Merge branch 'main' into filipi/lemonslice

# Conflicts:
#	README.md
#	uv.lock
This commit is contained in:
filipi87
2026-03-02 19:24:52 -03:00
351 changed files with 22154 additions and 4596 deletions

View File

@@ -1 +0,0 @@
- Fixed race condition where `RTVIObserver` could send messages before `DailyTransport` join completed. Outbound messages are now queued & delivered after the transport is ready.

View File

@@ -1 +0,0 @@
- Added `"timestampTransportStrategy": "ASYNC"` to `InworldAITTSService`. This allows timestamps info to trail audio chunks arrival, resulting in much better first audio chunk latency

View File

@@ -1 +0,0 @@
- Added model-specific `InputParams` to `RimeTTSService`: arcana params (`repetition_penalty`, `temperature`, `top_p`) and mistv2 params (`no_text_normalization`, `save_oovs`, `segment`). Model, voice, and param changes now trigger WebSocket reconnection.

View File

@@ -1 +0,0 @@
- ⚠️ `RimeTTSService` now defaults to `model="arcana"` and the `wss://users-ws.rime.ai/ws3` endpoint. `InputParams` defaults changed from mistv2-specific values to `None` — only explicitly-set params are sent as query params.

View File

@@ -1,3 +0,0 @@
- `AICFilter` now shares read-only AIC models via a singleton `AICModelManager` in `aic_filter.py`.
- Multiple filters using the same model path or `(model_id, model_download_dir)` share one loaded model, with reference counting and concurrent load deduplication.
- Model file I/O runs off the event loop so the filter does not block.

1
changelog/3696.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `TextAggregationMetricsData` metric measuring the time from the first LLM token to the first complete sentence, representing the latency cost of sentence aggregation in the TTS pipeline.

View File

@@ -0,0 +1 @@
- Added `text_aggregation_mode` parameter to `TTSService` and all TTS subclasses with a new `TextAggregationMode` enum (`SENTENCE`, `TOKEN`). All text now flows through text aggregators regardless of mode, enabling pattern detection and tag handling in TOKEN mode.

View File

@@ -0,0 +1 @@
- ⚠️ Deprecated `aggregate_sentences` parameter on `TTSService` and all TTS subclasses. Use `text_aggregation_mode=TextAggregationMode.SENTENCE` or `text_aggregation_mode=TextAggregationMode.TOKEN` instead.

View File

@@ -1 +0,0 @@
- Fixed async generator cleanup in OpenAI LLM streaming to prevent `AttributeError` with uvloop on Python 3.12+ (MagicStack/uvloop#699).

View File

@@ -1 +0,0 @@
- Added `X-User-Agent` and `X-Request-Id` headers to `InworldTTSService` for better traceability.

View File

@@ -1 +0,0 @@
- Fixed `SmallWebRTCTransport` input audio resampling to properly handle all sample rates, including 8kHz audio.

19
changelog/3714.added.md Normal file
View File

@@ -0,0 +1,19 @@
- Added support for using strongly-typed objects instead of dicts for updating service settings at runtime.
Instead of, say:
```python
await task.queue_frame(
STTUpdateSettingsFrame(settings={"language": Language.ES})
)
```
you'd do:
```python
await task.queue_frame(
STTUpdateSettingsFrame(delta=DeepgramSTTSettings(language=Language.ES))
)
```
Each service now vends strongly-typed classes like `DeepgramSTTSettings` representing the service's runtime-updatable settings.

View File

@@ -0,0 +1 @@
- ⚠️ Refactored runtime-updatable service settings to use strongly-typed classes (`TTSSettings`, `STTSettings`, `LLMSettings`, and service-specific subclasses) instead of plain dicts. Each service's `_settings` now holds these strongly-typed objects. For service maintainers, see changes in COMMUNITY_INTEGRATIONS.md.

View File

@@ -0,0 +1 @@
- Dict-based `*UpdateSettingsFrame(settings={...})` is deprecated in favor of passing typed settings delta objects with `*UpdateSettingsFrame(delta={...})`.

View File

@@ -0,0 +1,3 @@
- Deprecated `set_model()`, `set_voice()`, and `set_language()` on AI services in favor of runtime updates via `TTSUpdateSettingsFrame`, `STTUpdateSettingsFrame`, and `LLMUpdateSettingsFrame`.
⚠️ Note, too, a subtle behavior change in these deprecated methods. Whereas previously only `set_language()` caused the service to actually react to the update (e.g. by reconnecting to a remote service so it an pick up the change), now all these methods do. This change was made as part of a refactor making them all work the same way under the hood.

View File

@@ -1 +0,0 @@
- Fixed a race condition in `RTVIObserver` where bot output messages could be sent before the bot-started-speaking event.

View File

@@ -1 +0,0 @@
- Added `write_transport_frame()` hook to `BaseOutputTransport` allowing transport subclasses to handle custom frame types that flow through the audio queue.

View File

@@ -1 +0,0 @@
- Added `DailySIPTransferFrame` and `DailySIPReferFrame` to the Daily transport. These frames queue SIP transfer and SIP REFER operations with audio, so the operation executes only after the bot finishes its current utterance.

View File

@@ -1 +0,0 @@
- `DailyUpdateRemoteParticipantsFrame` is no longer deprecated and is now queued with audio like other transport frames.

View File

@@ -1 +0,0 @@
- Fixed Grok Realtime `session.updated` event parsing failure caused by the API returning prefixed voice names (e.g. `"human_Ara"` instead of `"Ara"`).

View File

@@ -1 +0,0 @@
- Bumped Pillow dependency upper bound from `<12` to `<13` to allow Pillow 12.x.

View File

@@ -1 +0,0 @@
- Fixed context ID reuse issue in `ElevenLabsTTSService`, `InworldTTSService`, `RimeTTSService`, `CartesiaTTSService`, `AsyncAITTSService`, and `PlayHTTTSService`. Services now properly reuse the same context ID across multiple `run_tts()` invocations within a single LLM turn, preventing context tracking issues and incorrect lifecycle signaling.

View File

@@ -1 +0,0 @@
- Fixed word timestamp interleaving issue in `ElevenLabsTTSService` when processing multiple sentences within a single LLM turn.

View File

@@ -1 +0,0 @@
- Added keepalive support to `SarvamSTTService` to prevent idle connection timeouts (e.g. when used behind a `ServiceSwitcher`).

View File

@@ -1 +0,0 @@
- Moved STT keepalive mechanism from `WebsocketSTTService` to the `STTService` base class, allowing any STT service (not just websocket-based ones) to use idle-connection keepalive via the `keepalive_timeout` and `keepalive_interval` parameters.

View File

@@ -1,3 +0,0 @@
- Improved audio context management in `AudioContextTTSService` by moving context ID tracking to the base class and adding `reuse_context_id_within_turn` parameter to control concurrent TTS request handling.
- Added helper methods: `has_active_audio_context()`, `get_active_audio_context_id()`, `remove_active_audio_context()`, `reset_active_audio_context()`
- Simplified Cartesia, ElevenLabs, Inworld, Rime, AsyncAI, and Gradium TTS implementations by removing duplicate context management code

View File

@@ -1 +0,0 @@
- Deprecated unused `Traceable`, `@traceable`, `@traced`, and `AttachmentStrategy` in `pipecat.utils.tracing.class_decorators`. This module will be removed in a future release.

View File

@@ -1 +0,0 @@
- Fixed tracing service decorators executing the wrapped function twice when the function itself raised an exception (e.g., LLM rate limit, TTS timeout).

View File

@@ -1 +0,0 @@
- Fixed `LLMUserAggregator` broadcasting mute events before `StartFrame` reaches downstream processors.

View File

@@ -1 +0,0 @@
- Fixed `UserIdleController` false idle triggers caused by gaps between user and bot activity frames. The idle timer now starts only after `BotStoppedSpeakingFrame` and is suppressed during active user turns and function calls.

View File

@@ -1 +0,0 @@
- Added `UserIdleTimeoutUpdateFrame` to enable or disable user idle detection at runtime by updating the timeout dynamically.

View File

@@ -1 +0,0 @@
- `UserIdleController` is now always created with a default timeout of 0 (disabled). The `user_idle_timeout` parameter changed from `Optional[float] = None` to `float = 0` in `UserTurnProcessor`, `LLMUserAggregatorParams`, and `UserIdleController`.

View File

@@ -1 +0,0 @@
- Change the version specifier from `>=0.2.8` to `~=0.2.8` for the `speechmatics-voice` package to ensure compatibility with future patch versions.

1
changelog/3764.added.md Normal file
View File

@@ -0,0 +1 @@
- Added support for specifying private endpoints for Azure Speech-to-Text, enabling use in private networks behind firewalls.

View File

@@ -1 +0,0 @@
- Updated `InworldTTSService` and `InworldHttpTTSService` to use `ASYNC` timestamp transport strategy by default

View File

@@ -1 +0,0 @@
- Fixed incorrect `sample_rate` assignment in `TavusInputTransport._on_participant_audio_data` (was using `audio.audio_frames` instead of `audio.sample_rate`).

View File

@@ -1 +0,0 @@
- Added `broadcast_sibling_id` field to the base `Frame` class. This field is automatically set by `broadcast_frame()` and `broadcast_frame_instance()` to the ID of the paired frame pushed in the opposite direction, allowing receivers to identify broadcast pairs.

View File

@@ -1 +0,0 @@
- Fixed `RTVIObserver` not processing upstream-only frames. Previously, all upstream frames were filtered out to avoid duplicate messages from broadcasted frames. Now only upstream copies of broadcasted frames are skipped.

View File

@@ -1 +0,0 @@
- Added `start_time` and `end_time` parameters to `start_ttfb_metrics()`, `stop_ttfb_metrics()`, `start_processing_metrics()`, and `stop_processing_metrics()` in `FrameProcessor` and `FrameProcessorMetrics`, allowing custom timestamps for metrics measurement. `STTService` now uses these instead of custom TTFB tracking.

View File

@@ -1 +0,0 @@
- Added `ignored_sources` parameter to `RTVIObserverParams` and `add_ignored_source()`/`remove_ignored_source()` methods to `RTVIObserver` to suppress RTVI messages from specific pipeline processors (e.g. a silent evaluation LLM).

View File

@@ -1 +0,0 @@
- Fixed mutable default arguments in `LLMContextAggregatorPair.__init__()` that could cause shared state across instances.

View File

@@ -1 +0,0 @@
- Fixed `DeepgramSageMakerSTTService` to properly track finalize lifecycle using `request_finalize()` / `confirm_finalize()` and use `is_final` (instead of `is_final and speech_final`) for final transcription detection, matching `DeepgramSTTService` behavior.

View File

@@ -1 +0,0 @@
- Added `DeepgramSageMakerTTSService` for running Deepgram TTS models deployed on AWS SageMaker endpoints via HTTP/2 bidirectional streaming. Supports the Deepgram TTS protocol (Speak, Flush, Clear, Close), interruption handling, and per-turn TTFB metrics.

View File

@@ -0,0 +1 @@
- Word timestamp support has been moved from `WordTTSService` into `TTSService` via a new `supports_word_timestamps` parameter. Services that previously extended `WordTTSService`, `AudioContextWordTTSService`, or `WebsocketWordTTSService` now pass `supports_word_timestamps=True` to their parent `__init__` instead.

View File

@@ -0,0 +1,5 @@
- Deprecated `WordTTSService`, `WebsocketWordTTSService`, `AudioContextWordTTSService`, and `InterruptibleWordTTSService`. Use their non-word counterparts with `supports_word_timestamps=True` instead:
- `WordTTSService``TTSService(supports_word_timestamps=True)`
- `WebsocketWordTTSService``WebsocketTTSService(supports_word_timestamps=True)`
- `AudioContextWordTTSService``AudioContextTTSService(supports_word_timestamps=True)`
- `InterruptibleWordTTSService``InterruptibleTTSService(supports_word_timestamps=True)`

View File

@@ -1 +0,0 @@
- Fixed a race condition in `AudioContextTTSService` where the audio context could time out between consecutive TTS requests within the same turn, causing audio to be discarded.

View File

@@ -1 +0,0 @@
- Fixed `push_interruption_task_frame_and_wait()` hanging indefinitely when the `InterruptionFrame` does not reach the pipeline sink within the timeout. Added a `timeout` keyword argument to customize the wait duration.

1
changelog/3794.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Added `LLMSpecificMessage` handling in `LLMContextSummarizationUtil` to skip provider-specific messages during context summarization.

1
changelog/3795.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Treated `response_cancel_not_active` as a non-fatal error in realtime services (`OpenAIRealtimeLLMService`, `GrokRealtimeLLMService`, `OpenAIRealtimeBetaLLMService`) to prevent WebSocket disconnection when cancelling an inactive response.

1
changelog/3803.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed Poetry compatibility by inlining `local-smart-turn-v3` dependencies (`transformers`, `onnxruntime`) into core dependencies instead of using a self-referential extra.

View File

@@ -0,0 +1 @@
- Removed `local-smart-turn-v3` optional extra from `pyproject.toml`. The `transformers` and `onnxruntime` packages are now always installed as core dependencies since they are required by the default turn stop strategy, `TurnAnalyzerUserTurnStopStrategy` which uses `LocalSmartTurnAnalyzerV3`.

1
changelog/3806.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `output_medium` parameter to `AgentInputParams` and `OneShotInputParams` in Ultravox service to control initial output medium (text or voice) at call creation time.

View File

@@ -0,0 +1 @@
- Improved Ultravox TTFB measurement accuracy by using VAD speech end time instead of `UserStoppedSpeakingFrame` timing.

View File

@@ -0,0 +1 @@
- Aligned `UltravoxRealtimeLLMService` frame handling with OpenAI/Gemini realtime services: added `InterruptionFrame` handling with metrics cleanup, processing metrics at response boundaries, and improved agent transcript handling for both voice and text output modalities.

View File

@@ -0,0 +1 @@
- Updated `OpenAIRealtimeLLMService` default model to `gpt-realtime-1.5`.

1
changelog/3808.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed `SentryMetrics` method signatures to match updated `FrameProcessorMetrics` base class, resolving `TypeError` when using `start_time`/`end_time` keyword arguments.

1
changelog/3809.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `TurnMetricsData` as a generic metrics class for turn detection, with e2e processing time measurement. `KrispVivaTurn` now emits `TurnMetricsData` with `e2e_processing_time_ms` tracking the interval from VAD speech-to-silence transition to turn completion.

View File

@@ -0,0 +1 @@
- Added `api_key` parameter to `KrispVivaSDKManager`, `KrispVivaTurn`, and `KrispVivaFilter` for Krisp SDK v1.6.1+ licensing. Falls back to `KRISP_VIVA_API_KEY` environment variable.

View File

@@ -0,0 +1 @@
- Deprecated `SmartTurnMetricsData` in favor of `TurnMetricsData`. `BaseSmartTurn` now emits `TurnMetricsData` directly.

View File

@@ -0,0 +1 @@
- Bumped `nltk` minimum version from 3.9.1 to 3.9.3 to resolve a security vulnerability.

1
changelog/3813.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed STT TTFB metrics not being reported for `SonioxSTTService` and `AWSTranscribeSTTService` due to missing `can_generate_metrics()` override.

1
changelog/3814.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `on_audio_context_interrupted()` and `on_audio_context_completed()` callbacks to `AudioContextTTSService`. Subclasses can override these to perform provider-specific cleanup instead of overriding `_handle_interruption()`.

1
changelog/3814.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed an issue where `AudioContextTTSService`-based providers (AsyncAI, ElevenLabs, Inworld, Rime) did not close or clean up their server-side audio contexts after normal speech completion, only on interruption.

View File

@@ -0,0 +1,4 @@
- `ServiceSettingsUpdateFrame`s are now `UninterruptibleFrame`s. Generally speaking, you don't want a user interruption to prevent a service setting change from going into effect. Note that you usually don't use `ServiceSettingsUpdateFrame` directly, you use one of its subclasses:
- `LLMUpdateSettingsFrame`
- `TTSUpdateSettingsFrame`
- `STTUpdateSettingsFrame`

1
changelog/3822.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed STT TTFB metrics measuring timeout expiry time instead of actual transcript arrival time.

1
changelog/3825.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed `InterimTranscriptionFrame` and `TranslationFrame` being unintentionally pushed downstream in `LLMUserAggregator`. They are now consumed like `TranscriptionFrame`.

1
changelog/3828.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed misleading "Empty audio frame received for STT service" warnings when using audio filters (e.g. `RNNoiseFilter`, `KrispVivaFilter`, `AICFilter`) that buffer audio internally.

1
changelog/3837.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed issues with `RimeNonJsonTTSService` where trailing punctuation is sometimes vocalized

View File

@@ -0,0 +1 @@
- ⚠️ Removed `PlayHTTTSService` and `PlayHTHttpTTSService`. PlayHT has been shut down and is no longer available.

1
changelog/3845.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed `TTSSpeakFrame` not committing spoken text to the conversation context when used outside of an LLM response (e.g., bot greetings or injected speech).

1
changelog/3850.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Removed verbose per-chunk audio logging from `GenesysAudioHookSerializer` that flooded production logs.

View File

@@ -0,0 +1 @@
- Added optional `llm` field to `LLMContextSummarizationConfig` for routing summarization to a dedicated LLM service (e.g., a cheaper/faster model) instead of the pipeline's primary model.

View File

@@ -0,0 +1 @@
- Added `summarization_timeout` to `LLMContextSummarizationConfig` (default 120s) to prevent hung LLM calls from permanently blocking future summarizations.

View File

@@ -0,0 +1 @@
- Added `on_summary_applied` event to `LLMContextSummarizer` for observability, providing message counts before and after context summarization.

1
changelog/3855.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `summary_message_template` to `LLMContextSummarizationConfig` for customizing how summaries are formatted when injected into context (e.g., wrapping in XML tags).

View File

@@ -0,0 +1 @@
- Updated context summarization to use `user` role instead of `assistant` for summary messages.

1
changelog/3857.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed `LocalSmartTurnAnalyzerV3` producing incorrect end-of-turn predictions at non-16kHz sample rates (e.g. 8kHz Twilio telephony) by adding automatic resampling to 16kHz before Whisper feature extraction.

View File

@@ -0,0 +1 @@
- Added `LLMContextSummaryConfig` (summary generation params: `target_context_tokens`, `min_messages_after_summary`, `summarization_prompt`) and `LLMAutoContextSummarizationConfig` (auto-trigger thresholds: `max_context_tokens`, `max_unsummarized_messages`, plus a nested `summary_config`). These replace the monolithic `LLMContextSummarizationConfig`.

1
changelog/3863.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `LLMSummarizeContextFrame` to trigger on-demand context summarization from anywhere in the pipeline (e.g. a function call tool). Accepts an optional `config: LLMContextSummaryConfig` to override summary generation settings per request.

View File

@@ -0,0 +1 @@
- ⚠️ Renamed `LLMAssistantAggregatorParams` fields: `enable_context_summarization``enable_auto_context_summarization` and `context_summarization_config``auto_context_summarization_config` (now accepts `LLMAutoContextSummarizationConfig`). The old names still work with a `DeprecationWarning` for one release cycle.

View File

@@ -0,0 +1 @@
- Deprecated `LLMContextSummarizationConfig`. Use `LLMAutoContextSummarizationConfig` with a nested `LLMContextSummaryConfig` instead. The old class emits a `DeprecationWarning`.

View File

@@ -0,0 +1 @@
- `ElevenLabsRealtimeSTTService` now sets `TranscriptionFrame.finalized` to `True` when using `CommitStrategy.MANUAL`.

1
changelog/3867.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed `PipelineTask` double-inserting `RTVIProcessor` into the frame chain when the user provides both an `RTVIProcessor` in the pipeline and a custom `RTVIObserver` subclass in observers.

View File

@@ -0,0 +1 @@
- Updated numba version pin from == to >=0.61.2

1
changelog/3873.added.md Normal file
View File

@@ -0,0 +1 @@
- Added support for the `speed_alpha` parameter to the `arcana` model in `RimeTTSService`.

View File

@@ -0,0 +1 @@
- Updated tracing code to use `ServiceSettings` dataclass API (`given_fields()`, attribute access) instead of dict-style access (`.items()`, `in`, subscript).

View File

@@ -0,0 +1 @@
- Added `ClientConnectedFrame`, a new `SystemFrame` pushed by all transports (Daily, LiveKit, FastAPI WebSocket, WebSocket Server, SmallWebRTC, HeyGen, Tavus) when a client connects. Enables observers to track transport readiness timing.

View File

@@ -0,0 +1 @@
Added `BotConnectedFrame` for SFU transports and `on_transport_timing_report` event to `StartupTimingObserver` with bot and client connection timing.

1
changelog/3881.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `StartupTimingObserver` for measuring how long each processor's `start()` method takes during pipeline startup. Also measures transport readiness — the time from `StartFrame` to first client connection — via the `on_transport_readiness_measured` event. Useful for diagnosing cold start slowness and identifying initialization bottlenecks.

1
changelog/3883.added.md Normal file
View File

@@ -0,0 +1 @@
- Added optional `direction` parameter to `PipelineTask.queue_frame()` and `PipelineTask.queue_frames()`, allowing frames to be pushed upstream from the end of the pipeline.

1
changelog/3886.other.md Normal file
View File

@@ -0,0 +1 @@
- Standardized Sarvam STT/TTS User-Agent header handling to consistently send Pipecat SDK identity in websocket requests.

1
changelog/3888.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed turn completion instructions being lost when `LLMMessagesUpdateFrame` replaces the LLM context. When `filter_incomplete_user_turns` is enabled, the turn completion system message is now re-injected after context replacement.

1
changelog/3893.fixed.md Normal file
View File

@@ -0,0 +1 @@
- Fixed Azure TTS and STT services silently swallowing cancellation errors (invalid API key, network failures, rate limiting) instead of propagating them as `ErrorFrame`s to the pipeline.

1
changelog/3896.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `broadcast_interruption()` to `FrameProcessor`. This method pushes an `InterruptionFrame` both upstream and downstream directly from the calling processor, avoiding the round-trip through the pipeline task that `push_interruption_task_frame_and_wait()` required.

View File

@@ -0,0 +1 @@
- ⚠️ Removed `event` field and `complete()` method from `InterruptionFrame`. Removed `event` field from `InterruptionTaskFrame`. These are no longer needed since `broadcast_interruption()` does not require a round-trip completion signal.

View File

@@ -0,0 +1 @@
- Deprecated `push_interruption_task_frame_and_wait()` in `FrameProcessor`. Use `broadcast_interruption()` instead. The old method now delegates to `broadcast_interruption()` and logs a deprecation warning.