wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add changelog fragments for realtime service mode

Browse Source 
Fragments use the +<name> prefix so they show up under "Unreleased"
without a PR-number suffix; rename to <PR#>.<type>.md before merge.
This commit is contained in:
Paul Kompfner
2026-05-20 15:09:38 -04:00
parent 1fe8cf5289
commit 11d7fcf174
6 changed files with 6 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
changelog/+realtime-no-user-turn-frames-log.added.md Normal file
View File

@@ -0,0 +1 @@
- Added a startup INFO log on realtime LLM services that don't emit `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame` (Gemini Live, AWS Nova Sonic, Ultravox). The log spells out which downstream processors depend on those frames (RTVI client speech events, `TurnTrackingObserver`, `AudioBufferProcessor` turn recording, `UserIdleController`, user mute strategies, voicemail detector) and how to opt into local VAD when needed.

1
changelog/+realtime-service-metadata-frame.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `RealtimeServiceMetadataFrame`, broadcast at pipeline start by realtime LLM services (OpenAI Realtime, Azure Realtime, Inworld, Grok/xAI Realtime, Gemini Live, AWS Nova Sonic, Ultravox). The context aggregator pair listens for it and, when `realtime_service_mode` isn't configured, logs a one-time INFO recommendation pointing users at the option and the `on_user_turn_stopped` timing change it implies.

1
changelog/+realtime-service-mode-config.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `RealtimeServiceModeConfig` and a new `realtime_service_mode` kwarg on `LLMContextAggregatorPair`, opting the pair into realtime (speech-to-speech) LLM behavior. When set, user messages are written to context when the assistant response starts rather than on user-turn-end frames — so context stays correct even when the realtime service emits no turn frames at all — and, by default, turn-end strategies stop waiting for transcripts before signalling end-of-turn, keeping transcript latency off the critical path in local-VAD-driven realtime pipelines. Both behaviors are individually controllable via the `context_writes_await_turns` and `turns_await_transcripts` fields. Cascade (non-realtime) behavior is unchanged when the kwarg is omitted.

1
changelog/+realtime-service-mode-events.added.md Normal file
View File

@@ -0,0 +1 @@
- Added `on_user_message_added` and `on_assistant_message_added` event handlers on `LLMUserAggregator` and `LLMAssistantAggregator`. Each fires when its respective message is flushed to context and carries the finalized content. In cascade mode they coincide with `on_user_turn_stopped` / `on_assistant_turn_stopped`; in realtime mode (where turn-stop fires before the message is finalized) they're the canonical way to subscribe to "context just updated, here's the text."

1
changelog/+user-turn-stopped-message-content-optional.changed.md Normal file
View File

@@ -0,0 +1 @@
- `UserTurnStoppedMessage.content` is now typed `str | None`. In realtime mode (`RealtimeServiceModeConfig(context_writes_await_turns=False)`) the user message isn't finalized at turn-stop time, so `content` is `None`; subscribers wanting the finalized text should use the new `on_user_message_added` event. Cascade behavior is unchanged.

1
changelog/+wait-for-transcript-stop-strategies.changed.md Normal file
View File

@@ -0,0 +1 @@
- `SpeechTimeoutUserTurnStopStrategy` and `TurnAnalyzerUserTurnStopStrategy` now accept a `wait_for_transcript: bool = True` kwarg. When set to `False`, the strategy signals end-of-turn as soon as VAD / the turn analyzer reports end-of-speech rather than waiting for a transcript — useful when local turn detection is the intended driver of a realtime conversation. `LLMContextAggregatorPair` flips this for you when `realtime_service_mode` is configured with the default `turns_await_transcripts=False`.