Document user-turn-frame behavior in realtime service docstrings

Each realtime LLM service docstring now states whether the service emits
UserStartedSpeakingFrame / UserStoppedSpeakingFrame from server-side turn
signals, and what that implies for the rest of the pipeline.

For the services that don't (Gemini Live, AWS Nova Sonic, Ultravox), the
docstring spells out which downstream processors won't activate (RTVI
client speech events, TurnTrackingObserver, AudioBufferProcessor turn
recording, UserIdleController, user mute strategies, voicemail detector),
points at realtime_service_mode for correct context-write semantics, and
notes the option of wiring local VAD plus the caveat that locally-
generated turn boundaries are a heuristic that may not match the
provider's server-side turn decisions.

For the services that do (OpenAI Realtime, Inworld, Grok/xAI Realtime),
the docstring confirms turn frames are emitted from server VAD and
points at realtime_service_mode.
This commit is contained in:
Paul Kompfner
2026-05-20 15:50:32 -04:00
parent a00211627f
commit 20d9bf4af6
6 changed files with 53 additions and 0 deletions

View File

@@ -241,6 +241,17 @@ class AWSNovaSonicLLMService(LLMService[AWSNovaSonicLLMAdapter]):
Provides bidirectional audio streaming, real-time transcription, text generation,
and function calling capabilities using AWS Nova Sonic model.
Does NOT emit ``UserStartedSpeakingFrame`` / ``UserStoppedSpeakingFrame``,
so pipeline processors that depend on those frames — RTVI client
speech events, ``TurnTrackingObserver``, ``AudioBufferProcessor`` turn
recording, ``UserIdleController``, user mute strategies, voicemail
detector — won't activate with the default server-VAD-only setup. Pair
with ``LLMContextAggregatorPair(..., realtime_service_mode=RealtimeServiceModeConfig())``
so context writes are correct anyway. To produce the turn frames
locally, wire ``vad_analyzer=SileroVADAnalyzer()`` (or similar) into
``LLMUserAggregatorParams``; locally-generated turn boundaries are a
heuristic and may not match Nova Sonic's server-side turn decisions.
"""
Settings = AWSNovaSonicLLMSettings

View File

@@ -361,6 +361,18 @@ class GeminiLiveLLMService(LLMService[GeminiLLMAdapter]):
This service enables real-time conversations with Gemini, supporting both
text and audio modalities. It handles voice transcription, streaming audio
responses, and tool usage.
Does NOT emit ``UserStartedSpeakingFrame`` / ``UserStoppedSpeakingFrame``
(the API exposes an ``interrupted`` event but no turn-start/-end), so
pipeline processors that depend on those frames — RTVI client speech
events, ``TurnTrackingObserver``, ``AudioBufferProcessor`` turn
recording, ``UserIdleController``, user mute strategies, voicemail
detector — won't activate with the default server-VAD-only setup. Pair
with ``LLMContextAggregatorPair(..., realtime_service_mode=RealtimeServiceModeConfig())``
so context writes are correct anyway. To produce the turn frames
locally, see ``examples/realtime/realtime-gemini-live-local-vad.py``;
note that locally-generated turn boundaries are a heuristic and may
not match Gemini Live's server-side turn decisions.
"""
Settings = GeminiLiveLLMSettings

View File

@@ -201,6 +201,11 @@ class InworldRealtimeLLMService(LLMService[InworldRealtimeLLMAdapter]):
Supports function calling, conversation management, and real-time
transcription.
Emits ``UserStartedSpeakingFrame`` / ``UserStoppedSpeakingFrame`` from
Inworld's server-side VAD events. Pair with
``LLMContextAggregatorPair(..., realtime_service_mode=RealtimeServiceModeConfig())``
so context writes are decoupled from those frames.
Example::
llm = InworldRealtimeLLMService(

View File

@@ -204,6 +204,15 @@ class OpenAIRealtimeLLMService(LLMService[OpenAIRealtimeLLMAdapter]):
Implements the OpenAI Realtime API with WebSocket communication for low-latency
bidirectional audio and text interactions. Supports function calling, conversation
management, and real-time transcription.
Emits ``UserStartedSpeakingFrame`` / ``UserStoppedSpeakingFrame`` from
OpenAI's server-side VAD events, so pipeline processors that depend on
those frames (RTVI client speech events, ``TurnTrackingObserver``,
``AudioBufferProcessor`` turn recording, ``UserIdleController``, user
mute strategies, voicemail detector) work out of the box. Pair with
``LLMContextAggregatorPair(..., realtime_service_mode=RealtimeServiceModeConfig())``
so context writes are decoupled from those frames; see the
``examples/realtime/realtime-openai.py`` example.
"""
Settings = OpenAIRealtimeLLMSettings

View File

@@ -174,6 +174,17 @@ class UltravoxRealtimeLLMService(LLMService):
Note: Ultravox is an audio-native model, so voice transcriptions are not used
by the model and may not always align with its understanding of user input.
Does NOT emit ``UserStartedSpeakingFrame`` / ``UserStoppedSpeakingFrame``,
so pipeline processors that depend on those frames — RTVI client
speech events, ``TurnTrackingObserver``, ``AudioBufferProcessor`` turn
recording, ``UserIdleController``, user mute strategies, voicemail
detector — won't activate with the default server-VAD-only setup. Pair
with ``LLMContextAggregatorPair(..., realtime_service_mode=RealtimeServiceModeConfig())``
so context writes are correct anyway. To produce the turn frames
locally, wire ``vad_analyzer=SileroVADAnalyzer()`` (or similar) into
``LLMUserAggregatorParams``; locally-generated turn boundaries are a
heuristic and may not match Ultravox's server-side turn decisions.
"""
Settings = UltravoxRealtimeLLMSettings

View File

@@ -195,6 +195,11 @@ class GrokRealtimeLLMService(LLMService[GrokRealtimeLLMAdapter]):
- Built-in tools (web_search, x_search, file_search)
- Custom function calling
- Server-side VAD (Voice Activity Detection)
Emits ``UserStartedSpeakingFrame`` / ``UserStoppedSpeakingFrame`` from
Grok's server-side VAD events. Pair with
``LLMContextAggregatorPair(..., realtime_service_mode=RealtimeServiceModeConfig())``
so context writes are decoupled from those frames.
"""
Settings = GrokRealtimeLLMSettings