realtime + local turn detection: drop the user-transcript wait
Add the configuration surface to drive a realtime service like Gemini Live from local turn detection without paying user-transcript latency. Cascaded pipelines wait for a transcript before ending the user's turn because the downstream LLM needs the user's words recorded in context — but that wait is pure latency in pipelines using local turn detection to drive a realtime service, which consumes user audio directly. Set `wait_for_transcript_to_end_user_turn=False` on `LLMUserAggregatorParams` to turn this on. With that single flag the aggregator: - drops `TranscriptionUserTurnStartStrategy` from the start strategies (so late-arriving realtime transcripts don't trigger new turns), - sets `wait_for_transcript=False` on any stop strategy that supports it (so the turn ends on the audible end of the turn, without waiting for a transcript), - fires `on_user_turn_stopped` on the audible end of the turn with empty `content` (since the transcript hasn't arrived), and - defers the context flush until the transcript arrives or a backstop timer fires. A new `on_user_turn_message_finalized` event fires when the user's message has been written to context. In the default mode it coincides with `on_user_turn_stopped`; in the delayed-transcript mode it fires later. Consumers that want the populated transcript should subscribe to `on_user_turn_message_finalized` — it's the event that always carries the user message, regardless of mode. Strategy mutations are logged: loudly when the user passed their own strategies (we're overwriting parts of their config), quietly otherwise. The strategy-level `wait_for_transcript` parameter on `TurnAnalyzerUserTurnStopStrategy` and `SpeechTimeoutUserTurnStopStrategy` remains exposed for advanced cases. The example `realtime-gemini-live-local-vad.py` demonstrates the full pattern.
This commit is contained in:
@@ -70,10 +70,26 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||
},
|
||||
],
|
||||
)
|
||||
# `wait_for_transcript_to_end_user_turn=False` configures the user
|
||||
# aggregator for realtime services like Gemini Live that emit user
|
||||
# transcripts after the audible end of the turn. With this flag the
|
||||
# aggregator:
|
||||
#
|
||||
# - drops `TranscriptionUserTurnStartStrategy` from the default start
|
||||
# strategies (so late-arriving realtime transcripts don't trigger
|
||||
# new turns),
|
||||
# - sets `wait_for_transcript=False` on the default stop strategy
|
||||
# (so the turn ends without waiting for a transcript),
|
||||
# - fires `on_user_turn_stopped` on the audible end of the turn with
|
||||
# empty `message.content` (the transcript hasn't arrived yet), and
|
||||
# - defers the context flush until the (late) transcript arrives, then
|
||||
# emits `on_user_turn_message_finalized` with the populated message
|
||||
# so the user's words land in the LLM context for audit/history.
|
||||
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
||||
context,
|
||||
user_params=LLMUserAggregatorParams(
|
||||
vad_analyzer=SileroVADAnalyzer(),
|
||||
wait_for_transcript_to_end_user_turn=False,
|
||||
),
|
||||
)
|
||||
|
||||
@@ -107,8 +123,19 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||
logger.info(f"Client disconnected")
|
||||
await task.cancel()
|
||||
|
||||
# With `wait_for_transcript_to_end_user_turn=False`, `on_user_turn_stopped`
|
||||
# fires on the audible end of the turn (before the transcript arrives), so
|
||||
# its `message.content` is empty. Logged here to make the timing of the
|
||||
# audible-stop signal visible alongside the later transcript event.
|
||||
@user_aggregator.event_handler("on_user_turn_stopped")
|
||||
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
|
||||
logger.info(f"User turn ended (audible, strategy: {type(strategy).__name__})")
|
||||
|
||||
# `on_user_turn_message_finalized` fires when the user transcript has
|
||||
# been written to context — later than `on_user_turn_stopped` in this
|
||||
# mode, since transcripts arrive after the audible turn end.
|
||||
@user_aggregator.event_handler("on_user_turn_message_finalized")
|
||||
async def on_user_turn_message_finalized(aggregator, strategy, message: UserTurnStoppedMessage):
|
||||
timestamp = f"[{message.timestamp}] " if message.timestamp else ""
|
||||
line = f"{timestamp}user: {message.content}"
|
||||
logger.info(f"Transcript: {line}")
|
||||
|
||||
Reference in New Issue
Block a user