Refactor delayed-transcript machinery; standardize vocabulary

Splits ``_maybe_emit_user_turn_stopped`` into three focused methods —
``_flush_user_message_to_context`` (push aggregation, return content +
timestamp), ``_finalize_user_turn`` (default-mode flow, emits both
events), and ``_finalize_delayed_user_message`` (delayed-mode flow,
emits only ``on_user_turn_message_finalized``). Fixes a side-issue
where ``on_user_turn_stopped`` could fire from non-end-of-turn paths
in delayed-transcript mode; that event now has a single origin (the
end-of-turn handler).

Standardizes vocabulary across docstrings and comments:

- "Default mode" / "Delayed-transcript mode" (with
  ``_expect_delayed_transcripts == False/True``)
- "End of turn" (not "audible stop" or "audible end of turn")
- "User message finalization" (the moment user-text is flushed to
  context + ``on_user_turn_message_finalized`` fires)
- "Pending finalization" (the in-between state in delayed mode)
- Transcripts (plural — the aggregator combines multiple per turn)

The timer that triggers user message finalization is no longer
described as a "backstop" — it's the sole trigger for finalization
in delayed-transcript mode, not a fallback. Renamed accordingly:
``_pending_finalization_task``, ``_pending_finalization_handler``,
``_run_pending_finalization``, ``_discard_pending_finalization``.

Adds a separate message class for the two events:
``UserTurnStoppedMessage.content`` is now ``str | None`` (``None``
at end-of-turn in delayed-transcript mode), and a new
``UserMessageFinalizedMessage`` carries the always-populated
``content`` for the finalization event.
This commit is contained in:
Paul Kompfner
2026-05-18 09:55:11 -04:00
parent 4479a3a6af
commit 8330c3487d
5 changed files with 224 additions and 146 deletions

View File

@@ -1 +1 @@
- Added the configuration surface to drive a realtime service like Gemini Live from local turn detection without paying user-transcript latency. Cascaded pipelines wait for a transcript before ending the user's turn because the downstream LLM needs the user's words recorded in context — but that wait is pure latency in pipelines using local turn detection to drive a realtime service, which consumes user audio directly. Set `wait_for_transcript_to_end_user_turn=False` on `LLMUserAggregatorParams` to turn this on: the aggregator drops `TranscriptionUserTurnStartStrategy` from the start strategies (so late-arriving realtime transcripts don't trigger new turns), sets `wait_for_transcript=False` on any stop strategies that support it (so the turn ends without waiting for a transcript), fires `on_user_turn_stopped` on the audible end of the turn (with empty content, since the transcript hasn't arrived), and defers the context flush until the transcript arrives. A new `on_user_turn_message_finalized` event fires when the user's message has been written to context — in the default mode it coincides with `on_user_turn_stopped`; in the delayed-transcript mode it fires later. Consumers that want the populated transcript should subscribe to `on_user_turn_message_finalized`. See `examples/realtime/realtime-gemini-live-local-vad.py` for the full pattern. - Added the configuration surface to drive a realtime service like Gemini Live from local turn detection without paying user-transcript latency. Cascaded pipelines wait for transcripts before ending the user's turn because the downstream LLM needs the user's words recorded in context — but that wait is pure latency in pipelines using local turn detection to drive a realtime service, which consumes user audio directly. Set `wait_for_transcript_to_end_user_turn=False` on `LLMUserAggregatorParams` to turn this on: the aggregator drops `TranscriptionUserTurnStartStrategy` from the start strategies (so late-arriving realtime transcripts don't trigger new turns), sets `wait_for_transcript=False` on any stop strategies that support it (so the turn ends without waiting for transcripts), fires `on_user_turn_stopped` at the end of turn (with empty content, since no transcripts have arrived yet), and defers the context flush until after a short wait that gives the realtime service time to emit its transcripts. A new `on_user_turn_message_finalized` event fires when the user's message has been written to context — in the default mode it coincides with `on_user_turn_stopped`; in the delayed-transcript mode it fires later. Consumers that want the populated transcript should subscribe to `on_user_turn_message_finalized`. See `examples/realtime/realtime-gemini-live-local-vad.py` for the full pattern.

View File

@@ -20,6 +20,7 @@ from pipecat.processors.aggregators.llm_response_universal import (
AssistantTurnStoppedMessage, AssistantTurnStoppedMessage,
LLMContextAggregatorPair, LLMContextAggregatorPair,
LLMUserAggregatorParams, LLMUserAggregatorParams,
UserMessageFinalizedMessage,
UserTurnStoppedMessage, UserTurnStoppedMessage,
) )
from pipecat.runner.types import RunnerArguments from pipecat.runner.types import RunnerArguments
@@ -72,19 +73,19 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
) )
# `wait_for_transcript_to_end_user_turn=False` configures the user # `wait_for_transcript_to_end_user_turn=False` configures the user
# aggregator for realtime services like Gemini Live that emit user # aggregator for realtime services like Gemini Live that emit user
# transcripts after the audible end of the turn. With this flag the # transcripts after the end of turn. With this flag the aggregator:
# aggregator:
# #
# - drops `TranscriptionUserTurnStartStrategy` from the default start # - drops `TranscriptionUserTurnStartStrategy` from the default start
# strategies (so late-arriving realtime transcripts don't trigger # strategies (so late-arriving realtime transcripts don't trigger
# new turns), # new turns),
# - sets `wait_for_transcript=False` on the default stop strategy # - sets `wait_for_transcript=False` on the default stop strategy
# (so the turn ends without waiting for a transcript), # (so the turn ends without waiting for transcripts),
# - fires `on_user_turn_stopped` on the audible end of the turn with # - fires `on_user_turn_stopped` at the end of turn with empty
# empty `message.content` (the transcript hasn't arrived yet), and # `message.content` (no transcripts have arrived yet), and
# - defers the context flush until the (late) transcript arrives, then # - defers the context flush until after a short wait that gives the
# emits `on_user_turn_message_finalized` with the populated message # realtime service time to emit its transcripts, then emits
# so the user's words land in the LLM context for audit/history. # `on_user_turn_message_finalized` with the populated message so
# the user's words land in the LLM context for audit/history.
user_aggregator, assistant_aggregator = LLMContextAggregatorPair( user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context, context,
user_params=LLMUserAggregatorParams( user_params=LLMUserAggregatorParams(
@@ -124,18 +125,20 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
await task.cancel() await task.cancel()
# With `wait_for_transcript_to_end_user_turn=False`, `on_user_turn_stopped` # With `wait_for_transcript_to_end_user_turn=False`, `on_user_turn_stopped`
# fires on the audible end of the turn (before the transcript arrives), so # fires at the end of turn (before any transcripts arrive), so its
# its `message.content` is empty. Logged here to make the timing of the # `message.content` is empty. Logged here to make the timing of the
# audible-stop signal visible alongside the later transcript event. # end-of-turn signal visible alongside the later finalization event.
@user_aggregator.event_handler("on_user_turn_stopped") @user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage): async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
logger.info(f"User turn ended (audible, strategy: {type(strategy).__name__})") logger.info(f"User turn ended (strategy: {type(strategy).__name__})")
# `on_user_turn_message_finalized` fires when the user transcript has # `on_user_turn_message_finalized` fires when the user message has
# been written to context — later than `on_user_turn_stopped` in this # been written to context — later than `on_user_turn_stopped` in this
# mode, since transcripts arrive after the audible turn end. # mode, since transcripts arrive after the end of turn.
@user_aggregator.event_handler("on_user_turn_message_finalized") @user_aggregator.event_handler("on_user_turn_message_finalized")
async def on_user_turn_message_finalized(aggregator, strategy, message: UserTurnStoppedMessage): async def on_user_turn_message_finalized(
aggregator, strategy, message: UserMessageFinalizedMessage
):
timestamp = f"[{message.timestamp}] " if message.timestamp else "" timestamp = f"[{message.timestamp}] " if message.timestamp else ""
line = f"{timestamp}user: {message.content}" line = f"{timestamp}user: {message.content}"
logger.info(f"Transcript: {line}") logger.info(f"Transcript: {line}")

View File

@@ -130,15 +130,16 @@ class LLMUserAggregatorParams:
vad_analyzer: Voice Activity Detection analyzer instance. vad_analyzer: Voice Activity Detection analyzer instance.
wait_for_transcript_to_end_user_turn: Defaults to True. Set to wait_for_transcript_to_end_user_turn: Defaults to True. Set to
False when using local turn detection to drive a realtime False when using local turn detection to drive a realtime
service (e.g. Gemini Live), where waiting for the transcript service (e.g. Gemini Live), where waiting for transcripts
before ending the turn is pure latency. When False: before ending the turn is pure latency. When False:
- the user turn ends as soon as the user stops speaking, - the user turn ends as soon as the user stops speaking,
without waiting for transcripts: ``on_user_turn_stopped`` without waiting for transcripts: ``on_user_turn_stopped``
fires immediately with empty content fires immediately with empty content
- the user-text flush to context and the - user message finalization (the user-text flush to context
``on_user_turn_message_finalized`` event are deferred and the ``on_user_turn_message_finalized`` event) is
until a backstop timer expires deferred and runs after a short wait, giving the realtime
service time to emit its transcripts
As part of configuring this behavior, the aggregator drops As part of configuring this behavior, the aggregator drops
``TranscriptionUserTurnStartStrategy`` from start strategies ``TranscriptionUserTurnStartStrategy`` from start strategies
@@ -278,13 +279,40 @@ class LLMAssistantAggregatorParams:
@dataclass @dataclass
class UserTurnStoppedMessage: class UserTurnStoppedMessage:
"""A user turn stopped message containing a user transcript update. """A message accompanying ``on_user_turn_stopped`` (end of user turn).
A message in a conversation transcript containing the user content. This is In default mode (``wait_for_transcript_to_end_user_turn=True``), the
the aggregated transcript that is then used in the context. user message is finalized at the end of the turn, so ``content``
carries the aggregated transcript. In delayed-transcript mode
(``wait_for_transcript_to_end_user_turn=False``), transcripts arrive
later, so ``content`` is ``None`` here — subscribe to
``on_user_turn_message_finalized`` for the populated message.
Parameters: Parameters:
content: The message content/text. content: The aggregated user transcript, or ``None`` if no
transcripts had arrived when end of turn fired (delayed-
transcript mode).
timestamp: When the user turn started.
user_id: Optional identifier for the user.
"""
content: str | None
timestamp: str
user_id: str | None = None
@dataclass
class UserMessageFinalizedMessage:
"""A message accompanying ``on_user_turn_message_finalized``.
Fired when the user message has been finalized into the context.
In default mode this coincides with ``on_user_turn_stopped``; in
delayed-transcript mode it fires later, once the realtime service
has emitted its transcripts. ``content`` is always populated.
Parameters:
content: The aggregated user transcript.
timestamp: When the user turn started. timestamp: When the user turn started.
user_id: Optional identifier for the user. user_id: Optional identifier for the user.
@@ -545,19 +573,23 @@ class LLMUserAggregator(LLMContextAggregator):
Event handlers available: Event handlers available:
- on_user_turn_started: Called when the user turn starts - on_user_turn_started: Called when the user turn starts.
- on_user_turn_stopped: Called when the user turn ends. With - on_user_turn_stopped: Called at the end of turn, with a
``wait_for_transcript_to_end_user_turn=True`` (default), this fires ``UserTurnStoppedMessage``. In default mode
with the populated ``UserTurnStoppedMessage.content``. With (``wait_for_transcript_to_end_user_turn=True``)
``wait_for_transcript_to_end_user_turn=False`` (realtime + local ``message.content`` carries the aggregated transcript. In
turn detection), this fires on the audible end of the turn — before delayed-transcript mode
the transcript has arrived — so ``content`` is empty; subscribe to (``wait_for_transcript_to_end_user_turn=False``) the user's
``on_user_turn_message_finalized`` to get the transcript. transcripts haven't arrived yet, so ``message.content`` is
- on_user_turn_message_finalized: Called once the user message has ``None``; subscribe to ``on_user_turn_message_finalized`` to get
been finalized into the context. Fires in both modes — in the the assembled message.
default mode it coincides with ``on_user_turn_stopped``; in the - on_user_turn_message_finalized: Called at user message
delayed-transcript path it fires later, when the transcript finalization — when the user message has been flushed to the
arrives. The ``UserTurnStoppedMessage.content`` is always populated. context — with a ``UserMessageFinalizedMessage``. In default
mode it coincides with ``on_user_turn_stopped``; in
delayed-transcript mode it fires a short time after, once the
realtime service has had time to emit its transcripts.
``message.content`` is always populated.
- on_user_turn_stop_timeout: Called when no user turn stop strategy triggers - on_user_turn_stop_timeout: Called when no user turn stop strategy triggers
- on_user_turn_idle: Called when the user has been idle for the configured timeout - on_user_turn_idle: Called when the user has been idle for the configured timeout
- on_user_mute_started: Called when the user becomes muted - on_user_mute_started: Called when the user becomes muted
@@ -574,7 +606,7 @@ class LLMUserAggregator(LLMContextAggregator):
... ...
@aggregator.event_handler("on_user_turn_message_finalized") @aggregator.event_handler("on_user_turn_message_finalized")
async def on_user_turn_message_finalized(aggregator, strategy: BaseUserTurnStopStrategy, message: UserTurnStoppedMessage): async def on_user_turn_message_finalized(aggregator, strategy: BaseUserTurnStopStrategy, message: UserMessageFinalizedMessage):
... ...
@aggregator.event_handler("on_user_turn_stop_timeout") @aggregator.event_handler("on_user_turn_stop_timeout")
@@ -662,14 +694,14 @@ class LLMUserAggregator(LLMContextAggregator):
# inferences fire before finalization. # inferences fire before finalization.
self._full_user_turn_aggregation: str | None = None self._full_user_turn_aggregation: str | None = None
# State for the deferred-finalization path that activates when # Pending-finalization state used in delayed-transcript mode
# `wait_for_transcript_to_end_user_turn=False`. The strategy fires # (`_expect_delayed_transcripts == True`): the end-of-turn event
# turn-stopped on the audible end of the user's turn; we defer the # `on_user_turn_stopped` has fired (with empty content) and user
# context flush and `on_user_turn_stopped` event until the transcript # message finalization is scheduled to run after the
# arrives (or a backstop timer fires). # pending-finalization timer expires.
self._pending_stop_strategy: BaseUserTurnStopStrategy | None = None self._pending_stop_strategy: BaseUserTurnStopStrategy | None = None
self._pending_inference_trigger: bool = False self._pending_inference_trigger: bool = False
self._delayed_transcript_backstop_task: asyncio.Task | None = None self._pending_finalization_task: asyncio.Task | None = None
self._user_turn_controller = UserTurnController( self._user_turn_controller = UserTurnController(
user_turn_strategies=user_turn_strategies, user_turn_strategies=user_turn_strategies,
@@ -715,12 +747,17 @@ class LLMUserAggregator(LLMContextAggregator):
@property @property
def _expect_delayed_transcripts(self) -> bool: def _expect_delayed_transcripts(self) -> bool:
"""Internal alias for the deferred-finalization mode. """True in delayed-transcript mode, False in default mode.
Active whenever ``wait_for_transcript_to_end_user_turn`` is False, In delayed-transcript mode the end of turn and user message
which is also when the strategy-mutation bundle is applied. The two finalization happen at different times: ``on_user_turn_stopped``
always travel together — there's no configuration where one is fires immediately at the end of turn (with empty content), and
active without the other. user message finalization is scheduled to run after the
pending-finalization timer expires, giving the realtime service
time to emit its transcripts.
Internal alias for ``wait_for_transcript_to_end_user_turn=False``.
Always travels with the strategy-mutation bundle applied at init.
""" """
return not self._params.wait_for_transcript_to_end_user_turn return not self._params.wait_for_transcript_to_end_user_turn
@@ -879,17 +916,20 @@ class LLMUserAggregator(LLMContextAggregator):
await self._cleanup() await self._cleanup()
async def _finalize_on_session_end(self): async def _finalize_on_session_end(self):
"""Flush any pending turn-stop on session end. """Flush any pending user message on session end.
If a deferred turn finalization is pending (because If user message finalization is pending (delayed-transcript mode,
``expect_delayed_transcripts`` is True and the transcript hadn't transcript hadn't arrived yet), replay the pending finalization
arrived yet), replay it first so the user message is captured before so the user message is captured before the session shuts down.
the session shuts down. Otherwise, run the mode-appropriate finalize path on whatever's
currently in the buffer.
""" """
if self._pending_stop_strategy is not None or self._pending_inference_trigger: if self._pending_stop_strategy is not None or self._pending_inference_trigger:
await self._replay_deferred_turn_finalization(on_session_end=True) await self._run_pending_finalization(on_session_end=True)
elif self._expect_delayed_transcripts:
await self._finalize_delayed_user_message(on_session_end=True)
else: else:
await self._maybe_emit_user_turn_stopped(on_session_end=True) await self._finalize_user_turn(on_session_end=True)
async def _cleanup(self): async def _cleanup(self):
if self._vad_controller: if self._vad_controller:
@@ -1021,16 +1061,17 @@ class LLMUserAggregator(LLMContextAggregator):
): ):
logger.debug(f"{self}: User started speaking (strategy: {strategy})") logger.debug(f"{self}: User started speaking (strategy: {strategy})")
# Precondition guard for `expect_delayed_transcripts`: if the previous # Precondition guard for delayed-transcript mode: if the previous
# turn's finalization is still pending, the precondition (turn N's # turn's user message finalization is still pending, the
# transcript arrives before turn N+1 starts) has been violated. # precondition (turn N's transcripts arrive before turn N+1
# Force-flush the previous turn before proceeding. # starts) has been violated. Force-finalize the previous turn
# before proceeding.
if self._pending_stop_strategy is not None or self._pending_inference_trigger: if self._pending_stop_strategy is not None or self._pending_inference_trigger:
logger.warning( logger.warning(
f"{self}: user turn started while previous turn's transcript was " f"{self}: user turn started while previous turn's transcript was "
f"still pending; flushing previous turn now" f"still pending; flushing previous turn now"
) )
await self._replay_deferred_turn_finalization() await self._run_pending_finalization()
self._user_turn_start_timestamp = time_now_iso8601() self._user_turn_start_timestamp = time_now_iso8601()
self._full_user_turn_aggregation = None self._full_user_turn_aggregation = None
@@ -1053,8 +1094,9 @@ class LLMUserAggregator(LLMContextAggregator):
logger.debug(f"{self}: User turn inference triggered (strategy: {strategy})") logger.debug(f"{self}: User turn inference triggered (strategy: {strategy})")
if self._expect_delayed_transcripts: if self._expect_delayed_transcripts:
# Defer the push_aggregation and event emission until the # Defer the push_aggregation and event emission; they'll run
# transcript arrives (or the backstop fires). # alongside user message finalization when the
# pending-finalization timer expires.
self._pending_inference_trigger = True self._pending_inference_trigger = True
return return
@@ -1083,60 +1125,67 @@ class LLMUserAggregator(LLMContextAggregator):
): ):
logger.debug(f"{self}: User stopped speaking (strategy: {strategy})") logger.debug(f"{self}: User stopped speaking (strategy: {strategy})")
# Audible-stop side effects fire on the strategy event regardless of # End-of-turn side effects always fire on the strategy event,
# whether the transcript-ready finalization is deferred. # regardless of whether user message finalization is deferred.
if params.enable_user_speaking_frames: if params.enable_user_speaking_frames:
await self.broadcast_frame(UserStoppedSpeakingFrame) await self.broadcast_frame(UserStoppedSpeakingFrame)
await self._user_idle_controller.process_frame(UserStoppedSpeakingFrame()) await self._user_idle_controller.process_frame(UserStoppedSpeakingFrame())
if self._expect_delayed_transcripts: if self._expect_delayed_transcripts:
# Fire `on_user_turn_stopped` now to signal the audible end of the # Delayed-transcript mode: fire `on_user_turn_stopped` now
# turn. `content` is empty because the transcript hasn't arrived # for the end of turncontent is empty because no
# yet — consumers that want the transcript subscribe to # transcripts have arrived yet. User message finalization
# `on_user_turn_message_finalized` instead, which fires later when # is scheduled to run when the pending-finalization timer
# the transcript lands (or the backstop timer fires). # expires; consumers wanting the assembled message subscribe
audible_stop_message = UserTurnStoppedMessage( # to `on_user_turn_message_finalized`.
content="", timestamp=self._user_turn_start_timestamp end_of_turn_message = UserTurnStoppedMessage(
content=None, timestamp=self._user_turn_start_timestamp
) )
await self._call_event_handler("on_user_turn_stopped", strategy, audible_stop_message) await self._call_event_handler("on_user_turn_stopped", strategy, end_of_turn_message)
self._pending_stop_strategy = strategy self._pending_stop_strategy = strategy
self._delayed_transcript_backstop_task = self.create_task( self._pending_finalization_task = self.create_task(
self._delayed_transcript_backstop_handler(DEFAULT_TTFS_P99), self._pending_finalization_handler(DEFAULT_TTFS_P99),
f"{self}::delayed_transcript_backstop", f"{self}::pending_finalization",
) )
return return
await self._maybe_emit_user_turn_stopped(strategy) await self._finalize_user_turn(strategy)
async def _delayed_transcript_backstop_handler(self, timeout: float): async def _pending_finalization_handler(self, timeout: float):
"""Backstop timer for `expect_delayed_transcripts`. """Pending-finalization timer for delayed-transcript mode.
Fires after ``timeout`` seconds and replays the deferred turn Waits ``timeout`` seconds — giving the realtime service time to
finalization with whatever transcript has been captured by then emit its transcripts — then runs the pending finalization with
(possibly nothing). whatever transcripts have been captured by then (possibly
nothing). Cancelled by reset / next-turn precondition guard /
session end.
""" """
try: try:
await asyncio.sleep(timeout) await asyncio.sleep(timeout)
except asyncio.CancelledError: except asyncio.CancelledError:
return return
finally: finally:
self._delayed_transcript_backstop_task = None self._pending_finalization_task = None
await self._replay_deferred_turn_finalization() await self._run_pending_finalization()
async def _replay_deferred_turn_finalization(self, *, on_session_end: bool = False): async def _run_pending_finalization(self, *, on_session_end: bool = False):
"""Replay the deferred halves of inference-triggered and turn-stopped. """Run the pending finalization for delayed-transcript mode.
Called from the backstop timer, the precondition guard in In delayed-transcript mode the end of turn fires
`_on_user_turn_started`, and the session-end paths. Clears all ``on_user_turn_stopped`` immediately and leaves user message
pending state before delegating to the normal finalization methods finalization (plus any pending inference-triggered segment)
so they run the unmodified code path. pending. This method runs that pending work: pushes the
accumulated user message to context and emits
``on_user_turn_message_finalized``. Called from the
pending-finalization timer (the normal path), the precondition
guard in ``_on_user_turn_started``, and the session-end paths.
""" """
if self._delayed_transcript_backstop_task: if self._pending_finalization_task:
await self.cancel_task(self._delayed_transcript_backstop_task) await self.cancel_task(self._pending_finalization_task)
self._delayed_transcript_backstop_task = None self._pending_finalization_task = None
pending_strategy = self._pending_stop_strategy pending_strategy = self._pending_stop_strategy
had_pending_inference = self._pending_inference_trigger had_pending_inference = self._pending_inference_trigger
@@ -1155,31 +1204,29 @@ class LLMUserAggregator(LLMContextAggregator):
await self._call_event_handler("on_user_turn_inference_triggered", pending_strategy) await self._call_event_handler("on_user_turn_inference_triggered", pending_strategy)
if pending_strategy is not None or on_session_end: if pending_strategy is not None or on_session_end:
# `on_user_turn_stopped` already fired on the audible stop; # `on_user_turn_stopped` already fired at the end of turn;
# only fire `on_user_turn_message_finalized` here. # this is the deferred user message finalization.
await self._maybe_emit_user_turn_stopped( await self._finalize_delayed_user_message(
pending_strategy, pending_strategy, on_session_end=on_session_end
on_session_end=on_session_end,
emit_stopped_event=False,
) )
async def _on_reset_aggregation( async def _on_reset_aggregation(
self, controller: UserTurnController, strategy: BaseUserTurnStartStrategy self, controller: UserTurnController, strategy: BaseUserTurnStartStrategy
): ):
logger.debug(f"{self}: Resetting aggregation (strategy: {strategy})") logger.debug(f"{self}: Resetting aggregation (strategy: {strategy})")
await self._discard_deferred_turn_finalization() await self._discard_pending_finalization()
await self.reset() await self.reset()
async def _discard_deferred_turn_finalization(self): async def _discard_pending_finalization(self):
"""Drop any pending deferred turn finalization without replaying. """Drop pending finalization state without running it.
Called from reset paths (interruption, explicit reset). "Reset" means Called from reset paths (interruption, explicit reset). "Reset"
"throw it away" — we don't want to flush a partial transcript that means "throw it away" — we don't flush a partial transcript that
was about to be invalidated anyway. was about to be invalidated anyway.
""" """
if self._delayed_transcript_backstop_task: if self._pending_finalization_task:
await self.cancel_task(self._delayed_transcript_backstop_task) await self.cancel_task(self._pending_finalization_task)
self._delayed_transcript_backstop_task = None self._pending_finalization_task = None
self._pending_stop_strategy = None self._pending_stop_strategy = None
self._pending_inference_trigger = False self._pending_inference_trigger = False
@@ -1189,32 +1236,22 @@ class LLMUserAggregator(LLMContextAggregator):
async def _on_user_turn_idle(self, controller): async def _on_user_turn_idle(self, controller):
await self._call_event_handler("on_user_turn_idle") await self._call_event_handler("on_user_turn_idle")
async def _maybe_emit_user_turn_stopped( async def _flush_user_message_to_context(
self, self, on_session_end: bool = False
strategy: BaseUserTurnStopStrategy | None = None, ) -> tuple[str, str] | None:
on_session_end: bool = False, """Push the aggregated user message to context, return ``(content, timestamp)``.
*,
emit_stopped_event: bool = True,
):
"""Finalize the user message and emit the corresponding events.
Earlier inference triggers in the same turn have already pushed Earlier inference triggers in the same turn already pushed their
their segments to the context and accumulated them into segments to the context and accumulated them in
``self._full_user_turn_aggregation``. Any aggregation that ``self._full_user_turn_aggregation``; whatever arrived after the
arrived after the last inference trigger is flushed here so last inference trigger is flushed here so end-of-turn content is
end-of-turn content is never lost from the public events. never lost.
Always emits ``on_user_turn_message_finalized``. Also emits Returns ``(content, timestamp)`` for the just-finalized user
``on_user_turn_stopped`` when ``emit_stopped_event=True`` (the message, or ``None`` when there's no content to flush and
default path); the delayed-transcript path passes False because it ``on_session_end=True`` (avoids emitting empty events during
already fired ``on_user_turn_stopped`` on the audible stop. session shutdown). Callers construct the appropriate message
dataclass for each event they emit.
Args:
strategy: The strategy that triggered the turn stop.
on_session_end: If True, only emit if there's unemitted content
(avoids duplicate events when session ends).
emit_stopped_event: Whether to fire ``on_user_turn_stopped``
alongside ``on_user_turn_message_finalized``.
""" """
segment = await self.push_aggregation() segment = await self.push_aggregation()
full_aggregation = self._full_user_turn_aggregation full_aggregation = self._full_user_turn_aggregation
@@ -1225,14 +1262,51 @@ class LLMUserAggregator(LLMContextAggregator):
else: else:
content = full_aggregation or segment content = full_aggregation or segment
if not on_session_end or content: if on_session_end and not content:
message = UserTurnStoppedMessage( return None
content=content, timestamp=self._user_turn_start_timestamp
) timestamp = self._user_turn_start_timestamp
if emit_stopped_event: self._user_turn_start_timestamp = ""
await self._call_event_handler("on_user_turn_stopped", strategy, message) return content, timestamp
await self._call_event_handler("on_user_turn_message_finalized", strategy, message)
self._user_turn_start_timestamp = "" async def _finalize_user_turn(
self,
strategy: BaseUserTurnStopStrategy | None = None,
on_session_end: bool = False,
):
"""Finalize the user turn: flush the message, emit both events.
Used in default mode (``_expect_delayed_transcripts == False``),
where end of turn and user message finalization coincide. Emits
both ``on_user_turn_stopped`` and ``on_user_turn_message_finalized``.
"""
result = await self._flush_user_message_to_context(on_session_end=on_session_end)
if result is None:
return
content, timestamp = result
stopped_msg = UserTurnStoppedMessage(content=content, timestamp=timestamp)
finalized_msg = UserMessageFinalizedMessage(content=content, timestamp=timestamp)
await self._call_event_handler("on_user_turn_stopped", strategy, stopped_msg)
await self._call_event_handler("on_user_turn_message_finalized", strategy, finalized_msg)
async def _finalize_delayed_user_message(
self,
strategy: BaseUserTurnStopStrategy | None = None,
on_session_end: bool = False,
):
"""Finalize the user message: flush to context, emit one event.
Used in delayed-transcript mode (``_expect_delayed_transcripts ==
True``), where user message finalization fires after the end of
turn. Emits ``on_user_turn_message_finalized`` only;
``on_user_turn_stopped`` was already emitted at the end of turn.
"""
result = await self._flush_user_message_to_context(on_session_end=on_session_end)
if result is None:
return
content, timestamp = result
finalized_msg = UserMessageFinalizedMessage(content=content, timestamp=timestamp)
await self._call_event_handler("on_user_turn_message_finalized", strategy, finalized_msg)
class LLMAssistantAggregator(LLMContextAggregator): class LLMAssistantAggregator(LLMContextAggregator):

View File

@@ -48,9 +48,10 @@ class SpeechTimeoutUserTurnStopStrategy(BaseUserTurnStopStrategy):
doesn't need the user's words recorded in context to respond — typically doesn't need the user's words recorded in context to respond — typically
when using local turn detection to drive a realtime service like Gemini when using local turn detection to drive a realtime service like Gemini
Live. In that case the strategy fires the turn stop as soon as the Live. In that case the strategy fires the turn stop as soon as the
user_speech_timeout elapses, without waiting for a transcript. Pair with user_speech_timeout elapses, without waiting for transcripts. Pair with
``expect_delayed_transcripts=True`` on ``LLMUserAggregatorParams`` so the ``wait_for_transcript_to_end_user_turn=False`` on
aggregator still captures the transcript when it arrives. ``LLMUserAggregatorParams`` so the aggregator still captures transcripts
when they arrive.
""" """
def __init__( def __init__(

View File

@@ -47,10 +47,10 @@ class TurnAnalyzerUserTurnStopStrategy(BaseUserTurnStopStrategy):
doesn't need the user's words recorded in context to respond — typically doesn't need the user's words recorded in context to respond — typically
when using local turn detection to drive a realtime service like Gemini when using local turn detection to drive a realtime service like Gemini
Live. In that case the strategy fires the turn stop as soon as the Live. In that case the strategy fires the turn stop as soon as the
analyzer concludes the turn is complete, without waiting for a analyzer concludes the turn is complete, without waiting for transcripts.
transcript. Pair with ``expect_delayed_transcripts=True`` on Pair with ``wait_for_transcript_to_end_user_turn=False`` on
``LLMUserAggregatorParams`` so the aggregator still captures the ``LLMUserAggregatorParams`` so the aggregator still captures transcripts
transcript when it arrives. when they arrive.
""" """
def __init__( def __init__(