Show commented-out local-VAD opt-in in no-turn-frames examples
For services that don't emit UserStarted/StoppedSpeakingFrame (Nova Sonic, Gemini Live, Ultravox), the absence of those frames means downstream consumers — including the Pipecat Prebuilt UI — can't group user transcripts into discrete turns. The Tier 1 comment block already called this out, but the fix required users to know to add the SileroVADAnalyzer import + LLMUserAggregatorParams kwarg themselves. Make it a copy-paste: include the relevant imports and `user_params=` argument as commented-out code, with a comment explaining that they're not strictly necessary for context aggregation but enable RTVI / turn- dependent processors when needed. Mirror the wording used in the LLMService startup log. Also fix line wrapping in the llm_service.py startup log for the no- turn-frames case (manual edit to that message left the last line over- length).
This commit is contained in:
@@ -148,24 +148,30 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
|
|
||||||
# Set up context and context management.
|
# Set up context and context management.
|
||||||
#
|
#
|
||||||
# AWS Nova Sonic drives the conversation server-side. It does NOT emit
|
# AWS Nova Sonic drives the conversation server-side and does not emit
|
||||||
# UserStartedSpeakingFrame / UserStoppedSpeakingFrame, so pipeline
|
# UserStartedSpeakingFrame / UserStoppedSpeakingFrame. Context
|
||||||
# processors that depend on those frames — RTVI client speech events,
|
# aggregation still works with realtime_service_mode, but pipeline
|
||||||
|
# processors that depend on those frames (RTVI client speech events,
|
||||||
# TurnTrackingObserver, AudioBufferProcessor turn recording,
|
# TurnTrackingObserver, AudioBufferProcessor turn recording,
|
||||||
# UserIdleController, user mute strategies, voicemail detector — won't
|
# UserIdleController, user mute strategies, voicemail detector) won't
|
||||||
# activate with the default server-VAD-only setup. Context aggregation
|
# activate. The Pipecat Prebuilt UI is one such consumer — without
|
||||||
# still works with realtime_service_mode.
|
# these frames it can't group user transcripts into discrete turns
|
||||||
|
# visually.
|
||||||
#
|
#
|
||||||
# To produce these frames locally, wire a VAD analyzer (e.g.
|
# If you need those frames, uncomment the SileroVADAnalyzer import
|
||||||
# SileroVADAnalyzer) into LLMUserAggregatorParams. Caveat: locally-
|
# above and the `user_params=` argument below. Note: local turn
|
||||||
# generated turn boundaries are a heuristic and may not match Nova
|
# detection may not match Nova Sonic's actual server-side turn
|
||||||
# Sonic's server-side turn decisions, which is what drives the
|
# decisions and can desynchronize in subtle ways.
|
||||||
# conversation; the two can drift apart in subtle ways especially
|
#
|
||||||
# around interruptions and overlapping speech.
|
# from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
|
# from pipecat.processors.aggregators.llm_response_universal import (
|
||||||
|
# LLMUserAggregatorParams,
|
||||||
|
# )
|
||||||
context = LLMContext(tools=tools)
|
context = LLMContext(tools=tools)
|
||||||
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
||||||
context,
|
context,
|
||||||
realtime_service_mode=RealtimeServiceModeConfig(),
|
realtime_service_mode=RealtimeServiceModeConfig(),
|
||||||
|
# user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Build the pipeline
|
# Build the pipeline
|
||||||
|
|||||||
@@ -131,22 +131,32 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
|||||||
llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation)
|
llm.register_function("get_restaurant_recommendation", fetch_restaurant_recommendation)
|
||||||
|
|
||||||
context = LLMContext()
|
context = LLMContext()
|
||||||
# Gemini Live drives the conversation server-side. It does NOT emit
|
# Gemini Live drives the conversation server-side and does not emit
|
||||||
# UserStartedSpeakingFrame / UserStoppedSpeakingFrame, so pipeline
|
# UserStartedSpeakingFrame / UserStoppedSpeakingFrame. Context
|
||||||
# processors that depend on those frames — RTVI client speech events,
|
# aggregation still works with realtime_service_mode, but pipeline
|
||||||
|
# processors that depend on those frames (RTVI client speech events,
|
||||||
# TurnTrackingObserver, AudioBufferProcessor turn recording,
|
# TurnTrackingObserver, AudioBufferProcessor turn recording,
|
||||||
# UserIdleController, user mute strategies, voicemail detector — won't
|
# UserIdleController, user mute strategies, voicemail detector) won't
|
||||||
# activate with the default server-VAD-only setup. Context aggregation
|
# activate. The Pipecat Prebuilt UI is one such consumer — without
|
||||||
# still works with realtime_service_mode.
|
# these frames it can't group user transcripts into discrete turns
|
||||||
|
# visually.
|
||||||
#
|
#
|
||||||
# To produce these frames locally, see `realtime-gemini-live-local-vad.py`.
|
# If you need those frames, uncomment the SileroVADAnalyzer import
|
||||||
# Caveat: locally-generated turn boundaries are a heuristic and may not
|
# above and the `user_params=` argument below. Note: local turn
|
||||||
# match Gemini Live's server-side turn decisions, which is what drives the
|
# detection may not match Gemini Live's actual server-side turn
|
||||||
# conversation; the two can drift apart in subtle ways especially around
|
# decisions and can desynchronize in subtle ways.
|
||||||
# interruptions and overlapping speech.
|
#
|
||||||
|
# For local VAD driving the conversation (server VAD disabled), see
|
||||||
|
# `realtime-gemini-live-local-vad.py` instead.
|
||||||
|
#
|
||||||
|
# from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
|
# from pipecat.processors.aggregators.llm_response_universal import (
|
||||||
|
# LLMUserAggregatorParams,
|
||||||
|
# )
|
||||||
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
||||||
context,
|
context,
|
||||||
realtime_service_mode=RealtimeServiceModeConfig(),
|
realtime_service_mode=RealtimeServiceModeConfig(),
|
||||||
|
# user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
|
||||||
)
|
)
|
||||||
|
|
||||||
pipeline = Pipeline(
|
pipeline = Pipeline(
|
||||||
|
|||||||
@@ -175,23 +175,29 @@ There is also a secret menu that changes daily. If the user asks about it, use t
|
|||||||
|
|
||||||
context = LLMContext([])
|
context = LLMContext([])
|
||||||
|
|
||||||
# Ultravox drives the conversation server-side. It does NOT emit
|
# Ultravox drives the conversation server-side and does not emit
|
||||||
# UserStartedSpeakingFrame / UserStoppedSpeakingFrame, so pipeline
|
# UserStartedSpeakingFrame / UserStoppedSpeakingFrame. Context
|
||||||
# processors that depend on those frames — RTVI client speech events,
|
# aggregation still works with realtime_service_mode, but pipeline
|
||||||
|
# processors that depend on those frames (RTVI client speech events,
|
||||||
# TurnTrackingObserver, AudioBufferProcessor turn recording,
|
# TurnTrackingObserver, AudioBufferProcessor turn recording,
|
||||||
# UserIdleController, user mute strategies, voicemail detector — won't
|
# UserIdleController, user mute strategies, voicemail detector) won't
|
||||||
# activate with this default setup. Context aggregation still works
|
# activate. The Pipecat Prebuilt UI is one such consumer — without
|
||||||
# with realtime_service_mode.
|
# these frames it can't group user transcripts into discrete turns
|
||||||
|
# visually.
|
||||||
#
|
#
|
||||||
# To produce these frames locally, wire a VAD analyzer (e.g.
|
# If you need those frames, uncomment the SileroVADAnalyzer import
|
||||||
# SileroVADAnalyzer) into LLMUserAggregatorParams. Caveat: locally-
|
# above and the `user_params=` argument below. Note: local turn
|
||||||
# generated turn boundaries are a heuristic and may not match
|
# detection may not match Ultravox's actual server-side turn
|
||||||
# Ultravox's server-side turn decisions, which is what drives the
|
# decisions and can desynchronize in subtle ways.
|
||||||
# conversation; the two can drift apart in subtle ways especially
|
#
|
||||||
# around interruptions and overlapping speech.
|
# from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
|
# from pipecat.processors.aggregators.llm_response_universal import (
|
||||||
|
# LLMUserAggregatorParams,
|
||||||
|
# )
|
||||||
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
||||||
context,
|
context,
|
||||||
realtime_service_mode=RealtimeServiceModeConfig(),
|
realtime_service_mode=RealtimeServiceModeConfig(),
|
||||||
|
# user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Build the pipeline
|
# Build the pipeline
|
||||||
|
|||||||
@@ -410,9 +410,9 @@ class LLMService(UserTurnCompletionLLMServiceMixin, AIService, Generic[TAdapter]
|
|||||||
"AudioBufferProcessor turn recording, UserIdleController, user "
|
"AudioBufferProcessor turn recording, UserIdleController, user "
|
||||||
"mute strategies, voicemail detector) will not activate. To "
|
"mute strategies, voicemail detector) will not activate. To "
|
||||||
"produce them locally, add `vad_analyzer=` to "
|
"produce them locally, add `vad_analyzer=` to "
|
||||||
"LLMUserAggregatorParams. Note: local turn detection is a "
|
"LLMUserAggregatorParams. Note: local turn detection may not "
|
||||||
"heuristic; its boundaries may not match the provider's actual "
|
"match the provider's actual server-side turn decisions and "
|
||||||
"server-side turn decisions and can desynchronize in subtle ways."
|
"can desynchronize in subtle ways."
|
||||||
)
|
)
|
||||||
|
|
||||||
async def stop(self, frame: EndFrame):
|
async def stop(self, frame: EndFrame):
|
||||||
|
|||||||
Reference in New Issue
Block a user