yukiobata1
32c6dccebe
Add word-level timestamp support to Azure TTS with cumulative PTS fix
...
This commit adds word boundary support to AzureTTSService and fixes
the race condition that causes scrambled TTS output across multiple
sentences.
## Features Added
- Change AzureTTSService to inherit from WordTTSService
- Subscribe to Azure SDK's synthesis_word_boundary event
- Emit word-level text with timing information via _words_queue
- Add synthesis lock for sequential sentence processing
## Race Condition Fix
Previously, each sentence's word boundary timestamps reset to 0,
causing downstream components to interleave words when reordering
frames by PTS. This resulted in scrambled output like:
'Hello ! I What am questions AI have assistant...'
The fix adds cumulative audio offset tracking to ensure monotonically
increasing PTS across all sentences:
Sentence 1: pts = 0.1s, 0.5s, 0.8s (cumulative at end: 0.8s)
Sentence 2: pts = 0.9s, 1.2s, 1.5s (0.8s + relative offset)
## Key Changes
- _cumulative_audio_offset: tracks total audio duration
- _handle_word_boundary: adds cumulative offset to timestamps
- _handle_completed: accumulates audio duration for next sentence
- flush_audio: resets cumulative offset at end of LLM response
- _handle_interruption: resets state on user interruption
- run_tts: uses synthesis lock for sequential processing
Fixes #2918
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2025-12-31 18:49:48 +09:00
Aleix Conchillo Flaqué
66a9dc70c7
LLMUserAggregator: fix turn strategies renaming
2025-12-30 20:59:48 -08:00
Aleix Conchillo Flaqué
846ca500d3
turns: update old turn_start_strategies deprecations
2025-12-30 19:50:10 -08:00
Aleix Conchillo Flaqué
eb5a797b12
turns: rename bot turn start to user turn stop strategies
2025-12-30 14:33:58 -08:00
Aleix Conchillo Flaqué
fb9a772e33
Merge pull request #3319 from pipecat-ai/aleix/openaillmcontext-backwards-compatibility
...
BaseInputTransport: fix OpenAILLMContext backwards compatibility
2025-12-30 09:35:43 -08:00
Aleix Conchillo Flaqué
4630e76942
ExternalUserTurnStartStrategy: disable interruptions
2025-12-30 09:32:31 -08:00
Aleix Conchillo Flaqué
4dba9ea329
BaseInputTransport: fix OpenAILLMContext backwards compatibility
2025-12-30 09:32:31 -08:00
Mark Backman
9c6d0f1be1
Merge pull request #3322 from gui217/fix/rnnoise_filter_handle_empty_audio
...
Fix/rnnoise filter handle empty audio
2025-12-30 08:39:42 -05:00
gui217
bad8f8aa51
Fix rnnoise filter to handle empty audio
2025-12-30 13:32:36 +02:00
Aleix Conchillo Flaqué
0a0ab51cc7
LLMUserAggregator: also read deprecated allow_interruptions
2025-12-29 17:57:18 -08:00
Aleix Conchillo Flaqué
ad4c22cf44
Merge pull request #3316 from pipecat-ai/aleix/llm-user-aggreagtor-enable-interruptions
...
turns(user): add support for enabling/disabling interruptions
2025-12-29 17:45:56 -08:00
Aleix Conchillo Flaqué
9fe99ed880
add and update changelog entries
2025-12-29 17:35:10 -08:00
Aleix Conchillo Flaqué
ffbbb1b3f5
turns(user): add support for enabling/disabling interruptions
2025-12-29 17:00:03 -08:00
Aleix Conchillo Flaqué
e22a6c9e4d
Merge pull request #3305 from omChauhanDev/fix_unregister_function
...
fix: missing key access by adding existence check
2025-12-29 14:52:41 -08:00
Aleix Conchillo Flaqué
09e79149ea
Merge pull request #3310 from omChauhanDev/fix-task-manager
...
fix: preserve asyncio.Task return value in create_task
2025-12-29 14:43:24 -08:00
Aleix Conchillo Flaqué
bd9a316d7a
transports: don't use interruptions_allowed to avoid deprecation warning
2025-12-29 14:40:00 -08:00
Aleix Conchillo Flaqué
c8f47b4b22
turns: add UserTurnStartedParams and BotTurnStartedParams
2025-12-29 14:32:08 -08:00
Mark Backman
c28ed2206c
DeepgramSTTService pushes user started/stopped speaking and interruption frames
2025-12-29 10:17:35 -08:00
Mark Backman
30e6a33930
Update VoicemailDetector to use ExternalTurnStartStrategies
2025-12-29 10:17:35 -08:00
Aleix Conchillo Flaqué
46db8e58d6
LLMUserAggregator: fix backwards compatibility with ExternalTurnStartStrategies
2025-12-29 10:17:35 -08:00
Aleix Conchillo Flaqué
c821e9f8fd
turns: add external user and bot turn start strategies
...
External strategies are strategies where the logic for user turn start and turn
end come from a different processors (e.g. an STT).
2025-12-29 10:17:35 -08:00
Mark Backman
01ce06c756
Merge pull request #3288 from pipecat-ai/mb/inworld-cleanup
...
Inworld TTS service clean up
2025-12-29 13:07:20 -05:00
Mark Backman
4bc490c843
Merge pull request #3289 from pipecat-ai/mb/audio-context-tts-service-base
...
Add AudioContextTTSService base class, update AudioContextWordTTSServ…
2025-12-29 13:05:06 -05:00
Mark Backman
7c0d897aa3
Merge pull request #3300 from omChauhanDev/nvidia-expose-use_ssl-param
...
exposed use_ssl param in nvidia services
2025-12-29 09:18:26 -05:00
Aleix Conchillo Flaqué
0e8e3afc85
Merge pull request #3307 from pipecat-ai/aleix/simplify-turns-package-imports
...
turns: simplify imports and don't require full strategy module path
2025-12-28 18:51:23 -08:00
Om Chauhan
a181e01310
fixed: create_task to return coroutine result
2025-12-29 07:46:15 +05:30
Aleix Conchillo Flaqué
5496aa722f
turns: simplify imports and don't require full strategy module path
2025-12-28 16:20:15 -08:00
Aleix Conchillo Flaqué
053f59ed6e
FrameProcessor: deprecated interruptions_allowed
2025-12-28 08:27:02 -08:00
Aleix Conchillo Flaqué
5b93fb9609
PipelineTask: deprecate allow_interruptions parameter
2025-12-28 08:27:02 -08:00
Aleix Conchillo Flaqué
192ede6e34
Merge pull request #3298 from pipecat-ai/aleix/push-user-started-speaking-first
...
push UserStartedSpeakingFrame before interruption
2025-12-28 08:24:50 -08:00
Aleix Conchillo Flaqué
8b861d9143
LLMUserAggregator: move turn_start_strategies from PipelineTask
2025-12-28 08:16:34 -08:00
Aleix Conchillo Flaqué
094d9fd7d7
turns(mute): make strategies available in __init__
2025-12-28 08:12:44 -08:00
Aleix Conchillo Flaqué
c7589663b5
deprecate STTMuteFilter in favor of LLMUSerAggregator user mute strategies
2025-12-28 08:12:44 -08:00
Om Chauhan
0f144f48cb
fix: missing key access by adding existence check
2025-12-28 10:28:37 +05:30
Aleix Conchillo Flaqué
a962c4eeba
STTMuteFilter: use FunctionCallsStartedFrame and support multiple function calls
2025-12-27 13:52:30 -08:00
Om Chauhan
604384b3ce
exposed use_ssl param
2025-12-25 20:09:42 +05:30
Aleix Conchillo Flaqué
260b7e7959
push UserStartedSpeakingFrame before interruption
2025-12-24 15:33:44 -08:00
Aleix Conchillo Flaqué
0abaae2f07
LLMUserAggregator: no need to reset strategies
...
Turn start strategies are already reset when triggered, so there's no need to
reset them again.
2025-12-24 11:08:43 -08:00
Aleix Conchillo Flaqué
30922d365f
minor turn start strategies cleanup
2025-12-24 11:08:43 -08:00
Aleix Conchillo Flaqué
c33c8d2195
LLMUserAggregator: add support for user mute strategies
2025-12-24 11:08:43 -08:00
Aleix Conchillo Flaqué
1d70275574
initial user mute strategies
2025-12-24 11:08:43 -08:00
Aleix Conchillo Flaqué
1f0357ae5e
LLMUserAggregator: add bot turn start strategies timeout fallback
2025-12-23 15:42:57 -08:00
Mark Backman
44a698cbcc
Add AudioContextTTSService base class, update AudioContextWordTTSService inheritance
2025-12-23 10:36:17 -05:00
Mark Backman
5038ebf205
Clean up _receive_messages to use WebsocketService class
2025-12-23 09:44:21 -05:00
Mark Backman
1da215f576
Inworld TTS service clean up
2025-12-23 09:24:29 -05:00
Aleix Conchillo Flaqué
40493e8ce8
Merge pull request #3286 from pipecat-ai/aleix/improve-turn-analyzer-bot-turn-start-strategy
...
improve turn analyzer bot turn start strategy
2025-12-22 21:46:48 -08:00
Aleix Conchillo Flaqué
4017bfa769
LLMUserAggregator: improve turn_analyzer warning
2025-12-22 21:44:49 -08:00
Aleix Conchillo Flaqué
480a9d092c
TurnAnalyzerBotTurnStartStrategy: make sure to use turn analyzer state
2025-12-22 16:29:48 -08:00
Aleix Conchillo Flaqué
b5fe1c9cd8
fix old interruption/speaking strategies docstrings
2025-12-22 16:19:25 -08:00
Mark Backman
49b53d72a9
Merge pull request #3276 from pipecat-ai/mb/grok-realtime-cleanup
...
GrokRealtimeLLMService cleanup
2025-12-22 18:13:23 -05:00
