wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Broad service settings refactor, with the primary aim of making service settings discoverable and strongly-typed. Service settings can be updated at runtime with *UpdateSettingsFrames.

Browse Source 
Does not (yet) touch `InputParams`, to avoid scope creep and touching something currently part of the public API. But there is a lot of overlap between `*Settings` object fields and `InputParams` fields.

Other than discoverability/typing, these are some other improvements brought by this refactor:
- There is now a single code path (see `_update_settings_from_typed`) where services can respond to settings changes (by, say, reconnecting if needed), improving maintainability and guaranteeing one and only one reconnection no matter which settings changed
- `set_language`/`set_model`/`set_voice`—which we're assuming are usable as public methods, though *not* recommended over `*UpdateSettingsFrame`—all use the same code path as settings updates. They're also now all consistent in that, if a service needs to respond to a change (by, say, reconnecting if needed), any of these methods will kick off that process. Note that this is technically a behavior change.
- Several services now properly react to changed settings by reconnecting:
  - `AWSTranscribeSTTService`
  - `AzureSTTService`
  - `SonioxSTTService`
  - `GladiaSTTService`
  - `SpeechmaticsSTTService`
  - `AssemblyAISTTService`
  - `CartesiaSTTService`
  - `FishAudioTTSService` (would previously only reconnect when `model` changed)
  - `GoogleSTTService`
  - `SpeechmaticsSTTService` (which previously only handled *some* settings updates through a nonstandard public `update_params` method)
  - `GradiumSTTService`
  - `NvidiaSegmentedSTTService` (which previously only handled changes to language)
- Bookkeeping across various services has been reduced, mostly by deduping ivars; the `self._settings` ivar is treated as the source of truth

NOTE: I pretty much guarantee that there are services missed in this PR in terms of bringing to consistency with how updates are handled (like whether changes in certain fields trigger reconnects when they need to). We can squash remaining inconsistencies as we stumble onto them, service by service. The goal here is to get things *mostly* in order, and establish the infrastructure and patterns we'll need going forward.
This commit is contained in:
Paul Kompfner
2026-02-11 10:50:11 -05:00
parent 0574167fbd
commit 8a4ab611be
69 changed files with 3943 additions and 1481 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.claude/skills/cleanup/SKILL.md
View File

@@ -293,7 +293,7 @@ class NewTTSService(TTSService):
""" """
super().__init__(**kwargs) super().__init__(**kwargs)
self._api_key = api_key self._api_key = api_key
self.set_voice(voice) self._voice_id = voice
``` ```
--- ---

2
examples/foundational/35-pattern-pair-voice-switching.py
View File

@@ -117,7 +117,7 @@ async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
# First flush any existing audio to finish the current context # First flush any existing audio to finish the current context
await tts.flush_audio() await tts.flush_audio()
# Then set the new voice # Then set the new voice
tts.set_voice(VOICE_IDS[voice_name]) await tts.set_voice(VOICE_IDS[voice_name])
logger.info(f"Switched to {voice_name} voice") logger.info(f"Switched to {voice_name} voice")
else: else:
logger.warning(f"Unknown voice: {voice_name}") logger.warning(f"Unknown voice: {voice_name}")

11
src/pipecat/frames/frames.py
View File

@@ -42,6 +42,7 @@ from pipecat.utils.utils import obj_count, obj_id
if TYPE_CHECKING: if TYPE_CHECKING:
from pipecat.processors.aggregators.llm_context import LLMContext, NotGiven from pipecat.processors.aggregators.llm_context import LLMContext, NotGiven
from pipecat.processors.frame_processor import FrameProcessor from pipecat.processors.frame_processor import FrameProcessor
from pipecat.services.settings import ServiceSettings
class DeprecatedKeypadEntry: class DeprecatedKeypadEntry:
@@ -2112,13 +2113,17 @@ class TTSStoppedFrame(ControlFrame):
class ServiceUpdateSettingsFrame(ControlFrame): class ServiceUpdateSettingsFrame(ControlFrame):
"""Base frame for updating service settings. """Base frame for updating service settings.
A control frame containing a request to update service settings. Supports both the legacy ``settings`` dict and the new typed ``update``
object. When both are provided, ``update`` takes precedence.
Parameters: Parameters:
settings: Dictionary of setting name to value mappings. settings: Dictionary of setting name to value mappings (legacy).
update: Typed :class:`~pipecat.services.settings.ServiceSettings`
object describing the delta to apply.
""" """
settings: Mapping[str, Any] settings: Mapping[str, Any] = field(default_factory=dict)
update: Optional["ServiceSettings"] = None
@dataclass @dataclass

41
src/pipecat/services/ai_service.py
View File

@@ -10,7 +10,7 @@ Provides the foundation for all AI services in the Pipecat framework, including
model management, settings handling, and frame processing lifecycle methods. model management, settings handling, and frame processing lifecycle methods.
""" """
from typing import Any, AsyncGenerator, Dict, Mapping from typing import Any, AsyncGenerator, Dict, Mapping, Set
from loguru import logger from loguru import logger
@@ -23,6 +23,7 @@ from pipecat.frames.frames import (
) )
from pipecat.metrics.metrics import MetricsData from pipecat.metrics.metrics import MetricsData
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.settings import ServiceSettings
class AIService(FrameProcessor): class AIService(FrameProcessor):
@@ -42,7 +43,7 @@ class AIService(FrameProcessor):
""" """
super().__init__(**kwargs) super().__init__(**kwargs)
self._model_name: str = "" self._model_name: str = ""
self._settings: Dict[str, Any] = {} self._settings: Dict[str, Any] | ServiceSettings = {}
self._session_properties: Dict[str, Any] = {} self._session_properties: Dict[str, Any] = {}
@property @property
@@ -135,6 +136,42 @@ class AIService(FrameProcessor):
else: else:
logger.warning(f"Unknown setting for {self.name} service: {key}") logger.warning(f"Unknown setting for {self.name} service: {key}")
async def _update_settings_from_typed(self, update: ServiceSettings) -> Set[str]:
"""Apply a typed settings update and return the set of changed field names.
If ``_settings`` is a :class:`ServiceSettings` object, the update is
applied to it and the changed-field set is returned. The ``model``
field is handled specially: when it changes, ``set_model_name`` is
called.
Services that have been migrated to typed settings should override
this method (calling ``super()``) to react to specific changed fields
(e.g. reconnect on voice change).
Args:
update: A typed settings delta.
Returns:
Set of field names whose values actually changed.
"""
if not isinstance(self._settings, ServiceSettings):
logger.warning(
f"{self.name}: received typed settings update but _settings "
f"is not a ServiceSettings — falling back to dict-based update"
)
await self._update_settings(update.to_dict())
return set()
changed = self._settings.apply_update(update)
if "model" in changed:
self.set_model_name(self._settings.model)
if changed:
logger.info(f"{self.name}: updated settings fields: {changed}")
return changed
async def process_frame(self, frame: Frame, direction: FrameDirection): async def process_frame(self, frame: Frame, direction: FrameDirection):
"""Process frames and handle service lifecycle. """Process frames and handle service lifecycle.

77
src/pipecat/services/anthropic/llm.py
View File

@@ -16,8 +16,8 @@ import copy
import io import io
import json import json
import re import re
from dataclasses import dataclass from dataclasses import dataclass, field
from typing import Any, Dict, List, Literal, Optional, Union from typing import Any, ClassVar, Dict, List, Literal, Optional, Union
import httpx import httpx
from loguru import logger from loguru import logger
@@ -42,7 +42,6 @@ from pipecat.frames.frames import (
LLMThoughtEndFrame, LLMThoughtEndFrame,
LLMThoughtStartFrame, LLMThoughtStartFrame,
LLMThoughtTextFrame, LLMThoughtTextFrame,
LLMUpdateSettingsFrame,
UserImageRawFrame, UserImageRawFrame,
) )
from pipecat.metrics.metrics import LLMTokenUsage from pipecat.metrics.metrics import LLMTokenUsage
@@ -59,6 +58,8 @@ from pipecat.processors.aggregators.openai_llm_context import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
from pipecat.services.settings import NOT_GIVEN as _NOT_GIVEN
from pipecat.services.settings import LLMSettings
from pipecat.utils.tracing.service_decorators import traced_llm from pipecat.utils.tracing.service_decorators import traced_llm
try: try:
@@ -69,6 +70,19 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class AnthropicLLMSettings(LLMSettings):
"""Typed settings for Anthropic LLM services.
Parameters:
enable_prompt_caching: Whether to enable prompt caching.
thinking: Extended thinking configuration.
"""
enable_prompt_caching: Any = field(default_factory=lambda: _NOT_GIVEN)
thinking: Any = field(default_factory=lambda: _NOT_GIVEN)
@dataclass @dataclass
class AnthropicContextAggregatorPair: class AnthropicContextAggregatorPair:
"""Pair of context aggregators for Anthropic conversations. """Pair of context aggregators for Anthropic conversations.
@@ -210,9 +224,10 @@ class AnthropicLLMService(LLMService):
self.set_model_name(model) self.set_model_name(model)
self._retry_timeout_secs = retry_timeout_secs self._retry_timeout_secs = retry_timeout_secs
self._retry_on_timeout = retry_on_timeout self._retry_on_timeout = retry_on_timeout
self._settings = { self._settings = AnthropicLLMSettings(
"max_tokens": params.max_tokens, model=model,
"enable_prompt_caching": ( max_tokens=params.max_tokens,
enable_prompt_caching=(
params.enable_prompt_caching params.enable_prompt_caching
if params.enable_prompt_caching is not None if params.enable_prompt_caching is not None
else ( else (
@@ -221,12 +236,12 @@ class AnthropicLLMService(LLMService):
else False else False
) )
), ),
"temperature": params.temperature, temperature=params.temperature,
"top_k": params.top_k, top_k=params.top_k,
"top_p": params.top_p, top_p=params.top_p,
"thinking": params.thinking, thinking=params.thinking,
"extra": params.extra if isinstance(params.extra, dict) else {}, extra=params.extra if isinstance(params.extra, dict) else {},
} )
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate usage metrics. """Check if this service can generate usage metrics.
@@ -280,7 +295,7 @@ class AnthropicLLMService(LLMService):
if isinstance(context, LLMContext): if isinstance(context, LLMContext):
adapter: AnthropicLLMAdapter = self.get_llm_adapter() adapter: AnthropicLLMAdapter = self.get_llm_adapter()
invocation_params = adapter.get_llm_invocation_params( invocation_params = adapter.get_llm_invocation_params(
context, enable_prompt_caching=self._settings["enable_prompt_caching"] context, enable_prompt_caching=self._settings.enable_prompt_caching
) )
messages = invocation_params["messages"] messages = invocation_params["messages"]
system = invocation_params["system"] system = invocation_params["system"]
@@ -294,20 +309,20 @@ class AnthropicLLMService(LLMService):
# Build params using the same method as streaming completions # Build params using the same method as streaming completions
params = { params = {
"model": self.model_name, "model": self.model_name,
"max_tokens": max_tokens if max_tokens is not None else self._settings["max_tokens"], "max_tokens": max_tokens if max_tokens is not None else self._settings.max_tokens,
"stream": False, "stream": False,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_k": self._settings["top_k"], "top_k": self._settings.top_k,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"messages": messages, "messages": messages,
"system": system, "system": system,
"tools": tools, "tools": tools,
"betas": ["interleaved-thinking-2025-05-14"], "betas": ["interleaved-thinking-2025-05-14"],
} }
if self._settings["thinking"]: if self._settings.thinking:
params["thinking"] = self._settings["thinking"].model_dump(exclude_unset=True) params["thinking"] = self._settings.thinking.model_dump(exclude_unset=True)
params.update(self._settings["extra"]) params.update(self._settings.extra)
# LLM completion # LLM completion
response = await self._client.beta.messages.create(**params) response = await self._client.beta.messages.create(**params)
@@ -358,14 +373,14 @@ class AnthropicLLMService(LLMService):
if isinstance(context, LLMContext): if isinstance(context, LLMContext):
adapter: AnthropicLLMAdapter = self.get_llm_adapter() adapter: AnthropicLLMAdapter = self.get_llm_adapter()
params = adapter.get_llm_invocation_params( params = adapter.get_llm_invocation_params(
context, enable_prompt_caching=self._settings["enable_prompt_caching"] context, enable_prompt_caching=self._settings.enable_prompt_caching
) )
return params return params
# Anthropic-specific context # Anthropic-specific context
messages = ( messages = (
context.get_messages_with_cache_control_markers() context.get_messages_with_cache_control_markers()
if self._settings["enable_prompt_caching"] if self._settings.enable_prompt_caching
else context.messages else context.messages
) )
return AnthropicLLMInvocationParams( return AnthropicLLMInvocationParams(
@@ -408,21 +423,21 @@ class AnthropicLLMService(LLMService):
params = { params = {
"model": self.model_name, "model": self.model_name,
"max_tokens": self._settings["max_tokens"], "max_tokens": self._settings.max_tokens,
"stream": True, "stream": True,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_k": self._settings["top_k"], "top_k": self._settings.top_k,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
} }
# Add thinking parameter if set # Add thinking parameter if set
if self._settings["thinking"]: if self._settings.thinking:
params["thinking"] = self._settings["thinking"].model_dump(exclude_unset=True) params["thinking"] = self._settings.thinking.model_dump(exclude_unset=True)
# Messages, system, tools # Messages, system, tools
params.update(params_from_context) params.update(params_from_context)
params.update(self._settings["extra"]) params.update(self._settings.extra)
# "Interleaved thinking" needed to allow thinking between sequences # "Interleaved thinking" needed to allow thinking between sequences
# of function calls, when extended thinking is enabled. # of function calls, when extended thinking is enabled.
@@ -576,11 +591,9 @@ class AnthropicLLMService(LLMService):
# NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal # NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal
# LLMContext with it # LLMContext with it
context = AnthropicLLMContext.from_messages(frame.messages) context = AnthropicLLMContext.from_messages(frame.messages)
elif isinstance(frame, LLMUpdateSettingsFrame):
await self._update_settings(frame.settings)
elif isinstance(frame, LLMEnablePromptCachingFrame): elif isinstance(frame, LLMEnablePromptCachingFrame):
logger.debug(f"Setting enable prompt caching to: [{frame.enable}]") logger.debug(f"Setting enable prompt caching to: [{frame.enable}]")
self._settings["enable_prompt_caching"] = frame.enable self._settings.enable_prompt_caching = frame.enable
else: else:
await self.push_frame(frame, direction) await self.push_frame(frame, direction)

60
src/pipecat/services/assemblyai/stt.py
View File

@@ -12,6 +12,7 @@ WebSocket API for streaming audio transcription.
import asyncio import asyncio
import json import json
from dataclasses import dataclass, field
from typing import Any, AsyncGenerator, Dict, Optional from typing import Any, AsyncGenerator, Dict, Optional
from urllib.parse import urlencode from urllib.parse import urlencode
@@ -29,6 +30,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import ASSEMBLYAI_TTFS_P99 from pipecat.services.stt_latency import ASSEMBLYAI_TTFS_P99
from pipecat.services.stt_service import WebsocketSTTService from pipecat.services.stt_service import WebsocketSTTService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -52,6 +54,19 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class AssemblyAISTTSettings(STTSettings):
"""Typed settings for the AssemblyAI STT service.
See :class:`AssemblyAIConnectionParams` for detailed parameter descriptions.
Parameters:
connection_params: Connection configuration parameters.
"""
connection_params: AssemblyAIConnectionParams = field(default_factory=lambda: NOT_GIVEN)
class AssemblyAISTTService(WebsocketSTTService): class AssemblyAISTTService(WebsocketSTTService):
"""AssemblyAI real-time speech-to-text service. """AssemblyAI real-time speech-to-text service.
@@ -96,9 +111,11 @@ class AssemblyAISTTService(WebsocketSTTService):
) )
self._api_key = api_key self._api_key = api_key
self._language = language self._settings: AssemblyAISTTSettings = AssemblyAISTTSettings(
language=language,
connection_params=connection_params,
)
self._api_endpoint_base_url = api_endpoint_base_url self._api_endpoint_base_url = api_endpoint_base_url
self._connection_params = connection_params
self._vad_force_turn_endpoint = vad_force_turn_endpoint self._vad_force_turn_endpoint = vad_force_turn_endpoint
self._termination_event = asyncio.Event() self._termination_event = asyncio.Event()
@@ -165,6 +182,35 @@ class AssemblyAISTTService(WebsocketSTTService):
""" """
return True return True
async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Apply a typed settings update and reconnect if anything changed.
Any change triggers a WebSocket reconnect since all connection
parameters are encoded in the WebSocket URL.
Args:
update: A :class:`STTSettings` (or ``AssemblyAISTTSettings``) delta.
Returns:
Set of field names whose values actually changed.
"""
changed = await super()._update_settings_from_typed(update)
if not changed:
return changed
# Re-apply manual turn mode config if vad_force_turn_endpoint is active
# and connection_params were updated.
if self._vad_force_turn_endpoint and "connection_params" in changed:
self._settings.connection_params = self._configure_manual_turn_mode(
self._settings.connection_params
)
await self._disconnect()
await self._connect()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the speech-to-text service. """Start the speech-to-text service.
@@ -239,7 +285,7 @@ class AssemblyAISTTService(WebsocketSTTService):
def _build_ws_url(self) -> str: def _build_ws_url(self) -> str:
"""Build WebSocket URL with query parameters using urllib.parse.urlencode.""" """Build WebSocket URL with query parameters using urllib.parse.urlencode."""
params = {} params = {}
for k, v in self._connection_params.model_dump().items(): for k, v in self._settings.connection_params.model_dump().items():
if v is not None: if v is not None:
if k == "keyterms_prompt": if k == "keyterms_prompt":
params[k] = json.dumps(v) params[k] = json.dumps(v)
@@ -415,18 +461,18 @@ class AssemblyAISTTService(WebsocketSTTService):
if not message.transcript: if not message.transcript:
return return
if message.end_of_turn and ( if message.end_of_turn and (
not self._connection_params.formatted_finals or message.turn_is_formatted not self._settings.connection_params.formatted_finals or message.turn_is_formatted
): ):
await self.push_frame( await self.push_frame(
TranscriptionFrame( TranscriptionFrame(
message.transcript, message.transcript,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._language, self._settings.language,
message, message,
) )
) )
await self._trace_transcription(message.transcript, True, self._language) await self._trace_transcription(message.transcript, True, self._settings.language)
await self.stop_processing_metrics() await self.stop_processing_metrics()
else: else:
await self.push_frame( await self.push_frame(
@@ -434,7 +480,7 @@ class AssemblyAISTTService(WebsocketSTTService):
message.transcript, message.transcript,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._language, self._settings.language,
message, message,
) )
) )

67
src/pipecat/services/asyncai/tts.py
View File

@@ -9,6 +9,7 @@
import asyncio import asyncio
import base64 import base64
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
import aiohttp import aiohttp
@@ -27,6 +28,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import AudioContextTTSService, TTSService from pipecat.services.tts_service import AudioContextTTSService, TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -72,6 +74,21 @@ def language_to_async_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class AsyncAITTSSettings(TTSSettings):
"""Typed settings for Async AI TTS services.
Parameters:
output_container: Audio container format (e.g. "raw").
output_encoding: Audio encoding format (e.g. "pcm_s16le").
output_sample_rate: Audio sample rate in Hz.
"""
output_container: str = field(default_factory=lambda: NOT_GIVEN)
output_encoding: str = field(default_factory=lambda: NOT_GIVEN)
output_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
class AsyncAITTSService(AudioContextTTSService): class AsyncAITTSService(AudioContextTTSService):
"""Async TTS service with WebSocket streaming. """Async TTS service with WebSocket streaming.
@@ -131,19 +148,21 @@ class AsyncAITTSService(AudioContextTTSService):
self._api_key = api_key self._api_key = api_key
self._api_version = version self._api_version = version
self._url = url self._url = url
self._settings = { self._settings: AsyncAITTSSettings = AsyncAITTSSettings(
"output_format": { model=model,
voice=voice_id,
output_format={
"container": container, "container": container,
"encoding": encoding, "encoding": encoding,
"sample_rate": 0, "sample_rate": 0,
}, },
"language": self.language_to_service_language(params.language) language=self.language_to_service_language(params.language)
if params.language if params.language
else None, else None,
} )
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice_id) self._voice_id = voice_id
self._receive_task = None self._receive_task = None
self._keepalive_task = None self._keepalive_task = None
@@ -179,7 +198,7 @@ class AsyncAITTSService(AudioContextTTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["output_format"]["sample_rate"] = self.sample_rate self._settings.output_sample_rate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -235,8 +254,12 @@ class AsyncAITTSService(AudioContextTTSService):
init_msg = { init_msg = {
"model_id": self._model_name, "model_id": self._model_name,
"voice": {"mode": "id", "id": self._voice_id}, "voice": {"mode": "id", "id": self._voice_id},
"output_format": self._settings["output_format"], "output_format": {
"language": self._settings["language"], "container": self._settings.output_container,
"encoding": self._settings.output_encoding,
"sample_rate": self._settings.output_sample_rate,
},
"language": self._settings.language,
} }
await self._get_websocket().send(json.dumps(init_msg)) await self._get_websocket().send(json.dumps(init_msg))
@@ -454,17 +477,17 @@ class AsyncAIHttpTTSService(TTSService):
self._api_key = api_key self._api_key = api_key
self._base_url = url self._base_url = url
self._api_version = version self._api_version = version
self._settings = { self._settings: AsyncAITTSSettings = AsyncAITTSSettings(
"output_format": { model=model,
"container": container, voice=voice_id,
"encoding": encoding, output_container=container,
"sample_rate": 0, output_encoding=encoding,
}, output_sample_rate=0,
"language": self.language_to_service_language(params.language) language=self.language_to_service_language(params.language)
if params.language if params.language
else None, else None,
} )
self.set_voice(voice_id) self._voice_id = voice_id
self.set_model_name(model) self.set_model_name(model)
self._session = aiohttp_session self._session = aiohttp_session
@@ -495,7 +518,7 @@ class AsyncAIHttpTTSService(TTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["output_format"]["sample_rate"] = self.sample_rate self._settings.output_sample_rate = self.sample_rate
@traced_tts @traced_tts
async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]: async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]:
@@ -517,8 +540,12 @@ class AsyncAIHttpTTSService(TTSService):
"model_id": self._model_name, "model_id": self._model_name,
"transcript": text, "transcript": text,
"voice": voice_config, "voice": voice_config,
"output_format": self._settings["output_format"], "output_format": {
"language": self._settings["language"], "container": self._settings.output_container,
"encoding": self._settings.output_encoding,
"sample_rate": self._settings.output_sample_rate,
},
"language": self._settings.language,
} }
yield TTSStartedFrame(context_id=context_id) yield TTSStartedFrame(context_id=context_id)
headers = { headers = {

56
src/pipecat/services/aws/llm.py
View File

@@ -18,8 +18,8 @@ import io
import json import json
import os import os
import re import re
from dataclasses import dataclass from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional from typing import Any, ClassVar, Dict, List, Optional
from loguru import logger from loguru import logger
from PIL import Image from PIL import Image
@@ -40,7 +40,6 @@ from pipecat.frames.frames import (
LLMFullResponseStartFrame, LLMFullResponseStartFrame,
LLMMessagesFrame, LLMMessagesFrame,
LLMTextFrame, LLMTextFrame,
LLMUpdateSettingsFrame,
UserImageRawFrame, UserImageRawFrame,
) )
from pipecat.metrics.metrics import LLMTokenUsage from pipecat.metrics.metrics import LLMTokenUsage
@@ -57,6 +56,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import LLMService from pipecat.services.llm_service import LLMService
from pipecat.services.settings import NOT_GIVEN, LLMSettings
from pipecat.utils.tracing.service_decorators import traced_llm from pipecat.utils.tracing.service_decorators import traced_llm
try: try:
@@ -71,6 +71,19 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class AWSBedrockLLMSettings(LLMSettings):
"""Typed settings for AWS Bedrock LLM services.
Parameters:
latency: Performance mode - "standard" or "optimized".
additional_model_request_fields: Additional model-specific parameters.
"""
latency: Any = field(default_factory=lambda: NOT_GIVEN)
additional_model_request_fields: Any = field(default_factory=lambda: NOT_GIVEN)
@dataclass @dataclass
class AWSBedrockContextAggregatorPair: class AWSBedrockContextAggregatorPair:
"""Container for AWS Bedrock context aggregators. """Container for AWS Bedrock context aggregators.
@@ -806,15 +819,16 @@ class AWSBedrockLLMService(LLMService):
self.set_model_name(model) self.set_model_name(model)
self._retry_timeout_secs = retry_timeout_secs self._retry_timeout_secs = retry_timeout_secs
self._retry_on_timeout = retry_on_timeout self._retry_on_timeout = retry_on_timeout
self._settings = { self._settings = AWSBedrockLLMSettings(
"max_tokens": params.max_tokens, model=model,
"temperature": params.temperature, max_tokens=params.max_tokens,
"top_p": params.top_p, temperature=params.temperature,
"latency": params.latency, top_p=params.top_p,
"additional_model_request_fields": params.additional_model_request_fields latency=params.latency,
additional_model_request_fields=params.additional_model_request_fields
if isinstance(params.additional_model_request_fields, dict) if isinstance(params.additional_model_request_fields, dict)
else {}, else {},
} )
logger.info(f"Using AWS Bedrock model: {model}") logger.info(f"Using AWS Bedrock model: {model}")
@@ -836,12 +850,12 @@ class AWSBedrockLLMService(LLMService):
Dictionary containing only the inference parameters that are not None. Dictionary containing only the inference parameters that are not None.
""" """
inference_config = {} inference_config = {}
if self._settings["max_tokens"] is not None: if self._settings.max_tokens is not None:
inference_config["maxTokens"] = self._settings["max_tokens"] inference_config["maxTokens"] = self._settings.max_tokens
if self._settings["temperature"] is not None: if self._settings.temperature is not None:
inference_config["temperature"] = self._settings["temperature"] inference_config["temperature"] = self._settings.temperature
if self._settings["top_p"] is not None: if self._settings.top_p is not None:
inference_config["topP"] = self._settings["top_p"] inference_config["topP"] = self._settings.top_p
return inference_config return inference_config
async def run_inference( async def run_inference(
@@ -879,7 +893,7 @@ class AWSBedrockLLMService(LLMService):
request_params = { request_params = {
"modelId": self.model_name, "modelId": self.model_name,
"messages": messages, "messages": messages,
"additionalModelRequestFields": self._settings["additional_model_request_fields"], "additionalModelRequestFields": self._settings.additional_model_request_fields,
} }
if inference_config: if inference_config:
@@ -1036,7 +1050,7 @@ class AWSBedrockLLMService(LLMService):
request_params = { request_params = {
"modelId": self.model_name, "modelId": self.model_name,
"messages": messages, "messages": messages,
"additionalModelRequestFields": self._settings["additional_model_request_fields"], "additionalModelRequestFields": self._settings.additional_model_request_fields,
} }
# Only add inference config if it has parameters # Only add inference config if it has parameters
@@ -1081,8 +1095,8 @@ class AWSBedrockLLMService(LLMService):
request_params["toolConfig"] = tool_config request_params["toolConfig"] = tool_config
# Add performance config if latency is specified # Add performance config if latency is specified
if self._settings["latency"] in ["standard", "optimized"]: if self._settings.latency in ["standard", "optimized"]:
request_params["performanceConfig"] = {"latency": self._settings["latency"]} request_params["performanceConfig"] = {"latency": self._settings.latency}
# Log request params with messages redacted for logging # Log request params with messages redacted for logging
if isinstance(context, LLMContext): if isinstance(context, LLMContext):
@@ -1207,8 +1221,6 @@ class AWSBedrockLLMService(LLMService):
# NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal # NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal
# LLMContext with it # LLMContext with it
context = AWSBedrockLLMContext.from_messages(frame.messages) context = AWSBedrockLLMContext.from_messages(frame.messages)
elif isinstance(frame, LLMUpdateSettingsFrame):
await self._update_settings(frame.settings)
else: else:
await self.push_frame(frame, direction) await self.push_frame(frame, direction)

73
src/pipecat/services/aws/stt.py
View File

@@ -14,6 +14,7 @@ import json
import os import os
import random import random
import string import string
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -28,6 +29,7 @@ from pipecat.frames.frames import (
TranscriptionFrame, TranscriptionFrame,
) )
from pipecat.services.aws.utils import build_event_message, decode_event, get_presigned_url from pipecat.services.aws.utils import build_event_message, decode_event, get_presigned_url
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import AWS_TRANSCRIBE_TTFS_P99 from pipecat.services.stt_latency import AWS_TRANSCRIBE_TTFS_P99
from pipecat.services.stt_service import WebsocketSTTService from pipecat.services.stt_service import WebsocketSTTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -43,6 +45,25 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class AWSTranscribeSTTSettings(STTSettings):
"""Typed settings for the AWS Transcribe STT service.
Parameters:
sample_rate: Audio sample rate in Hz (8000 or 16000).
media_encoding: Audio encoding format (e.g. "linear16").
number_of_channels: Number of audio channels.
show_speaker_label: Whether to show speaker labels.
enable_channel_identification: Whether to enable channel identification.
"""
sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
media_encoding: str = field(default_factory=lambda: NOT_GIVEN)
number_of_channels: int = field(default_factory=lambda: NOT_GIVEN)
show_speaker_label: bool = field(default_factory=lambda: NOT_GIVEN)
enable_channel_identification: bool = field(default_factory=lambda: NOT_GIVEN)
class AWSTranscribeSTTService(WebsocketSTTService): class AWSTranscribeSTTService(WebsocketSTTService):
"""AWS Transcribe Speech-to-Text service using WebSocket streaming. """AWS Transcribe Speech-to-Text service using WebSocket streaming.
@@ -78,21 +99,21 @@ class AWSTranscribeSTTService(WebsocketSTTService):
""" """
super().__init__(ttfs_p99_latency=ttfs_p99_latency, **kwargs) super().__init__(ttfs_p99_latency=ttfs_p99_latency, **kwargs)
self._settings = { self._settings: AWSTranscribeSTTSettings = AWSTranscribeSTTSettings(
"sample_rate": sample_rate, language=language,
"language": language, sample_rate=sample_rate,
"media_encoding": "linear16", # AWS expects raw PCM media_encoding="linear16",
"number_of_channels": 1, number_of_channels=1,
"show_speaker_label": False, show_speaker_label=False,
"enable_channel_identification": False, enable_channel_identification=False,
} )
# Validate sample rate - AWS Transcribe only supports 8000 Hz or 16000 Hz # Validate sample rate - AWS Transcribe only supports 8000 Hz or 16000 Hz
if sample_rate not in [8000, 16000]: if sample_rate not in [8000, 16000]:
logger.warning( logger.warning(
f"AWS Transcribe only supports 8000 Hz or 16000 Hz sample rates. Converting from {sample_rate} Hz to 16000 Hz." f"AWS Transcribe only supports 8000 Hz or 16000 Hz sample rates. Converting from {sample_rate} Hz to 16000 Hz."
) )
self._settings["sample_rate"] = 16000 self._settings.sample_rate = 16000
self._credentials = { self._credentials = {
"aws_access_key_id": aws_access_key_id or os.getenv("AWS_ACCESS_KEY_ID"), "aws_access_key_id": aws_access_key_id or os.getenv("AWS_ACCESS_KEY_ID"),
@@ -117,6 +138,20 @@ class AWSTranscribeSTTService(WebsocketSTTService):
} }
return encoding_map.get(encoding, encoding) return encoding_map.get(encoding, encoding)
async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Apply a typed settings update, reconnecting if needed.
Any change to connection-relevant settings (model, language, etc.)
triggers a WebSocket reconnect so the new configuration takes effect.
"""
changed = await super()._update_settings_from_typed(update)
if changed and self._websocket:
await self._disconnect()
await self._connect()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Initialize the connection when the service starts. """Initialize the connection when the service starts.
@@ -208,9 +243,9 @@ class AWSTranscribeSTTService(WebsocketSTTService):
logger.debug("Connecting to AWS Transcribe WebSocket") logger.debug("Connecting to AWS Transcribe WebSocket")
language_code = self.language_to_service_language(Language(self._settings["language"])) language_code = self.language_to_service_language(Language(self._settings.language))
if not language_code: if not language_code:
raise ValueError(f"Unsupported language: {self._settings['language']}") raise ValueError(f"Unsupported language: {self._settings.language}")
# Generate random websocket key # Generate random websocket key
websocket_key = "".join( websocket_key = "".join(
@@ -237,14 +272,14 @@ class AWSTranscribeSTTService(WebsocketSTTService):
}, },
language_code=language_code, language_code=language_code,
media_encoding=self.get_service_encoding( media_encoding=self.get_service_encoding(
self._settings["media_encoding"] self._settings.media_encoding
), # Convert to AWS format ), # Convert to AWS format
sample_rate=self._settings["sample_rate"], sample_rate=self._settings.sample_rate,
number_of_channels=self._settings["number_of_channels"], number_of_channels=self._settings.number_of_channels,
enable_partial_results_stabilization=True, enable_partial_results_stabilization=True,
partial_results_stability="high", partial_results_stability="high",
show_speaker_label=self._settings["show_speaker_label"], show_speaker_label=self._settings.show_speaker_label,
enable_channel_identification=self._settings["enable_channel_identification"], enable_channel_identification=self._settings.enable_channel_identification,
) )
logger.debug(f"{self} Connecting to WebSocket with URL: {presigned_url[:100]}...") logger.debug(f"{self} Connecting to WebSocket with URL: {presigned_url[:100]}...")
@@ -479,14 +514,14 @@ class AWSTranscribeSTTService(WebsocketSTTService):
transcript, transcript,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._settings["language"], self._settings.language,
result=result, result=result,
) )
) )
await self._handle_transcription( await self._handle_transcription(
transcript, transcript,
is_final, is_final,
self._settings["language"], self._settings.language,
) )
await self.stop_processing_metrics() await self.stop_processing_metrics()
else: else:
@@ -495,7 +530,7 @@ class AWSTranscribeSTTService(WebsocketSTTService):
transcript, transcript,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._settings["language"], self._settings.language,
result=result, result=result,
) )
) )

60
src/pipecat/services/aws/tts.py
View File

@@ -11,6 +11,7 @@ supporting multiple languages, voices, and SSML features.
""" """
import os import os
from dataclasses import dataclass, field
from typing import AsyncGenerator, List, Optional from typing import AsyncGenerator, List, Optional
from loguru import logger from loguru import logger
@@ -24,6 +25,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -121,6 +123,25 @@ def language_to_aws_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class AWSPollyTTSSettings(TTSSettings):
"""Typed settings for AWS Polly TTS service.
Parameters:
engine: TTS engine to use ('standard', 'neural', etc.).
pitch: Voice pitch adjustment (for standard engine only).
rate: Speech rate adjustment.
volume: Voice volume adjustment.
lexicon_names: List of pronunciation lexicons to apply.
"""
engine: str = field(default_factory=lambda: NOT_GIVEN)
pitch: str = field(default_factory=lambda: NOT_GIVEN)
rate: str = field(default_factory=lambda: NOT_GIVEN)
volume: str = field(default_factory=lambda: NOT_GIVEN)
lexicon_names: List[str] = field(default_factory=lambda: NOT_GIVEN)
class AWSPollyTTSService(TTSService): class AWSPollyTTSService(TTSService):
"""AWS Polly text-to-speech service. """AWS Polly text-to-speech service.
@@ -185,20 +206,21 @@ class AWSPollyTTSService(TTSService):
} }
self._aws_session = aioboto3.Session() self._aws_session = aioboto3.Session()
self._settings = { self._settings: AWSPollyTTSSettings = AWSPollyTTSSettings(
"engine": params.engine, voice=voice_id,
"language": self.language_to_service_language(params.language) engine=params.engine,
language=self.language_to_service_language(params.language)
if params.language if params.language
else "en-US", else "en-US",
"pitch": params.pitch, pitch=params.pitch,
"rate": params.rate, rate=params.rate,
"volume": params.volume, volume=params.volume,
"lexicon_names": params.lexicon_names, lexicon_names=params.lexicon_names,
} )
self._resampler = create_stream_resampler() self._resampler = create_stream_resampler()
self.set_voice(voice_id) self._voice_id = voice_id
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -222,19 +244,19 @@ class AWSPollyTTSService(TTSService):
def _construct_ssml(self, text: str) -> str: def _construct_ssml(self, text: str) -> str:
ssml = "<speak>" ssml = "<speak>"
language = self._settings["language"] language = self._settings.language
ssml += f"<lang xml:lang='{language}'>" ssml += f"<lang xml:lang='{language}'>"
prosody_attrs = [] prosody_attrs = []
# Prosody tags are only supported for standard and neural engines # Prosody tags are only supported for standard and neural engines
if self._settings["engine"] == "standard": if self._settings.engine == "standard":
if self._settings["pitch"]: if self._settings.pitch:
prosody_attrs.append(f"pitch='{self._settings['pitch']}'") prosody_attrs.append(f"pitch='{self._settings.pitch}'")
if self._settings["rate"]: if self._settings.rate:
prosody_attrs.append(f"rate='{self._settings['rate']}'") prosody_attrs.append(f"rate='{self._settings.rate}'")
if self._settings["volume"]: if self._settings.volume:
prosody_attrs.append(f"volume='{self._settings['volume']}'") prosody_attrs.append(f"volume='{self._settings.volume}'")
if prosody_attrs: if prosody_attrs:
ssml += f"<prosody {' '.join(prosody_attrs)}>" ssml += f"<prosody {' '.join(prosody_attrs)}>"
@@ -276,10 +298,10 @@ class AWSPollyTTSService(TTSService):
"TextType": "ssml", "TextType": "ssml",
"OutputFormat": "pcm", "OutputFormat": "pcm",
"VoiceId": self._voice_id, "VoiceId": self._voice_id,
"Engine": self._settings["engine"], "Engine": self._settings.engine,
# AWS only supports 8000 and 16000 for PCM. We select 16000. # AWS only supports 8000 and 16000 for PCM. We select 16000.
"SampleRate": "16000", "SampleRate": "16000",
"LexiconNames": self._settings["lexicon_names"], "LexiconNames": self._settings.lexicon_names,
} }
# Filter out None values # Filter out None values

52
src/pipecat/services/azure/stt.py
View File

@@ -11,6 +11,7 @@ Speech SDK for real-time audio transcription.
""" """
import asyncio import asyncio
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -25,6 +26,7 @@ from pipecat.frames.frames import (
TranscriptionFrame, TranscriptionFrame,
) )
from pipecat.services.azure.common import language_to_azure_language from pipecat.services.azure.common import language_to_azure_language
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import AZURE_TTFS_P99 from pipecat.services.stt_latency import AZURE_TTFS_P99
from pipecat.services.stt_service import STTService from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -48,6 +50,19 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class AzureSTTSettings(STTSettings):
"""Typed settings for the Azure STT service.
Parameters:
region: Azure region for the Speech service.
sample_rate: Audio sample rate in Hz.
"""
region: str = field(default_factory=lambda: NOT_GIVEN)
sample_rate: Optional[int] = field(default_factory=lambda: NOT_GIVEN)
class AzureSTTService(STTService): class AzureSTTService(STTService):
"""Azure Speech-to-Text service for real-time audio transcription. """Azure Speech-to-Text service for real-time audio transcription.
@@ -92,11 +107,11 @@ class AzureSTTService(STTService):
self._audio_stream = None self._audio_stream = None
self._speech_recognizer = None self._speech_recognizer = None
self._settings = { self._settings: AzureSTTSettings = AzureSTTSettings(
"region": region, region=region,
"language": language_to_azure_language(language), language=language_to_azure_language(language),
"sample_rate": sample_rate, sample_rate=sample_rate,
} )
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate performance metrics. """Check if this service can generate performance metrics.
@@ -106,6 +121,29 @@ class AzureSTTService(STTService):
""" """
return True return True
async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Apply a typed settings update, reconfiguring the recognizer if needed.
When ``language`` changes the ``SpeechConfig`` is updated and the
speech recognizer is restarted so that the new language takes effect.
"""
changed = await super()._update_settings_from_typed(update)
if "language" in changed:
# Convert Language enum to Azure language code if needed.
lang = self._settings.language
if isinstance(lang, Language):
lang = language_to_azure_language(lang)
self._settings.language = lang
self._speech_config.speech_recognition_language = lang
# Restart the recognizer with the new config.
if self._speech_recognizer:
self._speech_recognizer.stop_continuous_recognition_async()
self._speech_recognizer.start_continuous_recognition_async()
return changed
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]: async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
"""Process audio data for speech-to-text conversion. """Process audio data for speech-to-text conversion.
@@ -198,7 +236,7 @@ class AzureSTTService(STTService):
def _on_handle_recognized(self, event): def _on_handle_recognized(self, event):
if event.result.reason == ResultReason.RecognizedSpeech and len(event.result.text) > 0: if event.result.reason == ResultReason.RecognizedSpeech and len(event.result.text) > 0:
language = getattr(event.result, "language", None) or self._settings.get("language") language = getattr(event.result, "language", None) or self._settings.language
frame = TranscriptionFrame( frame = TranscriptionFrame(
event.result.text, event.result.text,
self._user_id, self._user_id,
@@ -213,7 +251,7 @@ class AzureSTTService(STTService):
def _on_handle_recognizing(self, event): def _on_handle_recognizing(self, event):
if event.result.reason == ResultReason.RecognizingSpeech and len(event.result.text) > 0: if event.result.reason == ResultReason.RecognizingSpeech and len(event.result.text) > 0:
language = getattr(event.result, "language", None) or self._settings.get("language") language = getattr(event.result, "language", None) or self._settings.language
frame = InterimTranscriptionFrame( frame = InterimTranscriptionFrame(
event.result.text, event.result.text,
self._user_id, self._user_id,

87
src/pipecat/services/azure/tts.py
View File

@@ -7,6 +7,7 @@
"""Azure Cognitive Services Text-to-Speech service implementations.""" """Azure Cognitive Services Text-to-Speech service implementations."""
import asyncio import asyncio
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -25,6 +26,7 @@ from pipecat.frames.frames import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.azure.common import language_to_azure_language from pipecat.services.azure.common import language_to_azure_language
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService, WordTTSService from pipecat.services.tts_service import TTSService, WordTTSService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -65,6 +67,31 @@ def sample_rate_to_output_format(sample_rate: int) -> SpeechSynthesisOutputForma
return sample_rate_map.get(sample_rate, SpeechSynthesisOutputFormat.Raw24Khz16BitMonoPcm) return sample_rate_map.get(sample_rate, SpeechSynthesisOutputFormat.Raw24Khz16BitMonoPcm)
@dataclass
class AzureTTSSettings(TTSSettings):
"""Typed settings for Azure TTS services.
Parameters:
emphasis: Emphasis level for speech ("strong", "moderate", "reduced").
language: Language for synthesis. Defaults to English (US).
pitch: Voice pitch adjustment (e.g., "+10%", "-5Hz", "high").
rate: Speech rate adjustment (e.g., "1.0", "1.25", "slow", "fast").
role: Voice role for expression (e.g., "YoungAdultFemale").
style: Speaking style (e.g., "cheerful", "sad", "excited").
style_degree: Intensity of the speaking style (0.01 to 2.0).
volume: Volume level (e.g., "+20%", "loud", "x-soft").
"""
emphasis: str = field(default_factory=lambda: NOT_GIVEN)
language: str = field(default_factory=lambda: NOT_GIVEN)
pitch: str = field(default_factory=lambda: NOT_GIVEN)
rate: str = field(default_factory=lambda: NOT_GIVEN)
role: str = field(default_factory=lambda: NOT_GIVEN)
style: str = field(default_factory=lambda: NOT_GIVEN)
style_degree: str = field(default_factory=lambda: NOT_GIVEN)
volume: str = field(default_factory=lambda: NOT_GIVEN)
class AzureBaseTTSService: class AzureBaseTTSService:
"""Base mixin class for Azure Cognitive Services text-to-speech implementations. """Base mixin class for Azure Cognitive Services text-to-speech implementations.
@@ -126,18 +153,18 @@ class AzureBaseTTSService:
""" """
params = params or AzureBaseTTSService.InputParams() params = params or AzureBaseTTSService.InputParams()
self._settings = { self._settings: AzureTTSSettings = AzureTTSSettings(
"emphasis": params.emphasis, emphasis=params.emphasis,
"language": self.language_to_service_language(params.language) language=self.language_to_service_language(params.language)
if params.language if params.language
else "en-US", else "en-US",
"pitch": params.pitch, pitch=params.pitch,
"rate": params.rate, rate=params.rate,
"role": params.role, role=params.role,
"style": params.style, style=params.style,
"style_degree": params.style_degree, style_degree=params.style_degree,
"volume": params.volume, volume=params.volume,
} )
self._api_key = api_key self._api_key = api_key
self._region = region self._region = region
@@ -156,7 +183,7 @@ class AzureBaseTTSService:
return language_to_azure_language(language) return language_to_azure_language(language)
def _construct_ssml(self, text: str) -> str: def _construct_ssml(self, text: str) -> str:
language = self._settings["language"] language = self._settings.language
# Escape special characters # Escape special characters
escaped_text = self._escape_text(text) escaped_text = self._escape_text(text)
@@ -169,38 +196,38 @@ class AzureBaseTTSService:
"<mstts:silence type='Sentenceboundary' value='20ms' />" "<mstts:silence type='Sentenceboundary' value='20ms' />"
) )
if self._settings["style"]: if self._settings.style:
ssml += f"<mstts:express-as style='{self._settings['style']}'" ssml += f"<mstts:express-as style='{self._settings.style}'"
if self._settings["style_degree"]: if self._settings.style_degree:
ssml += f" styledegree='{self._settings['style_degree']}'" ssml += f" styledegree='{self._settings.style_degree}'"
if self._settings["role"]: if self._settings.role:
ssml += f" role='{self._settings['role']}'" ssml += f" role='{self._settings.role}'"
ssml += ">" ssml += ">"
prosody_attrs = [] prosody_attrs = []
if self._settings["rate"]: if self._settings.rate:
prosody_attrs.append(f"rate='{self._settings['rate']}'") prosody_attrs.append(f"rate='{self._settings.rate}'")
if self._settings["pitch"]: if self._settings.pitch:
prosody_attrs.append(f"pitch='{self._settings['pitch']}'") prosody_attrs.append(f"pitch='{self._settings.pitch}'")
if self._settings["volume"]: if self._settings.volume:
prosody_attrs.append(f"volume='{self._settings['volume']}'") prosody_attrs.append(f"volume='{self._settings.volume}'")
# Only wrap in prosody tag if there are prosody attributes # Only wrap in prosody tag if there are prosody attributes
if prosody_attrs: if prosody_attrs:
ssml += f"<prosody {' '.join(prosody_attrs)}>" ssml += f"<prosody {' '.join(prosody_attrs)}>"
if self._settings["emphasis"]: if self._settings.emphasis:
ssml += f"<emphasis level='{self._settings['emphasis']}'>" ssml += f"<emphasis level='{self._settings.emphasis}'>"
ssml += escaped_text ssml += escaped_text
if self._settings["emphasis"]: if self._settings.emphasis:
ssml += "</emphasis>" ssml += "</emphasis>"
if prosody_attrs: if prosody_attrs:
ssml += "</prosody>" ssml += "</prosody>"
if self._settings["style"]: if self._settings.style:
ssml += "</mstts:express-as>" ssml += "</mstts:express-as>"
ssml += "</voice></speak>" ssml += "</voice></speak>"
@@ -314,7 +341,7 @@ class AzureTTSService(WordTTSService, AzureBaseTTSService):
subscription=self._api_key, subscription=self._api_key,
region=self._region, region=self._region,
) )
self._speech_config.speech_synthesis_language = self._settings["language"] self._speech_config.speech_synthesis_language = self._settings.language
self._speech_config.set_speech_synthesis_output_format( self._speech_config.set_speech_synthesis_output_format(
sample_rate_to_output_format(self.sample_rate) sample_rate_to_output_format(self.sample_rate)
) )
@@ -364,7 +391,7 @@ class AzureTTSService(WordTTSService, AzureBaseTTSService):
Returns: Returns:
True if the language is CJK, False otherwise. True if the language is CJK, False otherwise.
""" """
language = self._settings.get("language", "").lower() language = (self._settings.language if self._settings.language else "").lower()
# Check if language starts with CJK language codes # Check if language starts with CJK language codes
return language.startswith(("zh", "ja", "ko", "cmn", "yue", "wuu")) return language.startswith(("zh", "ja", "ko", "cmn", "yue", "wuu"))
@@ -735,7 +762,7 @@ class AzureHttpTTSService(TTSService, AzureBaseTTSService):
subscription=self._api_key, subscription=self._api_key,
region=self._region, region=self._region,
) )
self._speech_config.speech_synthesis_language = self._settings["language"] self._speech_config.speech_synthesis_language = self._settings.language
self._speech_config.set_speech_synthesis_output_format( self._speech_config.set_speech_synthesis_output_format(
sample_rate_to_output_format(self.sample_rate) sample_rate_to_output_format(self.sample_rate)
) )

33
src/pipecat/services/camb/tts.py
View File

@@ -16,7 +16,8 @@ Features:
- Model-specific sample rates: mars-pro (48kHz), mars-flash (22.05kHz) - Model-specific sample rates: mars-pro (48kHz), mars-flash (22.05kHz)
""" """
from typing import Any, AsyncGenerator, Dict, Optional from dataclasses import dataclass, field
from typing import AsyncGenerator, Dict, Optional
from camb import StreamTtsOutputConfiguration from camb import StreamTtsOutputConfiguration
from camb.client import AsyncCambAI from camb.client import AsyncCambAI
@@ -31,6 +32,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -133,6 +135,18 @@ def _get_aligned_audio(buffer: bytes) -> tuple[bytes, bytes]:
return buffer[:aligned_size], buffer[aligned_size:] return buffer[:aligned_size], buffer[aligned_size:]
@dataclass
class CambTTSSettings(TTSSettings):
"""Typed settings for Camb.ai TTS service.
Parameters:
user_instructions: Custom instructions for mars-instruct model only.
Ignored for other models. Max 1000 characters.
"""
user_instructions: str = field(default_factory=lambda: NOT_GIVEN)
class CambTTSService(TTSService): class CambTTSService(TTSService):
"""Camb.ai MARS text-to-speech service using the official SDK. """Camb.ai MARS text-to-speech service using the official SDK.
@@ -212,15 +226,16 @@ class CambTTSService(TTSService):
) )
# Build settings # Build settings
self._settings = { self._settings: CambTTSSettings = CambTTSSettings(
"language": ( model=model,
voice=voice_id,
language=(
self.language_to_service_language(params.language) if params.language else "en-us" self.language_to_service_language(params.language) if params.language else "en-us"
), ),
"user_instructions": params.user_instructions, user_instructions=params.user_instructions,
} )
self.set_model_name(model) self.set_model_name(model)
self.set_voice(str(voice_id))
self._voice_id = voice_id self._voice_id = voice_id
self._client = None self._client = None
@@ -283,14 +298,14 @@ class CambTTSService(TTSService):
tts_kwargs: Dict[str, Any] = { tts_kwargs: Dict[str, Any] = {
"text": text, "text": text,
"voice_id": self._voice_id, "voice_id": self._voice_id,
"language": self._settings["language"], "language": self._settings.language,
"speech_model": self.model_name, "speech_model": self.model_name,
"output_configuration": StreamTtsOutputConfiguration(format="pcm_s16le"), "output_configuration": StreamTtsOutputConfiguration(format="pcm_s16le"),
} }
# Add user instructions if using mars-instruct model # Add user instructions if using mars-instruct model
if self._model_name == "mars-instruct" and self._settings.get("user_instructions"): if self._model_name == "mars-instruct" and self._settings.user_instructions:
tts_kwargs["user_instructions"] = self._settings["user_instructions"] tts_kwargs["user_instructions"] = self._settings.user_instructions
await self.start_tts_usage_metrics(text) await self.start_tts_usage_metrics(text)
yield TTSStartedFrame(context_id=context_id) yield TTSStartedFrame(context_id=context_id)

41
src/pipecat/services/cartesia/stt.py
View File

@@ -12,6 +12,7 @@ the Cartesia Live transcription API for real-time speech recognition.
import json import json
import urllib.parse import urllib.parse
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -27,6 +28,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import CARTESIA_TTFS_P99 from pipecat.services.stt_latency import CARTESIA_TTFS_P99
from pipecat.services.stt_service import WebsocketSTTService from pipecat.services.stt_service import WebsocketSTTService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -42,6 +44,17 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class CartesiaSTTSettings(STTSettings):
"""Typed settings for the Cartesia STT service.
Parameters:
encoding: Audio encoding format (e.g. ``"pcm_s16le"``).
"""
encoding: str = field(default_factory=lambda: NOT_GIVEN)
class CartesiaLiveOptions: class CartesiaLiveOptions:
"""Configuration options for Cartesia Live STT service. """Configuration options for Cartesia Live STT service.
@@ -181,7 +194,11 @@ class CartesiaSTTService(WebsocketSTTService):
k: v for k, v in merged_options.items() if not isinstance(v, str) or v != "None" k: v for k, v in merged_options.items() if not isinstance(v, str) or v != "None"
} }
self._settings = merged_options self._settings: CartesiaSTTSettings = CartesiaSTTSettings(
model=merged_options["model"],
language=merged_options.get("language"),
encoding=merged_options.get("encoding", "pcm_s16le"),
)
self.set_model_name(merged_options["model"]) self.set_model_name(merged_options["model"])
self._api_key = api_key self._api_key = api_key
self._base_url = base_url or "api.cartesia.ai" self._base_url = base_url or "api.cartesia.ai"
@@ -275,13 +292,33 @@ class CartesiaSTTService(WebsocketSTTService):
await self._disconnect_websocket() await self._disconnect_websocket()
async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Apply a typed settings update and reconnect if anything changed.
Args:
update: A :class:`STTSettings` (or ``CartesiaSTTSettings``) delta.
Returns:
Set of field names whose values actually changed.
"""
changed = await super()._update_settings_from_typed(update)
if changed:
await self._disconnect()
await self._connect()
return changed
async def _connect_websocket(self): async def _connect_websocket(self):
try: try:
if self._websocket and self._websocket.state is State.OPEN: if self._websocket and self._websocket.state is State.OPEN:
return return
logger.debug("Connecting to Cartesia STT") logger.debug("Connecting to Cartesia STT")
params = self._settings params = {
"model": self._settings.model,
"language": self._settings.language,
"encoding": self._settings.encoding,
"sample_rate": str(self.sample_rate),
}
ws_url = f"wss://{self._base_url}/stt/websocket?{urllib.parse.urlencode(params)}" ws_url = f"wss://{self._base_url}/stt/websocket?{urllib.parse.urlencode(params)}"
headers = {"Cartesia-Version": "2025-04-16", "X-API-Key": self._api_key} headers = {"Cartesia-Version": "2025-04-16", "X-API-Key": self._api_key}

153
src/pipecat/services/cartesia/tts.py
View File

@@ -9,8 +9,9 @@
import base64 import base64
import json import json
import warnings import warnings
from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import AsyncGenerator, List, Literal, Optional from typing import Any, AsyncGenerator, List, Literal, Optional
from loguru import logger from loguru import logger
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
@@ -27,6 +28,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
from pipecat.services.tts_service import AudioContextWordTTSService, TTSService from pipecat.services.tts_service import AudioContextWordTTSService, TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.text.base_text_aggregator import BaseTextAggregator from pipecat.utils.text.base_text_aggregator import BaseTextAggregator
@@ -191,6 +193,31 @@ class CartesiaEmotion(str, Enum):
DETERMINED = "determined" DETERMINED = "determined"
@dataclass
class CartesiaTTSSettings(TTSSettings):
"""Typed settings for Cartesia TTS services.
Parameters:
output_container: Audio container format (e.g. "raw").
output_encoding: Audio encoding format (e.g. "pcm_s16le").
output_sample_rate: Audio sample rate in Hz.
speed: Voice speed control for non-Sonic-3 models (literal values).
emotion: List of emotion controls for non-Sonic-3 models.
generation_config: Generation configuration for Sonic-3 models. Includes volume,
speed (numeric), and emotion (string) parameters.
pronunciation_dict_id: The ID of the pronunciation dictionary to use for
custom pronunciations.
"""
output_container: str = field(default_factory=lambda: NOT_GIVEN)
output_encoding: str = field(default_factory=lambda: NOT_GIVEN)
output_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
speed: str = field(default_factory=lambda: NOT_GIVEN)
emotion: List[str] = field(default_factory=lambda: NOT_GIVEN)
generation_config: GenerationConfig = field(default_factory=lambda: NOT_GIVEN)
pronunciation_dict_id: str = field(default_factory=lambda: NOT_GIVEN)
class CartesiaTTSService(AudioContextWordTTSService): class CartesiaTTSService(AudioContextWordTTSService):
"""Cartesia TTS service with WebSocket streaming and word timestamps. """Cartesia TTS service with WebSocket streaming and word timestamps.
@@ -289,22 +316,20 @@ class CartesiaTTSService(AudioContextWordTTSService):
self._api_key = api_key self._api_key = api_key
self._cartesia_version = cartesia_version self._cartesia_version = cartesia_version
self._url = url self._url = url
self._settings = { self._settings: CartesiaTTSSettings = CartesiaTTSSettings(
"output_format": { output_container=container,
"container": container, output_encoding=encoding,
"encoding": encoding, output_sample_rate=0,
"sample_rate": 0, language=self.language_to_service_language(params.language)
},
"language": self.language_to_service_language(params.language)
if params.language if params.language
else None, else None,
"speed": params.speed, speed=params.speed,
"emotion": params.emotion, emotion=params.emotion,
"generation_config": params.generation_config, generation_config=params.generation_config,
"pronunciation_dict_id": params.pronunciation_dict_id, pronunciation_dict_id=params.pronunciation_dict_id,
} )
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice_id) self._voice_id = voice_id
self._context_id = None self._context_id = None
self._receive_task = None self._receive_task = None
@@ -317,16 +342,6 @@ class CartesiaTTSService(AudioContextWordTTSService):
""" """
return True return True
async def set_model(self, model: str):
"""Set the TTS model.
Args:
model: The model name to use for synthesis.
"""
self._model_id = model
await super().set_model(model)
logger.info(f"Switching TTS model to: [{model}]")
def language_to_service_language(self, language: Language) -> Optional[str]: def language_to_service_language(self, language: Language) -> Optional[str]:
"""Convert a Language enum to Cartesia language format. """Convert a Language enum to Cartesia language format.
@@ -391,7 +406,7 @@ class CartesiaTTSService(AudioContextWordTTSService):
Returns: Returns:
List of (word, start_time) tuples processed for the language. List of (word, start_time) tuples processed for the language.
""" """
current_language = self._settings.get("language") current_language = self._settings.language
# Check if this is a CJK language (if language is None, treat as non-CJK) # Check if this is a CJK language (if language is None, treat as non-CJK)
if current_language and self._is_cjk_language(current_language): if current_language and self._is_cjk_language(current_language):
@@ -414,7 +429,7 @@ class CartesiaTTSService(AudioContextWordTTSService):
voice_config["mode"] = "id" voice_config["mode"] = "id"
voice_config["id"] = self._voice_id voice_config["id"] = self._voice_id
if self._settings["emotion"]: if is_given(self._settings.emotion) and self._settings.emotion:
with warnings.catch_warnings(): with warnings.catch_warnings():
warnings.simplefilter("always") warnings.simplefilter("always")
warnings.warn( warnings.warn(
@@ -423,8 +438,7 @@ class CartesiaTTSService(AudioContextWordTTSService):
stacklevel=2, stacklevel=2,
) )
voice_config["__experimental_controls"] = {} voice_config["__experimental_controls"] = {}
if self._settings["emotion"]: voice_config["__experimental_controls"]["emotion"] = self._settings.emotion
voice_config["__experimental_controls"]["emotion"] = self._settings["emotion"]
msg = { msg = {
"transcript": text, "transcript": text,
@@ -432,24 +446,28 @@ class CartesiaTTSService(AudioContextWordTTSService):
"context_id": self._context_id, "context_id": self._context_id,
"model_id": self.model_name, "model_id": self.model_name,
"voice": voice_config, "voice": voice_config,
"output_format": self._settings["output_format"], "output_format": {
"container": self._settings.output_container,
"encoding": self._settings.output_encoding,
"sample_rate": self._settings.output_sample_rate,
},
"add_timestamps": add_timestamps, "add_timestamps": add_timestamps,
"use_original_timestamps": False if self.model_name == "sonic" else True, "use_original_timestamps": False if self.model_name == "sonic" else True,
} }
if self._settings["language"]: if is_given(self._settings.language) and self._settings.language:
msg["language"] = self._settings["language"] msg["language"] = self._settings.language
if self._settings["speed"]: if is_given(self._settings.speed) and self._settings.speed:
msg["speed"] = self._settings["speed"] msg["speed"] = self._settings.speed
if self._settings["generation_config"]: if is_given(self._settings.generation_config) and self._settings.generation_config:
msg["generation_config"] = self._settings["generation_config"].model_dump( msg["generation_config"] = self._settings.generation_config.model_dump(
exclude_none=True exclude_none=True
) )
if self._settings["pronunciation_dict_id"]: if is_given(self._settings.pronunciation_dict_id) and self._settings.pronunciation_dict_id:
msg["pronunciation_dict_id"] = self._settings["pronunciation_dict_id"] msg["pronunciation_dict_id"] = self._settings.pronunciation_dict_id
return json.dumps(msg) return json.dumps(msg)
@@ -460,7 +478,7 @@ class CartesiaTTSService(AudioContextWordTTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["output_format"]["sample_rate"] = self.sample_rate self._settings.output_sample_rate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -694,21 +712,21 @@ class CartesiaHttpTTSService(TTSService):
self._api_key = api_key self._api_key = api_key
self._base_url = base_url self._base_url = base_url
self._cartesia_version = cartesia_version self._cartesia_version = cartesia_version
self._settings = { self._settings: CartesiaTTSSettings = CartesiaTTSSettings(
"output_format": { model=model,
"container": container, voice=voice_id,
"encoding": encoding, output_container=container,
"sample_rate": 0, output_encoding=encoding,
}, output_sample_rate=0,
"language": self.language_to_service_language(params.language) language=self.language_to_service_language(params.language)
if params.language if params.language
else None, else None,
"speed": params.speed, speed=params.speed,
"emotion": params.emotion, emotion=params.emotion,
"generation_config": params.generation_config, generation_config=params.generation_config,
"pronunciation_dict_id": params.pronunciation_dict_id, pronunciation_dict_id=params.pronunciation_dict_id,
} )
self.set_voice(voice_id) self._voice_id = voice_id
self.set_model_name(model) self.set_model_name(model)
self._client = AsyncCartesia( self._client = AsyncCartesia(
@@ -742,7 +760,7 @@ class CartesiaHttpTTSService(TTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["output_format"]["sample_rate"] = self.sample_rate self._settings.output_sample_rate = self.sample_rate
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
"""Stop the Cartesia HTTP TTS service. """Stop the Cartesia HTTP TTS service.
@@ -778,7 +796,7 @@ class CartesiaHttpTTSService(TTSService):
try: try:
voice_config = {"mode": "id", "id": self._voice_id} voice_config = {"mode": "id", "id": self._voice_id}
if self._settings["emotion"]: if is_given(self._settings.emotion) and self._settings.emotion:
with warnings.catch_warnings(): with warnings.catch_warnings():
warnings.simplefilter("always") warnings.simplefilter("always")
warnings.warn( warnings.warn(
@@ -786,30 +804,39 @@ class CartesiaHttpTTSService(TTSService):
DeprecationWarning, DeprecationWarning,
stacklevel=2, stacklevel=2,
) )
voice_config["__experimental_controls"] = {"emotion": self._settings["emotion"]} voice_config["__experimental_controls"] = {"emotion": self._settings.emotion}
await self.start_ttfb_metrics() await self.start_ttfb_metrics()
output_format = {
"container": self._settings.output_container,
"encoding": self._settings.output_encoding,
"sample_rate": self._settings.output_sample_rate,
}
payload = { payload = {
"model_id": self._model_name, "model_id": self._model_name,
"transcript": text, "transcript": text,
"voice": voice_config, "voice": voice_config,
"output_format": self._settings["output_format"], "output_format": output_format,
} }
if self._settings["language"]: if is_given(self._settings.language) and self._settings.language:
payload["language"] = self._settings["language"] payload["language"] = self._settings.language
if self._settings["speed"]: if is_given(self._settings.speed) and self._settings.speed:
payload["speed"] = self._settings["speed"] payload["speed"] = self._settings.speed
if self._settings["generation_config"]: if is_given(self._settings.generation_config) and self._settings.generation_config:
payload["generation_config"] = self._settings["generation_config"].model_dump( payload["generation_config"] = self._settings.generation_config.model_dump(
exclude_none=True exclude_none=True
) )
if self._settings["pronunciation_dict_id"]: if (
payload["pronunciation_dict_id"] = self._settings["pronunciation_dict_id"] is_given(self._settings.pronunciation_dict_id)
and self._settings.pronunciation_dict_id
):
payload["pronunciation_dict_id"] = self._settings.pronunciation_dict_id
yield TTSStartedFrame(context_id=context_id) yield TTSStartedFrame(context_id=context_id)

10
src/pipecat/services/cerebras/llm.py
View File

@@ -68,14 +68,14 @@ class CerebrasLLMService(OpenAILLMService):
params = { params = {
"model": self.model_name, "model": self.model_name,
"stream": True, "stream": True,
"seed": self._settings["seed"], "seed": self._settings.seed,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"max_completion_tokens": self._settings["max_completion_tokens"], "max_completion_tokens": self._settings.max_completion_tokens,
} }
# Messages, tools, tool_choice # Messages, tools, tool_choice
params.update(params_from_context) params.update(params_from_context)
params.update(self._settings["extra"]) params.update(self._settings.extra)
return params return params

85
src/pipecat/services/deepgram/stt.py
View File

@@ -6,6 +6,7 @@
"""Deepgram speech-to-text service implementation.""" """Deepgram speech-to-text service implementation."""
from dataclasses import dataclass, field
from typing import AsyncGenerator, Dict, Optional from typing import AsyncGenerator, Dict, Optional
from loguru import logger from loguru import logger
@@ -23,6 +24,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import DEEPGRAM_TTFS_P99 from pipecat.services.stt_latency import DEEPGRAM_TTFS_P99
from pipecat.services.stt_service import STTService from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -45,6 +47,17 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class DeepgramSTTSettings(STTSettings):
"""Typed settings for the Deepgram STT service.
Parameters:
live_options: Deepgram ``LiveOptions`` for detailed configuration.
"""
live_options: LiveOptions = field(default_factory=lambda: NOT_GIVEN)
class DeepgramSTTService(STTService): class DeepgramSTTService(STTService):
"""Deepgram speech-to-text service. """Deepgram speech-to-text service.
@@ -129,11 +142,17 @@ class DeepgramSTTService(STTService):
merged_options["language"] = merged_options["language"].value merged_options["language"] = merged_options["language"].value
self.set_model_name(merged_options["model"]) self.set_model_name(merged_options["model"])
self._settings = merged_options merged_live_options = LiveOptions(**merged_options)
self._settings: DeepgramSTTSettings = DeepgramSTTSettings(
model=merged_options.get("model"),
language=merged_options.get("language"),
live_options=merged_live_options,
)
self._addons = addons self._addons = addons
self._should_interrupt = should_interrupt self._should_interrupt = should_interrupt
if merged_options.get("vad_events"): if merged_live_options.vad_events:
import warnings import warnings
with warnings.catch_warnings(): with warnings.catch_warnings():
@@ -164,7 +183,7 @@ class DeepgramSTTService(STTService):
Returns: Returns:
True if VAD events are enabled in the current settings. True if VAD events are enabled in the current settings.
""" """
return self._settings["vad_events"] return self._settings.live_options.vad_events
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -174,28 +193,48 @@ class DeepgramSTTService(STTService):
""" """
return True return True
async def set_model(self, model: str): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the Deepgram model and reconnect. """Apply a typed settings update, keeping ``live_options`` in sync.
Args: Top-level ``model`` and ``language`` are the source of truth. When
model: The Deepgram model name to use. they are given in *update* their values are propagated into
``live_options``. When only ``live_options`` is given, its ``model``
and ``language`` are propagated *up* to the top-level fields.
Any change triggers a WebSocket reconnect.
""" """
await super().set_model(model) # Determine which top-level fields are explicitly provided.
logger.info(f"Switching STT model to: [{model}]") model_given = isinstance(update, DeepgramSTTSettings) and is_given(
self._settings["model"] = model getattr(update, "model", NOT_GIVEN)
)
language_given = isinstance(update, DeepgramSTTSettings) and is_given(
getattr(update, "language", NOT_GIVEN)
)
changed = await super()._update_settings_from_typed(update)
if not changed:
return changed
# --- Sync model --------------------------------------------------
if model_given:
# Top-level model wins → push into live_options.
self._settings.live_options.model = self._settings.model
elif "live_options" in changed and self._settings.live_options.model is not None:
# Only live_options was given → pull model up.
self._settings.model = self._settings.live_options.model
self.set_model_name(self._settings.model)
# --- Sync language -----------------------------------------------
if language_given:
self._settings.live_options.language = self._settings.language
elif "live_options" in changed and self._settings.live_options.language is not None:
self._settings.language = self._settings.live_options.language
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
async def set_language(self, language: Language): return changed
"""Set the recognition language and reconnect.
Args:
language: The language to use for speech recognition.
"""
logger.info(f"Switching STT language to: [{language}]")
self._settings["language"] = language
await self._disconnect()
await self._connect()
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the Deepgram STT service. """Start the Deepgram STT service.
@@ -204,7 +243,7 @@ class DeepgramSTTService(STTService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["sample_rate"] = self.sample_rate self._settings.live_options.sample_rate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -257,7 +296,9 @@ class DeepgramSTTService(STTService):
self._on_utterance_end, self._on_utterance_end,
) )
if not await self._connection.start(options=self._settings, addons=self._addons): if not await self._connection.start(
options=self._settings.live_options, addons=self._addons
):
await self.push_error(error_msg=f"Unable to connect to Deepgram") await self.push_error(error_msg=f"Unable to connect to Deepgram")
else: else:
headers = { headers = {

83
src/pipecat/services/deepgram/stt_sagemaker.py
View File

@@ -14,6 +14,7 @@ languages, and various Deepgram features.
import asyncio import asyncio
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -31,6 +32,7 @@ from pipecat.frames.frames import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.aws.sagemaker.bidi_client import SageMakerBidiClient from pipecat.services.aws.sagemaker.bidi_client import SageMakerBidiClient
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import DEEPGRAM_SAGEMAKER_TTFS_P99 from pipecat.services.stt_latency import DEEPGRAM_SAGEMAKER_TTFS_P99
from pipecat.services.stt_service import STTService from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -47,6 +49,17 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class DeepgramSageMakerSTTSettings(STTSettings):
"""Typed settings for the Deepgram SageMaker STT service.
Parameters:
live_options: Deepgram ``LiveOptions`` for detailed configuration.
"""
live_options: LiveOptions = field(default_factory=lambda: NOT_GIVEN)
class DeepgramSageMakerSTTService(STTService): class DeepgramSageMakerSTTService(STTService):
"""Deepgram speech-to-text service for AWS SageMaker. """Deepgram speech-to-text service for AWS SageMaker.
@@ -129,7 +142,12 @@ class DeepgramSageMakerSTTService(STTService):
merged_options["language"] = merged_options["language"].value merged_options["language"] = merged_options["language"].value
self.set_model_name(merged_options["model"]) self.set_model_name(merged_options["model"])
self._settings = merged_options merged_live_options = LiveOptions(**merged_options)
self._settings: DeepgramSageMakerSTTSettings = DeepgramSageMakerSTTSettings(
model=merged_options.get("model"),
language=merged_options.get("language"),
live_options=merged_live_options,
)
self._client: Optional[SageMakerBidiClient] = None self._client: Optional[SageMakerBidiClient] = None
self._response_task: Optional[asyncio.Task] = None self._response_task: Optional[asyncio.Task] = None
@@ -143,35 +161,40 @@ class DeepgramSageMakerSTTService(STTService):
""" """
return True return True
async def set_model(self, model: str): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the Deepgram model and reconnect. """Apply a typed settings update, keeping ``live_options`` in sync.
Disconnects from the current session, updates the model setting, and Top-level ``model`` and ``language`` are the source of truth. When
establishes a new connection with the updated model. they change their values are propagated into ``live_options``.
Args: Any change triggers a reconnect.
model: The Deepgram model name to use (e.g., "nova-3").
""" """
await super().set_model(model) model_given = isinstance(update, DeepgramSageMakerSTTSettings) and is_given(
logger.info(f"Switching STT model to: [{model}]") getattr(update, "model", NOT_GIVEN)
self._settings["model"] = model )
await self._disconnect() language_given = isinstance(update, DeepgramSageMakerSTTSettings) and is_given(
await self._connect() getattr(update, "language", NOT_GIVEN)
)
async def set_language(self, language: Language):
"""Set the recognition language and reconnect. changed = await super()._update_settings_from_typed(update)
Disconnects from the current session, updates the language setting, and if not changed:
establishes a new connection with the updated language. return changed
Args: # Sync model into live_options
language: The language to use for speech recognition (e.g., Language.EN, if model_given and "model" in changed:
Language.ES). self._settings.live_options.model = self._settings.model
"""
logger.info(f"Switching STT language to: [{language}]") # Sync language into live_options
self._settings["language"] = language if language_given and "language" in changed:
lang = self._settings.language
if isinstance(lang, Language):
lang = lang.value
self._settings.live_options.language = lang
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the Deepgram SageMaker STT service. """Start the Deepgram SageMaker STT service.
@@ -180,7 +203,7 @@ class DeepgramSageMakerSTTService(STTService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["sample_rate"] = self.sample_rate self._settings.live_options.sample_rate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -226,12 +249,12 @@ class DeepgramSageMakerSTTService(STTService):
""" """
logger.debug("Connecting to Deepgram on SageMaker...") logger.debug("Connecting to Deepgram on SageMaker...")
# Update sample rate in settings # Update sample rate in live_options
self._settings["sample_rate"] = self.sample_rate self._settings.live_options.sample_rate = self.sample_rate
# Build query string from settings, converting booleans to strings # Build query string from live_options, converting booleans to strings
query_params = {} query_params = {}
for key, value in self._settings.items(): for key, value in self._settings.live_options.to_dict().items():
if value is not None: if value is not None:
# Convert boolean values to lowercase strings for Deepgram API # Convert boolean values to lowercase strings for Deepgram API
if isinstance(value, bool): if isinstance(value, bool):

37
src/pipecat/services/deepgram/tts.py
View File

@@ -11,6 +11,7 @@ for generating speech from text using various voice models.
""" """
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
import aiohttp import aiohttp
@@ -29,6 +30,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService, WebsocketTTSService from pipecat.services.tts_service import TTSService, WebsocketTTSService
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -43,6 +45,17 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class DeepgramTTSSettings(TTSSettings):
"""Typed settings for Deepgram TTS service.
Parameters:
encoding: Audio encoding format (linear16, mulaw, alaw).
"""
encoding: str = field(default_factory=lambda: NOT_GIVEN)
class DeepgramTTSService(WebsocketTTSService): class DeepgramTTSService(WebsocketTTSService):
"""Deepgram WebSocket-based text-to-speech service. """Deepgram WebSocket-based text-to-speech service.
@@ -91,10 +104,12 @@ class DeepgramTTSService(WebsocketTTSService):
self._api_key = api_key self._api_key = api_key
self._base_url = base_url self._base_url = base_url
self._settings = { self._settings: DeepgramTTSSettings = DeepgramTTSSettings(
"encoding": encoding, model=voice,
} voice=voice,
self.set_voice(voice) encoding=encoding,
)
self._voice_id = voice
self._receive_task = None self._receive_task = None
self._context_id: Optional[str] = None self._context_id: Optional[str] = None
@@ -177,7 +192,7 @@ class DeepgramTTSService(WebsocketTTSService):
# Build WebSocket URL with query parameters # Build WebSocket URL with query parameters
params = [] params = []
params.append(f"model={self._voice_id}") params.append(f"model={self._voice_id}")
params.append(f"encoding={self._settings['encoding']}") params.append(f"encoding={self._settings.encoding}")
params.append(f"sample_rate={self.sample_rate}") params.append(f"sample_rate={self.sample_rate}")
url = f"{self._base_url}/v1/speak?{'&'.join(params)}" url = f"{self._base_url}/v1/speak?{'&'.join(params)}"
@@ -357,10 +372,12 @@ class DeepgramHttpTTSService(TTSService):
self._api_key = api_key self._api_key = api_key
self._session = aiohttp_session self._session = aiohttp_session
self._base_url = base_url self._base_url = base_url
self._settings = { self._settings: DeepgramTTSSettings = DeepgramTTSSettings(
"encoding": encoding, model=voice,
} voice=voice,
self.set_voice(voice) encoding=encoding,
)
self._voice_id = voice
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if the service can generate metrics. """Check if the service can generate metrics.
@@ -390,7 +407,7 @@ class DeepgramHttpTTSService(TTSService):
params = { params = {
"model": self._voice_id, "model": self._voice_id,
"encoding": self._settings["encoding"], "encoding": self._settings.encoding,
"sample_rate": self.sample_rate, "sample_rate": self.sample_rate,
"container": "none", "container": "none",
} }

12
src/pipecat/services/deepseek/llm.py
View File

@@ -68,15 +68,15 @@ class DeepSeekLLMService(OpenAILLMService):
"model": self.model_name, "model": self.model_name,
"stream": True, "stream": True,
"stream_options": {"include_usage": True}, "stream_options": {"include_usage": True},
"frequency_penalty": self._settings["frequency_penalty"], "frequency_penalty": self._settings.frequency_penalty,
"presence_penalty": self._settings["presence_penalty"], "presence_penalty": self._settings.presence_penalty,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"max_tokens": self._settings["max_tokens"], "max_tokens": self._settings.max_tokens,
} }
# Messages, tools, tool_choice # Messages, tools, tool_choice
params.update(params_from_context) params.update(params_from_context)
params.update(self._settings["extra"]) params.update(self._settings.extra)
return params return params

203
src/pipecat/services/elevenlabs/stt.py
View File

@@ -14,6 +14,7 @@ transcription results directly.
import base64 import base64
import io import io
import json import json
from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
@@ -33,6 +34,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import ELEVENLABS_REALTIME_TTFS_P99, ELEVENLABS_TTFS_P99 from pipecat.services.stt_latency import ELEVENLABS_REALTIME_TTFS_P99, ELEVENLABS_TTFS_P99
from pipecat.services.stt_service import SegmentedSTTService, WebsocketSTTService from pipecat.services.stt_service import SegmentedSTTService, WebsocketSTTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -167,6 +169,44 @@ def language_to_elevenlabs_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class ElevenLabsSTTSettings(STTSettings):
"""Typed settings for the ElevenLabs file-based STT service.
Parameters:
tag_audio_events: Whether to include audio event tags in transcription.
"""
tag_audio_events: bool = field(default_factory=lambda: NOT_GIVEN)
@dataclass
class ElevenLabsRealtimeSTTSettings(STTSettings):
"""Typed settings for the ElevenLabs Realtime STT service.
See ``ElevenLabsRealtimeSTTService.InputParams`` for detailed descriptions.
Parameters:
commit_strategy: How to segment speech - manual (Pipecat VAD) or vad (ElevenLabs VAD).
vad_silence_threshold_secs: Seconds of silence before VAD commits (0.3-3.0).
vad_threshold: VAD sensitivity (0.1-0.9, lower is more sensitive).
min_speech_duration_ms: Minimum speech duration for VAD (50-2000ms).
min_silence_duration_ms: Minimum silence duration for VAD (50-2000ms).
include_timestamps: Whether to include word-level timestamps in transcripts.
enable_logging: Whether to enable logging on ElevenLabs' side.
include_language_detection: Whether to include language detection in transcripts.
"""
commit_strategy: CommitStrategy = field(default_factory=lambda: NOT_GIVEN)
vad_silence_threshold_secs: float = field(default_factory=lambda: NOT_GIVEN)
vad_threshold: float = field(default_factory=lambda: NOT_GIVEN)
min_speech_duration_ms: int = field(default_factory=lambda: NOT_GIVEN)
min_silence_duration_ms: int = field(default_factory=lambda: NOT_GIVEN)
include_timestamps: bool = field(default_factory=lambda: NOT_GIVEN)
enable_logging: bool = field(default_factory=lambda: NOT_GIVEN)
include_language_detection: bool = field(default_factory=lambda: NOT_GIVEN)
class ElevenLabsSTTService(SegmentedSTTService): class ElevenLabsSTTService(SegmentedSTTService):
"""Speech-to-text service using ElevenLabs' file-based API. """Speech-to-text service using ElevenLabs' file-based API.
@@ -223,13 +263,15 @@ class ElevenLabsSTTService(SegmentedSTTService):
self._base_url = base_url self._base_url = base_url
self._session = aiohttp_session self._session = aiohttp_session
self._model_id = model self._model_id = model
self._tag_audio_events = params.tag_audio_events
self._settings = { self._settings: ElevenLabsSTTSettings = ElevenLabsSTTSettings(
"language": self.language_to_service_language(params.language) model=model,
language=self.language_to_service_language(params.language)
if params.language if params.language
else "eng", else "eng",
} tag_audio_events=params.tag_audio_events,
)
self.set_model_name(model)
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if the service can generate processing metrics. """Check if the service can generate processing metrics.
@@ -250,27 +292,30 @@ class ElevenLabsSTTService(SegmentedSTTService):
""" """
return language_to_elevenlabs_language(language) return language_to_elevenlabs_language(language)
async def set_language(self, language: Language): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the transcription language. """Apply a typed settings update.
Converts language to ElevenLabs format before applying and keeps
``_model_id`` in sync with the model setting.
Args: Args:
language: The language to use for speech-to-text transcription. update: A :class:`STTSettings` (or ``ElevenLabsSTTSettings``) delta.
Returns:
Set of field names whose values actually changed.
""" """
logger.info(f"Switching STT language to: [{language}]") # Convert language to ElevenLabs format before applying
self._settings["language"] = self.language_to_service_language(language) if is_given(update.language) and isinstance(update.language, Language):
converted = self.language_to_service_language(update.language)
if converted is not None:
update.language = converted
async def set_model(self, model: str): changed = await super()._update_settings_from_typed(update)
"""Set the STT model.
Args: if "model" in changed:
model: The model name to use for transcription. self._model_id = self._settings.model
Note: return changed
ElevenLabs STT API does not currently support model selection.
This method is provided for interface compatibility.
"""
await super().set_model(model)
logger.info(f"Model setting [{model}] noted, but ElevenLabs STT uses default model")
async def _transcribe_audio(self, audio_data: bytes) -> dict: async def _transcribe_audio(self, audio_data: bytes) -> dict:
"""Upload audio data to ElevenLabs and get transcription result. """Upload audio data to ElevenLabs and get transcription result.
@@ -298,8 +343,8 @@ class ElevenLabsSTTService(SegmentedSTTService):
# Add required model_id, language_code, and tag_audio_events # Add required model_id, language_code, and tag_audio_events
data.add_field("model_id", self._model_id) data.add_field("model_id", self._model_id)
data.add_field("language_code", self._settings["language"]) data.add_field("language_code", self._settings.language)
data.add_field("tag_audio_events", str(self._tag_audio_events).lower()) data.add_field("tag_audio_events", str(self._settings.tag_audio_events).lower())
async with self._session.post(url, data=data, headers=headers) as response: async with self._session.post(url, data=data, headers=headers) as response:
if response.status != 200: if response.status != 200:
@@ -469,11 +514,22 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
self._api_key = api_key self._api_key = api_key
self._base_url = base_url self._base_url = base_url
self._model_id = model self._model_id = model
self._params = params
self._audio_format = "" # initialized in start() self._audio_format = "" # initialized in start()
self._receive_task = None self._receive_task = None
self._settings = {"language": params.language_code} self._settings: ElevenLabsRealtimeSTTSettings = ElevenLabsRealtimeSTTSettings(
model=model,
language=params.language_code,
commit_strategy=params.commit_strategy,
vad_silence_threshold_secs=params.vad_silence_threshold_secs,
vad_threshold=params.vad_threshold,
min_speech_duration_ms=params.min_speech_duration_ms,
min_silence_duration_ms=params.min_silence_duration_ms,
include_timestamps=params.include_timestamps,
enable_logging=params.enable_logging,
include_language_detection=params.include_language_detection,
)
self.set_model_name(model)
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if the service can generate processing metrics. """Check if the service can generate processing metrics.
@@ -483,42 +539,35 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
""" """
return True return True
async def set_language(self, language: Language): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the transcription language. """Apply a typed settings update and reconnect if anything changed.
Converts language to ElevenLabs format before applying and keeps
``_model_id`` in sync.
Args: Args:
language: The language to use for speech-to-text transcription. update: A :class:`STTSettings` (or ``ElevenLabsRealtimeSTTSettings``) delta.
Note: Returns:
Changing language requires reconnecting to the WebSocket. Set of field names whose values actually changed.
""" """
logger.info(f"Switching STT language to: [{language}]") # Convert language to ElevenLabs format before applying
new_language = ( if is_given(update.language) and isinstance(update.language, Language):
language_to_elevenlabs_language(language) converted = language_to_elevenlabs_language(update.language)
if isinstance(language, Language) if converted is not None:
else language update.language = converted
)
self._params.language_code = new_language changed = await super()._update_settings_from_typed(update)
self._settings["language"] = new_language
# Reconnect with new settings if not changed:
await self._disconnect() return changed
await self._connect()
if "model" in changed:
async def set_model(self, model: str): self._model_id = self._settings.model
"""Set the STT model.
Args:
model: The model name to use for transcription.
Note:
Changing model requires reconnecting to the WebSocket.
"""
await super().set_model(model)
logger.info(f"Switching STT model to: [{model}]")
self._model_id = model
# Reconnect with new settings
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the STT service and establish WebSocket connection. """Start the STT service and establish WebSocket connection.
@@ -566,7 +615,7 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
await self._start_metrics() await self._start_metrics()
elif isinstance(frame, VADUserStoppedSpeakingFrame): elif isinstance(frame, VADUserStoppedSpeakingFrame):
# Send commit when user stops speaking (manual commit mode) # Send commit when user stops speaking (manual commit mode)
if self._params.commit_strategy == CommitStrategy.MANUAL: if self._settings.commit_strategy == CommitStrategy.MANUAL:
if self._websocket and self._websocket.state is State.OPEN: if self._websocket and self._websocket.state is State.OPEN:
try: try:
commit_message = { commit_message = {
@@ -656,36 +705,40 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
# Build query parameters # Build query parameters
params = [f"model_id={self._model_id}"] params = [f"model_id={self._model_id}"]
if self._params.language_code: if self._settings.language:
params.append(f"language_code={self._params.language_code}") params.append(f"language_code={self._settings.language}")
params.append(f"audio_format={self._audio_format}") params.append(f"audio_format={self._audio_format}")
params.append(f"commit_strategy={self._params.commit_strategy.value}") params.append(f"commit_strategy={self._settings.commit_strategy.value}")
# Add optional parameters # Add optional parameters
if self._params.include_timestamps: if self._settings.include_timestamps:
params.append(f"include_timestamps={str(self._params.include_timestamps).lower()}")
if self._params.enable_logging:
params.append(f"enable_logging={str(self._params.enable_logging).lower()}")
if self._params.include_language_detection:
params.append( params.append(
f"include_language_detection={str(self._params.include_language_detection).lower()}" f"include_timestamps={str(self._settings.include_timestamps).lower()}"
)
if self._settings.enable_logging:
params.append(f"enable_logging={str(self._settings.enable_logging).lower()}")
if self._settings.include_language_detection:
params.append(
f"include_language_detection={str(self._settings.include_language_detection).lower()}"
) )
# Add VAD parameters if using VAD commit strategy and values are specified # Add VAD parameters if using VAD commit strategy and values are specified
if self._params.commit_strategy == CommitStrategy.VAD: if self._settings.commit_strategy == CommitStrategy.VAD:
if self._params.vad_silence_threshold_secs is not None: if self._settings.vad_silence_threshold_secs is not None:
params.append( params.append(
f"vad_silence_threshold_secs={self._params.vad_silence_threshold_secs}" f"vad_silence_threshold_secs={self._settings.vad_silence_threshold_secs}"
)
if self._settings.vad_threshold is not None:
params.append(f"vad_threshold={self._settings.vad_threshold}")
if self._settings.min_speech_duration_ms is not None:
params.append(f"min_speech_duration_ms={self._settings.min_speech_duration_ms}")
if self._settings.min_silence_duration_ms is not None:
params.append(
f"min_silence_duration_ms={self._settings.min_silence_duration_ms}"
) )
if self._params.vad_threshold is not None:
params.append(f"vad_threshold={self._params.vad_threshold}")
if self._params.min_speech_duration_ms is not None:
params.append(f"min_speech_duration_ms={self._params.min_speech_duration_ms}")
if self._params.min_silence_duration_ms is not None:
params.append(f"min_silence_duration_ms={self._params.min_silence_duration_ms}")
ws_url = f"wss://{self._base_url}/v1/speech-to-text/realtime?{'&'.join(params)}" ws_url = f"wss://{self._base_url}/v1/speech-to-text/realtime?{'&'.join(params)}"
@@ -817,7 +870,7 @@ class ElevenLabsRealtimeSTTService(WebsocketSTTService):
""" """
# If timestamps are enabled, skip this message and wait for the # If timestamps are enabled, skip this message and wait for the
# committed_transcript_with_timestamps message which contains all the data # committed_transcript_with_timestamps message which contains all the data
if self._params.include_timestamps: if self._settings.include_timestamps:
return return
text = data.get("text", "").strip() text = data.get("text", "").strip()

272
src/pipecat/services/elevenlabs/tts.py
View File

@@ -13,7 +13,19 @@ with support for streaming audio, word timestamps, and voice customization.
import asyncio import asyncio
import base64 import base64
import json import json
from typing import Any, AsyncGenerator, Dict, List, Literal, Mapping, Optional, Tuple, Union from dataclasses import dataclass, field
from typing import (
Any,
AsyncGenerator,
ClassVar,
Dict,
List,
Literal,
Mapping,
Optional,
Tuple,
Union,
)
import aiohttp import aiohttp
from loguru import logger from loguru import logger
@@ -32,6 +44,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
from pipecat.services.tts_service import ( from pipecat.services.tts_service import (
AudioContextWordTTSService, AudioContextWordTTSService,
WordTTSService, WordTTSService,
@@ -136,12 +149,12 @@ def output_format_from_sample_rate(sample_rate: int) -> str:
def build_elevenlabs_voice_settings( def build_elevenlabs_voice_settings(
settings: Dict[str, Any], settings: Union[Dict[str, Any], "TTSSettings"],
) -> Optional[Dict[str, Union[float, bool]]]: ) -> Optional[Dict[str, Union[float, bool]]]:
"""Build voice settings dictionary for ElevenLabs based on provided settings. """Build voice settings dictionary for ElevenLabs based on provided settings.
Args: Args:
settings: Dictionary containing voice settings parameters. settings: Dictionary or typed settings containing voice settings parameters.
Returns: Returns:
Dictionary of voice settings or None if no valid settings are provided. Dictionary of voice settings or None if no valid settings are provided.
@@ -150,8 +163,11 @@ def build_elevenlabs_voice_settings(
voice_settings = {} voice_settings = {}
for key in voice_setting_keys: for key in voice_setting_keys:
if key in settings and settings[key] is not None: val = (
voice_settings[key] = settings[key] getattr(settings, key, None) if isinstance(settings, TTSSettings) else settings.get(key)
)
if val is not None and is_given(val):
voice_settings[key] = val
return voice_settings or None return voice_settings or None
@@ -168,6 +184,75 @@ class PronunciationDictionaryLocator(BaseModel):
version_id: str version_id: str
@dataclass
class ElevenLabsTTSSettings(TTSSettings):
"""Typed settings for the ElevenLabs WebSocket TTS service.
Fields that appear in the WebSocket URL (``voice``, ``model``,
``language``) require a full reconnect when changed. Fields that
affect the voice character (``stability``, ``similarity_boost``,
``style``, ``use_speaker_boost``, ``speed``) can be applied by closing
the current audio context so a new one is opened with updated settings.
Parameters:
stability: Voice stability control (0.0 to 1.0).
similarity_boost: Similarity boost control (0.0 to 1.0).
style: Style control for voice expression (0.0 to 1.0).
use_speaker_boost: Whether to use speaker boost enhancement.
speed: Voice speed control (0.7 to 1.2).
auto_mode: Whether to enable automatic mode optimization.
enable_ssml_parsing: Whether to parse SSML tags in text.
enable_logging: Whether to enable ElevenLabs logging.
apply_text_normalization: Text normalization mode ("auto", "on", "off").
"""
stability: float = field(default_factory=lambda: NOT_GIVEN)
similarity_boost: float = field(default_factory=lambda: NOT_GIVEN)
style: float = field(default_factory=lambda: NOT_GIVEN)
use_speaker_boost: bool = field(default_factory=lambda: NOT_GIVEN)
speed: float = field(default_factory=lambda: NOT_GIVEN)
auto_mode: str = field(default_factory=lambda: NOT_GIVEN)
enable_ssml_parsing: bool = field(default_factory=lambda: NOT_GIVEN)
enable_logging: bool = field(default_factory=lambda: NOT_GIVEN)
apply_text_normalization: str = field(default_factory=lambda: NOT_GIVEN)
#: Fields in the WS URL — changing any of these requires a reconnect.
URL_FIELDS: ClassVar[frozenset[str]] = frozenset({"voice", "model", "language"})
#: Fields affecting voice character — changing these requires closing the
#: current audio context so the next one picks up new settings.
VOICE_SETTINGS_FIELDS: ClassVar[frozenset[str]] = frozenset(
{"stability", "similarity_boost", "style", "use_speaker_boost", "speed"}
)
_aliases: ClassVar[Dict[str, str]] = {"voice_id": "voice"}
@dataclass
class ElevenLabsHttpTTSSettings(TTSSettings):
"""Typed settings for the ElevenLabs HTTP TTS service.
Parameters:
optimize_streaming_latency: Latency optimization level (0-4).
stability: Voice stability control (0.0 to 1.0).
similarity_boost: Similarity boost control (0.0 to 1.0).
style: Style control for voice expression (0.0 to 1.0).
use_speaker_boost: Whether to use speaker boost enhancement.
speed: Voice speed control (0.25 to 4.0).
apply_text_normalization: Text normalization mode ("auto", "on", "off").
"""
optimize_streaming_latency: int = field(default_factory=lambda: NOT_GIVEN)
stability: float = field(default_factory=lambda: NOT_GIVEN)
similarity_boost: float = field(default_factory=lambda: NOT_GIVEN)
style: float = field(default_factory=lambda: NOT_GIVEN)
use_speaker_boost: bool = field(default_factory=lambda: NOT_GIVEN)
speed: float = field(default_factory=lambda: NOT_GIVEN)
apply_text_normalization: str = field(default_factory=lambda: NOT_GIVEN)
_aliases: ClassVar[Dict[str, str]] = {"voice_id": "voice"}
def calculate_word_times( def calculate_word_times(
alignment_info: Mapping[str, Any], alignment_info: Mapping[str, Any],
cumulative_time: float, cumulative_time: float,
@@ -316,22 +401,25 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
self._api_key = api_key self._api_key = api_key
self._url = url self._url = url
self._settings = { self._settings: ElevenLabsTTSSettings = ElevenLabsTTSSettings(
"language": self.language_to_service_language(params.language) model=model,
if params.language voice=voice_id,
else None, language=(
"stability": params.stability, self.language_to_service_language(params.language) if params.language else None
"similarity_boost": params.similarity_boost, ),
"style": params.style, stability=params.stability,
"use_speaker_boost": params.use_speaker_boost, similarity_boost=params.similarity_boost,
"speed": params.speed, style=params.style,
"auto_mode": str(params.auto_mode).lower(), use_speaker_boost=params.use_speaker_boost,
"enable_ssml_parsing": params.enable_ssml_parsing, speed=params.speed,
"enable_logging": params.enable_logging, auto_mode=str(params.auto_mode).lower(),
"apply_text_normalization": params.apply_text_normalization, enable_ssml_parsing=params.enable_ssml_parsing,
} enable_logging=params.enable_logging,
apply_text_normalization=params.apply_text_normalization,
)
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice_id) self._voice_id = voice_id
self._output_format = "" # initialized in start() self._output_format = "" # initialized in start()
self._voice_settings = self._set_voice_settings() self._voice_settings = self._set_voice_settings()
self._pronunciation_dictionary_locators = params.pronunciation_dictionary_locators self._pronunciation_dictionary_locators = params.pronunciation_dictionary_locators
@@ -366,54 +454,57 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
return language_to_elevenlabs_language(language) return language_to_elevenlabs_language(language)
def _set_voice_settings(self): def _set_voice_settings(self):
return build_elevenlabs_voice_settings(self._settings) ts = self._settings
voice_setting_keys = [
"stability",
"similarity_boost",
"style",
"use_speaker_boost",
"speed",
]
voice_settings = {}
for key in voice_setting_keys:
val = getattr(ts, key, None)
if val is not None and is_given(val):
voice_settings[key] = val
return voice_settings or None
async def set_model(self, model: str): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Set the TTS model and reconnect. """Apply a typed settings update, reconnecting as needed.
Uses the declarative ``URL_FIELDS`` and ``VOICE_SETTINGS_FIELDS``
sets on :class:`ElevenLabsTTSSettings` to decide whether to
reconnect the WebSocket or close the current audio context.
Args: Args:
model: The model name to use for synthesis. update: A :class:`TTSSettings` (or ``ElevenLabsTTSSettings``) delta.
Returns:
Set of field names whose values actually changed.
""" """
await super().set_model(model) changed = await super()._update_settings_from_typed(update)
logger.info(f"Switching TTS model to: [{model}]")
await self._disconnect()
await self._connect()
async def _update_settings(self, settings: Mapping[str, Any]): if not changed:
"""Update service settings and reconnect if voice, model, or language changed.""" return changed
# Track previous values for settings that require reconnection
prev_voice = self._voice_id
prev_model = self.model_name
prev_language = self._settings.get("language")
# Create snapshot of current voice settings to detect changes after update
prev_voice_settings = self._voice_settings.copy() if self._voice_settings else None
await super()._update_settings(settings) # Rebuild voice settings for next context
# Update voice settings for the next context creation
self._voice_settings = self._set_voice_settings() self._voice_settings = self._set_voice_settings()
# Check if URL-level settings changed (these require reconnection) url_changed = bool(changed & ElevenLabsTTSSettings.URL_FIELDS)
url_changed = ( voice_settings_changed = bool(changed & ElevenLabsTTSSettings.VOICE_SETTINGS_FIELDS)
prev_voice != self._voice_id
or prev_model != self.model_name
or prev_language != self._settings.get("language")
)
# Check if only voice settings changed (speed, stability, etc.)
voice_settings_changed = prev_voice_settings != self._voice_settings
if url_changed: if url_changed:
# These settings are in the WebSocket URL, so we need to reconnect
logger.debug( logger.debug(
f"URL-level setting changed (voice/model/language), reconnecting WebSocket" f"URL-level setting changed ({changed & ElevenLabsTTSSettings.URL_FIELDS}), "
f"reconnecting WebSocket"
) )
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
elif voice_settings_changed and self._context_id: elif voice_settings_changed and self._context_id:
# Voice settings can be updated by closing current context logger.debug(
# so new one gets created with updated voice settings f"Voice settings changed ({changed & ElevenLabsTTSSettings.VOICE_SETTINGS_FIELDS}), "
logger.debug(f"Voice settings changed, closing current context to apply changes") f"closing current context to apply changes"
)
try: try:
if self._websocket: if self._websocket:
await self._websocket.send( await self._websocket.send(
@@ -423,6 +514,8 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e) await self.push_error(error_msg=f"Unknown error occurred: {e}", exception=e)
self._context_id = None self._context_id = None
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the ElevenLabs TTS service. """Start the ElevenLabs TTS service.
@@ -505,19 +598,19 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
voice_id = self._voice_id voice_id = self._voice_id
model = self.model_name model = self.model_name
output_format = self._output_format output_format = self._output_format
url = f"{self._url}/v1/text-to-speech/{voice_id}/multi-stream-input?model_id={model}&output_format={output_format}&auto_mode={self._settings['auto_mode']}" url = f"{self._url}/v1/text-to-speech/{voice_id}/multi-stream-input?model_id={model}&output_format={output_format}&auto_mode={self._settings.auto_mode}"
if self._settings["enable_ssml_parsing"]: if self._settings.enable_ssml_parsing:
url += f"&enable_ssml_parsing={self._settings['enable_ssml_parsing']}" url += f"&enable_ssml_parsing={self._settings.enable_ssml_parsing}"
if self._settings["enable_logging"]: if self._settings.enable_logging:
url += f"&enable_logging={self._settings['enable_logging']}" url += f"&enable_logging={self._settings.enable_logging}"
if self._settings["apply_text_normalization"] is not None: if self._settings.apply_text_normalization is not None:
url += f"&apply_text_normalization={self._settings['apply_text_normalization']}" url += f"&apply_text_normalization={self._settings.apply_text_normalization}"
# Language can only be used with the ELEVENLABS_MULTILINGUAL_MODELS # Language can only be used with the ELEVENLABS_MULTILINGUAL_MODELS
language = self._settings["language"] language = self._settings.language
if model in ELEVENLABS_MULTILINGUAL_MODELS and language is not None: if model in ELEVENLABS_MULTILINGUAL_MODELS and language is not None:
url += f"&language_code={language}" url += f"&language_code={language}"
logger.debug(f"Using language code: {language}") logger.debug(f"Using language code: {language}")
@@ -809,20 +902,22 @@ class ElevenLabsHttpTTSService(WordTTSService):
self._params = params self._params = params
self._session = aiohttp_session self._session = aiohttp_session
self._settings = { self._settings: ElevenLabsHttpTTSSettings = ElevenLabsHttpTTSSettings(
"language": self.language_to_service_language(params.language) model=model,
voice=voice_id,
language=self.language_to_service_language(params.language)
if params.language if params.language
else None, else None,
"optimize_streaming_latency": params.optimize_streaming_latency, optimize_streaming_latency=params.optimize_streaming_latency,
"stability": params.stability, stability=params.stability,
"similarity_boost": params.similarity_boost, similarity_boost=params.similarity_boost,
"style": params.style, style=params.style,
"use_speaker_boost": params.use_speaker_boost, use_speaker_boost=params.use_speaker_boost,
"speed": params.speed, speed=params.speed,
"apply_text_normalization": params.apply_text_normalization, apply_text_normalization=params.apply_text_normalization,
} )
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice_id) self._voice_id = voice_id
self._output_format = "" # initialized in start() self._output_format = "" # initialized in start()
self._voice_settings = self._set_voice_settings() self._voice_settings = self._set_voice_settings()
self._pronunciation_dictionary_locators = params.pronunciation_dictionary_locators self._pronunciation_dictionary_locators = params.pronunciation_dictionary_locators
@@ -859,10 +954,19 @@ class ElevenLabsHttpTTSService(WordTTSService):
def _set_voice_settings(self): def _set_voice_settings(self):
return build_elevenlabs_voice_settings(self._settings) return build_elevenlabs_voice_settings(self._settings)
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
await super()._update_settings(settings) """Apply a typed settings update and rebuild voice settings.
# Update voice settings for the next context creation
self._voice_settings = self._set_voice_settings() Args:
update: A :class:`TTSSettings` (or ``ElevenLabsHttpTTSSettings``) delta.
Returns:
Set of field names whose values actually changed.
"""
changed = await super()._update_settings_from_typed(update)
if changed:
self._voice_settings = self._set_voice_settings()
return changed
def _reset_state(self): def _reset_state(self):
"""Reset internal state variables.""" """Reset internal state variables."""
@@ -999,10 +1103,13 @@ class ElevenLabsHttpTTSService(WordTTSService):
locator.model_dump() for locator in self._pronunciation_dictionary_locators locator.model_dump() for locator in self._pronunciation_dictionary_locators
] ]
if self._settings["apply_text_normalization"] is not None: if (
payload["apply_text_normalization"] = self._settings["apply_text_normalization"] is_given(self._settings.apply_text_normalization)
and self._settings.apply_text_normalization is not None
):
payload["apply_text_normalization"] = self._settings.apply_text_normalization
language = self._settings["language"] language = self._settings.language
if self._model_name in ELEVENLABS_MULTILINGUAL_MODELS and language: if self._model_name in ELEVENLABS_MULTILINGUAL_MODELS and language:
payload["language_code"] = language payload["language_code"] = language
logger.debug(f"Using language code: {language}") logger.debug(f"Using language code: {language}")
@@ -1020,8 +1127,11 @@ class ElevenLabsHttpTTSService(WordTTSService):
params = { params = {
"output_format": self._output_format, "output_format": self._output_format,
} }
if self._settings["optimize_streaming_latency"] is not None: if (
params["optimize_streaming_latency"] = self._settings["optimize_streaming_latency"] is_given(self._settings.optimize_streaming_latency)
and self._settings.optimize_streaming_latency is not None
):
params["optimize_streaming_latency"] = self._settings.optimize_streaming_latency
try: try:
await self.start_ttfb_metrics() await self.start_ttfb_metrics()

60
src/pipecat/services/fal/stt.py
View File

@@ -11,12 +11,14 @@ transcription using segmented audio processing.
""" """
import os import os
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
from pydantic import BaseModel from pydantic import BaseModel
from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import FAL_TTFS_P99 from pipecat.services.stt_latency import FAL_TTFS_P99
from pipecat.services.stt_service import SegmentedSTTService from pipecat.services.stt_service import SegmentedSTTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -146,6 +148,22 @@ def language_to_fal_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class FalSTTSettings(STTSettings):
"""Typed settings for the Fal Wizper STT service.
Parameters:
task: Task to perform ('transcribe' or 'translate'). Defaults to
'transcribe'.
chunk_level: Level of chunking ('segment'). Defaults to 'segment'.
version: Version of Wizper model to use. Defaults to '3'.
"""
task: str = field(default_factory=lambda: NOT_GIVEN)
chunk_level: str = field(default_factory=lambda: NOT_GIVEN)
version: str = field(default_factory=lambda: NOT_GIVEN)
class FalSTTService(SegmentedSTTService): class FalSTTService(SegmentedSTTService):
"""Speech-to-text service using Fal's Wizper API. """Speech-to-text service using Fal's Wizper API.
@@ -203,14 +221,14 @@ class FalSTTService(SegmentedSTTService):
) )
self._fal_client = fal_client.AsyncClient(key=api_key or os.getenv("FAL_KEY")) self._fal_client = fal_client.AsyncClient(key=api_key or os.getenv("FAL_KEY"))
self._settings = { self._settings: FalSTTSettings = FalSTTSettings(
"task": params.task, language=self.language_to_service_language(params.language)
"language": self.language_to_service_language(params.language)
if params.language if params.language
else "en", else "en",
"chunk_level": params.chunk_level, task=params.task,
"version": params.version, chunk_level=params.chunk_level,
} version=params.version,
)
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if the service can generate processing metrics. """Check if the service can generate processing metrics.
@@ -231,23 +249,17 @@ class FalSTTService(SegmentedSTTService):
""" """
return language_to_fal_language(language) return language_to_fal_language(language)
async def set_language(self, language: Language): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the transcription language. """Apply a typed settings update, converting language if changed."""
changed = await super()._update_settings_from_typed(update)
Args: if "language" in changed:
language: The language to use for speech-to-text transcription. # Convert the Language enum to a Fal language code.
""" lang = self._settings.language
logger.info(f"Switching STT language to: [{language}]") if isinstance(lang, Language):
self._settings["language"] = self.language_to_service_language(language) self._settings.language = self.language_to_service_language(lang)
async def set_model(self, model: str): return changed
"""Set the STT model.
Args:
model: The model name to use for transcription.
"""
await super().set_model(model)
logger.info(f"Switching STT model to: [{model}]")
@traced_stt @traced_stt
async def _handle_transcription( async def _handle_transcription(
@@ -276,19 +288,19 @@ class FalSTTService(SegmentedSTTService):
data_uri = fal_client.encode(audio, "audio/x-wav") data_uri = fal_client.encode(audio, "audio/x-wav")
response = await self._fal_client.run( response = await self._fal_client.run(
"fal-ai/wizper", "fal-ai/wizper",
arguments={"audio_url": data_uri, **self._settings}, arguments={"audio_url": data_uri, **self._settings.given_fields()},
) )
if response and "text" in response: if response and "text" in response:
text = response["text"].strip() text = response["text"].strip()
if text: # Only yield non-empty text if text: # Only yield non-empty text
await self._handle_transcription(text, True, self._settings["language"]) await self._handle_transcription(text, True, self._settings.language)
logger.debug(f"Transcription: [{text}]") logger.debug(f"Transcription: [{text}]")
yield TranscriptionFrame( yield TranscriptionFrame(
text, text,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
Language(self._settings["language"]), Language(self._settings.language),
result=response, result=response,
) )

12
src/pipecat/services/fireworks/llm.py
View File

@@ -68,15 +68,15 @@ class FireworksLLMService(OpenAILLMService):
params = { params = {
"model": self.model_name, "model": self.model_name,
"stream": True, "stream": True,
"frequency_penalty": self._settings["frequency_penalty"], "frequency_penalty": self._settings.frequency_penalty,
"presence_penalty": self._settings["presence_penalty"], "presence_penalty": self._settings.presence_penalty,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"max_tokens": self._settings["max_tokens"], "max_tokens": self._settings.max_tokens,
} }
# Messages, tools, tool_choice # Messages, tools, tool_choice
params.update(params_from_context) params.update(params_from_context)
params.update(self._settings["extra"]) params.update(self._settings.extra)
return params return params

81
src/pipecat/services/fish/tts.py
View File

@@ -11,6 +11,7 @@ for streaming text-to-speech synthesis with customizable voice parameters.
""" """
import uuid import uuid
from dataclasses import dataclass, field
from typing import AsyncGenerator, Literal, Optional from typing import AsyncGenerator, Literal, Optional
from loguru import logger from loguru import logger
@@ -28,6 +29,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import InterruptibleTTSService from pipecat.services.tts_service import InterruptibleTTSService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -45,6 +47,29 @@ except ModuleNotFoundError as e:
FishAudioOutputFormat = Literal["opus", "mp3", "pcm", "wav"] FishAudioOutputFormat = Literal["opus", "mp3", "pcm", "wav"]
@dataclass
class FishAudioTTSSettings(TTSSettings):
"""Typed settings for Fish Audio TTS service.
Parameters:
fish_sample_rate: Audio sample rate sent to the API.
latency: Latency mode ("normal" or "balanced"). Defaults to "normal".
format: Audio output format.
normalize: Whether to normalize audio output. Defaults to True.
prosody_speed: Speech speed multiplier (0.5-2.0). Defaults to 1.0.
prosody_volume: Volume adjustment in dB. Defaults to 0.
reference_id: Reference ID of the voice model.
"""
fish_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
latency: str = field(default_factory=lambda: NOT_GIVEN)
format: str = field(default_factory=lambda: NOT_GIVEN)
normalize: bool = field(default_factory=lambda: NOT_GIVEN)
prosody_speed: float = field(default_factory=lambda: NOT_GIVEN)
prosody_volume: int = field(default_factory=lambda: NOT_GIVEN)
reference_id: str = field(default_factory=lambda: NOT_GIVEN)
class FishAudioTTSService(InterruptibleTTSService): class FishAudioTTSService(InterruptibleTTSService):
"""Fish Audio text-to-speech service with WebSocket streaming. """Fish Audio text-to-speech service with WebSocket streaming.
@@ -136,17 +161,16 @@ class FishAudioTTSService(InterruptibleTTSService):
self._receive_task = None self._receive_task = None
self._request_id = None self._request_id = None
self._settings = { self._settings: FishAudioTTSSettings = FishAudioTTSSettings(
"sample_rate": 0, voice=reference_id,
"latency": params.latency, fish_sample_rate=0,
"format": output_format, latency=params.latency,
"normalize": params.normalize, format=output_format,
"prosody": { normalize=params.normalize,
"speed": params.prosody_speed, prosody_speed=params.prosody_speed,
"volume": params.prosody_volume, prosody_volume=params.prosody_volume,
}, reference_id=reference_id,
"reference_id": reference_id, )
}
self.set_model_name(model_id) self.set_model_name(model_id)
@@ -158,16 +182,22 @@ class FishAudioTTSService(InterruptibleTTSService):
""" """
return True return True
async def set_model(self, model: str): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Set the TTS model and reconnect. """Apply a typed settings update and reconnect if needed.
Any change to voice or model triggers a WebSocket reconnect.
Args: Args:
model: The model name to use for synthesis. update: A :class:`TTSSettings` (or ``FishAudioTTSSettings``) delta.
Returns:
Set of field names whose values actually changed.
""" """
await super().set_model(model) changed = await super()._update_settings_from_typed(update)
logger.info(f"Switching TTS model to: [{model}]") if changed:
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the Fish Audio TTS service. """Start the Fish Audio TTS service.
@@ -176,7 +206,7 @@ class FishAudioTTSService(InterruptibleTTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["sample_rate"] = self.sample_rate self._settings.fish_sample_rate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -225,7 +255,18 @@ class FishAudioTTSService(InterruptibleTTSService):
self._websocket = await websocket_connect(self._base_url, additional_headers=headers) self._websocket = await websocket_connect(self._base_url, additional_headers=headers)
# Send initial start message with ormsgpack # Send initial start message with ormsgpack
start_message = {"event": "start", "request": {"text": "", **self._settings}} request_settings = {
"sample_rate": self._settings.fish_sample_rate,
"latency": self._settings.latency,
"format": self._settings.format,
"normalize": self._settings.normalize,
"prosody": {
"speed": self._settings.prosody_speed,
"volume": self._settings.prosody_volume,
},
"reference_id": self._settings.reference_id,
}
start_message = {"event": "start", "request": {"text": "", **request_settings}}
await self._websocket.send(ormsgpack.packb(start_message)) await self._websocket.send(ormsgpack.packb(start_message))
logger.debug("Sent start message to Fish Audio") logger.debug("Sent start message to Fish Audio")

86
src/pipecat/services/gladia/stt.py
View File

@@ -14,6 +14,7 @@ import asyncio
import base64 import base64
import json import json
import warnings import warnings
from dataclasses import dataclass, field
from typing import Any, AsyncGenerator, Dict, Literal, Optional from typing import Any, AsyncGenerator, Dict, Literal, Optional
import aiohttp import aiohttp
@@ -32,6 +33,7 @@ from pipecat.frames.frames import (
UserStoppedSpeakingFrame, UserStoppedSpeakingFrame,
) )
from pipecat.services.gladia.config import GladiaInputParams from pipecat.services.gladia.config import GladiaInputParams
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import GLADIA_TTFS_P99 from pipecat.services.stt_latency import GLADIA_TTFS_P99
from pipecat.services.stt_service import WebsocketSTTService from pipecat.services.stt_service import WebsocketSTTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -178,6 +180,17 @@ class _InputParamsDescriptor:
return GladiaInputParams return GladiaInputParams
@dataclass
class GladiaSTTSettings(STTSettings):
"""Typed settings for Gladia STT service.
Parameters:
input_params: Gladia ``GladiaInputParams`` for detailed configuration.
"""
input_params: GladiaInputParams = field(default_factory=lambda: NOT_GIVEN)
class GladiaSTTService(WebsocketSTTService): class GladiaSTTService(WebsocketSTTService):
"""Speech-to-Text service using Gladia's API. """Speech-to-Text service using Gladia's API.
@@ -265,9 +278,8 @@ class GladiaSTTService(WebsocketSTTService):
self._region = region self._region = region
self._url = url self._url = url
self.set_model_name(model) self.set_model_name(model)
self._params = params
self._receive_task = None self._receive_task = None
self._settings = {} self._settings = GladiaSTTSettings(model=model, input_params=params)
# Session management # Session management
self._session_url = None self._session_url = None
@@ -307,31 +319,33 @@ class GladiaSTTService(WebsocketSTTService):
return language_to_gladia_language(language) return language_to_gladia_language(language)
def _prepare_settings(self) -> Dict[str, Any]: def _prepare_settings(self) -> Dict[str, Any]:
params = self._settings.input_params
settings = { settings = {
"encoding": self._params.encoding or "wav/pcm", "encoding": params.encoding or "wav/pcm",
"bit_depth": self._params.bit_depth or 16, "bit_depth": params.bit_depth or 16,
"sample_rate": self.sample_rate, "sample_rate": self.sample_rate,
"channels": self._params.channels or 1, "channels": params.channels or 1,
"model": self._model_name, "model": self._model_name,
} }
# Add custom_metadata if provided # Add custom_metadata if provided
settings["custom_metadata"] = dict(self._params.custom_metadata or {}) settings["custom_metadata"] = dict(params.custom_metadata or {})
settings["custom_metadata"]["pipecat"] = pipecat_version() settings["custom_metadata"]["pipecat"] = pipecat_version()
# Add endpointing parameters if provided # Add endpointing parameters if provided
if self._params.endpointing is not None: if params.endpointing is not None:
settings["endpointing"] = self._params.endpointing settings["endpointing"] = params.endpointing
if self._params.maximum_duration_without_endpointing is not None: if params.maximum_duration_without_endpointing is not None:
settings["maximum_duration_without_endpointing"] = ( settings["maximum_duration_without_endpointing"] = (
self._params.maximum_duration_without_endpointing params.maximum_duration_without_endpointing
) )
# Add language configuration (prioritize language_config over deprecated language) # Add language configuration (prioritize language_config over deprecated language)
if self._params.language_config: if params.language_config:
settings["language_config"] = self._params.language_config.model_dump(exclude_none=True) settings["language_config"] = params.language_config.model_dump(exclude_none=True)
elif self._params.language: # Backward compatibility for deprecated parameter elif params.language: # Backward compatibility for deprecated parameter
language_code = self.language_to_service_language(self._params.language) language_code = self.language_to_service_language(params.language)
if language_code: if language_code:
settings["language_config"] = { settings["language_config"] = {
"languages": [language_code], "languages": [language_code],
@@ -339,21 +353,18 @@ class GladiaSTTService(WebsocketSTTService):
} }
# Add pre_processing configuration if provided # Add pre_processing configuration if provided
if self._params.pre_processing: if params.pre_processing:
settings["pre_processing"] = self._params.pre_processing.model_dump(exclude_none=True) settings["pre_processing"] = params.pre_processing.model_dump(exclude_none=True)
# Add realtime_processing configuration if provided # Add realtime_processing configuration if provided
if self._params.realtime_processing: if params.realtime_processing:
settings["realtime_processing"] = self._params.realtime_processing.model_dump( settings["realtime_processing"] = params.realtime_processing.model_dump(
exclude_none=True exclude_none=True
) )
# Add messages_config if provided # Add messages_config if provided
if self._params.messages_config: if params.messages_config:
settings["messages_config"] = self._params.messages_config.model_dump(exclude_none=True) settings["messages_config"] = params.messages_config.model_dump(exclude_none=True)
# Store settings for tracing
self._settings = settings
return settings return settings
@@ -366,6 +377,31 @@ class GladiaSTTService(WebsocketSTTService):
await super().start(frame) await super().start(frame)
await self._connect() await self._connect()
async def _update_settings_from_typed(self, update: GladiaSTTSettings) -> set[str]:
"""Apply typed settings update.
Gladia sessions are fixed at creation time, so any change requires
a full session teardown and reconnect.
Args:
update: A typed settings delta.
Returns:
Set of field names whose values actually changed.
"""
changed = await super()._update_settings_from_typed(update)
if not changed:
return changed
# Gladia sessions are fixed — need to tear down and recreate
self._session_url = None
self._session_id = None
await self._disconnect()
await self._connect()
return changed
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
"""Stop the Gladia STT websocket connection. """Stop the Gladia STT websocket connection.
@@ -522,7 +558,7 @@ class GladiaSTTService(WebsocketSTTService):
Broadcasts UserStartedSpeakingFrame and optionally triggers interruption Broadcasts UserStartedSpeakingFrame and optionally triggers interruption
when VAD is enabled. when VAD is enabled.
""" """
if not self._params.enable_vad or self._is_speaking: if not self._settings.input_params.enable_vad or self._is_speaking:
return return
logger.debug(f"{self} User started speaking") logger.debug(f"{self} User started speaking")
@@ -537,7 +573,7 @@ class GladiaSTTService(WebsocketSTTService):
Broadcasts UserStoppedSpeakingFrame when VAD is enabled. Broadcasts UserStoppedSpeakingFrame when VAD is enabled.
""" """
if not self._params.enable_vad or not self._is_speaking: if not self._settings.input_params.enable_vad or not self._is_speaking:
return return
self._is_speaking = False self._is_speaking = False
await self.broadcast_frame(UserStoppedSpeakingFrame) await self.broadcast_frame(UserStoppedSpeakingFrame)

117
src/pipecat/services/google/gemini_live/llm.py
View File

@@ -17,9 +17,9 @@ import io
import time import time
import uuid import uuid
import warnings import warnings
from dataclasses import dataclass from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import Any, Dict, List, Optional, Union from typing import Any, ClassVar, Dict, List, Optional, Union
from loguru import logger from loguru import logger
from PIL import Image from PIL import Image
@@ -47,7 +47,6 @@ from pipecat.frames.frames import (
LLMThoughtEndFrame, LLMThoughtEndFrame,
LLMThoughtStartFrame, LLMThoughtStartFrame,
LLMThoughtTextFrame, LLMThoughtTextFrame,
LLMUpdateSettingsFrame,
StartFrame, StartFrame,
TranscriptionFrame, TranscriptionFrame,
TTSAudioRawFrame, TTSAudioRawFrame,
@@ -77,6 +76,7 @@ from pipecat.services.openai.llm import (
OpenAIAssistantContextAggregator, OpenAIAssistantContextAggregator,
OpenAIUserContextAggregator, OpenAIUserContextAggregator,
) )
from pipecat.services.settings import NOT_GIVEN, LLMSettings
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.string import match_endofsentence from pipecat.utils.string import match_endofsentence
from pipecat.utils.time import time_now_iso8601 from pipecat.utils.time import time_now_iso8601
@@ -602,6 +602,31 @@ class InputParams(BaseModel):
extra: Optional[Dict[str, Any]] = Field(default_factory=dict) extra: Optional[Dict[str, Any]] = Field(default_factory=dict)
@dataclass
class GeminiLiveLLMSettings(LLMSettings):
"""Typed settings for Gemini Live LLM services.
Parameters:
modalities: Response modalities.
language: Language for generation.
media_resolution: Media resolution setting.
vad: Voice activity detection parameters.
context_window_compression: Context window compression configuration.
thinking: Thinking configuration.
enable_affective_dialog: Whether to enable affective dialog.
proactivity: Proactivity configuration.
"""
modalities: Any = field(default_factory=lambda: NOT_GIVEN)
language: Any = field(default_factory=lambda: NOT_GIVEN)
media_resolution: Any = field(default_factory=lambda: NOT_GIVEN)
vad: Any = field(default_factory=lambda: NOT_GIVEN)
context_window_compression: Any = field(default_factory=lambda: NOT_GIVEN)
thinking: Any = field(default_factory=lambda: NOT_GIVEN)
enable_affective_dialog: Any = field(default_factory=lambda: NOT_GIVEN)
proactivity: Any = field(default_factory=lambda: NOT_GIVEN)
class GeminiLiveLLMService(LLMService): class GeminiLiveLLMService(LLMService):
"""Provides access to Google's Gemini Live API. """Provides access to Google's Gemini Live API.
@@ -714,25 +739,26 @@ class GeminiLiveLLMService(LLMService):
self._consecutive_failures = 0 self._consecutive_failures = 0
self._connection_start_time = None self._connection_start_time = None
self._settings = { self._settings = GeminiLiveLLMSettings(
"frequency_penalty": params.frequency_penalty, model=model,
"max_tokens": params.max_tokens, frequency_penalty=params.frequency_penalty,
"presence_penalty": params.presence_penalty, max_tokens=params.max_tokens,
"temperature": params.temperature, presence_penalty=params.presence_penalty,
"top_k": params.top_k, temperature=params.temperature,
"top_p": params.top_p, top_k=params.top_k,
"modalities": params.modalities, top_p=params.top_p,
"language": self._language_code, modalities=params.modalities,
"media_resolution": params.media_resolution, language=self._language_code,
"vad": params.vad, media_resolution=params.media_resolution,
"context_window_compression": params.context_window_compression.model_dump() vad=params.vad,
context_window_compression=params.context_window_compression.model_dump()
if params.context_window_compression if params.context_window_compression
else {}, else {},
"thinking": params.thinking or {}, thinking=params.thinking or {},
"enable_affective_dialog": params.enable_affective_dialog or False, enable_affective_dialog=params.enable_affective_dialog or False,
"proactivity": params.proactivity or {}, proactivity=params.proactivity or {},
"extra": params.extra if isinstance(params.extra, dict) else {}, extra=params.extra if isinstance(params.extra, dict) else {},
} )
self._file_api_base_url = file_api_base_url self._file_api_base_url = file_api_base_url
self._file_api: Optional[GeminiFileAPI] = None self._file_api: Optional[GeminiFileAPI] = None
@@ -798,7 +824,7 @@ class GeminiLiveLLMService(LLMService):
Args: Args:
modalities: The modalities to use for responses. modalities: The modalities to use for responses.
""" """
self._settings["modalities"] = modalities self._settings.modalities = modalities
def set_language(self, language: Language): def set_language(self, language: Language):
"""Set the language for generation. """Set the language for generation.
@@ -808,7 +834,7 @@ class GeminiLiveLLMService(LLMService):
""" """
self._language = language self._language = language
self._language_code = language_to_gemini_language(language) or "en-US" self._language_code = language_to_gemini_language(language) or "en-US"
self._settings["language"] = self._language_code self._settings.language = self._language_code
logger.info(f"Set Gemini language to: {self._language_code}") logger.info(f"Set Gemini language to: {self._language_code}")
async def set_context(self, context: OpenAILLMContext): async def set_context(self, context: OpenAILLMContext):
@@ -866,7 +892,7 @@ class GeminiLiveLLMService(LLMService):
async def _handle_interruption(self): async def _handle_interruption(self):
if self._bot_is_responding: if self._bot_is_responding:
await self._set_bot_is_responding(False) await self._set_bot_is_responding(False)
if self._settings.get("modalities") == GeminiModalities.AUDIO: if self._settings.modalities == GeminiModalities.AUDIO:
await self.push_frame(TTSStoppedFrame()) await self.push_frame(TTSStoppedFrame())
# Do not send LLMFullResponseEndFrame here - an interruption # Do not send LLMFullResponseEndFrame here - an interruption
# already tells the assistant context aggregator that the response # already tells the assistant context aggregator that the response
@@ -947,8 +973,6 @@ class GeminiLiveLLMService(LLMService):
# uses this frame *without* a user context aggregator still works # uses this frame *without* a user context aggregator still works
# (we have an example that does just that, actually). # (we have an example that does just that, actually).
await self._create_single_response(frame.messages) await self._create_single_response(frame.messages)
elif isinstance(frame, LLMUpdateSettingsFrame):
await self._update_settings(frame.settings)
elif isinstance(frame, LLMSetToolsFrame): elif isinstance(frame, LLMSetToolsFrame):
await self._update_settings() await self._update_settings()
else: else:
@@ -1074,20 +1098,20 @@ class GeminiLiveLLMService(LLMService):
# Assemble basic configuration # Assemble basic configuration
config = LiveConnectConfig( config = LiveConnectConfig(
generation_config=GenerationConfig( generation_config=GenerationConfig(
frequency_penalty=self._settings["frequency_penalty"], frequency_penalty=self._settings.frequency_penalty,
max_output_tokens=self._settings["max_tokens"], max_output_tokens=self._settings.max_tokens,
presence_penalty=self._settings["presence_penalty"], presence_penalty=self._settings.presence_penalty,
temperature=self._settings["temperature"], temperature=self._settings.temperature,
top_k=self._settings["top_k"], top_k=self._settings.top_k,
top_p=self._settings["top_p"], top_p=self._settings.top_p,
response_modalities=[Modality(self._settings["modalities"].value)], response_modalities=[Modality(self._settings.modalities.value)],
speech_config=SpeechConfig( speech_config=SpeechConfig(
voice_config=VoiceConfig( voice_config=VoiceConfig(
prebuilt_voice_config={"voice_name": self._voice_id} prebuilt_voice_config={"voice_name": self._voice_id}
), ),
language_code=self._settings["language"], language_code=self._settings.language,
), ),
media_resolution=MediaResolution(self._settings["media_resolution"].value), media_resolution=MediaResolution(self._settings.media_resolution.value),
), ),
input_audio_transcription=AudioTranscriptionConfig(), input_audio_transcription=AudioTranscriptionConfig(),
output_audio_transcription=AudioTranscriptionConfig(), output_audio_transcription=AudioTranscriptionConfig(),
@@ -1095,37 +1119,36 @@ class GeminiLiveLLMService(LLMService):
) )
# Add context window compression to configuration, if enabled # Add context window compression to configuration, if enabled
if self._settings.get("context_window_compression", {}).get("enabled", False): cwc = self._settings.context_window_compression or {}
if cwc.get("enabled", False):
compression_config = ContextWindowCompressionConfig() compression_config = ContextWindowCompressionConfig()
# Add sliding window (always true if compression is enabled) # Add sliding window (always true if compression is enabled)
compression_config.sliding_window = SlidingWindow() compression_config.sliding_window = SlidingWindow()
# Add trigger_tokens if specified # Add trigger_tokens if specified
trigger_tokens = self._settings.get("context_window_compression", {}).get( trigger_tokens = cwc.get("trigger_tokens")
"trigger_tokens"
)
if trigger_tokens is not None: if trigger_tokens is not None:
compression_config.trigger_tokens = trigger_tokens compression_config.trigger_tokens = trigger_tokens
config.context_window_compression = compression_config config.context_window_compression = compression_config
# Add thinking configuration to configuration, if provided # Add thinking configuration to configuration, if provided
if self._settings.get("thinking"): if self._settings.thinking:
config.thinking_config = self._settings["thinking"] config.thinking_config = self._settings.thinking
# Add affective dialog setting, if provided # Add affective dialog setting, if provided
if self._settings.get("enable_affective_dialog", False): if self._settings.enable_affective_dialog:
config.enable_affective_dialog = self._settings["enable_affective_dialog"] config.enable_affective_dialog = self._settings.enable_affective_dialog
# Add proactivity configuration to configuration, if provided # Add proactivity configuration to configuration, if provided
if self._settings.get("proactivity"): if self._settings.proactivity:
config.proactivity = self._settings["proactivity"] config.proactivity = self._settings.proactivity
# Add VAD configuration to configuration, if provided # Add VAD configuration to configuration, if provided
if self._settings.get("vad"): if self._settings.vad:
vad_config = AutomaticActivityDetection() vad_config = AutomaticActivityDetection()
vad_params = self._settings["vad"] vad_params = self._settings.vad
has_vad_settings = False has_vad_settings = False
# Only add parameters that are explicitly set # Only add parameters that are explicitly set
@@ -1604,7 +1627,7 @@ class GeminiLiveLLMService(LLMService):
text: The transcription text to push text: The transcription text to push
result: Optional LiveServerMessage that triggered this transcription result: Optional LiveServerMessage that triggered this transcription
""" """
await self._handle_user_transcription(text, True, self._settings["language"]) await self._handle_user_transcription(text, True, self._settings.language)
await self.push_frame( await self.push_frame(
TranscriptionFrame( TranscriptionFrame(
text=text, text=text,

52
src/pipecat/services/google/llm.py
View File

@@ -15,8 +15,8 @@ import io
import json import json
import os import os
import uuid import uuid
from dataclasses import dataclass from dataclasses import dataclass, field
from typing import Any, AsyncIterator, Dict, List, Literal, Optional from typing import Any, AsyncIterator, ClassVar, Dict, List, Literal, Optional
from loguru import logger from loguru import logger
from PIL import Image from PIL import Image
@@ -39,7 +39,6 @@ from pipecat.frames.frames import (
LLMThoughtEndFrame, LLMThoughtEndFrame,
LLMThoughtStartFrame, LLMThoughtStartFrame,
LLMThoughtTextFrame, LLMThoughtTextFrame,
LLMUpdateSettingsFrame,
) )
from pipecat.metrics.metrics import LLMTokenUsage from pipecat.metrics.metrics import LLMTokenUsage
from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_context import LLMContext
@@ -59,6 +58,7 @@ from pipecat.services.openai.llm import (
OpenAIAssistantContextAggregator, OpenAIAssistantContextAggregator,
OpenAIUserContextAggregator, OpenAIUserContextAggregator,
) )
from pipecat.services.settings import NOT_GIVEN, LLMSettings
from pipecat.utils.tracing.service_decorators import traced_llm from pipecat.utils.tracing.service_decorators import traced_llm
# Suppress gRPC fork warnings # Suppress gRPC fork warnings
@@ -673,6 +673,17 @@ class GoogleLLMContext(OpenAILLMContext):
self._messages = [m for m in self._messages if m.parts] self._messages = [m for m in self._messages if m.parts]
@dataclass
class GoogleLLMSettings(LLMSettings):
"""Typed settings for Google LLM services.
Parameters:
thinking: Thinking configuration.
"""
thinking: Any = field(default_factory=lambda: NOT_GIVEN)
class GoogleLLMService(LLMService): class GoogleLLMService(LLMService):
"""Google AI (Gemini) LLM service implementation. """Google AI (Gemini) LLM service implementation.
@@ -773,14 +784,15 @@ class GoogleLLMService(LLMService):
self._system_instruction = system_instruction self._system_instruction = system_instruction
self._http_options = update_google_client_http_options(http_options) self._http_options = update_google_client_http_options(http_options)
self._settings = { self._settings = GoogleLLMSettings(
"max_tokens": params.max_tokens, model=model,
"temperature": params.temperature, max_tokens=params.max_tokens,
"top_k": params.top_k, temperature=params.temperature,
"top_p": params.top_p, top_k=params.top_k,
"thinking": params.thinking, top_p=params.top_p,
"extra": params.extra if isinstance(params.extra, dict) else {}, thinking=params.thinking,
} extra=params.extra if isinstance(params.extra, dict) else {},
)
self._tools = tools self._tools = tools
self._tool_config = tool_config self._tool_config = tool_config
@@ -874,10 +886,10 @@ class GoogleLLMService(LLMService):
k: v k: v
for k, v in { for k, v in {
"system_instruction": system_instruction, "system_instruction": system_instruction,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"top_k": self._settings["top_k"], "top_k": self._settings.top_k,
"max_output_tokens": self._settings["max_tokens"], "max_output_tokens": self._settings.max_tokens,
"tools": tools, "tools": tools,
"tool_config": tool_config, "tool_config": tool_config,
}.items() }.items()
@@ -885,13 +897,13 @@ class GoogleLLMService(LLMService):
} }
# Add thinking parameters if configured # Add thinking parameters if configured
if self._settings["thinking"]: if self._settings.thinking:
generation_params["thinking_config"] = self._settings["thinking"].model_dump( generation_params["thinking_config"] = self._settings.thinking.model_dump(
exclude_unset=True exclude_unset=True
) )
if self._settings["extra"]: if self._settings.extra:
generation_params.update(self._settings["extra"]) generation_params.update(self._settings.extra)
return generation_params return generation_params
@@ -1190,8 +1202,6 @@ class GoogleLLMService(LLMService):
# NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal # NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal
# LLMContext with it # LLMContext with it
context = GoogleLLMContext(frame.messages) context = GoogleLLMContext(frame.messages)
elif isinstance(frame, LLMUpdateSettingsFrame):
await self._update_settings(frame.settings)
else: else:
await self.push_frame(frame, direction) await self.push_frame(frame, direction)

241
src/pipecat/services/google/stt.py
View File

@@ -15,13 +15,15 @@ import asyncio
import json import json
import os import os
import time import time
import warnings
from dataclasses import dataclass, field
from pipecat.utils.tracing.service_decorators import traced_stt from pipecat.utils.tracing.service_decorators import traced_stt
# Suppress gRPC fork warnings # Suppress gRPC fork warnings
os.environ["GRPC_ENABLE_FORK_SUPPORT"] = "false" os.environ["GRPC_ENABLE_FORK_SUPPORT"] = "false"
from typing import AsyncGenerator, List, Optional, Union from typing import Any, AsyncGenerator, List, Optional, Union
from loguru import logger from loguru import logger
from pydantic import BaseModel, Field, field_validator from pydantic import BaseModel, Field, field_validator
@@ -34,6 +36,7 @@ from pipecat.frames.frames import (
StartFrame, StartFrame,
TranscriptionFrame, TranscriptionFrame,
) )
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import GOOGLE_TTFS_P99 from pipecat.services.stt_latency import GOOGLE_TTFS_P99
from pipecat.services.stt_service import STTService from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -355,6 +358,44 @@ def language_to_google_stt_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class GoogleSTTSettings(STTSettings):
"""Typed settings for Google Cloud Speech-to-Text V2.
Parameters:
languages: List of ``Language`` enums for recognition
(e.g. ``[Language.EN_US]``). Preferred over ``language_codes``.
language_codes: List of Google STT language code strings
(e.g. ``["en-US"]``).
.. deprecated:: 0.0.103
Use ``languages`` instead. If both are provided, ``languages``
takes precedence. This field is here just for backward
compatibility with dict-based settings updates.
use_separate_recognition_per_channel: Process each audio channel separately.
enable_automatic_punctuation: Add punctuation to transcripts.
enable_spoken_punctuation: Include spoken punctuation in transcript.
enable_spoken_emojis: Include spoken emojis in transcript.
profanity_filter: Filter profanity from transcript.
enable_word_time_offsets: Include timing information for each word.
enable_word_confidence: Include confidence scores for each word.
enable_interim_results: Stream partial recognition results.
enable_voice_activity_events: Detect voice activity in audio.
"""
languages: Any = field(default_factory=lambda: NOT_GIVEN)
language_codes: Any = field(default_factory=lambda: NOT_GIVEN)
use_separate_recognition_per_channel: Any = field(default_factory=lambda: NOT_GIVEN)
enable_automatic_punctuation: Any = field(default_factory=lambda: NOT_GIVEN)
enable_spoken_punctuation: Any = field(default_factory=lambda: NOT_GIVEN)
enable_spoken_emojis: Any = field(default_factory=lambda: NOT_GIVEN)
profanity_filter: Any = field(default_factory=lambda: NOT_GIVEN)
enable_word_time_offsets: Any = field(default_factory=lambda: NOT_GIVEN)
enable_word_confidence: Any = field(default_factory=lambda: NOT_GIVEN)
enable_interim_results: Any = field(default_factory=lambda: NOT_GIVEN)
enable_voice_activity_events: Any = field(default_factory=lambda: NOT_GIVEN)
class GoogleSTTService(STTService): class GoogleSTTService(STTService):
"""Google Cloud Speech-to-Text V2 service implementation. """Google Cloud Speech-to-Text V2 service implementation.
@@ -508,21 +549,19 @@ class GoogleSTTService(STTService):
self._client = speech_v2.SpeechAsyncClient(credentials=creds, client_options=client_options) self._client = speech_v2.SpeechAsyncClient(credentials=creds, client_options=client_options)
self._settings = { self._settings = GoogleSTTSettings(
"language_codes": [ languages=list(params.language_list),
self.language_to_service_language(lang) for lang in params.language_list model=params.model,
], use_separate_recognition_per_channel=params.use_separate_recognition_per_channel,
"model": params.model, enable_automatic_punctuation=params.enable_automatic_punctuation,
"use_separate_recognition_per_channel": params.use_separate_recognition_per_channel, enable_spoken_punctuation=params.enable_spoken_punctuation,
"enable_automatic_punctuation": params.enable_automatic_punctuation, enable_spoken_emojis=params.enable_spoken_emojis,
"enable_spoken_punctuation": params.enable_spoken_punctuation, profanity_filter=params.profanity_filter,
"enable_spoken_emojis": params.enable_spoken_emojis, enable_word_time_offsets=params.enable_word_time_offsets,
"profanity_filter": params.profanity_filter, enable_word_confidence=params.enable_word_confidence,
"enable_word_time_offsets": params.enable_word_time_offsets, enable_interim_results=params.enable_interim_results,
"enable_word_confidence": params.enable_word_confidence, enable_voice_activity_events=params.enable_voice_activity_events,
"enable_interim_results": params.enable_interim_results, )
"enable_voice_activity_events": params.enable_voice_activity_events,
}
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if the service can generate metrics. """Check if the service can generate metrics.
@@ -545,6 +584,23 @@ class GoogleSTTService(STTService):
return [language_to_google_stt_language(lang) or "en-US" for lang in language] return [language_to_google_stt_language(lang) or "en-US" for lang in language]
return language_to_google_stt_language(language) or "en-US" return language_to_google_stt_language(language) or "en-US"
def _get_language_codes(self) -> List[str]:
"""Resolve the current language settings to Google STT language code strings.
Prefers ``languages`` (``Language`` enums) over the deprecated
``language_codes`` (raw strings). Falls back to ``["en-US"]``.
Returns:
List[str]: Google STT language code strings.
"""
from pipecat.services.settings import is_given
if is_given(self._settings.languages):
return [self.language_to_service_language(lang) for lang in self._settings.languages]
if is_given(self._settings.language_codes):
return list(self._settings.language_codes)
return ["en-US"]
async def _reconnect_if_needed(self): async def _reconnect_if_needed(self):
"""Reconnect the stream if it's currently active.""" """Reconnect the stream if it's currently active."""
if self._streaming_task: if self._streaming_task:
@@ -552,41 +608,65 @@ class GoogleSTTService(STTService):
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
async def set_language(self, language: Language):
"""Update the service's recognition language.
A convenience method for setting a single language.
Args:
language: New language for recognition.
"""
logger.debug(f"Switching STT language to: {language}")
await self.set_languages([language])
async def set_languages(self, languages: List[Language]): async def set_languages(self, languages: List[Language]):
"""Update the service's recognition languages. """Update the service's recognition languages.
.. deprecated::
Use ``STTUpdateSettingsFrame`` with ``GoogleSTTSettings(languages=...)``
instead.
Args: Args:
languages: List of languages for recognition. First language is primary. languages: List of languages for recognition. First language is primary.
""" """
with warnings.catch_warnings():
warnings.simplefilter("always")
warnings.warn(
"set_languages() is deprecated. Use STTUpdateSettingsFrame with "
"GoogleSTTSettings(languages=...) instead.",
DeprecationWarning,
)
logger.debug(f"Switching STT languages to: {languages}") logger.debug(f"Switching STT languages to: {languages}")
self._settings["language_codes"] = [ await self._update_settings_from_typed(GoogleSTTSettings(languages=list(languages)))
self.language_to_service_language(lang) for lang in languages
]
# Recreate stream with new languages
await self._reconnect_if_needed()
async def set_model(self, model: str): async def _update_settings_from_typed(self, update: GoogleSTTSettings) -> set[str]:
"""Update the service's recognition model. """Apply typed settings update and reconnect if anything changed.
Handles ``language`` from base ``set_language`` by converting it to
``languages``. Emits a deprecation warning if ``language_codes`` is
used. All other fields (model, boolean flags) are applied directly.
Reconnects the stream on any change.
Args: Args:
model: The new recognition model to use. update: A typed settings delta.
Returns:
Set of field names whose values actually changed.
""" """
logger.debug(f"Switching STT model to: {model}") from pipecat.services.settings import is_given
await super().set_model(model)
self._settings["model"] = model # If base set_language sent a Language value, convert to languages list
# Recreate stream with new model if is_given(update.language):
await self._reconnect_if_needed() update.languages = [update.language]
# Clear language so the base class doesn't try to store it
update.language = NOT_GIVEN
# Warn on deprecated language_codes usage
if is_given(update.language_codes):
with warnings.catch_warnings():
warnings.simplefilter("always")
warnings.warn(
"GoogleSTTSettings.language_codes is deprecated. "
"Use GoogleSTTSettings.languages (List[Language]) instead.",
DeprecationWarning,
stacklevel=2,
)
changed = await super()._update_settings_from_typed(update)
if changed:
await self._reconnect_if_needed()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the STT service and establish connection. """Start the STT service and establish connection.
@@ -632,6 +712,10 @@ class GoogleSTTService(STTService):
) -> None: ) -> None:
"""Update service options dynamically. """Update service options dynamically.
.. deprecated::
Use ``STTUpdateSettingsFrame`` with ``GoogleSTTSettings(...)``
instead.
Args: Args:
languages: New list of recognition languages. languages: New list of recognition languages.
model: New recognition model. model: New recognition model.
@@ -649,55 +733,42 @@ class GoogleSTTService(STTService):
Changes that affect the streaming configuration will cause Changes that affect the streaming configuration will cause
the stream to be reconnected. the stream to be reconnected.
""" """
# Update settings with new values with warnings.catch_warnings():
warnings.simplefilter("always")
warnings.warn(
"update_options() is deprecated. Use STTUpdateSettingsFrame with "
"GoogleSTTSettings(...) instead.",
DeprecationWarning,
)
# Build a typed settings delta from the provided options
update = GoogleSTTSettings()
if languages is not None: if languages is not None:
logger.debug(f"Updating language to: {languages}") update.languages = list(languages)
self._settings["language_codes"] = [
self.language_to_service_language(lang) for lang in languages
]
if model is not None: if model is not None:
logger.debug(f"Updating model to: {model}") update.model = model
self._settings["model"] = model
if enable_automatic_punctuation is not None: if enable_automatic_punctuation is not None:
logger.debug(f"Updating automatic punctuation to: {enable_automatic_punctuation}") update.enable_automatic_punctuation = enable_automatic_punctuation
self._settings["enable_automatic_punctuation"] = enable_automatic_punctuation
if enable_spoken_punctuation is not None: if enable_spoken_punctuation is not None:
logger.debug(f"Updating spoken punctuation to: {enable_spoken_punctuation}") update.enable_spoken_punctuation = enable_spoken_punctuation
self._settings["enable_spoken_punctuation"] = enable_spoken_punctuation
if enable_spoken_emojis is not None: if enable_spoken_emojis is not None:
logger.debug(f"Updating spoken emojis to: {enable_spoken_emojis}") update.enable_spoken_emojis = enable_spoken_emojis
self._settings["enable_spoken_emojis"] = enable_spoken_emojis
if profanity_filter is not None: if profanity_filter is not None:
logger.debug(f"Updating profanity filter to: {profanity_filter}") update.profanity_filter = profanity_filter
self._settings["profanity_filter"] = profanity_filter
if enable_word_time_offsets is not None: if enable_word_time_offsets is not None:
logger.debug(f"Updating word time offsets to: {enable_word_time_offsets}") update.enable_word_time_offsets = enable_word_time_offsets
self._settings["enable_word_time_offsets"] = enable_word_time_offsets
if enable_word_confidence is not None: if enable_word_confidence is not None:
logger.debug(f"Updating word confidence to: {enable_word_confidence}") update.enable_word_confidence = enable_word_confidence
self._settings["enable_word_confidence"] = enable_word_confidence
if enable_interim_results is not None: if enable_interim_results is not None:
logger.debug(f"Updating interim results to: {enable_interim_results}") update.enable_interim_results = enable_interim_results
self._settings["enable_interim_results"] = enable_interim_results
if enable_voice_activity_events is not None: if enable_voice_activity_events is not None:
logger.debug(f"Updating voice activity events to: {enable_voice_activity_events}") update.enable_voice_activity_events = enable_voice_activity_events
self._settings["enable_voice_activity_events"] = enable_voice_activity_events
if location is not None: if location is not None:
logger.debug(f"Updating location to: {location}") logger.debug(f"Updating location to: {location}")
self._location = location self._location = location
# Reconnect the stream for updates await self._update_settings_from_typed(update)
await self._reconnect_if_needed()
async def _connect(self): async def _connect(self):
"""Initialize streaming recognition config and stream.""" """Initialize streaming recognition config and stream."""
@@ -714,20 +785,20 @@ class GoogleSTTService(STTService):
sample_rate_hertz=self.sample_rate, sample_rate_hertz=self.sample_rate,
audio_channel_count=1, audio_channel_count=1,
), ),
language_codes=self._settings["language_codes"], language_codes=self._get_language_codes(),
model=self._settings["model"], model=self._settings.model,
features=cloud_speech.RecognitionFeatures( features=cloud_speech.RecognitionFeatures(
enable_automatic_punctuation=self._settings["enable_automatic_punctuation"], enable_automatic_punctuation=self._settings.enable_automatic_punctuation,
enable_spoken_punctuation=self._settings["enable_spoken_punctuation"], enable_spoken_punctuation=self._settings.enable_spoken_punctuation,
enable_spoken_emojis=self._settings["enable_spoken_emojis"], enable_spoken_emojis=self._settings.enable_spoken_emojis,
profanity_filter=self._settings["profanity_filter"], profanity_filter=self._settings.profanity_filter,
enable_word_time_offsets=self._settings["enable_word_time_offsets"], enable_word_time_offsets=self._settings.enable_word_time_offsets,
enable_word_confidence=self._settings["enable_word_confidence"], enable_word_confidence=self._settings.enable_word_confidence,
), ),
), ),
streaming_features=cloud_speech.StreamingRecognitionFeatures( streaming_features=cloud_speech.StreamingRecognitionFeatures(
enable_voice_activity_events=self._settings["enable_voice_activity_events"], enable_voice_activity_events=self._settings.enable_voice_activity_events,
interim_results=self._settings["enable_interim_results"], interim_results=self._settings.enable_interim_results,
), ),
) )
@@ -857,7 +928,7 @@ class GoogleSTTService(STTService):
if not transcript: if not transcript:
continue continue
primary_language = self._settings["language_codes"][0] primary_language = self._get_language_codes()[0]
if result.is_final: if result.is_final:
self._last_transcript_was_final = True self._last_transcript_was_final = True

201
src/pipecat/services/google/tts.py
View File

@@ -23,7 +23,8 @@ from pipecat.utils.tracing.service_decorators import traced_tts
# Suppress gRPC fork warnings # Suppress gRPC fork warnings
os.environ["GRPC_ENABLE_FORK_SUPPORT"] = "false" os.environ["GRPC_ENABLE_FORK_SUPPORT"] = "false"
from typing import Any, AsyncGenerator, List, Literal, Mapping, Optional from dataclasses import dataclass, field
from typing import AsyncGenerator, List, Literal, Optional
from loguru import logger from loguru import logger
from pydantic import BaseModel from pydantic import BaseModel
@@ -36,6 +37,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -474,6 +476,63 @@ def language_to_gemini_tts_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class GoogleHttpTTSSettings(TTSSettings):
"""Typed settings for Google HTTP TTS service.
Parameters:
pitch: Voice pitch adjustment (e.g., "+2st", "-50%").
rate: Speaking rate adjustment (e.g., "slow", "fast", "125%"). Used for
SSML prosody tags (non-Chirp voices).
speaking_rate: Speaking rate for AudioConfig (Chirp/Journey voices).
Range [0.25, 2.0].
volume: Volume adjustment (e.g., "loud", "soft", "+6dB").
emphasis: Emphasis level for the text.
language: Language for synthesis. Defaults to English.
gender: Voice gender preference.
google_style: Google-specific voice style.
"""
pitch: str = field(default_factory=lambda: NOT_GIVEN)
rate: str = field(default_factory=lambda: NOT_GIVEN)
speaking_rate: float = field(default_factory=lambda: NOT_GIVEN)
volume: str = field(default_factory=lambda: NOT_GIVEN)
emphasis: str = field(default_factory=lambda: NOT_GIVEN)
language: str = field(default_factory=lambda: NOT_GIVEN)
gender: str = field(default_factory=lambda: NOT_GIVEN)
google_style: str = field(default_factory=lambda: NOT_GIVEN)
@dataclass
class GoogleStreamTTSSettings(TTSSettings):
"""Typed settings for Google streaming TTS service.
Parameters:
language: Language for synthesis. Defaults to English.
speaking_rate: The speaking rate, in the range [0.25, 2.0].
"""
language: str = field(default_factory=lambda: NOT_GIVEN)
speaking_rate: float = field(default_factory=lambda: NOT_GIVEN)
@dataclass
class GeminiTTSSettings(TTSSettings):
"""Typed settings for Gemini TTS service.
Parameters:
language: Language for synthesis. Defaults to English.
prompt: Optional style instructions for how to synthesize the content.
multi_speaker: Whether to enable multi-speaker support.
speaker_configs: List of speaker configurations for multi-speaker mode.
"""
language: str = field(default_factory=lambda: NOT_GIVEN)
prompt: str = field(default_factory=lambda: NOT_GIVEN)
multi_speaker: bool = field(default_factory=lambda: NOT_GIVEN)
speaker_configs: List[dict] = field(default_factory=lambda: NOT_GIVEN)
class GoogleHttpTTSService(TTSService): class GoogleHttpTTSService(TTSService):
"""Google Cloud Text-to-Speech HTTP service with SSML support. """Google Cloud Text-to-Speech HTTP service with SSML support.
@@ -538,19 +597,19 @@ class GoogleHttpTTSService(TTSService):
params = params or GoogleHttpTTSService.InputParams() params = params or GoogleHttpTTSService.InputParams()
self._location = location self._location = location
self._settings = { self._settings: GoogleHttpTTSSettings = GoogleHttpTTSSettings(
"pitch": params.pitch, pitch=params.pitch,
"rate": params.rate, rate=params.rate,
"speaking_rate": params.speaking_rate, speaking_rate=params.speaking_rate,
"volume": params.volume, volume=params.volume,
"emphasis": params.emphasis, emphasis=params.emphasis,
"language": self.language_to_service_language(params.language) language=self.language_to_service_language(params.language)
if params.language if params.language
else "en-US", else "en-US",
"gender": params.gender, gender=params.gender,
"google_style": params.google_style, google_style=params.google_style,
} )
self.set_voice(voice_id) self._voice_id = voice_id
self._client: texttospeech_v1.TextToSpeechAsyncClient = self._create_client( self._client: texttospeech_v1.TextToSpeechAsyncClient = self._create_client(
credentials, credentials_path credentials, credentials_path
) )
@@ -619,21 +678,20 @@ class GoogleHttpTTSService(TTSService):
""" """
return language_to_google_tts_language(language) return language_to_google_tts_language(language)
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Override to handle speaking_rate updates for Chirp/Journey voices. """Override to handle speaking_rate validation.
Args: Args:
settings: Dictionary of settings to update. Can include 'speaking_rate' (float) update: Typed settings delta. Can include 'speaking_rate' (float).
""" """
if "speaking_rate" in settings: if isinstance(update, GoogleHttpTTSSettings) and is_given(update.speaking_rate):
rate_value = float(settings["speaking_rate"]) rate_value = float(update.speaking_rate)
if 0.25 <= rate_value <= 2.0: if not (0.25 <= rate_value <= 2.0):
self._settings["speaking_rate"] = rate_value
else:
logger.warning( logger.warning(
f"Invalid speaking_rate value: {rate_value}. Must be between 0.25 and 2.0" f"Invalid speaking_rate value: {rate_value}. Must be between 0.25 and 2.0"
) )
await super()._update_settings(settings) update.speaking_rate = NOT_GIVEN
return await super()._update_settings_from_typed(update)
def _construct_ssml(self, text: str) -> str: def _construct_ssml(self, text: str) -> str:
ssml = "<speak>" ssml = "<speak>"
@@ -641,39 +699,39 @@ class GoogleHttpTTSService(TTSService):
# Voice tag # Voice tag
voice_attrs = [f"name='{self._voice_id}'"] voice_attrs = [f"name='{self._voice_id}'"]
language = self._settings["language"] language = self._settings.language
voice_attrs.append(f"language='{language}'") voice_attrs.append(f"language='{language}'")
if self._settings["gender"]: if self._settings.gender:
voice_attrs.append(f"gender='{self._settings['gender']}'") voice_attrs.append(f"gender='{self._settings.gender}'")
ssml += f"<voice {' '.join(voice_attrs)}>" ssml += f"<voice {' '.join(voice_attrs)}>"
# Prosody tag # Prosody tag
prosody_attrs = [] prosody_attrs = []
if self._settings["pitch"]: if self._settings.pitch:
prosody_attrs.append(f"pitch='{self._settings['pitch']}'") prosody_attrs.append(f"pitch='{self._settings.pitch}'")
if self._settings["rate"]: if self._settings.rate:
prosody_attrs.append(f"rate='{self._settings['rate']}'") prosody_attrs.append(f"rate='{self._settings.rate}'")
if self._settings["volume"]: if self._settings.volume:
prosody_attrs.append(f"volume='{self._settings['volume']}'") prosody_attrs.append(f"volume='{self._settings.volume}'")
if prosody_attrs: if prosody_attrs:
ssml += f"<prosody {' '.join(prosody_attrs)}>" ssml += f"<prosody {' '.join(prosody_attrs)}>"
# Emphasis tag # Emphasis tag
if self._settings["emphasis"]: if self._settings.emphasis:
ssml += f"<emphasis level='{self._settings['emphasis']}'>" ssml += f"<emphasis level='{self._settings.emphasis}'>"
# Google style tag # Google style tag
if self._settings["google_style"]: if self._settings.google_style:
ssml += f"<google:style name='{self._settings['google_style']}'>" ssml += f"<google:style name='{self._settings.google_style}'>"
ssml += text ssml += text
# Close tags # Close tags
if self._settings["google_style"]: if self._settings.google_style:
ssml += "</google:style>" ssml += "</google:style>"
if self._settings["emphasis"]: if self._settings.emphasis:
ssml += "</emphasis>" ssml += "</emphasis>"
if prosody_attrs: if prosody_attrs:
ssml += "</prosody>" ssml += "</prosody>"
@@ -710,7 +768,7 @@ class GoogleHttpTTSService(TTSService):
synthesis_input = texttospeech_v1.SynthesisInput(ssml=ssml) synthesis_input = texttospeech_v1.SynthesisInput(ssml=ssml)
voice = texttospeech_v1.VoiceSelectionParams( voice = texttospeech_v1.VoiceSelectionParams(
language_code=self._settings["language"], name=self._voice_id language_code=self._settings.language, name=self._voice_id
) )
# Build audio config with conditional speaking_rate # Build audio config with conditional speaking_rate
audio_config_params = { audio_config_params = {
@@ -719,8 +777,8 @@ class GoogleHttpTTSService(TTSService):
} }
# For Chirp and Journey voices, include speaking_rate in AudioConfig # For Chirp and Journey voices, include speaking_rate in AudioConfig
if (is_chirp_voice or is_journey_voice) and self._settings["speaking_rate"] is not None: if (is_chirp_voice or is_journey_voice) and self._settings.speaking_rate is not None:
audio_config_params["speaking_rate"] = self._settings["speaking_rate"] audio_config_params["speaking_rate"] = self._settings.speaking_rate
audio_config = texttospeech_v1.AudioConfig(**audio_config_params) audio_config = texttospeech_v1.AudioConfig(**audio_config_params)
@@ -950,33 +1008,32 @@ class GoogleTTSService(GoogleBaseTTSService):
params = params or GoogleTTSService.InputParams() params = params or GoogleTTSService.InputParams()
self._location = location self._location = location
self._settings = { self._settings: GoogleStreamTTSSettings = GoogleStreamTTSSettings(
"language": self.language_to_service_language(params.language) language=self.language_to_service_language(params.language)
if params.language if params.language
else "en-US", else "en-US",
"speaking_rate": params.speaking_rate, speaking_rate=params.speaking_rate,
} )
self.set_voice(voice_id) self._voice_id = voice_id
self._voice_cloning_key = voice_cloning_key self._voice_cloning_key = voice_cloning_key
self._client: texttospeech_v1.TextToSpeechAsyncClient = self._create_client( self._client: texttospeech_v1.TextToSpeechAsyncClient = self._create_client(
credentials, credentials_path credentials, credentials_path
) )
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Override to handle speaking_rate updates for streaming API. """Override to handle speaking_rate validation.
Args: Args:
settings: Dictionary of settings to update. Can include 'speaking_rate' (float) update: Typed settings delta. Can include 'speaking_rate' (float).
""" """
if "speaking_rate" in settings: if isinstance(update, GoogleStreamTTSSettings) and is_given(update.speaking_rate):
rate_value = float(settings["speaking_rate"]) rate_value = float(update.speaking_rate)
if 0.25 <= rate_value <= 2.0: if not (0.25 <= rate_value <= 2.0):
self._settings["speaking_rate"] = rate_value
else:
logger.warning( logger.warning(
f"Invalid speaking_rate value: {rate_value}. Must be between 0.25 and 2.0" f"Invalid speaking_rate value: {rate_value}. Must be between 0.25 and 2.0"
) )
await super()._update_settings(settings) update.speaking_rate = NOT_GIVEN
return await super()._update_settings_from_typed(update)
@traced_tts @traced_tts
async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]: async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]:
@@ -1000,11 +1057,11 @@ class GoogleTTSService(GoogleBaseTTSService):
voice_cloning_key=self._voice_cloning_key voice_cloning_key=self._voice_cloning_key
) )
voice = texttospeech_v1.VoiceSelectionParams( voice = texttospeech_v1.VoiceSelectionParams(
language_code=self._settings["language"], voice_clone=voice_clone_params language_code=self._settings.language, voice_clone=voice_clone_params
) )
else: else:
voice = texttospeech_v1.VoiceSelectionParams( voice = texttospeech_v1.VoiceSelectionParams(
language_code=self._settings["language"], name=self._voice_id language_code=self._settings.language, name=self._voice_id
) )
# Create streaming config # Create streaming config
@@ -1013,7 +1070,7 @@ class GoogleTTSService(GoogleBaseTTSService):
streaming_audio_config=texttospeech_v1.StreamingAudioConfig( streaming_audio_config=texttospeech_v1.StreamingAudioConfig(
audio_encoding=texttospeech_v1.AudioEncoding.PCM, audio_encoding=texttospeech_v1.AudioEncoding.PCM,
sample_rate_hertz=self.sample_rate, sample_rate_hertz=self.sample_rate,
speaking_rate=self._settings["speaking_rate"], speaking_rate=self._settings.speaking_rate,
), ),
) )
@@ -1159,14 +1216,14 @@ class GeminiTTSService(GoogleBaseTTSService):
self._location = location self._location = location
self._model = model self._model = model
self._voice_id = voice_id self._voice_id = voice_id
self._settings = { self._settings: GeminiTTSSettings = GeminiTTSSettings(
"language": self.language_to_service_language(params.language) language=self.language_to_service_language(params.language)
if params.language if params.language
else "en-US", else "en-US",
"prompt": params.prompt, prompt=params.prompt,
"multi_speaker": params.multi_speaker, multi_speaker=params.multi_speaker,
"speaker_configs": params.speaker_configs, speaker_configs=params.speaker_configs,
} )
self._client: texttospeech_v1.TextToSpeechAsyncClient = self._create_client( self._client: texttospeech_v1.TextToSpeechAsyncClient = self._create_client(
credentials, credentials_path credentials, credentials_path
@@ -1183,7 +1240,7 @@ class GeminiTTSService(GoogleBaseTTSService):
""" """
return language_to_gemini_tts_language(language) return language_to_gemini_tts_language(language)
def set_voice(self, voice_id: str): async def set_voice(self, voice_id: str):
"""Set the voice for TTS generation. """Set the voice for TTS generation.
Args: Args:
@@ -1206,15 +1263,13 @@ class GeminiTTSService(GoogleBaseTTSService):
f"Current rate of {self.sample_rate}Hz may cause issues." f"Current rate of {self.sample_rate}Hz may cause issues."
) )
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Override to handle prompt updates. """Override to handle prompt updates.
Args: Args:
settings: Dictionary of settings to update. Can include 'prompt' (str) update: Typed settings delta. Can include 'prompt' (str).
""" """
if "prompt" in settings: return await super()._update_settings_from_typed(update)
self._settings["prompt"] = settings["prompt"]
await super()._update_settings(settings)
@traced_tts @traced_tts
async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]: async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]:
@@ -1234,10 +1289,10 @@ class GeminiTTSService(GoogleBaseTTSService):
await self.start_ttfb_metrics() await self.start_ttfb_metrics()
# Build voice selection params # Build voice selection params
if self._settings["multi_speaker"] and self._settings["speaker_configs"]: if self._settings.multi_speaker and self._settings.speaker_configs:
# Multi-speaker mode # Multi-speaker mode
speaker_voice_configs = [] speaker_voice_configs = []
for speaker_config in self._settings["speaker_configs"]: for speaker_config in self._settings.speaker_configs:
speaker_voice_configs.append( speaker_voice_configs.append(
texttospeech_v1.MultispeakerPrebuiltVoice( texttospeech_v1.MultispeakerPrebuiltVoice(
speaker_alias=speaker_config["speaker_alias"], speaker_alias=speaker_config["speaker_alias"],
@@ -1250,14 +1305,14 @@ class GeminiTTSService(GoogleBaseTTSService):
) )
voice = texttospeech_v1.VoiceSelectionParams( voice = texttospeech_v1.VoiceSelectionParams(
language_code=self._settings["language"], language_code=self._settings.language,
model_name=self._model, model_name=self._model,
multi_speaker_voice_config=multi_speaker_voice_config, multi_speaker_voice_config=multi_speaker_voice_config,
) )
else: else:
# Single speaker mode # Single speaker mode
voice = texttospeech_v1.VoiceSelectionParams( voice = texttospeech_v1.VoiceSelectionParams(
language_code=self._settings["language"], language_code=self._settings.language,
name=self._voice_id, name=self._voice_id,
model_name=self._model, model_name=self._model,
) )
@@ -1273,7 +1328,7 @@ class GeminiTTSService(GoogleBaseTTSService):
# Use base class streaming logic with prompt support # Use base class streaming logic with prompt support
async for frame in self._stream_tts( async for frame in self._stream_tts(
streaming_config, text, context_id, self._settings["prompt"] streaming_config, text, context_id, self._settings.prompt
): ):
yield frame yield frame

46
src/pipecat/services/gradium/stt.py
View File

@@ -12,6 +12,7 @@ WebSocket API for streaming audio transcription.
import base64 import base64
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -27,6 +28,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import GRADIUM_TTFS_P99 from pipecat.services.stt_latency import GRADIUM_TTFS_P99
from pipecat.services.stt_service import WebsocketSTTService from pipecat.services.stt_service import WebsocketSTTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -64,6 +66,18 @@ def language_to_gradium_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class GradiumSTTSettings(STTSettings):
"""Typed settings for the Gradium STT service.
Parameters:
delay_in_frames: Delay in audio frames (80ms each) before text is
generated. Higher delays allow more context but increase latency.
"""
delay_in_frames: int = field(default_factory=lambda: NOT_GIVEN)
class GradiumSTTService(WebsocketSTTService): class GradiumSTTService(WebsocketSTTService):
"""Gradium real-time speech-to-text service. """Gradium real-time speech-to-text service.
@@ -127,9 +141,15 @@ class GradiumSTTService(WebsocketSTTService):
self._api_key = api_key self._api_key = api_key
self._api_endpoint_base_url = api_endpoint_base_url self._api_endpoint_base_url = api_endpoint_base_url
self._websocket = None self._websocket = None
self._params = params or GradiumSTTService.InputParams()
self._json_config = json_config self._json_config = json_config
params = params or GradiumSTTService.InputParams()
self._settings: GradiumSTTSettings = GradiumSTTSettings(
language=params.language,
delay_in_frames=params.delay_in_frames if params.delay_in_frames else NOT_GIVEN,
)
self._receive_task = None self._receive_task = None
self._audio_buffer = bytearray() self._audio_buffer = bytearray()
@@ -149,16 +169,22 @@ class GradiumSTTService(WebsocketSTTService):
""" """
return True return True
async def set_language(self, language: Language): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the recognition language and reconnect. """Apply a typed settings update, sync params, and reconnect.
Args: Args:
language: The language to use for speech recognition. update: A :class:`STTSettings` (or ``GradiumSTTSettings``) delta.
Returns:
Set of field names whose values actually changed.
""" """
logger.info(f"Switching STT language to: [{language}]") changed = await super()._update_settings_from_typed(update)
self._params.language = language if not changed:
return changed
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the speech-to-text service. """Start the speech-to-text service.
@@ -298,12 +324,12 @@ class GradiumSTTService(WebsocketSTTService):
json_config = {} json_config = {}
if self._json_config: if self._json_config:
json_config = json.loads(self._json_config) json_config = json.loads(self._json_config)
if self._params.language: if is_given(self._settings.language) and self._settings.language:
gradium_language = language_to_gradium_language(self._params.language) gradium_language = language_to_gradium_language(self._settings.language)
if gradium_language: if gradium_language:
json_config["language"] = gradium_language json_config["language"] = gradium_language
if self._params.delay_in_frames: if is_given(self._settings.delay_in_frames) and self._settings.delay_in_frames:
json_config["delay_in_frames"] = self._params.delay_in_frames json_config["delay_in_frames"] = self._settings.delay_in_frames
if json_config: if json_config:
setup_msg["json_config"] = json_config setup_msg["json_config"] = json_config
await self._websocket.send(json.dumps(setup_msg)) await self._websocket.send(json.dumps(setup_msg))

47
src/pipecat/services/gradium/tts.py
View File

@@ -6,7 +6,8 @@
import base64 import base64
import json import json
from typing import Any, AsyncGenerator, Mapping, Optional from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
from pydantic import BaseModel from pydantic import BaseModel
@@ -22,6 +23,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import InterruptibleWordTTSService from pipecat.services.tts_service import InterruptibleWordTTSService
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -37,6 +39,17 @@ except ModuleNotFoundError as e:
SAMPLE_RATE = 48000 SAMPLE_RATE = 48000
@dataclass
class GradiumTTSSettings(TTSSettings):
"""Typed settings for the Gradium TTS service.
Parameters:
output_format: Audio output format.
"""
output_format: str = field(default_factory=lambda: NOT_GIVEN)
class GradiumTTSService(InterruptibleWordTTSService): class GradiumTTSService(InterruptibleWordTTSService):
"""Text-to-Speech service using Gradium's websocket API.""" """Text-to-Speech service using Gradium's websocket API."""
@@ -86,12 +99,11 @@ class GradiumTTSService(InterruptibleWordTTSService):
self._url = url self._url = url
self._voice_id = voice_id self._voice_id = voice_id
self._json_config = json_config self._json_config = json_config
self._model = model self._settings: GradiumTTSSettings = GradiumTTSSettings(
self._settings = { model=model,
"voice_id": voice_id, voice=voice_id,
"model_name": model, output_format="pcm",
"output_format": "pcm", )
}
# State tracking # State tracking
self._receive_task = None self._receive_task = None
@@ -105,24 +117,21 @@ class GradiumTTSService(InterruptibleWordTTSService):
""" """
return True return True
async def set_model(self, model: str): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Update the TTS model. """Apply a typed settings update and reconnect if voice changed.
Args: Args:
model: The model name to use for synthesis. update: A :class:`TTSSettings` (or ``GradiumTTSSettings``) delta.
"""
self._model = model
await super().set_model(model)
async def _update_settings(self, settings: Mapping[str, Any]): Returns:
"""Update service settings and reconnect if voice changed.""" Set of field names whose values actually changed.
"""
prev_voice = self._voice_id prev_voice = self._voice_id
await super()._update_settings(settings) changed = await super()._update_settings_from_typed(update)
if not prev_voice == self._voice_id: if self._voice_id != prev_voice:
self._settings["voice_id"] = self._voice_id
logger.info(f"Switching TTS voice to: [{self._voice_id}]")
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed
def _build_msg(self, text: str = "") -> dict: def _build_msg(self, text: str = "") -> dict:
"""Build JSON message for Gradium API.""" """Build JSON message for Gradium API."""

65
src/pipecat/services/grok/realtime/llm.py
View File

@@ -13,8 +13,8 @@ https://docs.x.ai/docs/guides/voice/agent
import base64 import base64
import json import json
import time import time
from dataclasses import dataclass from dataclasses import dataclass, field
from typing import Optional from typing import Any, Optional
from loguru import logger from loguru import logger
@@ -56,6 +56,7 @@ from pipecat.processors.aggregators.llm_response_universal import (
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
from pipecat.services.settings import NOT_GIVEN, LLMSettings
from pipecat.utils.time import time_now_iso8601 from pipecat.utils.time import time_now_iso8601
from . import events from . import events
@@ -85,6 +86,17 @@ class CurrentAudioResponse:
total_size: int = 0 total_size: int = 0
@dataclass
class GrokRealtimeLLMSettings(LLMSettings):
"""Typed settings for Grok Realtime LLM services.
Parameters:
session_properties: Grok Realtime session configuration.
"""
session_properties: Any = field(default_factory=lambda: NOT_GIVEN)
class GrokRealtimeLLMService(LLMService): class GrokRealtimeLLMService(LLMService):
"""Grok Realtime Voice Agent LLM service providing real-time audio and text communication. """Grok Realtime Voice Agent LLM service providing real-time audio and text communication.
@@ -134,9 +146,8 @@ class GrokRealtimeLLMService(LLMService):
self.api_key = api_key self.api_key = api_key
self.base_url = base_url self.base_url = base_url
# Initialize session_properties self._settings = GrokRealtimeLLMSettings(
self._session_properties: events.SessionProperties = ( session_properties=session_properties or events.SessionProperties(),
session_properties or events.SessionProperties()
) )
self._audio_input_paused = start_audio_paused self._audio_input_paused = start_audio_paused
@@ -186,13 +197,13 @@ class GrokRealtimeLLMService(LLMService):
Configured sample rate or None if not manually configured. Configured sample rate or None if not manually configured.
For PCMU/PCMA formats, returns 8000 Hz (G.711 standard). For PCMU/PCMA formats, returns 8000 Hz (G.711 standard).
""" """
if not self._session_properties.audio: if not self._settings.session_properties.audio:
return None return None
audio_config = ( audio_config = (
self._session_properties.audio.input self._settings.session_properties.audio.input
if direction == "input" if direction == "input"
else self._session_properties.audio.output else self._settings.session_properties.audio.output
) )
if audio_config and audio_config.format: if audio_config and audio_config.format:
@@ -222,8 +233,8 @@ class GrokRealtimeLLMService(LLMService):
def _is_turn_detection_enabled(self) -> bool: def _is_turn_detection_enabled(self) -> bool:
"""Check if server-side VAD is enabled.""" """Check if server-side VAD is enabled."""
if self._session_properties.turn_detection: if self._settings.session_properties.turn_detection:
return self._session_properties.turn_detection.type == "server_vad" return self._settings.session_properties.turn_detection.type == "server_vad"
return False return False
async def _handle_interruption(self): async def _handle_interruption(self):
@@ -290,18 +301,18 @@ class GrokRealtimeLLMService(LLMService):
await super().start(frame) await super().start(frame)
# Ensure audio configuration exists with both input and output # Ensure audio configuration exists with both input and output
if not self._session_properties.audio: if not self._settings.session_properties.audio:
self._session_properties.audio = events.AudioConfiguration() self._settings.session_properties.audio = events.AudioConfiguration()
# Fill in missing input configuration # Fill in missing input configuration
if not self._session_properties.audio.input: if not self._settings.session_properties.audio.input:
self._session_properties.audio.input = events.AudioInput( self._settings.session_properties.audio.input = events.AudioInput(
format=events.PCMAudioFormat(rate=frame.audio_in_sample_rate) format=events.PCMAudioFormat(rate=frame.audio_in_sample_rate)
) )
# Fill in missing output configuration # Fill in missing output configuration
if not self._session_properties.audio.output: if not self._settings.session_properties.audio.output:
self._session_properties.audio.output = events.AudioOutput( self._settings.session_properties.audio.output = events.AudioOutput(
format=events.PCMAudioFormat(rate=frame.audio_out_sample_rate) format=events.PCMAudioFormat(rate=frame.audio_out_sample_rate)
) )
@@ -336,6 +347,16 @@ class GrokRealtimeLLMService(LLMService):
frame: The frame to process. frame: The frame to process.
direction: The direction of frame flow in the pipeline. direction: The direction of frame flow in the pipeline.
""" """
# Legacy dict path: frame.settings contains SessionProperties fields,
# not our Settings fields, so we construct SessionProperties directly.
# The new typed path (frame.update) falls through to super, which calls
# _update_settings_from_typed → our override handles the rest.
if isinstance(frame, LLMUpdateSettingsFrame) and frame.update is None:
self._settings.session_properties = events.SessionProperties(**frame.settings)
await self._update_settings()
await self.push_frame(frame, direction)
return
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if isinstance(frame, TranscriptionFrame): if isinstance(frame, TranscriptionFrame):
@@ -355,9 +376,6 @@ class GrokRealtimeLLMService(LLMService):
await self._handle_bot_stopped_speaking() await self._handle_bot_stopped_speaking()
elif isinstance(frame, LLMMessagesAppendFrame): elif isinstance(frame, LLMMessagesAppendFrame):
await self._handle_messages_append(frame) await self._handle_messages_append(frame)
elif isinstance(frame, LLMUpdateSettingsFrame):
self._session_properties = events.SessionProperties(**frame.settings)
await self._update_settings()
elif isinstance(frame, LLMSetToolsFrame): elif isinstance(frame, LLMSetToolsFrame):
await self._update_settings() await self._update_settings()
@@ -436,9 +454,16 @@ class GrokRealtimeLLMService(LLMService):
return return
await self.push_error(error_msg=f"Error sending client event: {e}", exception=e) await self.push_error(error_msg=f"Error sending client event: {e}", exception=e)
async def _update_settings_from_typed(self, update):
"""Apply a typed settings update, sending a session update if needed."""
changed = await super()._update_settings_from_typed(update)
if "session_properties" in changed:
await self._update_settings()
return changed
async def _update_settings(self): async def _update_settings(self):
"""Update session settings on the server.""" """Update session settings on the server."""
settings = self._session_properties settings = self._settings.session_properties
adapter: GrokRealtimeLLMAdapter = self.get_llm_adapter() adapter: GrokRealtimeLLMAdapter = self.get_llm_adapter()
if self._context: if self._context:

33
src/pipecat/services/groq/tts.py
View File

@@ -8,6 +8,7 @@
import io import io
import wave import wave
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -20,6 +21,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -32,6 +34,21 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class GroqTTSSettings(TTSSettings):
"""Typed settings for the Groq TTS service.
Parameters:
output_format: Audio output format.
speed: Speech speed multiplier. Defaults to 1.0.
groq_sample_rate: Audio sample rate.
"""
output_format: str = field(default_factory=lambda: NOT_GIVEN)
speed: float = field(default_factory=lambda: NOT_GIVEN)
groq_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
class GroqTTSService(TTSService): class GroqTTSService(TTSService):
"""Groq text-to-speech service implementation. """Groq text-to-speech service implementation.
@@ -92,14 +109,14 @@ class GroqTTSService(TTSService):
self._voice_id = voice_id self._voice_id = voice_id
self._params = params self._params = params
self._settings = { self._settings: GroqTTSSettings = GroqTTSSettings(
"model": model_name, model=model_name,
"voice_id": voice_id, voice=voice_id,
"output_format": output_format, language=str(params.language) if params.language else "en",
"language": str(params.language) if params.language else "en", output_format=output_format,
"speed": params.speed, speed=params.speed,
"sample_rate": sample_rate, groq_sample_rate=sample_rate,
} )
self._client = AsyncGroq(api_key=self._api_key) self._client = AsyncGroq(api_key=self._api_key)

35
src/pipecat/services/hathora/stt.py
View File

@@ -8,6 +8,7 @@
import base64 import base64
import os import os
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
import aiohttp import aiohttp
@@ -18,6 +19,7 @@ from pipecat.frames.frames import (
Frame, Frame,
TranscriptionFrame, TranscriptionFrame,
) )
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import HATHORA_TTFS_P99 from pipecat.services.stt_latency import HATHORA_TTFS_P99
from pipecat.services.stt_service import SegmentedSTTService from pipecat.services.stt_service import SegmentedSTTService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -27,6 +29,19 @@ from pipecat.utils.tracing.service_decorators import traced_stt
from .utils import ConfigOption from .utils import ConfigOption
@dataclass
class HathoraSTTSettings(STTSettings):
"""Typed settings for the Hathora STT service.
Parameters:
config: Some models support additional config, refer to
`docs <https://models.hathora.dev>`_ for each model to see
what is supported.
"""
config: Optional[list] = field(default_factory=lambda: NOT_GIVEN)
class HathoraSTTService(SegmentedSTTService): class HathoraSTTService(SegmentedSTTService):
"""This service supports several different speech-to-text models hosted by Hathora. """This service supports several different speech-to-text models hosted by Hathora.
@@ -83,10 +98,11 @@ class HathoraSTTService(SegmentedSTTService):
params = params or HathoraSTTService.InputParams() params = params or HathoraSTTService.InputParams()
self._settings = { self._settings: HathoraSTTSettings = HathoraSTTSettings(
"language": params.language, model=model,
"config": params.config, language=params.language,
} config=params.config,
)
self.set_model_name(model) self.set_model_name(model)
@@ -123,12 +139,11 @@ class HathoraSTTService(SegmentedSTTService):
"model": self._model, "model": self._model,
} }
if self._settings["language"] is not None: if self._settings.language is not None:
payload["language"] = self._settings["language"] payload["language"] = self._settings.language
if self._settings["config"] is not None: if self._settings.config is not None:
payload["model_config"] = [ payload["model_config"] = [
{"name": option.name, "value": option.value} {"name": option.name, "value": option.value} for option in self._settings.config
for option in self._settings["config"]
] ]
base64_audio = base64.b64encode(audio).decode("utf-8") base64_audio = base64.b64encode(audio).decode("utf-8")
@@ -147,7 +162,7 @@ class HathoraSTTService(SegmentedSTTService):
if text: # Only yield non-empty text if text: # Only yield non-empty text
# Hathora's API currently doesn't return language info # Hathora's API currently doesn't return language info
# so we default to the requested language or "en" # so we default to the requested language or "en"
response_language = self._settings["language"] or "en" response_language = self._settings.language or "en"
await self._handle_transcription(text, True, response_language) await self._handle_transcription(text, True, response_language)
yield TranscriptionFrame( yield TranscriptionFrame(
text, text,

38
src/pipecat/services/hathora/tts.py
View File

@@ -9,6 +9,7 @@
import io import io
import os import os
import wave import wave
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional, Tuple from typing import AsyncGenerator, Optional, Tuple
import aiohttp import aiohttp
@@ -21,6 +22,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -45,6 +47,21 @@ def _decode_audio_payload(
return audio_bytes, fallback_sample_rate, fallback_channels return audio_bytes, fallback_sample_rate, fallback_channels
@dataclass
class HathoraTTSSettings(TTSSettings):
"""Typed settings for Hathora TTS service.
Parameters:
speed: Speech speed multiplier (if supported by model).
config: Some models support additional config, refer to
[docs](https://models.hathora.dev) for each model to see
what is supported.
"""
speed: float = field(default_factory=lambda: NOT_GIVEN)
config: list = field(default_factory=lambda: NOT_GIVEN)
class HathoraTTSService(TTSService): class HathoraTTSService(TTSService):
"""This service supports several different text-to-speech models hosted by Hathora. """This service supports several different text-to-speech models hosted by Hathora.
@@ -98,13 +115,15 @@ class HathoraTTSService(TTSService):
params = params or HathoraTTSService.InputParams() params = params or HathoraTTSService.InputParams()
self._settings = { self._settings: HathoraTTSSettings = HathoraTTSSettings(
"speed": params.speed, model=model,
"config": params.config, voice=voice_id,
} speed=params.speed,
config=params.config,
)
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice_id) self._voice_id = voice_id
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -135,12 +154,11 @@ class HathoraTTSService(TTSService):
if self._voice_id is not None: if self._voice_id is not None:
payload["voice"] = self._voice_id payload["voice"] = self._voice_id
if self._settings["speed"] is not None: if self._settings.speed is not None:
payload["speed"] = self._settings["speed"] payload["speed"] = self._settings.speed
if self._settings["config"] is not None: if self._settings.config is not None:
payload["model_config"] = [ payload["model_config"] = [
{"name": option.name, "value": option.value} {"name": option.name, "value": option.value} for option in self._settings.config
for option in self._settings["config"]
] ]
yield TTSStartedFrame(context_id=context_id) yield TTSStartedFrame(context_id=context_id)

4
src/pipecat/services/hume/tts.py
View File

@@ -117,7 +117,7 @@ class HumeTTSService(WordTTSService):
self._params = params or HumeTTSService.InputParams() self._params = params or HumeTTSService.InputParams()
# Store voice in the base class (mirrors other services) # Store voice in the base class (mirrors other services)
self.set_voice(voice_id) self._voice_id = voice_id
self._audio_bytes = b"" self._audio_bytes = b""
@@ -196,7 +196,7 @@ class HumeTTSService(WordTTSService):
key_l = (key or "").lower() key_l = (key or "").lower()
if key_l == "voice_id": if key_l == "voice_id":
self.set_voice(str(value)) await self.set_voice(str(value))
logger.debug(f"HumeTTSService voice_id set to: {self.voice}") logger.debug(f"HumeTTSService voice_id set to: {self.voice}")
elif key_l == "description": elif key_l == "description":
self._params.description = None if value is None else str(value) self._params.description = None if value is None else str(value)

120
src/pipecat/services/inworld/tts.py
View File

@@ -16,6 +16,7 @@ Inworld’s text-to-speech (TTS) models offer ultra-realistic, context-aware spe
import asyncio import asyncio
import base64 import base64
import json import json
from dataclasses import dataclass, field
from typing import Any, AsyncGenerator, Dict, List, Optional, Tuple from typing import Any, AsyncGenerator, Dict, List, Optional, Tuple
import aiohttp import aiohttp
@@ -23,6 +24,8 @@ import websockets
from loguru import logger from loguru import logger
from pydantic import BaseModel from pydantic import BaseModel
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
try: try:
from websockets.asyncio.client import connect as websocket_connect from websockets.asyncio.client import connect as websocket_connect
from websockets.protocol import State from websockets.protocol import State
@@ -47,6 +50,31 @@ from pipecat.services.tts_service import AudioContextWordTTSService, WordTTSServ
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@dataclass
class InworldTTSSettings(TTSSettings):
"""Typed settings for Inworld TTS services.
Parameters:
audio_encoding: Audio encoding format (e.g. LINEAR16).
audio_sample_rate: Audio sample rate in Hz.
speaking_rate: Speaking rate for speech synthesis.
temperature: Temperature for speech synthesis.
auto_mode: Whether to use auto mode. Recommended when texts are sent
in full sentences/phrases. When enabled, the server controls
flushing of buffered text to achieve minimal latency while
maintaining high quality audio output. If None (default),
automatically set based on aggregate_sentences.
apply_text_normalization: Whether to apply text normalization.
"""
audio_encoding: str = field(default_factory=lambda: NOT_GIVEN)
audio_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
speaking_rate: float = field(default_factory=lambda: NOT_GIVEN)
temperature: float = field(default_factory=lambda: NOT_GIVEN)
auto_mode: bool = field(default_factory=lambda: NOT_GIVEN)
apply_text_normalization: str = field(default_factory=lambda: NOT_GIVEN)
class InworldHttpTTSService(WordTTSService): class InworldHttpTTSService(WordTTSService):
"""Inworld AI HTTP-based TTS service. """Inworld AI HTTP-based TTS service.
@@ -110,23 +138,21 @@ class InworldHttpTTSService(WordTTSService):
else: else:
self._base_url = "https://api.inworld.ai/tts/v1/voice" self._base_url = "https://api.inworld.ai/tts/v1/voice"
self._settings = { self._settings: InworldTTSSettings = InworldTTSSettings(
"voiceId": voice_id, model=model,
"modelId": model, voice=voice_id,
"audioConfig": { audio_encoding=encoding,
"audioEncoding": encoding, audio_sample_rate=0,
"sampleRateHertz": 0, )
},
}
if params.temperature is not None: if params.temperature is not None:
self._settings["temperature"] = params.temperature self._settings.temperature = params.temperature
if params.speaking_rate is not None: if params.speaking_rate is not None:
self._settings["audioConfig"]["speakingRate"] = params.speaking_rate self._settings.speaking_rate = params.speaking_rate
self._cumulative_time = 0.0 self._cumulative_time = 0.0
self.set_voice(voice_id) self._voice_id = voice_id
self.set_model_name(model) self.set_model_name(model)
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
@@ -144,7 +170,7 @@ class InworldHttpTTSService(WordTTSService):
frame: The start frame. frame: The start frame.
""" """
await super().start(frame) await super().start(frame)
self._settings["audioConfig"]["sampleRateHertz"] = self.sample_rate self._settings.audio_sample_rate = self.sample_rate
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
"""Stop the Inworld TTS service. """Stop the Inworld TTS service.
@@ -223,15 +249,22 @@ class InworldHttpTTSService(WordTTSService):
""" """
logger.debug(f"{self}: Generating TTS [{text}] (streaming={self._streaming})") logger.debug(f"{self}: Generating TTS [{text}] (streaming={self._streaming})")
audio_config = {
"audioEncoding": self._settings.audio_encoding,
"sampleRateHertz": self._settings.audio_sample_rate,
}
if is_given(self._settings.speaking_rate):
audio_config["speakingRate"] = self._settings.speaking_rate
payload = { payload = {
"text": text, "text": text,
"voiceId": self._settings["voiceId"], "voiceId": self._settings.voice,
"modelId": self._settings["modelId"], "modelId": self._settings.model,
"audioConfig": self._settings["audioConfig"], "audioConfig": audio_config,
} }
if "temperature" in self._settings: if is_given(self._settings.temperature):
payload["temperature"] = self._settings["temperature"] payload["temperature"] = self._settings.temperature
# Use WORD timestamps for simplicity and correct spacing/capitalization # Use WORD timestamps for simplicity and correct spacing/capitalization
payload["timestampType"] = self._timestamp_type payload["timestampType"] = self._timestamp_type
@@ -470,27 +503,25 @@ class InworldTTSService(AudioContextWordTTSService):
self._api_key = api_key self._api_key = api_key
self._url = url self._url = url
self._settings: Dict[str, Any] = { self._settings: InworldTTSSettings = InworldTTSSettings(
"voiceId": voice_id, model=model,
"modelId": model, voice=voice_id,
"audioConfig": { audio_encoding=encoding,
"audioEncoding": encoding, audio_sample_rate=0,
"sampleRateHertz": 0, )
},
}
self._timestamp_type = "WORD" self._timestamp_type = "WORD"
if params.temperature is not None: if params.temperature is not None:
self._settings["temperature"] = params.temperature self._settings.temperature = params.temperature
if params.speaking_rate is not None: if params.speaking_rate is not None:
self._settings["audioConfig"]["speakingRate"] = params.speaking_rate self._settings.speaking_rate = params.speaking_rate
if params.apply_text_normalization is not None: if params.apply_text_normalization is not None:
self._settings["applyTextNormalization"] = params.apply_text_normalization self._settings.apply_text_normalization = params.apply_text_normalization
if params.auto_mode is not None: if params.auto_mode is not None:
self._settings["autoMode"] = params.auto_mode self._settings.auto_mode = params.auto_mode
else: else:
self._settings["autoMode"] = aggregate_sentences self._settings.auto_mode = aggregate_sentences
self._buffer_settings = { self._buffer_settings = {
"maxBufferDelayMs": params.max_buffer_delay_ms, "maxBufferDelayMs": params.max_buffer_delay_ms,
@@ -509,7 +540,7 @@ class InworldTTSService(AudioContextWordTTSService):
# Track the end time of the last word in the current generation # Track the end time of the last word in the current generation
self._generation_end_time = 0.0 self._generation_end_time = 0.0
self.set_voice(voice_id) self._voice_id = voice_id
self.set_model_name(model) self.set_model_name(model)
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
@@ -527,7 +558,7 @@ class InworldTTSService(AudioContextWordTTSService):
frame: The start frame. frame: The start frame.
""" """
await super().start(frame) await super().start(frame)
self._settings["audioConfig"]["sampleRateHertz"] = self.sample_rate self._settings.audio_sample_rate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -859,18 +890,25 @@ class InworldTTSService(AudioContextWordTTSService):
Args: Args:
context_id: The context ID. context_id: The context ID.
""" """
audio_config = {
"audioEncoding": self._settings.audio_encoding,
"sampleRateHertz": self._settings.audio_sample_rate,
}
if is_given(self._settings.speaking_rate):
audio_config["speakingRate"] = self._settings.speaking_rate
create_config: Dict[str, Any] = { create_config: Dict[str, Any] = {
"voiceId": self._settings["voiceId"], "voiceId": self._settings.voice,
"modelId": self._settings["modelId"], "modelId": self._settings.model,
"audioConfig": self._settings["audioConfig"], "audioConfig": audio_config,
} }
if "temperature" in self._settings: if is_given(self._settings.temperature):
create_config["temperature"] = self._settings["temperature"] create_config["temperature"] = self._settings.temperature
if "applyTextNormalization" in self._settings: if is_given(self._settings.apply_text_normalization):
create_config["applyTextNormalization"] = self._settings["applyTextNormalization"] create_config["applyTextNormalization"] = self._settings.apply_text_normalization
if "autoMode" in self._settings: if is_given(self._settings.auto_mode):
create_config["autoMode"] = self._settings["autoMode"] create_config["autoMode"] = self._settings.auto_mode
# Set buffer settings for timely audio generation. # Set buffer settings for timely audio generation.
# Use provided values or defaults that work well for streaming LLM output. # Use provided values or defaults that work well for streaming LLM output.

19
src/pipecat/services/kokoro/tts.py
View File

@@ -7,6 +7,7 @@
"""Kokoro TTS service implementation using kokoro-onnx.""" """Kokoro TTS service implementation using kokoro-onnx."""
import os import os
from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
@@ -22,6 +23,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -87,6 +89,17 @@ def language_to_kokoro_language(language: Language) -> str:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class KokoroTTSSettings(TTSSettings):
"""Typed settings for the Kokoro TTS service.
Parameters:
lang_code: Kokoro language code for synthesis.
"""
lang_code: str = field(default_factory=lambda: NOT_GIVEN)
class KokoroTTSService(TTSService): class KokoroTTSService(TTSService):
"""Kokoro TTS service implementation. """Kokoro TTS service implementation.
@@ -129,6 +142,12 @@ class KokoroTTSService(TTSService):
self._voice_id = voice_id self._voice_id = voice_id
self._lang_code = language_to_kokoro_language(params.language) self._lang_code = language_to_kokoro_language(params.language)
self._settings: KokoroTTSSettings = KokoroTTSSettings(
voice=voice_id,
language=language_to_kokoro_language(params.language),
lang_code=language_to_kokoro_language(params.language),
)
model = Path(model_path) if model_path else KOKORO_CACHE_DIR / "kokoro-v1.0.onnx" model = Path(model_path) if model_path else KOKORO_CACHE_DIR / "kokoro-v1.0.onnx"
voices = Path(voices_path) if voices_path else KOKORO_CACHE_DIR / "voices-v1.0.bin" voices = Path(voices_path) if voices_path else KOKORO_CACHE_DIR / "voices-v1.0.bin"

13
src/pipecat/services/llm_service.py
View File

@@ -44,6 +44,7 @@ from pipecat.frames.frames import (
LLMFullResponseEndFrame, LLMFullResponseEndFrame,
LLMFullResponseStartFrame, LLMFullResponseStartFrame,
LLMTextFrame, LLMTextFrame,
LLMUpdateSettingsFrame,
StartFrame, StartFrame,
UserImageRequestFrame, UserImageRequestFrame,
) )
@@ -58,6 +59,7 @@ from pipecat.processors.aggregators.llm_response import (
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_service import AIService from pipecat.services.ai_service import AIService
from pipecat.services.settings import ServiceSettings
from pipecat.turns.user_turn_completion_mixin import UserTurnCompletionLLMServiceMixin from pipecat.turns.user_turn_completion_mixin import UserTurnCompletionLLMServiceMixin
from pipecat.utils.context.llm_context_summarization import ( from pipecat.utils.context.llm_context_summarization import (
LLMContextSummarizationUtil, LLMContextSummarizationUtil,
@@ -351,6 +353,17 @@ class LLMService(UserTurnCompletionLLMServiceMixin, AIService):
await self._handle_interruptions(frame) await self._handle_interruptions(frame)
elif isinstance(frame, LLMConfigureOutputFrame): elif isinstance(frame, LLMConfigureOutputFrame):
self._skip_tts = frame.skip_tts self._skip_tts = frame.skip_tts
elif isinstance(frame, LLMUpdateSettingsFrame):
# New path: typed settings update object.
if frame.update is not None:
await self._update_settings_from_typed(frame.update)
# Legacy path: plain dict, but service uses typed settings — convert.
elif isinstance(self._settings, ServiceSettings):
update = type(self._settings).from_mapping(frame.settings)
await self._update_settings_from_typed(update)
# Legacy path: plain dict, service still uses dict-based settings.
else:
await self._update_settings(frame.settings)
elif isinstance(frame, LLMContextSummaryRequestFrame): elif isinstance(frame, LLMContextSummaryRequestFrame):
await self._handle_summary_request(frame) await self._handle_summary_request(frame)

29
src/pipecat/services/lmnt/tts.py
View File

@@ -7,6 +7,7 @@
"""LMNT text-to-speech service implementation.""" """LMNT text-to-speech service implementation."""
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -23,6 +24,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import InterruptibleTTSService from pipecat.services.tts_service import InterruptibleTTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -71,6 +73,17 @@ def language_to_lmnt_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class LmntTTSSettings(TTSSettings):
"""Typed settings for LMNT TTS service.
Parameters:
format: Audio output format. Defaults to "raw".
"""
format: str = field(default_factory=lambda: NOT_GIVEN)
class LmntTTSService(InterruptibleTTSService): class LmntTTSService(InterruptibleTTSService):
"""LMNT real-time text-to-speech service. """LMNT real-time text-to-speech service.
@@ -107,12 +120,14 @@ class LmntTTSService(InterruptibleTTSService):
) )
self._api_key = api_key self._api_key = api_key
self.set_voice(voice_id) self._voice_id = voice_id
self.set_model_name(model) self.set_model_name(model)
self._settings = { self._settings: LmntTTSSettings = LmntTTSSettings(
"language": self.language_to_service_language(language), model=model,
"format": "raw", # Use raw format for direct PCM data voice=voice_id,
} language=self.language_to_service_language(language),
format="raw",
)
self._receive_task = None self._receive_task = None
self._context_id: Optional[str] = None self._context_id: Optional[str] = None
@@ -202,9 +217,9 @@ class LmntTTSService(InterruptibleTTSService):
init_msg = { init_msg = {
"X-API-Key": self._api_key, "X-API-Key": self._api_key,
"voice": self._voice_id, "voice": self._voice_id,
"format": self._settings["format"], "format": self._settings.format,
"sample_rate": self.sample_rate, "sample_rate": self.sample_rate,
"language": self._settings["language"], "language": self._settings.language,
"model": self.model_name, "model": self.model_name,
} }

118
src/pipecat/services/minimax/tts.py
View File

@@ -11,6 +11,7 @@ for streaming text-to-speech synthesis.
""" """
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
import aiohttp import aiohttp
@@ -25,6 +26,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -85,6 +87,40 @@ def language_to_minimax_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class MiniMaxTTSSettings(TTSSettings):
"""Typed settings for MiniMax TTS service.
Parameters:
stream: Whether to use streaming mode.
speed: Speech speed (range: 0.5 to 2.0).
volume: Speech volume (range: 0 to 10).
pitch: Pitch adjustment (range: -12 to 12).
emotion: Emotional tone (options: "happy", "sad", "angry", "fearful",
"disgusted", "surprised", "calm", "fluent").
text_normalization: Enable text normalization (Chinese/English).
latex_read: Enable LaTeX formula reading.
audio_bitrate: Audio bitrate in bps.
audio_format: Audio output format.
audio_channel: Number of audio channels.
audio_sample_rate: Audio sample rate in Hz.
language_boost: Language boost string for multilingual support.
"""
stream: bool = field(default_factory=lambda: NOT_GIVEN)
speed: float = field(default_factory=lambda: NOT_GIVEN)
volume: float = field(default_factory=lambda: NOT_GIVEN)
pitch: int = field(default_factory=lambda: NOT_GIVEN)
emotion: str = field(default_factory=lambda: NOT_GIVEN)
text_normalization: bool = field(default_factory=lambda: NOT_GIVEN)
latex_read: bool = field(default_factory=lambda: NOT_GIVEN)
audio_bitrate: int = field(default_factory=lambda: NOT_GIVEN)
audio_format: str = field(default_factory=lambda: NOT_GIVEN)
audio_channel: int = field(default_factory=lambda: NOT_GIVEN)
audio_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
language_boost: str = field(default_factory=lambda: NOT_GIVEN)
class MiniMaxHttpTTSService(TTSService): class MiniMaxHttpTTSService(TTSService):
"""Text-to-speech service using MiniMax's T2A (Text-to-Audio) API. """Text-to-speech service using MiniMax's T2A (Text-to-Audio) API.
@@ -172,29 +208,27 @@ class MiniMaxHttpTTSService(TTSService):
self._voice_id = voice_id self._voice_id = voice_id
# Create voice settings # Create voice settings
self._settings = { self._settings: MiniMaxTTSSettings = MiniMaxTTSSettings(
"stream": True, model=model,
"voice_setting": { voice=voice_id,
"speed": params.speed, stream=True,
"vol": params.volume, speed=params.speed,
"pitch": params.pitch, volume=params.volume,
}, pitch=params.pitch,
"audio_setting": { audio_bitrate=128000,
"bitrate": 128000, audio_format="pcm",
"format": "pcm", audio_channel=1,
"channel": 1, )
},
}
# Set voice and model # Set voice and model
self.set_voice(voice_id) self._voice_id = voice_id
self.set_model_name(model) self.set_model_name(model)
# Add language boost if provided # Add language boost if provided
if params.language: if params.language:
service_lang = self.language_to_service_language(params.language) service_lang = self.language_to_service_language(params.language)
if service_lang: if service_lang:
self._settings["language_boost"] = service_lang self._settings.language_boost = service_lang
# Add optional emotion if provided # Add optional emotion if provided
if params.emotion: if params.emotion:
@@ -210,7 +244,7 @@ class MiniMaxHttpTTSService(TTSService):
"fluent", "fluent",
] ]
if params.emotion in supported_emotions: if params.emotion in supported_emotions:
self._settings["voice_setting"]["emotion"] = params.emotion self._settings.emotion = params.emotion
else: else:
logger.warning( logger.warning(
f"Unsupported emotion: {params.emotion}. Supported emotions: {supported_emotions}" f"Unsupported emotion: {params.emotion}. Supported emotions: {supported_emotions}"
@@ -226,15 +260,15 @@ class MiniMaxHttpTTSService(TTSService):
"Parameter `english_normalization` is deprecated and will be removed in a future version. Use `text_normalization` instead.", "Parameter `english_normalization` is deprecated and will be removed in a future version. Use `text_normalization` instead.",
DeprecationWarning, DeprecationWarning,
) )
self._settings["voice_setting"]["text_normalization"] = params.english_normalization self._settings.text_normalization = params.english_normalization
# Add text_normalization if provided (corrected parameter name) # Add text_normalization if provided (corrected parameter name)
if params.text_normalization is not None: if params.text_normalization is not None:
self._settings["voice_setting"]["text_normalization"] = params.text_normalization self._settings.text_normalization = params.text_normalization
# Add latex_read if provided # Add latex_read if provided
if params.latex_read is not None: if params.latex_read is not None:
self._settings["voice_setting"]["latex_read"] = params.latex_read self._settings.latex_read = params.latex_read
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -263,16 +297,6 @@ class MiniMaxHttpTTSService(TTSService):
""" """
self._model_name = model self._model_name = model
def set_voice(self, voice: str):
"""Set the voice to use.
Args:
voice: The voice identifier to use for synthesis.
"""
self._voice_id = voice
if "voice_setting" in self._settings:
self._settings["voice_setting"]["voice_id"] = voice
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the MiniMax TTS service. """Start the MiniMax TTS service.
@@ -280,7 +304,7 @@ class MiniMaxHttpTTSService(TTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["audio_setting"]["sample_rate"] = self.sample_rate self._settings.audio_sample_rate = self.sample_rate
logger.debug(f"MiniMax TTS initialized with sample_rate: {self.sample_rate}") logger.debug(f"MiniMax TTS initialized with sample_rate: {self.sample_rate}")
@traced_tts @traced_tts
@@ -302,10 +326,38 @@ class MiniMaxHttpTTSService(TTSService):
"Authorization": f"Bearer {self._api_key}", "Authorization": f"Bearer {self._api_key}",
} }
# Build voice_setting dict for API
voice_setting = {
"voice_id": self._voice_id,
"speed": self._settings.speed,
"vol": self._settings.volume,
"pitch": self._settings.pitch,
}
if is_given(self._settings.emotion):
voice_setting["emotion"] = self._settings.emotion
if is_given(self._settings.text_normalization):
voice_setting["text_normalization"] = self._settings.text_normalization
if is_given(self._settings.latex_read):
voice_setting["latex_read"] = self._settings.latex_read
# Build audio_setting dict for API
audio_setting = {
"bitrate": self._settings.audio_bitrate,
"format": self._settings.audio_format,
"channel": self._settings.audio_channel,
"sample_rate": self._settings.audio_sample_rate,
}
# Create payload from settings # Create payload from settings
payload = self._settings.copy() payload = {
payload["model"] = self._model_name "stream": self._settings.stream,
payload["text"] = text "voice_setting": voice_setting,
"audio_setting": audio_setting,
"model": self._model_name,
"text": text,
}
if is_given(self._settings.language_boost):
payload["language_boost"] = self._settings.language_boost
try: try:
await self.start_ttfb_metrics() await self.start_ttfb_metrics()

16
src/pipecat/services/mistral/llm.py
View File

@@ -185,19 +185,19 @@ class MistralLLMService(OpenAILLMService):
"messages": fixed_messages, "messages": fixed_messages,
"tools": params_from_context["tools"], "tools": params_from_context["tools"],
"tool_choice": params_from_context["tool_choice"], "tool_choice": params_from_context["tool_choice"],
"frequency_penalty": self._settings["frequency_penalty"], "frequency_penalty": self._settings.frequency_penalty,
"presence_penalty": self._settings["presence_penalty"], "presence_penalty": self._settings.presence_penalty,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"max_tokens": self._settings["max_tokens"], "max_tokens": self._settings.max_tokens,
} }
# Handle Mistral-specific parameter mapping # Handle Mistral-specific parameter mapping
# Mistral uses "random_seed" instead of "seed" # Mistral uses "random_seed" instead of "seed"
if self._settings["seed"]: if self._settings.seed:
params["random_seed"] = self._settings["seed"] params["random_seed"] = self._settings.seed
# Add any extra parameters # Add any extra parameters
params.update(self._settings["extra"]) params.update(self._settings.extra)
return params return params

61
src/pipecat/services/neuphonic/tts.py
View File

@@ -13,7 +13,8 @@ text-to-speech API for real-time audio synthesis.
import asyncio import asyncio
import base64 import base64
import json import json
from typing import Any, AsyncGenerator, Mapping, Optional from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional
import aiohttp import aiohttp
from loguru import logger from loguru import logger
@@ -34,6 +35,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import InterruptibleTTSService, TTSService from pipecat.services.tts_service import InterruptibleTTSService, TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -72,6 +74,23 @@ def language_to_neuphonic_lang_code(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class NeuphonicTTSSettings(TTSSettings):
"""Typed settings for Neuphonic TTS service.
Parameters:
lang_code: Neuphonic language code.
speed: Speech speed multiplier. Defaults to 1.0.
encoding: Audio encoding format.
sampling_rate: Audio sample rate.
"""
lang_code: str = field(default_factory=lambda: NOT_GIVEN)
speed: float = field(default_factory=lambda: NOT_GIVEN)
encoding: str = field(default_factory=lambda: NOT_GIVEN)
sampling_rate: int = field(default_factory=lambda: NOT_GIVEN)
class NeuphonicTTSService(InterruptibleTTSService): class NeuphonicTTSService(InterruptibleTTSService):
"""Neuphonic real-time text-to-speech service using WebSocket streaming. """Neuphonic real-time text-to-speech service using WebSocket streaming.
@@ -127,13 +146,13 @@ class NeuphonicTTSService(InterruptibleTTSService):
self._api_key = api_key self._api_key = api_key
self._url = url self._url = url
self._settings = { self._settings: NeuphonicTTSSettings = NeuphonicTTSSettings(
"lang_code": self.language_to_service_language(params.language), lang_code=self.language_to_service_language(params.language),
"speed": params.speed, speed=params.speed,
"encoding": encoding, encoding=encoding,
"sampling_rate": sample_rate, sampling_rate=sample_rate,
} )
self.set_voice(voice_id) self._voice_id = voice_id
self._cumulative_time = 0 self._cumulative_time = 0
@@ -160,15 +179,14 @@ class NeuphonicTTSService(InterruptibleTTSService):
""" """
return language_to_neuphonic_lang_code(language) return language_to_neuphonic_lang_code(language)
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Update service settings and reconnect with new configuration.""" """Apply a typed settings update and reconnect with new configuration."""
if "voice_id" in settings: changed = await super()._update_settings_from_typed(update)
self.set_voice(settings["voice_id"]) if changed:
await self._disconnect()
await super()._update_settings(settings) await self._connect()
await self._disconnect() logger.info(f"Switching TTS to settings: [{self._settings}]")
await self._connect() return changed
logger.info(f"Switching TTS to settings: [{self._settings}]")
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the Neuphonic TTS service. """Start the Neuphonic TTS service.
@@ -266,7 +284,10 @@ class NeuphonicTTSService(InterruptibleTTSService):
logger.debug("Connecting to Neuphonic") logger.debug("Connecting to Neuphonic")
tts_config = { tts_config = {
**self._settings, "lang_code": self._settings.lang_code,
"speed": self._settings.speed,
"encoding": self._settings.encoding,
"sampling_rate": self._settings.sampling_rate,
"voice_id": self._voice_id, "voice_id": self._voice_id,
} }
@@ -275,7 +296,7 @@ class NeuphonicTTSService(InterruptibleTTSService):
if value is not None: if value is not None:
query_params.append(f"{key}={value}") query_params.append(f"{key}={value}")
url = f"{self._url}/speak/{self._settings['lang_code']}" url = f"{self._url}/speak/{self._settings.lang_code}"
if query_params: if query_params:
url += f"?{'&'.join(query_params)}" url += f"?{'&'.join(query_params)}"
@@ -429,7 +450,7 @@ class NeuphonicHttpTTSService(TTSService):
self._lang_code = self.language_to_service_language(params.language) or "en" self._lang_code = self.language_to_service_language(params.language) or "en"
self._speed = params.speed self._speed = params.speed
self._encoding = encoding self._encoding = encoding
self.set_voice(voice_id) self._voice_id = voice_id
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.

128
src/pipecat/services/nvidia/stt.py
View File

@@ -8,6 +8,7 @@
import asyncio import asyncio
from concurrent.futures import CancelledError as FuturesCancelledError from concurrent.futures import CancelledError as FuturesCancelledError
from dataclasses import dataclass, field
from typing import AsyncGenerator, List, Mapping, Optional from typing import AsyncGenerator, List, Mapping, Optional
from loguru import logger from loguru import logger
@@ -22,6 +23,7 @@ from pipecat.frames.frames import (
StartFrame, StartFrame,
TranscriptionFrame, TranscriptionFrame,
) )
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import NVIDIA_TTFS_P99 from pipecat.services.stt_latency import NVIDIA_TTFS_P99
from pipecat.services.stt_service import SegmentedSTTService, STTService from pipecat.services.stt_service import SegmentedSTTService, STTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -89,6 +91,32 @@ def language_to_nvidia_riva_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class NvidiaSTTSettings(STTSettings):
"""Typed settings for the NVIDIA Riva streaming STT service."""
pass
@dataclass
class NvidiaSegmentedSTTSettings(STTSettings):
"""Typed settings for the NVIDIA Riva segmented STT service.
Parameters:
profanity_filter: Whether to filter profanity from results.
automatic_punctuation: Whether to add automatic punctuation.
verbatim_transcripts: Whether to return verbatim transcripts.
boosted_lm_words: List of words to boost in language model.
boosted_lm_score: Score boost for specified words.
"""
profanity_filter: bool = field(default_factory=lambda: NOT_GIVEN)
automatic_punctuation: bool = field(default_factory=lambda: NOT_GIVEN)
verbatim_transcripts: bool = field(default_factory=lambda: NOT_GIVEN)
boosted_lm_words: Optional[List[str]] = field(default_factory=lambda: NOT_GIVEN)
boosted_lm_score: float = field(default_factory=lambda: NOT_GIVEN)
class NvidiaSTTService(STTService): class NvidiaSTTService(STTService):
"""Real-time speech-to-text service using NVIDIA Riva streaming ASR. """Real-time speech-to-text service using NVIDIA Riva streaming ASR.
@@ -141,12 +169,6 @@ class NvidiaSTTService(STTService):
self._server = server self._server = server
self._api_key = api_key self._api_key = api_key
self._use_ssl = use_ssl self._use_ssl = use_ssl
self._profanity_filter = False
self._automatic_punctuation = True
self._no_verbatim_transcripts = False
self._language_code = params.language
self._boosted_lm_words = None
self._boosted_lm_score = 4.0
self._start_history = -1 self._start_history = -1
self._start_threshold = -1.0 self._start_threshold = -1.0
self._stop_history = -1 self._stop_history = -1
@@ -156,14 +178,9 @@ class NvidiaSTTService(STTService):
self._custom_configuration = "" self._custom_configuration = ""
self._function_id = model_function_map.get("function_id") self._function_id = model_function_map.get("function_id")
self._settings = { self._settings: NvidiaSTTSettings = NvidiaSTTSettings(
"language": str(params.language), language=params.language,
"profanity_filter": self._profanity_filter, )
"automatic_punctuation": self._automatic_punctuation,
"verbatim_transcripts": not self._no_verbatim_transcripts,
"boosted_lm_words": self._boosted_lm_words,
"boosted_lm_score": self._boosted_lm_score,
}
self.set_model_name(model_function_map.get("model_name")) self.set_model_name(model_function_map.get("model_name"))
@@ -186,22 +203,18 @@ class NvidiaSTTService(STTService):
config = riva.client.StreamingRecognitionConfig( config = riva.client.StreamingRecognitionConfig(
config=riva.client.RecognitionConfig( config=riva.client.RecognitionConfig(
encoding=riva.client.AudioEncoding.LINEAR_PCM, encoding=riva.client.AudioEncoding.LINEAR_PCM,
language_code=self._language_code, language_code=self._settings.language,
model="", model="",
max_alternatives=1, max_alternatives=1,
profanity_filter=self._profanity_filter, profanity_filter=False,
enable_automatic_punctuation=self._automatic_punctuation, enable_automatic_punctuation=True,
verbatim_transcripts=not self._no_verbatim_transcripts, verbatim_transcripts=True,
sample_rate_hertz=self.sample_rate, sample_rate_hertz=self.sample_rate,
audio_channel_count=1, audio_channel_count=1,
), ),
interim_results=True, interim_results=True,
) )
riva.client.add_word_boosting_to_config(
config, self._boosted_lm_words, self._boosted_lm_score
)
riva.client.add_endpoint_parameters_to_config( riva.client.add_endpoint_parameters_to_config(
config, config,
self._start_history, self._start_history,
@@ -318,14 +331,14 @@ class NvidiaSTTService(STTService):
transcript, transcript,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._language_code, self._settings.language,
result=result, result=result,
) )
) )
await self._handle_transcription( await self._handle_transcription(
transcript=transcript, transcript=transcript,
is_final=result.is_final, is_final=result.is_final,
language=self._language_code, language=self._settings.language,
) )
else: else:
await self.push_frame( await self.push_frame(
@@ -333,7 +346,7 @@ class NvidiaSTTService(STTService):
transcript, transcript,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._language_code, self._settings.language,
result=result, result=result,
) )
) )
@@ -445,18 +458,6 @@ class NvidiaSegmentedSTTService(SegmentedSTTService):
self._server = server self._server = server
self._use_ssl = use_ssl self._use_ssl = use_ssl
self._function_id = model_function_map.get("function_id") self._function_id = model_function_map.get("function_id")
self._model_name = model_function_map.get("model_name")
# Store the language as a Language enum and as a string
self._language_enum = params.language or Language.EN_US
self._language = self.language_to_service_language(self._language_enum) or "en-US"
# Configure transcription parameters
self._profanity_filter = params.profanity_filter
self._automatic_punctuation = params.automatic_punctuation
self._verbatim_transcripts = params.verbatim_transcripts
self._boosted_lm_words = params.boosted_lm_words
self._boosted_lm_score = params.boosted_lm_score
# Voice activity detection thresholds (use NVIDIA Riva defaults) # Voice activity detection thresholds (use NVIDIA Riva defaults)
self._start_history = -1 self._start_history = -1
@@ -467,10 +468,16 @@ class NvidiaSegmentedSTTService(SegmentedSTTService):
self._stop_threshold_eou = -1.0 self._stop_threshold_eou = -1.0
self._custom_configuration = "" self._custom_configuration = ""
# Create NVIDIA Riva client
self._config = None self._config = None
self._asr_service = None self._asr_service = None
self._settings = {"language": self._language_enum} self._settings: NvidiaSegmentedSTTSettings = NvidiaSegmentedSTTSettings(
language=params.language or Language.EN_US,
profanity_filter=params.profanity_filter,
automatic_punctuation=params.automatic_punctuation,
verbatim_transcripts=params.verbatim_transcripts,
boosted_lm_words=params.boosted_lm_words,
boosted_lm_score=params.boosted_lm_score,
)
def language_to_service_language(self, language: Language) -> Optional[str]: def language_to_service_language(self, language: Language) -> Optional[str]:
"""Convert pipecat Language enum to NVIDIA Riva's language code. """Convert pipecat Language enum to NVIDIA Riva's language code.
@@ -498,21 +505,25 @@ class NvidiaSegmentedSTTService(SegmentedSTTService):
auth = riva.client.Auth(None, self._use_ssl, self._server, metadata) auth = riva.client.Auth(None, self._use_ssl, self._server, metadata)
self._asr_service = riva.client.ASRService(auth) self._asr_service = riva.client.ASRService(auth)
def _get_language_code(self) -> str:
"""Resolve the current language enum to an NVIDIA Riva language code string."""
return self.language_to_service_language(self._settings.language) or "en-US"
def _create_recognition_config(self): def _create_recognition_config(self):
"""Create the NVIDIA Riva ASR recognition configuration.""" """Create the NVIDIA Riva ASR recognition configuration."""
# Create base configuration # Create base configuration
config = riva.client.RecognitionConfig( config = riva.client.RecognitionConfig(
language_code=self._language, # Now using the string, not a tuple language_code=self._get_language_code(),
max_alternatives=1, max_alternatives=1,
profanity_filter=self._profanity_filter, profanity_filter=self._settings.profanity_filter,
enable_automatic_punctuation=self._automatic_punctuation, enable_automatic_punctuation=self._settings.automatic_punctuation,
verbatim_transcripts=self._verbatim_transcripts, verbatim_transcripts=self._settings.verbatim_transcripts,
) )
# Add word boosting if specified # Add word boosting if specified
if self._boosted_lm_words: if self._settings.boosted_lm_words:
riva.client.add_word_boosting_to_config( riva.client.add_word_boosting_to_config(
config, self._boosted_lm_words, self._boosted_lm_score config, self._settings.boosted_lm_words, self._settings.boosted_lm_score
) )
# Add voice activity detection parameters # Add voice activity detection parameters
@@ -567,20 +578,21 @@ class NvidiaSegmentedSTTService(SegmentedSTTService):
self._config = self._create_recognition_config() self._config = self._create_recognition_config()
logger.debug(f"Initialized NvidiaSegmentedSTTService with model: {self.model_name}") logger.debug(f"Initialized NvidiaSegmentedSTTService with model: {self.model_name}")
async def set_language(self, language: Language): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the language for the STT service. """Apply a typed settings update and sync internal state.
Args: Args:
language: Target language for transcription. update: A :class:`STTSettings` (or ``NvidiaSegmentedSTTSettings``) delta.
"""
logger.info(f"Switching STT language to: [{language}]")
self._language_enum = language
self._language = self.language_to_service_language(language) or "en-US"
self._settings["language"] = language
# Update configuration with new language Returns:
if self._config: Set of field names whose values actually changed.
self._config.language_code = self._language """
changed = await super()._update_settings_from_typed(update)
if changed:
self._config = self._create_recognition_config()
return changed
@traced_stt @traced_stt
async def _handle_transcription( async def _handle_transcription(
@@ -633,11 +645,11 @@ class NvidiaSegmentedSTTService(SegmentedSTTService):
text, text,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._language_enum, self._settings.language,
) )
transcription_found = True transcription_found = True
await self._handle_transcription(text, True, self._language_enum) await self._handle_transcription(text, True, self._settings.language)
if not transcription_found: if not transcription_found:
logger.debug(f"{self}: No transcription results found in NVIDIA Riva response") logger.debug(f"{self}: No transcription results found in NVIDIA Riva response")

2
src/pipecat/services/nvidia/tts.py
View File

@@ -100,7 +100,7 @@ class NvidiaTTSService(TTSService):
self._function_id = model_function_map.get("function_id") self._function_id = model_function_map.get("function_id")
self._use_ssl = use_ssl self._use_ssl = use_ssl
self.set_model_name(model_function_map.get("model_name")) self.set_model_name(model_function_map.get("model_name"))
self.set_voice(voice_id) self._voice_id = voice_id
self._service = None self._service = None
self._config = None self._config = None

62
src/pipecat/services/openai/base_llm.py
View File

@@ -10,7 +10,8 @@ import asyncio
import base64 import base64
import json import json
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from typing import Any, Dict, List, Mapping, Optional from dataclasses import dataclass, field
from typing import Any, ClassVar, Dict, List, Mapping, Optional
import httpx import httpx
from loguru import logger from loguru import logger
@@ -32,7 +33,6 @@ from pipecat.frames.frames import (
LLMFullResponseStartFrame, LLMFullResponseStartFrame,
LLMMessagesFrame, LLMMessagesFrame,
LLMTextFrame, LLMTextFrame,
LLMUpdateSettingsFrame,
) )
from pipecat.metrics.metrics import LLMTokenUsage from pipecat.metrics.metrics import LLMTokenUsage
from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_context import LLMContext
@@ -42,9 +42,24 @@ from pipecat.processors.aggregators.openai_llm_context import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
from pipecat.services.settings import NOT_GIVEN as _NOT_GIVEN
from pipecat.services.settings import LLMSettings
from pipecat.utils.tracing.service_decorators import traced_llm from pipecat.utils.tracing.service_decorators import traced_llm
@dataclass
class OpenAILLMSettings(LLMSettings):
"""Typed settings for OpenAI-compatible LLM services.
Parameters:
max_completion_tokens: Maximum completion tokens to generate.
service_tier: Service tier to use (e.g., "auto", "flex", "priority").
"""
max_completion_tokens: Any = field(default_factory=lambda: _NOT_GIVEN)
service_tier: Any = field(default_factory=lambda: _NOT_GIVEN)
class BaseOpenAILLMService(LLMService): class BaseOpenAILLMService(LLMService):
"""Base class for all services that use the AsyncOpenAI client. """Base class for all services that use the AsyncOpenAI client.
@@ -120,17 +135,18 @@ class BaseOpenAILLMService(LLMService):
params = params or BaseOpenAILLMService.InputParams() params = params or BaseOpenAILLMService.InputParams()
self._settings = { self._settings = OpenAILLMSettings(
"frequency_penalty": params.frequency_penalty, model=model,
"presence_penalty": params.presence_penalty, frequency_penalty=params.frequency_penalty,
"seed": params.seed, presence_penalty=params.presence_penalty,
"temperature": params.temperature, seed=params.seed,
"top_p": params.top_p, temperature=params.temperature,
"max_tokens": params.max_tokens, top_p=params.top_p,
"max_completion_tokens": params.max_completion_tokens, max_tokens=params.max_tokens,
"service_tier": params.service_tier, max_completion_tokens=params.max_completion_tokens,
"extra": params.extra if isinstance(params.extra, dict) else {}, service_tier=params.service_tier,
} extra=params.extra if isinstance(params.extra, dict) else {},
)
self._retry_timeout_secs = retry_timeout_secs self._retry_timeout_secs = retry_timeout_secs
self._retry_on_timeout = retry_on_timeout self._retry_on_timeout = retry_on_timeout
self.set_model_name(model) self.set_model_name(model)
@@ -250,20 +266,20 @@ class BaseOpenAILLMService(LLMService):
"model": self.model_name, "model": self.model_name,
"stream": True, "stream": True,
"stream_options": {"include_usage": True}, "stream_options": {"include_usage": True},
"frequency_penalty": self._settings["frequency_penalty"], "frequency_penalty": self._settings.frequency_penalty,
"presence_penalty": self._settings["presence_penalty"], "presence_penalty": self._settings.presence_penalty,
"seed": self._settings["seed"], "seed": self._settings.seed,
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"max_tokens": self._settings["max_tokens"], "max_tokens": self._settings.max_tokens,
"max_completion_tokens": self._settings["max_completion_tokens"], "max_completion_tokens": self._settings.max_completion_tokens,
"service_tier": self._settings["service_tier"], "service_tier": self._settings.service_tier,
} }
# Messages, tools, tool_choice # Messages, tools, tool_choice
params.update(params_from_context) params.update(params_from_context)
params.update(self._settings["extra"]) params.update(self._settings.extra)
return params return params
async def run_inference( async def run_inference(
@@ -508,8 +524,6 @@ class BaseOpenAILLMService(LLMService):
# NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal # NOTE: LLMMessagesFrame is deprecated, so we don't support the newer universal
# LLMContext with it # LLMContext with it
context = OpenAILLMContext.from_messages(frame.messages) context = OpenAILLMContext.from_messages(frame.messages)
elif isinstance(frame, LLMUpdateSettingsFrame):
await self._update_settings(frame.settings)
else: else:
await self.push_frame(frame, direction) await self.push_frame(frame, direction)

60
src/pipecat/services/openai/realtime/llm.py
View File

@@ -10,8 +10,8 @@ import base64
import io import io
import json import json
import time import time
from dataclasses import dataclass from dataclasses import dataclass, field
from typing import Optional from typing import Any, Optional
from loguru import logger from loguru import logger
from PIL import Image from PIL import Image
@@ -59,6 +59,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
from pipecat.services.settings import NOT_GIVEN, LLMSettings
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601 from pipecat.utils.time import time_now_iso8601
from pipecat.utils.tracing.service_decorators import traced_openai_realtime, traced_stt from pipecat.utils.tracing.service_decorators import traced_openai_realtime, traced_stt
@@ -90,6 +91,17 @@ class CurrentAudioResponse:
total_size: int = 0 total_size: int = 0
@dataclass
class OpenAIRealtimeLLMSettings(LLMSettings):
"""Typed settings for OpenAI Realtime LLM services.
Parameters:
session_properties: OpenAI Realtime session configuration.
"""
session_properties: Any = field(default_factory=lambda: NOT_GIVEN)
class OpenAIRealtimeLLMService(LLMService): class OpenAIRealtimeLLMService(LLMService):
"""OpenAI Realtime LLM service providing real-time audio and text communication. """OpenAI Realtime LLM service providing real-time audio and text communication.
@@ -161,9 +173,9 @@ class OpenAIRealtimeLLMService(LLMService):
self.base_url = full_url self.base_url = full_url
self.set_model_name(model) self.set_model_name(model)
# Initialize session_properties self._settings = OpenAIRealtimeLLMSettings(
self._session_properties: events.SessionProperties = ( model=model,
session_properties or events.SessionProperties() session_properties=session_properties or events.SessionProperties(),
) )
self._audio_input_paused = start_audio_paused self._audio_input_paused = start_audio_paused
self._video_input_paused = start_video_paused self._video_input_paused = start_video_paused
@@ -227,12 +239,12 @@ class OpenAIRealtimeLLMService(LLMService):
def _is_modality_enabled(self, modality: str) -> bool: def _is_modality_enabled(self, modality: str) -> bool:
"""Check if a specific modality is enabled, "text" or "audio".""" """Check if a specific modality is enabled, "text" or "audio"."""
modalities = self._session_properties.output_modalities or ["audio", "text"] modalities = self._settings.session_properties.output_modalities or ["audio", "text"]
return modality in modalities return modality in modalities
def _get_enabled_modalities(self) -> list[str]: def _get_enabled_modalities(self) -> list[str]:
"""Get the list of enabled modalities.""" """Get the list of enabled modalities."""
modalities = self._session_properties.output_modalities or ["audio", "text"] modalities = self._settings.session_properties.output_modalities or ["audio", "text"]
# API only supports single modality responses: either ["text"] or ["audio"] # API only supports single modality responses: either ["text"] or ["audio"]
if "audio" in modalities: if "audio" in modalities:
return ["audio"] return ["audio"]
@@ -305,9 +317,9 @@ class OpenAIRealtimeLLMService(LLMService):
# None and False are different. Check for False. None means we're using OpenAI's # None and False are different. Check for False. None means we're using OpenAI's
# built-in turn detection defaults. # built-in turn detection defaults.
turn_detection_disabled = ( turn_detection_disabled = (
self._session_properties.audio self._settings.session_properties.audio
and self._session_properties.audio.input and self._settings.session_properties.audio.input
and self._session_properties.audio.input.turn_detection is False and self._settings.session_properties.audio.input.turn_detection is False
) )
if turn_detection_disabled: if turn_detection_disabled:
await self.send_client_event(events.InputAudioBufferClearEvent()) await self.send_client_event(events.InputAudioBufferClearEvent())
@@ -327,9 +339,9 @@ class OpenAIRealtimeLLMService(LLMService):
# None and False are different. Check for False. None means we're using OpenAI's # None and False are different. Check for False. None means we're using OpenAI's
# built-in turn detection defaults. # built-in turn detection defaults.
turn_detection_disabled = ( turn_detection_disabled = (
self._session_properties.audio self._settings.session_properties.audio
and self._session_properties.audio.input and self._settings.session_properties.audio.input
and self._session_properties.audio.input.turn_detection is False and self._settings.session_properties.audio.input.turn_detection is False
) )
if turn_detection_disabled: if turn_detection_disabled:
await self.send_client_event(events.InputAudioBufferCommitEvent()) await self.send_client_event(events.InputAudioBufferCommitEvent())
@@ -397,6 +409,16 @@ class OpenAIRealtimeLLMService(LLMService):
frame: The frame to process. frame: The frame to process.
direction: The direction of frame flow in the pipeline. direction: The direction of frame flow in the pipeline.
""" """
# Legacy dict path: frame.settings contains SessionProperties fields,
# not our Settings fields, so we construct SessionProperties directly.
# The new typed path (frame.update) falls through to super, which calls
# _update_settings_from_typed → our override handles the rest.
if isinstance(frame, LLMUpdateSettingsFrame) and frame.update is None:
self._settings.session_properties = events.SessionProperties(**frame.settings)
await self._update_settings()
await self.push_frame(frame, direction)
return
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if isinstance(frame, TranscriptionFrame): if isinstance(frame, TranscriptionFrame):
@@ -424,9 +446,6 @@ class OpenAIRealtimeLLMService(LLMService):
await self._handle_bot_stopped_speaking() await self._handle_bot_stopped_speaking()
elif isinstance(frame, LLMMessagesAppendFrame): elif isinstance(frame, LLMMessagesAppendFrame):
await self._handle_messages_append(frame) await self._handle_messages_append(frame)
elif isinstance(frame, LLMUpdateSettingsFrame):
self._session_properties = events.SessionProperties(**frame.settings)
await self._update_settings()
elif isinstance(frame, LLMSetToolsFrame): elif isinstance(frame, LLMSetToolsFrame):
await self._update_settings() await self._update_settings()
@@ -513,8 +532,15 @@ class OpenAIRealtimeLLMService(LLMService):
# treat a send-side error as fatal. # treat a send-side error as fatal.
await self.push_error(error_msg=f"Error sending client event: {e}", exception=e) await self.push_error(error_msg=f"Error sending client event: {e}", exception=e)
async def _update_settings_from_typed(self, update):
"""Apply a typed settings update, sending a session update if needed."""
changed = await super()._update_settings_from_typed(update)
if "session_properties" in changed:
await self._update_settings()
return changed
async def _update_settings(self): async def _update_settings(self):
settings = self._session_properties settings = self._settings.session_properties
adapter: OpenAIRealtimeLLMAdapter = self.get_llm_adapter() adapter: OpenAIRealtimeLLMAdapter = self.get_llm_adapter()
if self._context: if self._context:

51
src/pipecat/services/openai/stt.py
View File

@@ -16,6 +16,7 @@ Provides two STT services:
import base64 import base64
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Literal, Optional, Union from typing import AsyncGenerator, Literal, Optional, Union
from loguru import logger from loguru import logger
@@ -34,6 +35,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import OPENAI_REALTIME_TTFS_P99, OPENAI_TTFS_P99 from pipecat.services.stt_latency import OPENAI_REALTIME_TTFS_P99, OPENAI_TTFS_P99
from pipecat.services.stt_service import WebsocketSTTService from pipecat.services.stt_service import WebsocketSTTService
from pipecat.services.whisper.base_stt import BaseWhisperSTTService, Transcription from pipecat.services.whisper.base_stt import BaseWhisperSTTService, Transcription
@@ -123,6 +125,17 @@ class OpenAISTTService(BaseWhisperSTTService):
_OPENAI_SAMPLE_RATE = 24000 _OPENAI_SAMPLE_RATE = 24000
@dataclass
class OpenAIRealtimeSTTSettings(STTSettings):
"""Typed settings for the OpenAI Realtime STT service.
Parameters:
prompt: Optional prompt text to guide transcription style.
"""
prompt: Optional[str] = field(default_factory=lambda: NOT_GIVEN)
class OpenAIRealtimeSTTService(WebsocketSTTService): class OpenAIRealtimeSTTService(WebsocketSTTService):
"""OpenAI Realtime Speech-to-Text service using WebSocket transcription sessions. """OpenAI Realtime Speech-to-Text service using WebSocket transcription sessions.
@@ -213,12 +226,17 @@ class OpenAIRealtimeSTTService(WebsocketSTTService):
self._base_url = base_url self._base_url = base_url
self.set_model_name(model) self.set_model_name(model)
self._language_code = self._language_to_code(language) if language else None
self._prompt = prompt self._prompt = prompt
self._turn_detection = turn_detection self._turn_detection = turn_detection
self._noise_reduction = noise_reduction self._noise_reduction = noise_reduction
self._should_interrupt = should_interrupt self._should_interrupt = should_interrupt
self._settings: OpenAIRealtimeSTTSettings = OpenAIRealtimeSTTSettings(
model=model,
language=language,
prompt=prompt,
)
self._receive_task = None self._receive_task = None
self._session_ready = False self._session_ready = False
self._resampler = create_stream_resampler() self._resampler = create_stream_resampler()
@@ -248,19 +266,31 @@ class OpenAIRealtimeSTTService(WebsocketSTTService):
""" """
return True return True
async def set_language(self, language: Language): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the language for speech recognition. """Apply a typed settings update and send session update if needed.
If the session is already active, sends an updated configuration Keeps ``_language_code`` and ``_prompt`` in sync with typed settings
to the server. and sends a ``session.update`` to the server when the session is active.
Args: Args:
language: The language to use for speech recognition. update: A :class:`STTSettings` (or ``OpenAIRealtimeSTTSettings``) delta.
Returns:
Set of field names whose values actually changed.
""" """
self._language_code = self._language_to_code(language) changed = await super()._update_settings_from_typed(update)
if not changed:
return changed
if "prompt" in changed and isinstance(self._settings, OpenAIRealtimeSTTSettings):
self._prompt = self._settings.prompt
if self._session_ready: if self._session_ready:
await self._send_session_update() await self._send_session_update()
return changed
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the service and establish WebSocket connection. """Start the service and establish WebSocket connection.
@@ -407,8 +437,11 @@ class OpenAIRealtimeSTTService(WebsocketSTTService):
"""Send ``session.update`` to configure the transcription session.""" """Send ``session.update`` to configure the transcription session."""
transcription: dict = {"model": self.model_name} transcription: dict = {"model": self.model_name}
if self._language_code: language_code = (
transcription["language"] = self._language_code self._language_to_code(self._settings.language) if self._settings.language else None
)
if language_code:
transcription["language"] = language_code
if self._prompt: if self._prompt:
transcription["prompt"] = self._prompt transcription["prompt"] = self._prompt

44
src/pipecat/services/openai/tts.py
View File

@@ -10,6 +10,7 @@ This module provides integration with OpenAI's text-to-speech API for
generating high-quality synthetic speech from text input. generating high-quality synthetic speech from text input.
""" """
from dataclasses import dataclass, field
from typing import AsyncGenerator, Dict, Literal, Optional from typing import AsyncGenerator, Dict, Literal, Optional
from loguru import logger from loguru import logger
@@ -24,6 +25,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -60,6 +62,19 @@ VALID_VOICES: Dict[str, ValidVoice] = {
} }
@dataclass
class OpenAITTSSettings(TTSSettings):
"""Typed settings for OpenAI TTS service.
Parameters:
instructions: Instructions to guide voice synthesis behavior.
speed: Voice speed control (0.25 to 4.0, default 1.0).
"""
instructions: str = field(default_factory=lambda: NOT_GIVEN)
speed: float = field(default_factory=lambda: NOT_GIVEN)
class OpenAITTSService(TTSService): class OpenAITTSService(TTSService):
"""OpenAI Text-to-Speech service that generates audio from text. """OpenAI Text-to-Speech service that generates audio from text.
@@ -118,7 +133,7 @@ class OpenAITTSService(TTSService):
super().__init__(sample_rate=sample_rate, **kwargs) super().__init__(sample_rate=sample_rate, **kwargs)
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice) self._voice_id = voice
self._client = AsyncOpenAI(api_key=api_key, base_url=base_url) self._client = AsyncOpenAI(api_key=api_key, base_url=base_url)
if instructions or speed: if instructions or speed:
@@ -132,10 +147,12 @@ class OpenAITTSService(TTSService):
stacklevel=2, stacklevel=2,
) )
self._settings = { self._settings: OpenAITTSSettings = OpenAITTSSettings(
"instructions": params.instructions if params else instructions, model=model,
"speed": params.speed if params else speed, voice=voice,
} instructions=params.instructions if params else instructions,
speed=params.speed if params else speed,
)
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -145,15 +162,6 @@ class OpenAITTSService(TTSService):
""" """
return True return True
async def set_model(self, model: str):
"""Set the TTS model to use.
Args:
model: The model name to use for text-to-speech synthesis.
"""
logger.info(f"Switching TTS model to: [{model}]")
self.set_model_name(model)
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the OpenAI TTS service. """Start the OpenAI TTS service.
@@ -190,11 +198,11 @@ class OpenAITTSService(TTSService):
"response_format": "pcm", "response_format": "pcm",
} }
if self._settings["instructions"]: if self._settings.instructions:
create_params["instructions"] = self._settings["instructions"] create_params["instructions"] = self._settings.instructions
if self._settings["speed"]: if self._settings.speed:
create_params["speed"] = self._settings["speed"] create_params["speed"] = self._settings.speed
async with self._client.audio.speech.with_streaming_response.create( async with self._client.audio.speech.with_streaming_response.create(
**create_params **create_params

51
src/pipecat/services/openai_realtime_beta/openai.py
View File

@@ -10,8 +10,8 @@ import base64
import json import json
import time import time
import warnings import warnings
from dataclasses import dataclass from dataclasses import dataclass, field
from typing import Optional from typing import Any, Optional
from loguru import logger from loguru import logger
@@ -54,6 +54,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
from pipecat.services.openai.llm import OpenAIContextAggregatorPair from pipecat.services.openai.llm import OpenAIContextAggregatorPair
from pipecat.services.settings import NOT_GIVEN, LLMSettings
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
from pipecat.utils.time import time_now_iso8601 from pipecat.utils.time import time_now_iso8601
from pipecat.utils.tracing.service_decorators import traced_openai_realtime, traced_stt from pipecat.utils.tracing.service_decorators import traced_openai_realtime, traced_stt
@@ -91,6 +92,17 @@ class CurrentAudioResponse:
total_size: int = 0 total_size: int = 0
@dataclass
class OpenAIRealtimeBetaLLMSettings(LLMSettings):
"""Typed settings for OpenAI Realtime Beta LLM services.
Parameters:
session_properties: OpenAI Realtime session configuration.
"""
session_properties: Any = field(default_factory=lambda: NOT_GIVEN)
class OpenAIRealtimeBetaLLMService(LLMService): class OpenAIRealtimeBetaLLMService(LLMService):
"""OpenAI Realtime Beta LLM service providing real-time audio and text communication. """OpenAI Realtime Beta LLM service providing real-time audio and text communication.
@@ -146,8 +158,9 @@ class OpenAIRealtimeBetaLLMService(LLMService):
self.base_url = full_url self.base_url = full_url
self.set_model_name(model) self.set_model_name(model)
self._session_properties: events.SessionProperties = ( self._settings = OpenAIRealtimeBetaLLMSettings(
session_properties or events.SessionProperties() model=model,
session_properties=session_properties or events.SessionProperties(),
) )
self._audio_input_paused = start_audio_paused self._audio_input_paused = start_audio_paused
self._send_transcription_frames = send_transcription_frames self._send_transcription_frames = send_transcription_frames
@@ -187,12 +200,12 @@ class OpenAIRealtimeBetaLLMService(LLMService):
def _is_modality_enabled(self, modality: str) -> bool: def _is_modality_enabled(self, modality: str) -> bool:
"""Check if a specific modality is enabled, "text" or "audio".""" """Check if a specific modality is enabled, "text" or "audio"."""
modalities = self._session_properties.modalities or ["audio", "text"] modalities = self._settings.session_properties.modalities or ["audio", "text"]
return modality in modalities return modality in modalities
def _get_enabled_modalities(self) -> list[str]: def _get_enabled_modalities(self) -> list[str]:
"""Get the list of enabled modalities.""" """Get the list of enabled modalities."""
return self._session_properties.modalities or ["audio", "text"] return self._settings.session_properties.modalities or ["audio", "text"]
async def retrieve_conversation_item(self, item_id: str): async def retrieve_conversation_item(self, item_id: str):
"""Retrieve a conversation item by ID from the server. """Retrieve a conversation item by ID from the server.
@@ -259,7 +272,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
async def _handle_interruption(self): async def _handle_interruption(self):
# None and False are different. Check for False. None means we're using OpenAI's # None and False are different. Check for False. None means we're using OpenAI's
# built-in turn detection defaults. # built-in turn detection defaults.
if self._session_properties.turn_detection is False: if self._settings.session_properties.turn_detection is False:
await self.send_client_event(events.InputAudioBufferClearEvent()) await self.send_client_event(events.InputAudioBufferClearEvent())
await self.send_client_event(events.ResponseCancelEvent()) await self.send_client_event(events.ResponseCancelEvent())
await self._truncate_current_audio_response() await self._truncate_current_audio_response()
@@ -276,7 +289,7 @@ class OpenAIRealtimeBetaLLMService(LLMService):
async def _handle_user_stopped_speaking(self, frame): async def _handle_user_stopped_speaking(self, frame):
# None and False are different. Check for False. None means we're using OpenAI's # None and False are different. Check for False. None means we're using OpenAI's
# built-in turn detection defaults. # built-in turn detection defaults.
if self._session_properties.turn_detection is False: if self._settings.session_properties.turn_detection is False:
await self.send_client_event(events.InputAudioBufferCommitEvent()) await self.send_client_event(events.InputAudioBufferCommitEvent())
await self.send_client_event(events.ResponseCreateEvent()) await self.send_client_event(events.ResponseCreateEvent())
@@ -342,6 +355,16 @@ class OpenAIRealtimeBetaLLMService(LLMService):
frame: The frame to process. frame: The frame to process.
direction: The direction of frame flow in the pipeline. direction: The direction of frame flow in the pipeline.
""" """
# Legacy dict path: frame.settings contains SessionProperties fields,
# not our Settings fields, so we construct SessionProperties directly.
# The new typed path (frame.update) falls through to super, which calls
# _update_settings_from_typed → our override handles the rest.
if isinstance(frame, LLMUpdateSettingsFrame) and frame.update is None:
self._settings.session_properties = events.SessionProperties(**frame.settings)
await self._update_settings()
await self.push_frame(frame, direction)
return
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
if isinstance(frame, TranscriptionFrame): if isinstance(frame, TranscriptionFrame):
@@ -377,9 +400,6 @@ class OpenAIRealtimeBetaLLMService(LLMService):
await self._handle_messages_append(frame) await self._handle_messages_append(frame)
elif isinstance(frame, RealtimeMessagesUpdateFrame): elif isinstance(frame, RealtimeMessagesUpdateFrame):
self._context = frame.context self._context = frame.context
elif isinstance(frame, LLMUpdateSettingsFrame):
self._session_properties = events.SessionProperties(**frame.settings)
await self._update_settings()
elif isinstance(frame, LLMSetToolsFrame): elif isinstance(frame, LLMSetToolsFrame):
await self._update_settings() await self._update_settings()
elif isinstance(frame, RealtimeFunctionCallResultFrame): elif isinstance(frame, RealtimeFunctionCallResultFrame):
@@ -456,8 +476,15 @@ class OpenAIRealtimeBetaLLMService(LLMService):
# treat a send-side error as fatal. # treat a send-side error as fatal.
await self.push_error(error_msg=f"Error sending client event: {e}", exception=e) await self.push_error(error_msg=f"Error sending client event: {e}", exception=e)
async def _update_settings_from_typed(self, update):
"""Apply a typed settings update, sending a session update if needed."""
changed = await super()._update_settings_from_typed(update)
if "session_properties" in changed:
await self._update_settings()
return changed
async def _update_settings(self): async def _update_settings(self):
settings = self._session_properties settings = self._settings.session_properties
# tools given in the context override the tools in the session properties # tools given in the context override the tools in the session properties
if self._context and self._context.tools: if self._context and self._context.tools:
settings.tools = self._context.tools settings.tools = self._context.tools

20
src/pipecat/services/perplexity/llm.py
View File

@@ -72,16 +72,16 @@ class PerplexityLLMService(OpenAILLMService):
} }
# Add OpenAI-compatible parameters if they're set # Add OpenAI-compatible parameters if they're set
if self._settings["frequency_penalty"] is not NOT_GIVEN: if self._settings.frequency_penalty is not NOT_GIVEN:
params["frequency_penalty"] = self._settings["frequency_penalty"] params["frequency_penalty"] = self._settings.frequency_penalty
if self._settings["presence_penalty"] is not NOT_GIVEN: if self._settings.presence_penalty is not NOT_GIVEN:
params["presence_penalty"] = self._settings["presence_penalty"] params["presence_penalty"] = self._settings.presence_penalty
if self._settings["temperature"] is not NOT_GIVEN: if self._settings.temperature is not NOT_GIVEN:
params["temperature"] = self._settings["temperature"] params["temperature"] = self._settings.temperature
if self._settings["top_p"] is not NOT_GIVEN: if self._settings.top_p is not NOT_GIVEN:
params["top_p"] = self._settings["top_p"] params["top_p"] = self._settings.top_p
if self._settings["max_tokens"] is not NOT_GIVEN: if self._settings.max_tokens is not NOT_GIVEN:
params["max_tokens"] = self._settings["max_tokens"] params["max_tokens"] = self._settings.max_tokens
return params return params

88
src/pipecat/services/playht/tts.py
View File

@@ -14,6 +14,7 @@ import io
import json import json
import struct import struct
import warnings import warnings
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
import aiohttp import aiohttp
@@ -32,6 +33,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
from pipecat.services.tts_service import InterruptibleTTSService, TTSService from pipecat.services.tts_service import InterruptibleTTSService, TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -97,6 +99,25 @@ def language_to_playht_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class PlayHTTTSSettings(TTSSettings):
"""Typed settings for PlayHT TTS services.
Parameters:
output_format: Audio output format.
voice_engine: Voice engine to use.
speed: Speech speed multiplier. Defaults to 1.0.
seed: Random seed for voice consistency.
playht_sample_rate: Audio sample rate sent to the API.
"""
output_format: str = field(default_factory=lambda: NOT_GIVEN)
voice_engine: str = field(default_factory=lambda: NOT_GIVEN)
speed: float = field(default_factory=lambda: NOT_GIVEN)
seed: int = field(default_factory=lambda: NOT_GIVEN)
playht_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
class PlayHTTTSService(InterruptibleTTSService): class PlayHTTTSService(InterruptibleTTSService):
"""PlayHT WebSocket-based text-to-speech service. """PlayHT WebSocket-based text-to-speech service.
@@ -170,17 +191,19 @@ class PlayHTTTSService(InterruptibleTTSService):
self._receive_task = None self._receive_task = None
self._context_id = None self._context_id = None
self._settings = { self._settings: PlayHTTTSSettings = PlayHTTTSSettings(
"language": self.language_to_service_language(params.language) model=voice_engine,
voice=voice_url,
language=self.language_to_service_language(params.language)
if params.language if params.language
else "english", else "english",
"output_format": output_format, output_format=output_format,
"voice_engine": voice_engine, voice_engine=voice_engine,
"speed": params.speed, speed=params.speed,
"seed": params.seed, seed=params.seed,
} )
self.set_model_name(voice_engine) self.set_model_name(voice_engine)
self.set_voice(voice_url) self._voice_id = voice_url
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -304,13 +327,13 @@ class PlayHTTTSService(InterruptibleTTSService):
# Handle the new response format with multiple URLs # Handle the new response format with multiple URLs
if "websocket_urls" in data: if "websocket_urls" in data:
# Select URL based on voice_engine # Select URL based on voice_engine
if self._settings["voice_engine"] in data["websocket_urls"]: if self._settings.voice_engine in data["websocket_urls"]:
self._websocket_url = data["websocket_urls"][ self._websocket_url = data["websocket_urls"][
self._settings["voice_engine"] self._settings.voice_engine
] ]
else: else:
raise ValueError( raise ValueError(
f"Unsupported voice engine: {self._settings['voice_engine']}" f"Unsupported voice engine: {self._settings.voice_engine}"
) )
else: else:
raise ValueError("Invalid response: missing websocket_urls") raise ValueError("Invalid response: missing websocket_urls")
@@ -382,12 +405,12 @@ class PlayHTTTSService(InterruptibleTTSService):
tts_command = { tts_command = {
"text": text, "text": text,
"voice": self._voice_id, "voice": self._voice_id,
"voice_engine": self._settings["voice_engine"], "voice_engine": self._settings.voice_engine,
"output_format": self._settings["output_format"], "output_format": self._settings.output_format,
"sample_rate": self.sample_rate, "sample_rate": self.sample_rate,
"language": self._settings["language"], "language": self._settings.language,
"speed": self._settings["speed"], "speed": self._settings.speed,
"seed": self._settings["seed"], "seed": self._settings.seed,
"request_id": self._context_id, "request_id": self._context_id,
} }
@@ -499,17 +522,18 @@ class PlayHTHttpTTSService(TTSService):
# Extract the base engine name # Extract the base engine name
voice_engine = voice_engine.replace("-ws", "") voice_engine = voice_engine.replace("-ws", "")
self._settings = { self._settings: PlayHTTTSSettings = PlayHTTTSSettings(
"language": self.language_to_service_language(params.language) voice=voice_url,
language=self.language_to_service_language(params.language)
if params.language if params.language
else "english", else "english",
"output_format": output_format, output_format=output_format,
"voice_engine": voice_engine, voice_engine=voice_engine,
"speed": params.speed, speed=params.speed,
"seed": params.seed, seed=params.seed,
} )
self.set_model_name(voice_engine) self.set_model_name(voice_engine)
self.set_voice(voice_url) self._voice_id = voice_url
async def start(self, frame: StartFrame): async def start(self, frame: StartFrame):
"""Start the PlayHT HTTP TTS service. """Start the PlayHT HTTP TTS service.
@@ -518,7 +542,7 @@ class PlayHTHttpTTSService(TTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["sample_rate"] = self.sample_rate self._settings.playht_sample_rate = self.sample_rate
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -559,17 +583,17 @@ class PlayHTHttpTTSService(TTSService):
payload = { payload = {
"text": text, "text": text,
"voice": self._voice_id, "voice": self._voice_id,
"voice_engine": self._settings["voice_engine"], "voice_engine": self._settings.voice_engine,
"output_format": self._settings["output_format"], "output_format": self._settings.output_format,
"sample_rate": self.sample_rate, "sample_rate": self.sample_rate,
"language": self._settings["language"], "language": self._settings.language,
} }
# Add optional parameters if they exist # Add optional parameters if they exist
if self._settings["speed"] is not None: if self._settings.speed is not None:
payload["speed"] = self._settings["speed"] payload["speed"] = self._settings.speed
if self._settings["seed"] is not None: if self._settings.seed is not None:
payload["seed"] = self._settings["seed"] payload["seed"] = self._settings.seed
headers = { headers = {
"Authorization": f"Bearer {self._api_key}", "Authorization": f"Bearer {self._api_key}",

38
src/pipecat/services/resembleai/tts.py
View File

@@ -8,6 +8,7 @@
import base64 import base64
import json import json
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -24,6 +25,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import AudioContextWordTTSService from pipecat.services.tts_service import AudioContextWordTTSService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
from pipecat.utils.text.base_text_aggregator import BaseTextAggregator from pipecat.utils.text.base_text_aggregator import BaseTextAggregator
@@ -38,6 +40,21 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class ResembleAITTSSettings(TTSSettings):
"""Typed settings for Resemble AI TTS service.
Parameters:
precision: PCM bit depth (PCM_32, PCM_24, PCM_16, or MULAW).
output_format: Audio format (wav or mp3).
resemble_sample_rate: Audio sample rate sent to the API.
"""
precision: str = field(default_factory=lambda: NOT_GIVEN)
output_format: str = field(default_factory=lambda: NOT_GIVEN)
resemble_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
class ResembleAITTSService(AudioContextWordTTSService): class ResembleAITTSService(AudioContextWordTTSService):
"""Resemble AI TTS service with WebSocket streaming and word timestamps. """Resemble AI TTS service with WebSocket streaming and word timestamps.
@@ -76,11 +93,12 @@ class ResembleAITTSService(AudioContextWordTTSService):
self._api_key = api_key self._api_key = api_key
self._voice_id = voice_id self._voice_id = voice_id
self._url = url self._url = url
self._settings = { self._settings: ResembleAITTSSettings = ResembleAITTSSettings(
"precision": precision, voice=voice_id,
"output_format": output_format, precision=precision,
"sample_rate": sample_rate, output_format=output_format,
} resemble_sample_rate=sample_rate,
)
self._websocket = None self._websocket = None
self._request_id_counter = 0 self._request_id_counter = 0
@@ -101,7 +119,7 @@ class ResembleAITTSService(AudioContextWordTTSService):
self._jitter_buffer_bytes = 44100 # ~1000ms at 22050Hz to handle 400ms+ network gaps self._jitter_buffer_bytes = 44100 # ~1000ms at 22050Hz to handle 400ms+ network gaps
self._playback_started: dict[str, bool] = {} # Track if we've started playback per request self._playback_started: dict[str, bool] = {} # Track if we've started playback per request
self.set_voice(voice_id) self._voice_id = voice_id
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -125,9 +143,9 @@ class ResembleAITTSService(AudioContextWordTTSService):
"data": text, "data": text,
"binary_response": False, # Use JSON frames to get timestamps "binary_response": False, # Use JSON frames to get timestamps
"request_id": self._request_id_counter, # ResembleAI only accepts number "request_id": self._request_id_counter, # ResembleAI only accepts number
"output_format": self._settings["output_format"], "output_format": self._settings.output_format,
"sample_rate": self._settings["sample_rate"], "sample_rate": self._settings.resemble_sample_rate,
"precision": self._settings["precision"], "precision": self._settings.precision,
"no_audio_header": True, "no_audio_header": True,
} }
@@ -141,7 +159,7 @@ class ResembleAITTSService(AudioContextWordTTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["sample_rate"] = self.sample_rate self._settings.resemble_sample_rate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):

279
src/pipecat/services/rime/tts.py
View File

@@ -12,7 +12,8 @@ using Rime's API for streaming and batch audio synthesis.
import base64 import base64
import json import json
from typing import Any, AsyncGenerator, Mapping, Optional from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional
import aiohttp import aiohttp
from loguru import logger from loguru import logger
@@ -30,6 +31,7 @@ from pipecat.frames.frames import (
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
from pipecat.services.tts_service import ( from pipecat.services.tts_service import (
AudioContextWordTTSService, AudioContextWordTTSService,
InterruptibleTTSService, InterruptibleTTSService,
@@ -68,6 +70,62 @@ def language_to_rime_language(language: Language) -> str:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class RimeTTSSettings(TTSSettings):
"""Typed settings for Rime WS JSON and HTTP TTS services.
Parameters:
speaker: Voice speaker ID.
modelId: Rime model identifier.
audioFormat: Audio output format.
samplingRate: Audio sample rate.
lang: Rime language code.
speedAlpha: Speech speed multiplier. Defaults to 1.0.
reduceLatency: Whether to reduce latency at potential quality cost.
pauseBetweenBrackets: Whether to add pauses between bracketed content.
phonemizeBetweenBrackets: Whether to phonemize bracketed content.
inlineSpeedAlpha: Inline speed control markup.
"""
speaker: str = field(default_factory=lambda: NOT_GIVEN)
modelId: str = field(default_factory=lambda: NOT_GIVEN)
audioFormat: str = field(default_factory=lambda: NOT_GIVEN)
samplingRate: int = field(default_factory=lambda: NOT_GIVEN)
lang: str = field(default_factory=lambda: NOT_GIVEN)
speedAlpha: float = field(default_factory=lambda: NOT_GIVEN)
reduceLatency: bool = field(default_factory=lambda: NOT_GIVEN)
pauseBetweenBrackets: bool = field(default_factory=lambda: NOT_GIVEN)
phonemizeBetweenBrackets: bool = field(default_factory=lambda: NOT_GIVEN)
inlineSpeedAlpha: str = field(default_factory=lambda: NOT_GIVEN)
@dataclass
class RimeNonJsonTTSSettings(TTSSettings):
"""Typed settings for Rime non-JSON WS TTS service.
Parameters:
speaker: Voice speaker ID.
modelId: Rime model identifier.
audioFormat: Audio output format.
samplingRate: Audio sample rate.
lang: Rime language code.
segment: Text segmentation mode ("immediate", "bySentence", "never").
repetition_penalty: Token repetition penalty (1.0-2.0).
temperature: Sampling temperature (0.0-1.0).
top_p: Cumulative probability threshold (0.0-1.0).
"""
speaker: str = field(default_factory=lambda: NOT_GIVEN)
modelId: str = field(default_factory=lambda: NOT_GIVEN)
audioFormat: str = field(default_factory=lambda: NOT_GIVEN)
samplingRate: int = field(default_factory=lambda: NOT_GIVEN)
lang: str = field(default_factory=lambda: NOT_GIVEN)
segment: str = field(default_factory=lambda: NOT_GIVEN)
repetition_penalty: float = field(default_factory=lambda: NOT_GIVEN)
temperature: float = field(default_factory=lambda: NOT_GIVEN)
top_p: float = field(default_factory=lambda: NOT_GIVEN)
class RimeTTSService(AudioContextWordTTSService): class RimeTTSService(AudioContextWordTTSService):
"""Text-to-Speech service using Rime's websocket API. """Text-to-Speech service using Rime's websocket API.
@@ -149,19 +207,17 @@ class RimeTTSService(AudioContextWordTTSService):
self._url = url self._url = url
self._voice_id = voice_id self._voice_id = voice_id
self._model = model self._model = model
self._settings = { self._settings: RimeTTSSettings = RimeTTSSettings(
"speaker": voice_id, speaker=voice_id,
"modelId": model, modelId=model,
"audioFormat": "pcm", audioFormat="pcm",
"samplingRate": 0, samplingRate=0,
"lang": self.language_to_service_language(params.language) lang=self.language_to_service_language(params.language) if params.language else "eng",
if params.language speedAlpha=params.speed_alpha,
else "eng", reduceLatency=params.reduce_latency,
"speedAlpha": params.speed_alpha, pauseBetweenBrackets=json.dumps(params.pause_between_brackets),
"reduceLatency": params.reduce_latency, phonemizeBetweenBrackets=json.dumps(params.phonemize_between_brackets),
"pauseBetweenBrackets": json.dumps(params.pause_between_brackets), )
"phonemizeBetweenBrackets": json.dumps(params.phonemize_between_brackets),
}
# State tracking # State tracking
self._context_id = None # Tracks current turn self._context_id = None # Tracks current turn
@@ -188,15 +244,6 @@ class RimeTTSService(AudioContextWordTTSService):
""" """
return language_to_rime_language(language) return language_to_rime_language(language)
async def set_model(self, model: str):
"""Update the TTS model.
Args:
model: The model name to use for synthesis.
"""
self._model = model
await super().set_model(model)
# A set of Rime-specific helpers for text transformations # A set of Rime-specific helpers for text transformations
def SPELL(text: str) -> str: def SPELL(text: str) -> str:
"""Wrap text in Rime spell function.""" """Wrap text in Rime spell function."""
@@ -222,15 +269,15 @@ class RimeTTSService(AudioContextWordTTSService):
self._extra_msg_fields["inlineSpeedAlpha"] = ",".join(speed_vals + [str(speed)]) self._extra_msg_fields["inlineSpeedAlpha"] = ",".join(speed_vals + [str(speed)])
return f"[{text}]" return f"[{text}]"
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Update service settings and reconnect if voice changed.""" """Apply a typed settings update and reconnect if voice changed."""
prev_voice = self._voice_id prev_voice = self._voice_id
await super()._update_settings(settings) changed = await super()._update_settings_from_typed(update)
if not prev_voice == self._voice_id: if "voice" in changed:
self._settings["speaker"] = self._voice_id self._settings.speaker = self._voice_id
logger.info(f"Switching TTS voice to: [{self._voice_id}]")
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed
def _build_msg(self, text: str = "") -> dict: def _build_msg(self, text: str = "") -> dict:
"""Build JSON message for Rime API.""" """Build JSON message for Rime API."""
@@ -255,7 +302,7 @@ class RimeTTSService(AudioContextWordTTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["samplingRate"] = self.sample_rate self._settings.samplingRate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -301,7 +348,20 @@ class RimeTTSService(AudioContextWordTTSService):
if self._websocket and self._websocket.state is State.OPEN: if self._websocket and self._websocket.state is State.OPEN:
return return
params = "&".join(f"{k}={v}" for k, v in self._settings.items()) params = "&".join(
f"{k}={v}"
for k, v in {
"speaker": self._settings.speaker,
"modelId": self._settings.modelId,
"audioFormat": self._settings.audioFormat,
"samplingRate": self._settings.samplingRate,
"lang": self._settings.lang,
"speedAlpha": self._settings.speedAlpha,
"reduceLatency": self._settings.reduceLatency,
"pauseBetweenBrackets": self._settings.pauseBetweenBrackets,
"phonemizeBetweenBrackets": self._settings.phonemizeBetweenBrackets,
}.items()
)
url = f"{self._url}?{params}" url = f"{self._url}?{params}"
headers = {"Authorization": f"Bearer {self._api_key}"} headers = {"Authorization": f"Bearer {self._api_key}"}
self._websocket = await websocket_connect(url, additional_headers=headers) self._websocket = await websocket_connect(url, additional_headers=headers)
@@ -525,21 +585,17 @@ class RimeHttpTTSService(TTSService):
self._api_key = api_key self._api_key = api_key
self._session = aiohttp_session self._session = aiohttp_session
self._base_url = "https://users.rime.ai/v1/rime-tts" self._base_url = "https://users.rime.ai/v1/rime-tts"
self._settings = { self._settings: RimeTTSSettings = RimeTTSSettings(
"lang": self.language_to_service_language(params.language) lang=self.language_to_service_language(params.language) if params.language else "eng",
if params.language speedAlpha=params.speed_alpha,
else "eng", reduceLatency=params.reduce_latency,
"speedAlpha": params.speed_alpha, pauseBetweenBrackets=params.pause_between_brackets,
"reduceLatency": params.reduce_latency, phonemizeBetweenBrackets=params.phonemize_between_brackets,
"pauseBetweenBrackets": params.pause_between_brackets, inlineSpeedAlpha=params.inline_speed_alpha if params.inline_speed_alpha else NOT_GIVEN,
"phonemizeBetweenBrackets": params.phonemize_between_brackets, )
} self._voice_id = voice_id
self.set_voice(voice_id)
self.set_model_name(model) self.set_model_name(model)
if params.inline_speed_alpha:
self._settings["inlineSpeedAlpha"] = params.inline_speed_alpha
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -578,7 +634,15 @@ class RimeHttpTTSService(TTSService):
"Content-Type": "application/json", "Content-Type": "application/json",
} }
payload = self._settings.copy() payload = {
"lang": self._settings.lang,
"speedAlpha": self._settings.speedAlpha,
"reduceLatency": self._settings.reduceLatency,
"pauseBetweenBrackets": self._settings.pauseBetweenBrackets,
"phonemizeBetweenBrackets": self._settings.phonemizeBetweenBrackets,
}
if is_given(self._settings.inlineSpeedAlpha):
payload["inlineSpeedAlpha"] = self._settings.inlineSpeedAlpha
payload["text"] = text payload["text"] = text
payload["speaker"] = self._voice_id payload["speaker"] = self._voice_id
payload["modelId"] = self._model_name payload["modelId"] = self._model_name
@@ -699,26 +763,24 @@ class RimeNonJsonTTSService(InterruptibleTTSService):
self._url = url self._url = url
self._voice_id = voice_id self._voice_id = voice_id
self._model = model self._model = model
self._settings = { self._settings: RimeNonJsonTTSSettings = RimeNonJsonTTSSettings(
"speaker": voice_id, speaker=voice_id,
"modelId": model, modelId=model,
"audioFormat": audio_format, audioFormat=audio_format,
"samplingRate": sample_rate, samplingRate=sample_rate,
} lang=self.language_to_service_language(params.language)
if params.language
if params.language: else NOT_GIVEN,
self._settings["lang"] = self.language_to_service_language(params.language) segment=params.segment if params.segment is not None else NOT_GIVEN,
if params.segment is not None: repetition_penalty=params.repetition_penalty
self._settings["segment"] = params.segment if params.repetition_penalty is not None
if params.repetition_penalty is not None: else NOT_GIVEN,
self._settings["repetition_penalty"] = params.repetition_penalty temperature=params.temperature if params.temperature is not None else NOT_GIVEN,
if params.temperature is not None: top_p=params.top_p if params.top_p is not None else NOT_GIVEN,
self._settings["temperature"] = params.temperature )
if params.top_p is not None:
self._settings["top_p"] = params.top_p
# Add any extra parameters for future compatibility # Add any extra parameters for future compatibility
if params.extra: if params.extra:
self._settings.update(params.extra) self._settings.extra.update(params.extra)
self._receive_task = None self._receive_task = None
self._context_id: Optional[str] = None self._context_id: Optional[str] = None
@@ -750,7 +812,7 @@ class RimeNonJsonTTSService(InterruptibleTTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["samplingRate"] = self.sample_rate self._settings.samplingRate = self.sample_rate
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -794,8 +856,26 @@ class RimeNonJsonTTSService(InterruptibleTTSService):
try: try:
if self._websocket and self._websocket.state is State.OPEN: if self._websocket and self._websocket.state is State.OPEN:
return return
# Build URL with query parameters (only non-None values) # Build URL with query parameters (only given, non-None values)
params = "&".join(f"{k}={v}" for k, v in self._settings.items() if v is not None) settings_dict = {
"speaker": self._settings.speaker,
"modelId": self._settings.modelId,
"audioFormat": self._settings.audioFormat,
"samplingRate": self._settings.samplingRate,
}
if is_given(self._settings.lang):
settings_dict["lang"] = self._settings.lang
if is_given(self._settings.segment):
settings_dict["segment"] = self._settings.segment
if is_given(self._settings.repetition_penalty):
settings_dict["repetition_penalty"] = self._settings.repetition_penalty
if is_given(self._settings.temperature):
settings_dict["temperature"] = self._settings.temperature
if is_given(self._settings.top_p):
settings_dict["top_p"] = self._settings.top_p
# Include extras
settings_dict.update(self._settings.extra)
params = "&".join(f"{k}={v}" for k, v in settings_dict.items() if v is not None)
url = f"{self._url}?{params}" url = f"{self._url}?{params}"
headers = {"Authorization": f"Bearer {self._api_key}"} headers = {"Authorization": f"Bearer {self._api_key}"}
self._websocket = await websocket_connect( self._websocket = await websocket_connect(
@@ -889,68 +969,23 @@ class RimeNonJsonTTSService(InterruptibleTTSService):
except Exception as e: except Exception as e:
yield ErrorFrame(error=f"Unknown error occurred: {e}") yield ErrorFrame(error=f"Unknown error occurred: {e}")
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Update service settings and reconnect if necessary. """Apply a typed settings update and reconnect if necessary.
Since all settings are WebSocket URL query parameters, Since all settings are WebSocket URL query parameters,
any setting change requires reconnecting to apply the new values. any setting change requires reconnecting to apply the new values.
""" """
needs_reconnect = False changed = await super()._update_settings_from_typed(update)
# Track previous values from self._settings only # Sync voice and model to settings dict fields
prev_settings = self._settings.copy() if "voice" in changed:
self._settings.speaker = self._voice_id
if "model" in changed:
self._settings.modelId = self._model_name
# Let parent class handle standard settings (voice, model, language) if changed:
await super()._update_settings(settings)
# Check if voice changed and update settings dict
if "voice" in settings or "voice_id" in settings:
self._settings["speaker"] = self._voice_id
if prev_settings.get("speaker") != self._voice_id:
logger.info(f"Switching TTS voice to: [{self._voice_id}]")
needs_reconnect = True
# Check if model changed and update settings dict
if "model" in settings:
self._settings["modelId"] = self._model
if prev_settings.get("modelId") != self._model:
logger.info(f"Switching TTS model to: [{self._model}]")
needs_reconnect = True
# Handle language explicitly
if "language" in settings:
new_lang = self.language_to_service_language(settings["language"])
if new_lang and new_lang != prev_settings.get("lang"):
logger.info(f"Updating language to: [{new_lang}]")
self._settings["lang"] = new_lang
needs_reconnect = True
# Check other parameters
for key in ["segment", "repetition_penalty", "temperature", "top_p"]:
if key in settings and settings[key] != prev_settings.get(key):
logger.info(f"Updating {key} to: [{settings[key]}]")
self._settings[key] = settings[key]
needs_reconnect = True
# Handle extra parameters
for key, value in settings.items():
if key not in [
"voice",
"voice_id",
"model",
"language",
"segment",
"repetition_penalty",
"temperature",
"top_p",
]:
if value != prev_settings.get(key):
logger.info(f"Updating extra parameter {key} to: [{value}]")
self._settings[key] = value
needs_reconnect = True
# Reconnect if any setting changed
if needs_reconnect:
logger.debug("Settings changed, reconnecting WebSocket with new parameters") logger.debug("Settings changed, reconnecting WebSocket with new parameters")
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed

10
src/pipecat/services/sambanova/llm.py
View File

@@ -87,16 +87,16 @@ class SambaNovaLLMService(OpenAILLMService): # type: ignore
"model": self.model_name, "model": self.model_name,
"stream": True, "stream": True,
"stream_options": {"include_usage": True}, "stream_options": {"include_usage": True},
"temperature": self._settings["temperature"], "temperature": self._settings.temperature,
"top_p": self._settings["top_p"], "top_p": self._settings.top_p,
"max_tokens": self._settings["max_tokens"], "max_tokens": self._settings.max_tokens,
"max_completion_tokens": self._settings["max_completion_tokens"], "max_completion_tokens": self._settings.max_completion_tokens,
} }
# Messages, tools, tool_choice # Messages, tools, tool_choice
params.update(params_from_context) params.update(params_from_context)
params.update(self._settings["extra"]) params.update(self._settings.extra)
return params return params
@traced_llm # type: ignore @traced_llm # type: ignore

145
src/pipecat/services/sarvam/stt.py
View File

@@ -12,7 +12,7 @@ can handle multiple audio formats for Indian language speech recognition.
""" """
import base64 import base64
from dataclasses import dataclass from dataclasses import dataclass, field
from typing import AsyncGenerator, Dict, Literal, Optional from typing import AsyncGenerator, Dict, Literal, Optional
from loguru import logger from loguru import logger
@@ -32,6 +32,7 @@ from pipecat.frames.frames import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.sarvam._sdk import sdk_headers from pipecat.services.sarvam._sdk import sdk_headers
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import SARVAM_TTFS_P99 from pipecat.services.stt_latency import SARVAM_TTFS_P99
from pipecat.services.stt_service import STTService from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -130,6 +131,23 @@ MODEL_CONFIGS: Dict[str, ModelConfig] = {
} }
@dataclass
class SarvamSTTSettings(STTSettings):
"""Typed settings for the Sarvam STT service.
Parameters:
prompt: Optional prompt to guide transcription/translation style.
mode: Mode of operation (transcribe, translate, verbatim, etc.).
vad_signals: Enable VAD signals in response.
high_vad_sensitivity: Enable high VAD sensitivity.
"""
prompt: Optional[str] = field(default_factory=lambda: NOT_GIVEN)
mode: Optional[str] = field(default_factory=lambda: NOT_GIVEN)
vad_signals: Optional[bool] = field(default_factory=lambda: NOT_GIVEN)
high_vad_sensitivity: Optional[bool] = field(default_factory=lambda: NOT_GIVEN)
class SarvamSTTService(STTService): class SarvamSTTService(STTService):
"""Sarvam speech-to-text service. """Sarvam speech-to-text service.
@@ -207,22 +225,8 @@ class SarvamSTTService(STTService):
self.set_model_name(model) self.set_model_name(model)
self._api_key = api_key self._api_key = api_key
self._language_code: Optional[Language] = params.language
# Set language string: use provided language or model's default
if params.language:
self._language_string = language_to_sarvam_language(params.language)
else:
self._language_string = self._config.default_language
self._prompt = params.prompt
# Set mode: use provided mode or model's default
self._mode = params.mode if params.mode is not None else self._config.default_mode
# Store connection parameters # Store connection parameters
self._vad_signals = params.vad_signals
self._high_vad_sensitivity = params.high_vad_sensitivity
self._input_audio_codec = input_audio_codec self._input_audio_codec = input_audio_codec
# Initialize Sarvam SDK client # Initialize Sarvam SDK client
@@ -240,7 +244,19 @@ class SarvamSTTService(STTService):
self._socket_client = None self._socket_client = None
self._receive_task = None self._receive_task = None
if self._vad_signals: # Resolve mode default from model config
mode = params.mode if params.mode is not None else self._config.default_mode
self._settings: SarvamSTTSettings = SarvamSTTSettings(
model=model,
language=params.language,
prompt=params.prompt if params.prompt is not None else NOT_GIVEN,
mode=mode if mode is not None else NOT_GIVEN,
vad_signals=params.vad_signals,
high_vad_sensitivity=params.high_vad_sensitivity,
)
if params.vad_signals:
self._register_event_handler("on_speech_started") self._register_event_handler("on_speech_started")
self._register_event_handler("on_speech_stopped") self._register_event_handler("on_speech_stopped")
self._register_event_handler("on_utterance_end") self._register_event_handler("on_utterance_end")
@@ -258,6 +274,12 @@ class SarvamSTTService(STTService):
""" """
return language_to_sarvam_language(language) return language_to_sarvam_language(language)
def _get_language_string(self) -> Optional[str]:
"""Resolve the current language setting to a Sarvam language code string."""
if self._settings.language:
return language_to_sarvam_language(self._settings.language)
return self._config.default_language
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -275,42 +297,74 @@ class SarvamSTTService(STTService):
await super().process_frame(frame, direction) await super().process_frame(frame, direction)
# Only handle VAD frames when not using Sarvam's VAD signals # Only handle VAD frames when not using Sarvam's VAD signals
if not self._vad_signals: if not self._settings.vad_signals:
if isinstance(frame, VADUserStartedSpeakingFrame): if isinstance(frame, VADUserStartedSpeakingFrame):
await self._start_metrics() await self._start_metrics()
elif isinstance(frame, VADUserStoppedSpeakingFrame): elif isinstance(frame, VADUserStoppedSpeakingFrame):
if self._socket_client: if self._socket_client:
await self._socket_client.flush() await self._socket_client.flush()
async def set_language(self, language: Language): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the recognition language and reconnect. """Apply a typed settings update, validate, sync state, and reconnect.
Args: Args:
language: The language to use for speech recognition. update: A :class:`STTSettings` (or ``SarvamSTTSettings``) delta.
Returns:
Set of field names whose values actually changed.
Raises: Raises:
ValueError: If called on a model that auto-detects language. ValueError: If a setting is not supported by the current model.
""" """
if not self._config.supports_language: # Validate against model capabilities before applying
raise ValueError( if is_given(update.language) and update.language is not None:
f"Model '{self.model_name}' does not support language parameter " if not self._config.supports_language:
"(auto-detects language)." raise ValueError(
) f"Model '{self.model_name}' does not support language parameter "
"(auto-detects language)."
)
if isinstance(update, SarvamSTTSettings):
if is_given(update.prompt) and update.prompt is not None:
if not self._config.supports_prompt:
raise ValueError(
f"Model '{self.model_name}' does not support prompt parameter."
)
if is_given(update.mode) and update.mode is not None:
if not self._config.supports_mode:
raise ValueError(f"Model '{self.model_name}' does not support mode parameter.")
changed = await super()._update_settings_from_typed(update)
if not changed:
return changed
logger.info(f"Switching STT language to: [{language}]")
self._language_code = language
self._language_string = language_to_sarvam_language(language)
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
return changed
async def set_prompt(self, prompt: Optional[str]): async def set_prompt(self, prompt: Optional[str]):
"""Set the transcription/translation prompt and reconnect. """Set the transcription/translation prompt and reconnect.
.. deprecated::
Use ``STTUpdateSettingsFrame(SarvamSTTSettings(prompt=...))`` instead.
Args: Args:
prompt: Prompt text to guide transcription/translation style/context. prompt: Prompt text to guide transcription/translation style/context.
Pass None to clear/disable prompt. Pass None to clear/disable prompt.
Only applicable to models that support prompts. Only applicable to models that support prompts.
""" """
import warnings
with warnings.catch_warnings():
warnings.simplefilter("always")
warnings.warn(
f"{self.__class__.__name__}.set_prompt() is deprecated. "
"Use STTUpdateSettingsFrame(SarvamSTTSettings(prompt=...)) instead.",
DeprecationWarning,
stacklevel=2,
)
if not self._config.supports_prompt: if not self._config.supports_prompt:
if prompt is not None: if prompt is not None:
raise ValueError(f"Model '{self.model_name}' does not support prompt parameter.") raise ValueError(f"Model '{self.model_name}' does not support prompt parameter.")
@@ -318,7 +372,7 @@ class SarvamSTTService(STTService):
return return
logger.info(f"Updating {self.model_name} prompt.") logger.info(f"Updating {self.model_name} prompt.")
self._prompt = prompt self._settings.prompt = prompt
await self._disconnect() await self._disconnect()
await self._connect() await self._connect()
@@ -405,24 +459,25 @@ class SarvamSTTService(STTService):
# Enable flush signal when using Pipecat's VAD (not Sarvam's) so that # Enable flush signal when using Pipecat's VAD (not Sarvam's) so that
# the flush() call on user-stopped-speaking is honored by the server. # the flush() call on user-stopped-speaking is honored by the server.
if not self._vad_signals: if not self._settings.vad_signals:
connect_kwargs["flush_signal"] = "true" connect_kwargs["flush_signal"] = "true"
# Only send vad parameters when explicitly set (avoid overriding server defaults) # Only send vad parameters when explicitly set (avoid overriding server defaults)
if self._vad_signals is not None: if self._settings.vad_signals is not None:
connect_kwargs["vad_signals"] = "true" if self._vad_signals else "false" connect_kwargs["vad_signals"] = "true" if self._settings.vad_signals else "false"
if self._high_vad_sensitivity is not None: if self._settings.high_vad_sensitivity is not None:
connect_kwargs["high_vad_sensitivity"] = ( connect_kwargs["high_vad_sensitivity"] = (
"true" if self._high_vad_sensitivity else "false" "true" if self._settings.high_vad_sensitivity else "false"
) )
# Add language_code for models that support it # Add language_code for models that support it
if self._language_string is not None: language_string = self._get_language_string()
connect_kwargs["language_code"] = self._language_string if language_string is not None:
connect_kwargs["language_code"] = language_string
# Add mode for models that support it # Add mode for models that support it
if self._config.supports_mode and self._mode is not None: if self._config.supports_mode and is_given(self._settings.mode):
connect_kwargs["mode"] = self._mode connect_kwargs["mode"] = self._settings.mode
def _connect_with_sdk_headers(connect_fn, **kwargs): def _connect_with_sdk_headers(connect_fn, **kwargs):
# Different SDK versions may use different kwarg names. # Different SDK versions may use different kwarg names.
@@ -449,8 +504,8 @@ class SarvamSTTService(STTService):
self._socket_client = await self._websocket_context.__aenter__() self._socket_client = await self._websocket_context.__aenter__()
# Set prompt if provided (only for models that support prompts) # Set prompt if provided (only for models that support prompts)
if self._prompt is not None and self._config.supports_prompt: if is_given(self._settings.prompt) and self._config.supports_prompt:
await self._socket_client.set_prompt(self._prompt) await self._socket_client.set_prompt(self._settings.prompt)
# Register event handler for incoming messages # Register event handler for incoming messages
def _message_handler(message): def _message_handler(message):
@@ -544,10 +599,12 @@ class SarvamSTTService(STTService):
# Prefer language from message (auto-detected for translate models). Fallback to configured. # Prefer language from message (auto-detected for translate models). Fallback to configured.
if language_code: if language_code:
language = self._map_language_code_to_enum(language_code) language = self._map_language_code_to_enum(language_code)
elif self._language_string:
language = self._map_language_code_to_enum(self._language_string)
else: else:
language = Language.HI_IN language_string = self._get_language_string()
if language_string:
language = self._map_language_code_to_enum(language_string)
else:
language = Language.HI_IN
# Emit utterance end event # Emit utterance end event
await self._call_event_handler("on_utterance_end") await self._call_event_handler("on_utterance_end")

187
src/pipecat/services/sarvam/tts.py
View File

@@ -40,9 +40,9 @@ See https://docs.sarvam.ai/api-reference-docs/text-to-speech/stream for full API
import asyncio import asyncio
import base64 import base64
import json import json
from dataclasses import dataclass from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import Any, AsyncGenerator, Dict, List, Mapping, Optional, Tuple from typing import AsyncGenerator, Dict, List, Optional, Tuple
import aiohttp import aiohttp
from loguru import logger from loguru import logger
@@ -62,6 +62,7 @@ from pipecat.frames.frames import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.sarvam._sdk import sdk_headers from pipecat.services.sarvam._sdk import sdk_headers
from pipecat.services.settings import NOT_GIVEN, TTSSettings, is_given
from pipecat.services.tts_service import InterruptibleTTSService, TTSService from pipecat.services.tts_service import InterruptibleTTSService, TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -244,6 +245,80 @@ def language_to_sarvam_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=False) return resolve_language(language, LANGUAGE_MAP, use_base_code=False)
@dataclass
class SarvamHttpTTSSettings(TTSSettings):
"""Typed settings for Sarvam HTTP TTS service.
Parameters:
language: Sarvam language code.
enable_preprocessing: Whether to enable text preprocessing. Defaults to False.
**Note:** Always enabled for bulbul:v3-beta (cannot be disabled).
pace: Speech pace multiplier. Defaults to 1.0.
- bulbul:v2: Range 0.3 to 3.0
- bulbul:v3-beta: Range 0.5 to 2.0
pitch: Voice pitch adjustment (-0.75 to 0.75). Defaults to 0.0.
**Note:** Only supported for bulbul:v2. Ignored for v3 models.
loudness: Volume multiplier (0.3 to 3.0). Defaults to 1.0.
**Note:** Only supported for bulbul:v2. Ignored for v3 models.
temperature: Controls output randomness for bulbul:v3-beta (0.01 to 1.0).
Lower values = more deterministic, higher = more random. Defaults to 0.6.
**Note:** Only supported for bulbul:v3-beta. Ignored for v2.
sample_rate: Audio sample rate.
"""
language: str = field(default_factory=lambda: NOT_GIVEN)
enable_preprocessing: bool = field(default_factory=lambda: NOT_GIVEN)
pace: float = field(default_factory=lambda: NOT_GIVEN)
pitch: float = field(default_factory=lambda: NOT_GIVEN)
loudness: float = field(default_factory=lambda: NOT_GIVEN)
temperature: float = field(default_factory=lambda: NOT_GIVEN)
sarvam_sample_rate: int = field(default_factory=lambda: NOT_GIVEN)
@dataclass
class SarvamWSTTSSettings(TTSSettings):
"""Typed settings for Sarvam WebSocket TTS service.
Parameters:
target_language_code: Sarvam language code.
speaker: Voice speaker ID.
speech_sample_rate: Audio sample rate as string.
enable_preprocessing: Enable text preprocessing. Defaults to False.
**Note:** Always enabled for bulbul:v3-beta.
min_buffer_size: Minimum characters to buffer before generating audio.
Lower values reduce latency but may affect quality. Defaults to 50.
max_chunk_length: Maximum characters processed in a single chunk.
Controls memory usage and processing efficiency. Defaults to 150.
output_audio_codec: Audio codec format. Options: linear16, mulaw, alaw,
opus, flac, aac, wav, mp3. Defaults to "linear16".
output_audio_bitrate: Audio bitrate (32k, 64k, 96k, 128k, 192k).
Defaults to "128k".
pace: Speech pace multiplier. Defaults to 1.0.
- bulbul:v2: Range 0.3 to 3.0
- bulbul:v3-beta: Range 0.5 to 2.0
pitch: Voice pitch adjustment (-0.75 to 0.75). Defaults to 0.0.
**Note:** Only supported for bulbul:v2. Ignored for v3 models.
loudness: Volume multiplier (0.3 to 3.0). Defaults to 1.0.
**Note:** Only supported for bulbul:v2. Ignored for v3 models.
temperature: Controls output randomness for bulbul:v3-beta (0.01 to 1.0).
Lower = more deterministic, higher = more random. Defaults to 0.6.
**Note:** Only supported for bulbul:v3-beta. Ignored for v2.
"""
target_language_code: str = field(default_factory=lambda: NOT_GIVEN)
speaker: str = field(default_factory=lambda: NOT_GIVEN)
speech_sample_rate: str = field(default_factory=lambda: NOT_GIVEN)
enable_preprocessing: bool = field(default_factory=lambda: NOT_GIVEN)
min_buffer_size: int = field(default_factory=lambda: NOT_GIVEN)
max_chunk_length: int = field(default_factory=lambda: NOT_GIVEN)
output_audio_codec: str = field(default_factory=lambda: NOT_GIVEN)
output_audio_bitrate: str = field(default_factory=lambda: NOT_GIVEN)
pace: float = field(default_factory=lambda: NOT_GIVEN)
pitch: float = field(default_factory=lambda: NOT_GIVEN)
loudness: float = field(default_factory=lambda: NOT_GIVEN)
temperature: float = field(default_factory=lambda: NOT_GIVEN)
class SarvamHttpTTSService(TTSService): class SarvamHttpTTSService(TTSService):
"""Text-to-Speech service using Sarvam AI's API. """Text-to-Speech service using Sarvam AI's API.
@@ -403,35 +478,35 @@ class SarvamHttpTTSService(TTSService):
pace = max(pace_min, min(pace_max, pace)) pace = max(pace_min, min(pace_max, pace))
# Build base settings # Build base settings
self._settings = { self._settings: SarvamHttpTTSSettings = SarvamHttpTTSSettings(
"language": ( language=(
self.language_to_service_language(params.language) if params.language else "en-IN" self.language_to_service_language(params.language) if params.language else "en-IN"
), ),
"enable_preprocessing": ( enable_preprocessing=(
True if self._config.preprocessing_always_enabled else params.enable_preprocessing True if self._config.preprocessing_always_enabled else params.enable_preprocessing
), ),
"pace": pace, pace=pace,
"model": model, model=model,
} )
# Add parameters based on model support # Add parameters based on model support
if self._config.supports_pitch: if self._config.supports_pitch:
self._settings["pitch"] = params.pitch self._settings.pitch = params.pitch
elif params.pitch != 0.0: elif params.pitch != 0.0:
logger.warning(f"pitch parameter is ignored for {model}") logger.warning(f"pitch parameter is ignored for {model}")
if self._config.supports_loudness: if self._config.supports_loudness:
self._settings["loudness"] = params.loudness self._settings.loudness = params.loudness
elif params.loudness != 1.0: elif params.loudness != 1.0:
logger.warning(f"loudness parameter is ignored for {model}") logger.warning(f"loudness parameter is ignored for {model}")
if self._config.supports_temperature: if self._config.supports_temperature:
self._settings["temperature"] = params.temperature self._settings.temperature = params.temperature
elif params.temperature != 0.6: elif params.temperature != 0.6:
logger.warning(f"temperature parameter is ignored for {model}") logger.warning(f"temperature parameter is ignored for {model}")
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice_id) self._voice_id = voice_id
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.
@@ -459,7 +534,7 @@ class SarvamHttpTTSService(TTSService):
frame: The start frame containing initialization parameters. frame: The start frame containing initialization parameters.
""" """
await super().start(frame) await super().start(frame)
self._settings["sample_rate"] = self.sample_rate self._settings.sarvam_sample_rate = self.sample_rate
@traced_tts @traced_tts
async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]: async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]:
@@ -480,21 +555,25 @@ class SarvamHttpTTSService(TTSService):
# Build payload with common parameters # Build payload with common parameters
payload = { payload = {
"text": text, "text": text,
"target_language_code": self._settings["language"], "target_language_code": self._settings.language,
"speaker": self._voice_id, "speaker": self._voice_id,
"sample_rate": self.sample_rate, "sample_rate": self.sample_rate,
"enable_preprocessing": self._settings["enable_preprocessing"], "enable_preprocessing": self._settings.enable_preprocessing,
"model": self._model_name, "model": self._model_name,
"pace": self._settings.get("pace", 1.0), "pace": self._settings.pace if is_given(self._settings.pace) else 1.0,
} }
# Add model-specific parameters based on config # Add model-specific parameters based on config
if self._config.supports_pitch: if self._config.supports_pitch:
payload["pitch"] = self._settings.get("pitch", 0.0) payload["pitch"] = self._settings.pitch if is_given(self._settings.pitch) else 0.0
if self._config.supports_loudness: if self._config.supports_loudness:
payload["loudness"] = self._settings.get("loudness", 1.0) payload["loudness"] = (
self._settings.loudness if is_given(self._settings.loudness) else 1.0
)
if self._config.supports_temperature: if self._config.supports_temperature:
payload["temperature"] = self._settings.get("temperature", 0.6) payload["temperature"] = (
self._settings.temperature if is_given(self._settings.temperature) else 0.6
)
headers = { headers = {
"api-subscription-key": self._api_key, "api-subscription-key": self._api_key,
@@ -748,7 +827,7 @@ class SarvamTTSService(InterruptibleTTSService):
self._websocket_url = f"{url}?model={model}" self._websocket_url = f"{url}?model={model}"
self._api_key = api_key self._api_key = api_key
self.set_model_name(model) self.set_model_name(model)
self.set_voice(voice_id) self._voice_id = voice_id
# Validate and clamp pace to model's valid range # Validate and clamp pace to model's valid range
pace = params.pace pace = params.pace
@@ -758,36 +837,36 @@ class SarvamTTSService(InterruptibleTTSService):
pace = max(pace_min, min(pace_max, pace)) pace = max(pace_min, min(pace_max, pace))
# Build base settings # Build base settings
self._settings = { self._settings: SarvamWSTTSSettings = SarvamWSTTSSettings(
"target_language_code": ( target_language_code=(
self.language_to_service_language(params.language) if params.language else "en-IN" self.language_to_service_language(params.language) if params.language else "en-IN"
), ),
"speaker": voice_id, speaker=voice_id,
"speech_sample_rate": str(sample_rate), speech_sample_rate=str(sample_rate),
"enable_preprocessing": ( enable_preprocessing=(
True if self._config.preprocessing_always_enabled else params.enable_preprocessing True if self._config.preprocessing_always_enabled else params.enable_preprocessing
), ),
"min_buffer_size": params.min_buffer_size, min_buffer_size=params.min_buffer_size,
"max_chunk_length": params.max_chunk_length, max_chunk_length=params.max_chunk_length,
"output_audio_codec": params.output_audio_codec, output_audio_codec=params.output_audio_codec,
"output_audio_bitrate": params.output_audio_bitrate, output_audio_bitrate=params.output_audio_bitrate,
"pace": pace, pace=pace,
"model": model, model=model,
} )
# Add parameters based on model support # Add parameters based on model support
if self._config.supports_pitch: if self._config.supports_pitch:
self._settings["pitch"] = params.pitch self._settings.pitch = params.pitch
elif params.pitch != 0.0: elif params.pitch != 0.0:
logger.warning(f"pitch parameter is ignored for {model}") logger.warning(f"pitch parameter is ignored for {model}")
if self._config.supports_loudness: if self._config.supports_loudness:
self._settings["loudness"] = params.loudness self._settings.loudness = params.loudness
elif params.loudness != 1.0: elif params.loudness != 1.0:
logger.warning(f"loudness parameter is ignored for {model}") logger.warning(f"loudness parameter is ignored for {model}")
if self._config.supports_temperature: if self._config.supports_temperature:
self._settings["temperature"] = params.temperature self._settings.temperature = params.temperature
elif params.temperature != 0.6: elif params.temperature != 0.6:
logger.warning(f"temperature parameter is ignored for {model}") logger.warning(f"temperature parameter is ignored for {model}")
@@ -823,7 +902,7 @@ class SarvamTTSService(InterruptibleTTSService):
await super().start(frame) await super().start(frame)
# WebSocket API expects sample rate as string # WebSocket API expects sample rate as string
self._settings["speech_sample_rate"] = str(self.sample_rate) self._settings.speech_sample_rate = str(self.sample_rate)
await self._connect() await self._connect()
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
@@ -870,13 +949,12 @@ class SarvamTTSService(InterruptibleTTSService):
if isinstance(frame, (LLMFullResponseEndFrame, EndFrame)): if isinstance(frame, (LLMFullResponseEndFrame, EndFrame)):
await self.flush_audio() await self.flush_audio()
async def _update_settings(self, settings: Mapping[str, Any]): async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Update service settings and reconnect if voice changed.""" """Apply a typed settings update and resend config if voice changed."""
prev_voice = self._voice_id changed = await super()._update_settings_from_typed(update)
await super()._update_settings(settings) if "voice" in changed:
if not prev_voice == self._voice_id:
logger.info(f"Switching TTS voice to: [{self._voice_id}]")
await self._send_config() await self._send_config()
return changed
async def _connect(self): async def _connect(self):
"""Connect to Sarvam WebSocket and start background tasks.""" """Connect to Sarvam WebSocket and start background tasks."""
@@ -934,9 +1012,28 @@ class SarvamTTSService(InterruptibleTTSService):
"""Send initial configuration message.""" """Send initial configuration message."""
if not self._websocket: if not self._websocket:
raise Exception("WebSocket not connected") raise Exception("WebSocket not connected")
self._settings["speaker"] = self._voice_id self._settings.speaker = self._voice_id
logger.debug(f"Config being sent is {self._settings}") # Build config dict for the API
config_message = {"type": "config", "data": self._settings} config_data = {
"target_language_code": self._settings.target_language_code,
"speaker": self._settings.speaker,
"speech_sample_rate": self._settings.speech_sample_rate,
"enable_preprocessing": self._settings.enable_preprocessing,
"min_buffer_size": self._settings.min_buffer_size,
"max_chunk_length": self._settings.max_chunk_length,
"output_audio_codec": self._settings.output_audio_codec,
"output_audio_bitrate": self._settings.output_audio_bitrate,
"pace": self._settings.pace,
"model": self._settings.model,
}
if is_given(self._settings.pitch):
config_data["pitch"] = self._settings.pitch
if is_given(self._settings.loudness):
config_data["loudness"] = self._settings.loudness
if is_given(self._settings.temperature):
config_data["temperature"] = self._settings.temperature
logger.debug(f"Config being sent is {config_data}")
config_message = {"type": "config", "data": config_data}
try: try:
await self._websocket.send(json.dumps(config_message)) await self._websocket.send(json.dumps(config_message))

297
src/pipecat/services/settings.py Normal file
View File

@@ -0,0 +1,297 @@
#
# Copyright (c) 2024-2026, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""Typed settings infrastructure for Pipecat AI services.
This module provides typed dataclass-based settings objects that replace the
stringly-typed ``Mapping[str, Any]`` dictionaries previously used for service
configuration. Each service type has a corresponding settings class (e.g.
``TTSSettings``, ``LLMSettings``) whose fields use the ``NOT_GIVEN`` sentinel
to distinguish "leave unchanged" from an explicit ``None``.
Key concepts:
- **NOT_GIVEN sentinel**: A value meaning "this field was not provided in the
update". Distinct from ``None`` (which may be a valid value for a setting).
- **Settings as both state and delta**: The same class is used for the
service's current settings *and* for update objects. Fields set to
``NOT_GIVEN`` are simply skipped when applying an update.
- **apply_update**: Applies a delta onto a target settings object and returns
the set of field names that actually changed.
- **from_mapping**: Constructs a typed settings object from a plain dict,
supporting field aliases (e.g. ``"voice_id"`` → ``"voice"``).
- **Extras**: Unknown keys land in the ``extra`` dict so services that have
non-standard settings don't lose data.
"""
from __future__ import annotations
import copy
from dataclasses import dataclass, field, fields
from typing import Any, ClassVar, Dict, Mapping, Optional, Set, Type, TypeVar
from loguru import logger
# ---------------------------------------------------------------------------
# NOT_GIVEN sentinel
# ---------------------------------------------------------------------------
class _NotGiven:
"""Sentinel indicating a settings field was not provided.
``NOT_GIVEN`` means "the caller did not supply this value" — distinct from
``None``, which may be a legitimate setting value. It is used as the
default for every settings field so that ``apply_update`` can tell which
fields the caller actually wants to change.
"""
_instance: Optional[_NotGiven] = None
def __new__(cls) -> _NotGiven:
if cls._instance is None:
cls._instance = super().__new__(cls)
return cls._instance
def __repr__(self) -> str:
return "NOT_GIVEN"
def __bool__(self) -> bool:
return False
NOT_GIVEN: _NotGiven = _NotGiven()
"""Singleton sentinel meaning "this field was not included in the update"."""
def is_given(value: Any) -> bool:
"""Check whether a value was explicitly provided (i.e. is not ``NOT_GIVEN``).
Args:
value: The value to check.
Returns:
``True`` if *value* is anything other than ``NOT_GIVEN``.
"""
return not isinstance(value, _NotGiven)
# ---------------------------------------------------------------------------
# Base ServiceSettings
# ---------------------------------------------------------------------------
_S = TypeVar("_S", bound="ServiceSettings")
@dataclass
class ServiceSettings:
"""Base class for typed service settings.
Every AI service type (LLM, TTS, STT) extends this with its own fields.
Fields default to ``NOT_GIVEN`` so that an instance can represent either
the full current state **or** a sparse update delta.
Parameters:
model: The model identifier used by the service.
extra: Overflow dict for service-specific keys that don't map to a
declared field.
"""
# -- common fields -------------------------------------------------------
model: Any = field(default_factory=lambda: NOT_GIVEN)
"""AI model identifier (e.g. ``"gpt-4o"``, ``"eleven_turbo_v2_5"``)."""
extra: Dict[str, Any] = field(default_factory=dict)
"""Catch-all for service-specific keys that have no declared field."""
# -- class-level configuration -------------------------------------------
_aliases: ClassVar[Dict[str, str]] = {}
"""Map of alternative key names to canonical field names.
For example ``{"voice_id": "voice"}`` lets callers use either spelling.
Subclasses should override this as needed.
"""
# -- public API ----------------------------------------------------------
def given_fields(self) -> Dict[str, Any]:
"""Return a dict of only the fields that were explicitly provided.
Skips ``NOT_GIVEN`` values and the ``extra`` field itself. Entries
from ``extra`` are included at the top level.
Returns:
Dictionary mapping field names to their provided values.
"""
result: Dict[str, Any] = {}
for f in fields(self):
if f.name == "extra":
continue
val = getattr(self, f.name)
if is_given(val):
result[f.name] = val
result.update(self.extra)
return result
def apply_update(self: _S, update: _S) -> Set[str]:
"""Apply *update* onto this settings object, returning changed field names.
Only fields in *update* that are **given** (i.e. not ``NOT_GIVEN``)
are considered. A field is "changed" if its new value differs from
the current value.
The ``extra`` dicts are merged: keys present in the update overwrite
keys in the target.
Args:
update: A settings object of the same type containing the delta.
Returns:
The set of field names whose values actually changed.
Examples::
current = TTSSettings(voice="alice", language="en")
delta = TTSSettings(voice="bob")
changed = current.apply_update(delta)
# changed == {"voice"}
# current.voice == "bob", current.language == "en"
"""
changed: Set[str] = set()
for f in fields(self):
if f.name == "extra":
continue
new_val = getattr(update, f.name)
if not is_given(new_val):
continue
old_val = getattr(self, f.name)
if old_val != new_val:
setattr(self, f.name, new_val)
changed.add(f.name)
# Merge extra
for key, new_val in update.extra.items():
old_val = self.extra.get(key, NOT_GIVEN)
if old_val != new_val:
self.extra[key] = new_val
changed.add(key)
return changed
@classmethod
def from_mapping(cls: Type[_S], settings: Mapping[str, Any]) -> _S:
"""Construct a typed settings object from a plain dictionary.
Keys are matched to dataclass fields by name. Keys listed in
``_aliases`` are translated to their canonical name first. Any
remaining unrecognized keys are placed into ``extra``.
Args:
settings: A dictionary of setting names to values.
Returns:
A new settings instance with the corresponding fields populated.
Examples::
update = TTSSettings.from_mapping({"voice_id": "alice", "speed": 1.2})
# update.voice == "alice" (via alias)
# update.extra == {"speed": 1.2}
"""
field_names = {f.name for f in fields(cls)} - {"extra"}
kwargs: Dict[str, Any] = {}
extra: Dict[str, Any] = {}
for key, value in settings.items():
# Resolve aliases first
canonical = cls._aliases.get(key, key)
if canonical in field_names:
kwargs[canonical] = value
else:
extra[key] = value
instance = cls(**kwargs)
instance.extra = extra
return instance
def to_dict(self) -> Dict[str, Any]:
"""Serialize to a flat dictionary, including extra.
Only given (non-``NOT_GIVEN``) values are included. This is the
inverse of ``from_mapping`` and useful for passing settings to APIs
that expect plain dicts.
Returns:
A flat dictionary of all given settings.
"""
return self.given_fields()
def copy(self: _S) -> _S:
"""Return a deep copy of this settings instance.
Returns:
A new settings object with the same field values.
"""
return copy.deepcopy(self)
# ---------------------------------------------------------------------------
# Service-specific settings
# ---------------------------------------------------------------------------
@dataclass
class LLMSettings(ServiceSettings):
"""Typed settings for LLM services.
Parameters:
model: LLM model identifier.
temperature: Sampling temperature.
max_tokens: Maximum tokens to generate.
top_p: Nucleus sampling probability.
top_k: Top-k sampling parameter.
frequency_penalty: Frequency penalty.
presence_penalty: Presence penalty.
seed: Random seed for reproducibility.
"""
temperature: Any = field(default_factory=lambda: NOT_GIVEN)
max_tokens: Any = field(default_factory=lambda: NOT_GIVEN)
top_p: Any = field(default_factory=lambda: NOT_GIVEN)
top_k: Any = field(default_factory=lambda: NOT_GIVEN)
frequency_penalty: Any = field(default_factory=lambda: NOT_GIVEN)
presence_penalty: Any = field(default_factory=lambda: NOT_GIVEN)
seed: Any = field(default_factory=lambda: NOT_GIVEN)
@dataclass
class TTSSettings(ServiceSettings):
"""Typed settings for TTS services.
Parameters:
model: TTS model identifier.
voice: Voice identifier or name.
language: Language for speech synthesis.
"""
voice: Any = field(default_factory=lambda: NOT_GIVEN)
language: Any = field(default_factory=lambda: NOT_GIVEN)
_aliases: ClassVar[Dict[str, str]] = {"voice_id": "voice"}
@dataclass
class STTSettings(ServiceSettings):
"""Typed settings for STT services.
Parameters:
model: STT model identifier.
language: Language for speech recognition.
"""
language: Any = field(default_factory=lambda: NOT_GIVEN)

76
src/pipecat/services/soniox/stt.py
View File

@@ -8,7 +8,8 @@
import json import json
import time import time
from typing import AsyncGenerator, List, Optional from dataclasses import dataclass, field
from typing import Any, AsyncGenerator, List, Optional
from loguru import logger from loguru import logger
from pydantic import BaseModel from pydantic import BaseModel
@@ -23,6 +24,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import SONIOX_TTFS_P99 from pipecat.services.stt_latency import SONIOX_TTFS_P99
from pipecat.services.stt_service import WebsocketSTTService from pipecat.services.stt_service import WebsocketSTTService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -134,6 +136,17 @@ def _prepare_language_hints(
return list(set(prepared_languages)) return list(set(prepared_languages))
@dataclass
class SonioxSTTSettings(STTSettings):
"""Typed settings for Soniox STT service.
Parameters:
input_params: Soniox ``SonioxInputParams`` for detailed configuration.
"""
input_params: SonioxInputParams = field(default_factory=lambda: NOT_GIVEN)
class SonioxSTTService(WebsocketSTTService): class SonioxSTTService(WebsocketSTTService):
"""Speech-to-Text service using Soniox's WebSocket API. """Speech-to-Text service using Soniox's WebSocket API.
@@ -181,9 +194,13 @@ class SonioxSTTService(WebsocketSTTService):
self._api_key = api_key self._api_key = api_key
self._url = url self._url = url
self.set_model_name(params.model) self.set_model_name(params.model)
self._params = params
self._vad_force_turn_endpoint = vad_force_turn_endpoint self._vad_force_turn_endpoint = vad_force_turn_endpoint
self._settings = SonioxSTTSettings(
model=params.model,
input_params=params,
)
self._final_transcription_buffer = [] self._final_transcription_buffer = []
self._last_tokens_received: Optional[float] = None self._last_tokens_received: Optional[float] = None
@@ -198,6 +215,43 @@ class SonioxSTTService(WebsocketSTTService):
await super().start(frame) await super().start(frame)
await self._connect() await self._connect()
async def _update_settings_from_typed(self, update: SonioxSTTSettings) -> set[str]:
"""Apply a typed settings update, keeping ``input_params`` in sync.
Top-level ``model`` is the source of truth. When it is given in
*update* its value is propagated into ``input_params``. When only
``input_params`` is given, its ``model`` is propagated *up* to the
top-level field.
Any change triggers a WebSocket reconnect.
Args:
update: A typed settings delta.
Returns:
Set of field names whose values actually changed.
"""
model_given = is_given(getattr(update, "model", NOT_GIVEN))
changed = await super()._update_settings_from_typed(update)
if not changed:
return changed
# --- Sync model --------------------------------------------------
if model_given:
# Top-level model wins → push into input_params.
self._settings.input_params.model = self._settings.model
elif "input_params" in changed and self._settings.input_params.model is not None:
# Only input_params was given → pull model up.
self._settings.model = self._settings.input_params.model
self.set_model_name(self._settings.model)
await self._disconnect()
await self._connect()
return changed
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
"""Stop the Soniox STT websocket connection. """Stop the Soniox STT websocket connection.
@@ -311,7 +365,9 @@ class SonioxSTTService(WebsocketSTTService):
# Either one or the other is required. # Either one or the other is required.
enable_endpoint_detection = not self._vad_force_turn_endpoint enable_endpoint_detection = not self._vad_force_turn_endpoint
context = self._params.context params = self._settings.input_params
context = params.context
if isinstance(context, SonioxContextObject): if isinstance(context, SonioxContextObject):
context = context.model_dump() context = context.model_dump()
@@ -319,16 +375,16 @@ class SonioxSTTService(WebsocketSTTService):
config = { config = {
"api_key": self._api_key, "api_key": self._api_key,
"model": self._model_name, "model": self._model_name,
"audio_format": self._params.audio_format, "audio_format": params.audio_format,
"num_channels": self._params.num_channels or 1, "num_channels": params.num_channels or 1,
"enable_endpoint_detection": enable_endpoint_detection, "enable_endpoint_detection": enable_endpoint_detection,
"sample_rate": self.sample_rate, "sample_rate": self.sample_rate,
"language_hints": _prepare_language_hints(self._params.language_hints), "language_hints": _prepare_language_hints(params.language_hints),
"language_hints_strict": self._params.language_hints_strict, "language_hints_strict": params.language_hints_strict,
"context": context, "context": context,
"enable_speaker_diarization": self._params.enable_speaker_diarization, "enable_speaker_diarization": params.enable_speaker_diarization,
"enable_language_identification": self._params.enable_language_identification, "enable_language_identification": params.enable_language_identification,
"client_reference_id": self._params.client_reference_id, "client_reference_id": params.client_reference_id,
} }
# Send the configuration message. # Send the configuration message.

245
src/pipecat/services/speechmatics/stt.py
View File

@@ -8,8 +8,10 @@
import asyncio import asyncio
import os import os
import warnings
from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import Any, AsyncGenerator from typing import Any, AsyncGenerator, ClassVar
from dotenv import load_dotenv from dotenv import load_dotenv
from loguru import logger from loguru import logger
@@ -31,6 +33,7 @@ from pipecat.frames.frames import (
VADUserStoppedSpeakingFrame, VADUserStoppedSpeakingFrame,
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.settings import NOT_GIVEN, STTSettings, is_given
from pipecat.services.stt_latency import SPEECHMATICS_TTFS_P99 from pipecat.services.stt_latency import SPEECHMATICS_TTFS_P99
from pipecat.services.stt_service import STTService from pipecat.services.stt_service import STTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -80,6 +83,81 @@ class TurnDetectionMode(str, Enum):
SMART_TURN = "smart_turn" SMART_TURN = "smart_turn"
@dataclass
class SpeechmaticsSTTSettings(STTSettings):
"""Typed settings for Speechmatics STT service.
See ``SpeechmaticsSTTService.InputParams`` for detailed descriptions of each field.
Parameters:
model: The operating point / model name.
domain: Domain for Speechmatics API.
turn_detection_mode: Endpoint handling mode.
speaker_active_format: Formatter for active speaker ID.
speaker_passive_format: Formatter for passive speaker ID.
focus_speakers: List of speaker IDs to focus on.
ignore_speakers: List of speaker IDs to ignore.
focus_mode: Speaker focus mode for diarization.
known_speakers: List of known speaker labels and identifiers.
additional_vocab: List of additional vocabulary entries.
audio_encoding: Audio encoding format.
operating_point: Operating point for accuracy vs. latency.
max_delay: Maximum delay in seconds for transcription.
end_of_utterance_silence_trigger: Maximum delay for end of utterance trigger.
end_of_utterance_max_delay: Maximum delay for end of utterance.
punctuation_overrides: Punctuation overrides.
include_partials: Include partial segment fragments.
split_sentences: Emit finalized sentences mid-turn.
enable_diarization: Enable speaker diarization.
speaker_sensitivity: Diarization sensitivity.
max_speakers: Maximum number of speakers to detect.
prefer_current_speaker: Prefer current speaker ID.
extra_params: Extra parameters for the STT engine.
"""
domain: str = field(default_factory=lambda: NOT_GIVEN)
turn_detection_mode: TurnDetectionMode = field(default_factory=lambda: NOT_GIVEN)
speaker_active_format: str = field(default_factory=lambda: NOT_GIVEN)
speaker_passive_format: str = field(default_factory=lambda: NOT_GIVEN)
focus_speakers: list = field(default_factory=lambda: NOT_GIVEN)
ignore_speakers: list = field(default_factory=lambda: NOT_GIVEN)
focus_mode: Any = field(default_factory=lambda: NOT_GIVEN)
known_speakers: list = field(default_factory=lambda: NOT_GIVEN)
additional_vocab: list = field(default_factory=lambda: NOT_GIVEN)
audio_encoding: Any = field(default_factory=lambda: NOT_GIVEN)
operating_point: Any = field(default_factory=lambda: NOT_GIVEN)
max_delay: float = field(default_factory=lambda: NOT_GIVEN)
end_of_utterance_silence_trigger: float = field(default_factory=lambda: NOT_GIVEN)
end_of_utterance_max_delay: float = field(default_factory=lambda: NOT_GIVEN)
punctuation_overrides: dict = field(default_factory=lambda: NOT_GIVEN)
include_partials: bool = field(default_factory=lambda: NOT_GIVEN)
split_sentences: bool = field(default_factory=lambda: NOT_GIVEN)
enable_diarization: bool = field(default_factory=lambda: NOT_GIVEN)
speaker_sensitivity: float = field(default_factory=lambda: NOT_GIVEN)
max_speakers: int = field(default_factory=lambda: NOT_GIVEN)
prefer_current_speaker: bool = field(default_factory=lambda: NOT_GIVEN)
extra_params: dict = field(default_factory=lambda: NOT_GIVEN)
#: Fields that can be updated on a live connection via the Speechmatics
#: diarization-config API — no reconnect needed.
HOT_FIELDS: ClassVar[frozenset[str]] = frozenset(
{
"focus_speakers",
"ignore_speakers",
"focus_mode",
}
)
#: Fields that are purely local (formatting templates) — no reconnect
#: and no API call needed.
LOCAL_FIELDS: ClassVar[frozenset[str]] = frozenset(
{
"speaker_active_format",
"speaker_passive_format",
}
)
class SpeechmaticsSTTService(STTService): class SpeechmaticsSTTService(STTService):
"""Speechmatics STT service implementation. """Speechmatics STT service implementation.
@@ -327,30 +405,56 @@ class SpeechmaticsSTTService(STTService):
# Deprecation check # Deprecation check
self._check_deprecated_args(kwargs, params) self._check_deprecated_args(kwargs, params)
# Voice agent # Output formatting defaults
speaker_active_format = params.speaker_active_format
if speaker_active_format is None:
speaker_active_format = (
"@{speaker_id}: {text}" if params.enable_diarization else "{text}"
)
speaker_passive_format = params.speaker_passive_format or speaker_active_format
# Typed settings — seeded from InputParams
self._settings = SpeechmaticsSTTSettings(
language=params.language,
domain=params.domain,
turn_detection_mode=params.turn_detection_mode,
speaker_active_format=speaker_active_format,
speaker_passive_format=speaker_passive_format,
focus_speakers=params.focus_speakers,
ignore_speakers=params.ignore_speakers,
focus_mode=params.focus_mode,
known_speakers=params.known_speakers,
additional_vocab=params.additional_vocab,
audio_encoding=params.audio_encoding,
operating_point=params.operating_point,
max_delay=params.max_delay,
end_of_utterance_silence_trigger=params.end_of_utterance_silence_trigger,
end_of_utterance_max_delay=params.end_of_utterance_max_delay,
punctuation_overrides=params.punctuation_overrides,
include_partials=params.include_partials,
split_sentences=params.split_sentences,
enable_diarization=params.enable_diarization,
speaker_sensitivity=params.speaker_sensitivity,
max_speakers=params.max_speakers,
prefer_current_speaker=params.prefer_current_speaker,
extra_params=params.extra_params,
)
# Build SDK config from settings
self._client: VoiceAgentClient | None = None self._client: VoiceAgentClient | None = None
self._config: VoiceAgentConfig = self._prepare_config(params) self._config: VoiceAgentConfig = self._build_config()
# Outbound frame queue # Outbound frame queue
self._outbound_frames: asyncio.Queue[Frame] = asyncio.Queue() self._outbound_frames: asyncio.Queue[Frame] = asyncio.Queue()
# Output formatting
if params.speaker_active_format is None:
params.speaker_active_format = (
"@{speaker_id}: {text}" if params.enable_diarization else "{text}"
)
# Framework options # Framework options
self._enable_vad: bool = self._config.end_of_utterance_mode not in [ self._enable_vad: bool = self._config.end_of_utterance_mode not in [
EndOfUtteranceMode.FIXED, EndOfUtteranceMode.FIXED,
EndOfUtteranceMode.EXTERNAL, EndOfUtteranceMode.EXTERNAL,
] ]
self._speaker_active_format: str = params.speaker_active_format
self._speaker_passive_format: str = (
params.speaker_passive_format or params.speaker_active_format
)
# Model + metrics # Model + metrics (operating_point comes from the SDK config/preset)
self._settings.model = self._config.operating_point.value
self.set_model_name(self._config.operating_point.value) self.set_model_name(self._config.operating_point.value)
# Message queue # Message queue
@@ -374,6 +478,56 @@ class SpeechmaticsSTTService(STTService):
await super().start(frame) await super().start(frame)
await self._connect() await self._connect()
async def _update_settings_from_typed(self, update: SpeechmaticsSTTSettings) -> set[str]:
"""Apply typed settings update, reconnecting only when necessary.
Fields are classified into three categories (see
``SpeechmaticsSTTSettings``):
* **HOT_FIELDS** – diarization speaker settings that can be pushed
to a live Speechmatics connection without reconnecting.
* **LOCAL_FIELDS** – formatting templates evaluated locally; no
reconnect or API call needed.
* Everything else – baked into ``VoiceAgentConfig`` at connection
time and therefore require a full disconnect / reconnect.
Args:
update: A typed settings delta.
Returns:
Set of field names whose values actually changed.
"""
changed = await super()._update_settings_from_typed(update)
if not changed:
return changed
no_reconnect = SpeechmaticsSTTSettings.HOT_FIELDS | SpeechmaticsSTTSettings.LOCAL_FIELDS
needs_reconnect = bool(changed - no_reconnect)
if needs_reconnect:
# Connection-level fields changed — rebuild the SDK config
# from the now-updated self._settings, then reconnect.
self._config = self._build_config()
await self._disconnect()
await self._connect()
elif changed & SpeechmaticsSTTSettings.HOT_FIELDS:
if self._config.enable_diarization:
# Only hot-updatable fields changed — push to the live session.
self._config.speaker_config.focus_speakers = self._settings.focus_speakers
self._config.speaker_config.ignore_speakers = self._settings.ignore_speakers
self._config.speaker_config.focus_mode = self._settings.focus_mode
if self._client:
self._client.update_diarization_config(self._config.speaker_config)
else:
# Diarization not enabled — need a full reconnect to apply.
self._config = self._build_config()
await self._disconnect()
await self._connect()
# LOCAL_FIELDS: already applied by super(); nothing else to do.
return changed
async def stop(self, frame: EndFrame): async def stop(self, frame: EndFrame):
"""Called when the session ends.""" """Called when the session ends."""
await super().stop(frame) await super().stop(frame)
@@ -484,28 +638,35 @@ class SpeechmaticsSTTService(STTService):
# CONFIGURATION # CONFIGURATION
# ============================================================================ # ============================================================================
def _prepare_config(self, params: InputParams) -> VoiceAgentConfig: def _build_config(self) -> VoiceAgentConfig:
"""Parse the InputParams into VoiceAgentConfig.""" """Build a ``VoiceAgentConfig`` from the current ``self._settings``.
# Preset
config = VoiceAgentConfigPreset.load(params.turn_detection_mode.value) Used both at init time and before reconnecting so the connection
always reflects the latest settings.
"""
s = self._settings
# Preset from turn detection mode
config = VoiceAgentConfigPreset.load(s.turn_detection_mode.value)
# Language + domain # Language + domain
config.language = self._language_to_speechmatics_language(params.language) language = s.language
config.domain = params.domain config.language = self._language_to_speechmatics_language(language)
config.output_locale = self._locale_to_speechmatics_locale(config.language, params.language) config.domain = s.domain if is_given(s.domain) else None
config.output_locale = self._locale_to_speechmatics_locale(config.language, language)
# Speaker config # Speaker config
config.speaker_config = SpeakerFocusConfig( config.speaker_config = SpeakerFocusConfig(
focus_speakers=params.focus_speakers, focus_speakers=s.focus_speakers if is_given(s.focus_speakers) else [],
ignore_speakers=params.ignore_speakers, ignore_speakers=s.ignore_speakers if is_given(s.ignore_speakers) else [],
focus_mode=params.focus_mode, focus_mode=s.focus_mode if is_given(s.focus_mode) else SpeakerFocusMode.RETAIN,
) )
config.known_speakers = params.known_speakers config.known_speakers = s.known_speakers if is_given(s.known_speakers) else []
# Custom dictionary # Custom dictionary
config.additional_vocab = params.additional_vocab config.additional_vocab = s.additional_vocab if is_given(s.additional_vocab) else []
# Advanced parameters # Advanced parameters — only set if given (not NOT_GIVEN or None)
for param in [ for param in [
"operating_point", "operating_point",
"max_delay", "max_delay",
@@ -519,21 +680,20 @@ class SpeechmaticsSTTService(STTService):
"max_speakers", "max_speakers",
"prefer_current_speaker", "prefer_current_speaker",
]: ]:
if getattr(params, param) is not None: val = getattr(s, param)
setattr(config, param, getattr(params, param)) if is_given(val) and val is not None:
setattr(config, param, val)
# Extra parameters # Extra parameters
if isinstance(params.extra_params, dict): if is_given(s.extra_params) and isinstance(s.extra_params, dict):
for key, value in params.extra_params.items(): for key, value in s.extra_params.items():
if hasattr(config, key): if hasattr(config, key):
setattr(config, key, value) setattr(config, key, value)
# Enable sentences # Enable sentences
config.speech_segment_config = SpeechSegmentConfig( split = s.split_sentences if is_given(s.split_sentences) else False
emit_sentences=params.split_sentences or False config.speech_segment_config = SpeechSegmentConfig(emit_sentences=split or False)
)
# Return the complete config
return config return config
def update_params( def update_params(
@@ -542,12 +702,23 @@ class SpeechmaticsSTTService(STTService):
) -> None: ) -> None:
"""Updates the speaker configuration. """Updates the speaker configuration.
.. deprecated::
Use ``STTUpdateSettingsFrame`` with
``SpeechmaticsSTTSettings(...)`` instead.
This can update the speakers to listen to or ignore during an in-flight This can update the speakers to listen to or ignore during an in-flight
transcription. Only available if diarization is enabled. transcription. Only available if diarization is enabled.
Args: Args:
params: Update parameters for the service. params: Update parameters for the service.
""" """
with warnings.catch_warnings():
warnings.simplefilter("always")
warnings.warn(
"update_params() is deprecated. Use STTUpdateSettingsFrame with "
"SpeechmaticsSTTSettings(...) instead.",
DeprecationWarning,
)
# Check possible # Check possible
if not self._config.enable_diarization: if not self._config.enable_diarization:
raise ValueError("Diarization is not enabled") raise ValueError("Diarization is not enabled")
@@ -717,9 +888,9 @@ class SpeechmaticsSTTService(STTService):
def attr_from_segment(segment: dict[str, Any]) -> dict[str, Any]: def attr_from_segment(segment: dict[str, Any]) -> dict[str, Any]:
# Formats the output text based on the speaker and defined formats from the config. # Formats the output text based on the speaker and defined formats from the config.
text = ( text = (
self._speaker_active_format self._settings.speaker_active_format
if segment.get("is_active", True) if segment.get("is_active", True)
else self._speaker_passive_format else self._settings.speaker_passive_format
).format( ).format(
**{ **{
"speaker_id": segment.get("speaker_id", "UU"), "speaker_id": segment.get("speaker_id", "UU"),

2
src/pipecat/services/speechmatics/tts.py
View File

@@ -95,7 +95,7 @@ class SpeechmaticsTTSService(TTSService):
self._params = params or SpeechmaticsTTSService.InputParams() self._params = params or SpeechmaticsTTSService.InputParams()
# Set voice from constructor parameter # Set voice from constructor parameter
self.set_voice(voice_id) self._voice_id = voice_id
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Check if this service can generate processing metrics. """Check if this service can generate processing metrics.

52
src/pipecat/services/stt_service.py
View File

@@ -34,6 +34,7 @@ from pipecat.frames.frames import (
from pipecat.metrics.metrics import TTFBMetricsData from pipecat.metrics.metrics import TTFBMetricsData
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_service import AIService from pipecat.services.ai_service import AIService
from pipecat.services.settings import ServiceSettings, STTSettings
from pipecat.services.stt_latency import DEFAULT_TTFS_P99 from pipecat.services.stt_latency import DEFAULT_TTFS_P99
from pipecat.services.websocket_service import WebsocketService from pipecat.services.websocket_service import WebsocketService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
@@ -101,7 +102,6 @@ class STTService(AIService):
self._audio_passthrough = audio_passthrough self._audio_passthrough = audio_passthrough
self._init_sample_rate = sample_rate self._init_sample_rate = sample_rate
self._sample_rate = 0 self._sample_rate = 0
self._settings: Dict[str, Any] = {}
self._tracing_enabled: bool = False self._tracing_enabled: bool = False
self._muted: bool = False self._muted: bool = False
self._user_id: str = "" self._user_id: str = ""
@@ -166,18 +166,36 @@ class STTService(AIService):
async def set_model(self, model: str): async def set_model(self, model: str):
"""Set the speech recognition model. """Set the speech recognition model.
When the service has been migrated to typed settings this routes
through :meth:`_update_settings_from_typed` so that concrete
services can react (e.g. reconnect) in a single place.
Args: Args:
model: The name of the model to use for speech recognition. model: The name of the model to use for speech recognition.
""" """
self.set_model_name(model) logger.info(f"Switching STT model to: [{model}]")
if isinstance(self._settings, ServiceSettings):
settings_cls = type(self._settings)
await self._update_settings_from_typed(settings_cls(model=model))
else:
self.set_model_name(model)
async def set_language(self, language: Language): async def set_language(self, language: Language):
"""Set the language for speech recognition. """Set the language for speech recognition.
When the service has been migrated to typed settings this routes
through :meth:`_update_settings_from_typed` so that concrete
services can react (e.g. reconnect) in a single place.
Args: Args:
language: The language to use for speech recognition. language: The language to use for speech recognition.
""" """
pass logger.info(f"Switching STT language to: [{language}]")
if isinstance(self._settings, ServiceSettings):
settings_cls = type(self._settings)
await self._update_settings_from_typed(settings_cls(language=language))
else:
pass
@abstractmethod @abstractmethod
async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]: async def run_stt(self, audio: bytes) -> AsyncGenerator[Frame, None]:
@@ -224,6 +242,23 @@ class STTService(AIService):
else: else:
logger.warning(f"Unknown setting for STT service: {key}") logger.warning(f"Unknown setting for STT service: {key}")
async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Apply a typed STT settings update.
Handles ``model`` (via parent). Does **not** call ``set_language``
— concrete services should override this method and handle language
changes (including any reconnect logic) based on the returned
changed-field set.
Args:
update: A typed STT settings delta.
Returns:
Set of field names whose values actually changed.
"""
changed = await super()._update_settings_from_typed(update)
return changed
async def process_audio_frame(self, frame: AudioRawFrame, direction: FrameDirection): async def process_audio_frame(self, frame: AudioRawFrame, direction: FrameDirection):
"""Process an audio frame for speech recognition. """Process an audio frame for speech recognition.
@@ -285,7 +320,16 @@ class STTService(AIService):
await self._handle_vad_user_stopped_speaking(frame) await self._handle_vad_user_stopped_speaking(frame)
await self.push_frame(frame, direction) await self.push_frame(frame, direction)
elif isinstance(frame, STTUpdateSettingsFrame): elif isinstance(frame, STTUpdateSettingsFrame):
await self._update_settings(frame.settings) # New path: typed settings update object.
if frame.update is not None:
await self._update_settings_from_typed(frame.update)
# Legacy path: plain dict, but service uses typed settings — convert.
elif isinstance(self._settings, ServiceSettings):
update = type(self._settings).from_mapping(frame.settings)
await self._update_settings_from_typed(update)
# Legacy path: plain dict, service still uses dict-based settings.
else:
await self._update_settings(frame.settings)
elif isinstance(frame, STTMuteFrame): elif isinstance(frame, STTMuteFrame):
self._muted = frame.mute self._muted = frame.mute
logger.debug(f"STT service {'muted' if frame.mute else 'unmuted'}") logger.debug(f"STT service {'muted' if frame.mute else 'unmuted'}")

72
src/pipecat/services/tts_service.py
View File

@@ -52,6 +52,7 @@ from pipecat.frames.frames import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_service import AIService from pipecat.services.ai_service import AIService
from pipecat.services.settings import ServiceSettings, TTSSettings, is_given
from pipecat.services.websocket_service import WebsocketService from pipecat.services.websocket_service import WebsocketService
from pipecat.transcriptions.language import Language from pipecat.transcriptions.language import Language
from pipecat.utils.text.base_text_aggregator import BaseTextAggregator from pipecat.utils.text.base_text_aggregator import BaseTextAggregator
@@ -189,7 +190,6 @@ class TTSService(AIService):
self._init_sample_rate = sample_rate self._init_sample_rate = sample_rate
self._sample_rate = 0 self._sample_rate = 0
self._voice_id: str = "" self._voice_id: str = ""
self._settings: Dict[str, Any] = {}
self._text_aggregator: BaseTextAggregator = text_aggregator or SimpleTextAggregator() self._text_aggregator: BaseTextAggregator = text_aggregator or SimpleTextAggregator()
if text_aggregator: if text_aggregator:
import warnings import warnings
@@ -263,18 +263,40 @@ class TTSService(AIService):
async def set_model(self, model: str): async def set_model(self, model: str):
"""Set the TTS model to use. """Set the TTS model to use.
When the service has been migrated to typed settings this routes
through :meth:`_update_settings_from_typed` so that concrete
services can react (e.g. reconnect) in a single place.
Args: Args:
model: The name of the TTS model. model: The name of the TTS model.
""" """
self.set_model_name(model) logger.info(f"Switching TTS model to: [{model}]")
if isinstance(self._settings, ServiceSettings):
settings_cls = type(self._settings)
await self._update_settings_from_typed(settings_cls(model=model))
else:
self.set_model_name(model)
def set_voice(self, voice: str): async def set_voice(self, voice: str):
"""Set the voice for speech synthesis. """Set the voice for speech synthesis.
When the service has been migrated to typed settings this routes
through :meth:`_update_settings_from_typed` so that concrete
services can react (e.g. reconnect) in a single place.
.. versionchanged:: 0.0.103
Now ``async``. In ``__init__`` methods, set
``self._voice_id`` directly instead of calling this method.
Args: Args:
voice: The voice identifier or name. voice: The voice identifier or name.
""" """
self._voice_id = voice logger.info(f"Switching TTS voice to: [{voice}]")
if isinstance(self._settings, ServiceSettings):
settings_cls = type(self._settings)
await self._update_settings_from_typed(settings_cls(voice=voice))
else:
self._voice_id = voice
def create_context_id(self) -> str: def create_context_id(self) -> str:
"""Generate a unique context ID for a TTS request. """Generate a unique context ID for a TTS request.
@@ -416,13 +438,42 @@ class TTSService(AIService):
elif key == "model": elif key == "model":
self.set_model_name(value) self.set_model_name(value)
elif key == "voice" or key == "voice_id": elif key == "voice" or key == "voice_id":
self.set_voice(value) self._voice_id = value
elif key == "text_filter": elif key == "text_filter":
for filter in self._text_filters: for filter in self._text_filters:
await filter.update_settings(value) await filter.update_settings(value)
else: else:
logger.warning(f"Unknown setting for TTS service: {key}") logger.warning(f"Unknown setting for TTS service: {key}")
async def _update_settings_from_typed(self, update: TTSSettings) -> set[str]:
"""Apply a typed TTS settings update.
Handles ``model`` (via parent) and syncs ``_voice_id`` when voice
changes. Translates language values before applying. Does **not**
call ``set_voice`` or ``set_model`` directly — concrete services
should override this method and handle reconnect logic based on the
returned changed-field set.
Args:
update: A typed TTS settings delta.
Returns:
Set of field names whose values actually changed.
"""
# Translate language *before* applying so the stored value is canonical
if is_given(update.language) and update.language is not None:
converted = self.language_to_service_language(update.language)
if converted is not None:
update.language = converted
changed = await super()._update_settings_from_typed(update)
# Keep _voice_id in sync for code that reads it directly
if "voice" in changed and isinstance(self._settings, TTSSettings):
self._voice_id = self._settings.voice
return changed
async def say(self, text: str): async def say(self, text: str):
"""Immediately speak the provided text. """Immediately speak the provided text.
@@ -504,7 +555,16 @@ class TTSService(AIService):
await self.flush_audio() await self.flush_audio()
self._processing_text = processing_text self._processing_text = processing_text
elif isinstance(frame, TTSUpdateSettingsFrame): elif isinstance(frame, TTSUpdateSettingsFrame):
await self._update_settings(frame.settings) # New path: typed settings update object.
if frame.update is not None:
await self._update_settings_from_typed(frame.update)
# Legacy path: plain dict, but service uses typed settings — convert.
elif isinstance(self._settings, ServiceSettings):
update = type(self._settings).from_mapping(frame.settings)
await self._update_settings_from_typed(update)
# Legacy path: plain dict, service still uses dict-based settings.
else:
await self._update_settings(frame.settings)
elif isinstance(frame, BotStoppedSpeakingFrame): elif isinstance(frame, BotStoppedSpeakingFrame):
await self._maybe_resume_frame_processing() await self._maybe_resume_frame_processing()
await self.push_frame(frame, direction) await self.push_frame(frame, direction)

24
src/pipecat/services/ultravox/llm.py
View File

@@ -15,6 +15,7 @@ import asyncio
import datetime import datetime
import json import json
import uuid import uuid
from dataclasses import dataclass, field
from typing import Any, Dict, List, Literal, Optional, Union from typing import Any, Dict, List, Literal, Optional, Union
import aiohttp import aiohttp
@@ -34,7 +35,6 @@ from pipecat.frames.frames import (
LLMFullResponseEndFrame, LLMFullResponseEndFrame,
LLMFullResponseStartFrame, LLMFullResponseStartFrame,
LLMTextFrame, LLMTextFrame,
LLMUpdateSettingsFrame,
StartFrame, StartFrame,
TranscriptionFrame, TranscriptionFrame,
TTSAudioRawFrame, TTSAudioRawFrame,
@@ -56,6 +56,7 @@ from pipecat.processors.aggregators.openai_llm_context import (
) )
from pipecat.processors.frame_processor import FrameDirection from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import FunctionCallFromLLM, LLMService from pipecat.services.llm_service import FunctionCallFromLLM, LLMService
from pipecat.services.settings import NOT_GIVEN, LLMSettings
from pipecat.utils.time import time_now_iso8601 from pipecat.utils.time import time_now_iso8601
try: try:
@@ -66,6 +67,17 @@ except ModuleNotFoundError as e:
raise Exception(f"Missing module: {e}") raise Exception(f"Missing module: {e}")
@dataclass
class UltravoxRealtimeLLMSettings(LLMSettings):
"""Settings for UltravoxRealtimeLLMService.
Parameters:
output_medium: The output medium for the model ("voice" or "text").
"""
output_medium: str = field(default=NOT_GIVEN)
class AgentInputParams(BaseModel): class AgentInputParams(BaseModel):
"""Input parameters for Ultravox Realtime generation using a pre-defined Agent. """Input parameters for Ultravox Realtime generation using a pre-defined Agent.
@@ -163,6 +175,7 @@ class UltravoxRealtimeLLMService(LLMService):
**kwargs: Additional arguments passed to parent LLMService. **kwargs: Additional arguments passed to parent LLMService.
""" """
super().__init__(**kwargs) super().__init__(**kwargs)
self._settings = UltravoxRealtimeLLMSettings()
self._params = params self._params = params
if one_shot_selected_tools: if one_shot_selected_tools:
if not isinstance(self._params, OneShotInputParams): if not isinstance(self._params, OneShotInputParams):
@@ -310,6 +323,12 @@ class UltravoxRealtimeLLMService(LLMService):
await self.cancel_task(self._receive_task, timeout=1.0) await self.cancel_task(self._receive_task, timeout=1.0)
self._receive_task = None self._receive_task = None
async def _update_settings_from_typed(self, update: UltravoxRealtimeLLMSettings):
changed = await super()._update_settings_from_typed(update)
if "output_medium" in changed:
await self._update_output_medium(self._settings.output_medium)
return changed
# #
# frame processing # frame processing
# StartFrame, StopFrame, CancelFrame implemented in base class # StartFrame, StopFrame, CancelFrame implemented in base class
@@ -331,9 +350,6 @@ class UltravoxRealtimeLLMService(LLMService):
else LLMContext.from_openai_context(frame.context) else LLMContext.from_openai_context(frame.context)
) )
await self._handle_context(context) await self._handle_context(context)
elif isinstance(frame, LLMUpdateSettingsFrame):
if "output_medium" in frame.settings:
await self._update_output_medium(frame.settings.get("output_medium"))
elif isinstance(frame, InputTextRawFrame): elif isinstance(frame, InputTextRawFrame):
await self._send_user_text(frame.text) await self._send_user_text(frame.text)
await self.push_frame(frame, direction) await self.push_frame(frame, direction)

61
src/pipecat/services/whisper/base_stt.py
View File

@@ -10,6 +10,7 @@ This module provides common functionality for services implementing the Whisper
interface, including language mapping, metrics generation, and error handling. interface, including language mapping, metrics generation, and error handling.
""" """
from dataclasses import dataclass, field
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
from loguru import logger from loguru import logger
@@ -17,6 +18,7 @@ from openai import AsyncOpenAI
from openai.types.audio import Transcription from openai.types.audio import Transcription
from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_latency import WHISPER_TTFS_P99 from pipecat.services.stt_latency import WHISPER_TTFS_P99
from pipecat.services.stt_service import SegmentedSTTService from pipecat.services.stt_service import SegmentedSTTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
@@ -24,6 +26,22 @@ from pipecat.utils.time import time_now_iso8601
from pipecat.utils.tracing.service_decorators import traced_stt from pipecat.utils.tracing.service_decorators import traced_stt
@dataclass
class BaseWhisperSTTSettings(STTSettings):
"""Typed settings for Whisper API-based STT services.
Parameters:
base_url: API base URL.
prompt: Optional text to guide the model's style or continue
a previous segment.
temperature: Sampling temperature between 0 and 1.
"""
base_url: Optional[str] = field(default_factory=lambda: NOT_GIVEN)
prompt: Optional[str] = field(default_factory=lambda: NOT_GIVEN)
temperature: Optional[float] = field(default_factory=lambda: NOT_GIVEN)
def language_to_whisper_language(language: Language) -> Optional[str]: def language_to_whisper_language(language: Language) -> Optional[str]:
"""Maps pipecat Language enum to Whisper API language codes. """Maps pipecat Language enum to Whisper API language codes.
@@ -143,26 +161,36 @@ class BaseWhisperSTTService(SegmentedSTTService):
self._temperature = temperature self._temperature = temperature
self._include_prob_metrics = include_prob_metrics self._include_prob_metrics = include_prob_metrics
self._settings = { self._settings: BaseWhisperSTTSettings = BaseWhisperSTTSettings(
"base_url": base_url, model=model,
"language": self._language, language=self._language,
"prompt": self._prompt, base_url=base_url,
"temperature": self._temperature, prompt=self._prompt,
} temperature=self._temperature,
)
def _create_client(self, api_key: Optional[str], base_url: Optional[str]): def _create_client(self, api_key: Optional[str], base_url: Optional[str]):
return AsyncOpenAI(api_key=api_key, base_url=base_url) return AsyncOpenAI(api_key=api_key, base_url=base_url)
async def set_model(self, model: str): async def _update_settings_from_typed(self, update: STTSettings) -> set[str]:
"""Set the model name for transcription. """Apply a typed settings update, syncing instance variables.
Args: Keeps ``_language``, ``_prompt``, and ``_temperature`` in sync with
model: The name of the model to use. the typed settings fields.
""" """
self.set_model_name(model) changed = await super()._update_settings_from_typed(update)
if "language" in changed:
self._language = self.language_to_service_language(Language(self._settings.language))
if "prompt" in changed:
self._prompt = self._settings.prompt
if "temperature" in changed:
self._temperature = self._settings.temperature
return changed
def can_generate_metrics(self) -> bool: def can_generate_metrics(self) -> bool:
"""Indicates whether this service can generate metrics. """Whether this service can generate processing metrics.
Returns: Returns:
bool: True, as this service supports metric generation. bool: True, as this service supports metric generation.
@@ -180,15 +208,6 @@ class BaseWhisperSTTService(SegmentedSTTService):
""" """
return language_to_whisper_language(language) return language_to_whisper_language(language)
async def set_language(self, language: Language):
"""Set the language for transcription.
Args:
language: The Language enum value to use for transcription.
"""
logger.info(f"Switching STT language to: [{language}]")
self._language = self.language_to_service_language(language)
@traced_stt @traced_stt
async def _handle_transcription( async def _handle_transcription(
self, transcript: str, is_final: bool, language: Optional[Language] = None self, transcript: str, is_final: bool, language: Optional[Language] = None

79
src/pipecat/services/whisper/stt.py
View File

@@ -11,6 +11,7 @@ supporting both Faster Whisper and MLX Whisper backends for efficient inference.
""" """
import asyncio import asyncio
from dataclasses import dataclass, field
from enum import Enum from enum import Enum
from typing import AsyncGenerator, Optional from typing import AsyncGenerator, Optional
@@ -19,6 +20,7 @@ from loguru import logger
from typing_extensions import TYPE_CHECKING, override from typing_extensions import TYPE_CHECKING, override
from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame from pipecat.frames.frames import ErrorFrame, Frame, TranscriptionFrame
from pipecat.services.settings import NOT_GIVEN, STTSettings
from pipecat.services.stt_service import SegmentedSTTService from pipecat.services.stt_service import SegmentedSTTService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.time import time_now_iso8601 from pipecat.utils.time import time_now_iso8601
@@ -172,6 +174,36 @@ def language_to_whisper_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class WhisperSTTSettings(STTSettings):
"""Typed settings for the local Whisper (Faster Whisper) STT service.
Parameters:
device: Inference device ('cpu', 'cuda', or 'auto').
compute_type: Compute type for inference ('default', 'int8', etc.).
no_speech_prob: Probability threshold for filtering non-speech segments.
"""
device: str = field(default_factory=lambda: NOT_GIVEN)
compute_type: str = field(default_factory=lambda: NOT_GIVEN)
no_speech_prob: float = field(default_factory=lambda: NOT_GIVEN)
@dataclass
class WhisperMLXSTTSettings(STTSettings):
"""Typed settings for the MLX Whisper STT service.
Parameters:
no_speech_prob: Probability threshold for filtering non-speech segments.
temperature: Sampling temperature (0.0-1.0).
engine: Whisper engine identifier.
"""
no_speech_prob: float = field(default_factory=lambda: NOT_GIVEN)
temperature: float = field(default_factory=lambda: NOT_GIVEN)
engine: str = field(default_factory=lambda: NOT_GIVEN)
class WhisperSTTService(SegmentedSTTService): class WhisperSTTService(SegmentedSTTService):
"""Class to transcribe audio with a locally-downloaded Whisper model. """Class to transcribe audio with a locally-downloaded Whisper model.
@@ -206,12 +238,13 @@ class WhisperSTTService(SegmentedSTTService):
self._no_speech_prob = no_speech_prob self._no_speech_prob = no_speech_prob
self._model: Optional[WhisperModel] = None self._model: Optional[WhisperModel] = None
self._settings = { self._settings: WhisperSTTSettings = WhisperSTTSettings(
"language": language, model=model if isinstance(model, str) else model.value,
"device": self._device, language=language,
"compute_type": self._compute_type, device=self._device,
"no_speech_prob": self._no_speech_prob, compute_type=self._compute_type,
} no_speech_prob=self._no_speech_prob,
)
self._load() self._load()
@@ -234,15 +267,6 @@ class WhisperSTTService(SegmentedSTTService):
""" """
return language_to_whisper_language(language) return language_to_whisper_language(language)
async def set_language(self, language: Language):
"""Set the language for transcription.
Args:
language: The Language enum value to use for transcription.
"""
logger.info(f"Switching STT language to: [{language}]")
self._settings["language"] = language
def _load(self): def _load(self):
"""Loads the Whisper model. """Loads the Whisper model.
@@ -293,7 +317,7 @@ class WhisperSTTService(SegmentedSTTService):
# Divide by 32768 because we have signed 16-bit data. # Divide by 32768 because we have signed 16-bit data.
audio_float = np.frombuffer(audio, dtype=np.int16).astype(np.float32) / 32768.0 audio_float = np.frombuffer(audio, dtype=np.int16).astype(np.float32) / 32768.0
whisper_lang = self.language_to_service_language(self._settings["language"]) whisper_lang = self.language_to_service_language(self._settings.language)
segments, _ = await asyncio.to_thread( segments, _ = await asyncio.to_thread(
self._model.transcribe, audio_float, language=whisper_lang self._model.transcribe, audio_float, language=whisper_lang
) )
@@ -305,13 +329,13 @@ class WhisperSTTService(SegmentedSTTService):
await self.stop_processing_metrics() await self.stop_processing_metrics()
if text: if text:
await self._handle_transcription(text, True, self._settings["language"]) await self._handle_transcription(text, True, self._settings.language)
logger.debug(f"Transcription: [{text}]") logger.debug(f"Transcription: [{text}]")
yield TranscriptionFrame( yield TranscriptionFrame(
text, text,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._settings["language"], self._settings.language,
) )
@@ -347,12 +371,13 @@ class WhisperSTTServiceMLX(WhisperSTTService):
self._no_speech_prob = no_speech_prob self._no_speech_prob = no_speech_prob
self._temperature = temperature self._temperature = temperature
self._settings = { self._settings: WhisperMLXSTTSettings = WhisperMLXSTTSettings(
"language": language, model=model if isinstance(model, str) else model.value,
"no_speech_prob": self._no_speech_prob, language=language,
"temperature": self._temperature, no_speech_prob=self._no_speech_prob,
"engine": "mlx", temperature=self._temperature,
} engine="mlx",
)
# No need to call _load() as MLX Whisper loads models on demand # No need to call _load() as MLX Whisper loads models on demand
@@ -390,7 +415,7 @@ class WhisperSTTServiceMLX(WhisperSTTService):
# Divide by 32768 because we have signed 16-bit data. # Divide by 32768 because we have signed 16-bit data.
audio_float = np.frombuffer(audio, dtype=np.int16).astype(np.float32) / 32768.0 audio_float = np.frombuffer(audio, dtype=np.int16).astype(np.float32) / 32768.0
whisper_lang = self.language_to_service_language(self._settings["language"]) whisper_lang = self.language_to_service_language(self._settings.language)
chunk = await asyncio.to_thread( chunk = await asyncio.to_thread(
mlx_whisper.transcribe, mlx_whisper.transcribe,
audio_float, audio_float,
@@ -413,13 +438,13 @@ class WhisperSTTServiceMLX(WhisperSTTService):
await self.stop_processing_metrics() await self.stop_processing_metrics()
if text: if text:
await self._handle_transcription(text, True, self._settings["language"]) await self._handle_transcription(text, True, self._settings.language)
logger.debug(f"Transcription: [{text}]") logger.debug(f"Transcription: [{text}]")
yield TranscriptionFrame( yield TranscriptionFrame(
text, text,
self._user_id, self._user_id,
time_now_iso8601(), time_now_iso8601(),
self._settings["language"], self._settings.language,
) )
except Exception as e: except Exception as e:

32
src/pipecat/services/xtts/tts.py
View File

@@ -10,7 +10,8 @@ This module provides integration with Coqui XTTS streaming server for
text-to-speech synthesis using local Docker deployment. text-to-speech synthesis using local Docker deployment.
""" """
from typing import Any, AsyncGenerator, Dict, Optional from dataclasses import dataclass, field
from typing import AsyncGenerator, Dict, Optional
import aiohttp import aiohttp
from loguru import logger from loguru import logger
@@ -24,6 +25,7 @@ from pipecat.frames.frames import (
TTSStartedFrame, TTSStartedFrame,
TTSStoppedFrame, TTSStoppedFrame,
) )
from pipecat.services.settings import NOT_GIVEN, TTSSettings
from pipecat.services.tts_service import TTSService from pipecat.services.tts_service import TTSService
from pipecat.transcriptions.language import Language, resolve_language from pipecat.transcriptions.language import Language, resolve_language
from pipecat.utils.tracing.service_decorators import traced_tts from pipecat.utils.tracing.service_decorators import traced_tts
@@ -68,6 +70,17 @@ def language_to_xtts_language(language: Language) -> Optional[str]:
return resolve_language(language, LANGUAGE_MAP, use_base_code=True) return resolve_language(language, LANGUAGE_MAP, use_base_code=True)
@dataclass
class XTTSTTSSettings(TTSSettings):
"""Typed settings for XTTS TTS service.
Parameters:
base_url: Base URL of the XTTS streaming server.
"""
base_url: str = field(default_factory=lambda: NOT_GIVEN)
class XTTSService(TTSService): class XTTSService(TTSService):
"""Coqui XTTS text-to-speech service. """Coqui XTTS text-to-speech service.
@@ -98,11 +111,12 @@ class XTTSService(TTSService):
""" """
super().__init__(sample_rate=sample_rate, **kwargs) super().__init__(sample_rate=sample_rate, **kwargs)
self._settings = { self._settings: XTTSTTSSettings = XTTSTTSSettings(
"language": self.language_to_service_language(language), voice=voice_id,
"base_url": base_url, language=self.language_to_service_language(language),
} base_url=base_url,
self.set_voice(voice_id) )
self._voice_id = voice_id
self._studio_speakers: Optional[Dict[str, Any]] = None self._studio_speakers: Optional[Dict[str, Any]] = None
self._aiohttp_session = aiohttp_session self._aiohttp_session = aiohttp_session
@@ -138,7 +152,7 @@ class XTTSService(TTSService):
if self._studio_speakers: if self._studio_speakers:
return return
async with self._aiohttp_session.get(self._settings["base_url"] + "/studio_speakers") as r: async with self._aiohttp_session.get(self._settings.base_url + "/studio_speakers") as r:
if r.status != 200: if r.status != 200:
text = await r.text() text = await r.text()
await self.push_error( await self.push_error(
@@ -166,11 +180,11 @@ class XTTSService(TTSService):
embeddings = self._studio_speakers[self._voice_id] embeddings = self._studio_speakers[self._voice_id]
url = self._settings["base_url"] + "/tts_stream" url = self._settings.base_url + "/tts_stream"
payload = { payload = {
"text": text.replace(".", "").replace("*", ""), "text": text.replace(".", "").replace("*", ""),
"language": self._settings["language"], "language": self._settings.language,
"speaker_embedding": embeddings["speaker_embedding"], "speaker_embedding": embeddings["speaker_embedding"],
"gpt_cond_latent": embeddings["gpt_cond_latent"], "gpt_cond_latent": embeddings["gpt_cond_latent"],
"add_wav_header": False, "add_wav_header": False,

308
tests/test_settings.py Normal file
View File

@@ -0,0 +1,308 @@
#
# Copyright (c) 2024-2026, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""Tests for the typed settings infrastructure in pipecat.services.settings."""
import pytest
from pipecat.services.settings import (
NOT_GIVEN,
LLMSettings,
ServiceSettings,
STTSettings,
TTSSettings,
_NotGiven,
is_given,
)
# ---------------------------------------------------------------------------
# NOT_GIVEN sentinel
# ---------------------------------------------------------------------------
class TestNotGiven:
def test_singleton(self):
"""NOT_GIVEN is a singleton — every reference is the same object."""
assert _NotGiven() is _NotGiven()
assert NOT_GIVEN is _NotGiven()
def test_repr(self):
assert repr(NOT_GIVEN) == "NOT_GIVEN"
def test_bool_is_false(self):
assert not NOT_GIVEN
assert bool(NOT_GIVEN) is False
def test_is_given_with_not_given(self):
assert is_given(NOT_GIVEN) is False
def test_is_given_with_none(self):
assert is_given(None) is True
def test_is_given_with_values(self):
assert is_given(0) is True
assert is_given("") is True
assert is_given(False) is True
assert is_given(42) is True
assert is_given("hello") is True
# ---------------------------------------------------------------------------
# ServiceSettings base
# ---------------------------------------------------------------------------
class TestServiceSettings:
def test_default_fields_are_not_given(self):
s = ServiceSettings()
assert not is_given(s.model)
assert s.extra == {}
def test_given_fields_empty_by_default(self):
s = ServiceSettings()
assert s.given_fields() == {}
def test_given_fields_includes_set_values(self):
s = ServiceSettings(model="gpt-4o")
assert s.given_fields() == {"model": "gpt-4o"}
def test_given_fields_includes_extra(self):
s = ServiceSettings(model="gpt-4o")
s.extra = {"custom_key": 42}
result = s.given_fields()
assert result == {"model": "gpt-4o", "custom_key": 42}
def test_to_dict(self):
s = ServiceSettings(model="gpt-4o")
assert s.to_dict() == {"model": "gpt-4o"}
def test_copy_is_deep(self):
s = ServiceSettings(model="gpt-4o")
s.extra = {"nested": {"a": 1}}
c = s.copy()
assert c.model == "gpt-4o"
assert c.extra == {"nested": {"a": 1}}
# Mutating the copy shouldn't affect the original
c.extra["nested"]["a"] = 999
assert s.extra["nested"]["a"] == 1
# ---------------------------------------------------------------------------
# apply_update
# ---------------------------------------------------------------------------
class TestApplyUpdate:
def test_apply_update_basic(self):
current = TTSSettings(voice="alice", language="en")
delta = TTSSettings(voice="bob")
changed = current.apply_update(delta)
assert changed == {"voice"}
assert current.voice == "bob"
assert current.language == "en"
def test_apply_update_no_change(self):
current = TTSSettings(voice="alice", language="en")
delta = TTSSettings(voice="alice")
changed = current.apply_update(delta)
assert changed == set()
assert current.voice == "alice"
def test_apply_update_not_given_skipped(self):
current = TTSSettings(voice="alice", language="en")
delta = TTSSettings() # all NOT_GIVEN
changed = current.apply_update(delta)
assert changed == set()
assert current.voice == "alice"
assert current.language == "en"
def test_apply_update_multiple_fields(self):
current = LLMSettings(temperature=0.7, max_tokens=100)
delta = LLMSettings(temperature=0.9, max_tokens=200, top_p=0.95)
changed = current.apply_update(delta)
assert changed == {"temperature", "max_tokens", "top_p"}
assert current.temperature == 0.9
assert current.max_tokens == 200
assert current.top_p == 0.95
def test_apply_update_extra_merged(self):
current = TTSSettings(voice="alice")
current.extra = {"speed": 1.0, "stability": 0.5}
delta = TTSSettings()
delta.extra = {"speed": 1.2}
changed = current.apply_update(delta)
assert "speed" in changed
assert current.extra == {"speed": 1.2, "stability": 0.5}
def test_apply_update_extra_no_change(self):
current = TTSSettings(voice="alice")
current.extra = {"speed": 1.0}
delta = TTSSettings()
delta.extra = {"speed": 1.0}
changed = current.apply_update(delta)
assert changed == set()
def test_apply_update_model_field(self):
current = ServiceSettings(model="old-model")
delta = ServiceSettings(model="new-model")
changed = current.apply_update(delta)
assert changed == {"model"}
assert current.model == "new-model"
def test_apply_update_none_is_a_valid_value(self):
"""Setting a field to None should be treated as a change from NOT_GIVEN."""
current = TTSSettings()
delta = TTSSettings(language=None)
changed = current.apply_update(delta)
assert "language" in changed
assert current.language is None
def test_apply_update_none_to_value(self):
current = TTSSettings(language=None)
delta = TTSSettings(language="en")
changed = current.apply_update(delta)
assert "language" in changed
assert current.language == "en"
# ---------------------------------------------------------------------------
# from_mapping
# ---------------------------------------------------------------------------
class TestFromMapping:
def test_basic_mapping(self):
s = TTSSettings.from_mapping({"voice": "alice", "language": "en"})
assert s.voice == "alice"
assert s.language == "en"
assert not is_given(s.model)
def test_alias_resolution(self):
"""'voice_id' is an alias for 'voice' in TTSSettings."""
s = TTSSettings.from_mapping({"voice_id": "alice"})
assert s.voice == "alice"
def test_unknown_keys_go_to_extra(self):
s = TTSSettings.from_mapping({"voice": "alice", "speed": 1.2, "stability": 0.5})
assert s.voice == "alice"
assert s.extra == {"speed": 1.2, "stability": 0.5}
def test_model_field(self):
s = LLMSettings.from_mapping({"model": "gpt-4o", "temperature": 0.7})
assert s.model == "gpt-4o"
assert s.temperature == 0.7
def test_empty_mapping(self):
s = ServiceSettings.from_mapping({})
assert s.given_fields() == {}
def test_all_unknown_keys(self):
s = ServiceSettings.from_mapping({"foo": 1, "bar": 2})
assert not is_given(s.model)
assert s.extra == {"foo": 1, "bar": 2}
def test_llm_settings_from_mapping(self):
s = LLMSettings.from_mapping({"temperature": 0.5, "max_tokens": 1000, "custom_param": True})
assert s.temperature == 0.5
assert s.max_tokens == 1000
assert s.extra == {"custom_param": True}
def test_stt_settings_from_mapping(self):
s = STTSettings.from_mapping({"language": "fr", "model": "whisper-large"})
assert s.language == "fr"
assert s.model == "whisper-large"
# ---------------------------------------------------------------------------
# LLMSettings specifics
# ---------------------------------------------------------------------------
class TestLLMSettings:
def test_all_fields_not_given_by_default(self):
s = LLMSettings()
for name in (
"model",
"temperature",
"max_tokens",
"top_p",
"top_k",
"frequency_penalty",
"presence_penalty",
"seed",
):
assert not is_given(getattr(s, name)), f"{name} should be NOT_GIVEN"
def test_given_fields(self):
s = LLMSettings(temperature=0.7, seed=42)
assert s.given_fields() == {"temperature": 0.7, "seed": 42}
# ---------------------------------------------------------------------------
# TTSSettings specifics
# ---------------------------------------------------------------------------
class TestTTSSettings:
def test_all_fields_not_given_by_default(self):
s = TTSSettings()
for name in ("model", "voice", "language"):
assert not is_given(getattr(s, name)), f"{name} should be NOT_GIVEN"
def test_aliases_class_var(self):
assert TTSSettings._aliases == {"voice_id": "voice"}
def test_given_fields(self):
s = TTSSettings(voice="alice")
assert s.given_fields() == {"voice": "alice"}
# ---------------------------------------------------------------------------
# STTSettings specifics
# ---------------------------------------------------------------------------
class TestSTTSettings:
def test_all_fields_not_given_by_default(self):
s = STTSettings()
for name in ("model", "language"):
assert not is_given(getattr(s, name)), f"{name} should be NOT_GIVEN"
def test_given_fields(self):
s = STTSettings(language="en", model="whisper-large")
assert s.given_fields() == {"language": "en", "model": "whisper-large"}
# ---------------------------------------------------------------------------
# Integration: roundtrip from_mapping → apply_update
# ---------------------------------------------------------------------------
class TestRoundtrip:
def test_from_mapping_then_apply_update(self):
"""Simulate the real flow: dict arrives via frame, gets converted, applied."""
# Simulating current service state
current = TTSSettings(model="eleven_turbo_v2_5", voice="alice", language="en")
current.extra = {"stability": 0.5, "speed": 1.0}
# Incoming dict-based update
raw = {"voice_id": "bob", "speed": 1.2}
delta = TTSSettings.from_mapping(raw)
changed = current.apply_update(delta)
assert changed == {"voice", "speed"}
assert current.voice == "bob"
assert current.language == "en"
assert current.extra["speed"] == 1.2
assert current.extra["stability"] == 0.5
def test_from_mapping_preserves_model(self):
current = LLMSettings(model="gpt-4o", temperature=0.7)
delta = LLMSettings.from_mapping({"model": "gpt-4o-mini", "temperature": 0.9})
changed = current.apply_update(delta)
assert changed == {"model", "temperature"}
assert current.model == "gpt-4o-mini"
assert current.temperature == 0.9