wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #1370 from pipecat-ai/aleix/minor-ultravox-updates

Browse Source 
services(ultravox): CHANGELOG, formatting and minor changes
This commit is contained in:
Aleix Conchillo Flaqué
2025-03-13 12:05:13 -07:00
committed by GitHub
parent b426be3067 87004937be
commit c0c7c5d600
5 changed files with 120 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Added new `UltravoxSTTService`.
(see https://github.com/fixie-ai/ultravox)
- Added `on_frame_reached_upstream` and `on_frame_reached_downstream` event
handlers to `PipelineTask`. Those events will be called when a frame reaches
the beginning or end of the pipeline respectively.

4
README.md
View File

@@ -56,8 +56,8 @@ pip install "pipecat-ai[option,...]"
### Available services
| Category | Services | Install Command Example |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | `pip install "pipecat-ai[deepgram]"` |
|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/server/services/stt/assemblyai), [Azure](https://docs.pipecat.ai/server/services/stt/azure), [Deepgram](https://docs.pipecat.ai/server/services/stt/deepgram), [Gladia](https://docs.pipecat.ai/server/services/stt/gladia), [Google](https://docs.pipecat.ai/server/services/stt/google), [Groq (Whisper)](https://docs.pipecat.ai/server/services/stt/groq), [OpenAI (Whisper)](https://docs.pipecat.ai/server/services/stt/openai), [Ultravox](https://docs.pipecat.ai/server/services/stt/ultravox), [Whisper](https://docs.pipecat.ai/server/services/stt/whisper) | `pip install "pipecat-ai[deepgram]"` |
| LLMs | [Anthropic](https://docs.pipecat.ai/server/services/llm/anthropic), [Azure](https://docs.pipecat.ai/server/services/llm/azure), [Cerebras](https://docs.pipecat.ai/server/services/llm/cerebras), [DeepSeek](https://docs.pipecat.ai/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/server/services/llm/grok), [Groq](https://docs.pipecat.ai/server/services/llm/groq), [NVIDIA NIM](https://docs.pipecat.ai/server/services/llm/nim), [Ollama](https://docs.pipecat.ai/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/server/services/llm/openai), [OpenRouter](https://docs.pipecat.ai/server/services/llm/openrouter), [Perplexity](https://docs.pipecat.ai/server/services/llm/perplexity), [Together AI](https://docs.pipecat.ai/server/services/llm/together) | `pip install "pipecat-ai[openai]"` |
| Text-to-Speech | [AWS](https://docs.pipecat.ai/server/services/tts/aws), [Azure](https://docs.pipecat.ai/server/services/tts/azure), [Cartesia](https://docs.pipecat.ai/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/server/services/tts/elevenlabs), [Fish](https://docs.pipecat.ai/server/services/tts/fish), [Google](https://docs.pipecat.ai/server/services/tts/google), [LMNT](https://docs.pipecat.ai/server/services/tts/lmnt), [OpenAI](https://docs.pipecat.ai/server/services/tts/openai), [PlayHT](https://docs.pipecat.ai/server/services/tts/playht), [Rime](https://docs.pipecat.ai/server/services/tts/rime), [XTTS](https://docs.pipecat.ai/server/services/tts/xtts) | `pip install "pipecat-ai[cartesia]"` |
| Speech-to-Speech | [Gemini Multimodal Live](https://docs.pipecat.ai/server/services/s2s/gemini), [OpenAI Realtime](https://docs.pipecat.ai/server/services/s2s/openai) | `pip install "pipecat-ai[google]"` |

15
examples/foundational/07u-interruptible-ultravox.py
View File

@@ -17,9 +17,9 @@ from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.services.ultravox import UltravoxSTTService
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.services.ultravox import UltravoxSTTService
from pipecat.transports.services.daily import DailyParams, DailyTransport
load_dotenv(override=True)
@@ -30,13 +30,14 @@ load_dotenv(override=True)
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
#Want to initialize the ultravox processor since it takes time to load the model and dont
#want to load it every time the pipeline is run
# Want to initialize the ultravox processor since it takes time to load the model and dont
# want to load it every time the pipeline is run
ultravox_processor = UltravoxSTTService(
model_size="fixie-ai/ultravox-v0_4_1-llama-3_1-8b",
hf_token=os.getenv("HF_TOKEN"),
)
async def main():
async with aiohttp.ClientSession() as session:
(room_url, token) = await configure(session)
@@ -56,11 +57,9 @@ async def main():
tts = CartesiaTTSService(
api_key=os.environ.get("CARTESIA_API_KEY"),
voice_id='97f4b8fb-f2fe-444b-bb9a-c109783a857a',
voice_id="97f4b8fb-f2fe-444b-bb9a-c109783a857a",
)
pipeline = Pipeline(
[
transport.input(), # Transport user input
@@ -89,5 +88,3 @@ async def main():
if __name__ == "__main__":
asyncio.run(main())

1
pyproject.toml
View File

@@ -78,6 +78,7 @@ silero = [ "onnxruntime~=1.20.1" ]
simli = [ "simli-ai~=0.1.10"]
soundfile = [ "soundfile~=0.13.0" ]
together = []
ultravox = [ "transformers~=4.48.0", "vllm~=0.7.3" ]
websocket = [ "websockets~=13.1", "fastapi~=0.115.6" ]
whisper = [ "faster-whisper~=1.1.1" ]
openrouter = []

118
src/pipecat/services/ultravox.py
View File

@@ -1,40 +1,45 @@
#
# Copyright (c) 2024–2025, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""This module implements Ultravox speech-to-text with a locally-loaded model."""
import json
import time
import os
import time
from typing import AsyncGenerator, List, Optional
import numpy as np
from enum import Enum
from typing import AsyncGenerator, Optional, List
from loguru import logger
from pydantic import BaseModel
from huggingface_hub import login
from loguru import logger
from pipecat.frames.frames import (
Frame,
AudioRawFrame,
TranscriptionFrame,
TextFrame,
StartFrame,
EndFrame,
CancelFrame,
EndFrame,
ErrorFrame,
Frame,
StartFrame,
TranscriptionFrame,
UserStartedSpeakingFrame,
UserStoppedSpeakingFrame,
ErrorFrame
)
from pipecat.services.ai_services import AIService
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import AIService
from pipecat.utils.time import time_now_iso8601
try:
from vllm import SamplingParams, AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs
from transformers import AutoTokenizer
from vllm import AsyncLLMEngine, SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs
except ModuleNotFoundError as e:
logger.error(f"Exception: {e}")
logger.error("In order to use Ultravox, you need to `pip install pipecat-ai[ultravox]`.")
raise Exception(f"Missing module: {e}")
class AudioBuffer:
"""Buffer to collect audio frames before processing.
@@ -43,11 +48,13 @@ class AudioBuffer:
started_at: Timestamp when speech started
is_processing: Flag to prevent concurrent processing
"""
def __init__(self):
self.frames: List[AudioRawFrame] = []
self.started_at: Optional[float] = None
self.is_processing: bool = False
class UltravoxModel:
"""Model wrapper for the Ultravox multimodal model.
@@ -62,6 +69,7 @@ class UltravoxModel:
tokenizer: The tokenizer for the model
stop_token_ids: Optional token IDs to stop generation
"""
def __init__(self, model_name: str = "fixie-ai/ultravox-v0_4_1-llama-3_1-8b"):
self.model_name = model_name
self._initialize_engine()
@@ -74,7 +82,7 @@ class UltravoxModel:
model=self.model_name,
gpu_memory_utilization=0.9,
max_model_len=8192,
trust_remote_code=True
trust_remote_code=True,
)
self.engine = AsyncLLMEngine.from_engine_args(engine_args)
@@ -92,12 +100,16 @@ class UltravoxModel:
str: Formatted prompt string
"""
return self.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
messages, tokenize=False, add_generation_prompt=True
)
async def generate(self, messages: list, temperature: float = 0.7, max_tokens: int = 100, audio: np.ndarray = None):
async def generate(
self,
messages: list,
temperature: float = 0.7,
max_tokens: int = 100,
audio: np.ndarray = None,
):
"""Generate text from audio input using the model.
Args:
@@ -110,14 +122,10 @@ class UltravoxModel:
str: JSON chunks of the generated response
"""
sampling_params = SamplingParams(
temperature=temperature,
max_tokens=max_tokens,
stop_token_ids=self.stop_token_ids
temperature=temperature, max_tokens=max_tokens, stop_token_ids=self.stop_token_ids
)
mm_data = {
"audio": audio
}
mm_data = {"audio": audio}
inputs = {"prompt": self.format_prompt(messages), "multi_modal_data": mm_data}
results_generator = self.engine.generate(inputs, sampling_params, str(time.time()))
@@ -126,7 +134,7 @@ class UltravoxModel:
async for output in results_generator:
prompt_output = output.outputs
new_text = prompt_output[0].text[len(previous_text):]
new_text = prompt_output[0].text[len(previous_text) :]
previous_text = prompt_output[0].text
# Construct OpenAI-compatible chunk
@@ -160,6 +168,7 @@ class UltravoxModel:
yield json.dumps(chunk)
class UltravoxSTTService(AIService):
"""Service to transcribe audio using the Ultravox multimodal model.
@@ -180,10 +189,11 @@ class UltravoxSTTService(AIService):
max_tokens: Maximum tokens to generate
_connection_active: Flag indicating if service is active
"""
def __init__(
self,
*,
model_size: str = "fixie-ai/ultravox-v0_4_1-llama-3_1-8b",
model_size: str = "fixie-ai/ultravox-v0_4_1-llama-3_1-8b",
hf_token: Optional[str] = None,
temperature: float = 0.7,
max_tokens: int = 100,
@@ -201,12 +211,12 @@ class UltravoxSTTService(AIService):
# Initialize model
model_name = model_size if isinstance(model_size, str) else model_size.value
self.model = UltravoxModel(model_name=model_name)
self._model = UltravoxModel(model_name=model_name)
# Initialize service state
self.buffer = AudioBuffer()
self.temperature = temperature
self.max_tokens = max_tokens
self._buffer = AudioBuffer()
self._temperature = temperature
self._max_tokens = max_tokens
self._connection_active = False
logger.info(f"Initialized UltravoxSTTService with model: {model_name}")
@@ -247,7 +257,7 @@ class UltravoxSTTService(AIService):
"""
await super().cancel(frame)
self._connection_active = False
self.buffer = AudioBuffer()
self._buffer = AudioBuffer()
logger.info("UltravoxSTTService cancelled")
async def process_frame(self, frame: Frame, direction: FrameDirection):
@@ -263,14 +273,14 @@ class UltravoxSTTService(AIService):
if isinstance(frame, UserStartedSpeakingFrame):
logger.info("Speech started")
self.buffer = AudioBuffer()
self.buffer.started_at = time.time()
self._buffer = AudioBuffer()
self._buffer.started_at = time.time()
elif isinstance(frame, AudioRawFrame) and self.buffer.started_at is not None:
self.buffer.frames.append(frame)
elif isinstance(frame, AudioRawFrame) and self._buffer.started_at is not None:
self._buffer.frames.append(frame)
elif isinstance(frame, UserStoppedSpeakingFrame):
if self.buffer.frames and not self.buffer.is_processing:
if self._buffer.frames and not self._buffer.is_processing:
logger.info("Speech ended, processing buffer...")
await self.process_generator(self._process_audio_buffer())
return # Return early to avoid pushing None frame
@@ -289,18 +299,18 @@ class UltravoxSTTService(AIService):
Frame: TextFrame containing the transcribed text
"""
try:
self.buffer.is_processing = True
self._buffer.is_processing = True
# Check if we have valid frames before processing
if not self.buffer.frames:
if not self._buffer.frames:
logger.warning("No audio frames to process")
yield ErrorFrame("No audio frames to process")
return
# Process audio frames
audio_arrays = []
for f in self.buffer.frames:
if hasattr(f, 'audio') and f.audio:
for f in self._buffer.frames:
if hasattr(f, "audio") and f.audio:
# Handle bytes data - these are int16 PCM samples
if isinstance(f.audio, bytes):
try:
@@ -330,7 +340,7 @@ class UltravoxSTTService(AIService):
audio_data = np.concatenate(audio_arrays)
# Generate text using the model
if self.model:
if self._model:
try:
logger.info("Generating text from audio using model...")
full_response = ""
@@ -339,14 +349,11 @@ class UltravoxSTTService(AIService):
await self.start_ttfb_metrics()
await self.start_processing_metrics()
async for response in self.model.generate(
messages=[{
'role': 'user',
'content': "<|audio|>\n"
}],
temperature=self.temperature,
max_tokens=self.max_tokens,
audio=audio_data
async for response in self._model.generate(
messages=[{"role": "user", "content": "<|audio|>\n"}],
temperature=self._temperature,
max_tokens=self._max_tokens,
audio=audio_data,
):
# Stop TTFB metrics after first response
await self.stop_ttfb_metrics()
@@ -367,9 +374,9 @@ class UltravoxSTTService(AIService):
transcription = full_response.strip()
if transcription:
yield TranscriptionFrame(
user_id="",
text=transcription,
interim_text="",
timestamp=time_now_iso8601()
timestamp=time_now_iso8601(),
)
else:
logger.warning("Empty transcription result")
@@ -385,9 +392,10 @@ class UltravoxSTTService(AIService):
except Exception as e:
logger.error(f"Error processing audio buffer: {e}")
import traceback
logger.error(traceback.format_exc())
yield ErrorFrame(f"Error processing audio: {str(e)}")
finally:
self.buffer.is_processing = False
self.buffer.frames = []
self.buffer.started_at = None
self._buffer.is_processing = False
self._buffer.frames = []
self._buffer.started_at = None