Compare commits
1 Commits
hush/realt
...
hush/nova3
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f039ece2c0 |
15
.gitignore
vendored
15
.gitignore
vendored
@@ -32,21 +32,6 @@ fly.toml
|
|||||||
|
|
||||||
# Example files
|
# Example files
|
||||||
pipecat/examples/twilio-chatbot/templates/streams.xml
|
pipecat/examples/twilio-chatbot/templates/streams.xml
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/node_modules/
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/.expo/
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/dist/
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/npm-debug.*
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/*.jks
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/*.p8
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/*.p12
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/*.key
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/*.mobileprovision
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/*.orig.*
|
|
||||||
pipecat/examples/bot-ready-signalling/client/react-native/web-build/
|
|
||||||
|
|
||||||
# macOS
|
|
||||||
.DS_Store
|
|
||||||
|
|
||||||
|
|
||||||
# Documentation
|
# Documentation
|
||||||
docs/api/_build/
|
docs/api/_build/
|
||||||
|
|||||||
119
CHANGELOG.md
119
CHANGELOG.md
@@ -5,125 +5,6 @@ All notable changes to **Pipecat** will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
## [Unreleased]
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Pipecat version will now be logged on every application startup. This will
|
|
||||||
help us identify what version we are running in case of any issues.
|
|
||||||
|
|
||||||
- Added a new `StopFrame` which can be used to stop a pipeline task while
|
|
||||||
keeping the frame processors running. The frame processors could then be used
|
|
||||||
in a different pipeline. The difference between a `StopFrame` and a
|
|
||||||
`StopTaskFrame` is that, as with `EndFrame` and `EndTaskFrame`, the
|
|
||||||
`StopFrame` is pushed from the task and the `StopTaskFrame` is pushed upstream
|
|
||||||
inside the pipeline by any processor.
|
|
||||||
|
|
||||||
- Added a new `PipelineTask` parameter `observers` that replaces the previous
|
|
||||||
`PipelineParams.observers`.
|
|
||||||
|
|
||||||
- Added a new `PipelineTask` parameter `check_dangling_tasks` to enable or
|
|
||||||
disable checking for frame processors' dangling tasks when the Pipeline
|
|
||||||
finishes running.
|
|
||||||
|
|
||||||
- Added new `on_completion_timeout` event for LLM services (all OpenAI-based
|
|
||||||
services, Anthropic and Google). Note that this event will only get triggered
|
|
||||||
if LLM timeouts are setup and if the timeout was reached. It can be useful to
|
|
||||||
retrigger another completion and see if the timeout was just a blip.
|
|
||||||
|
|
||||||
- Added new log observers `LLMLogObserver` and `TranscriptionLogObserver` that
|
|
||||||
can be useful for debugging your pipelines.
|
|
||||||
|
|
||||||
- Added `room_url` property to `DailyTransport`.
|
|
||||||
|
|
||||||
- Added `addons` argument to `DeepgramSTTService`.
|
|
||||||
|
|
||||||
- Added `exponential_backoff_time()` to `utils.network` module.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- ⚠️ `PipelineTask` now requires keyword arguments (except for the first one for
|
|
||||||
the pipeline).
|
|
||||||
|
|
||||||
- The base `TTSService` class now strips leading newlines before sending text
|
|
||||||
to the TTS provider. This change is to solve issues where some TTS providers,
|
|
||||||
like Azure, would not output text due to newlines.
|
|
||||||
|
|
||||||
- `GrokLLMSService` now uses `grok-2` as the default model.
|
|
||||||
|
|
||||||
- `AnthropicLLMService` now uses `claude-3-7-sonnet-20250219` as the default
|
|
||||||
model.
|
|
||||||
|
|
||||||
- `RimeHttpTTSService` needs an `aiohttp.ClientSession` to be passed to the
|
|
||||||
constructor as all the other HTTP-based services.
|
|
||||||
|
|
||||||
- `RimeHttpTTSService` doesn't use a default voice anymore.
|
|
||||||
|
|
||||||
- `DeepgramSTTService` now uses the new `nova-3` model by default. If you want
|
|
||||||
to use the previous model you can pass `LiveOptions(model="nova-2-general")`.
|
|
||||||
(see https://deepgram.com/learn/introducing-nova-3-speech-to-text-api)
|
|
||||||
|
|
||||||
```python
|
|
||||||
stt = DeepgramSTTService(..., live_options=LiveOptions(model="nova-2-general"))
|
|
||||||
```
|
|
||||||
|
|
||||||
### Deprecated
|
|
||||||
|
|
||||||
- `PipelineParams.observers` is now deprecated, you the new `PipelineTask`
|
|
||||||
parameter `observers`.
|
|
||||||
|
|
||||||
### Removed
|
|
||||||
|
|
||||||
- Remove `TransportParams.audio_out_is_live` since it was not being used at all.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Fixed an `AudioContextWordTTSService` issue that would cause an `EndFrame` to
|
|
||||||
disconnect from the TTS service before audio from all the contexts was
|
|
||||||
received. This affected services like Cartesia and Rime.
|
|
||||||
|
|
||||||
- Fixed an issue that was not allowing to pass an `OpenAILLMContext` to create
|
|
||||||
`GoogleLLMService`'s context aggregators.
|
|
||||||
|
|
||||||
- Fixed a `ElevenLabsTTSService`, `FishAudioTTSService`, `LMNTTTSService` and
|
|
||||||
`PlayHTTTSService` issue that was resulting in audio requested before an
|
|
||||||
interruption being played after an interruption.
|
|
||||||
|
|
||||||
- Fixed `match_endofsentence` support for ellipses.
|
|
||||||
|
|
||||||
- Fixed an issue that would cause undesired interruptions via
|
|
||||||
`EmulateUserStartedSpeakingFrame` when only interim transcriptions (i.e. no
|
|
||||||
final transcriptions) where received.
|
|
||||||
|
|
||||||
- Fixed an issue where `EndTaskFrame` was not triggering
|
|
||||||
`on_client_disconnected` or closing the WebSocket in FastAPI.
|
|
||||||
|
|
||||||
- Fixed an issue in `DeepgramSTTService` where the `sample_rate` passed to the
|
|
||||||
`LiveOptions` was not being used, causing the service to use the default
|
|
||||||
sample rate of pipeline.
|
|
||||||
|
|
||||||
- Fixed a context aggregator issue that would not append the LLM text response
|
|
||||||
to the context if a function call happened in the same LLM turn.
|
|
||||||
|
|
||||||
- Fixed an issue that was causing HTTP TTS services to push `TTSStoppedFrame`
|
|
||||||
more than once.
|
|
||||||
|
|
||||||
- Fixed a `FishAudioTTSService` issue where `TTSStoppedFrame` was not being
|
|
||||||
pushed.
|
|
||||||
|
|
||||||
- Fixed an issue that `start_callback` was not invoked for some LLM services.
|
|
||||||
|
|
||||||
- Fixed an issue that would cause `DeepgramSTTService` to stop working after an
|
|
||||||
error occurred (e.g. sudden network loss). If the network recovered we would
|
|
||||||
not reconnect.
|
|
||||||
|
|
||||||
- Fixed a `STTMuteFilter` issue that would not mute user audio frames causing
|
|
||||||
transcriptions to be generated by the STT service.
|
|
||||||
|
|
||||||
### Other
|
|
||||||
|
|
||||||
- Added Gemini support to `examples/phone-chatbot`.
|
|
||||||
|
|
||||||
## [0.0.57] - 2025-02-14
|
## [0.0.57] - 2025-02-14
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
@@ -3,10 +3,10 @@ coverage~=7.6.12
|
|||||||
grpcio-tools~=1.67.1
|
grpcio-tools~=1.67.1
|
||||||
pip-tools~=7.4.1
|
pip-tools~=7.4.1
|
||||||
pre-commit~=4.0.1
|
pre-commit~=4.0.1
|
||||||
pyright~=1.1.394
|
pyright~=1.1.393
|
||||||
pytest~=8.3.4
|
pytest~=8.3.4
|
||||||
pytest-asyncio~=0.25.3
|
pytest-asyncio~=0.25.2
|
||||||
ruff~=0.9.7
|
ruff~=0.9.5
|
||||||
setuptools~=70.0.0
|
setuptools~=70.0.0
|
||||||
setuptools_scm~=8.1.0
|
setuptools_scm~=8.1.0
|
||||||
python-dotenv~=1.0.1
|
python-dotenv~=1.0.1
|
||||||
|
|||||||
@@ -18,9 +18,6 @@ AZURE_DALLE_API_KEY=...
|
|||||||
AZURE_DALLE_ENDPOINT=https://...
|
AZURE_DALLE_ENDPOINT=https://...
|
||||||
AZURE_DALLE_MODEL=...
|
AZURE_DALLE_MODEL=...
|
||||||
|
|
||||||
# Cartesia
|
|
||||||
CARTESIA_API_KEY=...
|
|
||||||
|
|
||||||
# Daily
|
# Daily
|
||||||
DAILY_API_KEY=...
|
DAILY_API_KEY=...
|
||||||
DAILY_SAMPLE_ROOM_URL=https://...
|
DAILY_SAMPLE_ROOM_URL=https://...
|
||||||
|
|||||||
@@ -1 +0,0 @@
|
|||||||
22.14
|
|
||||||
@@ -1,60 +0,0 @@
|
|||||||
# React Native Implementation
|
|
||||||
|
|
||||||
Basic implementation using the [Pipecat React Native SDK](https://docs.pipecat.ai/client/react-native/introduction).
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Expo requirements
|
|
||||||
|
|
||||||
This project cannot be used with an [Expo Go](https://docs.expo.dev/workflow/expo-go/) app because [it requires custom native code](https://docs.expo.io/workflow/customizing/).
|
|
||||||
|
|
||||||
When a project requires custom native code or a config plugin, we need to transition from using [Expo Go](https://docs.expo.dev/workflow/expo-go/)
|
|
||||||
to a [development build](https://docs.expo.dev/development/introduction/).
|
|
||||||
|
|
||||||
More details about the custom native code used by this demo can be found in [rn-daily-js-expo-config-plugin](https://github.com/daily-co/rn-daily-js-expo-config-plugin).
|
|
||||||
|
|
||||||
### Building remotely
|
|
||||||
|
|
||||||
If you do not have experience with Xcode and Android Studio builds or do not have them installed locally on your computer, you will need to follow [this guide from Expo to use EAS Build](https://docs.expo.dev/development/create-development-builds/#create-and-install-eas-build).
|
|
||||||
|
|
||||||
### Building locally
|
|
||||||
|
|
||||||
You will need to have installed locally on your computer:
|
|
||||||
- [Xcode](https://developer.apple.com/xcode/) to build for iOS;
|
|
||||||
- [Android Studio](https://developer.android.com/studio) to build for Android;
|
|
||||||
|
|
||||||
#### Install the demo dependencies
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Use the version of node specified in .nvmrc
|
|
||||||
nvm i
|
|
||||||
|
|
||||||
# Install dependencies
|
|
||||||
npm i
|
|
||||||
|
|
||||||
# Before a native app can be compiled, the native source code must be generated.
|
|
||||||
npx expo prebuild
|
|
||||||
|
|
||||||
# Configure the environment variable to connect to the local server
|
|
||||||
cp env.example .env
|
|
||||||
# edit .env and add your local ip address, for example: http://192.168.1.16:7860
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Running on Android
|
|
||||||
|
|
||||||
After plugging in an Android device [configured for debugging](https://developer.android.com/studio/debug/dev-options), run the following command:
|
|
||||||
|
|
||||||
```
|
|
||||||
npm run android
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Running on iOS
|
|
||||||
|
|
||||||
Run the following command:
|
|
||||||
|
|
||||||
```
|
|
||||||
npm run ios
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Connect to the server
|
|
||||||
Use the http://localhost:5173 in your app.
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
{
|
|
||||||
"expo": {
|
|
||||||
"name": "bot-ready-rn",
|
|
||||||
"slug": "bot-ready-rn",
|
|
||||||
"version": "1.0.0",
|
|
||||||
"orientation": "portrait",
|
|
||||||
"icon": "./assets/icon.png",
|
|
||||||
"userInterfaceStyle": "light",
|
|
||||||
"splash": {
|
|
||||||
"image": "./assets/splash.png",
|
|
||||||
"resizeMode": "contain",
|
|
||||||
"backgroundColor": "#ffffff"
|
|
||||||
},
|
|
||||||
"updates": {
|
|
||||||
"fallbackToCacheTimeout": 0
|
|
||||||
},
|
|
||||||
"assetBundlePatterns": [
|
|
||||||
"**/*"
|
|
||||||
],
|
|
||||||
"ios": {
|
|
||||||
"supportsTablet": true,
|
|
||||||
"bitcode": false,
|
|
||||||
"bundleIdentifier": "co.daily.expo.BotReady",
|
|
||||||
"infoPlist": {
|
|
||||||
"UIBackgroundModes": [
|
|
||||||
"voip"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"appleTeamId": "EEBGKV9N3N"
|
|
||||||
},
|
|
||||||
"android": {
|
|
||||||
"adaptiveIcon": {
|
|
||||||
"foregroundImage": "./assets/adaptive-icon.png",
|
|
||||||
"backgroundColor": "#FFFFFF"
|
|
||||||
},
|
|
||||||
"package": "co.daily.expo.BotReady",
|
|
||||||
"permissions": [
|
|
||||||
"android.permission.ACCESS_NETWORK_STATE",
|
|
||||||
"android.permission.BLUETOOTH",
|
|
||||||
"android.permission.CAMERA",
|
|
||||||
"android.permission.INTERNET",
|
|
||||||
"android.permission.MODIFY_AUDIO_SETTINGS",
|
|
||||||
"android.permission.RECORD_AUDIO",
|
|
||||||
"android.permission.SYSTEM_ALERT_WINDOW",
|
|
||||||
"android.permission.WAKE_LOCK",
|
|
||||||
"android.permission.FOREGROUND_SERVICE",
|
|
||||||
"android.permission.FOREGROUND_SERVICE_CAMERA",
|
|
||||||
"android.permission.FOREGROUND_SERVICE_MICROPHONE",
|
|
||||||
"android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION",
|
|
||||||
"android.permission.POST_NOTIFICATIONS"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"web": {
|
|
||||||
"favicon": "./assets/favicon.png"
|
|
||||||
},
|
|
||||||
"plugins": [
|
|
||||||
"@config-plugins/react-native-webrtc",
|
|
||||||
"@daily-co/config-plugin-rn-daily-js",
|
|
||||||
[
|
|
||||||
"expo-build-properties",
|
|
||||||
{
|
|
||||||
"android": {
|
|
||||||
"minSdkVersion": 24,
|
|
||||||
"compileSdkVersion": 35,
|
|
||||||
"targetSdkVersion": 34,
|
|
||||||
"buildToolsVersion": "35.0.0"
|
|
||||||
},
|
|
||||||
"ios": {
|
|
||||||
"deploymentTarget": "15.1"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Binary file not shown.
|
Before Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.4 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 22 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 46 KiB |
@@ -1,7 +0,0 @@
|
|||||||
module.exports = function(api) {
|
|
||||||
api.cache(true);
|
|
||||||
return {
|
|
||||||
presets: ['babel-preset-expo'],
|
|
||||||
plugins: [["module:react-native-dotenv"]],
|
|
||||||
};
|
|
||||||
};
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
API_BASE_URL=http://YOUR_LOCAL_IP:7860
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
import { registerRootComponent } from "expo";
|
|
||||||
|
|
||||||
import App from "./src/App";
|
|
||||||
|
|
||||||
// registerRootComponent calls AppRegistry.registerComponent('main', () => App);
|
|
||||||
// It also ensures that the environment is set up appropriately
|
|
||||||
registerRootComponent(App);
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
// Learn more https://docs.expo.io/guides/customizing-metro
|
|
||||||
const { getDefaultConfig } = require('expo/metro-config');
|
|
||||||
|
|
||||||
module.exports = getDefaultConfig(__dirname);
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,31 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "bot-ready-rn",
|
|
||||||
"version": "1.0.0",
|
|
||||||
"scripts": {
|
|
||||||
"start": "expo start --dev-client",
|
|
||||||
"android": "expo run:android --device",
|
|
||||||
"ios": "expo run:ios --device",
|
|
||||||
"web": "expo start --web"
|
|
||||||
},
|
|
||||||
"dependencies": {
|
|
||||||
"@config-plugins/react-native-webrtc": "^10.0.0",
|
|
||||||
"@daily-co/config-plugin-rn-daily-js": "0.0.7",
|
|
||||||
"@daily-co/react-native-daily-js": "^0.70.0",
|
|
||||||
"@daily-co/react-native-webrtc": "^118.0.3-daily.2",
|
|
||||||
"@react-native-async-storage/async-storage": "1.23.1",
|
|
||||||
"expo": "^52.0.0",
|
|
||||||
"expo-build-properties": "~0.13.1",
|
|
||||||
"expo-dev-client": "~5.0.5",
|
|
||||||
"expo-splash-screen": "~0.29.16",
|
|
||||||
"expo-status-bar": "~2.0.0",
|
|
||||||
"react": "18.3.1",
|
|
||||||
"react-native": "0.76.3",
|
|
||||||
"react-native-background-timer": "^2.4.1",
|
|
||||||
"react-native-dotenv": "^3.4.11",
|
|
||||||
"react-native-get-random-values": "^1.11.0"
|
|
||||||
},
|
|
||||||
"devDependencies": {
|
|
||||||
"@babel/core": "^7.12.9"
|
|
||||||
},
|
|
||||||
"private": true
|
|
||||||
}
|
|
||||||
@@ -1,121 +0,0 @@
|
|||||||
import React, { useState, useEffect } from 'react';
|
|
||||||
import {SafeAreaView, View, Text, Button, StyleSheet, ScrollView} from 'react-native';
|
|
||||||
import Daily from "@daily-co/react-native-daily-js";
|
|
||||||
import { API_BASE_URL } from "@env";
|
|
||||||
|
|
||||||
const CallScreen = () => {
|
|
||||||
const [connectionStatus, setConnectionStatus] = useState('Disconnected');
|
|
||||||
const [isConnected, setIsConnected] = useState(false);
|
|
||||||
const [callObject, setCallObject] = useState(null);
|
|
||||||
const [logs, setLogs] = useState([]);
|
|
||||||
|
|
||||||
useEffect(() => {
|
|
||||||
if (callObject) {
|
|
||||||
setupTrackListeners(callObject);
|
|
||||||
}
|
|
||||||
}, [callObject]);
|
|
||||||
|
|
||||||
const log = (message) => {
|
|
||||||
setLogs((prevLogs) => [...prevLogs, `${new Date().toISOString()} - ${message}`]);
|
|
||||||
console.log(message);
|
|
||||||
};
|
|
||||||
|
|
||||||
const setupTrackListeners = (callObject) => {
|
|
||||||
callObject.on("joined-meeting", () => {
|
|
||||||
setConnectionStatus('Connected');
|
|
||||||
setIsConnected(true);
|
|
||||||
log('Client connected');
|
|
||||||
});
|
|
||||||
callObject.on("left-meeting", () => {
|
|
||||||
setConnectionStatus('Disconnected');
|
|
||||||
setIsConnected(false);
|
|
||||||
log('Client disconnected');
|
|
||||||
});
|
|
||||||
callObject.on("participant-left", () => {
|
|
||||||
// When the bot leaves, we are also disconnecting from the call
|
|
||||||
disconnect().catch((err) => {
|
|
||||||
log(`Failed to disconnect ${err}`);
|
|
||||||
})
|
|
||||||
});
|
|
||||||
// Trigger so the bot can start sending audio
|
|
||||||
callObject.on("track-started", (evt) => {
|
|
||||||
if (evt.track.kind === "audio" && evt.participant.local === false) {
|
|
||||||
handleEventToConsole(evt)
|
|
||||||
log("Sending the message that will trigger the bot to play the audio.")
|
|
||||||
callObject.sendAppMessage("playable")
|
|
||||||
}
|
|
||||||
});
|
|
||||||
callObject.on("error", (evt) => log(`Error: ${evt.error}`));
|
|
||||||
// Other events just for awareness
|
|
||||||
callObject.on("track-stopped", handleEventToConsole);
|
|
||||||
callObject.on("participant-joined", handleEventToConsole);
|
|
||||||
callObject.on("participant-updated", handleEventToConsole);
|
|
||||||
};
|
|
||||||
|
|
||||||
const handleEventToConsole = (evt) => {
|
|
||||||
log(`Received event: ${evt.action}`);
|
|
||||||
};
|
|
||||||
|
|
||||||
const connect = async () => {
|
|
||||||
try {
|
|
||||||
const callObject = Daily.createCallObject({ subscribeToTracksAutomatically: true });
|
|
||||||
setCallObject(callObject);
|
|
||||||
const connectionUrl = `${API_BASE_URL}/connect`
|
|
||||||
const res = await fetch(connectionUrl, { method: "POST", headers: { "Content-Type": "application/json" } });
|
|
||||||
const roomInfo = await res.json();
|
|
||||||
await callObject.join({ url: roomInfo.room_url });
|
|
||||||
} catch (error) {
|
|
||||||
log(`Error connecting: ${error.message}`);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
const disconnect = async () => {
|
|
||||||
if (callObject) {
|
|
||||||
try {
|
|
||||||
await callObject.leave();
|
|
||||||
await callObject.destroy();
|
|
||||||
setCallObject(null);
|
|
||||||
} catch (error) {
|
|
||||||
log(`Error disconnecting: ${error.message}`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
return (
|
|
||||||
<SafeAreaView style={styles.safeArea}>
|
|
||||||
<View style={styles.container}>
|
|
||||||
<View style={styles.statusBar}>
|
|
||||||
<Text>Status: <Text style={styles.status}>{connectionStatus}</Text></Text>
|
|
||||||
<View style={styles.controls}>
|
|
||||||
<Button
|
|
||||||
title={isConnected ? "Disconnect" : "Connect"}
|
|
||||||
onPress={isConnected ? disconnect : connect}
|
|
||||||
/>
|
|
||||||
</View>
|
|
||||||
</View>
|
|
||||||
|
|
||||||
<View style={styles.debugPanel}>
|
|
||||||
<Text style={styles.debugTitle}>Debug Info</Text>
|
|
||||||
<ScrollView style={styles.debugLog}>
|
|
||||||
{logs.map((logEntry, index) => (
|
|
||||||
<Text key={index} style={styles.logText}>{logEntry}</Text>
|
|
||||||
))}
|
|
||||||
</ScrollView>
|
|
||||||
</View>
|
|
||||||
</View>
|
|
||||||
</SafeAreaView>
|
|
||||||
);
|
|
||||||
};
|
|
||||||
|
|
||||||
const styles = StyleSheet.create({
|
|
||||||
safeArea: { flex: 1, backgroundColor: '#f0f0f0', padding: 20 },
|
|
||||||
container: { flex: 1, margin: 20 },
|
|
||||||
statusBar: { flexDirection: 'row', justifyContent: 'space-between', alignItems: 'center', padding: 10, backgroundColor: '#fff', borderRadius: 8, marginBottom: 20 },
|
|
||||||
status: { fontWeight: 'bold' },
|
|
||||||
controls: { flexDirection: 'row', gap: 10 },
|
|
||||||
debugPanel: { height: '80%', backgroundColor: '#fff', borderRadius: 8, padding: 20},
|
|
||||||
debugTitle: { fontSize: 16, fontWeight: 'bold' },
|
|
||||||
debugLog: { height: '100%', overflow: 'scroll', backgroundColor: '#f8f8f8', padding: 10, borderRadius: 4, fontFamily: 'monospace', fontSize: 12, lineHeight: 1.4 },
|
|
||||||
});
|
|
||||||
|
|
||||||
export default CallScreen;
|
|
||||||
@@ -17,7 +17,7 @@ from runner import configure
|
|||||||
from pipecat.frames.frames import AudioRawFrame, EndFrame, OutputAudioRawFrame, TTSSpeakFrame
|
from pipecat.frames.frames import AudioRawFrame, EndFrame, OutputAudioRawFrame, TTSSpeakFrame
|
||||||
from pipecat.pipeline.pipeline import Pipeline
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineTask
|
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||||
from pipecat.services.cartesia import CartesiaTTSService
|
from pipecat.services.cartesia import CartesiaTTSService
|
||||||
from pipecat.transports.services.daily import DailyParams, DailyTransport
|
from pipecat.transports.services.daily import DailyParams, DailyTransport
|
||||||
|
|
||||||
|
|||||||
@@ -119,7 +119,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -124,7 +124,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@audiobuffer.event_handler("on_audio_data")
|
@audiobuffer.event_handler("on_audio_data")
|
||||||
async def on_audio_data(buffer, audio, sample_rate, num_channels):
|
async def on_audio_data(buffer, audio, sample_rate, num_channels):
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ async def main(room_url: str, token: str):
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -62,7 +62,7 @@ async def main(room_url: str, token: str):
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -18,7 +18,8 @@ from pipecat.pipeline.pipeline import Pipeline
|
|||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineTask
|
from pipecat.pipeline.task import PipelineTask
|
||||||
from pipecat.services.fal import FalImageGenService
|
from pipecat.services.fal import FalImageGenService
|
||||||
from pipecat.transports.local.tk import TkLocalTransport, TkTransportParams
|
from pipecat.transports.base_transport import TransportParams
|
||||||
|
from pipecat.transports.local.tk import TkLocalTransport
|
||||||
|
|
||||||
load_dotenv(override=True)
|
load_dotenv(override=True)
|
||||||
|
|
||||||
@@ -33,9 +34,7 @@ async def main():
|
|||||||
|
|
||||||
transport = TkLocalTransport(
|
transport = TkLocalTransport(
|
||||||
tk_root,
|
tk_root,
|
||||||
TkTransportParams(
|
TransportParams(camera_out_enabled=True, camera_out_width=1024, camera_out_height=1024),
|
||||||
camera_out_enabled=True, camera_out_width=1024, camera_out_height=1024
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
imagegen = FalImageGenService(
|
imagegen = FalImageGenService(
|
||||||
|
|||||||
@@ -44,8 +44,7 @@ async def main():
|
|||||||
runner = PipelineRunner()
|
runner = PipelineRunner()
|
||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
Pipeline([imagegen, transport.output()]),
|
Pipeline([imagegen, transport.output()]), PipelineParams(enable_metrics=True)
|
||||||
params=PipelineParams(enable_metrics=True),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
|
|||||||
@@ -30,7 +30,8 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
|||||||
from pipecat.services.cartesia import CartesiaHttpTTSService
|
from pipecat.services.cartesia import CartesiaHttpTTSService
|
||||||
from pipecat.services.fal import FalImageGenService
|
from pipecat.services.fal import FalImageGenService
|
||||||
from pipecat.services.openai import OpenAILLMService
|
from pipecat.services.openai import OpenAILLMService
|
||||||
from pipecat.transports.local.tk import TkLocalTransport, TkTransportParams
|
from pipecat.transports.base_transport import TransportParams
|
||||||
|
from pipecat.transports.local.tk import TkLocalTransport, TkOutputTransport
|
||||||
|
|
||||||
load_dotenv(override=True)
|
load_dotenv(override=True)
|
||||||
|
|
||||||
@@ -151,7 +152,7 @@ async def main():
|
|||||||
|
|
||||||
transport = TkLocalTransport(
|
transport = TkLocalTransport(
|
||||||
tk_root,
|
tk_root,
|
||||||
TkTransportParams(
|
TransportParams(
|
||||||
audio_out_enabled=True,
|
audio_out_enabled=True,
|
||||||
camera_out_enabled=True,
|
camera_out_enabled=True,
|
||||||
camera_out_width=1024,
|
camera_out_width=1024,
|
||||||
|
|||||||
@@ -105,10 +105,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(enable_metrics=True, enable_usage_metrics=True),
|
||||||
enable_metrics=True,
|
|
||||||
enable_usage_metrics=True,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
|
|||||||
@@ -127,7 +127,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -76,7 +76,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -103,7 +103,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -81,7 +81,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import os
|
|||||||
import sys
|
import sys
|
||||||
|
|
||||||
import aiohttp
|
import aiohttp
|
||||||
|
from deepgram import LiveOptions
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from runner import configure
|
from runner import configure
|
||||||
@@ -44,7 +45,23 @@ async def main():
|
|||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
|
# stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
|
||||||
|
stt = DeepgramSTTService(
|
||||||
|
api_key=os.getenv("DEEPGRAM_API_KEY"),
|
||||||
|
# url=deepgram_url,
|
||||||
|
live_options=LiveOptions(
|
||||||
|
encoding="linear16",
|
||||||
|
language="en-US",
|
||||||
|
model="nova-3",
|
||||||
|
channels=1,
|
||||||
|
interim_results=True,
|
||||||
|
# smart_format=smart_format,
|
||||||
|
# endpointing=endpointing,
|
||||||
|
vad_events=True,
|
||||||
|
diarize=True,
|
||||||
|
filler_words=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-helios-en")
|
tts = DeepgramTTSService(api_key=os.getenv("DEEPGRAM_API_KEY"), voice="aura-helios-en")
|
||||||
|
|
||||||
@@ -74,7 +91,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -75,7 +75,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -77,7 +77,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -83,7 +83,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -81,7 +81,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -81,7 +81,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -75,7 +75,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -80,7 +80,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -88,7 +88,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -81,7 +81,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -80,7 +80,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -76,7 +76,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -251,7 +251,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -78,11 +78,7 @@ async def main():
|
|||||||
runner = PipelineRunner()
|
runner = PipelineRunner()
|
||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline, PipelineParams(audio_in_sample_rate=24000, audio_out_sample_rate=24000)
|
||||||
params=PipelineParams(
|
|
||||||
audio_in_sample_rate=24000,
|
|
||||||
audio_out_sample_rate=24000,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
await runner.run(task)
|
await runner.run(task)
|
||||||
|
|||||||
@@ -24,7 +24,8 @@ from pipecat.pipeline.pipeline import Pipeline
|
|||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||||
from pipecat.transports.local.tk import TkLocalTransport, TkTransportParams
|
from pipecat.transports.base_transport import TransportParams
|
||||||
|
from pipecat.transports.local.tk import TkLocalTransport
|
||||||
from pipecat.transports.services.daily import DailyParams, DailyTransport
|
from pipecat.transports.services.daily import DailyParams, DailyTransport
|
||||||
|
|
||||||
load_dotenv(override=True)
|
load_dotenv(override=True)
|
||||||
@@ -66,7 +67,7 @@ async def main():
|
|||||||
|
|
||||||
tk_transport = TkLocalTransport(
|
tk_transport = TkLocalTransport(
|
||||||
tk_root,
|
tk_root,
|
||||||
TkTransportParams(
|
TransportParams(
|
||||||
audio_out_enabled=True,
|
audio_out_enabled=True,
|
||||||
camera_out_enabled=True,
|
camera_out_enabled=True,
|
||||||
camera_out_is_live=True,
|
camera_out_is_live=True,
|
||||||
@@ -82,11 +83,7 @@ async def main():
|
|||||||
pipeline = Pipeline([daily_transport.input(), MirrorProcessor(), tk_transport.output()])
|
pipeline = Pipeline([daily_transport.input(), MirrorProcessor(), tk_transport.output()])
|
||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline, PipelineParams(audio_in_sample_rate=24000, audio_out_sample_rate=24000)
|
||||||
params=PipelineParams(
|
|
||||||
audio_in_sample_rate=24000,
|
|
||||||
audio_out_sample_rate=24000,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
async def run_tk():
|
async def run_tk():
|
||||||
|
|||||||
@@ -76,7 +76,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -112,7 +112,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -99,13 +99,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
|
||||||
pipeline,
|
|
||||||
params=PipelineParams(
|
|
||||||
allow_interruptions=True,
|
|
||||||
enable_metrics=True,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -153,13 +153,7 @@ If you need to use a tool, simply use the tool. Do not tell the user the tool yo
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
|
||||||
pipeline,
|
|
||||||
params=PipelineParams(
|
|
||||||
allow_interruptions=True,
|
|
||||||
enable_metrics=True,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ from loguru import logger
|
|||||||
from runner import configure
|
from runner import configure
|
||||||
|
|
||||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
from pipecat.frames.frames import TTSSpeakFrame
|
|
||||||
from pipecat.pipeline.pipeline import Pipeline
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||||
@@ -31,12 +30,6 @@ logger.add(sys.stderr, level="DEBUG")
|
|||||||
video_participant_id = None
|
video_participant_id = None
|
||||||
|
|
||||||
|
|
||||||
async def start_fetch_weather(function_name, llm, context):
|
|
||||||
"""Push a frame to the LLM; this is handy when the LLM response might take a while."""
|
|
||||||
await llm.push_frame(TTSSpeakFrame("Let me check on that."))
|
|
||||||
logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")
|
|
||||||
|
|
||||||
|
|
||||||
async def get_weather(function_name, tool_call_id, arguments, llm, context, result_callback):
|
async def get_weather(function_name, tool_call_id, arguments, llm, context, result_callback):
|
||||||
location = arguments["location"]
|
location = arguments["location"]
|
||||||
await result_callback(f"The weather in {location} is currently 72 degrees and sunny.")
|
await result_callback(f"The weather in {location} is currently 72 degrees and sunny.")
|
||||||
@@ -70,7 +63,7 @@ async def main():
|
|||||||
)
|
)
|
||||||
|
|
||||||
llm = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"), model="gemini-2.0-flash-001")
|
llm = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"), model="gemini-2.0-flash-001")
|
||||||
llm.register_function("get_weather", get_weather, start_fetch_weather)
|
llm.register_function("get_weather", get_weather)
|
||||||
llm.register_function("get_image", get_image)
|
llm.register_function("get_image", get_image)
|
||||||
|
|
||||||
tools = [
|
tools = [
|
||||||
@@ -152,7 +145,7 @@ indicate you should use the get_image tool are:
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -113,7 +113,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -117,7 +117,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -123,7 +123,7 @@ Start by asking me for my location. Then, use 'get_weather_current' to give me a
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -123,7 +123,7 @@ Start by asking me for my location. Then, use 'get_weather_current' to give me a
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -117,7 +117,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -83,7 +83,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -133,7 +133,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -126,7 +126,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -85,13 +85,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True, enable_metrics=True))
|
||||||
pipeline,
|
|
||||||
params=PipelineParams(
|
|
||||||
allow_interruptions=True,
|
|
||||||
enable_metrics=True,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
# When a participant joins, start transcription for that participant so the
|
# When a participant joins, start transcription for that participant so the
|
||||||
# bot can "hear" and respond to them.
|
# bot can "hear" and respond to them.
|
||||||
|
|||||||
@@ -108,7 +108,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
report_only_initial_ttfb=True,
|
report_only_initial_ttfb=True,
|
||||||
|
|||||||
@@ -38,6 +38,7 @@ async def main():
|
|||||||
"GStreamer",
|
"GStreamer",
|
||||||
DailyParams(
|
DailyParams(
|
||||||
audio_out_enabled=True,
|
audio_out_enabled=True,
|
||||||
|
audio_out_is_live=True,
|
||||||
camera_out_enabled=True,
|
camera_out_enabled=True,
|
||||||
camera_out_width=1280,
|
camera_out_width=1280,
|
||||||
camera_out_height=720,
|
camera_out_height=720,
|
||||||
|
|||||||
@@ -16,13 +16,10 @@ from runner import configure
|
|||||||
|
|
||||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
from pipecat.audio.vad.vad_analyzer import VADParams
|
from pipecat.audio.vad.vad_analyzer import VADParams
|
||||||
from pipecat.frames.frames import TranscriptionMessage, TranscriptionUpdateFrame
|
|
||||||
from pipecat.pipeline.pipeline import Pipeline
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||||
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
||||||
from pipecat.processors.transcript_processor import TranscriptProcessor
|
|
||||||
from pipecat.services.deepgram import DeepgramSTTService
|
|
||||||
from pipecat.services.openai_realtime_beta import (
|
from pipecat.services.openai_realtime_beta import (
|
||||||
InputAudioTranscription,
|
InputAudioTranscription,
|
||||||
OpenAIRealtimeBetaLLMService,
|
OpenAIRealtimeBetaLLMService,
|
||||||
@@ -143,29 +140,21 @@ Remember, your responses should be short. Just one or two sentences, usually."""
|
|||||||
tools,
|
tools,
|
||||||
)
|
)
|
||||||
|
|
||||||
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
|
|
||||||
|
|
||||||
# Create transcript processor and handler
|
|
||||||
transcript = TranscriptProcessor()
|
|
||||||
|
|
||||||
context_aggregator = llm.create_context_aggregator(context)
|
context_aggregator = llm.create_context_aggregator(context)
|
||||||
|
|
||||||
pipeline = Pipeline(
|
pipeline = Pipeline(
|
||||||
[
|
[
|
||||||
transport.input(), # Transport user input
|
transport.input(), # Transport user input
|
||||||
stt,
|
|
||||||
transcript.user(), # User transcripts
|
|
||||||
context_aggregator.user(),
|
context_aggregator.user(),
|
||||||
llm, # LLM
|
llm, # LLM
|
||||||
context_aggregator.assistant(),
|
context_aggregator.assistant(),
|
||||||
transcript.assistant(), # Assistant transcripts
|
|
||||||
transport.output(), # Transport bot output
|
transport.output(), # Transport bot output
|
||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
@@ -173,16 +162,9 @@ Remember, your responses should be short. Just one or two sentences, usually."""
|
|||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Register event handler for transcript updates
|
|
||||||
@transcript.event_handler("on_transcript_update")
|
|
||||||
async def on_transcript_update(processor, frame):
|
|
||||||
logger.debug(f"Received transcript update with {len(frame.messages)} new messages")
|
|
||||||
for msg in frame.messages:
|
|
||||||
logger.debug(msg)
|
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
# await transport.capture_participant_transcription(participant["id"])
|
await transport.capture_participant_transcription(participant["id"])
|
||||||
# Kick off the conversation.
|
# Kick off the conversation.
|
||||||
await task.queue_frames([context_aggregator.user().get_context_frame()])
|
await task.queue_frames([context_aggregator.user().get_context_frame()])
|
||||||
|
|
||||||
|
|||||||
@@ -212,7 +212,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -237,7 +237,7 @@ Remember, your responses should be short. Just one or two sentences, usually."""
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -209,7 +209,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -263,7 +263,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -87,7 +87,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
# We just use 16000 because that's what Tavus is expecting and
|
# We just use 16000 because that's what Tavus is expecting and
|
||||||
# we avoid resampling.
|
# we avoid resampling.
|
||||||
audio_in_sample_rate=16000,
|
audio_in_sample_rate=16000,
|
||||||
|
|||||||
@@ -145,7 +145,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -138,7 +138,6 @@ class OutputGate(FrameProcessor):
|
|||||||
self._gate_open = start_open
|
self._gate_open = start_open
|
||||||
self._frames_buffer = []
|
self._frames_buffer = []
|
||||||
self._notifier = notifier
|
self._notifier = notifier
|
||||||
self._gate_task = None
|
|
||||||
|
|
||||||
def close_gate(self):
|
def close_gate(self):
|
||||||
self._gate_open = False
|
self._gate_open = False
|
||||||
@@ -179,13 +178,10 @@ class OutputGate(FrameProcessor):
|
|||||||
|
|
||||||
async def _start(self):
|
async def _start(self):
|
||||||
self._frames_buffer = []
|
self._frames_buffer = []
|
||||||
if not self._gate_task:
|
self._gate_task = self.create_task(self._gate_task_handler())
|
||||||
self._gate_task = self.create_task(self._gate_task_handler())
|
|
||||||
|
|
||||||
async def _stop(self):
|
async def _stop(self):
|
||||||
if self._gate_task:
|
await self.cancel_task(self._gate_task)
|
||||||
await self.cancel_task(self._gate_task)
|
|
||||||
self._gate_task = None
|
|
||||||
|
|
||||||
async def _gate_task_handler(self):
|
async def _gate_task_handler(self):
|
||||||
while True:
|
while True:
|
||||||
@@ -355,7 +351,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -342,7 +342,6 @@ class OutputGate(FrameProcessor):
|
|||||||
self._gate_open = start_open
|
self._gate_open = start_open
|
||||||
self._frames_buffer = []
|
self._frames_buffer = []
|
||||||
self._notifier = notifier
|
self._notifier = notifier
|
||||||
self._gate_task = None
|
|
||||||
|
|
||||||
def close_gate(self):
|
def close_gate(self):
|
||||||
self._gate_open = False
|
self._gate_open = False
|
||||||
@@ -383,13 +382,10 @@ class OutputGate(FrameProcessor):
|
|||||||
|
|
||||||
async def _start(self):
|
async def _start(self):
|
||||||
self._frames_buffer = []
|
self._frames_buffer = []
|
||||||
if not self._gate_task:
|
self._gate_task = self.create_task(self._gate_task_handler())
|
||||||
self._gate_task = self.create_task(self._gate_task_handler())
|
|
||||||
|
|
||||||
async def _stop(self):
|
async def _stop(self):
|
||||||
if self._gate_task:
|
await self.cancel_task(self._gate_task)
|
||||||
await self.cancel_task(self._gate_task)
|
|
||||||
self._gate_task = None
|
|
||||||
|
|
||||||
async def _gate_task_handler(self):
|
async def _gate_task_handler(self):
|
||||||
while True:
|
while True:
|
||||||
@@ -564,7 +560,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -25,8 +25,10 @@ from pipecat.frames.frames import (
|
|||||||
InputAudioRawFrame,
|
InputAudioRawFrame,
|
||||||
LLMFullResponseEndFrame,
|
LLMFullResponseEndFrame,
|
||||||
LLMFullResponseStartFrame,
|
LLMFullResponseStartFrame,
|
||||||
|
LLMMessagesFrame,
|
||||||
StartFrame,
|
StartFrame,
|
||||||
StartInterruptionFrame,
|
StartInterruptionFrame,
|
||||||
|
StopInterruptionFrame,
|
||||||
SystemFrame,
|
SystemFrame,
|
||||||
TextFrame,
|
TextFrame,
|
||||||
TranscriptionFrame,
|
TranscriptionFrame,
|
||||||
@@ -553,7 +555,6 @@ class OutputGate(FrameProcessor):
|
|||||||
self._notifier = notifier
|
self._notifier = notifier
|
||||||
self._context = context
|
self._context = context
|
||||||
self._transcription_buffer = user_transcription_buffer
|
self._transcription_buffer = user_transcription_buffer
|
||||||
self._gate_task = None
|
|
||||||
|
|
||||||
def close_gate(self):
|
def close_gate(self):
|
||||||
self._gate_open = False
|
self._gate_open = False
|
||||||
@@ -601,13 +602,10 @@ class OutputGate(FrameProcessor):
|
|||||||
|
|
||||||
async def _start(self):
|
async def _start(self):
|
||||||
self._frames_buffer = []
|
self._frames_buffer = []
|
||||||
if not self._gate_task:
|
self._gate_task = self.create_task(self._gate_task_handler())
|
||||||
self._gate_task = self.create_task(self._gate_task_handler())
|
|
||||||
|
|
||||||
async def _stop(self):
|
async def _stop(self):
|
||||||
if self._gate_task:
|
await self.cancel_task(self._gate_task)
|
||||||
await self.cancel_task(self._gate_task)
|
|
||||||
self._gate_task = None
|
|
||||||
|
|
||||||
async def _gate_task_handler(self):
|
async def _gate_task_handler(self):
|
||||||
while True:
|
while True:
|
||||||
@@ -742,7 +740,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -87,7 +87,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -122,7 +122,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -354,7 +354,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -63,7 +63,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -89,7 +89,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -120,7 +120,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -106,7 +106,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
#
|
#
|
||||||
# Copyright (c) 2024-2025, Daily
|
# Copyright (c) 2024, Daily
|
||||||
#
|
#
|
||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
#
|
#
|
||||||
@@ -34,7 +34,7 @@ search_tool = {"google_search": {}}
|
|||||||
tools = [search_tool]
|
tools = [search_tool]
|
||||||
|
|
||||||
system_instruction = """
|
system_instruction = """
|
||||||
You are an expert at providing the most recent news from any place. Your responses will be converted to audio, so avoid using special characters or overly complex formatting.
|
You are an expert at providing the most recent news from any place. Your responses will be converted to audio, so avoid using special characters or overly complex formatting.
|
||||||
|
|
||||||
Always use the google search API to retrieve the latest news. You must also use it to check which day is today.
|
Always use the google search API to retrieve the latest news. You must also use it to check which day is today.
|
||||||
|
|
||||||
@@ -93,7 +93,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -83,7 +83,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
),
|
),
|
||||||
|
|||||||
@@ -150,7 +150,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -150,7 +150,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -178,7 +178,7 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
@@ -18,10 +18,12 @@ from pipecat.frames.frames import (
|
|||||||
BotStartedSpeakingFrame,
|
BotStartedSpeakingFrame,
|
||||||
BotStoppedSpeakingFrame,
|
BotStoppedSpeakingFrame,
|
||||||
Frame,
|
Frame,
|
||||||
|
LLMFullResponseEndFrame,
|
||||||
|
LLMFullResponseStartFrame,
|
||||||
|
LLMTextFrame,
|
||||||
StartInterruptionFrame,
|
StartInterruptionFrame,
|
||||||
)
|
)
|
||||||
from pipecat.observers.base_observer import BaseObserver
|
from pipecat.observers.base_observer import BaseObserver
|
||||||
from pipecat.observers.loggers.llm_log_observer import LLMLogObserver
|
|
||||||
from pipecat.pipeline.pipeline import Pipeline
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
from pipecat.pipeline.runner import PipelineRunner
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||||
@@ -71,6 +73,38 @@ class DebugObserver(BaseObserver):
|
|||||||
logger.info(f"🤖 BOT STOP SPEAKING: {src} {arrow} {dst} at {time_sec:.2f}s")
|
logger.info(f"🤖 BOT STOP SPEAKING: {src} {arrow} {dst} at {time_sec:.2f}s")
|
||||||
|
|
||||||
|
|
||||||
|
class LLMLogObserver(BaseObserver):
|
||||||
|
"""Observer to log LLM activity to the console.
|
||||||
|
|
||||||
|
Logs all frame instances of:
|
||||||
|
- LLMFullResponseStartFrame (only from LLM service)
|
||||||
|
- LLMTextFrame
|
||||||
|
- LLMFullResponseEndFrame (only from LLM service)
|
||||||
|
|
||||||
|
This allows you to track when the LLM starts responding, what it generates, and when it finishes.
|
||||||
|
Log format: [LLM EVENT]: [details] at [timestamp]s
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def on_push_frame(
|
||||||
|
self,
|
||||||
|
src: FrameProcessor,
|
||||||
|
dst: FrameProcessor,
|
||||||
|
frame: Frame,
|
||||||
|
direction: FrameDirection,
|
||||||
|
timestamp: int,
|
||||||
|
):
|
||||||
|
time_sec = timestamp / 1_000_000_000
|
||||||
|
|
||||||
|
# Only log start/end frames from OpenAILLMService
|
||||||
|
if isinstance(frame, (LLMFullResponseStartFrame, LLMFullResponseEndFrame)):
|
||||||
|
if isinstance(src, OpenAILLMService):
|
||||||
|
event = "START" if isinstance(frame, LLMFullResponseStartFrame) else "END"
|
||||||
|
logger.info(f"🧠 LLM {event} RESPONSE at {time_sec:.2f}s")
|
||||||
|
# Log all LLMTextFrames
|
||||||
|
elif isinstance(frame, LLMTextFrame):
|
||||||
|
logger.info(f"🧠 LLM GENERATING: {frame.text!r} at {time_sec:.2f}s")
|
||||||
|
|
||||||
|
|
||||||
async def main():
|
async def main():
|
||||||
async with aiohttp.ClientSession() as session:
|
async with aiohttp.ClientSession() as session:
|
||||||
(room_url, token) = await configure(session)
|
(room_url, token) = await configure(session)
|
||||||
@@ -117,13 +151,13 @@ async def main():
|
|||||||
|
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
report_only_initial_ttfb=True,
|
report_only_initial_ttfb=True,
|
||||||
|
observers=[DebugObserver(), LLMLogObserver()],
|
||||||
),
|
),
|
||||||
observers=[DebugObserver(), LLMLogObserver()],
|
|
||||||
)
|
)
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ async def main():
|
|||||||
|
|
||||||
pipeline = Pipeline([NullProcessor()])
|
pipeline = Pipeline([NullProcessor()])
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(enable_heartbeats=True))
|
task = PipelineTask(pipeline, PipelineParams(enable_heartbeats=True))
|
||||||
|
|
||||||
runner = PipelineRunner()
|
runner = PipelineRunner()
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
#
|
#
|
||||||
# Copyright (c) 2024-2025, Daily
|
# Copyright (c) 2024, Daily
|
||||||
#
|
#
|
||||||
# SPDX-License-Identifier: BSD 2-Clause License
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
#
|
#
|
||||||
@@ -38,7 +38,7 @@ search_tool = {"google_search_retrieval": {}}
|
|||||||
tools = [search_tool]
|
tools = [search_tool]
|
||||||
|
|
||||||
system_instruction = """
|
system_instruction = """
|
||||||
You are an expert at providing the most recent news from any place. Your responses will be converted to audio, so avoid using special characters or overly complex formatting.
|
You are an expert at providing the most recent news from any place. Your responses will be converted to audio, so avoid using special characters or overly complex formatting.
|
||||||
|
|
||||||
Always use the google search API to retrieve the latest news. You must also use it to check which day is today.
|
Always use the google search API to retrieve the latest news. You must also use it to check which day is today.
|
||||||
|
|
||||||
@@ -117,7 +117,7 @@ async def main():
|
|||||||
]
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
|
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
|
||||||
|
|
||||||
@transport.event_handler("on_first_participant_joined")
|
@transport.event_handler("on_first_participant_joined")
|
||||||
async def on_first_participant_joined(transport, participant):
|
async def on_first_participant_joined(transport, participant):
|
||||||
|
|||||||
@@ -230,7 +230,7 @@ Your response will be turned into speech so use only simple words and punctuatio
|
|||||||
)
|
)
|
||||||
task = PipelineTask(
|
task = PipelineTask(
|
||||||
pipeline,
|
pipeline,
|
||||||
params=PipelineParams(
|
PipelineParams(
|
||||||
allow_interruptions=True,
|
allow_interruptions=True,
|
||||||
enable_metrics=True,
|
enable_metrics=True,
|
||||||
enable_usage_metrics=True,
|
enable_usage_metrics=True,
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user