Merge pull request #1290 from pipecat-ai/aiortc_example
P2P WebRTC transport option to Pipecat
This commit is contained in:
Filipi da Silva Fuchter
@@ -9,6 +9,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- Added `SmallWebRTCTransport`, a new P2P WebRTC transport.
|
||||
- Created two examples in `p2p-webrtc`:
|
||||
- **video-transform**: Demonstrates sending and receiving audio/video with `SmallWebRTCTransport` using `TypeScript`.
|
||||
Includes video frame processing with OpenCV.
|
||||
- **voice-agent**: A minimal example of creating a voice agent with `SmallWebRTCTransport`.
|
||||
|
||||
- Added support to `ProtobufFrameSerializer` to send the messages from `TransportMessageFrame` and `TransportMessageUrgentFrame`.
|
||||
|
||||
- Added support for a new TTS service, `PiperTTSService`.
|
||||
|
||||
59
examples/p2p-webrtc/video-transform/README.md
Normal file
59
examples/p2p-webrtc/video-transform/README.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# Video Transform
|
||||
|
||||
A Pipecat example demonstrating how to send and receive audio and video using `SmallWebRTCTransport`. This project also applies image processing to video frames using OpenCV.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 1️⃣ Start the Bot Server
|
||||
|
||||
#### 📂 Navigate to the Server Directory
|
||||
```bash
|
||||
cd server
|
||||
```
|
||||
|
||||
#### 🔧 Set Up the Environment
|
||||
1. Create and activate a virtual environment:
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
```
|
||||
|
||||
2. Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. Configure environment variables:
|
||||
- Copy `env.example` to `.env`
|
||||
```bash
|
||||
cp env.example .env
|
||||
```
|
||||
- Add your API keys
|
||||
|
||||
#### ▶️ Run the Server
|
||||
```bash
|
||||
python server.py
|
||||
```
|
||||
|
||||
### 2️⃣ Connect Using the Client App
|
||||
|
||||
For client-side setup, refer to the [JavaScript Guide](client/typescript/README.md).
|
||||
|
||||
## ⚠️ Important Note
|
||||
Ensure the bot server is running before using any client implementations.
|
||||
|
||||
## 📌 Requirements
|
||||
|
||||
- Python **3.10+**
|
||||
- Node.js **16+** (for JavaScript components)
|
||||
- Google API Key
|
||||
- Modern web browser with WebRTC support
|
||||
|
||||
---
|
||||
|
||||
### 💡 Notes
|
||||
- Ensure all dependencies are installed before running the server.
|
||||
- Check the `.env` file for missing configurations.
|
||||
- WebRTC requires a secure environment (HTTPS) for full functionality in production.
|
||||
|
||||
Happy coding! 🎉
|
||||
@@ -0,0 +1,27 @@
|
||||
# JavaScript Implementation
|
||||
|
||||
Basic implementation using the [Pipecat JavaScript SDK](https://docs.pipecat.ai/client/js/introduction).
|
||||
|
||||
## Setup
|
||||
|
||||
1. Run the bot server. See the [server README](../../README).
|
||||
|
||||
2. Navigate to the `client/typescript` directory:
|
||||
|
||||
```bash
|
||||
cd client/typescript
|
||||
```
|
||||
|
||||
3. Install dependencies:
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
4. Run the client app:
|
||||
|
||||
```
|
||||
npm run dev
|
||||
```
|
||||
|
||||
5. Visit http://localhost:5173 in your browser.
|
||||
@@ -0,0 +1,68 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8"/>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>WebRTC demo</title>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<div class="container">
|
||||
<!-- Settings Bar -->
|
||||
<div class="status-bar">
|
||||
<div class="option">
|
||||
<label>Audio</label>
|
||||
<select id="audio-input">
|
||||
<option value="" selected>Default device</option>
|
||||
</select>
|
||||
<select id="audio-codec">
|
||||
<option value="default" selected>Default codecs</option>
|
||||
<option value="opus/48000/2">Opus</option>
|
||||
<option value="PCMU/8000">PCMU</option>
|
||||
<option value="PCMA/8000">PCMA</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="option">
|
||||
<label>Video</label>
|
||||
<select id="video-input">
|
||||
<option value="" selected>Default device</option>
|
||||
</select>
|
||||
<select id="video-codec">
|
||||
<option value="default" selected>Default codecs</option>
|
||||
<option value="VP8/90000">VP8</option>
|
||||
<option value="H264/90000">H264</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Status Bar -->
|
||||
<div class="status-bar">
|
||||
<div class="status">
|
||||
Status: <span id="connection-status">Disconnected</span>
|
||||
</div>
|
||||
<div class="controls">
|
||||
<button id="connect-btn">Connect</button>
|
||||
<button id="disconnect-btn" disabled>Disconnect</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Main Content -->
|
||||
<div class="main-content">
|
||||
<div class="bot-container">
|
||||
<div id="bot-video-container">
|
||||
<video id="bot-video" autoplay="true" playsinline="true"></video>
|
||||
</div>
|
||||
<audio id="bot-audio" autoplay></audio>
|
||||
</div>
|
||||
<!-- Debug Panel -->
|
||||
<div class="debug-panel">
|
||||
<div id="debug-log"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script type="module" src="/src/app.ts"></script>
|
||||
<link rel="stylesheet" href="/src/style.css">
|
||||
|
||||
</body>
|
||||
</html>
|
||||
668
examples/p2p-webrtc/video-transform/client/typescript/package-lock.json
generated
Normal file
668
examples/p2p-webrtc/video-transform/client/typescript/package-lock.json
generated
Normal file
@@ -0,0 +1,668 @@
|
||||
{
|
||||
"name": "client",
|
||||
"version": "1.0.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "client",
|
||||
"version": "1.0.0",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"@pipecat-ai/client-js": "^0.3.2",
|
||||
"@pipecat-ai/small-webrtc-transport": "^0.0.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.13.1",
|
||||
"@vitejs/plugin-react-swc": "^3.7.2",
|
||||
"typescript": "^5.7.3",
|
||||
"vite": "^6.0.2"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/runtime": {
|
||||
"version": "7.27.0",
|
||||
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.27.0.tgz",
|
||||
"integrity": "sha512-VtPOkrdPHZsKc/clNqyi9WUA8TINkZ4cGk63UUE3u4pmB2k+ZMQRDuIOagv8UVd6j7k0T3+RRIb7beKTebNbcw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"regenerator-runtime": "^0.14.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@daily-co/daily-js": {
|
||||
"version": "0.73.0",
|
||||
"resolved": "https://registry.npmjs.org/@daily-co/daily-js/-/daily-js-0.73.0.tgz",
|
||||
"integrity": "sha512-Wz8c60hgmkx8fcEeDAi4L4J0rbafiihWKyXFyhYoFYPsw2OdChHpA4RYwIB+1enRws5IK+/HdmzFDYLQsB4A6w==",
|
||||
"license": "BSD-2-Clause",
|
||||
"dependencies": {
|
||||
"@babel/runtime": "^7.12.5",
|
||||
"@sentry/browser": "^8.33.1",
|
||||
"bowser": "^2.8.1",
|
||||
"dequal": "^2.0.3",
|
||||
"events": "^3.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/darwin-arm64": {
|
||||
"version": "0.24.0",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.24.0.tgz",
|
||||
"integrity": "sha512-CKyDpRbK1hXwv79soeTJNHb5EiG6ct3efd/FTPdzOWdbZZfGhpbcqIpiD0+vwmpu0wTIL97ZRPZu8vUt46nBSw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@pipecat-ai/client-js": {
|
||||
"version": "0.3.5",
|
||||
"resolved": "https://registry.npmjs.org/@pipecat-ai/client-js/-/client-js-0.3.5.tgz",
|
||||
"integrity": "sha512-qmhnDjwY2XUtLjww35ShsYf5TF9BCuAk0tIj0oHjpTe6v6QOlgKQt8JVCAdc32p5ycouzSZOeDFtBd2aNWuq1g==",
|
||||
"license": "BSD-2-Clause",
|
||||
"dependencies": {
|
||||
"@types/events": "^3.0.3",
|
||||
"clone-deep": "^4.0.1",
|
||||
"events": "^3.3.0",
|
||||
"typed-emitter": "^2.1.0",
|
||||
"uuid": "^10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@pipecat-ai/small-webrtc-transport": {
|
||||
"version": "0.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@pipecat-ai/small-webrtc-transport/-/small-webrtc-transport-0.0.1.tgz",
|
||||
"integrity": "sha512-WAOI7lT0V7cYOn0+qwUAryGxcOGe+wPVPEPzkR3qsM5GWIZ73spykZnuOndQGycq4UkcXVawCzERfNhpi+Uv7A==",
|
||||
"license": "BSD-2-Clause",
|
||||
"dependencies": {
|
||||
"@daily-co/daily-js": "^0.73.0",
|
||||
"dequal": "^2.0.3"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@pipecat-ai/client-js": "~0.3.5"
|
||||
}
|
||||
},
|
||||
"node_modules/@rollup/rollup-darwin-arm64": {
|
||||
"version": "4.28.0",
|
||||
"resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.28.0.tgz",
|
||||
"integrity": "sha512-lmKx9yHsppblnLQZOGxdO66gT77bvdBtr/0P+TPOseowE7D9AJoBw8ZDULRasXRWf1Z86/gcOdpBrV6VDUY36Q==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
]
|
||||
},
|
||||
"node_modules/@sentry-internal/browser-utils": {
|
||||
"version": "8.55.0",
|
||||
"resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-8.55.0.tgz",
|
||||
"integrity": "sha512-ROgqtQfpH/82AQIpESPqPQe0UyWywKJsmVIqi3c5Fh+zkds5LUxnssTj3yNd1x+kxaPDVB023jAP+3ibNgeNDw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@sentry/core": "8.55.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.18"
|
||||
}
|
||||
},
|
||||
"node_modules/@sentry-internal/feedback": {
|
||||
"version": "8.55.0",
|
||||
"resolved": "https://registry.npmjs.org/@sentry-internal/feedback/-/feedback-8.55.0.tgz",
|
||||
"integrity": "sha512-cP3BD/Q6pquVQ+YL+rwCnorKuTXiS9KXW8HNKu4nmmBAyf7urjs+F6Hr1k9MXP5yQ8W3yK7jRWd09Yu6DHWOiw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@sentry/core": "8.55.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.18"
|
||||
}
|
||||
},
|
||||
"node_modules/@sentry-internal/replay": {
|
||||
"version": "8.55.0",
|
||||
"resolved": "https://registry.npmjs.org/@sentry-internal/replay/-/replay-8.55.0.tgz",
|
||||
"integrity": "sha512-roCDEGkORwolxBn8xAKedybY+Jlefq3xYmgN2fr3BTnsXjSYOPC7D1/mYqINBat99nDtvgFvNfRcZPiwwZ1hSw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@sentry-internal/browser-utils": "8.55.0",
|
||||
"@sentry/core": "8.55.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.18"
|
||||
}
|
||||
},
|
||||
"node_modules/@sentry-internal/replay-canvas": {
|
||||
"version": "8.55.0",
|
||||
"resolved": "https://registry.npmjs.org/@sentry-internal/replay-canvas/-/replay-canvas-8.55.0.tgz",
|
||||
"integrity": "sha512-nIkfgRWk1091zHdu4NbocQsxZF1rv1f7bbp3tTIlZYbrH62XVZosx5iHAuZG0Zc48AETLE7K4AX9VGjvQj8i9w==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@sentry-internal/replay": "8.55.0",
|
||||
"@sentry/core": "8.55.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.18"
|
||||
}
|
||||
},
|
||||
"node_modules/@sentry/browser": {
|
||||
"version": "8.55.0",
|
||||
"resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-8.55.0.tgz",
|
||||
"integrity": "sha512-1A31mCEWCjaMxJt6qGUK+aDnLDcK6AwLAZnqpSchNysGni1pSn1RWSmk9TBF8qyTds5FH8B31H480uxMPUJ7Cw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@sentry-internal/browser-utils": "8.55.0",
|
||||
"@sentry-internal/feedback": "8.55.0",
|
||||
"@sentry-internal/replay": "8.55.0",
|
||||
"@sentry-internal/replay-canvas": "8.55.0",
|
||||
"@sentry/core": "8.55.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.18"
|
||||
}
|
||||
},
|
||||
"node_modules/@sentry/core": {
|
||||
"version": "8.55.0",
|
||||
"resolved": "https://registry.npmjs.org/@sentry/core/-/core-8.55.0.tgz",
|
||||
"integrity": "sha512-6g7jpbefjHYs821Z+EBJ8r4Z7LT5h80YSWRJaylGS4nW5W5Z2KXzpdnyFarv37O7QjauzVC2E+PABmpkw5/JGA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=14.18"
|
||||
}
|
||||
},
|
||||
"node_modules/@swc/core": {
|
||||
"version": "1.10.14",
|
||||
"resolved": "https://registry.npmjs.org/@swc/core/-/core-1.10.14.tgz",
|
||||
"integrity": "sha512-WSrnE6JRnH20ZYjOOgSS4aOaPv9gxlkI2KRkN24kagbZnPZMnN8bZZyzw1rrLvwgpuRGv17Uz+hflosbR+SP6w==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@swc/counter": "^0.1.3",
|
||||
"@swc/types": "^0.1.17"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/swc"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@swc/core-darwin-arm64": "1.10.14",
|
||||
"@swc/core-darwin-x64": "1.10.14",
|
||||
"@swc/core-linux-arm-gnueabihf": "1.10.14",
|
||||
"@swc/core-linux-arm64-gnu": "1.10.14",
|
||||
"@swc/core-linux-arm64-musl": "1.10.14",
|
||||
"@swc/core-linux-x64-gnu": "1.10.14",
|
||||
"@swc/core-linux-x64-musl": "1.10.14",
|
||||
"@swc/core-win32-arm64-msvc": "1.10.14",
|
||||
"@swc/core-win32-ia32-msvc": "1.10.14",
|
||||
"@swc/core-win32-x64-msvc": "1.10.14"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@swc/helpers": "*"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@swc/helpers": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/@swc/core-darwin-arm64": {
|
||||
"version": "1.10.14",
|
||||
"resolved": "https://registry.npmjs.org/@swc/core-darwin-arm64/-/core-darwin-arm64-1.10.14.tgz",
|
||||
"integrity": "sha512-Dh4VyrhDDb05tdRmqJ/MucOPMTnrB4pRJol18HVyLlqu1HOT5EzonUniNTCdQbUXjgdv5UVJSTE1lYTzrp+myA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"license": "Apache-2.0 AND MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
}
|
||||
},
|
||||
"node_modules/@swc/counter": {
|
||||
"version": "0.1.3",
|
||||
"resolved": "https://registry.npmjs.org/@swc/counter/-/counter-0.1.3.tgz",
|
||||
"integrity": "sha512-e2BR4lsJkkRlKZ/qCHPw9ZaSxc0MVUd7gtbtaB7aMvHeJVYe8sOB8DBZkP2DtISHGSku9sCK6T6cnY0CtXrOCQ==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@swc/types": {
|
||||
"version": "0.1.17",
|
||||
"resolved": "https://registry.npmjs.org/@swc/types/-/types-0.1.17.tgz",
|
||||
"integrity": "sha512-V5gRru+aD8YVyCOMAjMpWR1Ui577DD5KSJsHP8RAxopAH22jFz6GZd/qxqjO6MJHQhcsjvjOFXyDhyLQUnMveQ==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"@swc/counter": "^0.1.3"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/estree": {
|
||||
"version": "1.0.6",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.6.tgz",
|
||||
"integrity": "sha512-AYnb1nQyY49te+VRAVgmzfcgjYS91mY5P0TKUDCLEM+gNnA+3T6rWITXRLYCpahpqSQbN5cE+gHpnPyXjHWxcw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/events": {
|
||||
"version": "3.0.3",
|
||||
"resolved": "https://registry.npmjs.org/@types/events/-/events-3.0.3.tgz",
|
||||
"integrity": "sha512-trOc4AAUThEz9hapPtSd7wf5tiQKvTtu5b371UxXdTuqzIh0ArcRspRP0i0Viu+LXstIQ1z96t1nsPxT9ol01g==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "22.13.1",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-22.13.1.tgz",
|
||||
"integrity": "sha512-jK8uzQlrvXqEU91UxiK5J7pKHyzgnI1Qnl0QDHIgVGuolJhRb9EEl28Cj9b3rGR8B2lhFCtvIm5os8lFnO/1Ew==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~6.20.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@vitejs/plugin-react-swc": {
|
||||
"version": "3.7.2",
|
||||
"resolved": "https://registry.npmjs.org/@vitejs/plugin-react-swc/-/plugin-react-swc-3.7.2.tgz",
|
||||
"integrity": "sha512-y0byko2b2tSVVf5Gpng1eEhX1OvPC7x8yns1Fx8jDzlJp4LS6CMkCPfLw47cjyoMrshQDoQw4qcgjsU9VvlCew==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@swc/core": "^1.7.26"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"vite": "^4 || ^5 || ^6"
|
||||
}
|
||||
},
|
||||
"node_modules/bowser": {
|
||||
"version": "2.11.0",
|
||||
"resolved": "https://registry.npmjs.org/bowser/-/bowser-2.11.0.tgz",
|
||||
"integrity": "sha512-AlcaJBi/pqqJBIQ8U9Mcpc9i8Aqxn88Skv5d+xBX006BY5u8N3mGLHa5Lgppa7L/HfwgwLgZ6NYs+Ag6uUmJRA==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/clone-deep": {
|
||||
"version": "4.0.1",
|
||||
"resolved": "https://registry.npmjs.org/clone-deep/-/clone-deep-4.0.1.tgz",
|
||||
"integrity": "sha512-neHB9xuzh/wk0dIHweyAXv2aPGZIVk3pLMe+/RNzINf17fe0OG96QroktYAUm7SM1PBnzTabaLboqqxDyMU+SQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"is-plain-object": "^2.0.4",
|
||||
"kind-of": "^6.0.2",
|
||||
"shallow-clone": "^3.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/dequal": {
|
||||
"version": "2.0.3",
|
||||
"resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz",
|
||||
"integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/esbuild": {
|
||||
"version": "0.24.0",
|
||||
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.24.0.tgz",
|
||||
"integrity": "sha512-FuLPevChGDshgSicjisSooU0cemp/sGXR841D5LHMB7mTVOmsEHcAxaH3irL53+8YDIeVNQEySh4DaYU/iuPqQ==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"esbuild": "bin/esbuild"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@esbuild/aix-ppc64": "0.24.0",
|
||||
"@esbuild/android-arm": "0.24.0",
|
||||
"@esbuild/android-arm64": "0.24.0",
|
||||
"@esbuild/android-x64": "0.24.0",
|
||||
"@esbuild/darwin-arm64": "0.24.0",
|
||||
"@esbuild/darwin-x64": "0.24.0",
|
||||
"@esbuild/freebsd-arm64": "0.24.0",
|
||||
"@esbuild/freebsd-x64": "0.24.0",
|
||||
"@esbuild/linux-arm": "0.24.0",
|
||||
"@esbuild/linux-arm64": "0.24.0",
|
||||
"@esbuild/linux-ia32": "0.24.0",
|
||||
"@esbuild/linux-loong64": "0.24.0",
|
||||
"@esbuild/linux-mips64el": "0.24.0",
|
||||
"@esbuild/linux-ppc64": "0.24.0",
|
||||
"@esbuild/linux-riscv64": "0.24.0",
|
||||
"@esbuild/linux-s390x": "0.24.0",
|
||||
"@esbuild/linux-x64": "0.24.0",
|
||||
"@esbuild/netbsd-x64": "0.24.0",
|
||||
"@esbuild/openbsd-arm64": "0.24.0",
|
||||
"@esbuild/openbsd-x64": "0.24.0",
|
||||
"@esbuild/sunos-x64": "0.24.0",
|
||||
"@esbuild/win32-arm64": "0.24.0",
|
||||
"@esbuild/win32-ia32": "0.24.0",
|
||||
"@esbuild/win32-x64": "0.24.0"
|
||||
}
|
||||
},
|
||||
"node_modules/events": {
|
||||
"version": "3.3.0",
|
||||
"resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz",
|
||||
"integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.8.x"
|
||||
}
|
||||
},
|
||||
"node_modules/fsevents": {
|
||||
"version": "2.3.3",
|
||||
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
|
||||
"integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/is-plain-object": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/is-plain-object/-/is-plain-object-2.0.4.tgz",
|
||||
"integrity": "sha512-h5PpgXkWitc38BBMYawTYMWJHFZJVnBquFE57xFpjB8pJFiF6gZ+bU+WyI/yqXiFR5mdLsgYNaPe8uao6Uv9Og==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"isobject": "^3.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/isobject": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/isobject/-/isobject-3.0.1.tgz",
|
||||
"integrity": "sha512-WhB9zCku7EGTj/HQQRz5aUQEUeoQZH2bWcltRErOpymJ4boYE6wL9Tbr23krRPSZ+C5zqNSrSw+Cc7sZZ4b7vg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/kind-of": {
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz",
|
||||
"integrity": "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/nanoid": {
|
||||
"version": "3.3.8",
|
||||
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.8.tgz",
|
||||
"integrity": "sha512-WNLf5Sd8oZxOm+TzppcYk8gVOgP+l58xNy58D0nbUnOxOWRWvlcCV4kUF7ltmI6PsrLl/BgKEyS4mqsGChFN0w==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/ai"
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"nanoid": "bin/nanoid.cjs"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/picocolors": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
|
||||
"integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==",
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/postcss": {
|
||||
"version": "8.4.49",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.4.49.tgz",
|
||||
"integrity": "sha512-OCVPnIObs4N29kxTjzLfUryOkvZEq+pf8jTF0lg8E7uETuWHA+v7j3c/xJmiqpX450191LlmZfUKkXxkTry7nA==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/postcss/"
|
||||
},
|
||||
{
|
||||
"type": "tidelift",
|
||||
"url": "https://tidelift.com/funding/github/npm/postcss"
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/ai"
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"nanoid": "^3.3.7",
|
||||
"picocolors": "^1.1.1",
|
||||
"source-map-js": "^1.2.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^10 || ^12 || >=14"
|
||||
}
|
||||
},
|
||||
"node_modules/regenerator-runtime": {
|
||||
"version": "0.14.1",
|
||||
"resolved": "https://registry.npmjs.org/regenerator-runtime/-/regenerator-runtime-0.14.1.tgz",
|
||||
"integrity": "sha512-dYnhHh0nJoMfnkZs6GmmhFknAGRrLznOu5nc9ML+EJxGvrx6H7teuevqVqCuPcPK//3eDrrjQhehXVx9cnkGdw==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/rollup": {
|
||||
"version": "4.28.0",
|
||||
"resolved": "https://registry.npmjs.org/rollup/-/rollup-4.28.0.tgz",
|
||||
"integrity": "sha512-G9GOrmgWHBma4YfCcX8PjH0qhXSdH8B4HDE2o4/jaxj93S4DPCIDoLcXz99eWMji4hB29UFCEd7B2gwGJDR9cQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/estree": "1.0.6"
|
||||
},
|
||||
"bin": {
|
||||
"rollup": "dist/bin/rollup"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18.0.0",
|
||||
"npm": ">=8.0.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@rollup/rollup-android-arm-eabi": "4.28.0",
|
||||
"@rollup/rollup-android-arm64": "4.28.0",
|
||||
"@rollup/rollup-darwin-arm64": "4.28.0",
|
||||
"@rollup/rollup-darwin-x64": "4.28.0",
|
||||
"@rollup/rollup-freebsd-arm64": "4.28.0",
|
||||
"@rollup/rollup-freebsd-x64": "4.28.0",
|
||||
"@rollup/rollup-linux-arm-gnueabihf": "4.28.0",
|
||||
"@rollup/rollup-linux-arm-musleabihf": "4.28.0",
|
||||
"@rollup/rollup-linux-arm64-gnu": "4.28.0",
|
||||
"@rollup/rollup-linux-arm64-musl": "4.28.0",
|
||||
"@rollup/rollup-linux-powerpc64le-gnu": "4.28.0",
|
||||
"@rollup/rollup-linux-riscv64-gnu": "4.28.0",
|
||||
"@rollup/rollup-linux-s390x-gnu": "4.28.0",
|
||||
"@rollup/rollup-linux-x64-gnu": "4.28.0",
|
||||
"@rollup/rollup-linux-x64-musl": "4.28.0",
|
||||
"@rollup/rollup-win32-arm64-msvc": "4.28.0",
|
||||
"@rollup/rollup-win32-ia32-msvc": "4.28.0",
|
||||
"@rollup/rollup-win32-x64-msvc": "4.28.0",
|
||||
"fsevents": "~2.3.2"
|
||||
}
|
||||
},
|
||||
"node_modules/rxjs": {
|
||||
"version": "7.8.1",
|
||||
"resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.1.tgz",
|
||||
"integrity": "sha512-AA3TVj+0A2iuIoQkWEK/tqFjBq2j+6PO6Y0zJcvzLAFhEFIO3HL0vls9hWLncZbAAbK0mar7oZ4V079I/qPMxg==",
|
||||
"license": "Apache-2.0",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/shallow-clone": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/shallow-clone/-/shallow-clone-3.0.1.tgz",
|
||||
"integrity": "sha512-/6KqX+GVUdqPuPPd2LxDDxzX6CAbjJehAAOKlNpqqUpAqPM6HeL8f+o3a+JsyGjn2lv0WY8UsTgUJjU9Ok55NA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"kind-of": "^6.0.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/source-map-js": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
|
||||
"integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/tslib": {
|
||||
"version": "2.8.1",
|
||||
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
|
||||
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
|
||||
"license": "0BSD",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/typed-emitter": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/typed-emitter/-/typed-emitter-2.1.0.tgz",
|
||||
"integrity": "sha512-g/KzbYKbH5C2vPkaXGu8DJlHrGKHLsM25Zg9WuC9pMGfuvT+X25tZQWo5fK1BjBm8+UrVE9LDCvaY0CQk+fXDA==",
|
||||
"license": "MIT",
|
||||
"optionalDependencies": {
|
||||
"rxjs": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/typescript": {
|
||||
"version": "5.7.3",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.7.3.tgz",
|
||||
"integrity": "sha512-84MVSjMEHP+FQRPy3pX9sTVV/INIex71s9TL2Gm5FG/WG1SqXeKyZ0k7/blY/4FdOzI12CBy1vGc4og/eus0fw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.17"
|
||||
}
|
||||
},
|
||||
"node_modules/undici-types": {
|
||||
"version": "6.20.0",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.20.0.tgz",
|
||||
"integrity": "sha512-Ny6QZ2Nju20vw1SRHe3d9jVu6gJ+4e3+MMpqu7pqE5HT6WsTSlce++GQmK5UXS8mzV8DSYHrQH+Xrf2jVcuKNg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/uuid": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/uuid/-/uuid-10.0.0.tgz",
|
||||
"integrity": "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ==",
|
||||
"funding": [
|
||||
"https://github.com/sponsors/broofa",
|
||||
"https://github.com/sponsors/ctavan"
|
||||
],
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"uuid": "dist/bin/uuid"
|
||||
}
|
||||
},
|
||||
"node_modules/vite": {
|
||||
"version": "6.0.2",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-6.0.2.tgz",
|
||||
"integrity": "sha512-XdQ+VsY2tJpBsKGs0wf3U/+azx8BBpYRHFAyKm5VeEZNOJZRB63q7Sc8Iup3k0TrN3KO6QgyzFf+opSbfY1y0g==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"esbuild": "^0.24.0",
|
||||
"postcss": "^8.4.49",
|
||||
"rollup": "^4.23.0"
|
||||
},
|
||||
"bin": {
|
||||
"vite": "bin/vite.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/vitejs/vite?sponsor=1"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"fsevents": "~2.3.3"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0",
|
||||
"jiti": ">=1.21.0",
|
||||
"less": "*",
|
||||
"lightningcss": "^1.21.0",
|
||||
"sass": "*",
|
||||
"sass-embedded": "*",
|
||||
"stylus": "*",
|
||||
"sugarss": "*",
|
||||
"terser": "^5.16.0",
|
||||
"tsx": "^4.8.1",
|
||||
"yaml": "^2.4.2"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@types/node": {
|
||||
"optional": true
|
||||
},
|
||||
"jiti": {
|
||||
"optional": true
|
||||
},
|
||||
"less": {
|
||||
"optional": true
|
||||
},
|
||||
"lightningcss": {
|
||||
"optional": true
|
||||
},
|
||||
"sass": {
|
||||
"optional": true
|
||||
},
|
||||
"sass-embedded": {
|
||||
"optional": true
|
||||
},
|
||||
"stylus": {
|
||||
"optional": true
|
||||
},
|
||||
"sugarss": {
|
||||
"optional": true
|
||||
},
|
||||
"terser": {
|
||||
"optional": true
|
||||
},
|
||||
"tsx": {
|
||||
"optional": true
|
||||
},
|
||||
"yaml": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"name": "client",
|
||||
"version": "1.0.0",
|
||||
"main": "index.js",
|
||||
"scripts": {
|
||||
"dev": "node_modules/.bin/vite",
|
||||
"build": "node_modules/.bin/tsc && vite build",
|
||||
"preview": "node_modules/.bin/vite preview"
|
||||
},
|
||||
"keywords": [],
|
||||
"author": "",
|
||||
"license": "ISC",
|
||||
"description": "",
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.13.1",
|
||||
"@vitejs/plugin-react-swc": "^3.7.2",
|
||||
"typescript": "^5.7.3",
|
||||
"vite": "^6.0.2"
|
||||
},
|
||||
"dependencies": {
|
||||
"@pipecat-ai/client-js": "^0.3.2",
|
||||
"@pipecat-ai/small-webrtc-transport": "^0.0.1"
|
||||
}
|
||||
}
|
||||
212
examples/p2p-webrtc/video-transform/client/typescript/src/app.ts
Normal file
212
examples/p2p-webrtc/video-transform/client/typescript/src/app.ts
Normal file
@@ -0,0 +1,212 @@
|
||||
import {
|
||||
SmallWebRTCTransport
|
||||
} from "@pipecat-ai/small-webrtc-transport";
|
||||
import {Participant, RTVIClient, RTVIClientOptions} from "@pipecat-ai/client-js";
|
||||
|
||||
class WebRTCApp {
|
||||
|
||||
private declare connectBtn: HTMLButtonElement;
|
||||
private declare disconnectBtn: HTMLButtonElement;
|
||||
|
||||
private declare audioInput: HTMLSelectElement;
|
||||
private declare videoInput: HTMLSelectElement;
|
||||
private declare audioCodec: HTMLSelectElement;
|
||||
private declare videoCodec: HTMLSelectElement;
|
||||
|
||||
private declare videoElement: HTMLVideoElement;
|
||||
private declare audioElement: HTMLAudioElement;
|
||||
|
||||
private debugLog: HTMLElement | null = null;
|
||||
private statusSpan: HTMLElement | null = null;
|
||||
|
||||
private declare smallWebRTCTransport: SmallWebRTCTransport;
|
||||
private declare rtviClient: RTVIClient;
|
||||
|
||||
constructor() {
|
||||
this.setupDOMElements();
|
||||
this.setupDOMEventListeners();
|
||||
this.initializeRTVIClient()
|
||||
void this.populateDevices();
|
||||
}
|
||||
|
||||
private initializeRTVIClient(): void {
|
||||
const transport = new SmallWebRTCTransport();
|
||||
const RTVIConfig: RTVIClientOptions = {
|
||||
// need to understand why it is complaining
|
||||
// @ts-ignore
|
||||
transport,
|
||||
params: {
|
||||
baseUrl: "/api/offer"
|
||||
},
|
||||
enableMic: true,
|
||||
enableCam: true,
|
||||
callbacks: {
|
||||
onTransportStateChanged: (state) => {
|
||||
this.log(`Transport state: ${state}`)
|
||||
},
|
||||
onConnected: () => {
|
||||
this.onConnectedHandler()
|
||||
},
|
||||
onBotReady: () => {
|
||||
this.log("Bot is ready.")
|
||||
},
|
||||
onDisconnected: () => {
|
||||
this.onDisconnectedHandler()
|
||||
},
|
||||
onUserStartedSpeaking: () => {
|
||||
this.log("User started speaking.")
|
||||
},
|
||||
onUserStoppedSpeaking: () => {
|
||||
this.log("User stopped speaking.")
|
||||
},
|
||||
onBotStartedSpeaking: () => {
|
||||
this.log("Bot started speaking.")
|
||||
},
|
||||
onBotStoppedSpeaking: () => {
|
||||
this.log("Bot stopped speaking.")
|
||||
},
|
||||
onUserTranscript: (transcript) => {
|
||||
if (transcript.final) {
|
||||
this.log(`User transcript: ${transcript.text}`)
|
||||
}
|
||||
},
|
||||
onBotTranscript: (transcript) => {
|
||||
this.log(`Bot transcript: ${transcript.text}`)
|
||||
},
|
||||
onTrackStarted: (track: MediaStreamTrack, participant?: Participant) => {
|
||||
if (participant?.local) {
|
||||
return
|
||||
}
|
||||
this.onBotTrackStarted(track)
|
||||
},
|
||||
onServerMessage: (msg) => {
|
||||
this.log(`Server message: ${msg}`)
|
||||
}
|
||||
},
|
||||
}
|
||||
RTVIConfig.customConnectHandler = () => Promise.resolve();
|
||||
this.rtviClient = new RTVIClient(RTVIConfig);
|
||||
this.smallWebRTCTransport = transport
|
||||
}
|
||||
|
||||
private setupDOMElements(): void {
|
||||
this.connectBtn = document.getElementById('connect-btn') as HTMLButtonElement;
|
||||
this.disconnectBtn = document.getElementById('disconnect-btn') as HTMLButtonElement;
|
||||
|
||||
this.audioInput = document.getElementById('audio-input') as HTMLSelectElement;
|
||||
this.videoInput = document.getElementById('video-input') as HTMLSelectElement;
|
||||
this.audioCodec = document.getElementById('audio-codec') as HTMLSelectElement;
|
||||
this.videoCodec = document.getElementById('video-codec') as HTMLSelectElement;
|
||||
|
||||
this.videoElement = document.getElementById('bot-video') as HTMLVideoElement;
|
||||
this.audioElement = document.getElementById('bot-audio') as HTMLAudioElement;
|
||||
|
||||
this.debugLog = document.getElementById('debug-log');
|
||||
this.statusSpan = document.getElementById('connection-status');
|
||||
}
|
||||
|
||||
private setupDOMEventListeners(): void {
|
||||
this.connectBtn.addEventListener("click", () => this.start());
|
||||
this.disconnectBtn.addEventListener("click", () => this.stop());
|
||||
this.audioInput.addEventListener("change", (e) => {
|
||||
// @ts-ignore
|
||||
let audioDevice = e.target?.value
|
||||
this.rtviClient.updateMic(audioDevice)
|
||||
})
|
||||
this.videoInput.addEventListener("change", (e) => {
|
||||
// @ts-ignore
|
||||
let videoDevice = e.target?.value
|
||||
this.rtviClient.updateCam(videoDevice)
|
||||
})
|
||||
}
|
||||
|
||||
private log(message: string): void {
|
||||
if (!this.debugLog) return;
|
||||
const entry = document.createElement('div');
|
||||
entry.textContent = `${new Date().toISOString()} - ${message}`;
|
||||
if (message.startsWith('User: ')) {
|
||||
entry.style.color = '#2196F3';
|
||||
} else if (message.startsWith('Bot: ')) {
|
||||
entry.style.color = '#4CAF50';
|
||||
}
|
||||
this.debugLog.appendChild(entry);
|
||||
this.debugLog.scrollTop = this.debugLog.scrollHeight;
|
||||
}
|
||||
|
||||
private clearAllLogs() {
|
||||
this.debugLog!.innerText = ''
|
||||
}
|
||||
|
||||
private updateStatus(status: string): void {
|
||||
if (this.statusSpan) {
|
||||
this.statusSpan.textContent = status;
|
||||
}
|
||||
this.log(`Status: ${status}`);
|
||||
}
|
||||
|
||||
private onConnectedHandler() {
|
||||
this.updateStatus('Connected');
|
||||
if (this.connectBtn) this.connectBtn.disabled = true;
|
||||
if (this.disconnectBtn) this.disconnectBtn.disabled = false;
|
||||
}
|
||||
|
||||
private onDisconnectedHandler() {
|
||||
this.updateStatus('Disconnected');
|
||||
if (this.connectBtn) this.connectBtn.disabled = false;
|
||||
if (this.disconnectBtn) this.disconnectBtn.disabled = true;
|
||||
}
|
||||
|
||||
private onBotTrackStarted(track: MediaStreamTrack) {
|
||||
if (track.kind === 'video') {
|
||||
this.videoElement.srcObject = new MediaStream([track]);
|
||||
} else {
|
||||
this.audioElement.srcObject = new MediaStream([track]);
|
||||
}
|
||||
}
|
||||
|
||||
private async populateDevices(): Promise<void> {
|
||||
const populateSelect = (select: HTMLSelectElement, devices: MediaDeviceInfo[]): void => {
|
||||
let counter = 1;
|
||||
devices.forEach((device) => {
|
||||
const option = document.createElement('option');
|
||||
option.value = device.deviceId;
|
||||
option.text = device.label || ('Device #' + counter);
|
||||
select.appendChild(option);
|
||||
counter += 1;
|
||||
});
|
||||
};
|
||||
|
||||
try {
|
||||
const audioDevices = await this.rtviClient.getAllMics();
|
||||
populateSelect(this.audioInput, audioDevices);
|
||||
const videoDevices = await this.rtviClient.getAllCams();
|
||||
populateSelect(this.videoInput, videoDevices);
|
||||
} catch (e) {
|
||||
alert(e);
|
||||
}
|
||||
}
|
||||
|
||||
private async start(): Promise<void> {
|
||||
this.clearAllLogs()
|
||||
|
||||
this.connectBtn.disabled = true;
|
||||
this.updateStatus("Connecting")
|
||||
|
||||
this.smallWebRTCTransport.setAudioCodec(this.audioCodec.value)
|
||||
this.smallWebRTCTransport.setVideoCodec(this.videoCodec.value)
|
||||
try {
|
||||
await this.rtviClient.connect()
|
||||
} catch (e) {
|
||||
console.log(`Failed to connect ${e}`)
|
||||
this.stop()
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private stop(): void {
|
||||
void this.rtviClient.disconnect()
|
||||
}
|
||||
}
|
||||
|
||||
// Create the WebRTCConnection instance
|
||||
const webRTCConnection = new WebRTCApp();
|
||||
@@ -0,0 +1,120 @@
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
font-family: Arial, sans-serif;
|
||||
background-color: #f0f0f0;
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.container {
|
||||
margin: 0 auto;
|
||||
width: 90%;
|
||||
}
|
||||
|
||||
.option {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
label {
|
||||
margin: 5px;
|
||||
}
|
||||
|
||||
select {
|
||||
padding: 8px;
|
||||
margin: 10px;
|
||||
border-radius: 4px;
|
||||
border: 1px solid #ccc;
|
||||
}
|
||||
|
||||
.status-bar {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 10px;
|
||||
background-color: #fff;
|
||||
border-radius: 8px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.controls button {
|
||||
padding: 8px 16px;
|
||||
margin-left: 10px;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
#connect-btn {
|
||||
background-color: #4caf50;
|
||||
color: white;
|
||||
}
|
||||
|
||||
#disconnect-btn {
|
||||
background-color: #f44336;
|
||||
color: white;
|
||||
}
|
||||
|
||||
button:disabled {
|
||||
opacity: 0.5;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.main-content {
|
||||
background-color: #fff;
|
||||
border-radius: 8px;
|
||||
padding: 20px;
|
||||
margin-bottom: 20px;
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.bot-container {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
width: 50%;
|
||||
}
|
||||
|
||||
#bot-video-container {
|
||||
width: 640px;
|
||||
height: 360px;
|
||||
background-color: #e0e0e0;
|
||||
border-radius: 8px;
|
||||
overflow: hidden;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
#bot-video-container video {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
}
|
||||
|
||||
.debug-panel {
|
||||
background-color: #fff;
|
||||
border-radius: 8px;
|
||||
padding-left: 20px;
|
||||
width: 50%;
|
||||
}
|
||||
|
||||
.debug-panel h3 {
|
||||
margin: 0 0 10px 0;
|
||||
font-size: 16px;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
#debug-log {
|
||||
height: 500px;
|
||||
overflow-y: auto;
|
||||
background-color: #f8f8f8;
|
||||
padding: 10px;
|
||||
border-radius: 4px;
|
||||
font-family: monospace;
|
||||
font-size: 12px;
|
||||
line-height: 1.4;
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
/* Visit https://aka.ms/tsconfig to read more about this file */
|
||||
|
||||
/* Projects */
|
||||
// "incremental": true, /* Save .tsbuildinfo files to allow for incremental compilation of projects. */
|
||||
// "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */
|
||||
// "tsBuildInfoFile": "./.tsbuildinfo", /* Specify the path to .tsbuildinfo incremental compilation file. */
|
||||
// "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects. */
|
||||
// "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */
|
||||
// "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */
|
||||
|
||||
/* Language and Environment */
|
||||
"target": "es2016", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
|
||||
// "lib": [], /* Specify a set of bundled library declaration files that describe the target runtime environment. */
|
||||
// "jsx": "preserve", /* Specify what JSX code is generated. */
|
||||
// "experimentalDecorators": true, /* Enable experimental support for legacy experimental decorators. */
|
||||
// "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */
|
||||
// "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */
|
||||
// "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
|
||||
// "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */
|
||||
// "reactNamespace": "", /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */
|
||||
// "noLib": true, /* Disable including any library files, including the default lib.d.ts. */
|
||||
// "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */
|
||||
// "moduleDetection": "auto", /* Control what method is used to detect module-format JS files. */
|
||||
|
||||
/* Modules */
|
||||
"module": "commonjs", /* Specify what module code is generated. */
|
||||
// "rootDir": "./", /* Specify the root folder within your source files. */
|
||||
// "moduleResolution": "node10", /* Specify how TypeScript looks up a file from a given module specifier. */
|
||||
// "baseUrl": "./", /* Specify the base directory to resolve non-relative module names. */
|
||||
// "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */
|
||||
// "rootDirs": [], /* Allow multiple folders to be treated as one when resolving modules. */
|
||||
// "typeRoots": [], /* Specify multiple folders that act like './node_modules/@types'. */
|
||||
// "types": [], /* Specify type package names to be included without being referenced in a source file. */
|
||||
// "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */
|
||||
// "moduleSuffixes": [], /* List of file name suffixes to search when resolving a module. */
|
||||
// "allowImportingTsExtensions": true, /* Allow imports to include TypeScript file extensions. Requires '--moduleResolution bundler' and either '--noEmit' or '--emitDeclarationOnly' to be set. */
|
||||
// "rewriteRelativeImportExtensions": true, /* Rewrite '.ts', '.tsx', '.mts', and '.cts' file extensions in relative import paths to their JavaScript equivalent in output files. */
|
||||
// "resolvePackageJsonExports": true, /* Use the package.json 'exports' field when resolving package imports. */
|
||||
// "resolvePackageJsonImports": true, /* Use the package.json 'imports' field when resolving imports. */
|
||||
// "customConditions": [], /* Conditions to set in addition to the resolver-specific defaults when resolving imports. */
|
||||
// "noUncheckedSideEffectImports": true, /* Check side effect imports. */
|
||||
// "resolveJsonModule": true, /* Enable importing .json files. */
|
||||
// "allowArbitraryExtensions": true, /* Enable importing files with any extension, provided a declaration file is present. */
|
||||
// "noResolve": true, /* Disallow 'import's, 'require's or '<reference>'s from expanding the number of files TypeScript should add to a project. */
|
||||
|
||||
/* JavaScript Support */
|
||||
// "allowJs": true, /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */
|
||||
// "checkJs": true, /* Enable error reporting in type-checked JavaScript files. */
|
||||
// "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */
|
||||
|
||||
/* Emit */
|
||||
// "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
|
||||
// "declarationMap": true, /* Create sourcemaps for d.ts files. */
|
||||
// "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */
|
||||
// "sourceMap": true, /* Create source map files for emitted JavaScript files. */
|
||||
// "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */
|
||||
// "noEmit": true, /* Disable emitting files from a compilation. */
|
||||
// "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */
|
||||
// "outDir": "./", /* Specify an output folder for all emitted files. */
|
||||
// "removeComments": true, /* Disable emitting comments. */
|
||||
// "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
|
||||
// "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
|
||||
// "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */
|
||||
// "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */
|
||||
// "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */
|
||||
// "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
|
||||
// "newLine": "crlf", /* Set the newline character for emitting files. */
|
||||
// "stripInternal": true, /* Disable emitting declarations that have '@internal' in their JSDoc comments. */
|
||||
// "noEmitHelpers": true, /* Disable generating custom helper functions like '__extends' in compiled output. */
|
||||
// "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */
|
||||
// "preserveConstEnums": true, /* Disable erasing 'const enum' declarations in generated code. */
|
||||
// "declarationDir": "./", /* Specify the output directory for generated declaration files. */
|
||||
|
||||
/* Interop Constraints */
|
||||
// "isolatedModules": true, /* Ensure that each file can be safely transpiled without relying on other imports. */
|
||||
// "verbatimModuleSyntax": true, /* Do not transform or elide any imports or exports not marked as type-only, ensuring they are written in the output file's format based on the 'module' setting. */
|
||||
// "isolatedDeclarations": true, /* Require sufficient annotation on exports so other tools can trivially generate declaration files. */
|
||||
// "allowSyntheticDefaultImports": true, /* Allow 'import x from y' when a module doesn't have a default export. */
|
||||
"esModuleInterop": true, /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
|
||||
// "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
|
||||
"forceConsistentCasingInFileNames": true, /* Ensure that casing is correct in imports. */
|
||||
|
||||
/* Type Checking */
|
||||
"strict": true, /* Enable all strict type-checking options. */
|
||||
// "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied 'any' type. */
|
||||
// "strictNullChecks": true, /* When type checking, take into account 'null' and 'undefined'. */
|
||||
// "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
|
||||
// "strictBindCallApply": true, /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */
|
||||
// "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */
|
||||
// "strictBuiltinIteratorReturn": true, /* Built-in iterators are instantiated with a 'TReturn' type of 'undefined' instead of 'any'. */
|
||||
// "noImplicitThis": true, /* Enable error reporting when 'this' is given the type 'any'. */
|
||||
// "useUnknownInCatchVariables": true, /* Default catch clause variables as 'unknown' instead of 'any'. */
|
||||
// "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */
|
||||
// "noUnusedLocals": true, /* Enable error reporting when local variables aren't read. */
|
||||
// "noUnusedParameters": true, /* Raise an error when a function parameter isn't read. */
|
||||
// "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */
|
||||
// "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */
|
||||
// "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */
|
||||
// "noUncheckedIndexedAccess": true, /* Add 'undefined' to a type when accessed using an index. */
|
||||
// "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */
|
||||
// "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type. */
|
||||
// "allowUnusedLabels": true, /* Disable error reporting for unused labels. */
|
||||
// "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */
|
||||
|
||||
/* Completeness */
|
||||
// "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */
|
||||
"skipLibCheck": true /* Skip type checking all .d.ts files. */
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { defineConfig } from 'vite';
|
||||
import react from '@vitejs/plugin-react-swc';
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
server: {
|
||||
proxy: {
|
||||
// Proxy /api requests to the backend server
|
||||
'/api': {
|
||||
target: 'http://0.0.0.0:7860', // Replace with your backend URL
|
||||
changeOrigin: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
156
examples/p2p-webrtc/video-transform/server/bot.py
Normal file
156
examples/p2p-webrtc/video-transform/server/bot.py
Normal file
@@ -0,0 +1,156 @@
|
||||
#
|
||||
# Copyright (c) 2025, Daily
|
||||
#
|
||||
# SPDX-License-Identifier: BSD 2-Clause License
|
||||
#
|
||||
import os
|
||||
import sys
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
|
||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||
from pipecat.frames.frames import Frame, InputImageRawFrame, OutputImageRawFrame
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.pipeline.runner import PipelineRunner
|
||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||
from pipecat.processors.frameworks.rtvi import RTVIConfig, RTVIObserver, RTVIProcessor
|
||||
from pipecat.services.gemini_multimodal_live import GeminiMultimodalLiveLLMService
|
||||
from pipecat.transports.base_transport import TransportParams
|
||||
from pipecat.transports.network.small_webrtc import SmallWebRTCTransport
|
||||
|
||||
load_dotenv(override=True)
|
||||
|
||||
logger.remove(0)
|
||||
logger.add(sys.stderr, level="DEBUG")
|
||||
|
||||
|
||||
class EdgeDetectionProcessor(FrameProcessor):
|
||||
def __init__(self, camera_out_width, camera_out_height: int):
|
||||
super().__init__()
|
||||
self._camera_out_width = camera_out_width
|
||||
self._camera_out_height = camera_out_height
|
||||
|
||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
||||
await super().process_frame(frame, direction)
|
||||
|
||||
if isinstance(frame, InputImageRawFrame):
|
||||
# Convert bytes to NumPy array
|
||||
img = np.frombuffer(frame.image, dtype=np.uint8).reshape(
|
||||
(frame.size[1], frame.size[0], 3)
|
||||
)
|
||||
|
||||
# perform edge detection
|
||||
img = cv2.cvtColor(cv2.Canny(img, 100, 200), cv2.COLOR_GRAY2BGR)
|
||||
|
||||
# convert the size if needed
|
||||
desired_size = (self._camera_out_width, self._camera_out_height)
|
||||
if frame.size != desired_size:
|
||||
resized_image = cv2.resize(img, desired_size)
|
||||
frame = OutputImageRawFrame(resized_image.tobytes(), desired_size, frame.format)
|
||||
await self.push_frame(frame)
|
||||
else:
|
||||
await self.push_frame(
|
||||
OutputImageRawFrame(image=img.tobytes(), size=frame.size, format=frame.format)
|
||||
)
|
||||
else:
|
||||
await self.push_frame(frame, direction)
|
||||
|
||||
|
||||
SYSTEM_INSTRUCTION = f"""
|
||||
"You are Gemini Chatbot, a friendly, helpful robot.
|
||||
|
||||
Your goal is to demonstrate your capabilities in a succinct way.
|
||||
|
||||
Your output will be converted to audio so don't include special characters in your answers.
|
||||
|
||||
Respond to what the user said in a creative and helpful way. Keep your responses brief. One or two sentences at most.
|
||||
"""
|
||||
|
||||
|
||||
async def run_bot(webrtc_connection):
|
||||
transport_params = TransportParams(
|
||||
camera_in_enabled=True,
|
||||
camera_out_enabled=True,
|
||||
camera_out_is_live=True,
|
||||
audio_in_enabled=True,
|
||||
audio_out_enabled=True,
|
||||
vad_enabled=True,
|
||||
vad_analyzer=SileroVADAnalyzer(),
|
||||
vad_audio_passthrough=True,
|
||||
)
|
||||
|
||||
pipecat_transport = SmallWebRTCTransport(
|
||||
webrtc_connection=webrtc_connection, params=transport_params
|
||||
)
|
||||
|
||||
llm = GeminiMultimodalLiveLLMService(
|
||||
api_key=os.getenv("GOOGLE_API_KEY"),
|
||||
voice_id="Puck", # Aoede, Charon, Fenrir, Kore, Puck
|
||||
transcribe_user_audio=True,
|
||||
transcribe_model_audio=True,
|
||||
system_instruction=SYSTEM_INSTRUCTION,
|
||||
)
|
||||
|
||||
context = OpenAILLMContext(
|
||||
[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Start by greeting the user warmly and introducing yourself.",
|
||||
}
|
||||
],
|
||||
)
|
||||
context_aggregator = llm.create_context_aggregator(context)
|
||||
|
||||
# RTVI events for Pipecat client UI
|
||||
rtvi = RTVIProcessor(config=RTVIConfig(config=[]))
|
||||
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
pipecat_transport.input(),
|
||||
context_aggregator.user(),
|
||||
rtvi,
|
||||
llm, # LLM
|
||||
EdgeDetectionProcessor(
|
||||
transport_params.camera_out_width, transport_params.camera_out_height
|
||||
), # Sending the video back to the user
|
||||
pipecat_transport.output(),
|
||||
context_aggregator.assistant(),
|
||||
]
|
||||
)
|
||||
|
||||
task = PipelineTask(
|
||||
pipeline,
|
||||
params=PipelineParams(
|
||||
allow_interruptions=True,
|
||||
observers=[RTVIObserver(rtvi)],
|
||||
),
|
||||
)
|
||||
|
||||
@rtvi.event_handler("on_client_ready")
|
||||
async def on_client_ready(rtvi):
|
||||
logger.info("Pipecat client ready.")
|
||||
await rtvi.set_bot_ready()
|
||||
|
||||
@pipecat_transport.event_handler("on_client_connected")
|
||||
async def on_client_connected(transport, client):
|
||||
logger.info("Pipecat Client connected")
|
||||
# Kick off the conversation.
|
||||
await task.queue_frames([context_aggregator.user().get_context_frame()])
|
||||
|
||||
@pipecat_transport.event_handler("on_client_disconnected")
|
||||
async def on_client_disconnected(transport, client):
|
||||
logger.info("Pipecat Client disconnected")
|
||||
|
||||
@pipecat_transport.event_handler("on_client_closed")
|
||||
async def on_client_closed(transport, client):
|
||||
logger.info("Pipecat Client closed")
|
||||
await task.cancel()
|
||||
|
||||
runner = PipelineRunner(handle_sigint=False)
|
||||
|
||||
await runner.run(task)
|
||||
1
examples/p2p-webrtc/video-transform/server/env.example
Normal file
1
examples/p2p-webrtc/video-transform/server/env.example
Normal file
@@ -0,0 +1 @@
|
||||
GOOGLE_API_KEY=
|
||||
@@ -0,0 +1,6 @@
|
||||
python-dotenv
|
||||
fastapi[all]
|
||||
uvicorn
|
||||
aiortc
|
||||
opencv-python
|
||||
pipecat-ai[google,silero]
|
||||
79
examples/p2p-webrtc/video-transform/server/server.py
Normal file
79
examples/p2p-webrtc/video-transform/server/server.py
Normal file
@@ -0,0 +1,79 @@
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Dict
|
||||
|
||||
import uvicorn
|
||||
from bot import run_bot
|
||||
from dotenv import load_dotenv
|
||||
from fastapi import BackgroundTasks, FastAPI
|
||||
|
||||
from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv(override=True)
|
||||
|
||||
logger = logging.getLogger("pc")
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
# Store connections by pc_id
|
||||
pcs_map: Dict[str, SmallWebRTCConnection] = {}
|
||||
|
||||
ice_servers = ["stun:stun.l.google.com:19302"]
|
||||
|
||||
|
||||
@app.post("/api/offer")
|
||||
async def offer(request: dict, background_tasks: BackgroundTasks):
|
||||
pc_id = request.get("pc_id")
|
||||
|
||||
if pc_id and pc_id in pcs_map:
|
||||
pipecat_connection = pcs_map[pc_id]
|
||||
logger.info(f"Reusing existing connection for pc_id: {pc_id}")
|
||||
await pipecat_connection.renegotiate(
|
||||
sdp=request["sdp"], type=request["type"], restart_pc=request.get("restart_pc", False)
|
||||
)
|
||||
else:
|
||||
pipecat_connection = SmallWebRTCConnection(ice_servers)
|
||||
await pipecat_connection.initialize(sdp=request["sdp"], type=request["type"])
|
||||
|
||||
@pipecat_connection.on("closed")
|
||||
async def handle_disconnected(webrtc_connection: SmallWebRTCConnection):
|
||||
logger.info(f"Discarding peer connection for pc_id: {webrtc_connection.pc_id}")
|
||||
pcs_map.pop(webrtc_connection.pc_id, None)
|
||||
|
||||
background_tasks.add_task(run_bot, pipecat_connection)
|
||||
|
||||
answer = pipecat_connection.get_answer()
|
||||
# Updating the peer connection inside the map
|
||||
pcs_map[answer["pc_id"]] = pipecat_connection
|
||||
|
||||
return answer
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
yield # Run app
|
||||
coros = [pc.close() for pc in pcs_map.values()]
|
||||
await asyncio.gather(*coros)
|
||||
pcs_map.clear()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="WebRTC demo")
|
||||
parser.add_argument(
|
||||
"--host", default="localhost", help="Host for HTTP server (default: localhost)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port", type=int, default=7860, help="Port for HTTP server (default: 7860)"
|
||||
)
|
||||
parser.add_argument("--verbose", "-v", action="count")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.verbose:
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
else:
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
uvicorn.run(app, host=args.host, port=args.port)
|
||||
54
examples/p2p-webrtc/voice-agent/README.md
Normal file
54
examples/p2p-webrtc/voice-agent/README.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Voice Agent
|
||||
|
||||
A Pipecat example demonstrating the simplest way to create a voice agent using `SmallWebRTCTransport`.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 1️⃣ Start the Bot Server
|
||||
|
||||
#### 🔧 Set Up the Environment
|
||||
1. Create and activate a virtual environment:
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
```
|
||||
|
||||
2. Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. Configure environment variables:
|
||||
- Copy `env.example` to `.env`
|
||||
```bash
|
||||
cp env.example .env
|
||||
```
|
||||
- Add your API keys
|
||||
|
||||
#### ▶️ Run the Server
|
||||
```bash
|
||||
python server.py
|
||||
```
|
||||
|
||||
### 2️⃣ Connect Using the Client App
|
||||
|
||||
Open your browser and visit:
|
||||
```
|
||||
http://localhost:7860
|
||||
```
|
||||
|
||||
## 📌 Requirements
|
||||
|
||||
- Python **3.10+**
|
||||
- Node.js **16+** (for JavaScript components)
|
||||
- Google API Key
|
||||
- Modern web browser with WebRTC support
|
||||
|
||||
---
|
||||
|
||||
### 💡 Notes
|
||||
- Ensure all dependencies are installed before running the server.
|
||||
- Check the `.env` file for missing configurations.
|
||||
- WebRTC requires a secure environment (HTTPS) for full functionality in production.
|
||||
|
||||
Happy coding! 🎉
|
||||
102
examples/p2p-webrtc/voice-agent/bot.py
Normal file
102
examples/p2p-webrtc/voice-agent/bot.py
Normal file
@@ -0,0 +1,102 @@
|
||||
#
|
||||
# Copyright (c) 2025, Daily
|
||||
#
|
||||
# SPDX-License-Identifier: BSD 2-Clause License
|
||||
#
|
||||
import os
|
||||
import sys
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
|
||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.pipeline.runner import PipelineRunner
|
||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
|
||||
from pipecat.services.gemini_multimodal_live import GeminiMultimodalLiveLLMService
|
||||
from pipecat.transports.base_transport import TransportParams
|
||||
from pipecat.transports.network.small_webrtc import SmallWebRTCTransport
|
||||
|
||||
load_dotenv(override=True)
|
||||
|
||||
logger.remove(0)
|
||||
logger.add(sys.stderr, level="DEBUG")
|
||||
|
||||
|
||||
SYSTEM_INSTRUCTION = f"""
|
||||
"You are Gemini Chatbot, a friendly, helpful robot.
|
||||
|
||||
Your goal is to demonstrate your capabilities in a succinct way.
|
||||
|
||||
Your output will be converted to audio so don't include special characters in your answers.
|
||||
|
||||
Respond to what the user said in a creative and helpful way. Keep your responses brief. One or two sentences at most.
|
||||
"""
|
||||
|
||||
|
||||
async def run_bot(webrtc_connection):
|
||||
pipecat_transport = SmallWebRTCTransport(
|
||||
webrtc_connection=webrtc_connection,
|
||||
params=TransportParams(
|
||||
audio_in_enabled=True,
|
||||
audio_out_enabled=True,
|
||||
vad_enabled=True,
|
||||
vad_analyzer=SileroVADAnalyzer(),
|
||||
vad_audio_passthrough=True,
|
||||
),
|
||||
)
|
||||
|
||||
llm = GeminiMultimodalLiveLLMService(
|
||||
api_key=os.getenv("GOOGLE_API_KEY"),
|
||||
voice_id="Puck", # Aoede, Charon, Fenrir, Kore, Puck
|
||||
transcribe_user_audio=True,
|
||||
transcribe_model_audio=True,
|
||||
system_instruction=SYSTEM_INSTRUCTION,
|
||||
)
|
||||
|
||||
context = OpenAILLMContext(
|
||||
[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Start by greeting the user warmly and introducing yourself.",
|
||||
}
|
||||
],
|
||||
)
|
||||
context_aggregator = llm.create_context_aggregator(context)
|
||||
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
pipecat_transport.input(),
|
||||
context_aggregator.user(),
|
||||
llm, # LLM
|
||||
pipecat_transport.output(),
|
||||
context_aggregator.assistant(),
|
||||
]
|
||||
)
|
||||
|
||||
task = PipelineTask(
|
||||
pipeline,
|
||||
params=PipelineParams(
|
||||
allow_interruptions=True,
|
||||
),
|
||||
)
|
||||
|
||||
@pipecat_transport.event_handler("on_client_connected")
|
||||
async def on_client_connected(transport, client):
|
||||
logger.info("Pipecat Client connected")
|
||||
# Kick off the conversation.
|
||||
await task.queue_frames([context_aggregator.user().get_context_frame()])
|
||||
|
||||
@pipecat_transport.event_handler("on_client_disconnected")
|
||||
async def on_client_disconnected(transport, client):
|
||||
logger.info("Pipecat Client disconnected")
|
||||
|
||||
@pipecat_transport.event_handler("on_client_closed")
|
||||
async def on_client_closed(transport, client):
|
||||
logger.info("Pipecat Client closed")
|
||||
await task.cancel()
|
||||
|
||||
runner = PipelineRunner(handle_sigint=False)
|
||||
|
||||
await runner.run(task)
|
||||
1
examples/p2p-webrtc/voice-agent/env.example
Normal file
1
examples/p2p-webrtc/voice-agent/env.example
Normal file
@@ -0,0 +1 @@
|
||||
GOOGLE_API_KEY=
|
||||
100
examples/p2p-webrtc/voice-agent/index.html
Normal file
100
examples/p2p-webrtc/voice-agent/index.html
Normal file
@@ -0,0 +1,100 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>WebRTC Voice Agent</title>
|
||||
<style>
|
||||
body { font-family: Arial, sans-serif; text-align: center; margin-top: 50px; }
|
||||
#status { font-size: 20px; margin: 20px; }
|
||||
button { padding: 10px 20px; font-size: 16px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>WebRTC Voice Agent</h1>
|
||||
<p id="status">Disconnected</p>
|
||||
<button id="connect-btn">Connect</button>
|
||||
<audio id="audio-el" autoplay></audio>
|
||||
|
||||
<script>
|
||||
const statusEl = document.getElementById("status")
|
||||
const buttonEl = document.getElementById("connect-btn")
|
||||
const audioEl = document.getElementById("audio-el")
|
||||
|
||||
let connected = false
|
||||
let peerConnection = null
|
||||
|
||||
/*const waitForIceGatheringComplete = async (pc) => {
|
||||
if (pc.iceGatheringState === 'complete') return;
|
||||
return new Promise((resolve) => {
|
||||
const checkState = () => {
|
||||
if (pc.iceGatheringState === 'complete') {
|
||||
pc.removeEventListener('icegatheringstatechange', checkState);
|
||||
resolve();
|
||||
}
|
||||
};
|
||||
pc.addEventListener('icegatheringstatechange', checkState);
|
||||
});
|
||||
}*/
|
||||
|
||||
const createSmallWebRTCConnection = async (audioTrack) => {
|
||||
const pc = new RTCPeerConnection()
|
||||
pc.ontrack = e => audioEl.srcObject = e.streams[0]
|
||||
pc.addTransceiver(audioTrack, { direction: 'sendrecv' })
|
||||
await pc.setLocalDescription(await pc.createOffer())
|
||||
//await waitForIceGatheringComplete(pc)
|
||||
const offer = pc.localDescription
|
||||
const response = await fetch('/api/offer', {
|
||||
body: JSON.stringify({ sdp: offer.sdp, type: offer.type}),
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
method: 'POST',
|
||||
});
|
||||
const answer = await response.json()
|
||||
await pc.setRemoteDescription(answer)
|
||||
return pc
|
||||
}
|
||||
|
||||
const connect = async () => {
|
||||
const audioStream = await navigator.mediaDevices.getUserMedia({audio: true})
|
||||
peerConnection= await createSmallWebRTCConnection(audioStream.getAudioTracks()[0])
|
||||
peerConnection.onconnectionstatechange = () => {
|
||||
let connectionState = peerConnection?.connectionState
|
||||
if (connectionState === 'connected') {
|
||||
_onConnected()
|
||||
} else if (connectionState === 'disconnected') {
|
||||
_onDisconnected()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const _onConnected = () => {
|
||||
statusEl.textContent = "Connected"
|
||||
buttonEl.textContent = "Disconnect"
|
||||
connected = true
|
||||
}
|
||||
|
||||
const _onDisconnected = () => {
|
||||
statusEl.textContent = "Disconnected"
|
||||
buttonEl.textContent = "Connect"
|
||||
connected = false
|
||||
}
|
||||
|
||||
const disconnect = () => {
|
||||
if (!peerConnection) {
|
||||
return
|
||||
}
|
||||
peerConnection.close()
|
||||
peerConnection = null
|
||||
_onDisconnected()
|
||||
}
|
||||
|
||||
buttonEl.addEventListener("click", async () => {
|
||||
if (!connected) {
|
||||
await connect()
|
||||
} else {
|
||||
disconnect()
|
||||
}
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
5
examples/p2p-webrtc/voice-agent/requirements.txt
Normal file
5
examples/p2p-webrtc/voice-agent/requirements.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
python-dotenv
|
||||
fastapi[all]
|
||||
uvicorn
|
||||
aiortc
|
||||
pipecat-ai[google,silero]
|
||||
81
examples/p2p-webrtc/voice-agent/server.py
Normal file
81
examples/p2p-webrtc/voice-agent/server.py
Normal file
@@ -0,0 +1,81 @@
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Dict
|
||||
|
||||
import uvicorn
|
||||
from bot import run_bot
|
||||
from dotenv import load_dotenv
|
||||
from fastapi import BackgroundTasks, FastAPI
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv(override=True)
|
||||
|
||||
logger = logging.getLogger("pc")
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
# Store connections by pc_id
|
||||
pcs_map: Dict[str, SmallWebRTCConnection] = {}
|
||||
|
||||
|
||||
@app.post("/api/offer")
|
||||
async def offer(request: dict, background_tasks: BackgroundTasks):
|
||||
pc_id = request.get("pc_id")
|
||||
|
||||
if pc_id and pc_id in pcs_map:
|
||||
pipecat_connection = pcs_map[pc_id]
|
||||
logger.info(f"Reusing existing connection for pc_id: {pc_id}")
|
||||
await pipecat_connection.renegotiate(sdp=request["sdp"], type=request["type"])
|
||||
else:
|
||||
pipecat_connection = SmallWebRTCConnection()
|
||||
await pipecat_connection.initialize(sdp=request["sdp"], type=request["type"])
|
||||
|
||||
@pipecat_connection.on("closed")
|
||||
async def handle_disconnected(webrtc_connection: SmallWebRTCConnection):
|
||||
logger.info(f"Discarding peer connection for pc_id: {webrtc_connection.pc_id}")
|
||||
pcs_map.pop(webrtc_connection.pc_id, None)
|
||||
|
||||
background_tasks.add_task(run_bot, pipecat_connection)
|
||||
|
||||
answer = pipecat_connection.get_answer()
|
||||
# Updating the peer connection inside the map
|
||||
pcs_map[answer["pc_id"]] = pipecat_connection
|
||||
|
||||
return answer
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def serve_index():
|
||||
return FileResponse("index.html")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
yield # Run app
|
||||
coros = [pc.close() for pc in pcs_map.values()]
|
||||
await asyncio.gather(*coros)
|
||||
pcs_map.clear()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="WebRTC demo")
|
||||
parser.add_argument(
|
||||
"--host", default="localhost", help="Host for HTTP server (default: localhost)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port", type=int, default=7860, help="Port for HTTP server (default: 7860)"
|
||||
)
|
||||
parser.add_argument("--verbose", "-v", action="count")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.verbose:
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
else:
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
uvicorn.run(app, host=args.host, port=args.port)
|
||||
@@ -7,6 +7,7 @@
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Any, Dict, List, Mapping, Optional, Union
|
||||
@@ -177,6 +178,7 @@ class GeminiMultimodalLiveLLMService(LLMService):
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(base_url=base_url, **kwargs)
|
||||
self._last_sent_time = 0
|
||||
self.api_key = api_key
|
||||
self.base_url = base_url
|
||||
self.set_model_name(model)
|
||||
@@ -548,7 +550,13 @@ class GeminiMultimodalLiveLLMService(LLMService):
|
||||
async def _send_user_video(self, frame):
|
||||
if self._video_input_paused:
|
||||
return
|
||||
# logger.debug(f"Sending video frame to Gemini: {frame}")
|
||||
|
||||
now = time.time()
|
||||
if now - self._last_sent_time < 1:
|
||||
return # Ignore if less than 1 second has passed
|
||||
|
||||
self._last_sent_time = now # Update last sent time
|
||||
logger.debug(f"Sending video frame to Gemini: {frame}")
|
||||
evt = events.VideoInputMessage.from_image_frame(frame)
|
||||
await self.send_client_event(evt)
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ from pipecat.utils.base_object import BaseObject
|
||||
class TransportParams(BaseModel):
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True)
|
||||
|
||||
camera_in_enabled: bool = False
|
||||
camera_out_enabled: bool = False
|
||||
camera_out_is_live: bool = False
|
||||
camera_out_width: int = 1024
|
||||
|
||||
512
src/pipecat/transports/network/small_webrtc.py
Normal file
512
src/pipecat/transports/network/small_webrtc.py
Normal file
@@ -0,0 +1,512 @@
|
||||
#
|
||||
# Copyright (c) 2024–2025, Daily
|
||||
#
|
||||
# SPDX-License-Identifier: BSD 2-Clause License
|
||||
#
|
||||
|
||||
import asyncio
|
||||
import fractions
|
||||
import logging
|
||||
import time
|
||||
from collections import deque
|
||||
from typing import Any, Awaitable, Callable, Optional
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from aiortc import VideoStreamTrack
|
||||
from aiortc.mediastreams import AudioStreamTrack, MediaStreamError, VideoFrame
|
||||
from av import AudioFrame, AudioResampler
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel
|
||||
|
||||
# Get the logger for aiortc
|
||||
# aiortc_logger = logging.getLogger("aiortc")
|
||||
# aiortc_logger.setLevel(logging.DEBUG)
|
||||
from pipecat.frames.frames import (
|
||||
CancelFrame,
|
||||
EndFrame,
|
||||
InputAudioRawFrame,
|
||||
InputImageRawFrame,
|
||||
OutputImageRawFrame,
|
||||
StartFrame,
|
||||
TransportMessageFrame,
|
||||
TransportMessageUrgentFrame,
|
||||
)
|
||||
from pipecat.transports.base_input import BaseInputTransport
|
||||
from pipecat.transports.base_output import BaseOutputTransport
|
||||
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||
from pipecat.transports.network.webrtc_connection import SmallWebRTCConnection
|
||||
|
||||
|
||||
class SmallWebRTCCallbacks(BaseModel):
|
||||
on_app_message: Callable[[Any], Awaitable[None]]
|
||||
on_client_connected: Callable[[SmallWebRTCConnection], Awaitable[None]]
|
||||
on_client_disconnected: Callable[[SmallWebRTCConnection], Awaitable[None]]
|
||||
on_client_closed: Callable[[SmallWebRTCConnection], Awaitable[None]]
|
||||
|
||||
|
||||
class RawAudioTrack(AudioStreamTrack):
|
||||
def __init__(self, sample_rate):
|
||||
super().__init__()
|
||||
self._sample_rate = sample_rate
|
||||
self._samples_per_frame = self._sample_rate // 50 # 20ms per frame
|
||||
self._timestamp = 0
|
||||
self._audio_buffer = deque()
|
||||
self._start = time.time()
|
||||
|
||||
def add_audio_bytes(self, audio_bytes: bytes):
|
||||
"""
|
||||
Adds bytes to the audio buffer and returns a Future that completes when the data is processed.
|
||||
"""
|
||||
if len(audio_bytes) % 2 != 0:
|
||||
raise ValueError("Audio bytes length must be even (16-bit samples).")
|
||||
future = asyncio.get_running_loop().create_future()
|
||||
self._audio_buffer.append((audio_bytes, future))
|
||||
return future
|
||||
|
||||
async def recv(self):
|
||||
"""
|
||||
Returns the next audio frame, generating silence if needed.
|
||||
"""
|
||||
# Compute required wait time for synchronization
|
||||
if self._timestamp > 0:
|
||||
wait = self._start + (self._timestamp / self._sample_rate) - time.time()
|
||||
if wait > 0:
|
||||
await asyncio.sleep(wait)
|
||||
|
||||
# Check if we have enough data
|
||||
needed_bytes = self._samples_per_frame * 2 # 16-bit (2 bytes per sample)
|
||||
available_bytes = sum(len(audio_bytes) for audio_bytes, _ in self._audio_buffer)
|
||||
consumed_futures = [] # Track futures for processed data
|
||||
if available_bytes >= needed_bytes:
|
||||
# Extract data from deque
|
||||
chunk = bytearray()
|
||||
while len(chunk) < needed_bytes:
|
||||
audio_bytes, future = self._audio_buffer.popleft()
|
||||
chunk.extend(audio_bytes)
|
||||
consumed_futures.append(future) # Track the future
|
||||
chunk = bytes(chunk[:needed_bytes]) # Trim excess bytes
|
||||
else:
|
||||
chunk = bytes(needed_bytes) # Generate silent frame
|
||||
|
||||
# Convert the byte data to an ndarray of int16 samples
|
||||
samples = np.frombuffer(chunk, dtype=np.int16)
|
||||
|
||||
# Create AudioFrame
|
||||
frame = AudioFrame.from_ndarray(samples[None, :], layout="mono")
|
||||
self._timestamp += self._samples_per_frame
|
||||
frame.pts = self._timestamp
|
||||
frame.sample_rate = self._sample_rate
|
||||
frame.time_base = fractions.Fraction(1, self._sample_rate)
|
||||
|
||||
# Resolve all futures corresponding to consumed data
|
||||
for future in consumed_futures:
|
||||
if not future.done():
|
||||
future.set_result(True)
|
||||
|
||||
return frame
|
||||
|
||||
|
||||
class RawVideoTrack(VideoStreamTrack):
|
||||
def __init__(self, width, height):
|
||||
super().__init__()
|
||||
self._width = width
|
||||
self._height = height
|
||||
self._video_buffer = asyncio.Queue()
|
||||
|
||||
def add_video_frame(self, frame):
|
||||
"""Adds a raw video frame to the buffer."""
|
||||
self._video_buffer.put_nowait(frame)
|
||||
|
||||
async def recv(self):
|
||||
"""Returns the next video frame, waiting if the buffer is empty."""
|
||||
raw_frame = await self._video_buffer.get()
|
||||
|
||||
# Convert bytes to NumPy array
|
||||
frame_data = np.frombuffer(raw_frame.image, dtype=np.uint8).reshape(
|
||||
(self._height, self._width, 3)
|
||||
)
|
||||
|
||||
frame = VideoFrame.from_ndarray(frame_data, format="rgb24")
|
||||
|
||||
# Assign timestamp
|
||||
frame.pts, frame.time_base = await self.next_timestamp()
|
||||
|
||||
return frame
|
||||
|
||||
|
||||
class SmallWebRTCClient:
|
||||
def __init__(self, webrtc_connection: SmallWebRTCConnection, callbacks: SmallWebRTCCallbacks):
|
||||
self._webrtcConnection = webrtc_connection
|
||||
self._closing = False
|
||||
self._callbacks = callbacks
|
||||
|
||||
self._audio_output_track = None
|
||||
self._video_output_track = None
|
||||
self._audio_input_track: Optional[AudioStreamTrack] = None
|
||||
self._video_input_track: Optional[VideoStreamTrack] = None
|
||||
|
||||
self._params = None
|
||||
self._audio_in_channels = None
|
||||
self._in_sample_rate = None
|
||||
self._out_sample_rate = None
|
||||
|
||||
# We are always resampling it for 16000 if the sample_rate that we receive is bigger than that.
|
||||
# otherwise we face issues with Silero VAD
|
||||
self._pipecat_resampler = AudioResampler("s16", "mono", 16000)
|
||||
|
||||
@self._webrtcConnection.on("connected")
|
||||
async def on_connected(connection: SmallWebRTCConnection):
|
||||
logger.info("Peer connection established.")
|
||||
await self._handle_client_connected()
|
||||
|
||||
@self._webrtcConnection.on("disconnected")
|
||||
async def on_disconnected(connection: SmallWebRTCConnection):
|
||||
logger.info("Peer connection lost.")
|
||||
await self._handle_client_disconnected()
|
||||
|
||||
@self._webrtcConnection.on("closed")
|
||||
async def on_closed(connection: SmallWebRTCConnection):
|
||||
logger.info("Client connection closed.")
|
||||
await self._handle_client_closed()
|
||||
|
||||
@self._webrtcConnection.on("appMessage")
|
||||
async def on_app_message(message: Any):
|
||||
await self._handle_app_message(message)
|
||||
|
||||
async def read_video_frame(self):
|
||||
"""
|
||||
Reads a video frame from the given MediaStreamTrack, converts it to RGB,
|
||||
and creates an InputImageRawFrame.
|
||||
"""
|
||||
while True:
|
||||
if self._video_input_track is None:
|
||||
await asyncio.sleep(0.01)
|
||||
continue
|
||||
|
||||
try:
|
||||
frame = await asyncio.wait_for(self._video_input_track.recv(), timeout=2.0)
|
||||
except asyncio.TimeoutError:
|
||||
if self._webrtcConnection.is_connected():
|
||||
logger.warning("Timeout: No video frame received within the specified time.")
|
||||
# self._webrtcConnection.ask_to_renegotiate()
|
||||
frame = None
|
||||
except MediaStreamError:
|
||||
logger.warning("Received an unexpected media stream error while reading the audio.")
|
||||
frame = None
|
||||
|
||||
if frame is None or not isinstance(frame, VideoFrame):
|
||||
# If no valid frame, sleep for a bit
|
||||
await asyncio.sleep(0.01)
|
||||
continue
|
||||
|
||||
format_name = frame.format.name
|
||||
|
||||
# Convert frame to NumPy array in its native format
|
||||
frame_array = frame.to_ndarray(format=format_name)
|
||||
|
||||
# Handle different formats dynamically
|
||||
if format_name == "yuv420p":
|
||||
frame_rgb = cv2.cvtColor(frame_array, cv2.COLOR_YUV2RGB_I420)
|
||||
elif format_name == "nv12":
|
||||
frame_rgb = cv2.cvtColor(frame_array, cv2.COLOR_YUV2RGB_NV12)
|
||||
elif format_name == "gray":
|
||||
frame_rgb = cv2.cvtColor(frame_array, cv2.COLOR_GRAY2RGB)
|
||||
elif format_name.startswith("rgb"): # Already RGB, no conversion needed
|
||||
frame_rgb = frame_array
|
||||
else:
|
||||
raise ValueError(f"Unsupported format: {format_name}")
|
||||
|
||||
image_frame = InputImageRawFrame(
|
||||
image=frame_rgb.tobytes(),
|
||||
size=(frame.width, frame.height),
|
||||
format="RGB",
|
||||
)
|
||||
|
||||
yield image_frame
|
||||
|
||||
async def read_audio_frame(self):
|
||||
"""
|
||||
Reads 20ms of audio from the given MediaStreamTrack and creates an InputAudioRawFrame.
|
||||
"""
|
||||
while True:
|
||||
if self._audio_input_track is None:
|
||||
await asyncio.sleep(0.01)
|
||||
continue
|
||||
|
||||
try:
|
||||
frame = await asyncio.wait_for(self._audio_input_track.recv(), timeout=2.0)
|
||||
except asyncio.TimeoutError:
|
||||
if self._webrtcConnection.is_connected():
|
||||
logger.warning("Timeout: No audio frame received within the specified time.")
|
||||
frame = None
|
||||
except MediaStreamError:
|
||||
logger.warning("Received an unexpected media stream error while reading the audio.")
|
||||
frame = None
|
||||
|
||||
if frame is None or not isinstance(frame, AudioFrame):
|
||||
# If we don't read any audio let's sleep for a little bit (i.e. busy wait).
|
||||
await asyncio.sleep(0.01)
|
||||
continue
|
||||
|
||||
if frame.sample_rate > self._in_sample_rate:
|
||||
resampled_frames = self._pipecat_resampler.resample(frame)
|
||||
for resampled_frame in resampled_frames:
|
||||
# 16-bit PCM bytes
|
||||
pcm_bytes = resampled_frame.to_ndarray().astype(np.int16).tobytes()
|
||||
audio_frame = InputAudioRawFrame(
|
||||
audio=pcm_bytes,
|
||||
sample_rate=resampled_frame.sample_rate,
|
||||
num_channels=self._audio_in_channels,
|
||||
)
|
||||
yield audio_frame
|
||||
else:
|
||||
# 16-bit PCM bytes
|
||||
pcm_bytes = frame.to_ndarray().astype(np.int16).tobytes()
|
||||
audio_frame = InputAudioRawFrame(
|
||||
audio=pcm_bytes,
|
||||
sample_rate=frame.sample_rate,
|
||||
num_channels=self._audio_in_channels,
|
||||
)
|
||||
yield audio_frame
|
||||
|
||||
async def write_raw_audio_frames(self, data: bytes):
|
||||
if self._can_send() and self._audio_output_track:
|
||||
await self._audio_output_track.add_audio_bytes(data)
|
||||
|
||||
async def write_frame_to_camera(self, frame: OutputImageRawFrame):
|
||||
if self._can_send() and self._video_output_track:
|
||||
self._video_output_track.add_video_frame(frame)
|
||||
|
||||
async def setup(self, _params: TransportParams, frame):
|
||||
self._audio_in_channels = _params.audio_in_channels
|
||||
self._in_sample_rate = _params.audio_in_sample_rate or frame.audio_in_sample_rate
|
||||
self._out_sample_rate = _params.audio_out_sample_rate or frame.audio_out_sample_rate
|
||||
self._params = _params
|
||||
|
||||
async def connect(self):
|
||||
if self._webrtcConnection.is_connected():
|
||||
# already initialized
|
||||
return
|
||||
|
||||
logger.info(f"Connecting to Small WebRTC")
|
||||
await self._webrtcConnection.connect()
|
||||
|
||||
async def disconnect(self):
|
||||
if self.is_connected and not self.is_closing:
|
||||
logger.info(f"Disconnecting to Small WebRTC")
|
||||
self._closing = True
|
||||
await self._webrtcConnection.close()
|
||||
await self._handle_client_disconnected()
|
||||
|
||||
async def send_message(self, frame: TransportMessageFrame | TransportMessageUrgentFrame):
|
||||
if self._can_send():
|
||||
self._webrtcConnection.send_app_message(frame.message)
|
||||
|
||||
async def _handle_client_connected(self):
|
||||
# There is nothing to do here yet, the pipeline is still not ready
|
||||
if not self._params:
|
||||
return
|
||||
|
||||
self._audio_input_track = self._webrtcConnection.audio_input_track()
|
||||
self._video_input_track = self._webrtcConnection.video_input_track()
|
||||
if self._params.audio_out_enabled:
|
||||
self._audio_output_track = RawAudioTrack(sample_rate=self._out_sample_rate)
|
||||
self._webrtcConnection.replace_audio_track(self._audio_output_track)
|
||||
|
||||
if self._params.camera_out_enabled:
|
||||
self._video_output_track = RawVideoTrack(
|
||||
width=self._params.camera_out_width, height=self._params.camera_out_height
|
||||
)
|
||||
self._webrtcConnection.replace_video_track(self._video_output_track)
|
||||
|
||||
await self._callbacks.on_client_connected(self._webrtcConnection)
|
||||
|
||||
async def _handle_client_disconnected(self):
|
||||
self._audio_input_track = None
|
||||
self._video_input_track = None
|
||||
self._audio_output_track = None
|
||||
self._video_output_track = None
|
||||
await self._callbacks.on_client_disconnected(self._webrtcConnection)
|
||||
|
||||
async def _handle_client_closed(self):
|
||||
self._audio_input_track = None
|
||||
self._video_input_track = None
|
||||
self._audio_output_track = None
|
||||
self._video_output_track = None
|
||||
await self._callbacks.on_client_closed(self._webrtcConnection)
|
||||
|
||||
async def _handle_app_message(self, message: Any):
|
||||
await self._callbacks.on_app_message(message)
|
||||
|
||||
def _can_send(self):
|
||||
return self.is_connected and not self.is_closing
|
||||
|
||||
@property
|
||||
def is_connected(self) -> bool:
|
||||
return self._webrtcConnection.is_connected()
|
||||
|
||||
@property
|
||||
def is_closing(self) -> bool:
|
||||
return self._closing
|
||||
|
||||
|
||||
class SmallWebRTCInputTransport(BaseInputTransport):
|
||||
def __init__(
|
||||
self,
|
||||
client: SmallWebRTCClient,
|
||||
params: TransportParams,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(params, **kwargs)
|
||||
self._client = client
|
||||
self._params = params
|
||||
self._receive_audio_task = None
|
||||
self._receive_video_task = None
|
||||
|
||||
async def start(self, frame: StartFrame):
|
||||
await super().start(frame)
|
||||
await self._client.setup(self._params, frame)
|
||||
await self._client.connect()
|
||||
if not self._receive_audio_task and (
|
||||
self._params.audio_in_enabled or self._params.vad_enabled
|
||||
):
|
||||
self._receive_audio_task = self.create_task(self._receive_audio())
|
||||
if not self._receive_video_task and self._params.camera_in_enabled:
|
||||
self._receive_video_task = self.create_task(self._receive_video())
|
||||
|
||||
async def _stop_tasks(self):
|
||||
if self._receive_audio_task:
|
||||
await self.cancel_task(self._receive_audio_task)
|
||||
self._receive_audio_task = None
|
||||
if self._receive_video_task:
|
||||
await self.cancel_task(self._receive_video_task)
|
||||
self._receive_video_task = None
|
||||
|
||||
async def stop(self, frame: EndFrame):
|
||||
await super().stop(frame)
|
||||
await self._stop_tasks()
|
||||
await self._client.disconnect()
|
||||
|
||||
async def cancel(self, frame: CancelFrame):
|
||||
await super().cancel(frame)
|
||||
await self._stop_tasks()
|
||||
await self._client.disconnect()
|
||||
|
||||
async def _receive_audio(self):
|
||||
try:
|
||||
async for audio_frame in self._client.read_audio_frame():
|
||||
if audio_frame:
|
||||
await self.push_audio_frame(audio_frame)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"{self} exception receiving data: {e.__class__.__name__} ({e})")
|
||||
|
||||
async def _receive_video(self):
|
||||
try:
|
||||
async for video_frame in self._client.read_video_frame():
|
||||
if video_frame:
|
||||
await self.push_frame(video_frame)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"{self} exception receiving data: {e.__class__.__name__} ({e})")
|
||||
|
||||
async def push_app_message(self, message: Any):
|
||||
logger.info(f"Received app message inside SmallWebRTCInputTransport {message}")
|
||||
frame = TransportMessageUrgentFrame(message=message)
|
||||
await self.push_frame(frame)
|
||||
|
||||
|
||||
class SmallWebRTCOutputTransport(BaseOutputTransport):
|
||||
def __init__(
|
||||
self,
|
||||
client: SmallWebRTCClient,
|
||||
params: TransportParams,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(params, **kwargs)
|
||||
self._client = client
|
||||
self._params = params
|
||||
|
||||
async def start(self, frame: StartFrame):
|
||||
await super().start(frame)
|
||||
await self._client.setup(self._params, frame)
|
||||
await self._client.connect()
|
||||
|
||||
async def stop(self, frame: EndFrame):
|
||||
await super().stop(frame)
|
||||
await self._client.disconnect()
|
||||
|
||||
async def cancel(self, frame: CancelFrame):
|
||||
await super().cancel(frame)
|
||||
await self._client.disconnect()
|
||||
|
||||
async def send_message(self, frame: TransportMessageFrame | TransportMessageUrgentFrame):
|
||||
await self._client.send_message(frame)
|
||||
|
||||
async def write_raw_audio_frames(self, frames: bytes):
|
||||
await self._client.write_raw_audio_frames(frames)
|
||||
|
||||
async def write_frame_to_camera(self, frame: OutputImageRawFrame):
|
||||
await self._client.write_frame_to_camera(frame)
|
||||
|
||||
|
||||
class SmallWebRTCTransport(BaseTransport):
|
||||
def __init__(
|
||||
self,
|
||||
webrtc_connection: SmallWebRTCConnection,
|
||||
params: TransportParams,
|
||||
input_name: Optional[str] = None,
|
||||
output_name: Optional[str] = None,
|
||||
):
|
||||
super().__init__(input_name=input_name, output_name=output_name)
|
||||
self._params = params
|
||||
|
||||
self._callbacks = SmallWebRTCCallbacks(
|
||||
on_app_message=self._on_app_message,
|
||||
on_client_connected=self._on_client_connected,
|
||||
on_client_disconnected=self._on_client_disconnected,
|
||||
on_client_closed=self._on_client_closed,
|
||||
)
|
||||
|
||||
self._client = SmallWebRTCClient(webrtc_connection, self._callbacks)
|
||||
|
||||
self._input = SmallWebRTCInputTransport(self._client, self._params, name=self._input_name)
|
||||
self._output = SmallWebRTCOutputTransport(
|
||||
self._client, self._params, name=self._output_name
|
||||
)
|
||||
|
||||
# Register supported handlers. The user will only be able to register
|
||||
# these handlers.
|
||||
self._register_event_handler("on_app_message")
|
||||
self._register_event_handler("on_client_connected")
|
||||
self._register_event_handler("on_client_disconnected")
|
||||
self._register_event_handler("on_client_closed")
|
||||
|
||||
def input(self) -> SmallWebRTCInputTransport:
|
||||
if not self._input:
|
||||
self._input = SmallWebRTCInputTransport(
|
||||
self._client, self._params, name=self._input_name
|
||||
)
|
||||
return self._input
|
||||
|
||||
def output(self) -> SmallWebRTCOutputTransport:
|
||||
if not self._output:
|
||||
self._output = SmallWebRTCOutputTransport(
|
||||
self._client, self._params, name=self._input_name
|
||||
)
|
||||
return self._output
|
||||
|
||||
async def _on_app_message(self, message: Any):
|
||||
if self._input:
|
||||
await self._input.push_app_message(message)
|
||||
await self._call_event_handler("on_app_message", message)
|
||||
|
||||
async def _on_client_connected(self, webrtc_connection):
|
||||
await self._call_event_handler("on_client_connected", webrtc_connection)
|
||||
|
||||
async def _on_client_disconnected(self, webrtc_connection):
|
||||
await self._call_event_handler("on_client_disconnected", webrtc_connection)
|
||||
|
||||
async def _on_client_closed(self, webrtc_connection):
|
||||
await self._call_event_handler("on_client_closed", webrtc_connection)
|
||||
246
src/pipecat/transports/network/webrtc_connection.py
Normal file
246
src/pipecat/transports/network/webrtc_connection.py
Normal file
@@ -0,0 +1,246 @@
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
from enum import Enum
|
||||
from typing import Any, Optional
|
||||
|
||||
from aiortc import RTCConfiguration, RTCIceServer, RTCPeerConnection, RTCSessionDescription
|
||||
from loguru import logger
|
||||
|
||||
from pipecat.utils.event_emitter import EventEmitter
|
||||
|
||||
SIGNALLING_TYPE = "signalling"
|
||||
|
||||
|
||||
class SignallingMessage(Enum):
|
||||
RENEGOTIATE = "renegotiate"
|
||||
|
||||
|
||||
class SmallWebRTCConnection(EventEmitter):
|
||||
def __init__(self, ice_servers=None):
|
||||
super().__init__()
|
||||
if ice_servers:
|
||||
self.ice_servers = [RTCIceServer(urls=server) for server in ice_servers]
|
||||
else:
|
||||
self.ice_servers = []
|
||||
self._connect_invoked = False
|
||||
self._initialize()
|
||||
|
||||
def _initialize(self):
|
||||
logger.info("Initializing new peer connection")
|
||||
rtc_config = RTCConfiguration(iceServers=self.ice_servers)
|
||||
|
||||
self.answer: Optional[RTCSessionDescription] = None
|
||||
self.pc = RTCPeerConnection(rtc_config)
|
||||
self.pc_id = "PeerConnection(%s)" % uuid.uuid4()
|
||||
self._setup_listeners()
|
||||
self._tracks = set()
|
||||
self._data_channel = None
|
||||
self._renegotiation_in_progress = False
|
||||
self._last_received_time = None
|
||||
|
||||
def _setup_listeners(self):
|
||||
@self.pc.on("datachannel")
|
||||
def on_datachannel(channel):
|
||||
self._data_channel = channel
|
||||
|
||||
@channel.on("message")
|
||||
async def on_message(message):
|
||||
try:
|
||||
# aiortc does not provide any way so we can be aware when we are disconnected,
|
||||
# so we are using this keep alive message as a way to implement that
|
||||
if isinstance(message, str) and message.startswith("ping"):
|
||||
self._last_received_time = time.time()
|
||||
else:
|
||||
json_message = json.loads(message)
|
||||
await self.emit("appMessage", json_message)
|
||||
except Exception as e:
|
||||
logger.exception(f"Error parsing JSON message {message}, {e}")
|
||||
|
||||
# Despite the fact that aiortc provides this listener, they don't have a status for "disconnected"
|
||||
# So, in case we loose connection, this event will not be triggered
|
||||
@self.pc.on("connectionstatechange")
|
||||
async def on_connectionstatechange():
|
||||
await self._handle_new_connection_state()
|
||||
|
||||
# Despite the fact that aiortc provides this listener, they don't have a status for "disconnected"
|
||||
# So, in case we loose connection, this event will not be triggered
|
||||
@self.pc.on("iceconnectionstatechange")
|
||||
async def on_iceconnectionstatechange():
|
||||
logger.info(
|
||||
f"Ice connection state is {self.pc.iceConnectionState}, connection is {self.pc.connectionState}"
|
||||
)
|
||||
|
||||
@self.pc.on("icegatheringstatechange")
|
||||
async def on_icegatheringstatechange():
|
||||
logger.info(f"Ice gathering state is {self.pc.iceGatheringState}")
|
||||
|
||||
@self.pc.on("track")
|
||||
async def on_track(track):
|
||||
logger.info(f"Track {track.kind} received")
|
||||
self._tracks.add(track)
|
||||
await self.emit("track-started", track)
|
||||
|
||||
@track.on("ended")
|
||||
async def on_ended():
|
||||
logger.info(f"Track {track.kind} ended")
|
||||
self._tracks.discard(track)
|
||||
await self.emit("track-ended", track)
|
||||
|
||||
async def _create_answer(self, sdp: str, type: str):
|
||||
offer = RTCSessionDescription(sdp=sdp, type=type)
|
||||
await self.pc.setRemoteDescription(offer)
|
||||
|
||||
# For some reason, aiortc is not respecting the SDP for the transceivers to be sendrcv
|
||||
# so we are basically forcing it to act this way
|
||||
self.force_transceivers_to_send_recv()
|
||||
|
||||
# this answer does not contain the ice candidates, which will be gathered later, after the setLocalDescription
|
||||
logger.info(f"Creating answer")
|
||||
local_answer = await self.pc.createAnswer()
|
||||
await self.pc.setLocalDescription(local_answer)
|
||||
logger.info(f"Setting the answer after the local description is created")
|
||||
self.answer = self.pc.localDescription
|
||||
|
||||
async def initialize(self, sdp: str, type: str):
|
||||
await self._create_answer(sdp, type)
|
||||
|
||||
async def connect(self):
|
||||
self._connect_invoked = True
|
||||
# If we already connected, trigger again the connected event
|
||||
if self.is_connected():
|
||||
await self.emit("connected", self)
|
||||
# We are renegotiating here, because likely we have loose the first video frames
|
||||
# and aiortc does not handle that pretty well.
|
||||
self.ask_to_renegotiate()
|
||||
|
||||
async def renegotiate(self, sdp: str, type: str, restart_pc: bool = False):
|
||||
logger.info(f"Renegotiating {self.pc_id}")
|
||||
|
||||
if restart_pc:
|
||||
await self.emit("disconnected", self)
|
||||
logger.info("Closing old peer connection")
|
||||
# removing the listeners to prevent the bot from closing
|
||||
self.pc.remove_all_listeners()
|
||||
await self.close()
|
||||
# we are initializing a new peer connection in this case.
|
||||
self._initialize()
|
||||
|
||||
await self._create_answer(sdp, type)
|
||||
|
||||
# Maybe we should refactor to receive a message from the client side when the renegotiation is completed.
|
||||
# or look at the peer connection listeners
|
||||
# but this is good enough for now for testing.
|
||||
async def delayed_task():
|
||||
await asyncio.sleep(2)
|
||||
self._renegotiation_in_progress = False
|
||||
|
||||
asyncio.create_task(delayed_task())
|
||||
|
||||
def force_transceivers_to_send_recv(self):
|
||||
for transceiver in self.pc.getTransceivers():
|
||||
transceiver.direction = "sendrecv"
|
||||
# logger.info(
|
||||
# f"Transceiver: {transceiver}, Mid: {transceiver.mid}, Direction: {transceiver.direction}"
|
||||
# )
|
||||
# logger.info(f"Sender track: {transceiver.sender.track}")
|
||||
|
||||
def replace_audio_track(self, track):
|
||||
logger.info(f"Replacing audio track {track.kind}")
|
||||
# Transceivers always appear in creation-order for both peers
|
||||
# For now we are only considering that we are going to have 02 transceivers,
|
||||
# one for audio and one for video
|
||||
transceivers = self.pc.getTransceivers()
|
||||
if len(transceivers) > 0 and transceivers[0].sender:
|
||||
transceivers[0].sender.replaceTrack(track)
|
||||
else:
|
||||
logger.warning("Audio transceiver not found. Cannot replace audio track.")
|
||||
|
||||
def replace_video_track(self, track):
|
||||
logger.info(f"Replacing video track {track.kind}")
|
||||
# Transceivers always appear in creation-order for both peers
|
||||
# For now we are only considering that we are going to have 02 transceivers,
|
||||
# one for audio and one for video
|
||||
transceivers = self.pc.getTransceivers()
|
||||
if len(transceivers) > 1 and transceivers[1].sender:
|
||||
transceivers[1].sender.replaceTrack(track)
|
||||
else:
|
||||
logger.warning("Video transceiver not found. Cannot replace video track.")
|
||||
|
||||
async def close(self):
|
||||
if self.pc:
|
||||
await self.pc.close()
|
||||
|
||||
def get_answer(self):
|
||||
if not self.answer:
|
||||
return None
|
||||
|
||||
return {
|
||||
"sdp": self.answer.sdp,
|
||||
"type": self.answer.type,
|
||||
"pc_id": self.pc_id,
|
||||
}
|
||||
|
||||
async def _handle_new_connection_state(self):
|
||||
state = self.pc.connectionState
|
||||
logger.info(f"Connection state changed to: {state}")
|
||||
await self.emit(state, self)
|
||||
if state == "failed":
|
||||
logger.warning("Connection failed, closing peer connection.")
|
||||
await self.close()
|
||||
|
||||
# Despite the fact that aiortc provides this listener, they don't have a status for "disconnected"
|
||||
# So, there is no advantage in looking at self.pc.connectionState
|
||||
# That is why we are trying to keep our own state
|
||||
def is_connected(self):
|
||||
# If the small webrtc transport has never invoked to connect
|
||||
# we are acting like if we are not connected
|
||||
if not self._connect_invoked:
|
||||
return False
|
||||
|
||||
if self._last_received_time is None:
|
||||
# if we have never received a message, it is probably because the client has not created a data channel
|
||||
# so we are going to trust aiortc in this case
|
||||
return self.pc.connectionState == "connected"
|
||||
# Checks if the last received ping was within the last 3 seconds.
|
||||
return (time.time() - self._last_received_time) < 3
|
||||
|
||||
def audio_input_track(self):
|
||||
# Transceivers always appear in creation-order for both peers
|
||||
# For now we are only considering that we are going to have 02 transceivers,
|
||||
# one for audio and one for video
|
||||
transceivers = self.pc.getTransceivers()
|
||||
if len(transceivers) == 0 or not transceivers[0].receiver:
|
||||
logger.warning("No audio transceiver is available")
|
||||
return None
|
||||
|
||||
return transceivers[0].receiver.track
|
||||
|
||||
def video_input_track(self):
|
||||
# Transceivers always appear in creation-order for both peers
|
||||
# For now we are only considering that we are going to have 02 transceivers,
|
||||
# one for audio and one for video
|
||||
transceivers = self.pc.getTransceivers()
|
||||
if len(transceivers) <= 1 or not transceivers[1].receiver:
|
||||
logger.warning("No video transceiver is available")
|
||||
return None
|
||||
|
||||
return transceivers[1].receiver.track
|
||||
|
||||
def tracks(self):
|
||||
return self._tracks
|
||||
|
||||
def send_app_message(self, message: Any):
|
||||
if self._data_channel:
|
||||
json_message = json.dumps(message)
|
||||
self._data_channel.send(json_message)
|
||||
|
||||
def ask_to_renegotiate(self):
|
||||
if self._renegotiation_in_progress:
|
||||
return
|
||||
|
||||
self._renegotiation_in_progress = True
|
||||
self.send_app_message(
|
||||
{"type": SIGNALLING_TYPE, "message": SignallingMessage.RENEGOTIATE.value}
|
||||
)
|
||||
20
src/pipecat/utils/event_emitter.py
Normal file
20
src/pipecat/utils/event_emitter.py
Normal file
@@ -0,0 +1,20 @@
|
||||
class EventEmitter:
|
||||
def __init__(self):
|
||||
self._events = {}
|
||||
|
||||
def on(self, event_name):
|
||||
"""Decorator to register an event handler."""
|
||||
|
||||
def decorator(func):
|
||||
if event_name not in self._events:
|
||||
self._events[event_name] = []
|
||||
self._events[event_name].append(func)
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
async def emit(self, event_name, *args, **kwargs):
|
||||
"""Trigger all handlers for a given event."""
|
||||
if event_name in self._events:
|
||||
for handler in self._events[event_name]:
|
||||
await handler(*args, **kwargs)
|
||||
Reference in New Issue
Block a user