Integrate product-ws voice demo on port 8000 alongside REST API.
Add src/voice Pipecat pipeline, browser demo at /voice-demo, and config/voice.json. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
106
static/voice-demo/README.md
Normal file
106
static/voice-demo/README.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# Webpage Example — Realtime Voice Chat
|
||||
|
||||
A self-contained browser client for the engine's product websocket
|
||||
(`/ws-product`, protocol `va.ws.v1`).
|
||||
|
||||
## Features
|
||||
|
||||
- **Connect / Disconnect** to any `ws://` or `wss://` URL.
|
||||
- **Microphone selector + mic on/off toggle** — available input devices
|
||||
are listed with `enumerateDevices`, and getUserMedia is requested with
|
||||
`echoCancellation`, `noiseSuppression`, and `autoGainControl` so the
|
||||
browser handles AEC against the bot's voice.
|
||||
- **Text composer** — type a message and press <kbd>Enter</kbd> to send
|
||||
an `input.text` event (Shift+Enter for newline). Sending interrupts
|
||||
any in-flight bot audio so the next reply is heard cleanly.
|
||||
- **Chat history** rendered from `input.transcript.final` (you, when
|
||||
spoken), streamed `response.text.delta` / `response.text.final`
|
||||
(assistant — deltas arrive ahead of the synthesized audio), and locally
|
||||
for text you submit (the engine doesn't echo text input back as a
|
||||
transcript).
|
||||
- **WebSocket log** panel for connection state and compact send/receive
|
||||
events. Audio chunks are summarized so the UI does not flood.
|
||||
- **Gapless TTS playback** by scheduling each `response.audio.delta`
|
||||
chunk back-to-back on the AudioContext.
|
||||
- **Live VU meter** + mic and bot activity indicators.
|
||||
- **Clear** button to reset history.
|
||||
|
||||
No build step, no dependencies — just three files plus an AudioWorklet.
|
||||
|
||||
## Layout
|
||||
|
||||
```text
|
||||
examples/webpage/
|
||||
├── index.html
|
||||
├── styles.css
|
||||
├── app.js
|
||||
└── pcm-recorder.worklet.js
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
1. Start the engine (default port `8000`):
|
||||
|
||||
```bash
|
||||
cd AI-VideoAssistant-engine-v5-pipecat-minimal
|
||||
source .venv/bin/activate
|
||||
export OPENAI_API_KEY=...
|
||||
uvicorn engine.main:app --host 127.0.0.1 --port 8000
|
||||
```
|
||||
|
||||
2. Open the demo page served by the same process:
|
||||
|
||||
```text
|
||||
http://127.0.0.1:8000/demo/
|
||||
```
|
||||
|
||||
The default websocket URL is derived from the page host
|
||||
(`ws://127.0.0.1:8000/ws-product`). Click **Connect**, pick a
|
||||
microphone if needed, click **Enable mic**, and start speaking.
|
||||
|
||||
Mount path and on/off are controlled in `config.json`:
|
||||
|
||||
```json
|
||||
"server": {
|
||||
"serve_webpage": true,
|
||||
"webpage_mount": "/demo"
|
||||
}
|
||||
```
|
||||
|
||||
Set `"serve_webpage": false` in production if you serve the UI elsewhere.
|
||||
|
||||
### Standalone static server (optional)
|
||||
|
||||
You can still serve the files from another port for UI-only iteration.
|
||||
Add that origin to `server.cors_origins` in `config.json` if needed:
|
||||
|
||||
```bash
|
||||
cd AI-VideoAssistant-engine-v5-pipecat-minimal/examples/webpage
|
||||
python -m http.server 8080
|
||||
```
|
||||
|
||||
Then open <http://localhost:8080> and point the URL field at
|
||||
`ws://127.0.0.1:8000/ws-product`.
|
||||
|
||||
> The browser's mic API requires a secure context. `http://localhost`
|
||||
> qualifies; if you serve from another host, use HTTPS and a `wss://`
|
||||
> URL.
|
||||
|
||||
## Audio details
|
||||
|
||||
- Input: mono Float32 from `getUserMedia` is resampled in the
|
||||
AudioWorklet to PCM16 mono @ 16 kHz, framed into 20 ms chunks, and
|
||||
sent as **binary** websocket messages (the server accepts either
|
||||
binary or the JSON+base64 form).
|
||||
- Output: each `response.audio.delta` carries base64-encoded PCM16 @
|
||||
16 kHz; chunks are decoded and scheduled back-to-back through Web
|
||||
Audio. The browser handles resampling to the device rate.
|
||||
|
||||
## Notes
|
||||
|
||||
- Use headphones if you still hear echo despite browser AEC; the bot's
|
||||
voice leaking back into the open mic is the most common cause of
|
||||
feedback loops.
|
||||
- The engine's session has an inactivity timeout
|
||||
(`session.inactivity_timeout_sec` in `config.json`). If the bot
|
||||
doesn't respond after a long silence, reconnect.
|
||||
899
static/voice-demo/app.js
Normal file
899
static/voice-demo/app.js
Normal file
@@ -0,0 +1,899 @@
|
||||
/**
|
||||
* Minimal browser client for the AI VideoAssistant engine's product
|
||||
* websocket (`/ws-product`, protocol `va.ws.v1`).
|
||||
*
|
||||
* Responsibilities:
|
||||
* - Open/close the websocket and run the session handshake.
|
||||
* - List/select microphones and capture mic audio with browser AEC enabled.
|
||||
* - Downsample to PCM16 mono @ 16 kHz in an AudioWorklet and stream frames
|
||||
* as binary websocket messages.
|
||||
* - Play `response.audio.delta` frames gaplessly through Web Audio.
|
||||
* - Render a chat-style history of user transcripts and bot text deltas.
|
||||
*/
|
||||
|
||||
const SAMPLE_RATE = 16000;
|
||||
const CHANNELS = 1;
|
||||
const FRAME_MS = 20;
|
||||
const PROTOCOL = "va.ws.v1";
|
||||
const MAX_WS_LOG_LINES = 120;
|
||||
const AUDIO_DELTA_LOG_INTERVAL_MS = 1000;
|
||||
|
||||
function defaultWsUrl() {
|
||||
const scheme = location.protocol === "https:" ? "wss:" : "ws:";
|
||||
return `${scheme}//${location.host}/ws-product`;
|
||||
}
|
||||
|
||||
const els = {
|
||||
url: document.getElementById("ws-url"),
|
||||
connectBtn: document.getElementById("connect-btn"),
|
||||
statusDot: document.getElementById("status-dot"),
|
||||
statusText: document.getElementById("status-text"),
|
||||
chatLog: document.getElementById("chat-log"),
|
||||
micBtn: document.getElementById("mic-btn"),
|
||||
micSelect: document.getElementById("mic-select"),
|
||||
micLabel: document.querySelector(".mic-btn__label"),
|
||||
micIndicator: document.getElementById("mic-indicator"),
|
||||
botIndicator: document.getElementById("bot-indicator"),
|
||||
clearBtn: document.getElementById("clear-btn"),
|
||||
clearWsLogBtn: document.getElementById("clear-ws-log-btn"),
|
||||
wsLog: document.getElementById("ws-log"),
|
||||
meterFill: document.getElementById("meter-fill"),
|
||||
composer: document.getElementById("composer"),
|
||||
textInput: document.getElementById("text-input"),
|
||||
sendBtn: document.getElementById("send-btn"),
|
||||
};
|
||||
|
||||
const state = {
|
||||
ws: null,
|
||||
connected: false,
|
||||
connecting: false,
|
||||
|
||||
audioContext: null,
|
||||
micStream: null,
|
||||
micSourceNode: null,
|
||||
recorderNode: null,
|
||||
|
||||
micEnabled: false,
|
||||
micDevices: [],
|
||||
selectedMicDeviceId: "",
|
||||
|
||||
// Output scheduling.
|
||||
nextPlaybackTime: 0,
|
||||
playbackEndsAt: 0,
|
||||
scheduledSources: [],
|
||||
botActive: false,
|
||||
botUiTimer: null,
|
||||
|
||||
// Chat state.
|
||||
currentAssistantBubble: null,
|
||||
|
||||
// VU meter smoothing.
|
||||
meterLevel: 0,
|
||||
|
||||
// Compact websocket logging.
|
||||
audioDeltaLogCount: 0,
|
||||
audioDeltaLogBytes: 0,
|
||||
lastAudioDeltaLogAt: 0,
|
||||
audioSendLogCount: 0,
|
||||
audioSendLogBytes: 0,
|
||||
lastAudioSendLogAt: 0,
|
||||
};
|
||||
|
||||
/* ------------------------------------------------------------------ UI */
|
||||
|
||||
function setStatus(kind, text) {
|
||||
els.statusDot.className = `status__dot status__dot--${kind}`;
|
||||
els.statusText.textContent = text;
|
||||
}
|
||||
|
||||
function setConnectButton() {
|
||||
if (state.connecting) {
|
||||
els.connectBtn.textContent = "Connecting…";
|
||||
els.connectBtn.disabled = true;
|
||||
els.connectBtn.classList.remove("is-disconnect");
|
||||
} else if (state.connected) {
|
||||
els.connectBtn.textContent = "Disconnect";
|
||||
els.connectBtn.disabled = false;
|
||||
els.connectBtn.classList.add("is-disconnect");
|
||||
} else {
|
||||
els.connectBtn.textContent = "Connect";
|
||||
els.connectBtn.disabled = false;
|
||||
els.connectBtn.classList.remove("is-disconnect");
|
||||
}
|
||||
}
|
||||
|
||||
function setMicButton() {
|
||||
els.micBtn.disabled = !state.connected;
|
||||
els.micBtn.setAttribute("aria-pressed", state.micEnabled ? "true" : "false");
|
||||
els.micBtn.title = state.micEnabled ? "Mute mic" : "Unmute mic";
|
||||
els.micLabel.textContent = state.micEnabled ? "Mute mic" : "Enable mic";
|
||||
els.micIndicator.classList.toggle("is-active", state.micEnabled);
|
||||
}
|
||||
|
||||
function setMicSelectEnabled() {
|
||||
els.micSelect.disabled = !state.connected || !navigator.mediaDevices;
|
||||
}
|
||||
|
||||
function setComposerEnabled(enabled) {
|
||||
els.textInput.disabled = !enabled;
|
||||
els.sendBtn.disabled = !enabled || els.textInput.value.trim().length === 0;
|
||||
}
|
||||
|
||||
function setBotIndicator(active) {
|
||||
els.botIndicator.classList.toggle("is-active", active);
|
||||
}
|
||||
|
||||
function addBubble(role, text) {
|
||||
if (els.chatLog.querySelector(".chat__empty")) {
|
||||
els.chatLog.innerHTML = "";
|
||||
}
|
||||
const bubble = document.createElement("div");
|
||||
bubble.className = `bubble bubble--${role}`;
|
||||
if (role !== "system") {
|
||||
const tag = document.createElement("span");
|
||||
tag.className = "bubble__role";
|
||||
tag.textContent = role === "user" ? "You" : "Assistant";
|
||||
bubble.appendChild(tag);
|
||||
}
|
||||
const body = document.createElement("span");
|
||||
body.className = "bubble__text";
|
||||
body.textContent = text;
|
||||
bubble.appendChild(body);
|
||||
els.chatLog.appendChild(bubble);
|
||||
scrollChatToBottom();
|
||||
return bubble;
|
||||
}
|
||||
|
||||
function appendToBubble(bubble, text) {
|
||||
const body = bubble.querySelector(".bubble__text");
|
||||
body.textContent += text;
|
||||
scrollChatToBottom();
|
||||
}
|
||||
|
||||
function scrollChatToBottom() {
|
||||
els.chatLog.scrollTop = els.chatLog.scrollHeight;
|
||||
}
|
||||
|
||||
function clearChat() {
|
||||
els.chatLog.innerHTML = "";
|
||||
state.currentAssistantBubble = null;
|
||||
const empty = document.createElement("div");
|
||||
empty.className = "chat__empty";
|
||||
empty.innerHTML = "<p>Chat cleared.</p>";
|
||||
els.chatLog.appendChild(empty);
|
||||
}
|
||||
|
||||
function truncateLogValue(value, maxLength = 160) {
|
||||
const text = String(value);
|
||||
if (text.length <= maxLength) return text;
|
||||
return `${text.slice(0, maxLength - 1)}…`;
|
||||
}
|
||||
|
||||
function compactWsPayload(payload) {
|
||||
if (!payload || typeof payload !== "object") return String(payload);
|
||||
const compact = { ...payload };
|
||||
|
||||
if (typeof compact.audio === "string") {
|
||||
compact.audio = `<base64 ${compact.audio.length} chars>`;
|
||||
}
|
||||
if (typeof compact.data === "string" && compact.data.length > 160) {
|
||||
compact.data = `<string ${compact.data.length} chars>`;
|
||||
}
|
||||
if (typeof compact.text === "string") {
|
||||
compact.text = truncateLogValue(compact.text);
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.stringify(compact);
|
||||
} catch (_) {
|
||||
return payload.type || "unserializable websocket payload";
|
||||
}
|
||||
}
|
||||
|
||||
function addWsLog(direction, detail, kind = direction) {
|
||||
if (els.wsLog.querySelector(".ws-log__empty")) {
|
||||
els.wsLog.innerHTML = "";
|
||||
}
|
||||
|
||||
const entry = document.createElement("div");
|
||||
entry.className = `ws-log__entry ws-log__entry--${kind}`;
|
||||
|
||||
const time = document.createElement("span");
|
||||
time.className = "ws-log__time";
|
||||
time.textContent = new Date().toLocaleTimeString([], {
|
||||
hour12: false,
|
||||
hour: "2-digit",
|
||||
minute: "2-digit",
|
||||
second: "2-digit",
|
||||
});
|
||||
|
||||
const dir = document.createElement("span");
|
||||
dir.className = "ws-log__direction";
|
||||
dir.textContent =
|
||||
direction === "send"
|
||||
? "SEND"
|
||||
: direction === "recv"
|
||||
? "RECV"
|
||||
: direction.toUpperCase();
|
||||
|
||||
const body = document.createElement("span");
|
||||
body.className = "ws-log__detail";
|
||||
body.textContent = detail;
|
||||
|
||||
entry.append(time, dir, body);
|
||||
els.wsLog.appendChild(entry);
|
||||
|
||||
while (els.wsLog.children.length > MAX_WS_LOG_LINES) {
|
||||
els.wsLog.firstElementChild.remove();
|
||||
}
|
||||
els.wsLog.scrollTop = els.wsLog.scrollHeight;
|
||||
}
|
||||
|
||||
function flushAudioDeltaLog() {
|
||||
if (state.audioDeltaLogCount === 0) return;
|
||||
addWsLog(
|
||||
"recv",
|
||||
`response.audio.delta x${state.audioDeltaLogCount} (${state.audioDeltaLogBytes} bytes)`,
|
||||
);
|
||||
state.audioDeltaLogCount = 0;
|
||||
state.audioDeltaLogBytes = 0;
|
||||
state.lastAudioDeltaLogAt = performance.now();
|
||||
}
|
||||
|
||||
function flushAudioSendLog() {
|
||||
if (state.audioSendLogCount === 0) return;
|
||||
addWsLog(
|
||||
"send",
|
||||
`input.audio binary x${state.audioSendLogCount} (${state.audioSendLogBytes} bytes)`,
|
||||
);
|
||||
state.audioSendLogCount = 0;
|
||||
state.audioSendLogBytes = 0;
|
||||
state.lastAudioSendLogAt = performance.now();
|
||||
}
|
||||
|
||||
function flushPendingWsLogs() {
|
||||
flushAudioDeltaLog();
|
||||
flushAudioSendLog();
|
||||
}
|
||||
|
||||
function logWsPayload(direction, payload) {
|
||||
if (direction === "send") {
|
||||
flushAudioSendLog();
|
||||
} else {
|
||||
flushAudioDeltaLog();
|
||||
}
|
||||
|
||||
if (direction === "recv" && payload?.type === "response.audio.delta") {
|
||||
state.audioDeltaLogCount += 1;
|
||||
state.audioDeltaLogBytes += payload.bytes || payload.audio?.length || 0;
|
||||
const now = performance.now();
|
||||
if (now - state.lastAudioDeltaLogAt >= AUDIO_DELTA_LOG_INTERVAL_MS) {
|
||||
flushAudioDeltaLog();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
addWsLog(direction, compactWsPayload(payload));
|
||||
}
|
||||
|
||||
function logBinarySend(byteLength) {
|
||||
state.audioSendLogCount += 1;
|
||||
state.audioSendLogBytes += byteLength;
|
||||
const now = performance.now();
|
||||
if (now - state.lastAudioSendLogAt >= AUDIO_DELTA_LOG_INTERVAL_MS) {
|
||||
flushAudioSendLog();
|
||||
}
|
||||
}
|
||||
|
||||
function wsSend(data) {
|
||||
if (!state.ws || state.ws.readyState !== WebSocket.OPEN) return false;
|
||||
|
||||
if (typeof data === "string") {
|
||||
try {
|
||||
logWsPayload("send", JSON.parse(data));
|
||||
} catch (_) {
|
||||
flushAudioSendLog();
|
||||
flushAudioDeltaLog();
|
||||
addWsLog("send", truncateLogValue(data));
|
||||
}
|
||||
} else {
|
||||
const byteLength =
|
||||
data instanceof ArrayBuffer
|
||||
? data.byteLength
|
||||
: ArrayBuffer.isView(data)
|
||||
? data.byteLength
|
||||
: 0;
|
||||
if (byteLength > 0) {
|
||||
logBinarySend(byteLength);
|
||||
}
|
||||
}
|
||||
|
||||
state.ws.send(data);
|
||||
return true;
|
||||
}
|
||||
|
||||
function clearWsLog() {
|
||||
state.audioDeltaLogCount = 0;
|
||||
state.audioDeltaLogBytes = 0;
|
||||
state.audioSendLogCount = 0;
|
||||
state.audioSendLogBytes = 0;
|
||||
els.wsLog.innerHTML =
|
||||
'<div class="ws-log__empty">No websocket events yet.</div>';
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- Audio */
|
||||
|
||||
async function ensureAudioContext() {
|
||||
if (!state.audioContext) {
|
||||
const Ctx = window.AudioContext || window.webkitAudioContext;
|
||||
state.audioContext = new Ctx();
|
||||
await state.audioContext.audioWorklet.addModule("./pcm-recorder.worklet.js");
|
||||
}
|
||||
if (state.audioContext.state === "suspended") {
|
||||
await state.audioContext.resume();
|
||||
}
|
||||
return state.audioContext;
|
||||
}
|
||||
|
||||
function renderMicDevices() {
|
||||
const previousValue = state.selectedMicDeviceId || els.micSelect.value;
|
||||
els.micSelect.innerHTML = "";
|
||||
|
||||
const defaultOption = document.createElement("option");
|
||||
defaultOption.value = "";
|
||||
defaultOption.textContent = "Default microphone";
|
||||
els.micSelect.appendChild(defaultOption);
|
||||
|
||||
state.micDevices.forEach((device, index) => {
|
||||
const option = document.createElement("option");
|
||||
option.value = device.deviceId;
|
||||
option.textContent = device.label || `Microphone ${index + 1}`;
|
||||
els.micSelect.appendChild(option);
|
||||
});
|
||||
|
||||
const hasPrevious = state.micDevices.some(
|
||||
(device) => device.deviceId === previousValue,
|
||||
);
|
||||
state.selectedMicDeviceId = hasPrevious ? previousValue : "";
|
||||
els.micSelect.value = state.selectedMicDeviceId;
|
||||
setMicSelectEnabled();
|
||||
}
|
||||
|
||||
async function refreshMicDevices() {
|
||||
if (!navigator.mediaDevices?.enumerateDevices) {
|
||||
setMicSelectEnabled();
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const devices = await navigator.mediaDevices.enumerateDevices();
|
||||
state.micDevices = devices.filter((device) => device.kind === "audioinput");
|
||||
renderMicDevices();
|
||||
} catch (err) {
|
||||
console.warn("Could not enumerate microphones", err);
|
||||
setMicSelectEnabled();
|
||||
}
|
||||
}
|
||||
|
||||
async function startMic() {
|
||||
const ctx = await ensureAudioContext();
|
||||
const audioConstraints = {
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true,
|
||||
channelCount: 1,
|
||||
};
|
||||
if (state.selectedMicDeviceId) {
|
||||
audioConstraints.deviceId = { exact: state.selectedMicDeviceId };
|
||||
}
|
||||
|
||||
state.micStream = await navigator.mediaDevices.getUserMedia({
|
||||
audio: audioConstraints,
|
||||
video: false,
|
||||
});
|
||||
await refreshMicDevices();
|
||||
|
||||
state.micSourceNode = ctx.createMediaStreamSource(state.micStream);
|
||||
state.recorderNode = new AudioWorkletNode(ctx, "pcm-recorder", {
|
||||
numberOfInputs: 1,
|
||||
numberOfOutputs: 0,
|
||||
channelCount: 1,
|
||||
processorOptions: {
|
||||
targetSampleRate: SAMPLE_RATE,
|
||||
frameMs: FRAME_MS,
|
||||
},
|
||||
});
|
||||
state.recorderNode.port.onmessage = (event) => {
|
||||
const data = event.data;
|
||||
if (!data || data.type !== "frame") return;
|
||||
updateMeter(data.rms || 0);
|
||||
if (state.connected) {
|
||||
wsSend(data.buffer);
|
||||
}
|
||||
};
|
||||
|
||||
state.micSourceNode.connect(state.recorderNode);
|
||||
state.micEnabled = true;
|
||||
addWsLog("system", "mic capture started (binary input.audio frames)");
|
||||
setMicButton();
|
||||
}
|
||||
|
||||
function stopMic() {
|
||||
const wasEnabled = state.micEnabled;
|
||||
if (state.recorderNode) {
|
||||
try {
|
||||
state.recorderNode.port.onmessage = null;
|
||||
state.recorderNode.disconnect();
|
||||
} catch (_) {
|
||||
/* ignore */
|
||||
}
|
||||
state.recorderNode = null;
|
||||
}
|
||||
if (state.micSourceNode) {
|
||||
try {
|
||||
state.micSourceNode.disconnect();
|
||||
} catch (_) {
|
||||
/* ignore */
|
||||
}
|
||||
state.micSourceNode = null;
|
||||
}
|
||||
if (state.micStream) {
|
||||
for (const track of state.micStream.getTracks()) {
|
||||
try {
|
||||
track.stop();
|
||||
} catch (_) {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
state.micStream = null;
|
||||
}
|
||||
state.micEnabled = false;
|
||||
updateMeter(0);
|
||||
if (wasEnabled) {
|
||||
flushAudioSendLog();
|
||||
addWsLog("system", "mic capture stopped");
|
||||
}
|
||||
setMicButton();
|
||||
}
|
||||
|
||||
function updateMeter(rms) {
|
||||
// Smooth and convert to a 0..100 width. RMS ~0.3+ is loud speech.
|
||||
const target = Math.min(1, rms * 2.4);
|
||||
state.meterLevel = state.meterLevel * 0.5 + target * 0.5;
|
||||
els.meterFill.style.width = `${Math.round(state.meterLevel * 100)}%`;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------- Bot audio playback */
|
||||
|
||||
function schedulePlayback(int16) {
|
||||
const ctx = state.audioContext;
|
||||
if (!ctx) return;
|
||||
|
||||
const float32 = new Float32Array(int16.length);
|
||||
for (let i = 0; i < int16.length; i++) {
|
||||
float32[i] = int16[i] / (int16[i] < 0 ? 0x8000 : 0x7fff);
|
||||
}
|
||||
const buffer = ctx.createBuffer(CHANNELS, float32.length, SAMPLE_RATE);
|
||||
buffer.copyToChannel(float32, 0);
|
||||
|
||||
const src = ctx.createBufferSource();
|
||||
src.buffer = buffer;
|
||||
src.connect(ctx.destination);
|
||||
|
||||
const now = ctx.currentTime;
|
||||
// Schedule immediately after the previously scheduled chunk to keep
|
||||
// playback contiguous, with a tiny safety margin if we fell behind.
|
||||
const startAt = Math.max(now + 0.02, state.nextPlaybackTime);
|
||||
src.start(startAt);
|
||||
state.nextPlaybackTime = startAt + buffer.duration;
|
||||
state.playbackEndsAt = state.nextPlaybackTime;
|
||||
|
||||
src.onended = () => {
|
||||
const idx = state.scheduledSources.indexOf(src);
|
||||
if (idx >= 0) state.scheduledSources.splice(idx, 1);
|
||||
};
|
||||
state.scheduledSources.push(src);
|
||||
|
||||
setBotIndicator(true);
|
||||
if (state.botUiTimer) clearTimeout(state.botUiTimer);
|
||||
const msUntilEnd = Math.max(0, (state.playbackEndsAt - now) * 1000) + 120;
|
||||
state.botUiTimer = setTimeout(() => {
|
||||
if (state.audioContext &&
|
||||
state.audioContext.currentTime >= state.playbackEndsAt - 0.01) {
|
||||
setBotIndicator(false);
|
||||
}
|
||||
}, msUntilEnd);
|
||||
}
|
||||
|
||||
function stopPlaybackQueue() {
|
||||
for (const src of state.scheduledSources) {
|
||||
try {
|
||||
src.onended = null;
|
||||
src.stop();
|
||||
src.disconnect();
|
||||
} catch (_) {
|
||||
/* already stopped */
|
||||
}
|
||||
}
|
||||
state.scheduledSources = [];
|
||||
resetPlaybackClock();
|
||||
if (state.botUiTimer) {
|
||||
clearTimeout(state.botUiTimer);
|
||||
state.botUiTimer = null;
|
||||
}
|
||||
setBotIndicator(false);
|
||||
}
|
||||
|
||||
function resetPlaybackClock() {
|
||||
if (state.audioContext) {
|
||||
state.nextPlaybackTime = state.audioContext.currentTime;
|
||||
state.playbackEndsAt = state.audioContext.currentTime;
|
||||
}
|
||||
}
|
||||
|
||||
/* --------------------------------------------------------- Chat updates */
|
||||
|
||||
function handleUserTranscript(text) {
|
||||
if (!text) return;
|
||||
state.currentAssistantBubble = null;
|
||||
addBubble("user", text);
|
||||
}
|
||||
|
||||
function sendText(text) {
|
||||
const value = (text || "").trim();
|
||||
if (!value) return false;
|
||||
if (!state.ws || state.ws.readyState !== WebSocket.OPEN) return false;
|
||||
const message = {
|
||||
type: "input.text",
|
||||
text: value,
|
||||
interrupt: true,
|
||||
};
|
||||
|
||||
// The engine does not echo text input back as a transcript event, so we
|
||||
// render the user bubble locally. Also interrupt any in-flight bot audio
|
||||
// so the next reply is heard cleanly. We deliberately do NOT clear
|
||||
// `currentAssistantBubble` here — the engine will emit a
|
||||
// `response.text.final(interrupted=true)` for the in-flight assistant
|
||||
// turn, which finalizes that bubble in place. A brand-new bubble for the
|
||||
// reply will be created when `response.text.started` arrives.
|
||||
wsSend(JSON.stringify(message));
|
||||
stopPlaybackQueue();
|
||||
addBubble("user", value);
|
||||
return true;
|
||||
}
|
||||
|
||||
function handleAssistantDelta(text) {
|
||||
if (!text) return;
|
||||
if (!state.currentAssistantBubble) {
|
||||
state.currentAssistantBubble = addBubble("assistant", "");
|
||||
}
|
||||
appendToBubble(state.currentAssistantBubble, text);
|
||||
}
|
||||
|
||||
function handleAssistantStarted() {
|
||||
state.currentAssistantBubble = null;
|
||||
}
|
||||
|
||||
function handleAssistantFinal(text, interrupted) {
|
||||
if (!text) {
|
||||
state.currentAssistantBubble = null;
|
||||
return;
|
||||
}
|
||||
if (state.currentAssistantBubble) {
|
||||
const body = state.currentAssistantBubble.querySelector(".bubble__text");
|
||||
body.textContent = text;
|
||||
} else {
|
||||
state.currentAssistantBubble = addBubble("assistant", text);
|
||||
}
|
||||
if (interrupted) {
|
||||
state.currentAssistantBubble.classList.add("bubble--interrupted");
|
||||
}
|
||||
state.currentAssistantBubble = null;
|
||||
scrollChatToBottom();
|
||||
}
|
||||
|
||||
function finalizeAssistantBubble() {
|
||||
state.currentAssistantBubble = null;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------- Websocket IO */
|
||||
|
||||
function decodeBase64ToInt16(b64) {
|
||||
const binary = atob(b64);
|
||||
const len = binary.length;
|
||||
const bytes = new Uint8Array(len);
|
||||
for (let i = 0; i < len; i++) bytes[i] = binary.charCodeAt(i);
|
||||
return new Int16Array(bytes.buffer, bytes.byteOffset, bytes.byteLength / 2);
|
||||
}
|
||||
|
||||
function handleEvent(event) {
|
||||
switch (event.type) {
|
||||
case "response.audio.delta":
|
||||
if (typeof event.audio === "string") {
|
||||
schedulePlayback(decodeBase64ToInt16(event.audio));
|
||||
}
|
||||
break;
|
||||
case "response.audio.started":
|
||||
setBotIndicator(true);
|
||||
break;
|
||||
case "response.audio.stopped":
|
||||
finalizeAssistantBubble();
|
||||
// The indicator turns off automatically when the playback queue drains.
|
||||
break;
|
||||
case "response.text.delta":
|
||||
handleAssistantDelta(event.text);
|
||||
break;
|
||||
case "response.text.started":
|
||||
handleAssistantStarted();
|
||||
break;
|
||||
case "response.text.final":
|
||||
handleAssistantFinal(event.text, event.interrupted);
|
||||
break;
|
||||
case "input.transcript.final":
|
||||
handleUserTranscript(event.text);
|
||||
break;
|
||||
case "input.transcript.interim":
|
||||
// Ignore partial ASR updates; chat history renders committed user turns.
|
||||
break;
|
||||
case "transport.message":
|
||||
// Reserved for future structured messages; ignore silently.
|
||||
break;
|
||||
default:
|
||||
// Unknown event type: log for debugging.
|
||||
console.debug("ws event", event);
|
||||
}
|
||||
}
|
||||
|
||||
async function connect() {
|
||||
if (state.connected || state.connecting) return;
|
||||
const url = (els.url.value || "").trim();
|
||||
if (!url) {
|
||||
setStatus("error", "Missing URL");
|
||||
return;
|
||||
}
|
||||
|
||||
state.connecting = true;
|
||||
setStatus("connecting", "Connecting…");
|
||||
setConnectButton();
|
||||
addWsLog("system", `connecting ${url}`);
|
||||
|
||||
try {
|
||||
// Pre-warm audio context on user gesture so playback works on Safari.
|
||||
await ensureAudioContext();
|
||||
} catch (err) {
|
||||
console.error("AudioContext failed", err);
|
||||
state.connecting = false;
|
||||
setStatus("error", "Audio init failed");
|
||||
setConnectButton();
|
||||
addWsLog("error", `audio init failed: ${err.message || err}`, "error");
|
||||
return;
|
||||
}
|
||||
|
||||
let ws;
|
||||
try {
|
||||
ws = new WebSocket(url);
|
||||
} catch (err) {
|
||||
console.error("WebSocket constructor failed", err);
|
||||
state.connecting = false;
|
||||
setStatus("error", "Bad URL");
|
||||
setConnectButton();
|
||||
addWsLog("error", `bad websocket URL: ${err.message || err}`, "error");
|
||||
return;
|
||||
}
|
||||
ws.binaryType = "arraybuffer";
|
||||
state.ws = ws;
|
||||
|
||||
ws.addEventListener("open", () => {
|
||||
const startMessage = {
|
||||
type: "session.start",
|
||||
protocol: PROTOCOL,
|
||||
audio: {
|
||||
encoding: "pcm_s16le",
|
||||
sample_rate: SAMPLE_RATE,
|
||||
channels: CHANNELS,
|
||||
},
|
||||
};
|
||||
|
||||
state.connecting = false;
|
||||
state.connected = true;
|
||||
resetPlaybackClock();
|
||||
addWsLog("system", "websocket open");
|
||||
setStatus("connected", "Connected");
|
||||
setConnectButton();
|
||||
setMicButton();
|
||||
setMicSelectEnabled();
|
||||
refreshMicDevices();
|
||||
|
||||
wsSend(JSON.stringify(startMessage));
|
||||
addBubble("system", "Session started.");
|
||||
setComposerEnabled(true);
|
||||
els.textInput.focus();
|
||||
});
|
||||
|
||||
ws.addEventListener("message", (event) => {
|
||||
const data = event.data;
|
||||
if (typeof data === "string") {
|
||||
let parsed;
|
||||
try {
|
||||
parsed = JSON.parse(data);
|
||||
} catch (err) {
|
||||
console.warn("Bad JSON from server", err, data);
|
||||
addWsLog(
|
||||
"error",
|
||||
`invalid JSON from server: ${truncateLogValue(data)}`,
|
||||
"error",
|
||||
);
|
||||
return;
|
||||
}
|
||||
logWsPayload("recv", parsed);
|
||||
handleEvent(parsed);
|
||||
} else if (data instanceof ArrayBuffer) {
|
||||
// Server doesn't currently send binary, but handle it just in case.
|
||||
addWsLog("recv", `binary audio ${data.byteLength} bytes`);
|
||||
schedulePlayback(new Int16Array(data));
|
||||
}
|
||||
});
|
||||
|
||||
ws.addEventListener("error", (err) => {
|
||||
console.error("WebSocket error", err);
|
||||
setStatus("error", "Connection error");
|
||||
addWsLog("error", "websocket error", "error");
|
||||
});
|
||||
|
||||
ws.addEventListener("close", (event) => {
|
||||
const wasConnected = state.connected;
|
||||
state.ws = null;
|
||||
state.connected = false;
|
||||
state.connecting = false;
|
||||
if (state.micEnabled) stopMic();
|
||||
stopPlaybackQueue();
|
||||
setConnectButton();
|
||||
setMicButton();
|
||||
setMicSelectEnabled();
|
||||
setComposerEnabled(false);
|
||||
setBotIndicator(false);
|
||||
flushPendingWsLogs();
|
||||
addWsLog(
|
||||
"system",
|
||||
`websocket close code=${event.code}${
|
||||
event.reason ? ` reason=${event.reason}` : ""
|
||||
}`,
|
||||
);
|
||||
if (wasConnected) {
|
||||
addBubble(
|
||||
"system",
|
||||
`Session ended${event.reason ? ` — ${event.reason}` : ""}.`,
|
||||
);
|
||||
setStatus("idle", "Disconnected");
|
||||
} else {
|
||||
setStatus("error", "Connection closed");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function disconnect() {
|
||||
if (!state.ws) return;
|
||||
try {
|
||||
if (state.ws.readyState === WebSocket.OPEN) {
|
||||
const stopMessage = { type: "session.stop", reason: "client_disconnect" };
|
||||
wsSend(JSON.stringify(stopMessage));
|
||||
}
|
||||
} catch (_) {
|
||||
/* ignore */
|
||||
}
|
||||
try {
|
||||
state.ws.close(1000, "client_disconnect");
|
||||
} catch (_) {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- Wiring */
|
||||
|
||||
els.connectBtn.addEventListener("click", () => {
|
||||
if (state.connected) disconnect();
|
||||
else connect();
|
||||
});
|
||||
|
||||
els.micBtn.addEventListener("click", async () => {
|
||||
if (!state.connected) return;
|
||||
els.micBtn.disabled = true;
|
||||
try {
|
||||
if (state.micEnabled) {
|
||||
stopMic();
|
||||
} else {
|
||||
await startMic();
|
||||
}
|
||||
} catch (err) {
|
||||
console.error("Mic error", err);
|
||||
addBubble("system", `Mic error: ${err.message || err}`);
|
||||
} finally {
|
||||
els.micBtn.disabled = !state.connected;
|
||||
}
|
||||
});
|
||||
|
||||
els.micSelect.addEventListener("change", async () => {
|
||||
state.selectedMicDeviceId = els.micSelect.value;
|
||||
if (!state.micEnabled) return;
|
||||
|
||||
els.micSelect.disabled = true;
|
||||
els.micBtn.disabled = true;
|
||||
try {
|
||||
stopMic();
|
||||
await startMic();
|
||||
} catch (err) {
|
||||
console.error("Mic switch error", err);
|
||||
addBubble("system", `Mic switch error: ${err.message || err}`);
|
||||
} finally {
|
||||
setMicButton();
|
||||
setMicSelectEnabled();
|
||||
}
|
||||
});
|
||||
|
||||
if (navigator.mediaDevices?.addEventListener) {
|
||||
navigator.mediaDevices.addEventListener("devicechange", refreshMicDevices);
|
||||
}
|
||||
|
||||
els.clearBtn.addEventListener("click", () => {
|
||||
clearChat();
|
||||
});
|
||||
|
||||
els.clearWsLogBtn.addEventListener("click", () => {
|
||||
clearWsLog();
|
||||
});
|
||||
|
||||
function autosizeTextarea() {
|
||||
const ta = els.textInput;
|
||||
ta.style.height = "auto";
|
||||
ta.style.height = `${Math.min(ta.scrollHeight, 180)}px`;
|
||||
}
|
||||
|
||||
function submitText() {
|
||||
const value = els.textInput.value;
|
||||
if (!sendText(value)) return;
|
||||
els.textInput.value = "";
|
||||
autosizeTextarea();
|
||||
setComposerEnabled(state.connected);
|
||||
}
|
||||
|
||||
els.composer.addEventListener("submit", (event) => {
|
||||
event.preventDefault();
|
||||
submitText();
|
||||
});
|
||||
|
||||
els.textInput.addEventListener("input", () => {
|
||||
autosizeTextarea();
|
||||
setComposerEnabled(state.connected);
|
||||
});
|
||||
|
||||
els.textInput.addEventListener("keydown", (event) => {
|
||||
if (event.key === "Enter" && !event.shiftKey && !event.isComposing) {
|
||||
event.preventDefault();
|
||||
submitText();
|
||||
}
|
||||
});
|
||||
|
||||
window.addEventListener("beforeunload", () => {
|
||||
if (state.ws) {
|
||||
try {
|
||||
state.ws.close();
|
||||
} catch (_) {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
if (state.audioContext) {
|
||||
try {
|
||||
state.audioContext.close();
|
||||
} catch (_) {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
els.url.value = defaultWsUrl();
|
||||
|
||||
setStatus("idle", "Disconnected");
|
||||
setConnectButton();
|
||||
setMicButton();
|
||||
setMicSelectEnabled();
|
||||
setComposerEnabled(false);
|
||||
158
static/voice-demo/index.html
Normal file
158
static/voice-demo/index.html
Normal file
@@ -0,0 +1,158 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>VA Voice Chat — /ws-product</title>
|
||||
<link rel="stylesheet" href="./styles.css" />
|
||||
</head>
|
||||
<body>
|
||||
<main class="app">
|
||||
<header class="app__header">
|
||||
<div class="brand">
|
||||
<span class="brand__dot" aria-hidden="true"></span>
|
||||
<h1>VA Voice Chat</h1>
|
||||
</div>
|
||||
|
||||
<div class="connection">
|
||||
<label class="connection__field">
|
||||
<span>WebSocket URL</span>
|
||||
<input
|
||||
id="ws-url"
|
||||
type="text"
|
||||
placeholder="ws://host/ws-product"
|
||||
spellcheck="false"
|
||||
autocomplete="off"
|
||||
/>
|
||||
</label>
|
||||
<button id="connect-btn" class="btn btn--primary" type="button">
|
||||
Connect
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="status">
|
||||
<span id="status-dot" class="status__dot status__dot--idle"></span>
|
||||
<span id="status-text" class="status__text">Disconnected</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="app__body">
|
||||
<div class="app__main">
|
||||
<section class="chat" aria-label="Conversation history">
|
||||
<div id="chat-log" class="chat__log" role="log" aria-live="polite">
|
||||
<div class="chat__empty">
|
||||
<p>Connect to the engine, enable your mic, and start talking.</p>
|
||||
<p class="chat__hint">
|
||||
Audio is streamed as PCM16 mono @ 16 kHz over
|
||||
<code>/ws-product</code>.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<footer class="controls" aria-label="Chat controls">
|
||||
<div class="meter" aria-hidden="true">
|
||||
<div id="meter-fill" class="meter__fill"></div>
|
||||
</div>
|
||||
|
||||
<form id="composer" class="composer" autocomplete="off">
|
||||
<textarea
|
||||
id="text-input"
|
||||
class="composer__input"
|
||||
rows="1"
|
||||
placeholder="Type a message, or use the mic…"
|
||||
disabled
|
||||
></textarea>
|
||||
<button
|
||||
id="send-btn"
|
||||
class="btn btn--primary composer__send"
|
||||
type="submit"
|
||||
disabled
|
||||
title="Send message (Enter)"
|
||||
>
|
||||
Send
|
||||
</button>
|
||||
</form>
|
||||
|
||||
<div class="controls__row">
|
||||
<label class="device-picker">
|
||||
<span class="device-picker__label">Microphone</span>
|
||||
<select id="mic-select" class="device-picker__select" disabled>
|
||||
<option value="">Default microphone</option>
|
||||
</select>
|
||||
</label>
|
||||
|
||||
<button
|
||||
id="mic-btn"
|
||||
class="mic-btn"
|
||||
type="button"
|
||||
disabled
|
||||
aria-pressed="false"
|
||||
title="Mic is off"
|
||||
>
|
||||
<svg
|
||||
class="mic-btn__icon"
|
||||
viewBox="0 0 24 24"
|
||||
width="24"
|
||||
height="24"
|
||||
aria-hidden="true"
|
||||
>
|
||||
<path
|
||||
d="M12 14a3 3 0 0 0 3-3V6a3 3 0 1 0-6 0v5a3 3 0 0 0 3 3Z"
|
||||
fill="currentColor"
|
||||
/>
|
||||
<path
|
||||
d="M19 11a1 1 0 1 0-2 0 5 5 0 0 1-10 0 1 1 0 1 0-2 0 7 7 0 0 0 6 6.92V21a1 1 0 1 0 2 0v-3.08A7 7 0 0 0 19 11Z"
|
||||
fill="currentColor"
|
||||
/>
|
||||
</svg>
|
||||
<span class="mic-btn__label">Enable mic</span>
|
||||
</button>
|
||||
|
||||
<div class="indicators">
|
||||
<span id="mic-indicator" class="indicator">
|
||||
<span class="indicator__dot indicator__dot--mic"></span>
|
||||
<span class="indicator__label">Mic</span>
|
||||
</span>
|
||||
<span id="bot-indicator" class="indicator">
|
||||
<span class="indicator__dot indicator__dot--bot"></span>
|
||||
<span class="indicator__label">Bot</span>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<button id="clear-btn" class="btn btn--ghost" type="button">
|
||||
Clear
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<p class="hint">
|
||||
Press <kbd>Enter</kbd> to send, <kbd>Shift</kbd>+<kbd>Enter</kbd>
|
||||
for newline. Sending text will interrupt the bot if it's speaking.
|
||||
Browser echo cancellation is on; use headphones if echo persists.
|
||||
</p>
|
||||
</footer>
|
||||
</div>
|
||||
|
||||
<section class="ws-log" aria-label="WebSocket log">
|
||||
<div class="ws-log__header">
|
||||
<div class="ws-log__header-left">
|
||||
<h2>WebSocket Log</h2>
|
||||
<div class="ws-log__legend" aria-hidden="true">
|
||||
<span class="ws-log__legend-item ws-log__legend-item--send">Send</span>
|
||||
<span class="ws-log__legend-item ws-log__legend-item--recv">Recv</span>
|
||||
</div>
|
||||
</div>
|
||||
<button id="clear-ws-log-btn" class="btn btn--ghost" type="button">
|
||||
Clear log
|
||||
</button>
|
||||
</div>
|
||||
<div id="ws-log" class="ws-log__body" role="log" aria-live="polite">
|
||||
<div class="ws-log__empty">No websocket events yet.</div>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<script type="module" src="./app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
104
static/voice-demo/pcm-recorder.worklet.js
Normal file
104
static/voice-demo/pcm-recorder.worklet.js
Normal file
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* PCM Recorder AudioWorklet.
|
||||
*
|
||||
* Captures mono Float32 mic samples at the AudioContext's native rate,
|
||||
* resamples them to a target sample rate (default 16 kHz) with linear
|
||||
* interpolation, then ships PCM16 frames of a fixed duration (default 20 ms)
|
||||
* to the main thread via `port.postMessage(ArrayBuffer)`.
|
||||
*
|
||||
* It also computes a simple RMS level per frame for the UI VU meter so the
|
||||
* main thread doesn't have to re-process the audio.
|
||||
*/
|
||||
|
||||
class PcmRecorderProcessor extends AudioWorkletProcessor {
|
||||
constructor(options) {
|
||||
super();
|
||||
|
||||
const opts = (options && options.processorOptions) || {};
|
||||
this._targetSampleRate = opts.targetSampleRate || 16000;
|
||||
this._frameMs = opts.frameMs || 20;
|
||||
this._frameSamples = Math.round(
|
||||
(this._targetSampleRate * this._frameMs) / 1000,
|
||||
);
|
||||
|
||||
// Resampling state.
|
||||
// `ratio` is input samples per output sample.
|
||||
this._ratio = sampleRate / this._targetSampleRate;
|
||||
this._inputBuffer = new Float32Array(0);
|
||||
// Float position in `_inputBuffer` for the next output sample.
|
||||
this._inputOffset = 0;
|
||||
|
||||
// Output framing state.
|
||||
this._frameBuffer = new Int16Array(this._frameSamples);
|
||||
this._frameIndex = 0;
|
||||
|
||||
// VU meter accumulator.
|
||||
this._rmsSumSquares = 0;
|
||||
this._rmsCount = 0;
|
||||
}
|
||||
|
||||
process(inputs) {
|
||||
const input = inputs[0];
|
||||
if (!input || input.length === 0) return true;
|
||||
const channel = input[0];
|
||||
if (!channel || channel.length === 0) return true;
|
||||
|
||||
// Append new samples to the input buffer.
|
||||
const merged = new Float32Array(this._inputBuffer.length + channel.length);
|
||||
merged.set(this._inputBuffer, 0);
|
||||
merged.set(channel, this._inputBuffer.length);
|
||||
this._inputBuffer = merged;
|
||||
|
||||
const ratio = this._ratio;
|
||||
const inLen = this._inputBuffer.length;
|
||||
let pos = this._inputOffset;
|
||||
|
||||
while (pos + 1 < inLen) {
|
||||
const lo = Math.floor(pos);
|
||||
const hi = lo + 1;
|
||||
const w = pos - lo;
|
||||
const sample =
|
||||
this._inputBuffer[lo] * (1 - w) + this._inputBuffer[hi] * w;
|
||||
|
||||
this._rmsSumSquares += sample * sample;
|
||||
this._rmsCount += 1;
|
||||
|
||||
let s = sample;
|
||||
if (s > 1) s = 1;
|
||||
else if (s < -1) s = -1;
|
||||
this._frameBuffer[this._frameIndex++] =
|
||||
s < 0 ? Math.round(s * 0x8000) : Math.round(s * 0x7fff);
|
||||
|
||||
if (this._frameIndex === this._frameSamples) {
|
||||
const frame = new Int16Array(this._frameSamples);
|
||||
frame.set(this._frameBuffer);
|
||||
const rms =
|
||||
this._rmsCount > 0
|
||||
? Math.sqrt(this._rmsSumSquares / this._rmsCount)
|
||||
: 0;
|
||||
this.port.postMessage(
|
||||
{ type: "frame", buffer: frame.buffer, rms },
|
||||
[frame.buffer],
|
||||
);
|
||||
this._frameIndex = 0;
|
||||
this._rmsSumSquares = 0;
|
||||
this._rmsCount = 0;
|
||||
}
|
||||
|
||||
pos += ratio;
|
||||
}
|
||||
|
||||
// Trim consumed samples from the input buffer; keep at least the last
|
||||
// sample we still need to interpolate against on the next call.
|
||||
const consumed = Math.floor(pos);
|
||||
if (consumed > 0) {
|
||||
this._inputBuffer = this._inputBuffer.slice(consumed);
|
||||
pos -= consumed;
|
||||
}
|
||||
this._inputOffset = pos;
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor("pcm-recorder", PcmRecorderProcessor);
|
||||
739
static/voice-demo/styles.css
Normal file
739
static/voice-demo/styles.css
Normal file
@@ -0,0 +1,739 @@
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: #0b0d12;
|
||||
--bg-elevated: #131722;
|
||||
--bg-soft: #1a2030;
|
||||
--border: #232a3b;
|
||||
--text: #e6e9f2;
|
||||
--text-dim: #95a0bb;
|
||||
--accent: #4f8cff;
|
||||
--accent-strong: #6aa1ff;
|
||||
--user: #2f7ff0;
|
||||
--assistant: #2a3145;
|
||||
--danger: #ff5577;
|
||||
--success: #2dd28b;
|
||||
--warning: #ffb84d;
|
||||
--radius: 14px;
|
||||
--shadow: 0 10px 30px rgba(0, 0, 0, 0.35);
|
||||
font-family: ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto,
|
||||
"Helvetica Neue", Arial, sans-serif;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html,
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
height: 100%;
|
||||
background: radial-gradient(
|
||||
1200px 600px at 80% -10%,
|
||||
rgba(79, 140, 255, 0.18),
|
||||
transparent 60%
|
||||
),
|
||||
radial-gradient(
|
||||
900px 500px at -10% 110%,
|
||||
rgba(45, 210, 139, 0.12),
|
||||
transparent 60%
|
||||
),
|
||||
var(--bg);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
body {
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
padding: 12px;
|
||||
}
|
||||
|
||||
.app {
|
||||
display: grid;
|
||||
grid-template-rows: auto minmax(0, 1fr);
|
||||
gap: 12px;
|
||||
width: min(1440px, 100%);
|
||||
height: calc(100dvh - 24px);
|
||||
max-height: calc(100vh - 24px);
|
||||
background: var(--bg-elevated);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 20px;
|
||||
box-shadow: var(--shadow);
|
||||
padding: 14px;
|
||||
}
|
||||
|
||||
.app__body {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1fr) clamp(300px, 32vw, 420px);
|
||||
gap: 12px;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.app__main {
|
||||
display: grid;
|
||||
grid-template-rows: minmax(0, 1fr) auto;
|
||||
gap: 0;
|
||||
min-height: 0;
|
||||
overflow: hidden;
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
}
|
||||
|
||||
/* Header ---------------------------------------------------------------- */
|
||||
|
||||
.app__header {
|
||||
display: grid;
|
||||
grid-template-columns: auto minmax(0, 1fr) auto;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
padding-bottom: 10px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.brand {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.brand h1 {
|
||||
font-size: 16px;
|
||||
margin: 0;
|
||||
letter-spacing: 0.2px;
|
||||
}
|
||||
|
||||
.brand__dot {
|
||||
width: 10px;
|
||||
height: 10px;
|
||||
border-radius: 50%;
|
||||
background: var(--accent);
|
||||
box-shadow: 0 0 0 4px rgba(79, 140, 255, 0.18);
|
||||
}
|
||||
|
||||
.connection {
|
||||
display: flex;
|
||||
align-items: end;
|
||||
gap: 8px;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.connection__field {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
min-width: 0;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.connection__field span {
|
||||
font-size: 11px;
|
||||
color: var(--text-dim);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.8px;
|
||||
}
|
||||
|
||||
.connection__field input {
|
||||
background: var(--bg-soft);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
padding: 7px 10px;
|
||||
font: inherit;
|
||||
font-size: 12px;
|
||||
outline: none;
|
||||
width: 100%;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.connection__field input:focus {
|
||||
border-color: var(--accent);
|
||||
box-shadow: 0 0 0 3px rgba(79, 140, 255, 0.18);
|
||||
}
|
||||
|
||||
.status {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-size: 13px;
|
||||
color: var(--text-dim);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.status__dot {
|
||||
width: 9px;
|
||||
height: 9px;
|
||||
border-radius: 50%;
|
||||
background: var(--text-dim);
|
||||
}
|
||||
|
||||
.status__dot--idle {
|
||||
background: var(--text-dim);
|
||||
}
|
||||
|
||||
.status__dot--connecting {
|
||||
background: var(--warning);
|
||||
animation: pulse 1.2s ease-in-out infinite;
|
||||
}
|
||||
|
||||
.status__dot--connected {
|
||||
background: var(--success);
|
||||
}
|
||||
|
||||
.status__dot--error {
|
||||
background: var(--danger);
|
||||
}
|
||||
|
||||
/* Chat ------------------------------------------------------------------ */
|
||||
|
||||
.chat {
|
||||
overflow: hidden;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.chat__log {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 18px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
scroll-behavior: smooth;
|
||||
}
|
||||
|
||||
.chat__empty {
|
||||
margin: auto;
|
||||
text-align: center;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.chat__empty p {
|
||||
margin: 4px 0;
|
||||
}
|
||||
|
||||
.chat__hint code {
|
||||
background: var(--bg-soft);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
padding: 1px 6px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.bubble {
|
||||
max-width: 78%;
|
||||
padding: 10px 14px;
|
||||
border-radius: 14px;
|
||||
line-height: 1.45;
|
||||
font-size: 14px;
|
||||
white-space: pre-wrap;
|
||||
word-wrap: break-word;
|
||||
animation: bubble-in 0.16s ease-out;
|
||||
}
|
||||
|
||||
.bubble--user {
|
||||
background: var(--user);
|
||||
color: #fff;
|
||||
align-self: flex-end;
|
||||
border-bottom-right-radius: 4px;
|
||||
}
|
||||
|
||||
.bubble--assistant {
|
||||
background: var(--assistant);
|
||||
color: var(--text);
|
||||
align-self: flex-start;
|
||||
border-bottom-left-radius: 4px;
|
||||
}
|
||||
|
||||
.bubble--assistant.bubble--interrupted {
|
||||
opacity: 0.75;
|
||||
border-left: 2px solid var(--warning);
|
||||
}
|
||||
|
||||
.bubble--system {
|
||||
align-self: center;
|
||||
background: transparent;
|
||||
color: var(--text-dim);
|
||||
font-size: 12px;
|
||||
border: 1px dashed var(--border);
|
||||
padding: 6px 10px;
|
||||
border-radius: 999px;
|
||||
}
|
||||
|
||||
.bubble__role {
|
||||
display: block;
|
||||
font-size: 10px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.8px;
|
||||
opacity: 0.7;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
/* WebSocket log --------------------------------------------------------- */
|
||||
|
||||
.ws-log {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
min-height: 0;
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.ws-log__header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 8px;
|
||||
padding: 8px 12px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
flex-shrink: 0;
|
||||
background: var(--bg-soft);
|
||||
}
|
||||
|
||||
.ws-log__legend {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-size: 10px;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.ws-log__legend-item {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.ws-log__legend-item::before {
|
||||
content: "";
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 2px;
|
||||
}
|
||||
|
||||
.ws-log__legend-item--send::before {
|
||||
background: var(--success);
|
||||
}
|
||||
|
||||
.ws-log__legend-item--recv::before {
|
||||
background: var(--accent-strong);
|
||||
}
|
||||
|
||||
.ws-log__header h2 {
|
||||
margin: 0;
|
||||
font-size: 12px;
|
||||
color: var(--text-dim);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.8px;
|
||||
}
|
||||
|
||||
.ws-log__header-left {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.ws-log__body {
|
||||
flex: 1;
|
||||
min-height: 0;
|
||||
overflow-y: auto;
|
||||
overflow-x: hidden;
|
||||
padding: 6px 8px;
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
font-size: 10.5px;
|
||||
line-height: 1.4;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.ws-log__entry {
|
||||
display: grid;
|
||||
grid-template-columns: 58px 42px minmax(0, 1fr);
|
||||
gap: 6px;
|
||||
align-items: start;
|
||||
padding: 5px 4px;
|
||||
border-radius: 6px;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.ws-log__entry:hover {
|
||||
background: rgba(255, 255, 255, 0.03);
|
||||
}
|
||||
|
||||
.ws-log__time {
|
||||
color: var(--text-dim);
|
||||
opacity: 0.7;
|
||||
font-size: 10px;
|
||||
}
|
||||
|
||||
.ws-log__direction {
|
||||
font-weight: 700;
|
||||
font-size: 9px;
|
||||
letter-spacing: 0.4px;
|
||||
text-transform: uppercase;
|
||||
padding: 2px 0;
|
||||
}
|
||||
|
||||
.ws-log__detail {
|
||||
min-width: 0;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.ws-log__entry--send .ws-log__direction {
|
||||
color: var(--success);
|
||||
}
|
||||
|
||||
.ws-log__entry--recv .ws-log__direction {
|
||||
color: var(--accent-strong);
|
||||
}
|
||||
|
||||
.ws-log__entry--system .ws-log__direction {
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.ws-log__entry--error .ws-log__direction {
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
.ws-log__empty {
|
||||
color: var(--text-dim);
|
||||
opacity: 0.7;
|
||||
padding: 8px 4px;
|
||||
}
|
||||
|
||||
/* Controls -------------------------------------------------------------- */
|
||||
|
||||
.controls {
|
||||
display: grid;
|
||||
gap: 8px;
|
||||
padding: 10px 12px 8px;
|
||||
border-top: 1px solid var(--border);
|
||||
background: var(--bg-elevated);
|
||||
}
|
||||
|
||||
.meter {
|
||||
height: 6px;
|
||||
background: var(--bg-soft);
|
||||
border-radius: 999px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.meter__fill {
|
||||
height: 100%;
|
||||
width: 0%;
|
||||
background: linear-gradient(90deg, var(--success), var(--accent));
|
||||
transition: width 80ms linear;
|
||||
}
|
||||
|
||||
.composer {
|
||||
display: flex;
|
||||
align-items: flex-end;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.composer__input {
|
||||
flex: 1;
|
||||
resize: none;
|
||||
background: var(--bg-soft);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 12px;
|
||||
padding: 10px 12px;
|
||||
font: inherit;
|
||||
font-size: 14px;
|
||||
line-height: 1.4;
|
||||
outline: none;
|
||||
min-height: 42px;
|
||||
max-height: 180px;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
.composer__input:focus {
|
||||
border-color: var(--accent);
|
||||
box-shadow: 0 0 0 3px rgba(79, 140, 255, 0.18);
|
||||
}
|
||||
|
||||
.composer__input:disabled {
|
||||
opacity: 0.55;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.composer__send {
|
||||
padding: 10px 18px;
|
||||
border-radius: 12px;
|
||||
min-width: 84px;
|
||||
align-self: flex-end;
|
||||
}
|
||||
|
||||
.composer__send:disabled {
|
||||
opacity: 0.5;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.controls__row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.device-picker {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 3px;
|
||||
min-width: 0;
|
||||
flex: 1 1 160px;
|
||||
max-width: 240px;
|
||||
}
|
||||
|
||||
.device-picker__label {
|
||||
font-size: 11px;
|
||||
color: var(--text-dim);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.8px;
|
||||
}
|
||||
|
||||
.device-picker__select {
|
||||
appearance: none;
|
||||
background: var(--bg-soft);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
padding: 7px 28px 7px 10px;
|
||||
font: inherit;
|
||||
font-size: 12px;
|
||||
outline: none;
|
||||
width: 100%;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.device-picker__select:focus {
|
||||
border-color: var(--accent);
|
||||
box-shadow: 0 0 0 3px rgba(79, 140, 255, 0.18);
|
||||
}
|
||||
|
||||
.device-picker__select:disabled {
|
||||
opacity: 0.5;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.mic-btn {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
padding: 8px 12px;
|
||||
border-radius: 999px;
|
||||
background: var(--bg-soft);
|
||||
color: var(--text);
|
||||
border: 1px solid var(--border);
|
||||
font: inherit;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
flex-shrink: 0;
|
||||
transition: transform 0.08s ease, background 0.15s ease, color 0.15s ease,
|
||||
border-color 0.15s ease;
|
||||
}
|
||||
|
||||
.mic-btn:disabled {
|
||||
opacity: 0.5;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.mic-btn:not(:disabled):hover {
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.mic-btn:not(:disabled):active {
|
||||
transform: scale(0.98);
|
||||
}
|
||||
|
||||
.mic-btn[aria-pressed="true"] {
|
||||
background: var(--danger);
|
||||
border-color: var(--danger);
|
||||
color: #fff;
|
||||
box-shadow: 0 0 0 6px rgba(255, 85, 119, 0.18);
|
||||
}
|
||||
|
||||
.mic-btn__icon {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.indicators {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
margin-left: auto;
|
||||
}
|
||||
|
||||
.indicator {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
font-size: 12px;
|
||||
color: var(--text-dim);
|
||||
}
|
||||
|
||||
.indicator__dot {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 50%;
|
||||
background: var(--bg-soft);
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.indicator.is-active .indicator__dot--mic {
|
||||
background: var(--success);
|
||||
border-color: var(--success);
|
||||
box-shadow: 0 0 0 4px rgba(45, 210, 139, 0.2);
|
||||
}
|
||||
|
||||
.indicator.is-active .indicator__dot--bot {
|
||||
background: var(--accent);
|
||||
border-color: var(--accent);
|
||||
box-shadow: 0 0 0 4px rgba(79, 140, 255, 0.22);
|
||||
animation: pulse 1s ease-in-out infinite;
|
||||
}
|
||||
|
||||
.indicator.is-active .indicator__label {
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.btn {
|
||||
appearance: none;
|
||||
border: 1px solid var(--border);
|
||||
background: var(--bg-soft);
|
||||
color: var(--text);
|
||||
padding: 9px 14px;
|
||||
border-radius: 10px;
|
||||
font: inherit;
|
||||
font-size: 13px;
|
||||
cursor: pointer;
|
||||
transition: background 0.15s ease, border-color 0.15s ease;
|
||||
}
|
||||
|
||||
.btn:hover {
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.btn--primary {
|
||||
background: var(--accent);
|
||||
color: #fff;
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.btn--primary:hover {
|
||||
background: var(--accent-strong);
|
||||
border-color: var(--accent-strong);
|
||||
}
|
||||
|
||||
.btn--primary.is-disconnect {
|
||||
background: transparent;
|
||||
color: var(--danger);
|
||||
border-color: var(--danger);
|
||||
}
|
||||
|
||||
.btn--ghost {
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
.hint {
|
||||
margin: 0;
|
||||
font-size: 11px;
|
||||
color: var(--text-dim);
|
||||
opacity: 0.85;
|
||||
}
|
||||
|
||||
.hint kbd {
|
||||
background: var(--bg-soft);
|
||||
border: 1px solid var(--border);
|
||||
border-bottom-width: 2px;
|
||||
border-radius: 4px;
|
||||
padding: 0 5px;
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
font-size: 11px;
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
/* Animations ------------------------------------------------------------ */
|
||||
|
||||
@keyframes pulse {
|
||||
0%,
|
||||
100% {
|
||||
opacity: 1;
|
||||
}
|
||||
50% {
|
||||
opacity: 0.55;
|
||||
}
|
||||
}
|
||||
|
||||
@keyframes bubble-in {
|
||||
from {
|
||||
opacity: 0;
|
||||
transform: translateY(4px);
|
||||
}
|
||||
to {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
}
|
||||
}
|
||||
|
||||
/* Responsive ------------------------------------------------------------ */
|
||||
|
||||
@media (max-width: 820px) {
|
||||
.app__body {
|
||||
grid-template-columns: 1fr;
|
||||
grid-template-rows: minmax(0, 1fr) min(240px, 32vh);
|
||||
}
|
||||
|
||||
.ws-log__legend {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.hint {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 720px) {
|
||||
body {
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.app {
|
||||
height: 100dvh;
|
||||
max-height: 100vh;
|
||||
border-radius: 0;
|
||||
border: none;
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.app__header {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.connection {
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
}
|
||||
|
||||
.ws-log__entry {
|
||||
grid-template-columns: 54px 38px minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.status {
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.device-picker {
|
||||
flex: 1 1 100%;
|
||||
max-width: none;
|
||||
}
|
||||
|
||||
.indicators {
|
||||
margin-left: 0;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user