Add shopping-list UIWorker example (bridge-free voice + UI)
Demonstrates the 'every input acts, may speak' pattern without bridging: a standard voice pipeline (STT → LLM → TTS) whose LLM only converses, plus a separate UIWorker that does all the list work. The voice pipeline's user aggregator fires on_user_turn_stopped each turn and dispatches the transcript to the UIWorker as a respond job (a bus message); the UIWorker reads the auto-injected <ui_state> snapshot and drives the list silently via add_item / set_checked / remove_item commands (plus the standard highlight). Items are checkboxes whose label and checked state the snapshot exposes. Includes a vanilla-JS client following the existing UI-demo client style.
This commit is contained in:
92
examples/multi-worker/ui-worker/shopping-list/README.md
Normal file
92
examples/multi-worker/ui-worker/shopping-list/README.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# shopping-list
|
||||
|
||||
Every voice turn drives the UI; speech is incidental. The user builds a
|
||||
shopping list by talking — "add milk and eggs", "check off the bread",
|
||||
"drop the last one", "what's left?" — and the list updates on screen
|
||||
every turn. The assistant may also say something back. The two halves run
|
||||
in parallel on separate workers; neither calls the other as a tool.
|
||||
|
||||
This is the **bridge-free** pattern for "every input acts, may speak."
|
||||
|
||||
## What it shows
|
||||
|
||||
- **A standard voice pipeline + a UIWorker, no bridging.** The voice
|
||||
layer is an ordinary `transport → STT → LLM → TTS` pipeline whose LLM
|
||||
just converses (no tools, never touches the list). Its user aggregator
|
||||
fires `on_user_turn_stopped` once per user turn, and that handler
|
||||
dispatches the transcript to the UIWorker as a `respond` job
|
||||
(`worker.job("ui", name="respond", payload={"query": transcript})`) —
|
||||
a bus message. The UIWorker does all the list work, silently, on its
|
||||
own worker, so its LLM output never reaches TTS.
|
||||
- **Snapshot-driven action.** Before each UIWorker inference, the current
|
||||
`<ui_state>` is auto-injected (via the LLM's `on_before_process_frame`
|
||||
hook), so the worker resolves "the milk", "the last one", "the checked
|
||||
ones" against the live list.
|
||||
- **Custom UI commands.** A single bundled `update_list` tool maps the
|
||||
request to `add_item` / `set_checked` / `remove_item` commands (plus the
|
||||
standard `highlight`, used to *show* what's left). Each item is a
|
||||
checkbox whose accessible name is the item text, so the snapshot exposes
|
||||
every item's label and checked state.
|
||||
|
||||
## What it adds vs. the prior demos
|
||||
|
||||
The other demos use the **request/response delegation** shape: a voice LLM
|
||||
decides, via a tool call, when to consult the UIWorker. This one removes
|
||||
that decision — **every** user turn goes to the UIWorker automatically,
|
||||
and the voice LLM runs independently for conversation. The wiring is a
|
||||
single `on_user_turn_stopped` handler instead of a delegating tool.
|
||||
|
||||
## Run
|
||||
|
||||
Two terminals.
|
||||
|
||||
**Terminal 1 — bot:**
|
||||
|
||||
```bash
|
||||
cd examples/multi-worker/ui-worker/shopping-list
|
||||
uv run python bot.py
|
||||
```
|
||||
|
||||
The bot starts on `http://localhost:7860`.
|
||||
|
||||
**Terminal 2 — client:**
|
||||
|
||||
```bash
|
||||
cd examples/multi-worker/ui-worker/shopping-list/client
|
||||
npm install # one-time
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Open `http://localhost:5173` and click **Connect**.
|
||||
|
||||
## What to try
|
||||
|
||||
- _"Add milk and a dozen eggs."_ — both items appear; the voice
|
||||
acknowledges.
|
||||
- _"Add bread, butter, and coffee."_ — three more land.
|
||||
- _"Check off the bread."_ — it gets ticked and struck through.
|
||||
- _"Actually, drop the butter."_ — removed.
|
||||
- _"Clear the ones I've already got."_ — removes everything checked.
|
||||
- _"What's left?"_ — the unchecked items pulse, and the voice tells you.
|
||||
- _"Hi there!"_ — the voice greets you; the list is unchanged (the
|
||||
UIWorker no-ops).
|
||||
|
||||
You can also type in the box to add items by hand — they go through the
|
||||
same snapshot path, so the worker sees them next turn.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `OPENAI_API_KEY`
|
||||
- `DEEPGRAM_API_KEY`
|
||||
- `CARTESIA_API_KEY`
|
||||
|
||||
A `.env` in the example folder is the easiest way to set these (see
|
||||
`examples/multi-worker/env.example`).
|
||||
|
||||
## What this example _doesn't_ show
|
||||
|
||||
The voice layer is **blind to the screen** — it converses and answers from
|
||||
the dialogue so far, but it doesn't read the live list (only the UIWorker
|
||||
sees the snapshot). For "what's left?" the UIWorker highlights the answer
|
||||
visually while the voice gives a from-memory summary. The list is also not
|
||||
persisted — refresh and it's gone.
|
||||
325
examples/multi-worker/ui-worker/shopping-list/bot.py
Normal file
325
examples/multi-worker/ui-worker/shopping-list/bot.py
Normal file
@@ -0,0 +1,325 @@
|
||||
#
|
||||
# Copyright (c) 2026, Daily
|
||||
#
|
||||
# SPDX-License-Identifier: BSD 2-Clause License
|
||||
#
|
||||
|
||||
"""Shopping list — every voice turn drives the UI; speech is incidental.
|
||||
|
||||
The page renders a shopping list. The user talks: "add milk and eggs",
|
||||
"check off the bread", "drop the last one", "what's left?". Every user
|
||||
turn updates the list on screen; the assistant may also say something
|
||||
back. The two halves run in parallel on separate workers — neither calls
|
||||
the other as a tool.
|
||||
|
||||
This is the bridge-free pattern for "every input acts, may speak":
|
||||
|
||||
- The **voice layer** is an ordinary voice pipeline (STT → LLM → TTS).
|
||||
Its LLM just converses — it has no tools and never touches the list.
|
||||
- A **UIWorker** ("ui") does all the list work. It is *not* in the voice
|
||||
pipeline. Instead, the voice pipeline's user aggregator fires
|
||||
``on_user_turn_stopped`` once per user turn; that handler dispatches
|
||||
the transcript to the UIWorker as a ``respond`` job (a bus message).
|
||||
The UIWorker reads the current ``<ui_state>`` snapshot (auto-injected
|
||||
before its inference) and calls ``update_list`` to add / check /
|
||||
remove items. It acts silently — its LLM output never reaches TTS,
|
||||
because it lives on its own worker.
|
||||
|
||||
Architecture::
|
||||
|
||||
Voice pipeline (PipelineWorker "main", owns transport + RTVI):
|
||||
transport.in → STT → user_agg → LLM → TTS → transport.out → assistant_agg
|
||||
└── @on_user_turn_stopped: worker.job("ui", name="respond",
|
||||
payload={"query": transcript})
|
||||
|
||||
UIWorker "ui" (job-based, acts silently):
|
||||
└── @tool update_list(add, check, uncheck, remove, highlight)
|
||||
└── send_command("add_item" / "set_checked" / "remove_item")
|
||||
+ respond_to_job() # no speak
|
||||
|
||||
Run::
|
||||
|
||||
uv run python bot.py
|
||||
|
||||
Then open the client at ``http://localhost:5173`` (see ``README.md``).
|
||||
|
||||
Requirements:
|
||||
|
||||
- OPENAI_API_KEY
|
||||
- DEEPGRAM_API_KEY
|
||||
- CARTESIA_API_KEY
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
|
||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||
from pipecat.frames.frames import LLMRunFrame
|
||||
from pipecat.pipeline.job_context import JobError
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.pipeline.runner import PipelineRunner
|
||||
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
|
||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||
from pipecat.processors.aggregators.llm_response_universal import (
|
||||
LLMContextAggregatorPair,
|
||||
LLMUserAggregatorParams,
|
||||
)
|
||||
from pipecat.runner.types import RunnerArguments
|
||||
from pipecat.runner.utils import create_transport
|
||||
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||
from pipecat.services.deepgram.stt import DeepgramSTTService
|
||||
from pipecat.services.llm_service import FunctionCallParams
|
||||
from pipecat.services.openai.llm import OpenAILLMService
|
||||
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||
from pipecat.transports.daily.transport import DailyParams
|
||||
from pipecat.workers.llm import tool
|
||||
from pipecat.workers.ui import UIWorker
|
||||
|
||||
load_dotenv(override=True)
|
||||
|
||||
MAIN_NAME = "main"
|
||||
UI_NAME = "ui"
|
||||
|
||||
transport_params = {
|
||||
"daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||
"webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||
}
|
||||
|
||||
|
||||
VOICE_PROMPT = """\
|
||||
You are the voice of a shopping-list assistant. A separate UI layer \
|
||||
sees the list and updates it on screen; you do NOT touch the list \
|
||||
yourself and you cannot see the screen.
|
||||
|
||||
Your job is to be a brief, friendly companion. Acknowledge what the \
|
||||
user asked for ("Sure, adding milk and eggs"), react naturally, and \
|
||||
answer questions from the conversation so far ("So far you've asked \
|
||||
for milk, eggs, and bread"). Keep every reply to one short spoken \
|
||||
sentence. Don't describe how you're updating the list — the screen \
|
||||
shows that. For a plain greeting, greet back warmly."""
|
||||
|
||||
|
||||
# The UI wire-format guide (UI_STATE_PROMPT_GUIDE) is appended to the LLM's
|
||||
# system instruction automatically by UIWorker, so this prompt only needs the
|
||||
# app-specific behavior.
|
||||
UI_PROMPT = """\
|
||||
You maintain a shopping list by voice. The current ``<ui_state>`` \
|
||||
block is in your context. Each list item is a checkbox with a \
|
||||
snapshot ref (e.g. ``e5``), a label (the item text), and a checked / \
|
||||
unchecked state. Use the labels to resolve which item the user means.
|
||||
|
||||
## Tool: update_list
|
||||
|
||||
Every turn calls ``update_list`` exactly once — even when there's \
|
||||
nothing to change (call it with no arguments). One tool call per turn.
|
||||
|
||||
``update_list(add=None, check=None, uncheck=None, remove=None, highlight=None)``:
|
||||
|
||||
- ``add`` (OPTIONAL): a list of item texts to append. Split a spoken \
|
||||
list into separate items ("milk, eggs and bread" → \
|
||||
``["milk", "eggs", "bread"]``). Normalize lightly (lowercase, drop \
|
||||
filler like "some" / "a couple of").
|
||||
- ``check`` (OPTIONAL): a list of refs to mark done.
|
||||
- ``uncheck`` (OPTIONAL): a list of refs to mark not done.
|
||||
- ``remove`` (OPTIONAL): a list of refs to delete.
|
||||
- ``highlight`` (OPTIONAL): a list of refs to flash briefly. Use it to \
|
||||
*show* the user something when they ask (e.g. highlight the unchecked \
|
||||
items for "what's left?").
|
||||
|
||||
## Decision rules
|
||||
|
||||
- **"Add X (and Y)"** → ``add`` with one entry per item.
|
||||
- **"Check off / got / cross out X"** → resolve X's ref from \
|
||||
``<ui_state>``, set ``check``.
|
||||
- **"Uncheck / put back X"** → set ``uncheck``.
|
||||
- **"Remove / delete / never mind X"**, **"clear the checked ones"** → \
|
||||
set ``remove`` with the matching refs (read the checked state from \
|
||||
``<ui_state>`` for "the checked ones").
|
||||
- **"the last one"** → the last item in ``<ui_state>``.
|
||||
- **"What's left? / what do I still need?"** → ``highlight`` the \
|
||||
unchecked items. **"What's on my list?"** → ``highlight`` everything. \
|
||||
(The voice layer speaks; you just point.)
|
||||
- **A greeting or anything not about the list** → call ``update_list`` \
|
||||
with no arguments (no-op) so the turn completes.
|
||||
|
||||
## Examples
|
||||
|
||||
(refs are illustrative; use the actual refs from the current \
|
||||
``<ui_state>``)
|
||||
|
||||
- "Add milk and a dozen eggs." → \
|
||||
``update_list(add=["milk", "eggs"])``
|
||||
- "Check off the milk." → ``update_list(check=["e5"])``
|
||||
- "Actually drop the eggs." → ``update_list(remove=["e7"])``
|
||||
- "Clear the ones I've got." (e5, e9 checked) → \
|
||||
``update_list(remove=["e5", "e9"])``
|
||||
- "What's left?" (e7, e11 unchecked) → \
|
||||
``update_list(highlight=["e7", "e11"])``
|
||||
- "Hey there." → ``update_list()``"""
|
||||
|
||||
|
||||
class ListWorker(UIWorker):
|
||||
"""UIWorker that maintains the shopping list, silently.
|
||||
|
||||
A single bundled ``update_list`` tool, always called once per turn,
|
||||
maps the user's request to ``add_item`` / ``set_checked`` /
|
||||
``remove_item`` UI commands (plus the standard ``highlight``). The
|
||||
tool completes the ``respond`` job with no spoken reply
|
||||
(``respond_to_job()`` with no ``speak``) — the separate voice layer
|
||||
owns speech.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
llm = OpenAILLMService(
|
||||
api_key=os.environ["OPENAI_API_KEY"],
|
||||
settings=OpenAILLMService.Settings(system_instruction=UI_PROMPT),
|
||||
)
|
||||
super().__init__(UI_NAME, llm=llm)
|
||||
|
||||
@tool
|
||||
async def update_list(
|
||||
self,
|
||||
params: FunctionCallParams,
|
||||
add: list[str] | None = None,
|
||||
check: list[str] | None = None,
|
||||
uncheck: list[str] | None = None,
|
||||
remove: list[str] | None = None,
|
||||
highlight: list[str] | None = None,
|
||||
):
|
||||
"""Update the shopping list. Called exactly once per turn.
|
||||
|
||||
Args:
|
||||
add: New item texts to append (one entry per item).
|
||||
check: Snapshot refs of items to mark done.
|
||||
uncheck: Snapshot refs of items to mark not done.
|
||||
remove: Snapshot refs of items to delete.
|
||||
highlight: Snapshot refs of items to flash briefly (e.g. to
|
||||
show what's left when the user asks).
|
||||
"""
|
||||
logger.info(
|
||||
f"{self}: update_list(add={add!r}, check={check!r}, uncheck={uncheck!r}, "
|
||||
f"remove={remove!r}, highlight={highlight!r})"
|
||||
)
|
||||
# Defensive guards: skip malformed entries so a stray value can't
|
||||
# crash the tool before respond_to_job fires (which would hold the
|
||||
# single-flight lock until the requester's timeout).
|
||||
for text in add or []:
|
||||
if isinstance(text, str) and text.strip():
|
||||
await self.send_command("add_item", {"text": text.strip()})
|
||||
for ref in check or []:
|
||||
if isinstance(ref, str):
|
||||
await self.send_command("set_checked", {"ref": ref, "checked": True})
|
||||
for ref in uncheck or []:
|
||||
if isinstance(ref, str):
|
||||
await self.send_command("set_checked", {"ref": ref, "checked": False})
|
||||
for ref in remove or []:
|
||||
if isinstance(ref, str):
|
||||
await self.send_command("remove_item", {"ref": ref})
|
||||
for ref in highlight or []:
|
||||
if isinstance(ref, str):
|
||||
await self.highlight(ref)
|
||||
await self.respond_to_job()
|
||||
await params.result_callback(None)
|
||||
|
||||
|
||||
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||
logger.info("Starting shopping-list bot")
|
||||
|
||||
runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
|
||||
|
||||
stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
|
||||
tts = CartesiaTTSService(
|
||||
api_key=os.environ["CARTESIA_API_KEY"],
|
||||
settings=CartesiaTTSService.Settings(
|
||||
voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"),
|
||||
),
|
||||
)
|
||||
llm = OpenAILLMService(
|
||||
api_key=os.environ["OPENAI_API_KEY"],
|
||||
settings=OpenAILLMService.Settings(system_instruction=VOICE_PROMPT),
|
||||
)
|
||||
|
||||
context = LLMContext()
|
||||
aggregators = LLMContextAggregatorPair(
|
||||
context,
|
||||
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
|
||||
)
|
||||
user_aggregator = aggregators.user()
|
||||
assistant_aggregator = aggregators.assistant()
|
||||
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
transport.input(),
|
||||
stt,
|
||||
user_aggregator,
|
||||
llm,
|
||||
tts,
|
||||
transport.output(),
|
||||
assistant_aggregator,
|
||||
]
|
||||
)
|
||||
|
||||
worker = PipelineWorker(
|
||||
pipeline,
|
||||
name=MAIN_NAME,
|
||||
params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
|
||||
idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
|
||||
)
|
||||
|
||||
# Every user turn drives the UI: forward the transcript to the UIWorker
|
||||
# as a respond job (a bus message). Fire it from the turn-stopped event
|
||||
# so the voice LLM (which runs from the same turn) and the UIWorker act
|
||||
# in parallel — neither calls the other as a tool. This handler runs in
|
||||
# its own task, so awaiting the job here does not block the voice path.
|
||||
@user_aggregator.event_handler("on_user_turn_stopped")
|
||||
async def on_user_turn_stopped(aggregator, strategy, message):
|
||||
transcript = (message.content or "").strip()
|
||||
if not transcript:
|
||||
return
|
||||
logger.info(f"Dispatching turn to UI worker: {transcript!r}")
|
||||
try:
|
||||
async with worker.job(
|
||||
UI_NAME, name="respond", payload={"query": transcript}, timeout=15
|
||||
):
|
||||
pass
|
||||
except JobError as e:
|
||||
logger.warning(f"ui job failed: {e}")
|
||||
|
||||
@transport.event_handler("on_client_connected")
|
||||
async def on_client_connected(transport, client):
|
||||
logger.info("Client connected")
|
||||
context.add_message(
|
||||
{
|
||||
"role": "developer",
|
||||
"content": (
|
||||
"Greet the user briefly. Tell them they can build their "
|
||||
"shopping list by voice — add items, check things off, or "
|
||||
"ask what's left. One short sentence."
|
||||
),
|
||||
}
|
||||
)
|
||||
await worker.queue_frame(LLMRunFrame())
|
||||
|
||||
@transport.event_handler("on_client_disconnected")
|
||||
async def on_client_disconnected(transport, client):
|
||||
logger.info("Client disconnected")
|
||||
await runner.cancel()
|
||||
|
||||
await runner.launch_worker(ListWorker())
|
||||
await runner.launch_worker(worker)
|
||||
|
||||
await runner.run()
|
||||
|
||||
|
||||
async def bot(runner_args: RunnerArguments):
|
||||
"""Main bot entry point compatible with Pipecat Cloud."""
|
||||
transport = await create_transport(runner_args, transport_params)
|
||||
await run_bot(transport, runner_args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from pipecat.runner.run import main
|
||||
|
||||
main()
|
||||
26
examples/multi-worker/ui-worker/shopping-list/client/.gitignore
vendored
Normal file
26
examples/multi-worker/ui-worker/shopping-list/client/.gitignore
vendored
Normal file
@@ -0,0 +1,26 @@
|
||||
# Logs
|
||||
logs
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
pnpm-debug.log*
|
||||
lerna-debug.log*
|
||||
|
||||
node_modules
|
||||
dist
|
||||
dist-ssr
|
||||
*.local
|
||||
|
||||
# Editor directories and files
|
||||
.vscode/*
|
||||
!.vscode/extensions.json
|
||||
.idea
|
||||
.DS_Store
|
||||
*.suo
|
||||
*.ntvs*
|
||||
*.njsproj
|
||||
*.sln
|
||||
*.sw?
|
||||
|
||||
.vite
|
||||
@@ -0,0 +1,44 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Shopping list — Pipecat UIWorker</title>
|
||||
<link rel="stylesheet" href="styles.css" />
|
||||
</head>
|
||||
<body>
|
||||
<header class="topbar">
|
||||
<h1>🛒 Shopping list</h1>
|
||||
<div class="topbar-right">
|
||||
<span id="status" data-show="0"></span>
|
||||
<button id="connect">Connect</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<p class="hint">
|
||||
Talk to build your list — "add milk and eggs", "check off the
|
||||
bread", "what's left?". You can also type an item below.
|
||||
</p>
|
||||
|
||||
<form id="add-form" class="add-form">
|
||||
<input
|
||||
id="add-input"
|
||||
type="text"
|
||||
placeholder="Add an item…"
|
||||
autocomplete="off"
|
||||
aria-label="Add an item"
|
||||
/>
|
||||
<button type="submit">Add</button>
|
||||
</form>
|
||||
|
||||
<ul id="list" class="list" aria-label="Shopping list"></ul>
|
||||
<p id="list-empty" class="list-empty">
|
||||
Your list is empty. Say "add milk and eggs".
|
||||
</p>
|
||||
</main>
|
||||
|
||||
<audio id="bot-audio" autoplay></audio>
|
||||
<script type="module" src="main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
230
examples/multi-worker/ui-worker/shopping-list/client/main.js
Normal file
230
examples/multi-worker/ui-worker/shopping-list/client/main.js
Normal file
@@ -0,0 +1,230 @@
|
||||
/**
|
||||
* Shopping list — vanilla JS client.
|
||||
*
|
||||
* The bridge-free "every input acts, may speak" pattern:
|
||||
*
|
||||
* - A standard voice pipeline runs on the bot. Its LLM converses (you
|
||||
* hear it), and its user aggregator forwards each transcript to a
|
||||
* separate UIWorker.
|
||||
* - The UIWorker reads the snapshot of THIS page and drives the list
|
||||
* silently via three custom commands — ``add_item``, ``set_checked``,
|
||||
* ``remove_item`` — plus the standard ``highlight`` (used to show
|
||||
* "what's left").
|
||||
*
|
||||
* Each item is a checkbox with its text as the accessible name, so the
|
||||
* snapshot exposes every item's label and checked state to the worker.
|
||||
*/
|
||||
|
||||
import {
|
||||
PipecatClient,
|
||||
RTVIEvent,
|
||||
findElementByRef,
|
||||
} from "@pipecat-ai/client-js";
|
||||
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";
|
||||
|
||||
const BOT_URL = "http://localhost:7860/api/offer";
|
||||
|
||||
const connectButton = document.getElementById("connect");
|
||||
const status = document.getElementById("status");
|
||||
const botAudio = document.getElementById("bot-audio");
|
||||
const list = document.getElementById("list");
|
||||
const listEmpty = document.getElementById("list-empty");
|
||||
const addForm = document.getElementById("add-form");
|
||||
const addInput = document.getElementById("add-input");
|
||||
|
||||
let client;
|
||||
let unsubscribes = [];
|
||||
|
||||
function setStatus(text, autoHideMs = 0) {
|
||||
status.textContent = text;
|
||||
status.dataset.show = text ? "1" : "0";
|
||||
if (text && autoHideMs > 0) {
|
||||
setTimeout(() => {
|
||||
if (status.textContent === text) status.dataset.show = "0";
|
||||
}, autoHideMs);
|
||||
}
|
||||
}
|
||||
|
||||
function refreshEmptyState() {
|
||||
listEmpty.hidden = list.children.length > 0;
|
||||
}
|
||||
|
||||
// Render one item. The checkbox's accessible name is the item text, so
|
||||
// the snapshot walker exposes each item as a checkbox with its label and
|
||||
// checked state — that's what the UIWorker reads to resolve "the milk",
|
||||
// "the last one", "the checked ones", etc.
|
||||
function addItem(text, checked = false) {
|
||||
const trimmed = String(text ?? "").trim();
|
||||
if (!trimmed) return;
|
||||
|
||||
const li = document.createElement("li");
|
||||
li.className = "item";
|
||||
|
||||
const label = document.createElement("label");
|
||||
const cb = document.createElement("input");
|
||||
cb.type = "checkbox";
|
||||
cb.checked = !!checked;
|
||||
cb.setAttribute("aria-label", trimmed);
|
||||
cb.addEventListener("change", () => {
|
||||
li.classList.toggle("checked", cb.checked);
|
||||
});
|
||||
|
||||
const span = document.createElement("span");
|
||||
span.className = "item-text";
|
||||
span.textContent = trimmed;
|
||||
|
||||
label.appendChild(cb);
|
||||
label.appendChild(span);
|
||||
li.appendChild(label);
|
||||
li.classList.toggle("checked", !!checked);
|
||||
|
||||
list.appendChild(li);
|
||||
li.classList.add("item-flash");
|
||||
setTimeout(() => li.classList.remove("item-flash"), 1000);
|
||||
refreshEmptyState();
|
||||
}
|
||||
|
||||
function checkboxFor(ref) {
|
||||
const el = ref ? findElementByRef(ref) : null;
|
||||
if (!el) return null;
|
||||
if (el instanceof HTMLInputElement) return el;
|
||||
return el.querySelector?.("input[type=checkbox]") ?? null;
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────
|
||||
// Command handlers
|
||||
// ─────────────────────────────────────────────
|
||||
|
||||
function handleAddItem(payload) {
|
||||
addItem(payload?.text);
|
||||
}
|
||||
|
||||
function handleSetChecked(payload) {
|
||||
const cb = checkboxFor(payload?.ref);
|
||||
if (!cb) return;
|
||||
cb.checked = payload?.checked !== false;
|
||||
cb.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
}
|
||||
|
||||
function handleRemoveItem(payload) {
|
||||
const cb = checkboxFor(payload?.ref);
|
||||
const li = cb?.closest(".item");
|
||||
if (!li) return;
|
||||
li.remove();
|
||||
refreshEmptyState();
|
||||
}
|
||||
|
||||
// Standard ``highlight`` command: pulse the item row(s). The server may
|
||||
// send one ref per command; we flash the enclosing item.
|
||||
function handleHighlight(payload) {
|
||||
const cb = checkboxFor(payload?.ref);
|
||||
const row = cb?.closest(".item");
|
||||
if (!row) return;
|
||||
const duration = payload?.duration_ms ?? 1500;
|
||||
row.style.setProperty("--highlight-duration", `${duration}ms`);
|
||||
row.classList.remove("ui-highlight");
|
||||
void row.offsetWidth; // restart the animation if already running
|
||||
row.classList.add("ui-highlight");
|
||||
setTimeout(() => {
|
||||
row.classList.remove("ui-highlight");
|
||||
row.style.removeProperty("--highlight-duration");
|
||||
}, duration);
|
||||
}
|
||||
|
||||
function onUICommand(command, handler) {
|
||||
const listener = (data) => {
|
||||
if (data.command !== command) return;
|
||||
handler(data.payload);
|
||||
};
|
||||
client.on(RTVIEvent.UICommand, listener);
|
||||
return () => client.off(RTVIEvent.UICommand, listener);
|
||||
}
|
||||
|
||||
// Manual add (works without voice, and exercises the same snapshot path).
|
||||
addForm.addEventListener("submit", (e) => {
|
||||
e.preventDefault();
|
||||
addItem(addInput.value);
|
||||
addInput.value = "";
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────
|
||||
// Connection lifecycle
|
||||
// ─────────────────────────────────────────────
|
||||
|
||||
async function connect() {
|
||||
connectButton.disabled = true;
|
||||
setStatus("Connecting…");
|
||||
|
||||
client = new PipecatClient({
|
||||
transport: new SmallWebRTCTransport(),
|
||||
enableMic: true,
|
||||
enableCam: false,
|
||||
});
|
||||
|
||||
client.on(RTVIEvent.BotConnected, () => setStatus("Bot connected", 1500));
|
||||
client.on(RTVIEvent.Disconnected, () => {
|
||||
setStatus("Disconnected", 2000);
|
||||
connectButton.dataset.state = "";
|
||||
connectButton.textContent = "Connect";
|
||||
connectButton.disabled = false;
|
||||
teardownUI();
|
||||
});
|
||||
|
||||
client.on(RTVIEvent.TrackStarted, (track, participant) => {
|
||||
if (track.kind !== "audio") return;
|
||||
if (participant?.local) return;
|
||||
botAudio.srcObject = new MediaStream([track]);
|
||||
});
|
||||
|
||||
unsubscribes = [
|
||||
onUICommand("add_item", handleAddItem),
|
||||
onUICommand("set_checked", handleSetChecked),
|
||||
onUICommand("remove_item", handleRemoveItem),
|
||||
onUICommand("highlight", handleHighlight),
|
||||
];
|
||||
|
||||
try {
|
||||
await client.connect({ webrtcUrl: BOT_URL });
|
||||
client.startUISnapshotStream();
|
||||
connectButton.dataset.state = "connected";
|
||||
connectButton.textContent = "Disconnect";
|
||||
connectButton.disabled = false;
|
||||
setStatus("Connected. Try 'add milk and eggs'.", 5000);
|
||||
} catch (err) {
|
||||
console.error("Connect failed:", err);
|
||||
setStatus(`Connect failed: ${err.message ?? err}`, 4000);
|
||||
teardownUI();
|
||||
connectButton.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function disconnect() {
|
||||
connectButton.disabled = true;
|
||||
setStatus("Disconnecting…");
|
||||
try {
|
||||
await client?.disconnect();
|
||||
} finally {
|
||||
teardownUI();
|
||||
connectButton.dataset.state = "";
|
||||
connectButton.textContent = "Connect";
|
||||
connectButton.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
function teardownUI() {
|
||||
client?.stopUISnapshotStream();
|
||||
unsubscribes.forEach((unsubscribe) => unsubscribe());
|
||||
unsubscribes = [];
|
||||
if (botAudio.srcObject) botAudio.srcObject = null;
|
||||
client = undefined;
|
||||
}
|
||||
|
||||
connectButton.addEventListener("click", () => {
|
||||
if (connectButton.dataset.state === "connected") {
|
||||
disconnect();
|
||||
} else {
|
||||
connect();
|
||||
}
|
||||
});
|
||||
|
||||
refreshEmptyState();
|
||||
1128
examples/multi-worker/ui-worker/shopping-list/client/package-lock.json
generated
Normal file
1128
examples/multi-worker/ui-worker/shopping-list/client/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"name": "shopping-list-client",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@pipecat-ai/client-js": "1.9.0",
|
||||
"@pipecat-ai/small-webrtc-transport": "^1.10.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vite": "^8"
|
||||
}
|
||||
}
|
||||
216
examples/multi-worker/ui-worker/shopping-list/client/styles.css
Normal file
216
examples/multi-worker/ui-worker/shopping-list/client/styles.css
Normal file
@@ -0,0 +1,216 @@
|
||||
:root {
|
||||
--bg: #0f1115;
|
||||
--panel: #181b22;
|
||||
--panel-2: #1f242d;
|
||||
--text: #e8ebf0;
|
||||
--muted: #97a0ad;
|
||||
--accent: #5b8cff;
|
||||
--accent-soft: rgba(91, 140, 255, 0.18);
|
||||
--done: #5fd08a;
|
||||
--border: #2a2f3a;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
font-family:
|
||||
system-ui, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.topbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 16px 24px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
position: sticky;
|
||||
top: 0;
|
||||
background: rgba(15, 17, 21, 0.92);
|
||||
backdrop-filter: blur(6px);
|
||||
}
|
||||
|
||||
.topbar h1 {
|
||||
font-size: 20px;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.topbar-right {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
#status {
|
||||
font-size: 13px;
|
||||
color: var(--muted);
|
||||
transition: opacity 0.3s;
|
||||
}
|
||||
|
||||
#status[data-show="0"] {
|
||||
opacity: 0;
|
||||
}
|
||||
|
||||
#connect {
|
||||
font-size: 14px;
|
||||
padding: 8px 18px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid var(--accent);
|
||||
background: var(--accent-soft);
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
#connect[data-state="connected"] {
|
||||
border-color: var(--border);
|
||||
background: transparent;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
#connect:disabled {
|
||||
opacity: 0.5;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
main {
|
||||
max-width: 560px;
|
||||
margin: 0 auto;
|
||||
padding: 28px 24px 80px;
|
||||
}
|
||||
|
||||
.hint {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
margin: 0 0 18px;
|
||||
}
|
||||
|
||||
.add-form {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-bottom: 18px;
|
||||
}
|
||||
|
||||
.add-form input {
|
||||
flex: 1;
|
||||
font-size: 15px;
|
||||
padding: 10px 12px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid var(--border);
|
||||
background: var(--panel);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.add-form input:focus {
|
||||
outline: none;
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.add-form button {
|
||||
font-size: 14px;
|
||||
padding: 0 16px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid var(--border);
|
||||
background: var(--panel-2);
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.list {
|
||||
list-style: none;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.item {
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
transition:
|
||||
background 0.2s,
|
||||
opacity 0.2s;
|
||||
}
|
||||
|
||||
.item label {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
padding: 12px 14px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.item input[type="checkbox"] {
|
||||
width: 20px;
|
||||
height: 20px;
|
||||
accent-color: var(--done);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.item-text {
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.item.checked {
|
||||
opacity: 0.55;
|
||||
}
|
||||
|
||||
.item.checked .item-text {
|
||||
text-decoration: line-through;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.list-empty {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
text-align: center;
|
||||
padding: 32px 0;
|
||||
}
|
||||
|
||||
.list-empty[hidden] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
/* New-item flash so a voice-added item is easy to spot. */
|
||||
.item-flash {
|
||||
animation: item-flash 1s ease-out;
|
||||
}
|
||||
|
||||
@keyframes item-flash {
|
||||
0% {
|
||||
background: var(--accent-soft);
|
||||
border-color: var(--accent);
|
||||
}
|
||||
100% {
|
||||
background: var(--panel);
|
||||
border-color: var(--border);
|
||||
}
|
||||
}
|
||||
|
||||
/* Standard ``highlight`` command: a brief pulse. Duration is driven by
|
||||
the server-supplied duration_ms via the --highlight-duration var. */
|
||||
.ui-highlight {
|
||||
animation: ui-highlight var(--highlight-duration, 1500ms) ease-out;
|
||||
}
|
||||
|
||||
@keyframes ui-highlight {
|
||||
0% {
|
||||
transform: scale(1);
|
||||
box-shadow: 0 0 0 0 var(--accent-soft);
|
||||
}
|
||||
30% {
|
||||
transform: scale(1.02);
|
||||
box-shadow: 0 0 0 6px var(--accent-soft);
|
||||
border-color: var(--accent);
|
||||
}
|
||||
100% {
|
||||
transform: scale(1);
|
||||
box-shadow: 0 0 0 0 rgba(91, 140, 255, 0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
import { defineConfig } from "vite";
|
||||
|
||||
export default defineConfig({
|
||||
server: {
|
||||
port: 5173,
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user