Add shopping-list UIWorker example (bridge-free voice + UI)
Demonstrates the 'every input acts, may speak' pattern without bridging: a standard voice pipeline (STT → LLM → TTS) whose LLM only converses, plus a separate UIWorker that does all the list work. The voice pipeline's user aggregator fires on_user_turn_stopped each turn and dispatches the transcript to the UIWorker as a respond job (a bus message); the UIWorker reads the auto-injected <ui_state> snapshot and drives the list silently via add_item / set_checked / remove_item commands (plus the standard highlight). Items are checkboxes whose label and checked state the snapshot exposes. Includes a vanilla-JS client following the existing UI-demo client style.
This commit is contained in:
92
examples/multi-worker/ui-worker/shopping-list/README.md
Normal file
92
examples/multi-worker/ui-worker/shopping-list/README.md
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
# shopping-list
|
||||||
|
|
||||||
|
Every voice turn drives the UI; speech is incidental. The user builds a
|
||||||
|
shopping list by talking — "add milk and eggs", "check off the bread",
|
||||||
|
"drop the last one", "what's left?" — and the list updates on screen
|
||||||
|
every turn. The assistant may also say something back. The two halves run
|
||||||
|
in parallel on separate workers; neither calls the other as a tool.
|
||||||
|
|
||||||
|
This is the **bridge-free** pattern for "every input acts, may speak."
|
||||||
|
|
||||||
|
## What it shows
|
||||||
|
|
||||||
|
- **A standard voice pipeline + a UIWorker, no bridging.** The voice
|
||||||
|
layer is an ordinary `transport → STT → LLM → TTS` pipeline whose LLM
|
||||||
|
just converses (no tools, never touches the list). Its user aggregator
|
||||||
|
fires `on_user_turn_stopped` once per user turn, and that handler
|
||||||
|
dispatches the transcript to the UIWorker as a `respond` job
|
||||||
|
(`worker.job("ui", name="respond", payload={"query": transcript})`) —
|
||||||
|
a bus message. The UIWorker does all the list work, silently, on its
|
||||||
|
own worker, so its LLM output never reaches TTS.
|
||||||
|
- **Snapshot-driven action.** Before each UIWorker inference, the current
|
||||||
|
`<ui_state>` is auto-injected (via the LLM's `on_before_process_frame`
|
||||||
|
hook), so the worker resolves "the milk", "the last one", "the checked
|
||||||
|
ones" against the live list.
|
||||||
|
- **Custom UI commands.** A single bundled `update_list` tool maps the
|
||||||
|
request to `add_item` / `set_checked` / `remove_item` commands (plus the
|
||||||
|
standard `highlight`, used to *show* what's left). Each item is a
|
||||||
|
checkbox whose accessible name is the item text, so the snapshot exposes
|
||||||
|
every item's label and checked state.
|
||||||
|
|
||||||
|
## What it adds vs. the prior demos
|
||||||
|
|
||||||
|
The other demos use the **request/response delegation** shape: a voice LLM
|
||||||
|
decides, via a tool call, when to consult the UIWorker. This one removes
|
||||||
|
that decision — **every** user turn goes to the UIWorker automatically,
|
||||||
|
and the voice LLM runs independently for conversation. The wiring is a
|
||||||
|
single `on_user_turn_stopped` handler instead of a delegating tool.
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
Two terminals.
|
||||||
|
|
||||||
|
**Terminal 1 — bot:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd examples/multi-worker/ui-worker/shopping-list
|
||||||
|
uv run python bot.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The bot starts on `http://localhost:7860`.
|
||||||
|
|
||||||
|
**Terminal 2 — client:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd examples/multi-worker/ui-worker/shopping-list/client
|
||||||
|
npm install # one-time
|
||||||
|
npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
Open `http://localhost:5173` and click **Connect**.
|
||||||
|
|
||||||
|
## What to try
|
||||||
|
|
||||||
|
- _"Add milk and a dozen eggs."_ — both items appear; the voice
|
||||||
|
acknowledges.
|
||||||
|
- _"Add bread, butter, and coffee."_ — three more land.
|
||||||
|
- _"Check off the bread."_ — it gets ticked and struck through.
|
||||||
|
- _"Actually, drop the butter."_ — removed.
|
||||||
|
- _"Clear the ones I've already got."_ — removes everything checked.
|
||||||
|
- _"What's left?"_ — the unchecked items pulse, and the voice tells you.
|
||||||
|
- _"Hi there!"_ — the voice greets you; the list is unchanged (the
|
||||||
|
UIWorker no-ops).
|
||||||
|
|
||||||
|
You can also type in the box to add items by hand — they go through the
|
||||||
|
same snapshot path, so the worker sees them next turn.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- `OPENAI_API_KEY`
|
||||||
|
- `DEEPGRAM_API_KEY`
|
||||||
|
- `CARTESIA_API_KEY`
|
||||||
|
|
||||||
|
A `.env` in the example folder is the easiest way to set these (see
|
||||||
|
`examples/multi-worker/env.example`).
|
||||||
|
|
||||||
|
## What this example _doesn't_ show
|
||||||
|
|
||||||
|
The voice layer is **blind to the screen** — it converses and answers from
|
||||||
|
the dialogue so far, but it doesn't read the live list (only the UIWorker
|
||||||
|
sees the snapshot). For "what's left?" the UIWorker highlights the answer
|
||||||
|
visually while the voice gives a from-memory summary. The list is also not
|
||||||
|
persisted — refresh and it's gone.
|
||||||
325
examples/multi-worker/ui-worker/shopping-list/bot.py
Normal file
325
examples/multi-worker/ui-worker/shopping-list/bot.py
Normal file
@@ -0,0 +1,325 @@
|
|||||||
|
#
|
||||||
|
# Copyright (c) 2026, Daily
|
||||||
|
#
|
||||||
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
|
#
|
||||||
|
|
||||||
|
"""Shopping list — every voice turn drives the UI; speech is incidental.
|
||||||
|
|
||||||
|
The page renders a shopping list. The user talks: "add milk and eggs",
|
||||||
|
"check off the bread", "drop the last one", "what's left?". Every user
|
||||||
|
turn updates the list on screen; the assistant may also say something
|
||||||
|
back. The two halves run in parallel on separate workers — neither calls
|
||||||
|
the other as a tool.
|
||||||
|
|
||||||
|
This is the bridge-free pattern for "every input acts, may speak":
|
||||||
|
|
||||||
|
- The **voice layer** is an ordinary voice pipeline (STT → LLM → TTS).
|
||||||
|
Its LLM just converses — it has no tools and never touches the list.
|
||||||
|
- A **UIWorker** ("ui") does all the list work. It is *not* in the voice
|
||||||
|
pipeline. Instead, the voice pipeline's user aggregator fires
|
||||||
|
``on_user_turn_stopped`` once per user turn; that handler dispatches
|
||||||
|
the transcript to the UIWorker as a ``respond`` job (a bus message).
|
||||||
|
The UIWorker reads the current ``<ui_state>`` snapshot (auto-injected
|
||||||
|
before its inference) and calls ``update_list`` to add / check /
|
||||||
|
remove items. It acts silently — its LLM output never reaches TTS,
|
||||||
|
because it lives on its own worker.
|
||||||
|
|
||||||
|
Architecture::
|
||||||
|
|
||||||
|
Voice pipeline (PipelineWorker "main", owns transport + RTVI):
|
||||||
|
transport.in → STT → user_agg → LLM → TTS → transport.out → assistant_agg
|
||||||
|
└── @on_user_turn_stopped: worker.job("ui", name="respond",
|
||||||
|
payload={"query": transcript})
|
||||||
|
|
||||||
|
UIWorker "ui" (job-based, acts silently):
|
||||||
|
└── @tool update_list(add, check, uncheck, remove, highlight)
|
||||||
|
└── send_command("add_item" / "set_checked" / "remove_item")
|
||||||
|
+ respond_to_job() # no speak
|
||||||
|
|
||||||
|
Run::
|
||||||
|
|
||||||
|
uv run python bot.py
|
||||||
|
|
||||||
|
Then open the client at ``http://localhost:5173`` (see ``README.md``).
|
||||||
|
|
||||||
|
Requirements:
|
||||||
|
|
||||||
|
- OPENAI_API_KEY
|
||||||
|
- DEEPGRAM_API_KEY
|
||||||
|
- CARTESIA_API_KEY
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
|
from pipecat.frames.frames import LLMRunFrame
|
||||||
|
from pipecat.pipeline.job_context import JobError
|
||||||
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
|
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
|
||||||
|
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||||
|
from pipecat.processors.aggregators.llm_response_universal import (
|
||||||
|
LLMContextAggregatorPair,
|
||||||
|
LLMUserAggregatorParams,
|
||||||
|
)
|
||||||
|
from pipecat.runner.types import RunnerArguments
|
||||||
|
from pipecat.runner.utils import create_transport
|
||||||
|
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||||
|
from pipecat.services.deepgram.stt import DeepgramSTTService
|
||||||
|
from pipecat.services.llm_service import FunctionCallParams
|
||||||
|
from pipecat.services.openai.llm import OpenAILLMService
|
||||||
|
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||||
|
from pipecat.transports.daily.transport import DailyParams
|
||||||
|
from pipecat.workers.llm import tool
|
||||||
|
from pipecat.workers.ui import UIWorker
|
||||||
|
|
||||||
|
load_dotenv(override=True)
|
||||||
|
|
||||||
|
MAIN_NAME = "main"
|
||||||
|
UI_NAME = "ui"
|
||||||
|
|
||||||
|
transport_params = {
|
||||||
|
"daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||||
|
"webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
VOICE_PROMPT = """\
|
||||||
|
You are the voice of a shopping-list assistant. A separate UI layer \
|
||||||
|
sees the list and updates it on screen; you do NOT touch the list \
|
||||||
|
yourself and you cannot see the screen.
|
||||||
|
|
||||||
|
Your job is to be a brief, friendly companion. Acknowledge what the \
|
||||||
|
user asked for ("Sure, adding milk and eggs"), react naturally, and \
|
||||||
|
answer questions from the conversation so far ("So far you've asked \
|
||||||
|
for milk, eggs, and bread"). Keep every reply to one short spoken \
|
||||||
|
sentence. Don't describe how you're updating the list — the screen \
|
||||||
|
shows that. For a plain greeting, greet back warmly."""
|
||||||
|
|
||||||
|
|
||||||
|
# The UI wire-format guide (UI_STATE_PROMPT_GUIDE) is appended to the LLM's
|
||||||
|
# system instruction automatically by UIWorker, so this prompt only needs the
|
||||||
|
# app-specific behavior.
|
||||||
|
UI_PROMPT = """\
|
||||||
|
You maintain a shopping list by voice. The current ``<ui_state>`` \
|
||||||
|
block is in your context. Each list item is a checkbox with a \
|
||||||
|
snapshot ref (e.g. ``e5``), a label (the item text), and a checked / \
|
||||||
|
unchecked state. Use the labels to resolve which item the user means.
|
||||||
|
|
||||||
|
## Tool: update_list
|
||||||
|
|
||||||
|
Every turn calls ``update_list`` exactly once — even when there's \
|
||||||
|
nothing to change (call it with no arguments). One tool call per turn.
|
||||||
|
|
||||||
|
``update_list(add=None, check=None, uncheck=None, remove=None, highlight=None)``:
|
||||||
|
|
||||||
|
- ``add`` (OPTIONAL): a list of item texts to append. Split a spoken \
|
||||||
|
list into separate items ("milk, eggs and bread" → \
|
||||||
|
``["milk", "eggs", "bread"]``). Normalize lightly (lowercase, drop \
|
||||||
|
filler like "some" / "a couple of").
|
||||||
|
- ``check`` (OPTIONAL): a list of refs to mark done.
|
||||||
|
- ``uncheck`` (OPTIONAL): a list of refs to mark not done.
|
||||||
|
- ``remove`` (OPTIONAL): a list of refs to delete.
|
||||||
|
- ``highlight`` (OPTIONAL): a list of refs to flash briefly. Use it to \
|
||||||
|
*show* the user something when they ask (e.g. highlight the unchecked \
|
||||||
|
items for "what's left?").
|
||||||
|
|
||||||
|
## Decision rules
|
||||||
|
|
||||||
|
- **"Add X (and Y)"** → ``add`` with one entry per item.
|
||||||
|
- **"Check off / got / cross out X"** → resolve X's ref from \
|
||||||
|
``<ui_state>``, set ``check``.
|
||||||
|
- **"Uncheck / put back X"** → set ``uncheck``.
|
||||||
|
- **"Remove / delete / never mind X"**, **"clear the checked ones"** → \
|
||||||
|
set ``remove`` with the matching refs (read the checked state from \
|
||||||
|
``<ui_state>`` for "the checked ones").
|
||||||
|
- **"the last one"** → the last item in ``<ui_state>``.
|
||||||
|
- **"What's left? / what do I still need?"** → ``highlight`` the \
|
||||||
|
unchecked items. **"What's on my list?"** → ``highlight`` everything. \
|
||||||
|
(The voice layer speaks; you just point.)
|
||||||
|
- **A greeting or anything not about the list** → call ``update_list`` \
|
||||||
|
with no arguments (no-op) so the turn completes.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
(refs are illustrative; use the actual refs from the current \
|
||||||
|
``<ui_state>``)
|
||||||
|
|
||||||
|
- "Add milk and a dozen eggs." → \
|
||||||
|
``update_list(add=["milk", "eggs"])``
|
||||||
|
- "Check off the milk." → ``update_list(check=["e5"])``
|
||||||
|
- "Actually drop the eggs." → ``update_list(remove=["e7"])``
|
||||||
|
- "Clear the ones I've got." (e5, e9 checked) → \
|
||||||
|
``update_list(remove=["e5", "e9"])``
|
||||||
|
- "What's left?" (e7, e11 unchecked) → \
|
||||||
|
``update_list(highlight=["e7", "e11"])``
|
||||||
|
- "Hey there." → ``update_list()``"""
|
||||||
|
|
||||||
|
|
||||||
|
class ListWorker(UIWorker):
|
||||||
|
"""UIWorker that maintains the shopping list, silently.
|
||||||
|
|
||||||
|
A single bundled ``update_list`` tool, always called once per turn,
|
||||||
|
maps the user's request to ``add_item`` / ``set_checked`` /
|
||||||
|
``remove_item`` UI commands (plus the standard ``highlight``). The
|
||||||
|
tool completes the ``respond`` job with no spoken reply
|
||||||
|
(``respond_to_job()`` with no ``speak``) — the separate voice layer
|
||||||
|
owns speech.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
llm = OpenAILLMService(
|
||||||
|
api_key=os.environ["OPENAI_API_KEY"],
|
||||||
|
settings=OpenAILLMService.Settings(system_instruction=UI_PROMPT),
|
||||||
|
)
|
||||||
|
super().__init__(UI_NAME, llm=llm)
|
||||||
|
|
||||||
|
@tool
|
||||||
|
async def update_list(
|
||||||
|
self,
|
||||||
|
params: FunctionCallParams,
|
||||||
|
add: list[str] | None = None,
|
||||||
|
check: list[str] | None = None,
|
||||||
|
uncheck: list[str] | None = None,
|
||||||
|
remove: list[str] | None = None,
|
||||||
|
highlight: list[str] | None = None,
|
||||||
|
):
|
||||||
|
"""Update the shopping list. Called exactly once per turn.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
add: New item texts to append (one entry per item).
|
||||||
|
check: Snapshot refs of items to mark done.
|
||||||
|
uncheck: Snapshot refs of items to mark not done.
|
||||||
|
remove: Snapshot refs of items to delete.
|
||||||
|
highlight: Snapshot refs of items to flash briefly (e.g. to
|
||||||
|
show what's left when the user asks).
|
||||||
|
"""
|
||||||
|
logger.info(
|
||||||
|
f"{self}: update_list(add={add!r}, check={check!r}, uncheck={uncheck!r}, "
|
||||||
|
f"remove={remove!r}, highlight={highlight!r})"
|
||||||
|
)
|
||||||
|
# Defensive guards: skip malformed entries so a stray value can't
|
||||||
|
# crash the tool before respond_to_job fires (which would hold the
|
||||||
|
# single-flight lock until the requester's timeout).
|
||||||
|
for text in add or []:
|
||||||
|
if isinstance(text, str) and text.strip():
|
||||||
|
await self.send_command("add_item", {"text": text.strip()})
|
||||||
|
for ref in check or []:
|
||||||
|
if isinstance(ref, str):
|
||||||
|
await self.send_command("set_checked", {"ref": ref, "checked": True})
|
||||||
|
for ref in uncheck or []:
|
||||||
|
if isinstance(ref, str):
|
||||||
|
await self.send_command("set_checked", {"ref": ref, "checked": False})
|
||||||
|
for ref in remove or []:
|
||||||
|
if isinstance(ref, str):
|
||||||
|
await self.send_command("remove_item", {"ref": ref})
|
||||||
|
for ref in highlight or []:
|
||||||
|
if isinstance(ref, str):
|
||||||
|
await self.highlight(ref)
|
||||||
|
await self.respond_to_job()
|
||||||
|
await params.result_callback(None)
|
||||||
|
|
||||||
|
|
||||||
|
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||||
|
logger.info("Starting shopping-list bot")
|
||||||
|
|
||||||
|
runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
|
||||||
|
|
||||||
|
stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
|
||||||
|
tts = CartesiaTTSService(
|
||||||
|
api_key=os.environ["CARTESIA_API_KEY"],
|
||||||
|
settings=CartesiaTTSService.Settings(
|
||||||
|
voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
llm = OpenAILLMService(
|
||||||
|
api_key=os.environ["OPENAI_API_KEY"],
|
||||||
|
settings=OpenAILLMService.Settings(system_instruction=VOICE_PROMPT),
|
||||||
|
)
|
||||||
|
|
||||||
|
context = LLMContext()
|
||||||
|
aggregators = LLMContextAggregatorPair(
|
||||||
|
context,
|
||||||
|
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
|
||||||
|
)
|
||||||
|
user_aggregator = aggregators.user()
|
||||||
|
assistant_aggregator = aggregators.assistant()
|
||||||
|
|
||||||
|
pipeline = Pipeline(
|
||||||
|
[
|
||||||
|
transport.input(),
|
||||||
|
stt,
|
||||||
|
user_aggregator,
|
||||||
|
llm,
|
||||||
|
tts,
|
||||||
|
transport.output(),
|
||||||
|
assistant_aggregator,
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
worker = PipelineWorker(
|
||||||
|
pipeline,
|
||||||
|
name=MAIN_NAME,
|
||||||
|
params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
|
||||||
|
idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Every user turn drives the UI: forward the transcript to the UIWorker
|
||||||
|
# as a respond job (a bus message). Fire it from the turn-stopped event
|
||||||
|
# so the voice LLM (which runs from the same turn) and the UIWorker act
|
||||||
|
# in parallel — neither calls the other as a tool. This handler runs in
|
||||||
|
# its own task, so awaiting the job here does not block the voice path.
|
||||||
|
@user_aggregator.event_handler("on_user_turn_stopped")
|
||||||
|
async def on_user_turn_stopped(aggregator, strategy, message):
|
||||||
|
transcript = (message.content or "").strip()
|
||||||
|
if not transcript:
|
||||||
|
return
|
||||||
|
logger.info(f"Dispatching turn to UI worker: {transcript!r}")
|
||||||
|
try:
|
||||||
|
async with worker.job(
|
||||||
|
UI_NAME, name="respond", payload={"query": transcript}, timeout=15
|
||||||
|
):
|
||||||
|
pass
|
||||||
|
except JobError as e:
|
||||||
|
logger.warning(f"ui job failed: {e}")
|
||||||
|
|
||||||
|
@transport.event_handler("on_client_connected")
|
||||||
|
async def on_client_connected(transport, client):
|
||||||
|
logger.info("Client connected")
|
||||||
|
context.add_message(
|
||||||
|
{
|
||||||
|
"role": "developer",
|
||||||
|
"content": (
|
||||||
|
"Greet the user briefly. Tell them they can build their "
|
||||||
|
"shopping list by voice — add items, check things off, or "
|
||||||
|
"ask what's left. One short sentence."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
await worker.queue_frame(LLMRunFrame())
|
||||||
|
|
||||||
|
@transport.event_handler("on_client_disconnected")
|
||||||
|
async def on_client_disconnected(transport, client):
|
||||||
|
logger.info("Client disconnected")
|
||||||
|
await runner.cancel()
|
||||||
|
|
||||||
|
await runner.launch_worker(ListWorker())
|
||||||
|
await runner.launch_worker(worker)
|
||||||
|
|
||||||
|
await runner.run()
|
||||||
|
|
||||||
|
|
||||||
|
async def bot(runner_args: RunnerArguments):
|
||||||
|
"""Main bot entry point compatible with Pipecat Cloud."""
|
||||||
|
transport = await create_transport(runner_args, transport_params)
|
||||||
|
await run_bot(transport, runner_args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
from pipecat.runner.run import main
|
||||||
|
|
||||||
|
main()
|
||||||
26
examples/multi-worker/ui-worker/shopping-list/client/.gitignore
vendored
Normal file
26
examples/multi-worker/ui-worker/shopping-list/client/.gitignore
vendored
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
# Logs
|
||||||
|
logs
|
||||||
|
*.log
|
||||||
|
npm-debug.log*
|
||||||
|
yarn-debug.log*
|
||||||
|
yarn-error.log*
|
||||||
|
pnpm-debug.log*
|
||||||
|
lerna-debug.log*
|
||||||
|
|
||||||
|
node_modules
|
||||||
|
dist
|
||||||
|
dist-ssr
|
||||||
|
*.local
|
||||||
|
|
||||||
|
# Editor directories and files
|
||||||
|
.vscode/*
|
||||||
|
!.vscode/extensions.json
|
||||||
|
.idea
|
||||||
|
.DS_Store
|
||||||
|
*.suo
|
||||||
|
*.ntvs*
|
||||||
|
*.njsproj
|
||||||
|
*.sln
|
||||||
|
*.sw?
|
||||||
|
|
||||||
|
.vite
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>Shopping list — Pipecat UIWorker</title>
|
||||||
|
<link rel="stylesheet" href="styles.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header class="topbar">
|
||||||
|
<h1>🛒 Shopping list</h1>
|
||||||
|
<div class="topbar-right">
|
||||||
|
<span id="status" data-show="0"></span>
|
||||||
|
<button id="connect">Connect</button>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<p class="hint">
|
||||||
|
Talk to build your list — "add milk and eggs", "check off the
|
||||||
|
bread", "what's left?". You can also type an item below.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<form id="add-form" class="add-form">
|
||||||
|
<input
|
||||||
|
id="add-input"
|
||||||
|
type="text"
|
||||||
|
placeholder="Add an item…"
|
||||||
|
autocomplete="off"
|
||||||
|
aria-label="Add an item"
|
||||||
|
/>
|
||||||
|
<button type="submit">Add</button>
|
||||||
|
</form>
|
||||||
|
|
||||||
|
<ul id="list" class="list" aria-label="Shopping list"></ul>
|
||||||
|
<p id="list-empty" class="list-empty">
|
||||||
|
Your list is empty. Say "add milk and eggs".
|
||||||
|
</p>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<audio id="bot-audio" autoplay></audio>
|
||||||
|
<script type="module" src="main.js"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
230
examples/multi-worker/ui-worker/shopping-list/client/main.js
Normal file
230
examples/multi-worker/ui-worker/shopping-list/client/main.js
Normal file
@@ -0,0 +1,230 @@
|
|||||||
|
/**
|
||||||
|
* Shopping list — vanilla JS client.
|
||||||
|
*
|
||||||
|
* The bridge-free "every input acts, may speak" pattern:
|
||||||
|
*
|
||||||
|
* - A standard voice pipeline runs on the bot. Its LLM converses (you
|
||||||
|
* hear it), and its user aggregator forwards each transcript to a
|
||||||
|
* separate UIWorker.
|
||||||
|
* - The UIWorker reads the snapshot of THIS page and drives the list
|
||||||
|
* silently via three custom commands — ``add_item``, ``set_checked``,
|
||||||
|
* ``remove_item`` — plus the standard ``highlight`` (used to show
|
||||||
|
* "what's left").
|
||||||
|
*
|
||||||
|
* Each item is a checkbox with its text as the accessible name, so the
|
||||||
|
* snapshot exposes every item's label and checked state to the worker.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import {
|
||||||
|
PipecatClient,
|
||||||
|
RTVIEvent,
|
||||||
|
findElementByRef,
|
||||||
|
} from "@pipecat-ai/client-js";
|
||||||
|
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";
|
||||||
|
|
||||||
|
const BOT_URL = "http://localhost:7860/api/offer";
|
||||||
|
|
||||||
|
const connectButton = document.getElementById("connect");
|
||||||
|
const status = document.getElementById("status");
|
||||||
|
const botAudio = document.getElementById("bot-audio");
|
||||||
|
const list = document.getElementById("list");
|
||||||
|
const listEmpty = document.getElementById("list-empty");
|
||||||
|
const addForm = document.getElementById("add-form");
|
||||||
|
const addInput = document.getElementById("add-input");
|
||||||
|
|
||||||
|
let client;
|
||||||
|
let unsubscribes = [];
|
||||||
|
|
||||||
|
function setStatus(text, autoHideMs = 0) {
|
||||||
|
status.textContent = text;
|
||||||
|
status.dataset.show = text ? "1" : "0";
|
||||||
|
if (text && autoHideMs > 0) {
|
||||||
|
setTimeout(() => {
|
||||||
|
if (status.textContent === text) status.dataset.show = "0";
|
||||||
|
}, autoHideMs);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function refreshEmptyState() {
|
||||||
|
listEmpty.hidden = list.children.length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Render one item. The checkbox's accessible name is the item text, so
|
||||||
|
// the snapshot walker exposes each item as a checkbox with its label and
|
||||||
|
// checked state — that's what the UIWorker reads to resolve "the milk",
|
||||||
|
// "the last one", "the checked ones", etc.
|
||||||
|
function addItem(text, checked = false) {
|
||||||
|
const trimmed = String(text ?? "").trim();
|
||||||
|
if (!trimmed) return;
|
||||||
|
|
||||||
|
const li = document.createElement("li");
|
||||||
|
li.className = "item";
|
||||||
|
|
||||||
|
const label = document.createElement("label");
|
||||||
|
const cb = document.createElement("input");
|
||||||
|
cb.type = "checkbox";
|
||||||
|
cb.checked = !!checked;
|
||||||
|
cb.setAttribute("aria-label", trimmed);
|
||||||
|
cb.addEventListener("change", () => {
|
||||||
|
li.classList.toggle("checked", cb.checked);
|
||||||
|
});
|
||||||
|
|
||||||
|
const span = document.createElement("span");
|
||||||
|
span.className = "item-text";
|
||||||
|
span.textContent = trimmed;
|
||||||
|
|
||||||
|
label.appendChild(cb);
|
||||||
|
label.appendChild(span);
|
||||||
|
li.appendChild(label);
|
||||||
|
li.classList.toggle("checked", !!checked);
|
||||||
|
|
||||||
|
list.appendChild(li);
|
||||||
|
li.classList.add("item-flash");
|
||||||
|
setTimeout(() => li.classList.remove("item-flash"), 1000);
|
||||||
|
refreshEmptyState();
|
||||||
|
}
|
||||||
|
|
||||||
|
function checkboxFor(ref) {
|
||||||
|
const el = ref ? findElementByRef(ref) : null;
|
||||||
|
if (!el) return null;
|
||||||
|
if (el instanceof HTMLInputElement) return el;
|
||||||
|
return el.querySelector?.("input[type=checkbox]") ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─────────────────────────────────────────────
|
||||||
|
// Command handlers
|
||||||
|
// ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
function handleAddItem(payload) {
|
||||||
|
addItem(payload?.text);
|
||||||
|
}
|
||||||
|
|
||||||
|
function handleSetChecked(payload) {
|
||||||
|
const cb = checkboxFor(payload?.ref);
|
||||||
|
if (!cb) return;
|
||||||
|
cb.checked = payload?.checked !== false;
|
||||||
|
cb.dispatchEvent(new Event("change", { bubbles: true }));
|
||||||
|
}
|
||||||
|
|
||||||
|
function handleRemoveItem(payload) {
|
||||||
|
const cb = checkboxFor(payload?.ref);
|
||||||
|
const li = cb?.closest(".item");
|
||||||
|
if (!li) return;
|
||||||
|
li.remove();
|
||||||
|
refreshEmptyState();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Standard ``highlight`` command: pulse the item row(s). The server may
|
||||||
|
// send one ref per command; we flash the enclosing item.
|
||||||
|
function handleHighlight(payload) {
|
||||||
|
const cb = checkboxFor(payload?.ref);
|
||||||
|
const row = cb?.closest(".item");
|
||||||
|
if (!row) return;
|
||||||
|
const duration = payload?.duration_ms ?? 1500;
|
||||||
|
row.style.setProperty("--highlight-duration", `${duration}ms`);
|
||||||
|
row.classList.remove("ui-highlight");
|
||||||
|
void row.offsetWidth; // restart the animation if already running
|
||||||
|
row.classList.add("ui-highlight");
|
||||||
|
setTimeout(() => {
|
||||||
|
row.classList.remove("ui-highlight");
|
||||||
|
row.style.removeProperty("--highlight-duration");
|
||||||
|
}, duration);
|
||||||
|
}
|
||||||
|
|
||||||
|
function onUICommand(command, handler) {
|
||||||
|
const listener = (data) => {
|
||||||
|
if (data.command !== command) return;
|
||||||
|
handler(data.payload);
|
||||||
|
};
|
||||||
|
client.on(RTVIEvent.UICommand, listener);
|
||||||
|
return () => client.off(RTVIEvent.UICommand, listener);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Manual add (works without voice, and exercises the same snapshot path).
|
||||||
|
addForm.addEventListener("submit", (e) => {
|
||||||
|
e.preventDefault();
|
||||||
|
addItem(addInput.value);
|
||||||
|
addInput.value = "";
|
||||||
|
});
|
||||||
|
|
||||||
|
// ─────────────────────────────────────────────
|
||||||
|
// Connection lifecycle
|
||||||
|
// ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
async function connect() {
|
||||||
|
connectButton.disabled = true;
|
||||||
|
setStatus("Connecting…");
|
||||||
|
|
||||||
|
client = new PipecatClient({
|
||||||
|
transport: new SmallWebRTCTransport(),
|
||||||
|
enableMic: true,
|
||||||
|
enableCam: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
client.on(RTVIEvent.BotConnected, () => setStatus("Bot connected", 1500));
|
||||||
|
client.on(RTVIEvent.Disconnected, () => {
|
||||||
|
setStatus("Disconnected", 2000);
|
||||||
|
connectButton.dataset.state = "";
|
||||||
|
connectButton.textContent = "Connect";
|
||||||
|
connectButton.disabled = false;
|
||||||
|
teardownUI();
|
||||||
|
});
|
||||||
|
|
||||||
|
client.on(RTVIEvent.TrackStarted, (track, participant) => {
|
||||||
|
if (track.kind !== "audio") return;
|
||||||
|
if (participant?.local) return;
|
||||||
|
botAudio.srcObject = new MediaStream([track]);
|
||||||
|
});
|
||||||
|
|
||||||
|
unsubscribes = [
|
||||||
|
onUICommand("add_item", handleAddItem),
|
||||||
|
onUICommand("set_checked", handleSetChecked),
|
||||||
|
onUICommand("remove_item", handleRemoveItem),
|
||||||
|
onUICommand("highlight", handleHighlight),
|
||||||
|
];
|
||||||
|
|
||||||
|
try {
|
||||||
|
await client.connect({ webrtcUrl: BOT_URL });
|
||||||
|
client.startUISnapshotStream();
|
||||||
|
connectButton.dataset.state = "connected";
|
||||||
|
connectButton.textContent = "Disconnect";
|
||||||
|
connectButton.disabled = false;
|
||||||
|
setStatus("Connected. Try 'add milk and eggs'.", 5000);
|
||||||
|
} catch (err) {
|
||||||
|
console.error("Connect failed:", err);
|
||||||
|
setStatus(`Connect failed: ${err.message ?? err}`, 4000);
|
||||||
|
teardownUI();
|
||||||
|
connectButton.disabled = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function disconnect() {
|
||||||
|
connectButton.disabled = true;
|
||||||
|
setStatus("Disconnecting…");
|
||||||
|
try {
|
||||||
|
await client?.disconnect();
|
||||||
|
} finally {
|
||||||
|
teardownUI();
|
||||||
|
connectButton.dataset.state = "";
|
||||||
|
connectButton.textContent = "Connect";
|
||||||
|
connectButton.disabled = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function teardownUI() {
|
||||||
|
client?.stopUISnapshotStream();
|
||||||
|
unsubscribes.forEach((unsubscribe) => unsubscribe());
|
||||||
|
unsubscribes = [];
|
||||||
|
if (botAudio.srcObject) botAudio.srcObject = null;
|
||||||
|
client = undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
connectButton.addEventListener("click", () => {
|
||||||
|
if (connectButton.dataset.state === "connected") {
|
||||||
|
disconnect();
|
||||||
|
} else {
|
||||||
|
connect();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
refreshEmptyState();
|
||||||
1128
examples/multi-worker/ui-worker/shopping-list/client/package-lock.json
generated
Normal file
1128
examples/multi-worker/ui-worker/shopping-list/client/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"name": "shopping-list-client",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {
|
||||||
|
"dev": "vite",
|
||||||
|
"build": "vite build",
|
||||||
|
"preview": "vite preview"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@pipecat-ai/client-js": "1.9.0",
|
||||||
|
"@pipecat-ai/small-webrtc-transport": "^1.10.2"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"vite": "^8"
|
||||||
|
}
|
||||||
|
}
|
||||||
216
examples/multi-worker/ui-worker/shopping-list/client/styles.css
Normal file
216
examples/multi-worker/ui-worker/shopping-list/client/styles.css
Normal file
@@ -0,0 +1,216 @@
|
|||||||
|
:root {
|
||||||
|
--bg: #0f1115;
|
||||||
|
--panel: #181b22;
|
||||||
|
--panel-2: #1f242d;
|
||||||
|
--text: #e8ebf0;
|
||||||
|
--muted: #97a0ad;
|
||||||
|
--accent: #5b8cff;
|
||||||
|
--accent-soft: rgba(91, 140, 255, 0.18);
|
||||||
|
--done: #5fd08a;
|
||||||
|
--border: #2a2f3a;
|
||||||
|
}
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
font-family:
|
||||||
|
system-ui, -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||||
|
background: var(--bg);
|
||||||
|
color: var(--text);
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.topbar {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
padding: 16px 24px;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
position: sticky;
|
||||||
|
top: 0;
|
||||||
|
background: rgba(15, 17, 21, 0.92);
|
||||||
|
backdrop-filter: blur(6px);
|
||||||
|
}
|
||||||
|
|
||||||
|
.topbar h1 {
|
||||||
|
font-size: 20px;
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.topbar-right {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
#status {
|
||||||
|
font-size: 13px;
|
||||||
|
color: var(--muted);
|
||||||
|
transition: opacity 0.3s;
|
||||||
|
}
|
||||||
|
|
||||||
|
#status[data-show="0"] {
|
||||||
|
opacity: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
#connect {
|
||||||
|
font-size: 14px;
|
||||||
|
padding: 8px 18px;
|
||||||
|
border-radius: 8px;
|
||||||
|
border: 1px solid var(--accent);
|
||||||
|
background: var(--accent-soft);
|
||||||
|
color: var(--text);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
#connect[data-state="connected"] {
|
||||||
|
border-color: var(--border);
|
||||||
|
background: transparent;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
#connect:disabled {
|
||||||
|
opacity: 0.5;
|
||||||
|
cursor: default;
|
||||||
|
}
|
||||||
|
|
||||||
|
main {
|
||||||
|
max-width: 560px;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 28px 24px 80px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.hint {
|
||||||
|
color: var(--muted);
|
||||||
|
font-size: 14px;
|
||||||
|
margin: 0 0 18px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.add-form {
|
||||||
|
display: flex;
|
||||||
|
gap: 8px;
|
||||||
|
margin-bottom: 18px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.add-form input {
|
||||||
|
flex: 1;
|
||||||
|
font-size: 15px;
|
||||||
|
padding: 10px 12px;
|
||||||
|
border-radius: 8px;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
background: var(--panel);
|
||||||
|
color: var(--text);
|
||||||
|
}
|
||||||
|
|
||||||
|
.add-form input:focus {
|
||||||
|
outline: none;
|
||||||
|
border-color: var(--accent);
|
||||||
|
}
|
||||||
|
|
||||||
|
.add-form button {
|
||||||
|
font-size: 14px;
|
||||||
|
padding: 0 16px;
|
||||||
|
border-radius: 8px;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
background: var(--panel-2);
|
||||||
|
color: var(--text);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.list {
|
||||||
|
list-style: none;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.item {
|
||||||
|
background: var(--panel);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 10px;
|
||||||
|
transition:
|
||||||
|
background 0.2s,
|
||||||
|
opacity 0.2s;
|
||||||
|
}
|
||||||
|
|
||||||
|
.item label {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
padding: 12px 14px;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.item input[type="checkbox"] {
|
||||||
|
width: 20px;
|
||||||
|
height: 20px;
|
||||||
|
accent-color: var(--done);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.item-text {
|
||||||
|
font-size: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.item.checked {
|
||||||
|
opacity: 0.55;
|
||||||
|
}
|
||||||
|
|
||||||
|
.item.checked .item-text {
|
||||||
|
text-decoration: line-through;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
.list-empty {
|
||||||
|
color: var(--muted);
|
||||||
|
font-size: 14px;
|
||||||
|
text-align: center;
|
||||||
|
padding: 32px 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.list-empty[hidden] {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* New-item flash so a voice-added item is easy to spot. */
|
||||||
|
.item-flash {
|
||||||
|
animation: item-flash 1s ease-out;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes item-flash {
|
||||||
|
0% {
|
||||||
|
background: var(--accent-soft);
|
||||||
|
border-color: var(--accent);
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
background: var(--panel);
|
||||||
|
border-color: var(--border);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Standard ``highlight`` command: a brief pulse. Duration is driven by
|
||||||
|
the server-supplied duration_ms via the --highlight-duration var. */
|
||||||
|
.ui-highlight {
|
||||||
|
animation: ui-highlight var(--highlight-duration, 1500ms) ease-out;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes ui-highlight {
|
||||||
|
0% {
|
||||||
|
transform: scale(1);
|
||||||
|
box-shadow: 0 0 0 0 var(--accent-soft);
|
||||||
|
}
|
||||||
|
30% {
|
||||||
|
transform: scale(1.02);
|
||||||
|
box-shadow: 0 0 0 6px var(--accent-soft);
|
||||||
|
border-color: var(--accent);
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
transform: scale(1);
|
||||||
|
box-shadow: 0 0 0 0 rgba(91, 140, 255, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
import { defineConfig } from "vite";
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
server: {
|
||||||
|
port: 5173,
|
||||||
|
},
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user