Add hello-snapshot UIWorker example

Smallest UIWorker demo: a voice LLM in the main pipeline delegates
screen-relevant utterances to a UIWorker via a respond job; the UIWorker
auto-injects the current <ui_state> and answers grounded in what's on screen.
Includes a vanilla-JS client that streams accessibility snapshots over RTVI.
This commit is contained in:
Mark Backman
2026-05-21 16:51:38 -04:00
parent f1f5a986e8
commit 2254a8d0a2
8 changed files with 1828 additions and 0 deletions

View File

@@ -0,0 +1,89 @@
# hello-snapshot
The smallest possible `UIWorker` example. A static HTML page with a few
news cards and a sidebar. The user speaks; the worker answers grounded
in whatever's currently on screen.
## What it shows
- The accessibility-snapshot pipeline: the client walks the DOM and
streams a snapshot, which the `UIWorker` injects into its LLM context
as `<ui_state>`.
- The UIWorker delegate setup: the main pipeline's LLM (the
conversational layer) delegates every utterance to a `HelloWorker`
(`UIWorker`) via the `answer_about_screen` tool
(`params.pipeline_worker.job("hello", name="respond", ...)`) and speaks
the result.
- The native RTVI⇄bus UI bridge built into `PipelineWorker`: with
`enable_rtvi=True` (the default), inbound `ui-snapshot` messages are
broadcast on the bus and the `UIWorker` stores them — no decorator or
manual wiring.
## Architecture
```
Main worker (PipelineWorker, owns transport + RTVI):
transport.in → STT → user_agg → LLM → TTS → transport.out → assistant_agg
└── answer_about_screen(query) tool
└── params.pipeline_worker.job("hello", name="respond", payload={query})
HelloWorker (UIWorker):
└── @tool answer(text)
```
## Run
Two terminals.
**Terminal 1 — bot:**
```bash
cd examples/multi-worker/ui-worker/hello-snapshot
uv run python bot.py
```
The bot starts on `http://localhost:7860`.
**Terminal 2 — client:**
```bash
cd examples/multi-worker/ui-worker/hello-snapshot/client
npm install # one-time
npm run dev
```
Open `http://localhost:5173` and click **Connect**.
## What to try
Once connected, ask the worker:
- _"What's on this page?"_ — it summarizes the layout (heading, three
stories, trending tags sidebar).
- _"What was the second story about?"_ — sibling order in the snapshot
matches reading order, so "second" resolves cleanly.
- _"Which story was about energy?"_ — the worker grounds against the
actual content, not just titles.
- _"What tags are trending?"_ — exercises sidebar reading.
- _"What's the capital of France?"_ — the worker answers from general
knowledge when the question has nothing to do with the page.
If you scroll the page (in a smaller window) or resize, the snapshot
re-emits. Off-screen elements get an `[offscreen]` tag the worker
respects when answering positional questions like "what do I see right
now."
## Requirements
- `OPENAI_API_KEY`
- `DEEPGRAM_API_KEY`
- `CARTESIA_API_KEY`
A `.env` in the example folder is the easiest way to set these (see
`examples/multi-worker/env.example`).
## What this example _doesn't_ show
The read-side foundation only — no acting on the page (`scroll_to`,
`highlight`, ...), form filling, selection-based deixis, or async task
cards. Those build on this same skeleton.

View File

@@ -0,0 +1,274 @@
#
# Copyright (c) 2026, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""Hello UIWorker — the smallest possible accessibility-snapshot demo.
A voice bot whose LLM delegates every screen-relevant utterance to a
``UIWorker`` that sees the page and writes the spoken answer.
Architecture::
Main worker (PipelineWorker, owns transport + RTVI):
transport.in → STT → user_agg → LLM → TTS → transport.out → assistant_agg
└── answer_about_screen(query) tool
└── params.pipeline_worker.job("hello", name="respond", payload={query})
HelloWorker (UIWorker):
└── @tool answer(text)
The main LLM is the conversational layer: it forwards every utterance
to the UI worker via the ``answer_about_screen`` tool and speaks the
result. The UI worker's built-in ``respond`` job fires, which
auto-injects the latest ``<ui_state>`` block into its LLM context. The
UI worker's LLM picks the ``answer`` tool with a spoken reply grounded
in what's on screen.
The RTVI⇄bus UI bridge is built into ``PipelineWorker`` (active because
``enable_rtvi=True``), so inbound ``ui-snapshot`` messages from the
client are broadcast on the bus and the ``UIWorker`` stores them — no
decorator or manual wiring needed.
Why two LLMs for "hello world": this is the pattern UIWorker's
auto-inject is built for. The UI worker auto-injects the current screen
at the start of every delegated job, so the conversational LLM stays
small and screen-unaware. Later examples (pointing, form-fill, deixis,
async-tasks) compose new tools onto the same skeleton.
Run::
uv run python bot.py
Then open the client at ``http://localhost:5173`` (see ``README.md``).
Requirements:
- OPENAI_API_KEY
- DEEPGRAM_API_KEY
- CARTESIA_API_KEY
"""
import os
from dotenv import load_dotenv
from loguru import logger
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMRunFrame
from pipecat.pipeline.job_context import JobError
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.llm_service import FunctionCallParams
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.base_transport import BaseTransport, TransportParams
from pipecat.transports.daily.transport import DailyParams
from pipecat.workers.llm import tool
from pipecat.workers.ui import UIWorker
load_dotenv(override=True)
MAIN_NAME = "main"
transport_params = {
"daily": lambda: DailyParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
"webrtc": lambda: TransportParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
}
VOICE_PROMPT = """\
You are the voice layer of a screen-aware assistant. A separate UI \
layer sees the page the user is looking at and writes the spoken \
reply for any question that could plausibly involve the page.
## Routing rule
For every user utterance that could involve the page in any way — \
"what's on screen", "what does this say", "is X on the page", \
factual questions, navigational questions, anything where the page \
content might matter — call ``answer_about_screen`` with the user's \
request verbatim. The tool's response is the spoken reply, already \
TTS-ready; pass it through without paraphrasing.
If the request has nothing to do with the page, still call the \
tool — the UI layer falls back to general knowledge.
## When to answer directly
Only respond directly for pure pleasantries that don't need any \
content awareness:
- Greetings ("hi", "hello").
- Acknowledgements ("thanks", "got it").
- Goodbyes ("bye", "see you").
Keep direct replies to one short spoken sentence. No markdown, no \
lists, no symbols."""
# The UI wire-format guide (UI_STATE_PROMPT_GUIDE) is appended to the LLM's
# system instruction automatically by UIWorker, so this prompt only needs the
# app-specific behavior.
HELLO_PROMPT = """\
You answer the user's question grounded in the page they're looking \
at. The current ``<ui_state>`` block is in your context — use it for \
anything the user could be asking about on screen.
Always call exactly one tool: ``answer(text)``. Put the spoken reply \
in ``text``. Plain language, one or two short sentences, no markdown \
or symbols.
When the question is about something on the page, ground claims in \
the ``<ui_state>`` content. When it's general knowledge with no \
on-page referent (history, geography, definitions), answer from your \
own knowledge. Don't tell the user what you can't see — just answer \
or admit you don't know."""
class HelloWorker(UIWorker):
"""Snapshot-aware layer. Answers grounded in ``<ui_state>``.
``UIWorker`` defaults to ``active=True`` (unlike ``LLMWorker``)
because the canonical UIWorker role is an always-on delegate, so it
is online to receive snapshots and ``respond`` jobs as soon as its
pipeline starts.
"""
def __init__(self):
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAILLMService.Settings(system_instruction=HELLO_PROMPT),
)
super().__init__("hello", llm=llm)
@tool
async def answer(self, params: FunctionCallParams, text: str):
"""Speak ``text`` back to the user.
Args:
text: The spoken reply in plain language. One or two short
sentences. No markdown, no symbols, no lists.
"""
logger.info(f"{self}: answer('{text[:80]}')")
await self.respond_to_job(speak=text)
await params.result_callback(None)
async def answer_about_screen(params: FunctionCallParams, query: str):
"""Ask the screen-aware UI layer to answer about the current page.
Args:
query (str): The user's request, passed verbatim.
"""
logger.info(f"answer_about_screen('{query}')")
try:
async with params.pipeline_worker.job(
"hello", name="respond", payload={"query": query}, timeout=30
) as t:
pass
except JobError as e:
logger.warning(f"hello job failed: {e}")
await params.result_callback("Something went wrong on my side.")
return
speak = (t.response or {}).get("speak")
await params.result_callback(speak or "I'm not sure how to answer that.")
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
logger.info("Starting hello-snapshot bot")
runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
tts = CartesiaTTSService(
api_key=os.environ["CARTESIA_API_KEY"],
settings=CartesiaTTSService.Settings(
voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"),
),
)
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAILLMService.Settings(system_instruction=VOICE_PROMPT),
)
llm.register_direct_function(answer_about_screen, cancel_on_interruption=False, timeout_secs=60)
context = LLMContext(tools=ToolsSchema(standard_tools=[answer_about_screen]))
aggregators = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)
pipeline = Pipeline(
[
transport.input(),
stt,
aggregators.user(),
llm,
tts,
transport.output(),
aggregators.assistant(),
]
)
worker = PipelineWorker(
pipeline,
name=MAIN_NAME,
params=PipelineParams(
enable_metrics=True,
enable_usage_metrics=True,
),
idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
)
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
logger.info("Client connected")
context.add_message(
{
"role": "developer",
"content": (
"Greet the user briefly. Tell them they can ask about "
"anything on this page. One short sentence."
),
}
)
await worker.queue_frame(LLMRunFrame())
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
logger.info("Client disconnected")
await runner.cancel()
await runner.launch_worker(HelloWorker())
await runner.launch_worker(worker)
await runner.run()
async def bot(runner_args: RunnerArguments):
"""Main bot entry point compatible with Pipecat Cloud."""
transport = await create_transport(runner_args, transport_params)
await run_bot(transport, runner_args)
if __name__ == "__main__":
from pipecat.runner.run import main
main()

View File

@@ -0,0 +1,73 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Hello UIAgent</title>
<link rel="stylesheet" href="./styles.css" />
</head>
<body>
<header>
<h1>Local News Today</h1>
<button id="connect" type="button">Connect</button>
</header>
<main>
<section aria-label="Top stories">
<h2>Top stories</h2>
<article id="story-mediterranean" class="card">
<h3>Mediterranean diet linked to longer cognitive lifespan</h3>
<p>
A 12-year study following 50,000 adults across Italy and Spain
found that strict adherence to the Mediterranean diet
correlated with a 28% lower incidence of age-related cognitive
decline. Researchers attribute the effect to anti-inflammatory
compounds in olive oil and oily fish.
</p>
</article>
<article id="story-atacama" class="card">
<h3>Atacama solar field crosses 5 GW capacity</h3>
<p>
Chile's flagship desert solar project added another 600 MW of
generation this quarter, pushing total capacity past five
gigawatts. The expansion, completed three months ahead of
schedule, makes Atacama the largest solar generation site in
South America.
</p>
</article>
<article id="story-mars" class="card">
<h3>Mars sample return mission re-scoped to 2031</h3>
<p>
NASA and ESA jointly announced a revised timeline for the Mars
Sample Return mission, citing budget pressure and the need to
consolidate launch logistics. The first samples are now
expected back on Earth in late 2031, a two-year delay.
</p>
</article>
</section>
<aside aria-label="Trending tags">
<h2>Trending</h2>
<ul>
<li>climate</li>
<li>health</li>
<li>space</li>
<li>energy</li>
<li>longevity</li>
</ul>
</aside>
</main>
<div id="status" aria-live="polite"></div>
<!-- Bot audio sink. main.js attaches the bot's audio track to
this element on RTVIEvent.TrackStarted. Hidden via
data-a11y-exclude so it doesn't show up in the snapshot. -->
<audio id="bot-audio" autoplay data-a11y-exclude></audio>
<script type="module" src="./main.js"></script>
</body>
</html>

View File

@@ -0,0 +1,110 @@
/**
* Hello UIAgent — vanilla JS client.
*
* Wires three pieces of the SDK end to end:
* 1. PipecatClient + SmallWebRTCTransport for the voice session.
* 2. PipecatClient-managed accessibility snapshot streaming on
* every meaningful change (DOM mutations, focus, scroll-end,
* resize, visibility, selection).
*
* The agent has no tools — the snapshot is the entire input. The
* server's ``UIAgent`` auto-injects the latest ``<ui_state>`` block
* into the LLM context at the start of every turn, so the agent
* always answers grounded in what's currently on screen.
*/
import { PipecatClient, RTVIEvent } from "@pipecat-ai/client-js";
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";
const BOT_URL = "http://localhost:7860/api/offer";
const connectButton = document.getElementById("connect");
const status = document.getElementById("status");
const botAudio = document.getElementById("bot-audio");
let client;
function setStatus(text, autoHideMs = 0) {
status.textContent = text;
status.dataset.show = text ? "1" : "0";
if (text && autoHideMs > 0) {
setTimeout(() => {
if (status.textContent === text) status.dataset.show = "0";
}, autoHideMs);
}
}
async function connect() {
connectButton.disabled = true;
setStatus("Connecting…");
// 1. Construct the Pipecat client with the WebRTC transport.
client = new PipecatClient({
transport: new SmallWebRTCTransport(),
enableMic: true,
enableCam: false,
});
client.on(RTVIEvent.BotConnected, () => setStatus("Bot connected", 1500));
client.on(RTVIEvent.Disconnected, () => {
setStatus("Disconnected", 2000);
connectButton.dataset.state = "";
connectButton.textContent = "Connect";
connectButton.disabled = false;
teardownUI();
});
// Pipe the bot's audio track into the <audio> sink so the user
// hears it. Without this, the WebRTC track is alive but never
// routed to a playback element. The React kit ships
// `PipecatClientAudio` to do the same thing — this is the vanilla
// equivalent.
client.on(RTVIEvent.TrackStarted, (track, participant) => {
if (track.kind !== "audio") return;
if (participant?.local) return;
botAudio.srcObject = new MediaStream([track]);
});
// 3. Connect to the bot.
try {
await client.connect({ webrtcUrl: BOT_URL });
// 2. Start the managed snapshot stream once the RTVI transport is ready.
client.startUISnapshotStream();
connectButton.dataset.state = "connected";
connectButton.textContent = "Disconnect";
connectButton.disabled = false;
setStatus("Connected. Try asking the agent what's on screen.", 4000);
} catch (err) {
console.error("Connect failed:", err);
setStatus(`Connect failed: ${err.message ?? err}`, 4000);
teardownUI();
connectButton.disabled = false;
}
}
async function disconnect() {
connectButton.disabled = true;
setStatus("Disconnecting…");
try {
await client?.disconnect();
} finally {
teardownUI();
connectButton.dataset.state = "";
connectButton.textContent = "Connect";
connectButton.disabled = false;
}
}
function teardownUI() {
client?.stopUISnapshotStream();
if (botAudio.srcObject) botAudio.srcObject = null;
client = undefined;
}
connectButton.addEventListener("click", () => {
if (connectButton.dataset.state === "connected") {
disconnect();
} else {
connect();
}
});

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,18 @@
{
"name": "hello-snapshot-client",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview"
},
"dependencies": {
"@pipecat-ai/client-js": "1.9.0",
"@pipecat-ai/small-webrtc-transport": "^1.10.2"
},
"devDependencies": {
"vite": "^8"
}
}

View File

@@ -0,0 +1,129 @@
:root {
color-scheme: light;
font-family: system-ui, -apple-system, sans-serif;
--border: #d4d4d8;
--muted: #71717a;
}
* {
box-sizing: border-box;
}
body {
margin: 0;
background: #fafafa;
color: #18181b;
}
header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 1rem 1.5rem;
border-bottom: 1px solid var(--border);
background: #fff;
}
header h1 {
font-size: 1.25rem;
margin: 0;
}
#connect {
padding: 0.5rem 1rem;
border: 1px solid var(--border);
background: #fff;
border-radius: 6px;
cursor: pointer;
font-size: 0.875rem;
}
#connect:hover {
background: #f4f4f5;
}
#connect[data-state="connected"] {
background: #ef4444;
color: white;
border-color: #ef4444;
}
main {
display: grid;
grid-template-columns: 1fr 240px;
gap: 2rem;
padding: 1.5rem;
max-width: 1100px;
margin: 0 auto;
}
section,
aside {
display: flex;
flex-direction: column;
gap: 1rem;
}
section h2,
aside h2 {
font-size: 0.875rem;
text-transform: uppercase;
color: var(--muted);
letter-spacing: 0.04em;
margin: 0;
}
.card {
background: #fff;
border: 1px solid var(--border);
border-radius: 8px;
padding: 1rem;
}
.card h3 {
margin: 0 0 0.5rem;
font-size: 1rem;
}
.card p {
margin: 0;
font-size: 0.875rem;
line-height: 1.5;
color: #3f3f46;
}
aside ul {
list-style: none;
padding: 0;
margin: 0;
display: flex;
flex-direction: column;
gap: 0.5rem;
}
aside li {
font-size: 0.875rem;
padding: 0.375rem 0.75rem;
background: #fff;
border: 1px solid var(--border);
border-radius: 999px;
align-self: flex-start;
}
#status {
position: fixed;
bottom: 1rem;
right: 1rem;
padding: 0.5rem 0.75rem;
border-radius: 6px;
font-size: 0.8125rem;
background: #18181b;
color: white;
opacity: 0;
transition: opacity 0.2s;
pointer-events: none;
}
#status[data-show="1"] {
opacity: 1;
}

View File

@@ -0,0 +1,7 @@
import { defineConfig } from "vite";
export default defineConfig({
server: {
port: 5173,
},
});