2254a8d0a2
Smallest UIWorker demo: a voice LLM in the main pipeline delegates screen-relevant utterances to a UIWorker via a respond job; the UIWorker auto-injects the current <ui_state> and answers grounded in what's on screen. Includes a vanilla-JS client that streams accessibility snapshots over RTVI.
90 lines
2.7 KiB
Markdown
90 lines
2.7 KiB
Markdown
# hello-snapshot
|
|
|
|
The smallest possible `UIWorker` example. A static HTML page with a few
|
|
news cards and a sidebar. The user speaks; the worker answers grounded
|
|
in whatever's currently on screen.
|
|
|
|
## What it shows
|
|
|
|
- The accessibility-snapshot pipeline: the client walks the DOM and
|
|
streams a snapshot, which the `UIWorker` injects into its LLM context
|
|
as `<ui_state>`.
|
|
- The UIWorker delegate setup: the main pipeline's LLM (the
|
|
conversational layer) delegates every utterance to a `HelloWorker`
|
|
(`UIWorker`) via the `answer_about_screen` tool
|
|
(`params.pipeline_worker.job("hello", name="respond", ...)`) and speaks
|
|
the result.
|
|
- The native RTVI⇄bus UI bridge built into `PipelineWorker`: with
|
|
`enable_rtvi=True` (the default), inbound `ui-snapshot` messages are
|
|
broadcast on the bus and the `UIWorker` stores them — no decorator or
|
|
manual wiring.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Main worker (PipelineWorker, owns transport + RTVI):
|
|
transport.in → STT → user_agg → LLM → TTS → transport.out → assistant_agg
|
|
└── answer_about_screen(query) tool
|
|
└── params.pipeline_worker.job("hello", name="respond", payload={query})
|
|
|
|
HelloWorker (UIWorker):
|
|
└── @tool answer(text)
|
|
```
|
|
|
|
## Run
|
|
|
|
Two terminals.
|
|
|
|
**Terminal 1 — bot:**
|
|
|
|
```bash
|
|
cd examples/multi-worker/ui-worker/hello-snapshot
|
|
uv run python bot.py
|
|
```
|
|
|
|
The bot starts on `http://localhost:7860`.
|
|
|
|
**Terminal 2 — client:**
|
|
|
|
```bash
|
|
cd examples/multi-worker/ui-worker/hello-snapshot/client
|
|
npm install # one-time
|
|
npm run dev
|
|
```
|
|
|
|
Open `http://localhost:5173` and click **Connect**.
|
|
|
|
## What to try
|
|
|
|
Once connected, ask the worker:
|
|
|
|
- _"What's on this page?"_ — it summarizes the layout (heading, three
|
|
stories, trending tags sidebar).
|
|
- _"What was the second story about?"_ — sibling order in the snapshot
|
|
matches reading order, so "second" resolves cleanly.
|
|
- _"Which story was about energy?"_ — the worker grounds against the
|
|
actual content, not just titles.
|
|
- _"What tags are trending?"_ — exercises sidebar reading.
|
|
- _"What's the capital of France?"_ — the worker answers from general
|
|
knowledge when the question has nothing to do with the page.
|
|
|
|
If you scroll the page (in a smaller window) or resize, the snapshot
|
|
re-emits. Off-screen elements get an `[offscreen]` tag the worker
|
|
respects when answering positional questions like "what do I see right
|
|
now."
|
|
|
|
## Requirements
|
|
|
|
- `OPENAI_API_KEY`
|
|
- `DEEPGRAM_API_KEY`
|
|
- `CARTESIA_API_KEY`
|
|
|
|
A `.env` in the example folder is the easiest way to set these (see
|
|
`examples/multi-worker/env.example`).
|
|
|
|
## What this example _doesn't_ show
|
|
|
|
The read-side foundation only — no acting on the page (`scroll_to`,
|
|
`highlight`, ...), form filling, selection-based deixis, or async task
|
|
cards. Those build on this same skeleton.
|