pipecat/examples/multi-worker/ui-worker/deixis

deixis

The UIWorker grounds in what the user just selected. Highlight a paragraph in the article and ask "explain this" — the worker reads your selection from the snapshot and answers about that specific content.

What it shows

• The read direction: the client captures window.getSelection() and emits a <selection ref="...">selected text</selection> block inside <ui_state>. The UIWorker treats it as the deictic referent for "this", "that", "this paragraph". Asking "what does this mean?" with a paragraph selected resolves cleanly.
• The write direction: the worker says "this paragraph" and issues a select_text=ref command. The client puts the page's text selection on that element, so the user sees exactly which paragraph the worker means.
• ReplyToolMixin covering reading-style apps: the same bundled tool pointing uses also has select_text (durable selection) alongside highlight (brief flash).

What it adds vs. pointing

pointing proved the worker can act visually on the page (scroll, highlight). This one proves it can read the user's pointer (text selection) and point back in the same idiom (programmatic selection). Same skeleton; the new parts are the select_text command and the matching client handler.

Run

Two terminals.

Terminal 1 — bot:

cd examples/multi-worker/ui-worker/deixis
uv run python bot.py

The bot starts on http://localhost:7860.

Terminal 2 — client:

cd examples/multi-worker/ui-worker/deixis/client
npm install            # one-time
npm run dev

Open http://localhost:5173 and click Connect.

What to try

The page renders a short essay on octopus cognition with selectable paragraphs.

Read direction (user selects, worker grounds):

• Select the paragraph about RNA editing → "What does this mean?"
• Select any paragraph → "Explain this in one sentence."

Write direction (worker points back):

• "Where does it talk about how octopuses solve problems?" (no selection) — the worker finds the paragraph, speaks a brief reply, and selects it for you.
• "How many neurons does an octopus have?" — answers and selects the source paragraph.

Conversational without pointing:

• "What's this article about?" — a one-sentence summary, no selection.

Requirements

• OPENAI_API_KEY
• DEEPGRAM_API_KEY
• CARTESIA_API_KEY

A .env in the example folder is the easiest way to set these (see examples/multi-worker/env.example).

What this example doesn't show

Form filling (see form-fill/), async task cards (see async-tasks/), or custom command handlers beyond scroll_to / highlight / select_text.