Add deixis UIWorker example
A ReplyToolMixin UIWorker that grounds in the user's text selection (the <selection> block in the snapshot) and points back via select_text — both directions of deictic reference.
This commit is contained in:
Mark Backman
101
examples/multi-worker/ui-worker/deixis/client/index.html
Normal file
101
examples/multi-worker/ui-worker/deixis/client/index.html
Normal file
@@ -0,0 +1,101 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Deixis — UIAgent demo</title>
|
||||
<link rel="stylesheet" href="./styles.css" />
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Reading room</h1>
|
||||
<button id="connect" type="button">Connect</button>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<article aria-label="Octopus cognition">
|
||||
<h2>What octopuses can teach us about minds</h2>
|
||||
<p class="lede">
|
||||
A short tour of the strangest cognitive machine on Earth.
|
||||
Select any paragraph and ask the assistant to explain it,
|
||||
rephrase it, or place it in context.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
An octopus has roughly five hundred million neurons, about
|
||||
the same as a dog. But the wiring is unlike any vertebrate
|
||||
brain. Two thirds of those neurons live in the arms, not in
|
||||
the central brain. Each arm runs a substantial amount of
|
||||
its own motor planning and sensory integration locally,
|
||||
which is why a severed octopus arm will continue to react
|
||||
to touch and even grab nearby food for several minutes.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
Their skin is its own information system. Underneath are
|
||||
millions of pigment cells called chromatophores, each one
|
||||
opened or closed by a tiny ring of muscles. Layered below
|
||||
those are iridophores and leucophores, which scatter or
|
||||
reflect light. Together they let an octopus reproduce the
|
||||
color and texture of a coral, a rock, or a sandy bottom in
|
||||
fractions of a second. The remarkable detail is that
|
||||
octopuses are mostly colorblind, and yet they match colors
|
||||
accurately. The leading hypothesis is that the skin itself
|
||||
has photoreceptors and senses light directly.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
Cephalopods edit their RNA at extraordinary rates. Most
|
||||
animals make small, occasional substitutions in messenger
|
||||
RNA before it is translated into protein. Octopuses,
|
||||
squids, and cuttlefish make tens of thousands of edits, and
|
||||
a substantial fraction occur in the genes that build
|
||||
neurons. This may be how they fine-tune neural function in
|
||||
response to temperature without slow generations of natural
|
||||
selection. The trade-off appears to be a much slower rate
|
||||
of underlying genetic evolution.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
They solve novel problems. Captive octopuses learn to open
|
||||
screw-top jars, navigate mazes, distinguish individual
|
||||
human caretakers, and remember which ones have been
|
||||
unkind. There is at least one careful study in which an
|
||||
octopus, given a transparent box containing food, opened
|
||||
it the long way around rather than through the obvious
|
||||
flap, suggesting some kind of mental simulation rather than
|
||||
pure trial and error.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
Their relationship to time is strange. Most octopus species
|
||||
live one or two years, breed once, and die soon after, in a
|
||||
process driven by hormonal signals from the optic gland. A
|
||||
single animal can therefore acquire a remarkable range of
|
||||
skills, only to lose them on a schedule. From a human
|
||||
standpoint this looks tragic. From an evolutionary
|
||||
standpoint it is simply the cost of investing everything in
|
||||
a brief, intense life.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
Studying octopuses tends to widen the definition of mind.
|
||||
They evolved their cognition independently of vertebrates,
|
||||
starting from a common ancestor more than five hundred
|
||||
million years ago, when the most sophisticated neural
|
||||
tissue on the planet was probably a nerve net. Whatever
|
||||
they do with their distributed brains, they arrived at it
|
||||
on a separate evolutionary line. The result is a working
|
||||
example of intelligence built on a different plan, which
|
||||
is exactly the kind of comparison that a single-example
|
||||
field of study most needs.
|
||||
</p>
|
||||
</article>
|
||||
</main>
|
||||
|
||||
<div id="status" aria-live="polite"></div>
|
||||
<audio id="bot-audio" autoplay data-a11y-exclude></audio>
|
||||
|
||||
<script type="module" src="./main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
185
examples/multi-worker/ui-worker/deixis/client/main.js
Normal file
185
examples/multi-worker/ui-worker/deixis/client/main.js
Normal file
@@ -0,0 +1,185 @@
|
||||
/**
|
||||
* Deixis — vanilla JS client.
|
||||
*
|
||||
* Same base wiring as pointing (PipecatClient +
|
||||
* managed snapshot streaming + bot audio sink), with three command
|
||||
* handlers: ``scroll_to``, ``highlight``, and ``select_text``.
|
||||
*
|
||||
* The interesting one is ``select_text``: it puts the OS-level text
|
||||
* selection on the referenced element, so when the agent says
|
||||
* "this paragraph here" the user sees exactly which paragraph it
|
||||
* means. The READ direction (user selection) flows the other way —
|
||||
* Managed snapshot streaming automatically captures
|
||||
* ``window.getSelection()`` and emits a ``<selection ref=...>...
|
||||
* </selection>`` block in the snapshot the server sees.
|
||||
*/
|
||||
|
||||
import {
|
||||
PipecatClient,
|
||||
RTVIEvent,
|
||||
findElementByRef,
|
||||
} from "@pipecat-ai/client-js";
|
||||
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";
|
||||
|
||||
const BOT_URL = "http://localhost:7860/api/offer";
|
||||
|
||||
const connectButton = document.getElementById("connect");
|
||||
const status = document.getElementById("status");
|
||||
const botAudio = document.getElementById("bot-audio");
|
||||
|
||||
let client;
|
||||
let unsubscribes = [];
|
||||
|
||||
function setStatus(text, autoHideMs = 0) {
|
||||
status.textContent = text;
|
||||
status.dataset.show = text ? "1" : "0";
|
||||
if (text && autoHideMs > 0) {
|
||||
setTimeout(() => {
|
||||
if (status.textContent === text) status.dataset.show = "0";
|
||||
}, autoHideMs);
|
||||
}
|
||||
}
|
||||
|
||||
function resolveTarget(payload) {
|
||||
if (payload?.ref) {
|
||||
const el = findElementByRef(payload.ref);
|
||||
if (el) return el;
|
||||
}
|
||||
if (payload?.target_id) {
|
||||
return document.getElementById(payload.target_id);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function handleScrollTo(payload) {
|
||||
const el = resolveTarget(payload);
|
||||
if (!el) return;
|
||||
const behavior =
|
||||
payload?.behavior === "instant" || payload?.behavior === "smooth"
|
||||
? payload.behavior
|
||||
: "smooth";
|
||||
el.scrollIntoView({ behavior, block: "center", inline: "nearest" });
|
||||
}
|
||||
|
||||
function handleHighlight(payload) {
|
||||
// Pointing's pulse style isn't appropriate for an article
|
||||
// (paragraphs don't want to scale). Use a brief background flash
|
||||
// by briefly setting a class. Highlight is used here mostly for
|
||||
// emphasizing single phrases the agent named — see CSS.
|
||||
const el = resolveTarget(payload);
|
||||
if (!el) return;
|
||||
el.classList.remove("flash");
|
||||
void el.offsetWidth;
|
||||
el.classList.add("flash");
|
||||
const duration = payload?.duration_ms ?? 1500;
|
||||
setTimeout(() => el.classList.remove("flash"), duration);
|
||||
}
|
||||
|
||||
/**
|
||||
* Programmatically select an element's text on the page.
|
||||
*
|
||||
* Build a ``Range`` covering the element's children, replace the
|
||||
* window selection with it, and scroll the element into view. This
|
||||
* is the WRITE side of the deixis story: the agent says "this
|
||||
* paragraph" and the page shows the text actually selected.
|
||||
*/
|
||||
function handleSelectText(payload) {
|
||||
const el = resolveTarget(payload);
|
||||
if (!el) return;
|
||||
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(el);
|
||||
|
||||
const sel = window.getSelection();
|
||||
if (!sel) return;
|
||||
sel.removeAllRanges();
|
||||
sel.addRange(range);
|
||||
|
||||
// Scroll the selection into view if it isn't already, so the user
|
||||
// actually sees the agent's pointer.
|
||||
el.scrollIntoView({ behavior: "smooth", block: "center" });
|
||||
}
|
||||
|
||||
function onUICommand(command, handler) {
|
||||
const listener = (data) => {
|
||||
if (data.command !== command) return;
|
||||
handler(data.payload);
|
||||
};
|
||||
client.on(RTVIEvent.UICommand, listener);
|
||||
return () => client.off(RTVIEvent.UICommand, listener);
|
||||
}
|
||||
|
||||
async function connect() {
|
||||
connectButton.disabled = true;
|
||||
setStatus("Connecting…");
|
||||
|
||||
client = new PipecatClient({
|
||||
transport: new SmallWebRTCTransport(),
|
||||
enableMic: true,
|
||||
enableCam: false,
|
||||
});
|
||||
|
||||
client.on(RTVIEvent.BotConnected, () => setStatus("Bot connected", 1500));
|
||||
client.on(RTVIEvent.Disconnected, () => {
|
||||
setStatus("Disconnected", 2000);
|
||||
connectButton.dataset.state = "";
|
||||
connectButton.textContent = "Connect";
|
||||
connectButton.disabled = false;
|
||||
teardownUI();
|
||||
});
|
||||
|
||||
client.on(RTVIEvent.TrackStarted, (track, participant) => {
|
||||
if (track.kind !== "audio") return;
|
||||
if (participant?.local) return;
|
||||
botAudio.srcObject = new MediaStream([track]);
|
||||
});
|
||||
|
||||
unsubscribes = [
|
||||
onUICommand("scroll_to", handleScrollTo),
|
||||
onUICommand("highlight", handleHighlight),
|
||||
onUICommand("select_text", handleSelectText),
|
||||
];
|
||||
|
||||
try {
|
||||
await client.connect({ webrtcUrl: BOT_URL });
|
||||
client.startUISnapshotStream();
|
||||
connectButton.dataset.state = "connected";
|
||||
connectButton.textContent = "Disconnect";
|
||||
connectButton.disabled = false;
|
||||
setStatus("Connected. Try selecting a paragraph and asking 'explain this.'", 5000);
|
||||
} catch (err) {
|
||||
console.error("Connect failed:", err);
|
||||
setStatus(`Connect failed: ${err.message ?? err}`, 4000);
|
||||
teardownUI();
|
||||
connectButton.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function disconnect() {
|
||||
connectButton.disabled = true;
|
||||
setStatus("Disconnecting…");
|
||||
try {
|
||||
await client?.disconnect();
|
||||
} finally {
|
||||
teardownUI();
|
||||
connectButton.dataset.state = "";
|
||||
connectButton.textContent = "Connect";
|
||||
connectButton.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
function teardownUI() {
|
||||
client?.stopUISnapshotStream();
|
||||
unsubscribes.forEach((unsubscribe) => unsubscribe());
|
||||
unsubscribes = [];
|
||||
if (botAudio.srcObject) botAudio.srcObject = null;
|
||||
client = undefined;
|
||||
}
|
||||
|
||||
connectButton.addEventListener("click", () => {
|
||||
if (connectButton.dataset.state === "connected") {
|
||||
disconnect();
|
||||
} else {
|
||||
connect();
|
||||
}
|
||||
});
|
||||
1128
examples/multi-worker/ui-worker/deixis/client/package-lock.json
generated
Normal file
1128
examples/multi-worker/ui-worker/deixis/client/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
18
examples/multi-worker/ui-worker/deixis/client/package.json
Normal file
18
examples/multi-worker/ui-worker/deixis/client/package.json
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"name": "deixis-client",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@pipecat-ai/client-js": "1.9.0",
|
||||
"@pipecat-ai/small-webrtc-transport": "^1.10.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vite": "^8"
|
||||
}
|
||||
}
|
||||
138
examples/multi-worker/ui-worker/deixis/client/styles.css
Normal file
138
examples/multi-worker/ui-worker/deixis/client/styles.css
Normal file
@@ -0,0 +1,138 @@
|
||||
:root {
|
||||
color-scheme: light;
|
||||
font-family:
|
||||
Charter,
|
||||
Georgia,
|
||||
"Iowan Old Style",
|
||||
serif;
|
||||
--border: #d4d4d8;
|
||||
--muted: #71717a;
|
||||
--selection: #fde68a;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
background: #fafafa;
|
||||
color: #18181b;
|
||||
}
|
||||
|
||||
header {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 10;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 1rem 1.5rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: #fff;
|
||||
font-family: system-ui, -apple-system, sans-serif;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
font-size: 1.125rem;
|
||||
margin: 0;
|
||||
letter-spacing: 0.01em;
|
||||
}
|
||||
|
||||
#connect {
|
||||
padding: 0.5rem 1rem;
|
||||
border: 1px solid var(--border);
|
||||
background: #fff;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
font-size: 0.875rem;
|
||||
font-family: system-ui, -apple-system, sans-serif;
|
||||
}
|
||||
|
||||
#connect:hover {
|
||||
background: #f4f4f5;
|
||||
}
|
||||
|
||||
#connect[data-state="connected"] {
|
||||
background: #ef4444;
|
||||
color: white;
|
||||
border-color: #ef4444;
|
||||
}
|
||||
|
||||
main {
|
||||
max-width: 720px;
|
||||
margin: 0 auto;
|
||||
padding: 2rem 1.5rem 4rem;
|
||||
}
|
||||
|
||||
article h2 {
|
||||
margin: 0 0 1rem;
|
||||
font-size: 1.875rem;
|
||||
line-height: 1.2;
|
||||
letter-spacing: -0.01em;
|
||||
}
|
||||
|
||||
article p {
|
||||
margin: 0 0 1.25rem;
|
||||
font-size: 1.125rem;
|
||||
line-height: 1.7;
|
||||
color: #27272a;
|
||||
}
|
||||
|
||||
article p.lede {
|
||||
font-size: 1rem;
|
||||
line-height: 1.6;
|
||||
color: var(--muted);
|
||||
font-style: italic;
|
||||
margin-bottom: 2rem;
|
||||
border-left: 3px solid var(--border);
|
||||
padding-left: 1rem;
|
||||
}
|
||||
|
||||
/* Make the user's text selection (and the agent's programmatic
|
||||
selection) visually distinct. The same color is used for both —
|
||||
the agent and the user are pointing at the same thing. */
|
||||
::selection {
|
||||
background: var(--selection);
|
||||
color: #18181b;
|
||||
}
|
||||
|
||||
/* Brief background flash when the agent calls ``highlight`` on a
|
||||
paragraph. Distinct from ``select_text`` (which uses the OS-level
|
||||
text selection) so the agent has two different visual idioms:
|
||||
"I'm pointing at this content" (select) vs "look at this fact"
|
||||
(flash). */
|
||||
@keyframes flash-fade {
|
||||
0% {
|
||||
background: var(--selection);
|
||||
}
|
||||
100% {
|
||||
background: transparent;
|
||||
}
|
||||
}
|
||||
|
||||
.flash {
|
||||
animation: flash-fade 1.5s ease-out;
|
||||
border-radius: 4px;
|
||||
margin-left: -0.25rem;
|
||||
padding-left: 0.25rem;
|
||||
}
|
||||
|
||||
#status {
|
||||
position: fixed;
|
||||
bottom: 1rem;
|
||||
right: 1rem;
|
||||
padding: 0.5rem 0.75rem;
|
||||
border-radius: 6px;
|
||||
font-size: 0.8125rem;
|
||||
font-family: system-ui, -apple-system, sans-serif;
|
||||
background: #18181b;
|
||||
color: white;
|
||||
opacity: 0;
|
||||
transition: opacity 0.2s;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
#status[data-show="1"] {
|
||||
opacity: 1;
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
import { defineConfig } from "vite";
|
||||
|
||||
export default defineConfig({
|
||||
server: {
|
||||
port: 5173,
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user