Add form-fill UIWorker example
A ReplyToolMixin UIWorker that fills inputs (fills) and toggles checkboxes / presses submit (click) by voice — the state-changing half of the standard action set.
This commit is contained in:
76
examples/multi-worker/ui-worker/form-fill/README.md
Normal file
76
examples/multi-worker/ui-worker/form-fill/README.md
Normal file
@@ -0,0 +1,76 @@
|
||||
# form-fill
|
||||
|
||||
The UIWorker fills form inputs and clicks buttons by voice. The page
|
||||
renders a job application with text fields, a textarea, checkboxes, and
|
||||
a submit button. Tell the worker your name, email, and the rest; when
|
||||
you're ready, say "submit."
|
||||
|
||||
## What it shows
|
||||
|
||||
- The **state-changing actions**: `set_input_value` for writing into
|
||||
inputs, `click` for checkboxes and submit. Both are bundled into the
|
||||
same `ReplyToolMixin` that pointing and deixis use — `fills` is a list
|
||||
of `{"ref", "value"}` so the LLM can fill several fields in one turn
|
||||
("my name is John Smith" fills first AND last name in one call), and
|
||||
`click` is a list so checkboxes and submit run in order.
|
||||
- That `FormWorker` is a one-line composition:
|
||||
`class FormWorker(ReplyToolMixin, UIWorker)`. Same shape as pointing
|
||||
and deixis; the visual fields (`highlight`, `select_text`) just stay
|
||||
`null` here, and the prompt steers the LLM toward `fills` / `click`.
|
||||
|
||||
## What it adds vs. `pointing` and `deixis`
|
||||
|
||||
Those exercise the visual / attention-pointing fields of `reply`. This
|
||||
one exercises the state-changing fields (`fills`, `click`). Same
|
||||
composition, same mixin — different fields per turn, driven by the
|
||||
prompt.
|
||||
|
||||
## Run
|
||||
|
||||
Two terminals.
|
||||
|
||||
**Terminal 1 — bot:**
|
||||
|
||||
```bash
|
||||
cd examples/multi-worker/ui-worker/form-fill
|
||||
uv run python bot.py
|
||||
```
|
||||
|
||||
The bot starts on `http://localhost:7860`.
|
||||
|
||||
**Terminal 2 — client:**
|
||||
|
||||
```bash
|
||||
cd examples/multi-worker/ui-worker/form-fill/client
|
||||
npm install # one-time
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Open `http://localhost:5173` and click **Connect**.
|
||||
|
||||
## What to try
|
||||
|
||||
- _"My name is John Smith."_ — fills first and last name in one call.
|
||||
- _"My email is john at gmail dot com."_ — converts the spoken form to
|
||||
`mark@daily.co` and fills the email field.
|
||||
- _"I have five years of experience and I love working on real-time
|
||||
voice agents."_ — fills two fields in one call.
|
||||
- _"Agree to the terms."_ — clicks the terms checkbox.
|
||||
- _"What have I entered so far?"_ — reads back current values from
|
||||
`<ui_state>` (no fills, no clicks).
|
||||
- _"Submit it."_ — clicks submit. If terms isn't ticked yet, the worker
|
||||
clicks both in order: terms, then submit.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `OPENAI_API_KEY`
|
||||
- `DEEPGRAM_API_KEY`
|
||||
- `CARTESIA_API_KEY`
|
||||
|
||||
A `.env` in the example folder is the easiest way to set these (see
|
||||
`examples/multi-worker/env.example`).
|
||||
|
||||
## What this example _doesn't_ show
|
||||
|
||||
Selection-based deixis (see `deixis/`) or async task cards (see
|
||||
`async-tasks/`).
|
||||
278
examples/multi-worker/ui-worker/form-fill/bot.py
Normal file
278
examples/multi-worker/ui-worker/form-fill/bot.py
Normal file
@@ -0,0 +1,278 @@
|
||||
#
|
||||
# Copyright (c) 2026, Daily
|
||||
#
|
||||
# SPDX-License-Identifier: BSD 2-Clause License
|
||||
#
|
||||
|
||||
"""Form-fill — the UIWorker changes form state by voice.
|
||||
|
||||
The page renders a job-application form. The user dictates field values
|
||||
("my name is John Smith", "email is john at gmail dot com"), checks
|
||||
boxes ("I agree to the terms"), and submits ("send it"). The UIWorker
|
||||
maps each spoken value to the right input ref and writes it.
|
||||
|
||||
Same skeleton as ``pointing`` / ``deixis``. ``FormWorker`` composes
|
||||
``ReplyToolMixin``: the ``reply(answer, scroll_to, fills, click)`` bundle
|
||||
covers the state-changing actions — ``fills`` writes input values
|
||||
(many at once), ``click`` toggles checkboxes and presses submit.
|
||||
|
||||
Architecture::
|
||||
|
||||
Main worker (PipelineWorker, owns transport + RTVI):
|
||||
transport.in → STT → user_agg → LLM → TTS → transport.out → assistant_agg
|
||||
└── answer_about_screen(query) tool
|
||||
└── params.pipeline_worker.job("ui", name="respond", payload={query})
|
||||
|
||||
FormWorker (ReplyToolMixin + UIWorker):
|
||||
└── inherited: reply(answer, scroll_to, fills, click)
|
||||
|
||||
Run::
|
||||
|
||||
uv run python bot.py
|
||||
|
||||
Then open the client at ``http://localhost:5173`` (see ``README.md``).
|
||||
|
||||
Requirements:
|
||||
|
||||
- OPENAI_API_KEY
|
||||
- DEEPGRAM_API_KEY
|
||||
- CARTESIA_API_KEY
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
|
||||
from pipecat.adapters.schemas.tools_schema import ToolsSchema
|
||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||
from pipecat.frames.frames import LLMRunFrame
|
||||
from pipecat.pipeline.job_context import JobError
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.pipeline.runner import PipelineRunner
|
||||
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
|
||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||
from pipecat.processors.aggregators.llm_response_universal import (
|
||||
LLMContextAggregatorPair,
|
||||
LLMUserAggregatorParams,
|
||||
)
|
||||
from pipecat.runner.types import RunnerArguments
|
||||
from pipecat.runner.utils import create_transport
|
||||
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||
from pipecat.services.deepgram.stt import DeepgramSTTService
|
||||
from pipecat.services.llm_service import FunctionCallParams
|
||||
from pipecat.services.openai.llm import OpenAILLMService
|
||||
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||
from pipecat.transports.daily.transport import DailyParams
|
||||
from pipecat.workers.ui import ReplyToolMixin, UIWorker
|
||||
|
||||
load_dotenv(override=True)
|
||||
|
||||
MAIN_NAME = "main"
|
||||
|
||||
transport_params = {
|
||||
"daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||
"webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||
}
|
||||
|
||||
|
||||
VOICE_PROMPT = """\
|
||||
You are the voice layer of a form-fill assistant. A separate UI \
|
||||
layer sees the form and writes the spoken reply.
|
||||
|
||||
For every user utterance involving the form (filling fields, \
|
||||
checking boxes, submitting), call ``answer_about_screen`` with the \
|
||||
user's request verbatim. The tool's response is the spoken reply, \
|
||||
already TTS-ready.
|
||||
|
||||
Only respond directly for pure pleasantries (greetings, thanks, \
|
||||
goodbyes). Keep direct replies to one short spoken sentence."""
|
||||
|
||||
|
||||
# The UI wire-format guide (UI_STATE_PROMPT_GUIDE) is appended to the LLM's
|
||||
# system instruction automatically by UIWorker, so this prompt only needs the
|
||||
# app-specific behavior.
|
||||
UI_PROMPT = """\
|
||||
You help the user fill out a job application form by voice. The \
|
||||
current ``<ui_state>`` block is in your context. Each input has a \
|
||||
ref (e.g. ``e5``) and a label. Use the labels to decide which input \
|
||||
gets which value.
|
||||
|
||||
## Tool: reply
|
||||
|
||||
Every turn calls ``reply`` exactly once. One tool call per turn.
|
||||
|
||||
``reply(answer, scroll_to=None, fills=None, click=None)``:
|
||||
|
||||
- ``answer`` (REQUIRED): a short spoken reply confirming what you \
|
||||
did or asking for missing info. One short sentence. Plain language.
|
||||
- ``scroll_to`` (OPTIONAL): a single snapshot ref. Use when a field \
|
||||
the user wants to see is tagged ``[offscreen]``.
|
||||
- ``fills`` (OPTIONAL): a list of ``{"ref": "eN", "value": "..."}`` \
|
||||
objects. Each entry writes ``value`` into the input at ``ref``. \
|
||||
You can fill many fields in one turn (e.g. first name + last name \
|
||||
+ email when the user says "my name is John Smith, mark at \
|
||||
daily dot co").
|
||||
- ``click`` (OPTIONAL): a list of refs to click. Use for \
|
||||
checkboxes (terms, newsletter) and the submit button. Order matters: \
|
||||
click checkboxes before submit.
|
||||
|
||||
## Decision rules
|
||||
|
||||
- **User dictates field values** → match each value to the input \
|
||||
whose label fits, set ``fills``, confirm in ``answer``.
|
||||
- **User says "check" / "agree" / "yes" for a checkbox** → resolve \
|
||||
the matching checkbox ref, set ``click=[ref]``.
|
||||
- **User says "submit" / "send it"** → confirm any required fields \
|
||||
are filled (especially the terms checkbox if needed), then \
|
||||
``click=[submit_ref]``. If terms isn't checked yet but the user said \
|
||||
submit, click both: ``click=[terms_ref, submit_ref]``.
|
||||
- **User asks "what have I entered?" / "what's left?"** → read the \
|
||||
current values from ``<ui_state>`` (the walker emits each input's \
|
||||
current value), summarize in ``answer``. No fills, no clicks.
|
||||
|
||||
## Spelling and disambiguation
|
||||
|
||||
When the user says something like "john at gmail dot com", convert \
|
||||
to ``mark@daily.co``. "five five five one two three four" → \
|
||||
``5551234``. "five years" → ``5``. Don't read these conversions \
|
||||
back to the user verbatim; just confirm naturally ("got it, your \
|
||||
email is mark@daily.co").
|
||||
|
||||
## Examples
|
||||
|
||||
(refs are illustrative; use the actual refs from the current \
|
||||
``<ui_state>``)
|
||||
|
||||
- "My name is John Smith." → \
|
||||
``reply(answer="Got it, John Smith.", fills=[{"ref":"e5","value":"Mark"}, {"ref":"e7","value":"Backman"}])``
|
||||
- "Email is john at gmail dot com." → \
|
||||
``reply(answer="Email saved.", fills=[{"ref":"e9","value":"mark@daily.co"}])``
|
||||
- "I have five years of experience and I love working on \
|
||||
real-time voice agents." → \
|
||||
``reply(answer="Five years and your interest noted.", fills=[{"ref":"e15","value":"5"}, {"ref":"e17","value":"I love working on real-time voice agents."}])``
|
||||
- "I agree to the terms." → \
|
||||
``reply(answer="Terms accepted.", click=["e22"])``
|
||||
- "Submit it." (terms not yet checked) → \
|
||||
``reply(answer="Submitting.", click=["e22","e26"])``
|
||||
- "What have I entered?" → \
|
||||
``reply(answer="John Smith, mark@daily.co, 5 years experience. The cover letter and terms aren't done yet.")``"""
|
||||
|
||||
|
||||
class FormWorker(ReplyToolMixin, UIWorker):
|
||||
"""UIWorker that fills form fields and toggles controls via ``reply``.
|
||||
|
||||
Composes ``ReplyToolMixin``, which exposes a single
|
||||
``reply(answer, scroll_to=None, fills=None, click=None, ...)`` LLM
|
||||
tool. ``fills`` writes values into inputs (many in one turn) and
|
||||
``click`` toggles checkboxes / presses submit — the state-changing
|
||||
half of the standard action set.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
llm = OpenAILLMService(
|
||||
api_key=os.environ["OPENAI_API_KEY"],
|
||||
settings=OpenAILLMService.Settings(system_instruction=UI_PROMPT),
|
||||
)
|
||||
super().__init__("ui", llm=llm)
|
||||
|
||||
|
||||
async def answer_about_screen(params: FunctionCallParams, query: str):
|
||||
"""Ask the screen-aware UI worker to fill the form / answer about it.
|
||||
|
||||
Args:
|
||||
query (str): The user's request, passed verbatim.
|
||||
"""
|
||||
logger.info(f"answer_about_screen('{query}')")
|
||||
try:
|
||||
async with params.pipeline_worker.job(
|
||||
"ui", name="respond", payload={"query": query}, timeout=10
|
||||
) as t:
|
||||
pass
|
||||
except JobError as e:
|
||||
logger.warning(f"ui job failed: {e}")
|
||||
await params.result_callback("Something went wrong on my side.")
|
||||
return
|
||||
|
||||
speak = (t.response or {}).get("speak")
|
||||
await params.result_callback(speak or "I'm not sure how to answer that.")
|
||||
|
||||
|
||||
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||
logger.info("Starting form-fill bot")
|
||||
|
||||
runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
|
||||
|
||||
stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
|
||||
tts = CartesiaTTSService(
|
||||
api_key=os.environ["CARTESIA_API_KEY"],
|
||||
settings=CartesiaTTSService.Settings(
|
||||
voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"),
|
||||
),
|
||||
)
|
||||
llm = OpenAILLMService(
|
||||
api_key=os.environ["OPENAI_API_KEY"],
|
||||
settings=OpenAILLMService.Settings(system_instruction=VOICE_PROMPT),
|
||||
)
|
||||
llm.register_direct_function(answer_about_screen, cancel_on_interruption=False, timeout_secs=30)
|
||||
|
||||
context = LLMContext(tools=ToolsSchema(standard_tools=[answer_about_screen]))
|
||||
aggregators = LLMContextAggregatorPair(
|
||||
context,
|
||||
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
|
||||
)
|
||||
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
transport.input(),
|
||||
stt,
|
||||
aggregators.user(),
|
||||
llm,
|
||||
tts,
|
||||
transport.output(),
|
||||
aggregators.assistant(),
|
||||
]
|
||||
)
|
||||
|
||||
worker = PipelineWorker(
|
||||
pipeline,
|
||||
name=MAIN_NAME,
|
||||
params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
|
||||
idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
|
||||
)
|
||||
|
||||
@transport.event_handler("on_client_connected")
|
||||
async def on_client_connected(transport, client):
|
||||
logger.info("Client connected")
|
||||
context.add_message(
|
||||
{
|
||||
"role": "developer",
|
||||
"content": (
|
||||
"Greet the user briefly. Tell them they can dictate field "
|
||||
"values and you'll fill them in. One short sentence."
|
||||
),
|
||||
}
|
||||
)
|
||||
await worker.queue_frame(LLMRunFrame())
|
||||
|
||||
@transport.event_handler("on_client_disconnected")
|
||||
async def on_client_disconnected(transport, client):
|
||||
logger.info("Client disconnected")
|
||||
await runner.cancel()
|
||||
|
||||
await runner.launch_worker(FormWorker())
|
||||
await runner.launch_worker(worker)
|
||||
|
||||
await runner.run()
|
||||
|
||||
|
||||
async def bot(runner_args: RunnerArguments):
|
||||
"""Main bot entry point compatible with Pipecat Cloud."""
|
||||
transport = await create_transport(runner_args, transport_params)
|
||||
await run_bot(transport, runner_args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from pipecat.runner.run import main
|
||||
|
||||
main()
|
||||
88
examples/multi-worker/ui-worker/form-fill/client/index.html
Normal file
88
examples/multi-worker/ui-worker/form-fill/client/index.html
Normal file
@@ -0,0 +1,88 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Form fill — UIAgent demo</title>
|
||||
<link rel="stylesheet" href="./styles.css" />
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Application form</h1>
|
||||
<button id="connect" type="button">Connect</button>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<form id="application-form" aria-label="Job application">
|
||||
<h2>Apply for: Software Engineer</h2>
|
||||
<p class="hint">
|
||||
Tell the assistant what to put in any field. Try: "my name
|
||||
is John Smith, my email is john at gmail dot com". When
|
||||
you're ready, say "submit".
|
||||
</p>
|
||||
|
||||
<fieldset>
|
||||
<legend>Your details</legend>
|
||||
|
||||
<div class="field">
|
||||
<label for="first-name">First name</label>
|
||||
<input id="first-name" name="first_name" type="text" autocomplete="given-name" />
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<label for="last-name">Last name</label>
|
||||
<input id="last-name" name="last_name" type="text" autocomplete="family-name" />
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<label for="email">Email address</label>
|
||||
<input id="email" name="email" type="email" autocomplete="email" />
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<label for="phone">Phone (optional)</label>
|
||||
<input id="phone" name="phone" type="tel" autocomplete="tel" />
|
||||
</div>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>About you</legend>
|
||||
|
||||
<div class="field">
|
||||
<label for="experience">Years of relevant experience</label>
|
||||
<input id="experience" name="experience" type="text" inputmode="numeric" />
|
||||
</div>
|
||||
|
||||
<div class="field">
|
||||
<label for="cover">Why are you interested?</label>
|
||||
<textarea id="cover" name="cover" rows="5"></textarea>
|
||||
</div>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Confirm</legend>
|
||||
|
||||
<div class="field checkbox-field">
|
||||
<input id="terms" name="terms" type="checkbox" />
|
||||
<label for="terms">I agree to the terms of service.</label>
|
||||
</div>
|
||||
|
||||
<div class="field checkbox-field">
|
||||
<input id="newsletter" name="newsletter" type="checkbox" />
|
||||
<label for="newsletter">Send me product updates.</label>
|
||||
</div>
|
||||
</fieldset>
|
||||
|
||||
<div class="actions">
|
||||
<button id="submit" type="submit">Submit application</button>
|
||||
<span id="form-status" aria-live="polite"></span>
|
||||
</div>
|
||||
</form>
|
||||
</main>
|
||||
|
||||
<div id="status" aria-live="polite"></div>
|
||||
<audio id="bot-audio" autoplay data-a11y-exclude></audio>
|
||||
|
||||
<script type="module" src="./main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
202
examples/multi-worker/ui-worker/form-fill/client/main.js
Normal file
202
examples/multi-worker/ui-worker/form-fill/client/main.js
Normal file
@@ -0,0 +1,202 @@
|
||||
/**
|
||||
* Form fill — vanilla JS client.
|
||||
*
|
||||
* Same base wiring as pointing/deixis (PipecatClient
|
||||
* + managed snapshot streaming + bot audio sink). Three command
|
||||
* handlers: ``scroll_to``, ``set_input_value``, and ``click``.
|
||||
*
|
||||
* ``set_input_value`` writes a string into an ``<input>`` /
|
||||
* ``<textarea>``. Crucially it dispatches ``input`` and ``change``
|
||||
* events so React-controlled or other frameworks pick up the change
|
||||
* naturally; the React standard handler does the same. ``click``
|
||||
* is the catch-all for checkboxes, radios, and submit buttons.
|
||||
*
|
||||
* The submit button intercepts the form's submit event so the demo
|
||||
* stays on-page after the agent submits. Real apps would let it
|
||||
* through.
|
||||
*/
|
||||
|
||||
import {
|
||||
PipecatClient,
|
||||
RTVIEvent,
|
||||
findElementByRef,
|
||||
} from "@pipecat-ai/client-js";
|
||||
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";
|
||||
|
||||
const BOT_URL = "http://localhost:7860/api/offer";
|
||||
|
||||
const connectButton = document.getElementById("connect");
|
||||
const status = document.getElementById("status");
|
||||
const botAudio = document.getElementById("bot-audio");
|
||||
const form = document.getElementById("application-form");
|
||||
const formStatus = document.getElementById("form-status");
|
||||
|
||||
let client;
|
||||
let unsubscribes = [];
|
||||
|
||||
function setStatus(text, autoHideMs = 0) {
|
||||
status.textContent = text;
|
||||
status.dataset.show = text ? "1" : "0";
|
||||
if (text && autoHideMs > 0) {
|
||||
setTimeout(() => {
|
||||
if (status.textContent === text) status.dataset.show = "0";
|
||||
}, autoHideMs);
|
||||
}
|
||||
}
|
||||
|
||||
function resolveTarget(payload) {
|
||||
if (payload?.ref) {
|
||||
const el = findElementByRef(payload.ref);
|
||||
if (el) return el;
|
||||
}
|
||||
if (payload?.target_id) {
|
||||
return document.getElementById(payload.target_id);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function handleScrollTo(payload) {
|
||||
const el = resolveTarget(payload);
|
||||
if (!el) return;
|
||||
const behavior =
|
||||
payload?.behavior === "instant" || payload?.behavior === "smooth"
|
||||
? payload.behavior
|
||||
: "smooth";
|
||||
el.scrollIntoView({ behavior, block: "center", inline: "nearest" });
|
||||
}
|
||||
|
||||
/**
|
||||
* Write ``payload.value`` into the targeted input/textarea.
|
||||
*
|
||||
* Skips ``disabled`` / ``readonly`` / ``type="hidden"`` targets so
|
||||
* the agent can't bypass UI affordances. Dispatches ``input`` and
|
||||
* ``change`` events so framework-controlled inputs (React, Vue, etc.)
|
||||
* notice the change. Briefly flashes the field so the user sees
|
||||
* what the agent wrote.
|
||||
*/
|
||||
function handleSetInputValue(payload) {
|
||||
const el = resolveTarget(payload);
|
||||
if (!el) return;
|
||||
if (!(el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement))
|
||||
return;
|
||||
if (el.disabled || el.readOnly) return;
|
||||
if (el.type === "hidden") return;
|
||||
|
||||
const value = String(payload?.value ?? "");
|
||||
const replace = payload?.replace !== false;
|
||||
el.value = replace ? value : (el.value || "") + value;
|
||||
|
||||
el.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
el.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
|
||||
// Visual confirmation: a brief background flash so the user
|
||||
// notices the write happened.
|
||||
el.classList.remove("fill-flash");
|
||||
void el.offsetWidth;
|
||||
el.classList.add("fill-flash");
|
||||
setTimeout(() => el.classList.remove("fill-flash"), 1200);
|
||||
}
|
||||
|
||||
/**
|
||||
* Click the targeted element. Skips ``disabled`` targets so the
|
||||
* agent can't bypass disabled affordances; the standard React
|
||||
* handler does the same.
|
||||
*/
|
||||
function handleClick(payload) {
|
||||
const el = resolveTarget(payload);
|
||||
if (!el) return;
|
||||
if ("disabled" in el && el.disabled) return;
|
||||
el.click();
|
||||
}
|
||||
|
||||
// Don't actually submit the form on the demo; the agent says "I
|
||||
// submitted it" and we show a status message instead.
|
||||
form.addEventListener("submit", (e) => {
|
||||
e.preventDefault();
|
||||
formStatus.textContent = "Submitted (demo only — no network call).";
|
||||
formStatus.style.color = "#16a34a";
|
||||
});
|
||||
|
||||
function onUICommand(command, handler) {
|
||||
const listener = (data) => {
|
||||
if (data.command !== command) return;
|
||||
handler(data.payload);
|
||||
};
|
||||
client.on(RTVIEvent.UICommand, listener);
|
||||
return () => client.off(RTVIEvent.UICommand, listener);
|
||||
}
|
||||
|
||||
async function connect() {
|
||||
connectButton.disabled = true;
|
||||
setStatus("Connecting…");
|
||||
|
||||
client = new PipecatClient({
|
||||
transport: new SmallWebRTCTransport(),
|
||||
enableMic: true,
|
||||
enableCam: false,
|
||||
});
|
||||
|
||||
client.on(RTVIEvent.BotConnected, () => setStatus("Bot connected", 1500));
|
||||
client.on(RTVIEvent.Disconnected, () => {
|
||||
setStatus("Disconnected", 2000);
|
||||
connectButton.dataset.state = "";
|
||||
connectButton.textContent = "Connect";
|
||||
connectButton.disabled = false;
|
||||
teardownUI();
|
||||
});
|
||||
|
||||
client.on(RTVIEvent.TrackStarted, (track, participant) => {
|
||||
if (track.kind !== "audio") return;
|
||||
if (participant?.local) return;
|
||||
botAudio.srcObject = new MediaStream([track]);
|
||||
});
|
||||
|
||||
unsubscribes = [
|
||||
onUICommand("scroll_to", handleScrollTo),
|
||||
onUICommand("set_input_value", handleSetInputValue),
|
||||
onUICommand("click", handleClick),
|
||||
];
|
||||
|
||||
try {
|
||||
await client.connect({ webrtcUrl: BOT_URL });
|
||||
client.startUISnapshotStream();
|
||||
connectButton.dataset.state = "connected";
|
||||
connectButton.textContent = "Disconnect";
|
||||
connectButton.disabled = false;
|
||||
setStatus("Connected. Try: 'My name is John Smith'", 5000);
|
||||
} catch (err) {
|
||||
console.error("Connect failed:", err);
|
||||
setStatus(`Connect failed: ${err.message ?? err}`, 4000);
|
||||
teardownUI();
|
||||
connectButton.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function disconnect() {
|
||||
connectButton.disabled = true;
|
||||
setStatus("Disconnecting…");
|
||||
try {
|
||||
await client?.disconnect();
|
||||
} finally {
|
||||
teardownUI();
|
||||
connectButton.dataset.state = "";
|
||||
connectButton.textContent = "Connect";
|
||||
connectButton.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
function teardownUI() {
|
||||
client?.stopUISnapshotStream();
|
||||
unsubscribes.forEach((unsubscribe) => unsubscribe());
|
||||
unsubscribes = [];
|
||||
if (botAudio.srcObject) botAudio.srcObject = null;
|
||||
client = undefined;
|
||||
}
|
||||
|
||||
connectButton.addEventListener("click", () => {
|
||||
if (connectButton.dataset.state === "connected") {
|
||||
disconnect();
|
||||
} else {
|
||||
connect();
|
||||
}
|
||||
});
|
||||
1128
examples/multi-worker/ui-worker/form-fill/client/package-lock.json
generated
Normal file
1128
examples/multi-worker/ui-worker/form-fill/client/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"name": "form-fill-client",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@pipecat-ai/client-js": "1.9.0",
|
||||
"@pipecat-ai/small-webrtc-transport": "^1.10.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vite": "^8"
|
||||
}
|
||||
}
|
||||
207
examples/multi-worker/ui-worker/form-fill/client/styles.css
Normal file
207
examples/multi-worker/ui-worker/form-fill/client/styles.css
Normal file
@@ -0,0 +1,207 @@
|
||||
:root {
|
||||
color-scheme: light;
|
||||
font-family: system-ui, -apple-system, sans-serif;
|
||||
--border: #d4d4d8;
|
||||
--border-focus: #3b82f6;
|
||||
--muted: #71717a;
|
||||
--bg: #fafafa;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--bg);
|
||||
color: #18181b;
|
||||
}
|
||||
|
||||
header {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 10;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 1rem 1.5rem;
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: #fff;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
font-size: 1.125rem;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
#connect {
|
||||
padding: 0.5rem 1rem;
|
||||
border: 1px solid var(--border);
|
||||
background: #fff;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
font-size: 0.875rem;
|
||||
}
|
||||
|
||||
#connect:hover {
|
||||
background: #f4f4f5;
|
||||
}
|
||||
|
||||
#connect[data-state="connected"] {
|
||||
background: #ef4444;
|
||||
color: white;
|
||||
border-color: #ef4444;
|
||||
}
|
||||
|
||||
main {
|
||||
max-width: 640px;
|
||||
margin: 0 auto;
|
||||
padding: 2rem 1.5rem 4rem;
|
||||
}
|
||||
|
||||
form h2 {
|
||||
margin: 0 0 0.5rem;
|
||||
font-size: 1.5rem;
|
||||
}
|
||||
|
||||
.hint {
|
||||
margin: 0 0 1.5rem;
|
||||
font-size: 0.9375rem;
|
||||
line-height: 1.5;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
fieldset {
|
||||
margin: 0 0 1.5rem;
|
||||
padding: 1rem 1.25rem 1.25rem;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
background: #fff;
|
||||
}
|
||||
|
||||
legend {
|
||||
padding: 0 0.5rem;
|
||||
font-size: 0.8125rem;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.04em;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.field {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0.25rem;
|
||||
margin-top: 0.875rem;
|
||||
}
|
||||
|
||||
.field:first-child {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
.field label {
|
||||
font-size: 0.875rem;
|
||||
font-weight: 500;
|
||||
color: #27272a;
|
||||
}
|
||||
|
||||
.field input[type="text"],
|
||||
.field input[type="email"],
|
||||
.field input[type="tel"],
|
||||
.field textarea {
|
||||
font: inherit;
|
||||
font-size: 1rem;
|
||||
padding: 0.5rem 0.625rem;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: #fff;
|
||||
width: 100%;
|
||||
scroll-margin-top: 5rem;
|
||||
transition:
|
||||
border-color 0.15s,
|
||||
box-shadow 0.15s,
|
||||
background 0.4s;
|
||||
}
|
||||
|
||||
.field input:focus,
|
||||
.field textarea:focus {
|
||||
outline: none;
|
||||
border-color: var(--border-focus);
|
||||
box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
|
||||
}
|
||||
|
||||
/* Brief flash when the agent fills a field, so the user notices the
|
||||
write. The CSS class is added by the set_input_value handler and
|
||||
removed after the animation. */
|
||||
@keyframes field-fill-flash {
|
||||
0% {
|
||||
background: #fef3c7;
|
||||
}
|
||||
100% {
|
||||
background: #fff;
|
||||
}
|
||||
}
|
||||
|
||||
.field input.fill-flash,
|
||||
.field textarea.fill-flash {
|
||||
animation: field-fill-flash 1.2s ease-out;
|
||||
}
|
||||
|
||||
.field textarea {
|
||||
resize: vertical;
|
||||
}
|
||||
|
||||
.checkbox-field {
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.checkbox-field label {
|
||||
font-weight: 400;
|
||||
}
|
||||
|
||||
.actions {
|
||||
display: flex;
|
||||
gap: 1rem;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
#submit {
|
||||
font: inherit;
|
||||
font-size: 0.9375rem;
|
||||
font-weight: 500;
|
||||
padding: 0.625rem 1.25rem;
|
||||
background: #18181b;
|
||||
color: white;
|
||||
border: 1px solid #18181b;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
#submit:hover {
|
||||
background: #27272a;
|
||||
}
|
||||
|
||||
#form-status {
|
||||
font-size: 0.875rem;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
#status {
|
||||
position: fixed;
|
||||
bottom: 1rem;
|
||||
right: 1rem;
|
||||
padding: 0.5rem 0.75rem;
|
||||
border-radius: 6px;
|
||||
font-size: 0.8125rem;
|
||||
background: #18181b;
|
||||
color: white;
|
||||
opacity: 0;
|
||||
transition: opacity 0.2s;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
#status[data-show="1"] {
|
||||
opacity: 1;
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
import { defineConfig } from "vite";
|
||||
|
||||
export default defineConfig({
|
||||
server: {
|
||||
port: 5173,
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user