Add form-fill UIWorker example
A ReplyToolMixin UIWorker that fills inputs (fills) and toggles checkboxes / presses submit (click) by voice — the state-changing half of the standard action set.
This commit is contained in:
76
examples/multi-worker/ui-worker/form-fill/README.md
Normal file
76
examples/multi-worker/ui-worker/form-fill/README.md
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
# form-fill
|
||||||
|
|
||||||
|
The UIWorker fills form inputs and clicks buttons by voice. The page
|
||||||
|
renders a job application with text fields, a textarea, checkboxes, and
|
||||||
|
a submit button. Tell the worker your name, email, and the rest; when
|
||||||
|
you're ready, say "submit."
|
||||||
|
|
||||||
|
## What it shows
|
||||||
|
|
||||||
|
- The **state-changing actions**: `set_input_value` for writing into
|
||||||
|
inputs, `click` for checkboxes and submit. Both are bundled into the
|
||||||
|
same `ReplyToolMixin` that pointing and deixis use — `fills` is a list
|
||||||
|
of `{"ref", "value"}` so the LLM can fill several fields in one turn
|
||||||
|
("my name is John Smith" fills first AND last name in one call), and
|
||||||
|
`click` is a list so checkboxes and submit run in order.
|
||||||
|
- That `FormWorker` is a one-line composition:
|
||||||
|
`class FormWorker(ReplyToolMixin, UIWorker)`. Same shape as pointing
|
||||||
|
and deixis; the visual fields (`highlight`, `select_text`) just stay
|
||||||
|
`null` here, and the prompt steers the LLM toward `fills` / `click`.
|
||||||
|
|
||||||
|
## What it adds vs. `pointing` and `deixis`
|
||||||
|
|
||||||
|
Those exercise the visual / attention-pointing fields of `reply`. This
|
||||||
|
one exercises the state-changing fields (`fills`, `click`). Same
|
||||||
|
composition, same mixin — different fields per turn, driven by the
|
||||||
|
prompt.
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
Two terminals.
|
||||||
|
|
||||||
|
**Terminal 1 — bot:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd examples/multi-worker/ui-worker/form-fill
|
||||||
|
uv run python bot.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The bot starts on `http://localhost:7860`.
|
||||||
|
|
||||||
|
**Terminal 2 — client:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd examples/multi-worker/ui-worker/form-fill/client
|
||||||
|
npm install # one-time
|
||||||
|
npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
Open `http://localhost:5173` and click **Connect**.
|
||||||
|
|
||||||
|
## What to try
|
||||||
|
|
||||||
|
- _"My name is John Smith."_ — fills first and last name in one call.
|
||||||
|
- _"My email is john at gmail dot com."_ — converts the spoken form to
|
||||||
|
`mark@daily.co` and fills the email field.
|
||||||
|
- _"I have five years of experience and I love working on real-time
|
||||||
|
voice agents."_ — fills two fields in one call.
|
||||||
|
- _"Agree to the terms."_ — clicks the terms checkbox.
|
||||||
|
- _"What have I entered so far?"_ — reads back current values from
|
||||||
|
`<ui_state>` (no fills, no clicks).
|
||||||
|
- _"Submit it."_ — clicks submit. If terms isn't ticked yet, the worker
|
||||||
|
clicks both in order: terms, then submit.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- `OPENAI_API_KEY`
|
||||||
|
- `DEEPGRAM_API_KEY`
|
||||||
|
- `CARTESIA_API_KEY`
|
||||||
|
|
||||||
|
A `.env` in the example folder is the easiest way to set these (see
|
||||||
|
`examples/multi-worker/env.example`).
|
||||||
|
|
||||||
|
## What this example _doesn't_ show
|
||||||
|
|
||||||
|
Selection-based deixis (see `deixis/`) or async task cards (see
|
||||||
|
`async-tasks/`).
|
||||||
278
examples/multi-worker/ui-worker/form-fill/bot.py
Normal file
278
examples/multi-worker/ui-worker/form-fill/bot.py
Normal file
@@ -0,0 +1,278 @@
|
|||||||
|
#
|
||||||
|
# Copyright (c) 2026, Daily
|
||||||
|
#
|
||||||
|
# SPDX-License-Identifier: BSD 2-Clause License
|
||||||
|
#
|
||||||
|
|
||||||
|
"""Form-fill — the UIWorker changes form state by voice.
|
||||||
|
|
||||||
|
The page renders a job-application form. The user dictates field values
|
||||||
|
("my name is John Smith", "email is john at gmail dot com"), checks
|
||||||
|
boxes ("I agree to the terms"), and submits ("send it"). The UIWorker
|
||||||
|
maps each spoken value to the right input ref and writes it.
|
||||||
|
|
||||||
|
Same skeleton as ``pointing`` / ``deixis``. ``FormWorker`` composes
|
||||||
|
``ReplyToolMixin``: the ``reply(answer, scroll_to, fills, click)`` bundle
|
||||||
|
covers the state-changing actions — ``fills`` writes input values
|
||||||
|
(many at once), ``click`` toggles checkboxes and presses submit.
|
||||||
|
|
||||||
|
Architecture::
|
||||||
|
|
||||||
|
Main worker (PipelineWorker, owns transport + RTVI):
|
||||||
|
transport.in → STT → user_agg → LLM → TTS → transport.out → assistant_agg
|
||||||
|
└── answer_about_screen(query) tool
|
||||||
|
└── params.pipeline_worker.job("ui", name="respond", payload={query})
|
||||||
|
|
||||||
|
FormWorker (ReplyToolMixin + UIWorker):
|
||||||
|
└── inherited: reply(answer, scroll_to, fills, click)
|
||||||
|
|
||||||
|
Run::
|
||||||
|
|
||||||
|
uv run python bot.py
|
||||||
|
|
||||||
|
Then open the client at ``http://localhost:5173`` (see ``README.md``).
|
||||||
|
|
||||||
|
Requirements:
|
||||||
|
|
||||||
|
- OPENAI_API_KEY
|
||||||
|
- DEEPGRAM_API_KEY
|
||||||
|
- CARTESIA_API_KEY
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
from pipecat.adapters.schemas.tools_schema import ToolsSchema
|
||||||
|
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||||
|
from pipecat.frames.frames import LLMRunFrame
|
||||||
|
from pipecat.pipeline.job_context import JobError
|
||||||
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
|
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
|
||||||
|
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||||
|
from pipecat.processors.aggregators.llm_response_universal import (
|
||||||
|
LLMContextAggregatorPair,
|
||||||
|
LLMUserAggregatorParams,
|
||||||
|
)
|
||||||
|
from pipecat.runner.types import RunnerArguments
|
||||||
|
from pipecat.runner.utils import create_transport
|
||||||
|
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||||
|
from pipecat.services.deepgram.stt import DeepgramSTTService
|
||||||
|
from pipecat.services.llm_service import FunctionCallParams
|
||||||
|
from pipecat.services.openai.llm import OpenAILLMService
|
||||||
|
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||||
|
from pipecat.transports.daily.transport import DailyParams
|
||||||
|
from pipecat.workers.ui import ReplyToolMixin, UIWorker
|
||||||
|
|
||||||
|
load_dotenv(override=True)
|
||||||
|
|
||||||
|
MAIN_NAME = "main"
|
||||||
|
|
||||||
|
transport_params = {
|
||||||
|
"daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||||
|
"webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
VOICE_PROMPT = """\
|
||||||
|
You are the voice layer of a form-fill assistant. A separate UI \
|
||||||
|
layer sees the form and writes the spoken reply.
|
||||||
|
|
||||||
|
For every user utterance involving the form (filling fields, \
|
||||||
|
checking boxes, submitting), call ``answer_about_screen`` with the \
|
||||||
|
user's request verbatim. The tool's response is the spoken reply, \
|
||||||
|
already TTS-ready.
|
||||||
|
|
||||||
|
Only respond directly for pure pleasantries (greetings, thanks, \
|
||||||
|
goodbyes). Keep direct replies to one short spoken sentence."""
|
||||||
|
|
||||||
|
|
||||||
|
# The UI wire-format guide (UI_STATE_PROMPT_GUIDE) is appended to the LLM's
|
||||||
|
# system instruction automatically by UIWorker, so this prompt only needs the
|
||||||
|
# app-specific behavior.
|
||||||
|
UI_PROMPT = """\
|
||||||
|
You help the user fill out a job application form by voice. The \
|
||||||
|
current ``<ui_state>`` block is in your context. Each input has a \
|
||||||
|
ref (e.g. ``e5``) and a label. Use the labels to decide which input \
|
||||||
|
gets which value.
|
||||||
|
|
||||||
|
## Tool: reply
|
||||||
|
|
||||||
|
Every turn calls ``reply`` exactly once. One tool call per turn.
|
||||||
|
|
||||||
|
``reply(answer, scroll_to=None, fills=None, click=None)``:
|
||||||
|
|
||||||
|
- ``answer`` (REQUIRED): a short spoken reply confirming what you \
|
||||||
|
did or asking for missing info. One short sentence. Plain language.
|
||||||
|
- ``scroll_to`` (OPTIONAL): a single snapshot ref. Use when a field \
|
||||||
|
the user wants to see is tagged ``[offscreen]``.
|
||||||
|
- ``fills`` (OPTIONAL): a list of ``{"ref": "eN", "value": "..."}`` \
|
||||||
|
objects. Each entry writes ``value`` into the input at ``ref``. \
|
||||||
|
You can fill many fields in one turn (e.g. first name + last name \
|
||||||
|
+ email when the user says "my name is John Smith, mark at \
|
||||||
|
daily dot co").
|
||||||
|
- ``click`` (OPTIONAL): a list of refs to click. Use for \
|
||||||
|
checkboxes (terms, newsletter) and the submit button. Order matters: \
|
||||||
|
click checkboxes before submit.
|
||||||
|
|
||||||
|
## Decision rules
|
||||||
|
|
||||||
|
- **User dictates field values** → match each value to the input \
|
||||||
|
whose label fits, set ``fills``, confirm in ``answer``.
|
||||||
|
- **User says "check" / "agree" / "yes" for a checkbox** → resolve \
|
||||||
|
the matching checkbox ref, set ``click=[ref]``.
|
||||||
|
- **User says "submit" / "send it"** → confirm any required fields \
|
||||||
|
are filled (especially the terms checkbox if needed), then \
|
||||||
|
``click=[submit_ref]``. If terms isn't checked yet but the user said \
|
||||||
|
submit, click both: ``click=[terms_ref, submit_ref]``.
|
||||||
|
- **User asks "what have I entered?" / "what's left?"** → read the \
|
||||||
|
current values from ``<ui_state>`` (the walker emits each input's \
|
||||||
|
current value), summarize in ``answer``. No fills, no clicks.
|
||||||
|
|
||||||
|
## Spelling and disambiguation
|
||||||
|
|
||||||
|
When the user says something like "john at gmail dot com", convert \
|
||||||
|
to ``mark@daily.co``. "five five five one two three four" → \
|
||||||
|
``5551234``. "five years" → ``5``. Don't read these conversions \
|
||||||
|
back to the user verbatim; just confirm naturally ("got it, your \
|
||||||
|
email is mark@daily.co").
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
(refs are illustrative; use the actual refs from the current \
|
||||||
|
``<ui_state>``)
|
||||||
|
|
||||||
|
- "My name is John Smith." → \
|
||||||
|
``reply(answer="Got it, John Smith.", fills=[{"ref":"e5","value":"Mark"}, {"ref":"e7","value":"Backman"}])``
|
||||||
|
- "Email is john at gmail dot com." → \
|
||||||
|
``reply(answer="Email saved.", fills=[{"ref":"e9","value":"mark@daily.co"}])``
|
||||||
|
- "I have five years of experience and I love working on \
|
||||||
|
real-time voice agents." → \
|
||||||
|
``reply(answer="Five years and your interest noted.", fills=[{"ref":"e15","value":"5"}, {"ref":"e17","value":"I love working on real-time voice agents."}])``
|
||||||
|
- "I agree to the terms." → \
|
||||||
|
``reply(answer="Terms accepted.", click=["e22"])``
|
||||||
|
- "Submit it." (terms not yet checked) → \
|
||||||
|
``reply(answer="Submitting.", click=["e22","e26"])``
|
||||||
|
- "What have I entered?" → \
|
||||||
|
``reply(answer="John Smith, mark@daily.co, 5 years experience. The cover letter and terms aren't done yet.")``"""
|
||||||
|
|
||||||
|
|
||||||
|
class FormWorker(ReplyToolMixin, UIWorker):
|
||||||
|
"""UIWorker that fills form fields and toggles controls via ``reply``.
|
||||||
|
|
||||||
|
Composes ``ReplyToolMixin``, which exposes a single
|
||||||
|
``reply(answer, scroll_to=None, fills=None, click=None, ...)`` LLM
|
||||||
|
tool. ``fills`` writes values into inputs (many in one turn) and
|
||||||
|
``click`` toggles checkboxes / presses submit — the state-changing
|
||||||
|
half of the standard action set.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
llm = OpenAILLMService(
|
||||||
|
api_key=os.environ["OPENAI_API_KEY"],
|
||||||
|
settings=OpenAILLMService.Settings(system_instruction=UI_PROMPT),
|
||||||
|
)
|
||||||
|
super().__init__("ui", llm=llm)
|
||||||
|
|
||||||
|
|
||||||
|
async def answer_about_screen(params: FunctionCallParams, query: str):
|
||||||
|
"""Ask the screen-aware UI worker to fill the form / answer about it.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query (str): The user's request, passed verbatim.
|
||||||
|
"""
|
||||||
|
logger.info(f"answer_about_screen('{query}')")
|
||||||
|
try:
|
||||||
|
async with params.pipeline_worker.job(
|
||||||
|
"ui", name="respond", payload={"query": query}, timeout=10
|
||||||
|
) as t:
|
||||||
|
pass
|
||||||
|
except JobError as e:
|
||||||
|
logger.warning(f"ui job failed: {e}")
|
||||||
|
await params.result_callback("Something went wrong on my side.")
|
||||||
|
return
|
||||||
|
|
||||||
|
speak = (t.response or {}).get("speak")
|
||||||
|
await params.result_callback(speak or "I'm not sure how to answer that.")
|
||||||
|
|
||||||
|
|
||||||
|
async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
|
||||||
|
logger.info("Starting form-fill bot")
|
||||||
|
|
||||||
|
runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
|
||||||
|
|
||||||
|
stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
|
||||||
|
tts = CartesiaTTSService(
|
||||||
|
api_key=os.environ["CARTESIA_API_KEY"],
|
||||||
|
settings=CartesiaTTSService.Settings(
|
||||||
|
voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
llm = OpenAILLMService(
|
||||||
|
api_key=os.environ["OPENAI_API_KEY"],
|
||||||
|
settings=OpenAILLMService.Settings(system_instruction=VOICE_PROMPT),
|
||||||
|
)
|
||||||
|
llm.register_direct_function(answer_about_screen, cancel_on_interruption=False, timeout_secs=30)
|
||||||
|
|
||||||
|
context = LLMContext(tools=ToolsSchema(standard_tools=[answer_about_screen]))
|
||||||
|
aggregators = LLMContextAggregatorPair(
|
||||||
|
context,
|
||||||
|
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
|
||||||
|
)
|
||||||
|
|
||||||
|
pipeline = Pipeline(
|
||||||
|
[
|
||||||
|
transport.input(),
|
||||||
|
stt,
|
||||||
|
aggregators.user(),
|
||||||
|
llm,
|
||||||
|
tts,
|
||||||
|
transport.output(),
|
||||||
|
aggregators.assistant(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
worker = PipelineWorker(
|
||||||
|
pipeline,
|
||||||
|
name=MAIN_NAME,
|
||||||
|
params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
|
||||||
|
idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
|
||||||
|
)
|
||||||
|
|
||||||
|
@transport.event_handler("on_client_connected")
|
||||||
|
async def on_client_connected(transport, client):
|
||||||
|
logger.info("Client connected")
|
||||||
|
context.add_message(
|
||||||
|
{
|
||||||
|
"role": "developer",
|
||||||
|
"content": (
|
||||||
|
"Greet the user briefly. Tell them they can dictate field "
|
||||||
|
"values and you'll fill them in. One short sentence."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
await worker.queue_frame(LLMRunFrame())
|
||||||
|
|
||||||
|
@transport.event_handler("on_client_disconnected")
|
||||||
|
async def on_client_disconnected(transport, client):
|
||||||
|
logger.info("Client disconnected")
|
||||||
|
await runner.cancel()
|
||||||
|
|
||||||
|
await runner.launch_worker(FormWorker())
|
||||||
|
await runner.launch_worker(worker)
|
||||||
|
|
||||||
|
await runner.run()
|
||||||
|
|
||||||
|
|
||||||
|
async def bot(runner_args: RunnerArguments):
|
||||||
|
"""Main bot entry point compatible with Pipecat Cloud."""
|
||||||
|
transport = await create_transport(runner_args, transport_params)
|
||||||
|
await run_bot(transport, runner_args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
from pipecat.runner.run import main
|
||||||
|
|
||||||
|
main()
|
||||||
88
examples/multi-worker/ui-worker/form-fill/client/index.html
Normal file
88
examples/multi-worker/ui-worker/form-fill/client/index.html
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<title>Form fill — UIAgent demo</title>
|
||||||
|
<link rel="stylesheet" href="./styles.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<h1>Application form</h1>
|
||||||
|
<button id="connect" type="button">Connect</button>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<form id="application-form" aria-label="Job application">
|
||||||
|
<h2>Apply for: Software Engineer</h2>
|
||||||
|
<p class="hint">
|
||||||
|
Tell the assistant what to put in any field. Try: "my name
|
||||||
|
is John Smith, my email is john at gmail dot com". When
|
||||||
|
you're ready, say "submit".
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<fieldset>
|
||||||
|
<legend>Your details</legend>
|
||||||
|
|
||||||
|
<div class="field">
|
||||||
|
<label for="first-name">First name</label>
|
||||||
|
<input id="first-name" name="first_name" type="text" autocomplete="given-name" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="field">
|
||||||
|
<label for="last-name">Last name</label>
|
||||||
|
<input id="last-name" name="last_name" type="text" autocomplete="family-name" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="field">
|
||||||
|
<label for="email">Email address</label>
|
||||||
|
<input id="email" name="email" type="email" autocomplete="email" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="field">
|
||||||
|
<label for="phone">Phone (optional)</label>
|
||||||
|
<input id="phone" name="phone" type="tel" autocomplete="tel" />
|
||||||
|
</div>
|
||||||
|
</fieldset>
|
||||||
|
|
||||||
|
<fieldset>
|
||||||
|
<legend>About you</legend>
|
||||||
|
|
||||||
|
<div class="field">
|
||||||
|
<label for="experience">Years of relevant experience</label>
|
||||||
|
<input id="experience" name="experience" type="text" inputmode="numeric" />
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="field">
|
||||||
|
<label for="cover">Why are you interested?</label>
|
||||||
|
<textarea id="cover" name="cover" rows="5"></textarea>
|
||||||
|
</div>
|
||||||
|
</fieldset>
|
||||||
|
|
||||||
|
<fieldset>
|
||||||
|
<legend>Confirm</legend>
|
||||||
|
|
||||||
|
<div class="field checkbox-field">
|
||||||
|
<input id="terms" name="terms" type="checkbox" />
|
||||||
|
<label for="terms">I agree to the terms of service.</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="field checkbox-field">
|
||||||
|
<input id="newsletter" name="newsletter" type="checkbox" />
|
||||||
|
<label for="newsletter">Send me product updates.</label>
|
||||||
|
</div>
|
||||||
|
</fieldset>
|
||||||
|
|
||||||
|
<div class="actions">
|
||||||
|
<button id="submit" type="submit">Submit application</button>
|
||||||
|
<span id="form-status" aria-live="polite"></span>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<div id="status" aria-live="polite"></div>
|
||||||
|
<audio id="bot-audio" autoplay data-a11y-exclude></audio>
|
||||||
|
|
||||||
|
<script type="module" src="./main.js"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
202
examples/multi-worker/ui-worker/form-fill/client/main.js
Normal file
202
examples/multi-worker/ui-worker/form-fill/client/main.js
Normal file
@@ -0,0 +1,202 @@
|
|||||||
|
/**
|
||||||
|
* Form fill — vanilla JS client.
|
||||||
|
*
|
||||||
|
* Same base wiring as pointing/deixis (PipecatClient
|
||||||
|
* + managed snapshot streaming + bot audio sink). Three command
|
||||||
|
* handlers: ``scroll_to``, ``set_input_value``, and ``click``.
|
||||||
|
*
|
||||||
|
* ``set_input_value`` writes a string into an ``<input>`` /
|
||||||
|
* ``<textarea>``. Crucially it dispatches ``input`` and ``change``
|
||||||
|
* events so React-controlled or other frameworks pick up the change
|
||||||
|
* naturally; the React standard handler does the same. ``click``
|
||||||
|
* is the catch-all for checkboxes, radios, and submit buttons.
|
||||||
|
*
|
||||||
|
* The submit button intercepts the form's submit event so the demo
|
||||||
|
* stays on-page after the agent submits. Real apps would let it
|
||||||
|
* through.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import {
|
||||||
|
PipecatClient,
|
||||||
|
RTVIEvent,
|
||||||
|
findElementByRef,
|
||||||
|
} from "@pipecat-ai/client-js";
|
||||||
|
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";
|
||||||
|
|
||||||
|
const BOT_URL = "http://localhost:7860/api/offer";
|
||||||
|
|
||||||
|
const connectButton = document.getElementById("connect");
|
||||||
|
const status = document.getElementById("status");
|
||||||
|
const botAudio = document.getElementById("bot-audio");
|
||||||
|
const form = document.getElementById("application-form");
|
||||||
|
const formStatus = document.getElementById("form-status");
|
||||||
|
|
||||||
|
let client;
|
||||||
|
let unsubscribes = [];
|
||||||
|
|
||||||
|
function setStatus(text, autoHideMs = 0) {
|
||||||
|
status.textContent = text;
|
||||||
|
status.dataset.show = text ? "1" : "0";
|
||||||
|
if (text && autoHideMs > 0) {
|
||||||
|
setTimeout(() => {
|
||||||
|
if (status.textContent === text) status.dataset.show = "0";
|
||||||
|
}, autoHideMs);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function resolveTarget(payload) {
|
||||||
|
if (payload?.ref) {
|
||||||
|
const el = findElementByRef(payload.ref);
|
||||||
|
if (el) return el;
|
||||||
|
}
|
||||||
|
if (payload?.target_id) {
|
||||||
|
return document.getElementById(payload.target_id);
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function handleScrollTo(payload) {
|
||||||
|
const el = resolveTarget(payload);
|
||||||
|
if (!el) return;
|
||||||
|
const behavior =
|
||||||
|
payload?.behavior === "instant" || payload?.behavior === "smooth"
|
||||||
|
? payload.behavior
|
||||||
|
: "smooth";
|
||||||
|
el.scrollIntoView({ behavior, block: "center", inline: "nearest" });
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Write ``payload.value`` into the targeted input/textarea.
|
||||||
|
*
|
||||||
|
* Skips ``disabled`` / ``readonly`` / ``type="hidden"`` targets so
|
||||||
|
* the agent can't bypass UI affordances. Dispatches ``input`` and
|
||||||
|
* ``change`` events so framework-controlled inputs (React, Vue, etc.)
|
||||||
|
* notice the change. Briefly flashes the field so the user sees
|
||||||
|
* what the agent wrote.
|
||||||
|
*/
|
||||||
|
function handleSetInputValue(payload) {
|
||||||
|
const el = resolveTarget(payload);
|
||||||
|
if (!el) return;
|
||||||
|
if (!(el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement))
|
||||||
|
return;
|
||||||
|
if (el.disabled || el.readOnly) return;
|
||||||
|
if (el.type === "hidden") return;
|
||||||
|
|
||||||
|
const value = String(payload?.value ?? "");
|
||||||
|
const replace = payload?.replace !== false;
|
||||||
|
el.value = replace ? value : (el.value || "") + value;
|
||||||
|
|
||||||
|
el.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
el.dispatchEvent(new Event("change", { bubbles: true }));
|
||||||
|
|
||||||
|
// Visual confirmation: a brief background flash so the user
|
||||||
|
// notices the write happened.
|
||||||
|
el.classList.remove("fill-flash");
|
||||||
|
void el.offsetWidth;
|
||||||
|
el.classList.add("fill-flash");
|
||||||
|
setTimeout(() => el.classList.remove("fill-flash"), 1200);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Click the targeted element. Skips ``disabled`` targets so the
|
||||||
|
* agent can't bypass disabled affordances; the standard React
|
||||||
|
* handler does the same.
|
||||||
|
*/
|
||||||
|
function handleClick(payload) {
|
||||||
|
const el = resolveTarget(payload);
|
||||||
|
if (!el) return;
|
||||||
|
if ("disabled" in el && el.disabled) return;
|
||||||
|
el.click();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Don't actually submit the form on the demo; the agent says "I
|
||||||
|
// submitted it" and we show a status message instead.
|
||||||
|
form.addEventListener("submit", (e) => {
|
||||||
|
e.preventDefault();
|
||||||
|
formStatus.textContent = "Submitted (demo only — no network call).";
|
||||||
|
formStatus.style.color = "#16a34a";
|
||||||
|
});
|
||||||
|
|
||||||
|
function onUICommand(command, handler) {
|
||||||
|
const listener = (data) => {
|
||||||
|
if (data.command !== command) return;
|
||||||
|
handler(data.payload);
|
||||||
|
};
|
||||||
|
client.on(RTVIEvent.UICommand, listener);
|
||||||
|
return () => client.off(RTVIEvent.UICommand, listener);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function connect() {
|
||||||
|
connectButton.disabled = true;
|
||||||
|
setStatus("Connecting…");
|
||||||
|
|
||||||
|
client = new PipecatClient({
|
||||||
|
transport: new SmallWebRTCTransport(),
|
||||||
|
enableMic: true,
|
||||||
|
enableCam: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
client.on(RTVIEvent.BotConnected, () => setStatus("Bot connected", 1500));
|
||||||
|
client.on(RTVIEvent.Disconnected, () => {
|
||||||
|
setStatus("Disconnected", 2000);
|
||||||
|
connectButton.dataset.state = "";
|
||||||
|
connectButton.textContent = "Connect";
|
||||||
|
connectButton.disabled = false;
|
||||||
|
teardownUI();
|
||||||
|
});
|
||||||
|
|
||||||
|
client.on(RTVIEvent.TrackStarted, (track, participant) => {
|
||||||
|
if (track.kind !== "audio") return;
|
||||||
|
if (participant?.local) return;
|
||||||
|
botAudio.srcObject = new MediaStream([track]);
|
||||||
|
});
|
||||||
|
|
||||||
|
unsubscribes = [
|
||||||
|
onUICommand("scroll_to", handleScrollTo),
|
||||||
|
onUICommand("set_input_value", handleSetInputValue),
|
||||||
|
onUICommand("click", handleClick),
|
||||||
|
];
|
||||||
|
|
||||||
|
try {
|
||||||
|
await client.connect({ webrtcUrl: BOT_URL });
|
||||||
|
client.startUISnapshotStream();
|
||||||
|
connectButton.dataset.state = "connected";
|
||||||
|
connectButton.textContent = "Disconnect";
|
||||||
|
connectButton.disabled = false;
|
||||||
|
setStatus("Connected. Try: 'My name is John Smith'", 5000);
|
||||||
|
} catch (err) {
|
||||||
|
console.error("Connect failed:", err);
|
||||||
|
setStatus(`Connect failed: ${err.message ?? err}`, 4000);
|
||||||
|
teardownUI();
|
||||||
|
connectButton.disabled = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function disconnect() {
|
||||||
|
connectButton.disabled = true;
|
||||||
|
setStatus("Disconnecting…");
|
||||||
|
try {
|
||||||
|
await client?.disconnect();
|
||||||
|
} finally {
|
||||||
|
teardownUI();
|
||||||
|
connectButton.dataset.state = "";
|
||||||
|
connectButton.textContent = "Connect";
|
||||||
|
connectButton.disabled = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function teardownUI() {
|
||||||
|
client?.stopUISnapshotStream();
|
||||||
|
unsubscribes.forEach((unsubscribe) => unsubscribe());
|
||||||
|
unsubscribes = [];
|
||||||
|
if (botAudio.srcObject) botAudio.srcObject = null;
|
||||||
|
client = undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
connectButton.addEventListener("click", () => {
|
||||||
|
if (connectButton.dataset.state === "connected") {
|
||||||
|
disconnect();
|
||||||
|
} else {
|
||||||
|
connect();
|
||||||
|
}
|
||||||
|
});
|
||||||
1128
examples/multi-worker/ui-worker/form-fill/client/package-lock.json
generated
Normal file
1128
examples/multi-worker/ui-worker/form-fill/client/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"name": "form-fill-client",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {
|
||||||
|
"dev": "vite",
|
||||||
|
"build": "vite build",
|
||||||
|
"preview": "vite preview"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@pipecat-ai/client-js": "1.9.0",
|
||||||
|
"@pipecat-ai/small-webrtc-transport": "^1.10.2"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"vite": "^8"
|
||||||
|
}
|
||||||
|
}
|
||||||
207
examples/multi-worker/ui-worker/form-fill/client/styles.css
Normal file
207
examples/multi-worker/ui-worker/form-fill/client/styles.css
Normal file
@@ -0,0 +1,207 @@
|
|||||||
|
:root {
|
||||||
|
color-scheme: light;
|
||||||
|
font-family: system-ui, -apple-system, sans-serif;
|
||||||
|
--border: #d4d4d8;
|
||||||
|
--border-focus: #3b82f6;
|
||||||
|
--muted: #71717a;
|
||||||
|
--bg: #fafafa;
|
||||||
|
}
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
background: var(--bg);
|
||||||
|
color: #18181b;
|
||||||
|
}
|
||||||
|
|
||||||
|
header {
|
||||||
|
position: sticky;
|
||||||
|
top: 0;
|
||||||
|
z-index: 10;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
padding: 1rem 1.5rem;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
background: #fff;
|
||||||
|
}
|
||||||
|
|
||||||
|
header h1 {
|
||||||
|
font-size: 1.125rem;
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
#connect {
|
||||||
|
padding: 0.5rem 1rem;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
background: #fff;
|
||||||
|
border-radius: 6px;
|
||||||
|
cursor: pointer;
|
||||||
|
font-size: 0.875rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
#connect:hover {
|
||||||
|
background: #f4f4f5;
|
||||||
|
}
|
||||||
|
|
||||||
|
#connect[data-state="connected"] {
|
||||||
|
background: #ef4444;
|
||||||
|
color: white;
|
||||||
|
border-color: #ef4444;
|
||||||
|
}
|
||||||
|
|
||||||
|
main {
|
||||||
|
max-width: 640px;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 2rem 1.5rem 4rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
form h2 {
|
||||||
|
margin: 0 0 0.5rem;
|
||||||
|
font-size: 1.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.hint {
|
||||||
|
margin: 0 0 1.5rem;
|
||||||
|
font-size: 0.9375rem;
|
||||||
|
line-height: 1.5;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
fieldset {
|
||||||
|
margin: 0 0 1.5rem;
|
||||||
|
padding: 1rem 1.25rem 1.25rem;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 8px;
|
||||||
|
background: #fff;
|
||||||
|
}
|
||||||
|
|
||||||
|
legend {
|
||||||
|
padding: 0 0.5rem;
|
||||||
|
font-size: 0.8125rem;
|
||||||
|
font-weight: 600;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.04em;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
.field {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 0.25rem;
|
||||||
|
margin-top: 0.875rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.field:first-child {
|
||||||
|
margin-top: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.field label {
|
||||||
|
font-size: 0.875rem;
|
||||||
|
font-weight: 500;
|
||||||
|
color: #27272a;
|
||||||
|
}
|
||||||
|
|
||||||
|
.field input[type="text"],
|
||||||
|
.field input[type="email"],
|
||||||
|
.field input[type="tel"],
|
||||||
|
.field textarea {
|
||||||
|
font: inherit;
|
||||||
|
font-size: 1rem;
|
||||||
|
padding: 0.5rem 0.625rem;
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 6px;
|
||||||
|
background: #fff;
|
||||||
|
width: 100%;
|
||||||
|
scroll-margin-top: 5rem;
|
||||||
|
transition:
|
||||||
|
border-color 0.15s,
|
||||||
|
box-shadow 0.15s,
|
||||||
|
background 0.4s;
|
||||||
|
}
|
||||||
|
|
||||||
|
.field input:focus,
|
||||||
|
.field textarea:focus {
|
||||||
|
outline: none;
|
||||||
|
border-color: var(--border-focus);
|
||||||
|
box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Brief flash when the agent fills a field, so the user notices the
|
||||||
|
write. The CSS class is added by the set_input_value handler and
|
||||||
|
removed after the animation. */
|
||||||
|
@keyframes field-fill-flash {
|
||||||
|
0% {
|
||||||
|
background: #fef3c7;
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
background: #fff;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.field input.fill-flash,
|
||||||
|
.field textarea.fill-flash {
|
||||||
|
animation: field-fill-flash 1.2s ease-out;
|
||||||
|
}
|
||||||
|
|
||||||
|
.field textarea {
|
||||||
|
resize: vertical;
|
||||||
|
}
|
||||||
|
|
||||||
|
.checkbox-field {
|
||||||
|
flex-direction: row;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.checkbox-field label {
|
||||||
|
font-weight: 400;
|
||||||
|
}
|
||||||
|
|
||||||
|
.actions {
|
||||||
|
display: flex;
|
||||||
|
gap: 1rem;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
#submit {
|
||||||
|
font: inherit;
|
||||||
|
font-size: 0.9375rem;
|
||||||
|
font-weight: 500;
|
||||||
|
padding: 0.625rem 1.25rem;
|
||||||
|
background: #18181b;
|
||||||
|
color: white;
|
||||||
|
border: 1px solid #18181b;
|
||||||
|
border-radius: 6px;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
#submit:hover {
|
||||||
|
background: #27272a;
|
||||||
|
}
|
||||||
|
|
||||||
|
#form-status {
|
||||||
|
font-size: 0.875rem;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
#status {
|
||||||
|
position: fixed;
|
||||||
|
bottom: 1rem;
|
||||||
|
right: 1rem;
|
||||||
|
padding: 0.5rem 0.75rem;
|
||||||
|
border-radius: 6px;
|
||||||
|
font-size: 0.8125rem;
|
||||||
|
background: #18181b;
|
||||||
|
color: white;
|
||||||
|
opacity: 0;
|
||||||
|
transition: opacity 0.2s;
|
||||||
|
pointer-events: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
#status[data-show="1"] {
|
||||||
|
opacity: 1;
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
import { defineConfig } from "vite";
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
server: {
|
||||||
|
port: 5173,
|
||||||
|
},
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user