Add form-fill UIWorker example

A ReplyToolMixin UIWorker that fills inputs (fills) and toggles checkboxes /
presses submit (click) by voice — the state-changing half of the standard
action set.
This commit is contained in:
Mark Backman
2026-05-21 17:21:08 -04:00
parent f826da9ac9
commit 6b0e204d66
8 changed files with 2004 additions and 0 deletions

View File

@@ -0,0 +1,76 @@
# form-fill
The UIWorker fills form inputs and clicks buttons by voice. The page
renders a job application with text fields, a textarea, checkboxes, and
a submit button. Tell the worker your name, email, and the rest; when
you're ready, say "submit."
## What it shows
- The **state-changing actions**: `set_input_value` for writing into
inputs, `click` for checkboxes and submit. Both are bundled into the
same `ReplyToolMixin` that pointing and deixis use — `fills` is a list
of `{"ref", "value"}` so the LLM can fill several fields in one turn
("my name is John Smith" fills first AND last name in one call), and
`click` is a list so checkboxes and submit run in order.
- That `FormWorker` is a one-line composition:
`class FormWorker(ReplyToolMixin, UIWorker)`. Same shape as pointing
and deixis; the visual fields (`highlight`, `select_text`) just stay
`null` here, and the prompt steers the LLM toward `fills` / `click`.
## What it adds vs. `pointing` and `deixis`
Those exercise the visual / attention-pointing fields of `reply`. This
one exercises the state-changing fields (`fills`, `click`). Same
composition, same mixin — different fields per turn, driven by the
prompt.
## Run
Two terminals.
**Terminal 1 — bot:**
```bash
cd examples/multi-worker/ui-worker/form-fill
uv run python bot.py
```
The bot starts on `http://localhost:7860`.
**Terminal 2 — client:**
```bash
cd examples/multi-worker/ui-worker/form-fill/client
npm install # one-time
npm run dev
```
Open `http://localhost:5173` and click **Connect**.
## What to try
- _"My name is John Smith."_ — fills first and last name in one call.
- _"My email is john at gmail dot com."_ — converts the spoken form to
`mark@daily.co` and fills the email field.
- _"I have five years of experience and I love working on real-time
voice agents."_ — fills two fields in one call.
- _"Agree to the terms."_ — clicks the terms checkbox.
- _"What have I entered so far?"_ — reads back current values from
`<ui_state>` (no fills, no clicks).
- _"Submit it."_ — clicks submit. If terms isn't ticked yet, the worker
clicks both in order: terms, then submit.
## Requirements
- `OPENAI_API_KEY`
- `DEEPGRAM_API_KEY`
- `CARTESIA_API_KEY`
A `.env` in the example folder is the easiest way to set these (see
`examples/multi-worker/env.example`).
## What this example _doesn't_ show
Selection-based deixis (see `deixis/`) or async task cards (see
`async-tasks/`).