f1f5a986e8
UIWorker is an LLMContextWorker that observes and drives a client GUI over the RTVI UI channel: it stores accessibility snapshots, auto-injects <ui_state> at the start of each respond job, dispatches client events to @on_ui_event handlers, sends UI commands back to the client, and surfaces fan-out work as cancellable task cards via user_job_group(). The optional ReplyToolMixin exposes a bundled reply tool. The prompt_guide parameter auto-appends the UI wire-format guide to the LLM's system instruction (default UI_STATE_PROMPT_GUIDE; override with a string or disable with None), so the LLM can parse the injected <ui_state> / <ui_event> messages without the app concatenating the guide by hand.
1.2 KiB
-
Added
pipecat.workers.ui.UIWorker, anLLMContextWorkerthat observes and drives a client GUI over the RTVI UI channel: it stores live accessibility snapshots, auto-injects<ui_state>at the start of eachrespondjob, dispatches client events to@on_ui_eventhandlers, and sends UI commands (scroll_to,highlight,select_text,click,set_input_value) back to the client. The optionalReplyToolMixinexposes a bundledreplytool, anduser_job_group(...)surfaces fan-out work to the client as cancellable task cards. A native RTVI⇄bus UI bridge is built intoPipelineWorker(active whenever RTVI is enabled), so no decorator or manual wiring is needed: inbound UI messages are broadcast on the bus asBusUIEventMessage, and outboundBusUICommandMessage/BusUITask*carriers are translated into RTVI frames for the client. -
UIWorkerauto-injects the UI wire-format guide (UI_STATE_PROMPT_GUIDE) into its LLM's system instruction by default, via aprompt_guideparameter — pass your own string to override the guide, orNoneto disable. Apps no longer need to concatenateUI_STATE_PROMPT_GUIDEinto the LLM'ssystem_instructionby hand.