pipecat/changelog/xxxx.added.md at 0b9500aae4a4fd181a5b93c22033713fa9e800bf

Files

Mark Backman 1c94feaaff Inject <ui_state> via the LLM's on_before_process_frame hook

Move <ui_state> snapshot injection out of respond_with_llm into a
cross-cutting on_before_process_frame handler on the UIWorker's LLM, so it
appends the current snapshot to the context the request is built from, just
before each inference. Injection is gated to the user-turn-initiating
inference so a tool-calling turn never stacks duplicate <ui_state> blocks;
respond_with_llm no longer injects manually.

Also drop the bridged parameter from UIWorker: there is no viable way to
bridge a UIWorker between workers — a shared, teed context would be polluted
by the injection, and per-worker turn detection off teed frames isn't
supported. Other workers keep their PipelineWorker bridging.

2026-05-21 23:20:40 -04:00

1.2 KiB

Raw Blame History

Added pipecat.workers.ui.UIWorker, an LLMContextWorker that observes and drives a client GUI over the RTVI UI channel: it stores live accessibility snapshots, auto-injects <ui_state> into the LLM context before every inference (via the LLM's on_before_process_frame hook), dispatches client events to @on_ui_event handlers, and sends UI commands (scroll_to, highlight, select_text, click, set_input_value) back to the client. The optional ReplyToolMixin exposes a bundled reply tool, and user_job_group(...) surfaces fan-out work to the client as cancellable task cards. A native RTVI⇄bus UI bridge is built into PipelineWorker (active whenever RTVI is enabled), so no decorator or manual wiring is needed: inbound UI messages are broadcast on the bus as BusUIEventMessage, and outbound BusUICommandMessage / BusUITask* carriers are translated into RTVI frames for the client.
UIWorker auto-injects the UI wire-format guide (UI_STATE_PROMPT_GUIDE) into its LLM's system instruction by default, via a prompt_guide parameter — pass your own string to override the guide, or None to disable. Apps no longer need to concatenate UI_STATE_PROMPT_GUIDE into the LLM's system_instruction by hand.

1.2 KiB Raw Blame History

1.2 KiB

Raw Blame History