1c94feaaff
Move <ui_state> snapshot injection out of respond_with_llm into a cross-cutting on_before_process_frame handler on the UIWorker's LLM, so it appends the current snapshot to the context the request is built from, just before each inference. Injection is gated to the user-turn-initiating inference so a tool-calling turn never stacks duplicate <ui_state> blocks; respond_with_llm no longer injects manually. Also drop the bridged parameter from UIWorker: there is no viable way to bridge a UIWorker between workers — a shared, teed context would be polluted by the injection, and per-worker turn detection off teed frames isn't supported. Other workers keep their PipelineWorker bridging.
4 lines
1.2 KiB
Markdown
4 lines
1.2 KiB
Markdown
- Added `pipecat.workers.ui.UIWorker`, an `LLMContextWorker` that observes and drives a client GUI over the RTVI UI channel: it stores live accessibility snapshots, auto-injects `<ui_state>` into the LLM context before every inference (via the LLM's `on_before_process_frame` hook), dispatches client events to `@on_ui_event` handlers, and sends UI commands (`scroll_to`, `highlight`, `select_text`, `click`, `set_input_value`) back to the client. The optional `ReplyToolMixin` exposes a bundled `reply` tool, and `user_job_group(...)` surfaces fan-out work to the client as cancellable task cards. A native RTVI⇄bus UI bridge is built into `PipelineWorker` (active whenever RTVI is enabled), so no decorator or manual wiring is needed: inbound UI messages are broadcast on the bus as `BusUIEventMessage`, and outbound `BusUICommandMessage` / `BusUITask*` carriers are translated into RTVI frames for the client.
|
|
|
|
- `UIWorker` auto-injects the UI wire-format guide (`UI_STATE_PROMPT_GUIDE`) into its LLM's system instruction by default, via a `prompt_guide` parameter — pass your own string to override the guide, or `None` to disable. Apps no longer need to concatenate `UI_STATE_PROMPT_GUIDE` into the LLM's `system_instruction` by hand.
|