pipecat/changelog/4407.added.md at d18fe7c39cbacc4ecbec73c0bb92bbc399e8e54a

Files

Mark Backman 43abca0b06 feat(rtvi): add UI Agent Protocol as first-class RTVI message types

The UI Agent Protocol lets server-side AI agents observe and drive
a GUI app on the client side through structured RTVI messages.
Five new top-level RTVI types in kebab-case, in line with the rest
of the protocol:

  ui-event         client → server  (named event with payload)
  ui-command       server → client  (named command with payload)
  ui-snapshot      client → server  (accessibility tree of the page)
  ui-cancel-task   client → server  (cancel an in-flight task group)
  ui-task          server → client  (task lifecycle envelope)

Each ships paired ``*Data`` / ``*Message`` pydantic models in
``rtvi.models``, following the existing RTVI envelope convention
(``BotReady`` / ``BotReadyData``, ``Error`` / ``ErrorData``, etc.).
Built-in command payload models (``Toast``, ``Navigate``,
``ScrollTo``, ``Highlight``, ``Focus``, ``Click``, ``SetInputValue``,
``SelectText``) ship alongside; matching default React handlers
live in ``@pipecat-ai/client-react``.

Bumps the RTVI ``PROTOCOL_VERSION`` from ``1.2.0`` to ``1.3.0``.
Purely additive: only new top-level message types are introduced;
no existing wire shapes are changed. The major-version
compatibility check on ``client-ready`` still passes for older
1.x clients, so old clients continue to connect without warning;
they simply will not exercise the new types.

The ``RTVIProcessor`` registers a new ``on_ui_message`` event
handler that fires for inbound ``ui-event`` / ``ui-snapshot`` /
``ui-cancel-task`` with the parsed Message envelope, mirroring how
``on_client_message`` works for ``client-message``.

Five new pipeline frames let pipeline observers and processors see
UI traffic the same way they see other RTVI messages, mirroring
the frame-and-event pattern used by ``client-message``:

  RTVIUICommandFrame(command_name, payload)
    Pushed by downstream code (e.g. ``pipecat-ai-subagents``'s
    bridge) to send a UI command to the client. Wrapped by the
    observer into a ``UICommandMessage`` envelope.

  RTVIUITaskFrame(data: UITaskData)
    Same shape but for ``ui-task``; wrapped into ``UITaskMessage``.
    ``UITaskData`` is a discriminated union of the four lifecycle
    kinds (group_started / task_update / task_completed /
    group_completed).

  RTVIUIEventFrame(msg_id, event_name, payload)
  RTVIUISnapshotFrame(msg_id, tree)
  RTVIUICancelTaskFrame(msg_id, task_id, reason)
    Pushed by ``RTVIProcessor._handle_message`` whenever the
    matching inbound message arrives, alongside firing
    ``on_ui_message``. Pipeline observers and processors can match
    on the frame; subscribers like the subagents bridge keep using
    the event handler.

The data layer is the canonical authority for the wire format:
higher-level frameworks like ``pipecat-ai-subagents`` build the
agent abstractions on top, and single-LLM Pipecat apps can target
the same wire format directly via custom tools that emit these
typed messages.

2026-05-02 12:09:01 -04:00

1.3 KiB

Raw Blame History

Added the UI Agent Protocol to pipecat.processors.frameworks.rtvi.models:
- Five first-class RTVI message types carry the wire format: ui-event (client → server), ui-command (server → client), ui-snapshot (client → server, accessibility tree), ui-cancel-task (client → server), and ui-task (server → client task lifecycle envelopes). Each ships a paired *Data / *Message pydantic model following the existing RTVI convention.
- Built-in command payload models (Toast, Navigate, ScrollTo, Highlight, Focus, Click, SetInputValue, SelectText) ship alongside; matching default handlers live in @pipecat-ai/client-react.
- The RTVIProcessor registers a new on_ui_message event handler that fires for inbound ui-event / ui-snapshot / ui-cancel-task.
- Five new pipeline frames let pipeline observers and processors see UI traffic the same way they see other RTVI messages: RTVIUICommandFrame and RTVIUITaskFrame are pushed by downstream code to be wrapped by the observer into outbound UICommandMessage / UITaskMessage envelopes; RTVIUIEventFrame, RTVIUISnapshotFrame, and RTVIUICancelTaskFrame are pushed by the processor on inbound, alongside firing on_ui_message, mirroring the frame-and-event pattern used by client-message.
- Bumps the RTVI PROTOCOL_VERSION from 1.2.0 to 1.3.0.

1.3 KiB Raw Blame History

1.3 KiB

Raw Blame History