feat(rtvi): add UI Agent Protocol as first-class RTVI message types

The UI Agent Protocol lets server-side AI agents observe and drive
a GUI app on the client side through structured RTVI messages.
Five new top-level RTVI types in kebab-case, in line with the rest
of the protocol:

  ui-event         client → server  (named event with payload)
  ui-command       server → client  (named command with payload)
  ui-snapshot      client → server  (accessibility tree of the page)
  ui-cancel-task   client → server  (cancel an in-flight task group)
  ui-task          server → client  (task lifecycle envelope)

Each ships paired ``*Data`` / ``*Message`` pydantic models in
``rtvi.models``, following the existing RTVI envelope convention
(``BotReady`` / ``BotReadyData``, ``Error`` / ``ErrorData``, etc.).
Built-in command payload models (``Toast``, ``Navigate``,
``ScrollTo``, ``Highlight``, ``Focus``, ``Click``, ``SetInputValue``,
``SelectText``) ship alongside; matching default React handlers
live in ``@pipecat-ai/client-react``.

Bumps the RTVI ``PROTOCOL_VERSION`` from ``1.2.0`` to ``1.3.0``.
Purely additive: only new top-level message types are introduced;
no existing wire shapes are changed. The major-version
compatibility check on ``client-ready`` still passes for older
1.x clients, so old clients continue to connect without warning;
they simply will not exercise the new types.

The ``RTVIProcessor`` registers a new ``on_ui_message`` event
handler that fires for inbound ``ui-event`` / ``ui-snapshot`` /
``ui-cancel-task`` with the parsed Message envelope, mirroring how
``on_client_message`` works for ``client-message``.

Five new pipeline frames let pipeline observers and processors see
UI traffic the same way they see other RTVI messages, mirroring
the frame-and-event pattern used by ``client-message``:

  RTVIUICommandFrame(command_name, payload)
    Pushed by downstream code (e.g. ``pipecat-ai-subagents``'s
    bridge) to send a UI command to the client. Wrapped by the
    observer into a ``UICommandMessage`` envelope.

  RTVIUITaskFrame(data: UITaskData)
    Same shape but for ``ui-task``; wrapped into ``UITaskMessage``.
    ``UITaskData`` is a discriminated union of the four lifecycle
    kinds (group_started / task_update / task_completed /
    group_completed).

  RTVIUIEventFrame(msg_id, event_name, payload)
  RTVIUISnapshotFrame(msg_id, tree)
  RTVIUICancelTaskFrame(msg_id, task_id, reason)
    Pushed by ``RTVIProcessor._handle_message`` whenever the
    matching inbound message arrives, alongside firing
    ``on_ui_message``. Pipeline observers and processors can match
    on the frame; subscribers like the subagents bridge keep using
    the event handler.

The data layer is the canonical authority for the wire format:
higher-level frameworks like ``pipecat-ai-subagents`` build the
agent abstractions on top, and single-LLM Pipecat apps can target
the same wire format directly via custom tools that emit these
typed messages.
This commit is contained in:
Mark Backman
2026-05-02 08:38:51 -04:00
parent 30efd11e15
commit 43abca0b06
7 changed files with 788 additions and 2 deletions

View File

@@ -32,7 +32,12 @@ from pipecat.frames.frames import (
SystemFrame,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.frameworks.rtvi.frames import RTVIClientMessageFrame
from pipecat.processors.frameworks.rtvi.frames import (
RTVIClientMessageFrame,
RTVIUICancelTaskFrame,
RTVIUIEventFrame,
RTVIUISnapshotFrame,
)
from pipecat.processors.frameworks.rtvi.observer import RTVIObserver, RTVIObserverParams
from pipecat.services.llm_service import (
FunctionCallParams, # TODO(aleix): we shouldn't import `services` from `processors`
@@ -76,6 +81,7 @@ class RTVIProcessor(FrameProcessor):
self._register_event_handler("on_bot_started")
self._register_event_handler("on_client_ready")
self._register_event_handler("on_client_message")
self._register_event_handler("on_ui_message")
self._input_transport = None
self._transport = transport
@@ -288,6 +294,41 @@ class RTVIProcessor(FrameProcessor):
case "client-message":
data = RTVI.RawClientMessageData.model_validate(message.data)
await self._handle_client_message(message.id, data)
case "ui-event":
event_data = RTVI.UIEventData.model_validate(message.data or {})
await self.push_frame(
RTVIUIEventFrame(
msg_id=message.id,
event_name=event_data.name,
payload=event_data.payload,
)
)
await self._call_event_handler(
"on_ui_message",
RTVI.UIEventMessage(id=message.id, data=event_data),
)
case "ui-snapshot":
snapshot_data = RTVI.UISnapshotData.model_validate(message.data or {})
await self.push_frame(
RTVIUISnapshotFrame(msg_id=message.id, tree=snapshot_data.tree)
)
await self._call_event_handler(
"on_ui_message",
RTVI.UISnapshotMessage(id=message.id, data=snapshot_data),
)
case "ui-cancel-task":
cancel_data = RTVI.UICancelTaskData.model_validate(message.data or {})
await self.push_frame(
RTVIUICancelTaskFrame(
msg_id=message.id,
task_id=cancel_data.task_id,
reason=cancel_data.reason,
)
)
await self._call_event_handler(
"on_ui_message",
RTVI.UICancelTaskMessage(id=message.id, data=cancel_data),
)
case "llm-function-call-result":
data = RTVI.LLMFunctionCallResultData.model_validate(message.data)
await self._handle_function_call_result(data)