feat(rtvi): add UI Agent Protocol as first-class RTVI message types

The UI Agent Protocol lets server-side AI agents observe and drive
a GUI app on the client side through structured RTVI messages.
Five new top-level RTVI types in kebab-case, in line with the rest
of the protocol:

  ui-event         client → server  (named event with payload)
  ui-command       server → client  (named command with payload)
  ui-snapshot      client → server  (accessibility tree of the page)
  ui-cancel-task   client → server  (cancel an in-flight task group)
  ui-task          server → client  (task lifecycle envelope)

Each ships paired ``*Data`` / ``*Message`` pydantic models in
``rtvi.models``, following the existing RTVI envelope convention
(``BotReady`` / ``BotReadyData``, ``Error`` / ``ErrorData``, etc.).
Built-in command payload models (``Toast``, ``Navigate``,
``ScrollTo``, ``Highlight``, ``Focus``, ``Click``, ``SetInputValue``,
``SelectText``) ship alongside; matching default React handlers
live in ``@pipecat-ai/client-react``.

Bumps the RTVI ``PROTOCOL_VERSION`` from ``1.2.0`` to ``1.3.0``.
Purely additive: only new top-level message types are introduced;
no existing wire shapes are changed. The major-version
compatibility check on ``client-ready`` still passes for older
1.x clients, so old clients continue to connect without warning;
they simply will not exercise the new types.

The ``RTVIProcessor`` registers a new ``on_ui_message`` event
handler that fires for inbound ``ui-event`` / ``ui-snapshot`` /
``ui-cancel-task`` with the parsed Message envelope, mirroring how
``on_client_message`` works for ``client-message``.

Five new pipeline frames let pipeline observers and processors see
UI traffic the same way they see other RTVI messages, mirroring
the frame-and-event pattern used by ``client-message``:

  RTVIUICommandFrame(command_name, payload)
    Pushed by downstream code (e.g. ``pipecat-ai-subagents``'s
    bridge) to send a UI command to the client. Wrapped by the
    observer into a ``UICommandMessage`` envelope.

  RTVIUITaskFrame(data: UITaskData)
    Same shape but for ``ui-task``; wrapped into ``UITaskMessage``.
    ``UITaskData`` is a discriminated union of the four lifecycle
    kinds (group_started / task_update / task_completed /
    group_completed).

  RTVIUIEventFrame(msg_id, event_name, payload)
  RTVIUISnapshotFrame(msg_id, tree)
  RTVIUICancelTaskFrame(msg_id, task_id, reason)
    Pushed by ``RTVIProcessor._handle_message`` whenever the
    matching inbound message arrives, alongside firing
    ``on_ui_message``. Pipeline observers and processors can match
    on the frame; subscribers like the subagents bridge keep using
    the event handler.

The data layer is the canonical authority for the wire format:
higher-level frameworks like ``pipecat-ai-subagents`` build the
agent abstractions on top, and single-LLM Pipecat apps can target
the same wire format directly via custom tools that emit these
typed messages.

changelog/4407.added.md

@@ -0,0 +1,6 @@
+- Added the UI Agent Protocol to `pipecat.processors.frameworks.rtvi.models`:
+  - Five first-class RTVI message types carry the wire format: `ui-event` (client → server), `ui-command` (server → client), `ui-snapshot` (client → server, accessibility tree), `ui-cancel-task` (client → server), and `ui-task` (server → client task lifecycle envelopes). Each ships a paired `*Data` / `*Message` pydantic model following the existing RTVI convention.
+  - Built-in command payload models (`Toast`, `Navigate`, `ScrollTo`, `Highlight`, `Focus`, `Click`, `SetInputValue`, `SelectText`) ship alongside; matching default handlers live in `@pipecat-ai/client-react`.
+  - The `RTVIProcessor` registers a new `on_ui_message` event handler that fires for inbound `ui-event` / `ui-snapshot` / `ui-cancel-task`.
+  - Five new pipeline frames let pipeline observers and processors see UI traffic the same way they see other RTVI messages: `RTVIUICommandFrame` and `RTVIUITaskFrame` are pushed by downstream code to be wrapped by the observer into outbound `UICommandMessage` / `UITaskMessage` envelopes; `RTVIUIEventFrame`, `RTVIUISnapshotFrame`, and `RTVIUICancelTaskFrame` are pushed by the processor on inbound, alongside firing `on_ui_message`, mirroring the frame-and-event pattern used by `client-message`.
+  - Bumps the RTVI `PROTOCOL_VERSION` from `1.2.0` to `1.3.0`.

src/pipecat/processors/frameworks/rtvi/__init__.py

@@ -10,6 +10,11 @@ from pipecat.processors.frameworks.rtvi.frames import (
     RTVIClientMessageFrame,
     RTVIServerMessageFrame,
     RTVIServerResponseFrame,
+    RTVIUICancelTaskFrame,
+    RTVIUICommandFrame,
+    RTVIUIEventFrame,
+    RTVIUISnapshotFrame,
+    RTVIUITaskFrame,
 )
 from pipecat.processors.frameworks.rtvi.observer import (
     RTVIFunctionCallReportLevel,
@@ -26,4 +31,9 @@ __all__ = [
     "RTVIProcessor",
     "RTVIServerMessageFrame",
     "RTVIServerResponseFrame",
+    "RTVIUICancelTaskFrame",
+    "RTVIUICommandFrame",
+    "RTVIUIEventFrame",
+    "RTVIUISnapshotFrame",
+    "RTVIUITaskFrame",
 ]

src/pipecat/processors/frameworks/rtvi/frames.py

@@ -10,6 +10,7 @@ from dataclasses import dataclass
 from typing import Any
 
 from pipecat.frames.frames import SystemFrame
+from pipecat.processors.frameworks.rtvi.models import UITaskData
 
 
 @dataclass
@@ -27,6 +28,134 @@ class RTVIServerMessageFrame(SystemFrame):
         return f"{self.name}(data: {self.data})"
 
 
+@dataclass
+class RTVIUICommandFrame(SystemFrame):
+    """A frame for sending a UI command to the client.
+
+    Pipeline-side counterpart of the ``ui-command`` RTVI message.
+    The observer wraps the ``command_name`` + ``payload`` into a
+    ``UICommandMessage`` envelope before pushing it to the transport,
+    so the wire shape is:
+    ``{label, type: "ui-command", data: {name, payload}}``.
+
+    Parameters:
+        command_name: App-defined command name (e.g. ``"toast"``,
+            ``"navigate"``, or any app-specific name). The wire
+            field is ``data.name``; this avoids shadowing
+            ``SystemFrame.name`` (the per-instance debug label).
+        payload: App-defined payload. Pydantic command models
+            (``Toast``, ``Navigate``, ``ScrollTo``, ...) should be
+            converted to a plain dict via ``model_dump()`` before
+            being placed here; an arbitrary dict works as well.
+    """
+
+    command_name: str = ""
+    payload: Any = None
+
+    def __str__(self):
+        """String representation of the UI command frame."""
+        return f"{self.name}(command: {self.command_name})"
+
+
+@dataclass
+class RTVIUITaskFrame(SystemFrame):
+    """A frame for sending a UI task lifecycle envelope to the client.
+
+    Pipeline-side counterpart of the ``ui-task`` RTVI message. The
+    observer wraps the ``data`` into a ``UITaskMessage`` envelope
+    before pushing it to the transport, so the wire shape is:
+    ``{label, type: "ui-task", data: <one of the four kinds>}``.
+
+    Parameters:
+        data: One of the four task-lifecycle data models from
+            ``rtvi.models`` (``UITaskGroupStartedData``,
+            ``UITaskUpdateData``, ``UITaskCompletedData``, or
+            ``UITaskGroupCompletedData``). The ``kind`` field on
+            each discriminates which lifecycle phase this is.
+    """
+
+    data: UITaskData | None = None
+
+    def __str__(self):
+        """String representation of the UI task frame."""
+        kind = getattr(self.data, "kind", "?")
+        return f"{self.name}(kind: {kind})"
+
+
+@dataclass
+class RTVIUIEventFrame(SystemFrame):
+    """An inbound UI event from the client.
+
+    Pushed downstream by ``RTVIProcessor`` whenever a ``ui-event``
+    message arrives from the client, alongside firing the
+    ``on_ui_message`` event handler. Mirrors the
+    frame-and-event pattern used by ``client-message``: pipeline
+    observers and processors that want to react to UI events at the
+    pipeline level can match on this frame; code that subscribes to
+    events instead (like the bridge in ``pipecat-ai-subagents``)
+    keeps using the event handler.
+
+    Parameters:
+        msg_id: The RTVI message id, as set by the client.
+        event_name: App-defined event name (the ``data.name`` field).
+        payload: App-defined payload (the ``data.payload`` field).
+    """
+
+    msg_id: str = ""
+    event_name: str = ""
+    payload: Any = None
+
+    def __str__(self):
+        """String representation of the UI event frame."""
+        return f"{self.name}(event: {self.event_name})"
+
+
+@dataclass
+class RTVIUISnapshotFrame(SystemFrame):
+    """An inbound accessibility-snapshot from the client.
+
+    Pushed downstream by ``RTVIProcessor`` whenever a ``ui-snapshot``
+    message arrives, alongside firing ``on_ui_message``. Carries
+    the serialized accessibility tree the client took of its DOM.
+
+    Parameters:
+        msg_id: The RTVI message id, as set by the client.
+        tree: The serialized accessibility tree.
+    """
+
+    msg_id: str = ""
+    tree: Any = None
+
+    def __str__(self):
+        """String representation of the UI snapshot frame."""
+        return f"{self.name}"
+
+
+@dataclass
+class RTVIUICancelTaskFrame(SystemFrame):
+    """An inbound user-task-group cancellation request from the client.
+
+    Pushed downstream by ``RTVIProcessor`` whenever a
+    ``ui-cancel-task`` message arrives, alongside firing
+    ``on_ui_message``. The server-side framework should look up the
+    matching task group and cancel it (subject to whatever
+    cancellable policy the group was registered with).
+
+    Parameters:
+        msg_id: The RTVI message id, as set by the client.
+        task_id: The task group id the client wants cancelled.
+        reason: Optional human-readable reason.
+    """
+
+    msg_id: str = ""
+    task_id: str = ""
+    reason: str | None = None
+
+    def __str__(self):
+        """String representation of the UI cancel-task frame."""
+        return f"{self.name}(task_id: {self.task_id})"
+
+
 @dataclass
 class RTVIClientMessageFrame(SystemFrame):
     """A frame for sending messages from the client to the RTVI server.

src/pipecat/processors/frameworks/rtvi/models.py

@@ -27,7 +27,7 @@ from pipecat.frames.frames import (
 )
 
 # -- Constants --
-PROTOCOL_VERSION = "1.2.0"
+PROTOCOL_VERSION = "1.3.0"
 
 MESSAGE_LABEL = "rtvi-ai"
 MessageLiteral = Literal["rtvi-ai"]
@@ -549,3 +549,401 @@ class SystemLogMessage(BaseModel):
     label: MessageLiteral = MESSAGE_LABEL
     type: Literal["system-log"] = "system-log"
     data: TextMessageData
+
+
+# -- UI Agent Protocol -------------------------------------------------------
+#
+# A structured RTVI message vocabulary that lets server-side AI agents
+# observe and drive a GUI app on the client side. The protocol covers
+# five first-class RTVI message types:
+#
+#   ui-event         client-to-server event message
+#   ui-command       server-to-client command message
+#   ui-snapshot      client-to-server accessibility snapshot
+#   ui-cancel-task   client-to-server cancellation request
+#   ui-task          server-to-client task lifecycle envelope
+#
+# This section is data only (constants and payload models, no
+# behavior). Higher-level frameworks like ``pipecat-ai-subagents``
+# build the agent abstractions on top, and single-LLM Pipecat apps can
+# target the same wire format directly via custom tools that emit
+# typed RTVI messages with these types. The matching client-side
+# implementation lives in ``@pipecat-ai/client-js`` and
+# ``@pipecat-ai/client-react``.
+
+# The wire-format ``type`` strings (``"ui-event"``, ``"ui-command"``,
+# ``"ui-snapshot"``, ``"ui-cancel-task"``, ``"ui-task"``) are pinned
+# as ``Literal[...]`` field defaults on the corresponding ``*Message``
+# pydantic class below, matching the convention used for every other
+# RTVI message type in this module.
+
+# Each ``ui-task`` envelope carries a ``kind`` field that the client's
+# task reducer dispatches on. The four kinds form the lifecycle of a
+# user-facing task group:
+#
+#   group_started → task_update* → task_completed × N → group_completed
+#
+# where N is the number of workers in the group. The kind strings are
+# pinned as ``Literal[...]`` defaults on the matching ``UITask*Data``
+# class below.
+
+
+# -- UI envelope data classes --
+
+
+class UIEventData(BaseModel):
+    """Inner ``data`` for a ``ui-event`` message.
+
+    Parameters:
+        name: App-defined event name.
+        payload: App-defined payload, schemaless by design.
+    """
+
+    name: str
+    payload: Any | None = None
+
+
+class UICommandData(BaseModel):
+    """Inner ``data`` for a ``ui-command`` message.
+
+    Parameters:
+        name: App-defined command name.
+        payload: App-defined payload (already a plain dict by the
+            time it lands on the wire). The standard payload models
+            below produce the right shape via ``model_dump()``.
+    """
+
+    name: str
+    payload: Any | None = None
+
+
+class UISnapshotData(BaseModel):
+    """Inner ``data`` for a ``ui-snapshot`` message.
+
+    The accessibility snapshot tree is opaque on the server side.
+    The client owns its shape; the server stores it as-is for
+    rendering into the LLM context.
+
+    Parameters:
+        tree: The serialized accessibility tree.
+    """
+
+    tree: Any | None = None
+
+
+class UICancelTaskData(BaseModel):
+    """Inner ``data`` for a ``ui-cancel-task`` message.
+
+    Parameters:
+        task_id: The task group id the client wants cancelled.
+        reason: Optional human-readable reason.
+    """
+
+    task_id: str
+    reason: str | None = None
+
+
+class UITaskGroupStartedData(BaseModel):
+    """``data`` for a ``ui-task`` envelope with kind ``group_started``.
+
+    Parameters:
+        kind: Always ``"group_started"``.
+        task_id: Shared task identifier for the group.
+        agents: Names of the agents the work was dispatched to.
+        label: Optional human-readable label for the group.
+        cancellable: Whether the client may request cancellation.
+        at: Epoch milliseconds when the group started.
+    """
+
+    kind: Literal["group_started"] = "group_started"
+    task_id: str
+    agents: list[str] | None = None
+    label: str | None = None
+    cancellable: bool = True
+    at: int = 0
+
+
+class UITaskUpdateData(BaseModel):
+    """``data`` for a ``ui-task`` envelope with kind ``task_update``.
+
+    Parameters:
+        kind: Always ``"task_update"``.
+        task_id: The shared task identifier.
+        agent_name: The worker that produced the update.
+        data: The worker's update payload, forwarded verbatim.
+        at: Epoch milliseconds when the update was emitted.
+    """
+
+    kind: Literal["task_update"] = "task_update"
+    task_id: str
+    agent_name: str
+    data: Any | None = None
+    at: int = 0
+
+
+class UITaskCompletedData(BaseModel):
+    """``data`` for a ``ui-task`` envelope with kind ``task_completed``.
+
+    Parameters:
+        kind: Always ``"task_completed"``.
+        task_id: The shared task identifier.
+        agent_name: The worker that produced the response.
+        status: Completion status string.
+        response: The worker's response payload.
+        at: Epoch milliseconds when the response was received.
+    """
+
+    kind: Literal["task_completed"] = "task_completed"
+    task_id: str
+    agent_name: str
+    status: str
+    response: Any | None = None
+    at: int = 0
+
+
+class UITaskGroupCompletedData(BaseModel):
+    """``data`` for a ``ui-task`` envelope with kind ``group_completed``.
+
+    Parameters:
+        kind: Always ``"group_completed"``.
+        task_id: The shared task identifier.
+        at: Epoch milliseconds when the group completed.
+    """
+
+    kind: Literal["group_completed"] = "group_completed"
+    task_id: str
+    at: int = 0
+
+
+#: Discriminated union over the four task-lifecycle data shapes,
+#: keyed by the ``kind`` field.
+UITaskData = (
+    UITaskGroupStartedData | UITaskUpdateData | UITaskCompletedData | UITaskGroupCompletedData
+)
+
+
+# -- UI envelope message classes --
+
+
+class UIEventMessage(BaseModel):
+    """RTVI ``ui-event`` message (client → server)."""
+
+    label: MessageLiteral = MESSAGE_LABEL
+    type: Literal["ui-event"] = "ui-event"
+    id: str
+    data: UIEventData
+
+
+class UICommandMessage(BaseModel):
+    """RTVI ``ui-command`` message (server → client)."""
+
+    label: MessageLiteral = MESSAGE_LABEL
+    type: Literal["ui-command"] = "ui-command"
+    data: UICommandData
+
+
+class UISnapshotMessage(BaseModel):
+    """RTVI ``ui-snapshot`` message (client → server)."""
+
+    label: MessageLiteral = MESSAGE_LABEL
+    type: Literal["ui-snapshot"] = "ui-snapshot"
+    id: str
+    data: UISnapshotData
+
+
+class UICancelTaskMessage(BaseModel):
+    """RTVI ``ui-cancel-task`` message (client → server)."""
+
+    label: MessageLiteral = MESSAGE_LABEL
+    type: Literal["ui-cancel-task"] = "ui-cancel-task"
+    id: str
+    data: UICancelTaskData
+
+
+class UITaskMessage(BaseModel):
+    """RTVI ``ui-task`` message (server → client).
+
+    The ``data`` field is one of the four task-lifecycle
+    discriminated by the ``kind`` field.
+    """
+
+    label: MessageLiteral = MESSAGE_LABEL
+    type: Literal["ui-task"] = "ui-task"
+    data: UITaskData
+
+
+# -- UI command payloads --
+#
+# These models describe commands that have matching default React
+# handlers in ``@pipecat-ai/client-react``'s ``standardHandlers``.
+# Apps can use them as-is, override the client handler to customize
+# rendering, or ignore them entirely and define their own command
+# names.
+#
+# Server-side helpers that send commands accept these models directly.
+# ``BaseModel.model_dump()`` converts them to the plain-dict shape
+# that travels over the wire.
+
+
+class Toast(BaseModel):
+    """A transient notification surface shown on the client.
+
+    Parameters:
+        title: Required headline.
+        subtitle: Optional second line beneath the title.
+        description: Optional body text.
+        image_url: Optional leading image.
+        duration_ms: Optional dismiss timer. Client default applies
+            when None.
+    """
+
+    title: str
+    subtitle: str | None = None
+    description: str | None = None
+    image_url: str | None = None
+    duration_ms: int | None = None
+
+
+class Navigate(BaseModel):
+    """Client-side navigation to a named view.
+
+    Parameters:
+        view: App-defined view name (route, screen id, tab key, etc.).
+        params: Optional view-specific parameters.
+    """
+
+    view: str
+    params: dict | None = None
+
+
+class ScrollTo(BaseModel):
+    """Scroll a target element into view.
+
+    The client resolves the target by ``ref`` first (a snapshot ref
+    like ``"e42"`` assigned by the a11y walker), then falls back to
+    ``target_id`` (``document.getElementById``). Supply whichever you
+    have; ``ref`` is the normal choice when acting on a node from
+    ``<ui_state>``.
+
+    Parameters:
+        ref: Snapshot ref from ``<ui_state>``.
+        target_id: Element id registered on the client.
+        behavior: Optional scroll behavior hint. Typical values:
+            ``"smooth"`` or ``"instant"``. Clients may ignore.
+    """
+
+    ref: str | None = None
+    target_id: str | None = None
+    behavior: str | None = None
+
+
+class Highlight(BaseModel):
+    """Briefly emphasize a target element (flash, glow, pulse).
+
+    Parameters:
+        ref: Snapshot ref from ``<ui_state>``.
+        target_id: Element id registered on the client.
+        duration_ms: Optional highlight duration. Client default
+            applies when None.
+    """
+
+    ref: str | None = None
+    target_id: str | None = None
+    duration_ms: int | None = None
+
+
+class Focus(BaseModel):
+    """Move input focus to a target element.
+
+    Parameters:
+        ref: Snapshot ref from ``<ui_state>``.
+        target_id: Element id registered on the client.
+    """
+
+    ref: str | None = None
+    target_id: str | None = None
+
+
+class Click(BaseModel):
+    """Click an element on the client.
+
+    Closes the form-fill loop for non-text inputs (checkboxes, radios)
+    and exposes the rest of the action vocabulary (submit buttons,
+    links, app-specific clickable nodes). The standard handler
+    silently no-ops on ``disabled`` targets so the agent can't bypass
+    UI affordances the user is meant to control.
+
+    For native ``<select>``, prefer ``SetInputValue`` (clicking
+    options doesn't reliably change the selection); for custom
+    comboboxes (ARIA listbox + popup), apps wire their own command
+    matching the library's interaction model.
+
+    Parameters:
+        ref: Snapshot ref from ``<ui_state>``.
+        target_id: Element id registered on the client. Used as a
+            fallback when ``ref`` is not set or has gone stale.
+    """
+
+    ref: str | None = None
+    target_id: str | None = None
+
+
+class SetInputValue(BaseModel):
+    """Write a value into a text input or textarea on the client.
+
+    Use this for form-filling: the agent has decided what should go
+    into a field (clarifying answer, tax form entry, etc.) and asks
+    the client to populate it. With ``replace=True`` (the default),
+    the existing value is overwritten; with ``replace=False`` the
+    value is appended.
+
+    The standard handler silently no-ops on ``disabled``, ``readonly``,
+    and ``<input type="hidden">`` targets so the agent can't write
+    into fields the user can't.
+
+    Parameters:
+        value: The text to write.
+        ref: Snapshot ref from ``<ui_state>``. Typically the ref of
+            an ``<input>`` or ``<textarea>``.
+        target_id: Element id registered on the client. Used as a
+            fallback when ``ref`` is not set or has gone stale.
+        replace: When True (the default), overwrite the current
+            value. When False, append to it.
+    """
+
+    value: str = ""
+    ref: str | None = None
+    target_id: str | None = None
+    replace: bool = True
+
+
+class SelectText(BaseModel):
+    """Select text on the page so the user can see what the agent means.
+
+    Mirror of the ``selection`` field surfaced in the snapshot. Use
+    this to point the user's attention at a specific paragraph or
+    range after the agent has decided what it's referring to.
+
+    With ``start_offset`` and ``end_offset`` omitted, the entire
+    target's text content is selected (``Range.selectNodeContents``
+    for document elements; ``el.select()`` for ``<input>`` /
+    ``<textarea>``).
+
+    Parameters:
+        ref: Snapshot ref from ``<ui_state>``. Typically the ref of
+            a paragraph or input element.
+        target_id: Element id registered on the client. Used as a
+            fallback when ``ref`` is not set or has gone stale.
+        start_offset: Character offset within the target's text
+            where the selection should start. For ``<input>`` and
+            ``<textarea>`` this is the value offset; for document
+            elements it is computed against the concatenation of
+            descendant text nodes in document order.
+        end_offset: End character offset, exclusive. Same coordinate
+            system as ``start_offset``.
+    """
+
+    ref: str | None = None
+    target_id: str | None = None
+    start_offset: int | None = None
+    end_offset: int | None = None

src/pipecat/processors/frameworks/rtvi/observer.py

@@ -58,6 +58,8 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
 from pipecat.processors.frameworks.rtvi.frames import (
     RTVIServerMessageFrame,
     RTVIServerResponseFrame,
+    RTVIUICommandFrame,
+    RTVIUITaskFrame,
 )
 from pipecat.transports.base_output import BaseOutputTransport
 from pipecat.utils.string import match_endofsentence
@@ -430,6 +432,15 @@ class RTVIObserver(BaseObserver):
         elif isinstance(frame, RTVIServerMessageFrame):
             message = RTVI.ServerMessage(data=frame.data)
             await self.send_rtvi_message(message)
+        elif isinstance(frame, RTVIUICommandFrame):
+            message = RTVI.UICommandMessage(
+                data=RTVI.UICommandData(name=frame.command_name, payload=frame.payload)
+            )
+            await self.send_rtvi_message(message)
+        elif isinstance(frame, RTVIUITaskFrame):
+            if frame.data is not None:
+                message = RTVI.UITaskMessage(data=frame.data)
+                await self.send_rtvi_message(message)
         elif isinstance(frame, RTVIServerResponseFrame):
             if frame.error is not None:
                 await self._send_error_response(frame)

src/pipecat/processors/frameworks/rtvi/processor.py

@@ -32,7 +32,12 @@ from pipecat.frames.frames import (
     SystemFrame,
 )
 from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
-from pipecat.processors.frameworks.rtvi.frames import RTVIClientMessageFrame
+from pipecat.processors.frameworks.rtvi.frames import (
+    RTVIClientMessageFrame,
+    RTVIUICancelTaskFrame,
+    RTVIUIEventFrame,
+    RTVIUISnapshotFrame,
+)
 from pipecat.processors.frameworks.rtvi.observer import RTVIObserver, RTVIObserverParams
 from pipecat.services.llm_service import (
     FunctionCallParams,  # TODO(aleix): we shouldn't import `services` from `processors`
@@ -76,6 +81,7 @@ class RTVIProcessor(FrameProcessor):
         self._register_event_handler("on_bot_started")
         self._register_event_handler("on_client_ready")
         self._register_event_handler("on_client_message")
+        self._register_event_handler("on_ui_message")
 
         self._input_transport = None
         self._transport = transport
@@ -288,6 +294,41 @@ class RTVIProcessor(FrameProcessor):
                 case "client-message":
                     data = RTVI.RawClientMessageData.model_validate(message.data)
                     await self._handle_client_message(message.id, data)
+                case "ui-event":
+                    event_data = RTVI.UIEventData.model_validate(message.data or {})
+                    await self.push_frame(
+                        RTVIUIEventFrame(
+                            msg_id=message.id,
+                            event_name=event_data.name,
+                            payload=event_data.payload,
+                        )
+                    )
+                    await self._call_event_handler(
+                        "on_ui_message",
+                        RTVI.UIEventMessage(id=message.id, data=event_data),
+                    )
+                case "ui-snapshot":
+                    snapshot_data = RTVI.UISnapshotData.model_validate(message.data or {})
+                    await self.push_frame(
+                        RTVIUISnapshotFrame(msg_id=message.id, tree=snapshot_data.tree)
+                    )
+                    await self._call_event_handler(
+                        "on_ui_message",
+                        RTVI.UISnapshotMessage(id=message.id, data=snapshot_data),
+                    )
+                case "ui-cancel-task":
+                    cancel_data = RTVI.UICancelTaskData.model_validate(message.data or {})
+                    await self.push_frame(
+                        RTVIUICancelTaskFrame(
+                            msg_id=message.id,
+                            task_id=cancel_data.task_id,
+                            reason=cancel_data.reason,
+                        )
+                    )
+                    await self._call_event_handler(
+                        "on_ui_message",
+                        RTVI.UICancelTaskMessage(id=message.id, data=cancel_data),
+                    )
                 case "llm-function-call-result":
                     data = RTVI.LLMFunctionCallResultData.model_validate(message.data)
                     await self._handle_function_call_result(data)

tests/test_rtvi_ui.py

@@ -0,0 +1,191 @@
+#
+# Copyright (c) 2026, Daily
+#
+# SPDX-License-Identifier: BSD 2-Clause License
+#
+
+"""Smoke tests for the UI Agent Protocol wire format.
+
+The module under test is data only (constants, payload models, and
+envelope classes), so the goal is to pin the shapes: any accidental
+rename or type change to a wire-format field would break compatibility
+with existing client code, and we want a test that fails loudly.
+"""
+
+import unittest
+
+from pipecat.processors.frameworks.rtvi.models import (
+    Click,
+    Focus,
+    Highlight,
+    Navigate,
+    ScrollTo,
+    SelectText,
+    SetInputValue,
+    Toast,
+    UICancelTaskData,
+    UICancelTaskMessage,
+    UICommandData,
+    UICommandMessage,
+    UIEventData,
+    UIEventMessage,
+    UISnapshotData,
+    UISnapshotMessage,
+    UITaskCompletedData,
+    UITaskGroupCompletedData,
+    UITaskGroupStartedData,
+    UITaskMessage,
+    UITaskUpdateData,
+)
+
+
+class TestEnvelopeMessages(unittest.TestCase):
+    """Pin the on-the-wire envelope shapes for each first-class UI message."""
+
+    def test_ui_event_envelope(self):
+        msg = UIEventMessage(id="m1", data=UIEventData(name="nav_click", payload={"view": "home"}))
+        self.assertEqual(
+            msg.model_dump(),
+            {
+                "label": "rtvi-ai",
+                "type": "ui-event",
+                "id": "m1",
+                "data": {"name": "nav_click", "payload": {"view": "home"}},
+            },
+        )
+
+    def test_ui_command_envelope_no_id(self):
+        # Server-to-client push: no id field on the envelope (matches
+        # ServerMessage / LLMFunctionCallMessage shape).
+        msg = UICommandMessage(data=UICommandData(name="toast", payload={"title": "Saved"}))
+        self.assertEqual(
+            msg.model_dump(),
+            {
+                "label": "rtvi-ai",
+                "type": "ui-command",
+                "data": {"name": "toast", "payload": {"title": "Saved"}},
+            },
+        )
+
+    def test_ui_snapshot_envelope(self):
+        msg = UISnapshotMessage(id="m2", data=UISnapshotData(tree={"root": "..."}))
+        self.assertEqual(
+            msg.model_dump(),
+            {
+                "label": "rtvi-ai",
+                "type": "ui-snapshot",
+                "id": "m2",
+                "data": {"tree": {"root": "..."}},
+            },
+        )
+
+    def test_ui_cancel_task_envelope(self):
+        msg = UICancelTaskMessage(id="m3", data=UICancelTaskData(task_id="t-99", reason="user"))
+        self.assertEqual(
+            msg.model_dump(),
+            {
+                "label": "rtvi-ai",
+                "type": "ui-cancel-task",
+                "id": "m3",
+                "data": {"task_id": "t-99", "reason": "user"},
+            },
+        )
+
+    def test_ui_task_group_started(self):
+        msg = UITaskMessage(
+            data=UITaskGroupStartedData(task_id="t-1", agents=["a", "b"], label="Search", at=42)
+        )
+        self.assertEqual(msg.type, "ui-task")
+        self.assertEqual(msg.data.kind, "group_started")
+        self.assertEqual(msg.data.task_id, "t-1")
+
+    def test_ui_task_update(self):
+        msg = UITaskMessage(
+            data=UITaskUpdateData(task_id="t-1", agent_name="a", data={"progress": 0.5}, at=43)
+        )
+        self.assertEqual(msg.data.kind, "task_update")
+        self.assertEqual(msg.data.agent_name, "a")
+
+    def test_ui_task_completed(self):
+        msg = UITaskMessage(
+            data=UITaskCompletedData(
+                task_id="t-1", agent_name="a", status="completed", response={"ok": True}, at=44
+            )
+        )
+        self.assertEqual(msg.data.kind, "task_completed")
+        self.assertEqual(msg.data.status, "completed")
+
+    def test_ui_task_group_completed(self):
+        msg = UITaskMessage(data=UITaskGroupCompletedData(task_id="t-1", at=45))
+        self.assertEqual(msg.data.kind, "group_completed")
+
+
+class TestPayloadShapes(unittest.TestCase):
+    """Pin the on-the-wire dict shape of each command payload."""
+
+    def test_toast_required_only(self):
+        self.assertEqual(
+            dict(Toast(title="Saved")),
+            {
+                "title": "Saved",
+                "subtitle": None,
+                "description": None,
+                "image_url": None,
+                "duration_ms": None,
+            },
+        )
+
+    def test_navigate(self):
+        self.assertEqual(
+            dict(Navigate(view="home", params={"id": "42"})),
+            {"view": "home", "params": {"id": "42"}},
+        )
+
+    def test_scroll_to_with_ref(self):
+        self.assertEqual(
+            dict(ScrollTo(ref="e42", behavior="smooth")),
+            {"ref": "e42", "target_id": None, "behavior": "smooth"},
+        )
+
+    def test_highlight_with_target_id(self):
+        self.assertEqual(
+            dict(Highlight(target_id="cta", duration_ms=2000)),
+            {"ref": None, "target_id": "cta", "duration_ms": 2000},
+        )
+
+    def test_focus(self):
+        self.assertEqual(
+            dict(Focus(ref="e7")),
+            {"ref": "e7", "target_id": None},
+        )
+
+    def test_click(self):
+        self.assertEqual(
+            dict(Click(ref="e9")),
+            {"ref": "e9", "target_id": None},
+        )
+
+    def test_set_input_value_default_replace_true(self):
+        self.assertEqual(
+            dict(SetInputValue(ref="e3", value="Marie Curie")),
+            {"value": "Marie Curie", "ref": "e3", "target_id": None, "replace": True},
+        )
+
+    def test_set_input_value_append(self):
+        payload = dict(SetInputValue(ref="e3", value="more", replace=False))
+        self.assertFalse(payload["replace"])
+
+    def test_select_text_full(self):
+        self.assertEqual(
+            dict(SelectText(ref="e15", start_offset=4, end_offset=12)),
+            {"ref": "e15", "target_id": None, "start_offset": 4, "end_offset": 12},
+        )
+
+    def test_select_text_whole_target_when_offsets_omitted(self):
+        payload = dict(SelectText(ref="e15"))
+        self.assertIsNone(payload["start_offset"])
+        self.assertIsNone(payload["end_offset"])
+
+
+if __name__ == "__main__":
+    unittest.main()