wx44wx/pipecat
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(rtvi): add UI Agent Protocol as first-class RTVI message types

Browse Source 
The UI Agent Protocol lets server-side AI agents observe and drive
a GUI app on the client side through structured RTVI messages.
Five new top-level RTVI types in kebab-case, in line with the rest
of the protocol:

  ui-event         client → server  (named event with payload)
  ui-command       server → client  (named command with payload)
  ui-snapshot      client → server  (accessibility tree of the page)
  ui-cancel-task   client → server  (cancel an in-flight task group)
  ui-task          server → client  (task lifecycle envelope)

Each ships paired ``*Data`` / ``*Message`` pydantic models in
``rtvi.models``, following the existing RTVI envelope convention
(``BotReady`` / ``BotReadyData``, ``Error`` / ``ErrorData``, etc.).
Built-in command payload models (``Toast``, ``Navigate``,
``ScrollTo``, ``Highlight``, ``Focus``, ``Click``, ``SetInputValue``,
``SelectText``) ship alongside; matching default React handlers
live in ``@pipecat-ai/client-react``.

Bumps the RTVI ``PROTOCOL_VERSION`` from ``1.2.0`` to ``1.3.0``.
Purely additive: only new top-level message types are introduced;
no existing wire shapes are changed. The major-version
compatibility check on ``client-ready`` still passes for older
1.x clients, so old clients continue to connect without warning;
they simply will not exercise the new types.

The ``RTVIProcessor`` registers a new ``on_ui_message`` event
handler that fires for inbound ``ui-event`` / ``ui-snapshot`` /
``ui-cancel-task`` with the parsed Message envelope, mirroring how
``on_client_message`` works for ``client-message``.

Five new pipeline frames let pipeline observers and processors see
UI traffic the same way they see other RTVI messages, mirroring
the frame-and-event pattern used by ``client-message``:

  RTVIUICommandFrame(command_name, payload)
    Pushed by downstream code (e.g. ``pipecat-ai-subagents``'s
    bridge) to send a UI command to the client. Wrapped by the
    observer into a ``UICommandMessage`` envelope.

  RTVIUITaskFrame(data: UITaskData)
    Same shape but for ``ui-task``; wrapped into ``UITaskMessage``.
    ``UITaskData`` is a discriminated union of the four lifecycle
    kinds (group_started / task_update / task_completed /
    group_completed).

  RTVIUIEventFrame(msg_id, event_name, payload)
  RTVIUISnapshotFrame(msg_id, tree)
  RTVIUICancelTaskFrame(msg_id, task_id, reason)
    Pushed by ``RTVIProcessor._handle_message`` whenever the
    matching inbound message arrives, alongside firing
    ``on_ui_message``. Pipeline observers and processors can match
    on the frame; subscribers like the subagents bridge keep using
    the event handler.

The data layer is the canonical authority for the wire format:
higher-level frameworks like ``pipecat-ai-subagents`` build the
agent abstractions on top, and single-LLM Pipecat apps can target
the same wire format directly via custom tools that emit these
typed messages.
This commit is contained in:
Mark Backman
2026-05-02 08:38:51 -04:00
parent 30efd11e15
commit 43abca0b06
7 changed files with 788 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
changelog/4407.added.md Normal file
View File

@@ -0,0 +1,6 @@
- Added the UI Agent Protocol to `pipecat.processors.frameworks.rtvi.models`:
- Five first-class RTVI message types carry the wire format: `ui-event` (client → server), `ui-command` (server → client), `ui-snapshot` (client → server, accessibility tree), `ui-cancel-task` (client → server), and `ui-task` (server → client task lifecycle envelopes). Each ships a paired `*Data` / `*Message` pydantic model following the existing RTVI convention.
- Built-in command payload models (`Toast`, `Navigate`, `ScrollTo`, `Highlight`, `Focus`, `Click`, `SetInputValue`, `SelectText`) ship alongside; matching default handlers live in `@pipecat-ai/client-react`.
- The `RTVIProcessor` registers a new `on_ui_message` event handler that fires for inbound `ui-event` / `ui-snapshot` / `ui-cancel-task`.
- Five new pipeline frames let pipeline observers and processors see UI traffic the same way they see other RTVI messages: `RTVIUICommandFrame` and `RTVIUITaskFrame` are pushed by downstream code to be wrapped by the observer into outbound `UICommandMessage` / `UITaskMessage` envelopes; `RTVIUIEventFrame`, `RTVIUISnapshotFrame`, and `RTVIUICancelTaskFrame` are pushed by the processor on inbound, alongside firing `on_ui_message`, mirroring the frame-and-event pattern used by `client-message`.
- Bumps the RTVI `PROTOCOL_VERSION` from `1.2.0` to `1.3.0`.

10
src/pipecat/processors/frameworks/rtvi/__init__.py
View File

@@ -10,6 +10,11 @@ from pipecat.processors.frameworks.rtvi.frames import (
RTVIClientMessageFrame,
RTVIServerMessageFrame,
RTVIServerResponseFrame,
RTVIUICancelTaskFrame,
RTVIUICommandFrame,
RTVIUIEventFrame,
RTVIUISnapshotFrame,
RTVIUITaskFrame,
)
from pipecat.processors.frameworks.rtvi.observer import (
RTVIFunctionCallReportLevel,
@@ -26,4 +31,9 @@ __all__ = [
"RTVIProcessor",
"RTVIServerMessageFrame",
"RTVIServerResponseFrame",
"RTVIUICancelTaskFrame",
"RTVIUICommandFrame",
"RTVIUIEventFrame",
"RTVIUISnapshotFrame",
"RTVIUITaskFrame",
]

129
src/pipecat/processors/frameworks/rtvi/frames.py
View File

@@ -10,6 +10,7 @@ from dataclasses import dataclass
from typing import Any
from pipecat.frames.frames import SystemFrame
from pipecat.processors.frameworks.rtvi.models import UITaskData
@dataclass
@@ -27,6 +28,134 @@ class RTVIServerMessageFrame(SystemFrame):
return f"{self.name}(data: {self.data})"
@dataclass
class RTVIUICommandFrame(SystemFrame):
"""A frame for sending a UI command to the client.
Pipeline-side counterpart of the ``ui-command`` RTVI message.
The observer wraps the ``command_name`` + ``payload`` into a
``UICommandMessage`` envelope before pushing it to the transport,
so the wire shape is:
``{label, type: "ui-command", data: {name, payload}}``.
Parameters:
command_name: App-defined command name (e.g. ``"toast"``,
``"navigate"``, or any app-specific name). The wire
field is ``data.name``; this avoids shadowing
``SystemFrame.name`` (the per-instance debug label).
payload: App-defined payload. Pydantic command models
(``Toast``, ``Navigate``, ``ScrollTo``, ...) should be
converted to a plain dict via ``model_dump()`` before
being placed here; an arbitrary dict works as well.
"""
command_name: str = ""
payload: Any = None
def __str__(self):
"""String representation of the UI command frame."""
return f"{self.name}(command: {self.command_name})"
@dataclass
class RTVIUITaskFrame(SystemFrame):
"""A frame for sending a UI task lifecycle envelope to the client.
Pipeline-side counterpart of the ``ui-task`` RTVI message. The
observer wraps the ``data`` into a ``UITaskMessage`` envelope
before pushing it to the transport, so the wire shape is:
``{label, type: "ui-task", data: <one of the four kinds>}``.
Parameters:
data: One of the four task-lifecycle data models from
``rtvi.models`` (``UITaskGroupStartedData``,
``UITaskUpdateData``, ``UITaskCompletedData``, or
``UITaskGroupCompletedData``). The ``kind`` field on
each discriminates which lifecycle phase this is.
"""
data: UITaskData | None = None
def __str__(self):
"""String representation of the UI task frame."""
kind = getattr(self.data, "kind", "?")
return f"{self.name}(kind: {kind})"
@dataclass
class RTVIUIEventFrame(SystemFrame):
"""An inbound UI event from the client.
Pushed downstream by ``RTVIProcessor`` whenever a ``ui-event``
message arrives from the client, alongside firing the
``on_ui_message`` event handler. Mirrors the
frame-and-event pattern used by ``client-message``: pipeline
observers and processors that want to react to UI events at the
pipeline level can match on this frame; code that subscribes to
events instead (like the bridge in ``pipecat-ai-subagents``)
keeps using the event handler.
Parameters:
msg_id: The RTVI message id, as set by the client.
event_name: App-defined event name (the ``data.name`` field).
payload: App-defined payload (the ``data.payload`` field).
"""
msg_id: str = ""
event_name: str = ""
payload: Any = None
def __str__(self):
"""String representation of the UI event frame."""
return f"{self.name}(event: {self.event_name})"
@dataclass
class RTVIUISnapshotFrame(SystemFrame):
"""An inbound accessibility-snapshot from the client.
Pushed downstream by ``RTVIProcessor`` whenever a ``ui-snapshot``
message arrives, alongside firing ``on_ui_message``. Carries
the serialized accessibility tree the client took of its DOM.
Parameters:
msg_id: The RTVI message id, as set by the client.
tree: The serialized accessibility tree.
"""
msg_id: str = ""
tree: Any = None
def __str__(self):
"""String representation of the UI snapshot frame."""
return f"{self.name}"
@dataclass
class RTVIUICancelTaskFrame(SystemFrame):
"""An inbound user-task-group cancellation request from the client.
Pushed downstream by ``RTVIProcessor`` whenever a
``ui-cancel-task`` message arrives, alongside firing
``on_ui_message``. The server-side framework should look up the
matching task group and cancel it (subject to whatever
cancellable policy the group was registered with).
Parameters:
msg_id: The RTVI message id, as set by the client.
task_id: The task group id the client wants cancelled.
reason: Optional human-readable reason.
"""
msg_id: str = ""
task_id: str = ""
reason: str | None = None
def __str__(self):
"""String representation of the UI cancel-task frame."""
return f"{self.name}(task_id: {self.task_id})"
@dataclass
class RTVIClientMessageFrame(SystemFrame):
"""A frame for sending messages from the client to the RTVI server.

400
src/pipecat/processors/frameworks/rtvi/models.py
View File

@@ -27,7 +27,7 @@ from pipecat.frames.frames import (
)
# -- Constants --
PROTOCOL_VERSION = "1.2.0"
PROTOCOL_VERSION = "1.3.0"
MESSAGE_LABEL = "rtvi-ai"
MessageLiteral = Literal["rtvi-ai"]
@@ -549,3 +549,401 @@ class SystemLogMessage(BaseModel):
label: MessageLiteral = MESSAGE_LABEL
type: Literal["system-log"] = "system-log"
data: TextMessageData
# -- UI Agent Protocol -------------------------------------------------------
#
# A structured RTVI message vocabulary that lets server-side AI agents
# observe and drive a GUI app on the client side. The protocol covers
# five first-class RTVI message types:
#
# ui-event client-to-server event message
# ui-command server-to-client command message
# ui-snapshot client-to-server accessibility snapshot
# ui-cancel-task client-to-server cancellation request
# ui-task server-to-client task lifecycle envelope
#
# This section is data only (constants and payload models, no
# behavior). Higher-level frameworks like ``pipecat-ai-subagents``
# build the agent abstractions on top, and single-LLM Pipecat apps can
# target the same wire format directly via custom tools that emit
# typed RTVI messages with these types. The matching client-side
# implementation lives in ``@pipecat-ai/client-js`` and
# ``@pipecat-ai/client-react``.
# The wire-format ``type`` strings (``"ui-event"``, ``"ui-command"``,
# ``"ui-snapshot"``, ``"ui-cancel-task"``, ``"ui-task"``) are pinned
# as ``Literal[...]`` field defaults on the corresponding ``*Message``
# pydantic class below, matching the convention used for every other
# RTVI message type in this module.
# Each ``ui-task`` envelope carries a ``kind`` field that the client's
# task reducer dispatches on. The four kinds form the lifecycle of a
# user-facing task group:
#
# group_started → task_update* → task_completed × N → group_completed
#
# where N is the number of workers in the group. The kind strings are
# pinned as ``Literal[...]`` defaults on the matching ``UITask*Data``
# class below.
# -- UI envelope data classes --
class UIEventData(BaseModel):
"""Inner ``data`` for a ``ui-event`` message.
Parameters:
name: App-defined event name.
payload: App-defined payload, schemaless by design.
"""
name: str
payload: Any | None = None
class UICommandData(BaseModel):
"""Inner ``data`` for a ``ui-command`` message.
Parameters:
name: App-defined command name.
payload: App-defined payload (already a plain dict by the
time it lands on the wire). The standard payload models
below produce the right shape via ``model_dump()``.
"""
name: str
payload: Any | None = None
class UISnapshotData(BaseModel):
"""Inner ``data`` for a ``ui-snapshot`` message.
The accessibility snapshot tree is opaque on the server side.
The client owns its shape; the server stores it as-is for
rendering into the LLM context.
Parameters:
tree: The serialized accessibility tree.
"""
tree: Any | None = None
class UICancelTaskData(BaseModel):
"""Inner ``data`` for a ``ui-cancel-task`` message.
Parameters:
task_id: The task group id the client wants cancelled.
reason: Optional human-readable reason.
"""
task_id: str
reason: str | None = None
class UITaskGroupStartedData(BaseModel):
"""``data`` for a ``ui-task`` envelope with kind ``group_started``.
Parameters:
kind: Always ``"group_started"``.
task_id: Shared task identifier for the group.
agents: Names of the agents the work was dispatched to.
label: Optional human-readable label for the group.
cancellable: Whether the client may request cancellation.
at: Epoch milliseconds when the group started.
"""
kind: Literal["group_started"] = "group_started"
task_id: str
agents: list[str] | None = None
label: str | None = None
cancellable: bool = True
at: int = 0
class UITaskUpdateData(BaseModel):
"""``data`` for a ``ui-task`` envelope with kind ``task_update``.
Parameters:
kind: Always ``"task_update"``.
task_id: The shared task identifier.
agent_name: The worker that produced the update.
data: The worker's update payload, forwarded verbatim.
at: Epoch milliseconds when the update was emitted.
"""
kind: Literal["task_update"] = "task_update"
task_id: str
agent_name: str
data: Any | None = None
at: int = 0
class UITaskCompletedData(BaseModel):
"""``data`` for a ``ui-task`` envelope with kind ``task_completed``.
Parameters:
kind: Always ``"task_completed"``.
task_id: The shared task identifier.
agent_name: The worker that produced the response.
status: Completion status string.
response: The worker's response payload.
at: Epoch milliseconds when the response was received.
"""
kind: Literal["task_completed"] = "task_completed"
task_id: str
agent_name: str
status: str
response: Any | None = None
at: int = 0
class UITaskGroupCompletedData(BaseModel):
"""``data`` for a ``ui-task`` envelope with kind ``group_completed``.
Parameters:
kind: Always ``"group_completed"``.
task_id: The shared task identifier.
at: Epoch milliseconds when the group completed.
"""
kind: Literal["group_completed"] = "group_completed"
task_id: str
at: int = 0
#: Discriminated union over the four task-lifecycle data shapes,
#: keyed by the ``kind`` field.
UITaskData = (
UITaskGroupStartedData | UITaskUpdateData | UITaskCompletedData | UITaskGroupCompletedData
)
# -- UI envelope message classes --
class UIEventMessage(BaseModel):
"""RTVI ``ui-event`` message (client → server)."""
label: MessageLiteral = MESSAGE_LABEL
type: Literal["ui-event"] = "ui-event"
id: str
data: UIEventData
class UICommandMessage(BaseModel):
"""RTVI ``ui-command`` message (server → client)."""
label: MessageLiteral = MESSAGE_LABEL
type: Literal["ui-command"] = "ui-command"
data: UICommandData
class UISnapshotMessage(BaseModel):
"""RTVI ``ui-snapshot`` message (client → server)."""
label: MessageLiteral = MESSAGE_LABEL
type: Literal["ui-snapshot"] = "ui-snapshot"
id: str
data: UISnapshotData
class UICancelTaskMessage(BaseModel):
"""RTVI ``ui-cancel-task`` message (client → server)."""
label: MessageLiteral = MESSAGE_LABEL
type: Literal["ui-cancel-task"] = "ui-cancel-task"
id: str
data: UICancelTaskData
class UITaskMessage(BaseModel):
"""RTVI ``ui-task`` message (server → client).
The ``data`` field is one of the four task-lifecycle
discriminated by the ``kind`` field.
"""
label: MessageLiteral = MESSAGE_LABEL
type: Literal["ui-task"] = "ui-task"
data: UITaskData
# -- UI command payloads --
#
# These models describe commands that have matching default React
# handlers in ``@pipecat-ai/client-react``'s ``standardHandlers``.
# Apps can use them as-is, override the client handler to customize
# rendering, or ignore them entirely and define their own command
# names.
#
# Server-side helpers that send commands accept these models directly.
# ``BaseModel.model_dump()`` converts them to the plain-dict shape
# that travels over the wire.
class Toast(BaseModel):
"""A transient notification surface shown on the client.
Parameters:
title: Required headline.
subtitle: Optional second line beneath the title.
description: Optional body text.
image_url: Optional leading image.
duration_ms: Optional dismiss timer. Client default applies
when None.
"""
title: str
subtitle: str | None = None
description: str | None = None
image_url: str | None = None
duration_ms: int | None = None
class Navigate(BaseModel):
"""Client-side navigation to a named view.
Parameters:
view: App-defined view name (route, screen id, tab key, etc.).
params: Optional view-specific parameters.
"""
view: str
params: dict | None = None
class ScrollTo(BaseModel):
"""Scroll a target element into view.
The client resolves the target by ``ref`` first (a snapshot ref
like ``"e42"`` assigned by the a11y walker), then falls back to
``target_id`` (``document.getElementById``). Supply whichever you
have; ``ref`` is the normal choice when acting on a node from
``<ui_state>``.
Parameters:
ref: Snapshot ref from ``<ui_state>``.
target_id: Element id registered on the client.
behavior: Optional scroll behavior hint. Typical values:
``"smooth"`` or ``"instant"``. Clients may ignore.
"""
ref: str | None = None
target_id: str | None = None
behavior: str | None = None
class Highlight(BaseModel):
"""Briefly emphasize a target element (flash, glow, pulse).
Parameters:
ref: Snapshot ref from ``<ui_state>``.
target_id: Element id registered on the client.
duration_ms: Optional highlight duration. Client default
applies when None.
"""
ref: str | None = None
target_id: str | None = None
duration_ms: int | None = None
class Focus(BaseModel):
"""Move input focus to a target element.
Parameters:
ref: Snapshot ref from ``<ui_state>``.
target_id: Element id registered on the client.
"""
ref: str | None = None
target_id: str | None = None
class Click(BaseModel):
"""Click an element on the client.
Closes the form-fill loop for non-text inputs (checkboxes, radios)
and exposes the rest of the action vocabulary (submit buttons,
links, app-specific clickable nodes). The standard handler
silently no-ops on ``disabled`` targets so the agent can't bypass
UI affordances the user is meant to control.
For native ``<select>``, prefer ``SetInputValue`` (clicking
options doesn't reliably change the selection); for custom
comboboxes (ARIA listbox + popup), apps wire their own command
matching the library's interaction model.
Parameters:
ref: Snapshot ref from ``<ui_state>``.
target_id: Element id registered on the client. Used as a
fallback when ``ref`` is not set or has gone stale.
"""
ref: str | None = None
target_id: str | None = None
class SetInputValue(BaseModel):
"""Write a value into a text input or textarea on the client.
Use this for form-filling: the agent has decided what should go
into a field (clarifying answer, tax form entry, etc.) and asks
the client to populate it. With ``replace=True`` (the default),
the existing value is overwritten; with ``replace=False`` the
value is appended.
The standard handler silently no-ops on ``disabled``, ``readonly``,
and ``<input type="hidden">`` targets so the agent can't write
into fields the user can't.
Parameters:
value: The text to write.
ref: Snapshot ref from ``<ui_state>``. Typically the ref of
an ``<input>`` or ``<textarea>``.
target_id: Element id registered on the client. Used as a
fallback when ``ref`` is not set or has gone stale.
replace: When True (the default), overwrite the current
value. When False, append to it.
"""
value: str = ""
ref: str | None = None
target_id: str | None = None
replace: bool = True
class SelectText(BaseModel):
"""Select text on the page so the user can see what the agent means.
Mirror of the ``selection`` field surfaced in the snapshot. Use
this to point the user's attention at a specific paragraph or
range after the agent has decided what it's referring to.
With ``start_offset`` and ``end_offset`` omitted, the entire
target's text content is selected (``Range.selectNodeContents``
for document elements; ``el.select()`` for ``<input>`` /
``<textarea>``).
Parameters:
ref: Snapshot ref from ``<ui_state>``. Typically the ref of
a paragraph or input element.
target_id: Element id registered on the client. Used as a
fallback when ``ref`` is not set or has gone stale.
start_offset: Character offset within the target's text
where the selection should start. For ``<input>`` and
``<textarea>`` this is the value offset; for document
elements it is computed against the concatenation of
descendant text nodes in document order.
end_offset: End character offset, exclusive. Same coordinate
system as ``start_offset``.
"""
ref: str | None = None
target_id: str | None = None
start_offset: int | None = None
end_offset: int | None = None

11
src/pipecat/processors/frameworks/rtvi/observer.py
View File

@@ -58,6 +58,8 @@ from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.frameworks.rtvi.frames import (
RTVIServerMessageFrame,
RTVIServerResponseFrame,
RTVIUICommandFrame,
RTVIUITaskFrame,
)
from pipecat.transports.base_output import BaseOutputTransport
from pipecat.utils.string import match_endofsentence
@@ -430,6 +432,15 @@ class RTVIObserver(BaseObserver):
elif isinstance(frame, RTVIServerMessageFrame):
message = RTVI.ServerMessage(data=frame.data)
await self.send_rtvi_message(message)
elif isinstance(frame, RTVIUICommandFrame):
message = RTVI.UICommandMessage(
data=RTVI.UICommandData(name=frame.command_name, payload=frame.payload)
)
await self.send_rtvi_message(message)
elif isinstance(frame, RTVIUITaskFrame):
if frame.data is not None:
message = RTVI.UITaskMessage(data=frame.data)
await self.send_rtvi_message(message)
elif isinstance(frame, RTVIServerResponseFrame):
if frame.error is not None:
await self._send_error_response(frame)

43
src/pipecat/processors/frameworks/rtvi/processor.py
View File

@@ -32,7 +32,12 @@ from pipecat.frames.frames import (
SystemFrame,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.processors.frameworks.rtvi.frames import RTVIClientMessageFrame
from pipecat.processors.frameworks.rtvi.frames import (
RTVIClientMessageFrame,
RTVIUICancelTaskFrame,
RTVIUIEventFrame,
RTVIUISnapshotFrame,
)
from pipecat.processors.frameworks.rtvi.observer import RTVIObserver, RTVIObserverParams
from pipecat.services.llm_service import (
FunctionCallParams, # TODO(aleix): we shouldn't import `services` from `processors`
@@ -76,6 +81,7 @@ class RTVIProcessor(FrameProcessor):
self._register_event_handler("on_bot_started")
self._register_event_handler("on_client_ready")
self._register_event_handler("on_client_message")
self._register_event_handler("on_ui_message")
self._input_transport = None
self._transport = transport
@@ -288,6 +294,41 @@ class RTVIProcessor(FrameProcessor):
case "client-message":
data = RTVI.RawClientMessageData.model_validate(message.data)
await self._handle_client_message(message.id, data)
case "ui-event":
event_data = RTVI.UIEventData.model_validate(message.data or {})
await self.push_frame(
RTVIUIEventFrame(
msg_id=message.id,
event_name=event_data.name,
payload=event_data.payload,
)
)
await self._call_event_handler(
"on_ui_message",
RTVI.UIEventMessage(id=message.id, data=event_data),
)
case "ui-snapshot":
snapshot_data = RTVI.UISnapshotData.model_validate(message.data or {})
await self.push_frame(
RTVIUISnapshotFrame(msg_id=message.id, tree=snapshot_data.tree)
)
await self._call_event_handler(
"on_ui_message",
RTVI.UISnapshotMessage(id=message.id, data=snapshot_data),
)
case "ui-cancel-task":
cancel_data = RTVI.UICancelTaskData.model_validate(message.data or {})
await self.push_frame(
RTVIUICancelTaskFrame(
msg_id=message.id,
task_id=cancel_data.task_id,
reason=cancel_data.reason,
)
)
await self._call_event_handler(
"on_ui_message",
RTVI.UICancelTaskMessage(id=message.id, data=cancel_data),
)
case "llm-function-call-result":
data = RTVI.LLMFunctionCallResultData.model_validate(message.data)
await self._handle_function_call_result(data)

191
tests/test_rtvi_ui.py Normal file
View File

@@ -0,0 +1,191 @@
#
# Copyright (c) 2026, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
"""Smoke tests for the UI Agent Protocol wire format.
The module under test is data only (constants, payload models, and
envelope classes), so the goal is to pin the shapes: any accidental
rename or type change to a wire-format field would break compatibility
with existing client code, and we want a test that fails loudly.
"""
import unittest
from pipecat.processors.frameworks.rtvi.models import (
Click,
Focus,
Highlight,
Navigate,
ScrollTo,
SelectText,
SetInputValue,
Toast,
UICancelTaskData,
UICancelTaskMessage,
UICommandData,
UICommandMessage,
UIEventData,
UIEventMessage,
UISnapshotData,
UISnapshotMessage,
UITaskCompletedData,
UITaskGroupCompletedData,
UITaskGroupStartedData,
UITaskMessage,
UITaskUpdateData,
)
class TestEnvelopeMessages(unittest.TestCase):
"""Pin the on-the-wire envelope shapes for each first-class UI message."""
def test_ui_event_envelope(self):
msg = UIEventMessage(id="m1", data=UIEventData(name="nav_click", payload={"view": "home"}))
self.assertEqual(
msg.model_dump(),
{
"label": "rtvi-ai",
"type": "ui-event",
"id": "m1",
"data": {"name": "nav_click", "payload": {"view": "home"}},
},
)
def test_ui_command_envelope_no_id(self):
# Server-to-client push: no id field on the envelope (matches
# ServerMessage / LLMFunctionCallMessage shape).
msg = UICommandMessage(data=UICommandData(name="toast", payload={"title": "Saved"}))
self.assertEqual(
msg.model_dump(),
{
"label": "rtvi-ai",
"type": "ui-command",
"data": {"name": "toast", "payload": {"title": "Saved"}},
},
)
def test_ui_snapshot_envelope(self):
msg = UISnapshotMessage(id="m2", data=UISnapshotData(tree={"root": "..."}))
self.assertEqual(
msg.model_dump(),
{
"label": "rtvi-ai",
"type": "ui-snapshot",
"id": "m2",
"data": {"tree": {"root": "..."}},
},
)
def test_ui_cancel_task_envelope(self):
msg = UICancelTaskMessage(id="m3", data=UICancelTaskData(task_id="t-99", reason="user"))
self.assertEqual(
msg.model_dump(),
{
"label": "rtvi-ai",
"type": "ui-cancel-task",
"id": "m3",
"data": {"task_id": "t-99", "reason": "user"},
},
)
def test_ui_task_group_started(self):
msg = UITaskMessage(
data=UITaskGroupStartedData(task_id="t-1", agents=["a", "b"], label="Search", at=42)
)
self.assertEqual(msg.type, "ui-task")
self.assertEqual(msg.data.kind, "group_started")
self.assertEqual(msg.data.task_id, "t-1")
def test_ui_task_update(self):
msg = UITaskMessage(
data=UITaskUpdateData(task_id="t-1", agent_name="a", data={"progress": 0.5}, at=43)
)
self.assertEqual(msg.data.kind, "task_update")
self.assertEqual(msg.data.agent_name, "a")
def test_ui_task_completed(self):
msg = UITaskMessage(
data=UITaskCompletedData(
task_id="t-1", agent_name="a", status="completed", response={"ok": True}, at=44
)
)
self.assertEqual(msg.data.kind, "task_completed")
self.assertEqual(msg.data.status, "completed")
def test_ui_task_group_completed(self):
msg = UITaskMessage(data=UITaskGroupCompletedData(task_id="t-1", at=45))
self.assertEqual(msg.data.kind, "group_completed")
class TestPayloadShapes(unittest.TestCase):
"""Pin the on-the-wire dict shape of each command payload."""
def test_toast_required_only(self):
self.assertEqual(
dict(Toast(title="Saved")),
{
"title": "Saved",
"subtitle": None,
"description": None,
"image_url": None,
"duration_ms": None,
},
)
def test_navigate(self):
self.assertEqual(
dict(Navigate(view="home", params={"id": "42"})),
{"view": "home", "params": {"id": "42"}},
)
def test_scroll_to_with_ref(self):
self.assertEqual(
dict(ScrollTo(ref="e42", behavior="smooth")),
{"ref": "e42", "target_id": None, "behavior": "smooth"},
)
def test_highlight_with_target_id(self):
self.assertEqual(
dict(Highlight(target_id="cta", duration_ms=2000)),
{"ref": None, "target_id": "cta", "duration_ms": 2000},
)
def test_focus(self):
self.assertEqual(
dict(Focus(ref="e7")),
{"ref": "e7", "target_id": None},
)
def test_click(self):
self.assertEqual(
dict(Click(ref="e9")),
{"ref": "e9", "target_id": None},
)
def test_set_input_value_default_replace_true(self):
self.assertEqual(
dict(SetInputValue(ref="e3", value="Marie Curie")),
{"value": "Marie Curie", "ref": "e3", "target_id": None, "replace": True},
)
def test_set_input_value_append(self):
payload = dict(SetInputValue(ref="e3", value="more", replace=False))
self.assertFalse(payload["replace"])
def test_select_text_full(self):
self.assertEqual(
dict(SelectText(ref="e15", start_offset=4, end_offset=12)),
{"ref": "e15", "target_id": None, "start_offset": 4, "end_offset": 12},
)
def test_select_text_whole_target_when_offsets_omitted(self):
payload = dict(SelectText(ref="e15"))
self.assertIsNone(payload["start_offset"])
self.assertIsNone(payload["end_offset"])
if __name__ == "__main__":
unittest.main()