feat(rtvi): add UI Agent Protocol as first-class RTVI message types
The UI Agent Protocol lets server-side AI agents observe and drive
a GUI app on the client side through structured RTVI messages.
Five new top-level RTVI types in kebab-case, in line with the rest
of the protocol:
ui-event client → server (named event with payload)
ui-command server → client (named command with payload)
ui-snapshot client → server (accessibility tree of the page)
ui-cancel-task client → server (cancel an in-flight task group)
ui-task server → client (task lifecycle envelope)
Each ships paired ``*Data`` / ``*Message`` pydantic models in
``rtvi.models``, following the existing RTVI envelope convention
(``BotReady`` / ``BotReadyData``, ``Error`` / ``ErrorData``, etc.).
Built-in command payload models (``Toast``, ``Navigate``,
``ScrollTo``, ``Highlight``, ``Focus``, ``Click``, ``SetInputValue``,
``SelectText``) ship alongside; matching default React handlers
live in ``@pipecat-ai/client-react``.
Bumps the RTVI ``PROTOCOL_VERSION`` from ``1.2.0`` to ``1.3.0``.
Purely additive: only new top-level message types are introduced;
no existing wire shapes are changed. The major-version
compatibility check on ``client-ready`` still passes for older
1.x clients, so old clients continue to connect without warning;
they simply will not exercise the new types.
The ``RTVIProcessor`` registers a new ``on_ui_message`` event
handler that fires for inbound ``ui-event`` / ``ui-snapshot`` /
``ui-cancel-task`` with the parsed Message envelope, mirroring how
``on_client_message`` works for ``client-message``.
Five new pipeline frames let pipeline observers and processors see
UI traffic the same way they see other RTVI messages, mirroring
the frame-and-event pattern used by ``client-message``:
RTVIUICommandFrame(command_name, payload)
Pushed by downstream code (e.g. ``pipecat-ai-subagents``'s
bridge) to send a UI command to the client. Wrapped by the
observer into a ``UICommandMessage`` envelope.
RTVIUITaskFrame(data: UITaskData)
Same shape but for ``ui-task``; wrapped into ``UITaskMessage``.
``UITaskData`` is a discriminated union of the four lifecycle
kinds (group_started / task_update / task_completed /
group_completed).
RTVIUIEventFrame(msg_id, event_name, payload)
RTVIUISnapshotFrame(msg_id, tree)
RTVIUICancelTaskFrame(msg_id, task_id, reason)
Pushed by ``RTVIProcessor._handle_message`` whenever the
matching inbound message arrives, alongside firing
``on_ui_message``. Pipeline observers and processors can match
on the frame; subscribers like the subagents bridge keep using
the event handler.
The data layer is the canonical authority for the wire format:
higher-level frameworks like ``pipecat-ai-subagents`` build the
agent abstractions on top, and single-LLM Pipecat apps can target
the same wire format directly via custom tools that emit these
typed messages.
This commit is contained in:
@@ -32,7 +32,12 @@ from pipecat.frames.frames import (
|
||||
SystemFrame,
|
||||
)
|
||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||
from pipecat.processors.frameworks.rtvi.frames import RTVIClientMessageFrame
|
||||
from pipecat.processors.frameworks.rtvi.frames import (
|
||||
RTVIClientMessageFrame,
|
||||
RTVIUICancelTaskFrame,
|
||||
RTVIUIEventFrame,
|
||||
RTVIUISnapshotFrame,
|
||||
)
|
||||
from pipecat.processors.frameworks.rtvi.observer import RTVIObserver, RTVIObserverParams
|
||||
from pipecat.services.llm_service import (
|
||||
FunctionCallParams, # TODO(aleix): we shouldn't import `services` from `processors`
|
||||
@@ -76,6 +81,7 @@ class RTVIProcessor(FrameProcessor):
|
||||
self._register_event_handler("on_bot_started")
|
||||
self._register_event_handler("on_client_ready")
|
||||
self._register_event_handler("on_client_message")
|
||||
self._register_event_handler("on_ui_message")
|
||||
|
||||
self._input_transport = None
|
||||
self._transport = transport
|
||||
@@ -288,6 +294,41 @@ class RTVIProcessor(FrameProcessor):
|
||||
case "client-message":
|
||||
data = RTVI.RawClientMessageData.model_validate(message.data)
|
||||
await self._handle_client_message(message.id, data)
|
||||
case "ui-event":
|
||||
event_data = RTVI.UIEventData.model_validate(message.data or {})
|
||||
await self.push_frame(
|
||||
RTVIUIEventFrame(
|
||||
msg_id=message.id,
|
||||
event_name=event_data.name,
|
||||
payload=event_data.payload,
|
||||
)
|
||||
)
|
||||
await self._call_event_handler(
|
||||
"on_ui_message",
|
||||
RTVI.UIEventMessage(id=message.id, data=event_data),
|
||||
)
|
||||
case "ui-snapshot":
|
||||
snapshot_data = RTVI.UISnapshotData.model_validate(message.data or {})
|
||||
await self.push_frame(
|
||||
RTVIUISnapshotFrame(msg_id=message.id, tree=snapshot_data.tree)
|
||||
)
|
||||
await self._call_event_handler(
|
||||
"on_ui_message",
|
||||
RTVI.UISnapshotMessage(id=message.id, data=snapshot_data),
|
||||
)
|
||||
case "ui-cancel-task":
|
||||
cancel_data = RTVI.UICancelTaskData.model_validate(message.data or {})
|
||||
await self.push_frame(
|
||||
RTVIUICancelTaskFrame(
|
||||
msg_id=message.id,
|
||||
task_id=cancel_data.task_id,
|
||||
reason=cancel_data.reason,
|
||||
)
|
||||
)
|
||||
await self._call_event_handler(
|
||||
"on_ui_message",
|
||||
RTVI.UICancelTaskMessage(id=message.id, data=cancel_data),
|
||||
)
|
||||
case "llm-function-call-result":
|
||||
data = RTVI.LLMFunctionCallResultData.model_validate(message.data)
|
||||
await self._handle_function_call_result(data)
|
||||
|
||||
Reference in New Issue
Block a user