diff --git a/changelog/4407.added.md b/changelog/4407.added.md index dcae4bf31..c3f628389 100644 --- a/changelog/4407.added.md +++ b/changelog/4407.added.md @@ -1,6 +1,6 @@ -- Added the UI Agent Protocol to `pipecat.processors.frameworks.rtvi.models`: - - Five first-class RTVI message types carry the wire format: `ui-event` (client → server), `ui-command` (server → client), `ui-snapshot` (client → server, accessibility tree), `ui-cancel-task` (client → server), and `ui-task` (server → client task lifecycle envelopes). Each ships a paired `*Data` / `*Message` pydantic model following the existing RTVI convention. - - Built-in command payload models (`Toast`, `Navigate`, `ScrollTo`, `Highlight`, `Focus`, `Click`, `SetInputValue`, `SelectText`) ship alongside; matching default handlers live in `@pipecat-ai/client-react`. - - The `RTVIProcessor` registers a new `on_ui_message` event handler that fires for inbound `ui-event` / `ui-snapshot` / `ui-cancel-task`. - - Five new pipeline frames let pipeline observers and processors see UI traffic the same way they see other RTVI messages: `RTVIUICommandFrame` and `RTVIUITaskFrame` are pushed by downstream code to be wrapped by the observer into outbound `UICommandMessage` / `UITaskMessage` envelopes; `RTVIUIEventFrame`, `RTVIUISnapshotFrame`, and `RTVIUICancelTaskFrame` are pushed by the processor on inbound, alongside firing `on_ui_message`, mirroring the frame-and-event pattern used by `client-message`. +- Added first-class RTVI support for the UI Agent Protocol: + - Adds `ui-event`, `ui-snapshot`, and `ui-cancel-task` client-to-server messages, plus `ui-command` and `ui-task` server-to-client messages, with paired `*Data` / `*Message` pydantic models. + - Adds built-in command payload models for `Toast`, `Navigate`, `ScrollTo`, `Highlight`, `Focus`, `Click`, `SetInputValue`, and `SelectText`; matching default handlers live in `@pipecat-ai/client-react`. + - Adds `RTVIProcessor.on_ui_message` for inbound `ui-event`, `ui-snapshot`, and `ui-cancel-task` messages. + - Adds five UI pipeline frames, mirroring the `client-message` frame-and-event pattern: downstream code pushes `RTVIUICommandFrame` / `RTVIUITaskFrame` for the observer to wrap into outbound `UICommandMessage` / `UITaskMessage` envelopes, while the processor pushes inbound `RTVIUIEventFrame`, `RTVIUISnapshotFrame`, and `RTVIUICancelTaskFrame` alongside `on_ui_message`. - Bumps the RTVI `PROTOCOL_VERSION` from `1.2.0` to `1.3.0`.