Add input image to product protocol

This commit is contained in:
Xin Wang
2026-05-23 08:28:56 +08:00
parent 02db9545f0
commit 5b087f8610
4 changed files with 117 additions and 3 deletions

View File

@@ -95,6 +95,25 @@ Send audio:
The adapter also accepts raw binary websocket messages as PCM16 audio chunks. JSON/base64 is easier to inspect; binary is better for latency and bandwidth.
Send a camera snapshot for vision-capable LLM replies:
```json
{
"type": "input.image",
"image": "<base64 jpeg/png/webp bytes>",
"mime_type": "image/jpeg",
"width": 640,
"height": 360,
"text": "Answer using this camera image.",
"append_to_context": true
}
```
`input.image` appends the image to the Pipecat LLM context as a
`UserImageRawFrame` and immediately triggers the LLM. The reply returns through
the existing `response.text.*` and `response.audio.*` events. Prefer occasional
compressed camera snapshots over continuous video frames.
Stop:
```json