Add input image to product protocol
This commit is contained in:
19
README.md
19
README.md
@@ -95,6 +95,25 @@ Send audio:
|
||||
|
||||
The adapter also accepts raw binary websocket messages as PCM16 audio chunks. JSON/base64 is easier to inspect; binary is better for latency and bandwidth.
|
||||
|
||||
Send a camera snapshot for vision-capable LLM replies:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "input.image",
|
||||
"image": "<base64 jpeg/png/webp bytes>",
|
||||
"mime_type": "image/jpeg",
|
||||
"width": 640,
|
||||
"height": 360,
|
||||
"text": "Answer using this camera image.",
|
||||
"append_to_context": true
|
||||
}
|
||||
```
|
||||
|
||||
`input.image` appends the image to the Pipecat LLM context as a
|
||||
`UserImageRawFrame` and immediately triggers the LLM. The reply returns through
|
||||
the existing `response.text.*` and `response.audio.*` events. Prefer occasional
|
||||
compressed camera snapshots over continuous video frames.
|
||||
|
||||
Stop:
|
||||
|
||||
```json
|
||||
|
||||
Reference in New Issue
Block a user