116 lines
5.2 KiB
Markdown
116 lines
5.2 KiB
Markdown
# Plan: `@pipecat-ai/moq-transport`
|
||
|
||
A future Pipecat Client SDK transport plugin for Media-over-QUIC. Lets any
|
||
Pipecat JS UI — including the smallwebrtc playground — talk to a
|
||
MOQ-backed bot with zero UI changes, just by swapping the transport.
|
||
|
||
## Why this exists
|
||
|
||
The Pipecat Client SDK (`@pipecat-ai/client-js`) is transport-agnostic. UIs
|
||
talk to a `PipecatClient`, which delegates to a pluggable `Transport`. The
|
||
existing transports are:
|
||
|
||
- `@pipecat-ai/small-webrtc-transport` (WebRTC)
|
||
- `@pipecat-ai/daily-transport` (Daily)
|
||
- `@pipecat-ai/websocket-transport` (raw WebSocket / telephony)
|
||
- `@pipecat-ai/gemini-live-websocket-transport`
|
||
|
||
There is **no** `@pipecat-ai/moq-transport`. So MOQ bots today need a hand-
|
||
rolled client (which is what `moq_prebuilt/client/` is). Publishing a
|
||
proper transport plugin would:
|
||
|
||
1. Replace `moq_prebuilt/client/` with the prebuilt smallwebrtc UI — same
|
||
conversation panel, metrics, device pickers, transcript overlay, etc.
|
||
2. Let downstream Pipecat apps drop MOQ in as easily as WebRTC.
|
||
3. Centralize the MOQ wire-protocol work in one package instead of every
|
||
demo reinventing it.
|
||
|
||
## What we already have
|
||
|
||
About 80% of the wire-protocol work is done in
|
||
`moq_prebuilt/client/app.js` and `moq_prebuilt/client/moq-client.js`:
|
||
|
||
- moq-lite-02 codec (CLIENT_SETUP, SERVER_SETUP, ANNOUNCE_PLEASE /
|
||
ANNOUNCE_INIT, SUBSCRIBE / SUBSCRIBE_OK, GROUP / FRAME).
|
||
- WebTransport bring-up with certificate pinning.
|
||
- Per-participant broadcast paths (`<namespace>/<participant_id>`) and
|
||
multi-track routing (`bot-audio`, `user-audio`, `transcript`).
|
||
- Mic capture via AudioWorklet → 16 kHz PCM publish.
|
||
- Bot-audio 24 kHz PCM playback via Web Audio.
|
||
- RTVI message parsing on the transcript track (`bot-llm-*`,
|
||
`user-transcription`, etc.).
|
||
|
||
The package's job is to **wrap that in the `Transport` interface** and
|
||
emit the events `PipecatClient` expects.
|
||
|
||
## Effort estimate
|
||
|
||
### Usable v1 — 3–5 days
|
||
|
||
For a single bot / single client dev setup that runs against the
|
||
playground UI.
|
||
|
||
| Day | Work |
|
||
| --- | ---- |
|
||
| 1 | TypeScript package skeleton (build, tsconfig, ESM/CJS exports, npm scripts). Port existing protocol logic into a class. |
|
||
| 2 | Implement `Transport` interface: `connect()`, `disconnect()`, `sendMessage()`, `getDevices()`. Wire the RTVI message parsing on the transcript track into the event names `PipecatClient` expects (`connected`, `disconnected`, `trackStarted`, `userTranscript`, `botStartedSpeaking`, …). |
|
||
| 3 | Device handling — mic picker, mute, sample-rate negotiation, AudioWorklet inlined as a string blob. |
|
||
| 4 | Run against `ConsoleTemplate` from `@pipecat-ai/voice-ui-kit`. Fix lifecycle bugs (mid-call mute, reconnect, etc.). |
|
||
| 5 | Docs, example app, README. |
|
||
|
||
### Production-grade — ~2 weeks on top
|
||
|
||
- Multi-participant discovery via outbound `ANNOUNCE_PLEASE` + reacting to
|
||
`ANNOUNCE_UPDATE` so multiple bots / multiple clients can share a namespace.
|
||
- CA-signed cert path (not just self-signed `serverCertificateHashes`
|
||
pinning).
|
||
- Reconnection / network-blip recovery.
|
||
- Test suite (unit + at least one e2e against a real relay).
|
||
- Cross-browser sanity (Chrome reference, Safari WebTransport just
|
||
shipped, Firefox still behind a flag).
|
||
- npm publish workflow / CI.
|
||
|
||
## Known unknowns (could blow the estimate)
|
||
|
||
- **PipecatClient `Transport` surface area** — haven't done a full read of
|
||
`@pipecat-ai/client-js`. If the interface expects something MOQ can't
|
||
cleanly model (peer-connection introspection, ICE candidates, anything
|
||
WebRTC-specific that leaks into the abstraction), we need
|
||
workarounds/polyfills.
|
||
- **Audio interop with the voice-ui-kit's audio visualizers** —
|
||
`<VoicePresence>`, level meters, etc. read from a `MediaStreamTrack`. MOQ
|
||
delivers raw PCM uni-streams, not a media-stream-track. We'd need to
|
||
synthesize a `MediaStreamTrack` from the decoded PCM so the rest of the
|
||
kit just works. Could be a half-day rabbit hole or could be quick — there
|
||
are off-the-shelf libraries (`AudioWorklet` + `MediaStreamDestination`) but
|
||
the integration details aren't trivial.
|
||
|
||
## Suggested next step (cheap reconnaissance)
|
||
|
||
Spend ~2 hours reading:
|
||
- `@pipecat-ai/client-js/src/transport.ts` — the `Transport` abstract
|
||
interface.
|
||
- `@pipecat-ai/small-webrtc-transport` — closest analogue, since both
|
||
open a session and stream media bidirectionally.
|
||
- `@pipecat-ai/websocket-transport` — simpler reference; might be a closer
|
||
fit for MOQ since neither is WebRTC.
|
||
|
||
Output: a concrete TypeScript API surface for `MoqTransport` (constructor
|
||
options, method signatures, events emitted) + a sharper estimate. Two
|
||
hours of investment to either commit to the project or pass on it
|
||
informedly.
|
||
|
||
## Open questions for whoever picks this up
|
||
|
||
- One participant identity per `Transport` instance, or should the
|
||
transport manage multiple peers internally (matching how WebRTC SFUs do
|
||
it)?
|
||
- How should `getDevices()` map to MOQ — pure browser device enumeration
|
||
(since there's no SDK-side device negotiation)?
|
||
- Where does the per-call ID come from? Server-allocated via
|
||
`/start`, or client-generated UUID? (Today we hardcode `bot0` /
|
||
`client0`.)
|
||
- Cert pinning UX — for prod, can we get away with assuming the relay has
|
||
a real CA cert, or do we need an explicit `serverCertificateHashes`
|
||
config knob?
|