pipecat/moq_prebuilt/PLAN-moq-transport-package.md

Plan: @pipecat-ai/moq-transport

A future Pipecat Client SDK transport plugin for Media-over-QUIC. Lets any Pipecat JS UI — including the smallwebrtc playground — talk to a MOQ-backed bot with zero UI changes, just by swapping the transport.

Why this exists

The Pipecat Client SDK (@pipecat-ai/client-js) is transport-agnostic. UIs talk to a PipecatClient, which delegates to a pluggable Transport. The existing transports are:

• @pipecat-ai/small-webrtc-transport (WebRTC)
• @pipecat-ai/daily-transport (Daily)
• @pipecat-ai/websocket-transport (raw WebSocket / telephony)
• @pipecat-ai/gemini-live-websocket-transport

There is no @pipecat-ai/moq-transport. So MOQ bots today need a hand- rolled client (which is what moq_prebuilt/client/ is). Publishing a proper transport plugin would:

1. Replace moq_prebuilt/client/ with the prebuilt smallwebrtc UI — same conversation panel, metrics, device pickers, transcript overlay, etc.
2. Let downstream Pipecat apps drop MOQ in as easily as WebRTC.
3. Centralize the MOQ wire-protocol work in one package instead of every demo reinventing it.

What we already have

About 80% of the wire-protocol work is done in moq_prebuilt/client/app.js and moq_prebuilt/client/moq-client.js:

• moq-lite-02 codec (CLIENT_SETUP, SERVER_SETUP, ANNOUNCE_PLEASE / ANNOUNCE_INIT, SUBSCRIBE / SUBSCRIBE_OK, GROUP / FRAME).
• WebTransport bring-up with certificate pinning.
• Per-participant broadcast paths (<namespace>/<participant_id>) and multi-track routing (bot-audio, user-audio, transcript).
• Mic capture via AudioWorklet → 16 kHz PCM publish.
• Bot-audio 24 kHz PCM playback via Web Audio.
• RTVI message parsing on the transcript track (bot-llm-*, user-transcription, etc.).

The package's job is to wrap that in the Transport interface and emit the events PipecatClient expects.

Effort estimate

Usable v1 — 3–5 days

For a single bot / single client dev setup that runs against the playground UI.

Day	Work
1	TypeScript package skeleton (build, tsconfig, ESM/CJS exports, npm scripts). Port existing protocol logic into a class.
2	Implement Transport interface: connect(), disconnect(), sendMessage(), getDevices(). Wire the RTVI message parsing on the transcript track into the event names PipecatClient expects (connected, disconnected, trackStarted, userTranscript, botStartedSpeaking, …).
3	Device handling — mic picker, mute, sample-rate negotiation, AudioWorklet inlined as a string blob.
4	Run against ConsoleTemplate from @pipecat-ai/voice-ui-kit. Fix lifecycle bugs (mid-call mute, reconnect, etc.).
5	Docs, example app, README.

Production-grade — ~2 weeks on top

• Multi-participant discovery via outbound ANNOUNCE_PLEASE + reacting to ANNOUNCE_UPDATE so multiple bots / multiple clients can share a namespace.
• CA-signed cert path (not just self-signed serverCertificateHashes pinning).
• Reconnection / network-blip recovery.
• Test suite (unit + at least one e2e against a real relay).
• Cross-browser sanity (Chrome reference, Safari WebTransport just shipped, Firefox still behind a flag).
• npm publish workflow / CI.

Known unknowns (could blow the estimate)

• PipecatClient Transport surface area — haven't done a full read of @pipecat-ai/client-js. If the interface expects something MOQ can't cleanly model (peer-connection introspection, ICE candidates, anything WebRTC-specific that leaks into the abstraction), we need workarounds/polyfills.
• Audio interop with the voice-ui-kit's audio visualizers — <VoicePresence>, level meters, etc. read from a MediaStreamTrack. MOQ delivers raw PCM uni-streams, not a media-stream-track. We'd need to synthesize a MediaStreamTrack from the decoded PCM so the rest of the kit just works. Could be a half-day rabbit hole or could be quick — there are off-the-shelf libraries (AudioWorklet + MediaStreamDestination) but the integration details aren't trivial.

Suggested next step (cheap reconnaissance)

Spend ~2 hours reading:

• @pipecat-ai/client-js/src/transport.ts — the Transport abstract interface.
• @pipecat-ai/small-webrtc-transport — closest analogue, since both open a session and stream media bidirectionally.
• @pipecat-ai/websocket-transport — simpler reference; might be a closer fit for MOQ since neither is WebRTC.

Output: a concrete TypeScript API surface for MoqTransport (constructor options, method signatures, events emitted) + a sharper estimate. Two hours of investment to either commit to the project or pass on it informedly.

Open questions for whoever picks this up

• One participant identity per Transport instance, or should the transport manage multiple peers internally (matching how WebRTC SFUs do it)?
• How should getDevices() map to MOQ — pure browser device enumeration (since there's no SDK-side device negotiation)?
• Where does the per-call ID come from? Server-allocated via /start, or client-generated UUID? (Today we hardcode bot0 / client0.)
• Cert pinning UX — for prod, can we get away with assuming the relay has a real CA cert, or do we need an explicit serverCertificateHashes config knob?