diff --git a/docs/frame-progress.md b/docs/frame-progress.md index 5a83c4bdc..f4348bf88 100644 --- a/docs/frame-progress.md +++ b/docs/frame-progress.md @@ -1,4 +1,46 @@ # A Frame's Progress +1. A user says “Hello, LLM” and the cloud transcription service delivers a transcription to the Transport. ![A transcript frame arrives](images/frame-progress-01.png) +2. The Transport places a Transcription frame in the Pipeline’s source queue. +![Frame in source queue](images/frame-progress-02.png) + +3. The Pipeline passes the Transcription frame to the first Frame Processor in its list, the LLM User Message Aggregator. +![To UMA](images/frame-progress-03.png) + +4. The LLM User Message Aggregator updates the LLM Context with a `{“user”: “Hello LLM”}` message. +![Update context](images/frame-progress-04.png) + +5. The LLM User Message Aggregator yields an LLM Message Frame, containing the updated LLM Context. The Pipeline passes this frame to the LLM Frame Processor. +![Update context](images/frame-progress-05.png) + +6. The LLM Frame Processor creates a streaming chat completion based on the LLM context and yields the first chunk of a response, Text Frame with the value “Hi, “. The Pipeline passes this frame to the TTS Frame Processor. The TTS Frame Processor aggregates this response but doesn’t yield anything, yet, because it’s waiting for a full sentence. +![LLM yields Text](images/frame-progress-06.png) + +7. The LLM Frame Processor yields another Text Frame with the value “there.”. The Pipeline passes this frame to the TTS Frame Processor. +![LLM yields more Text](images/frame-progress-07.png) + +8. The TTS Frame Processor now has a full sentence, so it starts streaming audio based on “Hi, there.” It yields the first chunk of streaming audio as an Audio frame, which the Pipeline passes to the LLM Assistant Message Aggregator. +![TTS yields Audio](images/frame-progress-08.png) + +9. The LLM Assistant Message Aggregator doesn’t do anything with Audio frames, so it immediately yields the frame, unchanged. This is the convention for all Frame Processors: frames that the processor doesn’t process should be immediately yielded. +![pass-through](images/frame-progress-09.png) + +10. The Pipeline places the first Audio frame in its sink queue, which is being watched by the Transport. Since the frame is now in a queue, the Pipeline can continue processing other frames. Note that the source and sink queues form a sort of “boundary of concurrent processing” between a Pipeline and the outside world. In a Pipeline, Frames are processed sequentially; once a Frame is on a queue it can be processed in parallel with the frames being processed by the Pipeline. TODO: link to a more in-depth section about this. +![sink queue](images/frame-progress-10.png) + +11. The TTS Frame Processor yields another Audio frame as the Transport transmits the first Audio frame. +![parallel audio](images/frame-progress-11.png) + +12. As before, the LLM Assistant Message Aggregator immediately yields the Audio frame and the Pipeline places the Audio frame in the sink queue. +![sink queue 2](images/frame-progress-12.png) + +13. The TTS Frame Processor has no more frames to yield. The LLM Frame Processor emits an LLM Response End Frame, which the Pipeline passes to the TTS Frame Processor. +![response end](images/frame-progress-13.png) + +14. The TTS Frame Processor immediately yields the LLM Response End Frame, so the Pipeline passes it along to the LLM Assistant Message Aggregator. The LLM Assistant Message Aggregator updates the LLM Context with the full response from the LLM. TODO TODO: I realized I forgot that the TSS Frame Processor also yields the Text frames that the LLM emitted so that the LLM Assistant Message Aggregator could accumulate them, arrggh. +![response end](images/frame-progress-14.png) + +15. The system is quiet, and waiting for the next message from the Transport. +![response end](images/frame-progress-15.png) diff --git a/docs/images/frame-progress-01.png b/docs/images/frame-progress-01.png index 829ad58e8..f4f0a0e3e 100644 Binary files a/docs/images/frame-progress-01.png and b/docs/images/frame-progress-01.png differ diff --git a/docs/images/frame-progress-02.png b/docs/images/frame-progress-02.png index e90afba8d..7eddb0c1d 100644 Binary files a/docs/images/frame-progress-02.png and b/docs/images/frame-progress-02.png differ diff --git a/docs/images/frame-progress-03.png b/docs/images/frame-progress-03.png index ccfb22b4e..7579be4b0 100644 Binary files a/docs/images/frame-progress-03.png and b/docs/images/frame-progress-03.png differ diff --git a/docs/images/frame-progress-04.png b/docs/images/frame-progress-04.png index 377e971d3..b215c7ccc 100644 Binary files a/docs/images/frame-progress-04.png and b/docs/images/frame-progress-04.png differ diff --git a/docs/images/frame-progress-05.png b/docs/images/frame-progress-05.png index e1b7aae5c..5fb2ef967 100644 Binary files a/docs/images/frame-progress-05.png and b/docs/images/frame-progress-05.png differ diff --git a/docs/images/frame-progress-06.png b/docs/images/frame-progress-06.png index 791440615..d39510c7c 100644 Binary files a/docs/images/frame-progress-06.png and b/docs/images/frame-progress-06.png differ diff --git a/docs/images/frame-progress-07.png b/docs/images/frame-progress-07.png index 6a0e38512..cdfc8b0ef 100644 Binary files a/docs/images/frame-progress-07.png and b/docs/images/frame-progress-07.png differ diff --git a/docs/images/frame-progress-08.png b/docs/images/frame-progress-08.png index 3cd3d36a4..382882d52 100644 Binary files a/docs/images/frame-progress-08.png and b/docs/images/frame-progress-08.png differ diff --git a/docs/images/frame-progress-09.png b/docs/images/frame-progress-09.png index ef2ba2255..fd83bffa3 100644 Binary files a/docs/images/frame-progress-09.png and b/docs/images/frame-progress-09.png differ diff --git a/docs/images/frame-progress-10.png b/docs/images/frame-progress-10.png index 0b60ddeae..f4c25aaae 100644 Binary files a/docs/images/frame-progress-10.png and b/docs/images/frame-progress-10.png differ diff --git a/docs/images/frame-progress-11.png b/docs/images/frame-progress-11.png index ef987d4cf..f6f8f0fdc 100644 Binary files a/docs/images/frame-progress-11.png and b/docs/images/frame-progress-11.png differ diff --git a/docs/images/frame-progress-12.png b/docs/images/frame-progress-12.png index db420b20e..84b97555a 100644 Binary files a/docs/images/frame-progress-12.png and b/docs/images/frame-progress-12.png differ diff --git a/docs/images/frame-progress-13.png b/docs/images/frame-progress-13.png index 3664c278d..40835eb6d 100644 Binary files a/docs/images/frame-progress-13.png and b/docs/images/frame-progress-13.png differ diff --git a/docs/images/frame-progress-14.png b/docs/images/frame-progress-14.png index fa7e75fdb..228f0278d 100644 Binary files a/docs/images/frame-progress-14.png and b/docs/images/frame-progress-14.png differ diff --git a/docs/images/frame-progress-15.png b/docs/images/frame-progress-15.png index 6d66e6309..86ee47f8a 100644 Binary files a/docs/images/frame-progress-15.png and b/docs/images/frame-progress-15.png differ