Add Mermaid diagram support and update architecture documentation
- Included a new JavaScript file for Mermaid configuration to ensure consistent diagram sizing across documentation. - Enhanced architecture documentation to reflect the updated pipeline engine structure, including VAD, ASR, TD, LLM, and TTS components. - Updated various sections to clarify the integration of external services and tools within the architecture. - Improved styling for Mermaid diagrams to enhance visual consistency and usability.
This commit is contained in:
@@ -19,23 +19,25 @@
|
||||
|
||||
### 架构设计
|
||||
|
||||
管线式引擎将语音交互拆分为三个独立阶段:
|
||||
管线式引擎包含 **声音活动检测(VAD)**、**语音识别(ASR)**、**回合检测(TD)**、**大语言模型(LLM)**、**语音合成(TTS)**,各环节可对接**外部服务**(OpenAI、SiliconFlow、DashScope、本地模型)。LLM 可连接**工具**(Webhook、客户端工具、内建工具)。
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Input["输入处理"]
|
||||
Audio[用户音频] --> VAD[VAD 检测]
|
||||
VAD --> ASR[语音识别]
|
||||
Audio[用户音频] --> VAD[声音活动检测 VAD]
|
||||
VAD --> ASR[语音识别 ASR]
|
||||
ASR --> Text[转写文本]
|
||||
Text --> TD[回合检测 TD]
|
||||
end
|
||||
|
||||
subgraph Process["语义处理"]
|
||||
Text --> LLM[大语言模型]
|
||||
TD --> LLM[大语言模型 LLM]
|
||||
LLM --> Response[回复文本]
|
||||
LLM --> Tools[工具]
|
||||
end
|
||||
|
||||
subgraph Output["输出生成"]
|
||||
Response --> TTS[语音合成]
|
||||
Response --> TTS[语音合成 TTS]
|
||||
TTS --> OutputAudio[助手音频]
|
||||
end
|
||||
```
|
||||
@@ -55,7 +57,7 @@ sequenceDiagram
|
||||
Note over E: VAD 检测语音活动
|
||||
E->>E: 累积音频缓冲
|
||||
|
||||
Note over E: 检测到语音结束 (EOU)
|
||||
Note over E: 回合检测 (TD) 确定可送 LLM 的输入
|
||||
E->>ASR: 发送音频
|
||||
ASR-->>E: 转写文本 (流式)
|
||||
E-->>U: transcript.delta
|
||||
@@ -111,9 +113,21 @@ gantt
|
||||
|
||||
---
|
||||
|
||||
## 多模态引擎 (Multimodal)
|
||||
## 实时交互引擎与多模态
|
||||
|
||||
### 架构设计
|
||||
### 实时交互引擎连接
|
||||
|
||||
实时交互引擎可连接**实时交互引擎**后端,包括:
|
||||
|
||||
| 后端 | 说明 |
|
||||
|------|------|
|
||||
| **OpenAI Realtime** | OpenAI 实时语音模型 |
|
||||
| **Gemini Live** | Google 实时多模态 |
|
||||
| **Doubao 实时交互引擎** | 豆包实时交互 |
|
||||
|
||||
实时交互引擎与管线式引擎中的 LLM 一样,均可连接**工具**:Webhook、客户端工具、内建工具。
|
||||
|
||||
### 多模态引擎架构
|
||||
|
||||
多模态引擎使用端到端模型,直接处理音频输入输出:
|
||||
|
||||
@@ -127,12 +141,13 @@ flowchart LR
|
||||
subgraph Engine["引擎"]
|
||||
AudioIn --> RT[Realtime Model]
|
||||
RT --> AudioOut
|
||||
RT --> Tools[工具]
|
||||
end
|
||||
|
||||
subgraph Model["多模态模型"]
|
||||
RT --> GPT4o[GPT-4o Realtime]
|
||||
subgraph Model["实时交互引擎"]
|
||||
RT --> GPT4o[OpenAI Realtime]
|
||||
RT --> Gemini[Gemini Live]
|
||||
RT --> Step[Step Audio]
|
||||
RT --> Doubao[Doubao 实时]
|
||||
end
|
||||
```
|
||||
|
||||
@@ -155,13 +170,24 @@ sequenceDiagram
|
||||
Note over U,RT: 支持全双工<br/>用户可随时打断
|
||||
```
|
||||
|
||||
### 支持的模型
|
||||
### 外部服务(管线式)
|
||||
|
||||
管线式引擎各环节可选用以下**外部服务**:
|
||||
|
||||
| 服务 | 说明 |
|
||||
|------|------|
|
||||
| **OpenAI** | LLM / ASR / TTS 等 |
|
||||
| **SiliconFlow** | 国内 API 服务 |
|
||||
| **DashScope** | 阿里云灵积 |
|
||||
| **本地模型** | 私有化部署模型 |
|
||||
|
||||
### 支持的实时交互模型
|
||||
|
||||
| 模型 | 供应商 | 特点 |
|
||||
|------|--------|------|
|
||||
| **GPT-4o Realtime** | OpenAI | 最自然的语音,延迟极低 |
|
||||
| **OpenAI Realtime** | OpenAI | 最自然的语音,延迟极低 |
|
||||
| **Gemini Live** | Google | 多模态能力强 |
|
||||
| **Step Audio** | 阶跃星辰 | 国内可用,中文优化 |
|
||||
| **Doubao 实时交互** | 字节跳动 | 国内可用,中文优化 |
|
||||
|
||||
### 延迟对比
|
||||
|
||||
|
||||
Reference in New Issue
Block a user