Remove test files and testing documentation from PR
This commit is contained in:
zack
@@ -1,273 +0,0 @@
|
||||
# AssemblyAI u3-rt-pro Testing Checklist
|
||||
|
||||
## Test Environment Setup
|
||||
- [ ] Install dependencies: `uv sync --group dev --all-extras`
|
||||
- [ ] Set up `.env` file with API keys
|
||||
- [ ] Verify LiveKit connection
|
||||
- [ ] Run basic voice agent test
|
||||
|
||||
---
|
||||
|
||||
## Feature Testing Checklist
|
||||
|
||||
### ✅ Basic Configuration Tests
|
||||
|
||||
#### Test 1: Default u3-rt-pro Configuration
|
||||
- [ ] **Setup:** Create service with default params
|
||||
- [ ] **Expected:** No errors, uses u3-rt-pro model with 100ms min/max
|
||||
- [ ] **Verify:** Check logs for connection confirmation
|
||||
|
||||
#### Test 2: Custom min_turn_silence
|
||||
- [ ] **Setup:** Set `min_turn_silence=200`
|
||||
- [ ] **Expected:** Both min and max set to 200ms
|
||||
- [ ] **Verify:** Speak short phrases, observe turn detection timing
|
||||
|
||||
#### Test 3: User sets max_turn_silence (Warning Test)
|
||||
- [ ] **Setup:** Set `max_turn_silence=500` in connection params
|
||||
- [ ] **Expected:** Warning logged, value overridden to match min
|
||||
- [ ] **Verify:** Check logs for warning message
|
||||
|
||||
---
|
||||
|
||||
### ✅ Prompting Tests
|
||||
|
||||
#### Test 4: No Prompt (Default - Recommended)
|
||||
- [ ] **Setup:** Don't set prompt parameter
|
||||
- [ ] **Expected:** Uses default prompt, 88% accuracy, no warnings
|
||||
- [ ] **Verify:** Transcription quality is good
|
||||
|
||||
#### Test 5: Custom Prompt (Warning Test)
|
||||
- [ ] **Setup:** Set custom prompt in connection params
|
||||
- [ ] **Expected:** Warning logged about testing without prompt first
|
||||
- [ ] **Verify:** Check logs for prompt warning
|
||||
|
||||
#### Test 6: Prompt + Keyterms Conflict (Error Test)
|
||||
- [ ] **Setup:** Set both `prompt` and `keyterms_prompt` at init
|
||||
- [ ] **Expected:** ValueError raised with helpful error message
|
||||
- [ ] **Verify:** Service fails to initialize with clear error
|
||||
|
||||
---
|
||||
|
||||
### ✅ Keyterms Prompting Tests
|
||||
|
||||
#### Test 7: Basic Keyterms at Init
|
||||
- [ ] **Setup:** Set `keyterms_prompt=["Pipecat", "AssemblyAI", "Universal-3"]`
|
||||
- [ ] **Expected:** Terms are boosted in recognition
|
||||
- [ ] **Verify:** Say the boosted terms, check accuracy
|
||||
|
||||
#### Test 8: Empty Keyterms (No Boosting)
|
||||
- [ ] **Setup:** Set `keyterms_prompt=[]`
|
||||
- [ ] **Expected:** No boosting, default behavior
|
||||
- [ ] **Verify:** Normal transcription
|
||||
|
||||
---
|
||||
|
||||
### ✅ Diarization Tests
|
||||
|
||||
#### Test 9: Diarization Disabled (Default)
|
||||
- [ ] **Setup:** Don't set `speaker_labels` parameter
|
||||
- [ ] **Expected:** No speaker info in transcripts
|
||||
- [ ] **Verify:** TranscriptionFrame.user_id is default user_id
|
||||
|
||||
#### Test 10: Diarization Enabled (No Formatting)
|
||||
- [ ] **Setup:** Set `speaker_labels=True`
|
||||
- [ ] **Expected:** Speaker ID in user_id field, plain text
|
||||
- [ ] **Verify:** Multiple speakers show different IDs (Speaker A, Speaker B)
|
||||
|
||||
#### Test 11: Diarization with XML Formatting
|
||||
- [ ] **Setup:** Set `speaker_labels=True`, `speaker_format="<{speaker}>{text}</{speaker}>"`
|
||||
- [ ] **Expected:** Text includes speaker tags: `<Speaker A>Hello</Speaker A>`
|
||||
- [ ] **Verify:** Formatted text in transcript, speaker ID in user_id
|
||||
|
||||
#### Test 12: Diarization with Colon Prefix
|
||||
- [ ] **Setup:** Set `speaker_labels=True`, `speaker_format="{speaker}: {text}"`
|
||||
- [ ] **Expected:** Text includes prefix: `Speaker A: Hello`
|
||||
- [ ] **Verify:** Formatted text, multiple speakers distinguishable
|
||||
|
||||
---
|
||||
|
||||
### ✅ Dynamic Updates Tests
|
||||
|
||||
#### Test 13: Dynamic Keyterms Update (Stage 1 → Stage 2)
|
||||
- [ ] **Setup:** Start with empty keyterms, update mid-conversation
|
||||
- [ ] **Expected:** New keyterms take effect immediately
|
||||
- [ ] **Test Steps:**
|
||||
1. Start conversation with no keyterms
|
||||
2. Send update frame with `keyterms_prompt=["cardiology", "Dr. Smith"]`
|
||||
3. Say the new terms
|
||||
- [ ] **Verify:** Improved recognition after update
|
||||
|
||||
#### Test 14: Clear Keyterms (Reset Context)
|
||||
- [ ] **Setup:** Start with keyterms, clear them mid-stream
|
||||
- [ ] **Expected:** Context biasing removed
|
||||
- [ ] **Test Steps:**
|
||||
1. Start with `keyterms_prompt=["test", "words"]`
|
||||
2. Send update frame with `keyterms_prompt=[]`
|
||||
- [ ] **Verify:** No more boosting after clear
|
||||
|
||||
#### Test 15: Dynamic Silence Parameters
|
||||
- [ ] **Setup:** Update `max_turn_silence` mid-stream
|
||||
- [ ] **Expected:** Turn detection timing changes
|
||||
- [ ] **Test Steps:**
|
||||
1. Start with default (1200ms)
|
||||
2. Update to `max_turn_silence=5000` (for reading numbers)
|
||||
3. Pause longer between words
|
||||
4. Update back to `max_turn_silence=1200`
|
||||
- [ ] **Verify:** Longer pauses tolerated when increased
|
||||
|
||||
#### Test 16: Dynamic Prompt Update
|
||||
- [ ] **Setup:** Update prompt mid-stream
|
||||
- [ ] **Expected:** New instructions take effect
|
||||
- [ ] **Test Steps:**
|
||||
1. Start with default prompt
|
||||
2. Send update with custom prompt
|
||||
- [ ] **Verify:** Behavior changes according to new prompt
|
||||
|
||||
#### Test 17: Multiple Parameters at Once
|
||||
- [ ] **Setup:** Update keyterms, max_turn_silence, and min_end_of_turn together
|
||||
- [ ] **Expected:** All parameters updated in single WebSocket message
|
||||
- [ ] **Verify:** Check logs for single UpdateConfiguration message
|
||||
|
||||
#### Test 18: Dynamic Update - Prompt + Keyterms Conflict (Error)
|
||||
- [ ] **Setup:** Try to update both prompt and keyterms_prompt in same update
|
||||
- [ ] **Expected:** ValueError raised
|
||||
- [ ] **Verify:** Update fails with clear error message
|
||||
|
||||
---
|
||||
|
||||
### ✅ Turn Detection Mode Tests
|
||||
|
||||
#### Test 19: Pipecat Mode (vad_force_turn_endpoint=True) - Default
|
||||
- [ ] **Setup:** Use default settings (Pipecat mode)
|
||||
- [ ] **Expected:**
|
||||
- ForceEndpoint sent on VAD stop
|
||||
- Smart Turn Analyzer makes decisions
|
||||
- min=max=100ms for u3-rt-pro
|
||||
- [ ] **Verify:** Fast finals, Smart Turn handles completeness
|
||||
|
||||
#### Test 20: STT Mode (vad_force_turn_endpoint=False) - u3-rt-pro only
|
||||
- [ ] **Setup:** Set `vad_force_turn_endpoint=False` with u3-rt-pro
|
||||
- [ ] **Expected:**
|
||||
- AssemblyAI controls turn endings
|
||||
- SpeechStarted message triggers interruptions
|
||||
- UserStarted/StoppedSpeakingFrame emitted
|
||||
- [ ] **Verify:** Turn detection from AssemblyAI model
|
||||
|
||||
#### Test 21: STT Mode with universal-streaming (Error Test)
|
||||
- [ ] **Setup:** Set `vad_force_turn_endpoint=False` with universal-streaming
|
||||
- [ ] **Expected:** ValueError raised (requires u3-rt-pro)
|
||||
- [ ] **Verify:** Service fails with clear error
|
||||
|
||||
---
|
||||
|
||||
### ✅ Language Detection Tests (If Multilingual Model)
|
||||
|
||||
#### Test 22: Language Detection Enabled
|
||||
- [ ] **Setup:** Use `universal-streaming-multilingual` with `language_detection=True`
|
||||
- [ ] **Expected:** Language codes in transcripts
|
||||
- [ ] **Verify:** Speak different languages, check language_code field
|
||||
|
||||
#### Test 23: Language Confidence Threshold
|
||||
- [ ] **Setup:** Enable language detection
|
||||
- [ ] **Expected:** High confidence (≥0.7) → detected language, Low → fallback to English
|
||||
- [ ] **Verify:** Check logs for confidence warnings
|
||||
|
||||
---
|
||||
|
||||
### ✅ Edge Cases & Error Handling
|
||||
|
||||
#### Test 24: WebSocket Disconnect During Update
|
||||
- [ ] **Setup:** Simulate disconnect, try update
|
||||
- [ ] **Expected:** Error logged, update queued for reconnection
|
||||
- [ ] **Verify:** Graceful handling, no crash
|
||||
|
||||
#### Test 25: Invalid Parameter Types
|
||||
- [ ] **Setup:** Send update with wrong type (e.g., keyterms_prompt as string)
|
||||
- [ ] **Expected:** Warning logged, parameter skipped
|
||||
- [ ] **Verify:** Service continues, invalid param ignored
|
||||
|
||||
#### Test 26: Unknown Parameter in Update
|
||||
- [ ] **Setup:** Send update with unsupported parameter (e.g., `language`)
|
||||
- [ ] **Expected:** Warning logged about parameter
|
||||
- [ ] **Verify:** Other valid params still updated
|
||||
|
||||
---
|
||||
|
||||
### ✅ Integration Tests
|
||||
|
||||
#### Test 27: Full Voice Agent Flow (Multi-Stage)
|
||||
- [ ] **Setup:** Complete voice agent with stage transitions
|
||||
- [ ] **Test Steps:**
|
||||
1. Greeting stage (general keyterms)
|
||||
2. Name collection stage (name keyterms)
|
||||
3. Account number stage (number keyterms, longer silence)
|
||||
4. Medical info stage (medical keyterms)
|
||||
5. Closing stage (goodbye keyterms)
|
||||
- [ ] **Verify:** Each stage has appropriate keyterms and timing
|
||||
|
||||
#### Test 28: Diarization + Dynamic Updates
|
||||
- [ ] **Setup:** Enable diarization, update keyterms mid-stream
|
||||
- [ ] **Expected:** Both features work together
|
||||
- [ ] **Verify:** Speaker IDs persist, keyterms update correctly
|
||||
|
||||
#### Test 29: Interruption Handling
|
||||
- [ ] **Setup:** Bot speaking, user interrupts
|
||||
- [ ] **Expected:**
|
||||
- Pipecat mode: VAD + Smart Turn handles
|
||||
- STT mode: SpeechStarted triggers interrupt
|
||||
- [ ] **Verify:** Bot stops, user speech processed
|
||||
|
||||
---
|
||||
|
||||
## Testing Results Template
|
||||
|
||||
```
|
||||
| Test # | Feature | Status | Notes |
|
||||
|--------|---------|--------|-------|
|
||||
| 1 | Default Config | ✅ PASS | |
|
||||
| 2 | Custom min_silence | ✅ PASS | |
|
||||
| 3 | max_silence Warning | ✅ PASS | |
|
||||
| ... | ... | ... | ... |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Outcomes Summary
|
||||
|
||||
### ✅ Should Work (No Errors)
|
||||
- Default configuration
|
||||
- Custom min_turn_silence
|
||||
- Keyterms prompting
|
||||
- Diarization with/without formatting
|
||||
- Dynamic updates (one parameter or multiple)
|
||||
- Pipecat mode turn detection
|
||||
|
||||
### ⚠️ Should Warn (Logs Warning, Continues)
|
||||
- Custom prompt set at init
|
||||
- max_turn_silence set (overridden)
|
||||
- Invalid parameter types in updates
|
||||
- Language update attempted
|
||||
- Prompt used with universal-streaming
|
||||
|
||||
### ❌ Should Error (Raises Exception, Stops)
|
||||
- prompt + keyterms_prompt at init
|
||||
- prompt + keyterms_prompt in same update
|
||||
- vad_force_turn_endpoint=False with universal-streaming
|
||||
|
||||
---
|
||||
|
||||
## Quick Test Commands
|
||||
|
||||
```bash
|
||||
# Run basic test
|
||||
python test_assemblyai_u3pro.py --test basic
|
||||
|
||||
# Run specific test
|
||||
python test_assemblyai_u3pro.py --test diarization
|
||||
|
||||
# Run all tests
|
||||
python test_assemblyai_u3pro.py --test all
|
||||
|
||||
# Interactive mode
|
||||
python test_assemblyai_u3pro.py --interactive
|
||||
```
|
||||
310
TESTING_SETUP.md
310
TESTING_SETUP.md
@@ -1,310 +0,0 @@
|
||||
# AssemblyAI u3-rt-pro Testing Setup Guide
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Setup Environment
|
||||
|
||||
```bash
|
||||
# Copy API keys
|
||||
cp .env.testing .env
|
||||
|
||||
# Install dependencies
|
||||
uv sync --group dev --all-extras --no-extra gstreamer --no-extra krisp
|
||||
|
||||
# Make test script executable
|
||||
chmod +x test_assemblyai_u3pro.py
|
||||
```
|
||||
|
||||
### 2. Ensure Audio Devices
|
||||
|
||||
Make sure you have:
|
||||
- **Microphone** enabled and working
|
||||
- **Speakers/headphones** connected
|
||||
- Audio permissions granted (macOS will prompt on first run)
|
||||
|
||||
### 3. Run Tests
|
||||
|
||||
```bash
|
||||
# Run a specific test
|
||||
python test_assemblyai_u3pro.py --test basic
|
||||
|
||||
# Interactive mode (choose from menu)
|
||||
python test_assemblyai_u3pro.py --interactive
|
||||
|
||||
# Run all tests sequentially
|
||||
python test_assemblyai_u3pro.py --test all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Available Tests
|
||||
|
||||
### Basic Configuration Tests
|
||||
```bash
|
||||
# Test 1: Default configuration (min=max=100ms)
|
||||
python test_assemblyai_u3pro.py --test basic
|
||||
|
||||
# Test 2: Custom min_turn_silence
|
||||
python test_assemblyai_u3pro.py --test custom_min
|
||||
|
||||
# Test 3: max_turn_silence warning (should be overridden)
|
||||
python test_assemblyai_u3pro.py --test max_warning
|
||||
```
|
||||
|
||||
### Prompting Tests
|
||||
```bash
|
||||
# Test 5: Custom prompt warning
|
||||
python test_assemblyai_u3pro.py --test prompt_warning
|
||||
|
||||
# Test 6: Prompt + keyterms conflict (should error)
|
||||
python test_assemblyai_u3pro.py --test prompt_keyterms_conflict
|
||||
|
||||
# Test 7: Basic keyterms prompting
|
||||
python test_assemblyai_u3pro.py --test keyterms
|
||||
```
|
||||
|
||||
### Diarization Tests
|
||||
```bash
|
||||
# Test 10: Diarization without formatting
|
||||
python test_assemblyai_u3pro.py --test diarization
|
||||
|
||||
# Test 11: Diarization with XML formatting
|
||||
python test_assemblyai_u3pro.py --test diarization_xml
|
||||
```
|
||||
|
||||
### Dynamic Updates Tests
|
||||
```bash
|
||||
# Test 13: Dynamic keyterms (multi-stage)
|
||||
python test_assemblyai_u3pro.py --test dynamic_keyterms
|
||||
|
||||
# Test 15: Dynamic silence parameters
|
||||
python test_assemblyai_u3pro.py --test dynamic_silence
|
||||
|
||||
# Test 17: Multiple parameters at once
|
||||
python test_assemblyai_u3pro.py --test multi_param
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Execution Flow
|
||||
|
||||
### For Each Test:
|
||||
|
||||
1. **Start the test script**
|
||||
```bash
|
||||
python test_assemblyai_u3pro.py --test <test_name>
|
||||
```
|
||||
|
||||
2. **Wait for "started" message** indicating the bot is ready
|
||||
|
||||
3. **Speak into your microphone** to test - the bot will:
|
||||
- Transcribe your speech (you'll see `📝 TRANSCRIPTION:` logs)
|
||||
- Process through the LLM
|
||||
- Respond with voice through your speakers
|
||||
|
||||
4. **Observe logs** for:
|
||||
- ✅ Success indicators
|
||||
- ⚠️ Warning messages
|
||||
- ❌ Error messages
|
||||
- 📝 Transcription output
|
||||
|
||||
5. **Verify expected behavior** against checklist
|
||||
|
||||
6. **Stop test** with Ctrl+C
|
||||
|
||||
---
|
||||
|
||||
## Expected Test Outcomes
|
||||
|
||||
### Should Pass (✅)
|
||||
- Basic configuration creates service
|
||||
- Custom parameters are applied
|
||||
- Keyterms boost recognition
|
||||
- Diarization shows speaker IDs
|
||||
- Dynamic updates work without errors
|
||||
|
||||
### Should Warn (⚠️)
|
||||
Check logs for warnings:
|
||||
- "We recommend testing at first with no prompt"
|
||||
- "max_turn_silence is not used in Pipecat mode"
|
||||
- "Unknown setting for AssemblyAI STT service"
|
||||
|
||||
### Should Error (❌)
|
||||
Should raise ValueError and fail to start:
|
||||
- Both prompt and keyterms_prompt set at init
|
||||
- Both prompt and keyterms_prompt in same update
|
||||
- vad_force_turn_endpoint=False with universal-streaming
|
||||
|
||||
---
|
||||
|
||||
## Debugging Tips
|
||||
|
||||
### Check Logs
|
||||
```bash
|
||||
# Run with verbose logging
|
||||
LOGURU_LEVEL=DEBUG python test_assemblyai_u3pro.py --test <test_name>
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Issue: "WebSocket connection failed"**
|
||||
- Check ASSEMBLYAI_API_KEY is correct
|
||||
- Verify network connection
|
||||
- Check firewall settings
|
||||
|
||||
**Issue: "No audio input/output"**
|
||||
- Verify microphone permissions (System Preferences → Security & Privacy → Microphone)
|
||||
- Check default audio devices in System Preferences → Sound
|
||||
- Test microphone with another app first
|
||||
- Make sure no other app is using the microphone
|
||||
|
||||
**Issue: "No transcriptions appearing"**
|
||||
- Verify microphone permissions
|
||||
- Check audio levels (speak louder or move closer to mic)
|
||||
- Speak clearly and wait for VAD to detect
|
||||
- Check if microphone is muted
|
||||
|
||||
**Issue: "Can't hear bot responses"**
|
||||
- Check speaker/headphone volume
|
||||
- Verify correct output device is selected
|
||||
- Check terminal for TTS errors
|
||||
|
||||
**Issue: "Service fails to start"**
|
||||
- Check all API keys in .env
|
||||
- Run `uv sync` to ensure dependencies installed
|
||||
- Check Python version (3.10+)
|
||||
|
||||
---
|
||||
|
||||
## Manual Testing Checklist
|
||||
|
||||
After running automated tests, manually verify:
|
||||
|
||||
### ✅ Audio Quality
|
||||
- [ ] Transcriptions are accurate
|
||||
- [ ] No distortion or dropouts
|
||||
- [ ] Latency is acceptable
|
||||
|
||||
### ✅ Turn Detection
|
||||
- [ ] Bot waits for user to finish speaking
|
||||
- [ ] No premature cutoffs
|
||||
- [ ] Handles natural pauses correctly
|
||||
|
||||
### ✅ Interruptions
|
||||
- [ ] Can interrupt bot mid-sentence
|
||||
- [ ] Interruption is smooth
|
||||
- [ ] Bot stops speaking immediately
|
||||
|
||||
### ✅ Diarization (if enabled)
|
||||
- [ ] Multiple speakers detected correctly
|
||||
- [ ] Speaker IDs consistent
|
||||
- [ ] Speaker formatting works
|
||||
|
||||
### ✅ Dynamic Updates
|
||||
- [ ] Keyterms update without disconnection
|
||||
- [ ] Turn detection timing changes work
|
||||
- [ ] Updates logged correctly
|
||||
|
||||
---
|
||||
|
||||
## Test Results Recording
|
||||
|
||||
### Use this template:
|
||||
|
||||
```markdown
|
||||
## Test Run: YYYY-MM-DD
|
||||
|
||||
| Test # | Test Name | Status | Notes |
|
||||
|--------|-----------|--------|-------|
|
||||
| 1 | basic | ✅ PASS | Transcriptions working |
|
||||
| 2 | custom_min | ✅ PASS | Turn timing changed |
|
||||
| 3 | max_warning | ✅ PASS | Warning logged |
|
||||
| 5 | prompt_warning | ✅ PASS | Warning shown |
|
||||
| 6 | prompt_keyterms_conflict | ✅ PASS | ValueError raised |
|
||||
| 7 | keyterms | ✅ PASS | Terms boosted |
|
||||
| 10 | diarization | ✅ PASS | Speaker IDs correct |
|
||||
| 11 | diarization_xml | ✅ PASS | XML tags shown |
|
||||
| 13 | dynamic_keyterms | ✅ PASS | Updates worked |
|
||||
| 15 | dynamic_silence | ✅ PASS | Timing adjusted |
|
||||
| 17 | multi_param | ✅ PASS | All params updated |
|
||||
|
||||
### Issues Found:
|
||||
- None
|
||||
|
||||
### Notes:
|
||||
- All tests passed successfully
|
||||
- Latency is excellent (sub-300ms)
|
||||
- Diarization accuracy is good
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Testing
|
||||
|
||||
### Custom Test Scenarios
|
||||
|
||||
Create custom tests by modifying `test_assemblyai_u3pro.py`:
|
||||
|
||||
```python
|
||||
async def test_my_custom_scenario():
|
||||
"""My custom test scenario."""
|
||||
logger.info("Testing my specific use case")
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
# Your custom params here
|
||||
)
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
# Your test logic here
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
```
|
||||
|
||||
### Stress Testing
|
||||
|
||||
Test with:
|
||||
- Multiple simultaneous speakers
|
||||
- Long conversations (30+ minutes)
|
||||
- Rapid speech
|
||||
- Heavy accents
|
||||
- Background noise
|
||||
- Poor network conditions
|
||||
|
||||
---
|
||||
|
||||
## Reporting Issues
|
||||
|
||||
When reporting issues, include:
|
||||
|
||||
1. **Test name and number**
|
||||
2. **Full error message and stack trace**
|
||||
3. **Relevant log output** (use LOGURU_LEVEL=DEBUG)
|
||||
4. **Configuration used** (connection_params)
|
||||
5. **Expected vs actual behavior**
|
||||
6. **Steps to reproduce**
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
After testing:
|
||||
|
||||
1. ✅ Mark completed tests in `TESTING_CHECKLIST.md`
|
||||
2. 📝 Document any issues found
|
||||
3. 🐛 Create GitHub issues for bugs
|
||||
4. ✨ Suggest improvements
|
||||
5. 📊 Share results with team
|
||||
|
||||
---
|
||||
|
||||
## Contact
|
||||
|
||||
Questions? Issues?
|
||||
- Check `TESTING_CHECKLIST.md` for detailed test descriptions
|
||||
- Review logs with `LOGURU_LEVEL=DEBUG`
|
||||
- Reach out to the team with your findings
|
||||
|
||||
Happy testing! 🎯
|
||||
@@ -1,240 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Custom AssemblyAI u3-rt-pro Test Script
|
||||
Easy parameter tweaking for experimentation
|
||||
|
||||
Edit the CONFIGURATION section below to test different settings!
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
|
||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||
from pipecat.frames.frames import LLMRunFrame
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.pipeline.runner import PipelineRunner
|
||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||
from pipecat.processors.aggregators.llm_response_universal import (
|
||||
LLMContextAggregatorPair,
|
||||
LLMUserAggregatorParams,
|
||||
)
|
||||
from pipecat.services.assemblyai.models import AssemblyAIConnectionParams
|
||||
from pipecat.services.assemblyai.stt import AssemblyAISTTService
|
||||
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||
from pipecat.services.openai.llm import OpenAILLMService
|
||||
from pipecat.transports.local.audio import LocalAudioTransport, LocalAudioTransportParams
|
||||
|
||||
load_dotenv(override=True)
|
||||
|
||||
# ============================================================================
|
||||
# CONFIGURATION
|
||||
# ============================================================================
|
||||
|
||||
# Log Level: "DEBUG" for detailed logs, "INFO" for normal operation
|
||||
LOG_LEVEL = "INFO"
|
||||
|
||||
# ============================================================================
|
||||
# BOT IMPLEMENTATION
|
||||
# ============================================================================
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run the custom test bot with your configured parameters."""
|
||||
# Setup logging
|
||||
logger.remove(0)
|
||||
logger.add(sys.stderr, level=LOG_LEVEL)
|
||||
|
||||
logger.info("=" * 80)
|
||||
logger.info("AssemblyAI u3-rt-pro Custom Test")
|
||||
logger.info("=" * 80)
|
||||
logger.info("Starting bot... Speak after you hear the greeting!")
|
||||
logger.info("=" * 80)
|
||||
|
||||
# Create local audio transport
|
||||
transport = LocalAudioTransport(
|
||||
LocalAudioTransportParams(
|
||||
audio_in_enabled=True,
|
||||
audio_out_enabled=True,
|
||||
)
|
||||
)
|
||||
|
||||
# ========================================================================
|
||||
# EDIT PARAMETERS HERE
|
||||
# ========================================================================
|
||||
|
||||
# Build connection params
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
# ====================================================================
|
||||
# Model Selection
|
||||
# ====================================================================
|
||||
speech_model="u3-rt-pro",
|
||||
# speech_model="universal-streaming-english",
|
||||
# speech_model="universal-streaming-multilingual",
|
||||
# ====================================================================
|
||||
# Turn Detection Timing
|
||||
# ====================================================================
|
||||
# Minimum silence when confident about end of turn (milliseconds)
|
||||
# Default: 100ms | Higher = more patient | Lower = faster responses
|
||||
# Only used in Pipecat mode (vad_force_turn_endpoint=True)
|
||||
min_turn_silence=100000,
|
||||
# min_turn_silence=200,
|
||||
# min_turn_silence=300,
|
||||
# Maximum turn silence (milliseconds)
|
||||
# WARNING: In Pipecat mode (vad_force_turn_endpoint=True), this is
|
||||
# automatically set equal to min_turn_silence
|
||||
# to avoid double turn detection. Only used as-is in STT mode.
|
||||
max_turn_silence=500,
|
||||
# End of turn confidence threshold (0.0 to 1.0)
|
||||
# Higher = requires more confidence before ending turn
|
||||
# end_of_turn_confidence_threshold=0.8,
|
||||
# ====================================================================
|
||||
# Prompting & Boosting
|
||||
# ====================================================================
|
||||
# Custom Prompt (WARNING: test carefully, default is optimized!)
|
||||
# None = Use AssemblyAI's optimized default (recommended for 88% accuracy)
|
||||
prompt=None,
|
||||
# prompt="Transcribe speech with focus on technical terms.",
|
||||
# prompt="Context: Medical conversation. Transcribe accurately.",
|
||||
# Keyterms Prompting (boosts recognition for specific words)
|
||||
# NOTE: Cannot use both prompt and keyterms_prompt!
|
||||
keyterms_prompt=None,
|
||||
# keyterms_prompt=["Pipecat", "AssemblyAI", "OpenAI", "Cartesia"],
|
||||
# keyterms_prompt=["Python", "JavaScript", "TypeScript", "API"],
|
||||
# ====================================================================
|
||||
# Diarization (Speaker Identification)
|
||||
# ====================================================================
|
||||
# Enable speaker labels (identifies different speakers)
|
||||
speaker_labels=None, # None or True
|
||||
# speaker_labels=True,
|
||||
# ====================================================================
|
||||
# Audio Configuration
|
||||
# ====================================================================
|
||||
# Audio sample rate (Hz)
|
||||
# sample_rate=16000,
|
||||
# sample_rate=8000,
|
||||
# Audio encoding format
|
||||
# encoding="pcm_s16le", # Default: 16-bit PCM
|
||||
# encoding="pcm_mulaw", # μ-law encoding (telephony)
|
||||
# ====================================================================
|
||||
# Other Options
|
||||
# ====================================================================
|
||||
# Format transcript turns (applies formatting rules)
|
||||
# format_turns=True, # Default
|
||||
# format_turns=False,
|
||||
# Language detection (only for universal-streaming-multilingual)
|
||||
# language_detection=True,
|
||||
)
|
||||
|
||||
# Log connection parameters for debugging
|
||||
logger.info("=" * 80)
|
||||
logger.info("CONNECTION PARAMETERS:")
|
||||
logger.info(f" speech_model: {connection_params.speech_model}")
|
||||
logger.info(f" min_turn_silence: {connection_params.min_turn_silence}")
|
||||
logger.info(f" max_turn_silence: {connection_params.max_turn_silence}")
|
||||
logger.info(f" sample_rate: {connection_params.sample_rate}")
|
||||
logger.info(f" encoding: {connection_params.encoding}")
|
||||
logger.info(f" prompt: {connection_params.prompt}")
|
||||
logger.info(f" keyterms_prompt: {connection_params.keyterms_prompt}")
|
||||
logger.info(f" speaker_labels: {connection_params.speaker_labels}")
|
||||
logger.info(f" format_turns: {connection_params.format_turns}")
|
||||
logger.info(
|
||||
f" end_of_turn_confidence_threshold: {connection_params.end_of_turn_confidence_threshold}"
|
||||
)
|
||||
logger.info(f" language_detection: {connection_params.language_detection}")
|
||||
logger.info("=" * 80)
|
||||
|
||||
# AssemblyAI Speech-to-Text Service
|
||||
stt = AssemblyAISTTService(
|
||||
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
|
||||
connection_params=connection_params,
|
||||
# Turn Detection Mode
|
||||
# True = Pipecat mode (VAD + Smart Turn controls turns)
|
||||
# False = STT mode (u3-rt-pro model controls turns)
|
||||
vad_force_turn_endpoint=True,
|
||||
# Speaker Formatting (only used if speaker_labels=True)
|
||||
# None = Just log speaker IDs, don't modify transcript
|
||||
speaker_format=None,
|
||||
# speaker_format="<Speaker {speaker}>{text}</Speaker {speaker}>",
|
||||
# speaker_format="{speaker}: {text}",
|
||||
# speaker_format="[{speaker}] {text}",
|
||||
# Additional available parameters (uncomment to use):
|
||||
# should_interrupt=True, # Only for STT mode
|
||||
)
|
||||
|
||||
# ========================================================================
|
||||
|
||||
# Text-to-Speech
|
||||
tts = CartesiaTTSService(
|
||||
api_key=os.getenv("CARTESIA_API_KEY"),
|
||||
voice_id="a0e99841-438c-4a64-b679-ae501e7d6091", # Conversational English
|
||||
)
|
||||
|
||||
# LLM
|
||||
llm = OpenAILLMService(
|
||||
api_key=os.getenv("OPENAI_API_KEY"),
|
||||
model="gpt-4",
|
||||
)
|
||||
|
||||
# Conversation context
|
||||
messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a helpful voice assistant testing the AssemblyAI u3-rt-pro model. "
|
||||
"Keep responses very brief (1-2 sentences). "
|
||||
"Start by introducing yourself briefly and asking the user to speak."
|
||||
),
|
||||
},
|
||||
]
|
||||
|
||||
context = LLMContext(messages)
|
||||
|
||||
# Configure aggregator based on mode
|
||||
# In STT mode, don't use VAD (model handles turn detection)
|
||||
# In Pipecat mode, use VAD + Smart Turn
|
||||
vad_force_turn_endpoint = True # Must match the value in stt configuration above
|
||||
user_params = None
|
||||
if vad_force_turn_endpoint:
|
||||
user_params = LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer())
|
||||
|
||||
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
||||
context,
|
||||
user_params=user_params,
|
||||
)
|
||||
|
||||
# Pipeline
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
transport.input(),
|
||||
stt,
|
||||
user_aggregator,
|
||||
llm,
|
||||
tts,
|
||||
transport.output(),
|
||||
assistant_aggregator,
|
||||
]
|
||||
)
|
||||
|
||||
# Task
|
||||
task = PipelineTask(
|
||||
pipeline,
|
||||
params=PipelineParams(
|
||||
enable_metrics=True,
|
||||
enable_usage_metrics=True,
|
||||
),
|
||||
)
|
||||
|
||||
# Start the conversation
|
||||
await task.queue_frames([LLMRunFrame()])
|
||||
|
||||
# Run
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -1,749 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Interactive AssemblyAI u3-rt-pro Comprehensive Test Suite
|
||||
|
||||
Tests all features with detailed scenarios:
|
||||
- Basic configuration variations
|
||||
- Prompting and keyterms with difficult names
|
||||
- Diarization
|
||||
- Dynamic parameter updates (single and multiple)
|
||||
- Mode comparisons
|
||||
- STT mode timing experiments (testing silence parameters)
|
||||
- Edge cases
|
||||
|
||||
Usage:
|
||||
python test_assemblyai_interactive.py
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
|
||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||
from pipecat.frames.frames import LLMRunFrame, STTUpdateSettingsFrame
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.pipeline.runner import PipelineRunner
|
||||
from pipecat.pipeline.task import PipelineParams, PipelineTask
|
||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||
from pipecat.processors.aggregators.llm_response_universal import (
|
||||
LLMContextAggregatorPair,
|
||||
LLMUserAggregatorParams,
|
||||
)
|
||||
from pipecat.services.assemblyai.models import AssemblyAIConnectionParams
|
||||
from pipecat.services.assemblyai.stt import AssemblyAISTTService, AssemblyAISTTSettings
|
||||
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||
from pipecat.services.openai.llm import OpenAILLMService
|
||||
from pipecat.transports.local.audio import LocalAudioTransport, LocalAudioTransportParams
|
||||
|
||||
load_dotenv(override=True)
|
||||
|
||||
logger.remove(0)
|
||||
logger.add(sys.stderr, level="INFO")
|
||||
|
||||
|
||||
async def run_bot(
|
||||
connection_params: AssemblyAIConnectionParams,
|
||||
test_name: str,
|
||||
vad_force_turn_endpoint: bool = True,
|
||||
speaker_format: Optional[str] = None,
|
||||
test_dynamic_updates: Optional[callable] = None,
|
||||
):
|
||||
"""Run the voice bot with specified configuration."""
|
||||
logger.info("=" * 80)
|
||||
logger.info(f"TEST: {test_name}")
|
||||
logger.info("=" * 80)
|
||||
logger.info("Starting bot... Speak into your microphone after you hear the greeting!")
|
||||
logger.info("=" * 80)
|
||||
|
||||
# Create local audio transport
|
||||
transport = LocalAudioTransport(
|
||||
LocalAudioTransportParams(
|
||||
audio_in_enabled=True,
|
||||
audio_out_enabled=True,
|
||||
)
|
||||
)
|
||||
|
||||
# AssemblyAI Speech-to-Text
|
||||
stt = AssemblyAISTTService(
|
||||
api_key=os.getenv("ASSEMBLYAI_API_KEY"),
|
||||
connection_params=connection_params,
|
||||
vad_force_turn_endpoint=vad_force_turn_endpoint,
|
||||
speaker_format=speaker_format,
|
||||
)
|
||||
|
||||
# Text-to-Speech
|
||||
tts = CartesiaTTSService(
|
||||
api_key=os.getenv("CARTESIA_API_KEY"),
|
||||
voice_id="a0e99841-438c-4a64-b679-ae501e7d6091",
|
||||
)
|
||||
|
||||
# LLM
|
||||
llm = OpenAILLMService(
|
||||
api_key=os.getenv("OPENAI_API_KEY"),
|
||||
model="gpt-4",
|
||||
)
|
||||
|
||||
# Conversation context
|
||||
messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a helpful voice assistant testing the AssemblyAI u3-rt-pro model. "
|
||||
"Keep responses very brief (1-2 sentences). "
|
||||
"Start by introducing yourself briefly and asking the user to speak."
|
||||
),
|
||||
},
|
||||
]
|
||||
|
||||
context = LLMContext(messages)
|
||||
|
||||
# Configure aggregator based on mode
|
||||
user_params = None
|
||||
if vad_force_turn_endpoint:
|
||||
user_params = LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer())
|
||||
|
||||
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
||||
context,
|
||||
user_params=user_params,
|
||||
)
|
||||
|
||||
# Pipeline
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
transport.input(),
|
||||
stt,
|
||||
user_aggregator,
|
||||
llm,
|
||||
tts,
|
||||
transport.output(),
|
||||
assistant_aggregator,
|
||||
]
|
||||
)
|
||||
|
||||
# Task
|
||||
task = PipelineTask(
|
||||
pipeline,
|
||||
params=PipelineParams(
|
||||
enable_metrics=True,
|
||||
enable_usage_metrics=True,
|
||||
),
|
||||
)
|
||||
|
||||
# Handle dynamic updates if provided
|
||||
if test_dynamic_updates:
|
||||
asyncio.create_task(test_dynamic_updates(task))
|
||||
|
||||
# Start the conversation
|
||||
await task.queue_frames([LLMRunFrame()])
|
||||
|
||||
# Run
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test Configurations
|
||||
# ============================================================================
|
||||
|
||||
# === BASIC CONFIGURATION (1-3) ===
|
||||
|
||||
|
||||
async def test_01_basic_100ms():
|
||||
"""Test 1: Basic default configuration (100ms)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100,
|
||||
)
|
||||
await run_bot(connection_params, "Basic Default Configuration (100ms)")
|
||||
|
||||
|
||||
async def test_02_custom_200ms():
|
||||
"""Test 2: Custom min_end_of_turn_silence (200ms)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=200,
|
||||
)
|
||||
await run_bot(connection_params, "Custom Turn Silence (200ms)")
|
||||
|
||||
|
||||
async def test_03_custom_500ms():
|
||||
"""Test 3: Longer silence threshold (500ms)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=500,
|
||||
)
|
||||
await run_bot(connection_params, "Longer Turn Silence (500ms)")
|
||||
|
||||
|
||||
# === PROMPTING & WARNINGS (4-7) ===
|
||||
|
||||
|
||||
async def test_04_max_warning():
|
||||
"""Test 4: max_turn_silence warning (should be overridden)."""
|
||||
logger.warning("⚠️ EXPECT WARNING: max_turn_silence will be overridden")
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
max_turn_silence=500,
|
||||
)
|
||||
await run_bot(connection_params, "max_turn_silence Override Warning")
|
||||
|
||||
|
||||
async def test_05_prompt_warning():
|
||||
"""Test 5: Custom prompt warning."""
|
||||
logger.warning("⚠️ EXPECT WARNING: Custom prompts should be tested carefully")
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
prompt="Transcribe speech accurately with proper punctuation.",
|
||||
)
|
||||
await run_bot(connection_params, "Custom Prompt Warning Test")
|
||||
|
||||
|
||||
async def test_06_prompt_keyterms_conflict():
|
||||
"""Test 6: Prompt + keyterms conflict (should error)."""
|
||||
logger.error("❌ EXPECT ERROR: Cannot use both prompt and keyterms_prompt")
|
||||
try:
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
prompt="Custom prompt",
|
||||
keyterms_prompt=["test"],
|
||||
)
|
||||
await run_bot(connection_params, "Prompt + Keyterms Conflict (ERROR)")
|
||||
except ValueError as e:
|
||||
logger.error(f"✅ EXPECTED ERROR: {e}")
|
||||
input("\nPress Enter to continue...")
|
||||
return
|
||||
|
||||
|
||||
async def test_07_keyterms_difficult():
|
||||
"""Test 7: Keyterms with difficult/unusual names."""
|
||||
# Use names that STT wouldn't normally get right
|
||||
keyterms = ["Xiomara", "Saoirse", "Krzystof", "Nguyen", "Pipecat", "AssemblyAI"]
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
keyterms_prompt=keyterms,
|
||||
)
|
||||
logger.info("🎯 Boosted terms: Xiomara, Saoirse, Krzystof, Nguyen, Pipecat, AssemblyAI")
|
||||
logger.info(" Try saying these difficult names to test boosting!")
|
||||
await run_bot(connection_params, "Keyterms with Difficult Names")
|
||||
|
||||
|
||||
# === DIARIZATION (8-9) ===
|
||||
|
||||
|
||||
async def test_08_diarization_basic():
|
||||
"""Test 8: Basic diarization (speaker IDs logged)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
speaker_labels=True,
|
||||
)
|
||||
logger.info("🎤 Diarization enabled - speaker IDs will be logged")
|
||||
logger.info(" Try having multiple people speak!")
|
||||
await run_bot(connection_params, "Diarization - Basic")
|
||||
|
||||
|
||||
async def test_09_diarization_xml():
|
||||
"""Test 9: Diarization with XML formatting."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
speaker_labels=True,
|
||||
)
|
||||
logger.info("🎤 Diarization with XML tags")
|
||||
logger.info(" Transcripts will include <Speaker X>text</Speaker X>")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Diarization - XML Formatting",
|
||||
speaker_format="<Speaker {speaker}>{text}</Speaker {speaker}>",
|
||||
)
|
||||
|
||||
|
||||
# === DYNAMIC UPDATES - SINGLE PARAMETER (10-13) ===
|
||||
|
||||
|
||||
async def test_10_dynamic_keyterms():
|
||||
"""Test 10: Dynamic keyterms update with difficult names."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
)
|
||||
|
||||
async def dynamic_update(task):
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("PHASE 1: No keyterms boosting")
|
||||
logger.info(" Try saying: Xiomara, Saoirse, Krzystof")
|
||||
logger.info(" (May not transcribe correctly)")
|
||||
logger.info("=" * 80)
|
||||
await asyncio.sleep(15)
|
||||
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("🔄 UPDATING: Adding keyterms boost")
|
||||
logger.info("=" * 80)
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(
|
||||
keyterms_prompt=["Xiomara", "Saoirse", "Krzystof", "Nguyen"]
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("PHASE 2: Keyterms NOW boosted")
|
||||
logger.info(" Say the same names again: Xiomara, Saoirse, Krzystof")
|
||||
logger.info(" (Should transcribe better now!)")
|
||||
logger.info("=" * 80)
|
||||
|
||||
logger.info("🔄 This test has 2 phases:")
|
||||
logger.info(" Phase 1 (15s): No boosting - names may be wrong")
|
||||
logger.info(" Phase 2: Keyterms added - names should improve")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Dynamic Keyterms Update (Before/After)",
|
||||
test_dynamic_updates=dynamic_update,
|
||||
)
|
||||
|
||||
|
||||
async def test_11_dynamic_silence():
|
||||
"""Test 11: Dynamic silence parameter update (dramatic change)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100,
|
||||
)
|
||||
|
||||
async def dynamic_update(task):
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("PHASE 1: Quick responses (100ms silence threshold)")
|
||||
logger.info(" Speak normally - bot responds quickly")
|
||||
logger.info("=" * 80)
|
||||
await asyncio.sleep(10)
|
||||
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("🔄 UPDATING: Changing silence from 100ms → 3000ms (3 seconds!)")
|
||||
logger.info("=" * 80)
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(min_turn_silence=3000)
|
||||
)
|
||||
)
|
||||
)
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("PHASE 2: Patient responses (3 second silence threshold)")
|
||||
logger.info(" Bot will wait 3 full seconds before responding")
|
||||
logger.info(" Try pausing mid-sentence - bot should NOT interrupt")
|
||||
logger.info("=" * 80)
|
||||
|
||||
logger.info("🔄 Dramatic change: 100ms → 3000ms after 10 seconds")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Dynamic Silence Update (100ms → 3s)",
|
||||
test_dynamic_updates=dynamic_update,
|
||||
)
|
||||
|
||||
|
||||
async def test_12_dynamic_prompt():
|
||||
"""Test 12: Dynamic prompt update with keyterms in prompt."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
)
|
||||
|
||||
async def dynamic_update(task):
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("PHASE 1: Default prompt (no keyterms)")
|
||||
logger.info(" Try saying: Xiomara, Saoirse, Krzystof")
|
||||
logger.info(" (May not transcribe correctly)")
|
||||
logger.info("=" * 80)
|
||||
await asyncio.sleep(15)
|
||||
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("🔄 UPDATING: Adding custom prompt with keyterms")
|
||||
logger.info("=" * 80)
|
||||
custom_prompt = """Transcribe verbatim. Rules:
|
||||
1) Always include punctuation in output.
|
||||
2) Use period/question mark ONLY for complete sentences.
|
||||
3) Use comma for mid-sentence pauses.
|
||||
4) Use no punctuation for incomplete trailing speech.
|
||||
5) Filler words (um, uh, so, like) indicate speaker will continue.
|
||||
|
||||
Pay special attention to these names and transcribe them exactly: Xiomara, Saoirse, Krzystof, Nguyen."""
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(prompt=custom_prompt)
|
||||
)
|
||||
)
|
||||
)
|
||||
logger.info("\n" + "=" * 80)
|
||||
logger.info("PHASE 2: Prompt with keyterms NOW active")
|
||||
logger.info(" Say the same names again: Xiomara, Saoirse, Krzystof")
|
||||
logger.info(" (Should transcribe better now!)")
|
||||
logger.info("=" * 80)
|
||||
|
||||
logger.info("🔄 This test has 2 phases:")
|
||||
logger.info(" Phase 1 (15s): Default prompt - names may be wrong")
|
||||
logger.info(" Phase 2: Custom prompt with keyterms - names should improve")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Dynamic Prompt Update (with keyterms)",
|
||||
test_dynamic_updates=dynamic_update,
|
||||
)
|
||||
|
||||
|
||||
async def test_13_dynamic_clear_keyterms():
|
||||
"""Test 13: Clear keyterms dynamically."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
keyterms_prompt=["Pipecat", "AssemblyAI"],
|
||||
)
|
||||
|
||||
async def dynamic_update(task):
|
||||
await asyncio.sleep(10)
|
||||
logger.info("🔄 UPDATING: Clearing keyterms (empty array)")
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(keyterms_prompt=[])
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
logger.info("🎯 Initial: Pipecat, AssemblyAI boosted")
|
||||
logger.info("🔄 After 10s: Keyterms will be cleared")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Dynamic Clear Keyterms",
|
||||
test_dynamic_updates=dynamic_update,
|
||||
)
|
||||
|
||||
|
||||
# === DYNAMIC UPDATES - MULTIPLE PARAMETERS (14-15) ===
|
||||
|
||||
|
||||
async def test_14_multi_param_update():
|
||||
"""Test 14: Update multiple parameters at once."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100,
|
||||
)
|
||||
|
||||
async def dynamic_update(task):
|
||||
await asyncio.sleep(10)
|
||||
logger.info("🔄 UPDATING MULTIPLE: keyterms + silence")
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(
|
||||
keyterms_prompt=["Xiomara", "Pipecat"],
|
||||
min_turn_silence=250,
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
logger.info("🔄 After 10s: Will update BOTH keyterms AND silence threshold")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Multiple Parameter Update",
|
||||
test_dynamic_updates=dynamic_update,
|
||||
)
|
||||
|
||||
|
||||
async def test_15_complex_sequence():
|
||||
"""Test 15: Complex multi-stage update sequence."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
)
|
||||
|
||||
async def dynamic_update(task):
|
||||
logger.info("Stage 1: Initial (10s)")
|
||||
await asyncio.sleep(10)
|
||||
|
||||
logger.info("🔄 Stage 2: Add keyterms")
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(keyterms_prompt=["Pipecat"])
|
||||
)
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(10)
|
||||
|
||||
logger.info("🔄 Stage 3: Change silence")
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(min_turn_silence=200)
|
||||
)
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(10)
|
||||
|
||||
logger.info("🔄 Stage 4: Update both")
|
||||
await task.queue_frame(
|
||||
STTUpdateSettingsFrame(
|
||||
delta=AssemblyAISTTSettings(
|
||||
connection_params=AssemblyAIConnectionParams(
|
||||
keyterms_prompt=["AssemblyAI", "OpenAI"],
|
||||
min_turn_silence=150,
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
logger.info("🔄 Multi-stage: 4 configuration changes over 30 seconds")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Complex Update Sequence (4 stages)",
|
||||
test_dynamic_updates=dynamic_update,
|
||||
)
|
||||
|
||||
|
||||
# === MODE COMPARISON (16-17) ===
|
||||
|
||||
|
||||
async def test_16_pipecat_mode():
|
||||
"""Test 16: Pipecat mode (VAD + Smart Turn controls turns)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100,
|
||||
)
|
||||
logger.info("🎯 Pipecat Mode: VAD + Smart Turn control turn detection")
|
||||
logger.info(" Your min_end_of_turn_silence is sent but ForceEndpoint overrides it")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Pipecat Mode (VAD + Smart Turn)",
|
||||
vad_force_turn_endpoint=True,
|
||||
)
|
||||
|
||||
|
||||
async def test_17_stt_mode():
|
||||
"""Test 17: STT mode (model controls turns)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100,
|
||||
)
|
||||
logger.info("🎯 STT Mode: u3-rt-pro model controls turn detection")
|
||||
logger.info(" No ForceEndpoint - parameters are respected")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"STT Mode (Model Turn Detection)",
|
||||
vad_force_turn_endpoint=False,
|
||||
)
|
||||
|
||||
|
||||
# === STT MODE TIMING EXPERIMENTS (18-20) ===
|
||||
|
||||
|
||||
async def test_18_stt_long_max_short_min():
|
||||
"""Test 18: STT mode - Long max_turn_silence + Short min (5000ms + 100ms)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100, # Short - quick confident turns
|
||||
max_turn_silence=5000, # Long - allows pauses up to 5 seconds
|
||||
)
|
||||
logger.info("🎯 STT Mode: Testing max/min parameter interaction")
|
||||
logger.info(" min_turn_silence: 100ms (quick when confident)")
|
||||
logger.info(" max_turn_silence: 5000ms (allows up to 5 second pauses)")
|
||||
logger.info(" Try: Quick sentences (should respond fast) + Long pauses mid-thought")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"STT: Long Max (5s) + Short Min (100ms)",
|
||||
vad_force_turn_endpoint=False,
|
||||
)
|
||||
|
||||
|
||||
async def test_19_stt_long_min():
|
||||
"""Test 19: STT mode - Long min_turn_silence (3000ms)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=3000, # 3 seconds
|
||||
max_turn_silence=5000, # 5 seconds
|
||||
)
|
||||
logger.info("🎯 STT Mode: Testing long minimum silence requirement")
|
||||
logger.info(" min_turn_silence: 3000ms")
|
||||
logger.info(" max_turn_silence: 5000ms")
|
||||
logger.info(" Bot will wait 3 full seconds of silence before responding!")
|
||||
logger.info(" Try: Speaking with short pauses - bot should NOT interrupt")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"STT: Long Min (3s)",
|
||||
vad_force_turn_endpoint=False,
|
||||
)
|
||||
|
||||
|
||||
async def test_20_stt_both_short():
|
||||
"""Test 20: STT mode - Both short (max=300ms, min=100ms)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100, # 100ms
|
||||
max_turn_silence=300, # 300ms
|
||||
)
|
||||
logger.info("🎯 STT Mode: Testing aggressive/quick response timing")
|
||||
logger.info(" min_turn_silence: 100ms")
|
||||
logger.info(" max_turn_silence: 300ms")
|
||||
logger.info(" Bot will respond VERY quickly to any pause!")
|
||||
logger.info(" Try: Speaking with natural pauses - expect quick responses")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"STT: Both Short (300ms/100ms)",
|
||||
vad_force_turn_endpoint=False,
|
||||
)
|
||||
|
||||
|
||||
# === EDGE CASES (21-23) ===
|
||||
|
||||
|
||||
async def test_21_very_long_silence():
|
||||
"""Test 21: Very long silence threshold (STT mode only)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=10000, # 10 seconds
|
||||
)
|
||||
logger.warning("⚠️ STT Mode with 10 second silence threshold")
|
||||
logger.info(" Bot will wait 10 seconds of silence before responding!")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Very Long Silence (10s) - STT Mode",
|
||||
vad_force_turn_endpoint=False,
|
||||
)
|
||||
|
||||
|
||||
async def test_22_very_short_silence():
|
||||
"""Test 22: Very short silence threshold (50ms)."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=50,
|
||||
)
|
||||
logger.info("⚡ Very short silence threshold (50ms)")
|
||||
logger.info(" Bot will respond very quickly!")
|
||||
await run_bot(connection_params, "Very Short Silence (50ms)")
|
||||
|
||||
|
||||
async def test_23_keyterms_plus_diarization():
|
||||
"""Test 23: Keyterms + Diarization combined."""
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
keyterms_prompt=["Xiomara", "Saoirse", "Pipecat"],
|
||||
speaker_labels=True,
|
||||
)
|
||||
logger.info("🎯 Keyterms + 🎤 Diarization both enabled")
|
||||
logger.info(" Try multiple speakers saying difficult names!")
|
||||
await run_bot(
|
||||
connection_params,
|
||||
"Keyterms + Diarization Combined",
|
||||
speaker_format="[{speaker}] {text}",
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Interactive Menu
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def show_menu():
|
||||
"""Display the comprehensive test menu."""
|
||||
print("\n" + "=" * 80)
|
||||
print("AssemblyAI u3-rt-pro Comprehensive Test Suite")
|
||||
print("=" * 80)
|
||||
print("\n📋 BASIC CONFIGURATION (1-3)")
|
||||
print(" 1. Basic Default (100ms)")
|
||||
print(" 2. Custom Silence (200ms)")
|
||||
print(" 3. Longer Silence (500ms)")
|
||||
|
||||
print("\n⚠️ PROMPTING & WARNINGS (4-7)")
|
||||
print(" 4. max_turn_silence Warning")
|
||||
print(" 5. Custom Prompt Warning")
|
||||
print(" 6. Prompt + Keyterms Conflict (ERROR)")
|
||||
print(" 7. Keyterms with Difficult Names")
|
||||
|
||||
print("\n🎤 DIARIZATION (8-9)")
|
||||
print(" 8. Diarization - Basic")
|
||||
print(" 9. Diarization - XML Formatting")
|
||||
|
||||
print("\n🔄 DYNAMIC UPDATES - SINGLE (10-13)")
|
||||
print(" 10. Dynamic Keyterms (Before/After with difficult names)")
|
||||
print(" 11. Dynamic Silence (100ms → 3s DRAMATIC)")
|
||||
print(" 12. Dynamic Prompt with Keyterms (Before/After)")
|
||||
print(" 13. Dynamic Clear Keyterms")
|
||||
|
||||
print("\n🔄 DYNAMIC UPDATES - MULTIPLE (14-15)")
|
||||
print(" 14. Multiple Parameters at Once")
|
||||
print(" 15. Complex Update Sequence (4 stages)")
|
||||
|
||||
print("\n⚖️ MODE COMPARISON (16-17)")
|
||||
print(" 16. Pipecat Mode (VAD + Smart Turn)")
|
||||
print(" 17. STT Mode (Model Turn Detection)")
|
||||
|
||||
print("\n⏱️ STT MODE TIMING EXPERIMENTS (18-20)")
|
||||
print(" 18. STT: Long Max (5s) + Short Min (100ms)")
|
||||
print(" 19. STT: Long Min (3s)")
|
||||
print(" 20. STT: Both Short (300ms/100ms)")
|
||||
|
||||
print("\n🎯 EDGE CASES (21-23)")
|
||||
print(" 21. Very Long Silence (10s - STT Mode)")
|
||||
print(" 22. Very Short Silence (50ms)")
|
||||
print(" 23. Keyterms + Diarization Combined")
|
||||
|
||||
print("\n 0. Exit")
|
||||
print("\n" + "=" * 80)
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main interactive menu."""
|
||||
tests = {
|
||||
"1": test_01_basic_100ms,
|
||||
"2": test_02_custom_200ms,
|
||||
"3": test_03_custom_500ms,
|
||||
"4": test_04_max_warning,
|
||||
"5": test_05_prompt_warning,
|
||||
"6": test_06_prompt_keyterms_conflict,
|
||||
"7": test_07_keyterms_difficult,
|
||||
"8": test_08_diarization_basic,
|
||||
"9": test_09_diarization_xml,
|
||||
"10": test_10_dynamic_keyterms,
|
||||
"11": test_11_dynamic_silence,
|
||||
"12": test_12_dynamic_prompt,
|
||||
"13": test_13_dynamic_clear_keyterms,
|
||||
"14": test_14_multi_param_update,
|
||||
"15": test_15_complex_sequence,
|
||||
"16": test_16_pipecat_mode,
|
||||
"17": test_17_stt_mode,
|
||||
"18": test_18_stt_long_max_short_min,
|
||||
"19": test_19_stt_long_min,
|
||||
"20": test_20_stt_both_short,
|
||||
"21": test_21_very_long_silence,
|
||||
"22": test_22_very_short_silence,
|
||||
"23": test_23_keyterms_plus_diarization,
|
||||
}
|
||||
|
||||
while True:
|
||||
show_menu()
|
||||
choice = input("Enter test number (or 0 to exit): ").strip()
|
||||
|
||||
if choice == "0":
|
||||
print("\n👋 Goodbye!")
|
||||
break
|
||||
|
||||
if choice in tests:
|
||||
try:
|
||||
await tests[choice]()
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n⚠️ Test interrupted by user")
|
||||
except Exception as e:
|
||||
logger.error(f"Test failed with error: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
|
||||
input("\n\nPress Enter to return to menu...")
|
||||
else:
|
||||
print(f"\n❌ Invalid choice: {choice}")
|
||||
input("Press Enter to continue...")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n👋 Goodbye!")
|
||||
@@ -1,582 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""AssemblyAI u3-rt-pro Comprehensive Test Script
|
||||
|
||||
Tests all features:
|
||||
- Basic configuration
|
||||
- Prompting and keyterms
|
||||
- Diarization
|
||||
- Dynamic updates
|
||||
- Turn detection modes
|
||||
|
||||
Usage:
|
||||
python test_assemblyai_u3pro.py --test <test_name>
|
||||
python test_assemblyai_u3pro.py --interactive
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from typing import List
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
|
||||
# Add src to path
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "src"))
|
||||
|
||||
from pipecat.audio.vad.silero import SileroVADAnalyzer
|
||||
from pipecat.frames.frames import (
|
||||
EndFrame,
|
||||
Frame,
|
||||
LLMRunFrame,
|
||||
STTUpdateSettingsFrame,
|
||||
TranscriptionFrame,
|
||||
)
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.pipeline.runner import PipelineRunner
|
||||
from pipecat.pipeline.task import PipelineTask
|
||||
from pipecat.processors.aggregators.llm_context import LLMContext
|
||||
from pipecat.processors.aggregators.llm_response_universal import (
|
||||
LLMContextAggregatorPair,
|
||||
LLMUserAggregatorParams,
|
||||
)
|
||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||
from pipecat.services.assemblyai.models import AssemblyAIConnectionParams
|
||||
from pipecat.services.assemblyai.stt import AssemblyAISTTService
|
||||
from pipecat.services.cartesia.tts import CartesiaTTSService
|
||||
from pipecat.services.openai.llm import OpenAILLMService
|
||||
from pipecat.transports.local.audio import LocalAudioTransport, LocalAudioTransportParams
|
||||
|
||||
load_dotenv()
|
||||
|
||||
|
||||
# Test configuration
|
||||
class TestConfig:
|
||||
"""Centralized test configuration."""
|
||||
|
||||
ASSEMBLYAI_API_KEY = os.getenv("ASSEMBLYAI_API_KEY")
|
||||
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
|
||||
CARTESIA_API_KEY = os.getenv("CARTESIA_API_KEY")
|
||||
|
||||
@classmethod
|
||||
def validate(cls):
|
||||
"""Validate all required API keys are set."""
|
||||
missing = []
|
||||
if not cls.ASSEMBLYAI_API_KEY:
|
||||
missing.append("ASSEMBLYAI_API_KEY")
|
||||
if not cls.OPENAI_API_KEY:
|
||||
missing.append("OPENAI_API_KEY")
|
||||
if not cls.CARTESIA_API_KEY:
|
||||
missing.append("CARTESIA_API_KEY")
|
||||
|
||||
if missing:
|
||||
logger.error(f"Missing required environment variables: {', '.join(missing)}")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
class TranscriptionLogger(FrameProcessor):
|
||||
"""Log transcriptions for test verification."""
|
||||
|
||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
||||
if isinstance(frame, TranscriptionFrame):
|
||||
logger.info(f"📝 TRANSCRIPTION: {frame.text}")
|
||||
logger.info(f" Speaker: {frame.user_id}")
|
||||
logger.info(f" Finalized: {frame.finalized}")
|
||||
if hasattr(frame, "result") and frame.result:
|
||||
if hasattr(frame.result, "speaker"):
|
||||
logger.info(f" Diarization: {frame.result.speaker}")
|
||||
|
||||
await self.push_frame(frame, direction)
|
||||
|
||||
|
||||
async def create_basic_voice_agent(
|
||||
connection_params: AssemblyAIConnectionParams,
|
||||
vad_force_turn_endpoint: bool = True,
|
||||
speaker_format: str = None,
|
||||
) -> tuple[PipelineTask, LocalAudioTransport]:
|
||||
"""Create a basic voice agent for testing.
|
||||
|
||||
Args:
|
||||
connection_params: AssemblyAI connection parameters
|
||||
vad_force_turn_endpoint: Turn detection mode
|
||||
speaker_format: Optional speaker formatting string
|
||||
|
||||
Returns:
|
||||
Tuple of (PipelineTask, LocalAudioTransport)
|
||||
"""
|
||||
# Create local audio transport (uses your microphone and speakers)
|
||||
transport = LocalAudioTransport(
|
||||
params=LocalAudioTransportParams(
|
||||
audio_in_enabled=True,
|
||||
audio_out_enabled=True,
|
||||
)
|
||||
)
|
||||
|
||||
# Create STT
|
||||
stt = AssemblyAISTTService(
|
||||
api_key=TestConfig.ASSEMBLYAI_API_KEY,
|
||||
connection_params=connection_params,
|
||||
vad_force_turn_endpoint=vad_force_turn_endpoint,
|
||||
speaker_format=speaker_format,
|
||||
)
|
||||
|
||||
# Create TTS
|
||||
tts = CartesiaTTSService(
|
||||
api_key=TestConfig.CARTESIA_API_KEY,
|
||||
voice_id="a0e99841-438c-4a64-b679-ae501e7d6091", # Conversational English
|
||||
)
|
||||
|
||||
# Create LLM context and service
|
||||
messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a helpful voice assistant. Keep responses brief and natural. "
|
||||
"If you see speaker tags like <Speaker A>text</Speaker A>, acknowledge "
|
||||
"that you understand multiple speakers are present."
|
||||
),
|
||||
}
|
||||
]
|
||||
|
||||
context = LLMContext(messages)
|
||||
llm = OpenAILLMService(api_key=TestConfig.OPENAI_API_KEY, model="gpt-4")
|
||||
|
||||
# Create aggregators with VAD
|
||||
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
|
||||
context,
|
||||
user_params=LLMUserAggregatorParams(
|
||||
vad_analyzer=SileroVADAnalyzer(),
|
||||
),
|
||||
)
|
||||
|
||||
# Create transcription logger
|
||||
transcription_logger = TranscriptionLogger()
|
||||
|
||||
# Create pipeline
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
transport.input(),
|
||||
stt,
|
||||
transcription_logger,
|
||||
user_aggregator,
|
||||
llm,
|
||||
tts,
|
||||
transport.output(),
|
||||
assistant_aggregator,
|
||||
]
|
||||
)
|
||||
|
||||
# Create task
|
||||
task = PipelineTask(pipeline)
|
||||
|
||||
return task, transport
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Test Functions
|
||||
# ============================================================================
|
||||
|
||||
|
||||
async def test_basic_config():
|
||||
"""Test 1: Basic default configuration."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 1: Basic Default Configuration")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(speech_model="u3-rt-pro")
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
logger.info("✅ Service created successfully with default params")
|
||||
logger.info("Expected: min=max=100ms, u3-rt-pro model")
|
||||
logger.info("Speak into your microphone to test transcription")
|
||||
|
||||
# Trigger initial bot greeting
|
||||
await task.queue_frames([LLMRunFrame()])
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_custom_min_silence():
|
||||
"""Test 2: Custom min_turn_silence."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 2: Custom min_turn_silence")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(speech_model="u3-rt-pro", min_turn_silence=200)
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
logger.info("✅ Service created with min=200ms")
|
||||
logger.info("Expected: Both min and max set to 200ms")
|
||||
logger.info("Speak short phrases and observe turn detection timing")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_max_silence_warning():
|
||||
"""Test 3: Setting max_turn_silence should trigger warning."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 3: max_turn_silence Warning")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
min_turn_silence=100,
|
||||
max_turn_silence=500, # Should trigger warning
|
||||
)
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
logger.info("⚠️ Check logs above for warning about max_turn_silence being overridden")
|
||||
logger.info("Expected: Warning logged, max set to 100ms (same as min)")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_custom_prompt_warning():
|
||||
"""Test 5: Custom prompt should trigger warning."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 5: Custom Prompt Warning")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
prompt="Transcribe verbatim. Always include punctuation.",
|
||||
)
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
logger.info("⚠️ Check logs above for warning about testing without prompt first")
|
||||
logger.info("Expected: Warning logged, service continues with custom prompt")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_prompt_keyterms_conflict():
|
||||
"""Test 6: Prompt + keyterms_prompt should raise error."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 6: Prompt + Keyterms Conflict (Error)")
|
||||
logger.info("=" * 80)
|
||||
|
||||
try:
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
prompt="Custom prompt",
|
||||
keyterms_prompt=["test", "words"],
|
||||
)
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
logger.error("❌ TEST FAILED: Should have raised ValueError")
|
||||
except ValueError as e:
|
||||
logger.info(f"✅ TEST PASSED: ValueError raised as expected")
|
||||
logger.info(f" Error message: {e}")
|
||||
|
||||
|
||||
async def test_keyterms_basic():
|
||||
"""Test 7: Basic keyterms at initialization."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 7: Basic Keyterms Prompting")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(
|
||||
speech_model="u3-rt-pro",
|
||||
keyterms_prompt=["Pipecat", "AssemblyAI", "Universal-3", "streaming"],
|
||||
)
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
logger.info("✅ Service created with keyterms: Pipecat, AssemblyAI, Universal-3, streaming")
|
||||
logger.info("Expected: Boosted recognition for these terms")
|
||||
logger.info("Try saying: 'I'm testing Pipecat with AssemblyAI Universal-3 for streaming'")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_diarization_no_format():
|
||||
"""Test 10: Diarization enabled without formatting."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 10: Diarization Enabled (No Formatting)")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(speech_model="u3-rt-pro", speaker_labels=True)
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
logger.info("✅ Service created with speaker_labels=True")
|
||||
logger.info("Expected: Speaker IDs in user_id field, plain text in transcript")
|
||||
logger.info("Have multiple people speak to see different speaker labels")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_diarization_xml_format():
|
||||
"""Test 11: Diarization with XML formatting."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 11: Diarization with XML Formatting")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(speech_model="u3-rt-pro", speaker_labels=True)
|
||||
|
||||
task, transport = await create_basic_voice_agent(
|
||||
connection_params, speaker_format="<{speaker}>{text}</{speaker}>"
|
||||
)
|
||||
|
||||
logger.info("✅ Service created with XML speaker formatting")
|
||||
logger.info("Expected: Text like '<Speaker A>Hello</Speaker A>'")
|
||||
logger.info("Have multiple people speak to see formatted speaker tags")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_dynamic_keyterms():
|
||||
"""Test 13: Dynamic keyterms updates."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 13: Dynamic Keyterms Updates")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(speech_model="u3-rt-pro")
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
async def update_keyterms_stages():
|
||||
"""Simulate multi-stage conversation with keyterms updates."""
|
||||
await asyncio.sleep(5) # Wait for connection
|
||||
|
||||
# Stage 1: Greeting
|
||||
logger.info("🔄 STAGE 1: Greeting (general terms)")
|
||||
update1 = STTUpdateSettingsFrame(
|
||||
settings={"keyterms_prompt": ["hello", "hi", "good morning", "welcome"]}
|
||||
)
|
||||
await task.queue_frames([update1])
|
||||
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Stage 2: Name collection
|
||||
logger.info("🔄 STAGE 2: Name Collection")
|
||||
update2 = STTUpdateSettingsFrame(
|
||||
settings={
|
||||
"keyterms_prompt": [
|
||||
"first name",
|
||||
"last name",
|
||||
"John",
|
||||
"Jane",
|
||||
"Smith",
|
||||
"Johnson",
|
||||
]
|
||||
}
|
||||
)
|
||||
await task.queue_frames([update2])
|
||||
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Stage 3: Medical info
|
||||
logger.info("🔄 STAGE 3: Medical Information")
|
||||
update3 = STTUpdateSettingsFrame(
|
||||
settings={
|
||||
"keyterms_prompt": [
|
||||
"cardiology",
|
||||
"echocardiogram",
|
||||
"blood pressure",
|
||||
"Dr. Smith",
|
||||
"metoprolol",
|
||||
]
|
||||
}
|
||||
)
|
||||
await task.queue_frames([update3])
|
||||
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Stage 4: Clear keyterms
|
||||
logger.info("🔄 STAGE 4: Clear Keyterms")
|
||||
update4 = STTUpdateSettingsFrame(settings={"keyterms_prompt": []})
|
||||
await task.queue_frames([update4])
|
||||
|
||||
# Start update task
|
||||
asyncio.create_task(update_keyterms_stages())
|
||||
|
||||
logger.info("✅ Service created, will update keyterms every 10 seconds")
|
||||
logger.info("Expected: Different keyterms at each stage")
|
||||
logger.info("Watch logs for 'STAGE X' messages and test relevant terms")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_dynamic_silence_params():
|
||||
"""Test 15: Dynamic silence parameter updates."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 15: Dynamic Silence Parameters")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(speech_model="u3-rt-pro")
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
async def update_silence_params():
|
||||
"""Update silence parameters for different scenarios."""
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# Normal conversation
|
||||
logger.info("🔄 PHASE 1: Normal conversation (default timing)")
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Reading credit card
|
||||
logger.info("🔄 PHASE 2: Reading numbers (longer silence tolerance)")
|
||||
update1 = STTUpdateSettingsFrame(
|
||||
settings={
|
||||
"max_turn_silence": 5000,
|
||||
"min_turn_silence": 300,
|
||||
}
|
||||
)
|
||||
await task.queue_frames([update1])
|
||||
|
||||
await asyncio.sleep(15)
|
||||
|
||||
# Back to normal
|
||||
logger.info("🔄 PHASE 3: Back to normal conversation")
|
||||
update2 = STTUpdateSettingsFrame(
|
||||
settings={
|
||||
"max_turn_silence": 1200,
|
||||
"min_turn_silence": 100,
|
||||
}
|
||||
)
|
||||
await task.queue_frames([update2])
|
||||
|
||||
asyncio.create_task(update_silence_params())
|
||||
|
||||
logger.info("✅ Service will update silence parameters during conversation")
|
||||
logger.info("Expected: Longer pauses tolerated in Phase 2")
|
||||
logger.info("Try pausing between words to test")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
async def test_multi_param_update():
|
||||
"""Test 17: Update multiple parameters at once."""
|
||||
logger.info("=" * 80)
|
||||
logger.info("TEST 17: Multiple Parameter Update")
|
||||
logger.info("=" * 80)
|
||||
|
||||
connection_params = AssemblyAIConnectionParams(speech_model="u3-rt-pro")
|
||||
|
||||
task, transport = await create_basic_voice_agent(connection_params)
|
||||
|
||||
async def multi_update():
|
||||
await asyncio.sleep(5)
|
||||
|
||||
logger.info("🔄 Updating multiple parameters together")
|
||||
update = STTUpdateSettingsFrame(
|
||||
settings={
|
||||
"keyterms_prompt": ["account", "routing", "number"],
|
||||
"max_turn_silence": 3000,
|
||||
"min_turn_silence": 200,
|
||||
}
|
||||
)
|
||||
await task.queue_frames([update])
|
||||
|
||||
logger.info("✅ Check logs for single UpdateConfiguration message")
|
||||
|
||||
asyncio.create_task(multi_update())
|
||||
|
||||
logger.info("Expected: All params updated in single WebSocket message")
|
||||
|
||||
runner = PipelineRunner()
|
||||
await runner.run(task)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Main Test Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main():
|
||||
"""Main test runner."""
|
||||
parser = argparse.ArgumentParser(description="Test AssemblyAI u3-rt-pro integration")
|
||||
parser.add_argument(
|
||||
"--test",
|
||||
type=str,
|
||||
default="basic",
|
||||
help="Test to run (basic, custom_min, max_warning, prompt_warning, "
|
||||
"prompt_keyterms_conflict, keyterms, diarization, diarization_xml, "
|
||||
"dynamic_keyterms, dynamic_silence, multi_param, all)",
|
||||
)
|
||||
parser.add_argument("--interactive", action="store_true", help="Run in interactive mode")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Validate environment
|
||||
if not TestConfig.validate():
|
||||
logger.error("Please set all required environment variables in .env")
|
||||
sys.exit(1)
|
||||
|
||||
# Test mapping
|
||||
tests = {
|
||||
"basic": test_basic_config,
|
||||
"custom_min": test_custom_min_silence,
|
||||
"max_warning": test_max_silence_warning,
|
||||
"prompt_warning": test_custom_prompt_warning,
|
||||
"prompt_keyterms_conflict": test_prompt_keyterms_conflict,
|
||||
"keyterms": test_keyterms_basic,
|
||||
"diarization": test_diarization_no_format,
|
||||
"diarization_xml": test_diarization_xml_format,
|
||||
"dynamic_keyterms": test_dynamic_keyterms,
|
||||
"dynamic_silence": test_dynamic_silence_params,
|
||||
"multi_param": test_multi_param_update,
|
||||
}
|
||||
|
||||
if args.interactive:
|
||||
logger.info("Interactive mode - select test to run:")
|
||||
for i, (name, _) in enumerate(tests.items(), 1):
|
||||
logger.info(f"{i}. {name}")
|
||||
logger.info(f"{len(tests) + 1}. Run all tests")
|
||||
|
||||
choice = input("\nEnter test number: ")
|
||||
try:
|
||||
choice_num = int(choice)
|
||||
if choice_num == len(tests) + 1:
|
||||
args.test = "all"
|
||||
else:
|
||||
args.test = list(tests.keys())[choice_num - 1]
|
||||
except (ValueError, IndexError):
|
||||
logger.error("Invalid choice")
|
||||
sys.exit(1)
|
||||
|
||||
# Run test(s)
|
||||
if args.test == "all":
|
||||
logger.info("Running all tests sequentially...")
|
||||
for test_name, test_func in tests.items():
|
||||
try:
|
||||
asyncio.run(test_func())
|
||||
except KeyboardInterrupt:
|
||||
logger.info(f"Test '{test_name}' interrupted")
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error(f"Test '{test_name}' failed: {e}")
|
||||
else:
|
||||
if args.test not in tests:
|
||||
logger.error(f"Unknown test: {args.test}")
|
||||
logger.info(f"Available tests: {', '.join(tests.keys())}")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
asyncio.run(tests[args.test]())
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Test interrupted")
|
||||
except Exception as e:
|
||||
logger.error(f"Test failed: {e}")
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user