# AssemblyAI u3-rt-pro Testing Checklist

## Test Environment Setup
- [ ] Install dependencies: `uv sync --group dev --all-extras`
- [ ] Set up `.env` file with API keys
- [ ] Verify LiveKit connection
- [ ] Run basic voice agent test

---

## Feature Testing Checklist

### ✅ Basic Configuration Tests

#### Test 1: Default u3-rt-pro Configuration
- [ ] **Setup:** Create service with default params
- [ ] **Expected:** No errors, uses u3-rt-pro model with 100ms min/max
- [ ] **Verify:** Check logs for connection confirmation

#### Test 2: Custom min_turn_silence
- [ ] **Setup:** Set `min_turn_silence=200`
- [ ] **Expected:** Both min and max set to 200ms
- [ ] **Verify:** Speak short phrases, observe turn detection timing

#### Test 3: User sets max_turn_silence (Warning Test)
- [ ] **Setup:** Set `max_turn_silence=500` in connection params
- [ ] **Expected:** Warning logged, value overridden to match min
- [ ] **Verify:** Check logs for warning message

---

### ✅ Prompting Tests

#### Test 4: No Prompt (Default - Recommended)
- [ ] **Setup:** Don't set prompt parameter
- [ ] **Expected:** Uses default prompt, 88% accuracy, no warnings
- [ ] **Verify:** Transcription quality is good

#### Test 5: Custom Prompt (Warning Test)
- [ ] **Setup:** Set custom prompt in connection params
- [ ] **Expected:** Warning logged about testing without prompt first
- [ ] **Verify:** Check logs for prompt warning

#### Test 6: Prompt + Keyterms Conflict (Error Test)
- [ ] **Setup:** Set both `prompt` and `keyterms_prompt` at init
- [ ] **Expected:** ValueError raised with helpful error message
- [ ] **Verify:** Service fails to initialize with clear error

---

### ✅ Keyterms Prompting Tests

#### Test 7: Basic Keyterms at Init
- [ ] **Setup:** Set `keyterms_prompt=["Pipecat", "AssemblyAI", "Universal-3"]`
- [ ] **Expected:** Terms are boosted in recognition
- [ ] **Verify:** Say the boosted terms, check accuracy

#### Test 8: Empty Keyterms (No Boosting)
- [ ] **Setup:** Set `keyterms_prompt=[]`
- [ ] **Expected:** No boosting, default behavior
- [ ] **Verify:** Normal transcription

---

### ✅ Diarization Tests

#### Test 9: Diarization Disabled (Default)
- [ ] **Setup:** Don't set `speaker_labels` parameter
- [ ] **Expected:** No speaker info in transcripts
- [ ] **Verify:** TranscriptionFrame.user_id is default user_id

#### Test 10: Diarization Enabled (No Formatting)
- [ ] **Setup:** Set `speaker_labels=True`
- [ ] **Expected:** Speaker ID in user_id field, plain text
- [ ] **Verify:** Multiple speakers show different IDs (Speaker A, Speaker B)

#### Test 11: Diarization with XML Formatting
- [ ] **Setup:** Set `speaker_labels=True`, `speaker_format="<{speaker}>{text}</{speaker}>"`
- [ ] **Expected:** Text includes speaker tags: `<Speaker A>Hello</Speaker A>`
- [ ] **Verify:** Formatted text in transcript, speaker ID in user_id

#### Test 12: Diarization with Colon Prefix
- [ ] **Setup:** Set `speaker_labels=True`, `speaker_format="{speaker}: {text}"`
- [ ] **Expected:** Text includes prefix: `Speaker A: Hello`
- [ ] **Verify:** Formatted text, multiple speakers distinguishable

---

### ✅ Dynamic Updates Tests

#### Test 13: Dynamic Keyterms Update (Stage 1 → Stage 2)
- [ ] **Setup:** Start with empty keyterms, update mid-conversation
- [ ] **Expected:** New keyterms take effect immediately
- [ ] **Test Steps:**
  1. Start conversation with no keyterms
  2. Send update frame with `keyterms_prompt=["cardiology", "Dr. Smith"]`
  3. Say the new terms
- [ ] **Verify:** Improved recognition after update

#### Test 14: Clear Keyterms (Reset Context)
- [ ] **Setup:** Start with keyterms, clear them mid-stream
- [ ] **Expected:** Context biasing removed
- [ ] **Test Steps:**
  1. Start with `keyterms_prompt=["test", "words"]`
  2. Send update frame with `keyterms_prompt=[]`
- [ ] **Verify:** No more boosting after clear

#### Test 15: Dynamic Silence Parameters
- [ ] **Setup:** Update `max_turn_silence` mid-stream
- [ ] **Expected:** Turn detection timing changes
- [ ] **Test Steps:**
  1. Start with default (1200ms)
  2. Update to `max_turn_silence=5000` (for reading numbers)
  3. Pause longer between words
  4. Update back to `max_turn_silence=1200`
- [ ] **Verify:** Longer pauses tolerated when increased

#### Test 16: Dynamic Prompt Update
- [ ] **Setup:** Update prompt mid-stream
- [ ] **Expected:** New instructions take effect
- [ ] **Test Steps:**
  1. Start with default prompt
  2. Send update with custom prompt
- [ ] **Verify:** Behavior changes according to new prompt

#### Test 17: Multiple Parameters at Once
- [ ] **Setup:** Update keyterms, max_turn_silence, and min_end_of_turn together
- [ ] **Expected:** All parameters updated in single WebSocket message
- [ ] **Verify:** Check logs for single UpdateConfiguration message

#### Test 18: Dynamic Update - Prompt + Keyterms Conflict (Error)
- [ ] **Setup:** Try to update both prompt and keyterms_prompt in same update
- [ ] **Expected:** ValueError raised
- [ ] **Verify:** Update fails with clear error message

---

### ✅ Turn Detection Mode Tests

#### Test 19: Pipecat Mode (vad_force_turn_endpoint=True) - Default
- [ ] **Setup:** Use default settings (Pipecat mode)
- [ ] **Expected:**
  - ForceEndpoint sent on VAD stop
  - Smart Turn Analyzer makes decisions
  - min=max=100ms for u3-rt-pro
- [ ] **Verify:** Fast finals, Smart Turn handles completeness

#### Test 20: STT Mode (vad_force_turn_endpoint=False) - u3-rt-pro only
- [ ] **Setup:** Set `vad_force_turn_endpoint=False` with u3-rt-pro
- [ ] **Expected:**
  - AssemblyAI controls turn endings
  - SpeechStarted message triggers interruptions
  - UserStarted/StoppedSpeakingFrame emitted
- [ ] **Verify:** Turn detection from AssemblyAI model

#### Test 21: STT Mode with universal-streaming (Error Test)
- [ ] **Setup:** Set `vad_force_turn_endpoint=False` with universal-streaming
- [ ] **Expected:** ValueError raised (requires u3-rt-pro)
- [ ] **Verify:** Service fails with clear error

---

### ✅ Language Detection Tests (If Multilingual Model)

#### Test 22: Language Detection Enabled
- [ ] **Setup:** Use `universal-streaming-multilingual` with `language_detection=True`
- [ ] **Expected:** Language codes in transcripts
- [ ] **Verify:** Speak different languages, check language_code field

#### Test 23: Language Confidence Threshold
- [ ] **Setup:** Enable language detection
- [ ] **Expected:** High confidence (≥0.7) → detected language, Low → fallback to English
- [ ] **Verify:** Check logs for confidence warnings

---

### ✅ Edge Cases & Error Handling

#### Test 24: WebSocket Disconnect During Update
- [ ] **Setup:** Simulate disconnect, try update
- [ ] **Expected:** Error logged, update queued for reconnection
- [ ] **Verify:** Graceful handling, no crash

#### Test 25: Invalid Parameter Types
- [ ] **Setup:** Send update with wrong type (e.g., keyterms_prompt as string)
- [ ] **Expected:** Warning logged, parameter skipped
- [ ] **Verify:** Service continues, invalid param ignored

#### Test 26: Unknown Parameter in Update
- [ ] **Setup:** Send update with unsupported parameter (e.g., `language`)
- [ ] **Expected:** Warning logged about parameter
- [ ] **Verify:** Other valid params still updated

---

### ✅ Integration Tests

#### Test 27: Full Voice Agent Flow (Multi-Stage)
- [ ] **Setup:** Complete voice agent with stage transitions
- [ ] **Test Steps:**
  1. Greeting stage (general keyterms)
  2. Name collection stage (name keyterms)
  3. Account number stage (number keyterms, longer silence)
  4. Medical info stage (medical keyterms)
  5. Closing stage (goodbye keyterms)
- [ ] **Verify:** Each stage has appropriate keyterms and timing

#### Test 28: Diarization + Dynamic Updates
- [ ] **Setup:** Enable diarization, update keyterms mid-stream
- [ ] **Expected:** Both features work together
- [ ] **Verify:** Speaker IDs persist, keyterms update correctly

#### Test 29: Interruption Handling
- [ ] **Setup:** Bot speaking, user interrupts
- [ ] **Expected:**
  - Pipecat mode: VAD + Smart Turn handles
  - STT mode: SpeechStarted triggers interrupt
- [ ] **Verify:** Bot stops, user speech processed

---

## Testing Results Template

```
| Test # | Feature | Status | Notes |
|--------|---------|--------|-------|
| 1 | Default Config | ✅ PASS | |
| 2 | Custom min_silence | ✅ PASS | |
| 3 | max_silence Warning | ✅ PASS | |
| ... | ... | ... | ... |
```

---

## Expected Outcomes Summary

### ✅ Should Work (No Errors)
- Default configuration
- Custom min_turn_silence
- Keyterms prompting
- Diarization with/without formatting
- Dynamic updates (one parameter or multiple)
- Pipecat mode turn detection

### ⚠️ Should Warn (Logs Warning, Continues)
- Custom prompt set at init
- max_turn_silence set (overridden)
- Invalid parameter types in updates
- Language update attempted
- Prompt used with universal-streaming

### ❌ Should Error (Raises Exception, Stops)
- prompt + keyterms_prompt at init
- prompt + keyterms_prompt in same update
- vad_force_turn_endpoint=False with universal-streaming

---

## Quick Test Commands

```bash
# Run basic test
python test_assemblyai_u3pro.py --test basic

# Run specific test
python test_assemblyai_u3pro.py --test diarization

# Run all tests
python test_assemblyai_u3pro.py --test all

# Interactive mode
python test_assemblyai_u3pro.py --interactive
```