Mark Backman
845b4ad20e
Add 51 foundational to evals
2025-12-20 08:07:25 -05:00
Mark Backman
56c58f7302
Move Ultravox foundational example to 50, add to release evals
2025-12-18 13:38:12 -05:00
Aleix Conchillo Flaqué
d07b37b288
scripts(evals): more eval prompts improvements
2025-12-17 09:55:12 -08:00
Aleix Conchillo Flaqué
5b30f1b1ef
scripts(evals): improve prompts
2025-12-16 17:26:50 -08:00
Mark Backman
bd3bf9a00e
Inworld TTS services: Add websocket TTS class, add word-timestamp alignment
2025-12-16 13:47:24 -05:00
Aleix Conchillo Flaqué
21e346abe2
scripts(evals): improve eval prompts
2025-12-15 13:21:40 -08:00
Aleix Conchillo Flaqué
4f848e9631
Merge pull request #3227 from fixie-ai/mike/upstream
...
Add Ultravox service
2025-12-13 18:29:02 -08:00
Mike Depinet
4b81be7acf
Add Ultravox service ( #1 )
...
Adds support for using Ultravox Realtime as a speech-to-speech service.
Also removes the deprecated Ultravox speech-to-text vllm model integration to avoid confusion.
2025-12-12 10:16:15 -08:00
Paul Kompfner
12979293ad
Add thinking examples to eval suite
2025-12-11 15:58:48 -05:00
vipyne
acba544e6f
pr notes for nvidia service name change
2025-12-01 22:41:17 -06:00
Aleix Conchillo Flaqué
51ba245e10
scripts(evals): fix EVAL_CONVERSATION/EVAL_WEATHER eval
2025-11-18 21:14:27 -08:00
Aleix Conchillo Flaqué
38aac44a1e
scripts(evals): 26c should be a camera eval
2025-11-07 11:30:41 -08:00
Mark Backman
7eb880c5e8
Add DeepgramHttpTTSService
2025-10-31 11:39:32 -04:00
Aleix Conchillo Flaqué
19f046a338
examples(foundational): add 12d-describe-image-moondream
2025-10-30 14:02:17 -07:00
Aleix Conchillo Flaqué
74fb6e7676
scripts(evals): improve eval prompting
2025-10-30 13:08:15 -07:00
Aleix Conchillo Flaqué
a997655eac
scripts(evals): simplify eval configuration and allow RunnerArgs body
2025-10-30 13:08:15 -07:00
Aleix Conchillo Flaqué
e458d3edfe
scripts(evals): update 12-* for 14-*-video
2025-10-30 13:08:15 -07:00
Mark Backman
8902f1dc94
Bump openai, openpipe versions, add 14x foundational example
2025-10-15 15:17:22 -04:00
Mark Backman
4b415721e2
Update release evals for OpenAI Realtime, Gemini Live Vertex; shorten 26 foundational names
2025-10-10 12:26:23 -04:00
Mark Backman
eddce460da
Deprecate PlayHT TTS services
2025-10-07 16:40:01 -04:00
ivaaan
1b511557b2
upd evals
2025-10-02 13:48:30 -07:00
ivaaan
4ffdabcfde
add Hume example, small fixes
2025-10-02 13:38:36 -07:00
Filipi Fuchter
11bc4ea854
Adding deepgram flux to release evals.
2025-10-01 19:24:58 -03:00
Aleix Conchillo Flaqué
620b1f785c
examples: update Strands Agents with universal context and add evals
2025-09-23 11:37:57 -07:00
Paul Kompfner
272532a3ea
Update examples, wherever possible, to use LLMContext and associated machinery instead of OpenAILLMContext and associated machinery.
...
With all these examples updated, we no longer need dedicated examples illustrating `LLMContext`, so they're removed.
Here’s where we *don’t* yet use `LLMContext` and associated machinery:
- Realtime services: OpenAI Realtime, Gemini Live, and AWS Nova Sonic (support coming soon)
- `GoogleLLMOpenAIBetaService` (it’s deprecated, so we didn’t bother adding support)
- `LLMLogObserver` (support coming soon)
- `GatedOpenAILLMContextAggregator` (support coming soon)
- `LangchainProcessor` (support coming soon)
- `Mem0MemoryService` (support coming soon)
- Examples that use LLM-specific tools definitions as opposed to `ToolsSchema` (these will be updated soon)
- Examples that rely `GoogleLLMContext.upgrade_to_google` (TBD what to do with these)
Examples that use `LLMLogObserver`:
- 30-
Examples that use `GatedOpenAILLMContextAggregator`:
- 22-
Examples that use `LangchainProcessor`:
- 07b-
Examples that use `Mem0MemoryService`:
- 37-
Examples that need updating to use `ToolsSchema`:
- 15-
- 15a-
- 20a-
- 20c-
- 20d-
- 22b-
- 22c-
- 33-
- 36-
Examples that use `GoogleLLMContext.upgrade_to_google`:
- 22d-
- 25-
2025-09-22 16:21:35 -04:00
Aleix Conchillo Flaqué
82478be861
scripts(evals): add 19b-openai-realtime-text
2025-09-12 11:03:32 -07:00
Paul Kompfner
a4023d3915
Update evals to include examples that exercise the universal LLMContext
2025-09-11 14:32:56 -04:00
Mark Backman
977a57c8fb
Add 14k (CerebrasLLMService) to release evals
2025-09-03 17:11:38 -04:00
Aleix Conchillo Flaqué
9273ec0f25
scripts(evals): always require an eval prompt
2025-08-22 16:57:47 -07:00
Mark Backman
b30af3e155
Tests specify USER_SPEAKS_FIRST or BOT_SPEAKS_FIRST
2025-08-22 12:12:17 -04:00
Mark Backman
1c1ee94074
Add 44 to evals, update evals to support user speaking first
2025-08-22 12:12:17 -04:00
Mark Backman
b7ae2989ac
Add foundational 14w-function-calling.py
2025-08-14 10:00:46 -04:00
Mark Backman
901b6b5913
Add foundational 19b
2025-08-13 11:37:38 -04:00
Mark Backman
e720573e60
Added 07n-interruptible-gemini
2025-08-12 14:54:49 -04:00
Aleix Conchillo Flaqué
1cfbfcaf11
scripts(evals): add vision support
2025-08-11 20:06:24 -07:00
Aleix Conchillo Flaqué
b360cb31dc
scripts(evals): add multilinguag support and 15 series
2025-08-11 15:21:14 -07:00
Aleix Conchillo Flaqué
c6ef8de578
scripts(evals): fix 14v-function-calling-openai.py
2025-08-07 17:57:47 -07:00
Mark Backman
31acfaa091
Merge pull request #2388 from pipecat-ai/14v-adding-openai-stt-tts-llm-functioncalling
...
14v adding OpenAI stt tts llm functioncalling
2025-08-07 10:22:35 -07:00
Mark Backman
69541c8835
Linting fix, plus update eval suite with 14v and others, tiny fix for 14m, too
2025-08-07 13:20:45 -04:00
Aleix Conchillo Flaqué
7ed14ad91f
scripts(evals): added soniox, inworld and asyncai
2025-08-06 23:14:50 -07:00
Aleix Conchillo Flaqué
61f4dd2ff2
scripts(evals): fix 14e-function-calling-google
2025-08-05 17:44:45 -07:00
Mark Backman
ba5a5c7187
Update 14p, add 14p to evals, add Google creds to env.example
2025-08-05 13:30:36 -04:00
Mark Backman
b9a2a9b729
Add Tavus, HeyGen, Simli to release-evals
2025-08-02 09:35:06 -04:00
Aleix Conchillo Flaqué
baa878272d
scripts(evals): added 07a-interruptible-speechmatics.py
2025-07-03 13:44:21 -07:00
Mark Backman
43a24d15f6
Add 40-aws-nova-sonic to release evals list
2025-06-27 08:34:39 -04:00
Aleix Conchillo Flaqué
f1df079512
evals: allow running a single eval
2025-05-30 16:55:55 -07:00
Aleix Conchillo Flaqué
d77bedbafb
evals: move scripts/release to script/evals and add README
2025-05-30 15:04:05 -07:00
