Aleix Conchillo Flaqué
51ba245e10
scripts(evals): fix EVAL_CONVERSATION/EVAL_WEATHER eval
2025-11-18 21:14:27 -08:00
Aleix Conchillo Flaqué
38aac44a1e
scripts(evals): 26c should be a camera eval
2025-11-07 11:30:41 -08:00
Aleix Conchillo Flaqué
4f1468e0fa
scripts(evals): improve eval prompt
2025-11-07 10:05:46 -08:00
Mark Backman
7eb880c5e8
Add DeepgramHttpTTSService
2025-10-31 11:39:32 -04:00
Aleix Conchillo Flaqué
19f046a338
examples(foundational): add 12d-describe-image-moondream
2025-10-30 14:02:17 -07:00
Aleix Conchillo Flaqué
74fb6e7676
scripts(evals): improve eval prompting
2025-10-30 13:08:15 -07:00
Aleix Conchillo Flaqué
a997655eac
scripts(evals): simplify eval configuration and allow RunnerArgs body
2025-10-30 13:08:15 -07:00
Aleix Conchillo Flaqué
e458d3edfe
scripts(evals): update 12-* for 14-*-video
2025-10-30 13:08:15 -07:00
Mark Backman
8902f1dc94
Bump openai, openpipe versions, add 14x foundational example
2025-10-15 15:17:22 -04:00
Mark Backman
4b415721e2
Update release evals for OpenAI Realtime, Gemini Live Vertex; shorten 26 foundational names
2025-10-10 12:26:23 -04:00
Mark Backman
eddce460da
Deprecate PlayHT TTS services
2025-10-07 16:40:01 -04:00
ivaaan
1b511557b2
upd evals
2025-10-02 13:48:30 -07:00
ivaaan
4ffdabcfde
add Hume example, small fixes
2025-10-02 13:38:36 -07:00
Filipi Fuchter
11bc4ea854
Adding deepgram flux to release evals.
2025-10-01 19:24:58 -03:00
Paul Kompfner
6faa50ae5b
Remove remaining usage of OpenAILLMContext throughout the codebase in favor of LLMContext, except for:
...
- Usage in classes that are already deprecated
- Usage related to realtime LLMs, which don't yet support `LLMContext`
- Usage in (soon-to-be-deprecated) code paths related to `OpenAILLMContext` itself and associated machinery
2025-09-24 16:35:03 -04:00
Aleix Conchillo Flaqué
620b1f785c
examples: update Strands Agents with universal context and add evals
2025-09-23 11:37:57 -07:00
Paul Kompfner
272532a3ea
Update examples, wherever possible, to use LLMContext and associated machinery instead of OpenAILLMContext and associated machinery.
...
With all these examples updated, we no longer need dedicated examples illustrating `LLMContext`, so they're removed.
Here’s where we *don’t* yet use `LLMContext` and associated machinery:
- Realtime services: OpenAI Realtime, Gemini Live, and AWS Nova Sonic (support coming soon)
- `GoogleLLMOpenAIBetaService` (it’s deprecated, so we didn’t bother adding support)
- `LLMLogObserver` (support coming soon)
- `GatedOpenAILLMContextAggregator` (support coming soon)
- `LangchainProcessor` (support coming soon)
- `Mem0MemoryService` (support coming soon)
- Examples that use LLM-specific tools definitions as opposed to `ToolsSchema` (these will be updated soon)
- Examples that rely `GoogleLLMContext.upgrade_to_google` (TBD what to do with these)
Examples that use `LLMLogObserver`:
- 30-
Examples that use `GatedOpenAILLMContextAggregator`:
- 22-
Examples that use `LangchainProcessor`:
- 07b-
Examples that use `Mem0MemoryService`:
- 37-
Examples that need updating to use `ToolsSchema`:
- 15-
- 15a-
- 20a-
- 20c-
- 20d-
- 22b-
- 22c-
- 33-
- 36-
Examples that use `GoogleLLMContext.upgrade_to_google`:
- 22d-
- 25-
2025-09-22 16:21:35 -04:00
Aleix Conchillo Flaqué
82478be861
scripts(evals): add 19b-openai-realtime-text
2025-09-12 11:03:32 -07:00
Paul Kompfner
a4023d3915
Update evals to include examples that exercise the universal LLMContext
2025-09-11 14:32:56 -04:00
Aleix Conchillo Flaqué
46deaada70
scripts(evals): allow user to talk and only eval when needed
2025-09-06 19:19:08 -07:00
Mark Backman
977a57c8fb
Add 14k (CerebrasLLMService) to release evals
2025-09-03 17:11:38 -04:00
Aleix Conchillo Flaqué
aeb9f1ffca
transports: reorganize module
2025-09-02 17:31:39 -07:00
Aleix Conchillo Flaqué
5ebcea2a3b
scripts(eval): change "result" function call parameter
2025-08-28 11:38:59 -07:00
Paul Kompfner
189749b579
Add LLMRunFrame to trigger an LLM response, replacing context_aggregator.user().get_context_frame()
2025-08-28 09:53:33 -04:00
Aleix Conchillo Flaqué
a0128516ff
PipelineTask: handle cancellations gracefully
2025-08-22 19:04:31 -07:00
Aleix Conchillo Flaqué
9273ec0f25
scripts(evals): always require an eval prompt
2025-08-22 16:57:47 -07:00
Mark Backman
b30af3e155
Tests specify USER_SPEAKS_FIRST or BOT_SPEAKS_FIRST
2025-08-22 12:12:17 -04:00
Mark Backman
1c1ee94074
Add 44 to evals, update evals to support user speaking first
2025-08-22 12:12:17 -04:00
Aleix Conchillo Flaqué
28c729cc36
replace asyncio.wait_for for wait_for2.wait_for
2025-08-20 15:26:57 -07:00
Aleix Conchillo Flaqué
f387776985
add custom asyncio.wait_for()
...
This patch uses `wait_for2` package to implement `asyncio.wait_for()` for
Python < 3.12.
In Python 3.12, `asyncio.wait_for()` is implemented in terms of
`asyncio.timeout()` which fixed a bunch of issues. However, this was never
backported (because of the lack of `async.timeout()`) and there are still many
remainig issues, specially in Python 3.10, in `async.wait_for()`.
See https://github.com/python/cpython/pull/98518
2025-08-20 14:09:05 -07:00
Mark Backman
b7ae2989ac
Add foundational 14w-function-calling.py
2025-08-14 10:00:46 -04:00
Mark Backman
901b6b5913
Add foundational 19b
2025-08-13 11:37:38 -04:00
Mark Backman
e720573e60
Added 07n-interruptible-gemini
2025-08-12 14:54:49 -04:00
Aleix Conchillo Flaqué
1cfbfcaf11
scripts(evals): add vision support
2025-08-11 20:06:24 -07:00
Aleix Conchillo Flaqué
b360cb31dc
scripts(evals): add multilinguag support and 15 series
2025-08-11 15:21:14 -07:00
Aleix Conchillo Flaqué
c6ef8de578
scripts(evals): fix 14v-function-calling-openai.py
2025-08-07 17:57:47 -07:00
Mark Backman
31acfaa091
Merge pull request #2388 from pipecat-ai/14v-adding-openai-stt-tts-llm-functioncalling
...
14v adding OpenAI stt tts llm functioncalling
2025-08-07 10:22:35 -07:00
Mark Backman
69541c8835
Linting fix, plus update eval suite with 14v and others, tiny fix for 14m, too
2025-08-07 13:20:45 -04:00
Mark Backman
4288b5e780
Merge pull request #2381 from pipecat-ai/aleix/runner-args-pipeline-idle-timeout
...
allow specifying PipelineTask idle timeout to runner arguments
2025-08-07 04:47:08 -07:00
Aleix Conchillo Flaqué
7ed14ad91f
scripts(evals): added soniox, inworld and asyncai
2025-08-06 23:14:50 -07:00
Aleix Conchillo Flaqué
38951aab56
scripts(evals): use RunnerArguments.pipeline_idle_timeout_secs
2025-08-06 19:37:29 -07:00
Aleix Conchillo Flaqué
ed8b0655a8
scripts(evals): fix runner eval cancellation
...
We need to call asyncio.gather() just once, not for every cancelled task.
2025-08-06 19:36:42 -07:00
Aleix Conchillo Flaqué
61f4dd2ff2
scripts(evals): fix 14e-function-calling-google
2025-08-05 17:44:45 -07:00
Aleix Conchillo Flaqué
52a6d8013c
scripts(evals): pass runner arguments to run_bot()
2025-08-05 16:13:32 -07:00
Mark Backman
f26bbe4092
Merge pull request #2363 from pipecat-ai/mb/update-14p
...
Update 14p, add 14p to evals, add Google creds to env.example
2025-08-05 14:00:13 -07:00
Aleix Conchillo Flaqué
5546c8e01c
scripts(evals): update to use new runner function
2025-08-05 11:46:28 -07:00
Mark Backman
ba5a5c7187
Update 14p, add 14p to evals, add Google creds to env.example
2025-08-05 13:30:36 -04:00
Mark Backman
b9a2a9b729
Add Tavus, HeyGen, Simli to release-evals
2025-08-02 09:35:06 -04:00
Aleix Conchillo Flaqué
baa878272d
scripts(evals): added 07a-interruptible-speechmatics.py
2025-07-03 13:44:21 -07:00
Aleix Conchillo Flaqué
093285868e
scripts(evals): update timeout back to 90 seconds
2025-07-03 13:37:17 -07:00
