Aleix Conchillo Flaqué
82478be861
scripts(evals): add 19b-openai-realtime-text
2025-09-12 11:03:32 -07:00
Paul Kompfner
a4023d3915
Update evals to include examples that exercise the universal LLMContext
2025-09-11 14:32:56 -04:00
Aleix Conchillo Flaqué
46deaada70
scripts(evals): allow user to talk and only eval when needed
2025-09-06 19:19:08 -07:00
Mark Backman
977a57c8fb
Add 14k (CerebrasLLMService) to release evals
2025-09-03 17:11:38 -04:00
Aleix Conchillo Flaqué
aeb9f1ffca
transports: reorganize module
2025-09-02 17:31:39 -07:00
Aleix Conchillo Flaqué
5ebcea2a3b
scripts(eval): change "result" function call parameter
2025-08-28 11:38:59 -07:00
Paul Kompfner
189749b579
Add LLMRunFrame to trigger an LLM response, replacing context_aggregator.user().get_context_frame()
2025-08-28 09:53:33 -04:00
Aleix Conchillo Flaqué
a0128516ff
PipelineTask: handle cancellations gracefully
2025-08-22 19:04:31 -07:00
Aleix Conchillo Flaqué
9273ec0f25
scripts(evals): always require an eval prompt
2025-08-22 16:57:47 -07:00
Mark Backman
b30af3e155
Tests specify USER_SPEAKS_FIRST or BOT_SPEAKS_FIRST
2025-08-22 12:12:17 -04:00
Mark Backman
1c1ee94074
Add 44 to evals, update evals to support user speaking first
2025-08-22 12:12:17 -04:00
Aleix Conchillo Flaqué
28c729cc36
replace asyncio.wait_for for wait_for2.wait_for
2025-08-20 15:26:57 -07:00
Aleix Conchillo Flaqué
f387776985
add custom asyncio.wait_for()
...
This patch uses `wait_for2` package to implement `asyncio.wait_for()` for
Python < 3.12.
In Python 3.12, `asyncio.wait_for()` is implemented in terms of
`asyncio.timeout()` which fixed a bunch of issues. However, this was never
backported (because of the lack of `async.timeout()`) and there are still many
remainig issues, specially in Python 3.10, in `async.wait_for()`.
See https://github.com/python/cpython/pull/98518
2025-08-20 14:09:05 -07:00
Mark Backman
b7ae2989ac
Add foundational 14w-function-calling.py
2025-08-14 10:00:46 -04:00
Mark Backman
901b6b5913
Add foundational 19b
2025-08-13 11:37:38 -04:00
Mark Backman
e720573e60
Added 07n-interruptible-gemini
2025-08-12 14:54:49 -04:00
Aleix Conchillo Flaqué
1cfbfcaf11
scripts(evals): add vision support
2025-08-11 20:06:24 -07:00
Aleix Conchillo Flaqué
b360cb31dc
scripts(evals): add multilinguag support and 15 series
2025-08-11 15:21:14 -07:00
Aleix Conchillo Flaqué
c6ef8de578
scripts(evals): fix 14v-function-calling-openai.py
2025-08-07 17:57:47 -07:00
Mark Backman
31acfaa091
Merge pull request #2388 from pipecat-ai/14v-adding-openai-stt-tts-llm-functioncalling
...
14v adding OpenAI stt tts llm functioncalling
2025-08-07 10:22:35 -07:00
Mark Backman
69541c8835
Linting fix, plus update eval suite with 14v and others, tiny fix for 14m, too
2025-08-07 13:20:45 -04:00
Mark Backman
4288b5e780
Merge pull request #2381 from pipecat-ai/aleix/runner-args-pipeline-idle-timeout
...
allow specifying PipelineTask idle timeout to runner arguments
2025-08-07 04:47:08 -07:00
Aleix Conchillo Flaqué
7ed14ad91f
scripts(evals): added soniox, inworld and asyncai
2025-08-06 23:14:50 -07:00
Aleix Conchillo Flaqué
38951aab56
scripts(evals): use RunnerArguments.pipeline_idle_timeout_secs
2025-08-06 19:37:29 -07:00
Aleix Conchillo Flaqué
ed8b0655a8
scripts(evals): fix runner eval cancellation
...
We need to call asyncio.gather() just once, not for every cancelled task.
2025-08-06 19:36:42 -07:00
Aleix Conchillo Flaqué
61f4dd2ff2
scripts(evals): fix 14e-function-calling-google
2025-08-05 17:44:45 -07:00
Aleix Conchillo Flaqué
52a6d8013c
scripts(evals): pass runner arguments to run_bot()
2025-08-05 16:13:32 -07:00
Mark Backman
f26bbe4092
Merge pull request #2363 from pipecat-ai/mb/update-14p
...
Update 14p, add 14p to evals, add Google creds to env.example
2025-08-05 14:00:13 -07:00
Aleix Conchillo Flaqué
5546c8e01c
scripts(evals): update to use new runner function
2025-08-05 11:46:28 -07:00
Mark Backman
ba5a5c7187
Update 14p, add 14p to evals, add Google creds to env.example
2025-08-05 13:30:36 -04:00
Mark Backman
b9a2a9b729
Add Tavus, HeyGen, Simli to release-evals
2025-08-02 09:35:06 -04:00
Aleix Conchillo Flaqué
baa878272d
scripts(evals): added 07a-interruptible-speechmatics.py
2025-07-03 13:44:21 -07:00
Aleix Conchillo Flaqué
093285868e
scripts(evals): update timeout back to 90 seconds
2025-07-03 13:37:17 -07:00
Aleix Conchillo Flaqué
1a8d512abb
scripts(evals): make sure we cancel pending tasks after timeout
2025-07-03 10:01:42 -07:00
Mark Backman
43a24d15f6
Add 40-aws-nova-sonic to release evals list
2025-06-27 08:34:39 -04:00
Mark Backman
9146def21b
Update examples to use default allow_interruptions, fixes to align examples
2025-06-19 10:07:32 -04:00
Aleix Conchillo Flaqué
2fcfb0aa9f
evals: don't use Deepgram's smart formatting
2025-05-30 16:55:55 -07:00
Aleix Conchillo Flaqué
f1df079512
evals: allow running a single eval
2025-05-30 16:55:55 -07:00
Aleix Conchillo Flaqué
d77bedbafb
evals: move scripts/release to script/evals and add README
2025-05-30 15:04:05 -07:00
