Aleix Conchillo Flaqué
4f1468e0fa
scripts(evals): improve eval prompt
2025-11-07 10:05:46 -08:00
Aleix Conchillo Flaqué
74fb6e7676
scripts(evals): improve eval prompting
2025-10-30 13:08:15 -07:00
Aleix Conchillo Flaqué
a997655eac
scripts(evals): simplify eval configuration and allow RunnerArgs body
2025-10-30 13:08:15 -07:00
Paul Kompfner
6faa50ae5b
Remove remaining usage of OpenAILLMContext throughout the codebase in favor of LLMContext, except for:
...
- Usage in classes that are already deprecated
- Usage related to realtime LLMs, which don't yet support `LLMContext`
- Usage in (soon-to-be-deprecated) code paths related to `OpenAILLMContext` itself and associated machinery
2025-09-24 16:35:03 -04:00
Aleix Conchillo Flaqué
46deaada70
scripts(evals): allow user to talk and only eval when needed
2025-09-06 19:19:08 -07:00
Aleix Conchillo Flaqué
aeb9f1ffca
transports: reorganize module
2025-09-02 17:31:39 -07:00
Aleix Conchillo Flaqué
5ebcea2a3b
scripts(eval): change "result" function call parameter
2025-08-28 11:38:59 -07:00
Paul Kompfner
189749b579
Add LLMRunFrame to trigger an LLM response, replacing context_aggregator.user().get_context_frame()
2025-08-28 09:53:33 -04:00
Aleix Conchillo Flaqué
a0128516ff
PipelineTask: handle cancellations gracefully
2025-08-22 19:04:31 -07:00
Aleix Conchillo Flaqué
9273ec0f25
scripts(evals): always require an eval prompt
2025-08-22 16:57:47 -07:00
Mark Backman
1c1ee94074
Add 44 to evals, update evals to support user speaking first
2025-08-22 12:12:17 -04:00
Aleix Conchillo Flaqué
28c729cc36
replace asyncio.wait_for for wait_for2.wait_for
2025-08-20 15:26:57 -07:00
Aleix Conchillo Flaqué
f387776985
add custom asyncio.wait_for()
...
This patch uses `wait_for2` package to implement `asyncio.wait_for()` for
Python < 3.12.
In Python 3.12, `asyncio.wait_for()` is implemented in terms of
`asyncio.timeout()` which fixed a bunch of issues. However, this was never
backported (because of the lack of `async.timeout()`) and there are still many
remainig issues, specially in Python 3.10, in `async.wait_for()`.
See https://github.com/python/cpython/pull/98518
2025-08-20 14:09:05 -07:00
Aleix Conchillo Flaqué
1cfbfcaf11
scripts(evals): add vision support
2025-08-11 20:06:24 -07:00
Aleix Conchillo Flaqué
b360cb31dc
scripts(evals): add multilinguag support and 15 series
2025-08-11 15:21:14 -07:00
Aleix Conchillo Flaqué
38951aab56
scripts(evals): use RunnerArguments.pipeline_idle_timeout_secs
2025-08-06 19:37:29 -07:00
Aleix Conchillo Flaqué
ed8b0655a8
scripts(evals): fix runner eval cancellation
...
We need to call asyncio.gather() just once, not for every cancelled task.
2025-08-06 19:36:42 -07:00
Aleix Conchillo Flaqué
52a6d8013c
scripts(evals): pass runner arguments to run_bot()
2025-08-05 16:13:32 -07:00
Aleix Conchillo Flaqué
5546c8e01c
scripts(evals): update to use new runner function
2025-08-05 11:46:28 -07:00
Aleix Conchillo Flaqué
093285868e
scripts(evals): update timeout back to 90 seconds
2025-07-03 13:37:17 -07:00
Aleix Conchillo Flaqué
1a8d512abb
scripts(evals): make sure we cancel pending tasks after timeout
2025-07-03 10:01:42 -07:00
Mark Backman
9146def21b
Update examples to use default allow_interruptions, fixes to align examples
2025-06-19 10:07:32 -04:00
Aleix Conchillo Flaqué
2fcfb0aa9f
evals: don't use Deepgram's smart formatting
2025-05-30 16:55:55 -07:00
Aleix Conchillo Flaqué
f1df079512
evals: allow running a single eval
2025-05-30 16:55:55 -07:00
Aleix Conchillo Flaqué
d77bedbafb
evals: move scripts/release to script/evals and add README
2025-05-30 15:04:05 -07:00
