Mark Backman
1c1ee94074
Add 44 to evals, update evals to support user speaking first
2025-08-22 12:12:17 -04:00
Aleix Conchillo Flaqué
28c729cc36
replace asyncio.wait_for for wait_for2.wait_for
2025-08-20 15:26:57 -07:00
Aleix Conchillo Flaqué
f387776985
add custom asyncio.wait_for()
...
This patch uses `wait_for2` package to implement `asyncio.wait_for()` for
Python < 3.12.
In Python 3.12, `asyncio.wait_for()` is implemented in terms of
`asyncio.timeout()` which fixed a bunch of issues. However, this was never
backported (because of the lack of `async.timeout()`) and there are still many
remainig issues, specially in Python 3.10, in `async.wait_for()`.
See https://github.com/python/cpython/pull/98518
2025-08-20 14:09:05 -07:00
Mark Backman
b7ae2989ac
Add foundational 14w-function-calling.py
2025-08-14 10:00:46 -04:00
Mark Backman
901b6b5913
Add foundational 19b
2025-08-13 11:37:38 -04:00
Mark Backman
e720573e60
Added 07n-interruptible-gemini
2025-08-12 14:54:49 -04:00
Aleix Conchillo Flaqué
1cfbfcaf11
scripts(evals): add vision support
2025-08-11 20:06:24 -07:00
Aleix Conchillo Flaqué
b360cb31dc
scripts(evals): add multilinguag support and 15 series
2025-08-11 15:21:14 -07:00
Aleix Conchillo Flaqué
c6ef8de578
scripts(evals): fix 14v-function-calling-openai.py
2025-08-07 17:57:47 -07:00
Mark Backman
31acfaa091
Merge pull request #2388 from pipecat-ai/14v-adding-openai-stt-tts-llm-functioncalling
...
14v adding OpenAI stt tts llm functioncalling
2025-08-07 10:22:35 -07:00
Mark Backman
69541c8835
Linting fix, plus update eval suite with 14v and others, tiny fix for 14m, too
2025-08-07 13:20:45 -04:00
Mark Backman
4288b5e780
Merge pull request #2381 from pipecat-ai/aleix/runner-args-pipeline-idle-timeout
...
allow specifying PipelineTask idle timeout to runner arguments
2025-08-07 04:47:08 -07:00
Aleix Conchillo Flaqué
7ed14ad91f
scripts(evals): added soniox, inworld and asyncai
2025-08-06 23:14:50 -07:00
Aleix Conchillo Flaqué
38951aab56
scripts(evals): use RunnerArguments.pipeline_idle_timeout_secs
2025-08-06 19:37:29 -07:00
Aleix Conchillo Flaqué
ed8b0655a8
scripts(evals): fix runner eval cancellation
...
We need to call asyncio.gather() just once, not for every cancelled task.
2025-08-06 19:36:42 -07:00
Aleix Conchillo Flaqué
61f4dd2ff2
scripts(evals): fix 14e-function-calling-google
2025-08-05 17:44:45 -07:00
Aleix Conchillo Flaqué
52a6d8013c
scripts(evals): pass runner arguments to run_bot()
2025-08-05 16:13:32 -07:00
Mark Backman
f26bbe4092
Merge pull request #2363 from pipecat-ai/mb/update-14p
...
Update 14p, add 14p to evals, add Google creds to env.example
2025-08-05 14:00:13 -07:00
Aleix Conchillo Flaqué
5546c8e01c
scripts(evals): update to use new runner function
2025-08-05 11:46:28 -07:00
Mark Backman
ba5a5c7187
Update 14p, add 14p to evals, add Google creds to env.example
2025-08-05 13:30:36 -04:00
Mark Backman
b9a2a9b729
Add Tavus, HeyGen, Simli to release-evals
2025-08-02 09:35:06 -04:00
Mark Backman
a48f5d5796
Make fix-ruff.sh more flexible, use pyproject rules
2025-07-09 11:48:17 -07:00
Aleix Conchillo Flaqué
baa878272d
scripts(evals): added 07a-interruptible-speechmatics.py
2025-07-03 13:44:21 -07:00
Aleix Conchillo Flaqué
093285868e
scripts(evals): update timeout back to 90 seconds
2025-07-03 13:37:17 -07:00
Aleix Conchillo Flaqué
1a8d512abb
scripts(evals): make sure we cancel pending tasks after timeout
2025-07-03 10:01:42 -07:00
Filipi Fuchter
3de271161c
Fixing the ruff script to also try to fix docstrings.
2025-07-02 16:19:57 -03:00
Mark Backman
fd570b0377
Update the remaining docstrings, update pre-commit hook, add docstring formatting CI, update CONTRIBUTING with formatting guidance ( #2089 )
2025-07-01 00:37:04 -04:00
Mark Backman
43a24d15f6
Add 40-aws-nova-sonic to release evals list
2025-06-27 08:34:39 -04:00
Mark Backman
9146def21b
Update examples to use default allow_interruptions, fixes to align examples
2025-06-19 10:07:32 -04:00
Filipi Fuchter
fa15e64fc9
Test script that mimics the behavior expected to be supported by Tavus.
2025-06-18 07:45:38 -03:00
Aleix Conchillo Flaqué
2fcfb0aa9f
evals: don't use Deepgram's smart formatting
2025-05-30 16:55:55 -07:00
Aleix Conchillo Flaqué
f1df079512
evals: allow running a single eval
2025-05-30 16:55:55 -07:00
Aleix Conchillo Flaqué
d77bedbafb
evals: move scripts/release to script/evals and add README
2025-05-30 15:04:05 -07:00
Aleix Conchillo Flaqué
e9aeb2662b
scripts: allow specifying a name for the test run
2025-05-28 00:22:55 -07:00
Aleix Conchillo Flaqué
356f4039e4
scripts: allow storing logs for release evals
2025-05-27 21:10:22 -07:00
Aleix Conchillo Flaqué
736c7f1f30
scripts: allow storing audio for release evals
2025-05-27 18:09:25 -07:00
Aleix Conchillo Flaqué
2994448036
introduce release evals
...
This is an initial attempt to implement evals for all (or most) of our
foundational examples. Before we release, we want to make sure all of them work
and reply properly. Until now this has been done manually, hopefully this will
be useful to speed up our release process.
2025-05-27 17:42:52 -07:00
Filipi Fuchter
0e55db054e
Created script to fix ruff format issues.
2025-03-05 14:10:47 -03:00
Aleix Conchillo Flaqué
f78520f7d0
add pre-commit hooks
...
Fixes #945
2025-01-15 13:44:21 -08:00
