From d77bedbafb15ceedfda4c107193db7dd5c6eb774 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Aleix=20Conchillo=20Flaqu=C3=A9?= <aleix@daily.co>
Date: Fri, 30 May 2025 15:03:29 -0700
Subject: [PATCH] evals: move scripts/release to script/evals and add README

---
 scripts/evals/README.md                       | 45 +++++++++++++++++++
 scripts/{release => evals}/eval.py            |  0
 .../{release => evals}/run-release-evals.py   |  0
 scripts/{release => evals}/utils.py           |  0
 4 files changed, 45 insertions(+)
 create mode 100644 scripts/evals/README.md
 rename scripts/{release => evals}/eval.py (100%)
 rename scripts/{release => evals}/run-release-evals.py (100%)
 rename scripts/{release => evals}/utils.py (100%)

diff --git a/scripts/evals/README.md b/scripts/evals/README.md
new file mode 100644
index 000000000..d22102f4b
--- /dev/null
+++ b/scripts/evals/README.md
@@ -0,0 +1,45 @@
+# Pipecat Evals
+
+This directory contains a set of utilities to help test Pipecat, specifically
+its examples.
+
+## Release Evals
+
+Before any Pipecat release, we make sure that all (or most) of the examples work
+flawlessly.  We have 100+ examples, and checking each one manually was very
+time-consuming (and painful!), especially because we aim to release often.
+
+To make this process easier, we designed these "release evals," which do the
+following:
+
+- Start one of the foundational examples (the user bot)
+- Start an eval bot
+
+The user bot (i.e. the example) introduces itself, and the eval bot then asks a
+question. The user bot replies, and the eval bot verifies the response.
+
+For example, the eval bot might ask:
+
+"What's 2 plus 2?"
+
+The user bot replies:
+
+"2 plus 2 is 4."
+
+The eval bot (powered by an LLM) evaluates the response and emits a result.  It
+also explains why it thinks the answer is valid or invalid.
+
+To run the release evals:
+
+```sh
+python run-release-evals.py -a -v
+```
+
+This runs all the evals and stores logs and audio (`-a`) for each test.
+
+You can also specify which tests to run. For example, to run all `07` series
+tests:
+
+```sh
+python run-release-evals.py -p 07 -a -v
+```
diff --git a/scripts/release/eval.py b/scripts/evals/eval.py
similarity index 100%
rename from scripts/release/eval.py
rename to scripts/evals/eval.py
diff --git a/scripts/release/run-release-evals.py b/scripts/evals/run-release-evals.py
similarity index 100%
rename from scripts/release/run-release-evals.py
rename to scripts/evals/run-release-evals.py
diff --git a/scripts/release/utils.py b/scripts/evals/utils.py
similarity index 100%
rename from scripts/release/utils.py
rename to scripts/evals/utils.py