Move workflow evaluations to standalone runner

This commit is contained in:
Xin Wang
2026-07-29 10:35:45 +08:00
parent 3187713bfe
commit e6437ab332
9 changed files with 311 additions and 171 deletions

48
evals/README.md Normal file
View File

@@ -0,0 +1,48 @@
# Workflow evaluations
The evaluation runner calls a live FastAPI service backed by FastGPT. For now,
each conversation step checks only the SSE `stage_code.nextStageCode` value.
## Case format
Add YAML files anywhere under `evals/cases/`:
```yaml
name: browser_addon_example
clientMode: browser_addon
steps:
- input: hi
expect:
stageCode: browser_addon.1001
- input: 继续
expect:
stageCode: browser_addon.1002
```
Steps in one file share the same generated `sessionId`. Different cases always
use different session IDs.
## Run
Start the API, then execute:
```bash
python evals/run.py
```
Use another API address:
```bash
python evals/run.py --base-url http://192.168.1.10:8000
```
Run selected cases:
```bash
python evals/run.py --case browser_addon
```
The process exits with code `1` when an expectation fails and code `2` when
case configuration is invalid.