Refine prompts and evaluation cases for accident reporting AI

- Updated prompts to enhance clarity and consistency in user interactions, particularly focusing on safety prioritization and accurate information collection.
- Introduced new YAML evaluation cases to test various user input scenarios, ensuring robust handling of short, ambiguous, or irrelevant responses.
- Enhanced guidelines for transitioning to human assistance, emphasizing strict conditions for triggering state changes based on user responses.

evals/README.md

@@ -6,7 +6,7 @@ During the run it also prints the live dialogue and saves transcripts.
 
 ## Case format
 
-Add YAML files anywhere under `evals/cases/`:
+Add YAML files anywhere under `evals/cases/` (e.g. `direct/`, `browser_addon/`, `no_false/` for false-interrupt / false-transfer checks):
 
 ```yaml
 name: direct_initial_stage
@@ -45,6 +45,7 @@ Run selected cases:
 ```bash
 python evals/run.py --case direct
 python evals/run.py --case browser_addon
+python evals/run.py --case no_false
 ```
 
 ## Dialogue and transcripts

evals/cases/no_false/asr_no_false_0002.yaml

@@ -0,0 +1,25 @@
+name: direct_asr_no_false_0002
+clientMode: direct
+
+# Short/garbled answers while collecting accident description must stay on 1002,
+# not jump to 0002 after a few unclear turns.
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 不用管
+    expect:
+      stageCode: "1002"
+
+  - input: 你知吧
+    expect:
+      stageCode: "1002"

evals/cases/no_false/asr_no_false_0003.yaml

@@ -0,0 +1,20 @@
+name: direct_asr_no_false_0003
+clientMode: direct
+
+# ASR damage/garbled words must not be treated as injury/complex and jump to 0003.
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 知不懂
+    expect:
+      stageCode: "1002"
+
+  - input: 不，刚摧毁了
+    expect:
+      stageCode: "1002"

evals/cases/no_false/confirm_dual_no_false_0002.yaml

@@ -0,0 +1,89 @@
+name: direct_confirm_dual_no_false_0002
+clientMode: direct
+
+# During dual-car info confirm (3002 -> external 1002), short/garbled answers
+# must stay on 1002, not jump to 0002 after a few unclear turns.
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前两车追尾
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 是的
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "2010"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2011"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2012"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2013"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2014"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2015"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2016"
+
+  - input: 没错
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 李四
+    expect:
+      stageCode: "1002"
+
+  - input: 这个
+    expect:
+      stageCode: "1002"
+
+  - input: 试试
+    expect:
+      stageCode: "1002"
+
+  - input: 什么意思
+    expect:
+      stageCode: "1002"

evals/cases/no_false/confirm_dual_no_false_0003.yaml

@@ -0,0 +1,81 @@
+name: direct_confirm_dual_no_false_0003
+clientMode: direct
+
+# During dual-car info confirm, ASR damage/garbled injury-like words must not
+# jump to 0003; stay on 1002 (closed confirm / continue current question).
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前两车追尾
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 是的
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "2010"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2011"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2012"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2013"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2014"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2015"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2016"
+
+  - input: 没错
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 不，刚摧毁了
+    expect:
+      stageCode: "1002"
+
+  - input: 伤……没有吧
+    expect:
+      stageCode: "1002"

evals/cases/no_false/confirm_single_no_false_0002.yaml

@@ -0,0 +1,81 @@
+name: direct_confirm_single_no_false_0002
+clientMode: direct
+
+# During single-car info confirm (3001 -> external 1002), short/garbled answers
+# must stay on 1002, not jump to 0002 after a few unclear turns.
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前一辆车撞到护栏
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+      
+  - input: 对
+    expect:
+      stageCode: "2000"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2001"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2002"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2003"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2004"
+
+  - input: 是的
+    expect:
+      stageCode: "2005"
+
+  - input: 前方
+    expect:
+      stageCode: "1002"
+
+  - input: 是的
+    expect:
+      stageCode: "1002"
+
+  - input: 张三
+    expect:
+      stageCode: "1002"
+
+  - input: 这个
+    expect:
+      stageCode: "1002"
+
+  - input: 试试
+    expect:
+      stageCode: "1002"
+
+  - input: 什么意思
+    expect:
+      stageCode: "1002"

evals/cases/no_false/confirm_single_no_false_0003.yaml

@@ -0,0 +1,73 @@
+name: direct_confirm_single_no_false_0003
+clientMode: direct
+
+# During single-car info confirm, ASR damage/garbled injury-like words must not
+# jump to 0003; stay on 1002 (closed confirm / continue current question).
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前一辆车撞到护栏
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+      
+  - input: 对
+    expect:
+      stageCode: "2000"
+      
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2001"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2002"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2003"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2004"
+
+  - input: 是的
+    expect:
+      stageCode: "2005"
+
+  - input: 前方
+    expect:
+      stageCode: "1002"
+
+  - input: 是的
+    expect:
+      stageCode: "1002"
+
+  - input: 不，刚摧毁了
+    expect:
+      stageCode: "1002"
+
+  - input: 伤……没有吧
+    expect:
+      stageCode: "1002"

evals/cases/no_false/photo_dual_no_false_0002.yaml

@@ -0,0 +1,77 @@
+name: direct_photo_dual_no_false_0002
+clientMode: direct
+
+# At dual-car plate confirm (2016), short/garbled answers must stay on 2016,
+# not jump to 0002 after a few unclear turns.
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前两车追尾
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 是的
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "2010"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2011"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2012"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2013"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2014"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2015"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2016"
+
+  - input: 这个
+    expect:
+      stageCode: "2016"
+
+  - input: 试试
+    expect:
+      stageCode: "2016"
+
+  - input: 什么意思
+    expect:
+      stageCode: "2016"

evals/cases/no_false/photo_single_no_false_0002.yaml

@@ -0,0 +1,81 @@
+name: direct_photo_single_no_false_0002
+clientMode: direct
+
+# At single-car plate/damage confirm, short/garbled answers must stay on 2004/2005,
+# not jump to 0002 after a few unclear turns.
+steps:
+  - input: 新对话
+    expect:
+      stageCode: "1001"
+
+  - input: 继续办理
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前一辆车撞到护栏
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 没有
+    expect:
+      stageCode: "1002"
+
+  - input: 十分钟之前
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "1002"
+
+  - input: 是的
+    expect:
+      stageCode: "1002"
+
+  - input: 对
+    expect:
+      stageCode: "2000"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2001"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2002"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2003"
+
+  - input: 【拍摄完成】
+    expect:
+      stageCode: "2004"
+
+  - input: 这个
+    expect:
+      stageCode: "2004"
+
+  - input: 试试
+    expect:
+      stageCode: "2004"
+
+  - input: 是的
+    expect:
+      stageCode: "2005"
+
+  - input: 这个
+    expect:
+      stageCode: "2005"
+
+  - input: 试试
+    expect:
+      stageCode: "2005"
+
+  - input: 什么意思
+    expect:
+      stageCode: "2005"