Add floor-event timing objective and addressee diagnostics

This commit is contained in:
Xin Wang
2026-09-11 00:15:34 +08:00
parent 5937971222
commit a57e827606
14 changed files with 2144 additions and 234 deletions

View File

@@ -32,6 +32,7 @@ tests.
| [`easycom_muvap_stage3_baseline.md`](easycom_muvap_stage3_baseline.md) | MuVAP-only frozen VAP+ASD Stage-3 paper reconstruction on EgoSpeak/EasyCom targets |
| [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |
| [`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md) | Running and auditing the competing-risk entry/yield evaluator with training-side threshold calibration |
| [`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md) | Separating boundary observability, event-objective mismatch and addressee information with remote commands |
## Design history and implementation options

View File

@@ -46,6 +46,13 @@ For each event, predictions made from the same source time are max-pooled only
over the observable event window. This is a ranking score, not a calibrated
event probability.
The default `--decision-timing pre-boundary` uses the last row before the
aggregate-other transition and remains the primary anticipation protocol. The
optional `--decision-timing post-boundary` moves the decision source to the
first row after the transition. It is a separately identified reactive diagnostic,
not a replacement metric; see
[`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md).
## Confirmatory test command
Select each model's operating threshold on training-side predictions, then

View File

@@ -0,0 +1,142 @@
# EasyCom floor-event diagnosis: timing, objective, and addressee
These three experiments explain why a dense future-activity oracle can retain
reasonable mAP while response/yield event AP remains low. They are diagnostics,
not three proposed paper contributions. Keep the primary paper protocol as the
pre-boundary competing-risk evaluation until the evidence says otherwise.
## What each experiment isolates
| Diagnostic | Only intended change | If it improves clearly |
|---|---|---|
| pre-boundary vs post-boundary | whether the aggregate-other boundary has already been observed at the decision source | much of the error is boundary anticipation, not weak AV encoding |
| dense activity head vs direct event head | future-state loss vs response/yield loss, using the same frozen inputs | the dense objective/ranking conversion is the main mismatch |
| addressee vs zero-channel control | explicit past `Target_of_Speech == wearer_id` history | aggregate activity misses conversational routing information |
Do not compare raw AP across the pre- and post-boundary protocols as if they
were the same task. Compare methods within one timing protocol, and use the
pre/post difference only as a difficulty decomposition.
## Shared remote environment
```bash
cd /data1/yfzx/wangx/Code/icassp2027-experiment
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
```
Change only paths that differ on the remote host. The default feature mode is
`open-stream-replay`.
## 1. Post-boundary reactive diagnostic for the three existing runs
Point the wrapper at already aligned inference directories. Each directory
must contain `pred/Session_*.npy` and `valid/Session_*.npy`.
```bash
export POOLED_AV_RUN=/path/to/pooled_av_result
export ORACLE_RUN=/path/to/aggregate_visible_oracle_result
export MINIROAD_RUN=/path/to/converted_miniroada_result
bash run_easycom_post_boundary_eval.sh
```
Without `CALIBRATION_SESSIONS`, the wrapper explicitly marks threshold results
as exploratory and selects them in-sample. AP is still useful for this first
diagnosis. For a confirmatory result, provide training-side predictions in the
same run directories and use, for example:
```bash
export CALIBRATION_SESSIONS="Session_12"
bash run_easycom_post_boundary_eval.sh
```
The JSON protocol ID becomes
`wearer_floor_event_post_boundary_diagnostic_v1`. Simultaneous wearer changes
at the observed boundary and zero-length windows before an immediate competing
boundary are excluded and counted explicitly.
## 2. Direct response/yield event head
The direct probe uses frozen Omni states, the same mixed-audio vectors and the
same `aggregate-visible` past-activity oracle. Its layer fusion, audio
projection, oracle person attention and shared MLP trunk are the same
`TurnPredictionHead` components as the dense oracle; only the final two-event
readout and event loss differ. It trains on Sessions 4--11, selects epoch and
thresholds on Session 12, and initially evaluates only Session 12. First run a
one-epoch Session-11/12 shape, mask and checkpoint smoke test:
```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh smoke
```
Then run the single aggregate oracle diagnosis:
```bash
CUDA_VISIBLE_DEVICES=0 \
VARIANTS="aggregate-visible" \
bash run_easycom_floor_event_probes.sh all 0
```
This is the clean comparison against the existing dense aggregate-visible
oracle head. A large direct-head gain supports objective mismatch; a small gain
means the oracle information itself is probably weak for the event decision.
## 3. Addressee oracle with a capacity-matched control
Run all three direct-probe variants:
```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh all 0
```
The variants are:
- `aggregate-visible`: 2 s aggregate activity plus observation mask;
- `aggregate-visible-address-control`: the same input plus an all-zero channel;
- `aggregate-visible-addressee`: the same input plus past bins in which a
visible non-wearer was explicitly annotated as addressing wearer ID 2.
`Group`, `None`, unknown and other-person targets are not coerced into reliable
negative labels. The addressee channel is privileged, annotation-derived and
analysis-only; it is not an online feature or a deployable causal input.
The runner writes one `summary.json` and `metrics.csv` per variant and a compact
`comparison_dev.csv`. Inspect that comparison before touching Sessions 1--3.
## Optional frozen test evaluation
Only after the variants, epochs and interpretation rule are fixed from Session
12, apply the saved probes to official test Sessions 1--3:
```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh test 0
```
This reuses the saved Session-12 thresholds and writes `comparison_test.csv`.
Do not repeatedly inspect test results and redesign the probe.
## Minimum interpretation rule
For response, jointly inspect mean AP, 2 s AP, achieved response recall, false
entry rate, and short-resume trigger rate. For yield, inspect mean AP, 2 s AP,
achieved yield recall, and false-stop rate. Denominators and censoring counts
must match before comparing variants.
The three outcomes lead to different paper decisions:
1. **Only post-boundary improves:** retain the main anticipation task and frame
the difficulty as predicting a boundary before it is observable. Do not
claim that a larger event head solves grounding.
2. **Direct event supervision improves:** the main lesson is target alignment;
build the method around event-aware adaptation and keep dense mAP secondary.
3. **Addressee improves over both aggregate and zero control:** conversational
routing is the missing information. This motivates a deployable addressee or
addressedness estimator, but the oracle itself is not the method.
4. **None improves:** the current timing-derived event definition or available
observations are insufficient. Revisit annotation/task definition before a
heavier fusion architecture.

View File

@@ -15,10 +15,12 @@ This is an information upper-bound experiment, not a deployable input setting.
`0.2, 0.4, 0.6, ...` second cutoffs, with
`Start_Frame <= marker < End_Frame`. This is right-edge point sampling, not
an any-overlap reduction over a 200 ms bin.
- Each person vector stores the last 2 seconds of binary activity followed by
a visibility/availability mask. Activity is exposed only in bins where that
participant was visible according to the provided boxes. No future bin,
future wearer state, transcript text or `Target_of_Speech` value is input.
- Each original person vector stores the last 2 seconds of binary activity
followed by a visibility/availability mask. Activity is exposed only in bins
where that participant was visible according to the provided boxes. No future
bin, future wearer state or transcript text is input. The separately named
addressee diagnostic below additionally uses privileged `Target_of_Speech`;
it must not be described as an online/deployable feature.
- Only non-wearer participant IDs present in the latest provided-box frame
strictly before the cutoff receive slots. IDs route annotations during
preparation but are not exposed as embeddings to the head.
@@ -35,6 +37,8 @@ This is an information upper-bound experiment, not a deployable input setting.
| `audio` | Omni `G` plus the existing recent local audio `A` | deployable baseline |
| `visibility-only` | current visible-person set, but all activity bits are zero | parameter/person-count control |
| `aggregate-visible` | one OR-pooled history over currently visible people | aggregate floor-information upper bound |
| `aggregate-visible-address-control` | aggregate-visible plus an all-zero history channel | capacity control for addressee input |
| `aggregate-visible-addressee` | aggregate-visible plus past direct-to-wearer annotation bins | analysis-only addressee information upper bound |
| `per-person` | one correct activity history per currently visible participant | main oracle information test |
| `shuffled-id` | preserves the number active at every lag but independently reassigns activity between people | destroys identity continuity |
| `delayed` | correct per-person histories delayed by 1 second | destroys current temporal alignment without future leakage |
@@ -43,6 +47,12 @@ The decisive comparison is `per-person` against `aggregate-visible`,
`shuffled-id` and `delayed`, not only against `audio`. A gain over `audio` alone
could come from perfect aggregate VAD rather than person continuity.
For the separate addressee question, the decisive comparison is
`aggregate-visible-addressee` against both `aggregate-visible` and
`aggregate-visible-address-control`. Group/None/unknown targets are audited and
not treated as trustworthy semantic negatives. The full diagnostic commands
are in [`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md).
## Remote preparation and one-seed Session-12 screen
The source local sidecars provide only mixed audio and the visibility/box