103 lines
5.0 KiB
Markdown
103 lines
5.0 KiB
Markdown
# EasyCom causal oracle-person information screen
|
|
|
|
## Question
|
|
|
|
Before training a heavier visual grounding model, test whether perfect
|
|
participant-resolved **past** speaking activity contains useful information for
|
|
future wearer activity beyond the frozen Omni state and recent mixed audio.
|
|
This is an information upper-bound experiment, not a deployable input setting.
|
|
|
|
## Causal contract
|
|
|
|
- The task grid remains 5 Hz. State row `t` has cutoff `(t+1)/5` seconds.
|
|
- Oracle bin `t` summarizes transcript frames in
|
|
`[t/5,(t+1)/5)` and is therefore completely observed at that cutoff.
|
|
- Each person vector stores the last 2 seconds of binary activity followed by
|
|
a visibility/availability mask. Activity is exposed only in bins where that
|
|
participant was visible according to the provided boxes. No future bin,
|
|
future wearer state, transcript text or `Target_of_Speech` value is input.
|
|
- Only non-wearer participant IDs present in the latest provided-box frame
|
|
strictly before the cutoff receive slots. IDs route annotations during
|
|
preparation but are not exposed as embeddings to the head.
|
|
- Transcript-derived wearer/others activity must match `target_perframe` by
|
|
default. Any mismatch fails closed instead of being silently tolerated.
|
|
- Oracle sidecars use `easycom-oracle-person-history-v1` and are accepted only
|
|
by the explicit `oracle-frame-set-attention` mode. Normal local-feature
|
|
loading rejects them because they use speech labels.
|
|
|
|
## Conditions
|
|
|
|
| Condition | Information exposed | Purpose |
|
|
|---|---|---|
|
|
| `audio` | Omni `G` plus the existing recent local audio `A` | deployable baseline |
|
|
| `visibility-only` | current visible-person set, but all activity bits are zero | parameter/person-count control |
|
|
| `aggregate-visible` | one OR-pooled history over currently visible people | aggregate floor-information upper bound |
|
|
| `per-person` | one correct activity history per currently visible participant | main oracle information test |
|
|
| `shuffled-id` | preserves the number active at every lag but independently reassigns activity between people | destroys identity continuity |
|
|
| `delayed` | correct per-person histories delayed by 1 second | destroys current temporal alignment without future leakage |
|
|
|
|
The decisive comparison is `per-person` against `aggregate-visible`,
|
|
`shuffled-id` and `delayed`, not only against `audio`. A gain over `audio` alone
|
|
could come from perfect aggregate VAD rather than person continuity.
|
|
|
|
## Remote preparation and one-seed Session-12 screen
|
|
|
|
The source local sidecars provide only mixed audio and the visibility/box
|
|
timeline. Their learned visual vectors are not copied into the oracle input.
|
|
|
|
```bash
|
|
cd /data1/yfzx/wangx/Code/icassp2027-experiment
|
|
|
|
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
|
|
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
|
|
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
|
|
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
|
|
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
|
|
export CHECKPOINT_ROOT=checkpoints/easycom_oracle_information_4ep
|
|
export RESULT_ROOT=results/easycom_oracle_information_4ep
|
|
|
|
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh all 0
|
|
```
|
|
|
|
`all` prepares the five oracle conditions, trains on Sessions 4--11, selects
|
|
the checkpoint on Session 12, runs Session-12 inference and prints the standard
|
|
dense AP/mAP screen. Existing compatible oracle sidecars are validated and
|
|
reused, so interrupted preparation can resume. It deliberately does not run
|
|
Sessions 1--3.
|
|
|
|
If preparation reports any transcript/target mismatch, stop and inspect it.
|
|
Do not raise `--max-label-mismatch-rows` merely to make the run continue.
|
|
|
|
## Follow-up
|
|
|
|
If `per-person` is consistently better on Session 12, add seeds 1 and 2 using
|
|
the already prepared sidecars:
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh screen 1 2
|
|
```
|
|
|
|
Only after freezing the method and event evaluator should selected variants be
|
|
run once on Sessions 1--3:
|
|
|
|
```bash
|
|
VARIANTS="audio aggregate-visible per-person shuffled-id delayed" \
|
|
CUDA_VISIBLE_DEVICES=0 \
|
|
bash run_easycom_oracle_information.sh test 0 1 2
|
|
```
|
|
|
|
## Interpretation
|
|
|
|
- `per-person > aggregate-visible` and `per-person > shuffled-id`: speaker
|
|
continuity has information value; proceed to learned causal ASD/tracks.
|
|
- `aggregate-visible > audio`, but no per-person advantage: perfect aggregate
|
|
activity helps, but the current target does not justify an identity claim.
|
|
- oracle conditions do not improve: do not spend the main GPU budget on a more
|
|
complex person encoder until the task/head/evaluator is reconsidered.
|
|
- oracle improves but learned Qwen face features do not: this supports a
|
|
speaker-grounding bottleneck and motivates EgoASD-like AV evidence.
|
|
|
|
Dense 10-horizon mAP is only a screening signal. The publication claim still
|
|
requires the frozen competing-risk-censored entry/yield evaluator and matched
|
|
wearer-onset-recall false-entry analysis.
|