EasyCom causal oracle-person information screen

Question

Before training a heavier visual grounding model, test whether perfect participant-resolved past speaking activity contains useful information for future wearer activity beyond the frozen Omni state and recent mixed audio. This is an information upper-bound experiment, not a deployable input setting.

Causal contract

The task grid remains 5 Hz. State row t has cutoff (t+1)/5 seconds.
Oracle state t follows EgoSpeak's released-target conversion exactly: source-frame markers 4, 8, 12, ... are sampled at the corresponding 0.2, 0.4, 0.6, ... second cutoffs, with Start_Frame <= marker < End_Frame. This is right-edge point sampling, not an any-overlap reduction over a 200 ms bin.
Each person vector stores the last 2 seconds of binary activity followed by a visibility/availability mask. Activity is exposed only in bins where that participant was visible according to the provided boxes. No future bin, future wearer state, transcript text or Target_of_Speech value is input.
Only non-wearer participant IDs present in the latest provided-box frame strictly before the cutoff receive slots. IDs route annotations during preparation but are not exposed as embeddings to the head.
Transcript-derived wearer/others activity must match target_perframe exactly. Any mismatch fails closed; there is no tolerance flag.
Oracle sidecars use easycom-oracle-person-history-v1 and are accepted only by the explicit oracle-frame-set-attention mode. Normal local-feature loading rejects them because they use speech labels.

Conditions

Condition	Information exposed	Purpose
`audio`	Omni `G` plus the existing recent local audio `A`	deployable baseline
`visibility-only`	current visible-person set, but all activity bits are zero	parameter/person-count control
`aggregate-visible`	one OR-pooled history over currently visible people	aggregate floor-information upper bound
`per-person`	one correct activity history per currently visible participant	main oracle information test
`shuffled-id`	preserves the number active at every lag but independently reassigns activity between people	destroys identity continuity
`delayed`	correct per-person histories delayed by 1 second	destroys current temporal alignment without future leakage

The decisive comparison is per-person against aggregate-visible, shuffled-id and delayed, not only against audio. A gain over audio alone could come from perfect aggregate VAD rather than person continuity.

Remote preparation and one-seed Session-12 screen

The source local sidecars provide only mixed audio and the visibility/box timeline. Their learned visual vectors are not copied into the oracle input.

cd /data1/yfzx/wangx/Code/icassp2027-experiment

export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
export CHECKPOINT_ROOT=checkpoints/easycom_oracle_information_4ep
export RESULT_ROOT=results/easycom_oracle_information_4ep

CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh all 0

all prepares the five oracle conditions, trains on Sessions 4--11, selects the checkpoint on Session 12, runs Session-12 inference and prints the standard dense AP/mAP screen. Existing compatible oracle sidecars are validated and reused, so interrupted preparation can resume. It deliberately does not run Sessions 1--3.

If preparation reports any transcript/target mismatch, stop and inspect it; the preparation command has no mismatch-tolerance option.

Follow-up

If per-person is consistently better on Session 12, add seeds 1 and 2 using the already prepared sidecars:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh screen 1 2

Only after freezing the method and event evaluator should selected variants be run once on Sessions 1--3:

VARIANTS="audio aggregate-visible per-person shuffled-id delayed" \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_oracle_information.sh test 0 1 2

Interpretation

per-person > aggregate-visible and per-person > shuffled-id: speaker continuity has information value; proceed to learned causal ASD/tracks.
aggregate-visible > audio, but no per-person advantage: perfect aggregate activity helps, but the current target does not justify an identity claim.
oracle conditions do not improve: do not spend the main GPU budget on a more complex person encoder until the task/head/evaluator is reconsidered.
oracle improves but learned Qwen face features do not: this supports a speaker-grounding bottleneck and motivates EgoASD-like AV evidence.

Dense 10-horizon mAP is only a screening signal. The publication claim still requires the frozen competing-risk-censored entry/yield evaluator and matched wearer-onset-recall false-entry analysis.

5.2 KiB Raw Blame History