EasyCom no-box full-frame Qwen AV: F2/F3

Definition

This experiment adds full-frame Qwen vision-encoder evidence to the existing strictly causal global Omni plus recent Qwen-audio baseline. It never reads face/head boxes, track IDs, active-speaker labels or speech labels.

F2: G(t) + A(t) query latest full-frame Qwen spatial tokens before t

F3: same-grid-position tokens from the latest causal frames
      -> shared chronological GRU
      -> G(t) + A(t) queries the temporally encoded spatial grid

At the default 2 FPS, F3 keeps two observations per grid position: at most the latest one second of visual history. G already contains global audio/video context, so this tests whether an explicit recent encoder readout adds timely evidence. It is not an audio-free Omni baseline.

F2/F3 are no-box full-frame visual-context models. Improvement does not by itself establish speaker grounding, identity or addressee reasoning.

Causal contract

extract_fullframe_qwen_av_features.py creates qwen-fullframe-av-tokens-v1 sidecars:

a cutoff t sees only fixed-grid frames with timestamp strictly less than t; a frame exactly at t is excluded;
images are complete EasyCom camera frames after the standard session-banner crop;
Qwen post-merger spatial tokens are stored without mean pooling;
F3 links only the same merged-grid row/column within one source clip, and resets on clip boundaries, grid changes or excessive time gaps;
audio is copied from the existing audited [t-1,t) Qwen mixed-audio sidecar; its person-crop tensors are never consumed;
visual/audio model identity, config digest and Transformers version must match exactly;
partial smoke sidecars are rejected by training.

The targets remain the unchanged 5 Hz background/wearer/others labels and the ten 0.2--2.0-second horizons.

1. Four-second GPU smoke extraction

cd /data1/yfzx/wangx/Code/icassp2027-experiment

DATA_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
MODEL=/data1/yfzx/wangx/Models/Qwen2.5-Omni-3B
FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
AUDIO_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
SMOKE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_smoke

CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
  --data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
  --output-root "$SMOKE_ROOT" --samples 4 --max-steps 20 \
  --model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
  --flash-attention-2 --video-fps 2 --max-pixels 50176 \
  --image-batch-size 8

Expected: 20 states spanning 0.2--4.0 s; eight visual frames at 0.0,...,3.5; finite nonempty spatial tokens; matched 2048D audio; and partial=True. Never train on this smoke file.

If the audio sidecars were made with an explicit --encoder-id, pass that exact value here. Otherwise omit it, as above, so both branches use the same recorded local checkpoint path.

Run the regression tests in the remote environment:

python -m unittest -v test_fullframe_qwen_av test_local_turn_features

They verify strict boundary exclusion, same-grid F3 history, malformed and partial-contract rejection, and F2/F3 forward/backward behavior.

2. Extract all sessions

Use a fresh root. Existing files are not overwritten unless --overwrite is explicitly provided.

FULLFRAME_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_2fps_224

CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
  --data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
  --output-root "$FULLFRAME_ROOT" \
  --samples 1 2 3 4 5 6 7 8 9 10 11 12 \
  --model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
  --flash-attention-2 --video-fps 2 --max-pixels 50176 \
  --image-batch-size 8

On interruption, rerun only missing session numbers. Do not mix checkpoints, Transformers versions, FPS settings or pixel budgets in one root.

3. Train and evaluate

The launcher uses Sessions 4--12 for four fixed epochs and evaluates Sessions 3/2/1 with the existing LSTR-MCE ten-horizon protocol.

export FEATURE_ROOT TARGET_ROOT FULLFRAME_ROOT
export CHECKPOINT_ROOT=checkpoints/easycom_fullframe_qwen_av
export RESULT_ROOT=results/easycom_fullframe_qwen_av

CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 0

After seed 0 passes shape/loss/inference checks:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 1 2
CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 1 2

Defaults are train batch 64, inference batch 128, F3 history 2 frames and maximum inter-frame link gap 0.75 s. Lower TRAIN_BATCH_SIZE or INFER_BATCH_SIZE if needed; do not change the feature contract.

The checkpoint-compatible option remains named --person-history-frames; in F3 it means observations per spatial grid position, not person history.

Required comparison and claim boundary

Compare matched seeds for G+A, F2 and F3. Add a past time-shifted full-frame control before attributing gains to AV synchronization. F3 contains a GRU that F2 does not, so F3–F2 combines extra history and temporal capacity; a parameter-matched audio/query temporal control is required before claiming a visual-motion mechanism.

Dense AP/mAP is an architecture screen. Floor-entry/yield interpretations remain timing-derived proxies until the competing-risk real-data audit is frozen.

5.7 KiB Raw Blame History Unescape Escape