icassp2027-experiment/PROJECT_CONTEXT.md

AI Project Context

This is the short, authoritative repository map for humans and AI agents. The long README.md is an operational history and command reference; it is not the experiment scheduler.

One-sentence objective

Determine whether strictly causal global Omni context plus local audiovisual floor evidence organized around visible people can improve an invisible egocentric wearer's floor-entry and floor-yield behavior in natural multiparty conversation.

The paper is not simply “replace EgoSpeak's encoders with a newer backbone.” The decisive test is whether correct person-aligned vision reduces false wearer entries during other-to-other conversation at matched wearer-onset recall, without hurting addressed responses or yield behavior.

Source-of-truth order

Need	Authoritative file
Current paper thesis and claim boundary	docs/global_local_omni_paper_position_2026-09-07.md
Progressive feature-fusion implementation route	docs/progressive_feature_fusion_route_2026-09-02.md
Current hypotheses, experiment gates and stopping rules	docs/omni_speaker_floor_experiment_plan_2026-08-30.md
Novelty boundary and claim restrictions	docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md
Code/data map and current blocker	This file
Runnable commands and implementation history	README.md and docs/README.md
Archived measured results	results/README.md
Literature summaries	literature_notes/README.md

When files disagree, prefer the newest scoped document and verify behavior in code/tests. A planned experiment is not an implemented feature or a measured result.

Current state

Verified assets

• Strictly causal 5 Hz Qwen2.5-Omni global state extraction on EasyCom.
• Prepared-media and direct-original-media extraction paths.
• Frozen head training/inference with EgoSpeak-compatible 10-horizon AP/mAP.
• EasyCom local Qwen visual crops and recent mixed-audio sidecars.
• Encoder-only A/V/AV and G, G+A, G+V, G+A+V baselines.
• Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
• Timing-derived wearer floor-state and event diagnostics.

Measured screening result

• F1 Frame-set Attention uses fused G+A as one query over the latest half-open causal person set. On the existing three-seed, four-epoch test screen it obtained 81.77745 +/- 0.15638 0.2-second mAP and 64.95099 +/- 0.09451 ten-horizon mean mAP. This is stable, but is not a matched improvement over the historical no-dev pooled result; correct/wrong visual controls remain missing.

Measured local-fusion screens

• Two current-sidecar screening controls: audio-person-frame-set adds a zero-gated audio-to-person read before the F1 query, and gated-person-state-adapter adds a zero-gated person residual after the frozen Thinker readout. They are explicitly recorded as F5/F6 screening, not full history-conditioned F5 or an internal Thinker F6 adapter.
• track-gru-residual builds strictly causal histories from anonymous provided track IDs, applies one shared GRU per track, lets G+A query current track states, and adds a zero-gated correction to an exactly initialized pooled G+A+V checkpoint. The matched Session-12 three-seed screen was null (+0.01306 +/- 0.02368 percentage points in ten-horizon mean mAP). A seed-0 Sessions-1--3 audit reproduced pooled predictions to within -0.00030 points. Saved residual gates were only -0.0015 to +0.0108; because a zero scalar gate also blocks the initial branch gradient, this is recorded as an optimization failure rather than evidence that track continuity is useless. See the immutable result record and implementation note.

Measured oracle-information screen

• A seed-0, four-epoch Session-12 development screen supplied transcript-derived current/past speech activity only through the explicit analysis-only oracle contract. aggregate-visible improved over the matched local-audio baseline by +6.96649 points at 0.2 seconds and +2.91047 points in ten-horizon mean mAP. Visibility-only and 1-second-delayed controls stayed near audio.
• Person identity continuity is not supported: aggregate-visible exceeded per-person, while per-person and shuffled-id were effectively tied. The result motivates a learned causal local active-speaker/floor-evidence branch, but does not justify a track-continuity claim. See the immutable result record.

Current blocker

evaluate_floor_state_changes.py counts a wearer transition anywhere in a fixed response window. Before this becomes the paper event protocol, add competing-risk censoring:

• an other-offset response window ends at the next other onset;
• an other-onset yield window ends at the next other offset.

Until then, EOT/HOLD, response and yield results are proxy diagnostics, not semantic event ground truth.

Next experiment gates

The user has temporarily chosen to run the dense-AP model screen while P0 is unfinished. This does not relax the paper claim: event numbers remain proxy diagnostics until competing-risk censoring is fixed.

1. P0: fix/freeze event attribution, splits and scenario definitions.
2. P1: confirm the aggregate/per-person oracle screen over seeds 1--2, then train a causal visible-person active-speaker/floor-evidence estimator; compare correct, wrong-person and time-shifted visual inputs.
3. P2: re-evaluate G, G+A, G+A+pooled-V under the frozen event protocol.
4. P3: only if learned person-organized evidence beats aggregate/no-ID controls, repair the matched anonymous-track-GRU residual optimization by replacing the scalar zero gate with a zero-initialized residual output layer, and verify nonzero branch gradients on Session 12. Add broken-ID, wrong-time and remove-person controls only if that corrected screen improves.
5. P4: attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.

The detailed architecture, controls and stopping gates for P3 are frozen in the progressive feature-fusion route.

Stable data contracts

Time and labels

• Sampling rate: 5 Hz (STEP_SECONDS = 0.2).
• State row j uses media no later than (j+1)/5 seconds.
• Future horizons: 10, corresponding to 0.2, 0.4, ..., 2.0 seconds.
• Class order: background, camera_wearer_speaking, other_people_speaking.
• EasyCom targets are dense [T, 3]; predictions are [target_time, class, future_horizon].
• EasyCom split: train Session 4--12; test Session 1--3.

Global feature sidecar

Produced by streaming_turn_model.py or streaming_turn_model_directly.py as a .pt dictionary. Required training fields include:

• states: [T, 2048] or multi-layer [T, L, 2048];
• times: [0.2, 0.4, ...];
• label_indices: dense zero-origin indices into the target timeline;
• extraction mode, configuration and source metadata.

Never silently accept non-zero origins, missing label indices or mismatched time grids.

Local feature sidecar

Produced by extract_turn_local_features.py, validated by local_turn_features.py, and stored as qwen-local-pooled-v1. It contains native 2048D visual/audio vectors, validity masks, times, label indices, source metadata and the exact extraction config. A partial smoke sidecar must not be reused as a full-session feature file.

Inference result

infer_turn_head.py writes:

RESULT_DIR/
  metrics.json
  pred/<sample>.npy       # [T, 3, 10]
  valid/<sample>.npy      # [T, 10]
  transition_pred/        # only when the auxiliary head exists

evaluate_floor_state_changes.py consumes pred/ and valid/; it does not run a GPU model.

Canonical pipelines

Direct global features (preferred semantic path)

original EasyCom/Ego4D media
  -> streaming_turn_model_directly.py
  -> global feature .pt
  -> train_turn_head.py
  -> checkpoint .pth + .json
  -> infer_turn_head.py
  -> pred/ + valid/ + metrics.json
  -> evaluate_floor_state_changes.py

This path samples source video causally in Python and avoids requiring a re-encoded fixed-FPS MP4.

Prepared-media compatibility path

prepare_easycom_session.py / prepare_ego4d_2fps.py
  -> strict 2 FPS media + metadata
  -> streaming_turn_model.py
  -> the same training/inference pipeline

Keep it for reproducibility, cache-replay verification and comparisons with already extracted features.

Local evidence path

raw media + person boxes + existing global reference .pt
  -> extract_turn_local_features.py
     (or extract_ego4d_local_multigpu.py supervisor)
  -> local .pt sidecar
  -> train_turn_head.py --local-feature-root ...
  -> infer_turn_head.py --local-feature-root ...

The current local visual feature is pooled. It does not by itself prove speaker identity, active-speaker alignment or addressee understanding.

Root-code map

Root scripts remain flat because remote L20 commands call them directly. Do not reorganize them physically without compatibility wrappers.

Shared libraries

File	Responsibility
easycom_head.py	Dataset constants, feature loading, heads, losses and AP metrics
local_turn_features.py	Local-sidecar contract, alignment, crops and pooling
omni_cached_av.py	Cached AV structures used by Omni extraction/LoRA paths

Global feature extraction

File	Responsibility
streaming_turn_model_directly.py	Preferred direct raw-media causal extraction
streaming_turn_model.py	Prepared-media causal-prefix/open-stream extraction and replay checks
qwen_omni_features.py	Single-window feature inspection
cache_omni_av_pieces.py	Build reusable AV pieces for LoRA experiments
extract_ego4d_batch.py	Persistent-model Ego4D global batch extraction
analyze_block_phase.py	Streaming block-phase diagnostics
profile_streaming_bottleneck_gpu7.py	GPU extraction profiling

Data preparation and audit

File	Responsibility
easycom_manifest.py	EasyCom clip inventory and dense-timeline validation
prepare_easycom_session.py	Optional strict-grid EasyCom media preparation
prepare_ego4d_2fps.py	Optional strict-grid Ego4D media preparation
prepare_encoder_only_features.py	Convert local sidecars into encoder-only references
audit_egospeak_features.py	Audit released EgoSpeak feature layout
derive_easycom_interaction_events.py	Deterministic transcript/timing proxies
adjudicate_easycom_interaction_events.py	Optional ambiguous-event adjudication
audit_easycom_floor_events.py	Count EOT/takeover/backchannel candidates
prepare_easycom_oracle_features.py	Build explicit past-only oracle person-activity sidecars for information-value screening

Local features

File	Responsibility
extract_turn_local_features.py	Qwen person-crop visual and recent-audio sidecars
extract_ego4d_local_multigpu.py	Foreground supervisor for disjoint multi-GPU workers

Training, inference and evaluation

File	Responsibility
train_turn_head.py	Frozen-backbone head training, optional local fusion/auxiliary losses
infer_turn_head.py	Save EgoSpeak-compatible predictions and AP/mAP
evaluate_floor_state_changes.py	CPU floor-state/event proxy evaluation
summarize_local_baselines.py	Aggregate local baseline metrics.json files
run_easycom_track_gru_residual.sh	Matched pooled and anonymous track-GRU training/inference launcher
run_easycom_oracle_information.sh	Prepare and screen aggregate/per-person/shuffled/delayed oracle conditions on Session 12
train_omni_turn_lora.py	Thinker LoRA training from cached AV
eval_omni_turn_lora.py	Thinker LoRA evaluation

Launchers and tests

• run_*.sh and run_*.py are reproducible launchers, not libraries.
• smoke_test.py is the model/environment smoke check.
• test_*.py are CPU-oriented regression tests colocated at root for current import and command compatibility.

Terminology restrictions

• Global (G): a Thinker readout that may mix audio, video and language context; do not describe it as audio-free.
• Local audio (A): recent mixed audio encoded outside the Thinker.
• Local visual (V): person-box visual features; current pooling discards stable person structure.
• Encoder-only: Qwen modality encoder output without Thinker states.
• Speaker grounding: evidence that the correct visible person is linked to the correct speech/time. AV improvement alone is insufficient.
• Addressee grounding: evidence about whom speech targets; aggregate others labels cannot supervise it.
• Floor entry/yield: observed wearer OFF→ON / ON→OFF behavior under a defined causal event window; not automatically semantic response or interruption intent.

Environment boundaries

• The open-stream replay implementation is pinned to transformers==4.52.3 because it uses version-specific internals.
• The local encoder path has been tested with 4.52.3 and 4.57.6; sidecars from one experiment must share the same recorded contract/version.
• Remote paths differ (/data/... and /data1/...). Commands should expose dataset/model roots as variables rather than hard-code one machine.
• Large models, datasets, checkpoints and extracted features live outside Git.