icassp2027-experiment/PROJECT_CONTEXT.md

AI Project Context

This is the short, authoritative repository map for humans and AI agents. The long README.md is an operational history and command reference; it is not the experiment scheduler.

One-sentence objective

Determine whether strictly causal global Omni context plus local audiovisual floor evidence organized around visible people can improve an invisible egocentric wearer's floor-entry and floor-yield behavior in natural multiparty conversation.

The paper is not simply “replace EgoSpeak's encoders with a newer backbone.” The decisive test is whether correct person-aligned vision reduces false wearer entries during other-to-other conversation at matched wearer-onset recall, without hurting addressed responses or yield behavior.

Source-of-truth order

Need	Authoritative file
ICASSP submission story, pruning and acceptance evidence bar	docs/icassp2027_acceptance_storyline_2026-09-10.md
Current paper thesis and claim boundary	docs/global_local_omni_paper_position_2026-09-07.md
Progressive feature-fusion implementation route	docs/progressive_feature_fusion_route_2026-09-02.md
Current hypotheses, experiment gates and stopping rules	docs/omni_speaker_floor_experiment_plan_2026-08-30.md
Novelty boundary and claim restrictions	docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md
Code/data map and current blocker	This file
Runnable commands and implementation history	README.md and docs/README.md
Archived measured results	results/README.md
Literature summaries	literature_notes/README.md

When files disagree, prefer the newest scoped document and verify behavior in code/tests. A planned experiment is not an implemented feature or a measured result.

Current state

Verified assets

• Strictly causal 5 Hz Qwen2.5-Omni global state extraction on EasyCom.
• Prepared-media and direct-original-media extraction paths.
• Frozen head training/inference with EgoSpeak-compatible 10-horizon AP/mAP.
• EasyCom local Qwen visual crops and recent mixed-audio sidecars.
• Encoder-only A/V/AV and G, G+A, G+V, G+A+V baselines.
• Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
• Timing-derived wearer floor-state and event diagnostics.

Measured screening result

• F1 Frame-set Attention uses fused G+A as one query over the latest half-open causal person set. On the existing three-seed, four-epoch test screen it obtained 81.77745 +/- 0.15638 0.2-second mAP and 64.95099 +/- 0.09451 ten-horizon mean mAP. This is stable, but is not a matched improvement over the historical no-dev pooled result; correct/wrong visual controls remain missing.

Measured local-fusion screens

• Two current-sidecar screening controls: audio-person-frame-set adds a zero-gated audio-to-person read before the F1 query, and gated-person-state-adapter adds a zero-gated person residual after the frozen Thinker readout. They are explicitly recorded as F5/F6 screening, not full history-conditioned F5 or an internal Thinker F6 adapter.
• track-gru-residual builds strictly causal histories from anonymous provided track IDs, applies one shared GRU per track, lets G+A query current track states, and adds a zero-gated correction to an exactly initialized pooled G+A+V checkpoint. The matched Session-12 three-seed screen was null (+0.01306 +/- 0.02368 percentage points in ten-horizon mean mAP). A seed-0 Sessions-1--3 audit reproduced pooled predictions to within -0.00030 points. Saved residual gates were only -0.0015 to +0.0108; because a zero scalar gate also blocks the initial branch gradient, this is recorded as an optimization failure rather than evidence that track continuity is useless. See the immutable result record and implementation note.
• causal-person-adapter-v2 is now implemented but unmeasured. It preserves an exact matched pooled G+A+V base, reuses the same causal anonymous track histories, replaces the blocking zero scalar with a sample-dependent vector gate, and zero-initializes only the final residual-logit projection. Its isolated runbook is docs/easycom_causal_person_adapter_v2.md. This is an optimization-repair/incremental screen, not evidence that track identity matters.
• causal-person-adapter-egospeak-gru is an unmeasured controlled ablation of V2. It changes only the per-track visual temporal stem to EgoSpeak MiniROADA's Linear-LN-ReLU-Dropout-GRU-ReLU ordering while retaining V2's pooled base, G+A person query, set attention, dynamic gate and zero-output residual. It is an EgoSpeak-style per-track adaptation, not a reproduction of EgoSpeak's global 5 Hz RGB+audio GRU. See its runbook.
• highrate-person-av-attention is now implemented but unmeasured. A separate extractor runs the published, frozen MuVAP frontend at 25 Hz: mixed audio is encoded once as a shared CPC stream, while each face track contributes a causal visual state plus one AV active-speaker score. At each 5 Hz right edge G+U queries that variable person set. The first screen loads and freezes the matched trained global-Omni + traditional-audio base and learns a zero-initialized person-logit residual. This is global Omni context plus traditional high-rate audio/person perception; it is not the full MuVAP forecasting model and does not yet include online detection. See the runbook.
• A MuVAP-only Stage-3 baseline is now implemented as a clean-room paper reconstruction. It freezes the released 25 Hz mono role-VAP and causal ASD backbones, lets Z_VAP query the variable Z_ASD person set at each frame, applies a one-layer causal temporal Transformer, and trains only that fusion plus the EgoSpeak 10-horizon head. It deliberately excludes Omni and is not described as upstream Stage-3 code because that entry point remains unpublished. See the runbook.

Measured oracle-information screen

• A seed-0, four-epoch Session-12 development screen supplied transcript-derived current/past speech activity only through the explicit analysis-only oracle contract. aggregate-visible improved over the matched local-audio baseline by +6.96649 points at 0.2 seconds and +2.91047 points in ten-horizon mean mAP. Visibility-only and 1-second-delayed controls stayed near audio.
• Person identity continuity is not supported: aggregate-visible exceeded per-person, while per-person and shuffled-id were effectively tied. The result motivates a learned causal local active-speaker/floor-evidence branch, but does not justify a track-continuity claim. See the immutable result record.

Current blocker

evaluate_floor_state_changes.py now implements and unit-tests the required competing-risk attribution: an other-offset response window ends before the next other onset, and an other-onset yield window ends before the next other offset. It also supports disjoint training-side threshold calibration, rejects official-test calibration on Sessions 1--3, reports false-entry/false-stop counts, and prints auditable timelines.

P0 is still not frozen. The remaining blocker is a real-data audit: run the v2 evaluator on saved calibration/test predictions, inspect event timelines and censoring/invalid counts, independently review sampled events, and freeze the scenario definitions. These values remain timing-derived behavior proxies, not semantic EOT/INT or addressee ground truth. See the runbook.

Three explanation-oriented diagnostics are implemented but unmeasured: a separately identified post-boundary reactive evaluator, a direct competing-risk response/yield probe over the same frozen Omni/audio/oracle inputs, and a privileged direct-to-wearer Target_of_Speech oracle with an equal-width zero-channel control. They diagnose timing, objective and addressee-information limitations respectively; none changes or freezes P0. See docs/easycom_floor_event_diagnostics.md.

Next experiment gates

The user has temporarily chosen to run the dense-AP model screen while P0 is unfinished. This does not relax the paper claim: event numbers remain proxy diagnostics until the v2 attribution, calibration and scenario audit is frozen on real data.

1. P0: run and audit competing-risk v2, then freeze event attribution, calibration, splits and scenario definitions.
2. P1: confirm the aggregate/per-person oracle screen over seeds 1--2, then train a causal visible-person active-speaker/floor-evidence estimator; compare correct, wrong-person and time-shifted visual inputs.
3. P2: re-evaluate G, G+A, G+A+pooled-V under the frozen event protocol.
4. P3: only if learned person-organized evidence beats aggregate/no-ID controls, repair the matched anonymous-track-GRU residual optimization by replacing the scalar zero gate with a zero-initialized residual output layer, and verify nonzero branch gradients on Session 12. Add broken-ID, wrong-time and remove-person controls only if that corrected screen improves.
5. P4: attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.

The detailed architecture, controls and stopping gates for P3 are frozen in the progressive feature-fusion route.

Stable data contracts

Time and labels

• Sampling rate: 5 Hz (STEP_SECONDS = 0.2).
• State row j uses media no later than (j+1)/5 seconds.
• Future horizons: 10, corresponding to 0.2, 0.4, ..., 2.0 seconds.
• Class order: background, camera_wearer_speaking, other_people_speaking.
• EasyCom targets are dense [T, 3]; predictions are [target_time, class, future_horizon].
• EasyCom split: train Session 4--12; test Session 1--3.
• Locked final head protocol: exactly 4 epochs on all Sessions 4--12, no development checkpoint selection, followed by one evaluation on Sessions 1--3. Historical Session-12 screens remain development diagnostics and are not the final training protocol.

Global feature sidecar

Produced by streaming_turn_model.py or streaming_turn_model_directly.py as a .pt dictionary. Required training fields include:

• states: [T, 2048] or multi-layer [T, L, 2048];
• times: [0.2, 0.4, ...];
• label_indices: dense zero-origin indices into the target timeline;
• extraction mode, configuration and source metadata.

Never silently accept non-zero origins, missing label indices or mismatched time grids.

Local feature sidecar

Produced by extract_turn_local_features.py, validated by local_turn_features.py, and stored as qwen-local-pooled-v1. It contains native 2048D visual/audio vectors, validity masks, times, label indices, source metadata and the exact extraction config. A partial smoke sidecar must not be reused as a full-session feature file.

Inference result

infer_turn_head.py writes:

RESULT_DIR/
  metrics.json
  pred/<sample>.npy       # [T, 3, 10]
  valid/<sample>.npy      # [T, 10]
  transition_pred/        # only when the auxiliary head exists

evaluate_floor_state_changes.py consumes pred/ and valid/; it does not run a GPU model.

Canonical pipelines

Direct global features (preferred semantic path)

original EasyCom/Ego4D media
  -> streaming_turn_model_directly.py
  -> global feature .pt
  -> train_turn_head.py
  -> checkpoint .pth + .json
  -> infer_turn_head.py
  -> pred/ + valid/ + metrics.json
  -> evaluate_floor_state_changes.py

This path samples source video causally in Python and avoids requiring a re-encoded fixed-FPS MP4.

Prepared-media compatibility path

prepare_easycom_session.py / prepare_ego4d_2fps.py
  -> strict 2 FPS media + metadata
  -> streaming_turn_model.py
  -> the same training/inference pipeline

Keep it for reproducibility, cache-replay verification and comparisons with already extracted features.

Local evidence path

raw media + person boxes + existing global reference .pt
  -> extract_turn_local_features.py
     (or extract_ego4d_local_multigpu.py supervisor)
  -> local .pt sidecar
  -> train_turn_head.py --local-feature-root ...
  -> infer_turn_head.py --local-feature-root ...

The current local visual feature is pooled. It does not by itself prove speaker identity, active-speaker alignment or addressee understanding.

Root-code map

Root scripts remain flat because remote L20 commands call them directly. Do not reorganize them physically without compatibility wrappers.

Shared libraries

File	Responsibility
easycom_head.py	Dataset constants, feature loading, heads, losses and AP metrics
local_turn_features.py	Local-sidecar contract, alignment, crops and pooling
omni_cached_av.py	Cached AV structures used by Omni extraction/LoRA paths

Global feature extraction

File	Responsibility
streaming_turn_model_directly.py	Preferred direct raw-media causal extraction
streaming_turn_model.py	Prepared-media causal-prefix/open-stream extraction and replay checks
qwen_omni_features.py	Single-window feature inspection
cache_omni_av_pieces.py	Build reusable AV pieces for LoRA experiments
extract_ego4d_batch.py	Persistent-model Ego4D global batch extraction
analyze_block_phase.py	Streaming block-phase diagnostics
profile_streaming_bottleneck_gpu7.py	GPU extraction profiling

Data preparation and audit

File	Responsibility
easycom_manifest.py	EasyCom clip inventory and dense-timeline validation
prepare_easycom_session.py	Optional strict-grid EasyCom media preparation
prepare_ego4d_2fps.py	Optional strict-grid Ego4D media preparation
prepare_encoder_only_features.py	Convert local sidecars into encoder-only references
audit_egospeak_features.py	Audit released EgoSpeak feature layout
derive_easycom_interaction_events.py	Deterministic transcript/timing proxies
adjudicate_easycom_interaction_events.py	Optional ambiguous-event adjudication
audit_easycom_floor_events.py	Count EOT/takeover/backchannel candidates
prepare_easycom_eot_int_review.py	Build manuscript-aligned EOT-response/INT-yield positive and hard-negative review regions
adjudicate_easycom_eot_int_candidates.py	Add optional DeepSeek transcript/timing suggestions; never human ground truth
review_floor_events.py	Serve the local video/transcript workspace and persist human event reviews
prepare_easycom_oracle_features.py	Build explicit past-only oracle person-activity sidecars and analysis-only addressee controls

Local features

File	Responsibility
extract_turn_local_features.py	Qwen person-crop visual and recent-audio sidecars
extract_ego4d_local_multigpu.py	Foreground supervisor for disjoint multi-GPU workers

Training, inference and evaluation

File	Responsibility
train_turn_head.py	Frozen-backbone head training, optional local fusion/auxiliary losses
infer_turn_head.py	Save EgoSpeak-compatible predictions and AP/mAP
evaluate_floor_state_changes.py	CPU floor-state/event proxy evaluation
train_easycom_floor_event_probe.py	Analysis-only direct response/yield probe with shared competing-risk labels
summarize_easycom_floor_event_probes.py	Compare direct event-probe conditions under one evaluation contract
summarize_local_baselines.py	Aggregate local baseline metrics.json files
run_easycom_track_gru_residual.sh	Matched pooled and anonymous track-GRU training/inference launcher
run_easycom_oracle_information.sh	Prepare and screen aggregate/per-person/shuffled/delayed oracle conditions on Session 12
run_easycom_floor_event_probes.sh	Prepare, train and compare aggregate/addressee direct event probes
run_easycom_post_boundary_eval.sh	Compare existing dense runs after the aggregate-other boundary is observed
train_omni_turn_lora.py	Thinker LoRA training from cached AV
eval_omni_turn_lora.py	Thinker LoRA evaluation

Launchers and tests

• run_*.sh and run_*.py are reproducible launchers, not libraries.
• smoke_test.py is the model/environment smoke check.
• test_*.py are CPU-oriented regression tests colocated at root for current import and command compatibility.

Terminology restrictions

• Global (G): a Thinker readout that may mix audio, video and language context; do not describe it as audio-free.
• Local audio (A): recent mixed audio encoded outside the Thinker.
• Local visual (V): person-box visual features; current pooling discards stable person structure.
• Encoder-only: Qwen modality encoder output without Thinker states.
• Speaker grounding: evidence that the correct visible person is linked to the correct speech/time. AV improvement alone is insufficient.
• Addressee grounding: evidence about whom speech targets; aggregate others labels cannot supervise it.
• Floor entry/yield: observed wearer OFF→ON / ON→OFF behavior under a defined causal event window; not automatically semantic response or interruption intent.

Environment boundaries

• The open-stream replay implementation is pinned to transformers==4.52.3 because it uses version-specific internals.
• The local encoder path has been tested with 4.52.3 and 4.57.6; sidecars from one experiment must share the same recorded contract/version.
• Remote paths differ (/data/... and /data1/...). Commands should expose dataset/model roots as variables rather than hard-code one machine.
• Large models, datasets, checkpoints and extracted features live outside Git. The explicitly audited 72 MiB MuVAP ASD-CPC release and 7 MiB CPC frontend are a deliberate exception: they are vendored with their exact source and provenance under third_party/MuVAP for reproducible extraction.