icassp2027-experiment/PROJECT_CONTEXT.md

# AI Project Context

This is the short, authoritative repository map for humans and AI agents.
The long [`README.md`](README.md) is an operational history and command
reference; it is not the experiment scheduler.

## One-sentence objective

Determine whether strictly causal, speaker-grounded audiovisual evidence from
Qwen2.5-Omni can improve an invisible egocentric wearer's **floor-entry** and
**floor-yield** behavior in natural multiparty conversation.

The paper is not simply “replace EgoSpeak's encoders with a newer backbone.”
The decisive test is whether correct person-aligned vision reduces false
wearer entries during other-to-other conversation at matched wearer-onset
recall, without hurting addressed responses or yield behavior.

## Source-of-truth order

| Need | Authoritative file |
|---|---|
| Progressive feature-fusion implementation route | [docs/progressive_feature_fusion_route_2026-09-02.md](docs/progressive_feature_fusion_route_2026-09-02.md) |
| Current hypotheses, experiment gates and stopping rules | [`docs/omni_speaker_floor_experiment_plan_2026-08-30.md`](docs/omni_speaker_floor_experiment_plan_2026-08-30.md) |
| Novelty boundary and claim restrictions | [`docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md`](docs/multiparty_egocentric_floor_gap_audit_2026-08-30.md) |
| Code/data map and current blocker | This file |
| Runnable commands and implementation history | [`README.md`](README.md) and [`docs/README.md`](docs/README.md) |
| Archived measured results | [`results/README.md`](results/README.md) |
| Literature summaries | [`literature_notes/README.md`](literature_notes/README.md) |

When files disagree, prefer the newest scoped document and verify behavior in
code/tests. A planned experiment is not an implemented feature or a measured
result.

## Current state

### Verified assets

- Strictly causal 5 Hz Qwen2.5-Omni global state extraction on EasyCom.
- Prepared-media and direct-original-media extraction paths.
- Frozen head training/inference with EgoSpeak-compatible 10-horizon AP/mAP.
- EasyCom local Qwen visual crops and recent mixed-audio sidecars.
- Encoder-only A/V/AV and `G`, `G+A`, `G+V`, `G+A+V` baselines.
- Ego4D batch and 8-GPU local-feature launchers, with strict frame audits.
- Timing-derived wearer floor-state and event diagnostics.

### Implemented, awaiting remote runtime verification

- F1 Frame-set Attention loader/head/train/infer path. It uses fused G+A as
  one query over the latest half-open causal person set, retains direct A in
  the head, and dynamically pads people per batch. Do not report it as a
  measured method until the L20 smoke and full controls pass.
- Two current-sidecar screening controls: `audio-person-frame-set` adds a
  zero-gated audio-to-person read before the F1 query, and
  `gated-person-state-adapter` adds a zero-gated person residual after the
  frozen Thinker readout. They are explicitly recorded as F5/F6 screening,
  not full history-conditioned F5 or an internal Thinker F6 adapter.

### Current blocker

`evaluate_floor_state_changes.py` counts a wearer transition anywhere in a
fixed response window. Before this becomes the paper event protocol, add
competing-risk censoring:

- an other-offset response window ends at the next other onset;
- an other-onset yield window ends at the next other offset.

Until then, EOT/HOLD, response and yield results are proxy diagnostics, not
semantic event ground truth.

### Next experiment gates

1. **P0:** fix/freeze event attribution, splits and scenario definitions.
2. **P1:** probe aggregate activity and per-person active-speaker grounding;
   compare correct, wrong-person and time-shifted visual inputs.
3. **P2:** re-evaluate `G`, `G+A`, `G+A+pooled-V` under the frozen event protocol.
4. **P3:** begin the progressive person-fusion route with Frame-set Attention,
   then add spatial or track-temporal structure only after its controls pass.
5. **P4:** attempt person-memory adapter/Thinker LoRA only if P1/P3 pass.

The detailed architecture, controls and stopping gates for P3 are frozen in
[the progressive feature-fusion route](docs/progressive_feature_fusion_route_2026-09-02.md).

## Stable data contracts

### Time and labels

- Sampling rate: 5 Hz (`STEP_SECONDS = 0.2`).
- State row `j` uses media no later than `(j+1)/5` seconds.
- Future horizons: 10, corresponding to 0.2, 0.4, ..., 2.0 seconds.
- Class order: `background`, `camera_wearer_speaking`,
  `other_people_speaking`.
- EasyCom targets are dense `[T, 3]`; predictions are
  `[target_time, class, future_horizon]`.
- EasyCom split: train Session 4--12; test Session 1--3.

### Global feature sidecar

Produced by `streaming_turn_model.py` or
`streaming_turn_model_directly.py` as a `.pt` dictionary. Required training
fields include:

- `states`: `[T, 2048]` or multi-layer `[T, L, 2048]`;
- `times`: `[0.2, 0.4, ...]`;
- `label_indices`: dense zero-origin indices into the target timeline;
- extraction `mode`, configuration and source metadata.

Never silently accept non-zero origins, missing label indices or mismatched
time grids.

### Local feature sidecar

Produced by `extract_turn_local_features.py`, validated by
`local_turn_features.py`, and stored as `qwen-local-pooled-v1`. It contains
native 2048D visual/audio vectors, validity masks, times, label indices,
source metadata and the exact extraction config. A partial smoke sidecar must
not be reused as a full-session feature file.

### Inference result

`infer_turn_head.py` writes:

```text
RESULT_DIR/
  metrics.json
  pred/<sample>.npy       # [T, 3, 10]
  valid/<sample>.npy      # [T, 10]
  transition_pred/        # only when the auxiliary head exists
```

`evaluate_floor_state_changes.py` consumes `pred/` and `valid/`; it does not
run a GPU model.

## Canonical pipelines

### Direct global features (preferred semantic path)

```text
original EasyCom/Ego4D media
  -> streaming_turn_model_directly.py
  -> global feature .pt
  -> train_turn_head.py
  -> checkpoint .pth + .json
  -> infer_turn_head.py
  -> pred/ + valid/ + metrics.json
  -> evaluate_floor_state_changes.py
```

This path samples source video causally in Python and avoids requiring a
re-encoded fixed-FPS MP4.

### Prepared-media compatibility path

```text
prepare_easycom_session.py / prepare_ego4d_2fps.py
  -> strict 2 FPS media + metadata
  -> streaming_turn_model.py
  -> the same training/inference pipeline
```

Keep it for reproducibility, cache-replay verification and comparisons with
already extracted features.

### Local evidence path

```text
raw media + person boxes + existing global reference .pt
  -> extract_turn_local_features.py
     (or extract_ego4d_local_multigpu.py supervisor)
  -> local .pt sidecar
  -> train_turn_head.py --local-feature-root ...
  -> infer_turn_head.py --local-feature-root ...
```

The current local visual feature is pooled. It does not by itself prove
speaker identity, active-speaker alignment or addressee understanding.

## Root-code map

Root scripts remain flat because remote L20 commands call them directly.
Do not reorganize them physically without compatibility wrappers.

### Shared libraries

| File | Responsibility |
|---|---|
| `easycom_head.py` | Dataset constants, feature loading, heads, losses and AP metrics |
| `local_turn_features.py` | Local-sidecar contract, alignment, crops and pooling |
| `omni_cached_av.py` | Cached AV structures used by Omni extraction/LoRA paths |

### Global feature extraction

| File | Responsibility |
|---|---|
| `streaming_turn_model_directly.py` | Preferred direct raw-media causal extraction |
| `streaming_turn_model.py` | Prepared-media causal-prefix/open-stream extraction and replay checks |
| `qwen_omni_features.py` | Single-window feature inspection |
| `cache_omni_av_pieces.py` | Build reusable AV pieces for LoRA experiments |
| `extract_ego4d_batch.py` | Persistent-model Ego4D global batch extraction |
| `analyze_block_phase.py` | Streaming block-phase diagnostics |
| `profile_streaming_bottleneck_gpu7.py` | GPU extraction profiling |

### Data preparation and audit

| File | Responsibility |
|---|---|
| `easycom_manifest.py` | EasyCom clip inventory and dense-timeline validation |
| `prepare_easycom_session.py` | Optional strict-grid EasyCom media preparation |
| `prepare_ego4d_2fps.py` | Optional strict-grid Ego4D media preparation |
| `prepare_encoder_only_features.py` | Convert local sidecars into encoder-only references |
| `audit_egospeak_features.py` | Audit released EgoSpeak feature layout |
| `derive_easycom_interaction_events.py` | Deterministic transcript/timing proxies |
| `adjudicate_easycom_interaction_events.py` | Optional ambiguous-event adjudication |
| `audit_easycom_floor_events.py` | Count EOT/takeover/backchannel candidates |

### Local features

| File | Responsibility |
|---|---|
| `extract_turn_local_features.py` | Qwen person-crop visual and recent-audio sidecars |
| `extract_ego4d_local_multigpu.py` | Foreground supervisor for disjoint multi-GPU workers |

### Training, inference and evaluation

| File | Responsibility |
|---|---|
| `train_turn_head.py` | Frozen-backbone head training, optional local fusion/auxiliary losses |
| `infer_turn_head.py` | Save EgoSpeak-compatible predictions and AP/mAP |
| `evaluate_floor_state_changes.py` | CPU floor-state/event proxy evaluation |
| `summarize_local_baselines.py` | Aggregate local baseline `metrics.json` files |
| `train_omni_turn_lora.py` | Thinker LoRA training from cached AV |
| `eval_omni_turn_lora.py` | Thinker LoRA evaluation |

### Launchers and tests

- `run_*.sh` and `run_*.py` are reproducible launchers, not libraries.
- `smoke_test.py` is the model/environment smoke check.
- `test_*.py` are CPU-oriented regression tests colocated at root for current
  import and command compatibility.

## Terminology restrictions

- **Global (`G`)**: a Thinker readout that may mix audio, video and language
  context; do not describe it as audio-free.
- **Local audio (`A`)**: recent mixed audio encoded outside the Thinker.
- **Local visual (`V`)**: person-box visual features; current pooling discards
  stable person structure.
- **Encoder-only**: Qwen modality encoder output without Thinker states.
- **Speaker grounding**: evidence that the correct visible person is linked to
  the correct speech/time. AV improvement alone is insufficient.
- **Addressee grounding**: evidence about whom speech targets; aggregate
  `others` labels cannot supervise it.
- **Floor entry/yield**: observed wearer OFF→ON / ON→OFF behavior under a
  defined causal event window; not automatically semantic response or
  interruption intent.

## Environment boundaries

- The open-stream replay implementation is pinned to
  `transformers==4.52.3` because it uses version-specific internals.
- The local encoder path has been tested with 4.52.3 and 4.57.6; sidecars from
  one experiment must share the same recorded contract/version.
- Remote paths differ (`/data/...` and `/data1/...`). Commands should expose
  dataset/model roots as variables rather than hard-code one machine.
- Large models, datasets, checkpoints and extracted features live outside Git.