Merge branch 'main' of http://182.92.86.220:8300/wx44wx/icassp2027-experiment

PROJECT_CONTEXT.md

@@ -147,7 +147,16 @@ result.
   described as upstream Stage-3 code because that entry point remains
   unpublished. See the
   [`runbook`](docs/easycom_muvap_stage3_baseline.md).
-- `egospeak-omni-zero-residual` is implemented but unmeasured. It strictly
+- `egospeak-omni-zero-residual` has a user-reported seed-0 screen. Causal-valid
+  dense mean mAP increases from `61.193` to `62.714`; raw onset/offset mean
+  AP increases from `24.491/38.024` to `25.091/40.604`. Offset improves at all
+  ten horizons; onset gains are modest. Actual checkpoint provenance and
+  counts were not supplied for independent verification. Opportunity-based
+  semantic evaluation is temporarily paused because the user suspects
+  provisional annotation issues; historical metrics remain archived and P0
+  remains unfinished. See the
+  [`result record`](results/easycom_egospeak_omni_residual_2026-09-16/README.md).
+  The implementation strictly
   loads and freezes the released MiniROADA A+V/GRU/anticipation path, reuses
   existing causal 5 Hz Omni readouts, adds an explicit ten-bin partial-tail
   phase embedding, and learns only a zero-output residual over the original
@@ -157,6 +166,19 @@ result.
   states remains a future extraction ablation. See the
   [`runbook`](docs/easycom_egospeak_omni_residual.md).
 
+### Unmeasured dual-stream continuation
+
+Stage-two `egospeak-omni-selective-finetune` is implemented but unmeasured.
+It strictly continues the trained stage-one checkpoint, selectively adapts
+MiniROADA's GRU/forecast heads with a lower learning rate, and leaves original
+feature extractors, input projection and Omni frozen. It exports immutable
+stage-one/original references and automatic threshold-free raw endpoint AP.
+The user requested this exploratory continuation while semantic annotations
+are audited; P0 remains unfinished. Four additional epochs means eight total
+adaptation epochs, requiring the fusion-only continuation control for a
+budget-matched unfreezing comparison. See the
+[`stage-two runbook`](docs/easycom_egospeak_omni_finetune.md).
+
 ### Measured oracle-information screen
 
 - A seed-0, four-epoch Session-12 development screen supplied transcript-derived

docs/README.md

@@ -29,6 +29,7 @@ tests.
 | [`easycom_causal_person_adapter_v2.md`](easycom_causal_person_adapter_v2.md) | Dynamic-gated, zero-output causal person-track residual and matched L20 screen |
 | [`easycom_egospeak_style_person_gru.md`](easycom_egospeak_style_person_gru.md) | Controlled V2 temporal-stem ablation using EgoSpeak MiniROADA's layer order |
 | [`easycom_egospeak_omni_residual.md`](easycom_egospeak_omni_residual.md) | Stage-one frozen released MiniROADA plus zero-initialized tail-aware Omni residual fusion |
+| [`easycom_egospeak_omni_finetune.md`](easycom_egospeak_omni_finetune.md) | Stage-two selective MiniROADA GRU/head continuation with immutable stage-one/base references and raw endpoint AP |
 | [`easycom_highrate_person_av.md`](easycom_highrate_person_av.md) | Frozen 25 Hz causal ASD person evidence fused with 5 Hz global Omni context |
 | [`easycom_muvap_stage3_baseline.md`](easycom_muvap_stage3_baseline.md) | MuVAP-only frozen VAP+ASD Stage-3 paper reconstruction on EgoSpeak/EasyCom targets |
 | [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |

docs/easycom_egospeak_omni_finetune.md

@@ -0,0 +1,208 @@
+# Stage two: selective MiniROADA + Omni residual fine-tuning
+
+## Decision and claim boundary
+
+The user requested this continuation after the
+[`stage-one screen`](../results/easycom_egospeak_omni_residual_2026-09-16/README.md):
+causal-valid dense mean mAP improved by +1.521 points, raw onset endpoint AP
+by +0.600 and offset endpoint AP by +2.580. These are single-seed reported
+results, not statistically significant semantic floor-control evidence.
+Provisional ENTRY/YIELD evaluation is paused pending an annotation audit;
+historical provisional results remain archived. P0 is still unfinished.
+
+This stage means selective MiniROADA temporal/forecast adaptation, **not**
+Omni LoRA, a person-memory adapter, new feature extraction or a semantic head.
+It is an exploratory implementation of the earlier selective-unfreezing route.
+Do not use Sessions 1--3 to choose learning rates, windows, loss weights,
+unfreezing policy or the stopping epoch. A comparison observed on the exposed
+test split remains exploratory; choose any later settings on training-side
+development data and report the test exposure honestly.
+
+## What changes
+
+Load `checkpoint_last.pt` from stage one strictly. Preserve its learned
+nonzero residual and verify exact initial recovery of its fused/base logits
+on a training example. The original released checkpoint is not needed again:
+the first-stage checkpoint already contains its full frozen base parameters.
+
+| Component | Stage-two default |
+|---|---|
+| Released RGB and wav2vec2 feature extractors | Frozen; reuse existing arrays |
+| MiniROADA RGB+audio input projection (`layer1`) | Frozen, dropout remains off |
+| MiniROADA GRU | Trainable, learning rate 1e-5 |
+| Anticipation projection and shared classification head | Trainable, learning rate 1e-5 |
+| Ego/Omni projections, tail embedding and fusion residual | Continue training, learning rate 1e-4 |
+| Qwen2.5-Omni backbone | Frozen; reuse exactly the first-stage sidecars |
+
+The GRU state contributes to both the original anticipation logits and the
+feature-fusion residual. Unlike stage one, neither path is enclosed in
+`no_grad` or detached when selected for training. The default trainable
+MiniROADA modules contain 19,942,403 parameters, in addition to the existing
+residual branch. The RGB/audio encoders themselves are not loaded or trained;
+this is partial task-model fine-tuning with frozen feature extractors, not
+raw-media end-to-end training.
+
+Loss, window/stride, grid, class order and ten horizons stay unchanged.
+Window size and stride inherit the stage-one checkpoint and mismatched
+overrides fail. Readout mode/layer fusion/dimension must also match; when the
+stage-one sibling `metrics.json` is available, its full saved Omni extraction
+signature is checked. A relocated checkpoint without that provenance prints
+a warning, so the same original Omni root must be reused explicitly.
+
+Default optimizer: AdamW, weight decay 0.05, gradient clipping at norm 1.0.
+These defaults are specified before the new run, not tuned on test results.
+The first training batch logs separate fusion/base gradient norms. Final
+evaluation occurs only after the fixed additional epoch budget, with no
+test-side early stopping or best-checkpoint selection.
+
+## Session 4 smoke test
+
+From the repository root, reuse the same two feature roots as stage one:
+
+```bash
+EGO_ROOT=third_party/EgoSpeak/MiniROAD/data/EasyCom
+OMNI_ROOT=/path/to/the/same/first_stage/omni_features
+STAGE1=results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt
+
+CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
+  --egospeak-feature-root "$EGO_ROOT" \
+  --omni-feature-root "$OMNI_ROOT" \
+  --stage-one-checkpoint "$STAGE1" \
+  --train-sessions Session_4 \
+  --eval-sessions Session_4 \
+  --epochs 1 \
+  --max-train-examples 32 \
+  --batch-size 8 \
+  --num-workers 0 \
+  --output-dir results/easycom_egospeak_omni_finetune_smoke_2026-09-16
+```
+
+Check exact initialization, nonzero fusion/base gradients, finite loss,
+`[T,3,10]` predictions, `[T,10]` common valid masks and raw endpoint counts.
+The printed cutoff is `(source_row+1)/5` s; input stops at that row, and
+future labels are rows source+1 through source+10. Smoke predictions are not
+the final model's calibration predictions. Start the formal run from the
+original stage-one checkpoint, not the smoke checkpoint.
+
+## Fixed-budget continuation
+
+After the GPU smoke audit passes:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python3 -u train_egospeak_omni_finetune.py \
+  --egospeak-feature-root "$EGO_ROOT" \
+  --omni-feature-root "$OMNI_ROOT" \
+  --stage-one-checkpoint "$STAGE1" \
+  --base-trainable gru-heads \
+  --fusion-lr 1e-4 \
+  --base-lr 1e-5 \
+  --epochs 4 \
+  --batch-size 64 \
+  --num-workers 4 \
+  --seed 0 \
+  --output-dir results/easycom_egospeak_omni_finetune_seed0_2026-09-16
+```
+
+Defaults use Sessions 4--12 for training and Sessions 1--3 for final
+evaluation. Training or a first-stage checkpoint trained on Sessions 1--3 is
+rejected. Float32 is the first reproducibility run; do not enable AMP until
+its output/causality audit passes. Full-session GRU inference is the default;
+`--eval-chunk-size N` carries recurrent state across chunks if memory requires.
+Nonempty output directories are never overwritten.
+
+An additional four epochs means **4+4=8 total adaptation epochs**. It cannot
+isolate unfreezing against the original four-epoch first stage. The required
+matched-budget continuation control changes only these arguments:
+
+```bash
+  --base-trainable none \
+  --output-dir results/easycom_egospeak_omni_fusion_continue_seed0_2026-09-16
+```
+
+Keep the same stage-one checkpoint, seed, training split, four additional
+epochs, sampling, optimizer and fusion learning rate. `--base-trainable heads`
+is a separate predefined ablation that freezes the GRU and adapts only the
+forecast heads plus residual; do not select among policies on test scores.
+
+## Outputs and reading AP
+
+```text
+RESULT/
+  README.md
+  checkpoint_last.pt              # full adapted model, config, parent hash, budget
+  metrics.json                    # adapted dense + raw endpoint AP
+  raw_endpoint_metrics.csv        # three methods, ten horizons, AP and counts
+  pred/, valid/, raw_eval/         # second-stage predictions
+  stage_one_reference/
+    metrics.json, pred/, valid/, raw_eval/
+  frozen_egospeak_reference/
+    metrics.json, pred/, valid/, raw_eval/
+```
+
+Both references are immutable copies of the first-stage checkpoint: one
+includes its learned fusion, the other uses its original frozen MiniROADA
+path. The adapted base branch is never mislabeled as frozen EgoSpeak.
+Every method uses the same valid source rows. Reference directories are
+directly compatible with the existing evaluator. A loader for the new full
+model is `egospeak_omni_finetune.load_stage_two_checkpoint`; stage-one and
+stage-two checkpoint formats are intentionally distinct.
+
+Dense metrics retain both full-timeline and causal-valid protocols. Raw onset
+uses current wearer OFF, future ON versus future OFF with score p_wearer;
+offset uses current wearer ON, future OFF versus future ON with score
+1-p_wearer. AP pools eligible pairs over evaluated sessions separately for
+each horizon; the headline averages ten APs. This is endpoint anticipation,
+not timestamp detection or semantic ENTRY/YIELD. No Session 4 threshold
+calibration or annotation file is needed for these APs. An undefined horizon
+is preserved as null, not silently removed from the mean.
+
+The CSV/JSON store AP **fractions**, while console output uses percentages.
+To print the raw results:
+
+```bash
+python3 - <<'PY'
+import pandas as pd
+p = "results/easycom_egospeak_omni_finetune_seed0_2026-09-16/raw_endpoint_metrics.csv"
+x = pd.read_csv(p)
+cols = ["onset_endpoint_ap", "offset_endpoint_ap"]
+x[cols] *= 100
+print(x[["run", "nominal_seconds", *cols]].round(3).to_string(index=False))
+print(x.groupby("run")[cols].mean().round(3).to_string())
+PY
+```
+
+If any horizon is undefined, use the JSON headline (which stays null) instead
+of pandas' default mean that skips NaN. No semantic opportunity evaluation is
+run by this launcher.
+
+## Local validation
+
+### Evaluate a saved checkpoint without training again
+
+For a completed short continuation, use the separate inference-only CLI:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python3 -u infer_egospeak_omni_finetune.py \
+  --checkpoint results/easycom_egospeak_omni_finetune_low_lr_1ep_diagnostic_2026-09-16/checkpoint_last.pt \
+  --sessions Session_1 Session_2 Session_3 \
+  --output-dir results/easycom_egospeak_omni_finetune_low_lr_1ep_test_2026-09-16
+```
+
+Feature roots/readout settings inherit the checkpoint; optional
+`--egospeak-feature-root`/`--omni-feature-root` override relocated roots while
+saved extraction contracts are checked. Float32 inference, `pred/valid/raw_eval`,
+dense and threshold-free raw endpoint AP are exported. No optimizer is created
+and the checkpoint is unchanged. This repeated official-test evaluation is
+exploratory; do not choose subsequent parameters or epochs from its scores.
+
+### Tests
+
+```bash
+python3 -m unittest test_egospeak_omni_finetune test_egospeak_omni_residual -v
+```
+
+Tests cover trained residual preservation, selected-module gradients,
+reference immutability, policy masks, causal perturbation, carried-state
+chunk equivalence, checkpoint failure/success paths, raw AP and a tiny CPU
+training/export/reload smoke. They do not substitute for the real Session 4
+GPU smoke or establish second-stage performance; that run remains unmeasured.

docs/easycom_egospeak_omni_residual.md

@@ -154,6 +154,13 @@ so it remains readable by `summarize_local_baselines.py`.
 
 ## Interpretation and next gate
 
+The first seed-0 dense/raw screen is now
+[`recorded`](../results/easycom_egospeak_omni_residual_2026-09-16/README.md).
+At the user's request, provisional semantic evaluation is paused for an
+annotation audit. An exploratory selective GRU/head continuation is implemented
+in the [`stage-two runbook`](easycom_egospeak_omni_finetune.md); this does not
+imply that the semantic P0 gate below has passed.
+
 This first stage uses only the released dense future-activity objective.  It
 tests whether Omni adds useful context without discarding the task-specific
 EgoSpeak path.  It does not train semantic ENTRY/YIELD heads, and Sessions

egospeak_omni_finetune.py

@@ -0,0 +1,195 @@
+"""Stage-two selective MiniROADA adaptation from a trained Omni residual.
+
+The released input features and Omni states remain frozen. This module only
+enables gradients in selected MiniROADA temporal/forecast modules and the
+existing residual. Stage-one checkpoints are loaded strictly, without resetting
+the learned residual or requiring the original checkpoint to be available.
+"""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from pathlib import Path
+from typing import Any, Literal
+
+import torch
+
+from easycom_head import NUM_HORIZONS, OFFICIAL_TRAIN_SESSIONS
+from egospeak_omni_residual import (
+    EgoSpeakOmniResidual,
+    MiniROADABase,
+    load_torch_payload,
+    normalize_session_name,
+)
+
+BaseTrainable = Literal["none", "heads", "gru-heads"]
+STAGE_ONE_FORMAT = "egospeak-omni-zero-residual-v1"
+STAGE_TWO_FORMAT = "egospeak-omni-selective-finetune-v1"
+
+
+def validate_training_sessions(values: list[str]) -> None:
+    normalized = [normalize_session_name(value) for value in values]
+    if not normalized or len(normalized) != len(set(normalized)):
+        raise ValueError("Training sessions must be nonempty and unique")
+    if not set(normalized).issubset(OFFICIAL_TRAIN_SESSIONS):
+        raise ValueError("Training may only use EasyCom Sessions 4--12")
+
+
+def load_stage_one_checkpoint(
+    path: Path,
+) -> tuple[EgoSpeakOmniResidual, dict[str, Any]]:
+    path = path.expanduser().resolve()
+    if not path.is_file():
+        raise FileNotFoundError(path)
+    payload = load_torch_payload(path)
+    if not isinstance(payload, dict) or payload.get("format") != STAGE_ONE_FORMAT:
+        raise ValueError(
+            "--stage-one-checkpoint must be a stage-one residual checkpoint"
+        )
+    config = payload.get("model_config")
+    if not isinstance(config, dict):
+        raise TypeError("Stage-one checkpoint lacks model_config")
+    required = {
+        "omni_dim",
+        "fusion_dim",
+        "tail_embedding_dim",
+        "num_tail_bins",
+        "dropout",
+    }
+    if set(config) != required or config["num_tail_bins"] != NUM_HORIZONS:
+        raise ValueError("Incompatible stage-one residual model_config")
+    training = payload.get("training_config")
+    if not isinstance(training, dict) or int(training.get("epochs", 0)) <= 0:
+        raise ValueError("Stage-one checkpoint lacks a positive training epoch budget")
+    validate_training_sessions(payload.get("train_sessions", []))
+    base_config = payload.get("base_model_config", {})
+    if not isinstance(base_config, dict):
+        raise TypeError("Invalid base_model_config")
+    # Released v1 checkpoints omit base_model_config and use default MiniROADA.
+    model = EgoSpeakOmniResidual(MiniROADABase(**base_config), **config)
+    model.load_state_dict(payload["model_state_dict"], strict=True)
+    if any(not torch.isfinite(value).all() for value in model.state_dict().values()):
+        raise ValueError("Stage-one checkpoint contains nonfinite parameters")
+    model.requires_grad_(False)
+    model.eval()
+    return model, payload
+
+
+class EgoSpeakOmniFinetune(EgoSpeakOmniResidual):
+    """Continuation with explicit base policy and intact stage-one logits."""
+
+    def __init__(
+        self, *args: Any, base_trainable: BaseTrainable = "gru-heads", **kwargs: Any
+    ):
+        super().__init__(*args, **kwargs)
+        if base_trainable not in ("none", "heads", "gru-heads"):
+            raise ValueError(f"Invalid base-trainable policy: {base_trainable}")
+        self.base_trainable = base_trainable
+        self.requires_grad_(True)
+        self.base.requires_grad_(False)
+        if base_trainable in ("heads", "gru-heads"):
+            self.base.anticipation_layer.requires_grad_(True)
+            self.base.f_classification.requires_grad_(True)
+        if base_trainable == "gru-heads":
+            self.base.gru.requires_grad_(True)
+        self.train(False)
+
+    @classmethod
+    def from_stage_one(
+        cls,
+        reference: EgoSpeakOmniResidual,
+        config: dict[str, Any],
+        *,
+        base_trainable: BaseTrainable = "gru-heads",
+    ) -> EgoSpeakOmniFinetune:
+        model = cls(deepcopy(reference.base), **config, base_trainable=base_trainable)
+        # Copy rather than share weights with the immutable first-stage reference.
+        model.load_state_dict(reference.state_dict(), strict=True)
+        return model
+
+    def train(self, mode: bool = True) -> EgoSpeakOmniFinetune:
+        super().train(mode)
+        # Keep frozen layer1's dropout off. Enabling GRU/head train mode does
+        # not turn the original frozen RGB+audio projection back on.
+        if self.base_trainable in ("heads", "gru-heads"):
+            self.base.anticipation_layer.train(mode)
+            self.base.f_classification.train(mode)
+        if self.base_trainable == "gru-heads":
+            self.base.gru.train(mode)
+        return self
+
+    def optimizer_groups(
+        self, *, fusion_lr: float, base_lr: float
+    ) -> list[dict[str, Any]]:
+        if fusion_lr <= 0 or base_lr <= 0:
+            raise ValueError("Both learning rates must be positive")
+        fusion = [
+            p
+            for name, p in self.named_parameters()
+            if p.requires_grad and not name.startswith("base.")
+        ]
+        base = [
+            p
+            for name, p in self.named_parameters()
+            if p.requires_grad and name.startswith("base.")
+        ]
+        groups = [{"name": "fusion", "params": fusion, "lr": fusion_lr}]
+        if base:
+            groups.append({"name": "miniroada", "params": base, "lr": base_lr})
+        return groups
+
+    def forward_last(
+        self,
+        rgb: torch.Tensor,
+        audio: torch.Tensor,
+        omni: torch.Tensor,
+        tail_bin: torch.Tensor,
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        hidden, _ = self.base.encode(rgb, audio)
+        ego_hidden = hidden[:, -1, :]
+        base_logits = self.base.anticipation_logits(ego_hidden)
+        # No no_grad/detach: gradients must reach the selected MiniROADA GRU.
+        residual = self._residual_logits(ego_hidden, omni, tail_bin)
+        return base_logits + residual, base_logits
+
+    def forward_sequence_chunk(
+        self,
+        rgb: torch.Tensor,
+        audio: torch.Tensor,
+        omni: torch.Tensor,
+        tail_bin: torch.Tensor,
+        recurrent_state: torch.Tensor | None,
+    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        hidden, next_state = self.base.encode(rgb, audio, recurrent_state)
+        base_logits = self.base.anticipation_logits(hidden)
+        residual = self._residual_logits(hidden, omni, tail_bin)
+        return base_logits + residual, base_logits, next_state
+
+
+def base_model_config(base: MiniROADABase) -> dict[str, Any]:
+    return {
+        "rgb_dim": base.rgb_dim,
+        "audio_dim": base.audio_dim,
+        "embedding_dim": base.embedding_dim,
+        "hidden_dim": base.hidden_dim,
+        "num_layers": base.num_layers,
+        "num_horizons": base.num_horizons,
+        "num_classes": base.num_classes,
+        "dropout": base.layer1[-1].p,
+    }
+
+
+def load_stage_two_checkpoint(
+    path: Path,
+) -> tuple[EgoSpeakOmniFinetune, dict[str, Any]]:
+    payload = load_torch_payload(path)
+    if not isinstance(payload, dict) or payload.get("format") != STAGE_TWO_FORMAT:
+        raise ValueError("Not a stage-two fine-tuning checkpoint")
+    model = EgoSpeakOmniFinetune(
+        MiniROADABase(**payload["base_model_config"]),
+        **payload["model_config"],
+        base_trainable=payload["base_trainable"],
+    )
+    model.load_state_dict(payload["model_state_dict"], strict=True)
+    model.eval()
+    return model, payload

infer_egospeak_omni_finetune.py

@@ -0,0 +1,128 @@
+"""Evaluate an existing stage-two checkpoint without any optimization."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+
+from easycom_head import OFFICIAL_TEST_SESSIONS
+from egospeak_omni_finetune import load_stage_two_checkpoint
+from egospeak_omni_residual import align_source_predictions, sha256_file
+from train_egospeak_omni_finetune import raw_endpoint_metrics
+from train_egospeak_omni_residual import (
+    evaluate_predictions,
+    load_sessions,
+    normalize_sessions,
+    omni_contract_signature,
+    predict_session,
+    print_metrics,
+    resolve_device,
+)
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--checkpoint", type=Path, required=True)
+    parser.add_argument("--output-dir", type=Path, required=True)
+    parser.add_argument("--sessions", nargs="+", default=None)
+    parser.add_argument("--egospeak-feature-root", type=Path)
+    parser.add_argument("--omni-feature-root", type=Path)
+    parser.add_argument("--device", default="cuda")
+    parser.add_argument("--eval-chunk-size", type=int, default=0)
+    parser.add_argument("--no-progress", action="store_true")
+    args = parser.parse_args(argv)
+    if args.eval_chunk_size < 0:
+        raise ValueError("Evaluation chunk size must be nonnegative")
+    model, payload = load_stage_two_checkpoint(args.checkpoint)
+    config = payload["training_config"]
+    args.egospeak_feature_root = args.egospeak_feature_root or Path(
+        config["egospeak_feature_root"]
+    )
+    args.omni_feature_root = args.omni_feature_root or Path(config["omni_feature_root"])
+    args.omni_layer_fusion = config["omni_layer_fusion"]
+    args.expected_omni_mode = config["expected_omni_mode"]
+    names = normalize_sessions(args.sessions, OFFICIAL_TEST_SESSIONS)
+    sessions = load_sessions(args, names)
+    if {int(s.omni.shape[-1]) for s in sessions} != {model.omni_dim}:
+        raise ValueError("Inference Omni dimension differs from checkpoint")
+    known = payload.get("omni_sources", {})
+    for session in sessions:
+        signature = omni_contract_signature(session)
+        for source in known.values():
+            c = source["config"]
+            expected = (
+                source["model_id"],
+                source["mode"],
+                c.get("memory_seconds"),
+                c.get("video_fps"),
+                c.get("readout_last_n_layers"),
+                c.get("readout_layer_fusion"),
+                tuple(source["state_shape_before_layer_fusion"][1:]),
+            )
+            if signature != expected:
+                raise ValueError(
+                    "Inference Omni extraction differs from checkpoint provenance"
+                )
+    output = args.output_dir.expanduser().resolve()
+    if output.exists() and (not output.is_dir() or any(output.iterdir())):
+        raise FileExistsError(f"Refusing to overwrite output: {output}")
+    for name in ("pred", "valid", "raw_eval"):
+        (output / name).mkdir(parents=True)
+    device = resolve_device(args.device)
+    model.requires_grad_(False)
+    model.to(device).eval()
+    predictions, valid, targets = {}, {}, {}
+    for session in sessions:
+        raw, _ = predict_session(
+            model,
+            session,
+            device,
+            chunk_size=args.eval_chunk_size,
+            amp=False,
+            show_progress=not args.no_progress,
+        )
+        aligned, mask = align_source_predictions(raw, session.num_steps)
+        predictions[session.name], valid[session.name] = aligned, mask
+        targets[session.name] = np.asarray(session.targets, dtype=np.float32)
+        np.save(output / "raw_eval" / f"{session.name}.npy", raw)
+        np.save(output / "pred" / f"{session.name}.npy", aligned)
+        np.save(output / "valid" / f"{session.name}.npy", mask)
+    dense = evaluate_predictions(predictions, valid, targets)
+    endpoints = raw_endpoint_metrics(predictions, valid, targets)
+    metrics = {
+        **dense,
+        "raw_endpoint_anticipation": endpoints,
+        "sessions": names,
+        "train_sessions": payload["train_sessions"],
+        "seed": config["seed"],
+        "checkpoint": str(args.checkpoint.resolve()),
+        "checkpoint_sha256": sha256_file(args.checkpoint),
+        "inference_only": True,
+        "budgets": payload["budgets"],
+        "omni_sources": {s.name: s.omni_metadata for s in sessions},
+    }
+    (output / "metrics.json").write_text(json.dumps(metrics, indent=2) + "\n")
+    (output / "README.md").write_text(
+        "# Stage-two inference-only evaluation\n\n"
+        f"Checkpoint: {args.checkpoint}. Sessions: {names}. No optimization.\n\n"
+        "metrics.json stores dense/raw endpoint AP as fractions; console uses percent. "
+        "Repeated official-test evaluation is exploratory, not dev selection.\n"
+    )
+    print_metrics("stage_two inference only", dense)
+    print(
+        "Raw endpoint mean AP (percent):",
+        {
+            family: None
+            if endpoints[f"mean_{family}_ap"] is None
+            else endpoints[f"mean_{family}_ap"] * 100
+            for family in ("onset", "offset")
+        },
+    )
+    print(f"Saved {output / 'metrics.json'}; checkpoint unchanged")
+
+
+if __name__ == "__main__":
+    main()

results/README.md

@@ -6,6 +6,7 @@ readers to the right comparison.
 
 | Result record | Question answered |
 |---|---|
+| [`easycom_egospeak_omni_residual_2026-09-16/`](easycom_egospeak_omni_residual_2026-09-16/README.md) | Single-seed frozen EgoSpeak + Omni residual dense/raw endpoint gains; semantic annotation evaluation temporarily paused |
 | [`easycom_omni_floor_screen_2026-09-16/`](easycom_omni_floor_screen_2026-09-16/README.md) | Whether G+A, pooled-V, or current-frame F1 improve raw boundaries and provisional semantic opportunities over EgoSpeak |
 | [`easycom_egospeak_three_level_eval_2026-09-16/`](easycom_egospeak_three_level_eval_2026-09-16/README.md) | How dense forecasting, raw boundary anticipation, and provisional opportunity-conditioned ENTRY/YIELD differ for the EgoSpeak baseline |
 | [easycom_oracle_pooled_wearer_audit_2026-09-10/](easycom_oracle_pooled_wearer_audit_2026-09-10/README.md) | Fixed-four-epoch three-seed oracle test results and the limited wearer gains of historical pooled AV |

results/easycom_egospeak_omni_residual_2026-09-16/README.md

@@ -0,0 +1,71 @@
+# Stage-one frozen EgoSpeak + Omni residual screen
+
+## Provenance and scope
+
+These values were reported by the user on 2026-09-16, not recomputed on this
+host. The preceding launcher command specifies seed 0, four fixed epochs on
+EasyCom Sessions 4--12, and final evaluation on Sessions 1--3. The reported
+run is `results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt`; its
+actual checkpoint hash/config and sample counts were not supplied and remain
+unverified. The base is the released `miniroad_easycom_AV.pth` checkpoint.
+Implementation: [`stage-one runbook`](../../docs/easycom_egospeak_omni_residual.md).
+
+The original MiniROADA A+V path and Omni backbone remain frozen; only the
+tail-phase-aware residual fusion is trained. Existing Omni states use the
+8-s open-stream configuration and a fixed mean over the last-four stack.
+Both full-timeline and causal-valid dense numbers are retained in
+[`metrics.csv`](metrics.csv). All numbers in this record are percentages;
+deltas are percentage points. Displayed values are rounded console output.
+
+## Findings
+
+| Metric | Frozen EgoSpeak | Fused | Delta |
+|---|---:|---:|---:|
+| Causal-valid 0.2-s dense mAP | 74.204 | 77.739 | +3.535 |
+| Causal-valid 0.2-s wearer AP | 63.750 | 72.716 | +8.966 |
+| Causal-valid ten-horizon dense mean mAP | 61.193 | 62.714 | +1.521 |
+| Full-timeline ten-horizon dense mean mAP | 61.155 | 62.676 | +1.521 |
+| Ten-horizon raw onset endpoint AP | 24.491 | 25.091 | +0.600 |
+| Ten-horizon raw offset endpoint AP | 38.024 | 40.604 | +2.580 |
+
+Offset AP improves at all ten horizons, by about 2.8--3.5 points at
+0.4--1.2 s. Onset AP improves at nine horizons; its largest gain is +1.634
+points at 0.8 s, while long-horizon differences are small. This is a
+single-seed positive screen, strongest for future wearer cessation. It is
+not a significance test or evidence of semantic floor-yield understanding.
+A parameter-matched EgoSpeak-only residual control and additional seeds
+remain necessary to attribute the gain specifically to Omni.
+
+## Raw anticipation definition
+
+For each horizon h=0.2,...,2.0 s, use valid source/target pairs from the same
+session. Onset is evaluated only when wearer activity W(t)=0: positive if
+W(t+h)=1, negative if W(t+h)=0, with score p_wearer(t,h). Offset is evaluated
+only when W(t)=1: positive if W(t+h)=0, negative if W(t+h)=1, with score
+1-p_wearer(t,h). AP ranks all eligible samples pooled over Sessions 1--3;
+the headline averages ten horizon APs. This is endpoint state-change
+anticipation, not one-event-one-vote timestamp detection. Intermediate
+transitions can be invisible when source and endpoint states agree.
+
+The evaluator also computed exploratory/in-sample operating points, but
+only threshold-free endpoint APs are recorded here. They need no Session-4
+threshold calibration. Do not use the accompanying in-sample rate columns
+as confirmatory test results.
+
+## Annotation decision and next stage
+
+The user suspects issues with the provisional LLM-assisted semantic
+annotations. Opportunity-conditioned ENTRY/YIELD evaluation is therefore
+temporarily paused pending an annotation audit. Historical provisional
+metrics remain archived; they are neither human-reviewed ground truth nor
+discarded because of model performance. No new semantic result or semantic
+loss is introduced in this screen.
+
+The next requested implementation is selective MiniROADA GRU/anticipation
+fine-tuning from the stage-one checkpoint, keeping original feature
+extractors and Omni frozen. This is an exploratory continuation of the
+previously documented route, not passage of the still-unfinished P0 semantic
+evaluation gate. Do not tune windows, loss weights or stopping on Sessions
+1--3; compare fixed budgets and preserve the frozen stage-one/original base
+as references. A four-epoch continuation has eight total adaptation epochs
+and is not a matched-budget comparison against the four-epoch first stage.

results/easycom_egospeak_omni_residual_2026-09-16/metrics.csv

@@ -0,0 +1,21 @@
+run,horizon_seconds,dense_full_map,dense_causal_map,dense_causal_background_ap,dense_causal_wearer_ap,dense_causal_others_ap,onset_endpoint_ap,offset_endpoint_ap
+egospeak,0.2,74.124,74.204,68.189,63.750,90.673,5.139,9.696
+egospeak,0.4,68.291,68.349,58.596,59.595,86.857,9.878,18.846
+egospeak,0.6,64.267,64.313,52.373,56.619,83.947,15.389,27.730
+egospeak,0.8,61.091,61.128,47.629,54.172,81.585,21.093,34.971
+egospeak,1.0,59.734,59.768,46.172,52.916,80.216,27.225,40.646
+egospeak,1.2,58.774,58.805,45.840,51.488,79.088,30.997,45.112
+egospeak,1.4,57.471,57.498,43.587,50.765,78.143,33.563,47.880
+egospeak,1.6,57.064,57.090,43.518,49.739,78.012,34.046,50.444
+egospeak,1.8,55.933,55.954,42.227,48.605,77.031,34.030,52.051
+egospeak,2.0,54.802,54.821,40.349,47.393,76.720,33.550,52.862
+fused,0.2,77.658,77.739,69.105,72.716,91.396,5.722,11.756
+fused,0.4,71.510,71.570,59.729,67.888,87.093,10.749,21.920
+fused,0.6,66.855,66.903,53.297,63.553,83.858,16.613,31.248
+fused,0.8,63.212,63.251,48.874,59.713,81.167,22.727,38.383
+fused,1.0,60.961,60.996,46.574,56.953,79.460,27.946,43.741
+fused,1.2,59.410,59.441,45.243,54.917,78.163,31.260,47.922
+fused,1.4,58.375,58.404,43.983,53.446,77.782,33.502,50.269
+fused,1.6,57.336,57.361,42.934,51.712,77.439,34.053,52.173
+fused,1.8,56.222,56.244,41.284,50.607,76.840,34.202,53.814
+fused,2.0,55.216,55.235,39.820,49.470,76.415,34.137,54.812

results/easycom_egospeak_omni_residual_2026-09-16/summary.json

@@ -0,0 +1,23 @@
+{
+  "date": "2026-09-16",
+  "source": "user-reported console output; not locally recomputed",
+  "units": "percent; deltas are percentage points",
+  "split": {"train": [4, 5, 6, 7, 8, 9, 10, 11, 12], "test": [1, 2, 3]},
+  "seed": 0,
+  "epochs": 4,
+  "configuration_provenance": "preceding command; actual checkpoint/config not inspected",
+  "checkpoint": "results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt",
+  "checkpoint_sha256": null,
+  "base_checkpoint": "third_party/EgoSpeak/MiniROAD/checkpoints/miniroad_easycom_AV.pth",
+  "omni_memory_seconds": 8,
+  "omni_layer_fusion": "last-four fixed mean",
+  "num_seeds": 1,
+  "sample_counts": null,
+  "metrics": {
+    "egospeak": {"dense_causal_02_map": 74.204, "dense_causal_02_wearer_ap": 63.750, "dense_causal_mean_map": 61.193, "dense_full_mean_map": 61.155, "onset_mean_ap": 24.491, "offset_mean_ap": 38.024},
+    "fused": {"dense_causal_02_map": 77.739, "dense_causal_02_wearer_ap": 72.716, "dense_causal_mean_map": 62.714, "dense_full_mean_map": 62.676, "onset_mean_ap": 25.091, "offset_mean_ap": 40.604},
+    "fused_minus_egospeak": {"dense_causal_02_map": 3.535, "dense_causal_02_wearer_ap": 8.966, "dense_causal_mean_map": 1.521, "dense_full_mean_map": 1.521, "onset_mean_ap": 0.600, "offset_mean_ap": 2.580}
+  },
+  "semantic_evaluation": "paused: provisional annotation quality requires audit",
+  "claim_boundary": "single-seed dense/endpoint improvement; not semantic ENTRY/YIELD or Omni-specific causal attribution"
+}

test_egospeak_omni_finetune.py

@@ -0,0 +1,407 @@
+import csv
+import io
+import json
+import tempfile
+import unittest
+from contextlib import redirect_stdout
+from pathlib import Path
+from unittest.mock import patch
+
+import numpy as np
+import torch
+
+from egospeak_omni_finetune import (
+    STAGE_ONE_FORMAT,
+    STAGE_TWO_FORMAT,
+    EgoSpeakOmniFinetune,
+    base_model_config,
+    load_stage_one_checkpoint,
+    load_stage_two_checkpoint,
+    validate_training_sessions,
+)
+from egospeak_omni_residual import (
+    EgoSpeakOmniResidual,
+    MiniROADABase,
+    load_easycom_session_arrays,
+    miniroad_normalized_loss,
+    sha256_file,
+)
+from infer_egospeak_omni_finetune import main as infer_main
+from train_egospeak_omni_finetune import (
+    main,
+    parse_args,
+    prepare_output,
+    raw_endpoint_metrics,
+    validate_checkpoint_contract,
+)
+
+
+def reference_model():
+    torch.manual_seed(7)
+    base = MiniROADABase(
+        rgb_dim=4, audio_dim=6, embedding_dim=8, hidden_dim=5, dropout=0.4
+    )
+    config = {
+        "omni_dim": 7,
+        "fusion_dim": 4,
+        "tail_embedding_dim": 3,
+        "num_tail_bins": 10,
+        "dropout": 0.3,
+    }
+    model = EgoSpeakOmniResidual(base, **config)
+    # Simulate a trained, nonzero first-stage residual that must not be reset.
+    with torch.no_grad():
+        model.residual_output.weight.normal_(std=0.1)
+        model.residual_output.bias.normal_(std=0.1)
+    model.requires_grad_(False)
+    model.eval()
+    return model, config
+
+
+def stage_one_payload(reference, config):
+    return {
+        "format": STAGE_ONE_FORMAT,
+        "model_config": config,
+        "base_model_config": base_model_config(reference.base),
+        "model_state_dict": reference.state_dict(),
+        "train_sessions": ["Session_4"],
+        "training_config": {
+            "epochs": 4,
+            "window_size": 8,
+            "stride": 1,
+            "expected_omni_mode": "open-stream-replay",
+            "omni_layer_fusion": "mean",
+        },
+    }
+
+
+class EgoSpeakOmniFinetuneTest(unittest.TestCase):
+    def setUp(self):
+        # Small synthetic tests should not spin up many CPU BLAS workers.
+        self.previous_threads = torch.get_num_threads()
+        torch.set_num_threads(1)
+
+    def tearDown(self):
+        torch.set_num_threads(self.previous_threads)
+
+    def test_nonzero_stage_one_recovery_gradients_and_reference_immutability(self):
+        reference, config = reference_model()
+        model = EgoSpeakOmniFinetune.from_stage_one(reference, config)
+        inputs = (
+            torch.randn(2, 9, 4),
+            torch.randn(2, 9, 6),
+            torch.randn(2, 7),
+            torch.tensor([0, 9]),
+        )
+        expected, _ = reference.forward_last(*inputs)
+        actual, _ = model.forward_last(*inputs)
+        torch.testing.assert_close(actual, expected, rtol=0, atol=0)
+        self.assertGreater(float(reference.residual_output.weight.abs().sum()), 0)
+        original = {k: v.clone() for k, v in reference.state_dict().items()}
+        model.train()
+        self.assertFalse(model.base.layer1.training)
+        self.assertTrue(model.base.gru.training)
+        self.assertTrue(model.base.anticipation_layer.training)
+        logits, _ = model.forward_last(*inputs)
+        targets = torch.zeros_like(logits)
+        targets[..., 1] = 1
+        loss = miniroad_normalized_loss(logits, targets)
+        loss.backward()
+        for module in (
+            model.base.gru,
+            model.base.anticipation_layer,
+            model.base.f_classification,
+            model.residual_output,
+            model.omni_projection,
+        ):
+            norm = sum(
+                float(p.grad.abs().sum())
+                for p in module.parameters()
+                if p.grad is not None
+            )
+            self.assertGreater(norm, 0)
+        self.assertTrue(all(p.grad is None for p in model.base.layer1.parameters()))
+        self.assertTrue(all(p.grad is None for p in reference.parameters()))
+        groups = model.optimizer_groups(fusion_lr=1e-3, base_lr=1e-4)
+        ids = [id(p) for group in groups for p in group["params"]]
+        self.assertEqual(len(ids), len(set(ids)))
+        self.assertEqual(set(ids), {id(p) for p in model.trainable_parameters()})
+        before = model.base.gru.weight_ih_l0.detach().clone()
+        torch.optim.AdamW(groups).step()
+        self.assertGreater(
+            float((before - model.base.gru.weight_ih_l0).abs().max().detach()), 0
+        )
+        for key, value in reference.state_dict().items():
+            torch.testing.assert_close(value, original[key], rtol=0, atol=0)
+
+    def test_freezing_policies(self):
+        reference, config = reference_model()
+        for policy in ("none", "heads", "gru-heads"):
+            with self.subTest(policy=policy):
+                model = EgoSpeakOmniFinetune.from_stage_one(
+                    reference, config, base_trainable=policy
+                )
+                model.train()
+                for name, parameter in model.base.named_parameters():
+                    expected = (name.startswith("gru.") and policy == "gru-heads") or (
+                        name.startswith(("anticipation_layer.", "f_classification."))
+                        and policy != "none"
+                    )
+                    self.assertEqual(parameter.requires_grad, expected, name)
+                self.assertFalse(model.base.layer1[-1].training)
+                self.assertEqual(
+                    len(model.optimizer_groups(fusion_lr=1e-4, base_lr=1e-5)),
+                    1 if policy == "none" else 2,
+                )
+        with self.assertRaises(ValueError):
+            EgoSpeakOmniFinetune.from_stage_one(reference, config, base_trainable="all")
+
+    def test_continuation_inherits_and_locks_window_and_readout(self):
+        reference, config = reference_model()
+        payload = stage_one_payload(reference, config)
+        required = [
+            "--egospeak-feature-root",
+            "/tmp/ego",
+            "--omni-feature-root",
+            "/tmp/omni",
+            "--stage-one-checkpoint",
+            "/tmp/stage1.pt",
+            "--output-dir",
+            "/tmp/stage2",
+        ]
+        args = parse_args(required)
+        validate_checkpoint_contract(args, payload)
+        self.assertEqual((args.window_size, args.stride), (8, 1))
+        for options in (
+            ["--window-size", "16"],
+            ["--stride", "4"],
+            ["--omni-layer-fusion", "last"],
+            ["--expected-omni-mode", "any"],
+        ):
+            with self.subTest(options=options), self.assertRaises(ValueError):
+                validate_checkpoint_contract(parse_args(required + options), payload)
+
+    def test_sequence_is_causal_and_chunked_recurrent_inference_matches(self):
+        reference, config = reference_model()
+        model = EgoSpeakOmniFinetune.from_stage_one(reference, config)
+        rgb, audio = torch.randn(1, 15, 4), torch.randn(1, 15, 6)
+        omni, tail = torch.randn(1, 15, 7), torch.arange(15)[None] % 10
+        with torch.no_grad():
+            full, _, _ = model.forward_sequence_chunk(rgb, audio, omni, tail, None)
+            prefix, _, state = model.forward_sequence_chunk(
+                rgb[:, :6], audio[:, :6], omni[:, :6], tail[:, :6], None
+            )
+            suffix, _, _ = model.forward_sequence_chunk(
+                rgb[:, 6:], audio[:, 6:], omni[:, 6:], tail[:, 6:], state
+            )
+            torch.testing.assert_close(
+                full, torch.cat((prefix, suffix), dim=1), rtol=1e-5, atol=1e-6
+            )
+            rgb[:, 6:] += 100
+            audio[:, 6:] -= 100
+            omni[:, 6:] += 100
+            changed, _, _ = model.forward_sequence_chunk(rgb, audio, omni, tail, None)
+            torch.testing.assert_close(full[:, :6], changed[:, :6], rtol=0, atol=0)
+
+    def test_checkpoint_loaders_reject_wrong_formats_test_training_and_nonfinite(self):
+        reference, config = reference_model()
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory) / "checkpoint.pt"
+            payload = stage_one_payload(reference, config)
+            torch.save(payload, path)
+            loaded, _ = load_stage_one_checkpoint(path)
+            torch.testing.assert_close(
+                loaded.residual_output.weight, reference.residual_output.weight
+            )
+            for change in (
+                {"format": STAGE_TWO_FORMAT},
+                {"train_sessions": ["Session_1"]},
+                {"model_config": {**config, "num_tail_bins": 9}},
+            ):
+                torch.save({**payload, **change}, path)
+                with self.assertRaises(ValueError):
+                    load_stage_one_checkpoint(path)
+            with torch.no_grad():
+                payload["model_state_dict"]["residual_output.bias"][0] = float("nan")
+            torch.save(payload, path)
+            with self.assertRaisesRegex(ValueError, "nonfinite"):
+                load_stage_one_checkpoint(path)
+        with self.assertRaises(FileNotFoundError):
+            load_stage_one_checkpoint(Path("/does-not-exist/checkpoint.pt"))
+        for sessions in (["Session_1"], ["Session_4", "Session_4"], []):
+            with self.assertRaises(ValueError):
+                validate_training_sessions(sessions)
+
+    def test_raw_ap_uses_endpoint_labels_and_has_no_calibration_requirement(self):
+        targets = np.zeros((14, 3), dtype=np.float32)
+        targets[:, 0] = 1
+        wearer = np.array([0, 0, 1, 1] * 3 + [0, 0], dtype=bool)
+        targets[wearer, 0] = 0
+        targets[wearer, 1] = 1
+        predictions = np.zeros((14, 3, 10), dtype=np.float32)
+        predictions[:, 1, :] = targets[:, 1, None]
+        valid = np.ones((14, 10), dtype=bool)
+        metrics = raw_endpoint_metrics(
+            {"Session_1": predictions}, {"Session_1": valid}, {"Session_1": targets}
+        )
+        self.assertEqual(metrics["horizons"][0]["onset"]["average_precision"], 1)
+        self.assertEqual(metrics["horizons"][0]["offset"]["average_precision"], 1)
+        # Four-step horizons have no positives, so headline stays undefined.
+        self.assertIsNone(metrics["mean_onset_ap"])
+
+    def test_cpu_end_to_end_smoke_export_reload_and_fixed_reference_predictions(self):
+        reference, config = reference_model()
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            checkpoint = root / "stage1" / "checkpoint_last.pt"
+            checkpoint.parent.mkdir()
+            torch.save(stage_one_payload(reference, config), checkpoint)
+            ego, omni = root / "ego", root / "omni"
+            for sub in ("rgb_kinetics_resnet50", "audio_wav2vec2", "target_perframe"):
+                (ego / sub).mkdir(parents=True)
+            omni.mkdir()
+            steps = 36
+            rng = np.random.default_rng(3)
+            np.save(
+                ego / "rgb_kinetics_resnet50" / "Session_4.npy",
+                rng.standard_normal((steps, 4)).astype(np.float32),
+            )
+            np.save(
+                ego / "audio_wav2vec2" / "Session_4.npy",
+                rng.standard_normal((steps, 6)).astype(np.float32),
+            )
+            target = np.eye(3, dtype=np.float32)[np.arange(steps) % 3]
+            np.save(ego / "target_perframe" / "Session_4.npy", target)
+            torch.save(
+                {
+                    "mode": "open-stream-replay",
+                    "states": torch.randn(steps, 2, 7),
+                    "times": (torch.arange(steps, dtype=torch.float64) + 1) / 5,
+                    "label_indices": torch.arange(steps),
+                    "start": 0,
+                    "config": {"step_seconds": 0.2, "native_block_seconds": 2.0},
+                },
+                omni / "Session_4.pt",
+            )
+            session = load_easycom_session_arrays(
+                ego, omni, "Session_4", rgb_dim=4, audio_dim=6, omni_layer_fusion="mean"
+            )
+            output = root / "stage2"
+            arguments = [
+                "--egospeak-feature-root",
+                str(ego),
+                "--omni-feature-root",
+                str(omni),
+                "--stage-one-checkpoint",
+                str(checkpoint),
+                "--output-dir",
+                str(output),
+                "--train-sessions",
+                "Session_4",
+                "--eval-sessions",
+                "Session_4",
+                "--device",
+                "cpu",
+                "--epochs",
+                "1",
+                "--batch-size",
+                "2",
+                "--num-workers",
+                "0",
+                "--max-train-examples",
+                "4",
+                "--no-progress",
+            ]
+            with (
+                patch(
+                    "train_egospeak_omni_finetune.load_sessions", return_value=[session]
+                ),
+                redirect_stdout(io.StringIO()),
+            ):
+                main(arguments)
+            loaded, payload = load_stage_two_checkpoint(output / "checkpoint_last.pt")
+            self.assertEqual(payload["budgets"]["total_adaptation_epochs"], 5)
+            self.assertGreater(
+                payload["history"][0]["first_batch_gradient_norms"]["miniroada"], 0
+            )
+            self.assertEqual(loaded.base_trainable, "gru-heads")
+            self.assertTrue((output / "README.md").is_file())
+            infer_output = root / "infer_only"
+            before_hash = sha256_file(output / "checkpoint_last.pt")
+            with (
+                patch(
+                    "infer_egospeak_omni_finetune.load_sessions", return_value=[session]
+                ),
+                redirect_stdout(io.StringIO()),
+            ):
+                infer_main(
+                    [
+                        "--checkpoint",
+                        str(output / "checkpoint_last.pt"),
+                        "--output-dir",
+                        str(infer_output),
+                        "--sessions",
+                        "Session_4",
+                        "--device",
+                        "cpu",
+                        "--no-progress",
+                    ]
+                )
+            self.assertEqual(before_hash, sha256_file(output / "checkpoint_last.pt"))
+            np.testing.assert_array_equal(
+                np.load(output / "raw_eval" / "Session_4.npy"),
+                np.load(infer_output / "raw_eval" / "Session_4.npy"),
+            )
+            self.assertTrue(
+                json.loads((infer_output / "metrics.json").read_text())[
+                    "inference_only"
+                ]
+            )
+            for method in (
+                output,
+                output / "stage_one_reference",
+                output / "frozen_egospeak_reference",
+            ):
+                pred = np.load(method / "pred" / "Session_4.npy")
+                valid = np.load(method / "valid" / "Session_4.npy")
+                self.assertEqual(pred.shape, (steps, 3, 10))
+                self.assertEqual(valid.shape, (steps, 10))
+                self.assertEqual(valid.sum(), (steps - 10) * 10)
+                self.assertTrue(np.isfinite(pred).all())
+                np.testing.assert_array_equal(
+                    valid, np.load(output / "valid" / "Session_4.npy")
+                )
+            with torch.no_grad():
+                expected_stage1, expected_original, _ = (
+                    reference.forward_sequence_chunk(
+                        torch.from_numpy(np.asarray(session.rgb).copy())[
+                            None, : steps - 10
+                        ],
+                        torch.from_numpy(np.asarray(session.audio).copy())[
+                            None, : steps - 10
+                        ],
+                        session.omni[None, : steps - 10],
+                        session.tail_bins[None, : steps - 10],
+                        None,
+                    )
+                )
+            for name, expected in (
+                ("stage_one_reference", expected_stage1),
+                ("frozen_egospeak_reference", expected_original),
+            ):
+                np.testing.assert_allclose(
+                    np.load(output / name / "raw_eval" / "Session_4.npy"),
+                    expected.softmax(-1)[0].numpy(),
+                    rtol=0,
+                    atol=0,
+                )
+            metrics = json.loads((output / "metrics.json").read_text())
+            self.assertIn("raw_endpoint_anticipation", metrics)
+            with (output / "raw_endpoint_metrics.csv").open() as handle:
+                self.assertEqual(len(list(csv.DictReader(handle))), 30)
+            with self.assertRaises(FileExistsError):
+                prepare_output(output)
+
+
+if __name__ == "__main__":
+    unittest.main()

train_egospeak_omni_finetune.py

@@ -0,0 +1,520 @@
+#!/usr/bin/env python3
+"""Continue stage-one fusion with selective MiniROADA GRU/head fine-tuning.
+
+Frozen RGB/audio features and causal Omni states are reused. Final evaluation
+exports stage-two, immutable stage-one and original EgoSpeak predictions under
+identical masks, plus dense and threshold-free raw endpoint AP. No semantic
+annotations, dev checkpoint selection or test-side tuning are used.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import torch
+from tqdm.auto import tqdm
+
+from easycom_head import (
+    CLASS_NAMES,
+    NUM_HORIZONS,
+    OFFICIAL_TEST_SESSIONS,
+    OFFICIAL_TRAIN_SESSIONS,
+)
+from egospeak_omni_finetune import (
+    STAGE_TWO_FORMAT,
+    EgoSpeakOmniFinetune,
+    base_model_config,
+    load_stage_one_checkpoint,
+    validate_training_sessions,
+)
+from egospeak_omni_residual import (
+    EgoSpeakOmniResidual,
+    EgoSpeakOmniWindowDataset,
+    align_source_predictions,
+    miniroad_normalized_loss,
+    sha256_file,
+)
+from evaluate_floor_state_changes import evaluate_horizon
+from train_egospeak_omni_residual import (
+    autocast_context,
+    evaluate_predictions,
+    load_sessions,
+    make_loader,
+    normalize_sessions,
+    omni_contract_signature,
+    predict_session,
+    print_metrics,
+    resolve_device,
+    set_seed,
+)
+
+
+def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--egospeak-feature-root", type=Path, required=True)
+    parser.add_argument("--omni-feature-root", type=Path, required=True)
+    parser.add_argument("--stage-one-checkpoint", type=Path, required=True)
+    parser.add_argument("--output-dir", type=Path, required=True)
+    parser.add_argument("--train-sessions", nargs="+", default=None)
+    parser.add_argument("--eval-sessions", nargs="+", default=None)
+    parser.add_argument("--expected-omni-mode", default="open-stream-replay")
+    parser.add_argument(
+        "--omni-layer-fusion", choices=("single", "last", "mean"), default="mean"
+    )
+    parser.add_argument(
+        "--base-trainable",
+        choices=("none", "heads", "gru-heads"),
+        default="gru-heads",
+        help="none is a matched extra-epoch fusion-only control; layer1 always stays frozen.",
+    )
+    parser.add_argument(
+        "--epochs", type=int, default=4, help="Additional stage-two epochs."
+    )
+    parser.add_argument("--batch-size", type=int, default=64)
+    parser.add_argument("--num-workers", type=int, default=4)
+    parser.add_argument(
+        "--window-size", type=int, default=None, help="Inherit stage-one window."
+    )
+    parser.add_argument(
+        "--stride", type=int, default=None, help="Inherit stage-one sampling stride."
+    )
+    parser.add_argument("--fusion-lr", type=float, default=1e-4)
+    parser.add_argument("--base-lr", type=float, default=1e-5)
+    parser.add_argument("--weight-decay", type=float, default=0.05)
+    parser.add_argument("--grad-clip", type=float, default=1.0)
+    parser.add_argument("--seed", type=int, default=0)
+    parser.add_argument("--device", default="cuda")
+    parser.add_argument("--amp", action="store_true")
+    parser.add_argument("--eval-chunk-size", type=int, default=0)
+    parser.add_argument("--max-train-examples", type=int, default=None)
+    parser.add_argument("--no-progress", action="store_true")
+    return parser.parse_args(argv)
+
+
+def validate_checkpoint_contract(
+    args: argparse.Namespace, payload: dict[str, Any]
+) -> None:
+    training = payload["training_config"]
+    for key in ("omni_layer_fusion", "expected_omni_mode"):
+        if key not in training or getattr(args, key) != training[key]:
+            raise ValueError(f"Stage-two {key} must match stage-one extraction/readout")
+    if args.window_size is None:
+        args.window_size = int(training["window_size"])
+    if args.stride is None:
+        args.stride = int(training["stride"])
+    if args.window_size != training["window_size"] or args.stride != training["stride"]:
+        raise ValueError(
+            "Keep stage-one window_size/stride fixed for this continuation"
+        )
+
+
+@torch.inference_mode()
+def verify_stage_one_recovery(
+    model: EgoSpeakOmniFinetune,
+    reference: EgoSpeakOmniResidual,
+    dataset: EgoSpeakOmniWindowDataset,
+    device: torch.device,
+) -> float:
+    model.eval()
+    reference.eval()
+    rgb, audio, omni, tail, _, _, source = dataset[0]
+    inputs = (
+        rgb[None].to(device),
+        audio[None].to(device),
+        omni[None].to(device),
+        tail[None].to(device),
+    )
+    actual, actual_base = model.forward_last(*inputs)
+    expected, expected_base = reference.forward_last(*inputs)
+    maximum = max(
+        float((actual - expected).abs().max()),
+        float((actual_base - expected_base).abs().max()),
+    )
+    if maximum != 0.0:
+        raise RuntimeError(
+            f"Stage-two initialization changed stage-one logits: {maximum}"
+        )
+    print("Initialization audit: stage-two logits == trained stage-one logits exactly")
+    print(
+        f"Causal sample: source_row={int(source)}, cutoff={(int(source) + 1) / 5:.2f}s; "
+        "input ends at source row, labels are source+1,...,source+10"
+    )
+    return maximum
+
+
+def train_epoch(model, loader, optimizer, scaler, device, args) -> dict[str, Any]:
+    model.train()
+    total_loss = 0.0
+    num_examples = 0
+    first_batch_gradient_norms = {}
+    iterator = tqdm(
+        loader, desc="train stage2", disable=args.no_progress, dynamic_ncols=True
+    )
+    for rgb, audio, omni, tail, targets, _, _ in iterator:
+        rgb = rgb.to(device=device, dtype=torch.float32)
+        audio = audio.to(device=device, dtype=torch.float32)
+        omni = omni.to(device=device, dtype=torch.float32)
+        tail = tail.to(device)
+        targets = targets.to(device=device, dtype=torch.float32)
+        optimizer.zero_grad(set_to_none=True)
+        with autocast_context(device, args.amp):
+            logits, _ = model.forward_last(rgb, audio, omni, tail)
+            loss = miniroad_normalized_loss(logits.float(), targets)
+        if not torch.isfinite(loss):
+            raise RuntimeError("Nonfinite stage-two training loss")
+        scaler.scale(loss).backward()
+        scaler.unscale_(optimizer)
+        if not num_examples:
+            for group in optimizer.param_groups:
+                norms = [
+                    p.grad.detach().norm()
+                    for p in group["params"]
+                    if p.grad is not None
+                ]
+                norm = float(torch.stack(norms).norm()) if norms else 0.0
+                first_batch_gradient_norms[group["name"]] = norm
+            print(
+                f"First-batch gradient norms (before clipping): {first_batch_gradient_norms}"
+            )
+        torch.nn.utils.clip_grad_norm_(
+            model.trainable_parameters(),
+            args.grad_clip,
+            error_if_nonfinite=True,
+        )
+        scaler.step(optimizer)
+        scaler.update()
+        num_examples += len(rgb)
+        total_loss += float(loss.detach()) * len(rgb)
+        iterator.set_postfix_str(f"loss={total_loss / num_examples:.5f}")
+    return {
+        "loss": total_loss / num_examples,
+        "num_examples": num_examples,
+        "first_batch_gradient_norms": first_batch_gradient_norms,
+    }
+
+
+def raw_endpoint_metrics(predictions, valid, targets) -> dict[str, Any]:
+    names = list(predictions)
+    target = np.concatenate([targets[name] for name in names])
+    prediction = np.concatenate([predictions[name] for name in names])
+    mask = np.concatenate([valid[name] for name in names])
+    session_ids = np.concatenate(
+        [np.full(len(targets[name]), i) for i, name in enumerate(names)]
+    )
+    horizons = []
+    for horizon in range(NUM_HORIZONS):
+        item = evaluate_horizon(target, prediction, mask, session_ids, horizon, [])
+        horizons.append(
+            {
+                "seconds": item["nominal_seconds"],
+                "onset": item["wearer_off_to_on_endpoint"],
+                "offset": item["wearer_on_to_off_endpoint"],
+            }
+        )
+    means = {}
+    for family in ("onset", "offset"):
+        values = [h[family]["average_precision"] for h in horizons]
+        # Undefined horizons stay undefined rather than being silently dropped.
+        means[f"mean_{family}_ap"] = (
+            None if any(v is None for v in values) else float(np.mean(values))
+        )
+    return {
+        "protocol": "causal_valid_endpoint_state_change_ap",
+        "horizons": horizons,
+        **means,
+    }
+
+
+def prepare_output(path: Path) -> Path:
+    path = path.expanduser().resolve()
+    if path.exists() and (not path.is_dir() or any(path.iterdir())):
+        raise FileExistsError(f"Refusing to overwrite output: {path}")
+    path.mkdir(parents=True, exist_ok=True)
+    for method in (
+        path,
+        path / "stage_one_reference",
+        path / "frozen_egospeak_reference",
+    ):
+        for name in ("raw_eval", "pred", "valid"):
+            (method / name).mkdir(parents=True)
+    return path
+
+
+def main(argv: list[str] | None = None) -> None:
+    args = parse_args(argv)
+    if (
+        args.epochs <= 0
+        or args.batch_size <= 0
+        or args.eval_chunk_size < 0
+        or args.num_workers < 0
+        or args.grad_clip <= 0
+        or args.weight_decay < 0
+        or not all(
+            math.isfinite(value)
+            for value in (
+                args.fusion_lr,
+                args.base_lr,
+                args.grad_clip,
+                args.weight_decay,
+            )
+        )
+    ):
+        raise ValueError("Invalid stage-two epoch/batch/chunk/worker/gradient settings")
+    set_seed(args.seed)
+    device = resolve_device(args.device)
+    if args.amp and device.type != "cuda":
+        raise ValueError("--amp requires CUDA")
+    reference, parent = load_stage_one_checkpoint(args.stage_one_checkpoint)
+    validate_checkpoint_contract(args, parent)
+    train_names = normalize_sessions(args.train_sessions, OFFICIAL_TRAIN_SESSIONS)
+    eval_names = normalize_sessions(args.eval_sessions, OFFICIAL_TEST_SESSIONS)
+    validate_training_sessions(train_names)
+    train_sessions = load_sessions(args, train_names)
+    eval_sessions = load_sessions(args, eval_names)
+    sessions = (*train_sessions, *eval_sessions)
+    if {int(s.omni.shape[-1]) for s in sessions} != {reference.omni_dim}:
+        raise ValueError("Omni feature dimensions differ from stage-one checkpoint")
+    signatures = {omni_contract_signature(s) for s in sessions}
+    if len(signatures) != 1:
+        raise ValueError("Train/evaluation Omni extraction contracts differ")
+    # v1 saves readout/mode but not the entire extraction contract in its
+    # checkpoint. Verify sibling metrics provenance when it is available.
+    parent_metrics = (
+        args.stage_one_checkpoint.expanduser().resolve().parent / "metrics.json"
+    )
+    contract_verified = False
+    if parent_metrics.is_file():
+        metadata = json.loads(parent_metrics.read_text()).get("omni_sources", {})
+        for source in metadata.values():
+            config = source["config"]
+            signature = (
+                source["model_id"],
+                source["mode"],
+                config.get("memory_seconds"),
+                config.get("video_fps"),
+                config.get("readout_last_n_layers"),
+                config.get("readout_layer_fusion"),
+                tuple(source["state_shape_before_layer_fusion"][1:]),
+            )
+            if signature not in signatures:
+                raise ValueError(
+                    "Current Omni extraction differs from stage-one metrics provenance"
+                )
+        contract_verified = bool(metadata)
+    if not contract_verified:
+        print(
+            "WARNING: stage-one full extraction provenance unavailable; reuse its exact Omni root."
+        )
+
+    model = EgoSpeakOmniFinetune.from_stage_one(
+        reference,
+        parent["model_config"],
+        base_trainable=args.base_trainable,
+    ).to(device)
+    reference = reference.to(device)
+    groups = model.optimizer_groups(fusion_lr=args.fusion_lr, base_lr=args.base_lr)
+    for group in groups:
+        print(
+            f"parameters {group['name']}: {sum(p.numel() for p in group['params']):,}; lr={group['lr']:g}"
+        )
+    print(f"base_trainable={args.base_trainable}; layer1 frozen=True; Omni frozen=True")
+    print(f"train={train_names}; final evaluation={eval_names}")
+    dataset = EgoSpeakOmniWindowDataset(
+        train_sessions,
+        window_size=args.window_size,
+        stride=args.stride,
+        seed=args.seed,
+        max_examples=args.max_train_examples,
+    )
+    difference = verify_stage_one_recovery(model, reference, dataset, device)
+    output = prepare_output(args.output_dir)
+    optimizer = torch.optim.AdamW(groups, weight_decay=args.weight_decay)
+    try:
+        scaler = torch.amp.GradScaler("cuda", enabled=args.amp)
+    except (AttributeError, TypeError):  # Older PyTorch compatibility.
+        scaler = torch.cuda.amp.GradScaler(enabled=args.amp)
+    history = []
+    for epoch in range(1, args.epochs + 1):
+        dataset.resample(epoch - 1)
+        result = train_epoch(
+            model, make_loader(dataset, args, epoch), optimizer, scaler, device, args
+        )
+        result["epoch"] = epoch
+        history.append(result)
+        print(
+            f"stage2 epoch {epoch}/{args.epochs}: loss={result['loss']:.6f}; examples={result['num_examples']}"
+        )
+
+    training_config = vars(args).copy()
+    training_config = {
+        k: str(v) if isinstance(v, Path) else v for k, v in training_config.items()
+    }
+    budgets = {
+        "stage_one_epochs": int(parent["training_config"]["epochs"]),
+        "stage_two_epochs": args.epochs,
+        "total_adaptation_epochs": int(parent["training_config"]["epochs"])
+        + args.epochs,
+    }
+    checkpoint = output / "checkpoint_last.pt"
+    provenance = {
+        "stage_one_checkpoint": str(args.stage_one_checkpoint.expanduser().resolve()),
+        "stage_one_checkpoint_sha256": sha256_file(
+            args.stage_one_checkpoint.expanduser().resolve()
+        ),
+        "original_base_checkpoint": parent.get("base_checkpoint"),
+        "original_base_checkpoint_sha256": parent.get("base_checkpoint_sha256"),
+        "stage_one_full_extraction_contract_verified": contract_verified,
+    }
+    (output / "README.md").write_text(
+        "# Selective EgoSpeak + Omni fine-tuning\n\n"
+        f"Train: {train_names}. Final evaluation: {eval_names}. Seed: {args.seed}.\n\n"
+        f"Checkpoint: `checkpoint_last.pt`. Base policy: `{args.base_trainable}`. "
+        "RGB/audio feature extractors, input projection and Omni remain frozen.\n\n"
+        f"Epoch budgets: {budgets}. This continuation is not budget-matched "
+        "against stage one; use the `--base-trainable none` continuation control.\n\n"
+        "`metrics.json` reports dense AP/mAP and threshold-free raw endpoint AP "
+        "as fractions. `raw_endpoint_metrics.csv` has per-horizon AP and counts. "
+        "Raw onset uses OFF->ON versus OFF->OFF, scored by wearer probability; "
+        "offset uses ON->OFF versus ON->ON, scored by 1-wearer probability.\n\n"
+        "Immutable reference predictions are in `stage_one_reference/` and "
+        "`frozen_egospeak_reference/`, each with pred/valid/raw_eval and metrics. "
+        "No provisional semantic annotations are used.\n\n"
+        f"Run arguments:\n```json\n{json.dumps(training_config, indent=2)}\n```\n"
+    )
+    torch.save(
+        {
+            "format": STAGE_TWO_FORMAT,
+            "model_state_dict": model.state_dict(),
+            "base_model_config": base_model_config(model.base),
+            "model_config": parent["model_config"],
+            "base_trainable": args.base_trainable,
+            "training_config": training_config,
+            "train_sessions": train_names,
+            "eval_sessions": eval_names,
+            "stage_one_train_sessions": parent["train_sessions"],
+            "budgets": budgets,
+            "provenance": provenance,
+            "omni_sources": {item.name: item.omni_metadata for item in sessions},
+            "history": history,
+            "initialization_max_abs_stage_one_logit_difference": difference,
+        },
+        checkpoint,
+    )
+
+    directories = {
+        "stage_two": output,
+        "stage_one": output / "stage_one_reference",
+        "egospeak": output / "frozen_egospeak_reference",
+    }
+    predictions = {name: {} for name in directories}
+    valid = {}
+    targets = {}
+    for session in eval_sessions:
+        kwargs = {
+            "chunk_size": args.eval_chunk_size,
+            "amp": args.amp,
+            "show_progress": not args.no_progress,
+        }
+        fused, _ = predict_session(model, session, device, **kwargs)
+        first_stage, original = predict_session(reference, session, device, **kwargs)
+        for method, raw in (
+            ("stage_two", fused),
+            ("stage_one", first_stage),
+            ("egospeak", original),
+        ):
+            aligned, mask = align_source_predictions(raw, session.num_steps)
+            if session.name in valid and not np.array_equal(valid[session.name], mask):
+                raise RuntimeError("Reference and adapted valid masks differ")
+            valid[session.name] = mask
+            predictions[method][session.name] = aligned
+            directory = directories[method]
+            np.save(directory / "raw_eval" / f"{session.name}.npy", raw)
+            np.save(directory / "pred" / f"{session.name}.npy", aligned)
+            np.save(directory / "valid" / f"{session.name}.npy", mask)
+        targets[session.name] = np.asarray(session.targets, dtype=np.float32)
+
+    dense = {
+        name: evaluate_predictions(pred, valid, targets)
+        for name, pred in predictions.items()
+    }
+    endpoints = {
+        name: raw_endpoint_metrics(pred, valid, targets)
+        for name, pred in predictions.items()
+    }
+    for name, directory in directories.items():
+        metrics = {
+            **dense[name],
+            "format": "egospeak-omni-stage-two-eval-v1",
+            "classes": list(CLASS_NAMES),
+            "sessions": eval_names,
+            "train_sessions": train_names,
+            "checkpoint": str(checkpoint)
+            if name == "stage_two"
+            else provenance["stage_one_checkpoint"],
+            "method": name,
+            "seed": args.seed,
+            "prediction_layout": "[target_time, class, future_horizon]",
+            "raw_prediction_layout": "[source_time, future_horizon, class]",
+            "raw_endpoint_anticipation": endpoints[name],
+        }
+        if name == "stage_two":
+            metrics.update(
+                provenance=provenance,
+                budgets=budgets,
+                base_trainable=args.base_trainable,
+                training_config=training_config,
+                omni_sources={item.name: item.omni_metadata for item in sessions},
+                stage_one_reference=dense["stage_one"],
+                frozen_egospeak_baseline=dense["egospeak"],
+                semantic_evaluation="not run: provisional annotation audit pending",
+            )
+        (directory / "metrics.json").write_text(json.dumps(metrics, indent=2) + "\n")
+        print_metrics(name, dense[name])
+    csv_path = output / "raw_endpoint_metrics.csv"
+    with csv_path.open("w", newline="") as handle:
+        fields = [
+            "run",
+            "nominal_seconds",
+            "onset_endpoint_ap",
+            "offset_endpoint_ap",
+            "onset_positive",
+            "onset_negative",
+            "offset_positive",
+            "offset_negative",
+        ]
+        writer = csv.DictWriter(handle, fieldnames=fields)
+        writer.writeheader()
+        for name, metrics in endpoints.items():
+            for item in metrics["horizons"]:
+                writer.writerow(
+                    {
+                        "run": name,
+                        "nominal_seconds": item["seconds"],
+                        "onset_endpoint_ap": item["onset"]["average_precision"],
+                        "offset_endpoint_ap": item["offset"]["average_precision"],
+                        "onset_positive": item["onset"]["num_positive"],
+                        "onset_negative": item["onset"]["num_negative"],
+                        "offset_positive": item["offset"]["num_positive"],
+                        "offset_negative": item["offset"]["num_negative"],
+                    }
+                )
+    print("\nRaw endpoint ten-horizon means (percent; threshold-free):")
+    for name, metrics in endpoints.items():
+        print(
+            name,
+            {
+                k: None if metrics[k] is None else round(metrics[k] * 100, 3)
+                for k in ("mean_onset_ap", "mean_offset_ap")
+            },
+        )
+    print(f"Saved {checkpoint}; raw endpoint CSV (fractions)={csv_path}")
+
+
+if __name__ == "__main__":
+    main()