wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add competing-risk evaluation and align ICASSP story

Browse Source
This commit is contained in:
Xin Wang
2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
PROJECT_CONTEXT.md
View File

@@ -124,23 +124,29 @@ result.
### Current blocker ### Current blocker
`evaluate_floor_state_changes.py` counts a wearer transition anywhere in a `evaluate_floor_state_changes.py` now implements and unit-tests the required
fixed response window. Before this becomes the paper event protocol, add competing-risk attribution: an other-offset response window ends before the
competing-risk censoring: next other onset, and an other-onset yield window ends before the next other
offset. It also supports disjoint training-side threshold calibration, rejects
official-test calibration on Sessions 1--3, reports false-entry/false-stop
counts, and prints auditable timelines.
- an other-offset response window ends at the next other onset; P0 is still not frozen. The remaining blocker is a real-data audit: run the v2
- an other-onset yield window ends at the next other offset. evaluator on saved calibration/test predictions, inspect event timelines and
censoring/invalid counts, independently review sampled events, and freeze the
Until then, EOT/HOLD, response and yield results are proxy diagnostics, not scenario definitions. These values remain timing-derived behavior proxies,
semantic event ground truth. not semantic EOT/INT or addressee ground truth. See the
[`runbook`](docs/easycom_competing_risk_floor_evaluation.md).
### Next experiment gates ### Next experiment gates
The user has temporarily chosen to run the dense-AP model screen while P0 is The user has temporarily chosen to run the dense-AP model screen while P0 is
unfinished. This does not relax the paper claim: event numbers remain proxy unfinished. This does not relax the paper claim: event numbers remain proxy
diagnostics until competing-risk censoring is fixed. diagnostics until the v2 attribution, calibration and scenario audit is frozen
on real data.
1. **P0:** fix/freeze event attribution, splits and scenario definitions. 1. **P0:** run and audit competing-risk v2, then freeze event attribution,
calibration, splits and scenario definitions.
2. **P1:** confirm the aggregate/per-person oracle screen over seeds 1--2, 2. **P1:** confirm the aggregate/per-person oracle screen over seeds 1--2,
then train a causal visible-person active-speaker/floor-evidence estimator; then train a causal visible-person active-speaker/floor-evidence estimator;
compare correct, wrong-person and time-shifted visual inputs. compare correct, wrong-person and time-shifted visual inputs.

83
README.md
View File

@@ -46,8 +46,8 @@ It separates four questions that must not be conflated:
4. Does any suppression preserve addressed-to-wearer response recall/latency 4. Does any suppression preserve addressed-to-wearer response recall/latency
and wearer yield behavior, rather than making the model globally passive? and wearer yield behavior, rather than making the model globally passive?
The immediate order is: freeze the event protocol and competing-risk The immediate order is: run and audit the implemented competing-risk v2 event
attribution; run speaker-grounding probes and wrong-person/time-shift controls; protocol, then freeze it; run speaker-grounding probes and wrong-person/time-shift controls;
re-evaluate `G`, `G+A`, and `G+A+pooled-V`; then train an anonymous person-wise re-evaluate `G`, `G+A`, and `G+A+pooled-V`; then train an anonymous person-wise
causal side branch. A person-memory adapter or Thinker LoRA is a second-stage causal side branch. A person-memory adapter or Thinker LoRA is a second-stage
experiment only after the simpler branch passes those diagnostic gates. experiment only after the simpler branch passes those diagnostic gates.
@@ -392,6 +392,40 @@ bits on CPU (pass `--dataset easycom` for EasyCom; default remains Ego4D).
four-epoch heads with no dev set. This native-concatenation baseline is not four-epoch heads with no dev set. This native-concatenation baseline is not
capacity- or context-matched to the Thinker baseline. capacity- or context-matched to the Thinker baseline.
The first epoch-budget sweep shows severe capacity overfitting. Four epochs
reach **73.42020** at 0.2 seconds and a new frozen-Omni best of **60.87144**
mean mAP, whereas 12 and 50 epochs fall to `57.98809` and `53.03515` despite
lower training losses. The 4-epoch concat-MLP is effectively tied with the
scalar mixture at 0.2 seconds (`-0.03065`) but improves mean mAP by `+0.33721`,
mainly at 1.2--1.6 seconds. Because the runs used different total-length cosine
schedules and the official test sessions were inspected, this is a diagnostic
capacity result rather than a valid final epoch selection. Full tables and
protocol caveats are archived in
[`results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/`](results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/README.md).
### Frozen ROMA last-four-layer transfer
Three frozen ROMA/Qwen2.5-Omni-7B readout runs were recorded on 2026-08-27
using the same 8-second open-stream, 2 FPS, 5 Hz EasyCom evaluation path. A
four-epoch fixed mean over the last four `[3584]` query states is the best ROMA
control at **69.21863** mAP for 0.2 seconds and **58.90402** mean mAP. A
51.4M-parameter concat-MLP reaches only `68.65486 / 56.52425` after four
epochs; relative to the design- and epoch-matched frozen Qwen-3B concat model
it is `-4.76534` at 0.2 seconds and `-4.34719` in mean mAP. This is not
parameter-matched because the 3584-dimensional ROMA fusion is much wider.
This is a negative-transfer diagnostic, not evidence that ROMA is universally
worse: the project used Qwen-style open-stream replay rather than ROMA's native
one-second streaming training recipe. The practical decision is nevertheless
clear: retain frozen Qwen2.5-Omni-3B as the main backbone and stop tuning the
wide ROMA concat head. Full horizon tables, class AP, validity caveats, JSON,
and a formatted workbook are archived in
[`results/easycom_roma_frozen_last4_2026-08-27/`](results/easycom_roma_frozen_last4_2026-08-27/README.md).
The related-work synthesis and proposed next method—recent-200ms payload
readout plus structured future-floor projection—are documented in
[`reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md`](reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md).
## Current Ego4D context-length results ## Current Ego4D context-length results
The following single-seed snapshot was recorded on 2026-08-22. Both variants The following single-seed snapshot was recorded on 2026-08-22. Both variants
@@ -1678,6 +1712,44 @@ At 0.2 seconds, sparse supervision alone changes mAP from `71.69695` to
`71.08589`. This isolates the short-horizon regression primarily to Thinker `71.08589`. This isolates the short-horizon regression primarily to Thinker
updating rather than sparse target selection. updating rather than sparse target selection.
### Dense last-8 LoRA result
Restricting LoRA to `q_proj`, `v_proj`, and `o_proj` in Thinker layers 28--35,
reducing the learning rate to `5e-6`, using dense 5-Hz supervision, and freezing
the trained 8-second head substantially reduces but does not reverse negative
transfer. Five continuous epochs were selected on Session 12 by 0.2-second
mAP; epoch 3 / step 11910 was then evaluated once on Sessions 3, 2, and 1.
| Checkpoint | 0.2s full mAP | Full mean mAP | Delta from frozen dense mean |
|---|---:|---:|---:|
| Frozen Thinker dense-cache control | **71.69695** | **58.16091** | — |
| Dense last-8 q/v/o LoRA, frozen head, epoch 1, train 4--12 | 71.62100 | 58.03491 | -0.12600 |
| Dense last-8 q/v/o LoRA, frozen head, epoch 1, train 4--11 | 71.59585 | 58.01500 | -0.14591 |
| Dense last-8 q/v/o LoRA, frozen head, epoch 3, train 4--11 | 71.65000 | 57.88833 | -0.27258 |
| Prior frozen-head initialized sparse LoRA, 3 epochs | 71.08589 | 56.19608 | -1.96483 |
Epoch 1 is only `0.05415` below epoch 3 at 0.2 seconds but is `0.12667` higher
in mean mAP. It reduces the mean regression relative to frozen dense from
`-0.27258` to `-0.14591`, with better results than epoch 3 at every horizon
from 0.6 through 2.0 seconds. This supports fixing one epoch and retraining on
all official training Sessions 4--12 rather than selecting a later checkpoint.
That full-train run reaches `71.62100` at 0.2 seconds and `58.03491` mean,
improving the Sessions 4--11 epoch-1 result by only `0.02515` and `0.01991`.
It remains `0.12600` mean points below frozen dense, confirming that withholding
Session 12 was not the cause of the residual negative transfer. Its complete
record is in
[`results/easycom_dense_last8_qvo_full_train_epoch1_2026-08-25/`](results/easycom_dense_last8_qvo_full_train_epoch1_2026-08-25/README.md).
For epoch 3, the gap grows almost monotonically to `-0.45040` at 2.0 seconds
and camera-wearer AP is the main regression (`-0.62718` average).
The complete development table, per-horizon/class test table, configuration,
and provenance caveats are archived in
[`results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25/`](results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25/README.md).
This result should be reported as a negative-transfer ablation, not as an Omni
improvement. Any subsequent adapter interpolation or objective change must be
chosen without consulting Sessions 1--3; for a clean publication protocol,
the frozen head must also be retrained without the chosen development session.
For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU
producer-consumer benchmark. The producer GPU runs the frozen AV towers and producer-consumer benchmark. The producer GPU runs the frozen AV towers and
places current-tail embeddings in a bounded CPU queue while the consumer GPU places current-tail embeddings in a bounded CPU queue while the consumer GPU
@@ -1753,9 +1825,10 @@ The headline comparison cannot establish speaker grounding or social
understanding. Experiments therefore proceed through explicit gates: understanding. Experiments therefore proceed through explicit gates:
1. **P0 — protocol correctness:** preserve physical 5 Hz timing and causal 1. **P0 — protocol correctness:** preserve physical 5 Hz timing and causal
masks, then prevent an other-offset response from being attributed after a masks. The evaluator now prevents an other-offset response from being
new other onset (and symmetrically censor yield after the next other attributed at/after a new other onset and symmetrically censors yield at the
offset). Freeze scenario rules without tuning on EasyCom test Sessions 1--3. next other offset; run the real-data audit and freeze scenario rules without
tuning on EasyCom test Sessions 1--3.
2. **P1 — representation diagnosis:** compare EgoSpeak features, Qwen 2. **P1 — representation diagnosis:** compare EgoSpeak features, Qwen
encoder-only A/V/AV, the global Thinker state, and anonymous person slots on encoder-only A/V/AV, the global Thinker state, and anonymous person slots on
aggregate activity and per-person active-speaker probes. Include wrong-person, aggregate activity and per-person active-speaker probes. Include wrong-person,

1
docs/README.md
View File

@@ -31,6 +31,7 @@ tests.
| [`easycom_highrate_person_av.md`](easycom_highrate_person_av.md) | Frozen 25 Hz causal ASD person evidence fused with 5 Hz global Omni context | | [`easycom_highrate_person_av.md`](easycom_highrate_person_av.md) | Frozen 25 Hz causal ASD person evidence fused with 5 Hz global Omni context |
| [`easycom_muvap_stage3_baseline.md`](easycom_muvap_stage3_baseline.md) | MuVAP-only frozen VAP+ASD Stage-3 paper reconstruction on EgoSpeak/EasyCom targets | | [`easycom_muvap_stage3_baseline.md`](easycom_muvap_stage3_baseline.md) | MuVAP-only frozen VAP+ASD Stage-3 paper reconstruction on EgoSpeak/EasyCom targets |
| [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands | | [`easycom_oracle_information.md`](easycom_oracle_information.md) | Causal oracle per-person speaking-history upper bound and Session-12 screening commands |
| [`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md) | Running and auditing the competing-risk entry/yield evaluator with training-side threshold calibration |
## Design history and implementation options ## Design history and implementation options

109
docs/easycom_competing_risk_floor_evaluation.md Normal file
View File

@@ -0,0 +1,109 @@
# EasyCom competing-risk floor-event evaluation
This runbook evaluates whether a candidate model reduces **false wearer-entry
triggers** and **false wearer-yield triggers** relative to a baseline. It reads
the saved `pred/` and `valid/` arrays; no GPU is required.
The protocol is implemented in `evaluate_floor_state_changes.py`. It is a
timing-derived behavior evaluation, not semantic EOT/INT, addressee, or
individual-speaker ground truth: EasyCom's `others` target is aggregate
activity.
## Event attribution
At each horizon from 0.2 to 2.0 seconds, every eligible aggregate-other
boundary is counted once.
- **Response candidate:** `others` changes from active to inactive while the
wearer is inactive. A wearer onset is a response only if it occurs before
the next `others` onset.
- **Yield candidate:** `others` changes from inactive to active while the
wearer is active. A wearer offset is a yield only if it occurs before the
next `others` offset.
- The competing boundary itself is excluded. A simultaneous wearer transition
is therefore not assigned to the earlier candidate.
- An observed wearer outcome before the recording ends is retained. An
unresolved candidate without its full horizon is right-censored and omitted.
- Invalid future-score paths are omitted and counted separately.
For each event, predictions made from the same source time are max-pooled only
over the observable event window. This is a ranking score, not a calibrated
event probability.
## Confirmatory test command
Select each model's operating threshold on training-side predictions, then
apply that frozen threshold to official test Sessions 1--3. The run names in
`--calibration-run` must exactly match the names in `--run`.
```bash
python3 evaluate_floor_state_changes.py \
--target-root /path/to/easycom_target_frame \
--run baseline=/path/to/baseline/test_result \
--run candidate=/path/to/candidate/test_result \
--calibration-run baseline=/path/to/baseline/calibration_result \
--calibration-run candidate=/path/to/candidate/calibration_result \
--calibration-sessions Session_12 \
--sessions Session_3 Session_2 Session_1 \
--matched-recalls 0.7 0.8 0.9 \
--midturn-pause-max-seconds 2.0 \
--print-event-timelines 3 \
--output reports/easycom_floor_competing_risk/test_summary.json
```
If the calibration and test predictions already coexist under each `--run`
directory, omit `--calibration-run`. Official-test evaluation rejects missing
calibration and rejects calibration on Sessions 1--3. The
`--allow-in-sample-test-thresholds` option exists only for explicitly marked
exploratory audits and must not be used for a paper result.
Session 12 is shown as the current development convention. If the final model
has been trained on Session 12, prefer out-of-fold training-side predictions
or a pre-registered cross-fitting scheme; record exactly how calibration
predictions were produced.
## How to decide whether errors decreased
The command writes:
- `test_summary.json`: full protocol metadata, thresholds, counts, and metrics;
- `test_summary_events.csv`: one row per run and horizon for the primary event
evaluation;
- `test_summary_framewise.csv`: the older endpoint diagnostic, retained only
for secondary analysis.
At the headline recall (normally 0.8), compare the following event-CSV fields:
| Question | Primary field | Supporting checks |
|---|---|---|
| Did false entry decrease? | `no_response_trigger_rate_at_recall_0.8` | count, `response_recall_0.8`, response/no-response denominators |
| Did false entry during a short other-speaker restart decrease? | `short_resume_trigger_rate_at_recall_0.8` | trigger count and subgroup count |
| Did false yield/stop decrease? | `continue_false_stop_rate_at_recall_0.8` | count, `yield_recall_0.8`, yield/continue denominators |
Lower is better for all three error rates. A lower error rate is not evidence
of improvement if the candidate also obtains materially lower test response or
yield recall. The requested recall selects the threshold on calibration data;
`response_recall_0.8` and `yield_recall_0.8` report the recall actually achieved
on test and may differ from 0.8.
Before comparing models, confirm that their event denominators and invalid
window counts match. Then report both absolute percentage-point change and
relative error reduction, for example
`(baseline_rate - candidate_rate) / baseline_rate`. Event AP is a useful
threshold-free secondary result, but it does not replace the error-rate
comparison at a pre-registered operating point.
## Required manual audit before protocol freeze
Use `--print-event-timelines 3` first, then inspect a larger random sample with
an independent script or annotation view. For every printed timeline, verify:
1. the anchor row is the intended aggregate-other boundary;
2. no outcome at or after `C/excluded` is attributed to the earlier event;
3. the fixed-source horizon score and `valid` mask line up with physical time;
4. session/recording boundaries do not create transitions;
5. right-censored and invalid counts are plausible.
The current evaluator does not yet provide `third_party`, `addressed`, or
`group_or_unknown` strata, nor false entries per minute. Those require a
separately frozen addressee/scenario annotation contract and remain P0 TODOs.

9
docs/easycom_muvap_stage3_baseline.md
View File

@@ -206,6 +206,9 @@ CUDA_VISIBLE_DEVICES=4 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh tes
CUDA_VISIBLE_DEVICES=5 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh test 2 CUDA_VISIBLE_DEVICES=5 PERSON_EVIDENCE=none bash run_easycom_muvap_stage3.sh test 2
``` ```
Dense AP/mAP is currently the primary screening output. Floor-event results Dense AP/mAP is currently the primary screening output. For floor-event
remain timing-derived proxies until competing-risk censoring is frozen in comparison, run the competing-risk v2 evaluator with training-side threshold
`evaluate_floor_state_changes.py`. calibration as documented in
[`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md).
The resulting events remain timing-derived behavior proxies; a real-data audit
and scenario freeze are still required before using them in the paper.

6
docs/easycom_oracle_information.md
View File

@@ -81,8 +81,10 @@ checkpoint selection, and evaluates Sessions 1--3 once:
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh final 0 1 2 CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh final 0 1 2
``` ```
The `final` stage enforces `EPOCHS=4` and exactly Sessions 4--12, clears the The `final` stage overrides any inherited `EPOCHS` value with `4`, requires
development-session list internally, and saves the last epoch. Its artifacts exactly Sessions 4--12, clears the development-session list internally, and
saves the last epoch. It validates this configuration before preparing oracle
sidecars. Its artifacts
use separate `_train4_12_fixed4` checkpoint/result roots by default, so they use separate `_train4_12_fixed4` checkpoint/result roots by default, so they
cannot overwrite the historical Session-12 development screen. Override those cannot overwrite the historical Session-12 development screen. Override those
roots only with `FINAL_CHECKPOINT_ROOT` and `FINAL_RESULT_ROOT`. roots only with `FINAL_CHECKPOINT_ROOT` and `FINAL_RESULT_ROOT`.

30
docs/icassp2027_acceptance_storyline_2026-09-10.md
View File

@@ -193,3 +193,33 @@ Oracle audit、LoRA 和事件协议分别是动机、baseline 和验证工具，
这个结论只有在 learned causal method、错时/容量控制和冻结后的事件指标共同通过后才能写入 这个结论只有在 learned causal method、错时/容量控制和冻结后的事件指标共同通过后才能写入
摘要。 摘要。
## 10. 2026-09-10 manuscript 对齐决策
对 `manuscript/main/main.tex` 的逐段审查发现并记录以下冲突；稿件骨架已据此收缩：
1. 标题只写 `floor-entry`，正文却同时主张 entry/yield，统一改为更宽的 floor
forecasting；在 same-rate 对照通过前不在标题使用 `dual-rate`。
2. 单种子 oracle 不再充当摘要 headline；最终摘要必须以 learned method 和行为级结果为
主体。
3. gap 从“Omni 尚未用于这一任务组合”改为 global semantic representation 与 sub-second
wearer-relative floor evidence 之间的 temporal--role mismatch。
4. 四项并列贡献压缩为 adaptation mismatch、一个 local adapter、causal behavioral
evidence 三项；LoRA/oracle/event protocol 分别降为 baseline、动机和验证。
5. 主方法不再预设 track-GRU/identity continuity，改为匿名、no-ID 的短时 visible-person
set；track 只有超过 no-ID/shuffled/capacity controls 后才恢复。
6. wearer current state 默认只用于 entry/yield event interpretation。若作为模型输入，必须
先定义严格因果且部署可得的来源，并加入 no-state 对照。
7. Qwen2.5-Omni-3B 为主 backbone，7B 仅作参数规模控制；MiniCPM-o 和跨家族 sweep 从四页
主线删除。
8. Introduction 不得把尚未实现的 competing-risk censoring 写成已完成；P0 evaluator、测试
和人工时间轴是正式事件结论的硬门槛。
9. 已查看的 Sessions 1--3 单种子 exploratory pooled-vision 表从投稿骨架删除，不用于模型
选择或 confirmatory claim。
10. 最终只保留一个主结果表和一个机制消融表；oracle table、conditional track table 和
候选方法树不得并列占据正文。
11. Conclusion 必须报告最终 learned method、matched-recall false-entry、onset/yield 和
causal controls；若这些证据未通过，改写为 information audit 或停止方法型投稿。
当前 `manuscript/main/main.tex` 是带显式 TODO 的内容骨架。TODO 表示未实现、未冻结或未测量，
不得被引用为项目当前能力或实验结论。

5
docs/omni_speaker_floor_experiment_plan_2026-08-30.md
View File

@@ -86,8 +86,9 @@ speaker grounding 或 identity-aware reasoning。
2. 修正事件归因的 competing risk： 2. 修正事件归因的 competing risk：
- other offset 后的 wearer response 只在下一次 other onset 之前归因； - other offset 后的 wearer response 只在下一次 other onset 之前归因；
- other onset 后的 wearer yield 只在下一次 other offset 之前归因。 - other onset 后的 wearer yield 只在下一次 other offset 之前归因。
现有 `evaluate_floor_state_changes.py` 使用固定 horizon 内任意 transition，暂时只能作为 `evaluate_floor_state_changes.py` 已实现并通过合成单元测试；投稿前仍需在真实数据上打印
proxy，可能把后来无关的动作归给前一个边界。 时间轴、复核 censoring/invalid 计数并冻结协议。运行方式见
[`easycom_competing_risk_floor_evaluation.md`](easycom_competing_risk_floor_evaluation.md)。
3. 固定 EasyCom train/test（4--12 / 1--3）；阈值、窗口和规则不能在 test 上调。 3. 固定 EasyCom train/test（4--12 / 1--3）；阈值、窗口和规则不能在 test 上调。
4. 从训练 session 内留出 calibration/dev 子集，用来选择阈值和 early stopping；最终表格 4. 从训练 session 内留出 calibration/dev 子集，用来选择阈值和 early stopping；最终表格
仍在官方 test 上一次性报告。 仍在官方 test 上一次性报告。

886
evaluate_floor_state_changes.py
View File

File diff suppressed because it is too large Load Diff

48
manuscript/main/README.md
View File

@@ -25,32 +25,28 @@ main/
build/ # generated, ignored build/ # generated, ignored
``` ```
The author/affiliation, final results, Method, Experimental Setup, Results and The manuscript is intentionally a content skeleton with visible TODOs. Do not
Conclusion are still explicit TODOs. Do not turn planned experiments into turn planned implementations, controls, or results into completed claims.
reported findings.
The current paper thesis is model-agnostic and deliberately narrower than The frozen paper story is documented in
addressee prediction. It asks whether general-purpose omni representations [`../../docs/icassp2027_acceptance_storyline_2026-09-10.md`](../../docs/icassp2027_acceptance_storyline_2026-09-10.md).
expose the fine-grained evidence needed for 200 ms wearer floor-action The paper asks how long-context Omni understanding can be adapted into
projection, whether PEFT can recover it, and whether causal local audio/person sub-second, wearer-relative floor control for an invisible egocentric wearer.
evidence provides a reliable complement. The initial planned backbone set is: Its single candidate contribution is a causal local floor-evidence adapter:
- Qwen2.5-Omni-3B; - Qwen2.5-Omni-3B is the primary global backbone;
- Qwen2.5-Omni-7B; - matched LoRA/PEFT is a baseline, not a separate contribution;
- MiniCPM-o 4.5. - Qwen2.5-Omni-7B is an optional capacity-scaling control;
- the primary local method uses an anonymous, no-ID visible-person set;
- track continuity is conditional and must defeat no-ID/shuffled/capacity
controls before entering the paper;
- MiniCPM-o and a cross-family sweep are removed from the core plan;
- the oracle is analysis-only motivation, not a deployable method or headline;
- the publication gate is lower competing-risk-censored multi-other false
entry at matched wearer-onset recall without worse valid onset or yield.
The backbone and final fusion method are **not frozen yet**. Current Qwen-3B Competing-risk censoring and training-side threshold calibration are now
numbers are exploratory motivation, not evidence that every omni model has the implemented and unit-tested. The remaining P0 blocker is the real-data event
same limitation. Compare frozen, PEFT, and local-evidence protocols within each audit and protocol/scenario freeze; until then, event values are timing-derived
backbone; report common local encoders separately from model-native local proxies. Already viewed Sessions 1--3 numbers are exploratory internal evidence
encoders. Raw cross-model scores are secondary because the models expose and must not appear in the confirmatory paper table.
different streaming/readout interfaces.
Track-consistent person fusion remains one candidate after pooled and
capacity-matched controls. If selected, dataset participant IDs route
short-term causal state only; they are not semantic inputs. The publication
gate is an event-level reduction in multi-other false entries at matched
wearer-onset recall without degrading onset latency or wearer-yield behavior.
A general omni claim requires a consistent result across at least two backbone
families; otherwise the paper must state a model-specific finding. Selected
examples are explanatory evidence only.

110
manuscript/main/REVIEWER_CHECKLIST.md
View File

@@ -2,13 +2,11 @@
This is an internal execution checklist. It is not part of the submitted PDF. This is an internal execution checklist. It is not part of the submitted PDF.
## Reviewer verdict on the current draft ## Reviewer verdict and frozen paper shape
The potentially publishable question is now model-agnostic: **do The potentially publishable question is: **can a general-purpose Omni readout
general-purpose omni representations expose the fine-grained evidence needed be adapted into 200 ms, wearer-relative floor control by explicitly preserving
for 200 ms egocentric floor action, and can causal local evidence reduce false short-window causal audio and anonymous visible-person evidence?**
wearer entries at matched onset recall when frozen readout or PEFT is
insufficient?**
The contribution is not “the first audio-visual multiparty turn model.” VAP, The contribution is not “the first audio-visual multiparty turn model.” VAP,
EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible EgoSpeak, SICNet, and MuVAP already occupy adjacent territory. The defensible
@@ -17,17 +15,25 @@ wearer, combined with a controlled diagnosis of global omni states and a
behavior-aligned multi-other event protocol. behavior-aligned multi-other event protocol.
Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither Current Qwen2.5-Omni-3B results are exploratory motivation only. They neither
prove a limitation shared by omni models nor establish track-consistent fusion prove a limitation shared by Omni models nor establish track-consistent fusion
as the final method. The planned initial backbone set is Qwen2.5-Omni-3B, as the final method. Qwen2.5-Omni-3B is primary; Qwen2.5-Omni-7B is an optional
Qwen2.5-Omni-7B, and MiniCPM-o 4.5. capacity control. MiniCPM-o and a cross-family sweep are outside the core paper.
## P0 — Freeze the task before the backbone sweep The four-page paper has only three contributions: temporal--role adaptation
mismatch, one causal local floor adapter, and causal behavioral evidence.
LoRA, the oracle, and the event protocol are respectively a baseline,
motivation, and validation mechanism.
## P0 — Freeze the task before confirmatory adaptation runs
- [ ] Define the exact 5 Hz causal cutoff and physical target interval for each - [ ] Define the exact 5 Hz causal cutoff and physical target interval for each
of the ten 0.2 s horizons. of the ten 0.2 s horizons.
- [ ] Freeze wearer-onset/yield and multi-other exchange definitions. - [ ] Freeze wearer-onset/yield and multi-other exchange definitions.
- [ ] Add competing-risk rules: a candidate event ends at the first matched - [x] Implement and unit-test competing-risk rules: a response candidate ends
wearer action, incompatible floor change, timeout, or invalid source boundary. before the next other onset and a yield candidate before the next other
offset; right-censored and invalid windows are counted explicitly.
- [ ] Audit the competing-risk implementation on real timelines and freeze the
timeout/source-boundary convention before confirmatory reporting.
- [ ] Specify overlap, minimum silence, maximum response delay, duplicate - [ ] Specify overlap, minimum silence, maximum response delay, duplicate
detections, source discontinuities, track visibility, offscreen speakers, detections, source discontinuities, track visibility, offscreen speakers,
and whether target-of-speech strata are diagnostic only. and whether target-of-speech strata are diagnostic only.
@@ -41,9 +47,9 @@ Qwen2.5-Omni-7B, and MiniCPM-o 4.5.
least 100 randomly selected events. Publish the number of unique other IDs and least 100 randomly selected events. Publish the number of unique other IDs and
multi-other opportunity minutes before the main sweep. multi-other opportunity minutes before the main sweep.
## P1 — Audit each backbone interface ## P1 — Audit the primary backbone interface
For Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, record: For Qwen2.5-Omni-3B, and for 7B only if used as the scaling control, record:
- [ ] exact model/code revision, processor, dtype, and hardware; - [ ] exact model/code revision, processor, dtype, and hardware;
- [ ] native audio/video sampling and streaming/chunk semantics; - [ ] native audio/video sampling and streaming/chunk semantics;
@@ -57,9 +63,9 @@ a model cannot expose a comparable causal state, report that protocol as
unsupported. Raw scores are not a pure architecture ranking; emphasize unsupported. Raw scores are not a pure architecture ranking; emphasize
within-backbone deltas. within-backbone deltas.
## P2 — Establish the compact representation matrix ## P2 — Establish the compact adaptation matrix
Run three seeds per supported backbone with the same data, physical cutoffs, Run three seeds for the primary backbone with the same data, physical cutoffs,
target, prediction-head capacity, optimizer exposure, and evaluator: target, prediction-head capacity, optimizer exposure, and evaluator:
1. Frozen global causal state. 1. Frozen global causal state.
@@ -67,41 +73,39 @@ target, prediction-head capacity, optimizer exposure, and evaluator:
3. Global state plus recent local audio. 3. Global state plus recent local audio.
4. Global state plus pooled local audio and vision. 4. Global state plus pooled local audio and vision.
Use a common local encoder as the primary cross-backbone control. If native Add a parameter- and history-matched temporal local branch before attributing
local encoders are tested, report them in a separate within-model panel. Do not gain to audiovisual organization. Use the same local adapter for the optional
mix common-encoder and native-encoder deltas in one claim. 7B scaling row.
**Gate:** baseline standard deviations are known; the evaluator reproduces **Gate:** baseline standard deviations are known; the evaluator reproduces
identical scores from saved predictions; parameter counts and training budgets identical scores from saved predictions; parameter counts and training budgets
are reported. A single failed LoRA recipe is evidence about that recipe, not are reported. A single failed LoRA recipe is evidence about that recipe, not
proof that the model cannot adapt. proof that the model cannot adapt.
## P3 — Select local fusion only after P2 ## P3 — Train the no-ID local floor adapter
If local evidence gives a reliable event-level gain, compare candidates in The primary method is a short-window causal encoder over recent mixed audio and
increasing complexity: per-frame anonymous visible-person sets. It uses no participant-ID embedding or
cross-time track state. Compare it with:
1. Pooled local audio/vision. 1. Recent local audio only.
2. Parameter-matched temporal branch. 2. Global/full-frame visual pooling.
3. Per-frame person set with no cross-time identity. 3. A parameter- and history-matched temporal branch.
4. Shared track-wise temporal encoder. 4. Time-shifted local vision.
5. Person–audio cross-attention. 5. Removed local vision.
6. Optional per-person active-speaker auxiliary loss, only after selecting a 6. A same-rate control if the local path runs faster than 5 Hz.
person-aware activity-only model.
Keep the dense EgoSpeak wearer/others target as the primary task. Target of Keep the dense EgoSpeak wearer/others target as the primary task. Target of
speech is analysis-only and never model input. Describe only the train-side speech is analysis-only and never model input. Use ``dual-rate'' only if the
winner as the final method; move rejected candidates to ablations. local input/update rate differs explicitly and defeats the same-rate control.
For a track-aware candidate, temporary IDs route hidden state only: no ID Unit-test variable person count, empty sets, padding, person-set permutation,
embedding, cross-recording identity, or fixed seating slot. Unit-test variable
person count, empty sets, padding, slot permutation, global ID renaming,
source/gap resets, and strict causal cutoffs. source/gap resets, and strict causal cutoffs.
## P4 — Run visual counterfactuals conditionally ## P4 — Consider track continuity only after the no-ID method wins
Only if a track-aware candidate is selected, evaluate one frozen checkpoint Only if a track-aware candidate then beats the no-ID method on training-side
with: data, evaluate one frozen checkpoint with:
- correct person tracks; - correct person tracks;
- clip-consistent global renaming, which must be invariant; - clip-consistent global renaming, which must be invariant;
@@ -121,11 +125,14 @@ retrain test-time counterfactuals.
- [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8. - [ ] Operating points at wearer-onset recall 0.7, 0.8, and 0.9; headline 0.8.
- [ ] False wearer entries per multi-other opportunity minute, onset latency, - [ ] False wearer entries per multi-other opportunity minute, onset latency,
wearer-yield recall/latency, and original ten-horizon AP/mAP. wearer-yield recall/latency, and original ten-horizon AP/mAP.
- [ ] Error strata: backbone, visibility, number of visible people, overlap, - [ ] Error strata: visibility, number of visible people, overlap, offscreen
offscreen activity, track age/fragmentation, and transition proxy. activity, and transition proxy. Add track age/fragmentation only if a
track-aware method is selected.
- [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the - [ ] Ego4D AVD coverage audit on the exact EgoSpeak split before claiming the
same multi-other analysis. same multi-other analysis.
- [ ] Provided-track and online-tracker conditions named separately. - [ ] Name the causal box source and its limitations. Add separate
provided-track/online-tracker rows only if tracking enters the selected
method.
Qualitative cases follow a frozen selection rule and include the largest paired Qualitative cases follow a frozen selection rule and include the largest paired
improvement, a median case, a failure, and one preserved true wearer onset. improvement, a median case, a failure, and one preserved true wearer onset.
@@ -134,23 +141,20 @@ Examples explain rates; they never replace them.
## Paper assembly order ## Paper assembly order
1. Freeze and audit the event protocol. 1. Freeze and audit the event protocol.
2. Implement and causally audit all backbone readouts. 2. Audit the Qwen2.5-Omni-3B readout and matched PEFT baseline.
3. Complete the compact frozen/PEFT/local matrix. 3. Complete the frozen/PEFT/audio/matched-temporal/local-AV matrix.
4. Decide whether the paper can support a cross-family or only model-specific 4. Run correct/time-shift/remove-vision controls and the optional 7B scaling row.
representation claim. 5. Consider track continuity only if the no-ID method already passes.
5. Select a local fusion candidate using training-side event metrics. 6. Produce one confirmatory event table and one mechanism-ablation table.
6. Run conditional visual/track counterfactuals. 7. Draw one compact final system figure; remove rejected methods from the main
7. Produce the cross-backbone event table and paired intervals.
8. Draw one compact final system figure; remove rejected methods from the main
narrative. narrative.
9. Replace every visible TODO and verify the ICASSP 2027 kit/page limit. 8. Replace every visible TODO and verify the ICASSP 2027 kit/page limit.
## Stop/go decisions ## Stop/go decisions
- **General claim:** at least two backbone families show the same frozen/PEFT - **Claim boundary:** with Qwen-only evidence, describe a Qwen2.5-Omni case
weakness and a reliable local-evidence improvement. study and the tested adaptation mechanism; do not generalize to all Omni
- **Model-specific claim:** the effect appears only within Qwen; say so and do models.
not generalize to all omni models.
- **Audio-local reframe:** local audio improves but all visual variants fail; - **Audio-local reframe:** local audio improves but all visual variants fail;
claim a fine-grained acoustic readout bottleneck, not social visual reasoning. claim a fine-grained acoustic readout bottleneck, not social visual reasoning.
- **Track-aware method:** correct continuity beats pooled vision, temporal - **Track-aware method:** correct continuity beats pooled vision, temporal

787
manuscript/main/main.tex
View File

@@ -1,7 +1,6 @@
% Internal ICASSP 2027 draft using the official ICASSP 2026 template bundled % ICASSP 2027 content skeleton. The final paper must use the official 2027
% in this repository. Replace the class/style files when the 2027 author kit % author kit and contain no visible \todo{} markers. Planned experiments are
% is released, and remove every visible \todo{} marker before submission. % intentionally marked rather than presented as completed work.
% Compile from this directory with: make
\documentclass{article} \documentclass{article}
\usepackage{spconf} \usepackage{spconf}
\usepackage{amsmath,amssymb,graphicx,hyperref,xcolor} \usepackage{amsmath,amssymb,graphicx,hyperref,xcolor}
@@ -10,7 +9,8 @@
\newcommand{\vect}[1]{\boldsymbol{#1}} \newcommand{\vect}[1]{\boldsymbol{#1}}
\setlength{\emergencystretch}{2em} \setlength{\emergencystretch}{2em}
\title{Adapting Omni Models for Causal Egocentric Multiparty Floor-Entry Forecasting} \title{Is It My Turn? Causal Local Evidence Adaptation of Omni Models for
Egocentric Floor Forecasting}
% TODO: Insert the final author list and affiliations after the ICASSP 2027 % TODO: Insert the final author list and affiliations after the ICASSP 2027
% review policy and author order are finalized. % review policy and author order are finalized.
@@ -22,603 +22,340 @@
\maketitle \maketitle
\begin{abstract} \begin{abstract}
In multiparty conversation, a pause need not invite an egocentric participant Real-time audiovisual agents must decide not only what to say, but whether an
to speak: it may instead mark a handover between two other people. We study emerging floor opportunity belongs to them. This is difficult in multiparty
causal egocentric multiparty floor-entry forecasting, where the target is the egocentric interaction: the target wearer is behind the camera, while the
future speech participation of the first-person camera wearer---the one visible people provide only indirect evidence about whether the surrounding
participant who is necessarily absent from the visible speaker set. This floor remains occupied. We study strictly causal forecasting of the wearer's
wearer-relative target distinguishes our setting from recent multimodal VAP future speech activity every 200~ms. Although a general-purpose Omni model
models that forecast visible floor holders or observe human--human turn provides rich long-context audio--visual representations, its global readout is
dynamics. Following EgoSpeak, we use causal first-person audio--video and the not explicitly trained to expose sub-second, wearer-relative floor evidence.
human wearer's future speaking behavior as an imitation target, and ask how a We therefore study a candidate causal local floor adapter that combines the global
jointly pretrained audio--visual Omni model should be adapted for predictions Omni state with short-window mixed audio and an anonymous set of visible-person
every 200~ms. Our global--local formulation combines a causal Omni readout with observations, without assuming persistent speaker identity. On EasyCom,
a lightweight pathway for recent mixed audio and visible-person observations. \todo{report the final three-seed dense and competing-risk-censored event
On EasyCom development data, a single-seed oracle analysis finds that perfectly results: false-entry reduction at matched wearer-onset recall, onset latency,
aligned aggregate activity among visible non-wearers improves 0.2-s mAP by yield behavior, confidence intervals, and oracle-gap recovery}. Controlled
6.97 points and ten-horizon mean mAP by 2.91 points over global Omni plus recent \todo{time-shift/remove-vision/same-rate or capacity-matched} interventions
audio, whereas person-resolved histories do not outperform an show that \todo{state the supported mechanism, without claiming addressee or
identity-shuffled control. This motivates timely local floor evidence without speaker identity}. These results demonstrate \todo{the narrow conclusion
assuming that person continuity is useful. The full study compares frozen, actually supported by the final experiments}.
parameter-efficient, and learned local adaptation and evaluates false wearer
entry at matched wearer-onset recall together with yield behavior.
\todo{After the train-side study is locked, replace the final two sentences
with the selected learned local variant and insert: its oracle-gap recovery,
multi-other false-entry reduction at 80\% onset recall, onset/yield
non-degradation, and 95\% intervals. Mention track continuity only if its
controls separate.}
\end{abstract} \end{abstract}
\begin{keywords} \begin{keywords}
conversational agents, turn-taking, speech initiation, egocentric video, conversational agents, turn-taking, egocentric video, multimodal adaptation,
omni models voice activity projection
\end{keywords} \end{keywords}
\section{Introduction} \section{Introduction}
\label{sec:intro} \label{sec:intro}
Natural turn-taking requires a conversational system to anticipate not only Natural turn-taking requires a conversational system to anticipate not only
when speech will stop, but whether an emerging floor opportunity actually when speech will stop, but whether a floor opportunity actually belongs to it.
belongs to it. This distinction is particularly important in multiparty In a dyadic exchange, a pause after one interlocutor finishes often invites the
interaction. In a dyadic conversation, a pause after one interlocutor finishes other to respond. In a group conversation, the same pause may instead mark a
often provides an opportunity for the other to respond; in a group handover between two other participants. An egocentric agent that reacts only
conversation, the same pause may simply mark a handover between two other to silence or aggregate speech may therefore interrupt a conversation that
participants. An egocentric conversational agent that reacts only to silence should continue without it. We study \emph{egocentric floor forecasting}:
or aggregate speech activity may therefore enter the floor while the continuously predicting the near-future speech participation of the
surrounding conversation should continue without it. We study this problem as first-person wearer from strictly causal audio--visual observations.
\emph{egocentric multiparty floor-entry forecasting}: continuously predicting
the near-future speech participation of the first-person participant from
causal audio--visual observations.
Voice Activity Projection (VAP) provides a predictive formulation of Voice Activity Projection (VAP) introduced continuous prediction of future
turn-taking by continuously projecting the joint future voice activity of voice activity for turn-taking \cite{ekstedt2022vap}. EgoSpeak brought this
conversational participants, from which turn shifts, holds, and backchannels formulation to causal, untrimmed first-person streams, using future camera-
can be inferred \cite{ekstedt2022vap}. EgoSpeak extends predictive speech wearer activity as an imitation target \cite{kim2025egospeak}. Complementary
initiation to the first-person setting, using causal, untrimmed egocentric work adds visual gaze, head pose, and facial behavior to predictive turn-taking
audio--visual observations and the future speaking behavior of the camera \cite{russell2025mmvap}, while MuVAP combines global acoustic dynamics with
wearer as supervision \cite{kim2025egospeak}. This behavior is a useful visible face tracks to forecast visible floor holders in multiparty scenes
imitation target for the perceptual decision faced by an egocentric agent: \cite{qi2026muvap}. Our target has a different role structure: the participant
given everything observed so far, is the first-person participant likely to whose future action is predicted is the one participant absent from the
enter the floor next? EgoSpeak establishes that this prediction can be learned visible set. The visible people are contextual evidence for the unseen
from real-world egocentric streams, but leaves open how recent jointly wearer's decision, not candidate output identities.
pretrained audio--visual Omni models should be used for such fine-grained
causal forecasting.
In parallel, recent work has expanded predictive turn-taking along multimodal This role asymmetry exposes an adaptation problem for general-purpose Omni
and multiparty dimensions. MM-VAP incorporates facial expression, head pose, models. Omni pretraining favors globally integrated representations useful for
and gaze into predictive turn-taking for two-party visual interaction multimodal understanding and generation. Egocentric floor control instead
\cite{russell2025mmvap}. A recent social-robot MM-VAP framework adapts depends on timely prosodic and visible-person activity at sub-second scale,
voice-activity-related audio--visual encoders for anticipating human--human including whether other participants continue to occupy the floor. Increasing
turn dynamics in observer or mediation settings \cite{cano2026mmvap}. MuVAP model capacity or applying a generic parameter-efficient update does not
further extends VAP to multiparty interaction using a monaural mixed-audio explicitly preserve or route this evidence to a dense wearer-relative
stream and a single camera, grounding acoustic dynamics in visible face tracks prediction head. The open question is therefore not merely whether an Omni
and predicting current and future visible floor holders \cite{qi2026muvap}. backbone can be applied to EgoSpeak, but what task structure is still required
These formulations are complementary to our egocentric setting. In MuVAP, the to turn general audiovisual understanding into reliable floor control.
candidate future speakers belong to the set visible to the camera; in
mediation-oriented MM-VAP, the system observes turn dynamics between human
participants. Here, by contrast, the participant whose future floor entry must
be predicted is located \emph{behind the camera} and is therefore absent from
the visible participant set.
This leaves an underexplored intersection: continuous causal prediction of an Our candidate decomposition separates long-context understanding from
egocentric participant's own future floor participation in a changing short-window floor evidence. A
multiparty scene using a general-purpose audio--visual Omni model. The defining causal global Omni state represents the broader scene and conversation. In
difference is its viewpoint and prediction target: the model must project the parallel, a lightweight local path encodes recent mixed audio and a changing,
activity of the unseen first-person participant from the causal stream used to anonymous set of visible-person observations. A residual adapter makes this
observe everyone else. We investigate how pretrained Omni representations can local evidence accessible to the common ten-horizon activity head. The primary
be adapted under the EgoSpeak wearer-relative forecasting objective. Current variant deliberately avoids cross-time identity: the current development
Qwen2.5-Omni results motivate this study, but do not establish a limitation oracle supports timely aggregate visible-other activity but not person
shared by Omni models in general. continuity. Track-aware structure is considered only if it defeats no-ID,
identity-corrupted, and capacity-matched controls.
Our working hypothesis separates broad context from timely floor evidence. A The final paper will claim the following three contributions only if the
causal global Omni state can represent the scene and conversational history, corresponding TODO gates pass:
while a lightweight local pathway preserves recent mixed audio and a changing \begin{itemize}
set of visible-person observations. The latter may be aggregated per frame or \item We identify and test a temporal--role mismatch between a global Omni
organized into anonymous tracks; track continuity becomes a contribution only readout and causal egocentric floor forecasting for an invisible wearer.
if it outperforms aggregate, no-ID, shuffled, and capacity-matched controls. An \item We evaluate a wearer-relative causal local floor adapter that combines
analysis-only oracle screen on held-out EasyCom Session~12 supports the weaker long-context Omni information with short-window audio and anonymous
hypothesis: perfectly timed aggregate visible-other activity improves a visible-person evidence.
matched global-plus-audio model by 6.97 points at 0.2~s and 2.91 points in \item We evaluate whether the adapter reduces false wearer entries during
ten-horizon mean mAP, whereas person-resolved histories do not outperform an multi-other exchanges at matched valid-onset recall, using causal alignment
identity-shuffled control. Thus the current evidence motivates timely local interventions and competing-risk-censored entry/yield analysis.
floor information, not speaker identity or addressee understanding. \end{itemize}
We retain EgoSpeak-compatible dense future-activity prediction for
comparability and add a behavior-aligned event analysis. In particular,
per-frame average precision can hide the failure that motivates this work:
entering during an exchange among other participants. We therefore measure
false wearer entries during multi-other opportunities at matched wearer-onset
recall, together with response latency and wearer-yield behavior. These events
are observable timing-derived interaction proxies, not semantic EOT, INT, or
addressee labels; the final protocol uses competing-risk censoring so an event
cannot claim a later action across an incompatible floor change. This follows
the broader event-oriented emphasis on recall, false alarms, and latency
\cite{jiang2026turnbench}.
Our contributions are: (1) a controlled study of frozen and
parameter-efficient Omni adaptation for EgoSpeak-style causal forecasting;
(2) an oracle information audit that distinguishes timely aggregate local
floor evidence from person continuity; (3) a global--local adaptation selected
through causal, alignment, and capacity-matched controls; and (4) an event
protocol that tests whether dense gains reduce false first-person floor entry
without suppressing valid entry or yield. EasyCom supplies synchronized
egocentric audio--video, per-person tracks, and voice activity
\cite{donley2021easycom}; target-of-speech annotations, when used, serve only
as analysis strata and never as model input or required output.
\begin{figure}[t] \begin{figure}[t]
\centering \centering
\fbox{\parbox[c][1.05in][c]{0.94\columnwidth}{\centering \fbox{\parbox[c][1.00in][c]{0.94\columnwidth}{\centering
\todo{After model selection, draw the final system figure. Show the shared \todo{Draw only the selected method: causal AV to the long-context Omni path;
interface: raw causal AV to a global Omni readout; a parallel recent-audio and the short-window mixed-audio and anonymous person-set path; residual fusion;
visible-person path; aggregate/no-ID/track-organized variants; residual fusion; the common $10\times3$ head; and the wearer-silent/active event interpretation.
and the common $10\times3$ head. Draw the transcript-derived oracle only as a Mark physical cutoffs, trainable modules, person masks, and NULL/OFFSCREEN.
dashed, analysis-only upper-bound path. Mark causal cutoffs, provided versus Do not draw oracle labels, rejected track variants, or multiple backbones.}}}
online tracks, trainable modules, NULL/OFFSCREEN handling, and resets.}}} \caption{Long-context Omni understanding and short-window local floor evidence
\caption{Global context and causal local evidence for egocentric floor action. for wearer-relative future-activity prediction.}
Track organization is a controlled candidate, not a presupposed source of
speaker identity.}
\label{fig:method} \label{fig:method}
\end{figure} \end{figure}
\section{Methods} \section{Method}
\label{sec:method} \label{sec:method}
\subsection{Wearer-relative activity projection} \subsection{Causal wearer-relative projection}
At decision time $t$, the model observes only the causal stream At decision time $t$, the model observes only the stream
$\mathcal{X}_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s. $\mathcal X_{\leq t}$ and predicts $K=10$ future bins at $\Delta=0.2$~s. For
For bin $k$, the target bin $k$, the multilabel target
$\vect{y}_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes $\vect y_{t,k}=[y^{\mathrm{bg}},y^{\mathrm{w}},y^{\mathrm{o}}]$ denotes
background, camera-wearer activity, and activity by one or more other people. background, camera-wearer activity, and activity by one or more other people.
Wearer--other overlap is allowed; hence this is not a three-way softmax. The Wearer--other overlap is allowed. The model outputs
output is
\begin{equation} \begin{equation}
\hat{\vect{Y}}_t=f_\theta(\mathcal{X}_{\leq t})\in[0,1]^{K\times3}, \hat{\vect Y}_t=f_\theta(\mathcal X_{\leq t})
\qquad q_t=\hat y^{\mathrm{w}}_{t,1}, \in[0,1]^{K\times3}.
\label{eq:task} \label{eq:task}
\end{equation} \end{equation}
where $q_t$ is the immediate wearer-entry score used by the event protocol. The same wearer-activity forecast has two controller interpretations. In a
It is interpreted as an \emph{entry} score only in wearer-silent opportunity wearer-silent opportunity, $\hat y^{\mathrm w}_{t,1}$ is an entry score. While
windows; elsewhere it remains a future-activity probability. This retains the the wearer is active, $1-\hat y^{\mathrm w}_{t,1}$ is a near-term yield score.
EgoSpeak target and evaluation while exposing a deployable 5-Hz decision The current self-state is used to define these observable event strata, not as
signal. an unreported oracle model input. \todo{If the final adapter explicitly uses
self-state, define its strictly causal and deployment-available source and add
a matched no-state ablation; otherwise keep self-state evaluation-only.}
\subsection{Omni backbones and representation protocols} \subsection{Global Omni baseline}
Let $m$ index an omni backbone and let $\vect{g}^{(m)}_t$ be a Let $\vect g_t\in\mathbb R^{d_g}$ be the causal global readout of
backbone-specific causal global readout at the same physical cutoff $t$. Qwen2.5-Omni at the physical cutoff $t$ \cite{xu2025qwenomni}. The frozen
Qwen2.5-Omni \cite{xu2025qwenomni} is the currently implemented family; baseline trains only a dimension-matched prediction head. A matched PEFT/LoRA
Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o~4.5 baseline updates a pre-registered subset of the Omni model under the same
\cite{cui2026minicpmo45} form the planned initial comparison. These models have examples, context, optimizer budget, head, and evaluator. Its purpose is to
different native streaming APIs, temporal tokenization, hidden dimensions, and test whether generic parameter adaptation is sufficient; one negative recipe
adaptation interfaces. We therefore emphasize within-backbone deltas rather is not interpreted as a general limitation of LoRA or all Omni models.
than treating raw cross-model scores as a pure architecture ranking.
Every supported backbone is evaluated under three representation protocols: \todo{Freeze the exact Qwen2.5-Omni-3B revision, processor, input rates,
(i) a frozen global readout with a dimension-matched prediction head; streaming/cache semantics, readout layer, context, dtype, LoRA targets/rank,
(ii) a pre-registered parameter-efficient adaptation budget; and (iii) the trainable parameter budget, and cutoff audit. Add Qwen2.5-Omni-7B only as a
same global readout augmented by causal local evidence. Input cutoffs, EgoSpeak capacity-scaling control. Do not add another backbone family unless it is
targets, context duration, training examples, head capacity, and evaluation needed to support and experimentally earns a cross-family claim.}
are held fixed whenever the backbone permits it. A failure of one LoRA recipe
is reported as that recipe's result, not as proof that the backbone cannot
adapt.
\todo{Implement and audit one adapter for each planned backbone. For every \subsection{Causal local floor adapter}
model, document model revision, processor, native audio/video rates, exact
hidden-state readout, causal/streaming semantics, context limit, dtype,
trainable LoRA targets/rank, parameter count, peak memory, and P50/P95 latency.
Verify no future frames or samples enter $\vect g^{(m)}_t$. If a model cannot
expose a comparable causal state, report it as unsupported rather than inventing
a non-equivalent readout. Add future backbones only before the experiment
protocol is locked.}
\subsection{Global context and causal local evidence} The candidate local path operates on the most recent $L_\ell$ seconds. At local instant
$\tau\leq t$, a causal audio encoder produces $\vect a_\tau$ from mixed audio.
The global state $\vect g^{(m)}_t$ is responsible for broad scene and For the anonymous visible-person set
conversational context, but is not assumed to retain every localized cue needed $\mathcal P_\tau=\{(\vect x_{\tau,i},\vect b_{\tau,i})\}_{i=1}^{N_\tau}$,
for floor control. From the most a shared visual encoder and masked set operator produce
recent $L_a$ seconds, a causal local audio encoder produces $\vect a_t$. At
time $t$, provided boxes during method isolation---and an online detector and
tracker in the deployable condition---produce a variable-size anonymous set
$\mathcal P_t=\{(\vect v_{t,i},\vect b_{t,i},\vect r_{t,i})\}_{i=1}^{N_t}$,
where $\vect v_{t,i}$ is a crop feature, $\vect b_{t,i}$ is box geometry, and
$\vect r_{t,i}$ contains only causal reliability variables such as visibility,
track age, and time since observation. A shared causal track encoder updates
\begin{equation} \begin{equation}
\vect h_{t,i}=F_{\mathrm{trk}}\!\left(\vect h^-_{t,i}, \vect v_\tau=\mathrm{SetPool}
W_v\vect v_{t,i}+W_b\phi(\vect b_{t,i})+W_r\vect r_{t,i}\right). \left(\{E_v(\vect x_{\tau,i},\vect b_{\tau,i})}_{i=1}^{N_\tau}
\label{eq:track} \cup\{\vect v_\varnothing\}\right),
\label{eq:set}
\end{equation} \end{equation}
All tracks share $F_{\mathrm{trk}}$; the temporary ID only retrieves where $\vect v_\varnothing$ handles an empty or offscreen visible set. The
$\vect h^-_{t,i}$ and is never embedded. The global-audio query then reads the operator is permutation invariant and uses no participant-ID embedding or
current person set by masked cross-attention, cross-time identity. A causal short-window encoder summarizes local dynamics,
\begin{align} \begin{equation}
\vect u_t &= \mathrm{MHA}\!\left( \vect \ell_t=F_\ell
W_q[\vect g^{(m)}_t;\vect a_t], \left(\{[\vect a_\tau;\vect v_\tau]\}_{\tau\in(t-L_\ell,t]}\right).
[\vect h_{\varnothing};\vect h_{t,1};\ldots;\vect h_{t,N_t}] \label{eq:local}
\right), \\ \end{equation}
\vect z^{(m)}_t &= \vect g^{(m)}_t+W_o\vect u_t, The adapter preserves the global initialization through a zero-initialized
residual output,
\begin{equation}
\vect z_t=\vect g_t+W_o\,
\sigma\!\left(W_g\vect g_t+W_\ell\vect\ell_t\right),
\qquad W_o\vert_{\mathrm{init}}=0, \qquad W_o\vert_{\mathrm{init}}=0,
\label{eq:adapter} \label{eq:fusion}
\end{align} \end{equation}
where $\vect h_{\varnothing}$ is a learned NULL/OFFSCREEN token. Padding is and the common head maps $\vect z_t$ to Eq.~\eqref{eq:task}. This implements an
masked, so the same network handles $N_t=0$ and changing group size. For a evidence-access hypothesis rather than adding an unconstrained second model:
bounded implementation, $N_t>N_{\max}$ is resolved by a deterministic ranking the global path retains broad context, while the local path preserves timely
using detector confidence and track age available at $t$, and truncation is floor evidence.
reported. The operator is permutation equivariant
\cite{lee2019settransformer}; temporary tracker IDs provide continuity, not
identity semantics.
Equation~\eqref{eq:track} is a candidate organization of local evidence, not a \todo{Freeze $L_\ell$, local audio/video rates, crop source and resolution,
claim that recurrent identity is necessary. Simpler alternatives replace each set pooling, causal temporal encoder, masks, reset rules, parameter count, and
$\vect h_{t,i}$ with a current-frame observation or pool the set into one latency. If the local path runs faster than the 5-Hz global/output grid, compare
aggregate vector. This distinction is important because the current oracle it with a same-parameter 5-Hz variant before using the term ``dual-rate.'' Use
screen favors aggregate activity and does not separate correct from shuffled provided causal boxes only as a clearly labeled method-isolation condition;
identity histories. online detection/tracking is not a paper contribution unless actually run.}
This decomposition is informed by MuVAP: its global audio representation \subsection{Training}
queries same-frame causal ASD track representations, followed by temporal
modeling and gated global-to-speaker fusion \cite{qi2026muvap}. We retain the
useful separation between global floor dynamics and person-organized evidence,
but not its role-relative output: our fixed target is the invisible wearer's
entry/yield action, not the next visible speaker. The zero-initialized residual
output follows the conservative adapter principle used to inject visual
evidence without perturbing the base prediction at initialization
\cite{rouditchenko2024whisperflamingo}. Unlike a zero scalar gate, it permits
the residual output layer to receive a gradient immediately. Local evidence
is derived from the same causal raw media, not close microphones, transcripts,
speaker labels, global identity anchors, or future frames.
We evaluate candidates in increasing structural complexity: pooled recent The primary objective is class-balanced binary cross entropy over valid
audio/vision; a capacity-matched temporal branch; a per-frame person set with horizons and activity channels,
no cross-time identity; a shared track-wise temporal encoder; and person--audio
cross-attention as in Eqs.~\eqref{eq:track}--\eqref{eq:adapter}. Only a
candidate that improves the pre-registered event metric over simpler,
aggregate, identity-shuffled, and capacity-matched controls becomes the final
method. If person continuity does not separate from those controls, the final
model remains a global-plus-local aggregate floor-evidence model. A limited latent
bottleneck is tested only if large visible sets make full person attention a
runtime bottleneck; it is not assumed to improve accuracy.
\todo{Run the candidate ladder before fixing $H_\phi$. Match trainable
parameters and effective local history. Compare a shared local encoder across
backbones (cross-model control) with each backbone's native local encoder
(within-model control); do not mix these protocols in one claimed delta.
For track candidates, specify visibility/age masks, gap/source resets,
OFFSCREEN handling, and global-rename invariance. Describe only the selected
architecture as the final method; move rejected candidates to ablations.}
\subsection{Training objectives}
The common main loss is class-balanced binary cross entropy over all horizons
and activity channels. This dense EgoSpeak target is the only supervision
required by the task. If a person-aware candidate survives the activity-only
comparison, an optional auxiliary head predicts current active-speaker state:
\begin{equation} \begin{equation}
\mathcal{L}=\mathcal{L}_{\mathrm{act}}+ \mathcal L_{\mathrm{act}}=
\lambda_{\mathrm{asd}}\mathcal{L}_{\mathrm{asd}}. \sum_{t,k,c}m_{t,k}w_c\,
\mathrm{BCE}(\hat y^c_{t,k},y^c_{t,k}).
\label{eq:loss} \label{eq:loss}
\end{equation} \end{equation}
Participant ID, target-of-speech, transcript, close-microphone audio, and No transcript, target-of-speech label, close microphone, future frame, or
future frames are never model inputs. Target-of-speech may stratify qualitative per-person speech label enters the deployable model. \todo{Add an active-speaker
analysis but is not a predicted variable or a requirement for the primary or event auxiliary loss only if it improves the frozen primary protocol and
multi-other event definition. its label/input boundary is fully reported.}
\todo{First compare all representation/adaptation protocols with activity loss
only. Then ablate +ASD for the selected person-aware candidate. Add an
event-specific wearer onset/yield loss only as a separately reported extension.
Select all coefficients and early stopping on a training-side calibration
fold; do not inspect Sessions 1--3 during selection.}
\subsection{Track source and deployment contract}
The intended inference pipeline runs a face/head detector on each incoming
frame and associates detections with an online multi-object tracker using only
observations at or before $t$. Its temporary IDs route the shared recurrent
state in Eq.~\eqref{eq:track}; they do not denote names, fixed seats, or
cross-session identities. New tracks initialize from a learned state, short
misses retain a causally aged state, and expired tracks are removed. With no
visible face, the global/audio path and NULL/OFFSCREEN token remain valid. No
future-frame smoothing, full-video identity anchor, or cross-session
recognition is permitted. EasyCom state is continued only across genuinely
adjacent source segments and reset across redacted gaps; Ego4D state is local
to a clip.
During method isolation, dataset-provided boxes and anonymous IDs form a
separate \emph{provided-track} condition. This answers whether person
structure can help under accurate localization, but is not reported as online
detector performance. The deployable \emph{online-track} condition runs the
detector/tracker above and includes its errors and latency. This separation is
important because MuVAP used random full-video identity anchors to build its
dataset tracks; that preprocessing is useful for annotation but is not a
strictly online association algorithm \cite{qi2026muvap}.
\todo{Select and freeze the face/head detector, causal tracker, thresholds,
maximum age, and $N_{\max}$ on training-side data. Report detector recall,
provided/online-track coverage, mean/maximum people, truncations, ID switches,
fragments, occlusion gaps, source resets, and P50/P95 detector--tracker latency
for both datasets. Compare provided tracks, online tracks, and controlled box
jitter/dropout/fragmentation. Until the online row is run, limit deployment
claims to the intended pipeline and label all numbers ``provided-track.''}
\section{Experiments} \section{Experiments}
\label{sec:setup} \label{sec:experiments}
\textbf{Data and splits.} \textbf{Data and split.}
We use the EgoSpeak EasyCom protocol: Sessions 4--12 for model development and We follow the EgoSpeak EasyCom split: Sessions 4--12 for training-side model
Sessions 1--3 for final reporting. All architecture, checkpoint, threshold, development and Sessions 1--3 for final reporting \cite{donley2021easycom}.
and hyperparameter choices use training-side folds only. Inputs contain the Thresholds, windows, checkpoints, loss weights, and early stopping are selected
camera stream and shared array audio available to the wearer; close-talking without using the official test sessions. Already observed test results are
microphones, per-person voice labels, and target-of-speech labels are not model treated as exploratory internal evidence and are not included in the paper.
inputs. We also report aggregate activity forecasting on the EgoSpeak Ego4D \todo{Report duration, frame/event counts, class prevalence, visibility and
split \cite{grauman2022ego4d}. Ego4D Audio-Visual Diarization supplies box coverage, source discontinuities, and the frozen calibration fold.}
clip-local anonymous participant tracks; its per-person voice coverage on the
433-clip EgoSpeak split must be audited before reporting the same multi-other
event analysis.
\textbf{Oracle information audit.} \textbf{Baselines.}
Before selecting a learned local module, we measure what information could in The confirmatory comparison contains: an EgoSpeak reproduction; the frozen
principle complement the global state. Analysis-only sidecars expose Qwen2.5-Omni-3B global state $G$; matched LoRA/PEFT; $G$ plus recent local
transcript-derived speech activity no later than the current cutoff under six audio; a parameter- and history-matched temporal local branch; and the proposed
conditions: local audio, visibility only, one-second-delayed activity, timely audio--visible-person adapter. Qwen2.5-Omni-7B is a capacity-scaling control,
aggregate visible activity, per-person histories, and histories with their not a second paper storyline. \todo{Freeze all baseline implementations and
cross-time identity assignments shuffled. These labels never enter a ensure the EgoSpeak comparison uses identical targets, split, causal validity,
deployable model. The audit separates three hypotheses that ordinary feature and evaluation.}
ablation conflates: the value of seeing people, the value of knowing current
visible-other activity, and the incremental value of preserving who produced
that activity.
\textbf{Backbone comparison.} \textbf{Event protocol.}
The initial study covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Dense ten-horizon AP/mAP is retained for EgoSpeak comparability. The implemented
MiniCPM-o~4.5. For each backbone $m$, we first compare its frozen causal state competing-risk evaluator counts each eligible aggregate-other boundary once,
$G_m$, a parameter-efficiently adapted state $P_m$, and local-evidence ends a response window before the next other onset, and ends a yield window
augmentation $G_m+L$. The primary diagnostic is the within-backbone change; before the next other offset. Pending the remaining P0 real-data audit and
cross-backbone absolute scores are secondary because native tokenization, scenario freeze, the intended primary behavior metric is false wearer entries per
pretraining, streaming interfaces, and hidden-state semantics differ. Common minute of multi-other opportunity at a training-side selected wearer-onset
local encoders test whether adaptation transfers across global backbones, recall, with $R=.8$ as the headline operating point. The intended analysis also
whereas native local encoders are a separate, model-specific protocol. All reports onset latency and wearer-yield recall/latency, following the
rows share physical input cutoffs, targets, data splits, and evaluation. event-oriented emphasis on recall, false alarms, and latency
Unsupported causal readouts are disclosed rather than replaced by offline or \cite{jiang2026turnbench}. The protocol will censor a response window after an
future-conditioned features. other offset at the next other onset, and a yield window after an other onset
at the next other offset.
\todo{Implement the three backbone adapters and validate state timestamps with \todo{P0 blocker: run the tested competing-risk evaluator on real predictions,
an impulse/cutoff test. Pre-register one matched PEFT budget and one common manually audit sampled timelines, and freeze pairing before treating any event
local branch. Report native-vs-common local encoders in separate table panels. score as a paper result. Specify overlap,
Keep future model additions outside the confirmatory comparison once this minimum duration, timeouts, duplicate detections, source boundaries, offscreen
matrix is locked.} cases, and event counts. Until this is complete, call existing values
timing-derived proxies rather than final event results.}
\textbf{Event construction.} \textbf{Controls and statistics.}
We derive \emph{wearer onset} events directly from the wearer activity target. The main intervention compares correctly aligned local evidence with
A \emph{multi-other exchange} is a wearer-silent opportunity interval in which time-shifted evidence, removed vision, global-frame visual pooling, and the
at least two distinct non-wearer participant IDs contribute temporally ordered capacity-matched temporal branch. A same-rate control is required for a
speech activity. It is an observable interaction proxy, not a claim about the dual-rate claim. Track/no-ID/broken-continuity controls are added only if a
speakers' mental intent or addressee. Threshold $\tau_r$ is selected on the track-aware variant first wins on training-side data. We report three seeds,
development split to obtain wearer-onset recall mean$\pm$sample standard deviation, and paired cluster-bootstrap intervals over
$r\in\{0.7,0.8,0.9\}$ and is then frozen. The primary metric is false wearer sessions or conversation blocks. \todo{Freeze seeds, bootstrap unit/count,
entries per minute of multi-other opportunity at matched onset recall; the optimizer, epochs, hardware, throughput, P50/P95 latency, and trainable
headline operating point is $r=0.8$. We also report signed onset latency, parameters.}
wearer-yield recall/latency, and the original ten-horizon per-frame mAP.
\todo{Freeze the event pairing and competing-risk rules, including the
multi-other alternation window, minimum speech/silence, overlap, maximum onset
delay, duplicate detections, one detection per event, track visibility, source
boundaries, and group/offscreen cases. Publish train/test onset counts,
multi-other events/minutes, number of distinct visible speakers, and exclusion
counts. Manually audit at least 100 events. Treat all previously viewed
Sessions 1--3 scores as exploratory; lock the protocol and model before the
confirmatory three-seed run.}
\textbf{Baselines and controls.}
For each backbone we compare: frozen global state $G_m$; PEFT $P_m$; $G_m$
plus recent local audio; and $G_m$ plus pooled local audio--visual evidence.
This compact matrix tests the representation hypothesis before investing in a
complex fusion module. The first learned target after the oracle audit is a
causal estimator of timely aggregate visible-person floor evidence. Only if
person-organized features survive three seeds, visual time/person corruptions,
and the event gate do we compare a parameter-matched temporal branch without
person continuity, a separately trained per-frame/no-ID set branch, and a
track-consistent masked cross-attention branch. Provided-track and online-track
inputs are reported separately. On a frozen track-aware checkpoint, a clip-consistent
global renaming must leave outputs unchanged, whereas \emph{broken
continuity} periodically remaps IDs while preserving every crop, timestamp,
audio sample, and target. Time-shifted and removed vision provide additional
alignment controls.
\todo{Define the broken-continuity corruption before evaluation: remap only at
pre-registered 1-s boundaries or report a fixed ID-switch-rate curve. Do not
call a clip-consistent slot permutation ``broken ID.'' Verify unchanged crops,
frame counts, audio, labels, and visibility masks. Train the no-ID/reset
baseline from scratch with matched parameters; evaluate all test-time
counterfactuals using one frozen checkpoint and one frozen threshold.}
\textbf{Implementation and statistics.}
All prediction heads run at 5~Hz and use an 8-s causal observation window;
model-specific deviations required by a native streaming interface are stated
explicitly. In the frozen protocol the omni backbone does not update. In PEFT
and local protocols, trainable parameters and optimizer exposure are reported
separately. Every temporal state resets at non-contiguous source boundaries.
We report three seeds and mean$\pm$standard deviation. Confidence intervals for
event metrics use paired cluster bootstrap over sessions/conversation blocks
rather than treating adjacent frames as independent. Runtime includes P50/P95
decision latency, throughput, peak memory, and trainable parameter count.
\todo{Fill optimizer, learning rate, batch size, epochs, feature dimensions,
$L_a$, crop rate/resolution, number of heads/layers, $N_{\max}$, parameter
count, tracker timeout/ranking, hardware, and end-to-end real-time latency
after the architecture is frozen.}
\section{Results and Analysis} \section{Results and Analysis}
\label{sec:results} \label{sec:results}
\begin{table}[t] \begin{table}[t]
\caption{Causal oracle-information screen on EasyCom development Session~12 \caption{Confirmatory EasyCom results. Lower multi-other false entries are
(seed 0, four epochs). Oracle speech activity is analysis-only. Values are better; all event rows use a training-side selected operating point.}
percentages under the causal-valid protocol.}
\label{tab:oracle}
\centering
\small
\begin{tabular}{lcc}
\hline
Input condition & 0.2-s mAP & Mean mAP\\
\hline
Global + local audio & 72.58 & 58.77\\
Visibility only & 71.95 & 58.86\\
Activity delayed by 1 s & 72.02 & 58.90\\
Aggregate visible activity & \textbf{79.55} & \textbf{61.68}\\
Per-person activity histories & 78.80 & 61.11\\
Identity-shuffled histories & 78.86 & 61.19\\
\hline
\end{tabular}
\end{table}
Table~\ref{tab:oracle} establishes an information opportunity, not a deployable
result. Timely aggregate activity among visible non-wearers provides a large
gain, while visibility alone and stale activity do not. However, preserving
person-resolved history gives no advantage over destroying its cross-time
identity assignment, and both trail the aggregate representation. The learned
method must therefore first recover useful local floor evidence; person-track
continuity is retained only if subsequent no-ID and broken-track controls
separate.
\begin{table}[t]
\caption{Exploratory Qwen2.5-Omni-3B local-branch evidence on EasyCom test
Sessions 1--3. Values are percentages from one seed, not a cross-backbone or
confirmatory result.}
\label{tab:preliminary}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc}
\hline
Model & 0.2-s mAP & 0.2-s wearer AP & Mean mAP\\
\hline
Qwen-3B $G$+local A & 81.72 & 85.07 & 65.11\\
Qwen-3B $G$+local AV-pool & 82.03 & 85.09 & 65.40\\
\hline
$\Delta$ visual & +0.30 & +0.02 & +0.29\\
\hline
\end{tabular}}
\end{table}
Table~\ref{tab:preliminary} shows why aggregate mAP alone is insufficient.
Pooled vision gives a small gain, while immediate wearer AP is nearly
unchanged. This is compatible with either useful generic local context or
regularization; it is not evidence of person grounding. Only matched-recall
event evaluation and visual-alignment controls can distinguish these cases.
A separate Qwen-3B dense-cache diagnostic gave 58.16 mean mAP for its frozen
readout and 56.20 for one LoRA recipe. Because that diagnostic used a different
feature protocol, it is not combined numerically with
Table~\ref{tab:preliminary}; it only shows why a fair, protocol-matched PEFT
study remains necessary.
\begin{table}[t]
\caption{Planned cross-backbone EasyCom event evaluation. Lower
$\mathrm{FP}_{\mathrm{mo}}$ is better; all rows use the same
development-selected operating protocol. ``Local'' denotes the train-side
selected common local-evidence candidate.}
\label{tab:main} \label{tab:main}
\centering \centering
\small \small
\resizebox{\columnwidth}{!}{% \resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc} \begin{tabular}{lccccc}
\hline \hline
Backbone/protocol & $\mathrm{FP}_{\mathrm{mo}}$/min@$R=.8$ & Onset R & Method & 0.2-s mAP & Mean mAP & FP/min@$R=.8$ & Onset lat. & Yield R\\
Onset latency & Yield R & Mean mAP\\
\hline \hline
Qwen-3B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ EgoSpeak reproduction & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-3B + local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ $G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
Qwen-7B PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ $G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o frozen & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ $G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
MiniCPM-o PEFT/local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\ Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--}\\
\hline \hline
\end{tabular}} \end{tabular}}
\end{table} \end{table}
\todo{Report Table~\ref{tab:main} as mean$\pm$standard deviation over three \textbf{Information opportunity.}
seeds, plus 95\% paired cluster-bootstrap confidence intervals for frozen vs On development Session~12, an analysis-only seed-0 oracle supplied perfectly
PEFT and frozen vs local within each backbone. A general omni-representation timed visible-other speech activity through an explicit non-deployable path.
claim requires a consistent pattern across at least two backbone families; an Aggregate visible activity improved 0.2-s mAP by 6.97 points and ten-horizon
effect confined to Qwen must be stated as Qwen-specific. The behavior claim is mean mAP by 2.91 points over matched global-plus-local-audio. Visibility-only
supported only if multi-other FP/min decreases at matched wearer-onset recall and one-second-delayed activity remained near audio, while per-person histories
without worse onset latency or wearer-yield recall. If the selected local did not outperform identity-shuffled histories. This motivates timely local
candidate uses tracks, also report its paired intervals against no-ID and floor evidence, not participant identity. \todo{After seeds 1--2, report the
temporal-capacity controls.} interval or remove the exact oracle numbers from the four-page paper.}
\textbf{Main result.}
Table~\ref{tab:main} should establish one result: \todo{the selected local
adapter reduces multi-other false entry at matched valid-onset recall, retains
onset latency/yield behavior, improves dense forecasting, and recovers a
meaningful fraction of the oracle opportunity}. The 7B row should determine
whether \todo{capacity alone closes the gap or the adaptation structure remains
useful at larger scale}. No conclusion is drawn across all Omni families from
Qwen-only evidence.
\begin{table}[t] \begin{table}[t]
\caption{Conditional track-continuity counterfactual on the best track-aware \caption{Mechanism controls for the selected Omni-3B local adapter.}
model. This table is retained only if train-side selection chooses such a \label{tab:ablation}
candidate; all rows reuse one frozen checkpoint.}
\label{tab:counterfactual}
\centering \centering
\small \small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccc} \begin{tabular}{lccc}
\hline \hline
Track/visual evidence & $\mathrm{FP}_{\mathrm{mo}}$/min & Onset recall & 0.2-s mAP\\ Local condition & Mean mAP & FP/min@$R=.8$ & Yield R\\
\hline \hline
Correct tracks & \todo{--} & \todo{--} & \todo{--}\\ Correct causal local AV & \todo{--} & \todo{--} & \todo{--}\\
Global ID rename & \todo{--} & \todo{--} & \todo{--}\\ Remove local vision & \todo{--} & \todo{--} & \todo{--}\\
Broken continuity & \todo{--} & \todo{--} & \todo{--}\\ Time-shift local vision & \todo{--} & \todo{--} & \todo{--}\\
Time shifted & \todo{--} & \todo{--} & \todo{--}\\ Global/full-frame visual pool & \todo{--} & \todo{--} & \todo{--}\\
Removed & \todo{--} & \todo{--} & \todo{--}\\ Same-rate/capacity control & \todo{--} & \todo{--} & \todo{--}\\
\hline \hline
\end{tabular}} \end{tabular}
\end{table} \end{table}
If a track-aware candidate is selected, Table~\ref{tab:counterfactual} is the \textbf{Mechanism and limitations.}
continuity test. Global renaming must be Table~\ref{tab:ablation} should test whether gains require correctly timed local
numerically invariant because IDs carry no semantics. If the model exploits visual evidence rather than extra parameters, regularization, or generic scene
person history, correct tracks should outperform broken continuity even though context. \todo{Report paired intervals and the supported interpretation. If
both contain exactly the same per-frame people. If broken, time-shifted, and correct and shifted/removed vision do not separate, drop the audiovisual
removed variants perform similarly to correct tracks, the defensible evidence-routing claim. If dense mAP improves but event behavior does not, do
conclusion is additional capacity or generic visual context, not useful not claim improved floor control. If legitimate onset/yield degrades, stop the
person-continuous reasoning. adapter rather than describing lower false entry as success.}
\todo{Add one compact error analysis split by model family, track visibility, \todo{Add one compact, pre-selected qualitative timeline only if space permits:
number of visible people, overlap, track age/fragmentation, and one multi-other false-entry improvement paired with one preserved real wearer
backchannel/interruption proxy. Include event counts with every rate. Move onset or one failure. Examples illustrate aggregate rates and never establish
rejected PEFT recipes and fusion candidates to supplementary material unless the claim by themselves.}
they directly explain the selected method.}
\textbf{Qualitative evidence.}
For a multi-other exchange, visualize annotation-derived per-person activity
as an analysis reference, provided track IDs when applicable, wearer ground
truth, the frozen-global and selected-local wearer-entry scores, and the single
frozen threshold on one physical time axis. If the selected candidate is
track-aware, also show its broken-continuity score. A useful
example shows the frozen baseline rising at an other-speaker boundary while
the selected adapter remains below threshold; a companion positive example
must show that a real wearer onset is not suppressed.
\todo{Do not hand-pick only successful clips. Before viewing the final test
plots, freeze a deterministic selection rule and show: largest paired
improvement, near-median improvement, one failure, and one correctly retained
wearer onset. Include full local timelines and source frame times, not cropped
screenshots without context. State that examples illustrate the aggregate
event result and cannot establish it alone.}
\section{Conclusion} \section{Conclusion}
\label{sec:conclusion} \label{sec:conclusion}
We study when an egocentric agent should enter or yield a multiparty floor, not This study asks whether a general-purpose Omni representation can be adapted
merely whether someone will speak. The working decomposition is global Omni for causal floor forecasting of an invisible egocentric wearer. The candidate
context plus causal local audiovisual floor evidence. The current method separates long-context Omni understanding from short-window local floor
Qwen2.5-Omni-3B oracle audit supports the second term: timely aggregate evidence without assuming persistent speaker identity. \todo{State the final
visible-other activity is informative beyond the global state and recent learned-method improvement, matched-recall false-entry result, onset/yield
mixed audio. It does not support the stronger claim that participant identity non-degradation, causal-control evidence, and main limitation. If no learned
or track continuity adds value, because per-person histories do not beat an variant passes these tests, reframe the submission as an information audit or
identity-shuffled control. Dense activity forecasting supplies a comparable do not make the method claim.}
target, while matched-recall event evaluation must determine whether a learned
local estimator actually prevents intrusions during multi-other exchanges
without suppressing valid wearer entry or yield. \todo{After the controlled
study, state which learned local variant was selected, how much of the oracle
gap it recovered, whether the pattern generalized beyond Qwen, the
quantitative false-entry and onset/yield results, and the main deployment
limitation. If no learned variant passes these tests, conclude with the
information audit rather than claiming a track-based solution.}
\bibliographystyle{IEEEbib} \bibliographystyle{IEEEbib}
\bibliography{references} \bibliography{references}

353
reports/omni_turn_research_direction_2026-08-26.md Normal file
View File

@@ -0,0 +1,353 @@
# Frozen Omni turn anticipation: evidence, trends, and next direction
Date: **2026-08-26**
This memo combines the current EasyCom experiments, a read-only audit of the
local EasyCom/Ego4D/EgoSpeak copies under `/Volumes/MX500B/Code`, and recent
work on streaming multimodal interaction. Nothing on the NTFS volume was
modified.
## Executive decision
The most defensible ICASSP direction is **not** to continue scaling the
Qwen2.5-Omni backbone or to fine-tune more of it. The evidence instead supports:
> **A frozen causal Omni backbone with a fine-grained, transition-aware
> multimodal decision readout for egocentric future-speaker anticipation.**
The current 4-epoch last-four-layer concat-MLP reaches `60.87144` mean mAP,
slightly above EgoSpeak's reported EasyCom GRU A+V result (`60.6`), while using
the Omni Transformer itself for temporal modeling and no additional RNN. It is
not yet a clean paper result: it is single-seed and the epoch budget was chosen
after looking at Sessions 1--3. Its main scientific value is the diagnosis:
1. useful turn information is distributed across the upper Omni layers;
2. a final-layer query alone under-exposes that information;
3. unconstrained high-capacity readout training overfits the nine training
conversations very quickly;
4. larger 7B features and Thinker LoRA do not solve the problem.
The next contribution should therefore target **how to expose and supervise
the pretrained representation**, not how to make the backbone larger.
## Current result ladder
Selected EasyCom full-timeline results on Sessions 1--3:
| System | 0.2s mAP | Mean mAP | Interpretation |
|---|---:|---:|---|
| EgoSpeak Transformer A+V | 73.8 | 58.7 | best reported EgoSpeak 0.2s result |
| EgoSpeak GRU A+V | 73.5 | 60.6 | best reported EgoSpeak mean result |
| Frozen Omni-3B, final-layer query | 71.99768 | 58.51937 | initial 8s reference |
| Frozen Omni-3B, last-4 scalar mixture | **73.45085** | 60.53423 | recovers the short-horizon deficit |
| Frozen Omni-3B, last-4 concat-MLP, 4 epochs | 73.42020 | **60.87144** | exploratory project best |
| Same concat-MLP, 12 epochs | 72.44507 | 57.98809 | clear overfitting |
| Same concat-MLP, 50 epochs | 66.54645 | 53.03515 | severe overfitting |
| Frozen Omni-7B, reported run | 69.80133 | 57.67367 | larger backbone is worse |
The concat-MLP result is archived in
[`results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/`](../results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/README.md).
Compared with the scalar mixture, concat-MLP gains only `+0.33721` mean mAP
and loses `0.03065` at 0.2 seconds. The added capacity mainly helps around
1.2--1.6 seconds. This is useful but too small to justify a 16.8M-parameter
fusion module without a cleaner validation protocol and stronger regularization.
## What the local data actually imply
### EasyCom: many frames, few independent conversations
The EgoSpeak EasyCom targets contain:
| Split | Sessions | 5Hz steps | Duration | Background | Wearer | Others |
|---|---:|---:|---:|---:|---:|---:|
| Train (4--12) | 9 | 72,405 | 4.0225h | 22.57% | 24.93% | 60.50% |
| Test (1--3) | 3 | 23,048 | 1.2804h | 22.91% | 26.74% | 58.71% |
| Total | 12 | 95,453 | 5.3029h | -- | -- | -- |
Wearer and other-speaker labels overlap in about `8.08%` of all steps, so this
is intrinsically a multi-label conversational-state problem even though the
EgoSpeak implementation applies a three-way softmax objective.
This exposes an immediate objective mismatch. The observed target rows contain
exactly four conversational states:
1. `[1, 0, 0]`: silence/background;
2. `[0, 1, 0]`: wearer only;
3. `[0, 0, 1]`: other people only;
4. `[0, 1, 1]`: wearer and others overlap.
A three-way softmax cannot assign high probability to both active-speaker
classes. In the overlap state, the LSTR loss used by the current project sums
`-log p_wearer - log p_others`, forcing the two positive classes to compete.
This is especially relevant because wearer AP is the class that collapses most
under overfitting.
A structured four-state head is a cleaner formulation: predict a softmax over
`silence / wearer-only / others-only / overlap`, then expose evaluator scores
as `p_background=p_silence`, `p_wearer=p_wearer-only+p_overlap`, and
`p_others=p_others-only+p_overlap`. This retains a normalized state model while
matching the multi-positive EgoSpeak labels exactly.
The important limitation is the effective sample size. Adjacent 200ms anchors
are strongly correlated, and there are only nine independent training
sessions. Session-level variation is large:
- wearer prevalence ranges from `10.37%` to `37.70%`;
- other-speaker prevalence ranges from `47.71%` to `77.17%`;
- overlap ranges from `1.42%` to `20.35%`;
- label-change frequency ranges from about `26` to `35` events per minute.
This explains why a 16.8M concat projection can reduce training loss from
`0.806973` to `0.300277` while test mean mAP falls by `7.83629` points. The head
can learn conversation/session priors rather than a transferable switching
rule.
The data are nevertheless rich in transitions. Depending on the session,
roughly `54%`--`69%` of anchors have at least one label change in their next
two seconds. The next method should not simply oversample all transition-near
frames; it should explicitly model **persistence versus switching**.
### Existing annotations support stronger auxiliary supervision
The raw EasyCom copy contains 323 transcript JSON files with:
- 6,981 utterances and about 52,171 transcribed words;
- speaker ID and frame-accurate start/end timestamps;
- Target-of-Speech labels on 6,963 utterances;
- 6DoF tracked participant pose at the original 20 FPS;
- participant-linked head and face boxes.
These annotations permit auxiliary targets without new manual labeling:
- current conversational state;
- keep/switch state between adjacent 200ms steps;
- time-to-next-change bins;
- utterance completion/onset proximity;
- addressee type (group versus individual);
- head-orientation change and target-of-speech prediction.
Transcript text should be used as **training-only privileged information** or a
distillation target if the final online model must consume raw audio/video only.
That would demonstrate a semantic advantage without giving Omni unavailable
future text at test time.
### Ego4D is the more convincing generalization test
The split used by EgoSpeak's reported experiment contains 346 train and 87
test clips, almost all 5 minutes long:
| Split | Clips | 5Hz steps | Duration | Background | Wearer | Others | Overlap |
|---|---:|---:|---:|---:|---:|---:|---:|
| Train | 346 | 518,881 | 28.83h | 43.31% | 23.24% | 40.53% | 5.58% |
| Test | 87 | 130,472 | 7.25h | 42.41% | 26.30% | 38.81% | 5.63% |
This provides far more independent clips and substantially less split shift
than EasyCom at the session level. A method that improves only EasyCom may be
learning its fixed around-table conversational geometry. A method that also
improves Ego4D supports the stronger claim that the readout exposes general
Omni interaction cues.
## Research trend and what transfers to this project
### 1. Explicit decision heads are replacing implicit generation triggers
[ROMA](https://arxiv.org/abs/2601.10323) processes aligned streaming
audio-video units, preserves 40ms audio temporal position IDs, and places a
lightweight speak head parallel to the LM head. It forms the head input as a
learned mixture of the last four layers. This strongly validates the project's
last-four-layer query experiment, while also suggesting that the low-capacity
scalar mixture is a more natural primary model than concat-MLP.
[Proact-VL](https://arxiv.org/abs/2603.03447) inserts a dedicated FLAG token,
reads it with a small gated MLP, and separates the decision of when to respond
from response generation. [JoyAI-VL-Interaction](https://arxiv.org/abs/2606.14777)
similarly makes silence an explicit learned action rather than an external
heuristic. The shared trend is a small, explicit decision interface on top of
a general multimodal backbone.
### 2. Transition supervision is becoming central
Proact-VL argues that the important imbalance is not merely class frequency,
but **state transition versus state persistence**. It upweights switch steps
and adds two stability terms: local probability consistency during persistent
states and global speaking-rate calibration. This maps directly onto the
EasyCom structure and is the highest-priority idea for improving 0.2-second
prediction.
The [BadCat unit-based agent](https://arxiv.org/abs/2601.20230) reaches a
similar conclusion from a systems perspective: it represents dialogue control
as `keep-listen`, `listen-to-speak`, `keep-speak`, and `speak-to-listen` actions.
That is a strong conceptual justification for adding a keep/switch auxiliary
head rather than another generic hidden layer.
### 3. Frozen backbones plus task-specific fusion are credible
[MM-When2Speak](https://arxiv.org/abs/2505.14654) formulates timing as dense
response-type classification and uses adaptive attention over visual, audio,
and textual tokens. Its response fusion stage freezes the LLM and modality
encoders and optimizes a lightweight attention module. This is closely aligned
with our empirical finding that frozen Omni is more stable than Thinker LoRA.
The important next ablation is therefore not another larger query MLP. It is a
comparison between:
1. final query state;
2. learned last-four query mixture;
3. recent 200ms audio-token pooling;
4. recent 200ms audio-video token attention;
5. query plus recent-token pool.
For token pooling, only payload tokens in the causal interval should be used,
selected by TMRoPE time IDs and excluding padding/template tokens.
### 4. Fine temporal cadence is a differentiator
Most proactive video systems make decisions at one-second cadence. EgoSpeak
and this project operate at 200ms. That is not merely an implementation detail:
it creates a credible ICASSP contribution around fine-grained conversational
switch anticipation, especially because Omni audio tokens retain finer 40ms
temporal positions.
The paper should emphasize that a two-second multimodal ingestion unit does
not force a two-second decision rate. Causal readout at each 200ms anchor can
still exploit fine-grained audio timing, provided the state/readout does not
collapse the current-tail tokens.
## Recommended paper contribution
Working framing:
> **Fine-Grained Causal Readout of Frozen Omni Models for Egocentric Turn
> Anticipation**
Possible three-part contribution:
1. **Streaming formulation.** Raw egocentric audio/video is encoded by a
frozen Omni Transformer with bounded causal memory, producing decisions at
5Hz without an external RNN.
2. **Hierarchical temporal readout.** A parameter-efficient upper-layer query
mixture is fused with recent TMRoPE-localized audio/video payload tokens,
exposing both semantic context and immediate prosodic/nonverbal cues.
3. **Structured transition-aware decoding.** A four-state overlap-aware output
is marginalized to EgoSpeak's three AP classes, while existing frame labels
generate current, keep/switch, and time-to-change supervision with
persistence consistency and session-aware validation.
The negative results become useful evidence rather than discarded runs:
- 7B is worse than 3B;
- Thinker LoRA causes representation drift and worse test mAP;
- concat-MLP helps briefly but collapses with longer training;
- a tiny last-four mixture recovers almost all of the useful gain.
Together they support the thesis that **the bottleneck is task-aligned causal
readout and supervision, not foundation-model capacity**.
## Prioritized experiment plan
### P0: make the current gain publishable
1. Train on Sessions 4--11 and use Session 12 only as development data.
2. Use one fixed 12-epoch schedule and save every epoch. Do not compare runs
whose cosine schedule changes with the requested epoch count.
3. Select epoch and threshold using a predeclared development metric.
4. Retrain on Sessions 4--12 for the selected epoch count, then evaluate
Sessions 1--3 once.
5. Run at least three seeds; EgoSpeak reports five-seed summary statistics.
6. Match single-query, fixed mean, scalar mixture, and concat-MLP schedules.
Primary selection metric should be `0.5 * mAP@0.2s + 0.5 * mean_mAP`, or both
metrics should be reported as a Pareto pair. Selecting only mean mAP can hide
the exact responsiveness regression that matters most to this project.
Before changing the representation, run an identical-output-objective
ablation on the frozen final-layer and scalar-mixture features:
| Output formulation | Training loss | Evaluation scores |
|---|---|---|
| Official control | current three-logit LSTR MCE | three-way softmax |
| Independent multi-label | BCE on wearer/others | sigmoid scores; background derived from neither |
| Structured state | four-way CE on silence/wearer/others/overlap | three marginals |
The structured state variant is preferred because it represents overlap
without abandoning a coherent normalized conversational state. If it improves
the scalar-mixture model, carry it into every later experiment.
### P1: low-capacity temporal readout
Run these in order:
| Readout | Approx. new fusion parameters | Purpose |
|---|---:|---|
| Last-4 scalar mixture | 4 | robust reference |
| Feature-wise last-4 gate | 8,192 | per-channel layer choice |
| Low-rank concat, rank 128 | about 1.31M | cross-layer interaction |
| Low-rank concat, rank 256 | about 2.62M | capacity ablation |
| Current concat-MLP | about 16.8M | upper-capacity control |
Initialize feature-wise/low-rank fusion as a residual around the scalar mean,
so the first optimizer step preserves the strong pretrained readout. Use
weight decay, dropout, and dev early stopping, but do not make generic
regularization the main novelty.
### P1: transition-aware supervision
Convert each target vector to the corresponding four-state variable `q_t`,
then derive:
- `current_t = q_t`;
- `switch_{t,h} = 1[q_{t+h} != q_t]` for short horizons;
- `time_to_change_t` capped at 2 seconds;
- `persistence_t = 1 - switch_{t,1}`.
Train the existing ten-horizon head jointly with current/switch/time-to-change
heads. Apply larger weight only to true switch steps, plus a consistency loss
when the ground-truth state persists. This should be evaluated first on the
camera-wearer AP at 0.2 seconds, where representation drift and head
overfitting have repeatedly caused the largest damage.
### P2: expose recent token evidence
Add a readout query that attends only to payload tokens from `[t-0.2s, t]`,
identified through TMRoPE positions. Compare audio-only recent tokens, AV
recent tokens, and query-plus-token fusion. Keep the backbone frozen.
This is the most distinctive method contribution, but it should follow P0/P1
because token masking and timestamp correctness are easier to get wrong than
layer fusion.
### P2: use existing semantics without test leakage
Use EasyCom transcription and Target-of-Speech only during training:
- distill utterance-completion likelihood into the AV state;
- predict group-versus-individual addressee;
- predict a coarse semantic continuation state such as incomplete/complete;
- drop the privileged text branch at inference.
This makes the claimed "Omni semantic advantage" testable. Merely replacing
wav2vec/ResNet with an Omni hidden state does not demonstrate semantic use.
### P3: generalization and ablations
- EasyCom audio-only, video-only, and AV under identical readout capacity;
- Ego4D 346/87 split with the same method and no EasyCom-specific tuning;
- 4s, 8s, and 20s memory with fixed 5Hz output;
- 2 FPS versus 5 FPS video ablation;
- runtime, memory, and causal verification;
- per-phase and transition-conditioned AP in addition to EgoSpeak Table 2.
## Stop conditions
Do not spend another full training cycle on:
- full Thinker LoRA on EasyCom alone;
- Qwen2.5-Omni-7B without a readout change;
- 25--50 epoch high-capacity heads without development early stopping;
- more overlapping-window extraction as the main architecture;
- prompt wording as the primary method.
Those directions are weakly supported by the current evidence and make the
paper harder to interpret. The clean next experiment is the matched P0 readout
ablation followed by transition-aware supervision.

280
reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md Normal file
View File

@@ -0,0 +1,280 @@
# ROMA negative transfer and the next ICASSP direction
Date: **2026-08-27**
This memo combines the newly archived frozen-ROMA EasyCom results with recent
work on predictive turn-taking and proactive streaming multimodal agents. The
main conclusion is deliberately narrow:
> The next useful experiment is not another backbone or a larger readout. It
> is a **low-capacity multi-timescale causal readout plus structured future
> floor projection** at 5 Hz.
## Evidence from the current project
| System | 0.2s mAP | Mean mAP | What it establishes |
|---|---:|---:|---|
| Frozen Qwen-3B, final query, 8s | 71.99768 | 58.51937 | strong simple baseline |
| Frozen Qwen-3B, last-4 scalar mixture | 73.45085 | 60.53423 | upper-layer information is useful |
| Frozen Qwen-3B, last-4 concat, 4ep | 73.42020 | 60.87144 | small longer-horizon gain, high overfit risk |
| Frozen Qwen-7B, reported run | 69.80133 | 57.67367 | capacity alone is not the answer |
| Frozen ROMA, last-4 fixed mean, 4ep | 69.21863 | 58.90402 | best ROMA control |
| Frozen ROMA, last-4 concat, 4ep | 68.65486 | 56.52425 | wide fusion overfits strongly |
With the same last-4 concat design and epoch count, ROMA loses `4.76534` at
0.2 seconds and `4.34719` in mean mAP relative to Qwen-3B. This is a strong
diagnostic but not a parameter-matched attribution test: ROMA's wider hidden
state makes its concat fusion 51.4M parameters instead of about 16.8M. The
fixed mean removes that fusion-capacity confound and the worst overfitting, but
an exact Qwen-3B fixed-mean control is still missing.
This does **not** prove that ROMA is a worse streaming model. ROMA is adapted
for one-second multimodal units and proactive alert/narration supervision,
whereas this project extracts it with an eight-second Qwen-style replay and
asks for 200 ms future-speaker anticipation. The result instead shows a task
alignment mismatch: stronger generic proactive behavior does not guarantee a
better representation for fine-grained floor transfer.
## Strongly related work and what transfers
### EgoSpeak: the exact benchmark contract
[EgoSpeak (Findings of NAACL 2025)](https://github.com/jun297/EgoSpeak)
defines the exact problem and supplies LSTR and MiniROAD baselines. Its main
strength is the 5 Hz future-speaker evaluation on EasyCom and Ego4D. Its
limitation for the present question is architectural: pretrained ResNet and
wav2vec features are followed by a task-specific temporal model, so it does
not test whether an Omni Transformer already contains the necessary causal
state.
**Transfer:** preserve its split, ten horizons, target alignment, causality,
and AP evaluator exactly. The paper contribution must lie above this unchanged
contract.
### MM-VAP and MuVAP: model the future activity trajectory, not ten isolated labels
[MM-VAP (Findings of ACL 2025)](https://aclanthology.org/2025.findings-acl.12/)
adds facial expression, head pose, and gaze to an audio VAP model and reports
84% versus 79% hold/shift accuracy. Its ablation identifies facial expression
as the most useful visual cue.
[MuVAP (arXiv 2026)](https://arxiv.org/abs/2606.16731) is even closer to this
project: it is causal, multimodal, multiparty, single-camera, and single-audio
stream. It predicts a near-future joint voice-activity window and introduces
**Role-Relative Projection**, mapping arbitrary speakers to current versus
next floor-holder roles to avoid combinatorial growth.
The original VAP insight is crucial: near-future speaking states are jointly
structured. A backchannel, overlap, hold, and switch are different trajectories
through the next two seconds, not ten unrelated classification targets.
**Transfer:** replace the independent `10 x 3` output with a structured future
floor decoder. EasyCom naturally supplies four states at each horizon:
`silence`, `wearer-only`, `others-only`, and `overlap`. A small temporal decoder
can model their dependencies and marginalize back to EgoSpeak's three AP
scores. No new annotation is required.
### Interspeech 2025 gaze study: global video tokens may hide the useful cue
[Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations
(Interspeech 2025)](https://www.isca-archive.org/interspeech_2025/heo25_interspeech.html)
shows that gaze from one participant already improves prediction, and multiple
participants improve it further when gaze is grounded with speaker location.
**Transfer:** EasyCom already provides face/head boxes and tracked poses. Use
them to build participant- or role-conditioned visual pooling rather than
expecting a global video query state to preserve small facial and gaze changes.
This is more defensible than increasing the global MLP.
### ROMA: aligned streaming units and a small explicit decision head
[ROMA](https://arxiv.org/abs/2601.10323) uses one-second synchronized
audio-video units, chunked TMRoPE, 40 ms audio temporal positions, and a
two-layer speak head fed by a learned mixture of the final four layers. It also
uses a two-stage curriculum: streaming-format alignment followed by proactive
timing training.
**Transfer:** keep the explicit lightweight decision interface and last-layer
mixture. More importantly, retain the 40 ms audio time IDs when selecting the
recent 200 ms payload. The present negative result says that importing ROMA's
checkpoint alone is insufficient; its useful idea is the interface and
alignment structure.
### Proact-VL: transition imbalance is not ordinary class imbalance
[Proact-VL (2026 preprint)](https://arxiv.org/abs/2603.03447) reads a dedicated
`FLAG` token with a small gated MLP. It weights rare speaking/silence state
transitions more heavily, smooths probabilities only during persistent states,
and regularizes the global speaking rate.
**Transfer:** the project's previous transition auxiliary head is not the same
idea. It asks an extra branch to predict a transition but leaves the primary
future-activity probabilities unconstrained. A stronger experiment should:
1. upweight the main state loss at true hold/switch boundaries;
2. penalize probability jitter only inside persistence regions;
3. calibrate predicted per-class activity rates on the training split.
This directly shapes the output trajectory and is more likely to improve the
0.2-second boundary behavior than another auxiliary classifier.
### VideoLLM-online and Dispider: separate high-rate perception/decision from generation
[VideoLLM-online (CVPR 2024)](https://showlab.github.io/videollm-online/)
trains temporally aligned streaming dialogue and parallelizes video encoding,
LLM forwarding, and generation; it reports 5--10 FPS on an RTX 3090 and
10--15 FPS on an A100.
[Dispider (CVPR 2025)](https://github.com/Mark12Ding/Dispider) disentangles
perception, decision, and reaction. A compact module continuously decides
`wait/respond`, while generation runs asynchronously and does not block new
perception.
**Transfer:** for the benchmark, keep only frozen Omni perception plus the 5 Hz
floor controller. For a later interactive demo, run text/speech generation
asynchronously after a trigger. Do not train the heavy generator merely to
improve a 30-logit anticipation benchmark.
## Proposed method: Multi-timescale Future Floor Projection
At every 200 ms prediction time `t`, form two causal summaries:
1. **semantic context** `q_t`: a fixed or learned low-capacity mixture of the
final four query states over the bounded eight-second stream;
2. **immediate interaction cue** `r_t`: attention pooling over only audio/video
payload tokens whose TMRoPE timestamps fall in `(t-0.2s, t]`.
Exclude template tokens, padding, and tokens later than `t`. Keep audio and
video pools separate first, then fuse them through a small gate:
```text
q_t (long semantic state) -----------+
+--> gated low-rank fusion --> future-floor decoder
a_t (recent 200 ms audio tokens) ----+
v_t (current/role visual tokens) ----+
```
The future-floor decoder predicts a sequence of ten four-state distributions,
not ten independent three-class logits. A minimal version is:
```text
fused state -> 10 learned horizon queries -> 2-layer temporal Transformer
-> [10, 4] state logits
```
Convert it to the official evaluator scores with:
```text
p(background) = p(silence)
p(wearer) = p(wearer-only) + p(overlap)
p(others) = p(others-only) + p(overlap)
```
This design joins three complementary signals:
- Omni upper layers provide long semantic context;
- TMRoPE-local token pooling exposes recent prosody and motion;
- VAP-style structured projection models persistence, handover, overlap, and
backchannel trajectories across the two-second future.
## Highest-value experiment sequence
### P0: close attribution gaps before adding a method
1. Run Qwen-3B last-4 **fixed mean**, four epochs, same three seeds and schedule
as the existing readouts.
2. Run original Qwen2.5-Omni-7B versus ROMA with the same last-four fixed-mean
features and head. This isolates ROMA adaptation from model size.
3. Stop ROMA if the matched comparison is not positive. Do not spend more
budget tuning its 51.4M concat head.
### P1: test the central readout hypothesis
Use frozen Qwen-3B and the same dense 5 Hz anchors:
| Variant | Long state | Recent 200ms payload | Future decoder |
|---|---|---|---|
| A | final query | none | current `10 x 3` |
| B | last-4 mixture | none | current `10 x 3` |
| C | last-4 mixture | audio pool | current `10 x 3` |
| D | last-4 mixture | audio + role visual pool | current `10 x 3` |
| E | last-4 mixture | audio + role visual pool | structured `[10,4]` |
Variant C is the most informative first implementation. If C cannot improve
0.2-second wearer/background AP, the hypothesis that the single query hides
recent prosody is weakened before more complicated visual work.
### P2: add trajectory-aware loss only after P1
For E, compare:
- ordinary four-state CE;
- boundary-weighted CE;
- boundary-weighted CE plus persistence-only smoothness;
- the above plus class activity-rate calibration.
Keep loss weights small and predeclare them on a development session. Report
the main activity mAP and separate diagnostics on transition-near versus
persistence anchors.
### P3: use EasyCom annotations without new manual labels
Participant head/face boxes and tracked poses can supervise role-conditioned
visual pooling. Speech transcripts and Target-of-Speech labels should be
training-only privileged signals or distillation targets, not test-time input.
This gives a principled semantic/visual benefit while preserving the raw AV
online setting.
### P4: require Ego4D confirmation
EasyCom has only nine training conversations. A method selected on EasyCom
must be tested on the much larger Ego4D split. A gain on both datasets supports
a representation/readout claim; a gain only on EasyCom may reflect its fixed
around-table geometry.
## Evaluation contract for a publishable result
- Primary: 0.2-second mAP, because this is the current deficit and the claimed
fine-grained contribution.
- Secondary: mean mAP over all ten horizons.
- Diagnostics: per-class AP, transition-near/persistence mAP, overlap-state AP,
calibration, and end-to-end 5 Hz latency.
- Selection: Sessions 4--11 train and Session 12 development, then retrain on
Sessions 4--12 for the fixed budget and evaluate Sessions 1--3 once.
- Statistics: at least three seeds; five if matching EgoSpeak's reporting.
- Causality: no token, frame, transcript, or normalization statistic after
prediction time `t`.
## Paper angle
A defensible ICASSP framing is:
> **Fine-Grained Multimodal Future Floor Projection from Frozen Omni Models**
Possible contributions:
1. a strictly causal 5 Hz egocentric audio-video anticipation formulation on
top of a frozen Omni Transformer;
2. a multi-timescale readout that combines upper-layer semantic context with
TMRoPE-local recent payload cues;
3. an overlap-aware structured future-floor decoder with transition/persistence
supervision;
4. controlled evidence that larger backbones, ROMA transfer, and Thinker LoRA
do not solve the fine-grained task by themselves.
This is stronger than claiming a new head. The scientific claim is that
general streaming Omni states must be **read and supervised at the temporal
structure of conversational floor transfer**.
## Stop conditions
- Stop backbone scaling unless the exact same readout/schedule improves both
0.2-second and mean mAP.
- Stop high-capacity concat fusion; the current data already show severe
session-level overfitting.
- Stop transition auxiliary branches that do not change the primary activity
trajectory.
- Stop tuning on Sessions 1--3; use them once after the configuration is fixed.
The archived ROMA tables and provenance are in
[`results/easycom_roma_frozen_last4_2026-08-27/`](../results/easycom_roma_frozen_last4_2026-08-27/README.md).

1
results/README.md
View File

@@ -6,6 +6,7 @@ readers to the right comparison.
| Result record | Question answered | | Result record | Question answered |
|---|---| |---|---|
| [easycom_oracle_pooled_wearer_audit_2026-09-10/](easycom_oracle_pooled_wearer_audit_2026-09-10/README.md) | Fixed-four-epoch three-seed oracle test results and the limited wearer gains of historical pooled AV |
| [`easycom_open_stream_vs_causal_prefix_2026-08-19/`](easycom_open_stream_vs_causal_prefix_2026-08-19/README.md) | How prepared-media open-stream and causal-prefix features compare | | [`easycom_open_stream_vs_causal_prefix_2026-08-19/`](easycom_open_stream_vs_causal_prefix_2026-08-19/README.md) | How prepared-media open-stream and causal-prefix features compare |
| [`easycom_open_stream_8s_vs_20s_2026-08-23/`](easycom_open_stream_8s_vs_20s_2026-08-23/README.md) | Frozen open-stream context-length comparison | | [`easycom_open_stream_8s_vs_20s_2026-08-23/`](easycom_open_stream_8s_vs_20s_2026-08-23/README.md) | Frozen open-stream context-length comparison |
| [`easycom_sparse_supervision_vs_lora_2026-08-24/`](easycom_sparse_supervision_vs_lora_2026-08-24/README.md) | Sparse supervision versus LoRA comparison | | [`easycom_sparse_supervision_vs_lora_2026-08-24/`](easycom_sparse_supervision_vs_lora_2026-08-24/README.md) | Sparse supervision versus LoRA comparison |

159
results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25/README.md Normal file
View File

@@ -0,0 +1,159 @@
# EasyCom dense last-8 Thinker LoRA with a frozen head
Recorded on **2026-08-25** from user-reported remote evaluation output. This is
a single-run diagnostic result, not a multi-seed paper number.
## Experiment contract
- Model: Qwen2.5-Omni-3B Thinker.
- Inputs: dense cached AV phases, 2 FPS video, raw-audio tower embeddings, and
an 8-second causal memory budget.
- Prediction/label rate: 5 Hz with ten horizons from 0.2 to 2.0 seconds.
- LoRA scope: `q_proj`, `v_proj`, and `o_proj` in Thinker layers 28--35 only.
- LoRA configuration: rank 8, alpha 16, dropout 0.05, learning rate `5e-6`,
weight decay 0.01.
- Head: the trained frozen 8-second head, loaded from
`checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth` and kept frozen.
- LoRA training: dense Sessions 4--11, global batch 16, five continuous epochs.
- Development selection: Session 12, primary metric 0.2-second full-timeline
mAP and mean mAP as tie-breaker.
- Formal evaluation: epoch 3 was first evaluated on official EasyCom test
Sessions 3, 2, and 1 after development selection. Epoch 1 was subsequently
evaluated as a diagnostic before fixing the one-epoch full-training recipe.
- Selected checkpoint: epoch 3, step 11910.
The selected remote checkpoint was:
```text
checkpoints/omni_turn_lora_dense_8s_last8_qvo_frozen_head_epoch5/
checkpoint-step-11910/
```
The diagnostic epoch-1 checkpoint was:
```text
checkpoints/omni_turn_lora_dense_8s_last8_qvo_frozen_head_epoch5/
checkpoint-step-3970/
```
## Development selection on Session 12
All values are percentages.
| Epoch | Step | 0.2s mAP | Mean mAP |
|---:|---:|---:|---:|
| 1 | 3970 | 70.80918 | 63.06487 |
| 2 | 7940 | 70.96876 | **63.09318** |
| 3 | 11910 | **70.99153** | 63.01294 |
| 4 | 15880 | 70.87364 | 62.85465 |
| 5 | 19850 | 70.83634 | 62.81744 |
Epoch 3 was selected by the preregistered 0.2-second criterion. The five-run
range is only 0.18235 points at 0.2 seconds and 0.27574 mean points, indicating
that the restricted low-learning-rate LoRA is much more stable than the prior
full-Thinker sparse-LoRA runs. Epochs 4--5 nevertheless show mild overfitting.
## Official Sessions 1--3 results
Epoch 1 and epoch 3 are close at 0.2 seconds, but epoch 1 preserves the longer
horizons better and therefore has the higher mean mAP.
| Checkpoint | 0.2s mAP | Mean mAP | 0.2s delta vs frozen | Mean delta vs frozen |
|---|---:|---:|---:|---:|
| Frozen dense control | **71.69695** | **58.16091** | -- | -- |
| LoRA epoch 1 / step 3970 | 71.59585 | 58.01500 | -0.10110 | -0.14591 |
| LoRA epoch 3 / step 11910 | 71.65000 | 57.88833 | -0.04695 | -0.27258 |
Epoch 1 is only `0.05415` below epoch 3 at 0.2 seconds and is `0.12667`
higher in mean mAP. Relative to epoch 3, it improves every horizon from 0.6
through 2.0 seconds. This supports fixing one epoch and retraining once on all
official training Sessions 4--12, without development checkpoint selection.
### Epoch 1 / step 3970
| Horizon | LoRA mAP | Frozen dense mAP | Delta | Background AP | Wearer AP | Others AP |
|---:|---:|---:|---:|---:|---:|---:|
| 0.2s | 71.59585 | 71.69695 | -0.10110 | 54.40604 | 76.39165 | 83.98987 |
| 0.4s | 66.37987 | 66.46468 | -0.08481 | 47.56313 | 70.58803 | 80.98845 |
| 0.6s | 61.91962 | 62.03503 | -0.11541 | 42.34205 | 64.79174 | 78.62507 |
| 0.8s | 58.62537 | 58.77409 | -0.14872 | 39.04030 | 59.81340 | 77.02241 |
| 1.0s | 56.30013 | 56.43606 | -0.13593 | 37.19759 | 55.95535 | 75.74744 |
| 1.2s | 54.86526 | 55.00306 | -0.13780 | 36.31219 | 53.41217 | 74.87142 |
| 1.4s | 53.59386 | 53.73114 | -0.13728 | 35.39635 | 51.13887 | 74.24635 |
| 1.6s | 52.86682 | 53.04042 | -0.17360 | 35.19829 | 49.67869 | 73.72348 |
| 1.8s | 52.23019 | 52.42454 | -0.19435 | 34.71486 | 48.59750 | 73.37822 |
| 2.0s | 51.77300 | 52.00315 | -0.23015 | 34.68133 | 47.63528 | 73.00237 |
| **Mean** | **58.01500** | **58.16091** | **-0.14591** | **39.68521** | **57.80027** | **76.55951** |
### Epoch 3 / step 11910
The fair control is the frozen Thinker evaluated from the same dense AV cache
path with the same frozen head. `Delta` is selected LoRA minus that control.
| Horizon | LoRA mAP | Frozen dense mAP | Delta | Background AP | Wearer AP | Others AP |
|---:|---:|---:|---:|---:|---:|---:|
| 0.2s | 71.65000 | 71.69695 | -0.04695 | 54.59245 | 76.13410 | 84.22345 |
| 0.4s | 66.38669 | 66.46468 | -0.07799 | 47.57127 | 70.36283 | 81.22596 |
| 0.6s | 61.87698 | 62.03503 | -0.15805 | 42.33591 | 64.49864 | 78.79640 |
| 0.8s | 58.51356 | 58.77409 | -0.26053 | 38.97583 | 59.42986 | 77.13498 |
| 1.0s | 56.13447 | 56.43606 | -0.30159 | 37.07515 | 55.55303 | 75.77523 |
| 1.2s | 54.69993 | 55.00306 | -0.30313 | 36.20847 | 53.01781 | 74.87352 |
| 1.4s | 53.41760 | 53.73114 | -0.31354 | 35.30111 | 50.71685 | 74.23484 |
| 1.6s | 52.64243 | 53.04042 | -0.39799 | 35.00690 | 49.21422 | 73.70616 |
| 1.8s | 52.00887 | 52.42454 | -0.41567 | 34.55009 | 48.10873 | 73.36779 |
| 2.0s | 51.55275 | 52.00315 | -0.45040 | 34.55502 | 47.13447 | 72.96877 |
| **Mean** | **57.88833** | **58.16091** | **-0.27258** | **39.61722** | **57.41705** | **76.63071** |
The class-mean comparison is:
| Class | LoRA mean AP | Frozen dense mean AP | Delta |
|---|---:|---:|---:|
| Background | 39.61722 | 39.68434 | -0.06712 |
| Camera wearer speaking | 57.41705 | 58.04423 | **-0.62718** |
| Other people speaking | 76.63071 | 76.75417 | -0.12346 |
At 0.2 seconds specifically, background and other-person AP increase by
`+0.10051` and `+0.09021`, but wearer AP falls by `-0.33157`; the net mAP is
therefore `-0.04695`. The negative gap grows almost monotonically with horizon
and reaches `-0.45040` at 2.0 seconds.
## Interpretation
1. **Restricted LoRA nearly eliminates the short-horizon collapse but does not
create positive transfer.** The prior frozen-head initialized sparse LoRA
scored 71.08589 at 0.2 seconds and 56.19608 mean; the new recipe improves
those by `+0.56411` and `+1.69225`, respectively, but remains below the
frozen dense control.
2. **The remaining negative transfer is horizon-dependent.** The 0.2-second
representation is essentially preserved, while longer anticipation is
progressively damaged. This is inconsistent with a simple global
calibration error.
3. **Camera-wearer speech is the main failure mode.** Its average AP decreases
by 0.62718 points, much more than background or other-person speech.
4. **More epochs are not justified.** Epoch 1 is already within 0.05415 points
of epoch 3 at 0.2 seconds and is 0.12667 points better in mean mAP on the
official test sessions. The remaining official-split experiment therefore
fixes one epoch and trains on Sessions 4--12 without checkpoint selection.
5. **This checkpoint is an ablation, not a headline improvement.** It does not
beat frozen Omni, the 58.51937 open-stream frozen-8s result, or EgoSpeak GRU
A+V at 60.6 mean mAP.
The next defensible post-training diagnostic is adapter scaling/interpolation
on a clean development protocol. It should not be tuned from the already-seen
Sessions 1--3 result. For publication, retrain the frozen head without the
chosen development session; the reused frozen head was originally trained on
the official training split and may already have seen Session 12.
## Provenance and limitations
- Source: user-reported remote `eval_omni_turn_lora.py` console tables.
- Raw remote `metrics.json` archived here: no.
- Strict causal-valid table supplied: no.
- Exact remote git revision confirmed: no; the relevant implementation commits
include `7acab44`, `7fdda29`, and `aa4278e`.
- Multi-seed: no.
- Test-use caveat: epoch 1 was inspected after the epoch-3 test result; their
comparison is diagnostic rather than untouched test-set model selection.
- Development caveat: the frozen head may have been trained with Session 12;
this is adequate for diagnosis but is not a clean publication dev split.
- Publication status: negative-transfer ablation only.

93
results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25/summary.json Normal file
View File

@@ -0,0 +1,93 @@
{
"schema_version": 1,
"recorded_date": "2026-08-25",
"status": "single_run_negative_transfer_ablation",
"dataset": "EasyCom",
"prediction_step_seconds": 0.2,
"video_fps": 2,
"context_seconds": 8,
"train_sessions": [4, 5, 6, 7, 8, 9, 10, 11],
"development_sessions": [12],
"test_sessions": [3, 2, 1],
"configuration": {
"model": "Qwen2.5-Omni-3B Thinker",
"anchor_policy": "dense_5hz",
"lora_layers": [28, 29, 30, 31, 32, 33, 34, 35],
"lora_target_modules": ["q_proj", "v_proj", "o_proj"],
"lora_rank": 8,
"lora_alpha": 16,
"lora_dropout": 0.05,
"learning_rate": 0.000005,
"weight_decay": 0.01,
"head_frozen": true,
"epochs": 5,
"selected_epoch": 3,
"selected_step": 11910
},
"development_metrics_percent": [
{"epoch": 1, "step": 3970, "horizon_0.2_map": 70.80918, "mean_map": 63.06487},
{"epoch": 2, "step": 7940, "horizon_0.2_map": 70.96876, "mean_map": 63.09318},
{"epoch": 3, "step": 11910, "horizon_0.2_map": 70.99153, "mean_map": 63.01294},
{"epoch": 4, "step": 15880, "horizon_0.2_map": 70.87364, "mean_map": 62.85465},
{"epoch": 5, "step": 19850, "horizon_0.2_map": 70.83634, "mean_map": 62.81744}
],
"epoch1_test_metrics_percent": {
"protocol": "egospeak_full_timeline",
"num_timesteps": 23048,
"step": 3970,
"horizon_seconds": [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0],
"horizon_map": [71.59585, 66.37987, 61.91962, 58.62537, 56.30013, 54.86526, 53.59386, 52.86682, 52.23019, 51.77300],
"mean_map": 58.01500,
"background_ap": [54.40604, 47.56313, 42.34205, 39.04030, 37.19759, 36.31219, 35.39635, 35.19829, 34.71486, 34.68133],
"camera_wearer_speaking_ap": [76.39165, 70.58803, 64.79174, 59.81340, 55.95535, 53.41217, 51.13887, 49.67869, 48.59750, 47.63528],
"other_people_speaking_ap": [83.98987, 80.98845, 78.62507, 77.02241, 75.74744, 74.87142, 74.24635, 73.72348, 73.37822, 73.00237]
},
"test_metrics_percent": {
"protocol": "egospeak_full_timeline",
"num_timesteps": 23048,
"horizon_seconds": [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0],
"horizon_map": [71.65000, 66.38669, 61.87698, 58.51356, 56.13447, 54.69993, 53.41760, 52.64243, 52.00887, 51.55275],
"mean_map": 57.88833,
"background_ap": [54.59245, 47.57127, 42.33591, 38.97583, 37.07515, 36.20847, 35.30111, 35.00690, 34.55009, 34.55502],
"camera_wearer_speaking_ap": [76.13410, 70.36283, 64.49864, 59.42986, 55.55303, 53.01781, 50.71685, 49.21422, 48.10873, 47.13447],
"other_people_speaking_ap": [84.22345, 81.22596, 78.79640, 77.13498, 75.77523, 74.87352, 74.23484, 73.70616, 73.36779, 72.96877]
},
"frozen_dense_control_percent": {
"horizon_map": [71.69695, 66.46468, 62.03503, 58.77409, 56.43606, 55.00306, 53.73114, 53.04042, 52.42454, 52.00315],
"mean_map": 58.16091,
"class_mean_ap": {
"background": 39.68434,
"camera_wearer_speaking": 58.04423,
"other_people_speaking": 76.75417
}
},
"selected_lora_minus_frozen_dense_percent_points": {
"horizon_map": [-0.04695, -0.07799, -0.15805, -0.26053, -0.30159, -0.30313, -0.31354, -0.39799, -0.41567, -0.45040],
"mean_map": -0.27258,
"class_mean_ap": {
"background": -0.06712,
"camera_wearer_speaking": -0.62718,
"other_people_speaking": -0.12346
}
},
"epoch1_lora_minus_frozen_dense_percent_points": {
"horizon_map": [-0.10110, -0.08481, -0.11541, -0.14872, -0.13593, -0.13780, -0.13728, -0.17360, -0.19435, -0.23015],
"mean_map": -0.14591,
"class_mean_ap": {
"background": 0.00087,
"camera_wearer_speaking": -0.24396,
"other_people_speaking": -0.19466
}
},
"epoch1_minus_epoch3_percent_points": {
"horizon_0.2_map": -0.05415,
"mean_map": 0.12667
},
"provenance": {
"source": "User-reported remote evaluator console tables",
"raw_metrics_json_archived": false,
"strict_causal_metrics_supplied": false,
"multi_seed": false,
"clean_development_head_split": false
}
}

84
results/easycom_dense_last8_qvo_full_train_epoch1_2026-08-25/README.md Normal file
View File

@@ -0,0 +1,84 @@
# EasyCom dense last-8 LoRA, full official train split, one epoch
Recorded on **2026-08-25** from user-reported remote evaluator output. This is
a single-seed diagnostic experiment.
## Experiment contract
- Backbone: Qwen2.5-Omni-3B Thinker.
- Input representation: dense cached AV phases, 2 FPS video, 5 Hz prediction,
and 8 seconds of causal open-stream memory.
- Head: pretrained frozen 8-second head from
`checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth`.
- LoRA: `q_proj`, `v_proj`, and `o_proj` in Thinker layers 28--35, rank 8,
alpha 16, dropout 0.05, learning rate `5e-6`, and weight decay 0.01.
- Training: all official EasyCom train Sessions 4--12, one fixed epoch, 4526
updates, global batch 16, and no development checkpoint selection.
- Evaluation: official test Sessions 3, 2, and 1 using the EgoSpeak Table 2
full-timeline protocol.
- Training runtime: 1475.936 seconds on eight GPUs, 3.067 updates/second, with
8.86 GiB as the maximum per-rank CUDA allocation.
- Cached loss: 1.272857 before training and 1.269106 after training.
The evaluated checkpoint was:
```text
checkpoints/
omni_turn_lora_dense_8s_last8_qvo_frozen_head_train4to12_epoch1_ddp8/
```
## Official Sessions 1--3 result
| Horizon | LoRA mAP | Frozen dense mAP | Delta | Background AP | Wearer AP | Others AP |
|---:|---:|---:|---:|---:|---:|---:|
| 0.2s | 71.62100 | 71.69695 | -0.07595 | 54.41759 | 76.41473 | 84.03068 |
| 0.4s | 66.38400 | 66.46468 | -0.08068 | 47.56390 | 70.58441 | 81.00369 |
| 0.6s | 61.93324 | 62.03503 | -0.10179 | 42.35133 | 64.80023 | 78.64817 |
| 0.8s | 58.64198 | 58.77409 | -0.13211 | 39.04739 | 59.81664 | 77.06191 |
| 1.0s | 56.30504 | 56.43606 | -0.13102 | 37.17699 | 55.95400 | 75.78412 |
| 1.2s | 54.87690 | 55.00306 | -0.12616 | 36.30712 | 53.40678 | 74.91682 |
| 1.4s | 53.60541 | 53.73114 | -0.12573 | 35.40362 | 51.12561 | 74.28699 |
| 1.6s | 52.89234 | 53.04042 | -0.14808 | 35.22795 | 49.68491 | 73.76416 |
| 1.8s | 52.26201 | 52.42454 | -0.16253 | 34.76888 | 48.59595 | 73.42119 |
| 2.0s | 51.82720 | 52.00315 | -0.17595 | 34.76777 | 47.63587 | 73.07796 |
| **Mean** | **58.03491** | **58.16091** | **-0.12600** | **39.70325** | **57.80191** | **76.59957** |
## Controlled comparisons
| Checkpoint | Train sessions | Epoch | 0.2s mAP | Mean mAP | Mean delta vs frozen |
|---|---|---:|---:|---:|---:|
| Frozen dense control | 4--12 head | -- | **71.69695** | **58.16091** | -- |
| Restricted LoRA | 4--12 | 1 | 71.62100 | 58.03491 | -0.12600 |
| Restricted LoRA | 4--11 | 1 | 71.59585 | 58.01500 | -0.14591 |
| Restricted LoRA | 4--11 | 3 | 71.65000 | 57.88833 | -0.27258 |
Adding Session 12 to the one-epoch LoRA training improves 0.2-second mAP by
`0.02515` and mean mAP by `0.01991` relative to the Sessions 4--11 epoch-1
checkpoint. All ten horizons improve, but only by `0.00413--0.05420` points.
This is too small to explain or reverse the negative transfer.
Relative to the frozen dense control, background mean AP is unchanged
(`+0.01891`), while camera-wearer and other-person mean AP remain lower by
`-0.24232` and `-0.15460`. The final mean gap is `-0.12600`, less than half the
epoch-3 gap but still negative.
## Interpretation
1. Using the complete official training split helps consistently but only
marginally; withholding Session 12 was not the cause of the regression.
2. One epoch is preferable to three for this restricted adapter because it
preserves longer-horizon representations better.
3. The adapter still does not improve the frozen Omni representation. This
experiment should be reported as a negative-transfer ablation rather than
the main result.
4. The next primary comparison should use a stronger frozen backbone, such as
Qwen2.5-Omni-7B, without LoRA.
## Provenance and limitations
- Source: user-reported `eval_omni_turn_lora.py` console output.
- Raw remote `metrics.json` archived here: no.
- Strict causal-valid metrics supplied: no.
- Multi-seed: no.
- Test sessions had already been inspected in earlier LoRA diagnostics, so
this is not an untouched test-set selection result.

76
results/easycom_dense_last8_qvo_full_train_epoch1_2026-08-25/summary.json Normal file
View File

@@ -0,0 +1,76 @@
{
"schema_version": 1,
"recorded_date": "2026-08-25",
"status": "single_run_negative_transfer_ablation",
"dataset": "EasyCom",
"model": "Qwen2.5-Omni-3B Thinker",
"prediction_step_seconds": 0.2,
"video_fps": 2,
"context_seconds": 8,
"train_sessions": [7, 11, 10, 6, 9, 4, 8, 5, 12],
"development_sessions": [],
"test_sessions": [3, 2, 1],
"configuration": {
"anchor_policy": "dense_5hz",
"lora_layers": [28, 29, 30, 31, 32, 33, 34, 35],
"lora_target_modules": ["q_proj", "v_proj", "o_proj"],
"lora_rank": 8,
"lora_alpha": 16,
"lora_dropout": 0.05,
"learning_rate": 0.000005,
"weight_decay": 0.01,
"head_frozen": true,
"epochs": 1,
"updates": 4526,
"global_batch_size": 16,
"num_gpus": 8,
"training_seconds": 1475.936,
"updates_per_second": 3.067,
"peak_cuda_allocated_gib_max_across_ranks": 8.86,
"initial_cached_loss": 1.272857,
"final_cached_loss": 1.269106
},
"test_metrics_percent": {
"protocol": "egospeak_full_timeline",
"num_timesteps": 23048,
"horizon_seconds": [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0],
"horizon_map": [71.62100, 66.38400, 61.93324, 58.64198, 56.30504, 54.87690, 53.60541, 52.89234, 52.26201, 51.82720],
"mean_map": 58.03491,
"background_ap": [54.41759, 47.56390, 42.35133, 39.04739, 37.17699, 36.30712, 35.40362, 35.22795, 34.76888, 34.76777],
"camera_wearer_speaking_ap": [76.41473, 70.58441, 64.80023, 59.81664, 55.95400, 53.40678, 51.12561, 49.68491, 48.59595, 47.63587],
"other_people_speaking_ap": [84.03068, 81.00369, 78.64817, 77.06191, 75.78412, 74.91682, 74.28699, 73.76416, 73.42119, 73.07796],
"class_mean_ap": {
"background": 39.70325,
"camera_wearer_speaking": 57.80191,
"other_people_speaking": 76.59957
}
},
"frozen_dense_control_percent": {
"horizon_map": [71.69695, 66.46468, 62.03503, 58.77409, 56.43606, 55.00306, 53.73114, 53.04042, 52.42454, 52.00315],
"mean_map": 58.16091,
"class_mean_ap": {
"background": 39.68434,
"camera_wearer_speaking": 58.04423,
"other_people_speaking": 76.75417
}
},
"lora_minus_frozen_dense_percent_points": {
"horizon_map": [-0.07595, -0.08068, -0.10179, -0.13211, -0.13102, -0.12616, -0.12573, -0.14808, -0.16253, -0.17595],
"mean_map": -0.12600,
"class_mean_ap": {
"background": 0.01891,
"camera_wearer_speaking": -0.24232,
"other_people_speaking": -0.15460
}
},
"full_train_epoch1_minus_train4to11_epoch1_percent_points": {
"horizon_map": [0.02515, 0.00413, 0.01362, 0.01661, 0.00491, 0.01164, 0.01155, 0.02552, 0.03182, 0.05420],
"mean_map": 0.01991
},
"provenance": {
"source": "User-reported remote evaluator console table",
"raw_metrics_json_archived": false,
"strict_causal_metrics_supplied": false,
"multi_seed": false
}
}

121
results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/README.md Normal file
View File

@@ -0,0 +1,121 @@
# EasyCom last-4 concat-MLP epoch-budget sweep
Recorded on **2026-08-26** from three user-reported remote runs. This is an
exploratory single-seed capacity/overfitting diagnostic, not a clean
test-independent checkpoint-selection result.
## Shared experiment contract
- Dataset: EasyCom.
- Backbone: frozen Qwen2.5-Omni-3B Thinker.
- Features: final four query states from 8-second open-stream replay, 2 FPS
video, and 5 Hz prediction timestamps.
- Fusion: concatenate `[4, 2048]` to 8192 dimensions, then
`LayerNorm -> Linear(8192, 2048) -> GELU` before the unchanged `10 x 3`
anticipation MLP.
- Training examples: dense 5 Hz anchors from the official training sessions.
- Evaluation: EgoSpeak full-timeline AP/mAP on Sessions 3, 2, and 1.
- The three runs differ in total training budget: 4, 12, or 50 epochs.
## Main result
| Total epochs | Final train loss | 0.2s mAP | Mean mAP | Delta mean from 4 epochs |
|---:|---:|---:|---:|---:|
| **4** | 0.806973 | **73.42020** | **60.87144** | -- |
| 12 | 0.661751 | 72.44507 | 57.98809 | -2.88335 |
| 50 | **0.300277** | 66.54645 | 53.03515 | -7.83629 |
Training loss decreases monotonically while test mAP collapses. This is strong
evidence that the 16.8-million-parameter concat fusion overfits EasyCom's small,
highly correlated session-level training set when trained for conventional
50-epoch head budgets.
## Per-horizon mAP
| Horizon | 4 epochs | 12 epochs | 50 epochs | 4ep minus 12ep | 4ep minus 50ep |
|---:|---:|---:|---:|---:|---:|
| 0.2s | 73.42020 | 72.44507 | 66.54645 | +0.97513 | +6.87375 |
| 0.4s | 68.60449 | 66.95546 | 60.89720 | +1.64903 | +7.70729 |
| 0.6s | 64.61782 | 62.31847 | 56.41553 | +2.29935 | +8.20229 |
| 0.8s | 61.53765 | 58.54713 | 53.45517 | +2.99052 | +8.08248 |
| 1.0s | 59.41820 | 56.08389 | 51.56044 | +3.33431 | +7.85776 |
| 1.2s | 58.10069 | 54.58857 | 50.11650 | +3.51212 | +7.98419 |
| 1.4s | 57.24684 | 53.57918 | 48.95956 | +3.66766 | +8.28728 |
| 1.6s | 56.19807 | 52.66949 | 48.10094 | +3.52858 | +8.09713 |
| 1.8s | 55.18182 | 51.73658 | 47.49916 | +3.44524 | +7.68266 |
| 2.0s | 54.38864 | 50.95706 | 46.80053 | +3.43158 | +7.58811 |
| **Mean** | **60.87144** | **57.98809** | **53.03515** | **+2.88335** | **+7.83629** |
Longer horizons deteriorate earlier and more strongly than 0.2 seconds. The
12-epoch model loses `0.97513` at 0.2 seconds but more than 3.3 points from
1.0 through 2.0 seconds. The model is therefore not merely becoming globally
miscalibrated; it increasingly memorizes short-term training correlations that
do not support future-speaker anticipation.
## Average class AP over ten horizons
| Epoch budget | Background | Camera wearer | Other people |
|---:|---:|---:|---:|
| 4 | **42.12966** | **62.08079** | **78.40387** |
| 12 | 40.15120 | 56.70353 | 77.10954 |
| 50 | 34.28629 | 51.12830 | 73.69085 |
| 4ep minus 50ep | +7.84337 | **+10.95249** | +4.71302 |
The camera-wearer class is most sensitive to overfitting. This matches the
project's earlier LoRA diagnosis: wearer prediction repeatedly suffers first
when representation/head capacity is increased without enough independent
conversation sessions.
## Comparison with the lower-capacity readouts
| Readout | Training budget | 0.2s mAP | Mean mAP |
|---|---:|---:|---:|
| Single final-layer query | prior run | 71.99768 | 58.51937 |
| Last-4 learned scalar mixture | 25 epochs | **73.45085** | 60.53423 |
| Last-4 concat-MLP | 4 epochs | 73.42020 | **60.87144** |
Relative to scalar mixture, the 4-epoch concat-MLP is essentially tied at
0.2 seconds (`-0.03065`) and improves mean mAP by `+0.33721`. Its additional
gain is concentrated around 1.2--1.6 seconds, peaking at `+0.80974` at 1.4
seconds. The higher-capacity fusion therefore helps longer-term anticipation,
but does not improve the primary 0.2-second score beyond the scalar mixture.
## Validity caveats
1. The three commands wrote to the same misleadingly named checkpoint and
result paths containing `25ep`. Later runs therefore overwrite earlier
artifacts unless they were copied elsewhere. The 4-epoch model must be
rerun with a unique path if it was not preserved.
2. Each total-epoch setting creates a different cosine schedule. These are
training-budget comparisons, not checkpoints sampled from one identical
optimization trajectory.
3. The supplied excerpts do not contain complete optimizer/scheduler configs.
Confirm them from the remote checkpoint JSON before a paper table.
4. The 4-epoch budget was identified after evaluating Sessions 1--3. It is a
diagnostic result and cannot be presented as test-independent model
selection.
5. Only full-timeline metrics were supplied; strict causal-valid metrics and
remote `metrics.json` are not archived here.
6. All values are single-seed results.
## Clean follow-up protocol
- Use Sessions 4--11 for training and Session 12 for model selection.
- Save every epoch from one fixed 12-epoch schedule and select using a declared
development metric (0.2-second mAP for responsiveness, or mean mAP for Table
2 overall performance).
- Freeze the selected epoch count before retraining on Sessions 4--12 and
evaluating Sessions 1--3 once.
- Compare single-layer, fixed last-4 mean, scalar mixture, and concat-MLP with
identical seeds and schedules.
- For the concat model, test a smaller bottleneck or low-rank fusion before
adding stronger generic regularization; this preserves the cross-layer idea
while matching EasyCom's effective sample size.
Machine-readable values are stored in [`summary.json`](summary.json) and
[`metrics.csv`](metrics.csv). A cleaned transcription of the supplied console
tables is in [`console_transcription.txt`](console_transcription.txt).
The data audit, recent-work comparison, paper framing, and prioritized next
experiments are documented in
[`reports/omni_turn_research_direction_2026-08-26.md`](../../reports/omni_turn_research_direction_2026-08-26.md).

41
results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/console_transcription.txt Normal file
View File

@@ -0,0 +1,41 @@
4 epochs
final train loss: 0.806973
0.2s mAP=73.42020 background=56.72497 wearer=77.87909 others=85.65653
0.4s mAP=68.60449 background=49.88131 wearer=73.26704 others=82.66511
0.6s mAP=64.61782 background=44.66684 wearer=68.57660 others=80.61002
0.8s mAP=61.53765 background=41.31402 wearer=64.21804 others=79.08089
1.0s mAP=59.41820 background=39.34993 wearer=61.35721 others=77.54745
1.2s mAP=58.10069 background=38.91573 wearer=58.45949 others=76.92686
1.4s mAP=57.24684 background=38.68165 wearer=57.00022 others=76.05865
1.6s mAP=56.19807 background=37.71307 wearer=55.21953 others=75.66162
1.8s mAP=55.18182 background=36.99897 wearer=53.45848 others=75.08801
2.0s mAP=54.38864 background=37.05013 wearer=51.37220 others=74.74359
mean mAP=60.87144
12 epochs
final train loss: 0.661751
0.2s mAP=72.44507 background=55.62292 wearer=76.67027 others=85.04201
0.4s mAP=66.95546 background=48.28782 wearer=70.71200 others=81.86658
0.6s mAP=62.31847 background=42.64250 wearer=64.97839 others=79.33452
0.8s mAP=58.54713 background=38.96725 wearer=59.03811 others=77.63603
1.0s mAP=56.08389 background=37.02472 wearer=55.22792 others=75.99903
1.2s mAP=54.58857 background=36.78301 wearer=51.71494 others=75.26775
1.4s mAP=53.57918 background=36.51419 wearer=49.39741 others=74.82594
1.6s mAP=52.66949 background=35.89852 wearer=48.01650 others=74.09344
1.8s mAP=51.73658 background=35.09050 wearer=46.39426 others=73.72498
2.0s mAP=50.95706 background=34.68059 wearer=44.88548 others=73.30511
mean mAP=57.98809
50 epochs
final train loss: 0.300277
0.2s mAP=66.54645 background=46.77405 wearer=71.62111 others=81.24419
0.4s mAP=60.89720 background=39.76093 wearer=64.88690 others=78.04378
0.6s mAP=56.41553 background=35.39797 wearer=58.39212 others=75.45650
0.8s mAP=53.45517 background=33.13157 wearer=53.17203 others=74.06192
1.0s mAP=51.56044 background=32.04965 wearer=49.57623 others=73.05543
1.2s mAP=50.11650 background=31.56647 wearer=46.40684 others=72.37618
1.4s mAP=48.95956 background=31.27035 wearer=44.07047 others=71.53785
1.6s mAP=48.10094 background=31.17011 wearer=42.20836 others=70.92435
1.8s mAP=47.49916 background=31.06982 wearer=41.03746 others=70.39020
2.0s mAP=46.80053 background=30.67198 wearer=39.91152 others=69.81809
mean mAP=53.03515

31
results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/metrics.csv Normal file
View File

@@ -0,0 +1,31 @@
epochs,horizon_seconds,map,background_ap,camera_wearer_ap,other_people_ap,train_loss
4,0.2,73.42020,56.72497,77.87909,85.65653,0.806973
4,0.4,68.60449,49.88131,73.26704,82.66511,0.806973
4,0.6,64.61782,44.66684,68.57660,80.61002,0.806973
4,0.8,61.53765,41.31402,64.21804,79.08089,0.806973
4,1.0,59.41820,39.34993,61.35721,77.54745,0.806973
4,1.2,58.10069,38.91573,58.45949,76.92686,0.806973
4,1.4,57.24684,38.68165,57.00022,76.05865,0.806973
4,1.6,56.19807,37.71307,55.21953,75.66162,0.806973
4,1.8,55.18182,36.99897,53.45848,75.08801,0.806973
4,2.0,54.38864,37.05013,51.37220,74.74359,0.806973
12,0.2,72.44507,55.62292,76.67027,85.04201,0.661751
12,0.4,66.95546,48.28782,70.71200,81.86658,0.661751
12,0.6,62.31847,42.64250,64.97839,79.33452,0.661751
12,0.8,58.54713,38.96725,59.03811,77.63603,0.661751
12,1.0,56.08389,37.02472,55.22792,75.99903,0.661751
12,1.2,54.58857,36.78301,51.71494,75.26775,0.661751
12,1.4,53.57918,36.51419,49.39741,74.82594,0.661751
12,1.6,52.66949,35.89852,48.01650,74.09344,0.661751
12,1.8,51.73658,35.09050,46.39426,73.72498,0.661751
12,2.0,50.95706,34.68059,44.88548,73.30511,0.661751
50,0.2,66.54645,46.77405,71.62111,81.24419,0.300277
50,0.4,60.89720,39.76093,64.88690,78.04378,0.300277
50,0.6,56.41553,35.39797,58.39212,75.45650,0.300277
50,0.8,53.45517,33.13157,53.17203,74.06192,0.300277
50,1.0,51.56044,32.04965,49.57623,73.05543,0.300277
50,1.2,50.11650,31.56647,46.40684,72.37618,0.300277
50,1.4,48.95956,31.27035,44.07047,71.53785,0.300277
50,1.6,48.10094,31.17011,42.20836,70.92435,0.300277
50,1.8,47.49916,31.06982,41.03746,70.39020,0.300277
50,2.0,46.80053,30.67198,39.91152,69.81809,0.300277
1 epochs horizon_seconds map background_ap camera_wearer_ap other_people_ap train_loss
2 4 0.2 73.42020 56.72497 77.87909 85.65653 0.806973
3 4 0.4 68.60449 49.88131 73.26704 82.66511 0.806973
4 4 0.6 64.61782 44.66684 68.57660 80.61002 0.806973
5 4 0.8 61.53765 41.31402 64.21804 79.08089 0.806973
6 4 1.0 59.41820 39.34993 61.35721 77.54745 0.806973
7 4 1.2 58.10069 38.91573 58.45949 76.92686 0.806973
8 4 1.4 57.24684 38.68165 57.00022 76.05865 0.806973
9 4 1.6 56.19807 37.71307 55.21953 75.66162 0.806973
10 4 1.8 55.18182 36.99897 53.45848 75.08801 0.806973
11 4 2.0 54.38864 37.05013 51.37220 74.74359 0.806973
12 12 0.2 72.44507 55.62292 76.67027 85.04201 0.661751
13 12 0.4 66.95546 48.28782 70.71200 81.86658 0.661751
14 12 0.6 62.31847 42.64250 64.97839 79.33452 0.661751
15 12 0.8 58.54713 38.96725 59.03811 77.63603 0.661751
16 12 1.0 56.08389 37.02472 55.22792 75.99903 0.661751
17 12 1.2 54.58857 36.78301 51.71494 75.26775 0.661751
18 12 1.4 53.57918 36.51419 49.39741 74.82594 0.661751
19 12 1.6 52.66949 35.89852 48.01650 74.09344 0.661751
20 12 1.8 51.73658 35.09050 46.39426 73.72498 0.661751
21 12 2.0 50.95706 34.68059 44.88548 73.30511 0.661751
22 50 0.2 66.54645 46.77405 71.62111 81.24419 0.300277
23 50 0.4 60.89720 39.76093 64.88690 78.04378 0.300277
24 50 0.6 56.41553 35.39797 58.39212 75.45650 0.300277
25 50 0.8 53.45517 33.13157 53.17203 74.06192 0.300277
26 50 1.0 51.56044 32.04965 49.57623 73.05543 0.300277
27 50 1.2 50.11650 31.56647 46.40684 72.37618 0.300277
28 50 1.4 48.95956 31.27035 44.07047 71.53785 0.300277
29 50 1.6 48.10094 31.17011 42.20836 70.92435 0.300277
30 50 1.8 47.49916 31.06982 41.03746 70.39020 0.300277
31 50 2.0 46.80053 30.67198 39.91152 69.81809 0.300277

73
results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/summary.json Normal file
View File

@@ -0,0 +1,73 @@
{
"schema_version": 1,
"recorded_date": "2026-08-26",
"status": "single_seed_test_inspected_diagnostic",
"dataset": "EasyCom",
"model": {
"backbone": "Qwen2.5-Omni-3B",
"thinker_frozen": true,
"memory_seconds": 8,
"streaming_mode": "open-stream-replay",
"video_fps": 2,
"prediction_hz": 5,
"readout": "last_4_query_concat_mlp",
"fusion_parameters_approx": 16795648
},
"runs": [
{
"epochs": 4,
"final_train_loss": 0.806973,
"horizon_0.2_map": 73.42020,
"mean_map": 60.87144,
"mean_class_ap": {
"background": 42.12966,
"camera_wearer": 62.08079,
"other_people": 78.40387
}
},
{
"epochs": 12,
"final_train_loss": 0.661751,
"horizon_0.2_map": 72.44507,
"mean_map": 57.98809,
"mean_class_ap": {
"background": 40.15120,
"camera_wearer": 56.70353,
"other_people": 77.10954
}
},
{
"epochs": 50,
"final_train_loss": 0.300277,
"horizon_0.2_map": 66.54645,
"mean_map": 53.03515,
"mean_class_ap": {
"background": 34.28629,
"camera_wearer": 51.12830,
"other_people": 73.69085
}
}
],
"comparisons_percent": {
"epoch4_minus_epoch12_mean_map": 2.88335,
"epoch4_minus_epoch50_mean_map": 7.83629,
"epoch4_minus_scalar_mixture_mean_map": 0.33721,
"epoch4_minus_scalar_mixture_horizon_0.2_map": -0.03065,
"epoch4_minus_single_final_layer_mean_map": 2.35207,
"epoch4_minus_single_final_layer_horizon_0.2_map": 1.42252
},
"validity": {
"test_sessions": [3, 2, 1],
"test_was_used_to_compare_epoch_budgets": true,
"same_output_path_was_reused": true,
"cosine_schedule_changes_with_total_epochs": true,
"strict_causal_valid_reported": false,
"multi_seed": false
},
"provenance": {
"source": "user-reported remote console tables",
"shared_remote_checkpoint": "checkpoints/easycom_frozen_3b_8s_last4_concat_mlp_25ep.pth",
"shared_remote_result_dir": "results/easycom_frozen_3b_8s_last4_concat_mlp_25ep",
"complete_remote_checkpoint_json_archived": false
}
}

71
results/easycom_oracle_pooled_wearer_audit_2026-09-10/README.md Normal file
View File

@@ -0,0 +1,71 @@
# Pooled AV 与 oracle 的 wearer 增益审计
日期：2026-09-10。来源：用户粘贴的远程三种子控制台结果，以及已有的
[pooled/encoder-only 记录](../easycom_encoder_only_1s_4ep_2026-08-28/README.md)。
本记录没有重新运行训练或读取远程 checkpoint/predictions。
## 协议与来源
新 oracle：Sessions 4--12 全训练，固定 4 epochs，保存末轮，无 dev checkpoint
selection；Sessions 1--3 测试；seeds 0/1/2；`causal_valid_only`。
命令入口：`bash run_easycom_oracle_information.sh final 0 1 2`。
预测根目录：`results/easycom_oracle_information_4ep_train4_12_fixed4_test/`；
checkpoint 按 runner 默认应为
`checkpoints/easycom_oracle_information_4ep_train4_12_fixed4/<condition>_seed<seed>.pth`，
但实际环境覆盖值、checkpoint hash 和运行代码 revision 未随日志提供，尚未核验。
运行契约见 [oracle 手册](../../docs/easycom_oracle_information.md)。
`metrics.csv` 保存用户提供的全部 18 行数值，单位均为 AP/mAP 百分点。
BG/wearer/others AP 三列仅对应 0.2 秒；mean mAP 是三个类别、十个 horizon 的平均，
不能当成十 horizon wearer AP。
## 当前观测
三种子均值差值（百分点）：
| 比较 | 0.2s mAP | 0.2s wearer AP | 十 horizon mean mAP |
|---|---:|---:|---:|
| aggregate-visible − audio | +4.82609 | +0.57206 | +1.77462 |
| delayed − audio | +0.08185 | +0.00647 | +0.16443 |
| visibility-only − audio | +0.07598 | −0.06329 | +0.04018 |
| per-person − shuffled-id | +0.34857 | +0.03092 | +0.20013 |
| aggregate-visible − per-person | +0.42975 | +0.22818 | +0.29128 |
aggregate 的 0.2s 宏平均增益主要来自 BG（+10.45109）和 others（+3.45514），
wearer 增益为 +0.57206。三种子方向一致，但 seed 方差不能替代跨 session 泛化的
置信区间，也不能据此声称事件级改善。
## 与历史 pooled AV 的关系
历史 seed-0 `causal_valid_only`：G+A wearer AP 85.06554，G+A+V 85.08583，
局部视觉的 0.2s wearer 增量仅 +0.02029；mean mAP 增量 +0.29083。
历史完整曲线采用另一协议 `egospeak_full_timeline`：
- G+A+V − G+A：十 horizon wearer AP 平均 +0.06078，mean mAP +0.29073。
- A+V − A：十 horizon wearer AP 平均 −0.99993；0.2s wearer AP −1.40150。
因此普通 pooled V 尚无稳定改善 wearer 的证据；encoder-only A+V 还出现 wearer
退步。历史单种子与新 oracle 三种子不是一组完全匹配实验，不能用它们直接计算
oracle-gap recovery 或判定两个方法的统计显著差异。
## 可以和不可以得出的结论
1. 已支持：在当前实验设置下，真实的可见 others 活动标注具有互补预测价值；
aggregate 条件优于 visibility-only，收益不能仅由人物可见性控制解释。
2. 及时性有证据：per-person 优于同结构 delayed 条件；aggregate 与 delayed 同时
改变了聚合方式和时间，不能单独作为纯时间干预对照。
3. 身份连续性：per-person 对 shuffled-id 有小幅同向增益，不能继续称为完全持平或
断言 identity 无用；但 wearer 增量仅 +0.03，且 aggregate 更好，不支持其必要性。
4. 尚未证明：增益来自真实 RGB 可恢复的信息，或 pooling 是失败原因。Oracle 使用
说话标注，可能提供当前视觉/音频前端根本无法可靠估计的信号；信息缺失与访问困难
目前也不能区分。
5. 尚未证明：更少抢话、addressee 理解或更好的 entry/yield。短期活动延续也能改善
BG/others AP，需要直接分析已有 wearer predictions 的事件表现。
6. Oracle 是特权信息参照，不是数学性能上界；learned latent 可以携带额外线索而超过它。
## 对后续研究的含义
优先验证从真实因果 AV 估计的局部活动是否对 wearer 提供增量，保留简单 aggregate
对照；在证据出现前，不预设 track、pooling 修复或层间注入必然有效。
新 oracle 尚缺十 horizon wearer AP 曲线与事件结果，不能仅凭 mean mAP 批准强论文主张。
Sessions 1--3 已被查看；后续选择不能依赖这次 test 表，使用训练侧验证或预先固定配置。

19
results/easycom_oracle_pooled_wearer_audit_2026-09-10/metrics.csv Normal file
View File

@@ -0,0 +1,19 @@
condition,seed,map_0p2,bg_ap_0p2,wearer_ap_0p2,others_ap_0p2,mean_map
aggregate-visible,0,86.70353,78.68879,85.97697,95.44481,66.97086
aggregate-visible,1,86.60725,78.67625,85.68389,95.46161,66.95553
aggregate-visible,2,86.77584,79.03497,85.84636,95.44621,66.92697
audio,0,81.82018,68.32374,85.13784,91.99897,65.11495
audio,1,81.91804,68.45640,85.31319,91.98453,65.23095
audio,2,81.87011,68.26659,85.34002,92.00373,65.18361
delayed,0,81.98017,68.44804,85.42083,92.07165,65.36472
delayed,1,81.85499,68.35216,85.15280,92.06000,65.37526
delayed,2,82.01874,68.81110,85.23682,92.00832,65.28282
per-person,0,86.47493,78.23009,85.85070,95.34399,66.79960
per-person,1,86.14132,77.84909,85.42206,95.15281,66.64966
per-person,2,86.18113,77.91935,85.54992,95.07412,66.53026
shuffled-id,0,86.01447,77.05511,85.83326,95.15503,66.54441
shuffled-id,1,85.85603,77.03031,85.34384,95.19396,66.47007
shuffled-id,2,85.88116,77.04115,85.55283,95.04951,66.36466
visibility-only,0,81.99974,68.56855,85.35703,92.07364,65.23904
visibility-only,1,81.83181,68.36851,85.07484,92.05209,65.24955
visibility-only,2,82.00473,68.85219,85.16930,91.99268,65.16145
1 condition seed map_0p2 bg_ap_0p2 wearer_ap_0p2 others_ap_0p2 mean_map
2 aggregate-visible 0 86.70353 78.68879 85.97697 95.44481 66.97086
3 aggregate-visible 1 86.60725 78.67625 85.68389 95.46161 66.95553
4 aggregate-visible 2 86.77584 79.03497 85.84636 95.44621 66.92697
5 audio 0 81.82018 68.32374 85.13784 91.99897 65.11495
6 audio 1 81.91804 68.45640 85.31319 91.98453 65.23095
7 audio 2 81.87011 68.26659 85.34002 92.00373 65.18361
8 delayed 0 81.98017 68.44804 85.42083 92.07165 65.36472
9 delayed 1 81.85499 68.35216 85.15280 92.06000 65.37526
10 delayed 2 82.01874 68.81110 85.23682 92.00832 65.28282
11 per-person 0 86.47493 78.23009 85.85070 95.34399 66.79960
12 per-person 1 86.14132 77.84909 85.42206 95.15281 66.64966
13 per-person 2 86.18113 77.91935 85.54992 95.07412 66.53026
14 shuffled-id 0 86.01447 77.05511 85.83326 95.15503 66.54441
15 shuffled-id 1 85.85603 77.03031 85.34384 95.19396 66.47007
16 shuffled-id 2 85.88116 77.04115 85.55283 95.04951 66.36466
17 visibility-only 0 81.99974 68.56855 85.35703 92.07364 65.23904
18 visibility-only 1 81.83181 68.36851 85.07484 92.05209 65.24955
19 visibility-only 2 82.00473 68.85219 85.16930 91.99268 65.16145

123
results/easycom_roma_frozen_last4_2026-08-27/README.md Normal file
View File

@@ -0,0 +1,123 @@
# EasyCom frozen-ROMA last-four-layer readout study
Recorded on **2026-08-27** from three user-reported remote evaluations. These
are single-seed diagnostic results on official EasyCom test Sessions 1--3, not
test-independent paper numbers.
## Shared experiment contract
- Dataset: EasyCom; EgoSpeak full-timeline evaluator on Sessions 1--3.
- Backbone checkpoint: ROMA, based on Qwen2.5-Omni-7B.
- Backbone use: frozen representation extraction; no end-to-end ROMA update.
- Streaming path: this project's 8-second Qwen-style open-stream replay, 2 FPS
video, one prediction every 200 ms.
- Saved state: final four query states, shape `[T, 4, 3584]`.
- Prediction target: ten future horizons from 0.2 to 2.0 seconds, three
EgoSpeak AP classes.
ROMA's released model is trained with one-second aligned streaming units, but
these features were extracted with this project's matched Qwen open-stream
protocol. The result therefore tests the **frozen ROMA representation under
our protocol**; it does not reproduce ROMA's own streaming training recipe.
## Main result
| Readout | Epochs | Approx. fusion parameters | 0.2s mAP | Mean mAP |
|---|---:|---:|---:|---:|
| Last-4 concat-MLP | 1 | 51.4M | 68.27145 | 58.89219 |
| Last-4 concat-MLP | 4 | 51.4M | 68.65486 | 56.52425 |
| Last-4 fixed mean | 4 | 0 | **69.21863** | **58.90402** |
The fixed uniform mean is the strongest ROMA readout. It is effectively tied
with the one-epoch concat model in mean mAP (`+0.01183`) while gaining
`+0.94718` at 0.2 seconds. The 51.4M-parameter fusion is unnecessary here.
## Per-horizon mAP
| Horizon | Concat 1ep | Concat 4ep | Fixed mean 4ep |
|---:|---:|---:|---:|
| 0.2s | 68.27145 | 68.65486 | **69.21863** |
| 0.4s | 65.29126 | 64.32941 | **65.67454** |
| 0.6s | 62.15965 | 60.41845 | **62.52327** |
| 0.8s | **59.95476** | 57.35198 | 59.92499 |
| 1.0s | **58.25407** | 55.17851 | 58.14859 |
| 1.2s | **56.83849** | 53.70716 | 56.61984 |
| 1.4s | 55.65764 | 52.59370 | **55.68495** |
| 1.6s | **54.91836** | 51.83772 | 54.52641 |
| 1.8s | **54.20725** | 50.70519 | 53.67463 |
| 2.0s | **53.36895** | 50.46558 | 53.04431 |
| **Mean** | 58.89219 | 56.52425 | **58.90402** |
## Mean class AP over ten horizons
| Readout | Background | Camera wearer | Other people |
|---|---:|---:|---:|
| Concat 1ep | 42.17735 | **57.38537** | 77.11384 |
| Concat 4ep | 40.19574 | 52.75190 | 76.62513 |
| Fixed mean 4ep | **42.19246** | 57.26979 | **77.24980** |
Extending concat training from one to four epochs raises 0.2-second mAP by
only `+0.38341` but reduces mean mAP by `-2.36794`. Camera-wearer mean AP drops
by `-4.63347`, much more than background (`-1.98161`) or other people
(`-0.48871`). This is the same overfitting pattern observed with the 3B
concat-MLP, amplified by ROMA's wider 3584-dimensional state.
Replacing concat-MLP with a fixed mean at four epochs restores `+4.51789`
camera-wearer AP and `+2.37977` mean mAP. This isolates readout capacity as the
proximate cause of the four-epoch collapse.
## Comparison with the strongest frozen Qwen2.5-Omni-3B controls
| Frozen representation/readout | 0.2s mAP | Mean mAP |
|---|---:|---:|
| Qwen-3B final-layer query, 8s | 71.99768 | 58.51937 |
| Qwen-3B last-4 scalar mixture | **73.45085** | 60.53423 |
| Qwen-3B last-4 concat-MLP, 4ep | 73.42020 | **60.87144** |
| ROMA last-4 fixed mean, 4ep | 69.21863 | 58.90402 |
| ROMA last-4 concat-MLP, 4ep | 68.65486 | 56.52425 |
The same concat design and epoch count gives a strong diagnostic: ROMA is
`-4.76534` at 0.2 seconds and `-4.34719` in mean mAP relative to Qwen-3B.
It is not a parameter-matched attribution test, because the 3584-dimensional
ROMA fusion has 51.4M parameters versus about 16.8M for Qwen-3B. Together with
the fixed-mean result, it shows that ROMA's proactive streaming adaptation is
not automatically useful for 5 Hz egocentric future-speaker anticipation, but
does not yet isolate checkpoint adaptation from width and protocol.
The fixed-mean ROMA result is not directly architecture-matched to the 3B
scalar-mixture/concat models. An exact Qwen-3B fixed-mean four-epoch control is
still required. A second clean attribution test is ROMA versus its original
Qwen2.5-Omni-7B base using the same last-four stack and fixed-mean head.
## Interpretation and decision
1. **Do not continue tuning the ROMA concat head.** Its capacity is much too
high for nine EasyCom training conversations.
2. **Do not conclude that ROMA itself is universally worse.** The experiment
changes checkpoint family and uses a protocol different from ROMA's native
one-second training units.
3. **The practical EasyCom backbone remains frozen Qwen2.5-Omni-3B.** Its
last-four readouts are stronger, cheaper, and already validated throughout
the current pipeline.
4. **The next method should change readout structure and supervision.** The
strongest hypothesis is a low-capacity semantic query state fused with
recent 200 ms payload tokens, followed by a structured future-activity
projection rather than ten independent horizon classifiers.
## Validity caveats
- Only user-reported full-timeline metrics were available; strict
causal-valid tables and remote `metrics.json` files were not archived.
- Results are single-seed and Sessions 1--3 were inspected repeatedly.
- Exact optimizer settings and the remote checkpoint filenames for all three
runs were not supplied here.
- The train split is assumed to follow the project's Sessions 4--12 contract;
verify each remote command before using the values in a paper table.
- ROMA's official one-second streaming-unit path was not reproduced.
Machine-readable values are in [`summary.json`](summary.json), and a formatted
workbook with per-horizon metrics and comparison deltas is in
[`metrics.xlsx`](metrics.xlsx).
The literature-backed diagnosis and proposed ICASSP research plan are in
[`reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md`](../../reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md).

BIN
results/easycom_roma_frozen_last4_2026-08-27/metrics.xlsx Normal file
View File

Binary file not shown.

121
results/easycom_roma_frozen_last4_2026-08-27/summary.json Normal file
View File

@@ -0,0 +1,121 @@
{
"schema_version": 1,
"recorded_date": "2026-08-27",
"status": "single_seed_user_reported_test_inspected_diagnostic",
"dataset": "EasyCom",
"evaluation": {
"protocol": "EgoSpeak full timeline",
"test_sessions": [1, 2, 3],
"prediction_hz": 5,
"future_horizons_seconds": [0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0],
"classes": ["background", "camera_wearer_speaking", "other_people_speaking"],
"n": 23048
},
"representation": {
"checkpoint_family": "ROMA based on Qwen2.5-Omni-7B",
"thinker_frozen": true,
"streaming_mode": "open-stream-replay",
"memory_seconds": 8,
"video_fps": 2,
"state": "last_four_query_states",
"state_shape": [4, 3584],
"native_roma_one_second_streaming_recipe_reproduced": false
},
"runs": [
{
"id": "roma_last4_concat_mlp_1ep",
"layer_fusion": "concat-mlp",
"epochs": 1,
"fusion_parameters_approx": 51412480,
"reported_mean_map": 58.89219,
"mean_class_ap": {
"background": 42.17735,
"camera_wearer_speaking": 57.38537,
"other_people_speaking": 77.11384
},
"per_horizon": [
{"seconds": 0.2, "background": 52.51021, "camera_wearer_speaking": 70.83688, "other_people_speaking": 81.46725, "map": 68.27145},
{"seconds": 0.4, "background": 48.71387, "camera_wearer_speaking": 67.22686, "other_people_speaking": 79.93305, "map": 65.29126},
{"seconds": 0.6, "background": 45.13005, "camera_wearer_speaking": 62.85818, "other_people_speaking": 78.49071, "map": 62.15965},
{"seconds": 0.8, "background": 42.40835, "camera_wearer_speaking": 59.85171, "other_people_speaking": 77.60421, "map": 59.95476},
{"seconds": 1.0, "background": 41.04191, "camera_wearer_speaking": 56.82736, "other_people_speaking": 76.89293, "map": 58.25407},
{"seconds": 1.2, "background": 39.74672, "camera_wearer_speaking": 54.55271, "other_people_speaking": 76.21605, "map": 56.83849},
{"seconds": 1.4, "background": 38.80110, "camera_wearer_speaking": 52.54133, "other_people_speaking": 75.63049, "map": 55.65764},
{"seconds": 1.6, "background": 38.10108, "camera_wearer_speaking": 51.26856, "other_people_speaking": 75.38544, "map": 54.91836},
{"seconds": 1.8, "background": 37.64821, "camera_wearer_speaking": 49.94810, "other_people_speaking": 75.02545, "map": 54.20725},
{"seconds": 2.0, "background": 37.67202, "camera_wearer_speaking": 47.94197, "other_people_speaking": 74.49285, "map": 53.36895}
]
},
{
"id": "roma_last4_concat_mlp_4ep",
"layer_fusion": "concat-mlp",
"epochs": 4,
"fusion_parameters_approx": 51412480,
"reported_mean_map": 56.52425,
"mean_class_ap": {
"background": 40.19574,
"camera_wearer_speaking": 52.75190,
"other_people_speaking": 76.62513
},
"per_horizon": [
{"seconds": 0.2, "background": 52.12125, "camera_wearer_speaking": 71.51813, "other_people_speaking": 82.32518, "map": 68.65486},
{"seconds": 0.4, "background": 47.23227, "camera_wearer_speaking": 65.34609, "other_people_speaking": 80.40987, "map": 64.32941},
{"seconds": 0.6, "background": 42.97315, "camera_wearer_speaking": 59.62350, "other_people_speaking": 78.65869, "map": 60.41845},
{"seconds": 0.8, "background": 39.98155, "camera_wearer_speaking": 54.81910, "other_people_speaking": 77.25528, "map": 57.35198},
{"seconds": 1.0, "background": 38.30512, "camera_wearer_speaking": 50.99168, "other_people_speaking": 76.23874, "map": 55.17851},
{"seconds": 1.2, "background": 37.25808, "camera_wearer_speaking": 48.42603, "other_people_speaking": 75.43737, "map": 53.70716},
{"seconds": 1.4, "background": 36.84681, "camera_wearer_speaking": 46.09807, "other_people_speaking": 74.83621, "map": 52.59370},
{"seconds": 1.6, "background": 36.30368, "camera_wearer_speaking": 44.70198, "other_people_speaking": 74.50749, "map": 51.83772},
{"seconds": 1.8, "background": 35.37777, "camera_wearer_speaking": 43.20047, "other_people_speaking": 73.53734, "map": 50.70519},
{"seconds": 2.0, "background": 35.55772, "camera_wearer_speaking": 42.79393, "other_people_speaking": 73.04510, "map": 50.46558}
]
},
{
"id": "roma_last4_fixed_mean_4ep",
"layer_fusion": "fixed-mean",
"epochs": 4,
"fusion_parameters_approx": 0,
"reported_mean_map": 58.90402,
"mean_class_ap": {
"background": 42.19246,
"camera_wearer_speaking": 57.26979,
"other_people_speaking": 77.24980
},
"per_horizon": [
{"seconds": 0.2, "background": 53.55709, "camera_wearer_speaking": 72.09031, "other_people_speaking": 82.00848, "map": 69.21863},
{"seconds": 0.4, "background": 49.15182, "camera_wearer_speaking": 67.68575, "other_people_speaking": 80.18605, "map": 65.67454},
{"seconds": 0.6, "background": 45.55752, "camera_wearer_speaking": 63.19862, "other_people_speaking": 78.81368, "map": 62.52327},
{"seconds": 0.8, "background": 42.39609, "camera_wearer_speaking": 59.81524, "other_people_speaking": 77.56363, "map": 59.92499},
{"seconds": 1.0, "background": 41.02210, "camera_wearer_speaking": 56.50674, "other_people_speaking": 76.91692, "map": 58.14859},
{"seconds": 1.2, "background": 39.45131, "camera_wearer_speaking": 53.93851, "other_people_speaking": 76.46970, "map": 56.61984},
{"seconds": 1.4, "background": 38.78924, "camera_wearer_speaking": 52.30126, "other_people_speaking": 75.96435, "map": 55.68495},
{"seconds": 1.6, "background": 38.08597, "camera_wearer_speaking": 50.22392, "other_people_speaking": 75.26934, "map": 54.52641},
{"seconds": 1.8, "background": 37.22956, "camera_wearer_speaking": 49.02115, "other_people_speaking": 74.77319, "map": 53.67463},
{"seconds": 2.0, "background": 36.68386, "camera_wearer_speaking": 47.91641, "other_people_speaking": 74.53267, "map": 53.04431}
]
}
],
"comparisons_percent": {
"concat_4ep_minus_concat_1ep_horizon_0.2_map": 0.38341,
"concat_4ep_minus_concat_1ep_mean_map": -2.36794,
"fixed_mean_4ep_minus_concat_4ep_horizon_0.2_map": 0.56377,
"fixed_mean_4ep_minus_concat_4ep_mean_map": 2.37977,
"fixed_mean_4ep_minus_concat_1ep_horizon_0.2_map": 0.94718,
"fixed_mean_4ep_minus_concat_1ep_mean_map": 0.01183,
"roma_concat_4ep_minus_qwen3b_concat_4ep_horizon_0.2_map": -4.76534,
"roma_concat_4ep_minus_qwen3b_concat_4ep_mean_map": -4.34719
},
"validity": {
"single_seed": true,
"test_sessions_repeatedly_inspected": true,
"strict_causal_valid_reported": false,
"remote_metrics_json_archived": false,
"exact_training_commands_archived": false,
"train_sessions_assumed_from_project_protocol": [4, 5, 6, 7, 8, 9, 10, 11, 12]
},
"provenance": {
"source": "user-reported remote console tables",
"recorded_by": "Codex",
"values_are_percent": true
}
}

12
run_easycom_oracle_information.sh
View File

@@ -146,11 +146,11 @@ screen_variants() {
--protocol causal_valid_only --protocol causal_valid_only
} }
train_all_fixed_and_test() { configure_final_protocol() {
if [[ "${EPOCHS}" != "4" ]]; then if [[ "${EPOCHS}" != "4" ]]; then
echo "The final protocol requires EPOCHS=4, got ${EPOCHS}" >&2 echo "Final protocol: overriding inherited EPOCHS=${EPOCHS} with EPOCHS=4" >&2
exit 2
fi fi
EPOCHS=4
normalized_train=() normalized_train=()
for session in "${train_sessions[@]}"; do for session in "${train_sessions[@]}"; do
@@ -166,8 +166,6 @@ train_all_fixed_and_test() {
dev_sessions=() dev_sessions=()
CHECKPOINT_ROOT="${FINAL_CHECKPOINT_ROOT:-${CHECKPOINT_ROOT}_train4_12_fixed4}" CHECKPOINT_ROOT="${FINAL_CHECKPOINT_ROOT:-${CHECKPOINT_ROOT}_train4_12_fixed4}"
RESULT_ROOT="${FINAL_RESULT_ROOT:-${RESULT_ROOT}_train4_12_fixed4}" RESULT_ROOT="${FINAL_RESULT_ROOT:-${RESULT_ROOT}_train4_12_fixed4}"
train_variants
test_selected
} }
test_selected() { test_selected() {
@@ -219,8 +217,10 @@ case "${stage}" in
screen_variants screen_variants
;; ;;
final) final)
configure_final_protocol
prepare_oracles prepare_oracles
train_all_fixed_and_test train_variants
test_selected
;; ;;
test) test)
test_selected test_selected

386
test_evaluate_floor_state_changes.py
View File

@@ -1,8 +1,13 @@
from __future__ import annotations from __future__ import annotations
import io
import json
import sys
import tempfile import tempfile
import unittest import unittest
from contextlib import redirect_stdout
from pathlib import Path from pathlib import Path
from unittest import mock
import numpy as np import numpy as np
@@ -12,7 +17,9 @@ from evaluate_floor_state_changes import (
evaluate_event_horizon, evaluate_event_horizon,
evaluate_horizon, evaluate_horizon,
load_arrays, load_arrays,
main,
parse_runs, parse_runs,
print_event_timeline_examples,
validate_recalls, validate_recalls,
) )
@@ -22,6 +29,385 @@ def activity(wearer: int, others: int) -> list[int]:
class FloorStateMetricTests(unittest.TestCase): class FloorStateMetricTests(unittest.TestCase):
def test_response_window_stops_before_next_other_onset(self) -> None:
target = np.asarray(
[
activity(0, 1), # session 0: other offset anchor
activity(0, 0),
activity(0, 1), # competing other onset; window ends before here
activity(1, 1), # must not count as response to row-1 offset
activity(0, 1), # session 1: other offset anchor
activity(0, 0),
activity(1, 0), # valid response before any competing onset
activity(1, 1), # later competitor; response has already won
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
prediction[1, 1, 0] = 0.1
prediction[2, 1, 1] = 0.2
prediction[3, 1, 2] = 0.99 # excluded by competing-risk censoring
prediction[5, 1, 0] = 0.2
prediction[6, 1, 1] = 0.9
prediction[7, 1, 2] = 0.9
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.asarray([0, 0, 0, 0, 1, 1, 1, 1], dtype=np.int32)
result = evaluate_event_horizon(
target,
prediction,
valid,
session_ids,
2,
[0.8],
midturn_pause_max_steps=10,
)
response = result["other_offset_to_wearer_response_event"]
self.assertEqual(response["num_positive"], 1)
self.assertEqual(response["num_negative"], 1)
self.assertEqual(response["num_competing_risk_censored"], 1)
self.assertEqual(response["num_short_resume_hard_negative"], 1)
self.assertAlmostEqual(response["average_precision"], 1.0)
self.assertAlmostEqual(response["min_effective_window_seconds"], 0.2)
self.assertAlmostEqual(response["max_effective_window_seconds"], 0.4)
def test_yield_window_stops_before_next_other_offset(self) -> None:
target = np.asarray(
[
activity(1, 0), # session 0: other onset anchor
activity(1, 1),
activity(1, 0), # competing other offset; window ends before here
activity(0, 0), # must not count as yield to row-1 onset
activity(1, 0), # session 1: other onset anchor
activity(1, 1),
activity(0, 1), # valid yield while other remains active
activity(0, 0), # later competitor; yield has already won
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
prediction[1, 1, 0] = 0.9
prediction[2, 1, 1] = 0.2
prediction[3, 1, 2] = 0.01 # excluded by competing-risk censoring
prediction[5, 1, 0] = 0.8
prediction[6, 1, 1] = 0.1
prediction[7, 1, 2] = 0.1
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.asarray([0, 0, 0, 0, 1, 1, 1, 1], dtype=np.int32)
result = evaluate_event_horizon(
target,
prediction,
valid,
session_ids,
2,
[0.8],
midturn_pause_max_steps=10,
)
yield_metric = result["other_onset_to_wearer_yield_event"]
self.assertEqual(yield_metric["num_positive"], 1)
self.assertEqual(yield_metric["num_negative"], 1)
self.assertEqual(yield_metric["num_competing_risk_censored"], 1)
self.assertAlmostEqual(yield_metric["average_precision"], 1.0)
point = yield_metric["operating_points"][0]
self.assertEqual(point["negative_false_positive_count"], 0)
def test_observed_response_before_recording_end_is_not_right_censored(self) -> None:
target = np.asarray(
[
activity(0, 1),
activity(0, 0),
activity(1, 0), # observed response, but nominal 0.6-s end is absent
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.zeros(len(target), dtype=np.int32)
result = evaluate_event_horizon(
target,
prediction,
valid,
session_ids,
2,
[0.8],
midturn_pause_max_steps=10,
)
response = result["other_offset_to_wearer_response_event"]
self.assertEqual(response["num_positive"], 1)
self.assertEqual(response["num_right_censored"], 0)
def test_unresolved_event_at_recording_end_is_right_censored(self) -> None:
target = np.asarray(
[
activity(0, 1),
activity(0, 0),
activity(0, 0),
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.zeros(len(target), dtype=np.int32)
result = evaluate_event_horizon(
target,
prediction,
valid,
session_ids,
2,
[0.8],
midturn_pause_max_steps=10,
)
response = result["other_offset_to_wearer_response_event"]
self.assertEqual(response["num_evaluated"], 0)
self.assertEqual(response["num_right_censored"], 1)
def test_simultaneous_wearer_entry_at_competing_onset_is_excluded(self) -> None:
target = np.asarray(
[
activity(0, 1),
activity(0, 0),
activity(1, 1), # wearer and others restart simultaneously
activity(1, 1),
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.zeros(len(target), dtype=np.int32)
result = evaluate_event_horizon(
target,
prediction,
valid,
session_ids,
2,
[0.8],
midturn_pause_max_steps=10,
)
response = result["other_offset_to_wearer_response_event"]
self.assertEqual(response["num_positive"], 0)
self.assertEqual(response["num_negative"], 1)
self.assertEqual(response["num_competing_risk_censored"], 1)
self.assertEqual(response["num_short_resume_hard_negative"], 0)
def test_timeline_marks_competing_boundary_as_excluded(self) -> None:
target = np.asarray(
[
activity(0, 1),
activity(0, 0),
activity(0, 1),
activity(1, 1),
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.zeros(len(target), dtype=np.int32)
stream = io.StringIO()
with redirect_stdout(stream):
print_event_timeline_examples(
target,
prediction,
valid,
session_ids,
["Session_1"],
examples_per_type=1,
horizon_steps=3,
)
output = stream.getvalue()
self.assertIn("status=competing-censored negative", output)
self.assertIn("C/excluded", output)
def test_fixed_calibration_threshold_is_not_reselected(self) -> None:
target = np.asarray(
[
activity(0, 1),
activity(0, 0),
activity(1, 0),
activity(1, 0),
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.zeros(len(target), dtype=np.int32)
result = evaluate_event_horizon(
target,
prediction,
valid,
session_ids,
1,
[0.8],
midturn_pause_max_steps=10,
operating_thresholds={
"response": {0.8: 0.75},
"yield": {0.8: 0.25},
},
threshold_source="calibration_sessions:Session_12",
)
point = result["other_offset_to_wearer_response_event"][
"operating_points"
][0]
self.assertEqual(point["threshold"], 0.75)
self.assertEqual(
point["threshold_source"], "calibration_sessions:Session_12"
)
self.assertEqual(point["achieved_positive_recall"], 0.0)
def test_event_score_uses_full_observable_window(self) -> None:
target = np.asarray(
[
activity(0, 1),
activity(0, 0),
activity(1, 0), # response occurs here
activity(1, 0),
],
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
prediction[1, 1, 0] = 0.1
prediction[2, 1, 1] = 0.2
prediction[3, 1, 2] = 0.9 # still part of the by-horizon event score
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
session_ids = np.zeros(len(target), dtype=np.int32)
result = evaluate_event_horizon(
target,
prediction,
valid,
session_ids,
2,
[0.8],
midturn_pause_max_steps=10,
)
response = result["other_offset_to_wearer_response_event"]
self.assertAlmostEqual(response["max_effective_window_seconds"], 0.6)
self.assertAlmostEqual(response["operating_points"][0]["threshold"], 0.9)
def test_main_calibrates_on_disjoint_training_session(self) -> None:
target = np.asarray(
[
activity(0, 1),
activity(0, 0), # response anchor
activity(1, 0), # response
activity(1, 0),
activity(1, 1), # yield anchor
activity(0, 1), # yield
activity(0, 1),
activity(0, 0), # response anchor
activity(0, 0),
activity(0, 1), # competing onset, no response
activity(1, 0), # response anchor and response
activity(1, 0),
activity(1, 1), # yield anchor
activity(1, 1),
activity(1, 0), # competing offset, no yield
]
+ [activity(0, 1)] * 15,
dtype=np.float32,
)
prediction = np.full(
(len(target), NUM_CLASSES, NUM_HORIZONS), 0.5, dtype=np.float32
)
valid = np.ones((len(target), NUM_HORIZONS), dtype=bool)
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
target_root = root / "target"
run_root = root / "run"
(run_root / "pred").mkdir(parents=True)
(run_root / "valid").mkdir(parents=True)
target_root.mkdir()
for session in ("Session_1", "Session_12"):
np.save(target_root / f"{session}.npy", target)
np.save(run_root / "pred" / f"{session}.npy", prediction)
np.save(run_root / "valid" / f"{session}.npy", valid)
output = root / "report.json"
argv = [
"evaluate_floor_state_changes.py",
"--target-root",
str(target_root),
"--run",
f"model={run_root}",
"--sessions",
"Session_1",
"--calibration-sessions",
"Session_12",
"--matched-recalls",
"0.8",
"--output",
str(output),
]
with mock.patch.object(sys, "argv", argv), redirect_stdout(io.StringIO()):
main()
payload = json.loads(output.read_text())
self.assertEqual(
payload["protocol"], "wearer_floor_event_competing_risk_v2"
)
self.assertEqual(payload["calibration_sessions"], ["Session_12"])
point = payload["runs"]["model"]["event_horizons"][1][
"other_offset_to_wearer_response_event"
]["operating_points"][0]
self.assertEqual(
point["threshold_source"],
"calibration_sessions:Session_12",
)
framewise_point = payload["runs"]["model"]["horizons"][1][
"eot_response_vs_other_hold_proxy"
]["operating_points"][0]
self.assertEqual(
framewise_point["threshold_source"],
"calibration_sessions:Session_12",
)
self.assertTrue(output.with_name("report_events.csv").is_file())
def test_main_refuses_official_test_without_calibration(self) -> None:
argv = [
"evaluate_floor_state_changes.py",
"--target-root",
"unused",
"--run",
"model=unused",
"--sessions",
"Session_1",
]
with mock.patch.object(sys, "argv", argv):
with self.assertRaisesRegex(ValueError, "requires --calibration-sessions"):
main()
def test_main_refuses_test_session_as_official_calibration(self) -> None:
argv = [
"evaluate_floor_state_changes.py",
"--target-root",
"unused",
"--run",
"model=unused",
"--sessions",
"Session_1",
"--calibration-sessions",
"Session_2",
]
with mock.patch.object(sys, "argv", argv):
with self.assertRaisesRegex(ValueError, "Sessions 4--12"):
main()
def test_event_level_counts_each_boundary_once(self) -> None: def test_event_level_counts_each_boundary_once(self) -> None:
target = np.asarray( target = np.asarray(
[ [