3.8 KiB
EasyCom: open-stream replay versus causal-prefix
Recorded on 2026-08-19. These are the current single-run results reported from the remote EasyCom experiments. They are an experiment snapshot, not yet the final multi-seed paper numbers.
Experiment contract
- Dataset: EasyCom using EgoSpeak's official split.
- Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
- Test sessions: 3, 2, and 1.
- Model: frozen Qwen2.5-Omni-3B Thinker plus the trainable
10 x 3head. - Prediction rate: 5 Hz, one state every 0.2 seconds.
- Video sampling: 2 FPS.
- Causal context budget: 20 seconds.
- Targets: ten future horizons from 0.2 through 2.0 seconds.
- Metric: frame AP for all three classes, followed by per-horizon mAP and the mean over all ten horizons.
The two representations are trained and evaluated separately. A head trained on one representation must not be reused on the other.
Summary
All values below are percentages.
| Representation | EgoSpeak full-timeline mean mAP | Strict causal mean mAP | Difference from open stream |
|---|---|---|---|
| Open-stream replay | 58.316 | 58.326 | — |
| Causal-prefix | 58.830 | 58.841 | +0.514 / +0.515 |
For context, the EgoSpeak Table 2 EasyCom reference values currently recorded in the project README are 58.7 for Transformer A+V and 60.6 for GRU A+V. Accordingly, causal-prefix is 0.130 points above the Transformer reference, whereas open-stream replay is 0.384 points below it. Neither reaches the GRU A+V result.
Full-timeline result by horizon
| Horizon | Time | Open-stream mAP | Causal-prefix mAP | Causal minus open |
|---|---|---|---|---|
| 1 | 0.2 s | 69.756 | 72.149 | +2.393 |
| 2 | 0.4 s | 65.163 | 66.904 | +1.741 |
| 3 | 0.6 s | 61.578 | 62.468 | +0.890 |
| 4 | 0.8 s | 58.783 | 59.214 | +0.431 |
| 5 | 1.0 s | 56.913 | 57.037 | +0.124 |
| 6 | 1.2 s | 55.727 | 55.789 | +0.062 |
| 7 | 1.4 s | 54.819 | 54.845 | +0.026 |
| 8 | 1.6 s | 53.957 | 53.905 | -0.052 |
| 9 | 1.8 s | 53.459 | 53.265 | -0.194 |
| 10 | 2.0 s | 53.002 | 52.723 | -0.279 |
The complete per-class AP values for both evaluation protocols are stored in
metrics.csv. summary.json contains the
machine-readable aggregate comparison.
Current interpretation
- Causal-prefix is 0.514 mAP points better overall in the exact EgoSpeak full-timeline protocol.
- Almost all of that advantage occurs at 0.2--0.6 seconds. At 1.2--1.4 seconds the difference is negligible, and open-stream is slightly better at 1.6--2.0 seconds.
- The full-timeline and strict-causal means differ by only 0.010--0.011 points, so unavailable pre-session rows are not driving the comparison.
- The short-horizon pattern is consistent with block-local audio processing, partial-block AV/TMRoPE organization, or other 2-second boundary effects weakening precise speech-onset/offset cues. It does not look like a general loss of longer-horizon conversational information.
- Previous smoke tests showed exact open-stream cache replay but a lower cosine against the continuous-window representation. This supports a real representation-semantics difference rather than a cache-copy error.
Required follow-up before publication
- Repeat head training with at least three identical seeds for both modes and report mean and standard deviation.
- Group feature cosine and AP/mAP by the ten 0.2-second phases inside each 2-second block to test for a block-boundary effect.
- Compare samples before and after the 20-second memory budget fills to isolate input segmentation from rolling-memory behavior.
- Archive the original remote
metrics.json, checkpoint configuration, git commit, Transformers version, and Qwen model revision alongside the final paper results. The numbers in this snapshot were transcribed from the reported console tables.