wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reuse global history frames in overlap working memory

Browse Source
This commit is contained in:
Xin Wang
2026-08-24 11:17:39 +08:00
parent 1731f2b6f7
commit f6dbce30e4
3 changed files with 58 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
README.md
View File

@@ -1273,23 +1273,26 @@ The overlap is intentional. It presents recent evidence twice but does not add
AV-token duration beyond the matched 8-second budget. Thus this ablation trades
older unique context for stronger recent evidence instead of silently giving
the candidate more input tokens. Fixed blocks retain the global half-open 2 FPS
grid. The working recap uses an endpoint-aligned 2 FPS grid and includes the
current causal frame:
grid. The working recap reuses three causal global-grid history frames and
replaces the oldest visual slot with the current causal frame:
```text
fixed [18,20): 18.0 18.5 19.0 19.5
working [18.4,20.4]: 18.9 19.4 19.9 20.4
working [18.4,20.4]: 18.5 19.0 19.5 20.4
```
Keeping four frames means the oldest visual tick is replaced by the current
frame rather than adding a fifth frame; this isolates endpoint recency without
increasing visual tokens. Audio remains the full, exactly half-open
`[start,end)` slice. Consequently, call this an endpoint-aligned recent-video
intervention, not an exact timestamp-preserving resampling of both modalities.
The recap is serialized as the next TMRoPE chunk; it is deliberately a
recent-evidence recap, not a claim that duplicated samples occur later in
physical time. This mode is an ablation, not native Qwen streaming semantics,
and the Omni backbone stays frozen.
increasing visual tokens. Historical target times are causally floored to the
session-global 2 FPS grid. This lets them share the complete-block frame cache
and avoids making Decord seek backward through a long-GOP H.264 stream for four
new endpoint-shifted timestamps every 200 ms. Audio remains the full, exactly
half-open `[start,end)` slice. Consequently, call this a global-history plus
current-frame recent-video intervention, not an exact timestamp-preserving
resampling of both modalities. The recap is serialized as the next TMRoPE
chunk; it is deliberately a recent-evidence recap, not a claim that duplicated
samples occur later in physical time. This mode is an ablation, not native Qwen
streaming semantics, and the Omni backbone stays frozen.
One deliberate side effect is a 2-second phase cycle: overlap falls toward
0.2 seconds just before a fixed boundary and jumps to 2.0 seconds at the
@@ -1332,24 +1335,28 @@ python streaming_turn_model_directly.py \
--dtype bfloat16 \
--flash-attention-2 \
--device cuda:0 \
--verify-every 10 \
--output features/easycom_overlap_working_8s_smoke/Session_7_24s.pt
--verify-every 0 \
--output features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt
```
Expected shape is `(120, 2048)`. `overlap_cache_replay` should be numerically
exact or effectively exact. Inspect the saved timing policy with:
Expected shape is `(120, 2048)`. This speed smoke test disables replay
verification; after its throughput is acceptable, rerun a short four-second
check with `--verify-every 5`, where `overlap_cache_replay` should be
numerically exact or effectively exact. Inspect the saved timing policy and
decoder cache counters with:
```bash
python - <<'PY'
import torch
p = torch.load(
"features/easycom_overlap_working_8s_smoke/Session_7_24s.pt",
"features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt",
map_location="cpu",
weights_only=False,
)
print(p["mode"], tuple(p["states"].shape))
print(p["overlap_working_memory_policy"]["working_media"])
print(p["direct_media"]["runtime_counters"])
for row in p["overlap_working_memory_policy"]["layout"][98:102]:
print(row)
PY
@@ -1359,7 +1366,7 @@ For one complete session, replace `--end 24` with its target and use a
session-level output filename:
```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s
mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s
CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \
@@ -1383,7 +1390,7 @@ python streaming_turn_model_directly.py \
--dtype bfloat16 \
--flash-attention-2 \
--device cuda:0 \
--output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s/Session_7.pt
--output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s/Session_7.pt
```
After all train/test sessions have been extracted, train the unchanged frozen
@@ -1391,7 +1398,7 @@ head by setting only the expected feature mode:
```bash
python train_turn_head.py \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
--output checkpoints/easycom_overlap_working_frozen_8s.pt \
--expected-mode overlap-working-memory \
@@ -1400,7 +1407,7 @@ python train_turn_head.py \
--device cuda:0
python infer_turn_head.py \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
--checkpoint checkpoints/easycom_overlap_working_frozen_8s.pt \
--output-dir results/easycom_overlap_working_frozen_8s \