wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reuse global history frames in overlap working memory

Browse Source
This commit is contained in:
Xin Wang
2026-08-24 11:17:39 +08:00
parent 1731f2b6f7
commit f6dbce30e4
3 changed files with 58 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
README.md
View File

@@ -1273,23 +1273,26 @@ The overlap is intentional. It presents recent evidence twice but does not add
AV-token duration beyond the matched 8-second budget. Thus this ablation trades AV-token duration beyond the matched 8-second budget. Thus this ablation trades
older unique context for stronger recent evidence instead of silently giving older unique context for stronger recent evidence instead of silently giving
the candidate more input tokens. Fixed blocks retain the global half-open 2 FPS the candidate more input tokens. Fixed blocks retain the global half-open 2 FPS
grid. The working recap uses an endpoint-aligned 2 FPS grid and includes the grid. The working recap reuses three causal global-grid history frames and
current causal frame: replaces the oldest visual slot with the current causal frame:
```text ```text
fixed [18,20): 18.0 18.5 19.0 19.5 fixed [18,20): 18.0 18.5 19.0 19.5
working [18.4,20.4]: 18.9 19.4 19.9 20.4 working [18.4,20.4]: 18.5 19.0 19.5 20.4
``` ```
Keeping four frames means the oldest visual tick is replaced by the current Keeping four frames means the oldest visual tick is replaced by the current
frame rather than adding a fifth frame; this isolates endpoint recency without frame rather than adding a fifth frame; this isolates endpoint recency without
increasing visual tokens. Audio remains the full, exactly half-open increasing visual tokens. Historical target times are causally floored to the
`[start,end)` slice. Consequently, call this an endpoint-aligned recent-video session-global 2 FPS grid. This lets them share the complete-block frame cache
intervention, not an exact timestamp-preserving resampling of both modalities. and avoids making Decord seek backward through a long-GOP H.264 stream for four
The recap is serialized as the next TMRoPE chunk; it is deliberately a new endpoint-shifted timestamps every 200 ms. Audio remains the full, exactly
recent-evidence recap, not a claim that duplicated samples occur later in half-open `[start,end)` slice. Consequently, call this a global-history plus
physical time. This mode is an ablation, not native Qwen streaming semantics, current-frame recent-video intervention, not an exact timestamp-preserving
and the Omni backbone stays frozen. resampling of both modalities. The recap is serialized as the next TMRoPE
chunk; it is deliberately a recent-evidence recap, not a claim that duplicated
samples occur later in physical time. This mode is an ablation, not native Qwen
streaming semantics, and the Omni backbone stays frozen.
One deliberate side effect is a 2-second phase cycle: overlap falls toward One deliberate side effect is a 2-second phase cycle: overlap falls toward
0.2 seconds just before a fixed boundary and jumps to 2.0 seconds at the 0.2 seconds just before a fixed boundary and jumps to 2.0 seconds at the
@@ -1332,24 +1335,28 @@ python streaming_turn_model_directly.py \
--dtype bfloat16 \ --dtype bfloat16 \
--flash-attention-2 \ --flash-attention-2 \
--device cuda:0 \ --device cuda:0 \
--verify-every 10 \ --verify-every 0 \
--output features/easycom_overlap_working_8s_smoke/Session_7_24s.pt --output features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt
``` ```
Expected shape is `(120, 2048)`. `overlap_cache_replay` should be numerically Expected shape is `(120, 2048)`. This speed smoke test disables replay
exact or effectively exact. Inspect the saved timing policy with: verification; after its throughput is acceptable, rerun a short four-second
check with `--verify-every 5`, where `overlap_cache_replay` should be
numerically exact or effectively exact. Inspect the saved timing policy and
decoder cache counters with:
```bash ```bash
python - <<'PY' python - <<'PY'
import torch import torch
p = torch.load( p = torch.load(
"features/easycom_overlap_working_8s_smoke/Session_7_24s.pt", "features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt",
map_location="cpu", map_location="cpu",
weights_only=False, weights_only=False,
) )
print(p["mode"], tuple(p["states"].shape)) print(p["mode"], tuple(p["states"].shape))
print(p["overlap_working_memory_policy"]["working_media"]) print(p["overlap_working_memory_policy"]["working_media"])
print(p["direct_media"]["runtime_counters"])
for row in p["overlap_working_memory_policy"]["layout"][98:102]: for row in p["overlap_working_memory_policy"]["layout"][98:102]:
print(row) print(row)
PY PY
@@ -1359,7 +1366,7 @@ For one complete session, replace `--end 24` with its target and use a
session-level output filename: session-level output filename:
```bash ```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s
CUDA_VISIBLE_DEVICES=3 \ CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \ OMP_NUM_THREADS=4 \
@@ -1383,7 +1390,7 @@ python streaming_turn_model_directly.py \
--dtype bfloat16 \ --dtype bfloat16 \
--flash-attention-2 \ --flash-attention-2 \
--device cuda:0 \ --device cuda:0 \
--output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s/Session_7.pt --output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s/Session_7.pt
``` ```
After all train/test sessions have been extracted, train the unchanged frozen After all train/test sessions have been extracted, train the unchanged frozen
@@ -1391,7 +1398,7 @@ head by setting only the expected feature mode:
```bash ```bash
python train_turn_head.py \ python train_turn_head.py \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s \ --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \ --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
--output checkpoints/easycom_overlap_working_frozen_8s.pt \ --output checkpoints/easycom_overlap_working_frozen_8s.pt \
--expected-mode overlap-working-memory \ --expected-mode overlap-working-memory \
@@ -1400,7 +1407,7 @@ python train_turn_head.py \
--device cuda:0 --device cuda:0
python infer_turn_head.py \ python infer_turn_head.py \
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_working_2fps_8s \ --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \ --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
--checkpoint checkpoints/easycom_overlap_working_frozen_8s.pt \ --checkpoint checkpoints/easycom_overlap_working_frozen_8s.pt \
--output-dir results/easycom_overlap_working_frozen_8s \ --output-dir results/easycom_overlap_working_frozen_8s \

5
streaming_turn_model.py
View File

@@ -1659,8 +1659,9 @@ class OverlappingWorkingMemoryExtractor:
The final fixed block and the recap intentionally overlap. The final fixed block and the recap intentionally overlap.
Fixed and working media readers may implement different visual grids. In Fixed and working media readers may implement different visual grids. In
the direct-media experiment the fixed blocks remain half-open/global-grid, the direct-media experiment the fixed blocks remain half-open/global-grid.
while the recap contains a frame sampled exactly at current time ``t``. The recap reuses global-grid historical frames but replaces its oldest
visual slot with a frame sampled exactly at current time ``t``.
""" """
def __init__( def __init__(

49
streaming_turn_model_directly.py
View File

@@ -826,15 +826,17 @@ class CausalSourceAV:
} }
class EndpointAlignedAV: class GlobalHistoryCurrentAV:
"""Working-window view whose final visual sample is exactly current time. """Working-window view with cached global history plus the current frame.
Complete cached blocks use the source reader's fixed global half-open grid, Complete cached blocks use the source reader's fixed global half-open grid,
for example ``[18.0, 18.5, 19.0, 19.5]`` for ``[18,20)`` at 2 FPS. A for example ``[18.0, 18.5, 19.0, 19.5]`` for ``[18,20)`` at 2 FPS. A
two-second working interval ending at ``t`` instead uses two-second working interval ending at ``t`` uses the latest global-grid
``[t-1.5, t-1.0, t-0.5, t]``. The number of visual samples therefore samples at or before ``t-1.5``, ``t-1.0``, and ``t-0.5``, followed by the
stays matched while the working branch explicitly observes the current current causal frame at ``t``. At ``t=20.4`` this is
causal frame. Audio remains the exact half-open slice ``[start,end)``. ``[18.5, 19.0, 19.5, 20.4]``. Historical pixels therefore reuse the same
source-frame cache as complete blocks and only the current frame advances
at 5 Hz. Audio remains the exact half-open slice ``[start,end)``.
""" """
def __init__( def __init__(
@@ -868,13 +870,16 @@ class EndpointAlignedAV:
) )
count = max(1, int(math.ceil(duration * self.fps - EPSILON))) count = max(1, int(math.ceil(duration * self.fps - EPSILON)))
count = min(count, self.max_frames) count = min(count, self.max_frames)
first = end - (count - 1) / self.fps history_times = []
if first < start - EPSILON: for slots_before_current in range(count - 1, 0, -1):
raise RuntimeError( desired_time = end - slots_before_current / self.fps
f"Endpoint-aligned grid starts before its interval: " # Quantize only historical samples to the session-global grid.
f"{first:g} < {start:g}" # During the short initial warm-up, clamp to the causal interval
) # start rather than selecting media from before the requested
return [first + index / self.fps for index in range(count)] # working window.
grid_index = int(math.floor(desired_time * self.fps + EPSILON))
history_times.append(max(start, grid_index / self.fps))
return [*history_times, end]
def frames(self, start: float, end: float) -> list[Any]: def frames(self, start: float, end: float) -> list[Any]:
frames = [ frames = [
@@ -891,19 +896,23 @@ class EndpointAlignedAV:
def metadata(self) -> dict[str, Any]: def metadata(self) -> dict[str, Any]:
return { return {
"visual_sampling": "endpoint_aligned_latest_source_pts_not_after_tick", "visual_sampling": "global_grid_history_plus_current_causal_frame",
"audio_interval": "half_open_[start,end)", "audio_interval": "half_open_[start,end)",
"working_memory_seconds": self.max_interval_seconds, "working_memory_seconds": self.max_interval_seconds,
"output_video_fps": self.fps, "output_video_fps": self.fps,
"full_window_frame_count": self.max_frames, "full_window_frame_count": self.max_frames,
"full_window_relative_video_times": [ "desired_history_offsets_seconds": [
-(self.max_frames - 1 - index) / self.fps -(self.max_frames - 1 - index) / self.fps
for index in range(self.max_frames) for index in range(self.max_frames - 1)
], ],
"current_frame_offset_seconds": 0.0,
"history_timestamp_quantization": "floor_to_session_global_grid",
"includes_current_video_frame": True, "includes_current_video_frame": True,
"oldest_visual_tick_replaced_by_current": True, "oldest_visual_tick_replaced_by_current": True,
"timestamp_note": ( "timestamp_note": (
"video is endpoint-aligned while audio retains the full " "historical video targets are quantized causally to the "
"global grid for decoder/cache reuse; the final video sample "
"is exactly current time while audio retains the full "
"half-open working interval" "half-open working interval"
), ),
} }
@@ -1384,7 +1393,7 @@ def main(
show_progress=not args.no_progress, show_progress=not args.no_progress,
) )
else: else:
working_grid_av = EndpointAlignedAV( working_grid_av = GlobalHistoryCurrentAV(
reader, reader,
max_interval_seconds=args.working_memory_seconds, max_interval_seconds=args.working_memory_seconds,
) )
@@ -1427,7 +1436,7 @@ def main(
if args.mode == "open-stream-replay" if args.mode == "open-stream-replay"
else ( else (
"serialized_cached_complete_blocks_then_overlapping_" "serialized_cached_complete_blocks_then_overlapping_"
"endpoint_aligned_recent_recap" "global_history_current_frame_recent_recap"
) )
) )
), ),
@@ -1444,7 +1453,7 @@ def main(
"source_manifest_record": manifest_record, "source_manifest_record": manifest_record,
"direct_media": reader.metadata(), "direct_media": reader.metadata(),
"video_sampling_grid": ( "video_sampling_grid": (
"fixed_global_half_open_plus_endpoint_aligned_working_recap" "fixed_global_half_open_plus_global_history_current_frame_recap"
if args.mode == "overlap-working-memory" if args.mode == "overlap-working-memory"
else "direct_pts_causal_global_grid_half_open_with_tail_duplication" else "direct_pts_causal_global_grid_half_open_with_tail_duplication"
), ),