163 lines
7.1 KiB
Markdown
163 lines
7.1 KiB
Markdown
# Qwen partial-tail and chunk-size ablation
|
|
|
|
This runbook freezes the three-way comparison used to test whether causal
|
|
5 Hz adaptation needs an explicit partial tail. All modes use the same frozen
|
|
Qwen2.5-Omni backbone, right-edged 5 Hz output grid, 2 FPS causal video grid,
|
|
8-second memory budget, downstream head, split, and optimization.
|
|
|
|
| Mode | New media at output time `t` | 5 Hz freshness | Interpretation |
|
|
|---|---|---:|---|
|
|
| `open-stream-replay` | completed native 2 s blocks plus a temporary growing tail of 0.2--2.0 s | yes | proposed partial-tail representation |
|
|
| `completed-block-hold` | only completed native 2 s blocks; discard the incomplete tail and hold the last state | no | native-chunk stale control |
|
|
| `micro-chunk-200ms` | commit every new 0.2 s interval as an independent payload | yes | non-native micro-chunk stress control |
|
|
|
|
The 2-second hold mode does not backfill future information. Its first nine
|
|
rows (0.2--1.8 s) are zero initialization because no native block has yet
|
|
completed; the first model state appears at 2.0 s. Saved
|
|
`initialization_mask`, `state_source_times`, `state_update_mask`,
|
|
`payload_commit_mask`, and `native_boundary_mask` make this
|
|
auditable. Report a sensitivity analysis excluding initialization rows if
|
|
they have a non-negligible effect.
|
|
|
|
The micro-chunk mode is deliberately not described as native Qwen streaming.
|
|
It uses the model's physical-time TMRoPE rate to allocate five positions per
|
|
0.2-second step and refuses to run if ten micro steps do not tile one native
|
|
2-second/50-position chunk exactly. It tests whether freshness alone is
|
|
sufficient when the model's native chunk construction is removed.
|
|
|
|
Do not infer compute cost only from chunk duration. For a 2-second source
|
|
interval, partial-tail extraction prepares 0.2+...+2.0 = 11 seconds of media,
|
|
the hold control prepares 2 seconds in one call, and the micro control prepares
|
|
2 seconds across ten calls. Wall time, peak memory, processor calls, and saved
|
|
`production_prepared_media_seconds` should all be reported.
|
|
|
|
## Required short GPU smoke tests
|
|
|
|
Run each mode on the same four seconds before launching complete sessions:
|
|
|
|
```bash
|
|
cd /data1/yfzx/wangx/Code/icassp2027-experiment
|
|
|
|
MODE=open-stream-replay \
|
|
SESSION_ID=4 GPU=0 END=4 SHORT_MEMORY_SECONDS=8 \
|
|
FEATURE_ROOT=features/qwen_chunk_ablation_smoke \
|
|
LOG_ROOT=logs/qwen_chunk_ablation_smoke \
|
|
bash run_smoke_easycom_direct.sh
|
|
|
|
MODE=completed-block-hold \
|
|
SESSION_ID=4 GPU=0 END=4 SHORT_MEMORY_SECONDS=8 \
|
|
FEATURE_ROOT=features/qwen_chunk_ablation_smoke \
|
|
LOG_ROOT=logs/qwen_chunk_ablation_smoke \
|
|
bash run_smoke_easycom_direct.sh
|
|
|
|
MODE=micro-chunk-200ms \
|
|
SESSION_ID=4 GPU=0 END=4 SHORT_MEMORY_SECONDS=8 \
|
|
FEATURE_ROOT=features/qwen_chunk_ablation_smoke \
|
|
LOG_ROOT=logs/qwen_chunk_ablation_smoke \
|
|
bash run_smoke_easycom_direct.sh
|
|
```
|
|
|
|
The expected feature shape for every mode is `(20, D)`. Inspect the schedule
|
|
and processing accounting:
|
|
|
|
```bash
|
|
python - <<'PY'
|
|
from pathlib import Path
|
|
import torch
|
|
|
|
root = Path("features/qwen_chunk_ablation_smoke")
|
|
for mode in (
|
|
"open-stream-replay",
|
|
"completed-block-hold",
|
|
"micro-chunk-200ms",
|
|
):
|
|
path = root / f"Session_4_{mode}_gpu0.pt"
|
|
payload = torch.load(path, map_location="cpu", weights_only=False)
|
|
diagnostics = payload["extraction_diagnostics"]
|
|
print("\n", mode, tuple(payload["states"].shape))
|
|
print("processor calls:", diagnostics["production_processor_calls"])
|
|
print(
|
|
"prepared media seconds:",
|
|
diagnostics["production_prepared_media_seconds"],
|
|
)
|
|
print("source times:", diagnostics["state_source_times"].tolist())
|
|
print("initialization rows:", int(diagnostics["initialization_mask"].sum()))
|
|
print("state updates:", diagnostics["state_update_mask"].nonzero().flatten().tolist())
|
|
print("payload commits:", diagnostics["payload_commit_mask"].nonzero().flatten().tolist())
|
|
print("native boundaries:", diagnostics["native_boundary_mask"].nonzero().flatten().tolist())
|
|
print("verification:", payload["verification"])
|
|
PY
|
|
```
|
|
|
|
For four seconds, expected production accounting is:
|
|
|
|
| Mode | Processor calls | Prepared media seconds | State update rows | Payload commit rows |
|
|
|---|---:|---:|---|---|
|
|
| `open-stream-replay` | 20 | 22.0 | every row | 9, 19 |
|
|
| `completed-block-hold` | 2 | 4.0 | 9, 19 | 9, 19 |
|
|
| `micro-chunk-200ms` | 20 | 4.0 | every row | every row |
|
|
|
|
All three modes should report native 2-second boundaries at rows 9 and 19;
|
|
that timing fact is distinct from whether a mode updates or commits there.
|
|
|
|
`cache_replay` checks should be numerically exact or effectively exact.
|
|
`micro_chunk_vs_continuous_semantics` is a diagnostic comparison, not an
|
|
equivalence requirement: the micro representation intentionally changes input
|
|
segmentation.
|
|
|
|
## Extract all EasyCom sessions
|
|
|
|
The stable multi-session launcher already forwards `MODE`. Run the controls
|
|
separately so their outputs and logs cannot overwrite each other. Adjust GPU
|
|
IDs and roots to the remote host if needed.
|
|
|
|
```bash
|
|
cd /data1/yfzx/wangx/Code/icassp2027-experiment
|
|
|
|
PROJECT_ROOT=$PWD \
|
|
MODEL=/data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
|
VIDEO_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed \
|
|
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
|
FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/qwen_completed_block_hold_2fps_8s \
|
|
LOG_ROOT=$PWD/logs/qwen_completed_block_hold_2fps_8s \
|
|
MODE=completed-block-hold MEMORY_SECONDS=8 GPUS="0 1 2 3" \
|
|
bash run_extract_easycom_direct.sh
|
|
|
|
PROJECT_ROOT=$PWD \
|
|
MODEL=/data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
|
|
VIDEO_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed \
|
|
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
|
FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/qwen_micro_chunk_200ms_2fps_8s \
|
|
LOG_ROOT=$PWD/logs/qwen_micro_chunk_200ms_2fps_8s \
|
|
MODE=micro-chunk-200ms MEMORY_SECONDS=8 GPUS="0 1 2 3" \
|
|
bash run_extract_easycom_direct.sh
|
|
```
|
|
|
|
Each output is mode-tagged. Existing training code therefore fails closed if
|
|
the wrong directory is paired with `--expected-mode`.
|
|
|
|
## Train matched frozen heads
|
|
|
|
Use the same seeds and training hyperparameters as the partial-tail baseline.
|
|
The commands below show one matched run per control:
|
|
|
|
```bash
|
|
python train_turn_head.py \
|
|
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/qwen_completed_block_hold_2fps_8s \
|
|
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
|
--output checkpoints/easycom_qwen_completed_block_hold_8s.pt \
|
|
--expected-mode completed-block-hold \
|
|
--epochs 50 --batch-size 512 --device cuda:0
|
|
|
|
python train_turn_head.py \
|
|
--feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/qwen_micro_chunk_200ms_2fps_8s \
|
|
--target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
|
|
--output checkpoints/easycom_qwen_micro_chunk_200ms_8s.pt \
|
|
--expected-mode micro-chunk-200ms \
|
|
--epochs 50 --batch-size 512 --device cuda:0
|
|
```
|
|
|
|
This ablation is interpretable only when the partial-tail baseline is
|
|
re-extracted or otherwise confirmed to use the same 8-second memory, decoder,
|
|
2 FPS sampling, readout layers, split, seeds, and head training recipe.
|