wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add causal MiniCPM-o streaming feature extraction

Browse Source
This commit is contained in:
Xin Wang
2026-09-17 01:08:50 +08:00
parent 45facc8bf4
commit 6d0c8b733b
5 changed files with 1533 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
PROJECT_CONTEXT.md
View File

@@ -60,6 +60,15 @@ result.
the original wearer probabilities, excludes wearer backchannels from
headline ENTRY FPR, and records unanchored/ambiguous cases.
### Implemented but not GPU-verified
- MiniCPM-o 4.5 input-only feature extraction has two unmeasured modes: a
native 1 Hz state causally held on the 5 Hz grid, and a 5 Hz completed-cache
plus temporary-partial-tail branch. The latter refuses the normal
head-loading contract unless boundary, replay, snapshot-restoration, and
future-audio checks pass. A real-GPU smoke run is still required; see
[`docs/minicpmo_streaming_features.md`](docs/minicpmo_streaming_features.md).
### Measured screening result
- The released EgoSpeak MiniROADA A+V checkpoint on Sessions 1--3 obtains

1
README.md
View File

@@ -121,6 +121,7 @@ change what the Stage-0 result measures.
| `easycom_manifest.py` | Verified on EasyCom | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
| `streaming_turn_model.py` | Verified on EasyCom | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
| `streaming_minicpmo_features_directly.py` | Implemented; real-GPU smoke pending | Extract input-only MiniCPM-o 4.5 native 1 Hz held states or audited 5 Hz cache-plus-partial-tail states |
| `TurnPredictionHead` | Trained and evaluated | Map `[2048]` to `[10, 3]` logits |
| `train_turn_head.py` | Verified on EasyCom | Train the frozen-Omni anticipation head with the official EasyCom split |
| `infer_turn_head.py` | Verified on EasyCom | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |

204
docs/minicpmo_streaming_features.md Normal file
View File

@@ -0,0 +1,204 @@
# MiniCPM-o 4.5 causal global features
`streaming_minicpmo_features_directly.py` uses MiniCPM-o 4.5 only as a frozen
audio--visual feature extractor. It never calls `streaming_generate`, never
loads TTS, and never lets model-generated speech/listen tokens enter the
conversation history.
## Frozen modes
### `native-1hz-hold`
- The model commits one input-only AV chunk per completed second.
- The newest completed state is forward-held onto the 5 Hz target grid.
- The first four rows (0.2--0.8 s) use an all-zero initialization. The 1.0-s
state is never backfilled into those rows.
- `state_source_times`, `native_update_mask`, and `initialization_mask` make
this alignment auditable.
### `cached-partial-5hz`
- Completed one-second chunks remain in the official MiniCPM streaming cache.
- At a non-boundary 200-ms cutoff, the extractor snapshots the LLM cache,
audio-encoder cache, streaming Mel buffer, chunk index, and RNG state through
MiniCPM's official speculative-state API.
- It appends only the current half-open AV tail, reads the final normalized
Qwen3 hidden state, and restores the snapshot. Temporary closure never enters
the main cache.
- Native commits retain MiniCPM's normal streaming-window policy. Destructive
round pruning is suspended only inside the temporary transaction because the
upstream snapshot can truncate appended KV tokens but cannot recreate a
previously pruned round.
- At a one-second boundary, a complete native chunk is committed.
The 5 Hz branch is saved as the normal `mode=causal-prefix` contract only after
all mandatory checks pass:
1. a temporary full chunk matches the native committed state at a one-second
boundary;
2. a clean stream replay matches the cached result;
3. changing audio strictly after the cutoff leaves the current state exactly
unchanged;
4. snapshot restoration leaves LLM/audio cache lengths and checksums, chunk
history, position offset, and the audio chunk index unchanged.
With `--allow-unverified-partial-tail`, a failed or too-short audit may be
saved only as `mode=minicpmo-unverified-partial-tail`; the default head loader
rejects that artifact.
## Environment
The inspected upstream revision requires Python 3.10 and
`transformers==4.51.0`. Use a dedicated environment rather than changing the
working Qwen environment.
```bash
python -m pip install \
"transformers==4.51.0" \
accelerate \
"torch>=2.3.0,<=2.8.0" \
"torchaudio<=2.8.0" \
"minicpmo-utils>=1.0.5" \
"qwen-omni-utils[decord]"
```
The commands below use the upstream revision inspected during implementation,
`1f761131fa83f5ed3cd6f2f22b225c4501d154fa`. If the model is already stored
locally, replace `--model-id` with that snapshot directory and record its
immutable revision separately.
## Required smoke sequence
First validate the source timeline and decoder without loading MiniCPM:
```bash
python streaming_minicpmo_features_directly.py \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--end 3 \
--mode native-1hz-hold \
--decoder decord \
--media-only
```
Then extract three seconds with the native held-state baseline:
```bash
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--end 3 \
--mode native-1hz-hold \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_native_3s.pt
```
Run the verified cached partial-tail smoke test separately:
```bash
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--end 3 \
--mode cached-partial-5hz \
--verify-every 1 \
--future-intervention-every 1 \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_partial_3s.pt
```
Do not add `--allow-unverified-partial-tail` to a paper run. A verification
failure is a protocol failure to diagnose, not a tolerance to bypass.
Inspect both artifacts before a full session:
```bash
python - <<'PY'
from pathlib import Path
import torch
root = Path("/data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke")
for name in ("Session_4_native_3s.pt", "Session_4_partial_3s.pt"):
payload = torch.load(root / name, map_location="cpu", weights_only=False)
print(name)
print(" mode:", payload["mode"], payload["extractor_mode"])
print(" states:", tuple(payload["states"].shape))
print(" times:", payload["times"][:6].tolist())
print(" source times:", payload["state_source_times"][:6].tolist())
print(" verified:", payload["partial_tail_admissible"])
print(" checks:", payload["verification"])
PY
```
Expected alignment:
```text
native times: 0.2 0.4 0.6 0.8 1.0 1.2 ...
native source time: 0.0 0.0 0.0 0.0 1.0 1.0 ...
partial source time: 0.2 0.4 0.6 0.8 1.0 1.2 ...
```
## Full Session 4
Native 1 Hz held features:
```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
--mode native-1hz-hold \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--frame-cache-size 32 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold/Session_4.pt
```
Verified 5 Hz cached partial-tail features:
```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
--mode cached-partial-5hz \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--frame-cache-size 32 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz/Session_4.pt
```
The full run still executes the first clean-replay and future-intervention
checks plus every one-second boundary check. Add `--verify-every 250` and
`--future-intervention-every 250` for periodic 50-second audits; smaller values
make long sessions increasingly expensive because a clean audit replays the
prefix from the session start.
After extraction, either directory can be passed directly as
`train_turn_head.py --feature-root ...` because verified artifacts retain the
existing dense global-feature contract.

1089
streaming_minicpmo_features_directly.py Normal file
View File

File diff suppressed because it is too large Load Diff

230
test_streaming_minicpmo_features_directly.py Normal file
View File

@@ -0,0 +1,230 @@
"""CPU-only contract tests for MiniCPM-o streaming feature schedules."""
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
import numpy as np
import torch
from easycom_head import load_session_data
from local_turn_features import reference_alignment
from streaming_minicpmo_features_directly import (
FEATURE_FORMAT,
MiniCPMFeatureExtractor,
MiniCPMStreamConfig,
_comparison,
_time_grid,
validate_transformers_version,
)
class FakeReader:
def __init__(self, duration: float = 3.0) -> None:
self.audio = np.arange(
int(duration * 16_000), dtype=np.float32
) / 16_000.0
def frame_at_or_before(self, timestamp: float) -> float:
return float(timestamp)
def audio_slice(self, start: float, end: float) -> np.ndarray:
first = int(round(start * 16_000))
last = int(round(end * 16_000))
return self.audio[first:last].copy()
class FakeEngine:
hidden_size = 3
def __init__(self) -> None:
self.committed_until = 0.0
self.reset_count = 0
self.last_restore_audit = None
def reset(self) -> None:
self.committed_until = 0.0
self.reset_count += 1
@staticmethod
def _state(frame: float, audio: np.ndarray) -> torch.Tensor:
return torch.tensor(
[[float(audio[-1]), float(frame), float(len(audio))]],
dtype=torch.float32,
)
def commit_state(self, *, frame: float, audio: np.ndarray) -> torch.Tensor:
self.committed_until += len(audio) / 16_000.0
return self._state(frame, audio)
def temporary_state(self, *, frame: float, audio: np.ndarray) -> torch.Tensor:
before = self.committed_until
state = self._state(frame, audio)
if self.committed_until != before:
raise AssertionError("temporary state modified the fake main cache")
self.last_restore_audit = {
"kind": "snapshot_restore",
"llm_cache_length": 0,
"audio_cache_length": 0,
"audio_chunk_index": 0,
"streaming_position_offset": 0,
"window_pruning_suspended": True,
"window_policy_restored": True,
"llm_checksum_matches": True,
"audio_checksum_matches": True,
"history_matches": True,
"passed": True,
}
return state
class MiniCPMStreamingTests(unittest.TestCase):
def setUp(self) -> None:
self.config = MiniCPMStreamConfig(
boundary_max_abs=0.0,
boundary_min_cosine=1.0,
replay_max_abs=0.0,
replay_min_cosine=1.0,
)
def test_time_grid_is_right_edged_and_rejects_fractional_tail(self) -> None:
self.assertEqual(_time_grid(0.0, 1.0, 0.2), [0.2, 0.4, 0.6, 0.8, 1.0])
with self.assertRaisesRegex(ValueError, "integer number"):
_time_grid(0.0, 1.1, 0.2)
def test_native_hold_has_no_future_backfill(self) -> None:
extractor = MiniCPMFeatureExtractor(FakeReader(), FakeEngine(), self.config)
result = extractor.extract_native_hold(
start=0.0,
end=2.0,
show_progress=False,
)
self.assertEqual(tuple(result.states.shape), (10, 3))
self.assertTrue(torch.equal(result.states[:4], torch.zeros(4, 3)))
self.assertTrue(result.initialization_mask[:4].all())
self.assertFalse(result.initialization_mask[4:].any())
self.assertEqual(
result.native_update_mask.nonzero().flatten().tolist(),
[4, 9],
)
self.assertTrue(torch.equal(result.states[4], result.states[8]))
self.assertFalse(torch.equal(result.states[8], result.states[9]))
self.assertEqual(
result.state_source_times.tolist(),
[0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0],
)
def test_cached_partial_updates_every_row_and_passes_audits(self) -> None:
reader = FakeReader()
extractor = MiniCPMFeatureExtractor(reader, FakeEngine(), self.config)
original_audio = reader.audio.copy()
result = extractor.extract_cached_partial(
start=0.0,
end=2.0,
verify_every=0,
future_intervention_every=0,
allow_unverified=False,
show_progress=False,
)
self.assertTrue(result.partial_tail_admissible)
self.assertTrue(np.array_equal(reader.audio, original_audio))
self.assertFalse(result.initialization_mask.any())
self.assertTrue(torch.equal(result.state_source_times, result.times))
self.assertEqual(
result.native_update_mask.nonzero().flatten().tolist(),
[4, 9],
)
self.assertEqual(
{item["kind"] for item in result.verification},
{
"native_boundary_equivalence",
"clean_stream_replay",
"future_audio_intervention",
"snapshot_restore",
},
)
self.assertTrue(all(item["passed"] for item in result.verification))
self.assertFalse(torch.equal(result.states[0], result.states[1]))
self.assertLess(float(result.states[0, 1]), 0.2)
def test_short_partial_run_is_rejected_without_boundary_audit(self) -> None:
extractor = MiniCPMFeatureExtractor(FakeReader(), FakeEngine(), self.config)
with self.assertRaisesRegex(RuntimeError, "missing=.*native_boundary"):
extractor.extract_cached_partial(
start=0.0,
end=0.8,
verify_every=0,
future_intervention_every=0,
allow_unverified=False,
show_progress=False,
)
def test_comparison_rejects_different_states(self) -> None:
item = _comparison(
torch.zeros(1, 2),
torch.ones(1, 2),
kind="test",
current_time=0.2,
max_abs_tolerance=0.0,
min_cosine=1.0,
)
self.assertFalse(item["passed"])
self.assertEqual(item["max_abs"], 1.0)
def test_transformers_version_is_explicitly_gated(self) -> None:
self.assertEqual(validate_transformers_version("4.51.0"), "4.51.0")
with self.assertRaisesRegex(RuntimeError, "found 4.52.0"):
validate_transformers_version("4.52.0")
self.assertEqual(
validate_transformers_version("4.52.0", allow_untested=True),
"4.52.0",
)
def test_saved_payload_loads_in_existing_head_and_unverified_fails(self) -> None:
states = torch.randn(20, 8)
payload = {
"format": FEATURE_FORMAT,
"states": states,
"times": (torch.arange(20, dtype=torch.float64) + 1) / 5,
"label_indices": torch.arange(20),
"mode": "causal-prefix",
"extractor_mode": "native-1hz-hold",
"config": {"step_seconds": 0.2},
}
targets = np.tile(
np.asarray(
[[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 1, 1]],
dtype=np.float32,
),
(5, 1),
)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
feature_root = root / "features"
target_root = root / "targets"
feature_root.mkdir()
target_root.mkdir()
feature_path = feature_root / "Session_4.pt"
torch.save(payload, feature_path)
np.save(target_root / "Session_4.npy", targets)
loaded = load_session_data(feature_root, target_root, 4)
self.assertTrue(torch.equal(loaded.states, states))
times, indices = reference_alignment(payload)
self.assertTrue(torch.equal(times, payload["times"]))
self.assertTrue(torch.equal(indices, payload["label_indices"]))
bad_reference = dict(payload)
bad_reference["times"] = payload["times"] + 0.2
with self.assertRaisesRegex(ValueError, "Reference times"):
reference_alignment(bad_reference)
payload["mode"] = "minicpmo-unverified-partial-tail"
torch.save(payload, feature_path)
with self.assertRaisesRegex(ValueError, "expected 'causal-prefix'"):
load_session_data(feature_root, target_root, 4)
if __name__ == "__main__":
unittest.main()