MiniCPM-o 4.5 causal global features

streaming_minicpmo_features_directly.py uses MiniCPM-o 4.5 only as a frozen audio--visual feature extractor. It never calls streaming_generate, never loads TTS, and never lets model-generated speech/listen tokens enter the conversation history.

Frozen modes

`native-1hz-hold`

The model commits one input-only AV chunk per completed second.
The newest completed state is forward-held onto the 5 Hz target grid.
The first four rows (0.2--0.8 s) use an all-zero initialization. The 1.0-s state is never backfilled into those rows.
state_source_times, native_update_mask, and initialization_mask make this alignment auditable.

`cached-partial-5hz`

Completed one-second chunks remain in the official MiniCPM streaming cache.
At a non-boundary 200-ms cutoff, the extractor snapshots the LLM cache, audio-encoder cache, streaming Mel buffer, chunk index, and RNG state through MiniCPM's official speculative-state API.
It appends only the current half-open AV tail, reads the final normalized Qwen3 hidden state, and restores the snapshot. Temporary closure never enters the main cache.
Native commits retain MiniCPM's normal streaming-window policy. Destructive round pruning is suspended only inside the temporary transaction because the upstream snapshot can truncate appended KV tokens but cannot recreate a previously pruned round.
At a one-second boundary, a complete native chunk is committed.

The 5 Hz branch is saved as the normal mode=causal-prefix contract only after all mandatory checks pass:

a temporary full chunk matches the native committed state at a one-second boundary;
a clean stream replay matches the cached result;
changing audio strictly after the cutoff leaves the current state exactly unchanged;
snapshot restoration leaves LLM/audio cache lengths and checksums, chunk history, position offset, and the audio chunk index unchanged.

With --allow-unverified-partial-tail, a failed or too-short audit may be saved only as mode=minicpmo-unverified-partial-tail; the default head loader rejects that artifact.

Environment

The inspected upstream revision requires Python 3.10 and transformers==4.51.0. Use a dedicated environment rather than changing the working Qwen environment.

python -m pip install \
  "transformers==4.51.0" \
  accelerate \
  "torch>=2.3.0,<=2.8.0" \
  "torchaudio<=2.8.0" \
  "minicpmo-utils>=1.0.5" \
  "qwen-omni-utils[decord]"

The commands below use the upstream revision inspected during implementation, 1f761131fa83f5ed3cd6f2f22b225c4501d154fa. If the model is already stored locally, replace --model-id with that snapshot directory and record its immutable revision separately.

Required smoke sequence

First validate the source timeline and decoder without loading MiniCPM:

python streaming_minicpmo_features_directly.py \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
  --end 3 \
  --mode native-1hz-hold \
  --decoder decord \
  --media-only

Then extract three seconds with the native held-state baseline:

CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
  --model-id openbmb/MiniCPM-o-4_5 \
  --revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
  --end 3 \
  --mode native-1hz-hold \
  --dtype bfloat16 \
  --attention sdpa \
  --device cuda:0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_native_3s.pt

Run the verified cached partial-tail smoke test separately:

CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
  --model-id openbmb/MiniCPM-o-4_5 \
  --revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
  --end 3 \
  --mode cached-partial-5hz \
  --verify-every 1 \
  --future-intervention-every 1 \
  --dtype bfloat16 \
  --attention sdpa \
  --device cuda:0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_partial_3s.pt

Do not add --allow-unverified-partial-tail to a paper run. A verification failure is a protocol failure to diagnose, not a tolerance to bypass.

Inspect both artifacts before a full session:

python - <<'PY'
from pathlib import Path
import torch

root = Path("/data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke")
for name in ("Session_4_native_3s.pt", "Session_4_partial_3s.pt"):
    payload = torch.load(root / name, map_location="cpu", weights_only=False)
    print(name)
    print("  mode:", payload["mode"], payload["extractor_mode"])
    print("  states:", tuple(payload["states"].shape))
    print("  times:", payload["times"][:6].tolist())
    print("  source times:", payload["state_source_times"][:6].tolist())
    print("  verified:", payload["partial_tail_admissible"])
    print("  checks:", payload["verification"])
PY

Expected alignment:

native times:        0.2 0.4 0.6 0.8 1.0 1.2 ...
native source time:  0.0 0.0 0.0 0.0 1.0 1.0 ...
partial source time: 0.2 0.4 0.6 0.8 1.0 1.2 ...

Full Session 4

Native 1 Hz held features:

mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold

CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
  --model-id openbmb/MiniCPM-o-4_5 \
  --revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
  --target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
  --mode native-1hz-hold \
  --dtype bfloat16 \
  --attention sdpa \
  --device cuda:0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --frame-cache-size 32 \
  --output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold/Session_4.pt

Verified 5 Hz cached partial-tail features:

mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz

CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
  --model-id openbmb/MiniCPM-o-4_5 \
  --revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
  --target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
  --mode cached-partial-5hz \
  --dtype bfloat16 \
  --attention sdpa \
  --device cuda:0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --frame-cache-size 32 \
  --output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz/Session_4.pt

The full run still executes the first clean-replay and future-intervention checks plus every one-second boundary check. Add --verify-every 250 and --future-intervention-every 250 for periodic 50-second audits; smaller values make long sessions increasingly expensive because a clean audit replays the prefix from the session start.

After extraction, either directory can be passed directly as train_turn_head.py --feature-root ... because verified artifacts retain the existing dense global-feature contract.

7.4 KiB Raw Blame History