diff --git a/README.md b/README.md index 673ac99..63afab6 100644 --- a/README.md +++ b/README.md @@ -1020,6 +1020,16 @@ process uses its `LOCAL_RANK`. DDP intentionally rejects `--online-current-tail`; the online producer-consumer path remains a separate benchmark. +To continue from a completed run, pass both its `adapter/` directory through +`--adapter-checkpoint` and its `turn_head.pt` through `--head-checkpoint` while +writing to a new output directory. This is weights-only continuation: the +adapter and head weights resume, but AdamW moments and the cosine scheduler are +reinitialized. Preserve the previous global batch when changing GPU count; +for example, change 8 GPUs with batch 2 and accumulation 1 to 4 GPUs with batch +2 and accumulation 2. Use zero warmup and a reduced learning rate for a short +post-hoc continuation because a one-epoch cosine schedule has already reached +zero. + For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU producer-consumer benchmark. The producer GPU runs the frozen AV towers and places current-tail embeddings in a bounded CPU queue while the consumer GPU @@ -1112,6 +1122,43 @@ semantics. Run the following controlled experiments: The frozen-Omni head is the cleanest representation test. LoRA can be reported as a second-stage ceiling, not as the only result. +## Candidate improvement: explicit working and long memory + +Motivated by EgoSpeak's separation of long-term and working memory, test an +explicit two-timescale representation instead of compressing the entire causal +window into one undifferentiated final Thinker state. At prediction time `t`, +align long memory to fixed global 2-second boundaries and let working memory +slide with the current prediction time: + +```text +b = floor(t / 2 s) * 2 s +long memory: complete 2-second chunks ending at b, ..., [b-4, b-2], [b-2, b] +working memory: the sliding recent interval [t-2, t], updated every 0.2 s +``` + +For example, at `t = 20.4 s`, long memory contains the fixed chunks +`[16,18]`, `[18,20]`, and any earlier retained chunks, while the working +partial is `[18.4,20.4]`. The overlap between `[18,20]` and `[18.4,20.4]` is +intentional: long memory provides a stable completed-chunk representation, +whereas working memory preserves the exact high-resolution context ending at +the current prediction time. Cache one representation per completed +long-memory chunk, recompute the working-memory representation at each 5 Hz +prediction step, and fuse the two branches with a small gated/attention +pooling module before the same `10 x 3` prediction head. Keep the Omni +backbone frozen in the first test so that any gain can be attributed to memory +organization rather than additional backbone capacity. + +This is a candidate score-improvement idea, not yet an experimental result. +The key hypothesis is that a dedicated recent branch preserves local prosody, +speaker-change, and utterance-completion cues while long memory supplies +conversation context without diluting them in one final state. Evaluate it +against matched-compute baselines: (1) the current single-state 8-second model, +(2) working-memory-only `[t-2,t]`, and (3) long+working memory with the same +total causal context. Report parameter count, throughput, and per-horizon and +per-class AP over identical seeds. A useful result requires the fused model to +beat both controls consistently; otherwise a gain could merely come from the +extra fusion parameters or the stronger recent-window shortcut. + ## Single-window feature inspection `qwen_omni_features.py` is useful for inspecting a short window or layer, but