1069a0beee
Add cached AV pipeline for Omni turn LoRA
Xin Wang
2026-08-23 14:40:03 +08:00
2eb61e30de
Add README for Omni-only Turn-Taking Experiment detailing project goals, architecture, current status, and EasyCom results. Include comparisons with EgoSpeak and clarify dataset handling and evaluation processes.
Xin Wang
2026-08-19 23:09:56 +08:00
2e773706bd
Enhance EasyCom and Ego4D integration by adding dataset support and manifest handling. Update normalization functions to accommodate both datasets, and modify training and inference scripts to require manifest files for Ego4D. Improve argument parsing and validation for dataset-specific operations, ensuring proper session management and error handling. Update documentation to reflect changes in dataset usage and evaluation processes.
Xin Wang
2026-08-19 20:49:28 +08:00
92bf93c96b
Update README with verification status of EasyCom components and add results from recent experiments comparing open-stream replay and causal-prefix methods. Introduce new metrics CSV and detailed README for the EasyCom results, including current performance metrics and required follow-up actions before publication.
Xin Wang
2026-08-19 11:29:27 +08:00
5fa61eed23
Add extract_ego4d_batch.py for efficient EgoSpeak clip extraction. This script loads the Qwen model once and processes selected manifest rows sequentially, ensuring each clip is handled independently. Update README with usage instructions and details on the extraction process, including handling of completed outputs and memory management.
Xin Wang
2026-08-18 22:22:37 +08:00
609b2781d4
Add profiling capabilities for processor components in streaming model. Introduce _patch_component_call to time component invocations and enhance _install_profiling_hooks with new processor stages. Refactor tensor conversion and device transfer functions for improved clarity and performance. Update profiling summary to include new stages and advice for optimization.
Xin Wang
2026-08-18 11:20:55 +08:00
43f47e8f3a
Add caching mechanism for completed AV blocks in streaming model. Introduce CompletedBlockPieceCache class to manage reusable blocks during strict-window replay, enhancing efficiency. Update profiling hooks for cache operations and refine README documentation to reflect changes in block handling and replay strategies.
Xin Wang
2026-08-18 11:02:42 +08:00
d610305355
Add profiling script for GPU bottleneck analysis in EasyCom extraction. Introduce command-line options for GPU ID, log directory, and sampling period. Implement stage profiling to measure extraction performance and synchronize CUDA operations. Enhance CausalSourceAV class with new parameters for decoder threads and open readers, optimizing video decoding process.
Xin Wang
2026-08-18 10:30:19 +08:00
24cb92f1f4
Add support for NVDEC backend in video processing and enhance README documentation. Introduce a new script for extracting EasyCom sessions on specific GPUs, detailing session management and error handling. Update requirements.txt to include PyNvVideoCodec as an optional dependency. Refactor streaming_turn_model_directly.py to accommodate multiple decoder backends and improve frame indexing with ffprobe.
Xin Wang
2026-08-18 08:41:54 +08:00
e4fb62ba60
Add script for extracting EasyCom sessions with three workers per GPU. This script manages session assignments, ensures serialized worker launches, and includes error handling for session readiness and logging. Enhances the EasyCom pipeline for improved session management and feature extraction.
Xin Wang
2026-08-17 23:43:34 +08:00
d815b59def
Refactor README and streaming_turn_model.py to clarify fixed-grid frame ownership and update documentation. Remove merge conflict markers and enhance explanations regarding session launching conditions and audio slicing behavior.
Xin Wang
2026-08-17 22:59:02 +08:00
5ee371f0ae
Update README and streaming_turn_model.py to implement half-open interval ownership for fixed-grid AV frames. Adjust verification parameters to 1 for testing, and enhance documentation for clarity on audio slicing and frame selection. Add runtime checks to prevent overlapping temporal coordinates in open-stream intervals.
Xin Wang
2026-08-17 22:45:17 +08:00
8145be4108
Add scripts for extracting EasyCom session features: run_extract_easycom_direct.sh for full session extraction and run_smoke_easycom_direct.sh for smoke testing on a single GPU. Both scripts utilize streaming_turn_model_directly.py for processing and include necessary configurations and logging. This enhances the EasyCom pipeline for better session management and feature extraction.
WangXin932026-08-17 22:16:27 +08:00
de29e75734
Add scripts for preparing and extracting 5 FPS audio-video clips, including prepare_ego4d_2fps.py, run_extract_5fps_2pergpu.sh, and run_prepare_all_5fps.sh. Introduce streaming_turn_model_directly.py for direct extraction from original media. Update README.md to reflect changes in Qwen's streaming design and causal extraction modes. Add new AGENTS.md file for documentation.
Xin Wang
2026-08-17 15:17:11 +08:00
080c45f049
Add reproducible EasyCom causal head pipeline
Xin Wang
2026-08-16 22:19:43 +08:00
Xin Wang
WangXin93