Audio ML PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-18

RoughCut-Bench: A Benchmark for Automatic Video Rough-Cut with Large Vision-Language Models

Ruisi Zhao, You Li, Sihan Chen, Aosong Cheng, Chixiang Ma, Sen Pei, Zhen Yang, Hehe Fan, Dongliang He, Yi Yang

Zhejiang University +2 · arXiv

Affiliations: Zhejiang University, ByteDance, Peking University

Abstract

Video rough-cut is a core stage in media production workflows, organizing raw media materials into an initial video cut. Despite recent advances in large vision-language models (LVLMs) for video-related tasks, automatic rough-cut remains underexplored, without a comprehensive benchmark for evaluation or mature workflows for producing executable cut plans. To address the evaluation gap, we introduce RoughCut-Bench, which covers two practical settings: (1) a visually driven setting over video cases, covering highlight detection and selection, source-video ordering, and adherence to user requirements when provided; and (2) a speech-driven setting over automatic speech recognition (ASR) transcripts, covering filler removal, expression deduplication, and low-value content trimming. Across these settings, the benchmark contains 101 visually driven cases, including 831 raw videos with 1,476 highlight annotations. It also includes 84 speech-driven cases, each consisting of an ASR transcript with annotated cleanup spans. The evaluation compares cut plans with these annotations in both settings and additionally uses a human-aligned agentic evaluator to assess rendered-cut quality in the visual setting. To apply LVLMs to automatic rough-cut, we further propose agentic workflows that guide them to produce structured cut plans from input materials. Experiments show that current LVLMs remain limited in producing expert-aligned cut plans. Our benchmark, code, and prompts are released at https://whalesong-zrs.github.io/RoughCut-Bench-Project-Page.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper establishes a rigorous benchmark and evaluation framework for automatic video rough-cutting, demonstrating that current LVLMs exhibit uneven capabilities across compositional editing tasks and providing a scalable, human-aligned evaluation protocol for assessing rendered cut quality.

Methodology

The paper introduces RoughCut-Bench, a benchmark for automatic video rough-cutting, which is a practical and underexplored task in media production. The methodology is sound, defining two distinct settings: visually driven (highlight detection, ordering) and speech-driven (ASR transcript cleanup). The proposed agentic workflows (Video-Understanding Agent + Cut-Planning Agent) are logical decompositions of the complex task, addressing context length and reasoning stability issues inherent in long-video processing. The inclusion of a human-aligned agentic evaluator is a strong methodological choice, as it addresses the subjectivity of "quality" in video editing that standard IoU metrics cannot capture. The iterative refinement of the evaluator against professional scores is a rigorous approach to ensuring the automated metric is valid.

Results & evaluation

The experiments are comprehensive, evaluating 8 representative LVLMs (Gemini, GPT, Qwen, GLM, Seed) across both settings. The results provide valuable insights into the "capability dissociation" of current models, showing that strong performance in one aspect (e.g., filler removal) does not guarantee performance in another (e.g., source-video ordering). The ablation studies on frame sampling rates and input representations (time, frame, description) are well-designed and provide actionable insights for workflow optimization. The human-aligned evaluation results, showing high correlation with professional scores after refinement, validate the benchmark's utility.

Reproducibility

The paper commits to releasing code, prompts, and data, which is crucial for a benchmark paper. The detailed description of the annotation protocol (two-stage, professional annotators) and the evaluation metrics (IoU, Kendall's tau, specific precision/recall definitions for speech cleanup) enhances reproducibility. The specific model versions and workflow configurations are clearly stated.

Limitations

The benchmark size is moderate (101 visual cases, 84 speech cases), which may limit generalizability to all video types. The speech-driven setting relies on ASR transcripts, ignoring audio-visual cues like prosody or emotion, which the authors acknowledge as a limitation. The agentic workflows, while robust, may not represent the optimal performance of models that can handle longer contexts natively, as the authors prioritize cross-model comparability.

Broader impact

This work has significant potential impact on the video production industry and the development of multimodal agents. By providing a standardized benchmark and evaluation protocol, it enables fair comparison of LVLMs for creative tasks. The insights into model capabilities (e.g., the need for multi-granularity reasoning in speech cleanup) will guide future model training and architecture design. The release of professional annotations and a human-aligned evaluator will accelerate research in automated video editing. The paper establishes a rigorous benchmark and evaluation framework for automatic video rough-cutting, demonstrating that current LVLMs exhibit uneven capabilities across compositional editing tasks and providing a scalable, human-aligned evaluation protocol for assessing rendered cut quality.

Author of this paper? Send a correction.