StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding

From short clips to always-on multimodal agents

Xichen Zhang^1,2,*Guankai Li^2,*Yinghao Zhu³Shijian Wang²Sitong Wu⁴Shaozuo Yu⁴Meng Chu¹Yuan Lu²Jiaya Jia^1,†

¹ HKUST² Xiaohongshu Inc.³ The University of Hong Kong⁴ The Chinese University of Hong Kong

^* Equal contribution · ^† Corresponding author

Overview

Evaluating agents that never stop watching

StreamArena moves streaming-video evaluation beyond short clips and multiple-choice shortcuts toward continuous, causal, and interactive understanding.

Overview of StreamArena dataset statistics and its four evaluated capabilities — StreamArena covers seven real-world video domains and four capabilities, with every query and supporting evidence segment causally timestamped.

Why StreamArena?

Real-world multimodal agents must continuously ingest unbounded audio and video, remember what happened long ago, interact at low latency, and decide when to use tools or speak proactively. Existing benchmarks usually test only fragments of this behavior.

StreamArena provides full-length streams and open-ended tasks under strict causal access, exposing the trade-off between responsive interaction and persistent multimodal understanding.

Highlights

Hour-scale streams instead of isolated short clips.
Open-ended answers without answer-option cues.
Dual temporal grounding for queries and evidence.
Four deployment capabilities evaluated together.

243Full-length videos

88.8 minAverage duration

3,646Validated open-ended tasks

7Real-world domains

Abstract

Continuous video understanding as a system

Deploying autonomous multimodal agents in continuous, real-world environments requires them to ingest unbounded audio-visual streams and maintain hour-scale memory. Yet current evaluations predominantly rely on brief clips and multiple-choice formats, where recent-frame baselines and language shortcuts can hide fundamental limitations.

We introduce StreamArena, a benchmark for hour-scale, interactive streaming video understanding. It contains 243 full-length videos averaging 88.8 minutes and 3,646 rigorously annotated, open-ended question-answer pairs spanning real-time perception, historical retrospection, proactive interaction, and multimodal tool utilization.

We further propose StreamMind, a two-tier architecture that assigns latency-critical interaction and monitoring to frontend workers while backend workers asynchronously build persistent multimodal memory, retrieve historical evidence, and perform external search. StreamMind leads streaming baselines across all four capabilities while reducing query-to-answer latency through persistent state reuse.

The Benchmark

Four capabilities for always-on agents

Each capability targets a distinct failure mode that static, turn-based evaluation cannot fully reveal.

01RTP

Real-Time Perception

Reason over synchronized audio and video while events are still unfolding.

263 tasks

02HR

Historical Retrospection

Retrieve distant evidence from an hour-scale stream without seeing the future.

877 tasks

03TOOL

Multimodal Tool Use

Ground visible entities, identify information gaps, and search beyond memory.

1,732 tasks

04PRO

Proactive Interaction

Monitor future conditions and speak at the right moment without a new prompt.

774 tasks

01Draft

PhD-level annotators create multi-turn tasks.

02Cross-validate

Two annotators independently answer and correct.

03Blind audit

A third annotator verifies content and timestamps.

Benchmark Comparison

What StreamArena evaluates together

Existing benchmarks typically isolate one or two streaming abilities. StreamArena combines hour-scale video, causal access, multimodal input, dialogue, proactive triggering, and external tools in one protocol.

Comparison with existing video-understanding benchmarks
Benchmark	Venue	Anno.	Answer	Q/Vid	Dur. (min)	Long	Stream	Omni	MT	Pro.	Tool
Offline long / omni-modal video benchmarks
Video-MME	CVPR ’25	Manual	MC	3	17.0	✓	—	✓	—	—	—
LongVideoBench	NeurIPS ’24	Manual	MC	2	7.9	✓	—	—	—	—	—
MLVU	CVPR ’25	Manual	Mixed	2	15.5	✓	—	—	—	—	—
Online / streaming video benchmarks
StreamingBench‡	ICLR ’25	Hybrid	MC	5	9.7	—	✓	✓	✓	◐	—
OVO-Bench‡	CVPR ’25	Hybrid	Mixed	4	3.5	—	✓	—	—	◐	—
OVBench	CVPR ’25	Hybrid	MC	11	5.5	—	✓	—	✓	—	—
RTV-Bench	NeurIPS ’25	Hybrid	MC	8	18.2	—	✓	—	—	—	—
OST-Bench	NeurIPS ’25	Hybrid	Mixed	7	N/A	—	✓	—	—	—	—
Proactive / duplex streaming benchmarks
OmniMMI	CVPR ’25	Manual	Mixed	2	5.4	—	✓	✓	✓	✓	—
QIVD (Qualcomm)	ICLR ’26	Manual	Open	1	0.09	—	✓	✓	—	✓	—
Ours
StreamArena	—	Manual	Open	≈15	88.8	✓	✓	✓	✓	✓	✓

◐ denotes partial proactive support through separate queries rather than continuous monitoring. ‡ marks benchmarks with documented recency shortcuts. Values follow the paper comparison table.

Method

StreamMind: responsive interaction, persistent memory

A two-tier architecture keeps interaction responsive while long-horizon cognition runs asynchronously.

StreamMind architecture with frontend workers, backend workers, and persistent multimodal memory — StreamMind separates continuous ingestion, proactive monitoring, memory construction, historical recall, and external search into independently scheduled workers.

Responsive frontend

The Front Worker answers from recent context, delegates retrieval, or launches independent Monitor Workers for future conditions.

Persistent memory

The Memory Writer organizes events hierarchically while preserving entity relations and retrievable key frames.

Evidence-driven backend

Router, Recall, and Search Workers coordinate historical evidence and external information without blocking live interaction.

Key Results

Stronger coverage, lower response latency

StreamMind ranks first among evaluated streaming systems on every target capability.

Main Results

Accuracy across four streaming capabilities

Category averages from the full paper table, in percent.

Main StreamArena results
Class	Method	RTP ↑	HR ↑	Tool ↑	Pro. ↑
Human	Human (streaming)	91.8	63.4	95.2	91.5
Offline	Qwen3.5-397B-A17B	44.1	41.5	62.2	—
Offline	MiMo-V2.5	38.0	35.8	47.9	—
Offline	Kimi-K2.6	47.9	43.8	60.9	—
Offline	Gemini 3.5 Flash†	51.3	51.4	70.8	—
Offline	Qwen3.5-Omni†	41.8	35.8	49.4	—
Recent window	AURA‡	28.1	22.7	—	3.7
Recent window	MiniCPM-o-4.5‡	22.1	9.8	17.1	7.5
Text summary	VST‡	24.0	21.2	—	—
Internal compression	StreamForest	17.9	14.4	—	—
Internal compression	ThinkStream	8.0	7.5	1.8	1.2
Ours	StreamMind	44.5	34.9	56.1	11.6

HR is question-weighted across four evidence-to-query gaps; Pro. is weighted across three monitoring horizons. † closed-source model; ‡ author-finetuned backbone; — unsupported or not applicable.

Online Latency

Mean query-to-answer latency

Accuracy and mean latency in seconds.

Accuracy and mean latency in seconds
Method	ACC ↑	RTP ↓	HR ↓	Tool ↓
Gemini 3.5 Flash†	63.1	86.9	123.9	139.5
Kimi-K2.6	54.5	162.8	187.9	128.2
Qwen3.5-397B-A17B	54.2	83.8	117.4	62.8
Qwen3.5-Omni†	44.6	50.8	56.4	103.7
MiMo-V2.5	43.3	224.5	266.5	187.4
StreamMind	48.6	12.9	30.7	28.1

Experimental findings

Recent-window methods are responsive but cannot recover distant evidence once it leaves the context window.
Text summaries extend temporal coverage but discard visual details required by open-ended questions.
Persistent multimodal memory and explicit retrieval remain effective across all four historical time-gap buckets.
Proactive monitoring and reactive question answering benefit from distinct execution paths.

Conclusion

Streaming intelligence needs more than long context.

StreamArena evaluates the complete loop of continuous perception, long-horizon memory, tool use, and proactive interaction. StreamMind shows that separating real-time interaction from asynchronous multimodal memory is a promising path toward practical always-on agents.

Evaluation

Hour-scale streams, open-ended questions, timestamped evidence, and causal access expose shortcuts hidden by short clips and answer choices.

Architecture

Decoupling the responsive frontend from asynchronous memory, recall, monitoring, and search improves all four target capabilities.

What remains

Hour-scale recall, proactive monitoring, smaller backbones, and continuous-processing efficiency remain concrete open directions.

BibTeX

Cite StreamArena

BibTeX

@misc{zhang2026streamarenacontinuousinteractivelonghorizon,
  title={StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding},
  author={Xichen Zhang and Guankai Li and Yinghao Zhu and Shijian Wang and Sitong Wu and Shaozuo Yu and Meng Chu and Yuan Lu and Jiaya Jia},
  year={2026},
  eprint={2608.05703},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.05703},
}