3D-RFT Framework Brings Reinforcement Learning to Video-Based 3D Scene Understanding

3D-RFT Framework Brings Reinforcement Learning to Video-Based 3D Scene Understanding

Researchers have introduced 3D-RFT (Reinforcement Fine-Tuning for Video-based 3D Scene Understanding), the first framework to extend Reinforcement Learning with Verifiable Rewards (RLVR) to video-based 3D perception and reasoning, according to a paper accepted at ICML 2026 and published on arxiv.org.

According to the paper, existing approaches to 3D scene understanding rely on Supervised Fine-Tuning (SFT), where “the token-level cross-entropy loss acts as an indirect proxy for optimization, leading to a misalignment between training objectives and task performances.” 3D-RFT addresses this by directly optimizing models toward evaluation metrics.

The framework operates in two stages: first activating 3D-aware Multi-modal Large Language Models (MLLMs) via SFT, then applying reinforcement fine-tuning using Group Relative Policy Optimization (GRPO) with verifiable reward functions. According to arxiv.org, the researchers “design task-specific reward functions directly from metrics like 3D IoU and F1-Score to provide more effective signals to guide model training.”

Extensive experiments demonstrate that 3D-RFT-4B achieves state-of-the-art performance on various video-based 3D scene understanding tasks. Notably, according to the paper, “3D-RFT-4B significantly outperforms larger models (e.g., VGLLM-8B) on 3D video detection, 3D visual grounding, and spatial reasoning benchmarks.”

A project page for the research is available at 3d-rft.github.io.