According to arxiv.org, researchers have introduced Nested-ReFT, a novel reinforcement learning framework designed to reduce the computational costs of fine-tuning large language models for complex reasoning tasks.
The framework addresses a key challenge in reinforced fine-tuning (ReFT): while standard ReFT methods demonstrate significant performance improvements on challenging reasoning domains like mathematical reasoning, they require generating multiple completions during training, making the computational cost “non-trivial,” according to the paper.
Nested-ReFT draws inspiration from off-policy reinforcement learning and speculative decoding, using a subset of the target model’s layers as a “behavior model” to generate completions during training. According to the researchers, the behavior model is configured with dynamic layer skipping per batch, which decreases inference costs compared to standard ReFT frameworks.
The paper’s theoretical analysis shows that Nested-ReFT “yields unbiased gradient estimates with controlled variance.” Empirically, the framework demonstrates “improved computational efficiency measured as tokens/sec across multiple math reasoning benchmarks and model sizes,” according to arxiv.org.
To address potential off-policy issues, the researchers explored three variants of bias mitigation to minimize off-policyness in gradient updates, allowing the system to match baseline ReFT performance while maintaining the computational efficiency gains.