Researchers have introduced ART (Art-based Reinforcement Training), a novel fine-tuning method for Multimodal Large Language Models that optimizes visual inputs rather than modifying model parameters, according to a paper published on arxiv.org.
Unlike existing Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low-Rank Adaptation (LoRA), which introduces additional weights between LLM layers, or Soft Prompting, which adds fine-tuning-specific tokens to inputs, ART “injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input,” according to the paper. This approach works by “backpropagation of gradients back into a plain pixel array.”
According to the researchers, both LoRA and Soft Prompting “require modification to the computational graphs of precompiled, preoptimized LLMs” and consequently are not fully supported in high-throughput engines like vLLM. ART addresses this limitation by enabling “the soft-token approach on pre-compiled computational graphs.”
The paper reports that ART was tested on different sizes of the Qwen architecture across mathematics and structured-tool-use benchmarks. The method “reaches accuracy competitive with LoRA” on these benchmarks, according to the researchers. Additionally, the optimized visual inputs “can be stylized as task-relevant computational artworks,” the paper states.
The research supports any fine-tuning objective, according to arxiv.org.