DataFlow-Harness Platform Bridges Gap Between LLM Agents and Persistent Data Pipelines
Researchers have introduced DataFlow-Harness, a platform designed to address what they call the “NL2Pipeline gap”—the disconnect between LLM-generated scripts and persistent, editable platform artifacts, according to a paper published on arxiv.org.
According to the research, DataFlow-Harness guides LLM agents to construct platform-native directed acyclic graphs (DAGs) through “typed, incremental mutations rather than free-form scripts.” The platform combines three components: DataFlow-Skills for procedural guidance, a Model Context Protocol (MCP) layer exposing the live operator registry and current pipeline state, and DataFlow-WebUI, which synchronizes conversational authoring with a visual DAG editor.
On a 12-task data-engineering benchmark, DataFlow-Harness achieved a 93.3% observed end-to-end pass rate, according to the paper. Compared to Vanilla Claude Code, the platform reduced measured monetary cost by 72.5% and generation latency by 49.9%. The observed pass rate was within 0.9 percentage points of the Context-Aware Claude Code baseline while achieving 42.8% lower cost.
The research indicates that “Skills are most useful when construction depends on implicit procedural knowledge,” according to per-task analysis. The authors conclude that “live platform grounding can produce persistent, editable workflow artifacts with an observed reliability close to script-generation baselines and with lower measured construction cost and latency.”
The technical report spans 13 pages with 2 figures and 5 tables, published under the categories of Software Engineering and Artificial Intelligence.