VIA: Visual Interface Agent for Robot Control
Your coding/computer-use agent is secretly a robot control agent.
- Stanford University
Abstract
Core Idea
- Recast robot control as a visual tool-use task for agents, similar to general computer use.
- Design a visual interface and a small set of versatile MCP tools for agents to see and command the robot.
- Run off-the-shelf agents (Claude Code or Codex) to drive the interface and complete robot manipulation tasks.
Result Highlights
- VIA solves a diverse suite of tabletop manipulation tasks zero-shot with a high success rate.
- VIA inherits strong reasoning, long-horizon planning, and closed-loop control capabilities from frontier agents for free.
- VIA rides the wave of foundation model scaling, with VIA (Fable) significantly outperforming VIA (Opus).
Note
- VIA is not Code-as-Policies (CaP). It involves no CaP-style code generation and provides no perception or skill primitives for the model to compose. VIA observes the UI through raw pixels (screenshots) and acts via low-level tools akin to mouse and keyboard operations in computer-use.
- VIA opens up a new opportunity for leveraging state-of-the-art foundation models at scale in robotics, and demonstrates that with the right interface, robot control can become another economically valuable agentic task that benefits directly from modern models and agents.
VIA
execute_waypoint to invoke a simple PI
(Proportional-Integral) controller that moves the real gripper to that pose.
This observe-act loop repeats until the agent calls end_episode.
The UI reconstructs the 3D scene from RGB-D cameras. The MCP Tools section
below covers the documentation and illustrations of all available tools.
Videos showing episodes of VIA with Claude Code Fable completing different tasks zero-shot. The agent completes a task via a loop of observation, thinking, and finally a tool call that returns a new observation, until it reaches a terminal condition.
Highlights
Evaluation
We use a suite of six tasks drawn from different benchmarks, together covering diverse manipulations.
Task descriptions
- Stack: Pick up a red cube and place it on a larger green cube, from robosuite.
- Turn on stove: Turn on a stove by rotating its knob, from LIBERO-Goal.
- Open drawer: Open the middle drawer of a three-drawer cabinet, from LIBERO-Goal.
- Put bowl on plate: Pick up a bowl and place it on a plate, from LIBERO-Goal. This demands physical reasoning and precision, as the bowl must be grasped by pinching its side wall and placed at the exact center of the plate to succeed.
- Rainbow: Arrange seven randomly placed colored blocks into a rainbow, a task we created. This tests long-horizon planning and execution.
- T-block: Build a T-shape with three blocks of the same size: one blue block on the bottom and two green blocks resting directly on top of it, side by side; from BuilderBench, re-implemented. This requires both solving a physics puzzle and precise manipulation.
We evaluate VIA using two families of agents:
- Claude Code with Opus 4.8 (CC-Opus) and Fable 5 (CC-Fable),
- Codex with GPT-5.5 (Codex-5.5) and GPT-5.6-Sol (Codex-5.6-Sol).
The reasoning effort is set to xhigh for all agents.
On top of a shared system prompt on how to use the interface, we consider two variants of task-specific prompts for each task:
- minimal: states only the task's goal and success condition, testing the zero-shot performance of each agent.
- detailed: provides an example list of waypoints, simulating the effect of having a demonstration.
We disable memory read/write for the agents so that they cannot gain cross-episode experience.
| Task | CC-Opusminimal | CC-Fableminimal | Codex-5.5minimal | Codex-5.6-Solminimal | CC-Opusdetailed | Codex-5.5detailed | |
|---|---|---|---|---|---|---|---|
| Stack | 100% | 100% | 100% | 100% | — | — | |
| Turn on stove | 100% | 100% | 90% | 90% | 100% | 90% | |
| Open drawer | 70% | 90% | 30% | 30% | 100% | 30% | |
| Put bowl on plate | 60% | 100% | 40% | 80% | 100% | 30% | |
| Rainbow | 80% | 100% | 60% | 50% | — | — | |
| T-block | 10% | 40% | 40% | 20% | — | — | |
| Overall | 70% | 88% | 60% | 62% | 100% | 50% |
"—" marks combinations we do not evaluate; bold marks the best result per task within each prompt variant; the Overall row averages each column over its evaluated tasks.
Tool Call Efficiency and Cost
| Task | CC-Opusminimal | CC-Fableminimal | Codex-5.5minimal | Codex-5.6-Solminimal | CC-Opusdetailed | Codex-5.5detailed | |
|---|---|---|---|---|---|---|---|
| Stack | 28 | 40 | 26 | 32 | — | — | |
| Turn on stove | 36 | 39 | 33 | 31 | 43 | 32 | |
| Open drawer | 47 | 38 | 21 | 33 | 46 | 30 | |
| Put bowl on plate | 59 | 53 | 78 | 66 | 48 | 57 | |
| Rainbow | 159 | 143 | 176 | 217 | — | — | |
| T-block | 81 | 89 | 52 | 122 | — | — | |
| Overall | 68 | 67 | 64 | 84 | 46 | 40 |
| Task | CC-Opusminimal | CC-Fableminimal | Codex-5.5minimal | Codex-5.6-Solminimal | CC-Opusdetailed | Codex-5.5detailed | |
|---|---|---|---|---|---|---|---|
| Stack | $2.4 | $7.3 | $1.5 | $1.6 | — | — | |
| Turn on stove | $4.1 | $8.9 | $2.5 | $2.2 | $5.7 | $3.3 | |
| Open drawer | $7.4 | $7.5 | $1.4 | $1.1 | $6.5 | $1.7 | |
| Put bowl on plate | $7.5 | $11.8 | $8.1 | $2.9 | $5.0 | $4.9 | |
| Rainbow | $26.0 | $36.1 | $23.4 | $6.1 | — | — | |
| T-block | $9.7 | $19.2 | $6.1 | $10.5 | — | — | |
| Overall | $9.5 | $15.1 | $7.2 | $4.1 | $5.7 | $3.3 |
CC costs are read directly from each episode's session cost, while Codex costs are estimated from token usage and listed API pricing.
Task Rollouts
Task
MCP Tools