Generalized Planning Interfaces
for Agentic Robotics

Anonymous Authors

Supplementary material for peer review

Read the paper Watch the rollouts Explore failure recovery

Abstract

Agentic manipulation requires reasoning over semantic task structure and continuous robot motion. Existing robot interfaces determine which decisions an agent can express and which it must resolve explicitly. High-level skill interfaces restrict task coverage, while detailed low-level motion generation imposes substantial inference cost and is more prone to geometric errors. We introduce generalized planning interfaces (GPIs), which let agents specify what a robot should achieve and how it should act, at any level of abstraction. Agents flexibly combine declarative goals and motion constraints with imperative skill sequences and fully specified trajectories.

We instantiate a GPI with SPROUT (Sampling and Pooled Refinements of Underspecified Trajectories). SPROUT interleaves spatiotemporal constraint-graph synthesis with sampling and optimization, preserving agent-defined goals and constraints, while supplying the physical variables and constraints needed for execution. Compatible numerical queries are pooled across heterogeneous graphs for GPU execution. Across eight simulated tasks with three scenes each, SPROUT achieves a 29 percentage point increase in success, compared with less flexible agent-robot interfaces. SPROUT enables agents to solve complex multi-step manipulation tasks, reasoning at the level of abstraction required for the task.

Method

One interface, many levels of control

A goal is enough for some tasks. Others need a particular grasp, an orientation constraint, or an exact tool trajectory. A GPI lets the agent mix these specifications.

Goals, skills, constraints and trajectories flow from an agent into SPROUT, which synthesizes a constraint graph, executes robot motion, and returns observations.Expand figure
From specification to execution. Goals and prescribed motion can coexist across abstraction levels. SPROUT completes the request and returns trajectories or diagnostic feedback. Figure 1 in the paper.
1

Specify the essentials

The agent builds a structured request from goals, skills, constraints, waypoints, or trajectory segments. Decisions left open become variables for the planner.

2

Grow the missing structure

SPROUT synthesizes candidate constraint graphs, introducing actions and physical constraints while preserving the agent’s requirements.

3

Solve and execute

Sampling and numerical refinement solve the remaining choices. Compatible queries are pooled on the GPU. Execution observations inform the next request.

Choose where to spend the reasoning

For a drawing task, the agent can specify the pen path and let SPROUT find the grasp, approach, and robot motion. For object placement, a relational goal may be enough.

GoalsPlace on a plate
SkillsPick, then place
ConstraintsKeep upright
MotionTrace this path

Combine levels within the same request.

Figure 3, in motion

From a pen path to robot motion

View original figure ↗
0:24 / 0:24

Resolve the open choices

Sampling and refinement find values that satisfy the graph while preserving the agent’s requested pen path.

4 supplied values
8 completed variables
7 constraints

Scroll horizontally to explore the full graph.

Growing the drawing constraint graphFigure 3 reconstruction. The scene supplies the initial robot configuration and pen pose; the agent supplies pen identity and desired pen trajectory. SPROUT adds a shared grasp, end-effector poses and paths, robot configurations and connecting motions, constrained by grasp, kinematics, inverse kinematics and collision-free motion factors. Dotted lines connect variables to their constraints, not execution order. Provided valuesSPROUT grows the missing structure Observed sceneAgent specification A pen and a desired path.The physical choices are still open. Pen identity and grasp relation Initial pen pose, grasp and end-effector pose Pen path, shared grasp and end-effector path Inverse kinematics for the pick configuration Inverse kinematics for the drawing trajectory Collision-free robot motion from the initial configuration to the pick Collision-free robot motion connecting the pick and drawing configurations Initial robot configuration Initial pen pose Agent-selected pen Agent-specified pen trajectory Shared pen grasp End-effector pose for picking the pen End-effector drawing trajectory Robot configuration for picking the pen Robot drawing trajectory and drawing boundary configuration Robot motion to pick the pen Robot motion approaching the drawing Dotted edges connect variables to constraints.
Agent providesScene providesSPROUT variableConstraint
One feasible completion
tau pick-pen robotPick the pentau pre-draw robotApproach the boardtau draw robot and q drawFollow the pen path
The agent specifies the drawing; SPROUT completes its physical realization. Rounded nodes are variables; boxes are constraints. Kin relates object and end-effector motion; IK relates end-effector motion to robot configurations. Cfree Motion connects configurations without collisions. This is an explanatory sequence: synthesis and numerical search can interleave, and the animation is not a recorded solver trace. Figure 3 in the paper.

Clear the whiteboard. Then specify the drawing.

A task-level goal prepares the surface; a pen trajectory defines the drawing.

3× speed
SPROUT relocates the objects to clear the board.Simulation · 3× speed

Evaluation

Broader task coverage through flexible control

Eight manipulation tasks, three scenes per task, and the same agent across four interfaces. All reported benchmark results are from simulation.

16 / 24task–scene instances completed
+29.2 ppover Skill+EE and TAMP
8 / 8task families with a success

From pick-and-place to drawing and tool use

SPROUT completes six of the nine instances involving toast preparation, whiteboard drawing, and tool-assisted retrieval. The other interfaces record no successes on those nine instances.

The interface leaves routine geometric choices to the planner while allowing the agent to prescribe motion when the task demands it.

Success counts follow Section 5 and Figure 5 of the paper. A task family with a success may still have failed scenes.

Task completion over all 24 instances
Agent–robot interfaceSuccessesRate
End-effector control4 / 2416.7%
Skill + end-effector API9 / 2437.5%
Task and motion planning9 / 2437.5%
SPROUT16 / 2466.7%

Offload geometric reasoning. Keep expressive control.

The agent can spend its reasoning on the task-specific decisions and delegate geometric completion to the model-based planner. The timing results show that balance alongside the tasks each interface can solve.

Mean episode wall time

Five shared tasks · three scenes each · lower time is better

Mean wall time on the five shared tasks: EE approximately 558 seconds, Skill+EE 314 seconds, TAMP 274 seconds, SPROUT 315 seconds. Overall success across all 24 instances is shown in the table above.Expand chart
SPROUT averages 315 s, compared with 558 s for end-effector control, on the five shared tasks. Across the full 24-instance benchmark, SPROUT succeeds on 16 / 24, versus 4 / 24 for end-effector control.

Where the computation goes

Reported time allocation · Figure 6

Reported time shares: EE 95.6 percent reasoning and 0 planning; Skill+EE 90.9 reasoning and 2.1 planning; TAMP 69.3 reasoning and 16.4 planning; SPROUT 78.4 reasoning and 12.4 planning. Remaining time is execution and other work.Expand chart
Agent reasoning accounts for 78.4% of SPROUT’s reported time, versus 95.6% for EE. Model-based planning accounts for 12.4%, resolving choices left open by the agent.

Approximately 44% lower mean wall time than EE on the five shared tasks. SPROUT also completes 6/9 instances of toast preparation, drawing, and tool retrieval, which the other interfaces do not solve. The flexible specification lets the agent add motion detail when needed and return to high-level goals when it is not.

TAMP is faster on the shared tasks at 274 s, but cannot express the three more complex task types in this evaluation. Flexibility and task coverage matter alongside runtime.

Source: Section 5.2 and Figures 5–6. Episode means include unsuccessful runs. The time-allocation chart reproduces the paper’s reported percentages; it is not used to infer absolute planning times or per-replan latency.

View the timing values and measurement scope
Five shared tasks: 15 task–scene instances per interface
InterfaceMean wall time
EE≈558 s
Skill+EE≈314 s
TAMP≈274 s
SPROUT≈315 s
Time allocation reported in Figure 6
InterfaceAgent reasoningPlanningExecution + other
EE95.6%0.0%4.4%
Skill+EE90.9%2.1%7.0%
TAMP69.3%16.4%14.2%
SPROUT78.4%12.4%9.2%

The shared tasks are stacking blocks, stowing utensils, discarding bottles, aligning tools, and reorienting mugs. Figure 6 does not separately specify its aggregation basis, so its proportions are shown independently from the five-task means. TAMP’s percentages sum to 99.9% due to rounding.

Task-by-task results and evaluation scope
Successes out of three scenes per task
TaskEESkill+EETAMPSPROUT
Block stack3333
Utensils upright0201
Discard bottles1333
Tool alignment0011
Reorient mugs0122
Toast breakfast0002
Whiteboard smiley0001
Tube tools0003

The evaluation uses Isaac Sim with an FR3 and Robotiq 2F-85 gripper. Scene information and the agent are shared across conditions. TAMP cannot express the final three task types; these instances count as failures in its overall success rate.

Scope and limitations. This is a 24-instance evaluation. SPROUT does not solve every scene, and a restricted planner can be faster on tasks within its domain. Execution reliability, computation allocation, and real-world deployment remain directions for further work.

Agentic failure recovery

Notice the miss. Revise the request. Keep going.

Execution feedback lets the agent identify what needs to change. It revises the relevant part of the task, and SPROUT replans the remaining geometry and motion.

3× speed
Tool-use recovery. The initial attempt does not free the block. After a regrasp and another push, the robot retrieves it and completes the placement. 3× speed; chapter labels refer to playback time.
1

Monitor the outcome

The first rod interaction leaves the red block inside the tube. Execution observations give the agent feedback on the gap between the intended and observed result.

2

Edit what needs to change

The robot withdraws, reapproaches, and regrips the rod. A renewed push frees the block. The agent can revise a goal, constraint, or motion detail while the planner completes the rest.

3

Continue toward the goal

With the block accessible, the robot sets the rod aside, grasps the block, and places it on the plate. Recovery is part of the episode.

Efficient recovery through focused replanning. The agent specifies the necessary change instead of spelling out every grasp, approach, and robot trajectory again. SPROUT supplies those remaining choices using model-based search and pooled numerical queries.

The clip illustrates the physical retry and continuation. The agent–planner explanation follows the system design; individual replanning latency is not measured in the video.

Further examples of recovering and continuing

3× speed
Recover a banana placement · 3× speedThe banana misses the bowl and rests across its rim on earlier attempts. The robot regrips and places it into the bowl.
4× speed
Regrasp and stow utensils · 4× speedThe robot moves green utensils from a wire shelf toward a green bucket, then regrips displaced utensils from the tabletop and places them into the bucket.

BibTeX

Citation for the anonymous manuscript.

@misc{anonymous2026sprout,
  title = {Generalized Planning Interfaces for Agentic Robotics},
  author = {Anonymous Authors},
  year = {2026},
  note = {Anonymous manuscript}
}