Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition

About

Long-horizon household tasks require robots to compose many language-conditioned skills, yet the boundary between consecutive skills is rarely explicit. A skill may satisfy its own postcondition while leaving the robot, objects, or camera views in a state from which the next skill cannot reliably start. We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness. The harness invokes $\pi_{0.5}$-based skill checkpoints trained from cleaned BEHAVIOR-1K demonstrations, assigns each skill typed arguments and a step budget, and uses multi-view vision-language model verification to decide whether execution should advance, retry, or replan. To separate isolated skill competence from long-horizon compositional robustness, we evaluate the same checkpoints under two initial-state distributions: clean skill-boundary snapshots and chained terminal states produced by previous skills. Selected navigation, grasping, placement, and door-opening skills achieve 77--100% success from clean snapshots under human-reviewed verification, yet composed rollouts still frequently stall from chained states. The resulting traces attribute failures to next-skill readiness, target grounding, and control execution, turning nearzero task success into actionable diagnostics for what VLA skill libraries must learn next: robustness to the messy chained-state distribution that clean demonstrations underrepresent.

Ke Rui, Yushen Zuo, Jiawei Wang, Haoran Jia, Jinming Ma, Weitao Zhou, Minglei Li• 2026

Related benchmarks

TaskDatasetResultRank
Isolated Skill ExecutionBEHAVIOR-1K (val)
Success Count28
7
Bring in woodBehavior-1K
Progress (BEHAVIOR-1K)10.3
1
Cook hot dogsBehavior-1K
Progress (%)10.3
1
Freeze piesBehavior-1K
Progress6.4
1
Hide Easter eggsBEHAVIOR-1K three instances each
Progress0.00e+0
1
Make microwave popcornBEHAVIOR-1K three instances each
Progress (%)45.8
1
Move boxes to storageBEHAVIOR-1K three instances each
Progress30
1
Pick up trashBEHAVIOR-1K three instances each
Progress22.2
1
Put shoes on rackBehavior-1K
Progress (%)11.9
1
Set mousetrapsBehavior-1K
Progress (%)8.3
1
Showing 10 of 11 rows

Other info

Follow for update