Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition
About
Long-horizon household tasks require robots to compose many language-conditioned skills, yet the boundary between consecutive skills is rarely explicit. A skill may satisfy its own postcondition while leaving the robot, objects, or camera views in a state from which the next skill cannot reliably start. We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness. The harness invokes $\pi_{0.5}$-based skill checkpoints trained from cleaned BEHAVIOR-1K demonstrations, assigns each skill typed arguments and a step budget, and uses multi-view vision-language model verification to decide whether execution should advance, retry, or replan. To separate isolated skill competence from long-horizon compositional robustness, we evaluate the same checkpoints under two initial-state distributions: clean skill-boundary snapshots and chained terminal states produced by previous skills. Selected navigation, grasping, placement, and door-opening skills achieve 77--100% success from clean snapshots under human-reviewed verification, yet composed rollouts still frequently stall from chained states. The resulting traces attribute failures to next-skill readiness, target grounding, and control execution, turning nearzero task success into actionable diagnostics for what VLA skill libraries must learn next: robustness to the messy chained-state distribution that clean demonstrations underrepresent.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Isolated Skill Execution | BEHAVIOR-1K (val) | Success Count28 | 7 | |
| Bring in wood | Behavior-1K | Progress (BEHAVIOR-1K)10.3 | 1 | |
| Cook hot dogs | Behavior-1K | Progress (%)10.3 | 1 | |
| Freeze pies | Behavior-1K | Progress6.4 | 1 | |
| Hide Easter eggs | BEHAVIOR-1K three instances each | Progress0.00e+0 | 1 | |
| Make microwave popcorn | BEHAVIOR-1K three instances each | Progress (%)45.8 | 1 | |
| Move boxes to storage | BEHAVIOR-1K three instances each | Progress30 | 1 | |
| Pick up trash | BEHAVIOR-1K three instances each | Progress22.2 | 1 | |
| Put shoes on rack | Behavior-1K | Progress (%)11.9 | 1 | |
| Set mousetraps | Behavior-1K | Progress (%)8.3 | 1 |