TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents
About
Autonomous LLM agents can pursue hidden malicious objectives through sequences of individually benign actions, making sabotage difficult to detect using standard trajectory-level monitoring. Existing approaches either evaluate complete trajectories in a single pass or partition them into independently scored windows, limiting their ability to connect evidence across temporally distant actions. We propose TRACE, a monitoring framework for long-horizon LLM agent trajectories. TRACE operates through a TIJ (Triage-Inspect-Judge) loop that identifies high-signal regions, performs targeted inspection while maintaining accumulated evidence across reasoning steps, and synthesizes a trajectory-level verdict. We evaluate TRACE on ten task domains from SHADE-Arena against state-of-the-art baselines. TRACE achieves an aggregate F1 of 0.713 and recall of 0.844, with the largest gains on tasks requiring long-range evidence linking.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Sabotage detection | SHADE-Arena Election Tampering | Accuracy44.4 | 3 | |
| Sabotage detection | SHADE-Arena Performance Reviews | Accuracy67.3 | 3 | |
| Sabotage detection | SHADE-Arena Bank Transfer | Accuracy65.7 | 3 | |
| Sabotage detection | SHADE-Arena American Airlines | Accuracy72.3 | 3 | |
| Sabotage detection | SHADE-Arena Spam Filter | Accuracy64.4 | 3 | |
| Sabotage detection | SHADE-Arena Charity Allocation | Accuracy77.1 | 3 | |
| Sabotage detection | SHADE-Arena SSN Task | Accuracy46.3 | 3 | |
| Sabotage detection | SHADE-Arena Jira Tickets | Accuracy52.9 | 3 | |
| Sabotage detection | SHADE-Arena API Key Calendar | Accuracy57.4 | 3 | |
| Sabotage detection | SHADE-Arena Expenditure Accounting | Accuracy57.6 | 3 |