Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents

About

Autonomous LLM agents can pursue hidden malicious objectives through sequences of individually benign actions, making sabotage difficult to detect using standard trajectory-level monitoring. Existing approaches either evaluate complete trajectories in a single pass or partition them into independently scored windows, limiting their ability to connect evidence across temporally distant actions. We propose TRACE, a monitoring framework for long-horizon LLM agent trajectories. TRACE operates through a TIJ (Triage-Inspect-Judge) loop that identifies high-signal regions, performs targeted inspection while maintaining accumulated evidence across reasoning steps, and synthesizes a trajectory-level verdict. We evaluate TRACE on ten task domains from SHADE-Arena against state-of-the-art baselines. TRACE achieves an aggregate F1 of 0.713 and recall of 0.844, with the largest gains on tasks requiring long-range evidence linking.

Vijitha Mittapalli, Shreyaa Jayant Dani, Satya Srujana Pilli, Snigdha Ansu, Mohammadreza Teymoorianfard, Franck Dernoncourt, Hongjie Chen, Yu Wang, Ryan A. Rossi, Nesreen K. Ahmed• 2026

Related benchmarks

TaskDatasetResultRank
Sabotage detectionSHADE-Arena Election Tampering
Accuracy44.4
3
Sabotage detectionSHADE-Arena Performance Reviews
Accuracy67.3
3
Sabotage detectionSHADE-Arena Bank Transfer
Accuracy65.7
3
Sabotage detectionSHADE-Arena American Airlines
Accuracy72.3
3
Sabotage detectionSHADE-Arena Spam Filter
Accuracy64.4
3
Sabotage detectionSHADE-Arena Charity Allocation
Accuracy77.1
3
Sabotage detectionSHADE-Arena SSN Task
Accuracy46.3
3
Sabotage detectionSHADE-Arena Jira Tickets
Accuracy52.9
3
Sabotage detectionSHADE-Arena API Key Calendar
Accuracy57.4
3
Sabotage detectionSHADE-Arena Expenditure Accounting
Accuracy57.6
3
Showing 10 of 11 rows

Other info

Follow for update