Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Observable Channels, Not Just Storage: Evaluating Privacy Leakage in LLM Agent Pipelines

About

Privacy leakage in LLM agents is often studied through individual storage or execution components, such as memory modules, retrieval pipelines, or tool-mediated artifacts. However, these settings are typically analyzed in isolation, making it difficult to compare how private internal dependence becomes externally recoverable across heterogeneous agent pipelines. In this paper, we present CIPL (Channel Inversion for Privacy Leakage) as a unified channel-oriented measurement interface for evaluating privacy leakage in LLM agent pipelines. Rather than claiming a universally strongest attack recipe, CIPL provides a shared way to represent a target through its sensitive source, selection, assembly, execution, observation, and extraction stages, and to measure how internal exposure is transformed into attacker-recoverable leakage under a common protocol. Using memory-based, retrieval-mediated, and tool-mediated instantiations under this shared interface, we identify a distinct cross-target risk picture. Memory behaves as a near-saturated high-risk special case, while beyond-memory leakage exhibits a different regime: retrieval-mediated targets show frequent but often incomplete leakage, and tool-mediated targets are strongly shaped by the exposed observation surface and provider behavior. We further show that leakage is governed by channel conditions rather than by a universally dominant recipe: cleaned weak controls sharply suppress leakage, and semantic annotation reveals attacker-useful leakage beyond exact-match extraction. Together, these findings suggest that privacy risk in LLM agent pipelines is better understood through \emph{observable channels}, not just storage components. More broadly, our results motivate channel-oriented privacy evaluation as a necessary complement to component-local or exact-only analyses.

Tao Huang, Chen Hou, Guosen Wu, Jiayang Meng• 2026

Related benchmarks

TaskDatasetResultRank
Cross-Input Privacy Leakagememory_ehr--
3
Cross-Input Privacy Leakagememory_rap--
3
Cross-Input Privacy Leakagerag toy--
3
Cross-Input Privacy Leakagetool_mock (args_exfil, llm)--
3
Cross-Input Privacy Leakagetool_mock return_echo, llm--
3
Showing 5 of 5 rows

Other info

Follow for update