Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

About

Large Vision-Language Models (LVLMs) have achieved impressive performance across multimodal understanding and reasoning tasks, yet their internal safety mechanisms remain opaque and poorly controlled. In this work, we present a comprehensive framework for diagnosing and repairing unsafe channels within LVLMs (CARE). We first perform causal mediation analysis to identify neurons and layers that are causally responsible for unsafe behaviors. Based on these findings, we introduce a dual-modal safety subspace projection method that learns generalized safety subspaces for both visual and textual modalities through generalized eigen-decomposition between benign and malicious activations. During inference, activations are dynamically projected toward these safety subspaces via a hybrid fusion mechanism that adaptively balances visual and textual corrections, effectively suppressing unsafe features while preserving semantic fidelity. Extensive experiments on multiple safety benchmarks demonstrate that our causal-subspace repair framework significantly enhances safety robustness without degrading general multimodal capabilities, outperforming prior activation steering and alignment-based baselines. Additionally, our method exhibits good transferability, defending against unseen attacks.

Jinhu Fu, Yihang Lou, Qingyi Si, Shudong Zhang, Yan Bai, Sen Su• 2026

Related benchmarks

TaskDatasetResultRank
Safety EvaluationMM-Safety
ASR8.72
57
Safety EvaluationMI-FGSM-Toxic
ASR9.13
24
Safety EvaluationMI-FGSM-JailBreak
ASR8.03
24
Safety EvaluationJailBreakV
ASR6.55
18
Jailbreak DefenseJailBreakV
Attack Success Rate (ASR)6.55
14
Toxicity DefensePGD-Toxic
ASR (PGD-Toxic)2.37
14
Jailbreak DefensePGD-Jailbreak
ASR5.45
6
Safety EvaluationAPGD-JailBreak
ASR (Unconstrained)3.31
6
Safety EvaluationAPGD-Toxic
ASR (Unconstrained)4.43
6
Adversarial Robustness EvaluationPGDT64
Attack Success Rate (ASR)4.6
3
Showing 10 of 11 rows

Other info

Follow for update