Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

DeSRPA: Decoupled Speech Role-Playing Agent via Inference-Time Intervention

About

While Large Language Models (LLMs) have revolutionized text-based role-playing, creating immersive Speech Role-Playing Agents (SRPAs) requires a seamless bridge between cognitive reasoning and paralinguistic nuances. Current SRPAs primarily rely on end-to-end (E2E) fine-tuning. However, this paradigm suffers from poor generalization to unseen characters due to its reliance on role-specific data, while imposing a "modality alignment tax" that degrades intrinsic LLM reasoning capabilities. We propose DeSRPA, an agentic framework for character role play via inference-time intervention on frozen backbones. DeSRPA employs a dual-level control vector mechanism, Internal Cognitive Steering and External Expressive Rendering, to synchronize "mind" and "voice". Experiments on SpeechRole and OmniCharacter benchmarks demonstrate that DeSRPA significantly outperforms E2E baselines in personality and emotional consistency. It achieves high speech naturalness, narrowing the gap with proprietary models like GPT-4o Audio, while remaining a scalable and training-free paradigm.

Wenqiu Tang, Zhen Wan, Takahiro Komamizu, Ichiro Ide• 2026

Related benchmarks

TaskDatasetResultRank
Multimodal Role-playing Speech GenerationSpeechRole 4
TTFA (ms)561
9
Character-Role Speech GenerationOmniCharacter-10K (test)
Fluency8.7
3
Showing 2 of 2 rows

Other info

Follow for update