You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with a Multi-Agent Conversations
About
Meeting summarization suffers from limited high-quality data, mainly due to privacy restrictions and expensive collection processes. We address this gap with FAME, a dataset of 500 meetings in English and 300 in German produced by MIMIC, our new multi-agent meeting synthesis framework that generates meeting transcripts on a given knowledge source by defining psychologically grounded participant profiles, outlining the conversation, and orchestrating a large language model (LLM) debate. A modular post-processing step refines these outputs, mitigating potential repetitiveness and overly formal tones, ensuring coherent, credible dialogues at scale. We also propose a psychologically grounded evaluation framework assessing naturalness, social behavior authenticity, and transcript difficulties. Human assessments show that FAME approximates real-meeting spontaneity (4.5/5 in naturalness), preserves speaker-centric challenges (3/5 in spoken language), and introduces richer information-oriented difficulty (4/5 in difficulty). These findings highlight that FAME is a good and scalable proxy for real-world meeting conditions. It enables new test scenarios for meeting summarization research and other conversation-centric applications in tasks requiring conversation data or simulating social scenarios under behavioral constraints.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Abstractive Meeting Summarization | QMSum | -- | 4 | |
| Abstractive Meeting Summarization | FAME EN | -- | 4 | |
| Abstractive Meeting Summarization | FAME [GER] | -- | 4 | |
| Meeting Transcript Evaluation | FAME English (EN) | Coherence4.5 | 1 | |
| Meeting Transcript Evaluation | FAME German | Coherence (COH)4.5 | 1 | |
| Transcript Challenge Assessment | FAME English 1.0 (test) | Spoken Language Score3 | 1 | |
| Transcript Challenge Assessment | FAME German 1.0 (test) | Spoken Language Score3 | 1 | |
| Meeting Transcript Evaluation | AMI | -- | 1 | |
| Meeting Transcript Evaluation | ICSI | -- | 1 | |
| Meeting Transcript Evaluation | WPCP | -- | 1 |