Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Personalization and Evaluation of Conversational Information Access

About

Conversational interactions have reshaped information retrieval systems, as users increasingly favour direct answers over traditional hyperlinks. To build reliable Conversational Information Access (CIA) systems that account for personal context, this thesis addresses challenges: (1) personal context extraction, (2) personalized response generation, and (3) effective and interpretable system evaluation. First, we tackle personal context extraction by studying what Entity Linking (EL) in conversations entails, introducing a dataset for conversational entity linking (ConEL), and proposing CREL, a novel EL method tailored for conversational settings. Second, we focus on personalized response generation by proposing LAPS, a method for efficiently constructing large-scale, human-written, personalized conversational datasets, and using them to study how users' preferences can be utilized to generate personalized responses. Finally, we address the need for effective and interpretable system evaluation by introducing FACE, an automatic, reference-free method that assesses entire conversations and aligns closely with human judgments.

Hideaki Joko• 2026

Related benchmarks

TaskDatasetResultRank
Personal Entity LinkingConEL-2 (val)
Precision71.4
14
Personal Entity LinkingConEL-2 (test)
Precision67.3
14
Personal Entity LinkingConEL-PE
Precision61.3
14
Conversation EvaluationCRSArena-Eval Turn-level
Pearson Correlation (r)0.93
9
Conversation EvaluationCRSArena-Eval Dial-level
Pearson r0.913
9
Conversation EvaluationCRSArena-Eval (All)
Pearson Correlation (r)0.918
9
Concept and named entity linkingConEL-2 (test)
F_MD72.9
8
Concept and named entity linkingConEL
F1 (MD)55.9
8
Dialogue Evaluation AlignmentCRSArena-Eval RD (test)
Pearson r (Turn-level Rel)0.549
6
Dialogue Evaluation AlignmentCRSArena-Eval KG (test)
Pearson r (Turn-level Rel.)0.543
6
Showing 10 of 30 rows

Other info

Follow for update