Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

AI Can Learn Scientific Taste

About

Great scientists have strong judgement and foresight, closely tied to what we call scientific taste. Here, we use the term to refer to the capacity to judge and propose research ideas with high potential impact. However, most relative research focuses on improving an AI scientist's executive capability, while enhancing an AI's scientific taste remains underexplored. In this work, we propose Reinforcement Learning from Community Feedback (RLCF), a training paradigm that uses large-scale community signals as supervision, and formulate scientific taste learning as a preference modeling and alignment problem. For preference modeling, we train Scientific Judge on 700K field- and time-matched pairs of high- vs. low-citation papers to judge ideas. For preference alignment, using Scientific Judge as a reward model, we train a policy model, Scientific Thinker, to propose research ideas with high potential impact. Experiments show Scientific Judge outperforms SOTA LLMs (e.g., GPT-5.2, Gemini 3 Pro) and generalizes to future-year test, unseen fields, and peer-review preference. Furthermore, Scientific Thinker proposes research ideas with higher potential impact than baselines. Our findings show that AI can learn scientific taste, marking a key step toward reaching human-level AI scientists.

Jingqi Tong, Mingzhe Li, Hangcheng Li, Yongzhuo Yang, Yurong Mou, Weijie Ma, Zhiheng Xi, Hongji Chen, Xiaoran Liu, Qinyuan Cheng, Ming Zhang, Qiguang Chen, Weifeng Ge, Qipeng Guo, Tianlei Ying, Tianxiang Sun, Yining Zheng, Xinchi Chen, Jun Zhao, Ning Ding, Xuanjing Huang, Yugang Jiang, Xipeng Qiu• 2026

Related benchmarks

TaskDatasetResultRank
Scientific judgment (Pairwise citation prediction)SciJudgeBench in-domain (test)
CS Pairwise Accuracy87.9
22
Prospective paper impact forecastingarXiv October 2025
Top-5 Accuracy40
12
Prospective paper impact forecastingarXiv (November 2025)
Top-5 Accuracy26.7
12
Scientific Impact ForecastingarXiv June 2024 to November 2025 (temporal out-of-distribution (OOD))
Forecast Score (2025.08)0.226
12
Prospective paper impact forecastingarXiv June 2025 - November 2025 Average
Top-5 Accuracy22.2
12
Prospective paper impact forecastingarXiv August 2025
Top-5 Accuracy13.3
12
Prospective paper impact forecastingarXiv September 2025
Top-5 Accuracy0.00e+0
12
Prospective paper impact forecastingarXiv January 2025
Top-5 Accuracy40
11
Prospective paper impact forecastingarXiv July 2025
Top-5 Accuracy33.3
11
Prospective paper impact forecastingarXiv December 2024 - May 2025 Average
Top-5 Accuracy13.6
11
Showing 10 of 18 rows

Other info

GitHub

Follow for update