Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning
About
Recent studies have explored Vision-Language Models (VLMs) for food analysis. However, most existing methods rely primarily on supervised fine-tuning (SFT), which often limits reasoning and generalization capabilities. Moreover, high-quality large-scale nutritional annotations remain scarce. To address these issues, we introduce CalorieBench-80K, a large-scale benchmark with curated calorie labels and dietary advice annotations. To the best of our knowledge, it is the first food image benchmark to incorporate Chain-of-Thought (CoT) annotations for calorie reasoning. We also propose Food-R1, a unified food VLM trained in a multi-task learning paradigm to equip the model with broad capabilities. Food-R1 undergoes CoT-based cold-start instruction tuning, followed by reinforcement fine-tuning (RFT) using Group Relative Policy Optimization (GRPO) to improve reasoning and performance. Experiments on CalorieBench-80K and representative benchmarks show that Food-R1 consistently outperforms strong baselines across food-related tasks. The code, model weights, and benchmark annotations are available at the project repository.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Classification | Food101 | Top-1 Accuracy95.21 | 83 | |
| Recipe Generation | Recipe1M | SacreBLEU7.76 | 12 | |
| Calorie estimation | CalorieBench-80K (test) | MAE42.56 | 9 | |
| Dietary advice generation | CalorieBench 80K | ROUGE-10.65 | 9 | |
| Ingredient Recognition | VireoFood-172 | F1 Score84.61 | 4 | |
| Nutrition Estimation | Nutrition5k (test) | Calorie pMAE14.4 | 3 | |
| Referring Nutrition Estimation | Nutrition5k refer@3rd 1.0 (test) | Calorie MAE13.3 | 2 | |
| Referring Nutrition Estimation | Nutrition5k refer@1st 1.0 (test) | Calorie MAE28.2 | 2 | |
| Referring Nutrition Estimation | Nutrition5k refer@2nd 1.0 (test) | Calorie MAE24.4 | 2 |