ReEvalMed

Overview

ReEvalMed is a clinically grounded meta-evaluation framework designed to assess the reliability of radiology report generation metrics. While many existing metrics assign high scores to generated reports, they often fail to capture clinical semantics. ReEvalMed redefines how metrics should be evaluated by aligning with clinical needs and real-world decision-making.

Key Contributions

• A comprehensive, clinically aligned meta-evaluation framework for medical report metrics.
• Fine-grained GT–ME (Ground Truth – Meta Evaluation) dataset with targeted error types, clinical significance labels and explanations.
• Systematic analysis of weaknesses in widely used metrics (BLEU, ROUGE-L, BERTScore, RadGraph-F1, CheXbert F1, Green Score).
• Demonstration that no existing metric satisfies all essential criteria such as discrimination, robustness, and monotonicity.
• Guidelines for developing clinically reliable evaluation approaches.

Dataset

The ReEvalMed dataset is currently under formal review and approval. It will be publicly released on PhysioNet once the review process is completed.

If you need early access for research purposes, please feel free to contact: liruochen8@gmail.com.

Code

The official implementation will be released soon on GitHub.

If you require early access to the codebase, please contact: liruochen8@gmail.com

BibTeX

@inproceedings{li2025reevalmed,
  title={ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment},
  author={Li, Ruochen and Li, Jun and Jian, Bailiang and Yuan, Kun and Zhu, Youxiang},
  booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  pages={11823--11837},
  year={2025}
}