Percentile Ranking and Citation Impact of a Large Cohort of NHLBI-Funded Cardiovascular R01 Grants

I just ran across this interesting new study that evaluated the relationship between the score that an NIH R01 grant receives during peer-review and the future impact of the grant as measured by number and quality of publications. The bottom line is that a grant that receives a top score in the 10th percentile does not produce publications with impact above and beyond a grant in the 30th percentile that would not be funded by 2014 criteria.

Danthi N, Wu CO, Shi P, Lauer MS. Percentile Ranking and Citation Impact of a Large Cohort of NHLBI-Funded Cardiovascular R01 Grants. Circ Res. 2014 Jan 9. [PubMed]

Abstract

Rationale: Funding decisions for cardiovascular R01 grant applications at NHLBI largely hinge on percentile rankings. It is not known whether this approach enables the highest impact science.

Objective: To conduct an observational analysis of percentile rankings and bibliometric outcomes for a contemporary set of funded NHLBI cardiovascular R01 grants.

Methods and Results: We identified 1492 investigator-initiated de novo R01 grant applications that were funded between 2001 and 2008, and followed their progress for linked publications and citations to those publications. Our co-primary endpoints were citations received per million dollars of funding, citations obtained within 2-years of publication, and 2-year citations for each grant's maximally cited paper. In 7654 grant-years of funding that generated $3004 million of total NIH awards, the portfolio yielded 16,793 publications that appeared between 2001 and 2012 (median per grant 8, 25th and 75th percentiles 4 and 14, range 0 - 123), which received 2,224,255 citations (median per grant 1048, 25th and 75th percentiles 492 and 1,932, range 0 - 16,295). We found no association between percentile ranking and citation metrics; the absence of association persisted even after accounting for calendar time, grant duration, number of grants acknowledged per paper, number of authors per paper, early investigator status, human versus non-human focus, and institutional funding. An exploratory machine-learning analysis suggested that grants with the very best percentile rankings did yield more maximally cited papers.

Conclusions: In a large cohort of NHLBI-funded cardiovascular grants, we were unable to find a monotonic association between better percentile ranking and higher scientific impact as assessed by citation metrics.