An Unofficial Guide to Assessing Research Impact in NHMRC Grants

Tamika Heiden & Cathie Withyman – Research Impact Academy

*This information is based on our opinions and experiences in research impact. It has not been endorsed by the NHMRC and does not represent views of the NHMRC.

In 2018 the NHMRC introduced ‘research impact’ as part of the track record component of their Investigator and Synergy grant schemes. Despite three years of inclusion, and this element being worth 20% of the total scoring of applications, there remains significant confusion and inconsistency about how to review, assess and score research impact adequately and fairly.

Why have we developed this guide?

Since 2018, we have worked with over 300 researchers, including many peer-reviewers, to help them develop and write their impact track record sections for NHMRC applications. In doing so, we have become increasingly concerned about ‘how’ these sections are being scored. This is not a criticism of reviewers, we simply want to support them in this scoring task.

To help peer reviewers assess the impact elements of NHMRC applications, we have compiled a handy how to guide based on our experience and professional knowledge of research impact.

Before we get into this further, we want to highlight that there is no simple way to assess impact. It is a complex area that takes considerable time and effort to create, capture, evidence and communicate.

Download a PDF version of this guide

How to use the unofficial guide?

This guide begins with some basics of terminology and general impact review considerations. It then delves into how to score the sections that align with the three fields on impact within the researcher’s track record part of the application. The sections, hereon referred to as Fields are:

Reach and significance of the research impact (Field 1)
Programs contribution to research impact (Field 2)
Researcher’s contribution to research impact (Field 3)

Impact Terminology

We appreciate that many people still get confused with impact terminology, such as the difference between an output and an outcome, so here we are providing some of the terms that help with understanding research impact.

What is an output?

For the purpose of impact, an output is a deliverable from research. Essentially it is how the researcher packaged their findings for delivery to, and uptake by, others. An easy example is a journal article, this is an output, it allows others to access the new knowledge created by the research. Different types of research will have different outputs, such as tools, programs, products, patents, and datasets. Outputs are within the direct control of the program or researcher.

What is an outcome?

When talking about impact and its terminology, outcomes can differ from the traditional academic term for outcome. In research, we sometimes refer to study results as outcomes, however the term is used differently in the context of impact. An outcome happens when a research output is taken up or used in some way. Outcomes are the change that occurs because the output is used in some way. Outcomes are the differences made by the outputs or activities that delivered the program’s findings. Outcomes are early stage impacts.

NHMRC Definitions

Impact

Impact is the verifiable outcomes that research makes to knowledge, health, the economy and/or society. Impact is the effect of the research after it has been adopted, adapted for use, or used to inform further research.

Reach

Reach is the extent, spread, breadth, and/or diversity of the beneficiaries of the impact, relative to the type of research impact.

Significance

Significance is the degree to which the impact has enabled, enriched, influenced, informed or changed the performance of policies, practices, products, services, culture, understanding, awareness or well-being of the beneficiaries.

Assessing research impact overall

Firstly, we recommend that reviewers read all three impact sections before beginning to score them. There is no point providing a high score for reach and significance (Field 1) only to learn that the researcher was not talking about a program of work in which they were involved. This is an issue that stems from the NHMRC asking for impact in reverse order from what would usually be expected. Typically, in asking for impact information or a pathway to impact, we would see that a researcher was involved in a research program in some way (Field 3), they would then tell us the details of that research (Field 2) and go on to describe and demonstrate the resulting impact (Field 1).

When scoring the three impact fields, there are some tables from the guidelines that you must have on hand:

Table 2: Impact descriptors (page 45 of 2021 guidelines)
Table 1: Types of Research Impact and Examples of Evidence of Research Impact (page 104 of 2021 guidelines).

In assessing impact, first and foremost, we urge you to consider: Is there a description of impact (or not)? Has the applicant provided evidence of their impact? Simply put, they either have, or have not, made a case for impact. Importantly, one type of impact is no more important than another type of impact. Each research program, the discipline, its usefulness, and its ability to be applied more broadly will differ. Therefore, we must assess each impact statement on its own merits.

When looking at the assessment guidance we have provided below, you will notice we have used the NHMRC definitions as the fundamental guiding principles for what to expect within each of the three fields.

DOWNLOAD THE GUIDE

Field 1 – Reach and significance of the research impact

There are seven things to look for in Field 1. You can use these key indicators as a guide to score this first impact section. However, you should note that the section might not be written in a way that clearly separates and neatly compartmentalises these seven elements. If the section were written that way, it might not always make sense. Rather read the field as a whole while looking for these particular elements.

Note: We have seen many researchers write their whole Field 1 about the general impact of their research field more broadly, not their own particular research program’s impact. Applicants should not have done this; the guidelines clearly say impact from the program of research.

Seven things to look for:

Clearly defined problem and program of work outlined. Can you clearly understand what the program of work was addressing and how that led to impact?
- Have they told me what the problem was?
- Do I understand the program of work they are talking about?
- Is this a clear program/body of work?
Articulates the use, adoption, or adaptation of the program’s outputs, by others beyond the initial research work.
- Have they explained the use, adoption or adaptation by others? For example,
  - Used in/cited in policy, guidelines, education, practice, etc
  - Adopted and used/cited in other disciplines (knowledge)
  - Have they provided information on who used the output (organisation, Government Dept, other research groups, etc)?
Describes or shows the significance of the impact
- Have they described what has changed as a result of the use, adoption, adaptation of their program outputs?
- Have they told me who was affected by the change?
- Have they described the benefit of this change?
Provided verifiable evidence of all impact claims
- Have they provided references or the source of where they found the information of use/uptake? For example,
  - Full name of document and date or organisation that used findings
  - Testimonial – name/organisation and date of person
  - Scopus/scival/altmetric date of report, keyword used etc
  - Citations of other academic papers that cite the original work.
- Where did they source the information showing the stated use or change because of their program of work?
Shows reach of the outputs beyond the results of the research
- Has the applicant clearly shown, not only use of the work, but reach of the work?
  - Downloads, number of citations, media reach, number of users of guidelines, number of people accessing a report on a website, pageviews etc
Provided evidence of reach
- Where did they source the reach information?
  - Google Analytics report (date), media monitors (report, date)
  - Scopus, Sci Val, etc (key words, date)
  - Personal communications (pers comms, name, organisation, position of person, date)
Demonstrates the consequences of the adaption, adoption or use of their outputs.
- This is the long tail of impact, what has happened since the research was used by others leading to a change?
- What resulted from the use of the knowledge, program, practice, guideline and so on?
- Have they provided evidence to demonstrate the consequence?
  - Unfortunately, most researchers will not have this level of information, it is the hardest piece to track and capture.

Based on our above seven points, you can start to consider this as a rubric for scoring. As we previously stated, there is no perfect way to assess impact, but using these points as a base you can score either a yes or no answer. To be even more accurate, you could further divide each point to four parts (0.25) to provide a more precise score for applications where, for example, some elements are evidenced, but others are missing.

Field 2 – Program’s contribution to the research impact

This field is where the applicant must describe the degree to which the applicant’s research program was necessary to achieve the impact(s). Our interpretation – this field is where applicants write about their pathway to impact. This field should include the studies conducted, the papers published, the outputs delivered (e.g. in a knowledge impact example this is the publications and datasets). It should be noted that most applicants we have worked with tell us that Field 2 is the hardest one to understand and write.

We have identified six things that reviewers should expect to see in Field 2.

Six things to look for:

Clearly outlined or described the work that underpinned or led to the research outputs. (The use of these outputs and subsequent impact should have been discussed in Field 1).
- What was the research that was conducted leading to the development of the output?
- Have they told me the specifics of the studies within the program, how many, when were they conducted, what was explored/investigated? (Reviewers should not expect every detail of the research, such as participant numbers, methodology etc, but rather an overview of the program and the research conducted within the program.)
Underpinning research within the research program is clearly evidenced
- Have they provided details of the papers produced from the studies or that outline the outputs and key findings?
All work described within Field 2 is part of the program of work and is related to generating the outputs that were used for the impact described in Field 1
- They are only talking about the one research program and have not introduced new research topics.
- Do the outputs described align with those previously used in Field 1?
Translation of the outputs for impact
- Have they clearly shown or described the activities used to share the program outputs more broadly? (e.g conferences, collaborations, connections to policy etc)
Links to impact outlined in Field 1
- Have they provided a clear link between the underpinning research within the program and how it delivered the outputs that led to the impact in Field 1?
Verifiable evidence
- Have they provided verifiable evidence of the program’s contribution to the impacts in Field 1? For example:
  - Details of the programs conference presentations
  - Details of key connections and stakeholder involvement

Field 3 – Applicant’s contribution to the research program

This field should outline the role of the applicant within the research program. This field will differ between EL levels and Leadership levels due to opportunities and experience. It may be unlikely that someone at the EL level will have been lead CI or indeed senior author on all publications out of the program. In that instance, other considerations will need to be made regarding how the applicant describes their contribution. In reviewing over 300 draft versions of impact case studies, we identified seven elements that SHOW an applicant’s contribution, rather than just telling.

Authorship on the research program’s publications
- Have they told me how many of the program’s publications they have authored? (first or senior author papers)
Student/staff supervision
- Have they outlined the number and type of students or other staff they supervised as part of the program? (Eg number of PhD’s, Masters or Honors students and any research staff).
Grants and other funding sourced
- Were they part of, or responsible for winning grants that funded the program of work? (CIA, CIB etc)
- Have they told me where funding was sourced from?
Awards and peer recognition
- The applicant has won awards and accolades due to their contribution to the program.
- The applicant was invited to present a keynote/s on the program’s work.
Conference presentations and media
- Has the applicant played a lead role in sharing the program’s outputs more broadly? (e.g conference presentations, media exposure)
Collaborations
- Did the applicant form or manage key collaborations and partnerships with external stakeholders as part of the program?
Evidenced all claims
- Has the applicant provided sufficient verifiable evidence for all claims? For example,
  - Publications – citations of relevant papers
  - Grants – name of the funder and total dollar amount of funding, full titles of the grants or funding body and number of grant, date awarded
  - Awards – name of the award and the year won
  - Conference presentation – Name, location, date of conference
  - Collaborations – details of group, organisation, personal comms or testimonial

Finally remember, that as outlined in the NHMRC Investigator Grants 2021 Guidelines, “a poorly corroborated or non-corroborated research impact or contribution to impact will receive a score of one, in alignment with the category descriptors”.

Some common mistakes we have identified

Thinking that the impact has to be recent and in particular, that the research that led to impact needs to be recent! The guidelines explicitly state: “the research program that contributed to the research impact may be from any time in a researcher’s career – there are no time limits on when a researcher made a contribution to the research program or when the research program contributed to the research impact”.
Comparing applications to one another. This leads to further confusion and frustration as no two pathways and resulting research impact are the same.
Not reading the three impact sections as an entirety before beginning to score each section. The three impact sections must work together as an impact case study. Scores across the sections need to be considered within the context of a program of work and the elements undertaken by the applicant.

Overall, we hope that this unofficial guide helps you the reviewer to consider the impact case study in line with impact assessment methodologies. Our goal is to help, but we also recognise the value of the community in sharing their experiences, thought processes and indeed opinions on reviewing and scoring these sections. As such, we invite you to send your thoughts to us so to ensure an ongoing review and fine tuning of this guidance.

We invite your feedback and experience of reviewing as we continue to refine and develop this unofficial guide for assessing the Impact Track Record components within NHMRC grants.

Get a PDF version of the Guide

The information provided by Knowledge Translation Australia Pty Ltd in this blog post on www.researchimpactacademy.com is for general informational purposes only. All information in this blog post is provided in good faith, however, we make no representation or warranty of any kind, express or implied, regarding the accuracy, adequacy, validity, reliability, availability or completeness of this information.