ARC’s ERA 2023 Benchmarking and Rating Scale Consultation
Tony Willis | MARCH 2022
A key issue for the Australian Research Council’s Excellence in Research for Australia (ERA) research evaluation program has been discrepancy between outcomes for STEM and HASS disciplines, and that the ‘world standard’ benchmark has not been well-defined and may not have been applied consistently between citation analysis and peer review disciplines. These issues were recognised in last year’s ERA EI Review Final Report 2020–2021.
Photo by Adi Goldstein on Unsplash
Chaired by Prof. Michael Brooks, the ERA EI Review Advisory Committee recommended “convening an expert working group to consider the technical aspects of revising the ERA rating scale, benchmarks, and the definition and appropriateness of world standard, to be completed for ERA 2023” including to consider “…methodological and disciplinary-specific needs, such as equity between humanities and social sciences (HASS) disciplines and science, technology, engineering and mathematics (STEM) disciplines.”
The ERA Benchmarking and Rating Scale Working Group recently published its consultation paper (see www.arc.gov.au/excellence-research-australia/era-2023). The paper invites submissions against specific questions concerning two new rating scale options – A and B. Both provide greater granularity compared with previous ERA rounds and both are supported by two new benchmarks – the ‘High performer’ and ‘World benchmark’. In addition to providing increased granularity, an important, long overdue development, is that the rating scales will include clear definitions and be accompanied by characteristics for each rating within a ‘Rating scale matrix’. This matrix is like the NHMRC’s peer review ‘Category Descriptors’ – an advance that is welcome as it increases clarity and provides enhanced transparency.
Option A was developed by the Benchmarking and Rating Scale Working Group. It retains a 5-point rating scale, as with previous ERA rounds, but provides increased granularity at the highest level by splitting the old ERA rating of ‘5’ into two classes – ‘World leading’ and ‘Well above world standard’.
Option B was developed by ARC staff and proposes a new 6-point rating scale, with the previous ERA rating of ‘5’ effectively split into three classes. Like Option A, this option invokes the ‘World benchmark’ standard to represent universities that are around the average standard of universities for a given discipline internationally. It also invokes the ‘High performer’ benchmark but applies it, in the highest possible rating category (currently termed “AAA”), to universities that are among the very small number of the best of the high performing institutions internationally.
The ERA EI Review Final Report 2020–2021 concluded that “…dual methodologies – the use of peer review or citation analysis depending on the discipline under review – are appropriate and remain necessary to reflect disciplinary differences in publication practices…”. Further, it recommended that “the dual methodologies of citation analysis and peer review continue in ERA”. Consistent with this recommendation and supporting proposed changes to the ERA rating scale, new metrics and indices will be introduced that provide, in some instances, dynamic bibliometric comparators, for citation-based disciplines. Because peer review disciplines still will be rated by expert judgement, the peer reviewers will be supported with additional guidance, particularly in relation to ‘Approach’ and ‘Contribution’.
Responses to the ARC’s current consultation paper are due by 22 April 2022.
