How to measure the quality of research: who is DORA and why does it matter for PGRs?

In this post, Judith Hegenbarth, Head of Research Skills in Library Services, introduces the responsible use of research metrics and UoB’s Commitment to Responsible Research Assessment.

"We love DORA" badge

Any government minister will tell you that performing research costs money, and that public spending on it has to be justified.  The allocation of research funding is based on a perception of ‘quality’, and part of the equation is whether an individual, research group or institution has performed ‘quality’ research in the past.

Metrics

Measuring quality is a contentious issue, particularly when it concerns the ‘performance’ of an individual researcher or scholar.  In the past, the number of times a publication has been cited by other researchers has been used as a proxy for influence and thereby quality.  The h-index became a shorthand for author excellence.  This kind of metric has been shown to privilege certain fast publishing disciplines which produce multi-authored papers.  For those researchers who take career breaks to raise families, or lone scholars who publish larger works less frequently, a single measure isn’t helpful or fair.  There’s more discussion of this on our Influential Researcher intranet page (including Canvas course)

In a similar way, metrics that attempt to indicate which journals have the most prestige, such as the Journal Impact Factor (JIF), perpetuate a belief that if an article is published in a particular venue, it must be excellent.  Rather than looking at the quality of each piece of work, the ‘branding’ is used as a shortcut.  But what about journals which are used by practitioners and have real world impact, and what about new interdisciplinary areas of research, where new journal titles are springing up, and don’t yet have a build-up of citations?

UoB’s commitment

The University of Birmingham’s recent ‘Commitment to Responsible Research Assessment’ makes an explicit assurance that the measurement of the quality of research will not be reduced to single metrics.  The institution is a signatory to DORA (San Francisco Declaration on Research Assessment), which sets out how to use metrics responsibly. It also strongly endorses open research as a method of ensuring ‘global reach’.  

Why does this matter to PGRs?

As emergent researchers, you will not yet have an established body of work on which to be judged. Alongside publishing, you will also want to develop other aspects of your professional self; learning about how to conduct research as openly as possible to enable collaboration, mentoring others, beginning to peer review publications and possibly teaching undergraduates.  Wouldn’t it be great if the contributions to your discipline weren’t only measured by how many citations you generate at the end of a very long process?  Commitments to value a whole set of elements of a researcher’s career, rather than just their publications could be the first steps to changing our research culture for good.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Helen Kara

Writing and research

PostGradual: The PhD Careers Blog

Taking PhD careers one tip at a time

Think: Research

Because there's always room for improvement

Research & Scholarship Skills

Handy hints for PhD students

Think Ahead Blog

from the Researcher Development team at the University of Sheffield

Jobs on toast

Helping you move on in your career - with Chris Humphrey, PhD

patter

research education, academic writing, public engagement, funding, other eccentricities.

The Thesis Whisperer

Just like the horse whisperer - but with more pages

%d bloggers like this: