Creating a rapid reviewer pool…..and trying to use it
In our first post we introduced you to the COVID-19 cross-publisher collaboration and what we were aiming to solve by working together with at least 20 publishers, scholarly communication organizations and industry experts. Here we reflect on our initial experiences.
To recap, journals were overwhelmed with COVID-19 submissions and many of the most appropriate reviewers were busy dealing with the pandemic, so we created a group of ‘rapid reviewers’ for COVID-19 research papers. This group would agree, up front, to review rapidly (5 days), to have their reports made open (if applicable) and to have their reports transferred between journals in the group (if authors were rejected at one title and wished to submit to another journal in the collaboration).
Reviewers signed up in large numbers (~1800 at October 2), and so far 119 have been invited directly by publishers from the rapid reviewer list. At present we have no way of knowing how many have been contacted to review outside of the list – for example the researchers might be reviewing but just aren’t yet being tracked by the publishers as also being on the rapid reviewer list. There have also been challenges in making the pool of rapid reviewers available to the right people at the right time during the peer review process and identifying the most appropriate potential reviewer out of such a large group of people.
This leads us to ask whether there is anything that could be improved. We conducted a survey of those who signed up to be reviewers, receiving responses from around 15% of them. Respondents’ experiences mirror data collected by the publishers directly, although 21% of our respondents were invited to review for any one of the participating journals.
More than twice as many (47%) respondents have been invited to review C19 papers outside of the initiative. There is some variation in numbers geographically, but the trend holds. 75% of those contacted outside the initiative had not been asked to review papers through it.
Feedback suggests more information about their areas of expertise, or a link to their publication history, might have been helpful in persuading editors to invite them. This might explain why this group has been invited to review more outside of the initiative, than inside, where it is perhaps easier to connect with a wider range of data about the reviewer. An additional caveat here is that it is entirely possible that a reviewer has in fact been invited by an editor/journal because they are on this list, but the reviewer may themselves not be aware of this reason, since the journal may not necessarily be referring to the list in the review invitation.
C19 has acted as an accelerant in many areas including around open science but on this admittedly limited data we don’t see it definitively leading to a growth in new peer reviewers being invited, trained and mentored.
Outside of journal peer review 12% have actively commented on C19 preprints and 8% commented on PREreview, a partner in the initiative, directly. These global academics are more engaged in preprint reviewing than we might have anticipated and a minority are ready to help improve the work of their peers outside of a formal peer review process.
The experience of respondents in the initiative has been pretty positive – even if they haven’t been invited to review just yet. Reviewers from South America, Africa and Asia are the most positive on average.
Respondents who have been invited to review as part of C19RR are most positive on average, whereas those who’ve yet to be contacted aren’t as favourable. Both groups of respondents are equally happy about the initiative if they’ve been invited (or not) outside the rapid review collaboration.
The more we can engage with this group the better. They represent academics in global regions publishers don’t usually reach and as they remain positive about the collaboration (even if they haven’t been invited to review yet), we can potentially use this group of engaged, global academics to provide feedback on preprints in even greater numbers. We will be tracking their response to PREreview over the coming weeks and will be reporting on the data in the near future.