Evaluation of the 2022 Peer Circle Experiment of the Humboldt Foundation

In the context of digital transformation and the increasing stress on the academic peer review system, the Humboldt Foundation is looking to refine the peer review procedure on which all its selection results are based. The goal is to develop a technologically and academically appropriate procedure that guarantees high quality reviewing as a basis for fair selection.

In March 2022, the Humboldt Foundation launched the pilot phase of a new, digital, interactive review procedure. It was initially used in the Humboldt Research Fellowship Programme to assess applications in inorganic chemistry, materials science, zoology and modern history. Whilst the previous procedure usually involved the submission of two independent specialist reviews for each application, the peer-circle process now being tested is based on a group of six to ten reviewers comparing and discussing several applications on a protected platform.

During the pilot phase in 2022, the peer circle process was evaluated externally. The results were positive, and it achieved its goal of delivering statements on the validity, reliability and transparency of the procedure. In the evaluator’s view, the peer circle process is at least as effective as the classic procedure in every respect, and even has some definite advantages. In particular, the peer circle review process is well suited to programmes that sponsor individuals because all selection criteria (in the Humboldt Research Fellowship Programme: academic career, past performance, project proposal, future perspectives) are assessed in a balanced way.

Specifically, the evaluation focused on six questions or topics:

1. The quality of the peer review. This was at least as high as in the classic peer review procedure and tended to be even higher due to the greater objectivity resulting from a larger number of reviewers and the option to compare several applications. Subject coverage was generally appropriate; in specific cases, for which specialist expertise was not available, external experts were coopted for individual reviews.
2. The risk of premature consensus in the sense of suppressing dissenting views. This was recognised as a risk but did not occur. Rather, reading other people’s opinions tended to have a positive effect on the critical reflection of the reviewers’ own assessments. In general, discursive consensus building was judged to be a positive feature of the process, as it led to clearer recommendations for the committee members. 
3. The acceptance of the procedure. This was high in all four subject areas, i.e., amongst most of the participants. The peer circle was seen as a promising solution to the problems associated with the existing peer review system. The opportunity for collaborative and comparative review was considered an advantage in developing reviewers’ own reviewing and proposal skills.
4. The efficiency of the process. This, too, was ranked highly. In comparison with the classic procedure, the peer circle process meant less time was spent on each application. Although the overall workload for the individual was greater due to assessing several applications, the process saves the community a lot of time. During the pilot phase, a total of 27 experts reviewed all 89 applications, whereas up to 178 experts would have been needed under the traditional system. At the same time, it was possible to expand and diversify the pool of experts in the peer circle process without compromising quality. The procedure also tended to be more efficient than the classic approach regarding the length of selection committee discussions.
5. The quality of the selection results. Based on the committee's scores and the applicants’ bibliometric data (only in the field of chemistry), it was examined whether the review procedure had a potential impact on the decisions; no significant difference between the two procedures was found. Regarding bibliometrics, a surprisingly positive finding was that funded applications did not perform significantly better than rejected applications in either procedure, suggesting that other selection criteria are weighted more heavily than indexes and impact factors.
6. Gender balance in selection outcomes. Again, the peer circle process had no significant effect on this overall. However, since the results were of limited significance due to the low numbers at the level of individual disciplines, this would need to be evaluated further.

The results show that the peer circle process combines a lean evaluation procedure with high quality standards. The Humboldt Foundation now aims to extend the process to further research areas in the Humboldt Research Fellowship Programme.

• Evaluation report on the 2022 peer circle experiment at the Alexander von Humboldt Foundation (PDF, 1 MB)