ALMA Cycle 8 New Proposal Review Process Had Successful Results

20 April, 2022 / Read time: 3 minutes

Each ALMA observation cycle receives nearly two thousand proposals that need to pass through a selection process to prepare and optimize the observation schedule. In Cycle 8, ALMA implemented significant changes to the review process to make the process as fair as possible to the community and be better able to review the large number of proposals received every year. An extensive analysis of the results is published in two papers in the Bulletin of the American Astronomical Society and the Publication of the Astronomical Society of the Pacific. The results have been used to improve further the review process for ALMA Cycle 9, which will start in May 2022.

In the last cycle, a distributed system was implemented to make the review process more sustainable and inclusive: the same authors from the submitted proposals take part in the evaluation of other proposals. More than a thousand astronomers worldwide participated in the assessment of almost 1,500 proposals. This distributed system was previously tested successfully on the Cycle 7 ACA Supplemental Call for Proposals.

"This new distributed review process will help the Observatory face the challenge of an increasing number of submitted proposals by reducing reviewer workloads and making the review process more sustainable," said Jennifer Donovan Meyer, a scientist at the North American ALMA Science Center (NAASC), part of the US National Science Foundation's National Radio Astronomy Observatory (NRAO). After analyzing nearly 15,000 ranks and comments submitted by reviewers and collecting feedback from the principal investigators of the proposals through surveys, the Proposal Handling Team concluded that 90 percent of the proposal assignments matched the reviewer's expertise. Furthermore, principal investigators rated 73 percent of the review comments as helpful, with students and senior researchers providing equally valuable comments. "By correlating ranks and comments with various demographics, we were able to search exhaustively for systematics and identify areas in which the review process can be improved in the coming cycles," added Donovan Meyer.

"These results show distributed peer review is an excellent approach to reviewing a large number of proposals," emphasized Sean Dougherty, the ALMA Director, "with its much larger pool of independent reviewers providing a fairer review process, with a reduced reviewer workload, and it helps to reduce our carbon footprint. We are very grateful to the support and collaboration of the community in addressing these challenges so successfully."

"ALMA implemented dual-anonymous review in Cycle 8 to reduce biases in the peer review process," explained Observatory Scientist John Carpenter. Investigators do not know who reviewed their proposals, and reviewers do not know the identity of the investigators. "Our analysis suggests potential biases have been present related to the seniority of the principal investigator. This is a way of making the review process fairer for everyone."

Despite the significant process changes between Cycle 7 and Cycle 8 2021, the review process worked well. The Proposal Handling Team received only 153 support requests from over a thousand reviewers, and only nine proposals had significant violations of the dual-anonymous guidelines. "These results indicate that the community adapted well to the new proposal format and review process," explained Andrea Corvillón, Proposal Handling Team Lead. "The feedback from the community has been constructive, and we have made changes to make the review process even better in Cycle 9."

Additional Information

ALMA is a partnership of ESO (representing its member states), NSF (USA) and NINS (Japan), together with NRC (Canada), MOST and ASIAA (Taiwan), and KASI (Republic of Korea), in cooperation with the Republic of Chile. The Joint ALMA Observatory is operated by ESO, AUI/NRAO and NAOJ.