CHI’23 made significant progress in smoothing the R&R process down to a single track (removing the two tiers of acceptance), and bringing the reviewer pool workload of Revise and Resubmit to be more manageable, whilst giving more authors the chance to improve their paper through the revise and resubmit process (49%) (rather than just speculating about change in a rebuttal). However, there were still some pain points and things we learned:
- we are experiencing significant paper bloat, with nothing incentivising concise shorter papers and nothing penalizing sloppy, unedited text;
- the review period was too short and created unnecessary stress;
- the workload especially around the December holiday period in many parts of the world was too tight to properly protect people’s vacation time; and
- the overall number of papers going to R&R created a large drain on the reviewer pool, and most papers at threshold point were not accepted despite author effort to revise.
For CHI’24 we wanted to keep the strengths of the 2023 process (including protecting people’s weekends and holiday times), but make a number of improvements.
Encouraging shorter papers and concise writing.
No-limits CHI papers and the wording of recent CfPs has led to an expectation that longer is better and that shorter isn’t enough. In the meantime, having no specific length limits meant that we lost something really valuable: CHI Notes – short concise papers that make a succinct small contribution (see e.g. CHI2016 papers vs notes).
- For CHI’24, we strongly encourage the submission of short concise papers that are commensurate with the old CHI Note format.
- Such short papers will be reviewed alongside full-length papers in the same process, but by labeling them as short, the process will make sure that their contribution is handled appropriately for a short paper format.
- At submission time, authors will declare if their paper is a) short (less than 5,000 words), b) full-length (average 7,000-8,000), or c) excessively long (12,000+ words). Authors will be able to qualify their choice with a statement of commensurate contribution.
- Excessively long papers may still be reviewed and accepted, but will be given additional scrutiny for whether they make a commensurately larger contribution, or simply have not been edited to be clear and concisely written. (And to answer your question: yes, the phrase “excessively long” was a specific design choice intended to provoke authors to carefully consider the length of their papers!)
Changing the R&R threshold calculation
CHI’23 put 49% of papers through R&R, and all except 150 of those resubmitted. 60% of resubmitted papers were then accepted in round 2. This was a lot of work for authors and required a lot of reviewing workload for papers that were not then accepted (particularly those around the borderline). ACs also reported that the 2023 threshold rule of ‘any reviewer declaring R&R’ took power of decision away from the role of the AC. For 2024, we will use the threshold that a paper will be put through to R&R if any AC recommends it. 1ACs can make a judgment in relation to the reviews provided, and both ACs can also discuss the recommendation between them if needed.
- Considering CHI’23’s data, changing the threshold to any AC recommending R&R (instead of any reviewer) would bring the percentage of papers to 42% from 49%. In practice, looking more computationally, this would affect approximately 150 submissions where in 2023 a reviewer gave an R&R but no AC made this recommendation.
- This will reduce the expectation of authors to revise papers that are unlikely to be accepted, as well as reduce the overall workload on the reviewer pool.
- This threshold will be re-evaluated after CHI’24.
Timeline pain points
- Feedback was almost unanimous that the period of finding reviewers and for reviewers to produce a review was too short in CHI’23. This has been increased for 2024 by a week.
- Similarly, feedback was that despite trying to avoid the common December holidays in many parts of the world, the timing was too tight to wrap up final reviews and recommend decisions prior to the virtual PC Meeting. An additional week has been added, pushing decisions back by one week, in comparison to 2023.
Other changes
- The Quick Reject process has been removed (now that R&R allows for significant changes to papers), but the remit of the Desk Reject criteria has been slightly expanded. This reduces the task load prior to review requests being released, allowing us to create that additional week in the review process.
- Best Practice recommendations and expectations have been revised for both Accessibility and Inclusivity.
- We have altered the language around ‘Rigor’ to ‘Research Quality’ such that rigor is less likely to be confused for just experimental rigor. Research Quality should reflect what is expected in different disciplines across the CHI community.
- We plan to introduce greater oversight of review quality and recommended review length guidance for all reviewers in order to redress the few (but not insignificant) number of times that authors receive short and inadequate reviews.
- In conjunction with the TPCs, we plan to update the policy and guidance around anonymisation, and the use of tools like ChatGPT in submissions.
With these changes, we hope that the process is even smoother and more manageable, as we strive to maintain a high quality process and reasonable workload expectation on our community. More details will be available in the Call for Papers, which will be available on the CHI website soon.
Best,
Irina Shklovski, Phoebe Toups Dugas, Max L. Wilson
CHI2024 Papers Chairs