[!IMPORTANT] Please read this, and read it thoroughly. The community counts on you to provide constructive, insightful feedback, and these guidelines reflect the broader communities wishes about how peer review proceed. Thank you!

As a group of volunteers ourselves, the ACM TOCE editorial board is incredibly grateful for your volunteer effort! Without our collective effort, peer review would not be possible. We've written this guide to achieve two goals:

1. Make your job easier, by scaffolding reviewing tips and strategies
2. Ensure your review aligns with our reviewing standards, which aim to provide timely, constructive, expert feedback.

Most reviewers report needing anywhere from 1 to 3 hours to read a paper and write a sufficiently detailed and insightful review. We hope the guide below helps you share your expertise with minimal overhead.

Why did we choose you?

Associate Editors (AEs) are charged with finding reviewers that have expertise in some aspect of a submission. You likely don't have expertise for every aspect of the submission, and it's okay to only speak to the aspects for which you have expertise. Trust that the AE will try to solicit complementary expertise from other reviewers, and combine your various judgements into a holistic recommendation to the Editor-in-Chief (EiC) for a final decision.

Sometimes, our community does not have sufficient expertise to review something, or the experts aren't available to review. If we've asked you to review something that's on the edges of your expertise, do your best to use your expertise to make a judgement, but also be mindful to avoid speaking to issues for which you don't have expertise. There's nothing more frustrating than receiving a review that seems to be performing expertise where there is none, simply because a reviewer was asked to review.

May I engage a "shadow" or "ghost" reviewer, such as a student?

No, not as a ghostwriter. We asked you for your expertise and so the review should be from you.

However, we recognize the critical importance of helping educate doctoral students and other learners about journal reviews and so as long as you maintain the confidentiality of the paper, you are welcome to use the submission as a way of helping others learn to review (e.g., having them write their own opinion, offer feedback on their reviews, and even incorporate some of their critiques in your own review). In fact, we hope this review guide is a helpful resource in scaffolding any peer review learning you facilitate.

As an alternative to this, consider recommending your student or colleague as the reviewer, and then they are welcome to share the submission with you, so that you may better advise them on their review and the reviewing process. We welcome new reviewers into our community and your efforts to support their reviewing practice!

For more on the issue, see this 2019 Nature Article and the research it summarizes on review ghostwriting.

When is my review due?

Our goal is that every submission receives a recommendation within 90 days of the monthly submission deadline. The journal staff reviews for basic anonymity and formatting, which takes a few days. The EiC reviews for scope, plagiarism, and high level quality, and then assigns an AE to manage the paper; this takes at least a week, sometimes two if multiple AEs need to be asked. The AE reviews for quality, optimally making desk reject decisions with another AE of their choice, or if they believe it might be publishable, they begin recruiting reviewers. This can take a few weeks.

This means that by the time you receive the invite, the paper might have been received up to a month ago. Therefore, we prefer that all aim for all papers to be reviewed within 30 days of accepting an invite. This ensures that once all reviews are received, the AE has at least 2 weeks to read the reviews, ask for revisions, and write a recommendation to the EiC, and at least 2 weeks for the EiC to make the final decision, or further discuss the recommendation with the AE. Therefore, the 30 day review window is essential to keeping the review cycle to 90 days.

Of course, if the timing isn't right, and you need more time, it's entirely appropriate to ask for an extra week or two. We'd much rather have your expertise on a paper a few weeks later than not have at it all. After all, expertise is scarce, and so we're happy to work around your schedule to make it feasible for you to review.

We expect you to submit your review on the timeline you agreed to, whether 30 days or an extended timeline. When you're late, it creates work for AEs to remind you, it delays recommendations to the EiC, it leads authors to inquire to the EiC about the status of the paper. Because lateness creates work for everyone else, please do your best to avoid whenever possible. Obviously, this isn't always possible, and so please do your best to only commit to reviewing timelines that are actually feasible for you.

Here are some reminder practices that many reviewers have shared with us:

• Reserve time on your calendar to review the paper
• Add a reminder on a to do list with a due date one week prior to the review deadline
• Print the paper out somewhere you will see it regularly as a prominent reminder
• Rely on the Manuscript Central reminders of upcoming review deadlines

[!IMPORTANT] Please do not "ghost" our editors. They work very hard as volunteers to try to secure reviews, and not responding to their outreach with updates, declines, etc. makes their volunteer work more difficult.

Where do I find submissions assigned to me?

We manage submissions in Manuscript Central. To access a paper:

1. Log in. If you need to retrieve your account name and password, look for the "check for existing account" link on the login page and enter the email address used to send you the request for the review. This will then send to you your login and password.
2. If you have not yet reviewed for ACM TOCE, AE will have created an ACM Manuscript account for you, but you'll need to fill out your profile.
3. Click on Referee Center
4. Find a section titled Manuscripts Pending Review, where you'll see the papers you're assigned.
5. Click on the paper's title to view the submission and download its PDF if you like.

What if the submission isn't accessible?

If the PDF that authors have submitted is not sufficiently accessible, write the Editor-in-Chief and they will request an accessible version from the authors so that you may read the document. (We're working on long-term efforts to help the journal move away from PDFs to more accessible formats, but we're not there yet).

How do I review a submission?

First, note that there are many ACM peer review policies. It's important to be aware of these in general, but we will try to include the most relevant ones here.

Verify eligibility

Before reviewing a paper (but shortly after you agree to review it), there are some important eligibility checks to do. (Don't save these for right before the deadline, as they may require finding an alternate reviewer).

While all of our submissions are double-anonymous (you don't know the authors identity and the authors don't know yours), you may realize as you first start reading that you actually do know the authors. If you have a conflict of interest with them, email the AE immediately so that they can unassign you from the paper and invite someone else. However, if you merely know the identity of the authors, you do not necessarily need to recuse yourself from reviewing: breached anonymity risks introducing bias towards or against the authors, but perhaps your expertise is essential for reviewing and this is a worthwhile compromise. If you're not confident you can manage this bias, you are free to ask the AE to find a different reviewer.

Some source of bias aren't necessarily conflicts of interest, but gaps in expertise. Perhaps you start reading the paper and realize you don't have the expertise to evaluate it after all. If this is the case, write the AE and consult with them. Your expertise may still be valuable given the lack of other available expertise, or you may have a recommendation for a better reviewer. We can't optimize expertise alignment for every submission, but if you see a way to do so without significant delay, it may be worth it.

Another source of bias is categorical opposition. Perhaps now that you've seen the paper up closely, you realize you have a deep opposition to its premises, its methods, or its values. In some cases of conflict values, it might be appropriate for you to review anyway: for example, if a submission perpetuated hate in a way that the EiC and AE overlooked, it might be important for you to strongly critique the work for its positions and argue for its rejection on those grounds. But in other cases of conflicting opinions, it might not be appropriate for you to review: for example imagine a paper explored a learning context that involved the C++ programming language, which you despise. That might color your opinion of the work so much that you wouldn't be able to evaluate its merits, which might have nothing to do with the language choice. If you don't feel you can be objective because of such strong feelings, it might be appropriate to recuse yourself.

There are likely many other reasons why you might decide to decline after accepting. Consult with your AE as any arise (and remember to do so immediately after accepting, to prevent significant delays in the review process).

Reflect on your positionality

Reviewing is fraught with implicit biases, assumptions, and unexamined values. We all are prone to these things; it takes explicit effort to identify them, and manage them, to fairly evaluate others' research. One way to manage these biases is to ask yourself these set of questions before reviewing a paper:

• What epistemological frames are you bringing to the evaluation? Are they appropriate for the work?
• What expertise are you bringing to evaluating the paper? What expertise are you not bringing that might be needed?
• How does your identity and positionality enable you to interpret the work's methods and significance? How do they limit your ability to do this?
• What is your goal in reviewing the work? (It should be to find reasons to publish, not to demonstrate your expertise or find fatal flaws).

These are questions to ask yourself every time you review, as they change over time, and the answers may vary by each paper you review.

Remind yourself of the reviewing criteria

First, familiarize yourself with the Author Guidelines. Authors are instructed to comply with those and ideally you are consistent with them.

Before reading the paper, it's also important to remind yourself of the ACM TOCE reviewing criteria. Not all journals or research communities have the same criteria or define them in the same ways, so it's important not to simply assume that your own personal criteria align perfectly with the journal's. This is especially important if you think of criteria that are important to you but are not a consideration for the journal (e.g., how much the paper would increase the "prestige" of the journal), or criteria that you typically don't evaluate, but that the journal does consider.

There are a few other benefits to reading the review criteria before reading the paper:

• It can help you read the paper from the perspective of the criteria.
• It can help you structure notes while reading in ways that align with the reviewing criteria.
• It can help structure your review against the criteria, helping AEs and authors more easily under your critiques.

Read the paper

There are many different reading practices. Here are some that reviewers have shared with us:

• Printing the paper and marking it up with colored pens
• Using a PDF reader and annotating it with notes
• Creating a text file to store and organize notes
• Reading the paper from beginning to end, taking notes while reading
• Reading the paper non-linearly (e.g., abstract, intro, discussion, then methods and results)
• Reading the paper multiple times, once for each reviewing criterion

The journal doesn't advocate any particular strategy; use whatever practice is best for you, as long as it aligns with the journal's reviewing criteria.

Write a review that evaluates the paper against the reviewer criteria

We ask that you structure your review using the reviewing criteria above, with a heading for each criterion. This is important for a few reasons:

• Authors will have read the criteria and tried to align their submission with them.
• Recommendations should be based on the criteria, and so structuring your critiques against the criteria makes the rationale for your judgement more transparent
• It will help you focus your critiques on the criteria, without unintentionally letting other considerations enter into your reasoning.

[!IMPORTANT] If you choose to deviate from the template below, please do so intentionally, and with good reason. Authors who receive reviews in this format tell us that it helps them better understand reviewers' critiques and the basis for their recommendations.

A typical review might therefore be structured as:

SUMMARY OF CONTRIBUTION

A few sentences describing:
1. The question or topic the paper poses
2. The answers or insights it contributed

PRIOR WORK

A few paragraphs critiquing and/or praising:
1. How the paper addresses prior work
2. Any prior work it missed, or praise for its coverage and insightfulness

SOUNDNESS

A few paragraphs critiquing and/or praising:
1. How the submission answered its research questions
2. Methodological gaps, flaws, opportunities, and questions

SIGNIFICANCE

A few paragraphs critiquing and/or praising:
1. The importance, novelty, and interestingness of the submission relative to prior work
2. The submission's discussion of implications

CLARITY

A few paragraphs critiquing and/or praising:
1. The submission's writing
2. Structural issues
3. Possible errors
4. Spelling + grammar issues

RECOMMENDATION

1. A paragraph summarizing your recommendation and how the four criteria were weighed.
1. A list of concrete changes you would recommend authors make to better meet the criteria, if you view that as feasible.

See the Associate Editor guidelines for details on the four possible recommendation categories.

Note that many reviewers submit reviews that do not follow these guidelines, often submitting one sentence, one paragraph critiques that do not address the criteria above, or address other criteria outside our journal's consideration. This is unacceptable, harms the reviewer's reputation, harms the journal's reputation, and harms the quality of scholarship in our community. Please do not do this :)

What tone should my review have?

Write your review about the paper, not the authors. You job isn't to critique them, their decisions, or their knowledge or expertise; it's to critique the document in front of you and the contributions it represents. You likely don't enjoy personal critiques in the reviews you receive, so avoid doing it to others.

Be constructive; that means focusing on what makes the paper worthy of publication and what needs to change to ready it for publication. Your job is to help authors make a plan for eventual successful publication, not to tell them everything they did wrong. We are a community of scholars that supports each other and builds on each others work, and peer review is a part of that.

Finally, lead with love: when you review something, remember that you are potentially writing to a current or future colleague, someone who might look up to you, or someone you look up to. Peer review should be a place where we teach each other and help each other make collective progress, not where we tear each other down through criticism and competition.

May I sign my review?

Yes. The confidential review process is primarily to serve you as the reviewer; the upside of anonymity is that you may feel more free to fully share your opinion of a submission. However, if you feel that disclosing your identity better serves your ability to evaluate a work, or better aligns with your values as a scholar, feel free to sign your reviews.

How do I submit my review?

When you are ready to submit your review in Manuscript Central:

1. Return to your Referee Center.
2. Click on the Review button.
3. Click on Score Manuscript
4. Complete the online reviewer form using your draft review
5. Submit

Your AE will read your review, ensure that it is aligned with the journal's criteria and standards and is sufficiently constructive, and then your job is done. You will receive a notification on the decision on the paper after the AE has received all reviews, made a recommendation to the EiC, and the EiC has made the final decision.

[!IMPORTANT] Thank you again for reviewing! Peer review wouldn't be possible without our collective volunteer efforts.