An ACM TOCE Associate Editor (AE) has three responsibilities:

1. Editing submissions assigned by the EiC
2. Contributing to new and revised journal policy
3. Helping attract submissions and disseminate published work

Most of the role's effort goes to #1. However, #2 and #3, while small amounts of time, can be quite important to the broader mission of the journal. Below we discuss how to do each well.

What are the journal's reviewing standards?

All academic communities have evolving standards about rigor. The best way to understand the standards we try to uphold for ACM TOCE is to read two things:

• The Author Guidelines. Read these to understand how we communicate our expectations to authors. These evolve over time as the board discusses and debates new standards.
• The Reviewer Guidelines. Read these to understand how reviewers are guided to work with you and conduct their review.
• The Reviewing Criteria. Read these to understand the judgments we expect you and reviewers to make about submissions. These are mirrored in the author guidelines.

When you first join as an AE, read this carefully. But they will change, so also be sure to participate in discussions about changes, and refer back for the latest guidance.

Why was I assigned as Associate Editor for this paper?

A common question that AE's have is why they were assigned something (especially if they're feeling like they're being assigned too much!). Of course, trying to balance AE availability, expertise, and editing commitments with submission volume is far from a science. This is how the EiC tries to balance these things for each monthly submission cycle:

• First, the EiC desk rejects all papers out of scope, so we have a clean list of papers needing AE assignments
• Next, the EiC identifies all AEs who have 2 or fewer active assignments. (Assignments can go on for months, depending on how long it is taking to get reviews, but we aspire to ensure you aren't managing more than 3 at any given time).
• The EiC sorts this list first by the total number of submissions each AE has managed over the past 12 months, removing any AEs that have managed more than 12 from consideration. Within each group of number of submissions in the past 12 months, the EiC sorts from fewest active submissions to most.
• Going through each submission needing assignment, in order of submission, the EiC scans the sorted AE list for the first available AE with suitable expertise and without apparent conflicts of interest.
• After all submissions are tentatively assigned, the EiC sends assignments, with rationale for the assignment.

This whole process takes about 2 hours, for the roughly 12-20 submissions we receive each month, about 5-7 minutes per submission.

If there are not enough AEs for the number of submissions needing AEs, or there are not enough AEs with expertise on a topic, the EiC begs AEs with 3 current assignments to take a 4th, and then begins another surge of AE recruiting efforts to try to address the gap.

What is the Associate Editor's process?

This is the core job of an AE and builds upon an understanding of the journal's reviewing standards. It generally involves the following steps:

1. Read the paper
2. Decide whether to review
3. Invite reviewers
4. Remind reviewers
5. Review reviews
6. Make a recommendation
7. Handle revisions

Steps 1-6 above should ideally take no more than 90 days.

Each of these contains some important steps. Note: feel free to reach out to the EiC at any time for guidance on any of these steps (Discord or email). It's better that we collaborate on this process than do it alone.

Step 1. Read the paper

When you first receive an assignment from the EiC, log in to Manuscript Central:

Click the Associate Editor Center link:

Look for the "Awaiting Referee Selection" link in the Associate Editor list:

Find the View Submission link under the paper title to see the submission's PDF:

You want to first verify a few things:

• Do you have a conflict of interest with the authors? If you do, it's important to write back to the EiC as soon as you can to let them know, so they can assign a different AE to manage the submission. ACM details what counts as a conflict of interest in its detailed policy.

• Do you have the expertise to edit? You don't have to be the world's foremost expert on the paper's topic, but you should have enough expertise to be aware of the literature, and know who the world's foremost experts are. Skim the paper to see what methods it uses and what literature it cites. If you aren't at all sure who might review, write the EiC back and see if there's an AE available with more expertise. If there's not, the EiC may have you review it anyway, or they may try to recruit a new AE that does.

• Are you available to edit? As an AE, you should commit to editing assignments from the EiC, but there are many reasons why you might not be available. Don't hesitate to reply to the EiC and explain the situation; burnout is the last thing we want to drive a peer review process.

Step 2. Decide whether to review

Assuming you're eligible, available, and sufficiently expert, you'll next make one of two decisions.

The first possible decision is Reject without Review. The EiC is supposed to catch these prior to assigning an AE, but sometimes the case is ambiguous and they will assign an AE to get a second opinion. These are reasons you might reject without review:

• The paper seems out of scope.
• The paper is not research, based on the definition we provide in the journal scope.
• The paper is not sufficiently anonymized. (See the author guide on what "sufficient" means.)
• The paper is so incoherent that no reviewer will understand it.
• The paper appears to have a plagiarism issue within the scope of ACM plagiarism policy; Manuscript Central provides a report of potential plagiarism that might help detect some issues.

If you see one of these issues, consult with the EiC within one week of being assigned when possible. If they agree with the issue, they will unassign you from the paper and reject it without review, explaining the rationale to the authors.

The second possible decision is Reject. This is different from Reject without Review in that you will solicit a review from a second AE, and so it is technically reviewed, but only by one other person, to get at least one more opinion on your judgement. These are reasons you might choose to Reject:

• Unjustified length. (There is no set limit, but there are a variety of reasons why something might be appropriate, including verbosity, or something being the wrong genre, like a book or dissertation. Note that qualitative work is always longer than quantitative work, and this its extra length is justified).
• The paper is missing crucial relevant discourses (e.g., missing entire communities of work, as opposed to one or two references).
• The paper starts from premises that have been deeply, unequivocally refuted by a consensus of prior work (e.g., the Earth is flat)

If you choose to Reject, think of the review process as the same as with any other submission, but instead of finding three reviews, you're finding one, and instead of recruiting from the broader community, you're recruiting an Associate Editor from the editorial board. Once you get a review from another AE, you'll submit a recommendation to the Editor as normal, and the Editor will make the official Reject without Review decision. Authors will see your recommendation, the AE review you solicited, and the Editor's decision letter.

To do this, your first step is to invite an AE to review. Review the editorial board for a list. Use Manuscript Central's person search to find the secondary AE (email is the most reliable query):

Once you've found them, assign them as a reviewer by clicking the checkbox next to their result and pressing the add button:

Set the “# reviews needed to make decision" to “1”:

Find the referee list at the top of the page and press the invite button to invite the second AE. A popup window will appear with a draft email; modify it to state that you are seeking a second opinion on the paper to decide whether it is suitable for peer review. Request that the opinion be returned within 14 days. When you receive the secondary AE's opinion, submit the recommendation to the EiC for review and they will approve the recommendation.

Remember that when you desk reject, the author is only receiving feedback from two people; this puts a bigger burden on you to offer constructive critique, since you will not have external reviewers doing that work for you. Be thoughtful, kind, and helpful, offering authors concrete guidance on what they could have done differently.

If neither of the above decisions are appropriate, then you'll start a peer review process.

Step 3. Invite reviewers

If you've decided to review a paper, your first job is to solicit three high quality reviews within 1 month of submission. The goal is to find reviewers that can offer expert opinions on the work and do not have a conflict of interest with the author. Ideally, all three reviewers would be world experts on the topic (though this is frequently not possible).

To begin, start by making a list of at least 6 potential reviewers, ideally more, as some might be conflicts and some might decline. Construct your list entirely in Manuscript Central so that if someone else needs to take over your work, they have access to the research you've done to find potential reviewers.

You can identify reviewers in many ways:

• Your own network
• Suggested reviewers from the authors, which might be available in Manuscript Central
• Authors cited in the submission
• Suggestions from other AEs, which you can solicit on the journal's Discord #board channel
• Our Reviewer Pool, which is a simple spreadsheet of people who have expressed interest in reviewing
• Doing a lightweight literature review to understand the important work on the research topic
• A senior doctoral student you advise, if you commit to mentoring and guiding them on their review, you are careful to not coerce them into it (intentionally or unintentionally — be mindful of your power), and you do so only occasionally.

Once you have a list of potential reviewers, verify a few things for each one:

• They do not have an institutional conflict with an author
• They are not a TOCE AE (AE's are not expected to review, though they can if expertise is scarce)
• The group of reviewers cover the necessary expertise for evaluating the work (literature, methods, phenomena)
• They are not already reviewing more than 1 or 2 papers for the journal

Rank the list by expertise, and then assign the top three on your list in Manuscript Central. The assignment process in the system is a bit odd:

• First you must find the reviewer in the system by their name or email address
• If they don't exist in the system, you must create an account for them using their email address
• Once you've found them, you add them to the "Referee List"
• Once added to the list, you must press the "Invite" button, which will create a popup and a draft email. Revise the email, personalizing it if you know the reviewer, and send when you're ready.

Send all email through Manuscript Central. This ensures that if anyone needs to manage the paper other than you, they have a record of communication with reviewers that they can refer to before writing them again.

After sending out three invitations, your task is done until you hear back from reviewers.

• If someone declines, invite the next person on your ranked list as soon as your are able. Ask for suggestions you are out of people on your list.
• If someone accepts, thank them and say that you're looking forward to their review.
• If someone asks for more time, it is okay to grant it. It's more important that we get reviews on a timeline compatible with reviewers' lives than that we get it after a precise time period.
• If someone does not reply within a week, reach out again and see if you can get an answer either way.

If you run out of potential reviewers, and everyone is declining, there are a few options:

• Reduce the number of required reviews to 2. Before you do this, write the EiC to verify that this is okay in this case. You should have exhausted all of your options before doing this.
• Invite an expert AE to review. Explain the situation, and see if they are willing.
• Invite the EiC to review. If they have sufficient expertise, it is ultimately their responsibility to evaluate submissions, and so are one fallback.
• Unsubmit the paper. If it really is impossible to review the work, it is reasonable for the EiC to notify the authors that they were unable to find volunteers to review, and return the work to the authors for potential submission elsewhere.

Step 4. Remind reviewers

Ideally, reviewers would return reviews within 30 days. This allows the journal to make a recommendation within about 2 months of submission. However, this frequently does not happen. Manuscript Central will send out some automated reminders to reviewers, but if a reviewer is more than 2 weeks late, write them through Manuscript Central and see if they need an extension. It is okay to negotiate a new date; you can use the Grant an Extension link in the Referee List, so that Manuscript Central sends reminders at the appropriate time.

If they don't reply or submit a review within a month of when they first agreed, consider inviting another reviewer from your list, assuming that they will not ultimately submit a review.

If a substantial amount of time has passed and an AE has received two thoughtful reviews, it is acceptable to make a recommendation based on only those two reviews, provided that the two reviews are in good agreement. To make a recommendation with fewer than 3 reviewers, find the "# reviews required to make decision" field, change it to 2, and then submit a recommendation to the EiC.

If in the middle of the review process, you or an author notices an issue that should have been identified before inviting reviewers (plagiarism, anonymity), consult with the EiC about whether to continue the review process or whether the EiC needs to coordinate with the authors to potentially withdraw the paper from review. In general, anonymity issues not caught prior to review, while introducing some potential for bias toward or against the authors, is on the editorial board, not the authors, and so the paper will remain in review. This avoids punishing authors for the board's oversight.

Step 5. Review reviews

When a review arrives, check it for a few things:

• Ensure the review has a justification for its recommendation (empty reviews are unacceptable).
• Ensure it uses the review form integrated into the Manuscript Central System. If reviewers ignored certain criteria, write them back and ask them to complete the review.
• Ensure it complies with our (forthcoming) reviewer guidelines. If it doesn't, write them back and ask them to revise the review.
• Ensure it is free of major spelling and grammar errors.

Don't hesitate to write reviewers back and ask for revisions; it can be hard to get reviews at all, let alone high quality reviews, given the weak incentives to review, but it is our duty to ensure that reviews are sufficient, fair, and clear, so that authors see ACM TOCE, and our community in general, as a source of highly insightful, constructive, and fair critique.

Note that there is a difference between reviews that could be improved, and reviews that are harmful to authors. Examples of harm include:

• Personal attacks on authors' identities, expertise, knowledge, or choices (as compared to critiques on a manuscript)
• Racist, sexist, homophobic, transphobic, or ableist rationales for rejecting research

We should not share such reviews unfiltered — reviewers do not have a right to have their voice heard if it is ultimately destructive to our research community. That said, making judgements about such content is not always easy. Consult with the EiC if you have questions about ideas, statements, or phrasing of reviews. We can identify an appropriate remedy, including asking the reviewer to revise their review, discarding the review, or sharing the review with editorial caveats.

Step 6. Recommend

Once you receive all three reviews, within 1 week, re-read the paper and the reviews and write a meta-review, along with a recommendation to the EiC for publication. Your meta review should include:

• A summary of the reviews, including both strengths and weaknesses of the submission against the evaluation criteria.
• Your own opinion of the submission relative to the evaluation criteria, if you have sufficient expertise to weigh in.
• If you are recommending revisions, a clear enumeration of the revisions you believe are necessary to make the work acceptable.
• A recommendation to the EiC and rationale for the recommendation, linked to the reviews and your opinion. It is not necessary for every paper to satisfy all criteria. However, there should be a good reason if some criteria is weak or overlooked; there should be significant strengths elsewhere.
• If the paper is accepted, and you believe it deserves broader visibility by being presented at the next SIGCSE technical symposium, give a rationale for why. The EiC will use these nominations to select invited authors.

Try to frame your critiques as constructively as possible, focusing on the merits of the work and coupling any critiques with concrete suggestions for how to address them.

Possible recommendations include:

• Accept. You think the paper is ready to publish, without revision. You might recommend minor changes to fix typographical errors or improve clarity. This decision type is extremely rare on the first try, but is possible, especially for a paper that might have been a resubmission from a conference or different journal.

• Accept with Revisions. There are some revisions you think are necessary and would like to review them before publication, but not send them out again for review. Authors are given up to 60 days to resubmit a new version of the paper that incorporates the revisions recommended by the editors and reviewers. Because the revisions to the paper are minor, the revised manuscript is not sent out for a second round of peer review. Instead, the associate editor evaluates the changes and renders a decision. Note that a paper may generally receive a decision of "Accept with Revisions" only once.

• Revise and Resubmit. Revisions could make the submission acceptable, but they should be reviewed again, ideally by the same reviewers, before publishing (with the possibility that the revisions do not make the submission acceptable, and thus might be rejected. The authors may choose to submit a revised manuscript that addresses these issues. The revised manuscript will be sent out for a second round of peer review, preferably using the same reviewers as reviewed the original manuscript. Note: if the revised manuscript fails to receive a decision of “Accept with Revisions” or “Accept" after a second round of review, you should generally reject it, unless you're certain that yet another round of reviews will lead to a publishable work. Whether to choose Revise and Resubmit is ultimately a prediction: if the paper has a chance of becoming publishable, based on what you know now, I think it’s worth giving authors that chance. But if everything you know about a manuscript suggests that it either a) it wouldn’t be publishable, or b) you can’t know without seeing a substantial revision, then it’s okay to reject the paper. That sends a strong signal that the paper would have to be very different to be considered for publication.

• Reject. The submission will not be acceptable without substantial revision, and should not be further reviewed.

Use Manuscript Central to enter your meta-review and recommendation. You'll create a draft email that includes these, and then submit it to the EiC for review and approval. The EiC makes a final decision on the paper, based on the reviews and the meta-review, generally within one week of receipt of the AE’s recommendation. The EiC ultimately communicates this decision to the authors, along with the reviews and AE meta-review.

Step 7. Handle revisions

If the authors decide to make revisions, they submit a revised manuscript along with a detailed cover letter that summarizes changes and responds to the points raised in the reviews and meta-review. Read the cover letter, review the summary of changes, and repeat the process above again, defaulting to the original AE and reviewers, until either the author does not return with a satisfactory revision or the paper is accepted. If you need to recruit a new reviewer, return to your ranked list of potential reviewers for ideas.

How can I contribute to journal policy?

There are countless ideas about how to make peer review better; we've captured many of these priorities in this repositories issue tracker. The EiC captures, prioritizes, and works through these issues through a lightweight change process that involves feedback from the editorial board and the broader community.

Part of your role as an AE is responding to change proposals brought forth by the EiC. If a change proposal is particularly exciting to you, volunteer to collaborate with the EiC to help shape, revise, and implement the change.

How can I help make the journal visible?

On occasion, the EiC will ask the board to help share the journal to the computing education research community and also share work published in the journal. This might include:

• Sharing calls for submissions
• Sharing events and content related to published articles or issues
• Amplifying tweets by the @acmtoce Twitter account

Not everyone is active on social media, but if you are, your help in disseminating the journal's activities and work is greatly appreciated!