How to write a conference review - matthewdgreen/cryptoengineering GitHub Wiki

This page gives some notes and thoughts on how to write a good conference (sub)review.

My first piece of advice is: go read someone else's article on this -- there are many and all are much better than these quick notes. That said, here's a quick overview of the process that I will try to update and improve over time.

Step 0: Read the paper carefully. Maybe this seems obvious, but doing this right isn't so easy. Everyone does this differently, but my specific technique is as follows: first, I alway leave the abstract for last. I feel that reading an abstract is like reading the Cliff's Notes version of a book before reading the actual text. A paper should stand on its own without telegraphing its contributions in the abstract.

Next focus on the introduction. Did you come away with a clear idea of what the contributions are? Do you intuitively understand the techniques that are going to be presented in later technical sections of the paper? If you don't, that's a problem. You should include it in your review. This would be a good time to write a summary (paragraph) of the paper. (If it turns out that your summary at this stage does not match the actual body of the paper, that will probably be helpful in the rest of your review.)

Do focus on things like clarity of presentation. Make notes if you see a few obvious typos or mistakes. Include those in your detailed notes if they're helpful, but don't be a proofreader: if the paper is riddled with grammatical errors, you shouldn't waste your time finding them all -- just note that problem in your review comments. Remember that not every author speaks English as a first language; the best advice in this case is to advise the authors to find a proofreader. However, notice that this is not an excuse for a paper that is difficult for readers to understand.

Next, work through the technical portions of the paper. Are all the necessary ingredients explained? If there are elements that you don't understand because they're not described in the paper, make note of this. Look for obvious citations you'd expect to see, and examine the "related work" section. Do this carefully and make notes as you go: these will form the "detailed comments" section of your review.

Step 1: Summarize the paper. Ideally your summary shouldn't be more than a paragraph long. The point of this paragraph is to make sure that you've captured all of the contributions of the paper, and you've given your follow reviewers/authors a good idea of what's being captured in it. Don't assume that the person reading this paragraph has also read the paper.

Step 2: Give your overall impression of the paper. This is the most important portion of your review. It should ideally come immediately after the summary, and it should clearly indicate whether you think this is a valuable paper that should be accepted, or something that needs work and/or doesn't fit the conference. Nobody should finish reading this paragraph without a clear idea of where you stand on this paper's acceptance. You don't need to support all of your opinions in this paragraph (the support can go further down in the review). This is your thesis statement.

Also, I hope it goes without saying: be nice. If the paper isn't a fit, say so. If the paper is crummy, say that -- but in a polite way. You're speaking mainly to the conference PC and other reviewers here, but you are also speaking to the authors. Things that aren't ok to say: "I completely broke this idea", "the paper is terrible", "the ideas aren't that interesting". Try to be constructive because -- remember -- someone loves this paper the way you love your papers.

Step 3: Strengths and weaknesses. This varies from conference to conference, but many systems reviewers opt for a list of strengths and weaknesses. These should be meaningful technical comments on the paper's contribution, but "this paper is well written" (or vice versa) are also valid comments if you think that they apply.

Step 4: Summarize your thoughts on whether this paper should be accepted. This portion is optional, but this is generally where you give long-form prose comments on the paper. If you want to write multiple paragraphs justifying your conclusions in the previous paragraphs of the review, go for it. This is as much for the authors as the other reviewers.

Step 5: Detailed comments. I usually include these in the form of a bulleted list. A good review will have many of these. They can range from simple typos to major problems and/or questions you have for the authors. You should try to order these by importance. Don't give a list of 10 minor typos followed by a question like "By the way, [your main contribution of Section 5] seems to be broken, can you explain that?". But nothing is too small for this list. People are grateful when you find bugs in their paper.

This overview barely scratches the surface of what you need to do for a good conference review, but at least it should be helpful in getting the format and structure right.