How to Review
Be Sure to Include in your Review
Aspects to Consider
Whether to Correct Grammar and Spelling
Editors: What You Can Do in Return for Referees
Sample Referee Forms
Your Comments, Please
This guide concerns refereeing for conference and journal
articles. It follows the philosophy that the reviewing process
benefits the author as much as the editor. Authors deserve detailed
feedback in a
Authors also have a responsibility to reviewers. Reviewers volunteer time from busy schedules to conduct thorough reviews, and are happy to do so for well thought-through pieces. If you submit premature work, not only will you annoy your colleagues who take the time to referee, but you will gain an unsavory reputation over time. Researchers work in rather small communities and reputations spread.
Plan to read the submission three times: the first reading to get a feel for the paper, the second to read the paper in depth, and the third to actually mark it up. Fill out the refereeing form right after the third reading, while things are still fresh in memory.
You can split your review into three sections: (a) the actual refereeing form; (b) general comments on the paper; and (c) specific comments on individual passages in the paper. You could mark up section (c) directly on the paper. Unless disallowed, you can return the editor a copy of the paper with your markup, to be returned the author. Some reviewers make a photocopy of the paper and transcribe their comments clearly on the copy. Otherwise you can type your comments separately, clearly referencing the page, section, paragraph and/or line number.
You can write the editor a separate, confidential, note if you
wish. Be sure to mark it "confidential" so the editor doesn't forward
it by mistake. Many refereeing forms have a specific area for
confidential comments for the editor. Do not feel
Do not be afraid to reject a bad paper; i.e., do not feel obligated to accept a paper that you believe needs more work. Readers don't want to read poor pieces, and authors will benefit from revising and publishing quality work. Papers can be poor for many reasons. Authors may not explain themselves clearly, so the concepts, motivation, background, actual results and contributions do not come across well. The paper may require drastic proofreading. The research may be of low quality or not a significant contribution. The research may be premature to publish. The paper may have been "thrown together" (unsuccessfully) at the last minute to meet a deadline. It may not fit the characteristics of the intended journal or conference. It may too similar to other publications the author has made (though some journals will publish slight modifications of papers that have appeared in conference proceedings---check with the editor if you have any questions). For poor papers you still owe the editor and the author a review (or in the worst cases, a thorough explanation). If the paper is too premature to complete all parts of the review, state this in your comments.
Be professional and non-hostile in your review. If you get angry with an author (e.g., for doing a poor job or not for proofreading), you can say so -- either directly to the author in your comments or just to the editor.
As part of your review, be sure that you clearly justify your rating for each item on the refereeing form.
You should begin your comments by summarizing the paper in some detail. This will convey what you understood from the paper. This can show the author aspects that the reader misses or interprets differently from what the author expects.
You might choose to clarify to what extent you are (or are not) an expert in the paper's domain (especially if this is not asked on the review form). This can provide a level of context for your comments.
This is a partial list of what to look for when reviewing a submission. The items should help both to guide your evaluation, and form helpful comments and suggestions for the author. Not all are necessary or even desirable for all papers. Be sure to read the refereeing form, as this will give you additional criteria for the particular journal or conference.
Proofreading includes checking for correct grammar, correct spelling and overall, that a paper "reads well." As you know, spelling checkers check neither grammar nor comprehension. Authors should have enough respect for the reviewers and the editors to submit a paper that has been thoroughly proofread. Authors who are not native English speakers (or whatever language the forum allows) are responsible for ensuring that their submission is of the quality a native speaker would submit, even if they must pay someone to help in the editing process.
Nevertheless, as a reviewer you will often find small spelling or
grammatical mistakes the author has overlooked (e.g., a typo within a
A note for editors. Many referees appreciate feedback on their reviews. Like many other aspects of academia and research, reviewing is a learning process. You may consider sending each reviewer the same package you send the author, i.e., a copy of your correspondence with the paper's author, as well as a copy of each of the paper's reviews (including his or her own, in case the referee didn't make a copy). This makes the reviewer feel more a part of the process and gives valuable feedback.
In addition, many referees are building tenure and promotion files. A written acknowledgment (not email) of the referee's help looks good in these files and is much appreciated.
Here are some example referee forms, for your information.
HICSS'95 Minitrack on Hypermedia in Information Systems and Organizations
HICSS'96 Minitrack on Hypermedia Research
I would welcome your comments on these guidelines.
Please email them to email@example.com - thanks!