How to Review Michael Bieber Informatics Department Ying Wu College of Computing New Jersey Institute of Technology (version in Estonian and Swedish)

Outline

Introduction
Process Considerations
Be Sure to Include in your Review
Aspects to Consider
Whether to Correct Grammar and Spelling
Editors: What You Can Do in Return for Referees
Sample Referee Forms
Your Comments, Please

Introduction

This guide concerns refereeing for conference and journal articles. It follows the philosophy that the reviewing process benefits the author as much as the editor. Authors deserve detailed feedback in a non-hostile tone, regardless of whether you recommend rejecting, revising or accepting their work. Do the quality of review that you wish others would do for you. By agreeing to review, you take on the responsibility of doing a thorough job. If you cannot commit to this, you should notify the editor at once and return both the paper and reviewing package. Editors understand that you may not have the time to do a thorough review, but are unforgiving if you commit to a review and then do a poor job. Also, good reviewing does not go unnoticed by editors, and can establish a useful reputation for you over time.

Authors also have a responsibility to reviewers. Reviewers volunteer time from busy schedules to conduct thorough reviews, and are happy to do so for well thought-through pieces. If you submit premature work, not only will you annoy your colleagues who take the time to referee, but you will gain an unsavory reputation over time. Researchers work in rather small communities and reputations spread.

Process Considerations

Plan to read the submission three times: the first reading to get a feel for the paper, the second to read the paper in depth, and the third to actually mark it up. Fill out the refereeing form right after the third reading, while things are still fresh in memory.

You can split your review into three sections: (a) the actual refereeing form; (b) general comments on the paper; and (c) specific comments on individual passages in the paper. You could mark up section (c) directly on the paper. Unless disallowed, you can return the editor a copy of the paper with your markup, to be returned the author. Some reviewers make a photocopy of the paper and transcribe their comments clearly on the copy. Otherwise you can type your comments separately, clearly referencing the page, section, paragraph and/or line number.

You can write the editor a separate, confidential, note if you wish. Be sure to mark it "confidential" so the editor doesn't forward it by mistake. Many refereeing forms have a specific area for confidential comments for the editor. Do not feel obligated to write things to the editor that you do not share with the authors. In general authors benefit from as much as you can tell them.

Do not be afraid to reject a bad paper; i.e., do not feel obligated to accept a paper that you believe needs more work. Readers don't want to read poor pieces, and authors will benefit from revising and publishing quality work. Papers can be poor for many reasons. Authors may not explain themselves clearly, so the concepts, motivation, background, actual results and contributions do not come across well. The paper may require drastic proofreading. The research may be of low quality or not a significant contribution. The research may be premature to publish. The paper may have been "thrown together" (unsuccessfully) at the last minute to meet a deadline. It may not fit the characteristics of the intended journal or conference. It may too similar to other publications the author has made (though some journals will publish slight modifications of papers that have appeared in conference proceedings---check with the editor if you have any questions). For poor papers you still owe the editor and the author a review (or in the worst cases, a thorough explanation). If the paper is too premature to complete all parts of the review, state this in your comments.

Be professional and non-hostile in your review. If you get angry with an author (e.g., for doing a poor job or not for proofreading), you can say so -- either directly to the author in your comments or just to the editor.

Be Sure to Include in your Review

As part of your review, be sure that you clearly justify your rating for each item on the refereeing form.

You should begin your comments by summarizing the paper in some detail. This will convey what you understood from the paper. This can show the author aspects that the reader misses or interprets differently from what the author expects.

You might choose to clarify to what extent you are (or are not) an expert in the paper's domain (especially if this is not asked on the review form). This can provide a level of context for your comments.

Aspects to Consider

This is a partial list of what to look for when reviewing a submission. The items should help both to guide your evaluation, and form helpful comments and suggestions for the author. Not all are necessary or even desirable for all papers. Be sure to read the refereeing form, as this will give you additional criteria for the particular journal or conference.

• Does the abstract summarize the paper, the results and the contributions, or mostly motivate the work?
• Does the author cover everything promised in the introduction?
• Does the author provide adequate motivation?
• Does the author provide adequate background information?
• Are all descriptions clear? Are the tables and figures clear? Do they make sense on their own or only if one has read the text carefully? Are there too many? Would an additional table or figure help? Would a (running) scenario or an example help?
• Are the research contributions clear? Are the contributions significant?
• Is the approach clearly explained and well laid out? Does the author justify each of the points made?
• Are the equations, algorithms, methods, experiments, and conclusions: correct, robust, comprehensive and sensible?
• Is the research properly ground in the literature and appropriate theory?
• Has the author expressed the limitations of the research and the author's approach?
• Has the author performed a complete analysis and drawn insightful conclusions?
• Has the author described his or her future research plans? Is it clear where the research described in this paper will lead and what the next step will be?
• Is the conclusion significant? Is it just a rehash of the paper? Does it provide new synthesis or insights? Does it leave the reader excited about the research, the research domain or the future?
• Does the author use important, current and adequate citations? Are there too many citations? Are any irrelevant or insignificant? Are they at an adequate level for the publication (e.g., research oriented vs. trade journals; technical reports vs. journal articles)? Are there too many citations to the author's own work? Can you suggest any missing citations the author may have overlooked?
• How is the author's writing style? Is it too "dense" to make sense? Does it keep the reader's interest? Is it too informal? Note that an informal style in itself sometimes is very effective in getting a paper's ideas across. Similarly, many authors use humor very effectively in research papers. Only if the informality or humor gets in the way, should it be discouraged. (On the other hand, certain disciplines do enforce very formal writing styles, in which case an informal style is deemed inappropriate.)

Whether to Correct Grammar and Spelling

Proofreading includes checking for correct grammar, correct spelling and overall, that a paper "reads well." As you know, spelling checkers check neither grammar nor comprehension. Authors should have enough respect for the reviewers and the editors to submit a paper that has been thoroughly proofread. Authors who are not native English speakers (or whatever language the forum allows) are responsible for ensuring that their submission is of the quality a native speaker would submit, even if they must pay someone to help in the editing process.

Nevertheless, as a reviewer you will often find small spelling or grammatical mistakes the author has overlooked (e.g., a typo within a correction made after employing a spell-checker). And of course you may be able to suggest better ways to phrase certain passages in the paper. In all these cases, it is up to you to decide the extent to which you edit the paper. You may decide to correct the first couple of pages, or the first couple of cases of a recurring problem. If the paper requires major corrections and you know a corrected draft will be reviewed again, you may suggest the author undertakes such proofreading as part of the revision process.

Editors: What You Can Do in Return for Referees

A note for editors. Many referees appreciate feedback on their reviews. Like many other aspects of academia and research, reviewing is a learning process. You may consider sending each reviewer the same package you send the author, i.e., a copy of your correspondence with the paper's author, as well as a copy of each of the paper's reviews (including his or her own, in case the referee didn't make a copy). This makes the reviewer feel more a part of the process and gives valuable feedback.

In addition, many referees are building tenure and promotion files. A written acknowledgment (not email) of the referee's help looks good in these files and is much appreciated.

Sample Referee Forms

Here are some example referee forms, for your information.

HICSS'95 Minitrack on Hypermedia in Information Systems and Organizations

HICSS'96 Minitrack on Hypermedia Research

Your Comments, Please

I would welcome your comments on these guidelines.
Please email them to bieber@njit.edu - thanks!