RSS Board >> reviewer_instructions

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
[[:home|Home]]

#####RSS 2011 Reviewer Instructions#####

-----

===== Key dates =====
 
  * Jan 10 Abstracts due\\
  * Jan 17 Paper deadline\\
  * Jan 30 Reviews are assigned\\
  * Feb 28 Reviews due \\
  * Mar 7 Author response begins\\
  * Mar 14 Author response ends, discussion begins, reviewers revise initial reviews \\
  * Mar 28 Discussion period ends\\
  * Apr 7  Area chair meeting \\
  * Apr 15 Acceptance announced\\
  * Jun 27 RSS in Los Angeles 

Please be guided by __**this**__ schedule, not the deadlines listed by CMT. All CMT deadlines are listed as 12/25/25. The CMT deadline is the time at which CMT turns off access to parts of the system. In the course of the discussion and the acceptance decisions, the area chairs may ask your (the reviewer) to clarify a point in your review. This cannot happen after the CMT deadline. To allow revisions to rebuttals or the discussion, we put the CMT deadlines to a point substantially in the future after the acceptances are announced. 

-----

===== CMT Instructions =====

  * First go to https://cmt.research.microsoft.com/RSS2011/ and login.
  * If you have never logged in before, you will need to generate a password. Enter your email, click the "Reset your password" button and follow the instructions. (You should have received an email from CMT sometime in the last three months thanking you for reviewing for us, and inviting you to log in. The email address that received this email is the email address you need to use.) 
  * Once logged in, you should be presented with a webpage that looks very much like this image (click image to enlarge) :

{{gallery>:public_images:reviewer_desktop.png?200x49&lightbox}}

  * Click on "Download Submission Files for All Assigned Papers" to get a zip file of all your assigned papers. 
  * Once you are ready to write your review, we strongly recommend writing your review offline first in an emacs or vi buffer. (CMT provides a mechanism for preparing a review offline as an XML file and then uploading it. We discourage this practice because it is easy to make a small error in creating the xml file, and the error message that results is not helpful. It is a failure-prone process.)
  * To enter your review into CMT, move your cursor over the "Submit reviews for" link. A drop down menu should appear with a list of your papers for review. 
  * Click on the appropriate paper, and you should be presented with a webpage that looks very much like this image (click image to enlarge):

{{gallery>:public_images:reviewer-panel.png?200x50&lightbox}}

  * The ranking scores will not be available during Feedback, only the written comments. The red text <html><span style="color:red;">Visible To Authors Only After Decision Notification, Visible To Authors During Feedback</span></html> is contradictory. It should say "Visible To Authors Both During Feedback and After Decision Notification". (We were unable to get this corrected by CMT.) 
  * Once you have entered your review, do not forget to click "Submit" at the bottom of the page. After initially submitting a review, you will still be able to modify (or delete) your review later on. 

-----

===== Detailed Review Instructions =====

All RSS papers should be solid scientific papers, regardless of their specific area. We judge the merit of a paper based on six criteria: 
  - Technical Strength; 
  - Quality of Evaluation; 
  - Significance/Relevance; 
  - Clarity; 
  - References to Prior work
  - Novelty. 

For each criterion you have to assign an a rating. Below we provide a detailed explanation of the different criteria and ratings. While all criteria are important, we want to encourage technically strong and rigorously evaluated papers, and therefore would like you to pay special attention to the first two criterion. However, unlike a journal, a conference needs to be a venue for papers that are contain preliminary experimental work or preliminary theoretical results. In any case, your review should be constructive (that is: aimed to improve the paper) and thorough. For each of the criteria, you should provide detailed comments justifying the evaluation along with suggestions for improving the paper. Furthermore, please provide specific information on which issues you would like the authors to address in their rebuttal.

Comments to avoid:
  * **Less helpful: "This work is a reinvention of XX" without a citation.** More helpful: "The authors unfortunately appear to be unaware of the work of XX presented in YY which largely overlaps with the results in this paper." 
  * Less helpful: "This work contains no new theoretical results." While robotics is to a great extent the marriage of good theory and good experimentation, we welcome papers that provide insight into robotics through rigorous experimentation. If you are unconvinced that the experiments support the claim, then a more helpful comment would be: "This is an experimental paper, but unfortunately the experiments do not give sufficient confidence in the authors' central claim. We would need to see the following experiments to be convinced of the authors' contribution." Or perhaps if the experimental claim is uninteresting, a more helpful comment might be: "The authors provide experiments to support the claim of XX, but this claim is somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the experiments do not support that result."
  * **Less helpful: "This work contains little or no experimental results."** While robotics is to a great extent the marriage of good experimentation and good theory, we welcome papers that provide insight into robotics through novel theoretical or algorithmic contributions. If you are unconvinced that the theory is correct, then a more helpful comment would be: "This is a theoretical paper, but unfortunately the authors make the following assumptions XX, YY and ZZ that undermine confidence that this theory can be applied usefully. We would need to see either experimental evidence, or a relaxation of those assumptions to be convinced of the authors' contribution." Or perhaps if the theoretical claim is uninteresting, a more helpful comment might be: "The authors provide a proof to support the claim of XX, but this claim is somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the proofs do not support that result."
  * **Less helpful: "This is just a systems paper."** Again, while robotics is to a great extent the marriage of good theory and good experimentation, we welcome papers that provide insight into robotics through system development. This probably means that it is an experimental paper, but you are unconvinced by the experimentation, or even the claim of the paper. A more helpful comment would be: "This is an experimental paper, but unfortunately the experiments do not give sufficient confidence in the insight the authors describe. We would need to see the following experiments to be convinced of the authors' contribution." Or perhaps if the experimental claim is itself unclear or uninteresting, a more helpful comment might be: "The authors provide experiments but do not give a clear hypothesis being tested by  XX. The results are somewhat unsurprising given the work in YY and ZZ. A more interesting claim might be QQ, but the experiments do not support that result." 
  * **Less helpful: "This paper is basically uninteresting."** This comment is not helpful to the author. Even if you, as a reviewer, are tired of reading papers in a particular area, this comment gives no guidance to the authors. Please avoid comments of this form. It's possible to say that a paper is narrow in the problem it addresses, without actually calling it boring. Describing a problem as narrow encourages the authors to broaden the scope of the paper. Describing a paper as boring encourages the authors to set the building on fire out of frustration. 

Guidance on the meaning of the criteria and their numerical scores (Thanks to Dieter Fox). 

  * Technical soundness: Is the paper technically sound? Are the concepts correct and accurate?
  * Quality of Evaluation: Are claims well-supported by theoretical analysis or experimental results? How convincing is the evidence in support of the conclusions? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work?
  * Significance: Are the results important? Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique theoretical or pragmatic approach?
  * References to prior work: Do the authors situate the work in the literature correctly? Do they differentiate their work clearly? Are they missing, or perhaps unaware of important pieces of prior work?
  * Clarity: Is the paper clearly written? Is it well-organized? (If not, feel free to make suggestions to improve the manuscript.) Does it adequately inform the reader? (A superbly written paper provides enough information for the expert reader to reproduce its results.)
  * Novelty: Are the problems or approaches novel? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?
  * Overall Rating:
    * Excellent: Top 10% of accepted RSS papers, a seminal paper for the ages. Clearly an outstanding paper. I assume no further discussion is needed.
    * Strong accept: Top 25% of accepted RSS papers, an excellent paper, a strong accept. I will fight for acceptance.
    * Weak accept: Marginally above the acceptance threshold. Some weaknesses, but I think they can be fixed by a good and kindly shepherd. I vote for acceptance, although would not be upset if it were rejected.
    * Weak reject: Marginally below the acceptance threshold. Some weaknesses, and I doubt they can be fixed in time, even with a good and kindly shepherd. I vote for rejecting it, although would not be upset if it were accepted.
    * Strong reject: A clear rejection. I vote and argue for rejection.
    * Poor: Trivial or wrong or known. Clearly below RSS threshold, I assume no further discussion is needed.
  * Confidence: 
    - Excellent: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature.
    - Very good: The reviewer is confident but not absolutely certain that the evaluation is correct. It is unlikely but conceivable that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature.
    - Good: It is possible that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature. Mathematics and other details were not carefully checked.
    - Poor: The reviewer's evaluation is an educated guess and it is quite likely that the reviewer did not understand central parts of the paper. Either the paper is not in the reviewer's area, or it was extremely difficult to understand

-----

===== General Guidelines for Reviewers =====

(Notes and guidelines thanks to Oliver Brock and Greg Dudek.)

  * "I once foolishly (in an irritable stressed-out moment) wrote a very strong negative and sarcastic review. Some time later a person I knew, liked and respected came to me to ask advice on where to send a paper they had written and how to revise it, in light of this terrible review they had gotten. Of course the review they showed me was the one I had written. Naturally, I felt awful." The moral of the story is, even a bad paper should get a polite humane review (although in such a case it still needs to be very clear about the quality, or lack thereof, in the paper.) - now a famous roboticist. 

  * "I had written a rather blunt, negative review about a paper of one of the well-known researchers in the field. Due to a hard disk crash, the wrong configuration files were restored for an unnamed conference administration system. As a result, the reviews sent to authors included the names of the reviewers. Needless to say that I was extremely embarrassed." - a roboticist who almost became famous, had it not been for this incident.

  * Begin a review by summarizing the problem and the proposed contribution of the paper.

  * Point out some positive aspects of the paper.

  * Please keep in mind that the key objective of the review process is to make a binary decision. In your comments to the editors, please make a specific recommendation and justify it briefly.

  * Justify your decision with a brief statement of the most important points you make in the remainder of the review.

  * Provide feedback that enables the authors to turn the paper into a "winning" paper.

  * Criticize the most important points in detail. For example, don't spend much time discussing revisions and alternatives to the nomenclature when the fundamental premise of the paper is flawed—focus on explaining the assumptions that are wrong.

  * Be constructive: for everything you criticize, you should provide suggestions on how to improve it.

  * If you accept the paper, avoid suggestions that cannot reasonably be accomplished in a limited time frame. If you recommend to reject the paper, you may decide to include recommendations that require significant work.

  * Only write things that you would tell the author in person. Keep in mind that no matter how bad you think a paper is, the authors worked very hard and strongly identify with their work. A critique of the paper is likely to be seen as a critique of the authors.

  * Put your review aside for a day and then re-read it.

  * Would you change the review if the author was a famous colleague and you had to sign by name?

  * Will your comments allow the author to improve the paper in a limited time so that you would accept the paper?

  * [[http://www.eecs.berkeley.edu/Pubs/TechRpts/1989/CSD-89-511.pdf | Here]] is what Alan Jay Smith from the University of California at Berkeley thinks about The Task of The Referee.