Eliciting Informative Feedback: The Peer-Prediction Method

Nolan Miller, Paul Resnick, and Richard Zeckhauser.

Management Science, Sept. 2005. 51(9), pp. 1359-1373.

Final pre-publication draft

Abstract

Many recommendation and decision processes depend on eliciting evaluations of opportunities, products, and vendors. A scoring system is devised that induces honest reporting of feedback. Each rater merely reports a signal, and the system applies proper scoring rules to the implied posterior beliefs about another raters report. Honest reporting proves to be a Nash Equilibrium. The scoring schemes can be scaled to induce appropriate e¤ort by raters and can be extended to handle sequential interaction and continuous signals. We also address a number of practical implementation issues that arise in settings such as academic reviewing and on-line recommender and reputation systems.