Evaluation of Probabilities: A Level Playing Field?

Abstract

Scoring rules provide overall measures of the "goodness" of probabilities, and other measures (often from decompositions of scoring rules) relate to specific attributes such as calibration and sharpness. This paper briefly reviews scoring rules and probability evaluation and attempts to extend past work in two directions. The first extension generalizes the development of scoring rules that better reflect the difficulty of forecasting situations and the skill associated with probability forecasts, with the goal of obtaining measures that yield comparable scores. The second extension involves the use of scoring rules to evaluate probabilities without perfect knowledge of the actual outcomes of the events or variables to which the probabilities refer. This is in the spirit of attempting to create a level playing field for probability evaluation by making evaluation measures comparable and expanding the set of probabilities that can be evaluated.