Statistical power in observer-performance studies: Comparison of the receiver operating characteristic and free-response methods in tasks involving localization

Abstract

Rationale and Objectives. Statistical power, defined as the probability of detecting real differences between imaging modalities, determines the cost in terms of readers and cases of conducting receiver operating characteristic (ROC) studies. Neglect of location information in lesion-detection studies analyzed with the ROC method can compromise power. Use of the alternative free-response ROC (AFROC) method, which considers location information, has been discouraged, because it neglects intraimage correlations. The relative statistical powers of the two methods, however, have not been tested. The purpose of this study was to compare the statistical power of ROC and AFROC methods using simulations. Materials and Methods. A new model including intraimage correlations was developed to describe the decision variable sampling and to simulate data for ROC and AFROC analyses. Five readers and 200 cases (half of which contained one signal) were simulated for each trial. Two hundred trials, equally split between the null hypothesis and alternative hypothesis, were run. Ratings were analyzed with the Dorfman-Berbaum-Metz method, and separation of the null hypothesis and alternative hypothesis distributions was calculated. Results. The AFROC method yielded higher power than the ROC method. Separation of the null hypothesis and alternative hypothesis distributions was larger by a factor of 1.6 regardless of the presence or absence of intraimage correlations. The effect of the incorrect localizations during ROC analysis of localization data is believed to be the major reason for the enhanced power of the AFROC method. Conclusion. The AFROC method can yield higher power than the ROC method for studies involving lesion localization. Greater consideration of this methodology is warranted.