The challenges of ‘measuring long-term impacts of a science center on its community’: A methodological review

Abstract

This chapter assesses Falk and Needham’s (2011) study of the California Science Center’s long-term impact on the Los Angeles population’s scientific understanding, attitudes, and interests. The study has been put forward as a good model of long-term impact evaluation for other researchers and informal science learning institutions to emulate. Moreover, the study’s claims about the Science Center’s positive impacts have been widely cited. To contribute to methodological development in informal science learning research, we critically examine the authors’ methods and claims, identifying major challenges to the validity and reliability of the research approach. We focus on sampling, data collection and analysis practices, including the use of self-report data, asking parents to report impact on behalf of children and non-representative sampling methods. An important innovation claimed by the authors is an indicator-based impact measure (a ‘marker’) designed to limit their reliance on self-report data. Our essay highlights this measure’s limitations, while pointing to alternative approaches that could more validly assess long-term learning or attitudinal impacts. We also outline directions for improving the statistical analysis and its interpretation. Ultimately, we conclude that Falk and Needham’s sanguine conclusions about the Science Center’s impacts are not empirically justified. We recommend that future research employ more direct measurements of learning outcomes grounded in established social scientific methodology to evaluate informal science learning impacts.