Professor Thomas Pfeiffer and PhD student Michael Gordon.
Massey University are contributing to a year-long research project into a new system for assessing how likely social and behavioural science studies can be independently reproduced or replicated.
The Defense Advanced Research Projects Agency (DARPA) is developing the Systematizing Confidence in Open Research and Evidence (SCORE) program to develop and deploy automated tools to assign "confidence scores" to different social and behavioural science research results and claims. Confidence scores are quantitative measures that should enable someone to understand the degree to which a particular claim or result is likely to be reproducible and/or replicable.
Massey’s Professor Thomas Pfeiffer, who leads the experimental design for the project, says the team at Massey is incredibly proud to be part of this collaboration. “DARPA has an amazing history of supporting research on forecasting. And the team from KeyW has a cutting-edge market platform that proved successful in earlier forecasting competitions. We are confident that our research will yield novel insights into how reliably claims in scientific research are, and how the scientific process itself can be streamlined."
The Replication Markets effort is unique in that forecasts will be collected for claims from 3000 published studies. An independent team will test 5 -10 per cent of these studies by mid-2020. Forecasters will compete for over $100,000 of cash prizes. Similar projects conducted in the past, also involving Massey, operated at a scale of 20-40 claims.
Replication Markets is funded by DARPA’s SCORE project, with Principal Investigator Dr Charles Twardy of KeyW, a wholly owned subsidiary of Jacobs, and researchers from Harvard, Massey, and George Mason Universities, the University of California, Santa Cruz, the University of Virginia, and the Stockholm School of Economics. For more information.