Promoting Research Transparency in Social Science Research
Social scientists have begun promoting the adoption of higher standards of transparency in order to improve both the quality and credibility of research.
With a predominating “dysfunctional reward structure” that incentivizes researchers to report data that is more publishable than accurate, it is important to realign scholarly incentives with scholarly values, especially since policy decisions based on research impacts the lives of millions of people.
In 2014, some of my colleagues and I wrote an article for Science’s Policy Forum. In it, we explain:
“[t]here is growing appreciation for the advantages of experimentation in the social sciences. Policy-relevant claims that in the past were backed by theoretical arguments and inconclusive correlations are now being investigated using more credible methods. Changes have been particularly pronounced in development economics, where hundreds of randomized trials have been carried out over the last decade.”
My colleagues – researchers leading the research transparency and reproducibility movement – and I converge on three core practices: disclosure, registration and pre-analysis plans, and open data and materials.
1) Disclosure calls researchers to abide by reporting standards that require them to “document and disclose key details about the data collection and analysis”. This includes all information about measures, manipulations, data, exclusions, and how they arrived at final sample sizes.
2) Registration and Pre-analysis plans (or PAPs) credibly distinguish hypothesis testing from hypothesis generation and exploratory research. Pre-analysis plans can include documents specifying statistical models, variables, and testing corrections.
3) Having open data and materials allows other researchers to review, revise, and reproduce results to further assess a study’s external validity.
Along with the adoption of these steps, organizations exist to facilitate the movement towards transparent research. The Open Science Framework (or OSF) is a platform where researchers can make their data public, while other similar organizations exist to archive randomized controlled trials (RCTs) or assist with study pre-registration. The Berkeley Initiative for Transparency in the Social Sciences, or BITSS, was also established to provide tools and resources that promote and facilitate research transparency.
With this new direction of research, social science research has the potential to provide an overall higher quality of evidence to policy-makers and other decision-makers. Knowing this, what reservations might researchers have to adopting these practices? If these three steps can, so significantly, reduce fraud, why aren’t more people taking them?
You can read the whole paper by clicking on the link in the SEE ALSO section at the bottom of this page.
Also, if you have a spare 20 minutes, take a look at the clip from John Oliver’s Last Week Tonight on Scientific Studies. He takes a humorous, if slightly cynical, approach to the crisis of reproducibility and introduces a few concepts we’ll get into later. You can find the link in the Related Links section at the bottom of the page.
Miguel, E., C. Camerer, K. Casey, J. Cohen, K. M. Esterling, A. Gerber, R. Glennerster, et al. 2014. “Promoting Transparency in Social Science Research.” Science 343 (6166): 30–31. doi:10.1126/science.1245317.
© Center for Effective Global Action