Open Science Fundamentals
PNB 3EE3

Maya Flannery

2023-09-18

Objectives

  • What is open science?
  • Why care about open science?
  • How do you do open science?
  • Connection to this course!

Background: The Replication Crisis

The Replication Crisis

Evidence for precognition—extraordinary claims! (Bem, 2011)

Issue

ESP is real — or science is broken…

Psychologists’ methods become under extreme critique (Wagenmakers et al., 2011)

Follow up

(Munafò et al., 2017)

Many studies do not replicate! (Klein et al., 2018)

Problems: Questionable Practices (QRPs and QMPs)

Questionable research practices (John et al., 2012)

Questionable measurement practices (Flake & Fried, 2020)

Failing to report

  • all dependent measures
  • all conditions

Probability of rejecting null hypothesis increases with the number of tests.

Selective reporting

Collecting data

  • After seeing whether results are significant
  • Stopping after achieving desired result
  • Excluding data after seeing impact of doing do

See p-hacking demo

Analysis

  • Rounding down p-values
  • Falsifying data
  • Claim results are unaffected by demographics
    • Western Educated Industrialized Rich Democratic (WEIRD) populations

Claim to have predicted an unexpected finding

HARKing (Kerr, 1998)

Why do researchers do this?

  • Career depends on ability to “publish or perish”
    • It is very difficult to publish null results

Solution: Open Science

FOSTER Open Science

Open Access

Open Data

  • Promotes reproducibility
  • Saves resources
  • Must be prepared carefully

Open Reproducible Research

  • Data stored, organized, formatted appropriately
  • Code is clean, readable, well-documented
  • Instructions for reproduction

Open Science Tools

Incentives

(Munafò et al., 2017)

Resources

More Things to Think About

Metametascience

(Whitaker & Guest, 2020)

Knowledge?

(Hall & Tandon, 2017)

University of Oxford

Open Science in PNB 3EE3

Reproducible Research and Open Science Tools

Open repositories

Open source/notebooks

Preregistration

OSF Prereg Template

Assignment

Summary

(Munafò et al., 2017)

References

Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425. https://doi.org/10.1037/a0021524
Flake, J. K., & Fried, E. I. (2020). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393
Hall, B. L., & Tandon, R. (2017). Decolonization of knowledge, epistemicide, participatory research and higher education. Research for All, 1(1). https://doi.org/10.18546/RFA.01.1.02
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kerr, N. L. (1998). HARKing: Hypothesizing After the Results are Known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., … Nosek, B. A. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225
Lakens, D. (2014, May 23). The 20% Statistician: A pre-publication peer-review of the “Feeling The Future” meta-analysis. The 20% Statistician. http://daniellakens.blogspot.com/2014/05/a-pre-publication-peer-review-of-meta.html
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie Du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 0021. https://doi.org/10.1038/s41562-016-0021
Wagenmakers, E.-J. (2014, June 25). Open Science Collaboration Blog · Bem is Back: A Skeptic’s Review of a Meta-Analysis on Psi. Open Science Collaboration. http://osc.centerforopenscience.org/2014/06/25/a-skeptics-review/
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & Van Der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426–432. https://doi.org/10.1037/a0022790
Whitaker, K., & Guest, O. (2020, September). #bropenscience is broken science. The British Psychological Society. https://www.bps.org.uk/psychologist/bropenscience-broken-science