index.Rmd

---
title: "Making psychological science robust"
author: "Rick O. Gilmore"
bibliography: bib/psu-repro.bib
csl: bib/apa.csl
date: "`r Sys.time()`"
css: css/ioslides.css
logo: img/psichi_emblem2.gif
output: 
  ioslides_presentation:
    self_contained: false
    widescreen: true
    lib_dir: libs
    incremental: false
    transition: default
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = FALSE,
                      fig.align = "center")
```

# Preliminaries

## About me

- Professor of Psychology
- Founding Director of Human Imaging, PSU [SLEIC](http://imaging.psu.edu)
- Co-Founder/Co-Director [Databrary.org](http://databrary.org)

---

- B.A. in cognitive science, Brown University
- Ph.D. in cognitive neuroscience, Carnegie Mellon University
- vision, perception & action, brain development, open science
- ham radio operator (K3ROG), actor/singer, banjo-picker, hiker/cyclist, coder

## Overview

- Reproducibility in science
- Getting it wrong...and getting it right
- How to do better

# Reproducibility in science

---

```{r}
knitr::include_graphics("img/covid-19.png")
```

<div class="notes">
When scientists share things happen, we all benefit.
Scientists sequenced and openly published the genome of the Covid-19 virus early in 2020.
Vaccines of remarkable success were developed, tested, and deployed with astonishing speed.
Open sharing saved lives.
</div>

---

<iframe width="560" height="315" src="https://www.youtube.com/embed/66oNv_DJuPc" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<https://youtu.be/66oNv_DJuPc>

<!-- Data Sharing and Management Snafu in 3 Short Acts (Higher Quality) -->

## What proportion of studies in the published scientific literature are *actually true*?

>- 100%
>- 90%
>- 70%
>- 50%
>- 30%

## How do we define what "*actually true*" means?

## Is there a reproducibility crisis in science?

>- Yes, a significant crisis
>- Yes, a slight crisis
>- No crisis
>- Don't know

---

```{r}
knitr::include_graphics("http://www.nature.com/polopoly_fs/7.36716.1469695923!/image/reproducibility-graphic-online1.jpeg_gen/derivatives/landscape_630/reproducibility-graphic-online1.jpeg")
```

---

<!-- Scrolling final reference page -->
<!-- http://stackoverflow.com/q/38260799 -->
<style>
slides > slide { overflow: scroll; }
slides > slide:not(.nobackground):before {
  background: none;
  }
slides > slide:not(.nobackground):after {
  content: '';
  background: none;
  }
}
</style>

```{r, fig.cap="[[@baker_1500_2016]](http://doi.org/10.1038/533452a)", out.width="600px"}
knitr::include_graphics("https://media.springernature.com/w300/springer-static/image/art%3A10.1038%2F533452a/MediaObjects/41586_2016_BF533452a_Fige_HTML.jpg?as=webp")
```

## What does "reproducibility" mean?

- Are we all talking about the same thing?

## *Methods* reproducibility

- Enough details about materials & methods recorded (& reported) so that
- Same results with same materials & methods

[[@goodman_what_2016]](http://doi.org/10.1126/scitranslmed.aaf5027)

---

<div class="centered">
<img src="http://lh6.ggpht.com/_KVa1Tk_k1BU/TTjL-RSY_eI/AAAAAAAABtg/VQIfae1_wtQ/hit_thumb%5B3%5D.jpg?imgmax=800" height=550>
</div>

<div class="notes">
I like to call this the "hit by a truck" scenario.
</div>

## *Results* reproducibility

- Same results from independent study

[[@goodman_what_2016]](http://doi.org/10.1126/scitranslmed.aaf5027)

## *Inferential* reproducibility

- Same inferences from one or more studies or reanalyses
- Meta- or mega-analyses

[[@goodman_what_2016]](http://doi.org/10.1126/scitranslmed.aaf5027)

## Factors contributing to irreproducible research {.smaller}

```{r, fig.cap="[[@baker_1500_2016]](http://doi.org/10.1038/533452a)"}
knitr::include_graphics("http://www.nature.com/polopoly_fs/7.36719.1464174488!/image/reproducibility-graphic-online4.jpg_gen/derivatives/landscape_630/reproducibility-graphic-online4.jpg")
```

## Reproducibility crisis

- Not just psychology and related behavioral sciences
- "Hard" sciences, too
- Challenges affect data collection to statistical analysis to reporting to publishing

---

<div class="centered">
<img src="https://cdn.shopify.com/s/files/1/0877/5762/products/Rigor_Mortis_1024x1024.jpg?v=1491240110" height=550px/>
</div>

# Getting it wrong...and getting it right

## False positives & false negatives {.smaller}

```{r, fig.cap="https://i.reddituploads.com/cfb6336d162f4b908cb6715d3da752b5?fit=max&h=1536&w=1536&s=cb3b9e51ea5fef6fdc229fb24b740b7d"}
knitr::include_graphics("https://i.reddituploads.com/cfb6336d162f4b908cb6715d3da752b5?fit=max&h=1536&w=1536&s=cb3b9e51ea5fef6fdc229fb24b740b7d")
```

## Underpowered studies {.smaller}

```{r, fig.cap="[[@Szucs2017-fc]](http://dx.doi.org/10.1371/journal.pbio.2000797)", out.width="900px"}
knitr::include_graphics("http://journals.plos.org/plosbiology/article/file?id=10.1371/journal.pbio.2000797.g003&type=large")
```

---

> "*Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature.*"
>
> [[@Szucs2017-fc]](http://dx.doi.org/10.1371/journal.pbio.2000797)


## Getting the stats right...

```{r, out.width = "700px", fig.cap=="[[@Nuijten2015-ul]](https://doi.org10.3758/s13428-015-0664-2)"}
knitr::include_graphics("img/nuijten-etal.jpg")
```

---

```{r, out.height = "500px", fig.cap="[[@Nuijten2015-ul]](https://doi.org10.3758/s13428-015-0664-2)"}
knitr::include_graphics("https://static-content.springer.com/image/art%3A10.3758%2Fs13428-015-0664-2/MediaObjects/13428_2015_664_Fig3_HTML.gif")
```

## The toothbrush problem...

```{r}
knitr::include_graphics("https://www.rd.com/wp-content/uploads/2017/02/01-How-Bad-is-it-to-Share-a-Toothbrush-159311405-ABykov-760x506.jpg")
```

---

> "*...psychologists tend to treat other peoples’ theories like toothbrushes; no self-respecting individual wants to use anyone else’s.*"
>
> [[@Mischel2011-br]](https://www.psychologicalscience.org/observer/becoming-a-cumulative-science)

---

> "*The toothbrush culture undermines the building of a genuinely cumulative science, encouraging more parallel play and solo game playing, rather than building on each other’s directly relevant best work.*"
>
> [[@Mischel2011-br]](https://www.psychologicalscience.org/observer/becoming-a-cumulative-science)

## Openness, transparency, data and materials sharing are unrewarded...

> "*Reviewers and editors want novel, interesting results. Why would I waste my time doing careful direct replications?*"

> "*Reviewing papers is hard, unpaid work. If I have to check someone's stats, too, I'll quit.*"

-- Any number of researchers I've talked with

## To share or not to share... {.smaller}

```{r, out.height="500px", fig.cap="[[@wicherts_poor_2006]](http://doi.org/10.1037/0003-066X.61.7.726)"}
knitr::include_graphics("https://raw.githubusercontent.com/gilmore-lab/psu-data-repro-bootcamp-2017-07-10/master/img/wicherts_2006_amp_61_7_726_fig1a.jpg")
```

## [[@Vanpaemel2015-zr]](https://doi.org/10.1525/collabra.13)

| Response | Percent |
|----------|---------|
| No reply | 41% |
| Refused/unable to share data | 18% |
| No data despite promise | 4% |
| Data shared after reminder | 16%|
| Data shared after 1st request | 22% |

---

```{r, out.width="900px", fig.cap="https://doi.org/10.7554/eLife.67995"}
knitr::include_graphics("img/repro-cancer-bio.png")
```

## [[@Errington2021-dp]](https://doi.org/10.7554/eLife.67995) {.smaller}

```{r, out.width="900px"}
knitr::include_graphics("https://iiif.elifesciences.org/lax:67995%2Felife-67995-fig2-v1.tif/full/1500,/0/default.jpg")
```

## [[@Errington2021-dp]](https://doi.org/10.7554/eLife.67995)

> "*The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers.*"

---

> "*While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented.*"
>
> [[@Errington2021-dp]](https://doi.org/10.7554/eLife.67995)

---

> "*This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible*"
>
> [[@Errington2021-dp]](https://doi.org/10.7554/eLife.67995)

# How to do better

---

```{r, fig.cap='[[@munafo_manifesto_2017]](http://doi.org/10.1038/s41562-016-0021)', out.width="800px"}
knitr::include_graphics("https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs41562-016-0021/MediaObjects/41562_2016_Article_BFs415620160021_Fig1_HTML.jpg?as=webp")
```

<!-- Manifesto for reproducible science -->

<div class="notes">
This recent manifesto from Nature Human Behavior describes the risks to reproducible science at every step of the process.
I urge you to read it.
</div>

## Reproducibility in psychological science

- Bigger samples
- Multiple replications
- Registration
- Data, procedure, and materials sharing
- "Data blinding"
- Larg(er)-scale replication studies

## Reproducibility Project: Psychology (RPP)

[[@collaboration_estimating_2015]](http://science.sciencemag.org/content/349/6251/aac4716)

---

> "*We conducted replications of 100...studies published in three psychology journals using high-powered designs and original materials when available....The mean effect size (r) of the replication effects ...was half the magnitude of the mean effect size of the original effects...*"
>
> [[@collaboration_estimating_2015]](http://science.sciencemag.org/content/349/6251/aac4716)

---

> "*Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results.*"
>
> [[@collaboration_estimating_2015]](http://science.sciencemag.org/content/349/6251/aac4716)

---

> "*39% of effects were subjectively rated to have replicated the original result...*"
>
> [[@collaboration_estimating_2015]](http://science.sciencemag.org/content/349/6251/aac4716)

## [[@Camerer2018-tr]](Camerer2018-tr) {.smaller}

```{r fig.cap='[[@Camerer2018-tr]](Camerer2018-tr)', out.height="500px"}
knitr::include_graphics("https://mfr.osf.io/export?url=https://osf.io/fg4d3/?action=download%26mode=render%26direct%26public_file=True&initialWidth=711&childId=mfrIframe&parentTitle=OSF+%7C+F1+-+EffectSizes.png&parentUrl=https://osf.io/fg4d3/&format=2400x2400.jpeg")
```

---

```{r, fig.cap = '[[@Camerer2018-tr]](Camerer2018-tr)', out.height="500px"}
knitr::include_graphics("https://mfr.osf.io/export?url=https://osf.io/cefq7/?action=download%26mode=render%26direct%26public_file=True&initialWidth=680&childId=mfrIframe&parentTitle=OSF+%7C+F4+-+PeerBeliefs.png&parentUrl=https://osf.io/cefq7/&format=2400x2400.jpeg")
```

## Video as data and documentation

<div class="centered">
<img src="https://raw.githubusercontent.com/gilmore-lab/sips-2017-video-reproducibility/master/img/gilmore-adolph-nat-hum-beh.jpg" height=550px>
</div>

## Play & Learning Across a Year (PLAY Project)

- <https://play-project.org>
- "An hour in the life" of 1,000 families
- (Hyper)active data curation... [[@Soska2021-mh]](http://dx.doi.org/10.7191/jeslib.2021.1208)

## Improved statistical practices

- Automated checking of paper statistics (in American Psychological Association formats) via [Statcheck](http://statcheck.io)
- Redefine "statistical significance" as $p<.005$? [(Benjamin et al., 2017)](https://dx.doi.org/10.17605/OSF.IO/MKY9J)
- Or move away from [NHST](https://en.wikipedia.org/wiki/Statistical_hypothesis_testing) toward more robust and cumulative practices (Bayesian, CI/effect-size-driven)     - Better capture what we know or think we know

## Store data, materials, code in repositories

- Data libraries
- Funder, journal mandates for sharing increasing
- But no long-term, stable, funding sources for curation, archiving, sharing

## Script analyses, version control

- RMarkdown, Jupyter notebooks
- GitHub

## Open from the beginning...

---

```{r, fig.cap="http://cdn2.hubspot.net/hub/134568/file-1208368053-jpg/6-blind-men-hans.jpg", out.height="500px"}
knitr::include_graphics("http://cdn2.hubspot.net/hub/134568/file-1208368053-jpg/6-blind-men-hans.jpg")
```

## Science as a way of knowing...

- What do we know
- How do we know it?
- How do we share what we know (and how we know it)?
- Why should we (or others) care?

---

<div class="centered">

<video width="800" loop data-autoplay>
  <source src="https://github.com/gilmore-lab/DEVSEC-2018/blob/master/mov/databrary-splash.mp4?raw=true" type="video/mp4">
  </video>

rogilmore@psu.edu

<https://gilmore-lab.github.io>

<https://github.com/gilmore-lab/2022-02-23-psi-chi/>

</div>

## Resources {.smaller}

This talk was produced on `r Sys.Date()` in [RStudio](http://rstudio.com) using R Markdown.
The code and materials used to generate the slides may be found at <https://github.com/gilmore-lab/2022-02-23-psi-chi/>.
Information about the R Session that produced the code is as follows:

```{r session-info}
sessionInfo()
```

## References {.smaller}