Threats to validity

# The Four Horsemen<br>of Validity

**Session 6**

]

---

# Plan for today

.box-3.medium.sp-after-half[Construct validity]

.box-1.medium.sp-after-half[Statistical conclusion validity]

.box-5.medium.sp-after-half[Internal validity]

.box-6.medium.sp-after-half[External validity]

---

name: construct-validity
class: center middle section-title section-title-3 animated fadeIn

# Construct validity

---

.box-inv-3.medium[A new program hopes to<br>improve student commitment to school]

.box-inv-3.medium.sp-after[Participants score 200 points higher on the SAT and have a 0.3 higher GPA, on average]

.center.float-left[
.box-3.medium[Success!]&ensp;&ensp;.box-3.medium[Success?]
]

---

---

# The Streetlight Effect

.center[
<figure>
  <img src="img/06/streetlight.jpg" alt="Streetlight" title="Streetlight" width="70%">
</figure>
]

???

Drunk guy looking for keys in the light of the lamppost instead of over in the bushes where they lost them

Source: <https://pxhere.com/en/photo/488829>

---

# Construct validity

.box-inv-3.medium[Are you measuring what you want to measure?]

.box-3.sp-after[Test scores measure how good kids are at taking tests]

.box-inv-3.medium[This is why we spend so much time<br>on outcome measurement construction!]

---

layout: false
name: statistical-conclusion-validity
class: center middle section-title section-title-1 animated fadeIn

# Statistical conclusion<br>validity

---

---

# Statistical conclusion validity

.box-inv-1.medium[Are your statistics correct?]

---

# Power

.center.small[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Person </th>
   <th style="text-align:left;"> Group </th>
   <th style="text-align:center;"> Before </th>
   <th style="text-align:center;"> After </th>
   <th style="text-align:center;"> Difference </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 295 </td>
   <td style="text-align:left;"> Control </td>
   <td style="text-align:center;"> 122.09 </td>
   <td style="text-align:center;"> 229.04 </td>
   <td style="text-align:center;"> 106.95 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 126 </td>
   <td style="text-align:left;"> Treatment </td>
   <td style="text-align:center;"> 205.60 </td>
   <td style="text-align:center;"> 199.84 </td>
   <td style="text-align:center;"> -5.76 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 400 </td>
   <td style="text-align:left;"> Control </td>
   <td style="text-align:center;"> 133.25 </td>
   <td style="text-align:center;"> 130.40 </td>
   <td style="text-align:center;"> -2.85 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 94 </td>
   <td style="text-align:left;"> Treatment </td>
   <td style="text-align:center;"> 270.11 </td>
   <td style="text-align:center;"> 206.56 </td>
   <td style="text-align:center;"> -63.54 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 250 </td>
   <td style="text-align:left;"> Control </td>
   <td style="text-align:center;"> 344.37 </td>
   <td style="text-align:center;"> 222.89 </td>
   <td style="text-align:center;"> -121.49 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 59 </td>
   <td style="text-align:left;"> Treatment </td>
   <td style="text-align:center;"> 312.41 </td>
   <td style="text-align:center;"> 268.06 </td>
   <td style="text-align:center;"> -44.35 </td>
  </tr>
</tbody>
</table>
]

---

# Power

]

]

---

# What's the right sample size?

.center[
<figure>
  <img src="img/06/power-search.png" alt="Google power calculator" title="Google power calculator" width="50%">
</figure>
]

---

# Test assumptions

.box-1.smaller[For instance, for OLS:]

.center.float-left.smaller[
.box-1[Linearity]&ensp;.box-1[Homoscedasticity]&ensp;.box-1[Independence]&ensp;.box-1[Normality]
]

.box-inv-1.medium.sp-before[Make sure you're doing the stats correctly]

---

# Fishing and p-hacking

.box-inv-1[Wouldn't it be awesome to run thousands of models<br>with different combinations of variables<br>until you find coefficients that are statistically significant?]

.box-1.large[Don't!]

.center[
<figure>
  <img src="img/06/p-hacking.png" alt="p-hacking" title="p-hacking" width="60%">
</figure>
]

???

<https://projects.fivethirtyeight.com/p-hacking/>

---

# Spurious statistical significance

.box-inv-1.sp-after[If *p* threshold is 0.05 and you measure 20 outcomes,<br>1 will likely show correlation by chance]

.center[
<figure>
  <img src="img/06/xkcd.png" alt="xkcd: significance" title="xkcd: significance" width="60%">
</figure>
]

???

<https://xkcd.com/882/>

---

layout: false
name: internal-validity
class: center middle section-title section-title-5 animated fadeIn

# Internal validity

---

---

# Internal validity

.center.float-left[
.box-inv-5[Selection] .box-inv-5[Attrition]
]

.center.float-left[
.box-inv-5[Maturation] .box-inv-5[Secular trends] .box-inv-5[Seasonality] .box-inv-5[Testing] .box-inv-5[Regression]
]

.center.float-left[
.box-inv-5[Measurement error]

.center.float-left[
.box-inv-5[Hawthorne] .box-inv-5[John Henry]
]

.center.float-left[
.box-inv-5[Spillovers] .box-inv-5[Intervening events]
]
]

---

# Selection

.box-inv-5.less-medium.sp-after[If people can choose to enroll in a<br>program, those who enroll will be<br>different from those who do not]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Randomization into<br>treatment and control groups]

---

# Selection

.box-inv-5.less-medium.sp-after[If people can choose when to<br>enroll in a program, time might<br>influence the result]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Shift time around]

---

???

(happier people more likely to get married, so without randomly assigning marriage how would you study the impact of marriage on happiness?). They use a simple approach - since happiness varies over time set marriage equal to time zero and build a pre-post design around it. You essentially leverage the within-group variance and iron out across-age differences because of the varying ages of marriage. The whole insight it to change the time-line from calendar years to program years.

<https://doi.org/10.1016/j.socec.2005.11.043>

---

---

---

---

---

# Attrition

.box-inv-5.less-medium.sp-after[If the people who leave a program or<br>study are different than those who stay,<br>the effects will be biased]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Check characteristics of those<br>who stay and those who leave]

---

# Fake microfinance program results

.small.sp-after[
<table>
 <thead>
  <tr>
   <th style="text-align:center;"> ID </th>
   <th style="text-align:center;"> Increase in income </th>
   <th style="text-align:center;"> Remained in program </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:center;"> 1 </td>
   <td style="text-align:center;"> $3.00 </td>
   <td style="text-align:center;"> Yes </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 2 </td>
   <td style="text-align:center;"> $3.50 </td>
   <td style="text-align:center;"> Yes </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 3 </td>
   <td style="text-align:center;"> $2.00 </td>
   <td style="text-align:center;"> Yes </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 4 </td>
   <td style="text-align:center;"> $1.50 </td>
   <td style="text-align:center;"> No </td>
  </tr>
  <tr>
   <td style="text-align:center;"> 5 </td>
   <td style="text-align:center;"> $1.00 </td>
   <td style="text-align:center;"> No </td>
  </tr>
</tbody>
</table>
]

---

# Maturation

.box-inv-5.less-medium[Growth is expected naturally]

.box-inv-5.sp-after[e.g. programs targeted at childhood development<br>contend with the fact that children develop on their own too]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Use a comparison group to remove the trend]

---

???

https://www.wellesley.edu/sites/default/files/assets/dailyshot/2015/wellesleystudy_sesamestreet.pdf

https://umdrightnow.umd.edu/news/new-study-finds-sesame-street-improves-school-readiness

---

---

# Secular trends

.box-inv-5.less-medium[Patterns in data happen<br>because of larger global processes]

.center.float-left.sp-after[
.box-5[Recessions]&ensp;.box-5[Cultural shifts]&ensp;.box-5[Marriage equality]
]

.box-7.medium.sp-before[How to fix]

.box-inv-5.less-medium[Use a comparison group to remove the trend]

---

# Seasonal trends

.box-inv-5.less-medium.sp-after[Patterns in data happen because of<br>regular time-based trends]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Compare observations from same time period<br>or use yearly/monthly averages]

---

.center[
<img src="06-slides_files/figure-html/giving-2017-1.png" width="80%" style="display: block; margin: auto;" />
]

---

---

# Testing

.box-inv-5.less-medium.sp-after[Repeated exposure to questions or tasks<br>will make people improve naturally]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Change tests, maybe don't offer pre-tests,<br>use a control group that receives the test]

---

# Regression to the mean

.box-inv-5.less-medium[People in the extreme have a tendency to<br>become less extreme over time]

.center.float-left.sp-after[
.box-5[Luck]&ensp;.box-5[Crime and terrorism]&ensp;.box-5[Hot hand effect]
]

.box-7.medium.sp-before[How to fix]

.box-inv-5.less-medium[Don't select super high or<br>super low performers]

???

This isn’t because the universe trends toward some average; an extreme value is because of systematic and random extremes, which are rare. Luck goes away

---

# Measurement error

.box-inv-5.less-medium.sp-after[Measuring the outcome incorrectly<br>will bias the effect]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Measure the outcome well]

---

# Time frame

.box-inv-5.less-medium.sp-after[If the study is too short, the effect might not<br>be detectable yet; if the study is too long,<br>attrition becomes a problem]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Use prior knowledge about the thing<br>you're studying to choose the right length]

---

# Hawthorne effect

.box-inv-5.less-medium.sp-after[Observing people makes them behave differently]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Hide? Use completely unobserved control groups]

???

Experiments in 1924-1932 at Hawthorne Works

---

# John Henry effect

.box-inv-5.less-medium.sp-after[Control group works hard to prove<br>they're as good as the treatment group]

.box-7.medium[How to fix]

.box-inv-5.less-medium[Keep two groups separate]

---

# Spillover effect

.box-inv-5.less-medium[Control groups naturally pick up<br>what the treatment group is getting]

.center.float-left.sp-after[
.box-5[Externalities]&ensp;.box-5[Social interaction]&ensp;.box-5[Equilibrium effects]
]

.box-7.medium.sp-before[How to fix]

.box-inv-5.less-medium[Keep two groups separate;<br>use distant control groups]

---

# Intervening events

.box-inv-5.less-medium.sp-after[Something happens that affects one of<br>the groups and not the other]

.box-7.medium[How to fix]

.center.huge[🤷‍♂️]

---

# Internal validity

.center.float-left[
.box-inv-5[Selection] .box-inv-5[Attrition]
]

.center.float-left[
.box-inv-5[Maturation] .box-inv-5[Secular trends] .box-inv-5[Seasonality] .box-inv-5[Testing] .box-inv-5[Regression]
]

.center.float-left[
.box-inv-5[Measurement error]

.center.float-left[
.box-inv-5[Hawthorne] .box-inv-5[John Henry]
]

.center.float-left[
.box-inv-5[Spillovers] .box-inv-5[Intervening events]
]
]

---

# Fixing internal validity

.box-inv-5.medium[Randomization fixes a host of issues]

.center.float-left.sp-after[
.box-5[Selection]&ensp;.box-5[Maturation]&ensp;.box-5[Regression to the mean]
]

.box-inv-5.medium.sp-before[Randomization doesn't fix everything!]

.center.float-left[
.box-5[Attrition]&ensp;.box-5[Contamination]&ensp;.box-5[Measurement]
]

---

layout: false
name: external-validity
class: center middle section-title section-title-6 animated fadeIn

# External validity

---

---

# Generalizability

.box-inv-6.less-medium[Are your findings generalizable<br>to the whole population?]

.pull-left.right[
<figure>
  <img src="img/06/external-validity1.png" alt="Hospital lights increase risk of dying" title="Hospital lights increase risk of dying" width="80%">
</figure>
]

.pull-right.left[
<figure>
  <img src="img/06/external-validity2.png" alt="…in mice" title="…in mice" width="70%">
</figure>
]

---

# Lab conditions vs. real world

.box-inv-6.medium[Study volunteers are weird]

.box-6[<span style="color: #F6D645">W</span>estern, <span style="color: #F6D645">e</span>ducated, from <span style="color: #F6D645">i</span>ndustrialized,<br><span style="color: #F6D645">r</span>ich, and <span style="color: #F6D645">d</span>emocratic countries]

.box-inv-6.medium.sp-before[Not everyone takes surveys]

.float-left.center[
.box-6[Online surveys]&ensp;.box-6[Amazon Mechanical Turk]&ensp;.box-6[Random digit dialing]
]

---

# Different settings and circumstances

.box-inv-6.medium.sp-after[Does a study in one state<br>apply to other states?]

.box-inv-6.medium[Does the effect from a mosquito net trial<br>in Eritrea transfer to Bolivia?]