August 13, 2015

Researchers must understand selection bias when it comes to conducting and designing studies.

Researchers must understand selection bias when it comes to conducting and designing studies.

By Christopher Zoukis

Selection bias, tied to research design, can be a problem when determining the effectiveness of prison education and can skew results of studies. Researchers must understand selection bias when it comes to conducting and designing studies.

An Alarming Analysis of Study Design

Before Wilson and colleagues conducted a meta-analysis into the effects of prison education on recidivism and employment rates, they examined the design and conduct of each of the 33 studies in their analysis. But the study designs were so poor they couldn’t be certain results were due to pre-existing characteristics of the study subjects rather than the educational programs they had taken (Wilson, Gallagher, Coggeshall, & MacKenzie, 1999). 

What is Selection Bias?

Selection bias happens when there is an excess of people in a group more likely to succeed or fail, which can skew study results. The problem of comparing outcomes in two or more groups is ensuring that the only difference between the groups is the intervention under investigation. 

Suppose you conduct a study comparing the life expectancy of male smokers and female non-smokers.  You discover the female non-smokers live longer than the male smokers.  Is it because the men smoke, or do women just live longer? The correct comparison would be between smokers and non-smokers of each gender (males together and females together).

Correctional education is clearly associated with reduced recidivism and higher employment.  But is education causing lower recidivism, or is it because of some characteristics of the individuals participating in the education programs?

Prison education programs are mostly voluntary and usually have limited space. Participants tend to be motivated and persistent.  Even without education, those individuals may be less likely to be re-incarcerated and more likely to find a job.  Inmates with no interest in education or without enough motivation to fight for a place are more likely to have mental health problems or to anticipate re-offending.  The two groups may be very different.

Screen Shot 2015-10-02 at 4.31.44 PM.png

Once a study’s results are analyzed, it can be difficult to know the cause of differences in recidivism and employment rates between the two groups. Are they due to the education programs attended, or a result of the different characteristics of the two groups? These characteristics, though unrelated to the education received, may affect the outcome. 

Researchers should design studies so that these characteristics are evenly distributed between two comparison groups:

  • The intervention group is inmates who receive the education or training.
  • The control group is inmates who do not receive education or training.

Controlled characteristics would include things like age, sex, ethnicity, criminal history, offense type and severity, socioeconomic status, and presence of mental health problems.

Evidence of the Effects of Selection Bias in Studies is Conflicting

The degree to which studies are affected by selection bias varies. It is not clear how much selection bias affects data as a whole and the conclusions drawn from it. Reviews have been conflicting:

Wilson and Colleagues

These four researchers were scathing about the quality of studies used in their meta-analysis.  They discovered the better the study design, the lower the effect of correctional education on recidivism. 

A well-designed study accounts for the presence of other characteristics by ensuring they are evenly distributed between two groups. Part of the drop in recidivism was due to one or more of these characteristics, rather than education (Wilson et al., 1999).

Aos and Colleagues

This group included only the best designed studies in their meta-analysis.  They estimated 9% less recidivism due to vocational training, compared to a 36% reduction in the RAND meta-analysis.  Selection bias may account for the apparent reduction in recidivism (Aos, Miller, & Drake, 2006).

Wells

Wells' meta-analysis included 329 comparisons from 124 studies and had a more generous assessment. He rated:

  • 23% of studies as having a strong design.
  • 40% of studies as being moderately well-designed.
  • 20% of studies as being weak.
  • 6% of studies as having no scientific value.

In a finding not replicated by others, the better-designed studies showed the greatest reductions in recidivism, suggesting selection bias was not a problem (Wells, 2000).

Three State Recidivism Study

This study attempted to address selection bias by surveying study subjects about their motivations for participating in education programs. Researchers asked subjects about the importance of the following:

  • Preparing for a job.
  • Obtaining a job.
  • Getting higher pay.
  • Receiving better training.
  • Using new skills to contribute to family and/or community.
  • Being more independent.
  • Looking good to prison and parole officers, and so on.

They found no significant difference in these factors between the education and control groups (Steurer, Smith, & Tracy, 2001).

The RAND Corporation

RAND’s 2013 meta-analysis ranked studies using the Maryland Scientific Method Scale.  Of the 50 studies about the effect of prison education on recidivism:

  • 2 studies had an excellent design.
  • 5 studies had a good design.
  • 18 studies had a moderate design.
  • 25 had a weak design.

Only studies of scientific merit were included. To see how much selection bias affected the studies, they looked at the effect of the studies in each quality grade. Selection bias was not a significant factor (Davis, Bozick, Steele, Saunders, & Miles, 2013).

How Much Should We Worry About Selection Bias?

Expert statisticians conducted RAND’s meta-analysis, and so the results carry a lot of credibility (Davis et al., 2013).

It is still valid that correctional education reduces recidivism while increasing employment and wages.  However, when interpreting individual studies, remember the potential effects of selection bias. Examine study methodology details carefully, and focus on how well the two groups are matched for all pre-existing characteristics. The only difference between the groups should be the intervention under investigation.

As we seek to improve our understanding of the most effective types of training and education, it is important that researchers conducting future studies seek to understand and minimize selection bias.

__________

References:

Aos, S., Miller, M., & Drake, E. (2006). Evidence-based public policy options to reduce future prison construction, criminal justice costs, and crime rates. Washington State Institute for Public Policy. Olympia, Washington.

Davis, L. M., Bozick, R., Steele, J., Saunders, J., & Miles, J. N. (2013). Evaluating the effectiveness of correctional education - A meta-analysis of programs that provide education to incarcerated adults. RAND Corporation.

Steurer, S. J., Smith, L. G., & Tracy, A. (2001). Education reduces crime: Three state recidivism study. Correctional Education Association. Lanham, Maryland.

Wells, R. E. (2000). Education as prison reform: A meta-analysis. Unpublished dissertation, Baton Rouge, Louisiana: Louisiana State University.

Wilson, D. B., Gallagher, C. A., Coggeshall, M. B., & MacKenzie, D. L. (1999). A quantitative review and description of corrections-based education, vocation, and work programs. Corrections Management Quarterly, 3(4), 8-18.