This is a very similar type of sampling problem to asking those in a popular how many siblings they have and trying...

This is a very similar type of sampling problem to asking those in a popular how many siblings they have and trying to determine average family size.

If you do it wrong -- as this does -- then you wind up with much larger average family sizes then is real.

Originally shared by Andreas Schou

Oh. Delightful. It turns out we used cohort samples to determine the recidivism rate. Which means that we're overestimating the rate of recidivism by a tremendous amount. Why is this bad? I guess I'll quote myself, from two days ago: 

Let's say that you live in a society with two possible crimes: one with a year-long sentence, and one with a twenty-year sentence. Of criminals in your society, 95% are convicted of the crime with a year-long sentence, and 5% are convicted of the crime with a twenty-year sentence.

So, to determine what people in your society are incarcerated for, you dip into the prison population at a single point and see what people have been convicted of. You'll find that your prison population is divided straight down the middle: 50% of the population consists of people who committed the one-year crime, and 50% consists of people who committed the twenty-year crime.

This happens because the distribution of prisoners currently incarcerated is controlled by two factors: the base rate of crime in the underlying society, and the duration of prisoners' sentences. Each twenty-year inmate will appear in 20 different yearly samples. Each one-year inmate will only appear in the sample from the year he's incarcerated.

For similar reasons, prisoners with multiple nonconsecutive incarcerations in the study period will be vastly overrepresented in the population. Which means that, by studying people who are in prison at any single point in time, we're getting a view not of the typical person who has been in prison, but the person who is most likely to be in prison.

It's hard to overstate how awful this is: we've been making public policy based on studies like this for a very, very long time.
http://www.slate.com/articles/news_and_politics/crime/2015/10/why_do_so_many_prisoners_end_up_back_in_prison_a_new_study_says_maybe_they.html