A recent article in The Atlantic asks the question "Why Do Former High School Athletes Make More Money?" According to the article by Joe Pinsker:
Former high-school athletes generally go on to have higher status careers than those who didn't play a sport. On top of that, former athletes' wages are between 5 and 15 percent higher than those of the poor trombonists and Yearbook Club presidents. The earnings advantage doesn't appear to exist for any other extracurricular activity.
This struck Mrs. Everydata (my wife) as exactly the type of question that raised a flag for sound consumers of data, and at her urging, today I investigate the study and the numbers:
The basic premise of the article is based on a study by researchers at Cornell and Southern Illinois who looked at two unique data sets on biodata. In psychology, biodata is data collected and asked about one's self, i.e. "biographical data." There are two parts of the study. In part 1, a sample of 66 adults participated in a survey about leadership traits and past experience in extra-currriculars. This part of the survey was to capture subjective views about participation in athletics. For example, the study describes the questions asked as:
As the main set of questions, participants were asked to use a 9-point scale to indicate their level of disagreement (1) and agreement (9) with a set of 4 statements that varied according to whether a person participated in one of four different extracurricular activities. More specifically, participants were asked to indicate the degree to which they thought “A person who [played Varsity Basketball in high school; or, played Varsity Cross Country in high school; or played trombone in the high school Band; or, participated in the high school Yearbook club] is likely to display more” Self-Confidence, Leadership, Time-Management Skills, Volunteerism, Charitable Behavior, and Self-Respect."
From this small sample, the authors conclude that "people tend to expect former student-athletes to demonstrate greater leadership ability as well as organizationally beneficial personality traits; however, former studentathletes are not expected to be altruistic with respect to others." I will not spend a lot of time on Study 1, other than to say there is always controversy about these types of attitudinal surveys across fields. Economists tend to be very skeptical, but industrial organizational psychology utilizes them quite often. I will note from a statistical standpoint, it appears the key result is somewhat sensitive to whether all sports and all non-sports are pooled together.
That said, I'd like to focus on the second study--which conducted a statistical analysis of the 2000 University of Illinois Veteran's Survey. According to the study, this sample contained information on 931 World War II veterans who were ages 71-93 at the time of completion of the survey in 2000. The key conclusion from the authors here is that " we show that there appears to be long-term correlates of participation in competitive youth sports that persist for more than 55 years. More specifically, our results show a positive relationship between participation in competitive youth sports and several measures of long-term personal success and prosociality."
Here are a few observations:
First, there is no part of either study 1 or study 2 that looks or measures the actual effect of high school sports on wages. That data does not exist. Rather, the study looks at how self-reports of participating in a college sport 55 years ago correlates with leadership metrics, and with trade jobs versus upper-management jobs. This is a fairly crude tool to measure whether one gets a better job.
Second, the premise of the study with a somewhat limited set of explanatory variables is that sports participation more than 55 years ago correlates with future job outcomes. But, the only other explanations that the study can control for is age and size of home town. This immediately raises the question of omitted variables--what if high school sports participation by a subset of male Veterans age 71-93 were correlated with any number of other factors--GI Bill? Education Level? Participation in WW2? Ability? This is a classic issue in most studies of job performance.
Third, the authors caveat their results are potentially picking up correlations but no causation. There are a lot of two way analyses in this study- leadership correlated with athletics, self-confidence correlated with self-respect, etc. This is a complicated question, and it is highly unlikely that these particular metrics are capturing a true causal relationship.
That said, I love biometric data and think the study is very interesting. But, this is an example where what gets captured in the headlines may not quite match what is actually in the underlying study.