About that supposed weak link between test-taking and research quality...
One often hears that exams have little correlation with, well, anything. It’s a matter of faith among grad students that our very difficult prelims at the end of the first year reveal little about our potential as researchers. Good researchers are different than good test-takers, or so we think.
It turns out that an interesting paper addresses this very question. In What does Performance in Graduate School Predict? Graduate Economics Education and Student Outcomes, a top group of economists run regressions of the predictive power of admissions rank on first-year exam scores, and then of the effect of first-year scores on job placement. The rank of the economics department where a newly-minted PhD is first hired is used as a proxy for research quality.
Before going into the findings, some important caveats are necessary.
The sample consists of only five schools: Harvard, MIT, the University of Chicago, Princeton and Stanford.
The data is for students who entered these five schools in the years between 1990 and 1999.
The R-squared of the regressions is particularly low, no higher than .12 for any specification. This means that regressors with significant coefficients are still not explaining very much of the variation in job placement.
It’s worth pointing out that some people who are stars and get hired at top-ranked PhD departments upon graduation sometimes don’t get tenure, while others take many years to develop a reputation as a top-quality researcher. Job-placement is therefore an imperfect proxy for research quality. The advantage of examining job placement, though, is that it is much easier to assess than the quality of a researcher’s output.
And now for some interesting findings…
First-year test scores are highly correlated across subjects.
Students from foreign undergraduate institutions perform significantly better on first-year exams than do students from American schools.
Women at these five schools did not perform worse in the job market than men. Conditional on first-year test scores, women were placed at slightly higher-ranked institutions upon graduating.
First-year test scores are a significant predictor of job placement. In fact, first-year scores in micro and macro are just about the only significant predictors of job placement in the sample. Having attended a top-ranked undergraduate American university was the other significant factor affecting job placement in the top twenty economics departments. This raises important questions. Are people who attend elite undergraduate schools in America benefitting from a form of nepotism in the job market whereby they are hired at the institution they went to school? Or is there a systematic difference between these students and those who don’t attend elite undergraduate colleges that is not captured by the other variables in the model?
One finding is hard to explain. Admissions rank helps predict first-year exam scores and exam scores help predict job placement, but admissions rank does not help predict job placement. In fact, admissions rank does not predict job placement even if it is the only variable in the model.
The regression results presented here go against the stories we tell ourselves in graduate school. There are three possible interpretations. The truth most likely is a combination of the three.
1. The cynical view : exams serve as merely a signal of underlying characteristics such as intelligence and motivation. The fact that good test-takers end up becoming good researchers should not surprise us, because these same qualities are important in determining the quality of a researcher’s output.
2. The Panglossian view : courses in economics graduate programs are carefully designed to teach material that is imperative for economic research. The exams are comprehensive and therefore serve as effective arbiters of a given student’s knowledge of the material. Students who do well on these exams have mastered economic theory, and this superior knowledge explains why they become good researchers.
3. While both of the above explanations may contain a modicum of truth, with such a low R-squared, these regressions still aren’t explaining very much. As the authors conclude:
“The difficulty predicting job placement may, in part, result from the noise in our data, ambiguity in ranking jobs, the incompleteness of our measures, and inherent randomness in the academic job market. Diligence, perseverance, and creativity—factors that surely matter for successful research careers and job placement—are difficult to define and measure. Our results suggest that there is not an easily recognizable star profile or single path to success for an economics graduate student.”
Paper available here:
http://pubs.aeaweb.org/doi/pdfplus/10.1257/aer.97.2.512
Reference
Athey, Susan, Lawrence F. Katz, Alan B. Krueger, Steven Levitt, and James Poterba. "What does performance in graduate school predict? Graduate economics education and student outcomes." The American economic review 97, no. 2 (2007): 512-518.