Academic statistics can seem counterintuitive and confusing to outsiders, and this post describes why. I describe what the field values and how those incentives shape the kind of work statisticians end up doing—and not doing. My goal is to clarify the sometimes opaque institutional dynamics between academic statistics, data analysis, and the production of scientific knowledge.
I should note that I paint with a broad brush in this post. There’s immense variety within academia, and while I hope to capture common patterns, my goal is offer up some stylized facts rather than universal truths.
incentives in academic statistics
One very natural misconception is that academic statistics is the study of data analysis. This isn’t entirely wrong, but it is misleading. Academic statisticians, by and large, are not paid to analyze data, or even to explain how to analyze data. Rather, we are paid to describe how to estimate the parameters of probability models under assumptions that plausibly obtain in the real world. There are other contributions you can make to the statistics literature, but theory work for estimators is canonical. It is the work that most easily understood as a statistical contribution. So academic statisticians are, first and foremost, people who derive theoretical properties of estimators under probability models.
To be concrete, examples of this kind of work including things like:
- developing novel statistical models and estimators (as in Dunson 2009),
- deriving minimax rates of convergence and finding minimax estimators or statistical tests (as in Berrett, Kontoyiannis, and Samworth 2021), and
- developing new ways to sample from complicated probability distributions (as in Neal 2003),
amongst others1.
This emphasis on theoretical and methodological work has some surprising consequences. The first is that academic statistics has an ambivalent relationship to data analysis, by which I mean that we don’t necessarily analyze very much data in our research or in our training. Over the years, there have been numerous calls within the discipline to engage with data analysis more seriously. For instance, nearly 80 years ago Tukey took great care to describe himself as a data analyst first, and a statistician second, and to plead with statisticians to contribute more to data analysis (Tukey 1962). Breiman (2001) similarly lambasted statisticians for studying models with little practical relevance. Despite these calls, at tenure time, one of the best things you can do is have an Annals of Statistics paper on your CV, which is about as theoretical as it gets.
In statistics graduate programs, the emphasis on mathematical statistics over data analysis is clear, with coursework covering primarily probability, linear algebra and analysis. Most graduate programs in statistics do include a sequence on regression and experimental design, but emphasize derivation of
data analysis by statisticians
When statisticians do work with data, it’s usually in one of two ways: as data examples to illustrate how an estimator works, or a complicated standalone data analysis. Crucially, the point of data applications is to help readers understand the method, not the data or the world3. One plus side of this methodological focus is that statisticians are much less beholden to reviewers wanting p-values below 0.05 or splashy results. Rather, our data applications are evaluated on the correctness of the methodological application, with some indifference to scientific conclusions. This is really, really nice, and I do not envy publication pressures in more scientific fields.
Data analysis can also sometimes end up in statistics journals. Typically, this happens when there is a complicated dataset and someone comes up with a method specifically to analyze that data. For instance, my own paper Hayes and and Rohe (2025) does some methods work, but it’s all in the service of analyzing one very specific dataset. To get this type of work published, the analysis needs to be new or interesting or impactful or just very complicated. Biostatistics departments tend to be more applied than statistics departments, and have a lot more high impact data, and so they to do more of this kind of work.
When statisticians do careful analyses on interesting data without statistically novel elements, these papers do not end up in statistics journals. They end up in scientific or social scientific journals, and in a lot of departments, the tenure payoff is not proportional to the labor it takes to actually do good science. This ambivalence towards data analysis can be frustrating to people trying to do science or learn about the world.
service
Earlier I mentioned that the graduate curriculum in statistics prioritizes mathematical statistics over data analysis. So what happens in service courses designed for students in different departments, do those classes teach data analysis skills? My impression is that many service courses retain the heavy mathematical emphasis of courses designed for statistics students. Most service courses I’ve seen focus heavily on hypothesis testing and linear regression, with some limited programming instruction. It’s a start, but no matter how you slice it, it’s just not enough instruction to produce the level of applied competence that most scientific disciplines expect from their students. I often see graduate students go into these courses expecting to learn everything they need to know to analyze data, and then emerging later surprised and frustrated. Some explicit expectation setting might smooth out this experience.
The fact that service courses are not always sufficiently tailored to applied audiences comes up on occasion. A notable episode from several years back on Twitter involved a prominent psychologist making the claim that psychologists could teach data analysis better than statisticians. While I don’t fully agree, I think it’s valuable to consider how statistics courses are largely designed to train future tenured statistics faculty, and this is the wrong kind of training for scientists. Statisticians largely produce methods. Scientists largely produce knowledge about the world.
Another consequence of the methodological focus is that statisticians sometimes produce tools that are not particularly useful, since we are not always consumers of our own methods. This can also be the source of some frustration and confusion. The issue is not malice so much as prioritization. Actual science is just not on our critical path all that often. Because of this, statisticians are at risk of solving the wrong problem (i.e., a tractable mathematical problem rather than a scientific important one), or communicating insufficiently how our tools work, or never implementing our methods in usable software, or just generally developing a method so complicated that the effort to learn the method is not worthwhile4.
how other fields have responded
Since statisticians don’t really do service work, what tends to happen is that many applied departments, especially in the social sciences, will hire a small number of methodologists with a mixture of statistical and domain background. These methodologists teach statistics courses to graduate students in their own departments and do lots of translational labor, turning esoteric theory papers into tutorials and software. On one hand, this is a rational response to statistical indifference. On the other hand, the quality of work done by quantitative methodologists outside of statisticians is variable, so statisticians are not always particularly supportive of this approach.
An especially ironic consequence is that statisticians have more limited influence on the practice of data analysis than we might hope. This can be disheartening as a statistician, because we would very much like to be seen as and treated like experts. Which, notably, we are. It’s very easy to feel jaded about this, because a substantial amount of data analysis in science contains methodological errors that result in incorrect inferences. A decent chunk is sloppy but redeemable, and another portion is done very well5.
So when academic statistics is in fact interested the practice of statistics, we are often hamstrung by the fact that we are an insular institution, and that we have rarely spent time developing cultural capital. In a sense, statistics is in a state of crisis, on the verge of irrelevance. In another sense, the actual practice of statistics by scientists makes it clear that there is no real risk of irrelevance.
a summary in stylized facts
To sketch out the story all at once, I think a reasonable summary goes along these lines: statisticians respond strongly to tenure incentives, which pulls us strongly in a theoretical rather than applied direction. Consequently, academic statistics has largely opted out of service responsibilities to other disciplines. It’s not that we don’t do service, but that our service isn’t as helpful as others would hope. As a result, statistics can be seen as aloof or apathetic or indifferent (c.f. computer science, economics). Other disciplines have responded by building in-house statistical capacity of variable quality, and by ignoring statistics proper to some extent. This makes sense because a lot of statistics isn’t useful to scientists for various reasons. It’s also concerning because a lot of data analysis by scientists needs improvement. Frustrations abound on all sides because scientists want help with their data analysis and statisticians want people to analyze data well and also want to be taken seriously.
I should temper the sketch above with the note that most statisticians I know genuinely want to develop and teach useful methods, most scientists I know genuinely want to do good science, and science and statistics are enterprises that have made enormous progress over the last decades.
That said, it feels to me like a reasonable time to contemplate how statistics wants to relate to other fields. Are we interested in the technical problem of producing a rich statistical literature, or the social problem of producing a rich practice of statistics? Do our tenure practices reflect these priorities?
I don’t think there are necessarily right or wrong choices, but there are certainly tradeoffs. My view is that statistics currently prioritizes a rich statistical literature, and the cost of this is that we have a less rich statistical practice in the sciences6. Regardless of our precise priorities, I think it is important to articulate our values and to explain them to others, so that outsiders can understand what is on offer and what is not. Some sets of priorities may lead to conflicts that are difficult to resolve. There’s tension between pointing out methodological problems and actually helping people fix them.
My hope is that this tension, and some others, make more sense than they did at the beginning of the post. Tenure in statistics is about methods and theory work, and as a consequent statisticians often de-prioritize data analysis and scientific contribution. That said, things may be changing. Statistics departments grew substantially during the data science boom, and the field feels invigorated with new people and new ideas and new priorities. I’m curious to see where we head next.
acknowledgements
I would like to thank Ben Listyg for helpful comments on this post. Ben is consistently a sounding board for my half-formed ideas, and I am immensely grateful.
References
Footnotes
See Gelman and Vehtari (2020) for a short review of important recent ideas in statistics and Efron (2021) for a textbook-length treatment of influential methods.↩︎
Perhaps as a consequence, applied statisticians often describe a rule of thumb is that you aren’t capable of doing meaningful applied work until you’ve been working in and learning about a particular domain for at least three years.↩︎
Two representative examples of data applications in my own sub-field are Qing and Wang (2022) and Chen, Zhang, and Rohe (2020). Qing and Wang (2022) develops a new network formation model and shows it fits better data better than an older model. There is not really scientific inference on the data itself in the paper. Chen, Zhang, and Rohe (2020) develops a new method for locally clustering social networks, and then applies that method to Twitter data, and it’s certainly interesting, but the data analysis is not in support of any claims about how the world fundamentally works.↩︎
I have heard some complaints that econometrics is increasingly taking a turn in this direction.↩︎
Methodologists are on the end of a weird and limited inversion of Brandolini’s law, where it is far easier to recognize a bad data analysis than it is to produce a good one. There is a common adage that data science is the intersection of statistical expertise, data manipulation skills, and domain expertise, and that you need all three of these skills to analyze data well. To identify serious flaws is a data analysis, statistical expertise alone suffices.↩︎
What would statistics look like if tenure packets were evaluated for tutorial papers, for scientific contributions, for code, for outreach work?↩︎