Scientists sometimes feel that statisticians have a responsibility to analyze their data, or teach scientists how to analyze their data. However, statisticians spend little time doing either of these things. This can lead to friction. In this post, I explain how this tension emerges from potentially opaque institutional dynamics. Painting in broad strokes, I describe tenure considerations in statistics departments and explain how these incentives shape the work that statisticians end up doing, and, importantly, not doing.
incentives in academic statistics
A common misconception is that academic statistics is the study of data analysis. This isn’t entirely wrong, but it is misleading. Academic statisticians, by and large, are not paid to analyze data, or even to explain how to analyze data. Rather, we are paid to describe how to estimate the parameters of probability models under assumptions that plausibly obtain in the real world. There are other contributions one can make to the statistics literature, but theory work for estimators is canonical. It is the work that is most easily understood as a statistical contribution. So academic statisticians are, first and foremost, people who derive theoretical properties of estimators under probability models.
To be concrete, examples of this kind of work include things like:
- developing novel statistical models and estimators (as in Dunson 2009),
- deriving minimax rates of convergence and finding minimax estimators or statistical tests (as in Berrett et al. 2021), and
- developing new ways to sample from complicated probability distributions (as in Neal 2003),
amongst others1.
This emphasis on theoretical and methodological work has some surprising consequences. The first is an ambivalent relationship to data analysis, by which I mean that statisticians don’t necessarily analyze very much data, either as part of our research or during our training. There is some tension about this relationship within the discipline, and over the years, numerous calls have emerged to engage with data analysis more seriously. In the sixties, Tukey took great care to describe himself as a data analyst first, and a statistician second, and to plead with statisticians to contribute more to data analysis (Tukey 1962). Breiman (2001) similarly lambasted statisticians for studying models that he perceived as having little practical relevance. Recent years have seen an increased emphasis on workflow, visualization and data science. However, if we look to academic journals to see what the field values, the most prestigious journal in the field remains Annals of Statistics, a decidedly theoretical venue. Other top journals in the field also emphasize theoretical contributions over applied contributions2.
In statistics graduate programs, the emphasis on mathematical statistics over data analysis is clear, with coursework covering probability, linear algebra and analysis. Most graduate programs in statistics include a sequence on regression and experimental design, but emphasize derivation of \(F\)-tests and textbook data examples over, say, data cleaning. Most graduate programs also include a course on statistical consulting. For some graduate students, an exclusively theoretical dissertation means that this consulting course is their only experience working with data. The end result is that some graduate students are unpracticed at data analysis when they earn their degrees.
data analysis by statisticians
When statisticians do work with data, it’s usually in one of two ways: as data examples to illustrate how an estimator works, or as a complicated standalone data analysis. Crucially, the point of data applications is to help readers understand the method, not the data or the world3. Indeed, statisticians routinely distinguish between “real-world data” and “toy data,” and because statisticians rarely collect data themselves, we have real-world data less often than you might imagine. The focus on illustrative rather than scientific uses of data does have upsides: statisticians are much less beholden to reviewers wanting p-values below 0.05 or splashy results. Rather, our data applications are evaluated on how reasonable the methodological application is, with some indifference to scientific conclusions4.
Data analysis can also sometimes end up in statistics journals. Typically, this happens when there is a complicated dataset and someone comes up with a specialized method to analyze that data. For instance, my own paper Hayes and Rohe (2025) does some methods work, but it’s all in the service of analyzing one very specific dataset. To get this type of work published, the analysis needs to be new or interesting or impactful or just very complicated. Biostatistics departments tend to be more applied than statistics departments, and have a lot more high-impact data, and so they do more of this kind of work.
When statisticians do careful data analyses that are impactful but not statistically novel, these papers rarely end up in statistics journals5. They end up in scientific or social scientific journals. Doing good and careful science is time- and labor-intensive; the amount that hiring and tenure committees value this labor varies substantially by department. As a result, research projects focusing on data analysis may not be strategic relative to theoretical or methodological research. This can result in a professional ambivalence toward data analysis from statisticians, which can be frustrating to people trying to do science or learn about the world.
service
Earlier I mentioned that the graduate curriculum in statistics prioritizes mathematical statistics over data analysis. What happens in service courses for students in different departments? My impression is that many service courses retain the heavy mathematical emphasis of courses designed for statistics students, and when they do move away from math-stat content, they emphasize calculations rather than data analysis. The shift might be away from the properties of p-values to simply calculating p-values, lightening the mathematical content without necessarily being concretely useful.
Most service courses I’ve seen focus heavily on hypothesis testing and linear regression, with some limited programming instruction. It’s a start, but no matter how you slice it, service courses almost never produce the level of applied competence that most scientific disciplines expect from their students. I often see graduate students in STEM go into these courses expecting to learn to analyze data, and then to later emerge surprised and frustrated at their actual takeaways. I think this frustration is often two-fold: some frustration comes from content being poorly targeted to their needs, and some frustration is because students enter with the expectation that fifteen weeks is enough to learn to analyze data, and they don’t leave feeling like they know how to analyze data6.
The fact that service courses are not always sufficiently tailored to applied audiences comes up on occasion. A notable episode on Twitter from several years back involved a prominent psychologist making the claim that psychologists could teach data analysis better than statisticians. While I don’t fully agree, I think it’s valuable to consider how statistics courses are often implicitly designed to train future tenured statistics faculty, and this is the wrong kind of training for scientists. Statisticians largely produce methods. Scientists largely produce knowledge about the world.
Another consequence of methodological focus is that statisticians sometimes produce tools that are not particularly useful (again the famous argument of Breiman 2001), since we are not always consumers of our own methods. This can also be the source of some frustration and confusion. The issue is not malice so much as prioritization. Actual science is just not on our critical path all that often. Because of this, statisticians are at risk of solving the wrong problem (i.e., a tractable mathematical problem rather than a scientifically important one), or insufficiently communicating how our tools work, or never implementing our methods in usable software, or developing overcomplicated methods7.
how other fields have responded
Since statisticians don’t really do service work, what tends to happen is that many applied departments, especially in the social sciences, will hire a small number of methodologists with a mixture of statistical and domain background. These methodologists teach statistics courses to graduate students in their own departments and do lots of translational labor, turning esoteric theory papers into readable tutorials and software. On the one hand, this is a rational response to statistical indifference. On the other hand, the quality of work done by these quantitative methodologists is variable8. They typically publish outside of mainstream statistics journals in disciplinary methods journals, and so statisticians are often ignorant of their work, either on accident or on purpose.
The ironic consequence of this limited engagement with service work is that statisticians have far less influence on the practice of data analysis than we would like. This can be disheartening, because we would very much like to be seen as and treated like experts. Which, notably, we are. It’s very easy to feel jaded about this, because a substantial amount of data analysis in science is effectively wrong (another portion is sloppy but redeemable, and another portion is done very well). So when academic statistics is in fact interested in the practice of statistics, we are often hamstrung by the fact that we are an insular institution, and that we have rarely spent time developing cultural capital. In a sense, statistics is in a state of crisis, on the verge of irrelevance. In another sense, the actual practice of statistics by scientists makes it clear that there is no risk of irrelevance.
a summary in stylized facts
To sketch out the story all at once: statisticians respond strongly to tenure incentives, which pull us in a theoretical rather than applied direction. Consequently, academic statistics has largely opted out of service responsibilities to other disciplines. It’s not that we don’t do service, but that our service isn’t as helpful as others would hope. As a result, statistics can be seen as aloof or apathetic or indifferent (cf. computer science, economics). Other disciplines have responded by building in-house statistical capacity of variable quality, and by ignoring statistics proper to some extent. This makes sense because a lot of statistics isn’t useful to scientists for various reasons. It’s also concerning because a lot of data analysis by scientists needs improvement9. Frustrations occasionally abound because scientists want help with their data analysis and statisticians want people to analyze data well and also want to be taken seriously.
The story above is about incentives at the institutional level; at the individual level most statisticians I know genuinely want to develop (and occasionally teach) useful methods, and most scientists I know genuinely want to do good science. It’s really at the level of institutional incentives where there can be frictions, and I find it interesting and valuable to explain what those are.
I also want to highlight that, within statistics itself, there is some tension about priorities. Are we interested in the technical problem of producing a rich statistical literature, or the social problem of producing a rich practice of statistics? I don’t think there are necessarily right or wrong choices, but there are certainly tradeoffs. My impression is that statistics currently prioritizes a rich statistical literature, and the cost of this is that we have a less rich statistical practice in the sciences. When I entered grad school, I was very much interested in improving statistical practice and viewed statistics as a discipline in service to science; I still do in many ways, but years of professional acculturation have led me to value a rich statistical literature as well. Regardless of our precise priorities, I think it is valuable to both articulate and interrogate what our values are.
acknowledgements
I would like to thank Ben Listyg for helpful comments on this post. Ben is consistently a sounding board for my half-formed ideas, and I am immensely grateful.
References
Footnotes
See Gelman and Vehtari (2020) for a short review of important recent ideas in statistics and Efron (2021) for a textbook-length treatment of influential methods.↩︎
Here “applied” means empirical work, or work that otherwise centers data analysis.↩︎
Two representative examples of data applications in my own sub-field are Qing and Wang (2022) and Chen et al. (2020). Qing and Wang (2022) develops a new network formation model and shows it fits data better than an older model. There is not really scientific inference on the data itself in the paper. Chen et al. (2020) develops a new method for locally clustering social networks, and then applies that method to Twitter data, and it’s certainly interesting, but the data analysis is not in support of any claims about how the world fundamentally works.↩︎
I do not envy publication pressures in more scientific fields.↩︎
Statistics journals are uninterested in rote data analysis, but people typically don’t want to publish high-impact empirical work in statistical journals simply because no one reads them.↩︎
Learning to analyze data is analogous to learning to write, a process that takes years of practice and feedback. It would probably be helpful to tell students this early in their careers.↩︎
I have heard some complaints that econometrics is increasingly taking a turn in this direction.↩︎
Disciplinary methods journals are home to strands of literature that are very cool and effectively unknown in statistics proper, but they are also home to strands of literature that would never pass mainstream statistical peer review.↩︎
Methodologists are on the end of a weird and limited inversion of Brandolini’s law, where it is far easier to recognize a bad data analysis than it is to produce a good one. There is a common adage that data science is the intersection of statistical expertise, data manipulation skills, and domain expertise, and that you need all three of these skills to analyze data well. To identify serious flaws in a data analysis, statistical expertise alone suffices.↩︎