TBH I'm kinda long-term optimistic about academic slop. The current incentive structure has sort of funneled academics into churning out "minimally publishable" papers, the kind of thing you turn the crank on without having to think very hard about it. Then, when academics do have time to think, it's mostly for the purpose of overhyping their just-okay papers.
If "minimally publishable" becomes mass producible, then maybe the returns to thinking will improve.
"Top journals will focus on papers that are strikingly original or make important theoretical or empirical breakthroughs, while everyone else will publish the AI-produced papers that incrementally advance our understanding of narrow things."
Isn't it kind of like academia is structured already? Something that gets published in Slavic Review usually makes conclusions much larger than the narrow topic they focus on. And the "look, here's a document I found in the archives!" papers come out in a much less-prominent journal and are much less frequently read and cited.
It's true there's already a prestige divide -- but a lot of top journals are still heavily invested in causal inference papers that are very good methodologically but don't do much to advance theory. These are the kinds of papers that I think (hope) will become AI-automated and thus less prominent in top journals.
My experience reviewing papers for “top journals” (mostly genetic and genomic papers over a period of 50+ years) matches yours. Out of hundreds of papers reviewed, there were only a few for which I expressed serious concerns about the validity of the conclusions. Editors still valued my opinion as to how interesting the conclusions would be to the journal’s target audience.
Having just processed a lot of job applications and graduate student place applications, this is very similar there - out of (say) 100, about 2 looked individual and stood out - the rest were "merely competent"....
Brilliant post as ever. But the elephant in the room may be that KPI-driven academic publishing conventions have already strangled the life out of much innovative thought in the social sciences, and it’s only now that the scales are falling away from our eyes. All LLMs are doing is the kind of salami slicing that is already prolific.
So, with AI, academic slop could become just another way to masquerade progress in innovative thinking, when all that's actually happening is that metrics are being satisfied? This kind of reminds me of Mark Fisher's concept of market Stalinism and the "valuing of symbols of achievement over actual achievement" - just accelerated and somewhat more nightmarish.
Even before the age of AI there where many papers that will be considered as human generated slops. The only thing that has changed is the exchange of baton from man to machine.
This is basically what has happened to academic computer science over the past couple years. Submissions to top conferences (in most computer science fields conferences are the highest prestige venues) are up 50-200% and at a recent AI conference there were 23000 submissions. The peer review system is starting to buckle under the load.
AI models still can't quite churn out a decent paper in my field but they are an effective productive multiplier allowing groups that are more aggressive to double or triple their output. And that's before we get to arxiv. Arvix was already a firehouse on popular topics but with the rise of AI and a glut of technically sound but boring papers has moved conferences into more of taste-maker role, doubling down on potential impact and interest.
Of course that just exacerbates the review issue. Increasingly papers not in top conferences just aren't read so if you want to have any impact at all your work needs to be in a top conference. Some conferences have started putting per-author submission limits but I don't think anyone has a great solution yet.
Great write-up. I think you could go beyond the image of tiers in academic publishing and move towards truly thinking about a more automated "research floor" model where Automatenwissenschaft becomes a sort of factory constantly churning out updated findings and data. I agree the immediate concern here is that our academic institutions like scholarly journals are woefully unprepared for these developments.
I think the bigger concern revolves around how we are going to grapple with all of this information. And I think re-emphasizing theory-building only gets us so far. Similar to the Aristotelian phronesis, in 1998 Herbert Simon opined that "if one factor in a system (in this case, information) has become abundant, what has become scarce? Our time-your time and mine-to process that information." ("Information 101: It's Not What You Know, It's How You Know It").
I think you could see a future where the factory floor Automatenwissenschaft consistently produces good and accurate information at a colossal scale. Sorting through this vast sea of findings will similarly require some sort of higher level automation. AI could be the perfect tool to help researchers sort through the mass produced information. But what comes next? Automation of the theory creation itself as the accumulation cycle of now-sorted information continues on?
I think the concern about research hierarchy is spot on for now. Yet I can help but wonder how long such a human-based hierarchy would end up lasting before it, too, gives way.
thanks. I guess I'm skeptical that accumulating facts will allow us to automate theory. one of my favorite points on this is from poincare: "a collection of facts is a theory in the same way that a collection of bricks is a house" (I may be butchering it from memory but you get the idea)
Seva, greetings from long ago, Massachusetts Avenue in 2002.
All of these tentative conclusions at the moment appear to be based on the idea that most of the data needed is available on the Internet. Since 2007 or before, I've been dealing in my research with data that wasn't on the Internet, wasn't digitized, though it may have been in book form; and data that a researcher has to go and collect out of humans' minds. At present, because the data for half the interesting research is simply not accessible even in books, there appears to be a continuing place for research that is founded upon such data.
I'm not optimistic. As a frequent peer reviewer (mostly for highly ranked journals), I am constantly getting papers about which I end up asking "yes, but does it matter?" and answering, "not really." Sadly, a much too large share of the academic world is made up of people who have been trained to create technically competent - usually quantitative - but basically uncreative work. Phronesis is rare, and usually only learned gradually.
When competent output becomes cheap, institutions don’t just face overproduction. They lose the asymmetries that once protected judgment from optimization.
Peer review, like governance, depended on discretion, delay, and partial opacity. Once those erode, attempts to “filter for quality” start selecting for whatever can still be filtered not necessarily what matters.
At that point the problem isn’t slop. It’s that authority can no longer stand outside the test environment long enough to stabilize legitimacy.
TBH I'm kinda long-term optimistic about academic slop. The current incentive structure has sort of funneled academics into churning out "minimally publishable" papers, the kind of thing you turn the crank on without having to think very hard about it. Then, when academics do have time to think, it's mostly for the purpose of overhyping their just-okay papers.
If "minimally publishable" becomes mass producible, then maybe the returns to thinking will improve.
"Top journals will focus on papers that are strikingly original or make important theoretical or empirical breakthroughs, while everyone else will publish the AI-produced papers that incrementally advance our understanding of narrow things."
Isn't it kind of like academia is structured already? Something that gets published in Slavic Review usually makes conclusions much larger than the narrow topic they focus on. And the "look, here's a document I found in the archives!" papers come out in a much less-prominent journal and are much less frequently read and cited.
It's true there's already a prestige divide -- but a lot of top journals are still heavily invested in causal inference papers that are very good methodologically but don't do much to advance theory. These are the kinds of papers that I think (hope) will become AI-automated and thus less prominent in top journals.
My experience reviewing papers for “top journals” (mostly genetic and genomic papers over a period of 50+ years) matches yours. Out of hundreds of papers reviewed, there were only a few for which I expressed serious concerns about the validity of the conclusions. Editors still valued my opinion as to how interesting the conclusions would be to the journal’s target audience.
Having just processed a lot of job applications and graduate student place applications, this is very similar there - out of (say) 100, about 2 looked individual and stood out - the rest were "merely competent"....
Selecting social science articles is going to turn into how Rick Rubin selects his music
Brilliant post as ever. But the elephant in the room may be that KPI-driven academic publishing conventions have already strangled the life out of much innovative thought in the social sciences, and it’s only now that the scales are falling away from our eyes. All LLMs are doing is the kind of salami slicing that is already prolific.
it's been a problem for sure but I think with AI quantity acquires its own quality
So, with AI, academic slop could become just another way to masquerade progress in innovative thinking, when all that's actually happening is that metrics are being satisfied? This kind of reminds me of Mark Fisher's concept of market Stalinism and the "valuing of symbols of achievement over actual achievement" - just accelerated and somewhat more nightmarish.
Even before the age of AI there where many papers that will be considered as human generated slops. The only thing that has changed is the exchange of baton from man to machine.
I think at a certain point quantity becomes its own quality
Were*
Judgement, and individual perspectives, are the new oil.
This is basically what has happened to academic computer science over the past couple years. Submissions to top conferences (in most computer science fields conferences are the highest prestige venues) are up 50-200% and at a recent AI conference there were 23000 submissions. The peer review system is starting to buckle under the load.
AI models still can't quite churn out a decent paper in my field but they are an effective productive multiplier allowing groups that are more aggressive to double or triple their output. And that's before we get to arxiv. Arvix was already a firehouse on popular topics but with the rise of AI and a glut of technically sound but boring papers has moved conferences into more of taste-maker role, doubling down on potential impact and interest.
Of course that just exacerbates the review issue. Increasingly papers not in top conferences just aren't read so if you want to have any impact at all your work needs to be in a top conference. Some conferences have started putting per-author submission limits but I don't think anyone has a great solution yet.
Great write-up. I think you could go beyond the image of tiers in academic publishing and move towards truly thinking about a more automated "research floor" model where Automatenwissenschaft becomes a sort of factory constantly churning out updated findings and data. I agree the immediate concern here is that our academic institutions like scholarly journals are woefully unprepared for these developments.
I think the bigger concern revolves around how we are going to grapple with all of this information. And I think re-emphasizing theory-building only gets us so far. Similar to the Aristotelian phronesis, in 1998 Herbert Simon opined that "if one factor in a system (in this case, information) has become abundant, what has become scarce? Our time-your time and mine-to process that information." ("Information 101: It's Not What You Know, It's How You Know It").
I think you could see a future where the factory floor Automatenwissenschaft consistently produces good and accurate information at a colossal scale. Sorting through this vast sea of findings will similarly require some sort of higher level automation. AI could be the perfect tool to help researchers sort through the mass produced information. But what comes next? Automation of the theory creation itself as the accumulation cycle of now-sorted information continues on?
I think the concern about research hierarchy is spot on for now. Yet I can help but wonder how long such a human-based hierarchy would end up lasting before it, too, gives way.
thanks. I guess I'm skeptical that accumulating facts will allow us to automate theory. one of my favorite points on this is from poincare: "a collection of facts is a theory in the same way that a collection of bricks is a house" (I may be butchering it from memory but you get the idea)
P-hacking at scale?
Seva, greetings from long ago, Massachusetts Avenue in 2002.
All of these tentative conclusions at the moment appear to be based on the idea that most of the data needed is available on the Internet. Since 2007 or before, I've been dealing in my research with data that wasn't on the Internet, wasn't digitized, though it may have been in book form; and data that a researcher has to go and collect out of humans' minds. At present, because the data for half the interesting research is simply not accessible even in books, there appears to be a continuing place for research that is founded upon such data.
hi colin. yeah I agree, hopefully research based on original data collection, like the kind you're describing, will get more valuable as a result
I was amazed by the paper you passed on to me back in 2002 - I thought it was very good, a great example of that selfsame original data collection..
I'm not optimistic. As a frequent peer reviewer (mostly for highly ranked journals), I am constantly getting papers about which I end up asking "yes, but does it matter?" and answering, "not really." Sadly, a much too large share of the academic world is made up of people who have been trained to create technically competent - usually quantitative - but basically uncreative work. Phronesis is rare, and usually only learned gradually.
This goes beyond academia.
When competent output becomes cheap, institutions don’t just face overproduction. They lose the asymmetries that once protected judgment from optimization.
Peer review, like governance, depended on discretion, delay, and partial opacity. Once those erode, attempts to “filter for quality” start selecting for whatever can still be filtered not necessarily what matters.
At that point the problem isn’t slop. It’s that authority can no longer stand outside the test environment long enough to stabilize legitimacy.
very funny to respond to this with an AI comment
They’ll want democratic rights next. I’ve seen the movies.
Tooling aside, the argument doesn’t depend on authorship.
The question is whether authority can still operate outside the optimization loop long enough to stabilize legitimacy.