Orwell on AI

The author of 1984 predicted that literature would be written by machines. But he was wrong about why.

The Enigma of the Oracle, Giorgio de Chirico (1910)

Prescience is a fickle gift. We credit it to wisdom, but just as often it’s a byproduct of luck or well-timed anxiety. Sometimes the prediction is right, even remarkably so, but for the wrong reasons.

So it is with Orwell’s 1946 essay “The Prevention of Literature”. The essay is usually remembered for its central claim that literature and totalitarianism cannot coexist. Good art is like a panda: it will not breed in captivity.

But toward the end of the essay Orwell attempts something stranger. He tries to imagine what people will read in a future where literature was dying. And he comes up with an automaton that produces stories on demand, with no human intervention.

“It would probably not be beyond human ingenuity to write books by machinery,” he writes. The books would come off a “conveyor-belt process that reduces human initiative to the minimum,” each one “no more an individual product than a Ford car at the end of the assembly line.” It’s a soulless thing, a contraption of prefabricated phrases. “Imagination — even consciousness, so far as possible — would be eliminated from the process of writing.”

This wouldn’t even be that radical of a change, Orwell continues, since “a sort of mechanizing process can already be seen at work in the film and radio, in publicity and propaganda, and in the lower reaches of journalism.”

The Disney films, for instance, are produced by what is essentially a factory process, the work being done partly mechanically and partly by teams of artists who have to subordinate their individual style.

Radio features are commonly written by tired hacks to whom the subject and the manner of treatment are dictated beforehand: even so, what they write is merely a kind of raw material to be chopped into shape by producers and censors.

So also with the innumerable books and pamphlets commissioned by government departments. Even more machinelike is the production of short stories, serials and poems for the very cheap magazines. Papers such as the Writer abound with advertisements of Literary Schools, all of them offering you ready-made plots at a few shillings a time. Some, together with the plot, supply the opening and closing sentences of each chapter. Others furnish you with a sort of algebraical formula by the use of which you can construct your plots for yourself. Others offer packs of cards marked with characters and situations, which have only to be shuffled and dealt in order to produce ingenious stories automatically.

A pack of shuffled cards with pre-written plot points functions pretty well as the 1946 equivalent of LLM slop. AI did not invent the production of derivative, formulaic garbage; that industry was running strong decades before the transformer. LLMs just dropped the marginal cost of its production to zero.

What Orwell described from the production side, B.R. Myers documented, fifty-five years later, from the consumption side. I wrote in March about his remarkable 2001 demolition of the era’s prestige idols: Auster, DeLillo, McCarthy, Proulx et al. Completely by accident, his essay also anticipates the AI writing problem. Myers’s target was prose that performs difficulty without requiring concentration, sounds literary enough despite lacking substance, and can be skimmed without loss because there is nothing to lose. His conclusion was that the illusion of insight can be manufactured separately from insight itself, and that a large audience, including sophisticated critics, cannot tell the difference or maybe doesn’t care. As I wrote then:

Myers’ argument suggests that what made surface-level profundity popular with critics a quarter century ago is exactly what makes LLM writing popular today. Which means we can’t blame the machines without letting ourselves off the hook. In both cases the verbosity, the hedging, the faux-profundity are built in because the illusion of insight is the point; true understanding would spoil that illusion and is therefore unwelcome.

Put the two essays side by side and you have a nice division of intellectual labor. Orwell diagnosed the supply of slop and Myers diagnosed the demand. But Orwell placed the locus of mechanized writing inside cheap magazines, the “lower reaches” of journalism, and took for granted that the top of the market ran on real talent that only a police state could abolish. Myers, by contrast, found the assembly line operating at the top, in prize-winning novels that were classified as serious literature precisely because its prose was ornamental and vague.

Where Orwell’s prescience fails him is on the causes of this mechanization. He saw the rise of totalitarianism and the collapse of liberalism as the forces that would kill literature:

[I]f the liberal culture that we have lived in since the Renaissance actually comes to an end, the literary art will perish with it.

The death of literature would be imposed by a dictator. In the totalitarian future “the surviving literature of the past” would “have to be suppressed or at least elaborately rewritten.” But while book bans remain a sore subject in the culture wars, no one is really suppressing the literature of the past. It has never been so easily available: every novel Orwell loved is a free download on a sketchy Russian website. The books are preserved, freely available, and ignored. What did Rage Against the Machine say? “They don’t gotta burn the books they just remove ‘em.” Turns out they don’t even need to do that.

The strange thing is Orwell had the correct answer at first, but backed away from it. In the essay’s second paragraph he names two enemies of intellectual liberty: “its theoretical enemies, the apologists of totalitarianism, and its immediate, practical enemies, monopoly and bureaucracy.” That second category includes the press lords, the film magnates, the hack work every writer must do to eat: what Orwell calls the “vague economic forces” eating away the artist’s independence and livelihood. But then, in a quick sweep, he sets those “practical enemies” aside to concentrate on the ideological-totalitarian threat, which was foremost on his mind. He spent the following two years on Jura writing the novel about the other enemy.

But the mechanizing pull of the market, which Orwell treated as a nuisance in the foreground of a much larger catastrophe, turned out to be the real catastrophe. He assumed that mass-produced slop would stem from banned alternatives, that readers would have to be driven to the conveyor belt at gunpoint. We’re skipping to it on our own.

In one respect what we have now is a darker outcome than Orwell imagined. The decline of literature did not require a Big Brother to imprison authors. A regime that suppresses books at least believes in their power, the censor’s attention being a form of tribute to that power. But today it’s not a dictator but broad social and technological forces that are driving people toward slop. That kind of diffuse transformation may be even harder to reverse than a top-down ban. Orwell thought the death of democracy would kill reading. It turns out the death of reading may be killing democracy.