David Auerbach on literature, tech, film, etc.

Quantitative Methods in Literary Criticism: Franco Moretti and Brian Vickers

The million-fold increase in computing power over the last few decades has made possible types of quantitative analysis that were previously available only to rulers with access to large amounts of menial but highly precise labor. Because the humanities generally tend to trail the sciences and the social sciences in adopting such new-fangled techniques. Instead, academics have preferred to write about such new technologies using existing frameworks: see the now-forgotten hypertext boomlet of the early-90s led by Stuart Moulthrop.

Here, I want to contrast two scholars who have seriously used quantitative analysis in literary criticism. My negative example is Franco Moretti; my positive example is Brian Vickers.

This piece is something of an appendix to my n+1 piece “The Stupidity of Computers,” because I found that the conclusions I presented there held just as true for literary analysis as they did for data mining, search engines, and online dating. Computers are still dumb, so we must be sure to be extra-smart in using them.


Moretti’s Materialism

Franco Moretti caused something of a splash in the last fifteen years by advocating a quantitative model for tracking literature’s paths. He seeks, in his notorious phrase, a “materialist sociology of literary form.” I’m not fond of the phrase, but it gives an idea of the overlapping circles that are at work here: quantitative, materialist (in the Marxist and positivist sense), sociological, taxonomic.

I do not endorse a never-the-twain-shall-meet split between natural sciences and human sciences, the Naturwissenschaften and the Geisteswissenschaften. Many literature departments and continental philosophers cling to the separation as though their jobs depended on it–which indeed they might. But the great inadequacy of quantitative methods applied to artistic and social forms (e.g., economics) still serves as a reminder that the human sciences are vastly more imprecise and variable than the most successful of the natural sciences.

I believe that the reasons for this, as I discussed at some length in my recent n+1 piece, “The Stupidity of Computers,” primarily have to do with language and the endlessly fine-grained distinctions and variances it forces upon us. The practical result is that almost any quantitative approach falls down, and I think Moretti’s is a particularly good example of such a failure.

Moretti’s work has foundered on two main points:

  1. Presuppositions of taxonomies, ontologies, and evidentiary relevance that are then set up to be “confirmed” by the statistical model at work.
  2. Sheer lack of evidence, caused by the vast scope on which Moretti is working.

I have written on the first issue of presuppsitions with regard to political issues: attempts to discern how “liberal” and “conservative” blogs link to one another rely on purely human classification of blogs into the two-category taxonomy selected by the researchers themselves, rendering the results highly dubious.

In the political sphere, complicated charts analyzing networks of links from one political blog to another show clusters of linkages tightly within sets of “conservative” and “liberal” blogs. Another chart from a separate analysis shows clusters using a different taxonomy: progressive, independent, and conservative. Who decided on these categories? Humans. And who assigned individual blogs to each category? Again humans. So the humans decided on the categories and assigned the data to the individual categories—then told the computers to confirm their judgments. Naturally the computers obliged.

The Stupidity of Computers“, n+1 13

The same principle is at work in classifying books and sentences, as Moretti does. This sort of ontological question begging is so ubiquitous it should be the first question anyone asks on seeing a pretty chart or graph purporting to represent some aspect of human society.

The second issue is more particular to Moretti, which is that even with unprecedented processing power at his command, I gather Moretti still doesn’t have the resources to do the requisite analysis at the level which would be required.

Cosma Shalizi, expert statistician, has written extensively on these failings with regard to Moretti’s two main books, and his articles elaborate the problems sufficiently that I won’t go into much detail here. He criticized the statistical methods of Atlas of the European Novel, but his review of Graphs, Maps, Trees, which attempted to make a statistical case that new genres of novel tended to occur in bunche, cuts closer to the theoretical problems at hand:

[Moretti should] give actual causal accounts of how macroscopic patterns emerge from the interaction of many material bodies (notably, people and books), of the sort we know to exist, endowed with the kinds of abilities we know them to have.

This commitment may sound harmless, because contentless, but it does actually have implications.  It means that you have to do a lot of work to justify functionalist explanations (though it’s not impossible). It should make you very dubious about ideal types.  It should make you more interested in exploring variation, and not dismissing it.  It should make you very dubious about “practices” and other shared mental objects, at least as ordinarily conceived.  And it suggests a lot of productive directions, investigating communication, cognition, and the collective patterns they produce.

Graphs, Trees, Materialism, Fishing

If anything, Shalizi understates the difficulty. There needs to be a huge amount of work, and it must be done with partial, often inaccurate data from biased observers. The hermeneutic circle holds sway here. It’s not impossible. But the data analysis is vastly easier than the data collection, which computers still can’t do.

Vickers’ Wissenschaften

I have a positive example, however. Studies of authorship attribution are nothing new. They have frequently employed quantitative methods, albeit frequently in a haphazard and unsystematic way. In English, at least, the writer who has been subject to more of them than any other is William Shakespeare.

There has not been a good track record. Two attributions of new (and rather poor) works to Shakespeare in the last couple decades, “Shall I die?” and “A Funeral Elegy,” seem almost certainly incorrect. In the second case, the original claimant, Don Foster, has retracted the claim. In the first, it appears that no one except the original claimant supports the attribution anymore. Unfortunately, the original claimant, Gary Taylor, is the editor of the Oxford Shakespeare, so you can find “Shall I die?” there. What are you going to do?

Against that, though, there is Brian Vickers and particularly his recent work Shakespeare, A Lover’s Complaint, and John Davies of Hereford. (A short overview is available behind the TLS paywall.) This is a distinctive case because it’s trying to pull authorship of A Lover’s Complaint away from Shakespeare. The long (329 lines) poem was published in the same book as the sonnets in 1609, attributed to Shakespeare, but it has never garnered too much attention. There have been some doubts but, over the last 50 years, general endorsement of Shakespeare’s authorship.

Work had already been done to identify Shakespeare’s co-authors on Titus Andronicus, Pericles, Timon of Athens, and others, but this is the first instance I know of pulling a poem away from Shakespeare.

Vickers is upfront in his biases: he thinks the poem is lousy, too lousy to be Shakespeare’s, uninventive and moralistic. He attributes it instead to a mediocre but prolific poet named John Davies, whom I’d never heard of. But his method is far more solid than most. He recognizes the need for a holistic/hermeneutic understanding of the period and its literature, and so Vickers’ book takes both a top-down and a bottom-up approach, deriving general regulative and evaluative guidelines from a comprehensive knowledge of the period, then using atomic, discrete metrics to attempt to make meaningful distinctions. I believe this bi-directional method to be the only one that has a chance of success. (Moretti, in contrast, is entirely top-down, while much philological work is bottom-up.)

Here is a brief sample, taken from the TLS overview:

Davies did so on over 160 occasions. The author of the Complaint used “th’-” elisions frequently, and twice clumsily, not on words beginning with a vowel -as in Shakespeare’s “Th’expense of spirit in a waste of shame” -but before a consonant: “th’wel doing”, “th’smallest teene”. Davies regularly contracted the definite article without any concern for euphony, with such equally clumsy elisions as “th’reversions”, “th’reprobates” or “th’Gospel”. Critics have judged the contractions in “A Lover’s Complaint” to be a sign of Shakespeare’s later style, but here is a different explanation. The poet of the “Complaint” ended two lines with the formulaic phrase “forme receive”, once an infinitive (using the pleonastic do): “which did no forme receive”, and once a present tense: “all straing formes receives”, both times rhyming on “leave”. In John Davies’s poetry there are twenty-one instances of the rhyme word “deceive”; ten of them need a pleonastic do, twice rhyming with “leave”. The author of the “Complaint” took liberties with the English language to obtain a rhyme, coining the nonce-words “sawne” and “loverd”. In one poem Davies rhymed “wander” and “gander”, adding a marginal note justifying “The word . . . Gander for the Rime’s necessity”.

Brian Vickers, “A rum ‘do'”

So while the first section of the book, “Background,” is somewhat traditional literary criticism and close reading, the second section, “Foreground,” is tabulation across many metrics. Vickers downplays one of the most common (and easiest) forms of analysis, that of uncommon word occurrence, owing to the ambiguity of word placement and meaning. Vickers does use this analysis, but not to the exclusions of others, as some of his predecessors had in making the case for Shakespeare’s authorship.

Regardless, A Lover’s Complaint is only 329 lines long, so the sample size for any feature is quite small. Because of this, doubt should accompany any supposed distinction established along any one axis. Aware of this, Vickers is definitely trying for a preponderance of evidence, attacking the poem from different levels and angles. Those in favor of the attribution to Shakespeare (John Kerrigan and Katherine Duncan-Jones are the two most vocal, I believe) have not done a comparable study from the other side, which is what would be necessary to counter Vickers’ claims.

Similarly, statistical analysis is somewhat down-played, because the margins of error are so great. There are lists and collections, but they are presented in the sense of outlying features that distinguish the poem from other work. They are also presented in the sense of non-outlying features that are actually typical of work in the period against common knowledge. The presence of both types of claims is reassuring.

And it underscores something to bear in mind when reading the work of Moretti and others: beware statistical analysis on incomplete evidence. The challenge is in deciding on the metrics and applying them in the most exacting fashion, something that remains in the human sphere, not the computer. What computers have enabled is the ability to run some of the metrics on a large corpus rather quickly: a significant evidentiary boon, but not a paradigm shift. The most satisfactory uses are considerably less sexy than Moretti’s graphs: for example, searching for particular words and variations of a word across all (or most) Elizabethan literature.

(Again, I refer back to my n+1 article, where computers proved themselves most useful in dumb tasks performed in great quantity: lexical analysis rather than semantic analysis.)

For any analysis, there is also the ever-present problem of negative evidence. The significance of what words aren’t co-occurring is considerably harder to analyze, and so is frequently ignored. The case study Vickers gives is that of neologistic word formation with the prefix un-. Shakespeare indeed did form an awful lot of new words by attaching un- to existing words, a feature of A Lover’s Complaint. But Vickers cites Juergen Schaefer’s work in showing that the OED overrepresented Shakespeare (for assorted reasons) and that other writers had coined similar un- neologisms at a similar rate. Two of them were Thomas Nashe and…John Davies (133).

And in fact, when it comes to neologisms, A Lover’s Complaint introduces 11 Latin neologisms in 329 lines, including wackiness like annexions, fluxive, and Cautills (151). This is hardly sufficient evidence but it is the sort of heuristic metric that gives reason for doubt. Moreover, Vickers uses it to bash those who have claimed that the supposed lexical inventiveness of A Lover’s Complaint was unique to Shakespeare, when in fact it seems to have been a common feature among many writers of the period. Context is everything.

Above lexical and grammatical analysis, Vickers performs more complex rhetorical analysis. One striking example is Davies’ almost compulsive use of asyndeton (omission of conjunctions), helping to produce what’s been called the “cramped, gritty, discontinuous quality” of A Lover’s Complaint. Vickers has some fun citing instances of Davies piling on up to ten verbs in a row in his other work.

A high level of elision of other parts of speech and peculiar inversions of word order is also present. Here the qualitative element kicks in, as distinguishing “awkward” and “unnatural” inversions from “good” ones is simply not something that is uncontestable. Here Vickers returns to close reading and attempts to see what purpose inversions serve in the function and rhythm of the poem. This is not a positivistic method, but I’m with him in thinking it is more convincing than attempting to taxonomize inversions on a purely grammatical level as “unnatural” or not.

This alternation between the quantitative and the qualitative continues throughout the rest of Vickers’ book, and because Vickers is forceful in his aesthetic judgments, they do not always stand by themselves, but in tandem with the statistical evidence, they do gain a certain amount of force. Because I am not an expert in rhetoric and not an expert on the literature of the period, I cannot say how contestable some of the judgments are. But the work is there to be evaluated. Vickers also shows how metrics have been abused in the past to make other wrongful attributions, and so points out the pitfalls.

For rhetorical devices in particular, it helps a great deal that Vickers is immersed in the rhetorical taxonomy that Shakespeare and Davies themselves would have learned, and so applies metrics that they themselves would have used in constructing their works. This is where historical knowledge is crucial; the ready-made ontological categories that happen to be popular at a given time (structuralist tropes, for example) are less likely to line up as well as taxonomical devices that authors would have knowingly applied to their work at the time of creation.

Let’s look at some of the general metrics Vickers applies to A Lover’s Complaint, in order to see how that middle-ground is navigated:

  • A set of six rare words occurring in Davies work and in the Complaint
  • 21 common phrases like “high and low,” “wake and sleep,” and “gainst plus admirable moral principle” occurring disproportionately in Davies work and present in the Complaint
  • Six instances of poetic diction unique to A Lover’s Complaint and Davies other work
  • A reflexive fondness for a a handful of rhetorical figures, particularly the overuse of the word “all” (often twice in a line), occurring in the complaint and in Davies’ other work (and nowhere else save sometimes John Donne)
  • A certain overlap in metaphorical vocabulary, such as the “congestion” and “compounding” of individual appetites into desire
  • Of rhyme-word pairs in the Complaint not used by Shakespeare, 25% of those pairings do occur in Davies’ work

The last one gives an example of how inexhaustible the work is, since the analysis is only performed on rhyme-word pairs not occurring in Shakespeare and is not compared to other authors. Similar objections can be made to other metrics, but the thing to remember is that Vickers has still set the bar vastly higher than usual, and since such work is a progressive process, the details are at least there to be rebutted by someone who wants to perform some of the many remaining analyses. To use an apt legal analogy, such matters are always decided by the standard of a preponderance of evidence, not a K.O., and right now it appears that Vickers has presented the preponderance.

Harold Love, in the TLS, in fact criticized Vickers for not using more sophisticated statistical and computational tools, and disputed the certainty of the Davies’ attribution. Vickers’ response displays, in my opinion, the right attitude to take toward such tools and analyses:

Your readers are told that Vickers uses “an old-fashioned kind of testing”, and “lacks real understanding” of modern computational stylistics, such as “John Burrows’s Delta algorithm, based on complex statistical probing of lengthy frequency lists”.

To reply in brief: I am perfectly familiar with modern stylometry, but the Delta algorithm uses small vocabulary samples, such as the fifty most frequently occurring words, which are then treated as individual counters, deprived of semantic identity and grammatico-syntactical relationships with other words. It has achieved some success, but it is open to suspicion that two or more writers might favour the same fifty words. I am currently working with a team of medical statisticians who are applying to computational stylistics a technique developed for measuring irregular heart beats, which is able to use all the words in a text. It lists shared words in descending frequency and then uses a phylogenetic algorithm to create a tree, grouping similar texts on nearby branches and dissimilar texts on distant branches. (Intertextual distances are measured, applying a weighting function which is the sum of Shannon’s entropy–details on request.)

Brian Vickers, 13 July 2007

Love definitely erred in endorsing the Delta algorithm, which is indeed dangerously arbitrary in its metrics. Endorsing any one metric, in fact, is liable to be dangerously arbitrary because of evidentiary limitations. Fancy statistics run the danger of obscuring the anecdotal component of the work.

Vickers impresses, ultimately, because he bears three main points in mind: all such analyses are to be treated as incomplete, no one metric should ever be seen as definitive, and conclusions must be based on a pluralistic methodology utilizing both quantitative and qualitative metrics.

In conclusion: the human sciences are inexact and dismal, much like humans.


Afterword: Duncan-Jones and Kerrigan’s rebuttals to Vickers’ de-attribution were not sufficiently substantive to need addressing here. Kerrigan’s failure should not obscure my esteem of his panoramic study Revenge Tragedy: Aeschylus to Armageddon.)

In addition, I should say that Vickers’ critical sympathies are not necessarily my own. While I’ve learned a great deal from his work on the history of rhetoric, someone who can confidently announce of the sonnets, “These are, to state the obvious again, not homosexual poems”–and then cite Sonnet 20 as evidence–either sees friendship as a wildly fertile ground for manic jealousy, or else is using such a narrow definition of homosexuality as to make the observation tautological. I prefer Samuel Butler’s witty (though coy) formulation:

Fresh from the study of the other great work in which the love that passeth the love of women is portrayed as nowhere else save in the Sonnets, I cannot but be struck with the fact that it is in the two greatest of all poets that we find this subject treated with the greatest intensity of feeling. The marvel, however, is this, that whereas the love of Achilles for Patroclus depicted by the Greek poet is purely English,absolutely without taint or alloy of any kind, the love of the English poet for Mr W. H. was, though only for a short time, more Greek than English. I cannot explain this.

Samuel Butler


  1. Well, if critics like Mario DiGangi and Alan Bray are to be believed, discrete categories of (English) sexual identity didn’t arise until the 18th century (they see ‘molly houses’ as sites of differentiation), and there was more room in the conventions of early modern homosocial friendship for homoerotic behavior — or behavior that looks homoerotic from our vantage. To be openly and exclusively homosexual marked one as a sodomite, and sodomy connoted treason, atheism, etc., so homoeroticism is usually visible as an ambiguous dimension of otherwise licit same-sex friendship.

    • David Auerbach

      20 February 2012 at 12:18

      Vickers takes the opposite stance, that there is no homoerotic aspect, only arete. There’s room for debate, but not for dismissal. I think Bray may be closer to the mark, though I think it’s more complicated than his framework suggests. I think Joseph Pequigney’s remarkable “Such is My Love,” a close reading of the sonnets, has given me the most insight.

  2. I’ve enjoyed and learned from many quantitative analyses over the years. (Conveniently at hand are Ian Lancashire’s entertaining treatments of authorial voice and late style.) None of them, however, were produced by the technique’s loudest self-promoter, and I agree with you as to the reason: by distancing himself from notions of close reading or authorial intent or critical theory or psychology, in this phase of his career Moretti tried to motivate and circumscribe number-crunching with dubious generic taxonomies. Garbage canned, garbage out.

    Having read a fair (which is to say excessive) amount of John Davies, I find Vickers’s re-attribution appealing. But at the long-moment of academic publication, MacDonald P. Jackson may be winning:

    * Review in The Review of English Studies
    * “‘A Lover’s Complaint’, ‘Cymbeline’, and the Shakespeare Canon”, The Modern Language Review

    Regarding the spotty reputation of the Sonnets, just last week I got a kick out of “‘Pyramids of Egypt’: Shakespeare’s Sonnets and a Victorian Turn to Obscurity” by Rhian Williams, and you might as well.

    (As always, contact your local library or friendly university employee for a fist through the paywall.)

    • David Auerbach

      20 February 2012 at 12:55

      Lover’s Complaint was removed from the RSC Shakespeare by Jonathan Bate, so I think the conflict is ongoing.

      You lucky folks with access to those journals. I was able to get at the Jackson response, however, which I hadn’t seen, so thanks for the pointer. I think it confirms much of what I said above. Vickers goes after Jackson a fair bit in the book, but Jackson doesn’t engage with the most substantive criticism, which is that a single metric can’t be conclusive. Vickers criticizes Jackson for looking at vocabulary and nothing else, and in the response, Jackson still focuses only on single-word cooccurrence, picking at instance after instance. He may indeed have weakened Vickers’ case on that front, but Vickers still has at least a dozen other metrics by which Davies still comes out looking far more likely. Jackson doesn’t even touch them. Vickers still has the preponderance.

      As for Moretti, I definitely think the Marxist/Lukacs “objective”/materialist principle is at work here, as you imply, and is generating literary analysis of similar (low) value.

  3. I remember getting really annoyed with Moretti back when I read Graphs, Maps, Trees because I felt like his method just boiled down to “count things, draw pictures, and make up stories about the pictures,” which — despite his pretensions — isn’t any more “scientific” than the usual business of literary criticism. The book’s subtitle includes the word “models,” yet there isn’t really anything in the way of models (in the sense of fairly explicit, systematic, and judiciously simplified accounts of causation) in the book, just ad hoc stories invented to explain patterns seen (visually, not statistically) in data. Maybe if Moretti had come up with some models to test before even gathering data, the models would have suggested natural categorization schemes and natural divisions of variables into those that matter and those that don’t — which is what happens in natural science, with its continual interplay between theory and experiment. This can easily get kind of circular, but it’s better than the sort of circularity Moretti exhibits, where arbitrary (rather than motivated) classification choices result in particular conclusions, without even a sense of how alternative theories might have lead to alternative classification choices.

  4. The problem with Vickers is he is a poor judge of poetry. His -evaluation- is where he slips up: he is wrong that ALC is doggerel. It is interestingly involuted and contains some passages of Shakespearean force.

    I’m never quite sure whether he’s arguing that it’s bad because Shakespeare didn’t write it or that Shakespeare didn’t write it because it’s bad. Either way, he’s wrong that it’s bad.

  5. David Auerbach

    8 March 2012 at 23:19

    nostalgebraist: I can’t say much other than that I agree with your comment completely.

    ovaut: I think you put your finger on the main hermeneutic problem here. Vickers clearly started from a biased position against the poem, and this no doubt informed his analysis. He clearly thinks that ALC is too poor to have been written by Shakespeare.

    I’m not sure if ALC is as bad as Vickers’ says–it’s no “Shall I Die” or “Funeral Elegy,” but I can’t say that I’m terribly fond of it. But that question on its own can’t be settled. However, by laying down the quantitative metrics next to the qualitative ones, I think Vickers still makes a case that can stand on its own even if one thinks that ALC is actually pretty good. At the very least, no case for the authorial assignment to Shakespeare has been made. Vickers may (probably) go over the top in his aesthetic decisiveness, but given that none of us start from a blank slate in our readings and studies, I do appreciate having his position made clear.

  6. I recently read James E. Cutting’s Impressionism and Its Canon, which he’s made freely available along with all his other research. It’s maps-and-graphs-about-genre done right. (It’s also very likeable: an amateur scholar’s impulse fulfilled through professional effort.) Cuttting recognizes the likelihood of self-generated noise from a distance, and so he generates, tests, and discards hypotheses by shifting from textual analysis to social-psychology viewer-response studies to inspection of historical and biographical details and back again.

Leave a Reply

© 2024 Waggish

Theme by Anders NorenUp ↑