4. Stylometry, validated before it was used

The only published stylometric study of the letter is by Jack Grieve and six colleagues (Isobelle Clarke, Emily Chiang, Hannah Gideon, Annina Heini, Andrea Nini and Emily Waibel), "Attributing the Bixby Letter using n-gram tracing," in the journal Digital Scholarship in the Humanities 34 (2019), pp. 493–512. It concluded that Hay "was almost certainly its primary author." No replication has been published since, as far as we could find (section 8). We rebuilt the method from the paper's description, ran it on corpora we built ourselves, and then ran the tests the paper did not run: whether the method still works when the known texts resemble the letter in date and register, and whether its answer depends on which texts of each man are put in the comparison pool.

The corpora

Every Lincoln text comes from Basler's Collected Works, volumes I, IV, VII and VIII (the volumes for 1849 to 1859 and November 1861 to August 1863 are not available in full text; the 1849 to 1860 gap was filled from Arthur Brooks Lapsley's edition of 1905, an older printed collection). Basler's source codes let us keep only what survives in Lincoln's own hand (ALS, ADS, ADfS and the like) and set aside the items Basler says are in Hay's handwriting. Hay's writing comes from the three volumes of his letters and diary that his widow had printed privately in 1908 (Letters of John Hay and Extracts from Diary), from Thayer's biography, and from his books; only his 1860 to 1865 letters and diary are contemporary with the letter. Twenty other writers of the period supply a control set, used to check that a method separates Lincoln from Hay rather than from writers in general: William H. Seward, Salmon P. Chase and Gideon Welles of Lincoln's cabinet; Senator Charles Sumner; the secretaries John G. Nicolay and William O. Stoddard; the journalists Noah Brooks, Horace Greeley and George William Curtis; the orator Edward Everett; the writers Ralph Waldo Emerson, James Russell Lowell, John Lothrop Motley, Bayard Taylor, Mark Twain, Walt Whitman and Henry Adams; Generals Grant and Sherman; and Frederick Douglass.

The questioned text is the Transcript printing, 139 words after the salutation. Every copy of a test text was removed from the pools before it was tested, and the full document from which each register piece was cut was removed too.

Grieve's method, and what it reproduces

N-gram tracing takes every distinct string of one, two or three consecutive words and every distinct string of 3 to 16 consecutive characters (spaces included) in the questioned text, its word and character n-grams, and asks what share of them occurs anywhere in an equal-sized random sample of each candidate's writing. The candidate with the larger share wins, one vote per n-gram length, and the authors take the majority of the character 4- to 10-gram votes as the result. We reimplemented it exactly as described (capitals lowered, punctuation kept, no string running across a sentence boundary, samples of 0.95 times the smaller pool, and the random sampling repeated 10 times for validation and 50 times for the letter, with the shares averaged).

On known texts it behaves as the paper says. Under Grieve's design, Lincoln before May 1860 against all of Hay, leave-one-out attribution (each known text removed from its pool and attributed as if unknown) of 423 Lincoln and 588 Hay texts of 50 to 400 words is correct by the majority vote for every text. At single lengths the only misses are one Lincoln text at 3 characters and one or two Hay texts at 3 characters and at each length from 10 to 16. Under our own-hand design, Lincoln's autograph writing of 1860 to 1865 against Hay's letters and diary of the same years, the vote is right for 643 of the 645 Lincoln texts and for all 93 Hay texts. No single character length from 5 upward misses more than four Lincoln texts, and no length misses a Hay text. The 182 Lincoln and 31 Hay texts of 100 to 200 words are all attributed correctly by the vote. A 139-word text is not too short for the method when the text is an ordinary letter, telegram or diary entry. Grieve's confidence rests on this validation, and the validation holds.

The letter also comes out as the paper reports. Under Grieve's design all seventeen analyses give it to Hay: the 3-character analysis by four parts in a hundred thousand, the rest by margins of 0.01 to 0.14, with 0.04 to 0.10 for character 5- to 12-grams and 0.14 for word 2-grams, where the margin is Lincoln's share minus Hay's. (The paper reports the 5-word n-gram analysis for Lincoln on one phrase; we did not run 4- or 5-word n-grams.) Figure 1 plots the margin at each of the seventeen string lengths under this design and under the own-hand design described below.

Three tests the paper did not run

The authors had anticipated the second of these tests, on register. Their first presentation of the work (Aston University, 13 April 2016) noted that they had "not looked at detail at the effect of ... register variation" and that they "also need to test sensitivity to register variation", and its last slide listed testing "across different registers" as work still to do. The published paper says that "a systematic analysis of the writings of many authors and many registers is needed" and was being pursued. The tests below are new, and a second implementation of them, written independently, is reported at the end of this subsection.

Period. Grieve's Lincoln pool ends in May 1860 because the presidential correspondence is contaminated by secretaries. But Lincoln's style changed. When his own-hand writing of 1860 to 1865 is tested against Grieve's pool, 14 percent of the 182 texts of 100 to 200 words are attributed to Hay by the majority vote, and 18 percent of the 377 texts of 50 to 100 words; the figure for his pre-1860 texts is zero. Tested against each other, Lincoln before 1860 and Lincoln in the White House are separated by the method with 100 percent accuracy at character lengths 5 to 16. The method sees a difference between the two Lincolns as clearly as it sees a difference between Lincoln and Hay. The Bixby letter, tested against the two Lincoln periods alone, sits between them.

Register. Both pools are dominated by business prose, and the letter is not business prose. On the register-matched pieces the picture changes. Under Grieve's design the majority vote gives 30 of the 63 Lincoln pieces to Hay, including the Gettysburg Address, the Ellsworth letter and the McCullough letter, while 56 of the 57 Hay pieces go to Hay. The second implementation, on the same pieces with a smaller Lincoln pool, gives 19 of the 63 to Hay; the cause is traced below. At character lengths 4 to 6 the method is right about Lincoln's elevated prose only 21 to 38 percent of the time. Under the own-hand design the vote misses 17 of the 63 Lincoln pieces and none of Hay's. Short character strings, which carry function words (the, of, and), word endings and spelling, are the ones that fail; they mark the register, and the register of an elevated Lincoln text looks like Hay to a pool of Lincoln's telegrams. Long strings, which carry phrases, recover: at character lengths 11 to 16 the own-hand design is right about 89 to 94 percent of Lincoln's pieces. Figure 2 plots, for each string length and each design, how often the method attributes the register-matched pieces correctly, beside its rate on ordinary known texts.

The register set has limits of its own. The 63 Lincoln pieces come from 30 documents, and the 57 Hay pieces from three: two eulogies of 1861 and one short letter. The validation used below, which leaves out one source document at a time, therefore tests the classifier on only three Hay sources, and a Hay eulogy is not a Hay condolence letter. The test shows that the 2019 design misreads between a third and a half of these pieces of Lincoln's elevated prose, the share depending on the pool. It cannot show how that design would treat a wider range of Hay's writing in this register, because no such range survives from the 1860s.

Edition. Lincoln's own-hand texts are transcribed from manuscripts and keep his habits ("can not," "it's," dashes); the letter survives only in print, and the Lapsley texts used for 1849 to 1860 and for the 1862 to 1863 register pieces are print. We found 95 Lincoln texts that appear in both editions. The print version of the same words scores 0.010 to 0.016 nearer to Hay than the manuscript version at every n-gram length (at character 4-grams, in 85 of 95 pairs), and its majority-vote accuracy falls from 0.93 to 0.74 at 4-grams. In every design whose Lincoln pool is built from manuscripts, a printed Lincoln text therefore scores slightly nearer Hay than the same text in manuscript would.

A second implementation, and why its rate is lower

A second implementation of n-gram tracing was written from the paper alone, by a different hand, without sight of our code or our results. It rebuilt the two pools from our corpus files under the paper's design and reran the tests. Its report is cited in section 9 and its files are saved beside ours. On the letter it agrees with the paper and with us: Hay at all seventeen lengths, by margins of 0.002 at 3 characters to 0.145 at word 2-grams. Its leave-one-out validation of all 1,397 texts in its pools is right by the vote for 1,380, with the paper's accuracy curve (its F1 score, a standard accuracy measure that counts both wrong attributions and missed ones) at every length (the paper: all 1,662 of its texts; ours: all 1,011 of 50 to 400 words); its misses are Lincoln's verse, his 1842 eulogy, his temperance address and reported speech. On our 63 register pieces it sends 19 to Hay by the vote, 30 percent, against our 30, 48 percent; on 97 pieces of its own from 38 Lincoln documents of 1861 to 1865, 27, 28 percent. It agrees with us on which lengths fail (4 to 6 characters, at 25 to 44 percent accuracy) and on the Gettysburg and Ellsworth pieces, and it disagrees on the McCullough letter, which it gives to Lincoln four votes to three. Under period-matched pools of its own it finds what we found under the own-hand design: the letter splits, five lengths to two for Lincoln on the character vote and nine to eight for Hay over all seventeen, and all four of Hay's letters for Lincoln's signature, and Hay's own condolence, are called Lincoln.

The gap between 30 and 19 is in the Lincoln pool, not in the code. The two implementations agree piece by piece on 48 of the 63; on twelve of the fifteen they disagree about, one side or the other is a vote of four to three. Our code on the second implementation's exact pool gives 19 of 63 under the paper's rule for drawing the random samples of each man's writing, repeated 50 times, 19 under our rule repeated 20 times, and 20 repeated 10 times. Its code on our pool gives 26. Repeating our own run with three different random draws gives 30, 30 and 28, and under the paper's sampling rule (the whole smaller pool, repeated 50 times) 29. The one difference that matters is a set of 19 items from Lapsley's third and fourth volumes: 83,000 words of Lincoln's 1858 campaign speeches and his side of the seven debates with Stephen A. Douglas, his opponent for the Senate, which the second implementation dropped because Lapsley prints Douglas's letters and some interrogatories inside a few of them (a few hundred words at most that are not Lincoln's). With those 19 items removed, our code gives 18 to 20 of 63; with its selection of texts applied to our own copies, 18; and when the second implementation restored the substantive Lapsley items to its own pool, with Douglas's words cut out, its code gave 27 of 63. On its 97 pieces, our code gives 28 with its pool (its own figure is 27) and 40 with ours. Nothing else moved the figure.

The mechanism is the method's sampling rule. Each candidate is represented by a random sample the size of the smaller pool, so adding the debates raises Lincoln's pool from about 159,000 to about 243,000 words and Hay's sample by the same amount. Both halves of that matter. At an equal sample of 159,000 words, a Lincoln sample that includes the debates in proportion misses 24 of the 63 pieces where one without them misses 20: stump oratory supplies few of the strings of a condolence letter. Raising both samples to 243,000 takes the figure to 29: the extra words of Hay's mixed prose cover more of those strings than the same number of extra words of Lincoln's oratory do. (Raising Hay's sample alone, with Lincoln's held at 159,000, sends 54 of the 63 to Hay, which is why the method insists on equal samples.) Accuracy at 4 to 9 characters falls by 0.06 to 0.14 at every length when the debates are in. Which pool is the right one is a question about the paper's design, not about either implementation. The paper's Lincoln corpus, 1,085 texts and 400,747 words from Basler's first four volumes with texts up to 17,003 words long, included his speeches, and its samples were 261,000 words each, the size of its Hay corpus; our pool with the debates, 243,000 words, is the nearer reconstruction on both counts, and the second implementation's, 192,000, is the more cautious about Douglas. Neither is the paper's corpus, which was not obtainable. We therefore report a range: under the paper's design between 19 and 30 of the 63 register-matched Lincoln pieces go to Hay, 30 to 48 percent, and between 28 and 40 of the second implementation's 97, and the rate rises with the amount of Lincoln's oratory in the pool. The attribution therefore depends on which Lincoln texts are in the pool, and the range above measures the size of that dependence.

The letter under the own-hand design

Under the own-hand design the seventeen analyses split, eight to Hay and nine to Lincoln. Character 3- to 8-grams, 10-grams and word 1-grams go to Hay, by margins of 0.002 to 0.046; character 9-grams, 11- to 16-grams and word 2- and 3-grams go to Lincoln, by margins of 0.000 to 0.018. The 4- to 10-gram vote is Hay, six lengths to one. A rerun with punctuation removed gives the same split. Read against the register-matched pieces, the short-string margins are typical of Hay's eulogies and outside the range of Lincoln's elevated pieces (at 4-grams the letter is below every one of the 63 Lincoln pieces), and the long-string margins are typical of Lincoln's pieces and outside Hay's (at 15-grams it is above every one of the Hay pieces). A logistic regression model (a standard statistical classifier) trained on the full set of margins of the 120 register pieces, validated by leaving out one source document at a time, classifies 92 to 95 percent of the Lincoln pieces and 89 to 90 percent of the Hay pieces correctly, and gives the letter a probability of Hay of 0.62 with all Hay pieces and 0.50 when the sixteen Hay pieces that are more than a quarter quotation are dropped. The same model trained under Grieve's design gives 0.98, and under a third design (all of Lincoln's autograph writing against all of Hay's) 0.61 to 0.73, a design that also gives the Gettysburg Address to Hay at 0.79 to 0.87.

Other methods

Three methods that do not depend on shared strings were validated the same way and then applied.

A fourth check trains the same character classifier to choose among Lincoln, Hay and the twenty control writers. It ranks Lincoln first for the letter (0.12, with Hay at 0.05), first for all four letters Hay composed for Lincoln's signature, and first for Hay's own condolence; it also ranks Seward first for the Gettysburg Address and Lowell first for the McCullough letter, so it has nothing reliable to say about elevated prose and is not counted.

The last column is the important one. Hay wrote a condolence letter of his own in June 1864, and he composed at least four letters that went out over Lincoln's signature in 1863 to 1865 (to Boker, Garrison, Driggs and Charles Butler). The classifiers and Delta call all five Lincoln's. Own-hand n-gram tracing calls three of the four letters for Lincoln's signature Lincoln's by the majority vote; Garrison goes to Hay by margins of 0.004 to 0.021 at 4 to 9 characters and to Lincoln above that. It calls Hay's own condolence Lincoln's, four votes to three. The 19 documents in Basler's edition of 50 to 200 words that survive in Hay's hand, most of them telegrams and orders that Hay may have composed or only copied, go to Lincoln 18 times out of 19 under the own-hand design and 13 times under Grieve's. The methods are good at telling Hay's diary from Lincoln's telegrams. They have not been shown to detect Hay writing as the President, which is the hypothesis in question.

What the letter shares with each man

The margins can be read directly. The strings of the letter that occur in Lincoln's own-hand war writing and in none of Hay's are the office's: "in the files of the War Department," "a statement of the Adjutant General," "that may be found in," "should attempt to," "of mine which," "and leave you," "tendering to you," "and respectfully." The strings that occur in Hay's writing and in none of Lincoln's are the elegy's: "the altar of," "solemn pride," "gloriously on the field of battle," "the anguish of," "the consolation that," "from the grief of," "any words of mine," "refrain from," "memory of the loved." Nearly all of Hay's are in his post-war books, not his war-time letters. The letter's business idiom is Lincoln's office idiom, which any drafter in that office would have used; its elegiac idiom is one that Lincoln's surviving writing hardly ever had occasion for.

Raw counts. Lincoln's "our Heavenly Father" occurs in the proclamation of 10 April 1862 and the reply to Mrs Gurney of October 1862, neither of which survives in his hand and neither of which is in the volumes available to us; Bullard prints both. "Beguile" does not occur in Lincoln's Collected Works at all (Burlingame 1995); Hay used it at least thirty times in his life (Burlingame 1999), once in his war-time letters and twice in the "Colonel Baker" essay.

The word counts are single digits on both sides. "Beguile" is the strongest single item for Hay and has been since Burlingame noticed it, and it remains one word in a register in which Lincoln's surviving prose amounts to three letters. "Tender" as a verb, which Burlingame also claimed for Hay, is Lincoln's: 22 instances in his own-hand war writing against none in Hay's contemporary letters and one in all of Hay's half-million words, where the word appears 74 times as an adjective.

What 139 words can tell

On ordinary texts the answer is "a great deal": at 100 to 200 words the methods are 95 to 100 percent accurate. On texts of this letter's register the same methods are 73 to 95 percent accurate, the two designs give opposite answers for the letter, and the genuine Hay condolence and the four genuine Hay-for-Lincoln letters pass as Lincoln under every method but one, which catches one of the four. Taking the methods together, the likelihood ratio the stylometry supports for Hay against Lincoln lies somewhere between one to three and three to one. We treat it as one. On our tests the 2019 result depends on the corpus design as much as on the letter. A second implementation, written independently, agrees on the direction; the size of the register effect depends on the Lincoln pool, as shown above. Neither implementation used the paper's own corpus.