9. Sources and reproducibility

Primary sources read in full

• Roy P. Basler (ed.), The Collected Works of Abraham Lincoln (Rutgers, 1953), vols I, IV, VII and VIII, from the archive.org scans collectedworksof015581mbp, collectedworksof015582mbp, collectedworksof015583mbp, collectedworksof015584mbp. Vols II, III, V and VI are lending-only on archive.org and the University of Michigan online edition refused the connection; their absence is a stated limitation of the Lincoln corpus. The Bixby letter and note: VIII, 116 to 117.
• Letters of John Hay and Extracts from Diary (privately printed, 1908), 3 vols, archive.org lettersofjohnhay01hayj, 02, 03; William Roscoe Thayer, The Life and Letters of John Hay (1915), 2 vols, lifelettersofjoh01thay, 02; John Hay, Castilian Days (Project Gutenberg 7470), The Bread-Winners (16321), Pike County Ballads (6062), Poems (10518), Addresses of John Hay (1906, archive.org addressesofjohnh00hayj).
• William Schouler, A History of Massachusetts in the Civil War (1868), I, 576 to 578, archive.org massinthecivilwar01schorich; Annual Report of the Adjutant General of Massachusetts for 1864, archive.org annualreportofad1864mass.
• Nicholas Murray Butler, Across the Busy Years, II (1940), 390 to 393, archive.org acrossbusyyearsr02butl; John Morley, Recollections, II (1917), archive.org recollections0002morl.
• Kendall Banning, "The Case of Lydia Bixby", The Bookman, February 1922, archive.org caseoflydiabixby00bann.
• The Lincoln Log (thelincolnlog.org) for 28 October to 30 November 1864.

Secondary scholarship read in full

• F. Lauriston Bullard, Abraham Lincoln and the Widow Bixby (Rutgers, 1946), archive.org abrahamlincolnwi00bull.
• Sherman Day Wakefield, Abraham Lincoln and the Widow Bixby (1947) and Abraham Lincoln and the Bixby Letter (1948), archive.org abrahamlincolnwi00wake, abrahamlincolnbi00wake.
• Joe Nickell, "Lincoln's Bixby Letter: A Study in Authenticity", Lincoln Herald 91:4 (1989), 135 to 140, with Jean G. Pival's report; Jason Emerson, "New Evidence from an Ignored Voice: Robert Todd Lincoln and the Authorship of the Bixby Letter", Lincoln Herald 110:2 (2008), 86 to 116. Both from the Lincoln Memorial University digital collection (cdm15995.contentdm.oclc.org, collection myfirst, items 1946 and 3761), which serves page images without a text layer; we ran optical character recognition on the issues and read the articles from the output.
• Michael Burlingame, "New Light on the Bixby Letter", Journal of the Abraham Lincoln Association 16:1 (1995), 59 to 71; "The Trouble With the Bixby Letter", American Heritage 50:4 (1999); Abraham Lincoln: A Life, unedited manuscript, vol. 2, ch. 35 (Knox College Lincoln Studies Center). Burlingame's chapter "The Authorship of the Bixby Letter" in At Lincoln's Side (2000), 169 to 184, is lending-only on archive.org and is cited here through Emerson 2008 and the 1999 article; Hay's letter to Charles S. Spencer of 25 November 1864 is quoted from Burlingame's own quotations of it.
• Jason Emerson, "America's Most Famous Letter", American Heritage 57:1 (2006).
• Jack Grieve, Isobelle Clarke, Emily Chiang, Hannah Gideon, Annina Heini, Andrea Nini and Emily Waibel, "Attributing the Bixby Letter using n-gram tracing", Digital Scholarship in the Humanities 34:3 (2019), 493 to 512, doi 10.1093/llc/fqy042, read in the authors' accepted manuscript (Aston University repository, eprint 37182). The same group's first presentation of the work, "Attributing the Bixby Letter: A Case of Historical Disputed Authorship", LSS Seminar Series, Aston University, 13 April 2016 (slides linked from Jack Grieve's presentations page; a copy is saved with this report), and the Corpus Linguistics 2017 abstract and slides (Birmingham, 26 July 2017).
• "Independent rerun of the Bixby letter tests": the second implementation of n-gram tracing, written from the 2019 paper without sight of this report's code, with pools rebuilt from this report's corpus files under the paper's design, this report's 120 register pieces tested as given, and a second register set of 97 pieces from 38 Lincoln documents chosen independently. Report: https://historical-mysteries.netlify.app/bixby-letter/rerun/. Its code, data and per-piece tables are saved with this report in the folder independent-rerun/.
• Thomas Proisl, Stefan Evert, Fotis Jannidis, Christof Schöch, Leonard Konle and Steffen Pielström, "Delta vs. N-Gram Tracing: Evaluating the Robustness of Authorship Attribution Methods", Proceedings of LREC 2018 (aclanthology.org/L18-1523).

Code and data

Everything below is saved with this report. The corpus builder (corpus/scripts/parse_basler.py) reads Basler's source notes to tag every item with its manuscript code (ALS, ADfS, LS, Copy and so on) and to flag items whose surviving text is in Hay's hand. The analysis scripts (analysis/ngt.py, run_ngt.py, designs.py, run_other.py, features.py) reproduce every number in this report from the JSONL corpora; analysis/results/ holds the raw outputs; analysis/diag_gap.py and analysis/results/diag/ hold the runs that trace the gap between the two implementations' register results. The dossier folder holds the reading notes on the documentary record with page citations for every quotation used above.

The documentary record was read in full from the editions named above; the stylometry was re-implemented from the 2019 paper and validated on known texts before the letter was tested. Every number in sections 3 to 5 can be regenerated from the saved corpora and scripts.