{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of one row of the glossary table. The left cell holds the set-subtraction sign. The right cell reads: Set subtraction. A set-minus B denotes the set that contains all the elements in A except any that are also in B. Formally, A set-minus B is the set of s in A such that s is not in B. A set-minus B is called the relative complement of B in A, since A set-minus B equals A union B superscript C. A red outline surrounds the union sign in the last formula.", "caption": "Glossary of Mathematical Notation, row for set subtraction, p. 283 (PDF p. 301). The outline marks the union sign in $A\\backslash B = A \\cup B^C$, which should be an intersection sign.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T101", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T101"}], "headline": "The set difference is the intersection with the complement, $A\\backslash B = A \\cap B^C$; the glossary prints a union.", "id": "P001-BACK-E001", "location": "Glossary of Mathematical Notation, row for set subtraction, last sentence, p. 283 (PDF p. 301)", "pdf_page": 301, "printed_page": "283", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T101"], "text": {"as_printed": "\\emph{Set subtraction.} $A\\backslash B$ (read ``$A$ set-minus $B$'') denotes the set that contains all the elements in $A$ \\emph{except} any that are also in $B$. Formally, $A\\backslash B = \\{s \\in A : s \\notin B\\}$. $A\\backslash B$ is called the \\emph{relative complement} of $B$ in $A$, since $A\\backslash B = A \\hl{\\cup} B^C$.", "corrected": "\\emph{Set subtraction.} $A\\backslash B$ (read ``$A$ set-minus $B$'') denotes the set that contains all the elements in $A$ \\emph{except} any that are also in $B$. Formally, $A\\backslash B = \\{s \\in A : s \\notin B\\}$. $A\\backslash B$ is called the \\emph{relative complement} of $B$ in $A$, since $A\\backslash B = A \\hl{\\cap} B^C$.\n\nHere $A$ and $B$ are subsets of the universal set $U$, and $B^C = \\{s \\in U : s \\notin B\\}$, as in the glossary's entry for the complement.", "evidence": "I derive the corrected identity and give a counterexample to the printed one. Let $A$ and $B$ be subsets of the universal set $U$, and let $s \\in U$. By the definitions of set subtraction and of the complement in the glossary, I have\n\\[\ns \\in A\\backslash B \\iff (s \\in A \\text{ and } s \\notin B) \\iff (s \\in A \\text{ and } s \\in B^C) \\iff s \\in A \\cap B^C ,\n\\]\nwhich implies that $A\\backslash B = A \\cap B^C$. For the printed form, take $U = \\{1,2,3\\}$, $A = \\{1,2\\}$ and $B = \\{2,3\\}$. Then $A\\backslash B = \\{1\\}$ and $B^C = \\{1\\}$, so $A \\cup B^C = \\{1,2\\} \\neq \\{1\\}$, while $A \\cap B^C = \\{1\\}$. Among the $64$ pairs of subsets of $\\{1,2,3\\}$, the printed identity fails for $56$ and the corrected identity holds for all $64$.", "problem": "The last identity is false. The union $A \\cup B^C$ contains every element of $A$, including those that are in $B$, and every element outside $B$, including those that are outside $A$. It contradicts the set-builder definition given one sentence earlier in the same row, which is correct.", "what_survives": "The verbal and set-builder definitions in the row, the glossary entries for the complement, the union and the intersection, and the proof of the Addition Rule on p.~10, which uses $A \\cap B^C$, are correct. Nothing in the book depends on the misprinted identity."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of two reference entries from page 288. The second reads: Cameron, A. Colin and Pravin K. Trivedi. 2005. Microeconomics: Methods and applications. New York, NY: Cambridge University Press. A red outline surrounds the word Microeconomics.", "caption": "References, p. 288 (PDF p. 306). The outline marks “Microeconomics” in the entry for Cameron and Trivedi (2005); the book is Microeconometrics.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-E002.png"}, {"alt": "Crop of five lines from the references on page 291: the last line of one entry, the entry for Newey, and the entry for Pearl (2000). The entry for Newey reads: Newey, Whitney K. 1990. Semiparametric efficiency bounds. Journal of Applied Economics. 5(2): 99 to 135. A red outline surrounds the word Economics.", "caption": "References, p. 291 (PDF p. 309). The outline marks “Economics” in the journal name of the entry for Newey (1990); the journal is the Journal of Applied Econometrics.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-E002-b.png"}, {"alt": "Crop of five lines from the references on page 292: the entry for Van der Laan and Rose (2011), the entry for VanderWeele, and the first line of the entry for Varian (2014). The entry for VanderWeele reads: VanderWeele, Tyler J. 2015. Causal inference: Methods for mediation and interaction. New York, NY: Oxford University Press. A red outline surrounds the words Causal inference.", "caption": "References, p. 292 (PDF p. 310). The outline marks “Causal inference” at the start of the title in the entry for VanderWeele (2015); the title begins Explanation in Causal Inference.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-E002-c.png"}], "formal_release": null, "formal_statements": [], "headline": "Three reference entries misprint a book title or a journal name: the correct forms are Microeconometrics, Journal of Applied Econometrics and Explanation in Causal Inference.", "id": "P001-BACK-E002", "location": "References, entries for Cameron and Trivedi (2005), p. 288 (PDF p. 306), Newey (1990), p. 291 (PDF p. 309), and VanderWeele (2015), p. 292 (PDF p. 310)", "pdf_page": 306, "printed_page": "288", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Cameron, A. Colin and Pravin K. Trivedi. 2005. \\emph{\\hl{Microeconomics}: Methods and applications}. New York, NY: Cambridge University Press.\n\nNewey, Whitney K. 1990. Semiparametric efficiency bounds. \\emph{Journal of Applied \\hl{Economics}}. 5(2): 99--135.\n\nVanderWeele, Tyler J. 2015. \\emph{\\hl{Causal inference}: Methods for mediation and interaction}. New York, NY: Oxford University Press.", "corrected": "Cameron, A. Colin and Pravin K. Trivedi. 2005. \\emph{\\hl{Microeconometrics}: Methods and applications}. New York, NY: Cambridge University Press.\n\nNewey, Whitney K. 1990. Semiparametric efficiency bounds. \\emph{Journal of Applied \\hl{Econometrics}}. 5(2): 99--135.\n\nVanderWeele, Tyler J. 2015. \\emph{\\hl{Explanation in causal inference}: Methods for mediation and interaction}. New York, NY: Oxford University Press.", "evidence": "I checked the three entries on the page against the publishers' own listings: the Cambridge University Press page for the book by Cameron and Trivedi, the journal's page for the article by Newey, and the title page of the book by VanderWeele.", "problem": "Two book titles and one journal name are misprinted. The book by Cameron and Trivedi is \\emph{Microeconometrics: Methods and Applications}, the article by Newey appeared in the \\emph{Journal of Applied Econometrics}, and the book by VanderWeele is \\emph{Explanation in Causal Inference: Methods for Mediation and Interaction}. Anyone who follows the entries as printed searches for the wrong title or in the wrong journal; the authors and years still identify each work.", "what_survives": "The passages of the book that cite these works are unaffected, and no mathematical or empirical claim depends on the three entries."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the top of the Glossary of Common Abbreviations. A red outline surrounds the sentence Commonly abbreviated terms are defined when first used in the text. The next sentence reads We provide the unabbreviated terms here for reference. Below is a table with the headings Abbreviation, and Term and Reference in Text. Its first five rows are ATE, Average Treatment Effect, Section 7.1.1; BLP, Best Linear Predictor, Theorem 2.2.21; CDF, Cumulative Distribution Function, Definition 1.2.11; CEF, Conditional Expectation Function, Definition 2.2.15; and CMT, Continuous Mapping Theorem, Theorem 3.2.7.", "caption": "Glossary of Common Abbreviations, p. 286 (PDF p. 304). The outline marks the sentence saying that abbreviations are defined at first use. In the first rows of the table there is no row for APD before ATE, for ATC or ATT after it, or for CIA after CEF.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "The Glossary of Common Abbreviations omits CIA, APD, PDA, ATT and ATC, and CIA is not expanded anywhere in the text.", "id": "P001-BACK-N002", "location": "Glossary of Common Abbreviations, sentence above the table, and the table, pp. 286–287 (PDF pp. 304–305)", "pdf_page": 304, "printed_page": "286–287", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Glossary of Common Abbreviations (p.~286).} Commonly abbreviated terms are \\hl{defined when first used in the text}. We provide the unabbreviated terms here for reference.\n\n\\emph{Abbreviations in the table (pp.~286--287).} ATE, BLP, CDF, CEF, CMT, DR, i.i.d., IPW, KL, MAR, MCAR, ML, MSE, NRCM, OLS, PDF, PMF, WLLN.\n\n\\emph{Introduction (p.~xviii).} In addition, we provide a Glossary of Common Abbreviations, which gives the meanings of \\hl{all common abbreviations used in this book}, along with references to where they are defined in the text.\n\n\\emph{Section 7.3.1 (p.~273), first use.} The ATT is nonparametrically identified under the \\hl{CIA} and a weaker positivity assumption, $p_D(\\mathbf{X}_i) < \\epsilon < 1$.\n\n\\emph{Section 7.3.1 (p.~273), second use.} Thus, if the usual plug-in regularity conditions hold, the treatment propensity function is known (or can be estimated consistently), and the \\hl{CIA} holds, then regression will consistently estimate the average treatment effect for a reweighted population.", "corrected": "At its first use on p.~273, write ``\\hl{conditional independence assumption (CIA)}''. Add five rows to the table:\n\\begin{audittable}{ll}{Rows to add to the Glossary of Common Abbreviations.}\nAbbreviation & Term and Reference in Text \\\\\nAPD & Average Partial Derivative; Definition 4.3.6 \\\\\nATC & Average Treatment Effect among the Control Subjects; Section 7.1.1 \\\\\nATT & Average Treatment Effect among the Treated; Section 7.1.1 \\\\\nCIA & Conditional Independence Assumption; Definition 7.1.12 \\\\\nPDA & Partial Derivative at the Average; Definition 4.3.7 \\\\\n\\end{audittable}", "evidence": "I checked the whole table on pp.~286--287, the definitions on pp.~166--167 and 238, and the two uses of CIA on p.~273. I also searched the full text, finding CIA on p.~273 only.", "problem": "Five abbreviations that the text uses are missing from the table: APD and PDA (Definitions 4.3.6 and 4.3.7, pp.~166--167), ATT and ATC (expanded on p.~238), and CIA. Since CIA is used twice on p.~273 and is expanded nowhere, the statement that abbreviations are defined at first use is untrue for it. Its meaning, the conditional independence assumption of Definition 7.1.12, can be inferred from the discussion around it (see \\fid{P001-C07-N009}).", "what_survives": "The eighteen entries that the table lists and the statistical exposition are unaffected, and no result depends on the glossary."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Seven lines of the References. A red box marks 'In press.' in the entry 'Aronow, Peter M. and Cyrus Samii. In press. Estimating average causal effects under general interference'. The next entry begins 'Athey, Susan and Guido W. Imbens. Working paper.'", "caption": "References, p. 288 (PDF p. 306): the Aronow and Samii entry, with “In press.” marked. The Athey and Imbens entry follows it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N003.png"}, {"alt": "Six lines of the References. A red box marks 'In press.' in the entry 'Imbens, Guido W. and Michel Kolesar. In press. Robust standard errors in small samples'. The next entry reads 'Imbens, Guido W. and Donald B. Rubin. 2011. Causal inference in statistics and social sciences. Unpublished manuscript. Harvard University.'", "caption": "References, p. 290 (PDF p. 308): the Imbens and Kolesar entry, with “In press.” marked. The Imbens and Rubin entry follows it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N003-p290.png"}], "formal_release": null, "formal_statements": [], "headline": "Two entries marked “In press” are for articles that appeared in 2017 and 2016, before the book; two further entries cite a working paper and a 2011 manuscript of works that were published in 2017 and 2015.", "id": "P001-BACK-N003", "location": "References, p. 288 (PDF p. 306), entries for Aronow and Samii and for Athey and Imbens; p. 290 (PDF p. 308), entries for Imbens and Kolesar and for Imbens and Rubin", "pdf_page": 306, "printed_page": "288", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~288.} Aronow, Peter M. and Cyrus Samii. \\hl{In press.} Estimating average causal effects under general interference, with application to a social network experiment. \\emph{Annals of Applied Statistics}.\n\nAthey, Susan and Guido W. Imbens. Working paper. The state of applied econometrics: Causality and policy evaluation.\n\n\\emph{p.~290.} Imbens, Guido W. and Michel Kolesar. \\hl{In press.} Robust standard errors in small samples: Some practical advice. \\emph{The Review of Economics and Statistics}.\n\nImbens, Guido W. and Donald B. Rubin. 2011. \\emph{Causal inference in statistics and social sciences}. Unpublished manuscript. Harvard University.", "corrected": "\\emph{p.~288.} P. M. Aronow and Cyrus Samii. \\hl{2017.} Estimating average causal effects under general interference, with application to a social network experiment. \\emph{Annals of Applied Statistics}. \\hl{11(4): 1912--1947.}\n\n\\emph{p.~290.} Imbens, Guido W. and Michel Kolesar. \\hl{2016.} Robust standard errors in small samples: Some practical advice. \\emph{The Review of Economics and Statistics}. \\hl{98(4): 701--712.}\n\nThe citations on pp.~281 and~177 become Aronow and Samii (2017) and Imbens and Kolesar (2016). For the other two entries, the published versions are Athey and Imbens (2017), \\emph{Journal of Economic Perspectives} 31(2): 3--32, and Imbens and Rubin (2015), \\emph{Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction}, Cambridge University Press. If the earlier versions are the ones meant, the entries should say so explicitly.", "evidence": "I checked the pages against the published versions. The first page of the article in the \\emph{Annals of Applied Statistics} is headed ``2017, Vol.~11, No.~4, 1912--1947'' (DOI 10.1214/16-AOAS1005), and the first page of the article in \\emph{The Review of Economics and Statistics} carries ``October 2016, 98(4): 701--712''. The publisher's listing for the article by Athey and Imbens places it in vol.~31, no.~2 (Spring 2017) of the \\emph{Journal of Economic Perspectives}, and the listing of Cambridge University Press for the book by Imbens and Rubin gives 6 April 2015 as the publication date of the hardback.", "problem": "Both articles marked ``In press'' had appeared when the book was published in 2019, so the two labels were out of date, and the two entries lack the year, volume and pages. The citations of these articles on pp.~281 and~177 carry the same label. The other two entries cite earlier versions of works that had also been published by then: the article in 2017, and the book in 2015 under a different title. A citation may deliberately identify an earlier version. The later publication does not by itself imply that these two entries are wrong.", "what_survives": "Each of the four works can be identified from its authors and title. What the book says about these works does not depend on the publication details, and no result is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of the References. A red box marks '(28)2, 2014:' in the last line of the entry, 'Economic Perspectives. (28)2, 2014: 29-50.'", "caption": "References, p. 288 (PDF p. 306): the entry for Belloni, Chernozhukov and Hansen, with “(28)2, 2014:” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "In the entry for Belloni, Chernozhukov and Hansen (2014), “(28)2, 2014:” should read “28(2):”.", "id": "P001-BACK-N005", "location": "References, p. 288 (PDF p. 306), entry for Belloni, Chernozhukov and Hansen (2014)", "pdf_page": 306, "printed_page": "288", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Belloni, Alexandre, Victor Chernozhukov, and Christian Hansen. 2014. High-dimensional methods and inference on structural and treatment effects. \\emph{Journal of Economic Perspectives}. \\hl{(28)2, 2014:} 29--50.", "corrected": "Belloni, Alexandre, Victor Chernozhukov, and Christian Hansen. 2014. High-dimensional methods and inference on structural and treatment effects. \\emph{Journal of Economic Perspectives}. \\hl{28(2):} 29--50.", "evidence": "I checked pp.~288 and 292.", "problem": "The parentheses are set around the volume and not around the issue, and the year is printed a second time. Elsewhere, where an issue is given, the References set an article as volume(issue): pages. In particular, the Varian (2014) entry on p.~292, which is for an article in the same issue of the same journal, reads ``\\emph{Journal of Economic Perspectives}. 28(2): 3--28.''", "what_survives": "The volume, issue, year and pages are not in doubt, and the article can be identified."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of the References. A red box marks 'in estimation' in the title 'Moving the goalposts: Addressing limited overlap in estimation of average treatment effects by changing the estimand'.", "caption": "References, p. 289 (PDF p. 307): the entry for Crump, Hotz, Imbens and Mitnik, with “in estimation” marked, where the title has “in the estimation”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N006.png"}, {"alt": "Five lines of the References. A red box marks '972' in 'Biometrics. 61: 962-972.' at the end of the entry for Bang and Robins.", "caption": "References, p. 288 (PDF p. 306): the entry for Bang and Robins, with the last page, “972”, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N006-p288.png"}], "formal_release": null, "formal_statements": [], "headline": "The title of Crump et al. (2006) reads “in the estimation of”, and the article by Bang and Robins (2005) ends on p. 973, not p. 972; the year 2001 printed for Hayashi’s Econometrics may be that of a later printing of a book first published in 2000.", "id": "P001-BACK-N006", "location": "References, p. 289 (PDF p. 307), entry for Crump, Hotz, Imbens and Mitnik (2006); p. 288 (PDF p. 306), entry for Bang and Robins (2005); p. 290 (PDF p. 308), entry for Hayashi (2001)", "pdf_page": 307, "printed_page": "289", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~289.} Crump, Richard K., V. Joseph Hotz, Guido W. Imbens, and Oscar Mitnik. 2006. Moving the goalposts: Addressing limited overlap \\hl{in estimation} of average treatment effects by changing the estimand. NBER Technical Working Paper No. 330.\n\n\\emph{p.~288.} Bang, Heejung and James M. Robins. 2005. Doubly robust estimation in missing data and causal inference models. \\emph{Biometrics}. 61: 962--\\hl{972}.\n\n\\emph{p.~290.} Hayashi, Fumio. 2001. \\emph{Econometrics}. Princeton, NJ: Princeton University Press.", "corrected": "\\emph{p.~289.} Crump, Richard K., V. Joseph Hotz, Guido W. Imbens, and Oscar Mitnik. 2006. Moving the goalposts: Addressing limited overlap in \\hl{the} estimation of average treatment effects by changing the estimand. NBER Technical Working Paper No. 330.\n\n\\emph{p.~288.} Bang, Heejung and James M. Robins. 2005. Doubly robust estimation in missing data and causal inference models. \\emph{Biometrics}. 61: 962--\\hl{973}.\n\nIf the first edition of Hayashi's book is the one meant, then the year is 2000 both in the entry on p.~290 and in the citation on p.~204. If a printing of 2001 is meant, the entry should identify it.", "evidence": "I checked the pages against the publishers' listings. The NBER listing for Technical Working Paper 330 (\\url{https://www.nber.org/papers/t0330}) is headed ``Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand'', and the publisher's listing for the article in \\emph{Biometrics} reads ``Pages 962-973''. The economics catalogue of Princeton University Press for 2017 lists Hayashi's book as ``2000. 712 pages'', and the copyright page of the book carries the year 2000.", "problem": "The title of the working paper has ``in the estimation of'', and the entry omits ``the''. The article in \\emph{Biometrics} occupies pp.~962--973, and the entry stops one page short. The first edition of Hayashi's book is dated 2000. The entry and the citation ``Hayashi (2001)'' on p.~204 give 2001. Since a printing dated 2001 is not excluded, the evidence does not show that the year is an error.", "what_survives": "The three works can be identified from the entries as printed. No statement in the text and no result depends on these details."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of the References. A red box marks the surname 'Dasgupta' at the start of the entry 'Dasgupta, Anirban. 2008. Asymptotic theory of statistics and probability.'", "caption": "References, p. 289 (PDF p. 307): the entry for the book by DasGupta, with the surname “Dasgupta” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The surname in the entry should be “DasGupta”, as it is in the two citations in the text.", "id": "P001-BACK-N007", "location": "References, p. 289 (PDF p. 307), entry “Dasgupta, Anirban. 2008.”", "pdf_page": 307, "printed_page": "289", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\hl{Dasgupta}, Anirban. 2008. \\emph{Asymptotic theory of statistics and probability}. New York, NY: Springer-Verlag New York.", "corrected": "\\hl{DasGupta}, Anirban. 2008. \\emph{Asymptotic theory of statistics and probability}. New York, NY: Springer-Verlag New York.", "evidence": "I checked the pages and the author field of the publisher's listing for the book (\\url{https://link.springer.com/book/10.1007/978-0-387-75971-5}).", "problem": "The text cites the book as DasGupta (2008), with a capital G, in footnote~9 on p.~56 and on p.~141. The entry has a lower-case g. The publisher gives the author's name as Anirban DasGupta, and I conclude that the entry is the one that is misspelled.", "what_survives": "The book that is cited is not in doubt, and nothing else depends on the spelling."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of the References. A red box marks 'Vol. 5' at the start of the second line of the entry 'Davidson, Russell and James G. MacKinnon. 2004. Econometric theory and methods, Vol. 5. New York, NY: Oxford University Press.'", "caption": "References, p. 289 (PDF p. 307): the entry for Davidson and MacKinnon, with “Vol. 5” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "“Vol. 5” should be deleted from the entry for Davidson and MacKinnon (2004): the book is a single volume.", "id": "P001-BACK-N008", "location": "References, p. 289 (PDF p. 307), entry for Davidson and MacKinnon (2004)", "pdf_page": 307, "printed_page": "289", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Davidson, Russell and James G. MacKinnon. 2004. \\emph{Econometric theory and methods}\\hl{, Vol. 5}. New York, NY: Oxford University Press.", "corrected": "Davidson, Russell and James G. MacKinnon. 2004. \\emph{Econometric theory and methods}\\del{, Vol. 5}. New York, NY: Oxford University Press.", "evidence": "I checked the page against the description of the book in a library catalogue (Oxford University Press, 2004, xviii + 750 pages, ISBN 0195123727), and against the indexed front matter of the copy that the authors have posted, which carries the same ISBN and no volume number.", "problem": "The book is a single volume of xviii + 750 pages and is not the fifth volume of a larger work, so no volume number belongs in the entry. Where the ``5'' came from is not known.", "what_survives": "The citation on p.~177 gives authors and year only and is unaffected. The book can be identified from the rest of the entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of the References. A red box marks 'Sarah' in the entry 'Diaconis, Persi, Sarah Holmes, and Richard Montgomery. 2007. Dynamical bias in the coin toss.'", "caption": "References, p. 289 (PDF p. 307): the entry for Diaconis, Holmes and Montgomery, with the given name “Sarah” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "The second author of “Dynamical bias in the coin toss” is Susan Holmes, not Sarah Holmes.", "id": "P001-BACK-N009", "location": "References, p. 289 (PDF p. 307), entry for Diaconis, Holmes and Montgomery (2007)", "pdf_page": 307, "printed_page": "289", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Diaconis, Persi, \\hl{Sarah} Holmes, and Richard Montgomery. 2007. Dynamical bias in the coin toss. \\emph{Society for Industrial and Applied Mathematics Review}. 49(2): 211--235.", "corrected": "Diaconis, Persi, \\hl{Susan} Holmes, and Richard Montgomery. 2007. Dynamical bias in the coin toss. \\emph{Society for Industrial and Applied Mathematics Review}. 49(2): 211--235.", "evidence": "I checked the page against the journal's listing for the article (DOI 10.1137/S0036144504446436), whose list of authors reads ``Persi Diaconis, Susan Holmes, and Richard Montgomery''.", "problem": "The given name of the second author is wrong in the entry. The citation in footnote~3 on p.~5 gives surnames only and is unaffected.", "what_survives": "The article can be identified from its title, journal, volume and pages, and nothing else is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of the References. A red box marks the second '2016.' in the entry 'Greenland, Sander et al. 2016. Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. 2016. European Journal of Epidemiology.'", "caption": "References, p. 289 (PDF p. 307): the entry for Greenland et al., with the second “2016.” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N010.png"}, {"alt": "Four lines of the References. A red box marks 'applicationsin' in the title 'An introduction to statistical learning with applicationsin R.'", "caption": "References, p. 290 (PDF p. 308): the entry for James et al., with “applicationsin” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N010-p290.png"}, {"alt": "Five lines of the References. A red box marks 'Model- based', printed with a space after the hyphen, in the entry 'Rosenbaum, Paul R. 1987. Model- based direct adjustment.'", "caption": "References, p. 291 (PDF p. 309): the entry for Rosenbaum (1987), with “Model- based” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N010-p291.png"}], "formal_release": null, "formal_statements": [], "headline": "Three slips of typesetting: “2016.” is printed twice in the entry for Greenland et al., “applicationsin” should be “applications in” in the entry for James et al., and “Model- based” should be “Model-based” in the entry for Rosenbaum (1987).", "id": "P001-BACK-N010", "location": "References, p. 289 (PDF p. 307), entry for Greenland et al. (2016); p. 290 (PDF p. 308), entry for James et al. (2014); p. 291 (PDF p. 309), entry for Rosenbaum (1987)", "pdf_page": 307, "printed_page": "289", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~289.} Greenland, Sander et al. 2016. Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. \\hl{2016.} \\emph{European Journal of Epidemiology}. 31(4): 337--350.\n\n\\emph{p.~290.} James, Gareth et al. 2014. \\emph{An introduction to statistical learning with \\hl{applicationsin} R}. New York, NY: Springer Science+Business Media New York.\n\n\\emph{p.~291.} Rosenbaum, Paul R. 1987. \\hl{Model-~based} direct adjustment. \\emph{Journal of the American Statistical Association}. 82(398): 387--394.", "corrected": "\\emph{p.~289.} Greenland, Sander et al. 2016. Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. \\del{2016.} \\emph{European Journal of Epidemiology}. 31(4): 337--350.\n\n\\emph{p.~290.} James, Gareth et al. 2014. \\emph{An introduction to statistical learning with \\hl{applications in} R}. New York, NY: Springer Science+Business Media New York.\n\n\\emph{p.~291.} Rosenbaum, Paul R. 1987. \\hl{Model-based} direct adjustment. \\emph{Journal of the American Statistical Association}. 82(398): 387--394.", "evidence": "I checked the three pages.", "problem": "The year 2016 is printed after the authors and again after the title. A space is missing between ``applications'' and ``in''. A space follows the hyphen of ``Model-based'', in the middle of a line. The References also print Hernan and Kolesar without accents while they keep the accents of Hájek and Pötscher; I have not checked the accented spellings against a source, and I do not correct them here.", "what_survives": "The three works can be identified, and in each case the intended text is plain."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N010", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of the References. A red box marks '968' in 'Statistical Association. 81(396): 945-968.' at the end of the entry for Holland.", "caption": "References, p. 290 (PDF p. 308): the entry for Holland (1986), with the last page of the range, “968”, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N011.png"}], "formal_release": null, "formal_statements": [], "headline": "The article by Holland (1986) occupies pp. 945–960; the printed range, 945–968, also takes in the comments and the first page of Holland’s rejoinder.", "id": "P001-BACK-N011", "location": "References, p. 290 (PDF p. 308), entry for Holland (1986)", "pdf_page": 308, "printed_page": "290", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Holland, Paul W. 1986. Statistics and causal inference. \\emph{Journal of the American Statistical Association}. 81(396): 945--\\hl{968}.", "corrected": "Holland, Paul W. 1986. Statistics and causal inference. \\emph{Journal of the American Statistical Association}. 81(396): 945--\\hl{960}.\n\nIf the comments or the rejoinder are also meant, they should be cited separately, with their own pages.", "evidence": "I checked the page against an indexed version of the publisher's table of contents for the issue (\\url{https://www.tandfonline.com/toc/uasa20/81/396}), which has ``Pages: 945-960'' for the article and ``Pages: 967-968'' for the comment by Granger, and against the cover page of a scan of the rejoinder, which gives 968--970.", "problem": "The article named in the entry occupies pp.~945--960. The published comments follow on pp.~961--968, and Holland's rejoinder is on pp.~968--970, so the printed range reaches its first page. The printed range therefore fits neither the article alone nor the article together with the comments and the rejoinder. Whether the comments were meant to be included is not known.", "what_survives": "The citation on p.~236 gives author and year only. The article can be identified from its title, volume and first page, and nothing in the text depends on the range."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N011", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of the References. A red box marks '684' in 'Association. 47: 663-684.' at the end of the entry for Horvitz and Thompson.", "caption": "References, p. 290 (PDF p. 308): the entry for Horvitz and Thompson, with the last page, “684”, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N012.png"}], "formal_release": null, "formal_statements": [], "headline": "The article by Horvitz and Thompson (1952) ends on p. 685, not p. 684.", "id": "P001-BACK-N012", "location": "References, p. 290 (PDF p. 308), entry for Horvitz and Thompson (1952)", "pdf_page": 308, "printed_page": "290", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Horvitz, Daniel G. and Donovan J. Thompson. 1952. A generalization of sampling without replacement from a finite universe. \\emph{Journal of the American Statistical Association}. 47: 663--\\hl{684}.", "corrected": "Horvitz, Daniel G. and Donovan J. Thompson. 1952. A generalization of sampling without replacement from a finite universe. \\emph{Journal of the American Statistical Association}. 47: 663--\\hl{685}.", "evidence": "I checked the page against the cover page of a scan of the published article, which gives the source as Vol.~47, No.~260 (Dec., 1952), pp.~663--685.", "problem": "The last page of the article is 685, and the entry stops one page short.", "what_survives": "The citation on p.~227 gives authors and year only. The article can be identified, and what the book says about the estimator is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N012", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of the References. A red box marks 'population means' in the title 'Demystifying double robustness: A comparison of alternative strategies for estimating population means from incomplete data'.", "caption": "References, p. 290 (PDF p. 308): the entry for Kang and Schafer, with “population means” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N013.png"}], "formal_release": null, "formal_statements": [], "headline": "The title of Kang and Schafer (2007) has “estimating a population mean”, not “estimating population means”.", "id": "P001-BACK-N013", "location": "References, p. 290 (PDF p. 308), entry for Kang and Schafer (2007)", "pdf_page": 308, "printed_page": "290", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Kang, Joseph D. Y. and Joseph L. Schafer. 2007. Demystifying double robustness: A comparison of alternative strategies for estimating \\hl{population means} from incomplete data. \\emph{Statistical Science}. 22(4): 523--539.", "corrected": "Kang, Joseph D. Y. and Joseph L. Schafer. 2007. Demystifying double robustness: A comparison of alternative strategies for estimating \\hl{a population mean} from incomplete data. \\emph{Statistical Science}. 22(4): 523--539.", "evidence": "I checked the entry against the first page of the electronic reprint of the published article (\\url{https://arxiv.org/pdf/0804.2958}), whose title reads ``Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data''.", "problem": "The title of the article has the singular with the indefinite article, ``a population mean'', where the entry has the plural.", "what_survives": "The article can be identified. The citation on p.~234 does not quote the title, and no result depends on it."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N013", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of the References. A red box marks '4: 117' in 'Psychological Methods. 4: 117.' at the end of the entry for Reichardt and Gollob.", "caption": "References, p. 291 (PDF p. 309): the entry for Reichardt and Gollob, with “4: 117” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N014.png"}], "formal_release": null, "formal_statements": [], "headline": "The entry for Reichardt and Gollob (1999) gives the first page only and no issue; the article is in 4(1): 117–128.", "id": "P001-BACK-N014", "location": "References, p. 291 (PDF p. 309), entry for Reichardt and Gollob (1999)", "pdf_page": 309, "printed_page": "291", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Reichardt, Charles S. and Harry F. Gollob. 1999. Justifying the use and increasing the power of a \\emph{t} test for a randomized experiment with a convenience sample. \\emph{Psychological Methods}. \\hl{4: 117}.", "corrected": "Reichardt, Charles S. and Harry F. Gollob. 1999. Justifying the use and increasing the power of a \\emph{t} test for a randomized experiment with a convenience sample. \\emph{Psychological Methods}. \\hl{4(1): 117--128}.", "evidence": "I checked the entry against the bibliographic details in the publisher's preview of the article, which give 4(1): 117--128 (DOI 10.1037/1082-989X.4.1.117).", "problem": "The entry is incomplete and not false: 117 is the first page of the article, which runs to p.~128 in issue~1 of volume~4. Elsewhere the References give a page range for an article in a journal.", "what_survives": "Nothing that the entry prints is wrong, and the citation on p.~281 is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N014", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Eight lines from the reference list. In the entry for Richardson and Robins, a red box marks 'Working paper.' where other entries give a year, and a second red box marks the word 'the' in 'Center for the Statistics and the Social Sciences'.", "caption": "References, p. 291 (PDF p. 309): the entry for Richardson and Robins, with “Working paper.” in the position of the year and the extra “the” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N015.png"}], "formal_release": null, "formal_statements": [], "headline": "In the entry for Richardson and Robins, the issuing body is the Center for Statistics and the Social Sciences, with no “the” before “Statistics”, and the working paper is dated 2013.", "id": "P001-BACK-N015", "location": "References, p. 291 (PDF p. 309), entry for Richardson and Robins", "pdf_page": 309, "printed_page": "291", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Richardson, Thomas S. and James M. Robins. \\hl{Working paper.} Single-world intervention graphs (SWIGs): A unification of the counterfactual and graphical approaches to causality. Working Paper Number 128. Center for \\hl{the} Statistics and the Social Sciences, University of Washington, Seattle.", "corrected": "Richardson, Thomas S. and James M. Robins. \\hl{2013.} Single-world intervention graphs (SWIGs): A unification of the counterfactual and graphical approaches to causality. Working Paper Number 128. Center \\hl{for Statistics} and the Social Sciences, University of Washington, Seattle.", "evidence": "I checked the entry against the center's list of working papers, which is headed ``Center for Statistics and the Social Sciences'' and has the line ``WP 128 $\\cdot$ April 2013'' (\\url{https://csss.uw.edu/research/working-papers?items_per_page=25&page=1}).", "problem": "The name of the center has no ``the'' before ``Statistics''. The entry also gives ``Working paper'' in the position of the year, although the working paper is dated: the center lists it as Working Paper 128, April 2013. The entry omits the year; it does not give a false one.", "what_survives": "No result or argument depends on the name of the center or on the date, and the working paper remains identifiable by its authors, title and number. The two citations of it in the text, on pp.~247 and 281, give ``mimeo.'' in place of a year and still lead to the entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N015", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Six lines from the reference list: the last two lines of the preceding entry, the entry 'Richardson, Thomas S. and Andrea Rotnitzky. 2014. Causal etiology of the research of James M. Robins. Statistical Science. 29(4): 259-284.', and the entry that follows. A red box marks the page range '259-284'.", "caption": "References, p. 291 (PDF p. 309): the entry for Richardson and Rotnitzky (2014), with the page range “259–284” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N016.png"}], "formal_release": null, "formal_statements": [], "headline": "The article by Richardson and Rotnitzky (2014) is on pp. 459–484 of Statistical Science 29(4), not on pp. 259–284.", "id": "P001-BACK-N016", "location": "References, p. 291 (PDF p. 309), entry for Richardson and Rotnitzky (2014)", "pdf_page": 309, "printed_page": "291", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Richardson, Thomas S. and Andrea Rotnitzky. 2014. Causal etiology of the research of James M. Robins. \\emph{Statistical Science}. 29(4): \\hl{259--284}.", "corrected": "Richardson, Thomas S. and Andrea Rotnitzky. 2014. Causal etiology of the research of James M. Robins. \\emph{Statistical Science}. 29(4): \\hl{459--484}.", "evidence": "I checked the entry against the first page of the article (DOI 10.1214/14-STS505), read in the IMS electronic reprint at \\url{https://arxiv.org/pdf/1503.02894}, which carries the journal header ``2014, Vol. 29, No. 4, 459--484''.", "problem": "The first digit of both page numbers is wrong; the year, the volume and the issue are right.", "what_survives": "The entry still identifies the article by its authors, title, journal, volume and issue. The text cites the article by author and year only (p.~281), and nothing depends on the page numbers."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N016", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Eight lines from the reference list. In the entry 'Robins, James M. 1986. A new approach to causal inference in mortality studies with a sustained exposure period? Application to control of the healthy worker survivor effect', a small red box marks the question mark after 'period'.", "caption": "References, p. 291 (PDF p. 309): the entry for Robins (1986), with the question mark after “period” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N017.png"}], "formal_release": null, "formal_statements": [], "headline": "In the title of Robins (1986), the question mark after “period” should be a dash.", "id": "P001-BACK-N017", "location": "References, p. 291 (PDF p. 309), entry for Robins (1986)", "pdf_page": 309, "printed_page": "291", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Robins, James M. 1986. A new approach to causal inference in mortality studies with a sustained exposure period\\hl{? A}pplication to control of the healthy worker survivor effect. Mathematical models in medicine: Diseases and epidemics. Part 2. \\emph{Mathematical Modelling}. 7: 1393--1512.", "corrected": "Robins, James M. 1986. A new approach to causal inference in mortality studies with a sustained exposure period\\hl{---a}pplication to control of the healthy worker survivor effect. Mathematical models in medicine: Diseases and epidemics. Part 2. \\emph{Mathematical Modelling}. 7: 1393--1512.", "evidence": "I checked the entry against the publisher's listing of the article (\\url{https://www.sciencedirect.com/science/article/pii/0270025586900886}), which gives \\emph{Mathematical Modelling} 7(9--12): 1393--1512 and the DOI 10.1016/0270-0255(86)90088-6. The title there ends ``with a sustained exposure period---application to control of the healthy worker survivor effect''.", "problem": "The title of the article is not a question. In the publisher's listing of the article, a dash joins ``period'' to ``application'', which is in lower case. How the question mark entered the entry is not known.", "what_survives": "Only the punctuation and following capitalization of this title change, and no result or argument in the book depends on them."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N017", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four index lines: 'geometric distribution, 19'; 'Glivenko-Cantelli Theorem, 102n14, 117n27,' continued on the next line by '120n31'; and, after a gap, 'heteroskedasticity, 156'. A red box marks the locator '117n27'.", "caption": "Index, p. 295 (PDF p. 313), left column: the entry “Glivenko–Cantelli Theorem”, with the locator “117n27” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N019.png"}], "formal_release": null, "formal_statements": [], "headline": "The index locator for the Glivenko–Cantelli Theorem on p. 117 should give footnote 29, not footnote 27.", "id": "P001-BACK-N019", "location": "Index, p. 295 (PDF p. 313), left column, entry “Glivenko–Cantelli Theorem”", "pdf_page": 313, "printed_page": "295", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Glivenko--Cantelli Theorem, 102n14, 117n\\hl{27}, 120n31", "corrected": "Glivenko--Cantelli Theorem, 102n14, 117n\\hl{29}, 120n31", "evidence": "I checked the pages. Footnote 29 begins ``In fact, an even stronger claim holds. The Glivenko--Cantelli Theorem:''. Footnote 31 on p.~120 refers to ``the Glivenko--Cantelli Theorem in Footnote 29'', and the index entries for almost sure convergence on pp.~293 and 294 give 117n29.", "problem": "Footnote 29 on p.~117 names and states the theorem, while footnote 27 is on p.~116 and concerns the multivariate empirical CDF.", "what_survives": "The page number is right, and the other two locators lead to footnotes that name the theorem. The theorem is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N019", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Ten index lines: the heading 'independence' and its eight subentries, from 'CEFs and BLPs under, 82-84' to 'of random variables, 38-39'. A red box marks '59-60' in the subentry 'covariance and, 59-60', which stands directly below 'correlation and, 64-66'.", "caption": "Index, p. 295 (PDF p. 313), right column: the heading “independence” and its subentries, with the locator “59–60” of “covariance and” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N022.png"}], "formal_release": null, "formal_statements": [], "headline": "Under “independence”, the subentry “covariance and” should point to p. 64, not to pp. 59–60.", "id": "P001-BACK-N022", "location": "Index, p. 295 (PDF p. 313), right column, heading “independence”, subentry “covariance and”", "pdf_page": 313, "printed_page": "295", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Under ``independence''.} covariance and, \\hl{59--60}", "corrected": "\\emph{Under ``independence''.} covariance and, \\hl{64}", "evidence": "I checked the pages. Section 2.2.2 is titled ``Covariance, Correlation, and Independence'', and Theorem 2.2.8 states that $\\operatorname{Cov}[X,Y]=0$ for independent random variables $X$ and $Y$. The corresponding subentry under ``covariance'' on p.~294, ``and independence, 64'', gives the right page.", "problem": "Pages 59--60 define covariance, derive its alternative formula and state the variance rule; they do not discuss independence. The relation between covariance and independence is Theorem 2.2.8, in Section 2.2.2 on p.~64. The printed locator repeats that of the heading ``covariance, 59--60'' on p.~294.", "what_survives": "The subject of the subentry is right, and only its locator changes. Theorem 2.2.8 is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N022", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Eight index lines under the heading 'maximum likelihood (ML) estimation, 185-203, 233'. Red boxes mark the space and the comma in 'inference with , 202-203' and in 'of the classical linear model , 189-192'. The subentry below, 'of the probit model, 192-193', has no such space.", "caption": "Index, p. 296 (PDF p. 314), left column: the heading “maximum likelihood (ML) estimation”, with the space before the comma marked in two subentries.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N023.png"}, {"alt": "Five index lines: 'probit model, 183'; 'maximum likelihood estimation of ,' continued on the next line by '192-193'; 'propensity score'; and 'for causal inference, 250-252'. A red box marks the space and the comma after 'estimation of'.", "caption": "Index, p. 297 (PDF p. 315), right column: the heading “probit model”, with the space before the comma marked in its subentry.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N023-p297.png"}, {"alt": "Six index lines, all subentries of 'estimator': 'kernel plug-in, 123'; 'lasso, 169-170, 174, 202'; 'MSE of, 104-105'; 'plug in, see plug-in estimator'; 'post-stratification, 219, 256'; and 'regression, see regression'. A red box marks 'plug in', which is set without a hyphen.", "caption": "Index, p. 295 (PDF p. 313), left column: six subentries of the heading “estimator”, which begins on p. 294, with “plug in” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N023-p295.png"}], "formal_release": null, "formal_statements": [], "headline": "Three index subentries have a stray space before the comma, and the subentry “plug in” under “estimator” should be hyphenated.", "id": "P001-BACK-N023", "location": "Index, p. 296 (PDF p. 314), left column, heading “maximum likelihood (ML) estimation”; p. 297 (PDF p. 315), right column, heading “probit model”; p. 295 (PDF p. 313), left column, subentry “plug in” of the heading “estimator”", "pdf_page": 314, "printed_page": "296", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~296, under ``maximum likelihood (ML) estimation''.} inference with\\hl{ ,} 202--203 \\ldots\\ of the classical linear model\\hl{ ,} 189--192\n\n\\emph{p.~297, under ``probit model''.} maximum likelihood estimation of\\hl{ ,} 192--193\n\n\\emph{p.~295, under ``estimator''.} \\hl{plug in}, \\emph{see} plug-in estimator", "corrected": "\\emph{p.~296, under ``maximum likelihood (ML) estimation''.} inference with\\hl{,} 202--203 \\ldots\\ of the classical linear model\\hl{,} 189--192\n\n\\emph{p.~297, under ``probit model''.} maximum likelihood estimation of\\hl{,} 192--193\n\n\\emph{p.~295, under ``estimator''.} \\hl{plug-in}, \\emph{see} plug-in estimator", "evidence": "I checked the pages. The neighboring subentries ``of the probit model, 192--193'' on p.~296 and ``kernel plug-in, 123'' on p.~295 show the usual forms.", "problem": "In three subentries a space stands between the last word and the comma. Under ``estimator'', the subentry ``plug in'' lacks the hyphen of the entry ``plug-in estimator'' to which it refers.", "what_survives": "Only the typography changes, and no locator is affected. The abbreviation ``CLT'' occurs only in the index, where the heading ``Central Limit Theorem (CLT)'' on p.~293 expands it, and it needs no change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N023", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Eight index lines: the heading 'missing data, 208-219' and its six subentries, the last of which runs over two lines. A red box marks '230, 267' in the first subentry, 'Double Robustness Theorem for, 230, 267'.", "caption": "Index, p. 296 (PDF p. 314), right column: the heading “missing data” and its subentries, with the locators “230, 267” of “Double Robustness Theorem for” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N024.png"}], "formal_release": null, "formal_statements": [], "headline": "Under “missing data”, the subentry “Double Robustness Theorem for” should point to pp. 229–230, where the theorem for missing data appears, not to pp. 230 and 267.", "id": "P001-BACK-N024", "location": "Index, p. 296 (PDF p. 314), right column, heading “missing data”, subentry “Double Robustness Theorem for”", "pdf_page": 314, "printed_page": "296", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Under ``missing data, 208--219''.} Double Robustness Theorem for, \\hl{230, 267}", "corrected": "\\emph{Under ``missing data, 208--219''.} Double Robustness Theorem for, \\hl{229--230}", "evidence": "I checked the pages. Under the heading ``Double Robustness Theorem'' on p.~294, the index gives ``for causal inference, 267--269'' and ``for missing data, 229--230''.", "problem": "Since the subentry stands under ``missing data'', it refers to the Double Robustness Theorem for Missing Data. That theorem is Theorem 6.2.9, on pp.~229--230, and the printed locators omit p.~229, where it is stated. Page 267 states Theorem 7.2.8, the Double Robustness Theorem for Causal Inference. The passage that introduces Theorem 7.2.8 compares it with the case of missing data, but the theorem on that page is the causal one.", "what_survives": "Only this locator changes, and both theorems are unaffected. I examined five further cases in which an index entry differs from the entry for the same topic elsewhere in the index, or omits the page of a definition. They are ``regression estimator for, 224'' under ``missing data''; ``joint, 36, 40'' under ``probability density function (PDF)''; ``conditional, 42'' under ``probability mass function (PMF)''; ``kernel density, 122'' under ``estimator''; and the two entries for the conditional independence assumption, which omit p.~247. In each case the printed locator points to a page that treats its subject, and none is changed."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N024", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Eleven index lines: the heading 'plug-in estimator, 120', its eight subentries, and the next heading, 'plug-in principle, 116-124'. A red box marks '118' in the last subentry, 'sample mean as, 118, 210, 240'. Three lines above it is the subentry 'plug-in sample variance as, 119'.", "caption": "Index, p. 297 (PDF p. 315), left column: the heading “plug-in estimator” and its subentries, with the locator “118” of “sample mean as” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N026.png"}, {"alt": "Four index lines: the heading 'sample mean' and the subentries 'as plug-in estimator, 118, 210, 240', 'Chebyshev's Inequality for, 98-99' and 'consistency of, 105'. A red box marks '118'.", "caption": "Index, p. 298 (PDF p. 316), left column: the heading “sample mean”, with the locator “118” of “as plug-in estimator” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-BACK-N026-p298.png"}], "formal_release": null, "formal_statements": [], "headline": "In both index subentries for the sample mean as a plug-in estimator, the first locator should be p. 119, not p. 118.", "id": "P001-BACK-N026", "location": "Index, p. 297 (PDF p. 315), left column, heading “plug-in estimator”, subentry “sample mean as”; repeated on p. 298 (PDF p. 316), left column, heading “sample mean”, subentry “as plug-in estimator”", "pdf_page": 315, "printed_page": "297", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~297, under ``plug-in estimator, 120''.} sample mean as, \\hl{118}, 210, 240\n\n\\emph{p.~298, under ``sample mean''.} as plug-in estimator, \\hl{118}, 210, 240", "corrected": "\\emph{p.~297, under ``plug-in estimator, 120''.} sample mean as, \\hl{119}, 210, 240\n\n\\emph{p.~298, under ``sample mean''.} as plug-in estimator, \\hl{119}, 210, 240", "evidence": "I checked the pages. The subentry for the companion example on the same page, ``plug-in sample variance as, 119'', gives the right page. Pages 210 and 240 each use the sample mean as a plug-in estimator.", "problem": "Example 3.3.4, ``Sample Mean as a Plug-In Estimator'', is on p.~119. Page 118 defines a plug-in estimator in general and announces two examples, but it does not identify the sample mean as one.", "what_survives": "The locators 210 and 240 are right in both subentries. Example 3.3.4 and its later uses are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-BACK-N026", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of two sentences and a display. The second sentence reads 'So, for example, the probability that (X,Y) will fall within the triangular region {0 <= X <= 1, Y <= X} is', and the display reads 'Pr[0 <= X <= 1, Y <= X] = integral from 0 to 1 of the integral from 0 to x of f(x,y) dy dx'. Red boxes mark 'Y <= X' in the sentence and in the display.", "caption": "The illustration after Theorem 1.3.12, p. 37 (PDF p. 55). The two marked conditions $Y\\le X$ put no lower bound on $Y$, while the inner integral starts at $0$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T34", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T34"}, {"id": "P001-T35", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T35"}], "headline": "The event in the illustration after Theorem 1.3.12 omits the bound $0\\le Y$; the triangular region is $\\{0\\le X\\le 1,\\ 0\\le Y\\le X\\}$.", "id": "P001-C01-E001", "location": "p. 37 (PDF p. 55), the illustration after Theorem 1.3.12: the sentence beginning “So, for example,” and the display that follows it", "pdf_page": 55, "printed_page": "37", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T34", "P001-T35"], "text": {"as_printed": "So, for example, the probability that $(X,Y)$ will fall within the triangular region $\\{0\\le X\\le 1,\\hl{Y\\le X}\\}$ is\n\\[\n\\Pr[0\\le X\\le 1,\\hl{Y\\le X}]=\\int_0^1\\int_0^x f(x,y)\\,dy\\,dx.\n\\]", "corrected": "For jointly continuous random variables $X$ and $Y$ with joint PDF $f$, as in Theorem 1.3.12:\n\nSo, for example, the probability that $(X,Y)$ will fall within the triangular region $\\{0\\le X\\le 1,\\hl{0\\le Y}\\le X\\}$ is\n\\[\n\\Pr[0\\le X\\le 1,\\hl{0\\le Y}\\le X]=\\int_0^1\\int_0^x f(x,y)\\,dy\\,dx.\n\\]\nAlternatively, the event can be kept as printed. Then ``triangular region'' becomes ``region'', since the set is unbounded, and the display becomes $\\Pr[0\\le X\\le 1,\\ Y\\le X]=\\int_0^1\\int_{-\\infty}^{x}f(x,y)\\,dy\\,dx$.", "evidence": "The evidence is a derivation and an exact counterexample. Let $A=\\{0\\le X\\le 1,\\ Y\\le X\\}$, $T=\\{0\\le X\\le 1,\\ 0\\le Y\\le X\\}$ and $N=\\{0\\le X\\le 1,\\ Y<0\\}$. The events $T$ and $N$ are disjoint, and their union is $A$. By Theorem 1.3.12, I obtain\n\\[\n\\Pr[A]=\\int_0^1\\int_{-\\infty}^{x}f(x,y)\\,dy\\,dx=\\int_0^1\\int_0^x f(x,y)\\,dy\\,dx+\\Pr[0\\le X\\le 1,\\ Y<0].\n\\]\nThe printed equality therefore holds if and only if $\\Pr[0\\le X\\le 1,\\ Y<0]=0$. For the counterexample, let $X$ and $Y$ be independent, with PDFs $6x(1-x)$ on $(0,1)$ and $-6y(1+y)$ on $(-1,0)$, each zero elsewhere. Then $Y<0<X$ with probability one. The event as printed has probability $1$, and the displayed integral equals $0$. For independent $X$ and $Y$ that are uniform on $(-1,1)$, the two numbers are $3/8$ and $1/8$, and their difference is $\\Pr[0\\le X\\le 1,\\ Y<0]=1/4$.", "problem": "The set as printed contains every point with $0\\le X\\le 1$ and $Y<0$. It is unbounded below and is not a triangle, and the displayed integral, whose inner variable runs from $0$ to $x$, leaves out its part with $Y<0$. The printed equality holds only when $\\Pr[0\\le X\\le 1,\\ Y<0]=0$, which Section 1.3.3 does not assume. The word ``triangular'' and the limits of integration show which event is meant, namely $\\{0\\le X\\le 1,\\ 0\\le Y\\le X\\}$, and for that event the display is correct for every joint PDF.", "what_survives": "Theorem 1.3.12, with the measurability condition of its footnote 29, is unaffected, and so is the formula for the triangle that was meant. I found no later result that uses the illustration."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop showing Definition 1.2.14, which calls X continuous if there exists a non-negative function f such that F(x) equals the integral of f from minus infinity to x for all real x, followed by the sentence 'The function f is called the probability density function (PDF). The Fundamental Theorem of Calculus implies that, for any continuous random variable, the PDF is unique and can be defined as follows:'. A red box marks the words 'the PDF is unique and can be defined as follows:'.", "caption": "Definition 1.2.14 and the sentence after it, p. 24 (PDF p. 42). The marked words say that the PDF is unique and can be defined by the formula of Definition 1.2.15.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E002.png"}, {"alt": "Crop of Definition 1.2.15: for a continuous random variable X with CDF F, the probability density function of X is f(x) = dF(u)/du evaluated at u = x, for all x in R. A red box marks the quantifier 'for all x in R'.", "caption": "Definition 1.2.15, p. 25 (PDF p. 43). The marked quantifier requires the derivative of the CDF to exist, and to equal the PDF, at every real $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T29", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T29"}, {"id": "P001-T30", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T30"}], "headline": "The PDF of a continuous random variable is unique only up to equality almost everywhere, and it equals the derivative of the CDF at almost every point, not at every point.", "id": "P001-C01-E002", "location": "Section 1.2.4, pp. 24–25 (PDF pp. 42–43): the sentence after Definition 1.2.14, and Definition 1.2.15. The same qualification applies to Definition 1.3.11 (p. 36), to Definition 1.4.5 (p. 41), and to Definition 1.3.16 and Theorem 1.3.17 (pp. 38–39).", "pdf_page": 42, "printed_page": "24–25", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T29", "P001-T30"], "text": {"as_printed": "The function $f$ is called the \\emph{probability density function (PDF)}. The Fundamental Theorem of Calculus implies that, for any continuous random variable, the PDF is \\hl{unique} and can be defined as follows:\n\n\\textbf{Definition 1.2.15.} \\emph{Probability Density Function (PDF)}\n\nFor a continuous random variable $X$ with CDF $F$, the \\emph{probability density function} of $X$ is\n\\[\nf(x)=\\left.\\frac{dF(u)}{du}\\right|_{u=x},\\ \\hl{\\forall x\\in\\mathbb{R}}.\n\\]", "corrected": "The function $f$ is called the \\emph{probability density function (PDF)}. The Fundamental Theorem of Calculus\\hl{, in Lebesgue's form,} implies that, for any continuous random variable, the PDF is unique \\hl{up to equality almost everywhere} and \\hl{is related to the CDF} as follows:\n\n\\textbf{Definition 1.2.15.} \\emph{Probability Density Function (PDF)}\n\nFor a continuous random variable $X$ with CDF $F$, \\hl{a} \\emph{probability density function} of $X$ is \\hl{a non-negative function $f$ that satisfies Definition 1.2.14. Any two such functions are equal at Lebesgue-almost every point, and}\n\\[\nf(x)=\\left.\\frac{dF(u)}{du}\\right|_{u=x}\\quad\\hl{\\text{for Lebesgue-almost every }x\\in\\mathbb{R}}.\n\\]\n\\hl{At the remaining points the derivative may fail to exist, and $f$ may take any non-negative values there.}\n\nThe dependent statements change in the same way.\n\\begin{itemize}\n\\item Definition 1.3.11 (p.~36) and Definition 1.4.5 (p.~41): the joint PDF is the non-negative function $f$ of Definition 1.3.10 (respectively Definition 1.4.4), unique up to equality almost everywhere. It is the mixed derivative of the joint CDF $F$ in the weak sense, that is, $\\int F\\,\\partial_1\\cdots\\partial_K\\phi\\,dx=(-1)^K\\int f\\,\\phi\\,dx$ for every smooth function $\\phi$ with compact support. The formula stated at every point requires additional regularity of $f$.\n\\item Definition 1.3.16 and Theorem 1.3.17 (pp.~38--39), continuous case: $X$ and $Y$ are independent if $f(x,y)=f_X(x)f_Y(y)$ for Lebesgue-almost every $(x,y)$, or equivalently if some version of the joint PDF is the product of the marginal PDFs. The equality $f_{X|Y}(x|y)=f_X(x)$ holds for Lebesgue-almost every $x$, for almost every $y$ with respect to the distribution of $Y$. The statements for PMFs hold at every point, as printed.\n\\item Definition 1.2.20 (p.~31): the set $\\operatorname{Supp}[X]=\\{x\\in\\mathbb{R}:f(x)>0\\}$ refers to the specified version of the PDF.\n\\end{itemize}", "evidence": "The evidence is an exact counterexample taken from the book, followed by the derivation of the corrected statement. Let $X\\sim U(0,1)$, so that $F(x)=0$ for $x<0$, $F(x)=x$ for $0\\le x\\le 1$ and $F(x)=1$ for $x>1$. For $0<h<1$, I have\n\\[\n\\begin{aligned}\n\\frac{F(0-h)-F(0)}{-h}&=0, &\\qquad \\frac{F(0+h)-F(0)}{h}&=1,\\\\\n\\frac{F(1-h)-F(1)}{-h}&=1, &\\qquad \\frac{F(1+h)-F(1)}{h}&=0 .\n\\end{aligned}\n\\]\nThe one-sided limits differ at $0$ and at $1$, and $F$ has no derivative at either point. Note that no choice of the PDF changes $F$. As for uniqueness, the three functions $\\mathbf{1}_{[0,1]}$, $\\mathbf{1}_{(0,1)}$ and $\\mathbf{1}_{[0,1]}+(1/2)\\mathbf{1}_{\\{1/3\\}}$ are non-negative, and each integrates to $F(x)$ over $(-\\infty,x]$ for every $x$. They take the values $1$, $0$, $1$ at the point $0$ and the values $1$, $1$, $3/2$ at the point $1/3$.\n\nFor the joint statements, let $X$ and $Y$ be independent $U(0,1)$ random variables. Their joint CDF is $F(x)F(y)$. At the point $(0,1/3)$ the mixed derivative of Definition 1.3.11 does not exist in either order of differentiation, because the derivative in the first argument has one-sided slopes $0$ and a positive number ($1$ or $1/3$, according to the order). Next, the function\n\\[\nq(x,y)=\\mathbf{1}_{[0,1]}(x)\\,\\mathbf{1}_{[0,1]}(y)+\\frac{1}{2}\\,\\mathbf{1}_{\\{(1/3,\\,2/3)\\}}(x,y)\n\\]\nis a joint PDF of $(X,Y)$: it assigns to every event the same probability as the product of the marginal PDFs does, and its marginals are $f_X=f_Y=\\mathbf{1}_{[0,1]}$. The events $\\{X\\in D\\}$ and $\\{Y\\in E\\}$ are independent for all Borel sets $D$ and $E$, but $q(1/3,2/3)=3/2$, while $f_X(1/3)f_Y(2/3)=1$.\n\nThe corrected statement follows from the Fundamental Theorem of Calculus in Lebesgue's form: if $f$ is integrable and $F(x)=\\int_{-\\infty}^{x}f(u)\\,du$, then $F'(x)=f(x)$ for almost every $x$. If $f$ and $g$ both satisfy Definition 1.2.14 for the same $F$, then $\\int_{-\\infty}^{x}(f-g)(u)\\,du=0$ for all $x$. Applying the same theorem to $f-g$, I find that $f=g$ almost everywhere.", "problem": "Recall that Definition 1.2.14 calls $X$ continuous if there exists a non-negative function $f$ such that $F(x)=\\int_{-\\infty}^{x}f(u)\\,du$ for all $x$. Two claims in the passage do not follow from this definition. First, such a function is not unique: changing $f$ on a set of Lebesgue measure zero, for instance at one point, changes no integral. Second, the CDF need not be differentiable at every point, so the formula of Definition 1.2.15 cannot define $f(x)$ for all $x$. The uniform distribution of Example 1.2.18 (p.~28) shows both. Its CDF has corners at $0$ and at $1$, and Definition 1.2.15 assigns no value to $f(0)$ or to $f(1)$, while the example sets $f(0)=f(1)=1$. Footnote 22 requires absolute continuity only, which does not make the PDF continuous or the CDF differentiable everywhere.\n\nThe same omission recurs. Definition 1.3.11 (p.~36) and Definition 1.4.5 (p.~41) define the joint PDF as a mixed partial derivative of the joint CDF at every point. Definition 1.3.16 and Theorem 1.3.17 (pp.~38--39) state the factorization $f(x,y)=f_X(x)f_Y(y)$ and the equality $f_{X|Y}(x|y)=f_X(x)$ at every point; these hold for suitable versions of the PDFs, and not for every version. The proof of Theorem 1.2.16 uses $f=F'$ at every $x$. The set $\\operatorname{Supp}[X]$ of Definition 1.2.20 depends on which version of the PDF is specified.", "what_survives": "Definition 1.2.14, the displayed CDF and PDF of the uniform distribution, and both conclusions of Theorem 1.2.16 stand: non-negativity is part of Definition 1.2.14, and the integral equals one by letting $x\\to\\infty$ in the CDF. Every probability computed by integrating a PDF, every expectation that exists, the independence of the events $\\{X\\in D\\}$ and $\\{Y\\in E\\}$, and the statements for discrete random variables are unchanged. What changes is that uniqueness, and the identities stated at every point, hold almost everywhere. No assumption is added, and no later result or number is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the footnotes at the bottom of the page. A red box marks footnote 26: 'As Manski (1999, p. 15) intuitively describes it, \"A value x0 is said to be on the support of the probability distribution of x if there is a positive probability of observing x arbitrarily close to x0.\"' Footnote 27 follows below the box.", "caption": "Footnote 26, p. 31 (PDF p. 49), attached to the paragraph after Definition 1.2.20. The marked footnote describes support by neighbourhoods of a point.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E003.png"}, {"alt": "Crop of Definition 1.2.20, 'For a random variable X with PMF/PDF f, the support of X is Supp[X] = {x in R : f(x) > 0}', with a red box around the formula. The paragraph below says that for continuous X, Supp[X] is the set of values over which X has nonzero probability density, with footnote marker 26, and gives the example Supp[X] = [0,1] for X uniform on (0,1).", "caption": "Definition 1.2.20 and the paragraph after it, p. 31 (PDF p. 49). The marked formula is the definition of support used in the book, the set where the PMF/PDF is positive; the footnote marker 26 follows the words “nonzero probability density”. The definition is correct.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Footnote 26 describes support by neighbourhoods, which can include points where the PDF is zero; the support used in the book is the set where the PMF/PDF is positive, and the footnote should say that the two notions can differ.", "id": "P001-C01-E003", "location": "Definition 1.2.20 and footnote 26, p. 31 (PDF p. 49)", "pdf_page": 49, "printed_page": "31", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\textbf{Definition 1.2.20.} \\emph{Support}\n\nFor a random variable $X$ with PMF/PDF $f$, the \\emph{support} of $X$ is\n\\[\n\\operatorname{Supp}[X]=\\bigl\\{x\\in\\mathbb{R}:f(x)>0\\bigr\\}.\n\\]\nFootnote 26: \\hl{As Manski (1999, p.~15) intuitively describes it,} ``A value $x_0$ is said to be on the \\emph{support} of the probability distribution of $x$ if there is a positive probability of observing $x$ arbitrarily close to $x_0$.''", "corrected": "Footnote 26: \\hl{A related but distinct notion of support is described by Manski (1999, p.~15):} ``A value $x_0$ is said to be on the \\emph{support} of the probability distribution of $x$ if there is a positive probability of observing $x$ arbitrarily close to $x_0$.'' \\hl{This notion can include zero-density boundary or limit points that are excluded by the positivity-set convention used here.}", "intended_meaning": "Definition 1.2.20 governs: the support is the set where the specified PMF or PDF is positive, and Definitions 1.3.14 and 1.4.7 define conditional PDFs for conditioning values in this set, which keeps their denominators nonzero. The quotation in footnote 26 describes a different notion, the set of points whose every neighbourhood has positive probability, and the two sets can differ at points where the PDF is zero. For the continuous PDF $f(x)=6x(1-x)$ on $[0,1]$, zero elsewhere, the support of Definition 1.2.20 is $(0,1)$. Since $\\Pr[|X|<\\epsilon]=3\\epsilon^2-2\\epsilon^3>0$ for $0<\\epsilon<1$ and, by symmetry, $\\Pr[|X-1|<\\epsilon]>0$, the set described in the footnote is $[0,1]$. No formula in the book uses the footnote, and I did not check the quotation against Manski (1999)."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph beginning 'Loosely speaking, a random variable is continuous if its CDF is a continuous function.' A red box marks the sentence 'More specifically, X(Omega) must include some interval or union of intervals of the real line.'", "caption": "Section 1.2.4, p. 24 (PDF p. 42), the paragraph before Definition 1.2.14. The marked sentence says that the range $X(\\Omega)$ of a continuous random variable must include an interval.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T31", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T31"}, {"id": "P001-T32", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T32"}], "headline": "The range of a continuous random variable is uncountable, but it need not contain an interval; the sentence asserting that it must should be deleted.", "id": "P001-C01-E004", "location": "Section 1.2.4, p. 24 (PDF p. 42), paragraph before Definition 1.2.14, sentence beginning “More specifically”", "pdf_page": 42, "printed_page": "24", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T31", "P001-T32"], "text": {"as_printed": "This implies that a continuous random variable $X$ can take on an uncountably infinite number of different values; that is, $X(\\Omega)$ is an uncountable set (and therefore, so is $\\Omega$). More specifically, $X(\\Omega)$ \\hl{must include some interval or union of intervals of the real line}.", "corrected": "This implies that a continuous random variable $X$ can take on an uncountably infinite number of different values; that is, $X(\\Omega)$ is an uncountable set (and therefore, so is $\\Omega$). \\hl{Its range need not contain an interval of positive length.}\n\nThe sentence beginning ``More specifically'' is deleted. The highlighted sentence is an optional replacement for it.", "evidence": "The evidence is an exact counterexample. Let $C_0=[0,1]$. For $n\\ge 1$, let $C_n$ be the set obtained from $C_{n-1}$ by removing the open middle interval of length $4^{-n}$ from each of its component intervals, and define $C=\\bigcap_{n\\ge 0}C_n$. For example, $C_1=[0,3/8]\\cup[5/8,1]$. By induction, $C_n$ consists of $2^n$ disjoint closed intervals of common length $L_n=2^{-n-1}+2^{-2n-1}$. The removed intervals are disjoint, and their total length is\n\\[\n\\sum_{n=1}^{\\infty}2^{n-1}4^{-n}=\\frac{1}{2},\n\\]\nwhich gives $\\lambda(C)=1/2$, where $\\lambda$ denotes Lebesgue measure. An interval contained in $C$ lies in a single component of $C_n$ for every $n$, so its length is at most $L_n$. Since $L_n$ tends to zero, the set $C$ contains no interval of positive length.\n\nLet $\\Omega=C$ with its Borel sets, let $P(A)=2\\lambda(A)$, and let $X(\\omega)=\\omega$. Then $P(\\Omega)=1$, and for every $x\\in\\mathbb{R}$ I have\n\\[\nF(x)=\\Pr[X\\le x]=2\\lambda\\bigl(C\\cap(-\\infty,x]\\bigr)=\\int_{-\\infty}^{x}2\\cdot\\mathbf{1}_C(u)\\,du .\n\\]\nThis shows that $X$ is continuous in the sense of Definition 1.2.14, with PDF $f=2\\cdot\\mathbf{1}_C$, and its range is $X(\\Omega)=C$. The set where $f$ is positive is also $C$, and the example satisfies $X(\\Omega)=\\operatorname{Supp}[X]$ exactly.", "problem": "The marked sentence says that the range of a random variable contains an interval of positive length whenever the random variable is continuous. This is false. A random variable can satisfy Definition 1.2.14 while its range is a closed, nowhere-dense set of positive Lebesgue measure, which contains no interval of positive length. If ``interval'' were allowed to mean a single point, the sentence would hold for every random variable and would say nothing; I read it as a claim about intervals of positive length. The statement before it, that $X(\\Omega)$ is uncountable, is correct.", "what_survives": "The statement that $X(\\Omega)$ and $\\Omega$ are uncountable is unaffected, and so are Definition 1.2.14, the probabilities of intervals in Theorem 1.2.17 and the uniform and normal examples. I found no later result that relies on the sentence. Theorem 4.3.3 (p.~158), for instance, assumes $\\operatorname{Supp}[X]=[a,b]$ separately."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The heading 'Example 1.2.13. CDF of a Fair Die Roll' and five lines of text. Red boxes mark the word 'necessarily' in 'the CDF is necessarily a step function' and, on the next line, the words 'flat everywhere except where it jumps'.", "caption": "Example 1.2.13, p. 23 (PDF p. 41): the first sentence, with “necessarily” and the description of a step function marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E005.png"}, {"alt": "Four lines of text. A red box, spanning the end of the third line and the start of the fourth, marks the words 'like the CDF of a discrete random variable'.", "caption": "Paragraph after Definition 3.3.1, p. 116 (PDF p. 134): the comparison of the empirical CDF with the CDF of a discrete random variable is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E005-p116.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T27", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T27"}, {"id": "P001-T28", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T28"}], "headline": "The CDF of a discrete random variable is the sum of its jumps; it is flat between jumps when the range is finite, as for a die roll, but not for every discrete random variable.", "id": "P001-C01-E005", "location": "Example 1.2.13, first sentence, p. 23 (PDF p. 41); the comparison is repeated after Definition 3.3.1, p. 116 (PDF p. 134)", "pdf_page": 41, "printed_page": "23", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T27", "P001-T28"], "text": {"as_printed": "\\emph{p.~23.} For discrete random variables such as a die roll, the CDF is \\hl{necessarily} a step function, that is, a function that is \\hl{flat everywhere except where it} ``jumps'' from one value to another.\n\n\\emph{p.~116.} Note that the empirical CDF is always a nondecreasing step function, \\hl{like the CDF of a discrete random variable}.", "corrected": "\\emph{p.~23.} For \\hl{a discrete random variable, the CDF is the sum of the jumps contributed by its point masses. For finite range, as in a die roll, it is constant between the finitely many possible values.}\n\n\\emph{p.~116.} Note that the empirical CDF is always a nondecreasing step function, like the CDF of a discrete random variable \\hl{with finite range}.\n\nIn symbols: let $X$ be a discrete random variable with distinct possible values $q_i$ and masses $p_i=\\Pr[X=q_i]$, so that $\\sum_i p_i=1$. Then its CDF is\n\\[\nF(x)=\\sum_i p_i\\,\\mathbf{1}\\{q_i\\le x\\}\\quad\\text{for all }x\\in\\mathbb{R},\n\\]\nand the jump of $F$ at $q_i$ is $p_i$. If the range of $X$ is finite, then $F$ is constant between consecutive possible values.", "evidence": "An exact counterexample. Let $\\Omega$ be the set of pairs of integers $(m,k)$ such that $m\\ge 0$ and $0\\le k<2^m$, and define\n\\[\nP(\\{(m,k)\\})=2^{-(2m+1)},\\qquad X(m,k)=\\frac{2k+1}{2^{m+1}} .\n\\]\nThe masses are positive, and their sum is $\\sum_{m\\ge 0}2^m\\,2^{-(2m+1)}=\\sum_{m\\ge 0}2^{-(m+1)}=1$. The range of $X$ is the set of dyadic rationals in $(0,1)$, which is countable, so $X$ is discrete under Definition 1.2.4. Let $F$ denote the CDF of $X$, and fix real numbers $0\\le a<b\\le 1$. Fix an integer $m\\ge 0$ such that $2^{-m}<b-a$, and set $k=\\lfloor 2^m a+1/2\\rfloor$ and $q=(2k+1)/2^{m+1}$. Then $a<q<b$, and\n\\[\nF(b)-F(a)\\ge \\Pr[X=q]=2^{-(2m+1)}>0 .\n\\]\nThis implies that $F$ is strictly increasing on $(0,1)$ and constant on no subinterval. The point $1/3$ is not a value of $X$, and $F$ is continuous there: given $M$, every $y$ close enough to $1/3$ lies on the same side of each value $(2k+1)/2^{m+1}$ with $m\\le M$ as $1/3$ does, so $F(y)$ and $F(1/3)$ differ by at most the mass $2^{-(M+1)}$ of the remaining values. Evaluating the infinite sums exactly, I find $F(1/4)=3/16$, $F(1/3)=1/5$, $F(1/2)=3/4$, $F(2/3)=4/5$ and $F(3/4)=15/16$, with a jump of $1/2$ at $1/2$.", "problem": "Definition 1.2.4 calls $X$ discrete when its range is countable, and a countable range can be dense in an interval. The CDF of such a random variable still increases only by jumps, but it need not be constant on any interval. The description ``flat everywhere except where it jumps'', and the flat segments drawn in Figure 1.2.1, are correct when the range is finite, as for the die roll. They do not describe every discrete CDF.", "what_survives": "Figure 1.2.1 and the probabilities for the die roll, Definition 1.2.4, Theorems 1.2.10 and 1.2.12, and the empirical CDF of Definition 3.3.1 are unaffected; the empirical CDF has at most $n$ jumps and is a step function. I found no later proof or calculation that relies on the CDF of every discrete random variable being constant between its jumps. The correction weakens the general description: the CDF is said to be constant between jumps only when the range is finite."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "A paragraph of five lines followed by three bulleted formulas. A red box marks the lower limit 'x=1' under the summation sign in the third bullet, 'Pr[X < 4] = sum from x=1 to 3 of f(x)'.", "caption": "First bulleted list on p. 20 (PDF p. 38): the lower limit $x=1$ of the sum in the third bullet is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T25", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T25"}, {"id": "P001-T26", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T26"}], "headline": "For an integer-valued random variable $X$, the sum for $\\Pr[X<4]$ must start at $-\\infty$, not at $1$.", "id": "P001-C01-E006", "location": "First bulleted list on p. 20 (PDF p. 38), third bullet", "pdf_page": 38, "printed_page": "20", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T25", "P001-T26"], "text": {"as_printed": "To illustrate how the PMF can fully describe a discrete random variable, consider a random variable $X$ such that $f(x)=0$ for all $x\\notin\\mathbb{Z}$ (that is, $X$ takes on only integer values). Then\n\\begin{itemize}\n\\item $\\Pr[X\\ge 3]=\\sum_{x=3}^{\\infty}f(x)$.\n\\item $\\Pr[X\\ge 3\\text{ or }X=1]=f(1)+\\sum_{x=3}^{\\infty}f(x)$.\n\\item $\\Pr[X<4]=\\sum_{x=\\hl{1}}^{3}f(x)$.\n\\end{itemize}", "corrected": "Let $X$ be a discrete random variable with PMF $f$ such that $f(x)=0$ for all $x\\notin\\mathbb{Z}$. Then\n\\[\n\\Pr[X<4]=\\sum_{x=\\hl{-\\infty}}^{3}f(x).\n\\]\nThe sum from $1$ is valid exactly when $\\Pr[X\\le 0]=0$.", "evidence": "Exact counterexamples and a derivation. If $X=0$ with probability one, then $f(0)=1$ and $\\Pr[X<4]=1$, while the printed sum is $f(1)+f(2)+f(3)=0$. For the Bernoulli random variable of Example 1.2.7 with $p=1/3$, I have $f(0)=2/3$ and $f(1)=1/3$, so that $\\Pr[X<4]=1$ and the printed sum is $1/3$.\n\n\\begin{audittable}{lccc}{The third bullet as printed and as corrected, in four exact examples.}\nDistribution of $X$ & $\\Pr[X<4]$ & $\\sum_{x=1}^{3}f(x)$ (printed) & $\\sum_{x=-\\infty}^{3}f(x)$ (corrected) \\\\\n$X=0$ with probability one & $1$ & $0$ & $1$ \\\\\n$X=-2$ with probability one & $1$ & $0$ & $1$ \\\\\nBernoulli, $p=1/3$ (Example 1.2.7) & $1$ & $1/3$ & $1$ \\\\\nFair die (Example 1.2.6) & $1/2$ & $1/2$ & $1/2$ \\\\\n\\end{audittable}\n\nFor the corrected identity, write $A_n=\\{X=n\\}$ for each integer $n$. These events are pairwise disjoint, and since $X$ is integer-valued,\n\\[\n\\{X<4\\}=\\bigcup_{n\\le 3}A_n .\n\\]\nCountable additivity gives $\\Pr[X<4]=\\sum_{n=-\\infty}^{3}f(n)$. The sum is the limit of $\\sum_{n=-m}^{3}f(n)$ as $m\\to\\infty$, which exists because the terms are nonnegative and the partial sums are at most $1$.", "problem": "The passage restricts $X$ only to integer values, so $X$ may put mass at $0$ and at the negative integers, as the Bernoulli random variable of Example 1.2.7 does at $0$. The printed sum omits that mass. By countable additivity, I have\n\\[\n\\Pr[X<4]-\\sum_{x=1}^{3}f(x)=\\Pr[X\\le 0],\n\\]\nand the third bullet holds if and only if $\\Pr[X\\le 0]=0$.", "what_survives": "The first two bullets, the three probabilities for the die roll in the list that follows ($2/3$, $5/6$ and $1/2$), Definitions 1.2.4 and 1.2.5, and Theorems 1.2.9 and 1.2.10 are unaffected. For the die roll $\\Pr[X\\le 0]=0$, and the printed sum from $1$ is correct there. I found no later use of the third bullet in the book."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Five lines of text. A red box, spanning the end of the second line and the start of the third, marks the words 'the former always occurs whenever the latter does'.", "caption": "First paragraph of p. 9 (PDF p. 27): the parenthesis that explains monotonicity, with the reversed clause marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T23", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T23"}], "headline": "In the explanation of monotonicity, “former” and “latter” are interchanged: if one event is a subset of another, the latter occurs whenever the former does.", "id": "P001-C01-E007", "location": "First paragraph of p. 9 (PDF p. 27), the explanation of monotonicity that follows Theorem 1.1.4", "pdf_page": 27, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T23"], "text": {"as_printed": "Monotonicity implies that, if one event is a subset of another (so that \\hl{the former always occurs whenever the latter does}), then the probability of the former occurring is no greater than that of the latter.", "corrected": "Monotonicity implies that, if one event is a subset of another (so that \\hl{the latter always occurs whenever the former does}), then the probability of the former occurring is no greater than that of the latter.", "evidence": "An exact counterexample to the parenthesis: for a fair die with $A=\\{6\\}$ and $B=\\{4,5,6\\}$, the outcome $4$ lies in $B$ but not in $A$, although $A\\subseteq B$. Here $P(A)=1/6\\le 1/2=P(B)$, as the theorem requires.", "problem": "If $A\\subseteq B$, then $B$ occurs whenever $A$ does, and $B$ can occur without $A$. The parenthesis states the converse, which describes $B\\subseteq A$.", "what_survives": "Theorem 1.1.4 and its proof state the inclusion and the inequality correctly; only the parenthesis changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The last three lines of a proof, a displayed sum, and five lines of the next paragraph. A small red box marks the greater-than-or-equal sign in 'if P(A_i) >= 0 for i = 1, 2, 3, ...'. A second red box marks the words '{A, A^C} is a partition of Omega'.", "caption": "End of the proof of Theorem 1.1.13 and the next paragraph, p. 13 (PDF p. 31): the sign $\\ge$ in the condition on $P(A_i)$ and the remark on $\\{A,A^C\\}$ are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T24", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T24"}], "headline": "The proof of Theorem 1.1.13 should require $P(A_i)>0$, not $P(A_i)\\ge 0$, before conditioning on $A_i$, and $\\{A,A^C\\}$ is a partition of $\\Omega$ only when $A$ is neither empty nor all of $\\Omega$.", "id": "P001-C01-E008", "location": "Proof of Theorem 1.1.13 and the paragraph that follows it, p. 13 (PDF p. 31)", "pdf_page": 31, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T24"], "text": {"as_printed": "Additionally, if $P(A_i)\\hl{\\ge}0$ for $i=1,2,3,\\ldots$, then we can apply the Multiplicative Law of Probability to each term in the summation to obtain\n\\[\nP(B)=\\sum_i P(B\\mid A_i)P(A_i).\n\\]\n\\ldots\\ Notice that \\hl{for any event} $A$, $\\{A,A^C\\}$ is a partition of $\\Omega$, so the method we used to calculate $P(B)$ above was indeed a special case of the Law of Total Probability.", "corrected": "Additionally, if $P(A_i)\\hl{>}0$ for $i=1,2,3,\\ldots$, then we can apply the Multiplicative Law of Probability to each term in the summation to obtain\n\\[\nP(B)=\\sum_i P(B\\mid A_i)P(A_i).\n\\]\n\\ldots\\ Notice that for any event $A$, \\hl{the nonempty members of} $\\{A,A^C\\}$ \\hl{form} a partition of $\\Omega$, so the method we used to calculate $P(B)$ above was indeed a special case of the Law of Total Probability.\n\nThe step in full: if $\\{A_1,A_2,A_3,\\ldots\\}$ is a partition of $\\Omega$, $B\\in S$, and $P(A_i)>0$ for $i=1,2,3,\\ldots$, then $P(B\\cap A_i)=P(B\\mid A_i)P(A_i)$ for each $i$, and\n\\[\nP(B)=\\sum_i P(B\\cap A_i)=\\sum_i P(B\\mid A_i)P(A_i).\n\\]", "evidence": "An exact counterexample to the printed condition. Let $\\Omega=\\{a,b\\}$ with $P(\\{a\\})=1$ and $P(\\{b\\})=0$, and set $A_1=\\{a\\}$, $A_2=\\{b\\}$ and $B=\\Omega$. Then $\\{A_1,A_2\\}$ is a partition of $\\Omega$ and both members have nonnegative probability. The first form of the theorem holds:\n\\[\nP(B)=P(B\\cap A_1)+P(B\\cap A_2)=1+0=1.\n\\]\nThe second form cannot be written, because\n\\[\nP(B\\mid A_2)=\\frac{P(B\\cap A_2)}{P(A_2)}=\\frac{0}{0}\n\\]\nis undefined. For the remark, $A=\\varnothing$ and $A=\\Omega$ both give $\\{A,A^C\\}=\\{\\varnothing,\\Omega\\}$, which contains the empty set.", "problem": "By Definition 1.1.8, the conditional probability $P(B\\mid A_i)$ is defined only when $P(A_i)>0$, and the statement of Theorem 1.1.13 on p.~12 carries that condition. The proof restates the condition as $P(A_i)\\ge 0$. This holds for every event, and it would permit conditioning on an event of probability zero. Separately, Definition 1.1.12 requires the members of a partition to be nonempty, and $\\{A,A^C\\}$ has an empty member when $A=\\varnothing$ or $A=\\Omega$.", "what_survives": "Theorem 1.1.13 as stated on p.~12, the proof of its first form, and the calculation for the coin flip and die roll are unaffected. With $P(A_i)>0$, the proof of the second form holds as printed. Dropping an empty member repairs the remark, but it does not permit conditioning on a nonempty member of probability zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Footnote 6 in full, nine lines of small type. A red box marks the words 'is impossible to define the probability of every subset of Omega' on the fifth line.", "caption": "Footnote 6, p. 6 (PDF p. 24): the clause on defining the probability of every subset of $\\Omega$ is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "Footnote 6 should say that a specified probability law need not extend to every subset of the sample space, not that no probability can be defined on every subset.", "id": "P001-C01-E009", "location": "Footnote 6, p. 6 (PDF p. 24)", "pdf_page": 24, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Some readers well versed in set theory may wonder why we do not simply let $S=\\mathcal{P}(\\Omega)$, the power set (that is, the set of all subsets) of $\\Omega$. For reasons that we will not discuss in this book, this does not always work; for some sample spaces $\\Omega$, it is \\hl{impossible to define the probability of every subset} of $\\Omega$ in a manner consistent with the axioms of probability (see Definition 1.1.2).", "corrected": "For reasons that we will not discuss in this book, this does not always work; \\hl{a specified probability law need not extend to all subsets of the sample space while preserving its required properties}.", "intended_meaning": "The footnote explains why the event space is not always taken to be the power set. Read literally, the marked clause is too strong. For every nonempty $\\Omega$ and any fixed $\\omega_0\\in\\Omega$, the point mass $P(A)=\\mathbf{1}\\{\\omega_0\\in A\\}$ is defined for every $A\\subseteq\\Omega$ and satisfies the three axioms of Definition 1.1.2, since at most one set in a pairwise disjoint sequence contains $\\omega_0$. The intended statement is that a law with further required properties, such as the uniform law on $[0,1]$, need not extend to all subsets while keeping those properties. That statement is correct, and nothing in the book depends on the footnote."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the sentence 'From this definition, it follows that two functions of a random variable, g(X) and h(X), are equal as random variables if and only if they are equal as functions on X(Omega)', followed by Theorem 1.3.2: 'Let X be a random variable, and let f and g be functions of X. Then g(X) = h(X) if and only if, for all x in X(Omega), g(x) = h(x).' A red box marks the words 'f and g'.", "caption": "Theorem 1.3.2 and the sentence before it, p. 32 (PDF p. 50). The marked words introduce the functions as $f$ and $g$; the sentence above the theorem and the display use $g$ and $h$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T33", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T33"}], "headline": "Theorem 1.3.2 introduces its functions as $f$ and $g$ but states the equivalence for $g$ and $h$; the functions should be introduced as $g$ and $h$.", "id": "P001-C01-E010", "location": "Theorem 1.3.2, p. 32 (PDF p. 50), first line of the statement", "pdf_page": 50, "printed_page": "32", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T33"], "text": {"as_printed": "\\textbf{Theorem 1.3.2.} \\emph{Equality of Functions of a Random Variable}\n\nLet $X$ be a random variable, and let $\\hl{f}$ and $\\hl{g}$ be functions of $X$. Then\n\\[\ng(X)=h(X)\\iff\\forall x\\in X(\\Omega),\\ g(x)=h(x).\n\\]", "corrected": "Let $X$ be a random variable, and let $\\hl{g}$ and $\\hl{h}$ be functions of $X$. Then\n\\[\ng(X)=h(X)\\iff\\forall x\\in X(\\Omega),\\ g(x)=h(x).\n\\]", "evidence": "The evidence is a check of the page: the preceding paragraph, the theorem and its whole proof.", "problem": "The statement never introduces $h$ and never uses $f$. The sentence before the theorem, the displayed equivalence and both directions of the proof are written for $g$ and $h$.", "what_survives": "The theorem and its proof are correct once the names agree. Nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-E010", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of text. A red box marks 'Section 2.1.5' in the parenthesis '(see, e.g., Section 2.1.5, Expectation of a Function of a Random Variable)'.", "caption": "Paragraph after Definition 1.2.3, p. 17 (PDF p. 35): the cross-reference “Section 2.1.5” is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The cross-reference “Section 2.1.5” should read “Theorem 2.1.5”.", "id": "P001-C01-N003", "location": "Paragraph after Definition 1.2.3, p. 17 (PDF p. 35)", "pdf_page": 35, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Since $g(X)$ is itself a random variable, it is possible to define $A[g(X)]$ (see, e.g., \\hl{Section} 2.1.5, \\emph{Expectation of a Function of a Random Variable}).", "corrected": "Since $g(X)$ is itself a random variable, it is possible to define $A[g(X)]$ (see, e.g., \\hl{Theorem} 2.1.5, \\emph{Expectation of a Function of a Random Variable}).", "evidence": "A check of p.~17, of Theorem 2.1.5 on p.~47, and of the table of contents on p.~viii.", "problem": "There is no Section 2.1.5: Section 2.1 has the subsections 2.1.1 to 2.1.3. The title quoted is that of Theorem 2.1.5 on p.~47.", "what_survives": "The quoted title identifies the theorem, and nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-N003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of text. A red box marks the word 'Additonally', spelt without its second i, at the start of the last line.", "caption": "Example 1.2.13, p. 23 (PDF p. 41): the misspelt word is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N004.png"}, {"alt": "Three lines of text. Red boxes mark an upright capital B in 'B denote the event of observing 3' and an upright capital C in 'and C denote the event of observing 6'. The letter A on the same line and the letters B and C on the third line are italic.", "caption": "Example 1.1.11, p. 11 (PDF p. 29): the upright $\\mathrm{B}$ and $\\mathrm{C}$ are marked; the same letters are italic on the third line.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N004-p11.png"}, {"alt": "The last three lines of a shaded definition box. Red boxes mark an upright X and an upright Y in 'to denote that X and Y are independent'. The X and Y earlier on the same line are italic.", "caption": "Definition 1.3.16, p. 38 (PDF p. 56): the upright $\\mathrm{X}$ and $\\mathrm{Y}$ in the last sentence are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N004-p38.png"}, {"alt": "Four lines of text. A red box marks an upright Y in 'PMF/PDF of Y, that is'. The Y later on the same line is italic.", "caption": "Paragraph on the marginal PMF/PDF, p. 42 (PDF p. 60): the upright $\\mathrm{Y}$ is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N004-p42.png"}], "formal_release": null, "formal_statements": [], "headline": "“Additonally” should be “Additionally”, and five symbols on pp. 11, 38 and 42 are set upright where the same symbols nearby are italic.", "id": "P001-C01-N004", "location": "Example 1.2.13, p. 23 (PDF p. 41); also Example 1.1.11, p. 11 (PDF p. 29), Definition 1.3.16, p. 38 (PDF p. 56), and p. 42 (PDF p. 60)", "pdf_page": 41, "printed_page": "23", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~23.} \\hl{Additonally}, as usual, we can use the CDF to compute event probabilities:\n\n\\emph{p.~11.} Let $A$ denote the event of observing heads, $\\hl{\\mathrm{B}}$ denote the event of observing 3, and $\\hl{\\mathrm{C}}$ denote the event of observing 6.\n\n\\emph{p.~38.} We write $X\\perp\\!\\!\\!\\perp Y$ to denote that $\\hl{\\mathrm{X}}$ and $\\hl{\\mathrm{Y}}$ are independent.\n\n\\emph{p.~42.} \\ldots\\ to obtain the univariate PMF/PDF of $\\hl{\\mathrm{Y}}$, that is, the PMF/PDF of $Y$ ignoring the existence of the $X$s.", "corrected": "\\emph{p.~23.} \\hl{Additionally}, as usual, we can use the CDF to compute event probabilities:\n\n\\emph{p.~11.} Let $A$ denote the event of observing heads, $\\hl{B}$ denote the event of observing 3, and $\\hl{C}$ denote the event of observing 6.\n\n\\emph{p.~38.} We write $X\\perp\\!\\!\\!\\perp Y$ to denote that $\\hl{X}$ and $\\hl{Y}$ are independent.\n\n\\emph{p.~42.} \\ldots\\ to obtain the univariate PMF/PDF of $\\hl{Y}$, that is, the PMF/PDF of $Y$ ignoring the existence of the $X$s.", "evidence": "A check of the four pages at high magnification.", "problem": "The word lacks its second ``i''. The letters $B$ and $C$ on p.~11, $X$ and $Y$ after ``to denote that'' on p.~38, and the first $Y$ in ``univariate PMF/PDF of $Y$'' on p.~42 are set upright, unlike the same symbols in the surrounding text.", "what_survives": "These are slips of spelling and typeface, and no meaning changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-N004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the paragraph beginning 'For discrete X, Supp[X] is the set of values that X takes on with nonzero probability.' In the sentence 'Supp[X] = [0,1] = {x in R : 0 <= x <= 1}' a red box marks the first x inside the braces, which is printed in bold.", "caption": "Paragraph after Definition 1.2.20, p. 31 (PDF p. 49). The marked $\\mathbf{x}$ in the set-builder is bold; the $x$ later in the same braces is italic.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N005.png"}, {"alt": "Crop of Definition 1.3.14, the conditional PDF of Y given X = x, with domain 'for all y in R and for all x in Supp[X]' in italic x, followed by the sentence 'Again, the domain condition x in Supp[X] ensures that the denominator is nonzero.' A red box marks the x in that sentence, which is printed in bold.", "caption": "Definition 1.3.14 and the sentence after it, p. 38 (PDF p. 56). The marked $\\mathbf{x}$ is bold, while the definition above it uses the italic $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N005-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The scalar $x$ is printed in bold in two places; it should be the italic $x$ of the surrounding text.", "id": "P001-C01-N005", "location": "p. 31 (PDF p. 49), paragraph after Definition 1.2.20; p. 38 (PDF p. 56), first sentence after Definition 1.3.14", "pdf_page": 49, "printed_page": "31", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~31: For example, if $X\\sim U(0,1)$, then $\\operatorname{Supp}[X]=[0,1]=\\{\\hl{\\mathbf{x}}\\in\\mathbb{R}:0\\le x\\le 1\\}$.\n\np.~38: Again, the domain condition $\\hl{\\mathbf{x}}\\in\\operatorname{Supp}[X]$ ensures that the denominator is nonzero.", "corrected": "p.~31: For example, if $X\\sim U(0,1)$, then $\\operatorname{Supp}[X]=[0,1]=\\{\\hl{x}\\in\\mathbb{R}:0\\le x\\le 1\\}$.\n\np.~38: Again, the domain condition $\\hl{x}\\in\\operatorname{Supp}[X]$ ensures that the denominator is nonzero.", "evidence": "The evidence is a check of the two pages: the marked letters are set in bold, while Definition 1.2.20 and Definition 1.3.14 use the italic $x$.", "problem": "Both passages concern a value $x$ of a single random variable $X$, and p.~40 reserves boldface letters for vectors. In the first passage the bold $\\mathbf{x}$ and the italic $x$ are the same variable.", "what_survives": "Both statements are correct with a scalar $x$, and only the typeface changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Theorem 1.3.13, Marginal PDF: for jointly continuous random variables X and Y with joint PDF f, the marginal PDF of Y is f_Y(y) = integral from minus infinity to infinity of f(x,y) dx, for all y in R. A red box marks the quantifier 'for all y in R'.", "caption": "Theorem 1.3.13, p. 37 (PDF p. 55). The marked quantifier asserts the formula for the marginal PDF at every real $y$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N007.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T36", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T36"}], "headline": "The marginal PDF equals the integral of the joint PDF for almost every $y$, not for every $y$: at exceptional points the integral can be infinite.", "id": "P001-C01-N007", "location": "Theorem 1.3.13, p. 37 (PDF p. 55). The same qualification applies to the remark after Definition 1.3.14 (p. 38) and to the continuous case of Definition 1.4.6 (p. 42).", "pdf_page": 55, "printed_page": "37", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T36"], "text": {"as_printed": "\\textbf{Theorem 1.3.13.} \\emph{Marginal PDF}\n\nFor jointly continuous random variables $X$ and $Y$ with joint PDF $f$, the \\emph{marginal PDF} of $Y$ is\n\\[\nf_Y(y)=\\int_{-\\infty}^{\\infty}f(x,y)\\,dx,\\ \\hl{\\forall y\\in\\mathbb{R}}.\n\\]", "corrected": "\\textbf{Theorem 1.3.13.} \\emph{Marginal PDF}\n\nFor jointly continuous random variables $X$ and $Y$ with joint PDF $f$, \\hl{a} \\emph{marginal PDF} of $Y$ \\hl{satisfies}\n\\[\nf_Y(y)=\\int_{-\\infty}^{\\infty}f(x,y)\\,dx\\quad\\hl{\\text{for Lebesgue-almost every }y\\in\\mathbb{R}}.\n\\]\n\\hl{The integral is finite for Lebesgue-almost every $y$. On the null set where it is infinite, $f_Y$ may be given any finite non-negative values.}\n\nIn Definition 1.3.14 and the remark after it (p.~38), the conditional PDF of $Y$ given $X=x$ is the ratio $f(x,y)/\\int_{-\\infty}^{\\infty}f(x,v)\\,dv$ at every $x$ where this integral is positive and finite; at the remaining values of $x$, which have probability zero, \\hl{one fixed PDF may be chosen for all such $x$}. The conditional PDF of $X$ given $Y=y$ and the continuous case of Definition 1.4.6 (p.~42) are qualified in the same way.", "evidence": "The evidence is an exact counterexample, followed by the derivation of the corrected statement. Let $s(x)=e^x/(1+e^x)$, and define\n\\[\nf(x,y)=\\frac{s(x)^2}{\\pi\\,(1+y^2e^{2x})}\n\\]\nfor $(x,y)\\in\\mathbb{R}^2$. This function is smooth, strictly positive and bounded by $1/\\pi$. Substituting $z=e^xy$, I obtain $\\int_{-\\infty}^{\\infty}f(x,y)\\,dy=s(x)^2e^{-x}=e^x/(1+e^x)^2$. With $t=e^x$, the integral of this function over $x$ equals $\\int_0^\\infty(1+t)^{-2}\\,dt=1$, so $f$ is a joint PDF. Both mixed partial derivatives of its joint CDF exist at every point and equal $f$, and the example does not depend on the choice of a version of the joint PDF. At $y=0$, I have $f(x,0)=s(x)^2/\\pi\\ge 1/(4\\pi)$ for $x\\ge 0$, which gives\n\\[\n\\int_{-\\infty}^{\\infty}f(x,0)\\,dx=\\infty .\n\\]\nFor $y\\ne 0$ the integral is positive and at most $1/(2\\pi)+1/(2\\pi y^2)$. If the marginal PDF is given the value $f_Y(0)=1$, then $0\\in\\operatorname{Supp}[Y]$, and the ratio $f(x,0)/f_Y(0)$ has infinite integral over $x$ and is not a PDF.\n\nFor the corrected statement, let $f$ be any joint PDF and let $g(y)=\\int_{-\\infty}^{\\infty}f(x,y)\\,dx$. By Tonelli's theorem, I have $\\int_{-\\infty}^{\\infty}g(y)\\,dy=1$. Then $n\\,\\lambda\\{y:g(y)>n\\}\\le 1$ for every $n\\ge 1$, where $\\lambda$ denotes Lebesgue measure, and this implies that $\\lambda\\{y:g(y)=\\infty\\}=0$. Let $\\tilde g$ equal $g$ where $g$ is finite and take any finite non-negative values elsewhere. Then $\\Pr[Y\\in B]=\\int_B\\tilde g(y)\\,dy$ for every Borel set $B$, and $\\tilde g$ is a PDF of $Y$.", "problem": "Definition 1.2.14 makes a PDF a real-valued function, and Theorem 1.3.13 asserts that the integral of the joint PDF over $x$ is the value of the marginal PDF at every $y$. The hypotheses do not make this integral finite at every $y$. It is finite for almost every $y$, but it can be infinite at exceptional points even when the joint PDF is smooth, bounded and strictly positive. At such a point no real value of $f_Y(y)$ satisfies the printed equality. This is a qualification of the same kind as in \\fid{P001-C01-E002}.\n\nTwo later statements carry the same quantifier. The remark after Definition 1.3.14 (p.~38) says that conditional PDFs are univariate PDFs ``given any fixed value for the conditioning variable'', but the ratio is a PDF only where the integral in its denominator is positive and finite. The continuous case of Definition 1.4.6 (p.~42) states the formula with several variables integrated out, again for all $y$.", "what_survives": "Event probabilities and the marginal distributions are unchanged. Expectations and the Law of Iterated Expectations (Theorem 2.2.17) use these identities inside integrals, where equality almost everywhere suffices. What changes is the wording at fixed points. The conditional PDF of Definition 1.3.14 and the remark after it, Theorem 1.3.15, the continuous cases of Definitions 1.4.6 and 1.4.7, and the conditional expectations defined from them at a fixed conditioning value (Definitions 2.2.10, 2.2.15, 2.3.3 and 2.3.4) hold outside an exceptional set of conditioning values that has probability zero. No numerical result is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-N007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Theorem 1.3.17, Implications of Independence (Part I), with its five bulleted statements. A red box marks the last two: 'for all D, E subsets of R, the events {X in D} and {Y in E} are independent' and 'For all functions g of X and h of Y, g(X) is independent of h(Y).'", "caption": "Theorem 1.3.17, p. 39 (PDF p. 57). The two marked statements quantify over all subsets $D,E\\subseteq\\mathbb{R}$ and over all functions $g$ of $X$ and $h$ of $Y$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N008.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T47", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T47"}], "headline": "The last two statements of Theorem 1.3.17 are meant for Borel sets $D$ and $E$ and for Borel measurable functions $g$ and $h$, with the independence of $g(X)$ and $h(Y)$ understood in the general sense defined by events.", "id": "P001-C01-N008", "location": "Theorem 1.3.17, p. 39 (PDF p. 57), fourth and fifth statements, read with Definition 1.3.16, p. 38 (PDF p. 56)", "pdf_page": 57, "printed_page": "39", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P001-T47"], "text": {"as_printed": "\\textbf{Theorem 1.3.17.} \\emph{Implications of Independence (Part I)}\n\nLet $X$ and $Y$ be either two discrete random variables with joint PMF $f$ or two jointly continuous random variables with joint PDF $f$. Then the following statements are equivalent (that is, each one implies all the others):\n\\begin{itemize}\n\\item $X\\perp\\!\\!\\!\\perp Y$.\n\\item $\\forall x,y\\in\\mathbb{R}$, $f(x,y)=f_X(x)f_Y(y)$.\n\\item $\\forall x\\in\\mathbb{R}$ and $\\forall y\\in\\operatorname{Supp}[Y]$, $f_{X|Y}(x|y)=f_X(x)$.\n\\item $\\hl{\\forall D,E\\subseteq\\mathbb{R}}$, the events $\\{X\\in D\\}$ and $\\{Y\\in E\\}$ are independent.\n\\item \\hl{For all functions} $g$ of $X$ and $h$ of $Y$, $g(X)\\perp\\!\\!\\!\\perp h(Y)$.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item $\\forall D,E\\subseteq\\mathbb{R}$, the events $\\{X\\in D\\}$ and $\\{Y\\in E\\}$ are independent.\n\\item For all functions $g$ of $X$ and $h$ of $Y$, $g(X)\\perp\\!\\!\\!\\perp h(Y)$.\n\\end{itemize}\n\\hl{Footnote to these two statements: Sets and functions in these statements are understood to be Borel measurable. For arbitrary real-valued random variables $U$ and $V$, $U$ is independent of $V$ if $\\Pr[U\\in A,V\\in B]=\\Pr[U\\in A]\\Pr[V\\in B]$ for every pair of Borel sets $A,B\\subseteq\\mathbb{R}$.}", "intended_meaning": "The two marked statements are meant for Borel sets $D$ and $E$ and for Borel measurable functions $g$ and $h$, with $g(X)\\perp\\!\\!\\!\\perp h(Y)$ understood in the general sense that $\\Pr[g(X)\\in A,h(Y)\\in B]=\\Pr[g(X)\\in A]\\Pr[h(Y)\\in B]$ for all Borel sets $A$ and $B$. The qualifiers matter for two reasons: $\\{X\\in D\\}$ need not be an event when $D$ is not measurable, and $g(X)$ and $h(Y)$ can form a pair that is neither two discrete nor two jointly continuous random variables, for which Definition 1.3.16 does not define independence (for independent uniform $X$ and $Y$, let $g(x)=\\mathbf{1}\\{x>1/2\\}$ and $h(y)=y$). The book points to this reading: footnote 15 (p.~16) restricts a ``function of $X$'' to one for which $g(X)$ is a random variable, footnote 21 (p.~24) says that its definitions and theorems can almost always be generalized to all probability distributions, and the proofs of Theorem 3.2.9 (p.~102) and Theorem 3.2.19 (p.~106) cite ``a generalization of Theorem 1.3.17''. Under this reading the statements are correct, since for Borel sets $A$ and $B$ the event $\\{g(X)\\in A,h(Y)\\in B\\}$ equals $\\{X\\in g^{-1}(A),Y\\in h^{-1}(B)\\}$, the sets $g^{-1}(A)$ and $h^{-1}(B)$ are Borel, and by the fourth statement the probability of this event is the product of the two probabilities. The indicator and the square used in the two later proofs are Borel measurable, and the discrete case holds as printed.\n\nThe general measurable-transformation claim is supported by the preimage derivation above. The related formal statements in P001-T47 additionally assume finite second moments of the original and transformed variables."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-N008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Definition 1.4.7, Conditional PMF/PDF (Multivariate Case), with the discrete formula and the continuous formula for the conditional PMF/PDF of Y given the vector X = x. Red boxes mark 'Supp[X]' at the end of each formula, where X is printed in italic and not in bold.", "caption": "Definition 1.4.7, p. 42 (PDF p. 60). In both marked domain conditions the $X$ inside $\\operatorname{Supp}[\\,\\cdot\\,]$ is italic, while the conditioning vector $\\mathbf{X}$ is bold everywhere else in the definition.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C01-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "In Definition 1.4.7 the support is printed as $\\operatorname{Supp}[X]$ with an italic $X$; it should be $\\operatorname{Supp}[\\mathbf{X}]$, the support of the random vector.", "id": "P001-C01-N009", "location": "Definition 1.4.7, p. 42 (PDF p. 60), the two domain conditions", "pdf_page": 60, "printed_page": "42", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "For discrete random variables $X_{[1]}$, $X_{[2]}$, \\ldots, $X_{[K]}$, and $Y$ with joint PMF $f$, the \\emph{conditional PMF} of $Y$ given $\\mathbf{X}=\\mathbf{x}$ is\n\\[\nf_{Y|\\mathbf{X}}(y|\\mathbf{x})=\\Pr[Y=y|\\mathbf{X}=\\mathbf{x}]=\\frac{\\Pr[Y=y,\\mathbf{X}=\\mathbf{x}]}{\\Pr[\\mathbf{X}=\\mathbf{x}]}=\\frac{f(\\mathbf{x},y)}{f_{\\mathbf{X}}(\\mathbf{x})},\n\\]\n$\\forall y\\in\\mathbb{R}$ and $\\forall\\mathbf{x}\\in\\operatorname{Supp}[\\hl{X}]$.\n\nFor jointly continuous random variables $X_{[1]}$, $X_{[2]}$, \\ldots, $X_{[K]}$, and $Y$ with joint PDF $f$, the \\emph{conditional PDF} of $Y$ given $\\mathbf{X}=\\mathbf{x}$ is\n\\[\nf_{Y|\\mathbf{X}}(y|\\mathbf{x})=\\frac{f(\\mathbf{x},y)}{f_{\\mathbf{X}}(\\mathbf{x})},\\ \\forall y\\in\\mathbb{R}\\text{ and }\\forall\\mathbf{x}\\in\\operatorname{Supp}[\\hl{X}].\n\\]", "corrected": "In both displays, replace $\\operatorname{Supp}[X]$ with $\\operatorname{Supp}[\\hl{\\mathbf{X}}]$. An optional gloss is $\\operatorname{Supp}[\\mathbf{X}]=\\{\\mathbf{x}\\in\\mathbb{R}^K:f_{\\mathbf{X}}(\\mathbf{x})>0\\}$. \\hl{For the continuous formula, use the slice integral as the denominator where it is positive and finite, and choose one fixed PDF at the remaining conditioning values, which have probability zero.}", "evidence": "The evidence is a check of the page: in both displays the $\\mathbf{x}$ is bold and the $X$ inside the brackets is italic, while p.~40 reserves boldface letters for vectors.", "problem": "The conditioning object is the random vector $\\mathbf{X}=(X_{[1]},X_{[2]},\\ldots,X_{[K]})$, which is bold everywhere else in the definition, and no scalar $X$ appears in it. The set that is meant is the support of $\\mathbf{X}$, the set where its marginal PMF/PDF is positive.", "what_survives": "The typeface change identifies the intended vector support: Section 1.4 extends the constructions for one variable to vectors. The continuous formula also needs the qualification about exceptional conditioning values in \\fid{P001-C01-N007}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C01-N009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the box of Theorem 2.2.8, Implications of Independence (Part II), with its four bullets. A red box marks the third bullet, 'Correlation is zero: rho[X, Y] = 0.'", "caption": "Theorem 2.2.8, p. 64 (PDF p. 82). The marked bullet is stated for all independent $X$ and $Y$; the correlation in it is defined only when both variances are positive.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-A002.png"}, {"alt": "Crop of the box of Definition 2.2.5, Correlation, which defines rho[X, Y] as Cov[X, Y] divided by sigma[X] sigma[Y]. Red boxes mark the condition 'with sigma[X] > 0 and sigma[Y] > 0'. The paragraph below the box ends: '(Note that, unlike covariance, correlation is undefined when the variance of either variable is zero.)'", "caption": "Definition 2.2.5 and the paragraph after it, p. 62 (PDF p. 80). The marked condition is the domain on which the third bullet of Theorem 2.2.8 is read; the paragraph ends by saying that correlation is undefined when either variance is zero.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-A002-def.png"}], "formal_release": null, "formal_statements": [], "headline": "The third bullet of Theorem 2.2.8 holds when the correlation is defined, that is, when both variables have positive variance.", "id": "P001-C02-A002", "location": "Theorem 2.2.8, third bullet, p. 64 (PDF p. 82), with the last line of its proof on p. 65 (PDF p. 83); Definition 2.2.5, p. 62 (PDF p. 80)", "pdf_page": 82, "printed_page": "64", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\textbf{Theorem 2.2.8.} \\emph{Implications of Independence (Part II)}\n\nIf $X$ and $Y$ are independent random variables, then\n\\begin{itemize}\n\\item $\\mathrm{E}[XY]=\\mathrm{E}[X]\\mathrm{E}[Y]$.\n\\item Covariance is zero: $\\operatorname{Cov}[X,Y]=0$.\n\\item \\hl{Correlation is zero}: $\\rho[X,Y]=0$.\n\\item Variances are additive: $\\mathrm{V}[X+Y]=\\mathrm{V}[X]+\\mathrm{V}[Y]$.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Correlation, \\hl{when defined,} is zero: $\\rho[X,Y]=0$.\n\\end{itemize}\nThe correlation line of the proof on p.~65 applies when $\\sigma[X]>0$ and $\\sigma[Y]>0$, and Theorems 2.2.6 and 2.2.7 likewise use correlation on the domain of Definition 2.2.5.", "intended_meaning": "Definition 2.2.5 (p.~62) defines $\\rho[X,Y]$ only when $\\sigma[X]>0$ and $\\sigma[Y]>0$. The third bullet is read on that domain, where it is correct, since $\\operatorname{Cov}[X,Y]=0$ gives $\\rho[X,Y]=0$. Independence alone does not place $X$ and $Y$ in that domain. If $X$ is the constant $3$ and $Y$ is a fair die roll, then $X$ and $Y$ are independent and the other three bullets hold, but $\\mathrm{V}[X]=0$ and $\\rho[X,Y]$ is undefined."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-A002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Screenshot of the last line of footnote 20 and the four lines of footnote 21 at the foot of page 78. Two red boxes enclose the words: the BLP is the only linear function that satisfies, and, on the next line, the equation Cov[X, g(X)] = Cov[X, Y].", "caption": "Footnote 21 on p. 78 (PDF p. 96). The red boxes mark the claim that the BLP is the only linear function satisfying the covariance equation.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T44", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T44"}, {"id": "P001-T45", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T45"}], "headline": "The best linear predictor is the only linear function $g$ that satisfies both $\\operatorname{E}[g(X)]=\\operatorname{E}[Y]$ and $\\operatorname{Cov}[X,g(X)]=\\operatorname{Cov}[X,Y]$; the covariance equation alone fixes the slope and leaves the intercept free.", "historical_evidence_type": "lean_proof", "id": "P001-C02-E001", "location": "Footnote 21, p. 78 (PDF p. 96), first sentence", "pdf_page": 96, "printed_page": "78", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T44", "P001-T45"], "text": {"as_printed": "The BLP also demonstrates that covariance and correlation indeed measure the \\emph{linear} relationship between $X$ and $Y$ (as in Section 2.2.2): the BLP is \\hl{the only} linear function that satisfies $\\operatorname{Cov}[X,g(X)]=\\operatorname{Cov}[X,Y]$.", "corrected": "The BLP also demonstrates that covariance and correlation indeed measure the \\emph{linear} relationship between $X$ and $Y$ (as in Section 2.2.2): under the condition $\\operatorname{V}[X]>0$ of Theorem 2.2.21, the BLP is the only linear function $g$ satisfying \\hl{both} $\\hl{\\operatorname{E}[g(X)]=\\operatorname{E}[Y]}$ \\hl{and} $\\operatorname{Cov}[X,g(X)]=\\operatorname{Cov}[X,Y]$.", "evidence": "The evidence is an exact counterexample to the printed sentence and a derivation of the corrected one. Let $X$ and $Y$ have finite second moments, and write $\\mu=\\operatorname{E}[X]$, $\\nu=\\operatorname{E}[Y]$, $v=\\operatorname{V}[X]>0$ and $k=\\operatorname{Cov}[X,Y]$. For all real $a$ and $b$, linearity of expectation gives\n\\[\n\\operatorname{Cov}[X,a+bX]=a\\mu+b\\operatorname{E}[X^2]-\\mu(a+b\\mu)=bv .\n\\]\nThe covariance equation is therefore equivalent to $b=k/v$, and it puts no condition on $a$. The mean equation $\\operatorname{E}[g(X)]=\\operatorname{E}[Y]$ reads $a+b\\mu=\\nu$. Combining the two equations, I obtain\n\\[\nb=\\frac{k}{v},\\qquad a=\\nu-\\frac{k}{v}\\mu ,\n\\]\nwhich are the coefficients $\\beta$ and $\\alpha$ of Theorem 2.2.21. Next, set $\\epsilon=Y-\\alpha-\\beta X$, $u=a-\\alpha$ and $t=b-\\beta$. Then $\\operatorname{E}[\\epsilon]=0$ and $\\operatorname{E}[X\\epsilon]=0$. Expanding the square, I find\n\\[\n\\operatorname{E}\\bigl[(Y-a-bX)^2\\bigr]=\\operatorname{E}[\\epsilon^2]+(u+t\\mu)^2+t^2v .\n\\]\nSince $v>0$, the last two terms vanish only when $t=0$ and $u=0$. This implies that $\\alpha+\\beta X$ is the unique linear function of minimum MSE. A function that satisfies the covariance equation has $t=0$, and its MSE exceeds the minimum by $u^2$.\n\nThe table lists two counterexamples to the printed sentence. In the first, $X=Y$ takes the values $-1$ and $1$ with probability $1/2$ each, so that $\\operatorname{E}[Y]=0$ and $\\operatorname{V}[X]=\\operatorname{Cov}[X,Y]=1$; the function $x$ has mean $0$ and the function $x+1$ has mean $1$. This counterexample is formalized as a Lean proof. The second is the book's Example 2.2.23, where $X\\sim U(0,1)$ and $Y=10X^2+W$ with $W\\sim N(0,1)$ independent of $X$, so that $\\operatorname{V}[X]=1/12$ and $\\operatorname{Cov}[X,Y]=5/6$; its entries are computed in exact arithmetic. In each case both functions satisfy the covariance equation, and only the first is the BLP.\n\n\\begin{audittable}{llrr}{Two linear functions with the same covariance with $X$ and different mean squared errors, in each of two settings.}\nSetting & $g(x)$ & $\\operatorname{Cov}[X,g(X)]$ & MSE \\\\\n$X=Y=\\pm1$ & $x$ (the BLP) & $1$ & $0$ \\\\\n$X=Y=\\pm1$ & $x+1$ & $1$ & $1$ \\\\\nExample 2.2.23 & $-5/3+10x$ (the BLP) & $5/6$ & $14/9$ \\\\\nExample 2.2.23 & $10x$ & $5/6$ & $13/3$ \\\\\n\\end{audittable}", "problem": "In the book a linear function has the form $g(X)=a+bX$, with a free intercept (p.~76). The footnote presents the covariance equation as identifying both coefficients of the BLP, but the equation determines the slope and leaves the intercept free. For all real $a$ and $b$, I have $\\operatorname{Cov}[X,a+bX]=b\\operatorname{V}[X]$. The covariance equation therefore holds exactly when $b=\\operatorname{Cov}[X,Y]/\\operatorname{V}[X]$, whatever the value of $a$. Infinitely many linear functions satisfy it, and one of them is the BLP. Each of the others differs from the BLP by a nonzero constant $u$, and its mean squared error (MSE) exceeds that of the BLP by $u^2$. The equation that fixes the intercept is $\\operatorname{E}[g(X)]=\\operatorname{E}[Y]$. The proof of Theorem 2.2.21 uses this equation, and the footnote omits it.", "what_survives": "Theorem 2.2.21 with its intercept and slope formulas (pp.~76--78), the properties of deviations from the BLP in Theorem 2.2.22 (p.~79), the BLP $-5/3+10X$ of Example 2.2.23 (pp.~79--81) and the regression coefficient formulas of Section 4.1.1 (p.~144) are unaffected. The second sentence of footnote 21 needs the almost-sure qualification in \\fid{P001-C02-E004}. A search of the book's text finds no later passage that uses the covariance equation alone to characterize the BLP."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Figure 2.1.1, 'Minimum MSE Solution for a Fair Coin Flip'. A solid U-shaped curve is plotted against 'Choice of c' from -2 to 2. It starts at a height of about 13 at c = -2, reaches its lowest point, at a height of about 0.5, where a dashed vertical line marks c = 0.5, and ends at a height of about 5 at c = 2. A red box marks the vertical axis: the label MSE and the tick labels 0, 2, 4, 6, 8, 10, 12.", "caption": "Figure 2.1.1, p. 59 (PDF p. 77). The marked axis is labelled MSE, but its scale is twice the mean squared error of Example 2.1.25: the curve bottoms out at about $0.5$, where the mean squared error is $0.25$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T40", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T40"}], "headline": "The curve in Figure 2.1.1 approximates twice the mean squared error named on its vertical axis; the tick labels of that axis should be halved, which puts the minimum at $1/4$ for $c=1/2$.", "id": "P001-C02-E002", "location": "Figure 2.1.1, p. 59 (PDF p. 77), vertical axis and solid curve", "pdf_page": 77, "printed_page": "59", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T40"], "text": {"as_printed": "The figure, described: the vertical axis is labelled \\hl{MSE} and has the tick labels \\hl{0, 2, 4, 6, 8, 10, 12}. The horizontal axis is labelled ``Choice of c'' and has the tick labels $-2$, $-1$, $0$, $1$, $2$. The solid curve starts near $(-2,13)$, has its minimum near $(1/2,1/2)$ and ends near $(2,5)$, and a dashed vertical line marks $c=1/2$. These coordinates describe the drawn curve and are not printed on the figure. The caption reads:\n\nFIGURE 2.1.1. \\emph{Minimum MSE Solution for a Fair Coin Flip}\n\nExample 2.1.25, p.~58, for the fair coin flip $X\\in\\{0,1\\}$:\n\\[\n\\begin{aligned}\n\\mathrm{E}\\big[(X-c)^2\\big]&=\\frac12(0-c)^2+\\frac12(1-c)^2\\\\\n&=\\frac12(c^2+1+c^2-2c)\\\\\n&=\\frac12(1+2c^2-2c)\\\\\n&=\\frac12+c^2-c.\n\\end{aligned}\n\\]", "corrected": "In Figure 2.1.1 the vertical scale is twice the stated mean squared error. Halve the tick labels of the vertical axis, so that 0, 2, 4, 6, 8, 10, 12 become \\hl{0, 1, 2, 3, 4, 5, 6}, or redraw the curve with its heights divided by two. For the fair coin flip $X\\in\\{0,1\\}$ of Example 2.1.25,\n\\[\n\\mathrm{E}\\big[(X-c)^2\\big]=c^2-c+\\frac12=\\Big(c-\\frac12\\Big)^2+\\frac14 ,\n\\]\nwhose minimum is $\\hl{1/4}$ at $c=1/2$. The calculation in the example and the minimizing choice are unchanged. A repair of the label alone is also possible: keep the drawing and label the vertical axis $2\\times\\mathrm{MSE}$.", "evidence": "The evidence is an exact counterexample to the printed scale: the mean squared error of Example 2.1.25, computed exactly, set against the coordinates of the published curve. I took the coordinates of the curve and of the tick marks from the drawing commands of the published PDF and calibrated both axes on the printed tick labels.\n\\begin{audittable}{rrr}{Height of the curve in Figure 2.1.1 on the printed scale, and the mean squared error of Example 2.1.25.}\n$c$ & Height of the printed curve & $\\mathrm{E}[(X-c)^2]=c^2-c+1/2$ \\\\\n\\hline\n$-2$ & $13.00002$ & $13/2$ \\\\\n$1/2$ & $0.50010$ & $1/4$ \\\\\n$2$ & $5.00002$ & $5/2$ \\\\\n\\end{audittable}\nEvery point of the centre line of the drawn curve is within $0.011$ of $2(c^2-c+1/2)$ on the printed scale. Half the width of the stroke is about $0.023$ on that scale, and neither the thickness of the line nor rounding accounts for the factor of two.", "problem": "The figure illustrates Example 2.1.25, in which $X$ is a fair coin flip and the mean squared error about $c$ is $\\mathrm{E}[(X-c)^2]=c^2-c+1/2$ (Definition 2.1.22). This function equals $13/2$ at $c=-2$ and $5/2$ at $c=2$, and its minimum is $1/4$ at $c=1/2$. The printed curve has heights of about $13$, $5$ and $1/2$ at these points. On the printed scale it approximates twice the mean squared error across the plotted range. The unweighted sum $(0-c)^2+(1-c)^2=2\\,\\mathrm{E}[(X-c)^2]$ equals $13$, $5$ and $1/2$ at these points. The code that drew the figure is not available, and the cause of the factor is not established.", "what_survives": "Example 2.1.25, its solution $c=1/2$, Definition 2.1.22 and Theorems 2.1.23 and 2.1.24 do not depend on the figure. The figure still shows the minimizing choice, since doubling a function does not move its minimizer, and the dashed line stays at $c=1/2$. Only the heights read from the figure change. A search of the text finds the figure cited in the List of Figures, in the sentence on p.~58 that introduces it and in its caption, with no later reference."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Screenshot of the seven lines of Example 2.2.24, BLP under Nonlinearity. A red box encloses the term X squared in the equation E[Y|X] = X squared on the third line of the example.", "caption": "Example 2.2.24 on p. 82 (PDF p. 100). The red box marks the right-hand side $X^2$ of the conditional expectation.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T46", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T46"}], "headline": "The conditional expectation function in Example 2.2.24 is $10X^2$, not $X^2$.", "id": "P001-C02-E003", "location": "Example 2.2.24, p. 82 (PDF p. 100), second sentence", "pdf_page": 100, "printed_page": "82", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T46"], "text": {"as_printed": "Suppose that, as in Example 2.2.23, $Y=10X^2+W$, $W\\sim N(0,1)$ and $X\\perp\\!\\!\\!\\perp W$. If $X$ is uniformly distributed between zero and one, the CEF of $Y$ given $X$ is the same as above: $\\operatorname{E}[Y\\mid X]=\\hl{X^2}$.", "corrected": "Suppose that, as in Example 2.2.23, $Y=10X^2+W$, $W\\sim N(0,1)$ and $X\\perp\\!\\!\\!\\perp W$. If $X$ is uniformly distributed between zero and one, the CEF of $Y$ given $X$ is the same as above: $\\operatorname{E}[Y\\mid X]=\\hl{10X^2}$.", "evidence": "The evidence is a derivation. Since $W$ is independent of $X$ and $\\operatorname{E}[W]=0$, I have\n\\[\n\\operatorname{E}[Y\\mid X]=10X^2+\\operatorname{E}[W\\mid X]=10X^2 .\n\\]\nAt $X=1/2$ the CEF equals $5/2$ and the printed formula equals $1/4$.", "problem": "The conditional expectation drops the factor of ten in the stated outcome equation. Example 2.2.23 derives $\\operatorname{E}[Y\\mid X]=10X^2$ for the same model on p.~79, and the words ``the same as above'' refer to that formula.", "what_survives": "Everything else is unaffected. The BLP $-5/3+10X$ derived on pp.~80--81 and the black curve of Figures 2.2.1 and 2.2.2 use $10X^2$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the box of Theorem 2.2.6, Correlation and Linear Dependence, with its three bullets. Red boxes mark the equality Y = a + bX at the end of the second bullet and the equality Y = a - bX at the end of the third bullet. Below the box, the sentence 'We omit the proof of this theorem.'", "caption": "Theorem 2.2.6, p. 63 (PDF p. 81). The two marked equalities are stated without “almost surely”; by Definition 1.3.1 of the book they then hold at every outcome.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T41", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T41"}, {"id": "P001-T42", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T42"}], "headline": "Perfect correlation implies the affine relation only with probability one: in the second and third bullets of Theorem 2.2.6, $Y=a+bX$ and $Y=a-bX$ should hold almost surely.", "id": "P001-C02-E004", "location": "Theorem 2.2.6, p. 63 (PDF p. 81), second and third bullets", "pdf_page": 81, "printed_page": "63", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T41", "P001-T42"], "text": {"as_printed": "\\textbf{Theorem 2.2.6.} \\emph{Correlation and Linear Dependence}\n\nFor random variable $X$ and $Y$,\n\\begin{itemize}\n\\item $\\rho[X,Y]\\in[-1,1]$.\n\\item $\\rho[X,Y]=1\\Longleftrightarrow\\exists a,b\\in\\mathbb{R}$ with $b>0$ such that $\\hl{Y=a+bX}$.\n\\item $\\rho[X,Y]=-1\\Longleftrightarrow\\exists a,b\\in\\mathbb{R}$ with $b>0$ such that $\\hl{Y=a-bX}$.\n\\end{itemize}", "corrected": "\\textbf{Theorem 2.2.6.} \\emph{Correlation and Linear Dependence}\n\nFor random \\hl{variables} $X$ and $Y$,\n\\begin{itemize}\n\\item $\\rho[X,Y]\\in[-1,1]$.\n\\item $\\rho[X,Y]=1\\Longleftrightarrow\\exists a,b\\in\\mathbb{R}$ with $b>0$ such that $Y=a+bX$ \\hl{almost surely}.\n\\item $\\rho[X,Y]=-1\\Longleftrightarrow\\exists a,b\\in\\mathbb{R}$ with $b>0$ such that $Y=a-bX$ \\hl{almost surely}.\n\\end{itemize}\nHere $X$ and $Y$ have finite, strictly positive variances, as Definition 2.2.5 requires for $\\rho[X,Y]$ to be defined, and ``$Y=a+bX$ almost surely'' means that $\\Pr[Y=a+bX]=1$. The plural in the opening line is the correction of \\fid{P001-C02-N003}. The same two words are needed where the statement is repeated: in the description of perfect positive and negative linear dependence on p.~62, and in the last sentence of footnote 21 on p.~78.", "evidence": "The evidence is an exact counterexample for each sign, with a derivation of the corrected statement. Let $\\Omega=\\{u,v,n\\}$ with the probabilities and random variables below.\n\\begin{audittable}{lrrrr}{The counterexample: three outcomes, one of probability zero.}\nOutcome & Probability & $X$ & $Y_+$ & $Y_-$ \\\\\n\\hline\n$u$ & $1/2$ & $0$ & $0$ & $1$ \\\\\n$v$ & $1/2$ & $1$ & $1$ & $0$ \\\\\n$n$ & $0$ & $0$ & $1$ & $0$ \\\\\n\\end{audittable}\nEach of $X$, $Y_+$ and $Y_-$ takes the values $0$ and $1$ with probability $1/2$; each has mean $1/2$ and variance $1/4$, and its range equals its support $\\{0,1\\}$. A direct computation gives $\\mathrm{E}[XY_+]=1/2$, $\\operatorname{Cov}[X,Y_+]=1/4$ and $\\rho[X,Y_+]=1$. If $Y_+=a+bX$ at every outcome, then from the outcomes $u$ and $v$ I obtain $a=0$ and $b=1$, and the outcome $n$ requires $1=0$. Similarly, I have $\\mathrm{E}[XY_-]=0$, $\\operatorname{Cov}[X,Y_-]=-1/4$ and $\\rho[X,Y_-]=-1$. If $Y_-=a-bX$ at every outcome, then $u$ and $v$ give $a=1$ and $b=1$, and $n$ requires $0=1$. Both relations hold with probability one.\n\nFor the corrected statement, let $X$ and $Y$ have finite second moments with $\\sigma[X]>0$ and $\\sigma[Y]>0$, and set $A=(X-\\mathrm{E}[X])/\\sigma[X]$ and $B=(Y-\\mathrm{E}[Y])/\\sigma[Y]$. Then\n\\[\n\\mathrm{E}\\big[(A-B)^2\\big]=2-2\\rho[X,Y],\\qquad \\mathrm{E}\\big[(A+B)^2\\big]=2+2\\rho[X,Y].\n\\]\nBoth are nonnegative, and the first bullet follows. If $\\rho[X,Y]=1$, then $(A-B)^2$ is a nonnegative random variable with expectation zero. I conclude that $A=B$ with probability one, and\n\\[\nY=a+bX\\ \\text{almost surely},\\qquad b=\\frac{\\sigma[Y]}{\\sigma[X]}>0,\\quad a=\\mathrm{E}[Y]-b\\,\\mathrm{E}[X].\n\\]\nIf $\\rho[X,Y]=-1$, I apply the same argument to $(A+B)^2$ and find that $Y=a-bX$ almost surely with $b=\\sigma[Y]/\\sigma[X]$ and $a=\\mathrm{E}[Y]+b\\,\\mathrm{E}[X]$. Conversely, if $Y=a\\pm bX$ almost surely with $b>0$, then $\\mathrm{V}[Y]=b^2\\,\\mathrm{V}[X]$ and $\\operatorname{Cov}[X,Y]=\\pm b\\,\\mathrm{V}[X]$, which implies that $\\rho[X,Y]=\\pm1$.", "problem": "The book defines equality of random variables at every outcome: by Definition 1.3.1 (p.~31), $X=Y$ means $X(\\omega)=Y(\\omega)$ for all $\\omega\\in\\Omega$, and footnote 28 on that page distinguishes this from $\\Pr[X=Y]=1$, which it calls equality almost surely. With this definition, the second and third bullets say that $\\rho[X,Y]=\\pm1$ forces $Y=a\\pm bX$ at every outcome. Under perfect correlation the relation holds only with probability one, and $Y$ may differ from $a\\pm bX$ on an event of probability zero. The first bullet and the implications from the affine relation to $\\rho[X,Y]=\\pm1$ are correct.", "what_survives": "The bound $\\rho[X,Y]\\in[-1,1]$ and the implications from the affine relation to $\\rho[X,Y]=\\pm1$ are unaffected, and both equivalences hold in the corrected form. The conclusion drawn from $\\rho[X,Y]=\\pm1$ weakens from equality at every outcome to equality with probability one, and this changes no distribution, no moment and no expected prediction loss. Theorem 2.2.7, the derivation of the best linear predictor on pp.~76--78 and the variance calculation of Example 3.5.8 on pp.~140--141 rest on moment formulas and do not use equality at every outcome."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the box of Theorem 2.1.21, Properties of the Normal Distribution, whose first bullet reads: for all a, b in R with a not equal to 0, if W = aX + b, then W is normal. Below the box, the sentence 'We omit the proof. This theorem has an interesting and immediate implication: any linear combination of any number of mutually independent normal random variables must itself be normal.' A red box marks the words 'any linear combination'.", "caption": "Theorem 2.1.21 and the sentence after it, p. 56 (PDF p. 74). The marked words admit the combination whose coefficients are all zero, which is a constant; the theorem above them requires $a\\neq0$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E005.png"}, {"alt": "Crop of footnote 13, which gives the joint PDF of the bivariate normal distribution. Three red boxes mark: the expression rho(X, Y) in the first line, written with parentheses; the symbol mu with a lower-case subscript y in the last term of the exponent; and the range 'minus 1 less than or equal to rho less than or equal to 1' in the line after the formula. The last line of the footnote writes the correlation as rho[X, Y], with square brackets.", "caption": "Footnote 13, p. 66 (PDF p. 84). The marked range includes $\\rho=\\pm1$, where the displayed density is undefined; the other two marks are the correlation written with parentheses and the lower-case subscript in $\\mu_y$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E005-fn13.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T38", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T38"}, {"id": "P001-T39", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T39"}], "headline": "The linear combination in the sentence after Theorem 2.1.21 must have a coefficient that is not zero, and the bivariate normal density of footnote 13 requires $-1<\\rho<1$; the footnote also has two slips of notation.", "id": "P001-C02-E005", "location": "Sentence after Theorem 2.1.21, p. 56 (PDF p. 74); footnote 13, p. 66 (PDF p. 84)", "pdf_page": 74, "printed_page": "56", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T38", "P001-T39"], "text": {"as_printed": "Sentence after Theorem 2.1.21, p.~56:\n\nThis theorem has an interesting and immediate implication: \\hl{any linear combination} of any number of mutually independent normal random variables must itself be normal.\n\nFootnote 13, p.~66:\n\nWhile $\\hl{\\rho(X,Y)}=0$ does not generally imply $X\\perp\\!\\!\\!\\perp Y$ even when $X$ and $Y$ are both normal, it does in the special case where the joint distribution of $X$ and $Y$ is \\emph{bivariate normal}, that is, $X$ and $Y$ have the joint PDF\n\\[\nf(x,y)=\\frac{1}{2\\pi\\sigma_X\\sigma_Y\\sqrt{1-\\rho^2}}\\,e^{-\\frac{1}{2(1-\\rho^2)}\\left[\\frac{(x-\\mu_X)^2}{\\sigma_X^2}+\\frac{(y-\\mu_Y)^2}{\\sigma_Y^2}-\\frac{2\\rho(x-\\mu_X)(y-\\hl{\\mu_y})}{\\sigma_X\\sigma_Y}\\right]}\n\\]\nfor some $\\mu_X,\\mu_Y,\\sigma_X,\\sigma_Y,\\rho\\in\\mathbb{R}$ with $\\sigma_X>0$, $\\sigma_Y>0$, and $\\hl{-1\\le\\rho\\le1}$.", "corrected": "Sentence after Theorem 2.1.21, p.~56:\n\nThis theorem has an interesting and immediate implication: any \\hl{nontrivial} linear combination of any number of mutually independent normal random variables must itself be normal.\n\nHere a linear combination $\\sum_i c_iX_i$ is nontrivial if at least one coefficient $c_i$ is not zero.\n\nFootnote 13, p.~66:\n\nWhile $\\hl{\\rho[X,Y]}=0$ does not generally imply $X\\perp\\!\\!\\!\\perp Y$ even when $X$ and $Y$ are both normal, it does in the special case where the joint distribution of $X$ and $Y$ is \\emph{bivariate normal}, that is, $X$ and $Y$ have the joint PDF\n\\[\nf(x,y)=\\frac{1}{2\\pi\\sigma_X\\sigma_Y\\sqrt{1-\\rho^2}}\\,e^{-\\frac{1}{2(1-\\rho^2)}\\left[\\frac{(x-\\mu_X)^2}{\\sigma_X^2}+\\frac{(y-\\mu_Y)^2}{\\sigma_Y^2}-\\frac{2\\rho(x-\\mu_X)(y-\\hl{\\mu_Y})}{\\sigma_X\\sigma_Y}\\right]}\n\\]\nfor some $\\mu_X,\\mu_Y,\\sigma_X,\\sigma_Y,\\rho\\in\\mathbb{R}$ with $\\sigma_X>0$, $\\sigma_Y>0$, and $\\hl{-1<\\rho<1}$.\n\nAt $\\rho=\\pm1$ the joint distribution is singular and has no two-dimensional PDF.", "evidence": "The evidence is an exact counterexample for each passage, with a short derivation. Let $X_1,\\dots,X_n$ be mutually independent normal random variables with means $\\mu_1,\\dots,\\mu_n$ and variances $\\sigma_1^2,\\dots,\\sigma_n^2>0$, and let $c_1,\\dots,c_n\\in\\mathbb{R}$. Then\n\\[\n\\mathrm{V}\\Big[\\sum_{i=1}^n c_iX_i\\Big]=\\sum_{i=1}^n c_i^2\\sigma_i^2 .\n\\]\nThis variance is zero when every $c_i$ is zero, and the combination is then the constant $0$. Otherwise it is positive, and the combination is normal, since its characteristic function is $\\exp\\big(it\\sum_i c_i\\mu_i-t^2\\sum_i c_i^2\\sigma_i^2/2\\big)$.\n\nFor the footnote, let $X$ and $Y$ have standard deviations $\\sigma_X,\\sigma_Y>0$ and correlation $\\rho$, and set $A=(X-\\mu_X)/\\sigma_X$ and $B=(Y-\\mu_Y)/\\sigma_Y$. Then\n\\[\n\\mathrm{E}\\big[(A-B)^2\\big]=2-2\\rho,\\qquad \\mathrm{E}\\big[(A+B)^2\\big]=2+2\\rho .\n\\]\nAt $\\rho=1$ the first expectation is zero, and I have $A=B$ with probability one. At $\\rho=-1$ the second is zero, and I have $A=-B$ with probability one. In both cases the joint distribution is concentrated on a line and has no two-dimensional density.", "problem": "Since Definition 2.1.19 (p.~55) requires $\\sigma>0$, a constant is not a normal random variable. The linear combination with every coefficient equal to zero is the constant $0$, and the sentence on p.~56 includes it, although Theorem 2.1.21 itself requires $a\\neq0$. In footnote 13 the range $-1\\le\\rho\\le1$ includes $\\rho=\\pm1$, where $\\sqrt{1-\\rho^2}=0$ and the factor $1/(2(1-\\rho^2))$ is undefined, so that the displayed formula defines no density. The same footnote writes $\\rho(X,Y)$ where the book's notation is $\\rho[X,Y]$, and $\\mu_y$ where $\\mu_Y$ is meant.", "what_survives": "Theorems 2.1.20 and 2.1.21, the density of footnote 13 for $-1<\\rho<1$, and the footnote's statement that $\\rho[X,Y]=0$ implies independence for a bivariate normal pair are unaffected. Only the exclusions of degenerate distributions and the two symbols change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of the last two bullets of Theorem 2.2.25. The bullet reads: If g is a function of X and h is a function of Y, then. A red box encloses the first sub-bullet, E[g(Y) | h(X)] = E[g(Y)]. The second sub-bullet reads: The BLP of h(Y) given g(X) is E[h(Y)].", "caption": "Theorem 2.2.25, continued on p. 83 (PDF p. 101). The red box marks the first sub-bullet, where $g$ is applied to $Y$ and $h$ to $X$; the sub-bullet below it uses $h(Y)$ and $g(X)$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T47", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T47"}], "headline": "In the last part of Theorem 2.2.25 the names of the two functions are interchanged: the identity is $\\operatorname{E}[h(Y)\\mid g(X)]=\\operatorname{E}[h(Y)]$.", "id": "P001-C02-E006", "location": "Theorem 2.2.25, p. 83 (PDF p. 101), fourth bullet, first sub-bullet", "pdf_page": 101, "printed_page": "83", "reaches": [], "related_entries": [], "revisions": [{"classification_after": {"class": "error", "severity": "Minor"}, "classification_before": {"class": "error", "severity": "Minor"}, "note": "The measurability scope was revised; the classification is unchanged (error, Minor). How this came to light: the Lean work showed that measurability of the composed variables on an arbitrary sample space alone does not ensure that transformations preserve independence. Why it changed: the corrected identity needs a qualification on the transformations under the laws of the variables to which they are applied. The wording now requires agreement almost surely with Borel measurable functions under those laws, retaining measurable compositions and the required moments. Global Borel measurability is a sufficient special case. The function-name correction is unchanged. P. M. Aronow delegated this decision to the model session that runs the audit, which adopted the amendment after two checks; this note was written by that session."}], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T47"], "text": {"as_printed": "If $X$ and $Y$ are independent random variables, then [\\ldots] If $g$ is a function of $X$ and $h$ is a function of $Y$, then $\\operatorname{E}\\bigl[\\hl{g(Y)}\\bigm|\\hl{h(X)}\\bigr]=\\operatorname{E}\\bigl[\\hl{g(Y)}\\bigr]$. The BLP of $h(Y)$ given $g(X)$ is $\\operatorname{E}\\bigl[h(Y)\\bigr]$.", "corrected": "If $X$ and $Y$ are independent random variables, $g$ and $h$ agree almost surely under the respective laws of $X$ and $Y$ with Borel measurable functions, $g(X)$ and $h(Y)$ are random variables, and the displayed moments exist, then $\\operatorname{E}\\bigl[\\hl{h(Y)}\\bigm|\\hl{g(X)}\\bigr]=\\operatorname{E}\\bigl[\\hl{h(Y)}\\bigr]$, and the BLP of $h(Y)$ given $g(X)$ is $\\operatorname{E}\\bigl[h(Y)\\bigr]$.", "evidence": "The evidence is an exact counterexample to the printed expression. Let $X$ be uniform on $\\{0,1\\}$ and $Y$ uniform on $\\{2,3\\}$, with $X$ and $Y$ independent, and let $g$ and $h$ be the identity functions defined on $\\{0,1\\}$ and on $\\{2,3\\}$. Here $g$ is a function of $X$ and $h$ is a function of $Y$, but $g(Y)$ and $h(X)$ are undefined at every outcome. The corrected expression is defined, and $\\operatorname{E}[h(Y)\\mid g(X)]=5/2=\\operatorname{E}[h(Y)]$. If $g$ and $h$ agree almost surely under the respective laws of $X$ and $Y$ with Borel measurable functions, $g(X)$ and $h(Y)$ are random variables, and the displayed moments exist, then $g(X)$ and $h(Y)$ are independent when $X$ and $Y$ are, and the conditional mean of $h(Y)$ given $g(X)$ equals its mean.", "problem": "The theorem declares $g$ a function of $X$ and $h$ a function of $Y$, and the first sub-bullet then applies $g$ to $Y$ and $h$ to $X$. By Definition 1.2.2 (p.~16), a function of $X$ need only be defined on a set that contains the range of $X$, and hence $g(Y)$ and $h(X)$ need not be defined. The next sub-bullet uses $h(Y)$ and $g(X)$, as declared. When $g(Y)$ and $h(X)$ are random variables with the required moments and $g$ and $h$ agree almost surely with Borel measurable functions under the laws of $Y$ and $X$, respectively, the printed identity is true, and the error lies in the names.", "what_survives": "The conclusions of Theorem 2.2.25 as intended and the BLP sub-bullet that follows are unaffected. The printed identity also holds whenever $g(Y)$ and $h(X)$ are random variables with the required moments and $g$ and $h$ agree almost surely with Borel measurable functions under the laws of $Y$ and $X$, respectively."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the sentence 'By linearity of expectations and the chain rule, as well as the fact that derivatives pass through expectations (because they pass through sums and integrals), this becomes:' followed by two displayed lines in which the derivative of E[epsilon squared] is moved inside the expectation, ending in minus 2 E[epsilon] and minus 2 E[epsilon X]. Red boxes mark the words from 'derivatives' to 'integrals)'.", "caption": "Proof of Theorem 2.2.21, p. 77 (PDF p. 95). The marked words justify moving each derivative inside the expectation in the display below them.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E007.png"}], "formal_release": null, "formal_statements": [], "headline": "The remark that derivatives pass through expectations is not a general rule; the step holds here because $\\mathrm{E}[(Y-a-bX)^2]$ is a quadratic polynomial in $(a,b)$ with finite coefficients.", "id": "P001-C02-E007", "location": "Proof of Theorem 2.2.21, p. 77 (PDF p. 95), the sentence before the displayed normal equations", "pdf_page": 95, "printed_page": "77", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "By linearity of expectations and the chain rule, as well as the fact that \\hl{derivatives pass through expectations (because they pass through sums and integrals)}, this becomes:\n\\[\n\\begin{aligned}\n0&=\\frac{\\partial\\mathrm{E}[\\varepsilon^2]}{\\partial\\alpha}=\\mathrm{E}\\left[\\frac{\\partial\\varepsilon^2}{\\partial\\alpha}\\right]=\\mathrm{E}\\left[2\\varepsilon\\frac{\\partial\\varepsilon}{\\partial\\alpha}\\right]=-2\\mathrm{E}[\\varepsilon],\\\\\n0&=\\frac{\\partial\\mathrm{E}[\\varepsilon^2]}{\\partial\\beta}=\\mathrm{E}\\left[\\frac{\\partial\\varepsilon^2}{\\partial\\beta}\\right]=\\mathrm{E}\\left[2\\varepsilon\\frac{\\partial\\varepsilon}{\\partial\\beta}\\right]=-2\\mathrm{E}[\\varepsilon X].\n\\end{aligned}\n\\]", "corrected": "\\hl{Expanding the finite quadratic $\\mathrm{E}[(Y-a-bX)^2]$, differentiating in $a$ and $b$, and evaluating at $(a,b)=(\\alpha,\\beta)$ gives:}", "intended_meaning": "The marked words describe the step taken in this proof, and that step is valid. Under the chapter's convention that the expectations it uses converge absolutely (footnote 2, p.~45), the objective is the quadratic polynomial\n\\[\n\\mathrm{E}\\big[(Y-a-bX)^2\\big]=\\mathrm{E}[Y^2]-2a\\,\\mathrm{E}[Y]-2b\\,\\mathrm{E}[XY]+a^2+2ab\\,\\mathrm{E}[X]+b^2\\,\\mathrm{E}[X^2],\n\\]\nand differentiating it term by term gives the displayed equations. As a general statement the words are too broad, since a derivative does not pass through every sum or integral without further conditions. The normal equations and Theorem 2.2.21 are unaffected; the letters $\\alpha$ and $\\beta$ in the derivatives are the subject of \\fid{P001-C02-N006}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of Definition 2.3.1, Covariance Matrix, with the displayed K by K matrix of variances and covariances. A red box encloses the symbols for all i, j in the second line, which follow an entry written with the indices k and k prime.", "caption": "Definition 2.3.1 on p. 84 (PDF p. 102). The red box marks the quantifier over $i,j$; the entry before it and the displayed matrix use the indices of the components.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T48", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T48"}], "headline": "The quantifier in Definition 2.3.1 runs over $k,k'$, the indices of the entry, not over $i,j$.", "id": "P001-C02-E008", "location": "Definition 2.3.1, p. 84 (PDF p. 102), first sentence", "pdf_page": 102, "printed_page": "84", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T48"], "text": {"as_printed": "For a random vector $\\mathbf{X}$ of length $K$, the \\emph{covariance matrix} $\\operatorname{V}[\\mathbf{X}]$ is a matrix whose $(k,k')^{\\text{th}}$ entry is $\\operatorname{Cov}[X_{[k]},X_{[k']}]$, $\\forall\\,\\hl{i,j}\\in\\{1,2,\\ldots,K\\}$.", "corrected": "For a random vector $\\mathbf{X}$ of length $K$, the \\emph{covariance matrix} $\\operatorname{V}[\\mathbf{X}]$ is a matrix whose $(k,k')^{\\text{th}}$ entry is $\\operatorname{Cov}[X_{[k]},X_{[k']}]$, $\\forall\\,\\hl{k,k'}\\in\\{1,2,\\ldots,K\\}$.", "evidence": "The evidence is a check of the page. The displayed matrix and the sentences after the definition, on symmetry and on the diagonal entries, index the components as intended.", "problem": "The quantifier uses $i,j$ although the entries of the covariance matrix are indexed by $k,k'$. The letters $i$ and $j$ appear nowhere else in the definition.", "what_survives": "Everything else is unaffected; the displayed matrix fixes the meaning."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-E008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the display that begins 'Likewise, if X and Y are jointly continuous, then E[Y] = ...'. Four lines are shown. A red box marks the fourth, the double integral from minus infinity to infinity of y times f of y given x times f_X(x), dy dx.", "caption": "Proof of Theorem 2.2.17, jointly continuous case, p. 72 (PDF p. 90); the display continues on p. 73. The marked line is the first in which the conditional PDF appears under an integral over all real $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "In the proof of the Law of Iterated Expectations, the integrals over $x$ whose integrands contain $f_{Y|X}(y|x)$ or $\\mathrm{E}[Y|X=x]$ are taken over $\\operatorname{Supp}[X]$, where these are defined.", "id": "P001-C02-N002", "location": "Proof of Theorem 2.2.17, jointly continuous case, pp. 72–73 (PDF pp. 90–91)", "pdf_page": 90, "printed_page": "72–73", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Likewise, if $X$ and $Y$ are jointly continuous, then\n\\[\n\\begin{aligned}\n\\mathrm{E}[Y]&=\\int_{-\\infty}^{\\infty}yf_Y(y)dy\\\\\n&=\\int_{-\\infty}^{\\infty}y\\left(\\int_{-\\infty}^{\\infty}f(x,y)dx\\right)dy\\\\\n&=\\int_{-\\infty}^{\\infty}\\int_{-\\infty}^{\\infty}yf(x,y)dydx\\\\\n&=\\hl{\\int_{-\\infty}^{\\infty}}\\int_{-\\infty}^{\\infty}yf_{Y|X}(y|x)f_X(x)dydx\\\\\n&=\\hl{\\int_{-\\infty}^{\\infty}}\\left(\\int_{-\\infty}^{\\infty}yf_{Y|X}(y|x)dy\\right)f_X(x)dx\\\\\n&=\\hl{\\int_{-\\infty}^{\\infty}}\\mathrm{E}[Y|X=x]f_X(x)dx\\\\\n&=\\mathrm{E}\\big[\\mathrm{E}[Y|X]\\big].\n\\end{aligned}\n\\]", "corrected": "\\[\n\\begin{aligned}\n\\mathrm{E}[Y]&=\\int_{-\\infty}^{\\infty}\\int_{-\\infty}^{\\infty}yf(x,y)dydx\\\\\n&=\\hl{\\int_{\\operatorname{Supp}[X]}}\\int_{-\\infty}^{\\infty}yf_{Y|X}(y|x)f_X(x)dydx\\\\\n&=\\hl{\\int_{\\operatorname{Supp}[X]}}\\left(\\int_{-\\infty}^{\\infty}yf_{Y|X}(y|x)dy\\right)f_X(x)dx\\\\\n&=\\hl{\\int_{\\operatorname{Supp}[X]}}\\mathrm{E}[Y|X=x]f_X(x)dx\\\\\n&=\\mathrm{E}\\big[\\mathrm{E}[Y|X]\\big].\n\\end{aligned}\n\\]\nThe sums over $x$ in the discrete case are likewise over $\\operatorname{Supp}[X]$.", "intended_meaning": "Definition 1.3.14 (p.~38) defines $f_{Y|X}(y|x)$ only for $x\\in\\operatorname{Supp}[X]$, and $\\mathrm{E}[Y|X=x]$ is defined on the same set, while the marked integrals run over all real $x$. These integrals are meant over $\\operatorname{Supp}[X]$. That reading is correct: for $x$ outside $\\operatorname{Supp}[X]$, I have $\\int_{-\\infty}^{\\infty}f(x,y)dy=f_X(x)=0$ with $f\\ge0$, so these $x$ contribute nothing to $\\int\\int yf(x,y)dydx$. The theorem and the rest of the proof are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the boxes of Theorem 2.2.6 and Theorem 2.2.7. In the opening line of Theorem 2.2.6, 'For random variable X and Y,', a red box marks the word 'variable'. In the opening line of Theorem 2.2.7, 'For random variables X, Y, and Z,', a red box marks the letter Z. The four bullets of Theorem 2.2.7 involve only X, Y and the constants a, b, c, d.", "caption": "Theorems 2.2.6 and 2.2.7, p. 63 (PDF p. 81). The marked word should be plural, and the marked $Z$ appears in none of the four bullets below it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "Theorem 2.2.6 should begin “For random variables $X$ and $Y$”, and Theorem 2.2.7 should not introduce $Z$, which it does not use.", "id": "P001-C02-N003", "location": "Opening lines of Theorem 2.2.6 and of Theorem 2.2.7, p. 63 (PDF p. 81)", "pdf_page": 81, "printed_page": "63", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Theorem 2.2.6: For random \\hl{variable} $X$ and $Y$,\n\nTheorem 2.2.7: For random variables $X$, $Y$, and $\\hl{Z}$,", "corrected": "Theorem 2.2.6: For random \\hl{variables} $X$ and $Y$,\n\nTheorem 2.2.7: For random variables $X$ \\hl{and} $Y$,", "evidence": "The evidence is a check of the page: I read both theorems and the whole proof of Theorem 2.2.7 on pp.~63--64, and $Z$ does not occur after the opening line.", "problem": "Theorem 2.2.6 concerns the two random variables $X$ and $Y$ and has ``variable'' in the singular. Theorem 2.2.7 introduces a third random variable $Z$, while its four bullets and its proof involve only $X$, $Y$ and the constants $a,b,c,d$.", "what_survives": "Nothing else in either theorem changes. The second and third bullets of Theorem 2.2.6 have their own correction, \\fid{P001-C02-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the start of Example 2.2.16, Flipping a Coin and Rolling a Die: 'Consider, again, the generative process from Example 1.1.11. Recall from Example 1.3.5 that the conditional PMF of Y given X = x is', followed by the conditional PMF, equal to 1/4 for x = 0 and y in 1 to 4, to 1/6 for x = 1 and y in 1 to 6, and to 0 otherwise. A red box marks the words 'Example 1.3.5'.", "caption": "Example 2.2.16, p. 70 (PDF p. 88). The marked reference should be to Example 1.3.8, where this conditional PMF is derived.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N004.png"}], "formal_release": null, "formal_statements": [], "headline": "The conditional PMF recalled at the start of Example 2.2.16 comes from Example 1.3.8, not from Example 1.3.5.", "id": "P001-C02-N004", "location": "Example 2.2.16, second sentence, p. 70 (PDF p. 88)", "pdf_page": 88, "printed_page": "70", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Consider, again, the generative process from Example 1.1.11. Recall from \\hl{Example 1.3.5} that the conditional PMF of $Y$ given $X=x$ is\n\\[\nf_{Y|X}(y|x)=\\begin{cases}\\frac14 &:\\ x=0,\\,y\\in\\{1,2,3,4\\}\\\\[0.5ex] \\frac16 &:\\ x=1,\\,y\\in\\{1,2,3,4,5,6\\}\\\\[0.5ex] 0 &:\\ \\text{otherwise.}\\end{cases}\n\\]", "corrected": "Consider, again, the generative process from Example 1.1.11. Recall from \\hl{Example 1.3.8} that the conditional PMF of $Y$ given $X=x$ is", "evidence": "The evidence is a check of the pages: I compared the sentence on p.~70 with Examples 1.3.5 and 1.3.8 on pp.~33--35.", "problem": "Example 1.3.5 (p.~33) gives the joint PMF of $X$ and $Y$ and one value of the joint CDF. The conditional PMF is defined only afterwards, in Definition 1.3.7, and the function quoted here is displayed in Example 1.3.8 (pp.~34--35).", "what_survives": "The displayed conditional PMF is correct, and the rest of Example 2.2.16 is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four rows of a display, each beginning with an equals sign: E[epsilon squared given X] + 2W E[epsilon given X] + W squared; E[epsilon squared given X] + 0 + W squared; E[epsilon squared given X] minus E[epsilon given X] squared + W squared; V[Y given X] + W squared. Below, the sentence 'where the third and fifth lines follow from Theorem 2.2.19.' A red box marks the words 'third and fifth lines'.", "caption": "Proof of Theorem 2.2.20, p. 76 (PDF p. 94): the last four rows of the display and the sentence after it. The marked words are right if the opening row of the display, on p. 75, is not counted.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N005.png"}, {"alt": "Crop of the end of a paragraph and the first two rows of a display. A red box marks the first row, E[U squared given X] = E[(epsilon + W) squared given X]. The second row reads = E[epsilon squared + 2 epsilon W + W squared given X].", "caption": "Proof of Theorem 2.2.20, foot of p. 75 (PDF p. 93): the first two rows of the same display. The marked row is the opening row, which the count leaves out.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N005-p75.png"}], "formal_release": null, "formal_statements": [], "headline": "“The third and fifth lines” counts the lines after the opening row of the display; the two steps that use Theorem 2.2.19 are $\\mathrm{E}[\\varepsilon|X]=0$ and $\\mathrm{V}[\\varepsilon|X]=\\mathrm{V}[Y|X]$.", "id": "P001-C02-N005", "location": "Proof of Theorem 2.2.20, pp. 75–76 (PDF pp. 93–94), the sentence after the display that ends “$=\\mathrm{V}[Y|X]+W^2$”", "pdf_page": 93, "printed_page": "75–76", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\[\n\\begin{aligned}\n\\mathrm{E}\\big[U^2\\,\\big|\\,X\\big]&=\\mathrm{E}\\big[(\\varepsilon+W)^2\\,\\big|\\,X\\big]\\\\\n&=\\mathrm{E}\\big[\\varepsilon^2+2\\varepsilon W+W^2\\,\\big|\\,X\\big]\\\\\n&=\\mathrm{E}\\big[\\varepsilon^2\\,\\big|\\,X\\big]+2W\\mathrm{E}[\\varepsilon|X]+W^2\\\\\n&=\\mathrm{E}\\big[\\varepsilon^2\\,\\big|\\,X\\big]+0+W^2\\\\\n&=\\mathrm{E}\\big[\\varepsilon^2\\,\\big|\\,X\\big]-\\mathrm{E}[\\varepsilon|X]^2+W^2\\\\\n&=\\mathrm{V}[Y|X]+W^2,\n\\end{aligned}\n\\]\nwhere the \\hl{third and fifth lines} follow from Theorem 2.2.19.", "corrected": "where $\\hl{\\mathrm{E}[\\varepsilon|X]=0}$ and $\\hl{\\mathrm{V}[\\varepsilon|X]=\\mathrm{V}[Y|X]}$ follow from Theorem 2.2.19.", "intended_meaning": "The display has six rows, two on p.~75 and four on p.~76, and Theorem 2.2.19 (p.~74) states that $\\mathrm{E}[\\varepsilon|X]=0$ and $\\mathrm{V}[\\varepsilon|X]=\\mathrm{V}[Y|X]$. Counting from the row after the opening one, the third line is the row with $+0$ and the fifth is the last row, and both use the theorem. If the opening row is counted, the same rows are the fourth and the sixth. The proof is correct under either count."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the start of the proof: 'Let epsilon = Y - (a + bX). If g(X) = alpha + beta X is the best linear predictor of Y given X, then (alpha, beta) = arg min over (a, b) in R squared of E[epsilon squared]. The first-order conditions yield the system of equations', followed by two equations setting to zero the partial derivatives of E[epsilon squared]. Red boxes mark the definition epsilon = Y - (a + bX) and the two denominators, partial alpha and partial beta.", "caption": "Proof of Theorem 2.2.21, p. 77 (PDF p. 95). The first mark is the definition of $\\varepsilon$ in terms of $a$ and $b$; the other two are the derivatives, which are written in $\\alpha$ and $\\beta$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N006.png"}, {"alt": "Crop of footnote 19, 'The second-order partial derivatives are', with three displayed lines. In the second line, the second derivative of E[epsilon squared] in beta, a red box marks a bold upright X inside the expression minus 2 E[epsilon X]. The other occurrences of X in that line are italic.", "caption": "Footnote 19, p. 77 (PDF p. 95). The marked $\\mathbf{X}$ is set in bold, the book’s notation for a random vector; every other $X$ in the line is the italic scalar.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N006-fn19.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T43", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T43"}], "headline": "In the proof of Theorem 2.2.21 the first-order conditions are derivatives in the free arguments $a$ and $b$, evaluated at $(\\alpha,\\beta)$, and not derivatives in $\\alpha$ and $\\beta$; in footnote 19 one bold $\\mathbf{X}$ should be the scalar $X$.", "id": "P001-C02-N006", "location": "Proof of Theorem 2.2.21, p. 77 (PDF p. 95), from its first sentence to the first-order conditions; footnote 19, second displayed line", "pdf_page": 95, "printed_page": "77", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T43"], "text": {"as_printed": "\\textbf{Proof:} Let $\\varepsilon=Y-(a+bX)$. If $g(X)=\\alpha+\\beta X$ is the best linear predictor of $Y$ given $X$, then\n\\[\n(\\alpha,\\beta)=\\operatorname*{arg\\,min}_{(a,b)\\in\\mathbb{R}^2}\\mathrm{E}\\big[\\varepsilon^2\\big].\n\\]\nThe first-order conditions yield the system of equations\n\\[\n\\begin{aligned}\n0&=\\frac{\\partial\\mathrm{E}[\\varepsilon^2]}{\\hl{\\partial\\alpha}},\\\\\n0&=\\frac{\\partial\\mathrm{E}[\\varepsilon^2]}{\\hl{\\partial\\beta}}.\n\\end{aligned}\n\\]\nFootnote 19, second displayed line:\n\\[\n\\frac{\\partial^2\\mathrm{E}[\\varepsilon^2]}{\\partial\\beta^2}=\\frac{\\partial}{\\partial\\beta}\\big(-2\\mathrm{E}[\\varepsilon\\hl{\\mathbf{X}}]\\big)=-2\\mathrm{E}\\left[\\frac{\\partial\\varepsilon}{\\partial\\beta}X\\right]=-2\\mathrm{E}\\big[-X^2\\big]=2\\mathrm{E}\\big[X^2\\big],\n\\]", "corrected": "\\textbf{Proof:} Let $\\hl{Q(a,b)=\\mathrm{E}\\big[(Y-(a+bX))^2\\big]}$ for $(a,b)\\in\\mathbb{R}^2$. If $g(X)=\\alpha+\\beta X$ is the best linear predictor of $Y$ given $X$, then\n\\[\n(\\alpha,\\beta)=\\operatorname*{arg\\,min}_{(a,b)\\in\\mathbb{R}^2}\\hl{Q(a,b)}.\n\\]\nThe first-order conditions yield the system of equations\n\\[\n\\begin{aligned}\n0&=\\hl{\\frac{\\partial Q}{\\partial a}(\\alpha,\\beta)},\\\\\n0&=\\hl{\\frac{\\partial Q}{\\partial b}(\\alpha,\\beta)}.\n\\end{aligned}\n\\]\nThe later displays of the proof and footnote 19 are read in the same way: each derivative is taken in $a$ or $b$ and evaluated at $(\\alpha,\\beta)$, where $\\varepsilon=Y-(\\alpha+\\beta X)$. In the second line of footnote 19, $-2\\mathrm{E}[\\varepsilon\\mathbf{X}]$ becomes $-2\\mathrm{E}[\\varepsilon\\hl{X}]$, with the scalar $X$.", "evidence": "The evidence is a derivation of the first-order conditions in the corrected notation, and a check of the page for the bold letter. I write\n\\[\nQ(a,b)=\\mathrm{E}\\big[(Y-a-bX)^2\\big]=\\mathrm{E}[Y^2]-2a\\,\\mathrm{E}[Y]-2b\\,\\mathrm{E}[XY]+a^2+2ab\\,\\mathrm{E}[X]+b^2\\,\\mathrm{E}[X^2].\n\\]\nDifferentiating, I obtain\n\\[\n\\begin{aligned}\n\\frac{\\partial Q}{\\partial a}(a,b)&=2a+2b\\,\\mathrm{E}[X]-2\\,\\mathrm{E}[Y]=-2\\,\\mathrm{E}[Y-a-bX],\\\\\n\\frac{\\partial Q}{\\partial b}(a,b)&=2a\\,\\mathrm{E}[X]+2b\\,\\mathrm{E}[X^2]-2\\,\\mathrm{E}[XY]=-2\\,\\mathrm{E}\\big[X(Y-a-bX)\\big].\n\\end{aligned}\n\\]\nSetting both derivatives to zero at $(a,b)=(\\alpha,\\beta)$, I find $\\beta=\\operatorname{Cov}[X,Y]/\\mathrm{V}[X]$ and $\\alpha=\\mathrm{E}[Y]-\\beta\\,\\mathrm{E}[X]$, as in the theorem. The second derivatives of $Q$ are $2$ in $a$, $2\\,\\mathrm{E}[X^2]$ in $b$ and $2\\,\\mathrm{E}[X]$ mixed. The determinant of the Hessian is $4\\,\\mathrm{E}[X^2]-4\\,\\mathrm{E}[X]^2=4\\,\\mathrm{V}[X]>0$, which implies that $(\\alpha,\\beta)$ is the unique minimizer. In the published PDF the $X$ of $-2\\mathrm{E}[\\varepsilon X]$ in footnote 19 is set in the bold font, and the $X$ that follow it in the same line are in italic.", "problem": "The proof defines $\\varepsilon=Y-(a+bX)$, a function of the free arguments $a$ and $b$, and $(\\alpha,\\beta)$ as the fixed pair that minimizes $\\mathrm{E}[\\varepsilon^2]$. It then writes the first-order conditions as derivatives in $\\alpha$ and $\\beta$, on which $\\mathrm{E}[\\varepsilon^2]$, as defined, does not depend. The values $\\partial\\varepsilon/\\partial\\alpha=-1$ and $\\partial\\varepsilon/\\partial\\beta=-X$ used in the next display and in footnote 19 require $\\varepsilon=Y-(\\alpha+\\beta X)$ with $\\alpha$ and $\\beta$ treated as variables. The proof means to differentiate in $a$ and $b$ and to evaluate at $(a,b)=(\\alpha,\\beta)$. In the second line of footnote 19, the $X$ in $-2\\mathrm{E}[\\varepsilon X]$ is set in bold where the scalar $X$ is meant.", "what_survives": "Theorem 2.2.21, the formulas for $\\alpha$ and $\\beta$, the first-order equations read at the minimizer and the second-order calculation of footnote 19 are unaffected. Only the letters change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Screenshot of panel (a) of Figure 2.2.2. A black parabola with minimum at zero runs across the whole plot from x = -1 to x = 1, and a magenta straight line rises to the right. The density strip under the plot is filled only between 0 and 1. A red box encloses the part of the parabola to the left of x = 0.", "caption": "Figure 2.2.2, panel (a), on p. 83 (PDF p. 101). The red box marks the stretch of the black curve over negative values of $x$, where the density of $X$ shown in the strip below is zero.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "In panel (a) of Figure 2.2.2 the black curve is the CEF only on the support of $X$; over $[-1,0)$ it continues the formula $10x^2$.", "id": "P001-C02-N007", "location": "Figure 2.2.2, panel (a), p. 83 (PDF p. 101), with the sentences of Example 2.2.24 on p. 82 (PDF p. 100) that describe it", "pdf_page": 101, "printed_page": "83", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "However, we see in panel (a) of Figure 2.2.2 that the BLP approximates the CEF reasonably well only over certain areas. Additionally, if the distribution of $X$ changes, the BLP changes. Figure 2.2.2 demonstrates how, with \\hl{the same (nonlinear) CEF (in black)}, the BLP (in red) depends on the distribution of $X$.\n\nFIGURE 2.2.2. \\emph{Plotting the CEF and the BLP over Different Distributions of $X$}", "corrected": "Caption: FIGURE 2.2.2. \\emph{Plotting the CEF and the BLP over Different Distributions of $X$.} \\hl{The black curve is} $\\hl{10x^2}$\\hl{; it equals the CEF on the support of} $\\hl{X}$ \\hl{and shows a polynomial continuation elsewhere.}\n\nExample 2.2.24: Figure 2.2.2 demonstrates how, with \\hl{the same CEF formula on each support} (in black), the BLP (in red) depends on the distribution of $X$.", "intended_meaning": "The black curve is the function $10x^2$ given by the model $Y=10X^2+W$, drawn on a common axis while the distribution of $X$ changes from panel to panel. The book defines the CEF only on the support of $X$ (Definition 2.2.15, p.~69). In panel (a) $X$ is uniform on $[0,1]$. At $x=-1$ the density of $X$ is zero, and the ratio that defines the conditional PDF (Definition 1.3.14, p.~38) is $0/0$. The stretch of the curve over $[-1,0)$ is therefore the same formula continued, not a conditional expectation. On the support the figure is correct. For $x\\in[0,1]$ the conditional density of $Y$ given $X=x$ is the $N(10x^2,1)$ density, and the CEF is $10x^2$. The BLP is $-5/3+10x$, as derived on p.~81. The comparison across panels also holds, since for $X$ uniform on $[-1,1]$ the BLP in the same model is the constant $10/3$. A search of the book's text finds no later use of Example 2.2.24 or of Figure 2.2.2."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Screenshot of the four panels of Figure 2.2.2 with its caption. Each panel shows a black parabola, a magenta straight line and a density strip below. A red box encloses the horizontal magenta line of panel (b), which sits near 3.4, and another encloses the descending magenta line of panel (c), which runs from about 6 at x = -1 to about -2 at x = 1.", "caption": "Figure 2.2.2 on p. 83 (PDF p. 101). The red boxes mark the BLP lines of panels (b) and (c), which are drawn slightly away from $10/3$ and from $2-4x$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "In panels (b) and (c) of Figure 2.2.2 the red lines for the BLP lie 2 to 3 percent away from the exact BLPs, $10/3$ and $2-4x$, for the densities drawn beneath them.", "id": "P001-C02-N008", "location": "Figure 2.2.2, panels (b) and (c), p. 83 (PDF p. 101), the red lines", "pdf_page": 101, "printed_page": "83", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "FIGURE 2.2.2. \\emph{Plotting the CEF and the BLP over Different Distributions of $X$}", "corrected": "For the densities drawn, the red line of panel (b) is the horizontal line at height $\\hl{10/3}$, and the red line of panel (c) is $\\hl{2-4x}$. Alternatively, the caption can say that the plotted lines are numerical approximations.", "evidence": "The evidence is a recalculation from the published figure. The endpoints of the two red lines were read from the vector drawing in the PDF and converted to the units of the axes with the tick marks of each panel. For the exact BLPs, let $W$ be independent of $X$ with mean zero. Then the slope and the intercept of the BLP are\n\\[\n\\beta=\\frac{10\\bigl(\\operatorname{E}[X^3]-\\operatorname{E}[X]\\operatorname{E}[X^2]\\bigr)}{\\operatorname{E}[X^2]-\\operatorname{E}[X]^2},\\qquad \\alpha=10\\operatorname{E}[X^2]-\\beta\\operatorname{E}[X] .\n\\]\nIn panel (b) the moments $\\operatorname{E}[X]$, $\\operatorname{E}[X^2]$, $\\operatorname{E}[X^3]$ are $0$, $1/3$, $0$, and I obtain $\\beta=0$ and $\\alpha=10/3$. In panel (c) they are $-1/3$, $1/3$, $-1/5$, so that $\\operatorname{V}[X]=2/9$ and $\\operatorname{Cov}[X,X^2]=-4/45$, and I obtain $\\beta=-4$ and $\\alpha=2$.\n\n\\begin{audittable}{llll}{The red lines of panels (b) and (c) as drawn and the exact BLPs for the densities drawn beneath them.}\nPanel & Density of $X$ on $[-1,1]$ & Line as drawn & Exact BLP \\\\\n(b) & $1/2$ & $3.396$ & $10/3$ \\\\\n(c) & $(1-x)/2$ & $2.037-3.879x$ & $2-4x$ \\\\\n\\end{audittable}\n\nThe figure's code is not available, so the distributions actually used and the cause of the difference are not established. An equally weighted grid of 101 points on $[-1,1]$ yields the constant $17/5=3.4$ for panel (b), which is consistent with a numerical approximation, but the check does not show that such a grid was used.", "problem": "The two panels show the BLP of $Y$ given $X$ in the model $Y=10X^2+W$ for two distributions of $X$ that the text does not name. The density strips show the uniform density $1/2$ on $[-1,1]$ in panel (b) and the density $(1-x)/2$ on $[-1,1]$ in panel (c). For these densities the BLP is the constant $10/3$ in panel (b) and the line $2-4x$ in panel (c). The red lines as drawn are approximately $3.40$ and $2.04-3.88x$. The constant of panel (b) is off by 1.87 percent, and the intercept and slope of panel (c) are off by 1.84 and 3.01 percent. In panel (c) the red line differs from the exact BLP by $-0.084$ at $x=-1$ and by $0.157$ at $x=1$. The text reports no coefficient for either panel.", "what_survives": "The CEF, the formula for the BLP, the worked calculation for $X$ uniform on $[0,1]$ and the point of the figure, that the BLP changes with the distribution of $X$, are unaffected. Only the positions of the two red lines change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of Definition 2.3.6, BLP (Multivariate Case). Three red boxes enclose the ellipses in the sums b0 + b1 X[1] + b2 X[2] ... + bK X[K], in the same sum with beta coefficients, and in the sum inside the arg min display; each ellipsis directly follows X[2] with no plus sign.", "caption": "Definition 2.3.6 on p. 86 (PDF p. 104). The red boxes mark the three ellipses that have no plus sign before them.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N009.png"}, {"alt": "Screenshot of the one-line proof of Theorem 2.3.7. In the numerator of the right-hand fraction a red box encloses the ellipsis in beta0 + beta1 X[1] + beta2 X[2] + ... betaK X[K], which is followed directly by betaK.", "caption": "Proof of Theorem 2.3.7 on p. 87 (PDF p. 105). The red box marks the ellipsis that has no plus sign after it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N009-p87.png"}, {"alt": "Screenshot of Theorem 2.3.2, Multivariate Variance Rule. Two red boxes enclose the italic letters V at the start of the first and second expressions of the displayed equation.", "caption": "Theorem 2.3.2 on p. 84 (PDF p. 102). The red boxes mark the two variance symbols set in italic.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N009-p84.png"}, {"alt": "Screenshot of one line of text and the displayed fraction for beta that follows it. The line reads: X is distributed U(0,1), so its PDF is f sub X of x = 1, for all x in [0,1], and f(x) = 0 otherwise, so. A red box encloses f(x).", "caption": "Example 2.2.23 on p. 80 (PDF p. 98). The red box marks the density written without its subscript.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N009-p80.png"}], "formal_release": null, "formal_statements": [], "headline": "Four formulas carry small misprints: a plus sign is missing beside an ellipsis in Definition 2.3.6 and in the proof of Theorem 2.3.7, the variance in Theorem 2.3.2 is set as an italic $V$, and a density in Example 2.2.23 has lost its subscript.", "id": "P001-C02-N009", "location": "Definition 2.3.6, p. 86 (PDF p. 104), the three sums; also the proof of Theorem 2.3.7, p. 87 (PDF p. 105); Theorem 2.3.2, p. 84 (PDF p. 102); Example 2.2.23, p. 80 (PDF p. 98)", "pdf_page": 104, "printed_page": "86", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item Definition 2.3.6, p.~86, in the two formulas for $g(\\mathbf{X})$ and again inside the arg min: $g(\\mathbf{X})=b_0+b_1X_{[1]}+b_2X_{[2]}\\hl{\\ldots}+b_KX_{[K]}$ and $g(\\mathbf{X})=\\beta_0+\\beta_1X_{[1]}+\\beta_2X_{[2]}\\hl{\\ldots}+\\beta_KX_{[K]}$.\n\\item Proof of Theorem 2.3.7, p.~87, in the numerator of the derivative: $\\beta_0+\\beta_1X_{[1]}+\\beta_2X_{[2]}+\\hl{\\ldots}\\beta_KX_{[K]}$.\n\\item Theorem 2.3.2, p.~84, with both variances set in italic: $\\hl{V}\\bigl[X_{[1]}+X_{[2]}+\\cdots+X_{[K]}\\bigr]=\\hl{V}\\Bigl[\\sum_{k=1}^{K}X_{[k]}\\Bigr]=\\sum_{k=1}^{K}\\sum_{k'=1}^{K}\\operatorname{Cov}\\bigl[X_{[k]},X_{[k']}\\bigr]$.\n\\item Example 2.2.23, p.~80: its PDF is $f_X(x)=1$, $\\forall x\\in[0,1]$, and $\\hl{f(x)}=0$ otherwise.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Definition 2.3.6, in all three places: $g(\\mathbf{X})=b_0+b_1X_{[1]}+b_2X_{[2]}\\mathbin{\\hl{+}}\\cdots+b_KX_{[K]}$ and $g(\\mathbf{X})=\\beta_0+\\beta_1X_{[1]}+\\beta_2X_{[2]}\\mathbin{\\hl{+}}\\cdots+\\beta_KX_{[K]}$.\n\\item Proof of Theorem 2.3.7: $\\beta_0+\\beta_1X_{[1]}+\\beta_2X_{[2]}+\\cdots\\mathbin{\\hl{+}}\\beta_KX_{[K]}$.\n\\item Theorem 2.3.2: $\\hl{\\operatorname{V}}\\bigl[X_{[1]}+X_{[2]}+\\cdots+X_{[K]}\\bigr]=\\hl{\\operatorname{V}}\\Bigl[\\sum_{k=1}^{K}X_{[k]}\\Bigr]=\\sum_{k=1}^{K}\\sum_{k'=1}^{K}\\operatorname{Cov}\\bigl[X_{[k]},X_{[k']}\\bigr]$.\n\\item Example 2.2.23: its PDF is $f_X(x)=1$, $\\forall x\\in[0,1]$, and $\\hl{f_X(x)}=0$ otherwise.\n\\end{itemize}", "evidence": "The evidence is a check of pp.~80, 84, 86 and 87. The paragraph after Definition 2.3.6 prints the same sums with every plus sign, and Definition 2.3.1 on p.~84 uses the upright $\\operatorname{V}$.", "problem": "Each ellipsis continues a sum, and the plus sign before it (p.~86) or after it (p.~87) is missing. In Theorem 2.3.2 the variance is set as an italic $V$, where the book otherwise uses the upright $\\operatorname{V}$. On p.~80 the second $f$ denotes the same density $f_X$ and has lost its subscript. Each formula keeps its evident meaning.", "what_survives": "Everything else is unaffected; nothing depends on these formulas as misprinted."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of five lines of text from the top of page 87. Four red boxes enclose, in each of the two expansions beta0 + beta1 x[1] + beta2 x[2] ... + betaK x[k], the ellipsis that follows x[2] and the lower-case subscript [k] of the last term. The argument lists of g on the same lines end in x[K] with a capital K.", "caption": "First paragraph of p. 87 (PDF p. 105). The red boxes mark, in each of the two expansions of $g$, the ellipsis with no plus sign before it and the lower-case subscript $[k]$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C02-N010.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T49", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T49"}], "headline": "In the two expansions of $g$ on p. 87 the last term is $\\beta_Kx_{[K]}$, not $\\beta_Kx_{[k]}$, and a plus sign belongs before each ellipsis.", "id": "P001-C02-N010", "location": "p. 87 (PDF p. 105), first paragraph, the two expansions of $g$", "pdf_page": 105, "printed_page": "87", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T49"], "text": {"as_printed": "[\\ldots] so that $g(x_{[1]},x_{[2]},\\ldots,x_{[K]})=\\beta_0+\\beta_1x_{[1]}+\\beta_2x_{[2]}\\hl{\\ldots}+\\beta_K\\hl{x_{[k]}}$, and $g(x_{[1]}+1,x_{[2]},\\ldots,x_{[K]})=\\beta_0+\\beta_1(x_{[1]}+1)+\\beta_2x_{[2]}\\hl{\\ldots}+\\beta_K\\hl{x_{[k]}}$.", "corrected": "[\\ldots] so that $g(x_{[1]},x_{[2]},\\ldots,x_{[K]})=\\beta_0+\\beta_1x_{[1]}+\\beta_2x_{[2]}\\mathbin{\\hl{+}}\\cdots+\\beta_K\\hl{x_{[K]}}$, and $g(x_{[1]}+1,x_{[2]},\\ldots,x_{[K]})=\\beta_0+\\beta_1(x_{[1]}+1)+\\beta_2x_{[2]}\\mathbin{\\hl{+}}\\cdots+\\beta_K\\hl{x_{[K]}}$.", "evidence": "The evidence is a check of the page. Definition 2.3.6 on p.~86 and the derivative display on p.~87 end the same sum in $[K]$. The difference stated in the next sentence, $g(x_{[1]}+1,x_{[2]},\\ldots,x_{[K]})-g(x_{[1]},x_{[2]},\\ldots,x_{[K]})=\\beta_1$, makes the intended sums clear.", "problem": "The last subscript of each expansion is the lower-case $[k]$, although the argument lists of $g$ on the same lines end in $x_{[K]}$. Each sum also lacks the plus sign before its ellipsis.", "what_survives": "Everything else is unaffected, including the conclusion that the difference equals $\\beta_1$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C02-N010", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of three lines of text and the shaded box of Theorem 3.3.8, Consistency of the Kernel Density Estimator (Rectangular Kernel). A red box marks the quantifier 'for all x in R' in the statement: For i.i.d. continuous random variables X1, X2, ..., Xn, for all x in R, if K(x) is one half times the indicator that x lies between -1 and 1 (that is, the kernel is rectangular), and if h tends to 0 and nh tends to infinity as n tends to infinity, then f-hat sub K (x) converges in probability to f(x).", "caption": "Theorem 3.3.8, p. 122 (PDF p. 140). The marked quantifier covers every real $x$; the box does not repeat the condition that the PDF is continuous.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-A001.png"}, {"alt": "Crop of four lines of text. Red boxes mark the words 'as long as the PDF really is continuous' in the sentence 'Then, as long as the PDF really is continuous, kernel density estimation will be consistent.'", "caption": "P. 121 (PDF p. 139), last paragraph. The marked words state the condition under which the estimator is consistent; Theorem 3.3.8, on the next page, uses it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-A001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T127", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T127"}], "headline": "Theorem 3.3.8 holds at each point $x$ at which the density $f$ is continuous; the box says “$\\forall x\\in\\mathbb{R}$” and does not repeat the continuity condition stated on p. 121.", "id": "P001-C03-A001", "location": "Theorem 3.3.8, p. 122 (PDF p. 140), boxed statement; the continuity condition is in the last paragraph of p. 121 (PDF p. 139)", "pdf_page": 140, "printed_page": "122", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P001-T127"], "text": {"as_printed": "P.~121: ``Then, as long as the PDF really is continuous, kernel density estimation will be consistent.''\n\nP.~122: ``Theorem 3.3.8. \\emph{Consistency of the Kernel Density Estimator (Rectangular Kernel)}. For i.i.d.\\ continuous random variables $X_1, X_2, \\ldots, X_n$, $\\hl{\\forall x\\in\\mathbb{R}}$, if $K(x)=\\frac{1}{2}\\mathrm{I}(-1\\le x\\le 1)$ (that is, the kernel is rectangular), and if $h\\to 0$ and $nh\\to\\infty$ as $n\\to\\infty$, then $\\hat f_K(x)\\xrightarrow{p}f(x)$.''", "corrected": "Theorem 3.3.8. For i.i.d.\\ continuous random variables $X_1, X_2, \\ldots, X_n$ with common PDF $f$, \\hl{and for each $x\\in\\mathbb{R}$ at which $f$ is continuous}, if $K(x)=\\frac{1}{2}\\mathrm{I}(-1\\le x\\le 1)$ (that is, the kernel is rectangular), and if $h\\to 0$ and $nh\\to\\infty$ as $n\\to\\infty$, then $\\hat f_K(x)\\xrightarrow{p}f(x)$.\n\nHere $f$ is the density, $h$ the bandwidth and $n$ the sample size. To keep the conclusion for every real $x$, assume that $f$ is continuous on $\\mathbb{R}$.", "intended_meaning": "The theorem is a pointwise statement, at each fixed $x$, and it uses the condition stated in the sentence before it on p.~121: the PDF is continuous. Under that condition the theorem is correct. Fix $x$, and set $q_h=\\Pr[x-h\\le X_i\\le x+h]$ and $a_h=q_h/(2h)$. The count $2nh\\hat f_K(x)$ has the binomial distribution with parameters $n$ and $q_h$, so $\\hat f_K(x)$ has expectation $a_h$ and variance $a_h(1-q_h)/(2nh)$. If $f$ is continuous at $x$, then $a_h\\to f(x)$ as $h\\to 0$, and the variance tends to zero since $nh\\to\\infty$. Chebyshev's inequality then implies that $\\hat f_K(x)\\xrightarrow{p}f(x)$. The limit $a_h\\to f(x)$ is the last display of the proof sketch on p.~123, and it holds whenever $F'(x)$ exists and equals $f(x)$.\n\nRead without p.~121, the box covers every continuous random variable in the sense of Definition 1.2.14 and every real $x$, and the statement fails at a point where the density jumps. For the standard uniform distribution of Example 1.2.18, I have $f(0)=f(1)=1$. For $0<h<1$, the count $2nh\\hat f_K(0)$ has the binomial distribution with parameters $n$ and $h$. Then $\\hat f_K(0)$ has expectation $1/2$ and variance $(1-h)/(4nh)$, and $\\hat f_K(0)\\xrightarrow{p}1/2$. The same holds at $x=1$. A search of the text of the book finds no later reference to Theorem 3.3.8 and no numerical result that uses it at such a point."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-A001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "A paragraph of eight lines that begins 'We omit the proof, which requires empirical process theory.' Red boxes mark the clause 'our assumption about finite variance was sufficient to imply that the population mean is well behaved.', which runs over the third and fourth lines.", "caption": "Section 3.3.1, paragraph after Theorem 3.3.6, p. 120 (PDF p. 138). The marked clause says that finite variance makes the population mean well behaved.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E001.png"}, {"alt": "The seven lines of footnote 31. Red boxes mark its first sentence, 'We say T is well behaved if it is Hadamard differentiable at F with respect to the Kolmogorov distance.', and the expression 'sup over x of |F(x) - G(x)|' in its second sentence.", "caption": "Footnote 31, p. 120 (PDF p. 138). The marked sentence defines “well behaved” as Hadamard differentiability with respect to the Kolmogorov distance, and the marked expression is that distance.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E001-b.png"}, {"alt": "Two line charts with a common vertical scale from 0 to 1.2 and the index n = 2, 3, 5, 7, 11 on the horizontal axis. Left panel, 'Differentiability at F': blue circles for the size of the direction fall from 0.375 to 0.091, and orange squares for the difference quotient of the mean stay at 1. Right panel, 'Continuity at F': blue circles for the Kolmogorov distance fall from 0.234 to 0.008, and orange squares for the change in the mean stay at 1. Each panel is labeled 'stays at 1' beside the orange line and 'tends to 0' beside the blue line.", "caption": "Exact values for the two families of perturbations of $F$ described in the Evidence section above, at $n=2,3,5,7,11$. Left: the size $\\lVert h_n\\rVert_\\infty$ of the direction tends to $0$ and the difference quotient of the mean equals $1$. Right: the Kolmogorov distance $d_K(F,\\widetilde G_n)$ tends to $0$ and the mean changes by $1$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E001-quotient.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T1", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T1"}, {"id": "P001-T2", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T2"}, {"id": "P001-T118", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T118"}], "headline": "Finite variance does not make the population mean well behaved in the sense of footnote 31; the asymptotic normality of the sample mean follows from the Central Limit Theorem (Theorem 3.2.24), not from Hadamard differentiability with respect to the Kolmogorov distance.", "id": "P001-C03-E001", "location": "Section 3.3.1, paragraph after Theorem 3.3.6, p. 120 (PDF p. 138), read with footnote 31", "pdf_page": 138, "printed_page": "120", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T1", "P001-T118", "P001-T2"], "text": {"as_printed": "We omit the proof, which requires empirical process theory. The conditions for $T$ to be well behaved at $F$ are often implied by other assumptions; for instance, \\hl{our assumption about finite variance was sufficient to imply that the population mean is well behaved}. We will refer to these assumptions, as they apply to the functional under consideration, as the \\emph{usual plug-in regularity conditions}. The usual plug-in regularity conditions can be satisfied by assuming (and even verifying, in some cases) that the functional of interest will be well behaved in the neighborhoods of plausible population distributions.\n\nFootnote 31, first two sentences: We say $T$ is well behaved if it is Hadamard differentiable at $F$ with respect to the Kolmogorov distance. Formally, the Kolmogorov distance between two distributions $F$ and $G$ is given by the value of the largest divergence between their two CDFs, $\\sup_x|F(x)-G(x)|$.", "corrected": "We omit the proof, which requires empirical process theory. The conditions for $T$ to be well behaved at $F$ are often implied by other assumptions. \\hl{For the sample mean, our finite-variance assumption was sufficient for asymptotic normality by the scalar central limit theorem (Theorem 3.2.24). This does not imply that the population-mean functional is Hadamard differentiable with respect to the Kolmogorov distance of footnote 31.} We will refer to \\hl{such differentiability assumptions}, as they apply to the functional under consideration, as the \\emph{usual plug-in regularity conditions}. The usual plug-in regularity conditions can be satisfied by assuming (and even verifying, in some cases) that the functional of interest will be well behaved in the neighborhoods of plausible population distributions.\n\nTheorem 3.2.24 is the statement that replaces the clause. If $X_1,X_2,\\ldots,X_n$ are i.i.d. random variables with finite $\\mathrm E[X]=\\mu$ and finite $\\mathrm V[X]=\\sigma^2>0$, then\n\\[\n\\sqrt n\\,(\\overline X-\\mu)\\xrightarrow{d}N(0,\\sigma^2).\n\\]\nAn argument for the mean through Theorem 3.3.6 would need a topology and a set of directions that control first moments, with the derivative and the convergence of the empirical process established in that setting. I do not give one.", "evidence": "The evidence is an exact counterexample, with a derivation that holds for every $n\\ge2$, and a check of the definition in Wasserman (2006), to which the book refers, and in van der Vaart (1998). Let $F$ put mass $2^{-(k+1)}$ on each integer $k\\ge0$, so that $\\mathrm E_F[X]=1$, $\\mathrm E_F[X^2]=3$ and $\\operatorname{Var}_F X=2$. For an integer $L\\ge1$ and $0<p<1$, let $H_L$ be the distribution of $X+L$ and set $G=(1-p)F+pH_L$. The distribution $G$ has the same support as $F$, and a direct computation gives\n\\[\n\\mathrm E_G[X]=1+pL,\\qquad\\operatorname{Var}_G X=2+p(1-p)L^2,\\qquad d_K(F,G)=\\sup_x|F(x)-G(x)|=p\\,(1-2^{-L}).\n\\]\n\nIf $T_{\\mathrm E}$ is Hadamard differentiable at $F$ with derivative $A$, then $(T_{\\mathrm E}(F+t_nh_n)-T_{\\mathrm E}(F))/t_n\\to A(h)$ for all $t_n>0$ tending to $0$ and all $h_n\\to h$ uniformly such that $F+t_nh_n$ is a CDF with finite mean. For $h=0$ the limit is $A(0)=0$. Set $p_n=n^{-3}$, $L_n=n$, $t_n=n^{-2}$, $G_n=(1-p_n)F+p_nH_{L_n}$ and $h_n=(G_n-F)/t_n$. Substituting into the display, I obtain\n\\[\n\\begin{aligned}\n\\lVert h_n\\rVert_\\infty&=\\frac{1-2^{-n}}{n}\\longrightarrow0,\\qquad\\operatorname{Var}_{G_n}X=2+n^{-1}-n^{-4}\\longrightarrow2,\\\\\n\\frac{T_{\\mathrm E}(F+t_nh_n)-T_{\\mathrm E}(F)}{t_n}&=\\frac{n^{-2}}{n^{-2}}=1.\n\\end{aligned}\n\\]\nThe difference quotient equals $1$ for all $n$ and does not tend to $0$. I conclude that no such derivative exists. Note that each $G_n$ has finite variance, the same support as $F$, and second moment at most $4$.\n\\begin{audittable}{rrrrr}{The perturbations $G_n$ of $F$: exact values.}\n$n$ & $d_K(F,G_n)$ & $\\lVert h_n\\rVert_\\infty$ & $\\operatorname{Var}_{G_n}X$ & Difference quotient \\\\\n2 & $3/32$ & $3/8$ & $39/16$ & $1$ \\\\\n3 & $7/216$ & $7/24$ & $188/81$ & $1$ \\\\\n5 & $31/4000$ & $31/160$ & $1374/625$ & $1$ \\\\\n7 & $127/43904$ & $127/896$ & $5144/2401$ & $1$ \\\\\n11 & $2047/2725888$ & $2047/22528$ & $30612/14641$ & $1$ \\\\\n\\end{audittable}\nThe mean is not continuous at $F$ either. Set $\\widetilde G_n=(1-n^{-2})F+n^{-2}H_{n^2}$. Then $d_K(F,\\widetilde G_n)=n^{-2}(1-2^{-n^2})\\to0$ and $\\operatorname{Var}_{\\widetilde G_n}X=n^2+1<\\infty$, while $T_{\\mathrm E}(\\widetilde G_n)=2$ and $T_{\\mathrm E}(F)=1$ for every $n$.\n\nThe book gives no formal definition of Hadamard differentiability and refers to Wasserman (2006, Ch.~2). That chapter uses the distance $\\sup_x|F(x)-G(x)|$ over all distribution functions (Theorem 2.27 and Section 2.6), with no restriction on support, moments or directions under which the mean would be differentiable. Its appendix states the definition less precisely than the standard one and does not explicitly require the derivative to be continuous. The counterexample does not refute the clause if that weaker formula is read literally. Under the standard definition, which requires a continuous linear derivative (van der Vaart 1998, Section 20.2), the clause is false.", "problem": "The marked clause concerns the functional $T_{\\mathrm E}(F)=\\int x\\,dF(x)$. Read with footnote 31, it says that finite variance of $F$ makes $T_{\\mathrm E}$ Hadamard differentiable at $F$ with respect to the Kolmogorov distance. Under the standard definition of Hadamard differentiability, which requires a continuous linear derivative, this is false. The Kolmogorov distance does not control the tails of a distribution. A mixture that moves a small fraction of the mass far to the right changes the CDF uniformly by a small amount, and it can move the mean by a fixed amount. Finite variance of $F$ does not prevent this, and neither does finite variance of every perturbed distribution. The clause is the one concrete case that the book offers for the usual plug-in regularity conditions, and finite variance does not ensure that the mean satisfies them. Theorem 3.3.6 cannot be used, on the strength of finite variance alone, to justify the asymptotic normality of the sample mean. Under i.i.d. sampling with finite positive variance, the sample mean is asymptotically normal by Theorem 3.2.24 (p.~109).", "what_survives": "By Theorem 3.2.24 (p.~109), the sample mean is asymptotically normal under i.i.d. sampling with finite positive variance, and that theorem requires no differentiability of the mean functional. Example 3.3.4 (p.~119), which identifies the sample mean as a plug-in estimator, is unchanged, and so are the results on the consistency, unbiasedness and sampling variance of the sample mean. No estimator, estimand, theorem or reported number changes, and no simulation or application estimate was recalculated for this point. The clause is withdrawn. Where a later argument relies on the usual plug-in regularity conditions, they have to be established for the functional under consideration, and no later result is shown to be false. The missing condition in Theorem 3.3.6 itself is the subject of \\fid{P001-C03-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Three lines of running text that announce an informal statement of a key result (Wasserman 2006, Theorem 2.27), followed by Theorem 3.3.6 in a shaded box: 'If T is well behaved at F and theta-hat = T(F-hat), then theta-hat is asymptotically normal.' A red box marks the words 'is asymptotically normal.'", "caption": "Theorem 3.3.6, p. 120 (PDF p. 138), with the sentence that introduces it. The marked conclusion is stated with no condition on the limiting variance.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E002.png"}, {"alt": "The seven lines of footnote 31. Red boxes mark its first sentence: 'We say T is well behaved if it is Hadamard differentiable at F with respect to the Kolmogorov distance.'", "caption": "Footnote 31, p. 120 (PDF p. 138). The marked sentence is the definition of “well behaved”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E002-c.png"}, {"alt": "Definition 3.2.28, Asymptotic Normality, in a shaded box: an estimator is asymptotically normal if root n times (theta-hat minus theta) converges in distribution to N(0, phi squared), for some finite phi greater than 0. Red boxes mark the words 'for some' and 'finite phi > 0.'", "caption": "Definition 3.2.28, p. 112 (PDF p. 130). The marked condition requires a positive limiting standard deviation.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T56", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T56"}, {"id": "P001-T57", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T57"}, {"id": "P001-T118", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T118"}], "headline": "Theorem 3.3.6 needs a positive limiting variance: Hadamard differentiability of $T$ at $F$ allows a degenerate limit, which is not asymptotic normality as Definition 3.2.28 defines it.", "id": "P001-C03-E002", "location": "Theorem 3.3.6 and footnote 31, p. 120 (PDF p. 138); Definition 3.2.28, p. 112 (PDF p. 130)", "pdf_page": 138, "printed_page": "120", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T118", "P001-T56", "P001-T57"], "text": {"as_printed": "Theorem 3.3.6. Asymptotic Normality of Plug-In Estimators. If $T$ is well behaved$^{31}$ at $F$ and $\\hat\\theta=T(\\hat F)$, then $\\hat\\theta$ \\hl{is asymptotically normal}.\n\nFootnote 31, first sentence: We say $T$ is well behaved if it is Hadamard differentiable at $F$ with respect to the Kolmogorov distance.\n\nDefinition 3.2.28. Asymptotic Normality. An estimator $\\hat\\theta$ is \\emph{asymptotically normal} if $\\sqrt n(\\hat\\theta-\\theta)\\xrightarrow{d}N(0,\\phi^2)$, for some finite $\\phi>0$.", "corrected": "Theorem 3.3.6. Asymptotic Normality of Plug-In Estimators. For i.i.d. observations with CDF $F$, let $\\hat\\theta=T(\\hat F)$. If $T$ is Hadamard differentiable at $F$ with respect to the Kolmogorov distance, with derivative $\\dot T_F$, \\hl{and $0<\\operatorname{Var}[\\dot T_F(\\mathbb B_F)]<\\infty$, where $\\mathbb B_F$ is the limiting empirical-CDF process}, then $\\hat\\theta$ is asymptotically normal in the sense of Definition 3.2.28. \\hl{Without the positive-variance condition the delta-method limit may be degenerate.}\n\nHere $\\mathbb B_F$ is the centered Gaussian limit of $\\sqrt n(\\hat F-F)$, with covariance $F(\\min(x,y))-F(x)F(y)$. A derivative $\\dot T_F$ that is not zero does not suffice: the variance of $\\dot T_F(\\mathbb B_F)$ must be positive. The book omits the proof of Theorem 3.3.6, and I do not give a proof of the corrected statement.", "evidence": "The evidence is two exact counterexamples. Let $X_1,X_2,\\ldots,X_n$ be independent with $\\Pr[X_i=0]=\\Pr[X_i=1]=1/2$, and let $F$ be their CDF. Both functionals below are defined by their formulas on every bounded function $G$. Let $t_j\\to0$ with $t_j\\ne0$, and let $h_j\\to h$ uniformly.\n\nFirst, set $T_0(G)=7$ for every $G$. For all $j$, I have $(T_0(F+t_jh_j)-T_0(F))/t_j=0$, and $T_0$ is Hadamard differentiable at $F$ with derivative zero. Since $\\theta=\\hat\\theta=7$, it follows that $\\sqrt n(\\hat\\theta-\\theta)=0$ for all $n$. In particular, $\\Pr[\\sqrt n(\\hat\\theta-\\theta)\\le0]=1$ for every $n$, while the CDF of $N(0,\\phi^2)$ is continuous and equals $1/2$ at $0$ for every $\\phi>0$.\n\nSecond, set $T_1(G)=(G(0)-1/2)^2$, which is not constant. Since $F(0)=1/2$, I have\n\\[\n\\frac{T_1(F+t_jh_j)-T_1(F)}{t_j}=t_jh_j(0)^2\\longrightarrow0,\n\\]\nbecause $|h_j(0)|\\le\\lVert h\\rVert_\\infty+1$ for all large $j$. This implies that $T_1$ is Hadamard differentiable at $F$ with derivative zero. Let $K_n$ be the number of observations equal to $0$, and set $S_n=2K_n-n$. Then $\\theta=0$, $\\hat\\theta=(K_n/n-1/2)^2$ and $Y_n=\\sqrt n\\,\\hat\\theta=S_n^2/(4n^{3/2})$. Since $S_n$ is a sum of $n$ independent signs, $\\mathrm E[S_n^4]=3n^2-2n$, and I obtain\n\\[\n\\mathrm E[Y_n^2]=\\frac{3n-2}{16n^2}\\le\\frac{3}{16n}.\n\\]\nSince $\\Pr[|Y_n|\\ge a]\\le\\mathrm E[Y_n^2]/a^2$ for every $a>0$, I conclude that $Y_n\\xrightarrow{p}0$. Further, since $Y_n\\ge0$, I have $\\Pr[Y_n\\le-1]=0$ for all $n$, and $Y_n$ does not converge in distribution to $N(0,\\phi^2)$ for any $\\phi>0$. Summing over the binomial distribution in rational arithmetic for $n=1,\\ldots,64$, I find the same values of $\\mathrm E[Y_n^2]$, for example $1/16$ at $n=1$, $11/512$ at $n=8$ and $95/32768$ at $n=64$.", "problem": "Definition 3.2.28 requires that $\\sqrt n(\\hat\\theta-\\theta)$ converge in distribution to $N(0,\\phi^2)$ for some finite $\\phi>0$, and Definition 2.1.19 (p.~55) defines a normal distribution with positive standard deviation. The hypothesis of Theorem 3.3.6 is that $T$ is Hadamard differentiable at $F$ with respect to the Kolmogorov distance. This does not exclude a derivative that vanishes on the limit of $\\sqrt n(\\hat F-F)$. In that case $\\sqrt n(\\hat\\theta-\\theta)$ converges to the point mass at zero, and the conclusion fails in the sense that the book defines. The paragraph before the theorem calls it an informal statement, but neither that paragraph nor footnote 31 adds a condition on the limiting variance.", "what_survives": "The definitions of the empirical CDF and of plug-in estimators, and the cases in which the limiting variance is positive, are unchanged. Theorem 3.4.2 (p.~125) and Theorem 3.4.6 (p.~129) assume asymptotic normality separately, and the counterexamples do not bear on them. The usual plug-in regularity conditions of Section 3.3.1 are invoked for normal-approximation inference in the bootstrap discussion (p.~131), for regression (pp.~152--154), for the bounds with missing data (p.~212), for doubly robust estimation with missing data (p.~231), for the causal bounds (p.~244) and for doubly robust causal estimation (p.~270). Where these passages need a normal limit with positive variance, the positive-variance condition has to be added. No reported estimate, interval or simulation is shown to be wrong, and I did not examine each of these uses. The sentence about the population mean in the paragraph after the theorem is the subject of \\fid{P001-C03-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Screenshot of the two bullets that end Example 3.1.3: E[X] = 5 and V[X] = 17/2. A red box encloses the first expression after V[X] =, which reads E[(x - E[X])], with a lower-case x and no exponent.", "caption": "Example 3.1.3 (continued) on p. 94 (PDF p. 112). The red box marks the first expression for the variance, which has a lower-case $x$ and no square.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E003.png"}, {"alt": "Screenshot of the four-line display that begins V[X] = T_V(F). The third line is the integral of x squared dF(x) minus E[X] squared. In the fourth line a red box encloses a plus sign between the integral of x squared dF(x) and the square of the integral of x dF(x).", "caption": "The display for the variance functional on p. 118 (PDF p. 136). The red box marks the plus sign in the last line; the line above it has the minus sign.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E003-p118.png"}, {"alt": "Screenshot of the end of Example 3.3.4 and the whole of Example 3.3.5. One red box encloses the summand x in the sum from i = 1 to n that is set equal to X bar. A second red box encloses the plus sign in V[X] = T_V(F) = integral of x squared dF(x) + (integral of x dF(x)) squared. The last display, for the plug-in estimator, has a minus sign in the same position.", "caption": "Examples 3.3.4 and 3.3.5 on p. 119 (PDF p. 137). The red boxes mark the unindexed summand $x$ in the last line of Example 3.3.4 and the plus sign in the first display of Example 3.3.5; the plug-in display below it has the minus sign.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E003-p119.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T50", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T50"}], "headline": "Three displays misstate quantities that are defined correctly beside them: the variance on p. 94 is $\\operatorname{E}[(X-\\operatorname{E}[X])^2]$, the variance functional on pp. 118–119 subtracts the squared mean, and the last sum for the sample mean on p. 119 is over $X_i$.", "id": "P001-C03-E003", "location": "Example 3.1.3 (continued), p. 94 (PDF p. 112), variance bullet; the display for the variance functional, p. 118 (PDF p. 136), repeated in Example 3.3.5, p. 119 (PDF p. 137); Example 3.3.4, p. 119 (PDF p. 137), last line", "pdf_page": 112, "printed_page": "94", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T50"], "text": {"as_printed": "Example 3.1.3 (continued), p.~94:\n\\[\n\\operatorname{V}[X]=\\operatorname{E}\\bigl[\\hl{(x-\\operatorname{E}[X])}\\bigr]=\\sum_{x}\\bigl(x-\\operatorname{E}[X]\\bigr)^2f(x)=\\sum_{x\\in\\mathcal{X}}(x-\\mu)^2f_{FP}(x)=\\sigma^2=\\frac{17}{2}.\n\\]\nThe variance functional, p.~118, whose last expression is repeated in Example 3.3.5 on p.~119:\n\\[\n\\begin{aligned}\n\\operatorname{V}[X]&=T_{\\mathrm{V}}(F)\\\\\n&=\\int_{-\\infty}^{\\infty}\\bigl(x-\\operatorname{E}[X]\\bigr)^2dF(x)\\\\\n&=\\int_{-\\infty}^{\\infty}x^2dF(x)-\\operatorname{E}[X]^2\\\\\n&=\\int_{-\\infty}^{\\infty}x^2dF(x)\\mathbin{\\hl{+}}\\Bigl(\\int_{-\\infty}^{\\infty}x\\,dF(x)\\Bigr)^2.\n\\end{aligned}\n\\]\nExample 3.3.4, p.~119, last two lines of the display:\n\\[\n=\\frac{1}{n}\\sum_{x}x\\,(\\text{number of observations }i\\text{ with outcome }X_i=x)=\\frac{1}{n}\\sum_{i=1}^{n}\\hl{x}=\\overline{X},\n\\]", "corrected": "Example 3.1.3 (continued), p.~94:\n\\[\n\\operatorname{V}[X]=\\operatorname{E}\\bigl[(\\hl{X}-\\operatorname{E}[X])^{\\hl{2}}\\bigr]=\\sum_{x}\\bigl(x-\\operatorname{E}[X]\\bigr)^2f(x)=\\sum_{x\\in\\mathcal{X}}(x-\\mu)^2f_{FP}(x)=\\sigma^2=\\frac{17}{2}.\n\\]\nThe variance functional, p.~118, and the first display of Example 3.3.5, p.~119:\n\\[\n\\begin{aligned}\n\\operatorname{V}[X]&=T_{\\mathrm{V}}(F)\\\\\n&=\\int_{-\\infty}^{\\infty}\\bigl(x-\\operatorname{E}[X]\\bigr)^2dF(x)\\\\\n&=\\int_{-\\infty}^{\\infty}x^2dF(x)-\\operatorname{E}[X]^2\\\\\n&=\\int_{-\\infty}^{\\infty}x^2dF(x)\\mathbin{\\hl{-}}\\Bigl(\\int_{-\\infty}^{\\infty}x\\,dF(x)\\Bigr)^2.\n\\end{aligned}\n\\]\nExample 3.3.4, p.~119, last two lines of the display:\n\\[\n=\\frac{1}{n}\\sum_{x}x\\,(\\text{number of observations }i\\text{ with outcome }X_i=x)=\\frac{1}{n}\\sum_{i=1}^{n}\\hl{X_i}=\\overline{X},\n\\]", "evidence": "The evidence is a recalculation with the population $(3,4,3,10)$ of Example 3.1.3. I have $\\operatorname{E}[X]=5$ and $\\operatorname{E}[X^2]=(9+16+9+100)/4=67/2$, so that\n\\[\n\\operatorname{V}[X]=\\frac{67}{2}-25=\\frac{17}{2},\n\\]\nwhich is the value printed on p.~94. For the same population $\\operatorname{E}[X-\\operatorname{E}[X]]=0$, and the expression with the plus sign equals $67/2+25=117/2$. For the sample mean, I group the observations by their value and obtain\n\\[\n\\frac{1}{n}\\sum_{x}x\\,\\#\\{i:X_i=x\\}=\\frac{1}{n}\\sum_{i=1}^{n}X_i .\n\\]\nFor the sample $(1,3)$ both sides equal $2$. The third line of the display on p.~118 and the plug-in display of Example 3.3.5 on p.~119 have the minus sign.", "problem": "On p.~94 the first expression for $\\operatorname{V}[X]$ is the expectation of the centered variable without the square, and it has the lower-case $x$ where the random variable $X$ belongs. Since the expectation of $X-\\operatorname{E}[X]$ is zero, the first equality of the line is false as printed. On p.~118 the last line of the variance functional adds the squared mean instead of subtracting it, against the line above it, and Example 3.3.5 repeats the plus sign. In the last line of Example 3.3.4 the sum runs over $i$ and its summand is an unindexed $x$, which makes the sum equal to $x$ and not to the sample mean. In each case the adjacent expressions state the intended quantity.", "what_survives": "The value $\\sigma^2=17/2$ on p.~94, the remaining expressions of each display and everything else are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-E003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the opening of Example 3.5.8, Consequences of Clustering for the Sample Mean. Red boxes mark the quantifier 'for all i, j, j-prime' in two places: in the second bulleted assumption, 'for all i, j, j-prime, rho[X ij, X ij-prime] = rho > 0', and in the sentence 'Then, for all i, j, j-prime, Cov[X ij, X ij-prime] = rho[X ij, X ij-prime] sigma[X ij] sigma[X ij-prime] = rho sigma squared, so'. The first bulleted assumption reads 'for all i, j, V[X ij] = sigma squared'.", "caption": "Example 3.5.8, p. 140 (PDF p. 158). The two marked quantifiers cover every pair $j,j'$, including $j=j'$; the common correlation and the covariance $\\rho\\sigma^2$ are meant for $j\\neq j'$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E004.png"}, {"alt": "Crop of a numbered list of four steps and the paragraph after it. A red box marks the letter m in step 1: 'Take a with replacement sample of m clusters from our sample, keeping clusters intact.' Step 2 reads 'Calculate our would-be estimate using this collected n observations in our block bootstrap sample.' The paragraph after the list reads 'Since we drew a sample of l i.i.d. clusters to begin with, we are again replicating the sampling process, taking with-replacement samples of l clusters from our sample.'", "caption": "Block bootstrap, p. 141 (PDF p. 159). The marked symbol in step 1 is the number of clusters to resample; the paragraph after the four steps says “samples of $l$ clusters”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T64", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T64"}], "headline": "In Example 3.5.8 the common correlation $\\rho$ applies to distinct observations of a cluster ($j\\neq j'$), and step 1 of the block bootstrap resamples $l$ clusters, not $m$.", "id": "P001-C03-E004", "location": "Example 3.5.8, p. 140 (PDF p. 158), second assumption and the sentence after the two assumptions; block bootstrap, step 1, p. 141 (PDF p. 159)", "pdf_page": 158, "printed_page": "140", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T64"], "text": {"as_printed": "P.~140, Example 3.5.8: ``Suppose that:\n\\begin{itemize}\n\\item $\\forall i,j$, $\\mathrm{V}[X_{ij}]=\\sigma^2$.\n\\item $\\hl{\\forall i,j,j'}$, $\\rho[X_{ij},X_{ij'}]=\\rho>0$.\n\\end{itemize}\nIn other words, the variance of each observation is identical, and every unit within a cluster shares the same correlation. Then, $\\hl{\\forall i,j,j'}$, $\\mathrm{Cov}[X_{ij},X_{ij'}]=\\rho[X_{ij},X_{ij'}]\\sigma[X_{ij}]\\sigma[X_{ij'}]=\\rho\\sigma^2$, so \\ldots''\n\nP.~141, the block bootstrap, steps 1 and 2:\n\\begin{enumerate}\n\\item ``Take a \\emph{with replacement} sample of $\\hl{m}$ clusters from our sample, keeping clusters intact.\n\\item Calculate our would-be estimate using this collected $n$ observations in our block bootstrap sample.''\n\\end{enumerate}", "corrected": "P.~140, Example 3.5.8: ``Suppose that:\n\\begin{itemize}\n\\item $\\forall i,j$, $\\mathrm{V}[X_{ij}]=\\sigma^2$.\n\\item $\\forall i,j,j'$ \\hl{with $j\\neq j'$}, $\\rho[X_{ij},X_{ij'}]=\\rho>0$.\n\\end{itemize}\nIn other words, the variance of each observation is identical, and every unit within a cluster shares the same correlation. Then, $\\forall i,j,j'$ \\hl{with $j\\neq j'$}, $\\mathrm{Cov}[X_{ij},X_{ij'}]=\\rho[X_{ij},X_{ij'}]\\sigma[X_{ij}]\\sigma[X_{ij'}]=\\rho\\sigma^2$, so\n\\[\n\\begin{aligned}\n\\mathrm{V}\\bigl[\\overline{X}\\bigr]&=\\frac{1}{lm^2}\\sum_{j=1}^{m}\\sum_{j'=1}^{m}\\mathrm{Cov}[X_{ij},X_{ij'}]\\\\\n&=\\frac{1}{lm^2}\\bigl(m\\sigma^2+m(m-1)\\rho\\sigma^2\\bigr)\\\\\n&=\\frac{\\sigma^2}{lm}\\bigl(1+\\rho(m-1)\\bigr).\\text{''}\n\\end{aligned}\n\\]\n\nP.~141, the block bootstrap, steps 1 and 2:\n\\begin{enumerate}\n\\item ``Take a \\emph{with replacement} sample of $\\hl{l}$ clusters from our sample, keeping clusters intact.\n\\item Calculate our would-be estimate using this collected $n$ observations in our block bootstrap sample.''\n\\end{enumerate}\n\nHere $l$ is the number of clusters, $m$ is the number of observations in each cluster and $n=lm$.", "evidence": "The evidence is a derivation and a check of pp.~135--136 and 140--141. With the restriction $j\\neq j'$, Theorem 3.5.5 implies that\n\\[\n\\mathrm{V}\\bigl[\\overline{X}\\bigr]=\\frac{1}{lm^2}\\sum_{j=1}^{m}\\sum_{j'=1}^{m}\\mathrm{Cov}[X_{ij},X_{ij'}]=\\frac{1}{lm^2}\\bigl(m\\sigma^2+m(m-1)\\rho\\sigma^2\\bigr)=\\frac{\\sigma^2}{lm}\\bigl(1+\\rho(m-1)\\bigr),\n\\]\nwhich is the printed formula. At $\\rho=0$ it equals $\\sigma^2/(lm)$, and at $\\rho=1$ it equals $\\sigma^2/l$. As a check, let $m=3$, $l=5$, $\\sigma^2=2$ and $\\rho=1/4$. The covariance matrix of a cluster has diagonal elements $2$ and off-diagonal elements $1/2$. Computing directly from this matrix, I find $\\mathrm{V}[\\overline{X}]=1/5$, which agrees with the formula.\n\nFor the block bootstrap, let $w_1,w_2,\\ldots,w_l$ be the observed cluster averages, with mean $\\overline{w}$, and set $v=(1/l)\\sum_{i=1}^{l}(w_i-\\overline{w})^2$. A bootstrap sample of $k$ intact clusters holds $km$ observations, and its mean is the average of $k$ independent draws from $w_1,w_2,\\ldots,w_l$. Conditional on the data, the variance of this mean is $v/k$. For the intended choice $k=l$, the sample holds $n$ observations and the variance is $v/l$. For the printed choice $k=m$, the sample holds $m^2$ observations and the variance is $v/m$, which is $l/m$ times the intended variance. For $l=100$ and $m=2$, the printed step collects $4$ observations in place of $200$, and the variance is $50$ times the intended one.", "problem": "On p.~140, the condition of a common correlation within clusters lacks the restriction to distinct observations. As printed, the condition and the covariance statement after it hold for all $j$ and $j'$, including $j=j'$. An observation with positive variance has correlation $1$ with itself and covariance $\\sigma^2$ with itself, so the condition as printed forces $\\rho=1$, and the covariance statement puts $\\rho\\sigma^2$ on the diagonal in place of $\\sigma^2$. The calculation after the statement uses the intended condition: it takes $\\sigma^2$ for the $m$ terms with $j=j'$ and $\\rho\\sigma^2$ for the $m(m-1)$ terms with $j\\neq j'$.\n\nIn step 1 of the block bootstrap on p.~141, the number of resampled clusters is mislabeled. Recall that Section 3.5 has $l$ clusters of size $m$ and total sample size $n=lm$. A sample of $m$ clusters holds $m^2$ observations, which equals the $n=lm$ observations of step 2 only if $m=l$, and the standard deviation in step 4 is then that of an estimate from $m$ clusters. The paragraph after the steps states the intended procedure: ``taking with-replacement samples of $l$ clusters from our sample''.", "what_survives": "The formula $\\mathrm{V}[\\overline{X}]=\\sigma^2(1+\\rho(m-1))/(lm)$, its two extreme cases and the block bootstrap as described in the paragraph after the steps are correct as printed. The example assumes $\\rho>0$ and then considers $\\rho=0$ as an extreme case; this extends the formula to the boundary and is not an error. The condition may be stated as $0\\le\\rho\\le 1$ to cover both extreme cases."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-E004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Seven lines of running text that explain convergence in probability. A red box marks the expression 'p in [0,1]', with a closed bracket at 1, in the sentence that begins 'More specifically'.", "caption": "Paragraph after Definition 3.2.6, p. 99 (PDF p. 117). The marked interval includes the endpoint $p=1$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T51", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T51"}], "headline": "The verbal description of convergence in probability holds for every $p\\in[0,1)$; at the endpoint $p=1$ it is false.", "id": "P001-C03-E005", "location": "Paragraph after Definition 3.2.6, p. 99 (PDF p. 117), sentence beginning “More specifically”", "pdf_page": 117, "printed_page": "99", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T51"], "text": {"as_printed": "More specifically, for any $p\\in[0,1\\hl{]}$ and any $\\epsilon>0$, there is always an $N$ large enough that, for every $n\\ge N$, the probability that $T_{(n)}$ will be within $\\epsilon$ of $c$ is at least $p$.", "corrected": "More specifically, for any $p\\in[0,1\\hl{)}$ and any $\\epsilon>0$, there is always an $N$ large enough that, for every $n\\ge N$, the probability that $T_{(n)}$ will be within $\\epsilon$ of $c$ is at least $p$.", "evidence": "The evidence is an exact counterexample to the case $p=1$ and a derivation of the case $p<1$. Let $T_{(n)}$ take the value $1$ with probability $1/(n+1)$ and the value $0$ otherwise, and set $c=0$. For $0<\\epsilon\\le1$, I have $\\Pr[|T_{(n)}|\\ge\\epsilon]=1/(n+1)$, which tends to $0$, and for $\\epsilon>1$ this probability is $0$. I conclude that $T_{(n)}\\xrightarrow{p}0$. For $\\epsilon=1/2$, I have $\\Pr[|T_{(n)}|<\\epsilon]=n/(n+1)<1$ for all $n$, and there is no $N$ such that the sentence holds with $p=1$.\n\nNow fix $p<1$ and $\\epsilon>0$, and set $q_n=\\Pr[|T_{(n)}-c|<\\epsilon]$. Since $q_n\\to1$, there exists $N$ such that $|q_n-1|<1-p$ for all $n\\ge N$, which implies that $q_n>p$. The sentence holds for every $p\\in[0,1)$ with no further hypothesis.", "problem": "By Definition 3.2.6, the statement $T_{(n)}\\xrightarrow{p}c$ means that $\\Pr[|T_{(n)}-c|<\\epsilon]$ tends to $1$ as $n\\to\\infty$, for all $\\epsilon>0$. A sequence of probabilities that tends to $1$ is eventually larger than any fixed $p<1$, but it need not equal $1$ for any finite $n$. For $p=1$ the sentence asserts that the probability equals $1$ for every $n\\ge N$, and convergence in probability does not imply this.", "what_survives": "Definition 3.2.6, the results on convergence in probability and their proofs are unchanged. Only the bracket in this sentence changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-E005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Two lines of text and one displayed formula. A red box marks 'by the WLLN, X-squared-bar converges in probability to E[X squared]'. The display below applies the continuous mapping theorem and ends with V[X] and a proof-end square.", "caption": "End of the proof of Theorem 3.2.19, p. 107 (PDF p. 125). The marked step applies the WLLN to the sample mean of the squares.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E006.png"}, {"alt": "The statement of Theorem 3.2.8, Weak Law of Large Numbers (WLLN), in a shaded box. Red boxes mark the words 'finite variance V[X] >' at the end of the first line and the '0' that begins the second line. The conclusion is that the sample mean converges in probability to E[X].", "caption": "Theorem 3.2.8, p. 100 (PDF p. 118). The marked hypothesis is finite variance of the variables that are averaged.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E006-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T53", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T53"}, {"id": "P001-T54", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T54"}], "headline": "The proof of Theorem 3.2.19 applies the weak law of large numbers for variables with finite variance (Theorem 3.2.8) to the squares $X_i^2$, which need not have finite variance; the step follows from the weak law for integrable variables, and the theorem holds as stated.", "id": "P001-C03-E006", "location": "Proof of Theorem 3.2.19, p. 107 (PDF p. 125), sentence beginning “Now, by the WLLN”; Theorem 3.2.8, p. 100 (PDF p. 118)", "pdf_page": 125, "printed_page": "107", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T53", "P001-T54"], "text": {"as_printed": "Now, \\hl{by the WLLN}, $\\overline{X^2}\\xrightarrow{p}\\mathrm E[X^2]$ and $\\overline X\\xrightarrow{p}\\mathrm E[X]$. Let $g(u,v)=u-v^2$. Then, by the CMT,\n\\[\n\\overline{X^2}-\\overline X^{\\,2}=g\\bigl(\\overline{X^2},\\overline X\\bigr)\\xrightarrow{p}g\\bigl(\\mathrm E[X^2],\\mathrm E[X]\\bigr)=\\mathrm E[X^2]-\\mathrm E[X]^2=\\mathrm V[X].\\quad\\square\n\\]", "corrected": "Since $\\mathrm E[X^2]<\\infty$, \\hl{the WLLN for i.i.d. integrable random variables} gives $\\overline{X^2}\\xrightarrow{p}\\mathrm E[X^2]$. \\hl{Theorem 3.2.8} gives $\\overline X\\xrightarrow{p}\\mathrm E[X]$. Let $g(u,v)=u-v^2$. Then, by the CMT,\n\\[\n\\overline{X^2}-\\overline X^{\\,2}=g\\bigl(\\overline{X^2},\\overline X\\bigr)\\xrightarrow{p}g\\bigl(\\mathrm E[X^2],\\mathrm E[X]\\bigr)=\\mathrm E[X^2]-\\mathrm E[X]^2=\\mathrm V[X].\\quad\\square\n\\]\nThe law used in the first sentence is the following statement, which the book does not state. Let $Y_1,Y_2,\\ldots,Y_n$ be i.i.d. random variables with $\\mathrm E|Y|<\\infty$, and let $\\overline Y_{(n)}=n^{-1}\\sum_{i=1}^nY_i$. Then $\\overline Y_{(n)}\\xrightarrow{p}\\mathrm E[Y]$.\n\nTheorem 3.2.19 itself is unchanged: for i.i.d. random variables $X_1,X_2,\\ldots,X_n$ with finite variance $\\mathrm V[X]>0$,\n\\[\n\\mathrm E\\bigl[\\overline{X^2}-\\overline X^{\\,2}\\bigr]=\\frac{n-1}{n}\\,\\mathrm V[X]\\qquad\\text{and}\\qquad\\overline{X^2}-\\overline X^{\\,2}\\xrightarrow{p}\\mathrm V[X].\n\\]", "evidence": "The evidence is an exact example in which the cited theorem does not apply, and a derivation of the limit without a fourth moment. Let $\\Pr[X=2^k]=7/8^{k+1}$ for $k=0,1,2,\\ldots$. Summing geometric series, I obtain $\\mathrm E[X]=7/6$, $\\mathrm E[X^2]=7/4$ and $\\mathrm V[X]=7/18>0$. The partial sums of the fourth moment are\n\\[\n\\sum_{k=0}^{N}\\frac{7}{8^{k+1}}\\,2^{4k}=\\frac78\\bigl(2^{N+1}-1\\bigr),\n\\]\nwhich tend to infinity, so that $\\mathrm V[X^2]=\\infty$. This distribution satisfies the assumption of Theorem 3.2.19, and its squares do not satisfy the assumption of Theorem 3.2.8.\n\nFor the limit, let $Y_1,Y_2,\\ldots$ be i.i.d. nonnegative random variables with $m=\\mathrm E[Y]<\\infty$. Fix $M>0$ and set $U_i=\\min(Y_i,M)$, $R_i=Y_i-U_i$, $m_M=\\mathrm E[U_i]$ and $r_M=\\mathrm E[R_i]=m-m_M$. I write $\\overline Y$, $\\overline U$ and $\\overline R$ for the sample means of the first $n$ terms. By monotone convergence, $r_M\\to0$ as $M\\to\\infty$. Since $0\\le U_i\\le M$, I have $\\mathrm V[U_i]\\le M^2$. By Chebyshev's inequality, I obtain $\\Pr[|\\overline U-m_M|\\ge\\epsilon/2]\\le4M^2/(n\\epsilon^2)$ for all $\\epsilon>0$. Markov's inequality implies that $\\Pr[\\overline R\\ge\\epsilon/4]\\le4r_M/\\epsilon$. Since $|\\overline Y-m|\\le|\\overline U-m_M|+\\overline R+r_M$, for all $M$ such that $r_M\\le\\epsilon/4$ I find\n\\[\n\\Pr\\bigl[|\\overline Y-m|\\ge\\epsilon\\bigr]\\le\\frac{4M^2}{n\\epsilon^2}+\\frac{4r_M}{\\epsilon}.\n\\]\nGiven $\\eta>0$, I choose $M$ such that $r_M<\\min(\\epsilon/4,\\eta\\epsilon/8)$ and then $n_0>8M^2/(\\eta\\epsilon^2)$. For all $n\\ge n_0$ the right side is less than $\\eta$, which implies that $\\overline Y\\xrightarrow{p}m$. For integrable variables of either sign, I apply the same argument to the positive and negative parts. By the bound $X^2\\le2(X-\\mathrm E[X])^2+2\\mathrm E[X]^2$, I have $\\mathrm E[X^2]<\\infty$ under the assumption of Theorem 3.2.19, and the argument applies to $Y_i=X_i^2$.", "problem": "Theorem 3.2.19 assumes i.i.d. random variables $X_1,X_2,\\ldots,X_n$ with finite variance $\\mathrm V[X]>0$. The weak law of large numbers stated in the book is Theorem 3.2.8 (p.~100), which the glossary (p.~287) gives as the meaning of the abbreviation WLLN. It assumes that the variables being averaged have finite variance. The proof applies it to the squares $X_i^2$, and this requires $\\mathrm V[X^2]<\\infty$, that is, a finite fourth moment of $X$, which Theorem 3.2.19 does not assume. The cited theorem does not cover the step. Note that the limit $\\overline{X^2}\\xrightarrow{p}\\mathrm E[X^2]$ is true under the stated assumption: the proof has a gap, and the theorem is not false. The consistency parts of Theorems 3.2.21, 3.2.33 and 3.2.35 are deduced from this step, and the interval of Example 3.4.3 (p.~127) and the variance estimator of Section 3.5.2 (p.~140) use it through them.", "what_survives": "Theorem 3.2.19 holds as stated, with no fourth moment and no added assumption, and the computation of the expectation in its proof needs no repair. With the repaired step, the consistency parts of Theorems 3.2.21, 3.2.33 and 3.2.35, the interval of Example 3.4.3 and the consistency claim of Section 3.5.2 hold under their stated conditions. No estimator, estimand or number changes. The simulations of Example 3.4.7 use uniform and Bernoulli distributions, which are bounded, and the gap does not concern them."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-E006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of text. Red boxes mark the two occurrences of 89 percent in the sentences 'For example, when n = 10, the actual coverage of our 95 percent confidence intervals lies below the nominal level; they in fact have only 89 percent coverage. That is, across 89 percent of our simulated random draws of n = 10 values from the distribution X, the (nominally) 95 percent normal approximation-based confidence interval actually covers E[X].'", "caption": "Example 3.4.7, p. 134 (PDF p. 152), the paragraph that interprets the coverage columns of Table 3.4.1. Both marked figures should read $91.6\\%$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E007.png"}, {"alt": "Crop of Table 3.4.1, Implications of Micronumerosity (Standard Uniform Distribution), and Table 3.4.2, Implications of Micronumerosity (Bernoulli Distribution with p = 0.5). Each table has rows for n = 10, 100 and 1,000 and coverage columns for the 90, 95 and 99 percent confidence intervals. Red boxes mark the 95 percent entry of the n = 10 row in each table: 91.6 percent in Table 3.4.1 and 89.0 percent in Table 3.4.2. The coverage entries of the n = 10 row are 86.6, 91.6 and 96.5 percent in Table 3.4.1, and 89.0, 89.0 and 97.8 percent in Table 3.4.2.", "caption": "Tables 3.4.1 and 3.4.2, p. 134 (PDF p. 152). The marked entry of Table 3.4.1 (uniform, $n=10$, $95\\%$ CI) is $91.6\\%$. The marked entry of Table 3.4.2 (Bernoulli, $n=10$, $95\\%$ CI) is $89.0\\%$, the figure that appears in the text.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-E007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "On p. 134, the text assigns to the uniform experiment the coverage of about $89\\%$ that belongs to the Bernoulli experiment; for the nominal $95\\%$ interval with ten observations, it should report $91.6\\%$, as in Table 3.4.1.", "id": "P001-C03-E007", "location": "Example 3.4.7, p. 134 (PDF p. 152), paragraph beginning “In the remaining columns”, below Tables 3.4.1 and 3.4.2", "pdf_page": 152, "printed_page": "134", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``For example, when $n=10$, the actual coverage of our $95\\%$ confidence intervals lies below the nominal level---they in fact have only $\\hl{89\\%}$ coverage. That is, across $\\hl{89\\%}$ of our simulated random draws of $n=10$ values from the distribution $X$, the (nominally) $95\\%$ normal approximation--based confidence interval actually covers $\\mathrm{E}[X]$.''", "corrected": "``For example, when $n=10$, the actual coverage of our $95\\%$ confidence intervals lies below the nominal level---they in fact have only $\\hl{91.6\\%}$ coverage. That is, across $\\hl{91.6\\%}$ of our simulated random draws of $n=10$ values from the distribution $X$, the (nominally) $95\\%$ normal approximation--based confidence interval actually covers $\\mathrm{E}[X]$.''", "evidence": "The evidence is a check of the page and a recalculation. A simulation of the stated design ($U(0,1)$, $n=10$, $1{,}000{,}000$ replications, critical value $1.96$) gives coverage $91.6406\\%$, with a Monte Carlo standard error of $0.02768$ percentage points. For the Bernoulli design with $p=0.5$ and $n=10$, exact enumeration yields coverage $57/64=89.0625\\%$. No replication code is available for the book, and the original random draws cannot be recovered. Note that the agreement with Table 3.4.2 does not establish how the figure entered the text.", "problem": "The paragraph explains Table 3.4.1, the simulation with i.i.d.\\ draws from $U(0,1)$. The $n=10$ row of that table reports coverage $86.6\\%$, $91.6\\%$ and $96.5\\%$ for the nominal $90\\%$, $95\\%$ and $99\\%$ intervals, and no entry of that row is $89\\%$. The entry $89.0\\%$ is in Table 3.4.2, the Bernoulli simulation, which the text introduces in the next paragraph. The two figures differ by $2.6$ percentage points, more than rounding allows. With the correct figure, the coverage is $3.4$ points below the nominal level and not $6.0$.", "what_survives": "Tables 3.4.1 and 3.4.2 are unchanged. The conclusions of the example stand: the interval covers too rarely at $n=10$, the shortfall shrinks as $n$ grows, and the Bernoulli results differ from the uniform ones. A search of the text of the book finds no later use of the figure $89\\%$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-E007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of the last six lines of Section 3.1.1. A red box encloses the words Dubins and Freedman 1979 inside the parenthesis that follows: Some technical finesse is required to formalize this intuition.", "caption": "Last sentence of Section 3.1.1 on p. 94 (PDF p. 112). The red box marks the citation of Dubins and Freedman (1979).", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "For the approximation of sampling without replacement from a large population by i.i.d. sampling, the fitting citation is Diaconis and Freedman (1980), not Dubins and Freedman (1979).", "id": "P001-C03-N001", "location": "Section 3.1.1, p. 94 (PDF p. 112), last sentence", "pdf_page": 112, "printed_page": "94", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Some technical finesse is required to formalize this intuition (e.g., \\hl{Dubins and Freedman 1979}), but it implies that sampling units at random from a hypothetical, infinite \\emph{superpopulation} yields data that can be treated as i.i.d.", "corrected": "Some technical finesse is required to formalize this intuition (e.g., \\hl{Diaconis and Freedman 1980}), but it implies that sampling units at random from a hypothetical, infinite \\emph{superpopulation} yields data that can be treated as i.i.d.\n\nThe reference to add is: Diaconis, P., and Freedman, D. (1980). Finite exchangeable sequences. \\emph{Annals of Probability} 8(4): 745--764. Dubins and Freedman (1979) can be kept only as a reference for the caveat that de Finetti's representation can fail on general measurable spaces.", "intended_meaning": "The sentence points to the technical work behind the preceding claim, that draws without replacement from a large population are approximately independent and identically distributed. It is not false, and nothing in the book depends on the citation. The paper cited, Dubins and Freedman (1979), ``Exchangeable processes need not be mixtures of independent, identically distributed random variables'', constructs spaces on which de Finetti's representation fails; its summary (p.~115) states: ``This paper demonstrates the existence of a separable metric space for which the conclusion fails.'' It does not give the approximation for finite populations. That approximation is the subject of Diaconis and Freedman (1980), ``Finite exchangeable sequences'', whose abstract states: ``The main tool is a bound on the variation distance between sampling with and without replacement.'' I checked the published 1979 paper and the abstract of the 1980 paper, not the body of the 1980 paper."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Footnote 12, one line, followed by the three lines of footnote 13. A red box marks 'for all D in R' in footnote 12, printed with the membership sign.", "caption": "Footnotes 12 and 13, p. 102 (PDF p. 120). The marked quantifier makes $D$ an element of $\\mathbb R$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T52", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T52"}], "headline": "In footnote 12 the quantifier ranges over sets $D\\subseteq\\mathbb R$; the footnote prints $D\\in\\mathbb R$.", "id": "P001-C03-N004", "location": "Footnote 12, p. 102 (PDF p. 120)", "pdf_page": 120, "printed_page": "102", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T52"], "text": {"as_printed": "Note that for any random variable $X$, $\\forall D\\hl{\\in}\\mathbb R$, $\\mathrm E[\\mathrm I(X\\in D)]=\\Pr[X\\in D]$.", "corrected": "Note that for any random variable $X$ \\hl{and any measurable set} $D\\hl{\\subseteq}\\mathbb R$, $\\mathrm E[\\mathrm I(X\\in D)]=\\Pr[X\\in D]$.", "evidence": "The evidence is a check of the page: the printed symbol is $\\in$ and not $\\subseteq$. For a measurable set $D\\subseteq\\mathbb R$, the indicator $\\mathrm I(X\\in D)$ takes the value $1$ with probability $\\Pr[X\\in D]$ and the value $0$ otherwise, so that\n\\[\n\\mathrm E[\\mathrm I(X\\in D)]=1\\cdot\\Pr[X\\in D]+0\\cdot\\Pr[X\\notin D]=\\Pr[X\\in D].\n\\]\nThe proof of Theorem 3.2.9 on the same page computes this expectation for the event $X\\le x$.", "problem": "The event $X\\in D$ is defined when $D$ is a set of real numbers. The footnote quantifies over $D\\in\\mathbb R$, which makes $D$ a real number. Theorem 1.3.17 (p.~39) writes the corresponding quantifier as $\\forall D,E\\subseteq\\mathbb R$.", "what_survives": "The identity is correct for sets. Theorem 3.2.9 and its proof use only the event $X_i\\le x$ and are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The three lines of footnote 19. A red box marks the word 'Sterling's' in the phrase 'primarily relies on Sterling's approximation'.", "caption": "Footnote 19, p. 109 (PDF p. 127). The marked name is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "The approximation named in footnote 19 is Stirling’s approximation.", "id": "P001-C03-N005", "location": "Footnote 19, p. 109 (PDF p. 127)", "pdf_page": 127, "printed_page": "109", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Full proofs of the Central Limit Theorem invariably require fairly advanced mathematics, though the proof of a special case, the De Moivre--Laplace Theorem, is relatively accessible and primarily relies on \\hl{Sterling's} approximation. See, e.g., Feller (1968, Sections II.9, VII.3).", "corrected": "\\ldots\\ is relatively accessible and primarily relies on \\hl{Stirling's} approximation.", "evidence": "The evidence is a check of the page.", "problem": "The approximation $n!\\sim\\sqrt{2\\pi n}\\,(n/e)^n$ for the factorial, which the proof of the De Moivre--Laplace Theorem uses, is Stirling's approximation. The footnote prints the name as ``Sterling's''.", "what_survives": "The content of the footnote and the statement of the Central Limit Theorem are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The heading '3.2.5 Asymptotic Estimation Theory' and five lines of text. A red box marks the words 'in probability' in the first line, which reads 'We noted above that when a sample statistic T(n) converges in probability to a'.", "caption": "Opening of Section 3.2.5, p. 111 (PDF p. 129): the first sentence, with “in probability” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N006.png"}, {"alt": "Four lines of text. A red box marks the words 'in distribution' in the sentence 'The distribution of the random variable T to which a sequence of random variables T(n) converges in distribution is referred to as the limit distribution (or asymptotic distribution) of T(n).' The sentence after it, in parentheses, reads 'This notion will be important in Section 3.2.5.'", "caption": "Paragraph after Definition 3.2.22, p. 109 (PDF p. 127): the passage that the sentence on p. 111 recalls, with “in distribution” marked. The sentence after it refers ahead to Section 3.2.5.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The first sentence of Section 3.2.5 should say “converges in distribution”, as does the passage on p. 109 that it recalls; it is printed with “converges in probability”.", "id": "P001-C03-N006", "location": "Section 3.2.5, first sentence, p. 111 (PDF p. 129)", "pdf_page": 129, "printed_page": "111", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We noted above that when a sample statistic $T_{(n)}$ converges \\hl{in probability} to a random variable $T$, the distribution of $T$ is known as the limit distribution (or asymptotic distribution) of $T_{(n)}$.", "corrected": "We noted above that when a sample statistic $T_{(n)}$ converges \\hl{in distribution} to a random variable $T$, the distribution of $T$ is known as the limit distribution (or asymptotic distribution) of $T_{(n)}$.", "evidence": "The evidence is a check of the pages and a derivation. The sentence is true as printed, since convergence in probability implies convergence in distribution. Let $F_{(n)}$ and $F$ be the CDFs of $T_{(n)}$ and $T$, and suppose that $T_{(n)}$ converges in probability to $T$. For every continuity point $t$ of $F$ and every $\\delta>0$, I have\n\\[\nF(t-\\delta)-\\Pr\\bigl[|T_{(n)}-T|>\\delta\\bigr]\\le F_{(n)}(t)\\le F(t+\\delta)+\\Pr\\bigl[|T_{(n)}-T|>\\delta\\bigr].\n\\]\nLetting $n\\to\\infty$ and then $\\delta\\to0$, I find that $F_{(n)}(t)\\to F(t)$, which is the convergence required by Definition 3.2.22. The printed sentence states a sufficient condition, and the passage on p.~109 covers more cases. The standardized sample mean $Z$ of Theorem 3.2.24 is one of them: when each sample consists of the first $n$ terms of one i.i.d.\\ sequence, $Z$ converges in distribution to $N(0,1)$ and does not converge in probability to any random variable.", "problem": "The sentence recalls the paragraph after Definition 3.2.22 (p.~109), which names the limit distribution for a sequence that converges in distribution and refers ahead to Section 3.2.5. The two passages use the same symbols and the same two names, but different modes of convergence. The Central Limit Theorem (p.~109) and Definitions 3.2.26, 3.2.28, 3.2.29, 3.2.30 and 3.2.32 (pp.~112--113) are stated with convergence in distribution. The word ``probability'' on p.~111 is a misprint for ``distribution''.", "what_survives": "No result changes. The sentence is true as printed, and the passage on p.~109, the Central Limit Theorem and the definitions of Section 3.2.5 already use convergence in distribution. The correction changes one word and adds no assumption."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The first half of Definition 3.2.32 in a shaded box. The sentence above the display names the sampling variance estimator V-hat of theta-hat. A red box marks 'n V-hat[theta]', with no hat on theta, at the left of the displayed convergence to V[T].", "caption": "Definition 3.2.32, p. 113 (PDF p. 131), first display. The introducing sentence writes $\\hat{\\mathrm V}[\\hat\\theta]$, and the marked term has no hat on $\\theta$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N007.png"}, {"alt": "The second half of Definition 3.2.32. The sentence names the standard error estimator, the square root of V-hat of theta-hat. In the display below, a red box marks 'V-hat[theta]' under the square root, with no hat on theta.", "caption": "Definition 3.2.32, continued, p. 114 (PDF p. 132), second display. The marked term has no hat on $\\theta$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N007-b.png"}, {"alt": "The two lines of footnote 23. Red boxes mark 'n V-hat[theta]' in the first equivalent expression and 'V-hat[theta]' under the square root in the second.", "caption": "Footnote 23, p. 114 (PDF p. 132). Both marked terms have no hat on $\\theta$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N007-c.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T55", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T55"}, {"id": "P001-T128", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T128"}], "headline": "In the two displays of Definition 3.2.32 and in footnote 23, the sampling variance estimator is $\\hat{\\mathrm V}[\\hat\\theta]$; the hat on $\\theta$ is missing in four places.", "id": "P001-C03-N007", "location": "Definition 3.2.32, pp. 113–114 (PDF pp. 131–132), both displays; footnote 23, p. 114 (PDF p. 132)", "pdf_page": 131, "printed_page": "113–114", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T128", "P001-T55"], "text": {"as_printed": "For an estimator $\\hat\\theta$ such that $\\sqrt n(\\hat\\theta-\\theta)\\xrightarrow{d}T$, a sampling variance estimator $\\hat{\\mathrm V}[\\hat\\theta]$ is \\emph{consistent} if\n\\[\nn\\hat{\\mathrm V}[\\hl{\\theta}]\\xrightarrow{p}\\mathrm V[T]\n\\]\nand a standard error estimator $\\sqrt{\\hat{\\mathrm V}[\\hat\\theta]}$ is \\emph{consistent} if\n\\[\n\\sqrt n\\sqrt{\\hat{\\mathrm V}[\\hl{\\theta}]}\\xrightarrow{p}\\sqrt{\\mathrm V[T]}.\n\\]\nFootnote 23: Note that it is equivalent to define these as $n\\hat{\\mathrm V}[\\hl{\\theta}]-\\mathrm V[T]\\xrightarrow{p}0$ and $\\sqrt n\\sqrt{\\hat{\\mathrm V}[\\hl{\\theta}]}-\\sqrt{\\mathrm V[T]}\\xrightarrow{p}0$, respectively.", "corrected": "For an estimator $\\hat\\theta$ such that $\\sqrt n(\\hat\\theta-\\theta)\\xrightarrow{d}T$, a sampling variance estimator $\\hat{\\mathrm V}[\\hat\\theta]$ is \\emph{consistent} if\n\\[\nn\\hat{\\mathrm V}[\\hl{\\hat\\theta}]\\xrightarrow{p}\\mathrm V[T]\n\\]\nand a standard error estimator $\\sqrt{\\hat{\\mathrm V}[\\hat\\theta]}$ is \\emph{consistent} if\n\\[\n\\sqrt n\\sqrt{\\hat{\\mathrm V}[\\hl{\\hat\\theta}]}\\xrightarrow{p}\\sqrt{\\mathrm V[T]}.\n\\]\nFootnote 23: Note that it is equivalent to define these as $n\\hat{\\mathrm V}[\\hl{\\hat\\theta}]-\\mathrm V[T]\\xrightarrow{p}0$ and $\\sqrt n\\sqrt{\\hat{\\mathrm V}[\\hl{\\hat\\theta}]}-\\sqrt{\\mathrm V[T]}\\xrightarrow{p}0$, respectively.", "evidence": "The evidence is a check of the two pages. Theorem 3.4.2 (p.~125), which uses this definition, writes $\\hat{\\mathrm V}[\\hat\\theta]$ throughout.", "problem": "The sentence that introduces each display names $\\hat{\\mathrm V}[\\hat\\theta]$, the estimator of the sampling variance of the estimator $\\hat\\theta$. In both displays and in both expressions of footnote 23 the argument is printed as $\\theta$, the fixed target, which has no sampling variance.", "what_survives": "The definition as intended and Theorem 3.4.2 are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The last line of a proof and a paragraph of six lines. Red boxes mark the phrase 'the sampling variance of the sample mean' in the first line of the paragraph and the phrase 'limit distribution of the sample mean' in its third line.", "caption": "Paragraph after the proof of Theorem 3.2.33, p. 115 (PDF p. 133). The two marked phrases name the sampling variance where its estimator is meant, and the sample mean where its centered and scaled version is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "The sentence after Theorem 3.2.33 describes the estimator of the sampling variance of the sample mean, and the limit distribution it refers to is that of $\\sqrt n(\\overline X-\\mathrm E[X])$.", "id": "P001-C03-N008", "location": "Paragraph after the proof of Theorem 3.2.33, p. 115 (PDF p. 133)", "pdf_page": 133, "printed_page": "115", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The first property states that \\hl{the sampling variance of the sample mean} is unbiased, and the second is equivalent to stating that it is consistent (as the \\hl{limit distribution of the sample mean} has variance $\\mathrm V[X]$).", "corrected": "The first property states that \\hl{the estimator of} the sampling variance of the sample mean is unbiased, and the second is equivalent to stating that it is consistent (as the limit distribution of $\\hl{\\sqrt n(\\overline X-\\mathrm E[X])}$ has variance $\\mathrm V[X]$).", "intended_meaning": "The two properties in Theorem 3.2.33 concern the estimator $\\hat{\\mathrm V}[\\overline X]=\\hat{\\mathrm V}[X]/n$: its expectation is $\\mathrm V[\\overline X]$, and $n\\hat{\\mathrm V}[\\overline X]-n\\mathrm V[\\overline X]\\xrightarrow{p}0$. Unbiasedness and consistency are properties of this estimator. The sampling variance $\\mathrm V[\\overline X]=\\mathrm V[X]/n$ is a fixed number. The second property is consistency in the sense of Definition 3.2.32, which refers to the limit distribution of $\\sqrt n(\\overline X-\\mathrm E[X])$. By the Central Limit Theorem (Theorem 3.2.24), this limit is $N(0,\\mathrm V[X])$, with variance $\\mathrm V[X]$. The sample mean without centering and scaling converges in probability to $\\mathrm E[X]$, and its limit distribution has variance zero. Theorem 3.2.33 and Definition 3.2.32, directly above the sentence, give the intended meaning, which is correct. The theorem, its proof and the standard error formulas are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The end of Example 3.3.5: the words 'the plug-in estimator of V[X] is', a displayed formula, and a closing sentence. A red box marks the symbol 'V-hat[X]' at the left of the display, which is set equal to T sub V of F-hat, to a difference of two integrals, and to X-squared-bar minus X-bar squared.", "caption": "Example 3.3.5, p. 119 (PDF p. 137), second display. The marked symbol denotes the plug-in sample variance here.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N009.png"}, {"alt": "Definition 3.2.20, Unbiased Sample Variance, in a shaded box. A red box marks the symbol 'V-hat[X]' at the left of the display, which is set equal to n over (n minus 1) times the difference of X-squared-bar and X-bar squared.", "caption": "Definition 3.2.20, p. 107 (PDF p. 125). The same symbol is defined as the unbiased sample variance.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Example 3.3.5 writes $\\hat{\\mathrm V}[X]$ for the plug-in sample variance, and Definition 3.2.20 fixes the same symbol as the unbiased sample variance; the two differ by the factor $n/(n-1)$.", "id": "P001-C03-N009", "location": "Example 3.3.5, p. 119 (PDF p. 137), second display; Definition 3.2.20, p. 107 (PDF p. 125)", "pdf_page": 137, "printed_page": "119", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "the plug-in estimator of $\\mathrm V[X]$ is\n\\[\n\\hl{\\hat{\\mathrm V}[X]}=T_{\\mathrm V}(\\hat F)=\\int_{-\\infty}^{\\infty}x^2\\,d\\hat F(x)-\\left(\\int_{-\\infty}^{\\infty}x\\,d\\hat F(x)\\right)^2=\\overline{X^2}-\\overline X^{\\,2}.\n\\]", "corrected": "the plug-in estimator of $\\mathrm V[X]$ is\n\\[\n\\hl{\\hat{\\mathrm V}_{\\mathrm{PI}}[X]}=T_{\\mathrm V}(\\hat F)=\\int_{-\\infty}^{\\infty}x^2\\,d\\hat F(x)-\\left(\\int_{-\\infty}^{\\infty}x\\,d\\hat F(x)\\right)^2=\\overline{X^2}-\\overline X^{\\,2}.\n\\]", "intended_meaning": "The example derives the plug-in sample variance, which its title and its last sentence name, and the formula is correct. The symbol $\\hat{\\mathrm V}[X]$ is defined in Definition 3.2.20 as the unbiased sample variance, $n(\\overline{X^2}-\\overline X^{\\,2})/(n-1)$, and p.~114 uses it with that meaning. In the example the symbol denotes $(n-1)/n$ times that quantity. The later formulas for the standard error of the sample mean use $\\hat{\\mathrm V}[X]$ with the meaning of Definition 3.2.20. With the plug-in sample variance in its place, the standard error would be smaller by the factor $\\sqrt{(n-1)/n}$. No calculation in the book is shown to use the plug-in sample variance where the unbiased one is meant."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Three lines of running text. A red box marks the words 'estimate theta', with no hat on theta, in the sentence 'Then, as n grows large, the estimate theta tends to more and more closely approximate theta'.", "caption": "Section 3.3, p. 119 (PDF p. 137). The marked estimate is printed as $\\theta$ without its hat.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010.png"}, {"alt": "Three lines of running text. A red box marks 'i = 1,2,...N' in the phrase 'a finite population U consisting of N units, indexed i = 1,2,...N'.", "caption": "Section 3.1.1, p. 92 (PDF p. 110). The marked list has no comma after the ellipsis.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-b.png"}, {"alt": "Three lines of running text. A red box marks the words 'allow to us to' at the start of the second line.", "caption": "Section 3.2.2, p. 103 (PDF p. 121). The marked phrase has a misplaced “to”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-c.png"}, {"alt": "Three lines of running text. A red box marks the words 'defined it in' in the sentence 'Indeed, this is the same MSE that we defined it in Definition 2.1.22'.", "caption": "Section 3.2.2, p. 104 (PDF p. 122). The marked phrase has an extra “it”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-d.png"}, {"alt": "Four lines of running text. A red box marks 'Section 3.2.2).' at the end of the sentence 'We begin with a simple case: the asymptotic variant of the unbiasedness property from Section 3.2.2).' The parenthesis in the sentence before it is opened and closed.", "caption": "Section 3.2.5, p. 111 (PDF p. 129). The marked closing parenthesis has no opening parenthesis in its sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-e.png"}, {"alt": "Four lines of running text. A red box marks the words 'a way to a represent' in the sentence 'It is simply a way to a represent a feature of a random variable'.", "caption": "Section 3.3, p. 118 (PDF p. 136). The marked phrase has an extra “a”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-f.png"}, {"alt": "Three lines of running text followed by the heading of Theorem 3.3.8, Consistency of the Kernel Density Estimator (Rectangular Kernel), in a shaded box. A red box marks the word 'example' in 'The following example shows how'.", "caption": "Section 3.3.2, p. 122 (PDF p. 140). The marked word introduces a result that is labeled Theorem 3.3.8.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-g.png"}, {"alt": "The heading and first three lines of Theorem 3.4.2, Normal Approximation-Based Confidence Interval, in a shaded box. A red box marks the words 'an consistent' at the end of the first line of the statement.", "caption": "Theorem 3.4.2, p. 125 (PDF p. 143). The marked article should be “a”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-h.png"}, {"alt": "Three lines of running text. A red box marks the words 'an result' in 'we would infrequently encounter an result as extreme as the one that we saw'.", "caption": "Section 3.4.2, p. 128 (PDF p. 146). The marked article should be “a”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N010-i.png"}], "formal_release": null, "formal_statements": [], "headline": "The running text of Chapter 3 has nine typographical slips; none changes a meaning.", "id": "P001-C03-N010", "location": "Running text at nine places: p. 92 (PDF p. 110); p. 103 (PDF p. 121); p. 104 (PDF p. 122); p. 111 (PDF p. 129); p. 118 (PDF p. 136); p. 119 (PDF p. 137); p. 122 (PDF p. 140); Theorem 3.4.2, p. 125 (PDF p. 143); p. 128 (PDF p. 146)", "pdf_page": 110, "printed_page": "92", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item p.~92: indexed $i=1,2,\\hl{\\ldots N}$.\n\\item p.~103: Our usual definitions that allow us to characterize random variables also \\hl{allow to us to} characterize the distributions of estimators.\n\\item p.~104: Indeed, this is the same MSE that we \\hl{defined it} in Definition 2.1.22, just applied to estimators.\n\\item p.~111: We begin with a simple case: the asymptotic variant of the unbiasedness property from Section 3.2.2\\hl{)}.\n\\item p.~118: It is simply a way \\hl{to a represent} a feature of a random variable.\n\\item p.~119: Then, as $n$ grows large, the estimate $\\hl{\\theta}$ tends to more and more closely approximate $\\theta$.\n\\item p.~122: The following \\hl{example} shows how, for continuous $X$, the kernel density estimate (with a rectangular kernel) will be consistent for the true PDF. The proof itself leans on the plug-in principle.\n\\item p.~125, Theorem 3.4.2: Let $\\hat\\theta$ be an asymptotically normal estimator of $\\theta$, $\\hat{\\mathrm V}[\\hat\\theta]$ be \\hl{an} consistent estimator of the sampling variance, and $\\alpha\\in(0,1)$.\n\\item p.~128: we would infrequently encounter \\hl{an} result as extreme as the one that we saw.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item p.~92: indexed $i=1,2,\\ldots\\hl{,}N$.\n\\item p.~103: Our usual definitions that allow us to characterize random variables also \\hl{allow us to} characterize the distributions of estimators.\n\\item p.~104: Indeed, this is the same MSE that we \\hl{defined} in Definition 2.1.22, just applied to estimators.\n\\item p.~111: We begin with a simple case: the asymptotic variant of the unbiasedness property from Section 3.2.2\\hl{.}\n\\item p.~118: It is simply a way \\hl{to represent} a feature of a random variable.\n\\item p.~119: Then, as $n$ grows large, the estimate $\\hl{\\hat\\theta}$ tends to more and more closely approximate $\\theta$.\n\\item p.~122: The following \\hl{theorem} shows how, for continuous $X$, the kernel density estimate (with a rectangular kernel) will be consistent for the true PDF. The proof itself leans on the plug-in principle.\n\\item p.~125, Theorem 3.4.2: Let $\\hat\\theta$ be an asymptotically normal estimator of $\\theta$, $\\hat{\\mathrm V}[\\hat\\theta]$ be \\hl{a} consistent estimator of the sampling variance, and $\\alpha\\in(0,1)$.\n\\item p.~128: we would infrequently encounter \\hl{a} result as extreme as the one that we saw.\n\\end{itemize}", "evidence": "The evidence is a check of the nine pages. On p.~119 the estimator is defined in the sentences directly above, and on p.~122 the result that follows is labeled Theorem 3.3.8.", "problem": "Each item is a slip of typesetting or copy-editing. On p.~92 the comma after the ellipsis is missing. On p.~103 a ``to'' is misplaced, p.~104 has an extra ``it'', and p.~118 has an extra ``a''. On p.~111 a closing parenthesis has no opening one. On p.~119 the estimate is printed as $\\theta$ without its hat. On p.~122 the result that the sentence introduces is Theorem 3.3.8 and not an example. On pp.~125 and 128 the article ``an'' stands before a consonant.", "what_survives": "No statement, formula or result changes. The intended reading is immediate in each case."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N010", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the shaded box of Definition 3.3.7, Kernel Density Estimator. A red box marks the clause 'Let K : R to R be an even function satisfying integral from minus infinity to infinity of K(x) dx = 1'. The rest of the definition gives the rescaled kernel K sub h and the estimator f-hat sub K (x), the average of K sub h (x - X i) over the n observations.", "caption": "Definition 3.3.7, p. 121 (PDF p. 139). The marked clause holds the only conditions placed on the kernel $K$: it is even and it integrates to one. The definition does not require $K\\ge 0$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N011.png"}, {"alt": "Crop of a paragraph and the shaded box of Definition 3.3.9, Kernel Plug-In Estimator. Red boxes mark the words 'and thus we can apply any statistical functional to that estimate' in the paragraph, and the expression T(F-hat sub K) in the displayed formula theta-hat sub K = T(F-hat sub K). The last line of the definition reads: where F-hat sub K (x) = integral from minus infinity to x of f-hat(u) du.", "caption": "P. 123 (PDF p. 141), the sentence before Definition 3.3.9 and the definition. The marked words apply a statistical functional $T$, which Definition 3.3.2 defines on CDFs, to the integrated estimate $\\hat F_K$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N011-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T58", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T58"}, {"id": "P001-T59", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T59"}], "headline": "The kernel plug-in estimator of Definition 3.3.9 requires a kernel that is nonnegative almost everywhere: for a kernel with negative values, which Definition 3.3.7 admits, the integrated estimate $\\hat F_K$ need not be a CDF.", "id": "P001-C03-N011", "location": "Definition 3.3.7, p. 121 (PDF p. 139), the conditions on the kernel $K$; read with Definition 3.3.9, p. 123 (PDF p. 141), and Definition 3.3.2, p. 118 (PDF p. 136)", "pdf_page": 139, "printed_page": "121", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T58", "P001-T59"], "text": {"as_printed": "Definition 3.3.7 (p.~121): ``Let $X_1, X_2, \\ldots, X_n$ be i.i.d.\\ continuous random variables with common PDF $f$. Let $K:\\mathbb{R}\\to\\mathbb{R}$ be \\hl{an even function}$^{32}$ satisfying $\\hl{\\int_{-\\infty}^{\\infty}K(x)dx=1}$, and let $K_h(x)=\\frac{1}{h}K(\\frac{x}{h})$, $\\forall x\\in\\mathbb{R}$ and $h>0$. Then a \\emph{kernel density estimator} of $f(x)$ is\n\\[\n\\hat f_K(x)=\\frac{1}{n}\\sum_{i=1}^{n}K_h(x-X_i),\\ \\forall x\\in\\mathbb{R}.\\text{''}\n\\]\n\nP.~123, before Definition 3.3.9: ``We can integrate our estimate of the PDF to obtain an estimate of the full CDF, and thus we can apply any statistical functional to that estimate.''\n\nDefinition 3.3.9 (p.~123): ``The \\emph{kernel plug-in estimator} of $\\theta=T(F)$ is\n\\[\n\\hat\\theta_K=T\\bigl(\\hat F_K\\bigr),\n\\]\nwhere $\\hat F_K(x)=\\int_{-\\infty}^{x}\\hat f(u)du$.''\n\nDefinition 3.3.2 (p.~118): ``A \\emph{statistical functional} is a function $T:\\mathcal{F}\\to\\mathbb{R}$, where $\\mathcal{F}$ is the set of all possible CDFs.''", "corrected": "Definition 3.3.9. Let $X_1, X_2, \\ldots, X_n$ be i.i.d.\\ continuous random variables with common PDF $f$ and CDF $F$, let $K:\\mathbb{R}\\to\\mathbb{R}$ be an even function satisfying $\\int_{-\\infty}^{\\infty}K(x)\\,dx=1$, let $h>0$, and let $\\hat f_K$ be the kernel density estimator of Definition 3.3.7. \\hl{For kernel plug-in estimation, assume additionally that $K$ is nonnegative almost everywhere.} The \\emph{kernel plug-in estimator} of $\\theta=T(F)$ is\n\\[\n\\hat\\theta_K=T\\bigl(\\hat F_K\\bigr),\n\\]\nwhere $\\hat F_K(x)=\\int_{-\\infty}^{x}\\hat f_K(u)\\,du$.\n\nThe subscript $K$ on the integrand is the correction of \\fid{P001-C03-N012}. The condition may instead be added to Definition 3.3.7, which then excludes kernels with negative values from density estimation as well.", "evidence": "The evidence is an exact counterexample, computed in rational arithmetic, and a derivation for the repair. Define\n\\[\nK(t)=\\begin{cases}1,&|t|\\le 1,\\\\ -1/2,&1<|t|\\le 2,\\\\ 0,&|t|>2.\\end{cases}\n\\]\nSince $K$ is even and $\\int_{-\\infty}^{\\infty}K(t)\\,dt=-1/2+2-1/2=1$, the function $K$ satisfies the conditions of Definition 3.3.7. Set $H(t)=\\int_{-\\infty}^{t}K(u)\\,du$. Integrating piece by piece, I find $H(-2)=0$, $H(-1)=-1/2$, $H(1)=3/2$ and $H(2)=1$. Let $n=1$ and $h=1$, and let $a$ be the value of the observation $X_1$. Then $\\hat f_K(x)=K(x-a)$ and $\\hat F_K(x)=H(x-a)$, and I have\n\\[\n\\hat F_K(a-2)=0>\\hat F_K(a-1)=-\\frac{1}{2},\\qquad \\hat F_K(a+1)=\\frac{3}{2}.\n\\]\nNote that this holds for all real $a$.\n\nThe estimator can also be undefined for every sample. Let $T(G)=\\sqrt{G(0)}$, which is defined for any CDF $G$. Let $f(a)=4(1-4|a-3/2|)$ for $5/4\\le a\\le 7/4$ and $f(a)=0$ otherwise, so that $f$ is a continuous density and $T(F)=0$. For $1\\le a\\le 2$, I have $H(-a)=a/2-1$. For $h=1$, every $n$ and every sample from $f$, I obtain\n\\[\n\\hat F_K(0)=\\frac{1}{n}\\sum_{i=1}^{n}H(-X_i)\\in\\Bigl[-\\frac{3}{8},-\\frac{1}{8}\\Bigr].\n\\]\nFor such a sample, $T(\\hat F_K)$ is the square root of a negative number.\n\nSuppose now that $K\\ge 0$ almost everywhere. Substituting $v=(x-X_i)/h$, I find that each function $x\\mapsto K_h(x-X_i)$ is nonnegative almost everywhere and integrates to one, and the same holds for their average $\\hat f_K$. This implies that $\\hat F_K$ is nondecreasing and continuous, with limit $0$ at $-\\infty$ and limit $1$ at $+\\infty$. I conclude that $\\hat F_K$ is a CDF for every sample and every $h>0$.", "problem": "Definition 3.3.7 admits every even function $K$ with integral one, including a function $K$ that takes negative values. Recall that Definition 3.3.2 defines a statistical functional $T$ on the set of CDFs only. If $K$ takes negative values, then the integrated estimate $\\hat F_K$ of Definition 3.3.9 can decrease, can be negative and can exceed one. In that case $\\hat F_K$ is not a CDF and $T(\\hat F_K)$ is not defined. The four kernels of Figure 3.3.2 are nonnegative, but the definitions are not restricted to them.", "what_survives": "The four kernels of Figure 3.3.2 (rectangular, normal, triangular and Epanechnikov) are nonnegative, and the missing condition does not affect Theorem 3.3.8, which concerns the rectangular kernel. Plug-in estimation with the empirical CDF is unaffected, as is plug-in estimation by maximum likelihood, which Section 5.2.4 (p.~197) introduces by analogy with Definition 3.3.9. A kernel with negative values can still be used to estimate a density alone. A search of the text of the book finds no simulation, empirical estimate or reported number computed with such a kernel. What changes is the claim on p.~123 that any statistical functional can be applied to the integrated estimate, together with the general plug-in discussion on p.~124 that continues it. Both need the estimated distribution to be a CDF, or a stated extension of the functional to functions that are not CDFs."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N011", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the shaded box of Definition 3.3.9, Kernel Plug-In Estimator. A red box marks the symbol f-hat, printed with no subscript, in the last line: where F-hat sub K (x) = integral from minus infinity to x of f-hat(u) du.", "caption": "Definition 3.3.9, p. 123 (PDF p. 141). The marked integrand is printed as $\\hat f$, without the subscript $K$ of the kernel density estimator $\\hat f_K$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N012.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T60", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T60"}], "headline": "The integrand in Definition 3.3.9 is the kernel density estimator $\\hat f_K$; the subscript $K$ is missing.", "id": "P001-C03-N012", "location": "Definition 3.3.9, p. 123 (PDF p. 141), last line", "pdf_page": 141, "printed_page": "123", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T60"], "text": {"as_printed": "``The \\emph{kernel plug-in estimator} of $\\theta=T(F)$ is\n\\[\n\\hat\\theta_K=T\\bigl(\\hat F_K\\bigr),\n\\]\nwhere $\\hat F_K(x)=\\int_{-\\infty}^{x}\\hl{\\hat f}(u)du$.''", "corrected": "``The \\emph{kernel plug-in estimator} of $\\theta=T(F)$ is\n\\[\n\\hat\\theta_K=T\\bigl(\\hat F_K\\bigr),\n\\]\nwhere $\\hat F_K(x)=\\int_{-\\infty}^{x}\\hl{\\hat f_K}(u)du$.''", "evidence": "The evidence is a check of pp.~119, 121 and 123.", "problem": "The integrand is printed as $\\hat f$. The kernel density estimator is written $\\hat f_K$ in Definition 3.3.7 and in the proof sketch on the same page, and $\\hat f$ without a subscript denotes the empirical PMF in Example 3.3.4 (p.~119).", "what_survives": "The definition as intended is unchanged, and nothing depends on the misprint. A condition on the kernel $K$ in the same definition is the subject of \\fid{P001-C03-N011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N012", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text. Red boxes mark the word 'implies' and the expression 'p converges in distribution to U(0,1)' in the sentence 'Loosely speaking, however, asymptotic validity in the context of p-values implies that, under the null hypothesis that theta = theta 0, as n tends to infinity, p converges in distribution to U(0,1).'", "caption": "First paragraph after Theorem 3.4.6, p. 130 (PDF p. 148). The marked word presents the uniform limit, also marked, as a consequence of asymptotic validity.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N014.png"}], "formal_release": null, "formal_statements": [], "headline": "The limit $p\\xrightarrow{d}U(0,1)$ under the null hypothesis is a property of the three $p$-values of Theorem 3.4.6; asymptotic validity does not imply it in general.", "id": "P001-C03-N014", "location": "First paragraph after Theorem 3.4.6, p. 130 (PDF p. 148), sentence beginning “Loosely speaking”", "pdf_page": 148, "printed_page": "130", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``We omit the proof of this theorem, as well as a formal definition of asymptotic validity. Loosely speaking, however, asymptotic validity in the context of $p$-values \\hl{implies} that, under the null hypothesis that $\\theta=\\theta_0$, as $n\\to\\infty$, $p\\xrightarrow{d}U(0,1)$.''", "corrected": "``Loosely speaking, \\hl{under the null hypothesis that $\\theta=\\theta_0$, the $p$-values in Theorem 3.4.6 satisfy $p\\xrightarrow{d}U(0,1)$ as $n\\to\\infty$}.''", "intended_meaning": "The sentence describes the $p$-values of Theorem 3.4.6, and for them it is correct. Under the null hypothesis that $\\theta=\\theta_0$, the $t$-statistic satisfies $t\\xrightarrow{d}Z$ with $Z$ standard normal. The three $p$-values are $\\Phi(t)$, $1-\\Phi(t)$ and $2(1-\\Phi(|t|))$, and they converge in distribution to $\\Phi(Z)$, $1-\\Phi(Z)$ and $2(1-\\Phi(|Z|))$. Each limit is uniform on $(0,1)$; for the third, I have $\\Pr[2(1-\\Phi(|Z|))\\le u]=\\Pr[|Z|\\ge\\Phi^{-1}(1-u/2)]=u$ for $0<u<1$.\n\nAs a statement about validity in general, ``implies'' is too strong. A valid $p$-value may be conservative. The $p$-value that equals $1$ for every sample satisfies $\\Pr[p\\le\\alpha]=0\\le\\alpha$ for all $\\alpha\\in(0,1)$ and all $n$, and its limit is a point mass at $1$. Validity requires only that $\\limsup_{n\\to\\infty}\\Pr[p\\le\\alpha]\\le\\alpha$. Theorem 3.4.6 and its three formulas are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N014", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of three lines of text. A red box marks the words 'one-to-one' in the sentence 'One direct implication of Theorem 3.4.6 is that an asymptotically valid p-value is given by a one-to-one function of t.' The next sentence reads 'For example, if |t| = 1.96, then an asymptotically valid two-tailed p-value is given by p = 0.05.'", "caption": "Second paragraph after Theorem 3.4.6, p. 130 (PDF p. 148). The marked words call the $p$-value a one-to-one function of $t$; the example in the next sentence is two-tailed and depends on $|t|$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N015.png"}], "formal_release": null, "formal_statements": [], "headline": "The two-tailed $p$-value of Theorem 3.4.6 is a function of $t$, but not a one-to-one function: $t$ and $-t$ give the same $p$.", "id": "P001-C03-N015", "location": "Second paragraph after Theorem 3.4.6, p. 130 (PDF p. 148), sentence beginning “One direct implication”", "pdf_page": 148, "printed_page": "130", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``One direct implication of Theorem 3.4.6 is that an asymptotically valid $p$-value is given by a \\hl{one-to-one} function of $t$. For example, if $|t|=1.96$, then an asymptotically valid two-tailed $p$-value is given by $p=0.05$.''", "corrected": "``One direct implication of Theorem 3.4.6 is that an asymptotically valid $p$-value is given by a \\del{one-to-one} function of $t$.''", "intended_meaning": "The sentence means that, by Theorem 3.4.6, the $p$-value is a function of $t$ once the tail is chosen. The example and the formula of the theorem use $|t|$ correctly. The two-tailed $p$-value is $g(t)=2(1-\\Phi(|t|))$. Note that $g(t)=g(-t)$: both $t=1.96$ and $t=-1.96$ give $p=0.049996$, which the book rounds to $0.05$. The function $g$ is one-to-one in $|t|$, and the one-tailed $p$-values $\\Phi(t)$ and $1-\\Phi(t)$ are one-to-one in $t$. No number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N015", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the numbered list of four bootstrap steps and the four lines after it. Red boxes mark the sentence 'Given the usual plug-in regularity conditions (Section 3.3.1), this standard deviation will consistently estimate the standard error.' Step 4 of the list reads 'Using the resulting collection of bootstrap estimates, calculate the standard deviation of the bootstrap distribution of our estimator.'", "caption": "Section 3.4.3, p. 131 (PDF p. 149), the four bootstrap steps and the sentence after them. The marked sentence gives the usual plug-in regularity conditions as sufficient for the bootstrap standard deviation to be consistent.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N016.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T61", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T61"}, {"id": "P001-T62", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T62"}], "headline": "The bootstrap standard deviation is consistent for the standard error under the usual plug-in regularity conditions together with a moment condition on the bootstrap distribution; the regularity conditions alone do not suffice.", "id": "P001-C03-N016", "location": "Section 3.4.3, p. 131 (PDF p. 149), the sentence after the four bootstrap steps", "pdf_page": 149, "printed_page": "131", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T61", "P001-T62"], "text": {"as_printed": "``\\hl{Given the usual plug-in regularity conditions (Section 3.3.1)}, this standard deviation will consistently estimate the standard error.''", "corrected": "``Given the usual plug-in regularity conditions (Section 3.3.1) \\hl{and a moment condition on the bootstrap distribution}, this standard deviation will consistently estimate the standard error.''\n\nIn full: let $X_1,\\ldots,X_n$ be i.i.d.\\ with CDF $F$, let $\\theta=T(F)$ and $\\hat\\theta=T(\\hat F)$, and suppose that $\\sqrt n(\\hat\\theta-\\theta)\\xrightarrow{d}N(0,\\phi^2)$ with $0<\\phi<\\infty$. Let $\\hat\\theta^*$ be the estimate computed from a sample of size $n$ drawn with replacement from $X_1,\\ldots,X_n$, let $\\mathrm{E}_*$ and $\\mathrm{V}_*$ denote expectation and variance conditional on $X_1,\\ldots,X_n$, and set $Z_n^*=\\sqrt n(\\hat\\theta^*-\\hat\\theta)$. Suppose that the conditional distribution of $Z_n^*$ converges weakly to $N(0,\\phi^2)$ in probability, and that for every $\\varepsilon>0$,\n\\[\n\\lim_{K\\to\\infty}\\limsup_{n\\to\\infty}\\Pr\\Bigl[\\mathrm{E}_*\\bigl[(Z_n^*)^2\\mathbf{1}\\{|Z_n^*|>K\\}\\bigr]>\\varepsilon\\Bigr]=0.\\qquad\\text{(M)}\n\\]\nThen $\\sqrt n\\sqrt{\\mathrm{V}_*[\\hat\\theta^*]}\\xrightarrow{p}\\phi$.\n\nCondition (M) holds with no further assumption when $\\hat\\theta$ is the sample mean of a distribution with finite variance. When $\\hat\\theta$ is the sample median and $F$ has a positive continuous density near its unique median, (M) holds if $\\mathrm{E}|X|^{\\eta}<\\infty$ for some $\\eta>0$. If the standard deviation is computed from finitely many bootstrap samples, the conclusion also requires that $n$ times the error of the Monte Carlo variance tends to zero in probability.", "evidence": "The evidence is an exact counterexample and a derivation. Let $X_1,\\ldots,X_n$ be i.i.d.\\ with CDF $F$, and suppose that $\\sqrt n(\\hat\\theta-\\theta)\\xrightarrow{d}N(0,\\phi^2)$ with $0<\\phi<\\infty$. Let $\\hat\\theta^*$ be the estimate computed from a sample of size $n$ drawn with replacement from $X_1,\\ldots,X_n$. I write $\\mathrm{E}_*$ and $\\mathrm{V}_*$ for the expectation and the variance conditional on $X_1,\\ldots,X_n$, and I set $Z_n^*=\\sqrt n(\\hat\\theta^*-\\hat\\theta)$. At the scale of Definition 3.2.32, the printed sentence states that $n\\mathrm{V}_*[\\hat\\theta^*]\\xrightarrow{p}\\phi^2$.\n\nFor the counterexample, let $U$ be uniform on $(0,1)$ and independently choose one of three components: let $X$ be uniform on $[-1,1]$ with probability $1/2$, equal to $2^{1/U^2}$ with probability $1/4$, and equal to $-2^{1/U^2}$ with probability $1/4$. The distribution of $X$ is symmetric, its unique median is $0$, its density equals $1/4$ on $[-1,1]$, and $\\Pr[X\\ge 2^{j^2}]=1/(4j)$ for all integers $j\\ge 1$. The median is Hadamard differentiable at $F$ along the directions that are continuous at $0$, which are the usual directions for a quantile. The sample median $M_n$ satisfies $\\sqrt n M_n\\xrightarrow{d}N(0,4)$, and the conditional distribution of $\\sqrt n(M_n^*-M_n)$ converges to $N(0,4)$ in probability. The finite-sample median has infinite second moment here, so it has no finite sampling variance; the example concerns the finite variance of its limiting law, which Definition 3.2.32 uses. Fix an odd $n\\ge 5$, and let $A_n$ be the event that the sample contains an observation $a\\ge 2^{n^2}$ and an observation $b\\in[-1,1]$. By Bonferroni's inequality, I have\n\\[\n\\Pr[A_n]\\ge\\frac{7}{32}+\\frac{1}{32n}-2^{-n}\\ge\\frac{3}{16}.\n\\]\nOn $A_n$, the bootstrap sample that repeats $a$ in all $n$ draws has probability $n^{-n}$ and median $a$, and the bootstrap sample that repeats $b$ has the same probability and median $b$. Using $n\\le 2^{n-1}$, I obtain\n\\[\n\\mathrm{V}_*[M_n^*]\\ge n^{-2n}\\bigl(2^{n^2}-1\\bigr)^2\\ge 2^{2n-2}.\n\\]\nI conclude that $\\Pr\\bigl[n\\mathrm{V}_*[M_n^*]>5\\bigr]\\ge 3/16$ for all odd $n\\ge 5$, while the target is $4$.\n\nA second functional shows that the failure does not depend on the directions allowed in the derivative. Let $T_0(G)=G(0)-1/2$ if $1/4\\le G(0)\\le 3/4$, and let $T_0(G)$ be the median of $G$ otherwise. The functional $T_0$ is affine on the CDFs within Kolmogorov distance $1/4$ of $F$, and it is Hadamard differentiable at $F$ in every direction. For the same distribution of $X$, I have $\\sqrt n\\,T_0(\\hat F)\\xrightarrow{d}N(0,1/4)$, the conditional distribution of the bootstrap statistic has the same normal limit, and the bound above holds with $T_0$ in place of the median. Its target is $1/4$.\n\nFor the repair, suppose that the conditional distribution of $Z_n^*$ converges weakly to $N(0,\\phi^2)$ in probability and that condition (M) of the corrected statement below holds. The functions $z\\mapsto\\min(z^2,K^2)$ and $z\\mapsto\\max(-K,\\min(z,K))$ are bounded and continuous, so their conditional expectations at $Z_n^*$ converge in probability to the corresponding expectations under $N(0,\\phi^2)$. Removing the truncation changes these conditional expectations by at most $\\mathrm{E}_*[(Z_n^*)^2\\mathbf{1}\\{|Z_n^*|>K\\}]$ and $K^{-1}$ times this quantity, respectively. Letting $n\\to\\infty$ and then $K\\to\\infty$, and using (M), I find that $\\mathrm{E}_*[Z_n^*]\\xrightarrow{p}0$ and $\\mathrm{E}_*[(Z_n^*)^2]\\xrightarrow{p}\\phi^2$, which implies that $n\\mathrm{V}_*[\\hat\\theta^*]\\xrightarrow{p}\\phi^2$. Conversely, if the weak limit holds and the conditional second moments are finite, then this convergence implies (M).\n\nFor the sample median, the abstract of Ghosh, Parr, Singh and Babu, ``A note on bootstrapping the sample median'', \\emph{Annals of Statistics} 12 (1984), 1130--1135 (\\url{https://doi.org/10.1214/aos/1176346731}), says that the conjecture of almost-sure convergence of the bootstrap variance estimator to the asymptotic variance is ``false unless a tail condition is imposed on $F$''.", "problem": "Section 3.3.1 (p.~120) calls ``the usual plug-in regularity conditions'' the conditions under which the functional $T$ is well behaved at $F$, and footnote 31 defines ``well behaved'' as Hadamard differentiable at $F$ with respect to the Kolmogorov distance. These conditions concern the estimand $\\theta=T(F)$. They control limiting distributions: with positive limiting variance, they give asymptotic normality of $\\hat\\theta=T(\\hat F)$ as in the correction of \\fid{P001-C03-E002}. They do not control the second moment of the bootstrap distribution. The standard deviation in step 4 is consistent, at the scale of Definition 3.2.32, only if that second moment converges, and this requires a moment condition. Without one, a few extreme bootstrap samples can keep the bootstrap variance far from its target while the bootstrap distribution converges to the correct normal limit.", "what_survives": "The four bootstrap steps are unchanged, as is the bootstrap approximation to the sampling distribution. Bootstrap standard errors of sample means and of cluster means with finite variance are consistent with no further assumption. The moment condition also holds for the uniform and Bernoulli designs of Section 3.4.4, and for the sample means of bounded variables that give the bounds in Sections 6.3 and 7.5. Theorems 3.4.2 and 3.4.6 assume a consistent variance estimator and stand as stated. Percentile intervals use only quantiles of the bootstrap distribution, and this point does not affect them. No standard error, estimate or conclusion in the book is shown to be wrong. What changes is the general assurance on p.~131 and the statements that use the same rationale: the block bootstrap for an arbitrary estimator (p.~141), and the bootstrap standard errors for regression (pp.~152 and 154) and for maximum likelihood (p.~203). Each needs the moment condition. For the estimators in the applications that refit a regression, a propensity score or a match in every bootstrap sample, the book does not give enough to decide whether the condition holds."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N016", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the four lines of footnote 42. A red box marks the words 'Example 7.2.4 (Matching Estimation)' in the last sentence, 'We will briefly discuss a practical situation in Example 7.2.4 (Matching Estimation) where the bootstrap may also fail.'", "caption": "Footnote 42, p. 131 (PDF p. 149). The marked reference gives Example 7.2.4 the title “Matching Estimation”; the example is titled “Propensity Score Matching” and the section “Matching”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N017.png"}], "formal_release": null, "formal_statements": [], "headline": "Footnote 42 should refer to Section 7.2.4 (Matching): Example 7.2.4 is titled “Propensity Score Matching”, and the remark on the bootstrap is in the text of the section after the example.", "id": "P001-C03-N017", "location": "Footnote 42, p. 131 (PDF p. 149), last sentence", "pdf_page": 149, "printed_page": "131", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``We will briefly discuss a practical situation in \\hl{Example 7.2.4 (\\emph{Matching Estimation})} where the bootstrap may also fail.''", "corrected": "``We will briefly discuss a practical situation in \\hl{Section 7.2.4 (\\emph{Matching})} where the bootstrap may also fail.''", "evidence": "The evidence is a check of pp.~131, 262 and 264. A search of the text of the book finds ``Matching Estimation'' only in this footnote.", "problem": "No item of the book is titled ``Matching Estimation''. On p.~262, Section 7.2.4 is titled ``Matching'' and Example 7.2.4 is titled ``Propensity Score Matching''. The statement that the usual methods for constructing confidence intervals, including the bootstrap, do not generally work is on p.~264, in the text of Section 7.2.4 after the example.", "what_survives": "The rest of the footnote is unchanged, and the printed reference still points to the right pages."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N017", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of text. A red box marks 'Theorem 3.2.34' in the sentence 'Theorem 3.2.34 establishes that the standard error of the sample mean is sigma[X-bar] = the square root of V[X]/n.'", "caption": "Section 3.4.4, p. 132 (PDF p. 150), second paragraph. The marked reference calls item 3.2.34 a theorem; it is a definition, and the formula also uses Theorem 3.2.4.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N018.png"}, {"alt": "Crop of four lines of text. A red box marks 'Theorem 3.2.34' in the sentence 'The second column reports the true standard error (which can be exactly derived via Theorem 3.2.34).'", "caption": "Example 3.4.7, p. 133 (PDF p. 151), last paragraph of the page. The same reference recurs in the description of the second column of Table 3.4.1.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N018-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Item 3.2.34 is a definition, and the formula $\\sigma[\\overline{X}]=\\sqrt{\\mathrm{V}[X]/n}$ follows from Theorem 3.2.4 together with Definition 3.2.34.", "id": "P001-C03-N018", "location": "Section 3.4.4, p. 132 (PDF p. 150), second paragraph; repeated in Example 3.4.7, p. 133 (PDF p. 151), last paragraph of the page", "pdf_page": 150, "printed_page": "132", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~132: ``\\hl{Theorem 3.2.34} establishes that the standard error of the sample mean is $\\sigma[\\overline{X}]=\\sqrt{\\mathrm{V}[X]/n}$.''\n\nP.~133: ``The second column reports the true standard error (which can be exactly derived via \\hl{Theorem 3.2.34}).''", "corrected": "P.~132: ``\\hl{Theorem 3.2.4 together with Definition 3.2.34} establishes that the standard error of the sample mean is $\\sigma[\\overline{X}]=\\sqrt{\\mathrm{V}[X]/n}$.''\n\nP.~133: ``The second column reports the true standard error (which can be exactly derived via \\hl{Theorem 3.2.4 and Definition 3.2.34}).''", "evidence": "The evidence is a check of pp.~98, 115, 132 and 133.", "problem": "Item 3.2.34 (p.~115) is Definition 3.2.34, \\emph{Standard Error of the Sample Mean}, which states that $\\sigma[\\overline{X}]=\\sqrt{\\mathrm{V}[\\overline{X}]}$. The identity $\\mathrm{V}[\\overline{X}]=\\mathrm{V}[X]/n$ is Theorem 3.2.4 (p.~98). Substituting the second into the first, I obtain the printed formula, which is correct.", "what_survives": "The formula and the second column of Table 3.4.1 are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N018", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of a paragraph of seven lines. A red box marks the symbol E[X] in the sentence 'Alternatively, we can think about the estimator as the average of i.i.d. random variables, with mean E[X] and common variance V[W].' The rest of the paragraph says that a version of the WLLN still holds for the cluster sample mean when the number of clusters l tends to infinity with m not growing in n.", "caption": "Paragraph before Theorem 3.5.6, p. 138 (PDF p. 156). The marked symbol is the mean of the i.i.d. random variables that the sentence describes; the same sentence gives their variance as $\\mathrm{V}[W]$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C03-N019.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T63", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T63"}], "headline": "The i.i.d. random variables averaged by the cluster sample mean are the cluster averages $W_i$, and their mean is $\\mathrm{E}[W]$, not $\\mathrm{E}[X]$.", "id": "P001-C03-N019", "location": "Paragraph between the proof of Theorem 3.5.5 and Theorem 3.5.6, p. 138 (PDF p. 156), first sentence", "pdf_page": 156, "printed_page": "138", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T63"], "text": {"as_printed": "``Alternatively, we can think about the estimator as the average of i.i.d.\\ random variables, with mean $\\mathrm{E}[\\hl{X}]$ and common variance $\\mathrm{V}[W]$.''", "corrected": "``Alternatively, we can think about the estimator as the average of i.i.d.\\ random variables, with mean $\\mathrm{E}[\\hl{W}]$ and common variance $\\mathrm{V}[W]$.''\n\nHere the estimator is the cluster sample mean $\\overline{X}$ of Definition 3.5.2, and the i.i.d.\\ random variables are the cluster averages $W_1,W_2,\\ldots,W_l$ of Definition 3.5.1.", "evidence": "The evidence is a check of pp.~136--138 and a short derivation. Definition 3.5.1 sets $W_i=(1/m)\\sum_{j=1}^{m}X_{ij}$, so that\n\\[\n\\mathrm{E}[W_i]=\\frac{1}{m}\\sum_{j=1}^{m}\\mathrm{E}[X_{ij}].\n\\]\nTheorem 3.5.4 states that $\\mathrm{E}[\\overline{X}]=\\mathrm{E}[W]$, and Theorem 3.5.6 states that $\\overline{X}\\xrightarrow{p}\\mathrm{E}[W]$. An example shows that the coordinates need not have a common mean. Let $m=2$, let $U_1,U_2,\\ldots,U_l$ be independent Bernoulli random variables with parameter $1/2$, and set $(X_{i1},X_{i2})=(U_i,U_i+2)$. Then $W_i=U_i+1$ and $\\mathrm{E}[W]=3/2$, while $\\mathrm{E}[X_{i1}]=1/2$ and $\\mathrm{E}[X_{i2}]=5/2$.", "problem": "The estimator in the sentence is the cluster sample mean $\\overline{X}$, and Theorem 3.5.3 shows that $\\overline{X}$ equals the sample mean $\\overline{W}$ of the cluster averages $W_1,W_2,\\ldots,W_l$. The cluster averages are the i.i.d.\\ random variables that the sentence describes, and their common mean is $\\mathrm{E}[W]$. The symbol $\\mathrm{E}[X]$ has no meaning in the cluster model: the random vectors $(X_{i1},X_{i2},\\ldots,X_{im})$ are i.i.d.\\ across clusters $i$, but the coordinates $X_{ij}$ need not have the same distribution across $j$.", "what_survives": "Theorems 3.5.4 and 3.5.6 are stated with $\\mathrm{E}[W]$, and the rest of the paragraph is unchanged. Nothing depends on the misprint."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C03-N019", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The last part of a shaded definition box: the display of beta-hat as an arg min over b in R to the power J_n + 1, and below it the line 'where J_n tends to infinity and (1/n) J_n tends to 0', with the two conditions in a red box. Below the box, a paragraph of ten lines. Two further red boxes, one at the end of a line and one at the start of the next, mark the words 'will be consistent across regions of the data with nonzero probability mass'.", "caption": "End of Definition 4.3.9 and the paragraph that follows it, p. 169 (PDF p. 187): the two conditions on the degree and the claim of consistency are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T5", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T5"}, {"id": "P001-T6", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T6"}, {"id": "P001-T70", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T70"}, {"id": "P001-T119", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T119"}], "headline": "The conditions $J_n\\to\\infty$ and $J_n/n\\to0$ do not imply that the polynomial least squares sieve estimator is consistent; the claim holds under a stronger condition on $J_n$, which for uniform $X$ reads $(J_n+1)^3/n\\to0$.", "id": "P001-C04-E001", "location": "Definition 4.3.9, pp. 168–169 (PDF pp. 186–187), and the paragraph that follows it on p. 169", "pdf_page": 186, "printed_page": "168–169", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T119", "P001-T5", "P001-T6", "P001-T70"], "text": {"as_printed": "\\textbf{Definition 4.3.9.} \\emph{Polynomial Least Squares Sieve Estimator}\nFor i.i.d. random vectors $(Y_1,X_1)$, $(Y_2,X_2)$, \\ldots, $(Y_n,X_n)$, the \\emph{polynomial least squares sieve estimator} of the conditional expectation function is\n\\[\n\\hat{\\mathrm{E}}_n[Y\\mid X=x]=\\sum_{k=0}^{J_n}\\hat{\\beta}_kx^k,\n\\]\nwhere\n\\[\n\\hat{\\boldsymbol{\\beta}}=\\underset{\\mathbf{b}\\in\\mathbb{R}^{J_n+1}}{\\arg\\min}\\sum_{i=1}^n\\left(Y_i-\\sum_{k=0}^{J_n}b_kX_i^k\\right),\n\\]\nwhere $J_n\\to\\infty$ and $\\hl{\\tfrac{1}{n}J_n\\to0}$.\n\n[p.~169, next paragraph] As long as the CEF is smooth with appropriate support across $X$, then $\\hat{\\mathrm{E}}[Y\\mid X=x]$ \\hl{will be consistent across regions of the data with nonzero probability mass} (by the logic of Theorem 4.3.3, \\emph{Polynomial Approximation of the CEF}).", "corrected": "\\textbf{Definition 4.3.9.} \\emph{Polynomial Least Squares Sieve Estimator}\nFor i.i.d. random vectors $(Y_1,X_1)$, $(Y_2,X_2)$, \\ldots, $(Y_n,X_n)$, the \\emph{polynomial least squares sieve estimator} of the conditional expectation function is\n\\[\n\\hat{\\mathrm{E}}_n[Y\\mid X=x]=\\sum_{k=0}^{J_n}\\hat{\\beta}_kx^k,\n\\]\nwhere\n\\[\n\\hat{\\boldsymbol{\\beta}}=\\underset{\\mathbf{b}\\in\\mathbb{R}^{J_n+1}}{\\arg\\min}\\sum_{i=1}^n\\left(Y_i-\\sum_{k=0}^{J_n}b_kX_i^k\\right)^2,\n\\]\nwhere $J_n\\to\\infty$ and $\\hl{\\zeta_{J_n+1}^2\\,(J_n+1)/n\\to0}$. Here, for each $K\\ge1$, $p_K=(p_0,\\ldots,p_{K-1})^T$ is a basis of the polynomials of degree at most $K-1$ such that $\\mathrm{E}[p_K(X)p_K(X)^T]=I_K$, and $\\zeta_K^2=\\sup_{x\\in\\operatorname{Supp}[X]}\\|p_K(x)\\|^2$. If $X$ is uniform on a compact interval, then $\\zeta_K^2=K^2$, and the condition reads \\hl{$(J_n+1)^3/n\\to0$}.\n\n[p.~169, next paragraph] As long as the CEF is \\hl{continuous}, \\hl{$\\operatorname{Supp}[X]$ is a compact interval} and \\hl{$\\mathrm{E}[(Y-\\mathrm{E}[Y\\mid X])^2]<\\infty$}, then $\\hat{\\mathrm{E}}[Y\\mid X=x]$ will be consistent \\hl{in mean square}, that is,\n\\[\n\\int\\bigl(\\hat{\\mathrm{E}}[Y\\mid X=x]-\\mathrm{E}[Y\\mid X=x]\\bigr)^2\\,dF(x)\\to0\\quad\\text{in probability,}\n\\]\nwhere $F$ is the CDF of $X$, and so it will be consistent across regions of the data with nonzero probability mass. \\hl{Theorem 4.3.3, \\emph{Polynomial Approximation of the CEF}, bounds the error of approximation, and the condition on $J_n$ bounds the sampling error.}\n\nThe square in the objective is the correction \\fid{P001-C04-E003}.", "evidence": "The evidence is an exact counterexample to the printed conditions and a derivation of the corrected ones.\n\nLet $(X_i,Y_i)$ be i.i.d.\\ with $X_i\\sim U[0,1]$ and $Y_i\\sim N(0,1)$ independent of $X_i$, so that $m(x)=\\mathrm{E}[Y\\mid X=x]=0$ and a polynomial of any degree represents $m$ exactly. Set $J_n=\\lfloor n^{3/4}\\rfloor$. Then $J_n\\to\\infty$ and $J_n/n\\le n^{-1/4}\\to0$. Consider $n=t^4$ for integers $t\\ge2$, so that the degree is $d=t^3$, and write $\\hat m_n$ for the least squares fit. Set $a=1/(2n)$, and let $B_t$ be the event that $X_i\\ge a$ for all $i$. By a union bound, I have $\\Pr(B_t)\\ge1-na=1/2$. Let $T_d$ be the Chebyshev polynomial of degree $d$, and define\n\\[\nr_t(x)=T_d\\!\\left(\\frac{2x-1-a}{1-a}\\right).\n\\]\nFor $x\\in[a,1]$, I have $|r_t(x)|\\le1$, so that $\\sum_ir_t(X_i)^2\\le n$ on $B_t$. For $x\\in[0,a/4]$, the bound $T_d(1+u)\\ge d^6u^3/360$, valid for $u\\ge0$ and $d\\ge3$, implies that $|r_t(x)|\\ge3t^6/2560$. Fix $\\delta\\in(0,1]$ and define\n\\[\nZ=\\left(\\int_0^\\delta r_t(x)^2\\,dx\\right)^{-1/2}\\int_0^\\delta\\hat m_n(x)\\,r_t(x)\\,dx.\n\\]\nConditional on $X_1,\\ldots,X_n$, the coefficients of $\\hat m_n$ are jointly normal with mean zero, and so is $Z$. By the Cauchy--Schwarz inequality for the matrix of sums of squares and cross products of the regressors, I have on $B_t$, for all $t$ such that $a/4\\le\\delta$,\n\\[\n\\operatorname{Var}(Z\\mid X_1,\\ldots,X_n)\\ge\\frac{\\int_0^\\delta r_t(x)^2\\,dx}{\\sum_ir_t(X_i)^2}\\ge\\frac{1}{n}\\cdot\\frac{a}{4}\\left(\\frac{3t^6}{2560}\\right)^2=\\frac{9t^4}{52428800}=V_t.\n\\]\nBy the Cauchy--Schwarz inequality for integrals, I have $|Z|\\le\\bigl(\\int_0^\\delta\\hat m_n(x)^2\\,dx\\bigr)^{1/2}$. Further, a normal variable with mean zero and variance $V$ lies in $[-\\eta,\\eta]$ with probability at most $2\\eta/\\sqrt{2\\pi V}$. Since $V_t\\to\\infty$, I obtain, for every $\\varepsilon>0$,\n\\[\n\\liminf_{t\\to\\infty}\\Pr\\left(\\int_0^\\delta\\hat m_{t^4}(x)^2\\,dx>\\varepsilon\\right)\\ge\\frac{1}{2}.\n\\]\n\n\\begin{audittable}{rrrrrr}{Exact lower bounds $V_t$ on the conditional variance of $Z$ in the counterexample, along $n=t^4$ with $J_n=t^3$. The printed condition holds, since $J_n/n\\to0$, and the bound grows without limit.}\n$t$ & $n=t^4$ & $J_n=t^3$ & $J_n/n$ & $V_t=9t^4/52428800$ & $V_t$ as a decimal \\\\\n\\hline\n$2$ & $16$ & $8$ & $1/2$ & $9/3276800$ & $2.75\\times10^{-6}$ \\\\\n$4$ & $256$ & $64$ & $1/4$ & $9/204800$ & $4.39\\times10^{-5}$ \\\\\n$8$ & $4{,}096$ & $512$ & $1/8$ & $9/12800$ & $7.03\\times10^{-4}$ \\\\\n$16$ & $65{,}536$ & $4{,}096$ & $1/16$ & $9/800$ & $0.01125$ \\\\\n$32$ & $1{,}048{,}576$ & $32{,}768$ & $1/32$ & $9/50$ & $0.18$ \\\\\n$64$ & $16{,}777{,}216$ & $262{,}144$ & $1/64$ & $72/25$ & $2.88$ \\\\\n$128$ & $268{,}435{,}456$ & $2{,}097{,}152$ & $1/128$ & $1152/25$ & $46.08$ \\\\\n$256$ & $4{,}294{,}967{,}296$ & $16{,}777{,}216$ & $1/256$ & $18432/25$ & $737.28$ \\\\\n$1024$ & $1{,}099{,}511{,}627{,}776$ & $1{,}073{,}741{,}824$ & $1/1024$ & $4718592/25$ & $188{,}743.68$ \\\\\n\\end{audittable}\n\nThe constants in $V_t$ come from crude bounds. The argument uses only the divergence of $V_t$, not its size at a given $n$. A second derivation with other constants gives $\\Pr\\bigl(\\int_0^{1/2}\\hat m_n(x)^2\\,dx>1\\bigr)\\ge1/48$ for every $n\\ge372827023$, and it applies also when $Y_i=\\pm1$ with probability $1/2$ each. In the same example the average of $\\hat m_n(X_i)^2$ over the observations has conditional expectation $(J_n+1)/n$ and tends to zero in probability. The fit is consistent in average squared error at the observed points, while population mean-square consistency fails. The counterexample does not show that the fit fails at every interior point, and it does not identify the fastest growth of $J_n$ that preserves consistency.\n\nFor the corrected conditions, set $K=J_n+1$ and $\\hat G=n^{-1}\\sum_ip_K(X_i)p_K(X_i)^T$, with $p_K$ and $\\zeta_K$ as in the corrected definition below. Since $\\mathrm{E}[p_K(X)p_K(X)^T]=I_K$, I have\n\\[\n\\mathrm{E}\\bigl\\|\\hat G-I_K\\bigr\\|_F^2=\\frac{1}{n}\\Bigl(\\mathrm{E}\\|p_K(X)\\|^4-K\\Bigr)\\le\\frac{\\zeta_K^2K}{n}\\to0,\n\\]\nwhich implies that the smallest eigenvalue of $\\hat G$ is at least $1/2$ with probability tending to one. Fix $\\varepsilon>0$ and a polynomial $q$ such that $\\sup_x|m(x)-q(x)|\\le\\varepsilon$, which exists by the Weierstrass Approximation Theorem. On that event, for all large $n$,\n\\[\n\\int\\bigl(\\hat m_n(x)-m(x)\\bigr)^2\\,dF(x)\\le9\\varepsilon^2+12\\|w\\|^2,\\qquad w=\\frac{1}{n}\\sum_{i=1}^np_K(X_i)\\bigl(Y_i-m(X_i)\\bigr),\n\\]\nwhere $F$ is the CDF of $X$, and $\\mathrm{E}\\|w\\|^2\\le\\zeta_K^2\\sigma^2/n\\to0$ with $\\sigma^2=\\mathrm{E}[(Y-m(X))^2]$. Since $\\varepsilon$ is arbitrary, the integral tends to zero in probability. For $X$ uniform on $[0,1]$, the polynomials $p_k(x)=\\sqrt{2k+1}\\,P_k(2x-1)$, where $P_k$ is the Legendre polynomial of degree $k$, satisfy $\\mathrm{E}[p_j(X)p_k(X)]=\\mathbf{1}\\{j=k\\}$ and $\\zeta_K^2=\\sum_{k=0}^{K-1}(2k+1)=K^2$. The condition is sufficient; I do not claim that it is necessary.", "problem": "The paragraph presents the two conditions on $J_n$, with a smooth CEF and suitable support, as enough for the estimated CEF to be consistent over regions of nonzero probability. They are not enough. Theorem 4.3.3, to which the text appeals, concerns the best polynomial predictor in the population: it states that $\\mathrm{E}[(\\mathrm{E}[Y\\mid X]-g(X,X^2,\\ldots,X^K))^2]$ is small for large $K$, where $g$ is the BLP of $Y$ given $X,X^2,\\ldots,X^K$. It does not bound the sampling error of a least squares fit whose number of coefficients grows with $n$, and this error need not vanish when $J_n/n\\to0$. Let $X$ be uniform on $[0,1]$, let $Y$ be standard normal and independent of $X$, so that the CEF is zero, and set $J_n=\\lfloor n^{3/4}\\rfloor$. Both printed conditions hold. For every $\\delta\\in(0,1]$, the integral of the squared fit over $[0,\\delta]$, a region of probability $\\delta$, does not converge to zero in probability. The sentence on p.~168 that the estimator is consistent if flexibility grows ``slowly'' enough is correct; the error is to present $J_n/n\\to0$ as slow enough. (The printed objective also lacks a square, which is the subject of \\fid{P001-C04-E003}. I restore the square throughout.)", "what_survives": "Theorem 4.3.3, the consistency of OLS with a fixed specification, and the statement on p.~168 that the estimator is consistent when flexibility grows slowly enough are unchanged. If the CEF is continuous, the support is compact and the conditional variance is bounded, then under the printed conditions the average squared error at the observed values of $X$ still converges to zero in probability. The regressions of Section 4.4 use fixed specifications with $n=184$, and I found no later displayed theorem invoking Definition 4.3.9 or its conditions on $J_n$. What changes is the condition on $J_n$ in the definition and the hypotheses of the claim of consistency that follows it."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Four lines of text and a shaded box with Definition 4.3.7. The definition displays PDA with subscript X[k] as the partial derivative of E[Y|X] with respect to X[k], followed by a vertical bar. A red box marks the subscript of the bar, X = E[X].", "caption": "Definition 4.3.7, p. 167 (PDF p. 185), with the paragraph that introduces it: the point of evaluation $\\mathbf{X}=\\mathrm{E}[\\mathbf{X}]$ is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E002.png"}, {"alt": "A paragraph of eight lines that compares the average partial derivative with the partial derivative at the average. A red box marks the words 'has a well-defined meaning' at the start of the fourth line; the sentence continues 'it is limited to just one particular point on the probability distribution of the explanatory variables'.", "caption": "First paragraph of p. 168 (PDF p. 186): the words “has a well-defined meaning” are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E002-p168.png"}, {"alt": "A line chart titled 'The same distribution of (Y, X) gives a partial derivative at the average of 0 or of 1, depending on the extension', with ticks from -2 to 2 on the horizontal axis, labeled as a value of the explanatory variable X, and the conditional mean of Y on the vertical axis. Two shaded bands, from -2 to -1 and from 1 to 2, are labeled 'X has density 1/2', and a thick black horizontal line at height 0 runs across each band. Between -1 and 1, labeled 'no observations', a solid blue line stays at 0 and a dashed orange curve dips to about -0.29 near x = -0.45, passes through 0 at x = 0 and rises to about 0.29 near x = 0.45 before returning to 0 at x = 1. A dotted vertical line marks E[X] = 0. Labels read 'slope of m1 at E[X]: 1' and 'slope of m0 at E[X]: 0'.", "caption": "The counterexample to Definition 4.3.7 (p. 167). The explanatory variable $X$ has density $1/2$ on $[-2,-1]\\cup[1,2]$ (shaded), and $\\mathrm{E}[Y\\mid X=x]=0$ there (black). Two continuously differentiable extensions across the interval with no observations, $m_0$ (blue, solid) and $m_1$ (orange, dashed), have derivatives $0$ and $1$ at $\\mathrm{E}[X]=0$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E002-extensions.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T3", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T3"}, {"id": "P001-T4", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T4"}, {"id": "P001-T102", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T102"}], "headline": "Definition 4.3.7 evaluates the derivative of the CEF at $\\mathrm{E}[\\mathbf{X}]$ without requiring that the derivative be defined there and determined by the distribution of $(Y,\\mathbf{X})$; a sufficient condition is that $\\mathrm{E}[\\mathbf{X}]$ have an open neighborhood inside the support of $\\mathbf{X}$ on which the CEF is continuous, and that the CEF be differentiable at $\\mathrm{E}[\\mathbf{X}]$.", "id": "P001-C04-E002", "location": "Definition 4.3.7, p. 167 (PDF p. 185); first paragraph of p. 168 (PDF p. 186)", "pdf_page": 185, "printed_page": "167", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T102", "P001-T3", "P001-T4"], "text": {"as_printed": "\\textbf{Definition 4.3.7.} \\emph{Partial Derivative at the Average}\nFor a random variable $Y$ and random vector $\\mathbf{X}$, the \\emph{partial derivative at the average} of the CEF (of $Y$ given $\\mathbf{X}$) with respect to $X_{[k]}$ is\n\\[\nPDA_{X_{[k]}}=\\left.\\frac{\\partial\\mathrm{E}[Y\\mid\\mathbf{X}]}{\\partial X_{[k]}}\\right|_{\\hl{\\scriptstyle\\mathbf{X}=\\mathrm{E}[\\mathbf{X}]}}.\n\\]\n\n[p.~168] While the partial derivative at the average also \\hl{has a well-defined meaning}, it is limited to just one particular point on the probability distribution of the explanatory variables.", "corrected": "\\textbf{Definition 4.3.7.} \\emph{Partial Derivative at the Average}\nFor a random variable $Y$ and random vector $\\mathbf{X}$, write $m(\\mathbf{x})=\\mathrm{E}[Y\\mid\\mathbf{X}=\\mathbf{x}]$ for the CEF. Suppose that $\\boldsymbol{\\mu}=\\mathrm{E}[\\mathbf{X}]$ exists and \\hl{has an open neighborhood $U$ contained in $\\operatorname{Supp}[\\mathbf{X}]$}, and that \\hl{$m$ has a version that is continuous on $U$ and differentiable with respect to $x_{[k]}$ at $\\boldsymbol{\\mu}$}. Then the \\emph{partial derivative at the average} of the CEF (of $Y$ given $\\mathbf{X}$) with respect to $X_{[k]}$ is\n\\[\nPDA_{X_{[k]}}=\\frac{\\partial m}{\\partial x_{[k]}}(\\boldsymbol{\\mu}).\n\\]\n\n[p.~168] While the partial derivative at the average also has a well-defined meaning \\hl{under these conditions}, it is limited to just one particular point on the probability distribution of the explanatory variables. \\hl{A full open neighborhood is not necessary in every case. The derivative at $\\mathrm{E}[\\mathbf{X}]$ of a chosen extrapolating predictor can still be reported, but is not in general determined by the distribution of $(Y,\\mathbf{X})$.}", "evidence": "The evidence is an exact counterexample to the printed definition and a derivation of the corrected one.\n\nLet $X$ have density $1/2$ on $S=[-2,-1]\\cup[1,2]$, and let $Y\\sim U(-1,1)$ be independent of $X$. Then $\\mathrm{E}[X]=0$, which is not in $S$, and the interval $(-1,1)$ has probability zero. For $x\\in S$, I have $\\mathrm{E}[Y\\mid X=x]=0$. Define $m_0(x)=0$ for all $x$, and define $m_1(x)=x(1-x^2)^2$ for $|x|<1$ and $m_1(x)=0$ for $|x|\\ge1$. Both functions equal the CEF at every point of $S$. Set $h(x)=x(1-x^2)^2=x-2x^3+x^5$, so that $h'(x)=1-6x^2+5x^4$. Since $h(\\pm1)=0$ and $h'(\\pm1)=0$, the function $m_1$ is continuously differentiable on $\\mathbb{R}$. At the mean,\n\\[\nm_0'(0)=0,\\qquad m_1'(0)=h'(0)=1,\n\\]\nand replacing $h$ by $ah$ gives the value $a$ for any real $a$. The distribution of $(Y,X)$ is the same whichever extension is used, and each extension has average partial derivative $0$ and mean squared prediction error $\\mathrm{E}[Y^2]=1/3$. A statistic of an i.i.d.\\ sample that converged in probability to both $0$ and $1$ would lie within $1/3$ of each with probability tending to one, which is impossible.\n\nFor a vector, let $(X_1,X_2)$ be uniform on $[-2,2]^2\\setminus(-1,1)^2$, and let $Y\\sim U(-1,1)$ be independent. The support is connected, each coordinate is supported on $[-2,2]$, and the mean $(0,0)$ lies in the hole. The CEF is $0$ on the support. The function equal to $x_1(1-x_1^2)^2(1-x_2^2)^2$ on $(-1,1)^2$ and to $0$ elsewhere is continuously differentiable, and its partial derivative in $x_1$ at $(0,0)$ is $1$; the zero extension gives $0$.\n\nUnder the corrected definition the derivative is determined by the distribution. Let $m$ and $\\tilde m$ be two versions of the CEF that are continuous on the open set $U$. Then $m(\\mathbf{X})=\\tilde m(\\mathbf{X})$ almost surely. If $m$ and $\\tilde m$ differed at a point of $U$, they would differ on a ball around it, and this ball has positive probability because $U\\subseteq\\operatorname{Supp}[\\mathbf{X}]$. Therefore $m=\\tilde m$ on $U$, the difference quotients of $m$ and $\\tilde m$ at $\\boldsymbol{\\mu}$ coincide, and so do the derivatives.", "problem": "The book defines the conditional expectation function (CEF) only at points of the support of $\\mathbf{X}$ (Definitions 2.2.10 and 2.2.15 on pp.~67 and 69, and Definitions 2.3.3 and 2.3.4 on p.~85). Footnote 13 on p.~166 adds that $\\mathbf{X}$ is continuous and that the CEF is differentiable at every value of $\\mathbf{X}$ with positive support. None of these conditions places $\\mathrm{E}[\\mathbf{X}]$ in the support. When $\\mathrm{E}[\\mathbf{X}]$ lies in a region where $\\mathbf{X}$ has no probability, the CEF has no value there, and a derivative at $\\mathrm{E}[\\mathbf{X}]$ is a derivative of whichever extension of the CEF one chooses. Different differentiable extensions give different derivatives, while the distribution of $(Y,\\mathbf{X})$ is the same. In that case the quantity in Definition 4.3.7 is not a property of the distribution of $(Y,\\mathbf{X})$, and the data cannot determine it. The sentence on p.~168, which says that the partial derivative at the average ``has a well-defined meaning'' at a point ``on the probability distribution of the explanatory variables'', also needs a derivative defined and determined by the distribution. A gap in the support of one explanatory variable is not the only way this can happen: the support of $\\mathbf{X}$ can be connected, with every explanatory variable supported on a full interval, and still omit a neighborhood of $\\mathrm{E}[\\mathbf{X}]$.", "what_survives": "Example 4.3.8 is unaffected: its mean $(1/2,1/2)$ is an interior point of the support $[0,1]^2$, and $PDA_{X_{[1]}}=-1$ and $APD_{X_{[1]}}=1$ hold. Definition 4.3.6 does not evaluate the CEF at the mean and is unchanged. The values in the PDA row of Table 4.4.5 (p.~173) are derivatives of the fitted specifications at the sample means, and no number there is shown to be wrong. What changes is their interpretation as estimates of a partial derivative at the average of the CEF, which needs a derivative defined and determined by the distribution; the corrected definition gives sufficient conditions. Since the data for the application are not available, I cannot say whether these conditions hold there. Outside this discussion and the application of Section 4.4, I found no substantive use of the partial derivative at the average and no theorem that depends on it."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The top of a shaded definition box. The display reads beta-hat equals the arg min over b in R to the power J_n + 1 of the sum over i from 1 to n of (Y_i minus the sum over k from 0 to J_n of b_k X_i to the power k). A red box marks the parenthesized residual, which carries no exponent. The next line reads 'where J_n tends to infinity and (1/n) J_n tends to 0'.", "caption": "Definition 4.3.9 (continued), p. 169 (PDF p. 187): the residual in the objective is marked; no exponent follows the closing parenthesis.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T5", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T5"}, {"id": "P001-T70", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T70"}, {"id": "P001-T119", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T119"}], "headline": "The objective in Definition 4.3.9 is the sum of squared residuals; the printed display omits the square.", "id": "P001-C04-E003", "location": "Definition 4.3.9 (continued), p. 169 (PDF p. 187), display that defines $\\hat{\\boldsymbol{\\beta}}$", "pdf_page": 187, "printed_page": "169", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T119", "P001-T5", "P001-T70"], "text": {"as_printed": "\\[\n\\hat{\\boldsymbol{\\beta}}=\\underset{\\mathbf{b}\\in\\mathbb{R}^{J_n+1}}{\\arg\\min}\\sum_{i=1}^n\\hl{\\left(Y_i-\\sum_{k=0}^{J_n}b_kX_i^k\\right)},\n\\]", "corrected": "\\[\n\\hat{\\boldsymbol{\\beta}}=\\underset{\\mathbf{b}\\in\\mathbb{R}^{J_n+1}}{\\arg\\min}\\sum_{i=1}^n\\left(Y_i-\\sum_{k=0}^{J_n}b_kX_i^k\\right)^{\\hl{\\scriptstyle2}},\n\\]", "evidence": "The evidence is a derivation and a check of the page. Set $b_1=\\cdots=b_{J_n}=0$. Since $X_i^0=1$, the printed objective equals $\\sum_{i=1}^nY_i-nb_0$, which tends to $-\\infty$ as $b_0\\to\\infty$, for every $n\\ge1$. The objective is unbounded below, and the arg min does not exist. The corresponding displays of Definition 4.1.2 (p.~147) and Definition 4.3.10 (p.~170) print the square.", "problem": "The printed objective sums the residuals without squaring them. It is then not a least squares criterion, and it has no minimizer. The name of the estimator and the text around the definition show that the sum of squared residuals is meant.", "what_survives": "The polynomial least squares estimator that the text describes is unchanged. Restoring the square does not repair the conditions on $J_n$ in the same definition, which are corrected in \\fid{P001-C04-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-E003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Three lines at the foot of a page: the heading 'Definition 4.3.10. Lasso Estimator' and two lines of text. Red boxes mark 'X =' at the end of the second line and '(1, X[1], X[2], ..., X[K])' at the start of the third.", "caption": "Start of Definition 4.3.10, p. 169 (PDF p. 187): the vector $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, which has $K+1$ entries, is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E004-p169.png"}, {"alt": "The top of a shaded definition box. The display reads beta-hat-lambda equals the arg min over b in R to the power K of (1/n) times the sum of (Y_i minus X_i b) squared, plus lambda times the norm of b with subscript 1. A red box marks 'b in R^K' under the arg min. A second red box, in the line below, marks 'norm of b with subscript 1 equals the sum from k = 1 to K of the absolute value of b_k'.", "caption": "Definition 4.3.10 (continued), p. 170 (PDF p. 188): the domain $\\mathbf{b}\\in\\mathbb{R}^K$ and the line that defines $\\|\\mathbf{b}\\|_1$ are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T71", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T71"}], "headline": "In Definition 4.3.10 the minimization is over $\\mathbb{R}^{K+1}$, and the penalty is the $\\ell^1$ norm of the slope coefficients, without the intercept.", "id": "P001-C04-E004", "location": "Definition 4.3.10, pp. 169–170 (PDF pp. 187–188): the vector of explanatory variables, the arg min display and the line that defines the norm", "pdf_page": 187, "printed_page": "169–170", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T71"], "text": {"as_printed": "[p.~169] For i.i.d. random vectors $(Y_1,\\mathbf{X}_1)$, $(Y_2,\\mathbf{X}_2)$, \\ldots, $(Y_n,\\mathbf{X}_n)$ where $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, the \\emph{lasso estimator} with \\emph{penalty parameter} $\\lambda$ is\n\n[p.~170]\n\\[\n\\hat{\\boldsymbol{\\beta}}_\\lambda=\\underset{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^K}}{\\arg\\min}\\left(\\frac{1}{n}\\sum_{i=1}^n(Y_i-\\mathbf{X}_i\\mathbf{b})^2+\\lambda\\|\\mathbf{b}\\|_1\\right),\n\\]\nwhere $\\|\\cdot\\|_1$ is the $\\ell^1$ norm, that is, $\\hl{\\textstyle\\|\\mathbf{b}\\|_1=\\sum_{k=1}^K|b_k|}$ (the sum of the absolute value of the coefficients).", "corrected": "[p.~169] For i.i.d. random vectors $(Y_1,\\mathbf{X}_1)$, $(Y_2,\\mathbf{X}_2)$, \\ldots, $(Y_n,\\mathbf{X}_n)$ where $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, the \\emph{lasso estimator} with \\emph{penalty parameter} $\\lambda$ is\n\n[p.~170]\n\\[\n\\hat{\\boldsymbol{\\beta}}_\\lambda=\\underset{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K+1}}}{\\arg\\min}\\left(\\frac{1}{n}\\sum_{i=1}^n(Y_i-\\mathbf{X}_i\\mathbf{b})^2+\\lambda\\hl{\\sum_{k=1}^K|b_k|}\\right),\n\\]\nwhere \\hl{$\\mathbf{b}=(b_0,b_1,\\ldots,b_K)^T$ and $b_0$, the coefficient of the constant, is not penalized}. The penalty $\\sum_{k=1}^K|b_k|$ is the $\\ell^1$ norm of \\hl{the slope coefficients $(b_1,\\ldots,b_K)$} (the sum of the absolute value of the \\hl{slope} coefficients).", "evidence": "The evidence is a derivation and a check of the two pages. For $K=1$, the row $\\mathbf{X}_i=(1,X_{[1]i})$ has two entries and cannot multiply a vector in $\\mathbb{R}^1$. For $\\mathbf{b}=(1,0,\\ldots,0)^T\\in\\mathbb{R}^{K+1}$, the printed sum equals $0$, whereas $\\|\\mathbf{b}\\|_1=1$. A norm does not vanish at a nonzero vector, and the sum is not a norm on $\\mathbb{R}^{K+1}$. Definition 4.1.2 (p.~147), with the same vector of explanatory variables, takes its arg min over $\\mathbb{R}^{K+1}$.", "problem": "The vector $\\mathbf{X}=(1,X_{[1]},\\ldots,X_{[K]})$ has $K+1$ entries, and the arg min is taken over $\\mathbf{b}\\in\\mathbb{R}^K$. The product $\\mathbf{X}_i\\mathbf{b}$ is then not defined. Once $\\mathbf{b}$ has $K+1$ entries, the sum $\\sum_{k=1}^K|b_k|$ omits the coefficient of the constant and is not the $\\ell^1$ norm of $\\mathbf{b}$. The text before the definition says that the penalty typically excludes the intercept. The sum is the intended penalty, and the name is wrong.", "what_survives": "The lasso with an unpenalized intercept, its equality with OLS at $\\lambda=0$ and the discussion of the choice of $\\lambda$ are unchanged. Only the domain of the arg min and the description of the penalty change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-E004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of one line of text, 'We can stack all n of these e i's to form a system of equations', and three displayed equations separated by vertical dots: e 1 = Y 1 - (...), e 2 = Y 2 - (...), and e n = Y 3 - (b 0 + b 1 X[1]n + b 2 X[2]n + ... b K X[K]n). A red box marks the symbol Y 3 in the last equation.", "caption": "Proof of Theorem 4.1.4, p. 148 (PDF p. 166), the stacked system of equations. The marked outcome in the equation for $e_n$ has the subscript $3$; every regressor in that equation has the subscript $n$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T65", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T65"}], "headline": "The last equation of the stacked system is the residual of observation $n$, and its outcome is $Y_n$, not $Y_3$.", "id": "P001-C04-E005", "location": "Proof of Theorem 4.1.4, p. 148 (PDF p. 166), last line of the stacked system of equations", "pdf_page": 166, "printed_page": "148", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T65"], "text": {"as_printed": "\\[\ne_n=Y_{\\hl{3}}-\\bigl(b_0+b_1X_{[1]n}+b_2X_{[2]n}+\\ldots b_KX_{[K]n}\\bigr).\n\\]", "corrected": "\\[\ne_n=Y_{\\hl{n}}-\\bigl(b_0+b_1X_{[1]n}+b_2X_{[2]n}+\\ldots+b_KX_{[K]n}\\bigr).\n\\]\n\nThe plus sign before the last term is the correction of \\fid{P001-C04-N002}.", "evidence": "The evidence is a check of p.~148. The proof defines $e_i=Y_i-\\bigl(b_0+b_1X_{[1]i}+b_2X_{[2]i}+\\ldots+b_KX_{[K]i}\\bigr)$ for every observation $i$. Substituting $i=n$, I obtain the outcome $Y_n$. The matrix equation directly below the stacked system has $Y_n$ in the last row of the vector of outcomes.", "problem": "On p.~148, the last equation of the stacked system uses $Y_3$ where the last row of a sample of size $n$ requires $Y_n$. The equation is the residual of observation $n$: its left side is $e_n$, and every regressor on its right side has the subscript $n$.", "what_survives": "The matrix equation $\\mathbf{e}=\\mathbf{Y}-\\mathbb{X}\\mathbf{b}$, the normal equations and the OLS solution of Theorem 4.1.4 are unaffected, since the proof continues from the matrix equation, which is correct."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-E005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Part of a figure with two three-dimensional surface plots of infant mortality against access to clean water and electricity. Above the plots are the labels (5) OLS: alpha plus beta X plus gamma Z plus omega X Z, and (6) OLS: alpha plus beta X plus gamma Z plus a double sum, over kappa from 1 to 2 and l from 1 to 2, of omega kappa l times X to the kappa times Z to the l. Below the plots are the labels (7) OLS and (8) Lasso, with the same formula and kappa running from 1 to 3. Red boxes mark the terms beta X plus gamma Z in labels (6), (7) and (8).", "caption": "Figure 4.4.4, p. 176 (PDF p. 194), lower part: the labels of panels (5) and (6), and panels (7) and (8) with their labels. In labels (6), (7) and (8) the marked terms $\\beta X+\\gamma Z$ are the only nonconstant terms that do not contain both variables; the pure powers $X^2$, $X^3$ and $Z^2$ do not appear.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E006.png"}, {"alt": "Two formula labels. On the left, (3) OLS: alpha plus beta X plus gamma Z. On the right, (4) OLS: alpha plus the sum over kappa from 1 to 2 of beta sub kappa times X to the kappa, plus the sum over kappa from 1 to 2 of gamma kappa Z to the kappa. A red box marks gamma kappa, where the kappa is printed at full size on the line and not as a subscript.", "caption": "Figure 4.4.4, p. 176 (PDF p. 194), labels of panels (3) and (4). In label (4) the marked $\\kappa$ after $\\gamma$ is set on the line, where the subscript of $\\gamma_\\kappa$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E006-b.png"}, {"alt": "Upper part of the regression table for cross-national infant mortality, with columns (1) to (8) and rows Intercept, Water, Water squared, Water cubed, Elec and Elec squared, standard errors in parentheses. Red boxes mark, in columns (6), (7) and (8), the Water squared coefficients -0.006, -0.076 and 0.000; in columns (7) and (8), the Water cubed coefficients 0.001 and 0.000; and in columns (6), (7) and (8), the Elec squared coefficients -0.004, -0.010 and -0.003.", "caption": "Table 4.4.5, p. 173 (PDF p. 191), upper rows. The marked coefficients on Water$^2$, Water$^3$ and Elec$^2$ in columns (6)–(8) belong to pure powers that the labels of Figure 4.4.4 omit.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E006-c.png"}, {"alt": "Text of Example 4.3.5, Overfitting with a Nonlinear CEF. It supposes that Y equals g of X plus epsilon for a nonlinear function g, with X uniform on minus 10 to 10, epsilon normal with mean 0 and variance 25, and X independent of epsilon. A red box marks g(X), with a capital X, in the sentence: The CEF of Y given X is thus E[Y|X = x] = g(X). A second red box marks Figure 4.3.4 in the sentence: Let g be the function shown in black in Figure 4.3.4. The next sentence begins: As in Figure 4.3.4, panel (a) Figure 4.3.5 plots ten random draws.", "caption": "Example 4.3.5, p. 163 (PDF p. 181). The marked $g(X)$ should be $g(x)$, and the marked reference should be to Figure 4.3.5.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-E006-d.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T68", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T68"}], "headline": "The labels of panels (6)–(8) of Figure 4.4.4 omit the pure powers of $X$ and $Z$ that Table 4.4.5 reports for the same regressions; label (4) has a misplaced subscript, and Example 4.3.5 should read $g(x)$ and refer to Figure 4.3.5.", "id": "P001-C04-E006", "location": "Figure 4.4.4, labels of panels (4) and (6)–(8), p. 176 (PDF p. 194), against Table 4.4.5, columns (4) and (6)–(8), p. 173 (PDF p. 191); Example 4.3.5, p. 163 (PDF p. 181)", "pdf_page": 194, "printed_page": "176", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T68"], "text": {"as_printed": "Figure 4.4.4, p.~176, labels of panels (4), (6), (7) and (8):\n\\[\n\\begin{aligned}\n&\\text{(4) OLS: }\\alpha+\\sum_{\\kappa=1}^{2}\\beta_\\kappa X^\\kappa+\\sum_{\\kappa=1}^{2}\\hl{\\gamma\\kappa}Z^\\kappa\\\\\n&\\text{(6) OLS: }\\alpha+\\hl{\\beta X+\\gamma Z}+\\sum_{\\kappa=1}^{2}\\sum_{l=1}^{2}\\omega_{\\kappa l}X^\\kappa Z^l\\\\\n&\\text{(7) OLS: }\\alpha+\\hl{\\beta X+\\gamma Z}+\\sum_{\\kappa=1}^{3}\\sum_{l=1}^{2}\\omega_{\\kappa l}X^\\kappa Z^l\\\\\n&\\text{(8) Lasso: }\\alpha+\\hl{\\beta X+\\gamma Z}+\\sum_{\\kappa=1}^{3}\\sum_{l=1}^{2}\\omega_{\\kappa l}X^\\kappa Z^l\n\\end{aligned}\n\\]\nExample 4.3.5, p.~163:\n\nSuppose we had a random vector $(X,Y)$ such that\n\\[\nY=g(X)+\\varepsilon,\n\\]\nfor some (nonlinear) function $g$, where $X\\sim U(-10,10)$, $\\varepsilon\\sim N(0,25)$, and $X\\perp\\!\\!\\!\\perp\\varepsilon$. The CEF of $Y$ given $X$ is thus $\\mathrm{E}[Y|X=x]=\\hl{g(X)}$. Let $g$ be the function shown in black in \\hl{Figure 4.3.4}.", "corrected": "Figure 4.4.4, labels of panels (4), (6), (7) and (8), with $X$ the percentage with access to clean water and $Z$ the percentage with access to electricity:\n\\[\n\\begin{aligned}\n&\\text{(4) OLS: }\\alpha+\\sum_{\\kappa=1}^{2}\\beta_\\kappa X^\\kappa+\\sum_{\\kappa=1}^{2}\\hl{\\gamma_\\kappa}Z^\\kappa\\\\\n&\\text{(6) OLS: }\\alpha+\\hl{\\sum_{\\kappa=1}^{2}\\beta_\\kappa X^\\kappa+\\sum_{l=1}^{2}\\gamma_lZ^l}+\\sum_{\\kappa=1}^{2}\\sum_{l=1}^{2}\\omega_{\\kappa l}X^\\kappa Z^l\\\\\n&\\text{(7) OLS: }\\alpha+\\hl{\\sum_{\\kappa=1}^{3}\\beta_\\kappa X^\\kappa+\\sum_{l=1}^{2}\\gamma_lZ^l}+\\sum_{\\kappa=1}^{3}\\sum_{l=1}^{2}\\omega_{\\kappa l}X^\\kappa Z^l\\\\\n&\\text{(8) Lasso: }\\alpha+\\hl{\\sum_{\\kappa=1}^{3}\\beta_\\kappa X^\\kappa+\\sum_{l=1}^{2}\\gamma_lZ^l}+\\sum_{\\kappa=1}^{3}\\sum_{l=1}^{2}\\omega_{\\kappa l}X^\\kappa Z^l\n\\end{aligned}\n\\]\nThe highlighted sums replace the terms $\\beta X+\\gamma Z$; they are not added to them.\n\nExample 4.3.5, p.~163:\n\nSuppose we had a random vector $(X,Y)$ such that\n\\[\nY=g(X)+\\varepsilon,\n\\]\nfor some (nonlinear) function $g$, where $X\\sim U(-10,10)$, $\\varepsilon\\sim N(0,25)$, and $X\\perp\\!\\!\\!\\perp\\varepsilon$. The CEF of $Y$ given $X$ is thus $\\mathrm{E}[Y|X=x]=\\hl{g(x)}$. Let $g$ be the function shown in black in \\hl{Figure 4.3.5}.", "evidence": "The evidence is a check of the pages and an exact calculation from the printed coefficients. For any coefficients, the formula in labels (6)--(8) reduces at $X=0$ to $h(z)=\\alpha+\\gamma z$, and the second difference $h(0)-2h(1)+h(2)$ of a linear function is zero. For $h(z)=a+bz+cz^2$ the same difference is $2c$. From the coefficients printed in Table 4.4.5, I obtain at $X=0$ the polynomials $T(0,z)$ in the table below. Each has a nonzero second difference, and I conclude that no choice of $\\alpha$, $\\beta$, $\\gamma$ and $\\omega_{\\kappa l}$ in the labels reproduces the polynomial of the table.\n\\begin{audittable}{llrrrr}{Table 4.4.5 at $X=0$, computed exactly from the printed coefficients. Every formula of the printed form has second difference $0$.}\nColumn & $T(0,z)$ & $T(0,0)$ & $T(0,1)$ & $T(0,2)$ & $T(0,0)-2T(0,1)+T(0,2)$ \\\\\n(6) & $67.385+0.137z-0.004z^2$ & $67.385$ & $67.518$ & $67.643$ & $-0.008$ \\\\\n(7) & $50.900+0.990z-0.010z^2$ & $50.900$ & $51.880$ & $52.840$ & $-0.020$ \\\\\n(8) & $68.631+0.062z-0.003z^2$ & $68.631$ & $68.690$ & $68.743$ & $-0.006$ \\\\\n\\end{audittable}\nThe pure powers of $X$ are missing in the same way. At $Z=0$ the polynomial of the table is $67.385-0.432x-0.006x^2$ in column (6) and $50.900+1.808x-0.076x^2+0.001x^3$ in column (7). Its second difference in $x$ at $0,1,2$ is $-0.012$ in column (6) and $-0.146$ in column (7), and its third difference at $0,1,2,3$ in column (7) is $0.006$. Each of these differences is zero for the labels. In column (8) the table prints $0.000$ for Water$^2$ and Water$^3$, which does not show that these terms are active; for that label the mismatch follows from the coefficient $-0.003$ on Elec$^2$. For label (4), the literal reading requires the coefficient on Elec$^2$ to be $2\\times0.157=0.314$, and the table prints $-0.004$. For Example 4.3.5, since $\\varepsilon$ is independent of $X$ and has mean zero, I have\n\\[\n\\mathrm{E}[Y|X=x]=\\mathrm{E}[g(X)|X=x]+\\mathrm{E}[\\varepsilon|X=x]=g(x)+\\mathrm{E}[\\varepsilon]=g(x).\n\\]\nI did not refit the regressions, because the data are not available.", "problem": "Here $X$ is the percentage with access to clean water and $Z$ the percentage with access to electricity, and the six panels of Figure 4.4.4 show the fits of regressions (3)--(8) of Table 4.4.5. In the labels of panels (6)--(8), every term of the double sum has positive powers of both $X$ and $Z$, and none is a pure power $X^2$, $X^3$ or $Z^2$. Table 4.4.5 reports coefficients on these pure powers in the same columns: Water$^2$ and Elec$^2$ in column (6), and Water$^2$, Water$^3$ and Elec$^2$ in columns (7) and (8). Then the labels describe different regressors from the table, and someone who fits the regressions as labelled does not fit the ones that the table reports. The specifications to follow are those of the table. In label (4), the $\\kappa$ after $\\gamma$ is set on the line and not as a subscript; read literally, $\\gamma\\kappa Z^\\kappa$ makes the coefficient on $Z^2$ twice the coefficient on $Z$, which the table contradicts. In Example 4.3.5, the conditional expectation at a fixed $x$ is written as $g(X)$, a random variable. The example also says that $g$ is shown in Figure 4.3.4, but that figure shows the linear CEF of Example 4.3.4, and the nonlinear $g$ is the black curve of Figure 4.3.5.", "what_survives": "These are corrections to labels, a subscript, a letter and a figure number. No coefficient, plotted surface or estimate of the APD or PDA is shown to be wrong, and the discussion of Table 4.4.5 on p.~175 rests on the table, which I have not recomputed. The description of regression (6) on p.~174 lists the interactions that it includes and needs no change. In Example 4.3.5, the next sentence, which begins ``As in Figure 4.3.4'', is a correct comparison and stays as printed."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-E006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three lines of displayed mathematics. The first line reads g(X[1], X[2], ..., X[K]) = beta 0 + beta 1 X[1] + beta 2 X[2] + ... + beta K X[K] such that. The next lines define (beta 0, beta 1, ..., beta K) as the argmin over (b 0, b 1, ..., b K) of E[(Y - (b 0 + b 1 X[1] + b 2 X[2] + ... b K X[K])) squared]. A red box marks the ellipsis and b K, which are printed with no plus sign between them.", "caption": "Definition of the multivariate BLP, p. 146 (PDF p. 164). In the marked place the ellipsis is followed by $b_K$ with no plus sign; the first line of the crop prints the same sum with the sign, “$+\\ldots+\\beta_KX_{[K]}$”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N002.png"}, {"alt": "Crop of the end of the shaded box of Definition 4.1.2 and two lines of text below it. Red boxes mark two places. The first is the ellipsis and b K in the displayed argmin of one over n times the sum over i of (Y i - (b 0 + b 1 X[1]i + b 2 X[2]i + ... b K X[K]i)) squared. The second is the ellipsis and beta-hat K in the sentence 'The quantity e i = Y i - (beta-hat 0 + beta-hat 1 X[1]i + beta-hat 2 X[2]i + ... beta-hat K X[K]i) is known as the i-th residual.' In both places no plus sign follows the ellipsis.", "caption": "Definition 4.1.2 and the sentence after it, p. 147 (PDF p. 165). The two marked places lack the plus sign before the last term; the first line of the crop has it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N002-b.png"}, {"alt": "Crop of four displayed equations and one line of text. The first equation reads e i = Y i - (b 0 + b 1 X[1]i + b 2 X[2]i + ... + b K X[K]i). After the sentence 'We can stack all n of these e i's to form a system of equations', three equations follow, for e 1, e 2 and e n. A red box in each marks the ellipsis and b K, printed with no plus sign between them. The last equation begins e n = Y 3.", "caption": "Proof of Theorem 4.1.4, p. 148 (PDF p. 166). The three equations of the stacked system lack the plus sign in the marked places; the definition of $e_i$ in the first line of the crop has it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N002-c.png"}], "formal_release": null, "formal_statements": [], "headline": "A plus sign is missing before the last term of the linear predictor in six formulas on pp. 146–148.", "id": "P001-C04-N002", "location": "Display defining the multivariate BLP, p. 146 (PDF p. 164); Definition 4.1.2 and the sentence after it, p. 147 (PDF p. 165); proof of Theorem 4.1.4, stacked system of equations, p. 148 (PDF p. 166)", "pdf_page": 164, "printed_page": "146", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~146:\n\\[\n(\\beta_0,\\beta_1,\\ldots,\\beta_K)=\\operatorname*{arg\\,min}_{(b_0,b_1,\\ldots,b_K)\\in\\mathbb{R}^{K+1}}\\mathrm{E}\\Bigl[\\bigl(Y-(b_0+b_1X_{[1]}+b_2X_{[2]}+\\hl{\\ldots b_K}X_{[K]})\\bigr)^2\\Bigr].\n\\]\n\nP.~147, Definition 4.1.2:\n\\[\n\\hat{\\boldsymbol\\beta}=(\\hat\\beta_0,\\hat\\beta_1,\\ldots,\\hat\\beta_K)=\\operatorname*{arg\\,min}_{(b_0,b_1,\\ldots,b_K)\\in\\mathbb{R}^{K+1}}\\frac{1}{n}\\sum_{i=1}^{n}\\bigl(Y_i-(b_0+b_1X_{[1]i}+b_2X_{[2]i}+\\hl{\\ldots b_K}X_{[K]i})\\bigr)^2.\n\\]\n\nP.~147, after Definition 4.1.2: ``The quantity $e_i=Y_i-(\\hat\\beta_0+\\hat\\beta_1X_{[1]i}+\\hat\\beta_2X_{[2]i}+\\hl{\\ldots\\hat\\beta_K}X_{[K]i})$ is known as the $i^{\\text{th}}$ \\emph{residual}.''\n\nP.~148, proof of Theorem 4.1.4:\n\\[\n\\begin{aligned}\ne_1&=Y_1-\\bigl(b_0+b_1X_{[1]1}+b_2X_{[2]1}+\\hl{\\ldots b_K}X_{[K]1}\\bigr)\\\\\ne_2&=Y_2-\\bigl(b_0+b_1X_{[1]2}+b_2X_{[2]2}+\\hl{\\ldots b_K}X_{[K]2}\\bigr)\\\\\n&\\ \\ \\vdots\\\\\ne_n&=Y_3-\\bigl(b_0+b_1X_{[1]n}+b_2X_{[2]n}+\\hl{\\ldots b_K}X_{[K]n}\\bigr).\n\\end{aligned}\n\\]", "corrected": "P.~146:\n\\[\n(\\beta_0,\\beta_1,\\ldots,\\beta_K)=\\operatorname*{arg\\,min}_{(b_0,b_1,\\ldots,b_K)\\in\\mathbb{R}^{K+1}}\\mathrm{E}\\Bigl[\\bigl(Y-(b_0+b_1X_{[1]}+b_2X_{[2]}+\\ldots\\hl{+}b_KX_{[K]})\\bigr)^2\\Bigr].\n\\]\n\nP.~147, Definition 4.1.2:\n\\[\n\\hat{\\boldsymbol\\beta}=(\\hat\\beta_0,\\hat\\beta_1,\\ldots,\\hat\\beta_K)=\\operatorname*{arg\\,min}_{(b_0,b_1,\\ldots,b_K)\\in\\mathbb{R}^{K+1}}\\frac{1}{n}\\sum_{i=1}^{n}\\bigl(Y_i-(b_0+b_1X_{[1]i}+b_2X_{[2]i}+\\ldots\\hl{+}b_KX_{[K]i})\\bigr)^2.\n\\]\n\nP.~147, after Definition 4.1.2: ``The quantity $e_i=Y_i-(\\hat\\beta_0+\\hat\\beta_1X_{[1]i}+\\hat\\beta_2X_{[2]i}+\\ldots\\hl{+}\\hat\\beta_KX_{[K]i})$ is known as the $i^{\\text{th}}$ \\emph{residual}.''\n\nP.~148, proof of Theorem 4.1.4:\n\\[\n\\begin{aligned}\ne_1&=Y_1-\\bigl(b_0+b_1X_{[1]1}+b_2X_{[2]1}+\\ldots\\hl{+}b_KX_{[K]1}\\bigr)\\\\\ne_2&=Y_2-\\bigl(b_0+b_1X_{[1]2}+b_2X_{[2]2}+\\ldots\\hl{+}b_KX_{[K]2}\\bigr)\\\\\n&\\ \\ \\vdots\\\\\ne_n&=Y_n-\\bigl(b_0+b_1X_{[1]n}+b_2X_{[2]n}+\\ldots\\hl{+}b_KX_{[K]n}\\bigr).\n\\end{aligned}\n\\]\n\nThe outcome $Y_n$ in the last equation is the correction of \\fid{P001-C04-E005}.", "evidence": "I checked pp.~146--148 at high magnification. The six formulas are the argmin display on p.~146, the argmin display and the residual on p.~147, and the three equations of the stacked system on p.~148. The signs are absent from the printed page and not only from text extracted from the PDF.", "problem": "In each of the six formulas the ellipsis is followed directly by the last term of the sum, $b_KX_{[K]}$, $b_KX_{[K]i}$ or $\\hat\\beta_KX_{[K]i}$. The sum is the linear predictor, and its last term is added like the others. The same sum is printed with the plus sign in the BLP function on p.~146, in the fitted function at the top of p.~147, and in the definition of $e_i$ before the stacked system on p.~148.", "what_survives": "The linear predictor, the OLS regression estimator and the proof of Theorem 4.1.4 are unchanged. Nothing depends on the missing signs."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of four lines of text. Red boxes mark the bold symbol beta k and the ordinal k-th in the sentence 'the robust standard error estimator for beta k is the square root of the k-th diagonal element of V-hat[beta-hat].'", "caption": "Paragraph after Definition 4.2.1, p. 153 (PDF p. 171). The marked coefficient is set in bold, and the marked ordinal counts diagonal elements from $1$ while the coefficients are numbered from $0$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N003.png"}, {"alt": "Crop of five lines of text. A red box marks the ordinal k-th in the sentence 'The classical standard error estimator for beta k is then just defined as square root of the k-th diagonal element of V-hat C[beta-hat].'", "caption": "Sentence after Definition 4.2.3, p. 155 (PDF p. 173). The marked ordinal is the same as on p. 153.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T66", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T66"}], "headline": "The standard error estimator for $\\hat\\beta_k$ is the square root of the $(k+1)^{\\text{th}}$ diagonal element of the estimated covariance matrix, since the coefficients are numbered from $0$; on p. 153 the scalar coefficient is also set in bold.", "id": "P001-C04-N003", "location": "Paragraph after Definition 4.2.1, p. 153 (PDF p. 171); sentence after Definition 4.2.3, p. 155 (PDF p. 173)", "pdf_page": 171, "printed_page": "153", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T66"], "text": {"as_printed": "P.~153: ``the robust standard error estimator for $\\hl{\\boldsymbol\\beta_k}$ is the square root of the $\\hl{k^{\\text{th}}}$ diagonal element of $\\hat{\\mathrm{V}}[\\hat{\\boldsymbol\\beta}]$.''\n\nP.~155: ``The classical standard error estimator for $\\beta_k$ is then just defined as square root of the $\\hl{k^{\\text{th}}}$ diagonal element of $\\hat{\\mathrm{V}}_{\\mathrm{C}}[\\hat{\\boldsymbol\\beta}]$.''", "corrected": "P.~153: ``the robust standard error estimator for $\\hl{\\hat\\beta_k}$ is the square root of the $\\hl{(k+1)^{\\text{th}}}$ diagonal element of $\\hat{\\mathrm{V}}[\\hat{\\boldsymbol\\beta}]$\\hl{, for $k=0,1,\\ldots,K$}.''\n\nP.~155: ``The classical standard error estimator for $\\hl{\\hat\\beta_k}$ is then just defined as square root of the $\\hl{(k+1)^{\\text{th}}}$ diagonal element of $\\hat{\\mathrm{V}}_{\\mathrm{C}}[\\hat{\\boldsymbol\\beta}]$\\hl{, for $k=0,1,\\ldots,K$}.''\n\nHere $\\hat{\\boldsymbol\\beta}=(\\hat\\beta_0,\\hat\\beta_1,\\ldots,\\hat\\beta_K)$ is the OLS coefficient estimator, and $\\hat{\\mathrm{V}}[\\hat{\\boldsymbol\\beta}]$ and $\\hat{\\mathrm{V}}_{\\mathrm{C}}[\\hat{\\boldsymbol\\beta}]$ are the $(K+1)\\times(K+1)$ matrices of Definitions 4.2.1 and 4.2.3.", "evidence": "The evidence is a derivation from the ordering in Definitions 4.1.2, 4.2.1 and 4.2.3, with an illustration that is not taken from the book. Let $K=1$, and suppose that the estimated covariance matrix has diagonal elements $4$ and $9$. The standard error estimate for the slope $\\hat\\beta_1$ is $3$, the square root of the second diagonal element. Read literally, the sentence takes the first diagonal element, which gives $2$, the standard error estimate for the intercept $\\hat\\beta_0$.", "problem": "Recall that Definition 4.1.2 orders the coefficients as $\\hat{\\boldsymbol\\beta}=(\\hat\\beta_0,\\hat\\beta_1,\\ldots,\\hat\\beta_K)$, and Definitions 4.2.1 and 4.2.3 order the regressors as $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$. The estimated covariance matrix is therefore $(K+1)\\times(K+1)$, with the variance estimate for $\\hat\\beta_0$ in the first diagonal position and the variance estimate for $\\hat\\beta_k$ in position $k+1$. Read literally, ``the $k^{\\text{th}}$ diagonal element'' is the variance estimate for $\\hat\\beta_{k-1}$, and for $k=0$ there is no such element. On p.~153 the scalar coefficient is also printed in bold, the typeface of the vector $\\boldsymbol\\beta$.", "what_survives": "The robust and classical covariance estimators of Definitions 4.2.1 and 4.2.3 are correct as printed, and their interpretation is unchanged. Only the position of the diagonal element for a given coefficient changes. Note that the phrase ``standard error estimator for $\\beta_k$'', without a hat, is common shorthand and not an error; the corrected sentences write $\\hat\\beta_k$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of two lines of text and a display of three lines. The first line of the display reads n V[beta-hat] is approximately (E[X transpose X]) inverse times E[epsilon squared X transpose X] times (E[X i transpose X]) inverse. A red box marks X i transpose X in the last factor, where the first X carries a subscript i. The second line reads (E[X transpose X]) inverse times (sigma squared E[X transpose X]) times (E[X transpose X]) inverse, and the third reads sigma squared (E[X transpose X]) inverse.", "caption": "Section 4.2.3, p. 155 (PDF p. 173), the display for $n\\mathrm{V}[\\hat{\\boldsymbol\\beta}]$. The marked factor carries a subscript $i$; the first factor of the same line and both inverse factors of the second line have none.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N005.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T67", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T67"}], "headline": "The last factor in the first line of the display is $(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}])^{-1}$; the subscript $i$ on $\\mathbf{X}^T$ is a misprint.", "id": "P001-C04-N005", "location": "Section 4.2.3, p. 155 (PDF p. 173), first line of the display for $n\\mathrm{V}[\\hat{\\boldsymbol\\beta}]$, last factor", "pdf_page": 173, "printed_page": "155", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T67"], "text": {"as_printed": "\\[\n\\begin{aligned}\nn\\mathrm{V}[\\hat{\\boldsymbol\\beta}]&\\approx\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1}\\mathrm{E}[\\varepsilon^2\\mathbf{X}^T\\mathbf{X}]\\bigl(\\mathrm{E}[\\mathbf{X}_{\\hl{i}}^T\\mathbf{X}]\\bigr)^{-1}\\\\\n&=\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1}\\bigl(\\sigma^2\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1}\\\\\n&=\\sigma^2\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1},\n\\end{aligned}\n\\]", "corrected": "Suppose that $\\mathrm{E}[\\varepsilon\\mid X_{[1]},X_{[2]},\\ldots,X_{[K]}]=0$ and $\\mathrm{V}[\\varepsilon\\mid X_{[1]},X_{[2]},\\ldots,X_{[K]}]=\\sigma^2$ for all possible values of $X_{[1]},X_{[2]},\\ldots,X_{[K]}$, and let $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$. Then\n\\[\n\\begin{aligned}\nn\\mathrm{V}[\\hat{\\boldsymbol\\beta}]&\\approx\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1}\\mathrm{E}[\\varepsilon^2\\mathbf{X}^T\\mathbf{X}]\\bigl(\\mathrm{E}[\\hl{\\mathbf{X}^T}\\mathbf{X}]\\bigr)^{-1}\\\\\n&=\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1}\\bigl(\\sigma^2\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1}\\\\\n&=\\sigma^2\\bigl(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]\\bigr)^{-1}.\n\\end{aligned}\n\\]", "evidence": "The evidence is a check of pp.~152 and 155 and a derivation. Set $Q=\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]$, and assume that $Q$ is nonsingular. Under the two assumptions of Section 4.2.3, I have $\\mathrm{E}[\\varepsilon^2\\mid\\mathbf{X}]=\\sigma^2$. By the Law of Iterated Expectations,\n\\[\n\\mathrm{E}[\\varepsilon^2\\mathbf{X}^T\\mathbf{X}]=\\mathrm{E}\\bigl[\\mathbf{X}^T\\mathbf{X}\\,\\mathrm{E}[\\varepsilon^2\\mid\\mathbf{X}]\\bigr]=\\sigma^2Q.\n\\]\nSubstituting this into $Q^{-1}\\mathrm{E}[\\varepsilon^2\\mathbf{X}^T\\mathbf{X}]Q^{-1}$, I obtain $Q^{-1}(\\sigma^2Q)Q^{-1}=\\sigma^2Q^{-1}$, which is the third line of the display. The reduction requires the same matrix $Q$ in both inverse factors.", "problem": "The last factor of the first line carries an observation subscript $i$ on $\\mathbf{X}^T$ and none on $\\mathbf{X}$. The matrix meant is $\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]$, the second-moment matrix of the regressor row $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$. It is printed without a subscript in the first factor of the same line, in both inverse factors of the second line, and in the same expression in Section 4.2.2 (p.~152).", "what_survives": "The simplification $n\\mathrm{V}[\\hat{\\boldsymbol\\beta}]\\approx\\sigma^2(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}])^{-1}$ under the two assumptions and Definition 4.2.3 are unaffected. Nothing depends on the misprint."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of a paragraph of eight lines. A red box marks the words 'If homoskedasticity does hold,' in the sentence 'If homoskedasticity does hold, then in large samples, the classical and robust sampling variance estimates will agree.' The next sentence reads 'With large n and i.i.d. sampling, there is little reason not to use robust standard errors or the bootstrap to characterize the uncertainty of a regression fit.'", "caption": "Last paragraph of Section 4.2.3, p. 156 (PDF p. 174). The marked condition names homoskedasticity alone.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N006.png"}, {"alt": "Crop of six lines of text. Red boxes mark the two assumptions in the sentence 'Suppose we assume that E[epsilon | X[1], X[2], ..., X[K]] = 0 and V[epsilon | X[1], X[2], ..., X[K]] = sigma squared for all possible values of X[1], X[2], ..., X[K].' The next sentence reads 'The first assumption is equivalent to linearity of E[Y | X[1], X[2], ..., X[K]], and the second assumption is known as (conditional) homoskedasticity.'", "caption": "Opening of Section 4.2.3, p. 155 (PDF p. 173). The two marked assumptions are both in force in the section: the first is linearity of the CEF, and the second is homoskedasticity.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The classical and robust sampling variance estimates agree in large samples when both assumptions of Section 4.2.3 hold, linearity of the CEF and homoskedasticity; the closing sentence of the section names only homoskedasticity.", "id": "P001-C04-N006", "location": "Section 4.2.3, last paragraph, p. 156 (PDF p. 174); the two assumptions of the section are on p. 155 (PDF p. 173)", "pdf_page": 174, "printed_page": "156", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~155: ``Suppose we \\emph{assume} that $\\mathrm{E}[\\varepsilon\\mid X_{[1]},X_{[2]},\\ldots,X_{[K]}]=0$ and $\\mathrm{V}[\\varepsilon\\mid X_{[1]},X_{[2]},\\ldots,X_{[K]}]=\\sigma^2$ for all possible values of $X_{[1]},X_{[2]},\\ldots,X_{[K]}$. The first assumption is equivalent to linearity of $\\mathrm{E}[Y\\mid X_{[1]},X_{[2]},\\ldots,X_{[K]}]$, and the second assumption is known as (conditional) \\emph{homoskedasticity}.''\n\nP.~156: ``\\hl{If homoskedasticity \\emph{does} hold}, then in large samples, the classical and robust sampling variance estimates will agree.''", "corrected": "``If \\hl{both the linearity and homoskedasticity assumptions} hold, then in large samples, the classical and robust sampling variance estimates will agree.''", "intended_meaning": "The sentence closes a section that assumes both $\\mathrm{E}[\\varepsilon\\mid\\mathbf{X}]=0$ and $\\mathrm{V}[\\varepsilon\\mid\\mathbf{X}]=\\sigma^2$, and the derivation on p.~155 obtains $\\mathrm{E}[\\varepsilon^2\\mid\\mathbf{X}]=\\sigma^2$ ``by the linearity and homoskedasticity assumptions''. With both assumptions in force the sentence is correct. Since $\\mathrm{E}[\\varepsilon^2\\mid\\mathbf{X}]=\\sigma^2$, I have $\\mathrm{E}[\\varepsilon^2\\mathbf{X}^T\\mathbf{X}]=\\sigma^2\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}]$ and $\\mathrm{E}[\\varepsilon^2]=\\sigma^2$. The limit of $n$ times the robust estimate of the sampling variance is $(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}])^{-1}\\mathrm{E}[\\varepsilon^2\\mathbf{X}^T\\mathbf{X}](\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}])^{-1}$, and the limit of $n$ times the classical one is $\\mathrm{E}[\\varepsilon^2](\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}])^{-1}$. Both equal $\\sigma^2(\\mathrm{E}[\\mathbf{X}^T\\mathbf{X}])^{-1}$.\n\nRead alone, with homoskedasticity as its only premise, the sentence is too broad. The error $\\varepsilon$ is the deviation from the BLP, and\n\\[\n\\mathrm{E}[\\varepsilon^2\\mid\\mathbf{X}]=\\mathrm{V}[\\varepsilon\\mid\\mathbf{X}]+\\bigl(\\mathrm{E}[\\varepsilon\\mid\\mathbf{X}]\\bigr)^2,\n\\]\nwhich can vary with $\\mathbf{X}$ when the CEF is not linear. This occurs in Example 4.3.1 on the same page. Let $X\\sim U(0,1)$ and $Y=10X^2+W$, with $W\\sim N(0,1)$ independent of $X$, and regress $Y$ on $(1,X)$. The BLP is $-5/3+10X$, so that $\\mathrm{V}[\\varepsilon\\mid X]=1$ for all $X$ and $\\mathrm{E}[\\varepsilon\\mid X]=10X^2-10X+5/3$. An exact calculation gives the limit $472/21$ for $n$ times the robust estimate of the sampling variance of the slope, and $56/3$ for $n$ times the classical one. I found no calculation that uses the broader reading: the derivation of Section 4.2.3 uses both assumptions, and Table 4.4.5 reports bootstrap standard errors."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Five lines of text from Example 4.3.1. A red box marks the words 'slope of the CEF' in the sentence 'With bivariate regression, the estimated slope of the CEF of Y with respect to X was beta-hat-1.'", "caption": "Example 4.3.1, p. 157 (PDF p. 175): the sentence on bivariate regression, with “slope of the CEF” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "Bivariate regression estimates the slope of the BLP, which this sentence calls the slope of the CEF.", "id": "P001-C04-N007", "location": "Example 4.3.1, p. 157 (PDF p. 175), paragraph before the display of the derivative", "pdf_page": 175, "printed_page": "157", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "With bivariate regression, the estimated slope of the \\hl{CEF} of $Y$ with respect to $X$ was $\\hat{\\beta}_1$.", "corrected": "With bivariate regression, the estimated slope of the \\hl{BLP} of $Y$ \\hl{given} $X$ was $\\hat{\\beta}_1$.", "intended_meaning": "The sentence refers to the regression of $Y$ on $X$ alone. The coefficient $\\hat{\\beta}_1$ estimates the slope of the best linear predictor (BLP) of $Y$ given $X$, and the preceding page says that OLS regression estimates the BLP. Under this reading the sentence is correct. Read as a statement about the derivative of the conditional expectation function (CEF), it would be false. In the example, $X\\sim U(0,1)$ and $\\mathrm{E}[Y\\mid X]=10X^2$. A direct computation gives $\\operatorname{Cov}[X,10X^2]=5/6$ and $\\mathrm{V}[X]=1/12$. The BLP is $-5/3+10X$, with slope $10$. The derivative of the CEF is $20x$, which equals $10$ only at $x=1/2$. The formula $\\beta_1+2\\beta_2X$ that follows in the text, and its estimate $\\hat{\\beta}_1+2\\hat{\\beta}_2x$, are correct."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Two lines of text that end 'For example,' and a displayed equation in two lines for E[Y | X[1], X[2]]. The second line of the equation reads: plus beta-6 X[1] squared X[1], plus beta-7 X[1] X[2] squared, plus beta-8 X[1] squared X[2] squared. A red box marks the second factor, X[1], of the beta-6 term.", "caption": "Section 4.3.3, p. 165 (PDF p. 183): the display that combines interactions with higher-order polynomial terms, with the second factor of the $\\beta_6$ term marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N009.png"}, {"alt": "Three lines of text: 'Regression (6) includes higher-order interaction terms: in addition to an interaction term for X and Z, the specification accounts for the interaction of X and Z squared, Z and X squared, and X squared and Z squared. Regression (7) includes even more'. A red box marks the words 'Z and X squared'.", "caption": "Section 4.4, p. 174 (PDF p. 192): the description of regression (6), with the interaction of $Z$ and $X^2$ marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In the display on p. 165 that combines interactions with higher-order polynomial terms, the $\\beta_6$ term should be $\\beta_6X_{[1]}^2X_{[2]}$; it is printed as $\\beta_6X_{[1]}^2X_{[1]}$.", "id": "P001-C04-N009", "location": "Section 4.3.3, p. 165 (PDF p. 183), display that combines interactions with higher-order polynomial terms, second line, the $\\beta_6$ term", "pdf_page": 183, "printed_page": "165", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "For example,\n\\[\n\\begin{aligned}\n\\mathrm{E}\\bigl[Y\\mid X_{[1]},X_{[2]}\\bigr]&=\\beta_0+\\beta_1X_{[1]}+\\beta_2X_{[2]}+\\beta_3X_{[1]}X_{[2]}+\\beta_4X_{[1]}^2+\\beta_5X_{[2]}^2\\\\\n&\\quad+\\beta_6X_{[1]}^2X_{[\\hl{\\scriptscriptstyle 1}]}+\\beta_7X_{[1]}X_{[2]}^2+\\beta_8X_{[1]}^2X_{[2]}^2.\n\\end{aligned}\n\\]", "corrected": "For example,\n\\[\n\\begin{aligned}\n\\mathrm{E}\\bigl[Y\\mid X_{[1]},X_{[2]}\\bigr]&=\\beta_0+\\beta_1X_{[1]}+\\beta_2X_{[2]}+\\beta_3X_{[1]}X_{[2]}+\\beta_4X_{[1]}^2+\\beta_5X_{[2]}^2\\\\\n&\\quad+\\beta_6X_{[1]}^2X_{[\\hl{\\scriptscriptstyle 2}]}+\\beta_7X_{[1]}X_{[2]}^2+\\beta_8X_{[1]}^2X_{[2]}^2.\n\\end{aligned}\n\\]", "evidence": "The evidence is a check of the pages and an exact calculation. I write each regressor as $X_{[1]}^aX_{[2]}^b$. The printed display has eight of the nine pairs $(a,b)$ with $a,b\\in\\{0,1,2\\}$, lacks $(2,1)$, and has $(3,0)$. With $(2,1)$ in place of $(3,0)$, the nine pairs are those of the regressors in column (6) of Table 4.4.5. Subtracting the corrected right-hand side from the printed one, I obtain $\\beta_6X_{[1]}^2(X_{[1]}-X_{[2]})$. When $X_{[1]}$ and $X_{[2]}$ are independent and $U(0,1)$, neither list of regressors reproduces the term that only the other has: the BLP of $X_{[1]}^2X_{[2]}$ given the nine printed regressors has mean squared error $1/34560$, and the BLP of $X_{[1]}^3$ given the nine corrected regressors has mean squared error $1/2800$.", "problem": "As printed, the $\\beta_6$ term equals $\\beta_6X_{[1]}^3$, a cubic in $X_{[1]}$ alone. The other eight terms are products of one of $1$, $X_{[1]}$, $X_{[1]}^2$ with one of $1$, $X_{[2]}$, $X_{[2]}^2$, and the only such product that the display lacks is $X_{[1]}^2X_{[2]}$. The nine such products are the regressors of regression (6) in Section 4.4, with $X$ in place of $X_{[1]}$ and $Z$ in place of $X_{[2]}$. The description of that regression on p.~174 lists the interactions of $X$ and $Z$, of $X$ and $Z^2$, of $Z$ and $X^2$, and of $X^2$ and $Z^2$, and column (6) of Table 4.4.5 (p.~173) has an estimate in the row ``Elec $\\times$ Water$^2$'' and none in the row ``Water$^3$''. The second factor $X_{[1]}$ of the $\\beta_6$ term is a misprint for $X_{[2]}$. A cubic term is a legitimate regressor, and the display as printed is a possible specification. It is a different specification from the corrected one.", "what_survives": "The discussion of interactions and of partial derivatives in Section 4.3.3, footnote 11, and the statement that OLS consistently estimates the BLP for the specification chosen are unchanged. Table 4.4.5 and the description of regression (6) already give the corrected list of regressors, and no reported estimate is shown to change. The code for Section 4.4 is not available, and I could not check which regressors it used. A fit that copies the display as printed has $X_{[1]}^3$ in place of $X_{[1]}^2X_{[2]}$, and its predictions and partial derivatives can differ from those of the corrected specification."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "A shaded box with Definition 4.3.6, which displays APD with subscript X[k] equal to the expectation of the partial derivative of E[Y|X] with respect to X[k]. Below it, a sentence beginning 'A natural plug-in estimator for', in which a red box marks APD with subscript X[k]i. Below the sentence, the display of the estimator, written APD-hat with subscript X[k].", "caption": "Definition 4.3.6 and the sentence after it, p. 166 (PDF p. 184): the label $APD_{X_{[k]i}}$ is marked. The definition above it and the estimator below it both write $APD_{X_{[k]}}$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N010.png"}, {"alt": "Two displayed equations from Example 4.3.8, each introduced by a line of text. In the first, for APD with subscript X[1], a red box marks X[2]i inside the second probability, Pr[X[2]i > .8]; the first probability reads Pr[X[2] <= .8]. In the second, a red box marks the left side, PDA with subscript X[k], which is set equal to the partial derivative of minus X[1] with respect to X[1], equal to minus 1.", "caption": "Example 4.3.8, p. 167 (PDF p. 185): the index $i$ in the second probability and the label $PDA_{X_{[k]}}$ are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N010-p167.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T69", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T69"}], "headline": "Three subscripts are misprinted: the estimator after Definition 4.3.6 targets $APD_{X_{[k]}}$, the second probability in Example 4.3.8 is $\\Pr[X_{[2]}>.8]$, and the last display of the example computes $PDA_{X_{[1]}}$.", "id": "P001-C04-N010", "location": "Sentence after Definition 4.3.6, p. 166 (PDF p. 184); Example 4.3.8, p. 167 (PDF p. 185), the displays for $APD_{X_{[1]}}$ and for the PDA", "pdf_page": 184, "printed_page": "166", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T69"], "text": {"as_printed": "[p.~166] A natural plug-in estimator for $APD_{X_{[k]\\hl{i}}}$ is obtained by replacing the expected value with a sample mean and the CEF with the estimated CEF:\n\n[p.~167] Then, by the Law of Iterated Expectations,\n\\[\n\\begin{aligned}\nAPD_{X_{[1]}}&=\\frac{\\partial\\bigl(-X_{[1]}\\bigr)}{\\partial X_{[1]}}\\Pr\\bigl[X_{[2]}\\le.8\\bigr]+\\frac{\\partial\\bigl(9X_{[1]}\\bigr)}{\\partial X_{[1]}}\\Pr\\bigl[X_{[2]\\hl{i}}>.8\\bigr]\\\\\n&=-1\\times.8+9\\times.2=1.\n\\end{aligned}\n\\]\nBut the partial derivative at the average gives a very different result:\n\\[\nPDA_{X_{[\\hl{\\scriptscriptstyle k}]}}=\\frac{\\partial\\bigl(-X_{[1]}\\bigr)}{\\partial X_{[1]}}=-1.\n\\]", "corrected": "[p.~166] A natural plug-in estimator for $APD_{\\hl{\\scriptstyle X_{[k]}}}$ is obtained by replacing the expected value with a sample mean and the CEF with the estimated CEF:\n\n[p.~167] Then, by the Law of Iterated Expectations,\n\\[\n\\begin{aligned}\nAPD_{X_{[1]}}&=\\frac{\\partial\\bigl(-X_{[1]}\\bigr)}{\\partial X_{[1]}}\\Pr\\bigl[X_{[2]}\\le.8\\bigr]+\\frac{\\partial\\bigl(9X_{[1]}\\bigr)}{\\partial X_{[1]}}\\Pr\\bigl[\\hl{X_{[2]}}>.8\\bigr]\\\\\n&=-1\\times.8+9\\times.2=1.\n\\end{aligned}\n\\]\nBut the partial derivative at the average gives a very different result:\n\\[\nPDA_{X_{[\\hl{\\scriptscriptstyle 1}]}}=\\frac{\\partial\\bigl(-X_{[1]}\\bigr)}{\\partial X_{[1]}}=-1.\n\\]", "evidence": "The evidence is a check of the two pages and a recalculation of the example. In Example 4.3.8, $X_{[1]}$ and $X_{[2]}$ are independent and $U(0,1)$, and the CEF is $-X_{[1]}$ when $X_{[2]}\\le.8$ and $9X_{[1]}$ otherwise. The partial derivative with respect to $X_{[1]}$ is $-1$ with probability $4/5$ and $9$ with probability $1/5$, and I obtain\n\\[\nAPD_{X_{[1]}}=(-1)(4/5)+9(1/5)=1.\n\\]\nThe mean of $(X_{[1]},X_{[2]})$ is $(1/2,1/2)$, which lies in the region $X_{[2]}\\le.8$, where the derivative is $-1$. I conclude that $PDA_{X_{[1]}}=-1$. Both printed values are correct. Note that the probability printed with the index $i$ also has the right value, since each observation of $X_{[2]}$ has the $U(0,1)$ distribution.", "problem": "Definition 4.3.6 and the estimator displayed below it write the average partial derivative with respect to the $k$th variable as $APD_{X_{[k]}}$. The sentence between them adds an observation index $i$, which a population quantity does not carry. In Example 4.3.8 the second probability has the same index, although the example involves the random variable $X_{[2]}$ and no sample. Since the last display of the example differentiates with respect to $X_{[1]}$, its label is $PDA_{X_{[1]}}$ and not the generic $PDA_{X_{[k]}}$.", "what_survives": "Definitions 4.3.6 and 4.3.7, the two plug-in estimators and the comparison of $APD_{X_{[1]}}=1$ with $PDA_{X_{[1]}}=-1$ in Example 4.3.8 are unchanged. Only the three labels change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N010", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The heading '4.4 Application: Access to Clean Water and Infant Mortality' and the first four lines of the section. A red box marks the italic words 'Quality of Governance' at the end of the first line, before 'data (Teorell et al. 2016)'.", "caption": "Opening of Section 4.4, p. 170 (PDF p. 188): the name “Quality of Governance” is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N012.png"}], "formal_release": null, "formal_statements": [], "headline": "The dataset is the Quality of Government data; the text calls it the “Quality of Governance” data.", "id": "P001-C04-N012", "location": "Section 4.4, p. 170 (PDF p. 188), first sentence", "pdf_page": 188, "printed_page": "170", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We now consider an application of regression using the \\emph{Quality of \\hl{Governance}} data (Teorell et al. 2016), a dataset that compiles country-level information from a variety of sources.", "corrected": "We now consider an application of regression using the \\emph{Quality of \\hl{Government}} data (Teorell et al. 2016), a dataset that compiles country-level information from a variety of sources.", "evidence": "The evidence is a check of the page against the reference list of the book. The entry on p.~292 reads ``Teorell, Jan et al. 2016. The Quality of Government standard dataset, version Jan 16. University of Gothenburg: The Quality of Government Institute.''", "problem": "The work cited is the Quality of Government standard dataset. The text writes ``Governance'' for ``Government''.", "what_survives": "The citation identifies the dataset, and nothing else changes. The same name is misprinted in Section 6.3; see \\fid{P001-C06-N006}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N012", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Bottom of a regression table with eight columns. In the last column the rows for the estimated APD and PDA of water give the estimates -0.345 and -0.394, and red boxes mark the standard errors (0.075) and (0.087) printed beneath them. The two coefficient rows above show dashes in the last column where columns six and seven have standard errors. The note below the table says that regression (8) reports lasso coefficients and that bootstrap standard error estimates are reported in parentheses under each estimate.", "caption": "Table 4.4.5, p. 173 (PDF p. 191), last rows and note. The rightmost column is regression (8), the lasso fit. The marked entries are the bootstrap standard errors printed for its estimated APD and PDA; the coefficient rows of the same column have dashes.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N015.png"}, {"alt": "Paragraph of text. A red outline marks the sentence: However, as we increase the complexity of the specification, the standard errors grow quickly for both the estimated APD and PDA, suggesting overfitting. The sentence before it says that in columns (3) through (8) the estimate of the APD is largely unchanged by introducing nonlinear specifications.", "caption": "p. 175 (PDF p. 193), the paragraph below Figure 4.4.3. The marked sentence compares the standard errors of the estimated APD and PDA across the columns of Table 4.4.5.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N015-b.png"}, {"alt": "Footnote 14. It says that the lasso has a sparsity property and that most coefficients are set to exactly zero. A red outline marks the words: This sparsity property implies that standard errors cannot generally be computed for the lasso coefficients without very strong assumptions. The footnote ends: though this is a topic of ongoing inquiry. See Leeb and Pötscher (2006).", "caption": "Footnote 14, p. 174 (PDF p. 192). The marked words are the book’s own caution about standard errors for lasso coefficients.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N015-c.png"}], "formal_release": null, "formal_statements": [], "headline": "Column (8) of Table 4.4.5 prints bootstrap standard errors, $0.075$ and $0.087$, for the lasso estimates of the APD and PDA, with no argument that the bootstrap is valid for a lasso fit; they should be described as unvalidated, and neither is shown to be wrong.", "id": "P001-C04-N015", "location": "Table 4.4.5, column (8), rows $\\widehat{APD}_{\\text{Water}}$ and $\\widehat{PDA}_{\\text{Water}}$, p. 173 (PDF p. 191); footnote 14, p. 174 (PDF p. 192); the comparison of standard errors on p. 175 (PDF p. 193)", "pdf_page": 191, "printed_page": "173", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lr}{Table 4.4.5, column (8), p.~173 (excerpt).}\n & (8) \\\\\n$\\widehat{APD}_{\\text{Water}}$ & $-0.345$ \\\\\n & \\hl{$(0.075)$} \\\\\n$\\widehat{PDA}_{\\text{Water}}$ & $-0.394$ \\\\\n & \\hl{$(0.087)$} \\\\\n\\end{audittable}\nThe note to the table:\n\nRegressions (1)--(7) report OLS coefficients and regression (8) reports lasso coefficients. $n=184$ for all regressions. The final two rows report the estimated average partial derivatives and partial derivatives at the average with respect to access to clean water. Bootstrap standard error estimates are reported in parentheses under each estimate.\n\nFootnote 14, p.~174:\n\nNote that the lasso has a ``sparsity'' property that is manifest here---most coefficients are set to exactly zero. This sparsity property implies that standard errors cannot generally be computed for the lasso coefficients without very strong assumptions, though this is a topic of ongoing inquiry. See Leeb and Pötscher (2006).\n\np.~175:\n\nHowever, as we increase the complexity of \\hl{the specification}, the standard errors grow quickly for both the estimated APD and PDA, suggesting overfitting.", "corrected": "Note to Table 4.4.5, last sentence:\n\nBootstrap standard error estimates are reported in parentheses under each estimate. \\hl{For regression (8), the values in parentheses describe the variability of the two estimates across bootstrap resamples; the bootstrap has not been shown to be valid for these functions of a lasso fit.}\n\np.~175:\n\nHowever, as we increase the complexity of the specification \\hl{in the OLS regressions (3) through (7)}, the standard errors grow quickly for both the estimated APD and PDA, suggesting overfitting.", "evidence": "The evidence is a published result and a derivation; it is not a recalculation of the table. It was shown in Camponovo (2015, \\emph{Biometrika} 102(4), 981--987) that the naive pairs bootstrap is not in general a valid method for approximating the distribution of the lasso estimator. The following example shows that the same failure can occur for the APD and PDA. Let $X\\sim U[1,2]$, let $\\varepsilon$ equal $1$ or $-1$ with probability $1/2$ each, independently of $X$, and let $Y=X\\varepsilon$, so that $\\mathbb{E}[Y\\mid X=x]=0$. The lasso fit of $f_b(x)=bx$ minimizes $n^{-1}\\sum_{i=1}^n(Y_i-X_ib)^2+\\lambda_n|b|$ over $b$. Since $\\partial_xf_b(x)=b$ for every $x$, the estimated APD and the estimated PDA both equal the fitted slope $\\widehat b_n$. Set $q=\\mathbb{E}[X^2]=7/3$ and $V=\\operatorname{Var}(X^2\\varepsilon)=31/5$, and define $v(z)=\\operatorname{Var}(s(z+G))$, where $G\\sim N(0,1)$ and $s(u)=\\operatorname{sign}(u)(|u|-1)_+$. With $\\lambda_n=2\\sqrt{V}/\\sqrt{n}$, I have\n\\[\n\\frac{nq^2}{V}\\operatorname{Var}(\\widehat b_n)\\longrightarrow v(0)=4\\Phi(-1)-2\\phi(1)\\approx 0.1507,\n\\]\nwhere $\\Phi$ and $\\phi$ are the standard normal distribution function and density. With pairs $(X_i,Y_i)$ resampled with replacement and the same deterministic penalty used in each resample, the bootstrap variance, scaled in the same way, converges in distribution to $v(Z)$ with $Z\\sim N(0,1)$. Note that $v$ is not constant: $v(3)\\approx 0.9602$, and $v(z)\\to1$ as $z\\to\\infty$. This implies that the ratio of the bootstrap standard error to the true standard error has a nondegenerate limit and does not converge to one. With the smaller penalty $\\lambda_n=2\\sqrt{V}/n$ in the same population, both scaled variances converge to $1$, and the bootstrap standard error is consistent in ratio. Neither case uses cross-validation, and the data behind the table are not available. The derivation does not determine whether $0.075$ and $0.087$ are valid.", "problem": "Column (8) reports a lasso fit whose penalty $\\lambda$ is chosen by leave-one-out cross-validation (p.~174). The table reports no standard errors for the coefficients of this fit, and footnote 14 says why: most coefficients are set to exactly zero, and inference for such estimates is difficult. For two functions of those coefficients, the estimated APD and PDA, the table does print bootstrap standard errors, and the text on p.~175 reads the standard errors across the columns as a sign of overfitting. In Section 3.4.3 (p.~131), the book states that the bootstrap consistently estimates the standard error under the usual regularity conditions for plug-in estimation, and it does not argue that these two functions of the lasso coefficients satisfy them. A function of the lasso coefficients can be smooth in the coefficients and still fail these conditions. The book also does not say how the bootstrap was carried out for column (8), in particular whether $\\lambda$ was chosen again in each resample.", "what_survives": "No point estimate is shown to be wrong, and neither standard error is shown to be wrong: for some contrasts and some penalties the bootstrap is valid. The comparison of the APD point estimates across columns (3)--(8), the OLS columns (1)--(7) and the definitions of the APD and PDA do not depend on the two standard errors. What changes is the support for the remark on p.~175 that the standard errors grow with the complexity of the specification: it can rest on the OLS columns only. Justifying the two standard errors requires an argument covering these functions of the coefficients, penalty selection and resampling; checking their computation also requires the data and the resampling procedure."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N015", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Paragraph on kernel regression and locally weighted regression. A red box marks the words: 2006 provides a clear treatment. They follow a semicolon after the words closer to the point of prediction, and no author's name stands before the year. The next sentence recommends James et al. (2014) and Hastie, Tibshirani, and Friedman (2009).", "caption": "Section 4.5, p. 177 (PDF p. 195). The marked clause begins with a year; the author’s name that should stand before it is missing.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C04-N016.png"}], "formal_release": null, "formal_statements": [], "headline": "The author’s name has dropped out of a citation in Section 4.5, which now reads “; 2006 provides a clear treatment”; the name should be restored, and the page does not show which work was meant.", "id": "P001-C04-N016", "location": "Section 4.5 (Further Readings), p. 177 (PDF p. 195), the sentence on kernel and locally weighted regression", "pdf_page": 195, "printed_page": "177", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "These can be thought of as locally weighted regression fits of observations near a given set of values for the explanatory variables, with larger weights assigned to observations closer to the point of prediction; \\hl{2006} provides a clear treatment.", "corrected": "Restore the author's name before the year from the manuscript. If the name cannot be recovered, delete the clause and end the sentence earlier:\n\nThese can be thought of as locally weighted regression fits of observations near a given set of values for the explanatory variables, with larger weights assigned to observations closer to the point of prediction\\hl{.}", "evidence": "The evidence is a check of the page and of the reference list. The reference list has several entries dated 2006. One of them, Wasserman (2006), \\emph{All of Nonparametric Statistics}, is recommended on p.~141 for the theory of statistical estimation and is a plausible match, but neither the page nor the reference list establishes that it is the work meant here. The manuscript was not consulted.", "problem": "The clause after the semicolon is the remainder of an author--year citation that recommends a treatment of kernel or locally weighted regression. No name stands between the semicolon and the year, and the sentence does not identify the work.", "what_survives": "The description of kernel and locally weighted regression and the other references of Section 4.5 are unaffected. One pointer to further reading is lost."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C04-N016", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the box of Theorem 5.2.8, The ML Estimator Is Consistent for the Minimum KL Divergence Predictor. A red box marks the expression 'KL[ f_{Y|X}( . | x) || g( . , x; t) ]' in the display that defines theta-tilde as the arg min over t in Theta; the x in it is lower case and bold. Below the display, the theorem concludes that theta-hat-ML converges in probability to theta-tilde. Under the box, a paragraph says the proof is omitted and compares the result with OLS, using KL divergence rather than MSE.", "caption": "Theorem 5.2.8, p. 195 (PDF p. 213). The marked divergence is evaluated at a fixed value $\\mathbf x$ of the explanatory variables, with no expectation over $\\mathbf X$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E001.png"}, {"alt": "Line chart. The horizontal axis is Pr[X = 1], with the values 1/4, 1/2 and 3/4. The vertical axis is the parameter value t, with ticks from 1/4 to 3/4. Two horizontal dashed orange lines, labeled 'as printed, at x = 1: t = 3/4' and 'as printed, at x = 0: t = 1/4', stay level across the chart. A solid blue line with three round markers rises from t = 3/8 at Pr[X = 1] = 1/4, through t = 1/2 at 1/2, to t = 5/8 at 3/4. The legend calls the dashed lines the target as printed, the minimizer of the divergence at a fixed x, and the blue line the limit of the ML estimator, the minimizer of the divergence averaged over X. Every blue point lies strictly between the two dashed lines.", "caption": "The counterexample to Theorem 5.2.8 (p. 195) as printed, for three distributions of $X$ and the same conditional distributions of $Y$ given $X$. The dashed lines are the minimizers at a fixed $x$, which are $1/4$ at $x=0$ and $3/4$ at $x=1$. The points are the limit of the ML estimator, which is $3/8$, $1/2$ and $5/8$ for $\\Pr[X=1]=1/4$, $1/2$ and $3/4$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E001-targets.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T7", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T7"}, {"id": "P001-T8", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T8"}, {"id": "P001-T120", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T120"}], "headline": "In Theorem 5.2.8 the limit of the ML estimator is the minimizer of the Kullback–Leibler divergence averaged over the distribution of $\\mathbf X$, not the minimizer at a fixed value $\\mathbf x$.", "id": "P001-C05-E001", "location": "Theorem 5.2.8, p. 195 (PDF p. 213), the display that defines $\\tilde{\\boldsymbol\\theta}$; the explanation continues on p. 196 (PDF p. 214)", "pdf_page": 213, "printed_page": "195", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T120", "P001-T7", "P001-T8"], "text": {"as_printed": "\\textbf{Theorem 5.2.8.} \\emph{The ML Estimator Is Consistent for the Minimum KL Divergence Predictor}\n\nLet $Y$ be a random variable and $\\mathbf X$ be a random vector. Let $(Y_1,\\mathbf X_1)$, $(Y_2,\\mathbf X_2)$, \\ldots, $(Y_n,\\mathbf X_n)$ be i.i.d. observations of $(Y,\\mathbf X)$. Then the ML estimate of $f_{Y|\\mathbf X}(y|\\mathbf x)$, $g(y,\\mathbf x;\\hat{\\boldsymbol\\theta}_{ML})$ consistently estimates the minimum Kullback--Leibler divergence approximation of $f_{Y|\\mathbf X}(y|\\mathbf x)$ among all probability distributions of the form $g(y,\\mathbf x;\\mathbf t)$. That is, assuming that there exists a unique\n\\[\n\\tilde{\\boldsymbol\\theta}=\\operatorname*{arg\\,min}_{\\mathbf t\\in\\boldsymbol\\Theta}\\mathrm{KL}\\big[f_{Y|\\mathbf X}(\\cdot\\,|\\hl{\\mathbf x})\\,\\big\\|\\,g(\\cdot,\\hl{\\mathbf x};\\mathbf t)\\big],\n\\]\nthen\n\\[\n\\hat{\\boldsymbol\\theta}_{ML}\\xrightarrow{p}\\tilde{\\boldsymbol\\theta}.\n\\]", "corrected": "\\textbf{Theorem 5.2.8.} \\emph{The ML Estimator Is Consistent for the Minimum KL Divergence Predictor}\n\nLet $Y$ be a random variable and $\\mathbf X$ be a random vector. Let $(Y_1,\\mathbf X_1)$, $(Y_2,\\mathbf X_2)$, \\ldots, $(Y_n,\\mathbf X_n)$ be i.i.d. observations of $(Y,\\mathbf X)$. Let $L_n(\\mathbf t)=n^{-1}\\sum_{i=1}^n\\log g(Y_i,\\mathbf X_i;\\mathbf t)$ and $L(\\mathbf t)=\\mathrm{E}[\\log g(Y,\\mathbf X;\\mathbf t)]$, and suppose that $L(\\mathbf t)$ is finite for every $\\mathbf t\\in\\boldsymbol\\Theta$ and that the following conditions hold.\n\\begin{itemize}\n\\item $\\sup_{\\mathbf t\\in\\boldsymbol\\Theta}|L_n(\\mathbf t)-L(\\mathbf t)|\\xrightarrow{p}0$.\n\\item $L$ has a unique maximizer $\\tilde{\\boldsymbol\\theta}\\in\\boldsymbol\\Theta$, and $L(\\tilde{\\boldsymbol\\theta})-\\sup_{\\|\\mathbf t-\\tilde{\\boldsymbol\\theta}\\|\\ge\\epsilon}L(\\mathbf t)>0$ for every $\\epsilon>0$ such that the set $\\{\\mathbf t\\in\\boldsymbol\\Theta:\\|\\mathbf t-\\tilde{\\boldsymbol\\theta}\\|\\ge\\epsilon\\}$ is not empty.\n\\item $L_n(\\hat{\\boldsymbol\\theta}_{ML})\\ge\\sup_{\\mathbf t\\in\\boldsymbol\\Theta}L_n(\\mathbf t)-\\eta_n$, where $\\eta_n\\ge0$ and $\\eta_n\\xrightarrow{p}0$. An exact maximizer has $\\eta_n=0$.\n\\end{itemize}\nThen\n\\[\n\\hat{\\boldsymbol\\theta}_{ML}\\xrightarrow{p}\\tilde{\\boldsymbol\\theta}.\n\\]\nIf, in addition, $\\mathrm{E}|\\log f_{Y|\\mathbf X}(Y|\\mathbf X)|<\\infty$ and $\\mathrm{E}|\\log g(Y,\\mathbf X;\\mathbf t)|<\\infty$ for each $\\mathbf t\\in\\boldsymbol\\Theta$, then\n\\[\n\\tilde{\\boldsymbol\\theta}=\\operatorname*{arg\\,min}_{\\mathbf t\\in\\boldsymbol\\Theta}\\hl{\\mathrm{E}_{\\mathbf X}}\\Big[\\mathrm{KL}\\big[f_{Y|\\mathbf X}(\\cdot\\,|\\hl{\\mathbf X})\\,\\big\\|\\,g(\\cdot,\\hl{\\mathbf X};\\mathbf t)\\big]\\Big].\n\\]\nThe parameter $\\tilde{\\boldsymbol\\theta}$ gives the closest fit averaged over the distribution of $\\mathbf X$. It need not give the closest fit at each fixed value $\\mathbf x$.\n\nOn p.~196, ``closest'' (in terms of KL divergence) and the population KL divergence in the paragraph on the idea behind the proof refer to this averaged divergence.", "evidence": "I give an exact counterexample to the theorem as printed, with a derivation of the corrected target. Let $X$ take the values $0$ and $1$ with probability $1/2$ each, and let\n\\[\nY\\mid X=0\\sim\\operatorname{Bernoulli}(1/4),\\qquad Y\\mid X=1\\sim\\operatorname{Bernoulli}(3/4).\n\\]\nConsider the model $g(y,x;t)=t^y(1-t)^{1-y}$ for $y\\in\\{0,1\\}$, with $t\\in\\Theta=[1/10,9/10]$. The parameter space is compact, and each observation's log-likelihood is continuous in $t$ and at most $\\log 10$ in absolute value. For $0<p<1$, the KL divergence with true Bernoulli parameter $p$ and model parameter $t$ is\n\\[\nK_p(t)=p\\log\\frac{p}{t}+(1-p)\\log\\frac{1-p}{1-t},\\qquad K_p'(t)=\\frac{t-p}{t(1-t)}.\n\\]\nSince the derivative is negative for $t<p$ and positive for $t>p$, the function $K_p$ has the unique minimizer $t=p$ on $\\Theta$ when $p\\in\\Theta$. The target printed in Theorem 5.2.8 is then $\\tilde\\theta=1/4$ at $x=0$ and $\\tilde\\theta=3/4$ at $x=1$, and it is unique in both cases.\n\nLet $\\bar Y_n=n^{-1}\\sum_{i=1}^nY_i$. The average log-likelihood is $L_n(t)=\\bar Y_n\\log t+(1-\\bar Y_n)\\log(1-t)$, with derivative $(\\bar Y_n-t)/(t(1-t))$, and the ML estimator is\n\\[\n\\hat\\theta_{ML}=\\min\\{9/10,\\max\\{1/10,\\bar Y_n\\}\\}.\n\\]\nI have $\\mathrm{E}[Y]=1/2$ and $\\mathrm{V}[\\bar Y_n]=1/(4n)$, and restricting $\\bar Y_n$ to $\\Theta$ does not increase its distance from $1/2$. By Chebyshev's inequality, I have\n\\[\n\\Pr\\big[|\\hat\\theta_{ML}-1/2|\\ge\\epsilon\\big]\\le\\frac{1}{4n\\epsilon^2}\n\\]\nfor every $\\epsilon>0$, so that $\\hat\\theta_{ML}\\xrightarrow{p}1/2$. Taking $\\epsilon=1/8$, I obtain $\\Pr[|\\hat\\theta_{ML}-1/4|>1/8]\\ge1-16/n$, and the same bound holds with $3/4$ in place of $1/4$. The ML estimator converges to neither printed target.\n\nThe averaged divergence is $\\bar K(t)=K_{1/4}(t)/2+K_{3/4}(t)/2$, with $\\bar K'(t)=(t-1/2)/(t(1-t))$. Its unique minimizer is $1/2$, the limit of the ML estimator, and $\\bar K(1/4)-\\bar K(1/2)=\\bar K(3/4)-\\bar K(1/2)=\\log(4/3)/2>0$. If $\\Pr[X=1]=q$ and the conditional distributions are unchanged, the minimizer of the averaged divergence is $1/4+q/2$. It equals $3/8$ for $q=1/4$ and $5/8$ for $q=3/4$, and the minimizers at a fixed $x$ remain $1/4$ and $3/4$.\n\nFor the corrected target, suppose that $\\mathrm{E}|\\log f_{Y|\\mathbf X}(Y|\\mathbf X)|<\\infty$ and $\\mathrm{E}|\\log g(Y,\\mathbf X;\\mathbf t)|<\\infty$ for each $\\mathbf t$. By the Law of Iterated Expectations,\n\\[\n\\mathrm{E}_{\\mathbf X}\\Big[\\mathrm{KL}\\big[f_{Y|\\mathbf X}(\\cdot\\,|\\mathbf X)\\,\\big\\|\\,g(\\cdot,\\mathbf X;\\mathbf t)\\big]\\Big]=\\mathrm{E}\\big[\\log f_{Y|\\mathbf X}(Y|\\mathbf X)\\big]-L(\\mathbf t).\n\\]\nThe first term on the right does not depend on $\\mathbf t$. Minimizing the averaged divergence is then the same as maximizing $L(\\mathbf t)$, and it is not in general the same as minimizing the divergence at each $\\mathbf x$. Consistency under the three conditions of the corrected statement follows from a comparison of $L_n$ and $L$. Set $D_n=\\sup_{\\mathbf t\\in\\boldsymbol\\Theta}|L_n(\\mathbf t)-L(\\mathbf t)|$ and $\\delta_\\epsilon=L(\\tilde{\\boldsymbol\\theta})-\\sup_{\\|\\mathbf t-\\tilde{\\boldsymbol\\theta}\\|\\ge\\epsilon}L(\\mathbf t)$. On the event $\\|\\hat{\\boldsymbol\\theta}_{ML}-\\tilde{\\boldsymbol\\theta}\\|\\ge\\epsilon$, I have\n\\[\nL(\\tilde{\\boldsymbol\\theta})-D_n-\\eta_n\\le L_n(\\tilde{\\boldsymbol\\theta})-\\eta_n\\le L_n(\\hat{\\boldsymbol\\theta}_{ML})\\le L(\\hat{\\boldsymbol\\theta}_{ML})+D_n\\le L(\\tilde{\\boldsymbol\\theta})-\\delta_\\epsilon+D_n,\n\\]\nwhich implies that $\\delta_\\epsilon\\le2D_n+\\eta_n$. Since $\\delta_\\epsilon>0$, and $D_n$ and $\\eta_n$ tend to zero in probability, the probability of this event tends to zero. The counterexample satisfies these conditions, with $D_n\\le|\\bar Y_n-1/2|\\log 9$.", "problem": "By Definition 5.2.7 the Kullback--Leibler (KL) divergence integrates over the outcome alone. The displayed quantity therefore compares the true conditional distribution of $Y$ with the model at one fixed value $\\mathbf x$ of the explanatory variables, and the display contains no expectation over $\\mathbf X$. The ML estimator fits a single parameter to the whole sample, and under the usual consistency conditions its limit is the maximizer of the expected log-likelihood $L(\\mathbf t)=\\mathrm{E}[\\log g(Y,\\mathbf X;\\mathbf t)]$ of p.~194. This maximizer minimizes the divergence averaged over the distribution of $\\mathbf X$, and it can change when that distribution changes. A minimizer at a fixed $\\mathbf x$ need not equal it, even when that minimizer is unique at every $\\mathbf x$, the parameter space is compact, the likelihood is continuous, and the sample and population maxima are unique. Read as printed, the theorem names the wrong parameter, and it promises a best approximation at each value of $\\mathbf x$ separately, which ML estimation does not provide. The expected log-likelihood on p.~194 and the comparison with OLS on pp.~195--196 point to the intended averaged target, but the printed formula does not state it. The two targets do not differ in every misspecified model with explanatory variables: there exist false models in which the minimizers at each $\\mathbf x$ and the minimizer of the averaged divergence coincide.", "what_survives": "The definition of the likelihood, the ML estimates themselves, the equivalence of OLS and ML in the classical linear model, and Theorem 5.2.6 with its regularity conditions are unchanged, and no number in the book is shown to depend on the printed target. Examples 5.2.9 and 5.2.11 and Figures 5.2.2--5.2.5 have no explanatory variables, and the correction does not alter their targets; I did not recompute those figures. The claim that ML estimation consistently estimates a best-fitting distribution in the parametric family survives, with ``best'' defined by the averaged divergence, and what is given up is the closest fit at each value of $\\mathbf x$ separately. The reference in Section 5.2.6 (p.~202) to this predictor should also use the averaged target (see \\fid{P001-C05-N006} for its wording), and Section 7.2.3 (pp.~261--262) assumes that the parametric model is true and is not affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the top of p. 199 with the running head 'Parametric Models' and the page number 199. The first paragraph reads 'where theta = (p_1, theta_1, p_2, theta_2, ..., p_{M-1}, theta_{M-1}, theta_M). Then, as before, the product over i from 1 to n of g(Y_i, X_i; theta) can be maximized to find the ML estimate of theta. (This can sometimes be computationally tricky.)' A red box marks the words 'can be maximized to find the ML estimate of theta'. The second paragraph begins 'If any subset of the models' and ends 'our estimates should be interpreted explicitly as an approximation.'", "caption": "Section 5.2.5, p. 199 (PDF p. 217). The marked clause says that the likelihood of the mixture can be maximized; for the mixture of Example 5.2.11 it has no maximum on a sample of positive observations.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E002.png"}, {"alt": "Crop of the end of Example 5.2.11. After the line 'Consider a mixture model that incorporates all three models:', the display gives g(y; theta) as p_N times the normal density, plus p_LN times the log-normal density, plus (1 - p_N - p_LN) times the exponential density. Small red boxes mark sigma_N in the denominator of the normal term and sigma_LN in the denominator of the log-normal term. Below the display, the text reads 'where theta = (p_N, p_LN, mu_N, sigma_N, mu_LN, sigma_LN, lambda_E). Figure 5.2.5 visualizes the ML estimate of this mixture model, which provides a markedly better approximation.' Red boxes mark the words 'Figure 5.2.5 visualizes the ML estimate of this mixture model'. The next paragraph begins 'Technically, mixture models are parametric'.", "caption": "Example 5.2.11, p. 200 (PDF p. 218). The marked scale parameters $\\sigma_N$ and $\\sigma_{LN}$ are free. On positive samples, the likelihood grows without bound if either component is centered at an observation and its scale tends to zero while another component stays fixed with positive weight. The marked sentence presents the curve of Figure 5.2.5 as the ML estimate of this mixture.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E002-2.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T9", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T9"}, {"id": "P001-T10", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T10"}, {"id": "P001-T103", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T103"}], "headline": "On every sample of positive observations, the likelihood of the normal, log-normal and exponential mixture of Example 5.2.11 is unbounded and the ML estimate of the unrestricted mixture does not exist; Figure 5.2.5 cannot show a global ML estimate of that mixture.", "historical_evidence_type": "lean_proof", "id": "P001-C05-E002", "location": "Section 5.2.5, p. 199 (PDF p. 217), “can be maximized to find the ML estimate”; Example 5.2.11, p. 200 (PDF p. 218); Figure 5.2.5, p. 201 (PDF p. 219)", "pdf_page": 217, "printed_page": "199", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T10", "P001-T103", "P001-T9"], "text": {"as_printed": "Section 5.2.5, p.~199:\n\nwhere $\\boldsymbol\\theta=(p_1,\\boldsymbol\\theta_1,p_2,\\boldsymbol\\theta_2,\\ldots,p_{M-1},\\boldsymbol\\theta_{M-1},\\boldsymbol\\theta_M)$. Then, as before, $\\prod_{i=1}^ng(Y_i,\\mathbf X_i;\\boldsymbol\\theta)$ \\hl{can be maximized} to find the ML estimate of $\\boldsymbol\\theta$. (This can sometimes be computationally tricky.)\n\nIf any subset of the models in $(g_1(y,\\mathbf x;\\boldsymbol\\theta_1),\\ldots,g_M(y,\\mathbf x;\\boldsymbol\\theta_M))$ jointly characterize the conditional distribution of $Y$, then this estimator is consistent and asymptotically efficient. When the models are not valid (even jointly), our estimates should be interpreted explicitly as an approximation.\n\nExample 5.2.11, p.~200:\n\nConsider a mixture model that incorporates all three models:\n\\[\n\\begin{aligned}\ng(y;\\boldsymbol\\theta)&=p_Ng_N\\big(y;(\\mu_N,\\sigma_N)\\big)+p_{LN}g_{LN}\\big(y;(\\mu_{LN},\\sigma_{LN})\\big)\\\\\n&\\quad+(1-p_N-p_{LN})g_E(y;\\lambda_E)\\\\\n&=p_N\\frac{1}{\\sigma_N\\sqrt{2\\pi}}e^{-\\frac{(y-\\mu_N)^2}{2\\sigma_N^2}}+p_{LN}\\frac{1}{y\\sigma_{LN}\\sqrt{2\\pi}}e^{-\\frac{(\\log y-\\mu_{LN})^2}{2\\sigma_{LN}^2}}\\\\\n&\\quad+(1-p_N-p_{LN})\\lambda_Ee^{-\\lambda_Ey},\n\\end{aligned}\n\\]\nwhere $\\boldsymbol\\theta=(p_N,p_{LN},\\mu_N,\\sigma_N,\\mu_{LN},\\sigma_{LN},\\lambda_E)$. Figure 5.2.5 visualizes the \\hl{ML estimate} of this mixture model, which provides a markedly better approximation. $\\triangle$\n\nFigure 5.2.5, p.~201, has the plot title ``Mixture \\hl{ML} Fit'' and the caption:\n\nFIGURE 5.2.5. \\emph{Mixture Model \\hl{Maximum Likelihood} Approximation}", "corrected": "Section 5.2.5, p.~199:\n\nThen, as before, $\\prod_{i=1}^ng(Y_i,\\mathbf X_i;\\boldsymbol\\theta)$ can be maximized to find the ML estimate of $\\boldsymbol\\theta$\\hl{, provided that a maximum exists}. (This can sometimes be computationally tricky. \\hl{A maximum need not exist: when a component has a free scale parameter, as the normal and log-normal components of Example 5.2.11 do, the likelihood can be unbounded. The parameters must then be restricted to ensure an attained likelihood maximum, for example to a nonempty compact subset of the admissible parameter space. Alternatively, a penalty must ensure an attained maximum of the penalized log-likelihood before a penalized ML estimate is defined.})\n\nThe sentence on consistency and efficiency that follows applies only when a maximum exists and the regularity conditions hold; its correction is given in \\fid{P001-C05-N005}.\n\nExample 5.2.11, p.~200:\n\n\\hl{Figure 5.2.5 shows a fitted mixture of this form}, which provides a markedly better approximation.\n\nThe example should state the objective and the restrictions used for the curve: for example, a population (minimum KL divergence) approximation as in Example 5.2.9, or a fit to a sample with the scale parameters restricted. Until then the title and caption of Figure 5.2.5 should describe a fitted mixture approximation, not a maximum likelihood estimate.\n\nThe mathematical statement is as follows. Consider the mixture of Example 5.2.11 with $p_N,p_{LN}\\ge0$, $p_N+p_{LN}\\le1$, $\\mu_N,\\mu_{LN}\\in\\mathbb{R}$ and $\\sigma_N,\\sigma_{LN},\\lambda_E>0$, and let $y_1,\\ldots,y_n$ be positive observations with $n\\ge1$. Then $\\sup_{\\boldsymbol\\theta}\\prod_{i=1}^ng(y_i;\\boldsymbol\\theta)=\\infty$, and no parameter value attains the supremum. If the parameters are restricted to a nonempty compact set on which $\\sigma_N$, $\\sigma_{LN}$ and $\\lambda_E$ are bounded below by a positive constant, then the likelihood of the sample attains its maximum on that set. This restriction does not by itself give a unique maximizer or asymptotic efficiency, and when the model is not true it can change the fitted density and the KL divergence target. A fixed $\\ell^1$ penalty on the seven parameters, as in Section 5.2.6, does not restore a maximum.", "evidence": "The evidence is a Lean proof of the counterexample on the sample $(1,2,3)$, with a derivation that extends it to every sample of positive observations. I use the parameter space described above. For $\\sigma>0$, I set, in the printed order of the parameters,\n\\[\n\\boldsymbol\\theta_\\sigma=(1/2,\\,1/4,\\,1,\\,\\sigma,\\,0,\\,1,\\,1).\n\\]\nThe three weights are $1/2$, $1/4$ and $1/4$, and for $y>0$ the density is\n\\[\ng(y;\\boldsymbol\\theta_\\sigma)=\\frac{e^{-(y-1)^2/(2\\sigma^2)}}{2\\sigma\\sqrt{2\\pi}}+\\frac{e^{-(\\log y)^2/2}}{4y\\sqrt{2\\pi}}+\\frac{e^{-y}}{4}.\n\\]\nEach term is nonnegative, so that $g(1;\\boldsymbol\\theta_\\sigma)\\ge1/(2\\sigma\\sqrt{2\\pi})$, $g(2;\\boldsymbol\\theta_\\sigma)\\ge e^{-2}/4$ and $g(3;\\boldsymbol\\theta_\\sigma)\\ge e^{-3}/4$. Multiplying these bounds, I obtain\n\\[\ng(1;\\boldsymbol\\theta_\\sigma)\\,g(2;\\boldsymbol\\theta_\\sigma)\\,g(3;\\boldsymbol\\theta_\\sigma)\\ge\\frac{e^{-5}}{32\\sqrt{2\\pi}\\,\\sigma}.\n\\]\nThe right side exceeds any given real number when $\\sigma$ is small enough. Since the likelihood is finite at every admissible parameter value, no admissible parameter value maximizes it. The Lean proof establishes these two statements for the sample $(1,2,3)$. Using $e<3$ and $\\sqrt{2\\pi}<3$, I find that the likelihood at $\\sigma=1/k$ exceeds $k/23328$ for every positive integer $k$. The table lists five values of this bound.\n\\begin{audittable}{rr}{Likelihood of the sample $(1,2,3)$ at $\\boldsymbol\\theta_\\sigma=(1/2,1/4,1,\\sigma,0,1,1)$: an exact lower bound as $\\sigma_N=\\sigma$ decreases.}\n$\\sigma_N$ & The likelihood exceeds \\\\\n\\hline\n$1/23328$ & $1$ \\\\\n$1/46656$ & $2$ \\\\\n$1/256608$ & $11$ \\\\\n$1/2356128$ & $101$ \\\\\n$1/23328023328$ & $1000001$ \\\\\n\\end{audittable}\nFor positive observations $y_1,\\ldots,y_n$, the same weights with $\\mu_N=y_1$, $\\sigma_N=\\sigma$, $\\mu_{LN}=0$, $\\sigma_{LN}=1$ and $\\lambda_E=1$ give\n\\[\n\\prod_{i=1}^ng(y_i;\\boldsymbol\\theta)\\ge\\frac{e^{-\\sum_{i=2}^ny_i}}{2\\cdot4^{n-1}\\sqrt{2\\pi}\\,\\sigma},\n\\]\nwhich tends to infinity as $\\sigma\\to0$. A sample that contains negative values and at least one positive value is treated in the same way, with the log-normal component concentrated at a positive observation and the normal component kept as a positive background. The likelihood is not unbounded on every real sample: on the sample $(-1,-2)$ only the normal component is positive, and the likelihood is at most $2/(\\pi e)$, attained at $p_N=1$, $\\mu_N=-3/2$ and $\\sigma_N=1/2$. The failure does not come from misspecification. If the data are drawn independently from this mixture with $\\boldsymbol\\theta=(1/2,1/4,0,1,0,1,1)$, then $\\Pr[Y\\le0]=1/4$, and a sample of size $n$ contains a positive observation, and has an unbounded likelihood, with probability $1-4^{-n}$.\n\nFor the restricted statement in the correction, the likelihood of a sample of positive observations is a continuous function of $\\boldsymbol\\theta$ where $\\sigma_N$, $\\sigma_{LN}$ and $\\lambda_E$ are positive, and a continuous function attains its maximum on a nonempty compact set. Along $\\boldsymbol\\theta_\\sigma$ the $\\ell^1$ norm of the seven parameters is $15/4+\\sigma$, which stays bounded while the log-likelihood diverges.", "problem": "Definition 5.2.2 defines the ML estimate as the global maximizer of the likelihood of the sample. For the mixture of Example 5.2.11 the example states no restriction on the parameters beyond those of the three distributions: nonnegative $p_N$ and $p_{LN}$ with $p_N+p_{LN}\\le1$ and remaining weight $1-p_N-p_{LN}$, real $\\mu_N$ and $\\mu_{LN}$, and positive $\\sigma_N$, $\\sigma_{LN}$ and $\\lambda_E$. On this parameter space the likelihood has no global maximizer when every observation is positive. Keep the normal and exponential weights fixed and positive, let $\\mu_N$ equal one observation, and let $\\sigma_N$ tend to zero. Then the density at that observation grows like $1/\\sigma_N$, while the exponential component keeps the density at every other observation bounded away from zero, and the likelihood exceeds every bound. Since the likelihood is finite at every admissible parameter value, no parameter value attains the supremum. For this mixture the product on p.~199 cannot ``be maximized'' as printed, the consistency and efficiency claimed for ``this estimator'' have no object until the parameter space is restricted, and a curve with finite parameter values, such as the curve in Figure 5.2.5, cannot be the global ML estimate of the unrestricted mixture from a sample of positive observations. The ML estimate fails to exist here, which is different from being hard to compute.\n\nTheorem 5.2.6 is not contradicted: it requires regularity conditions, a compact parameter space and a unique maximum among them, and the glossary's convention for $\\arg\\max$ assumes that a unique maximum exists (p.~285). The mixture of the example does not satisfy these conditions as printed. No data or code for computing Figure 5.2.5 accompany the source available to me, and the book does not state the objective or the restrictions used to compute it. Example 5.2.11 refers back to Example 5.2.9, which describes an asymptotic ML estimate, and the curve may be a population (minimum KL divergence) approximation, or a restricted or local fit. The evidence does not show that the curve is wrong.", "what_survives": "Theorem 5.2.6 with its regularity conditions and the use of mixtures to approximate a distribution are unaffected. The curve of Figure 5.2.5 is not shown to be wrong, and its closer fit to the histogram is not in question; what changes is its description as the ML estimate of the unrestricted mixture, and the consistency and efficiency attached to that estimator. The later references to mixtures for binary propensity models (pp.~226 and 264) are not affected, since every factor of a binary likelihood is at most one, and the propensity model in the application of Chapter 6 is a logit model (p.~233). No reported number is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph that begins 'The idea behind the proof of Theorem 5.2.8 is again related to the plug-in principle.' Red boxes mark the phrase 'KL divergence of the hypothesized distribution from the observed data', which runs from the end of the fifth line to the sixth, in the sentence 'It can be shown that the ML estimate is equal to the sample analog: the parameter values that minimize the KL divergence of the hypothesized distribution from the observed data.' The paragraph ends 'the ML estimate converges to the value that minimizes the KL divergence of the hypothesized distribution from the true population distribution.'", "caption": "Section 5.2.3, p. 196 (PDF p. 214), the paragraph on the idea behind the proof of Theorem 5.2.8. The marked phrase describes the sample objective of ML estimation as a KL divergence from the observed data.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E003.png"}], "formal_release": null, "formal_statements": [], "headline": "The sample analog on p. 196 is the minimizer of the empirical cross-entropy, that is, the maximizer of the average log-likelihood; it minimizes a KL divergence from the observed data only when the outcome is discrete.", "id": "P001-C05-E003", "location": "Section 5.2.3, p. 196 (PDF p. 214), second paragraph, the sentence that begins “It can be shown”", "pdf_page": 214, "printed_page": "196", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "It can be shown that the ML estimate is equal to the sample analog: the parameter values that minimize the \\hl{KL divergence of the hypothesized distribution from the observed data}. So, since the distribution of the observed data converges to the population distribution as $n$ grows large, the ML estimate converges to the value that minimizes the KL divergence of the hypothesized distribution from the true population distribution.", "corrected": "It can be shown that the ML estimate is equal to the sample analog: the parameter values that minimize the \\hl{empirical cross-entropy}\n\\[\n\\hl{H_n(\\mathbf t)=-\\frac{1}{n}\\sum_{i=1}^n\\log g(Y_i,\\mathbf X_i;\\mathbf t)},\n\\]\n\\hl{the sample analog of the part of the population KL divergence that depends on $\\mathbf t$, when the population entropy and expected log-density are finite}. So, since \\hl{$H_n(\\mathbf t)$ converges to $-\\mathrm{E}[\\log g(Y,\\mathbf X;\\mathbf t)]$} as $n$ grows large, the ML estimate converges, \\hl{under the requisite regularity conditions}, to the value that minimizes the KL divergence of the hypothesized distribution from the true population distribution.", "intended_meaning": "The sample analog meant here is the maximizer of the average log-likelihood $L_n(\\mathbf t)$ of p.~193, which is the minimizer of the empirical cross-entropy $-L_n(\\mathbf t)$, and with this meaning the sentence is correct. For a discrete outcome, such as the coin flip with $y=(1,0,1)$ on the same page, minimizing $-L_n$ is the same as minimizing the KL divergence from the empirical distribution, since the two criteria differ by a term that does not depend on $\\mathbf t$. For a continuous model the description is not literal: the empirical distribution has no density, Definition 5.2.7 does not apply to it, and under the general definition the divergence of a model with a density from the empirical distribution is infinite for every $\\mathbf t$. With explanatory variables, the population criterion is the divergence averaged over the distribution of $\\mathbf X$ (see \\fid{P001-C05-E001}), which differs from $-\\mathrm{E}[\\log g(Y,\\mathbf X;\\mathbf t)]$ by a constant when both are finite. The estimator and the calculations of the chapter are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Paragraph that follows the display Y = X beta + epsilon. Red boxes mark the words independent of X and the citation (Theorem 2.2.19) in the sentence: The fact that the deviations from the linear component are mean zero and independent of X should be somewhat familiar; these are the properties of the deviations from the CEF (Theorem 2.2.19).", "caption": "Section 5.1.1, p. 181 (PDF p. 199), the paragraph after the display $Y=\\mathbf{X}\\boldsymbol{\\beta}+\\varepsilon$. The marked words attribute independence from $\\mathbf{X}$ to deviations from the CEF and cite Theorem 2.2.19 for it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T73", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T73"}, {"id": "P001-T74", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T74"}], "headline": "Independence from $\\mathbf{X}$ is a property of the errors of the classical linear model and not of deviations from the CEF in general; Theorem 2.2.19, which the sentence cites, states only conditional mean zero and zero covariance.", "id": "P001-C05-E004", "location": "Section 5.1.1, p. 181 (PDF p. 199), the sentence before Theorem 5.1.5 that begins “The fact that the deviations”", "pdf_page": 199, "printed_page": "181", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T73", "P001-T74"], "text": {"as_printed": "The fact that the deviations from the linear component are mean zero and \\hl{independent of} $\\mathbf{X}$ should be somewhat familiar; these are the properties of the deviations from the CEF \\hl{(Theorem 2.2.19)}.", "corrected": "\\hl{In the classical linear model,} the deviations from the linear component are mean zero and independent of $\\mathbf{X}$. \\hl{More generally, deviations from the CEF have conditional mean zero and are uncorrelated with functions of the covariates whenever the relevant expectations exist} (Theorem 2.2.19)\\hl{; independence is not a property of deviations from the CEF in general.}\n\nThe smallest change with the same effect replaces ``independent of'' by ``uncorrelated with'' in the printed sentence.", "evidence": "The evidence is an exact counterexample. Let $(X,Y)$ take the four pairs of values in the table below with probability $1/4$ each, and let $\\varepsilon=Y-\\mathrm{E}[Y|X]$.\n\\begin{audittable}{rrrrr}{A joint distribution with a linear CEF whose deviation is not independent of $X$.}\n$X$ & $Y$ & $\\mathrm{E}[Y|X]$ & $\\varepsilon$ & Probability \\\\\n$0$ & $1$ & $2$ & $-1$ & $1/4$ \\\\\n$0$ & $3$ & $2$ & $1$ & $1/4$ \\\\\n$1$ & $0$ & $3$ & $-3$ & $1/4$ \\\\\n$1$ & $6$ & $3$ & $3$ & $1/4$ \\\\\n\\end{audittable}\nThen $\\mathrm{E}[Y|X=x]=2+x$, which is linear in $x$. The deviation $\\varepsilon$ has all five properties of Theorem 2.2.19. Indeed, $\\mathrm{E}[\\varepsilon|X=0]=\\mathrm{E}[\\varepsilon|X=1]=0$, and $\\mathrm{E}[g(X)\\varepsilon]=0$ for every function $g$, which implies that $\\operatorname{Cov}[g(X),\\varepsilon]=0$. Further, $\\mathrm{V}[\\varepsilon|X=0]=\\mathrm{V}[Y|X=0]=1$ and $\\mathrm{V}[\\varepsilon|X=1]=\\mathrm{V}[Y|X=1]=9$, and $\\mathrm{V}[\\varepsilon]=5=\\mathrm{E}[\\mathrm{V}[Y|X]]$. However,\n\\[\n\\Pr(X=0,\\ |\\varepsilon|=1)=\\frac12\\neq\\frac14=\\Pr(X=0)\\Pr(|\\varepsilon|=1),\n\\]\nwhich shows that $\\varepsilon$ and $X$ are dependent. In the classical linear model with $\\sigma>0$, the conditional density of $\\varepsilon=Y-\\mathbf{X}\\boldsymbol{\\beta}$ given $\\mathbf{X}=\\mathbf{x}$ is $\\phi(t;(0,\\sigma^2))$ for every $\\mathbf{x}$. Since this density does not depend on $\\mathbf{x}$, the error is independent of $\\mathbf{X}$ in that model.", "problem": "Under the classical linear model, the errors $\\varepsilon=Y-\\mathbf{X}\\boldsymbol{\\beta}$ are independent of $\\mathbf{X}$, and the first clause is correct. The clause after the semicolon says that mean zero and independence are the properties of deviations from the CEF, and it cites Theorem 2.2.19. For $\\varepsilon=Y-\\mathrm{E}[Y|X]$, that theorem states that $\\mathrm{E}[\\varepsilon|X]=0$, that $\\mathrm{E}[\\varepsilon]=0$, that $\\operatorname{Cov}[g(X),\\varepsilon]=0$ for any function $g$ of $X$, that $\\mathrm{V}[\\varepsilon|X]=\\mathrm{V}[Y|X]$, and that $\\mathrm{V}[\\varepsilon]=\\mathrm{E}[\\mathrm{V}[Y|X]]$. It does not state independence, and independence does not hold in general, since by the fourth property the conditional variance of the deviation can depend on $X$. In the classical linear model, independence comes from the conditional normal density with constant variance that Definition 5.1.4 assumes.", "what_survives": "Theorem 2.2.19, Definition 5.1.4 and Theorem 5.1.5 are unaffected. The proof of Theorem 5.1.5 uses the conditional normal density directly and does not use this sentence. The statement that the errors of the classical linear model are independent of $\\mathbf{X}$ stands. Only the comparison with deviations from the CEF in general is weakened, from independence to conditional mean zero and zero covariance."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Footnote 10: We do not need to estimate sigma here, but simple algebraic manipulations show that its ML estimator will equal 1 over n times the sum over i from 1 to n of (Y i minus X i beta hat) squared, or the plug-in sample variance of the residuals. Red boxes mark the symbol sigma in the first line and the closing parenthesis after the final period. The first two lines of footnote 11 follow.", "caption": "Footnote 10, p. 191 (PDF p. 209). The marked $\\sigma$ should be $\\sigma^2$, and the marked closing parenthesis has no partner.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T78", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T78"}], "headline": "The expression in footnote 10 is the ML estimator of $\\sigma^2$, not of $\\sigma$; the footnote should begin “We do not need to estimate $\\sigma^2$ here”, and its final parenthesis is unmatched.", "id": "P001-C05-E005", "location": "Footnote 10, Example 5.2.4, p. 191 (PDF p. 209)", "pdf_page": 209, "printed_page": "191", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T78"], "text": {"as_printed": "We do not need to estimate \\hl{$\\sigma$} here, but simple algebraic manipulations show that its ML estimator will equal $\\frac{1}{n}\\sum_{i=1}^{n}(Y_i-\\mathbf{X}_i\\hat{\\boldsymbol{\\beta}})^2$, or the plug-in sample variance of the residuals.\\hl{)}", "corrected": "We do not need to estimate \\hl{$\\sigma^2$} here, but \\hl{when the sum of squared residuals is positive,} simple algebraic manipulations show that its ML estimator will equal $\\frac{1}{n}\\sum_{i=1}^{n}(Y_i-\\mathbf{X}_i\\hat{\\boldsymbol{\\beta}})^2$, or the plug-in sample variance of the residuals.\n\nThe unmatched parenthesis is deleted. If the sum of squared residuals is zero, no maximum exists with $s>0$. If the footnote is to keep $\\sigma$, the estimator is the square root of the displayed expression, and ``sample variance'' becomes ``sample standard deviation''.", "evidence": "The evidence is a derivation. Set $S=\\sum_{i=1}^{n}(Y_i-\\mathbf{X}_i\\hat{\\boldsymbol{\\beta}})^2$ and suppose that $S>0$. As a function of the scale $s>0$, the log-likelihood at $\\mathbf{b}=\\hat{\\boldsymbol{\\beta}}$ and its derivative are\n\\[\n\\ell(s)=-n\\log s-\\frac{S}{2s^2}-\\frac{n}{2}\\log(2\\pi),\\qquad\n\\ell'(s)=-\\frac{n}{s}+\\frac{S}{s^3}=\\frac{S-ns^2}{s^3}.\n\\]\nThe derivative is positive for $s<\\sqrt{S/n}$ and negative for $s>\\sqrt{S/n}$. Then the unique maximum is at $\\hat\\sigma=\\sqrt{S/n}$, and the ML estimator of the variance is $\\hat\\sigma^2=S/n$, which is the expression in the footnote. For the two observations $0$ and $4$ and a model with an intercept only, the fitted mean is $2$ and $S=8$. I obtain $\\hat\\sigma=2$, while the expression in the footnote equals $4$; the log-likelihood is $-4.224$ at $s=2$ and $-4.860$ at $s=4$.", "problem": "In Example 5.2.4, $\\sigma$ is the standard deviation of the errors: the model has $\\varepsilon\\sim N(0,\\sigma^2)$, and the likelihood is written with the scale $s$ and the variance $s^2$. The expression in the footnote is the ML estimator of the variance $\\sigma^2$, as the footnote's own gloss, ``the plug-in sample variance of the residuals'', says. When the sum of squared residuals is positive, the ML estimator of $\\sigma$ is its square root. The footnote also ends with a closing parenthesis that has no opening one.", "what_survives": "No later calculation uses the footnote. The result of Example 5.2.4 that the ML estimator of $\\boldsymbol{\\beta}$ minimizes the sum of squared residuals whenever a positive-scale ML solution exists is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Footnote 5: Formally, we would say that the likelihood function is itself a random function, script L from Omega to script L, where script L is the space of functions L from R to the interval [0, 1]. A red box marks L : R to [0, 1].", "caption": "Footnote 5, p. 186 (PDF p. 204). The marked type gives the realized likelihood a scalar argument and values in $[0,1]$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T77", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T77"}], "headline": "In footnote 5 the realized likelihood has the type $L:\\mathbb{R}\\to[0,1]$; in general its domain is the parameter space $\\boldsymbol{\\Theta}$, and its values are nonnegative but need not be at most one.", "id": "P001-C05-E006", "location": "Footnote 5, p. 186 (PDF p. 204)", "pdf_page": 204, "printed_page": "186", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T77"], "text": {"as_printed": "Formally, we would say that the likelihood function is itself a random function, $\\mathcal{L}:\\Omega\\to\\mathscr{L}$, where $\\mathscr{L}$ is the space of functions $L:\\hl{\\mathbb{R}}\\to\\hl{[0,1]}$.", "corrected": "Formally, we would say that the likelihood function is itself a random function, $\\mathcal{L}:\\Omega\\to\\mathscr{L}$, where $\\mathscr{L}$ is the space of functions $L:\\hl{\\boldsymbol{\\Theta}}\\to\\hl{[0,\\infty)}$.\n\nA finite-valued likelihood is a nonnegative function on its parameter space $\\boldsymbol{\\Theta}$; its values need not be bounded by one.", "evidence": "The evidence is an exact counterexample and a check of the pages. Definition 5.1.1 (p.~179) takes $\\boldsymbol{\\Theta}\\subseteq\\mathbb{R}^{\\tau}$, and in the normal model of Example 5.2.4 (p.~190) the parameter $(\\mathbf{b},s)$ has $K+2$ coordinates. In that model, one observation with $Y_i=\\mathbf{X}_i\\mathbf{b}$ and $s=0.1$ has likelihood\n\\[\n\\phi\\bigl(0;(0,0.1^2)\\bigr)=\\frac{1}{0.1\\sqrt{2\\pi}}\\approx3.989>1.\n\\]", "problem": "The likelihood of Definition 5.2.1 is evaluated at a parameter value $\\mathbf{t}\\in\\boldsymbol{\\Theta}\\subseteq\\mathbb{R}^{\\tau}$, which is a vector unless $\\tau=1$. For a continuous outcome its value is a density, and a density can exceed one. A discrete outcome ensures that likelihood values lie in $[0,1]$, but density values need not obey this bound. The description of the likelihood as a random function is correct.", "what_survives": "The definition of the likelihood in the body of the page and the likelihood calculations that follow are unaffected, and nothing uses the footnote. A fully general statement would also specify measurability and allow for densities that take the value $+\\infty$; the correction below is for finite-valued likelihoods."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the box of Definition 5.2.10, Maximum Likelihood Plug-In Estimator, to the end of p. 197. In the display for F-hat-ML(y, x), the indicator reads 'I(X_[1]i <= x_[1]i, X_[2] <= x_[2]i, ..., X_[K]i <= x_[x])', followed by the sum over v <= y of g(v, X_i; theta-hat-ML). One red box marks 'x_[1]i, X_[2] <= x_[2]i' and a second red box marks 'x_[x]'.", "caption": "Definition 5.2.10, p. 197 (PDF p. 215), the display for a PMF. The marked subscripts are wrong: the thresholds $x_{[1]i}$ and $x_{[2]i}$ carry an observation index, $X_{[2]}$ lacks one, and the last threshold is printed $x_{[x]}$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E007.png"}, {"alt": "Crop of the end of the box of Definition 5.2.10 at the top of p. 198. The first line reads 'if f_{Y|X}(y|x) is a PMF, and'. In the display for F-hat-ML(y, x), the indicator reads 'I(X_[1]i <= x_[1]i, X_[2] <= x_[2]i, ..., X_[K]i <= x_[x])', followed by the integral from minus infinity to y of g(v, X_i; theta-hat-ML) dv. One red box marks 'x_[1]i, X_[2] <= x_[2]i' and a second red box marks 'x_[x]'. The last line reads 'if f_{Y|X}(y|x) is a PDF.'", "caption": "Definition 5.2.10, continued on p. 198 (PDF p. 216), the display for a PDF. The indicator repeats the subscripts of the first display.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E007-2.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T79", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T79"}], "headline": "In both displays of Definition 5.2.10 the indicator should read $\\mathrm{I}(X_{[1]i}\\le x_{[1]},X_{[2]i}\\le x_{[2]},\\ldots,X_{[K]i}\\le x_{[K]})$.", "id": "P001-C05-E007", "location": "Definition 5.2.10, pp. 197–198 (PDF pp. 215–216), the indicator in both displays for $\\hat F_{ML}$", "pdf_page": 215, "printed_page": "197–198", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T79"], "text": {"as_printed": "\\textbf{Definition 5.2.10.} \\emph{Maximum Likelihood Plug-In Estimator}\n\nFor i.i.d. random vectors $(Y_1,\\mathbf X_1)$, $(Y_2,\\mathbf X_2)$, \\ldots, $(Y_n,\\mathbf X_n)$ with common CDF $F$, the \\emph{ML plug-in estimator} of $\\psi=T(F)$ is\n\\[\n\\hat\\psi_{ML}=T\\big(\\hat F_{ML}\\big),\n\\]\nwhere\n\\[\n\\hat F_{ML}(y,\\mathbf x)=\\frac{1}{n}\\sum_{i=1}^n\\left(\\mathrm{I}\\big(X_{[1]i}\\le\\hl{x_{[1]i}},\\hl{X_{[2]}}\\le\\hl{x_{[2]i}},\\ldots,X_{[K]i}\\le\\hl{x_{[x]}}\\big)\\sum_{v\\le y}g(v,\\mathbf X_i;\\hat{\\boldsymbol\\theta}_{ML})\\right)\n\\]\nif $f_{Y|\\mathbf X}(y|\\mathbf x)$ is a PMF, and\n\\[\n\\hat F_{ML}(y,\\mathbf x)=\\frac{1}{n}\\sum_{i=1}^n\\left(\\mathrm{I}\\big(X_{[1]i}\\le\\hl{x_{[1]i}},\\hl{X_{[2]}}\\le\\hl{x_{[2]i}},\\ldots,X_{[K]i}\\le\\hl{x_{[x]}}\\big)\\int_{-\\infty}^{y}g(v,\\mathbf X_i;\\hat{\\boldsymbol\\theta}_{ML})dv\\right)\n\\]\nif $f_{Y|\\mathbf X}(y|\\mathbf x)$ is a PDF.", "corrected": "\\textbf{Definition 5.2.10.} \\emph{Maximum Likelihood Plug-In Estimator}\n\nFor i.i.d. random vectors $(Y_1,\\mathbf X_1)$, $(Y_2,\\mathbf X_2)$, \\ldots, $(Y_n,\\mathbf X_n)$ with common CDF $F$, the \\emph{ML plug-in estimator} of $\\psi=T(F)$ is\n\\[\n\\hat\\psi_{ML}=T\\big(\\hat F_{ML}\\big),\n\\]\nwhere\n\\[\n\\hat F_{ML}(y,\\mathbf x)=\\frac{1}{n}\\sum_{i=1}^n\\left(\\mathrm{I}\\big(X_{[1]i}\\le\\hl{x_{[1]}},\\hl{X_{[2]i}}\\le\\hl{x_{[2]}},\\ldots,X_{[K]i}\\le\\hl{x_{[K]}}\\big)\\sum_{v\\le y}g(v,\\mathbf X_i;\\hat{\\boldsymbol\\theta}_{ML})\\right)\n\\]\nif $f_{Y|\\mathbf X}(y|\\mathbf x)$ is a PMF, and\n\\[\n\\hat F_{ML}(y,\\mathbf x)=\\frac{1}{n}\\sum_{i=1}^n\\left(\\mathrm{I}\\big(X_{[1]i}\\le\\hl{x_{[1]}},\\hl{X_{[2]i}}\\le\\hl{x_{[2]}},\\ldots,X_{[K]i}\\le\\hl{x_{[K]}}\\big)\\int_{-\\infty}^{y}g(v,\\mathbf X_i;\\hat{\\boldsymbol\\theta}_{ML})dv\\right)\n\\]\nif $f_{Y|\\mathbf X}(y|\\mathbf x)$ is a PDF. Here $\\mathbf x=(x_{[1]},\\ldots,x_{[K]})$ is the point at which the CDF is evaluated, and $X_{[k]i}$ is the $k$th component of $\\mathbf X_i$.", "evidence": "The evidence is a derivation, with a check of both pages. Let $A_{\\mathbf x}$ be the event that $X_{[1]}\\le x_{[1]},\\ldots,X_{[K]}\\le x_{[K]}$. By the Law of Iterated Expectations, which the book invokes below the definition, I have\n\\[\nF(y,\\mathbf x)=\\mathrm{E}\\big[\\mathrm{I}(A_{\\mathbf x})\\,\\mathrm{I}(Y\\le y)\\big]=\\mathrm{E}\\big[\\mathrm{I}(A_{\\mathbf x})\\Pr[Y\\le y\\,|\\,\\mathbf X]\\big].\n\\]\nSubstituting the empirical distribution of $\\mathbf X$ for its population distribution, and the fitted model for the conditional distribution of $Y$, I obtain\n\\[\n\\hat F_{ML}(y,\\mathbf x)=\\frac{1}{n}\\sum_{i=1}^n\\mathrm{I}\\big(X_{[1]i}\\le x_{[1]},\\ldots,X_{[K]i}\\le x_{[K]}\\big)\\,G(y|\\mathbf X_i),\n\\]\nwhere $G(y|\\mathbf X_i)$ is $\\sum_{v\\le y}g(v,\\mathbf X_i;\\hat{\\boldsymbol\\theta}_{ML})$ for a PMF and $\\int_{-\\infty}^yg(v,\\mathbf X_i;\\hat{\\boldsymbol\\theta}_{ML})dv$ for a PDF. Every summand uses the $i$th observation in each coordinate and the same fixed thresholds. As a check, take $K=2$ and $n=2$, with $\\mathbf X_1=(0,2)$ and $\\mathbf X_2=(2,1)$, a fitted Bernoulli model with success probability $1/4$ at $\\mathbf X_1$ and $3/4$ at $\\mathbf X_2$, $y=0$ and $\\mathbf x=(1,3)$. Only $\\mathbf X_1$ satisfies both inequalities, and the corrected formula gives $\\hat F_{ML}(0,(1,3))=(1-1/4)/2=3/8$.", "problem": "The indicator should compare the $i$th observation with the fixed point $\\mathbf x=(x_{[1]},\\ldots,x_{[K]})$ in every coordinate. As printed in both displays, the thresholds $x_{[1]i}$ and $x_{[2]i}$ carry an observation index although they are coordinates of the fixed point, $X_{[2]}$ lacks the index $i$, and the last threshold is $x_{[x]}$ where $x_{[K]}$ is meant. The printed expression is not defined in the notation of the book and cannot be evaluated as written.", "what_survives": "The ML plug-in estimator, its construction from the Law of Iterated Expectations and the plug-in estimates of the CEF on p.~198 are unaffected. Only the subscripts of the two indicators change."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of the page. The first line reads 'Let L(t) = E[log g(Y, X; t)] = E[log L(t | Y, X)], so that we can write the above statement as', followed by the display 'L_n(t) converges in probability to L(t)'. In the sentence below it, red boxes mark the word 'likelihood' in 'the sample mean of the likelihood of each observation' and again in 'the expected likelihood of a single observation'. The crop ends with the lemma 'L(t) is at most L(theta)'.", "caption": "Section 5.2.2, p. 194 (PDF p. 212), the sentence after the display $L_n(\\mathbf t)\\xrightarrow{p}L(\\mathbf t)$. The two marked words stand for the log-likelihood: the line above defines $L(\\mathbf t)$ as the expectation of $\\log g$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E008.png"}, {"alt": "Crop of the seven-line paragraph that begins 'What this proof sketch illustrates is that the ML estimator is itself a type of plug-in estimator.' A red box marks the word 'likelihood' at the end of the fifth line, in 'the parameter values that maximize the sample mean of the likelihood function'. Two lines above, the paragraph reads 'the parameter values that maximize the expected value of the log-likelihood function'.", "caption": "Section 5.2.2, p. 194 (PDF p. 212), the paragraph after the proof sketch. The marked word stands for the log-likelihood; the preceding sentence of the paragraph speaks of “the expected value of the log-likelihood function”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E008-2.png"}], "formal_release": null, "formal_statements": [], "headline": "In three places on p. 194, “likelihood” means the log-likelihood, as the adjacent definitions of $L_n$ and $L$ show.", "id": "P001-C05-E008", "location": "Section 5.2.2, p. 194 (PDF p. 212): the sentence after the display $L_n(\\mathbf t)\\xrightarrow{p}L(\\mathbf t)$, and the paragraph after the proof sketch", "pdf_page": 212, "printed_page": "194", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Let $L(\\mathbf t)=\\mathrm{E}[\\log g(Y,\\mathbf X;\\mathbf t)]=\\mathrm{E}[\\log\\mathcal L(\\mathbf t|Y,\\mathbf X)]$, so that we can write the above statement as\n\\[\nL_n(\\mathbf t)\\xrightarrow{p}L(\\mathbf t),\n\\]\nthat is, the sample mean of the \\hl{likelihood} of each observation given parameter values $\\mathbf t$ converges to the expected \\hl{likelihood} of a single observation given parameter values $\\mathbf t$.\n\nIn the paragraph after the proof sketch:\n\nTo estimate this population quantity, we use the sample analog: the parameter values that maximize the sample mean of the \\hl{likelihood} function.", "corrected": "that is, the sample mean of the \\hl{log-likelihood} of each observation given parameter values $\\mathbf t$ converges to the expected \\hl{log-likelihood} of a single observation given parameter values $\\mathbf t$.\n\nTo estimate this population quantity, we use the sample analog: the parameter values that maximize the sample mean of the \\hl{log-likelihood} function.", "intended_meaning": "The quantities in these sentences are averages of logarithms. On p.~193 the book sets $L_n(\\mathbf t)=n^{-1}\\sum_{i=1}^n\\log g(Y_i,\\mathbf X_i;\\mathbf t)$, and on p.~194 it sets $L(\\mathbf t)=\\mathrm{E}[\\log g(Y,\\mathbf X;\\mathbf t)]$. Between the two marked passages, the text speaks of ``the expected value of the log-likelihood function''. The intended meaning is the sample mean and the expectation of the log-likelihood, and with this meaning these sentences agree with the displayed criteria. The difference matters only if the prose is read apart from the equations: for the coin data $(1,0,1)$ of the chapter, the mean of the three likelihoods is $(1+p)/3$, which is largest at $p=1$, and the mean log-likelihood is largest at the ML estimate $p=2/3$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "End of Definition 5.1.4 in a shaded box. It says that beta is the vector of beta 0 to beta K, that phi of y with parameters mu and sigma squared denotes the PDF of the normal distribution with mean mu and variance sigma squared, and that the parameter space Theta is the set of pairs beta, sigma in R to the K plus 2 with sigma greater than or equal to 0. A red box marks the condition sigma greater than or equal to 0.", "caption": "Definition 5.1.4, last lines, p. 181 (PDF p. 199). The marked condition $\\sigma\\ge0$ should be $\\sigma>0$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T72", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T72"}], "headline": "The parameter space of the classical linear model should require $\\sigma>0$; at $\\sigma=0$ the normal density that defines the model does not exist.", "id": "P001-C05-E009", "location": "Definition 5.1.4 (Classical Linear Model), last line, p. 181 (PDF p. 199); the definition begins on p. 180 (PDF p. 198)", "pdf_page": 199, "printed_page": "181", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T72"], "text": {"as_printed": "For a random variable $Y$ and random vector $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, the \\emph{classical linear model} is a parametric model with\n\\[\ng\\bigl(y,\\mathbf{x};(\\boldsymbol{\\beta},\\sigma)\\bigr)=\\phi\\bigl(y;(\\mathbf{x}\\boldsymbol{\\beta},\\sigma^2)\\bigr),\n\\]\nwhere $\\boldsymbol{\\beta}=(\\beta_0,\\beta_1,\\ldots,\\beta_K)^T$, $\\phi(y;(\\mu,\\sigma^2))$ denotes the PDF of the normal distribution with mean $\\mu$ and variance $\\sigma^2$, and the parameter space is $\\boldsymbol{\\Theta}=\\{(\\boldsymbol{\\beta},\\sigma)\\in\\mathbb{R}^{K+2}:\\sigma\\hl{\\ge}0\\}$.", "corrected": "For a random variable $Y$ and random vector $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, the \\emph{classical linear model} is a parametric model with\n\\[\ng\\bigl(y,\\mathbf{x};(\\boldsymbol{\\beta},\\sigma)\\bigr)=\\phi\\bigl(y;(\\mathbf{x}\\boldsymbol{\\beta},\\sigma^2)\\bigr),\n\\]\nwhere $\\boldsymbol{\\beta}=(\\beta_0,\\beta_1,\\ldots,\\beta_K)^T$, $\\phi(y;(\\mu,\\sigma^2))$ denotes the PDF of the normal distribution with mean $\\mu$ and variance $\\sigma^2$, and the parameter space is $\\boldsymbol{\\Theta}=\\{(\\boldsymbol{\\beta},\\sigma)\\in\\mathbb{R}^{K+2}:\\sigma\\hl{>}0\\}$.", "evidence": "The evidence is a check of the pages. In Definition 2.1.19, the normal PDF is\n\\[\nf(x)=\\frac{1}{\\sigma\\sqrt{2\\pi}}e^{-\\frac{(x-\\mu)^2}{2\\sigma^2}},\\qquad\\forall x\\in\\mathbb{R},\n\\]\nfor constants $\\mu,\\sigma\\in\\mathbb{R}$ with $\\sigma>0$. At $\\sigma=0$ both denominators vanish. The distribution at that boundary is a point mass at $\\mathbf{x}\\boldsymbol{\\beta}$, which has no density with respect to Lebesgue measure.", "problem": "The model is defined through the normal PDF $\\phi(y;(\\mathbf{x}\\boldsymbol{\\beta},\\sigma^2))$, and Definition 2.1.19 (p.~55) defines that PDF for $\\sigma>0$ only, since its formula divides by $\\sigma$. The printed parameter space includes $\\sigma=0$, where the density is undefined. The inequality should be strict.", "what_survives": "The model with $\\sigma>0$, Theorem 5.1.5 and the likelihood calculations that follow are unaffected. One could treat $\\sigma=0$ separately as a point mass, but the correction does not need it."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Theorem 5.1.8, CEF of a Binary Choice Model, in a shaded box. Its display reads E[Y|X = x] = h(X beta), for all x in the support of X. A red box marks h(X beta), which is printed with a capital bold X although the left side conditions on X = x.", "caption": "Theorem 5.1.8, p. 183 (PDF p. 201). In the marked term the argument of $h$ is the random vector $\\mathbf{X}$, where the fixed value $\\mathbf{x}$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E010.png"}, {"alt": "Definition 5.1.7, Binary Choice Model, in a shaded box. The display defines g(y, X; beta) as 1 minus h(X beta) when y = 0, as h(X beta) when y = 1, and as 0 otherwise. Red boxes mark g(y, X; beta) and the two occurrences of h(X beta), each printed with a capital bold X.", "caption": "Definition 5.1.7, p. 182 (PDF p. 200). The three marked terms have the random vector $\\mathbf{X}$ where the argument $\\mathbf{x}$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E010-b.png"}, {"alt": "A four-line display that begins: beta hat ML = arg max of log L(b | Y, X). Under each of the three arg max operators and under the final arg min operator stands the domain b in R to the K, and each of the four is marked with a red box. A fifth red box marks L(b | Y, X) in the first line; the sentence above the display writes log L((b, s) | Y, X). The second line contains s in the terms 1 over s root 2 pi and 2 s squared.", "caption": "Example 5.2.4, p. 191 (PDF p. 209), the display for $\\hat{\\boldsymbol{\\beta}}_{ML}$. The four marked domains read $\\mathbf{b}\\in\\mathbb{R}^{K}$, where $\\mathbb{R}^{K+1}$ is meant, and the marked likelihood in the first line omits $s$ from its argument.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E010-c.png"}, {"alt": "Paragraph on l1-penalized ML estimation. A red box marks the word flexibility in the sentence: Mirroring the language of Section 4.3.6, it is a way of flexibility estimating the conditional density function when we have a very large number of parameters to estimate.", "caption": "Section 5.2.6, p. 202 (PDF p. 220). The marked word should be “flexibly”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-E010-d.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T75", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T75"}, {"id": "P001-T76", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T76"}], "headline": "Three local slips: Theorem 5.1.8 should read $h(\\mathbf{x}\\boldsymbol{\\beta})$, the maximization in Example 5.2.4 is over $\\mathbf{b}\\in\\mathbb{R}^{K+1}$, and “flexibility estimating” on p. 202 should read “flexibly estimating”.", "id": "P001-C05-E010", "location": "Theorem 5.1.8, p. 183 (PDF p. 201), with Definition 5.1.7, p. 182 (PDF p. 200); Example 5.2.4, the display for $\\hat{\\boldsymbol{\\beta}}_{ML}$, p. 191 (PDF p. 209); Section 5.2.6, second paragraph, p. 202 (PDF p. 220)", "pdf_page": 201, "printed_page": "183", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T75", "P001-T76"], "text": {"as_printed": "(a) Theorem 5.1.8, p.~183:\n\nFor a binary random variable $Y$ and random vector $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, if the binary choice model with mean function $h$ is true, the CEF of $Y$ given $\\mathbf{X}$ is\n\\[\n\\mathrm{E}[Y|\\mathbf{X}=\\mathbf{x}]=h(\\hl{\\mathbf{X}}\\boldsymbol{\\beta}),\\ \\forall\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}].\n\\]\nDefinition 5.1.7, p.~182, the display:\n\\[\ng(y,\\hl{\\mathbf{X}};\\boldsymbol{\\beta})=\n\\begin{cases}\n1-h(\\hl{\\mathbf{X}}\\boldsymbol{\\beta}) & :\\ y=0\\\\\nh(\\hl{\\mathbf{X}}\\boldsymbol{\\beta}) & :\\ y=1\\\\\n0 & :\\ \\text{otherwise,}\n\\end{cases}\n\\]\n(b) Example 5.2.4, p.~191:\n\nWe want to find the $\\mathbf{b}$ that maximizes $\\log\\mathcal{L}((\\mathbf{b},s)\\,|\\,\\mathbf{Y},\\mathbb{X})$:\n\\[\n\\begin{aligned}\n\\hat{\\boldsymbol{\\beta}}_{ML}&=\\operatorname*{arg\\,max}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K}}}\\bigl(\\log\\mathcal{L}(\\hl{\\mathbf{b}}\\,|\\,\\mathbf{Y},\\mathbb{X})\\bigr)\\\\\n&=\\operatorname*{arg\\,max}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K}}}\\sum_{i=1}^{n}\\left(\\log\\left(\\frac{1}{s\\sqrt{2\\pi}}\\right)+\\frac{-(Y_i-\\mathbf{X}_i\\mathbf{b})^2}{2s^2}\\right)\\\\\n&=\\operatorname*{arg\\,max}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K}}}\\sum_{i=1}^{n}\\bigl(-(Y_i-\\mathbf{X}_i\\mathbf{b})^2\\bigr)\\\\\n&=\\operatorname*{arg\\,min}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K}}}\\sum_{i=1}^{n}(Y_i-\\mathbf{X}_i\\mathbf{b})^2.\n\\end{aligned}\n\\]\n(c) Section 5.2.6, p.~202:\n\nMirroring the language of Section 4.3.6, it is a way of \\hl{flexibility} estimating the conditional density function when we have a very large number of parameters to estimate.", "corrected": "(a) Theorem 5.1.8:\n\nFor a binary random variable $Y$ and random vector $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, if the binary choice model with mean function $h$ is true, the CEF of $Y$ given $\\mathbf{X}$ is\n\\[\n\\mathrm{E}[Y|\\mathbf{X}=\\mathbf{x}]=h(\\hl{\\mathbf{x}}\\boldsymbol{\\beta}),\\ \\forall\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}].\n\\]\nDefinition 5.1.7:\n\nFor a binary random variable $Y$ and random vector $\\mathbf{X}=(1,X_{[1]},X_{[2]},\\ldots,X_{[K]})$, a \\emph{binary choice model} is a parametric model with\n\\[\ng(y,\\hl{\\mathbf{x}};\\boldsymbol{\\beta})=\n\\begin{cases}\n1-h(\\hl{\\mathbf{x}}\\boldsymbol{\\beta}) & :\\ y=0\\\\\nh(\\hl{\\mathbf{x}}\\boldsymbol{\\beta}) & :\\ y=1\\\\\n0 & :\\ \\text{otherwise,}\n\\end{cases}\n\\]\nwhere $\\boldsymbol{\\beta}=(\\beta_0,\\beta_1,\\ldots,\\beta_K)^T$, the mean function $h:\\mathbb{R}\\to[0,1]$ is known, and the parameter space is $\\boldsymbol{\\Theta}=\\mathbb{R}^{K+1}$.\n\n(b) Example 5.2.4, with $s>0$ held fixed:\n\\[\n\\begin{aligned}\n\\hat{\\boldsymbol{\\beta}}_{ML}&=\\operatorname*{arg\\,max}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K+1}}}\\bigl(\\log\\mathcal{L}(\\hl{(\\mathbf{b},s)}\\,|\\,\\mathbf{Y},\\mathbb{X})\\bigr)\\\\\n&=\\operatorname*{arg\\,max}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K+1}}}\\sum_{i=1}^{n}\\left(\\log\\left(\\frac{1}{s\\sqrt{2\\pi}}\\right)+\\frac{-(Y_i-\\mathbf{X}_i\\mathbf{b})^2}{2s^2}\\right)\\\\\n&=\\operatorname*{arg\\,max}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K+1}}}\\sum_{i=1}^{n}\\bigl(-(Y_i-\\mathbf{X}_i\\mathbf{b})^2\\bigr)\\\\\n&=\\operatorname*{arg\\,min}_{\\mathbf{b}\\in\\hl{\\scriptstyle\\mathbb{R}^{K+1}}}\\sum_{i=1}^{n}(Y_i-\\mathbf{X}_i\\mathbf{b})^2.\n\\end{aligned}\n\\]\n(c) Section 5.2.6:\n\nMirroring the language of Section 4.3.6, it is a way of \\hl{flexibly} estimating the conditional density function when we have a very large number of parameters to estimate.", "evidence": "The evidence is a check of the pages and a short derivation. For a fixed $\\mathbf{x}$, under the binary choice model I have\n\\[\n\\mathrm{E}[Y|\\mathbf{X}=\\mathbf{x}]=\\sum_{y\\in\\{0,1\\}}y\\,g(y,\\mathbf{x};\\boldsymbol{\\beta})=0\\cdot\\bigl(1-h(\\mathbf{x}\\boldsymbol{\\beta})\\bigr)+1\\cdot h(\\mathbf{x}\\boldsymbol{\\beta})=h(\\mathbf{x}\\boldsymbol{\\beta}).\n\\]\nThis is a number. The printed $h(\\mathbf{X}\\boldsymbol{\\beta})$ is in general random. The book writes $h(\\mathbf{x}\\hat{\\boldsymbol{\\beta}}_{ML})$ when it applies the theorem on p.~198. For (b), p.~190 defines $\\mathbf{b}$ with the entries $b_0,\\ldots,b_K$ and $\\mathbb{X}$ with $K+1$ columns. The product $\\mathbf{X}_i\\mathbf{b}$ then requires $\\mathbf{b}\\in\\mathbb{R}^{K+1}$, and the parallel display for the probit model on p.~193 has $\\mathbb{R}^{K+1}$. At any fixed $s>0$, the log-likelihood equals\n\\[\n-n\\log\\bigl(s\\sqrt{2\\pi}\\bigr)-\\frac{1}{2s^2}\\sum_{i=1}^{n}(Y_i-\\mathbf{X}_i\\mathbf{b})^2,\n\\]\nand its maximizers in $\\mathbf{b}$ are the minimizers of the sum of squared residuals. The chain of equalities holds once the domain is corrected.", "problem": "(a) The left side of the display in Theorem 5.1.8 conditions on $\\mathbf{X}=\\mathbf{x}$ and is a number for each $\\mathbf{x}$, but the right side is written with the random vector $\\mathbf{X}$. Definition 5.1.7 has the same slip in the function $g$, whose covariate argument is the lower-case $\\mathbf{x}$ in Definitions 5.1.1 and 5.1.4. (b) The vector $\\mathbf{b}=(b_0,b_1,\\ldots,b_K)^T$ has $K+1$ entries, one for each column of the matrix $\\mathbb{X}$, whose first column consists of ones, but the four domains are printed as $\\mathbb{R}^{K}$. The first line also drops $s$ from the argument of the likelihood, which the sentence above the display writes as $\\mathcal{L}((\\mathbf{b},s)\\,|\\,\\mathbf{Y},\\mathbb{X})$, although $s$ appears in the second line. (c) ``Flexibility'' is printed for ``flexibly''. Section 4.3.6, whose language the sentence mirrors, speaks of ``a way of flexibly estimating the CEF''.", "what_survives": "The CEF of the binary choice model, the result of Example 5.2.4 that the ML estimator of $\\boldsymbol{\\beta}$ minimizes the sum of squared residuals whenever a positive-scale ML solution exists, and the discussion of penalized ML estimation are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-E010", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Paragraph of text. A red box marks Section 7.2.3 in the sentence: In settings where respecting the bounds of predictions is important (we will have such a case in Section 7.2.3), we will generally prefer to use a binary choice model rather than an approach (like regression) that might yield nonsensical predictions.", "caption": "The paragraph after Theorem 5.1.8, p. 183 (PDF p. 201). The marked forward reference should be to Section 7.2.5.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "The forward reference to Section 7.2.3 should point to Section 7.2.5, where propensity scores are estimated with a binary choice model so that the estimates stay in $[0,1]$.", "id": "P001-C05-N002", "location": "p. 183 (PDF p. 201), the paragraph after Theorem 5.1.8", "pdf_page": 201, "printed_page": "183", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "In settings where respecting the bounds of predictions is important (we will have such a case in \\hl{Section 7.2.3}), we will generally prefer to use a binary choice model rather than an approach (like regression) that might yield nonsensical predictions.", "corrected": "In settings where respecting the bounds of predictions is important (we will have such a case in \\hl{Section 7.2.5}), we will generally prefer to use a binary choice model rather than an approach (like regression) that might yield nonsensical predictions.\n\nThe reference could also be given as ``Sections 6.2.4 and 7.2.5''.", "evidence": "The evidence is a check of the pages of Sections 7.2.3, 7.2.5 and 6.2.4. Both Section 7.2.5 and Section 6.2.4 fit the sentence, and the intended target is not certain. Section 7.2.5 is the one that refers back to Section 5.1.2.", "problem": "Section 7.2.3 (pp.~261--262) discusses conditional outcome models and causal effects in general terms. It contains no case in which predictions must respect bounds and no binary choice model. The case that the sentence promises is in Section 7.2.5 (p.~264), which says that a linear approximation could predict propensity score estimates ``greater than one or less than zero'' and turns to a binary choice model, with a reference back to Section 5.1.2. Section 6.2.4 (pp.~225--226) treats the same point for response propensities.", "what_survives": "Nothing else depends on the reference."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-N002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Text of Example 5.2.3 with two displays of the likelihood of p hat given Y. Red boxes mark g(Y i | p hat) in two places: at the start of the line g(Y i | p hat) = 1 minus p hat, so, and at the start of the line g(Y i | p hat) = p hat, so.", "caption": "Example 5.2.3, p. 188 (PDF p. 206). In the two marked terms a bar separates $\\hat p$ from the argument of $g$, where the book’s convention is a semicolon.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N003.png"}, {"alt": "Display: L((b, s) | Y, X) equals the product over i from 1 to n of phi(Y i, (X i b, s squared)), which equals the product over i of 1 over s root 2 pi times e to the power minus (Y i minus X i b) squared over 2 s squared. A red box marks Y i and the comma that follows it inside phi.", "caption": "Example 5.2.4, the likelihood display, p. 190 (PDF p. 208). The marked comma after $Y_i$ stands where Definition 5.1.4 has a semicolon.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N003-b.png"}, {"alt": "Start of Example 5.2.5, Maximum Likelihood Estimation of the Probit Model. Its second display reads: f of Y given X, evaluated at (Y|X), = g(Y, X; beta) = Phi(X beta) to the power Y times (1 minus Phi(X beta)) to the power 1 minus Y. A red box marks g(Y, X; beta), where the X is italic and not bold, unlike the other occurrences of X in the display.", "caption": "Example 5.2.5, p. 192 (PDF p. 210). In the marked term the covariate vector is an italic $X$; it is bold elsewhere in the display.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N003-c.png"}, {"alt": "Paragraph of text. A red box marks an italic X in the sentence: If we have estimates of the full distribution of Y conditional on X, we can use that information to construct a standard error estimate.", "caption": "Section 5.2.7, p. 203 (PDF p. 221). The marked $X$ is italic, where the bold vector $\\mathbf{X}$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N003-d.png"}], "formal_release": null, "formal_statements": [], "headline": "Four notational inconsistencies: a bar for the semicolon in $g(Y_i;\\hat p)$, twice; a comma for the semicolon in $\\phi(Y_i;(\\mathbf{X}_i\\mathbf{b},s^2))$; and the covariate vector $\\mathbf{X}$ set without bold face in two places.", "id": "P001-C05-N003", "location": "Example 5.2.3, p. 188 (PDF p. 206); Example 5.2.4, the likelihood display, p. 190 (PDF p. 208); Example 5.2.5, p. 192 (PDF p. 210); Section 5.2.7, p. 203 (PDF p. 221)", "pdf_page": 206, "printed_page": "188", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "(a) Example 5.2.3, p.~188, in two places:\n\\[\n\\mathcal{L}(\\hat p\\,|\\,Y_i)=g(Y_i\\hl{|}\\hat p)=1-\\hat p\n\\qquad\\text{and}\\qquad\n\\mathcal{L}(\\hat p\\,|\\,Y_i)=g(Y_i\\hl{|}\\hat p)=\\hat p\n\\]\n(b) Example 5.2.4, p.~190:\n\\[\n\\mathcal{L}\\bigl((\\mathbf{b},s)\\,|\\,\\mathbf{Y},\\mathbb{X}\\bigr)=\\prod_{i=1}^{n}\\phi\\bigl(Y_i\\hl{,}(\\mathbf{X}_i\\mathbf{b},s^2)\\bigr)\n\\]\n(c) Example 5.2.5, p.~192:\n\\[\nf_{Y|\\mathbf{X}}(Y|\\mathbf{X})=g(Y,\\hl{X};\\boldsymbol{\\beta})=\\Phi(\\mathbf{X}\\boldsymbol{\\beta})^{Y}\\bigl(1-\\Phi(\\mathbf{X}\\boldsymbol{\\beta})\\bigr)^{1-Y}.\n\\]\n(d) Section 5.2.7, p.~203:\n\nIf we have estimates of the full distribution of $Y$ conditional on \\hl{$X$}, we can use that information to construct a standard error estimate.", "corrected": "(a) $g(Y_i\\hl{;}\\hat p)$, in both places.\n\n(b) $\\phi\\bigl(Y_i\\hl{;}(\\mathbf{X}_i\\mathbf{b},s^2)\\bigr)$.\n\n(c) $g(Y,\\hl{\\mathbf{X}};\\boldsymbol{\\beta})$.\n\n(d) If we have estimates of the full distribution of $Y$ conditional on \\hl{$\\mathbf{X}$}, we can use that information to construct a standard error estimate.", "evidence": "The evidence is a check of the pages. Footnote 1 reads ``We separate $\\boldsymbol{\\theta}$ from the arguments of $g$ with a semicolon.'' On p.~192 the italic $X$ stands in the same display as the bold $\\mathbf{X}$.", "problem": "Footnote 1 on p.~179 fixes the convention that $\\boldsymbol{\\theta}$ is separated from the arguments of $g$ by a semicolon, and Definition 5.1.4 writes the normal PDF as $\\phi(y;(\\mu,\\sigma^2))$. Items (a) and (b) depart from these conventions. In (c) and (d) the covariate vector, which is the bold $\\mathbf{X}$ elsewhere, is set in italic. In each case the intended meaning is clear, and no mathematics is affected.", "what_survives": "Nothing depends on these four places. I do not count as an error the sentence on p.~195 that writes $P$ and $Q$ for the distributions with densities $p(\\cdot)$ and $q(\\cdot)$. The letters are not introduced there, but the reading is standard and nothing false follows from it."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-N003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Figure 5.2.2. The left panel, titled 'Histogram of Y', shows a histogram on the range 0 to 5 with its tallest bar, of height about 0.74, just above Y = 1. The right panel, titled 'Normal ML Fit', shows the same histogram with a red bell-shaped curve that peaks at a height of about 0.52 near Y = 1.6. Red boxes mark the title 'Normal ML Fit' and the caption 'FIGURE 5.2.2. Minimum KL Divergence Approximation of an Unusual Distribution'.", "caption": "Figure 5.2.2, p. 197 (PDF p. 215). The marked title and caption present the red curve as the normal ML fit to the histogram; the book gives no data, bin heights or fitted mean and standard deviation for it.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N004.png"}], "formal_release": null, "formal_statements": [], "headline": "The fitted curves in Figures 5.2.2–5.2.5 illustrate the text and cannot be reproduced exactly, since the book reports neither the distribution behind the histogram nor the fitted parameters.", "id": "P001-C05-N004", "location": "Figures 5.2.2–5.2.5, pp. 197–201 (PDF pp. 215–219), with Example 5.2.9, p. 196 (PDF p. 214), and Example 5.2.11, pp. 199–200 (PDF pp. 217–218)", "pdf_page": 215, "printed_page": "197–201", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Example 5.2.9, p.~196:\n\nSuppose that the true distribution of $Y$ were given by the histogram on the lefthand side of Figure 5.2.2. [\\ldots] Then the asymptotic ML estimate of the distribution would be \\hl{as shown in red} on the righthand side of the figure.\n\nFIGURE 5.2.2. \\emph{Minimum KL Divergence Approximation of an Unusual Distribution}\n\nExample 5.2.11, p.~200:\n\nThe ML estimates from each of these models are \\hl{visualized in Figure 5.2.4}. [\\ldots] \\hl{Figure 5.2.5 visualizes the ML estimate} of this mixture model, which provides a markedly better approximation.\n\nFIGURE 5.2.4. \\emph{Normal, Log-Normal, and Exponential Maximum Likelihood Approximations}\n\nFIGURE 5.2.5. \\emph{Mixture Model Maximum Likelihood Approximation}", "corrected": "No wording has to change. To make the figures reproducible, Example 5.2.9 could add: ``\\hl{The histogram and the fitted curves in Figures 5.2.2--5.2.5 are illustrations; the underlying distribution, the fitted parameter values and the fitting procedure are not reported here.}'' A full account would give the distribution or data behind the histogram, the fitted parameters of each curve, and the fitting procedure, including any restriction placed on the mixture.", "intended_meaning": "These figures are illustrations. A normal distribution, and then a log-normal distribution, an exponential distribution and a mixture of the three, are fitted to the positive distribution drawn as a histogram, and the mixture follows its shape most closely. This reading is clear, and no curve is shown to be wrong. The book gives no observations, bin heights or generating distribution for the histogram, no fitted parameter values and no account of the fitting, and no replication files were available to me. The curves therefore cannot be recomputed exactly, and I did not check that each is the best fit in its family. For Figure 5.2.5 the objective and the restrictions on the parameters also matter (see \\fid{P001-C05-E002})."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-N004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the first two paragraphs of p. 199 and the opening of Example 5.2.11. The first paragraph defines theta = (p_1, theta_1, p_2, theta_2, ..., p_{M-1}, theta_{M-1}, theta_M) and says that the product of the g(Y_i, X_i; theta) can be maximized to find the ML estimate of theta. In the second paragraph, 'If any subset of the models in (g_1(y, x; theta_1), ..., g_M(y, x; theta_M)) jointly characterize the conditional distribution of Y, then this estimator is consistent and asymptotically efficient.', red boxes mark the words 'this estimator is consistent and asymptotically efficient'. Below, the heading 'Example 5.2.11. Mixture Model Approximation' and its first two lines.", "caption": "Section 5.2.5, p. 199 (PDF p. 217). The marked clause promises consistency and asymptotic efficiency for the ML estimate of the full vector $\\boldsymbol\\theta$ of the first line whenever a subset of the component models characterizes the distribution.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N005.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T11", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T11"}, {"id": "P001-T12", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T12"}, {"id": "P001-T103", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T103"}, {"id": "P001-T121", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T121"}], "headline": "When only some of the component models of a mixture are needed, the parameters of the unused components are not identified, and the ML estimator of the full parameter vector is consistent and asymptotically efficient only under identification and regularity conditions that the sentence does not state.", "id": "P001-C05-N005", "location": "Section 5.2.5, p. 199 (PDF p. 217), second paragraph", "pdf_page": 217, "printed_page": "199", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T103", "P001-T11", "P001-T12", "P001-T121"], "text": {"as_printed": "where $\\boldsymbol\\theta=(p_1,\\boldsymbol\\theta_1,p_2,\\boldsymbol\\theta_2,\\ldots,p_{M-1},\\boldsymbol\\theta_{M-1},\\boldsymbol\\theta_M)$. Then, as before, $\\prod_{i=1}^ng(Y_i,\\mathbf X_i;\\boldsymbol\\theta)$ can be maximized to find the ML estimate of $\\boldsymbol\\theta$. (This can sometimes be computationally tricky.)\n\nIf any subset of the models in $(g_1(y,\\mathbf x;\\boldsymbol\\theta_1),\\ldots,g_M(y,\\mathbf x;\\boldsymbol\\theta_M))$ jointly characterize the conditional distribution of $Y$, then \\hl{this estimator is consistent and asymptotically efficient}. When the models are not valid (even jointly), our estimates should be interpreted explicitly as an approximation.", "corrected": "If any subset of the models in $(g_1(y,\\mathbf x;\\boldsymbol\\theta_1),\\ldots,g_M(y,\\mathbf x;\\boldsymbol\\theta_M))$ jointly characterize the conditional distribution of $Y$, \\hl{then the mixture model is true. This alone does not make the ML estimator of $\\boldsymbol\\theta$ consistent or asymptotically efficient, since a component with weight zero leaves its own parameters unidentified. If in addition the true value of $\\boldsymbol\\theta$ is identified and lies in the interior of the parameter space, and the regularity conditions for ML estimation hold, among them a unique maximizer of the expected log-likelihood and a nonsingular information matrix,} then this estimator is consistent and asymptotically efficient. \\hl{A component with weight zero must not retain free parameters in the vector to which this conclusion is applied.}\n\nA sufficient set of conditions is as follows. The observations are i.i.d., the dimension of $\\boldsymbol\\theta$ is fixed, and $f_{Y|\\mathbf X}(y|\\mathbf x)=g(y,\\mathbf x;\\boldsymbol\\theta^0)$ for some $\\boldsymbol\\theta^0\\in\\boldsymbol\\Theta$. \\hl{A measurable global maximizer of the sample likelihood exists and is used as the ML estimator.} The parameter space $\\boldsymbol\\Theta$ is compact, $L(\\mathbf t)=\\mathrm{E}[\\log g(Y,\\mathbf X;\\mathbf t)]$ is continuous on $\\boldsymbol\\Theta$ and uniquely maximized at $\\boldsymbol\\theta^0$, and $\\sup_{\\mathbf t\\in\\boldsymbol\\Theta}|L_n(\\mathbf t)-L(\\mathbf t)|\\xrightarrow{p}0$, where $L_n(\\mathbf t)=n^{-1}\\sum_{i=1}^n\\log g(Y_i,\\mathbf X_i;\\mathbf t)$. The point $\\boldsymbol\\theta^0$ is interior to $\\boldsymbol\\Theta$. Near $\\boldsymbol\\theta^0$ the density $g$ is positive on a common support and twice differentiable in $\\mathbf t$, the identity $\\int g(y,\\mathbf x;\\mathbf t)\\,dy=1$ (with a sum in place of the integral for a PMF) may be differentiated twice under the integral, the score $s=\\nabla_{\\mathbf t}\\log g$ has finite second moments, the information matrix $I=\\mathrm{E}[ss^{T}]$ at $\\boldsymbol\\theta^0$ is positive definite, and the averages of the second derivatives of $\\log g$ converge uniformly near $\\boldsymbol\\theta^0$ to their expectations, which are continuous. Then $\\hat{\\boldsymbol\\theta}_{ML}\\xrightarrow{p}\\boldsymbol\\theta^0$, and $\\sqrt n(\\hat{\\boldsymbol\\theta}_{ML}-\\boldsymbol\\theta^0)$ converges in distribution to $N(0,I^{-1})$.\n\nFor the mixture of Example 5.2.11 these conditions hold when the distribution of $Y$ is this mixture with all three weights positive, and the fit is restricted to a compact set on which the three weights are at least some $\\epsilon>0$, the parameters $\\mu_N$ and $\\mu_{LN}$ are bounded, and $\\sigma_N$, $\\sigma_{LN}$ and $\\lambda_E$ are bounded above and below by positive constants, with the true parameter in the interior of that set. They fail when a weight is zero and the parameters of that component are left free. A different repair changes the target from $\\boldsymbol\\theta$ to the fitted conditional distribution, which requires conditions of its own.", "evidence": "The evidence is an exact counterexample, with a derivation that holds for every sample size. For the general mixture, let $p_j=0$ and let $\\boldsymbol\\theta^A$ and $\\boldsymbol\\theta^B$ differ only in the parameter of component $j$. Then\n\\[\ng(y,\\mathbf x;\\boldsymbol\\theta^A)-g(y,\\mathbf x;\\boldsymbol\\theta^B)=p_j\\big\\{g_j(y,\\mathbf x;\\boldsymbol\\theta_j^A)-g_j(y,\\mathbf x;\\boldsymbol\\theta_j^B)\\big\\}=0,\n\\]\nwhich implies that the two vectors give the same likelihood for every sample.\n\nFor a bounded example, let $Y$ take values in $\\{0,1,2,3\\}$, with no explanatory variables. Writing a distribution as its vector of probabilities at $y=0,1,2,3$, consider the two component models\n\\[\ng_1(\\cdot\\,;a)=(1-a,\\;a,\\;0,\\;0),\\qquad g_2(\\cdot\\,;b)=(0,\\;0,\\;1-b,\\;b),\n\\]\nand the mixture $g=p\\,g_1+(1-p)\\,g_2$ with $\\boldsymbol\\theta=(p,a,b)\\in[0,1]\\times[0,1]\\times[1/4,3/4]$. The mixture has probabilities $q=(p(1-a),\\,pa,\\,(1-p)(1-b),\\,(1-p)b)$. Since the parameter space is compact and the likelihood is continuous and at most one, a maximum exists for every sample. For $0<p<1$ the parameter is recovered from $q$ by $p=q_0+q_1$, $a=q_1/p$ and $b=q_3/(1-p)$. Set\n\\[\n\\boldsymbol\\theta_A=(1,\\,1/3,\\,1/4),\\qquad\\boldsymbol\\theta_B=(1,\\,1/3,\\,3/4).\n\\]\nThe first component alone characterizes the distribution at both, and the hypothesis of the printed sentence holds.\n\\begin{audittable}{lcc}{Two parameter vectors of the mixture $p\\,g_1+(1-p)\\,g_2$ that no sample can distinguish.}\n & $\\boldsymbol\\theta_A$ & $\\boldsymbol\\theta_B$ \\\\\n\\hline\n$(p,a,b)$ & $(1,1/3,1/4)$ & $(1,1/3,3/4)$ \\\\\nProbabilities of $y=0,1,2,3$ & $(2/3,1/3,0,0)$ & $(2/3,1/3,0,0)$ \\\\\nLikelihood of the sample $(0,0,0,0,1,1)$ & $16/729$ & $16/729$ \\\\\nDerivative of the four probabilities with respect to $b$ & $(0,0,0,0)$ & $(0,0,0,0)$ \\\\\n\\end{audittable}\nA sample of size $n$ with $k$ ones and $n-k$ zeros has probability $(1/3)^k(2/3)^{n-k}$ under both vectors, and every other sample has probability zero under both. Let $T_n$ be any estimator of $\\boldsymbol\\theta$, and let $B_n$ be its third coordinate. The events $\\{|B_n-1/4|<1/8\\}$ and $\\{|B_n-3/4|<1/8\\}$ are disjoint. Their probabilities under the common distribution of the sample then sum to at most one, and $T_n$ cannot be consistent at both $\\boldsymbol\\theta_A$ and $\\boldsymbol\\theta_B$. The likelihood of a sample of zeros and ones is $p^na^k(1-a)^{n-k}$, which does not involve $b$. Every $(1,k/n,b)$ with $b\\in[1/4,3/4]$ is then a global maximizer. The ML estimator that selects $b=1/4$ has an error of $1/2$ in the third coordinate at $\\boldsymbol\\theta_B$ for every $n$. The matrix of second moments of the score at $\\boldsymbol\\theta_A$ is $\\operatorname{diag}(1,9/2,0)$, which has rank two. For positive weights, a mixture of a $\\operatorname{Bernoulli}(a)$ and a $\\operatorname{Bernoulli}(b)$ component with weight $p$ is $\\operatorname{Bernoulli}(pa+(1-p)b)$, and $(p,a,b)=(1/2,1/5,4/5)$ and $(1/2,3/10,7/10)$ both give $\\operatorname{Bernoulli}(1/2)$.\n\nThe corrected statement is the standard result for regular ML estimation, and I sketch its derivation in the notation of that statement. By the inequality $\\log z\\le z-1$, I have $L(\\mathbf t)\\le L(\\boldsymbol\\theta^0)$ for every $\\mathbf t$, with equality only if $g(\\cdot,\\cdot;\\mathbf t)$ and $g(\\cdot,\\cdot;\\boldsymbol\\theta^0)$ agree almost surely, and identification then implies that $\\mathbf t=\\boldsymbol\\theta^0$. Consistency follows from the uniform convergence of $L_n$ to $L$ and the gap between $L(\\boldsymbol\\theta^0)$ and the maximum of $L$ outside a neighborhood of $\\boldsymbol\\theta^0$. Expanding the score equation of an interior maximizer about $\\boldsymbol\\theta^0$, I obtain $\\sqrt n(\\hat{\\boldsymbol\\theta}_{ML}-\\boldsymbol\\theta^0)=-A_n^{-1}n^{-1/2}\\sum_{i=1}^ns_i$, where $s_i$ is the score of observation $i$ at $\\boldsymbol\\theta^0$ and $A_n\\xrightarrow{p}-I$. The central limit theorem then gives the limit $N(0,I^{-1})$. For a regular, asymptotically linear estimator with influence function $\\psi$ such that $\\mathrm{E}[\\psi s^{T}]$ is the identity matrix, $\\mathrm{E}[\\psi\\psi^{T}]-I^{-1}=\\mathrm{E}[(\\psi-I^{-1}s)(\\psi-I^{-1}s)^{T}]$ is positive semidefinite, and the ML estimator attains this bound.", "problem": "``This estimator'' is the ML estimate of the full vector $\\boldsymbol\\theta=(p_1,\\boldsymbol\\theta_1,\\ldots,p_{M-1},\\boldsymbol\\theta_{M-1},\\boldsymbol\\theta_M)$ of the preceding sentence. Suppose that a proper subset of the component models characterizes the conditional distribution of $Y$. The distribution can then be written as the mixture with weight zero on every other component. If such a component $j$ has a parameter $\\boldsymbol\\theta_j$ that is free to vary, then every value of $\\boldsymbol\\theta_j$ gives the same density at $p_j=0$, the same likelihood for every sample and every $n$, and the same expected log-likelihood. The data cannot distinguish these parameter vectors, and three consequences follow. No estimator is consistent for $\\boldsymbol\\theta$ at all of them. The expected log-likelihood has no unique maximizer. The score in the direction of $\\boldsymbol\\theta_j$ is zero and the information matrix is singular, and the usual meaning of asymptotic efficiency does not apply to the full vector. The mixture of Example 5.2.11 has this form: at $p_N=0$ the density does not depend on $(\\mu_N,\\sigma_N)$, at $p_{LN}=0$ it does not depend on $(\\mu_{LN},\\sigma_{LN})$, and at $p_N+p_{LN}=1$ it does not depend on $\\lambda_E$. Positive weights are not enough either, since the mixture itself must be identified. The sentence thus promises consistency and efficiency in the case that it names, where both can fail.\n\nTheorem 5.2.6 requires a unique maximum (pp.~193--194), and the book labels each distribution ``distinctly'' with its parameter vector (p.~179). These conditions exclude an unused component with a free parameter; they do not make the sentence true for one. If the sentence is read as carrying every condition of Theorem 5.2.6 with it, then it is not false, but it says nothing about the case it names whenever an unused component has a free parameter. The text does not settle which reading was intended. The mathematics above does not depend on the reading.", "what_survives": "Theorem 5.2.6 with its conditions, the algebra of mixture densities, and the use of mixtures as approximations are unaffected, and no theorem, figure or reported number is shown to depend on recovering the parameters of a mixture. Sections 6.2.4 and 7.2.5 mention mixtures only as optional models for a propensity score, and the propensity models fitted in the applications are logit models. In the example above the fitted distribution remains consistent: every ML estimate has fitted probabilities $(1-K/n,\\,K/n,\\,0,\\,0)$ with $K=\\sum_{i=1}^nY_i$, and $\\mathrm{E}[K/n]=1/3$ and $\\mathrm{V}[K/n]=2/(9n)$. What changes is the promise for the parameter vector $\\boldsymbol\\theta$ itself, and any inference on mixture weights or component parameters that rests on it when a component may be unused."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the eight-line paragraph that begins 'As with the case of penalized regression, choosing lambda is not easy'. A red box marks the words 'minimum MKLD Predictor' in the fifth line, in the sentence 'As long as we assume that, as n tends to infinity, lambda does not grow too quickly, then the l1-penalized estimator will usually be consistent for the minimum MKLD Predictor (see Section 5.2.3).'", "caption": "Section 5.2.6, last paragraph, p. 202 (PDF p. 220). The marked term uses the abbreviation MKLD, which the book does not define.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C05-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "“Minimum MKLD Predictor” should read “minimum KL divergence predictor”, the term of Theorem 5.2.8.", "id": "P001-C05-N006", "location": "Section 5.2.6, last paragraph, p. 202 (PDF p. 220)", "pdf_page": 220, "printed_page": "202", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "As long as we assume that, as $n\\to\\infty$, $\\lambda$ does not grow too quickly, then the $\\ell^1$-penalized estimator will usually be consistent for the \\hl{minimum MKLD Predictor} (see Section 5.2.3).", "corrected": "As long as we assume that, as $n\\to\\infty$, $\\lambda$ does not grow too quickly, then the $\\ell^1$-penalized estimator will usually be consistent for the \\hl{minimum KL divergence predictor} (see Section 5.2.3).", "evidence": "The evidence is a check of the page and a search of the text of the book. The string ``MKLD'' occurs only on p.~202, and the glossary entry for KL is on p.~286.", "problem": "The abbreviation MKLD is not defined. It occurs once in the book, on this line, and the glossary of abbreviations lists KL and no MKLD. Section 5.2.3, to which the sentence refers, names the target the ``Minimum KL Divergence Predictor'' in the title of Theorem 5.2.8. If the M stands for ``minimum'', the word is also doubled.", "what_survives": "The cross-reference to Section 5.2.3 identifies the target, and nothing else changes. That target is the minimizer of the KL divergence averaged over the distribution of $\\mathbf X$, as corrected in \\fid{P001-C05-E001}. I examined the label only, not the consistency of the penalized estimator."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C05-N006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Three lines of text, a table and a closing line from Example 6.1.2. A red box marks the word 'lowest' in 'we replace these missing values with the lowest possible values'. In the table below, units 2 and 6 have R equal to 0 and Y set to 1, printed in magenta. The last line gives the estimated upper bound as five sixths.", "caption": "Example 6.1.2, p. 210 (PDF p. 228): the sentence that introduces the upper-bound table, with “lowest” marked, followed by the table and the bound.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T80", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T80"}], "headline": "On p. 210, the upper bound is described as assigning the lowest possible values to the missing outcomes, although it assigns the highest; the reported upper bound of $5/6$ is unchanged.", "id": "P001-C06-E001", "location": "Example 6.1.2 (Bounding the Expected Value), p. 210 (PDF p. 228), sentence between the second and third tables", "pdf_page": 228, "printed_page": "210", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T80"], "text": {"as_printed": "Then, using the sample mean as a plug-in estimator for the expected value, our estimated lower bound for $\\mathrm{E}[Y_i]$ is $\\frac{1}{2}$. Likewise, to estimate an upper bound, we replace these missing values with the \\hl{lowest} possible values:", "corrected": "Likewise, to estimate an upper bound, we replace these missing values with the \\hl{highest} possible values:", "evidence": "A recalculation from the tables of the example. The observed outcomes are $1,1,0,1$ for units 1, 3, 4 and 5, and the outcomes of units 2 and 6 are missing. Replacing the two missing outcomes by $0$, I obtain the sample mean $3/6=1/2$, and replacing them by $1$ gives $5/6$. The printed table and the printed bound $5/6$ use $1$.", "problem": "The sentence repeats the instruction for the lower bound, which is given a few lines earlier. For an upper bound, the missing outcomes are replaced by the highest possible value. The rule is stated correctly on p.~209 (``For an upper bound, assume that all missing values of $Y_i$ equal the highest possible value, $b$''), and the table that follows the sentence sets both missing outcomes to $1$.", "what_survives": "Both tables, the bounds $1/2$ and $5/6$, and the results on bounds that follow are unaffected. Only the word changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Four lines of text. One red box marks the membership sign between 'Supp[E[R i | X i]]' and '[0, 1]'. A second red box marks the words 'greater than zero or less than one' inside the closing parenthesis.", "caption": "Section 6.2.4, p. 225 (PDF p. 243): the last sentence on the page, with the membership sign and the phrase “greater than zero or less than one” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T82", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T82"}], "headline": "On p. 225, the thresholds that describe impossible predictions of a probability are reversed, and the support is written as an element of $[0,1]$ where it is a subset: impossible predictions are greater than one or less than zero, and $\\operatorname{Supp}[\\mathrm{E}[R_i|\\mathbf{X}_i]]\\subseteq[0,1]$.", "id": "P001-C06-E002", "location": "Section 6.2.4, p. 225 (PDF p. 243), first paragraph, last sentence on the page", "pdf_page": 243, "printed_page": "225", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T82"], "text": {"as_printed": "We know that $\\operatorname{Supp}[\\mathrm{E}[R_i|\\mathbf{X}_i]]\\hl{\\in}[0,1]$, and thus we might be concerned that a linear approximation would yield impossible propensity score estimates (predictions \\hl{greater than zero or less than one}).", "corrected": "We know that $\\operatorname{Supp}[\\mathrm{E}[R_i|\\mathbf{X}_i]]\\hl{\\subseteq}[0,1]$, and thus we might be concerned that a linear approximation would yield impossible propensity score estimates (predictions \\hl{greater than one or less than zero}).", "evidence": "A derivation. Since $R_i$ takes the values $0$ and $1$, I have $\\mathrm{E}[R_i|\\mathbf{X}_i]=\\Pr[R_i=1|\\mathbf{X}_i]$, which lies in $[0,1]$. Then the support of this random variable is a set contained in $[0,1]$. For example, if $R_i$ is independent of $\\mathbf{X}_i$ and $\\Pr[R_i=1]=1/2$, then $\\mathrm{E}[R_i|\\mathbf{X}_i]$ is the constant $1/2$ and its support is the set $\\{1/2\\}$, which is a subset of $[0,1]$ and not an element of it. For the thresholds, the negation of ``$z>0$ or $z<1$'' is ``$z\\le0$ and $z\\ge1$'', which no real number $z$ satisfies. The printed condition therefore holds for every real $z$, including the valid probability $z=1/2$.", "problem": "The sentence has two slips. First, since the support of a random variable is a set (Definition 1.2.20, p.~31), the support of $\\mathrm{E}[R_i|\\mathbf{X}_i]$ is a subset of $[0,1]$ and not an element of it. The chapter writes $\\operatorname{Supp}[Y_i]\\subseteq[a,b]$ on p.~209. Second, every real number is greater than zero or less than one, so the parenthesis as printed describes all predictions, including valid ones. The impossible values of a probability are those greater than one or less than zero. The parallel sentence on p.~264 has the thresholds in this order.", "what_survives": "The reason for using a binary choice model and the estimators that follow are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The statement of Theorem 6.2.6 in a shaded box: three lines of text and a displayed equation. A red box marks the symbol R i in 'Then, if R i is MAR conditional on X i'.", "caption": "Theorem 6.2.6, p. 226 (PDF p. 244): the statement of the theorem, with $R_i$ marked in the hypothesis.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T83", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T83"}], "headline": "In Theorem 6.2.6 on p. 226, the missing-at-random assumption is attributed to the response indicator $R_i$ instead of the outcome $Y_i$; the hypothesis should read “if $Y_i$ is MAR conditional on $\\mathbf{X}_i$”.", "id": "P001-C06-E003", "location": "Theorem 6.2.6 (Reweighting under MAR), p. 226 (PDF p. 244), hypothesis", "pdf_page": 244, "printed_page": "226", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T83"], "text": {"as_printed": "\\textbf{Theorem 6.2.6.} \\emph{Reweighting under MAR}\n\nLet $Y_i$ and $R_i$ be random variables with $\\operatorname{Supp}[R_i]=\\{0,1\\}$. Let $Y_i^*=Y_iR_i+(-99)(1-R_i)$ and let $\\mathbf{X}_i$ be a random vector. Then, if $\\hl{R_i}$ is MAR conditional on $\\mathbf{X}_i$,\n\\[\n\\mathrm{E}[Y_i]=\\mathrm{E}\\left[\\frac{Y_i^*R_i}{p_R(\\mathbf{X}_i)}\\right].\n\\]", "corrected": "\\textbf{Theorem 6.2.6.} \\emph{Reweighting under MAR}\n\nLet $Y_i$ and $R_i$ be random variables with $\\operatorname{Supp}[R_i]=\\{0,1\\}$. Let $Y_i^*=Y_iR_i+(-99)(1-R_i)$ and let $\\mathbf{X}_i$ be a random vector. Then, if $\\hl{Y_i}$ is MAR conditional on $\\mathbf{X}_i$,\n\\[\n\\mathrm{E}[Y_i]=\\mathrm{E}\\left[\\frac{Y_i^*R_i}{p_R(\\mathbf{X}_i)}\\right].\n\\]", "evidence": "A check of the pages (Definition 6.1.8, the theorem and its proof), and a derivation of the identity under the corrected hypothesis, which follows the printed proof. By the definition of $Y_i^*$, I have $Y_i^*R_i=Y_iR_i$. By the two conditions of Definition 6.1.8,\n\\[\n\\mathrm{E}\\left[\\frac{Y_i^*R_i}{p_R(\\mathbf{X}_i)}\\,\\Big|\\,\\mathbf{X}_i\\right]\n=\\frac{\\mathrm{E}[Y_iR_i|\\mathbf{X}_i]}{p_R(\\mathbf{X}_i)}\n=\\frac{\\mathrm{E}[Y_i|\\mathbf{X}_i]\\,\\mathrm{E}[R_i|\\mathbf{X}_i]}{p_R(\\mathbf{X}_i)}\n=\\mathrm{E}[Y_i|\\mathbf{X}_i].\n\\]\nIn the second equality, I used the conditional independence of $Y_i$ and $R_i$, and in the third, I used that $\\mathrm{E}[R_i|\\mathbf{X}_i]=\\Pr[R_i=1|\\mathbf{X}_i]=p_R(\\mathbf{X}_i)>0$. Then the Law of Iterated Expectations implies that $\\mathrm{E}[Y_i]=\\mathrm{E}[Y_i^*R_i/p_R(\\mathbf{X}_i)]$.", "problem": "The chapter defines MAR as a property of the outcome. Definition 6.1.8 (p.~215) says that $Y_i$ is missing at random conditional on $\\mathbf{X}_i$ if $Y_i$ is independent of $R_i$ conditional on $\\mathbf{X}_i$ and there exists $\\epsilon>0$ such that $\\Pr[R_i=1|\\mathbf{X}_i]>\\epsilon$. Theorems 6.1.9, 6.1.10 and 6.1.12 write ``if $Y_i$ is MAR conditional on $\\mathbf{X}_i$''. The phrase ``$R_i$ is MAR'' is not defined, and $R_i$ is observed for every unit. The proof on p.~227 uses the intended hypothesis: ``Then, since MAR holds, $Y_i\\perp\\!\\!\\!\\perp R_i|\\mathbf{X}_i$''.", "what_survives": "The theorem and its proof hold under the intended hypothesis, and nothing else in the chapter changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-E003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Seven lines of text. A red box marks the words 'logically equivalent' in the sentence 'In the case where there is a discrete and finite number of values for X i, the IPW estimator is logically equivalent to the plug-in estimator from Section 6.2.1 (Plug-In Estimation)'.", "caption": "Section 6.2.5, p. 228 (PDF p. 246): the paragraph that follows Definition 6.2.7, with “logically equivalent” marked in its last sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T84", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T84"}, {"id": "P001-T85", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T85"}], "headline": "On p. 228, a finite number of covariate values is treated as sufficient for the IPW estimator to equal the post-stratification estimator of Section 6.2.1; the two are equal when each estimated response probability is the observed fraction of respondents in its covariate cell.", "id": "P001-C06-E004", "location": "Section 6.2.5 (Weighting Estimators), p. 228 (PDF p. 246), first paragraph on the page, last sentence", "pdf_page": 246, "printed_page": "228", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T84", "P001-T85"], "text": {"as_printed": "In the case where there is a discrete and finite number of values for $\\mathbf{X}_i$, the IPW estimator is \\hl{logically equivalent} to the plug-in estimator from Section 6.2.1 (\\emph{Plug-In Estimation}).", "corrected": "\\hl{When the covariates have finite discrete support, every observed covariate cell has at least one respondent, and the estimated response probability in each cell is its observed respondent fraction,} the IPW estimator \\hl{equals} the plug-in estimator from Section 6.2.1 (\\emph{Plug-In Estimation}).\n\nIn symbols: let $\\mathbf{X}_i$ take finitely many values, and for each observed value $\\mathbf{x}$ let $n_{\\mathbf{x}}=\\sum_{i=1}^nI(\\mathbf{X}_i=\\mathbf{x})$ and $r_{\\mathbf{x}}=\\sum_{i=1}^nR_i\\,I(\\mathbf{X}_i=\\mathbf{x})$. If $r_{\\mathbf{x}}>0$ and $\\hat p_R(\\mathbf{x})=r_{\\mathbf{x}}/n_{\\mathbf{x}}$ for every observed $\\mathbf{x}$, then\n\\[\n\\frac{1}{n}\\sum_{i=1}^n\\frac{Y_i^*R_i}{\\hat p_R(\\mathbf{X}_i)}\n=\\sum_{\\mathbf{x}}\\frac{\\sum_{i=1}^nY_i^*R_i\\,I(\\mathbf{X}_i=\\mathbf{x})}{\\sum_{i=1}^nR_i\\,I(\\mathbf{X}_i=\\mathbf{x})}\\cdot\\frac{\\sum_{i=1}^nI(\\mathbf{X}_i=\\mathbf{x})}{n},\n\\]\nwhere the sum on the right is over the observed values of $\\mathbf{X}_i$. A maximum likelihood fit of a binary choice model that is not saturated in $\\mathbf{X}_i$, or known response probabilities, need not satisfy the condition on $\\hat p_R$.", "evidence": "A derivation and an exact counterexample. For each observed covariate value $\\mathbf{x}$, let $n_{\\mathbf{x}}$ be the number of sample units with $\\mathbf{X}_i=\\mathbf{x}$, let $r_{\\mathbf{x}}>0$ be the number of respondents among them, let $S_{\\mathbf{x}}=\\sum_{i=1}^nY_i^*R_i\\,I(\\mathbf{X}_i=\\mathbf{x})$, and let $q_{\\mathbf{x}}=\\hat p_R(\\mathbf{x})>0$. Grouping the sum in Definition 6.2.7 by cells and subtracting the post-stratification estimator of Section 6.2.1, I obtain\n\\[\n\\frac{1}{n}\\sum_{\\mathbf{x}}\\frac{S_{\\mathbf{x}}}{q_{\\mathbf{x}}}-\\frac{1}{n}\\sum_{\\mathbf{x}}\\frac{n_{\\mathbf{x}}S_{\\mathbf{x}}}{r_{\\mathbf{x}}}\n=\\frac{1}{n}\\sum_{\\mathbf{x}}S_{\\mathbf{x}}\\left(\\frac{1}{q_{\\mathbf{x}}}-\\frac{n_{\\mathbf{x}}}{r_{\\mathbf{x}}}\\right).\n\\]\nIf $q_{\\mathbf{x}}=r_{\\mathbf{x}}/n_{\\mathbf{x}}$ in every cell, the difference is zero for every vector of outcomes. Conversely, if the difference is zero for every vector of outcomes, with the covariates, the responses and the $q_{\\mathbf{x}}$ held fixed, then $q_{\\mathbf{x}}=r_{\\mathbf{x}}/n_{\\mathbf{x}}$ in every cell, since the outcomes can be chosen so that $S_{\\mathbf{x}}\\neq0$ in one cell only. For a particular vector of outcomes the two estimates can agree with other values of $q_{\\mathbf{x}}$.\n\nFor the counterexample, let $n=12$ and $\\mathbf{X}_i=(1,x_i)$ with $x_i\\in\\{-1,0,1\\}$.\n\\begin{audittable}{rrrlrrr}{A sample of twelve units in which the IPW and post-stratification estimates differ.}\n$x$ & $n_x$ & $r_x$ & Outcomes of respondents & $S_x$ & $S_x/r_x$ & $r_x/n_x$ \\\\\n$-1$ & $4$ & $2$ & $1/4$, $1/2$ & $3/4$ & $3/8$ & $1/2$ \\\\\n$0$ & $4$ & $3$ & $1/4$, $1/2$, $3/4$ & $3/2$ & $1/2$ & $3/4$ \\\\\n$1$ & $4$ & $2$ & $1/4$, $1/2$ & $3/4$ & $3/8$ & $1/2$\n\\end{audittable}\nThe logit model $\\Pr[R_i=1|x_i]=e^{a+bx_i}/(1+e^{a+bx_i})$ has a unique maximum likelihood fit in this sample, $(a,b)=(\\log(7/5),0)$, with estimated response probability $7/12$ in every cell. Both scores vanish there: the intercept score is $7-12\\cdot(7/12)=0$, and the slope score is $-(2-7/3)+(2-7/3)=0$. Since the log-likelihood is strictly concave, this stationary point is its only maximizer.\n\\begin{audittable}{llr}{The estimates of $\\mathrm{E}[Y_i]$ in this sample.}\nEstimator & Response probabilities & Estimate \\\\\nPost-stratification (Section 6.2.1) & none & $5/12$ \\\\\nIPW (Definition 6.2.7) & logit maximum likelihood fit, $7/12$ in every cell & $3/7$ \\\\\nIPW (Definition 6.2.7) & known probability $7/12$ & $3/7$ \\\\\nIPW (Definition 6.2.7) & observed fractions $1/2$, $3/4$, $1/2$ & $5/12$\n\\end{audittable}\nThe post-stratification estimate is $(3/8+1/2+3/8)/3=5/12$, and the IPW estimate with the logit fit is $(1/12)\\cdot3/(7/12)=3/7$. The two estimates differ by $1/84$. The stabilized IPW estimate is also $3/7$, because its denominator equals $1$. The sample has positive probability under i.i.d.\\ sampling from a population in which $x_i$ is uniform on $\\{-1,0,1\\}$, $Y_i$ is uniform on $\\{1/4,1/2,3/4\\}$, $R_i$ is Bernoulli with parameter $7/12$, and the three are independent. In this population MAR holds, the logit model is correctly specified, and $\\mathrm{E}[Y_i]=1/2$.", "problem": "The sentence follows the definition of the IPW estimator (Definition 6.2.7) and reads as a statement about two sample estimators: when $\\mathbf{X}_i$ takes finitely many values, the IPW estimator and the post-stratification estimator of Section 6.2.1 give the same estimate. This holds when $\\hat p_R(\\mathbf{x})$ is the observed fraction of respondents among the sample units with $\\mathbf{X}_i=\\mathbf{x}$, and it does not hold in general. Section 6.2.4 (p.~226) has just adopted ``an ML plug-in estimate from a binary choice model'' for the propensity score, whatever the exact specification, and the paragraph that contains the sentence mentions known response probabilities. With either choice the two estimates generally differ in a given sample. The two estimators share a population target: under MAR, both $\\mathrm{E}[Y_i^*R_i/p_R(\\mathbf{X}_i)]$ and $\\sum_{\\mathbf{x}}\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i=\\mathbf{x}]\\Pr[\\mathbf{X}_i=\\mathbf{x}]$ equal $\\mathrm{E}[Y_i]$.", "what_survives": "Theorem 6.2.6, the definitions of the two estimators and the double robustness result (Theorem 6.2.9) do not use the equality. Both estimators have the same target under MAR, and the counterexample does not show that a correctly specified IPW estimator is inconsistent. Section 6.3 uses a logit model for the propensity score and reports the IPW and stabilized IPW estimates separately; its numbers do not rely on the remark, and they were not recomputed, because the data and code were not available. The correction restricts the remark to one way of estimating the response probabilities. The parallel sentence in Section 7.2.6 (p.~266) is treated in \\fid{P001-C07-E005}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-E004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Eight lines of text. A red box marks the expression 'sum from i = 1 to n of 1 over p R of X i', which has no R i in its numerator. Four lines above, the text gives 'E[R i over p R of X i] = 1'.", "caption": "Paragraph after Definition 6.2.8, p. 228 (PDF p. 246): the explanation of the stabilized IPW estimator, with the sum over all units marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T86", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T86"}], "headline": "On p. 228, the prose that describes stabilization omits the response indicator that the displayed estimator includes: the sum should read $\\sum_{i=1}^nR_i/\\hat p_R(\\mathbf{X}_i)$, with $R_i=1$ for a respondent and $\\hat p_R(\\mathbf{X}_i)$ the fitted propensity score.", "id": "P001-C06-E005", "location": "Paragraph after Definition 6.2.8, p. 228 (PDF p. 246), sentence beginning “Because the stabilized IPW estimator accounts for”", "pdf_page": 246, "printed_page": "228", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T86"], "text": {"as_printed": "Because the stabilized IPW estimator accounts for this between-sample variation in $\\sum_{i=1}^{n}\\frac{\\hl{1}}{p_R(\\mathbf{X}_i)}$, it typically confers efficiency gains over the IPW estimator in practice.", "corrected": "Because the stabilized IPW estimator accounts for this between-sample variation in $\\sum_{i=1}^{n}\\frac{\\hl{R_i}}{\\hl{\\hat p_R}(\\mathbf{X}_i)}$, it typically confers efficiency gains over the IPW estimator in practice.", "evidence": "An exact counterexample. Let $n=2$, and let the true and the fitted propensity score equal $1/2$ for both units. The printed sum $\\sum_{i=1}^n1/p_R(\\mathbf{X}_i)$ equals $4$ for each of the four response patterns. The sum over respondents, $\\sum_{i=1}^nR_i/p_R(\\mathbf{X}_i)$, equals $0$, $2$, $2$ and $4$ for the patterns $(0,0)$, $(0,1)$, $(1,0)$ and $(1,1)$. Here the printed sum does not vary between samples, while the sum that normalizes the estimator does.", "problem": "Definition 6.2.8 divides by $n^{-1}\\sum_{i=1}^nR_i/\\hat p_R(\\mathbf{X}_i)$, the total weight of the respondents divided by $n$, and the text a few sentences earlier states that $\\mathrm{E}[R_i/p_R(\\mathbf{X}_i)]=1$. The sum named in the sentence runs over all $n$ units and has no $R_i$. This sum does not enter the estimator, and it differs from the sum over respondents even when the true propensity score is used.", "what_survives": "Definition 6.2.8 and the statement that its denominator converges in probability to $1$ are correct. The counterexample concerns the sum only, and it neither supports nor contradicts the remark that stabilization typically improves efficiency. The sum with the true propensity score, $\\sum_{i=1}^nR_i/p_R(\\mathbf{X}_i)$, is a smaller change to the sentence and is also correct. The parallel sentence in Section 7.2.6 (p.~267) is treated in \\fid{P001-C07-E006}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-E005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of nine lines of text. Red boxes mark the symbol R i in three conditional expectations: 'E[Y* i | R i, X i] is approximately beta 0 + X i beta', 'E[Y* i | R i, X i] is approximately beta 0 + X i beta + X [2]i beta 2', and 'Estimating E[Y* i | R i, X i] with OLS'. The paragraph reports the OLS (linear) estimate 8.997 and the OLS (polynomial) estimate 8.084.", "caption": "Section 6.3, paragraph below Table 6.3.1, p. 233 (PDF p. 251). The three marked symbols condition on the response indicator $R_i$ without fixing its value at one.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T87", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T87"}], "headline": "The two regressions of Section 6.3 approximate the CEF among respondents, $\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i]$; “$=1$” is missing after $R_i$ in three places.", "id": "P001-C06-E006", "location": "Section 6.3, paragraph below Table 6.3.1, p. 233 (PDF p. 251): two working formulas and one inline reference", "pdf_page": 251, "printed_page": "233", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T87"], "text": {"as_printed": "``As we might expect, the first regression uses OLS under the working assumption that the CEF of the observed data is linear, $\\mathrm{E}[Y_i^*|\\hl{R_i},\\mathbf{X}_i]\\approx\\beta_0+\\mathbf{X}_i\\boldsymbol{\\beta}$. The resulting estimate, denoted OLS (linear), is $\\hat{\\mathrm{E}}[Y_i]=8.997$, somewhat lower than the MCAR plug-in estimate. We also use a more flexible specification, with $\\mathrm{E}[Y_i^*|\\hl{R_i},\\mathbf{X}_i]\\approx\\beta_0+\\mathbf{X}_i\\boldsymbol{\\beta}+\\mathbf{X}_{[2]i}\\boldsymbol{\\beta}_2$, where $\\mathbf{X}_{[2]i}$ is a vector consisting of the squared values of each element $\\mathbf{X}_i$. Estimating $\\mathrm{E}[Y_i^*|\\hl{R_i},\\mathbf{X}_i]$ with OLS, we compute the OLS (polynomial) estimate $\\hat{\\mathrm{E}}[Y_i]=8.084$, again, lower than the MCAR-based estimate.''", "corrected": "``As we might expect, the first regression uses OLS under the working assumption that the CEF of the observed data is linear, $\\mathrm{E}[Y_i^*|R_i\\hl{=1},\\mathbf{X}_i]\\approx\\beta_0+\\mathbf{X}_i\\boldsymbol{\\beta}$. The resulting estimate, denoted OLS (linear), is $\\hat{\\mathrm{E}}[Y_i]=8.997$, somewhat lower than the MCAR plug-in estimate. We also use a more flexible specification, with $\\mathrm{E}[Y_i^*|R_i\\hl{=1},\\mathbf{X}_i]\\approx\\beta_0+\\mathbf{X}_i\\boldsymbol{\\beta}+\\mathbf{X}_{[2]i}\\boldsymbol{\\beta}_2$, where $\\mathbf{X}_{[2]i}$ is a vector consisting of the squared values of each element $\\mathbf{X}_i$. Estimating $\\mathrm{E}[Y_i^*|R_i\\hl{=1},\\mathbf{X}_i]$ with OLS, we compute the OLS (polynomial) estimate $\\hat{\\mathrm{E}}[Y_i]=8.084$, again, lower than the MCAR-based estimate.''\n\nHere $Y_i^*=Y_iR_i+(-99)(1-R_i)$ is the recorded outcome, $R_i$ is the response indicator and $\\mathbf{X}_i$ is the vector of covariates.", "evidence": "The evidence is a derivation from Definition 6.1.1 and a small example. On the event $R_i=1$, I have $Y_i^*=Y_i$, and on the event $R_i=0$, I have $Y_i^*=-99$. This implies that\n\\[\n\\mathrm{E}[Y_i^*|R_i,\\mathbf{X}_i]=R_i\\,\\mathrm{E}[Y_i|R_i=1,\\mathbf{X}_i]-99(1-R_i).\n\\]\nFor example, let $\\mathbf{X}_i$ be constant, let $Y_i=0$, and let $R_i$ be Bernoulli with parameter $1/2$. Then $\\mathrm{E}[Y_i^*|R_i,\\mathbf{X}_i]$ takes the values $-99$ and $0$, while $\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i]$ is the constant $0$. A formula in $\\mathbf{X}_i$ alone can represent the second and cannot represent the first.", "problem": "The three expressions condition on the random variable $R_i$, where the event $R_i=1$ is meant. By the stable outcomes model (Definition 6.1.1), I have $Y_i^*=Y_iR_i+(-99)(1-R_i)$. The conditional expectation given $R_i$ then equals $-99$ whenever $R_i=0$. It can depend on $R_i$ even after $\\mathbf{X}_i$ is fixed, and it is not the respondent CEF that the working formulas are intended to approximate. The regressions are fitted to the countries with observed outcomes and approximate the CEF among respondents, which is the object of Definition 6.2.4 (p.~223).", "what_survives": "This notation correction does not change the reported estimates 8.997 and 8.084. The text says that the regressions are used to impute missing responses, as in Section 6.2.2, and nothing indicates that the value $-99$ entered the fits. I did not have access to the data or code for the application to confirm this."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-E006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of three lines of text ending 'with E-hat[Y i] = 8.538' and footnote mark 15, followed by the four lines of footnote 15. Red boxes mark two clauses of the footnote: 'since the bias of the estimator may converge slowly' and 'but the bootstrap standard errors are likely to adequately characterize sampling variability'. Between them the footnote cites Abadie and Imbens 2008.", "caption": "Footnote 15 and the sentence that carries it, p. 233 (PDF p. 251). The first marked clause gives slowly converging bias as a possible reason for invalid bootstrap confidence intervals; the second says that the bootstrap standard errors are likely to be adequate.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E007.png"}, {"alt": "Crop of Table 6.3.1, Estimates of the Cross-National Average of Clean Energy Usage with Missing Data, with columns Estimate and Bootstrap SE and ten rows. A red box marks the row 'Hot Deck Imputation, 8.538, 1.204'. The other rows are Lower Bound Plug-In 7.110, 1.021; Upper Bound Plug-In 34.283, 3.128; MCAR Plug-In 9.762, 1.329; OLS (Linear) 8.997, 1.203; OLS (Polynomial) 8.084, 1.260; IPW 8.361, 1.111; Stabilized IPW 8.737, 1.154; Doubly Robust (Linear) 8.575, 1.151; Doubly Robust (Polynomial) 8.716, 1.187.", "caption": "Table 6.3.1, p. 233 (PDF p. 251). The marked row reports the hot deck estimate with a bootstrap standard error of 1.204.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-E007-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T13", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T13"}, {"id": "P001-T14", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T14"}], "headline": "Footnote 15 should not say that bootstrap standard errors are likely to be adequate for hot deck imputation: for one-to-one, nearest-neighbor hot deck imputation, the bootstrap variance can be inconsistent even when the estimator has no bias.", "id": "P001-C06-E007", "location": "Footnote 15, p. 233 (PDF p. 251), attached to the hot deck estimate 8.538; Table 6.3.1, row “Hot Deck Imputation”", "pdf_page": 251, "printed_page": "233", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T13", "P001-T14"], "text": {"as_printed": "Footnote 15: ``The bootstrap may not work as a method for computing valid confidence intervals for hot deck imputation \\hl{since the bias of the estimator may converge slowly} (see, e.g., Abadie and Imbens 2008), \\hl{but the bootstrap standard errors are likely to adequately characterize sampling variability}.''\n\nThe sentence that carries the footnote: ``Our estimate using one-to-one, nearest-neighbor hot deck imputation based on the propensity score largely agrees with that of the regression estimates, with $\\hat{\\mathrm{E}}[Y_i]=8.538$.''\n\nTable 6.3.1, row ``Hot Deck Imputation'': Estimate 8.538, Bootstrap SE 1.204.", "corrected": "Footnote 15: ``The \\hl{ordinary bootstrap can fail to estimate the sampling variability of one-to-one, nearest-neighbor hot deck imputation, even when the bias of the estimator is negligible} (see, e.g., Abadie and Imbens 2008). \\hl{The bootstrap standard error for hot deck imputation in Table 6.3.1 is reported descriptively; its validity for this implementation, which matches on a fitted propensity score, has not been established.}''\n\nThe correction supplies no replacement standard error. A valid one requires a variance argument for the imputation and resampling procedure that was used.", "evidence": "I checked the cited paper and derived an exact example. Abadie and Imbens (2008, \\emph{Econometrica} 76(6), 1537--1557) state on p.~1537 that ``the standard bootstrap fails to provide asymptotically valid standard errors'', and they prove this for a matching estimator with a scalar covariate and no bias.\n\nThe example is a missing data problem with a known propensity score. Let $P$ be uniform on $[1/4,3/4]$. Conditional on $P$, let $R$ be Bernoulli with parameter $P$. Let $\\varepsilon$ equal $+1$ or $-1$ with probability $1/2$ each, independently of $(P,R)$, and set $Y=50+\\varepsilon$. The covariate is $X=P$, so that $\\Pr[R=1|X]=X$. MAR and positivity hold. Given $n$ i.i.d.\\ draws, impute to each missing outcome the outcome of the respondent with the nearest value of $P$, and let $T_n$ be the mean of the $n$ completed outcomes (set $T_n=50$ if there is no respondent). Let $W_j$ be the number of times that the outcome of unit $j$ is used, with $W_j=0$ if $R_j=0$. Then\n\\[\nT_n-50=\\frac{1}{n}\\sum_{j=1}^{n}W_j\\varepsilon_j,\n\\]\nand the weights $W_j$ depend only on the values of $P$ and $R$. This implies that $\\mathrm{E}[T_n]=50$ for every $n$. Set $r=(1-p)/p$. A direct computation with the limiting Poisson process of respondents and nonrespondents near a respondent with $P=p$ gives $\\mathrm{E}[W_j^2\\,|\\,P_j=p,R_j=1]\\to 1+3r+(3/2)r^2$, and I obtain\n\\[\nn\\operatorname{Var}(T_n)\\longrightarrow V=\\int_{1/4}^{3/4}2p\\Bigl(1+3r+\\frac{3}{2}r^2\\Bigr)\\,dp=3\\log 3-\\frac{1}{4}\\approx 3.0458 .\n\\]\nThe bootstrap resamples $n$ units with replacement and repeats the imputation, using the same convention if there is no respondent. Let $v_n$ be $n$ times the bootstrap variance of the resulting estimate, given the data. Applying the same computation to the bootstrap weights, centered at the bootstrap mean, I find\n\\[\n\\mathrm{E}[v_n]\\longrightarrow V_b=\\int_{1/4}^{3/4}2p\\bigl(1+Ar+Br^2\\bigr)\\,dp\\approx 2.6471,\n\\]\nwhere $s=1-e^{-1}$, $A=2/s-s/(2-s)$ and $B=3/(2s)-(4-s)/(2(2-s))$. Since $e>8/3$, I have $A<3$ and $B<3/2$, and hence $V_b<V$. The ratio $V_b/V$ is about $0.869$. Since the fourth moments of the weights are bounded uniformly in $n$, the variables $v_n$ are uniformly integrable. If $v_n$ converged in probability to $V$, then $\\mathrm{E}[v_n]$ would converge to $V$, which is false. I conclude that the bootstrap variance is inconsistent, although $T_n$ is unbiased and root-$n$ consistent.\n\n\\begin{audittable}{lrrr}{The example: $n$ times the variance of the hot deck estimator $T_n$ and $n$ times the expected bootstrap variance. Monte Carlo standard errors in parentheses.}\n & True variance & Bootstrap variance & Difference \\\\\nLimit (derivation) & 3.0458 & 2.6471 & $-0.3988$ \\\\\nSimulation, $n=1500$ & 3.0534 (0.0064) & 2.6533 (0.0047) & $-0.4002$ (0.0022) \\\\\nSimulation, $n=1500$, second seed & 3.0457 (0.0061) & 2.6489 (0.0044) & $-0.3968$ (0.0022) \\\\\n\\end{audittable}\n\nEach simulation draws the values of $(P,R)$ for 600 independent samples of size $n=1500$, takes 300 bootstrap draws per sample and integrates over the outcomes exactly. The example matches on a known propensity score. The application matches on a score fitted by a logit model. The example does not show that the standard error 1.204 is wrong, and its variance ratio does not transfer to the application.", "problem": "The footnote states one reason why the bootstrap may fail for hot deck imputation, namely that the bias of the estimator may converge slowly. It then separates confidence intervals, which may be invalid, from standard errors, which it expects to be adequate. I find this methodological reassurance unsupported. The paper that the footnote cites, Abadie and Imbens (2008), shows that the standard errors of the standard bootstrap can be asymptotically invalid for matching with a fixed number of neighbors, even when the matching estimator has no asymptotic bias. What fails in their example is the bootstrap variance, despite the absence of bias. I obtain the same failure for the one-to-one, nearest-neighbor hot deck imputation described in Section 6.2.3: in the example below, the estimator is unbiased for every sample size, and the bootstrap variance is inconsistent. The standard error 1.204 in the hot deck row of Table 6.3.1 was computed by bootstrap. I have not shown that this number is wrong; the footnote does not provide a separate variance justification for it.", "what_survives": "My variance argument does not invalidate the point estimate 8.538 for hot deck imputation, the other nine point estimates in Table 6.3.1, the bounds, the identification results under MAR or the comparisons of point estimates. The caution of the footnote about confidence intervals also stands, and slow bias remains a possible reason for it. The bootstrap standard error 1.204 has not been shown to be calibrated; the precision and uncertainty comparisons on p.~234 involving the hot deck row therefore require a separate variance justification. Footnote 20 of Chapter 7 (p.~279) repeats the assurance about matching standard errors; see \\fid{P001-C07-E011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-E007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of text. A red box marks the italic title '(Missing Data with Binary Outcomes)' that follows 'from Example 6.1.2'.", "caption": "Paragraph after the proof of Theorem 6.1.4, p. 212 (PDF p. 230): the first of the four references to Example 6.1.2, with the quoted title marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N001.png"}, {"alt": "The heading and first two lines of Example 6.1.2. A red box marks the italic title 'Bounding the Expected Value' after the bold label 'Example 6.1.2.'", "caption": "Section 6.1.1, p. 209 (PDF p. 227): the heading of Example 6.1.2, with its title marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N001-p209.png"}], "formal_release": null, "formal_statements": [], "headline": "Four cross-references give Example 6.1.2 the title “Missing Data with Binary Outcomes”, although its title is “Bounding the Expected Value”, and two cross-references misquote the titles of Sections 6.2 and 5.1.2.", "id": "P001-C06-N001", "location": "Paragraph after the proof of Theorem 6.1.4, p. 212 (PDF p. 230); the same title on pp. 214, 220 and 222 (PDF pp. 232, 238 and 240); section titles on p. 215 (PDF p. 233) and p. 233 (PDF p. 251)", "pdf_page": 230, "printed_page": "212", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~212.} Note that these plug-in estimators will reproduce our bounds estimate of $[\\frac{1}{2},\\frac{5}{6}]$ from Example 6.1.2 (\\hl{\\emph{Missing Data with Binary Outcomes}}).\n\n\\emph{p.~214.} For instance, in Example 6.1.2 (\\hl{\\emph{Missing Data with Binary Outcomes}}), our estimate of $\\mathrm{E}[Y_i]$ assuming MCAR would be $\\frac{3}{4}$.\n\n\\emph{p.~220.} Consider again Example 6.1.2 (\\hl{\\emph{Missing Data with Binary Outcomes}}), but now assume that we also observe a binary covariate $X_i$ for every unit.\n\n\\emph{p.~222.} Once again, consider Example 6.1.2 (\\hl{\\emph{Missing Data with Binary Outcomes}}), but now assume that we observe two covariates for every unit:\n\n\\emph{p.~215.} But the utility of this way of thinking about estimating expectations with missing data will become clear in Section 6.2 (\\hl{\\emph{Estimation with Missing Data}}).\n\n\\emph{p.~233.} \\ldots\\ or a \\emph{logit} binary choice model (see Section 5.1.2, \\hl{\\emph{Binary Choice Model}}, for more details).", "corrected": "\\emph{pp.~212, 214, 220 and 222.} Example 6.1.2 (\\hl{\\emph{Bounding the Expected Value}})\n\n\\emph{p.~215.} \\ldots\\ will become clear in Section 6.2 (\\emph{Estimation with Missing Data \\hl{under MAR}}).\n\n\\emph{p.~233.} \\ldots\\ (see Section 5.1.2, \\emph{Binary Choice \\hl{Models}}, for more details).", "evidence": "A check of the pages, and a search of the text of the whole book for the phrase ``Missing Data with Binary Outcomes'', which finds it in the four references and nowhere else.", "problem": "The heading on p.~209 reads ``Example 6.1.2. Bounding the Expected Value''. No example in the book has the title ``Missing Data with Binary Outcomes'', and the phrase occurs only in these four references. The heading of Section 6.2 on p.~219 is ``Estimation with Missing Data under MAR'', and the heading of Section 5.1.2 on p.~182 is ``Binary Choice Models''.", "what_survives": "The number in each cross-reference is right, and each reference leads to the intended example or section. The two section titles differ from the headings only in the omitted words ``under MAR'' and in the singular ``Model'', and they identify the correct sections. Nothing else depends on the titles."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-N001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The heading of Section 6.1.4 and its first paragraph of six lines. A red box marks the words 'strong ignorability' in the sentence 'If strong ignorability holds, we need not condition on all the covariates'.", "caption": "Section 6.1.4, p. 217 (PDF p. 235): the opening paragraph, with “strong ignorability” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N002.png"}, {"alt": "Four lines of text. A red box marks the words 'potential outcomes' in 'the sample mean with imputed values for potential outcomes'.", "caption": "Example 6.2.3, p. 223 (PDF p. 241): the last paragraph of the example, with “potential outcomes” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N002-p223.png"}, {"alt": "Four lines of text. A red box marks the words 'strong ignorability' in 'Valid asymptotic inference for IPW estimators can be achieved under strong ignorability'.", "caption": "Paragraph after Definition 6.2.8, p. 228 (PDF p. 246): the sentence on inference for IPW estimators, with “strong ignorability” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N002-p228.png"}], "formal_release": null, "formal_statements": [], "headline": "Three sentences of Chapter 6 use terms that belong to Chapter 7: “strong ignorability” should read “MAR” on pp. 217 and 228, and “imputed values for potential outcomes” should read “imputed outcome values” on p. 223.", "id": "P001-C06-N002", "location": "Section 6.1.4, p. 217 (PDF p. 235), second sentence; also Example 6.2.3, p. 223 (PDF p. 241), last paragraph, and the paragraph after Definition 6.2.8, p. 228 (PDF p. 246)", "pdf_page": 235, "printed_page": "217", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~217.} If \\hl{strong ignorability} holds, we need not condition on all the covariates; instead, we can condition on a summary measure of response and the covariates.\n\n\\emph{p.~223.} In either case, the estimate of $\\mathrm{E}[Y_i]$ is then the sample mean with imputed values for \\hl{potential outcomes}.\n\n\\emph{p.~228.} Valid asymptotic inference for IPW estimators can be achieved under \\hl{strong ignorability}, proper specification of $\\hat p_R(\\mathbf{X}_i)$, and the usual plug-in regularity conditions via the bootstrap, re-estimating the propensity scores in each bootstrap sample.", "corrected": "\\emph{p.~217.} If \\hl{MAR} holds, we need not condition on all the covariates; instead, we can condition on a summary measure of response and the covariates.\n\n\\emph{p.~223.} In either case, the estimate of $\\mathrm{E}[Y_i]$ is then the sample mean with \\hl{imputed outcome values}.\n\n\\emph{p.~228.} Valid asymptotic inference for IPW estimators can be achieved under \\hl{MAR}, proper specification of $\\hat p_R(\\mathbf{X}_i)$, and the usual plug-in regularity conditions via the bootstrap, re-estimating the propensity scores in each bootstrap sample.", "evidence": "A check of the pages: the statement about terms on p.~215 and Definition 6.1.8, Theorem 6.1.12 and its proof on pp.~217--218, the two tables on p.~223, the sentence on p.~228, and Definition 7.1.12 on p.~247.", "problem": "The chapter says on p.~215 that it uses the term ``MAR'', and that ``strong ignorability'' is the name of the analogous assumption in Section 7.1.5. Strong ignorability is defined in Definition 7.1.12 (p.~247) for the potential outcomes $Y_i(0)$ and $Y_i(1)$ and a treatment indicator $D_i$, and Chapter 6 has neither. The result that the sentence on p.~217 introduces, Theorem 6.1.12, is stated under MAR. In Chapter 6 the outcome $Y_i$ is a single stable outcome, not a potential outcome. The two tables on p.~223 impute values of $Y_i$, for the units with missing outcomes in the first table and for all units in the second.", "what_survives": "The result on the propensity score under MAR and the two imputation procedures are unaffected. The sentence on p.~228 keeps its conditions on specification and regularity, and replacing the term does not by itself establish its claim about the bootstrap."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-N002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of main text, including the footnote mark 10, and below them the two lines of footnote 10. A red box marks the last sentence of the footnote, 'See Footnote 15.'", "caption": "Section 6.2.1, p. 219 (PDF p. 237): the text that calls footnote 10, and footnote 10 itself, with “See Footnote 15.” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N003.png"}, {"alt": "The four lines of footnote 15, which begins 'The bootstrap may not work as a method for computing valid confidence intervals for hot deck imputation'. A red box marks the footnote number 15.", "caption": "Section 6.3, p. 233 (PDF p. 251): footnote 15 of Chapter 6, with its number marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N003-p233.png"}], "formal_release": null, "formal_statements": [], "headline": "The pointer “See Footnote 15” at the end of footnote 10 leads to a footnote about the bootstrap, not about covariate cells without respondents, and should be deleted.", "id": "P001-C06-N003", "location": "Footnote 10, p. 219 (PDF p. 237), last sentence", "pdf_page": 237, "printed_page": "219", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Footnote 10.} This requires that we have at least one unit who responded with $\\mathbf{X}_i=\\mathbf{x}$ for every covariate profile $\\mathbf{x}\\in\\widehat{\\operatorname{Supp}}[\\mathbf{X}_i]$, otherwise we will have a zero in the denominator. \\hl{See Footnote 15.}", "corrected": "\\emph{Footnote 10.} This requires that we have at least one unit who responded with $\\mathbf{X}_i=\\mathbf{x}$ for every covariate profile $\\mathbf{x}\\in\\widehat{\\operatorname{Supp}}[\\mathbf{X}_i]$, otherwise we will have a zero in the denominator. \\del{See Footnote 15.}\n\nAlternatively, the pointer can be kept and redirected: \\hl{See Section 7.3 and its Footnote 15 for the analogous population positivity issue.}", "evidence": "A check of the pages: footnote 10 on p.~219, footnote 15 of Chapter 6 on p.~233, and footnotes 12 and 15 of Chapter 7 on pp.~256 and 271.", "problem": "Footnote 10 warns that a covariate cell without respondents puts a zero in the denominator of the post-stratification estimator. Footnote 15 of the chapter (p.~233) concerns the bootstrap for hot deck imputation and says nothing about such cells. Chapter 7 has a footnote on a related question: footnote 15 on p.~271, in Section 7.3, reads ``In the analogous problem of missing data, we might worry that $\\exists\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}_i]$ such that $p_R(\\mathbf{x})=0$. Our results in this section apply analogously to Chapter 6.'' Footnote 12 of Chapter 7 (p.~256), which is parallel to footnote 10, also ends ``See Footnote 15''. The pointer may have been carried over from there, but the pages do not establish this. The footnote in Chapter 7 concerns positivity in the population, which is not the same as an empty cell in a finite sample.", "what_survives": "The warning about empty cells and the post-stratification estimator are correct. The displayed estimator has the number of respondents in the cell in its denominator, which makes the warning clear without the pointer."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-N003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Seven lines of text. A red box marks the term p R of X 1 at the end of the line 'But p R of X 4 = 0.35 is almost as close to p R of X 1'. The preceding text gives p R of X 1 = 0.33 and p R of X 2 = 0.14.", "caption": "Paragraph after Example 6.2.5, p. 225 (PDF p. 243): the argument that unit 4 is nearly as good a donor for unit 2 as unit 1, with $p_R(\\mathbf{X}_1)$ marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N004.png"}, {"alt": "Three lines of text. A red box marks the words 'can viewed' in 'causal inference can viewed as a missing data problem'.", "caption": "Chapter 6, introduction, p. 208 (PDF p. 226), with “can viewed” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N004-p208.png"}, {"alt": "Five lines of text. A red box marks the words 'note what' in 'Furthermore, note what the second line of the above proof implies that'.", "caption": "Paragraph after the proof of Theorem 6.1.9, p. 216 (PDF p. 234), with “note what” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N004-p216.png"}, {"alt": "Four lines of text. Two red boxes mark the word 'as' at the end of one line and the words 'the some of the' at the start of the next, in 'as the some of the plug-in estimators developed in the next section'.", "caption": "End of Section 6.1.4, p. 219 (PDF p. 237), with “as the some of the” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N004-p219.png"}, {"alt": "Four lines of text. A red box marks the words 'units with closest' in 'the observed units with closest propensity score is unit 3'. Two lines above, the parallel sentence reads 'The observed unit with the closest propensity score is unit 1'.", "caption": "Example 6.2.5, p. 225 (PDF p. 243), with “units with closest” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N004-p225.png"}, {"alt": "Three lines of text. A red box marks the words 'have can' in 'we have can use a ML estimator to obtain propensity score estimates'.", "caption": "Section 6.2.4, p. 226 (PDF p. 244), with “have can” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N004-p226.png"}, {"alt": "Four lines of text. A red box marks the word 'unverifable' in 'And MAR remains unverifable, with no clear theoretical heuristic'.", "caption": "Section 6.3, last paragraph, p. 234 (PDF p. 252), with “unverifable” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N004-p234.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T81", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T81"}], "headline": "In the discussion of hot deck imputation, $p_R(\\mathbf{X}_4)=0.35$ is almost as close to $p_R(\\mathbf{X}_2)$, not to $p_R(\\mathbf{X}_1)$; six misprints in the prose of the chapter are also corrected.", "id": "P001-C06-N004", "location": "Paragraph after Example 6.2.5, p. 225 (PDF p. 243), sentence beginning “But $p_R(\\mathbf{X}_4)=0.35$”; six misprints in running prose on pp. 208, 216, 219, 225, 226 and 234 (PDF pp. 226, 234, 237, 243, 244 and 252)", "pdf_page": 243, "printed_page": "225", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T81"], "text": {"as_printed": "\\emph{p.~225.} In the above example, we used $Y_1=2$ to impute $Y_2$ because $p_R(\\mathbf{X}_1)=0.33$ is closest to $p_R(\\mathbf{X}_2)=0.14$. But $p_R(\\mathbf{X}_4)=0.35$ is almost as close to $p_R(\\mathbf{X}_{\\hl{1}})$, so $Y_4=10$ probably also gives us some information about $\\mathrm{E}[Y_i|p_R(\\mathbf{X}_i)=0.14]$.\n\n\\emph{p.~208.} Rather, we will show that causal inference \\hl{can viewed} as a missing data problem (see further discussion in Section 7.1.2).\n\n\\emph{p.~216.} Furthermore, \\hl{note what} the second line of the above proof implies that the conditional expectation function (CEF) of $Y_i$ given $\\mathbf{X}_i$ is point identified under MAR.\n\n\\emph{p.~219.} We consider this in Section 6.2.4, \\hl{as the some of the} plug-in estimators developed in the next section rely on having estimates that well approximate the true propensity score.\n\n\\emph{p.~225.} Likewise, unit 6 has the propensity score $p_R(\\mathbf{X}_6)=0.70$, and the observed \\hl{units with closest} propensity score is unit 3 with $p_R(\\mathbf{X}_3)=0.73$, so we use $Y_3=3$ to impute $Y_6$.\n\n\\emph{p.~226.} If this model is true, then following Section 5.2.4, we \\hl{have can} use \\hl{a} ML estimator to obtain propensity score estimates: $\\hat p_R(\\mathbf{X}_i)=h(\\mathbf{X}_i\\hat{\\boldsymbol{\\beta}}_{ML})$.\n\n\\emph{p.~234.} And MAR remains \\hl{unverifable}, with no clear theoretical heuristic for ascertaining its validity in this setting.", "corrected": "\\emph{p.~225.} But $p_R(\\mathbf{X}_4)=0.35$ is almost as close to $p_R(\\mathbf{X}_{\\hl{2}})$, so $Y_4=10$ probably also gives us some information about $\\mathrm{E}[Y_i|p_R(\\mathbf{X}_i)=0.14]$.\n\n\\emph{p.~208.} Rather, we will show that causal inference \\hl{can be viewed} as a missing data problem (see further discussion in Section 7.1.2).\n\n\\emph{p.~216.} Furthermore, \\hl{note that} the second line of the above proof implies that the conditional expectation function (CEF) of $Y_i$ given $\\mathbf{X}_i$ is point identified under MAR.\n\n\\emph{p.~219.} We consider this in Section 6.2.4, \\hl{as some of the} plug-in estimators developed in the next section rely on having estimates that well approximate the true propensity score.\n\n\\emph{p.~225.} Likewise, unit 6 has the propensity score $p_R(\\mathbf{X}_6)=0.70$, and the observed \\hl{unit with the closest} propensity score is unit 3 with $p_R(\\mathbf{X}_3)=0.73$, so we use $Y_3=3$ to impute $Y_6$.\n\n\\emph{p.~226.} If this model is true, then following Section 5.2.4, we \\hl{can} use \\hl{an} ML estimator to obtain propensity score estimates: $\\hat p_R(\\mathbf{X}_i)=h(\\mathbf{X}_i\\hat{\\boldsymbol{\\beta}}_{ML})$.\n\n\\emph{p.~234.} And MAR remains \\hl{unverifiable}, with no clear theoretical heuristic for ascertaining its validity in this setting.", "evidence": "A recalculation from the table of Example 6.2.5, whose propensity scores for units 1 to 6 are $0.33$, $0.14$, $0.73$, $0.35$, $0.78$ and $0.70$, with the outcomes of units 2 and 6 missing. The distances to unit 2 are $|0.33-0.14|=0.19$ for unit 1 and $|0.35-0.14|=0.21$ for unit 4, and the argument uses that these two distances are nearly equal. The distance from unit 4 to unit 1 is $|0.35-0.33|=0.02$. The nearest unit with an observed outcome is unit 1 for unit 2 and unit 3 for unit 6, as printed. The six misprints were checked on the pages.", "problem": "The outcome that is imputed is that of unit 2, whose propensity score is $0.14$. Unit 1 is the donor because its score, $0.33$, is the closest among the units with observed outcomes. The sentence then argues that unit 4, with score $0.35$, is nearly as good a donor for unit 2. The comparison that supports this is between the distances of units 1 and 4 from unit 2, and the last term of the sentence should be $p_R(\\mathbf{X}_2)$. As printed, the sentence names unit 1 as the target of the comparison. Unit 4 is close to unit 1 as well, but the argument needs its distance from unit 2. The other six items are slips of grammar or spelling.", "what_survives": "Both donor assignments, the imputed outcomes, the estimate $32/6=16/3\\approx5.33$, and the argument that a second donor carries information about the missing outcome are unaffected. No number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-N004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The first lines of the statement of Theorem 6.2.9 in a shaded box, down to condition (i). A red box marks the expression 'E tilde of Y i given R i = 1, X i = x', in which Y i has no star. In condition (i), below, the same expression is printed with Y i star.", "caption": "Theorem 6.2.9, p. 229 (PDF p. 247): the opening of the statement, with the label $\\tilde{\\mathrm{E}}[Y_i|R_i=1,\\mathbf{X}_i=\\mathbf{x}]$ marked; condition (i), below, writes $Y_i^*$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N005.png"}, {"alt": "The statement of Theorem 6.1.10 in a shaded box: three lines of text and a displayed equation. A red box marks an italic X i in 'conditional on X i'. Every other X i in the statement is bold.", "caption": "Theorem 6.1.10, p. 217 (PDF p. 235): the statement of the theorem, with the italic $X_i$ marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N005-p217.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T113", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T113"}], "headline": "Theorem 6.2.9 introduces the approximation of the CEF as $\\tilde{\\mathrm{E}}[Y_i|R_i=1,\\mathbf{X}_i=\\mathbf{x}]$ and then writes it as $\\tilde{\\mathrm{E}}[Y_i^*|R_i=1,\\mathbf{X}_i=\\mathbf{x}]$, and the hypothesis of Theorem 6.1.10 prints $X_i$ for $\\mathbf{X}_i$; one label and the bold symbol should be used throughout.", "id": "P001-C06-N005", "location": "Theorem 6.2.9, p. 229 (PDF p. 247), third sentence of the statement; also Theorem 6.1.10, p. 217 (PDF p. 235), hypothesis", "pdf_page": 247, "printed_page": "229", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T113"], "text": {"as_printed": "\\emph{Theorem 6.2.9, p.~229.} Let $Y_i$ and $R_i$ be random variables with $\\operatorname{Supp}[R_i]=\\{0,1\\}$. Let $Y_i^*=Y_iR_i+(-99)(1-R_i)$ and let $\\mathbf{X}_i$ be a random vector. Let $\\tilde{\\mathrm{E}}[\\hl{Y_i}|R_i=1,\\mathbf{X}_i=\\mathbf{x}]$ be an approximation of the CEF evaluated at $R_i=1$ and $\\mathbf{X}_i=\\mathbf{x}$, and let $\\tilde p_R(\\mathbf{X}_i)$ be an approximation of the propensity score. If MAR holds and if either (i) $\\tilde{\\mathrm{E}}[Y_i^*|R_i=1,\\mathbf{X}_i=\\mathbf{x}]=\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i=\\mathbf{x}]$ $\\forall\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}_i]$, and $\\exists\\epsilon>0$ such that $\\epsilon<\\tilde p_R(\\mathbf{x})<1-\\epsilon$, or \\ldots\n\n\\emph{Theorem 6.1.10, p.~217.} Let $Y_i$ and $R_i$ be random variables with $\\operatorname{Supp}[R_i]=\\{0,1\\}$, let $Y_i^*=Y_iR_i+(-99)(1-R_i)$, and let $\\mathbf{X}_i$ be a random vector. Then, if $Y_i$ is MAR conditional on $\\hl{X_i}$,\n\\[\n\\mathrm{E}[Y_i|\\mathbf{X}_i]=\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i].\n\\]", "corrected": "\\emph{Theorem 6.2.9, p.~229.} Let $Y_i$ and $R_i$ be random variables with $\\operatorname{Supp}[R_i]=\\{0,1\\}$. Let $Y_i^*=Y_iR_i+(-99)(1-R_i)$ and let $\\mathbf{X}_i$ be a random vector. Let $\\tilde{\\mathrm{E}}[\\hl{Y_i^*}|R_i=1,\\mathbf{X}_i=\\mathbf{x}]$ be an approximation of the CEF evaluated at $R_i=1$ and $\\mathbf{X}_i=\\mathbf{x}$, and let $\\tilde p_R(\\mathbf{X}_i)$ be an approximation of the propensity score. If MAR holds and if either (i) $\\tilde{\\mathrm{E}}[Y_i^*|R_i=1,\\mathbf{X}_i=\\mathbf{x}]=\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i=\\mathbf{x}]$, $\\forall\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}_i]$, and $\\exists\\epsilon>0$ such that $\\epsilon<\\tilde p_R(\\mathbf{x})<1-\\epsilon$, or (ii) $\\tilde p_R(\\mathbf{x})=p_R(\\mathbf{x})$, $\\forall\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}_i]$, then\n\\[\n\\mathrm{E}[Y_i]=\\mathrm{E}\\left[\\tilde{\\mathrm{E}}[Y_i^*|R_i=1,\\mathbf{X}_i]+\\frac{R_i\\left(Y_i^*-\\tilde{\\mathrm{E}}[Y_i^*|R_i=1,\\mathbf{X}_i]\\right)}{\\tilde p_R(\\mathbf{X}_i)}\\right].\n\\]\n\n\\emph{Theorem 6.1.10, p.~217.} Let $Y_i$ and $R_i$ be random variables with $\\operatorname{Supp}[R_i]=\\{0,1\\}$, let $Y_i^*=Y_iR_i+(-99)(1-R_i)$, and let $\\mathbf{X}_i$ be a random vector. Then, if $Y_i$ is MAR conditional on $\\hl{\\mathbf{X}_i}$,\n\\[\n\\mathrm{E}[Y_i|\\mathbf{X}_i]=\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i].\n\\]", "evidence": "A check of the pages, and a derivation. By Definition 6.1.1, I have $Y_i^*=Y_iR_i+(-99)(1-R_i)$, which gives $Y_i^*R_i=Y_iR_i$ for $R_i\\in\\{0,1\\}$ and $Y_i^*=Y_i$ when $R_i=1$. This implies that $\\mathrm{E}[Y_i^*|R_i=1,\\mathbf{X}_i]=\\mathrm{E}[Y_i|R_i=1,\\mathbf{X}_i]$, and an approximation of one is an approximation of the other.", "problem": "The statement of Theorem 6.2.9 uses two labels for one function. It introduces the approximation with $Y_i$ and uses $Y_i^*$ in condition (i) and in the displayed identity, without saying that the two labels denote the same object. The exact CEFs are equal because $Y_i^*=Y_i$ when $R_i=1$; one label makes clear that a single approximation is intended. In Theorem 6.1.10 the vector of covariates is bold everywhere in the statement except once in the hypothesis, where it is italic.", "what_survives": "Every printed expression equals the intended one, and both theorems, their proofs and the estimators are unchanged. The expressions written with $Y_i$ in Definition 6.2.8, in the last lines of the proof of Theorem 6.2.6 and in the text on pp.~230--231 equal those written with $Y_i^*$ and need no correction."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading '6.3 Application: Estimating the Cross-National Average of Clean Energy Use' and three lines of text. A red box marks the words 'Quality of Governance' in the sentence 'We now consider an application, again using the Quality of Governance data (Teorell et al. 2016).'", "caption": "Section 6.3, heading and first sentence, p. 231 (PDF p. 249). The marked words name the data source.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N006.png"}, {"alt": "Crop of the last four lines of footnote 14. A red box marks the words 'Quality of Governance' in the sentence 'For further details, please refer to the Quality of Governance Standard Data codebook (Teorell et al. 2016).'", "caption": "Footnote 14, last four lines, p. 232 (PDF p. 250). The marked words repeat the name.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The data are the Quality of Government dataset of Teorell et al. (2016); “Quality of Governance” is printed in both places.", "id": "P001-C06-N006", "location": "Section 6.3, first sentence, p. 231 (PDF p. 249); footnote 14, last sentence, p. 232 (PDF p. 250)", "pdf_page": 249, "printed_page": "231", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~231: ``We now consider an application, again using the \\hl{Quality of Governance} data (Teorell et al.\\ 2016).''\n\nP.~232, footnote 14: ``For further details, please refer to the \\hl{Quality of Governance} Standard Data codebook (Teorell et al.\\ 2016).''", "corrected": "P.~231: ``We now consider an application, again using the \\hl{Quality of Government} data (Teorell et al.\\ 2016).''\n\nP.~232, footnote 14: ``For further details, please refer to the \\hl{Quality of Government} Standard Data codebook (Teorell et al.\\ 2016).''", "evidence": "The evidence is a check of pp.~231 and 232 against the bibliography entry on p.~292 (PDF p.~310).", "problem": "The bibliography (p.~292) lists the source as ``The Quality of Government standard dataset, version Jan 16'', from the Quality of Government Institute. Section 6.3 calls it ``Quality of Governance'' twice. The citation identifies the dataset. Only the name is wrong.", "what_survives": "Nothing depends on the name. The same name is printed in Section 4.4 (p.~170), which is the subject of \\fid{P001-C04-N012}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-N006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text. A red box marks the word 'production' in the sentence 'Alternative and nuclear energy use is measured as the percentage of total energy production in a country generated from non-carbon dioxide emitting sources, including, but not limited to, nuclear, wind, hydroelectric, geothermal, and solar power.'", "caption": "Section 6.3, first paragraph, p. 231 (PDF p. 249). The marked word gives the denominator of the outcome variable as total energy production.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N007.png"}, {"alt": "Crop of four lines of footnote 14. Red boxes mark the words 'the number of individuals who have used the Internet in the last 12 months' in the clause 'Internet use is measured as the number of individuals who have used the Internet in the last 12 months;'. The clauses before and after it describe GDP per capita and particulate matter.", "caption": "Footnote 14, p. 232 (PDF p. 250). The marked words describe Internet use as a number of individuals, with no denominator.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C06-N007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "If the application uses the indicators described in the Quality of Government Basic codebook of January 2016, the energy denominator should be total energy use and Internet users should be measured per 100 people.", "id": "P001-C06-N007", "location": "Section 6.3, first paragraph, p. 231 (PDF p. 249); footnote 14, the clause on Internet use, p. 232 (PDF p. 250)", "pdf_page": 249, "printed_page": "231", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~231: ``Alternative and nuclear energy use is measured as the percentage of total energy \\hl{production} in a country generated from non--carbon dioxide emitting sources, including, but not limited to, nuclear, wind, hydroelectric, geothermal, and solar power.''\n\nP.~232, footnote 14: ``Internet use is measured as \\hl{the number of individuals who have used the Internet in the last 12 months};''", "corrected": "If the outcome is \\texttt{wdi\\_altnucen} and the covariate is \\texttt{wdi\\_internetuse}, the two passages read as follows.\n\nP.~231: ``Alternative and nuclear energy use is measured as the percentage of total energy \\hl{use} in a country generated from non--carbon dioxide emitting sources, including, but not limited to, nuclear, wind, hydroelectric, geothermal, and solar power.''\n\nP.~232, footnote 14: ``Internet use is measured as the number of individuals who have used the Internet in the last 12 months\\hl{, per 100 people};''", "evidence": "The evidence is a check of pp.~231 and 232 against a public source. The codebook of the Quality of Government Basic Dataset, version Jan16, lists \\texttt{wdi\\_altnucen} as ``Alternative and nuclear energy (\\% of total energy use)'' and \\texttt{wdi\\_internetuse} as ``Internet users (per 100 people)'' (\\url{https://www.qogdata.pol.gu.se/dataarchive/qog_bas_jan16.pdf}). The World Bank metadata consulted gives the unit of the energy indicator as ``\\% (share) of energy use''. The book cites the Standard dataset of the same release. I did not read the codebook of the Standard dataset or the data. It is not confirmed that these two indicators are the variables used in the application.", "problem": "On p.~231 the outcome is a percentage of total energy production, and footnote 14 describes Internet use as a number of individuals, with no denominator. The codebook of the Quality of Government data of January 2016 carries two indicators of the World Bank that fit these descriptions, with other units: alternative and nuclear energy is a percentage of total energy \\emph{use}, and Internet use is the number of users \\emph{per 100 people}. The book does not name the variables that it analyzes. If these two indicators were used without transformation, correcting their descriptions would not change the estimates.", "what_survives": "I did not recompute the ten estimates of Table 6.3.1 or their standard errors, and I establish no numerical error in them. Whether the application used these two indicators without transformation remains to be confirmed from the data."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C06-N007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of two lines of text and the shaded box of Definition 7.1.19, Average Marginal Causal Effect. The text ends 'If we assume that Y i (d) is differentiable with respect to d, then the following definition allows us to derive a concise measure.' The formula reads: AMCE = E of the fraction with numerator 'partial E[Y i (D i) | D i, X i]' and denominator 'partial D i'. Red boxes mark the D i inside Y i (D i), the D i after the conditioning bar, and the D i in the denominator.", "caption": "Definition 7.1.19, p. 255 (PDF p. 273). The three marked symbols are the same $D_i$: the argument of the potential outcome, the conditioning treatment and the variable of differentiation.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-A001.png"}, {"alt": "Crop of four lines of text, a displayed formula and two more lines. Red boxes mark the words 'under the conditional independence assumption, the CEF is causal, hence'. The formula reads: AMCE = APD sub D i = E of the fraction with numerator 'partial E[Y i | D i, X i]' and denominator 'partial D i'. The text after it reads 'In other words, under strong ignorability, the average partial derivative with respect to D i is the average marginal causal effect of D i.'", "caption": "P. 255 (PDF p. 273), the paragraph after Definition 7.1.19. The marked words state the condition under which the display equates the AMCE with the average partial derivative.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-A001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In Definition 7.1.19 the derivative is taken in the treatment level $d$ of the potential outcome $Y_i(d)$, with the conditioning values held fixed, and is then evaluated at $d=D_i$; the printed formula writes $D_i$ for both.", "id": "P001-C07-A001", "location": "Definition 7.1.19, p. 255 (PDF p. 273), and the display $AMCE=APD_{D_i}$ below it", "pdf_page": 273, "printed_page": "255", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Definition 7.1.19. \\emph{Average Marginal Causal Effect}. The \\emph{average marginal causal effect} of $D_i$ is\n\\[\nAMCE=\\mathrm{E}\\left[\\frac{\\partial\\mathrm{E}[Y_i(\\hl{D_i})|D_i,\\mathbf{X}_i]}{\\partial \\hl{D_i}}\\right].\\text{''}\n\\]\n\n``Furthermore, under the conditional independence assumption, the CEF is causal, hence\n\\[\nAMCE=APD_{D_i}=\\mathrm{E}\\left[\\frac{\\partial\\mathrm{E}[Y_i|D_i,\\mathbf{X}_i]}{\\partial D_i}\\right].\n\\]\nIn other words, under strong ignorability, the average partial derivative with respect to $D_i$ is the average marginal causal effect of $D_i$.''", "corrected": "``Definition 7.1.19. \\emph{Average Marginal Causal Effect}. The \\emph{average marginal causal effect} of $D_i$ is\n\\[\nAMCE=\\mathrm{E}\\left[\\left.\\frac{\\partial\\mathrm{E}[Y_i(\\hl{d})|D_i,\\mathbf{X}_i]}{\\partial \\hl{d}}\\right|_{\\hl{d=D_i}}\\right],\n\\]\n\\hl{where the conditioning values $D_i$ and $\\mathbf{X}_i$ are held fixed during differentiation, whenever this derivative and its expectation exist}.''\n\n``Furthermore, under \\hl{generalized strong ignorability and the requisite positivity, with differentiable conditional mean functions}, the CEF is causal, hence\n\\[\nAMCE=APD_{D_i}=\\mathrm{E}\\left[\\frac{\\partial\\mathrm{E}[Y_i|D_i,\\mathbf{X}_i]}{\\partial D_i}\\right].\\text{''}\n\\]", "intended_meaning": "The sentence before the definition assumes that $Y_i(d)$ is differentiable with respect to $d$, and the definition is meant to differentiate in that argument. Set $M(d,t,\\mathbf{x})=\\mathrm{E}[Y_i(d)|D_i=t,\\mathbf{X}_i=\\mathbf{x}]$. The intended quantity is the expectation of $\\partial M/\\partial d$ at $d=t=D_i$ and $\\mathbf{x}=\\mathbf{X}_i$: the average change in the expected potential outcome, for units at their realized treatment levels, when the treatment level alone is varied. With this meaning the statement below the definition is correct. Suppose that generalized strong ignorability holds, $(Y_i(d))_{d\\in\\operatorname{Supp}[D_i]}\\perp\\!\\!\\!\\perp D_i\\,|\\,\\mathbf{X}_i$, with the requisite positivity. Then $M(d,t,\\mathbf{x})=\\mathrm{E}[Y_i(d)|\\mathbf{X}_i=\\mathbf{x}]$ does not depend on $t$; write $g(d,\\mathbf{x})$ for it. Since $Y_i=Y_i(D_i)$ by Definition 7.1.18, I have $\\mathrm{E}[Y_i|D_i=t,\\mathbf{X}_i=\\mathbf{x}]=M(t,t,\\mathbf{x})=g(t,\\mathbf{x})$. The derivative of $M$ in $d$ at $d=t$ and the derivative of the CEF in $t$ are then both $\\partial g/\\partial t$. Averaging over $(D_i,\\mathbf{X}_i)$, I obtain $AMCE=APD_{D_i}$, provided that the conditional means are differentiable and the expectations are finite.\n\nRead literally, with the convention of Definition 4.3.6, the printed formula differentiates $M(t,t,\\mathbf{x})$ along the diagonal. Since $Y_i(D_i)=Y_i$, this is the average partial derivative of the CEF, with or without ignorability, and it need not be causal. For example, let there be no covariates, let $U\\in\\{0,1\\}$ and $D_i\\in[0,1]$ have joint density $3/4-t/2$ at $(0,t)$ and $1/4+t/2$ at $(1,t)$, and let $Y_i(d)=U$ for every $d$. Every causal effect is zero, while $\\mathrm{E}[Y_i|D_i=t]=1/4+t/2$, and the literal formula has the value $1/2$. Note that this example violates ignorability and does not contradict the identification statement. The AMCE appears only on p.~255 and in the index, and no later result uses the formula."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-A001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of a displayed formula and three lines of text. The formula reads: tau i L bar = (1/n) times the sum over i from 1 to n of (Y i D i + a(1 - D i) - Y i (1 - D i) - b D i). A red box marks the symbol Y i L under a bar in the sentence 'By Theorem 3.2.3 (The Expected Value of the Sample Mean is the Population Mean) and the Weak Law of Large Numbers (WLLN), Y i L bar is unbiased and consistent for E[tau i L], and by the Law of Iterated Expectations,'.", "caption": "Proof of Theorem 7.1.8, p. 243 (PDF p. 261). The display defines the sample mean $\\overline{\\tau_i^L}$; the marked symbol in the next sentence is $\\overline{Y_i^L}$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E001.png"}, {"alt": "Crop of one line of text and a displayed formula. The text reads 'Similarly, let tau i U = Y i D i + b(1 - D i) - Y i (1 - D i) - a D i, so that'. The formula reads: tau i U bar = (1/n) times the sum over i from 1 to n of (Y i D i + b(1 - D i) - Y i (x1 - D i) - a D i). A red box marks the letter x.", "caption": "Proof of Theorem 7.1.8, p. 244 (PDF p. 262). The line above the display defines $\\tau_i^U$ with the factor $(1-D_i)$; the marked letter $x$ in the display does not belong.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T88", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T88"}], "headline": "In the proof of Theorem 7.1.8, the sample mean that is unbiased and consistent for $\\mathrm{E}[\\tau_i^L]$ is $\\overline{\\tau_i^L}$, not $\\overline{Y_i^L}$, and the summand of $\\overline{\\tau_i^U}$ contains $Y_i(1-D_i)$, with no $x$.", "id": "P001-C07-E001", "location": "Proof of Theorem 7.1.8: p. 243 (PDF p. 261), sentence beginning “By Theorem 3.2.3”; p. 244 (PDF p. 262), display defining $\\overline{\\tau_i^U}$", "pdf_page": 261, "printed_page": "243", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T88"], "text": {"as_printed": "P.~243: ``By Theorem 3.2.3 (\\emph{The Expected Value of the Sample Mean is the Population Mean}) and the Weak Law of Large Numbers (WLLN), $\\hl{\\overline{Y_i^L}}$ is unbiased and consistent for $\\mathrm{E}[\\tau_i^L]$, and by the Law of Iterated Expectations,''\n\nP.~244:\n\\[\n\\overline{\\tau_i^U}=\\frac{1}{n}\\sum_{i=1}^{n}\\bigl(Y_iD_i+b(1-D_i)-Y_i(\\hl{x}1-D_i)-aD_i\\bigr).\n\\]", "corrected": "P.~243: ``By Theorem 3.2.3 (\\emph{The Expected Value of the Sample Mean is the Population Mean}) and the Weak Law of Large Numbers (WLLN), $\\hl{\\overline{\\tau_i^L}}$ is unbiased and consistent for $\\mathrm{E}[\\tau_i^L]$, and by the Law of Iterated Expectations,''\n\nP.~244:\n\\[\n\\overline{\\tau_i^U}=\\frac{1}{n}\\sum_{i=1}^{n}\\bigl(Y_iD_i+b(1-D_i)-Y_i\\hl{(1-D_i)}-aD_i\\bigr).\n\\]\n\nHere $\\tau_i^L=Y_iD_i+a(1-D_i)-Y_i(1-D_i)-bD_i$ and $\\overline{\\tau_i^L}=n^{-1}\\sum_{i=1}^{n}\\tau_i^L$, as defined in the proof, and $[a,b]$ contains the supports of $Y_i(0)$ and $Y_i(1)$.", "evidence": "The evidence is a check of pp.~243 and 244. Observe that the statement of Theorem 7.1.8 prints both summands without the $x$, and that the definitions of $\\tau_i^L$ and $\\tau_i^U$ in the proof determine both replacements. Since the variables $\\tau_i^L$ are i.i.d.\\ and bounded, Theorem 3.2.3 and the WLLN apply to their sample mean $\\overline{\\tau_i^L}$, which is the claim of the sentence on p.~243.", "problem": "Chapter 7 defines no variable $Y_i^L$. The line before the sentence on p.~243 defines $\\tau_i^L=Y_iD_i+a(1-D_i)-Y_i(1-D_i)-bD_i$ and its sample mean $\\overline{\\tau_i^L}$, and the calculation that follows computes $\\mathrm{E}[\\tau_i^L]$. The symbol $\\overline{Y_i^L}$ belongs to the proof of Theorem 6.1.4 (p.~212), which this proof follows. On p.~244, the line before the display defines $\\tau_i^U=Y_iD_i+b(1-D_i)-Y_i(1-D_i)-aD_i$, and the display should average this summand. The letter $x$ is extraneous.", "what_survives": "Theorem 7.1.8, both bounds in its statement and the rest of the proof are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the line 'The distribution of observables is' and a displayed piecewise function f(Y i, X i, D i) with seven rows. The first four rows give probability 1/6 to the cells (Y i, X i, D i) = (0,0,0), (1,0,0), (1,1,0) and (0,0,1). Red boxes mark the fifth and sixth rows, which both read '1/6 : Y i = 1, X i = 1, D i = 1'. The last row reads '0 : otherwise.'", "caption": "Example 7.1.17, p. 253 (PDF p. 271), the distribution of the observables. The two marked rows are identical: each gives probability $1/6$ to the cell $Y_i=1$, $X_i=1$, $D_i=1$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T89", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T89"}], "headline": "In Example 7.1.17 the cell $(Y_i,X_i,D_i)=(1,1,1)$ has probability $1/3$; the display lists it twice with probability $1/6$.", "id": "P001-C07-E002", "location": "Example 7.1.17, p. 253 (PDF p. 271), display after “The distribution of observables is”", "pdf_page": 271, "printed_page": "253", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T89"], "text": {"as_printed": "``The distribution of observables is\n\\[\nf(Y_i,X_i,D_i)=\\begin{cases}\n\\tfrac{1}{6} &:\\ Y_i=0,X_i=0,D_i=0\\\\\n\\tfrac{1}{6} &:\\ Y_i=1,X_i=0,D_i=0\\\\\n\\tfrac{1}{6} &:\\ Y_i=1,X_i=1,D_i=0\\\\\n\\tfrac{1}{6} &:\\ Y_i=0,X_i=0,D_i=1\\\\\n\\hl{\\tfrac{1}{6}} &:\\ \\hl{Y_i=1,X_i=1,D_i=1}\\\\\n\\hl{\\tfrac{1}{6}} &:\\ \\hl{Y_i=1,X_i=1,D_i=1}\\\\\n0 &:\\ \\text{otherwise.''}\n\\end{cases}\n\\]", "corrected": "``The distribution of observables is\n\\[\nf(Y_i,X_i,D_i)=\\begin{cases}\n\\tfrac{1}{6} &:\\ Y_i=0,X_i=0,D_i=0\\\\\n\\tfrac{1}{6} &:\\ Y_i=1,X_i=0,D_i=0\\\\\n\\tfrac{1}{6} &:\\ Y_i=1,X_i=1,D_i=0\\\\\n\\tfrac{1}{6} &:\\ Y_i=0,X_i=0,D_i=1\\\\\n\\hl{\\tfrac{1}{3}} &:\\ Y_i=1,X_i=1,D_i=1\\\\\n0 &:\\ \\text{otherwise.''}\n\\end{cases}\n\\]", "evidence": "The evidence is a recalculation from the joint distribution of the example. Recall that $X_i=X_i(1)D_i+X_i(0)(1-D_i)$ and that $Y_i(1)=Y_i(0)$ for every type. The six types then have the observables $(0,0,0)$, $(1,0,0)$, $(1,1,0)$, $(0,0,1)$, $(1,1,1)$ and $(1,1,1)$. Adding the probabilities, I find that the cells $(0,0,0)$, $(1,0,0)$, $(1,1,0)$ and $(0,0,1)$ have probability $1/6$ each and that the cell $(1,1,1)$ has probability $1/3$, with total one. With these probabilities, $\\Pr[X_i=0]=\\Pr[X_i=1]=1/2$, and I have\n\\[\n\\begin{aligned}\n\\mathrm{E}[Y_i|D_i=1,X_i=0]-\\mathrm{E}[Y_i|D_i=0,X_i=0]&=0-\\frac{1}{2}=-\\frac{1}{2},\\\\\n\\mathrm{E}[Y_i|D_i=1,X_i=1]-\\mathrm{E}[Y_i|D_i=0,X_i=1]&=1-1=0 .\n\\end{aligned}\n\\]\nThe adjusted contrast is then $(1/2)(-1/2)+(1/2)\\cdot 0=-1/4$, as printed.", "problem": "The joint distribution printed above this display has six equally likely types. Two of them give the same observables $(Y_i,X_i,D_i)=(1,1,1)$: the types with $(Y_i(0),Y_i(1),X_i(1),X_i(0),D_i)$ equal to $(1,1,1,0,1)$ and to $(1,1,1,1,1)$. Their probabilities must be added. The display instead lists this cell twice. Read as a function, it assigns the cell probability $1/6$, and the five distinct cells have total probability $5/6$.", "what_survives": "The joint distribution of potential outcomes and treatment, the random assignment of $D_i$, the absence of any effect on $Y_i$ and the adjusted contrast $-1/4$ are unchanged. The example still shows that conditioning on a post-treatment variable can produce a nonzero contrast when the ATE is zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of a table with columns Unit, Y i(0), Y i(1), D i, X [1]i and X [2]i and six rows. Missing outcomes are filled with expressions of the form beta-hat 0 + beta-hat 1 times d + beta-hat 2 times the first covariate + beta-hat 3 times the second covariate. In the row for unit 3, red boxes mark the 9 in 'beta-hat 2 times 9' and the 9 in the column X [1]i.", "caption": "Example 7.2.2, p. 259 (PDF p. 277), second table. In the row for unit 3, the marked entries give the first covariate as $9$, in the imputed outcome and in the covariate column.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E003.png"}, {"alt": "Crop of the heading 'Example 7.2.2. Causal Inference with OLS Regression', the sentence 'Suppose we had the following data:' and a table of six units with columns Unit, Y i(0), Y i(1), D i, X [1]i and X [2]i. The row for unit 3 reads: question mark, 3, 1, 10, 3. A red box marks the 10.", "caption": "Example 7.2.2, p. 259 (PDF p. 277), first table. The marked entry gives the first covariate of unit 3 as $10$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E003-b.png"}, {"alt": "Crop of a table in which both potential outcomes of all six units are written as fitted expressions. In the row for unit 3, red boxes mark the 9 in 'beta-hat 2 times 9' in the column Y i(0), the 9 in 'beta-hat 2 times 9' in the column Y i(1), and the 9 in the column X [1]i.", "caption": "Example 7.2.2, p. 259 (PDF p. 277), third table. In the row for unit 3, the three marked entries again give the first covariate as $9$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E003-c.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T92", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T92"}, {"id": "P001-T93", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T93"}], "headline": "In the two imputation tables of Example 7.2.2, the first covariate of unit 3 is $10$, as in the data table, not $9$.", "id": "P001-C07-E003", "location": "Example 7.2.2, p. 259 (PDF p. 277), row for unit 3 in the three tables", "pdf_page": 277, "printed_page": "259", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T92", "P001-T93"], "text": {"as_printed": "First table (the data), unit 3: $Y_i(0)={?}$, $Y_i(1)=3$, $D_i=1$, $X_{[1]i}=10$, $X_{[2]i}=3$.\n\nSecond table (missing outcomes imputed), unit 3: $Y_i(0)=\\hat\\beta_0+\\hat\\beta_1\\cdot 0+\\hat\\beta_2\\cdot\\hl{9}+\\hat\\beta_3\\cdot 3$, $Y_i(1)=3$, $D_i=1$, $X_{[1]i}=\\hl{9}$, $X_{[2]i}=3$.\n\nThird table (all outcomes imputed), unit 3: $Y_i(0)=\\hat\\beta_0+\\hat\\beta_1\\cdot 0+\\hat\\beta_2\\cdot\\hl{9}+\\hat\\beta_3\\cdot 3$, $Y_i(1)=\\hat\\beta_0+\\hat\\beta_1\\cdot 1+\\hat\\beta_2\\cdot\\hl{9}+\\hat\\beta_3\\cdot 3$, $D_i=1$, $X_{[1]i}=\\hl{9}$, $X_{[2]i}=3$.", "corrected": "Second table, unit 3: $Y_i(0)=\\hat\\beta_0+\\hat\\beta_1\\cdot 0+\\hat\\beta_2\\cdot\\hl{10}+\\hat\\beta_3\\cdot 3$ and $X_{[1]i}=\\hl{10}$.\n\nThird table, unit 3: $Y_i(0)=\\hat\\beta_0+\\hat\\beta_1\\cdot 0+\\hat\\beta_2\\cdot\\hl{10}+\\hat\\beta_3\\cdot 3$, $Y_i(1)=\\hat\\beta_0+\\hat\\beta_1\\cdot 1+\\hat\\beta_2\\cdot\\hl{10}+\\hat\\beta_3\\cdot 3$ and $X_{[1]i}=\\hl{10}$.", "evidence": "The evidence is a recalculation from the first table. The OLS regression of $Y_i$ on $(1,D_i,X_{[1]i},X_{[2]i})$ has coefficients\n\\[\n(\\hat\\beta_0,\\hat\\beta_1,\\hat\\beta_2,\\hat\\beta_3)=\\frac{1}{2437}\\,(13563,\\,6178,\\,-759,\\,-1778).\n\\]\nWith $9$ in the prediction for unit 3, I find the estimate $12103/4874$ of $\\mathrm{E}[\\tau_i]$ when only the missing outcomes are imputed, and $6178/2437$ when all outcomes are imputed. The second is larger by $253/4874$, although the example says that the two procedures yield the same estimate. With $10$, both estimates equal $6178/2437=\\hat\\beta_1$.", "problem": "The three tables describe the same six units, but the first covariate of unit 3 changes from $10$ to $9$ after the first table. The value $9$ appears in the covariate column of both imputation tables and in three terms $\\hat\\beta_2\\cdot 9$. Example 7.2.4 (pp.~262--263) uses the same six units and has $10$ in all three of its tables.", "what_survives": "The regression estimator of Definition 7.2.3 is unchanged, and the two imputation estimates are equal for a single data set. The example reports no numerical estimate, and no printed number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of four lines of text and a displayed formula. The text ends 'the maximum likelihood plug-in estimator for E[Y i(d)] is computed by substituting theta-hat ML for theta and replacing the expected value with respect to X i with a sample mean:'. The display reads E-hat[Y i(d)] = (1/n) times the sum over i from 1 to n of the integral of y g(y, (d, x), theta-hat ML) dy. A red box marks the bold lower-case x.", "caption": "Section 7.2.3, p. 261 (PDF p. 279), last display. The marked argument is a lower-case $\\mathbf{x}$ with no index; the sentence above the display says that the expected value with respect to $\\mathbf{X}_i$ is replaced with a sample mean.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T94", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T94"}], "headline": "In the maximum likelihood plug-in estimator of $\\mathrm{E}[Y_i(d)]$, the $i$th summand is evaluated at the observed covariates $\\mathbf{X}_i$, not at a free $\\mathbf{x}$.", "id": "P001-C07-E004", "location": "Section 7.2.3, p. 261 (PDF p. 279), last display", "pdf_page": 279, "printed_page": "261", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T94"], "text": {"as_printed": "\\[\n\\hat{\\mathrm{E}}\\bigl[Y_i(d)\\bigr]=\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\int_{-\\infty}^{\\infty}y\\,g\\bigl(y,(d,\\hl{\\mathbf{x}}),\\hat{\\boldsymbol{\\theta}}_{ML}\\bigr)\\,dy\\right).\n\\]", "corrected": "\\[\n\\hat{\\mathrm{E}}\\bigl[Y_i(d)\\bigr]=\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\int_{-\\infty}^{\\infty}y\\,g\\bigl(y,(d,\\hl{\\mathbf{X}_i}),\\hat{\\boldsymbol{\\theta}}_{ML}\\bigr)\\,dy\\right).\n\\]\nHere, as in Section 7.2.3, strong ignorability holds, $g(y,(d,\\mathbf{x}),\\boldsymbol{\\theta})$ is a correctly specified parametric model for the conditional density of $Y_i$ given $(D_i,\\mathbf{X}_i)=(d,\\mathbf{x})$, and $\\hat{\\boldsymbol{\\theta}}_{ML}$ is the maximum likelihood estimate from $n$ i.i.d.\\ observations of $(Y_i,D_i,\\mathbf{X}_i)$.", "evidence": "The evidence is a derivation. Set\n\\[\nm_{\\boldsymbol{\\theta}}(d,\\mathbf{x})=\\int_{-\\infty}^{\\infty}y\\,g\\bigl(y,(d,\\mathbf{x}),\\boldsymbol{\\theta}\\bigr)\\,dy .\n\\]\nThe first display of Section 7.2.3 states that $\\mathrm{E}[Y_i(d)]=\\int_{-\\infty}^{\\infty}m_{\\boldsymbol{\\theta}}(d,\\mathbf{x})f_{\\mathbf{X}_i}(\\mathbf{x})\\,d\\mathbf{x}$. Replacing the distribution of $\\mathbf{X}_i$ by the empirical distribution of $\\mathbf{X}_1,\\ldots,\\mathbf{X}_n$, and $\\boldsymbol{\\theta}$ by $\\hat{\\boldsymbol{\\theta}}_{ML}$, I obtain\n\\[\n\\frac{1}{n}\\sum_{i=1}^{n}m_{\\hat{\\boldsymbol{\\theta}}_{ML}}(d,\\mathbf{X}_i).\n\\]\nThe printed expression equals $m_{\\hat{\\boldsymbol{\\theta}}_{ML}}(d,\\mathbf{x})$. For example, suppose that the fitted conditional distribution of $Y_i$ is normal with mean $x$ and variance $1$, and that the observed covariates are $0$ and $1$. Then the intended estimate is $1/2$, and the printed expression is $x$.", "problem": "As printed, the summand does not depend on $i$, and $\\mathbf{x}$ is a free variable. The right side is then the fitted conditional mean at $\\mathbf{x}$, not an average over the observed covariates. The sentence before the display says that the expected value with respect to $\\mathbf{X}_i$ is replaced with a sample mean. The display for the estimator of $\\mathrm{E}[\\tau_i]$ at the top of p.~262 evaluates $g$ at $(1,\\mathbf{X}_i)$ and $(0,\\mathbf{X}_i)$.", "what_survives": "The identification display above it and the construction of the estimator are unchanged. The estimator of $\\mathrm{E}[\\tau_i]$ on p.~262 is printed with $\\mathbf{X}_i$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text. The last sentence reads: 'In the case where we have a discrete and finite number of values for X i, the IPW estimator is logically equivalent to the plug-in estimator from Section 7.2.1 (Plug-In Estimation).' Red boxes mark the words 'logically' and 'equivalent', which fall on two lines.", "caption": "Section 7.2.6, p. 266 (PDF p. 284), end of the paragraph after Definition 7.2.6. The marked words state the equivalence with no condition on the fitted propensity scores.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T96", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T96"}, {"id": "P001-T97", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T97"}], "headline": "With discrete covariates, the IPW estimator equals the plug-in estimator of Section 7.2.1 when each fitted propensity score is the observed treated fraction in its covariate cell, not for every fit.", "id": "P001-C07-E005", "location": "Section 7.2.6, p. 266 (PDF p. 284), paragraph after Definition 7.2.6, last sentence", "pdf_page": 284, "printed_page": "266", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T96", "P001-T97"], "text": {"as_printed": "``In the case where we have a discrete and finite number of values for $\\mathbf{X}_i$, the IPW estimator is \\hl{logically equivalent} to the plug-in estimator from Section 7.2.1 (\\emph{Plug-In Estimation}).''", "corrected": "``With discrete covariates taking finitely many values, the IPW estimator \\hl{equals} the post-stratification plug-in estimator in Section 7.2.1 \\hl{when each fitted propensity equals the observed treated fraction in its covariate cell (as under saturated propensity estimation), provided both treatment arms are represented in every observed cell}.''\n\nIn symbols: if every observed covariate value $\\mathbf{x}$ has $N_{1\\mathbf{x}}\\ge 1$ and $N_{0\\mathbf{x}}\\ge 1$, and $\\hat p_D(\\mathbf{x})=N_{1\\mathbf{x}}/N_{\\mathbf{x}}$, then\n\\[\n\\frac{1}{n}\\sum_{i=1}^{n}\\left(\\frac{Y_iD_i}{\\hat p_D(\\mathbf{X}_i)}-\\frac{Y_i(1-D_i)}{1-\\hat p_D(\\mathbf{X}_i)}\\right)=\\sum_{\\mathbf{x}}\\frac{N_{\\mathbf{x}}}{n}\\bigl(\\overline{Y}_{1\\mathbf{x}}-\\overline{Y}_{0\\mathbf{x}}\\bigr).\n\\]\nFor other fitted propensity scores the two sides can differ.", "evidence": "The evidence is a derivation and an exact counterexample. For each observed covariate value $\\mathbf{x}$, let $N_{\\mathbf{x}}$ be the number of observations with $\\mathbf{X}_i=\\mathbf{x}$, let $N_{1\\mathbf{x}}$ and $N_{0\\mathbf{x}}$ be the numbers of treated and control observations among them, and let $\\overline{Y}_{1\\mathbf{x}}$ and $\\overline{Y}_{0\\mathbf{x}}$ be their mean outcomes. With both treatment arms observed in every cell and $0<\\hat p_D(\\mathbf{x})<1$, grouping the sum in Definition 7.2.6 by covariate value, I obtain\n\\[\n\\hat{\\mathrm{E}}_{IPW}[\\tau_i]=\\sum_{\\mathbf{x}}\\left(\\frac{N_{1\\mathbf{x}}}{n\\,\\hat p_D(\\mathbf{x})}\\,\\overline{Y}_{1\\mathbf{x}}-\\frac{N_{0\\mathbf{x}}}{n\\,(1-\\hat p_D(\\mathbf{x}))}\\,\\overline{Y}_{0\\mathbf{x}}\\right),\n\\]\nand the estimator of Section 7.2.1 is $\\sum_{\\mathbf{x}}(N_{\\mathbf{x}}/n)(\\overline{Y}_{1\\mathbf{x}}-\\overline{Y}_{0\\mathbf{x}})$. The two agree for every vector of outcomes if and only if $\\hat p_D(\\mathbf{x})=N_{1\\mathbf{x}}/N_{\\mathbf{x}}$ for every observed $\\mathbf{x}$.\n\nThe counterexample has 18 observations and one covariate $Z_i\\in\\{-1,0,1\\}$, with six observations at each value. The propensity score is fitted by maximum likelihood in the logit model with an intercept and a slope in $Z_i$. The fitted coefficients are $0$ and $\\log 2$. Indeed, the score equations hold at this point, and the maximum is unique because the log likelihood is strictly concave. The estimator of Section 7.2.1 equals $1$, and the IPW estimator of Definition 7.2.6 equals $59/36$. This sample has positive probability under i.i.d.\\ sampling from a population in which strong ignorability holds and the true propensity score is this logit model. A correctly specified model therefore does not restore the equality. On the data of Example 7.2.1, the estimator of Section 7.2.1 is $4/9$, as printed. The IPW estimator equals $1/4$ with the maximum likelihood fit $\\hat p_D=2/3$ of the model with an intercept only, and $4/9$ with the treated fractions $1/2$ and $3/4$.\n\\begin{audittable}{rrrrrrr}{The counterexample: 18 observations in three covariate cells, with the fitted propensity score and the observed treated fraction in each cell.}\n$Z_i$ & $N_{\\mathbf{x}}$ & $N_{1\\mathbf{x}}$ & fitted $\\hat p_D(\\mathbf{x})$ & $N_{1\\mathbf{x}}/N_{\\mathbf{x}}$ & $\\overline{Y}_{1\\mathbf{x}}$ & $\\overline{Y}_{0\\mathbf{x}}$ \\\\\n$-1$ & 6 & 3 & $1/3$ & $1/2$ & 4 & 3 \\\\\n$0$ & 6 & 1 & $1/2$ & $1/6$ & 3 & 2 \\\\\n$1$ & 6 & 5 & $2/3$ & $5/6$ & 4 & 3 \\\\\n\\end{audittable}", "problem": "Definition 7.2.6 allows any estimate $\\hat p_D$ of the propensity score, and Section 7.2.5 proceeds with a maximum likelihood estimate from a binary choice model. Such a fit need not reproduce the treated fraction in each covariate cell, and the IPW estimate can then differ from the estimate of Section 7.2.1. Finite support of $\\mathbf{X}_i$, strong ignorability and a correctly specified model for the propensity score do not make the two estimators equal. They are equal when $\\hat p_D(\\mathbf{x})$ is the treated fraction among the observations with $\\mathbf{X}_i=\\mathbf{x}$, as under a saturated model. The two estimators do rest on the same reweighting argument, and that part of the remark is correct.", "what_survives": "Theorem 7.1.13, Theorem 7.2.5, the definitions of the IPW and stabilized IPW estimators, and the consistency of each estimator under its own assumptions do not use this sentence. Example 7.2.1 and its answer $4/9$ are unchanged. The application fits a logit model and Winsorizes the fitted propensity scores. It does not use the equality, and this correction requires no change to its reported numbers."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of a paragraph of nine lines. It contains the sentence 'with the true propensity score, E[D i / p D(X i)] = E[(1 minus D i) / (1 minus p D(X i))] = 1'. Red boxes mark two expressions in the later sentence 'Because the stabilized IPW estimator accounts for this between-sample variation in ... and ..., it typically confers efficiency gains over the IPW estimator in practice': the sum over i from 1 to n of 1 / p D(X i), and the sum over i from 1 to n of 1 / (1 minus p D(X i)).", "caption": "Section 7.2.6, p. 267 (PDF p. 285), paragraph after Definition 7.2.7. The two marked sums have numerator $1$ and run over all observations; two sentences earlier the expectations are written with $D_i$ and $1-D_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T98", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T98"}], "headline": "In the explanation of the stabilized IPW estimator, the numerators of the two sums are the treatment indicators $D_i$ and $1-D_i$.", "id": "P001-C07-E006", "location": "Section 7.2.6, p. 267 (PDF p. 285), paragraph after Definition 7.2.7, sentence beginning “Because the stabilized IPW estimator accounts for”", "pdf_page": 285, "printed_page": "267", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T98"], "text": {"as_printed": "``Because the stabilized IPW estimator accounts for this between-sample variation in $\\sum_{i=1}^{n}\\frac{\\hl{1}}{p_D(\\mathbf{X}_i)}$ and $\\sum_{i=1}^{n}\\frac{\\hl{1}}{1-p_D(\\mathbf{X}_i)}$, it typically confers efficiency gains over the IPW estimator in practice.''", "corrected": "``Because the stabilized IPW estimator accounts for this between-sample variation in $\\sum_{i=1}^{n}\\frac{\\hl{D_i}}{\\hl{\\hat p_D}(\\mathbf{X}_i)}$ and $\\sum_{i=1}^{n}\\frac{\\hl{1-D_i}}{1-\\hl{\\hat p_D}(\\mathbf{X}_i)}$, it typically confers efficiency gains over the IPW estimator in practice.''\n\nThe hats follow Definition 7.2.7, which uses the estimated propensity score. With the true propensity score, the same two sums are written with $p_D$.", "evidence": "The evidence is a check of the page and a short computation. Two sentences earlier the text states that $\\mathrm{E}[D_i/p_D(\\mathbf{X}_i)]=\\mathrm{E}[(1-D_i)/(1-p_D(\\mathbf{X}_i))]=1$, with the indicators. Suppose that $p_D(\\mathbf{X}_i)=1/2$ for every unit. Then each printed sum equals $2n$ in every sample, while the sums with the indicators equal $2n_T$ and $2(n-n_T)$, where $n_T$ is the number of treated units.", "problem": "The two sums in this sentence run over all $n$ observations and have no treatment indicators. The denominators of the stabilized IPW estimator in Definition 7.2.7, on the same page, are $n^{-1}\\sum_{i=1}^{n}D_i/\\hat p_D(\\mathbf{X}_i)$ and $n^{-1}\\sum_{i=1}^{n}(1-D_i)/(1-\\hat p_D(\\mathbf{X}_i))$. These are the quantities whose variation between samples the estimator accounts for.", "what_survives": "Definition 7.2.7 and the two expectation identities are printed correctly. Only the notation of the explanation changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the displayed null hypothesis 'W i independent of D i given X i' and the paragraph after it, which describes an F-test that compares a regression of D i on W i and X i with a regression of D i on X i alone. A red box marks the last sentence: 'If so, then we can reject the null hypothesis that W i is independent of D i conditional on X i.'", "caption": "Section 7.2.8, p. 270 (PDF p. 288). The marked sentence draws the conclusion that the $F$-test does not support: rejection of the conditional independence of $\\mathbf{W}_i$ and $D_i$ given $\\mathbf{X}_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E007.png"}, {"alt": "Line chart with the covariate value X i (minus 1, 0, 1) on the horizontal axis and the probability of treatment, from 0 to 1, on the vertical axis. Blue circles for the true propensity score lie at 0.75, 0.25 and 0.75. Orange squares for the OLS fit of D i on (1, X i) lie on a flat line at 7/12, about 0.58. Green triangles for the OLS fit of D i on (1, X i, W i), averaged given X i, lie at 45/68, 29/68 and 45/68, about 0.66, 0.43 and 0.66. The title reads 'The regression that is linear in X i misses the curve; the placebo W i picks it up'.", "caption": "The counterexample to the $F$-test of Section 7.2.8. The propensity score is $3/4$, $1/4$, $3/4$ at $X_i=-1,0,1$; the regression of $D_i$ on $(1,X_i)$ fits $7/12$ at every $X_i$; the regression that adds $W_i$ fits $45/68$, $29/68$, $45/68$ on average, although $W_i$ is independent of $D_i$ given $X_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E007-placebo-fit.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T17", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T17"}, {"id": "P001-T18", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T18"}, {"id": "P001-T124", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T124"}], "headline": "The $F$-test for adding the placebo variables $\\mathbf{W}_i$ to a regression of $D_i$ on $\\mathbf{X}_i$ can reject $\\mathbf{W}_i\\perp\\!\\!\\!\\perp D_i\\mid\\mathbf{X}_i$ with probability tending to one even when it holds.", "id": "P001-C07-E007", "location": "Section 7.2.8 (Placebo Testing), p. 270 (PDF p. 288), sentences beginning “One simple way to implement this is by performing an $F$-test”", "pdf_page": 288, "printed_page": "270", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T124", "P001-T17", "P001-T18"], "text": {"as_printed": "``Then we can test the null hypothesis that\n\\[\n\\mathbf{W}_i\\perp\\!\\!\\!\\perp D_i\\mid\\mathbf{X}_i.\n\\]\nThis is known as a \\emph{placebo test}.$^{14}$ One simple way to implement this is by performing an $F$-test. An $F$-test compares a regression of $D_i$ on $\\mathbf{W}_i$ and $\\mathbf{X}_i$ with a regression of $D_i$ on $\\mathbf{X}_i$ alone, and asks: does adding $\\mathbf{W}_i$ to the regression explain more of the remaining variation in $D_i$ than we would expect by chance? \\hl{If so, then we can reject the null hypothesis that $\\mathbf{W}_i$ is independent of $D_i$ conditional on $\\mathbf{X}_i$.}''", "corrected": "``One simple regression-based placebo check is a joint test of the coefficients on $\\mathbf{W}_i$ in the OLS regression of $D_i$ on $\\mathbf{W}_i$ and $\\mathbf{X}_i$. \\hl{If, under the null, $\\mathrm{E}[D_i\\mid\\mathbf{X}_i]$ is correctly represented by the included $\\mathbf{X}_i$ regressors and inference uses a consistent heteroskedasticity-robust covariance estimator under the usual large-sample conditions}, conditional independence of $\\mathbf{W}_i$ and $D_i$ given $\\mathbf{X}_i$ implies that these coefficients are zero. A rejection can then be evidence against conditional independence, \\hl{subject to the conditional-mean specification assumption}. \\hl{Without it, $\\mathbf{W}_i$ may simply capture omitted nonlinear functions of $\\mathbf{X}_i$}; use a procedure justified for the conditional-independence null if that specification assumption is unavailable.''\n\nIn full: let the observations $(D_i,\\mathbf{X}_i,\\mathbf{W}_i)$ be i.i.d.\\ with finite second moments, let $\\mathbf{X}_i$ include a constant, and let the matrix of second moments of $(\\mathbf{X}_i,\\mathbf{W}_i)$ be nonsingular. If $\\mathrm{E}[D_i\\mid\\mathbf{X}_i]$ is a linear combination of the components of $\\mathbf{X}_i$ and $\\mathbf{W}_i\\perp\\!\\!\\!\\perp D_i\\mid\\mathbf{X}_i$, then the coefficients on $\\mathbf{W}_i$ in the best linear predictor of $D_i$ given $\\mathbf{X}_i$ and $\\mathbf{W}_i$ are zero. The linearity condition is sufficient for this conclusion; it is a restriction on the propensity score that an application may not satisfy.", "evidence": "The evidence is an exact counterexample and a derivation of the corrected statement. Let $X_i$ be uniform on $\\{-1,0,1\\}$, let $Z_i$ be uniform on $\\{-1,1\\}$ and independent of $X_i$, and set\n\\[\nW_i=X_i^2+\\frac{Z_i}{2},\\qquad \\Pr[D_i=1\\mid X_i,Z_i]=\\frac{1}{4}+\\frac{X_i^2}{2}.\n\\]\nGiven $X_i$, the placebo $W_i$ is a function of $Z_i$, and $Z_i$ is independent of $D_i$ given $X_i$, so that $W_i\\perp\\!\\!\\!\\perp D_i\\mid X_i$ holds exactly. The propensity score takes the values $1/4$ and $3/4$, so positivity holds. With $Y_i(0)=2+Z_i$ and $Y_i(1)=3+Z_i$, strong ignorability holds given $X_i$, and $\\mathrm{E}[Y_i\\mid D_i,X_i]=2+D_i$ is linear.\n\nIn the population, the regression of $D_i$ on $(1,X_i)$ has coefficients $(7/12,0)$, and the regression of $D_i$ on $(1,X_i,W_i)$ has coefficients $(29/68,0,4/17)$. The coefficient on $W_i$ is not zero. Let $F_n$ denote the $F$-statistic that compares the two regressions. By the strong law of large numbers, I have\n\\[\n\\frac{F_n}{n}\\longrightarrow\\frac{35/144-59/272}{59/272}=\\frac{64}{531}\\quad\\text{almost surely},\n\\]\nso that the test rejects the true null hypothesis with probability tending to one at every fixed level. The table lists the other exact values. A second example, with $X_i$ uniform on $[-1,1]$, $W_i=X_i^2$ and the same propensity score, has $F_n/n\\to 16/159$.\n\\begin{audittable}{lr}{The counterexample: exact values for the two regressions and the two test statistics.}\nquantity & value \\\\\ncoefficient on $W_i$ in the regression of $D_i$ on $(1,X_i,W_i)$ & $4/17$ \\\\\nresidual variance, regression of $D_i$ on $(1,X_i)$ & $35/144$ \\\\\nresidual variance, regression of $D_i$ on $(1,X_i,W_i)$ & $59/272$ \\\\\nlimit of $F_n/n$ & $64/531$ \\\\\nlimit of the robust Wald statistic for the coefficient on $W_i$, divided by $n$ & $18496/135315$ \\\\\n$F$-statistic in a sample of $240$ observations with the population frequencies & $5056/177$ \\\\\ncoefficient on $W_i$ after $X_i^2$ is added to both regressions & $0$ \\\\\n\\end{audittable}\n\nFor the corrected statement, suppose that $\\mathbf{X}_i$ includes a constant and that $\\mathrm{E}[D_i\\mid\\mathbf{X}_i]=\\mathbf{X}_i\\boldsymbol{\\gamma}$ for some $\\boldsymbol{\\gamma}$. Under the null hypothesis, I have $\\mathrm{E}[D_i\\mid\\mathbf{X}_i,\\mathbf{W}_i]=\\mathbf{X}_i\\boldsymbol{\\gamma}$, and the error $e_i=D_i-\\mathbf{X}_i\\boldsymbol{\\gamma}$ satisfies $\\mathrm{E}[e_i\\mid\\mathbf{X}_i,\\mathbf{W}_i]=0$. Then $e_i$ is uncorrelated with $\\mathbf{X}_i$ and with $\\mathbf{W}_i$. These are the normal equations of the regression of $D_i$ on $\\mathbf{X}_i$ and $\\mathbf{W}_i$, solved with zero coefficients on $\\mathbf{W}_i$. If the matrix of second moments of the regressors is nonsingular, this solution is unique, and conditional independence implies that the coefficients on $\\mathbf{W}_i$ are zero.", "problem": "Recall that, in the book, a regression without qualification is an OLS regression (p.~147, footnote 3). The $F$-test therefore asks whether the coefficients on $\\mathbf{W}_i$ are zero in the best linear predictor of $D_i$ given $\\mathbf{W}_i$ and $\\mathbf{X}_i$. Under the null hypothesis, $\\mathrm{E}[D_i\\mid\\mathbf{X}_i,\\mathbf{W}_i]=\\mathrm{E}[D_i\\mid\\mathbf{X}_i]$, but this conditional mean need not be linear in $\\mathbf{X}_i$. The chapter does not assume that it is, and a propensity score in general is not. When it is not linear, $\\mathbf{W}_i$ can carry a nonlinear function of $\\mathbf{X}_i$ that the regression omits. The coefficient on $\\mathbf{W}_i$ can therefore be nonzero in the population although the null hypothesis holds; when it is, the test rejects with probability tending to one as $n$ grows. The failure concerns the size of the test, not its power. In that case, heteroskedasticity-robust standard errors do not help, because the coefficient that they are attached to is not zero.", "what_survives": "The identification results, the estimators and the estimates of the application are not affected; the application does not use this test. The point that ignorability cannot be tested directly remains, and so does the value of placebo evidence from a test that is valid for the null hypothesis of conditional independence. The $F$-test of p.~270 changes: the linearity condition with valid large-sample inference is a sufficient justification, or it can be read as a test of whether $\\mathbf{W}_i$ improves the linear prediction of the treatment."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first two paragraphs of Section 7.3, fifteen lines. The first paragraph gives the positivity assumption as 'there exists epsilon greater than 0 such that epsilon is less than p D(X i), which is less than 1 minus epsilon'. Red boxes mark the words 'a failure of the positivity assumption logically implies incomplete (or limited) overlap' in the sentence that begins 'We proceed by illustrating that'.", "caption": "Opening of Section 7.3, p. 271 (PDF p. 289). The first paragraph states the positivity assumption with a single $\\epsilon$; the marked words say that its failure implies incomplete overlap.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E008.png"}, {"alt": "Crop of the shaded box of Theorem 7.3.2, 'Positivity Implies Complete Population Overlap', and the two lines after it. Red boxes mark the words 'Furthermore, the converse of this claim (that population overlap implies positivity) can almost be stated here', which are followed by 'but requires some measure-theoretic'.", "caption": "Theorem 7.3.2 and the sentence after it, p. 271 (PDF p. 289); the sentence continues on p. 272. The marked words say that the converse of the theorem can almost be stated.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E008-b.png"}, {"alt": "Two charts with the covariate value k, from 1 to 16, on the horizontal axis and logarithmic vertical axes. Left panel, 'Both arms put positive probability on every k': blue circles for the treated arm, Pr[X i = k given D i = 1] = 2 divided by (3 to the power k), fall from 0.67 to about 5 times 10 to the minus 8, and orange squares for the control arm, Pr[X i = k given D i = 0] = 1/2 to the power k, fall from 0.5 to about 1.5 times 10 to the minus 5; every value is positive. Right panel, 'The propensity score falls below every epsilon greater than 0': blue circles for p D(k) fall from 0.57 at k = 1 to 0.003 at k = 16, crossing a horizontal line at epsilon = 1/10 at k = 8 and a horizontal line at epsilon = 1/100 at k = 14.", "caption": "The counterexample to the converse of Theorem 7.3.2. Left: the conditional distributions of $X_i$ in the two treatment arms have the same support $\\{1,2,\\ldots\\}$. Right: the propensity score $p_D(k)$ is below $1/10$ from $k=8$ and below $1/100$ from $k=14$. Both vertical axes are logarithmic.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E008-overlap-positivity.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T19", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T19"}, {"id": "P001-T20", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T20"}, {"id": "P001-T104", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T104"}], "headline": "A failure of the positivity assumption does not imply incomplete overlap: for discrete covariates, complete population overlap implies $0<p_D(\\mathbf{x})<1$ at each covariate value in the support, but need not give a single bound $\\epsilon$ for all of them.", "id": "P001-C07-E008", "location": "Section 7.3, pp. 271–272 (PDF pp. 289–290): the paragraph before Definition 7.3.1, and the sentence after Theorem 7.3.2", "pdf_page": 289, "printed_page": "271–272", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T104", "P001-T19", "P001-T20"], "text": {"as_printed": "P.~271, before Definition 7.3.1: ``We proceed by illustrating that \\hl{a failure of the positivity assumption logically implies incomplete (or ``limited'') overlap} in the population of interest. \\hl{When the positivity assumption fails}, then the distributions of $\\mathbf{X}_i$ for treated and control units have regions with positive probability that do not completely overlap with one another.''\n\nP.~271: ``\\textbf{Theorem 7.3.2.} \\emph{Positivity Implies Complete Population Overlap} Let $D_i$ be a random variable with $\\operatorname{Supp}[D_i]=\\{0,1\\}$, and let $\\mathbf{X}_i$ be a random vector. If $\\exists\\epsilon>0$ such that $\\epsilon<p_D(\\mathbf{X}_i)<1-\\epsilon$, then $D_i$ has \\emph{complete population overlap} with respect to $\\mathbf{X}_i$.''\n\nPp.~271--272: ``Furthermore, \\hl{the converse of this claim (that population overlap implies positivity) can almost be stated here}, but requires some measure-theoretic conditions on support.''", "corrected": "P.~271, before Definition 7.3.1: ``When a propensity is \\hl{zero or one on part of the covariate support}, complete population overlap fails. \\hl{Uniform positivity can also fail with complete overlap} if propensities approach zero or one.''\n\nPp.~271--272, after Theorem 7.3.2: ``The converse of Theorem 7.3.2 \\hl{does not hold in general}: complete population overlap does not imply a uniform positivity bound. In the discrete case, complete population overlap is equivalent to \\hl{pointwise positivity} on the covariate support.''\n\nIn full: let $D_i$ be a random variable with $\\operatorname{Supp}[D_i]=\\{0,1\\}$, and let $\\mathbf{X}_i$ be a discrete random vector. Then $D_i$ has complete population overlap with respect to $\\mathbf{X}_i$ if and only if $0<p_D(\\mathbf{x})<1$ for every $\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}_i]$. If in addition $\\operatorname{Supp}[\\mathbf{X}_i]$ is finite, then complete population overlap implies that there exists $\\epsilon>0$ such that $\\epsilon<p_D(\\mathbf{X}_i)<1-\\epsilon$.", "evidence": "The evidence is an exact counterexample and a derivation for discrete covariates. Let $X_i$ take the values $k=1,2,\\ldots$ with\n\\[\n\\Pr[X_i=k,D_i=1]=3^{-k},\\qquad \\Pr[X_i=k,D_i=0]=2^{-(k+1)}.\n\\]\nThese probabilities sum to one, and $\\Pr[D_i=1]=\\Pr[D_i=0]=1/2$. I have $\\Pr[X_i=k\\mid D_i=1]=2/3^k$ and $\\Pr[X_i=k\\mid D_i=0]=1/2^k$, which are positive for every $k\\ge 1$, so that\n\\[\n\\operatorname{Supp}[X_i\\mid D_i=1]=\\operatorname{Supp}[X_i\\mid D_i=0]=\\{1,2,3,\\ldots\\}\n\\]\nand $D_i$ has complete population overlap with respect to $X_i$. The propensity score is\n\\[\np_D(k)=\\frac{2}{(3/2)^k+2},\n\\]\nwhich lies strictly between $0$ and $1$ for every $k$. Since $(3/2)^k\\ge 1+k/2$, I have $p_D(k)\\le 4/(k+6)$. For any $\\epsilon>0$, the value $k=\\lceil 4/\\epsilon\\rceil$ satisfies $p_D(k)<\\epsilon$ and $\\Pr[X_i=k]>0$. Hence there exists no $\\epsilon>0$ such that $\\epsilon<p_D(X_i)$. For example, $p_D(8)=512/7073<1/10$ and $p_D(14)=32768/4815737<1/100$. Under conditional independence and consistency, identification is not lost in this example: every cell $(k,d)$ has positive probability. With $Y_i(1)=1$ and $Y_i(0)=0$, the IPW estimator with the true propensity score has variance $5/(4n)$ for $n$ independent draws.\n\nFor discrete $\\mathbf{X}_i$, set $w(\\mathbf{x})=\\Pr[\\mathbf{X}_i=\\mathbf{x}]$ and $\\pi=\\Pr[D_i=1]$. For $\\mathbf{x}$ such that $w(\\mathbf{x})>0$, Bayes' rule gives\n\\[\n\\Pr[\\mathbf{X}_i=\\mathbf{x}\\mid D_i=1]=\\frac{p_D(\\mathbf{x})\\,w(\\mathbf{x})}{\\pi},\\qquad \\Pr[\\mathbf{X}_i=\\mathbf{x}\\mid D_i=0]=\\frac{(1-p_D(\\mathbf{x}))\\,w(\\mathbf{x})}{1-\\pi}.\n\\]\nThe two conditional supports are equal if and only if both probabilities are positive for every such $\\mathbf{x}$, that is, if and only if $0<p_D(\\mathbf{x})<1$ for every $\\mathbf{x}\\in\\operatorname{Supp}[\\mathbf{X}_i]$. If $\\operatorname{Supp}[\\mathbf{X}_i]$ is finite, then $m=\\min_{\\mathbf{x}}\\min\\{p_D(\\mathbf{x}),1-p_D(\\mathbf{x})\\}$ is positive, and $\\epsilon=m/2$ satisfies $\\epsilon<p_D(\\mathbf{X}_i)<1-\\epsilon$.", "problem": "Recall that the positivity assumption of the chapter (Definition 7.1.12, restated at the opening of Section 7.3) asks for one $\\epsilon>0$ such that $\\epsilon<p_D(\\mathbf{X}_i)<1-\\epsilon$ for all covariate values. Complete population overlap (Definition 7.3.1) asks that the supports of $\\mathbf{X}_i$ in the two treatment arms be equal. For discrete covariates, equal supports imply that $0<p_D(\\mathbf{x})<1$ at each covariate value $\\mathbf{x}$ in the support, but need not supply a uniform bound. The propensity score can approach $0$ or $1$ along a sequence of covariate values, and then the positivity assumption fails while overlap is complete. The first marked sentence is therefore false as a general statement; it is the contrapositive of the converse of Theorem 7.3.2. The sentence after the theorem names the wrong obstacle to that converse: what is missing is the uniform bound, not a measure-theoretic condition on supports. Note that the implication does hold in the case that the section goes on to illustrate, where $p_D(\\mathbf{x})$ equals $0$ or $1$ on part of the support (Example 7.3.3).", "what_survives": "Definition 7.3.1 and Theorem 7.3.2 stand, and so do the earlier identification results, which assume positivity. Example 7.3.3 with Figure 7.3.1 illustrates a propensity score equal to $0$ or $1$ on part of the support, where overlap does fail, and the empirical illustration in Example 7.3.4 with Figure 7.3.2 is unchanged. Restricting attention to $\\epsilon<p_D(\\mathbf{X}_i)<1-\\epsilon$, the discussion of extreme weights, Winsorization and Table 7.5.1 do not use the converse. The framing of Section 7.3 changes: the discussion of nonidentification on pp.~272--273 applies where one treatment arm is absent on part of the support, not to every failure of the uniform bound."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text. A red box marks the expression E[p D(X i) tau i] in the sentence 'For example, the estimand E[p D(X i) tau i], which weights by the propensity score itself, is equivalent to the average treatment effect on the treated (ATT), E[tau i given D i = 1].' The sentence before it says that one might choose an estimand that weights members of the population by some function of the propensity score.", "caption": "Section 7.3.1, p. 273 (PDF p. 291). The marked expectation is printed without a denominator and is said to be equivalent to the ATT.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T99", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T99"}, {"id": "P001-T100", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T100"}], "headline": "The propensity-weighted estimand that equals the ATT is $\\mathrm{E}[p_D(\\mathbf{X}_i)\\tau_i]/\\mathrm{E}[p_D(\\mathbf{X}_i)]$; the printed expression lacks the denominator.", "id": "P001-C07-E009", "location": "Section 7.3.1, p. 273 (PDF p. 291), second paragraph, sentence beginning “For example, the estimand”", "pdf_page": 291, "printed_page": "273", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T100", "P001-T99"], "text": {"as_printed": "``For example, the estimand $\\hl{\\mathrm{E}[p_D(\\mathbf{X}_i)\\tau_i]}$, which weights by the propensity score itself, is equivalent to the \\emph{average treatment effect on the treated (ATT)}, $\\mathrm{E}[\\tau_i\\mid D_i=1]$.''", "corrected": "``For example, the estimand $\\mathrm{E}[p_D(\\mathbf{X}_i)\\tau_i]\\hl{/\\mathrm{E}[p_D(\\mathbf{X}_i)]}$, which weights by the propensity score itself, is equivalent to the \\emph{average treatment effect on the treated (ATT)}, $\\mathrm{E}[\\tau_i\\mid D_i=1]$.''\n\nIn full: if $(Y_i(0),Y_i(1))\\perp\\!\\!\\!\\perp D_i\\mid\\mathbf{X}_i$, $\\operatorname{Supp}[D_i]=\\{0,1\\}$ and $\\mathrm{E}|\\tau_i|<\\infty$, then\n\\[\n\\mathrm{E}[\\tau_i\\mid D_i=1]=\\frac{\\mathrm{E}[p_D(\\mathbf{X}_i)\\tau_i]}{\\mathrm{E}[p_D(\\mathbf{X}_i)]},\\qquad \\mathrm{E}[p_D(\\mathbf{X}_i)]=\\Pr[D_i=1]>0.\n\\]", "evidence": "The evidence is a derivation and an exact counterexample. Set $q=\\Pr[D_i=1]$. By the Law of Iterated Expectations, I have $\\mathrm{E}[p_D(\\mathbf{X}_i)]=\\mathrm{E}[\\mathrm{E}[D_i\\mid\\mathbf{X}_i]]=q$. Under the conditional independence assumption, $\\tau_i$ and $D_i$ are independent given $\\mathbf{X}_i$, and I obtain\n\\[\n\\mathrm{E}[p_D(\\mathbf{X}_i)\\tau_i]=\\mathrm{E}\\bigl[\\mathrm{E}[D_i\\mid\\mathbf{X}_i]\\,\\mathrm{E}[\\tau_i\\mid\\mathbf{X}_i]\\bigr]=\\mathrm{E}[D_i\\tau_i]=q\\,\\mathrm{E}[\\tau_i\\mid D_i=1].\n\\]\nFor example, let $\\mathbf{X}_i$ be constant, $Y_i(0)=0$, $Y_i(1)=1$ and $q=1/4$. The printed expression equals $1/4$, and the ATT equals $1$. In a second example, $X_i$ takes the values $0$ and $1$ with probabilities $1/3$ and $2/3$, the propensity score is $p_D(0)=1/4$ and $p_D(1)=3/8$, and $\\tau_i$ equals $1$ when $X_i=0$ and $4$ when $X_i=1$. Then $q=1/3$, the printed expression equals $13/12$, and the ATT equals $13/4$.", "problem": "Under the conditional independence assumption, $\\mathrm{E}[p_D(\\mathbf{X}_i)\\tau_i]$ equals $\\Pr[D_i=1]$ times the ATT. The two are equal only when the ATT is zero, since $\\Pr[D_i=1]<1$. The neighboring sentences speak of weighting the members of a population and of the average effect among those who took up the treatment. The normalized weighted average is meant, and the printed formula omits the normalization.", "what_survives": "The ATT as a target, its identification under the conditional independence assumption and the weaker positivity condition, and the discussion of changing the target population are unchanged. I found no later theorem or calculation that uses this equality; Table 7.5.1 reports estimates of the ATE."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of ten lines from page 278, from the sentence The fact that random assignment is unlikely to hold does not mean that all hope is lost, to the first two lines of the next paragraph. A red outline in two parts surrounds the end of the first paragraph: by definition, D sub i must be conditionally independent of potential outcomes. The next paragraph begins: If we assume ignorability, then we again have a variety of estimators available.", "caption": "Section 7.5, p. 278 (PDF p. 296). The outline marks the conclusion that $D_i$ is conditionally independent of the potential outcomes “by definition”; it runs over a line break. The next paragraph begins “If we assume ignorability”.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E010.png"}, {"alt": "Line chart. The horizontal axis is the described gender of the applicant: female (D sub i equal to 0) and male (D sub i equal to 1). The vertical axis is the mean percentage of no votes, from 0 to 95. A blue line with round markers, for electorates of type 0, rises from 20 to 25; its markers are labelled with the probabilities 3/8 at female and 1/8 at male. An orange line with square markers, for electorates of type 1, rises from 60 to 65; its markers are labelled 1/8 at female and 3/8 at male. A dashed black line with diamond markers, for all electorates given X, rises from 30 to 55. Labels at the right read 60 to 65: difference 5; 30 to 55: difference 25, in bold; and 20 to 25: difference 5. The title reads: A male description adds 5 points in every electorate; the contrast given the coded descriptions is 25 points.", "caption": "The counterexample for P001-C07-E010. Mean percentage of “no” votes by described gender, for each type of electorate (solid lines) and for all electorates together, which is the comparison given $\\mathbf{X}_i$ (dashed line). The fraction beside each solid-line marker is the probability of that combination of electorate type and gender, and the area of that marker is proportional to it. The difference between male and female descriptions is $5$ points within each type of electorate and $25$ points given $\\mathbf{X}_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E010-counterexample.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T21", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T21"}, {"id": "P001-T22", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T22"}, {"id": "P001-T105", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T105"}], "headline": "Coding all the information in the applicant descriptions does not make gender conditionally independent of the potential outcomes by definition; conditional independence given $\\mathbf{X}_i$ remains a substantive assumption.", "id": "P001-C07-E010", "location": "Section 7.5, paragraph beginning “The fact that random assignment is unlikely to hold”, last sentence, p. 278 (PDF p. 296)", "pdf_page": 296, "printed_page": "278", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T105", "P001-T21", "P001-T22"], "text": {"as_printed": "This is a setting where the ignorability assumption, implying $(Y_i(0), Y_i(1)) \\mathbin{\\perp\\!\\!\\!\\perp} D_i \\mid \\mathbf{X}_i$, is quite plausible. As long as voters do not directly know the applicant, the applicant description contains \\emph{all} of the information available to voters. Assuming that the researchers' coding of the applicant descriptions---that is, the variables in $\\mathbf{X}_i$---does indeed account for all the information presented to the voters, then, \\hl{\\emph{by definition}, $D_i$ must be} conditionally independent of potential outcomes.", "corrected": "This is a setting where the ignorability assumption, implying $(Y_i(0), Y_i(1)) \\mathbin{\\perp\\!\\!\\!\\perp} D_i \\mid \\mathbf{X}_i$, is quite plausible. As long as voters do not directly know the applicant, the applicant description contains \\emph{all} of the information available to voters. \\hl{Recording all information about applicants presented to voters can address selection associated with those characteristics, but does not establish ignorability by definition. The additional assumption is that $(Y_i(0),Y_i(1)) \\mathbin{\\perp\\!\\!\\!\\perp} D_i \\mid \\mathbf{X}_i$. Differences in electorate preferences or decision context can violate this assumption even if voters do not personally know the applicants.}", "evidence": "The evidence is an exact counterexample in which both premises hold and conditional independence fails. All descriptions are identical apart from the gender shown, and no voter knows an applicant. Then $\\mathbf{X}_i$ is constant and records all non-gender applicant information; gender is recorded separately as $D_i$. Electorates are of two types, $U_i \\in \\{0,1\\}$, each with probability $1/2$. An electorate of type $u$ returns $Y_i(d) = 20 + 40u + 5d$ per cent ``no'' votes on a description showing gender $d$, which implies that $Y_i(1) - Y_i(0) = 5$ for every referendum and that the ATE is $5$. Male applicants make up a quarter of the applications before electorates of type $0$ and three quarters of those before electorates of type $1$. The table shows the joint distribution.\n\\begin{audittable}{cccccc}{The counterexample: the joint distribution of electorate type, described gender and potential outcomes (percentage of ``no'' votes). The descriptions are otherwise identical, so $\\mathbf{X}_i$ is constant.}\nElectorate type $U_i$ & Gender $D_i$ & Probability & $Y_i(0)$ & $Y_i(1)$ & Observed $Y_i$ \\\\\n$0$ & $0$ (female) & $3/8$ & $20$ & $25$ & $20$ \\\\\n$0$ & $1$ (male) & $1/8$ & $20$ & $25$ & $25$ \\\\\n$1$ & $0$ (female) & $1/8$ & $60$ & $65$ & $60$ \\\\\n$1$ & $1$ (male) & $3/8$ & $60$ & $65$ & $65$ \\\\\n\\end{audittable}\nBy the table, I have $\\Pr[D_i = 1 \\mid \\mathbf{X}_i] = 1/8 + 3/8 = 1/2$, so positivity holds, and\n\\[\n\\begin{aligned}\n\\Pr\\bigl[(Y_i(0),Y_i(1)) = (60,65),\\ D_i = 1 \\mid \\mathbf{X}_i\\bigr] &= \\frac{3}{8} , \\\\\n\\Pr\\bigl[(Y_i(0),Y_i(1)) = (60,65) \\mid \\mathbf{X}_i\\bigr] \\, \\Pr[D_i = 1 \\mid \\mathbf{X}_i] &= \\frac{1}{2} \\cdot \\frac{1}{2} = \\frac{1}{4} .\n\\end{aligned}\n\\]\nSince the two probabilities differ, conditional independence given $\\mathbf{X}_i$ fails. The contrast that Theorem 7.1.13 equates with the ATE under strong ignorability is\n\\[\n\\mathrm{E}[Y_i \\mid D_i = 1, \\mathbf{X}_i] - \\mathrm{E}[Y_i \\mid D_i = 0, \\mathbf{X}_i] = 55 - 30 = 25 ,\n\\]\nwhich is $20$ points above the ATE of $5$. Conditioning on $U_i$ as well as $\\mathbf{X}_i$ restores conditional independence, and the standardized contrast is then $5$. The example settles the implication only. It does not show that ignorability fails in the Swiss data, and it determines neither the sign nor the size of any bias there.", "problem": "The last sentence derives conditional independence from two premises: voters do not know the applicants, and $\\mathbf{X}_i$ records all non-gender information in the applicant descriptions. The conclusion does not follow. The potential outcomes $Y_i(0)$ and $Y_i(1)$ are the ``no'' vote shares that the electorate of referendum $i$ would return if the same description carried a female or a male applicant. They depend on who is voting as well as on the description. Conditioning on the recorded applicant characteristics holds them fixed. It does not exclude that electorates more inclined to vote ``no'' also see a different proportion of male applicants among descriptions with the same $\\mathbf{X}_i$. The covariates listed on p.~277 include the decade of the referendum and not the municipality. Conditional independence is a substantive assumption here, as the book says in general on p.~248 (``we recommend only invoking strong ignorability when justifiable by substantive knowledge''), and the passage presents it as true by definition. The causal reading of the six estimators in Table 7.5.1 that rest on ignorability (OLS linear and unpooled, matching, IPW, stabilized IPW and doubly robust) depends on this assumption.", "what_survives": "Definition 7.1.12 and the identification and estimation results that assume ignorability are untouched. So are the bounds earlier in the application, which do not use ignorability, and the random-assignment estimate, which rests on its own stated assumption. The correction changes no estimate or standard error in Table 7.5.1; I did not recompute them from the data, which were not available to me. The causal statements on pp.~279--280 are prefaced by ``If we accept the ignorability assumption'' and stand as conditional statements. What changes is that the assumption has to be argued for the electorates and the decision context as well as for the applicant descriptions."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E010", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of a paragraph of nine lines. A red box marks the clause 'the bias (and therefore MSE) of the estimator shrinks too slowly as n grows large, and, thus,' in the sentence 'For example, even if we have a consistent estimator of p D(X i), matching with estimated propensity scores is not generally root-n consistent ... the usual methods for constructing confidence intervals, including the bootstrap, do not generally work.' The sentence ends with the footnote mark 13.", "caption": "End of Section 7.2.4, p. 264 (PDF p. 282). The marked clause gives slowly shrinking bias as the reason that the bootstrap does not work for matching.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E011.png"}, {"alt": "Crop of footnote 20 and the first line of footnote 21. Footnote 20 reads: 'As with hot deck imputation, the bootstrap may not work as a method for computing valid confidence intervals for matching estimators since the bias of the estimator may converge very slowly (see, e.g., Abadie and Imbens 2008), but the bootstrap standard errors are likely to adequately characterize sampling variability.' A red box marks the clause that begins 'but the bootstrap standard errors'.", "caption": "Footnote 20, p. 279 (PDF p. 297). The marked clause is the assurance about the bootstrap standard errors of the matching estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E011-b.png"}, {"alt": "Crop of Table 7.5.1, 'Estimates for Effect of Gender on Percentage No Votes', with columns Estimator, Target, Estimate and SE and two rows for each of nine estimators. Red boxes mark the standard errors 1.951 and 1.988 in the Matching rows, whose estimates are 34.732 for E[Y i(0)] and 2.009 for E[tau i]. The OLS (Linear) rows have standard errors 0.487 and 0.508.", "caption": "Table 7.5.1, p. 280 (PDF p. 298). The two marked standard errors of the Matching row are the ones that rest on the assurance in footnote 20; neither is shown to be wrong.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E011-c.png"}, {"alt": "Chart with sample size n (500, 2000, 8000) on the horizontal axis and n times variance, from 2.5 to 3.2, on the vertical axis. Blue circles for estimates of the sampling variance lie at 3.059, 3.054 and 3.048, near a horizontal line labeled 'exact limit 3.046'. Orange squares for estimates of the expected bootstrap variance lie at 2.660, 2.652 and 2.648, near a horizontal line labeled 'exact limit 2.647, which is 0.869 of the true limit'. Bars show two Monte Carlo standard errors at each sample size; most are partly hidden by the markers. The title reads 'The bootstrap variance stays about 13 percent below the true variance'.", "caption": "Counterexample to the assurance in footnote 20: simulation estimates of $n$ times the sampling variance and expected bootstrap variance at $n=500$, $2000$ and $8000$, with the exact limits $3.046$ and $2.647$. The vertical axis starts at $2.5$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-E011-bootstrap-variance.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T15", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T15"}, {"id": "P001-T16", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T16"}], "headline": "For nearest-neighbor matching the bootstrap can fail through its variance even when the estimator has no bias, so the assurance that the bootstrap standard errors “are likely to adequately characterize sampling variability” is unsupported.", "id": "P001-C07-E011", "location": "End of Section 7.2.4, p. 264 (PDF p. 282), with footnote 13; footnote 20, p. 279 (PDF p. 297); Matching row of Table 7.5.1, p. 280 (PDF p. 298)", "pdf_page": 282, "printed_page": "264", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P001-T15", "P001-T16"], "text": {"as_printed": "P.~264: ``For example, even if we have a consistent estimator of $p_D(\\mathbf{X}_i)$, matching with estimated propensity scores is not generally root-$n$ consistent\\hl{---the bias (and therefore MSE) of the estimator shrinks ``too slowly'' as $n$ grows large---and, thus,} the usual methods for constructing confidence intervals, \\emph{including the bootstrap}, do not generally work.$^{13}$'' Footnote 13: ``See, e.g., Abadie and Imbens (2008).''\n\nP.~279, footnote 20: ``As with hot deck imputation, the bootstrap may not work as a method for computing valid confidence intervals for matching estimators since the bias of the estimator may converge very slowly (see, e.g., Abadie and Imbens 2008), \\hl{but the bootstrap standard errors are likely to adequately characterize sampling variability}.''\n\nP.~280, Table 7.5.1, Matching: $\\mathrm{E}[Y_i(0)]$, estimate $34.732$, SE $1.951$; $\\mathrm{E}[\\tau_i]$, estimate $2.009$, SE $1.988$.", "corrected": "P.~264: ``For example, a consistent estimator of $p_D(\\mathbf{X}_i)$ does not by itself guarantee that matching with estimated propensity scores is root-$n$ consistent. The bootstrap can also fail for nearest-neighbor matching when the estimator is root-$n$ consistent and has no bias, \\hl{because the bootstrap variance itself may be inconsistent}.$^{13}$ Under their \\hl{assumptions for parametric propensity-score estimation}, Abadie and Imbens (2016) give a root-$n$ normal approximation whose variance accounts for the estimation of the score.''\n\nP.~279, footnote 20: ``As with hot deck imputation, the bootstrap may not give valid confidence intervals \\hl{or standard errors} for matching estimators (see, e.g., Abadie and Imbens 2008). The bootstrap standard errors reported for matching \\hl{have not been justified for the matching and propensity-fitting procedure used here}, and they do not support comparisons of precision with the other estimators.''\n\nP.~280, Table 7.5.1, Matching: the estimates $34.732$ and $2.009$ are unchanged. The standard errors $1.951$ and $1.988$ are \\hl{bootstrap values without a demonstrated justification}; no replacement values are established.", "evidence": "The evidence is the paper cited in footnotes 13 and 20, an exact counterexample with a derivation, and a simulation of the counterexample. In its introduction, the cited paper describes the matching estimator that it studies as ``root-N consistent and asymptotically normal with no asymptotic bias'' (Abadie and Imbens 2008, p.~1537), and it states on the same page that ``the standard bootstrap fails to provide asymptotically valid standard errors''.\n\nIn the counterexample, $X_i$ is uniform on $[0,1/2]$, $D_i$ given $X_i=x$ is Bernoulli with parameter $1/4+x$, and $\\varepsilon_i$ equals $1$ or $-1$ with probability $1/2$ each, independently of $(X_i,D_i)$. Set $Y_i(0)=\\varepsilon_i$ and $Y_i(1)=0$, so that strong ignorability holds and $\\mathrm{E}[\\tau_i]=0$. The propensity score is estimated by $\\hat p_D(x)=\\hat a+x$, where $\\hat a$ is $\\overline{D}-\\overline{X}$ restricted to $[1/8,3/8]$. This estimator is consistent. Since $|\\hat p_D(x)-\\hat p_D(z)|=|x-z|$, the matches do not depend on $\\hat a$. The matching estimator of Section 7.2.4 (one-to-one, nearest-neighbor, with replacement) equals\n\\[\n\\hat\\theta_n=-\\frac{1}{n}\\sum_{i:D_i=0}(1+K_i)\\,\\varepsilon_i ,\n\\]\nwhere $K_i$ is the number of treated units matched to the control unit $i$. In samples or bootstrap resamples with an empty treatment arm, I impute the missing-arm outcomes as zero. Since the $\\varepsilon_i$ have mean zero and are independent of $(X_i,D_i)$, I have $\\mathrm{E}[\\hat\\theta_n]=0$ for all $n$. A computation with the spacings between neighboring covariate values gives\n\\[\n\\lim_{n\\to\\infty}n\\operatorname{Var}(\\hat\\theta_n)=3\\log 3-\\frac{1}{4}\\approx 3.046 ,\n\\]\nso that $\\hat\\theta_n$ is root-$n$ consistent. Let $V_n^*$ denote the bootstrap variance of $\\hat\\theta_n$ for the bootstrap of Section 3.4.3, which draws $n$ observations with replacement, refits the propensity score, and recomputes the matches. Then\n\\[\n\\lim_{n\\to\\infty}\\mathrm{E}[nV_n^*]=\\frac{1}{2}+\\frac{c_1}{2}+c_2\\Bigl(2\\log 3-\\frac{3}{2}\\Bigr)\\approx 2.647,\n\\]\nwhere\n\\[\nc_1=\\frac{2}{q}-\\frac{q}{2-q},\\qquad c_2=\\frac{3}{2q}-\\frac{4-q}{2(2-q)},\\qquad q=1-e^{-1}.\n\\]\nThe second limit is $0.869$ times the first. Since $nV_n^*\\ge 0$, Fatou's lemma implies that $nV_n^*$ does not converge in probability to $3\\log 3-1/4$. The bootstrap variance is inconsistent, although the estimator is unbiased and root-$n$ consistent. A simulation integrating out the $\\varepsilon_i$, with 400 independent samples at each sample size and 120 bootstrap resamples of each sample, agrees with the two limits.\n\\begin{audittable}{rrr}{Simulation of the counterexample: $n$ times the variance of the matching estimator and $n$ times the expected bootstrap variance, with Monte Carlo standard errors in parentheses, and the exact limits.}\n$n$ & true variance & bootstrap variance \\\\\n500 & 3.059 (0.013) & 2.660 (0.010) \\\\\n2000 & 3.054 (0.007) & 2.652 (0.005) \\\\\n8000 & 3.048 (0.003) & 2.648 (0.002) \\\\\nlimit & 3.046 & 2.647 \\\\\n\\end{audittable}", "problem": "Both passages attribute the failure of the bootstrap for nearest-neighbor matching to bias that shrinks too slowly. Footnote 20 then separates the confidence intervals, which may fail, from the bootstrap standard errors, which it expects to be adequate. That separation has no general justification. The paper cited in both footnotes concerns a matching estimator with a root-$n$ rate and no asymptotic bias, and for that estimator the standard bootstrap does not provide valid standard errors: the failure is in the variance, not in bias. Bias and rate therefore do not decide whether bootstrap standard errors for matching are valid. That needs an argument about the variance for the matching and propensity-fitting procedure in use, and the book gives none. The standard errors $1.951$ and $1.988$ in the Matching row of Table 7.5.1 are left without a stated justification. The same holds for the comparison on p.~279 that rests on them, that the standard error of matching ``is nearly 4 times as large as that associated with the OLS (linear) specification''.\n\nTwo points are not established. Neither standard error is shown to be wrong: the data and code of the application were not available, and the words ``may'' and ``likely'' make the footnote a judgment, not a theorem. The warning on p.~264 that a consistent estimator of the propensity score does not by itself ensure a root-$n$ rate also stands. Under their assumptions, Abadie and Imbens (2016) obtain a root-$n$ normal approximation for matching on a propensity score estimated by a regular parametric model, with a variance that accounts for the first estimation step, but their assumptions do not cover every consistent estimator.", "what_survives": "The matching estimates $34.732$ and $2.009$ in Table 7.5.1, the arithmetic of the comparisons on p.~279, the identification results, and the other rows of the table are not affected. The reading of the two standard errors in the Matching row changes, and with it the comparison of precision and its part in the remark that matching ``appears to perform abysmally''. No replacement values are established. The parallel sentence about hot deck imputation in Chapter 6 is the subject of \\fid{P001-C06-E007}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-E011", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three lines of text. A red box marks a garbled word, 'person' followed by a-circumflex, A-breve, Z-acute and 's', in the sentence 'The problem is that we cannot observe a person's counterfactual outcome.'", "caption": "Section 7.1, second paragraph, p. 236 (PDF p. 254). The marked word is printed with three accented letters where the apostrophe belongs.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "The word “person’s” is printed with three stray characters in place of the apostrophe.", "id": "P001-C07-N001", "location": "Section 7.1, second paragraph, p. 236 (PDF p. 254)", "pdf_page": 254, "printed_page": "236", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``The problem is that we cannot observe a person\\hl{âĂŹ}s counterfactual outcome.''", "corrected": "``The problem is that we cannot observe a person\\hl{'}s counterfactual outcome.''", "evidence": "The evidence is a check of p.~236.", "problem": "Three accented letters replace the apostrophe of ``person's'' on the printed page.", "what_survives": "The meaning is plain, and nothing depends on the misprint."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of text. Red boxes mark the italic words '(Ignorability and the Propensity Score)' in the sentence 'We will again prove the first two properties and omit a proof of the third; the proof follows nearly identically from Theorem 6.1.12 (Ignorability and the Propensity Score).'", "caption": "Proof of Theorem 7.1.16, first sentence, p. 251 (PDF p. 269). The marked title is quoted for Theorem 6.1.12.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "Theorem 6.1.12 is titled “MAR and the Propensity Score”; the proof of Theorem 7.1.16 cites it as “Ignorability and the Propensity Score”.", "id": "P001-C07-N002", "location": "Proof of Theorem 7.1.16, first sentence, p. 251 (PDF p. 269)", "pdf_page": 269, "printed_page": "251", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``We will again prove the first two properties and omit a proof of the third; the proof follows nearly identically from Theorem 6.1.12 (\\hl{\\emph{Ignorability and the Propensity Score}}).''", "corrected": "``We will again prove the first two properties and omit a proof of the third; the proof follows nearly identically from Theorem 6.1.12 (\\hl{\\emph{MAR and the Propensity Score}}).''", "evidence": "The evidence is a check of p.~251 against pp.~217 and 224.", "problem": "The theorem number is right and the title is not. Theorem 6.1.12 (p.~217) is titled ``MAR and the Propensity Score'', and Example 6.2.5 (p.~224) cites it under that title.", "what_survives": "The reference points to the intended theorem, and nothing depends on the title."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of two lines of text and a displayed formula. A red box marks the words 'Definition 7.1.13' in the sentence 'Plugging into Definition 7.1.13 (The ATE under Strong Ignorability), we have'. The formula reads: the sum over x of E[Y i | D i = 1, X i = x] Pr[X i = x], minus the sum over x of E[Y i | D i = 0, X i = x] Pr[X i = x], equals -1/4.", "caption": "Example 7.1.17, p. 253 (PDF p. 271), the sentence before the adjusted contrast. The marked words call item 7.1.13 a definition.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "Item 7.1.13 is a theorem: “Definition 7.1.13” should read “Theorem 7.1.13”.", "id": "P001-C07-N003", "location": "Example 7.1.17, p. 253 (PDF p. 271), sentence before the last display", "pdf_page": 271, "printed_page": "253", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Plugging into \\hl{Definition} 7.1.13 (\\emph{The ATE under Strong Ignorability}), we have''", "corrected": "``Plugging into \\hl{Theorem} 7.1.13 (\\emph{The ATE under Strong Ignorability}), we have''", "evidence": "The evidence is a check of p.~253 against p.~248 (PDF p.~266).", "problem": "The item numbered 7.1.13 (p.~248) is Theorem 7.1.13, \\emph{ATE under Strong Ignorability}, and its formula is the one used in the example.", "what_survives": "The number and the title identify the result, and nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N003", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of a displayed formula and the paragraph after it. The display writes E[tau i(d, d-prime)] as a sum over x of E[Y i given D i = d-prime, X i = x] Pr[X i = x] minus the same sum with d in place of d-prime. A red box marks the sentence 'Thus, under ignorability, for any d, d-prime in Supp[D i], E[tau i(d, d-prime)] is point identified.' The sentence before it says that E[Y i given D i = d-prime, X i = x], E[Y i given D i = d, X i = x] and Pr[X i = x] can be point identified because the random vector (Y i, D i, X i) is observed.", "caption": "Section 7.1.8, p. 255 (PDF p. 273). The marked sentence claims that $\\mathrm{E}[\\tau_i(d,d')]$ is point identified for every pair of doses in the support of $D_i$; the display above it is the generalized formula for the ATE.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N004.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T90", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T90"}, {"id": "P001-T91", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T91"}], "headline": "For a continuous treatment, ignorability and overlap identify fixed-dose mean effects if the conditional mean potential outcome is continuous in the dose.", "id": "P001-C07-N004", "location": "Section 7.1.8, p. 255 (PDF p. 273), paragraph after the first display, sentence beginning “Thus, under ignorability”", "pdf_page": 273, "printed_page": "255", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T90", "P001-T91"], "text": {"as_printed": "``Since we can observe the random vector $(Y_i,D_i,\\mathbf{X}_i)$, we can point identify $\\mathrm{E}[Y_i\\mid D_i=d',\\mathbf{X}_i=\\mathbf{x}]$, $\\mathrm{E}[Y_i\\mid D_i=d,\\mathbf{X}_i=\\mathbf{x}]$, and $\\Pr[\\mathbf{X}_i=\\mathbf{x}]$. Thus, under ignorability, \\hl{for any $d,d'\\in\\operatorname{Supp}[D_i]$}, $\\mathrm{E}[\\tau_i(d,d')]$ is point identified.''", "corrected": "``Since we can observe the random vector $(Y_i,D_i,\\mathbf{X}_i)$, we can point identify $\\Pr[\\mathbf{X}_i=\\mathbf{x}]$, \\hl{and} $\\mathrm{E}[Y_i\\mid D_i=d,\\mathbf{X}_i=\\mathbf{x}]$ \\hl{is identified as a function of $d$ up to sets of doses of conditional probability zero}. Thus, under ignorability, \\hl{if $d$ and $d'$ have positive conditional probability given each $\\mathbf{x}$, or if the mean of $Y_i(d)$ given $\\mathbf{X}_i=\\mathbf{x}$ is continuous in the dose on the common support of the treatment}, $\\mathrm{E}[\\tau_i(d,d')]$ is point identified.''\n\nIn full: let $\\mathbf{X}_i$ be discrete, let $Y_i=Y_i(D_i)$, let the process $(Y_i(d))_{d}$ be \\hl{a random function with jointly measurable evaluation}, independent of $D_i$ given $\\mathbf{X}_i$, and let the outcomes be integrable (bounded outcomes suffice). Let $S$ be a \\hl{Borel} set of doses such that the following two conditions hold for every $\\mathbf{x}$ with $\\Pr[\\mathbf{X}_i=\\mathbf{x}]>0$.\n\\begin{enumerate}\n\\item Every neighborhood in $S$ of every point of $S$ has positive probability under the conditional distribution of $D_i$ given $\\mathbf{X}_i=\\mathbf{x}$. This holds, for instance, when $S$ is an interval \\hl{of positive length} on which the conditional density of $D_i$ is positive.\n\\item The function $d\\mapsto\\mathrm{E}[Y_i(d)\\mid\\mathbf{X}_i=\\mathbf{x}]$ is continuous on $S$.\n\\end{enumerate}\nThen, for any $d,d'\\in S$,\n\\[\n\\mathrm{E}[\\tau_i(d,d')]=\\sum_{\\mathbf{x}}\\bigl(m_{\\mathbf{x}}(d')-m_{\\mathbf{x}}(d)\\bigr)\\Pr[\\mathbf{X}_i=\\mathbf{x}]\n\\]\nis point identified, where $m_{\\mathbf{x}}$ is the version of $d\\mapsto\\mathrm{E}[Y_i\\mid D_i=d,\\mathbf{X}_i=\\mathbf{x}]$ that is continuous on $S$. The second condition restricts the mean of the potential outcome: a continuous version of the observed regression does not suffice without it. The statement on p.~255 for $\\mathrm{E}[\\tau_i(d,d')\\mid\\mathbf{X}_i=\\mathbf{x}]$ needs the same two conditions.", "evidence": "The evidence is an exact counterexample, followed by a derivation of the corrected statement. Let $\\mathbf{X}_i$ be constant and let $D_i$ be uniform on $[0,1]$. Consider two models for the potential outcomes:\n\\[\n\\text{model A: } Y_i(d)=0 \\text{ for all } d,\\qquad \\text{model B: } Y_i(d)=\\mathbf{1}\\{d=1/2\\}.\n\\]\nIn each model the potential outcomes are not random, so that the whole process $(Y_i(d))_{d}$ is independent of $D_i$, and the density of $D_i$ equals $1$ on $[0,1]$. The observed outcome $Y_i=Y_i(D_i)$ is $0$ in model A. In model B it is $0$ except on the event $\\{D_i=1/2\\}$, which has probability zero. The joint CDF of $(Y_i,D_i,\\mathbf{X}_i)$ is the same in the two models, but\n\\[\n\\mathrm{E}[\\tau_i(0,1/2)]=0 \\text{ in model A},\\qquad \\mathrm{E}[\\tau_i(0,1/2)]=1 \\text{ in model B},\n\\]\nand the same two values hold for the doses $(1/4,1/2)$. By Definition 6.0.1, $\\mathrm{E}[\\tau_i(d,d')]$ is not point identified. The example does not need a degenerate outcome. If $Z_i$ is uniform on $[0,1]$ and independent of $D_i$, then the models $Y_i(d)=Z_i$ and $Y_i(d)=Z_i+\\mathbf{1}\\{d=1/2\\}$ have the same observable distribution, and $\\mathrm{E}[Y_i(1/2)]$ equals $1/2$ in the first and $3/2$ in the second.\n\nFor the corrected statement, let $S$ be the set of doses in that statement, and fix $\\mathbf{x}$ such that $\\Pr[\\mathbf{X}_i=\\mathbf{x}]>0$. Let $\\nu_{\\mathbf{x}}$ denote the conditional distribution of $D_i$ given $\\mathbf{X}_i=\\mathbf{x}$, and set $m_{\\mathbf{x}}(d)=\\mathrm{E}[Y_i(d)\\mid\\mathbf{X}_i=\\mathbf{x}]$. By the generalized potential outcomes model and the independence of the process $(Y_i(d))_{d}$ and $D_i$ given $\\mathbf{X}_i$, using jointly measurable evaluation, I have\n\\[\n\\mathrm{E}\\bigl[Y_i\\mathbf{1}\\{D_i\\in B\\}\\mid\\mathbf{X}_i=\\mathbf{x}\\bigr]=\\int_B m_{\\mathbf{x}}(d)\\,\\nu_{\\mathbf{x}}(\\mathrm{d}d)\n\\]\nfor every Borel set $B$. The left side depends only on the observable distribution. Two functions that are compatible with it in the role of $m_{\\mathbf{x}}$ then agree outside a set of $\\nu_{\\mathbf{x}}$-measure zero. Suppose that both are continuous on $S$ and differ at a dose $d_0\\in S$ by $\\delta>0$. Then there exists a neighborhood of $d_0$ in $S$ on which they differ by more than $\\delta/2$. By the first condition of the corrected statement, this neighborhood has positive $\\nu_{\\mathbf{x}}$-measure, which is a contradiction. I conclude that $m_{\\mathbf{x}}(d_0)$ is determined by the observable distribution. Summing $(m_{\\mathbf{x}}(d')-m_{\\mathbf{x}}(d))\\Pr[\\mathbf{X}_i=\\mathbf{x}]$ over $\\mathbf{x}$, I obtain $\\mathrm{E}[\\tau_i(d,d')]$.", "problem": "Section 7.1.8 allows the treatment $D_i$ to be continuous. Every single treatment level, which I call a dose, then has probability zero, and the distribution of $(Y_i,D_i,\\mathbf{X}_i)$ determines the function $d\\mapsto\\mathrm{E}[Y_i\\mid D_i=d,\\mathbf{X}_i=\\mathbf{x}]$ only up to a set of doses of probability zero. The value of this function at one specified dose is not determined by the observable distribution. Ignorability does not determine it either, even when the treatment has a positive density at that dose. The claim holds for a dose that has positive conditional probability, and it holds when the mean of $Y_i(d)$ given $\\mathbf{X}_i=\\mathbf{x}$ is continuous in $d$. The section states no such condition. Note that it assumes that $Y_i(d)$ is differentiable in $d$ only afterwards, for the average marginal causal effect (AMCE).", "what_survives": "Theorem 7.1.13 for a binary treatment, the estimators of Section 7.2, and the application in Section 7.5 with Table 7.5.1 do not use this claim. The counterexample does not contradict the identity for the AMCE, which is stated under differentiability. The two statements on p.~255 that the unconditional and the conditional mean effects are point identified for any $d,d'\\in\\operatorname{Supp}[D_i]$ need the continuity condition, and so does the formula for $\\mathrm{E}[Y_i(d)]$ in Section 7.2.3 (p.~261) when the treatment is continuous."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N004", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading '7.2.3 Maximum Likelihood Plug-In Estimation of Causal Effects' and the first three lines of the section. A red box marks the parenthesis '(Plug-In Estimation and Conditional Expectation Functions)', which follows the words 'Using the tools developed in Section 5.2.4'.", "caption": "Opening of Section 7.2.3, p. 261 (PDF p. 279). The marked words are given as the title of Section 5.2.4.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "Section 5.2.4 is titled “Maximum Likelihood Plug-In Estimation”.", "id": "P001-C07-N005", "location": "Section 7.2.3, p. 261 (PDF p. 279), first sentence", "pdf_page": 279, "printed_page": "261", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Using the tools developed in Section 5.2.4 (\\hl{\\emph{Plug-In Estimation and Conditional Expectation Functions}}), it is possible to directly wed the causal identification assumptions here to the parametric models discussed in Chapter 5.''", "corrected": "``Using the tools developed in Section 5.2.4 (\\hl{\\emph{Maximum Likelihood Plug-In Estimation}}), it is possible to directly wed the causal identification assumptions here to the parametric models discussed in Chapter 5.''", "evidence": "The evidence is a check of the heading of Section 5.2.4 on p.~197 and of the table of contents. The printed title occurs nowhere else in the book.", "problem": "The title in parentheses is not the title of Section 5.2.4, which is ``Maximum Likelihood Plug-In Estimation'' (p.~197). The section number is right.", "what_survives": "The reference points to the right section, and nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N005", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the last four lines of the page. The text reads: 'In the above example, we used Y 1(1) = 2 to impute Y 2(1) because P 1 = 0.33 is closest to P 2 = 0.14. But P 4 = 0.35 is almost as close to P 1, so Y 4(1) = 10 probably'. A red box marks the second P 1.", "caption": "Foot of p. 263 (PDF p. 281), paragraph after Example 7.2.4. The marked symbol names unit 1 where the comparison is with unit 2.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N006.png"}], "formal_release": null, "formal_statements": [{"id": "P001-T95", "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-T95"}], "headline": "The propensity score of unit 4 is compared with that of unit 2, whose outcome is imputed, not with that of unit 1.", "id": "P001-C07-N006", "location": "Paragraph after Example 7.2.4, p. 263 (PDF p. 281), last line; the sentence continues on p. 264", "pdf_page": 281, "printed_page": "263", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P001-T95"], "text": {"as_printed": "``In the above example, we used $Y_1(1)=2$ to impute $Y_2(1)$ because $P_1=0.33$ is closest to $P_2=0.14$. But $P_4=0.35$ is almost as close to $\\hl{P_1}$, so $Y_4(1)=10$ probably also gives us some information about $\\mathrm{E}[Y_i(1)\\mid p_D(\\mathbf{X}_i)=0.14]$.''", "corrected": "``But $P_4=0.35$ is almost as close to $\\hl{P_2}$, so $Y_4(1)=10$ probably also gives us some information about $\\mathrm{E}[Y_i(1)\\mid p_D(\\mathbf{X}_i)=0.14]$.''\n\nHere $P_j$ denotes $p_D(\\mathbf{X}_j)$.", "evidence": "The evidence is a recalculation from the propensity scores in the table of Example 7.2.4, which are $0.33$, $0.14$ and $0.35$ for units 1, 2 and 4.", "problem": "The outcome being imputed is that of unit 2. The distance that matters is the distance from $P_2=0.14$, which is $0.19$ for unit 1 and $0.21$ for unit 4. The distance between $P_4$ and $P_1$ is $0.02$, which is a different comparison. The symbols $P_1$, $P_2$ and $P_4$ are not introduced: the tables of Example 7.2.4 write $p_D(\\mathbf{X}_i)$.", "what_survives": "The matching example, its donors and its estimate $7/6$ are unchanged, and so is the point that nearest-neighbor matching does not use the information in the other units."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N006", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of footnote 14, five lines. It says that placebo tests should not be confused with balance tests after matching or inverse probability weighting, and ends: 'By Theorem 7.1.16 (Strong Ignorability and the Propensity Score), if we have the right propensity score specification, we know that the balancing property ensures that we will have statistical independence of D i and X i.' A red box marks the words 'statistical independence of D i and X i'.", "caption": "Footnote 14, p. 270 (PDF p. 288). The marked words state the independence of $D_i$ and $\\mathbf{X}_i$ without the conditioning on the propensity score.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The balancing property makes $D_i$ and $\\mathbf{X}_i$ independent conditional on the propensity score $p_D(\\mathbf{X}_i)$, not unconditionally.", "id": "P001-C07-N007", "location": "Footnote 14, p. 270 (PDF p. 288), last sentence", "pdf_page": 288, "printed_page": "270", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``By Theorem 7.1.16 (\\emph{Strong Ignorability and the Propensity Score}), if we have the right propensity score specification, we know that the ``balancing property'' ensures that we will have \\hl{statistical independence of $D_i$ and $\\mathbf{X}_i$}.''", "corrected": "``By Theorem 7.1.16 (\\emph{Strong Ignorability and the Propensity Score}), if we have the right propensity score specification, we know that the ``balancing property'' ensures that we will have statistical independence of $D_i$ and $\\mathbf{X}_i$ \\hl{conditional on $p_D(\\mathbf{X}_i)$}.''", "intended_meaning": "The footnote contrasts placebo tests with balance tests after matching or inverse probability weighting. In that setting the independence meant is the one that Theorem 7.1.16 states, $D_i\\perp\\!\\!\\!\\perp\\mathbf{X}_i\\mid p_D(\\mathbf{X}_i)$: among units with the same propensity score, treatment is independent of the covariates. For weighting, the corresponding statement is that $D_i$ and $\\mathbf{X}_i$ are independent in the population reweighted by the inverse probability weights of the true propensity score. The sentence does not mean independence in the original population, which fails when the propensity score varies with $\\mathbf{X}_i$. For example, let $X_i$ be Bernoulli with parameter $1/2$, and let $\\Pr[D_i=1\\mid X_i]$ equal $1/4$ when $X_i=0$ and $3/4$ when $X_i=1$; then $D_i$ and $X_i$ are dependent. The theorem concerns the population and does not ensure exact balance in each matched or weighted sample."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N007", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines from page 273. A red outline surrounds the words variation in X sub i, at the end of the sentence that begins We could even define alternative populations. The line above the sentence ends treatment assignments along all salient dimensions. The lines below restrict attention to observations whose propensity score lies between epsilon and one minus epsilon, for a value of epsilon in the interval from 0 to .5.", "caption": "Section 7.3.1, second paragraph, p. 273 (PDF p. 291). The outline marks “variation in $\\mathbf{X}_i$”. The line above speaks of variation in treatment assignments, and the next sentence trims on the propensity score $p_D(\\mathbf{X}_i)$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "In Section 7.3.1, the covariate profiles to keep are those with variation in the treatment $D_i$, not in the covariates $\\mathbf{X}_i$.", "id": "P001-C07-N008", "location": "Section 7.3.1, second paragraph, first sentence, p. 273 (PDF p. 291)", "pdf_page": 291, "printed_page": "273", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We could even define alternative populations from the population under study, and condition on the types of observations that have some probability of having variation in $\\hl{\\mathbf{X}_i}$.", "corrected": "We could even define alternative populations from the population under study, and condition on the types of observations that have some probability of having variation in $\\hl{D_i}$.", "evidence": "The evidence is a check of the page and a small example: if $\\mathbf{X}_i$ is constant and $D_i$ is Bernoulli with parameter $1/2$, then $p_D(\\mathbf{X}_i) = 1/2$, and positivity holds although $\\mathbf{X}_i$ does not vary.", "problem": "The sentence names the wrong variable. A type of observation is a covariate profile, and overlap at a profile requires that both values of the treatment $D_i$ occur there with positive probability, whether or not $\\mathbf{X}_i$ varies. The preceding paragraph ends with ``variation in treatment assignments along all salient dimensions'', and the next sentence restricts attention to $\\epsilon < p_D(\\mathbf{X}_i) < 1-\\epsilon$, which is a condition on the probability of treatment.", "what_survives": "The trimming rule $\\epsilon < p_D(\\mathbf{X}_i) < 1-\\epsilon$, the positivity of the retained subpopulation and the reasons given for changing the target population are unaffected; only the variable named in this sentence changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N008", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of nine lines from page 273, with three red outlines. The first surrounds epsilon in (0, .5) at the start of the second line. The second surrounds the abbreviation CIA in the sentence The ATT is nonparametrically identified under the CIA and a weaker positivity assumption. The third surrounds the condition that follows it: p sub D of X sub i is less than epsilon, which is less than 1.", "caption": "Section 7.3.1, second paragraph, p. 273 (PDF p. 291). The lower outlines mark the abbreviation CIA and the condition $p_D(\\mathbf{X}_i) < \\epsilon < 1$ in the sentence on the ATT. The upper outline marks the earlier use of $\\epsilon$, as a trimming margin in $(0,.5)$.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "In the positivity condition for the ATT, $\\epsilon$ is a new upper bound below one and not the trimming margin introduced two sentences earlier; CIA abbreviates the conditional independence assumption.", "id": "P001-C07-N009", "location": "Section 7.3.1, second paragraph, sentence beginning “The ATT is nonparametrically identified”, p. 273 (PDF p. 291)", "pdf_page": 291, "printed_page": "273", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The ATT is nonparametrically identified under the \\hl{CIA} and a weaker positivity assumption, $p_D(\\mathbf{X}_i) < \\hl{\\epsilon} < 1$.", "corrected": "The ATT is nonparametrically identified under the \\hl{conditional independence assumption (CIA)} and a weaker positivity assumption: \\hl{for some $\\delta > 0$, $p_D(\\mathbf{X}_i) < 1 - \\delta$ on the treated covariate support}.", "intended_meaning": "CIA is the conditional independence assumption of Definition 7.1.12; the abbreviation occurs only twice in the book, both times on this page, and is never expanded (see \\fid{P001-BACK-N002}). The condition bounds the propensity score above by a constant less than one, so that control units occur at every covariate profile of the treated. This is sufficient for the ATT: under conditional independence, $\\mathrm{E}[Y_i(0) \\mid D_i = 1]$ is the average of $\\mathrm{E}[Y_i \\mid D_i = 0, \\mathbf{X}_i = x]$ over the distribution of $\\mathbf{X}_i$ among the treated, which requires no lower bound on the propensity score. Two sentences earlier, $\\epsilon \\in (0,.5)$ is the trimming margin in $\\epsilon < p_D(\\mathbf{X}_i) < 1-\\epsilon$. With that value carried over, the printed condition is not the earlier one with its lower bound removed: for $\\epsilon = 1/4$, a constant propensity score of $3/5$ satisfies the two-sided condition and fails $p_D(\\mathbf{X}_i) < \\epsilon$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N009", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of ten lines from page 279, with two red outlines, each running over a line break. The upper one surrounds the words With the exception of the (potentially pathological) matching and IPW estimators. The lower one surrounds the words no estimated effect of gender that approaches traditional levels of statistical significance. The crop ends in the middle of that sentence, at the foot of the main text of the page.", "caption": "Section 7.5, p. 279 (PDF p. 297). The lower outline marks the statement that no estimated effect “approaches traditional levels of statistical significance”. The upper outline marks the exception for matching and IPW at the head of the paragraph. The sentence continues on p. 280.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N012.png"}], "formal_release": null, "formal_statements": [], "headline": "The statement that no estimate approaches significance covers the four ignorability-based estimates that remain once matching and IPW are set aside; none of them is significant at the 5% level.", "id": "P001-C07-N012", "location": "Section 7.5, paragraph beginning “Table 7.5.1 summarizes these estimates”, sentence beginning “After conditioning”, pp. 279–280 (PDF pp. 297–298)", "pdf_page": 297, "printed_page": "279–280", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "After conditioning on their observable differences from women, we find \\hl{no estimated effect of gender that approaches traditional levels of statistical significance}, though our estimates still suggest that---all else equal---being male decreases voter support for one's immigration application.", "corrected": "After conditioning on their observable differences from women \\hl{and setting aside the potentially pathological matching and IPW estimates, none of the remaining estimates is statistically significant at the 5\\% level}, though our estimates still suggest that---all else equal---being male decreases voter support for one's immigration application.", "intended_meaning": "The paragraph begins by excepting ``the (potentially pathological) matching and IPW estimators'', and the sentence summarizes the four estimates that remain: OLS (Linear), OLS (Unpooled), stabilized IPW and doubly robust. Their two-tailed normal $p$-values, computed from the printed estimates and standard errors, are $0.165$, $0.076$, $0.675$ and $0.094$. None is significant at the level $0.05$ named on p.~278, and all four estimates are positive. With that scope the sentence is correct, although ``approaches'' has no stated cutoff. Taken over all six adjusted rows it would not be, since for the IPW row ($3.731$, standard error $1.088$) I find $p = 0.0006$."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N012", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Table 7.5.1, Estimates for Effect of Gender on Percentage No Votes. The columns are Estimator, Target, Estimate and SE, and there are two rows for each of nine estimators: Lower Bound Plug-In, Upper Bound Plug-In, Random Assignment, OLS (Linear), OLS (Unpooled), Matching, IPW, Stabilized IPW and Doubly Robust. A red outline surrounds the SE column, whose eighteen entries run from 0.388 to 1.988. The note under the table describes the two targets and cites Hainmueller and Hangartner 2013; it does not mention the standard errors.", "caption": "Table 7.5.1, p. 280 (PDF p. 298). The outline marks the SE column. Neither the table nor the note under it says how the standard errors were computed.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-C07-N013.png"}], "formal_release": null, "formal_statements": [], "headline": "Section 7.5 does not say how the standard errors in Table 7.5.1 and the two $p$-values in the text were computed.", "id": "P001-C07-N013", "location": "Table 7.5.1, SE column, p. 280 (PDF p. 298); footnote 20, p. 279 (PDF p. 297); the $p$-values on pp. 278–279", "pdf_page": 298, "printed_page": "280", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{llrr}{Table 7.5.1 (p.~280): the column headings and four of the eighteen rows.}\nEstimator & Target & Estimate & \\hl{SE} \\\\\nRandom Assignment & $\\mathrm{E}[Y_i(0)]$ & 35.780 & 0.557 \\\\\n & $\\mathrm{E}[\\tau_i]$ & 1.335 & 0.685 \\\\\nOLS (Linear) & $\\mathrm{E}[Y_i(0)]$ & 36.206 & 0.487 \\\\\n & $\\mathrm{E}[\\tau_i]$ & 0.706 & 0.508 \\\\\n\\end{audittable}\n\n\\emph{Text, p.~278.} \\ldots\\ we have a point estimate of $\\widehat{\\mathrm{E}}[\\tau_i] = 1.335$, with a two-tailed $p$-value (against the null hypothesis that $\\mathrm{E}[\\tau_i] = 0$) of 0.051\\ldots\n\n\\emph{Text, p.~279.} \\ldots\\ as the $p$-value against the null hypothesis of zero ATE is 0.165.\n\n\\emph{Footnote 20, p.~279.} \\ldots\\ but the bootstrap standard errors are likely to adequately characterize sampling variability.", "corrected": "I would add a note under Table 7.5.1 stating how the standard errors were computed (the resampling scheme, the number of replications, and whether the regressions, the propensity scores and the Winsorization were recomputed in each replicate) and which reference distribution the $p$-values in the text use. These details must come from the original analysis; they are not supplied here.", "intended_meaning": "The chapter's discussion of inference points to the bootstrap: for the IPW estimators it prescribes ``re-estimating the propensity scores in each bootstrap sample'' (p.~267), and footnote 20 speaks of bootstrap standard errors. The two $p$-values agree with the two-tailed normal approximation applied to the printed estimates and standard errors: the ratios $1.335/0.685$ and $0.706/0.508$ give $0.051$ and $0.165$. I find no error in these reported values. The application leaves out the resampling scheme, the number of replications, whether the propensity scores and the Winsorization were recomputed in each replicate, and the reference distribution of the $p$-values."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-C07-N013", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of text from the Introduction. A red box marks the words 'all common abbreviations' in the sentence 'In addition, we provide a Glossary of Common Abbreviations, which gives the meanings of all common abbreviations used in this book'.", "caption": "Introduction, p. xviii (PDF p. 18): the sentence that describes the Glossary of Common Abbreviations, with “all common abbreviations” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-FRONT-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "The Introduction describes the Glossary of Common Abbreviations as covering all common abbreviations, although it lists only some of them.", "id": "P001-FRONT-N001", "location": "Introduction, p. xviii (PDF p. 18), last sentence of the first paragraph", "pdf_page": 18, "printed_page": "xviii", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~xviii.} In addition, we provide a Glossary of Common Abbreviations, which gives the meanings of \\hl{all common abbreviations} used in this book, along with references to where they are defined in the text.", "corrected": "\\emph{p.~xviii.} In addition, we provide a Glossary of Common Abbreviations, which gives the meanings of \\hl{selected} common abbreviations used in this book, along with references to where they are defined in the text.", "evidence": "A check of the pages, with a count of the abbreviations that stand alone in the text of pp.~3--281: CI has 18 uses, APD 19 and PDA 14, against 10 for NRCM, 12 for CMT and 19 for KL. Which abbreviations count as common is a judgment, not a computed fact.", "problem": "The glossary on pp.~286--287 has 18 entries. The text of the book uses 17 abbreviations that the glossary does not list (CI, APD, PDA, ATT, ATC, PATE, ACE, AMCE, MAE, IQR, ECDF, LOESS, ANOVA, CIA, MKLD, and the estimator labels DM and SIPW), and CI, APD and PDA occur about as often as the listed NRCM, CMT and KL.", "what_survives": "No result is affected. This entry corrects only the description of the glossary on p.~xviii. The missing glossary entries and the expansion of CIA are treated in \\fid{P001-BACK-N002}; the wording on p.~202 is treated in \\fid{P001-C05-N006}."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-FRONT-N001", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Eight lines of the list of names in the acknowledgments. Red boxes mark 'Josh Kalla', printed between Anand Gupta and Sarah Hamerling, and 'Tchetgen-Tchetgen' in 'Eric Tchetgen-Tchetgen'.", "caption": "Introduction, p. xviii (PDF p. 18): the list of names in the acknowledgments, with the two names marked.", "url": "https://pmaronow.github.io/llm-audit/works/P001/exhibits/P001-FRONT-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "In the acknowledgments, “Tchetgen-Tchetgen” should be “Tchetgen Tchetgen”, and Josh Kalla is out of alphabetical order.", "id": "P001-FRONT-N002", "location": "Introduction, “Who helped to write this book?”, p. xviii (PDF p. 18), list of names in the first paragraph", "pdf_page": 18, "printed_page": "xviii", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\ldots\\ Robin Gomila, Don Green, Anand Gupta, \\hl{Josh Kalla}, Sarah Hamerling, Erin Hartman, Jennifer Hill, \\ldots\\ Brandon Stewart, Eric \\hl{Tchetgen-Tchetgen}, Dustin Tingley, Teppei Yamamoto, \\ldots", "corrected": "\\ldots\\ Robin Gomila, Don Green, Anand Gupta, Sarah Hamerling, Erin Hartman, Jennifer Hill, Will Hunt, Jun Hwang, \\hl{Josh Kalla}, Donald Lee, \\ldots\\ Brandon Stewart, Eric \\hl{Tchetgen Tchetgen}, Dustin Tingley, Teppei Yamamoto, \\ldots", "evidence": "A check of the page, and a public source for the spelling: the faculty profile at the Wharton School of the University of Pennsylvania is headed ``Eric J. Tchetgen Tchetgen'' (\\url{https://statistics.wharton.upenn.edu/profile/ett/}).", "problem": "The surname is printed with a hyphen that it does not carry. The rest of the list is in alphabetical order by surname, and Kalla stands between Gupta and Hamerling.", "what_survives": "No one is misidentified, and nothing depends on either point."}, "url": "https://pmaronow.github.io/llm-audit/works/P001/#P001-FRONT-N002", "work": {"arxiv_id": null, "doi": "10.1017/9781316831762", "title": "Foundations of Agnostic Statistics", "url": "https://pmaronow.github.io/llm-audit/works/P001/"}, "work_id": "P001"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Theorem 1 and Remark 3. Red boxes mark '= 1 minus alpha.' at the end of the displayed limit in Theorem 1, the words 'is equal to 1 minus alpha' in the first sentence of Remark 3, and the words 'is exactly 1 minus alpha' in its last sentence.", "caption": "Theorem 1 and Remark 3, p. 8 (PDF p. 8). The marked terms are the equality in the theorem and the two phrases of the remark that assert it.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E001.png"}, {"alt": "Crop of Theorem 2. A red box marks '= 1 minus alpha.' at the end of the displayed limit of the infimum of coverage over the parameter space Xi n.", "caption": "Theorem 2, p. 11 (PDF p. 11). The marked term is the equality with $1-\\alpha$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T4", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T4"}, {"id": "P002-T5", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T5"}, {"id": "P002-T29", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T29"}, {"id": "P002-T30", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T30"}], "headline": "The limiting infimum of coverage in Theorems 1 and 2 is at least $1-\\alpha$; it does not equal $1-\\alpha$ for every admitted choice of the constants.", "id": "P002-E001", "location": "Theorem 1 and Remark 3, p. 8 (PDF p. 8); Theorem 2, p. 11 (PDF p. 11)", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T29", "P002-T30", "P002-T4", "P002-T5"], "text": {"as_printed": "Theorem 1. Fix $\\alpha\\in(0,1)$, $\\delta<1$, $r\\in(0,1/2]$ and $A>0$. Let $\\Theta_n(\\delta,r,A)$ be the parameter space of models that satisfy Assumptions 1--4 with these constants. Then\n\\[\n\\lim_{n\\to\\infty}\\,\\inf_{\\beta\\in\\mathbb R}\\,\\inf_{\\{\\theta\\in\\Theta_n(\\delta,r,A):\\,\\bar\\beta(\\theta)=\\beta\\}}\\mathbb P_\\theta\\{\\beta\\in I^*_{1-\\alpha}(Z)\\}\\;\\hl{=}\\;1-\\alpha.\n\\]\n\nRemark 3. The infimum of the coverage rates of the confidence sets \\hl{is equal to} $1-\\alpha$ because some constant-additive-effect models belong to the set $\\Theta_n(\\delta,r,A)$. In the design-based setting, the variance is not identified, and a conservative variance bound must instead be estimated (Aronow et al., 2014; Imbens \\& Rubin, 2015). The asymptotic coverage rate is bounded below by $1-\\alpha$ for many data-generating processes, and \\hl{is exactly} $1-\\alpha$ when the data are generated under a constant-additive-effect model.\n\nTheorem 2. Fix $k\\in\\mathbb N$, $\\alpha\\in(0,1)$, $r\\in(0,1/2]$, $\\delta<1$ and $\\tilde\\varepsilon,A,B>0$. Let $\\Xi_n(k,\\delta,\\tilde\\varepsilon,A,B,r)$ be the parameter space of models that satisfy Assumptions 1, 2, 4 and 5 with these constants. Then\n\\[\n\\lim_{n\\to\\infty}\\,\\inf_{\\beta\\in\\mathbb R}\\,\\inf_{\\{\\theta\\in\\Xi_n(k,\\delta,\\tilde\\varepsilon,A,B,r):\\,\\bar\\beta(\\theta)=\\beta\\}}\\mathbb P_\\theta\\{\\beta\\in I^{\\mathrm{adj},*}_{1-\\alpha}(Z)\\}\\;\\hl{=}\\;1-\\alpha.\n\\]", "corrected": "Theorem 1. Fix $\\alpha\\in(0,1)$, $\\delta<1$, $r\\in(0,1/2]$ and $A>0$. Let $\\Theta_n(\\delta,r,A)$ be the parameter space of models that satisfy Assumptions 1--4 with these constants. Then, \\hl{along the sample sizes $n$ for which $\\Theta_n(\\delta,r,A)$ is nonempty, provided these sample sizes are unbounded,}\n\\[\n\\hl{\\liminf_{n\\to\\infty}}\\,\\inf_{\\beta\\in\\mathbb R}\\,\\inf_{\\{\\theta\\in\\Theta_n(\\delta,r,A):\\,\\bar\\beta(\\theta)=\\beta\\}}\\mathbb P_\\theta\\{\\beta\\in I^*_{1-\\alpha}(Z)\\}\\;\\hl{\\ge}\\;1-\\alpha.\n\\]\n\nRemark 3. The \\hl{limiting} infimum of the coverage rates of the confidence sets \\hl{is at least} $1-\\alpha$. \\hl{It is equal to $1-\\alpha$ when the set $\\Theta_n(\\delta,r,A)$ contains a sequence of models whose coverage tends to $1-\\alpha$, for instance constant-additive-effect models with positive variances $\\sigma^2_{w(a)}$ that satisfy every assumption with the fixed constants. When the constants force $\\sigma^2_{w(a)}=0$, the coverage is one.} In the design-based setting, the variance is not identified, and a conservative variance bound must instead be estimated (Aronow et al., 2014; Imbens \\& Rubin, 2015).\n\nTheorem 2. Fix $k\\in\\mathbb N$, $\\alpha\\in(0,1)$, $r\\in(0,1/2]$, $\\delta<1$ and $\\tilde\\varepsilon,A,B>0$. Let $\\Xi_n(k,\\delta,\\tilde\\varepsilon,A,B,r)$ be the parameter space of models that satisfy Assumptions 1, 2, 4 and 5 with these constants. Then, \\hl{along the sample sizes $n$ for which this parameter space is nonempty, provided these sample sizes are unbounded,}\n\\[\n\\hl{\\liminf_{n\\to\\infty}}\\,\\inf_{\\beta\\in\\mathbb R}\\,\\inf_{\\{\\theta\\in\\Xi_n(k,\\delta,\\tilde\\varepsilon,A,B,r):\\,\\bar\\beta(\\theta)=\\beta\\}}\\mathbb P_\\theta\\{\\beta\\in I^{\\mathrm{adj},*}_{1-\\alpha}(Z)\\}\\;\\hl{\\ge}\\;1-\\alpha.\n\\]\n\nThe last two sentences of the proof of Theorem 1 (Supplement, p.~31), which pass from the inequality to the equality, are deleted. I do not give a condition on the constants of Theorem 2 under which equality holds.", "evidence": "The evidence is an exact counterexample to each equality, with a derivation that covers every model in the parameter space. Let $n\\ge2$ and $v\\in\\mathbb R^n$, and set $S_2=\\sum_i v_i^2$ and $S_4=\\sum_i v_i^4$. Then\n\\[\nnS_4-S_2^2=\\sum_{i<j}(v_i^2-v_j^2)^2\\ge0 .\n\\]\nIf $(S_4/n)^{1/4}\\le(1/2)\\{S_2/(n-1)\\}^{1/2}$, then $16(n-1)^2S_4\\le nS_2^2$. Combining the two inequalities, I obtain\n\\[\n\\{16(n-1)^2-n^2\\}S_2^2\\le0,\\qquad 16(n-1)^2-n^2=15(n-2)^2+28(n-2)+12>0,\n\\]\nwhich implies that $S_2=0$.\n\nFor Theorem 1, set $\\delta=0$, $r=1/4$ and $A=1/2$. By Assumption 3(ii) and the previous display with $v_i=w_i(a)-\\mu_{w(a)}$, I have $w_i(a)=c_a$ for all $i$ and each $a\\in\\{0,1\\}$. Under exclusion and monotonicity, $w_i(1)-w_i(0)$ equals $y_i(1)-y_i(0)-\\bar\\beta$ for a complier and $0$ for every other unit, so that $n(c_1-c_0)=\\sum_{i\\in\\mathcal C}\\{y_i(1)-y_i(0)\\}-|\\mathcal C|\\bar\\beta=0$. Then $Y_i-\\bar\\beta D_i$ is the same constant for every unit and every assignment. The observed statistic and every randomization statistic are $0/0=0$, the critical value is $0$, and $\\bar\\beta\\in I^*_{1-\\alpha}(Z)$. For every $\\beta\\in\\mathbb R$ and every $n\\ge2$, there exists a model in the parameter space such that $\\bar\\beta=\\beta$: all units are compliers, $y_i(0)=0$, $y_i(1)=\\beta$ and $n_1=\\lfloor n/2\\rfloor$. I conclude that the infimum in Theorem 1 is $1$ for every $n\\ge2$. Enumerating all assignments in rational arithmetic for $n=4,6,8$ (6, 20 and 70 assignments), I find coverage $1$ in each case.\n\nFor Theorem 2, set $k=1$, $\\delta=0$, $\\tilde\\varepsilon=1/4$, $A=4$, $B=1/2$ and $r=1/4$. The population regression residuals have mean zero. By Assumption 5(ii) and the same argument, they vanish, so that $w_i(a)=\\varphi_a+\\gamma_ax_i$. Since the covariates are centered, $\\varphi_a$ is the mean of $w(a)$, and the identity used for Theorem 1 implies that $\\varphi_1=\\varphi_0$. When the regression in each arm has full rank, the fitted residuals are zero and the adjusted numerator is the difference of the two fitted intercepts, which is $0$. Then $\\bar\\beta\\in I^{\\mathrm{adj},*}_{1-\\alpha}(Z)$, since the critical value is nonnegative. By Chebyshev's inequality for the sample mean and the sample second moment of the covariate, the probability that the regression in either arm is rank deficient is at most $1548/(n-1)$ for every model in the parameter space. For every $\\beta$ and every $n\\ge4$, there exists a model in the parameter space such that $\\bar\\beta=\\beta$: $x_i=(2i-n-1)/n$, all units are compliers, $y_i(0)=0$, $y_i(1)=\\beta$ and $n_1=\\lfloor n/2\\rfloor$. The infimum in Theorem 2 lies between $1-1548/(n-1)$ and $1$, and it tends to $1$.\n\nSeparately, Assumption 5(iv)--(v) imply that $\\tilde\\varepsilon\\le A^{1/2}$, so that $\\Xi_n$ is empty for some admitted constants (for example $A=4$ and $\\tilde\\varepsilon=3$).", "problem": "Both theorems assert that the smallest coverage over the parameter space converges to $1-\\alpha$ for every admitted choice of the constants. For some admitted choices the limit is $1$. Assumption 3(ii) bounds the standardized fourth moment of each vector $w(a)=(w_i(a))_{i=1}^n$, where $w_i(a)=y_i\\{d_i(a)\\}-\\bar\\beta d_i(a)$, by the constant $A$. Assumption 5(ii) bounds the standardized fourth moment of the population regression residuals by $B$. For a vector that is not constant, the standardized fourth moment is at least $\\{(n-1)/n\\}^{1/2}$. A constant below $1$ then admits only vectors with zero variance when $n$ is large. For $A=1/2$ this holds for every $n\\ge2$, and every model in $\\Theta_n(\\delta,r,1/2)$ has constant $w(1)$ and $w(0)$. The statistic at the true effect is $0/0$, which the Supplement (p.~23) sets to $0$, and the confidence set contains the true effect under every assignment. Coverage is $1$ for every model, and the infimum is $1$.\n\nThe proof of Theorem 1 ends (Supplement, p.~31) by passing from the lower bound to the equality, since ``a sequence of constant-additive-effect models belongs to our parameter set''. This step needs constant-additive-effect models with positive variances whose coverage tends to $1-\\alpha$, and for these constants the parameter space contains none. The last sentence of Remark 3 fails in the same way, since the models with $y_i(0)=0$ and $y_i(1)=\\beta$ have a constant additive effect and coverage $1$. Note that coverage is above $1-\\alpha$ in both counterexamples. They do not show that coverage can fall below $1-\\alpha$.", "what_survives": "This error does not affect the lower bound on coverage. The estimand, the confidence sets (7) and (11), their finite-sample validity under a constant additive effect, the algorithms and Theorem 3 do not use the equality. The simulations and the applications do not use it either, and none was recomputed for this point. Theorems 1 and 2 now state a lower bound, Remark 3 asserts equality only when a suitable sequence of models is exhibited, and Remark 5 extends the lower bound."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E001", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the statement of Lemma 5: the variance of tau-hat at beta n is at least (1 minus delta) times the sum of n0 over n n1 times S1 squared and n1 over n n0 times S0 squared. A red box marks the factor (1 minus delta).", "caption": "Supplement, Lemma 5, p. 23 (PDF p. 3). The marked factor is $1-\\delta$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E002.png"}, {"alt": "Crop of the chain of inequalities (C.3) to (C.6) and the two lines of text below it. Red boxes mark the coefficients delta n0 over n1 and delta n1 over n0 in line (C.4), and the phrase 'the elementary inequality 2ab less than or equal to a squared plus b squared' in the sentence that explains the steps.", "caption": "Supplement, proof of Lemma 5, p. 24 (PDF p. 4), steps (C.3)–(C.6). The marked coefficients of (C.4) come from multiplying $2ab\\le a^2+b^2$ by $\\delta$, and the marked phrase is the justification given for the step.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E002-b.png"}, {"alt": "Crop of the opening of Lemma 16 and its first claim: the variance of tau-tilde-adj is at least (1 minus delta) n0 over n1 n times script S1 squared plus (1 minus delta) n1 over n0 n times script S0 squared, numbered (E.10). Red boxes mark the two factors (1 minus delta).", "caption": "Supplement, Lemma 16, claim 1, bound (E.10), p. 34 (PDF p. 14). The marked factors are $1-\\delta$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E002-c.png"}, {"alt": "Crop of Theorem 1. A red box marks 'delta less than 1' in the list of constants 'Fix alpha in (0,1), delta less than 1, r in (0, 1/2] and A greater than 0'.", "caption": "Theorem 1, p. 8 (PDF p. 8). The marked condition is the only restriction on $\\delta$. Theorem 2 (p. 11) and Assumptions 3 and 5 state the same condition.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E002-d.png"}, {"alt": "Two panels, each with a horizontal axis of variance and three rows. Left, Lemma 5: the printed lower bound 65/4752 (about 0.0137; orange square) lies to the right of the exact variance 5/396 (about 0.0126; black diamond on a vertical line), and the corrected lower bound 65/7128 (about 0.0091; blue circle) lies to its left. Right, bound (E.10): printed lower bound 39/55 (about 0.7091), exact variance 36/55 (about 0.6545), corrected lower bound 26/55 (about 0.4727), in the same arrangement.", "caption": "Exact variance, printed lower bound and corrected lower bound for the population with $n=12$ and $\\delta=-1/2$ described in the Evidence section above, for Lemma 5 (left) and for (E.10) (right). In both panels the printed bound is above the exact variance and the corrected bound is below it. The variances are computed by enumerating all 66 assignments, and the lower bounds from the displayed formulas.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E002-bounds.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T1", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T1"}, {"id": "P002-T2", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T2"}, {"id": "P002-T3", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T3"}], "headline": "The variance lower bounds of Lemma 5 and (E.10) hold with the factor $1-\\max\\{\\delta,0\\}$; with the printed factor $1-\\delta$ they fail for negative $\\delta$, which Theorems 1 and 2 admit.", "id": "P002-E002", "location": "Supplement, Lemma 5, p. 23 (PDF p. 3), and its proof, steps (C.3)–(C.8), p. 24 (PDF p. 4); Lemma 16, claim 1, bound (E.10), p. 34 (PDF p. 14), and its proof, p. 35 (PDF p. 15)", "pdf_page": 3, "printed_page": "23", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T1", "P002-T2", "P002-T3"], "text": {"as_printed": "Lemma 5. For all $\\theta_n\\in\\Theta_n$, let $\\beta_n=\\bar\\beta(\\theta_n)$. Then\n\\[\n\\operatorname{var}\\bigl(\\widehat\\tau(\\beta_n)\\bigr)\\ge(1-\\hl{\\delta})\\left(\\frac{n_0}{nn_1}S_1^2(\\beta_n)+\\frac{n_1}{nn_0}S_0^2(\\beta_n)\\right).\n\\]\n\nProof of Lemma 5, p.~24:\n\\[\n\\begin{aligned}\n\\cdots&\\le\\sigma^2_{w(1),\\mathcal C}+\\sigma^2_{w(0),\\mathcal C}+2\\delta\\sigma_{w(1),\\mathcal C}\\cdot\\sigma_{w(0),\\mathcal C}&&\\text{(C.3)}\\\\\n&{}\\hl{\\le}\\;\\sigma^2_{w(1),\\mathcal C}+\\sigma^2_{w(0),\\mathcal C}+\\frac{\\delta n_0}{n_1}\\sigma^2_{w(1),\\mathcal C}+\\frac{\\delta n_1}{n_0}\\sigma^2_{w(0),\\mathcal C}&&\\text{(C.4)}\n\\end{aligned}\n\\]\nIn (C.3) we used Assumption 3-(i), in (C.4) we used the elementary inequality $2ab\\le a^2+b^2$, \\ldots\n\nLemma 16, claim 1. We have\n\\[\n\\operatorname{var}\\bigl(\\tilde\\tau_{\\mathrm{adj}}(\\beta_{n_k})\\bigr)\\ge(1-\\hl{\\delta})\\frac{n_0}{n_1n}\\mathcal S_1^2(\\beta_{n_k})+(1-\\hl{\\delta})\\frac{n_1}{n_0n}\\mathcal S_0^2(\\beta_{n_k}).\\qquad\\text{(E.10)}\n\\]", "corrected": "Let $\\delta_+=\\max\\{\\delta,0\\}$.\n\nLemma 5. For all $\\theta_n\\in\\Theta_n$, let $\\beta_n=\\bar\\beta(\\theta_n)$. Then\n\\[\n\\operatorname{var}\\bigl(\\widehat\\tau(\\beta_n)\\bigr)\\ge(1-\\hl{\\delta_+})\\left(\\frac{n_0}{nn_1}S_1^2(\\beta_n)+\\frac{n_1}{nn_0}S_0^2(\\beta_n)\\right).\n\\]\n\nLemma 16, claim 1. Consider a sequence $(\\theta_n)_{n=1}^\\infty$ such that $\\theta_n\\in\\Xi_n$ and let $\\beta_n=\\bar\\beta(\\theta_n)$. Let $(\\theta_{n_k})_{k=1}^\\infty$ be any of its subsequences. We have\n\\[\n\\operatorname{var}\\bigl(\\tilde\\tau_{\\mathrm{adj}}(\\beta_{n_k})\\bigr)\\ge(1-\\hl{\\delta_+})\\frac{n_0}{n_1n}\\mathcal S_1^2(\\beta_{n_k})+(1-\\hl{\\delta_+})\\frac{n_1}{n_0n}\\mathcal S_0^2(\\beta_{n_k}).\\qquad\\text{(E.10)}\n\\]\n\nIn the two proofs, $\\delta$ is replaced by $\\delta_+$ from (C.3) to (C.8) and throughout the proof of (E.10). The same replacement is made in the constants of the steps that use these bounds: Lemma 6 for Lemma 5, and (E.13) and (E.26)--(E.27) for (E.10). Alternatively, the two lemmas can be stated for $0\\le\\delta<1$ and applied with $\\delta=0$ to the models with $\\delta<0$.", "evidence": "The evidence is an exact counterexample and a derivation of the corrected bounds. Set $n=12$, $n_1=2$, $n_0=10$, $\\delta=-1/2$, $r=1/6$, $A=B=2$, $k=1$ and $\\tilde\\varepsilon=1/2$. Every unit is a complier, with $d_i(0)=0$, $d_i(1)=1$ and $y_i(1)=y_i(0)=y_i$, and the four pairs $(x_i,y_i)=(-1,-1),(-1,1),(1,-1),(1,1)$ each occur three times. Then $\\bar\\beta=0$ and $w_i(1)=w_i(0)=y_i$. The complier covariance and both complier variances are $12/11$, so that Assumption 3(i) reads $12/11\\ge6/11$. The remaining conditions of Assumptions 1--5 hold, the population regression coefficients are zero, and the residuals are $y_i$. Enumerating the 66 assignments in rational arithmetic, I obtain the following values.\n\\begin{audittable}{lrrr}{Population with $n=12$, $n_1=2$, $n_0=10$ and $\\delta=-1/2$: exact variance and the two lower bounds.}\nBound & Exact variance & Printed lower bound & Corrected lower bound \\\\\nLemma 5 & $5/396\\approx0.01263$ & $65/4752\\approx0.01368$ & $65/7128\\approx0.00912$ \\\\\n(E.10) & $36/55\\approx0.6545$ & $39/55\\approx0.7091$ & $26/55\\approx0.4727$ \\\\\n\\end{audittable}\nFor this population the right side of (C.3) is $12/11$ and the right side of (C.4) is $-36/55$. In the proof of (E.10), the left side of the first inequality is $0$ and its right side is $-36/5$. Repeating the four pairs $3m$ times, I obtain a population with $n=12m$ that satisfies the same conditions, and for it the right side of (E.10) is $13/12$ times the variance for every $m\\ge1$. The failure persists along a sequence with $n\\to\\infty$.\n\nFor the corrected bounds, set $\\delta_+=\\max\\{\\delta,0\\}$. Since standard deviations are nonnegative, Assumption 3(i) implies that $\\sigma_{w(1),w(0),\\mathcal C}\\ge-\\delta_+\\sigma_{w(1),\\mathcal C}\\sigma_{w(0),\\mathcal C}$. Since $\\delta_+\\ge0$ and both coefficients are at least $1$, steps (C.3)--(C.8) hold with $\\delta_+$ in place of $\\delta$, and I have\n\\[\nS_{10}^2(\\beta_n)\\le\\Bigl(1+\\frac{\\delta_+n_0}{n_1}\\Bigr)S_1^2(\\beta_n)+\\Bigl(1+\\frac{\\delta_+n_1}{n_0}\\Bigr)S_0^2(\\beta_n).\n\\]\nInserting this bound into (B.1), which reads $\\operatorname{var}(\\widehat\\tau(\\beta))=S_1^2(\\beta)/n_1+S_0^2(\\beta)/n_0-S_{10}^2(\\beta)/n$, I obtain\n\\[\n\\begin{aligned}\n\\operatorname{var}\\bigl(\\widehat\\tau(\\beta_n)\\bigr)&\\ge\\Bigl(\\frac1{n_1}-\\frac1n-\\frac{\\delta_+n_0}{nn_1}\\Bigr)S_1^2(\\beta_n)+\\Bigl(\\frac1{n_0}-\\frac1n-\\frac{\\delta_+n_1}{nn_0}\\Bigr)S_0^2(\\beta_n)\\\\\n&=(1-\\delta_+)\\Bigl(\\frac{n_0}{nn_1}S_1^2(\\beta_n)+\\frac{n_1}{nn_0}S_0^2(\\beta_n)\\Bigr).\n\\end{aligned}\n\\]\nIn the last equality, I used that $n=n_0+n_1$. The same computation with the regression residuals, Assumption 5(i) and (D.5) gives (E.10) with $\\delta_+$.", "problem": "Theorems 1 and 2 and Assumptions 3 and 5 require only that $\\delta<1$, and negative values are admitted. For $\\delta<0$, the covariance condition of Assumption 3(i), $\\sigma_{w(1),w(0),\\mathcal C}\\ge-\\delta\\sigma_{w(1),\\mathcal C}\\sigma_{w(0),\\mathcal C}$, says that the correlation of $w(1)$ and $w(0)$ among compliers is at least $-\\delta$ when both complier variances are positive. For $-1\\le\\delta<0$ every population with a constant additive effect satisfies it, since then $w_i(1)=w_i(0)$. For negative $\\delta$ both lower bounds can fail. The proof of Lemma 5 passes from (C.3) to (C.4) by multiplying the inequality $2ab\\le a^2+b^2$, with $a=(n_0/n_1)^{1/2}\\sigma_{w(1),\\mathcal C}$ and $b=(n_1/n_0)^{1/2}\\sigma_{w(0),\\mathcal C}$, by $\\delta$. The direction of the inequality is preserved only if $\\delta\\ge0$. Steps (C.6) and (C.8) also use that the coefficients $1+\\delta n_0/n_1$ and $1+\\delta n_1/n_0$ are nonnegative, which can fail for negative $\\delta$. The first inequality in the proof of (E.10), on p.~35, makes the same step for the regression residuals. When $\\delta<0$, the printed proofs of Theorems 1 and 2 rely on bounds that need not hold, and the printed right sides are $1-\\delta$ times the corrected right sides.", "what_survives": "Every model admitted by the original assumptions is still admitted, since a model that satisfies the covariance condition with a negative $\\delta$ also satisfies it with $\\delta=0$. The corrected factor $1-\\delta_+$ is positive, which is what the later steps of the two proofs use. This error does not by itself weaken the coverage conclusions of Theorems 1 and 2 (for the equality asserted there, see \\fid{P002-E001}). For $0\\le\\delta<1$ nothing changes. The repair leaves the confidence-set definitions, algorithms, simulations and applications unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E002", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Item 3 of Lemma 20 with display (E.60), whose square root contains var-star of tau-tilde-star-adj, then item 4 with display (E.61), a limit of P-star of the event that sigma-hat-squared-adj-star over var-star of tau-hat-star-adj minus 1 is at least kappa in absolute value. A red outline marks tau-hat in the denominator of (E.61). The proof paragraph follows and ends: The proof of (E.61) uses Lemma 19 and is similar to the proof of (E.18), so we omit it.", "caption": "Supplement, p. 42 (PDF p. 22 of the Supplement), items 3 and 4 of Lemma 20 and its proof. Outlined: $\\widehat\\tau$, printed with a hat in the denominator of (E.61); the line above, (E.60), has $\\tilde\\tau^*_{\\mathrm{adj}}$ under the square root.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T6", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T6"}, {"id": "P002-T7", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T7"}], "headline": "In (E.61) the denominator should be the reassignment variance of the infeasible statistic $\\tilde\\tau^*_{\\mathrm{adj}}$, as in (E.58)–(E.60), not that of the feasible statistic $\\widehat\\tau^*_{\\mathrm{adj}}$.", "id": "P002-E003", "location": "Supplement, Lemma 20, item 4, display (E.61), p. 42 (PDF p. 22 of the Supplement)", "pdf_page": 22, "printed_page": "42", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T6", "P002-T7"], "text": {"as_printed": "4. For every $\\kappa>0$,\n\\[\n\\lim_{k\\to\\infty}\\mathbb P^*\\left(\\left|\\frac{\\widehat\\sigma^2_{\\mathrm{adj},*}(\\beta_{n_k})}{\\operatorname{var}^*\\big(\\hl{\\widehat\\tau}^*_{\\mathrm{adj}}(\\beta_{n_k})\\big)}-1\\right|\\ge\\kappa\\right)=0,\\qquad\\text{(E.61)}\n\\]\nFrom the proof: ``The proof of (E.61) uses Lemma 19 and is similar to the proof of (E.18), so we omit it.''", "corrected": "Lemma~20, item~4. Let $(\\theta_n)_{n=1}^\\infty$ be a sequence of models with covariates such that $\\theta_n\\in\\Xi_n$ for all $n\\in\\mathbb N$, and denote $\\beta_n=\\bar\\beta(\\theta_n)$. Let $(\\theta_{n_k})_{k=1}^\\infty$ be any of its subsequences. Then, almost surely with respect to the randomness in $Z$, for every $\\kappa>0$,\n\\[\n\\lim_{k\\to\\infty}\\mathbb P^*\\left(\\left|\\frac{\\widehat\\sigma^2_{\\mathrm{adj},*}(\\beta_{n_k})}{\\operatorname{var}^*\\big(\\hl{\\tilde\\tau}^*_{\\mathrm{adj}}(\\beta_{n_k})\\big)}-1\\right|\\ge\\kappa\\right)=0.\\qquad\\text{(E.61)}\n\\]\nHere $\\Xi_n=\\Xi_n(k,\\delta,\\tilde\\varepsilon,A,B,r)$ is the parameter space of Theorem~2, and $\\tilde\\tau^*_{\\mathrm{adj}}$ is the infeasible statistic of (D.4) under reassignment, whose regression coefficients are those of the regression on the full observed population and are held fixed across reassignments. Its variance $\\operatorname{var}^*(\\tilde\\tau^*_{\\mathrm{adj}})$ is given by (D.6). The limit $k\\to\\infty$ is as printed.", "evidence": "The evidence is an exact counterexample to the printed statement and a derivation of the corrected one. Recall that $\\widehat\\sigma^2_{\\mathrm{adj},*}=RSS_1/n_1^2+RSS_0/n_0^2$, where $RSS_a$ is the residual sum of squares of the regression in arm $a$ after reassignment.\n\n\\emph{Counterexample.} Let $k=1$, $\\delta=0$, $\\tilde\\varepsilon=1/4$, $A=16$, $B=5$ and $r=1/3$. For even $n=2m\\ge8$, let $t_j=(2j-(m-1))/(m-1)$ for $j=0,\\ldots,m-1$, $\\eta_n=2^{-n^2}$ and $v_m=m^{-1}\\sum_j t_j^2$. The population has two units for each $j$, indexed by $s\\in\\{1,-1\\}$, with\n\\[\nx_{s,j}=s+\\eta_nt_j,\\qquad y_{s,j}(0)=y_{s,j}(1)=t_j+\\frac{1}{10}\\big(t_j^2-v_m\\big),\\qquad d_{s,j}(0)=0,\\qquad d_{s,j}(1)=1,\n\\]\nand $n_1=n_0=m$ under complete randomization. Every unit is a complier, the local average treatment effect is $\\beta=0$, Assumptions 1, 2, 4 and 5 hold with the constants above, and every arm regression has full rank. On the reassignment that treats the whole block $s=1$, both fitted slopes equal $1/\\eta_n$ and $\\widehat\\tau^*_{\\mathrm{adj}}=-2/\\eta_n$; on its complement, $\\widehat\\tau^*_{\\mathrm{adj}}=2/\\eta_n$. Exchanging the two arms changes the sign of $\\widehat\\tau^*_{\\mathrm{adj}}$, so its mean over reassignments is zero, and\n\\[\n\\operatorname{var}^*\\big(\\widehat\\tau^*_{\\mathrm{adj}}\\big)\\ge\\frac{8}{\\binom{n}{m}\\eta_n^2}\\ge8\\cdot2^{2n^2-n}.\n\\]\nSince the fit with zero intercept and zero slope is available in each arm and $|y_i|\\le11/10$, every reassignment has $\\widehat\\sigma^2_{\\mathrm{adj},*}\\le121/(25n)<5/n$. Combining the two bounds, I find that the ratio in (E.61) is at most $(5/(8n))\\,2^{n-2n^2}<1/2$ on every reassignment. This implies that\n\\[\n\\mathbb P^*\\left(\\left|\\frac{\\widehat\\sigma^2_{\\mathrm{adj},*}}{\\operatorname{var}^*\\big(\\widehat\\tau^*_{\\mathrm{adj}}\\big)}-1\\right|\\ge\\frac12\\right)=1\n\\]\nfor every even $n\\ge8$ and every original assignment, while (E.61) asserts that this probability tends to $0$.\n\n\\begin{audittable}{lrrrr}{Exact enumeration of the 70 reassignments at $n=8$ in the construction above, for three values of $\\eta$. The decimals are rounded displays of exact rational values.}\n$\\eta$ & $\\operatorname{var}^*(\\widehat\\tau^*_{\\mathrm{adj}})$ & Largest $\\widehat\\sigma^2_{\\mathrm{adj},*}$ & Largest ratio in (E.61) & $\\mathbb P^*(|\\text{ratio}-1|\\ge1/2)$ \\\\\n$1/100$ & $1143.27$ & $0.27776$ & $2.43\\times10^{-4}$ & $1$ \\\\\n$1/1000$ & $114286.13$ & $0.27778$ & $2.43\\times10^{-6}$ & $1$ \\\\\n$2^{-64}$ & $3.889\\times10^{37}$ & $0.27778$ & $7.14\\times10^{-39}$ & $1$ \\\\\n\\end{audittable}\n\n\\emph{Derivation of the corrected statement,} taking Lemma~19 as given. I write $e_i$ for the residuals of the regression on the full observed population, and I set $H=n^{-1}\\sum_{i=1}^ne_i^2$ and $V=\\operatorname{var}^*(\\tilde\\tau^*_{\\mathrm{adj}})$. By (D.6), I have $V=(1/n_1+1/n_0)\\,nH/(n-1)$. Let $Q_a$ and $h_a$ be the averages of $\\tilde x_i\\tilde x_i'$ and of $\\tilde x_ie_i$ over the units reassigned to arm $a$. When $Q_a$ is invertible, I obtain from the normal equations of least squares that\n\\[\n\\frac{RSS_a}{n_a}=\\frac{1}{n_a}\\sum_{i\\in\\mathcal T^*(a)}e_i^2-h_a'Q_a^{-1}h_a.\n\\]\nLet $Q=n^{-1}\\sum_{i=1}^n\\tilde x_i\\tilde x_i'$. Its smallest eigenvalue is at least $\\min\\{1,\\tilde\\varepsilon\\}$ by Assumption~5(iii) and~(v). Using Assumption~5(iv), the variance formula for sampling without replacement and the identity $\\sum_{i=1}^n\\tilde x_ie_i=0$, I find that $Q_a-Q\\to0$ and $\\|h_a\\|_2^2/H\\to0$ in $\\mathbb P^*$-probability. By Lemma~19 and Chebyshev's inequality, I have $n_a^{-1}\\sum_{i\\in\\mathcal T^*(a)}e_i^2/H\\to1$ in probability. Then $RSS_a/(n_aH)\\to1$ in probability for each $a\\in\\{0,1\\}$, and I conclude that\n\\[\n\\frac{\\widehat\\sigma^2_{\\mathrm{adj},*}}{V}=\\frac{n-1}{n}\\cdot\\frac{n_1^{-1}\\,RSS_1/(n_1H)+n_0^{-1}\\,RSS_0/(n_0H)}{n_1^{-1}+n_0^{-1}}\\longrightarrow1\n\\]\nin probability, which is the corrected (E.61).", "problem": "Lemma~20 concerns a sequence of models with covariates $\\theta_n\\in\\Xi_n$, and its claims hold almost surely with respect to the randomness in $Z$. The symbols $\\mathbb P^*$ and $\\operatorname{var}^*$ refer to reassignment. The feasible statistic $\\widehat\\tau^*_{\\mathrm{adj}}$ estimates the regression coefficients anew in each arm for every reassignment (equation (9) of the article). The infeasible statistic $\\tilde\\tau^*_{\\mathrm{adj}}$ of (D.4) holds them fixed at the coefficients of the regression on the full observed population, and its variance is stated in (D.6). Items 1--3 of the lemma, (E.58)--(E.60), are normalized by $\\sqrt{\\operatorname{var}^*(\\tilde\\tau^*_{\\mathrm{adj}})}$, and the proof of Theorem~2 needs item~4 on the same scale. As printed, item~4 asserts instead that $\\widehat\\sigma^2_{\\mathrm{adj},*}$ is consistent for the variance of the feasible statistic. This is a different claim, and it is false under the stated assumptions. Assumption~5 controls the feasible statistic in probability, but it does not control what rare reassignments with ill-conditioned arm regressions contribute to its second moment.", "what_survives": "With the tilde in the denominator, (E.60) and (E.61) are on the same scale, and Slutsky's theorem, applied conditionally, gives the normal critical values that this step of the proof of Theorem~2 uses. The correction adds no assumption and changes neither the test statistic, nor the confidence set, nor the coverage statement. Equations (9)--(11) of the article, (F.1)--(F.4) and Algorithms 3 and 4 use residual sums of squares and need no change. Theorem~1 and the unadjusted procedure do not use Lemma~20. The counterexample does not contradict the coverage conclusion of Theorem~2, since all its treatment effects are zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E003", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of two displays and the text between them. In the identity that begins with one over n1 times the sum of Z i times v-hat i squared, a red box marks the last term inside the large bracket, 'minus beta n mu d(1)'. In the numerator of the expectation in (C.17), a red box marks the same term.", "caption": "Supplement, proof of Lemma 7, p. 26 (PDF p. 6): the identity before (C.17), and (C.17). The marked term $-\\beta_n\\mu_{d(1)}$ should carry a plus sign in both displays.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T10", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T10"}, {"id": "P002-T11", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T11"}], "headline": "In the identity before (C.17) and in (C.17), the centering term is $-\\mu_{y(d(1))}+\\beta_n\\mu_{d(1)}$; the sign of $\\beta_n\\mu_{d(1)}$ is printed wrong.", "id": "P002-E004", "location": "Supplement, proof of Lemma 7, p. 26 (PDF p. 6): the identity before (C.17) and the numerator of (C.17)", "pdf_page": 6, "printed_page": "26", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T10", "P002-T11"], "text": {"as_printed": "We have the identity\n\\[\n\\frac1{n_1}\\sum_{i=1}^nZ_i\\widehat v_i^2(1)=\\frac1{n_1}\\sum_{i=1}^nZ_iv_i^2(1)-\\left(\\frac1{n_1}\\sum_{i=1}^nZ_i\\bigl(y_i(\\hl{d}(1))-\\beta_nd_i(1)\\bigr)-\\mu_{y(d(1))}\\;\\hl{-}\\;\\beta_n\\mu_{d(1)}\\right)^2,\n\\]\nwhich we use to expand the numerator in (C.16), creating two terms. The second term in this expansion converges to zero in probability, by Markov's inequality and the estimate\n\\[\n\\begin{aligned}\nE\\left[\\frac{\\bigl(n_1^{-1}\\sum_{i=1}^nZ_i\\bigl(y_i(\\hl{d}(1))-\\beta_nd_i(1)\\bigr)-\\mu_{y(d(1))}\\;\\hl{-}\\;\\beta_n\\mu_{d(1)}\\bigr)^2}{S_1^2(\\beta_n)}\\right]&=\\frac{n_0}{n_1n(n-1)}\\frac{\\sum_{i=1}^nv_i^2(1)}{S_1^2(\\beta_n)}\\\\\n&=O\\left(\\frac{n_0}{n_1n}\\right)=o(1).\\qquad\\text{(C.17)}\n\\end{aligned}\n\\]", "corrected": "We have the identity\n\\[\n\\frac1{n_1}\\sum_{i=1}^nZ_i\\widehat v_i^2(1)=\\frac1{n_1}\\sum_{i=1}^nZ_iv_i^2(1)-\\left(\\frac1{n_1}\\sum_{i=1}^nZ_i\\bigl(y_i(\\hl{d_i}(1))-\\beta_nd_i(1)\\bigr)-\\mu_{y(d(1))}\\;\\hl{+}\\;\\beta_n\\mu_{d(1)}\\right)^2,\n\\]\nand the estimate\n\\[\n\\begin{aligned}\nE\\left[\\frac{\\bigl(n_1^{-1}\\sum_{i=1}^nZ_i\\bigl(y_i(\\hl{d_i}(1))-\\beta_nd_i(1)\\bigr)-\\mu_{y(d(1))}\\;\\hl{+}\\;\\beta_n\\mu_{d(1)}\\bigr)^2}{S_1^2(\\beta_n)}\\right]&=\\frac{n_0}{n_1n(n-1)}\\frac{\\sum_{i=1}^nv_i^2(1)}{S_1^2(\\beta_n)}\\\\\n&=O\\left(\\frac{n_0}{n_1n}\\right)=o(1).\\qquad\\text{(C.17)}\n\\end{aligned}\n\\]\nHere $\\mu_{y(d(1))}$ and $\\mu_{d(1)}$ are the population means of $y_i(d_i(1))$ and $d_i(1)$.", "evidence": "The evidence is a derivation and an exact example. Set $\\mu_w=\\mu_{y(d(1))}-\\beta_n\\mu_{d(1)}$ and $\\bar w_1=n_1^{-1}\\sum_iZ_iw_i$. For a treated unit, $\\widehat v_i(1)=w_i-\\bar w_1$ by equation (3) of the article and $v_i(1)=w_i-\\mu_w$ by (B.3). Expanding the square, I obtain\n\\[\n\\frac1{n_1}\\sum_{i=1}^nZ_i\\widehat v_i^2(1)=\\frac1{n_1}\\sum_{i=1}^nZ_iv_i^2(1)-(\\bar w_1-\\mu_w)^2.\n\\]\nUnder complete randomization, I have $E[\\bar w_1]=\\mu_w$ and\n\\[\n\\operatorname{var}(\\bar w_1)=\\frac{n_0}{n_1n(n-1)}\\sum_{i=1}^nv_i^2(1),\n\\]\nwhich is the right side of (C.17) after division by $S_1^2(\\beta_n)$. For the example, set $n=4$, $n_1=2$ and $\\beta_n=2$, let all units be compliers with $y(0)=(-1,-1,1,1)$ and $y(1)=y(0)+2$, and treat the first two units. The left side of the identity is $0$. The printed right side is $1-(-5)^2=-24$, and the corrected right side is $1-(-1)^2=0$.", "problem": "The quantity that is centered is $w_i=y_i(d_i(1))-\\beta_nd_i(1)$, whose population mean is $\\mu_{y(d(1))}-\\beta_n\\mu_{d(1)}$. By (B.3), I have $v_i(1)=w_i-\\mu_{y(d(1))}+\\beta_n\\mu_{d(1)}$. The term subtracted in the bracket is then $\\mu_{y(d(1))}-\\beta_n\\mu_{d(1)}$. With the printed sign the bracket is the error of the treated sample mean minus $2\\beta_n\\mu_{d(1)}$. The identity is then false, and the expectation of the numerator in (C.17) exceeds the printed value by $4\\beta_n^2\\mu_{d(1)}^2$, which need not vanish. Both displays also print $y_i(d(1))$ for $y_i(d_i(1))$.", "what_survives": "With the corrected sign the identity holds and (C.17) holds with its printed right side. Lemma 7 and the rest of its proof are unchanged, and no assumption is added."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E004", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the end of the proof of Lemma 11 around display (C.31). One red box marks the fraction n 0 n 1 over n in the first line of the display. A second red box marks the whole second line, which reads: greater than or equal up to a constant to the squared difference of the arm means.", "caption": "Supplementary Material, p. 29 (PDF p. 9), display (C.31). The marked factor $n_0n_1/n$ belongs to the bound for $S_1^{*2}(\\beta_n)$ and cancels in the bound for the variance; the marked second line then lacks a factor $1/n$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T12", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T12"}, {"id": "P002-T13", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T13"}], "headline": "Display (C.31) omits the factor $1/n_1+1/n_0$ of (C.25): the lower bound for the randomization variance is $\\pi^2(1-\\pi)^2\\Delta_n^2/(n-1)$, which is of order $\\Delta_n^2/n$ and not of order $\\Delta_n^2$.", "id": "P002-E005", "location": "Supplementary Material, proof of Lemma 11, display (C.31), p. 29 (PDF p. 9 of the supplement)", "pdf_page": 9, "printed_page": "29", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T12", "P002-T13"], "text": {"as_printed": "Recalling (C.25), this shows that\n\\[\n\\begin{aligned}\n\\operatorname{var}^*(\\widehat\\tau^*(\\beta_n))&\\ge\\frac{\\pi^2(1-\\pi)^2}{n-1}\\,\\hl{\\frac{n_0n_1}{n}}\\,\\Big(\\widehat\\mu_{y(d(1))}-\\widehat\\mu_{y(d(0))}-\\beta_n\\big(\\widehat\\mu_{d(1)}-\\widehat\\mu_{d(0)}\\big)\\Big)^2\\\\\n&{}\\hl{\\gtrsim\\Big(\\widehat\\mu_{y(d(1))}-\\widehat\\mu_{y(d(0))}-\\beta_n\\big(\\widehat\\mu_{d(1)}-\\widehat\\mu_{d(0)}\\big)\\Big)^2}.\\qquad\\text{(C.31)}\n\\end{aligned}\n\\]", "corrected": "Recalling (C.25), this shows that\n\\[\n\\operatorname{var}^*(\\widehat\\tau^*(\\beta_n))\\ge\\frac{\\pi^2(1-\\pi)^2}{n-1}\\,\\Delta_n^2\\;\\hl{\\gtrsim\\frac1n}\\,\\Delta_n^2,\\qquad\\Delta_n=\\widehat\\mu_{y(d(1))}-\\widehat\\mu_{y(d(0))}-\\beta_n\\big(\\widehat\\mu_{d(1)}-\\widehat\\mu_{d(0)}\\big).\\qquad\\text{(C.31)}\n\\]\nThe first inequality holds for every $n\\ge2$ and every data set, with $\\pi=n_1/n$. The second uses Assumption 4.", "evidence": "The evidence is a derivation and an exact example. Let $w_i=Y_i-\\beta_nD_i$ and $\\bar w=n^{-1}\\sum_{i=1}^nw_i$. By (C.25), (C.26) and the decomposition on p.~29, I have\n\\[\n\\operatorname{var}^*(\\widehat\\tau^*(\\beta_n))=\\frac{\\pi^2(1-\\pi)^2}{n-1}\\Big(\\frac1{n_1}+\\frac1{n_0}\\Big)\\sum_{i=1}^n(w_i-\\bar w)^2\\ge\\frac{\\pi^2(1-\\pi)^2}{n-1}\\cdot\\frac{n}{n_0n_1}\\cdot\\frac{n_0n_1}{n}\\,\\Delta_n^2=\\frac{\\pi^2(1-\\pi)^2}{n-1}\\,\\Delta_n^2 .\n\\]\nSince Assumption 4 keeps $\\pi$ away from $0$ and $1$, the right-hand side is at least a positive constant times $\\Delta_n^2/n$. For the example, let $n=8$, $n_1=n_0=4$ and $\\beta_n=0$, with observed outcomes $(-1,-1,-1,-1,1,1,1,1)$, where the first four units are treated. Then $\\Delta_n=-2$ and $\\pi^2(1-\\pi)^2=1/16$. The variance of $\\widehat\\tau^*(0)$ over the $70$ assignments of four units to treatment is $1/28$. The right-hand side of the first inequality in (C.31) equals $1/14$, which exceeds the variance, and the corrected bound equals $1/28$.", "problem": "Let $\\Delta_n=\\widehat\\mu_{y(d(1))}-\\widehat\\mu_{y(d(0))}-\\beta_n(\\widehat\\mu_{d(1)}-\\widehat\\mu_{d(0)})$. The display before (C.31) decomposes the total sum of squares of $Y_i-\\beta_nD_i$ into the sums of squares within the two arms and the term $(n_0n_1/n)\\Delta_n^2$ between the arms. Dropping the sums within the arms bounds the quantity $S_1^{*2}(\\beta_n)$ of (C.26) from below by $\\pi^2(1-\\pi)^2(n_0n_1/n)\\Delta_n^2/(n-1)$, which is the right-hand side of the first inequality in (C.31). That side is therefore a lower bound for $S_1^{*2}(\\beta_n)$ and not for the variance. By (C.25) the variance is $S_1^{*2}(\\beta_n)(1/n_1+1/n_0)$, and $1/n_1+1/n_0=n/(n_0n_1)$ cancels the factor $n_0n_1/n$. The second bound in (C.31) is then too large by a factor of order $n$.", "what_survives": "Lemma 11 holds as stated. The estimate after (C.32) needs the between-arm contribution $\\Delta_n^2/\\{n^2\\operatorname{var}^*(\\widehat\\tau^*(\\beta_n))\\}$ to be $O(1/n)$. This holds by the corrected bound, and the last term displayed on p.~29 already has that order. Nothing else in the paper changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E005", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the statement of Lemma 12 and the two lines of its proof. A red box marks the numerator of the ratio, sigma hat star of beta n k, printed without a square, above the denominator var star of tau hat star of beta n k.", "caption": "Supplementary Material, p. 30 (PDF p. 10), Lemma 12. The marked numerator carries no square, while the denominator is a variance.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T14", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T14"}, {"id": "P002-T15", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T15"}], "headline": "The numerator of the ratio in Lemma 12 is the estimated variance $\\widehat\\sigma^{*2}(\\beta_{n_k})$, and not the estimated standard deviation $\\widehat\\sigma^{*}(\\beta_{n_k})$.", "id": "P002-E006", "location": "Supplementary Material, statement of Lemma 12, p. 30 (PDF p. 10 of the supplement)", "pdf_page": 10, "printed_page": "30", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T14", "P002-T15"], "text": {"as_printed": "Lemma 12. Consider a sequence $(\\theta_n)_{n=1}^\\infty$ such that $\\theta_n\\in\\Theta_n$. Let $\\beta_n=\\bar\\beta(\\theta_n)$. Let $(\\theta_{n_k})_{k=1}^\\infty$ be any of its subsequences. Then for every $\\kappa>0$, we have\n\\[\n\\mathbb P^*\\left(\\left|\\frac{\\hl{\\widehat\\sigma^*(\\beta_{n_k})}}{\\operatorname{var}^*(\\widehat\\tau^*(\\beta_{n_k}))}-1\\right|\\ge\\kappa\\right)=o_{a.s.}(1).\n\\]", "corrected": "Lemma 12. Consider a sequence $(\\theta_n)_{n=1}^\\infty$ such that $\\theta_n\\in\\Theta_n$. Let $\\beta_n=\\bar\\beta(\\theta_n)$. Let $(\\theta_{n_k})_{k=1}^\\infty$ be any of its subsequences. Then for every $\\kappa>0$, we have\n\\[\n\\mathbb P^*\\left(\\left|\\frac{\\hl{\\widehat\\sigma^{*2}(\\beta_{n_k})}}{\\operatorname{var}^*(\\widehat\\tau^*(\\beta_{n_k}))}-1\\right|\\ge\\kappa\\right)=o_{a.s.}(1).\n\\]", "evidence": "The evidence is an exact example in which the printed statement fails, and a derivation of the corrected one. Let $n$ be even and $n_1=n_0=n/2$. Suppose that every unit is a complier with $y_i(1)=y_i(0)$, that half of these outcomes equal $-1$ and half equal $1$, and that $\\beta_n=0$. Let $q$ be the mean outcome in a reassigned treatment group, so that the control mean is $-q$. Computing the three quantities directly, I find\n\\[\n\\operatorname{var}^*(\\widehat\\tau^*(0))=\\frac1{4(n-1)},\\qquad\\widehat\\sigma^{*2}(0)=\\frac{1-q^2}{4n},\\qquad E^*q^2=\\frac1{n-1}.\n\\]\nThe printed ratio equals $2(n-1)n^{-1/2}(1-q^2)^{1/2}$, which diverges in $\\mathbb P^*$-probability for every original assignment. The ratio with the squared numerator is $R=(n-1)(1-q^2)/n$. Since $0\\le R\\le1$ and $E^*(1-R)=2/n$, Markov's inequality implies that $\\mathbb P^*(|R-1|\\ge\\kappa)\\le2/(n\\kappa)$. With $\\kappa=1/2$ and $n=8,32,128,512$, the exact probability for the printed ratio equals $1$ up to rounding error in each case, and for the corrected ratio it is $0.029$, $4.9\\times10^{-5}$, $6.5\\times10^{-17}$ and $1.6\\times10^{-63}$.\n\nIn general, let $a_i=Y_i-\\beta_nD_i$ minus its mean, $S^2=(n-1)^{-1}\\sum_ia_i^2$ and $M^2=\\max_ia_i^2$. Lemma 11 implies that $M^2/(nS^2)\\to0$ for almost every sequence of original assignments. In a reassigned arm of size $m$, the mean of the $a_i^2$ has expectation $(n-1)S^2/n$ and variance at most a constant times $M^2S^2/n$, and the squared arm mean has expectation $O(S^2/n)$. By Chebyshev's and Markov's inequalities, the centred second moment in each arm divided by $S^2$ tends to $1$ in $\\mathbb P^*$-probability. Substituting into the definition of $\\widehat\\sigma^{*2}$ on p.~6 of the article and comparing with (C.25), I obtain the corrected statement.", "problem": "The ratio divides an estimated standard deviation by a variance. In the balanced, bounded-outcome example below, the denominator is of order $1/n$ and $\\widehat\\sigma^*$ is of order $n^{-1/2}$, so the printed ratio is of order $n^{1/2}$ and does not approach $1$. The sentence that introduces the lemma, the parallel statement in Lemma 7 and the use of the lemma to studentize in Lemma 13 all concern the ratio of the estimated variance to the variance.", "what_survives": "Lemma 13 and the studentized conditional central limit theorem use the ratio of variances and are unchanged. The correction needs no further assumption, and the coverage results are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E006", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of display (E.14) with the line before it and the two lines after it. A red box marks the equals sign between 'max over i of the squared norm of x tilde i' and '1 plus the sum over s of max over i of x s i squared'.", "caption": "Supplementary Material, p. 35 (PDF p. 15), display (E.14). The marked sign equates the maximum of $\\|\\tilde x_i\\|_2^2$ with a sum of coordinatewise maxima.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T16", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T16"}, {"id": "P002-T17", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T17"}], "headline": "The middle relation in (E.14) is an inequality: the maximum over units of a sum is at most the sum of the maxima, and equality can fail.", "id": "P002-E007", "location": "Supplementary Material, proof of Lemma 16, display (E.14), p. 35 (PDF p. 15 of the supplement)", "pdf_page": 15, "printed_page": "35", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T16", "P002-T17"], "text": {"as_printed": "For (E.11), for all $s\\in[k]$, Assumption 5-(iv) implies that\n\\[\n\\max_{i\\in[n]}|x_{si}|\\le(An)^{1/4},\\qquad\\max_{i\\in[n]}\\|\\tilde x_i\\|_2^2\\;\\hl{=}\\;1+\\sum_{s=1}^k\\max_{i\\in[n]}|x_{si}|^2\\lesssim n^{1/2},\\qquad\\text{(E.14)}\n\\]\nsince $k$ does not depend on $n$.", "corrected": "For (E.11), for all $s\\in[k]$, Assumption 5-(iv) implies that\n\\[\n\\max_{i\\in[n]}|x_{si}|\\le(An)^{1/4},\\qquad\\max_{i\\in[n]}\\|\\tilde x_i\\|_2^2\\;\\hl{\\le}\\;1+\\sum_{s=1}^k\\max_{i\\in[n]}|x_{si}|^2\\lesssim n^{1/2},\\qquad\\text{(E.14)}\n\\]\nsince $k$ does not depend on $n$.", "evidence": "The evidence is a derivation and an exact example. Since $\\tilde x_i=[1,x_i']'$, for every $i$ I have\n\\[\n\\|\\tilde x_i\\|_2^2=1+\\sum_{s=1}^k|x_{si}|^2\\le1+\\sum_{s=1}^k\\max_{j\\in[n]}|x_{sj}|^2,\n\\]\nand I obtain the inequality by taking the maximum over $i$. By Assumption 5-(iv), I have $\\max_i|x_{si}|\\le(An)^{1/4}$. The right-hand side is then at most $1+k(An)^{1/2}$, which is $O(n^{1/2})$ because $k$ is fixed. For the example, let $k=2$ and $n=4$, with the centred rows $(1,0)$, $(-1,0)$, $(0,1)$ and $(0,-1)$. The left-hand side equals $2$ and the right-hand side equals $3$.", "problem": "Since different coordinates can attain their maxima at different units, the two sides of the marked relation differ in general. Only the upper bound is used in the proof.", "what_survives": "The bound $\\max_i\\|\\tilde x_i\\|_2^2\\lesssim n^{1/2}$, claim (E.11) and Lemma 16 are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E007", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the start of the proof of claim (E.15) in Lemma 17: four lines of text, a display, two lines of text, and displays (E.21) to (E.23). Two red boxes mark the left-hand side 'tau hat adj of beta n minus tau tilde adj of beta n', once in the display before (E.21) and once in (E.21).", "caption": "Supplementary Material, p. 36 (PDF p. 16), the display before (E.21) and displays (E.21)–(E.23). The marked left-hand sides are the difference whose sign is opposite to that of the right-hand sides.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T18", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T18"}], "headline": "The difference $\\widehat\\tau_{\\mathrm{adj}}(\\beta_n)-\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)$ is the negative of the right-hand side printed before (E.21) and in (E.21)–(E.23); the right-hand sides equal $\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)-\\widehat\\tau_{\\mathrm{adj}}(\\beta_n)$.", "id": "P002-E008", "location": "Supplementary Material, proof of Lemma 17, the display before (E.21) and displays (E.21)–(E.23), p. 36 (PDF p. 16 of the supplement)", "pdf_page": 16, "printed_page": "36", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T18"], "text": {"as_printed": "We begin with (E.15). By the definitions of $\\widehat\\tau_{\\mathrm{adj}}(\\beta)$ and $\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)$,\n\\[\n\\hl{\\widehat\\tau_{\\mathrm{adj}}(\\beta_n)-\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)}=\\frac1{n_1}\\sum_{i=1}^nZ_ix_i'\\big(\\widehat\\gamma(1,\\beta_n)-\\gamma(1,\\beta_n)\\big)-\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)x_i'\\big(\\widehat\\gamma(0,\\beta_n)-\\gamma(0,\\beta_n)\\big).\n\\]\nRecall the notation $x_i^\\circ=[0,x_i']'\\in\\mathbb R^{k+1}$ and $\\tilde x_i=[1,x_i']'\\in\\mathbb R^{k+1}$. Using (E.2), the previous expression becomes\n\\[\n\\begin{aligned}\n&{}\\hl{\\widehat\\tau_{\\mathrm{adj}}(\\beta_n)-\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)}&&\\text{(E.21)}\\\\\n&\\quad=\\frac1{n_1}\\sum_{i=1}^nZ_ix_i^{\\circ\\prime}\\Big(\\frac1{n_1}\\sum_{i=1}^nZ_i\\tilde x_i\\tilde x_i'\\Big)^{-1}\\frac1{n_1}\\sum_{i=1}^nZ_i\\tilde x_i\\varepsilon_i(1,\\beta_n)&&\\text{(E.22)}\\\\\n&\\quad\\quad-\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)x_i^{\\circ\\prime}\\Big(\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\tilde x_i'\\Big)^{-1}\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\varepsilon_i(0,\\beta_n).&&\\text{(E.23)}\n\\end{aligned}\n\\]", "corrected": "We begin with (E.15). By the definitions of $\\widehat\\tau_{\\mathrm{adj}}(\\beta)$ and $\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)$,\n\\[\n\\hl{\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)-\\widehat\\tau_{\\mathrm{adj}}(\\beta_n)}=\\frac1{n_1}\\sum_{i=1}^nZ_ix_i'\\big(\\widehat\\gamma(1,\\beta_n)-\\gamma(1,\\beta_n)\\big)-\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)x_i'\\big(\\widehat\\gamma(0,\\beta_n)-\\gamma(0,\\beta_n)\\big).\n\\]\nRecall the notation $x_i^\\circ=[0,x_i']'\\in\\mathbb R^{k+1}$ and $\\tilde x_i=[1,x_i']'\\in\\mathbb R^{k+1}$. Using (E.2), the previous expression becomes\n\\[\n\\begin{aligned}\n&{}\\hl{\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)-\\widehat\\tau_{\\mathrm{adj}}(\\beta_n)}&&\\text{(E.21)}\\\\\n&\\quad=\\frac1{n_1}\\sum_{i=1}^nZ_ix_i^{\\circ\\prime}\\Big(\\frac1{n_1}\\sum_{i=1}^nZ_i\\tilde x_i\\tilde x_i'\\Big)^{-1}\\frac1{n_1}\\sum_{i=1}^nZ_i\\tilde x_i\\varepsilon_i(1,\\beta_n)&&\\text{(E.22)}\\\\\n&\\quad\\quad-\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)x_i^{\\circ\\prime}\\Big(\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\tilde x_i'\\Big)^{-1}\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\varepsilon_i(0,\\beta_n).&&\\text{(E.23)}\n\\end{aligned}\n\\]\nEquivalently, the left-hand sides can be kept as printed and every right-hand side negated.", "evidence": "The evidence is a derivation from (9) of the article and (D.4). Subtracting (D.4) from (9), the terms $Y_i-\\beta_nD_i$ cancel, and I obtain\n\\[\n\\widehat\\tau_{\\mathrm{adj}}(\\beta_n)-\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)=-\\frac1{n_1}\\sum_{i=1}^nZ_ix_i'\\big(\\widehat\\gamma(1,\\beta_n)-\\gamma(1,\\beta_n)\\big)+\\frac1{n_0}\\sum_{i=1}^n(1-Z_i)x_i'\\big(\\widehat\\gamma(0,\\beta_n)-\\gamma(0,\\beta_n)\\big).\n\\]\nEquations (E.2) and (E.3) express each slope error as the inverse of the arm design matrix applied to the sum of $\\tilde x_i\\varepsilon_i$ over the arm, with a positive sign. Substituting these expressions, I find (E.22) with a minus sign and (E.23) with a plus sign when the left-hand side is kept.", "problem": "Both statistics subtract the fitted covariate term from the adjusted outcome, $\\widehat\\tau_{\\mathrm{adj}}$ with the estimated slopes $\\widehat\\gamma$ and $\\tilde\\tau_{\\mathrm{adj}}$ with the population slopes $\\gamma$. Their difference carries a minus sign in the treated arm and a plus sign in the control arm, which is the opposite of what is printed. Substituting (E.2) and (E.3) does not change a sign, and (E.21)--(E.23) carry the same reversal.", "what_survives": "The proof uses only the absolute value of this difference, and claim (E.15) is stated for $|\\tilde\\tau_{\\mathrm{adj}}(\\beta_{n_k})-\\widehat\\tau_{\\mathrm{adj}}(\\beta_{n_k})|$. The bounds that follow, the other claims of Lemma 17 and the results on the confidence sets are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E008", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of display (E.41): a column vector with entries phi(a, beta_n) minus phi-hat(a, beta_n) and gamma(a, beta_n) minus gamma-hat(a, beta_n), set equal to the inverse of an arm average of x-tilde_i x-tilde_i-prime times an arm average of x-tilde_i epsilon(a, beta_n). Red boxes mark the equality sign and the term epsilon(a, beta_n).", "caption": "Supplementary Material, p. 39 (PDF p. 19 of the supplement): display (E.41). The right side, which begins after the marked equality sign, lacks a minus sign, and the marked residual $\\varepsilon(a,\\beta_n)$ lacks its index $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E009.png"}, {"alt": "Crop of the sentence 'Next, (E.39) is, after applying (E.41),' followed by a display: the number 2 times a product of three arm averages, the middle one inverted. A red box marks the leading 2.", "caption": "Supplementary Material, p. 40 (PDF p. 20 of the supplement): the display after (E.46), which evaluates the cross term (E.39). The marked coefficient $2$ should be $-2$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E009-b.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T19", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T19"}], "headline": "Two signs in the proof of Lemma 18 are wrong: the right side of (E.41) lacks a minus sign, and the cross term displayed after (E.46) has coefficient $-2$, not $2$; the residual in (E.41) also lacks its index $i$.", "id": "P002-E009", "location": "Supplementary Material, proof of Lemma 18: display (E.41), p. 39 (PDF p. 19 of the supplement), and the unnumbered display after (E.46), p. 40 (PDF p. 20 of the supplement)", "pdf_page": 19, "printed_page": "39", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T19"], "text": {"as_printed": "Recalling the definition of $\\tilde x_i$, we have by a representation analogous to (E.2) that, asymptotically almost surely,\n\\[\n\\begin{bmatrix}\\varphi(a,\\beta_n)-\\widehat\\varphi(a,\\beta_n)\\\\ \\gamma(a,\\beta_n)-\\widehat\\gamma(a,\\beta_n)\\end{bmatrix}\\hl{=}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'\\right)^{-1}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\hl{\\varepsilon}(a,\\beta_n)\\right).\\qquad\\text{(E.41)}\n\\]\nNext, (E.39) is, after applying (E.41),\n\\[\n\\hl{2}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\varepsilon_i(a,\\beta_n)\\tilde x_i'\\right)\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'\\right)^{-1}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\varepsilon_i(a,\\beta_n)\\right).\n\\]", "corrected": "Recalling the definition of $\\tilde x_i$, we have by a representation analogous to (E.2) that, asymptotically almost surely,\n\\[\n\\begin{bmatrix}\\varphi(a,\\beta_n)-\\widehat\\varphi(a,\\beta_n)\\\\ \\gamma(a,\\beta_n)-\\widehat\\gamma(a,\\beta_n)\\end{bmatrix}=\\hl{-}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'\\right)^{-1}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\hl{\\varepsilon_i}(a,\\beta_n)\\right).\\qquad\\text{(E.41)}\n\\]\nThe identity holds whenever the matrix $n_a^{-1}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'$ is invertible, which the proof shows to hold asymptotically almost surely.\n\nNext, (E.39) is, after applying (E.41),\n\\[\n\\hl{-2}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\varepsilon_i(a,\\beta_n)\\tilde x_i'\\right)\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'\\right)^{-1}\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\varepsilon_i(a,\\beta_n)\\right).\n\\]\nThe sign in (E.47) may be changed to match; its statement holds with either sign.", "evidence": "The evidence is a derivation from the least-squares normal equations. Fix an arm $a$ and write\n\\[\nG=\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i',\\qquad h=\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\varepsilon_i(a,\\beta_n).\n\\]\nFor $i\\in\\mathcal T(a)$ the regressand is $w_i(a,\\beta_n)=\\tilde x_i'\\psi+\\varepsilon_i(a,\\beta_n)$. The normal equations are\n\\[\n0=\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\bigl(w_i(a,\\beta_n)-\\tilde x_i'\\widehat\\psi\\bigr)=h-G(\\widehat\\psi-\\psi).\n\\]\nWhen $G$ is invertible, the normal equations imply that $\\widehat\\psi-\\psi=G^{-1}h$ and $\\psi-\\widehat\\psi=-G^{-1}h$, in agreement with (E.2)--(E.3). Since $x_i'(\\widehat\\gamma-\\gamma)+\\widehat\\varphi-\\varphi=\\tilde x_i'(\\widehat\\psi-\\psi)$, I find that the term (E.39) equals $-2h'(\\widehat\\psi-\\psi)=-2h'G^{-1}h$.\n\nThe positive sign in (E.47) is not a further error. That display states that a quantity is $o_{a.s.}(1)$, and a quantity is $o_{a.s.}(1)$ exactly when its negative is.", "problem": "Let $\\psi=(\\varphi,\\gamma')'$ denote the population coefficients and $\\widehat\\psi$ their least-squares estimates in arm $a$. The least-squares identity (E.2)--(E.3) of Lemma 14 gives the difference $\\widehat\\psi-\\psi$, estimate minus population. Display (E.41) prints the same right side for the difference $\\psi-\\widehat\\psi$, which has the opposite sign. The term (E.39) carries the factor $-2/n_a$. When the correct identity is substituted into it, the resulting product has coefficient $-2$, and the display after (E.46) prints $2$.", "what_survives": "The bounds (E.44), (E.46) and (E.47) and the conclusion (E.37) of Lemma 18 are unaffected, because they use only the fact that these quantities tend to zero. Only the two displays change."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E009", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The display that follows the words Then Assumption 5-(v): a quadratic form in the difference of psi-hat(a, beta-n) and psi-star(beta-n) is bounded below by epsilon-tilde plus an almost surely vanishing term, times a squared norm. Red outlines mark epsilon-tilde and the term psi-hat(a) inside the norm. Below, the text continues: Using (E.11), we find that (E.57) is bounded by, with a further display.", "caption": "Supplement, p. 42 (PDF p. 22 of the Supplement), first display, in the proof of Lemma 19. Outlined: the constant $\\tilde\\varepsilon$, and $\\widehat\\psi(a)$ printed without its argument $\\beta_n$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T8", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T8"}, {"id": "P002-T9", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T9"}], "headline": "The lower bound on the quadratic form holds with the constant $\\min\\{1,\\tilde\\varepsilon\\}$ in place of $\\tilde\\varepsilon$, because the matrix includes the intercept.", "id": "P002-E010", "location": "Supplement, proof of Lemma 19, p. 42 (PDF p. 22 of the Supplement), first display, after “Then Assumption 5-(v),”", "pdf_page": 22, "printed_page": "42", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T8", "P002-T9"], "text": {"as_printed": "Then Assumption 5-(v),\n\\[\n\\begin{aligned}\n&\\big[\\widehat\\psi(a,\\beta_n)-\\psi^*(\\beta_n)\\big]'\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'\\right)\\big[\\widehat\\psi(a,\\beta_n)-\\psi^*(\\beta_n)\\big]\\\\\n&\\qquad\\qquad\\ge\\big(\\hl{\\tilde\\varepsilon}+o_{a.s.}(1)\\big)\\,\\|\\hl{\\widehat\\psi(a)}-\\psi^*(\\beta_n)\\|_2^2.\n\\end{aligned}\n\\]", "corrected": "Then Assumption 5-(v),\n\\[\n\\begin{aligned}\n&\\big[\\widehat\\psi(a,\\beta_n)-\\psi^*(\\beta_n)\\big]'\\left(\\frac{1}{n_a}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'\\right)\\big[\\widehat\\psi(a,\\beta_n)-\\psi^*(\\beta_n)\\big]\\\\\n&\\qquad\\qquad\\ge\\big(\\hl{\\min\\{1,\\tilde\\varepsilon\\}}+o_{a.s.}(1)\\big)\\,\\|\\widehat\\psi(a\\hl{,\\beta_n})-\\psi^*(\\beta_n)\\|_2^2.\n\\end{aligned}\n\\]\nHere $a\\in\\{0,1\\}$, $\\tilde x_i=[1,x_i']'$, and $\\tilde\\varepsilon>0$ is the constant of Assumption~5(v), on which no upper bound is imposed. The bound uses Assumption~5(iii), $\\sum_{i=1}^nx_i=0_k$, and (E.40) together with Assumption~5(v).", "evidence": "The evidence is a derivation, with a counterexample to the printed step. Let $Q_n=n^{-1}\\sum_{i=1}^n x_ix_i'$. By Assumption~5(iii), I have $n^{-1}\\sum_{i=1}^n\\tilde x_i\\tilde x_i'=\\operatorname{diag}(1,Q_n)$, and $\\lambda_{\\min}(Q_n)\\ge\\tilde\\varepsilon$ by Assumption~5(v). The smallest eigenvalue of $\\operatorname{diag}(1,Q_n)$ is $\\min\\{1,\\lambda_{\\min}(Q_n)\\}$, which is at least $\\min\\{1,\\tilde\\varepsilon\\}$. By (E.40), the matrix $n_a^{-1}\\sum_{i\\in\\mathcal T(a)}\\tilde x_i\\tilde x_i'$ equals $\\operatorname{diag}(1,Q_n)+o_{a.s.}(1)$. Weyl's inequality then implies that its smallest eigenvalue is at least $\\min\\{1,\\tilde\\varepsilon\\}+o_{a.s.}(1)$, and the corrected bound follows for every vector in place of $\\widehat\\psi(a,\\beta_n)-\\psi^*(\\beta_n)$.\n\nFor the counterexample, suppose that there is a single covariate, whose values alternate between $2$ and $-2$, so that $Q_n=4$, and let $\\tilde\\varepsilon=2$ and $v=(1,0)'$. Then\n\\[\nv'\\operatorname{diag}(1,4)\\,v=1<2=\\tilde\\varepsilon\\,\\|v\\|_2^2.\n\\]\nThe matrix $\\operatorname{diag}(1,Q_n)$ does not satisfy the bound with the constant $\\tilde\\varepsilon$. This does not contradict Lemma~19.", "problem": "Here $\\tilde x_i=[1,x_i']'$ is the covariate vector with an intercept. Assumption~5(v) bounds below by $\\tilde\\varepsilon$ the smallest eigenvalue of $n^{-1}\\sum_{i=1}^n x_ix_i'$, the second-moment matrix of the covariates alone. Since the covariates are centred by Assumption~5(iii), the matrix $n^{-1}\\sum_{i=1}^n\\tilde x_i\\tilde x_i'$ has the additional eigenvalue $1$ from the intercept, and its smallest eigenvalue is bounded below by $\\min\\{1,\\tilde\\varepsilon\\}$. Assumption~5 allows $\\tilde\\varepsilon>1$, so the printed constant is justified only when $\\tilde\\varepsilon\\le1$. The norm on the right-hand side also omits the argument $\\beta_n$ of $\\widehat\\psi$.", "what_survives": "The step needs only a lower bound by a fixed positive constant, and $\\min\\{1,\\tilde\\varepsilon\\}$ is such a constant. The bound on (E.57), Lemma~19 and the results that rest on it hold under the assumptions as stated."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E010", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Algorithm 2 from 'Set e i equal to min of C i and record the pair of the interval from e i minus 1 to e i and m i' to Step 6. Red boxes mark, in Step 5, the pair 'interval from e k to e k plus 1, m k' and, in the next line, the interval 'e k to e k plus 1' at the end of the definition of CS k.", "caption": "Algorithm 2, end of Step 4 and Steps 5–6, p. 16. Step 4 records $\\{[e_{i-1},e_i],m_i\\}$ (first line of the crop); the marked pair and interval in Step 5 are shifted by one position.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E011.png"}, {"alt": "Crop of Steps 2 to 4 of Algorithm 4. A red box marks 'm i' in 'Step 3: Set e 0 equal to minus infinity and let m i be any element in I of beta 1 minus 1'.", "caption": "Supplementary Material, p. 44 (PDF p. 24), Algorithm 4, Step 3. The marked index is $m_i$, where Algorithm 2 has $m_1$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E011-b.png"}, {"alt": "Crop of the last two paragraphs of the proof of Theorem 3. Red boxes mark 'm k in I of beta for all beta in the interval from e k to V k', 'CS i equals CS star intersected with the interval from e i to e i plus 1', and the union sign with limits i equal to 0 and n I.", "caption": "Supplementary Material, p. 45 (PDF p. 25), Section H.2, last paragraph. The three marked expressions use the indexing of Step 5.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E011-c.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T22", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T22"}, {"id": "P002-T28", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T28"}], "headline": "Step 5 of Algorithms 2 and 4 and the proof of Theorem 3 in Section H.2 label the recorded pairs one position off: Step 4 records the pairs $([e_{i-1},e_i],m_i)$, and Step 3 of Algorithm 4 should set $m_1$.", "id": "P002-E011", "location": "Algorithm 2, Steps 3–6, p. 16; Supplementary Material, Algorithm 4, Steps 3 and 5, p. 44 (PDF p. 24 of the supplement), and Section H.2, last paragraph, p. 45 (PDF p. 25)", "pdf_page": 16, "printed_page": "16", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T22", "P002-T28"], "text": {"as_printed": "Algorithm 2, p.~16: ``Step 5: For each interval-index pair in $\\hl{([e_k,e_{k+1}],m_k)}$ recorded in the previous steps: Find $\\mathrm{CS}_k=\\{\\beta:|\\Delta^2(\\beta)|\\leqslant|\\Delta^{*2}_{m_k}(\\beta)|,\\ \\beta\\in\\hl{[e_k,e_{k+1}]}\\}$.'' Step 5 of Algorithm 4 has the same indexing.\n\nAlgorithm 4, Supplementary Material p.~44: ``Step 3: Set $e_0=-\\infty$ and let $\\hl{m_i}$ be any element in $\\mathcal I(\\beta_1-1)$.''\n\nSection H.2, Supplementary Material p.~45: ``For $k\\notin\\{0,n_I\\}$, we have by construction that $[e_k,e_{k+1}]=[e_k,V_k]\\cup[V_k,e_{k+1}]$. We have $\\hl{m_k\\in\\mathcal I(\\beta)\\text{ for all }\\beta\\in[e_k,V_k]}$ by construction and Lemma 1. If $V_k=e_{k+1}$ then we are done. If $V_k\\ne e_{k+1}$, on $(V_k,e_{k+1})$, there is no distinct AR function that intersects the AR function that realizes the $1-\\alpha$ quantile. Hence $m_k\\in\\mathcal I(\\beta)$ for all $\\beta\\in[V_k,e_{k+1})$ and by continuity $m_k\\in I(e_{k+1})$ if $e_{k+1}<\\infty$. By the same argument as in (H.1), the set $\\mathrm{CS}_i$ returned in Step 5 of Algorithm 2 satisfies $\\hl{\\mathrm{CS}_i=\\mathrm{CS}^*\\cap[e_i,e_{i+1}]}$, and hence $\\hl{\\bigcup_{i=0}^{n_I}}\\mathrm{CS}_i=\\mathrm{CS}^*$.''", "corrected": "Algorithms 2 and 4: ``Step 5: For each interval-index pair $\\hl{([e_{k-1},e_k],m_k)}$, $k=1,\\ldots,n_I$, recorded in Step 4, where $e_{n_I}=\\infty$ for the last pair: Find $\\mathrm{CS}_k=\\{\\beta:|\\Delta^2(\\beta)|\\leqslant|\\Delta^{*2}_{m_k}(\\beta)|,\\ \\beta\\in\\hl{[e_{k-1},e_k]}\\}$.'' Step 6 is unchanged.\n\nAlgorithm 4: ``Step 3: Set $e_0=-\\infty$ and let $\\hl{m_1}$ be any element in $\\mathcal I(\\beta_1-1)$.''\n\nSection H.2: ``For $\\hl{1\\le k\\le n_I-2}$, we have by construction that $[e_k,e_{k+1}]=[e_k,V_k]\\cup[V_k,e_{k+1}]$. We have $\\hl{m_{k+1}}\\in\\mathcal I(\\beta)$ for all $\\beta\\in[e_k,V_k]$ by construction and Lemma 1. If $V_k=e_{k+1}$ then we are done. If $V_k\\ne e_{k+1}$, on $(V_k,e_{k+1})$, there is no distinct AR function that intersects the AR function that realizes the $1-\\alpha$ quantile. Hence $\\hl{m_{k+1}}\\in\\mathcal I(\\beta)$ for all $\\beta\\in[V_k,e_{k+1})$ and by continuity $\\hl{m_{k+1}}\\in I(e_{k+1})$ if $e_{k+1}<\\infty$. By the same argument as in (H.1), the set $\\mathrm{CS}_i$ returned in Step 5 of Algorithm 2 satisfies $\\mathrm{CS}_i=\\mathrm{CS}^*\\cap\\hl{[e_{i-1},e_i]}$, and hence $\\hl{\\bigcup_{i=1}^{n_I}}\\mathrm{CS}_i=\\mathrm{CS}^*$.''", "evidence": "The evidence is a check of the pages and a trace of the algorithm. The first recorded pair is $([e_0,e_1],m_1)$. After the first finite intersection, Step 4 chooses $m_2$ in $\\mathcal I((e_1+V_1)/2)$ or in $\\mathcal I(e_1+1)$, and the second recorded pair is $([e_1,e_2],m_2)$ or $([e_1,\\infty),m_2)$. Each iteration records one pair, including the last. There are $n_I$ pairs indexed $1,\\ldots,n_I$, and the union in Step 6 is correct. Both branches of Step 4 assign $m_{i+1}$, and no index is left undefined. In the research code, the routine for Algorithm 2 pairs the $i$th recorded index with the interval whose left endpoint is the $i$th entry of $(-\\infty,e_1,e_2,\\ldots)$ and whose right endpoint is the $i$th entry of $(e_1,e_2,\\ldots,\\infty)$, which is the pairing of Step 4. I read this part of the code and did not run it for this point.", "problem": "Step 3 chooses $m_1$ for the interval $(-\\infty,e_1]$. In iteration $i$, Step 4 records $([e_{i-1},e_i],m_i)$, or $([e_{i-1},\\infty),m_i)$ when it exits, and chooses $m_{i+1}$ for the next interval. The paragraph below the algorithm says the same: the index $m_{i+1}$ realizes the quantile on $[e_i,e_{i+1}]$. Step 5 instead pairs $m_k$ with $[e_k,e_{k+1}]$, on which the recorded index is $m_{k+1}$, and it leaves $[e_0,e_1]$ without an index. The last paragraph of Section H.2 uses the indexing of Step 5 for the index and for $\\mathrm{CS}_i$, and takes a union from $i=0$, while Step 6 takes it from $i=1$. Step 5 says to process the pairs ``recorded in the previous steps'', and those pairs are correct. An implementation that used the displayed formula of Step 5 in place of the recorded pairs could compare with the wrong simulated AR function.", "what_survives": "The algorithm determined by the recorded pairs, the argument of Section H.2 once its indices are aligned, Lemma 1 and Theorem 3 are unchanged. The correction is one of labels."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E011", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Three lines of text that introduce Algorithm 1, then the four-line display (H.1) for the set CS-i. The first line compares the absolute value of Delta squared with the absolute value of Delta-star squared with index m-i. One red outline encloses eta-hat-star with subscript 1 minus alpha, evaluated at (beta, Z), in the second line and again in the third line. The last line reads CS-star intersected with the interval from beta-i to beta-(i+1). Below: where the second equality is justified by Lemma 1. Hence we have.", "caption": "Supplement, p. 45 (PDF p. 25 of the Supplement), display (H.1) in the proof of Theorem 3. Outlined: the critical value $\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)$ in the second and third lines, printed without a square.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E012.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T23", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T23"}], "headline": "In the second and third lines of (H.1) the critical value $\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)$ should be squared, since it is compared with the squared statistic.", "id": "P002-E012", "location": "Supplement, Section H.2, display (H.1), second and third lines, p. 45 (PDF p. 25 of the Supplement)", "pdf_page": 25, "printed_page": "45", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T23"], "text": {"as_printed": "\\[\n\\begin{aligned}\n\\mathrm{CS}_i&=\\big\\{\\beta:|\\Delta^2(\\beta)|\\le|\\Delta^{*2}_{m_i}(\\beta)|,\\ \\beta\\in[\\beta_i,\\beta_{i+1}]\\big\\}\\\\\n&=\\big\\{\\beta:|\\Delta^2(\\beta)|\\le\\hl{\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)},\\ \\beta\\in[\\beta_i,\\beta_{i+1}]\\big\\}\\\\\n&=\\big\\{\\beta:|\\Delta^2(\\beta)|\\le\\hl{\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)}\\big\\}\\cap[\\beta_i,\\beta_{i+1}]\\\\\n&=\\mathrm{CS}^*\\cap[\\beta_i,\\beta_{i+1}],\\qquad\\qquad\\text{(H.1)}\n\\end{aligned}\n\\]\nwhere the second equality is justified by Lemma 1.", "corrected": "\\[\n\\begin{aligned}\n\\mathrm{CS}_i&=\\big\\{\\beta:|\\Delta^2(\\beta)|\\le|\\Delta^{*2}_{m_i}(\\beta)|,\\ \\beta\\in[\\beta_i,\\beta_{i+1}]\\big\\}\\\\\n&=\\big\\{\\beta:|\\Delta^2(\\beta)|\\le\\big[\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)\\big]\\hl{{}^{2}},\\ \\beta\\in[\\beta_i,\\beta_{i+1}]\\big\\}\\\\\n&=\\big\\{\\beta:|\\Delta^2(\\beta)|\\le\\big[\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)\\big]\\hl{{}^{2}}\\big\\}\\cap[\\beta_i,\\beta_{i+1}]\\\\\n&=\\mathrm{CS}^*\\cap[\\beta_i,\\beta_{i+1}],\\qquad\\qquad\\text{(H.1)}\n\\end{aligned}\n\\]\nwhere the second equality is justified by Lemma 1. Here $i\\in\\{0,\\ldots,K\\}$, the intervals $[\\beta_i,\\beta_{i+1}]$ are those of Step~2 of Algorithm~1, $m_i$ is the index selected in Step~3, $\\mathrm{CS}_i$ is the set returned in Step~4, $\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)$ is the critical value defined in (13) of the article, and $\\mathrm{CS}^*$ is the confidence set (14).", "evidence": "The evidence is a derivation. By Lemma~1, there exists an index $m_i$ such that, on $[\\beta_i,\\beta_{i+1}]$, I have $\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)=|\\Delta^*_{m_i}(\\beta)|$, so that $|\\Delta^{*2}_{m_i}(\\beta)|=\\big[\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)\\big]^2$. For nonnegative $u$ and $q$, the inequality $u\\le q$ holds if and only if $u^2\\le q^2$. With the squares in place, the four sets in (H.1) are therefore equal. Without them the comparison is a different one: if $|\\Delta(\\beta)|=2$ and $\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)=3$, then (14) accepts $\\beta$ because $2\\le3$, while the printed lines reject it because $4>3$.", "problem": "By (13) of the article, $\\widehat\\eta^*_{1-\\alpha}(\\beta,Z)$ is an empirical quantile of the simulated absolute values $|\\Delta^*_m(\\beta)|$, and the confidence set $\\mathrm{CS}^*$ of (14) compares $|\\Delta(\\beta)|$ with it. The first line of (H.1) compares the squared statistic with a squared simulated statistic. The second and third lines keep the square on $\\Delta$ and omit it on the critical value. As printed, the sets in these two lines do not equal $\\mathrm{CS}^*\\cap[\\beta_i,\\beta_{i+1}]$ in general.", "what_survives": "Algorithm~1 and the display on p.~15 of the article already use the squared comparison. With the squares restored, (H.1) holds and the union over $i$ equals $\\mathrm{CS}^*$, which is the part of the proof of Theorem~3 that concerns Algorithm~1."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E012", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Theorem 7: 'For every alpha in (0, 1)', followed by display (E.36), a double limit of a supremum over beta and a supremum over theta in Theta_n with beta-bar(theta) equal to beta, of the probability that the absolute value of Delta_adj(beta) exceeds z_(1 - alpha/2) minus kappa, bounded by alpha. A red box marks Theta_n in the subscript of the inner supremum. The proof below reads 'Given Lemma 17, the proof of (E.36) is nearly identical to the proof of Theorem 6, so we omit it.'", "caption": "Supplementary Material, p. 39 (PDF p. 19 of the supplement): Theorem 7 and the first lines of its proof. The marked symbol $\\Theta_n$ under the inner supremum of (E.36) should be $\\Xi_n$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E013.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T21", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T21"}], "headline": "The inner supremum of (E.36) is taken over $\\Theta_n$, the parameter space of the theory without covariates; it should be taken over $\\Xi_n$, the parameter space of the regression-adjusted theory. If the population size ranges over all positive integers, the inner limit should also be a limit superior.", "id": "P002-E013", "location": "Supplementary Material, Theorem 7, display (E.36), p. 39 (PDF p. 19 of the supplement), inner supremum", "pdf_page": 19, "printed_page": "39", "reaches": [], "related_entries": [], "revisions": [{"classification_after": {"class": "error", "severity": "Minor"}, "classification_before": {"class": "error", "severity": "Minor"}, "note": "The wording was revised; the classification is unchanged. How this came to light: stating Theorem 7 in Lean showed that the display asserts an ordinary inner limit, and that with $r=1/2$ the model classes are empty at every odd population size. Admissible models exist whose rejection probabilities stay above one half along the sizes divisible by four, for every $\\kappa$ near zero, while the supremum is zero at odd sizes. Why it changed: the correction had said that only the symbol changes and that Theorem 2 is unaffected; neither assurance holds as stated. The corrected display now uses a limit superior, which the argument of the paper supports. P. M. Aronow delegated this decision to the model session that runs the audit, which adopted the amendment recommended by the two checks; this note was written by that session."}], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T21"], "text": {"as_printed": "Theorem 7. For every $\\alpha\\in(0,1)$,\n\\[\n\\lim_{\\kappa\\to0^+}\\lim_{n\\to\\infty}\\sup_{\\beta\\in\\mathbb R}\\ \\sup_{\\{\\theta\\in\\hl{\\scriptstyle\\Theta_n}:\\,\\bar\\beta(\\theta)=\\beta\\}}\\mathbb P_\\theta\\bigl(\\lvert\\Delta_{\\mathrm{adj}}(\\beta)\\rvert>z_{1-\\alpha/2}-\\kappa\\bigr)\\le\\alpha.\\qquad\\text{(E.36)}\n\\]", "corrected": "Theorem 7. For every $\\alpha\\in(0,1)$,\n\\[\n\\lim_{\\kappa\\to0^+}\\hl{\\limsup_{n\\to\\infty}}\\sup_{\\beta\\in\\mathbb R}\\ \\sup_{\\{\\theta\\in\\hl{\\scriptstyle\\Xi_n}:\\,\\bar\\beta(\\theta)=\\beta\\}}\\mathbb P_\\theta\\bigl(\\lvert\\Delta_{\\mathrm{adj}}(\\beta)\\rvert>z_{1-\\alpha/2}-\\kappa\\bigr)\\le\\alpha.\\qquad\\text{(E.36)}\n\\]\nHere $\\Xi_n=\\Xi_n(k,\\delta,\\tilde\\varepsilon,A,B,r)$ is the parameter space of models that satisfy Assumptions 1, 2, 4 and 5, with the constants fixed in the statement of Theorem 2, as in Section D.1 of the Supplementary Material. The population size ranges over all positive integers, and an empty supremum of probabilities is zero. The outer limit exists because the inner limit superior is nondecreasing in $\\kappa$ and lies in $[0,1]$.", "evidence": "The evidence is a check of the pages: the definitions of $\\Theta_n$ (p.~7) and of $\\Xi_n$ (p.~11) in the article, and Section D.1 (p.~31) and the statement of Lemma 17 (p.~35) in the Supplementary Material. I checked the four-cell example by exact enumeration of the assignment distribution for $n=4$, $8$ and $12$.", "problem": "The space $\\Theta_n$ collects the models without covariates that satisfy Assumptions 1--4 (p.~7 of the article). Theorem 7 concerns the regression-adjusted statistic $\\Delta_{\\mathrm{adj}}$. Section D.1 of the Supplementary Material fixes $\\Xi_n=\\Xi_n(k,\\delta,\\tilde\\varepsilon,A,B,r)$, the parameter space of Theorem 2, for Sections D and E, and the proof of Theorem 7 rests on Lemma 17, which is stated for sequences $\\theta_n\\in\\Xi_n$.\n\nThere is also an issue with the inner limit. Assumption 4 (p.~7 of the article) allows $r=1/2$, which forces $n_1=n/2$, so both $\\Theta_n$ and $\\Xi_n$ are empty at every odd population size. The paper neither restricts the size index to feasible sizes nor says what value an empty supremum of probabilities takes. If that value is zero, the inner limit can fail to exist, even for $\\kappa$ arbitrarily close to zero. Take $m$ units in each of the four cells $(x_i,z_i)\\in\\{-1,1\\}^2$, all compliers, with $y_i(0)=y_i(1)=z_i$ and $n_1=n/2$; these models belong to $\\Xi_n$ with $k=1$, $\\delta=0$, $\\tilde\\varepsilon=1/2$ and $A=B=2$. For $\\alpha=1/2$ and every $0<\\kappa<z_{1-\\alpha/2}$, the rejection probability tends to $2\\{1-\\Phi(z_{1-\\alpha/2}-\\kappa)\\}>1/2$ along the sizes divisible by four, where $\\Phi$ is the standard normal distribution function, while the supremum is zero at every odd size. The argument of Lemma 17 and Theorem 6 bounds the limit superior; it does not show that the ordinary inner limit exists. Restricting the index to feasible sizes removes this example but would need its own argument for existence.", "what_survives": "The model-space correction and the conservative bound on the adjusted tail are retained in the limit-superior form below. That the ordinary inner limits exist is not established."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E013", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the statement of Theorem 3. A red box marks the words 'in the Supplementary Material' that follow 'Without covariates and under Assumption 6'.", "caption": "Theorem 3, p. 17. The marked phrase places Assumption 6 in the Supplementary Material; it is stated in Section 5.2 of the article, p. 14.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E014.png"}, {"alt": "Crop of the last line of a displayed inequality and the sentence after it. A red box marks 'Assumption 5-(ii)' in 'where the inequality follows from Assumption 5-(ii). Therefore, recalling (D.5),'.", "caption": "Supplementary Material, p. 35 (PDF p. 15), proof of Lemma 16. The marked pointer should be to Assumption 5-(i); the printed inequality follows for $\\delta\\ge0$, but needs $\\delta_+=\\max\\{\\delta,0\\}$ in place of $\\delta$ over the full stated range.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E014-b.png"}, {"alt": "Crop of display (E.14) and the two lines after it. A red box marks '(using Assumption 5-(iv))' in 'Claim (E.12) can be proved as in Lemma 4 (using Assumption 5-(iv)), and claim (E.13) can be proved as in Lemma 6'.", "caption": "Supplementary Material, p. 35 (PDF p. 15), end of the proof of Lemma 16. The marked pointer is given for claim (E.12), which concerns residuals and follows from Assumption 5-(ii).", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E014-c.png"}, {"alt": "Crop of the first lines of Section I, Algebra for Remark 4, to display (I.1) and the sentence after it. Red boxes mark '(4.12)' in 'We first show that (4.12) satisfies', '(4.8)' in 'introduced below equation (4.8)', the two right-hand sides of display (I.1), 'tau hat y,1 minus beta tau hat d,1' and 'tau hat y,0 minus beta tau hat d,0', and the four symbols 'tau hat y,1, tau hat y,0, tau hat d,1, and tau hat d,0' in the sentence 'were introduced in Remark 4'.", "caption": "Supplementary Material, p. 45 (PDF p. 25), start of Section I. The marked pointers (4.12) and (4.8) are to equations (12) and (8) of the article, and the marked symbols $\\widehat\\tau_{y,z}$, $\\widehat\\tau_{d,z}$ are the slopes that Remark 4 calls $\\widehat\\gamma_{y,z}$, $\\widehat\\gamma_{d,z}$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E014-d.png"}, {"alt": "Crop of the first line of printed page 46 and the first line of the display after it. A red box marks 'tau hat of beta' in 'The quantity tau hat of beta can be rewritten as'; the display below begins 'tau hat adj of beta equals'.", "caption": "Supplementary Material, p. 46 (PDF p. 26). The marked symbol lacks the subscript of the statistic $\\widehat\\tau_{\\mathrm{adj}}(\\beta)$ that the display defines.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E014-e.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T24", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T24"}, {"id": "P002-T28", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T28"}], "headline": "Several cross-references point to the wrong place: the Supplementary Material cites equations of the article by section-prefixed numbers that the article does not use, Theorem 3 places Assumption 6 in the Supplementary Material, two steps in the proof of Lemma 16 cite the wrong parts of Assumption 5, its covariance and variance bounds can fail for allowed negative $\\delta$, and displays (I.1) and (I.2) write the regression slopes of Remark 4 as $\\widehat\\tau$.", "id": "P002-E014", "location": "Theorem 3, p. 17; Supplementary Material, pp. 23, 25, 32, 34, 35, 42, 43, 45, 46 and 48 (PDF pp. 3, 5, 12, 14, 15, 22, 23, 25, 26 and 28 of the supplement)", "pdf_page": 17, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T24", "P002-T28"], "text": {"as_printed": "\\begin{itemize}\n\\item Theorem 3, p.~17: ``Without covariates and under Assumption 6 \\hl{in the Supplementary Material}, the confidence set CS returned by Algorithm 1 or 2 is equal to that in (14).''\n\\item Supplementary Material, pointers to equations of the article: ``following \\hl{(2.2)}'' (p.~23); ``recalling the definitions in \\hl{(3.3)}'' (p.~25); ``between expressions \\hl{(4.9)} and (D.4)'' (p.~32); ``similar to \\hl{(5.15)} and \\hl{(5.16)}'' (p.~42); ``the definition of $\\Delta_{\\mathrm{adj}}(\\beta)$ from \\hl{(4.10)}'' (p.~43); ``the definition of $\\mathcal I$ in \\hl{(5.17)}'', ``the confidence set defined in \\hl{(5.14)}'', ``We first show that \\hl{(4.12)} satisfies'' and ``introduced below equation \\hl{(4.8)}'' (p.~45); ``generated according to \\hl{(6.18)} and \\hl{(6.19)}'' and ``generated according to \\hl{(6.20)} and \\hl{(6.21)}'' (p.~48).\n\\item Lemma 16, p.~34:\n\\[\n\\operatorname{var}\\big(\\widetilde\\tau_{\\mathrm{adj}}(\\beta_{n_k})\\big)\\ge(1-\\hl{\\delta})\\frac{n_0}{n_1n}S_1^2(\\beta_{n_k})+(1-\\hl{\\delta})\\frac{n_1}{n_0n}S_0^2(\\beta_{n_k}).\\qquad\\text{(E.10)}\n\\]\nIts proof, p.~35, uses\n\\[\n\\begin{aligned}\n\\sum_{i=1}^n\\big(\\varepsilon_i(1,\\beta_n)-\\varepsilon_i(0,\\beta_n)\\big)^2\n&=\\sum_{i=1}^n\\varepsilon_i^2(1,\\beta_n)+\\sum_{i=1}^n\\varepsilon_i^2(0,\\beta_n)-2\\sum_{i=1}^n\\varepsilon_i(1,\\beta_n)\\varepsilon_i(0,\\beta_n)\\\\\n&\\le\\Big(1+\\frac{\\hl{\\delta}n_0}{n_1}\\Big)\\sum_{i=1}^n\\varepsilon_i^2(1,\\beta_n)+\\Big(1+\\frac{\\hl{\\delta}n_1}{n_0}\\Big)\\sum_{i=1}^n\\varepsilon_i^2(0,\\beta_n),\n\\end{aligned}\n\\]\nfollowed by ``where the inequality follows from \\hl{Assumption 5-(ii)}'', and ends with ``Claim (E.12) can be proved as in Lemma 4 (using \\hl{Assumption 5-(iv)})''.\n\\item Section I, p.~45:\n\\[\n\\widehat\\gamma(1,\\beta)=\\hl{\\widehat\\tau_{y,1}}-\\beta\\,\\hl{\\widehat\\tau_{d,1}},\\qquad\\widehat\\gamma(0,\\beta)=\\hl{\\widehat\\tau_{y,0}}-\\beta\\,\\hl{\\widehat\\tau_{d,0}},\\qquad\\text{(I.1)}\n\\]\n``where $\\widehat\\tau_{y,1}$, $\\widehat\\tau_{y,0}$, $\\widehat\\tau_{d,1}$, and $\\widehat\\tau_{d,0}$ were introduced in Remark 4.'' The same symbols recur in (I.2) and in the text after it.\n\\item Section I, p.~46: ``The quantity $\\hl{\\widehat\\tau(\\beta)}$ can be rewritten as'', followed by a display for $\\widehat\\tau_{\\mathrm{adj}}(\\beta)$.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Theorem 3: ``Without covariates and under Assumption 6, the confidence set CS returned by Algorithm 1 or 2 is equal to that in (14).'' The reference to the Supplementary Material after Assumption 7 stays.\n\\item In the Supplementary Material, replace (2.2), (3.3), (4.8), (4.9), (4.10), (4.12), (5.14), (5.15), (5.16), (5.17), (6.18), (6.19), (6.20) and (6.21) by \\hl{(2), (3), (8), (9), (10), (12), (14), (15), (16), (17), (18), (19), (20) and (21)}.\n\\item Lemma 16 and its proof: define $\\delta_+=\\max\\{\\delta,0\\}$ and replace $\\delta$ by $\\hl{\\delta_+}$ in the covariance inequality, the ensuing variance bounds including (E.10), and subsequent uses of this lower bound. The pointer sentences become ``where the inequality follows from \\hl{Assumption 5-(i)}'' and ``Claim (E.12) can be proved as in Lemma 4 (using \\hl{Assumption 5-(ii)})''.\n\\item Section I:\n\\[\n\\widehat\\gamma(1,\\beta)=\\hl{\\widehat\\gamma_{y,1}}-\\beta\\,\\hl{\\widehat\\gamma_{d,1}},\\qquad\\widehat\\gamma(0,\\beta)=\\hl{\\widehat\\gamma_{y,0}}-\\beta\\,\\hl{\\widehat\\gamma_{d,0}},\\qquad\\text{(I.1)}\n\\]\n``where $\\widehat\\gamma_{y,1}$, $\\widehat\\gamma_{y,0}$, $\\widehat\\gamma_{d,1}$, and $\\widehat\\gamma_{d,0}$ were introduced in Remark 4.'' The same replacement applies in (I.2) and in the text after it.\n\\item Section I, p.~46: ``The quantity $\\hl{\\widehat\\tau_{\\mathrm{adj}}(\\beta)}$ can be rewritten as''.\n\\end{itemize}", "evidence": "I checked the pages and the bounds, including an exact counterexample. The targets of the equation pointers are:\n\\begin{audittable}{lll}{Pointers in the Supplementary Material to equations of the article.}\nPrinted page of the supplement & Printed pointer & Equation of the article \\\\\n23 & (2.2) & (2) \\\\\n25 & (3.3) & (3) \\\\\n32 & (4.9) & (9) \\\\\n42 & (5.15), (5.16) & (15), (16) \\\\\n43 & (4.10) & (10) \\\\\n45 & (5.17), (5.14), (4.12), (4.8) & (17), (14), (12), (8) \\\\\n48 & (6.18), (6.19), (6.20), (6.21) & (18), (19), (20), (21)\n\\end{audittable}\nFor the first step of Lemma 16, let $U=\\sum_i\\varepsilon_i^2(1,\\beta_n)$, $W=\\sum_i\\varepsilon_i^2(0,\\beta_n)$ and $C=\\sum_i\\varepsilon_i(1,\\beta_n)\\varepsilon_i(0,\\beta_n)$. The residuals have mean zero, and Assumption 5-(i) implies that $C\\ge-\\delta(UW)^{1/2}$. Set $\\delta_+=\\max\\{\\delta,0\\}$. Then $C\\ge-\\delta_+(UW)^{1/2}$, and since $\\delta_+\\ge0$ and $2(UW)^{1/2}\\le(n_0/n_1)U+(n_1/n_0)W$, I have\n\\[\nU+W-2C\\le\\Big(1+\\frac{\\delta_+n_0}{n_1}\\Big)U+\\Big(1+\\frac{\\delta_+n_1}{n_0}\\Big)W.\n\\]\nThis yields (E.10) with $1-\\delta_+$ in place of $1-\\delta$. To see why the replacement is needed, take $n=6$, $n_1=2$, $n_0=4$, $\\delta=-9/10$, all compliers, $x=(1,-1,1,-1,0,0)$ and $y(0)=y(1)=(1,1,-1,-1,0,0)$. These satisfy the stated assumptions with $r=1/3$, $\\widetilde\\varepsilon=1/2$, $A=1$ and $B=2$, and give $\\overline\\beta=0$ and $U=W=C=4$: the printed inequality reads $0\\le-1$, and (E.10) claims the variance $3/5$ is at least $19/30$. For (E.12), Assumption 5-(ii) gives $\\max_i\\varepsilon_i^2(a,\\beta_n)\\le\\{\\sum_i\\varepsilon_i^4(a,\\beta_n)\\}^{1/2}\\le B^2n^{1/2}\\sigma^2_{\\varepsilon(a,\\beta_n)}$, and $n^{-1}\\sum_i\\varepsilon_i^2(a,\\beta_n)=(n-1)\\sigma^2_{\\varepsilon(a,\\beta_n)}/n$. The ratio in (E.12) is then at most $B^2n^{3/2}/(n-1)$, which is $o(n)$. The use of Assumption 5-(iv) for (E.14) is correct. For (I.1), the outcome $Y_i-\\beta D_i$ is linear in $(Y_i,D_i)$ and the regressions share one design. The fitted slope is then $\\widehat\\gamma_{y,a}-\\beta\\widehat\\gamma_{d,a}$.", "problem": "The article numbers its equations (1) to (21), and the numbers with a section prefix that the Supplementary Material cites do not exist in it. Assumption 6 is stated in Section 5.2 of the article, p.~14; the Supplementary Material states only Assumption 7. In the proof of Lemma 16, the inequality that leads to (E.10) uses a bound on the sum of products of the residuals $\\varepsilon_i(1,\\beta_n)\\varepsilon_i(0,\\beta_n)$, which is the subject of the correlation bound 5-(i) and not of the fourth-moment bound 5-(ii). Changing this pointer alone is insufficient: the printed inequality and (E.10) can fail when $\\delta<0$, which Assumption 5 permits. Claim (E.12) concerns the largest squared residual, which is the subject of 5-(ii) and not of the covariate bound 5-(iv). Remark 4 introduces the slopes $\\widehat\\gamma_{y,1}$, $\\widehat\\gamma_{y,0}$, $\\widehat\\gamma_{d,1}$ and $\\widehat\\gamma_{d,0}$, and no symbols $\\widehat\\tau_{y,z}$ or $\\widehat\\tau_{d,z}$. Each intended target can be recovered from the content.", "what_survives": "The definitions and regression algebra are unaffected. The variance lower bound (E.10) and its uses need $1-\\delta_+$, which remains strictly positive throughout the stated range $\\delta<1$."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E014", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the start of Algorithm 2: the title, the Require line, Step 1 and Step 2. A red box marks the two lines 'If J is empty, choose any beta in R and m in I of beta, return CS equal to the set of beta such that the absolute value of Delta squared of beta is at most the absolute value of Delta star squared m of beta, and break'.", "caption": "Algorithm 2, Step 1, p. 16. The marked instruction is the comparison that the code omits.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E015.png"}, {"alt": "Listing of lines 188 to 202 of the function AR_permutation_test_poly2_cpp. A red box marks lines 194 to 200: an if statement on intersects_list counter equal to 0, a print statement, a comment saying that the observed AR curve must lie below the alpha-quantile AR curve because the Wald estimate must always be in the interval, the assignment CS_region_output=c(-Inf,Inf), and return(CS_region_output).", "caption": "Research code, 2_AR_inversion_algo2.R, lines 188–202 (a listing of the replication code, with line numbers added). The marked branch returns the whole real line when no intersection is found.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E015-b.png"}, {"alt": "Listing of lines 104 to 110 of the function AR_algo2. A red box marks lines 107 to 110: an if statement on length of intersections_grid equal to 0, a message 'No intersections found - returning full real line', and return(list(c(-Inf, Inf))).", "caption": "The rilate package, R/AR_algo2.R, lines 104–110 (a listing of the replication code, with line numbers added). The marked branch returns the whole real line.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E015-c.png"}], "formal_release": null, "formal_statements": [], "headline": "When no two simulated AR functions intersect, both implementations of Algorithm 2 in the replication code return the whole real line without the comparison of the observed AR function with the quantile function that Step 1 prescribes.", "id": "P002-E015", "location": "Algorithm 2, Step 1, p. 16, as implemented in the replication code: research code, 2_AR_inversion_algo2.R, lines 194–200; rilate package, R/AR_algo2.R, lines 107–110", "pdf_page": null, "printed_page": "16", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Algorithm 2, Step 1, p.~16: ``If $\\mathcal J$ is empty, choose any $\\beta\\in\\mathbb R$ and $m\\in\\mathcal I(\\beta)$, return $\\mathrm{CS}=\\{\\beta:|\\Delta^2(\\beta)|\\leqslant|\\Delta^{*2}_m(\\beta)|\\}$ and break.''\n\nResearch code, \\texttt{2\\_AR\\_inversion\\_algo2.R}, lines 194, 197, 198 and 200:\n\n\\texttt{if (intersects\\_list\\$counter==0)\\{}\n\n\\texttt{\\#if there is no intersections amont the AR curves. The observed AR curve must lies below the alpha-quantile AR curve becasue the Wald-estimate must always be the interval}\n\n\\texttt{\\hl{CS\\_region\\_output=c(-Inf,Inf)}}\n\n\\texttt{return(CS\\_region\\_output)}\n\nThe \\texttt{rilate} package, \\texttt{R/AR\\_algo2.R}, lines 107 and 109:\n\n\\texttt{if (length(intersections\\_grid) == 0) \\{}\n\n\\texttt{\\hl{return(list(c(-Inf, Inf)))}}", "corrected": "The printed text needs no change. When $\\mathcal J$ is empty, the code should choose an index $m$ that attains the simulated quantile at any finite $\\beta_0$ and \\hl{return $\\{\\beta\\in\\mathbb R:\\Delta^2(\\beta)\\le\\Delta_m^{*2}(\\beta)\\}$}, with its endpoints, in place of $(-\\infty,\\infty)$. In the \\texttt{rilate} package the result should be returned in the usual list, with \\texttt{confidence\\_set} and the p-value that the function has already computed.", "evidence": "The evidence is an exact counterexample run through both unmodified routines. Let $n=6$, $n_1=n_0=3$, $Y=(0,1,2,3,4,5)$, $D=(1,0,1,0,1,0)$ and $Z=(1,1,1,0,0,0)$, and let the $M=1000$ simulated assignments all equal $(1,0,1,0,1,0)$, at confidence level $0.95$. The data are consistent with exclusion, monotonicity, two compliers and a local average treatment effect of zero, and Assumption 6 holds for all $20$ assignments. A direct computation gives\n\\[\n\\Delta^2(\\beta)=\\frac{3(\\beta+9)^2}{4(\\beta^2+3)},\\qquad\\Delta_m^{*2}(\\beta)=\\frac{9(\\beta+1)^2}{16}\\quad\\text{for every }m.\n\\]\nSince all simulated AR functions coincide, $\\mathcal J$ is empty. Clearing the positive denominators, I have\n\\[\n\\Delta^2(\\beta)\\le\\Delta_m^{*2}(\\beta)\\iff(\\beta^2-9)\\big(3(\\beta+1)^2+32\\big)\\ge0\\iff|\\beta|\\ge3,\n\\]\nso (14) equals $(-\\infty,-3]\\cup[3,\\infty)$. At $\\beta=0$ the observed value $81/4$ exceeds the critical value $9/16$. Both routines return the whole real line. The package returns it as a bare list, without the fields \\texttt{confidence\\_set} and \\texttt{p\\_value} of its documented result, while its p-value at $\\beta=0$ is $1/1001$. In a second example, with $Y=(0,1,3,2,4,7)$ and the same $D$, $Z$ and simulated assignments, (14) again equals $(-\\infty,-3]\\cup[3,\\infty)$, while both routines again return the whole real line.", "problem": "When the simulated AR functions have no intersection, Lemma 1 implies that one index $m$ realizes the quantile on all of $\\mathbb R$, and (14) is the set where the observed AR function is at most the simulated one with index $m$. This set need not be the whole real line. The comment in the research code argues that the observed AR function must lie below the quantile function because the Wald estimate belongs to the set. The Wald estimate belongs to the set because $\\Delta$ vanishes there, and this says nothing about other values of $\\beta$.", "what_survives": "Step 1 of Algorithm 2 as printed, Lemma 1 and Theorem 3 are correct. Since the whole real line contains the intended set, coverage cannot fall on a run that reaches this branch; rejections and precision are lost. The example uses $1000$ identical simulated assignments, an event of probability $20^{-1000}$, and no reported result is shown to have reached the branch. The branch is also reached when the fixed tolerance of \\fid{P002-N015} leaves the grid of intersections empty."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E015", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the end of Algorithm 1, from the last lines of Step 3 to Step 5. A red box marks the less-than-or-equal sign in Step 4, 'Find CS i equal to the set of beta such that the absolute value of Delta squared of beta is at most the absolute value of Delta star squared m i of beta, beta in the interval from beta i to beta i plus 1'.", "caption": "Algorithm 1, Step 4, p. 15. The marked relation is the weak inequality with which the paper accepts a value of $\\beta$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E016.png"}, {"alt": "Listing of lines 244 to 254 of the function compare_AR. A red box marks line 252, 'return(AR1 < AR2)', under the comment 'Return TRUE if observed is below permutation'.", "caption": "The rilate package, R/find_intervals.R, lines 244–254 (a listing of the replication code, with line numbers added). The marked line is the strict comparison of the observed with the simulated AR statistic.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E016-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The routine of the replication code that builds an acceptance region from two AR functions tests the intervals between their intersections with a strict inequality; it returns nothing when the two functions coincide and it drops isolated points of equality, although the confidence sets of the paper accept with $\\le$.", "id": "P002-E016", "location": "Algorithm 1, Step 4, p. 15 (also equation (14), p. 12, and Algorithm 2, Step 5, p. 16), as implemented in the replication code: rilate package, R/find_intervals.R, lines 228 and 252; research code, 2_AR_inversion_algo1.R, lines 413 and 426", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Algorithm 1, Step 4, p.~15: ``For each interval $\\{[\\beta_i,\\beta_{i+1}]\\}_{i=0}^K$: Find $\\mathrm{CS}_i=\\{\\beta:|\\Delta^2(\\beta)|\\;\\leqslant\\;|\\Delta^{*2}_{m_i}(\\beta)|,\\ \\beta\\in[\\beta_i,\\beta_{i+1}]\\}$.''\n\nThe \\texttt{rilate} package, \\texttt{R/find\\_intervals.R}, line 252 (line 228 is the same; the research code has the same comparison):\n\n\\texttt{return(AR1 \\hl{<} AR2)}", "corrected": "The printed text needs no change. In the code, the interval routine should return $\\{\\beta:N_1(\\beta)Q_2(\\beta)-N_2(\\beta)Q_1(\\beta)\\;\\hl{\\le}\\;0\\}$: the whole real line when the polynomial $P=N_1Q_2-N_2Q_1$ is identically zero, and otherwise the closures of the intervals on which $P<0$ together with every real zero of $P$, kept as a one-point interval $[r,r]$ where it is not already an endpoint. These pieces must be kept by the later intersection and merging steps. The strict comparison may remain for the interior of the intervals.", "evidence": "The evidence is a set of exact counterexamples run through the unmodified code of both sources. For the coefficient vector $c_1=(1,0,0,1,0,1)$, which represents $\\beta^2/(1+\\beta^2)$, both routines return an empty result when $c_1$ is compared with itself, where the acceptance set is $\\mathbb R$. They also return an empty result when $c_1$ is compared with $c_0=(0,0,0,1,0,1)$, where $\\beta^2/(1+\\beta^2)\\le0$ holds exactly at $\\beta=0$. For $c_1$ against $(0,0,1,1,0,1)$ they return $[-1,1]$, which is correct.\n\nBoth cases arise from data. Consider eight units with $Y=(0,1,2,3,0,1,2,3)$ and $D=Z=(1,1,1,1,0,0,0,0)$. Assumption 6 holds for all $70$ assignments of four units. The observed assignment and its complement both have $\\Delta^2(\\beta)=8\\beta^2/5$, and the assignment $W=(1,0,0,1,1,0,0,1)$ and its complement have squared AR function identically $0$. Let the $M=20$ simulated assignments be $Z$, $1-Z$, and nine copies of the pair $W$, $1-W$. At level $0.95$ the quantile function of (13) is the observed AR function, so (14) is $\\mathbb R$. Both \\texttt{AR\\_algo1} and \\texttt{AR\\_algo2} of the package return an empty confidence set. When only the interval routine is replaced by one that keeps the two cases, both return $(-\\infty,\\infty)$.", "problem": "The routine computes the real zeros of $P=N_1Q_2-N_2Q_1$ for the observed and the simulated AR function, evaluates the strict comparison at one point of each interval between consecutive zeros, and returns the intervals where the comparison holds, with their endpoints. Ordinary intersections are handled correctly, because the endpoints are included. Two cases are lost. If the two functions are identical, $P$ has no isolated zero, the strict comparison fails, and the routine returns nothing, where the set $\\{\\beta:\\Delta^2(\\beta)\\le\\Delta_m^{*2}(\\beta)\\}$ is the whole real line. If the two functions touch at a point without crossing, the comparison fails on both sides and the point is dropped. Note that Assumption 6 does not exclude identical functions. With arms of equal size, an assignment and its complement have the same squared AR function, and a simulated assignment can equal the observed one.", "what_survives": "The definitions (13) and (14), the steps of Algorithms 1 and 2 as printed, and the theory are correct. No reported table entry or application result is shown to change. How often the two cases occur in the designs of the paper is not established, and the application was not recomputed."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E016", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Top of page 20 of the article with the running head. A red outline marks the sentence: In the Supplementary Material we report an additional set of results based on the Get Out the Vote experiment in Green et al. (2003). The paragraph on Project STAR follows.", "caption": "Article, p. 20 (PDF p. 20), first two lines. Outlined: the sentence that points to the get-out-the-vote application in the Supplementary Material, which the script 1_data_analysis_GGN2003.R computes.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E017.png"}, {"alt": "Monospaced listing of code lines 115 to 132, headed ORIGINAL CODE TEXT (not a paper screenshot). A red outline marks line 122, which stores a first-stage F statistic computed by coeftest from fs1. A second red outline marks the statement on line 127 that assigns a linear model to fs1.", "caption": "Lines 115–132 of 1_data_analysis_GGN2003.R in the replication files, rendered as text (not a page of the article). Outlined: line 122, which uses fs1, and the statement on line 127 that first creates it.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E017-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The script for the get-out-the-vote application uses the first-stage regression fs1 on line 122, five lines before line 127 creates it, so the first city stops with an error; line 122 should be deleted.", "id": "P002-E017", "location": "Replication code, 1_data_analysis_GGN2003.R, lines 122 and 127; the article points to this application on p. 20 (PDF p. 20), and its results are Supplement Tables 6 and 7, pp. 49–50", "pdf_page": 20, "printed_page": "20", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Replication code (not a passage of the article):\n\nline 2: \\texttt{rm(list=ls())}\n\nline 122: \\texttt{output\\_list\\_temp[['first\\_stage\\_F']] = coeftest(\\hl{fs1},param=c('assignment'),vcov. = vcovHC(\\hl{fs1},type=\"HC0\"))['assignment','t value']$\\texttt{\\textasciicircum}$2}\n\nline 127: \\texttt{fs1 = lm(D\\_observed $\\texttt{\\textasciitilde}$ assignment + xz1 + xz0, data = data\\_temp2)}\n\nline 128: \\texttt{output\\_list\\_temp[['first\\_stage\\_F\\_cov']] = coeftest(fs1,param=c('assignment'),vcov. = vcovHC(fs1,type=\"HC0\"))['assignment','t value']$\\texttt{\\textasciicircum}$2}", "corrected": "Delete line 122 of \\texttt{1\\_data\\_analysis\\_GGN2003.R}. Keep lines 127--128, which fit \\texttt{fs1} and compute \\texttt{first\\_stage\\_F\\_cov}, and lines 151--153, which fit \\texttt{fs2} and compute \\texttt{first\\_stage\\_F\\_no\\_cov} and \\texttt{compliance\\_rate}. The F statistic printed in Tables~6 and~7 is the one without covariates, which lines 151--152 compute from \\texttt{fs2}. The deletion repairs the order of the statements only; a complete run still needs the paths, the packages and the six-city data.", "evidence": "The evidence is an exact counterexample obtained by running the code, and a recalculation. With the Bridgeport data supplied in the replication files (1307 voters), running the unchanged lines 109--113 and then the unchanged line 122 produces the error \\texttt{object 'fs1' not found}, and nothing is stored. With line 122 omitted, lines 127--128 and 151--153 run for Bridgeport and for Columbus. The saved output of the application in the replication files is an empty list. Exact rational arithmetic on the two city files reproduces the entries of Tables~6 and~7 that require no simulation.\n\n\\begin{audittable}{lrrrr}{Entries of Supplement Tables~6 and~7 recomputed in exact rational arithmetic from the two city files. The columns headed by a city give the recomputed values, with decimals rounded to six places; the columns headed by a table give the printed values.}\nQuantity & Bridgeport & Table 6 & Columbus & Table 7 \\\\\nSample size & 1307 & 1307 & 2424 & 2424 \\\\\nCompliance rate & $2/7=0.285714$ & 0.286 & $53/377=0.140584$ & 0.141 \\\\\nFirst-stage F, no covariate & $1274/5=254.8$ & 254.8 & $19981/108=185.009259$ & 185.0 \\\\\nPoint estimate, no covariate & 0.163425 & 0.163 & 0.103971 & 0.104 \\\\\nPoint estimate, with covariate & 0.173315 & 0.173 & 0.111099 & 0.111 \\\\\n\\end{audittable}\n\nThe whole script could not be run. It also needs directory paths that exist only on the machine where it was written, the \\texttt{ivreg} package, and the data file for the six cities, which is not among the replication files. Without those, a run stops near the top of the script, at the statement on line 7 that sets the working directory.", "problem": "The script clears its workspace on line 2 and then loops over the six cities, beginning with Bridgeport. Inside the loop, line 122 computes a first-stage F statistic from the fitted model \\texttt{fs1}, but no earlier statement creates \\texttt{fs1}; its first assignment is on line 127. When R reaches line 122 for the first city, it stops with the error \\texttt{object 'fs1' not found}. The loop then does not finish Bridgeport, does not reach the other five cities or the randomization-based confidence sets, and does not reach the statement on line 164 that saves the results. As released, the script cannot regenerate Supplement Tables~6 and~7. The entry that line 122 would store, \\texttt{first\\_stage\\_F}, is used nowhere else in the replication files.", "what_survives": "No printed number is shown to be wrong, and the defect does not show what state the code was in when the tables were produced. For Bridgeport and Columbus, the sample sizes, compliance rates, first-stage F statistics and both point estimates recompute to the printed values. The randomization-based confidence sets and the other four cities were not recomputed; the two-city 2SLS intervals were recalculated separately (see \\fid{P002-N014}). The theory, the simulation scripts and the STAR application do not use this line."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E017", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text beginning 'where W_2i is a standard normal variable. We further centre the average treatment effect'. A red box marks the words 'for the complier group' at the start of the second line, which continues 'so that the LATE is exactly zero.'", "caption": "Section 6, p. 18 (PDF p. 18), first lines: the sentence that follows equation (19). The marked words name the complier group as the group in which the treatment effects are centred.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E018.png"}, {"alt": "Listing of source code with line numbers. Line 73 is the comment 'calibrate the effects to match exactly'. Red boxes mark lines 75 and 76 under the comment 'For always-takers', which compute factor_at and subtract it from the second column of outcome_raw, and lines 79 and 80 under the comment 'For never-takers', which do the same with factor_nt. Lines 83 and 84, under the comment 'For compliers', compute factor_c and subtract it, and are not marked.", "caption": "Source code of the research repository, not a page of the article: 2_gen_data.R, lines 71–84. The marked lines subtract the mean drawn effect from the treated potential outcomes of the always-takers (lines 75–76) and of the never-takers (lines 79–80); lines 83–84 do the same for the compliers.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E018-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The description of the first simulation design states that the treatment effects are centred in the complier group and omits that the generator also centres them within the always-takers and within the never-takers; the sentence should describe all three centrings.", "id": "P002-E018", "location": "Section 6, p. 18 (PDF p. 18), first lines: the sentence after equation (19); equations (18) and (19) are on p. 17. Research repository RandomizaionIV_ResearchCode (commit 4d919c4): 2_gen_data.R, lines 73–84. The code is not printed in the article.", "pdf_page": 18, "printed_page": "18", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``where $W_{2i}$ is a standard normal variable. We further centre the average treatment effect \\hl{for the complier group} so that the LATE is exactly zero.''\n\n\\texttt{2\\_gen\\_data.R}, lines 75--76 and 79--80:\n\n\\texttt{factor\\_at = mean(outcome\\_raw[1:N\\_at,2]-outcome\\_raw[1:N\\_at,3]) -tau\\_at}\n\n\\texttt{outcome\\_raw[1:N\\_at,2] = outcome\\_raw[1:N\\_at,2] - factor\\_at}\n\n\\texttt{factor\\_nt = mean(outcome\\_raw[(N\\_at+1):(N\\_at+N\\_nt),2]-outcome\\_raw[(N\\_at+1):(N\\_at+N\\_nt),3]) -tau\\_nt}\n\n\\texttt{outcome\\_raw[(N\\_at+1):(N\\_at+N\\_nt),2] = outcome\\_raw[(N\\_at+1):(N\\_at+N\\_nt),2] -factor\\_nt}", "corrected": "``where $W_{2i}$ is a standard normal variable. We further centre the treatment effects \\hl{separately within the always-taker, never-taker and complier groups, so that the average treatment effect of each group is exactly zero; in particular,} the LATE is exactly zero.''\n\nEquations (18) and (19) then describe the draws before this centring.", "evidence": "The evidence is an exact example, run through the unchanged generator. I write $e_i=0.1W_{2i}$ for the drawn effect of unit $i$ and $\\bar e_g$ for the mean of the drawn effects in group $g$. The printed steps give $y_i(1)=y_i(0)+e_i-\\mathbf{1}\\{i\\in\\mathcal C\\}\\bar e_{\\mathcal C}$, and the generator gives $y_i(1)=y_i(0)+e_i-\\bar e_g$ for $i$ in group $g$. The two populations have the same $y_i(0)$. Their observed outcomes differ by $-\\bar e_{\\mathcal A}$ for each always-taker and agree for every other unit. Suppose that $n_1$ units are assigned to treatment and $n_0$ to control, and that $a_1$ of the $m_{\\mathcal A}$ always-takers are assigned to treatment. Then the difference in mean outcomes between the two arms changes by\n\\[\n-\\bar e_{\\mathcal A}\\left(\\frac{a_1}{n_1}-\\frac{m_{\\mathcal A}-a_1}{n_0}\\right).\n\\]\nFor the example, suppose that there are $100$ units, of which $25$ are always-takers, $25$ never-takers and $50$ compliers, and that every $W_{1i}$ equals $0$. Let the drawn effects have mean $1/4$ among the always-takers and $-1/8$ among the never-takers, and let them equal $-1/8$ for $25$ compliers and $3/8$ for the other $25$. Suppose further that the $25$ always-takers and the $25$ compliers whose drawn effect is $3/8$ are assigned to treatment. The difference in mean outcomes is $-3/4$ under the printed steps and $-7/8$ under the generator. Every unit assigned to treatment takes it and no other unit does. The Wald estimate then changes by the same amount, $-1/8$. The LATE is exactly zero in both populations. The generator, run on these draws, returns the second population.\n\nThe example does not show that any published coverage rate, length or computation time is wrong, and it does not measure how far the simulation results would move under the printed steps.", "problem": "Someone who builds the first design from equations (18) and (19) and from this sentence centres the drawn treatment effects of the compliers and leaves those of the always-takers and never-takers as drawn. The generator does more. In each of the three groups it subtracts the mean drawn effect of the group from the treated potential outcomes, so that the average effect is exactly zero in every group. The two procedures give different populations. Since an always-taker takes the treatment under either assignment, the additional centring changes the observed outcomes of the always-takers. The additional centring of the never-takers changes only a potential outcome that is never observed. The printed sentence is true as far as it goes: the complier effects are centred, and the LATE is exactly zero.", "what_survives": "Under both procedures the LATE is exactly zero, the average effect of assignment is zero and the treatment effects are heterogeneous. The theorems and the algorithms are unaffected. The simulation scripts call the generator as it stands, so correcting the sentence requires no new simulation. I could not determine which version of the code produced each published entry. If the centring of the always-takers were removed from the generator instead, the summaries for that design would have to be computed again."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E018", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the caption of Table 1. A red box marks the words '(ii) the nominal 95 per cent 2SLS intervals from the IV regression', which are followed on the next line by the formula Y ~ 1 + D + X + X squared, with instruments 1, Z, X, X squared.", "caption": "Caption of Table 1, p. 19 (PDF p. 19). The marked words describe the 2SLS comparison as the interval from an IV regression that includes the covariate $X$ and its square.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E019.png"}, {"alt": "Listing of source code with line numbers, under the comment 'tsls wald CI'. A red box marks line 76, which sets denominator to the mean of D_observed among units with assignment 1 minus the mean among units with assignment 0. A red box marks the condition abs(denominator)<1e-8 in line 78. A red box marks tsls_t_CI_lo=-Inf,tsls_t_CI_up=Inf in line 80. In the else branch, line 84 calls ivreg with formula Y_observed ~ D_observed+x1+x2 | assignment+x1+x2 and line 87 computes a 95 per cent confidence interval with HC0 standard errors.", "caption": "Source code of the research repository, not a page of the article: 2_main_function_with_covariates.R, lines 69–92. The marked line 76 computes the unadjusted difference in treatment rates, the marked condition on line 78 tests it, and the marked assignment on line 80 stores the whole real line; the regression with covariates on line 84 is fitted only otherwise.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E019-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The simulation code stores the covariate-adjusted 2SLS comparison interval as the whole real line whenever the unadjusted difference in treatment rates between the two arms is zero, although the adjusted regression can then still be fitted; the exception should depend on whether the adjusted regression can be computed.", "id": "P002-E019", "location": "Research repository RandomizaionIV_ResearchCode (commit 4d919c4): 2_main_function_with_covariates.R, lines 76–80. The code is not printed in the article; the comparison that it computes is described in the caption of Table 1, p. 19 (PDF p. 19).", "pdf_page": 19, "printed_page": "19", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{2\\_main\\_function\\_with\\_covariates.R}, lines 76, 78, 80, 82 and 84:\n\n\\texttt{denominator = mean(data\\_temp[which(data\\_temp[,'assignment']==1),'D\\_observed'])- mean(data\\_temp[which(data\\_temp[,'assignment']==0),'D\\_observed'])}\n\n\\texttt{if (\\hl{abs(denominator)<1e-8})\\{}\n\n\\texttt{output=c(output, tsls\\_t\\_CI\\_lo=-Inf,tsls\\_t\\_CI\\_up=Inf,tsls=NA,Fstat=NA)}\n\n\\texttt{\\}else\\{}\n\n$\\texttt{iv\\_result=ivreg( formula=Y\\_observed\\textasciitilde{}D\\_observed+x1+x2|assignment+x1+x2, data = data\\_tsls)}$\n\nCaption of Table 1, p.~19: ``Rows 1--4 report the coverage rates of (i) our randomization-based confidence sets, (ii) the nominal 95\\% 2SLS intervals from the IV regression $Y\\sim 1+D+X+X^2\\mid 1,Z,X,X^2$, (iii) confidence sets based on the AR statistic with a normal critical value using the same IV regression, and (iv) the tF confidence intervals from Lee et al. (2022) with the same IV regression.''", "corrected": "\\texttt{2\\_main\\_function\\_with\\_covariates.R}, lines 76--80: \\hl{the condition on the unadjusted difference is replaced by a test of whether the adjusted regression of line 84 can be computed}, on the same observations and with the same constant and covariates as the fit. In the notation above, the regression can be computed when $C$ has rank three, $z'M_Cz>0$ and $z'M_Cd\\ne0$. In floating-point arithmetic these conditions are tested with a rank tolerance scaled to the matrices. The Wald interval is computed whenever the conditions hold, and another interval is stored only when they fail, under a stated rule. The 2SLS rows of Table 1 and of Table 3 in the Supplementary Material are computed again before any entry is replaced.", "evidence": "The evidence is a pair of exact counterexamples, each run through the unchanged code.\n\nIn the first example, eight units have covariate $X=(-3,-2,-1,0,1,2,3,4)$, assignments $z=(1,1,1,1,0,0,0,0)$, treatments $d=(1,1,0,0,1,1,0,0)$ and outcomes $y=2d+e$, where $e=(14,-22,-3,11,-1,3,2,-4)/6000$. Let $x_1$ and $x_2$ be $X$ and $X^2$ minus their means, as in the simulation script. In each arm two of the four units take the treatment, and the unadjusted difference is zero. The residual of $z$ after projection on $C$ is $M_Cz=(-7,1,9,17,-17,-9,-1,7)/42$, so that $z'M_Cz=10/21$ and $z'M_Cd=-16/21$. The first-stage coefficient of the adjusted regression is $-8/5$. The 2SLS coefficient is exactly $2$, and its HC0 variance is $661/512000000$. The $95\\%$ Wald interval is approximately $[1.998,2.002]$ with the normal critical value and $[1.997,2.003]$ with the Student critical value. The data are consistent with a monotone compliance pattern, in which units $1$, $2$, $7$ and $8$ are the compliers, and with potential outcomes $y_i(0)=y_i(1)=y_i$. The LATE is then zero, and the interval excludes it.\n\nThe second example has $100$ units with the counts of the design L5\\%: $48$ always-takers, $47$ never-takers and $5$ compliers, with $50$ units in each arm. The treated arm holds $22$ always-takers, $24$ never-takers and $4$ compliers, and $26$ units take the treatment in each arm. The first-stage coefficient of the adjusted regression is $-85037548/309802841$, the 2SLS coefficient is $2$, and the $95\\%$ Wald interval is approximately $[1.9996,2.0004]$ while the LATE is zero. On both examples the code returns $(-\\infty,\\infty)$ and does not call the regression.\n\nIn the design L5\\%, with $50$ of $100$ units assigned to treatment, the unadjusted difference is zero with probability\n\\[\n\\frac{971765151348859865}{13607964513309364889}\\approx0.0714.\n\\]\nThis is the probability that the exception applies. It is not the size of any change in a coverage rate.", "problem": "The code computes \\texttt{denominator}, the difference between the two arms in the mean of the observed treatment. No covariate enters it, and I call it the unadjusted difference. When its absolute value is below $10^{-8}$, the code stores the interval $(-\\infty,\\infty)$, stores \\texttt{NA} for the estimate and for the first-stage $F$ statistic, and does not call the IV regression. The regression that the caption describes, and that the code fits otherwise, includes a constant, the covariate and its square. A zero unadjusted difference does not make this regression singular. Let $C=[\\mathbf 1,x_1,x_2]$ have rank three, let $M_C=I-C(C'C)^{-1}C'$, and let $z$, $d$ and $y$ be the vectors of assignments, treatments and outcomes. The 2SLS coefficient of the treatment is $z'M_Cy/z'M_Cd$, and it is defined whenever $z'M_Cd\\ne0$. The code therefore replaces some finite Wald intervals by the whole real line. Since the whole line always covers the true value and has infinite length, the exception can raise the coverage rate and the median length reported for 2SLS and cannot lower them.", "what_survives": "The randomization-based confidence sets, their coverage rates, lengths and computation times, and the AR and tF comparisons do not pass through this exception. Tables 4 and 5 of the Supplementary Material have no 2SLS rows, and the 2SLS intervals of the applications are computed by separate calls. No entry of Table 1 or of Table 3 in the Supplementary Material is shown to change, and the size of any change is not established. Since the exception cannot lower the coverage rate reported for 2SLS, the coverage below the nominal level that Table 1 reports for 2SLS at low compliance rates does not come from it."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E019", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Figure 1 of the article: a plot of log AR squared against a transformed beta axis, with four dashed curves labelled AR_sim1 to AR_sim4, one dot-dash curve labelled AR_obs, a solid quantile curve, and four vertical lines labelled e_i, V_i, S_i and e_i+1. A red box marks the label AR_obs. Below the plot is the caption, in which a red box marks the words 'Illustration of Algorithm 2'.", "caption": "Figure 1 and its caption, p. 17 (PDF p. 17). The marked label AR_obs names the dot-dash curve, which the script computes from the unadjusted statistic; the dashed curves and the solid quantile curve come from the regression-adjusted statistic. The marked words of the caption present the figure as an illustration of Algorithm 2.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E020.png"}, {"alt": "Listing of source code with line numbers. In line 103 of 4_plot_AR.R a red box marks AR_coef in the assignment AR_obs_coef = AR_coef(data_temp,N1,N0). In line 104 a red box marks AR_permutation_test_calculate_coef_adj in the assignment to AR_sim_coef. Below, lines of 2_AR_inversion3.R show that AR_coef constructs the coefficients without covariates, that AR_coef_adj constructs them with covariates, and that AR_permutation_test_calculate_coef_adj returns AR_coef_adj(data_table, N1, N0), which is marked by a red box in line 126.", "caption": "Source code of the research repository, not a page of the article: 4_plot_AR.R, lines 100–104 and 108–109, above the definitions that these lines call in 2_AR_inversion3.R. The marked call on line 103 uses the function without covariates, and the marked call on line 104 uses a function that returns AR_coef_adj, the function with covariates.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-E020-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The script that draws Figure 1 computes the observed curve from the unadjusted AR statistic and the four simulated curves from the regression-adjusted statistic, so the region below the quantile curve in the figure is not the confidence set (14); line 104 should call the unadjusted function, as Algorithm 2 requires.", "id": "P002-E020", "location": "Figure 1 and its caption, p. 17 (PDF p. 17), with the description of the figure on p. 16 (PDF p. 16). Research repository RandomizaionIV_ResearchCode (commit 4d919c4): 4_plot_AR.R, block “Plot for Paper”, lines 103–104. The code is not printed in the article.", "pdf_page": 17, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Caption of Figure 1, p.~17: ``Fig. 1. Illustration of Algorithm 2: the dashed lines represent the AR functions from simulated assignments, while the dot-dash line is the AR function from the observed data; the solid curve is the quantile function. We use four simulated assignments and the 75th percentile; the vertical lines denote intersections of the simulated AR functions.''\n\nPage 16: ``The confidence set is the region where the dot-dash curve lies below the solid curve.''\n\n\\texttt{4\\_plot\\_AR.R}, lines 103--104:\n\n\\texttt{AR\\_obs\\_coef = AR\\_coef(data\\_temp,N1,N0)}\n\n\\texttt{AR\\_sim\\_coef = pbsapply(1:nsim\\_permu,AR\\_permutation\\_test\\_calculate\\_coef\\hl{\\_adj},data\\_table=data\\_temp,N1=N1,N0=N0,assignment\\_random=zsim)}", "corrected": "\\texttt{4\\_plot\\_AR.R}, line 104 (line 103 is kept):\n\n\\texttt{AR\\_sim\\_coef = pbsapply(1:nsim\\_permu,\\hl{AR\\_permutation\\_test\\_calculate\\_coef},data\\_table=data\\_temp,N1=N1,N0=N0,assignment\\_random=zsim)}\n\nThe figure is then drawn again, and the labels $e_i$, $V_i$, $S_i$ and $e_{i+1}$ are checked against the new crossings. Alternatively, line 103 may call \\texttt{AR\\_coef\\_adj}; the caption and the description on p.~16 should then refer to Algorithm 4 in the Supplementary Material and to its regression-adjusted confidence set.", "evidence": "The evidence is an exact counterexample and an exact calculation on the inputs of the figure.\n\nLet eight units have outcomes $Y=(0,1,2,4,0,2,1,3)$, treatments $D=(0,1,1,1,0,0,1,0)$, assignments $Z=(1,1,1,1,0,0,0,0)$ and one covariate $x=(-3,-1,1,3,-3,-1,1,3)$. For this assignment the squared unadjusted statistic and the squared adjusted statistic are\n\\[\n\\frac{4(2\\beta-1)^2}{6\\beta^2-10\\beta+55}\n\\qquad\\text{and}\\qquad\n\\frac{10(2\\beta-1)^2}{10\\beta^2+22\\beta+21}.\n\\]\nBoth denominators are positive for every real $\\beta$. At $\\beta=0$ the two values are $4/55$ and $10/21$. The two functions of the script therefore give different curves for the same data and the same assignment.\n\nWhen I run the block ``Plot for Paper'' with its own seed, the curves and their crossings agree by eye with those of the published figure. The block generates $30$ units, of which $15$ are assigned to treatment, and four simulated assignments. At $\\beta=-89/100$, exact rational arithmetic on these inputs gives the following values of the squared statistics.\n\n\\begin{audittable}{lr}{Inputs of Figure 1 at $\\beta=-0.89$: squared AR statistics, rounded to four decimals. The figure draws the second and third rows.}\nQuantity & Value \\\\\n75th percentile of the four simulated curves, unadjusted & $0.8140$ \\\\\nObserved curve, unadjusted & $1.5902$ \\\\\n75th percentile of the four simulated curves, adjusted & $1.6715$ \\\\\nObserved curve, adjusted & $3.2551$ \\\\\n\\end{audittable}\n\nThe rule of the figure compares the second row with the third and includes $\\beta=-0.89$. The rule (14) compares the second row with the first, the regression-adjusted rule compares the fourth with the third, and both exclude this value.", "problem": "Algorithm 2 and the set (14) compare the observed AR statistic with the same statistic computed under each simulated assignment. The data that the script generates include three covariates. Line 103 computes the observed curve with \\texttt{AR\\_coef}, which ignores the covariates and returns the unadjusted statistic of equation (4). Line 104 computes the four simulated curves with \\texttt{AR\\_permutation\\_test\\_calculate\\_coef\\_adj}, which calls \\texttt{AR\\_coef\\_adj} and returns the regression-adjusted statistic of equation (10). The two statistics are different functions of $\\beta$, and no change of normalization turns one into the other. The dot-dash curve of Figure 1 is therefore not the observed counterpart of the dashed curves. The region where it lies below the solid curve is neither the set (14) of Algorithm 2 nor the regression-adjusted set of Algorithm 4 in the Supplementary Material.", "what_survives": "The four simulated curves all use the adjusted statistic. Their intersections, their order, the 75th-percentile curve and the illustration of the skipped intersection $S_i$ do not involve the observed curve and are unaffected. Lemma 1, Algorithms 1--4 and Theorem 3 are unaffected, and no table or application result is shown to depend on this block of the script. In the simulation and application scripts that I read, the routines in use apply one statistic to the observed and the simulated assignments. What changes is the dot-dash curve of Figure 1 and, with it, the region that p.~16 describes as the confidence set."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-E020", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the second half of Step 4 of Algorithm 2, from 'Else: Set e i equal to min of C i and record' to 'Set m i plus 1 to be any element in I of e i plus V i over 2'. A red box marks the two lines 'If E i is empty: Set m i plus 1 to be any element of I of e i plus 1'.", "caption": "Algorithm 2, Step 4, p. 16. The marked branch chooses a new index after the last intersection; the code omits this choice.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N002.png"}, {"alt": "Listing of lines 287 to 297 of the function AR_permutation_test_poly2_cpp. An if statement tests whether ind_int equals the length of intersections_grid, with the comment 'if the intersection is the last point, break'. A red box marks line 290, 'indices = c(indices,s)', followed by break. In the else branch, line 297 sets s to find_quantile_R of ind_int+1.", "caption": "Research code, 2_AR_inversion_algo2.R, lines 287–297 (a listing of the replication code, with line numbers added). At the last intersection the marked line records the current index s for the final interval; the call that selects a new index is made only in the other branch, on line 297.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N002-b.png"}, {"alt": "Listing of lines 177 to 187 of the function AR_algo2. Under the comment 'Check if we reached the end', an if statement tests whether ind_int equals the length of intersections_grid. A red box marks line 179, 'indices <- c(indices, s)', followed by break. After the if block, lines 185 to 187 set s to find_quantile_index of ind_int + 1.", "caption": "The rilate package, R/AR_algo2.R, lines 177–187 (a listing of the replication code, with line numbers added). The marked line records the current index for the final interval.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N002-c.png"}, {"alt": "Line chart with a strip below it. The horizontal axis is beta from minus 10 to 24; the vertical axis is the squared AR statistic from 0 to 5.2. A blue curve, the squared critical value used in (14), has two tall peaks, falls to about 2.5 at beta equal to 8, and then stays near 2.8. A black curve, the observed AR function, falls from about 4.6 near beta equal to minus 5 to zero at beta equal to 4 and rises slowly to about 1.8 at beta equal to 24. A dashed orange curve, the AR function kept by the code, starts at beta equal to 8, where a dotted vertical line marks the last intersection, and falls towards zero; an open circle marks where the black and orange curves meet at beta equal to 12. At beta equal to 16 three dots are labelled 128/45 on the blue curve, 18/13 on the black curve and 32/57 on the orange curve. In the strip, a blue bar shows the set (14), from minus 4 to minus 2 and from 2 to the right edge, and an orange bar shows the returned set, from minus 4 to minus 2 and from 2 to 12.", "caption": "The eight-unit example at $M=1000$. Upper panel: the observed squared AR function, the squared critical value used in (14), and the simulated AR function that the code keeps beyond the last intersection of two simulated AR functions, at $\\beta=8$. Lower panel: the set (14) and the set returned by the code. To the right of $\\beta=12$ the observed function lies above the kept function and below the squared critical value, so the code rejects values that (14) accepts.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N002-last-intersection.png"}], "formal_release": null, "formal_statements": [], "headline": "When the intersection reached by the loop of Algorithm 2 is the largest of all intersections, both implementations in the replication code keep the previous quantile index for the final interval instead of choosing $m_{i+1}\\in\\mathcal I(e_i+1)$; the returned set can then differ from (14) to the right of the last intersection and can be bounded where (14) is unbounded.", "id": "P002-N002", "location": "Algorithm 2, Step 4, p. 16 (the same step in Algorithm 4, Supplementary Material p. 44), as implemented in the replication code: research code, 2_AR_inversion_algo2.R, lines 287–292 (and 103–108); rilate package, R/AR_algo2.R, lines 178–180", "pdf_page": null, "printed_page": "16", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Algorithm 2, Step 4, p.~16: ``Set $e_i=\\min(\\mathcal C_i)$ and record $\\{[e_{i-1},e_i],m_i\\}$. Set $\\mathcal E_i=\\{j\\in\\mathcal J,j>e_i\\}$. If $\\mathcal E_i$ is empty: Set $m_{i+1}$ to be any element of $\\mathcal I(e_i+1)$.''\n\nResearch code, \\texttt{2\\_AR\\_inversion\\_algo2.R}, lines 287, 290 and 292, where \\texttt{s} is the current quantile index:\n\n\\texttt{if (ind\\_int == length(intersections\\_grid))\\{}\n\n\\texttt{\\hl{indices = c(indices,s)}}\n\n\\texttt{break}\n\nThe \\texttt{rilate} package, \\texttt{R/AR\\_algo2.R}, lines 178--180:\n\n\\texttt{if (ind\\_int == length(intersections\\_grid)) \\{}\n\n\\texttt{\\hl{indices <- c(indices, s)}}\n\n\\texttt{break}", "corrected": "The printed text needs no change. In the code, the branch for the last intersection should select the index for the final interval before recording it. In the research code:\n\n\\texttt{if (ind\\_int == length(intersections\\_grid))\\{}\n\n\\texttt{\\hl{s = as.integer(find\\_quantile\\_R(ind\\_int+1,AR\\_sim\\_coef,intersections\\_grid,alpha=1-alpha,tol=tol))}}\n\n\\texttt{indices = c(indices,s)}\n\n\\texttt{break}\n\nIn the \\texttt{rilate} package:\n\n\\texttt{if (ind\\_int == length(intersections\\_grid)) \\{}\n\n\\texttt{\\hl{s <- as.integer(find\\_quantile\\_index(ind\\_int + 1L, AR\\_sim\\_coef, intersections\\_grid, alpha = 1 - alpha, tol = tol))}}\n\n\\texttt{indices <- c(indices, s)}\n\n\\texttt{break}\n\nThe second wrapper of the research code, written in R only (lines 103--108), needs the same line with its fixed significance level $0.05$. The index already recorded for the preceding interval is kept.", "evidence": "The evidence is an exact counterexample, run through the unmodified code of both sources, and a derivation. I take eight units, four of them treated in every assignment, with the observed assignment $Z$ and four other assignments $A$, $B$, $H$ and $L$:\n\\begin{audittable}{rrrrrrr}{Data of the example: outcome, treatment received, the observed assignment and four assignments used for the simulation.}\n$Y$ & $D$ & $Z$ & $A$ & $B$ & $H$ & $L$ \\\\\n$3$ & $1$ & $1$ & $1$ & $1$ & $1$ & $1$ \\\\\n$3$ & $0$ & $1$ & $1$ & $0$ & $0$ & $0$ \\\\\n$-1$ & $1$ & $1$ & $0$ & $1$ & $1$ & $1$ \\\\\n$-1$ & $1$ & $1$ & $0$ & $0$ & $1$ & $0$ \\\\\n$1$ & $1$ & $0$ & $1$ & $1$ & $1$ & $0$ \\\\\n$1$ & $0$ & $0$ & $1$ & $0$ & $0$ & $1$ \\\\\n$-3$ & $0$ & $0$ & $0$ & $1$ & $0$ & $1$ \\\\\n$-3$ & $0$ & $0$ & $0$ & $0$ & $0$ & $0$\n\\end{audittable}\nFor an assignment $j$ let $t_j(\\beta)$ be the difference of the arm means of $Y-\\beta D$, and let $T(\\beta)=2(\\beta-1)^2+38$. The squared AR function of $j$ is $F_j=16t_j^2/(T-2t_j^2)$, with positive denominator, and\n\\[\nt_Z=2-\\tfrac\\beta2,\\qquad t_A=4,\\qquad t_B=-\\tfrac\\beta2,\\qquad t_H=1-\\beta,\\qquad t_L=0 .\n\\]\nSince $F_j$ is increasing in $t_j^2$, every comparison reduces to one between squares of these linear functions. The simulated AR functions intersect at $-8,-3,0,2/3,1,2,5$ and $8$, and the largest intersection, at $\\beta=8$, is a crossing of $F_A$ and $F_B$. Further,\n\\[\nF_Z\\le F_A\\iff-4\\le\\beta\\le12,\\qquad F_Z\\le F_B\\iff\\beta\\ge2,\\qquad F_Z\\le F_H\\iff|\\beta|\\ge2 .\n\\]\nLet $r$ be the rank of the quantile, and let the $M$ simulated assignments be $r-2$ copies of $L$, one copy each of $A$ and $B$, and $M-r$ copies of $H$. The squared critical value at rank $r$ is then $\\min\\{F_H,\\max(F_A,F_B)\\}$, and $\\max(F_A,F_B)$ equals $F_A$ for $|\\beta|\\le8$ and $F_B$ otherwise. For (13), $r=\\lceil0.95M\\rceil$, and I obtain\n\\[\n\\widehat I^*_{0.95}(Z)=[-4,-2]\\cup[2,\\infty).\n\\]\nJust left of $\\beta=8$ the quantile is realized by $A$, and just right of it by $B$. The code keeps $A$ on $(8,\\infty)$ and returns $[-4,-2]\\cup[2,12]$. At $\\beta=16$ the kept function equals $32/57$, the observed function $18/13$ and the squared critical value $128/45$.\n\\begin{audittable}{llll}{Sets returned at $M=1000$ and confidence level $0.95$. The last column is the intended inversion at each routine's own rank; (13) uses rank $950$, while the package uses $951$.}\nRoutine & Rank $r$ & Returned & After the repair below \\\\\nResearch code (C++ path), no covariate & $950$ & $[-4,-2]\\cup[2,12]$ & $[-4,-2]\\cup[2,\\infty)$ \\\\\n\\texttt{rilate}, no covariate & $951$ & $[-4,-2]\\cup[2,12]$ & $[-4,-2]\\cup[2,\\infty)$ \\\\\nResearch code (C++ path), one covariate & $950$ & $[-20,-18]\\cup[-14,-4]$ & $[-20,-18]\\cup[-14,\\infty)$ \\\\\n\\texttt{rilate}, one covariate & $951$ & $[-20,-18]\\cup[-14,-4]$ & $[-20,-18]\\cup[-14,\\infty)$\n\\end{audittable}\nThe rows with a covariate use $32$ units, obtained by repeating each unit four times, with a covariate taking the values $-1,-1,1,1$ within each group of four and with $Y$ replaced by $Y-16D$. The endpoints in these rows agree with the stated values to $10^{-12}$. In these rows the returned set excludes the value $0$, which the intended set contains. With the same construction, the stated sets also hold at $M=5000$ and $M=10^4$, with ranks $4750$ and $9500$ for the research code and $4751$ and $9501$ for the package; the code was run at $M=1000$ only. A second, independent example with eight simulated assignments has set $(-\\infty,-3]\\cup[-1,\\infty)$, and both routines return only $(-\\infty,-3]$.\n\nWith correctly computed intersections and quantile indices, the repair is justified as follows. Under Assumption 6, or Assumption 7 with covariates, the squared AR functions are continuous, and two distinct ones differ by a rational function with at most four real zeros. To the right of the largest intersection no difference changes sign, so the order of the simulated values is constant there. Any index in $\\mathcal I(e_i+1)$ realizes the quantile on the whole final interval, including its left endpoint by continuity.\n\nThe effect of the defect on the reported numbers is not established. The simulated assignments of the published runs are not in the replication files, and neither are the data of the application to Angrist et al.\\ or of four of the six cities. An argument that the branch is reached with probability of order $2/M$ does not hold in general: for four units with $Y=(0,1,2,4)$, $D=(1,1,0,0)$ and two treated, under complete randomization, the code keeps a wrong index with probability at least $1-2\\times10^{-106}$ at $M=1000$. Coverage over $5000$ data sets moves in steps of $0.0002$, and one changed decision can change a printed third decimal.", "problem": "Step 4 follows the simulated AR function that currently realizes the quantile, with index $m_i$, to its next intersection $e_i$, and then chooses a new index for the interval to the right of $e_i$. The printed step does this in both cases: when later intersections exist it evaluates $\\mathcal I$ between $e_i$ and the next one, and when $e_i$ is the largest of all intersections it evaluates $\\mathcal I$ at $e_i+1$. The code carries out the first case. In the second case it appends the current index \\texttt{s}, which is $m_i$, for the final interval and leaves the loop. If the crossing changes the function that realizes the quantile, the code compares the observed AR function with the wrong simulated one on $(e_i,\\infty)$. The returned set then differs from the intended inversion where the observed function is strictly above the smaller of the two critical functions and at or below the larger. The difference between the two sets consists either only of added points or only of deleted points, and it can be unbounded.\n\nThe defect is confined to the interval to the right of the largest intersection of simulated AR functions. Its effect on the reported numbers is not known. When the last intersection is a unique simple crossing of two singly occurring functions, the kept function is one rank away from the correct one, but this bounds neither how often the branch is reached nor how far an endpoint moves: in the example below the kept function is exactly one rank away at $M=1000$, $5000$ and $10^4$, and the right endpoint moves from $\\infty$ to $12$.", "what_survives": "Step 4 as printed, Lemma 1, the definitions (13)--(17), Theorem 3 and the theory for the confidence sets are unaffected by this code defect. This defect does not change membership at or to the left of the largest intersection of simulated AR functions. A value $\\beta$ to the right of it changes status only if the observed AR function is strictly above the smaller of the kept function and the correct one and at or below the larger; in particular a changed coverage decision at the true value $0$ in Section 6 requires both a negative largest intersection and this bracket at $0$. The p-value of the \\texttt{rilate} package at $\\beta=0$ is computed separately and is unaffected. The procedures that do not use randomization critical values are unaffected. The simulation and application scripts of the research code call the affected routine. I have not established whether any reported entry changes: this applies to the coverage rates and lengths of the randomization sets in Table 1 and Supplementary Tables 3--5 and 12, to their endpoints in Table 2 and Supplementary Tables 6 and 7, and to the relative lengths in Supplementary Table 10."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N002", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 1 with seven columns, headed L5, L10, L25, L50 and L90 per cent, IR1 and IR2, and twelve rows of coverage rates, median lengths and times. A red box marks the column heading L25 per cent. A second red box marks the last four entries of that column: 50, 7, 28 571 and 2919, in the rows 'Time (s, Algorithm 3, M = 10^3)', 'Time (s, Algorithm 4, M = 10^3)', 'Time (s, Algorithm 3, M = 10^4)' and 'Time (s, Algorithm 4, M = 10^4)'.", "caption": "Table 1, p. 19 (PDF p. 19). The marked entries are the four computation times of column L25%. The script selects runs with $20$ compliers for this column, but whether it produced these published values is not established.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The script that assembles the computation times of Table 1 fills the column L25% from the output directory of the runs with $20$ compliers; it should read the directory of the runs with $25$ compliers. Whether the published column was assembled in this way is not established.", "id": "P002-N003", "location": "Table 1, p. 19 (PDF p. 19), rows 9–12, column L25%. Research repository RandomizaionIV_ResearchCode (commit 4d919c4): 3_merge_results_grace2.R, lines 110 and 129. The code is not printed in the article.", "pdf_page": 19, "printed_page": "19", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{3\\_merge\\_results\\_grace2.R}, line 110, final directory name in the argument \\texttt{path} for \\texttt{output\\_25\\_time}:\n\n\\texttt{sim\\_12172024\\_randomization\\_cov\\_\\hl{20}}\n\nLine 129:\n\n\\texttt{output\\_table\\_temp = cbind(output\\_5\\_time,output\\_10\\_time,output\\_25\\_time,output\\_50\\_time,output\\_90\\_time,output\\_IR2dx\\_time,output\\_IR2dx2\\_time)}\n\n\\begin{audittable}{lr}{Table 1, p.~19: rows 9--12 of column L25\\%.}\nRow & L25\\% \\\\\nTime (s, Algorithm 3, $M=10^3$) & $50$ \\\\\nTime (s, Algorithm 4, $M=10^3$) & $7$ \\\\\nTime (s, Algorithm 3, $M=10^4$) & $28\\,571$ \\\\\nTime (s, Algorithm 4, $M=10^4$) & $2919$ \\\\\n\\end{audittable}\n\nCaption of Table 1: ``Rows 9--12 report the median computation times (in seconds) for constructing randomization-based confidence sets using Algorithms 3 and 4 in the Supplementary Material, with $10^3$ and $10^4$ simulated assignments. The times are obtained separately with 50 additional simulations; all times are rounded to the nearest integer''", "corrected": "\\texttt{3\\_merge\\_results\\_grace2.R}, line 110: replace the final directory name in the argument \\texttt{path} for \\texttt{output\\_25\\_time} by\n\n\\texttt{sim\\_12172024\\_randomization\\_cov\\_\\hl{25}}\n\nThe comment on line 108 should also read 25. The four medians are then computed from the timing files of the design with $25$ compliers, and an entry of Table 1 is replaced if it changes. The replication files do not contain the times from which replacement values could be computed.", "evidence": "The evidence is an exact check of the code on constructed inputs. I ran the unchanged function and the unchanged lines on $50$ constructed timing files in each directory. The medians of the four times are $(200,20,20000,2000)$ in the directory \\texttt{cov\\_20} and $(250,25,25000,2500)$ in the directory \\texttt{cov\\_25}. The third column of the assembled table is $(200,20,20000,2000)$. When the suffix on line 110 is changed to \\texttt{25}, the third column is $(250,25,25000,2500)$ and the other six columns are unchanged.\n\nThe inputs are constructed, and they are not the times of the paper. The original timing files are not in the replication files, and nothing there identifies the run that produced the published column. I cannot say whether any of the entries $50$, $7$, $28\\,571$ and $2919$ differs from the median for the design with $25$ compliers.", "problem": "\\begin{audittable}{rl}{Selected lines of \\texttt{3\\_merge\\_results\\_grace2.R}. An ellipsis replaces the directory prefix in each \\texttt{path}; line numbers are from the source.}\nLine & Source code \\\\\n106 & \\texttt{output\\_10\\_time=extract\\_results3(path='\\ldots/sim\\_12172024\\_randomization\\_cov\\_10',tau=0)} \\\\\n110 & \\texttt{output\\_25\\_time=extract\\_results3(path='\\ldots/sim\\_12172024\\_randomization\\_cov\\_20',tau=0)} \\\\\n114 & \\texttt{output\\_50\\_time=extract\\_results3(path='\\ldots/sim\\_12172024\\_randomization\\_cov\\_50',tau=0)} \\\\\n129 & \\texttt{output\\_table\\_temp = cbind(output\\_5\\_time,output\\_10\\_time,output\\_25\\_time,output\\_50\\_time,output\\_90\\_time,output\\_IR2dx\\_time,output\\_IR2dx2\\_time)} \\\\\n\\end{audittable}\nLine 110 assigns to \\texttt{output\\_25\\_time} the medians that \\texttt{extract\\_results3} computes from the directory whose name ends in \\texttt{cov\\_20}. The timing script with $40$ always-takers, $40$ never-takers and $20$ compliers (\\texttt{1\\_simulation\\_grace\\_cov\\_20\\_nmc1000.R}) writes to that directory. The timing script with $38$ always-takers, $37$ never-takers and $25$ compliers (\\texttt{1\\_simulation\\_grace\\_cov\\_25\\_nmc1000.R}) writes to the directory whose name ends in \\texttt{cov\\_25}. Line 129 places \\texttt{output\\_25\\_time} third among the seven timing columns, which is the position of L25\\% in Table 1. If this block assembled the published column from the outputs of these scripts, its four entries describe the design with $20$ compliers and not the design with $25$.", "what_survives": "Line 110 affects neither the coverage rates and lengths of column L25\\%, which the script reads from a separate directory of runs with $25$ compliers, nor the other six timing columns. The other six designs support the comparison of the speeds of Algorithms 3 and 4. No theorem, algorithm or application result uses these times."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N003", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text beginning 'quickly. Tables 4 and 5 in the Supplementary Material report coverage rates'. Red boxes mark the letter K in 'randomization draws with K in {1000, 5000, 10 000}' and in 'this range of K'.", "caption": "Section 6, p. 19 (PDF p. 19), paragraph after Table 1. The two marked symbols $K$ denote the number of simulated assignments, which the paper elsewhere writes $M$.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N004.png"}, {"alt": "Crop of the four-line caption of Table 4. Red boxes mark N_1 in 'N_1 = 50', N_0 in 'N_0 = 50', and K in '(K = 1000, 5000, 10000)'.", "caption": "Supplementary Material, p. 48 (PDF p. 28 of the supplement): caption of Table 4. The marked symbols $N_1$ and $N_0$ are the arm sizes $n_1$ and $n_0$, and the marked $K$ is the number of simulated assignments.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N004-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The number of simulated assignments is written $K$ in the paragraph after Table 1 and in Section J of the Supplementary Material, where the rest of the paper writes $M$; the sizes of the two arms are written $N_1$ and $N_0$ in the tables of the Supplementary Material and in Table 2, where the paper defines $n_1$ and $n_0$.", "id": "P002-N004", "location": "Section 6, p. 19 (PDF p. 19), paragraph after Table 1; also Supplementary Material, Section J and the captions of Tables 4 and 5, pp. 47–49 (PDF pp. 27–29 of the supplement), and Table 2, p. 21 (PDF p. 21)", "pdf_page": 19, "printed_page": "19", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Page 19: ``Tables 4 and 5 in the Supplementary Material report coverage rates, median lengths and median computation times for different numbers of randomization draws with $\\hl{K}\\in\\{1000,5000,10\\,000\\}$ using Algorithm 4. We do not observe substantial differences across this range of $\\hl{K}$.''\n\nSupplementary Material, caption of Table 4, p.~48: ``Coverage rates, median lengths, and average computation times for randomization-based confidence sets across simulation designs with $\\hl{N_1}=50$ and $\\hl{N_0}=50$ and with varying numbers of randomization draws ($\\hl{K}=1000,5000,10000$).''", "corrected": "Page 19: ``\\ldots for different numbers of randomization draws with $\\hl{M}\\in\\{1000,5000,10\\,000\\}$ using Algorithm 4. We do not observe substantial differences across this range of $\\hl{M}$.''\n\nSupplementary Material, Section J and the captions of Tables 4 and 5: $\\hl{n_1}$ and $\\hl{n_0}$ for the arm sizes, and $\\hl{M}=1000,5000,10000$ for the numbers of randomization draws. Table 2: row labels $\\hl{n_1}$ and $\\hl{n_0}$.", "intended_meaning": "In these passages $K$ counts the simulated assignments, which Section 5, Algorithms 1--4 and Table 1 denote by $M$. In Section 5.3 and in the algorithms $K$ is the number of intersection points. The symbols $N_1$ and $N_0$ are the numbers of units assigned to treatment and to control, which the paper defines as $n_1$ and $n_0$. Each meaning is stated where the symbol is used, and no reported number depends on the choice of letter."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N004", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Three entries of the reference list: Kang, Peck and Keele (2018); Keane and Neal; Keele, Small and Grieve (2017). A red outline marks the year (2021) in the entry: Keane, M. P. and Neal, T. (2021). A practical guide to weak instruments. Annu. Rev. Econ. 16, 185 to 212.", "caption": "Reference list, p. 22 (PDF p. 22), entry for Keane & Neal with its neighbours. Outlined: the year 2021, printed with volume 16 of Annu. Rev. Econ.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N007.png"}, {"alt": "Three entries of the reference list: Moreira (2009); Nelson and Startz, J. Bus. 63; Nelson and Startz (1990), Econometrica 58. A red outline marks the year (1988) in the entry: Nelson, C. and Startz, R. (1988). The distribution of the instrumental variables estimator and its t-ratio when the instrument is a poor one. J. Bus. 63, S125 to 40.", "caption": "Reference list, p. 22 (PDF p. 22), the two entries for Nelson & Startz with the entry before them. Outlined: the year 1988, printed with volume 63 of J. Bus.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Two references pair the year of a working paper with the volume and pages of the journal article: the Keane & Neal article appeared in 2024, and the Journal of Business article of Nelson & Startz in 1990.", "id": "P002-N007", "location": "Reference list, p. 22 (PDF p. 22), entries “Keane, M. P. & Neal, T. (2021)” and “Nelson, C. & Startz, R. (1988)”; citations on pp. 3 and 18", "pdf_page": 22, "printed_page": "22", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Keane, M. P. \\& Neal, T. (\\hl{2021}). A practical guide to weak instruments. \\emph{Annu. Rev. Econ.} \\textbf{16}, 185--212.\n\nNelson, C. \\& Startz, R. (\\hl{1988}). The distribution of the instrumental variables estimator and its \\emph{t}-ratio when the instrument is a poor one. \\emph{J. Bus.} \\textbf{63}, S125--40.\n\nIn the text: ``Keane \\& Neal (\\hl{2021})'' (p.~3) and ``(Nelson \\& Startz, \\hl{1988}, 1990; Maddala \\& Jeong, 1992; Angrist \\& Kolesár, 2024)'' (p.~18).", "corrected": "Keane, M. P. \\& Neal, T. (\\hl{2024}). A practical guide to weak instruments. \\emph{Annu. Rev. Econ.} \\textbf{16}, 185--212.\n\nNelson, C. \\& Startz, R. (\\hl{1990a}). The distribution of the instrumental variables estimator and its \\emph{t}-ratio when the instrument is a poor one. \\emph{J. Bus.} \\textbf{63}, S125--40.\n\nNelson, C. R. \\& Startz, R. (\\hl{1990b}). Some further results on the exact small sample properties of the instrumental variable estimator. \\emph{Econometrica} \\textbf{58}, 967--76.\n\nIn the text: ``Keane \\& Neal (\\hl{2024})'' on p.~3 and ``(Nelson \\& Startz, \\hl{1990a,b}; Maddala \\& Jeong, 1992; Angrist \\& Kolesár, 2024)'' on p.~18. Alternatively, the years 2021 and 1988 may stand if the entries cite the working papers themselves, without the journal volumes and pages.", "evidence": "The evidence is a check against the publishers' records. The Annual Reviews page for the article reads ``Vol. 16:185--212 (Volume publication date August 2024)''. The JSTOR contents of the \\emph{Journal of Business} issue are headed ``Vol. 63, No. 1, Jan., 1990, Part 2'' and list the article on pp.~S125--S140. The working papers are CEPAR Working Paper 2021/24, dated 27 September 2021, and NBER Technical Working Paper 69, issued in September 1988.", "problem": "Each of the two entries prints the volume and pages of the journal article with the year of an earlier working paper. Volume 16 of \\emph{Annual Review of Economics}, pp.~185--212, was published in 2024. Volume 63 of \\emph{Journal of Business}, no.~1, part~2, pp.~S125--S140, is dated January 1990. The printed years, 2021 and 1988, are those of the working-paper versions, and the citations on pp.~3 and 18 carry them into the text.", "what_survives": "Both works remain identifiable from their authors, titles, volumes and pages, and nothing in the paper depends on the years."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N007", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of text from the notation section of the Supplement. A red box marks the citation 'van der Vaart (2000)' in the sentence which says that the conventions for stochastic o and O symbols follow that work.", "caption": "Supplement, Section A, p. 21 (PDF p. 1). The marked citation has no entry in a reference list.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N008.png"}, {"alt": "Five lines of text from the proof of Lemma 13. A red box marks the words 'Lemma 21.2 of van der Vaart (2000)', which the sentence says states that convergence in distribution implies weak convergence of quantile functions.", "caption": "Supplement, proof of Lemma 13, p. 30 (PDF p. 10). The same work is cited for a numbered lemma.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N008-c.png"}, {"alt": "A displayed identity for alpha-hat, the sentence that begins 'By the OLS representation in Lin (2013)' and the display that follows it. A red box marks the citation 'Lin (2013)'.", "caption": "Supplement, Section I, p. 47 (PDF p. 27). The marked citation has no entry in a reference list.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The Supplement cites van der Vaart (2000) and Lin (2013), and neither work has an entry in a reference list.", "id": "P002-N008", "location": "Supplement, Section A, p. 21 (PDF p. 1); proof of Lemma 13, p. 30 (PDF p. 10); Section I, p. 47 (PDF p. 27)", "pdf_page": 1, "printed_page": "21", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "For stochastic $o$ and $O$ symbols, we follow the conventions in \\hl{van der Vaart (2000)}. (p.~21)\n\nAs a consequence of this convergence in distribution and \\hl{Lemma 21.2 of van der Vaart (2000)}, which states that convergence in distribution implies weak convergence of quantile functions, \\ldots\\ (p.~30)\n\nBy the OLS representation in \\hl{Lin (2013)}, we have: (p.~47)", "corrected": "Add a reference list to the Supplement, or two entries to the reference list of the article, with the full references for van der Vaart (2000) and Lin (2013). I do not supply the entries here, since the two works were not consulted.", "evidence": "The evidence is a check of the pages and a search of the text of the article and of the Supplement. The names of the two authors occur only in the three citations above. The two works were not consulted, and no statement taken from them was checked.", "problem": "The Supplement has no reference list: it ends with Table 12 on p.~52. The reference list of the article (pp.~21--23) has no entry for either work. The first is cited for a numbered lemma that the proof of Lemma 13 uses, and the second for a representation used in Section I.", "what_survives": "Every formula, proof and result is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N008", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of a page of mathematics. In display (E.29) a red box marks Z_i x_i-circle-prime at the start of the first sum. In the three-line expansion of tau-tilde-adj(beta_n) below it, four red boxes mark the letter a in the four occurrences of w_i(a, beta_n), and in the last line a red box marks the factor (1 - Z_i), which stands before the summation sign.", "caption": "Supplementary Material, p. 37 (PDF p. 17 of the supplement): display (E.29) and the expansion of $\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)$ that follows it. The marked terms are the factor $Z_i$ in the first sum of (E.29), the four indices $a$, and the factor $(1-Z_i)$ outside the last sum.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N009.png"}], "formal_release": null, "formal_statements": [{"id": "P002-T20", "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-T20"}], "headline": "Three slips in the proof of Lemma 17: the first sum of (E.29) should carry $(1-Z_i)$ in place of $Z_i$, the outcomes $w_i(a,\\beta_n)$ of the next display should be $w_i(1,\\beta_n)$ in the treated sums and $w_i(0,\\beta_n)$ in the control sums, and the factor $(1-Z_i)$ of its last line belongs inside the sum.", "id": "P002-N009", "location": "Supplementary Material, proof of Lemma 17, p. 37 (PDF p. 17 of the supplement): display (E.29) and the display that follows “For the second claim, in (E.16), first notice that”", "pdf_page": 17, "printed_page": "37", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P002-T20"], "text": {"as_printed": "A similar argument shows that\n\\[\n\\frac{1}{n_0\\sqrt{\\operatorname{var}(\\tilde\\tau_{\\mathrm{adj}}(\\beta_n))}}\\sum_{i=1}^n\\hl{Z_i}x_i^{\\circ\\prime}\\left(\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\tilde x_i'\\right)^{-1}\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\varepsilon_i(0,\\beta_n)=o_p(1).\\qquad\\text{(E.29)}\n\\]\nThis proves the first claim. For the second claim, in (E.16), first notice that\n\\[\n\\begin{aligned}\n\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)&=\\frac{1}{n_1}\\sum_{i=1}^nZ_i\\bigl(w_i(\\hl{a},\\beta_n)-x_i'\\gamma(1,\\beta_n)\\bigr)-\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\bigl(w_i(\\hl{a},\\beta_n)-x_i'\\gamma(0,\\beta_n)\\bigr)\\\\\n&=\\frac{1}{n_1}\\sum_{i=1}^nZ_i\\bigl(w_i(\\hl{a},\\beta_n)-\\varphi(1,\\beta_n)-x_i'\\gamma(1,\\beta_n)\\bigr)\\\\\n&\\qquad-\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\bigl(w_i(\\hl{a},\\beta_n)-\\varphi(0,\\beta_n)-x_i'\\gamma(0,\\beta_n)\\bigr)\\\\\n&=\\frac{1}{n_1}\\sum_{i=1}^nZ_i\\varepsilon_i(1,\\beta_n)-\\frac{1}{n_0}\\hl{(1-Z_i)}\\sum_{i=1}^n\\varepsilon_i(0,\\beta_n),\n\\end{aligned}\n\\]\nwhere we use the fact that \\ldots", "corrected": "A similar argument shows that\n\\[\n\\frac{1}{n_0\\sqrt{\\operatorname{var}(\\tilde\\tau_{\\mathrm{adj}}(\\beta_n))}}\\sum_{i=1}^n\\hl{(1-Z_i)}x_i^{\\circ\\prime}\\left(\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\tilde x_i'\\right)^{-1}\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\tilde x_i\\varepsilon_i(0,\\beta_n)=o_p(1).\\qquad\\text{(E.29)}\n\\]\nThis proves the first claim. For the second claim, in (E.16), first notice that\n\\[\n\\begin{aligned}\n\\tilde\\tau_{\\mathrm{adj}}(\\beta_n)&=\\frac{1}{n_1}\\sum_{i=1}^nZ_i\\bigl(w_i(\\hl{1},\\beta_n)-x_i'\\gamma(1,\\beta_n)\\bigr)-\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\bigl(w_i(\\hl{0},\\beta_n)-x_i'\\gamma(0,\\beta_n)\\bigr)\\\\\n&=\\frac{1}{n_1}\\sum_{i=1}^nZ_i\\bigl(w_i(\\hl{1},\\beta_n)-\\varphi(1,\\beta_n)-x_i'\\gamma(1,\\beta_n)\\bigr)\\\\\n&\\qquad-\\frac{1}{n_0}\\sum_{i=1}^n(1-Z_i)\\bigl(w_i(\\hl{0},\\beta_n)-\\varphi(0,\\beta_n)-x_i'\\gamma(0,\\beta_n)\\bigr)\\\\\n&=\\frac{1}{n_1}\\sum_{i=1}^nZ_i\\varepsilon_i(1,\\beta_n)-\\frac{1}{n_0}\\sum_{i=1}^n\\hl{(1-Z_i)}\\varepsilon_i(0,\\beta_n).\n\\end{aligned}\n\\]\nHere $\\beta_n=\\bar\\beta(\\theta_n)$ for a sequence $\\theta_n\\in\\Xi_n$, as in the statement of Lemma 17, and the second equality uses $\\varphi(1,\\beta_n)=\\varphi(0,\\beta_n)$.", "evidence": "The evidence is a derivation from the definitions of the Supplementary Material. By the definition (D.1), I have $w_i(a,\\beta)=y_i(d_i(a))-\\beta d_i(a)$, so that $Z_i(Y_i-\\beta D_i)=Z_iw_i(1,\\beta)$ and $(1-Z_i)(Y_i-\\beta D_i)=(1-Z_i)w_i(0,\\beta)$. Substituting these identities into the definition (D.4) of $\\tilde\\tau_{\\mathrm{adj}}(\\beta)$, I obtain the first line of the corrected expansion below. The proof shows that $\\varphi(1,\\beta_n)=\\varphi(0,\\beta_n)$. Since $n_1^{-1}\\sum_{i=1}^nZ_i=n_0^{-1}\\sum_{i=1}^n(1-Z_i)=1$, subtracting $\\varphi(1,\\beta_n)$ inside the first average and $\\varphi(0,\\beta_n)$ inside the second leaves the difference unchanged, which gives the second line. The third line follows from the definition $\\varepsilon_i(a,\\beta)=w_i(a,\\beta)-\\varphi(a,\\beta)-x_i'\\gamma(a,\\beta)$.\n\nFor (E.29), Assumption 5(iii) states that $\\sum_{i=1}^nx_i=0$, which implies that $\\sum_{i=1}^nZ_ix_i^{\\circ}=-\\sum_{i=1}^n(1-Z_i)x_i^{\\circ}$. The printed left side of (E.29) is then the negative of the intended one, and the printed statement is true as it stands. The slip is in the expression and not in the limit.", "problem": "Display (E.29) is introduced as the control-arm analogue of the display before it, which concerns the treated-arm remainder (E.22). The control-arm remainder (E.23) begins with $n_0^{-1}\\sum_{i=1}^n(1-Z_i)x_i^{\\circ\\prime}$, and (E.29) prints $Z_i$ in that sum. In the next display the index $a$ is not bound: a treated unit reveals $w_i(1,\\beta_n)$ and a control unit reveals $w_i(0,\\beta_n)$. In the last line of that display the factor $(1-Z_i)$ stands outside the sum over $i$.", "what_survives": "Lemma 17 and every result that uses it are unaffected. Only the three expressions change."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N009", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The three timing rows of Table 4, labelled Median Time (1000), Median Time (5000) and Median Time (10000), followed by the caption. A red outline marks the word average in the phrase: Coverage rates, median lengths, and average computation times. The caption ends: Median time is reported in seconds, rounded to the nearest integer.", "caption": "Supplement, p. 48 (PDF p. 28 of the Supplement), last rows and caption of Table 4. Outlined: the word “average” in the first sentence of the caption; the rows above and the last sentence say “Median”.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N011.png"}], "formal_release": null, "formal_statements": [], "headline": "The captions of Supplement Tables 4 and 5 call the computation times “average” in their first sentence, while the row labels and the last sentence call them “median”; the median is meant.", "id": "P002-N011", "location": "Supplement, captions of Tables 4 and 5, pp. 48–49 (PDF pp. 28–29 of the Supplement)", "pdf_page": 28, "printed_page": "48–49", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Table 4: Coverage rates, median lengths, and \\hl{average} computation times for randomization-based confidence sets across simulation designs with $N_1=50$ and $N_0=50$ and with varying numbers of randomization draws ($K=1000,5000,10000$). Median time is reported in seconds, rounded to the nearest integer.\n\nRow labels of the same table: ``Median Time (1000)'', ``Median Time (5000)'', ``Median Time (10000)''. The caption of Table~5 has the same wording with $N_1=30$ and $N_0=70$.", "corrected": "Table 4: Coverage rates, median lengths, and \\hl{median} computation times for randomization-based confidence sets across simulation designs with $N_1=50$ and $N_0=50$ and with varying numbers of randomization draws ($K=1000,5000,10000$). Median time is reported in seconds, rounded to the nearest integer.\n\nThe caption of Table~5 changes in the same way.", "intended_meaning": "The row labels and the last sentence of each caption name the median, and p.~19 of the article describes the same rows as median computation times. The word ``average'' in the first sentence is loose wording for that summary. This note concerns the wording only. Whether the printed times are medians over the simulations is a separate question, treated in \\fid{P002-N012}."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N011", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 4 with seven columns, headed L5, L10, L25, L50 and L90 per cent, IR1 and IR2, and nine rows: Coverage, Median Length and Median Time, each for 1000, 5000 and 10000 draws. A red box marks the three rows Median Time (1000), Median Time (5000) and Median Time (10000), with entries 9, 8, 8, 11, 8, 9, 8; 511, 511, 472, 640, 563, 552, 529; and 3640, 3651, 3561, 4863, 3904, 3227, 3650. The caption below ends 'Median time is reported in seconds, rounded to the nearest integer.'", "caption": "Supplementary Material, p. 48 (PDF p. 28 of the supplement): Table 4 and its caption. The marked rows are the three rows “Median Time”.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N012.png"}, {"alt": "Crop of Table 5, laid out as Table 4. A red box marks the three rows Median Time (1000), Median Time (5000) and Median Time (10000), with entries 9, 10, 9, 6, 10, 8, 7; 520, 614, 507, 350, 529, 546, 493; and 3297, 3835, 3639, 2670, 3872, 3560, 3303. The caption below ends 'Median time is reported in seconds, rounded to the nearest integer.'", "caption": "Supplementary Material, p. 49 (PDF p. 29 of the supplement): Table 5 and its caption. The marked rows are the three rows “Median Time”.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N012-b.png"}, {"alt": "Listing of source code with line numbers. Line 111 stores the elapsed time of each file in the first column of output3. Red boxes mark median(output1[1]) in line 130, median(output2[1]) in line 132 and median(output3[1]) in line 134. Below, line 171 of extract_results3 reads return(apply(output,2,median)).", "caption": "Source code of the research repository, not a page of the article: 3_merge_results_functions.R, lines 110–111 and 130–135 of extract_results2, and lines 170–171 of extract_results3. Each marked expression takes the median of a single element of the matrix of results; the function that assembles the times of Table 1 takes the median of every column.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N012-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The function that assembles the rows “Median Time” of Tables 4 and 5 in the Supplementary Material returns the time of the first replication and not the median over replications; the three lines should take the median of the first column. Whether the published rows were produced by this function is not established.", "id": "P002-N012", "location": "Supplementary Material, Tables 4 and 5, rows “Median Time”, pp. 48–49 (PDF pp. 28–29 of the supplement). Research repository RandomizaionIV_ResearchCode (commit 4d919c4): 3_merge_results_functions.R, function extract_results2, lines 130, 132 and 134. The code is not printed in the article.", "pdf_page": 28, "printed_page": "48–49", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{3\\_merge\\_results\\_functions.R}, lines 130, 132 and 134:\n\n\\texttt{K1000=c(median(output1\\hl{[1]}), apply(output1[,2:3],2,mean))}\n\n\\texttt{K5000=c(median(output2\\hl{[1]}), apply(output2[,2:3],2,mean))}\n\n\\texttt{K10000=c(median(output3\\hl{[1]}), apply(output3[,2:3],2,mean))}\n\n\\begin{audittable}{lrrrrrrr}{Supplementary Material, Tables 4 and 5, pp.~48--49: the rows ``Median Time''. Both captions end ``Median time is reported in seconds, rounded to the nearest integer.''}\nRow & L5\\% & L10\\% & L25\\% & L50\\% & L90\\% & IR1 & IR2 \\\\\nTable 4, Median Time (1000) & 9 & 8 & 8 & 11 & 8 & 9 & 8 \\\\\nTable 4, Median Time (5000) & 511 & 511 & 472 & 640 & 563 & 552 & 529 \\\\\nTable 4, Median Time (10000) & 3640 & 3651 & 3561 & 4863 & 3904 & 3227 & 3650 \\\\\nTable 5, Median Time (1000) & 9 & 10 & 9 & 6 & 10 & 8 & 7 \\\\\nTable 5, Median Time (5000) & 520 & 614 & 507 & 350 & 529 & 546 & 493 \\\\\nTable 5, Median Time (10000) & 3297 & 3835 & 3639 & 2670 & 3872 & 3560 & 3303 \\\\\n\\end{audittable}", "corrected": "\\texttt{3\\_merge\\_results\\_functions.R}, lines 130, 132 and 134:\n\n\\texttt{K1000=c(median(output1\\hl{[,1]}), apply(output1[,2:3],2,mean))}\n\n\\texttt{K5000=c(median(output2\\hl{[,1]}), apply(output2[,2:3],2,mean))}\n\n\\texttt{K10000=c(median(output3\\hl{[,1]}), apply(output3[,2:3],2,mean))}\n\nIf this function produced the published rows, the six rows are computed again from the elapsed times of all replications, and an entry is replaced if it changes. The replication files do not contain the times from which replacement values could be computed.", "evidence": "The evidence is an exact check of the code on constructed inputs. I ran the unchanged function on three constructed replications. In the order in which the files are read, their times are $(9,1,2)$ for $1000$ draws, $(90,10,20)$ for $5000$ draws and $(900,100,200)$ for $10000$ draws. The function returns $9$, $90$ and $900$. The medians are $2$, $20$ and $200$. When the second file is read first, the function returns $1$, $10$ and $100$, and the medians are the same. With the three corrected lines the function returns $2$, $20$ and $200$, and its coverage and length summaries are unchanged.\n\nThe original times of the individual replications are not in the replication files, and nothing there ties this version of the function to the published tables. I cannot say which of the $42$ published entries, if any, differs from its median.", "problem": "In R, a single index applied to a matrix selects one element. The expression \\texttt{output1[1]} is the element in the first row and first column of the matrix \\texttt{output1}, which holds the elapsed time of the first file read, and the median of one number is that number. The three timing entries that \\texttt{extract\\_results2} returns are therefore the times of the first replication, whatever the times of the other replications. The script \\texttt{3\\_merge\\_results\\_grace2.R} (lines 73--75 and 87--89) takes these entries for seven designs and two splits of the arms, which makes $42$ entries laid out as the rows ``Median Time'' of Tables 4 and 5.", "what_survives": "The error concerns the timing entries alone; the coverage and length summaries of the same function do not use the expressions in question. The times of Table 1 come from a different function, \\texttt{extract\\_results3}, which returns the median of every column. The statement on p.~19 that Algorithm 4 finishes in under an hour with $10^4$ random assignments describes Table 1, in which all seven entries for Algorithm 4 with $M=10^4$ are below $3600$ seconds (the largest is $3163$). Eight of the $14$ entries for $10000$ draws in Tables 4 and 5 exceed $3600$ seconds. This neither shows that they are times of single replications nor contradicts the statement about Table 1."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N012", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Header and six rows of Table 5 for the seven simulation designs L5, L10, L25, L50, L90, IR1 and IR2. In the row Coverage (10000), a red outline marks 0.949 and 0.950 in columns IR1 and IR2; the row Coverage (5000) above has the same two values. In the row Median Length (10000), a red outline marks 0.425 in column IR2; the row Median Length (5000) above also has 0.425.", "caption": "Supplement Table 5, p. 49 (PDF p. 29 of the Supplement), coverage and median-length rows. Outlined: the IR1 and IR2 coverage rates and the IR2 median length in the 10000-draw rows, which equal the entries of the 5000-draw rows above them.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N013.png"}, {"alt": "The eight rows of Table 3. In the row Coverage Rate (Rand), a red outline marks 0.952 and 0.951 in columns IR1 and IR2. In the row Median Length (Rand), a red outline marks 0.422 in column IR2. The L columns of these two rows read 0.949, 0.949, 0.954, 0.948, 0.952 and Inf, Inf, 13.667, 1.495, 0.339.", "caption": "Supplement Table 3, p. 48 (PDF p. 28 of the Supplement), for the same design with $10^4$ randomization draws. Outlined: the IR1 and IR2 coverage rates and the IR2 median length of the randomization-based sets, which differ from the outlined entries of Table 5.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N013-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Three entries in the 10000-draw rows of Supplement Table 5 repeat the 5000-draw values; with the values of Table 3, the IR1 and IR2 coverage rates are 0.952 and 0.951 and the IR2 median length is 0.422.", "id": "P002-N013", "location": "Supplement Table 5, p. 49 (PDF p. 29 of the Supplement), rows “Coverage (10000)” and “Median Length (10000)”, columns IR1 and IR2; compare Supplement Table 3, p. 48 (PDF p. 28 of the Supplement)", "pdf_page": 29, "printed_page": "49", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lrrrrrrr}{Supplement Table~5 ($N_1=30$, $N_0=70$), four of its rows.}\nSimulation Design & L5\\% & L10\\% & L25\\% & L50\\% & L90\\% & IR1 & IR2 \\\\\nCoverage (5000) & 0.953 & 0.956 & 0.953 & 0.955 & 0.955 & 0.949 & 0.950 \\\\\nCoverage (10000) & 0.949 & 0.949 & 0.954 & 0.948 & 0.952 & \\hl{0.949} & \\hl{0.950} \\\\\nMedian Length (5000) & Inf & Inf & 14.899 & 1.526 & 0.341 & Inf & 0.425 \\\\\nMedian Length (10000) & Inf & Inf & 13.667 & 1.495 & 0.339 & Inf & \\hl{0.425} \\\\\n\\end{audittable}\n\n\\begin{audittable}{lrrrrrrr}{Supplement Table~3 (the same design, $10^4$ randomization draws), two of its rows.}\nSimulation Design & L5\\% & L10\\% & L25\\% & L50\\% & L90\\% & IR1 & IR2 \\\\\nCoverage Rate (Rand) & 0.949 & 0.949 & 0.954 & 0.948 & 0.952 & 0.952 & 0.951 \\\\\nMedian Length (Rand) & Inf & Inf & 13.667 & 1.495 & 0.339 & Inf & 0.422 \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lrrrrrrr}{Supplement Table~5, the two 10000-draw rows, with the IR entries of Table~3.}\nSimulation Design & L5\\% & L10\\% & L25\\% & L50\\% & L90\\% & IR1 & IR2 \\\\\nCoverage (10000) & 0.949 & 0.949 & 0.954 & 0.948 & 0.952 & \\hl{0.952} & \\hl{0.951} \\\\\nMedian Length (10000) & Inf & Inf & 13.667 & 1.495 & 0.339 & Inf & \\hl{0.422} \\\\\n\\end{audittable}\n\nIn \\texttt{3\\_merge\\_results\\_grace2.R}, replace \\texttt{[[2]]} by \\texttt{[[3]]} for the two IR designs on lines 83 and 86, and likewise on line 72. If Table~5 was meant to report a separate simulation run, its IR entries should instead be regenerated from the summaries of that run for $10^4$ draws.", "evidence": "The evidence is an exact counterexample for the code and a recalculation for the tables. Four synthetic files of results were built in which the 5000-draw sets have coverage $1/4$ and median length $4$, and the sets for $10^4$ draws have coverage $3/4$ and median length $8$. On these files, the unmodified aggregation function \\texttt{extract\\_results2} and the row assignments of the script return $(3/4,3/4,3/4,3/4,3/4,1/4,1/4)$ for the 10000-draw coverage row of Table~5 and $(8,8,8,8,8,4,4)$ for its median-length row. With \\texttt{[[3]]} in the IR positions they return $3/4$ and $8$ in every column. Applying the selectors of the script to the printed summaries, with the 5000-draw rows of Table~5 as element 2 and the randomization rows of Table~3 as element 3, reproduces all 14 printed cells of the two 10000-draw rows. The simulation outputs behind the tables are not among the replication files. The replacement values are taken from Table~3 and are not recomputed from the simulations.", "problem": "Table~3 and the 10000-draw rows of Table~5 report the same quantities for the same design, and they agree in the five L columns. In columns IR1 and IR2 they differ, and Table~5 prints the values of its own 5000-draw rows. The script that assembles both tables, \\texttt{3\\_merge\\_results\\_grace2.R}, accounts for this pattern. For the 10000-draw coverage and median-length rows of Table~5, it selects the list element \\texttt{[[3]]}, the summary for $10^4$ draws, for the five L designs and the element \\texttt{[[2]]}, the summary for 5000 draws, for the two IR designs (lines 83 and 86). The rows of Table~3 use \\texttt{[[3]]} throughout (lines 55 and 57). Four cells of Table~5 are selected from the wrong summary. Three of them differ from Table~3 in print; the fourth, the IR1 median length, is Inf in both tables. The median-length row for 10000 draws in Table~4 has the same slip on line 72, with no difference visible at the printed precision.", "what_survives": "The three entries change by $0.003$ and $0.001$ in coverage and by $0.003$ in median length. The remark on p.~19 of the article, that the results do not differ substantially across the numbers of draws, is unchanged. Table~3, the coverage and length rows of Table~1, the theorems, the algorithms and the data applications do not use these selectors."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N013", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The last three rows of Table 6 (Wald Est., 2SLS CI, Randomization CI) for Bridgeport, Raleigh and Minneapolis, followed by the first three lines of the caption. A red outline marks the opening words: Table 6: Applications to the Green et al. (2003) dataset.", "caption": "Supplement, p. 49 (PDF p. 29 of the Supplement), last rows and caption of Table 6, the first of the five tables concerned. Outlined: the opening words of the caption.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N014.png"}], "formal_release": null, "formal_statements": [], "headline": "Supplement Tables 6–10 cannot be regenerated in full from the replication files, which lack the saved output of the application, the data for four of the six cities and the outputs of the subsampling exercise; no printed entry is shown to be wrong.", "id": "P002-N014", "location": "Supplement Tables 6–10 and Sections K–L, pp. 49–52 (PDF pp. 29–32 of the Supplement)", "pdf_page": 29, "printed_page": "49–52", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Table 6: Applications to the Green et al. (2003) dataset for cities Bridgeport, Raleigh, and Minneapolis.\n\nTable 7: Applications to the Green et al. (2003) dataset for cities Detroit, Columbus, and St. Paul.\n\nTable 8: Average sample sizes as a function of the treated-group subsampling rate.\n\nTable 9: Average compliance rates as a function of the treated complier subsampling rate.\n\nTable 10: The median of the difference in length between the randomization-based confidence sets and the 2SLS confidence intervals, relative to the length of the 2SLS confidence intervals.", "corrected": "A note with the replication files could read: ``The files include the derived data for Bridgeport and Columbus only. Regenerating Supplement Tables~6--10 also requires the six-city data of Green et al. (2003), the saved results of the six-city analysis with their assignment draws or seeds, and the outputs of the Bridgeport subsampling for each design, with the code and seeds that generated them.'' No change to an entry of the tables is called for.", "intended_meaning": "These tables report the application to six cities and Monte Carlo summaries of a subsampling exercise, and nothing in them is shown to be false. The note concerns what the replication files allow. The only saved output of the application is an empty list. The files hold the derived data for Bridgeport and Columbus, but not the data file for the six cities or the data for Raleigh, Minneapolis, Detroit and St.~Paul, and they hold none of the subsampling outputs that the summarizing script reads. The entries that can be recomputed agree with the tables. For Bridgeport and Columbus, the point estimates, the 2SLS intervals, the compliance rates and the first-stage F statistics recompute to the printed rounding. The sample sizes of Table~8 (509.7, 408.0, 306.0) recompute at the printed precision, and the compliance rates of Table~9 (0.167, 0.108, 0.039) are compatible with the expectations under the design. The randomization-based sets, the results for the other four cities and the medians of Table~10 were not recomputed. A separate defect that stops the application script is recorded as \\fid{P002-E017}."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N014", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the second half of Remark 7, from 'which, by Assumption 6, is equivalent to' and the displayed product equation to the end of the remark. Red boxes mark the sentence 'Hence, the solution set is the zero set of a polynomial function in beta with degree at most 4.'", "caption": "Remark 7, p. 14. The marked sentence is the statement that the code is intended to implement: the intersections are the real zeros of a polynomial of degree at most 4.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N015.png"}, {"alt": "Listing of lines 137 to 165 of the function AR_intersection. A red box marks line 140, 'tol <- 1e-8'. A second red box marks lines 146 and 147, an if statement that tests whether the absolute values of v0, v1, v2, v3 and v4 are all below tol and, if so, sets real_roots to an empty vector. The following else-if branches test fewer coefficients against tol and call polyroot on a polynomial of lower degree.", "caption": "The rilate package, R/calculate_intersections.R, lines 137–165 (a listing of the replication code, with line numbers added). The marked lines fix the threshold and return no intersection when all five coefficients are below it.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N015-b.png"}, {"alt": "Two line charts with logarithmic axes. The horizontal axis is the sample size n and the vertical axis is the largest absolute coefficient over all pairs, labelled from 10 to the minus 10 to 10 to the minus 3. A horizontal black line marks the tolerance 10 to the minus 8, and the region below it is shaded. In each panel a blue line with circles is M equal to 1000 and an orange line with squares is M equal to 10000; both fall steadily as n grows. In the left panel, Columbus composition, the points at n equal to 2424, 4848 and 9696 are filled and above the tolerance, and the points from n equal to 19392 to 155136 are open and below it. In the right panel, equal numbers of the four types, the points are filled up to n equal to 25600; at 51200 the M equal to 1000 point is open and the M equal to 10000 point is filled, and at 102400 both are open.", "caption": "The largest absolute coefficient of the intersection polynomials, over all pairs of simulated assignments, against the sample size, for two populations with binary outcome and treatment and balanced assignment. Left: the composition of the Columbus data, replicated $1$ to $64$ times. Right: equal numbers of the four types $(Y,D)$. Below the horizontal line every pair is classified as identical, the code finds no intersection, and its empty-grid branch returns the whole real line. The exact set (14) is a bounded interval at every point shown.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N015-tolerance.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication code uses a fixed absolute threshold of $10^{-8}$ to decide the degree and identity of intersection polynomials, and it can then miss intersections of AR functions and treat distinct AR functions as identical; with binary outcome and treatment and a large sample it returns the whole real line where the confidence set (14) is a bounded interval.", "id": "P002-N015", "location": "Remark 7, p. 14, and Step 1 of Algorithms 1 and 2, pp. 15–16, as implemented in the replication code: rilate package, R/calculate_intersections.R, lines 140–165; research code, AR_intersection in 2_AR_inversion_algo1.R and find_root in cpp_implementations/auxiliary_functions.cpp", "pdf_page": null, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Remark 7, p.~14: ``Hence, the solution set is the zero set of a polynomial function in $\\beta$ with degree at most 4. The solution set may be empty, a discrete set of size at most 4, or the real line. When the solution set is the real line, the two functions agree over the real line and are the same function.''\n\nThe \\texttt{rilate} package, \\texttt{R/calculate\\_intersections.R}, lines 140 and 146--147, where \\texttt{v0}, \\ldots, \\texttt{v4} are the coefficients of the polynomial:\n\n\\texttt{tol <- \\hl{1e-8}}\n\n\\texttt{if (abs(v0)<tol \\& abs(v1)<tol \\& abs(v2)<tol \\& abs(v3)<tol \\& abs(v4)<tol)\\{}\n\n\\texttt{real\\_roots <- c()}\n\nThe branches that follow use the same threshold to choose the degree of the polynomial that is solved. The research code has the same tests with the same threshold.", "corrected": "The printed text needs no change. In the code, replace the fixed threshold by \\hl{exact or certified tests of $v_j=0$}, both for the identity of two AR functions and for the degree of $P_{ms}$; keep every nonzero coefficient; and compute all real zeros of $N_mQ_s-N_sQ_m$. The coefficients may be rescaled for the numerical solution, but no term may be dropped. When no pair of simulated AR functions intersects, the code must carry out the comparison of Step 1 of Algorithm 2 (\\fid{P002-E015}).\n\nThe set (14) can also be computed without the pairwise intersections. Under Assumption 6, with $P_{om}=N_oQ_m-N_mQ_o$ and $k=\\lceil(1-\\alpha)M\\rceil$,\n\\[\n\\beta\\in\\widehat I^*_{1-\\alpha}(Z)\\iff\\#\\{m:\\Delta_m^{*2}(\\beta)<\\Delta^2(\\beta)\\}\\le k-1\\iff\\#\\{m:P_{om}(\\beta)\\le0\\}\\ge M-k+1,\n\\]\nso the set is determined by the signs of the \\hl{nonzero} polynomials $P_{om}$ between their real zeros \\hl{and at those zeros}; \\hl{identically zero polynomials count toward the inequality for every $\\beta$}.", "evidence": "I use exact counterexamples, run through the unmodified R and C++ code, and recalculations using exact zeros and roots of the calculated AR functions.\n\nFirst, consider the coefficient vectors $(1,0,0,1,0,1)$ and $(1,-2,1,1,0,1)$, which represent $\\beta^2/(\\beta^2+1)$ and $(\\beta-1)^2/(\\beta^2+1)$. Their only intersection is $\\beta=1/2$, and $P=(2\\beta-1)(\\beta^2+1)$. After both vectors are multiplied by $10^{-5}$ each function is unchanged, the coefficients of $P$ are $(-10^{-10},2\\times10^{-10},-10^{-10},2\\times10^{-10},0)$, and the root finders of the \\texttt{rilate} package and of the research code (R and C++) return no intersection. The interval routine of the package then returns the whole real line where the first function is at most the second exactly on $(-\\infty,1/2]$.\n\nSecond, I take eight units with $Y=(0,1,2,3,4,5,6,8)$ and $D=Z=(0,0,0,0,1,1,1,1)$, and $M=40$ simulated assignments consisting of $20$ copies each of $(1,0,0,0,0,1,1,1)$ and $(0,1,0,0,0,1,1,1)$. All denominators are positive, the two simulated AR functions intersect only at $\\beta=5$, and the set (14) at level $0.95$ is $[3,14/3]$. The package returns this interval to within $10^{-12}$. When the outcomes are multiplied by $10^{-7}$, the set (14) is $[3\\times10^{-7},\\,(14/3)\\times10^{-7}]$, the code loses the intersection at $5\\times10^{-7}$, and \\texttt{AR\\_algo2} returns the whole real line.\n\nThird, no rescaling is needed. Let $n$ be a multiple of $8$ such that $n>8$, and consider $n/4$ units of each type $(Y,D)\\in\\{(0,0),(1,0),(0,1),(1,1)\\}$ with $n/2$ treated. Let the observed treatment group contain $0$, $n/8$, $n/8$ and $n/4$ units of these types, the first simulated one $n/8-1$, $n/8+1$, $n/8$ and $n/8$, and the second $n/8-1$, $n/8$, $n/8+1$ and $n/8$, with $20$ copies of each simulated assignment. In the representation used by the code, the numerators are $N_o=(\\beta-1)^2/4$, $N_1=16/n^2$ and $N_2=16\\beta^2/n^2$, the denominators are $Q_j=(\\beta^2+1-N_j)/n>0$, and\n\\[\nP_{12}(\\beta)=\\frac{16}{n^3}\\,(1-\\beta^4),\\qquad\\widehat I^*_{0.95}(Z)=\\Big[1-\\frac8n,\\ \\frac{n}{n-8}\\Big].\n\\]\nThe two simulated AR functions intersect at $\\pm1$. The code discards $P_{12}$ when $16/n^3<10^{-8}$, that is, for $n\\ge1170$. At $n=1168$ it finds both intersections and returns a bounded set. At $n=1176$ the set (14) is $[146/147,\\,147/146]$, and the package returns $(-\\infty,\\infty)$.\n\nFourth, independent simulated assignments lead to the same state. For the pooled composition of the Columbus data ($2075$, $190$, $130$ and $29$ units of the four types) replicated $8$ times, $n=19\\,392$, with balanced assignment and $M=1000$ or $M=10^4$ independent simulated assignments, the largest $|v_j|$ over all pairs is below $10^{-8}$, while (14) equals $[-14/323,\\,50/629]$ and $[-14/323,\\,50/621]$ respectively. At $n=9696$ some pair is still above the threshold. There is no threshold in $n$ alone: for equal numbers of the four types the first tested size is $51\\,200$ at $M=1000$ and $102\\,400$ at $M=10^4$.\n\\begin{audittable}{lll}{The confidence set (14), or (F.4) with the covariate, and the set returned by the code, for the same data and simulated assignments.}\nExample & Set (14) or (F.4) & Returned by the code \\\\\nEight units, outcomes multiplied by $10^{-7}$ & $[3\\times10^{-7},\\,(14/3)\\times10^{-7}]$ & $(-\\infty,\\infty)$ \\\\\nBinary family, $n=1176$, $M=40$ & $[146/147,\\,147/146]$ & $(-\\infty,\\infty)$ \\\\\nColumbus composition, $n=19\\,392$, $M=1000$ & $[-14/323,\\,50/629]$ & $(-\\infty,\\infty)$ \\\\\nColumbus data with covariate, $M=10^4$ (rounded) & $[-0.041574,\\,0.266049]$ & $[-0.041607,\\,0.266049]$\n\\end{audittable}\nThe first and second rows were run through the unmodified code. The fourth used the same root and inversion decisions with optimized storage and grid searches. In the third row the returned set follows from the classification of all pairs, which was computed exactly. In the fourth row the set is the regression-adjusted analogue (F.4) of (14).", "problem": "Let $\\Delta_m^{*2}=N_m/Q_m$ be the squared AR function of a simulated assignment $m$, where $N_m$ and $Q_m$ are the quadratics of (16). Under Assumption 6, two such functions meet exactly at the real zeros of $P_{ms}=N_mQ_s-N_sQ_m=\\sum_{j=0}^4v_j\\beta^j$, and they are the same function exactly when all five coefficients $v_j$ are zero. This is Remark 7. The code replaces ``$v_j=0$'' by ``$|v_j|<10^{-8}$'', both to decide whether two functions are the same and to choose the degree of the polynomial that it solves. The coefficients $v_j$ have no fixed scale. Multiplying the coefficient vectors of both functions by $t>0$ multiplies every $v_j$ by $t^2$ and leaves the functions unchanged, and in the binary examples below the $v_j$ decrease as $n$ grows. The code discards nonzero terms that are below the threshold, which can change the set of real zeros. When all five coefficients of a pair are below the threshold, it reports no intersection for that pair. When this happens for every pair, the grid of intersections is empty and the code for Algorithm 2 returns the whole real line (\\fid{P002-E015}).\n\nThe threshold is reached on unrescaled data in the settings of the paper. In recalculations with the same simulated assignments, nonzero terms were discarded in one saved data set for each of the fourteen combinations of design and allocation of Section 6 ($M=1000$) and in the Bridgeport and Columbus data, with and without the covariate. In one recalculation for Columbus without the covariate at $M=1000$, $51\\,705$ pairs of distinct functions are classified as identical and the code reports $621\\,548$ real roots, compared with $977\\,137$ for the full polynomials, counted separately for each pair. Every endpoint returned in these runs still agrees with the exact set (14), or (F.4) with the covariate, to three decimals. The largest error is approximately $0.0000336$, at the lower endpoint for Columbus with the covariate at $M=10^4$, where the code returns approximately $-0.041607$ and the exact value is approximately $-0.041574$. The failure is of a different kind when outcome and treatment are binary and the sample is large: the code returns the whole real line where (14) is a short interval. This occurs from $n=1176$ in a constructed family and at $n=19\\,392$ for the composition of the Columbus data with independent simulated assignments.", "what_survives": "Remark 7, Lemma 1, Algorithms 1 to 4 as printed and Theorem 3 use the actual intersections and are not contradicted by this code defect. The definitions of the confidence sets and the finite-sample and asymptotic theory are unaffected. No reported value is shown to change: every endpoint checked in the Section 6 and city recalculations agrees with exact inversion at the three decimals printed in Supplementary Tables 6 and 7. The checks do not cover the application to the data of Angrist et al., the cities Raleigh, Minneapolis, Detroit and St.~Paul, or the Bridgeport subsamples, whose inputs are not in the replication files, and they do not use the simulated assignments of the published runs. The research code also contains a C++ routine in exact integer arithmetic. It tests for exact zero and is not affected. A user who applies the replication code to a large experiment with binary outcome and treatment can obtain the whole real line where (14) is bounded."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N015", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of the first seventeen lines of the function SolveP3, which solves the cubic x cubed plus a x squared plus b x plus c = 0. A red box marks line 83, 'x[0] = root3(-2 * r);', inside the branch 'if (fabs(q) less than eps)'. Line 78 in the same branch reads 'x[0] = x[1] = x[2] = -a/3;'.", "caption": "Research code, cpp_implementations/poly34.cpp, lines 71–87, typeset from the source file with line numbers added; the e-mail addresses in two comments are replaced by <...>. The marked line assigns the real root without subtracting $a/3$. Line 78, for a triple root, includes the shift.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N016.png"}], "formal_release": null, "formal_statements": [], "headline": "In the research code, the branch of the cubic solver for $q=0$ and $r\\ne0$ must subtract $a/3$ from the real root it returns; without this shift an intersection of two AR functions is misplaced.", "id": "P002-N016", "location": "Research code in the replication files (commit 4d919c4), folder cpp_implementations, file poly34.cpp, function SolveP3, lines 76–87. No passage of the article; the step implemented is the root finding of Remark 7, p. 14", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Lines 83--86 of \\texttt{poly34.cpp}, comments omitted:\n\n\\texttt{x[0] = \\hl{root3(-2 * r)};}\n\n\\texttt{x[1] = x[0] * 0.5;}\n\n\\texttt{x[2] = x[0] * sqrt(3.) / 2;}\n\n\\texttt{return(1);}", "corrected": "\\texttt{x[0] = root3(-2 * r);}\n\n\\texttt{x[1] = x[0] * 0.5;}\n\n\\texttt{x[2] = x[0] * sqrt(3.) / 2;}\n\n\\texttt{\\hl{x[0] -= a / 3;}}\n\n\\texttt{return(1);}\n\nThe added line corrects the real root, which is the value the intersection routines use. I did not examine the complex pair returned in \\texttt{x[1]} and \\texttt{x[2]}.", "evidence": "The evidence is an exact counterexample. The function, compiled as distributed and called with $(a,b,c)=(3,3,2)$, for which $q=0$ and $r=1/2$, returns one real root, exactly $-1$. Since\n\\[\nx^3+3x^2+3x+2=(x+2)(x^2+x+1),\n\\]\nthe only real root is $-2$, and the cubic equals $1$ at $-1$. This cubic arises from data. For a data set of 16 units with binary $D$ and two assignments with 8 treated units each, the squared AR functions are $16(\\beta+3)^2/(15\\beta^2+10\\beta+7)$ and $16(\\beta+1)^2/(15\\beta^2+14\\beta+15)$. Both denominators are positive on the real line. Cross-multiplying, I find the crossing polynomial $1024(\\beta^3+3\\beta^2+3\\beta+2)$. The two functions meet only at $\\beta=-2$, where both equal $16/47$. For this pair the intersection routine passes $(a,b,c)=(3,3,2)$ to the solver and obtains the point $-1$, where the two functions equal $16/3$ and $0$. With the line of the correction inserted, the compiled function returns $-2$.", "problem": "The function solves $x^3+ax^2+bx+c=0$ by the substitution $x=y-a/3$, which gives $y^3-3qy+2r=0$ with $q=(a^2-3b)/9$ and $r=(2a^3-9ab+27c)/54$. When $q=0$ and $r\\ne0$, the real root is $y=(-2r)^{1/3}$, and the real root of the original cubic is $x=(-2r)^{1/3}-a/3$. The branch for $|q|<10^{-14}$ returns $(-2r)^{1/3}$, and the other branches of the function subtract $a/3$. The routines that compute the intersections of two AR functions take the returned value as an intersection. When the crossing polynomial of two AR functions is a cubic of this kind, they record a point that is not an intersection and omit the true one. The solver is third-party code included in the research code; its header attributes it to S.~I.~Khashin.", "what_survives": "The cross-multiplication argument of the article, Lemma 1, the algorithms as printed and the theorems on coverage are not affected. The R package \\texttt{rilate} finds the roots with \\texttt{polyroot} and returns $-2.0000000000000124$ for this example. The branch is reached only when $q$ vanishes to within $10^{-14}$. No reported number is shown to change, and the simulations and applications were not recomputed with a corrected solver."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N016", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Header and eight rows of Table 1 for the seven simulation designs L5, L10, L25, L50, L90, IR1 and IR2. Red outlines mark the row Coverage rate (2SLS), with entries 0.697, 0.836, 0.922, 0.956, 0.952, 0.840, 0.942, and the row Median length (2SLS), with entries 6.618, 6.943, 2.750, 1.139, 0.298, 3.034, 0.322.", "caption": "Table 1, p. 19 (PDF p. 19), coverage and median-length rows. Outlined: the two 2SLS rows, which have the layout of the rows that the aggregation function assembles.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N018.png"}, {"alt": "Monospaced listing of code lines 88 to 113, headed ORIGINAL CODE TEXT (not a paper screenshot). The block is labelled K10000. Red outlines mark the word result2 in the two assignments of the 2SLS interval endpoints on lines 100 and 101. Lines 111 and 113 read the time and the randomization interval from result3.", "caption": "Lines 88–113 of 3_merge_results_functions.R in the replication files, rendered as text (not a page of the article). Outlined: result2 on lines 100 and 101, in the block for $10^4$ draws; lines 111 and 113 of the same block read result3.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N018-code.png"}], "formal_release": null, "formal_statements": [], "headline": "In the replication code, the summary built for the runs with $10^4$ randomization draws takes its 2SLS interval from the 5000-draw run; lines 100 and 101 should read result3.", "id": "P002-N018", "location": "Replication code, 3_merge_results_functions.R, function extract_results2, lines 100–101; the related published rows are “Coverage rate (2SLS)” and “Median length (2SLS)” of Table 1, p. 19 (PDF p. 19)", "pdf_page": 19, "printed_page": "19", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Replication code (not a passage of the article), in the block that builds \\texttt{output3}, the summary for $10^4$ draws:\n\nline 100: \\texttt{tsls\\_lo = \\hl{result2}[[1]]['tsls\\_t\\_CI\\_lo']\\$tsls\\_t\\_CI\\_lo}\n\nline 101: \\texttt{tsls\\_up = \\hl{result2}[[1]]['tsls\\_t\\_CI\\_up']\\$tsls\\_t\\_CI\\_up}\n\nline 111: \\texttt{output3[i,1] = result3[[1]]\\$time4['elapsed']}\n\nline 113: \\texttt{permu\\_CI = result3[[1]]\\$permu\\_CI4}\n\nCaption of Table~1: ``Rows 5--8 report the median sizes of these confidence sets. For the randomization-based confidence sets, we use $10^4$ randomization draws.''", "corrected": "line 100: \\texttt{tsls\\_lo = \\hl{result3}[[1]]['tsls\\_t\\_CI\\_lo']\\$tsls\\_t\\_CI\\_lo}\n\nline 101: \\texttt{tsls\\_up = \\hl{result3}[[1]]['tsls\\_t\\_CI\\_up']\\$tsls\\_t\\_CI\\_up}\n\nIf this function supplied the 2SLS rows of Table~1 or of Supplement Table~3, those rows should be recomputed from the saved \\texttt{result3} outputs; the size and direction of any change are not known. Alternatively, the code may keep \\texttt{result2}, provided its documentation says that the 2SLS summaries come from the 5000-draw run.", "evidence": "The evidence is a check of the code and an exact counterexample. The unmodified function was run on two identical test files in which the 2SLS interval is $[-4,4]$ in \\texttt{result1}, $[1,3]$ in \\texttt{result2} and $[-2,2]$ in \\texttt{result3}, with true value $\\tau=0$. In the third summary it returns 2SLS coverage $0$ and median 2SLS length $2$, which are the values of \\texttt{result2}. The values of \\texttt{result3} are $1$ and $4$. With \\texttt{result3} on lines 100 and 101, the function returns $1$ and $4$, and no other entry of the three summaries changes.\n\n\\begin{audittable}{lrrrr}{Summaries returned for the test files (true value $\\tau=0$). ``As released'' is the unmodified function; ``corrected'' has \\texttt{result3} on lines 100 and 101. Lengths are medians; ``Rand.'' is the randomization-based set.}\nVersion, draws & 2SLS coverage & 2SLS length & Rand. coverage & Rand. length \\\\\nAs released, 5000 & 0 & 2 & 1 & 12 \\\\\nAs released, 10000 & \\hl{0} & \\hl{2} & 1 & 14 \\\\\nCorrected, 5000 & 0 & 2 & 1 & 12 \\\\\nCorrected, 10000 & \\hl{1} & \\hl{4} & 1 & 14 \\\\\n\\end{audittable}\n\nThe scripts' own code for generating and saving the observed data was also run on a population of six units. The three calls produced three different observed assignments, and the data file written for the Anderson--Rubin and tF computations held the third. No entry of Table~1 or of Supplement Table~3 is shown to be wrong, and the replication files do not establish which outputs produced the published rows.", "problem": "For each simulated population, the simulation scripts make three calls, with 1000, 5000 and $10^4$ randomization draws, and save the results as \\texttt{result1}, \\texttt{result2} and \\texttt{result3}. Each call draws a new observed assignment. The function \\texttt{extract\\_results2} builds one summary for each number of draws. In the third summary, the 2SLS coverage indicator and the 2SLS interval length are read from \\texttt{result2}, while the computation time and the randomization-based confidence set are read from \\texttt{result3}. The 2SLS coverage and the quantiles of the 2SLS length in the third summary therefore repeat those of the second summary, and use the second run's assignment while the randomization entries beside them use the third run's. The script \\texttt{3\\_merge\\_results\\_grace2.R} reads these entries of the third summary into its 2SLS coverage and median-length rows (lines 48, 50, 56 and 58), which have the layout of the 2SLS rows of Table~1 and of Supplement Table~3. Other indexing slips in the same scripts are recorded as \\fid{P002-N003}, \\fid{P002-N012} and \\fid{P002-N013}.", "what_survives": "Under complete randomization, each observed assignment is uniformly distributed over the assignments with the given number of treated units. Then the expected 2SLS coverage and the distribution of the 2SLS interval length are the same whichever of the two runs supplies the interval, and the 2SLS rows estimate the same marginal quantities in either case. This defect does not invalidate the comparison of coverage and length across methods in Table~1. The correction changes only the 2SLS entries of the third summary. Supplement Tables~4 and~5 have no 2SLS rows, and the theoretical results, the algorithms and the data applications do not use this code."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N018", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of eleven lines of C++ code. Red boxes mark 'round(nsim*(nsim-1)/2/20)' at line 35, the value assigned to the integer variable named progress denominator, and the remainder of count by that variable, written with the percent sign, in the condition of the if statement at line 42, inside two nested for loops over i and j.", "caption": "Research code, cpp_implementations/main2.cpp, lines 35–45, typeset from the source file with line numbers added. The marked expression at line 35 is evaluated in integer arithmetic and is zero for fewer than seven simulated assignments. The marked remainder at line 42 then has divisor zero.", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N019.png"}], "formal_release": null, "formal_statements": [], "headline": "The progress counter of the research C++ routines takes a remainder by zero when there are 2 to 6 simulated assignments; the condition needs a guard.", "id": "P002-N019", "location": "Research code in the replication files (commit 4d919c4), folder cpp_implementations: file main2.cpp, function calculate_intersections_adj, lines 35 and 42; file main.cpp, function calculate_intersections, lines 420, 427 and 500. No passage of the article", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Lines 35 and 42 of \\texttt{main2.cpp} (lines 420 and 427 of \\texttt{main.cpp} are the same):\n\n\\texttt{int progress\\_denominator = \\hl{round(nsim*(nsim-1)/2/20)};}\n\n\\texttt{if (\\hl{count \\% progress\\_denominator} == 0)\\{}", "corrected": "\\texttt{if (\\hl{progress\\_denominator > 0 \\&\\&} count \\% progress\\_denominator == 0)\\{}\n\nIn \\texttt{main.cpp}, the routine should also return an empty intersection result of the expected shape when there is no intersection, before the vectors are read at line 500. The caller should then use the no-intersection branch of the printed algorithm.", "evidence": "The evidence is exact arithmetic on the expression and a run of the code. The routine of \\texttt{main2.cpp}, compiled unchanged with a sanitizer for undefined behavior and called with two simulated assignments, stops at line 42 with a division by zero. Its input was the $6\\times2$ matrix of coefficients computed from six observations, and both denominators are positive on the real line. The progress loops of both routines, compiled separately, stop in the same way for $M=2,\\ldots,6$ and complete for $M=7$ after 21 calls of the root finder. The full routine of \\texttt{main.cpp} was not compiled.", "problem": "With $M$ simulated assignments, the expression \\texttt{nsim*(nsim-1)/2/20} is evaluated in integer arithmetic and equals $\\lfloor M(M-1)/40\\rfloor$, which is $0$ for $M\\le6$. For $2\\le M\\le6$ the first pair of assignments evaluates \\texttt{0 \\% 0}, whose behavior is undefined in C++, before any intersection is computed. For $M=1$ the loop is skipped, and the older routine in \\texttt{main.cpp} then reads element 0 of two empty vectors at line 500.", "what_survives": "The definitions, theorems and algorithms of the article contain no such step. The article uses $10^3$, $5\\times10^3$ and $10^4$ simulated assignments, for which the divisor is $24975$, $624875$ and $2499750$. No reported result is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N019", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of eight comment lines that document the fast algorithm, titled 'Anderson-Rubin permutation test, Algorithm 2 (fast, jumping)'. Red boxes mark 'Computes the same Anderson-Rubin confidence set and randomization p-value as' at line 28 and the name of the slow algorithm's function at the start of line 29.", "caption": "R package rilate, R/AR_algo2.R, lines 26–33, typeset from the source file with line numbers added. The marked words state that the fast algorithm computes the same confidence set and $p$-value as AR_algo1().", "url": "https://pmaronow.github.io/llm-audit/works/P002/exhibits/P002-N022.png"}], "formal_release": null, "formal_statements": [], "headline": "The fast algorithm of the rilate package does not always return the confidence set of the slow algorithm: in the one case examined it excludes a point that the package’s own test accepts.", "id": "P002-N022", "location": "R package rilate in the replication files (commit e5085c8), R/AR_algo2.R, lines 28–32, and the saved simulation results in sim_slurm/sim_coverage_power/aggregated/. No passage of the article; the corresponding statement is Theorem 3, p. 17", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Documentation of \\texttt{AR\\_algo2}, lines 28--32:\n\nComputes \\hl{the same Anderson-Rubin confidence set} and randomization p-value as \\texttt{AR\\_algo1()}, but instead of checking every interval it jumps between crossing points (via \\texttt{find\\_crossings\\_with\\_all()}), which is substantially faster. This is the default algorithm. Internal -- dispatched to by \\texttt{run\\_rilate()}, not called directly.", "corrected": "The correction is to the code of \\texttt{AR\\_algo2}, and I have not identified the line to change. For the strictly positive denominators in this example, a construction that returns the set of the package's test is the following. Partition the real line at every isolated intersection of the observed AR function with a simulated one; identical functions count as ties throughout. On each open interval and at each point of the partition, count the simulated statistics that are at least as large as the observed one. Keep the intervals and points at which the package's $p$-value exceeds the level of the test, which is $1/20$ in these files. The $p$-value is one plus this count, divided by one plus the number of simulated assignments.\n\nUntil the code is corrected, the documentation should read: Computes the Anderson-Rubin confidence set and randomization p-value by jumping between crossing points, which is substantially faster than checking every interval. \\hl{The confidence set can differ from that of} \\texttt{AR\\_algo1()}.", "evidence": "The evidence is an exact counterexample in one case. In the file \\texttt{combo06\\_gamma0.5\\_tau1.0.csv}, case number 5688 has no covariates, 100 units, 50 of them treated, and 1000 simulated assignments. I regenerated it from the seeds in the package's scripts, and both algorithms reproduce the saved widths and $p$-values. The slow algorithm returns $[0.15575,\\,2.09403]$ and the fast algorithm returns $[0.15575,\\,2.07933]$. At $\\beta=417/200$, I find by integer arithmetic that 950 of the 1000 simulated squared statistics are below the observed one, none is equal to it, and 50 are above it. The package takes the 951st order statistic as its critical value, so its test accepts this $\\beta$, with $p$-value $51/1001>1/20$, while the set of the fast algorithm excludes it. All 1001 denominators are positive on the real line. The other 93 flagged cases were not examined, and the line of code responsible was not identified.", "problem": "The package implements two algorithms that invert the same randomization test. Its saved simulation results, nine files of 10,000 cases each, flag a difference between the two returned sets in 48 cases without covariates and in 46 cases with covariates, which are 94 distinct cases out of 90,000. In all 90,000 cases the two algorithms agree on whether the true effect is covered, on whether zero is rejected, and on the $p$-value, so the summaries of coverage and power do not show the difference. In 87 of the 94 flagged comparisons the saved distance between the two sets is finite. In the one flagged case that I reconstructed, the two sets differ in their right endpoints by about $0.0147$, and the fast algorithm excludes a point that the package's test accepts.", "what_survives": "Theorem 3 and the algorithms as printed are not contradicted. The critical value of (13) is the 950th order statistic, under which $\\beta=417/200$ is rejected. The example concerns the package's rule and not the set (14). The saved summaries of coverage, rejection and $p$-values are unaffected. The designs in these files are not those of Section 6, and no table, simulation or application of the article is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P002/#P002-N022", "work": {"arxiv_id": "2404.18786", "doi": "10.1093/biomet/asag010", "title": "Randomization-based confidence sets for the local average treatment effect", "url": "https://pmaronow.github.io/llm-audit/works/P002/"}, "work_id": "P002"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Algorithm B.1 with the paragraph before it and Lemma B.2 after it. Three red boxes: the input condition 'k1, k2 in the positive integers such that k2 is greater than k1' in the first line, and, over two lines of Step 3, the sentence 'If a = k1, evaluate f(k1) and return k1 - 1 if f(k1) = 1, and k1 if f(k1) = 0.'", "caption": "Appendix B.1, Algorithm B.1 and Lemma B.2, p. 20 (PDF p. 20). The marked passages are the input condition and the clause of Step 3 that returns when $a=k_1$ without evaluating $f(k_2)$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T4", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T4"}, {"id": "P003-T5", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T5"}, {"id": "P003-T7", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T7"}, {"id": "P003-T23", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T23"}, {"id": "P003-T27", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T27"}, {"id": "P003-T32", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T32"}, {"id": "P003-T34", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T34"}], "headline": "Algorithm B.1 returns $k_1$ in place of $k_2$ on a two-point range where $f$ is zero, can fail to stop on a two-point range, and excludes ranges that Algorithm 4.3 supplies; the search should start from the points $k_1-1$ and $k_2+1$, which are never evaluated.", "id": "P003-E001", "location": "Appendix B.1, Algorithm B.1 (Steps 2 and 3) and Lemma B.2, p. 20 (PDF p. 20); the search is invoked by Algorithm 4.3, p. 8, and in the proof of Theorem 4.1, p. 21", "pdf_page": 20, "printed_page": "20", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P003-T23", "P003-T27", "P003-T32", "P003-T34", "P003-T4", "P003-T5", "P003-T7"], "text": {"as_printed": "``Algorithm B.1. This binary search algorithm takes as input $k_1,k_2\\in\\hl{\\mathbb Z_{>0}}$ such that $k_2\\hl{>}k_1$, and a function $f:[\\![k_1,k_2]\\!]\\to\\{0,1\\}$ such that $f(x)=0$ if $x\\le r$ and $f(x)=1$ if $x>r$, where $r\\in[\\![k_1-1,k_2]\\!]$ is unknown. The algorithm returns $r$ through the following steps.\n\\begin{enumerate}\n\\item Initialize $a\\leftarrow\\hl{k_1}$ and $b\\leftarrow\\hl{k_2}$.\n\\item Set $c\\leftarrow\\lfloor(a+b)/2\\rfloor$ and evaluate $f(c)$. If $f(c)=0$, set $a\\leftarrow c$. If $f(c)=1$, set $b\\leftarrow c$.\n\\item Repeat the previous step until $b=a+1$. In this case, return $a$ if $a>k_1$ and $b<k_2$. If $a=k_1$, evaluate $f(k_1)$ and return $k_1-1$ if $f(k_1)=1$, and $\\hl{k_1}$ if $f(k_1)=0$. If $b=k_2$, evaluate $f(k_2)$, and return $k_2$ if $f(k_2)=0$ and $k_2-1$ if $f(k_2)=1$.\n\\end{enumerate}\nThe following fact is well known.\n\nLemma B.2. Algorithm B.1 \\hl{terminates} in at most $\\hl{\\lfloor\\log_2(k_2-k_1+1)+2\\rfloor}$ evaluations of $f$.''", "corrected": "``Algorithm B.1. This binary search algorithm takes as input $k_1,k_2\\in\\hl{\\mathbb Z}$ such that $k_2\\hl{\\ge}k_1$, and a function $f:[\\![k_1,k_2]\\!]\\to\\{0,1\\}$ such that $f(x)=0$ if $x\\le r$ and $f(x)=1$ if $x>r$, where $r\\in[\\![k_1-1,k_2]\\!]$ is unknown. The algorithm returns $r$ through the following steps.\n\\begin{enumerate}\n\\item Initialize $a\\leftarrow\\hl{k_1-1}$ and $b\\leftarrow\\hl{k_2+1}$. \\hl{The function is never evaluated at these two points.}\n\\item Set $c\\leftarrow\\lfloor(a+b)/2\\rfloor$ and evaluate $f(c)$. If $f(c)=0$, set $a\\leftarrow c$. If $f(c)=1$, set $b\\leftarrow c$.\n\\item Repeat the previous step until $b=a+1$. \\hl{Then return} $\\hl{a}$.\n\\end{enumerate}\nLemma B.2. Algorithm B.1 terminates \\hl{and returns} $\\hl{r}$ in at most $\\hl{\\lfloor\\log_2(k_2-k_1+1)\\rfloor+1}$ evaluations of $f$.''\n\nThe bound in Lemma B.2 is at most the printed bound $\\lfloor\\log_2(k_2-k_1+1)+2\\rfloor$. The search for the left boundary and the search in Algorithm 4.5 are corrected in the same way.", "evidence": "The evidence is an exact counterexample, computed in integer and rational arithmetic, and a derivation of the corrected search.\n\n\\begin{audittable}{llll}{Inputs on which the printed search fails. The third row is the upper search of Algorithm 4.3 for $n=16$, $m=8$, observed counts $\\mathbf n=(7,1,0,8)$ and $\\alpha=1/20$.}\nRange & Values of $f$ & Printed search & Required value $r$ \\\\\n$\\{1,2\\}$ & $f(1)=f(2)=0$ & returns 1 & 2 \\\\\n$\\{1,2\\}$ & $f(1)=f(2)=1$ & never returns & 0 \\\\\n$\\{14,15\\}$ & $f(14)=f(15)=0$ & returns 14 & 15 \\\\\n\\end{audittable}\n\nIn the third row, I have $T(\\mathbf n)=7/8$ and $n\\max\\mathcal C(\\mathbf n)=15$, and the range is $\\{14,15\\}$. The count vector $\\mathbf v=(0,15,0,1)$ is possible given $\\mathbf n$ and has $\\tau(\\mathbf v)=15/16$. Over the 12870 assignments, $T$ takes the values $7/8$ and $1$, each 6435 times, and both are at distance $1/16$ from $15/16$, which is also the observed distance. I obtain $p(\\mathbf v,\\mathbf n)=1$ and $U_\\alpha(\\mathbf n)=15/16$. Enumerating all 144 possible count vectors gives $\\mathcal I_\\alpha(\\mathbf n)=[1/2,15/16]$. The value $14/16=T(\\mathbf n)$ is accepted as well, through the table $(0,14,0,2)$, and I have $f(14)=f(15)=0$. The printed search evaluates $f(14)$ twice, never evaluates $f(15)$, and returns 14. The upper endpoint of Algorithm 4.3 as printed is then $14/16$. If $(0,15,0,1)$ is the true table, these are the observed counts at 6435 of the 12870 assignments, and at each of them the printed upper endpoint is below the true effect $15/16$.\n\nFor the corrected search (given under Corrected), let $N=k_2-k_1+1$. The inequalities $k_1-1\\le a\\le r<b\\le k_2+1$ hold at the start. If $b-a\\ge2$, then $a<c<b$, and $f$ is evaluated only at points of $[\\![k_1,k_2]\\!]$. If $f(c)=0$, then $c\\le r$, and if $f(c)=1$, then $r<c$. In both cases the update preserves the inequalities. Each update replaces $b-a$ by at most $\\lceil(b-a)/2\\rceil$, and the search stops with $b=a+1$. Then $a\\le r<a+1$, and I obtain $r=a$. The width $b-a$ is $N+1$ at the start and at most $\\lceil(N+1)/2^t\\rceil$ after $t$ evaluations. Since $N+1\\le2^q$ for $q=\\lfloor\\log_2N\\rfloor+1$, the search uses at most $\\lfloor\\log_2N\\rfloor+1$ evaluations.", "problem": "Algorithm 4.3 defines its upper endpoint through this search (``The precise definition of this binary search is given as Algorithm B.1''), and the printed steps are part of the algorithm. The steps fail on a range with two points. Suppose that $k_2=k_1+1$. Step 2 evaluates $f$ at $c=k_1$. If $f(k_1)=0$, then $a=k_1$ and $b=k_2=a+1$, and both end conditions of Step 3 hold. The clause for $a=k_1$ comes first and returns $k_1$ without evaluating $f(k_2)$, which is wrong when $f(k_2)=0$, that is, when $r=k_2$. If $f(k_1)=1$, then Step 2 sets $b=k_1$, the pair $(a,b)=(k_1,k_1)$ repeats, and the condition $b=a+1$ never holds. The search does not stop, which contradicts Lemma B.2. Both inputs satisfy the stated conditions.\n\nThe stated conditions also exclude ranges that Algorithm 4.3 passes to the search. Its range is $[\\![nT(\\mathbf n),n\\max\\mathcal C(\\mathbf n)]\\!]$. The left endpoint can be zero or negative (it is $-12$ in the first row of Table 1), and the range is the single point $\\{n\\}$ when $n_{11}=m$ and $n_{01}=0$. In a balanced design the range of the upper search is $[\\![2(n_{11}-n_{01}),\\,n_{11}-n_{01}+m]\\!]$. It has two points exactly when $n_{11}-n_{01}=m-1$, and then Algorithm 4.3 as printed returns $T(\\mathbf n)=(m-1)/m$ as its upper endpoint, although $U_\\alpha(\\mathbf n)=(2m-1)/(2m)$ for every $\\alpha\\in(0,1)$. This occurs for every $m\\ge2$, including the range $n\\ge15$ of Theorem 4.1.", "what_survives": "The interval $\\mathcal I_\\alpha$, Proposition 3.2, Lemma 4.2 and the compatibility test inside Algorithm 4.3 are unaffected. The corrected search retains the bound in Theorem 4.1. The range has at most $n+1$ points, an evaluation of $f$ at a nonzero point costs at most $n+1$ permutation tests, and the evaluation at $0$, which occurs at most once, costs at most $2(n+1)$. Each endpoint costs at most $(n+1)(\\lfloor\\log_2(n+1)\\rfloor+2)\\le(n+1)\\lfloor\\log_2(n+1)+3\\rfloor$ tests, and (B.10) implies the bound $4n\\log_2n$. Algorithm 4.5 inherits the search and needs the same correction (see \\fid{P003-E003}); Algorithm 5.3 does not use it. The replication code treats short ranges differently: it skips the midpoint loop when the range has one or two points and always tests the outer upper bound (see \\fid{P003-N007}). No reported simulation or application number is shown to be affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E001", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of steps (b) to (e) of Algorithm 5.3 and of Remark 5.4. A red box marks step (e), 'Set I_{alpha,K} <- I_{alpha,K} union J(v).', after which a blank space precedes Remark 5.4. Step (b) above contains the sentence 'If no value of v10 leads to a possible table, set j <- j + 1 and return to (2).'", "caption": "Algorithm 5.3, Steps 2(b)–2(e), and Remark 5.4, p. 12 (PDF p. 12). The marked step is the last instruction of the algorithm. Only the branch of Step 2(b) in which no table is possible increases $j$ and returns to Step 2.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T22", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T22"}, {"id": "P003-T35", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T35"}], "headline": "After Step 2(e) of Algorithm 5.3, $j$ must be increased by one and the algorithm must return to Step 2.", "id": "P003-E002", "location": "Algorithm 5.3, pp. 11–12 (PDF pp. 11–12), after Step 2(e)", "pdf_page": 11, "printed_page": "11–12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T22", "P003-T35"], "text": {"as_printed": "P.~11: ``For every $\\tau_0\\in\\mathcal C(\\mathbf n)$, in descending order from $\\max(\\mathcal C(\\mathbf n))$, we perform the following steps. 1. Initialize $j\\leftarrow 0$. 2. If $j\\le n$, perform the following steps.''\n\nP.~12, Step 2(b): ``If no value of $v_{10}$ leads to a possible table, set $j\\leftarrow j+1$ and return to (2).''\n\nP.~12, last step: ``\\hl{(e)} Set $\\mathcal I_{\\alpha,K}\\leftarrow\\mathcal I_{\\alpha,K}\\cup\\mathcal J(\\mathbf v)$.'' The algorithm ends with this step.", "corrected": "``(e) Set $\\mathcal I_{\\alpha,K}\\leftarrow\\mathcal I_{\\alpha,K}\\cup\\mathcal J(\\mathbf v)$. \\hl{Set} $\\hl{j\\leftarrow j+1}$ \\hl{and return to (2).}''\n\nWhen $j>n$, the steps are repeated for the next $\\tau_0\\in\\mathcal C(\\mathbf n)$. After the last $\\tau_0$, the algorithm returns $\\mathcal I_{\\alpha,K}$.", "evidence": "The evidence is a check of pp.~11--12 and a small exact example of the literal reading. Let $n=2$ and $m=1$, with observed counts $(0,1,0,1)$, the estimand $\\theta(\\mathbf v)=v_{10}+v_{01}$ and the constant statistic $R=0$, so that every test accepts. If the search stops at the first $j$ with a possible table, I obtain the tables $(0,0,1,1)$, $(0,0,0,2)$ and $(0,1,0,1)$, and the set $\\{0,1\\}$. The full loop also reaches $(0,1,1,0)$ and returns $\\{0,1,2\\}$. The table with one subject of type $(0,1)$ and one of type $(1,0)$ produces these data and has $\\theta=2$. The released code loops over every $j$ (\\texttt{for (int j=0; j<n+1; j++)}).", "problem": "The loop over $j\\in[[0,n]]$ is advanced only in Step 2(b), in the case where no table is possible. After a possible table has been processed in Steps 2(c)--2(e), nothing increases $j$ or returns to Step 2. Read literally, the algorithm treats only the first $j$ with a possible table for each $\\tau_0$. Remark 5.4, the running-time count in Appendix C and the released code all assume the loop over every $j$.", "what_survives": "The algorithm as intended, its coverage (Theorem 5.5) and its $O(Kn^3)$ running time are unaffected once the instruction is restored. The released code already performs the loop over $j$."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E002", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph of the proof of Lemma B.15 that begins 'The event {U alpha(Y, Z) is at most U alpha, epsilon, K(Y, Z)} is equal to the event that b_J = 0' and of the display that follows it, which ends with (B.81) and (B.82). Three red boxes: that first sentence; the first equality of the display, '= P(b_J = 1)'; and the line 'at most N exp(-2 K epsilon squared)', numbered (B.81).", "caption": "Appendix B.5, proof of Lemma B.15, p. 27 (PDF p. 27). The marked passages are the asserted identity of events, the equality of probabilities drawn from it, and the bound (B.81). The same step is repeated at (B.85)–(B.89) on p. 28.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T6", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T6"}, {"id": "P003-T7", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T7"}], "headline": "The proof of Lemma B.15 rests on a false identity between the order of the two upper endpoints and the direction of the first disagreement between the exact and the approximate tests; with the corrected binary search, the bounds of the lemma hold, with the same constants, by a union bound over the tests that the approximate search performs.", "id": "P003-E003", "location": "Appendix B.5, proof of Lemma B.15, pp. 27–28 (PDF pp. 27–28): the paragraph before (B.81), the display that ends with (B.81) and (B.82), and (B.85)–(B.89)", "pdf_page": 27, "printed_page": "27–28", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P003-T6", "P003-T7"], "text": {"as_printed": "P.~27: ``The event $\\{U_\\alpha(\\mathbf Y,\\mathbf Z)\\le U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\}$ \\hl{is equal to the event that} $b_J=0$. That is, in order that $U_\\alpha\\le U_{\\alpha,\\varepsilon,K}$, at the first time the result of an approximate permutation test in Algorithm 4.5 differs from the result of an exact permutation test, the approximate test \\hl{must accept}. We therefore compute\n\\[\n\\begin{aligned}\n\\mathbb P\\bigl(\\{U_\\alpha(\\mathbf Y,\\mathbf Z)\\le U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\}^c\\bigr)&{}\\hl{=}\\mathbb P(b_J=1)\\\\\n&=\\sum_{i=1}^N\\mathbb P(b_i=1\\text{ and }J=i)\\\\\n&\\le\\sum_{i=1}^N\\mathbb P(b_i=1\\text{ and }a_i=0)\\\\\n&{}\\hl{\\le N\\exp(-2K\\varepsilon^2)}\\qquad\\text{(B.81)}\\\\\n&\\le(n+1)\\lfloor\\log_2(n+1)+3\\rfloor\\exp(-2K\\varepsilon^2).\\qquad\\text{(B.82)''}\n\\end{aligned}\n\\]\n\nP.~28: ``The event $\\{U_{\\alpha-2\\varepsilon}(\\mathbf Y,\\mathbf Z)\\ge U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\}$ \\hl{is equal to} the event\n\\[\n\\{b_J=1\\}\\cup\\{J=N+1\\}.\\qquad\\text{(B.85)}\n\\]\nWe compute\n\\[\n\\mathbb P\\bigl(\\{U_{\\alpha-2\\varepsilon}(\\mathbf Y,\\mathbf Z)\\ge U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\}^c\\bigr)\\hl{=}\\mathbb P(b_J=0\\text{ and }J\\le N)\\qquad\\text{(B.86)''}\n\\]", "corrected": "The statement of Lemma B.15 is unchanged. In its proof, replace the passage from ``Let $N$ be the total number of permutation tests'' through the sentence following (B.83), ending ``which was shown in the proof of Theorem 4.1'', by the following.\n\n``Condition on the observed data, and \\hl{number the table tests performed by Algorithm 4.5} in the course of finding $U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)$. There are at most $(n+1)\\lfloor\\log_2(n+1)+3\\rfloor$ of them. The table $\\mathbf w$ tested at the $i$th step is determined by the results of the earlier tests, and Algorithm 4.4 uses new draws at each step. If $p(\\mathbf w,\\mathbf Y,\\mathbf Z)\\ge\\alpha$, then by (B.75), Algorithm 4.4 rejects $\\mathbf w$ with \\hl{conditional probability} at most $\\exp(-2K\\varepsilon^2)$. By a \\hl{union bound}, with probability at least\n\\[\n1-(n+1)\\lfloor\\log_2(n+1)+3\\rfloor\\exp(-2K\\varepsilon^2),\n\\]\nAlgorithm 4.5 rejects no table $\\mathbf w$ such that $p(\\mathbf w,\\mathbf Y,\\mathbf Z)\\ge\\alpha$. On this event, every $\\tau_0\\le U_\\alpha(\\mathbf Y,\\mathbf Z)$ that the search tests is declared compatible, and therefore $U_\\alpha(\\mathbf Y,\\mathbf Z)\\le U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)$.''\n\nFor the second part, remove the sentence that redefines $\\mathbf a$ and $\\mathbf b$, and replace the passage from ``The event $\\{U_{\\alpha-2\\varepsilon}(\\mathbf Y,\\mathbf Z)\\ge U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\}$ is equal to the event'' through (B.93) and its justification by the analogous argument. If $p(\\mathbf w,\\mathbf Y,\\mathbf Z)<\\alpha-2\\varepsilon$, acceptance implies $S-p(\\mathbf w,\\mathbf Y,\\mathbf Z)\\hl{>}\\varepsilon$, so (B.76) gives the same conditional bound. Thus, with probability at least $1-(n+1)\\lfloor\\log_2(n+1)+3\\rfloor\\exp(-2K\\varepsilon^2)$, Algorithm 4.5 accepts no such table, and on this event $U_{\\alpha-2\\varepsilon}(\\mathbf Y,\\mathbf Z)\\ge U_{\\alpha,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)$. Both passages use the binary search as corrected in \\fid{P003-E001}.", "evidence": "I base this finding on an exact counterexample, computed in rational arithmetic, and a derivation of the lemma by a different argument.\n\nLet $n=30$, $m=15$, $\\alpha=1/20$, $\\varepsilon=1/200$ and $K=1$, and let the observed counts be $\\mathbf n=(2,13,1,14)$. With $K=1$, an approximate test accepts a table with probability equal to its exact $p$-value. The upper search runs over the effects $x/30$ with $x\\in[\\![2,16]\\!]$, and $U_\\alpha=U_{\\alpha-2\\varepsilon}=10/30$.\n\n\\begin{audittable}{lll}{The exact upper search at level $\\alpha=1/20$ and one path of the approximate search, which has positive probability.}\nEffect tested & Exact search & Approximate search, one path \\\\\n$9/30$ & rejects 1 table, accepts $(2,9,0,19)$ & accepts $(1,9,0,20)$ \\\\\n$12/30$ & rejects 5 tables & rejects 5 tables \\\\\n$10/30$ & rejects 2 tables, accepts $(3,10,0,17)$ & rejects 7 tables \\\\\n$11/30$ & rejects 6 tables & not tested \\\\\nEndpoint returned & $10/30$ & $9/30$ \\\\\n\\end{audittable}\n\nAt $9/30$ the exact search rejects $(1,9,0,20)$, whose $p$-value is $412/10005<1/20$, and accepts $(2,9,0,19)$, whose $p$-value is $752/10005$. On the path in the table, the approximate search accepts $(1,9,0,20)$, so $J=1$ and $b_J=0$. It then rejects every table that it tests at $12/30$ and at $10/30$, including $(3,10,0,17)$, which the exact test accepts with $p$-value $7143/126730$. It returns $9/30<U_\\alpha$. The same decisions have positive probability for every $K$.\n\nFor (B.85), the sequence $\\mathbf a$ is computed at level $\\alpha-2\\varepsilon=1/25$, and the first exact test accepts $(1,9,0,20)$. Consider the path on which the approximate search rejects this table, then accepts $(2,9,0,19)$, and then accepts the first table tested at each of $12/30$, $14/30$, $15/30$ and $16/30$. On this path $J=1$ and $b_J=1$, and the search returns $16/30>U_{\\alpha-2\\varepsilon}$. The event (B.85) holds, and the event $\\{U_{\\alpha-2\\varepsilon}\\ge U_{\\alpha,\\varepsilon,K}\\}$ does not. Summing over the whole search tree gives the probabilities below.\n\n\\begin{audittable}{lr}{Probabilities given the observed data in the same example, rounded to four places. The proof equates the first with the second, and the third with the fourth.}\nEvent & Probability \\\\\n$U_{\\alpha,\\varepsilon,K}<U_\\alpha$ & 0.8628 \\\\\n$b_J=1$, with $\\mathbf a$ at level $\\alpha$ & 0.9489 \\\\\n$U_{\\alpha,\\varepsilon,K}>U_{\\alpha-2\\varepsilon}$ & 0.0300 \\\\\n$b_J=0$ and $J\\le N$, with $\\mathbf a$ at level $\\alpha-2\\varepsilon$ & 0.0038 \\\\\n\\end{audittable}\n\nFor (B.81), I keep the data and $\\alpha$, and set $\\varepsilon=1/50$ and $K=5000$. Then $N=16$ and $a_2=0$, and an approximate test accepts when $S\\ge\\alpha-\\varepsilon=3/100$. The $p$-value $412/10005$ of $(1,9,0,20)$ is above this threshold. If the first approximate test accepts this table, the second approximate test is the test of $(1,12,0,17)$ at $12/30$, whose $p$-value $407/190095$ is below the threshold. Applying Chebyshev's inequality to each of the two tests, I find that $\\mathbb P(b_2=1\\text{ and }a_2=0)>9/10$, while $N\\exp(-2K\\varepsilon^2)=16e^{-4}<48/103$.\n\nFor the replacement argument, fix the observed data, and set $D=(n+1)\\lfloor\\log_2(n+1)+3\\rfloor$ and $\\delta=\\exp(-2K\\varepsilon^2)$. With the corrected search, the upper search performs at most $D$ table tests, whatever their outcomes. I number the tests that the approximate search performs. The table $\\mathbf W_i$ tested at step $i$ is determined by the outcomes of the earlier tests, and the draws used at step $i$ are independent of them. Conditional on the earlier tests, (B.75) implies that a table with $p(\\mathbf W_i,\\mathbf n)\\ge\\alpha$ is rejected with probability at most $\\delta$, and (B.76) implies that a table with $p(\\mathbf W_i,\\mathbf n)<\\alpha-2\\varepsilon$ is accepted with probability at most $\\delta$. By a union bound over at most $D$ steps, each of the following events has probability at least $1-D\\delta$: no table with $p$-value at least $\\alpha$ is rejected, and no table with $p$-value below $\\alpha-2\\varepsilon$ is accepted.\n\nOn the first event, every value $x/n\\le U_\\alpha$ in the range is declared compatible when it is tested. Indeed, by the proof of Theorem 4.1, the list of tables tested for such a value contains a table with $p$-value at least $\\alpha$, and the approximate search accepts either an earlier table of the list or that one. The upper pointer $b$ of the search then stays above $nU_\\alpha$, and the search returns $U_{\\alpha,\\varepsilon,K}\\ge U_\\alpha$. On the second event, every table that the approximate search accepts has $p$-value at least $\\alpha-2\\varepsilon$, and every value it declares compatible is at most $U_{\\alpha-2\\varepsilon}$. The lower pointer $a$ then stays at or below $nU_{\\alpha-2\\varepsilon}$, and the search returns $U_{\\alpha,\\varepsilon,K}\\le U_{\\alpha-2\\varepsilon}$. The lower endpoints are treated in the same way after every outcome is replaced by its complement, which changes the signs of $T$ and $\\tau$ and preserves the $p$-values. By a union bound over the two endpoints, each part of Lemma B.15 holds with probability at least $1-2D\\delta$.", "problem": "In Algorithm 4.3, the compatibility of one value $\\tau_0$ is decided by testing several tables in turn and stopping at the first acceptance. Algorithm 4.5 does the same with a fresh set of Monte Carlo draws for each table. The sequences $\\mathbf a$ and $\\mathbf b$ record the decisions on single tables, and $J$ is the first index at which they differ. The proof asserts that the direction of this first disagreement decides the order of the final upper endpoints. It does not. Suppose that $b_J=0$, that is, the approximate test accepts a table that the exact test rejects. The exact search may accept the next table for the same $\\tau_0$. Both searches then declare $\\tau_0$ compatible and continue from the same midpoint, and a later rejection by the approximate test can still produce $U_{\\alpha,\\varepsilon,K}<U_\\alpha$. In the other direction, a first disagreement with $b_J=1$ can be followed by the acceptance of the next table for the same $\\tau_0$, after which the approximate search can end above the exact endpoint. The identity before (B.81) and the identity (B.85) are therefore false, and so are the equalities of probabilities drawn from them, including (B.86).\n\nThe argument has a second error. After the two searches separate, the $i$th tests of the two sequences can concern different tables. The event $\\{b_i=1\\text{ and }a_i=0\\}$ is then not the rejection of a fixed table whose $p$-value is at least $\\alpha$, Lemma B.14 does not bound its probability, and the bound (B.81) on the sum can fail.\n\nThe printed argument therefore does not prove Lemma B.15, and the proof of Proposition 4.6 cites the lemma. The counterexamples to the first-disagreement argument do not refute either probability bound. With the corrected search of \\fid{P003-E001}, both bounds hold with the same constants by the argument given under Evidence. The literal printed search has a separate defect: if an execution that returns no endpoint is assigned the empty interval, then (B.78) fails at $\\mathbf v=(0,2,0,0)$, $m=1$, $\\alpha=3/4$, $\\varepsilon=1/4$ and $K=200$. Its inclusion event has probability zero, whereas its claimed lower bound is $1-24e^{-25}\\ge1/13>0$. Without this convention, the printed interval is undefined at this input. This example does not refute (B.79) or either bound for the corrected search.", "what_survives": "Lemma B.15 and Proposition 4.6 hold as stated, with the same constants, when the search is the corrected one and the intervals in their statements are defined (the second part of the lemma needs $\\alpha-2\\varepsilon>0$). This proof error does not affect the statements of Lemma B.14, Lemma 4.2 or Theorem 4.1, or the exact test. The simulations and the application do not use the false identities, and no reported number is shown to be affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E003", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the second half of the proof of Proposition 6.2, with displays (D.3) to (D.7). Two red boxes: display (D.3), 'J alpha(M, Z) is a subset of I circle alpha(M, Z)', and the final 'less than alpha' at the end of display (D.7), which bounds a probability involving T(Y(+), Z). The last line reads 'so w is incompatible with (Y prime, Z), as desired.'", "caption": "Appendix D, proof of Proposition 6.2, p. 29 (PDF p. 29), from (D.3) to the end. The marked passages are the inclusion (D.3), which fails in the example below, and the last inequality of (D.7), which does not follow from (D.5) when $\\mathbf w$ is not possible given $(\\mathbf Y^{(+)},\\mathbf Z)$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E004.png"}, {"alt": "Crop of the end of Definition 6.1, Proposition 6.2 and Remark 6.3. Two red boxes: display (6.2), which defines I circle alpha(M, Z) as the interval from min(L alpha(Y(-), Z), T(Y(-), Z)) to max(U alpha(Y(+), Z), T(Y(+), Z)); and the words 'Suppose m less than n in the positive integers,' at the start of Proposition 6.2, whose display (6.3) states that the interval contains tau(y) with probability at least 1 - alpha.", "caption": "Definition 6.1, Proposition 6.2 and Remark 6.3, p. 14 (PDF p. 14). The marked passages are the interval (6.2), built from the two extremal imputations, and the hypothesis of Proposition 6.2, which allows every $m<n$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E004-b.png"}, {"alt": "Chart with a horizontal axis for the sample average treatment effect, with labelled ticks from -0.7 to -0.1, and four horizontal intervals. Lower imputation, which here equals the complete data: from -7/10 to -3/20, in grey. Upper imputation: from -13/20 to -1/5, in grey. As printed, interval (6.2): from -7/10 to -1/5, in orange. Corrected, hull over all completions: from -7/10 to -3/20, in blue. A vertical line marks the true effect -3/20. The printed interval stops one step of 1/20 short of the line, and the corrected interval ends on it.", "caption": "The intervals at the one assignment of the example at which an outcome is hidden ($n=20$, $m=3$, $\\alpha=109/500$, treated units 12, 13, 14). The interval (6.2) as printed is $[-7/10,-1/5]$ and excludes the true effect $-3/20$; the hull over the completions is $[-7/10,-3/20]$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E004-intervals.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T1", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T1"}, {"id": "P003-T2", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T2"}, {"id": "P003-T3", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T3"}, {"id": "P003-T26", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T26"}], "headline": "Proposition 6.2 is false as stated: in an unbalanced design, the interval (6.2) can cover $\\tau(\\mathbf y)$ with probability below $1-\\alpha$. The proposition holds when $n=2m$; for general $m<n$, the interval can be replaced by the hull of the exact intervals over all completions of the missing outcomes.", "id": "P003-E004", "location": "Proposition 6.2 and equation (6.2) of Definition 6.1, p. 14 (PDF p. 14); Appendix D, displays (D.3) and (D.7), p. 29 (PDF p. 29)", "pdf_page": 14, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P003-T1", "P003-T2", "P003-T26", "P003-T3"], "text": {"as_printed": "P.~14: ``Finally, for $\\alpha\\in(0,1)$, we set\n\\[\n\\mathcal I^\\circ_\\alpha(\\mathbf M,\\mathbf Z)=\\Bigl[\\hl{\\min\\bigl(L_\\alpha(\\mathbf Y^{(-)},\\mathbf Z),T(\\mathbf Y^{(-)},\\mathbf Z)\\bigr)},\\ \\hl{\\max\\bigl(U_\\alpha(\\mathbf Y^{(+)},\\mathbf Z),T(\\mathbf Y^{(+)},\\mathbf Z)\\bigr)}\\Bigr].\\qquad\\text{(6.2)}\n\\]\nThe following proposition is proved in Appendix D.\n\nProposition 6.2. Suppose $\\hl{m<n\\in\\mathbb Z_{>0}}$, and fix potential outcomes $\\mathbf y$ and $\\alpha\\in(0,1)$. Then\n\\[\n\\mathbb P\\bigl(\\tau(\\mathbf y)\\in\\mathcal I^\\circ_\\alpha(\\mathbf M,\\mathbf Z)\\bigr)\\ge1-\\alpha,\\qquad\\text{(6.3)}\n\\]\nwhere the probability is with respect to the variable $\\mathbf Z$.''\n\nP.~29: ``First, note that by Proposition 3.2,\n\\[\n\\mathbb P\\bigl(\\tau(\\mathbf y)\\in\\mathcal J_\\alpha(\\mathbf M,\\mathbf Z)\\bigr)\\ge1-\\alpha,\\qquad\\text{(D.1)}\n\\]\nwhere\n\\[\n\\mathcal J_\\alpha(\\mathbf M,\\mathbf Z)=\\bigcup_{\\mathbf Y'\\in\\mathcal Y(\\mathbf M)}\\mathcal I_\\alpha(\\mathbf Y',\\mathbf Z),\\qquad\\text{(D.2)}\n\\]\nand $\\mathcal Y(\\mathbf M)$ consists of all vectors of realized outcomes $\\mathbf Y'$ compatible with the partially observed vector $\\mathbf M$. To complete the proof of the theorem it then suffices to show that\n\\[\n\\hl{\\mathcal J_\\alpha(\\mathbf M,\\mathbf Z)\\subset\\mathcal I^\\circ_\\alpha(\\mathbf M,\\mathbf Z)},\\qquad\\text{(D.3)}\n\\]\n[\\ldots] Fix some $\\tau_0\\in\\mathcal I^\\circ_\\alpha(\\mathbf M,\\mathbf Z)^c$. We may suppose that\n\\[\n\\tau_0>\\max\\bigl(U_\\alpha(\\mathbf Y^{(+)},\\mathbf Z),T(\\mathbf Y^{(+)},\\mathbf Z)\\bigr),\\qquad\\text{(D.5)}\n\\]\nsince the analogous argument in the complementary case is similar. Then we must show that all tables $\\mathbf w$ such that $\\tau(\\mathbf w)=\\tau_0$ are incompatible with all choices of $(\\mathbf Y',\\mathbf Z)$ with $\\mathbf Y'\\in\\mathcal Y(\\mathbf M)$. Fix such a $\\mathbf w$ and a choice of $\\mathbf Y'$. By the definition of $\\mathbf Y^{(+)}$, and since $T(\\mathbf Y^{(+)},\\mathbf Z)$ and $T(\\mathbf Y',\\mathbf Z)$ are coupled through $\\mathbf Z$, we have\n\\[\n\\tau_0\\ge T(\\mathbf Y^{(+)},\\mathbf Z)\\ge T(\\mathbf Y',\\mathbf Z).\\qquad\\text{(D.6)}\n\\]\nThen\n\\[\n\\mathbb P\\Bigl(\\bigl|T(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})-\\tau_0\\bigr|\\ge\\bigl|T(\\mathbf Y',\\mathbf Z)-\\tau_0\\bigr|\\Bigr)\\le\\mathbb P\\Bigl(\\bigl|T(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})-\\tau_0\\bigr|\\ge\\bigl|T(\\mathbf Y^{(+)},\\mathbf Z)-\\tau_0\\bigr|\\Bigr)\\hl{<\\alpha},\\qquad\\text{(D.7)}\n\\]\nso $\\mathbf w$ is incompatible with $(\\mathbf Y',\\mathbf Z)$, as desired.''", "corrected": "Either of the following two statements is correct. The first keeps the interval (6.2) and restricts the design.\n\n``Proposition 6.2. Suppose $\\hl{n=2m}$ \\hl{for some} $\\hl{m\\in\\mathbb Z_{>0}}$, and fix potential outcomes $\\mathbf y$ and $\\alpha\\in(0,1)$. Then\n\\[\n\\mathbb P\\bigl(\\tau(\\mathbf y)\\in\\mathcal I^\\circ_\\alpha(\\mathbf M,\\mathbf Z)\\bigr)\\ge1-\\alpha,\\qquad\\text{(6.3)}\n\\]\nwhere the probability is with respect to the variable $\\mathbf Z$.''\n\nHere $\\mathbf J$ may have any distribution and may depend on $\\mathbf Y$ and $\\mathbf Z$, as in Section 6. In Appendix D, the argument after (D.3) is replaced by the argument for the balanced case given under Evidence.\n\nThe second keeps every $m<n$ and changes the interval in Definition 6.1.\n\n``Finally, for $\\alpha\\in(0,1)$, we set\n\\[\n\\mathcal I^\\circ_\\alpha(\\mathbf M,\\mathbf Z)=\\Bigl[\\hl{\\min_{\\mathbf Y'\\in\\mathcal Y(\\mathbf M)}L_\\alpha(\\mathbf Y',\\mathbf Z)},\\ \\hl{\\max_{\\mathbf Y'\\in\\mathcal Y(\\mathbf M)}U_\\alpha(\\mathbf Y',\\mathbf Z)}\\Bigr],\\qquad\\text{(6.2)}\n\\]\n\\hl{where} $\\mathcal Y(\\mathbf M)$ \\hl{is the set of vectors in} $\\{0,1\\}^n$ \\hl{that agree with} $\\mathbf M$ \\hl{at every observed entry.}''\n\nWith this definition, Proposition 6.2 holds as printed for every $m<n$, and its proof ends at (D.2): the interval contains $\\mathcal J_\\alpha(\\mathbf M,\\mathbf Z)$, and (D.1) gives (6.3).", "evidence": "The evidence is an exact counterexample, obtained by enumerating all assignments in integer and rational arithmetic, and derivations of the two corrections.\n\nFor the counterexample, let $n=20$, $m=3$ and $\\alpha=109/500$. Units 1 to 11 have potential outcomes $(y_i(1),y_i(0))=(1,1)$, units 12 to 14 have $(0,1)$, and units 15 to 20 have $(0,0)$. The count vector is $(11,0,3,6)$ and $\\tau(\\mathbf y)=-3/20$. The outcome of unit 1 is missing exactly when the treated units are 12, 13 and 14, and no other outcome is ever missing. This rule depends on $\\mathbf Z$ only and is allowed by Section 6. There are 1140 assignments. Since $1140\\alpha=248.52$, a table is accepted when the two-sided tail in (2.12) contains at least 249 assignments.\n\nAt the assignment $\\{12,13,14\\}$, the complete data have observed counts $(0,3,11,6)$. Unit 1 is a control with outcome 1. The lower imputation restores this outcome, and the upper imputation replaces it by 0, which gives the counts $(0,3,10,7)$. The exact intervals are $\\mathcal I_\\alpha=[-7/10,-3/20]$ for the counts $(0,3,11,6)$ and $\\mathcal I_\\alpha=[-13/20,-1/5]$ for the counts $(0,3,10,7)$. Since $T(\\mathbf Y^{(-)},\\mathbf Z)=-11/17$ and $T(\\mathbf Y^{(+)},\\mathbf Z)=-10/17$, the interval (6.2) is\n\\[\n\\bigl[\\min(-7/10,-11/17),\\max(-1/5,-10/17)\\bigr]=[-7/10,-1/5],\n\\]\nwhich excludes $\\tau(\\mathbf y)=-3/20$. The tail of the true table contains the assignments in which none or all three of the treated units are among units 1 to 11, for both sets of counts, and their number is $\\binom93+\\binom{11}3=249$. The true table is possible given the complete data. It is not possible given the upper imputation: since no treated unit has outcome 1, all eleven units of type $(1,1)$ are controls and show eleven control outcomes equal to 1, while the upper imputation has ten. Among the tables that are possible given the upper imputation and have effect at least $-3/20$, the largest tail count is 248, attained by $(8,0,3,9)$ and $(9,0,3,8)$. With $\\mathbf w=\\mathbf y$, $\\tau_0=-3/20$ and $\\mathbf Y'=\\mathbf Y$, the hypothesis (D.5) holds and both probabilities in (D.7) equal $249/1140>\\alpha$.\n\n\\begin{audittable}{lrrlrr}{The 1140 assignments of the example, grouped by the observed counts $\\mathbf n$ of the complete data. Tail: number of assignments in the two-sided tail of the true table (accepted if at least 249). Last two columns: number of assignments at which $\\tau(\\mathbf y)=-3/20$ is covered, with complete data and with the interval (6.2) under the rule for the missing outcome.}\n$\\mathbf n$ & Assignments & Tail & $\\mathcal I_\\alpha(\\mathbf n)$ & Complete data & Interval (6.2) \\\\\n$(0,3,11,6)$ & 1 & 249 & $[-7/10,\\,-3/20]$ & 1 & 0 \\\\\n$(0,3,12,5)$ & 18 & 83 & $[-3/4,\\,-3/10]$ & 0 & 0 \\\\\n$(0,3,13,4)$ & 45 & 65 & $[-4/5,\\,-7/20]$ & 0 & 0 \\\\\n$(0,3,14,3)$ & 20 & 20 & $[-17/20,\\,-2/5]$ & 0 & 0 \\\\\n$(1,2,11,6)$ & 33 & 810 & $[-11/20,\\,1/20]$ & 33 & 33 \\\\\n$(1,2,12,5)$ & 198 & 612 & $[-3/5,\\,0]$ & 198 & 198 \\\\\n$(1,2,13,4)$ & 165 & 414 & $[-13/20,\\,-1/20]$ & 165 & 165 \\\\\n$(2,1,11,6)$ & 165 & 777 & $[-7/20,\\,1/4]$ & 165 & 165 \\\\\n$(2,1,12,5)$ & 330 & 1140 & $[-2/5,\\,1/5]$ & 330 & 330 \\\\\n$(3,0,11,6)$ & 165 & 248 & $[-1/10,\\,2/5]$ & 0 & 0 \\\\\nTotal & 1140 & & & 892 & 891 \\\\\n\\end{audittable}\n\nWith complete data, $\\tau(\\mathbf y)$ is covered at 892 assignments. The missing outcome changes the interval at one assignment, and the interval (6.2) covers at 891. The coverage is $891/1140=297/380$, and $391/500-297/380=1/2375>0$.\n\nFor the balanced case, suppose that $n=2m$. Fix an assignment, a pattern of missing outcomes, a completion $\\mathbf Y'$, and a table $\\mathbf w$ that is possible given $(\\mathbf Y',\\mathbf Z)$ with $p(\\mathbf w,\\mathbf Y',\\mathbf Z)\\ge\\alpha$. Set $\\tau_0=\\tau(\\mathbf w)$. If $\\tau_0\\le T(\\mathbf Y^{(+)},\\mathbf Z)$, then $\\tau_0$ does not exceed the upper endpoint of (6.2). Otherwise, let $d$ be the number of units at which $\\mathbf Y'$ and $\\mathbf Y^{(+)}$ differ; their outcomes are missing. Define $\\mathbf w^+$ from $\\mathbf w$ by replacing $w_i(Z_i)$ by $Y^{(+)}_i$ at each of these units. Then $\\mathbf w^+$ is possible given $(\\mathbf Y^{(+)},\\mathbf Z)$. Each replacement raises the effect of its unit by 1 and raises $T$ by $1/m=2/n$. I have $\\tau(\\mathbf w^+)=\\tau_0+d/n$ and $T(\\mathbf Y^{(+)},\\mathbf Z)=T(\\mathbf Y',\\mathbf Z)+2d/n$. For an assignment $\\mathbf z$, let $N(\\mathbf z)$ be the number of replaced entries that are realized under $\\mathbf z$. Then $0\\le N(\\mathbf z)\\le d$ and $T(\\mathbf w^+,\\mathbf z)=T(\\mathbf w,\\mathbf z)+2N(\\mathbf z)/n$, and I obtain\n\\[\n\\bigl|\\bigl(T(\\mathbf w^+,\\mathbf z)-\\tau(\\mathbf w^+)\\bigr)-\\bigl(T(\\mathbf w,\\mathbf z)-\\tau_0\\bigr)\\bigr|=\\frac{|2N(\\mathbf z)-d|}{n}\\le\\frac dn.\n\\]\nSet $\\Delta=\\tau_0-T(\\mathbf Y',\\mathbf Z)$, which satisfies $\\Delta>2d/n$. The observed distance is $\\Delta$ for $\\mathbf w$ and $\\Delta-d/n$ for $\\mathbf w^+$. If $|T(\\mathbf w,\\mathbf z)-\\tau_0|\\ge\\Delta$, then $|T(\\mathbf w^+,\\mathbf z)-\\tau(\\mathbf w^+)|\\ge\\Delta-d/n$. The tail of $\\mathbf w$ is contained in the tail of $\\mathbf w^+$, and I find that $p(\\mathbf w^+,\\mathbf Y^{(+)},\\mathbf Z)\\ge p(\\mathbf w,\\mathbf Y',\\mathbf Z)\\ge\\alpha$. Since $\\mathbf w^+$ is possible and accepted, Algorithm 2.3 implies that $U_\\alpha(\\mathbf Y^{(+)},\\mathbf Z)\\ge\\tau(\\mathbf w^+)\\ge\\tau_0$. The lower endpoint is treated in the same way after every outcome and potential outcome is replaced by its complement. I conclude that (D.3) holds when $n=2m$, and (D.1) then implies (6.3).\n\nFor general $m<n$, the complete vector $\\mathbf Y$ is one of the completions of $\\mathbf M$ at every assignment, whatever outcomes are missing. The interval $\\mathcal I_\\alpha(\\mathbf Y,\\mathbf Z)$ is then contained in the hull over all completions, and the coverage follows from Proposition 3.2. In the example, the hull at the assignment $\\{12,13,14\\}$ is $[-7/10,-3/20]$, and the hull covers $\\tau(\\mathbf y)$ at 892 assignments. If $r_1$ treated and $r_0$ control outcomes are missing, the exact interval of a completion depends only on how many missing outcomes in each group are set to 1, and there are $(r_1+1)(r_0+1)$ such choices.", "problem": "Proposition 6.2 claims coverage for every allocation $m<n$ and for every distribution allowed in Section 6 for the indicator $\\mathbf J$ of the missing outcomes, which may depend on $\\mathbf Y$ and $\\mathbf Z$. The claim fails in an unbalanced design. In the example given under Evidence, with $n=20$ and $m=3$, the interval (6.2) covers $\\tau(\\mathbf y)$ with probability $891/1140$, which is below $1-\\alpha=391/500$.\n\nThe proof fails at the last inequality of (D.7). I call the vectors $\\mathbf Y'\\in\\mathcal Y(\\mathbf M)$ of (D.2) the completions of $\\mathbf M$. By (D.5), the value $\\tau_0$ lies above $U_\\alpha(\\mathbf Y^{(+)},\\mathbf Z)$. By Algorithm 2.3, this means that every table with effect $\\tau_0$ that is possible given $(\\mathbf Y^{(+)},\\mathbf Z)$ has a $p$-value below $\\alpha$. The table $\\mathbf w$ fixed in the proof may be possible given the completion $(\\mathbf Y',\\mathbf Z)$. When $\\mathbf Y'\\ne\\mathbf Y^{(+)}$, it need not be possible given $(\\mathbf Y^{(+)},\\mathbf Z)$, and then (D.5) says nothing about its $p$-value. In the example, the true table is possible given the complete data, not possible given the upper imputation, and has the $p$-value $249/1140>\\alpha$ for both. The inclusion (D.3) fails as well, since the exact interval of one completion, the complete data, is not contained in (6.2).\n\nThe correction changes the missing-outcome extension in Section 6; the complete-data results of Sections 2--5 do not depend on Proposition 6.2. One option is to add the hypothesis $n=2m$, under which Proposition 6.2 holds for the interval (6.2), whatever outcomes are missing; the guarantee then does not extend to unbalanced experiments. The other option is to keep general $m<n$ and replace (6.2) by the hull of the exact intervals over all completions. This interval can be wider, and it may require $(r_1+1)(r_0+1)$ interval computations in place of two, where $r_1$ and $r_0$ are the numbers of missing outcomes in the treatment and control groups.\n\nThe shortfall in the example is $1/2375$, at the level $\\alpha=109/500$. No failure at $\\alpha=0.05$ or $\\alpha=0.10$ is established, nor a smallest example, nor a description of the allocations for which (6.2) fails.", "what_survives": "Display (D.1) and the union (D.2) stand: the union of the exact intervals over all completions covers $\\tau(\\mathbf y)$ with probability at least $1-\\alpha$, and so does its hull. For balanced designs, Proposition 6.2 holds with the interval (6.2), and Remark 6.3 stands. Sections 2 to 5 and the comparisons of Section 7 use complete data and do not depend on Proposition 6.2. The guarantee for the interval (6.2) in unbalanced designs is lost. Remark 6.4 has a different target, the effect in the original $2m-1$ subjects, and the example neither proves nor refutes it."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E004", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the bullet 'To target the sample average treatment effect, we have already seen that we may use the estimand' and of equation (5.6), theta(v) = tau(v) = (v11 - v01)/n = (v11 - v01)/(v11 + v10 + v01 + v00). Red boxes mark v11 in each of the two numerators.", "caption": "Equation (5.6), p. 11 (PDF p. 11). The two marked terms are the count $v_{11}$ of subjects whose outcome is 1 under both treatment and control; the sample average treatment effect has $v_{10}$ in both places.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T13", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T13"}], "headline": "The sample average treatment effect in (5.6) is $(v_{10}-v_{01})/n$: both numerators should read $v_{10}-v_{01}$.", "id": "P003-E005", "location": "Equation (5.6), p. 11 (PDF p. 11), both numerators", "pdf_page": 11, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T13"], "text": {"as_printed": "``To target the sample average treatment effect, we have already seen that we may use the estimand\n\\[\n\\theta(\\mathbf v)=\\tau(\\mathbf v)=\\frac{\\hl{v_{11}}-v_{01}}{n}=\\frac{\\hl{v_{11}}-v_{01}}{v_{11}+v_{10}+v_{01}+v_{00}}.\\qquad\\text{(5.6)''}\n\\]", "corrected": "``To target the sample average treatment effect, we have already seen that we may use the estimand\n\\[\n\\theta(\\mathbf v)=\\tau(\\mathbf v)=\\frac{\\hl{v_{10}}-v_{01}}{n}=\\frac{\\hl{v_{10}}-v_{01}}{v_{11}+v_{10}+v_{01}+v_{00}},\\qquad\\text{(5.6)''}\n\\]\nwhere $\\mathbf v=(v_{11},v_{10},v_{01},v_{00})$ is the count vector (2.2) of a potential outcome table of $n=v_{11}+v_{10}+v_{01}+v_{00}$ subjects.", "evidence": "The evidence is a derivation and an exact counterexample. By (2.2), I have $\\sum_j w_j(1)=v_{11}+v_{10}$ and $\\sum_j w_j(0)=v_{11}+v_{01}$. Substituting these sums into (2.6), I obtain\n\\[\n\\tau(\\mathbf w)=\\frac{(v_{11}+v_{10})-(v_{11}+v_{01})}{n}=\\frac{v_{10}-v_{01}}{n},\n\\]\nand $n=v_{11}+v_{10}+v_{01}+v_{00}$. For the counterexample, let $n=20$ and $\\mathbf v=(10,0,0,10)$. Every subject has the same outcome under treatment and control. Then $\\tau=0$, while both printed fractions equal $1/2$. For $\\mathbf v=(0,20,0,0)$ I have $\\tau=1$, and both printed fractions equal $0$.", "problem": "Equation (5.6) restates the sample average treatment effect as the estimand for Algorithm 5.1. By (2.2), $v_{ab}$ is the number of subjects whose outcomes under treatment and control are $(a,b)$. Then the effect (2.6) is the number of subjects with outcomes $(1,0)$ minus the number with outcomes $(0,1)$, divided by $n$. The display has $v_{11}$, the number of subjects with outcome 1 under both conditions, where $v_{10}$ is required. The printed quantity differs from $\\tau(\\mathbf v)$ by $(v_{11}-v_{10})/n$, and the two agree only when $v_{11}=v_{10}$.", "what_survives": "The definitions (2.2) and (2.6), the constraint (4.2), the construction (5.16) and the calculation (A.7) use the correct expression and are unaffected. The released code computes $(v_{10}-v_{01})/(v_{11}+v_{10}+v_{01}+v_{00})$, so the misprint does not enter the simulations. Theorem 5.5 is stated for a general estimand $\\theta$ and does not depend on (5.6)."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E005", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Definition A.1 and the paragraph that follows. The definition says that a probability mass function f on the integers is symmetric about k0, a half-integer, if f(k0 - x) = f(k0 + x) for all half-integers x such that k0 + x is an integer, display (A.2), and that f is decreasing away from k0 if f(k0 + y) is at most f(k0 + x) for all half-integers x, y such that y > x and k0 + x, k0 + y are integers, display (A.3). A red outline surrounds y > x. After a sentence that defines symmetric-decreasing, abbreviated SD, the next paragraph reads: Given a positive integer k and a half-integer k0, we say that a probability mass function f on the lattice of multiples of 1/k is symmetric about k0/k if the function g on the integers defined by g(x) = f(kx) is symmetric about k0. Similarly, we say that f is SD if g(x) = f(kx) is. Red outlines surround both occurrences of g(x) = f(kx).", "caption": "Appendix A, Definition A.1 and the paragraph after it, p. 18. The outlines mark the condition $y > x$ in (A.3), which lacks $x \\ge 0$, and the two occurrences of $g(x) = f(kx)$, which should read $g(x) = f(x/k)$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T9", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T9"}, {"id": "P003-T10", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T10"}, {"id": "P003-T14", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T14"}, {"id": "P003-T15", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T15"}], "headline": "In Definition A.1, display (A.3) needs the restriction $0 \\le x < y$, and the rescaled function on $\\mathbb{Z}$ is $g(x) = f(x/k)$, not $g(x) = f(kx)$.", "id": "P003-E006", "location": "Appendix A, Definition A.1, display (A.3), and the paragraph after the definition, p. 18 (PDF p. 18)", "pdf_page": 18, "printed_page": "18", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T10", "P003-T14", "P003-T15", "P003-T9"], "text": {"as_printed": "If $f$ is symmetric about $k_0$, we say that $f$ is \\emph{decreasing away from} $k_0$ if\n\\[\nf(k_0 + y) \\le f(k_0 + x) \\text{ for all } x, y \\in 2^{-1}\\mathbb{Z} \\text{ such that } \\hl{y > x} \\text{ and } k_0 + x, k_0 + y \\in \\mathbb{Z}. \\qquad \\text{(A.3)}\n\\]\n[\\ldots] Given $k \\in \\mathbb{Z}_{>0}$ and $k_0 \\in 2^{-1}\\mathbb{Z}$, we say that a probability mass function $f : k^{-1}\\mathbb{Z} \\to \\mathbb{R}_{\\ge 0}$ is symmetric about $k^{-1}k_0 \\in (2k)^{-1}\\mathbb{Z}$ if the function $g : \\mathbb{Z} \\to \\mathbb{R}_{\\ge 0}$ defined by $g(x) = \\hl{f(kx)}$ is symmetric about $k_0$. Similarly, we say that $f$ is SD if $g(x) = \\hl{f(kx)}$ is.", "corrected": "\\textbf{Definition A.1.} Let $f : \\mathbb{Z} \\to \\mathbb{R}_{\\ge 0}$ be a probability mass function. We say that $f$ is \\emph{symmetric about} $k_0$ for some $k_0 \\in 2^{-1}\\mathbb{Z}$ if\n\\[\nf(k_0 - x) = f(k_0 + x) \\text{ for all } x \\in 2^{-1}\\mathbb{Z} \\text{ such that } k_0 + x \\in \\mathbb{Z}. \\qquad \\text{(A.2)}\n\\]\nIf $f$ is symmetric about $k_0$, we say that $f$ is \\emph{decreasing away from} $k_0$ if\n\\[\nf(k_0 + y) \\le f(k_0 + x) \\text{ for all } x, y \\in 2^{-1}\\mathbb{Z} \\text{ such that } \\hl{0 \\le x < y} \\text{ and } k_0 + x, k_0 + y \\in \\mathbb{Z}. \\qquad \\text{(A.3)}\n\\]\nWe now suppose that $\\sum_{k=-\\infty}^{\\infty} k f(k)$ converges absolutely, so that the mean of the distribution corresponding to $f$ exists. In this case, we say that $f$ is \\emph{symmetric-decreasing} (abbreviated SD) if $f$ is symmetric about $k_0 = \\sum_{k=-\\infty}^{\\infty} k f(k)$, and decreasing away from $k_0$.\n\nGiven $k \\in \\mathbb{Z}_{>0}$ and $k_0 \\in 2^{-1}\\mathbb{Z}$, we say that a probability mass function $f : k^{-1}\\mathbb{Z} \\to \\mathbb{R}_{\\ge 0}$ is symmetric about $k^{-1}k_0 \\in (2k)^{-1}\\mathbb{Z}$ if the function $g : \\mathbb{Z} \\to \\mathbb{R}_{\\ge 0}$ defined by $g(x) = \\hl{f(x/k)}$ is symmetric about $k_0$. Similarly, we say that $f$ is SD if $g(x) = \\hl{f(x/k)}$ is.", "evidence": "The evidence is a derivation with two exact counterexamples. Suppose that $f$ satisfies (A.3) as printed. For integers $a < b$, taking $x = a - k_0$ and $y = b - k_0$, I have $f(b) \\le f(a)$. If $f(b) > 0$ for some $b$, then $f(a) \\ge f(b) > 0$ for the infinitely many integers $a < b$, and $f$ is not summable. This implies that no probability mass function satisfies the printed condition; the point mass at $0$ in the paragraph above is one instance. With $0 \\le x < y$, the condition says that $f(k_0 + x)$ is non-increasing in $x \\ge 0$, and by symmetry the same holds on the other side of $k_0$.\n\nFor the rescaling, let $X$ have probability mass function $f$ on $k^{-1}\\mathbb{Z}$. Then $kX$ takes values in $\\mathbb{Z}$ and $P(kX = j) = f(j/k)$. The function $g(j) = f(j/k)$ keeps every point mass, has total mass $1$, and moves the centre $k^{-1}k_0$ to $k_0$. For $k = 2$ and the point mass $f$ at $1/2$, the printed $g(x) = f(2x)$ vanishes at every integer $x$ and has total mass $0$. The corrected $g(x) = f(x/2)$ is the point mass at $1$, which is the centre $k_0 = 1$ that the definition requires. The same direction appears in the proof of Lemma~B.5 on p.~23, which studies $mX$ on $\\mathbb{Z}$ for $X$ on $m^{-1}\\mathbb{Z}$.", "problem": "As printed, (A.3) requires $f(k_0 + y) \\le f(k_0 + x)$ for every pair $x < y$, including negative $x$. It then asks $f$ to be non-increasing on all of $\\mathbb{Z}$, and no probability mass function has this property. For the point mass at $0$, with $k_0 = 0$, $x = -1$ and $y = 0$, the condition reads $1 \\le 0$. The intended condition is that $f$ decreases with the distance from $k_0$, that is, $0 \\le x < y$. In the paragraph after the definition, a probability mass function $f$ on the lattice $k^{-1}\\mathbb{Z}$ is carried to $\\mathbb{Z}$ by $g(x) = f(kx)$. The map that does this is $g(x) = f(x/k)$, the probability mass function of $kX$ when $X$ has probability mass function $f$. As printed, $g$ takes the values of $f$ only at integer multiples of $k$. Every later use follows the intended definitions.", "what_survives": "Both changes repair the wording of Definition~A.1 and of its extension to the lattice $k^{-1}\\mathbb{Z}$. The later arguments use the corrected form, and no result changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E006", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Lemma B.5 and its proof, with display (B.43). Three red boxes: the words 'let X2 be uniformly distributed on {0, 1/m}' in the statement; 'a in Z' in the sentence 'Suppose that m X1 is symmetric about a in Z'; and 'If b = a + 1/2,' in the last sentence of the proof.", "caption": "Appendix B.3, Lemma B.5 and its proof, p. 23 (PDF p. 23). The marked passages are the hypothesis on $X_2$, which does not mention independence, the restriction $a\\in\\mathbb Z$, and the case $b=a+1/2$, which arises only for a half-integer $a$. The citation of the lemma in the proof of Lemma B.9, on p. 25, is not shown.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T9", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T9"}, {"id": "P003-T10", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T10"}], "headline": "Lemma B.5 needs $X_2$ to be independent of $X_1$, its proof should allow the centre $a$ to be a half-integer, and the proof of Lemma B.9 needs a separate convolution step for a distribution on a lattice of spacing two.", "id": "P003-E007", "location": "Appendix B.3, Lemma B.5 and its proof, p. 23 (PDF p. 23); proof of Lemma B.9, pp. 24–25 (PDF pp. 24–25), where Lemma B.5 is cited", "pdf_page": 23, "printed_page": "23", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T10", "P003-T9"], "text": {"as_printed": "P.~23: ``Lemma B.5. Let $X_1$ be SD on $m^{-1}\\mathbb Z$, and let $X_2$ be \\hl{uniformly distributed} on $\\{0,m^{-1}\\}$. Then $X_1+X_2$ is SD on $m^{-1}\\mathbb Z$.\n\nProof. It suffices to show that $m(X_1+X_2)$ is SD on $\\mathbb Z$. Denote the pdf of $mX_1$ by $f_1$. Then the pdf $g$ of $m(X_1+X_2)$ is given by $g(x)=\\tfrac12\\bigl(f_1(x)+f_1(x-1)\\bigr)$. Suppose that $mX_1$ is symmetric about $\\hl{a\\in\\mathbb Z}$. Then it is clear that $g(x)$ is symmetric about $a+\\tfrac12$. To show that $g(x)$ is decreasing, it suffices to show that for any $b\\ge a+1/2$, we have $g(b)\\ge g(b+1)$. We find\n\\[\ng(b)=\\frac12\\bigl(f_1(b)+f_1(b-1)\\bigr)\\ge\\frac12\\bigl(f_1(b+1)+f_1(b)\\bigr)=g(b+1),\\qquad\\text{(B.43)}\n\\]\nas desired. In the previous equation, we used $f_1(b)\\ge f_1(b+1)$, which is true since $f_1$ is SD, and $f_1(b-1)\\ge f_1(b)$. The latter is true by the SD property of $f_1$ if $b\\ge a+1$. If $b=a+1/2$, then this follows from the symmetric of $f_1$ about $a$.''\n\nPp.~24--25, proof of Lemma B.9: ``By Lemma B.7, the conditional distribution of $T(\\mathbf v^\\circ,\\mathbf Z')$ for the remaining $n-2$ people is SD on $(m-1)^{-1}(2\\mathbb Z)$ or $(m-1)^{-1}(2\\mathbb Z+1)$. Similarly, the conditional distribution of (B.47) for the partnership is always uniform on $\\{0,1\\}$, since $\\mathbf w_j=(0,0)$ or $\\mathbf w_j=(1,1)$. Then the conditional pmf for $T(\\mathbf v,\\mathbf Z)$ is SD on $m^{-1}\\mathbb Z$ \\hl{by Lemma B.5}, with mean independent of the choice of $\\mathbf w_j$ and equal to the unconditional mean $\\mathbb E[T(\\mathbf v,\\mathbf Z)]$.''", "corrected": "``Lemma B.5. Let $X_1$ be SD on $m^{-1}\\mathbb Z$, and let $X_2$ be \\hl{independent of} $X_1$ \\hl{and} uniformly distributed on $\\{0,m^{-1}\\}$. Then $X_1+X_2$ is SD on $m^{-1}\\mathbb Z$.''\n\nIn the proof: ``Suppose that $mX_1$ is symmetric about $\\hl{a\\in\\tfrac12\\mathbb Z}$. [\\ldots] it suffices to show that for any \\hl{integer} $b\\ge a+1/2$, we have $g(b)\\ge g(b+1)$.''\n\nIn the proof of Lemma B.9, the citation of Lemma B.5 is replaced by the following statement: let $Y$ be SD on $2\\mathbb Z+p$ for some $p\\in\\{0,1\\}$, and let $U$ be independent of $Y$ and uniformly distributed on $\\{0,1\\}$; then $Y+U$ is SD on $\\mathbb Z$. It is applied with $Y=(m-1)T(\\mathbf v^\\circ,\\mathbf Z')$ and with $U$ the signed contribution of the pair, so that $mT(\\mathbf v,\\mathbf Z)=Y+U$. For the second form of $\\mathbf v$ in (B.52), the signed contribution of the pair is $U-1$ with $U$ uniform on $\\{0,1\\}$, so that $mT(\\mathbf v,\\mathbf Z)=Y+U-1$, and a shift by an integer preserves the SD property. When $m=1$, there are no remaining subjects, and $Y$ is taken to be $0$.", "evidence": "The evidence consists of exact counterexamples to the three printed steps and derivations of the corrected statements.\n\nFor the first error, let $m=1$, let $X_1$ be uniform on $\\{0,1\\}$, and set $X_2=X_1$. Then $X_1$ is SD on $\\mathbb Z$ and $X_2$ is uniform on $\\{0,1\\}$, but $X_1+X_2$ takes the values $0$ and $2$ with probability $1/2$ each and has no mass at its mean $1$. For the second, let $n=6$, $m=3$ and $\\mathbf v=(1,4,0,1)$. Removing $\\mathbf a=(1,1,0,0)$ leaves $(0,3,0,1)$, for which $(m-1)T$ takes the values $1$ and $2$ with probability $1/2$ each, with centre $3/2$. For the third, let $n=4$, $m=2$ and $\\mathbf v=(1,1,0,2)$, and pair the subject of type $(1,0)$ with a subject of type $(0,0)$. The remaining table $(1,0,0,1)$ has $(m-1)T$ uniform on $\\{-1,1\\}$, which is SD on $2\\mathbb Z+1$ and not on $\\mathbb Z$.\n\nFor the corrected Lemma B.5, let $f$ be the mass function of $mX_1$, symmetric about a point $a$ such that $2a$ is an integer. By independence, the mass function of $m(X_1+X_2)$ is $g(x)=(f(x)+f(x-1))/2$. Set $c=a+1/2$. Using the symmetry of $f$ about $a$, I obtain\n\\[\ng(2c-x)=\\tfrac12\\bigl(f(2a+1-x)+f(2a-x)\\bigr)=\\tfrac12\\bigl(f(x-1)+f(x)\\bigr)=g(x).\n\\]\nLet $b\\ge c$ be an integer and set $t=b-a\\ge1/2$. Then $|t-1|\\le t+1$. By the SD property, I have $f(b-1)\\ge f(b+1)$, which gives\n\\[\ng(b)-g(b+1)=\\tfrac12\\bigl(f(b-1)-f(b+1)\\bigr)\\ge0.\n\\]\nTogether with the symmetry, this shows that $m(X_1+X_2)$ is SD on $\\mathbb Z$.\n\nFor the step in Lemma B.9, let $Y$ be SD on $2\\mathbb Z+p$ with $p\\in\\{0,1\\}$, with mass function $f$ and centre $a$, and let $U$ be independent of $Y$ and uniform on $\\{0,1\\}$. Reflection about $a$ maps each point of the support to a point of the support, and both lie in $2\\mathbb Z+p$. It follows that $a$ is an integer. The mass function of $Y+U$ is again $g(x)=(f(x)+f(x-1))/2$, and it is symmetric about $a+1/2$ by the computation above. Let $b\\ge a+1$ be an integer. If $b-p$ is even, then $b-1$ and $b+1$ lie outside $2\\mathbb Z+p$, and $g(b)-g(b+1)=0$. If $b-p$ is odd, then $b-1$ and $b+1$ lie in $2\\mathbb Z+p$ and $b+1>b-1\\ge a$. The SD property implies that $f(b-1)\\ge f(b+1)$, and hence $g(b)\\ge g(b+1)$. I conclude that $Y+U$ is SD on $\\mathbb Z$.", "problem": "Throughout this correction, SD is read as in Definition A.1 with the corrections of \\fid{P003-E006}: a distribution is SD if it is symmetric about its mean and its mass does not increase with the distance from the mean. The pair sum (B.47) is read with the signs corrected in \\fid{P003-E009}. There are three errors.\n\nFirst, the proof of Lemma B.5 writes the mass function of the sum as $g(x)=(f_1(x)+f_1(x-1))/2$. This is the convolution formula for independent variables, but the statement does not assume that $X_1$ and $X_2$ are independent, and without independence the conclusion can fail. Second, the proof takes the centre $a$ of $mX_1$ to be an integer. A distribution that is SD on $\\mathbb Z$ can be centred at a half-integer, and such centres occur where the lemma is applied, in case 1 of the proof of Lemma B.6. The last sentence of the proof, on $b=a+1/2$, concerns exactly this case, because $b$ is an integer. Third, the proof of Lemma B.9 applies Lemma B.5 to a conditional distribution that Lemma B.7 describes as SD on a lattice of spacing two. Such a distribution has no mass on the other half of the lattice $(m-1)^{-1}\\mathbb Z$ and need not be SD on that lattice. The hypothesis of Lemma B.5 is then not met.\n\nIn each place the intended statement is true. The variables are independent where the lemma is applied, the proof of Lemma B.5 goes through for half-integer centres, and if an independent variable that is uniform on two adjacent points is added to an SD distribution on a lattice of spacing two, the sum is SD on the full lattice.", "what_survives": "Lemmas B.5, B.6 and B.9 hold with these changes. No hypothesis is added where the lemmas are used, because in the proofs of Lemmas B.6 and B.9 the pair and the remaining $n-2$ subjects are randomized independently. Lemma B.12, Lemma 4.2, Theorem 4.1, the algorithms and the reported numbers are unchanged. Section 3 does not use these lemmas."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E007", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the last line of Lemma B.12, 'and suppose v10 + v01 is at least 1. Then the pmf of T(v, Z) is SD on the lattice of multiples of 1/m', its proof, and the one-line proof of Lemma 4.2. Six red boxes in the proof: 'min(v10, v01) is at least 1'; 'v01 + v01 = 2'; 'v01 + v01 greater than 2'; 'v10 at least 2' continued on the next line by 'and v01 at least 1'; and 'v10 at least 1 and v01 at least 2'.", "caption": "Appendix B.3, end of the statement of Lemma B.12 and its proof, p. 25 (PDF p. 25). The marked conditions restrict the induction to tables with both $v_{10}$ and $v_{01}$ positive, although the lemma assumes only $v_{10}+v_{01}\\ge1$; the sum $v_{01}+v_{01}$ is printed twice for $v_{10}+v_{01}$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T15", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T15"}], "headline": "The induction in the proof of Lemma B.12 should fix any $\\mathbf v$ with $v_{10}+v_{01}\\ge1$, compare $v_{10}+v_{01}$ with 2, and, when $v_{10}+v_{01}>2$, require only that one of $v_{10}$ and $v_{01}$ be at least 2.", "historical_evidence_type": "lean_proof", "id": "P003-E008", "location": "Appendix B.3, proof of Lemma B.12, p. 25 (PDF p. 25)", "pdf_page": 25, "printed_page": "25", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T15"], "text": {"as_printed": "``Lemma B.12. Let $n=2m$ for some $m\\in\\mathbb Z_{>0}$. Fix any potential outcome count vector\n\\[\n\\mathbf v=(v_{11},v_{10},v_{01},v_{00}),\\qquad\\text{(B.56)}\n\\]\nand suppose $v_{10}+v_{01}\\ge1$. Then the pmf of $T(\\mathbf v,\\mathbf Z)$ is SD on $m^{-1}\\mathbb Z$.\n\nProof. We induct on $n$. The base case $n=0$ is trivial. For the induction step, suppose that the claim is true for $n-2$, and fix any $\\mathbf v$ corresponding to a potential outcome table with $n$ subjects such that $\\hl{\\min(v_{10},v_{01})\\ge1}$. If $(v_{10},v_{01})=(1,0)$ or $(v_{10},v_{01})=(0,1)$, we are done by Lemma B.9. If $\\hl{v_{01}+v_{01}}=2$, we are done by Lemma B.10 and Corollary B.11.\n\nIn the case $\\hl{v_{01}+v_{01}}>2$, we will use Lemma B.6 to show the claim is true for the given $\\mathbf v$. In this case, suppose first that $v_{10}\\ge2$ and $\\hl{v_{01}\\ge1}$. Then by the induction hypothesis, Lemmas B.9, B.10, and Corollary B.11, the first condition given in the statement of Lemma B.6 holds. The case where $\\hl{v_{10}\\ge1}$ and $v_{01}\\ge2$ is analogous. This completes the proof.''", "corrected": "``Proof. We induct on $n$. The base case $n=0$ is trivial. For the induction step, suppose that the claim is true for $n-2$, and fix any $\\mathbf v$ corresponding to a potential outcome table with $n$ subjects such that $\\hl{v_{10}+v_{01}\\ge1}$. If $(v_{10},v_{01})=(1,0)$ or $(v_{10},v_{01})=(0,1)$, we are done by Lemma B.9. If $\\hl{v_{10}+v_{01}}=2$, we are done by Lemma B.10 and Corollary B.11.\n\nIn the case $\\hl{v_{10}+v_{01}}>2$, we will use Lemma B.6 to show the claim is true for the given $\\mathbf v$. In this case, \\hl{at least one of} $v_{10}$ \\hl{and} $v_{01}$ \\hl{is at least 2.} Suppose first that $v_{10}\\ge2$. \\hl{For every} $\\mathbf a$ \\hl{in (B.45) such that} $\\mathbf v-\\mathbf a$ \\hl{has nonnegative entries, the vector} $\\mathbf v-\\mathbf a$ \\hl{has} $n-2$ \\hl{subjects, and the sum of its second and third entries is at least} $\\hl{v_{10}+v_{01}-2\\ge1}$. Then by the induction hypothesis, the first condition given in the statement of Lemma B.6 holds. The case where $\\hl{v_{01}\\ge2}$ is analogous, \\hl{with (B.46) and the second condition}. This completes the proof.''", "evidence": "The evidence is a check of the page, a Lean proof of the arithmetic of the cases, an exact enumeration of the examples, and a derivation of the corrected induction. Here SD is read as in Definition A.1 with the corrections of \\fid{P003-E006}.\n\nThe table $(0,4,0,0)$, with $n=4$, satisfies $v_{10}+v_{01}\\ge1$ and lies outside the printed restriction, outside Lemma B.9, Lemma B.10 and Corollary B.11, and outside both cases with sum above 2. The same holds for $(1,3,0,0)$. Neither contradicts the lemma. For $(0,4,0,0)$, all six assignments give $T=1$, and for $(1,3,0,0)$, three assignments give $T=1/2$ and three give $T=1$. The Lean proof covers these statements about the cases, and two facts used below. If $v_{10}+v_{01}\\ge3$, then $v_{10}\\ge2$ or $v_{01}\\ge2$, and removing at most two of the subjects of types $(1,0)$ and $(0,1)$ leaves at least one of them.\n\nFor the corrected induction, set $d=v_{10}+v_{01}\\ge1$. If $d=1$, Lemma B.9 applies, and if $d=2$, Lemma B.10 or Corollary B.11 applies. Suppose that $d\\ge3$. Then $n\\ge4$ and one of $v_{10}$ and $v_{01}$ is at least 2. If $v_{10}\\ge2$, every vector $\\mathbf a$ in (B.45) removes at most two subjects of types $(1,0)$ and $(0,1)$. Each $\\mathbf v-\\mathbf a$ with nonnegative entries has $n-2\\ge2$ subjects and satisfies the hypothesis of the lemma. By the induction hypothesis, the first condition of Lemma B.6 holds. If $v_{01}\\ge2$, the same argument with (B.46) shows that the second condition holds. No table with $n=0$ satisfies $v_{10}+v_{01}\\ge1$, and the base case holds vacuously.", "problem": "Lemma B.12 assumes only that $v_{10}+v_{01}\\ge1$. The proof fixes $\\mathbf v$ with $\\min(v_{10},v_{01})\\ge1$, and when the sum exceeds 2 it treats only the cases $v_{10}\\ge2$, $v_{01}\\ge1$ and $v_{10}\\ge1$, $v_{01}\\ge2$. Tables in which one of $v_{10}$ and $v_{01}$ is zero and the other is at least 3 satisfy the hypothesis of the lemma and fall under no case of the proof. The induction itself needs them: removing $\\mathbf a=(0,1,1,0)$ from $(0,4,1,1)$ leaves $(0,3,0,1)$. The sum $v_{01}+v_{01}$ is printed twice where $v_{10}+v_{01}$ is meant. Read literally, the table $(1,2,1,0)$ has $v_{01}+v_{01}=2$ and is sent to Lemma B.10 and Corollary B.11, which do not cover it. The lemma is true, and the lemmas already cited close the induction once the cases are stated correctly.", "what_survives": "Lemma B.12 holds as stated, with SD read as above. The proof of Lemma 4.2 applies it to $\\mathbf v-(0,1,1,0)$, in which one of the two counts can be zero, and the corrected induction covers this case. Lemma 4.2, Theorem 4.1, the balanced algorithms and the reported numbers are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E008", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the first case in the proof of Lemma B.7. Six red boxes: the symbol w2 in 'We consider first the case that w2 = (1,1)'; the plus signs before (1 - Z1) w1(0) and before (1 - Zj) wj(0) in display (B.49); the plus sign before (1 - Zi) wi(0) in displays (B.50) and (B.51); and the words 'm inverse (2Z) or m inverse (2Z + 1)' in the sentence that follows the displays.", "caption": "Appendix B.3, proof of Lemma B.7, p. 24 (PDF p. 24), displays (B.49)–(B.51). The marked plus signs stand before the control outcomes, which the estimator subtracts. The partner is called $\\mathbf w_2$ in the text and $\\mathbf w_j$ in (B.49), and the marked lattices require the sums (B.50) and (B.51) to be divided by $m$. Display (B.47), earlier on the page, has the same two plus signs.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T16", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T16"}], "headline": "In (B.47) and (B.49)–(B.51) the control outcomes are subtracted, not added, and the sums (B.50) and (B.51) carry the factor $1/m$.", "id": "P003-E009", "location": "Appendix B.3, displays (B.47), (B.49), (B.50) and (B.51), p. 24 (PDF p. 24), in the proofs of Lemmas B.6 and B.7", "pdf_page": 24, "printed_page": "24", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T16"], "text": {"as_printed": "Proof of Lemma B.6: ``Then $w_1(1)-w_j(0)=0$, and $w_j(1)-w_1(0)=1$, so the distribution of\n\\[\nZ_1w_1(1)\\hl{+}(1-Z_1)w_1(0)+Z_jw_j(1)\\hl{+}(1-Z_j)w_j(0)\\qquad\\text{(B.47)}\n\\]\nis uniform on the set $\\{0,1\\}$.''\n\nProof of Lemma B.7: ``We consider first the case that $\\mathbf w_2=(1,1)$. Then the distribution of\n\\[\nZ_1w_1(1)\\hl{+}(1-Z_1)w_1(0)+Z_jw_j(1)\\hl{+}(1-Z_j)w_j(0)\\qquad\\text{(B.49)}\n\\]\nconditional on the choice of partner and that fact that subjects 1 and 2 are assigned to different groups is constant and equal to 0. This implies we are done by induction, since $v_{11}-2$, the number of $(1,1)$ potential outcome vectors in the remaining $n-2$ subjects, has the same parity as $v_{11}$, and the sampling distributions of\n\\[\n\\sum_{i=1}^nZ_iw_i(1)\\hl{+}(1-Z_i)w_i(0)\\qquad\\text{(B.50)}\n\\]\nand\n\\[\n\\sum_{i=3}^nZ_iw_i(1)\\hl{+}(1-Z_i)w_i(0)\\qquad\\text{(B.51)}\n\\]\nare then both supported and SD on either $m^{-1}(2\\mathbb Z)$ or $m^{-1}(2\\mathbb Z+1)$, as desired.''", "corrected": "\\[\nZ_1w_1(1)\\hl{-}(1-Z_1)w_1(0)+Z_jw_j(1)\\hl{-}(1-Z_j)w_j(0)\\qquad\\text{(B.47), (B.49)}\n\\]\n\\[\n\\hl{\\frac1m}\\sum_{i=1}^n\\bigl[Z_iw_i(1)\\hl{-}(1-Z_i)w_i(0)\\bigr]\\qquad\\text{(B.50)}\n\\]\n\\[\n\\hl{\\frac1m}\\sum_{i=3}^n\\bigl[Z_iw_i(1)\\hl{-}(1-Z_i)w_i(0)\\bigr]\\qquad\\text{(B.51)}\n\\]\nThe sums (B.47) and (B.49) keep the integer-valued distributions stated in the proofs, and they are divided by $m$ when they are combined with the remaining subjects to form $T$. In the proof of Lemma B.7, after the partner is chosen: ``\\hl{Conditional on the choice of partner, the partner is relabelled as subject 2, so that} $\\hl{j=2}$.''", "evidence": "The evidence is a recalculation of the pair sums in every case of the two proofs. The two values in each cell correspond to $(Z_1,Z_j)=(1,0)$ and $(0,1)$.\n\n\\begin{audittable}{llll}{Values of the pair sum, as printed and with the control outcomes subtracted, beside the distribution stated in the proof.}\nTypes $\\mathbf w_1$, $\\mathbf w_j$ & Printed sum & Sum with minus signs & Stated in the proof \\\\\n$(1,0)$, $(1,1)$ & $2$, $1$ & $0$, $1$ & uniform on $\\{0,1\\}$ \\\\\n$(1,0)$, $(1,0)$ & $1$, $1$ & $1$, $1$ & constant $1$ \\\\\n$(1,0)$, $(0,1)$ & $2$, $0$ & $0$, $0$ & constant $0$ \\\\\n$(1,0)$, $(0,0)$ & $1$, $0$ & $1$, $0$ & uniform on $\\{0,1\\}$ \\\\\n$(1,1)$, $(1,1)$ & $2$, $2$ & $0$, $0$ & constant $0$ \\\\\n$(1,1)$, $(0,0)$ & $1$, $1$ & $1$, $-1$ & uniform on $\\{-1,1\\}$ \\\\\n\\end{audittable}\n\nFor the normalization, let $A$ be the pair sum with minus signs and $R$ the corresponding sum over the remaining $n-2$ subjects. By (2.7), I have $T(\\mathbf v,\\mathbf Z)=(A+R)/m$, and, when $m>1$, the estimator for the remaining subjects is $R/(m-1)$. The remaining subjects contribute $R/m$ to $T$, and this contribution lies in $m^{-1}\\mathbb Z$; when $m=1$, the empty sum is $R=0$.", "problem": "The four displays are contributions to the estimator $T$ of (2.7), in which control outcomes enter with a minus sign. The sentence before (B.47) uses the differences $w_1(1)-w_j(0)$ and $w_j(1)-w_1(0)$, and the distributions stated in the two proofs are those of the sums with minus signs. With the printed plus signs they are wrong. For $\\mathbf w_1=(1,0)$ and $\\mathbf w_j=(1,1)$, the printed sum (B.47) takes the values $2$ and $1$, not $0$ and $1$. For $\\mathbf w_1=\\mathbf w_j=(1,1)$, the printed sum (B.49) is the constant $2$, not $0$. The sums (B.50) and (B.51) are integers, and the claim that they are SD on $m^{-1}(2\\mathbb Z)$ or $m^{-1}(2\\mathbb Z+1)$ requires the factor $1/m$. The proof of Lemma B.7 also writes $\\mathbf w_2$ for the partner that (B.49) calls $\\mathbf w_j$, and (B.51) starts at $i=3$. This is consistent once the partner is relabelled as subject 2, which the proof does not say.", "what_survives": "Lemmas B.6 and B.7 hold, and their proofs go through with the corrected displays. Nothing else in the paper changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E009", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the proof of Proposition 3.2, with displays (A.9) and (A.10). A red box, split over two lines, marks the words 'smallest real number such that' in the sentence 'Let x denote the smallest real number such that', which is followed by display (A.10): the probability that the absolute value of T minus tau(y) is at least x is less than alpha.", "caption": "Appendix A.2, proof of Proposition 3.2, p. 19 (PDF p. 19). The marked words define $x$ as the smallest real number that satisfies (A.10).", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T11", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T11"}, {"id": "P003-T12", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T12"}, {"id": "P003-T29", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T29"}], "headline": "In the proof of Proposition 3.2, the threshold $x$ should be the smallest value of the absolute deviation whose tail probability is below $\\alpha$, if such a value exists; a smallest real number with that property does not exist.", "historical_evidence_type": "lean_proof", "id": "P003-E010", "location": "Appendix A.2, proof of Proposition 3.2, p. 19 (PDF p. 19), the sentence that introduces (A.10)", "pdf_page": 19, "printed_page": "19", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T11", "P003-T12", "P003-T29"], "text": {"as_printed": "``Here probability is with respect to $\\widetilde{\\mathbf Z}$, and we consider the left side of (A.9) as a function of $T(\\mathbf Y,\\mathbf Z)$. Let $x$ denote the \\hl{smallest real number} such that\n\\[\n\\mathbb P\\Bigl(\\bigl|T(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})-\\tau(\\mathbf y)\\bigr|\\ge x\\Bigr)<\\alpha,\\qquad\\text{(A.10)}\n\\]\nwhich exists because $T(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})$ is a discrete random variable supported on a bounded set. Then the event where (A.9) occurs is the same as the event where $\\bigl|T(\\mathbf Y,\\mathbf Z)-\\tau(\\mathbf y)\\bigr|\\ge x$. By the definition of $x$, and the fact that $(\\mathbf Y,\\mathbf Z)$ and $(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})$ are identically distributed, this event has probability at most $\\alpha$. This completes the proof.''", "corrected": "``Here probability is with respect to $\\widetilde{\\mathbf Z}$, and we consider the left side of (A.9) as a function of $T(\\mathbf Y,\\mathbf Z)$. \\hl{Let} $S$ \\hl{be the finite set of values of} $\\bigl|T(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})-\\tau(\\mathbf y)\\bigr|$. \\hl{If no} $x\\in S$ \\hl{satisfies (A.10) below, the event where (A.9) occurs is empty. Otherwise,} let $x$ denote the \\hl{smallest element of} $S$ such that\n\\[\n\\mathbb P\\Bigl(\\bigl|T(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})-\\tau(\\mathbf y)\\bigr|\\ge x\\Bigr)<\\alpha.\\qquad\\text{(A.10)}\n\\]\nThen the event where (A.9) occurs is the same as the event where $\\bigl|T(\\mathbf Y,\\mathbf Z)-\\tau(\\mathbf y)\\bigr|\\ge x$. \\hl{Since} $(\\mathbf Y,\\mathbf Z)$ and $(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})$ are identically distributed, \\hl{the probability of this event is the left side of (A.10), which is less than} $\\alpha$. This completes the proof.''", "evidence": "The evidence is an exact counterexample to the existence of $x$, with a Lean proof of its statement about real thresholds, and a derivation of the corrected step.\n\nFor the counterexample, let $n=2$, $m=1$, $\\alpha=1/2$ and $y_1=y_2=(0,0)$. Both assignments give $\\mathbf Y=(0,0)$ and $T(\\mathbf Y,\\mathbf Z)=0=\\tau(\\mathbf y)$. Then $D=0$ with probability one, and I have $q(t)=1$ for $t\\le0$ and $q(t)=0$ for $t>0$. The real numbers $t$ such that $q(t)<1/2$ are the positive numbers, and if $t$ is one of them, then so is $t/2$. A Lean proof establishes that this set is $(0,\\infty)$, that it has no smallest element, and that $t=0$ does not satisfy the strict inequality. The coverage claim is not violated in this example, since the interval is $[-1/2,1/2]$ for both assignments.\n\nIn general, let $d_1<\\dots<d_k$ be the values of $D$, and let $r$ be the largest index such that $q(d_r)\\ge\\alpha$, which exists because $q(d_1)=1$. For $t\\le d_r$, I have $q(t)\\ge q(d_r)\\ge\\alpha$. For $t>d_r$, every value of $D$ that is at least $t$ has index at least $r+1$, and I have $q(t)\\le q(d_{r+1})<\\alpha$ if $r<k$, and $q(t)=0$ if $r=k$. I conclude that $\\{t:q(t)<\\alpha\\}=(d_r,\\infty)$.\n\nFor the corrected step, let $S=\\{d_1,\\dots,d_k\\}$ and $R=\\{d\\in S:q(d)<\\alpha\\}$. Let $D_0=|T(\\mathbf Y,\\mathbf Z)-\\tau(\\mathbf y)|$ be the observed deviation, which takes values in $S$ and has the same distribution as $D$. The event in (A.9) is $\\{q(D_0)<\\alpha\\}$. If $R$ is empty, this event is empty. Otherwise, set $x=\\min R$. For $d\\in S$ with $d<x$, I have $q(d)\\ge\\alpha$, since $d\\notin R$. For $d\\in S$ with $d\\ge x$, I have $q(d)\\le q(x)<\\alpha$. The event in (A.9) is then $\\{D_0\\ge x\\}$, and its probability is $q(x)<\\alpha$.", "problem": "Let $D=|T(\\widetilde{\\mathbf Y},\\widetilde{\\mathbf Z})-\\tau(\\mathbf y)|$ and $q(t)=\\mathbb P(D\\ge t)$. The variable $D$ takes finitely many values, and the set of real numbers $t$ such that $q(t)<\\alpha$ is an open half-line $(d,\\infty)$, where $d$ is the largest value of $D$ such that $q(d)\\ge\\alpha$. This set has no smallest element. The number $x$ is therefore not defined, and the two sentences that use it do not prove the claim. The error is confined to this step, and Proposition 3.2 holds as stated.", "what_survives": "Proposition 3.2 holds under its stated hypotheses, including unbalanced designs. The number $x$ appears only in this proof, and no algorithm computes it. The correction changes no interval, no bound and no reported number. The same argument with $R=\\{d\\in S:q(d)\\le\\alpha\\}$ covers the acceptance rule $p>\\alpha$ of footnote 8."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E010", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the bullet 'For inference on tau, we could also use the studentized test statistic' with equations (5.8) to (5.11) and the sentence 'The function sigma squared is the usual variance estimator written in terms of n.' A red box marks the denominator sigma(n) of equation (5.8), R_std(n, v) = (T(n) - tau(v))/sigma(n).", "caption": "Equations (5.8)–(5.11), p. 11 (PDF p. 11). The marked denominator $\\sigma(\\mathbf n)$ is zero whenever the observed outcomes are constant within each arm.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E011.png"}, {"alt": "Crop of the two lines of footnote 13. A red box marks the opening words 'In each of these expressions (and the ones in the next example)'. The footnote continues: 'we interpret division by zero as returning some arbitrarily selected real number, such as 10 to the power 6. This case is rare: as long as one treated unit has outcome 0 and one control unit has outcome 1, (5.12) and (5.13) will have nonzero denominators.'", "caption": "Footnote 13, p. 11 (PDF p. 11), attached to the risk-ratio statistic after (5.13). The marked words limit the convention on division by zero to the risk-ratio expressions and to the next example, the odds ratio.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E011-b.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T17", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T17"}], "headline": "The studentized statistic (5.8) has no value when $\\sigma(\\mathbf n)=0$, and footnote 13 does not cover it; a fixed finite value must be assigned in that case and used for the observed and the simulated assignments alike.", "id": "P003-E011", "location": "Section 5.1, equations (5.8)–(5.11), p. 11 (PDF p. 11); footnote 13, p. 11", "pdf_page": 11, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T17"], "text": {"as_printed": "``For inference on $\\tau$, we could also use the studentized test statistic\n\\[\nR_{\\mathrm{std}}(\\mathbf n,\\mathbf v)=\\frac{T(\\mathbf n)-\\tau(\\mathbf v)}{\\hl{\\sigma(\\mathbf n)}},\\qquad\\text{(5.8)}\n\\]\nwhere\n\\[\n\\sigma^2(\\mathbf n)=\\frac{Q_1}{(n_{11}+n_{10})^2}+\\frac{Q_0}{(n_{01}+n_{00})^2},\\qquad\\text{(5.9)}\n\\]\n\\[\nQ_1=n_{11}\\left(\\frac{n_{11}}{n_{11}+n_{10}}-1\\right)^2+n_{10}\\left(\\frac{n_{11}}{n_{11}+n_{10}}\\right)^2,\\qquad\\text{(5.10)}\n\\]\n\\[\nQ_0=n_{01}\\left(\\frac{n_{01}}{n_{01}+n_{00}}-1\\right)^2+n_{00}\\left(\\frac{n_{01}}{n_{01}+n_{00}}\\right)^2.\\qquad\\text{(5.11)}\n\\]\nThe function $\\sigma^2$ is the usual variance estimator written in terms of $\\mathbf n$.''\n\nFootnote 13, attached to the risk-ratio statistic $R_{\\mathrm{RR}}$ that follows: ``In each of these expressions (and the ones in the next example), we interpret division by zero as returning some arbitrarily selected real number, such as $10^6$. This case is rare: as long as one treated unit has outcome 0 and one control unit has outcome 1, (5.12) and (5.13) will have nonzero denominators.''", "corrected": "``For inference on $\\tau$, we could also use the studentized test statistic\n\\[\nR_{\\mathrm{std}}(\\mathbf n,\\mathbf v)=\n\\begin{cases}\n\\dfrac{T(\\mathbf n)-\\tau(\\mathbf v)}{\\sigma(\\mathbf n)}, & \\sigma(\\mathbf n)>0,\\\\[6pt]\n{}\\hl{C}, & {}\\hl{\\sigma(\\mathbf n)=0,}\n\\end{cases}\n\\qquad\\text{(5.8)}\n\\]\nwhere $\\sigma(\\mathbf n)\\ge 0$ is defined by (5.9)--(5.11) for count vectors $\\mathbf n$ with both arms nonempty, and \\hl{$C$ is a finite real constant, such as $10^6$, fixed before the tests are run. The same function is applied to the observed assignment and to every simulated or reused assignment.}''", "evidence": "The evidence is an exact counterexample, found by enumerating every assignment. Let $n=5$ and $m=2$. Suppose that all five subjects have potential outcomes $(0,0)$. For each of the 10 assignments, I have $\\mathbf n=(0,2,0,3)$, $T=0$, $Q_1=Q_0=0$ and $\\tau=0$, and (5.8) is $0/0$ with probability one. If one subject has outcomes $(0,1)$ and the other four have $(0,0)$, then $\\tau=-1/5$, and in the 4 of 10 assignments that treat this subject the numerator is $1/5$ and $\\sigma=0$.\n\nThe repair keeps (5.4). If one real-valued function is applied to the observed assignment and to the $K$ simulated ones, the $K+1$ values are exchangeable under the null. Let $r_i$ be the number of these values that are at least as large in absolute value as the $i$th one, so that $S=r_0/(K+1)$ in (5.3). For every integer $t\\ge0$, at most $t$ indices have $r_i\\le t$, ties included. Taking $t=\\lfloor\\alpha(K+1)\\rfloor$, I obtain $\\mathbb P(S\\le\\alpha)\\le\\lfloor\\alpha(K+1)\\rfloor/(K+1)\\le\\alpha$. The constant matters for the test itself: in the second example, the exact $p$-value of the true table at the all-zero observation is $1$ if the assigned value is $0$ and $2/5$ if it is $10^6$.", "problem": "Section 5.1 defines a test statistic as a real-valued function of $(\\mathbf n,\\mathbf v)$, and the design allows every potential outcome table (2.1). Expanding (5.10) and (5.11), I obtain $Q_1=n_{11}n_{10}/(n_{11}+n_{10})$ and $Q_0=n_{01}n_{00}/(n_{01}+n_{00})$. This implies that $\\sigma(\\mathbf n)=0$ exactly when $n_{11}n_{10}=0$ and $n_{01}n_{00}=0$, that is, when the observed outcomes are constant within each arm. For such data (5.8) is $0/0$, or a nonzero number divided by zero. Footnote 13 is attached to the risk-ratio example and extends to ``the next example'', the odds ratio; it does not reach back to (5.8). Algorithms 5.1 and 5.3 and Theorem 5.5 take a real-valued statistic. They do not apply to $R_{\\mathrm{std}}$ as printed until a value is fixed for this case.", "what_survives": "Remark 5.2 and Theorem 5.5 hold for every real-valued statistic, and in particular for $R_{\\mathrm{std}}$ once the value at $\\sigma(\\mathbf n)=0$ is fixed. The procedures based on the difference in means, the balanced algorithms and the missing-data construction do not use (5.8). In the six populations simulated for Table 3, the variance estimate is positive under every assignment. The true table never meets this case there, but other tables tested during the inversion can. No reported number is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E011", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Proposition 4.6 on page 9, from its first line through equation (4.10). Part 1 begins: Fix epsilon in (0, alpha). It then gives the lower bound (4.9) on K and the coverage statement (4.10) for the interval with subscripts alpha minus epsilon, epsilon, K. A red outline surrounds the range (0, alpha).", "caption": "Proposition 4.6, part 1, p. 9 (PDF p. 9). The outline marks the range of $\\varepsilon$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E012.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T34", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T34"}], "headline": "Part 1 of Proposition 4.6 should take $\\varepsilon\\in(0,\\alpha/2)$, the range in which its call to Algorithm 4.5 with level $\\alpha-\\varepsilon$ and tolerance $\\varepsilon$ meets the input condition of that algorithm.", "id": "P003-E012", "location": "Proposition 4.6, part 1, p. 9 (PDF p. 9); input conditions of Algorithms 4.4 and 4.5, p. 9", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P003-T34"], "text": {"as_printed": "1. Fix $\\varepsilon\\in\\hl{(0,\\alpha)}$. If\n\\[\nK\\ge\\frac{1}{2\\varepsilon^2}\\log\\Bigl(\\frac{4n\\log_2(n)}{\\varepsilon}\\Bigr),\n\\]\nthen\n\\[\n\\mathbb P\\bigl(\\tau(\\mathbf y)\\in\\mathcal I_{\\alpha-\\varepsilon,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\bigr)\\ge1-\\alpha.\n\\]", "corrected": "1. Fix $\\varepsilon\\in\\hl{(0,\\alpha/2)}$. If\n\\[\nK\\ge\\frac{1}{2\\varepsilon^2}\\log\\Bigl(\\frac{4n\\log_2(n)}{\\varepsilon}\\Bigr),\n\\]\nthen\n\\[\n\\mathbb P\\bigl(\\tau(\\mathbf y)\\in\\mathcal I_{\\alpha-\\varepsilon,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\bigr)\\ge1-\\alpha.\n\\]", "intended_meaning": "Algorithms 4.4 and 4.5 take a level in $(0,1)$ and a tolerance that is nonnegative and less than the level. Part 1 calls them with level $\\alpha-\\varepsilon$ and tolerance $\\varepsilon$, and their input condition becomes $\\varepsilon<\\alpha-\\varepsilon$, that is, $\\varepsilon<\\alpha/2$. For example, $\\alpha=1/10$ and $\\varepsilon=3/40$ lie in the printed range, and the level is then $1/40<3/40$. The coverage statement is not false for $\\varepsilon\\ge\\alpha/2$. If the rule of Algorithm 4.4 is applied as written, then it accepts when $S+\\varepsilon\\ge\\alpha-\\varepsilon$, which holds for every table because $S\\ge0$. The interval is then the whole range of possible effects, which contains $\\tau(\\mathbf y)$. The proposition is correct as intended, and the range $(0,\\alpha/2)$ states it within the domain of the algorithm. Part 2 already requires $\\varepsilon<\\alpha/3$."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E012", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the six lines of Remark 5.7. Red boxes mark 'O(n x polylog(n))' in 'The time-complexity of the balanced algorithm is O(n x polylog(n))' and 'log^2 n' inside the formula K = log(4n log^2 n / epsilon)/(2 epsilon^2). The sentence continues '(following (4.9)), and for the unbalanced algorithm, the time complexity is O(K n^3). Hence, for large n, the balanced algorithm will be more efficient than the unbalanced algorithm.'", "caption": "Remark 5.7, p. 13 (PDF p. 13). The first marked term is the stated run time of the balanced algorithm, which lacks a factor of $n$; the second is the squared logarithm in $K$, where (4.9) has the logarithm to base 2.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E013.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T18", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T18"}], "headline": "The operation count of the balanced algorithm is $O(n^2\\,\\mathrm{polylog}(n))$ for fixed $\\epsilon$, as in (1.6), not $O(n\\times\\mathrm{polylog}(n))$; and the choice of $K$ in (4.9) has $\\log_2 n$, not $\\log^2 n$.", "id": "P003-E013", "location": "Remark 5.7, second sentence, p. 13 (PDF p. 13); compare (1.6), p. 4, and (4.9), p. 9", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T18"], "text": {"as_printed": "``The time-complexity of the balanced algorithm is $O(\\hl{n}\\times\\mathrm{polylog}(n))$, by choosing $K=\\log(4n\\hl{\\log^2 n}/\\epsilon)/(2\\epsilon^2)$ (following (4.9)), and for the unbalanced algorithm, the time complexity is $O(Kn^3)$.''", "corrected": "``The time-complexity of the balanced algorithm, \\hl{for fixed} $\\hl{0<\\epsilon<\\alpha}$, is $O(\\hl{n^2}\\times\\mathrm{polylog}(n))$ \\hl{(more precisely} $\\hl{O(n^2\\log^2 n)}$\\hl{)}, by choosing $K=\\hl{\\lceil}\\log(4n\\hl{\\log_2 n}/\\epsilon)/(2\\epsilon^2)\\hl{\\rceil}$ (following (4.9)), and for the unbalanced algorithm, the time complexity is $O(Kn^3)$.''", "evidence": "The evidence is a derivation of the upper bound and an exact family of executions that excludes the printed order. For the upper bound, I multiply the $O(n\\log n)$ tests of Theorem 4.1 by the $O(Kn)$ operations of one test and insert $K=\\lceil\\log(4n\\log_2 n/\\epsilon)/(2\\epsilon^2)\\rceil$, which is of order $\\log n$ for fixed $0<\\epsilon<\\alpha$. This gives $O(n^2\\log^2 n)$.\n\nFor the lower bound, let $n=2^r$ with $r\\ge 4$, $m=n/2$, $\\alpha=1/2$ and $\\epsilon=1/8$, and let every observed outcome be zero, so that $\\mathbf n=(0,m,0,m)$. The first query of the search for the upper endpoint in Algorithm 4.3 is $\\tau_0=1/4$. At this query the possible tables are $\\mathbf v_j=(0,j,j-n/4,n-2j+n/4)$ for $n/4\\le j\\le n/2$, and there are $n/4+1$ tests if all of them reject. For each $\\mathbf v_j$ there exists an assignment such that $V=0$ in (4.6), since its statistic equals $\\tau_0$. If every draw is this assignment, then $S=0$ and the test rejects, for every $K$. On this event the first query alone takes $(n/4+1)K(n-1)$ iterations of the Fisher--Yates shuffle. For $n=16$ this is 5 tests and, with $K=244$ as in (4.9), 1220 draws and 18300 iterations. More generally, by Chebyshev's inequality each exact $p$-value is at most $4/(n-1)$; a concentration bound and a union bound show that, for $r\\ge 6$ and $K$ as in (4.9), all $n/4+1$ tests reject with probability greater than $1/2$. I conclude that the expected number of operations is at least of order $n^2\\log n$, which is not $O(n\\,\\mathrm{polylog}(n))$.\n\nThe squared logarithm does not endanger coverage. Since $\\log^2 n/\\log_2 n=\\log n\\,\\log 2>1$ for $n\\ge 15$, the printed $K$ is larger than (4.9) requires, and it has the same order.", "problem": "The paper measures run time in arithmetic operations (p.~4, footnote 7), and a Monte Carlo test with $K$ samples costs $O(Kn)$ operations when the assignments are generated by the Fisher--Yates algorithm (p.~13). By Theorem 4.1, there are $O(n\\log n)$ tests, and (4.9) implies that $K$ is of order $\\log n$ for fixed $\\epsilon$. The product is $O(n^2\\log^2 n)$, which is the bound (1.6). The order $n\\,\\mathrm{polylog}(n)$ printed in the remark is that of the number of Monte Carlo draws, $O(Kn\\log n)$. The bound $O(Kn^3)$ for the unbalanced algorithm is a number of operations. The remark also has $\\log^2 n$ inside $K$ where (4.9) has $\\log_2 n$.", "what_survives": "The bound (1.6), Theorem 4.1, Proposition 4.6 and the coverage results do not depend on the remark. Its conclusion also stands: for fixed $\\epsilon$ the corrected bound for the balanced algorithm is of smaller order than $n^3$, against $O(Kn^3)$ for the unbalanced one. The timings of Tables 4 and 5 were measured and do not use the sentence. The corrected bound is weaker than the printed one. It concerns the implementation that the paper analyzes, which generates each assignment by a shuffle, and it says nothing about an implementation that samples counts directly, as the released code does."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E013", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of footnote 16, two lines under a short rule: Without the covariate, there are 4 coordinates to choose from, with the single linear restriction that they sum to n. With the binary covariate, there are 8 coordinates and a single linear restriction. A red outline surrounds the last words: a single linear restriction.", "caption": "Footnote 16, p. 17. The outline marks the statement that the eight counts satisfy a single linear restriction.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E014.png"}, {"alt": "Crop of two paragraphs of Section 8. The first begins: To understand the difficulty that strata create, suppose that each subject comes equipped with a binary covariate X, and we conduct complete randomization separately among the subjects with X = 0 and the subjects with X = 1. It ends: increasing the number of potential outcome count vectors v to examine from O(n cubed) to O(n to the power 7), followed by the footnote mark 16. A red outline surrounds the words: O(n cubed) to O(n to the power 7), with the footnote mark. The second paragraph begins: In Section 5, when we examined the case without covariates, and continues: One can similarly gain a factor of n when there is a binary covariate, but this only brings the time complexity to O(n to the power 6), which is far from practical. A second red outline surrounds O(n to the power 6).", "caption": "Section 8, p. 17, the two paragraphs that use the count. The outlines mark the bounds that rest on footnote 16: the number of count vectors, $O(n^7)$, with the call to the footnote, and the running time $O(n^6)$ in the next paragraph. Both are valid upper bounds, each one power of $n$ above the bound that the two restrictions give.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E014-b.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T19", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T19"}], "headline": "With a known binary covariate, the eight counts satisfy two linear restrictions, one for each stratum size, and not the single restriction of footnote 16; the number of count vectors is $O(n^6)$, and the printed $O(n^7)$ and $O(n^6)$ are valid but loose upper bounds.", "id": "P003-E014", "location": "Section 8, paragraph beginning “To understand the difficulty that strata create”, and footnote 16, p. 17 (PDF p. 17)", "pdf_page": 17, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T19"], "text": {"as_printed": "``[\\ldots] with an additional degree of freedom coming from the new binary covariate, increasing the number of potential outcome count vectors $\\mathbf{v}$ to examine from $O(n^3)$ to $O(n^7)$.''\n\nFootnote~16: ``Without the covariate, there are 4 coordinates to choose from, with the single linear restriction that they sum to $n$. With the binary covariate, there are 8 coordinates and \\hl{a single linear restriction}.''\n\nNext paragraph: ``One can similarly gain a factor of $n$ when there is a binary covariate, but this only brings the time complexity to $O(n^6)$, which is far from practical.''", "corrected": "Footnote~16: ``Without the covariate, there are 4 coordinates to choose from, with the single linear restriction that they sum to $n$. With the binary covariate, there are 8 coordinates and \\hl{two linear restrictions}: the four counts for the subjects with $X = 0$ sum to the number $N_0$ of these subjects, and the four counts for the subjects with $X = 1$ sum to the number $N_1$ of these subjects. Before compatibility with the observed outcomes is imposed, there are $\\binom{N_0+3}{3}\\binom{N_1+3}{3} = O(n^6)$ count vectors, and this order is attained when $N_0$ and $N_1$ are both proportional to $n$.''\n\nIn the text, the bounds may stand as loose upper bounds, or they may be sharpened: ``[\\ldots] increasing the number of potential outcome count vectors $\\mathbf{v}$ to examine from $O(n^3)$ to $\\hl{O(n^6)}$'', and, in the next paragraph, ``One can similarly gain a factor of $n$ when there is a binary covariate, \\hl{which brings the running time to the equivalent of} $\\hl{O(n^5)}$ approximate permutation tests, or $O(Kn^6)$ arithmetic operations [\\ldots]''.", "evidence": "The evidence is a derivation and an exact enumeration. I write $v_{abs}$ for the number of subjects with $X = s$ and potential outcomes $(w(1), w(0)) = (a, b)$. Each subject of stratum $s$ has exactly one pair $(a, b)$, so\n\\[\nv_{110} + v_{100} + v_{010} + v_{000} = N_0, \\qquad v_{111} + v_{101} + v_{011} + v_{001} = N_1 .\n\\]\nThe two equations involve disjoint sets of counts and are independent, and their sum is the restriction that the eight counts sum to $n$. The four counts of stratum $s$ range over the $\\binom{N_s+3}{3}$ nonnegative integer solutions of the corresponding equation. Each solution is realized by some table, and the two strata are chosen independently. Before the observed outcomes are used, the number of count vectors is\n\\[\n\\binom{N_0+3}{3}\\binom{N_1+3}{3} \\le \\frac{(n+3)^6}{36} = O(n^6),\n\\]\nand it is at least $c^6 n^6/36$ when $N_0 \\ge cn$ and $N_1 \\ge cn$ for a constant $c > 0$. With the single restriction of the footnote, the count is $\\binom{n+7}{7}$, of order $n^7/5040$, and it includes vectors with other stratum sizes. For $n = 4$ and $N_0 = N_1 = 2$, there are 330 nonnegative integer vectors with sum 4, and 100 of them have both stratum sums equal to 2. These 100 are the count vectors of the $4^4 = 256$ tables. The vector $(3,0,0,0;1,0,0,0)$ satisfies the restriction of the footnote and is the count vector of no table.\n\nFor the running time, I split the count vectors of stratum 0 into lines. Along a line, consecutive vectors differ by $(-1, +1, +1, -1)$ in the order $(v_{110}, v_{100}, v_{010}, v_{000})$. There are at most $(N_0+1)(2N_0+1)$ lines, and each has at most $N_0 + 1$ vectors. With the counts of stratum 1 held fixed, I obtain at most\n\\[\n(N_0+1)(2N_0+1)\\binom{N_1+3}{3} = O(n^5)\n\\]\nlines in all. Along a line, the $K$ draws are updated in $O(K)$ operations per vector, as in Section~5.1.1 applied within stratum 0. A line then costs $O(Kn)$ operations, and the whole computation costs $O(Kn^6)$. Since one Monte Carlo permutation test with $K$ samples costs $O(Kn)$, this is equivalent to $O(n^5)$ tests. Both figures are upper bounds, and no lower bound is claimed.", "problem": "In the experiment of this paragraph, the subjects with $X = 0$ and the subjects with $X = 1$ are randomized separately. Since the covariate of every subject is known, the numbers $N_0$ and $N_1$ of subjects in the two strata are fixed. A potential outcome table can change the outcomes of a subject but not its stratum. The four counts within each stratum then sum to the size of that stratum. The eight counts satisfy two independent linear restrictions, and six of them are free. Footnote~16 states a single restriction, which is the sum of these two. The count that the footnote supports is one power of $n$ too large: there are $O(n^6)$ count vectors, and the bound $O(n^7)$ in the text is valid but not sharp. The next paragraph inherits the extra power. If the Monte Carlo draws are reused as in Section~5, the running time is equivalent to $O(n^5)$ approximate permutation tests. The printed $O(n^6)$ is an upper bound for the cost of that approach and not its order.", "what_survives": "The correction concerns the discussion of future work in Section~8 and needs no new assumption, since the stratum sizes are part of the stated design. The printed $O(n^7)$ and $O(n^6)$ remain valid upper bounds. The algorithms and results of Sections~3--6 and the examples and simulations of Section~7 concern complete randomization without strata and do not use this count. The interest in more efficient methods for stratified designs is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E014", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of page 5 with equations (2.2), (2.3) and (2.4). Equation (2.3) reads: Z equals the vector (Z1, Z1, ..., Zn), an element of the 0-1 vectors of length n. A red outline surrounds the second Z1.", "caption": "Equations (2.2)–(2.4), p. 5 (PDF p. 5). The outline marks the second coordinate of (2.3), printed as $Z_1$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E015.png"}, {"alt": "Crop of three lines of page 20. The first line reads: Note that the last condition is always satisfied. Considering j minus v10 minus n01, we get. Display (B.8) follows with four inequalities, the first of which is n10 at least 0. A red outline surrounds n10 in that first inequality.", "caption": "Display (B.8), p. 20 (PDF p. 20), in the proof of Lemma B.3. The outline marks the count $n_{10}$ in the first inequality.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E015-B8.png"}, {"alt": "Crop of nine lines of the proof of Theorem 4.1 on page 21, including display (B.13). The text reads in part: Let j = u11 + u01, and let v01 be the value determined in Step (2b). Let v be the vector determined by (2a) for this value of j, using this value of v01. Further down: constructed from v prime 10 = 1 using (4.4) and j = u11 + u01. Red outlines surround u01 in both formulas for j and both occurrences of v01.", "caption": "Proof of Theorem 4.1, p. 21 (PDF p. 21), paragraph beginning “Therefore, we may suppose $\\mathbf u$ was not tested”. The outlines mark the two occurrences of $u_{01}$ in the index $j$ and the two occurrences of $v_{01}$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E015-p21.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T20", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T20"}], "headline": "Eight groups of local slips are corrected: $Z_2$ in (2.3), the vector $(-1,1,1,-1)$ below (5.20), $n_{01}\\ge0$ in (B.8), the index $j=u_{11}+u_{10}$ and the parameter $v_{10}$ in the proof of Theorem 4.1, the parity of $v_{00}$ in the proof of Lemma B.10, and four sentences that are reworded.", "id": "P003-E015", "location": "Equation (2.3), p. 5 (PDF p. 5); also pp. 12, 13, 14, 18, 20, 21 and 26", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T20"], "text": {"as_printed": "\\emph{(a) Equation (2.3), p.~5.}\n\\[\n\\mathbf Z=(Z_1,\\hl{Z_1},\\ldots,Z_n)\\in\\{0,1\\}^n\n\\]\n\n\\emph{(b) Below (5.20), p.~12.} \\ldots\\ for the count vector $\\mathbf v'=\\mathbf v+(-1,1,\\hl{1-1})$.\n\n\\emph{(c) Display (B.8), p.~20.}\n\\[\n\\hl{n_{10}}\\ge0,\\quad v_{10}\\ge j-n_{11}-n_{01},\\quad v_{10}\\ge n\\tau_0,\\quad n\\ge j+n_{10},\n\\]\n\n\\emph{(d) Proof of Theorem 4.1, p.~21.} Let $j=u_{11}+\\hl{u_{01}}$, and let $\\hl{v_{01}}$ be the value determined in Step (2b). Let $\\mathbf v$ be the vector determined by (2a) for this value of $j$, using this value of $\\hl{v_{01}}$. \\ldots\\ Let $\\mathbf v'$ be the vector constructed from $v'_{10}=1$ using (4.4) and $j=u_{11}+\\hl{u_{01}}$.\n\n\\emph{(e) Proof of Lemma B.10, p.~26.} Case II: $v_{11}$ is odd Then $\\hl{v_{11}}$ is odd, since $n$ is even.\n\n\\emph{(f) Appendix A, p.~18.} Since this quantity \\hl{depends on $\\mathbf w$ only through} its associated count vector $\\mathbf v$, we also write $T(\\mathbf v,\\mathbf Z)$ for $T(\\mathbf n,\\mathbf Z)$.\n\n\\emph{(g) Proposition 6.2, p.~14.} \\ldots\\ where the probability is with respect to \\hl{the variable $\\mathbf Z$}.\n\n\\emph{(h) Section 5.2, p.~13, and Definition 6.1, p.~14.} We now state \\hl{on} our main result on Algorithm 5.3. \\ldots\\ Given \\hl{a} observed data $(\\mathbf M,\\mathbf Z)$ \\ldots", "corrected": "\\emph{(a) Equation (2.3).}\n\\[\n\\mathbf Z=(Z_1,\\hl{Z_2},\\ldots,Z_n)\\in\\{0,1\\}^n\n\\]\n\n\\emph{(b) Below (5.20).} \\ldots\\ for the count vector $\\mathbf v'=\\mathbf v+(-1,1,\\hl{1,-1})$.\n\n\\emph{(c) Display (B.8).}\n\\[\n\\hl{n_{01}}\\ge0,\\quad v_{10}\\ge j-n_{11}-n_{01},\\quad v_{10}\\ge n\\tau_0,\\quad n\\ge j+n_{10},\n\\]\n\n\\emph{(d) Proof of Theorem 4.1.} Let $j=u_{11}+\\hl{u_{10}}$, and let $\\hl{v_{10}}$ be the value determined in Step (2b). Let $\\mathbf v$ be the vector determined by (2a) for this value of $j$, using this value of $\\hl{v_{10}}$. \\ldots\\ Let $\\mathbf v'$ be the vector constructed from $v'_{10}=1$ using (4.4) and $j=u_{11}+\\hl{u_{10}}$.\n\n\\emph{(e) Proof of Lemma B.10.} Case II: $v_{11}$ is odd. Then $\\hl{v_{00}}$ is \\hl{also} odd, since $n$ is even.\n\n\\emph{(f) Appendix A.} \\hl{The distribution of this quantity under uniform randomization depends on $\\mathbf w$ only through} its associated count vector $\\mathbf v$.\n\n\\emph{(g) Proposition 6.2.} \\ldots\\ where the probability is with respect to \\hl{the joint distribution of $\\mathbf Z$ and $\\mathbf J$}.\n\n\\emph{(h) Section 5.2 and Definition 6.1.} We now state \\del{on} our main result on Algorithm 5.3. \\ldots\\ Given \\del{a} observed data $(\\mathbf M,\\mathbf Z)$ \\ldots", "evidence": "I checked the pages and, for (c)--(f), a short derivation or example.\n\nFor (c), subtracting $j-v_{10}-n_{01}$ from $j-v_{10}$ leaves $n_{01}$.\n\nFor (d), let $j=u_{11}+u_{10}$ and write $v_{10}=u_{10}-k$. Using $n\\tau_0=u_{10}-u_{01}$ and substituting into (4.4), I obtain\n\\[\n\\mathbf v=(u_{11}+k,\\ u_{10}-k,\\ u_{01}-k,\\ u_{00}+k)=\\mathbf u+k(1,-1,-1,1),\n\\]\nwhich is the relation that (B.13) needs. For the printed index, let $\\mathbf u=(1,2,0,1)$, with $n=4$ and $n\\tau_0=2$. Then $u_{11}+u_{01}=1$, while $u_{11}+u_{10}=3$. At $j=1$ and $n\\tau_0=2$, formula (4.4) implies that $v_{01}\\ge0$ only if $v_{10}\\ge2$, and that $v_{11}\\ge0$ only if $v_{10}\\le1$. No table can be constructed.\n\nFor (e), by (B.53), I have $n=v_{11}+2+v_{00}$. Since $n$ is even, $v_{00}=n-2-v_{11}$ has the parity of $v_{11}$.\n\nFor (f), let $n=2$, $m=1$, $\\mathbf w=((1,1),(0,0))$, and let $\\mathbf w'$ be $\\mathbf w$ with the two subjects exchanged. The two tables have the same count vector. At $\\mathbf Z=(1,0)$ they give $T=1$ and $T=-1$. Under uniform randomization, both have $T=\\pm1$ with probability $1/2$ each.", "problem": "Each slip is local, and the intended reading is plain from the surrounding formulas.\n\nIn (a), the vector $\\mathbf Z$ has one indicator for each subject, and its second coordinate is $Z_2$. In (b), the shift has four coordinates, as in (5.17), and a comma is missing.\n\nIn (c), the display compares the third argument of the maximum in (B.4), $j-v_{10}-n_{01}$, with the first argument of the minimum in (B.5), $j-v_{10}$. The resulting condition is $n_{01}\\ge0$, not $n_{10}\\ge0$. Both hold for every observed count vector, and the criterion (B.2)--(B.3) is unchanged.\n\nIn (d), Step 2(a) of Algorithm 4.3 sets $v_{11}=j-v_{10}$, and Step 2(b) determines $v_{10}$. Then the line of count vectors through $\\mathbf u$ has index $j=u_{11}+u_{10}$. With the printed index $j=u_{11}+u_{01}$, the vector $\\mathbf v$ need not lie on that line, and the relation $\\mathbf v=\\mathbf u+k(1,-1,-1,1)$, which the proof uses for (B.13), fails.\n\nIn (e), the printed sentence repeats the assumption of the case. It is meant to state the parity of $v_{00}$, which is odd.\n\nIn (f), the quantity $T(\\mathbf w,\\mathbf Z)$ is a function of the labeled table and the assignment, and two tables with the same count vector can give different values at the same assignment. What depends on $\\mathbf w$ only through $\\mathbf v$ is its distribution under uniform randomization, and this is what the arguments that follow use. The symbol $T(\\mathbf n,\\mathbf Z)$ at the end of the same sentence is the subject of \\fid{P003-N017}.\n\nIn (g), Section 6 lets the missingness indicator $\\mathbf J$ depend on $\\mathbf Z$ and $\\mathbf Y$ and carry randomness of its own. The probability in Proposition 6.2 is then over the joint distribution of $\\mathbf Z$ and $\\mathbf J$. This concerns only the description of the probability space. The statement of the proposition is corrected in \\fid{P003-E004}. The two sentences in (h) have a superfluous word each.", "what_survives": "These local repairs do not change the intended results. With the corrected symbols, (B.8), the argument for (B.13) and Case II of the proof of Lemma B.10 read as the surrounding formulas require."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E015", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first-scenario block of Table 2, with columns Sample size, Permutation Cov., Permutation Med. Length, Wald Cov. and Wald Med. Length. A red box marks the four entries of the row for sample size 50: 0.97, 0.37, 0.93, 0.39. The row for sample size 100 directly below has the same entries. The rows for 200 and 1000 read 0.96, 0.26, 0.93, 0.28 and 0.96, 0.12, 0.95, 0.12.", "caption": "Table 2, first scenario, p. 15. The marked row, for sample size 50, has the same four entries as the row for sample size 100 below it.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E016.png"}, {"alt": "Two stacked panels with a common horizontal axis, k from 0 to 25, the number of treated units with potential outcomes (1,1). Upper panel: the length of the 95 percent Wald interval for each k, rising from 0 at k = 0 to about 0.565 at k = 12 and 13 and falling symmetrically. Six blue circles at k = 10 to 15 mark intervals that cover 0, with lengths between 0.554 and 0.565; orange squares mark the other twenty values. A solid line at 0.554 is labelled 'Shortest interval that covers 0: 0.554 (exact median length: 0.562)'. A dashed red line at 0.395 is labelled 'Printed median length 0.39: before rounding, at most 0.395'. Lower panel: bars for the probability of each k, concentrated on k = 8 to 17; the bars for k = 10 to 15 are blue and the label reads 'Intervals that cover 0: total probability 0.911 (printed coverage 0.93)'.", "caption": "Wald intervals in the stated 50-unit design of Table 2, first scenario. Upper panel: length of the interval for each value of $k$; the intervals that cover 0 (circles) all lie on or above the solid line at 0.554, and the dashed line at 0.395 is an upper bound on any median that rounds to 0.39. Lower panel: probability of each $k$; the values of $k$ whose interval covers 0 have total probability 0.911.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E016-wald-lengths.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T8", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T8"}], "headline": "The first-scenario row of Table 2 for $n=50$ repeats the row for $n=100$, and its Wald coverage 0.93 and Wald median length 0.39 cannot both come from the stated 50-unit design; the row has to be recomputed from recovered or regenerated 50-unit outputs.", "id": "P003-E016", "location": "Table 2, first scenario, row for sample size 50, p. 15 (PDF p. 15)", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P003-T8"], "text": {"as_printed": "\\begin{audittable}{rrrrr}{Table 2, first scenario, first two rows.}\nSample size & First Scenario & & & \\\\\n & Permutation & & Wald & \\\\\n & Cov. & Med. Length & Cov. & Med. Length \\\\\n50 & 0.97 & 0.37 & \\hl{0.93} & \\hl{0.39} \\\\\n100 & 0.97 & 0.37 & 0.93 & 0.39\n\\end{audittable}", "corrected": "Table 2, first scenario, $n=50$: the printed row repeats the values for $n=100$ and is incompatible with the stated 50-unit experiment. \\hl{The row is to be recomputed from recovered or regenerated outputs for $n=50$, with the Wald coverage and the Wald median length computed from the same intervals, and the two permutation entries checked against the same outputs.}\n\nFor the stated design (50 units, 25 with potential outcomes $(1,1)$ and 25 with $(0,0)$, 25 units treated) and the 95\\% Wald interval with the HC1 variance of the replication code, the exact coverage is $0.9112578194$ and the exact median length is $0.5617043865$, which are $0.91$ and $0.56$ to two decimal places. These exact values establish the inconsistency. They are not the values of the original $10^4$ simulations and are not offered as replacements for them.", "evidence": "The evidence is an exact computation over all assignments of the stated design, with an independent calculation in R that uses the regression and the HC1 variance of the replication code. Let $k$ be the number of treated units with potential outcomes $(1,1)$. Then\n\\[\nP(k)=\\frac{\\binom{25}{k}\\binom{25}{25-k}}{\\binom{50}{25}},\\qquad \\widehat\\tau_k=\\frac{2k-25}{25},\\qquad k=0,\\ldots,25.\n\\]\nThe proportions of ones in the two arms are $p_1=k/25$ and $p_0=(25-k)/25$, and the HC1 variance estimate is\n\\[\nV_k=\\frac{50}{48}\\left\\{\\frac{p_1(1-p_1)}{25}+\\frac{p_0(1-p_0)}{25}\\right\\}=\\frac{k(25-k)}{7500}.\n\\]\nLet $z=\\Phi^{-1}(0.975)$; then $1.95996398454005<z<1.95996398454006$. The interval $\\widehat\\tau_k\\pm z\\sqrt{V_k}$ has length $w_k=2z\\sqrt{V_k}$, and it covers 0 exactly when $\\widehat\\tau_k^2\\le z^2V_k$. With $d=2k-25$, this condition reads $48d^2\\le z^2(625-d^2)$. It holds for $|d|\\le 5$ and fails for $|d|\\ge 7$. Since $d$ is odd, I conclude that the interval covers 0 exactly when $10\\le k\\le 15$. For these $k$, I have $k(25-k)\\ge 150$ and $V_k\\ge 1/50$, which gives\n\\[\nw_k\\ge 2z\\sqrt{1/50}\\approx 0.5544.\n\\]\nConsider any finite set of runs whose coverage rounds to 0.93. More than half of its intervals cover 0. Then the middle length, or each of the two middle lengths when the number of runs is even, is at least $2z\\sqrt{1/50}$, and so is the median. A median printed as 0.39 is at most $0.395$, which is a contradiction. Summing $P(k)$ over $10\\le k\\le 15$, I obtain the exact coverage\n\\[\n\\frac{14399081696900}{15801325804719}\\approx 0.9113,\n\\]\nand the median of $w_k$ under $P$ is $2z\\sqrt{77/3750}\\approx 0.5617$, attained at $k=11$ and $k=14$. The same argument applies to the variance printed in (5.9)--(5.11), which omits the factor $50/48$: the interval covers 0 for the same values of $k$, and every such interval has length at least $2z\\sqrt{12/625}\\approx 0.5432$.\n\n\\begin{audittable}{rrrrr}{Wald columns of Table 2, first scenario: printed values and exact values for the stated designs.}\nSample size & Cov., printed & Cov., exact & Med. length, printed & Med. length, exact \\\\\n50 & 0.93 & 0.9113 & 0.39 & 0.5617 \\\\\n100 & 0.93 & 0.9287 & 0.39 & 0.3947\n\\end{audittable}", "problem": "The row for sample size 50 in the first scenario prints the same four values as the row for sample size 100. The caption fixes the design behind the row: 50 units, of which 25 have potential outcomes $(1,1)$ and 25 have $(0,0)$, with 25 units treated. The sample average treatment effect is 0. In this design the data depend on the assignment only through the number $k$ of treated units with potential outcomes $(1,1)$, and the 95\\% Wald interval covers 0 only when $10\\le k\\le 15$. Each of these intervals has length at least $0.554$. A coverage that rounds to 0.93 implies that more than half of the simulated intervals cover 0. Their median length is then at least $0.554$, and it cannot round to 0.39. At least one of the two Wald entries is wrong, whatever the random seed and the number of simulations. For the stated design the exact coverage is $0.9113$ and the exact median length is $0.5617$. The printed pair agrees with the exact values for $n=100$, which are $0.9287$ and $0.3947$.", "what_survives": "The algorithms, their proofs and the coverage and complexity results do not use this row. This finding does not concern the other rows of Table 2, the sparse-outcome example of Section 7.1 (reported coverage 89\\% at $n=50$ in the second scenario) or the timing tables; I have not examined those results here. Section 7.1 uses Table 2 to compare the two methods, and this row must be corrected before it is used for a quantitative comparison. The two permutation entries of the row also repeat the row for $n=100$. In a reduced run of the released balanced routine for the stated 50-unit design, with exact assignment weights and four independent Monte Carlo blocks, I obtain coverage $0.9773$ and median length $0.46$, against $0.97$ and $0.37$ printed. These results describe the released routine; they do not recover the original simulation outputs and are not replacement values for the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E016", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the body of Table 5, headed Second Scenario, with the columns Sample size, Balanced, and Unbalanced with K = 1000, 2000, 5000, 10,000 and 20,000. The row for sample size 500 reads 1182.66, 439.95, 282.34, 439.95, 1590.83, 3247.27. Red outlines surround the two entries 439.95, under K = 1000 and under K = 5000. The rows above read 14.08, 0.24, 0.47, 1.14, 2.31, 4.71 for sample size 50; 78.20, 1.39, 2.81, 7.05, 14.41, 29.40 for 100; and 215.11, 11.98, 24.29, 61.24, 125.94, 258.15 for 200. The row below, for 1000, reads 2850.90, 833.64, 1657.10, 4313.17, 9819.47, 21568.74.", "caption": "Table 5, second scenario, p. 16. The outlines mark the two entries 439.95 in the row for sample size 500, under $K = 1000$ and $K = 5000$. The entry between them, for $K = 2000$, is 282.34.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E017.png"}], "formal_release": null, "formal_statements": [], "headline": "For $n = 500$ in the second scenario, the table script pools the runs with $K = 1000$ and $K = 5000$ and reports the pooled mean under both headings, consistent with Table 5 printing 439.95 seconds in both columns; the separate run times and widths have to be regenerated from the original outputs.", "id": "P003-E017", "location": "Table 5, row for sample size 500, columns $K = 1000$ and $K = 5000$, p. 16 (PDF p. 16), and the same cells of Table 7, p. 17; replication code 5_time_comparisons.R, lines 315–328", "pdf_page": 16, "printed_page": "16", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lrrrrrr}{Tables~5 and~7, Second Scenario, rows for sample size 500. The first numerical column is the balanced algorithm and the other five are the unbalanced algorithm.}\n & Balanced & $K = 1000$ & $K = 2000$ & $K = 5000$ & $K = 10{,}000$ & $K = 20{,}000$ \\\\\nTable~5 (seconds) & $1182.66$ & $\\hl{439.95}$ & $282.34$ & $\\hl{439.95}$ & $1590.83$ & $3247.27$ \\\\\nTable~7 (width) & $0.12$ & $\\hl{0.12}$ & $0.12$ & $\\hl{0.12}$ & $0.11$ & $0.11$\n\\end{audittable}\n\nReplication code (not printed in the article), \\texttt{5\\_time\\_comparisons.R}, lines 321, 323 and 328 (line 328 is shown with a break after each argument):\n\n\\texttt{\\ \\ \\hl{output\\_500\\_s2\\_5000}=rbind(\\hl{output\\_500\\_s2\\_5000}, read.csv(files\\_s2\\_500\\_1000[i]))}\n\n\\texttt{\\ \\ output\\_500\\_s2\\_5000=rbind(output\\_500\\_s2\\_5000, read.csv(files\\_s2\\_500\\_5000[i]))}\n\n\\texttt{time\\_500\\_s2=cbind(apply(\\hl{output\\_500\\_s2\\_5000},2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_2000,2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_5000,2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_50000,2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_20000,2,mean))}", "corrected": "Replication code, \\texttt{5\\_time\\_comparisons.R}, lines 315, 321 and 328 (line 328 is shown with a break after each argument):\n\n\\texttt{\\hl{output\\_500\\_s2\\_1000}=c()}\n\n\\texttt{\\ \\ \\hl{output\\_500\\_s2\\_1000}=rbind(\\hl{output\\_500\\_s2\\_1000}, read.csv(files\\_s2\\_500\\_1000[i]))}\n\n\\texttt{time\\_500\\_s2=cbind(apply(\\hl{output\\_500\\_s2\\_1000},2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_2000,2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_5000,2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_50000,2,mean),}\n\n\\texttt{\\ \\ apply(output\\_500\\_s2\\_20000,2,mean))}\n\nWith these three edits, each column holds the mean of its own setting. The entries for $K = 1000$ and $K = 5000$ in the rows for sample size 500 of Tables~5 and~7 are then to be regenerated from the original output files. No replacement values can be given here.", "evidence": "I checked this with an exact counterexample and a derivation. Lines 309--328 of the script and its table assembly were executed without change on five test files of one row each, one for each $K$, with run times $1, 2, 5, 10, 20$ and widths $1/2, 1/4, 1/8, 1/16, 1/32$. The script reports the run time $3 = (1+5)/2$ and the width $5/16 = (1/2 + 1/8)/2$ in both the column for $K = 1000$ and the column for $K = 5000$, as the table below shows. After the three edits of the correction, it returns each input in its own column.\n\n\\begin{audittable}{rrrrr}{What lines 309--328 of the table script report on the five test files. The two pooled values in each reported column are marked.}\n$K$ & Run time put in & Run time reported & Width put in & Width reported \\\\\n$1000$ & $1$ & $\\hl{3}$ & $1/2$ & $\\hl{5/16}$ \\\\\n$2000$ & $2$ & $2$ & $1/4$ & $1/4$ \\\\\n$5000$ & $5$ & $\\hl{3}$ & $1/8$ & $\\hl{5/16}$ \\\\\n$10{,}000$ & $10$ & $10$ & $1/16$ & $1/16$ \\\\\n$20{,}000$ & $20$ & $20$ & $1/32$ & $1/32$\n\\end{audittable}\n\nIn general, let $a$ be the mean of a column over the $N_A$ rows read for $K = 1000$, and let $b$ be its mean over the $N_B$ rows read for $K = 5000$. After the loop, the accumulator holds all $N_A + N_B$ rows, and both positions receive\n\\[\np = \\frac{N_A a + N_B b}{N_A + N_B}, \\qquad p - a = \\frac{N_B (b - a)}{N_A + N_B}, \\qquad p - b = \\frac{N_A (a - b)}{N_A + N_B} .\n\\]\nIf $a \\neq b$, neither position holds the mean of its own setting. The test values are not simulation results. The original simulation outputs are not supplied, so I cannot establish how the printed entries were produced or give corrected values.", "problem": "Each unbalanced column of Tables~5 and~7 is meant to give the mean run time, or the mean width, of the unbalanced algorithm over the simulated experiments with one value of $K$. For every other sample size, and for the first scenario, the script \\texttt{5\\_time\\_comparisons.R} keeps one accumulator for each $K$. The block for the second scenario at $n = 500$ has no accumulator for $K = 1000$. Line 321 appends the output for $K = 1000$ to the accumulator for $K = 5000$, and line 323 appends the output for $K = 5000$ to the same object. Line 328 places the column means of that object in the first and in the third position. Both positions then hold the mean over the runs with $K = 1000$ and with $K = 5000$ together, for the run time and for the width. Table~5 prints 439.95 seconds under both headings, which exceeds the 282.34 seconds printed for $K = 2000$, and Table~7 prints 0.12 under both. With equal numbers of runs, the pooled value is the average $(a+b)/2$ of the two separate means $a$ and $b$. One such number does not determine the two: the pairs $(100, 779.90)$ and $(200, 679.90)$ both give $439.95$. The run times and widths of the two settings cannot be recovered from the printed tables.", "what_survives": "The correction repairs the table script and adds no assumption. The columns for $K = 2000$, $10{,}000$ and $20{,}000$, the balanced column, the other sample sizes and the first scenario do not change. The simulations, the coverage results of Tables~2 and~3 and the theoretical results do not pass through this block. Section~7.2 says that for $n = 500$ or $1000$ and $K \\ge 5000$ the balanced algorithm ``typically outperforms'' the unbalanced one. Among the twelve printed comparisons of this kind, the balanced algorithm is faster in eleven, and the exception is the cell in question (439.95 against 1182.66). Suppose that 439.95 is the rounded pooled mean from this script with equal numbers of one-row output files. Since run times are nonnegative, each separate mean is at most $2(439.955) = 879.91$ seconds, below the balanced mean of at least 1182.655 seconds, so the unbalanced algorithm remains faster in that cell. The qualified statement of Section~7.2 is not refuted. It is not known whether the two widths 0.12 of Table~7 change after rounding."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E017", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 5 with its caption. The caption reads: Average computation times (in seconds) for the balanced algorithm and unbalanced algorithm with the difference-in-means statistic and sample sizes 50, 100, 200, 500, and 1000 in Simulation Scenario 2. The numbers of treated units are 25, 50, 100, 250, and 500, respectively. A red outline surrounds the number 50 in this list. The table has the columns Sample size, Balanced, and Unbalanced with K = 1000, 2000, 5000, 10,000 and 20,000. A second red outline surrounds the five Unbalanced entries of the row for sample size 100: 1.39, 2.81, 7.05, 14.41, 29.40. The Balanced entry of that row is 78.20.", "caption": "Table 5, p. 16. The outlines mark the number of treated units that the caption states for sample size 100 and the five run times of the unbalanced algorithm in that row.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E018.png"}, {"alt": "Crop of Table 7 with its caption. The caption reads: Average widths for the balanced algorithm and unbalanced algorithm with the difference-in-means statistic and sample sizes 50, 100, 200, 500, and 1000 in Simulation Scenario 1. The numbers of treated units are 25, 50, 100, 250, and 500, respectively. A red outline surrounds the number 50 in this list. The table is headed Second Scenario and has the columns Sample size, Balanced, and Unbalanced with K = 1000, 2000, 5000, 10,000 and 20,000. A second red outline surrounds the five Unbalanced entries of the row for sample size 100: 0.29, 0.29, 0.28, 0.28, 0.28. The Balanced entry of that row is 0.26. The other rows read 0.35 throughout for sample size 50; 0.18, 0.19, 0.18, 0.18, 0.18, 0.18 for 200; 0.12, 0.12, 0.12, 0.12, 0.11, 0.11 for 500; and 0.08, 0.09, 0.08, 0.08, 0.08, 0.08 for 1000.", "caption": "Table 7, p. 17. The outlines mark the same number of treated units in the caption and the five widths of the unbalanced algorithm in the row for sample size 100. The width of the balanced algorithm in that row is 0.26.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E018-b.png"}, {"alt": "Line chart. The horizontal axis is the number of randomizations K in the unbalanced algorithm, with the five values 1000, 2000, 5000, 10,000 and 20,000. The vertical axis is the mean width of the 95 percent interval, from 0.24 to 0.30. The title reads: The printed widths at n = 100 match 30 treated units, not the 50 of the caption. Orange diamonds mark the entries printed in Table 7, row 100, Unbalanced: 0.29, 0.29, 0.28, 0.28, 0.28, each with a pale bar from 0.005 below to 0.005 above it. A blue line with circles, labelled recalculated with 30 treated and 70 control units, as in the replication files, runs from 0.2926 at K = 1000 through 0.2861 and 0.2845 to 0.2841 at K = 20,000, and every circle lies inside the bar of the printed entry. A green line with squares, labelled recalculated with 50 treated and 50 control units, as in the caption, runs from 0.2510 at K = 1000 to 0.2457 at K = 20,000, about 0.03 to 0.04 below the printed entries.", "caption": "Mean width of the interval of the unbalanced algorithm at $n = 100$ in the second scenario, by number of randomizations $K$. The entries printed in Table 7 (orange diamonds; each bar covers the values that round to the entry) agree, after rounding, with Monte Carlo estimates for 30 treated units (blue circles) and lie well above those for 50 treated units (green squares).", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E018-widths.png"}], "formal_release": null, "formal_statements": [], "headline": "At $n = 100$ in the second scenario, the simulation script in the replication files runs the unbalanced algorithm with 30 treated units where the captions of Tables 5 and 7 state 50; the widths printed in Table 7 match recalculations with 30 treated units and are incompatible with ordinary 50/50 replication.", "id": "P003-E018", "location": "Tables 5 and 7, rows for sample size 100, with the sentence on treated units in both captions, pp. 16–17 (PDF pp. 16–17); replication code 1_simulation_unbalanced_case_balanced_design_2.R, lines 24 and 50–53", "pdf_page": 16, "printed_page": "16–17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Table~5, caption: ``Average computation times (in seconds) for the balanced algorithm and unbalanced algorithm with the difference-in-means statistic and sample sizes 50, 100, 200, 500, and 1000 in Simulation Scenario 2. The numbers of treated units are 25, 50, 100, 250, and 500, respectively.'' The caption of Table~7 contains the same sentence on treated units.\n\n\\begin{audittable}{lrrrrrr}{Tables~5 and~7, Second Scenario, rows for sample size 100. The first numerical column is the balanced algorithm and the other five are the unbalanced algorithm.}\n & Balanced & $K = 1000$ & $K = 2000$ & $K = 5000$ & $K = 10{,}000$ & $K = 20{,}000$ \\\\\nTable~5 (seconds) & $78.20$ & $1.39$ & $2.81$ & $7.05$ & $14.41$ & $29.40$ \\\\\nTable~7 (width) & $0.26$ & $0.29$ & $0.29$ & $0.28$ & $0.28$ & $0.28$\n\\end{audittable}\n\nReplication code (not printed in the article), lines 24, 50 and 51 of the script\n\n\\texttt{1\\_simulation\\_unbalanced\\_case\\_balanced\\_design\\_2.R}:\n\n\\texttt{\\#The number of treated units is nt=0.5*n.}\n\n\\texttt{n=100}\n\n\\texttt{nt=\\hl{30}}", "corrected": "Replication code, lines 50 and 51 of the script\n\n\\texttt{1\\_simulation\\_unbalanced\\_case\\_balanced\\_design\\_2.R}:\n\n\\texttt{n=100}\n\n\\texttt{nt=\\hl{50}}\n\nThe five ``Unbalanced'' entries of the row for sample size 100 in Table~5 and the five in Table~7 are then to be recomputed for $K = 1000$, $2000$, $5000$, $10{,}000$ and $20{,}000$, with the computing environment of the run times recorded. If the original output files exist, they should be examined before the ten printed entries are attributed to a particular version of the script. The script for a third scenario, which the paper does not report, sets \\texttt{nt=30} at $n = 100$ in the same way; that is a separate correction.", "evidence": "The evidence is a recalculation from the replication files, together with an exact enumeration. A trace of the R functions in the replication files shows that the script generates 30 treated and 70 control observations at $n = 100$ and passes 30 to the permutation routine for all five values of $K$. The script for the balanced algorithm generates 50 and 50. Let $n = 100$, let the count vector be $\\mathbf{v} = (8, 0, 0, 92)$, and take the difference-in-means statistic with $\\alpha = 0.05$. I enumerate the nine possible observed tables with their hypergeometric probabilities. With exact $p$-values, I find that the mean width of the interval is\n\\[\n\\frac{13}{50} = 0.2600 \\quad \\text{for } m = 50, \\qquad \\frac{17647336669}{62029298100} \\approx 0.2845 \\quad \\text{for } m = 30 .\n\\]\nFor $m = 50$ the width is $0.26$ for every observed table. The permutation routine of the replication files was also run at each $K$ on the nine tables, and the widths were averaged with the same probabilities. The results are in the table below. For each $K$, the value for 30 treated units rounds to the printed entry and the value for 50 treated units rounds to 0.25. These values come from one to six runs of the routine per observed table, and they are not replacement entries for Table~7. For $m = 50$ and $K = 20{,}000$ the expected width is at most $0.260002$, and the probability that the mean over $10^4$ independent simulated experiments rounds to 0.28 or more is below $1.2 \\times 10^{-25}$. These calculations do not supply replacement run times for Table~5.\n\n\\begin{audittable}{rrrr}{Mean width of the interval of the unbalanced algorithm at $n = 100$ in the second scenario: the entries of Table~7 and Monte Carlo estimates from the permutation routine of the replication files.}\n$K$ & Printed in Table~7 & 30 treated units & 50 treated units \\\\\n$1000$ & $0.29$ & $0.2926$ & $0.2510$ \\\\\n$2000$ & $0.29$ & $0.2861$ & $0.2466$ \\\\\n$5000$ & $0.28$ & $0.2845$ & $0.2457$ \\\\\n$10{,}000$ & $0.28$ & $0.2846$ & $0.2457$ \\\\\n$20{,}000$ & $0.28$ & $0.2841$ & $0.2457$\n\\end{audittable}", "problem": "Tables~5 and~7 describe a comparison of the balanced and the unbalanced algorithm on balanced designs. For $n = 100$ the captions state 50 treated units, and the script for the balanced algorithm sets 50. The script that runs the unbalanced algorithm in the second scenario sets 30 treated units at $n = 100$, below a comment that says the number of treated units is $n/2$. It sets 25 and 100 at $n = 50$ and $n = 200$, and the script for the first scenario sets 50 at $n = 100$. If generated by these scripts, the ``Unbalanced'' entries at $n = 100$ come from trials with 30 treated and 70 control units, and the ``Balanced'' entry comes from trials with 50 and 50.\n\nThe widths printed in Table~7 match recalculations with 30 treated units. In the second scenario at $n = 100$, there are 8 units with potential outcomes $(1,1)$ and 92 with $(0,0)$. Only nine observed tables are possible, and the mean width can be computed exactly. With exact $p$-values and 30 treated units it is about $0.2845$, in agreement with the scale of the printed 0.28 to 0.29. With 50 treated units it is $0.2600$, and the permutation routine of the replication files returns about $0.25$; neither value rounds to 0.28. In Table~7 the intervals of the unbalanced algorithm appear wider than those of the balanced one at $n = 100$ (0.28 to 0.29 against 0.26). I take these recalculations as strong evidence that the allocation mismatch explains this excess width. The table script takes the run times of Table~5 from the same output records as the widths. If the scripts in the replication files produced that row, its five run times belong to the design with 30 treated units as well. The original outputs are not in the replication files, so I cannot establish which script version produced either printed row.", "what_survives": "The repair changes one number in a simulation script and adds no assumption. In both designs the sample average treatment effect is zero, and Algorithm~5.3 is valid for either allocation. The other sample sizes of Tables~5 and~7, the first scenario in Tables~4 and~6, Tables~1--3 and the theoretical results are unaffected by this allocation error. One row does not refute the recommendation of Section~7.2 on the choice of algorithm. The row for $n = 100$ cannot serve as a comparison on a common design until it is recomputed, and the replacement run times and widths are not known."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E018", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the foot of page 7 with footnotes 9 and 10. Footnote 9 ends: Setting alpha = 17/18, it can be shown by direct combinatorial analysis (or simulation) that only -1/9, 0, and 1/9 are accepted. Then the realized confidence interval is [-1/9, 1/9], excluding the observed estimate of 1/7. Red outlines surround -1/9 in the list of accepted values and -1/9 as the lower endpoint.", "caption": "Footnote 9, p. 7 (PDF p. 7). The outlines mark the value $-1/9$ in the list of accepted values and as the lower endpoint of the interval.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E019.png"}, {"alt": "Listing of lines 40 to 56 of the R script 1_cheking_lemma3_1.R, with line numbers. Line 45 reads: dim_temp = sum(y1*assignment[i,])/7 - sum(y0*assignment[i,])/2. A red outline surrounds the second term, sum(y0*assignment[i,])/2.", "caption": "Replication script 1_cheking_lemma3_1.R, lines 40–56 (a listing of the released code, not a page of the article). The outline marks the control-group term of line 45, which sums over the treated units.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E019-code.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T21", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T21"}], "headline": "In the example of footnote 9, the value $-1/9$ is rejected at $\\alpha=17/18$: the accepted values are $0$ and $1/9$, and the interval is $[0,1/9]$, which still excludes the estimate $1/7$.", "id": "P003-E019", "location": "Footnote 9, p. 7 (PDF p. 7); replication script 1_cheking_lemma3_1.R, lines 45 and 94", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T21"], "text": {"as_printed": "This claim is not always true for unbalanced trials. For example, consider an experiment with 9 units, where 7 are assigned to treatment and the true potential outcome table is $v_{11}=0$, $v_{10}=1$, $v_{01}=0$, and $v_{00}=8$. Suppose we observe $n_{11}=1$, $n_{10}=6$, $n_{01}=0$, and $n_{00}=2$, and therefore $T(\\mathbf n)=1/7$. Setting $\\alpha=17/18$, it can be shown by direct combinatorial analysis (or simulation) that only $\\hl{-1/9}$, $0$, and $1/9$ are accepted. Then the realized confidence interval is $[\\hl{-1/9},1/9]$, excluding the observed estimate of $1/7$.", "corrected": "This claim is not always true for unbalanced trials. For example, consider an experiment with 9 units, where 7 are assigned to treatment and the true potential outcome table is $v_{11}=0$, $v_{10}=1$, $v_{01}=0$, and $v_{00}=8$. Suppose we observe $n_{11}=1$, $n_{10}=6$, $n_{01}=0$, and $n_{00}=2$, and therefore $T(\\mathbf n)=1/7$. Setting $\\alpha=17/18$, it can be shown by direct combinatorial analysis (or simulation) that only \\hl{$0$ and $1/9$} are accepted. Then the realized confidence interval is $[\\hl{0},1/9]$, excluding the observed estimate of $1/7$.\n\nIn the script \\texttt{1\\_cheking\\_lemma3\\_1.R}, line 45 becomes\n\n\\texttt{dim\\_temp = sum(y1*assignment[i,])/7 - sum(y0*\\hl{(1-assignment[i,])})/2}\n\nand line 94 becomes\n\n\\texttt{sate\\_grid = ((\\hl{n11}-n01-nt):(\\hl{n11}-n01-nt+n))}", "evidence": "I checked an exact enumeration, in rational arithmetic, of the $42$ possible count vectors and the $36$ assignments. The table lists the largest tail count for each value of $9\\tau_0$.\n\\begin{audittable}{rrrl}{Example of footnote 9: for each value of $9\\tau_0$, the largest tail count over the possible tables, out of $36$ assignments, and the largest $p$-value. Acceptance at $\\alpha=17/18$ needs a tail count of at least $34$.}\n$9\\tau_0$ & Largest tail count & Largest $p$-value & Accepted \\\\\n$-6$ & $1$ & $1/36$ & no \\\\\n$-5$ & $3$ & $1/12$ & no \\\\\n$-4$ & $6$ & $1/6$ & no \\\\\n$-3$ & $16$ & $4/9$ & no \\\\\n$-2$ & $18$ & $1/2$ & no \\\\\n$-1$ & $29$ & $29/36$ & no \\\\\n$0$ & $36$ & $1$ & yes \\\\\n$1$ & $36$ & $1$ & yes \\\\\n$2$ & $24$ & $2/3$ & no \\\\\n$3$ & $3$ & $1/12$ & no \\\\\n\\end{audittable}\nFor $9\\tau_0=-1$ the maximum is attained by the table $(v_{11},v_{10},v_{01},v_{00})=(1,0,1,7)$, for which the observed deviation is $|1/7+1/9|=16/63$. Of the $36$ choices of the two control units, the $7$ pairs formed by the $(0,1)$ unit and a $(0,0)$ unit give $T=-5/14$ and a deviation of $31/126<16/63$. The other $29$ pairs lie in the tail. Run unchanged, the script accepts $9\\tau_0\\in\\{-1,0,1\\}$. With lines 45 and 94 corrected, it reproduces the ten tail counts of the table.", "problem": "By Definition 2.1 and Algorithms 2.2--2.3, a value $\\tau_0$ is accepted if there exists a table, possible given the data, such that its effect is $\\tau_0$ and its $p$-value is at least $\\alpha$. For $\\tau_0=-1/9$, I find that the largest $p$-value over the six possible count vectors is $29/36$, which is less than $17/18=34/36$. Hence $-1/9$ is rejected. The values $0$ and $1/9$ are each accepted through a possible table with $p$-value $1$, and no other value reaches $17/18$. The interval is $[0,1/9]$. The printed set $\\{-1/9,0,1/9\\}$ is the accepted set exactly when $2/3<\\alpha\\le29/36$.\n\nThe script of the replication files for this footnote reproduces the printed set and has two errors. Line 45 computes the mean of the control group over the treated units. Line 94 starts the grid of values of $9\\tau_0$ at $n_{10}-n_{01}-m=-1$, where (2.10) gives $n_{11}-n_{01}-m=-6$.", "what_survives": "The point of the footnote survives. With the same data and the same $\\alpha$, the interval $[0,1/9]$ excludes the estimate $1/7$, since $1/7-1/9=2/63>0$. Lemma 3.1 and the results after it do not use the numbers of the example, and the simulation scripts do not load this script."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E019", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the caption and the head of Table 7. The caption begins: Average widths for the balanced algorithm and unbalanced algorithm with the difference-in-means statistic and sample sizes 50, 100, 200, 500, and 1000 in Simulation Scenario 1. A red outline surrounds the words Simulation Scenario 1. Below the caption, the head of the table reads Sample size, Second Scenario, then Balanced and Unbalanced, then K = 1000, K = 2000, K = 5000, K = 10,000 and K = 20,000.", "caption": "Table 7, caption and head of the table, p. 17. The outline marks the scenario named in the caption. The header of the table below the caption reads “Second Scenario”.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E020.png"}], "formal_release": null, "formal_statements": [], "headline": "The caption of Table 7 should read “Simulation Scenario 2”: the table reports the second scenario, as its header and Section 7.2 say.", "id": "P003-E020", "location": "Table 7, caption, p. 17 (PDF p. 17)", "pdf_page": 17, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Table~7, caption: ``Average widths for the balanced algorithm and unbalanced algorithm with the difference-in-means statistic and sample sizes 50, 100, 200, 500, and 1000 in \\hl{Simulation Scenario 1}.'' Header of the table: ``Second Scenario''.", "corrected": "Table~7, caption: ``Average widths for the balanced algorithm and unbalanced algorithm with the difference-in-means statistic and sample sizes 50, 100, 200, 500, and 1000 in \\hl{Simulation Scenario 2}.''", "evidence": "The evidence is a check of the page. The table script in the replication files also builds this table from the output for the second scenario.", "problem": "The caption of Table~7 repeats the caption of Table~6 and names the first scenario. The header of the table reads ``Second Scenario'', and Section~7.2 says that Tables~6 and~7 give the widths for the two scenarios of Tables~4 and~5.", "what_survives": "The entries of the table are those of the second scenario, and nothing else changes with this correction. Two rows of the same table are the subject of \\fid{P003-E017} and \\fid{P003-E018}."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E020", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the opening of Algorithm 4.4 on page 9 with equation (4.6), which defines the indicator V i as 1 when the absolute value of T(n(i)) minus tau(w tilde) is greater than or equal to the absolute value of T(n) minus tau(w tilde). A red outline surrounds the greater-than-or-equal sign.", "caption": "Algorithm 4.4, p. 9 (PDF p. 9). The outline marks the weak inequality in (4.6): a permutation that ties with the observed statistic belongs to the tail.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E021.png"}, {"alt": "Listing of lines 624 to 661 of the C++ file permutation_test.cpp, with line numbers. Line 625 sets term_obs to the absolute value of test_statistics minus parameter_value. Line 641 sets term_mc in the same way from the simulated statistic. Line 643 reads: if (term_mc >= term_obs), followed by count += 1. A red outline surrounds the comparison term_mc >= term_obs.", "caption": "Function perm_test_inner of permutation_test.cpp, lines 624–661 (a listing of the released code, not a page of the article). The outline marks the comparison on line 643; its two sides are the double-precision quantities set on lines 625 and 641.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E021-code.png"}, {"alt": "Two line charts of the largest p-value against the hypothesized effect, titled: The released code drops tied permutations, so its p-values fall below those of definition (4.6). The left chart covers all 51 effects from -0.58 to 0.42: three curves rise from 0 to 1 near -0.16 and fall back to 0, and the two curves labelled as coded lie on or below the curve labelled as defined. The right chart shows effects from -0.40 to -0.28 with a horizontal line at alpha = 0.05. At -0.34 the curve as defined is at 0.078, above the line, and the two curves as coded are at 0.045 and 0.035, below the line. A subtitle states that inverting at 0.05 gives the interval from -0.34 to 0.02 as defined and from -0.32 to 0.02 as coded.", "caption": "The largest $p$-value over the possible tables, for each hypothesized effect $\\tau_0$, for the data $(0,25,4,21)$ with $n=50$ and $m=25$: with the tail as defined in (4.6) (circles), and with the comparison of the released code in double precision, for the observed statistic computed from the two proportions (squares) and with mean in R (triangles). At $\\tau_0=-17/50$ the values are $0.078$, $0.045$ and $0.035$, so this effect is accepted at level $0.05$ under the definition and rejected by the coded comparison.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E021-tie-pvalues.png"}], "formal_release": null, "formal_statements": [], "headline": "The released code compares the permuted and the observed statistic in double precision and so drops permutations that tie with the observed statistic, which definitions (4.6) and (5.1) count; the comparison should be made on integer counts.", "id": "P003-E021", "location": "Algorithm 4.4, equation (4.6), p. 9 (PDF p. 9), and equation (5.1), p. 10; released code permutation_test.cpp, function perm_test_inner, lines 625 and 641–645", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Algorithm 4.4, equation (4.6), p.~9.}\n\\[\nV_i=\\mathbb{1}\\Bigl(\\bigl|T(\\mathbf n^{(i)})-\\tau(\\widetilde{\\mathbf w})\\bigr|\\ge\\bigl|T(\\mathbf n)-\\tau(\\widetilde{\\mathbf w})\\bigr|\\Bigr),\\qquad i\\in[\\![1,K]\\!],\n\\]\n\n\\emph{Section 1.5, p.~4.} Code \\hl{implementing our proposed algorithms} and replicating the tables provided in Section 7 is available at \\ldots\n\n\\emph{Released code, \\texttt{permutation\\_test.cpp}, lines 625, 641 and 643--645 (comments omitted; all three quantities are of type \\texttt{double}).}\n\n\\texttt{term\\_obs = abs(test\\_statistics - parameter\\_value);}\n\n\\texttt{term\\_mc = abs(test\\_statistics\\_mc - parameter\\_value);}\n\n\\texttt{if (\\hl{term\\_mc >= term\\_obs})\\{ count += 1; \\}}", "corrected": "Definition (4.6) stands. In the released code, the tail event for the difference in means is decided on integers. Let $u$ and $w$ be the numbers of successes in the treatment group and in the control group, and let $d=v_{10}-v_{01}$, so that $T=u/m-w/(n-m)$ and $\\tau=d/n$. Multiplying $|T-\\tau|$ by $nm(n-m)$ gives the integer\n\\[\nC(u,w;d)=\\bigl|n\\bigl((n-m)u-mw\\bigr)-d\\,m(n-m)\\bigr|,\n\\]\nand the indicator of (4.6) is computed as\n\\[\nV_i=\\mathbf 1\\bigl(\\hl{C(u^{(i)},w^{(i)};d)\\ge C(u,w;d)}\\bigr),\n\\]\nwhere $(u^{(i)},w^{(i)})$ are the success counts of the $i$th permuted assignment and $(u,w)$ are those of the observed data. The integer type must be wide enough to avoid overflow. A fixed floating-point tolerance is not a general guarantee of exactness. The simulation results that use this function must then be recomputed.", "evidence": "The evidence is an exact counterexample. I summed exact integer assignment counts over all possible count states, using rational arithmetic for the probabilities and double precision only for the coded comparison. I also checked the $37158$ indicators of the examples against the unchanged function \\texttt{perm\\_test\\_inner}, compiled from the released source.\n\\begin{audittable}{lrr}{The table $\\mathbf v=(4,0,17,29)$ for the data $(0,25,4,21)$, $n=50$, $m=25$: numbers of assignments and the corresponding probabilities.}\nSet of assignments & Number & Probability \\\\\nAll assignments & $126410606437752$ & $1$ \\\\\nTail as defined in (4.6) & $9816673905402$ & $0.0777$ \\\\\nTies with the observed statistic & $6531550843350$ & $0.0517$ \\\\\nTail as coded, observed statistic from the two proportions & $4838194239567$ & $0.0383$ \\\\\nTail as coded, observed statistic from \\texttt{mean} in R & $3571151604852$ & $0.0283$ \\\\\n\\end{audittable}\nOver the $55$ possible tables with $\\tau=-17/50$, the largest $p$-value is $0.0777$ as defined, and $0.0447$ or $0.0348$ as coded. Both coded values are below $0.045$, the threshold that the simulation settings apply to the Monte Carlo fraction. In a second example, with the data $(8,4,5,7)$ of the third row of Table 1 and the possible table $(10,0,3,11)$, the $p$-value is $57388/676039=0.0849$ as defined and $881/29393=0.0300$ as coded with the observed statistic from \\texttt{mean} in R 4.4.2: the code drops all $148500$ tied assignments out of $2704156$.\n\nThese checks do not establish undercoverage of an interval. They also do not determine which entries of Tables 2--7 the defect changed: the original simulation outputs are not in the replication files, and I have not repeated the full simulations.", "problem": "Definitions (2.12), (4.6) and (5.1) place in the tail every permutation whose centered statistic is at least as large in absolute value as the observed one, ties included. For the difference in means, both sides lie on a lattice, and ties have positive probability. The released function \\texttt{perm\\_test\\_inner} forms the two sides as double-precision numbers and compares them with \\texttt{>=}. Two quantities that are equal as rational numbers can then differ in the last binary digit, and a tie is counted or dropped according to the rounding.\n\nFor example, let $n=50$ and $m=25$, with observed counts $(n_{11},n_{10},n_{01},n_{00})=(0,25,4,21)$ and the possible table $\\mathbf v=(4,0,17,29)$, for which $\\tau=-17/50$. For the permuted counts $(0,25,13,12)$, I have $|T-\\tau|=9/50$, the same value as for the observed counts. In double precision, with the observed statistic computed from the two proportions, the permuted side is smaller by $2^{-55}$, and the permutation is not counted. Summing over all assignments, I find that the $p$-value of this table is $0.0777$ as defined. The indicator as coded has expectation $0.0383$ when the observed statistic is computed from the two proportions, and $0.0283$ when it is computed with the R function \\texttt{mean}, as the released wrapper does (R 4.4.2). Since the expectation of the coded indicator differs from the $p$-value, more Monte Carlo draws do not remove the difference.\n\nThe interval changes as well. Inverting the fully enumerated tails over all $2860$ possible tables at level $0.05$, I obtain $[-17/50,1/50]$ with the tail as defined and $[-8/25,1/50]$ with the coded comparison. The latter interval is a strict subset, of width $0.34$ against $0.36$. Since Proposition 4.6 and Theorem 5.5 concern the procedures with the tail as defined, their coverage guarantees do not carry over to the released code as it stands. The routines for the difference in means, in balanced and in general designs, both call this function. The permutation columns of Tables 2--7 may be affected, but which printed entries change is not established.", "what_survives": "Definitions (2.12), (4.6) and (5.1), the algorithms as printed and the theorems about them are untouched. The estimand, the construction for missing outcomes and the Wald columns of the tables do not involve this comparison. What changes is the released code, and the permutation columns of Tables 2--7 should be recomputed after the repair."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E021", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph that ends with equation (5.27), J = {theta(v) : v in V}. Red boxes mark the words 'Iterating this procedure allows us to search the entire line' at the end of one line and 'segment L' at the start of the next. The sentence continues 'to find the set V consisting of v_k in L that are both possible and would be accepted by an application of Algorithm 5.1 (where possibility is checked using Lemma B.3).'", "caption": "Section 5.1.1, p. 13 (PDF p. 13), the paragraph before (5.27). The marked sentence says that the whole line segment $\\mathcal L$ is searched. This is the intended search; the released code’s shrinking bound can stop it early.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E022.png"}, {"alt": "Listing of source code with line numbers. Line 990 computes v10_max. A red box marks the condition 'k<(v10_max-v10+1)' in line 995, a while loop with the two conditions (k<(v10_max-v10+1)) and (abs(result)<1e-10). A second red box marks line 999, 'v10 = v10+1;', among the four updates v11 = v11-1; v10 = v10+1; v01 = v01+1; v00 = v00-1; of lines 998 to 1001. Line 1019 is 'k = k +1;'.", "caption": "Source code of the replication files, not a page of the article: permutation_test.cpp, lines 990–1001 and 1019–1020 (routine perm_test_SATE). The marked bound of line 995 contains v10, which the marked line 999 increases at every step, so the bound falls by one each time $k$ rises by one.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E022-code.png"}], "formal_release": null, "formal_statements": [], "headline": "Five loops in the released code compute their bound from the count that they advance, allowing at most about half the steps along each line of count vectors; the code can stop before completing the search described in Section 5.1.1.", "id": "P003-E022", "location": "Section 5.1.1, p. 13 (PDF p. 13), paragraph before (5.27); replication files, permutation_test.cpp, lines 475, 803, 906, 995 and 1204; Table 3 and Tables 4–7, pp. 16–17", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~13: ``Iterating this procedure allows us to search the entire line segment $\\mathcal L$ to find the set $\\mathcal V$ consisting of $\\boldsymbol v_k\\in\\mathcal L$ that are both possible and would be accepted by an application of Algorithm 5.1 (where possibility is checked using Lemma B.3).''\n\nReplication files, \\texttt{permutation\\_test.cpp}, routine \\texttt{perm\\_test\\_SATE}, lines 995, 998--1001 and 1019:\n\n\\texttt{while ( (k<(v10\\_max-\\hl{v10}+1)) \\&\\& (abs(result)<1e-10))\\{}\n\n\\texttt{v11 = v11-1;}\n\n\\texttt{v10 = v10+1;}\n\n\\texttt{v01 = v01+1;}\n\n\\texttt{v00 = v00-1;}\n\n\\texttt{k = k +1;}\n\nLine 1204 has the same condition. Lines 475, 803 and 906 read \\texttt{for (int k=1; k<(v10\\_max-\\hl{v10}+1); k++)\\{}, with the same four updates in the body.", "corrected": "In all five loops, compare the count with its fixed maximum. Lines 995 and 1204 become\n\n\\texttt{while ( (\\hl{v10<v10\\_max}) \\&\\& (abs(result)<1e-10))\\{}\n\nand lines 475, 803 and 906 become\n\n\\texttt{for (int k=1; \\hl{v10<v10\\_max}; k++)\\{}\n\nAn equivalent change is to store $d$, the difference between \\texttt{v10\\_max} and \\texttt{v10}, before the loop and to iterate over $k=1,\\ldots,d$. The sizing routine must be changed together with the enumeration, and the two routines for the sample average treatment effect keep their stop at an acceptance. The sentence of Section 5.1.1 needs no change.", "evidence": "The evidence is a derivation and an exact counterexample, confirmed by compiling the original routines. Let $a$ be the value of \\texttt{v10} on entering the loop and $b$ the value of \\texttt{v10\\_max}, so that $d=b-a$. After $k-1$ steps, \\texttt{v10} equals $a+k-1$, and the length condition for step $k$ is\n\\[\nk<b-(a+k-1)+1=d-k+2 .\n\\]\nFor integers this is equivalent to $2k\\le d+1$, and hence to $k\\le\\lceil d/2\\rceil$.\n\nFor the counterexample, take the observed counts $(1,1,1,1)$, $\\tau_0=0$ and $j=2$. The line is $(2,0,0,2)$, $(1,1,1,1)$, $(0,2,2,0)$, and \\texttt{v10\\_max} is 2.\n\\begin{audittable}{lllll}{The loop as coded on the line $\\tau_0=0$, $j=2$ for the observed counts $(1,1,1,1)$.}\nstep $k$ & \\texttt{v10} before the step & bound \\texttt{v10\\_max-v10+1} & $k<{}$bound & vector reached \\\\\n1 & 0 & 3 & true & $(1,1,1,1)$ \\\\\n2 & 1 & 2 & false & none: $(0,2,2,0)$ is not tested \\\\\n\\end{audittable}\nOf the 15 count vectors that are possible given these data, the code visits 14. When compiled, the sizing routine returns 14 and the set routine returns 14 rows; both counts become 15 once the five bounds are replaced. The omitted vector $(0,2,2,0)$ can be the count vector of the true table: with two subjects of type $(1,0)$ and two of type $(0,1)$, the observed counts $(1,1,1,1)$ have probability $2/3$. Its risk ratio is 1 and its exact $p$-value is 1 for the risk-ratio test.\n\nFor the data set with $n=50$, $m=30$ and observed counts $(1,29,3,17)$, I find the following counts.\n\\begin{audittable}{rrr}{Count vectors with a given $\\tau_0$ for the observed counts $(1,29,3,17)$: those possible, and those reached by the loop as coded before any stop at an acceptance.}\n$n\\tau_0$ & possible count vectors & reached by the loop as coded \\\\\n$3$ & 73 & 46 \\\\\n$-16$ & 78 & 49 \\\\\n$2$ & 78 & 49 \\\\\n$-15$ & 83 & 52 \\\\\n\\end{audittable}\nNote that an omitted vector need not remove its value of the estimand. In the four-unit example the tested vector $(2,0,0,2)$ also has risk ratio 1 and $p$-value 1. For the four values of $\\tau_0$ in the second table, the largest exact $p$-value of the difference-in-means test is the same over the full and the truncated sets of vectors. I have not established a changed interval, coverage or printed number; the original simulation outputs are unavailable.", "problem": "Section 5.1.1 tests every possible additional vector $\\mathbf v_k$ on $\\mathcal L$, starting from the vector $\\mathbf v$ already tested in Step 2(c), updating the Monte Carlo draws from one vector to the next. The proof of Theorem 5.5 in Appendix C relies on the true count vector being among the vectors tested. In the code, each of the five loops runs while $k$ is less than \\texttt{v10\\_max-v10+1}, and the body of the loop increases \\texttt{v10} by one. The bound falls by one at every step while $k$ rises by one. Let $d$ be the difference between \\texttt{v10\\_max} and \\texttt{v10} on entering the loop, which is the number of steps needed to reach the end of the line. Without an earlier stop at acceptance, the loop makes only $\\lceil d/2\\rceil$ of them. The bound alone omits the last $\\lfloor d/2\\rfloor$ vectors, and for $d\\ge 2$ the end of the line is not reached. The bound does not involve $K$, and more Monte Carlo draws do not restore the omitted tests. The same bound appears in the routine that sizes the output (line 475), in the two routines for general estimands that serve the risk ratio and the odds ratio (lines 803 and 906), and in the two routines for the sample average treatment effect (lines 995 and 1204). The supplied replication workflows call the last two for the permutation columns of Table 3, and \\texttt{perm\\_test\\_SATE} for the ``Unbalanced'' columns of Tables 4--7. The full-search coverage argument therefore does not by itself establish coverage for this code, and the affected interval and timing comparisons need checking with corrected code.", "what_survives": "This code defect does not contradict the full-search procedure of Section 5.1.1, Algorithm 5.3, Theorem 5.5 or the $O(Kn^3)$ bound. The routines for the difference in means in balanced designs contain none of the five loops. Table 2, the ``Balanced'' columns of Tables 4--7 and the Wald columns are unaffected by these bounds. The supplied code for the permutation columns of Table 3 and the ``Unbalanced'' columns of Tables 4--7 makes fewer tests than Algorithm 5.3 requires whenever a line with $d\\ge 2$ must be searched to its end: a full search takes $d$ steps where the code takes $\\lceil d/2\\rceil$. The interval and timing comparisons must be checked again with corrected code."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E022", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of steps (b) to (e) of Algorithm 5.3 and of Remark 5.4. A red box marks, in step (c), the instruction 'If v is accepted, set I_{alpha,K} <- I_{alpha,K} union {theta(v)}'. Remark 5.4 below says that for the sample average treatment effect the second step may be terminated as soon as some value of tau_0 is deemed compatible.", "caption": "Algorithm 5.3, Steps 2(b)–2(e), and Remark 5.4, p. 12 (PDF p. 12). The marked instruction adds $\\theta(\\mathbf v)$ to the confidence set whenever $\\mathbf v$ is accepted, and nothing later removes it. The passage is correct; the released code departs from it.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E023.png"}, {"alt": "Listing of source code with line numbers. Red boxes mark 'int result = 0;' in line 959, the condition '(abs(result)<1e-10)' of the while loop in line 962, the words 'double result' at the start of line 980, and line 982, 'my_matrix(i,1)=result;'. Lines 1006 to 1011 show the later test, 'result = perm_test_inner(...)' without a declaration, followed by 'my_matrix(i,1) = result;'.", "caption": "Source code of the replication files, not a page of the article: permutation_test.cpp, lines 957–962, 978–982 and 1006–1011 (routine perm_test_SATE). The loop of line 962 runs while the flag of line 959 is zero. Line 980 declares a second variable of the same name, so the flag stays at zero and line 982 stores a decision that later tests overwrite; line 1009 assigns to the flag itself.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E023-code.png"}, {"alt": "Chart with two rows of dots over a horizontal axis of tested values tau_0 from minus 0.38 to 0.12 in steps of 0.02. Top row, Algorithm 5.3 with every accepted value kept: blue dots from minus 0.32 to 0.06, labelled width 0.38. Bottom row, the search as coded in the replication files: blue dots from minus 0.30 to 0.04, labelled width 0.34, and orange diamonds at minus 0.32 and at 0.06, which the legend describes as accepted at a possible table, then overwritten by a rejection. Small grey dots at the other values mark values accepted at no possible table.", "caption": "The interval for one data set ($n=50$, $m=30$, observed counts $(1,29,3,17)$, $\\alpha=0.05$), with exact $p$-values in place of Monte Carlo ones. Algorithm 5.3 returns $[-0.32,0.06]$. The search as coded returns $[-0.30,0.04]$: the values $-0.32$ and $0.06$ are accepted at a possible table, and a later rejection overwrites the acceptance. The data set has probability $342/2303$ in the paper’s second scenario with $n=50$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E023-interval.png"}], "formal_release": null, "formal_statements": [], "headline": "In the released routines for the sample average treatment effect, a variable declared inside a block hides the flag that should hold an acceptance; a later rejection can then erase an accepted table, and the code can exclude values that Algorithm 5.3 keeps.", "id": "P003-E023", "location": "Algorithm 5.3, Step 2(c), and Remark 5.4, p. 12 (PDF p. 12); replication files, permutation_test.cpp, lines 959–982 and 1167–1190; Table 3 and Tables 4–7, pp. 16–17", "pdf_page": 12, "printed_page": "12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~12, Algorithm 5.3, Step 2(c): ``Monte Carlo permutation test the count vector $\\mathbf v$ coming from the choice of $v_{10}$ in the previous step using Algorithm 5.1 with the given values of $\\alpha$, $K$, $\\theta$, and $R$. If $\\mathbf v$ is accepted, set $\\mathcal I_{\\alpha,K}\\leftarrow\\mathcal I_{\\alpha,K}\\cup\\{\\theta(\\mathbf v)\\}$ using the returned value of $\\theta(\\mathbf v)$.''\n\nReplication files, \\texttt{permutation\\_test.cpp}, routine \\texttt{perm\\_test\\_SATE}, lines 959, 962, 980 and 982:\n\n\\texttt{int result = 0; //result = 0 means it is currently being rejected}\n\n\\texttt{while( (j<n+1) \\&\\& (abs(result)<1e-10) )\\{ //as soon as we find a j that is not rejected, we stop}\n\n\\texttt{\\hl{double} result = perm\\_test\\_inner(q, K, test\\_statistics, alpha, epsilon,v11,v10,v01,v00,test\\_type,warning\\_msg);}\n\n\\texttt{my\\_matrix(i,1)=result;}\n\nRoutine \\texttt{perm\\_test\\_SATE\\_STD}, line 1188:\n\n\\texttt{\\hl{double} result = perm\\_test\\_inner\\_STD(q, K, test\\_statistics\\_num, test\\_statistics\\_den, alpha, epsilon,v11,v10,v01,v00,test\\_type,warning\\_msg);}", "corrected": "In \\texttt{perm\\_test\\_SATE} (line 980) and \\texttt{perm\\_test\\_SATE\\_STD} (line 1188), assign the decision to the flag that already exists:\n\n\\texttt{\\del{double} result = perm\\_test\\_inner(q, K, test\\_statistics, alpha, epsilon,v11,v10,v01,v00,test\\_type,warning\\_msg);}\n\n\\texttt{\\del{double} result = perm\\_test\\_inner\\_STD(q, K, test\\_statistics\\_num, test\\_statistics\\_den, alpha, epsilon,v11,v10,v01,v00,test\\_type,warning\\_msg);}\n\nAn accepted first table then stops the search for that $\\tau_0$, as Remark 5.4 allows, and cannot be overwritten. The outputs behind Table 3 and the ``Unbalanced'' columns of Tables 4--7 should be regenerated after this and the other corrections to the code, among them \\fid{P003-E022} and \\fid{P003-E024}. Algorithm 5.3 needs no change.", "evidence": "The evidence is a set of exact counterexamples. I computed the $p$-values in rational arithmetic and checked the compiled routines using all assignments in the small examples, with and without the two declarations.\n\nFirst, let $n=3$, $m=1$, $\\alpha=1/2$, $\\varepsilon=0$ and $\\tau_0=1/3$, with observed counts $(0,1,0,2)$. Exactly two possible tables have this $\\tau_0$, and neither line contains a further table.\n\\begin{audittable}{llllll}{The search as coded for $\\tau_0=1/3$. The value $1/3$ is accepted at $j=1$ and returned as rejected.}\n$j$ & table $\\mathbf v$ & exact $p$-value & test at $\\alpha=1/2$ & stored decision & flag after the test \\\\\n1 & $(0,1,0,2)$ & $1$ & accept & 1 & 0 \\\\\n2 & $(0,2,1,0)$ & $1/3$ & reject & 0 & 0 \\\\\n\\end{audittable}\nWith \\texttt{double} removed, the search stops at $j=1$ and returns acceptance, as Algorithm 5.3 does. For the population with one subject of type $(1,0)$ and two of type $(0,0)$, whose effect is $1/3$, the interval of the search as coded covers the effect with probability $1/3$ at nominal level $1/2$ when the tests are exact, and with probability below $0.334$ when $K=300$.\n\nSecond, let $n=50$, $m=30$, $\\alpha=0.05$ and $\\varepsilon=0$, with observed counts $(1,29,3,17)$, using exact $p$-values in place of Monte Carlo ones. At $\\tau_0=3/50$ the ten accepted possible tables $\\mathbf v=(4,j-4,j-7,57-2j)$, $j=10,\\ldots,19$, have exact $p$-values between $1962369/38116060\\approx 0.0515$ and $6295858813/102454809226\\approx 0.0615$, and the search as coded returns rejection. The same happens at $\\tau_0=-16/50$. The search as coded returns $[-15/50,2/50]$, of width $0.34$. Algorithm 5.3 returns $[-16/50,3/50]$, of width $0.38$, and so does the code with only the two declarations changed. This data set has probability $342/2303$ in the paper's second scenario with $n=50$ (four subjects of type $(1,1)$ and 46 of type $(0,0)$).\n\nThird, for the studentized routine with $\\varepsilon=0$ and $\\alpha=0.05$, take nine subjects whose potential outcomes are all zero, and $m=5$. The returned interval covers the effect $0$ with probability at most $50/63\\approx 0.794$ for every positive $K$ divisible by 20. Note that this bound also depends on how the code treats a zero variance. It is not an effect of the hidden flag alone.\n\nNo entry of Tables 3--7 has been recomputed. The replication files contain no simulation outputs.", "problem": "Algorithm 5.3 adds $\\theta(\\mathbf v)$ to the confidence set whenever a possible table $\\mathbf v$ is accepted, and no later step removes it. Remark 5.4 allows the search over the tables with a given $\\tau_0$ to stop at the first acceptance. The routines \\texttt{perm\\_test\\_SATE} and \\texttt{perm\\_test\\_SATE\\_STD} implement this search with an integer flag \\texttt{result}, set to 0 before the loop, and a loop that runs while the flag is 0. At the first table of each line of tables, the outcome of the test is assigned to a new variable, also named \\texttt{result}, that exists only inside the enclosing block (line 980, and line 1188 in the studentized routine). After an acceptance the flag is still 0. The search continues, and each later test overwrites the stored decision. A value $\\tau_0$ that has an accepted possible table can therefore be returned as rejected if a later rejection overwrites the acceptance and no subsequent acceptance survives. For given decisions of the individual tests, the set returned by the code is contained in the set of Algorithm 5.3, and it can be strictly smaller. The replication files use these routines for the ``Permutation (DiM)'' and ``Permutation (Std)'' columns of Table 3, and the first of them for the ``Unbalanced'' columns of Tables 4--7. Theorem 5.5 describes a procedure in which an accepted value is never removed. Its coverage statement does not transfer to these routines as released.", "what_survives": "The set-union instruction in Algorithm 5.3 and the stopping rule in Remark 5.4 need no change; this defect does not refute Theorem 5.5. The balanced routines, Tables 1 and 2, the ``Balanced'' columns of Tables 4--7, the routines for general estimands, the Wald comparisons and the missing-data construction are not affected by the two declarations. What changes is the standing of the two permutation columns of Table 3 and of the ``Unbalanced'' columns of Tables 4--7: their intervals can be narrower than those of the method, and their widths and run times must be regenerated with corrected code. Which entries change, and by how much, is not established."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E023", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the end of Algorithm 5.1 on page 10. Equation (5.3) sets S equal to 1 over (K + 1) times the quantity 1 plus the sum of the V i. The next sentence reads: Return the decision accept if S > alpha, and return the decision reject otherwise. Red outlines surround the right-hand side of (5.3) and the condition S > alpha.", "caption": "Algorithm 5.1, p. 10 (PDF p. 10). The outlines mark the definition (5.3) of $S$ and the acceptance rule $S>\\alpha$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E024.png"}, {"alt": "Listing of lines 624 to 661 of the C++ file permutation_test.cpp, with line numbers. Line 652 reads: p_value = count/n_mc. Line 655 reads: if ( (p_value + epsilon )< alpha), followed by return 0 with the comment reject null hypothesis. Red outlines surround the statement on line 652 and the condition on line 655.", "caption": "Function perm_test_inner of permutation_test.cpp, lines 624–661 (a listing of the released code, not a page of the article). The outlines mark the fraction on line 652 and the rejection rule on line 655.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E024-code.png"}], "formal_release": null, "formal_statements": [], "headline": "For the tests of Section 5, the released code uses the fraction $c/K$ of tail draws and accepts when $c/K+\\varepsilon\\ge\\alpha$, not the rule $(1+c)/(K+1)>\\alpha$ of Algorithm 5.1 on which validity for every $K$ rests; the two rules agree at every reported Section 5 simulation setting.", "id": "P003-E024", "location": "Algorithm 5.1, equation (5.3) and the acceptance rule, p. 10 (PDF p. 10); Remark 5.6, p. 13; released code permutation_test.cpp, lines 652–655 and 728–731", "pdf_page": 10, "printed_page": "10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Algorithm 5.1, p.~10.}\n\\[\nS=\\frac{1}{K+1}\\Bigl(1+\\sum_{i=1}^{K}V_i\\Bigr).\n\\]\n\\ldots\\ Return the decision accept if $S>\\alpha$, and return the decision reject otherwise.\n\n\\emph{Remark 5.6, p.~13.} We note that Theorem 5.5 holds for all values of $K$, so that the validity of the confidence interval procedure does not depend on $K$.\n\n\\emph{Released code, \\texttt{permutation\\_test.cpp}, lines 652 and 655 (the same statements are on lines 728 and 731).}\n\n\\texttt{p\\_value = \\hl{count/n\\_mc};}\n\n\\texttt{if ( \\hl{(p\\_value + epsilon )< alpha})\\{ return 0; \\}}", "corrected": "Algorithm 5.1 stands as printed. In the released code, the tests of Section 5 compute\n\n\\texttt{p\\_value = \\hl{(1 + count)/(n\\_mc + 1)};}\n\nand reject exactly when \\texttt{\\hl{p\\_value <= alpha}}, with no tolerance \\texttt{epsilon}. The tests of Section 4 keep the fraction \\texttt{count/n\\_mc} and the rule of Algorithm 4.4 on a separate path, since \\texttt{perm\\_test\\_inner} is called by both.", "evidence": "I checked an exact counterexample and its comparison with the unchanged functions compiled from the released source. Let three units have potential outcomes $(1,0)$, $(0,0)$ and $(0,0)$, and let one of them be treated, so that $\\mathbf v=(0,1,0,2)$ and $\\tau=1/3$. Let $K=1$, $\\alpha=1/20$ and $\\varepsilon=0$, with the difference-in-means statistic. If the $(1,0)$ unit is treated, which has probability $1/3$, then $|T-\\tau|=2/3$. A simulated assignment that treats another unit, which has probability $2/3$, has $|T-\\tau|=1/3$. Then $c=0$ and the code rejects. In every other case $c=1$ and the code accepts. The code therefore rejects the true table with probability $2/9>1/20$, while Algorithm 5.1 has $S\\ge1/2$ and never rejects. In a second example, for the studentized function with $n=5$, $m=2$ and $\\mathbf v=(1,0,0,4)$, the rejection probability is $6/25$.\n\nFor the agreement at the reported settings, let $\\varepsilon=0$. The code accepts when $c\\ge\\lceil K\\alpha\\rceil$. Algorithm 5.1 accepts when $c>\\alpha(K+1)-1$, that is, when $c\\ge\\lfloor\\alpha(K+1)\\rfloor$. If $K\\alpha=r$ is an integer and $0<\\alpha<1$, then both thresholds equal $r$.\n\nNote that the example concerns a single test. It does not show undercoverage of the interval that the code returns, because another possible table with the same value of the estimand can be accepted.", "problem": "Remark 5.2 states that $S$ of (5.3) is a valid $p$-value for every $K$, and the proof of Theorem 5.5 applies this to the true table. The released functions \\texttt{perm\\_test\\_inner} and \\texttt{perm\\_test\\_inner\\_STD}, which the code labels as Algorithm 5.1, compute $c/K$, where $c=\\sum_{i=1}^KV_i$, and reject when $c/K+\\varepsilon<\\alpha$. This test is not valid for every $K$: in the example below it rejects a true table with probability $2/9$ at $\\alpha=1/20$. The guarantee of Theorem 5.5 for every $K$ does not carry over to the released code.\n\nAt the settings of the Section 5 simulations, $\\alpha=0.05$ and $\\varepsilon=0$, the two rules accept for the same values of $c$ whenever $K$ is a multiple of $20$. This covers every reported Section 5 value of $K$, namely $1000$, $2000$, $5000$, $10000$ and $20000$, and the difference alone changes no reported result.", "what_survives": "Algorithm 5.1, Remark 5.2 and Theorem 5.5 are correct as printed. For Table 3, with $K=20000$, and for Tables 4--7, with $K$ from $1000$ to $20000$, changing this rule alone leaves every decision unchanged. What changes is the released code, which for general $K$ is not the procedure that Theorem 5.5 covers."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E024", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the bullet 'For inference on tau, we could also use the studentized test statistic' with equations (5.8) to (5.11). Red boxes mark the two denominators of equation (5.9): (n11 + n10) squared under Q1 and (n01 + n00) squared under Q0.", "caption": "Equations (5.8)–(5.11), p. 11 (PDF p. 11). The marked denominators of (5.9) divide the sums of squares $Q_1$ and $Q_0$ by the squared arm sizes; the released code uses them for the simulated assignments only.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E025.png"}, {"alt": "Listing of source code with line numbers. In line 78 of 2_permutation_test.R, denominator = sqrt(var(data[data$treatment==1,'outcome'])/nt + var(data[data$treatment==0,'outcome'])/(n-nt)), red boxes mark the two terms that call var(). In line 171 of permutation_test.cpp, double sigma = sqrt(Q1/((n11+n10)*(n11+n10)) + Q0/((n01+n00)*(n01+n00))), red boxes mark the two quotients. Lines 76 and 168 to 170 compute the difference in means, T, Q1 and Q0.", "caption": "Source code of the replication files, not a page of the article: 2_permutation_test.R, lines 76–78, and permutation_test.cpp, lines 168–171. The marked terms of line 78 use R’s var(), which divides by the arm size minus one; the marked terms of line 171 divide $Q_1$ and $Q_0$ by the squared arm sizes, as (5.9) does.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E025-code.png"}], "formal_release": null, "formal_statements": [], "headline": "In the released code, the studentized test divides the observed statistic by sample variances (divisors $m-1$ and $n-m-1$) and the simulated statistics by the variance (5.9) (divisors $m$ and $n-m$); one statistic should be used for both.", "id": "P003-E025", "location": "Equations (5.8)–(5.11), p. 11 (PDF p. 11); Table 3, p. 16, columns “Permutation (Std)”; replication files, 2_permutation_test.R, line 78, and permutation_test.cpp, line 171", "pdf_page": 11, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~11: ``For inference on $\\tau$, we could also use the studentized test statistic\n\\[\nR_{\\mathrm{std}}(\\mathbf n,\\mathbf v)=\\frac{T(\\mathbf n)-\\tau(\\mathbf v)}{\\sigma(\\mathbf n)},\\qquad\\text{(5.8)}\n\\]\nwhere\n\\[\n\\sigma^2(\\mathbf n)=\\frac{Q_1}{(n_{11}+n_{10})^2}+\\frac{Q_0}{(n_{01}+n_{00})^2},\\text{''}\\qquad\\text{(5.9)}\n\\]\nwith $Q_1$ and $Q_0$ as in (5.10) and (5.11).\n\nReplication files, \\texttt{2\\_permutation\\_test.R}, line 78 (the observed statistic):\n\n\\texttt{denominator = sqrt(\\hl{var}(data[data\\$treatment==1,'outcome'])/nt + \\hl{var}(data[data\\$treatment==0,'outcome'])/(n-nt)) \\#estimated std}\n\nReplication files, \\texttt{permutation\\_test.cpp}, line 171 (the statistic of each simulated assignment):\n\n\\texttt{double sigma = sqrt(Q1/((n11+n10)*(n11+n10)) + Q0/((n01+n00)*(n01+n00)));}", "corrected": "Compute the observed statistic with the denominator of (5.9), as the simulated statistics are computed:\n\\[\n\\sigma(\\mathbf n)=\\sqrt{\\frac{Q_1}{\\hl{m^2}}+\\frac{Q_0}{\\hl{(n-m)^2}}},\\qquad m=n_{11}+n_{10},\n\\]\nwith $Q_1$ and $Q_0$ as in (5.10) and (5.11), in place of the two calls to \\texttt{var()} in line 78 of \\texttt{2\\_permutation\\_test.R} (and in line 210, which repeats it). Apply one rule to both sides when a standard error is zero or an arm has a single subject. Then recompute the ``Permutation (Std)'' columns of Table 3. The other choice is to use the sample variances on both sides and to change (5.9) to match.", "evidence": "The evidence is an exact counterexample, reproduced with the original R line and the original C++ routine. For the counts $\\mathbf n=(1,2,1,1)$ and $\\tau=0$, I have $T=-1/6$, $Q_1=2/3$, $Q_0=1/2$, $V_P=43/216$ and $V_O=13/36$. The squared statistic is then $6/43$ by (5.8) and $1/13$ in the observed path of the code.\n\nNext, I consider five subjects, one with potential outcomes $(1,1)$ and four with $(0,0)$, and $m=2$. Both variances are positive under each of the 10 assignments. The squared statistic of the observed path equals $1$ under every assignment. The squared statistic of a simulated assignment equals $2$ with probability $2/5$ and $3/2$ with probability $3/5$. The two are not identically distributed. If the subject of type $(1,1)$ and one other subject are treated, the exact $p$-value of the true table is $2/5$ with (5.8) on both sides and $1$ with the comparison as coded. At $\\alpha=1/2$ the first rejects and the second accepts.\n\nThe direction is the same in general. If $m,l>1$ and $V_P>0$, then $V_O>V_P$, since $m/(m-1)>1$ and $l/(l-1)>1$. The observed statistic of the code is then no larger in absolute value than (5.8). Each tail indicator can only change from 0 to 1, and the $p$-value can only rise. In the design of Table 3 with arms of 30 and 20, I have $30/29\\le V_O/V_P\\le 20/19$.", "problem": "Algorithm 5.1 compares $|R(\\mathbf n^{(i)},\\mathbf v)|$ with $|R(\\mathbf n,\\mathbf v)|$ for one statistic $R$, and Remark 5.2 uses that the $K+1$ values in (5.5) are identically distributed. The replication files compute the two sides differently. I write $m=n_{11}+n_{10}$ and $l=n_{01}+n_{00}$. The observed statistic is formed in R with \\texttt{var()}, which divides each sum of squares by the arm size minus one. Its squared denominator is $V_O=Q_1/(m(m-1))+Q_0/(l(l-1))$. The statistic of each simulated assignment is formed in C++ with (5.9), and its squared denominator is $V_P=Q_1/m^2+Q_0/l^2$. These are two different functions of the data, and the released routine for the ``Permutation (Std)'' columns of Table 3 uses a comparison other than the one printed.", "what_survives": "This discrepancy does not refute equations (5.8)--(5.11), Algorithm 5.1, Remark 5.2 or Theorem 5.5. The routines for the difference in means, the ``Permutation (DiM)'' and Wald columns of Table 3 and the balanced algorithm do not use these denominators. Where both variances are positive and both arms have more than one subject, the mismatch can only raise $p$-values, and it is not evidence of undercoverage. Whether a printed entry of the ``Permutation (Std)'' columns changes has not been computed. The case of zero variance is the subject of \\fid{P003-E011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E025", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 4.6 on page 9, from its first line through equation (4.10). Equation (4.10) states that the probability that tau(y) lies in the interval with subscripts alpha minus epsilon, epsilon, K is at least 1 minus alpha. A red outline surrounds the subscripts alpha minus epsilon, epsilon, K.", "caption": "Proposition 4.6, part 1, p. 9 (PDF p. 9). The outline marks the subscript of (4.10): the algorithm is run at level $\\alpha-\\varepsilon$ to obtain coverage of at least $1-\\alpha$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E026.png"}, {"alt": "Crop of the caption of Table 2 on page 15. The caption says that the second column reports the coverage probability of the 95 per cent confidence intervals based on the permutation test, and that the number of randomizations is chosen according to the formula (4.9) with epsilon = 0.005. Red outlines surround 95 per cent and epsilon = 0.005.", "caption": "Caption of Table 2, p. 15 (PDF p. 15). The outlines mark the stated confidence level and the value of $\\epsilon$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E026-caption.png"}, {"alt": "Listing of lines 26 to 53 of the R script 2_sim_raw.R, with line numbers. Line 37 reads: result_test_case = permutation_test(data, test_type, alpha=0.05, epsilon=0.005, nperm=0, return_interval = TRUE, and further arguments. A red outline surrounds alpha=0.05, epsilon=0.005.", "caption": "Script 2_sim_raw.R, lines 26–53 (a listing of the released code, not a page of the article). The outline marks the level and the tolerance passed to the balanced routine on line 37.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E026-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The 95% in the caption of Table 2 is nominal: for Algorithm 4.5 as defined, level $0.05$ with $\\varepsilon=0.005$ gives the Proposition 4.6 coverage bound $0.945$, while the $0.95$ bound needs level $0.045$; these bounds do not certify the released implementation (P003-E021).", "id": "P003-E026", "location": "Proposition 4.6, part 1, equation (4.10), p. 9 (PDF p. 9); caption of Table 2, p. 15 (PDF p. 15); released code 2_sim_raw.R, line 37", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Proposition 4.6, part 1, equation (4.10), p.~9.}\n\\[\n\\mathbb P\\bigl(\\tau(\\mathbf y)\\in\\mathcal I_{\\alpha-\\varepsilon,\\varepsilon,K}(\\mathbf Y,\\mathbf Z)\\bigr)\\ge1-\\alpha.\n\\]\n\n\\emph{Caption of Table 2, p.~15.} The second column (Permutation Cov.) reports the coverage probability of the 95\\% confidence intervals (CIs) based on the permutation test with the difference-in-means statistic. \\ldots\\ The number of simulations is $10^4$, and the number of randomizations is chosen according to the formula (4.9) with $\\epsilon=0.005$.\n\n\\emph{Released code, \\texttt{2\\_sim\\_raw.R}, line 37.}\n\n\\texttt{permutation\\_test(data, test\\_type , alpha=0.05, epsilon=0.005, nperm=0,\\ldots)}", "corrected": "\\emph{Caption of Table 2.} The second column (Permutation Cov.) reports the coverage probability of the \\hl{nominal} 95\\% confidence intervals (CIs) based on the permutation test with the difference-in-means statistic. \\ldots\\ The number of simulations is $10^4$, and the number of randomizations is chosen according to the formula (4.9) with $\\epsilon=0.005$\\hl{; the released simulation scripts pass level $0.05$. For Algorithm 4.5 as defined, these parameters give the Proposition 4.6 coverage lower bound $94.5\\%$. This bound does not certify the released implementation because of the defect described in} \\fid{P003-E021}.", "intended_meaning": "I read the caption's 95\\% as the nominal confidence level of the intervals. In (4.10), the level passed to Algorithm 4.5 is $\\alpha-\\varepsilon$, and $1-\\alpha$ is the guaranteed lower bound on coverage. For the $0.95$ guarantee with $\\varepsilon=0.005$, the algorithm is run at level $0.045$, and Algorithm 4.4 then accepts when $S\\ge0.04$. The released scripts for Table 2 and for the balanced columns of Tables 4--7 pass the level $0.05$ unchanged to the test, which then accepts when $S\\ge0.045$. For Algorithm 4.5 as defined, these parameters give the Proposition 4.6 lower bound $1-(0.05+0.005)=0.945$. A comment in the released wrapper says the same: ``The implied confidence level has 1-alpha-epsilon.'' This is a lower bound for the intended algorithm, not the coverage attained or a guarantee for the released implementation: the tail-comparison defect of \\fid{P003-E021} prevents that transfer. Applying the proposition requires a faithful implementation, including the repaired search of \\fid{P003-E001} and the tie-inclusive tail comparison. The coverages reported in Table 2 are $0.96$ to $1.00$, and no undercoverage is shown. The replication files contain no simulation outputs, and I could not trace the published entries to a run of these scripts.\n\nThe two levels can return different intervals. Let $n=16$ and $m=8$, with observed counts $(0,8,0,8)$. The largest exact $p$-value over possible tables at $\\tau_0=\\pm5/16$ is $29/715\\approx0.0406$, which lies between $0.04$ and $0.045$. Let $K=411840$, which satisfies (4.9). Then $S$ equals the exact $p$-value in every test with positive probability, and on this event level $0.05$ returns $[-1/4,1/4]$ and level $0.045$ returns $[-5/16,5/16]$. To obtain the $95\\%$ lower bound of Proposition 4.6 after implementing the repaired search and the tail comparison of \\fid{P003-E001} and \\fid{P003-E021}, line 37 of \\texttt{2\\_sim\\_raw.R} would pass \\texttt{alpha=0.045}, with $K$ still chosen by (4.9). Changing the level alone does not establish this guarantee for the released code. The balanced coverage, length and running-time results of Tables 2 and 4--7 would then have to be recomputed."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E026", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the second-scenario block of Table 2, with columns Sample size, Permutation Cov., Permutation Med. Length, Wald Cov. and Wald Med. Length. A red box marks the last row, sample size 1000, with entries 0.99, 0.08, 0.96, 0.07. The rows above it are for sample sizes 50, 100 and 200.", "caption": "Table 2, second scenario, p. 15. The marked row, for sample size 1000, is the one computed by the block of the script that contains the undefined name.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E027.png"}], "formal_release": null, "formal_statements": [], "headline": "The script that assembles Table 2 stops at an undefined variable name, files_s12_1000 for files_s2_1000, and the script for Table 3 stops when two kinds of output file share its input folders.", "id": "P003-E027", "location": "Replication script 5_create_table_balanced.R, line 115, which builds the second-scenario row for sample size 1000 of Table 2, p. 15 (PDF p. 15); 5_create_table_unbalanced.R, lines 11–14, for Table 3, p. 16", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Replication code (not printed in the article), \\texttt{5\\_create\\_table\\_balanced.R}, lines 113 and 115:\n\n\\texttt{files\\_s2\\_1000=list.files()}\n\n\\texttt{for (i in 1:length(\\hl{files\\_s12\\_1000}))\\{}\n\n\\texttt{5\\_create\\_table\\_unbalanced.R}, lines 11 and 14:\n\n\\texttt{files\\_s1\\_50=\\hl{list.files()}}\n\n\\texttt{output\\_50\\_s1=rbind(output\\_50\\_s1, read.csv(files\\_s1\\_50[i])[,2:15])}", "corrected": "\\texttt{5\\_create\\_table\\_balanced.R}, line 115:\n\n\\texttt{for (i in 1:length(\\hl{files\\_s2\\_1000}))\\{}\n\nIn \\texttt{5\\_create\\_table\\_unbalanced.R}, the blocks for the coverage and length entries of Table 3 should read only the output files of the unbalanced-trial simulations. \\hl{Either these files are kept in folders separate from the output of the comparison on balanced designs, or the intended files are selected by design and number of draws and their columns are checked before they are extracted.} These repairs remove the undefined name in the Table~2 script and the column-selection failure in the Table~3 blocks. They do not establish whether any published table value is right or wrong.", "evidence": "I base these corrections on an exact counterexample for each failure. The scripts were executed on small input files constructed with the correct shapes, and only the folder paths were changed. The Table 2 script stops at line 115 with the message above, and it completes all 216 lines once the name is corrected. The Table 3 blocks stop at line 14 when a 15-column file and a 4-column file are in the same folder, and they complete when the four-column files are moved or filtered out. No simulation was run, and no published number was recomputed.", "problem": "The script \\texttt{5\\_create\\_table\\_balanced.R} clears the workspace on its first line, defines \\texttt{files\\_s2\\_1000} on line 113, and then uses \\texttt{files\\_s12\\_1000}, which is defined nowhere, as the bound of the loop on line 115. R stops with the message \\texttt{object 'files\\_s12\\_1000' not found}. The script cannot form the second-scenario row for $n=1000$ of Table 2 or the complete table for the second scenario, and its later timing code is not reached. The second failure is in the first blocks of the script for Table 3, which compute the coverage and length entries of that table. These blocks read every file in their folders and take columns 2 to 15. The simulation scripts for the unbalanced trials and those for the comparison of the two algorithms on balanced designs write to the same folders, and the files of the second kind have four columns. When both kinds are present, R stops with the message \\texttt{undefined columns selected}.", "what_survives": "Both defects are in the scripts that assemble the tables, and both have local repairs. No published value is shown to be wrong, and the algorithms and proofs are not involved. I cannot trace the entries of Tables 2--7 to the original simulation outputs, which are absent from the replication files. The Table 3 failure occurs only when both kinds of output are present in the shared folders. The same blocks also lack a selection by number of draws and design, which is the subject of \\fid{P003-N014}."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E027", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the caption of Table 4: Average computation times (in seconds) for the balanced algorithm and unbalanced algorithm with the difference-in-means statistic and sample sizes 50, 100, 200, 500, and 1000 in Simulation Scenario 1. The numbers of treated units are 25, 50, 100, 250, and 500, respectively. The number of simulations is 10 to the power 4 for sample sizes 50, 100, and 200, and 10 to the power 3 for sample sizes 500 and 1000. The number of randomizations for the balanced algorithm is chosen according to the formula (4.9) with epsilon = 0.005. All numbers are rounded to two digits after the decimal point. Two red outlines surround the words: 10 to the power 3 for sample sizes 500, at the end of one line, and: and 1000, at the start of the next.", "caption": "Table 4, caption, p. 16. The outlines mark the number of simulations stated for sample sizes 500 and 1000 (the phrase breaks across two lines). The captions of Tables 5, 6 and 7 contain the same sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E028.png"}], "formal_release": null, "formal_statements": [], "headline": "For sample sizes 500 and 1000, the submission scripts in the replication files configure 100 simulated experiments per submission, against the $10^3$ stated in the captions of Tables 4–7 and the $10^4$ stated in the caption of Table 2; the number of experiments run for the paper is not known.", "id": "P003-E028", "location": "Table 4, caption, p. 16 (PDF p. 16), and the same sentence in the captions of Tables 5–7, pp. 16–17; Table 2, caption, p. 15; replication code, simulation and submission scripts for sample sizes 500 and 1000", "pdf_page": 16, "printed_page": "16", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Tables~4, 5, 6 and~7, captions: ``The number of simulations is $10^4$ for sample sizes 50, 100, and 200, and $10^3$ for sample sizes 500 and 1000.''\n\nTable~2, caption: ``The number of simulations is $10^4$, and the number of randomizations is chosen according to the formula (4.9) with $\\epsilon = 0.005$.''\n\nReplication code (not printed in the article), \\texttt{1\\_simulation\\_balanced\\_case\\_1\\_n1000.R}, line 32, and \\texttt{submit\\_job\\_balanced\\_1000\\_s1.sh}, line 10:\n\n\\texttt{\\hl{nsim=1}}\n\n\\texttt{\\#SBATCH -}\\texttt{-array=\\hl{1-100}}", "corrected": "The replication files should give, for each published scenario and each $K$, submission settings in which the number of distinct task indices times the number of experiments per task equals the count in the caption, with the output of every task kept. For a cell of Tables~4--7 at $n = 500$ or $n = 1000$:\n\n\\texttt{nsim=1} and \\texttt{\\#SBATCH -}\\texttt{-array=\\hl{1-1000}}\n\nFor a row of Table~2 at $n = 1000$:\n\n\\texttt{\\hl{nsim=10}} and \\texttt{\\#SBATCH -}\\texttt{-array=\\hl{1-1000}}\n\nThe scripts for Table~2 and the tables of run times and widths read the same output folders, and the settings should also say which runs enter which table. The counts in the captions should change only if the original outputs or job records show different numbers.", "evidence": "I checked the counts configured in the replication files. Each of the six submission scripts for the balanced algorithm at $n = 500$ and $n = 1000$ requests tasks 1 to 100 and calls a script with \\texttt{nsim=1}. The outer simulation functions were evaluated unchanged, with the inner computation replaced by a counter. They return one row per task for each of the twelve scripts for the large samples, and ten rows per task and sample size for the scripts for the small samples. The counts are $100 \\times 1 = 100$ for a large-sample cell and $1000 \\times 10 = 10{,}000$ for a small-sample cell. No simulation was run.", "problem": "For sample sizes 500 and 1000, the simulation scripts for these tables specify one simulated experiment per task, and each submission script for the balanced algorithm requests an array of 100 tasks. One submission configures 100 experiments for a given scenario and sample size. This is one tenth of the $10^3$ that the captions of Tables~4--7 state for these sample sizes. It is one hundredth of the $10^4$ that the caption of Table~2 states, and the replication code assembles the rows of Table~2 for $n = 1000$ from the output of the same simulation scripts. The two submission scripts for the unbalanced algorithm at these sample sizes also request 100 tasks. Their active lines run a third scenario that the paper does not report, and the lines for the two published scenarios are commented out. Submitting an array a second time adds no experiments, because the task index sets both the random seed and the name of the output file. For sample sizes 50, 100 and 200 the task counts agree with the captions: 1000 tasks with 10 experiments each give $10^4$ for the scenario that is selected. For the large samples, the replication files neither reproduce nor document the stated counts. They do not show how many experiments were run for the paper. The files say that paths and indices may need to be changed, and they include no outputs and no job records.", "what_survives": "No table entry is shown to be wrong, and the captions are not shown to be false. What is missing is a configuration, or a record of the runs, that supports the stated counts for $n = 500$ and $n = 1000$ in Tables~4--7 and for $n = 1000$ in Table~2. The caption of Table~2 also omits that sample size from its list, which is the subject of \\fid{P003-N012}. The algorithms and the theoretical results do not depend on these settings."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E028", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of the second paragraph of Section 7.2. Two red boxes mark the sentence 'Second, in Tables 4 and 5, for n = 50, 100, and 200, the unbalanced algorithm is practically faster than the balanced algorithm.', which runs over a line break. The next sentence begins 'However, when n = 500 or 1000 and K at least 5000, the balanced algorithm typically outperforms the unbalanced one.'", "caption": "Section 7.2, second paragraph, p. 15. The marked sentence compares the run times of the two algorithms for $n=50$, $100$ and $200$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E030.png"}, {"alt": "Crop of Table 4 (first scenario) and Table 5 (second scenario, with its caption), each with columns Sample size, Balanced, and Unbalanced for K = 1000, 2000, 5000, 10,000 and 20,000, and rows for sample sizes 50, 100, 200, 500 and 1000. Red boxes mark four entries in the rows for sample size 200: in Table 4, Balanced 249.15 and K = 20,000 497.06; in Table 5, Balanced 215.11 and K = 20,000 258.15.", "caption": "Tables 4 and 5, p. 16, average computation times in seconds. In the rows for sample size 200, the marked entries are the time of the balanced algorithm and the time of the unbalanced algorithm at $K=20{,}000$, which is the larger of the two in both tables.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-E030-tables.png"}], "formal_release": null, "formal_statements": [], "headline": "The statement that the unbalanced algorithm is faster for $n=50$, $100$ and $200$ needs an exception: at $n=200$ and $K=20{,}000$ the balanced algorithm is faster in both Table 4 and Table 5.", "id": "P003-E030", "location": "Section 7.2, second paragraph, sentence beginning “Second, in Tables 4 and 5”, p. 15 (PDF p. 15); Tables 4 and 5, rows for sample size 200, p. 16", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Second, in Tables 4 and 5, for $n = 50$, $100$, and $\\hl{200}$, the unbalanced algorithm is practically faster than the balanced algorithm.", "corrected": "Second, in Tables 4 and 5, for $n = 50$, $100$, and $200$, the unbalanced algorithm is practically faster than the balanced algorithm\\hl{, except at $n=200$ and $K=20{,}000$, where the balanced algorithm is faster in both scenarios}.", "evidence": "I checked the comparisons by recalculating from the printed tables. Among the 30 comparisons for $n=50$, $100$ and $200$ (three sample sizes, five values of $K$, two scenarios), the unbalanced algorithm is faster in 28. The two exceptions are at $n=200$ and $K=20{,}000$, with differences of 247.91 s in Table 4 and 43.04 s in Table 5, which rounding to two decimal places cannot reverse.\n\n\\begin{audittable}{lrrr}{Average computation times in seconds at $n=200$, from Tables 4 and 5.}\nTable & Balanced & Unbalanced, $K=10{,}000$ & Unbalanced, $K=20{,}000$ \\\\\nTable 4 (first scenario) & 249.15 & 226.55 & 497.06 \\\\\nTable 5 (second scenario) & 215.11 & 125.94 & 258.15\n\\end{audittable}", "problem": "Tables 4 and 5 report average run times for the balanced algorithm and for the unbalanced algorithm with $K\\in\\{1000,2000,5000,10000,20000\\}$. At $n=200$ and $K=20{,}000$ the unbalanced algorithm is slower than the balanced algorithm in both tables. The sentence states the comparison for $n=200$ without this exception.", "what_survives": "The entries of Tables 4 and 5 are unchanged. The unbalanced algorithm is faster in every listed comparison at $n=50$ and $n=100$, and at $n=200$ for the listed values $K\\le 10{,}000$. The recommendation to change algorithms at a sample size in the low hundreds is compatible with the tables; at $n=200$ the faster algorithm depends on $K$."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-E030", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph on general designs in Section 1.3 on page 4. The marked sentence reads: Unlike the O(n squared) procedure in Li and Ding (2016), our procedure provably returns the intervals in Rigdon and Hudgens (2015), up to Monte Carlo error (which may be made arbitrarily small), and extends to other causal estimands. Red outlines surround the words provably returns and the words from the intervals in Rigdon and Hudgens to arbitrarily small.", "caption": "Section 1.3, p. 4 (PDF p. 4), paragraph on general designs. The outlines mark the claim that the procedure returns the Rigdon–Hudgens intervals up to Monte Carlo error.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "The general procedure returns the Rigdon–Hudgens interval up to small Monte Carlo error only when no possible table has a $p$-value equal to $\\alpha$, the exception that footnote 12 already describes.", "id": "P003-N001", "location": "Section 1.3, p. 4 (PDF p. 4), paragraph beginning “For a general unbalanced trial”; abstract, p. 1", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Section 1.3, p.~4.} Unlike the $O(n^2)$ procedure in Li and Ding (2016), our procedure \\emph{provably} \\hl{returns the intervals in Rigdon and Hudgens (2015), up to Monte Carlo error (which may be made arbitrarily small)}, and extends to other causal estimands.\n\n\\emph{Abstract, p.~1.} We show that these confidence intervals can be found in $O(n\\log n)$ permutation tests in the case of balanced designs, where the treatment and control groups have equal sizes, and \\hl{$O(n^2)$ permutation tests} in the general case.", "corrected": "\\emph{Section 1.3.} Unlike the $O(n^2)$ procedure in Li and Ding (2016), our procedure \\emph{provably} \\hl{returns valid Monte Carlo confidence sets for every $K$, whose interval hulls converge in probability to the intervals in Rigdon and Hudgens (2015) as $K$ increases when no possible table has a $p$-value equal to $\\alpha$ (ties at the threshold need the conservative handling discussed in footnote 12)}, and extends to other causal estimands.\n\n\\emph{Abstract.} \\ldots\\ and \\hl{the equivalent of $O(n^2)$ Monte Carlo permutation tests} in the general case.", "intended_meaning": "The sentence of Section 1.3 is meant with the exception of footnote 12 (p.~10): when a $p$-value equals $\\alpha$, its Monte Carlo approximations fluctuate above and below $\\alpha$ however large $K$ is. With that exception the claim is correct. For fixed data, if no possible table has a $p$-value equal to $\\alpha$, then the hull of the set returned by Algorithm 5.3 equals $\\mathcal I_\\alpha$ with probability tending to one as $K$ grows. At a tie it does not. Let $n=10$ and $m=3$, with observed counts $(n_{11},n_{10},n_{01},n_{00})=(1,2,2,5)$, and let $\\alpha=1/20$. By exact enumeration, I have $\\mathcal I_\\alpha=[-2/5,3/5]$, and the upper endpoint is attained only by the count vector $(2,6,0,2)$, whose $p$-value is $6/120=\\alpha$. For $K=20r$, the rule $S>\\alpha$ of Algorithm 5.1 rejects this table with probability $\\sum_{x=0}^{r-1}\\binom{20r}{x}19^{20r-x}/20^{20r}$, which is $0.36$ at $K=20$ and $0.48$ at $K=1000$ and tends to $1/2$. The upper endpoint of the hull is then at most $1/2$. In the abstract, the tests of the general case are Monte Carlo tests, as Section 1.3 says. Theorem 5.5, on coverage for every $K$ and on the $O(Kn^3)$ running time, does not depend on this point."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N001", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the foot of page 3 showing footnotes 3 and 4 and the page number. Footnote 4 reads: See Lemma 3.1 in Section 3, below. A red outline surrounds the words Lemma 3.1.", "caption": "Footnote 4, p. 3 (PDF p. 3). The outline marks the cross-reference.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "Footnote 4 should refer to Proposition 3.2, the coverage result, and not to Lemma 3.1.", "id": "P003-N002", "location": "Footnote 4, p. 3 (PDF p. 3)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "See \\hl{Lemma 3.1} in Section 3, below.", "corrected": "See \\hl{Proposition 3.2} in Section 3, below.", "evidence": "I checked footnote 4 on p.~3, and Lemma 3.1 and Proposition 3.2 on p.~7.", "problem": "The footnote is attached to the statement that $\\mathcal I_\\alpha$ contains $\\tau(\\mathbf y)$ with probability at least $1-\\alpha$. Lemma 3.1 states that $L_\\alpha(\\mathbf n)\\le T(\\mathbf n)\\le U_\\alpha(\\mathbf n)$ when $n=2m$. The coverage statement is Proposition 3.2, equation (3.2).", "what_survives": "Nothing depends on the cross-reference, and both results are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N002", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Section 1.5 on page 4, headed Code. The text reads: Code implementing our proposed algorithms and replicating the tables provided in Section 7 is available at a GitHub address ending in hchang1508/BinaryOutcomePermutationTest_replication. A red outline surrounds the words implementing our proposed algorithms and replicating the tables provided in Section 7.", "caption": "Section 1.5, p. 4 (PDF p. 4). The outline marks the description of what the linked code does.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N004.png"}], "formal_release": null, "formal_statements": [], "headline": "The linked replication files do not replicate every table of Section 7 as released: the Table 2 script stops at an undefined name, there is no script for Table 1, and a floating-point feasibility check can skip a possible table.", "id": "P003-N004", "location": "Section 1.5, p. 4 (PDF p. 4); replication files at the linked repository, commit 9805051", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Code \\hl{implementing our proposed algorithms and replicating the tables provided in Section 7} is available at \\url{https://github.com/hchang1508/BinaryOutcomePermutationTest_replication}.", "corrected": "For the defects described here, I would repair the replication files as follows.\n\\begin{itemize}\n\\item In \\texttt{5\\_create\\_table\\_balanced.R}, line 115, write \\hl{\\texttt{files\\_s2\\_1000}} for \\texttt{files\\_s12\\_1000}.\n\\item In \\texttt{1\\_simulation\\_unbalanced\\_case\\_1\\_1000.R}, line 40, pass a positive value of \\texttt{nperm}, and remove the steps for $n=1000$ from the README instructions for Table 3.\n\\item Add a script that produces Table 1, or state that the files cover Tables 2--7.\n\\item In \\texttt{check\\_j}, in \\texttt{generate\\_v10} and where the count vector is rebuilt, carry $k=n\\tau_0$ as an integer.\n\\item In \\texttt{test\\_tau0\\_balanced}, test the extra table of Step 2(c) only if it is possible, as Algorithm 4.3 prescribes.\n\\end{itemize}", "evidence": "I checked the released code and an exact example. When the unchanged lines of the two R scripts are evaluated, R reports that the object \\texttt{files\\_s12\\_1000} is not found and that the argument \\texttt{nperm} is missing with no default. For the feasibility check, let $n=50$ and $m=25$, with observed counts $(n_{11},n_{10},n_{01},n_{00})=(3,22,0,25)$, and let $\\tau_0=7/50$ and $j=7$. The inequalities (B.2) hold, the range (B.3) is the single value $v_{10}=7$, and the table $\\mathbf v=(0,7,0,43)$ is possible. Its exact permutation $p$-value is $1$. Indeed, with $X$ the number of treated $(1,0)$ units, I have $|T-\\tau_0|=|2X-7|/50\\ge1/50$, which is the observed deviation. In double precision the product $50\\cdot(7/50)$ equals $7+2^{-50}$. The unchanged \\texttt{check\\_j} then returns false, and the unchanged \\texttt{generate\\_v10} returns its empty-range value $51$. At $n=50$, the computed product $n\\cdot(k/n)$ differs from $k$ for $k=7$, $14$, $28$ and $29$. The example shows that the code omits a possible table. It does not show that $\\tau_0=7/50$ is rejected, since other values of $j$ can supply an accepted table.", "problem": "The sentence is too broad for the replication files at commit 9805051, in four respects. First, the script that assembles Table 2, \\texttt{5\\_create\\_table\\_balanced.R}, defines \\texttt{files\\_s2\\_1000} on line 113 and then uses the undefined name \\texttt{files\\_s12\\_1000} on line 115, where it stops (see also \\fid{P003-E027}). Second, the driver \\texttt{1\\_simulation\\_unbalanced\\_case\\_1\\_1000.R} calls \\texttt{compare\\_ATE\\_unbalanced} without its required argument \\texttt{nperm} and cannot run (see also \\fid{P003-N011}). Third, the files contain no script for Table 1, which Section 7.1 presents; their README covers Tables 2--7. Fourth, the C++ functions \\texttt{check\\_j} and \\texttt{generate\\_v10}, which apply the feasibility conditions (B.2) and (B.3), compare integers with the product $n\\tau_0$ computed in double precision. When the computed product is not exactly an integer, a possible table at the edge of the range (B.3) can be skipped.\n\nTwo further points were examined and are not errors of this kind. The README lists steps for $n=1000$ under Table 3, one of them for a script that is absent, but the published Table 3 has only $n=50$, $100$ and $200$. The function \\texttt{test\\_tau0\\_balanced} runs the extra test of Step 2(c) of Algorithm 4.3 without first checking that the extra table is possible. For $0\\le\\varepsilon<\\alpha<1$ this cannot make the code accept an impossible table, although it spends random draws that the algorithm does not. Other departures of the released code from the algorithms are the subject of \\fid{P003-E021}, \\fid{P003-E022}, \\fid{P003-E023}, \\fid{P003-E024} and \\fid{P003-E025}.", "what_survives": "The feasibility conditions (B.2)--(B.3), the possibility condition of Algorithm 4.3 and the mathematical results are unaffected. No entry of a table and no interval is shown to be wrong. The statement of Section 1.5 needs qualification until the replication files are repaired."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N004", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of text. A red box marks the reference (4.4) in the sentence 'We note that Algorithm 4.5 is trivially parallelizable, for example by dividing the computation of the V_i in (4.4) across distinct processors.' The next line begins Remark 4.7.", "caption": "Section 4.2, p. 10 (PDF p. 10), the paragraph before Remark 4.7. The marked reference (4.4) is the parametrization of the count vector on p. 8; the $V_i$ are defined in (4.6).", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N005.png"}, {"alt": "Crop of the four lines that state Algorithm 4.5. Red boxes mark three places: the list of inputs 'alpha in (0,1), epsilon in [0, alpha), and a vector', which does not contain K; the words 'Section 4' in 'The algorithm is identical to Section 4'; and the symbol Z with superscript (k) at the end of 'A new set of independent vectors Z(1), ..., Z(k)'.", "caption": "Algorithm 4.5, p. 9 (PDF p. 9). The marked list of inputs has no $K$, although the output is indexed by $K$; “Section 4” stands for Algorithm 4.3; and the last vector carries a lowercase $k$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N005-b.png"}, {"alt": "Crop of five lines of Section 7.1. A red box marks 'Algorithm 4.3' in the sentence 'We compare the confidence intervals (CIs) returned by Algorithm 4.3, based on permutation tests with the difference-in-means statistic, with asymptotic intervals based on the studentized difference-in-means statistic and a normal critical value.'", "caption": "Section 7.1, p. 14 (PDF p. 14), the description of Table 2. The marked reference names the exact algorithm, whereas the caption of Table 2 chooses the number of randomizations by (4.9).", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N005-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The $V_i$ are defined in (4.6), not (4.4); Algorithm 4.5 modifies Algorithm 4.3 and takes $K$ as an input; and Table 2 reports the intervals of Algorithm 4.5.", "id": "P003-N005", "location": "Section 4.2, p. 10 (PDF p. 10), paragraph before Remark 4.7; Algorithm 4.5, p. 9 (PDF p. 9); Section 7.1, p. 14 (PDF p. 14), description of Table 2", "pdf_page": 10, "printed_page": "10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~10: ``We note that Algorithm 4.5 is trivially parallelizable, for example by dividing the computation of the $V_i$ in \\hl{(4.4)} across distinct processors.''\n\nP.~9: ``Algorithm 4.5. This algorithm takes as input $\\alpha\\in(0,1)$, $\\varepsilon\\in[0,\\alpha)$, \\hl{and a vector} $\\mathbf n\\in\\mathbb Z_{\\ge 0}^{\\{0,1\\}^2}$. It returns an interval $\\mathcal I_{\\alpha,\\varepsilon,K}(\\mathbf n)=[L_{\\alpha,\\varepsilon,K}(\\mathbf n),U_{\\alpha,\\varepsilon,K}(\\mathbf n)]$. The algorithm is identical to \\hl{Section 4}, except every use of an exact permutation test (Algorithm 2.2) is replaced with an approximate permutation test with the given parameters (Algorithm 4.4). A new set of independent vectors $\\mathbf Z^{(1)},\\ldots,\\mathbf Z^{(\\hl{k})}$ is generated for each approximate permutation test (independently of the permutation vectors used for any previous tests).''\n\nP.~14: ``We compare the confidence intervals (CIs) returned by \\hl{Algorithm 4.3}, based on permutation tests with the difference-in-means statistic, with asymptotic intervals based on the studentized difference-in-means statistic and a normal critical value.''", "corrected": "P.~10: ``\\ldots by dividing the computation of the $V_i$ in \\hl{(4.6)} across distinct processors.''\n\nP.~9: ``Algorithm 4.5. This algorithm takes as input $\\alpha\\in(0,1)$, $\\varepsilon\\in[0,\\alpha)$, $\\hl{K\\in\\mathbb Z_{>0}}$, and a vector $\\mathbf n\\in\\mathbb Z_{\\ge 0}^{\\{0,1\\}^2}$. \\ldots\\ The algorithm is identical to \\hl{Algorithm 4.3}, except every use of an exact permutation test (Algorithm 2.2) is replaced with an approximate permutation test with the given parameters (Algorithm 4.4). A new set of independent vectors $\\mathbf Z^{(1)},\\ldots,\\mathbf Z^{(\\hl{K})}$ is generated for each approximate permutation test \\ldots''\n\nP.~14: ``We compare the confidence intervals (CIs) returned by \\hl{Algorithm 4.5}, \\ldots''", "evidence": "I checked pp.~8--10 and 14--15 of the article.", "problem": "Equation (4.4), on p.~8, is the parametrization of the count vector. The indicators $V_i$ are defined in (4.6), on p.~9. Algorithm 4.5 is Algorithm 4.3 with each exact test replaced by the Monte Carlo test of Algorithm 4.4. That test needs the number of samples $K$, which also indexes the output, but $K$ is not among the inputs, and the last vector is written with a lowercase $k$. In the caption of Table 2, the number of randomizations is chosen by (4.9), which implies that the intervals described in Section 7.1 are those of Algorithm 4.5.", "what_survives": "Nothing else changes: the remark on parallel computation, Proposition 4.6 and Table 2 are unaffected. The reference in Remark 4.7 to ``Steps 1 through 3 of Algorithm 4.5'' stands, since Algorithm 4.5 inherits the steps of Algorithm 4.3."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N005", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the second paragraph of Remark 5.6. Red boxes mark three places: the symbol U_{alpha,epsilon,K}(n) in 'correctly returning U_{alpha,epsilon,K}(n) (or some larger value) requires'; the words 'smaller U_{alpha,epsilon,K}(n)' in 'that no value smaller U_{alpha,epsilon,K}(n) is rejected during the search process'; and the symbol U_{alpha,epsilon,K}(n) in 'An incorrect rejection early in the search may lead to U_{alpha,epsilon,K}(n) (or larger values) never being considered.'", "caption": "Remark 5.6, second paragraph, p. 13 (PDF p. 13). The three marked symbols denote the endpoint returned by the Monte Carlo search itself; the argument concerns the exact endpoint $U_\\alpha(\\mathbf n)$. The second mark also shows “smaller” without “than”.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N006.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T23", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T23"}], "headline": "In Remark 5.6 the endpoint that a false rejection can cut off is the exact endpoint $U_\\alpha(\\mathbf n)$, not the Monte Carlo endpoint $U_{\\alpha,\\varepsilon,K}(\\mathbf n)$, and “than” is missing after “smaller”.", "id": "P003-N006", "location": "Remark 5.6, second paragraph, p. 13 (PDF p. 13)", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T23"], "text": {"as_printed": "``For example, when searching for the upper bound, correctly returning $\\hl{U_{\\alpha,\\varepsilon,K}(\\mathbf n)}$ (or some larger value) requires that no value \\hl{smaller} $\\hl{U_{\\alpha,\\varepsilon,K}(\\mathbf n)}$ is rejected during the search process. An incorrect rejection early in the search may lead to $\\hl{U_{\\alpha,\\varepsilon,K}(\\mathbf n)}$ (or larger values) never being considered.''", "corrected": "``For example, when searching for the upper bound, correctly returning $\\hl{U_\\alpha(\\mathbf n)}$ (or some larger value) requires that no value smaller \\hl{than} $\\hl{U_\\alpha(\\mathbf n)}$ is rejected during the search process. An incorrect rejection early in the search may lead to $\\hl{U_\\alpha(\\mathbf n)}$ (or larger values) never being considered.''", "evidence": "I checked the page against the definition of $U_{\\alpha,\\varepsilon,K}$ in Algorithm 4.5 (p.~9), the definition of $U_\\alpha$ (pp.~3 and 6) and the binary search of Algorithm 4.3 (p.~8) and Algorithm B.1 (p.~20).", "problem": "Algorithm 4.5 defines $U_{\\alpha,\\varepsilon,K}(\\mathbf n)$ as the value returned by the randomized binary search. The phrase ``correctly returning $U_{\\alpha,\\varepsilon,K}(\\mathbf n)$'' then names no target. The remark compares the search with the exact endpoint $U_\\alpha(\\mathbf n)$: if a value below $U_\\alpha(\\mathbf n)$ is rejected by a Monte Carlo test, the binary search discards $U_\\alpha(\\mathbf n)$ and every larger value.", "what_survives": "The rest of Remark 5.6 is unaffected, including the statement that Theorem 5.5 holds for every $K$. The remark explains a difficulty of this binary search; it does not claim that no procedure for the balanced case can be valid for every $K$."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N006", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 1 with its caption. One red box marks the caption phrase 'number of permutation tests required to return I alpha for Algorithm 4.3'. A second red box marks the column headed Algorithm 4.3, with entries 24, 16 and 26. The neighbouring columns give the intervals [-14, -5], [-4, 10], [-3, 13] and the counts 113, 308, 421 and 189, 1225, 2160 for the two earlier methods.", "caption": "Table 1, p. 14. The marked phrase of the caption says what the column “Algorithm 4.3” counts, and the marked column gives the three counts.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The counts 24, 16 and 26 in the “Algorithm 4.3” column of Table 1 are those of the binary search as implemented in the replication code; Algorithm B.1 as stated uses fewer tests.", "id": "P003-N007", "location": "Table 1, column “Algorithm 4.3” and third sentence of the caption, p. 14 (PDF p. 14); Algorithm 4.3, p. 8; Algorithm B.1, p. 20", "pdf_page": 14, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The remaining columns indicate the \\hl{number of permutation tests required to return $\\mathcal{I}_\\alpha$ for Algorithm 4.3} and the algorithms from Rigdon and Hudgens (2015) and Li and Ding (2016).\n\n\\begin{audittable}{llr}{Table 1, first three columns.}\n$\\boldsymbol{n}$ & Confidence interval & Algorithm 4.3 \\\\\n$(2,6,8,0)$ & $[-14,-5]$ & 24 \\\\\n$(6,4,4,6)$ & $[-4,10]$ & 16 \\\\\n$(8,4,5,7)$ & $[-3,13]$ & 26\n\\end{audittable}", "corrected": "The remaining columns indicate the number of permutation tests required to return $\\mathcal{I}_\\alpha$ for \\hl{the implementation of Algorithm 4.3 in the replication code} and the algorithms from Rigdon and Hudgens (2015) and Li and Ding (2016). \\hl{The implementation rounds each midpoint of the binary search to the nearest integer (halfway values away from zero), always evaluates the final outer bounds, and computes the statistic with the R function \\texttt{mean()}; Algorithm B.1 as stated requires fewer tests.}", "intended_meaning": "The column gives the number of permutation tests run by the implementation of the binary-search method, and under that reading the three counts are correct. Using the search in the replication code with exact permutation $p$-values, I obtain 24, 16 and 26 tests and the three printed intervals. That search is not Algorithm B.1, to which Algorithm 4.3 refers for the precise definition of the binary search. The code rounds each midpoint to the nearest integer, where Algorithm B.1 takes $c=\\lfloor (a+b)/2\\rfloor$, and it always evaluates the final outer bounds, where Algorithm B.1 returns an interior final bound without a further evaluation. Algorithm B.1 as stated uses fewer tests (15, 4 and 5 for the three upper endpoints). Since the paper does not specify the search for the lower endpoint, the text determines no single column of counts for Algorithm B.1; two natural choices, a search with floor midpoints and Algorithm B.1 applied after the reflection $x\\mapsto -x$, give the totals in the table below. In the third row the count depends on the floating-point rounding of a midpoint equal to $-1.5$: it is 25 in exact arithmetic, and 26 when the statistic is computed, as in the replication code, from two calls to the R function \\texttt{mean()}. Every procedure in the table returns the printed intervals, and a direct inversion over all possible count vectors gives the same intervals. Every procedure also uses fewer tests than either of the two earlier methods in Table 1.\n\n\\begin{audittable}{lrrr}{Number of permutation tests for the three data sets of Table 1, by search procedure.}\nSearch procedure & $(2,6,8,0)$ & $(6,4,4,6)$ & $(8,4,5,7)$ \\\\\nAlgorithm B.1, lower search with floor midpoints & 17 & 10 & 18 \\\\\nAlgorithm B.1, lower search by reflection & 17 & 11 & 19 \\\\\nReplication code, exact midpoints & 24 & 16 & 25 \\\\\nReplication code, statistic from R \\texttt{mean()} & 24 & 16 & 26 \\\\\nPrinted in Table 1 & 24 & 16 & 26\n\\end{audittable}"}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N007", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the three-line caption of Table 1. A red box marks the expression n tau of bold n in the first sentence, which reads: 95 percent confidence intervals for n tau(n).", "caption": "Table 1, caption, p. 14. The marked expression is the target of the intervals as printed, $n\\tau(\\boldsymbol{n})$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "The caption of Table 1 should name the target of the intervals as $n\\tau(\\boldsymbol{y})$, the scaled sample average treatment effect of the potential outcomes, and not $n\\tau(\\boldsymbol{n})$.", "id": "P003-N008", "location": "Table 1, caption, first sentence, p. 14 (PDF p. 14)", "pdf_page": 14, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "95\\% confidence intervals for $n\\tau(\\hl{\\boldsymbol{n}})$.", "corrected": "95\\% confidence intervals for $n\\tau(\\hl{\\boldsymbol{y}})$.", "evidence": "I checked the page against the definitions (2.2), (2.6) and (2.8) and Algorithm 2.3. If the first observed vector $(2,6,8,0)$ were read as a count vector of potential outcomes, then $n\\tau$ would equal $6-8=-2$, which lies outside the interval $[-14,-5]$ printed in that row.", "problem": "Equation (2.6) defines $\\tau(\\boldsymbol{w})$ for a potential outcome table $\\boldsymbol{w}$, and (2.8) defines $\\boldsymbol{n}$ as the vector of observed counts. The intervals in Table 1 are computed from $\\boldsymbol{n}$, and their target is the effect $\\tau(\\boldsymbol{y})$ of the unknown table $\\boldsymbol{y}$ of the subjects, multiplied by $n$.", "what_survives": "The intervals and the counts in Table 1 are unchanged, and nothing else depends on the caption."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N008", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines at the end of Section 7.1. Two red boxes mark the words 'the unbalanced algorithm takes around 14.5 h.', which run over a line break. The sentence begins 'With n = 1000 and a single thread on a contemporary processor, the balanced algorithm takes around 45 min'.", "caption": "Section 7.1, last paragraph, p. 14. The marked words give the run time of the unbalanced algorithm at $n=1000$, which the script quoted above would have to reproduce.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N011.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication script for the unbalanced design with $n=1000$ omits the required argument nperm and stops with an error; as supplied, it cannot reproduce the run time of around 14.5 h.", "id": "P003-N011", "location": "Replication script 1_simulation_unbalanced_case_1_1000.R, line 40, for the run time quoted in Section 7.1, last paragraph, p. 14 (PDF p. 14)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "With $n = 1000$ and a single thread on a contemporary processor, the balanced algorithm takes around 45 min, and the unbalanced algorithm takes around 14.5 h.\n\nReplication code (not printed in the article), \\texttt{1\\_simulation\\_unbalanced\\_case\\_1\\_1000.R}, line 40 (one line in the file, broken here after the seventh argument):\n\n\\texttt{result\\_1000=\\hl{compare\\_ATE\\_unbalanced(nsim,n,nt,v11\\_share,v10\\_share,v01\\_share,1,}}\n\n\\texttt{\\hl{display\\_progress=FALSE,warning\\_msg=FALSE)}}", "corrected": "\\texttt{result\\_1000=compare\\_ATE\\_unbalanced(nsim,n,nt,v11\\_share,v10\\_share,v01\\_share,1,}\n\n\\texttt{\\hl{nperm=K,}display\\_progress=FALSE,warning\\_msg=FALSE)}\n\nHere \\texttt{K} is the number of Monte Carlo draws of the run that produced the reported time, a positive integer to be recovered from that run and documented. The value 0 is not a repair: the inner call passes \\texttt{epsilon=0}, and with \\texttt{nperm=0} the default formula for the number of draws then returns an infinite value.", "evidence": "I base this correction on a recalculation from the replication files. The unchanged function definitions were executed in R with the inputs of the script, and the call fails with the message above before the compiled routine is reached; a global variable named \\texttt{nperm} does not prevent the failure. With \\texttt{nperm=20000} added to the call, the same code passes 20000 draws to a stand-in for the compiled routine. This tested argument propagation only; the number of draws behind the published run time is not known. In the public history of the repository, the number of draws was fixed at 20000 inside the helper function until a change of June 2025 made \\texttt{nperm} a required argument without updating this call. The omission therefore does not show that the run time was never measured.", "problem": "The function \\texttt{compare\\_ATE\\_unbalanced} in \\texttt{2\\_sim\\_raw.R} takes the number of Monte Carlo draws as its eighth argument, \\texttt{nperm}, and gives it no default value. The call on line 40 passes seven positional arguments, the last of which is matched to \\texttt{test\\_type}, and two named arguments. Nothing is matched to \\texttt{nperm}. R stops with the message \\texttt{argument \"nperm\" is missing, with no default} before any permutation test is computed, and no output file is written. The script sets $n=1000$ with 600 treated units and the potential outcomes of the first scenario. As supplied, it cannot reproduce the run time quoted in Section 7.1.", "what_survives": "No theorem, coverage statement or complexity bound depends on this script. Table 3 reports only $n=50$, $100$ and $200$, and the scripts for those sample sizes pass \\texttt{nperm}, as do the separate scripts behind Tables 4--7. I have not shown that the figure of 14.5 h is wrong, and the missing argument does not establish whether the entries of those tables are correct."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N011", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the caption of Table 2 and the first-scenario block of the table. Red boxes mark the words '50, 100, and 200.' and '25, 50, and 100,' in the first two sentences of the caption, and the table row for sample size 1000, with entries 0.96, 0.12, 0.95, 0.12.", "caption": "Table 2, caption and first scenario, p. 15. The marked lists in the caption give three sample sizes and three numbers of treated units; the marked row is the fourth sample size, 1000.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N012.png"}], "formal_release": null, "formal_statements": [], "headline": "The caption of Table 2 lists the sample sizes 50, 100 and 200 and omits the sample size 1000, with 500 treated units, for which the table has a row in each scenario.", "id": "P003-N012", "location": "Table 2, caption, first two sentences, p. 15 (PDF p. 15)", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Simulation results for the balanced case with sample sizes \\hl{50, 100, and 200}. The numbers of treated units are \\hl{25, 50, and 100}, respectively.", "corrected": "Simulation results for the balanced case with sample sizes \\hl{50, 100, 200, and 1000}. The numbers of treated units are \\hl{25, 50, 100, and 500}, respectively.", "evidence": "I checked the caption against the table and text on the page. The caption also gives the number of simulations as $10^4$ without restriction. Whether the rows for $n=1000$ rest on $10^4$ simulations is not established, because the original outputs are not in the replication files; the settings of the job arrays for these rows are the subject of \\fid{P003-E028}.", "problem": "Table 2 has four rows in each scenario, for sample sizes 50, 100, 200 and 1000, and the text below it says ``We also report results in the balanced case with a sample size of 1000.'' The caption lists only the first three sample sizes and their numbers of treated units. In the balanced design the row for $n=1000$ has 500 treated units.", "what_survives": "The rows for $n=1000$ and the description of the design in the text are unchanged, and the omission in the caption does not show that any table entry is wrong."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N012", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last four lines of the caption of Table 3 and the head of the table. The caption ends: The number of simulation is 10 to the power 4, and the number of randomizations is 2 times 10 to the power 4. All numbers are rounded two digits after the decimal point. A red outline surrounds the words: the number of randomizations is 2 times 10 to the power 4. Below the caption are the column heads Sample size, First Scenario, Permutation (DiM), Permutation (Std) and Wald, each with Cov. and Med. Length, and the row for sample size 50: 0.96, 0.48, 0.96, 0.52, 0.96, 0.57.", "caption": "Table 3, end of the caption and head of the table, p. 16. The outline marks the number of randomizations stated for the table. The script that assembles the table does not select the runs with this number.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N014.png"}], "formal_release": null, "formal_statements": [], "headline": "The script that assembles Table 3 reads every file in its input folders, with no selection by the number of randomizations $K$ or by the design, and returns the table at the stated $K = 2 \\times 10^4$ only when the folders hold those runs and nothing else.", "id": "P003-N014", "location": "Table 3, caption, p. 16 (PDF p. 16); replication code 5_create_table_unbalanced.R, lines 9–99", "pdf_page": 16, "printed_page": "16", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Table~3, caption: ``The number of simulation is $10^4$, and the number of randomizations is $2 \\times 10^4$. All numbers are rounded two digits after the decimal point.''\n\nReplication code (not printed in the article), \\texttt{5\\_create\\_table\\_unbalanced.R}, lines 11--15:\n\n\\texttt{files\\_s1\\_50=\\hl{list.files()}}\n\n\\texttt{output\\_50\\_s1=c()}\n\n\\texttt{for (i in 1:length(files\\_s1\\_50))\\{}\n\n\\texttt{\\ \\ output\\_50\\_s1=rbind(output\\_50\\_s1, read.csv(files\\_s1\\_50[i])[,2:15])}\n\n\\texttt{\\}}", "corrected": "In \\texttt{5\\_create\\_table\\_unbalanced.R}, each of the six blocks for Table~3 should select its files by name before it reads them. For sample size $N$, the block \\hl{reads only the files whose names consist of a numerical job index, the sample size and 20000}, in the form \\texttt{INDEX\\_N\\_20000.csv}. This excludes the runs with other values of $K$ and the output of the timing comparison, whose file names begin with \\texttt{balanced\\_design}. Each block should also check that every file has the 15 columns of the simulation output and that each group has $10^4$ rows, and the runs for the difference-in-means test should be made with $K = 20{,}000$ stated explicitly. If the later comparison blocks are kept, each should take its files from its own list, loop over that list, and append to its own accumulator. The repair concerns the selection of files. The entries of Table~3 still have to be checked against the original outputs for $K = 2 \\times 10^4$ or against a new run.", "evidence": "I base this correction on an exact counterexample. Lines 9--99 of the script were executed on constructed files that have the columns of the simulation output and ten rows each. Only the folder paths and the call that prints the table were changed. With one file for $K = 20{,}000$ in each folder, the six groups of Table~3 are summarized correctly. With one file for $K = 10{,}000$ and one for $K = 20{,}000$ in each folder, the script pools the twenty rows. In the test files the coverage indicator is $0$ in the first file and $1$ in the second, and the widths are $1/2$ and $1$. The script reports coverage $1/2$ and median width $3/4$, where the file for $K = 20{,}000$ alone gives $1$ and $1$. With the file for $K = 10{,}000$ alone, it reports coverage $0$ and width $1/2$ and raises no error. These inputs are test values and not simulation results.", "problem": "The replication files name \\texttt{5\\_create\\_table\\_unbalanced.R} as the script that assembles Table~3. For each scenario and each sample size, the script lists every file in one output folder and reads all of them. The simulation scripts write a separate file for each $K$ to that folder, and the script does not select among them. The submission script in the replication files requests $K = 10{,}000$ in its active lines, where the caption states $2 \\times 10^4$. This $K$ is the number of randomizations of the test with the difference-in-means statistic; the studentized test always uses 20,000. The output of the timing comparison on balanced designs is written to the same folders, and the script does not set it aside either. The script returns the summaries at $K = 2 \\times 10^4$ only if each folder holds the runs with that $K$ and no other file. If a folder holds runs with two values of $K$, the script pools them, and if it holds runs with $K = 10{,}000$ alone, the script reports them without a warning. Later blocks of the same script, which do not enter Table~3, index the wrong lists of files and rebuild their accumulators from an unrelated object at every step.", "what_survives": "No entry of Table~3 is shown to be wrong, and the caption is not shown to be false. The original outputs are not in the replication files, and it is not known which files the folders held when the table was made. The algorithms, the estimands and the coverage proofs do not depend on this script. The same blocks stop with an error when the two kinds of output share a folder, which is the subject of \\fid{P003-E027}."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N014", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of three lines of text: Recall from (2.1) that w denotes a generic potential outcome table. We use the notation T(w, Z) to stand for T(Y, Z) with Y generated using the potential outcomes w and the treatment assignments given by Z. Since this quantity depends on w only through its associated count vector v, we also write T(v, Z) for T(n, Z). A red outline surrounds the final symbol T(n, Z).", "caption": "Appendix A, notation paragraph before Section A.1, p. 18. The outline marks the symbol $T(\\mathbf{n}, \\mathbf{Z})$, which is not defined; the sentence before it defines $T(\\mathbf{w}, \\mathbf{Z})$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N017.png"}, {"alt": "Crop of the start of the proof of Lemma A.2: Let Z dagger be the random vector defined by Z dagger i = 1 - Z i. Then for any randomization given by Z, the vector Z dagger represents a randomization that exchanges the control and treatment groups. We have, in display (A.4), one half of (T(w, Z) + T(w, Z dagger)) = 1 over 2m times the sum from i = 1 to n of (w j (1) - w j (0)) = the expectation of T(w, Z). A red outline surrounds the sum: the index under the summation sign is i and the two subscripts in the summand are j.", "caption": "Proof of Lemma A.2, display (A.4), p. 18. The outline marks the sum, whose index is $i$ while its summand is indexed by $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N017-b.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T24", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T24"}], "headline": "Two symbols on p. 18 are misprinted: $T(\\mathbf{n}, \\mathbf{Z})$ should read $T(\\mathbf{w}, \\mathbf{Z})$, and the summand of (A.4) should carry the summation index $i$.", "id": "P003-N017", "location": "Appendix A, last sentence of the notation paragraph before Section A.1, and display (A.4) in the proof of Lemma A.2, p. 18 (PDF p. 18)", "pdf_page": 18, "printed_page": "18", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T24"], "text": {"as_printed": "``Since this quantity depends on $\\mathbf{w}$ only through its associated count vector $\\mathbf{v}$, we also write $T(\\mathbf{v}, \\mathbf{Z})$ for $T(\\hl{\\mathbf{n}}, \\mathbf{Z})$.''\n\\[\n\\frac{1}{2}\\big(T(\\mathbf{w}, \\mathbf{Z}) + T(\\mathbf{w}, \\mathbf{Z}^\\dagger)\\big) = \\frac{1}{2m}\\sum_{i=1}^{n}\\big(w_{\\hl{j}}(1) - w_{\\hl{j}}(0)\\big) = \\mathbb{E}\\big[T(\\mathbf{w}, \\mathbf{Z})\\big]. \\qquad \\text{(A.4)}\n\\]", "corrected": "``Since \\hl{the distribution of} this quantity \\hl{under uniform randomization} depends on $\\mathbf{w}$ only through its associated count vector $\\mathbf{v}$, we also write $T(\\mathbf{v}, \\mathbf{Z})$ for $T(\\hl{\\mathbf{w}}, \\mathbf{Z})$.''\n\nDisplay (A.4), with its hypotheses: let $n = 2m$ for some $m \\in \\mathbb{Z}_{>0}$, let $\\mathbf{w}$ be a potential outcome table, and let $\\mathbf{Z}^\\dagger$ be defined by $Z_i^\\dagger = 1 - Z_i$. Then\n\\[\n\\frac{1}{2}\\big(T(\\mathbf{w}, \\mathbf{Z}) + T(\\mathbf{w}, \\mathbf{Z}^\\dagger)\\big) = \\frac{1}{2m}\\sum_{i=1}^{n}\\big(w_{\\hl{i}}(1) - w_{\\hl{i}}(0)\\big) = \\mathbb{E}\\big[T(\\mathbf{w}, \\mathbf{Z})\\big]. \\qquad \\text{(A.4)}\n\\]", "evidence": "The evidence is a check of the page and a derivation of the corrected (A.4). Let $n = 2m$ and $Z_i^\\dagger = 1 - Z_i$. By (2.5) and (2.7), I have\n\\[\nT(\\mathbf{w}, \\mathbf{Z}) = \\frac{1}{m}\\sum_{i=1}^{n}\\big(Z_i w_i(1) - (1 - Z_i) w_i(0)\\big).\n\\]\nReplacing $Z_i$ by $1 - Z_i$ and adding the two expressions, I obtain\n\\[\nT(\\mathbf{w}, \\mathbf{Z}) + T(\\mathbf{w}, \\mathbf{Z}^\\dagger) = \\frac{1}{m}\\sum_{i=1}^{n}\\big(w_i(1) - w_i(0)\\big).\n\\]\nHalf of the right side is $\\tau(\\mathbf{w})$, by (2.6) with $n = 2m$. Since $\\mathbf{Z}$ and $\\mathbf{Z}^\\dagger$ have the same distribution, the expectation of the left side is $2\\,\\mathbb{E}[T(\\mathbf{w}, \\mathbf{Z})]$, which implies the last equality of (A.4).", "problem": "The notation paragraph defines $T(\\mathbf{w}, \\mathbf{Z})$ in the sentence before, and the shorthand that it introduces replaces the table $\\mathbf{w}$ by its count vector $\\mathbf{v}$. The symbol $T(\\mathbf{n}, \\mathbf{Z})$ is defined nowhere: p.~5 introduces only $T(\\mathbf{n})$, for the observed count vector $\\mathbf{n}$. In (A.4), the sum runs over $i$ and the summand is indexed by $j$. Both are slips of notation, and the identity (A.4) holds with matching indices.", "what_survives": "The symmetry argument of Lemma~A.2 is unchanged, and nothing else depends on the two symbols. The correction of the first symbol concerns the notation only. The count-vector shorthand concerns the distribution under uniform randomization, as explained in \\fid{P003-E015}."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N017", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the proof of Theorem 4.1. Three red boxes: the words 'there exists a table u' inside display (B.11), which defines the set J alpha of n; the words 'is an interval equal to I alpha of n' in the line after the display; and the right side '[T(n), U alpha(n)]' of display (B.12).", "caption": "Appendix B.2, proof of Theorem 4.1, p. 21 (PDF p. 21), displays (B.11) and (B.12). The marked passages are the phrase that admits every table $\\mathbf u$, the assertion that the set is an interval equal to $\\mathcal I_\\alpha(\\mathbf n)$, and the right side of (B.12).", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N018.png"}], "formal_release": null, "formal_statements": [{"id": "P003-T25", "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-T25"}], "headline": "In (B.11) the table $\\mathbf u$ must be possible given $\\mathbf n$, and the set $\\mathcal J_\\alpha(\\mathbf n)$ equals $\\mathcal I_\\alpha(\\mathbf n)\\cap\\mathcal C(\\mathbf n)$, not the interval $\\mathcal I_\\alpha(\\mathbf n)$.", "id": "P003-N018", "location": "Appendix B.2, proof of Theorem 4.1, p. 21 (PDF p. 21), display (B.11), the sentence after it, and display (B.12)", "pdf_page": 21, "printed_page": "21", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P003-T25"], "text": {"as_printed": "``Recall that by Lemma 2.5, the set of elements\n\\[\n\\mathcal J_\\alpha(\\mathbf n)=\\{\\tau_0\\in\\mathcal C(\\mathbf n):\\text{there exists }\\hl{\\text{a table}}\\ \\mathbf u\\text{ such that }\\tau(\\mathbf u)=\\tau_0\\text{ and }p(\\mathbf u,\\mathbf n)\\ge\\alpha\\}\\qquad\\text{(B.11)}\n\\]\nis \\hl{an interval equal to} $\\mathcal I_\\alpha(\\mathbf n)$, so that (by Lemma 3.1)\n\\[\n\\bigl[T(\\mathbf n),C_+\\bigr]\\cap\\mathcal J_\\alpha(\\mathbf n)=\\hl{\\bigl[T(\\mathbf n),U_\\alpha(\\mathbf n)\\bigr]}.\\qquad\\text{(B.12)''}\n\\]", "corrected": "``Recall that by Lemma 2.5, the set of elements\n\\[\n\\begin{aligned}\n\\mathcal J_\\alpha(\\mathbf n)=\\{\\tau_0\\in\\mathcal C(\\mathbf n):{}&\\text{there exists a }\\hl{\\text{possible}}\\text{ table }\\mathbf u\\ \\hl{\\text{given }\\mathbf n}\\\\\n&\\text{such that }\\tau(\\mathbf u)=\\tau_0\\text{ and }p(\\mathbf u,\\mathbf n)\\ge\\alpha\\}\\qquad\\text{(B.11)}\n\\end{aligned}\n\\]\nis an interval \\hl{in the lattice} $\\mathcal C(\\mathbf n)$\\hl{,} equal to $\\mathcal I_\\alpha(\\mathbf n)\\hl{\\cap\\,\\mathcal C(\\mathbf n)}$, so that (by Lemma 3.1)\n\\[\n\\bigl[T(\\mathbf n),C_+\\bigr]\\cap\\mathcal J_\\alpha(\\mathbf n)=\\bigl[T(\\mathbf n),U_\\alpha(\\mathbf n)\\bigr]\\hl{\\cap\\,\\mathcal C(\\mathbf n)}.\\qquad\\text{(B.12)''}\n\\]\nHere $n=2m$ and $\\alpha\\in(0,1)$, as in Theorem 4.1 and Lemma 2.5.", "evidence": "The evidence is an exact counterexample to the display as printed. Let $n=4$, $m=2$ and $\\alpha=1/2$, and let the observed counts be $\\mathbf n=(0,2,0,2)$. In this example $T(\\mathbf n)=0$ and $\\mathcal C(\\mathbf n)=\\{-1/2,-1/4,0,1/4,1/2\\}$. The count vector $\\mathbf v=(1,0,2,1)$ has $\\tau(\\mathbf v)=-1/2$. It is not possible given $\\mathbf n$, since its subject of type $(1,1)$ would show the outcome 1 in either group and every observed outcome is 0. Over the six assignments, $T$ takes the values $-1,-1/2,-1,-1/2,0,0$, and four of them are at distance at least $1/2$ from $-1/2$. I find that $p(\\mathbf v,\\mathbf n)=2/3\\ge\\alpha$, and the set (B.11) as printed contains $-1/2$. The only possible count vector with effect $-1/2$ is $(0,0,2,2)$, with $p=1/3<\\alpha$. The effects of accepted possible tables are $-1/4$, $0$ and $1/4$, and I have $\\mathcal I_\\alpha(\\mathbf n)=[-1/4,1/4]$. The point $1/8$ lies in this interval and not in $\\mathcal C(\\mathbf n)$.", "problem": "Read literally, (B.11) admits every table $\\mathbf u$ such that $\\tau(\\mathbf u)=\\tau_0$ and $p(\\mathbf u,\\mathbf n)\\ge\\alpha$, including tables that are not possible given $\\mathbf n$. Algorithm 2.3 uses only possible tables, and the set as printed can contain effects outside $\\mathcal I_\\alpha(\\mathbf n)$. Further, $\\mathcal J_\\alpha(\\mathbf n)$ is a subset of the finite set $\\mathcal C(\\mathbf n)$, while $\\mathcal I_\\alpha(\\mathbf n)$ is the closed real interval of (2.16). Lemma 2.5 implies that the set defined with possible tables equals $\\mathcal I_\\alpha(\\mathbf n)\\cap\\mathcal C(\\mathbf n)$. Both qualifications are supplied by Lemma 2.5 and by Algorithm 4.3, which tests only possible tables and searches over points of $\\mathcal C(\\mathbf n)$.", "what_survives": "Algorithm 4.3, the endpoints it returns and the argument for Theorem 4.1 are unchanged once $\\mathcal J_\\alpha(\\mathbf n)$ is defined with possible tables and compared with $\\mathcal I_\\alpha(\\mathbf n)\\cap\\mathcal C(\\mathbf n)$."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N018", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the proof of Theorem 5.5. Three red boxes: the symbol I with subscript K, alpha in 'will be included the returned interval'; the words 'q(i) can be computed from q prime (i)' in the paragraph on step 2(d); and the reference '(4.8)' in 'the approximate p-value (4.8)'.", "caption": "Appendix C, proof of Theorem 5.5, p. 28 (PDF p. 28). The marked terms are the interval written $\\mathcal I_{K,\\alpha}$ (the first line of the crop has $\\mathcal I_{\\alpha,K}$), the reversed direction of the update, and the reference (4.8).", "url": "https://pmaronow.github.io/llm-audit/works/P003/exhibits/P003-N020.png"}], "formal_release": null, "formal_statements": [], "headline": "In the proof of Theorem 5.5, the approximate $p$-value is (5.3), not (4.8); $\\mathbf q'^{(i)}$ is computed from $\\mathbf q^{(i)}$; and the returned interval is $\\mathcal I_{\\alpha,K}$.", "id": "P003-N020", "location": "Appendix C, proof of Theorem 5.5, p. 28 (PDF p. 28), last sentence of the first paragraph and the paragraph on step 2(d)", "pdf_page": 28, "printed_page": "28", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Then the value $\\theta(\\mathbf v)$ corresponding to the count vector $\\mathbf v$ such that $\\mathbf v=\\hat{\\mathbf v}$ will be included the returned interval $\\hl{\\mathcal I_{K,\\alpha}}$ with probability at least $1-\\alpha$, by (5.4). This shows (5.28).''\n\n``We next consider the time required for step 2(d). The time complexity of computing $\\{\\mathbf q'^{(i)}\\}_{i=1}^K$ given $\\{\\mathbf q^{(i)}\\}_{i=1}^K$ is $O(K)$, since for each $i\\in[\\![1,K]\\!]$, $\\hl{\\mathbf q^{(i)}}$ can be computed from $\\hl{\\mathbf q'^{(i)}}$ in constant time. Additionally, the time complexity of computing the approximate $p$-value \\hl{(4.8)} given the $K$ vectors $\\{\\mathbf q'^{(i)}\\}_{i=1}^K$ is also $O(K)$.''", "corrected": "``Then the value $\\theta(\\mathbf v)$ corresponding to the count vector $\\mathbf v$ such that $\\mathbf v=\\hat{\\mathbf v}$ will be included \\hl{in} the returned interval $\\hl{\\mathcal I_{\\alpha,K}}$ with probability at least $1-\\alpha$, by (5.4).''\n\n``The time complexity of computing $\\{\\mathbf q'^{(i)}\\}_{i=1}^K$ given $\\{\\mathbf q^{(i)}\\}_{i=1}^K$ is $O(K)$, since for each $i\\in[\\![1,K]\\!]$, $\\hl{\\mathbf q'^{(i)}}$ can be computed from $\\hl{\\mathbf q^{(i)}}$ in constant time. Additionally, the time complexity of computing the approximate $p$-value \\hl{(5.3)} given the $K$ vectors $\\{\\mathbf q'^{(i)}\\}_{i=1}^K$ is also $O(K)$.''", "evidence": "The evidence is a check of pp.~9--13 and 28 and a count of operations. Each of the $K$ updates changes a fixed number of entries, and the sum of $K$ indicators costs $O(K)$ operations for either formula. The cost $O(K)$ for each table in step 2(d) and the total $O(Kn^3)$ are therefore the same after the correction.", "problem": "There are three misprints. Algorithm 5.3 computes its $p$-values by Algorithm 5.1, whose $p$-value is (5.3), namely $(1+\\sum_iV_i)/(K+1)$. Equation (4.8) is the average $\\sum_iV_i/K$ of Algorithm 4.4, which is used in the balanced case, and only (5.3) has the validity (5.4) for every $K$ that the first paragraph of the proof invokes. The update of Section 5.1.1 computes $\\mathbf q'$ from $\\mathbf q$, as the first half of the same sentence says. The interval of Theorem 5.5 is written $\\mathcal I_{\\alpha,K}$ in (5.28).", "what_survives": "Theorem 5.5 and the rest of its proof are unchanged. The $p$-value computed by the replication code is the subject of \\fid{P003-E024}."}, "url": "https://pmaronow.github.io/llm-audit/works/P003/#P003-N020", "work": {"arxiv_id": "2305.09906", "doi": "10.1016/j.jeconom.2025.106056", "title": "Fast computation of exact confidence intervals for randomized experiments with binary outcomes", "url": "https://pmaronow.github.io/llm-audit/works/P003/"}, "work_id": "P003"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the start of Proposition 4. After the words We have the following linearization results and Proposition 4. Define, a two-line display defines tau-hat-L sub IPW of d as the average of O sub i prime beta sub 1,N minus the average of O sub i prime beta sub 0,N, plus the average of Z sub i times mu sub i of Y and d minus O sub i prime beta sub 1,N, over p of O sub i given theta-zero, minus the average of 1 minus Z sub i times mu sub i of Y and d minus O sub i prime beta sub 0,N, over 1 minus p of O sub i given theta-zero. The two last sums have the upper limit n. Then: Under C2, C3, C4, C6, and C1, equation (36): tau-hat-L sub IPW of d minus tau-hat sub IPW of d equals o-p of N to the minus one half. A red outline surrounds equation (36).", "caption": "Supplement A, Proposition 4 of Appendix A.6, p. 15. The outline marks equation (36), which asserts that the printed linear form and the IPW estimator differ by $o_p(N^{-1/2})$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E001.png"}, {"alt": "Crop of equations (42) and (43) with the text between them. Equation (42) expands the average of Z sub i mu sub i over p of O sub i given theta-hat-MLE as the same average at theta-zero, minus the average of Z sub i mu sub i over p of O sub i given theta-zero, times 1 minus p of O sub i given theta-zero, times O sub i prime, times theta-hat-MLE minus theta-zero, plus a second-order term. Equation (43) states that H inverse times the average of Z sub i mu sub i over p of O sub i given theta-zero, times p of O sub i given theta-zero times 1 minus p of O sub i given theta-zero times O sub i, minus beta sub 1,N, is o-p of 1. One red outline surrounds the factor 1 minus p times O sub i prime in (42). A second surrounds the factor p times 1 minus p times O sub i in (43).", "caption": "Supplement A, proof of Proposition 4, equations (42) and (43), p. 16. The outlines mark the weight $(1-p(O_i\\mid\\theta_0))O_i'$ in the first-order term of (42) and the different weight $p(O_i\\mid\\theta_0)(1-p(O_i\\mid\\theta_0))O_i$ in (43).", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E001-b.png"}, {"alt": "Crop of one line of text and a displayed equation. The text reads: Hence (42) becomes. The equation states that the average of Z sub i mu sub i over p of O sub i given theta-hat-MLE equals the average of Z sub i mu sub i over p of O sub i given theta-zero, minus the average of Z sub i minus p of O sub i given theta-zero, over p of O sub i given theta-zero, times O sub i prime beta sub 1,N, plus o-p of N to the minus one half. A red outline surrounds the second term on the right, from its minus sign to beta sub 1,N.", "caption": "Supplement A, proof of Proposition 4, top of p. 17. The outline marks the correction term that is substituted into (42).", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E001-c.png"}, {"alt": "Crop of a line of text, a heading and a displayed inequality. The text reads: A similar homophily condition is needed for the HAC variance estimation. C 2. For all sample size N. The inequality states that one over N times the sum over i of tau sub i minus the difference of O sub i prime beta sub 1,N and O sub i prime beta sub 0,N, times the sum over j in B of i and d of tau sub j minus the difference of O sub i prime beta sub 1,N and O sub i prime beta sub 0,N, is at least 0. Red outlines surround the two symbols O sub i prime in the second factor.", "caption": "Supplement A, condition C 2, p. 14. The outlines mark the two places in the second factor where $O_i'$ is printed and $O_j'$ is needed.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E001-d.png"}, {"alt": "Crop of five lines of text. They read: In practice, p of O sub i is unknown to researchers. In such cases, parametric methods, such as logistic regression, and nonparametric methods, such as the sieve estimator in Hirano, Imbens and Ridder (2003), can be used to estimate the propensity score. In Appendix A.6, we develop a complete inferential theory for the inverse probability weighted (IPW) estimator where propensity scores are modeled using a logistic model. A red outline surrounds the line that begins we develop a complete inferential theory.", "caption": "Main text, Section 6.4, p. 759 (PDF p. 16). The outline marks the line that announces the theory of Appendix A.6.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E001-e.png"}, {"alt": "Two panels under the title: Counterexample to (36): the printed variance is about half the variance of the IPW estimator, and an interval built on it covers less often than its nominal level. Left panel, three bars on an axis labelled N times the variance: a blue bar of height 1753/750000 = 0.00234 for N times the variance of the printed linear form, an orange bar of height 3/625 = 0.00480 for N times the variance of the IPW estimator to first order, and a grey bar of height 10369/750000 = 0.01383 for the variance of their difference scaled by the square root of N. The panel is headed: Equation (36) asserts that the third bar is 0. Right panel, a line chart of limiting coverage in percent against the nominal level of the interval in percent, from 80 to 99: a grey straight line for the nominal level and a blue curve below it for the limiting coverage with the printed variance, with marked points 74.9 percent at nominal 90 and 82.9 percent at nominal 95.", "caption": "The counterexample to (36). Left: $N$ times the variance of the printed linear form ($1753/750000$) and of the first-order form of the IPW estimator ($3/625$), and the variance of $\\sqrt N$ times their difference ($10369/750000$), which (36) asserts to vanish in the limit. Right: the limiting coverage $2\\Phi(z\\sqrt{1753/3600})-1$ of a normal interval for the IPW estimator whose variance estimate converges to the printed variance, against its nominal level.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E001-expansion.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T1", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T1"}, {"id": "P004-T2", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T2"}, {"id": "P004-T4", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T4"}, {"id": "P004-T35", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T35"}], "headline": "Equation (36) can fail when the assignment probabilities differ across nodes: to first order, estimating the logistic coefficient changes the IPW estimator by $-N^{-1}\\sum_i(Z_i-p_i)O_i'\\gamma_N$ with $\\gamma_N=H_N^{-1}N^{-1}\\sum_i[(1-p_i)m_{i1}+p_im_{i0}]O_i$, and the linear form, the variance, the two variance estimators and the coverage statements of Proposition 4 of A.6 are rebuilt on this term.", "id": "P004-E001", "location": "Supplement A, Appendix A.6: Proposition 4 and equation (36), p. 15; its proof, equations (42) and (43), p. 16, and the display after “Hence (42) becomes”, p. 17; condition C 2, p. 14. Announced in Section 6.4, p. 759 (PDF p. 16)", "pdf_page": 15, "printed_page": "15", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P004-T1", "P004-T2", "P004-T35", "P004-T4"], "text": {"as_printed": "\\emph{Main text, Section 6.4, p.~759.} In Appendix A.6, we develop \\hl{a complete inferential theory} for the inverse probability weighted (IPW) estimator where propensity scores are modeled using a logistic model.\n\n\\emph{Supplement A, p.~14.} For brevity, we denote $p(O_i\\mid\\theta_0)=p_0(O_i)$. We define the following coefficients:\n\\[\n\\beta_{1,N}=\\Bigl(\\frac1N\\sum_{i=1}^{N}p_0(O_i)(1-p_0(O_i))O_iO_i'\\Bigr)^{-1}\\Bigl(\\frac1N\\sum_{i=1}^{N}\\mathrm E\\bigl[\\mu_i(1;d)\\,p_0(O_i)(1-p_0(O_i))O_i\\bigr]\\Bigr),\n\\]\nand,\n\\[\n\\beta_{0,N}=\\Bigl(\\frac1N\\sum_{i=1}^{N}p_0(O_i)(1-p_0(O_i))O_iO_i'\\Bigr)^{-1}\\Bigl(\\frac1N\\sum_{i=1}^{N}\\mathrm E\\bigl[\\mu_i(0;d)\\,p_0(O_i)(1-p_0(O_i))O_i\\bigr]\\Bigr).\n\\]\n\n\\emph{Supplement A, p.~15.} Proposition 4. Define:\n\\[\n\\begin{aligned}\n\\widehat\\tau^{\\mathrm{L}}_{\\mathrm{IPW}}(d)={}&\\frac1N\\sum_{i=1}^{N}O_i'\\beta_{1,N}-\\frac1N\\sum_{i=1}^{N}O_i'\\beta_{0,N}\\\\\n&+\\frac1N\\sum_{i=1}^{n}\\frac{Z_i\\bigl(\\mu_i(\\mathbf Y;d)-O_i'\\beta_{1,N}\\bigr)}{p(O_i\\mid\\theta_0)}-\\frac1N\\sum_{i=1}^{n}\\frac{(1-Z_i)\\bigl(\\mu_i(\\mathbf Y;d)-O_i'\\beta_{0,N}\\bigr)}{1-p(O_i\\mid\\theta_0)}.\n\\end{aligned}\n\\]\nUnder C2, C3, C4, C6, and C1,\n\\[\n\\widehat\\tau^{\\mathrm{L}}_{\\mathrm{IPW}}(d)-\\widehat\\tau_{\\mathrm{IPW}}(d)=\\hl{o_p(N^{-\\frac12})}.\n\\]\n\n\\emph{Supplement A, proof, pp.~16--17, equation (43) and the next display.}\n\\[\nH^{-1}\\Bigl(\\frac1N\\sum_{i=1}^{N}\\frac{Z_i\\mu_i(\\mathbf Y;d)}{p(O_i\\mid\\theta_0)}\\,p(O_i\\mid\\theta_0)(1-p(O_i\\mid\\theta_0))O_i\\Bigr)-\\beta_{1,N}=o_p(1).\n\\]\nHence (42) becomes\n\\[\n\\frac1N\\sum_{i=1}^{N}\\frac{Z_i\\mu_i(\\mathbf Y;d)}{p(O_i\\mid\\widehat\\theta_{\\mathrm{MLE}})}=\\frac1N\\sum_{i=1}^{N}\\frac{Z_i\\mu_i(\\mathbf Y;d)}{p(O_i\\mid\\theta_0)}\\hl{-\\frac1N\\sum_{i=1}^{N}\\frac{Z_i-p(O_i\\mid\\theta_0)}{p(O_i\\mid\\theta_0)}O_i'\\beta_{1,N}}+o_p(N^{-\\frac12}).\n\\]\n\n\\emph{Supplement A, p.~14, condition C~2.} For all sample size $N$,\n\\[\n\\frac1N\\sum_{i=1}^{N}\\Bigl(\\tau_i(d;\\eta)-\\bigl(O_i'\\beta_{1,N}-O_i'\\beta_{0,N}\\bigr)\\Bigr)\\sum_{j\\in\\mathcal B(i;d)}\\Bigl(\\tau_j(d;\\eta)-\\bigl(\\hl{O_i'}\\beta_{1,N}-\\hl{O_i'}\\beta_{0,N}\\bigr)\\Bigr)\\ge0.\n\\]", "corrected": "\\emph{Condition C~1(iv).} $\\Theta_0\\subset\\mathbb R^k$ is a fixed compact set, and $\\theta_0$ is an \\hl{interior point} of $\\Theta_0$ (\\fid{P004-E003}).\n\n\\emph{Condition C~2.} For all sample sizes $N$, with $\\delta_i=O_i'(\\beta_{1,N}-\\beta_{0,N})$,\n\\[\n\\frac1N\\sum_{i=1}^{N}\\sum_{j\\in\\mathcal B(i;d)}\\bigl(\\tau_i(d;\\eta)-\\delta_i\\bigr)\\bigl(\\tau_j(d;\\eta)-\\hl{\\delta_j}\\bigr)\\ge0 .\n\\]\nHere $\\mathcal B(i;d)$ is the neighbourhood (12), which contains $i$ (see also \\fid{P004-E015}).\n\n\\emph{Proposition 4 of A.6.} Write $p_i=p(O_i\\mid\\theta_0)$, $m_{ia}=E[\\mu_i(a;d)]$ and $H_N=N^{-1}\\sum_ip_i(1-p_i)O_iO_i'$. Define\n\\[\n\\hl{\\gamma_N=H_N^{-1}\\,\\frac1N\\sum_{i=1}^{N}\\bigl[(1-p_i)m_{i1}+p_im_{i0}\\bigr]O_i},\\qquad L_i=O_i'\\gamma_N,\n\\]\n\\[\n\\hl{f_{i1}=p_i\\delta_i+p_i(1-p_i)L_i},\\qquad\\hl{f_{i0}=-(1-p_i)\\delta_i+p_i(1-p_i)L_i},\n\\]\n\\[\n\\widehat\\tau^{L}_{IPW}(d)=\\frac1N\\sum_{i=1}^{N}\\delta_i+\\frac1N\\sum_{i=1}^{N}T_i,\\qquad T_i=\\frac{Z_i\\bigl(\\mu_i(\\mathbf Y;d)-\\hl{f_{i1}}\\bigr)}{p_i}-\\frac{(1-Z_i)\\bigl(\\mu_i(\\mathbf Y;d)-\\hl{f_{i0}}\\bigr)}{1-p_i}.\n\\]\nUnder C2, C3, C4, C6 and C~1 with item (iv) as above,\n\\[\n\\widehat\\tau^{L}_{IPW}(d)-\\widehat\\tau_{IPW}(d)=o_p(N^{-\\frac12}).\n\\]\nThe variance $V_N=V(\\widehat\\tau^{L}_{IPW}(d))$ is\n\\[\n\\begin{aligned}\nV_N={}&\\frac{1}{N^2}\\sum_{i=1}^{N}\\Bigl\\{\\frac{E\\bigl[(\\mu_i(\\mathbf 1;d)-f_{i1})^2\\bigr]}{p_i}+\\frac{E\\bigl[(\\mu_i(\\mathbf 0;d)-f_{i0})^2\\bigr]}{1-p_i}\\Bigr\\}\\\\\n&+\\frac{1}{N^2}\\sum_{i;\\,j\\in\\mathcal B(i;d),\\,j\\ne i}\\ \\sum_{a=0}^{1}\\sum_{b=0}^{1}(-1)^{a+b}E\\bigl[\\bigl(\\mu_i(\\mathbf a_i,\\mathbf b_j;d)-f_{ia}\\bigr)\\bigl(\\mu_j(\\mathbf a_i,\\mathbf b_j;d)-f_{jb}\\bigr)\\bigr]\\\\\n&-\\frac{1}{N^2}\\sum_{i=1}^{N}\\sum_{j\\in\\mathcal B(i;d)}(\\tau_i-\\delta_i)(\\tau_j-\\delta_j).\n\\end{aligned}\n\\]\nIf \\hl{$N\\times V_N$} is uniformly bounded below by a positive constant for all large $N$, then\n\\[\n\\frac{\\widehat\\tau_{IPW}(d)-\\mathrm{AME}(d;\\eta)}{\\sqrt{V_N}}\\xrightarrow{d}N(0,1).\n\\]\nLet $\\widehat p_i=p(O_i\\mid\\widehat\\theta_{MLE})$, $\\widehat H=N^{-1}\\sum_i\\widehat p_i(1-\\widehat p_i)O_iO_i'$, and\n\\[\n\\widehat\\gamma=\\widehat H^{-1}\\,\\frac1N\\sum_{i=1}^{N}\\Bigl[\\frac{Z_i\\mu_i(\\mathbf Y;d)(1-\\widehat p_i)}{\\widehat p_i}+\\frac{(1-Z_i)\\mu_i(\\mathbf Y;d)\\,\\widehat p_i}{1-\\widehat p_i}\\Bigr]O_i .\n\\]\nLet $\\widehat\\beta_{1,N}$ and $\\widehat\\beta_{0,N}$ be as in footnote 9 of the supplement, $\\widehat\\delta_i=O_i'(\\widehat\\beta_{1,N}-\\widehat\\beta_{0,N})$ and $\\widehat L_i=O_i'\\widehat\\gamma$, and let $\\widehat f_{i1}$, $\\widehat f_{i0}$ and $\\widehat T_i$ be defined as $f_{i1}$, $f_{i0}$ and $T_i$ with $\\widehat p_i$, $\\widehat\\delta_i$ and $\\widehat L_i$ in place of $p_i$, $\\delta_i$ and $L_i$. Define\n\\[\n\\begin{aligned}\n\\widehat V^{obs}_{HAC}(d)&=\\frac{1}{N^2}\\sum_{i=1}^{N}\\sum_{j\\in\\mathcal B(i;d)}\\widehat T_i\\widehat T_j,\\\\\n\\widehat V^{obs}_{SAH}(d)&=\\frac{1}{N^2}\\sum_{i=1}^{N}c_i(d)\\Bigl[\\frac{Z_i\\bigl(\\mu_i(\\mathbf Y;d)-\\widehat f_{i1}\\bigr)^2}{\\widehat p_i^{\\,2}}+\\frac{(1-Z_i)\\bigl(\\mu_i(\\mathbf Y;d)-\\widehat f_{i0}\\bigr)^2}{(1-\\widehat p_i)^2}\\Bigr].\n\\end{aligned}\n\\]\nThen, for each $\\hl{0<}\\alpha<1/2$,\n\\[\n\\liminf_{N\\to\\infty}\\Pr\\Bigl(\\bigl|\\widehat\\tau_{IPW}(d)-\\mathrm{AME}(d;\\eta)\\bigr|\\le z_{1-\\alpha/2}\\sqrt{\\widehat V^{obs}_{SAH}(d)}\\Bigr)\\ge1-\\alpha,\n\\]\nand, additionally under C~2 as corrected above, the same inequality holds with $\\hl{\\max\\{\\widehat V^{obs}_{HAC}(d),0\\}}$ in place of $\\widehat V^{obs}_{SAH}(d)$.\nWhen \\hl{both assignment arms are nonempty}, the estimator $\\widehat V^{obs}_{HAC}(d)$ is the (2,2) entry of the sandwich printed on p.~15 when the residual there is taken to be\n\\[\n\\widehat\\epsilon_i=\\frac{N_1}{N}\\,\\frac{Z_i\\bigl(\\mu_i(\\mathbf Y;d)-\\widehat f_{i1}\\bigr)}{\\widehat p_i}+\\frac{N_0}{N}\\,\\frac{(1-Z_i)\\bigl(\\mu_i(\\mathbf Y;d)-\\widehat f_{i0}\\bigr)}{1-\\widehat p_i}.\n\\]\nIn $\\widehat V^{obs}_{SAH}(d)$ the residuals $\\mu_i(\\mathbf Y;d)-\\widehat f_{ia}$ are not weighted before they are squared, so that the inverse probability enters once, squared (see \\fid{P004-E002}).\n\n\\emph{Main text, Section 6.4, p.~759.} The sentence stands with this proposition in place of the printed one.", "evidence": "The evidence is an exact counterexample to (36), computed in rational arithmetic, and a derivation of the corrected statement.\n\n\\emph{The expansion of the IPW estimator.} For fixed outcomes and assignments,\n\\[\n\\nabla_\\theta\\,\\frac{Z_iR_i}{p(O_i\\mid\\theta)}=-\\frac{Z_iR_i\\,(1-p(O_i\\mid\\theta))}{p(O_i\\mid\\theta)}\\,O_i,\\qquad\\nabla_\\theta\\Bigl[-\\frac{(1-Z_i)R_i}{1-p(O_i\\mid\\theta)}\\Bigr]=-\\frac{(1-Z_i)R_i\\,p(O_i\\mid\\theta)}{1-p(O_i\\mid\\theta)}\\,O_i.\n\\]\nSince $Z_i$ is independent of the other assignments, I have $E[Z_iR_i]=p_im_{i1}$ and $E[(1-Z_i)R_i]=(1-p_i)m_{i0}$. The expected gradient of $\\widehat\\tau_{IPW}$ at $\\theta_0$ is therefore $-D_N$, where $D_N=N^{-1}\\sum_i[(1-p_i)m_{i1}+p_im_{i0}]O_i$. Under C2, C3, C4, C6 and C~1, the summands are bounded and each shares assignments with a bounded number of others, which implies that the gradient differs from $-D_N$ by $O_p(N^{-1/2})$. The second derivatives are bounded near $\\theta_0$. Inserting the expansion $\\widehat\\theta_{MLE}-\\theta_0=H_N^{-1}S_N+O_p(N^{-1})$ from the logistic score equation, with $\\theta_0$ an interior point (\\fid{P004-E003}), I obtain $\\widehat\\tau_{IPW}=\\widehat\\tau_{HT}-S_N'\\gamma_N+O_p(N^{-1})$ with $\\gamma_N=H_N^{-1}D_N$.\n\n\\emph{The counterexample.} Let $N=3K$. The nodes are the points $(10i,0)$, $i=1,\\dots,N$, and the outcome points are the same points, so that at $d=0$ the circle average of a node is the outcome at its own point. The nodes have the types $t=-1,0,1$, repeated in this order. A node of type $t$ has $O_t=(1,t)'$ and the outcome $q_t$ at its own point under every assignment, with $q_{-1}=1$, $q_0=21/20$ and $q_1=1$. The assignments are independent with $\\theta_0=(0,\\log2)'$ and $\\Theta_0=[-2,2]^2$. Then $p_t=2^t/(1+2^t)$, that is, $1/3$, $1/2$ and $2/3$. Every hypothesis of the proposition holds. The outcomes are bounded. There is no interference, and $\\mathcal B(i;0)=\\{i\\}$. The nodes are 10 apart, the probabilities lie in $(1/10,9/10)$, the Gram matrix of the confounders is $\\operatorname{diag}(1,2/3)$, and $\\theta_0$ is an interior point of $\\Theta_0$. Further, $\\tau_i=0$ and $\\beta_{1,N}=\\beta_{0,N}$, which implies that the sum in C~2 is 0, in the printed form and in the corrected one. The variance condition of the proposition also holds, since $V_P>0$ below. A direct computation gives\n\\[\nH_N=\\operatorname{diag}\\Bigl(\\frac{25}{108},\\frac{4}{27}\\Bigr),\\qquad\\beta_{1,N}=\\beta_{0,N}=\\Bigl(\\frac{509}{500},0\\Bigr)',\\qquad\\gamma_N=\\Bigl(\\frac{549}{125},0\\Bigr)'.\n\\]\nSince the two arms have the same outcome, both linear forms are averages over the nodes of multiples of $Z_i-p_i$. At a node of type $t$, the first-order form of the IPW estimator has the coefficient $a_t$ and the printed linear form has the coefficient $b_t$, where\n\\[\na_t=\\frac{q_t}{p_t(1-p_t)}-O_t'\\gamma_N,\\qquad b_t=\\frac{q_t-O_t'\\beta_{1,N}}{p_t(1-p_t)}.\n\\]\n\\begin{audittable}{crrrr}{The three types of node and the coefficient of $Z_i-p_i$ in the two linear forms.}\nType $t$ & $p_t$ & $q_t$ & IPW estimator, $a_t$ & Printed linear form, $b_t$ \\\\\n$-1$ & $1/3$ & $1$ & $27/250$ & $-81/1000$ \\\\\n$0$ & $1/2$ & $21/20$ & $-24/125$ & $16/125$ \\\\\n$1$ & $2/3$ & $1$ & $27/250$ & $-81/1000$ \\\\\n\\end{audittable}\nNote that the two sets of coefficients have opposite signs. Writing $w_t=p_t(1-p_t)$, I find that $N$ times the variances of the two linear forms and their difference are\n\\[\nV_A=\\frac13\\sum_tw_ta_t^2=\\frac{3}{625},\\qquad V_P=\\frac13\\sum_tw_tb_t^2=\\frac{1753}{750000},\\qquad V_D=\\frac13\\sum_tw_t(a_t-b_t)^2=\\frac{10369}{750000}\n\\]\nfor the first-order form of the IPW estimator, for the printed linear form, and for their difference, respectively. By the expansion above, $\\sqrt N(\\widehat\\tau_{IPW}-\\widehat\\tau^{L}_{IPW})=U_K+o_p(1)$, where $U_K$ is a normalized sum of $K$ independent and identically distributed terms, one for each block of three nodes. Its second moment is $V_D$ and its fourth moment is at most $2205448981/2250000000000$ for every $K$. By the Cauchy--Schwarz inequality, I have\n\\[\n\\Pr\\bigl(U_K^2>V_D/2\\bigr)\\ge\\frac{107516161}{2205448981}\\quad\\text{for every }K\\ge1.\n\\]\nHence the difference in (36) is not $o_p(N^{-1/2})$. Further, $\\sqrt N\\,\\widehat\\tau_{IPW}$ converges in distribution to a normal variable with variance $3/625$, and the AME is 0. Then the ratio that Proposition 4 standardizes by the printed variance has limiting variance $V_A/V_P=3600/1753$, not 1. In this example the printed HAC estimator, multiplied by $N$, converges to $V_P$. A normal interval with critical value $z$ then has limiting coverage $2\\Phi(z\\sqrt{1753/3600})-1$, which is about $0.83$ at the nominal level $0.95$ and about $0.75$ at the nominal level $0.90$.\n\n\\emph{The corrected linear form.} With $L_i=O_i'\\gamma_N$, the fits $f_{i1}$ and $f_{i0}$ of the corrected statement satisfy $f_{i1}-f_{i0}=\\delta_i$ and $f_{i1}/p_i+f_{i0}/(1-p_i)=L_i$. For every assignment,\n\\[\nf_{i1}-f_{i0}-\\frac{Z_if_{i1}}{p_i}+\\frac{(1-Z_i)f_{i0}}{1-p_i}=-(Z_i-p_i)\\Bigl[\\frac{f_{i1}}{p_i}+\\frac{f_{i0}}{1-p_i}\\Bigr]=-(Z_i-p_i)L_i.\n\\]\nSumming over $i$, I obtain $N^{-1}\\sum_i\\delta_i+N^{-1}\\sum_iT_i=\\widehat\\tau_{HT}-S_N'\\gamma_N$ as an identity, with $T_i$ as in the corrected statement, and $E[T_i]=\\tau_i-\\delta_i$. The corrected form of (36) then follows from the expansion of the IPW estimator.\n\n\\emph{The variance estimators.} Let $V_N$ be the variance of $N^{-1}\\sum_iT_i$, and let $W$ be the sum of the products $T_iT_j$ over $i$ and $j\\in\\mathcal B(i;d)$, divided by $N^2$. By C3, the covariance of $T_i$ and $T_j$ vanishes when $j\\notin\\mathcal B(i;d)$, which implies that\n\\[\nE[W]=V_N+\\frac{1}{N^2}\\sum_{i=1}^{N}\\sum_{j\\in\\mathcal B(i;d)}(\\tau_i-\\delta_i)(\\tau_j-\\delta_j).\n\\]\nThe last sum is nonnegative under the corrected C~2, which is the homophily condition of the paper with $O_j$ in its second factor. For the SAH estimator, the inequality $\\operatorname{Cov}(T_i,T_j)\\le(\\operatorname{Var}T_i+\\operatorname{Var}T_j)/2$ and the symmetry of the neighbourhoods give $\\operatorname{Var}(\\sum_iT_i)\\le\\sum_ic_i(d)E[T_i^2]$, with no homophily condition, and $T_i^2$ is the bracket in the corrected SAH estimator. The $T_i$ are bounded, and each depends on the assignments of a bounded number of nodes. By Chebyshev's inequality, $N$ times each of the two sums concentrates around its expectation. The estimated fits $\\widehat f_{ia}$ are consistent uniformly in $i$, since $\\widehat\\theta_{MLE}$, $\\widehat\\gamma$ and $\\widehat\\beta_{a,N}$ have errors of order $N^{-1/2}$. Then replacing $T_i$ by $\\widehat T_i$ changes $N$ times each sum by $o_p(1)$. Finally, a sum of bounded variables whose dependency graph has bounded degree is asymptotically normal when its variance is of order $N$, which is the condition that $NV_N$ is bounded below.\n\n\\emph{Three remarks on the correction.} First, the index in C~2 matters. In a second exact example, with three types on a path, the printed sum of C~2 equals $7/339>0$ and the corrected sum equals $-775/38307<0$. The corrected HAC estimator, multiplied by $N$, then converges to $6053/153228$, below the value $27/452$ of $NV_N$. Second, a positive printed variance does not imply that $V_N$ is positive. With $q=(1,9/8,1)$ in the counterexample, the first-order form of the IPW estimator vanishes identically, while $N$ times the printed variance is $1753/120000$. Third, the fits are not unique. The fits $g_{i1}=p_iO_i'\\gamma_1$ and $g_{i0}=(1-p_i)O_i'\\gamma_0$, with $\\gamma_1=H_N^{-1}N^{-1}\\sum_i(1-p_i)m_{i1}O_i$ and $\\gamma_0=H_N^{-1}N^{-1}\\sum_ip_im_{i0}O_i$, yield the same linear form. With them, however, the bound for the HAC estimator needs homophily of $\\tau_i-g_{i1}+g_{i0}$, which C~2 does not imply: in a third example the sum for $g$ equals $-9/2000$ while the sum of C~2 is 0. The fits $f$ avoid this additional condition.\n\n\\emph{What is not established.} Equation (36) does not fail in every design with unequal probabilities. With two types of node, a saturated logistic model and outcomes that depend only on the type and the arm, (36) holds although the probabilities differ. No published figure, estimate or interval of the observational application has been shown to change, and the corrected conditions have not been checked for that application.", "problem": "Throughout, $d$ is fixed, and I write $R_i=\\mu_i(\\mathbf Y;d)$, $p_i=p(O_i\\mid\\theta_0)$, $m_{ia}=E[\\mu_i(a;d)]$ for $a\\in\\{0,1\\}$, $\\tau_i=m_{i1}-m_{i0}$ and $\\delta_i=O_i'(\\beta_{1,N}-\\beta_{0,N})$. Let\n\\[\nH_N=\\frac1N\\sum_{i=1}^{N}p_i(1-p_i)O_iO_i',\\qquad S_N=\\frac1N\\sum_{i=1}^{N}(Z_i-p_i)O_i,\\qquad\\widehat\\tau_{HT}=\\frac1N\\sum_{i=1}^{N}\\Bigl[\\frac{Z_iR_i}{p_i}-\\frac{(1-Z_i)R_i}{1-p_i}\\Bigr],\n\\]\nso that $\\beta_{a,N}=H_N^{-1}N^{-1}\\sum_ip_i(1-p_i)m_{ia}O_i$. The upper limits $n$ in the two residual sums of the proposition are read as $N$, and ``C1'' in its hypotheses is the condition C~1 of A.6 (see \\fid{P004-N016}).\n\nThe first-order term of the expansion (42) has the coefficient $N^{-1}\\sum_iZ_iR_i(1-p_i)O_i'/p_i$, whose expectation is $N^{-1}\\sum_i(1-p_i)m_{i1}O_i'$. Equation (43) concerns a different sum, in which $Z_iR_i/p_i$ is multiplied by $p_i(1-p_i)O_i$ and not by $(1-p_i)O_i$, and it is this sum that converges to $H_N\\beta_{1,N}$. The display after ``Hence (42) becomes'' puts $\\beta_{1,N}$ in place of the coefficient of (42) and moves a factor $1/p_i$ into the score sum. Lemma A.6 expresses $\\widehat\\theta_{MLE}-\\theta_0$ through $H_N^{-1}S_N$, in which every node has the same weight, so no factor $1/p_i$ can arise there. The two expressions agree when all $p_i$ are equal and differ in general. Expanding the definition, I find that the printed linear form is\n\\[\n\\widehat\\tau^{L}_{IPW}=\\widehat\\tau_{HT}-\\frac1N\\sum_{i=1}^{N}(Z_i-p_i)\\Bigl[\\frac{O_i'\\beta_{1,N}}{p_i}+\\frac{O_i'\\beta_{0,N}}{1-p_i}\\Bigr],\n\\]\nwhereas the IPW estimator satisfies $\\widehat\\tau_{IPW}=\\widehat\\tau_{HT}-S_N'\\gamma_N+O_p(N^{-1})$ with $\\gamma_N$ as in the headline. In general the two differ at order $N^{-1/2}$, and then (36) fails. The variance calculation (37)--(40) is based on the printed linear form rather than the correct first-order form of the IPW estimator; the HAC and SAH estimators built on the residuals $R_i-O_i'\\widehat\\beta_{a,N}$ can estimate the wrong quantity, and the normal limit and the two coverage statements for the IPW estimator do not follow. As printed, Appendix A.6 does not supply the inferential theory that Section 6.4 announces.", "what_survives": "The definition of the AME, its identification under independent assignment, and the Horvitz--Thompson estimator with known probabilities (with the weight corrected in \\fid{P004-E004}) are not affected. The theory of Sections 4 and 5 for experiments with a constant probability does not use Proposition 4 of A.6. The IPW estimator remains consistent for the AME. The defect concerns its first-order approximation and the variance derived from it. The printed expansion is correct when the logistic model has an intercept only, since all probabilities are then equal. What changes is the inference for the IPW estimator with a fitted logistic model: the linear form, its variance, the two variance estimators and the conditions are replaced as below. The correction keeps the estimand, the estimator and the homophily condition in its intended form. It adds two regularity conditions, an interior $\\theta_0$ and a positive lower bound on $N$ times the corrected variance."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E001", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the line that defines epsilon-hat_i as N_1 over N times Z_i over p(O_j | theta-hat_MLE) times a treated residual, plus N_0 over N times (1 minus Z_i) over (1 minus p(O_j | theta-hat_MLE)) times a control residual; the sentence 'The formula is similar to (6), except now that the residuals are weighted with the inverse propensity scores'; and the display of V-hat-obs-SAH(d). Red boxes mark the denominators p squared (O_i | theta-hat_MLE) and (1 minus p(O_i | theta-hat_MLE)) squared of the two sums.", "caption": "Supplement A, p. 15, the residual $\\widehat\\epsilon_i$ of A.6 and the observational SAH estimator. The residual in the first line already carries the inverse propensity weights; the two marked denominators divide its square by the squared propensities again.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E002.png"}, {"alt": "Bar chart with three groups, N = 32, N = 64 and the limit as N tends to infinity. The vertical axis is the expected variance estimate divided by the true variance, from 0 to 2.3, with a horizontal line at 1 labelled 'true variance'. Orange bars, as printed: 0.21, 0.30 and 0.32, all far below the line. Blue bars, corrected: 1.07, 1.58 and 2.00, all above the line.", "caption": "The observational SAH estimator, as printed and as corrected, in the example described under Evidence. Each bar is the expected value of the estimator divided by the variance of $\\widehat\\tau_{\\mathrm{IPW}}$, computed exactly at $N=32$ and $N=64$; the third group is the ratio of the limits of $N$ times each quantity. As printed: $0.21$, $0.30$, $0.32$. Corrected: $1.07$, $1.58$, $2.00$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E002-variance.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T3", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T3"}, {"id": "P004-T4", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T4"}, {"id": "P004-T36", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T36"}], "headline": "The observational SAH estimator divides the square of an inverse-propensity-weighted residual by the squared propensity a second time; the corrected estimator is $N_1^{-2}\\sum_iZ_ic_i(d)\\widehat\\epsilon_i^2+N_0^{-2}\\sum_i(1-Z_i)c_i(d)\\widehat\\epsilon_i^2$.", "id": "P004-E002", "location": "Supplement A, A.6, p. 15 (supplement PDF p. 15): the residual $\\widehat\\epsilon_i$ and the display of $\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}(d)$", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P004-T3", "P004-T36", "P004-T4"], "text": {"as_printed": "``where $\\widehat\\epsilon_i=\\frac{N_1}{N}\\frac{Z_i}{p(\\hl{O_j}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{1,N}\\big)+\\frac{N_0}{N}\\frac{1-Z_i}{1-p(\\hl{O_j}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{0,N}\\big)$.$^{9}$ The formula is similar to (6), except now that the residuals are weighted with the inverse propensity scores. Note that the covariates $O_i$ do not appear in the matrix $\\mathbf X$. Similarly, we can similarly define a Sävje, Aronow and Hudgens (2018) type variance estimator as in (20):\n\\[\n\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}(d)=\\frac1{N^2}\\sum_{i=1}^N\\frac{Z_ic_i(d)\\widehat\\epsilon_i^2}{\\hl{p^2(O_i|\\widehat\\theta_{\\mathrm{MLE}})}}+\\frac1{N^2}\\sum_{i=1}^N\\frac{(1-Z_i)\\,c_i(d)\\widehat\\epsilon_i^2}{\\hl{\\big(1-p(O_i|\\widehat\\theta_{\\mathrm{MLE}})\\big)^2}},\\text{''}\n\\]", "corrected": "``where $\\widehat\\epsilon_i=\\frac{N_1}{N}\\frac{Z_i}{p(\\hl{O_i}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{1,N}\\big)+\\frac{N_0}{N}\\frac{1-Z_i}{1-p(\\hl{O_i}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{0,N}\\big)$. \\ldots\\ Similarly, we can similarly define a Sävje, Aronow and Hudgens (2018) type variance estimator as in (20):\n\\[\n\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}(d)=\\hl{\\frac1{N_1^2}}\\sum_{i=1}^NZ_ic_i(d)\\widehat\\epsilon_i^2+\\hl{\\frac1{N_0^2}}\\sum_{i=1}^N(1-Z_i)\\,c_i(d)\\widehat\\epsilon_i^2,\\text{''}\n\\]\nwhere $N_1=\\sum_{i=1}^NZ_i$ and $N_0=N-N_1$, and the entire contribution from an empty arm is set to $0$. Equivalently, with the unweighted residuals $\\widehat r_{ia}=\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{a,N}$ and $\\widehat p_i=p(O_i|\\widehat\\theta_{\\mathrm{MLE}})$,\n\\[\n\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}(d)=\\frac1{N^2}\\sum_{i=1}^Nc_i(d)\\left\\{\\frac{Z_i\\widehat r_{i1}^2}{\\widehat p_i^{\\,2}}+\\frac{(1-Z_i)\\widehat r_{i0}^2}{(1-\\widehat p_i)^2}\\right\\}.\n\\]\nThe two forms agree whenever both arms are nonempty. These displays isolate the double weighting. For the corrected Proposition 4 of A.6, use the fitted values $\\widehat f_{ia}$ of \\fid{P004-E001} in place of $O_i'\\widehat\\beta_{a,N}$ in the residuals.", "evidence": "The evidence is an exact counterexample. Let $N$ be a multiple of $8$. Under independent assignment, a quarter of the nodes have propensity $1/2$ and treated circle averages $a_i=+1$ or $-1$, in equal numbers; the others have propensity $1/10$ and treated circle average $0$. All control circle averages are $0$, there is no interference at $d=0$, and $c_i(0)=1$. The covariates are the two group indicators, for which the logistic model holds, and the parameter space is the compact set on which the fitted propensities lie in $[1/20,19/20]$. The AME equals $0$ and the coefficients $\\beta_{1,N}$ and $\\beta_{0,N}$ equal $0$. Let $S$ be the sum of $a_iZ_i$ over the first group and $\\widehat p_H$ the fitted propensity of that group. Then $\\widehat\\tau_{\\mathrm{IPW}}=S/(N\\widehat p_H)$. A direct computation gives $\\widehat\\tau_{\\mathrm{IPW}}-2S/N=O_p(N^{-1})$ and $N\\operatorname{Var}(\\widehat\\tau_{\\mathrm{IPW}})\\to1/4$. The printed estimator satisfies $N\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}\\to2/25$ in probability, a ratio of $8/25$ to the variance. For the corrected estimator the limit is $1/2$. By an exact calculation over the assignment distribution at $N=32$, the variance of $\\widehat\\tau_{\\mathrm{IPW}}$ is $0.01212$, the expectation of the printed estimator is $0.00256$, and that of the corrected estimator is $0.01298$. At $N=64$ the three values are $0.00483$, $0.00145$ and $0.00764$. The nominal $95\\%$ interval built on the printed estimator has limiting coverage $2\\Phi(1.96\\sqrt{8/25})-1\\approx0.73$.", "problem": "The residual $\\widehat\\epsilon_i$ defined above the display already contains the inverse propensity weight. I write $\\widehat p_i=p(O_i|\\widehat\\theta_{\\mathrm{MLE}})$ and $\\widehat r_{ia}=\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{a,N}$, and I read the unbound $O_j$ of the residual as $O_i$ (see \\fid{P004-E015}). For a treated node, $\\widehat\\epsilon_i=(N_1/N)\\,\\widehat r_{i1}/\\widehat p_i$. The SAH display divides $\\widehat\\epsilon_i^2$ by $\\widehat p_i^2$ again. Substituting the residual into the display, I obtain\n\\[\n\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}(d)=\\frac1{N^2}\\sum_{i=1}^Nc_i(d)\\left[\\Big(\\frac{N_1}{N}\\Big)^2\\frac{Z_i\\widehat r_{i1}^2}{\\widehat p_i^{\\,4}}+\\Big(\\frac{N_0}{N}\\Big)^2\\frac{(1-Z_i)\\widehat r_{i0}^2}{(1-\\widehat p_i)^4}\\right],\n\\]\nwhereas the analogue of (20) has $Z_i\\widehat r_{i1}^2/\\widehat p_i^{\\,2}$ and $(1-Z_i)\\widehat r_{i0}^2/(1-\\widehat p_i)^2$ in the brackets. The two differ by the factors $(N_1/(N\\widehat p_i))^2$ and $(N_0/(N(1-\\widehat p_i)))^2$. These factors equal $1$ for an intercept-only logistic model with an interior maximum-likelihood estimate, where $\\widehat p_i=N_1/N$, and they can be far below $1$ when the propensities vary across nodes. The example below shows that the printed estimator need not be a conservative bound for the variance, and the intended IPW interval of Proposition 4(i) of A.6 can cover less often than its nominal level. A.6 defines no unweighted residual for the SAH display, and the page offers no other reading of $\\widehat\\epsilon_i$ there.", "what_survives": "The experimental SAH estimator (20) of the article, which is (24) of Supplement A, uses unweighted residuals with a common $p$; it is a different formula and is unaffected. The experimental simulations, the AME, the point estimators and the randomization tests are also unaffected by this error, and the observational HAC display is not touched by this point. The forest application of Section 8.2 reports HAC intervals, not observational SAH intervals; this error does not establish that any reported interval changes. What changes is the SAH display of A.6 and, with it, the intended IPW interval of Proposition 4(i) there. The corrected display removes the double weighting and adds no assumption. The general interval also requires the corrected linearization and fitted values in \\fid{P004-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E002", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of items (iv), (v) and (vi) of a condition and of Lemma A.6. Item (iv) reads: Compact Parameter Space: theta-zero in Theta-zero where Theta-zero, a subset of R to the k, is a compact set. Item (vi) defines theta-hat-MLE as the maximizer over theta in Theta-zero of the logistic log-likelihood. Lemma A.6 reads: Under C6 and C1, theta-hat-MLE minus theta-zero is o-p of 1, and theta-hat-MLE minus theta-zero equals the inverse of the average of p times 1 minus p times O sub i O sub i prime, times the average of Z sub i minus p times O sub i, plus o-p of N to the minus one half. In particular, theta-hat-MLE minus theta-zero is O-p of N to the minus one half. One red outline surrounds theta-zero in Theta-zero in item (iv). A second surrounds the term plus o-p of N to the minus one half.", "caption": "Supplement A, condition C 1, items (iv) to (vi), and Lemma A.6, p. 14. The outlines mark the condition on $\\theta_0$ and the remainder of the expansion.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T9", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T9"}, {"id": "P004-T10", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T10"}], "headline": "Lemma A.6 needs the true coefficient $\\theta_0$ to be an interior point of the compact set $\\Theta_0$ over which the likelihood is maximized; when $\\theta_0$ lies on the boundary of $\\Theta_0$, the expansion of the estimator fails.", "id": "P004-E003", "location": "Supplement A, Appendix A.6, condition C 1, items (iv) and (vi), and Lemma A.6, p. 14 (C 1 begins on p. 13)", "pdf_page": 14, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T10", "P004-T9"], "text": {"as_printed": "(iv) Compact Parameter Space: $\\hl{\\theta_0\\in\\Theta_0}$ where $\\Theta_0\\subset\\mathbb R^k$ is a compact set.\n\n(vi) The MLE estimator is used to estimate the coefficient vector $\\theta$:\n\\[\n\\widehat\\theta_{MLE}=\\arg\\max_{\\theta\\in\\Theta_0}\\sum_{i=1}^{N}\\bigl(Z_iO_i'\\theta-\\log(1+\\exp(O_i'\\theta))\\bigr).\n\\]\nLemma A.6. Under C6 and C1, $\\widehat\\theta_{MLE}-\\theta_0=o_p(1)$ and\n\\[\n\\widehat\\theta_{MLE}-\\theta_0=\\Bigl(\\frac1N\\sum_{i=1}^{N}p(O_i\\mid\\theta_0)(1-p(O_i\\mid\\theta_0))O_iO_i'\\Bigr)^{-1}\\frac1N\\sum_{i=1}^{N}\\bigl((Z_i-p(O_i\\mid\\theta_0))O_i\\bigr)\\hl{+o_p(N^{-\\frac12})}.\n\\]\nIn particular, $\\widehat\\theta_{MLE}-\\theta_0=O_p(N^{-\\frac12})$.", "corrected": "(iv) Compact Parameter Space: \\hl{$\\theta_0$ is an interior point of $\\Theta_0$}, where $\\Theta_0\\subset\\mathbb R^k$ is a fixed compact set. That is, there exists a fixed $r>0$ such that the ball $B(\\theta_0,r)$ is contained in $\\Theta_0$.\n\nLemma A.6. Under C6 and C~1 with item (iv) as above, $\\widehat\\theta_{MLE}-\\theta_0=o_p(1)$ and\n\\[\n\\widehat\\theta_{MLE}-\\theta_0=\\Bigl(\\frac1N\\sum_{i=1}^{N}p(O_i\\mid\\theta_0)(1-p(O_i\\mid\\theta_0))O_iO_i'\\Bigr)^{-1}\\frac1N\\sum_{i=1}^{N}(Z_i-p(O_i\\mid\\theta_0))O_i+o_p(N^{-\\frac12}).\n\\]\nIn particular, $\\widehat\\theta_{MLE}-\\theta_0=O_p(N^{-\\frac12})$. The remainder in the display is $O_p(N^{-1})$.", "evidence": "The evidence is an exact counterexample and a derivation of the lemma under the corrected condition.\n\n\\emph{Counterexample.} Let $k=1$, $O_i=1$, $\\Theta_0=[0,1]$ and $\\theta_0=0$, and let the $Z_i$ be independent with probability $1/2$. Every item of C~1 holds (with bound 1 on the confounder and Gram matrix 1), and C6 holds with $\\epsilon=1/4$. Let $T=\\sum_i(2Z_i-1)$. The matrix of the lemma equals $1/4$, and the leading term is $2T/N$. The derivative of the log-likelihood at 0 is $T/2$, and the log-likelihood is strictly concave. If $T<0$, it is decreasing on $[0,1]$ and $\\widehat\\theta_{MLE}=0$ exactly. On this event the remainder of the lemma is $-2T/N$, and $\\sqrt N$ times its absolute value exceeds 1 when in addition $T^2>N/4$. Since $E[T^2]=N$ and $E[T^4]=3N^2-2N$, the Cauchy--Schwarz inequality implies that\n\\[\n\\Pr\\bigl(T^2>N/4\\bigr)\\ge\\frac{(3N/4)^2}{3N^2-2N}\\ge\\frac{3}{16},\n\\]\nand the distribution of $T$ is symmetric. I conclude that\n\\[\n\\Pr\\Bigl(\\sqrt N\\,\\bigl|\\widehat\\theta_{MLE}-\\theta_0-2T/N\\bigr|>1\\Bigr)\\ge\\frac{3}{32}\\quad\\text{for every }N\\ge1,\n\\]\nso that the remainder is not $o_p(N^{-1/2})$. The limit of $\\sqrt N\\,\\widehat\\theta_{MLE}$ is $2\\max\\{U,0\\}$ with $U$ standard normal, and the leading term has the limit $2U$.\n\n\\emph{The lemma for an interior $\\theta_0$.} Let $g_N=N^{-1}\\sum_i(Z_i-p_i)O_i$ and $\\delta=\\widehat\\theta_{MLE}-\\theta_0$, and let $\\ell_N$ be the log-likelihood divided by $N$. Under C~1 and C6, I have $g_N=O_p(N^{-1/2})$, and there exists a constant $c>0$ such that the negative Hessian of $\\ell_N$ is at least $c$ times the identity on every segment between two points of $\\Theta_0$. Since $\\theta_0\\in\\Theta_0$, I obtain\n\\[\n0\\le\\ell_N(\\widehat\\theta_{MLE})-\\ell_N(\\theta_0)\\le g_N'\\delta-\\frac c2\\|\\delta\\|^2,\n\\]\nwhich implies that $\\|\\delta\\|\\le2\\|g_N\\|/c=O_p(N^{-1/2})$. This part uses the printed conditions only. If the ball $B(\\theta_0,r)$ lies in $\\Theta_0$, then $\\|\\delta\\|<r/2$ with probability tending to one. On that event the maximizer is an interior point and the score vanishes there, so that $0=g_N-H_N\\delta+e_N$ with $\\|e_N\\|\\le C\\|\\delta\\|^2$, where $H_N$ is the matrix of the lemma. Since $H_N^{-1}$ is bounded, I find $\\delta=H_N^{-1}g_N+O_p(N^{-1})$.", "problem": "The estimator is defined as the maximizer of the likelihood over $\\Theta_0$, and item (iv) only places $\\theta_0$ in $\\Theta_0$. A value of $\\theta_0$ on the boundary of $\\Theta_0$ is therefore allowed. There the constrained maximizer need not solve the score equation, and the expansion of Lemma A.6, which is the expansion of an unconstrained maximizer, fails. Condition C6 concerns independence and overlap and does not restrict $\\Theta_0$.", "what_survives": "The first and the last assertion of Lemma A.6, consistency and the rate $O_p(N^{-1/2})$, hold under the printed conditions, boundary included. The results for experiments with a known assignment probability do not use the lemma. The script of the observational application fits the logistic model without a constraint on the coefficient. This example does not establish any change to its reported numbers. What depends on the corrected condition is the expansion of Lemma A.6 and the results of A.6 that use it. If fixed outcomes are added to the boundary example, equation (36) and the normal limit of Proposition 4 of A.6 fail as well, although $N$ times the variance of the printed linear form equals 4. That proposition has a second and separate defect, treated in \\fid{P004-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E003", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of three lines of text, a displayed formula and one more line. The text ends: Under C6 the Horvitz-Thompson estimator with known propensity scores, defined as. The formula reads: tau-hat-obs sub HT of d equals one over N times the sum over i of Z sub i over p of O sub i times mu sub i of Y and d, minus one over N times the sum over i of 1 minus Z sub i over p of O sub i times mu sub i of Y and d. The line below reads: is unbiased for AME of d and eta. A red outline surrounds the denominator p of O sub i of the second fraction.", "caption": "Section 6.4, p. 759 (PDF p. 16). The outline marks the denominator of the control term, printed as $p(O_i)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E004.png"}, {"alt": "Crop of three displayed lines and one line of text. The first two lines carry, for the control arm, the fraction 1 minus p of O sub i over 1 minus p of O sub i. The third line reads: equals E of one over N times the sum of Z sub i over p of O sub i times mu sub i of Y and d, minus one over N times the sum of 1 minus Z sub i over p of O sub i times mu sub i of Y and d. A red outline surrounds the denominator p of O sub i of the second fraction in the third line. The text below reads: The fourth equality uses C6-(ii) and the law of total expectations.", "caption": "Supplement A, A.1.3, p. 4, last three lines of the display. The line before the last has $1-p(O_i)$ for the control arm; the outline marks the denominator $p(O_i)$ in the last line.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T11", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T11"}], "headline": "In the Horvitz–Thompson estimator for observational studies, the control observations are weighted by $1/(1-p(O_i))$, not by $1/p(O_i)$.", "id": "P004-E004", "location": "Section 6.4, display that defines $\\widehat\\tau^{obs}_{HT}(d)$, p. 759 (PDF p. 16); Supplement A, A.1.3, last line of the display, p. 4", "pdf_page": 16, "printed_page": "759", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T11"], "text": {"as_printed": "\\emph{Main text, p.~759.} Under C6 the Horvitz--Thompson estimator with known propensity scores, defined as\n\\[\n\\widehat\\tau^{\\mathrm{obs}}_{\\mathrm{HT}}(d)=\\frac1N\\sum_{i=1}^{N}\\frac{Z_i}{p(O_i)}\\mu_i(\\mathbf Y;d)-\\frac1N\\sum_{i=1}^{N}\\frac{1-Z_i}{\\hl{p(O_i)}}\\mu_i(\\mathbf Y;d),\n\\]\nis unbiased for $\\mathrm{AME}(d;\\eta)$.\n\n\\emph{Supplement A, p.~4, last line of the display.}\n\\[\n=\\mathrm E\\Bigl[\\frac1N\\sum_{i=1}^{N}\\frac{Z_i}{p(O_i)}\\mu_i(\\mathbf Y;d)-\\frac1N\\sum_{i=1}^{N}\\frac{1-Z_i}{\\hl{p(O_i)}}\\mu_i(\\mathbf Y;d)\\Bigr]\n\\]", "corrected": "Under C6 the Horvitz--Thompson estimator with known propensity scores, defined as\n\\[\n\\widehat\\tau^{obs}_{HT}(d)=\\frac1N\\sum_{i=1}^{N}\\frac{Z_i}{p(O_i)}\\mu_i(\\mathbf Y;d)-\\frac1N\\sum_{i=1}^{N}\\frac{1-Z_i}{\\hl{1-p(O_i)}}\\mu_i(\\mathbf Y;d),\n\\]\nis unbiased for $\\mathrm{AME}(d;\\eta)$. The last line of the display in Supplement A.1.3 (p.~4) has the same denominator $1-p(O_i)$ in its second sum.", "evidence": "The evidence is a derivation. Let $p_i=p(O_i)$ and $m_{ia}=E[\\mu_i(\\mathbf Y;d)\\mid Z_i=a]$, which is $\\mu_i(a;d,\\eta)$ in the notation of A.1.3. Then $\\mathrm{AME}(d;\\eta)=N^{-1}\\sum_i(m_{i1}-m_{i0})$. Under C6, I have $0<p_i<1$ and\n\\[\nE\\Bigl[\\frac{Z_i\\mu_i(\\mathbf Y;d)}{p_i}\\Bigr]=m_{i1},\\qquad E\\Bigl[\\frac{(1-Z_i)\\mu_i(\\mathbf Y;d)}{p_i}\\Bigr]=\\frac{(1-p_i)m_{i0}}{p_i},\\qquad E\\Bigl[\\frac{(1-Z_i)\\mu_i(\\mathbf Y;d)}{1-p_i}\\Bigr]=m_{i0}.\n\\]\nThe printed estimator has expectation $N^{-1}\\sum_i[m_{i1}-(1-p_i)m_{i0}/p_i]$, and the corrected one has expectation $\\mathrm{AME}(d;\\eta)$. For example, if $\\mu_i(\\mathbf Y;d)=1$ for every assignment and $p_i=1/4$, then the AME is 0, the printed estimator has expectation $1-3=-2$, and the corrected one has expectation 0.", "problem": "As printed, the control observations are weighted by the inverse of the probability of treatment. The estimator so defined is not unbiased for the AME in general. The line before the last one in the proof of A.1.3 has the factor $(1-p(O_i))/(1-p(O_i))$ for the control arm, and the IPW estimator of A.6 (Supplement A, p.~14) has the denominator $1-p(O_i\\mid\\widehat\\theta_{MLE})$. The denominator $p(O_i)$ in the control sum is a misprint that occurs in both places.", "what_survives": "The corrected estimator is unbiased under C6, and the proof in A.1.3 is correct up to its last line. The IPW estimator of A.6 is defined with the correct weight. The inference for that estimator is a separate matter, treated in \\fid{P004-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E004", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of text from Section 6.3. The fourth and fifth lines read: under the sharp null. As a result, rejecting the null if one over P times the sum over p of the indicator that the absolute value of T of Y and Z sub p is at least T of Y and Z, is at most alpha, for some fixed large enough P gives an alpha-level exact test of the sharp null hypothesis. One red outline surrounds T of Y and Z on the right of the inequality inside the indicator. A second surrounds the words alpha-level exact test.", "caption": "Section 6.3, second paragraph, p. 758 (PDF p. 15). The outlines mark the observed statistic, which is printed without an absolute value, and the words “$\\alpha$-level exact test”.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T12", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T12"}, {"id": "P004-T13", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T13"}], "headline": "The Monte Carlo rule compares $|T(\\mathbf Y,\\mathbf Z_p)|$ with $|T(\\mathbf Y,\\mathbf Z)|$, and it has level $\\alpha$ for every number $P$ of draws when the observed assignment is counted, that is, when one rejects if $(1+B)/(P+1)\\le\\alpha$, where $B$ is the number of draws whose statistic is at least as extreme as the observed one.", "id": "P004-E005", "location": "Section 6.3, second paragraph, p. 758 (PDF p. 15)", "pdf_page": 15, "printed_page": "758", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [{"id": "P008-E002", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E002", "work_id": "P008"}], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T12", "P004-T13"], "text": {"as_printed": "As a result, rejecting the null if $\\frac{1}{P}\\sum_p \\mathbf 1\\{|T(\\mathbf Y,\\mathbf Z_p)|\\ge \\hl{T(\\mathbf Y,\\mathbf Z)}\\}\\le\\alpha$ for some fixed large enough $P$ gives an \\hl{$\\alpha$-level exact test} of the sharp null hypothesis.", "corrected": "Under the sharp null hypothesis, draw $\\mathbf Z_1,\\dots,\\mathbf Z_P$ independently from the assignment distribution of the experiment, independently of the observed assignment $\\mathbf Z$. Reject the null if\n\\[\n\\frac{\\hl{1+}\\sum_{p=1}^{P}\\mathbf 1\\{|T(\\mathbf Y,\\mathbf Z_p)|\\ge \\hl{|T(\\mathbf Y,\\mathbf Z)|}\\}}{\\hl{P+1}}\\le\\alpha .\n\\]\nFor every $P\\ge1$ this is a test of the sharp null hypothesis whose size is at most $\\alpha$ in finite samples. Ties and the discreteness of the randomization distribution can make it conservative. A larger $P$ gives a finer $p$-value and is not needed for the level.", "evidence": "The evidence is an exact counterexample to the level of the rule $B/P\\le\\alpha$, computed in rational arithmetic, and a derivation of the level of the corrected rule.\n\n\\emph{Example.} I take eight intervention nodes at the points $(3i,0)$, $i=1,\\dots,8$, with one outcome point at each node, and the outcomes $y_i=2^{i-8}$, that is, $1/128,1/64,\\dots,1$, under every assignment. The sharp null hypothesis holds. The assignments are independent with probability $1/2$, and the statistic is the absolute value of the Horvitz--Thompson estimator (13) at $d=0$. The 256 assignments yield the 128 values $(2r-1)/512$, $r=1,\\dots,128$, each with probability $1/128$. Let $\\alpha=1/20$ and $P=100$. If the observed statistic is the $m$th largest value, then $B$ is binomial with parameters $100$ and $m/128$, and the rule rejects when $B\\le5$. Summing over $m$, I obtain the size\n\\[\n\\frac{1}{128}\\sum_{m=1}^{128}\\Pr\\bigl(\\operatorname{Bin}(100,m/128)\\le5\\bigr)=0.0555\\ldots>\\frac1{20}.\n\\]\nThe rule with $(1+B)/(P+1)$ rejects when $B\\le4$ and has size $0.0456\\ldots$ in the same design. For the signed estimator with the comparison as printed, the 128 assignments with a negative estimate never reject, and the size is $0.0277\\ldots$, half of the first value.\n\n\\emph{A large number of draws.} With 17 nodes and $y_i=2^{i-17}$ there are $M=65536$ equally likely values of the statistic. For $P=10000$ and $\\alpha=1/20$ the rule rejects when $B\\le500$. If the observed statistic is the $m$th largest value, the rule rejects with probability $f(m/M)$, where $f(q)=\\Pr(\\operatorname{Bin}(10000,q)\\le500)$ is decreasing in $q$ and has integral $501/10001$ over $[0,1]$. Comparing the sum over $m$ with the integral, I find that the size is at least\n\\[\n\\frac{501}{10001}-\\frac{1}{65536}=\\frac{32823535}{655425536}=0.05008\\ldots>\\frac1{20}.\n\\]\n\n\\emph{The corrected rule.} Let $S_0=|T(\\mathbf Y,\\mathbf Z)|$ and $S_p=|T(\\mathbf Y,\\mathbf Z_p)|$ for $p=1,\\dots,P$. Under the sharp null hypothesis the outcomes are fixed, and $S_0,S_1,\\dots,S_P$ are independent and identically distributed. Let $R_i$ be the number of indices $j$ with $S_j\\ge S_i$. Then the corrected $p$-value is $R_0/(P+1)$. For any values and any integer $k$, at most $k$ indices have $R_i\\le k$, since such an index is among the $k$ largest values, ties included. Taking $k=\\lfloor\\alpha(P+1)\\rfloor$ and using that the $P+1$ indices are exchangeable, I have\n\\[\n(P+1)\\Pr\\Bigl(\\frac{R_0}{P+1}\\le\\alpha\\Bigr)=E\\Bigl[\\sum_{i=0}^{P}\\mathbf 1\\{R_i\\le k\\}\\Bigr]\\le k\\le\\alpha(P+1).\n\\]\nApplying the same argument to the event $B\\le\\lfloor\\alpha P\\rfloor$, I obtain $\\Pr(B/P\\le\\alpha)\\le(\\lfloor\\alpha P\\rfloor+1)/(P+1)\\le\\alpha+(1-\\alpha)/(P+1)$.\n\n\\emph{What the examples do not show.} The rule $B/P\\le\\alpha$ is not invalid for every $P$. For $\\alpha=1/20$ and $P=9999$ both rules reject exactly when $B\\le499$. For a fixed finite design the rule becomes conservative when $P$ is large enough: in the eight-node example its size is at most $1997/40000$ for all $P\\ge8{,}000{,}000$. If the printed sentence only asserts that a suitable $P$ exists, these examples do not contradict it. What fails is the guarantee for an arbitrary large $P$.", "problem": "There are two points. First, the indicator compares the absolute value of each simulated statistic with the signed observed statistic. If $T(\\mathbf Y,\\mathbf Z)<0$, then every indicator equals 1, the proportion is 1, and the rule cannot reject. The statistics of the paper are signed. In Section 8.1 the observed statistic is $-0.034$ and the reported $p$-value is about 0.065, which the printed rule cannot produce. The comparison that is meant has $|T(\\mathbf Y,\\mathbf Z)|$ on the right.\n\nSecond, let $B$ be the number of draws with $|T(\\mathbf Y,\\mathbf Z_p)|\\ge|T(\\mathbf Y,\\mathbf Z)|$. The rule that rejects when $B/P\\le\\alpha$ does not have level $\\alpha$ merely because $P$ is large. Its size can exceed $\\alpha$ at a given $P$, by at most $(1-\\alpha)/(P+1)$.", "what_survives": "The definitions of the AME, the point estimators, their limit theory and the confidence intervals do not use this rule. The simulation code in the replication files takes absolute values on both sides and compares the observed statistic with a quantile of the simulated ones, so the printed comparison is not what was computed, and no curve of Figure 9 is shown to be wrong. The separate recalculations in \\fid{P004-N006} do not identify the computation behind the reported $p$-value of Section 8.1. If it is $B/P$ with $P=10{,}000$, then the corrected value exceeds it by at most $1/10001$, and the rejection at the 10\\% level and the non-rejection at the 5\\% level are unchanged. The corrected rule needs no new assumption. The $p$-value increases by $(1-B/P)/(P+1)$."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E005", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of equation (26), sAME(d; eta) equal to the integral over t of AME(t; eta) times K_h((d minus t) over h), and of the six lines of text after it. A red box marks the sentence 'The donut and disk AMEs, defined in Section 3, are special cases with (properly-normalized) uniform kernels.'", "caption": "Section 6.2, p. 757 (PDF p. 14), equation (26) and the paragraph after it. The marked sentence calls the donut and disk AMEs special cases of the smoothed AME (26) with uniform kernels.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T14", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T14"}, {"id": "P004-T15", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T15"}], "headline": "The disk and donut AMEs are not special cases of the smoothed AME (26) with uniform kernels: they weight each distance by the mass of outcome points at that distance from each node.", "id": "P004-E006", "location": "Section 6.2, p. 757 (PDF p. 14), the sentence after equation (26)", "pdf_page": 14, "printed_page": "757", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T14", "P004-T15"], "text": {"as_printed": "``The donut and disk AMEs, defined in Section 3, \\hl{are special cases with (properly-normalized) uniform kernels}.''", "corrected": "``\\hl{The donut and disk AMEs average the node-specific circle effects over their distance bands using the normalized shell-mass measures induced by the spatial measure. For complete Euclidean disks about point interventions under area weighting, these radial weights are proportional to distance; a common uniform radial kernel applies as an identity only when the normalized shell-mass measures coincide with it.}''\n\nPrecisely, for a node $i$ and a set $A$ of distances let $\\nu_i(A)=\\zeta\\{x:d_i(x)\\in A\\}$, and let $\\tau_i(r;\\eta)$ be the effect of node $i$ on the outcome points at distance $r$ (for a measure without atoms, the conditional effect given the distance). For a band $D$ such that $0<\\nu_i(D)<\\infty$ for all $i$,\n\\[\n\\mathrm{AME}_D(\\eta)=\\frac1N\\sum_{i=1}^N\\frac{\\int_D\\tau_i(r;\\eta)\\,\\nu_i(dr)}{\\nu_i(D)}.\n\\]\nFor a complete Euclidean disk about a point node under area measure, the normalized measure $\\nu_i(dr)/\\nu_i(D)$ has density $2r/d^2$ on $D=[0,d]$ and density $2r/(b^2-a^2)$ on an annulus $D=[a,b]$.", "evidence": "The evidence is three exact calculations and a derivation. First, I consider one node at the origin, independent Bernoulli$(1/2)$ assignment, and unit mass at each of the outcome points $(1,0)$, $(0,2)$ and $(-2,0)$, with effects $0$, $1$ and $1$. The circle AMEs at radii $1$ and $2$ are $0$ and $1$, and their uniform average is $1/2$. The disk $[0,2]$ and the donut $[1,2]$ each have AME $2/3$. Second, I consider nodes at $(0,0)$ and $(3,0)$ and unit mass at $(0,4)$, $(3,4)$ and $(3,-4)$, with $Y_x(\\mathbf z)=1+z_1g_1(x)+z_2g_2(x)$, $g_1=(0,1,1)$ and $g_2=(-1,0,0)$. Then $\\mathrm{AME}(4)=\\mathrm{AME}(5)=0$, and every choice of common weights on the two radii gives $0$. The AME of the disk $[0,5]$ and of the donut $[4,5]$ equals $1/6$. The normalized masses at radii $4$ and $5$ are $(1/3,2/3)$ for the first node and $(2/3,1/3)$ for the second. Third, for one point node, area measure on the unit disk and $Y_x(z)=z\\|x\\|$, the circle AME at radius $r$ is $r$. Its uniform average over $[0,1]$ is $1/2$, and the disk AME is $2/3$.\n\nFor the derivation, let $\\delta_i(x)$ be the marginalized effect of node $i$ on the point $x$. By (6), (8) and (9), the effect of node $i$ on a band $D$ is $\\int_{d_i(x)\\in D}\\delta_i(x)\\,\\zeta(dx)$ divided by $\\zeta\\{x:d_i(x)\\in D\\}$. Rewriting the integral over $x$ as an integral over the distance $r=d_i(x)$, I obtain the formula of the corrected passage.", "problem": "By (2), a disk or donut average is the integral of the outcome over the points whose distance to the node lies in a band, divided by the mass of those points under the measure $\\zeta$. Each distance in the band then enters with a weight equal to the mass of outcome points at that distance from the node. Equation (26) averages the circle AMEs over distance with one kernel, common to all nodes. A kernel that is constant on the band gives every distance the same weight, and no normalization of it supplies weights that vary with the distance or with the node. For a point node under area measure, the weight of radius $r$ in the disk $[0,d]$ is $2r/d^2$, not $1/d$.", "what_survives": "Definitions (2), (6), (8) and (9), the inference results of Section 5 and the smoothed quantities (26) and (27) are unaffected. Equations (26) and (27) still define smoothed quantities, and the applications report smooths with a triangular kernel. No reported number depends on the sentence. What is withdrawn is the identity between a disk or donut AME and the smoothed AME (26) with a uniform kernel. When the masses at each distance differ across nodes, the circle AMEs alone need not determine the AME of a disk or donut. The reading of a sum of ring estimates as an area effect in Section 8.1 is the subject of \\fid{P004-E008}."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E006", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of the paragraph that begins 'For an interpretable sufficient condition'. A red box marks the sentence 'In this case, h(d) in C3 can be set to 2 d-bar + 2d.'", "caption": "Section 4, p. 752 (PDF p. 9), the paragraph after C3. The marked sentence sets the cutoff of C3 to $2\\bar d+2d$ with no restriction to point nodes.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E007.png"}, {"alt": "Crop of five lines of text. A red box marks the parenthesis '(thus ensuring adequate spacing between nonadjacent polygons)' in the sentence 'For polygon-intervention experiments, one may require that the size of each polygon is larger than a threshold value'.", "caption": "Section 4, p. 753 (PDF p. 10), the end of the paragraph after C4. The marked parenthesis says that a lower bound on polygon size ensures adequate spacing.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E007-b.png"}, {"alt": "Crop of footnotes 24 and 25. A red box marks '2 d-tilde + 2d' in footnote 25, 'To be precise, for AME at distance value d, we set the bandwidth at 2 d-tilde + 2d, as discussed in Section 4.'", "caption": "Footnote 25, p. 763 (PDF p. 20). The marked bandwidth applies the cutoff rule of Section 4 to the village polygons of the first application.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E007-c.png"}, {"alt": "Diagram in the plane, drawn to scale, with horizontal coordinates from 0 to 16 and vertical coordinates from 0 to 3. Two unit squares labelled A and B sit on the horizontal axis, A from 0 to 1 and B from 10 to 11. A double arrow between them is labelled 'd AB = 9'. A long grey rectangle labelled 'C, diameter D = square root of 122, about 11.05' lies above them, from 0 to 11 across and from 2 to 3 in height. A vertical double arrow from the lower edge of C to the lower edge of B is labelled 'effect radius d-bar = 2'. Below the squares, two horizontal bars start at the right edge of A. A short orange bar of length 4 is labelled 'printed cutoff 2 d-bar + 2d = 4: ends before B'. A long blue bar of length about 15.05 is labelled 'corrected cutoff 2 d-bar + 2d + D = 4 + square root of 122, about 15.05: reaches B'. Dashed vertical lines mark the right edge of A and the left edge of B. The title reads: 'Polygon nodes A and B, at distance 9, both depend on the treatment at C: the printed cutoff is 4, and the corrected cutoff is 15.05'.", "caption": "The first counterexample, drawn to scale. The squares $A=[0,1]\\times[0,1]$ and $B=[10,11]\\times[0,1]$ are $9$ apart, and the treatment at the polygon $C=[0,11]\\times[2,3]$ sets the outcome on both of them, within the effect radius $\\bar d=2$. Each bar is a cutoff, drawn as a length from $A$ toward $B$. The printed cutoff $2\\bar d+2d=4$ ends before $B$. The corrected cutoff $2\\bar d+2d+D=4+\\sqrt{122}\\approx15.05$, where $D=\\sqrt{122}$ is the diameter of $C$, reaches $B$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E007-counterexample.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T16", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T16"}, {"id": "P004-T17", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T17"}], "headline": "The cutoff $h(d)=2\\bar d+2d$ is sufficient for C3 with point nodes; with polygon nodes adding a uniform bound on the source polygon’s diameter gives a sufficient cutoff, and a lower bound on polygon area does not give C4.", "id": "P004-E007", "location": "Section 4, p. 752 (PDF p. 9), the paragraph after C3; p. 753 (PDF p. 10), the sentence on polygon interventions after C4; footnote 25, p. 763 (PDF p. 20)", "pdf_page": 9, "printed_page": "752", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P004-T16", "P004-T17"], "text": {"as_printed": "P.~752: ``For an interpretable sufficient condition, suppose that the intervention at every node has no influence on the outcome points more than $\\bar d$ away from it. Then C3 would be satisfied if the distance between the boundary of the radius-d circle around node i and the radius-d circle around node j is larger than $2\\bar d$. \\hl{In this case, $h(d)$ in C3 can be set to $2\\bar d+2d$.}''\n\nP.~753: ``For polygon-intervention experiments, one may require that the size of each polygon is larger than a threshold value \\hl{(thus ensuring adequate spacing between nonadjacent polygons)}.''\n\nFootnote 25, p.~763: ``To be precise, for AME at distance value d, we set the bandwidth at \\hl{$2\\tilde d+2d$}, as discussed in Section 4.''", "corrected": "P.~752: ``For an interpretable sufficient condition, suppose that the intervention at every node has no influence on the outcome points more than $\\bar d$ away from it. Then, \\hl{for point interventions}, C3 would be satisfied if the distance between the boundary of the radius-d circle around node i and the radius-d circle around node j is larger than $2\\bar d$. In this case, $h(d)$ in C3 can be set to $2\\bar d+2d$. \\hl{For polygon interventions, if every polygon has diameter at most $D<\\infty$, uniformly in the sample size, $h(d)$ in C3 can be set to $2\\bar d+2d+D$.}''\n\nStated in full, the cutoff rule for polygon interventions is as follows. Let $d_{ij}$ be the minimum distance between the points of node $i$ and the points of node $j$, as in footnote 3, and let the diameter of a node be the largest distance between two of its points. Suppose that the circle average (2) at distance value $d$ uses only outcome points within distance $d$ of its node, which holds for the circle, the donut and the disk. Suppose further that the intervention at every node has no influence on the outcome points more than $\\bar d$ away from it, and that every node has diameter at most $D<\\infty$, where $\\bar d$ and $D$ do not depend on the sample size. Then for all nodes $i\\ne j$,\n\\[\ns_{ij}(d)=1\\quad\\text{implies}\\quad d_{ij}\\le2\\bar d+2d+\\hl{D},\n\\]\nand C3 holds with $h(d)=2\\bar d+2d+\\hl{D}$. For point interventions every node has diameter $0$, and the rule is the printed $h(d)=2\\bar d+2d$.\n\nP.~753: ``For polygon-intervention experiments, \\hl{a lower bound on the area of each polygon does not ensure C4. In the Euclidean plane, if the polygons have pairwise disjoint interiors, area at least $a>0$ and diameter at most $D<\\infty$, uniformly in the sample size, then $|S_N(i;r)|\\le\\pi(r+2D)^2/a$ for all $i$ and all $r\\ge0$, and C4 holds with $b(r)=\\pi(r+2D)^2/a$.}''\n\nFootnote 25: ``To be precise, for AME at distance value d, we set the bandwidth at $2\\tilde d+2d$, \\hl{the rule of Section 4 for point interventions. For village polygons this bandwidth is a sensitivity choice; the premise of a local effect radius does not establish that it is sufficient.}''\n\nAlternatively, C3 and C4 may be assumed directly for polygon interventions, without the claim that a local effect radius and a lower bound on area establish them.", "evidence": "The evidence is two exact counterexamples and a derivation of the corrected bounds. First, let $A=[0,1]\\times[0,1]$, $B=[10,11]\\times[0,1]$ and $C=[0,11]\\times[2,3]$, with independent Bernoulli$(1/2)$ assignments, $\\bar d=2$, $d=0$ and Lebesgue measure. Let $Y_x(\\mathbf z)=z_C$ for $x\\in A\\cup B$ and $Y_x(\\mathbf z)=0$ elsewhere. Every affected point is within distance $2$ of $C$, and $A$ and $B$ affect nothing. The circle averages at $A$ and $B$ both equal $Z_C$. Then $s_{AB}(0)=1$ and their covariance is $1/4$, while $d_{AB}=9>2\\bar d+2d=4$. A variant with $d=\\bar d=1$ has $C=[0,11]\\times[3,4]$ and unit mass on the outcome points $(1/2,2)$ and $(21/2,2)$, where the outcome equals $z_C$. In it $d_{AB}=9>4$, the radius-$1$ contours of $A$ and $B$ are $7>2\\bar d=2$ apart, and again $s_{AB}(1)=1$ with covariance $1/4$.\n\nThe figure draws the first example to scale, and the table gives the distances in both examples. In each example the printed cutoff is $4$, and $A$ and $B$ are $9$ apart. The polygon $C$ has diameter $D=\\sqrt{11^2+1^2}=\\sqrt{122}\\approx11.05$, which is the largest of the three diameters. With this diameter, I obtain the corrected cutoff $2\\bar d+2d+D=4+\\sqrt{122}\\approx15.05$, which is larger than $d_{AB}=9$.\n\\begin{audittable}{lcc}{The two counterexamples to the printed cutoff rule.}\nQuantity & First example & Second example \\\\\nDistance value $d$ & $0$ & $1$ \\\\\nEffect radius $\\bar d$ & $2$ & $1$ \\\\\nPolygon $C$ & $[0,11]\\times[2,3]$ & $[0,11]\\times[3,4]$ \\\\\nPrinted cutoff $2\\bar d+2d$ & $4$ & $4$ \\\\\nDistance $d_{AB}$ & $9$ & $9$ \\\\\n$s_{AB}(d)$ & $1$ & $1$ \\\\\nCovariance of the circle averages at $A$ and $B$ & $1/4$ & $1/4$ \\\\\nLargest diameter $D$, that of $C$ & $\\sqrt{122}\\approx11.05$ & $\\sqrt{122}\\approx11.05$ \\\\\nCorrected cutoff $2\\bar d+2d+D$ & $4+\\sqrt{122}\\approx15.05$ & $4+\\sqrt{122}\\approx15.05$ \\\\\n\\end{audittable}\n\nSecond, for $n\\ge1$ let $P_0=[0,1]\\times[0,1]$ and $P_{k+1}=[2,2+n]\\times[2k/n,(2k+1)/n]$ for $0\\le k<n$. These $n+1$ polygons are pairwise disjoint, each has area $1$, and each lies within distance less than $2$ of $P_0$. Then $|S_{n+1}(P_0;2)|=n+1$, which is unbounded in $n$.\n\nFor the corrected cutoff, suppose that $s_{ij}(d)=1$ through a third node $\\ell$ of diameter at most $D$. There exist outcome points $x$ and $y$, both affected by the third node, in the averaging regions of $i$ and $j$, points $a\\in i$ and $b\\in j$ such that $\\|a-x\\|\\le d$ and $\\|b-y\\|\\le d$, and points $u,v\\in\\ell$ such that $\\|x-u\\|\\le\\bar d$, $\\|y-v\\|\\le\\bar d$ and $\\|u-v\\|\\le D$. By the triangle inequality along $a,x,u,v,y,b$, I have $d_{ij}\\le2\\bar d+2d+D$. If $\\ell=i$ or $\\ell=j$, I have $d_{ij}\\le d+\\bar d$ by the same argument. For the count, suppose every polygon has diameter at most $D$, and fix a point $q$ of polygon $i$. If $d_{ij}\\le r$, every point of $j$ is within $r+2D$ of $q$. The polygons of $S_N(i;r)$ have disjoint interiors, each has area at least $a$, and all lie in the disk of radius $r+2D$ about $q$. Comparing areas, I obtain $a\\,|S_N(i;r)|\\le\\pi(r+2D)^2$.", "problem": "The distance between two polygon nodes is the minimum distance between them (footnote 3). With that distance, a third polygon $\\ell$ can lie within $\\bar d$ of the averaging regions of both $i$ and $j$ while $i$ and $j$ are farther apart than $2\\bar d+2d$: the point of $\\ell$ nearest to one region and the point of $\\ell$ nearest to the other may be far from each other. The printed rule treats them as one point, which is right for a point node. For polygons a sufficient rule adds a uniform bound on the diameter of the common source polygon. Separately, a lower bound on the area of each polygon does not bound the number of polygons within a fixed distance of a given polygon, because long, thin polygons of fixed area can lie arbitrarily close together. The parenthesis on p.~753 does not give C4. The paper suggests setting the cutoff value $\\tilde d$ of the spatial HAC variance estimator (19) equal to $h(d)$ (p.~755, PDF p.~12). With polygon nodes, a cutoff value set by the printed rule can leave out pairs of nodes whose circle averages depend on a common assignment. Footnote 25 applies the cutoff rule to the village polygons of the first application.", "what_survives": "Proposition 1 uses neither C3 nor C4, and Propositions 2--4 assume both directly; these counterexamples do not refute them or the implication from C3 and C4 to C4A. The rule $h(d)=2\\bar d+2d$ stands for point nodes. No simulation result or application interval is shown to change: I did not recompute the application, which reports its intervals for a range of assumed interference radii. What changes is the justification of the cutoff for polygon nodes. For polygons, C3 and C4 are either assumed directly or derived under the additional geometric conditions below."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E007", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Four lines of body text that define the test statistic as the sum of the point Hajek estimates between zero and five km and give its value, -0.034. A red box marks the parenthesis: (a 3.4 percentage point decline in deforested area).", "caption": "Section 8.1, last paragraph of p. 763 (PDF p. 20). The marked parenthesis reads the sum of the point estimates as a decline in deforested area.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E008.png"}, {"alt": "Three lines of body text. Two red boxes mark the words: These results indicate that the net gains from the intervention were indeed beneficial. The sentence ends: with no indication of substantial leakage.", "caption": "Continuation of the same paragraph on p. 764 (PDF p. 21), below Figure 11. The marked clause draws a net gain from the test.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E008-b.png"}, {"alt": "Dot plot with three rows, one for each example, on a horizontal axis labelled from -4 to 1 percentage points. In every row an orange diamond sits at -3.4, the sum of the six distance-specific estimates. A blue circle shows the effect on the deforested share of the whole region: -0.57 in example A, +0.67 in example B and -0.39 in example C. A grey line joins the two marks in each row, and a vertical line marks zero.", "caption": "The three examples of the table. In each, the sum of the six distance-specific estimates is $-0.034$, which p. 763 reads as a decline of 3.4 percentage points; the effect on the deforested share of the whole region is $-0.57$, $+0.67$ or $-0.39$ percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E008-sum-vs-area.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T5", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T5"}, {"id": "P004-T6", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T6"}, {"id": "P004-T37", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T37"}], "headline": "The statistic $-0.034$ is an unweighted sum of six distance-specific Hajek estimates, not a 3.4 percentage point decline in deforested area, and its sign does not by itself show a net gain over the area.", "id": "P004-E008", "location": "Section 8.1, last paragraph of p. 763 (PDF p. 20), continuing on p. 764 (PDF p. 21) after Figure 11", "pdf_page": 20, "printed_page": "763", "reaches": [{"effect": "same_error_restated", "effect_text": "a corresponding statement is printed there", "explanation": null, "package": null, "recorded_as": "P008-N016", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P008-N016", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N016", "work_id": "P008"}], "severity": "Moderate", "subject": "paper", "target_ids": ["P004-T37", "P004-T5", "P004-T6"], "text": {"as_printed": "``Alternatively, we use a randomization test to evaluate the cumulative effects between zero km and five km. The test statistic is the sum of the point Hajek estimates between zero and five km. The estimated value is $-0.034$ \\hl{(a 3.4 percentage point decline in deforested area)}. For the randomization test, we use 10,000 random draws, and the p-value is around 0.065. Thus, we would reject with 90\\% confidence the sharp null hypothesis. \\hl{These results indicate that the net gains from the intervention were indeed beneficial}, with no indication of substantial leakage.''", "corrected": "``Alternatively, we use a randomization test to evaluate the cumulative effects between zero km and five km. The test statistic is the sum of the point Hajek estimates between zero and five km. \\hl{The sum of the distance-specific estimates is $-0.034$; this is a summed contrast rather than an estimate of the area-average change in deforestation.} For the randomization test, we use 10,000 random draws, and the p-value is around 0.065. Thus, we would reject with 90\\% confidence the sharp null hypothesis. \\hl{The randomization test provides evidence against the sharp null. Together with the displayed point estimates, the results suggest beneficial effects,} with no indication of substantial leakage \\hl{at the examined distances}.''\n\nA statement about deforested area needs a spatial aggregate that is defined explicitly and computed from the data with its own weights.", "evidence": "The evidence is a set of exact examples with binary outcomes. The village is a disc of radius 1 km, with five rings of width 1 km around it. The six areas are proportional to $1,3,5,7,9,11$. The deforested share is $1/10$ in every bin of a control village. In each of the three examples in the table, the six differences sum to $-17/500=-0.034$. The effects on the deforested share of the whole region are $-0.57$, $+0.67$ and $-0.39$ percentage points. In the first example, if each ring is split in two, there are eleven bins and the sum is $-187/3000\\approx-0.062$, with the same outcomes and the same effect on the region.\n\n\\begin{audittable}{lllr}{Three examples with the published sum. Effects are differences in deforested share, treated minus control.}\nExample & Effect in the village & Effects in the five rings & Effect on the region \\\\\nA & $-17/3000$ & $-17/3000$ in each & $-17/3000$ \\\\\nB & $-2/25$ & $+23/2500$ in each & $+121/18000$ \\\\\nC & $-1/100$ & $(-9,-6,-5,-3,-1)/1000$ & $-7/1800$ \\\\\n\\end{audittable}", "problem": "Each Hajek estimate $\\widehat{\\tau}_{\\mathrm{HA}}(d)$ of (16) is a difference between treated and control villages in the mean of a circle average (2). In this application it is a difference in the share of deforested pixels at distance $d$. The test statistic adds six of these differences, for $d=0,1,\\dots,5$ km, with no weights and no denominator.\n\nConsider a region made of disjoint distance bins with areas $a_k$ and effects $\\tau_k$, and set $A=\\sum_k a_k$. The effect on the deforested share of the region is the weighted mean $\\sum_k(a_k/A)\\tau_k$. Since its weights are nonnegative and sum to one, this mean lies between the smallest and the largest $\\tau_k$. In the sum every weight equals one. With equal areas the sum is the mean multiplied by the number of bins. Refining the distance grid can change the sum without changing the effect on the region. The sum therefore does not estimate a change in deforested area, and a negative sum is compatible with an increase in the deforested share of the region. For village polygons, the weights would also depend on the areas of the buffers, on missing pixels and on overlaps.", "what_survives": "The sum remains a valid statistic for the randomization test of the sharp null hypothesis, and its p-value does not depend on how the statistic is described. The estimates and intervals of Figure 11, and the reading that they show no substantial leakage, are unaffected. What is lost is the magnitude of 3.4 points and the statement about net gains so far as it rests on the sum. I have not computed an effect on deforested area for the Uganda data, and the examples say nothing about its sign or size. Whether the replication files give the values $-0.034$ and $0.065$ is a separate matter (\\fid{P004-N006})."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E008", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The first paragraph of Section 8.2, nine lines of body text. A red box marks the words before 1980 in the sentence: Each parcel has a value of 1 if deforestation occurred inside it before 1980.", "caption": "Section 8.2, first paragraph, p. 764 (PDF p. 21): the definition of the outcome of the Costa Rica application.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "The Costa Rica outcome is deforestation by 1997; the year 1980 is the cutoff for the establishment of the protected areas, not the date of the outcome.", "id": "P004-E009", "location": "Section 8.2, first paragraph, p. 764 (PDF p. 21)", "pdf_page": 21, "printed_page": "764", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``The outcome is measured at the level of parcels with a size of three hectares. Each parcel has a value of 1 if deforestation occurred inside it \\hl{before 1980}.''", "corrected": "``Each parcel has a value of 1 if deforestation occurred inside it \\hl{by 1997}.''\n\nIf the sample is described further, the protected areas are those established before 1980.", "evidence": "The evidence is a check of the cited source. Ferraro, Hanauer and Sims (2011), p.~13914, write: ``Each parcel is classified as deforested or forested by the end year (Costa Rica in 1997 and Thailand in 2000)'', and they describe the protected areas as ``established before 1980 (Costa Rica)''. The application script in the replication archive takes its outcome from a field named \\texttt{defor97}.", "problem": "The sentence assigns to the outcome the date that defines the treatment. In the study that Section 8.2 cites, each parcel is classified as deforested or forested at the end year, 1997 for Costa Rica, and 1980 is the cutoff for the establishment of the protected areas.", "what_survives": "This correction changes the description only. The script uses \\texttt{defor97}, and no estimate, interval or figure has been shown wrong by this date check. I did not compute the Costa Rica results again."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E009", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of equation (11), which sets s_ij(d) to 1 if I_il(d) I_jl(d) = 1 for some node l and to 0 otherwise, and of the four lines of text below it. A red box marks the clause 'meaning nonindependence over possible values of Z.' at the end of the first sentence.", "caption": "Section 4, p. 751 (PDF p. 8), equation (11) and the sentence after it. The marked clause asserts nonindependence where possible dependence is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E010.png"}], "formal_release": null, "formal_statements": [], "headline": "A shared assignment input, $s_{ij}(d)=1$, permits dependence between the circle averages at $i$ and $j$; it does not imply it.", "id": "P004-E010", "location": "Section 4, p. 751 (PDF p. 8), the sentence after equation (11)", "pdf_page": 8, "printed_page": "751", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``If $s_{ij}(d)=1$, then circle averages at $i$ and $j$ will vary together whenever there is variation in treatment values at the relevant $\\ell$s, \\hl{meaning nonindependence over possible values of $\\mathbf Z$}.''", "corrected": "``If $s_{ij}(d)=1$, circle averages at $i$ and $j$ share a possible source of variation and \\hl{may be dependent} over possible values of $\\mathbf Z$.''", "intended_meaning": "The sentence explains what an edge of the dependency graph records: the two circle averages have an assignment input in common and may be dependent. Read as a claim that $s_{ij}(d)=1$ forces dependence, it is too strong. Under independent Bernoulli$(1/2)$ assignments, let $\\mu_1=\\mathbf 1\\{Z_1\\ne Z_2\\}$ and $\\mu_2=Z_2$ at $d=0$. Both depend on $Z_2$, which implies that $s_{12}(0)=1$. Each of the four pairs of values has probability $1/4$, and the two averages are independent. Definitions (10)--(11) and the arguments that follow use only the other direction: circle averages with no assignment input in common are independent. That direction is correct."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E010", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The statement of Lemma A.3 with a displayed expansion of the linearised Hajek estimator: AME plus 1 over N p times a sum over treated nodes, minus 1 over N p times a sum over control nodes. A red box marks the second denominator, N p. Below, the first line of the proof defines the treated average with 1 over N p and the control average with 1 over N(1-p).", "caption": "Supplement A, p. 4: the statement of Lemma A.3 and the first line of its proof. The marked denominator of the control sum reads $Np$; the proof below normalises the control average by $N(1-p)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E011.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T18", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T18"}, {"id": "P004-T42", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T42"}, {"id": "P004-T44", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T44"}], "headline": "In Lemma A.3 the control sum of the linearised Hajek estimator should be divided by $N(1-p)$, not by $Np$.", "id": "P004-E011", "location": "Supplement A, Lemma A.3, p. 4, displayed linear expansion", "pdf_page": 4, "printed_page": "4", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T18", "P004-T42", "P004-T44"], "text": {"as_printed": "\\textbf{Lemma A.3.} Consider the estimator $\\widehat\\tau_{\\mathrm{HA}}(d)$ defined in (16). It has the following asymptotic linear expansion:\n\\[\n\\widehat\\tau^{\\mathrm{L}}_{\\mathrm{HA}}(d)=\\mathrm{AME}(d;\\eta)+\\frac{1}{Np}\\sum_{i=1}^N Z_i\\left(\\mu_i(\\mathbf{Y}(\\mathbf{Z});d)-\\bar\\mu^1(d)\\right)-\\frac{1}{\\hl{Np}}\\sum_{i=1}^N(1-Z_i)\\left(\\mu_i(\\mathbf{Y}(\\mathbf{Z});d)-\\bar\\mu^0(d)\\right).\n\\]\nSuch an expansion satisfies $\\sqrt{N}\\left(\\widehat\\tau^{\\mathrm{L}}_{\\mathrm{HA}}(d)-\\widehat\\tau_{\\mathrm{HA}}(d)\\right)=o_p(1)$.", "corrected": "\\textbf{Lemma A.3.} Consider the estimator $\\widehat\\tau_{\\mathrm{HA}}(d)$ defined in (16). It has the following asymptotic linear expansion:\n\\[\n\\widehat\\tau^{\\mathrm{L}}_{\\mathrm{HA}}(d)=\\mathrm{AME}(d;\\eta)+\\frac{1}{Np}\\sum_{i=1}^N Z_i\\left(\\mu_i(\\mathbf{Y}(\\mathbf{Z});d)-\\bar\\mu^1(d)\\right)-\\frac{1}{\\hl{N(1-p)}}\\sum_{i=1}^N(1-Z_i)\\left(\\mu_i(\\mathbf{Y}(\\mathbf{Z});d)-\\bar\\mu^0(d)\\right).\n\\]\nSuch an expansion satisfies $\\sqrt{N}\\left(\\widehat\\tau^{\\mathrm{L}}_{\\mathrm{HA}}(d)-\\widehat\\tau_{\\mathrm{HA}}(d)\\right)=o_p(1)$.\n\nHere, as in the proof,\n\\[\n\\bar\\mu^1(d)=\\frac{1}{N}\\sum_{i=1}^N\\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)\\right],\\qquad \\bar\\mu^0(d)=\\frac{1}{N}\\sum_{i=1}^N\\mathrm{E}\\left[\\mu_i(\\mathbf{0};d)\\right].\n\\]\nThe lemma states no conditions of its own; its proof bounds the remainder by the argument of Lemma A.2, which uses C1--C4.", "evidence": "The evidence is a derivation and an exact counterexample to the printed display. In the notation of the proof,\n\\[\n\\begin{aligned}\n\\widehat\\mu^1(d)&=\\frac{1}{Np}\\sum_{i=1}^N Z_i\\mu_i, &\\qquad \\widehat\\mu^0(d)&=\\frac{1}{N(1-p)}\\sum_{i=1}^N(1-Z_i)\\mu_i,\\\\\n\\widehat N_1&=\\frac{1}{Np}\\sum_{i=1}^N Z_i, &\\qquad \\widehat N_0&=\\frac{1}{N(1-p)}\\sum_{i=1}^N(1-Z_i),\n\\end{aligned}\n\\]\nwhere $\\mu_i$ stands for $\\mu_i(\\mathbf{Y}(\\mathbf{Z});d)$. The gradient of $f$ at $(\\bar\\mu^1(d),\\bar\\mu^0(d),1,1)$ is $(1,-1,-\\bar\\mu^1(d),\\bar\\mu^0(d))$. Inserting the definitions above into the first-order term, I obtain\n\\[\n\\left(\\widehat\\mu^1-\\bar\\mu^1\\right)-\\left(\\widehat\\mu^0-\\bar\\mu^0\\right)-\\bar\\mu^1\\left(\\widehat N_1-1\\right)+\\bar\\mu^0\\left(\\widehat N_0-1\\right)=\\frac{1}{Np}\\sum_{i=1}^N Z_i\\left(\\mu_i-\\bar\\mu^1\\right)-\\frac{1}{N(1-p)}\\sum_{i=1}^N(1-Z_i)\\left(\\mu_i-\\bar\\mu^0\\right).\n\\]\nFor the counterexample, let $p=1/4$, let there be no interference, and let the outcome of node $i$ be $a_i$ under both conditions, with $a_i=1$ for half of the nodes and $a_i=-1$ for the others. Then $\\bar\\mu^1(d)=\\bar\\mu^0(d)=0$. Subtracting the corrected expansion from the printed one, I find\n\\[\nD_N=-\\left(\\frac{1}{p}-\\frac{1}{1-p}\\right)\\frac{1}{N}\\sum_{i=1}^N(1-Z_i)a_i,\\qquad N\\operatorname{Var}(D_N)=p(1-p)\\left(\\frac{1}{p}-\\frac{1}{1-p}\\right)^2=\\frac{4}{3}.\n\\]\nAn exact enumeration for $N=2$, $4$ and $8$ gives $4/3$ each time. The summands are independent, bounded and centred. The central limit theorem then implies that $\\sqrt{N}D_N$ has a nondegenerate normal limit, and this quantity is not $o_p(1)$.", "problem": "The proof of the lemma writes the Hajek estimator as $f(a,b,c,d)=a/c-b/d$ evaluated at the treated and control averages and the two normalised counts, and it defines the control average and the control count with the factor $1/(N(1-p))$. The first-order term of that expansion divides the control residuals by $N(1-p)$. The statement prints $Np$. The two expressions agree when $p=1/2$. For other values of $p$ the printed expansion does not in general satisfy the last sentence of the lemma: its difference from the Hajek estimator is of order $N^{-1/2}$ and not smaller.", "what_survives": "The proof of Lemma A.3, the variance in Lemma A.4 and equation (27) already use $N(1-p)$ for the control terms. The results on asymptotic normality and inference that rest on the lemma are not affected by the misprint. Only the display in the statement changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E011", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Lemma A.5 with the line of text above it. A red box marks the clause 'If c_i(d) is at most c-tilde for all i and c-tilde squared over N tends to 0, then', which precedes the displayed conclusion that the sum of the U_i divided by the square root of its variance tends to N(0,1).", "caption": "Supplement A, p. 6, Lemma A.5. The marked clause is the lemma’s condition on the sizes of the neighborhoods; no condition on the variance of the sum appears.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E012.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T19", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T19"}, {"id": "P004-T20", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T20"}], "headline": "Lemma A.5 needs a lower bound on the variance of the sum; without one, bounded mean-zero summands with small dependency neighborhoods need not have a normal limit.", "id": "P004-E012", "location": "Supplement A, Lemma A.5, p. 6 (supplement PDF p. 6)", "pdf_page": 6, "printed_page": "6", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T19", "P004-T20"], "text": {"as_printed": "``LEMMA A.5. (Ogburn et al. (2020), Lemma 1 and 2) Consider a set of $N$ units. Let $U_1,\\ldots,U_N$ be bounded mean-zero random variables with finite fourth moments and dependency neighborhoods $\\mathcal B(i;d)$. \\hl{If $c_i(d)\\le\\tilde c$ for all $i$ and $\\tilde c^2/N\\to0$}, then\n\\[\n\\frac{\\sum_{i=1}^NU_i}{\\sqrt{\\operatorname{Var}\\big(\\sum_{i=1}^NU_i\\big)}}\\to N(0,1).\\text{''}\n\\]", "corrected": "``LEMMA A.5. Consider a set of $N$ units. Let $V_1,\\ldots,V_N$ be mean-zero random variables with dependency neighborhoods $\\mathcal B(i;d)$, \\hl{such that $|V_i|\\le M$ for a constant $M$ that does not depend on $i$ or $N$}. If $c_i(d)\\le\\bar c$ for all $i$, \\hl{where $\\bar c$ does not depend on $N$, and $\\operatorname{Var}\\big(\\sum_{i=1}^NV_i\\big)\\ge vN$ for a constant $v>0$ and all sufficiently large $N$}, then\n\\[\n\\frac{\\sum_{i=1}^NV_i}{\\sqrt{\\operatorname{Var}\\big(\\sum_{i=1}^NV_i\\big)}}\\Rightarrow N(0,1).\\text{''}\n\\]\nHere a dependency neighborhood means that the variables indexed by two sets of units are independent whenever no unit of one set lies in the neighborhood of a unit of the other. In the proof of Proposition 2 the lemma is applied to the unnormalized summands $V_i$ such that $\\widehat\\tau_{\\mathrm{HT}}(d)-\\mathrm{AME}(d;\\eta)=N^{-1}\\sum_iV_i$, and in the proof of Proposition 3 to the summands of the linearized Hajek estimator. Equivalently, for the normalized summands $U_i$ of the printed proof, the conditions are $\\operatorname{Var}(\\sum_iU_i)=1$, $c_i(d)\\le\\bar c$ and $|U_i|\\le K/\\sqrt N$ for a constant $K$.", "evidence": "The evidence is an exact counterexample and a derivation of the corrected statement. For every $N$, let $U_1$ equal $+1$ or $-1$ with probability $1/2$ each, and let $U_2=\\cdots=U_N=0$. These variables are independent, bounded by $1$ and have mean zero, and $\\mathcal B(i;d)=\\{i\\}$ is a dependency neighborhood. Then $c_i(d)=1$ and $\\tilde c^2/N=1/N\\to0$. The sum equals $U_1$ and has variance $1$, and the standardized sum takes the values $\\pm1$ with probability $1/2$ each for every $N$. Its distribution function at $-1/2$ equals $1/2$, and $\\Phi(-1/2)<1/2$. A second example has every marginal variance equal to $1$. Let $N=2m+1$, let $R_0,\\ldots,R_m$ be independent signs, and let $(U_1,\\ldots,U_N)=(R_0,R_1,-R_1,\\ldots,R_m,-R_m)$. The neighborhoods have at most two elements, and the sum is again $R_0$.\n\nFor the corrected statement, let $\\bar c$ be a constant that bounds the sizes of the neighborhoods for all sample sizes. A joint cumulant of $r$ summands vanishes unless their indices are connected through the neighborhoods. There are at most $Nr^{r-2}\\bar c^{\\,r-1}$ connected $r$-tuples of indices, and each cumulant is at most $C_rM^r$ in absolute value, where $M$ bounds the summands and $C_r$ depends only on $r$. With $X_N$ the standardized sum and $v$ the constant in the lower bound on the variance, I obtain\n\\[\n|\\kappa_r(X_N)|\\le\\frac{r^{r-2}\\bar c^{\\,r-1}C_rM^r}{v^{r/2}}\\,N^{1-r/2}\\to0\\qquad\\text{for each }r\\ge3.\n\\]\nSince $\\kappa_1(X_N)=0$ and $\\kappa_2(X_N)=1$, this bound implies that the moments of $X_N$ converge to those of $N(0,1)$, and these moments determine the normal law. Note that I did not compare the lemma with Lemmas 1 and 2 of Ogburn et al.\\ (2020); nothing here concerns the original result.", "problem": "The lemma states a central limit theorem for bounded, centered summands under a single condition, on the sizes $c_i(d)$ of the dependency neighborhoods. Nothing in it keeps the variance of the sum from being small relative to $N$, and without such a condition the conclusion fails. A positive lower bound on the variance of the sum is not enough either: the standardized sum does not change when every summand is multiplied by a constant. The propositions that use the lemma assume that $N$ times the variance of the estimator is bounded below. The condition is available where the lemma is applied, but it is missing from the lemma.", "what_survives": "Propositions 2 and 3 keep their normal limits. In their proofs the unnormalized summands are bounded by a constant multiple of $b/\\min(p,1-p)$, conditions C3 and C4 bound the neighborhoods (C4A), and each proposition assumes that $N$ times the variance is bounded below. Proposition 4 uses these limits and assumes the same bound. The observational extension in A.6 also states its limit under a lower bound on $N$ times the variance. The corrected lemma covers only neighborhoods of bounded size: I do not claim a version for neighborhoods whose sizes grow while $\\tilde c^2/N\\to0$. No estimand, simulation or reported number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E012", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 2, 'Under C1-C4, we have', with the displayed bound on the variance of the linearized Hajek estimator. Red boxes mark, in the second sum, the bold 1 in mu_i(1; d) and the exponent 2 that follows the closing bracket of the expectation.", "caption": "Supplement A, p. 9, the statement of Proposition 2. In the control term, the marked argument $\\mathbf 1$ should be $\\mathbf 0$, and the marked outer exponent should be absent.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E013.png"}, {"alt": "Crop of four lines of a displayed computation. Red boxes mark the terms 'minus (E[(mu_i(1; d) minus mu-bar-1(d)) squared]) squared' and 'minus (E[(mu_i(0; d) minus mu-bar-0(d)) squared]) squared'. The last line is plus 2 E[mu_i(1; d) minus mu-bar-1(d)] times E[mu_i(0; d) minus mu-bar-0(d)].", "caption": "Supplement A, p. 9, the first lines of the proof of Proposition 2. The two marked terms subtract the squares of second moments; the squares of the means are required.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E013-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T21", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T21"}, {"id": "P004-T43", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T43"}], "headline": "In the bound $\\bar{\\mathrm V}_{\\mathrm{HA}}(d)$ the control term is $\\mathrm E[(\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d))^2]/(1-p)$, with no outer square, and the proof subtracts squared means, not squared second moments.", "id": "P004-E013", "location": "Supplement A, A.4.2, Proposition 2, p. 9 (supplement PDF p. 9): the bound $\\bar{\\mathrm V}_{\\mathrm{HA}}(d)$ and the first lines of its proof", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T21", "P004-T43"], "text": {"as_printed": "``PROPOSITION 2. Under C1-C4, we have,\n\\[\n\\begin{aligned}\n&\\operatorname{Var}\\big(\\hat\\tau^{\\mathrm L}_{\\mathrm{HA}}(d)\\big)\\le\\bar{\\mathrm V}_{\\mathrm{HA}}(d)\\\\\n&\\quad=\\frac1{N^2}\\sum_{i=1}^Nc_i(d)\\frac{\\mathrm E[(\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d))^2]}{p}+\\frac1{N^2}\\sum_{i=1}^Nc_i(d)\\frac{\\mathrm E[(\\mu_i(\\hl{\\mathbf 1};d)-\\bar\\mu^0(d))^2]\\hl{{}^2}}{1-p}.\\text{''}\n\\end{aligned}\n\\]\nIn the proof, the variance of the summand of node $i$ is printed as\n\\[\n\\begin{aligned}\n={}&\\frac{\\mathrm E[(\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d))^2]}{p}-\\hl{(\\mathrm E[(\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d))^2])^2}\\\\\n&+\\frac{\\mathrm E[(\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d))^2]}{1-p}-\\hl{(\\mathrm E[(\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d))^2])^2}\\\\\n&+2\\mathrm E[\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d)]\\times\\mathrm E[\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d)]\n\\end{aligned}\n\\]", "corrected": "``PROPOSITION 2. Under C1-C4, we have,\n\\[\n\\begin{aligned}\n&\\operatorname{Var}\\big(\\hat\\tau^{\\mathrm L}_{\\mathrm{HA}}(d)\\big)\\le\\bar{\\mathrm V}_{\\mathrm{HA}}(d)\\\\\n&\\quad=\\frac1{N^2}\\sum_{i=1}^Nc_i(d)\\frac{\\mathrm E[(\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d))^2]}{p}+\\frac1{N^2}\\sum_{i=1}^Nc_i(d)\\frac{\\mathrm E[(\\mu_i(\\hl{\\mathbf 0};d)-\\bar\\mu^0(d))^2]}{1-p}.\\text{''}\n\\end{aligned}\n\\]\nIn the proof:\n\\[\n\\begin{aligned}\n={}&\\frac{\\mathrm E[(\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d))^2]}{p}-\\hl{(\\mathrm E[\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d)])^2}\\\\\n&+\\frac{\\mathrm E[(\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d))^2]}{1-p}-\\hl{(\\mathrm E[\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d)])^2}\\\\\n&+2\\mathrm E[\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d)]\\times\\mathrm E[\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d)]\n\\end{aligned}\n\\]", "evidence": "The evidence is two exact counterexamples and a derivation. Let $N=2$ and $p=1/4$, with no interference, $c_i(d)=1$, treated outcomes $0$ and control outcomes $+1$ and $-1$. Both marginal means equal $0$. The printed bound equals $0$, the variance of the linearized estimator equals $1/6$, and the corrected bound equals $2/3$. For the proof step, let $p=1/4$ and consider a node such that $\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d)=1/2$ and $\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d)=0$. The variance of its summand is $3/4$, and the printed expression equals $15/16$.\n\nFor the derivation, I write $a_i=\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d)$ and $b_i=\\mu_i(\\mathbf 0;d)-\\bar\\mu^0(d)$. Let $A_i=Z_ia_i/p$ and $B_i=(1-Z_i)b_i/(1-p)$, as on p.~10. Since $A_iB_i=0$, I have\n\\[\n\\operatorname{Var}(A_i-B_i)=\\frac{\\mathrm E[a_i^2]}{p}+\\frac{\\mathrm E[b_i^2]}{1-p}-\\big(\\mathrm E[a_i]-\\mathrm E[b_i]\\big)^2\\le\\frac{\\mathrm E[a_i^2]}{p}+\\frac{\\mathrm E[b_i^2]}{1-p}.\n\\]\nThe covariance of the summands of $i$ and $j$ vanishes for $j\\notin\\mathcal B(i;d)$ and is otherwise at most half the sum of the two variances. Since $j\\in\\mathcal B(i;d)$ if and only if $i\\in\\mathcal B(j;d)$, the variance of the linearized estimator is at most $N^{-2}\\sum_ic_i(d)\\operatorname{Var}(A_i-B_i)$, which is at most the corrected bound.", "problem": "The term for the control arm is printed with the treated outcome $\\mu_i(\\mathbf 1;d)$ and with its expectation squared. As printed, the bound is false. The first lines of the proof compute the variance of the summand of node $i$ and subtract the squares of two second moments where the squares of the two means are required. The continuation of the proof on p.~10 has the correct identity and the correct bound, and the estimator (24) on p.~9, which is equation (20) of the article, uses the control residual.", "what_survives": "The bound as corrected, the SAH estimator (24) and the inference results that rest on the bound are unaffected. The correct expressions are printed on p.~10 of Supplement A."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E013", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of equations (39) and (40). Equation (39) reads: plus one over N squared times the sum over i and over j in B of i and d with j not equal to i, of the sums over a and b from 0 to 1, of E of the product of mu sub i of a sub i, b sub j and d minus O sub i prime beta sub a,N, and mu sub j of a sub i, b sub j and d minus O sub i prime beta sub b,N. Equation (40) reads: minus the same sums of minus 1 to the a plus b times E of mu sub i of a and d minus O sub i prime beta sub a,N, times E of mu sub j of b and d minus O sub i prime beta sub b,N. Red outlines surround the letter E that opens the expectation in (39) and the symbol O sub i prime in the second factor of each equation.", "caption": "Supplement A, equations (39) and (40), p. 15. The outline at the expectation sign of (39) marks the place where the factor $(-1)^{a+b}$ is missing; the two other outlines mark $O_i'$ in the residual of node $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E014.png"}, {"alt": "Crop of a displayed inequality under the words In addition under C2. It bounds V of tau-hat-L sub IPW of d by the two diagonal sums of (37) plus one over N squared times the sum over i and over j in B of i and d with j not equal to i, of the sums over a and b, of E of the product of mu sub i of a sub i, b sub j and d minus O sub i prime beta sub a,N, and mu sub j of a sub i, b sub j and d minus O sub i prime beta sub b,N. Red outlines surround the letter E that opens this expectation and the symbol O sub i prime in its second factor.", "caption": "Supplement A, the bound on the variance, p. 16. The outlines mark the same two places in the repeated term.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E014-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T22", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T22"}], "headline": "The joint term (39) of the variance carries the factor $(-1)^{a+b}$, and in (39), in (40) and in the bound on p. 16 the residual of node $j$ is formed with $O_j$, not with $O_i$.", "id": "P004-E014", "location": "Supplement A, Appendix A.6, equations (39) and (40), p. 15, and the bound that repeats (39), p. 16", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T22"], "text": {"as_printed": "\\emph{Equation (39), repeated in the bound on p.~16.}\n\\[\n+\\frac{1}{N^2}\\sum_{i;\\,j\\in\\mathcal B(i;d),\\,j\\ne i}\\ \\sum_{a=0}^{1}\\sum_{b=0}^{1}\\hl{\\mathrm E}\\Bigl[\\bigl(\\mu_i(\\mathbf a_i,\\mathbf b_j;d)-O_i'\\beta_{a,N}\\bigr)\\bigl(\\mu_j(\\mathbf a_i,\\mathbf b_j;d)-\\hl{O_i'}\\beta_{b,N}\\bigr)\\Bigr]\n\\]\n\\emph{Equation (40).}\n\\[\n-\\frac{1}{N^2}\\sum_{i;\\,j\\in\\mathcal B(i;d),\\,j\\ne i}\\ \\sum_{a=0}^{1}\\sum_{b=0}^{1}(-1)^{a+b}\\,\\mathrm E\\bigl[\\mu_i(\\mathbf a;d)-O_i'\\beta_{a,N}\\bigr]\\,E\\bigl[\\mu_j(\\mathbf b;d)-\\hl{O_i'}\\beta_{b,N}\\bigr].\n\\]", "corrected": "\\emph{Equation (39), and the same term in the bound on p.~16.}\n\\[\n+\\frac{1}{N^2}\\sum_{i;\\,j\\in\\mathcal B(i;d),\\,j\\ne i}\\ \\sum_{a=0}^{1}\\sum_{b=0}^{1}\\hl{(-1)^{a+b}}\\,\\mathrm E\\Bigl[\\bigl(\\mu_i(\\mathbf a_i,\\mathbf b_j;d)-O_i'\\beta_{a,N}\\bigr)\\bigl(\\mu_j(\\mathbf a_i,\\mathbf b_j;d)-\\hl{O_j'}\\beta_{b,N}\\bigr)\\Bigr]\n\\]\n\\emph{Equation (40).}\n\\[\n-\\frac{1}{N^2}\\sum_{i;\\,j\\in\\mathcal B(i;d),\\,j\\ne i}\\ \\sum_{a=0}^{1}\\sum_{b=0}^{1}(-1)^{a+b}\\,\\mathrm E\\bigl[\\mu_i(\\mathbf a;d)-O_i'\\beta_{a,N}\\bigr]\\,E\\bigl[\\mu_j(\\mathbf b;d)-\\hl{O_j'}\\beta_{b,N}\\bigr].\n\\]", "evidence": "The evidence is a derivation and an exact example. I write $r_{ia}=\\mu_i(\\mathbf Y;d)-O_i'\\beta_{a,N}$ and $S_i=\\sum_{a=0}^{1}(2a-1)\\mathbf 1\\{Z_i=a\\}\\,r_{ia}/\\Pr(Z_i=a)$. Then the printed linear form is a constant plus $N^{-1}\\sum_iS_i$. For $i\\ne j$, the assignments $Z_i$ and $Z_j$ are independent, their joint probability cancels the two denominators, and I have\n\\[\nE[S_iS_j]=\\sum_{a=0}^{1}\\sum_{b=0}^{1}(-1)^{a+b}E\\Bigl[\\bigl(\\mu_i(\\mathbf a_i,\\mathbf b_j;d)-O_i'\\beta_{a,N}\\bigr)\\bigl(\\mu_j(\\mathbf a_i,\\mathbf b_j;d)-O_j'\\beta_{b,N}\\bigr)\\Bigr].\n\\]\nThe example has three nodes with probability $1/2$ and $O_i=1$, and the circle averages are $\\mu_1(\\mathbf Y;d)=\\mu_2(\\mathbf Y;d)=Z_3-1/2$ and $\\mu_3(\\mathbf Y;d)=0$. All arm means and both coefficients vanish. For the pair of nodes 1 and 2, each of the four conditional products equals $1/4$, so that the signed sum is 0 and the sum without signs is 1. Enumerating the eight assignments, I find that the variance of the linear form is $2/9$. The expression (37)--(40) as printed equals $4/9$, and with the sign in (39) it equals $2/9$.", "problem": "Equations (37)--(40) expand the variance of a contrast between the treated and the control arm. In such an expansion the term for arm $a$ at node $i$ and arm $b$ at node $j$ has the sign $(-1)^{a+b}$, as in Lemma A.4 (Supplement A, p.~5), to which the proof refers, and as in (40). Equation (39) and its repetition in the bound on p.~16 omit the sign. In addition, the residual of node $j$ is written with $O_i'\\beta_{b,N}$ in (39), in (40) and in the bound. The fitted value of node $j$ is $O_j'\\beta_{b,N}$.", "what_survives": "Lemma A.4 has the signed form, and the sign in (40) is correct. With the two corrections, (37)--(40) is the variance of the printed linear form of Proposition 4 of A.6. That linear form is itself replaced in \\fid{P004-E001}, where the fitted values $O_i'\\beta_{a,N}$ of these displays become $f_{ia}$."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E014", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the sentence 'A similar homophily condition is needed for the HAC variance estimation.' and of condition C 2, a displayed inequality: one over N times the sum over i of (tau_i minus (O_i' beta_1 minus O_i' beta_0)) times the sum over j in B(i; d) of (tau_j minus (O_i' beta_1 minus O_i' beta_0)) is at least 0. Red boxes mark the two O_i' in the second factor.", "caption": "Supplement A, p. 14, condition “C 2” of A.6. The two marked $O_i'$ stand in the factor that is summed over the neighbor $j$, where $O_j'$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E015.png"}, {"alt": "Crop of the definition of the N by 2 matrix X, the display V-hat-obs-HAC(d) = (X'X) inverse times a double sum times (X'X) inverse, and the line that defines epsilon-hat_i. Red boxes mark the left-hand side V-hat-obs-HAC(d) and the symbol O_j in each of the two denominators p(O_j | theta-hat_MLE) and 1 minus p(O_j | theta-hat_MLE).", "caption": "Supplement A, p. 15, the HAC-type estimator of A.6 and the residual $\\widehat\\epsilon_i$. The marked left-hand side names a $2\\times2$ matrix as a variance; the two marked $O_j$ stand in the weights of the residual of node $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E015-b.png"}, {"alt": "Crop of the line 'Further more, for each alpha less than one half,' and of items (i) and (ii), each a limit of the probability that the absolute value of a studentized estimator lies between z_(alpha/2) and z_(1 minus alpha/2). Red boxes mark tau-hat_HA(d) in the numerator of each item.", "caption": "Supplement A, p. 16, items (i) and (ii) of Proposition 4 of A.6. The two marked numerators carry $\\widehat\\tau_{\\mathrm{HA}}(d)$; the estimator defined and analyzed in A.6 is $\\widehat\\tau_{\\mathrm{IPW}}(d)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E015-c.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T23", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T23"}], "headline": "In A.6 the neighbor factor of “C 2” takes $O_j$, the residual $\\widehat\\epsilon_i$ takes $p(O_i|\\widehat\\theta_{\\mathrm{MLE}})$, the scalar HAC variance is the $(2,2)$ entry of the displayed matrix, and the intervals are for $\\widehat\\tau_{\\mathrm{IPW}}$.", "id": "P004-E015", "location": "Supplement A, A.6: condition “C 2”, p. 14; the display of $\\widehat{\\mathrm V}^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)$ and the residual $\\widehat\\epsilon_i$, p. 15; items (i)–(ii) of Proposition 4, p. 16 (supplement PDF pp. 14–16)", "pdf_page": 14, "printed_page": "14", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T23"], "text": {"as_printed": "P.~14: ``C 2. For all sample size $N$,\n\\[\n\\frac1N\\sum_{i=1}^N\\Big(\\tau_i(d;\\eta)-\\big(O_i'\\beta_{1,N}-O_i'\\beta_{0,N}\\big)\\Big)\\sum_{j\\in\\mathcal B(i;d)}\\Big(\\tau_j(d;\\eta)-\\big(\\hl{O_i'}\\beta_{1,N}-\\hl{O_i'}\\beta_{0,N}\\big)\\Big)\\ge0.\\text{''}\n\\]\nP.~15:\n\\[\n\\hl{\\widehat{\\mathrm V}^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)}=(\\mathbf X'\\mathbf X)^{-1}\\left(\\sum_{i=1}^N\\sum_{j=1}^N\\mathbf X_i\\mathbf X_j'\\hat\\epsilon_i\\hat\\epsilon_j\\mathbf 1\\{j\\in\\mathcal B(i;d)\\}\\right)(\\mathbf X'\\mathbf X)^{-1},\n\\]\n``where $\\widehat\\epsilon_i=\\frac{N_1}{N}\\frac{Z_i}{p(\\hl{O_j}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{1,N}\\big)+\\frac{N_0}{N}\\frac{1-Z_i}{1-p(\\hl{O_j}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{0,N}\\big)$.''\n\nP.~16:\n\\[\n\\begin{aligned}\n&\\text{``(i) }\\lim_{N\\to\\infty}\\mathbf{Prob}\\Bigg(z_{\\frac\\alpha2}\\le\\Bigg|\\frac{\\hl{\\widehat\\tau_{\\mathrm{HA}}(d)}-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}(d)}}\\Bigg|\\le z_{1-\\frac\\alpha2}\\Bigg)\\ge1-\\alpha;\\\\\n&\\text{(ii) additionally under C2, }\\lim_{N\\to\\infty}\\mathbf{Prob}\\Bigg(z_{\\frac\\alpha2}\\le\\Bigg|\\frac{\\hl{\\widehat\\tau_{\\mathrm{HA}}(d)}-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat V^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)}}\\Bigg|\\le z_{1-\\frac\\alpha2}\\Bigg)\\ge1-\\alpha.\\text{''}\n\\end{aligned}\n\\]", "corrected": "P.~14: ``C 2. For all sample size $N$,\n\\[\n\\frac1N\\sum_{i=1}^N\\Big(\\tau_i(d;\\eta)-\\big(O_i'\\beta_{1,N}-O_i'\\beta_{0,N}\\big)\\Big)\\sum_{j\\in\\mathcal B(i;d)}\\Big(\\tau_j(d;\\eta)-\\big(\\hl{O_j'}\\beta_{1,N}-\\hl{O_j'}\\beta_{0,N}\\big)\\Big)\\ge0.\\text{''}\n\\]\nP.~15:\n\\[\n\\begin{aligned}\n{}\\hl{\\widehat\\Sigma^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)}&=(\\mathbf X'\\mathbf X)^{-1}\\left(\\sum_{i=1}^N\\sum_{j=1}^N\\mathbf X_i\\mathbf X_j'\\hat\\epsilon_i\\hat\\epsilon_j\\mathbf 1\\{j\\in\\mathcal B(i;d)\\}\\right)(\\mathbf X'\\mathbf X)^{-1},\\\\\n{}\\hl{\\widehat{\\mathrm V}^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)}&{}\\hl{{}=w'\\widehat\\Sigma^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)\\,w,\\quad w=(0,1)',}\n\\end{aligned}\n\\]\nthat is, the $(2,2)$ entry of the matrix, ``where $\\widehat\\epsilon_i=\\frac{N_1}{N}\\frac{Z_i}{p(\\hl{O_i}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{1,N}\\big)+\\frac{N_0}{N}\\frac{1-Z_i}{1-p(\\hl{O_i}|\\widehat\\theta_{\\mathrm{MLE}})}\\big(\\mu_i(\\mathbf Y;d)-O_i'\\widehat\\beta_{0,N}\\big)$.''\n\nP.~16:\n\\[\n\\begin{aligned}\n&\\text{``(i) }\\lim_{N\\to\\infty}\\mathbf{Prob}\\Bigg(z_{\\frac\\alpha2}\\le\\Bigg|\\frac{\\hl{\\widehat\\tau_{\\mathrm{IPW}}(d)}-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat V^{\\mathrm{obs}}_{\\mathrm{SAH}}(d)}}\\Bigg|\\le z_{1-\\frac\\alpha2}\\Bigg)\\ge1-\\alpha;\\\\\n&\\text{(ii) additionally under C2, }\\lim_{N\\to\\infty}\\mathbf{Prob}\\Bigg(z_{\\frac\\alpha2}\\le\\Bigg|\\frac{\\hl{\\widehat\\tau_{\\mathrm{IPW}}(d)}-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat V^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)}}\\Bigg|\\le z_{1-\\frac\\alpha2}\\Bigg)\\ge1-\\alpha.\\text{''}\n\\end{aligned}\n\\]", "evidence": "The evidence is a check of pp.~14--16 of Supplement A and a short derivation. With $h_i=\\tau_i(d;\\eta)-O_i'(\\beta_{1,N}-\\beta_{0,N})$, the homophily condition is a sum of products $h_ih_j$, and the second factor must contain $O_j$. The weight of the residual of node $i$ is the propensity of node $i$, as in the IPW estimator on p.~14. Since the matrix $\\mathbf X$ is $N$-by-$2$, the displayed matrix is $2\\times2$; the variance of the treatment contrast is $w'\\Sigma w=\\Sigma_{22}$ with $w=(0,1)'$, as on p.~755 of the article. The limit stated directly before item (i) is for $\\widehat\\tau_{\\mathrm{IPW}}$.", "problem": "Four displays of A.6 carry a wrong index or label. In ``C 2'' the factor summed over the neighbor $j$ repeats $O_i$ where the covariate of node $j$ is meant. In $\\widehat\\epsilon_i$ the propensity is written with $O_j$, although no $j$ is bound there and the residual belongs to node $i$. The display named $\\widehat{\\mathrm V}^{\\mathrm{obs}}_{\\mathrm{HAC}}(d)$ is a $2\\times2$ matrix, while items (i)--(ii) need the scalar variance of the treatment contrast. Items (i)--(ii) are written for $\\widehat\\tau_{\\mathrm{HA}}$, while A.6 defines and analyzes $\\widehat\\tau_{\\mathrm{IPW}}$. The absolute value in (i)--(ii) is harmless: since $z_{\\alpha/2}<0$, the lower inequality always holds, and the event is the usual symmetric interval.", "what_survives": "These corrections concern notation only, and nothing in the article's experimental results changes. They do not repair the linearization of Proposition 4 of A.6 (\\fid{P004-E001}), its covariance displays (\\fid{P004-E014}) or its SAH display (\\fid{P004-E002}). They also do not establish an interval for the Hajek estimator in the observational setting; such a claim would need its own argument."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E015", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of three lines of text and equation (44). The text reads: Define lambda equals w-prime times the inverse of X-prime X, times X, in R to the 1 by N; M equals I minus X-prime times the inverse of X-prime X, times X, in R to the N by N; and the weight matrix Omega in R to the N by N where Omega sub ij is the indicator that j is in B of i and d. Young (2015)'s effective degree of freedom adjustment in our setting is calculated as. Equation (44) defines mu as a trace. Red outlines surround the expression for lambda and the expression for M.", "caption": "Supplement A, Appendix A.7, p. 17. The outlines mark the two products with misplaced transposes.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E016.png"}, {"alt": "Crop of equation (45) and two lines of text. Equation (45) defines nu as 2 times the trace of the square of the matrix in (44). The text reads: We use the alpha over 2 quantile and the 1 minus alpha over 2 quantile of the t-distribution with 2 mu squared over v degree of freedom to construct the confidence interval. A red outline surrounds the letter v in the denominator.", "caption": "Supplement A, equation (45) and the last sentence of Appendix A.7, p. 17. The outline marks the denominator $v$, where the $\\nu$ of (45) is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E016-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T24", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T24"}], "headline": "In Appendix A.7 the transposes are on the wrong factors: $\\lambda=w'(X'X)^{-1}X'$ and $M=I-X(X'X)^{-1}X'$; the degrees of freedom are $2\\mu^2/\\nu$, with the $\\nu$ of (45).", "id": "P004-E016", "location": "Supplement A, Appendix A.7, the sentence “Define $\\lambda=\\dots$” before equation (44) and the last sentence of the section, p. 17", "pdf_page": 17, "printed_page": "17", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T24"], "text": {"as_printed": "Define $\\lambda=\\mathbf w'(\\mathbf X'\\mathbf X)^{-1}\\hl{\\mathbf X}\\in\\mathbb R^{1\\times N}$, $\\mathbf M=\\mathbf I-\\hl{\\mathbf X'}(\\mathbf X'\\mathbf X)^{-1}\\hl{\\mathbf X}\\in\\mathbb R^{N\\times N}$, and the weight matrix $\\Omega\\in\\mathbb R^{N\\times N}$ where $\\Omega_{ij}=\\mathbf 1\\{j\\in\\mathcal B(i;d)\\}$.\n\nWe use the $\\frac\\alpha2$-quantile and $\\bigl(1-\\frac\\alpha2\\bigr)$-quantile of the t-distribution with $\\frac{2\\mu^2}{\\hl{v}}$ degree of freedom to construct the confidence interval.", "corrected": "Define $\\lambda=\\mathbf w'(\\mathbf X'\\mathbf X)^{-1}\\hl{\\mathbf X'}\\in\\mathbb R^{1\\times N}$, $\\mathbf M=\\mathbf I_N-\\hl{\\mathbf X}(\\mathbf X'\\mathbf X)^{-1}\\hl{\\mathbf X'}\\in\\mathbb R^{N\\times N}$, and the weight matrix $\\Omega\\in\\mathbb R^{N\\times N}$ where $\\Omega_{ij}=\\mathbf 1\\{j\\in\\mathcal B(i;d)\\}$.\n\nWe use the $\\frac\\alpha2$-quantile and $\\bigl(1-\\frac\\alpha2\\bigr)$-quantile of the t-distribution with $\\frac{2\\mu^2}{\\hl{\\nu}}$ degree of freedom to construct the confidence interval.", "evidence": "The evidence is a check of the dimensions. With $X\\in\\mathbb R^{N\\times2}$, the row $w'(X'X)^{-1}X'$ has dimension $1\\times N$, and $I_N-X(X'X)^{-1}X'$ has dimension $N\\times N$, is symmetric and idempotent, and annihilates $X$. The package in the replication files uses these forms: \\texttt{z\\_vec <- c(0, 1) \\%*\\% XX\\_mat\\_inv \\%*\\% t(X\\_mat)} and \\texttt{M\\_mat <- diag(1, nz, nz) - X\\_mat \\%*\\% XX\\_mat\\_inv \\%*\\% t(X\\_mat)} (\\texttt{SpatialEffect/R/SpatialEffect.R}, lines 196--197).", "problem": "The matrix $X$ has dimension $N\\times2$ and $w=(0,1)'$. The product $w'(X'X)^{-1}$ is a row of length 2 and cannot multiply $X$, and $X'(X'X)^{-1}X$ is not a product of conforming matrices for general $N$. The dimensions stated in the sentence, $1\\times N$ and $N\\times N$, identify what is meant: the row of weights whose product with the outcomes is the coefficient of the treatment in the regression, and the matrix that maps the outcomes to the residuals. In the last sentence, $v$ stands for the $\\nu$ defined in (45).", "what_survives": "Equations (44) and (45) are as intended once $\\lambda$ and $M$ are read in the corrected form, and the code computes them from the corrected matrices. This correction concerns the two definitions and the symbol only. It does not bear on the accuracy of the adjustment in finite samples."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E016", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of page 26 of Supplement A with the definition of tau hat i N set in the text, the sentence Note E of tau hat i N equals 0, the statement of Lemma C.2 and equation (57). Red boxes mark the last term of the definition, minus tau i of d and eta, and the last term of equation (57), minus tau i of d.", "caption": "Supplement A, p. 26: the definition of $\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)$ before Lemma C.2, and equation (57). The two marked terms lack the factor $1/N$ that every other term carries.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E017.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T25", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T25"}], "headline": "The centring term of the summand $\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)$ is $\\tau_i(d;\\eta)/N$ and not $\\tau_i(d;\\eta)$, both in the definition before Lemma C.2 and in equation (57).", "id": "P004-E017", "location": "Supplement A, Appendix C, definition of $\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)$ before Lemma C.2 and equation (57), p. 26 (PDF p. 26 of the supplement)", "pdf_page": 26, "printed_page": "26", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T25"], "text": {"as_printed": "The following lemma states that provided $\\{\\mu_i(\\mathbf Y;d),i\\in\\mathcal S_N\\}$ is uniform $L_2$-NED with size $-4$, so is the transformation\n\\[\n\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)=\\Big(\\frac{\\mathbf Z_i}{Np}-\\frac{(1-\\mathbf Z_i)}{N(1-p)}\\Big)\\mu_i(\\mathbf Y;d)+\\frac1N(\\mu_1-\\mu_0)-\\Big(\\frac{\\mathbf Z_i\\mu_1}{Np}-\\frac{(1-\\mathbf Z_i)\\mu_0}{N(1-p)}\\Big)\\hl{-\\tau_i(d;\\eta)}.\n\\]\nNote $\\mathrm E[\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)]=0$.\n\nProof. Note $\\mathrm E[\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)|\\mathcal F_i(s)]$ is equal to:\n\\[\n\\Big(\\frac{\\mathbf Z_i}{Np}-\\frac{(1-\\mathbf Z_i)}{N(1-p)}\\Big)\\mathrm E[\\mu_i(\\mathbf Y;d)|\\mathcal F_i(s)]+\\frac1N(\\mu_1-\\mu_0)-\\Big(\\frac{\\mathbf Z_i\\mu_1}{Np}-\\frac{(1-\\mathbf Z_i)\\mu_0}{N(1-p)}\\Big)\\hl{-\\tau_i(d)},\\qquad\\text{(57)}\n\\]", "corrected": "The following lemma states that provided $\\{\\mu_i(\\mathbf Y;d),i\\in\\mathcal S_N\\}$ is uniform $L_2$-NED with size $-4$, so is the transformation\n\\[\n\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)=\\Big(\\frac{\\mathbf Z_i}{Np}-\\frac{(1-\\mathbf Z_i)}{N(1-p)}\\Big)\\mu_i(\\mathbf Y;d)+\\frac1N(\\mu_1-\\mu_0)-\\Big(\\frac{\\mathbf Z_i\\mu_1}{Np}-\\frac{(1-\\mathbf Z_i)\\mu_0}{N(1-p)}\\Big)-\\hl{\\frac1N}\\tau_i(d;\\eta).\n\\]\nNote $\\mathrm E[\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)]=0$ under C1, for each pair $\\mu_1,\\mu_0\\in\\mathbb R$.\n\nProof. Note $\\mathrm E[\\widehat\\tau_{i,N}(\\mu_1,\\mu_0;d)|\\mathcal F_i(s)]$ is equal to:\n\\[\n\\Big(\\frac{\\mathbf Z_i}{Np}-\\frac{(1-\\mathbf Z_i)}{N(1-p)}\\Big)\\mathrm E[\\mu_i(\\mathbf Y;d)|\\mathcal F_i(s)]+\\frac1N(\\mu_1-\\mu_0)-\\Big(\\frac{\\mathbf Z_i\\mu_1}{Np}-\\frac{(1-\\mathbf Z_i)\\mu_0}{N(1-p)}\\Big)-\\hl{\\frac1N}\\tau_i(d),\\qquad\\text{(57)}\n\\]", "evidence": "The evidence is a derivation and an exact example. Let $m_{iz}=\\mathrm E[\\mu_i(\\mathbf Y;d)\\mid\\mathbf Z_i=z]$. Under the independent Bernoulli assignments of C1, I have $\\tau_i(d;\\eta)=m_{i1}-m_{i0}$ and\n\\[\n\\mathrm E\\Big[\\Big(\\frac{\\mathbf Z_i}{Np}-\\frac{1-\\mathbf Z_i}{N(1-p)}\\Big)\\mu_i(\\mathbf Y;d)\\Big]=\\frac{m_{i1}-m_{i0}}{N}=\\frac{\\tau_i(d;\\eta)}{N},\\qquad\n\\mathrm E\\Big[\\frac{\\mathbf Z_i\\mu_1}{Np}-\\frac{(1-\\mathbf Z_i)\\mu_0}{N(1-p)}\\Big]=\\frac{\\mu_1-\\mu_0}{N}.\n\\]\nThe second expectation cancels the constant $(\\mu_1-\\mu_0)/N$. The printed summand therefore has mean $\\tau_i(d;\\eta)/N-\\tau_i(d;\\eta)$, and the summand with $\\tau_i(d;\\eta)/N$ subtracted has mean zero. For the example, suppose that there is no interference and that $\\mu_i(\\mathbf Y;d)=\\mathbf Z_i$, with $N=8$, $p=1/2$ and $\\mu_1=\\mu_0=0$. Then $\\tau_i(d;\\eta)=1$, the printed summand has mean $-7/8$, and the corrected summand has mean $0$. By (56) and the definition (9) of the AME in the article, the corrected summands sum to $\\widehat\\tau(\\mu_1,\\mu_0;d)-\\mathrm{AME}(d;\\eta)$.", "problem": "The summand is meant to have mean zero and order $1/N$, and the sentence after the definition asserts that its mean is zero. Every other term of the definition carries the factor $1/N$, and the last term does not. As printed, the mean of the summand is $(1/N-1)\\tau_i(d;\\eta)$. The same term appears without the factor in (57). The proofs of Lemmas C.3 and C.7 use summands of mean zero and order $1/N$, and they need the corrected definition.", "what_survives": "Lemma C.2 and the bound (58)--(59) are unchanged, since the centring term is deterministic and cancels in the difference between $\\widehat\\tau_{i,N}$ and its conditional expectation. Covariances are unchanged for the same reason. The asymptotic arguments of Appendix C that use these summands are unaffected once the definition is corrected."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E017", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of assumption C 4 and the two sentences that introduce it. Red boxes mark s greater than or equal to 0 and the formula b of s equals C b times s squared.", "caption": "Supplement A, p. 27: assumption C4 of Appendix C. The marked range of radii includes $s=0$, where the marked bound $C_bs^2$ equals zero.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E018.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T26", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T26"}, {"id": "P004-T27", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T27"}, {"id": "P004-T28", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T28"}, {"id": "P004-T29", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T29"}, {"id": "P004-T44", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T44"}], "headline": "Assumption C4 of Appendix C is a quadratic bound on the number of nodes within distance $s$ of a node and is meant for radii $s\\ge1$; at smaller radii, down to the printed $s\\ge0$, the ball still contains its own centre and the bound $C_bs^2$ cannot hold.", "id": "P004-E018", "location": "Supplement A, Appendix C, assumption C4, p. 27 (PDF p. 27 of the supplement); used in equations (69)–(73), p. 28", "pdf_page": 27, "printed_page": "27", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P004-T26", "P004-T27", "P004-T28", "P004-T29", "P004-T44"], "text": {"as_printed": "C 4. For each radius $\\hl{s\\ge0}$ and all large sample sizes $N$, there exists a $C_b>0$ such that $\\sup_{i\\in\\mathcal S_N}|\\mathcal B_N(i;s)|\\le b(s)$, with $\\hl{b(s)=C_bs^2}$.", "corrected": "C 4. \\hl{Let} $\\hl{\\mathcal B_N(i;s)=\\{j\\in\\mathcal S_N:d_{ij}\\le s\\}}$\\hl{.} There exists \\hl{one constant} $C_b>0$\\hl{, the same for all radii and all sample sizes,} such that for each radius $\\hl{s\\ge1}$ and all large sample sizes $N$, $\\sup_{i\\in\\mathcal S_N}|\\mathcal B_N(i;s)|\\le C_bs^2$. In the displays (69)--(73), $b(0)=0$ and $b(k)=C_bk^2$ for integers $k\\ge1$.", "intended_meaning": "The assumption is meant to bound the number of nodes within distance $s$ of any node by a constant times $s^2$, with one constant for all radii and all sample sizes. The counting arguments in (69)--(73) and (76)--(79) use nothing more, and the bound holds for nodes that are spaced apart: a square grid with unit spacing satisfies it with $C_b=9$. The printed range of radii is too wide. At $s=0$ the set $\\mathcal B_N(i;0)$ contains $i$, and its size is at least $1$, while $b(0)=0$. Two further points concern the notation. First, $\\mathcal B_N(i;s)$ here denotes the ball $\\{j\\in\\mathcal S_N:d_{ij}\\le s\\}$, which the article writes $\\mathcal S_N(i;s)$, while in equation (12) of the article the same symbol denotes a neighbourhood of radius $h(d)$. Second, note that the constant $C_b$ must not depend on $s$ or $N$. With the wording below, the weighted bound (69) holds with the formal values $b(0)=0$ and $b(k)=C_bk^2$ for integers $k\\ge1$. This follows by summation by parts applied to the number of nodes within distance $k+1$ of a node, which is at most $C_b(k+1)^2$."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E018", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the statement of Lemma C.10 with display (123) and the concluding display. A red box marks the subscript j in B of i and b N under the sum in the second hypothesis.", "caption": "Supplement A, p. 35: the statement of Lemma C.10. The marked index set puts the sum over the nodes inside the ball $\\mathcal B(i,b_N)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E019.png"}, {"alt": "Crop of the last two lines of the chain of displays in the proof of Lemma C.10. A red box marks the subscript j not in B of i and b tilde N under the final sum, which is followed by equals o of 1.", "caption": "Supplement A, p. 35: the last two lines of the proof of Lemma C.10. The marked index set puts the sum over the nodes outside the ball $\\mathcal B(i,\\tilde b_N)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E019-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T26", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T26"}], "headline": "The second hypothesis of Lemma C.10 should bound the sum of $\\phi_d(d_{ij})$ over the nodes $j$ outside the ball $\\mathcal B(i,b_N)$, as its proof and Remark 1 require, and not over the nodes inside it.", "id": "P004-E019", "location": "Supplement A, Appendix C, Lemma C.10, second hypothesis after (123), and the proof of the lemma, p. 35 (PDF p. 35 of the supplement)", "pdf_page": 35, "printed_page": "35", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T26"], "text": {"as_printed": "Lemma C.10. Under C2, C4 and C5, and suppose for all large $N$ and any pair $i$ and $j$ we have\n\\[\n|\\tau_i(d)-\\mathrm{AME}(d;\\eta)|\\times|\\tau_j(d)-\\mathrm{AME}(d;\\eta)|\\le\\phi_d(d_{ij}),\\qquad\\text{(123)}\n\\]\nand $\\lim_N\\sup_{i\\in\\mathcal S_N}\\sum_{\\hl{\\scriptstyle j\\in\\mathcal B(i,b_N)}}\\phi_d(d_{ij})\\to0$ for any $b_N\\to\\infty$. We have\n\\[\n\\liminf_N\\Big(\\frac1N\\sum_{i=1}^N\\sum_{j=1}^NK(\\frac{d_{ij}}{b_N})(\\tau_i(d)-\\mathrm{AME}(d;\\eta))(\\tau_j(d)-\\mathrm{AME}(d;\\eta))\\Big)=0.\n\\]\nProof. [\\ldots] Let $\\tilde b_N=o(b_N^{\\frac13})$. Then, for large $N$, [\\ldots]\n\\[\n\\lesssim\\lim_N\\frac{\\tilde b_N^3}{b_N}+\\lim_N\\sup_{i\\in\\mathcal S_N}\\sum_{j\\notin\\mathcal B(i,\\tilde b_N)}|(\\tau_i(d)-\\mathrm{AME}(d;\\eta))(\\tau_j(d)-\\mathrm{AME}(d;\\eta))|=o(1).\n\\]", "corrected": "Lemma C.10. Under C2, C4 and C5, and suppose for all large $N$ and any pair $i$ and $j$ we have\n\\[\n|\\tau_i(d)-\\mathrm{AME}(d;\\eta)|\\times|\\tau_j(d)-\\mathrm{AME}(d;\\eta)|\\le\\phi_d(d_{ij}),\\qquad\\text{(123)}\n\\]\nand $\\lim_N\\sup_{i\\in\\mathcal S_N}\\sum_{\\hl{\\scriptstyle j\\notin\\mathcal B(i,b_N)}}\\phi_d(d_{ij})\\to0$ for any $b_N\\to\\infty$. We have\n\\[\n\\liminf_N\\Big(\\frac1N\\sum_{i=1}^N\\sum_{j=1}^NK(\\frac{d_{ij}}{b_N})(\\tau_i(d)-\\mathrm{AME}(d;\\eta))(\\tau_j(d)-\\mathrm{AME}(d;\\eta))\\Big)=0.\n\\]\nProof. [\\ldots] Let $\\tilde b_N=\\hl{b_N^{\\frac16}}$\\hl{, so that} $\\hl{\\tilde b_N\\to\\infty}$ \\hl{and} $\\hl{\\tilde b_N^3/b_N\\to0}$. Then, for large $N$, [\\ldots]", "evidence": "The evidence is a check of the page and a derivation of both versions. On p.~35 the hypothesis is printed with $j\\in\\mathcal B(i,b_N)$ and the last line of the proof with $j\\notin\\mathcal B(i,\\tilde b_N)$. Let $a_i=\\tau_i(d)-\\mathrm{AME}(d;\\eta)$. For the printed version, recall that $K$ is supported on $[0,1]$ and bounded by $K_{\\max}$. By (123), I have\n\\[\n\\Big|\\frac1N\\sum_{i=1}^N\\sum_{j=1}^NK\\Big(\\frac{d_{ij}}{b_N}\\Big)a_ia_j\\Big|\\le K_{\\max}\\sup_{i\\in\\mathcal S_N}\\sum_{j\\in\\mathcal B(i,b_N)}\\phi_d(d_{ij}),\n\\]\nwhich tends to zero. For the corrected version, let $\\tilde b_N=b_N^{1/6}$. Since $\\sum_ia_i=0$, the double sum is unchanged when $K(d_{ij}/b_N)$ is replaced by $K(d_{ij}/b_N)-1$. For $d_{ij}\\le\\tilde b_N$ and large $N$, C5 implies that $|K(d_{ij}/b_N)-1|\\le C\\tilde b_N/b_N$, C2 bounds $|a_ia_j|$, and C4 bounds the number of such $j$ by $C_b\\tilde b_N^2$. This part is at most a constant times $\\tilde b_N^3/b_N=b_N^{-1/2}$. For $d_{ij}>\\tilde b_N$, I have $|K(d_{ij}/b_N)-1|\\le K_{\\max}+1$ and $|a_ia_j|\\le\\phi_d(d_{ij})$. This part is at most $(K_{\\max}+1)\\sup_i\\sum_{j\\notin\\mathcal B(i,\\tilde b_N)}\\phi_d(d_{ij})$, which tends to zero by the corrected hypothesis applied to the diverging sequence $\\tilde b_N$.", "problem": "As printed, the hypothesis bounds the sum of $\\phi_d(d_{ij})$ over the nodes inside the growing ball. That sum contains the term $j=i$, and (123) then implies that $\\sup_i(\\tau_i(d)-\\mathrm{AME}(d;\\eta))^2$ is at most $\\sup_i\\sum_{j\\in\\mathcal B(i,b_N)}\\phi_d(d_{ij})$, which tends to zero. With the single function $\\phi_d$ of the statement, the condition forces $\\phi_d(0)=0$ and hence $\\tau_i(d)=\\mathrm{AME}(d;\\eta)$ for every $i$ and all large $N$. The effects are then homogeneous. If $\\phi_d$ were allowed to vary with $N$, the effects would be homogeneous asymptotically, uniformly in $i$. The conclusion of the lemma holds in that case. The printed lemma is true, but it covers only this case. Remark 1 announces a condition on heterogeneity ``when nodes are far apart'', and the last line of the proof uses the sum over $j\\notin\\mathcal B(i,\\tilde b_N)$, which the printed hypothesis does not control. Note that the proof also sets $\\tilde b_N=o(b_N^{1/3})$ without requiring that $\\tilde b_N$ diverge, which the bound on the outside sum needs.", "what_survives": "The conclusion of Lemma C.10 holds under either hypothesis, and C6 and Proposition 6 are unaffected. The corrected hypothesis restores the case of heterogeneous effects that Remark 1 describes. Its proof uses C4 as worded in \\fid{P004-E018}."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E019", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the two sentences that introduce Proposition 6 and of its statement, with parts (i) to (v). Red boxes mark the phrase Under C1, C2, C3, C4, C5 and C6 and the symbol lim as N tends to infinity at the start of part (v).", "caption": "Supplement A, p. 35: Proposition 6. The marked list of assumptions contains no lower bound on $N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$, and the marked conclusion of part (v) is a limit.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E020.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T27", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T27"}, {"id": "P004-T28", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T28"}], "headline": "Part (v) of Proposition 6 needs the condition that $N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$ stays bounded away from zero, which Proposition 5 imposes, and its conclusion should be stated with a limit inferior.", "id": "P004-E020", "location": "Supplement A, Appendix C, Proposition 6, the list of assumptions and part (v), p. 35 (PDF p. 35 of the supplement); compare Proposition 5, p. 29", "pdf_page": 35, "printed_page": "35", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T27", "P004-T28"], "text": {"as_printed": "Proposition 6. Let $b_N=o(N^{\\frac16})$ and $b_N\\to\\infty$. \\hl{Under C1, C2, C3, C4, C5 and C6,}\n\n(i) $N\\times\\big(\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d;b_N)-\\widehat{\\mathrm V}_1(d,b_N)\\big)=o_p(1)$.\n\n(ii) $N\\times\\big(\\widehat{\\mathrm V}_1(d;b_N)-\\widehat{\\mathrm V}_2(d,b_N)\\big)=o_p(1)$.\n\n(iii) $N\\times\\big(\\widehat{\\mathrm V}_2(d;b_N)-\\mathrm E[\\widehat{\\mathrm V}_2(d;b_N)]\\big)=o_p(1)$.\n\n(iv) $\\liminf_N\\big(N\\times\\big(\\mathrm E[\\widehat{\\mathrm V}_2(d;b_N)]-\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))\\big)\\big)\\ge0$.\n\n(v) $\\hl{\\lim_{N\\to\\infty}}\\operatorname{Prob}\\Big(z_{\\frac\\alpha2}\\le\\dfrac{\\widehat\\tau_{\\mathrm{HA}}(d)-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d,b_N)}}\\le z_{1-\\frac\\alpha2}\\Big)\\ge1-\\alpha$", "corrected": "Proposition 6. Let $b_N=o(N^{\\frac16})$ and $b_N\\to\\infty$. Under C1, C2, C3, C4, C5 and C6,\n\n(i) $N\\times\\big(\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d;b_N)-\\widehat{\\mathrm V}_1(d,b_N)\\big)=o_p(1)$.\n\n(ii) $N\\times\\big(\\widehat{\\mathrm V}_1(d;b_N)-\\widehat{\\mathrm V}_2(d,b_N)\\big)=o_p(1)$.\n\n(iii) $N\\times\\big(\\widehat{\\mathrm V}_2(d;b_N)-\\mathrm E[\\widehat{\\mathrm V}_2(d;b_N)]\\big)=o_p(1)$.\n\n(iv) $\\liminf_N\\big(N\\times\\big(\\mathrm E[\\widehat{\\mathrm V}_2(d;b_N)]-\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))\\big)\\big)\\ge0$.\n\n(v) \\hl{If in addition} $\\hl{\\liminf_{N\\to\\infty}N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))>0}$\\hl{, then for each} $\\hl{0<\\alpha<1}$\\hl{,}\n\\[\n\\hl{\\liminf_{N\\to\\infty}}\\operatorname{Prob}\\Big(z_{\\frac\\alpha2}\\le\\frac{\\widehat\\tau_{\\mathrm{HA}}(d)-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d,b_N)}}\\le z_{1-\\frac\\alpha2}\\Big)\\ge1-\\alpha.\n\\]\nHere C4 is read as worded in \\fid{P004-E018}.", "evidence": "The evidence is an exact counterexample to part (v), a second example for the limit, and a derivation of the corrected statement. Both examples satisfy C4 as worded in \\fid{P004-E018}. With C4 exactly as printed, no nonempty set of nodes satisfies the assumptions, and the failure concerns the proposition as it is meant.\n\nFor each $N\\ge3$, let the nodes be the points $(iN^2,0)$ for $1\\le i\\le N$, with independent assignments of probability $p=1/2$. Let the outcome points be the nodes themselves, with counting measure, and let the distance be $d=1$, with $\\Omega_d=[0,d]$, so that each disc contains its own node only. Let $Y_1(\\mathbf z)=1/4+(3/4)z_1$ and $Y_i(\\mathbf z)=0$ for $i>1$. Let $K(u)=\\mathbf 1\\{0\\le u\\le1\\}$ and $b_N=\\lfloor N^{1/12}\\rfloor$. Then C1, C2, C3 and C5 hold, a ball of radius $s\\ge1$ contains at most $2s+1\\le3s^2$ nodes, and the expression in C6 equals $9(N-1)/(16N^2)\\ge0$. A direct computation gives\n\\[\nN\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))=\\frac{25(N-1)}{16N^2},\n\\]\nwhich tends to zero, and parts (i)--(iv) hold. Since the spacing $N^2$ exceeds $b_N$, the estimator $\\widehat{\\mathrm V}_{\\mathrm{HAC}}$ has no cross terms. Let $q_N$ be the number of nodes in the arm that contains node 1. Outside four assignments, of total probability $4/2^N$, I have\n\\[\n\\widehat\\tau_{\\mathrm{HA}}(d)=\\frac1{q_N},\\quad\\widehat{\\mathrm V}_{\\mathrm{HAC}}=\\frac{q_N-1}{q_N^3}\\quad\\text{if }Z_1=1,\\qquad\\widehat\\tau_{\\mathrm{HA}}(d)=-\\frac1{4q_N},\\quad\\widehat{\\mathrm V}_{\\mathrm{HAC}}=\\frac{q_N-1}{16q_N^3}\\quad\\text{if }Z_1=0.\n\\]\nHere $\\mathrm{AME}(d;\\eta)=3/(4N)$, and $q_N/N$ tends to $1/2$ in probability. The studentized statistic of part (v) then tends in probability to $5/8$ given $Z_1=1$ and to $-5/2$ given $Z_1=0$. Since $5/8<z_{0.975}<5/2$, the coverage of the nominal 95\\% interval tends to $1/2$. The table lists the exact coverage for the critical values $\\pm2$.\n\n\\begin{audittable}{rl}{Exact coverage probability in the first example of the interval with critical values $\\pm2$. The four assignments on which the statistic is undefined are counted as failures to cover.}\n$N$ & Coverage \\\\\n4 & $3/8=0.375$ \\\\\n8 & $63/128\\approx0.4922$ \\\\\n16 & $0.5018$ \\\\\n32 & $0.5027$ \\\\\n64 & $0.5006$ \\\\\n128 & $0.500009$ \\\\\n256 & $0.500000007$ \\\\\n\\end{audittable}\n\nThe second example has the same nodes, kernel, bandwidth and assignments, with sample sizes $N$ divisible by $4$. Let $a_i$ take the values $1$ and $-1$ in equal numbers. Let $Y_i(1)=Y_i(0)=a_i$ when $N$ is a multiple of $8$, and let $Y_i(1)=2a_i$ and $Y_i(0)=0$ when $N\\equiv4\\pmod 8$. Then C1--C6 hold and $N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$ tends to $4$. The studentized statistic converges in distribution to a standard normal variable $G$ along the first sequence and to $G/\\sqrt2$ along the second. The coverage of the nominal 95\\% interval has the limits $0.95$ and about $0.9944$ along the two sequences, and it does not converge.\n\nFor the corrected statement, let $\\sigma_N^2=\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$ and $m_N=\\mathrm E[\\widehat{\\mathrm V}_2(d;b_N)]$, and suppose that $N\\sigma_N^2\\ge c>0$ for all large $N$. By (i)--(iii), I have $R_N=N(\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d;b_N)-m_N)=o_p(1)$, and (iv) implies that $D_N=N(m_N-\\sigma_N^2)$ satisfies $\\liminf_ND_N\\ge0$. Fix $\\varepsilon\\in(0,1)$. For large $N$, I have $D_N\\ge-\\varepsilon c/2$, and $R_N\\ge-\\varepsilon c/2$ with probability tending to one. On this event,\n\\[\n\\frac{\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d;b_N)}{\\sigma_N^2}=1+\\frac{D_N+R_N}{N\\sigma_N^2}\\ge1-\\varepsilon.\n\\]\nLet $T_N$ denote the studentized statistic of part (v) and let $a=z_{1-\\alpha/2}$. By Proposition 5, the ratio $(\\widehat\\tau_{\\mathrm{HA}}(d)-\\mathrm{AME}(d;\\eta))/\\sigma_N$ converges in distribution to $G$. It follows that $\\liminf_N\\operatorname{Prob}(|T_N|\\le a)\\ge\\operatorname{Prob}(|G|\\le a\\sqrt{1-\\varepsilon})$. Letting $\\varepsilon$ tend to zero, I obtain the bound $1-\\alpha$.", "problem": "Part (v) asserts that the Wald interval built on $\\widehat{\\mathrm V}_{\\mathrm{HAC}}$ has asymptotic coverage at least $1-\\alpha$ under C1--C6 alone. Its proof (``(v) follows similarly as in Proposition 1'', p.~39) rests on the normal limit of Proposition 5, and Proposition 5 assumes in addition that $N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$ is uniformly bounded below for all large $N$. Proposition 6 does not state this condition on the variance, and C1--C6 do not imply it. Parts (i)--(iv) control the error of the variance estimator at the scale $1/N$ only. When $N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$ tends to zero, they say nothing about the ratio of $\\widehat{\\mathrm V}_{\\mathrm{HAC}}$ to the variance, and the studentized statistic need not be approximately normal. The conclusion is also written as a limit, and the coverage probability need not converge even when the condition on the variance holds. Propositions 3 and 4 of the article, the second of which is its coverage result, state the condition on the variance.", "what_survives": "The estimator, the estimand, Proposition 5 with its stated premise and parts (i)--(iv) of Proposition 6 are not contradicted. Proposition 4 of the article already carries the condition on the variance. What changes is part (v), and with it the coverage claim of the extension announced in Section 6.5 (p.~759). The added condition excludes sequences with sparse effects such as the first example, and the conclusion is a lower bound on the limit inferior of the coverage and not a limit. No later result uses part (v). I did not recalculate the simulations or the applications, and no change to them is established."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E020", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of parts (i), (ii) and (iii) of Lemma C.9 with displays (108) and (109). A red box marks the condition s greater than 3 b N at the start of part (iii).", "caption": "Supplement A, p. 33: Lemma C.9, parts (i)–(iii). The marked condition of part (iii) is on $s$, while the bound (109) below it depends on $d_{ij}$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E021.png"}, {"alt": "Crop of displays (114) to (116), the expansion of the HAC variance estimator. A red box marks the factor Z i in the second sum of (115), the sum that is divided by N 0 squared.", "caption": "Supplement A, p. 34: equations (114)–(116). The marked factor in the second sum of (115), the diagonal term of the control arm, should be $1-Z_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E021-b.png"}, {"alt": "Crop of equation (120), the sentence Note V hat 1 equals the sum of v i, and equation (121), which defines V hat 2 as a sum of v i. Red boxes mark the symbol v i, printed without a hat, in both places.", "caption": "Supplement A, p. 34: the sentence after (120), and equation (121). The marked $v_i$ is the expectation of $\\widehat v_i$, and the sums should run over $\\widehat v_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E021-c.png"}, {"alt": "Crop of displays (135) and (136). Red boxes mark the denominator N squared times the square of 1 minus p, under the numerators (1 minus Z i) Z j and (1 minus Z j) Z i.", "caption": "Supplement A, p. 37: equations (135) and (136). The marked denominators should be $N^2p(1-p)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E021-d.png"}, {"alt": "Crop of displays (146), (147) and (148). A red box marks the whole of (148): equals the double sum over i and j of K of d i j over b N, times tau i of d minus AME, times tau j of d minus AME.", "caption": "Supplement A, p. 38: equations (146)–(148). The marked right-hand side of (148) lacks the factor $N^{-2}$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E021-e.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T29", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T29"}], "headline": "Five displays in the proof that the HAC variance estimator is consistent are misprinted: Lemma C.9(iii) requires $d_{ij}>3b_N$, the diagonal term of the control arm in (115) carries $1-Z_i$, the sums that define $\\widehat{\\mathrm V}_1$ and $\\widehat{\\mathrm V}_2$ run over the random $\\widehat v_{i,N}$, the mixed terms (135)–(136) have denominator $N^2p(1-p)$, and (148) carries the factor $N^{-2}$.", "id": "P004-E021", "location": "Supplement A, Appendix C: Lemma C.9(iii), p. 33; equation (115) and the definitions after (120) and in (121), p. 34; equations (135)–(136), p. 37; equation (148), p. 38 (PDF pages of the supplement with the same numbers)", "pdf_page": 33, "printed_page": "33", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T29"], "text": {"as_printed": "\\begin{enumerate}\n\\item Lemma C.9, p.~33: (iii) If $\\hl{s}>3b_N$,\n\\[\n\\begin{aligned}\n|\\operatorname{Cov}(\\widehat v_{i,N}(d,b_N),\\widehat v_{j,N}(d,b_N))|\\lesssim{}&\\frac{b_N^2}{N^4}\\times\\max\\Big\\{\\sum_{\\{k:d_{ik}\\le b_N\\}}\\psi_d(\\frac{d_{ij}}3-d_{ik}),\\\\\n&\\sum_{\\{l:d_{jl}\\le b_N\\}}\\psi_d(\\frac{d_{ij}}3-d_{jl})\\Big\\},\\qquad\\text{(109)}\n\\end{aligned}\n\\]\nwhere by using the symbol $\\lesssim$ we suppress a constant dependent of $B$ but independent of $N$.\n\\item Equations (114)--(116), p.~34:\n\\[\n\\begin{aligned}\n\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d;b_N)={}&\\frac1{N_1^2}\\sum_{i=1}^NZ_i\\hat e_i^2(d)+\\frac1{N_0^2}\\sum_{i=1}^N\\hl{Z_i}\\hat e_i^2(d)+\\frac1{N_1^2}\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})\\\\\n&-\\frac2{N_1N_0}\\sum_{i=1}^N\\sum_{j\\ne i}(1-Z_i)Z_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})\\\\\n&+\\frac1{N_0^2}\\sum_{i=1}^N\\sum_{j\\ne i}(1-Z_i)(1-Z_j)\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N}).\n\\end{aligned}\n\\]\n\\item After (120), p.~34: Note $\\widehat{\\mathrm V}_1(d;b_N)=\\sum_{i=1}^N\\hl{v_i}(\\widehat{\\bar\\mu}_1(d),\\widehat{\\bar\\mu}_0(d);d,b_N)$. Also define\n\\[\n\\widehat{\\mathrm V}_2(d;b_N)=\\sum_{i=1}^N\\hl{v_i}(\\bar\\mu_1(d),\\bar\\mu_0(d);d,b_N)\\qquad\\text{(121)}\n\\]\n\\item Equations (135)--(136), p.~37, two of the terms of $u_{ij}$:\n\\[\n\\begin{aligned}\n&-\\frac{(1-Z_i)Z_j}{\\hl{N^2(1-p)^2}}\\Big(-\\mu_i(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^1(d)\\big)-\\mu_j(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^0(d)-\\bar\\mu^0(d)\\big)\\\\\n&\\qquad\\qquad\\qquad\\qquad+\\widehat{\\bar\\mu}^0(d)\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^0(d)\\bar\\mu^1(d)\\Big)&&\\text{(135)}\\\\\n&-\\frac{(1-Z_j)Z_i}{\\hl{N^2(1-p)^2}}\\Big(-\\mu_i(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^0(d)-\\bar\\mu^0(d)\\big)-\\mu_j(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^1(d)\\big)\\\\\n&\\qquad\\qquad\\qquad\\qquad+\\widehat{\\bar\\mu}^0(d)\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^0(d)\\bar\\mu^1(d)\\Big)&&\\text{(136)}\n\\end{aligned}\n\\]\n\\item Equations (147)--(148), p.~38:\n\\[\n\\begin{aligned}\n&=\\sum_{i=1}^N\\sum_{j=1}^NK\\Big(\\frac{d_{ij}}{b_N}\\Big)\\mathrm E[\\widehat\\epsilon_{i,N}(d)]\\mathrm E[\\widehat\\epsilon_{j,N}(d)]&&\\text{(147)}\\\\\n&=\\hl{\\sum_{i=1}^N\\sum_{j=1}^N}K\\Big(\\frac{d_{ij}}{b_N}\\Big)(\\tau_i(d)-\\mathrm{AME}(d;\\eta))(\\tau_j(d)-\\mathrm{AME}(d;\\eta)).&&\\text{(148)}\n\\end{aligned}\n\\]\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item Lemma C.9, p.~33: (iii) If $\\hl{d_{ij}}>3b_N$,\n\\[\n\\begin{aligned}\n|\\operatorname{Cov}(\\widehat v_{i,N}(d,b_N),\\widehat v_{j,N}(d,b_N))|\\lesssim{}&\\frac{b_N^2}{N^4}\\times\\max\\Big\\{\\sum_{\\{k:d_{ik}\\le b_N\\}}\\psi_d(\\frac{d_{ij}}3-d_{ik}),\\\\\n&\\sum_{\\{l:d_{jl}\\le b_N\\}}\\psi_d(\\frac{d_{ij}}3-d_{jl})\\Big\\},\\qquad\\text{(109)}\n\\end{aligned}\n\\]\nwhere by using the symbol $\\lesssim$ we suppress a constant dependent of $B$ but independent of $N$.\n\\item Equations (114)--(116), p.~34:\n\\[\n\\begin{aligned}\n\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d;b_N)={}&\\frac1{N_1^2}\\sum_{i=1}^NZ_i\\hat e_i^2(d)+\\frac1{N_0^2}\\sum_{i=1}^N\\hl{(1-Z_i)}\\hat e_i^2(d)+\\frac1{N_1^2}\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})\\\\\n&-\\frac2{N_1N_0}\\sum_{i=1}^N\\sum_{j\\ne i}(1-Z_i)Z_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})\\\\\n&+\\frac1{N_0^2}\\sum_{i=1}^N\\sum_{j\\ne i}(1-Z_i)(1-Z_j)\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N}).\n\\end{aligned}\n\\]\n\\item After (120), p.~34: Note $\\widehat{\\mathrm V}_1(d;b_N)=\\sum_{i=1}^N\\hl{\\widehat v_{i,N}}(\\widehat{\\bar\\mu}_1(d),\\widehat{\\bar\\mu}_0(d);d,b_N)$. Also define\n\\[\n\\widehat{\\mathrm V}_2(d;b_N)=\\sum_{i=1}^N\\hl{\\widehat v_{i,N}}(\\bar\\mu_1(d),\\bar\\mu_0(d);d,b_N)\\qquad\\text{(121)}\n\\]\n\\item Equations (135)--(136), p.~37:\n\\[\n\\begin{aligned}\n&-\\frac{(1-Z_i)Z_j}{N^2\\hl{p(1-p)}}\\Big(-\\mu_i(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^1(d)\\big)-\\mu_j(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^0(d)-\\bar\\mu^0(d)\\big)\\\\\n&\\qquad\\qquad\\qquad\\qquad+\\widehat{\\bar\\mu}^0(d)\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^0(d)\\bar\\mu^1(d)\\Big)&&\\text{(135)}\\\\\n&-\\frac{(1-Z_j)Z_i}{N^2\\hl{p(1-p)}}\\Big(-\\mu_i(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^0(d)-\\bar\\mu^0(d)\\big)-\\mu_j(\\mathbf Y;d)\\big(\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^1(d)\\big)\\\\\n&\\qquad\\qquad\\qquad\\qquad+\\widehat{\\bar\\mu}^0(d)\\widehat{\\bar\\mu}^1(d)-\\bar\\mu^0(d)\\bar\\mu^1(d)\\Big)&&\\text{(136)}\n\\end{aligned}\n\\]\n\\item Equations (147)--(148), p.~38:\n\\[\n\\begin{aligned}\n&=\\sum_{i=1}^N\\sum_{j=1}^NK\\Big(\\frac{d_{ij}}{b_N}\\Big)\\mathrm E[\\widehat\\epsilon_{i,N}(d)]\\mathrm E[\\widehat\\epsilon_{j,N}(d)]&&\\text{(147)}\\\\\n&=\\hl{\\frac1{N^2}}\\sum_{i=1}^N\\sum_{j=1}^NK\\Big(\\frac{d_{ij}}{b_N}\\Big)(\\tau_i(d)-\\mathrm{AME}(d;\\eta))(\\tau_j(d)-\\mathrm{AME}(d;\\eta)).&&\\text{(148)}\n\\end{aligned}\n\\]\n\\end{enumerate}", "evidence": "The evidence is a derivation from the adjacent definitions, with two exact checks. In the regression of the circle averages on the assignment, node $i$ enters the sandwich formula for the variance of the coefficient through $(Z_i/N_1-(1-Z_i)/N_0)\\hat e_i(d)$. Since $Z_i(1-Z_i)=0$, its square is $Z_i\\hat e_i^2(d)/N_1^2+(1-Z_i)\\hat e_i^2(d)/N_0^2$, which is the diagonal of the corrected (115). With $N=4$, assignments $(1,1,0,0)$ and residuals $(0,0,1,-1)$, which have mean zero in each arm, the diagonal terms sum to $0$ as printed and to $1/2$ as corrected. Multiplying the two terms of (102) for a control node $i$ and a treated node $j$, I obtain the factor $-(1-Z_i)Z_j/(N^2p(1-p))$ of (135), and exchanging $i$ and $j$ gives (136). At $p=1/4$ the printed factor $1/(1-p)^2$ equals $16/9$ and the correct factor $1/(p(1-p))$ equals $16/3$. Substituting footnote 22 into (147), I obtain (148) with the factor $N^{-2}$. Multiplying by $N$ as in (157) then gives the normalization $1/N$ of C6 in (122). The expansions (117)--(120) and (130)--(132) are sums of the random products in (107), that is, of the $\\widehat v_{i,N}$. The proof of Lemma C.9(iii) uses the separation $d_{ij}/3>b_N$.", "problem": "Each display disagrees with a definition or an identity next to it.\n\\begin{enumerate}\n\\item In Lemma C.9(iii) the radius $s$ plays no role in the bound (109), which depends on $d_{ij}$, and the proof of that part begins ``if $d_{ij}>3b_N$''.\n\\item The second sum of (115) is the diagonal term of the control arm and carries $1-Z_i$, as in (117). As printed, the residuals of control nodes do not enter the diagonal.\n\\item The line after (107) defines $v_{i,N}$ as the expectation of $\\widehat v_{i,N}$. The printed sums are then constants, while $\\widehat{\\mathrm V}_1$ and $\\widehat{\\mathrm V}_2$ are random quadratic forms, and the proof bounds the variance of $\\widehat{\\mathrm V}_2$ through $\\operatorname{Cov}(\\widehat v_{i,N},\\widehat v_{j,N})$ in (138).\n\\item The mixed terms multiply a factor $Z/(Np)$ for a treated node by a factor $(1-Z)/(N(1-p))$ for a control node. Their denominator is $N^2p(1-p)$, as in (119).\n\\item By footnote 22, $\\mathrm E[\\widehat\\epsilon_{i,N}(d)]=(\\tau_i(d)-\\mathrm{AME}(d;\\eta))/N$, and (147) equals $N^{-2}$ times the right-hand side of (148) as printed.\n\\end{enumerate}", "what_survives": "The variance estimator, the orders in probability asserted in the proof and the proof of Proposition 6(i)--(iv) are unchanged after these substitutions, for fixed $0<p<1$. Part (v) of that proposition is the subject of \\fid{P004-E020}."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E021", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of four lines of text that end Section 6.3. The last sentence reads: We include simulation results for a class of randomization tests in Section 6.3. A red outline surrounds the words Section 6.3.", "caption": "Main text, end of Section 6.3, p. 758 (PDF p. 15). The outline marks the reference to Section 6.3, which is the section that contains the sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E022.png"}, {"alt": "Crop of three lines of text. They read: C6-(i) assumes that the treatments are assigned independently, C6-(ii) states that the ith node's treatment probability can be fully described by the confounders O sub i, independent of the potential outcomes, and C6-(ii) assures overlap with the target set of potential outcomes. A red outline surrounds the words C6-(ii) assures overlap.", "caption": "Main text, paragraph after condition C6, p. 758 (PDF p. 15). The outline marks the second reference to C6-(ii), where the overlap condition C6-(iii) is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E022-b.png"}, {"alt": "Crop of Figure 13 with its caption. Two line plots stand side by side. The left one is titled Hajek Estimator and the right one Smoothed Hajek Estimator (Triangular Kernel, bw=5km); each shows an effect curve and interval bounds against distance in km from 0 to about 14. The caption below reads in part: The top plot presents results from the Hajek estimator. The bottom plot presents results from the smoothed Hajek estimator with a triangular kernel and a bandwidth of five km. Red outlines surround the words The top plot and The bottom plot.", "caption": "Figure 13 and its caption, p. 765 (PDF p. 22). The two panels are side by side; the outlines mark the words “The top plot” and “The bottom plot”.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E022-c.png"}, {"alt": "Crop of six lines of text. They read in part: under completely randomized assignment, where a fixed number N sub 1 of nodes are assigned to treatment. Then for Y sub ix of 1, one marginalizes over assignments with N minus 1 units assigned to treatment, while for Y sub ix of 0, one marginalizes over assignments with N units assigned to treatments. Red outlines surround N minus 1 and the following N.", "caption": "Main text, top of p. 751 (PDF p. 8). The outlines mark the two counts of treated units, printed as $N-1$ and $N$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E022-d.png"}, {"alt": "Crop of equation (33) with two lines of text above and below. The equation reads: V of tau-hat-L sub HA of d equals V of tau-hat of mu-bar-1 of d and mu-bar-1 of d, approximately 0 plus C. A red outline surrounds the second mu-bar-1 of d.", "caption": "Supplement A, equation (33), p. 13. The outline marks the second argument, printed as $\\bar\\mu_1(d)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E022-e.png"}, {"alt": "Crop of the first five lines of Section A.5, Efficiency Comparison between the Hajek and HT estimators. The second sentence begins: Although it is not true that the Hajek estimator is not always more efficient than the HT estimator in our setting. A red outline surrounds the words it is not true that the Hajek estimator is not always.", "caption": "Supplement A, opening of A.5, p. 11. The outline marks the clause with two negations.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E022-f.png"}, {"alt": "Crop of six lines of text. The last sentence reads: Finally, we show that the spatial HAC standard errors estimator estimates a quantity that is probably larger than the true asymptomatic variance under C5, enabling conservative Wald-inference. A red outline surrounds the word asymptomatic.", "caption": "Supplement A, second paragraph of p. 1. The outline marks the word “asymptomatic”.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E022-g.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T30", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T30"}], "headline": "Seven local slips are corrected: the counts under complete randomization are $N_1-1$ and $N_1$; the simulations of the randomization tests are in Section 7; the overlap condition is C6-(iii); the panels of Figure 13 are left and right; “asymptomatic” is “asymptotic”; the opening of A.5 has one negation too many; and the second argument in (33) is $\\bar\\mu_0(d)$.", "id": "P004-E022", "location": "Main text: top of p. 751 (PDF p. 8); end of Section 6.3 and the paragraph after C6, p. 758 (PDF p. 15); caption of Figure 13, p. 765 (PDF p. 22). Supplement A: second paragraph of p. 1; opening sentence of A.5, p. 11; equation (33), p. 13", "pdf_page": 8, "printed_page": "751", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T30"], "text": {"as_printed": "\\emph{Main text, p.~751.} Then for $Y_{ix}(1;\\eta)$, one marginalizes over assignments with $\\hl{N-1}$ units assigned to treatment, while for $Y_{ix}(0;\\eta)$, one marginalizes over assignments with $\\hl{N}$ units assigned to treatments.\n\n\\emph{Main text, p.~758, end of Section 6.3.} We include simulation results for a class of randomization tests in \\hl{Section 6.3}.\n\n\\emph{Main text, p.~758, after C6.} C6-(i) assumes that the treatments are assigned independently, C6-(ii) states that the $i$th node's treatment probability can be fully described by the confounders $O_i$, independent of the potential outcomes, and \\hl{C6-(ii)} assures overlap with the target set of potential outcomes.\n\n\\emph{Main text, p.~765, caption of Figure 13.} The \\hl{top} plot presents results from the Hajek estimator. The \\hl{bottom} plot presents results from the smoothed Hajek estimator with a triangular kernel and a bandwidth of five km.\n\n\\emph{Supplement A, p.~1.} Finally, we show that the spatial HAC standard errors estimator estimates a quantity that is probably larger than the true \\hl{asymptomatic} variance under C5, enabling conservative Wald-inference.\n\n\\emph{Supplement A, p.~11.} Although \\hl{it is not true that} the Hajek estimator is \\emph{not always} more efficient than the HT estimator in our setting, the Hajek estimator has some efficiency properties that make it appealing in practice:\n\n\\emph{Supplement A, p.~13, equation (33).}\n\\[\nV(\\widehat\\tau^{L}_{HA}(d)) = V\\bigl(\\widehat\\tau(\\bar\\mu_1(d),\\hl{\\bar\\mu_1(d)})\\bigr)\\approx 0+C,\n\\]", "corrected": "\\emph{Main text, p.~751.} Then for $Y_{ix}(1;\\eta)$, one marginalizes over assignments with $\\hl{N_1-1}$ units assigned to treatment, while for $Y_{ix}(0;\\eta)$, one marginalizes over assignments with $\\hl{N_1}$ units assigned to treatments. In both cases the count is taken among the nodes other than $i$.\n\n\\emph{Main text, p.~758, end of Section 6.3.} We include simulation results for a class of randomization tests in \\hl{Section 7}.\n\n\\emph{Main text, p.~758, after C6.} \\ldots and \\hl{C6-(iii)} assures overlap with the target set of potential outcomes.\n\n\\emph{Main text, p.~765, caption of Figure 13.} The \\hl{left} plot presents results from the Hajek estimator. The \\hl{right} plot presents results from the smoothed Hajek estimator with a triangular kernel and a bandwidth of five km.\n\n\\emph{Supplement A, p.~1.} \\ldots a quantity that is probably larger than the true \\hl{asymptotic} variance under C5, enabling conservative Wald-inference.\n\n\\emph{Supplement A, p.~11.} Although the Hajek estimator is \\emph{not always} more efficient than the HT estimator in our setting, the Hajek estimator has some efficiency properties that make it appealing in practice:\n\n\\emph{Supplement A, p.~13, equation (33).}\n\\[\nV(\\widehat\\tau^{L}_{HA}(d)) = V\\bigl(\\widehat\\tau(\\bar\\mu_1(d),\\hl{\\bar\\mu_0(d)})\\bigr)\\approx 0+C.\n\\]", "evidence": "The evidence is a check of the pages against one another: Section 7 (p.~761) and Figure 9 (p.~762), condition C6 (p.~758), the panels of Figure 13, the list after the opening sentence of A.5, and equations (27) and (56) of Supplement A, which give the two centres of the Hajek estimator. For the counts, a direct enumeration with $N=4$ and $N_1=2$ gives one treated node among the other three when node $i$ is treated and two when it is not.", "problem": "Each item is a local slip, and the context fixes the intended meaning. Under complete randomization with $N_1$ treated nodes, the nodes other than $i$ contain $N_1-1$ treated nodes when node $i$ is treated and $N_1$ when it is not. The simulations of the randomization tests are reported in Section 7 with Figure 9, and Section 6.3 is the section in which the sentence stands. In C6, item (ii) defines the probability model and item (iii) bounds the probabilities away from 0 and 1, which is the overlap condition. The two panels of Figure 13 are side by side. The list that follows the opening sentence of A.5 ends by saying that either estimator can be the more efficient one, so the sentence means that the Hajek estimator is not always more efficient. The linearized Hajek estimator is centred at $\\bar\\mu_1(d)$ in the treated arm and at $\\bar\\mu_0(d)$ in the control arm.", "what_survives": "These local corrections do not change the estimators or reported numbers. In the sentence on p.~1 of Supplement A, ``probably larger'' may be meant as ``provably larger'', in line with the bound that the supplement derives. The printed wording is not false, and I leave it."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E022", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Eight lines of body text. Red boxes mark the beginning of the probability-fitting sentence, through Ferraro et al., and the inferential-approach sentence through Jayachandran et al.; its final word, study, is on the next page.", "caption": "Section 8.2, end of p. 764 (PDF p. 21). The marked passages describe the fitted treatment probabilities and say that inference follows the approach of the first application; the latter sentence continues on p. 765.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E023.png"}], "formal_release": null, "formal_statements": [], "headline": "The Costa Rica script in the replication archive fits the propensity scores but omits the argument that activates probability weighting, so, if estimation is reached, its three calls compute an unweighted unsmoothed contrast, with variances that ignore the scores; each call should set prob_treatment = \"prob_treatment\".", "id": "P004-E023", "location": "Section 8.2, last paragraph of p. 764 (PDF p. 21), continuing on p. 765, and Figure 13; replication archive (Supplement B), application/scripts/Ferraro_etal_2015.R, lines 134–137, 161 and 238–249, and SpatialEffect/R/SpatialEffect.R, lines 2 and 51–55", "pdf_page": 21, "printed_page": "764", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 8.2, p.~764: ``We estimate the probability for each node to be treated by running a logistic regression model of the treatment indicator on the three covariates used by Ferraro et al., aggregated to the level of intervention nodes: soil quality, distance to the nearest road, and distance to the nearest large city.'' And, at the end of p.~764, continuing on p.~765: ``We use the same inferential approach as with the Jayachandran et al.\\ study.''\n\n\\texttt{Ferraro\\_etal\\_2015.R}, line 161 and the first call, lines 238--241:\n\n\\texttt{prob\\_treatment <- \"prob\\_treatment\"}\n\n\\texttt{result.list <- SpatialEffect(ras = ras, Zdata = Zdata, x\\_coord\\_Z = x\\_coord\\_Z, y\\_coord\\_Z = y\\_coord\\_Z, ras\\_Z = NULL, treatment = treatment, dVec = dVec[-1], numpts = 10000, only.unique = only.unique, smooth = smooth, per.se = per.se, conley.se = conley.se, cutoff = cutoff, alpha = alpha, edf = edf, nPerms = nPerms, smooth.conley.se = smooth.conley.se, dist.metric = dist.metric, cType = \"edge\", kernel = kernel)}\n\n\\texttt{SpatialEffect.R}, lines 51--55 (the highlighted unit weights are selected by these calls):\n\n\\texttt{if (is.null(prob\\_treatment))\\{}\n\n\\texttt{pweight <- \\hl{rep(1, nz)}}\n\n\\texttt{\\}else\\{}\n\n\\texttt{pweight <- Zup / Zdata[, \"prob\\_treatment\"] + (1 - Zup) / (1 - Zdata[, \"prob\\_treatment\"])}\n\n\\texttt{\\}}", "corrected": "In each of the three calls (lines 238, 243 and 248):\n\n\\texttt{result.list <- SpatialEffect(\\hl{prob\\_treatment = \"prob\\_treatment\"}, ras = ras, Zdata = Zdata, ...)}\n\nCheck that the rows of the probabilities, the treatment indicator, the node geometry and the circle averages are aligned, and compute the application again.", "evidence": "I checked the original point-estimation code on small examples with synthetic geometry and verified the contrasts by exact arithmetic. With four nodes, $(z_i,y_i,p_i)=(1,0,1/4)$, $(1,1,3/4)$, $(0,0,1/4)$ and $(0,0,3/4)$, each of the three calls gives $1/2$ as written and $1/4$ with the argument; the R-level variance calculation with a supplied identity covariance kernel gives $1/8$ as written and $9/128$ with the argument. The other two examples in the table use scores that are the exact maximum likelihood fit of a logistic model on three covariates. These checks did not execute the compiled covariance routine or the complete Costa Rica analysis, and no saved result for that application was available for comparison.\n\n\\begin{audittable}{lrr}{Exact contrasts in three examples.}\nExample & Calls as written & With \\texttt{prob\\_treatment} supplied \\\\\nFour nodes & $1/2$ & $1/4$ \\\\\nSixteen nodes, fitted logistic scores & $1/6$ & $1/12$ \\\\\nEight nodes, fitted logistic scores & $1/2$ & $0$ \\\\\n\\end{audittable}", "problem": "Section 8.2 presents the Costa Rica analysis as an instance of the observational extension of Section 6.4, in which estimated treatment probabilities weight the estimator. The script fits the logistic model, stores the fitted probabilities in the table of nodes (lines 134--137), and assigns the name of that column to a variable (line 161). None of its three calls to \\texttt{SpatialEffect} (lines 238--249) passes the argument \\texttt{prob\\_treatment}. In R an omitted argument takes its default, here \\texttt{NULL}, whatever variable of that name exists in the calling script. With \\texttt{NULL}, the function sets every weight to one (lines 51--55).\n\nAt a fixed distance, let $y_i$ be the circle average of node $i$, $z_i$ its treatment and $p_i$ its fitted probability, with all circle averages observed, both treatment groups nonempty and $0<p_i<1$. If estimation is reached, the calls as written give the unsmoothed contrast\n\\[\n\\frac{\\sum_i z_iy_i}{\\sum_i z_i}-\\frac{\\sum_i(1-z_i)y_i}{\\sum_i(1-z_i)},\n\\]\nand with the argument supplied the function returns the normalized weighted contrast\n\\[\n\\frac{\\sum_i z_iy_i/p_i}{\\sum_i z_i/p_i}-\\frac{\\sum_i(1-z_i)y_i/(1-p_i)}{\\sum_i(1-z_i)/(1-p_i)}.\n\\]\nThe HAC and smoothing code receive the same probability weights, and they too ignore the scores in the calls as written. The released code does not carry out the propensity-adjusted analysis that the text describes. I have not established that these calls produced Figure 13.", "what_survives": "This omission does not invalidate the logistic fit or the density of the scores in Figure 12. The theory, the simulations and the Uganda application do not depend on these calls. This code defect does not show any value in Figure 13 or any conclusion about leakage to be wrong, since it is not known whether the published figure was computed with or without the weights. Supplying the argument activates the weighted contrast and the weighted smoothing and HAC paths that the function already has. It does not make the variance account for the estimation of the scores, which is a separate matter (\\fid{P004-E001})."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E023", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the two lines that introduce the variance-bound estimator of Sävje, Aronow and Hudgens, equation (20), and the three lines after it, which say that the estimates tend to be extremely conservative. Red boxes mark the denominators p squared and (1 minus p) squared of the two sums in equation (20).", "caption": "Equation (20), p. 755 (PDF p. 12). The marked denominators are $p^2$ and $(1-p)^2$; the equation is correct, and the code does not match it.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E024.png"}, {"alt": "Listing of source code with line numbers. Lines 119 to 121 of 6_functions3.R compute mu_d, the least squares coefficient beta and the residual res. Line 194 reads SAHvar=(sum(D* res^2 / prob * cid[,2]) + sum( (1-D)* (res^2 / (1-prob) * cid[,2])))/N^2. Red boxes mark '/ prob' and '/ (1-prob)' in line 194 and in the identical lines 95 and 123 of 6_functions2.R.", "caption": "Source code of the replication archive, not a page of the article: 6_functions3.R, lines 119–121 and 194, and 6_functions2.R, lines 95 and 123. The marked divisors are prob and (1-prob), to the first power; res is the ordinary least squares residual of line 121.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E024-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The simulation code divides the squared residuals by $p$ and $1-p$ where the SAH estimator (20) has $p^2$ and $(1-p)^2$; with $p=1/2$ it returns half of (20).", "id": "P004-E024", "location": "Equation (20), p. 755 (PDF p. 12); replication archive (Supplement B), simulation/scripts/6_functions3.R, line 194, and 6_functions2.R, lines 95 and 123; Figures 7–8, p. 761 (PDF p. 18)", "pdf_page": 12, "printed_page": "755", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Equation (20), which is correct as printed:\n\\[\n\\widehat{\\mathrm V}_{\\mathrm{SAH}}(d)=\\frac1{N^2}\\sum_{i=1}^N\\frac{Z_ic_i(d)\\widehat e_i^2}{p^2}+\\frac1{N^2}\\sum_{i=1}^N\\frac{(1-Z_i)c_i(d)\\widehat e_i^2}{(1-p)^2}.\n\\]\nReplication code, \\texttt{6\\_functions3.R}, line 194, in the routine \\texttt{Regression\\_AME} (the same expression stands at lines 95 and 123 of \\texttt{6\\_functions2.R}):\n\\[\n\\begin{aligned}\n&\\texttt{SAHvar=(sum(D* res\\textasciicircum 2 }\\hl{\\texttt{/ prob}}\\texttt{ * cid[,2])}\\\\\n&\\qquad\\texttt{+ sum( (1-D)* (res\\textasciicircum 2 }\\hl{\\texttt{/ (1-prob)}}\\texttt{ * cid[,2])))/N\\textasciicircum 2}\n\\end{aligned}\n\\]", "corrected": "Equation (20) stands. In \\texttt{Regression\\_AME} (\\texttt{6\\_functions3.R}, line 194) and in both definitions of \\texttt{calculate\\_var} (\\texttt{6\\_functions2.R}, lines 95 and 123):\n\\[\n\\begin{aligned}\n&\\texttt{SAHvar=(sum(D* res\\textasciicircum 2 }\\hl{\\texttt{/ prob\\textasciicircum 2}}\\texttt{ * cid[,2])}\\\\\n&\\qquad\\texttt{+ sum( (1-D)* (res\\textasciicircum 2 }\\hl{\\texttt{/ (1-prob)\\textasciicircum 2}}\\texttt{ * cid[,2])))/N\\textasciicircum 2}\n\\end{aligned}\n\\]\nAt $p=1/2$ this doubles each SAH variance and multiplies each half-length by $\\sqrt2$ on identical inputs. The SAH summaries of the simulations, and any figure produced from the earlier values, are to be recomputed.", "evidence": "The evidence is an exact calculation on one input and an execution of the original lines on the same input. Let $N=8$, with nodes at $(i-1,0)$, $d=0$ and cutoff $0$, so that $c_i(0)=1$. Let $p=1/2$, $Z=(0,0,0,0,1,1,1,1)$ and circle averages $(9,11,9,11,11,13,11,13)$. The least squares fit has intercept $10$ and slope $2$, and the residuals are $(-1,1,-1,1,-1,1,-1,1)$. I write $A=\\sum_iZ_ic_i(d)\\widehat e_i^2$ and $B=\\sum_i(1-Z_i)c_i(d)\\widehat e_i^2$; here $A=B=4$. The code returns $(A/p+B/(1-p))/N^2=16/64=1/4$, and equation (20) equals $(A/p^2+B/(1-p)^2)/N^2=32/64=1/2$. The three code expressions, executed as they stand, return $0.25$. For every input with $p=1/2$, the value of (20) is twice the value of the code.", "problem": "Equation (20) is correct. The routine that computes the SAH variance in the simulations forms the residuals of the least squares regression (18). It divides their squares by \\texttt{prob} and \\texttt{1-prob}, the assignment probability and its complement, where (20) divides by the squares of these. The simulation script sets the assignment probability to $1/2$. The value returned is then half of (20) for every assignment with a nonzero residual. The plotting script multiplies the square root of this value by $1.96$. The SAH half-lengths that it plots are then $1/\\sqrt2\\approx0.707$ times those of (20). Note that the archive does not contain the saved simulation inputs and results. I cannot confirm that the published SAH curves in Figures 7--8 and in the supplement were produced by this code without a later change, and I have not recomputed their coverage.", "what_survives": "Equation (20) and the result stated for it are untouched. The Hajek point estimates, their mean squared errors, the HAC and HAC-PD variances, the degrees-of-freedom adjustment and the randomization tests do not use this line. The intervals of the two applications (Figures 11 and 13) are HAC-PD intervals and do not use it either. On the same simulated draws, the corrected code lengthens every SAH interval by the factor $\\sqrt2$ and cannot lower its coverage. The statement that the SAH estimates are extremely conservative is not weakened. To the extent that the supplied code produced them, the SAH half-lengths in Figures 7--8 and in the supplement's comparison figures change by this factor."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E024", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the nine-line caption of Figure 7. It reads in part: HAC refers to the CI with the HAC variance estimator in (19) and a normal critical value. The length and coverage of the HAC CI are assessed with respect to the cases where HAC estimator returns a nonnegative value. HAC_PD refers to the CI with positive-semidefinite HAC variance estimator in (22) and a normal critical value. HAC (edof) refers to the CI with HAC variance estimator in (19) using a t distribution with an approximated empirical degrees of freedom. Red outlines surround the words HAC variance estimator in (19) and a normal critical value, and the words estimator in (22) and a normal critical value.", "caption": "Caption of Figure 7, p. 761 (PDF p. 18). The outlines mark the two definitions of the plain curves.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E025.png"}, {"alt": "Crop of seven lines of text. The marked sentence reads: The confidence interval generated by the positive semidefinite HAC variance estimator is not substantially longer than the one with the HAC variance estimator. It is followed by: Based on these observations, we recommend researchers to use the positive semidefinite HAC variance estimator with the empirical degree of adjustment when applying our method. Red outlines surround the marked sentence, which runs over three lines.", "caption": "Main text below Figure 7, p. 761 (PDF p. 18). The outline marks the sentence that compares the lengths of the two intervals.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E025-b.png"}, {"alt": "Crop of equation (44) and two lines of text. Equation (44) defines mu as the trace of N sub 1 N sub 0 over N times M times the Hadamard product of Omega and lambda-prime lambda, times M. The text reads: where the circle denotes the pointwise (Hadamard) matrix product. The estimated variance is inflated by one over mu, becoming w-prime Sigma-hat sub HAC w over mu. We further define the quantity. A red outline surrounds the sentence The estimated variance is inflated by one over mu, becoming w-prime Sigma-hat sub HAC w over mu.", "caption": "Supplement A, Appendix A.7, p. 17. The outline marks the only place where the division of the variance by $\\mu$ is described.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E025-c.png"}, {"alt": "Two bar charts under the title: Two exact examples: dividing each variance by its own mu reverses the order of the HAC and HAC_PD variances (left) and changes whether the normal interval covers (right). Left panel, two unit squares: for HAC, equation (19), a blue bar of height 3/2 (as defined) and an orange bar of height 3 (divided by mu = 1/2); for HAC_PD, equation (22), a blue bar of height 2 and an orange bar of height 8/3 (mu = 3/4). The vertical axis is the variance estimate in squared outcome units. Right panel, identity kernel: a blue bar of height 2401/1250 = 1.921 for the variance as defined in (19) and an orange bar of height 4802/1875 = 2.561 for the variance divided by mu = 3/4, with a horizontal line at the squared error 9/4. The vertical axis is the squared half-length of the interval in squared outcome units. Only the orange bar reaches above the line.", "caption": "The two exact examples. Left: in the second example the HAC_PD variance is larger than the HAC variance as defined ($2$ against $3/2$) and smaller after each is divided by its own $\\mu$ ($8/3$ against $3$). Right: in the first example the squared half-length of the normal interval is below the squared error $9/4$ with the variance of (19) and above it after the division by $\\mu=3/4$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E025-rescaling.png"}], "formal_release": null, "formal_statements": [], "headline": "The curves labelled HAC and HAC_PD in Figures 7 and 8 use the variances of (19) and (22), each divided by its own factor $\\mu$ from Appendix A.7; they differ from the “edof” curves only in the critical value, and the remarks on the length of the intervals hold for these rescaled variances.", "id": "P004-E025", "location": "Caption of Figure 7 and the labels inherited by Figure 8, p. 761 (PDF p. 18); captions of Figures 2, 3, 5, 6, 8 and 9 of Supplement A (pp. 19–23); the two observations drawn from these figures, p. 756 and p. 761", "pdf_page": 18, "printed_page": "761", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Caption of Figure 7, p.~761.} HAC refers to the CI with the \\hl{HAC variance estimator in (19)} and a normal critical value. The length and coverage of the HAC CI are assessed with respect to the cases where HAC estimator returns a nonnegative value. HAC\\_PD refers to the CI with positive-semidefinite HAC variance \\hl{estimator in (22)} and a normal critical value. HAC (edof) refers to the CI with HAC variance estimator in (19) using a $t$ distribution with an approximated empirical degrees of freedom. HAC\\_PD (edof) refers to the CI with HAC variance estimator in (22) and empirical degree of freedom adjustment.\n\n\\emph{Main text, p.~761.} The confidence interval generated by the positive semidefinite HAC variance estimator is \\hl{not substantially longer} than the one with the HAC variance estimator.\n\n\\emph{Main text, p.~756.} We find the HAC-PD estimator returns nonnegative variance estimates with \\hl{little loss in efficiency}.", "corrected": "\\emph{Caption of Figure 7.} HAC refers to the CI with the HAC variance estimator in (19), \\hl{divided by its factor $\\mu$ of (44)}, and a normal critical value. \\ldots\\ HAC\\_PD refers to the CI with positive-semidefinite HAC variance estimator in (22), \\hl{divided by its own factor $\\mu$ computed from the positive semidefinite kernel}, and a normal critical value. HAC (edof) and HAC\\_PD (edof) \\hl{use the same two rescaled variances} with \\hl{the empirical $t$ critical values computed by the simulation code}.\n\n\\emph{Main text, p.~761.} The confidence interval generated by the positive semidefinite HAC variance estimator is not substantially longer than the one with the HAC variance estimator, \\hl{when each variance is divided by its own factor $\\mu$}.\n\n\\emph{Main text, p.~756.} We find the HAC-PD estimator returns nonnegative variance estimates with little loss in efficiency, \\hl{when each variance is divided by its own factor $\\mu$}.\n\nAlternatively, the definitions of the plain curves can stand as printed if those curves are recomputed from the variances of (19) and (22) without the division by $\\mu$; the length and efficiency comparisons would then need to be checked again.", "evidence": "The evidence is a check of the replication code and two exact examples, computed in rational arithmetic and also run through the published R and C++ routines, which return the same numbers.\n\n\\emph{The code.} In \\texttt{6\\_functions3.R} (line 196), the function \\texttt{Regression\\_AME} returns a vector whose first two entries are \\texttt{Conley\\_var2/mu2} and \\texttt{Conley\\_var2\\_pd/mu2\\_pd}. In \\texttt{5\\_functions\\_run\\_simulations.R} (lines 264 and 279), it is called with \\texttt{if\\_edof=1}, and these two entries are stored as the HAC and HAC\\_PD variances. In \\texttt{5\\_functions\\_generate\\_figures.R} (lines 71--99), the square root \\texttt{std1} of the stored HAC variance is computed once. The coverage of the plain HAC curve is the share of draws with \\texttt{abs(tau\\_est-tau) <= std1 * 1.96}, and that of the ``(edof)'' curve is the share with \\texttt{abs(tau\\_est-tau) <= std1 * qt(0.975,edof)}. The HAC\\_PD curves are computed in the same way from \\texttt{std2}.\n\n\\emph{First example.} The eight nodes are at $(0,0),(2,0),\\dots,(14,0)$, and the kernel is uniform with cutoff 1. Then the kernel matrix and its positive semidefinite part are both the identity. Let $Z=(0,0,0,0,1,1,1,1)$ and $Y=(1,-1,1,-1,5/2,1/2,5/2,1/2)$. The estimate is $3/2$, the variance of (19) is $1/2$, the factor is $\\mu=3/4$, and the stored variance, that is, the variance divided by this factor, is $2/3$. Suppose that the true value is 0, so that the squared error is $9/4$. With the critical value $1.96=49/25$, the squared half-length of the interval is\n\\[\n\\begin{aligned}\n\\Bigl(\\frac{49}{25}\\Bigr)^2\\cdot\\frac12&=\\frac{2401}{1250}<\\frac94&&\\text{with the variance of (19)},\\\\\n\\Bigl(\\frac{49}{25}\\Bigr)^2\\cdot\\frac23&=\\frac{4802}{1875}>\\frac94&&\\text{with the stored variance}.\n\\end{aligned}\n\\]\nThe interval of the caption does not cover the true value, and the interval that is plotted does.\n\n\\emph{Second example.} The eight nodes are the corners of two unit squares, and the cutoff is 1. In this order, the nodes are $(0,0)$, $(1,0)$, $(1,1)$, $(0,1)$ and $(10,0)$, $(11,0)$, $(11,1)$, $(10,1)$. Let\n\\[\nZ=(0,0,1,1,0,0,1,1),\\qquad Y=(3,-3,1,-1,3,-3,1,-1).\n\\]\n\\begin{audittable}{lrr}{Second example: the variances as defined and as stored by the simulation code.}\n & HAC, (19) & HAC\\_PD, (22) \\\\\nVariance as defined & $3/2$ & $2$ \\\\\nFactor $\\mu$ & $1/2$ & $3/4$ \\\\\nVariance divided by $\\mu$, as stored & $3$ & $8/3$ \\\\\n\\end{audittable}\nThe ratio of the HAC\\_PD variance to the HAC variance is $4/3$ as defined and $8/9$ as stored. The order of the two variances is reversed.\n\n\\emph{Direction for the uniform kernel.} For the uniform kernel the factor satisfies $0\\le\\mu<1$. On the draws where $\\mu>0$ and the variance of (19) is nonnegative, the plotted normal interval contains the interval built on (19), and the coverage of the latter cannot exceed that of the plotted curve on those draws.\n\n\\emph{What is not established.} The simulation outputs are not in the replication archive, and I did not regenerate the simulations. The size of the differences in Figures 7 and 8 is not known, and I do not claim that the intervals built on (22) are substantially longer than those built on (19) in the designs of the paper. The two examples show what the rescaling can do. They are not reconstructions of the simulations.", "problem": "Equations (19) and (22) define the two variance estimators without any rescaling. Appendix A.7 introduces a separate adjustment with two parts: the variance is divided by the factor $\\mu$ of (44), and the normal critical value is replaced by a quantile of a $t$ distribution. According to the captions, the curves HAC and HAC\\_PD use the variances of (19) and (22) with the critical value 1.96, and the curves marked ``(edof)'' carry the adjustment. The simulation code in the replication files computes something else. The function that returns the variances divides each of them by its own factor and returns $\\widehat V_{HAC}/\\mu_{HAC}$ and $\\widehat V_{PD}/\\mu_{PD}$. The simulation stores these two numbers, and the plotting code uses them for all four curves, with 1.96 for the curves HAC and HAC\\_PD, which I call the plain curves, and a $t$ quantile for the ``(edof)'' curves.\n\nThere are three consequences. The figures do not show how the intervals built on (19) and (22) behave without the adjustment. The difference between a plain curve and its ``(edof)'' curve shows the effect of the critical value only, not of the whole adjustment of A.7. Finally, the remarks that the positive semidefinite interval is ``not substantially longer'' (p.~761) and comes with ``little loss in efficiency'' (p.~756) compare two variances that are each divided by a different factor. Since\n\\[\n\\frac{\\widehat V_{PD}/\\mu_{PD}}{\\widehat V_{HAC}/\\mu_{HAC}}=\\frac{\\widehat V_{PD}}{\\widehat V_{HAC}}\\cdot\\frac{\\mu_{HAC}}{\\mu_{PD}},\n\\]\nthis is not the comparison of (22) with (19).", "what_survives": "This discrepancy does not refute the estimators (19) and (22), the positive semidefinite construction, the asymptotic results, the figures for the mean squared error or the randomization tests. The plotted curves describe the procedures that were computed, once they are relabelled. The captions of the two applications already name the positive semidefinite estimator with the adjustment; I have not established a change in their numerical results. The recommendation on p.~761 can stand as a statement about the adjusted procedures. What changes is the scope of the two observations: they hold for the rescaled variances, and the figures do not separate the whole adjustment of A.7 from the intervals built on (19) and (22). Whether the positive semidefinite variance may be divided by its factor, and the degrees of freedom used for HAC\\_PD (edof), are treated in \\fid{P004-N004} and \\fid{P004-E026}."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E025", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Five lines from the italic caption of Figure 7. Red boxes mark the sentence: HAC_PD (edof) refers to the CI with HAC variance estimator in (22) and empirical degree of freedom adjustment.", "caption": "Caption of Figure 7, p. 761 (PDF p. 18). The marked sentence defines the HAC_PD (edof) series as the interval with the positive semidefinite HAC variance estimator (22) and the degrees-of-freedom adjustment of Supplement A.7.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E026.png"}, {"alt": "Equation (45): nu equals 2 times the trace of the square of the matrix (N1 N0 / N) M (Omega circ lambda-prime lambda) M, with a red box around the factor 2 and the star after it. Below it, the sentence: We use the alpha/2-quantile and (1 - alpha/2)-quantile of the t-distribution with 2 mu squared over v degree of freedom to construct the confidence interval. A red box marks the fraction 2 mu squared over v.", "caption": "Supplement A, Section A.7, p. 17: equation (45) and the sentence after it. The marks show the factor 2 in $\\nu$ and the stated degrees of freedom $2\\mu^2/v$; the $v$ of that sentence is the $\\nu$ of (45).", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E026-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The simulation code uses $\\mu_{\\mathrm{PD}}^2/\\nu_{\\mathrm{PD}}$ degrees of freedom for the HAC_PD (edof) interval, half of the $2\\mu^2/\\nu$ stated in Supplement A.7, so the coverage and half-lengths plotted for this interval in Figures 7 and 8 belong to an interval at least as wide as the one described; the code should use $2\\mu_{\\mathrm{PD}}^2/\\nu_{\\mathrm{PD}}$, and the series should be computed again.", "id": "P004-E026", "location": "Figure 7 (caption) and Figure 8, p. 761 (PDF p. 18); Supplement A, Section A.7, equation (45) and the sentence after it, p. 17; replication archive (Supplement B), simulation/scripts/6_functions3.R, lines 196 and 312, and simulation/scripts/5_functions_generate_figures.R, lines 78–79, 87 and 97", "pdf_page": 18, "printed_page": "761", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Caption of Figure 7, p.~761: ``HAC\\_PD (edof) refers to the CI with HAC variance estimator in (22) and empirical degree of freedom adjustment.''\n\nSupplement A, Section A.7, p.~17:\n\\[\n\\nu = 2 * \\mathbf{Trace}\\left(\\left(\\frac{N_1N_0}{N}\\mathbf{M}\\left(\\Omega\\circ\\left(\\lambda'\\lambda\\right)\\right)\\mathbf{M}\\right)\\left(\\frac{N_1N_0}{N}\\mathbf{M}\\left(\\Omega\\circ\\left(\\lambda'\\lambda\\right)\\right)\\mathbf{M}\\right)\\right). \\tag{45}\n\\]\n``We use the $\\frac{\\alpha}{2}$-quantile and $\\left(1-\\frac{\\alpha}{2}\\right)$-quantile of the t-distribution with $\\frac{2\\mu^2}{v}$ degree of freedom to construct the confidence interval.''\n\n\\texttt{6\\_functions3.R}, line 196 (line 312 is identical), shown here on two lines:\n\n$\\texttt{return(c(Conley\\_var2/mu2,Conley\\_var2\\_pd/mu2\\_pd,SAHvar,}$\n\n$\\texttt{edof=2*mu2\\textasciicircum 2/v2,edof\\_pd=}\\hl{\\texttt{mu2\\_pd\\textasciicircum 2}}\\texttt{/v2\\_pd))}$\n\n\\texttt{5\\_functions\\_generate\\_figures.R}, lines 78--79:\n\n\\texttt{edof[which(edof<=1)]=1}\n\n\\texttt{edof\\_pd[which(edof\\_pd<=1)]=1}", "corrected": "In both return statements of \\texttt{6\\_functions3.R} (lines 196 and 312):\n\n$\\texttt{edof\\_pd=}\\hl{\\texttt{2*}}\\texttt{mu2\\_pd\\textasciicircum 2/v2\\_pd}$\n\nThen compute again the coverage and the half-length of the HAC\\_PD (edof) series in Figures 7 and 8 and in Figures 2, 3, 5, 6, 8 and 9 of Supplement A. In \\texttt{5\\_functions\\_generate\\_figures.R}, remove the reset of degrees of freedom below one (lines 78--79), or state the rule in A.7 if it is kept for the HAC series. After the factor is restored, the reset has no effect on HAC\\_PD whenever the quadratic form is nonzero. The effect of the correction on the published figures remains to be computed.", "evidence": "The evidence is an exact example, computed in rational arithmetic and reproduced by running the unchanged R and C++ routines in the archive. I take six nodes at $0,10,\\dots,50$ on a line, with $Z=(0,0,0,1,1,1)$, observed circle averages $Y=(0,0,3,1,1,1)$ and a kernel cutoff of $1$. Since the kernel matrix is the identity, the HAC and HAC\\_PD estimators coincide. Here $\\lambda=(-1,-1,-1,1,1,1)/3$, and the matrix in (45) is $\\mathbf{M}/6$, where $\\mathbf{M}$ is a projection of rank $4$. A direct computation gives $\\mu=2/3$ and $\\nu=2/9$. By A.7, I have $2\\mu^2/\\nu=4$ for both estimators. The routines return $4$ for HAC and $2$ for HAC\\_PD. The estimate is $0$, and both adjusted standard errors equal $1$. The half-length of the 95\\% interval is $2.776$ with $4$ degrees of freedom and $4.303$ with $2$. The coded interval contains the value $3$, and the stated interval does not.\n\nThe table adds two examples. In the second, the kernel is positive semidefinite with two clusters of three nodes, and the reset in the figure code restores the stated value. The third comes from a separate check with eight nodes, where the half-lengths are $2.234$ as stated and $2.905$ as coded.\n\nThe C++ routine stores $\\mathbf{M}A$ where A.7 has $\\mathbf{M}A\\mathbf{M}$, with $A$ the kernel-weighted matrix. Since $\\mathbf{M}^2=\\mathbf{M}$, the two matrices have the same trace, and their squares have the same trace. The routine computes the $\\mu$ and $\\nu$ of A.7, and no other normalization accounts for the missing factor.\n\n\\begin{audittable}{lrrrrrr}{Degrees of freedom in three exact examples: the value $2\\mu^2/\\nu$ of A.7, the values returned by the simulation routines, and the value used in the figure code. In each example the HAC and HAC\\_PD kernels are equal.}\nExample & $\\mu$ & $\\nu$ & A.7 & HAC, code & HAC\\_PD, code & HAC\\_PD, plotted \\\\\nSix nodes, identity kernel & $2/3$ & $2/9$ & $4$ & $4$ & $2$ & $2$ \\\\\nSix nodes, two clusters & $4/9$ & $32/81$ & $1$ & $1$ & $1/2$ & $1$ \\\\\nEight nodes, identity kernel & $3/4$ & $3/16$ & $6$ & $6$ & $3$ & $3$ \\\\\n\\end{audittable}", "problem": "Supplement A.7 chooses the degrees of freedom by matching two moments. Suppose that a quadratic form has mean $\\mu$ and variance $\\nu$. Divided by $\\mu$, it has mean $1$ and variance $\\nu/\\mu^2$. A chi-square variable with $f$ degrees of freedom, divided by $f$, has mean $1$ and variance $2/f$. Equating the two variances, I obtain $f=2\\mu^2/\\nu$, where $\\nu$ is twice the trace of the squared matrix, as in (45). The caption of Figure 7 applies this adjustment to the HAC interval and to the HAC\\_PD interval alike.\n\nThe two simulation routines, \\texttt{Regression\\_AME} and \\texttt{Regression\\_AME\\_obs}, return $2\\mu^2/\\nu$ for the HAC estimator and $\\mu_{\\mathrm{PD}}^2/\\nu_{\\mathrm{PD}}$ for the positive semidefinite one. One C++ routine computes $\\mu$ and $\\nu$ for both kernels, and its $\\nu$ already contains the factor 2 of (45). The value returned for HAC\\_PD is half of the stated one. The figure code takes this value as the degrees of freedom of the $t$ quantile, in the coverage and in the median half-length of the HAC\\_PD (edof) series. Since the positive upper $t$ critical value decreases as the degrees of freedom grow, the plotted interval is at least as wide as the interval that A.7 describes, and its coverage is at least as high.\n\nThe figure code also resets any degrees of freedom below one to one, a rule that A.7 does not state. Let $b_i\\ge 0$ be the eigenvalues of the matrix in (45) for a positive semidefinite kernel, not all zero. Then the stated value is $(\\sum_i b_i)^2/\\sum_i b_i^2\\ge 1$, which implies that the reset has no effect on HAC\\_PD once the factor is restored. With the factor missing, the reset raises a coded value below one back to one, and it then hides part of the error.\n\nThe affected series are HAC\\_PD (edof) in Figures 7 and 8 and in Figures 2, 3, 5, 6, 8 and 9 of Supplement A. These series are the simulation evidence for the interval that Section 7 recommends. The replication archive does not contain the simulation results needed to recompute these series, and I have not computed the series again.", "what_survives": "No formula in the paper changes. The moment-matching formula in A.7 is correct, and the HAC return expression already uses $2\\mu^2/\\nu$. Restoring the factor changes no variance estimate, and so it changes no series computed with a normal critical value and no SAH series. The point estimators, the asymptotic results, the mean squared errors and the randomization test do not involve the degrees of freedom. The application scripts compute their intervals by a separate path (\\fid{P004-N007}). What changes is the HAC\\_PD (edof) series: with the stated degrees of freedom its intervals are no wider, and its coverage no higher, than plotted. The size of the change is not known, and nothing here shows that the recommendation of Section 7 fails."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E026", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Equation (46): f_x(d) equals 3 alpha_x times (Gamma(d;1,1) minus Gamma(d;5,0.5)) times max((1 - d squared over 36), 0), with a red box around the max factor. Two lines below, a red box marks the words: is used to guarantee that there is no treatment effect beyond 6 units of distance.", "caption": "Supplement A, Appendix B.1, p. 18: equation (46) and the sentence that explains its last factor. The marks show the factor $\\max((1-d^2/36),0)$ and the statement that no effect extends beyond 6 units.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E027.png"}], "formal_release": null, "formal_statements": [], "headline": "The generator of the additive-effect simulations multiplies by $1-d^2/36$ without the maximum with zero that equation (46) prescribes, and its effects do not vanish beyond distance 6; the code should use $\\max(1-d^2/36,0)$.", "id": "P004-E027", "location": "Supplement A, Appendix B.1, equation (46) and the sentence after it, p. 18; replication archive (Supplement B), SpatialEffect/src/effect_nonmo.cpp, line 14", "pdf_page": 18, "printed_page": "18", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Supplement A, p.~18:\n\\[\nf_x(d)=3\\alpha_x\\times\\left(\\Gamma(d;1,1)-\\Gamma(d;5,0.5)\\right)\\times\\max\\left(\\left(1-\\frac{d^2}{36}\\right),0\\right), \\tag{46}\n\\]\n``The term $\\max\\left(\\left(1-\\frac{d^2}{36}\\right),0\\right)$ is used to guarantee that there is no treatment effect beyond 6 units of distance.''\n\n\\texttt{effect\\_nonmo.cpp}, line 14:\n\n\\texttt{return ( (Rcpp::dgamma(d, sh, 2*sc,0) - a*Rcpp::dgamma(d, 5*sh, sc,0))*\\hl{((-1)*d*d/36+1)});}", "corrected": "\\texttt{return ( (Rcpp::dgamma(d, sh, 2*sc,0) - a*Rcpp::dgamma(d, 5*sh, sc,0))*\\hl{Rcpp::pmax(1-d*d/36, 0.0)});}\n\nEquation (46) stays as printed. The additive simulation data, and the results computed from them, should be generated again before corrected figures are reported.", "evidence": "The evidence is an exact calculation and a compilation of the unchanged source. Fix $\\alpha_x=1$ and one treated node at $d=7$. Since $\\Gamma(d;1,1)=e^{-d}$ and $\\Gamma(d;5,0.5)=(4/3)d^4e^{-2d}$, the printed increment is $0$, and the coded increment is\n\\[\n3\\left(e^{-7}-\\frac{9604}{3}e^{-14}\\right)\\left(1-\\frac{49}{36}\\right)=\\frac{13}{12}\\left(\\frac{9604}{3}-e^{7}\\right)e^{-14},\n\\]\nwhich lies between $0.001895961$ and $0.001895963$. The compiled source returns $0.0018960$ at $d=7$, $0.00065$ at $d=8$ and $-0.000096$ at $d=10$, and it agrees with (46) for $d\\le 6$. For $d\\ge 13$ the coded function is negative. With $\\alpha_x=1$, a treated node at distance 14 then lowers every outcome on the circle of radius $1/2$ around another node. A copy of the source with the correction below agrees with (46) at every distance tested.", "problem": "Equation (46) is correct as printed. The code that generates the additive outcomes of (47) evaluates the effect function with the factor $1-d^2/36$, which is negative beyond $d=6$, where the printed factor is zero. The generator for the interactive case does truncate. In the additive simulations the effect of a treated node extends beyond 6 units. The effect there is small: about $0.0019\\,\\alpha_x$ at $d=7$, against about $1.7\\,\\alpha_x$ at $d=0.5$. The simulations set the HAC cutoff to $2\\cdot 6+2d$ because no effect is supposed to extend beyond 6 units. With the code as supplied this reason fails: two nodes at distance 14 have dependent circle averages at $d=1/2$, although $h(1/2)=13$.", "what_survives": "Equation (46), the theory, the interactive and null-effect generators and the two applications are unaffected by this omission. The supplied generator for the additive simulations illustrated in Figures 5, 6, 7 and 9 and Appendix B differs from (46) by this tail; I have not traced every published panel to its data. At $d=7$ the tail is about a thousandth of the effect at $d=0.5$. I have not computed the simulations again, and no conclusion is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E027", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Listing of source code with line numbers, in four parts. In line 7 of Conley.cpp a red box marks the name ConleySE in a definition that returns Rcpp::List and takes eight parameters. In line 7 of Conley_debug.cpp a red box marks the name ConleySE in a definition that returns void and takes the same eight parameters. In line 11 of effect_nonmo.cpp and in line 11 of effect_interactive.cpp red boxes mark the name nonmonoeffect_c, each with the parameters NumericVector d, double sh, double sc, double a. The first body returns a single expression; the second builds a vector of weights and returns the product of an output vector and the weights.", "caption": "Source code of the package in the replication archive, not a page of the article: src/Conley.cpp and src/Conley_debug.cpp, lines 6–7, and src/effect_nonmo.cpp and src/effect_interactive.cpp, from line 11, typeset from the files with line numbers added. Each marked name is defined in two files with the same list of parameter types. The two helper functions have different bodies.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E028.png"}, {"alt": "Listing of source code with line numbers, in three parts. In line 15 of RcppExports.cpp a red box marks the declaration ConleySE(arma::colvec res, arma::mat W_meat, arma::mat dist, double cutoff, int kernel). In line 25 a red box marks the call ConleySE(res, W_meat, dist, cutoff, kernel). Line 44 registers _SpatialEffect_ConleySE with 5 arguments. Lines 4 to 6 of RcppExports.R define the R function ConleySE with the arguments res, W_meat, dist, cutoff, kernel. Line 7 of Conley.cpp defines ConleySE with eight parameters, including XXinv, trim and if_edof.", "caption": "The generated bindings in the same package: src/RcppExports.cpp, lines 14–16, 25 and 43–44, and R/RcppExports.R, lines 4–6, above the definition in src/Conley.cpp, line 7. The marked declaration and call have five arguments; the definition has eight.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E028-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The source of the SpatialEffect package in the replication archive does not build as distributed: two C++ functions are each defined twice, and the generated bindings call a five-argument ConleySE that no file defines; the debug file should be left out, the two helper functions kept under distinct names, and the bindings and callers brought into agreement with one ConleySE interface.", "id": "P004-E028", "location": "Replication archive (Supplement B), package folder SpatialEffect/: src/Conley.cpp, line 7; src/Conley_debug.cpp, line 7; src/effect_nonmo.cpp and src/effect_interactive.cpp, line 11; src/RcppExports.cpp, lines 15, 25 and 44; R/RcppExports.R, lines 4–6. The code is not printed in the article; the archive is named on p. 766 (PDF p. 23).", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "result_fails", "effect_text": "a result or code path there relies on it and fails with it", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}, {"effect": "result_fails", "effect_text": "a result or code path there relies on it and fails with it", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{src/Conley.cpp}, line 7:\n\n\\texttt{Rcpp::List \\hl{ConleySE}(arma::colvec res, arma::mat W\\_meat, arma::mat dist, arma::mat XXinv, double cutoff, int kernel, int trim,int if\\_edof)\\{}\n\n\\texttt{src/Conley\\_debug.cpp}, line 7:\n\n\\texttt{void \\hl{ConleySE}(arma::colvec res, arma::mat W\\_meat, arma::mat dist, arma::mat XXinv, double cutoff, int kernel, int trim,int if\\_edof)\\{}\n\n\\texttt{src/effect\\_nonmo.cpp} and \\texttt{src/effect\\_interactive.cpp}, line 11 of each:\n\n\\texttt{NumericVector \\hl{nonmonoeffect\\_c}(NumericVector d, double sh, double sc, double a)\\{}\n\n\\texttt{src/RcppExports.cpp}, lines 15, 25 and 44:\n\n\\texttt{Rcpp::List ConleySE(\\hl{arma::colvec res, arma::mat W\\_meat, arma::mat dist, double cutoff, int kernel});}\n\n\\texttt{rcpp\\_result\\_gen = Rcpp::wrap(ConleySE(res, W\\_meat, dist, cutoff, kernel));}\n\n\\texttt{\\{\"\\_SpatialEffect\\_ConleySE\", (DL\\_FUNC) \\&\\_SpatialEffect\\_ConleySE, 5\\},}\n\n\\texttt{R/RcppExports.R}, line 4:\n\n\\texttt{ConleySE <- function(res, W\\_meat, dist, cutoff, kernel) \\{}", "corrected": "\\begin{itemize}\n\\item Leave \\texttt{\\hl{src/Conley\\_debug.cpp}} out of the package, and keep \\texttt{src/Conley.cpp}.\n\\item Keep both bodies of \\texttt{nonmonoeffect\\_c}, under \\hl{distinct names} or with internal linkage, and remove the export annotation above each where it is not needed.\n\\item Choose one interface for \\texttt{ConleySE}, and bring \\texttt{src/RcppExports.cpp}, \\texttt{R/RcppExports.R} and the callers into agreement with it. The callers in \\texttt{R/SpatialEffect.R} (lines 218--219) and \\texttt{R/SpatialEffect\\_internal.R} (lines 208--209) pass five arguments. An explicitly specified five-argument adapter is an alternative to changing every caller.\n\\item Check that the package installs from clean source and that representative calls of the replication run.\n\\end{itemize}", "evidence": "The evidence is a compilation of the unchanged source, with the binaries shipped in the folder removed. All eleven C++ files compile. Linking the resulting objects reports both duplicate symbols, and linking each pair of files alone reproduces each collision. The symbol tables show that the object built from \\texttt{RcppExports.cpp} requires a five-argument \\texttt{ConleySE} that no object defines; the entry point that the file registers, \\texttt{\\_SpatialEffect\\_ConleySE}, is itself defined. A library linked without the duplicates, for diagnosis only, fails to load for want of that function. A complete installation with \\texttt{R CMD INSTALL} stopped earlier, at a Fortran library missing from the machine I used. For that reason I showed the defects by linking the compiled objects directly.", "problem": "The ReadMe of the archive presents this folder as a standalone R package and says that installing it is essential for replicating the empirical results. When R builds a package, it compiles every \\texttt{.cpp} file in \\texttt{src/}, and neither \\texttt{Makevars} file excludes any of them. The files \\texttt{Conley.cpp} and \\texttt{Conley\\_debug.cpp} each define \\texttt{ConleySE} with eight parameters of the same types. Since a return type does not distinguish two C++ functions, the two definitions are the same external symbol, and the library cannot be linked. The same holds for \\texttt{nonmonoeffect\\_c}, which is defined in both effect files. Its two bodies differ: the one in \\texttt{effect\\_interactive.cpp} sets the effect to zero beyond distance 6, and the other does not. Apart from the duplicates, \\texttt{RcppExports.cpp} declares and calls a \\texttt{ConleySE} with five arguments, and the only definitions have eight. The library therefore needs a function that no source file provides.", "what_survives": "This finding shows no error in a theorem, estimand or printed number. The simulation scripts compile selected files one at a time with \\texttt{sourceCpp}, which generates its own bindings and avoids both collisions. Results produced earlier with a different installed binary are not shown to be wrong, and I did not compute them again. What changes is that the package cannot be installed from the archive without repair. I did not build a repaired package."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E028", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. In line 178, inside a call to SpatialEffect, a red box marks the argument trim = trim. In line 330, at the end of a call to ConleySE, a red box marks a sixth argument, as.integer(1). In line 366, inside a call to LocalReg, a red box marks the arguments as.integer(1), edf = TRUE; they are followed by dist.metric and bias_correction = 0.", "caption": "Source code in the replication archive, not a page of the article: application/scripts/Jayachandran_etal_2017.R, lines 176–179, 329–330 and 364–366, typeset from the file with line numbers added; long lines continue on the next row. The marked arguments are the ones that the called functions do not accept.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E029.png"}, {"alt": "Listing of source code with line numbers, in three parts. Lines 1 to 6 of SpatialEffect.R give the arguments of SpatialEffect; a red box marks the end of the list, edf = FALSE, m = 2, lambda = 0.02, nPerms = 1000, and the list has no argument named trim and no three dots. In line 4 of RcppExports.R a red box marks the five arguments of ConleySE: res, W_meat, dist, cutoff, kernel. In line 93 of SpatialEffect_internal.R red boxes mark the end of the arguments of LocalReg, dist.metric = \"Euclidean\", bias_correction = TRUE; the list has no argument named edf.", "caption": "The interfaces that these calls meet, in the package folder of the same archive: R/SpatialEffect.R, lines 1–6; R/RcppExports.R, lines 4–6; R/SpatialEffect_internal.R, lines 92–93. The marks are on the end of each list of arguments: SpatialEffect has no trim and no ..., ConleySE has five arguments, and LocalReg has no edf.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E029-b.png"}, {"alt": "Listing of source code with line numbers. Lines 127 to 137 build the data frame Zdata with the columns treatment, cov1, cov2, cov3 and prob_treatment. Lines 153 and 154 are comments that would have named columns x_coord and y_coord. Red boxes mark line 158, x_coord_Z assigned \"x_coord\", and line 159, y_coord_Z assigned \"y_coord\". In line 238 a red box marks the arguments x_coord_Z = x_coord_Z, y_coord_Z = y_coord_Z, ras_Z = NULL in the first call to SpatialEffect.", "caption": "application/scripts/Ferraro_etal_2015.R, lines 127–137, 153–154, 158–159 and 238–241. The table Zdata receives the columns treatment, cov1, cov2, cov3 and prob_treatment; the two lines that would name coordinate columns are commented out. The marked lines request the columns x_coord and y_coord in a call with ras_Z = NULL.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E029-c.png"}], "formal_release": null, "formal_statements": [], "headline": "Four calls in the two application scripts of the replication archive do not run against the package supplied with them: three calls in the Uganda script pass arguments that the functions do not have, and the first call in the Costa Rica script asks for coordinate columns that the script never creates.", "id": "P004-E029", "location": "Replication archive (Supplement B): application/scripts/Jayachandran_etal_2017.R, lines 176–179, 329–330 and 364–368; application/scripts/Ferraro_etal_2015.R, lines 127–137, 158–159 and 238–241; compared with SpatialEffect/R/SpatialEffect.R, lines 1–6, SpatialEffect/R/RcppExports.R, lines 4–6, and SpatialEffect/R/SpatialEffect_internal.R, lines 92–93. The code is not printed in the article; the applications are in Section 8, pp. 763–765 (PDF pp. 20–22).", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "result_fails", "effect_text": "a result or code path there relies on it and fails with it", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}, {"effect": "result_fails", "effect_text": "a result or code path there relies on it and fails with it", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{Jayachandran\\_etal\\_2017.R}, line 178, inside the call to \\texttt{SpatialEffect} on lines 176--179:\n\n\\texttt{per.se = per.se, conley.se = conley.se, cutoff = cutoff, alpha = alpha, edf = edf, \\hl{trim = trim}, bw = 10.996, bw\\_debias = 25.13,}\n\nLines 329--330:\n\n\\texttt{Conley\\_result <- ConleySE(as.vector(res\\_d), as.matrix(W\\_meat), as.matrix(dist),}\n\n\\texttt{as.double(c), as.integer(k), \\hl{as.integer(1)})}\n\nLines 364--366:\n\n\\texttt{wls\\_results <- LocalReg(dVec, Sdata, bw = 18.707, bw\\_debias = 25.13, dist = dist, Zup,}\n\n\\texttt{xevals = NULL, smooth.conley.se = 1, kernel, as.numeric(cutoff),}\n\n\\texttt{\\hl{as.integer(1), edf = TRUE}, dist.metric, bias\\_correction = 0)}\n\n\\texttt{Ferraro\\_etal\\_2015.R}, lines 158--159, and line 238, the first line of the call on lines 238--241:\n\n\\texttt{x\\_coord\\_Z <- \"x\\_coord\"}\n\n\\texttt{y\\_coord\\_Z <- \"y\\_coord\"}\n\n\\texttt{result.list <- SpatialEffect(ras = ras, Zdata = Zdata, \\hl{x\\_coord\\_Z = x\\_coord\\_Z, y\\_coord\\_Z = y\\_coord\\_Z, ras\\_Z = NULL},}", "corrected": "\\begin{itemize}\n\\item Bring the package and the two scripts to one interface, keeping the positive semidefinite trimming and the degrees-of-freedom calculation that the article describes. If the R interface of \\texttt{SpatialEffect} is kept as printed, remove \\texttt{\\hl{trim = trim}} from the call.\n\\item Give \\texttt{ConleySE} one definition, one generated binding and callers that agree with it (see \\fid{P004-E028}).\n\\item Call \\texttt{LocalReg} with named arguments that it has: omit \\texttt{\\hl{edf = TRUE}} and the extra \\texttt{\\hl{as.integer(1)}}, name \\texttt{dist.metric}, and keep \\texttt{bias\\_correction}. Supply the quantity \\texttt{edof} that line 368 uses.\n\\item In the Costa Rica script, skip the first call, whose result the later polygon calls overwrite, or pass it the polygons through \\texttt{\\hl{ras\\_Z = ras\\_Z}}. Coordinate columns are needed only if an analysis of point nodes is intended.\n\\item Run the repaired applications and compare them with the reported results.\n\\end{itemize}", "evidence": "The evidence is an exact check of the calls. I read each of the three Uganda calls from the script and evaluated it against the unchanged R functions of the package. The errors are ``unused argument (trim = trim)'', ``unused argument (as.integer(1))'' and ``unused arguments (edf = TRUE, dist.metric)''; they arise before any data are used. For Costa Rica, I ran the script's own construction of \\texttt{Zdata} and its first call on a small explicit input, a treatment raster of four cells and an outcome raster of nine cells equal to 1. The sampling points are empty, the four circle averages are \\texttt{NaN}, and the regression stops with ``0 (non-NA) cases''. With coordinate columns added the same call completes. The call on lines 364--366 also passes \\texttt{bias\\_correction = 0}; \\texttt{LocalReg} has that argument, and it is not part of this finding.", "problem": "The function \\texttt{SpatialEffect} of the supplied package has no argument \\texttt{trim} and no \\texttt{...}, and R rejects the first Uganda call with ``unused argument (trim = trim)''. The R function \\texttt{ConleySE} has five arguments and the script passes six. \\texttt{LocalReg} has no argument \\texttt{edf}; in the call on lines 364--366 the unnamed \\texttt{as.integer(1)} takes the place of \\texttt{dist.metric}, and both \\texttt{edf = TRUE} and the unnamed \\texttt{dist.metric} are left without a match. Removing \\texttt{edf} alone is therefore not enough. Line 368 then reads \\texttt{wls\\_results[[\"edof\"]]}, which \\texttt{LocalReg} never returns. In the Costa Rica script, \\texttt{Zdata} has the five columns \\texttt{treatment}, \\texttt{cov1}, \\texttt{cov2}, \\texttt{cov3} and \\texttt{prob\\_treatment}. Since the first call sets \\texttt{ras\\_Z = NULL}, the function treats the nodes as points and looks up the columns \\texttt{x\\_coord} and \\texttt{y\\_coord}, which do not exist. The two later calls pass the polygons, but they run only after the first call returns.", "what_survives": "No estimand, proposition or proof depends on these calls, and no reported estimate or interval is shown to be wrong. The maps and the plot of the propensity scores come before the failing calls. What changes is that the application estimates and intervals of Section 8 cannot be regenerated by running these scripts as supplied. I did not run a repaired application, and I did not test whether the output of each script matches Figures 11 and 13."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E029", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. A red box marks line 1, rm(list=ls()). In line 20 a red box marks today_date inside paste0, which builds the name of the first output file. Lines 22 and 23 load the null results and read simulation_result. Lines 50 and 51 load the interactive results and read simulation_result. Line 77 builds the output name with nonmono in it. A red box marks line 79, temp=simulation_result[['output_rt_list']], which is not preceded by a load. Line 93 sets the title Rejection Probability (Additive Effect).", "caption": "Source code in the replication archive, not a page of the article: simulation/scripts/2_generate_figures_03.R, lines 1, 20–23, 48–51, 77–82 and 93, typeset from the file with line numbers added. Line 1 clears the workspace and line 20 uses today_date. The block for the additive panel reads simulation_result on line 79 without a load, unlike the blocks on lines 20–23 and 48–51.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-E030.png"}], "formal_release": null, "formal_statements": [], "headline": "The script for the randomization-test panels of Figure 9 clears the workspace and then uses a date variable that it never defines, and it would draw the additive panel from the interactive results; two inserted lines repair both defects.", "id": "P004-E030", "location": "Replication archive (Supplement B): simulation/scripts/2_generate_figures_03.R, lines 1, 20, 48, 50 and 77–96. The code is not printed in the article; the script is associated with the three panels of Figure 9, p. 762 (PDF p. 19).", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{2\\_generate\\_figures\\_03.R}, lines 1, 20, 50, 77, 79 and 93:\n\n\\texttt{\\hl{rm(list=ls())}}\n\n\\texttt{figure\\_title\\_null =paste0('../graph\\_new/Randomization\\_test\\_null\\_', \\hl{today\\_date},'.png') \\#figure name}\n\n\\texttt{load('../data\\_new/Paired\\_randomization\\_test\\_interactive\\_10\\_Jun\\_12\\_points.RData')}\n\n\\texttt{figure\\_title\\_nonmono =paste0('../graph\\_new/Randomization\\_test\\_nonmono\\_', today\\_date,'.png') \\#figure name}\n\n\\texttt{\\hl{temp=simulation\\_result[['output\\_rt\\_list']]}}\n\n\\texttt{ggtitle('Rejection Probability (Additive Effect)')+}\n\nNo line of the file assigns \\texttt{today\\_date}, and no \\texttt{load} stands between line 50 and line 96.", "corrected": "Insert after line 1:\n\n\\texttt{\\hl{today\\_date = format(Sys.Date(), format=\"\\%b\\_\\%d\")}}\n\nInsert immediately before line 79:\n\n\\texttt{\\hl{load('../data\\_new/Paired\\_randomization\\_test\\_nonmono\\_10\\_Jun\\_12\\_points.RData')}}", "evidence": "The evidence is an exact counterexample, obtained by running lines 1--96 of the script itself; only line 9, which sets the working directory through RStudio, was left out. I replaced the three input files by small ones in which every rejection rate equals $0$ (null), $1/4$ (interactive) or $3/4$ (additive). As shipped, the script stops at line 20. With a date supplied after line 1, the panel titled ``Additive Effect'' shows $1/4$ at each of the 39 distances and three sample sizes, the same 117 values as the interactive panel. With the missing \\texttt{load} inserted it shows $3/4$. The name of the missing file appears in the neighbouring script \\texttt{2\\_generate\\_figures\\_02.R}.", "problem": "Line 1 clears the workspace, and nothing defines \\texttt{today\\_date} before line 20 needs it for the name of the first output file. Once the working directory on line 9 is set through RStudio, the script stops at line 20 with ``object 'today\\_date' not found''. Once a date is supplied, a second defect appears. The third block has the output name \\texttt{nonmono} and the title ``Rejection Probability (Additive Effect)'', but it reads \\texttt{simulation\\_result} on line 79 without loading the additive results. The object still holds the interactive results loaded on line 50, and the additive panel repeats the interactive one.", "what_survives": "The published Figure 9 has distinct additive and interactive curves, and this finding does not show it to be wrong. The simulation results that the script loads are not in the archive, and I could not regenerate its rejection probabilities. The null and interactive blocks load their own results. Later, the script summarises the test based on the supremum statistic and loads each scenario explicitly. The missing \\texttt{load} does not affect that summary, but the date error stops the script before it is reached."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E030", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of text. A red box marks the upper limit n of the sum in 'the Hajek estimator replaces them with N_1 = sum from i = 1 to n of Z_i and N_0 = N minus N_1, respectively'.", "caption": "Section 5, p. 754 (PDF p. 11), the sentence before equation (16). The marked upper limit of the sum is $n$; the number of intervention nodes is $N$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N001.png"}, {"alt": "Crop of the line 'First, note that the HAC variance estimator can be expressed as', of equation (21) and of four lines of text. Red boxes mark 'e-hat_n)' in 'where e-hat = (e-hat_1, ..., e-hat_n)' and the words 'n-by-n' in 'K is a n-by-n symmetric matrix'. The next two lines write the eigenvalue decomposition of K and the matrix K^PD with sums from i = 1 to n.", "caption": "Section 5, p. 756 (PDF p. 13), equation (21) and the paragraph after it. The marked $\\widehat e_n$ and “$n$-by-$n$” use $n$ for the number of nodes, as do the three sums and index sets that follow.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N001-b.png"}, {"alt": "Crop of footnote 22, 'That is, Y_i(Z) = Y_i(0) for all Z.' A red box marks 'Y_i(Z) = Y_i(0)'.", "caption": "Footnote 22, p. 761 (PDF p. 18). The marked null hypothesis is indexed by the node index $i$; outcome points are indexed by $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N001-c.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T31", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T31"}], "headline": "The number of intervention nodes is $N$, not $n$, in the treatment count and in the eigenvalue decomposition of $\\mathcal K$, and footnote 22 should index outcome points by $x$.", "id": "P004-N001", "location": "Section 5, p. 754 (PDF p. 11), the sentence before equation (16); p. 756 (PDF p. 13), the paragraph after equation (21); footnote 22, p. 761 (PDF p. 18)", "pdf_page": 11, "printed_page": "754", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T31"], "text": {"as_printed": "P.~754: ``the Hajek estimator replaces them with $N_1=\\textstyle\\sum_{i=1}^{\\hl{n}}Z_i$ and $N_0=N-N_1$, respectively,''.\n\nP.~756: ``where $\\widehat e=(\\widehat e_1,\\ldots,\\widehat e_{\\hl{n}})$, $\\mathcal K$ is a \\hl{$n$}-by-\\hl{$n$} symmetric matrix with $\\mathcal K_{ij}=K(\\frac{d_{ij}}{\\tilde d})$, and $\\circ$ denotes the pointwise (Hadamard) matrix product. Denote the eigenvalue decomposition of the matrix $\\mathcal K$ as $\\mathcal K=\\textstyle\\sum_{i=1}^{\\hl{n}}\\lambda_iv_iv_i'$, where $\\{\\lambda_i\\}_{i=1}^{\\hl{n}}$ are the eigenvalues and $\\{v_i\\}_{i=1}^{\\hl{n}}$ are the eigenvectors. We define $\\mathcal K^{\\mathrm{PD}}=\\textstyle\\sum_{i=1}^{\\hl{n}}\\max\\{\\lambda_i,0\\}v_iv_i'$''.\n\nFootnote 22, p.~761: ``That is, $Y_{\\hl{i}}(\\mathbf Z)=Y_{\\hl{i}}(0)$ for all $\\mathbf Z$.''", "corrected": "P.~754: ``the Hajek estimator replaces them with $N_1=\\textstyle\\sum_{i=1}^{\\hl{N}}Z_i$ and $N_0=N-N_1$, respectively,''.\n\nP.~756: ``where $\\widehat e=(\\widehat e_1,\\ldots,\\widehat e_{\\hl{N}})$, $\\mathcal K$ is a \\hl{$N$}-by-\\hl{$N$} symmetric matrix with $\\mathcal K_{ij}=K(\\frac{d_{ij}}{\\tilde d})$, and $\\circ$ denotes the pointwise (Hadamard) matrix product. Denote the eigenvalue decomposition of the matrix $\\mathcal K$ as $\\mathcal K=\\textstyle\\sum_{i=1}^{\\hl{N}}\\lambda_iv_iv_i'$, where $\\{\\lambda_i\\}_{i=1}^{\\hl{N}}$ are the eigenvalues and $\\{v_i\\}_{i=1}^{\\hl{N}}$ are the eigenvectors. We define $\\mathcal K^{\\mathrm{PD}}=\\textstyle\\sum_{i=1}^{\\hl{N}}\\max\\{\\lambda_i,0\\}v_iv_i'$''.\n\nFootnote 22: ``That is, $Y_{\\hl{x}}(\\mathbf Z)=Y_{\\hl{x}}(0)$ \\hl{for all $x\\in\\mathcal X$ and} all $\\mathbf Z$.''", "evidence": "The evidence is a check of pp.~751, 753--756, 758, 759 and 761 of the article.", "problem": "The article writes $N$ for the number of intervention nodes: the sums in (16) run to $N$, and the design matrix $\\mathbf X$ on p.~755 is $N$-by-$2$. The letter $n$ is introduced only in Section 7. In footnote 22 the sharp null hypothesis is written with the node index $i$, while outcomes are indexed by points $x\\in\\mathcal X$; Section 6.3 states the same hypothesis with $x$. The meaning is clear in each place. The letter $b$ denotes a constant in C2 and a function in C4; each is defined where it is used, and no correction is needed there.", "what_survives": "Nothing else changes. The estimators, the positive semidefinite variance estimator (22) and the randomization tests are as intended."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N001", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text. They read: Figure 6 shows the mean squared errors of the Hajek estimator for both additive-effect (31) and interactive-effect cases (32), with intervention-node sample sizes of 64, 100, and 144. In both cases the MSEs decrease as sample sizes increase, as predicted by our theory. Figure 7 and Figure 8 report coverage rates and median half-lengths for the Hajek estimator with different confidence interval constructions in the additive-effect case and the interactive-effect case, respectively. For brevity, we only display the case with a sample size of 144. The results are illustrated for AMEs at different distance values. Red outlines surround the words the mean squared errors of the Hajek estimator and the words coverage rates.", "caption": "Section 7, p. 760 (PDF p. 17). The outlines mark the two quantities that are computed against the approximated AME.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N003.png"}, {"alt": "Crop of three lines of text, equation (50) and one more line. The text reads: For our simulation, we use a Bernoulli design where Z sub i equals 1 with probability 0.5 for all intervention nodes. We run the simulation for 2000 times. Denote the pth assignment of Z as Z sub p and the value of Z sub i under the assignment as Z sub pi. To obtain AME, we first calculate. Equation (50) gives tau sub ix of eta as approximately the ratio of the sum over p from 1 to 2000 of Z sub pi times Y sub x of Z sub p to the sum of Z sub pi, minus the same ratio with 1 minus Z sub pi. The last line reads: Then tau sub i of d and eta and the AME can be constructed following their definitions. One red outline surrounds the words To obtain AME, we first calculate, and another surrounds the equation.", "caption": "Supplement A, Appendix B.1, p. 18. The outlines mark the sentence that introduces the approximation and equation (50).", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The mean squared errors and coverage rates of Figures 6 to 8 are computed against an AME that is itself estimated, by (50), from the same 2000 draws of the assignment that are being scored; the text does not say so.", "id": "P004-N003", "location": "Section 7, the two paragraphs on Figures 6 to 8, p. 760 (PDF p. 17); Supplement A, Appendix B.1, “Experimental Designs” and equation (50), p. 18", "pdf_page": 17, "printed_page": "760", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Main text, p.~760.} Figure 6 shows \\hl{the mean squared errors} of the Hajek estimator for both additive-effect (31) and interactive-effect cases (32), with intervention-node sample sizes of 64, 100, and 144. In both cases the MSEs decrease as sample sizes increase, as predicted by our theory.\n\nFigure 7 and Figure 8 report \\hl{coverage rates} and median half-lengths for the Hajek estimator with different confidence interval constructions in the additive-effect case and the interactive-effect case, respectively.\n\n\\emph{Supplement A, p.~18.} We run the simulation for 2000 times. Denote the $p$th assignment of $\\mathbf Z$ as $\\mathbf Z_p$ and the value of $Z_i$ under the assignment as $Z_{pi}$. \\hl{To obtain AME, we first calculate}\n\\[\n\\tau_{ix}(\\eta)\\approx\\frac{\\sum_{p=1}^{2000}Z_{pi}Y_x(\\mathbf Z_p)}{\\sum_{p=1}^{2000}Z_{pi}}-\\frac{\\sum_{p=1}^{2000}(1-Z_{pi})Y_x(\\mathbf Z_p)}{\\sum_{p=1}^{2000}(1-Z_{pi})}.\n\\]\nThen $\\tau_i(d;\\eta)$ and the AME can be constructed following their definitions.", "corrected": "Add after (50): \\hl{The mean squared errors and coverage rates reported below are computed relative to this approximation of the AME, which is obtained from the same 2000 draws. The error of the approximation is not assessed separately.}\n\nThe approximation can be avoided, since both designs have a closed form. Under (47), $\\tau_{ix}(\\eta)=f_x(d_{ix})$. Under (49) with assignment probability $1/2$,\n\\[\n\\tau_{ix}(\\eta)=f_x(d_{ix})+\\tfrac12\\,g_x(d_{ix})+\\tfrac12\\sum_{j:\\,\\mathcal N(j)=i}g_x(d_{jx}).\n\\]", "intended_meaning": "The figures give Monte Carlo approximations to the mean squared error and the coverage for the fixed, design-based AME. Equation (50) states that the AME is approximated from the simulated draws, and the formula is a correct approximation: under independent Bernoulli assignment, $\\tau_{ix}$ is the difference of the conditional means of $Y_x(\\mathbf Z)$ given $Z_i=1$ and given $Z_i=0$, and (50) is the difference of the corresponding sample means. Nothing printed is false.\n\nWhat the text leaves out is that the replication code computes this benchmark from the same outcomes and assignments on which the estimator is then evaluated. Let $B$ be the benchmark, $\\theta$ the AME, $\\delta=B-\\theta$, and $\\bar e$ the mean error of the estimates over the draws. Expanding the square, I find that the mean squared error computed against $B$ minus the one computed against $\\theta$ equals $\\delta^2-2\\delta\\bar e$ in every run. In a constructed example with 64 nodes and 2000 draws, the mean squared error is 4 against the AME and 3 against the benchmark, and an interval of half-width 1 covers the AME in no draw and the benchmark in three quarters of the draws. This example is not the design of the paper. The simulation outputs are not in the replication archive, and neither the size nor the sign of any effect on the published curves is known. The half-lengths of the intervals do not depend on the benchmark."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N003", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of body text. A red box marks the sentence: we recommend researchers to use the positive semidefinite HAC variance estimator with the empirical degree of adjustment when applying our method.", "caption": "Section 7, p. 761 (PDF p. 18): the recommendation that closes the discussion of Figures 7 and 8.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N004.png"}, {"alt": "Equation (44): mu equals the trace of (N1 N0 / N) M (Omega circ lambda-prime lambda) M. Below it a red box marks the sentence: The estimated variance is inflated by 1 over mu, becoming w-prime Sigma-hat HAC w over mu. Equation (45) for nu follows.", "caption": "Supplement A, Section A.7, p. 17: equation (44) and the sentence that describes the rescaling of the variance. A.7 defines $\\mu$ with the uniform kernel matrix $\\Omega$ only.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T7", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T7"}, {"id": "P004-T8", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T8"}], "headline": "The recommended interval divides the positive semidefinite HAC variance by a factor $\\mu$ computed from the trimmed kernel, and this factor can converge to a limit above one; under C1–C5 the adjusted variance can then stay below the variance of the estimator, and the interval can cover too rarely; the guarantee of Proposition 4 carries over once the factor is capped at one.", "id": "P004-N004", "location": "Section 7, p. 761 (PDF p. 18), end of the paragraph between Figures 7 and 8; Section 5, p. 756 (PDF p. 13), equation (22) and footnote 15; Supplement A, Section A.7, equation (44) and the sentence after it, p. 17; replication archive (Supplement B), simulation/scripts/6_functions3.R, line 196", "pdf_page": 18, "printed_page": "761", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [{"classification_after": {"class": "error", "severity": "Moderate"}, "classification_before": {"class": "error", "severity": "Moderate"}, "note": "One sentence of what survives was qualified; the classification is unchanged (error, Moderate). How this came to light: a Lean formalization of Proposition 4 showed that, under its conditions, the coverage probabilities need not have limits, which the clarity note P004-N024 now records. Why it changed: the sentence said that Proposition 4 is correct as printed; it now refers to the proposition read with limits inferior, which is what the proof of the proposition supports and what this entry relies on. P. M. Aronow decided to add the clarity note; this note was written by the model session that runs the audit."}], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P004-T7", "P004-T8"], "text": {"as_printed": "Section 7, p.~761: ``Based on these observations, we recommend researchers to use the positive semidefinite HAC variance estimator \\hl{with the empirical degree of adjustment} when applying our method.''\n\nSection 5, p.~756: ``Nonetheless, it is possible to design a biased-upward positive definite estimator.'' After (22): ``In addition, $\\widehat{\\Sigma}^{\\mathrm{PD}}_{\\mathrm{HAC}}(d)\\geq\\widehat{\\Sigma}_{\\mathrm{HAC}}(d)$.''\n\nSupplement A, Section A.7, p.~17: ``The estimated variance is inflated by $\\frac{1}{\\mu}$, becoming $\\frac{\\mathbf{w}'\\widehat{\\Sigma}_{\\mathrm{HAC}}\\mathbf{w}}{\\mu}$.''\n\n\\texttt{6\\_functions3.R}, line 196, second returned value: \\texttt{Conley\\_var2\\_pd/\\hl{mu2\\_pd}}", "corrected": "Recommendation of Section 7, with the adjustment of A.7 written out for the positive semidefinite estimator. Let $K^{\\mathrm{PD}}$ be the positive part of the kernel matrix $K$ of (21), let $\\widehat{V}_{\\mathrm{PD}}(d)$ be the $(2,2)$ entry of $\\widehat{\\Sigma}^{\\mathrm{PD}}_{\\mathrm{HAC}}(d)$ in (22), and let $\\mu_{\\mathrm{PD}}$ and $\\nu_{\\mathrm{PD}}$ be the quantities (44) and (45) computed with $K^{\\mathrm{PD}}$ in place of $\\Omega$. The recommended variance estimate is\n\\[\n\\widehat{V}^{\\mathrm{adj}}_{\\mathrm{PD}}(d)=\\frac{\\widehat{V}_{\\mathrm{PD}}(d)}{\\hl{\\min\\{\\mu_{\\mathrm{PD}},1\\}}}\\quad\\text{if }\\mu_{\\mathrm{PD}}>0,\\qquad \\widehat{V}^{\\mathrm{adj}}_{\\mathrm{PD}}(d)=\\widehat{V}_{\\mathrm{PD}}(d)\\quad\\text{otherwise.}\n\\]\nIn the first case the critical value $c$ is the $(1-\\alpha/2)$-quantile of the $t$ distribution with $2\\mu_{\\mathrm{PD}}^2/\\nu_{\\mathrm{PD}}$ degrees of freedom, and in the second it is the normal quantile $z_{1-\\alpha/2}$. The interval is $\\widehat{\\tau}_{\\mathrm{HA}}(d)\\pm c\\,\\widehat{V}^{\\mathrm{adj}}_{\\mathrm{PD}}(d)^{1/2}$.\n\nLet $K(\\cdot)$ in (19) be the uniform kernel with cutoff $\\tilde d=h(d)$. Under C1--C5, and if $N\\times\\operatorname{AVar}(\\widehat{\\tau}_{\\mathrm{HA}}(d))$ is uniformly bounded away from zero for all large $N$, the limit inferior of this interval's coverage probability for $\\operatorname{AME}(d;\\eta)$ is at least $1-\\alpha$, for each $0<\\alpha<1$. Indeed, $\\widehat{V}^{\\mathrm{adj}}_{\\mathrm{PD}}(d)\\ge\\widehat{V}_{\\mathrm{PD}}(d)\\ge\\widehat{V}_{\\mathrm{HAC}}(d)$ and $c\\ge z_{1-\\alpha/2}$, so the interval contains the interval of Proposition 4(ii) whenever $\\widehat{V}_{\\mathrm{HAC}}(d)\\ge 0$.\n\nI have not assessed the finite-sample coverage of the capped interval. If the uncapped division is kept, Section 7 should present the interval as a finite-sample device supported by the simulations, and it should say that Proposition 4 does not cover it.", "evidence": "The evidence is a pair of exact examples that satisfy C1--C5, with the limits derived and the finite matrices computed in rational arithmetic.\n\n\\emph{First example.} For each $m\\ge 1$, the design has $N=5m$ nodes in $m$ blocks. Block $b$ has a centre at $(10b,0)$ and four leaves at $(10b\\pm 1,0)$ and $(10b,\\pm 1)$. The assignments are independent with probability $1/2$. The circle averages at $d=0$ are $y=(0,1,1,-1,-1)$ in every block under every assignment, so that no interference is present and every treatment effect is zero. Conditions C1--C5 hold, and $h(0)=6/5$ is a valid cutoff in C3. With this cutoff the uniform kernel links the centre to each leaf and links no two leaves. In each block the kernel has eigenvalues $-1,1,1,1,3$. After the eigenvalue $-1$ is trimmed, the matrix $K^{\\mathrm{PD}}$ has diagonal $(3/2,9/8,9/8,9/8,9/8)$ and trace $6$.\n\nI write $s_i=2Z_i-1$. Since $\\sum_i y_i=0$ in each block, I have $\\widehat{\\tau}_{\\mathrm{HA}}=2N^{-1}\\sum_i s_iy_i+O_p(N^{-3/2})$, and hence $N\\operatorname{Var}(\\widehat{\\tau}_{\\mathrm{HA}})\\to (4/N)\\sum_i y_i^2=16/5$. For a kernel $Q$ made of identical blocks, $N\\widehat{V}_Q$ equals $4/N$ times the sum over blocks of $t'Qt$, up to a term that tends to zero in probability, where $t_i=s_iy_i$. Using the independence of the signs, I obtain $\\mathbb{E}[t'Qt]=\\sum_i Q_{ii}y_i^2$. This sum is $4$ for the uniform kernel and $4\\cdot 9/8=9/2$ for $K^{\\mathrm{PD}}$. By the law of large numbers, I find that $N\\widehat{V}_{\\mathrm{HAC}}\\to 16/5$ and $N\\widehat{V}_{\\mathrm{PD}}\\to 18/5$. Finally, $\\mu_Q$ equals $N^{-1}\\operatorname{Trace}(Q)$ up to a term that tends to zero, which implies that $\\mu\\to 1$ for the uniform kernel and $\\mu_{\\mathrm{PD}}\\to 6/5$. Combining these limits, I obtain\n\\[\nN\\,\\frac{\\widehat{V}_{\\mathrm{PD}}}{\\mu_{\\mathrm{PD}}}\\to\\frac{18/5}{6/5}=3<\\frac{16}{5},\n\\]\nso the adjusted variance converges to $15/16$ of the variance of the estimator. The degrees of freedom $2\\mu_{\\mathrm{PD}}^2/\\nu_{\\mathrm{PD}}$ tend to infinity, and the coverage of the nominal 95\\% interval tends to $2\\Phi(z_{0.975}\\sqrt{15/16})-1=0.9423$.\n\n\\emph{Second example.} The blocks now have four nodes at the corners of a unit square, $(10b,0)$, $(10b+1,0)$, $(10b+1,1)$ and $(10b,1)$. I use disk averages with counting measure. Each node $i$ has outcome $2Z_j-1$, where $j$ is its horizontal neighbour; at the reflection of $j$ across $i$, the outcome is $1-2Z_i$. The two effects of $Z_i$ are both at distance one from $i$ and cancel in every disk centred on $i$, so every marginal effect is zero. Here the cutoff $h(0)=1$ equals the range of the dependence. The trimmed kernel has diagonal $5/4$. Both unadjusted estimators are asymptotically exact, and the adjusted one converges to $4/5$ of the variance of the estimator.\n\n\\begin{audittable}{lrrrrrr}{Limits in the two examples as $N\\to\\infty$. The last column is the limiting coverage of the nominal 95\\% interval with variance $\\widehat{V}_{\\mathrm{PD}}/\\mu_{\\mathrm{PD}}$.}\nExample & $\\mu_{\\mathrm{PD}}$ & $N\\operatorname{Var}(\\widehat{\\tau}_{\\mathrm{HA}})$ & $N\\widehat{V}_{\\mathrm{PD}}$ & $N\\widehat{V}_{\\mathrm{PD}}/\\mu_{\\mathrm{PD}}$ & Ratio & Coverage \\\\\nFive-node blocks, no interference & $6/5$ & $16/5$ & $18/5$ & $3$ & $15/16$ & $0.9423$ \\\\\nFour-node blocks, paired interference & $5/4$ & $8$ & $8$ & $32/5$ & $4/5$ & $0.9204$ \\\\\n\\end{audittable}\n\nThe ordering also fails in finite samples. In the first example with $N=20$ and a balanced assignment, $\\widehat{V}_{\\mathrm{PD}}=\\widehat{V}_{\\mathrm{HAC}}=4/25$, $\\mu=41/50$ and $\\mu_{\\mathrm{PD}}=101/100$, and the adjusted values are $16/101<8/41$. In the second with $N=16$, both variances equal $3/4$, $\\mu=13/16$ and $\\mu_{\\mathrm{PD}}=33/32$, and the adjusted values are $8/11<12/13$. The unchanged R and C++ functions in the archive reproduce the values of the second case. The missing factor of two in the coded degrees of freedom (\\fid{P004-E026}) plays no part here, because the stated and the coded degrees of freedom both tend to infinity.", "problem": "Section 5 shows that the positive semidefinite estimator is at least as large as the HAC estimator, and under the conditions of Proposition 4(ii) the unadjusted HAC interval has asymptotic coverage of at least $1-\\alpha$. The interval with the positive semidefinite variance $\\widehat{V}_{\\mathrm{PD}}$ and a normal critical value inherits this guarantee. The recommended interval is a different one. Section A.7 defines the adjustment for the uniform kernel matrix $\\Omega$ only. The replication code extends it to the positive semidefinite estimator: it computes $\\mu_{\\mathrm{PD}}$ and $\\nu_{\\mathrm{PD}}$ from the trimmed kernel $K^{\\mathrm{PD}}$ of Section 5 in place of $\\Omega$, and it returns $\\widehat{V}_{\\mathrm{PD}}/\\mu_{\\mathrm{PD}}$.\n\nThe diagonal entries of $\\Omega$ equal one, and in the examples below $\\mu$ tends to one. Trimming the negative eigenvalues adds a positive semidefinite matrix to the kernel, which raises its trace. In the examples below, $\\mu_{\\mathrm{PD}}$ tends to the average diagonal entry of $K^{\\mathrm{PD}}$, which is $6/5$ in one and $5/4$ in the other. Dividing by a factor above one does not inflate the variance. It deflates it, and the upward bias of $\\widehat{V}_{\\mathrm{PD}}$ need not make up the difference. The ordering $\\widehat{V}_{\\mathrm{PD}}/\\mu_{\\mathrm{PD}}\\ge\\widehat{V}_{\\mathrm{HAC}}/\\mu$ can then fail, and $N\\widehat{V}_{\\mathrm{PD}}/\\mu_{\\mathrm{PD}}$ can converge to a limit below the limit of $N\\operatorname{Var}(\\widehat{\\tau}_{\\mathrm{HA}})$. In the examples the $t$ critical value does not compensate, because the degrees of freedom tend to infinity.\n\nThe paper states no theorem for the adjusted estimator; this failure does not contradict Proposition 4 or the printed unadjusted inequality. What fails is the recommendation when it is read as a procedure valid under C1--C5. Figures 7 and 8 report this interval, and the captions of Figures 11 and 13 describe the intervals of the two applications in the same terms.", "what_survives": "Proposition 4, read with limits inferior of its coverage probabilities as in \\fid{P004-N024}, the inequality $\\widehat{\\Sigma}^{\\mathrm{PD}}_{\\mathrm{HAC}}(d)\\geq\\widehat{\\Sigma}_{\\mathrm{HAC}}(d)$ and footnote 15 are correct; they concern the unadjusted estimators. The definition and identification of the AME, the point estimators with their limit theorems, and the randomization test are unaffected. The examples do not show that the coverage plotted in Figures 7 and 8 is wrong for those designs, or that an interval in Figure 11 or 13 is too short, and they do not show that $\\mu_{\\mathrm{PD}}$ exceeds one in every design. What changes is the recommendation. As implemented, the adjusted positive semidefinite interval has no asymptotic guarantee under C1--C5. The capped version below has one, and it is wider wherever $\\mu_{\\mathrm{PD}}>1$."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N004", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Five lines of body text. A red box marks the words: cutoff values (d-tilde = 2 km, 5 km and 10 km). A footnote mark 25 follows the parenthesis.", "caption": "Section 8.1, p. 763 (PDF p. 20): the values 2, 5 and 10 km are introduced as cutoff values and written $\\tilde{d}$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N005.png"}, {"alt": "Footnotes 24 and 25. In footnote 25 a red box marks the words: bandwidth at 2 d-tilde + 2d.", "caption": "Footnote 25, p. 763 (PDF p. 20): the bandwidth actually used is $2\\tilde{d}+2d$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N005-b.png"}, {"alt": "Three lines of body text. A red box marks the sentence: d-tilde is a cutoff value, and our setup suggests setting it at d-tilde = h(d).", "caption": "Section 5, p. 755 (PDF p. 12), after equation (19): $\\tilde{d}$ is defined as the cutoff of the kernel, with the suggested value $h(d)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N005-c.png"}], "formal_release": null, "formal_statements": [], "headline": "In Section 8 and in the legends of Figures 11 and 13, $\\tilde{d}$ and “cutoff” denote the assumed range of interference, not the cutoff of the HAC kernel that Section 5 calls $\\tilde{d}$; the kernel cutoff used at distance $d$ is twice the stated value plus $2d$.", "id": "P004-N005", "location": "Section 8.1, p. 763 (PDF p. 20), and footnote 25; Section 5, p. 755 (PDF p. 12), after equation (19); legends of Figures 11 and 13, pp. 764–765", "pdf_page": 20, "printed_page": "763", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 5, p.~755: ``$\\tilde{d}$ is a cutoff value, and our setup suggests setting it at $\\tilde{d}=h(d)$.''\n\nSection 8.1, p.~763: ``we assess the robustness of inferences by considering a range of \\hl{cutoff values ($\\tilde{d}$ = 2 km, 5 km and 10 km)}.''\n\nFootnote 25: ``To be precise, for AME at distance value d, we set the \\hl{bandwidth at $2\\tilde{d}+2d$}, as discussed in Section 4.''\n\nLegends of Figures 11 and 13: cutoff=2km, cutoff=5km, cutoff=10km.", "corrected": "Section 8.1: ``we assess the robustness of inferences by considering a range of \\hl{assumed ranges of interference ($\\bar{d}$ = 2 km, 5 km and 10 km)}.''\n\nFootnote 25: ``To be precise, for AME at distance value $d$, we set the \\hl{kernel cutoff at $\\tilde{d}=2\\bar{d}+2d$}, as discussed in Section 4.''\n\nIn the legends and captions of Figures 11 and 13, and in the sentence of Section 8.2 that sets the cutoff value, write ``assumed interference range'' in place of ``cutoff''.", "intended_meaning": "Footnote 25 settles the meaning. The values 2, 5 and 10 km are assumed ranges of interference, the quantity written $\\bar{d}$ in Section 4, where $h(d)$ is set to $2\\bar{d}+2d$. The kernel cutoff of (19) at distance $d$ is then $4+2d$, $10+2d$ or $20+2d$ km. The application script in the replication archive computes its kernel cutoff in the same way, which supports this reading. No bandwidth or interval is wrong because of the notation."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N005", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Four lines of body text and two footnotes. Red boxes mark the value -0.034, the words 10,000 random draws, and the p-value is around 0.065, and in footnote 24 the words a triangular kernel with a bandwidth of five km.", "caption": "Section 8.1, last paragraph of p. 763 (PDF p. 20), with footnotes 24 and 25. The marks show the reported statistic, the number of draws with the p-value, and the smoothing bandwidth.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N006.png"}, {"alt": "Paragraph headed Replication archive, with the DOI 10.1214/24-AOAS1986SUPPB. Two red boxes mark the words: All code and data to replicate simulation and empirical results are available in the Supplementary Material.", "caption": "Supplementary Material, p. 766 (PDF p. 23): the statement that the archive contains all code and data to replicate the empirical results.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The supplied Uganda results and script do not reproduce the published cumulative statistic $-0.034$: the stored results give $-0.052$ over 0–5 km and $-0.039$ over 1–5 km, and the script differs from the article in the range of the statistic, the number of draws, the smoothing bandwidth and the cutoffs; the computation behind the published values needs to be identified and supplied.", "id": "P004-N006", "location": "Section 8.1, last paragraph of p. 763 (PDF p. 20), continuing on p. 764, and footnote 24; Supplementary Material statement, p. 766 (PDF p. 23); replication archive (Supplement B), Jayachandran_etal_2017.R, lines 116, 178, 193, 291 and 364, and forestResult_reproj.RData", "pdf_page": 20, "printed_page": "763", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 8.1, p.~763: ``The test statistic is the sum of the point Hajek estimates between zero and five km. The estimated value is \\hl{$-0.034$} (a 3.4 percentage point decline in deforested area). For the randomization test, we use \\hl{10,000 random draws}, and the p-value is around \\hl{0.065}.''\n\nFootnote 24: ``We use a triangular kernel with a bandwidth of \\hl{five km} to construct the smoothed Hajek estimator.''\n\nSupplementary Material, p.~766: ``Replication archive (DOI: 10.1214/24-AOAS1986SUPPB; .zip). All code and data to replicate simulation and \\hl{empirical results are available in the Supplementary Material} as well as on the following github repository: \\url{https://github.com/YeWang1576/SpatialEffectPackage}''\n\n\\texttt{Jayachandran\\_etal\\_2017.R}, line 116 and the call on line 193:\n\n\\texttt{nPerms <- \\hl{500}}\n\n\\texttt{test.result <- SpatialEffectTest(result.list, test.range = \\hl{c(1, 5)}, smooth = 0, alpha = 0.1)}", "corrected": "No replacement value is proposed. For the results of Section 8.1, the archive should contain:\n\\begin{itemize}\n\\item the processing of the outcome raster, the geometry and projection, the distance grid, the matrix of circle averages, the estimates and the saved result object that give \\hl{$-0.034$};\n\\item the randomization design, the tail convention, the seed and the package versions that give the reported p-value of \\hl{around $0.065$};\n\\item a script whose settings agree with the article: range \\hl{zero to five km}, \\hl{10,000} draws, smoothing bandwidth \\hl{five km}, cutoff values \\hl{2, 5 and 10 km}.\n\\end{itemize}\nThe published numbers should not be replaced by the stored ones before the final computation is identified and checked.", "evidence": "The evidence is a recalculation from the archive in exact arithmetic. I read the stored circle averages and the six stored estimates as binary fractions and computed (16) again from the 726 circle averages. Each recomputed estimate agrees with the stored one to within $10^{-16}$.\n\n\\begin{audittable}{rrrr}{Stored results of the Uganda application: mean circle average in control and treated villages, and the Hajek estimate (16).}\nDistance (km) & Control & Treated & Estimate \\\\\n0 & $0.02608$ & $0.01263$ & $-0.01344$ \\\\\n1 & $0.02353$ & $0.01566$ & $-0.00786$ \\\\\n2 & $0.02283$ & $0.01472$ & $-0.00811$ \\\\\n3 & $0.02352$ & $0.01389$ & $-0.00963$ \\\\\n4 & $0.02181$ & $0.01541$ & $-0.00640$ \\\\\n5 & $0.02308$ & $0.01610$ & $-0.00698$ \\\\\nSum, 0--5 km & & & $-0.05242$ \\\\\nSum, 1--5 km & & & $-0.03898$ \\\\\nPublished & & & $-0.034$ \\\\\n\\end{audittable}\n\n\\begin{audittable}{lll}{Settings of Section 8.1 in the article and in the script of the archive.}\nSetting & Article & Script \\\\\nRange of the test statistic & zero to five km & \\texttt{test.range = c(1, 5)} \\\\\nRandom draws & 10,000 & \\texttt{nPerms <- 500} \\\\\nSmoothing bandwidth & five km & \\texttt{bw = 10.996} and \\texttt{bw = 18.707} \\\\\nCutoff values & 2, 5 and 10 km & \\texttt{seq(2, 10, 2)} \\\\\n\\end{audittable}\n\nAs a further check, I held the stored outcomes fixed and drew 10,000 assignments of 60 treated villages among the 121. The absolute value of the sum over 0--5 km is at least the observed one in 626 draws, and over 1--5 km in 857. These are counts for one Monte Carlo sample.\n\nSeparately, I recalculated the circle averages from the supplied outcome raster using the authors' functions, with changes to the supplied script. The estimates over 0--5 km sum to $-0.0427592$, with a two-sided Monte Carlo fraction of $0.0928$ from 10,000 draws (Monte Carlo standard error $0.00290$). This calculation uses newly computed circle averages, rather than the stored result object above. All eight selected unsmoothed and smoothed point estimates agree with Figure 11 within the stated graph-reading tolerances; selected interval endpoints were also compared. At 0 km the lower endpoint using the 5 km cutoff and the degrees of freedom of A.7 is $-0.027219$, against approximately $-0.023$ in the figure, a difference just beyond the graph-reading tolerance of $0.004$. These comparisons do not establish an exact discrepancy in an endpoint that the figure does not print numerically.", "problem": "The archive contains the result object that the Uganda script saves and then loads. It holds the circle averages of 121 villages, 60 treated and 61 control, at distances $0,1,\\dots,30$ km, and the Hajek estimates computed from them. The six stored estimates for 0 to 5 km sum to $-0.0524$. Over 1 to 5 km, the range that the script passes to the test, they sum to $-0.0390$. Both sums are below $-0.0345$, so neither rounds to $-0.034$. The script also differs from the article in four settings, listed in the second table. These stored results and script settings do not reproduce the published calculation.\n\nWhich computation produced $-0.034$ and the p-value around $0.065$ is not identified. I do not conclude that the published values are wrong, because nothing ties the stored object to the final computation.", "what_survives": "Equation (16), the estimator and the procedure of the randomization test are not in question. The statistic from the stored results is still negative, and the Monte Carlo fraction for 0--5 km is $0.0626$, below $0.10$. This calculation gives no reversal of the reported rejection at the 10\\% level. The published statistic $-0.034$ and p-value around $0.065$ are not shown to be wrong. They are not reproduced by the supplied Uganda results and script. The simulations and the Costa Rica application are not affected by this finding. The interpretation of the statistic is a separate matter (\\fid{P004-E008})."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N006", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Five lines of body text and two footnotes. Red boxes mark the sentence: For the randomization test, we use 10,000 random draws, and the p-value is around 0.065.", "caption": "Section 8.1, last paragraph of p. 763 (PDF p. 20): the randomization test and its p-value.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N007.png"}, {"alt": "Two lines of body text and footnote 3. Red boxes mark the sentence: Under polygon intervention, d_ij = min over x in i and x-prime in j of gamma(x, x-prime).", "caption": "Footnote 3, p. 751 (PDF p. 8): the distance between two polygons is the minimal distance between their points.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N007-b.png"}, {"alt": "Equation (45): nu equals 2 times the trace of the square of the matrix (N1 N0 / N) M (Omega circ lambda-prime lambda) M, with a red box around the factor 2 and the star after it. Below it, the sentence that sets the degrees of freedom of the t-distribution, with a red box around the fraction 2 mu squared over v.", "caption": "Supplement A, Section A.7, p. 17: equation (45) and the stated degrees of freedom, $2\\mu^2/v$ with $v$ the $\\nu$ of (45).", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N007-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The Uganda script in the replication archive draws its randomization distribution by permuting treatment with a fixed count and no blocks, which is not the lottery within subcounties among covariate-balanced lists that assigned the villages; the same script measures HAC distances between centroids where footnote 3 prescribes minimal distances between polygons, and it uses twice the degrees of freedom of Supplement A.7.", "id": "P004-N007", "location": "Section 8.1, last paragraph of p. 763 (PDF p. 20), and Figure 11, p. 764; footnote 3, p. 751 (PDF p. 8); Supplement A, Section A.7, equation (45) and the sentence after it, p. 17; replication archive (Supplement B), application/scripts/Jayachandran_etal_2017.R, lines 111, 170–173, 286–289 and 339–342", "pdf_page": 20, "printed_page": "763", "reaches": [{"effect": "result_needs_same_repair", "effect_text": "a result or code path there needs the same repair", "explanation": null, "package": null, "recorded_as": "P008-N012", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P008-N012", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N012", "work_id": "P008"}], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 8.1, p.~763: ``For the randomization test, we use 10,000 random draws, and the p-value is around 0.065.''\n\nFootnote 3, p.~751: ``As before, under point intervention, $d_{ij}=\\gamma(x(i),x(j))$. Under polygon intervention, $d_{ij}=\\min_{x\\in i,x'\\in j}\\gamma(x,x')$.''\n\nSupplement A, Section A.7, p.~17: equation (45),\n\\[\n\\nu = 2 * \\mathbf{Trace}\\left(\\left(\\frac{N_1N_0}{N}\\mathbf{M}\\left(\\Omega\\circ(\\lambda'\\lambda)\\right)\\mathbf{M}\\right)\\left(\\frac{N_1N_0}{N}\\mathbf{M}\\left(\\Omega\\circ(\\lambda'\\lambda)\\right)\\mathbf{M}\\right)\\right).\n\\]\n``We use the $\\frac{\\alpha}{2}$-quantile and $\\left(1-\\frac{\\alpha}{2}\\right)$-quantile of the t-distribution with $\\frac{2\\mu^2}{v}$ degree of freedom to construct the confidence interval.''\n\n\\texttt{Jayachandran\\_etal\\_2017.R}, lines 111, 170, 286 and 341--342:\n\n\\texttt{blockvar <- \\hl{NULL}}\n\n\\texttt{centr <- st\\_coordinates(\\hl{st\\_centroid}(map))}\n\n\\texttt{Zmat <- Zdata[, c(1:2)]}\n\n$\\texttt{nu <- (nz/((nz-2)*XX\\_mat\\_inv[2, 2]))\\textasciicircum 2*sum(diag(dof\\_mat \\%*\\% dof\\_mat))}$\n\n$\\texttt{cvs[d] <- (}\\hl{\\texttt{2*}}\\texttt{dofs[d]\\textasciicircum 2) / nu}$", "corrected": "\\begin{itemize}\n\\item Draw reference assignments from the trial's lottery: in each of the seven subcounties, \\hl{choose with equal probability between its two original balanced lists}, conditioning on those paired lists where available; otherwise reproduce their original selection procedure before the lottery. Setting \\texttt{blockvar} to the subcounty alone does not reproduce this lottery.\n\\item In the loop over cutoffs, take the distances between villages as \\hl{minimal distances between polygons}, as in footnote 3, for the HAC kernel and for the degrees of freedom.\n\\item Replace line 342 by $\\texttt{cvs[d] <- }\\hl{\\texttt{dofs[d]\\textasciicircum 2}}\\texttt{ / nu}$.\n\\item Compute the intervals and the p-value again before any published number is revised.\n\\end{itemize}", "evidence": "For the reference distribution, the evidence is a check of the trial's published design against the code. The article of Jayachandran et al.\\ (2017) also states that the randomization was stratified by subcounty.\n\nFor the other two parts, the evidence is a pair of exact examples and a recalculation from the data in the archive. I take six nodes with $Z=(0,1,0,1,0,1)$ and $Y=(0,2,1,1,2,0)$ and an identity kernel. By A.7, I have $4$ degrees of freedom, and the expressions in the script return $8$. Next, I keep this assignment and these outcomes, place six rectangles of width 3 and height 1 on a line, with centres at $0,4,12,20,28,36$, and take a cutoff of 2. The first two rectangles are at distance $1$ as polygons and $4$ as centroids, and no other pair is within the cutoff. The unadjusted HAC variance is $2/3$ with polygon distances and $4/9$ with centroid distances.\n\nFor the Uganda data at $d=0$, the script's calculation, done again from the data, reproduces the five stored pairs of interval endpoints to within $10^{-12}$. The first table compares the script's degrees of freedom with those of A.7 and lists the half-widths at the stored standard errors. At the largest cutoff the upper endpoint of the stored interval moves from $-0.00037$ to $+0.00051$, so that this interval contains zero after the change. The second table counts the pairs of villages within the kernel cutoff. The cutoffs of the first table are those of the script, not those of Figure 11.\n\n\\begin{audittable}{rrrrrr}{Uganda data, $d=0$, stored estimate $-0.01344$: degrees of freedom and half-widths of the 90\\% interval at the stored standard errors.}\nNominal cutoff (km) & df, script & df, A.7 & Half-width, script & Half-width, A.7 & Ratio \\\\\n2 & $77.48$ & $38.74$ & $0.01144$ & $0.01158$ & $1.012$ \\\\\n4 & $39.28$ & $19.64$ & $0.01480$ & $0.01516$ & $1.025$ \\\\\n6 & $26.51$ & $13.26$ & $0.01555$ & $0.01613$ & $1.038$ \\\\\n8 & $18.05$ & $9.03$ & $0.01443$ & $0.01526$ & $1.057$ \\\\\n10 & $15.58$ & $7.79$ & $0.01307$ & $0.01395$ & $1.067$ \\\\\n\\end{audittable}\n\n\\begin{audittable}{rrr}{Pairs of distinct villages within the kernel cutoff at $d=0$, out of 7,260 pairs, by the script's ellipsoidal centroid distances and by minimal spherical distances between polygons.}\nKernel cutoff (km) & By centroid distance & By polygon distance \\\\\n4 & 309 & 837 \\\\\n10 & 1,480 & 2,154 \\\\\n20 & 3,378 & 3,902 \\\\\n\\end{audittable}", "problem": "I find three discrepancies in the script that the archive offers for Section 8.1.\n\n\\emph{The reference distribution.} The test of Section 6.3 compares the observed statistic with its distribution over assignments drawn from the design. The script draws assignments uniformly among those with 60 treated villages out of 121, with no blocks. Note that fixing the number treated is not in itself an error: under Bernoulli assignment with a common probability, the assignment is uniform given its count. The trial was not assigned in that way. Its supplementary materials (Jayachandran et al., 2017, \\href{https://seemajayachandran.com/supplementary_material_pes.pdf}{Section 2.2}) describe a public lottery in each of seven subcounties. The villages of a subcounty were divided into two lists of nearly equal size, and only divisions balanced on four covariates were admitted: ``For each subcounty, among the balanced permutations, we randomly chose one to use in the public lottery.'' The lottery then selected the list to be treated. Permutations without blocks ignore both the subcounties and the balance restriction. The reference distribution of the script is not that of the trial, and the p-value it yields is not shown to be valid for the trial's design.\n\n\\emph{The distances.} Footnote 3 defines the distance between two polygons as the minimal distance between their points. The script computes these distances (lines 85--88) and then rebuilds the table of nodes \\texttt{Zdata} from the centroids of the polygons (lines 170--173). The loop that computes the HAC intervals takes its distances from these centroids (lines 286--289). In the Uganda data, hundreds of pairs of villages that are within the kernel cutoff by polygon distance are beyond it by centroid distance.\n\n\\emph{The degrees of freedom.} Section A.7 prescribes $2\\mu^2/\\nu$ with $\\nu$ equal to twice the trace of the squared matrix, that is, the squared trace divided by the trace of the square. The script computes \\texttt{nu} without the factor 2 and then sets \\texttt{cvs} to $2\\,\\texttt{dofs}^2/\\texttt{nu}$. The scalar normalization cancels in this ratio, and the result is twice the stated value, with or without the positive semidefinite trimming. More degrees of freedom give a smaller $t$ critical value and a narrower interval.\n\nIt is not established that this script produced Figure 11 or the p-value $0.065$. The stored results in the archive differ from the published ones (\\fid{P004-N006}).", "what_survives": "The polygon-distance rule in footnote 3 and the degrees-of-freedom formula in Section A.7 are correct; the estimand, the estimators and the theory under its stated assumptions are unaffected. Correcting the distances and the degrees of freedom changes neither the test statistic nor its reference distribution, and correcting the sampler does not change the intervals. The published p-value and the conclusion of no substantial leakage are not shown to be wrong: the computation behind the published numbers is not identified, and the direction of any change in those results is unknown. The theory in the paper assumes Bernoulli assignment. I have not examined how it extends to the restricted design of the trial."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N007", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of body text. Red boxes mark the two sentences: To construct the circle averages, we generate buffers around each of the village polygons. The distance range for estimating the AME is set to run from zero km to 15 km.", "caption": "Section 8.1, p. 763 (PDF p. 20): the buffers around the village polygons and the range of distances, in km.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N009.png"}, {"alt": "Figure 11 with its caption. Two panels, Hajek Estimator on the left and Smoothed Hajek Estimator (Triangular Kernel, bw=5km) on the right, each showing an effect curve near zero with upper and lower interval curves for cutoffs of 2, 5 and 10 km, over distances from 0 to about 14 km. Red boxes mark the label Distance (km) under each horizontal axis.", "caption": "Figure 11, p. 764 (PDF p. 21). Both panels plot the estimates against distance in km; the marks show the two axis labels.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The Uganda script in the replication archive projects the raster and the village polygons with a UTM zone of the central United States and then sends projected coordinates to a geodesic routine; its buffers of one unit are not buffers of one kilometre on the ground, and a projection for the study area with Euclidean distances, or geographic coordinates with geodesic distances, is needed.", "id": "P004-N009", "location": "Section 8.1, p. 763 (PDF p. 20), and Figure 11, p. 764 (PDF p. 21); replication archive (Supplement B), Jayachandran_etal_2017.R, lines 117 and 166–168, and SpatialEffect.R, lines 23–41 and 168–176", "pdf_page": 20, "printed_page": "763", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 8.1, p.~763: ``To construct the circle averages, we generate buffers around each of the village polygons. The distance range for estimating the AME is set to run from zero km to 15 km.'' Axes of Figure 11, p.~764: ``Distance (km)''.\n\n\\texttt{Jayachandran\\_etal\\_2017.R}, lines 117 and 166--168:\n\n\\texttt{dist.metric <- \\hl{\"Geodesic\"}}\n\n\\texttt{sr <- \"+proj=utm \\hl{+zone=15 +ellps=GRS80 +datum=NAD83} +units=km +no\\_defs\"}\n\n\\texttt{ras <- projectRaster(ras, crs = sr)}\n\n\\texttt{ras\\_Z <- spTransform(ras\\_Z, crs(ras))}", "corrected": "\\texttt{sr <- \"+proj=utm \\hl{+zone=36 +datum=WGS84} +units=km +no\\_defs\"}\n\n\\texttt{dist.metric <- \\hl{\"Euclidean\"}}\n\nTransform the raster and the polygons with the same projection, using transformations appropriate to their classes, and compute distances and buffers in the projected kilometres. \\hl{This is a local approximation to ground distances.} The alternative is to keep geographic coordinates with geodesic distances and to bring the buffer widths (in metres), the distances and the cutoffs to the same unit. The separate script/package interface errors also need repair (\\fid{P004-E029}). Any result obtained from the buffers of the supplied script should be computed again.", "evidence": "I checked the projection's returned coordinates by exact comparisons, with an analytic upper bound for distances on the ground, after bypassing the separate execution errors; I did not reproduce the estimates. The first centroid is sent to the routine as about $(8019, 19696)$, and all 121 fail its bounds of 360 and 90. The table lists the two comparisons of distances. The first uses the points at longitudes $31.2^\\circ$ and $31.21^\\circ$ E on the parallel $1.2^\\circ$ N. With a projection for UTM zone 36 the same two points are $1.113$ units apart, and their geodesic distance is $1.113$ km. The second uses a boundary vertex of the first village and the point $0.006^\\circ$ west of it on the same parallel.\n\n\\begin{audittable}{lll}{Distances on the ground and in the projected coordinates of the script.}\nPair & On the ground & In the script's projection \\\\\nTwo points $0.01^\\circ$ of longitude apart & less than $1.135$ km & more than $1.99$ units \\\\\nA point and the first village & less than $0.681$ km & more than $1$ unit (outside the buffer) \\\\\n\\end{audittable}", "problem": "The villages lie near $31^\\circ$ E and $1^\\circ$ N, in UTM zone 36. The script projects the outcome raster and the polygons to UTM zone 15, which covers the central United States, with kilometre units. So far from its own zone the projection distorts lengths, and the package draws its buffers in the projected coordinates. A buffer of width one is then not a buffer of one km on the ground. Two points of the study area that are less than $1.135$ km apart on the ground are more than $1.99$ units apart after projection, a ratio above $7/4$. A point less than $0.681$ km from the boundary of the first village lies outside that village's buffer of width one.\n\nThe script also selects geodesic distances. The package then takes the centroids of the projected polygons and passes their coordinates, in projected kilometres, to a routine that expects longitude and latitude in degrees. All 121 centroids have a first coordinate above 360, and the routine stops with the error \\texttt{longitude > 360}. This path cannot produce HAC distances. The later block of the script, which computes the intervals for the varying cutoffs, uses the original geographic coordinates and valid geodesic distances, and this defect does not reach it.\n\nNo dependence of Figure 11 on this path is established. If the published analysis used these buffers, every estimate at a nonzero distance and the cumulative statistic refer to regions other than those stated.", "what_survives": "These code defects do not refute the definitions of distance or the statistical theory. The region at distance zero is the village itself and needs no positive buffer width; I have not verified its published numerical estimate. Figure 11, the cumulative statistic, its p-value and the conclusion about leakage are not shown to depend on this path: the script as supplied does not run against the package supplied with it (\\fid{P004-E029}), and the code that produced Figure 11 is not identified. What changes is that the archive cannot confirm the distance scale of Figure 11."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N009", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text that begin: study. The distance range is set to run from zero km and 20 km. A red box marks the word and, between zero km and 20 km.", "caption": "Section 8.2, p. 765 (PDF p. 22), the first lines of text below the caption of Figure 12. The marked word should be “to”.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N011.png"}, {"alt": "Crop of Figure 13 with its caption. Two panels, titled Hajek Estimator and Smoothed Hajek Estimator (Triangular Kernel, bw=5km), plot an effect curve and interval curves against distance in km. Red boxes mark the horizontal axis of each panel, with the tick labels 0, 5 and 10 and the label Distance (km). The curves stop a little before the right edge of each panel, near 14 km.", "caption": "Figure 13, p. 765 (PDF p. 22). The marked horizontal axes carry ticks at 0, 5 and 10 km. The curves in both panels end about four units beyond the tick at 10 km, short of the 20 km stated in the text.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N011-b.png"}, {"alt": "Crop of Figure 11 with its caption. Two panels, titled Hajek Estimator and Smoothed Hajek Estimator (Triangular Kernel, bw=5km), plot an effect curve and interval curves against distance in km. Red boxes mark the horizontal axis of each panel, with the tick labels 0, 5 and 10 and the label Distance (km). The curves stop a little before the right edge of each panel, near 14 km.", "caption": "Figure 11, p. 764 (PDF p. 21). The marked horizontal axes carry ticks at 0, 5 and 10 km. The curves in both panels end about four units beyond the tick at 10 km, short of the 15 km stated in Section 8.1.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N011-c.png"}], "formal_release": null, "formal_statements": [], "headline": "In Section 8.2, “from zero km and 20 km” should read “from zero km to 20 km”, and Figures 11 and 13 display estimates only up to about 14 km, a shorter range than the 15 km and 20 km stated in the text.", "id": "P004-N011", "location": "Section 8.2, p. 765 (PDF p. 22), first line of text; Figure 13, p. 765; Figure 11, p. 764 (PDF p. 21); compare Section 8.1, p. 763 (PDF p. 20)", "pdf_page": 22, "printed_page": "765", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 8.2, p.~765: ``The distance range is set to run from zero km \\hl{and} 20 km.''\n\nSection 8.1, p.~763: ``The distance range for estimating the AME is set to run from zero km to 15 km.''\n\nFigures 11 and 13: the horizontal axes carry ticks at 0, 5 and 10 km, and the curves end at about 14 km.", "corrected": "Section 8.2, p.~765: ``The distance range is set to run from zero km \\hl{to} 20 km.''\n\nCaptions of Figures 11 and 13, added sentence: ``\\hl{Displayed estimates cover distances from 0 to about 14 km.}''", "evidence": "The evidence is a check of pp.~763--765. In each panel the curves end about four units beyond the tick at 10 km, and no label is printed at the endpoint. The Costa Rica script in the replication files sets the distances to run from 0 to 20 km in steps of 500 m, which agrees with the text of Section 8.2. The Uganda script in the same files sets \\texttt{len=30}, which does not confirm the range of 15 km stated in Section 8.1. I recomputed no estimate.", "problem": "The word ``and'' is a slip for ``to''. The two sentences give the range of distances at which the AME is estimated, 15 km in Section 8.1 and 20 km in Section 8.2. Both figures show curves that end near 14 km, and neither caption says that the display is shorter than the range given in the text.", "what_survives": "The stated ranges of estimation are not shown to be wrong, and nothing displayed in the two figures changes. The shorter display does not by itself establish an error in the statements about nearby areas and about leakage."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N011", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Three displayed lines of a proof. The first is an upper bound whose first term has the coefficient 1 over N squared p. After the words We also have, the next display repeats the expression with a third term subtracted; a red box marks the coefficient of its first term, 1 over N p, and a second red box marks the right-hand side, O(1). The text continues: since C2 implies that all the moments are bounded.", "caption": "Supplement A, p. 3, proof of Lemma A.2: the bound that ends the expansion of the diagonal terms and the display after “We also have:”. The marked coefficient reads $1/(Np)$ where the line above has $1/(N^2p)$, and the marked rate reads $O(1)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N013.png"}, {"alt": "A displayed difference of two sums over pairs i and j with j in the neighbourhood B(i;d) and j different from i, each divided by N squared. A red box marks the right-hand side, O(1). The text continues: since the size of B(i;d) is bounded by (C4a). Other covariance terms have similar forms.", "caption": "Supplement A, p. 3, end of the proof of Lemma A.2: the display after “Moreover,”. The marked rate reads $O(1)$.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N013-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T32", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T32"}], "headline": "In the proof of Lemma A.2, the first term of the display after “We also have:” should carry the factor $1/(N^2p)$, and that display and the one after “Moreover,” should conclude $O(1/N)$, the rate that the lemma states.", "id": "P004-N013", "location": "Supplement A, proof of Lemma A.2, p. 3, the displays after “We also have:” and after “Moreover,”", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T32"], "text": {"as_printed": "We also have:\n\\[\n\\frac{1}{\\hl{Np}}\\sum_{i=1}^N \\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)^2\\right]+\\frac{1}{N^2(1-p)}\\sum_{i=1}^N \\mathrm{E}\\left[\\mu_i(\\mathbf{0};d)^2\\right]-\\frac{1}{N^2}\\sum_{i=1}^N\\left(\\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)-\\mu_i(\\mathbf{0};d)\\right]\\right)^2=\\hl{O(1)},\n\\]\nsince C2 implies that all the moments are bounded. [\\ldots] Moreover,\n\\[\n\\frac{1}{N^2}\\sum_{i;j\\in\\mathcal{B}(i;d),j\\ne i}\\mathrm{E}\\left[\\mu_i(\\mathbf{1}_{\\mathbf{i}},\\mathbf{1}_{\\mathbf{j}};d)\\mu_j(\\mathbf{1}_{\\mathbf{i}},\\mathbf{1}_{\\mathbf{j}};d)\\right]-\\frac{1}{N^2}\\sum_{i;j\\in\\mathcal{B}(i;d),j\\ne i}\\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)\\right]\\mathrm{E}\\left[\\mu_j(\\mathbf{1};d)\\right]=\\hl{O(1)},\n\\]\nsince $|\\mathcal{B}(i;d)|$ is bounded by (C4a).", "corrected": "We also have:\n\\[\n\\frac{1}{\\hl{N^2p}}\\sum_{i=1}^N \\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)^2\\right]+\\frac{1}{N^2(1-p)}\\sum_{i=1}^N \\mathrm{E}\\left[\\mu_i(\\mathbf{0};d)^2\\right]-\\frac{1}{N^2}\\sum_{i=1}^N\\left(\\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)-\\mu_i(\\mathbf{0};d)\\right]\\right)^2=\\hl{O\\left(\\frac{1}{N}\\right)},\n\\]\nsince C2 implies that all the moments are bounded. [\\ldots] Moreover,\n\\[\n\\frac{1}{N^2}\\sum_{i;j\\in\\mathcal{B}(i;d),j\\ne i}\\mathrm{E}\\left[\\mu_i(\\mathbf{1}_{\\mathbf{i}},\\mathbf{1}_{\\mathbf{j}};d)\\mu_j(\\mathbf{1}_{\\mathbf{i}},\\mathbf{1}_{\\mathbf{j}};d)\\right]-\\frac{1}{N^2}\\sum_{i;j\\in\\mathcal{B}(i;d),j\\ne i}\\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)\\right]\\mathrm{E}\\left[\\mu_j(\\mathbf{1};d)\\right]=\\hl{O\\left(\\frac{1}{N}\\right)},\n\\]\nsince $|\\mathcal{B}(i;d)|$ is bounded by (C4a).", "evidence": "The evidence is a derivation under the conditions of the lemma. By C2, there exists a constant $b$ such that $|\\mu_i(\\cdot\\,;d)|\\le b$. The corrected first expression is a sum of $N$ variances divided by $N^2$. It is nonnegative, and since its last term is subtracted I have\n\\[\n0\\le\\frac{1}{N^2p}\\sum_{i=1}^N \\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)^2\\right]+\\frac{1}{N^2(1-p)}\\sum_{i=1}^N \\mathrm{E}\\left[\\mu_i(\\mathbf{0};d)^2\\right]-\\frac{1}{N^2}\\sum_{i=1}^N\\left(\\mathrm{E}\\left[\\mu_i(\\mathbf{1};d)-\\mu_i(\\mathbf{0};d)\\right]\\right)^2\\le\\frac{b^2}{Np}+\\frac{b^2}{N(1-p)}.\n\\]\nBy C4A, there exists a constant $c$ such that $|\\mathcal{B}(i;d)|\\le c$. Each sum in the second display then has at most $N(c-1)$ terms, and each term is bounded by $b^2$ in absolute value. The difference of the two sums, divided by $N^2$, is at most $2b^2(c-1)/N$ in absolute value. Both quantities are $O(1/N)$.", "problem": "Lemma A.2 asserts that $\\operatorname{Var}(\\widehat\\tau_{\\mathrm{HT}}(d))=O(1/N)$ under C1--C4. The proof expands the diagonal part of the variance, and both the exact expression and the bound printed just above ``We also have:'' carry $1/(N^2p)$ on the treated sum. The next display repeats the expression with $1/(Np)$. Neither printed statement is false. With the factor $1/(Np)$ the first expression is of order one and no smaller: if $\\mu_i(\\mathbf{1};d)=\\mu_i(\\mathbf{0};d)=1$ for every $i$, it equals $1/p+1/(N(1-p))$. But it is then not the diagonal part of the variance, and a bound of order one on each part does not imply the rate that the lemma states. Once the factor is restored, both displays are $O(1/N)$. Replacing only the first $O(1)$ by $O(1/N)$ would make that display false.", "what_survives": "Lemma A.2, its variance bound and its conclusion $O(1/N)$ under C1--C4 are not affected, and neither is any later use of the lemma. The exact expansion printed above the first display is correct. Only the two displays that justify the rate change."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N013", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The last display of a proof, a double sum that is shown to be at least zero, followed by the sentence: by C5. Hence the term combining expressions in (2) and (5) is non-positive in the limit under C5, which proves the lemma. A red box marks (2).", "caption": "Supplement A, p. 5, last lines of the proof of Lemma A.4. The marked reference reads (2); the display above it combines lines (3) and (5).", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N014.png"}, {"alt": "A paragraph of text. A red box marks the words Proposition A.4 in the sentence: The Hajek estimator's asymptotic distribution can be then obtained via the linear expansion in Proposition A.4 and a similar application of the CLT result.", "caption": "Supplement A, p. 6, Section A.3, first paragraph. The marked reference reads “Proposition A.4”; the linear expansion is Lemma A.3.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N014-b.png"}, {"alt": "A paragraph of text. Red boxes mark the words Lemma, at the end of one line, and A.3, at the start of the next, in the sentence: we show that the HAC variance estimator is consistent for the variance bound defined in Lemma A.3 and lines (6)-(7).", "caption": "Supplement A, p. 6, Section A.4.1, first paragraph. The marked reference reads “Lemma A.3”; lines (6)–(7) belong to Lemma A.4.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N014-c.png"}, {"alt": "Three lines of text. A red box marks the words Proposition A.4 in the sentence: The characterization of the variance is similar to that in Proposition A.4.", "caption": "Supplement A, p. 12, first line of the proof of Proposition 3. The marked reference reads “Proposition A.4”.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N014-d.png"}, {"alt": "Text and one display. A red box marks (6) in the sentence: We can similarly define a HAC type estimator as in (6). The display defines a HAC estimator as a sandwich of the inverse of X prime X around a double sum of X_i X_j prime times two residuals and a neighbourhood indicator. A second red box marks (6) in the sentence: The formula is similar to (6), except now that the residuals are weighted with the inverse propensity scores.", "caption": "Supplement A, p. 15, the observational HAC estimator. Both marked references read (6); the estimator displayed between them has the form of equation (19) of the article.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N014-e.png"}, {"alt": "Five lines of text. A red box marks (5) in the sentence: Recall the regression interpretation of the Hajek estimator discussed in (5) and the HAC variance estimator in (19).", "caption": "Supplement A, p. 17, Section A.7, second paragraph. The marked reference reads (5); the regression form of the Hajek estimator is equation (18) of the article.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N014-f.png"}], "formal_release": null, "formal_statements": [], "headline": "Seven cross-references in Supplement A name the wrong item; each should name the lemma or equation listed in the correction.", "id": "P004-N014", "location": "Supplement A: end of the proof of Lemma A.4, p. 5; first paragraphs of Sections A.3 and A.4.1, p. 6; proof of Proposition 3, p. 12; the text around the observational HAC estimator, p. 15; Section A.7, p. 17.", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item p.~5: ``Hence the term combining expressions in \\hl{(2)} and (5) is non-positive in the limit under C5''\n\\item p.~6: ``via the linear expansion in \\hl{Proposition A.4}''\n\\item p.~6: ``the variance bound defined in \\hl{Lemma A.3} and lines (6)-(7)''\n\\item p.~12: ``The characterization of the variance is similar to that in \\hl{Proposition A.4}.''\n\\item p.~15: ``We can similarly define a HAC type estimator as in \\hl{(6)}.'' and ``The formula is similar to \\hl{(6)}''\n\\item p.~17: ``Recall the regression interpretation of the Hajek estimator discussed in \\hl{(5)}''\n\\end{itemize}", "corrected": "\\begin{audittable}{lll}{The seven references of Supplement A, as printed and corrected.}\nPlace & As printed & Corrected \\\\\np.~5, end of the proof of Lemma A.4 & (2) and (5) & \\hl{(3)} and (5) \\\\\np.~6, Section A.3 & the linear expansion in Proposition A.4 & the linear expansion in \\hl{Lemma A.3} \\\\\np.~6, Section A.4.1 & Lemma A.3 and lines (6)-(7) & \\hl{Lemma A.4} and lines (6)-(7) \\\\\np.~12, proof of Proposition 3 & similar to that in Proposition A.4 & similar to that in \\hl{Lemma A.4} \\\\\np.~15, first sentence & as in (6) & as in \\hl{(19) of the article} \\\\\np.~15, after the display & similar to (6) & similar to \\hl{(19) of the article} \\\\\np.~17, Section A.7 & discussed in (5) & discussed in \\hl{(18) of the article} \\\\\n\\end{audittable}", "evidence": "The evidence is a check of the pages. I compared each reference with the item that it names and with the item that matches the description in the sentence, on pp.~4--6, 12, 15 and 17 of Supplement A and on pp.~754--755 of the article.", "problem": "On p.~5 the proof combines lines (3) and (5) a few lines earlier; line (2) is the positive sum of second moments on the diagonal. The linear expansion of the Hajek estimator is Lemma A.3, its variance and the bound in lines (6)--(7) are Lemma A.4, and the supplement has no Proposition A.4. The spatial HAC estimator is equation (19) of the article. In the supplement, (6) is the variance bound of Lemma A.4. The regression form of the Hajek estimator is equation (18) of the article; equation (5) there defines the marginalised circle average. Each intended item can be identified from its content.", "what_survives": "Replacing these labels leaves the arguments and formulas unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N014", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the variance bound of Lemma A.4, lines (6) and (7), with the sentence that introduces it and the first two lines of the proof. Red boxes mark the fragment '; d)' that follows mu_i(0; d) in the control term of line (6), and the fragment '; d)' that follows mu_j(a_i, b_j; d) in the second factor of line (7).", "caption": "Supplement A, p. 5, lines (6)–(7) of Lemma A.4. The two marked fragments “$;d)$” repeat the distance argument after the outcome argument is already closed.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N015.png"}, {"alt": "Crop of the line 'Proof. We first prove the case for the HT estimator. Define U_i as' and the display that defines U_i. A red box marks the first 'mu_i(' in the numerator (1 - Z_i) mu_i( mu_i(0; d) of the second fraction.", "caption": "Supplement A, p. 6, the definition of $U_i$ in the proof of Propositions 2 and 3. The marked “$\\mu_i($” is the stray fragment in the control term.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N015-b.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T33", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T33"}], "headline": "Lines (6) and (7) each carry a stray “$;d)$”, and the control term of $U_i$ carries a stray “$\\mu_i($”.", "id": "P004-N015", "location": "Supplement A, Lemma A.4, p. 5 (supplement PDF p. 5), lines (6)–(7); A.3.1, p. 6 (supplement PDF p. 6), the definition of $U_i$", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T33"], "text": {"as_printed": "P.~5: ``Under conditions C1-C5, we have the following variance bound for $\\operatorname{Var}\\big(\\widehat\\tau^{\\mathrm L}_{\\mathrm{HA}}(d)\\big)$:\n\\[\n\\begin{aligned}\n\\tilde{\\mathrm V}_{\\mathrm{HA}}={}&\\frac{1}{N^2p}\\sum_{i=1}^N\\mathrm E\\Big[\\big(\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d)\\big)^2\\Big]+\\frac{1}{N^2(1-p)}\\sum_{i=1}^N\\mathrm E\\Big[\\big(\\mu_i(\\mathbf 0;d)\\hl{;d)}-\\bar\\mu^0(d)\\big)^2\\Big]\\quad\\text{(6)}\\\\\n&+\\frac{1}{N^2}\\sum_{i;j\\in\\mathcal B(i;d),j\\neq i}\\ \\sum_{a,b=0}^{1}(-1)^{a+b}\\mathrm E\\big[(\\mu_i(\\mathbf{a_i},\\mathbf{b_j};d)-\\bar\\mu^a(d))\\big(\\mu_j(\\mathbf{a_i},\\mathbf{b_j};d)\\hl{;d)}-\\bar\\mu^b(d)\\big)\\big].\\quad\\text{(7)''}\n\\end{aligned}\n\\]\nP.~6: the control term inside the parentheses of $U_i$ is printed as\n\\[\n-\\frac{(1-Z_i)\\hl{\\mu_i(}\\mu_i(\\mathbf 0;d)}{N(1-p)}.\n\\]", "corrected": "P.~5: ``Under conditions C1-C5, we have the following variance bound for $\\operatorname{Var}\\big(\\widehat\\tau^{\\mathrm L}_{\\mathrm{HA}}(d)\\big)$:\n\\[\n\\begin{aligned}\n\\tilde{\\mathrm V}_{\\mathrm{HA}}={}&\\frac{1}{N^2p}\\sum_{i=1}^N\\mathrm E\\Big[\\big(\\mu_i(\\mathbf 1;d)-\\bar\\mu^1(d)\\big)^2\\Big]+\\frac{1}{N^2(1-p)}\\sum_{i=1}^N\\mathrm E\\Big[\\big(\\hl{\\mu_i(\\mathbf 0;d)}-\\bar\\mu^0(d)\\big)^2\\Big]\\quad\\text{(6)}\\\\\n&+\\frac{1}{N^2}\\sum_{i;j\\in\\mathcal B(i;d),j\\neq i}\\ \\sum_{a,b=0}^{1}(-1)^{a+b}\\mathrm E\\big[(\\mu_i(\\mathbf{a_i},\\mathbf{b_j};d)-\\bar\\mu^a(d))\\big(\\hl{\\mu_j(\\mathbf{a_i},\\mathbf{b_j};d)}-\\bar\\mu^b(d)\\big)\\big].\\quad\\text{(7)''}\n\\end{aligned}\n\\]\nP.~6:\n\\[\nU_i=\\frac{1}{\\sqrt{\\operatorname{Var}(\\widehat\\tau_{\\mathrm{HT}}(d))}}\\left(\\frac{Z_i\\mu_i(\\mathbf 1;d)}{Np}-\\frac{(1-Z_i)\\hl{\\mu_i(\\mathbf 0;d)}}{N(1-p)}-\\frac{\\mathrm E[\\mu_i(\\mathbf 1;d)]-\\mathrm E[\\mu_i(\\mathbf 0;d)]}{N}\\right).\n\\]", "evidence": "The evidence is a check of pp.~5--6 of Supplement A against lines (2) and (4) of Lemma A.4 and equation (13) of the article. The identity stated directly below the definition of $U_i$ fixes the summand that is meant.", "problem": "The three fragments are typesetting slips. In (6) and (7) the argument of the outcome is closed before the second ``$;d)$''. Line (2) of the same lemma has the control term without the fragment, and line (4) has the second-node term with one distance argument. In $U_i$ the function $\\mu_i$ does not take another $\\mu_i$ as its argument: the Horvitz--Thompson estimator of equation (13), p.~754, has one control outcome in that term.", "what_survives": "The variance bound of Lemma A.4 and the centered sum used for asymptotic normality are unaffected. Only the three displays change."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N015", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading 'A.6. Theoretical Results on Observational Studies', its introductory paragraph, and the line 'C 1. The following properties hold for all sample size N and for all i in S_N:' followed by item (i). A red box marks the label 'C 1.'", "caption": "Supplement A, p. 13, the start of A.6. The marked label “C 1.” names a new block of conditions on the logistic assignment model; C1 of the article is the Bernoulli design.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N016.png"}, {"alt": "Crop of the line 'Under C2, C3, C4, C6, and C1,' and of equation (36), which states that the linearized and the fitted IPW estimators differ by o_p(N to the power minus one half). Red boxes mark 'C2' and 'C1'.", "caption": "Supplement A, p. 15, Proposition 4 of A.6, the hypothesis of (36). The marked “C2” is the article’s bounded-outcomes condition and the marked “C1” is the logistic block of p. 13.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N016-b.png"}, {"alt": "Crop of the line 'In addition under C2,' and the first line of the displayed upper bound on the variance of the linearized IPW estimator. A red box marks 'C2'.", "caption": "Supplement A, p. 16, the continuation of Proposition 4. The marked “C2” is the homophily condition of p. 14, not the condition of the same name on p. 15.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N016-c.png"}, {"alt": "Crop of the sentence 'The following assumption is similar as C5.', the line 'C 6. Given a kernel function K and a sequence of bandwidth b_N,' and equation (122), a liminf condition on a kernel-weighted sum of products of centered node effects. Red boxes mark 'C5' and the label 'C 6.'", "caption": "Supplement A, p. 34, Appendix C. The marked “C5” refers to the homophily condition of the article; the marked “C 6.” is a new condition local to Appendix C.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N016-d.png"}], "formal_release": null, "formal_statements": [], "headline": "Supplement A reuses the labels C1–C6 for different assumptions: in A.6, “C 1” and “C 2” are a logistic-model block and a homophily condition, and in Appendix C, “C 3”–“C 6” are four new conditions.", "id": "P004-N016", "location": "Supplement A, A.6, pp. 13–16, and Appendix C, pp. 26–35 (the same supplement PDF pages)", "pdf_page": 13, "printed_page": "13–16", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "A.6, p.~13: ``\\hl{C 1.} The following properties hold for all sample size $N$ and for all $i\\in\\mathcal S_N$: \\ldots''; p.~14: ``\\hl{C 2.} For all sample size $N$, \\ldots''; Proposition 4, pp.~15--16: ``Under \\hl{C2}, C3, C4, C6, and \\hl{C1}, \\ldots'' and ``In addition under \\hl{C2}, \\ldots''.\n\nAppendix C, p.~26: ``We shall inherit C1 and C2. We relax C3 with the following weak-dependence assumption. \\hl{C 3.} \\ldots''; p.~27: ``\\hl{C 4.} \\ldots''; p.~32: ``\\hl{C 5.} The kernel function \\ldots''; p.~34: ``The following assumption is similar as \\hl{C5}. \\hl{C 6.} \\ldots'', with footnote 20: ``Note if $b_N$ is fixed and $K$ is the uniform kernel, this reduces to \\hl{C5}.''; Proposition 6, p.~35: ``Under C1, C2, C3, C4, C5 and C6''.", "corrected": "Give the two conditions of A.6 the labels \\hl{O1} (logistic model) and \\hl{O2} (homophily), and state Proposition 4 as ``Under C2, C3, C4, C6, and \\hl{O1}, \\ldots\\ In addition under \\hl{O2}, \\ldots''. Give the conditions of Appendix C the labels \\hl{W3}--\\hl{W6}, and state its results under ``C1, C2, \\hl{W3, W4, W5 and W6}''. On p.~34, write ``similar to C5 \\hl{of the article}'' and ``this reduces to C5 \\hl{of the article}''.", "intended_meaning": "In A.6, C6 is the article's assignment condition and ``C 1'' is the new block of conditions on the logistic model. In Proposition 4 the first ``C2'' is the article's bounded outcomes: the proof on p.~16 uses it, with bounded confounders, to control remainders. The ``C2'' of ``In addition under C2'' and of item (ii) is the homophily condition of p.~14. Appendix C states that it keeps C1 and C2 of the article and replaces C3. Its ``C 3''--``C 6'' are local conditions (near-epoch dependence, a quadratic bound on neighborhood counts, kernel conditions and a homophily limit), and Proposition 6 uses that local set. The two comparisons on p.~34, ``similar as C5'' and ``this reduces to C5'', refer to the homophily condition C5 of the article, not to the kernel condition labeled ``C 5'' on p.~32. With these readings the hypotheses of the supplement's propositions are recoverable, and no result depends on the labels."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N016", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of displays (127), (128) and (129) and the sentence that follows them. Red boxes mark the factor N squared over N 1 squared minus 1 over p squared, set in parentheses, at the start of (128) and at the start of (129).", "caption": "Supplement A, p. 36: equations (127)–(129). The marked factor is printed without absolute value bars, while (127) and the second factor of (128) are inside them.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N018.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T34", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T34"}], "headline": "The leading factor $N^2/N_1^2-1/p^2$ of the bounds (128) and (129) should appear in absolute value.", "id": "P004-N018", "location": "Supplement A, Appendix C, proof of Proposition 6(i), equations (128)–(129), p. 36 (PDF p. 36 of the supplement)", "pdf_page": 36, "printed_page": "36", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P004-T34"], "text": {"as_printed": "\\[\n\\begin{aligned}\n&\\Big|N\\times(\\frac1{N_1^2}\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})-\\frac1{N^2p^2}\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N}))\\Big|&&\\text{(127)}\\\\\n&\\le\\hl{\\Big(\\frac{N^2}{N_1^2}-\\frac1{p^2}\\Big)}\\Big|\\frac1N\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})\\Big|&&\\text{(128)}\\\\\n&\\lesssim\\hl{\\Big(\\frac{N^2}{N_1^2}-\\frac1{p^2}\\Big)}\\big(b_N^2\\times K_{max}\\big)=O_p(N^{-\\frac12}b_N^2)=o_p(1)&&\\text{(129)}\n\\end{aligned}\n\\]", "corrected": "\\[\n\\begin{aligned}\n&\\Big|N\\times(\\frac1{N_1^2}\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})-\\frac1{N^2p^2}\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N}))\\Big|&&\\text{(127)}\\\\\n&\\le\\hl{\\Big|\\frac{N^2}{N_1^2}-\\frac1{p^2}\\Big|}\\,\\Big|\\frac1N\\sum_{i=1}^N\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(\\frac{d_{ij}}{b_N})\\Big|&&\\text{(128)}\\\\\n&\\lesssim\\hl{\\Big|\\frac{N^2}{N_1^2}-\\frac1{p^2}\\Big|}\\big(b_N^2\\times K_{max}\\big)=O_p(N^{-\\frac12}b_N^2)=o_p(1)&&\\text{(129)}\n\\end{aligned}\n\\]", "evidence": "The evidence is an exact counterexample and a derivation of the corrected bound. Let $N=8$, $p=1/2$ and $N_1=5$, with residuals $(1,-1,0,0,0)$ for the treated nodes and zero for the control nodes, and with $K(d_{ij}/b_N)=1$ for all pairs. The residuals have mean zero in each arm. Then $A=-36/25$ and $B=-1/4$. The quantity (127) equals $9/25$, and the right-hand side of (128), as printed, equals $-9/25$. For the corrected bound, note that $|A|=O_p(N^{-1/2})$ by (124). The residuals are bounded by C2, and by C4 and C5 each node has at most a constant times $b_N^2$ neighbours with nonzero kernel weight, each weight being at most $K_{\\max}$. I obtain $|A|\\,|B|\\lesssim|A|\\,b_N^2K_{\\max}=O_p(N^{-1/2}b_N^2)$, which is $o_p(1)$ because $b_N=o(N^{1/6})$.", "problem": "Let $A=N^2/N_1^2-1/p^2$ and $B=N^{-1}\\sum_{i}\\sum_{j\\ne i}Z_iZ_j\\hat e_i(d)\\hat e_j(d)K(d_{ij}/b_N)$. The quantity (127) is an absolute value and equals $|A|\\,|B|$, while the printed bound (128) is $A|B|$. The factor $A$ is negative whenever $N_1/N>p$, which happens on ordinary assignments, and the right-hand side as printed is then negative when $B\\ne0$. The same factor appears without bars in (129).", "what_survives": "The rate $O_p(N^{-1/2}b_N^2)$ and Proposition 6(i) are unchanged. Only the two displayed bounds change."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N018", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the notation table with the columns Notations, Definitions and First Appear in, from the row for S to the row for tau i of d and eta. Red boxes mark the symbol Y x of z i and eta in the row Marginalized potential outcome at point x, and the entry Page 6 in the row Marginalized circle averages.", "caption": "Supplement A, p. 39: the first rows of the notation table of Appendix D. The marked symbol lacks the index $i$, and the marked entry should be Page 7.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N019.png"}], "formal_release": null, "formal_statements": [], "headline": "In the notation table, the marginalized potential outcome is $Y_{ix}(z_i;\\eta)$, with the index $i$ of the node, and the marginalized circle averages first appear on page 7 of the article and not on page 6.", "id": "P004-N019", "location": "Supplement A, Appendix D, notation table, p. 39 (PDF p. 39 of the supplement); compare equations (4) and (5) of the article, pp. 749–750 (PDF pp. 6–7)", "pdf_page": 39, "printed_page": "39", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lll}{Three rows of the notation table, as printed.}\nNotations & Definitions & First Appear in \\\\\n$\\hl{Y_x}(z_i;\\eta)$ & Marginalized potential outcome at point x & Page 6 \\\\\n$\\mu_i(z_i;d,\\eta)$ & Marginalized circle averages & Page \\hl{6} \\\\\n$\\tau_{ix}(\\eta)$ & Individual marginalized effect & Page 7 \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lll}{The same rows, corrected.}\nNotations & Definitions & First Appear in \\\\\n$\\hl{Y_{ix}}(z_i;\\eta)$ & Marginalized potential outcome at point x & Page 6 \\\\\n$\\mu_i(z_i;d,\\eta)$ & Marginalized circle averages & Page \\hl{7} \\\\\n$\\tau_{ix}(\\eta)$ & Individual marginalized effect & Page 7 \\\\\n\\end{audittable}", "evidence": "The evidence is a check of p.~39 of the supplement against pp.~749--750 of the article.", "problem": "Equation (4) of the article defines the marginalized potential outcome $Y_{ix}(z_i;\\eta)$ at the point $x$ for the node $i$ whose assignment is held fixed. The table drops the index $i$ from the symbol and keeps it in $z_i$. The table refers to pages of the article's PDF: equation (4) is on PDF p.~6 (p.~749). The marginalized circle average $\\mu_i(z_i;d,\\eta)$ first appears in equation (5), on PDF p.~7 (p.~750). The row for $\\tau_{ix}(\\eta)$ is correct as printed.", "what_survives": "The definitions (4)--(8) of the article are unaffected, and nothing depends on the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N019", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 21 to 34 of Conley.cpp with line numbers. In the branch metric == 1, line 24 reads dist(j) = sum(pow(point1 - point2, 2)). In the branch metric == 2, lines 26 to 28 call distGeo from the package geosphere, and a red box marks line 29, dist(j) = geo_distance[0]. In line 33 a red box marks the test sqrt(dist) < cutoff, whose result is assigned to dist_u.", "caption": "Source code of the package in the GitHub repository, not a page of the article: SpatialEffect/src/Conley.cpp, lines 21–34, typeset from the file with line numbers added. Line 24 stores a squared Euclidean distance. The marked line 29 stores the geodesic distance itself, and the marked test on line 33 takes the square root in both cases.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N020.png"}], "formal_release": null, "formal_statements": [], "headline": "In the older source of the package, the spatial HAC routine compares the square root of a geodesic distance with the cutoff, so that a pair of nodes is weighted when its distance is below the square of the cutoff; the geodesic branch should store the squared distance.", "id": "P004-N020", "location": "R package SpatialEffect in the GitHub repository named on p. 766 (commit 7c873da): SpatialEffect/src/Conley.cpp at the top level of the repository, lines 23–33. The code is not printed in the article; the estimator that it implements is equation (19), p. 755 (PDF p. 12).", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{SpatialEffect/src/Conley.cpp}, lines 24, 29 and 33:\n\n\\texttt{dist(j) = sum(pow(point1 - point2, 2));}\n\n\\texttt{dist(j) = \\hl{geo\\_distance[0]};}\n\n\\texttt{arma::uvec dist\\_u = (\\hl{sqrt(dist)} < cutoff);}\n\nLine 24 belongs to the branch \\texttt{metric == 1} (Euclidean) and line 29 to the branch \\texttt{metric == 2} (geodesic).", "corrected": "Line 29:\n\n\\texttt{dist(j) = \\hl{pow(geo\\_distance[0], 2)};}\n\nBoth branches then store squared distances, and line 33 compares $d_{ij}$ with the cutoff under either metric. The kernel of footnote 12 includes the endpoint $d_{ij}=\\tilde d$; to agree with it there as well, line 33 would read \\texttt{sqrt(dist) <= cutoff}. That is a separate choice.", "evidence": "The evidence is an exact counterexample. Let four nodes lie on the equator at arc offsets $0$, $2500$, $50{,}000$ and $52{,}500$ metres, and let $c=100$, $Z=(0,0,1,1)$ and $Y=(-1,1,0,2)$. The regression of $Y$ on an intercept and $Z$ has coefficients $(0,1)$ and residuals $e=(-1,1,-1,1)$, and\n\\[\n\\mathbf{X}'\\mathbf{X}=\\begin{pmatrix}4&2\\\\2&2\\end{pmatrix},\\qquad (\\mathbf{X}'\\mathbf{X})^{-1}=\\begin{pmatrix}1/2&-1/2\\\\-1/2&1\\end{pmatrix}.\n\\]\nNo two nodes are within 100 metres, and the kernel matrix of equation (19) is the identity. Since every $e_i^2$ equals 1, the double sum in (19) equals $\\mathbf{X}'\\mathbf{X}$, and I obtain the estimate $(\\mathbf{X}'\\mathbf{X})^{-1}$, in which the treatment coefficient has variance $1$. The code also weights the pairs $(1,2)$ and $(3,4)$, since $\\sqrt{2500}=50<100$. In each pair the rows of $\\mathbf{X}$ agree and the residuals sum to zero, so the double sum vanishes and the variance is $0$. When I compile the unchanged file and run it on these coordinates, it returns the zero matrix; with the corrected line it returns the variance $1$.\n\\begin{audittable}{lll}{The four-node example at a cutoff of 100 metres.}\nQuantity & Equation (19) & Code as written \\\\\nPairs with weight one, off the diagonal & none & $(1,2)$ and $(3,4)$ \\\\\nDistances given weight one & $d_{ij}\\le 100$ m & $d_{ij}<10{,}000$ m \\\\\nVariance of the treatment coefficient & $1$ & $0$ \\\\\n\\end{audittable}", "problem": "Equation (19) weights the pair $(i,j)$ by $K(d_{ij}/\\tilde d)$, and for the uniform kernel of footnote 12, $K(x)=\\mathbf{1}\\{x\\in[0,1]\\}$, a pair counts when $d_{ij}\\le\\tilde d$. In the Euclidean branch the routine stores $d_{ij}^2$, and the square root on line 33 recovers $d_{ij}$. In the geodesic branch it stores $d_{ij}$ itself, in metres. The test on line 33 is then $\\sqrt{d_{ij}}<c$ for the cutoff $c$, which is $d_{ij}<c^2$. With a cutoff of 100 metres, pairs less than 10,000 metres apart receive weight one. For a cutoff below 1 the radius is too small.", "what_survives": "Equation (19) and footnote 12 are correct as printed, and the Euclidean branch of the routine has the right radius. The copy of the package inside the replication archive has a different routine, which divides a matrix of distances by the cutoff and returns the kernel and the variance of equation (19) on the same example. The simulation scripts compile those files, and the application scripts use the newer interface. No published number is shown to depend on the older routine; I ran no simulation or application again. Standard errors and intervals computed with the older source, geodesic distances and a positive cutoff should be recomputed, and the direction of the change is not determined in general."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N020", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 33 to 39 and 50 to 54 of Conley.cpp with line numbers. In line 35, meat_EHW = meat_EHW + res(i) * X_mat.row(i).t() * X_mat.row(i), a red box marks res(i). Line 39 forms VCE_EHW as bread.t() * meat_EHW * bread. Lines 50 to 54 return VCE_EHW if cutoff == 0 and VCE otherwise.", "caption": "Source code of the package in the GitHub repository, not a page of the article: SpatialEffect/src/Conley.cpp, lines 33–39 and 50–54, typeset from the file with line numbers added. The marked factor on line 35 is the residual, not its square. Lines 50–51 return the matrix built from it when the cutoff is zero.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N021.png"}], "formal_release": null, "formal_statements": [], "headline": "In the older source of the package, the heteroskedasticity-robust covariance returned at cutoff zero is built from residuals where squared residuals are needed, and it can be zero or negative; the residual on line 35 should be squared.", "id": "P004-N021", "location": "R package SpatialEffect in the GitHub repository named on p. 766 (commit 7c873da): SpatialEffect/src/Conley.cpp at the top level of the repository, line 35, with lines 39 and 50–51. The code is not printed in the article; the corresponding formula is equation (19), p. 755 (PDF p. 12).", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P008/", "where": null, "work_id": "P008"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P004"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{SpatialEffect/src/Conley.cpp}, line 35:\n\n\\texttt{meat\\_EHW = meat\\_EHW + \\hl{res(i)} * X\\_mat.row(i).t() * X\\_mat.row(i);}", "corrected": "\\texttt{meat\\_EHW = meat\\_EHW + \\hl{pow(res(i), 2)} * X\\_mat.row(i).t() * X\\_mat.row(i);}\n\nWith this line the routine returns $(\\mathbf{X}'\\mathbf{X})^{-1}\\bigl(\\sum_i e_i^2x_ix_i'\\bigr)(\\mathbf{X}'\\mathbf{X})^{-1}$ at cutoff zero.", "evidence": "The evidence is a set of exact counterexamples. In the first, $N=8$, $z=(0,0,0,0,1,1,1,1)$ and $y=(2,4,2,4,4,6,4,6)$. The coefficients are $(3,2)$ and the residuals alternate between $-1$ and $1$. Then $M_{\\mathrm{old}}=0$. Since every squared residual equals 1, I have $M=\\mathbf{X}'\\mathbf{X}$, and the correct covariance is $(\\mathbf{X}'\\mathbf{X})^{-1}$, in which the slope coefficient has variance $1/2$. In the second, the rows of $\\mathbf{X}$ are $(1,0)$, $(1,1)$, $(1,2)$ and $y=(2,7,6)$. The residuals are $(-1,2,-1)$, and I obtain\n\\[\nM_{\\mathrm{old}}=\\begin{pmatrix}0&0\\\\0&-2\\end{pmatrix},\\qquad (\\mathbf{X}'\\mathbf{X})^{-1}M_{\\mathrm{old}}(\\mathbf{X}'\\mathbf{X})^{-1}=\\begin{pmatrix}-1/2&1/2\\\\1/2&-1/2\\end{pmatrix},\n\\]\nwhere the correct covariance has diagonal $(7/6,\\,1/2)$. In the third, eight units have the fixed outcomes $Y_i(0)=(-1,1,-1,1,-1,1,-1,1)$ and $Y_i(1)=Y_i(0)+2$, and four are treated at random. Over the 70 assignments the difference in means has variance $4/7$, and the returned variance is $0$ on every assignment. A negative value, $-1/2$ in place of $1/2$, also arises with a design matrix prepared by the package's own function \\texttt{RegRep}. When compiled, the unchanged file returns the faulty matrices of these examples, and with the corrected line it returns the correct ones.\n\\begin{audittable}{lll}{Variance of the slope coefficient in the three examples.}\nExample & Returned at cutoff zero & Reference value \\\\\nEight units, binary treatment & $0$ & $1/2$, with squared residuals \\\\\nThree units, regressor $0,1,2$ & $-1/2$ & $1/2$, with squared residuals \\\\\nFour of eight units treated at random & $0$ on each of 70 assignments & $4/7$, variance of the difference in means \\\\\n\\end{audittable}", "problem": "Let $x_i'$ denote row $i$ of the design matrix $\\mathbf{X}$ and $e_i$ the regression residual. The loop accumulates $M_{\\mathrm{old}}=\\sum_i e_i x_i x_i'$, and at cutoff zero the routine returns $(\\mathbf{X}'\\mathbf{X})^{-1}M_{\\mathrm{old}}(\\mathbf{X}'\\mathbf{X})^{-1}$. The Eicker--Huber--White covariance uses $M=\\sum_i e_i^2 x_i x_i'$. The matrix $M_{\\mathrm{old}}$ changes sign when every outcome is multiplied by $-1$, and it need not be positive semidefinite. For a regression on an intercept and one binary treatment $z_i$, the normal equations and $z_i^2=z_i$ make every entry of $M_{\\mathrm{old}}$ zero, and the returned variance is zero whatever the data. The public function \\texttt{SpatialEffect} of this source has the defaults \\texttt{conley.se = 1} and \\texttt{cutoff = 0}, and it takes the square root of the diagonal, which is undefined when an entry is negative.", "what_survives": "Equation (19) has the product $\\hat e_i\\hat e_j$ and hence squared residuals on the diagonal; it is correct as printed. For a nonzero cutoff the routine returns its other matrix, which does not use \\texttt{meat\\_EHW}. In the copy of the package inside the replication archive the corresponding line squares the residual. The main simulation scripts use a different routine, and the application scripts set positive cutoffs through a different interface. No published number is shown to depend on this branch; I did not run the archive again in full. Standard errors and intervals computed with the older source at cutoff zero, the default of its public function, change."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N021", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Top of a plot titled Hajek Estimator with Its CIs. The legend has three entries: a solid line labelled AME estimates, a black dotted line labelled Spatial HAC 95 per cent CI, and a blue dotted line labelled Quantiles under sharp null. A red box marks the level, 95 per cent.", "caption": "Figure in the replication archive, not printed in the article: application/graphs/J2017_AME_polygon_unsmoothed_reproj_90CI.pdf, title and legend. The marked level is 95%; the file name says 90.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N022.png"}, {"alt": "Top of a plot titled Kernel Regression Estimator with Its CIs. The legend has two entries: a solid line labelled AME estimates and a black dotted line labelled Spatial HAC 95 per cent CI. A red box marks the level, 95 per cent.", "caption": "The second figure, application/graphs/J2017_AME_polygon_smoothed_reproj_90CI.pdf, title and legend, with the same marked level.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N022-b.png"}, {"alt": "Listing of source code with line numbers. A red box marks line 114, alpha <- 0.1. Line 198 opens the file J2017_AME_polygon_unsmoothed_reproj_90CI.pdf. Lines 210 and 211 draw columns 1 and 2 of Conley.CI as black dotted lines. In line 216 a red box marks the level, 95 per cent, in the legend text Spatial HAC 95 per cent CI. Line 219 opens the file J2017_AME_polygon_smoothed_reproj_90CI.pdf. Lines 231 and 232 draw columns 1 and 2 of smoothed.Conley.CI. In line 235 a red box marks the level, 95 per cent, in the same legend text.", "caption": "The plotting code: application/scripts/Jayachandran_etal_2017.R, lines 114, 198, 210–216, 219 and 231–235, typeset from the file with line numbers added. Line 114 sets alpha to 0.1. The dotted lines drawn on lines 210–211 and 231–232 are the two ends of the interval, and the marked legends call it a 95% interval.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N022-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The plotting code for two figures of the Uganda application in the replication archive labels nominal two-sided 90% intervals “Spatial HAC 95% CI”; the two legend strings should read 90%.", "id": "P004-N022", "location": "Replication archive (Supplement B): application/scripts/Jayachandran_etal_2017.R, lines 216 and 235, and the legends of application/graphs/J2017_AME_polygon_unsmoothed_reproj_90CI.pdf and J2017_AME_polygon_smoothed_reproj_90CI.pdf. These two figures are not printed in the article; the published counterpart is Figure 11, p. 764 (PDF p. 21).", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Legend of both figures:\n\nSpatial HAC \\hl{95\\%} CI\n\n\\texttt{Jayachandran\\_etal\\_2017.R}, lines 216 and 235:\n\n\\texttt{legend = c(\"AME estimates\", \"Spatial HAC \\hl{95\\%} CI\", \"Quantiles under sharp null\"), cex=2, bty = \"n\")}\n\n\\texttt{legend = c(\"AME estimates\", \"Spatial HAC \\hl{95\\%} CI\"), cex=2, bty = \"n\")}", "corrected": "On lines 216 and 235:\n\n\\texttt{\"Spatial HAC \\hl{90\\%} CI\"}\n\nIf the two figure files are replaced, they should be drawn again from a documented 90\\% calculation and not only relabelled, since their curves differ from the stored results.", "evidence": "The evidence is a derivation from the code and a recalculation from the results stored in the archive. In both stored interval matrices, each of the 31 rows equals the stored estimate plus or minus \\texttt{qnorm(0.95)} times the stored standard error, with no difference in R's arithmetic. The 95\\% formula differs from them by up to $0.0025$ for the unsmoothed intervals and $0.0043$ for the smoothed ones. The check has a limit. The curves saved in the two figure files do not match the stored matrices: the point estimates agree within the rounding of the files, about $10^{-6}$, but the interval ends differ by up to $0.0034$ and $0.0133$. The level of the saved curves is therefore not determined, and the finding is the label in the code.", "problem": "The script sets \\texttt{alpha <- 0.1} on line 114 and passes it to \\texttt{SpatialEffect}, which forms each interval as the estimate plus or minus \\texttt{qnorm(1-alpha/2)} standard errors. The critical value is $\\Phi^{-1}(0.95)=1.645$, and under the normal reference the two ends enclose probability $2(0.95)-1=0.90$. A two-sided 95\\% interval would use $\\Phi^{-1}(0.975)=1.960$. Lines 210--211 and 231--232 plot both ends, and lines 216 and 235 label them ``Spatial HAC 95\\% CI''. Each end taken alone is a one-sided 95\\% bound, which is the use that the article describes on p.~763, but the pair is a 90\\% interval.", "what_survives": "Figure 11 is a separate graphic, and its caption states pointwise two-sided 90\\% intervals. No estimate, test or conclusion uses the legend text. Only the two legend strings in the code change."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N022", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. Lines 128 and 129 set len to 30 and dVec to the sequence from 0 to len by 1. A red box marks line 195, dVec_real <- dVec[-1]. In line 200, plot( AMR_est[1:17, 2] ~ dVec_real[1:17], a red box marks dVec_real[1:17]. Lines 210 and 211 plot Conley.CI[1:17, 1] and Conley.CI[1:17, 2] against dVec_real[1:17]. In line 221, plot( AMR_est_smoothed[1:17, 2] ~ dVec_real[1:17], a red box marks dVec_real[1:17].", "caption": "Source code in the replication archive, not a page of the article: application/scripts/Jayachandran_etal_2017.R, lines 128–129, 195, 198–200, 210–211 and 219–221, typeset from the file with line numbers added. The marked line 195 removes the distance 0 from the horizontal vector; the marked terms on lines 200 and 221 plot rows 1 to 17 of the estimates against it.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N023.png"}, {"alt": "Plot titled Hajek Estimator with Its CIs, with horizontal axis Distance (kilometers), ticks at 5, 10 and 15, and vertical axis Estimates from -0.04 to 0.04. A solid black curve of estimates rises from about -0.013 at the left to about 0 at the right, between black dotted and blue dotted curves. A red box marks the left end of the solid curve, which lies four tick units to the left of 5, at 1 kilometre. All curves end two units to the right of 15.", "caption": "Figure in the replication archive, not printed in the article: application/graphs/J2017_AME_polygon_unsmoothed_reproj_90CI.pdf. The marked point is the first estimate. It belongs to distance 0, the interior of the villages, and it is drawn at 1 kilometre; the curves end at 17.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N023-b.png"}, {"alt": "Plot titled Kernel Regression Estimator with Its CIs, with horizontal axis Distance (kilometers), ticks at 5, 10 and 15, and vertical axis Estimates from -0.04 to 0.04. A smooth solid curve rises from about -0.012 at the left to about 0 at the right, between two black dotted curves. A red box marks the left end of the solid curve, at 1 kilometre.", "caption": "The second figure, application/graphs/J2017_AME_polygon_smoothed_reproj_90CI.pdf, with the same shift: the marked first estimate belongs to distance 0 and is drawn at 1 kilometre.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N023-c.png"}], "formal_release": null, "formal_statements": [], "headline": "Two figures of the Uganda application in the replication archive draw every estimate and interval end one kilometre to the right of its distance, because the script drops the first entry of the distance vector but not of the estimates; the full distance vector should be used.", "id": "P004-N023", "location": "Replication archive (Supplement B): application/scripts/Jayachandran_etal_2017.R, lines 195–232, and the figures application/graphs/J2017_AME_polygon_unsmoothed_reproj_90CI.pdf and J2017_AME_polygon_smoothed_reproj_90CI.pdf. These two figures are not printed in the article; the published counterpart is Figure 11, p. 764 (PDF p. 21).", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{Jayachandran\\_etal\\_2017.R}, lines 195, 200 and 221:\n\n\\texttt{dVec\\_real <- \\hl{dVec[-1]}}\n\n\\texttt{plot( AMR\\_est[1:17, 2] $\\sim$ dVec\\_real[1:17],}\n\n\\texttt{plot( AMR\\_est\\_smoothed[1:17, 2] $\\sim$ dVec\\_real[1:17],}\n\nBoth horizontal axes are labelled ``Distance (kilometers)''.", "corrected": "Line 195:\n\n\\texttt{dVec\\_real <- \\hl{dVec}}\n\nThe two figures should then be drawn again.", "evidence": "The evidence is an exact check of the stored results and of the two saved figures. In the results stored in the archive, the distance grid and the first column of both arrays of estimates equal $(0,1,\\dots,30)$. The first unsmoothed estimate, $-0.013444575$, has distance $0$, and the plotting formula pairs it with \\texttt{dVec[-1][1]} $=1$. In both figure files the ticks 5, 10 and 15 stand at 188, 313 and 438 points, which is 25 points per kilometre from an origin at 63 points. Each curve has its vertices at $88+25(i-1)$ points for $i=1,\\dots,17$, that is, at $x=i$. The first vertex is at $(88-63)/25=1$ kilometre and the last at 17. With line 195 changed as below, the same code places the 136 values at the distances $0,\\dots,16$ and leaves their heights unchanged.", "problem": "The distances are \\texttt{dVec} $=(0,1,\\dots,30)$, and row $i$ of each array of estimates belongs to the distance $i-1$. Line 195 deletes the first entry, so that \\texttt{dVec\\_real[i]} $=i$. The eight plotted series are the two curves of estimates, the two ends of each spatial HAC interval and the two quantile curves under the sharp null. They keep rows 1 to 17 and are drawn against \\texttt{dVec\\_real[1:17]}. Row $i$, computed for the distance $i-1$, therefore appears at $x=i$, and each of the 136 plotted values lies one kilometre too far to the right. The article separates the effect inside the villages, at distance zero, from spillovers at positive distances; in these two figures the interior effect appears at 1 kilometre.", "what_survives": "The estimates, the intervals and the test are computed before line 195 and do not change. Later plots in the script use \\texttt{dVec} directly, line 378 restores \\texttt{dVec\\_real <- dVec}, and the later figures of the archive have their curves at $0,\\dots,16$ kilometres. Figure 11 is a separate graphic and starts at zero. Only the horizontal position of the curves in these two files changes. The heights of the interval curves in the same files are the subject of \\fid{P004-N022} and do not bear on the shift."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N023", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of Proposition 4 and its two displays. Red boxes surround lim and its subscript N at the start of part (i) and of part (ii); the arrow and infinity sign of each subscript lie just outside the boxes.", "caption": "Proposition 4, p. 755 (PDF p. 12): both parts state an ordinary limit.", "url": "https://pmaronow.github.io/llm-audit/works/P004/exhibits/P004-N024.png"}], "formal_release": null, "formal_statements": [{"id": "P004-T43", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-T43"}], "headline": "Proposition 4 states that the coverage probabilities of the two intervals have limits of at least $1-\\alpha$; under its conditions the limits need not exist, and the guarantee to read is that their limits inferior are at least $1-\\alpha$.", "historical_evidence_type": "lean_proof", "id": "P004-N024", "location": "Proposition 4, p. 755 (PDF p. 12)", "pdf_page": 12, "printed_page": "755", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P004-T43"], "text": {"as_printed": "Proposition 4, p.~755: ``Let $K(\\cdot)$ in (19) be the uniform kernel. Under C1--C4 and if $N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$ is uniformly bounded below for all large $N$, [\\ldots] for each $\\alpha<1$:\n\\[\n\\text{(i) }\\hl{\\lim_{N\\to\\infty}}\\mathrm{Prob}\\Biggl(z_{\\frac\\alpha2}\\le\\frac{\\widehat\\tau_{\\mathrm{HA}}(d)-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat{\\mathrm V}_{\\mathrm{SAH}}(d)}}\\le z_{1-\\frac\\alpha2}\\Biggr)\\ge1-\\alpha;\n\\]\n\\[\n\\text{(ii) additionally under C5, }\\hl{\\lim_{N\\to\\infty}}\\mathrm{Prob}\\Biggl(z_{\\frac\\alpha2}\\le\\frac{\\widehat\\tau_{\\mathrm{HA}}(d)-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d)}}\\le z_{1-\\frac\\alpha2}\\Biggr)\\ge1-\\alpha.\\text{''}\n\\]", "corrected": "Proposition 4, restated: let $K(\\cdot)$ in (19) be the uniform kernel. Under C1--C4, and if $N\\times\\operatorname{AVar}(\\widehat\\tau_{\\mathrm{HA}}(d))$ is bounded below by a positive constant for all large $N$, then for each $0<\\alpha<1$:\n\\[\n\\text{(i) }\\hl{\\liminf_{N\\to\\infty}}\\mathrm{Prob}\\Biggl(z_{\\frac\\alpha2}\\le\\frac{\\widehat\\tau_{\\mathrm{HA}}(d)-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat{\\mathrm V}_{\\mathrm{SAH}}(d)}}\\le z_{1-\\frac\\alpha2}\\Biggr)\\ge1-\\alpha;\n\\]\n\\[\n\\text{(ii) additionally under C5, }\\hl{\\liminf_{N\\to\\infty}}\\mathrm{Prob}\\Biggl(z_{\\frac\\alpha2}\\le\\frac{\\widehat\\tau_{\\mathrm{HA}}(d)-\\mathrm{AME}(d;\\eta)}{\\sqrt{\\widehat{\\mathrm V}_{\\mathrm{HAC}}(d)}}\\le z_{1-\\frac\\alpha2}\\Biggr)\\ge1-\\alpha.\n\\]", "intended_meaning": "The conditions of Proposition 4 bound the coverage probabilities from below as $N$ grows, but they do not make them converge. In a design that satisfies C1--C4 and the variance bound, with units spaced so that each forms its own neighbourhood, treatment probability $1/2$ and bounded outcomes whose form differs between even and odd $N$, the coverage of the interval in part (i) at $\\alpha=1/2$ tends to $1/2$ along even $N$ and to about $0.66$ along odd $N$, so it has no limit. Both values are at least $1-\\alpha$, and the interval does not cover too rarely. Read as the statement that the limit inferior of each coverage probability is at least $1-\\alpha$, which is what an asymptotically conservative interval requires, the proposition is not affected by the example. The example concerns part (i); part (ii) asserts a limit of the same form."}, "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N024", "work": {"arxiv_id": "2010.13599", "doi": "10.1214/24-AOAS1986", "title": "Design-Based Inference for Spatial Experiments under Unknown Interference", "url": "https://pmaronow.github.io/llm-audit/works/P004/"}, "work_id": "P004"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of page 1510. The first line reads 'If kappa log d = o(1),' with the footnote mark 8. The display below gives ATE-hat_I as ATE plus the treated-arm average of y_i(1) minus its mean minus x_i' beta_{I,T}, minus the corresponding control-arm average, plus O_p of the square root of kappa d over n. A red box marks the condition 'kappa log d = o(1)'.", "caption": "Theorem 4.1, continued, p. 1510 (PDF p. 9): the condition and the expansion. The marked condition is $\\kappa\\log d=o(1)$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T7", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T7"}, {"id": "P005-T8", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T8"}, {"id": "P005-T9", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T9"}, {"id": "P005-T20", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T20"}, {"id": "P005-T24", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T24"}, {"id": "P005-T27", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T27"}], "headline": "The condition for the expansion in Theorem 4.1 should read $\\kappa\\log\\max\\{d,2\\}=o(1)$: for a single covariate the printed condition $\\kappa\\log d=o(1)$ is empty, and the expansion can fail.", "id": "P005-E001", "location": "Theorem 4.1, p. 1510 (PDF p. 9), the condition “If $\\kappa\\log d=o(1)$” and the expansion that follows it; the theorem begins on p. 1509 (PDF p. 8)", "pdf_page": 9, "printed_page": "1510", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T20", "P005-T24", "P005-T27", "P005-T7", "P005-T8", "P005-T9"], "text": {"as_printed": "Theorem 4.1 is stated ``Under Assumptions 1--3''. Its last paragraph reads ``If $\\kappa\\hl{\\log d}=o(1)$,'' with footnote 8 attached, and continues with the display\n\\[\n\\widehat{\\mathrm{ATE}}_I=\\mathrm{ATE}+\\frac{1}{n_T}\\sum_{i\\in[T]}\\bigl(y_i(1)-\\overline{y(1)}-x_i'\\beta_{I,T}\\bigr)-\\frac{1}{n_C}\\sum_{i\\in[C]}\\bigl(y_i(0)-\\overline{y(0)}-x_i'\\beta_{I,C}\\bigr)+O_p\\Bigl(\\sqrt{\\frac{\\kappa d}{n}}\\Bigr).\n\\]\nFootnote 8: ``The upper bound on the stochastic order of the bias of the interacted case is characterized in equation (15) in Lei and Ding (2021).''", "corrected": "Theorem 4.1, last paragraph. Let Assumptions 1--3 hold, and let all design matrices be invertible, as footnote 4 assumes. If $\\kappa\\hl{\\log\\max\\{d,2\\}}=o(1)$, then\n\\[\n\\widehat{\\mathrm{ATE}}_I=\\mathrm{ATE}+\\frac{1}{n_T}\\sum_{i\\in[T]}\\bigl(y_i(1)-\\overline{y(1)}-x_i'\\beta_{I,T}\\bigr)-\\frac{1}{n_C}\\sum_{i\\in[C]}\\bigl(y_i(0)-\\overline{y(0)}-x_i'\\beta_{I,C}\\bigr)+O_p\\Bigl(\\sqrt{\\frac{\\kappa d}{n}}\\Bigr).\n\\]\nEquivalently, the printed condition $\\kappa\\log d=o(1)$ is kept, and $\\kappa=o(1)$ is required in addition when $d=1$.", "evidence": "The evidence is an exact counterexample and a derivation. For even $n\\ge6$, let $d=1$, $n_T=n_C=n/2$ and $\\epsilon=n^{-2}$. Let $v_1=0$, and let $v_2,\\ldots,v_n$ be the $n-1$ equally spaced points from $-1$ to $1$. I define $a_1=1$ and $a_i=\\epsilon v_i$ for $i\\ge2$, and I write $x$ for the array $a$ after centering and scaling to $\\overline{x^2}=1$. The potential outcomes are $y_i(1)=y_i(0)=v_i$. Assumptions 1--3 hold, with $K=1$ in Assumption 1. The average treatment effect is zero, and both arm regressions are invertible under every assignment. The maximum leverage is\n\\[\n\\kappa=\\frac{3n(n-1)(n-2)}{3n^2(n-2)+1}\\longrightarrow1,\n\\]\nwhile $\\kappa\\log d=0$ for every $n$. In the arm without unit 1, the relation $y_i=a_i/\\epsilon$ holds exactly, so the fitted value at $x=0$, that is, at $a=1/n$, equals $n$. In the arm with unit 1, the fitted value at $x=0$ is less than $3$ in absolute value. Hence $|\\widehat{\\mathrm{ATE}}_I|\\ge n-3$ under every assignment. The linear term of the expansion is at most $13/6$ in absolute value. Combining these two bounds, I find that the remainder $R_n$ satisfies $|R_n|\\ge n-6$ under every assignment, while the claimed order is $\\sqrt{\\kappa/n}\\le n^{-1/2}$. The ratio is at least $(n-6)\\sqrt n$, which implies that $R_n$ is not $O_p(\\sqrt{\\kappa d/n})$. I enumerated all assignments in exact arithmetic for $n=6,8,10,12,14$, and the results agree with these bounds. At $n=12$, the smallest value of $|R_n|$ over the 924 assignments is $8690712840/752921287$, which is larger than $11$. Note that in this example the estimator and its remainder each have mean zero over the assignments. The failure concerns the stochastic order of the remainder and not the expected bias.\n\nFor the derivation, let $d=1$, and suppose that Assumptions 1--3 hold, that the arm regressions are invertible, and that $\\kappa=o(1)$. Let $S$ be an arm of size $m$ and $w$ a fixed array. Then\n\\[\nE\\bar w_S=\\bar w,\\qquad \\operatorname{Var}(\\bar w_S)=\\frac{n-m}{m(n-1)}\\,\\overline{(w-\\bar w)^2},\n\\]\nand the coefficient in the formula for the variance is $O(1/n)$ by Assumption 3. Since $\\bar x=0$, $\\overline{x^2}=1$ and $\\max_ix_i^2=n\\kappa$, I have $\\bar x_S=O_p(n^{-1/2})$ and $\\overline{x^4}\\le n\\kappa$. This gives $\\overline{x^2}_S=1+O_p(\\sqrt\\kappa)$ and $\\widehat L_S=\\overline{x^2}_S-\\bar x_S^2=1+O_p(\\sqrt\\kappa)$, so that $\\widehat L_S^{-1}=O_p(1)$. Let $y$ be the potential outcome of the arm, $\\beta=\\overline{xy}$ and $e_i=y_i-\\bar y-x_i\\beta$. Then $\\bar e=\\overline{xe}=0$ and $\\overline{e^2}\\le K$. Applying the formula for the variance to the residuals, I find $\\bar e_S=O_p(n^{-1/2})$. The same formula yields $\\overline{xe}_S=O_p(\\sqrt\\kappa)$, since $\\overline{x^2e^2}\\le n\\kappa\\,\\overline{e^2}$. The fitted slope satisfies\n\\[\n\\widehat\\beta_S-\\beta=\\widehat L_S^{-1}\\bigl(\\overline{xe}_S-\\bar x_S\\bar e_S\\bigr)=O_p(\\sqrt\\kappa),\n\\]\nwhere I used that $\\kappa\\ge1/n$. Subtracting the linear term from the estimator, I obtain\n\\[\nR_n=-\\bar x_T\\bigl(\\widehat\\beta_{I,T}-\\beta_{I,T}\\bigr)+\\bar x_C\\bigl(\\widehat\\beta_{I,C}-\\beta_{I,C}\\bigr)=O_p\\Bigl(\\sqrt{\\frac{\\kappa}{n}}\\Bigr).\n\\]\nThis is the stated remainder for $d=1$. For $d\\ge2$ the corrected condition is the printed one; the expansion in that range is the subject of \\fid{P005-N004}.", "problem": "The number of covariates $d$ may equal $1$: Assumption 1 allows every $d\\le n$, and the paper gives $\\log$ no special meaning. Since $\\log1=0$, the condition $\\kappa\\log d=o(1)$ holds for every sequence with $d=1$, whatever the maximum leverage $\\kappa$. The theorem therefore asserts the expansion with remainder $O_p(\\sqrt{\\kappa/n})$ for sequences with a single covariate such that $\\kappa$ does not tend to zero. On such sequences the expansion can fail. One unit whose leverage stays away from zero is enough, as the example below shows.", "what_survives": "The finite-sample decomposition in Theorem 4.1, the exact bias formulas, the noninteracted results and the case $d\\ge2$ are unaffected. Theorem 4.2 already assumes $\\kappa=o(1)$ for its expansion. Its proof in Appendix B.4 refers to the stochastic orders of Theorem 4.1, but that assumption excludes the example. The condition of Remark 4, a fixed $d$ with bounded fourth moments, also excludes it, since $\\overline{x^4}$ grows at least as $n\\kappa^2$ in the example. In the example the mean of the debiased estimator of Theorem 4.2 over all assignments is exactly zero, which is the average treatment effect. What changes is the domain of the expansion for $d=1$: a sequence in which the leverage of one unit does not vanish is excluded. The estimators, the estimand and the stated remainder do not change."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E001", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the statement of Lemma A.1: the line 'Lemma A.1: Let t in {T, C} and n at least 4. Then,', the identity for the expectation of the covariance estimator, and the variance bound, followed by the first two lines of the proof. A red box marks the whole right side of the variance bound, from the inequality sign to the sum of the squares b_i^2.", "caption": "Lemma A.1, p. 1511 (PDF p. 10). The marked right side of the variance bound is written with the vectors $a$ and $b$ themselves, which the paragraph before the lemma allows to be arbitrary.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E002.png"}, {"alt": "Crop of the second paragraph of the proof of Lemma A.1 and the two-line display after it. A red box marks the clause 'we can assume, without loss of generality, a-bar = 0 and b-bar = 0', with its footnote mark 9. The display bounds Var(ab-bar_T minus a-bar_T b-bar_T) by the same right side as in the lemma.", "caption": "Proof of Lemma A.1, p. 1511 (PDF p. 10). The marked clause centers $a$ and $b$; the display below it then derives the bound for centered vectors only.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E002-b.png"}, {"alt": "Crop of item 2, which begins 'Since the first-order term in (11) is canceled' and ends 'First notice, by Lemma A.1 and Assumption 1,', and of the four-line chain of inequalities for the variance of the treated-arm covariance of h and y(1). A red box marks the first line of the chain, whose terms contain h_i y_i(1) minus the mean of hy(1), y_i(1) squared and h_i squared. The chain ends with O(kappa squared n plus kappa d over n).", "caption": "Proof of Theorem 3.2, item 2, p. 1517 (PDF p. 16). The marked line applies Lemma A.1 to the leverage $h$, whose mean is $d$, and the outcome $y(1)$, whose mean need not be zero.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E002-c.png"}, {"alt": "Dot plot with three rows, one for each example: arbitrary a and b with n = 6 and n_t = 4; a_i b_i = 1 for all i with n = 14 and n_t = 7; leverage h and outcome y(1) with n = 16 and n_T = 8. The horizontal axis is the bound divided by the exact variance of the estimator, from 0 to about 2.3, with a vertical line at ratio 1 labeled exact variance. In each row a blue circle for the bound as printed lies to the left of the line, at 0.77, 0.90 and 0.93, and an orange square for the centered bound lies to the right, at about 1.89, 2.05 and 2.07.", "caption": "The printed and the centered bound of Lemma A.1, each divided by the exact variance of the covariance estimator, in the three examples of the table. A ratio below 1 means that the bound fails. The printed bound has ratios 0.77, 0.90 and 0.93.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E002-bounds.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T1", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T1"}, {"id": "P005-T2", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T2"}, {"id": "P005-T3", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T3"}, {"id": "P005-T4", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T4"}, {"id": "P005-T19", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T19"}, {"id": "P005-T26", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T26"}], "headline": "The variance bound of Lemma A.1 holds when its right side is computed from the centered vectors $a-\\bar a$ and $b-\\bar b$, not for arbitrary $a$ and $b$ as printed; with this change, the order $O(\\kappa^2n+\\kappa d/n)$ in the proof of Theorem 3.2 is the same.", "id": "P005-E002", "location": "Appendix A.2, Lemma A.1, p. 1511 (PDF p. 10), the variance bound and the second paragraph of its proof; applied in Appendix B.2, proof of Theorem 3.2, item 2, p. 1517 (PDF p. 16)", "pdf_page": 10, "printed_page": "1511", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T1", "P005-T19", "P005-T2", "P005-T26", "P005-T3", "P005-T4"], "text": {"as_printed": "P.~1510: ``Let $a=(a_1,\\ldots,a_n)\\in\\mathbb R^n$ and $b=(b_1,\\ldots,b_n)\\in\\mathbb R^n$ be arbitrary $n$-vectors with fixed elements. We define the covariance estimator for the treated group as $\\widehat{\\operatorname{Cov}}_T(a,b)=\\overline{ab}_T-\\bar a_T\\bar b_T$, and similarly for the control group as $\\widehat{\\operatorname{Cov}}_C(a,b)=\\overline{ab}_C-\\bar a_C\\bar b_C$.''\n\nP.~1511: ``LEMMA A.1: Let $t\\in\\{T,C\\}$ and $n\\ge4$. Then,\n\\[\n\\mathbb E\\bigl[\\widehat{\\operatorname{Cov}}_t(a,b)\\bigr]=\\frac{(n_t-1)n}{n_t(n-1)}\\bigl(\\overline{ab}-\\bar a\\bar b\\bigr),\n\\]\n\\[\n\\operatorname{Var}\\bigl(\\widehat{\\operatorname{Cov}}_t(a,b)\\bigr)\\le\\frac{2(n-n_t)}{n_t(n-1)}\\frac1n\\sum_{i=1}^n\\bigl(\\hl{a_ib_i-\\overline{ab}}\\bigr)^2+\\frac{2(n-n_t)n}{n_t^3(n-1)}\\frac1n\\sum_{i=1}^n\\hl{a_i^2}\\times\\frac1n\\sum_{i=1}^n\\hl{b_i^2}.\\text{''}\n\\]\n\nP.~1511, proof: ``To upper-bound the variance, we first note that the covariance estimator is location invariant and we can assume, \\hl{without loss of generality}, $\\bar a=0$ and $\\bar b=0$.$^{9}$''\n\nP.~1517, item 2: ``First notice, by Lemma A.1 and Assumption 1,\n\\[\n\\begin{aligned}\n&\\operatorname{Var}\\bigl[\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\bigr]\\\\\n&\\quad\\le\\frac{2(n-n_T)}{n_T(n-1)}\\frac1n\\sum_{i=1}^n\\bigl(\\hl{h_iy_i(1)-\\overline{hy(1)}}\\bigr)^2+\\frac{2(n-n_T)n}{n_T^3(n-1)}\\frac1n\\sum_{i=1}^n\\hl{y_i^2(1)}\\,\\frac1n\\sum_{i=1}^n\\hl{h_i^2}\\\\\n&\\quad\\le\\frac{2(n-n_T)}{n_T(n-1)}\\frac1n\\sum_{i=1}^n\\hl{h_i^2y_i^2(1)}+\\frac{2(n-n_T)n}{n_T^3(n-1)}\\frac1n\\sum_{i=1}^n\\hl{y_i^2(1)}\\,\\frac1n\\sum_{i=1}^n\\hl{h_i^2}\\\\\n&\\quad\\le\\frac{2(n-n_T)}{n_T(n-1)}n^2\\kappa^2\\frac1n\\sum_{i=1}^ny_i^2(1)+\\frac{2(n-n_T)n}{n_T^3(n-1)}\\frac1n\\sum_{i=1}^ny_i^2(1)\\,\\frac1n\\sum_{i=1}^nh_i\\times n\\kappa\\\\\n&\\quad=O\\Bigl(\\kappa^2n+\\frac{\\kappa d}{n}\\Bigr).\\text{''}\n\\end{aligned}\n\\]", "corrected": "Lemma A.1: Let $a$ and $b$ be arbitrary $n$-vectors with fixed elements, and write $a_i^c=a_i-\\bar a$ and $b_i^c=b_i-\\bar b$. Let $t\\in\\{T,C\\}$ and $n\\ge4$. Then,\n\\[\n\\mathbb E\\bigl[\\widehat{\\operatorname{Cov}}_t(a,b)\\bigr]=\\frac{(n_t-1)n}{n_t(n-1)}\\bigl(\\overline{ab}-\\bar a\\bar b\\bigr),\n\\]\n\\[\n\\operatorname{Var}\\bigl(\\widehat{\\operatorname{Cov}}_t(a,b)\\bigr)\\le\\frac{2(n-n_t)}{n_t(n-1)}\\frac1n\\sum_{i=1}^n\\bigl(\\hl{a_i^cb_i^c-\\overline{a^cb^c}}\\bigr)^2+\\frac{2(n-n_t)n}{n_t^3(n-1)}\\frac1n\\sum_{i=1}^n\\hl{(a_i^c)^2}\\times\\frac1n\\sum_{i=1}^n\\hl{(b_i^c)^2}.\n\\]\n\nP.~1517, item 2: ``First notice, by Lemma A.1 applied to $h$ and $y(1)$, whose centered versions are $h-d$ and $y(1)-\\overline{y(1)}$, and by Assumptions 1--3,\n\\[\n\\begin{aligned}\n&\\operatorname{Var}\\bigl[\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\bigr]\\\\\n&\\quad\\le\\frac{2(n-n_T)}{n_T(n-1)}\\frac1n\\sum_{i=1}^n\\Bigl(\\hl{(h_i-d)\\bigl(y_i(1)-\\overline{y(1)}\\bigr)-\\overline{(h-d)\\bigl(y(1)-\\overline{y(1)}\\bigr)}}\\Bigr)^2\\\\\n&\\qquad+\\frac{2(n-n_T)n}{n_T^3(n-1)}\\frac1n\\sum_{i=1}^n\\hl{\\bigl(y_i(1)-\\overline{y(1)}\\bigr)^2}\\,\\frac1n\\sum_{i=1}^n\\hl{(h_i-d)^2}\\\\\n&\\quad\\le\\frac{2(n-n_T)}{n_T(n-1)}n^2\\kappa^2\\hl{K}+\\frac{2(n-n_T)n}{n_T^3(n-1)}n\\kappa\\hl{dK}\\\\\n&\\quad=O\\Bigl(\\kappa^2n+\\frac{\\kappa d}{n}\\Bigr),\n\\end{aligned}\n\\]\nwhere $K$ is the constant of Assumption 1.'' The next sentence of the proof, which gives the order $O(\\kappa^2/n+\\kappa d/n^3)$ for the variance of $n^{-1}(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T)$, is unchanged.", "evidence": "The evidence is an exact counterexample to the printed bound and a derivation of the centered bound. All variances below are over the random assignment, in which the group is a uniformly random subset of size $n_t$. For each numerical example, the variance is computed in exact rational arithmetic over every assignment.\n\n\\emph{The lemma.} Let $n=6$, $n_t=4$, $a=(0,-3,-1,2,-4,-3)$ and $b=(2,1,3,-3,1,1)$. Over the 15 assignments, $\\widehat{\\operatorname{Cov}}_t(a,b)$ has mean $-69/40$, in agreement with the expectation identity, and variance $16607/4800$. The printed right side is $3829/1440$, which is smaller. The centered bound is $835/128$; the means of $a$ and $b$ are $-3/2$ and $5/6$.\n\n\\emph{The application.} Let $n=16$, $n_T=8$, $d=1$ and $x=(8c,-8c,c,-c,\\ldots,c,-c)$ with $c=\\sqrt{8/71}$, so that $\\bar x=0$ and $\\overline{x^2}=1$. Then $h_i=x_i^2$ equals $512/71$ for the first two units and $8/71$ for the others, $\\bar h=1=d$ and $\\kappa=32/71$. Let $y(1)=(1/64,1/64,1,\\ldots,1)$ and $y(0)=0$. The mean squares of $x$, $y(1)$ and $y(0)$ are at most 1, and $n_T/n=1/2$. Since every product $h_iy_i(1)$ equals $8/71$, the first term of the printed bound vanishes. Over the 12870 assignments, the variance of $\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T$ is $1678565511/8259174400$. The first line of the display on p.~1517 equals $117645319/619438080$ and the second line equals $334349/1744896$; both are smaller than the variance. The centered bound equals $5550293259/13214679040$. In a third example, with $n=14$, $n_t=7$, $a=(30,1,\\ldots,1)$ and $b_i=1/a_i$, the variance is $707281/240100$, the printed right side is $10683013/4013100$ and the centered bound is $365664277/60505200$. The table collects the three examples.\n\n\\begin{audittable}{lrrr}{Exact variance of $\\widehat{\\operatorname{Cov}}_t(a,b)$ and the two bounds in three examples, rounded to four decimals; the exact values are in the text. A valid upper bound is at least as large as the variance.}\nExample & Exact variance & Bound as printed & Centered bound \\\\\n$n=6$, $n_t=4$, arbitrary $a$ and $b$ & 3.4598 & 2.6590 & 6.5234 \\\\\n$n=14$, $n_t=7$, $a_ib_i=1$ for all $i$ & 2.9458 & 2.6620 & 6.0435 \\\\\n$n=16$, $n_T=8$, $a=h$ and $b=y(1)$ & 0.2032 & 0.1899 & 0.4200 \\\\\n\\end{audittable}\n\n\\emph{The centered bound.} Let $n\\ge4$ and $1\\le m\\le n-1$, where $m=n_t$, let $S$ be the random group, and write $u=a-\\bar a$ and $v=b-\\bar b$. Expanding the sample means shows that $\\widehat{\\operatorname{Cov}}_t(a,b)=\\overline{uv}_S-\\bar u_S\\bar v_S$ for every $S$. By the variance of a sample mean under simple random sampling, I have\n\\[\n\\operatorname{Var}\\bigl(\\overline{uv}_S\\bigr)=\\frac{n-m}{m(n-1)}\\frac1n\\sum_{i=1}^n\\bigl(u_iv_i-\\overline{uv}\\bigr)^2.\n\\]\nFor the product of the two means, set $Z=\\bigl(\\sum_{i\\in S}u_i\\bigr)\\bigl(\\sum_{i\\in S}v_i\\bigr)$, $A=\\sum_iu_i^2$, $B=\\sum_iv_i^2$, $G=\\sum_iu_iv_i$ and $H=\\sum_iu_i^2v_i^2$, and define\n\\[\n\\alpha=\\frac{m(n-m)}{n(n-1)},\\qquad\\ell=\\frac{m(m-1)(n-m)(n-m-1)}{n(n-1)(n-2)(n-3)}.\n\\]\nExpanding $Z^2$, grouping the terms by the number of distinct indices and using $\\sum_iu_i=\\sum_iv_i=0$, I obtain\n\\[\n\\operatorname{Var}Z=(\\alpha-6\\ell)H+\\ell AB+(2\\ell-\\alpha^2)G^2.\n\\]\nHere $0\\le H\\le AB$, $G^2\\le AB$ and $G^2\\le nH$, and $\\ell\\le\\alpha(n-2)/(4(n-3))$ because the factors $m-1$ and $n-m-1$ sum to $n-2$. If $\\alpha\\ge6\\ell$, the identity implies that $\\operatorname{Var}Z\\le(\\alpha-3\\ell)AB\\le\\alpha AB$. If $\\alpha<6\\ell$, using $H\\ge G^2/n$, I find\n\\[\n\\operatorname{Var}Z\\le\\Bigl[\\frac\\alpha n+\\Bigl(3-\\frac6n\\Bigr)\\ell\\Bigr]AB\\le\\alpha\\Bigl[1-\\frac{n-4}{4(n-3)}\\Bigr]AB\\le\\alpha AB.\n\\]\nIn the second inequality, I used the bound on $\\ell$. Since $\\bar u_S\\bar v_S=Z/m^2$, in both cases\n\\[\n\\operatorname{Var}\\bigl(\\bar u_S\\bar v_S\\bigr)\\le\\frac{(n-m)n}{m^3(n-1)}\\Bigl(\\frac1n\\sum_{i=1}^nu_i^2\\Bigr)\\Bigl(\\frac1n\\sum_{i=1}^nv_i^2\\Bigr).\n\\]\nThe centered bound follows from $\\operatorname{Var}(U-V)\\le2\\operatorname{Var}U+2\\operatorname{Var}V$. Both of its sides are unchanged when constants are added to $a$ and $b$. The reduction to $\\bar a=\\bar b=0$ in the printed proof is then legitimate.\n\n\\emph{The order on p.~1517.} By Assumption 2, I have $\\bar h=d$ and $0\\le h_i\\le n\\kappa$. Then $|h_i-d|\\le n\\kappa$ and $n^{-1}\\sum_i(h_i-d)^2\\le n^{-1}\\sum_ih_i^2\\le n\\kappa d$. Assumption 1 gives $n^{-1}\\sum_i(y_i(1)-\\overline{y(1)})^2\\le K$. Subtracting a mean lowers a mean square, so the first sum in the centered bound is at most $n^{-1}\\sum_i(h_i-d)^2(y_i(1)-\\overline{y(1)})^2\\le n^2\\kappa^2K$. Inserting these bounds into the centered bound, and using Assumption 3 for the coefficients, I obtain the chain in the correction below.", "problem": "Lemma A.1 is stated for arbitrary fixed vectors $a$ and $b$, and the bars in its variance bound denote their population means. The variance argument begins by assuming $\\bar a=\\bar b=0$, on the ground that the covariance estimator is location invariant. The estimator is unchanged when a constant is added to $a$ or to $b$, but the right side of the bound is not. The proof therefore establishes the bound only when the right side is computed from $a-\\bar a$ and $b-\\bar b$. I refer to the bound with this right side as the centered bound. For vectors that are not centered, the printed right side can be smaller than the variance. Item 2 of the proof of Theorem 3.2 applies the lemma to the leverage $h$, whose population mean is $d$ by Assumption 2, and to the outcome $y(1)$, whose mean need not be zero. The first two purported upper bounds in the display on p.~1517 are not valid in general: both fail for a design that Assumptions 1--3 allow.", "what_survives": "The expectation identity of Lemma A.1 is correct as printed. The covariance estimators, the bias corrections and the unbiasedness claims of Theorems 3.2 and 4.2 do not depend on the variance bound. The asymptotic expansions of Theorems 3.2 and 4.2 use the bound only for the order of the variance of the covariance between leverage and outcome. Under Assumptions 1--3 this order is the same with the centered bound, and this variance error does not affect the expansions. This correction requires no change to an assumption, estimand, estimator or theorem conclusion, or to a computation in the replication files."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E002", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the lower part of Theorem 4.1 on page 1509. Two rows define nu_1T, nu_2T, nu_3T and nu_1C, nu_2C, nu_3C. Here nu_3T is minus x-bar_T times the treated mean of y star of 1, and nu_3C is minus x-bar_C times the control mean of y star of 0. Two displays follow. In the last display, the expectation of x-bar_C prime nu_3C minus the expectation of x-bar_T prime nu_3T is set equal to n_T (n_T - n_C) over (n-1)(n-2) n_C squared, times the covariance term for y(0), minus n_C (n_C - n_T) over (n-1)(n-2) n_T squared, times the covariance term for y(1). Red boxes mark the first coefficient and the second coefficient together with its minus sign.", "caption": "Theorem 4.1, p. 1509 (PDF p. 8): the definitions of $\\nu_{3T}$ and $\\nu_{3C}$ and the two displays for the bias components. The marked coefficients in the last display carry the reversed signs.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E003.png"}, {"alt": "Chart with three positions on the horizontal axis: n_T = 2 and n_C = 4 with 15 assignments, n_T = 3 and n_C = 3 with 20 assignments, and n_T = 4 and n_C = 2 with 15 assignments. The vertical axis is the third bias component in units of the outcome, from -0.2 to 0.2. Filled blue circles, the exact expectations, are at 217/2500, 0 and -3689/25000. Open orange squares, the printed display, are at -217/2500, 0 and 3689/25000. At each unbalanced allocation the square is the mirror image of the circle about zero, and at the balanced allocation both are zero.", "caption": "The third bias component of $\\widehat{\\mathrm{ATE}}_I$ in the six-unit example, for three allocations. Filled circles give the exact expectation over all assignments, which equals the corrected display. Open squares give the last display of Theorem 4.1 as printed.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E003-third-component.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T5", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T5"}, {"id": "P005-T6", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T6"}, {"id": "P005-T13", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T13"}, {"id": "P005-T20", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T20"}, {"id": "P005-T24", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T24"}], "headline": "Both signs on the right-hand side of the last display of Theorem 4.1 are reversed.", "id": "P005-E003", "location": "Theorem 4.1, p. 1509 (PDF p. 8), last display", "pdf_page": 8, "printed_page": "1509", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P005-T13", "P005-T20", "P005-T24", "P005-T5", "P005-T6"], "text": {"as_printed": "Theorem 4.1 defines $\\nu_{3T}=-(\\bar x_T\\overline{y^*(1)}_T)$ and $\\nu_{3C}=-(\\bar x_C\\overline{y^*(0)}_C)$, and its last display reads\n\\[\n\\begin{aligned}\nE\\bigl[\\bar x_C'\\nu_{3C}\\bigr]-E\\bigl[\\bar x_T'\\nu_{3T}\\bigr]\n={}&{}\\hl{\\frac{n_T(n_T-n_C)}{(n-1)(n-2)n_C^2}}\\bigl(\\overline{hy(0)}-\\bar h\\times\\overline{y(0)}\\bigr)\\\\\n&{}\\hl{-}\\frac{n_C(n_C-n_T)}{(n-1)(n-2)n_T^2}\\bigl(\\overline{hy(1)}-\\bar h\\times\\overline{y(1)}\\bigr).\n\\end{aligned}\n\\]", "corrected": "Theorem 4.1, last display. Let Assumptions 1--3 hold, \\hl{with Assumption 2's normalization holding at the sample size considered}, let $y_i^*(t)=y_i(t)-\\overline{y(t)}$ and $h_i=\\|x_i\\|_2^2$, and let $\\nu_{3T}=-(\\bar x_T\\overline{y^*(1)}_T)$ and $\\nu_{3C}=-(\\bar x_C\\overline{y^*(0)}_C)$, as in the theorem. The expectation is over the complete randomization of $n_T$ of the $n\\ge3$ units to treatment, with $n_C=n-n_T$. Then\n\\[\n\\begin{aligned}\nE\\bigl[\\bar x_C'\\nu_{3C}\\bigr]-E\\bigl[\\bar x_T'\\nu_{3T}\\bigr]\n={}&{}\\hl{-}\\frac{n_T(n_T-n_C)}{(n-1)(n-2)n_C^2}\\bigl(\\overline{hy(0)}-\\bar h\\times\\overline{y(0)}\\bigr)\\\\\n&{}\\hl{+}\\frac{n_C(n_C-n_T)}{(n-1)(n-2)n_T^2}\\bigl(\\overline{hy(1)}-\\bar h\\times\\overline{y(1)}\\bigr).\n\\end{aligned}\n\\]\nIn the notation of Appendix A.1, the right-hand side is $-N_{CCC}S_0+N_{TTT}S_1$. The definitions of $\\nu_{3T}$ and $\\nu_{3C}$ and the estimator of Theorem 4.2 stay as printed.", "evidence": "The evidence is a derivation and an exact counterexample. For the derivation, let $A$ be a uniformly chosen subset of size $m$ of the $n\\ge3$ units, and let $a$, $b$ and $c$ be fixed arrays that each sum to zero. I set $s=\\sum_i a_ib_ic_i$. In the sum $\\sum_{i,j,k}a_ib_jc_k$, the terms in which all three indices are equal contribute $s$, and the terms in which exactly two indices are equal contribute $-3s$. The whole sum is zero, so the terms in which the three indices are distinct contribute $2s$. One, two and three distinct units are included in $A$ with probabilities $q_1=m/n$, $q_2=m(m-1)/(n(n-1))$ and $q_3=m(m-1)(m-2)/(n(n-1)(n-2))$. I obtain\n\\[\nE\\bigl[\\bar a_A\\bar b_A\\bar c_A\\bigr]=\\frac{q_1-3q_2+2q_3}{m^3}\\,s=\\frac{(n-m)(n-2m)}{(n-1)(n-2)m^2}\\,\\overline{abc}.\n\\]\nThis is the first identity of Lemma A.2. Note that the coefficient is $N_{TTT}$ for $m=n_T$ and $N_{CCC}$ for $m=n_C$. Let $a=b$ be a coordinate of $x$, which is centered by Assumption 2, and let $c=y^*(t)$. Summing over the coordinates and using $\\overline{hy^*(t)}=S_t$, I have\n\\[\nE\\bigl[\\bar x_T'\\bar x_T\\,\\overline{y^*(1)}_T\\bigr]=N_{TTT}S_1,\\qquad\nE\\bigl[\\bar x_C'\\bar x_C\\,\\overline{y^*(0)}_C\\bigr]=N_{CCC}S_0.\n\\]\nBy the definitions of $\\nu_{3T}$ and $\\nu_{3C}$, each of which has a minus sign, I have $E[\\bar x_C'\\nu_{3C}]=-N_{CCC}S_0$ and $E[\\bar x_T'\\nu_{3T}]=-N_{TTT}S_1$. This implies that the left-hand side of the last display is $-N_{CCC}S_0+N_{TTT}S_1$.\n\nFor the counterexample, let $n=6$, $d=1$, $x=(-8,-5,0,3,4,6)/5$, $y_i(1)=x_i^2$ and $y_i(0)=2x_i^2+3$. Then $\\bar x=0$, $\\overline{x^2}=1$, $S_1=434/625$ and $S_0=868/625$, and every arm regression is invertible. For $n_T=2$ and $n_C=4$, the constants are $N_{TTT}=1/10$ and $N_{CCC}=-1/80$. Enumerating the 15 assignments in exact arithmetic, I find $217/2500$ for the left-hand side, while the right-hand side as printed is $-217/2500$. For $n_T=4$ and $n_C=2$ the two values are $-3689/25000$ and $3689/25000$, and for $n_T=n_C=3$ both are zero. If this population is repeated $r$ times, with $n_T=2r$ and $n_C=4r$, the resulting sequence satisfies Assumptions 1--3. On this sequence the third component is $5S_1/(2(6r-1)(6r-2))>0$, and the printed display is the negative of this value.\n\nA second example shows the effect on the total. Let $n=5$, $x=(-3/2,-1/2,0,1/2,3/2)$, $y(1)=(0,0,1,0,0)$, $y(0)=0$, $n_T=2$ and $n_C=3$. Enumerating the 10 assignments, I obtain the entries of the table. The three components add up to the bias of $\\widehat{\\mathrm{ATE}}_I$ only with the third component as defined.\n\\begin{audittable}{lrr}{Five-unit example: exact expectations of the components of the bias of $\\widehat{\\mathrm{ATE}}_I$.}\nQuantity & Third component as defined & Third component as printed \\\\\nFirst component & $3/40$ & $3/40$ \\\\\nSecond component & $11/80$ & $11/80$ \\\\\nThird component & $-1/80$ & $1/80$ \\\\\nSum of the three components & $1/5$ & $9/40$ \\\\\nBias of $\\widehat{\\mathrm{ATE}}_I$, by enumeration & $1/5$ & $1/5$ \\\\\n\\end{audittable}", "problem": "Theorem 4.1 writes the bias of the interacted estimator as a sum of three components, $E[\\bar x_C'\\nu_{jC}]-E[\\bar x_T'\\nu_{jT}]$ for $j=1,2,3$, and its last display states the third component in closed form. The theorem defines $\\nu_{3T}$ and $\\nu_{3C}$ with a minus sign each. With these definitions, the right-hand side of the last display is the negative of the left-hand side. I write $S_t=\\overline{hy(t)}-\\bar h\\,\\overline{y(t)}$ for $t=0,1$, and I let $N_{TTT}$ and $N_{CCC}$ denote the constants of Appendix A.1. The left-hand side is $-N_{CCC}S_0+N_{TTT}S_1$, and the display prints $N_{CCC}S_0-N_{TTT}S_1$. A bias computed by adding the three displayed components is off by twice the third component. The printed and the correct value agree only when the component is zero. This is the case when $n_T=n_C$, so a balanced design does not reveal the error. With unequal arm sizes, the component can also vanish by cancellation between nonzero $S_0$ and $S_1$.", "what_survives": "The decomposition of the coefficients in Theorem 4.1 and the display for its first component are unaffected, and no stochastic order changes. Theorem 4.2 is unaffected. Its coefficients $n_T/((n-2)n_C)$ and $-n_C/((n-2)n_T)$ follow from the corrected signs, and the opening display of its proof in Appendix B.4 carries the correct sign through an outer minus. In the six-unit example the mean of the debiased estimator over all assignments equals the average treatment effect, $-4$, for each of the three allocations. The analogous equation (6) for the noninteracted estimator is correct as printed. The interacted correction in the replication files uses the signs of Theorem 4.2. What changes is the last display of Theorem 4.1, and with it any bias computed from that display when the third component is nonzero."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E003", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop from page 1506. Two lines introduce the interacted OLS regression. Equation (2) follows: Y_i ~ alpha_I + tau_I D_i + x_i' beta_{I,C} + D_i x_i' beta_{I,T}. Two further lines name the OLS estimators of the interacted case. A red box marks the last two terms of the equation, x_i' beta_{I,C} + D_i x_i' beta_{I,T}.", "caption": "Equation (2), p. 1506 (PDF p. 5). The marked terms are $x_i'\\beta_{I,C}+D_ix_i'\\beta_{I,T}$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E004.png"}, {"alt": "Crop of the opening of Section 4 on page 1509. The first lines define beta-hat I,T as L-hat T inverse times N-hat T and beta-hat I,C as L-hat C inverse times N-hat C. Below the display for the interacted estimator, the sentence reads 'where beta-hat I,T and beta-hat I,C are the OLS coefficients on covariates D_i x_i and x_i, repsectively.' Red boxes mark the words 'OLS coefficients on covariates D_i x_i' and, on the next line, 'and x_i'.", "caption": "Section 4, p. 1509 (PDF p. 8): the definitions of $\\widehat\\beta_{I,T}$ and $\\widehat\\beta_{I,C}$, the display for $\\widehat{\\mathrm{ATE}}_I$ and the sentence after it. The marked words call the two estimators the coefficients on $D_ix_i$ and $x_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T22", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T22"}], "headline": "In Section 4, $\\widehat\\beta_{I,T}$ and $\\widehat\\beta_{I,C}$ are the slopes fitted within the treatment arm and within the control arm; under equation (2) as printed, the coefficient on $D_ix_i$ is the difference of these two slopes.", "id": "P005-E004", "location": "Equation (2), p. 1506 (PDF p. 5); Section 4, p. 1509 (PDF p. 8), the definitions of $\\widehat\\beta_{I,T}$ and $\\widehat\\beta_{I,C}$ and the sentence after the display for $\\widehat{\\mathrm{ATE}}_I$", "pdf_page": 5, "printed_page": "1506", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P005-T22"], "text": {"as_printed": "Equation (2), p.~1506:\n\\[\nY_i\\sim\\alpha_I+\\tau_ID_i+\\hl{x_i'\\beta_{I,C}+D_ix_i'\\beta_{I,T}}.\\qquad\\text{(2)}\n\\]\nSection 4, p.~1509: ``We write the regression coefficient estimators of the pretreatment covariates in the interacted case as $\\widehat\\beta_{I,T}=\\widehat L_T^{-1}\\widehat N_T$ and $\\widehat\\beta_{I,C}=\\widehat L_C^{-1}\\widehat N_C$ \\ldots'' and, after the display\n\\[\n\\widehat{\\mathrm{ATE}}_I=\\overline{y(1)}_T-\\overline{y(0)}_C-\\bigl(\\bar x_T'\\widehat\\beta_{I,T}-\\bar x_C'\\widehat\\beta_{I,C}\\bigr)\n\\]\n``for the interacted case, where $\\widehat\\beta_{I,T}$ and $\\widehat\\beta_{I,C}$ are \\hl{the OLS coefficients on covariates $D_ix_i$ and $x_i$}, repsectively.''", "corrected": "Equation (2):\n\\[\nY_i\\sim\\alpha_I+\\tau_ID_i+\\hl{(1-D_i)}x_i'\\beta_{I,C}+D_ix_i'\\beta_{I,T}.\\qquad\\text{(2)}\n\\]\nSection 4: ``for the interacted case, where $\\widehat\\beta_{I,T}$ and $\\widehat\\beta_{I,C}$ are \\hl{the OLS slopes within the treatment arm and within the control arm, that is, the coefficients on $D_ix_i$ and $(1-D_i)x_i$ in (2)}, respectively.''", "intended_meaning": "Section 4 defines the two estimators as $\\widehat L_T^{-1}\\widehat N_T$ and $\\widehat L_C^{-1}\\widehat N_C$. These are the least-squares slopes within the treatment arm and within the control arm. Every formula that follows, including Theorems 4.1 and 4.2, uses these definitions, and the display for $\\widehat{\\mathrm{ATE}}_I$ is correct with them. This is the intended meaning. The labels alone do not match. In equation (2) as printed, the fitted line is $\\alpha_I+x_i'\\beta_{I,C}$ in the control arm and $\\alpha_I+\\tau_I+x_i'(\\beta_{I,C}+\\beta_{I,T})$ in the treatment arm. The OLS coefficient on $x_i$ is the control slope, and the coefficient on $D_ix_i$ is the difference of the two slopes, $\\widehat L_T^{-1}\\widehat N_T-\\widehat L_C^{-1}\\widehat N_C$. Substituting that interaction coefficient for $\\widehat\\beta_{I,T}$ in the display gives a different number. For example, let $x=\\sqrt{3/20}\\,(-3,-2,-1,0,1,5)$ with the first two units in the treatment arm, and let $y_i(0)=x_i$ and $y_i(1)=2x_i$. The coefficients of (2) are $(0,0,1,1)$, and the slopes within the treatment arm and within the control arm are $2$ and $1$. The display equals $0$ with these two slopes and $-\\sqrt{15}/4$ with the two coefficients of (2). The parameterization below makes the labels agree and changes neither the fitted values nor $\\widehat\\tau_I$."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E004", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the opening of the proof of Lemma A.2 and its first display, which expands the expectation of the sum over i of D_i x_i y_i times the sum over j not equal to i of D_j z_j. A red box marks the bracket [D_i D_j] in the double sum after the first equality sign.", "caption": "Proof of Lemma A.2, p. 1511 (PDF p. 10), first display. The marked bracket has no expectation sign; the next equality replaces it by the probability $n_T(n_T-1)/(n(n-1))$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E005.png"}, {"alt": "Crop of the last display of the proof of Lemma A.2, which writes E[x-bar_T y-bar_T z-bar_T] as one over n_T cubed times three expectations. A red box marks the capital N above the first summation sign of the third expectation, whose lower limit is i = 1.", "caption": "Proof of Lemma A.2, p. 1512 (PDF p. 11), last display. The marked upper limit of the outer sum is $N$; the sums in the lines above run to $n$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E005-b.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T16", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T16"}], "headline": "In the proof of Lemma A.2, the bracket $[D_iD_j]$ should read $\\mathbb E[D_iD_j]$, and the upper limit $N$ should read $n$.", "id": "P005-E005", "location": "Appendix A.2, proof of Lemma A.2: first display, p. 1511 (PDF p. 10); last display, p. 1512 (PDF p. 11)", "pdf_page": 10, "printed_page": "1511", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T16"], "text": {"as_printed": "P.~1511:\n\\[\n\\mathbb E\\Bigl[\\sum_{i=1}^nD_ix_iy_i\\sum_{j\\ne i}D_jz_j\\Bigr]=\\sum_{i=1}^n\\sum_{j\\ne i}\\hl{[D_iD_j]}x_iy_iz_j=\\frac{n_T(n_T-1)}{n(n-1)}\\sum_{i=1}^n\\sum_{j\\ne i}x_iy_iz_j\n\\]\n\nP.~1512, last display:\n\\[\n\\begin{aligned}\n\\mathbb E[\\bar x_T\\bar y_T\\bar z_T]=\\frac1{n_T^3}\\Bigl(&\\mathbb E\\Bigl[\\sum_iD_ix_iy_iz_i\\Bigr]+\\mathbb E\\Bigl[\\sum_{i=1}^n\\sum_{j\\ne i}D_iD_j(x_iy_iz_j+x_iy_jz_i+x_jy_iz_i)\\Bigr]\\\\\n&+\\mathbb E\\Bigl[\\sum_{i=1}^{\\hl{N}}D_ix_i\\sum_{j\\ne i}D_jy_j\\sum_{s\\notin\\{i,j\\}}D_sz_s\\Bigr]\\Bigr)\n\\end{aligned}\n\\]", "corrected": "P.~1511:\n\\[\n\\mathbb E\\Bigl[\\sum_{i=1}^nD_ix_iy_i\\sum_{j\\ne i}D_jz_j\\Bigr]=\\sum_{i=1}^n\\sum_{j\\ne i}\\hl{\\mathbb E[D_iD_j]}x_iy_iz_j=\\frac{n_T(n_T-1)}{n(n-1)}\\sum_{i=1}^n\\sum_{j\\ne i}x_iy_iz_j\n\\]\n\nP.~1512, last display:\n\\[\n\\begin{aligned}\n\\mathbb E[\\bar x_T\\bar y_T\\bar z_T]=\\frac1{n_T^3}\\Bigl(&\\mathbb E\\Bigl[\\sum_iD_ix_iy_iz_i\\Bigr]+\\mathbb E\\Bigl[\\sum_{i=1}^n\\sum_{j\\ne i}D_iD_j(x_iy_iz_j+x_iy_jz_i+x_jy_iz_i)\\Bigr]\\\\\n&+\\mathbb E\\Bigl[\\sum_{i=1}^{\\hl{n}}D_ix_i\\sum_{j\\ne i}D_jy_j\\sum_{s\\notin\\{i,j\\}}D_sz_s\\Bigr]\\Bigr)\n\\end{aligned}\n\\]", "evidence": "The evidence is a check of pp.~1511--1512 and a derivation. With both readings restored, the coefficient of $\\sum_ix_iy_iz_i$ in the last display is\n\\[\n\\frac1{n_T^3}\\Bigl(\\frac{n_T}{n}-\\frac{3n_T(n_T-1)}{n(n-1)}+\\frac{2n_T(n_T-1)(n_T-2)}{n(n-1)(n-2)}\\Bigr)=\\frac{(n-n_T)(n-2n_T)}{n\\,n_T^2(n-1)(n-2)},\n\\]\nwhich equals $N_{\\mathrm{TTT}}/n$ with $N_{\\mathrm{TTT}}$ as defined on p.~1510.", "problem": "The first equality on p.~1511 takes the expectation inside the double sum. The bracket then needs the expectation sign, and the next equality replaces it by $n_T(n_T-1)/(n(n-1))$, which is $\\mathbb E[D_iD_j]$ for $i\\ne j$. On p.~1512 the outer sum of the third term runs to $N$. The population has $n$ units, and the same triple sum runs to $n$ in the display above.", "what_survives": "Lemma A.2, its constants and the rest of its proof are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E005", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the three-line display that bounds Var[(*)] and of the four lines of text below it. A red box marks, in the third line, the sum over i of n to the minus 1 times the squared norm of x_i times the square of e_j(1) minus e_j(0). The text below ends with the identity for the sum over i and j of (x_i prime x_j a_j) squared, which equals n times the sum over j of a_j squared times the squared norm of x_j.", "caption": "Appendix B.1.1, p. 1514 (PDF p. 13), the bound for $\\operatorname{Var}[(*)]$ and the explanation below it. In the marked sum the index of summation and the leverage carry $i$, while the residuals carry $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T17", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T17"}], "headline": "In the last line of the variance bound for $(*)$, the leverage and the residual difference should carry the same summation index.", "id": "P005-E007", "location": "Appendix B.1.1, p. 1514 (PDF p. 13), display bounding $\\operatorname{Var}[(*)]$, third line", "pdf_page": 13, "printed_page": "1514", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T17"], "text": {"as_printed": "\\[\n=\\Bigl(\\frac1{p_Cn_Tn}\\Bigr)^2n^2\\frac{n_Tn_C}{n(n-1)}\\sum_{\\hl{i}}n^{-1}\\|x_{\\hl{i}}\\|_2^2\\bigl(e_j(1)-e_j(0)\\bigr)^2=O\\Bigl(\\frac\\kappa n\\Bigr),\n\\]\nand, below the display: ``For the second to last equality, we use the fact\n\\[\n\\textstyle\\sum_{i,j}(x_i'x_ja_j)^2=\\sum_ja_j^2x_j'(\\sum_ix_ix_i')x_j=n\\sum_ja_j^2\\|x_j\\|_2^2\n\\]\nby Assumption 2.''", "corrected": "\\[\n=\\Bigl(\\frac1{p_Cn_Tn}\\Bigr)^2n^2\\frac{n_Tn_C}{n(n-1)}\\sum_{\\hl{j}}n^{-1}\\|x_{\\hl{j}}\\|_2^2\\bigl(e_j(1)-e_j(0)\\bigr)^2=O\\Bigl(\\frac\\kappa n\\Bigr),\n\\]", "evidence": "The evidence is a check of the page and a derivation. I write $\\delta_j=e_j(1)-e_j(0)$ and $h_j=\\|x_j\\|_2^2$. Since $\\sum_ix_ix_i'=nI_d$ by Assumption 2, I have\n\\[\n\\sum_{i,j}(x_i'x_j\\delta_j)^2=\\sum_j\\delta_j^2x_j'\\Bigl(\\sum_ix_ix_i'\\Bigr)x_j=n\\sum_jh_j\\delta_j^2.\n\\]\nWith the index matched, the last line equals $\\overline{h\\delta^2}/(p_Tp_Cn(n-1))$. Since $h_j\\le n\\kappa$, this is at most $\\kappa\\,\\overline{\\delta^2}/(p_Tp_C(n-1))$, which is $O(\\kappa/n)$ by Assumption 3 and the bounds on the residuals stated on p.~1513.", "problem": "The sum binds $i$, but the residuals carry $j$, which is then a free index. The identity quoted below the display contracts the double sum over $i$ and $j$ to a single sum in which the leverage and the squared coefficient have the same index.", "what_survives": "The contraction identity and the rate $O(\\kappa/n)$ are correct, and only the index changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E007", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the first lines of page 1516: the sentence 'Collecting the constants in front of' the treated-arm covariance with its display, which equals minus 1 over n minus 2, then the sentence 'Collecting the constants in front of' the control-arm expression with its display, which equals 1 over n minus 2. A red box marks the second term of the control-arm expression, a single overline over hy(0)y(0) with subscript C.", "caption": "Proof of Theorem 3.2, p. 1516 (PDF p. 15). The marked term is printed as a single mean of $hy(0)y(0)$; the parallel sentence for the treated arm, two lines above, has the product of two sample means.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T15", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T15"}], "headline": "The control-arm covariance whose coefficient is collected is $\\overline{hy(0)}_C-\\bar h_C\\,\\overline{y(0)}_C$; the coefficient $1/(n-2)$ is correct.", "id": "P005-E009", "location": "Appendix B.2, proof of Theorem 3.2, p. 1516 (PDF p. 15), the second sentence that begins “Collecting the constants in front of”", "pdf_page": 15, "printed_page": "1516", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T15"], "text": {"as_printed": "``Collecting the constants in front of $\\overline{hy(0)}_C-\\hl{\\overline{hy(0)y(0)}_C}$ gives\n\\[\n\\frac1{n-1}\\frac{n_C(n-1)}{(n_C-1)n}-\\frac{(n_T-n_C)}{(n-1)(n-2)n_T}\\frac{n_T(n-1)}{(n_C-1)n}=\\frac1{n-2}.\\text{''}\n\\]", "corrected": "``Collecting the constants in front of $\\overline{hy(0)}_C-\\hl{\\bar h_C\\,\\overline{y(0)}_C}$ gives\n\\[\n\\frac1{n-1}\\frac{n_C(n-1)}{(n_C-1)n}-\\frac{(n_T-n_C)}{(n-1)(n-2)n_T}\\frac{n_T(n-1)}{(n_C-1)n}=\\frac1{n-2}.\\text{''}\n\\]", "evidence": "The evidence is a check of the page against equations (7), (11) and (12), which print $\\overline{hy(0)}_C-\\bar h_C\\overline{y(0)}_C$, and a recalculation of the coefficient. With $n=n_T+n_C$, the left side of the display is\n\\[\n\\frac{n_C}{(n_C-1)n}-\\frac{n_T-n_C}{(n-2)(n_C-1)n}=\\frac{n_C(n-2)-n_T+n_C}{(n-2)(n_C-1)n}=\\frac1{n-2}.\n\\]", "problem": "The second term is printed as one overline over $hy(0)y(0)$. The sentence collects the coefficient of the control-arm covariance in equations (11) and (12). The second term of that covariance is the product of the sample means of $h$ and $y(0)$, as in the parallel sentence for the treated arm.", "what_survives": "The coefficient, the estimator of Theorem 3.2 and the rest of the proof are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E009", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the sentence 'Now we derive the stochastic expansion for the debiased estimator:' and the four-line display for the debiased ATE estimator. One red box marks the factor (x-bar_T minus x-bar_C), printed without a prime, before L-hat inverse N-hat in the first line. A second red box marks the index set 'i in [T]' under the second sum of the third line, the sum of y_i(0) minus x_i prime N with factor one over n_C.", "caption": "Proof of Theorem 3.2, p. 1516 (PDF p. 15), the expansion of $\\widehat{\\mathrm{ATE}}_{\\mathrm{Debiased}}$. The first marked factor has no transpose; the second marked index set belongs to a sum of control outcomes with factor $1/n_C$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E010.png"}, {"alt": "Crop of the sentence 'We can characterize the stochastic expansion for the debiased estimator of the treated group as' and the three-line display for the treated-arm debiased estimator. Red boxes mark four occurrences of x-bar_T printed without a prime: before L-hat_T inverse N-hat_T in the first line, before N_T and before (N-hat_T minus N_T) in the second line, and before N_T in the third line.", "caption": "Proof of Theorem 4.2, p. 1518 (PDF p. 17), last display. The four marked factors $\\bar x_T$ have no transpose; the factor inside the large parentheses of the first line has one.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E010-b.png"}, {"alt": "Crop of the heading 'B.2. Proof of Theorem 3.2' and the first three lines of the proof. A red box marks the factor (x-bar_C minus x-bar_T), printed without a prime, in the expectation E[(x-bar_C minus x-bar_T)(nu_1 + nu_2 + nu_3)].", "caption": "Proof of Theorem 3.2, p. 1515 (PDF p. 14), first sentence. The marked factor has no transpose; item 1, two lines below, writes $(\\bar x_C-\\bar x_T)'\\nu_1$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E010-c.png"}, {"alt": "Crop of Remark 2 with one line of text above and one below. Red boxes mark the symbol v_2, in a Latin letter, in both of its occurrences: in 'The bias in E[(x-bar_C minus x-bar_T) prime v_2] has been discussed previously' and in '(x-bar_C minus x-bar_T) prime v_2 contains no unknowns'.", "caption": "Remark 2, p. 1507 (PDF p. 6). Both marked symbols are a Latin $v_2$; Theorem 3.1, above the remark, defines the Greek $\\nu_2$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E010-d.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T10", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T10"}, {"id": "P005-T11", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T11"}, {"id": "P005-T14", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T14"}], "headline": "Six covariate means lack a transpose, one control-arm sum runs over $[T]$ where $[C]$ is meant, and Remark 2 prints $v_2$ for $\\nu_2$.", "id": "P005-E010", "location": "Appendix B.2, proof of Theorem 3.2, p. 1516 (PDF p. 15), display for $\\widehat{\\mathrm{ATE}}_{\\mathrm{Debiased}}$; Appendix B.4, p. 1518 (PDF p. 17), last display; p. 1515 (PDF p. 14), first sentence of the proof of Theorem 3.2; Remark 2, p. 1507 (PDF p. 6)", "pdf_page": 15, "printed_page": "1516", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T10", "P005-T11", "P005-T14"], "text": {"as_printed": "P.~1516, first and third lines of the display:\n\\[\n\\begin{aligned}\n\\widehat{\\mathrm{ATE}}_{\\mathrm{Debiased}}&=\\bigl(\\overline{y(1)}_T-\\overline{y(0)}_C-\\hl{(\\bar x_T-\\bar x_C)}\\widehat L^{-1}\\hat N\\bigr)-(\\bar x_C-\\bar x_T)'\\bigl(\\widehat L^{-1}-I_d^{-1}\\bigr)\\hat N+\\cdots\\\\\n&=\\frac1{n_T}\\sum_{i\\in[T]}\\bigl(y_i(1)-x_i'N\\bigr)-\\frac1{n_C}\\sum_{\\hl{i\\in[T]}}\\bigl(y_i(0)-x_i'N\\bigr)-\\cdots\n\\end{aligned}\n\\]\n\nP.~1518, last display:\n\\[\n\\begin{aligned}\n\\widehat{\\mathrm{ATE}}_{T,I,\\mathrm{Debiased}}&=\\overline{y(1)}_T-\\hl{\\bar x_T}\\widehat L_T^{-1}\\hat N_T-\\Bigl(-\\bar x_T'\\bigl(\\widehat L_T^{-1}-I_d^{-1}\\bigr)\\hat N_T-\\frac{n_T}{(n-2)n_C}\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\Bigr)\\\\\n&=\\overline{y(1)}_T-\\hl{\\bar x_T}N_T-\\hl{\\bar x_T}(\\hat N_T-N_T)+\\frac{n_T}{(n-2)n_C}\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\\\\n&=\\overline{y(1)}_T-\\hl{\\bar x_T}N_T+O_p\\Bigl(\\sqrt{\\frac\\kappa n}\\Bigr),\n\\end{aligned}\n\\]\n\nP.~1515: ``We first propose estimators for the bias $\\mathbb E[\\hl{(\\bar x_C-\\bar x_T)}(\\nu_1+\\nu_2+\\nu_3)]$.''\n\nP.~1507, Remark 2: ``The bias in $E[(\\bar x_C-\\bar x_T)'\\hl{v_2}]$ has been discussed previously. $(\\bar x_C-\\bar x_T)'\\hl{v_2}$ contains no unknowns and hence can be subtracted directly for debiasing purposes.''", "corrected": "P.~1516, first and third lines of the display:\n\\[\n\\begin{aligned}\n\\widehat{\\mathrm{ATE}}_{\\mathrm{Debiased}}&=\\bigl(\\overline{y(1)}_T-\\overline{y(0)}_C-\\hl{(\\bar x_T-\\bar x_C)'}\\widehat L^{-1}\\hat N\\bigr)-(\\bar x_C-\\bar x_T)'\\bigl(\\widehat L^{-1}-I_d^{-1}\\bigr)\\hat N+\\cdots\\\\\n&=\\frac1{n_T}\\sum_{i\\in[T]}\\bigl(y_i(1)-x_i'N\\bigr)-\\frac1{n_C}\\sum_{\\hl{i\\in[C]}}\\bigl(y_i(0)-x_i'N\\bigr)-\\cdots\n\\end{aligned}\n\\]\n\nP.~1518, last display: write $\\hl{\\bar x_T'}$ for $\\bar x_T$ before $\\widehat L_T^{-1}\\hat N_T$ in the first line, before $N_T$ and before $(\\hat N_T-N_T)$ in the second line, and before $N_T$ in the third line. The display with these transposes and with the corrected weights is given under \\fid{P005-E013}.\n\nP.~1515: ``We first propose estimators for the bias $\\mathbb E[\\hl{(\\bar x_C-\\bar x_T)'}(\\nu_1+\\nu_2+\\nu_3)]$.''\n\nP.~1507, Remark 2: ``The bias in $E[(\\bar x_C-\\bar x_T)'\\hl{\\nu_2}]$ has been discussed previously. $(\\bar x_C-\\bar x_T)'\\hl{\\nu_2}$ contains no unknowns and hence can be subtracted directly for debiasing purposes.''", "evidence": "The evidence is a check of the pages against Section 2 and the statements of Theorems 3.1, 3.2 and 4.2. With the transposes restored, the inverse terms on p.~1516 cancel:\n\\[\n-(\\bar x_T-\\bar x_C)'\\widehat L^{-1}\\hat N-(\\bar x_C-\\bar x_T)'\\bigl(\\widehat L^{-1}-I_d^{-1}\\bigr)\\hat N=-(\\bar x_T-\\bar x_C)'\\hat N.\n\\]\nFurther, $\\overline{y(0)}_C-\\bar x_C'N=n_C^{-1}\\sum_{i\\in[C]}(y_i(0)-x_i'N)$.", "problem": "Since the covariates $x_i$ are column vectors (Section 2), each of these scalar adjustments needs the transpose of the covariate mean. The same expressions carry it in Theorems 3.1, 3.2 and 4.2. In the second equality on p.~1516, the sum with factor $1/n_C$ expands $\\overline{y(0)}_C-\\bar x_C'N$ and must run over the control group. Remark 2 refers to the component $\\nu_2$ that Theorem 3.1 defines immediately above it.", "what_survives": "No result changes. The weights in the last display on p.~1518 are a separate misprint, corrected under \\fid{P005-E013}."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E010", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of two sentences with their displays: 'Thus, an unbiased estimator for E[minus x-bar_T prime (nu_1T + nu_2T + nu_3T)] is' with the treated-arm estimator, and 'Similarly, an unbiased estimator for E[x-bar_C prime (nu_1C + nu_2C + nu_3C)] is' with the control-arm estimator. A red box marks L-hat_T inverse in the control-arm display, in the factor (L-hat_T inverse minus I_d inverse) before N-hat_C.", "caption": "Proof of Theorem 4.2, p. 1518 (PDF p. 17). The marked inverse in the control-arm estimator carries the subscript $T$; the treated-arm estimator above it is correct.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E012.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T12", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T12"}, {"id": "P005-T13", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T13"}], "headline": "The estimator of the control-arm bias in the proof of Theorem 4.2 should have $\\widehat L_C^{-1}$, not $\\widehat L_T^{-1}$.", "id": "P005-E012", "location": "Appendix B.4, proof of Theorem 4.2, p. 1518 (PDF p. 17), the display after “Similarly, an unbiased estimator for”", "pdf_page": 17, "printed_page": "1518", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T12", "P005-T13"], "text": {"as_printed": "``Similarly, an unbiased estimator for $\\mathbb E[\\bar x_C'(\\nu_{1C}+\\nu_{2C}+\\nu_{3C})]$ is\n\\[\n\\frac{n_T}{(n-2)n_C}\\bigl(\\overline{hy(0)}_C-\\bar h_C\\overline{y(0)}_C\\bigr)+\\bar x_C'\\bigl(\\widehat L_{\\hl{T}}^{-1}-I_d^{-1}\\bigr)\\hat N_C.\\text{''}\n\\]", "corrected": "``Similarly, an unbiased estimator for $\\mathbb E[\\bar x_C'(\\nu_{1C}+\\nu_{2C}+\\nu_{3C})]$ is\n\\[\n\\frac{n_T}{(n-2)n_C}\\bigl(\\overline{hy(0)}_C-\\bar h_C\\overline{y(0)}_C\\bigr)+\\bar x_C'\\bigl(\\widehat L_{\\hl{C}}^{-1}-I_d^{-1}\\bigr)\\hat N_C.\\text{''}\n\\]", "evidence": "The evidence is a check of the page against Theorems 4.1 and 4.2 and an exact counterexample. Let $n=6$, $n_T=2$, $d=1$, $x=\\sqrt{3/20}\\,(-3,-2,-1,0,1,5)$ and $y(0)=(9,4,1,0,1,25)$, with $h_i=x_i^2$. Then $\\bar x=0$ and $\\overline{x^2}=1$, and $\\widehat L_T$ and $\\widehat L_C$ are invertible for each of the 15 assignments. The control-arm term $\\bar x_C'\\widehat\\beta_{I,C}=\\bar x_C'\\widehat L_C^{-1}\\hat N_C$ of $\\widehat{\\mathrm{ATE}}_I$ has mean $31085815448171/17062566567774\\approx1.822$ over the assignments. Since $\\mathbb E[\\bar x_C]=0$, this mean is the bias $\\mathbb E[\\bar x_C'(\\nu_{1C}+\\nu_{2C}+\\nu_{3C})]$ to be estimated. The expression with $\\widehat L_C^{-1}$ has the same mean. The expression as printed has mean $12010469/846720\\approx14.185$.", "problem": "The display is the control-arm analogue of the treated-arm estimator printed above it. By Theorem 4.1, $\\nu_{2C}=(\\widehat L_C^{-1}-I_d^{-1})\\hat N_C$, so the term that is subtracted directly must use the control-arm matrix $\\widehat L_C$. With $\\widehat L_T^{-1}$ the expression differs from the intended one by $\\bar x_C'(\\widehat L_T^{-1}-\\widehat L_C^{-1})\\hat N_C$. Since the expectation of this difference need not vanish, the expression as printed is not in general unbiased.", "what_survives": "Theorem 4.2 prints $\\widehat L_C^{-1}$ in the control-arm term and is unaffected, as is the definition of $\\nu_{2C}$ in Theorem 4.1. Only this display of the proof changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E012", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the sentence 'We can characterize the stochastic expansion for the debiased estimator of the treated group as' and the three-line display for the treated-arm debiased estimator. Red boxes mark the fraction n_T over (n minus 2) n_C in front of the treated-arm covariance of h and y(1), once in the first line and once in the second line.", "caption": "Proof of Theorem 4.2, p. 1518 (PDF p. 17), last display. The two marked weights have $n_T$ over $n_C$; the weight derived two displays earlier on the same page, and stated in Theorem 4.2, is $n_C/((n-2)n_T)$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E013.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T10", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T10"}, {"id": "P005-T11", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T11"}], "headline": "In the final expansion of the treated-arm debiased estimator, the weight on the treated-arm covariance should be $n_C/((n-2)n_T)$, not $n_T/((n-2)n_C)$.", "id": "P005-E013", "location": "Appendix B.4, proof of Theorem 4.2, p. 1518 (PDF p. 17), last display, first and second lines", "pdf_page": 17, "printed_page": "1518", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T10", "P005-T11"], "text": {"as_printed": "\\[\n\\begin{aligned}\n\\widehat{\\mathrm{ATE}}_{T,I,\\mathrm{Debiased}}&=\\overline{y(1)}_T-\\bar x_T\\widehat L_T^{-1}\\hat N_T-\\Bigl(-\\bar x_T'\\bigl(\\widehat L_T^{-1}-I_d^{-1}\\bigr)\\hat N_T-\\hl{\\frac{n_T}{(n-2)n_C}}\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\Bigr)\\\\\n&=\\overline{y(1)}_T-\\bar x_TN_T-\\bar x_T(\\hat N_T-N_T)+\\hl{\\frac{n_T}{(n-2)n_C}}\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\\\\n&=\\overline{y(1)}_T-\\bar x_TN_T+O_p\\Bigl(\\sqrt{\\frac\\kappa n}\\Bigr),\n\\end{aligned}\n\\]\nTwo displays earlier on the same page: ``Collecting the constants in front of $(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T)$ gives\n\\[\n\\frac{n_C}{n(n_T-1)}-\\frac{(n-n_T)(n-2n_T)}{(n-1)(n-2)n_T^2}\\times\\frac{n_T(n-1)}{(n_T-1)n}=\\frac{n_C}{(n-2)n_T}.\\text{''}\n\\]", "corrected": "With the weight restored, and with the transposes of \\fid{P005-E010}:\n\\[\n\\begin{aligned}\n\\widehat{\\mathrm{ATE}}_{T,I,\\mathrm{Debiased}}&=\\overline{y(1)}_T-\\bar x_T'\\widehat L_T^{-1}\\hat N_T-\\Bigl(-\\bar x_T'\\bigl(\\widehat L_T^{-1}-I_d^{-1}\\bigr)\\hat N_T-\\hl{\\frac{n_C}{(n-2)n_T}}\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\Bigr)\\\\\n&=\\overline{y(1)}_T-\\bar x_T'N_T-\\bar x_T'(\\hat N_T-N_T)+\\hl{\\frac{n_C}{(n-2)n_T}}\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr)\\\\\n&=\\overline{y(1)}_T-\\bar x_T'N_T+O_p\\Bigl(\\sqrt{\\frac\\kappa n}\\Bigr).\n\\end{aligned}\n\\]\nAfter the cancellation, the treated component is\n\\[\n\\overline{y(1)}_T-\\bar x_T'\\hat N_T+\\frac{n_C}{(n-2)n_T}\\bigl(\\overline{hy(1)}_T-\\bar h_T\\overline{y(1)}_T\\bigr),\\qquad\\hat N_T=\\overline{y(1)x}_T-\\overline{y(1)}_T\\bar x_T.\n\\]", "evidence": "The evidence is a recalculation and an exact counterexample. With $N_{\\mathrm{TTT}}$ as defined on p.~1510, the coefficient collected on p.~1518 is\n\\[\n\\frac{n_C}{n(n_T-1)}-N_{\\mathrm{TTT}}\\frac{n_T(n-1)}{(n_T-1)n}=\\frac{n_C}{n(n_T-1)}\\Bigl(1-\\frac{n_C-n_T}{(n-2)n_T}\\Bigr)=\\frac{n_C}{(n-2)n_T}.\n\\]\nFor the counterexample, let $n=6$, $n_T=2$, $d=1$, $x=\\sqrt{3/20}\\,(-3,-2,-1,0,1,5)$, $y(1)=(9,4,1,0,1,25)$ and $y(0)=0$, with $h_i=x_i^2$. Over the 15 assignments, the second line of the display has mean $4913/1200\\approx4.094$ with the printed weight $1/8$, and mean $20/3$ with the weight $1/2$ derived on the page. The population mean $\\overline{y(1)}$ is $20/3$.", "problem": "The display expands the debiased estimator for the treated arm by subtracting the bias estimator derived just above it, whose covariance weight is $n_C/((n-2)n_T)$. Both lines print the weight with $n_T$ and $n_C$ interchanged. The two weights agree only when $n_T=n_C$. In an unbalanced design the expression as printed is not unbiased for $\\overline{y(1)}$.", "what_survives": "Theorem 4.2 states the correct weight, and the coefficient derived two displays earlier is correct. Only the two repetitions of the weight in this display change. The same display also lacks four transposes, listed under \\fid{P005-E010}."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E013", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three lines of text on page 1505, followed by equation (1), Y_i ~ alpha + tau D_i + x_i' beta. The third line reads 'results from a OLS regression:'. A red box marks the words 'a OLS'.", "caption": "p. 1505 (PDF p. 4), the sentence before equation (1). The marked words are “a OLS”.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E014.png"}, {"alt": "Crop from page 1509. It shows the display for the interacted estimator, the sentence 'for the interacted case, where beta-hat I,T and beta-hat I,C are the OLS coefficients on covariates D_i x_i and x_i, repsectively.', and the first three lines of Theorem 4.1. A red box marks the word 'repsectively'.", "caption": "p. 1509 (PDF p. 8), Section 4, the sentence after the display for $\\widehat{\\mathrm{ATE}}_I$. The marked word is “repsectively”.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E014-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Read “an OLS regression” on p. 1505 and “respectively” on p. 1509.", "id": "P005-E014", "location": "p. 1505 (PDF p. 4), the sentence before equation (1); p. 1509 (PDF p. 8), Section 4, the sentence after the display for $\\widehat{\\mathrm{ATE}}_I$", "pdf_page": 4, "printed_page": "1505", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~1505: ``The first estimator results from \\hl{a} OLS regression:''\n\np.~1509: ``for the interacted case, where $\\widehat\\beta_{I,T}$ and $\\widehat\\beta_{I,C}$ are the OLS coefficients on covariates $D_ix_i$ and $x_i$, \\hl{repsectively}.''", "corrected": "p.~1505: ``The first estimator results from \\hl{an} OLS regression:''\n\np.~1509: ``\\ldots\\ on covariates $D_ix_i$ and $x_i$, \\hl{respectively}.''", "evidence": "The evidence is a check of the two pages.", "problem": "The indefinite article before ``OLS'' should be ``an'', and ``repsectively'' is a misspelling of ``respectively''.", "what_survives": "Nothing else changes. The description of the two coefficients in the second sentence is the subject of \\fid{P005-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E014", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 606 to 614 of an R function in monospaced type with line numbers. A red box marks the name unbiased.reg.interact_new_formula in line 612, which assigns to debiased_estimator the value of that function at (y,d,z). Line 608 fits lm of y on d times z, and line 614 replaces the coefficient on d by debiased_estimator.", "caption": "Replication files, 30_functions2.R, lines 606–614, typeset from the source file with line numbers added. The marked name is the function called at line 612; the replication files define no function of this name.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E015.png"}, {"alt": "Listing of lines 345 to 350 in monospaced type with line numbers. A red box marks the name unbiased.reg.interact_appl_new_formula at the start of line 345, where it is assigned a function of y, d and z.", "caption": "Replication files, 30_functions2.R, lines 345–350, typeset from the source file with line numbers added. The marked name is the function that the file defines.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E015-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The function interacted_unbiased_regression calls a function under the name unbiased.reg.interact_new_formula, which the replication files do not define; the name they define is unbiased.reg.interact_appl_new_formula.", "id": "P005-E015", "location": "Replication files (public repository Debias, commit c9cfb0c1d006), folder 1_main_functions_for_simulations, file 30_functions2.R, function interacted_unbiased_regression (lines 593–658), line 612. No passage of the article", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Line 612 of \\texttt{30\\_functions2.R}:\n\n\\texttt{debiased\\_estimator= \\hl{unbiased.reg.interact\\_new\\_formula}(y,d,z) \\#debiased estimator}", "corrected": "Line 612 of \\texttt{30\\_functions2.R}:\n\n\\texttt{debiased\\_estimator= unbiased.reg.interact\\hl{\\_appl}\\_new\\_formula(y,d,z) \\#debiased estimator}", "evidence": "This finding rests on a run of the unmodified file and on a search of the replication files. In a new R session I called the function on eight units with two covariates, for which the interacted regression has full rank. The call stops with the message \\texttt{could not find function} followed by the name \\texttt{unbiased.reg.interact\\_new\\_formula}. The name occurs once in the text of the replication files, at line 612, and none of the 43 files of saved R objects contains a function of that name.", "problem": "Line 612 calls a function that the replication files do not define. The function that computes the bias-corrected estimator of Theorem 4.2 is defined at line 345 of the same file under the name \\texttt{unbiased.reg.interact\\_appl\\_new\\_formula}, and line 467 calls it by that name. When the file is loaded on its own, \\texttt{interacted\\_unbiased\\_regression} stops at line 612 and returns no estimate, standard error or interval.", "what_survives": "The article is not affected. It reports no simulation or application, and its estimators, theorems and proofs do not involve this function. I found no direct call to \\texttt{interacted\\_unbiased\\_regression} in the replication scripts, and the simulation code calls \\texttt{unbiased.reg.interact\\_appl\\_new\\_formula} directly. The change below corrects the name only. With more than one covariate the function then stops at the error described in \\fid{P005-E016}."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E015", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 593 to 606 in monospaced type with line numbers: the header of the function interacted_unbiased_regression, comments that describe its arguments, and three assignments. A red box marks line 603, 'd=as.matrix(d)'. Lines 602 and 604 apply as.matrix to y and to z.", "caption": "Replication files, 30_functions2.R, lines 593–606, typeset from the source file with line numbers added. The marked statement at line 603 converts the assignment to a matrix; the comment at line 597 describes d as an assignment vector.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E016.png"}, {"alt": "Listing of lines 358 to 370 in monospaced type with line numbers. Line 358 assigns to z.tr the columns of z minus their means. A red box marks the product 'd*z.tr' inside the formula of the regression at line 370, lm of y on d plus z.tr plus I(d*z.tr). The lines between are comments and two constants, CA and CB.", "caption": "Replication files, 30_functions2.R, lines 358–370, typeset from the source file with line numbers added. The marked product at line 370 multiplies the assignment by the centered covariate matrix z.tr defined at line 358.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-E016-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The function interacted_unbiased_regression converts the assignment d to an $n\\times1$ matrix, and with more than one covariate the product d*z.tr at line 370 then fails; d should remain a numeric vector.", "id": "P005-E016", "location": "Replication files (public repository Debias, commit c9cfb0c1d006), folder 1_main_functions_for_simulations, file 30_functions2.R: function interacted_unbiased_regression, lines 602–604 and 612; function unbiased.reg.interact_appl_new_formula, lines 358 and 370. No passage of the article", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Lines 602--604 of \\texttt{30\\_functions2.R}, in \\texttt{interacted\\_unbiased\\_regression}:\n\n\\texttt{y=as.matrix(y)}\n\n\\texttt{d=\\hl{as.matrix}(d)}\n\n\\texttt{z=as.matrix(z)}\n\nLines 358 and 370, in \\texttt{unbiased.reg.interact\\_appl\\_new\\_formula}:\n\n\\texttt{z.tr <- apply(z,2,function(y) y - mean(y))}\n\n\\texttt{ols<-coef(ols.model<-lm(y$\\sim$d+z.tr+I(d*z.tr)))[2]}", "corrected": "Line 603 of \\texttt{30\\_functions2.R}:\n\n\\texttt{d=\\hl{as.numeric}(d)}\n\nLines 602 and 604 are unchanged. Line 612 also needs the name given in \\fid{P005-E015}.", "evidence": "This finding rests on a run of the unmodified file on twelve units with two covariates and alternating assignment, for which both arm regressions have full rank. With the missing name defined in the session as in \\fid{P005-E015} and the file unchanged, the call of \\texttt{interacted\\_unbiased\\_regression} stops with \\texttt{non-conformable arrays}. The function \\texttt{unbiased.reg.interact\\_appl\\_new\\_formula} stops with the same message when called with \\texttt{as.matrix(d)}, and returns $2.39214301337333$ when called with the vector \\texttt{d}.", "problem": "After line 603, \\texttt{d} is an $n\\times1$ matrix. Line 612 passes it to the function that computes the bias-corrected estimate, under the name corrected in \\fid{P005-E015}. That function centers the $k$ covariates in the $n\\times k$ matrix \\texttt{z.tr} and fits a regression whose formula contains the elementwise product \\texttt{d*z.tr}. For $k>1$, R does not multiply an $n\\times1$ matrix by an $n\\times k$ matrix elementwise, and the call stops with the error \\texttt{non-conformable arrays}. This failure is separate from the undefined name and appears once the name is corrected.", "what_survives": "The article is not affected. The formulas of the estimators and the function \\texttt{unbiased.reg.interact\\_appl\\_new\\_formula}, called with a vector \\texttt{d}, are unchanged. I found no direct call to \\texttt{interacted\\_unbiased\\_regression} in the replication scripts, and the failure is an error message, not a wrong estimate. I did not test whether the function runs to its end once both changes are made."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-E016", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Assumption 1 on page 1505. The first line reads 'Assumption 1: For all n, there exists a finite constant K such that'. Three displayed bounds follow, each an average of squares that is at most K, and then the line 'for all k = 1, ..., d and d <= n.' A red box marks the words 'For all n, there exists a finite constant K'.", "caption": "Assumption 1, p. 1505 (PDF p. 4). The marked words give the order of the quantifiers.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "In Assumption 1 the constant $K$ is one bound for all $n$; the printed order of the quantifiers lets it depend on $n$.", "id": "P005-N002", "location": "Assumption 1, p. 1505 (PDF p. 4), opening words", "pdf_page": 4, "printed_page": "1505", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``ASSUMPTION 1: \\hl{For all $n$, there exists a finite constant $K$} such that\n\\[\n\\frac1n\\sum_{i=1}^n y_i(1)^2\\le K,\\qquad \\frac1n\\sum_{i=1}^n y_i(0)^2\\le K,\\qquad \\frac1n\\sum_{i=1}^n x_{ik}^2\\le K,\n\\]\nfor all $k=1,\\ldots,d$ and $d\\le n$.''", "corrected": "``ASSUMPTION 1: \\hl{There exists a finite constant $K$ such that, for all $n$,}\n\\[\n\\frac1n\\sum_{i=1}^n y_i(1)^2\\le K,\\qquad \\frac1n\\sum_{i=1}^n y_i(0)^2\\le K,\\qquad \\frac1n\\sum_{i=1}^n x_{ik}^2\\le K,\n\\]\nfor all $k=1,\\ldots,d$ and $d\\le n$.''", "intended_meaning": "The intended statement is that there exists a single finite constant $K$ that bounds the three second moments for every $n$. This is the standard form of the assumption and the form that the proofs use. For example, the bounds $n^{-1}\\sum_{i=1}^n e_i^2(1)=O(1)$ and $n^{-1}\\sum_{i=1}^n e_i^2(0)=O(1)$ on p.~1513 need it. Read literally, the printed order lets $K$ depend on $n$, and the condition then holds for every sequence of finite populations. For instance, let $n=4m$, and let the population consist of the four units with $x=(-1,-1,1,1)$ and $q=(-1,1,-1,1)$, repeated $m$ times, with $y(1)=nq$, $y(0)=0$ and $p_T=1/2$. Assumptions 2 and 3 hold, the printed Assumption 1 holds with $K=n^2$, and the second moment of the residuals $e_i(1)$ is $n^2$. Assumption 3, on the same page, places its constants before the words ``for all $n$''. The asymptotic arguments of the paper are made under the uniform bound. The finite-sample identities for the bias and for unbiasedness do not need a bound that is uniform in $n$."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-N002", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of page 1510. The first line reads 'If kappa log d = o(1),' with the footnote mark 8. The display below gives ATE-hat_I as ATE plus the treated-arm average of y_i(1) minus its mean minus x_i' beta_{I,T}, minus the corresponding control-arm average, and its second line is 'plus O_p of the square root of kappa d over n'. Red boxes mark the condition 'kappa log d = o(1)' and the remainder term.", "caption": "Theorem 4.1, continued, p. 1510 (PDF p. 9). The marked terms are the condition $\\kappa\\log d=o(1)$ and the remainder $O_p(\\sqrt{\\kappa d/n})$.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-N004.png"}, {"alt": "Crop of section B.3, Proof of Theorem 4.1, on page 1517. The section has three sentences: 'The decomposition of the bias is similar to the one in Theorem 3.1. We omit the details. The stochastic order of the bias is derived in Lei and Ding (2021).' A red box marks the last sentence.", "caption": "Appendix B.3, p. 1517 (PDF p. 16): the proof of Theorem 4.1 in full. The marked sentence refers the stochastic order to Lei and Ding (2021).", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-N004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T9", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T9"}], "headline": "The expansion of Theorem 4.1, which the paper cites from Lei and Ding (2021) without proof, holds with the stated remainder for $d\\ge2$ under the printed condition, and for $d=1$ only if $\\kappa=o(1)$ is added.", "id": "P005-N004", "location": "Theorem 4.1, p. 1510 (PDF p. 9), the expansion with remainder $O_p(\\sqrt{\\kappa d/n})$ and footnote 8; its proof, Appendix B.3, p. 1517 (PDF p. 16)", "pdf_page": 9, "printed_page": "1510", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T9"], "text": {"as_printed": "Theorem 4.1, p.~1510: ``If $\\kappa\\hl{\\log d}=o(1)$,'' with footnote 8 attached, followed by the display\n\\[\n\\widehat{\\mathrm{ATE}}_I=\\mathrm{ATE}+\\frac{1}{n_T}\\sum_{i\\in[T]}\\bigl(y_i(1)-\\overline{y(1)}-x_i'\\beta_{I,T}\\bigr)-\\frac{1}{n_C}\\sum_{i\\in[C]}\\bigl(y_i(0)-\\overline{y(0)}-x_i'\\beta_{I,C}\\bigr)+O_p\\Bigl(\\sqrt{\\frac{\\kappa d}{n}}\\Bigr).\n\\]\nFootnote 8: ``The upper bound on the stochastic order of the bias of the interacted case is characterized in equation (15) in Lei and Ding (2021).''\n\nAppendix B.3, p.~1517: ``The decomposition of the bias is similar to the one in Theorem 3.1. We omit the details. The stochastic order of the bias is derived in Lei and Ding (2021).''", "corrected": "Theorem 4.1, last paragraph. Let Assumptions 1--3 hold, and let all design matrices be invertible, as footnote 4 assumes. If $\\kappa\\hl{\\log\\max\\{d,2\\}}=o(1)$, then\n\\[\n\\widehat{\\mathrm{ATE}}_I=\\mathrm{ATE}+\\frac{1}{n_T}\\sum_{i\\in[T]}\\bigl(y_i(1)-\\overline{y(1)}-x_i'\\beta_{I,T}\\bigr)-\\frac{1}{n_C}\\sum_{i\\in[C]}\\bigl(y_i(0)-\\overline{y(0)}-x_i'\\beta_{I,C}\\bigr)+O_p\\Bigl(\\sqrt{\\frac{\\kappa d}{n}}\\Bigr).\n\\]\nThis is the statement given under \\fid{P005-E001}. The condition $\\kappa\\log(2d)=o(1)$, under which the derivation above is written, is equivalent to it: both reduce to the printed condition for $d\\ge2$ and to $\\kappa=o(1)$ for $d=1$. The reference to Lei and Ding (2021) in Appendix B.3 and in footnote 8 can stay, with the condition as corrected.", "evidence": "The evidence is a derivation and an exact counterexample. For the derivation, suppose that Assumptions 1--3 hold and that $\\kappa\\log(2d)=o(1)$. Let $S$ be an arm of size $m$, and define $G_S=m^{-1}\\sum_{i\\in S}x_ix_i'$. Note that $0\\preceq x_ix_i'\\preceq n\\kappa I_d$ and $\\overline{xx'}=I_d$. By a Chernoff bound for sums of positive semidefinite matrices, which also holds for sampling without replacement, I have\n\\[\nP\\bigl\\{\\lambda_{\\min}(G_S)\\le\\tfrac12\\bigr\\}\\le d\\exp\\Bigl(-\\frac{m/n}{8\\kappa}\\Bigr),\n\\]\nand the right-hand side tends to zero, because $\\kappa\\log(2d)=o(1)$ and $m/n$ is bounded below by Assumption 3. Since $E\\|\\bar x_S\\|^2=O(d/n)$ and $d/n\\le\\kappa$, I have $\\widehat L_S=G_S-\\bar x_S\\bar x_S'\\succeq I_d/4$ with probability tending to one. Let $y$ be the potential outcome of the arm, $\\beta=\\overline{xy}$ and $e_i=y_i-\\bar y-x_i'\\beta$, so that $\\bar e=0$, $\\overline{xe}=0$ and $\\overline{e^2}\\le K$. The fitted intercept of the arm is\n\\[\n\\bar y+\\bar e_S-\\bar x_S'\\widehat L_S^{-1}\\bigl(\\overline{xe}_S-\\bar x_S\\bar e_S\\bigr),\n\\]\nwith $\\|\\bar x_S\\|=O_p(\\sqrt{d/n})$, $\\bar e_S=O_p(n^{-1/2})$ and $\\|\\overline{xe}_S\\|=O_p(\\sqrt\\kappa)$. The last term is $O_p(\\sqrt{\\kappa d/n})+O_p(d/n^{3/2})=O_p(\\sqrt{\\kappa d/n})$. Since $\\widehat{\\mathrm{ATE}}_I$ is the difference of the two fitted intercepts, I obtain the expansion. For $d\\ge2$, the inequality $\\log(2d)\\le2\\log d$ shows that the printed condition suffices.\n\nThe counterexample for $d=1$ is a second sequence, different from the one in \\fid{P005-E001}. Let $r=2,4,6,\\ldots$, $n=r^2+1$ and $\\ell=(n-1)/2$. One unit has the covariate value $n-1$ before scaling and both potential outcomes equal to $0$. The other $2\\ell$ units have the labels $t_i\\in\\{\\pm1/\\ell,\\pm2/\\ell,\\ldots,\\pm1\\}$, the covariate values $-1+\\epsilon t_i$ before scaling and the potential outcomes $y_i(1)=y_i(0)=t_i$. Here $\\epsilon=2rn^{-3}/(1-qn^{-6})$ with $q=(\\ell+1)(2\\ell+1)/(3n\\ell)$, so that $\\epsilon$ is asymptotic to $2n^{-5/2}$. The arm sizes are $n_T=\\ell$ and $n_C=\\ell+1$. Assumptions 1--3 hold with $K=1$, every regression is invertible, and $\\kappa\\to1$. Under every assignment the remainder satisfies $|R_n|\\ge1/\\epsilon-6$. At $n=5$, where $\\epsilon=1000/31249$, the bound is $25249/1000$, and enumeration of the 10 assignments agrees. At $n=17$ the bound is $191213957/314432$, and enumeration of the 24,310 assignments agrees.\n\nThe identities and inequalities that the proofs take from the other cited results were derived directly: the mean and the variance of the quadratic forms under sampling without replacement that are used on pp.~1511, 1514 and 1517, the mean identities behind equation (5), and the leverage inequality on p.~1517. I did not compare these with the cited papers, and I do not confirm the wording or the numbering of the cited results.", "problem": "The paper gives no proof of this expansion. Appendix B.3 omits the details and refers to Lei and Ding (2021), and footnote 8 does the same. The question is whether the cited order holds under Assumptions 1--3 with the condition $\\kappa\\log d=o(1)$. It holds for $d\\ge2$. It does not hold for $d=1$, where the condition is empty. This is the defect described in \\fid{P005-E001}, and it is the only failure of this expansion that I found. The proofs also rely on results from other papers: Lemma A.5 and (B.6) of Lei and Ding (2021), Proposition 1 of Freedman (2008b) and Proposition 1 of Wu and Ding (2021). I found no failure in the use of these.", "what_survives": "The algebraic decomposition of the bias in Theorem 4.1, the expansion for $d\\ge2$ under the printed condition, and the exact unbiasedness of the two corrections are unaffected. The asymptotic statements for the debiased estimators already assume $\\kappa=o(1)$, which excludes the example. What changes is the condition for $d=1$, as in \\fid{P005-E001}. This entry adds no second error."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-N004", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the sentence 'We bound the stochastic order of (*):' and the three-line display below it. Each line is one over p_C times x-bar_T prime times a bracketed sum over the treated group. A red box marks the small closing parenthesis at the very end of the second line, after the two large closing parentheses.", "caption": "Appendix B.1.1, p. 1514 (PDF p. 13), the three-line display that rewrites the term $(*)$. The marked parenthesis at the end of the second line closes nothing.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-N005.png"}], "formal_release": null, "formal_statements": [{"id": "P005-T18", "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-T18"}], "headline": "The second line of the display that rewrites the term $(*)$ has one closing parenthesis too many.", "id": "P005-N005", "location": "Appendix B.1.1, p. 1514 (PDF p. 13), first display, second line", "pdf_page": 13, "printed_page": "1514", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P005-T18"], "text": {"as_printed": "\\[\n=\\frac1{p_C}\\bar x_T'\\Bigl(\\frac1n\\sum_{i\\in[T]}\\Bigl(x_i\\bigl(e_i(1)-e_i(0)\\bigr)+\\frac1{p_T}\\frac1n\\sum_{i=1}^nx_ie_i(0)\\Bigr)\\Bigr)\\hl{)}\n\\]", "corrected": "The second line, with its last parenthesis deleted:\n\\[\n=\\frac1{p_C}\\bar x_T'\\Bigl(\\frac1n\\sum_{i\\in[T]}\\Bigl(x_i\\bigl(e_i(1)-e_i(0)\\bigr)+\\frac1{p_T}\\frac1n\\sum_{i=1}^nx_ie_i(0)\\Bigr)\\Bigr)\\del{)}\n\\]", "evidence": "The evidence is a check of the page and a derivation. Let $\\delta_i=e_i(1)-e_i(0)$, $A=n^{-1}\\sum_{i=1}^nx_i\\delta_i$, $B=n^{-1}\\sum_{i=1}^nx_ie_i(0)$ and $U=n^{-1}\\sum_{i\\in[T]}x_i\\delta_i$. The relation $p_T\\sum_ix_ie_i(1)+p_C\\sum_ix_ie_i(0)=0$, quoted below the display, gives $B=-p_TA$. The three bracketed expressions are $U+B$, $n^{-1}\\sum_{i\\in[T]}(x_i\\delta_i+B/p_T)$ and $n^{-1}\\sum_{i\\in[T]}(x_i\\delta_i-A)$. The last two equal $U+B$ because $n_T/n=p_T$.", "problem": "The line closes one parenthesis more than it opens. Nothing else in the display is wrong. On the second and third lines the population average stands inside the parenthesis that opens after the sum over $i\\in[T]$. It is added $n_T$ times and multiplied by $1/n$, which supplies the factor $p_T$.", "what_survives": "The algebra for $(*)$, its expectation and its stated order are unchanged, and no factor needs to be added."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-N005", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the running head of page 1519 and the two lines that end the proof of Theorem 4.2: 'using the stochastic order estimates we derived in the proof of Theorem 4.1 and the fact that kappa is in [d/n, 1].' A red box marks 'Theorem 4.1'.", "caption": "End of the proof of Theorem 4.2, first line of p. 1519 (PDF p. 18). The marked reference points to Theorem 4.1, whose proof derives no stochastic order estimates.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The stochastic order estimates used at the end of the proof of Theorem 4.2 are derived in the proof of Theorem 3.2, not of Theorem 4.1.", "id": "P005-N007", "location": "Appendix B.4, end of the proof of Theorem 4.2, first line of p. 1519 (PDF p. 18)", "pdf_page": 18, "printed_page": "1519", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``using the stochastic order estimates we derived in the proof of \\hl{Theorem 4.1} and the fact that $\\kappa\\in[\\frac dn,1]$.''", "corrected": "``using the stochastic order estimates we derived in the proof of \\hl{Theorem 3.2} and the fact that $\\kappa\\in[\\frac dn,1]$.''", "evidence": "The evidence is a check of pp.~1516--1519.", "problem": "The proof of Theorem 4.1 (Appendix B.3, p.~1517) is three sentences long. It omits the details and refers to Lei and Ding (2021) for the stochastic order of the bias; it derives no estimates. The estimates used here are items 1--3 of the proof of Theorem 3.2 (Appendix B.2, pp.~1516--1517).", "what_survives": "The estimates and the step that uses them are unaffected, and only the reference changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-N007", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three entries of the reference list: Tan (2014), the entry 'Tibshirani, Robert J., and Bradley Efron (1993): An Introduction to the Bootstrap, Monographs on statistics and applied probability, 57, 1-436. [1504]', and White (1980). A red box marks the author heading 'Tibshirani, Robert J., and Bradley Efron'.", "caption": "Reference list, p. 1519 (PDF p. 18). The marked heading gives Tibshirani before Efron; the entry stands between Tan and White.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-N008.png"}, {"alt": "Crop of four lines of the Introduction, at the start of the paragraph on the practice of debiasing estimators. A red box marks the citation 'Tibshirani and Efron (1993)' in the sentence 'Tibshirani and Efron (1993) have warned that the bias correction could have costs in practice due to its high variability in finite samples.'", "caption": "Introduction, p. 1504 (PDF p. 3). The marked citation gives the two authors in the same reversed order.", "url": "https://pmaronow.github.io/llm-audit/works/P005/exhibits/P005-N008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The authors of An Introduction to the Bootstrap are Efron and Tibshirani, in that order; the citation and the reference entry reverse them.", "id": "P005-N008", "location": "References, p. 1519 (PDF p. 18), the entry for An Introduction to the Bootstrap; cited in the Introduction, p. 1504 (PDF p. 3)", "pdf_page": 18, "printed_page": "1519", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~1519, reference list: \\hl{TIBSHIRANI, ROBERT J., AND BRADLEY EFRON} (1993): ``An Introduction to the Bootstrap,'' \\emph{Monographs on statistics and applied probability}, 57, 1--436. [1504]\n\nP.~1504: ``\\hl{Tibshirani and Efron} (1993) have warned that the bias correction could have costs in practice due to its high variability in finite samples.''", "corrected": "P.~1519, reference list, between the entries for Deaton and Cartwright (2018) and for Finucan, Galbraith, and Stone (1974): \\hl{EFRON, BRADLEY, AND ROBERT J. TIBSHIRANI} (1993): ``An Introduction to the Bootstrap,'' \\emph{Monographs on statistics and applied probability}, 57, 1--436. [1504]\n\nP.~1504: ``\\hl{Efron and Tibshirani} (1993) have warned that the bias correction could have costs in practice due to its high variability in finite samples.''", "evidence": "The evidence is a check of pp.~1504 and 1519 and the WorldCat catalogue listing of the book (OCLC 481704221), which gives Bradley Efron before Robert J. Tibshirani. I read the listing at \\url{https://search.worldcat.org/title/An-introduction-to-the-bootstrap/oclc/481704221}. The catalogue does not show whether the book contains the warning that the Introduction attributes to it; I did not check that.", "problem": "The book \\emph{An Introduction to the Bootstrap} (Chapman \\& Hall, 1993, Monographs on Statistics and Applied Probability 57) is by Bradley Efron and Robert J. Tibshirani, in that order. The reference entry and the citation in the Introduction give the two authors in the reverse order. With the order corrected, the entry belongs between Deaton and Finucan in the alphabetical list.", "what_survives": "The book is identified by its title and year, and nothing in the argument depends on the order of its authors. Only the citation and the reference list change."}, "url": "https://pmaronow.github.io/llm-audit/works/P005/#P005-N008", "work": {"arxiv_id": "2110.08425", "doi": "10.3982/ECTA20289", "title": "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials", "url": "https://pmaronow.github.io/llm-audit/works/P005/"}, "work_id": "P005"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 6 (ATO Best Response), its displayed rule, and the first display of the proof. One red box marks the two cases of the rule: 1 if x is in the arg max of mu 1 and mu 1 of x is positive, 0 otherwise. A second red box marks the middle expression of the proof's display, 'equals E of E of Y i given T i equals 1 and X i, given T i equals 1', which follows 'E of w ATO'.", "caption": "Appendix B.3, PDF p. 15: Proposition 6 and the first display of its proof. The upper mark is the printed best response to $w_{\\mathrm{ATO}}$. The lower mark is the equality that replaces the expected reward by the mean among the treated units.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001.png"}, {"alt": "Crop of Proposition 7 (ATT Best Response), its displayed rule, and the first two lines of the proof. One red box marks the two cases of the rule: 1 if x is in the arg max of tau and tau of x is positive, 0 otherwise. A second red box marks the right side of the first line of the proof, 'equals E of Y i minus mu hat 0 of X i, given T i equals 1'.", "caption": "Appendix B.3, PDF p. 16: Proposition 7 and the first display of its proof. The upper mark is the printed best response to $w_{\\mathrm{ATT}}$, and the lower mark is the corresponding equality.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-b.png"}, {"alt": "Crop of the left column showing 'Reward Function 1 (ATO)', whose definition has two cases, the average of the treated outcomes when at least one unit is treated and 0 otherwise, followed by the statement of Proposition 1. A red box marks the sentence 'The agent's unconstrained best response is pi of x equals the indicator that x is in the arg max of mu 1 and mu 1 of x is positive.'", "caption": "Section 3, PDF p. 4, left column: Reward Function 1 and the best response stated after it. The second case of the definition sets the reward to $0$ when nobody is treated. The marked sentence gives the rule.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-c.png"}, {"alt": "Crop of the first seven lines of the left column. A red box marks the sentence 'The agent's unconstrained best response is pi of x equals the indicator that x is in the arg max of tau and tau of x is positive.' Below it are the sentence 'The resulting regret is bounded, but still not zero.' and the statement of Proposition 2 (ATT Regret).", "caption": "Section 3, PDF p. 5, left column, top: the best response stated for Reward Function 2 (marked), followed by Proposition 2.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-d.png"}, {"alt": "Crop of six lines of the left column. Two red boxes mark the words 'the agent only treats patients with the single value' in the sentence 'This means that in the best response, the agent only treats patients with the single value x with maximum treatment effect tau of x.'", "caption": "Section 3, PDF p. 5, left column: the sentence that describes the best response to $w_{\\mathrm{ATT}}$ in words.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-e.png"}, {"alt": "Crop of Table 1 with its caption: Utility and regret comparisons for different reward functions. For each reward function w, we report utility V of pi w, regret R of pi w, and the realized treatment rate. The table has the columns Utility, Regret and Treat rate for the Horse Colic dataset and for the Stroke Trial dataset, and five rows. A red box surrounds the first two rows: w ATO with 0.00000, 0.1470, 0.1922 and 0.00004, 0.0251, 0.0001; w ATT with 0.00784, 0.1391, 0.0039 and 0.00013, 0.0250, 0.0001. Below the box are the rows w TT, with 0.14695, 0.000, 0.2431 and 0.02518, 0.000, 0.1872, w TT (no info) and w TT (demographic info).", "caption": "Table 1, PDF p. 9, with its caption. The mark surrounds the rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$, which the caption describes as the utility, the regret and the treatment rate of the best response $\\pi^w$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-f.png"}, {"alt": "Line chart. The horizontal axis is the probability of treating the lower type, from 0 to 1, and the vertical axis is the reward, with labelled ticks from 0.70 to 1.00. A solid blue curve, the expected reward as defined, rises from 3/4 at 0 to a maximum of 1369/1440 at 17/18 and ends at 19/20 at 1. A dashed orange curve, the mean among the treated units, falls from 1 at 0 to 19/20 at 1. At 0 an open orange circle is labelled 'printed rule, value in the printed proofs: 1' and a filled blue circle is labelled 'printed rule, expected reward: 3/4'. A blue diamond marks the maximum and a blue square marks 'treat everyone: 19/20'.", "caption": "The counterexample with two units: the reward as a function of the probability $\\pi(0)$ of treating type $0$, when type $1$ is always treated. The solid curve is the expected reward as defined. The dashed curve is the mean among the treated units, which is the expression in the printed proofs. The printed rule is the point $\\pi(0)=0$, where the printed proofs give $1$ and the expected reward is $3/4$; the maximum of the expected reward, $1369/1440$, is at $\\pi(0)=17/18$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-reward.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T1", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T1"}, {"id": "P006-T2", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T2"}, {"id": "P006-T3", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T3"}, {"id": "P006-T18", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T18"}, {"id": "P006-T26", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T26"}], "headline": "Because the rewards $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ are zero when nobody is treated, their expected value is $[1-(1-q)^n]A/q$, where $q=E[\\pi(X)]$, $A=E[s(X)\\pi(X)]$ and $s$ is $\\mu_1$ or $\\tau$; the rule printed as the agent’s best response (treat only the values of $x$ with the largest positive $s(x)$) need not maximize it, and on the Horse Colic data the best response to $w_{\\mathrm{ATT}}$ keeps 48.03% of the achievable welfare, where the printed rule keeps 5.34%.", "id": "P006-E001", "location": "Section 3, best responses stated after Reward Functions 1 and 2, PDF pp. 4–5; Appendix B.3, Propositions 6 and 7 and the first display of each proof, PDF pp. 15–16; proof of Proposition 2, PDF pp. 17–18; rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ of Table 1 (PDF p. 9) and Table 2 (PDF p. 27)", "pdf_page": 4, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": ["P006-T1", "P006-T18", "P006-T2", "P006-T26", "P006-T3"], "text": {"as_printed": "Section 3, PDF p.~4:\n\\[\nw_{\\mathrm{ATO}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})=\\begin{cases}\\dfrac{\\sum_{i=1}^nY_iT_i^\\pi}{\\sum_{i=1}^nT_i^\\pi} & \\sum_{i=1}^nT_i^\\pi>0,\\\\[1ex] 0 & \\text{otherwise.}\\end{cases}\n\\]\n``The agent's unconstrained best response is \\hl{$\\pi^{w_{\\mathrm{ATO}}}(x)=\\mathbb{1}(x\\in\\arg\\max_x\\mu_1(x)\\text{ and }\\mu_1(x)>0)$}.''\n\nSection 3, PDF pp.~4--5:\n\\[\nw_{\\mathrm{ATT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})=\\begin{cases}\\dfrac{\\sum_{i=1}^n(Y_i-\\hat\\mu_0(X_i))T_i^\\pi}{\\sum_{i=1}^nT_i^\\pi} & \\sum_{i=1}^nT_i^\\pi>0,\\\\[1ex] 0 & \\text{otherwise.}\\end{cases}\n\\]\n``The agent's unconstrained best response is \\hl{$\\pi^{w_{\\mathrm{ATT}}}(x)=\\mathbb{1}(x\\in\\arg\\max_x\\tau(x)\\text{ and }\\tau(x)>0)$}. \\ldots\\ This means that in the best response, \\hl{the agent only treats patients with the single value $x$} with maximum treatment effect $\\tau(x)$.''\n\nAppendix B.3, PDF p.~15: \\textbf{Proposition 6} (ATO Best Response). Suppose $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$, and suppose $X$ is a discrete random variable supported on $\\mathcal{X}$. Then the agent's best response to $w_{\\mathrm{ATO}}$ is:\n\\[\n\\pi^{w_{\\mathrm{ATO}}}(x)=\\hl{\\begin{cases}1 & \\text{if }x\\in\\arg\\max_x\\mu_1(x)\\text{ and }\\mu_1(x)>0\\\\ 0 & \\text{otherwise.}\\end{cases}}\n\\]\n\\emph{Proof.} By the tower property,\n\\[\nE[w_{\\mathrm{ATO}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]\\hl{{}=E[E[Y_i\\mid T_i^\\pi=1,X_i]\\mid T_i^\\pi=1]}=\\int E[Y_i\\mid T_i^\\pi=1,X_i]\\,dP_{X_i\\mid T_i^\\pi=1}.\n\\]\n\nAppendix B.3, PDF p.~16: \\textbf{Proposition 7} (ATT Best Response). Suppose $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$, and suppose $X$ is a discrete random variable supported on $\\mathcal{X}$. Then the agent's best response to $w_{\\mathrm{ATT}}$ is:\n\\[\n\\pi^{w_{\\mathrm{ATT}}}(x)=\\hl{\\begin{cases}1 & \\text{if }x\\in\\arg\\max_x\\tau(x)\\text{ and }\\tau(x)>0\\\\ 0 & \\text{otherwise.}\\end{cases}}\n\\]\n\\emph{Proof.}\n\\[\nE[w_{\\mathrm{ATT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]\\hl{{}=E[Y_i-\\hat\\mu_0(X_i)\\mid T_i^\\pi=1]}\n\\]\n\nTable 1, PDF p.~9: ``For each reward function $w$, we report utility $V(\\pi^w)$, regret $R(\\pi^w)$, and the realized treatment rate $P(T^{\\pi^w}=1)$.'' The rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ are the same in Table 2, PDF p.~27.\n\\begin{audittable}{lrrrrrr}{Tables 1 and 2, first two rows. The first three columns of numbers are headed ``Horse Colic dataset'' and the last three ``Stroke Trial dataset''.}\nReward function & Utility & Regret & Treat rate & Utility & Regret & Treat rate \\\\\n$w_{\\mathrm{ATO}}$ & 0.00000 & 0.1470 & 0.1922 & \\hl{0.00004} & \\hl{0.0251} & \\hl{0.0001} \\\\\n$w_{\\mathrm{ATT}}$ & \\hl{0.00784} & \\hl{0.1391} & \\hl{0.0039} & \\hl{0.00013} & \\hl{0.0250} & \\hl{0.0001}\n\\end{audittable}", "corrected": "\\textbf{Propositions 6 and 7.} Suppose that $\\hl{n\\ge1}$, the units are independent and identically distributed with \\hl{integrable potential outcomes}, and the treatment indicators are drawn independently given the covariates. For $w_{\\mathrm{ATT}}$, suppose also that, for every admissible policy, $\\hat\\mu_0(X_i)$ is \\hl{integrable} and $E[\\hat\\mu_0(X_i)\\mid\\mathbf{X},\\mathbf{T}^\\pi]=\\mu_0(X_i)$, as holds for a pointwise unbiased estimator computed from independent auxiliary data with policies fixed independently of the estimator (this condition is the subject of \\fid{P006-E002}). Let $s=\\mu_1$ for $w_{\\mathrm{ATO}}$ and $s=\\tau$ for $w_{\\mathrm{ATT}}$, and for a policy $\\pi$ write $q=E[\\pi(X)]$ and $A=E[s(X)\\pi(X)]$. Then\n\\[\nE[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=\\begin{cases}{}\\hl{\\bigl[1-(1-q)^n\\bigr]}\\,\\dfrac{A}{q} & q>0,\\\\[1ex] 0 & q=0,\\end{cases}\n\\]\nand a best response, when one exists, is \\hl{a policy in $\\Pi$ that maximizes this expression}. It need not treat only the values of $x$ that maximize $s$.\n\nSuppose in addition that $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$ and that $X$ takes finitely many values, with probabilities $p_1,\\dots,p_K>0$ and with the values of $s$ ordered as $s_1\\ge\\dots\\ge s_K$. Set $Q_j=p_1+\\dots+p_j$ and $B_j=p_1s_1+\\dots+p_js_j$, with $Q_0=B_0=0$, and $h_n(q)=\\sum_{k=0}^{n-1}(1-q)^k$, which equals $[1-(1-q)^n]/q$ for $q>0$. Among the policies with $E[\\pi(X)]=q\\in[Q_{j-1},Q_j]$, the largest value of $A$ is $A_{\\max}(q)=B_{j-1}+s_j(q-Q_{j-1})$. It is attained by the policy that treats the first $j-1$ values, treats the $j$th value with probability $(q-Q_{j-1})/p_j$, and treats no other value. A best response is a policy of this form with $q$ chosen to maximize $h_n(q)A_{\\max}(q)$ over $0\\le q\\le1$.\n\nLet $M=s_1$ and $p=P(s(X)=M)$, and suppose that $M>0$. The rule that treats exactly the values of $x$ with $s(x)=M$ is a best response if and only if every value of $x$ with $s(x)<M$ satisfies\n\\[\ns(x)\\le M\\Bigl[1-\\frac{np(1-p)^{n-1}}{1-(1-p)^n}\\Bigr].\n\\]\nThis condition holds when $s(x)\\le0$ for every such $x$, and for $n=1$ it holds only then. If $M\\le0$, the policy that treats nobody is a best response.\n\nThe printed rule maximizes the mean among the treated units, $A/q$, when the maximum of $s$ is positive and is attained with positive probability. It can be kept as a best-response rule for that explicitly stated alternative objective.\n\n\\textbf{Section 3.} The two sentences beginning ``The agent's unconstrained best response is'' are replaced by the statement above, and the sentence that the agent ``only treats patients with the single value $x$ with maximum treatment effect $\\tau(x)$'' is withdrawn.\n\n\\textbf{Proposition 2} (bound unchanged). \\emph{Proof.} Suppose the conditions stated above for $w_{\\mathrm{ATT}}$. For every policy, $V(\\pi)=E[\\pi(X)\\tau(X)]=A$, so $E[w_{\\mathrm{ATT}}]=h_n(q)V(\\pi)$ with $h_n(q)>0$. The policy $\\pi\\equiv0$ belongs to $\\Pi$ and has expected reward $0$, so a best response $\\pi^{w_{\\mathrm{ATT}}}$ has expected reward at least $0$ and therefore $V(\\pi^{w_{\\mathrm{ATT}}})\\ge0$. Hence $R(\\pi^{w_{\\mathrm{ATT}}})=\\max_{\\pi\\in\\Pi}V(\\pi)-V(\\pi^{w_{\\mathrm{ATT}}})\\le\\max_{\\pi\\in\\Pi}V(\\pi)$.\n\n\\textbf{Tables 1 and 2.} The two rows are computed for the best responses to the rewards as defined, using the fitted values in the double-precision arithmetic of the notebooks, with the $n$ units drawn independently from the empirical distribution of the analyzed cases: $n=255$ for the Horse Colic data, where $n=300$ gives the same row, and $n=7264$ for the Stroke Trial data.\n\\begin{audittable}{lrrrrrr}{Tables 1 and 2, first two rows, for the best responses to the rewards as defined. The first three columns of numbers are for the Horse Colic dataset and the last three for the Stroke Trial dataset.}\nReward function & Utility & Regret & Treat rate & Utility & Regret & Treat rate \\\\\n$w_{\\mathrm{ATO}}$ & 0.00000 & 0.1470 & 0.1922 & \\hl{0.00036} & \\hl{0.0248} & \\hl{0.0007} \\\\\n$w_{\\mathrm{ATT}}$ & \\hl{0.07058} & \\hl{0.0764} & \\hl{0.0353} & \\hl{0.00063} & \\hl{0.0246} & \\hl{0.0007}\n\\end{audittable}\nThe rows as printed report the utility, the regret and the treatment rate of the rule that treats only the units with the largest fitted $\\mu_1$, or the largest fitted $\\mu_1-\\mu_0$. With the corrected rows, the utility for $w_{\\mathrm{ATO}}$ is close to zero for both datasets, as Section 6 states. For $w_{\\mathrm{ATT}}$, the best response keeps 2.50\\% of the achievable welfare on the Stroke Trial data and 48.03\\% on the Horse Colic data, where its regret is $0.0764$ and not $0.1391$.", "evidence": "The evidence is a derivation of the expected reward, an exact counterexample, and a recalculation of the two rows of the tables from the replication files.\n\nLet the units be independent and identically distributed with integrable potential outcomes, and let the treatment indicators be drawn independently given the covariates. For $w_{\\mathrm{ATT}}$, suppose that $\\hat\\mu_0(X_i)$ is integrable and $E[\\hat\\mu_0(X_i)\\mid\\mathbf{X},\\mathbf{T}^\\pi]=\\mu_0(X_i)$. I write $s=\\mu_1$ for $w_{\\mathrm{ATO}}$ and $s=\\tau$ for $w_{\\mathrm{ATT}}$, and set $q=E[\\pi(X)]$, $A=E[s(X)\\pi(X)]$ and $N=\\sum_{i=1}^nT_i^\\pi$. Each unit is treated with probability $q$. By independence, I have $P(N=0)=(1-q)^n$. Let $t\\in\\{0,1\\}^n$ be a treatment vector of positive probability such that at least one unit is treated. Given $\\mathbf{T}^\\pi=t$, the term of each treated unit in the numerator of the reward ($Y_i$, or $Y_i-\\hat\\mu_0(X_i)$) has mean $A/q$. The reward is the average of these terms, so its conditional mean is also $A/q$. The reward is $0$ when $N=0$. Summing over $t$, I obtain\n\\[\nE[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=\\bigl[1-(1-q)^n\\bigr]\\frac{A}{q}\\qquad\\text{for }q>0,\n\\]\nand $E[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=0$ for $q=0$. Note that the first display in each printed proof keeps only $A/q$.\n\nFor the counterexample, I take $n=2$ and two equally likely types $x\\in\\{0,1\\}$, with $Y(0)=0$ and $\\hat\\mu_0=0$, and with $Y(1)=9/10$ for type $0$ and $Y(1)=1$ for type $1$. Then $\\mu_1=\\tau$, the two rewards are equal, and $\\mu_1$ has a unique maximum, which is positive, at $x=1$. The printed rule treats type $1$ only.\n\\begin{audittable}{llrr}{The counterexample with two units. Each pair of types has probability $1/4$.}\nTypes $(X_1,X_2)$ & Treated under the printed rule & Reward, printed rule & Reward, everyone treated \\\\\n$(0,0)$ & nobody & $0$ & $9/10$ \\\\\n$(0,1)$ & unit 2 & $1$ & $19/20$ \\\\\n$(1,0)$ & unit 1 & $1$ & $19/20$ \\\\\n$(1,1)$ & both units & $1$ & $1$ \\\\\nExpected reward & & $3/4$ & $19/20$\n\\end{audittable}\nThe policy that treats everyone has expected reward $19/20$, which exceeds the expected reward $3/4$ of the printed rule by $1/5$. For the printed rule, the right side of the first display of the printed proofs equals $1$, and the left side equals $3/4$.\n\nNeither of these two policies is the best response. With $u=\\pi(0)$ and $v=\\pi(1)$, the formula above gives $E[w]=(9u+10v)(4-u-v)/40$, and\n\\[\n\\frac{1369}{1440}-\\frac{(9u+10v)(4-u-v)}{40}=\\frac{9}{40}\\Bigl(u-\\frac{17}{18}\\Bigr)^2+\\frac{(1-v)(30-19u-10v)}{40}\\ge0\n\\]\nfor $0\\le u,v\\le1$, with equality only at $u=17/18$ and $v=1$. The best response treats type $0$ with probability $17/18$ and type $1$ with probability $1$, and it has expected reward $1369/1440$.\n\nOn a finite support, the best response is found as follows. In the notation of the corrected statement below, let $F(q)=h_n(q)A_{\\max}(q)$ be the largest expected reward among the policies with $E[\\pi(X)]=q$, and let $g(q)=1-(1-q)^n$, so that $F(q)=g(q)A_{\\max}(q)/q$ for $q>0$. On the segment $Q_{j-1}\\le q\\le Q_j$, I have $A_{\\max}(q)=a_j+s_jq$ with $a_j=\\sum_{i<j}p_i(s_i-s_j)\\ge0$, and a direct computation gives\n\\[\nF'(q)=g'(q)\\bigl[s_j-a_jH_n(q)\\bigr],\\qquad H_n(q)=\\frac{g(q)-qg'(q)}{q^2g'(q)}=\\frac{1}{n}\\sum_{l=1}^{n-1}(n-l)(1-q)^{-l},\n\\]\nfor $0<q<1$. The function $H_n$ is nonnegative and nondecreasing, and at $q=Q_j$ the slope of $A_{\\max}$ falls from $s_j$ to $s_{j+1}$. These two facts imply that $F'(q)/g'(q)$ is nonincreasing on $(0,1)$. A value of $q$ at which $F'$ changes sign from positive to negative then maximizes $F$ over $[0,1]$. Let $M=s_1>0$ and $p=P(s(X)=M)$, and, if $p<1$, let $s'$ be the largest value of $s$ below $M$. The printed rule is the policy of this form with $q=p$. On $0<q\\le p$, I have $F(q)=Mg(q)$, which increases with $q$, and for $p<1$ the right derivative of $F$ at $p$ equals\n\\[\n\\frac{g(p)}{p}\\Bigl[s'-M\\bigl(1-r_n(p)\\bigr)\\Bigr],\\qquad r_n(p)=\\frac{pg'(p)}{g(p)}=\\frac{np(1-p)^{n-1}}{1-(1-p)^n}.\n\\]\nFor $p<1$, the printed rule is therefore a best response if and only if $s'\\le M(1-r_n(p))$. If $p=1$, treating everyone is a best response.\n\nIn the replication files, both notebooks compute the row $w_{\\mathrm{ATO}}$ from the indicator that the fitted $\\mu_1$ of a unit equals the largest fitted $\\mu_1$ in the sample, and the row $w_{\\mathrm{ATT}}$ from the indicator that its fitted $\\mu_1-\\mu_0$ equals the largest such difference. For the recalculation, I computed the best responses to the rewards as defined on the same fitted values. The population is the empirical distribution of the cases that the notebook analyzes, with the fitted $\\mu_0$ and $\\mu_1$ of the notebook. There are 255 cases for the Horse Colic data and 7264 patients for the Stroke Trial data. The $n$ units are independent draws from this distribution, with $n$ equal to the number of cases, and $\\Pi$ is the set of all functions from the covariate values to $[0,1]$. For each data set and each reward, I located the change in sign of $F'$ and checked the signs on either side in exact rational arithmetic.\n\\begin{audittable}{llrrrrr}{Rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ of Tables 1 and 2 as printed, and the same rows for the best response to the rewards as defined, rounded as in the tables. The last two columns are computed from unrounded values: the expected number of units treated, and the welfare as a percentage of the achievable welfare. In the rows as printed, these two columns are those of the printed rule in the replication files.}\nData and reward & Row & Utility & Regret & Treat rate & Expected units treated & Welfare kept (\\%) \\\\\nHorse Colic, $w_{\\mathrm{ATO}}$ & as printed & 0.00000 & 0.1470 & 0.1922 & 49 & 0.00 \\\\\n & best response & 0.00000 & 0.1470 & 0.1922 & 49 & 0.00 \\\\\nHorse Colic, $w_{\\mathrm{ATT}}$ & as printed & 0.00784 & 0.1391 & 0.0039 & 1 & 5.34 \\\\\n & best response & 0.07058 & 0.0764 & 0.0353 & 9 & 48.03 \\\\\nStroke Trial, $w_{\\mathrm{ATO}}$ & as printed & 0.00004 & 0.0251 & 0.0001 & 1 & 0.16 \\\\\n & best response & 0.00036 & 0.0248 & 0.0007 & 4.845 & 1.45 \\\\\nStroke Trial, $w_{\\mathrm{ATT}}$ & as printed & 0.00013 & 0.0250 & 0.0001 & 1 & 0.52 \\\\\n & best response & 0.00063 & 0.0246 & 0.0007 & 5 & 2.50\n\\end{audittable}\nOn the Horse Colic data, 49 cases have a fitted $\\mu_1$ equal to $1$ in the double-precision arithmetic of the notebook. The printed rule treats these 49 cases, and the best response to $w_{\\mathrm{ATO}}$ treats the same cases. The nine largest values of the fitted $\\mu_1-\\mu_0$ lie between $1.9994$ and $2.0000$, and the tenth is $1.9062$. The printed rule treats the one case with the largest value, and the best response to $w_{\\mathrm{ATT}}$ treats the nine. The welfare of this best response is $0.0706$, its regret is $0.0764$, and it treats 3.53\\% of the cases. For $n=255$, the expected reward is $1.2657$ under the printed rule and $1.9996$ under the best response. On the Stroke Trial data, the best response to $w_{\\mathrm{ATO}}$ treats four patients with probability $1$ and a fifth with probability $0.8452$, and the best response to $w_{\\mathrm{ATT}}$ treats five patients. The printed rule treats one patient under each reward. The condition $s'\\le M(1-r_n(p))$ holds for $w_{\\mathrm{ATO}}$ on the Horse Colic data and fails for the other three rows.\n\nThe paper gives $n=300$ for the Horse Colic data, and the notebook analyzes the 255 cases that remain after its exclusions. With $n=300$ and the same 255 cases, I find that the best responses, the welfare, the regret and the treatment rate are the same.\n\nThe notebook for the Stroke Trial data fits a linear model by least squares. I fitted it again from the data, and the coefficients agree with those printed in the stored output of the notebook to within $5\\times10^{-10}$. For the Horse Colic data I did not fit the logistic model again. I reconstructed the fitted values from the coefficients, which the stored output of the notebook gives to eight decimal places. The intercept is not printed, and I recovered it from the group means in the stored output. Bounds for the rounding of these numbers were carried through the calculation. For every set of fitted values within the bounds, the best response to $w_{\\mathrm{ATT}}$ treats the same nine cases, and it keeps between 48.0296\\% and 48.0297\\% of the achievable welfare. The tie among the 49 cases is an effect of rounding in double precision. With the logistic function evaluated to 90 decimal places, the best response to $w_{\\mathrm{ATO}}$ for $n=255$ treats 41 cases, and its welfare is still $0$ to fifteen decimal places. The best response to $w_{\\mathrm{ATT}}$ does not change.", "problem": "Reward Functions 1 and 2 are averages over the treated units in a sample of $n$ units, and each is defined to be $0$ when no unit is treated. Section 2 and Appendix B.1 define the agent's best response as the policy that maximizes the expected reward $E[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]$, with no conditioning. The first equality in the proof of Proposition 6, and the first in the proof of Proposition 7, replace this expectation by the mean among the treated units. That mean is the expected reward given that at least one unit is treated. The expected reward has the further factor $1-(1-q)^n$, the probability that at least one unit is treated, where $q=E[\\pi(X)]$ is the probability that a given unit is treated. This factor depends on the policy and increases with $q$: a policy that treats more units is left less often with the reward $0$. The printed rule maximizes the mean among the treated units and does not account for this factor. It therefore need not maximize the expected reward.\n\nSection 3 states the same rule for both rewards and adds that under $w_{\\mathrm{ATT}}$ the agent ``only treats patients with the single value $x$ with maximum treatment effect''. The proof of Proposition 2 substitutes the printed rule for the best response. The rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ of Tables 1 and 2 are computed for the printed rule: the replication files treat the units whose fitted $\\mu_1$, or fitted $\\mu_1-\\mu_0$, equals the largest value in the sample. The captions of the tables present these rows as the utility, the regret and the treatment rate of the best response $\\pi^w$.\n\nA recalculation from the replication files shows that the printed rule is the best response for one of the four rows, $w_{\\mathrm{ATO}}$ on the Horse Colic data, where 49 cases share the largest fitted $\\mu_1$. For the other three rows the best response treats more units than the printed rule. I call the largest welfare of any policy on the fitted data the achievable welfare; it is the utility in the row $w_{\\mathrm{TT}}$. For $w_{\\mathrm{ATT}}$ on the Horse Colic data, the best response treats nine cases where the printed rule treats one, and it keeps 48.03\\% of the achievable welfare where the printed rule keeps 5.34\\%. The printed row therefore understates the welfare under this reward on these data by a factor of about nine. On the Stroke Trial data the best responses keep 1.45\\% and 2.50\\% of the achievable welfare, and the printed rows still show correctly that welfare is low under both rewards.\n\nThe printed rule does maximize a different objective, the mean among the treated units in the population, when the maximum of $\\mu_1$ or $\\tau$ is positive and is attained with positive probability. For a fixed policy with $q>0$, the expected reward tends to that mean as $n\\to\\infty$. The paper states neither qualification.", "what_survives": "The definitions of the two rewards stand, and so does the bound of Proposition 2, $R(\\pi^{w_{\\mathrm{ATT}}})\\le\\max_{\\pi\\in\\Pi}V(\\pi)$, with the proof given below. The conclusion of Proposition 1 also stands. With $P(X=1)=p\\in(0,1)$, $\\mu_1(0)=0$, $\\mu_1(1)=1$ and $n\\ge2$, the unique best response to $w_{\\mathrm{ATO}}$ as defined treats type $1$ only, and with $\\mu_0(0)=0$ and $\\mu_0(1)=B>1$ its regret is $p(B-1)$, which is unbounded in $B$; for $n=1$ every best response has this regret. Under the same conditional unbiasedness condition, Proposition 3 does not depend on the error, because $E[w_{\\mathrm{TT}}]=nV(\\pi)$ involves no division by the number of treated units. The ranking and information-asymmetry results and their rows in the tables use total-effect rewards and do not depend on it either. These are statements about dependence on this error, not a confirmation of those results.\n\nWhat changes is the description of the agent under the two average rewards and the reading of the rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ of Tables 1 and 2. The agent need not treat only the best type. The printed rows summarize the printed rule, subject to the rounding corrections in \\fid{P006-E018}. They are the rows of the best response only for $w_{\\mathrm{ATO}}$ on the Horse Colic data. The statement of Section 6 that the utility for $w_{\\mathrm{ATO}}$ is close to zero for both datasets stands. On the Stroke Trial data both rewards still lose almost all of the achievable welfare: the best responses keep 1.45\\% and 2.50\\% of it and treat about five of the 7264 patients. On the Horse Colic data the best response to $w_{\\mathrm{ATT}}$ still has positive regret, but it keeps 48.03\\% of the achievable welfare, with welfare $0.0706$, regret $0.0764$ and 3.53\\% of the cases treated. The printed row, with utility $0.00784$ and regret $0.1391$, corresponds to 5.34\\%."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E001", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the right column. Two lines end a paragraph: also have access to an unbiased estimator of the mean conditional untreated potential outcome. Then: Assumption 1. The principal accesses an estimator mu-hat-zero of x which is unbiased: E of mu-hat-zero of x equals mu-zero of x, for all x in the set X. A red outline surrounds this equation. Below, the next paragraph begins: In general, obtaining an unbiased estimator mu-hat-zero of x can be difficult, but circumstances under which causal in-.", "caption": "Assumption 1, PDF p. 4, right column. The outline marks the condition, which is stated for each fixed $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002.png"}, {"alt": "Crop of the start of a proof. A two-line display gives the expected ATT reward as E of Y sub i minus mu-hat-zero of X sub i given T sub i equals 1, and then as E of Y sub i given T sub i equals 1, minus E of the conditional expectation of mu-hat-zero of X sub i given T sub i equals 1 and X sub i, given T sub i equals 1. The next line reads: Since E of mu-hat-zero of x equals mu-zero of x for all x in X, we have E of mu-hat-zero of X sub i given T sub i equals 1 and X sub i, equals mu-zero of X sub i. Thus, followed by a display in which mu-zero has replaced mu-hat-zero. A red outline surrounds the equation that follows the words we have.", "caption": "Proof of Proposition 7, PDF p. 16. The outline marks the conditional statement that is inferred from unbiasedness at each fixed $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002-b.png"}, {"alt": "Crop of the start of a proof. A display gives the expected TT reward as n times E of Y sub i minus mu-hat-zero of X sub i given T sub i equals 1, times the probability that T sub i equals 1. The text continues: Since we have ignorability in X sub i, we have E of Y sub i given T sub i equals 1 equals E of mu-one of X sub i given T sub i equals 1 (shown in more detail in the above proofs). Since E of mu-hat-zero of x equals mu-zero of x for all x in X, we have E of mu-hat-zero of X sub i given T sub i equals 1, equals E of mu-zero of X sub i given T sub i equals 1. Thus, followed by a display that ends with n V of pi. A red outline surrounds the equation that follows the second we have.", "caption": "Proof of Proposition 9, PDF p. 17. The outline marks the same inference for the total-treatment-effect reward.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002-c.png"}, {"alt": "Crop of three displayed lines under the words By the tower property. First line: E of pi of X and U times mu-hat-zero of X equals E of the conditional expectation of mu-hat-zero of X sub i times pi of X sub i and U sub i, given X sub i and U sub i. Second line: equals E of pi of X sub i and U sub i times the conditional expectation of mu-hat-zero of X sub i given X sub i and U sub i. Third line: equals E of pi of X sub i and U sub i times mu-zero of X sub i. A red outline surrounds the third line.", "caption": "Proof of Theorem 3, PDF p. 22. The outline marks the last equality, which replaces $E[\\hat\\mu_0(X_i)\\mid X_i,U_i]$ by $\\mu_0(X_i)$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002-d.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T4", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T4"}, {"id": "P006-T5", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T5"}, {"id": "P006-T6", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T6"}, {"id": "P006-T7", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T7"}, {"id": "P006-T8", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T8"}, {"id": "P006-T27", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T27"}], "headline": "Assumption 1, unbiasedness of $\\hat\\mu_0(x)$ at each fixed $x$, does not imply the unbiasedness at the agent’s treated observations that the proofs use; conditional unbiasedness at those observations repairs the estimator substitution in the zero-regret, ranking and $2\\gamma_{\\mathrm{marg}}$ proofs, and is supplied by a pointwise unbiased estimator from independent auxiliary data.", "id": "P006-E002", "location": "Assumption 1, PDF p. 4, right column; proofs of Proposition 7 (PDF p. 16), Proposition 9 (PDF p. 17) and Theorem 3 (PDF p. 22)", "pdf_page": 4, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [{"classification_after": {"class": "error", "severity": "Moderate"}, "classification_before": {"class": "error", "severity": "Moderate"}, "note": "The integrability conditions were revised; the classification is unchanged (error, Moderate). How this came to light: comparing the Lean statement with the audit target exposed a gap between integrability of each weighted reward summand and finiteness of the total expected reward. Two checks confirmed a cancellation counterexample in which the weighted identity and the asserted optimality fail despite finite total expectations. Why it changed: the checked repair explicitly requires integrable treated estimator residuals, integrable $g\\tau$, and integrable individual weighted summands. Integrability of the estimator at untreated evaluations is not required. The best-response and zero-regret conclusions hold under these sufficient conditions. P. M. Aronow delegated this decision to the model session that runs the audit, which adopted the amendment recommended by the two checks; this note was written by that session."}], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P006-T27", "P006-T4", "P006-T5", "P006-T6", "P006-T7", "P006-T8"], "text": {"as_printed": "\\emph{Assumption 1 (PDF p.~4).} The principal accesses an estimator $\\hat\\mu_0(x)$ which is unbiased:\n\\[\n\\hl{E[\\hat\\mu_0(x)] = \\mu_0(x)\\ \\ \\forall x\\in\\mathcal X}.\n\\]\n\n\\emph{Proof of Proposition 7 (PDF p.~16).} Since $E[\\hat\\mu_0(x)] = \\mu_0(x)$ for all $x\\in\\mathcal X$, we have\n\\[\n\\hl{E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1,X_i] = \\mu_0(X_i)}.\n\\]\nThus, \\ldots\n\n\\emph{Proof of Proposition 9 (PDF p.~17).} Since $E[\\hat\\mu_0(x)] = \\mu_0(x)$ for all $x\\in\\mathcal X$, we have\n\\[\n\\hl{E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1] = E[\\mu_0(X_i)\\mid T_i^\\pi=1]}.\n\\]\nThus, \\ldots\n\n\\emph{Proof of Theorem 3 (PDF p.~22).} By the tower property,\n\\[\n\\begin{aligned}\nE[\\pi(X,U)\\hat\\mu_0(X)] &= E[E[\\hat\\mu_0(X_i)\\pi(X_i,U_i)\\mid X_i,U_i]] \\\\\n&= E[\\pi(X_i,U_i)E[\\hat\\mu_0(X_i)\\mid X_i,U_i]] \\\\\n&= \\hl{E[\\pi(X_i,U_i)\\mu_0(X_i)]}.\n\\end{aligned}\n\\]", "corrected": "\\emph{Assumption 1.} The principal accesses an estimator $\\hat\\mu_0$ which is unbiased at the agent's treated observations: for every feasible treatment rule $\\pi$ and every unit $i$,\n\\[\n\\hl{E[\\hat\\mu_0(X_i)\\mid X_i,T_i^\\pi=1]=\\mu_0(X_i)\\quad\\text{whenever }\\pi(X_i)>0.}\n\\]\nUnder information asymmetry (Section 5), the condition is $E[\\hat\\mu_0(X_i)\\mid X_i,U_i,T_i^\\pi=1]=\\mu_0(X_i)$ whenever $\\pi(X_i,U_i)>0$. For the baseline substitution in the average reward $w_{\\mathrm{ATT}}$, the condition is imposed given the full vectors: $E[\\hat\\mu_0(X_i)\\mid\\mathbf X,\\mathbf T^\\pi]=\\mu_0(X_i)$ for every treated unit $i$. Here $\\pi$ is a fixed function, as in Section 2, and the expectations are over the sample, the treatment randomization and the estimator; if the estimator uses data generated under $\\pi$, they are taken under the joint distribution that $\\pi$ induces.\n\nIt suffices that $\\hat\\mu_0$ is unbiased at each fixed $x$, that is, $E[\\hat\\mu_0(x)]=\\mu_0(x)$ for all $x\\in\\mathcal X$, and is computed from auxiliary data independent of the agent's covariates, private information and treatment randomization. For an estimator that uses the agent's own sample, such as the one in Appendix A.1, item 2, the condition has to be checked separately.\n\n\\emph{Proofs of Propositions 7 and 9 (PDF pp.~16--17).} \\hl{By Assumption 1}, we have $E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1,X_i]=\\mu_0(X_i)$, and hence $E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1]=E[\\mu_0(X_i)\\mid T_i^\\pi=1]$.\n\n\\emph{Proof of Theorem 3 (PDF p.~22).} Writing $\\widehat V(\\pi)=E[w_{\\mathrm{TT}}]/n$, apply Assumption 1 and the tower property directly to the total reward:\n\\[\n\\hl{E[T_i^\\pi\\hat\\mu_0(X_i)]=E\\bigl[E[T_i^\\pi\\hat\\mu_0(X_i)\\mid X_i,U_i]\\bigr]=E[\\pi(X_i,U_i)\\mu_0(X_i)].}\n\\]\nIn the preceding regret inequality, use $\\hl{\\widehat V(\\pi)-V(\\pi)=E[\\pi(X_i,U_i)(\\mu_0(X_i,U_i)-\\mu_0(X_i))]}$. Jensen's inequality and Assumption 2 then give the printed bound.\n\nThe treated residual $T_i^\\pi(\\hat\\mu_0(X_i)-\\mu_0(X_i))$ is assumed absolutely integrable for every feasible policy and unit, so its conditional expectation is an ordinary conditional expectation. For the weighted identity used in Theorems 1 and 2, assume also that $g(X_i)\\tau(X_i)$ and each individual reward summand $(Y_i-\\hat\\mu_0(X_i))T_i^\\pi g(X_i)$ are absolutely integrable, for every applicable weight, feasible policy and unit. Finiteness of the expectation of the total weighted reward alone does not suffice: cancellations between nonintegrable summands can invalidate the weighted identity and the optimality asserted by Theorem 1. Integrability of the estimator at untreated evaluations is not required.\n\nPropositions 2, 3 and 9 and Theorems 1, 2 and 3 are read with this Assumption 1 in place of the printed one, with these additional weighted integrability conditions for Theorems 1 and 2.", "evidence": "I use exact counterexamples to the printed implication, computed in rational arithmetic, and derive the total-effect identities and regret bound under the corrected assumption.\n\n\\emph{Example A.} Let $n=1$, let $X_1$ take the values 0 and 1 with probability $1/2$ each, and let $Y_1(0)=0$ and $Y_1(1)=1/2$. Then $\\mu_0(x)=0$ and $\\tau(x)=1/2$ for both values of $x$. Define the estimator by $\\hat\\mu_0(x)=2\\cdot\\mathbf{1}\\{x=X_1\\}-1$. Note that it uses only the covariate, which the principal observes, and that it does not depend on the policy.\n\\begin{audittable}{ccccc}{Example A: the sampling distribution of the estimator.}\nProbability & $X_1$ & $\\hat\\mu_0(0)$ & $\\hat\\mu_0(1)$ & $\\hat\\mu_0(X_1)$ \\\\\n$1/2$ & 0 & $1$ & $-1$ & $1$ \\\\\n$1/2$ & 1 & $-1$ & $1$ & $1$ \\\\\n\\end{audittable}\nFor each fixed $x$, I have $E[\\hat\\mu_0(x)]=(1/2)(1)+(1/2)(-1)=0=\\mu_0(x)$, and Assumption 1 holds. However, $\\hat\\mu_0(X_1)=1$ in both states, so under the treat-all policy $E[\\hat\\mu_0(X_1)\\mid X_1,T_1^\\pi=1]=1$, not $\\mu_0(X_1)=0$. For a policy with $\\pi(0)=p_0$ and $\\pi(1)=p_1$, a direct computation gives\n\\[\nE[w_{\\mathrm{TT}}]=-\\tfrac14(p_0+p_1),\\qquad V(\\pi)=\\tfrac14(p_0+p_1).\n\\]\nThe reward is maximized only at $p_0=p_1=0$, and welfare only at $p_0=p_1=1$. The best response has regret $1/2$, which contradicts Propositions 3 and 9, and Theorem 1 with $g\\equiv1$. Since $n=1$, the rewards $w_{\\mathrm{ATT}}$ and $w_{\\mathrm{TT}}$ coincide, and the best response stated in Proposition 7 also fails. If $U_1$ is constant, then $\\gamma_{\\mathrm{marg}}=0$, and the bound of Theorem 3 would read $1/2\\le0$.\n\\begin{audittable}{cccc}{Example A: expected reward and welfare of the four deterministic policies. The last column replaces $X_1$ in the estimator by an independent copy $Z$ of the covariate.}\nPolicy $(p_0,p_1)$ & $E[w_{\\mathrm{TT}}]$, estimator from $X_1$ & Welfare $V(\\pi)$ & $E[w_{\\mathrm{TT}}]$, estimator from $Z$ \\\\\n$(0,0)$ & $0$ & $0$ & $0$ \\\\\n$(0,1)$ & $-1/4$ & $1/4$ & $1/4$ \\\\\n$(1,0)$ & $-1/4$ & $1/4$ & $1/4$ \\\\\n$(1,1)$ & $-1/2$ & $1/2$ & $1/2$ \\\\\n\\end{audittable}\n\n\\emph{Example B.} In the setting of Example A, I change two signs: $Y_1(1)=-1/2$ and $\\hat\\mu_0(x)=1-2\\cdot\\mathbf{1}\\{x=X_1\\}$. The estimator is again unbiased at each fixed $x$, and it equals $-1$ at $X_1$. Then $E[w_{\\mathrm{TT}}]=E[w_{\\mathrm{ATT}}]=(p_0+p_1)/4$ and $V(\\pi)=-(p_0+p_1)/4$. The best response treats everyone, and its regret $1/2$ exceeds $\\max_{\\pi}V(\\pi)=0$, which contradicts Proposition 2.\n\n\\emph{Example C.} Let $j$ and $k$ be two agents with one unit each. Their covariates $X_j$ and $X_k$ are independent and take the values 0 and 1 with probability $1/2$ each, as does the reference covariate, so that all weights $g$ equal 1. The untreated outcomes are 0, and the treated outcomes are $1/4$ for agent $j$ and $1/8$ for agent $k$. Both agents are scored with the estimator $\\hat\\mu_0(x)=\\mathbf{1}\\{x=X_j\\}-\\mathbf{1}\\{x=X_k\\}$, which has mean 0 at each fixed $x$. Then $\\max_{\\pi_j}E[w_j]=0$ and $\\max_{\\pi_k}E[w_k]=5/8$, although $\\tau^j(x)>\\tau^k(x)>0$ for every $x$. The ranking of Theorem 2 is reversed.\n\n\\emph{The corrected assumption.} Suppose that $E[\\hat\\mu_0(X_i)\\mid X_i,U_i,T_i^\\pi=1]=\\mu_0(X_i)$ whenever $\\pi(X_i,U_i)>0$. Since $P(T_i^\\pi=1\\mid X_i,U_i)=\\pi(X_i,U_i)$, I have\n\\[\nE[T_i^\\pi\\hat\\mu_0(X_i)\\mid X_i,U_i]=\\pi(X_i,U_i)\\,\\mu_0(X_i),\n\\]\nand ignorability and consistency imply that $E[T_i^\\pi Y_i\\mid X_i,U_i]=\\pi(X_i,U_i)\\,\\mu_1(X_i,U_i)$. Subtracting, summing over $i$ and taking expectations, I obtain\n\\[\n\\tfrac1n E[w_{\\mathrm{TT}}]=E\\bigl[\\pi(X_i,U_i)\\bigl(\\mu_1(X_i,U_i)-\\mu_0(X_i)\\bigr)\\bigr].\n\\]\nWithout $U_i$ the right side is $E[\\pi(X_i)\\tau(X_i)]=V(\\pi)$. Reward and welfare then have the same maximizers over any feasible set, which is Proposition 3. The following weighted identity uses the per-unit integrability conditions stated below. With a weight $g(X_i)$, I find in the same way that $E[w^g_{\\mathrm{TT}}]=nE[g(X_i)\\pi(X_i)\\tau(X_i)]$, which is the identity used for Theorems 1 and 2. With $U_i$, I write $\\widehat V(\\pi)=E[w_{\\mathrm{TT}}]/n$. Then\n\\[\n|\\widehat V(\\pi)-V(\\pi)|=\\bigl|E\\bigl[\\pi(X_i,U_i)\\bigl(\\mu_0(X_i,U_i)-\\mu_0(X_i)\\bigr)\\bigr]\\bigr|\\le E\\bigl|\\mu_0(X_i,U_i)-\\mu_0(X_i)\\bigr|\\le\\gamma_{\\mathrm{marg}}.\n\\]\nLet $\\pi^*$ maximize welfare and $\\pi^w$ maximize expected reward. Since $\\widehat V(\\pi^*)\\le\\widehat V(\\pi^w)$, I conclude that\n\\[\nV(\\pi^*)-V(\\pi^w)=\\bigl[V(\\pi^*)-\\widehat V(\\pi^*)\\bigr]+\\bigl[\\widehat V(\\pi^*)-\\widehat V(\\pi^w)\\bigr]+\\bigl[\\widehat V(\\pi^w)-V(\\pi^w)\\bigr]\\le2\\gamma_{\\mathrm{marg}},\n\\]\nwhich is the bound of Theorem 3. If $\\hat\\mu_0$ is unbiased at each fixed $x$ and independent of the agent's covariates, private information and treatment randomization, then $E[\\hat\\mu_0(X_i)\\mid X_i=x,U_i=u,T_i^\\pi=1]=E[\\hat\\mu_0(x)]=\\mu_0(x)$, and the corrected assumption holds. The last column of the second table shows this for Example A.", "problem": "Assumption 1 fixes the mean of the random function $\\hat\\mu_0$ at each fixed argument $x$. The proofs use the mean of $\\hat\\mu_0$ at the agent's own treated observations, that is, of $\\hat\\mu_0(X_i)$ given $T_i^\\pi=1$. The first does not imply the second when the estimator depends on the sample on which the agent is scored. In that case $\\hat\\mu_0(X_i)$ can be biased for $\\mu_0(X_i)$ although $\\hat\\mu_0(x)$ is unbiased for every fixed $x$, so the three highlighted steps need not hold. If Assumption 1 is the only condition on the estimator, then the conclusions drawn from these steps fail as well: the best responses of Propositions 7 and 9, the bound of Proposition 2, the zero regret of Proposition 3 and Theorem 1, the ranking property of Theorem 2, and the bound $2\\gamma_{\\mathrm{marg}}$ of Theorem 3.\n\nThe paper does not exclude such estimators. The auxiliary data on PDF p.~4 are introduced as ``a concrete example'', and Appendix A.1, item 2, proposes to add the agent's own untreated units to the estimate of $\\mu_0$. Assumption 1 is a legitimate assumption. What fails is the step from it to the conditional statements.\n\nThe examples below do not require an agent who observes the realized estimate before choosing a policy. In each of them the policy is a fixed function of the covariates, as in Section 2, and the expected reward is taken over the estimator as well.", "what_survives": "Without information asymmetry, the corrected assumption, with integrable treated estimator residuals, gives the identity $E[w_{\\mathrm{TT}}]=nV(\\pi)$ and the zero regret of Proposition 3 for any feasible set. With the weighted integrability conditions below, it also gives Theorem 1 for unconstrained policies and positive weights, by the derivation above. Under information asymmetry, it gives the bound $2\\gamma_{\\mathrm{marg}}$ of Theorem 3. The corrected assumption restricts the joint distribution of the estimator and the agent's sample. An unbiased estimator from independent auxiliary data, as in the example on PDF p.~4 and in Appendix D.2, satisfies it. An estimator that reuses the agent's sample need not. The rewards for the average and the total treated outcome do not use the estimator. The experiments of Section 6 take the fitted means as given and do not simulate estimation of a separate baseline, so no numerical correction follows from this defect. This correction concerns only the replacement of $\\hat\\mu_0$ by $\\mu_0$. The best responses to the average rewards are corrected in \\fid{P006-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E002", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Assumption 3 (Non-degenerate w) and Theorem 4. Two red boxes mark, in the theorem, the words 'Suppose w satisfies Assumption 3.' and the words 'then regret is unbounded.'", "caption": "Appendix A.2, PDF p. 14: Assumption 3 and Theorem 4. The marked hypothesis places no condition on the feasible set $\\Pi$, and the marked conclusion does not say for which best response the regret is unbounded.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E003.png"}, {"alt": "Crop of the second half of the proof of Theorem 4, from 'Denote the welfare maximizing policy for distribution D as' to the last line. A red box marks the sentence 'Then the set of welfare maximizing policies contains a pi star with pi star of x bar equal to 0 for all x bar in X bar.' The regret display below it ends with 'greater than or equal to minus delta times (alpha minus beta) times rho'.", "caption": "Proof of Theorem 4, PDF p. 14. The marked step takes a welfare-maximizing policy that withholds treatment on $\\bar{\\mathcal{X}}$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T10", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T10"}, {"id": "P006-T11", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T11"}], "headline": "Adding feasibility of the policy that treats nobody repairs Theorem 4, with the conclusion that some best response has arbitrarily large regret; the printed proof assumes that treatment can be withheld.", "id": "P006-E003", "location": "Appendix A.2, Assumption 3, Theorem 4 and its proof, PDF p. 14", "pdf_page": 14, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T10", "P006-T11"], "text": {"as_printed": "\\textbf{Theorem 4.} \\hl{Suppose $w$ satisfies Assumption 3.} Let $\\mathcal{S}$ be the set of all pairs of distributions $\\mathcal{D},\\mathcal{D}'$ such that the distributions of $X_i,Y_i(1)$ remain unchanged between $\\mathcal{D}$ and $\\mathcal{D}'$. If for all pairs of distributions $\\mathcal{D},\\mathcal{D}'\\in\\mathcal{S}$, $E_{\\mathcal{D}}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=E_{\\mathcal{D}'}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]$ for all $\\pi$, \\hl{then regret is unbounded.}\n\nFrom the proof: ``Choose $\\beta>\\alpha$. \\hl{Then the set of welfare maximizing policies contains a $\\pi^*$ with $\\pi^*(\\bar x)=0$ for all $\\bar x\\in\\bar{\\mathcal{X}}$.} The regret is then\n\\[\nR(\\pi')=E_{\\mathcal{D}}[(\\pi^*(X_i)-\\pi'(X_i))(\\mu_1(X_i)-\\mu_0(X_i))]\\ge-\\delta(\\alpha-\\beta)\\rho.\n\\]\nFixing $\\delta>0,\\rho>0,\\alpha<\\infty$ and choosing $\\beta$ to be arbitrarily high results in arbitrarily high regret.''", "corrected": "\\textbf{Theorem 4.} Suppose \\hl{that the policy $\\pi\\equiv0$ belongs to $\\Pi$ and} that $w$ satisfies Assumption 3. Suppose that, for every $\\pi\\in\\Pi$, the expected reward $E_{\\mathcal{D}}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]$ takes the same value for all distributions $\\mathcal{D}$ with the same joint distribution of $(X_i,Y_i(1))$, all expectations being finite. Then \\hl{for every $M>0$ there exist a distribution $\\mathcal{D}$ and a best response $\\pi_*\\in\\arg\\max_{\\pi\\in\\Pi}E_{\\mathcal{D}}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]$ such that $R_{\\mathcal{D}}(\\pi_*)>M$.}\n\nIn the proof, the steps from ``Let $\\pi'\\in\\arg\\max$'' to the regret display are replaced by the construction with $Y(0)=Y(1)+B$, which gives $R_{\\mathcal{D}_B}(\\pi_*)=B\\,E[\\pi_*(X)]$.", "evidence": "The evidence is an exact counterexample and a derivation of the corrected statement. Let $n=1$, let $X=0$ with probability one, let $\\Pi$ consist of the single policy $\\pi(0)=1$, and let $w=w_{\\mathrm{ATO}}$. For $B>0$, let $Y(1)=1$ and $Y(0)=1+B$. Assumption 3 holds because the only feasible policy treats with probability $1$. For $n=1$, I have $E[w_{\\mathrm{ATO}}]=E[\\pi(X)Y(1)]$, which does not involve $Y(0)$. The expected reward is then unchanged when the distribution of $Y(0)$ changes. The welfare of the only feasible policy is $-B$, which is also the maximum of the welfare over $\\Pi$, and I find that the regret is $0$ for every $B$. With the constants of the printed proof, $\\bar{\\mathcal{X}}=\\{0\\}$, $\\rho=\\delta=\\alpha=1$ and $\\beta=1+B$, the printed bound reads $0\\ge B$.\n\nFor the corrected statement, by Assumption 3 there exist a distribution $\\mathcal{D}_*$ and a best response $\\pi_*$ such that $c=E_{\\mathcal{D}_*}[\\pi_*(X)]>0$. For $B>0$, let $\\mathcal{D}_B$ be the distribution with the same joint distribution of $(X,Y(1))$ as $\\mathcal{D}_*$ and with $Y(0)=Y(1)+B$. Since the expected reward of each feasible policy is the same under $\\mathcal{D}_B$ and $\\mathcal{D}_*$, the policy $\\pi_*$ is a best response under $\\mathcal{D}_B$. Under $\\mathcal{D}_B$, the treatment effect is $\\tau(x)=-B$ for all $x$. Then every feasible $\\pi$ has welfare $V(\\pi)=-B\\,E[\\pi(X)]\\le0$, and the policy that treats nobody attains $0$. I obtain\n\\[\nR_{\\mathcal{D}_B}(\\pi_*)=0-(-Bc)=Bc,\n\\]\nand for $B=(M+1)/c$ I have $R_{\\mathcal{D}_B}(\\pi_*)>M$. Note that the conclusion cannot be extended to every best response: with $w=0$ and $\\Pi$ consisting of the policy that treats nobody and the policy that treats everyone, both policies are best responses, and for $Y(1)=1$ and $Y(0)=1+B$ the first has regret $0$ and the second has regret $B$.", "problem": "Section 2 measures regret against the best policy in the feasible set $\\Pi$, and Theorem 4 places no condition on $\\Pi$. Its proof takes a welfare-maximizing policy $\\pi^*$ that does not treat on the set $\\bar{\\mathcal{X}}$. Such a policy exists only if $\\Pi$ contains a policy that withholds treatment there. If $\\Pi$ contains only the policy that treats everyone, then that policy is both the best response and the welfare maximizer, and the regret is $0$ for every distribution. The proof also takes $\\pi'$ and $\\mu_1$ to be constant on $\\bar{\\mathcal{X}}$, and Assumption 3 does not supply a set of positive probability on which they are constant. The conclusion is written for the regret of a best response without saying which one, and when several policies tie, some of them can have regret $0$.", "what_survives": "Proposition 1 assumes that $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$, and its conclusion that the regret under $w_{\\mathrm{ATO}}$ can be arbitrarily large is not affected. No other result in the paper uses Theorem 4 or Assumption 3. What changes is the scope of Theorem 4 and of the two paragraphs that introduce it in Appendix A.2. The theorem does not cover a feasible set that requires some treatment, such as a fixed positive share of treated units, and it asserts large regret for some best response and not for all of them."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E003", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the left column. Two lines end a paragraph: bution P sub X sub 0 satisfies both ranking properties in Definitions 1 and 2. Then: Theorem 2 (Ranking Desiderata Satisfied). Let P sub X superscript k be absolutely continuous with respect to P sub X sub 0, and let g sub k equal one over n sub k times d P sub X sub 0 over d P sub X superscript k be the normalized Radon-Nikodym derivative of the reference distribution P sub X sub 0 with respect. Red outlines surround P sub X superscript k and the words be absolutely continuous with respect to P sub X sub 0.", "caption": "Theorem 2, PDF p. 6, foot of the left column; the statement continues at the top of the right column. The outlines mark the condition of absolute continuity.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E005.png"}, {"alt": "Crop of three lines. The text reads: Theorem 2 (Ranking Desiderata Satisfied). Let P sub X superscript (k) be absolutely continuous with respect to P sub X sub 0, and let g sub k equal one over n sub k times d P sub X sub 0 over d P sub X superscript (k) be the normalized Radon-Nikodym derivative of the reference distribution P sub X sub 0 with respect to agent k's covariate distribution P sub X superscript (k). Then. A red outline surrounds the words from P sub X superscript (k) to P sub X sub 0.", "caption": "Restatement of Theorem 2 in Appendix C.2, PDF p. 20. The outline marks the same condition.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E005-b.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T5", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T5"}, {"id": "P006-T12", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T12"}], "headline": "In Theorem 2 the direction of absolute continuity is reversed: the weight $g_k$ is defined when $P_{X_0}$ is absolutely continuous with respect to $P_{X^k}$.", "id": "P006-E005", "location": "Theorem 2, PDF p. 6, foot of the left column; restatement in Appendix C.2, PDF p. 20", "pdf_page": 6, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T12", "P006-T5"], "text": {"as_printed": "\\textbf{Theorem 2} (Ranking Desiderata Satisfied). Let $\\hl{P_{X^k}}$ be absolutely continuous with respect to $\\hl{P_{X_0}}$, and let $g_k=\\frac{1}{n_k}\\frac{dP_{X_0}}{dP_{X^k}}$ be the normalized Radon--Nikodym derivative of the reference distribution $P_{X_0}$ with respect to agent $k$'s covariate distribution $P_{X^k}$. Then setting $w_k$ to be $w_{\\mathrm{TT}}^{g_k}$ for agent $k$'s treatment population satisfies both ranking properties in Definitions 1 and 2 as long as $\\Pi_k$ is unconstrained and treatment effects are nonnegative, $\\tau^k(x)\\geq0$, for all $k\\in\\{1,\\ldots,K\\}$.", "corrected": "\\textbf{Theorem 2} (Ranking Desiderata Satisfied). Let $\\hl{P_{X_0}}$ be absolutely continuous with respect to $\\hl{P_{X^k}}$, and let $g_k=\\frac{1}{n_k}\\frac{dP_{X_0}}{dP_{X^k}}$ be the normalized Radon--Nikodym derivative of the reference distribution $P_{X_0}$ with respect to agent $k$'s covariate distribution $P_{X^k}$. Then setting $w_k$ to be $w_{\\mathrm{TT}}^{g_k}$ for agent $k$'s treatment population satisfies both ranking properties in Definitions 1 and 2 as long as $\\Pi_k$ is unconstrained and treatment effects are nonnegative, $\\tau^k(x)\\geq0$, for all $k\\in\\{1,\\ldots,K\\}$.\n\nThe same change applies to the restatement on PDF p.~20, where the distribution is written $P_{X^{(k)}}$. If the weights $g_k$ must also be strictly positive almost everywhere under $P_{X^k}$, as the proof of Theorem 1 assumes of its weight, then the two distributions must be mutually absolutely continuous. The ranking conclusion does not need this.", "evidence": "The evidence is an exact counterexample to the sufficiency of the printed condition, and a derivation of the theorem under the corrected one.\n\n\\emph{Counterexample.} On $[0,1]$, let $L$ be Lebesgue measure, and let $P_{X^k}=L$ and $P_{X_0}=(L+\\delta_{1/2})/2$. For every Borel set $C$, I have $P_{X_0}(C)\\ge P_{X^k}(C)/2$, which implies that $P_{X^k}$ is absolutely continuous with respect to $P_{X_0}$, as the theorem assumes. Both distributions have support $[0,1]$. However, $P_{X^k}(\\{1/2\\})=0$ and $P_{X_0}(\\{1/2\\})=1/2$. If a density $f=dP_{X_0}/dP_{X^k}$ existed, then I would have $1/2=P_{X_0}(\\{1/2\\})=\\int_{\\{1/2\\}}f\\,dP_{X^k}=0$. The example shows that the printed condition does not suffice for the construction. It is not a counterexample to the ranking conclusion in a case where the derivative exists.\n\n\\emph{Derivation.} Suppose that $P_{X_0}$ is absolutely continuous with respect to $P_{X^k}$, and write $f=dP_{X_0}/dP_{X^k}$, so that $g_k=f/n_k$. Then $\\int h\\,dP_{X_0}=\\int hf\\,dP_{X^k}$ for every nonnegative measurable function $h$. Using the expected score of Theorem 1 and this identity with $h=\\tau^k\\pi_k$, I obtain\n\\[\nE[w_k]=n_k\\int\\tau^k(x)\\pi_k(x)g_k(x)\\,dP_{X^k}(x)=\\int\\tau^k(x)\\pi_k(x)\\,dP_{X_0}(x).\n\\]\nSince $0\\le\\pi_k\\le1$ and $\\tau^k\\ge0$, the maximum over $\\pi_k$ is $E[\\tau^k(X_0)]$, attained at $\\pi_k\\equiv1$. If $\\tau^j(x)\\ge\\tau^k(x)$ for all $x$, then $E[\\tau^j(X_0)]\\ge E[\\tau^k(X_0)]$, and Definition 1 holds. The hypothesis of Definition 2 is this inequality itself.", "problem": "The derivative $dP_{X_0}/dP_{X^k}$ of the whole reference distribution exists when $P_{X_0}$ is absolutely continuous with respect to $P_{X^k}$. The theorem assumes the opposite direction, which does not guarantee that the weight $g_k$ is defined. Section 4 assumes that all covariate distributions have the same support $\\mathcal X$. If support is understood in the topological sense, this assumption does not supply the missing direction. The intended condition is clear from the derivative that the theorem names, and with it the change-of-measure argument is valid.", "what_survives": "The ranking conclusion and its proof are unchanged once the condition is reversed. Theorem 1, the results of Sections 3 and 5 and the experiments of Section 6 do not use the condition. Section 6 contains no experiment with reweighting to a reference distribution. The expected score used in the derivation requires the corrected Assumption 1 of \\fid{P006-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E005", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first paragraph of Appendix D.3, six lines, beginning 'Suppose the principal estimates mu hat 0 of x from the untreated data from the agent's treatment population'. A red box marks 'pi of x greater than 0 for all x in X' after the words 'maintains positivity'.", "caption": "Appendix D.3, first paragraph, PDF p. 23. The marked condition is stated as positivity for identifying the untreated outcome from the agent’s untreated units.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E006.png"}, {"alt": "Crop of the statement of Theorem 5 with its display 'R of pi w is at most 2 gamma max'. A red box marks the hypothesis 'pi of x, u greater than 0 for all x in X, u in U'.", "caption": "Theorem 5, PDF p. 23. The marked hypothesis requires a positive probability of treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E006-b.png"}, {"alt": "Crop of the end of the proof of Theorem 5 and the statement of Lemma 3. A red box marks 'pi of x, u greater than 0 for all x in X, u in U' in the sentence 'Since for all pi in Pi, ..., we can apply Lemma 3 below'. Lemma 3 begins 'Suppose that for all pi in Pi, P of U given X equals x and T pi equals 0 is a well defined probability distribution.'", "caption": "Proof of Theorem 5, PDF p. 24. The marked condition is given as the reason that Lemma 3 applies. Lemma 3, at the bottom of the crop, assumes that the distribution of $U$ given $X=x$ and $T^\\pi=0$ is well defined.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E006-c.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T15", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T15"}, {"id": "P006-T16", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T16"}], "headline": "The positivity condition of Appendix D.3 and Theorem 5 is that untreated units occur with positive probability at each $x$, $P(T^\\pi=0\\mid X=x)>0$, for which $\\pi(x,u)<1$ suffices; the printed condition $\\pi>0$ concerns the treated units.", "id": "P006-E006", "location": "Appendix D.3, first paragraph and Theorem 5, PDF p. 23; proof of Theorem 5, sentence before Lemma 3, PDF p. 24", "pdf_page": 23, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T15", "P006-T16"], "text": {"as_printed": "Appendix D.3, first paragraph, PDF p.~23: ``As discussed in Sections 5 and A, under full information symmetry, the mean conditional untreated potential outcome is identifiable as long as the agent's treatment rule class $\\Pi$ maintains positivity, \\hl{$\\pi(x)>0$} for all $x\\in\\mathcal{X}$.''\n\nPDF p.~23: ``\\textbf{Theorem 5.} Suppose for all $\\pi\\in\\Pi$, \\hl{$\\pi(x,u)>0$} for all $x\\in\\mathcal{X}$, $u\\in\\mathcal{U}$. If the principal applies the reward function $w_{\\mathrm{TT}}$ using an estimate $\\hat\\mu_0^\\pi(x)$, then under Assumption 4, the regret is upper bounded as $R(\\pi^w)\\le2\\gamma_{\\max}$.''\n\nProof, PDF p.~24: ``Since for all $\\pi\\in\\Pi$, \\hl{$\\pi(x,u)>0$} for all $x\\in\\mathcal{X}$, $u\\in\\mathcal{U}$, we can apply Lemma 3 below, which says that $\\hat\\mu_0^\\pi(x)\\in[\\min_u\\mu_0(x,u),\\max_u\\mu_0(x,u)]$.''", "corrected": "Appendix D.3, first paragraph: ``\\ldots\\ the mean conditional untreated potential outcome is identifiable as long as the agent's treatment rule class $\\Pi$ maintains positivity, \\hl{$\\pi(x)<1$} for all $x\\in\\mathcal{X}$.''\n\n\\textbf{Theorem 5.} Suppose for all $\\pi\\in\\Pi$, \\hl{$P(T^\\pi=0\\mid X=x)=E[1-\\pi(x,U)\\mid X=x]>0$ for almost every $x$ with respect to the distribution of $X$}. If the principal applies the reward function $w_{\\mathrm{TT}}$ using the estimate $\\hat\\mu_0^\\pi(x)=E[Y_i\\mid T^\\pi=0,X=x]$, then under Assumption 4, the regret is upper bounded as $R(\\pi^w)\\le2\\gamma_{\\max}$. The hypothesis holds if $\\pi(x,u)<1$ for all $x\\in\\mathcal{X}$, $u\\in\\mathcal{U}$.\n\nProof: ``Since for all $\\pi\\in\\Pi$, \\hl{$P(T^\\pi=0\\mid X=x)>0$ for almost every $x$ with respect to the distribution of $X$}, the distribution $P_{U\\mid X=x,T^\\pi=0}$ is well defined and we can apply Lemma 3 below \\ldots''", "evidence": "The evidence is an exact counterexample to the printed hypothesis and a derivation of the bound under the corrected one. Let $X=0$ and $U=0$ with probability one, let $\\Pi$ consist of the single policy $\\pi(0,0)=1$, and let $Y(1)=1$. Consider $Y(0)=0$ and $Y(0)=2$. In both cases $\\pi>0$, Assumption 4 holds with $\\gamma_{\\max}=0$, and the observed data are the single point $(X,T,Y)=(0,1,1)$. The untreated means are $0$ and $2$, and the untreated mean is not identified. Further, $P(T^\\pi=0\\mid X=0)=0$, and the conditional mean in $\\hat\\mu_0^\\pi(0)$ is the ratio $0/0$. Untreated units elsewhere do not help: with $P(X=0)=P(X=1)=1/2$, $\\pi(0)=1$ and $\\pi(1)=1/2$, I have $P(T^\\pi=0)=1/4$ and $P(T^\\pi=0\\mid X=0)=0$.\n\nFor the corrected hypothesis, I write $q_\\pi(x)=P(T^\\pi=0\\mid X=x)=E[1-\\pi(x,U)\\mid X=x]$ and suppose that $q_\\pi(x)>0$. Then the distribution of $U$ given $X=x$ and $T^\\pi=0$ has density $(1-\\pi(x,u))/q_\\pi(x)$ with respect to the distribution of $U$ given $X=x$, and $\\hat\\mu_0^\\pi(x)$ is the mean of $\\mu_0(x,\\cdot)$ under it. This implies that $|\\hat\\mu_0^\\pi(x)-\\mu_0(x,u)|\\le\\Delta(x,u)$ for all $u$, where $\\Delta(x,u)=\\max_{\\tilde u\\in\\mathcal{U}}|\\mu_0(x,u)-\\mu_0(x,\\tilde u)|$ is the quantity defined in Assumption 4. Let $S(\\pi)=E[\\pi(X,U)(\\mu_1(X,U)-\\hat\\mu_0^\\pi(X))]$ denote the expected reward per unit. Since $0\\le\\pi\\le1$, I have $|S(\\pi)-V(\\pi)|\\le E[\\Delta(X,U)]\\le\\gamma_{\\max}$ for every $\\pi\\in\\Pi$. Let $\\pi^*$ maximize $V$ over $\\Pi$. Since $\\pi^w$ maximizes $S$ over $\\Pi$, I obtain\n\\[\nR(\\pi^w)=[V(\\pi^*)-S(\\pi^*)]+[S(\\pi^*)-S(\\pi^w)]+[S(\\pi^w)-V(\\pi^w)]\\le2\\gamma_{\\max}.\n\\]", "problem": "In the paper $\\pi$ is the probability of treatment. The estimate $\\hat\\mu_0^\\pi(x)=E[Y_i\\mid T^\\pi=0,X=x]$ is a mean over untreated units. It is defined only if untreated units occur with positive probability at $x$, and Lemma 3 assumes that the distribution of $U$ given $X=x$ and $T^\\pi=0$ is well defined. The printed condition $\\pi>0$ ensures instead that treated units occur, and it allows a policy that treats everyone. The condition is printed in this form in three places. Appendix A.1 states the condition correctly, as $P(T_i^\\pi=0\\mid X_i=x)>0$, and notes that it can be enforced by restricting $\\Pi$ to treatment rules with $\\pi(x)<1$.", "what_survives": "The bound $R(\\pi^w)\\le2\\gamma_{\\max}$ of Theorem 5 holds under the corrected hypothesis, with the same constant, and Lemma 3 holds under its own hypothesis. The results of Appendix D.2, Theorem 3 and Proposition 5 do not use the condition. Appendix E.3 states that the experiments do not simulate the estimation of $\\hat\\mu_0$ from a subset of the data, and no reported number depends on the condition. The corrected hypothesis restricts $\\Pi$: it excludes every policy that treats all units at a value of $x$ of positive probability. Appendix A.1 notes that such a restriction may not always be possible or ethical."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E006", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph 'A second failure mode would be if alpha equals minus 1, and beta is greater than 1. Then the regret is given by', the display, and the three lines that follow. A red box marks the right part of the display, 'minus p times (beta minus 1) equals 1 minus p beta', which follows 'R of pi w equals (1 minus p)'.", "caption": "Appendix B.4, proof of Proposition 1, PDF p. 17: the second failure mode. The marked terms are the welfare of the agent’s policy, entered with the wrong sign, and the resulting value of the regret.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T18", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T18"}, {"id": "P006-T19", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T19"}], "headline": "The regret in the second failure mode is $(1-p)-p(1-\\beta)=1-2p+p\\beta$; the printed display subtracts $p(\\beta-1)$ and has the value $1-p\\beta$.", "id": "P006-E007", "location": "Appendix B.4, proof of Proposition 1, second failure mode, display after “Then the regret is given by”, PDF p. 17", "pdf_page": 17, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T18", "P006-T19"], "text": {"as_printed": "A second failure mode would be if $\\alpha=-1$, and $\\beta>1$. Then the regret is given by\n\\[\nR(\\pi^w)=(1-p)\\hl{{}-p(\\beta-1)=1-p\\beta}.\n\\]\nThis is also unbounded for unbounded $\\beta$.", "corrected": "Let $\\mathcal{X}=\\{0,1\\}$ with $P(X_i=1)=p$, \\hl{$0<p<1$}, $\\mu_1(0)=0$, $\\mu_1(1)=1$, $\\mu_0(0)=\\alpha=-1$ and $\\mu_0(1)=\\beta>1$, and let $\\pi^w(0)=0$ and $\\pi^w(1)=1$. Then the regret is given by\n\\[\nR(\\pi^w)=(1-p)-p(\\hl{1-\\beta})=\\hl{1-2p+p\\beta}.\n\\]\nThis is also unbounded for unbounded $\\beta$.", "evidence": "The evidence is a derivation and an exact example. With $\\alpha=-1$ and $\\beta>1$, the treatment effects are $\\tau(0)=0-(-1)=1$ and $\\tau(1)=1-\\beta<0$, which yields $V(\\pi)=(1-p)\\pi(0)+p(1-\\beta)\\pi(1)$. For every policy,\n\\[\n(1-p)-V(\\pi)=(1-p)(1-\\pi(0))+p(\\beta-1)\\pi(1)\\ge0,\n\\]\nwhich implies that the largest welfare is $1-p$, attained by treating $x=0$ only. The agent's policy has $V(\\pi^w)=p(1-\\beta)$. I obtain\n\\[\nR(\\pi^w)=(1-p)-p(1-\\beta)=1-2p+p\\beta,\n\\]\nwhich exceeds the printed expression by $2p(\\beta-1)>0$. For $p=1/2$ and $\\beta=4$, the largest welfare is $1/2$, the welfare of $\\pi^w$ is $-3/2$, and the regret is $2$; the printed expression equals $-1$. In this example with $n=2$, a direct enumeration shows that $\\pi^w$ maximizes the expected reward under $w_{\\mathrm{ATO}}$ as defined. The example does not depend on the rule corrected in \\fid{P006-E001}.", "problem": "For $0<p<1$, the proof takes $\\mathcal{X}=\\{0,1\\}$ with $P(X_i=1)=p$, $\\mu_1(0)=0$, $\\mu_1(1)=1$, $\\mu_0(0)=\\alpha$ and $\\mu_0(1)=\\beta$, and the agent's policy $\\pi^w(0)=0$, $\\pi^w(1)=1$. The regret is the largest welfare minus the welfare of $\\pi^w$. In the second failure mode the welfare of $\\pi^w$ is $p(1-\\beta)$, which is negative. The display subtracts $p(\\beta-1)$, its absolute value. The printed expression $1-p\\beta$ decreases in $\\beta$ and is negative for $\\beta>1/p$, and a regret is never negative.", "what_survives": "Proposition 1 is unchanged. With $p=1/2$ and $\\beta=2m$, the corrected regret equals $m$. The regret in the second failure mode is then unbounded, and the first failure mode, with regret $-\\alpha(1-p)$, is printed correctly. The replication files compute the regret under $w_{\\mathrm{ATO}}$ as the difference between the largest utility and the utility of the agent's policy, both computed from $\\mu_1-\\mu_0$, and no reported number depends on this display."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E007", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of eleven lines of Appendix E.2.1. The text reads: We removed all examples in which the horse was euthanized, and used the remaining outcome variable as Y, where we set Y sub i = 1 if the horse lived, and Y sub i = -1 if the horse died. After a sentence on the data files it continues: To simulate mu sub t of x, we assume that the outcome distribution takes the parametric form, P(Y(t) = 1 given X = x) = sigma(beta 0 + beta 1 transpose x + beta 2 t + beta 3 transpose x t), where sigma(x) = 1 over (1 + e to the minus x) is the standard logistic function. We trained a logistic regression model of Y on X, T, and the interaction term XT to estimate parameters beta hat, and used the resulting estimate to compute mu sub t of x = sigma(beta hat 0 + beta hat 1 transpose x + beta hat 2 t + beta hat 3 transpose x t). A red outline surrounds this last formula. The crop also shows the AUC and accuracy, the statement that mu sub t of x is taken as synthetic mean conditional potential outcomes, and the beginning of a sentence on clinical validity.", "caption": "Appendix E.2.1 (PDF p. 25). The outline marks the printed formula for $\\mu_t(x)$. Its right side is the survival probability of the display above it, and the second line of the crop codes the outcome as $1$ and $-1$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T20", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T20"}, {"id": "P006-T21", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T21"}], "headline": "With survival coded $1$ and death coded $-1$, the fitted conditional mean is $\\mu_t(x) = 2\\sigma(\\cdot) - 1$ and not the survival probability $\\sigma(\\cdot)$ that the appendix prints; the released code uses the correct formula.", "id": "P006-E008", "location": "Appendix E.2.1, paragraph beginning “We trained a logistic regression model” (PDF p. 25)", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T20", "P006-T21"], "text": {"as_printed": "We removed all examples in which the horse was euthanized, and used the remaining ``outcome'' variable as $Y$, where we set $Y_i = 1$ if the horse lived, and $Y_i = -1$ if the horse died. [\\ldots] To simulate $\\mu_t(x)$, we assume that the outcome distribution takes the parametric form,\n\\[\nP(Y(t) = 1 | X = x) = \\sigma(\\beta_0 + \\beta_1^\\top x + \\beta_2 t + \\beta_3^\\top x t)\n\\]\nwhere $\\sigma(x) = \\frac{1}{1+e^{-x}}$ is the standard logistic function.\n\nWe trained a logistic regression model of $Y$ on $X$, $T$, and the interaction term $XT$ to estimate parameters $\\hat\\beta$, and used the resulting estimate to compute $\\mu_t(x) = \\hl{\\sigma(\\hat\\beta_0 + \\hat\\beta_1^\\top x + \\hat\\beta_2 t + \\hat\\beta_3^\\top x t)}$. [\\ldots] When computing agent best responses and regret, we take this function $\\mu_t(x)$ to be given as synthetic mean conditional potential outcomes.", "corrected": "We trained a logistic regression model of $Y$ on $X$, $T$, and the interaction term $XT$ to estimate parameters $\\hat\\beta$, and used the resulting estimate to compute\n\\[\n\\mu_t(x) = \\hl{2}\\,\\sigma(\\hat\\beta_0 + \\hat\\beta_1^\\top x + \\hat\\beta_2 t + \\hat\\beta_3^\\top x t) \\hl{{} - 1} .\n\\]\nHere $Y_i = 1$ if the horse lived and $Y_i = -1$ if the horse died, and $\\sigma(\\hat\\beta_0 + \\hat\\beta_1^\\top x + \\hat\\beta_2 t + \\hat\\beta_3^\\top x t)$ is the fitted probability $P(Y(t) = 1 \\mid X = x)$ of survival. The two probability equations, in Section~6 (PDF p.~7) and in Appendix~E.2.1, stay as printed, and Section~6 may state the same conversion to the mean.", "evidence": "The evidence is a derivation and an exact counterexample. Fix $t$ and $x$, and write $p_t(x) = P(Y(t) = 1 \\mid X = x)$. Since $Y(t)$ takes only the values $1$ and $-1$, I have\n\\[\n\\mathbb{E}[Y(t) \\mid X = x] = 1 \\cdot p_t(x) + (-1) \\cdot (1 - p_t(x)) = 2 p_t(x) - 1 .\n\\]\nSubstituting the fitted probability into this identity, I obtain the corrected formula. For the counterexample, let $X = 0$ with probability one, and let $(Y(0), Y(1))$ equal $(1, 1)$, $(1, -1)$, $(-1, 1)$ and $(-1, -1)$ with probabilities $1/8$, $3/8$, $1/8$ and $3/8$. This law belongs to the printed model, with $\\beta_0 = 0$, $\\beta_2 = -\\log 3$ and $\\beta_1 = \\beta_3 = 0$: the survival probabilities are $\\sigma(0) = 1/2$ without treatment and $\\sigma(-\\log 3) = 1/4$ with treatment. The means are $\\mu_0 = 0$ and $\\mu_1 = -1/2$, and the printed formula gives $\\mu_1 = 1/4$. In this example the rule $\\mu_1(x) > 0$ treats nobody when the correct mean is used and everybody when the printed one is used. The first response has regret $0$, and the second has regret $1/2$. In the released notebook for the Horse Colic data, both prediction functions return \\texttt{(Y\\_probs*2.) - 1.}, and the optimal utility stored there, $0.146954$, agrees with the entry $0.14695$ of Table~1.", "problem": "The appendix codes $Y$ on the scale $\\{-1, 1\\}$, and Section~2 defines $\\mu_t(X)$ as the conditional mean $\\mathbb{E}[Y_i(t) \\mid X]$. The logistic model specifies the probability of survival, $P(Y(t) = 1 \\mid X = x)$, and the printed sentence sets the mean equal to that probability. For an outcome with the values $1$ and $-1$, the mean is twice the probability minus one. The two agree only if the probability is one, a value that the logistic function does not attain. The released notebook computes the means as $2p - 1$ from the fitted probability $p$. The numbers in the paper are then on the stated scale, and the defect is in the description. If the printed formula were followed, the treatment effects would be half as large, and so would the utility and the regret of a fixed treatment rule. The response to the total-outcome reward $w_{\\mathrm{TO}}$ of the appendix could also change. Its rule $\\mu_1(x) > 0$ means $P(Y(1) = 1 \\mid X = x) > 1/2$ on the stated scale, and it holds for every $x$ when the mean is replaced by the probability.", "what_survives": "The correction changes one formula in the description of the experiment. It adds no assumption and changes no estimand or theorem. The released code already applies it. The Horse Colic entries of Tables~1 and~2 need no change for this correction, and they should not be doubled. The probability equations in Section~6 and in the appendix are correct as printed, and the Stroke Trial model and the theoretical results do not involve this formula."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E008", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of eight lines of the right column. The text reads: take these as given to produce mu-zero of x and mu-one of x. The fitted mu-zero of x and mu-one of x show 62 horses benefiting from surgery and 146 being better off without surgery. On the horses that would benefit, the average benefit was 0.147, which is fairly significant. The clinical validity of these estimated potential outcomes cannot be verified from this data alone, and we instead take these estimates as synthetic potential outcomes. Red outlines surround the words the horses that would benefit and the number 0.147.", "caption": "Section 6, paragraph “Horse Colic dataset”, PDF p. 7, right column. The outlines mark the group to which the average is attributed and the number.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E009.png"}, {"alt": "Crop of five lines of the left column. The text reads: 1360 patients benefiting from heparin and 5904 being better off without heparin. On the patients that would benefit, the average benefit of treatment was 0.025. This tracks with the study's findings that the benefit of heparin was non-significant and inconclusive. Red outlines surround the words the patients that would benefit, which run over a line break, and the number 0.025.", "caption": "Section 6, paragraph “International Stroke Trial dataset”, PDF p. 8, top of the left column. The outlines mark the group to which the average is attributed and the number.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E009-b.png"}, {"alt": "Two bar charts side by side under the title: The printed averages are taken over the whole sample, not over the units that would benefit. Left panel, Horse Colic (surgery): a blue bar of height 0.147 labelled all 255 horses and an orange bar of height 0.604 labelled the 62 horses that would benefit. Right panel, Stroke Trial (heparin): a blue bar of height 0.025 labelled all 7264 patients and an orange bar of height 0.135 labelled the 1360 patients that would benefit. A legend says that blue is the average over the whole analysis sample (the printed number) and orange the average over the units with a positive fitted effect. The vertical axis is the average fitted benefit of treatment in units of the outcome Y.", "caption": "The printed numbers (blue) and the averages over the units that would benefit (orange), from the stored outputs of the released notebooks. The two panels have different vertical scales, since the outcome is $\\pm1$ for the Horse Colic data and a score between $-4$ and $3$ for the Stroke data.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E009-benefit.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T9", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T9"}, {"id": "P006-T28", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T28"}], "headline": "The numbers 0.147 and 0.025 are averages of the positive fitted effects over all horses and all patients in the analysis samples; among the horses and patients that would benefit, the average fitted benefit is about 0.604 and 0.135.", "historical_evidence_type": "lean_proof", "id": "P006-E009", "location": "Section 6, paragraph “Horse Colic dataset”, PDF p. 7, right column; paragraph “International Stroke Trial dataset”, PDF p. 8, top of the left column", "pdf_page": 7, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T28", "P006-T9"], "text": {"as_printed": "\\emph{PDF p.~7.} The fitted $\\mu_0(x)$ and $\\mu_1(x)$ show 62 horses benefiting from surgery and 146 being better off without surgery. On \\hl{the horses that would benefit}, the average benefit was \\hl{0.147}, which is fairly significant.\n\n\\emph{PDF pp.~7--8.} The fitted $\\mu_0(x)$ and $\\mu_1(x)$ showed 1360 patients benefiting from heparin and 5904 being better off without heparin. On \\hl{the patients that would benefit}, the average benefit of treatment was \\hl{0.025}. This tracks with the study's findings that the benefit of heparin was non-significant and inconclusive.", "corrected": "\\emph{PDF p.~7.} \\ldots \\hl{Under the fitted model, treating only the horses with positive fitted effects gives an average welfare gain of} 0.147 \\hl{over all 255 horses in the analysis sample, with zero contributions from the horses not treated.}\n\n\\emph{PDF pp.~7--8.} \\ldots \\hl{Under the fitted model, treating only the patients with positive fitted effects gives an average welfare gain of} 0.025 \\hl{over all 7264 patients in the analysis sample, with zero contributions from the patients not treated.}\n\nAlternatively, the two sentences keep their wording and the numbers become \\hl{0.604} and \\hl{0.135}. In general, if $r\\hl{>0}$ is the fraction of units with a positive fitted effect and $V^*$ is the \\hl{unconstrained} optimal utility, then the mean fitted benefit among those units is $V^*/r$.", "evidence": "The evidence is a recalculation from the stored outputs of the released notebooks, in exact arithmetic. Let $\\hat\\tau_i=\\mu_1(X_i)-\\mu_0(X_i)$ for the $n$ units of the analysis sample, let $k$ be the number of units with $\\hat\\tau_i>0$, and let $S$ be the sum of $\\hat\\tau_i$ over those units. The notebooks compute $V=S/n$. The average over the units that would benefit is\n\\[\nb=\\frac{S}{k}=V\\cdot\\frac{n}{k}.\n\\]\nThe stored values of $V$ are rounded to six decimals. Allowing for this rounding, I obtain the intervals in the table.\n\\begin{audittable}{lrrrr}{The printed numbers and the averages over the units that would benefit.}\nData set & $n$ & $k$ & $V=S/n$ (printed as 0.147 and 0.025) & $b=S/k$ \\\\\nHorse Colic & 255 & 62 & 0.146954 & between 0.604405 and 0.604410 \\\\\nStroke Trial & 7264 & 1360 & 0.025186 & between 0.134520 and 0.134526 \\\\\n\\end{audittable}\nNeither interval contains a number that rounds to 0.147 or to 0.025. The identity for $b$ and the two intervals are also proved in Lean, with the stored values and the sample sizes as hypotheses. For an independent least-squares fit of the Stroke model, I find the counts 1360 and 5904 again, with $V=0.025185778762$ and $b=0.134521688917$. The Horse Colic model was not fitted again. Its value of $b$ follows from the stored outputs.", "problem": "The two sentences describe 0.147 and 0.025 as averages over the units that would benefit, that is, over the 62 horses and the 1360 patients whose fitted effect $\\mu_1(x)-\\mu_0(x)$ is positive. The released notebooks compute a different quantity. They multiply each fitted effect by the indicator that it is positive and take the mean over the whole analysis sample: \\texttt{V\\_OPT = ((mu\\_1s - mu\\_0s)* T\\_OPT).mean()}, where \\texttt{T\\_OPT} is the indicator. The units with a fitted effect that is not positive enter this mean as zeros. The stored values are 0.146954 for the 255 horses and 0.025186 for the 7264 patients, and they are the $w_{\\mathrm{TT}}$ utilities of Table 1 (0.14695 and 0.02518). The average over the units that would benefit divides the same sums by 62 and by 1360. It is larger than the printed number by the factors $255/62\\approx4.1$ and $7264/1360\\approx5.3$.", "what_survives": "The utility $V$ as defined in Section 2, Table 1's interpretation as population utilities and regrets, and the theoretical results require no change because of this wording error. If the numbers are kept, then the remarks that follow them (``fairly significant'', and the comparison with the trial's finding on heparin) refer to gains averaged over the whole samples. These are fitted effects of a synthetic model, and my check does not establish clinical or statistical significance."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E009", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of Appendix E.2.1. The text reads: We trained a logistic regression model of Y on X, T, and the interaction term XT to estimate parameters beta hat, and used the resulting estimate to compute mu sub t of x = sigma(beta hat 0 + beta hat 1 transpose x + beta hat 2 t + beta hat 3 transpose x t). For the fitted model, the AUC was 0.9924, and the accuracy was 0.6824. When computing agent best responses and regret, we take this function mu sub t of x to be given as synthetic mean conditional potential outcomes. A red outline surrounds the words accuracy was 0.6824.", "caption": "Appendix E.2.1 (PDF p. 25). The outline marks the reported accuracy.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E010.png"}, {"alt": "Chart with three rows and a horizontal axis from 0.6 to 1.0, labelled Accuracy: share of the 255 horses counted as correctly classified. The title reads: The printed accuracy, 0.6824, is below every value that correct scoring allows. In the first row, labelled Printed in Appendix E.2.1 (the score the notebook computes), an orange diamond stands at 0.6824 with the note 174 of 255. In the second row, labelled Scored correctly: range implied by the stored outputs of the fit, a blue bar runs from 0.7804 to 0.9843. In the third row, labelled Scored correctly: fit reconstructed from the released coefficients, a blue dot stands at 0.9647 with the note 246 of 255.", "caption": "The accuracy printed in Appendix E.2.1 (orange diamond) beside the correctly scored accuracy of the same fit. The blue bar is the range from $199/255$ to $251/255$ that the score and the AUC stored in the released notebook allow. The blue dot is the value $246/255$ in the reconstruction of the fit from the released coefficients.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E010-accuracy.png"}], "formal_release": null, "formal_statements": [], "headline": "The accuracy $0.6824$ reported for the outcome model of the Horse Colic data is the share of horses that were predicted to live and did live, $174/255$, because the code compares predictions coded $0$ and $1$ with outcomes coded $-1$ and $1$; the correctly scored accuracy of the same fit is at least $199/255 \\approx 0.7804$, and it is $246/255 \\approx 0.9647$ in a reconstruction of the fit.", "id": "P006-E010", "location": "Appendix E.2.1, sentence “For the fitted model, the AUC was 0.9924, and the accuracy was 0.6824” (PDF p. 25); Horse Colic notebook of the replication files, eighth cell, methods fit_outcome and print_metrics", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Appendix~E.2.1.} For the fitted model, the AUC was 0.9924, and the accuracy was \\hl{0.6824}.\n\n\\emph{Notebook \\texttt{horse\\_colic\\_experiments.ipynb}, method \\texttt{fit\\_outcome} (an excerpt: the lines between those shown and the arguments of the constructor are omitted, and the print statement is broken here after the format string).}\n\\[\n\\begin{aligned}\n&\\texttt{target = self.get\\_target\\_column(df\\_input)}\\\\\n&\\texttt{clf\\_outcome = LogisticRegression(}\\ldots\\texttt{).fit(inputs, target)}\\\\\n&\\texttt{y\\_true = target}\\\\\n&\\texttt{print(\"Accuracy for outcome model: \\%f\"}\\\\\n&\\qquad\\texttt{\\% clf\\_outcome.score(inputs, }\\hl{\\texttt{df\\_input[self.y\\_column]}}\\texttt{))}\n\\end{aligned}\n\\]\n\n\\emph{The same notebook, method \\texttt{print\\_metrics} (an excerpt, with the same line break).}\n\\[\n\\begin{aligned}\n&\\texttt{y\\_true = self.get\\_target\\_column(df\\_input)}\\\\\n&\\texttt{print(\"Accuracy for outcome model: \\%f\"}\\\\\n&\\qquad\\texttt{\\% self.clf\\_outcome.score(inputs, }\\hl{\\texttt{df\\_input[self.y\\_column]}}\\texttt{))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Notebook.} The score is taken against the recoded outcome, \\texttt{target} in \\texttt{fit\\_outcome} and \\texttt{y\\_true} in \\texttt{print\\_metrics}:\n\\[\n\\begin{aligned}\n&\\texttt{clf\\_outcome.score(inputs, }\\hl{\\texttt{target}}\\texttt{)}\\\\\n&\\texttt{self.clf\\_outcome.score(inputs, }\\hl{\\texttt{y\\_true}}\\texttt{)}\n\\end{aligned}\n\\]\n\n\\emph{Appendix~E.2.1.} For the fitted model, the AUC was 0.9924, and the correctly scored accuracy was \\hl{at least $199/255$ (about $0.7804$)}. To obtain its exact accuracy, rescore the original fit on its training sample with the corrected call. The reconstruction of the fit gives $0.9647$; a replacement taken from a reconstructed or newly fitted model should be identified as such.", "evidence": "The evidence is an exact identity, a bound from the outputs stored in the notebook, and a reconstruction of the fit. For a horse with outcome $y \\in \\{-1, 1\\}$ and prediction $h \\in \\{0, 1\\}$, the code tests whether $h = y$, and a correct classification is $h = (y + 1)/2$. There are four cases.\n\\begin{audittable}{llllr}{The four cases of outcome and prediction, and the number of horses in each case in the reconstruction of the fit.}\nOutcome $y$ & Prediction $h$ & Counted by the code & Correctly classified & Horses \\\\\nlived ($1$) & lives ($1$) & yes & yes & 174 \\\\\nlived ($1$) & dies ($0$) & no & no & 4 \\\\\ndied ($-1$) & dies ($0$) & no & yes & 72 \\\\\ndied ($-1$) & lives ($1$) & no & no & 5 \\\\\n\\end{audittable}\nThe table shows that the score of the code is the number of correctly predicted survivors divided by $n$, and that the accuracy adds the number of correctly predicted deaths to the numerator. The sample has $n = 255$ horses, of which 178 lived and 77 died. The notebook stores the score $0.682353$, and $174$ is the only integer $k$ such that $k/255$ rounds to this value. Hence 174 survivors are predicted correctly and 4 are not. The stored AUC, $0.992412$, bounds the number $F$ of deaths that are predicted as survivals. Each of the $4F$ pairs of a misclassified survivor and a misclassified death loses at least half a credit in the AUC, and the total loss over the $178 \\cdot 77 = 13706$ pairs of a survivor and a death is at most $104.01$. This implies that $F \\le 52$. At least 25 deaths are then predicted correctly, and the accuracy of the fit is at least $199/255 \\approx 0.7804$. No such value rounds to $0.6824$.\n\nThe notebook does not store the fitted model or its intercept. I reconstruct the fit from the rounded coefficients that the notebook prints, with the intercept equal to the sum of the coefficients of the two age indicators. The reconstruction reproduces the stored score and the stored AUC to six decimals. Its counts are those in the last column of the table, and its accuracy is $246/255 \\approx 0.9647$. This value belongs to the reconstruction; it is not a rescoring of the original fit.", "problem": "Accuracy is the share of horses whose outcome the fitted classifier predicts correctly, and a correctly predicted death counts as much as a correctly predicted survival. The method \\texttt{get\\_target\\_column} recodes the outcome from $-1$ and $1$ to $0$ and $1$. The classifier is fitted to the recoded outcome, and its predictions take the values $0$ and $1$. Both accuracy statements then compare these predictions with the original outcome, coded $-1$ and $1$. Since a prediction of death is $0$ and a death is coded $-1$ in the original outcome, no death is ever counted as correctly classified. The score that the code prints is the share of horses that were predicted to live and did live. Whatever the model, this share cannot exceed the share of survivors, $178/255 \\approx 0.698$. The printed value $0.6824$ understates the fit of the outcome model to the sample, since the correctly scored accuracy is at least $199/255$. The AUC in the same sentence is computed against the recoded outcome and is not affected. The accuracy is printed and not used again, and no other number in the paper depends on it.", "what_survives": "The classifier is trained on the recoded outcome, and the AUC is computed against it. The accuracy appears only inside two print statements. The correction leaves unchanged the fitted probabilities, the synthetic potential outcomes and every utility, regret and treatment rate for the Horse Colic data, including the Horse Colic entries of Tables~1 and~2 and the figures. The theoretical results and the Stroke Trial calculations do not involve this diagnostic."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E010", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the definition of the Stroke Trial outcome. The text reads: outcome variable Y, we apply the negative of the scalarized composite outcome score from Kallus and Zhou (2018). Specifically, Y = -2 times the indicator of death, minus the indicator of recurrent stroke, minus 0.5 times the indicator of pulmonary embolism or intracranial bleeding, minus 0.5 times the indicator of other side effects, plus 2 times the indicator of full recovery at 6 months, plus the indicator of discharge within 14 days. This results in Y in the interval from -4 to 3. A red outline surrounds the last term, the indicator of discharge within 14 days.", "caption": "Appendix E.2.2 (PDF p. 25): the definition of the outcome score. The outline marks the last term, which the released code computes without the 14-day condition.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E011.png"}, {"alt": "Bar chart. The horizontal axis is days from randomization to discharge alive from hospital, from 0 to 60, and the vertical axis is the number of patients discharged alive on that day, from 0 to about 350. The title reads: The code credits 971 discharges after day 14 that the printed score excludes. Blue bars for days 0 to 14 rise from 20 patients on day 0 to between 167 and 268 patients a day from day 3 to day 13, with a peak of 339 on day 14; the legend reads Day 0 to 14: 2,921 patients, credited by the printed score and by the code. A vertical line after day 14 is labelled end of the 14-day window of the printed score. Orange bars for the later days fall from 139 patients on day 15 to fewer than 10 a day after day 36; the legend reads After day 14: 971 patients, credited by the code only. A note at the right says: 52 more patients after day 60 (latest: day 206).", "caption": "Day of discharge of the 3,892 patients of the analyzed sample who were discharged alive, in days from randomization. The released code gives each of them the point for discharge. The printed score gives it to the 2,921 patients discharged on days 0 to 14 (blue) and not to the 971 discharged later (orange). The 52 discharges after day 60 are not drawn.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E011-discharge-days.png"}], "formal_release": null, "formal_statements": [], "headline": "The released code credits every discharge alive from hospital, while the printed score credits only a discharge within 14 days; 971 of the 7,264 outcomes are one point higher than under the printed score, and the Stroke Trial results in the paper are those of the outcome as coded.", "id": "P006-E011", "location": "Appendix E.2.2, displayed outcome formula, last term (PDF p. 25); Stroke Trial notebook of the replication files, seventh and thirteenth cells; Stroke Trial columns of Table 1 (PDF p. 9) and Table 2 (PDF p. 27), Figure 1 (PDF p. 8) and the Stroke Trial panel of Figure 2 (PDF p. 27)", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Appendix~E.2.2.} For the outcome variable $Y$, we apply the negative of the scalarized composite outcome score from Kallus and Zhou (2018). Specifically,\n\\[\n\\begin{aligned}\nY = {} & -2\\,\\mathbb{1}(\\text{death}) - \\mathbb{1}(\\text{recurrent stroke}) - 0.5\\,\\mathbb{1}(\\text{pulmonary embolism or intracranial bleeding})\\\\\n& - 0.5\\,\\mathbb{1}(\\text{other side effects}) + 2\\,\\mathbb{1}(\\text{full recovery at 6 months}) + \\mathbb{1}(\\text{discharge within 14 days}).\n\\end{aligned}\n\\]\nThis results in $Y \\in [-4, 3]$.\n\n\\emph{Notebook \\texttt{stroke\\_trial\\_experiments.ipynb}, thirteenth cell.}\n\\[\n\\texttt{df['discharge'] = }\\hl{\\texttt{(df['DALIVE'] == 'Y')}}\\texttt{.astype(int)}\n\\]", "corrected": "\\emph{Notebook, thirteenth cell (one line of code, broken here after the first condition).}\n\\[\n\\begin{aligned}\n&\\texttt{df['discharge'] = ((df['DALIVE'] == 'Y')}\\\\\n&\\qquad\\hl{\\texttt{\\& df['DALIVED'].between(0, 14, inclusive='both')}}\\texttt{).astype(int)}\n\\end{aligned}\n\\]\nThe cells that follow are then run again. The recomputed values replace the fit statistics of Appendix~E.2.2, the counts and averages for the Stroke Trial data in Section~6, the Stroke Trial columns of Tables~1 and~2, Figure~1 and the Stroke Trial panel of Figure~2. The printed score stays as it is.", "evidence": "The evidence is an exact counterexample in the trial data and an exact refit. The patient on line 30 of the data file \\texttt{IST\\_corrected.csv} passes every filter of the notebook, was discharged alive on day 20, and has none of the other events of the score. For this patient, I have $Y = 1$ under the code and $Y = 0$ under the printed score. For every patient of the sample who was discharged alive, the day of discharge is recorded as an integer from 0 to 206. Each of the 971 late discharges lowers the outcome by exactly one point under the printed score, and the sum of the outcomes falls from 2,765 to 1,794.\n\nI then compare two least-squares fits in exact rational arithmetic, one for the outcome as coded and one with the discharge term restricted to days 0 to 14. The sample, the covariates, the treatment coding and the calculation of the conditional means are the same in both. For the first fit, $\\gamma_{\\mathrm{marg}}$ and the individual and cumulative regrets match the stored notebook values to within $10^{-12}$. Between the two fits, the optimal utility moves from $0.025186$ to $0.024790$, the number of patients with a positive fitted effect of treatment moves from 1,360 to 1,313, and the optimal decision changes for 355 patients. The table sets the Stroke Trial columns of Table~2 beside the recomputed values; the rows other than $w_{\\mathrm{TO}}$ also appear in Table~1. The printed table truncates some of its entries (see \\fid{P006-E018}).\n\\begin{audittable}{llrrr}{Stroke Trial columns of Table~2. For each reward function, the first row gives the entries as printed, and the second row gives the values recomputed with the discharge term restricted to days 0 to 14.}\nReward function & Entry & Utility & Regret & Treat rate \\\\\n$w_{\\mathrm{ATO}}$ & printed & $0.00004$ & $0.0251$ & $0.0001$ \\\\\n & recomputed & $0.00013$ & $0.0247$ & $0.0001$ \\\\\n$w_{\\mathrm{ATT}}$ & printed & $0.00013$ & $0.0250$ & $0.0001$ \\\\\n & recomputed & $0.00013$ & $0.0247$ & $0.0001$ \\\\\n$w_{\\mathrm{TO}}$ & printed & $-0.08278$ & $0.1079$ & $0.6689$ \\\\\n & recomputed & $-0.07492$ & $0.0997$ & $0.6195$ \\\\\n$w_{\\mathrm{TT}}$ & printed & $0.02518$ & $0.000$ & $0.1872$ \\\\\n & recomputed & $0.02479$ & $0.000$ & $0.1808$ \\\\\n$w_{\\mathrm{TT}}$ (no info) & printed & $-0.04888$ & $0.0741$ & $0.4829$ \\\\\n & recomputed & $-0.04912$ & $0.0739$ & $0.4774$ \\\\\n$w_{\\mathrm{TT}}$ (demographic info) & printed & $-0.06392$ & $0.0891$ & $0.5041$ \\\\\n & recomputed & $-0.06287$ & $0.0877$ & $0.4934$ \\\\\n\\end{audittable}\nThe refit changes the discharge term alone and keeps every other coding of the notebook as released. It uses the released notebook and the corrected second version of the trial data, and it does not speak for any earlier version of the code.", "problem": "The last term of the printed score adds one point for a discharge alive from hospital within 14 days of randomization. The seventh cell of the notebook states the score with the same condition. The code sets the discharge indicator from the variable \\texttt{DALIVE} (discharged alive from hospital) alone, and it never reads \\texttt{DALIVED}, the number of days from randomization to that discharge. A patient discharged alive on day 20, or on day 206, receives the point. In the analyzed sample 3,892 patients were discharged alive, 2,921 of them on days 0 to 14 and 971 later. For these 971 patients, of 7,264, the coded outcome is one point above its value under the printed score. The linear model is fitted to this outcome. The synthetic potential outcomes, the Stroke Trial columns of Tables~1 and~2, Figure~1 and the Stroke Trial panel of Figure~2 are all computed from that fit. The numbers published for the Stroke Trial data are therefore those of an experiment whose outcome differs from the one that the appendix defines. The printed score is a coherent definition, and the repair is to make the code follow it. To change the text to ``discharged alive'' would replace the stated 14-day endpoint by a different one.", "what_survives": "The theoretical results, the Horse Colic experiment, and the sample, covariates and treatment allocation of the Stroke Trial experiment do not depend on the discharge term. Every quantity for the Stroke Trial data that is computed from the fitted outcomes has to be recomputed. The changes are small, and the qualitative findings hold in the refit. The optimal utility remains positive, and the regret of $w_{\\mathrm{TT}}$ with full information is exactly zero. The regret with demographic information, $0.0877$, still exceeds the regret with no information, $0.0739$, and regret is still amplified for the individual features Age, Delay, Sleep symptoms and CT done. Some entries under the printed rule do not change (see \\fid{P006-E001}): $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ each still treat one patient of 7,264, the utility of $w_{\\mathrm{ATT}}$ still rounds to $0.00013$, and the last four cumulative regrets of Figure~1 remain zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E011", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the definition of the Stroke Trial outcome. The text reads: outcome variable Y, we apply the negative of the scalarized composite outcome score from Kallus and Zhou (2018). Specifically, Y = -2 times the indicator of death, minus the indicator of recurrent stroke, minus 0.5 times the indicator of pulmonary embolism or intracranial bleeding, minus 0.5 times the indicator of other side effects, plus 2 times the indicator of full recovery at 6 months, plus the indicator of discharge within 14 days. This results in Y in the interval from -4 to 3. A red outline surrounds the third term, -0.5 times the indicator of pulmonary embolism or intracranial bleeding.", "caption": "Appendix E.2.2 (PDF p. 25): the definition of the outcome score. The outline marks the term for pulmonary embolism or intracranial bleeding, which the released code computes as the sum of two indicators.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E012.png"}], "formal_release": null, "formal_statements": [], "headline": "The released code adds the indicators of pulmonary embolism and of intracranial bleeding where the printed score has the indicator of either event; no patient in the analyzed sample has both events, and no outcome changes.", "id": "P006-E012", "location": "Appendix E.2.2, displayed outcome formula, third term (PDF p. 25); Stroke Trial notebook of the replication files, tenth cell", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Appendix~E.2.2.}\n\\[\n\\begin{aligned}\nY = {} & -2\\,\\mathbf{1}(\\text{death}) - \\mathbf{1}(\\text{recurrent stroke}) - 0.5\\,\\mathbf{1}(\\text{pulmonary embolism or intracranial bleeding})\\\\\n& - 0.5\\,\\mathbf{1}(\\text{other side effects}) + 2\\,\\mathbf{1}(\\text{full recovery at 6 months}) + \\mathbf{1}(\\text{discharge within 14 days}).\n\\end{aligned}\n\\]\n\n\\emph{Notebook \\texttt{stroke\\_trial\\_experiments.ipynb}, tenth cell.}\n\\[\n\\texttt{df['emb\\_bleeding'] = df['PE14'] }\\hl{\\texttt{+}}\\texttt{ df['H14']}\n\\]", "corrected": "\\emph{Notebook, tenth cell.}\n\\[\n\\texttt{df['emb\\_bleeding'] = }\\hl{\\texttt{((df['PE14'] > 0) | (df['H14'] > 0)).astype(int)}}\n\\]\nThe printed score stays as it is.", "evidence": "The evidence is a recalculation from the trial data with the filters of the notebook. Of the 7,264 patients in the analyzed sample, 44 have \\texttt{PE14} equal to one, 68 have \\texttt{H14} equal to one, and none has both. Replacing the sum by the indicator of the union leaves all 7,264 outcomes exactly unchanged, and the range of the outcome remains $[-4, 3]$.", "problem": "The printed term is the indicator of the union of two events, and it carries one penalty of $0.5$. The code forms the sum of the two indicators \\texttt{PE14} and \\texttt{H14}. A patient with a pulmonary embolism and intracranial bleeding within 14 days would then receive a penalty of $1$. The mismatch has no effect on the paper: no patient in the analyzed sample has both events, and the sum equals the indicator for every patient.", "what_survives": "Every Stroke Trial outcome is unchanged, so no fitted value or result needs to be recomputed for this correction. The code should follow the printed score all the same, since the sum and the union differ on any data in which the two events occur together."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E012", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Definition 1 (Uniform Rank Preservation). The text reads: A set of score functions w sub k, k from 1 to K, preserves treatment effect ordering uniformly over X if for all j, k in 1 to K, tau superscript j of x is at least tau superscript k of x for all x in X implies that the maximum over pi sub j of E of w sub j of X superscript k, T superscript pi sub j, Y superscript j is at least the maximum over pi sub k of E of w sub k of X superscript k, T superscript pi sub k, Y superscript k. A red outline surrounds X superscript k in the score of agent j.", "caption": "Definition 1, PDF p. 5, right column. The outline marks the covariates of agent $k$ in the score of agent $j$. Definition 2, below it on the page, has the same display.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013.png"}, {"alt": "Crop of the start of a proof. The text reads: Proof. Suppose for agents j and k, Pi sub j and Pi sub k are both unconstrained. For w sub k as defined above. First display: the maximum over pi sub j of E of w sub j of X superscript (j), T superscript pi sub j, Y superscript (k) equals n sub j times E of tau superscript (j) of X superscript (j) times the indicator that it is positive. Second display: the same for agent k, with every superscript (k). A red outline surrounds Y superscript (k) in the first display.", "caption": "Proof of Proposition 4, PDF p. 19, first two displays. The outline marks the outcomes of agent $k$ in the score of agent $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-b.png"}, {"alt": "Crop of the start of a proof. The text reads: Proof. By the tower property. A two-line display begins with E of w superscript g sub TT of Y, T superscript pi, X, and equals n times the probability that T sub i equals 1 times a conditional expectation of Y sub i minus mu-hat-zero of X sub i, times g of X sub i. A red outline surrounds the arguments Y, T superscript pi, X.", "caption": "Proof of Theorem 1, PDF p. 19. The outline marks the arguments of the score, written in the order $(\\mathbf Y,\\mathbf T^\\pi,\\mathbf X)$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-c.png"}, {"alt": "Crop of three lines of the theorem and a display. The theorem begins: Theorem 2 (Ranking Desiderata Satisfied). Let P sub X superscript (k) be absolutely continuous with respect to P sub X sub 0, and let g sub k equal one over n sub k times the derivative of P sub X sub 0 with respect to P sub X superscript (k). The display, numbered (2), gives w sub k as the sum over i from 1 to n sub k of Y sub i superscript (k) minus mu-hat-zero of X sub i superscript (k), times T sub i, times g sub k of X superscript (k). A red outline surrounds g sub k of X superscript (k), which has no index i.", "caption": "Restatement of Theorem 2 with equation (2), PDF p. 20. The outline marks the weight $g_k(X^{(k)})$ inside the sum over $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-d.png"}, {"alt": "Crop of part of a proof. The text reads: Let Pi sub j, Pi sub k both be unconstrained. Then two displays give the maximal expected scores of agents j and k as E of tau of X sub 0 times the indicator that it is positive, with superscripts (j) and (k); in the first, the score of agent j is written with Y superscript (k), inside a red outline. Below: Definition 1 is immediately satisfied, since tau superscript (j) of x is at least tau superscript (k) of x for all x in X implies E of tau superscript (k) of X sub 0 times the indicator that it is positive, is at least the same expression with superscript (k). A second red outline surrounds the left expression.", "caption": "Proof of Theorem 2, PDF p. 20. The outlines mark the outcomes of agent $k$ in the score of agent $j$, and the left side of the comparison, which has $\\tau^{(k)}$ where $\\tau^{(j)}$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-e.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T14", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T14"}, {"id": "P006-T30", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T30"}], "headline": "In the ranking definitions and their proofs, each agent’s score takes that agent’s own covariates and outcomes, in the order $w(\\mathbf X,\\mathbf T,\\mathbf Y)$; the weight in equation (2) is $g_k(X_i^{(k)})$; and the inequality from which Definition 1 follows has agent $j$ on its left side.", "id": "P006-E013", "location": "Definitions 1 and 2, PDF p. 5, right column; Appendix C, proofs of Proposition 4 and Theorem 1, PDF p. 19; equation (2) and proof of Theorem 2, PDF p. 20", "pdf_page": 5, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T14", "P006-T30"], "text": {"as_printed": "\\emph{Definitions 1 and 2 (PDF p.~5), right side of each implication.}\n\\[\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\hl{\\mathbf X^k},\\mathbf T^{\\pi_j},\\mathbf Y^j)]\\ \\ge\\ \\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^k,\\mathbf T^{\\pi_k},\\mathbf Y^k)]\n\\]\n\n\\emph{Proof of Proposition 4 (PDF p.~19), first and last displays.}\n\\[\n\\begin{aligned}\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(k)}})]&=n_jE[\\tau^{(j)}(X^{(j)})\\,\\mathbf{1}(\\tau^{(j)}(X^{(j)})>0)],\\\\\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(k)}})]&<\\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^{(k)},\\mathbf T^{\\pi_k},\\mathbf Y^{(k)})].\n\\end{aligned}\n\\]\n\n\\emph{Proof of Theorem 2 (PDF p.~20), first display.}\n\\[\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(k)}})]=E[\\tau^{(j)}(X_0)\\,\\mathbf{1}(\\tau^{(j)}(X_0)>0)],\n\\]\n\n\\emph{Proof of Theorem 1 (PDF p.~19), left side of three displays.} $E[w^g_{\\mathrm{TT}}(\\hl{\\mathbf Y,\\mathbf T^\\pi,\\mathbf X})]$\n\n\\emph{Equation (2) (PDF p.~20).}\n\\[\nw_k(\\mathbf X^{(k)},\\mathbf T^{\\pi_k},\\mathbf Y^{(k)})=\\sum_{i=1}^{n_k}(Y_i^{(k)}-\\hat\\mu_0(X_i^{(k)}))T_i^{\\pi_k}\\hl{g_k(X^{(k)})}\\qquad(2)\n\\]\n\n\\emph{Proof of Theorem 2 (PDF p.~20).} Definition 1 is immediately satisfied, since $\\tau^{(j)}(x)\\ge\\tau^{(k)}(x)$ for all $x\\in\\mathcal X$ implies $\\hl{E[\\tau^{(k)}(X_0)\\,\\mathbf{1}(\\tau^{(k)}(X_0)>0)]}\\ge E[\\tau^{(k)}(X_0)\\,\\mathbf{1}(\\tau^{(k)}(X_0)>0)]$.", "corrected": "\\emph{Definition 1 (Uniform Rank Preservation).} A set of score functions $\\{w_k\\}_{k=1}^K$ preserves treatment effect ordering uniformly over $\\mathcal X$ if for all $j,k\\in\\{1,\\ldots,K\\}$,\n\\[\n\\tau^j(x)\\ge\\tau^k(x)\\ \\ \\forall x\\in\\mathcal X\\ \\implies\\ \\max_{\\pi_j\\in\\Pi_j}E[w_j(\\hl{\\mathbf X^j},\\mathbf T^{\\pi_j},\\mathbf Y^j)]\\ \\ge\\ \\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^k,\\mathbf T^{\\pi_k},\\mathbf Y^k)].\n\\]\n\n\\emph{Definition 2 (Relative Rank Preservation).} A set of score functions $\\{w_k\\}_{k=1}^K$ preserves treatment effect ordering relative to a reference population $P_{X_0}$ with support $\\mathcal X$ if for all $j,k\\in\\{1,\\ldots,K\\}$,\n\\[\nE[\\tau^j(X_0)]\\ge E[\\tau^k(X_0)]\\ \\implies\\ \\max_{\\pi_j\\in\\Pi_j}E[w_j(\\hl{\\mathbf X^j},\\mathbf T^{\\pi_j},\\mathbf Y^j)]\\ \\ge\\ \\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^k,\\mathbf T^{\\pi_k},\\mathbf Y^k)].\n\\]\n\n\\emph{Appendix C (PDF pp.~19--20).} In the three displays for agent $j$, the score is $w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(j)}})$. In the proof of Theorem 1, the left sides read $E[w^g_{\\mathrm{TT}}(\\hl{\\mathbf X,\\mathbf T^\\pi,\\mathbf Y})]$. Equation (2) reads\n\\[\nw_k(\\mathbf X^{(k)},\\mathbf T^{\\pi_k},\\mathbf Y^{(k)})=\\sum_{i=1}^{n_k}(Y_i^{(k)}-\\hat\\mu_0(X_i^{(k)}))T_i^{\\pi_k}\\hl{g_k(X_i^{(k)})}.\\qquad(2)\n\\]\nIn the proof of Theorem 2: Definition 1 is immediately satisfied, since $\\tau^{(j)}(x)\\ge\\tau^{(k)}(x)$ for all $x\\in\\mathcal X$ implies $\\hl{E[\\tau^{(j)}(X_0)\\,\\mathbf{1}(\\tau^{(j)}(X_0)>0)]}\\ge E[\\tau^{(k)}(X_0)\\,\\mathbf{1}(\\tau^{(k)}(X_0)>0)]$.", "evidence": "The evidence is a check of the pages against the paper's own definitions, and a derivation. Appendix C.1 (PDF p.~18) defines $w_k$ on agent $k$'s own three vectors of length $n_k$, and Reward Function 4 (PDF p.~6) has $g(X_i)$ inside the sum and the arguments in the order $(\\mathbf X,\\mathbf T^\\pi,\\mathbf Y)$. With the corrected indices, the proof on PDF p.~20 has $E[w_k]=n_kE[\\tau^{(k)}(X_i^{(k)})\\pi_k(X_i^{(k)})g_k(X_i^{(k)})]=E[\\tau^{(k)}(X_0)\\pi_k(X_0)]$, where the second equality is the change of measure under the condition of Theorem 2 as corrected in \\fid{P006-E005}. Over unconstrained policies the maximum is $E[\\max(\\tau^{(k)}(X_0),0)]$. If $\\tau^{(j)}(x)\\ge\\tau^{(k)}(x)$ for all $x$, then $\\max(\\tau^{(j)}(x),0)\\ge\\max(\\tau^{(k)}(x),0)$ for all $x$. Integrating against $P_{X_0}$, I obtain the inequality with $j$ on the left and $k$ on the right.", "problem": "These are slips of index and of argument. In Definitions 1 and 2 the score of agent $j$ is evaluated on the covariates of agent $k$. In three displays of Appendix C it is evaluated on the outcomes of agent $k$. When $n_j\\ne n_k$, the arguments then have different lengths. In the proof of Theorem 1 the arguments of the score are written in the reverse order. In equation (2) the weight is applied to $X^{(k)}$, without the index $i$ of the sum. In the last passage the two sides of the inequality are identical, and the statement to be derived has $\\tau^{(j)}$ on the left.", "what_survives": "These slips do not change the intended ranking results. The intended formulas are fixed by Appendix C.1 and Reward Function 4, and the proofs go through with the indices corrected under the estimator and absolute-continuity conditions of \\fid{P006-E002} and \\fid{P006-E005}."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E013", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the six-line paragraph beginning 'Assumption 4 is stronger than Assumption 2'. In its last sentence, two red boxes mark the words 'Assumption 4' after 'Intuitively,' and the words 'allocate treatment to single values of u tilde in U'.", "caption": "Appendix D.3, paragraph after Assumption 4, PDF p. 23. The first mark is the assumption named in the sentence beginning “Intuitively”. The second mark is the description of the agent’s policy.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E014.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T17", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T17"}], "headline": "The sentence beginning “Intuitively” refers to Assumption 2 and not Assumption 4, and the agent concentrates the estimate $\\hat\\mu_0^\\pi(x)$ on one type $\\tilde u$ by leaving only that type untreated, not by treating only that type.", "id": "P006-E014", "location": "Appendix D.3, paragraph after Assumption 4, sentence beginning “Intuitively”, PDF p. 23", "pdf_page": 23, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T17"], "text": {"as_printed": "Assumption 4 is stronger than Assumption 2 in the sense that Assumption 2 is not sufficient to bound the regret when the principal estimates the mean conditional untreated potential outcome from the agent's data using $\\hat\\mu_0^\\pi(X)$. Furthermore, the bound in Assumption 4 implies that Assumption 2 is satisfied with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$. Intuitively, \\hl{Assumption 4} is not sufficient to bound the regret when the principal uses $\\hat\\mu_0^\\pi(X)$ because the agent can choose $\\pi$ to \\hl{allocate treatment to single values of $\\tilde u\\in\\mathcal{U}$}, such that $\\hat\\mu_0^\\pi(x)$ ends up matching a single value $\\mu_0(x,\\tilde u)$.", "corrected": "Intuitively, \\hl{Assumption 2} is not sufficient to bound the regret when the principal uses $\\hat\\mu_0^\\pi(X)$, because the agent can \\hl{leave only a selected type $\\tilde u$ of positive conditional probability untreated}, so that $\\hat\\mu_0^\\pi(x)=\\mu_0(x,\\tilde u)$.", "evidence": "This is a check of the page and a short computation. If $U$ takes finitely many values with conditional probabilities $p_u$ given $X=x$, then $\\hat\\mu_0^\\pi(x)=\\sum_u\\mu_0(x,u)p_u(1-\\pi(x,u))/\\sum_up_u(1-\\pi(x,u))$. For three equally likely types with $\\mu_0(x,u)$ equal to $0$, $1$ and $3$, I find $\\hat\\mu_0^\\pi(x)=2$ when only the first type is treated, and $\\hat\\mu_0^\\pi(x)=0$ when only the first type is left untreated.", "problem": "The sentence means Assumption 2: the preceding sentence says that Assumption 2 is not sufficient, and Theorem 5, which follows, shows that Assumption 4 is sufficient. Since the estimate $\\hat\\mu_0^\\pi(x)=E[Y_i\\mid T^\\pi=0,X=x]$ is a mean over the untreated units, it equals $\\mu_0(x,\\tilde u)$ when the type $\\tilde u$ is the only one left untreated, not when it is the only one treated.", "what_survives": "No result depends on this sentence; the hypothesis of Theorem 5 is the subject of \\fid{P006-E006}."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E014", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of four lines of text and one displayed formula. The text reads: When simulating information asymmetry, we compute empirical estimates of gamma marg over the data. For each individual feature in Figures 1 and 3, let X represent the individual feature, and let U represent the set of all other features. We use the full regression result as mu 0 of (x, u). To calculate mu 0 of x where x represents the individual feature, we take the empirical conditional mean over the dataset. The display sets mu 0 of x equal to a fraction. Its numerator is the sum over i from 1 to n of mu 0 of (X sub i, U sub i) times the indicator of X sub i = x. Its denominator is the indicator of X sub i = x alone, and a red outline surrounds it.", "caption": "Appendix E.4 (PDF p. 26). The outline marks the denominator of the empirical conditional mean, a single indicator with no sum over $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E015.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T22", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T22"}], "headline": "The empirical conditional mean $\\mu_0(x)$ divides by the number of observations with $X_i = x$, which is $\\sum_{i=1}^n \\mathbf{1}(X_i = x)$; the printed denominator is a single indicator with no sum.", "id": "P006-E015", "location": "Appendix E.4, first displayed formula (PDF p. 26)", "pdf_page": 26, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T22"], "text": {"as_printed": "To calculate $\\mu_0(x)$ where $x$ represents the individual feature, we take the empirical conditional mean over the dataset:\n\\[\n\\mu_0(x) = \\frac{\\sum_{i=1}^n \\mu_0(X_i, U_i)\\,\\mathbf{1}(X_i = x)}{\\hl{\\mathbf{1}(X_i = x)}}.\n\\]", "corrected": "To calculate $\\mu_0(x)$ where $x$ represents the individual feature, we take the empirical conditional mean over the dataset: for every value $x$ such that $\\sum_{i=1}^n \\mathbf{1}(X_i = x) > 0$,\n\\[\n\\mu_0(x) = \\frac{\\sum_{i=1}^n \\mu_0(X_i, U_i)\\,\\mathbf{1}(X_i = x)}{\\hl{\\sum_{i=1}^n} \\mathbf{1}(X_i = x)}.\n\\]", "evidence": "The evidence is a derivation and a check of the code. Let $m_i = \\mu_0(X_i, U_i)$, and let $N(x) = \\sum_{i=1}^n \\mathbf{1}(X_i = x)$ be the number of observations with $X_i = x$. Suppose that $N(x) > 0$. Conditioned on the event $X = x$, the empirical distribution of the data puts mass $1/N(x)$ on each of these observations. The conditional mean of $m$ is then\n\\[\n\\frac{1}{N(x)} \\sum_{i=1}^n m_i\\,\\mathbf{1}(X_i = x) .\n\\]\nFor two observations with $X_i = x$ and fitted means $1$ and $3$, I obtain $2$ from this formula and $4$ from the printed quotient. Both released notebooks compute the group mean, with the statement \\texttt{grouped = df\\_train.groupby(x\\_obs).mean()}, and the values of $\\gamma_{\\mathrm{marg}}$ and the regrets in the paper were computed with the correct denominator.", "problem": "The denominator is one indicator, and no sum binds its index $i$, so the display is not a well-defined formula. If the indicator is read at an observation with $X_i = x$, the display divides the sum of the fitted means by $1$ and returns their sum instead of their mean. The text calls the quantity an empirical conditional mean, which fixes the intended denominator: the number of observations whose feature equals $x$.", "what_survives": "No number in the paper changes. The second display of Appendix~E.4, which defines $\\gamma_{\\mathrm{marg}}$ as the average of $|\\mu_0(X_i) - \\mu_0(X_i, U_i)|$ over the dataset, stands as printed. It evaluates $\\mu_0$ only at observed values of the feature, where the count is positive."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E015", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of seven lines of the left column. The text reads: Status quo: average treated outcome. We begin by analyzing regret under the current common quality measure that rewards the average treated outcome, as done by the mortality measures in the New York and Philidelphia health departments in the 1990s analyzed by (Dranove et al., 2003), and in many CMS quality measures (Institute, 2022). Lazzarini et al. (2022) also. A red outline surrounds the words Philidelphia health departments.", "caption": "Section 3, paragraph “Status quo: average treated outcome”, PDF p. 4, left column. The outline marks the phrase that names the second jurisdiction and its issuing body.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E016.png"}], "formal_release": null, "formal_statements": [], "headline": "The report cards studied by Dranove et al. (2003) were published by the New York Department of Health and, for the second jurisdiction, by the Pennsylvania Health Care Cost Containment Council, not by a Philadelphia health department.", "id": "P006-E016", "location": "Section 3, paragraph “Status quo: average treated outcome”, PDF p. 4, left column", "pdf_page": 4, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We begin by analyzing regret under the current common quality measure that rewards the average treated outcome, as done by the mortality measures in the New York and \\hl{Philidelphia health departments} in the 1990s analyzed by (Dranove et al., 2003), and in many CMS quality measures (Institute, 2022).", "corrected": "\\ldots as done by the \\hl{mortality report cards published by the New York Department of Health and the Pennsylvania Health Care Cost Containment Council} in the 1990s analyzed by (Dranove et al., 2003), and in many CMS quality measures (Institute, 2022).", "evidence": "The evidence is a check of the page and of the cited article (Dranove, Kessler, McClellan and Satterthwaite, \\emph{Journal of Political Economy} 111(3), 2003, pp.~555--588), which I read in the copy of the journal version posted by one of its authors. Its abstract (p.~556) names ``New York and Pennsylvania''. Its Section II has ``the N.Y. Department of Health publicly released'' (p.~559) and ``the Pennsylvania Health Care Cost Containment Council published'' (p.~560).", "problem": "The second jurisdiction of the cited study is the state of Pennsylvania, and the body that published its report cards was a council, not a health department. The printed word is also a misspelling of Philadelphia. If only the spelling were corrected, the attribution would still be wrong.", "what_survives": "The sentence is a historical example, and nothing depends on it. The regret results of Section 3 are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E016", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the right column. The text reads: untreated potential outcomes is bounded. Assumption 2 (Bounded Heteogeneity). The effect of U sub i on the conditional untreated potential outcome is bounded as E of the absolute value of mu-zero of X sub i minus mu-zero of X sub i and U sub i, at most gamma sub marg. Note that mu-zero of X sub i equals E of mu-zero of X sub i and U sub i given X sub i. Thus, the het-. A red outline surrounds the word Heteogeneity.", "caption": "Assumption 2, PDF p. 6, right column. The outline marks the misspelled word in the name of the assumption.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017.png"}, {"alt": "Crop of a bar chart titled gamma sub marg per feature. One orange bar labelled None and twenty blue bars, with heights between about 0.44 and 0.65, carry rotated labels: Conciousness, Visuospatial disorder, Hemianopia, Stroke subtype, Dysphasia, Age, Leg/foot deficit, Brainstem/cerebellar signs, Face deficit, Arm/hand deficit, Other deficit, Atrial fibrillation, Sex, Blood pressure, Delay, Infact visible, CT done, Sleep symptoms, Aspirin prior, Heparin prior. Red outlines surround the labels Conciousness and Infact visible.", "caption": "Figure 1, top panel, PDF p. 8. The outlines mark the feature labels “Conciousness” and “Infact visible”. Figure 2 (PDF p. 27) has the same labels.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-b.png"}, {"alt": "Crop of a centered two-line title in bold: Supplementary Materials for Operationalizing Counterfactual Metrics: Incentives, Ranking, and Information Asymmetry. A red outline surrounds the word Operationalizing.", "caption": "Title of the supplement, PDF p. 13. The outline marks the word at which the title departs from the title of the paper, “On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry”.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-c.png"}, {"alt": "Crop of five lines of a numbered list. Item 1, Auxiliary untreated data, says that the principal has an auxiliary dataset of X prime sub j, T prime sub j, Y prime sub j, and may produce an unbiased estimator from it if: (a) mu prime sub zero of x, equal to E of Y prime sub j of 0 given X prime sub j equals x, is identifiable from this dataset. A set of sufficient conditions for this would be if ignorability was satisfied, and the probability that T prime sub j equals 0 given X sub j equals x is positive for all x in X, and SUTVA was satisfied. A red outline surrounds X sub j, which has no prime.", "caption": "Appendix A.1, item 1(a), PDF p. 13. The outline marks $X_j$ in the positivity condition; the other covariates of the item carry a prime.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-d.png"}, {"alt": "Crop of three lines. The text reads: outcomes. However, medical treatment protocols are also generally heavily codified, with online clinical decision support tools becoming increasingly widely used upt (2022). Still, if the regulatory agency does not have full access to patents' medical records, then information asymmetry will arise. Red outlines surround upt (2022) and the word patents'.", "caption": "Appendix F, PDF p. 28. The outlines mark the citation printed as “upt (2022)” and the word “patents”'.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-e.png"}], "formal_release": null, "formal_statements": [], "headline": "The text, the figure labels and the reference list contain the misspellings and typographical slips listed below; none of them affects a result.", "id": "P006-E017", "location": "Assumption 2, PDF p. 6, right column; Figures 1–3, PDF pp. 8, 27 and 28; reference list, PDF p. 11; title of the supplement and Appendix A.1, PDF p. 13; PDF pp. 17, 23, 26 and 28", "pdf_page": 6, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item PDF p.~6: Assumption 2 (Bounded \\hl{Heteogeneity}).\n\\item Figures 1 and 2 (PDF pp.~8 and 27), feature labels: \\hl{Conciousness}; \\hl{Infact} visible.\n\\item PDF p.~11, reference list: \\hl{Jersey} Neyman.\n\\item PDF p.~13, title of the supplement: Supplementary Materials for \\hl{Operationalizing Counterfactual Metrics}: Incentives, Ranking, and Information Asymmetry\n\\item PDF p.~13, Appendix A.1, item 1(a): $P(T'_j=0\\mid\\hl{X_j}=x)>0\\ \\ \\forall x\\in\\mathcal X$\n\\item PDF p.~17, proof of Proposition 1: We construct \\hl{an} family of distributions for which the regret is unbounded.\n\\item PDF p.~23: bounds from \\hl{senstivity} analysis on expected outcome functions\n\\item PDF p.~26: agent's \\hl{preceived} $\\mu_0(x)$ function\n\\item Figures 2 and 3 (PDF pp.~27 and 28), feature label: \\hl{Abdomcentesis} total protein.\n\\item PDF p.~28: increasingly widely used \\hl{upt (2022)}; access to \\hl{patents'} medical records.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Assumption 2 (Bounded \\hl{Heterogeneity}).\n\\item Feature labels: \\hl{Consciousness}; \\hl{Infarct} visible; \\hl{Abdominocentesis} total protein.\n\\item Reference list: \\hl{Jerzy} Neyman.\n\\item Supplementary Materials for \\hl{On Counterfactual Metrics for Social Welfare}: Incentives, Ranking, and Information Asymmetry\n\\item Appendix A.1, item 1(a): $P(T'_j=0\\mid\\hl{X'_j}=x)>0\\ \\ \\forall x\\in\\mathcal X$\n\\item We construct \\hl{a} family of distributions for which the regret is unbounded.\n\\item bounds from \\hl{sensitivity} analysis on expected outcome functions\n\\item agent's \\hl{perceived} $\\mu_0(x)$ function\n\\item increasingly widely used \\hl{(Wolters Kluwer, 2022)}; access to \\hl{patients'} medical records.\n\\end{itemize}", "evidence": "The evidence is a check of the pages. For the feature labels I compared the documentation of the two data sets. The Stroke variables are the patient's conscious state and whether an infarct is visible. The label ``abdomcentesis'' is printed in the variable list of the Horse Colic data, which the paper follows.", "problem": "Each item has an evident intended reading. Three of them are not misspellings. The supplement is headed with a title that differs from the title of the paper on PDF p.~1. In Appendix A.1, item 1(a), the positivity condition is written with $X_j$, although the item concerns the auxiliary data, whose covariate is $X'_j$ everywhere else in the item. On PDF p.~28 a citation is printed as ``upt (2022)'', which identifies no source. The reference list (PDF p.~9) has one entry of 2022 on UpToDate, published by Wolters Kluwer. The misspelling ``Philidelphia'' on PDF p.~4 is corrected in \\fid{P006-E016}.", "what_survives": "Nothing depends on these slips. No numerical result changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E017", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Table 2 with its caption: Utility and regret comparisons for different reward functions. For each reward function w, we report utility, regret, and the realized treatment rate. The table has six rows, w ATO, w ATT, w TO, w TT, w TT (no info) and w TT (demographic info), and for each of the Horse Colic and Stroke Trial datasets the columns Utility, Regret and Treat rate. The Horse Colic entries of the last two rows are 0.09761, 0.0493 and 0.6274 in both rows, and a red outline surrounds the two values 0.6274. The Stroke Trial entries are 0.00004, 0.0251, 0.0001 for w ATO; 0.00013, 0.0250, 0.0001 for w ATT; -0.08278, 0.1079, 0.6689 for w TO; 0.02518, 0.000, 0.1872 for w TT; -0.04888, 0.0741, 0.4829 for w TT (no info); and -0.06392, 0.0891, 0.5041 for w TT (demographic info).", "caption": "Table 2 (PDF p. 27). The outline marks the Horse Colic treatment rates of the last two rows, printed as 0.6274.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E018.png"}, {"alt": "Crop of Table 1 with its caption: Utility and regret comparisons for different reward functions. For each reward function w, we report utility, regret, and the realized treatment rate. The table has five rows, w ATO, w ATT, w TT, w TT (no info) and w TT (demographic info), with the same columns as Table 2. The Horse Colic entries of the last two rows are 0.09761, 0.0493 and 0.6275 in both rows, and a red outline surrounds the two values 0.6275. The Stroke Trial entries of the five rows are the same as in Table 2.", "caption": "Table 1 (PDF p. 9). The outline marks the same two treatment rates, printed as 0.6275.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E018-b.png"}], "formal_release": null, "formal_statements": [], "headline": "For the Horse Colic data, Table 2 prints the treatment rate of the two rows with information asymmetry as $0.6274$ where Table 1 prints $0.6275$ for the same rate, $160/255$; several Stroke Trial entries and the two fit statistics of Appendix E.2.2 are truncated where neighboring entries are rounded.", "id": "P006-E018", "location": "Table 2, Horse Colic “Treat rate” in the rows $w_{\\mathrm{TT}}$ (no info) and $w_{\\mathrm{TT}}$ (demographic info) (PDF p. 27), against the same cells of Table 1 (PDF p. 9); Stroke Trial entries of Tables 1 and 2, and the fit statistics of Appendix E.2.2 (PDF p. 25)", "pdf_page": 27, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table~1 (PDF p.~9), Horse Colic dataset, Treat rate.} $w_{\\mathrm{TT}}$ (no info): 0.6275. $w_{\\mathrm{TT}}$ (demographic info): 0.6275.\n\n\\emph{Table~2 (PDF p.~27), the same cells.} $w_{\\mathrm{TT}}$ (no info): \\hl{0.6274}. $w_{\\mathrm{TT}}$ (demographic info): \\hl{0.6274}.\n\n\\emph{Tables~1 and~2, Stroke Trial dataset.} $w_{\\mathrm{TT}}$, Utility: 0.0251\\hl{8}. $w_{\\mathrm{TT}}$ (no info), Utility: $-0.0488\\hl{8}$. $w_{\\mathrm{ATT}}$, Regret: 0.025\\hl{0}.\n\n\\emph{Table~2, Stroke Trial dataset.} $w_{\\mathrm{TO}}$, Utility: $-0.0827\\hl{8}$; Regret: 0.10\\hl{79}.\n\n\\emph{Appendix~E.2.2 (PDF p.~25).} The fitted RMSE was 1.3\\hl{4} and the $R^2$ was 0.2\\hl{6}.", "corrected": "\\emph{Table~2, Horse Colic dataset, Treat rate.} $w_{\\mathrm{TT}}$ (no info): \\hl{0.6275}. $w_{\\mathrm{TT}}$ (demographic info): \\hl{0.6275}.\n\nIf every entry is to be rounded to nearest, the Stroke Trial entries become $0.0251\\hl{9}$, $-0.0488\\hl{9}$ and $0.025\\hl{1}$ in both tables and $-0.0827\\hl{7}$ and $0.10\\hl{80}$ in Table~2, and the sentence of Appendix~E.2.2 reads: The fitted RMSE was 1.3\\hl{5} and the $R^2$ was 0.2\\hl{7}.", "evidence": "The evidence is a recalculation from the outputs stored in the notebooks and an exact refit of the Stroke Trial model. The notebook for the Horse Colic data prints both rates as $0.627451$, and 160 is the only count out of 255 that agrees with this output. For the Stroke Trial data the computed values are $0.0251858$, $-0.0488860$ and $0.0250547$ for the three entries common to both tables, $-0.0827749$ and $0.1079606$ for the $w_{\\mathrm{TO}}$ row, and $1.346663$ and $0.269755$ for the RMSE and the $R^2$. All the printed entries are the truncated values except $-0.08278$, which is neither the rounded nor the truncated value of $-0.0827749$. A second rounding of the six-decimal output of the notebook, $-0.082775$, would produce it.", "problem": "Table~2 repeats the rows of Table~1 and adds the row $w_{\\mathrm{TO}}$. The two treatment rates are the same quantities in both tables. Each rate is $160/255 = 0.62745\\ldots$, which rounds to $0.6275$, and Table~2 shows the truncated value. The other entries listed above also differ from the nearest rounding at the printed precision, while most entries of the two tables are rounded to nearest. The paper states no rounding rule, so for these entries the defect is an inconsistency of presentation and not a wrong value.", "what_survives": "No sign, ordering or conclusion depends on these digits, and the calculations and figures use the unrounded values. The Stroke Trial values here are those of the outcome as coded in the released notebook, which is the subject of \\fid{P006-E011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E018", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the left column. The text reads: pal were to estimate mu-hat-zero of x from the agent's untreated units with T sub i equal to 0. We discuss this case in detail in the Appendix, and show that a stronger assumption yields a similar bound to Theorem 5. Most importantly, our key result is that even without confounding, informa-. A red outline surrounds the words Theorem 5.", "caption": "Section 5, paragraph “Information asymmetry and confounding”, PDF p. 7, left column. The outline marks the theorem number.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "The bound to which the appendix result is compared is that of Theorem 3, not Theorem 5.", "id": "P006-N001", "location": "Section 5, paragraph “Information asymmetry and confounding”, PDF p. 7, left column", "pdf_page": 7, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We discuss this case in detail in the Appendix, and show that a stronger assumption yields a similar bound to \\hl{Theorem 5}.", "corrected": "We discuss this case in detail in the Appendix, and show that a stronger assumption yields a similar bound to \\hl{Theorem 3}.", "evidence": "The evidence is a check of the pages. The introduction to Appendix D (PDF p.~20) has the reference right: ``we can get similar regret bounds to Theorem 3''.", "problem": "Theorem 5 is the appendix result that the sentence announces: under the stronger Assumption 4, the regret is at most $2\\gamma_{\\max}$ (PDF p.~23). The bound with which it is compared is that of Theorem 3, $R(\\pi^{w_{\\mathrm{TT}}})\\le2\\gamma_{\\mathrm{marg}}$, stated on the same page as the sentence.", "what_survives": "Nothing depends on the cross-reference."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N001", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of the right column. The text reads: setup by comparing treating with high doses of heparin and aspirin (T = 1) with aspirin alone (T = 0). This leaves n = 7264 patients and a treatment rate of 0.33. The 20 covariates include each patient's age. A red outline surrounds the words n = 7264 patients.", "caption": "Section 6, paragraph “International Stroke Trial dataset”, PDF p. 7, right column. The outline marks the size of the analysis sample.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The Stroke sample of 7264 patients includes 35 whose status at six months is missing; the released notebook keeps them and scores them as neither dead nor recovered, and without them the sample has 7229 patients.", "id": "P006-N003", "location": "Section 6, paragraph “International Stroke Trial dataset”, PDF p. 7, right column; replication notebook stroke_trial_experiments.ipynb, the cell headed “Filter missing outcomes”", "pdf_page": 7, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{PDF p.~7.} This leaves \\hl{$n=7264$ patients} and a treatment rate of 0.33.\n\n\\emph{Replication notebook.} Under the comment \\texttt{\\# Filter missing outcomes}, the filter is \\texttt{df = df[df['OCCODE'] \\hl{< 5}]}. Later cells set \\texttt{df['death'] = (df['OCCODE'] == 1).astype(int)} and \\texttt{df['recovery'] = (df['OCCODE'] == 4).astype(int)}.", "corrected": "\\emph{Replication notebook.} \\texttt{df = df[df['OCCODE']\\hl{.isin([1, 2, 3, 4])}]}\n\n\\emph{PDF p.~7.} This leaves \\hl{$n=7229$ patients} and a treatment rate of 0.33, where patients whose status at six months is missing are excluded; the fitted Stroke model, its tables and its figures are computed again on this sample.", "evidence": "The evidence is a recalculation from the replication data, in exact arithmetic. The data file has 19435 rows, with \\texttt{OCCODE} equal to 0 in 97, to 9 in 53, and to 1, 2, 3 or 4 in the others. When the filter is restricted to the codes 1 to 4, exactly the 35 patients are removed, of whom 11 were treated. The least-squares model was then fitted again with no other change.\n\\begin{audittable}{lrr}{Stroke results with the released filter and with the 35 patients removed.}\nQuantity & As released & Codes 1 to 4 only \\\\\nPatients in the analysis sample & 7264 & 7229 \\\\\nPatients with a positive fitted effect & 1360 & 1390 \\\\\nUtility under $w_{\\mathrm{TT}}$ & 0.025186 & 0.026224 \\\\\nRegret under $w_{\\mathrm{TT}}$ & 0 & 0 \\\\\nRegret with no information & 0.074072 & 0.072990 \\\\\nRegret with demographic information & 0.089102 & 0.087954 \\\\\n\\end{audittable}\nThe recalculation does not show what the missing outcomes were, and the direction of any bias is not established.", "problem": "The article that documents the trial database defines the values 0 and 9 of \\texttt{OCCODE} as missing status at six months. The filter removes the code 9 and keeps the code 0. Of the 7264 patients analysed, 35 have the code 0, and their six-month fields for death, dependency and recovery are blank or unknown. The two equality tests then score all 35 as neither dead nor recovered, and neither the paper nor the notebook states a rule for missing outcomes. The printed count 7264 is the size of the sample that the notebook analyses. The defect is in the filter, which does not do what its comment says.", "what_survives": "The theoretical results and the Horse Colic experiment are unaffected. Without the 35 patients the Stroke entries of Tables 1 and 2, Figure 1, the Stroke panel of Figure 2 and the fit statistics of Appendix E.2.2 change slightly or not at all, and the conclusions drawn from them stand: the utility under $w_{\\mathrm{TT}}$ is positive, the regret with demographic information exceeds the regret with no information, and the order of the features is unchanged. The observed treatment rate still rounds to 0.33. The table changes the filter only. A second correction to the Stroke outcome is the subject of \\fid{P006-E011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N003", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of ten lines of the left column, beginning 'We also study a more continuous spectrum of information asymmetry'. A red box marks the words 'ascending order of feature importance'. Four lines below, not marked, the text reads 'building up starting from the most important feature'.", "caption": "Section 6, PDF p. 8, left column. The marked phrase gives the order in which the features are sorted. The last sentence in the crop says that the subsets are built up from the most important feature.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N004.png"}], "formal_release": null, "formal_statements": [], "headline": "“Ascending order of feature importance” means ascending order of the individual $\\gamma_{\\mathrm{marg}}$, which puts the most important feature first.", "id": "P006-N004", "location": "Section 6, PDF p. 8, left column, paragraph beginning “We also study a more continuous spectrum”", "pdf_page": 8, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Relating this to Section 5, we first sort the features in \\hl{ascending order of feature importance}, as measured by estimating $\\gamma_{\\mathrm{marg}}$ when the principal knows only the individual feature. Then, Figure 1 shows regret when the principal knows increasingly large feature subsets, building up starting from the most ``important'' feature.", "corrected": "Relating this to Section 5, we first sort the features in \\hl{ascending order of individual $\\gamma_{\\mathrm{marg}}$ (from most to least important)}, as measured by estimating $\\gamma_{\\mathrm{marg}}$ when the principal knows only the individual feature.", "intended_meaning": "The features are sorted by their individual values of $\\gamma_{\\mathrm{marg}}$, from the smallest to the largest. Assumption 2 defines $\\gamma_{\\mathrm{marg}}$ as a bound on $E[|\\mu_0(X_i)-\\mu_0(X_i,U_i)|]$. A feature with a small $\\gamma_{\\mathrm{marg}}$ leaves little of the untreated outcome unexplained when the principal knows that feature alone, and the feature with the smallest value is the most important one. The order is ascending in $\\gamma_{\\mathrm{marg}}$ and descending in importance. The next sentence and the caption of Figure 1 (``accumulates features from the left (most important)'') state this order, and the replication files sort the values of $\\gamma_{\\mathrm{marg}}$ upward for both datasets. Figure 1 is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N004", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of item 2, 'Untreated patients in the treatment population', with its conditions (a) No information asymmetry and (b) Positivity. Two red boxes mark the end of condition (b): 'and Y i is observed for a nonzero proportion of untreated units for all x.'", "caption": "Appendix A.1, item 2, PDF p. 13. The marked clause is the only condition placed on which untreated outcomes are observed.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N005.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T23", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T23"}, {"id": "P006-T24", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T24"}], "headline": "The conditions listed for identifying $\\mu_0(x)$ from a fraction of the untreated outcomes are not sufficient; adding mean representativeness of the observed untreated outcomes at each $x$ makes them sufficient.", "id": "P006-N005", "location": "Appendix A.1, item 2 (“Untreated patients in the treatment population”), conditions (a) and (b), PDF p. 13", "pdf_page": 13, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T23", "P006-T24"], "text": {"as_printed": "2. \\textbf{Untreated patients in the treatment population.} In the main paper, we have assumed that the principal only observed outcomes $Y_i$ for treated units with $T_i^\\pi=1$. If the principal also observed a fraction of outcomes for untreated units, with $T_i^\\pi=0$, then these may also be incorporated into the estimate for $\\hat\\mu_0(x)$. Sufficient conditions for identifying $\\mu_0(x)$ from a dataset of the agent's untreated units are:\n\n(a) No information asymmetry: the agent's treatment policy $\\pi$ only depends on $X_i$. Under this assumption, we have no confounding: $\\{Y_i(0),Y_i(1)\\}\\perp\\!\\!\\!\\perp T_i^\\pi\\mid X_i$.\n\n(b) Positivity: $P(T_i^\\pi=0\\mid X_i=x)>0$ for all $x\\in\\mathcal{X}$, \\hl{and $Y_i$ is observed for a nonzero proportion of untreated units for all $x$.}", "corrected": "Sufficient conditions for identifying $\\mu_0(x)$ from a dataset of the agent's untreated units are:\n\n(a) No information asymmetry: the agent's treatment policy $\\pi$ only depends on $X_i$. Under this assumption, we have no confounding: $\\{Y_i(0),Y_i(1)\\}\\perp\\!\\!\\!\\perp T_i^\\pi\\mid X_i$.\n\n(b) Positivity: $P(T_i^\\pi=0\\mid X_i=x)>0$ for all $x\\in\\mathcal{X}$, and $Y_i$ is observed for a nonzero proportion of untreated units for all $x$.\n\n\\hl{(c) Representative observation:} with $R_i=1$ when $Y_i$ is observed,\n\\[\n\\hl{E[Y_i(0)\\mid X_i=x,T_i^\\pi=0,R_i=1]=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0]}\\quad\\text{for all }x\\in\\mathcal{X}.\n\\]\n\nUnder (a), (b), (c) and consistency, $\\mu_0(x)=E[Y_i\\mid X_i=x,T_i^\\pi=0,R_i=1]$. Condition (c) holds when $Y_i(0)$ is independent of $R_i$ given $X_i$ and $T_i^\\pi=0$.", "evidence": "The evidence is an exact counterexample and a derivation of the corrected statement. Let $X=0$ and $Y(1)=1$ for every unit, and let treatment be assigned with probability $1/2$ independently of the potential outcomes. Every treated outcome is observed. Let $R=1$ when the outcome of a unit is observed. I consider two distributions of the untreated outcome $Y(0)\\in\\{0,1\\}$ and of $R$ among the untreated units.\n\\begin{audittable}{lrr}{Two distributions that satisfy (a) and (b) and give the same observed data.}\nQuantity & First distribution & Second distribution \\\\\n$P(Y(0)=1)$ & $1/2$ & $3/4$ \\\\\n$P(R=1\\mid Y(0)=0,\\,T=0)$ & $1/4$ & $1/2$ \\\\\n$P(R=1\\mid Y(0)=1,\\,T=0)$ & $3/4$ & $1/2$ \\\\\n$P(R=1\\mid T=0)$ & $1/2$ & $1/2$ \\\\\nMean of the observed untreated outcomes & $3/4$ & $3/4$ \\\\\n$\\mu_0(0)$ & $1/2$ & $3/4$\n\\end{audittable}\nIn both distributions the observed data have the same law: an untreated unit with a missing outcome has probability $1/4$, an untreated unit with observed outcome $0$ has probability $1/16$, an untreated unit with observed outcome $1$ has probability $3/16$, and a treated unit, whose outcome is $1$, has probability $1/2$. Since the two values of $\\mu_0(0)$ differ, $\\mu_0(0)$ is not a function of the law of the observed data.\n\nFor the corrected statement, suppose in addition that $E[Y_i(0)\\mid X_i=x,T_i^\\pi=0,R_i=1]=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0]$. By consistency, this condition and (a), I have\n\\[\n\\begin{aligned}\nE[Y_i\\mid X_i=x,T_i^\\pi=0,R_i=1]&=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0,R_i=1]\\\\\n&=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0]=E[Y_i(0)\\mid X_i=x]=\\mu_0(x),\n\\end{aligned}\n\\]\nand the conditioning event has positive probability by (b).", "problem": "Item 2 supposes that the principal observes the outcomes of only a fraction of the untreated units. Among the listed conditions, (a) concerns how treatment is assigned, and (b) requires only that some untreated outcomes are observed at each $x$. Neither condition says which untreated outcomes are observed. If the probability that an untreated outcome is observed depends on its value, then the mean of the observed untreated outcomes at $x$ can differ from the mean of all untreated outcomes there. In that case (a) and (b) hold and $\\mu_0(x)=E[Y_i(0)\\mid X_i=x]$ is not identified.", "what_survives": "The correction concerns one way of obtaining the estimate $\\hat\\mu_0$. It does not affect the results that take an unbiased estimate as given, the auxiliary data of item 1 and Appendix D.2, or the analysis in Appendix D.3, which uses the mean over all untreated units. Appendix E.3 states that the experiments do not simulate the estimation of $\\hat\\mu_0$ from a subset of the data, and no reported number changes. The added condition is an assumption about how outcomes come to be observed, and a data source in which observation depends on the outcome can fail to satisfy it. The first paragraph of Appendix D.3 needs the same condition when only a fraction of the untreated outcomes is observed."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N005", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of four lines of the proof of Proposition 5 around the display 'E of w TT equals n, open parenthesis, E of pi times (mu 1 minus mu 0)'. A red box marks the letter n and the opening parenthesis that follows it.", "caption": "Appendix D.2, proof of Proposition 5, PDF p. 22. The marked parenthesis after $n$ has no closing partner.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N006.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T25", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T25"}], "headline": "The display for $E[w_{\\mathrm{TT}}]$ in the proof of Proposition 5 has an opening parenthesis after $n$ that is never closed.", "historical_evidence_type": "lean_proof", "id": "P006-N006", "location": "Appendix D.2, proof of Proposition 5, display before “The resulting policy $\\pi^{w_{\\mathrm{TT}}}$ that maximizes this is”, PDF p. 22", "pdf_page": 22, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P006-T25"], "text": {"as_printed": "Then by Lemma 2, and assuming $\\hat\\mu_0(x)$ is unbiased, we have\n\\[\nE[w_{\\mathrm{TT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=n\\hl{(}E[\\pi(X_i,U_i)(\\mu_1(X_i,U_i)-\\mu_0(X_i))].\n\\]\nThe resulting policy $\\pi^{w_{\\mathrm{TT}}}$ that maximizes this is", "corrected": "\\[\nE[w_{\\mathrm{TT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=\\hl{nE}[\\pi(X_i,U_i)(\\mu_1(X_i,U_i)-\\mu_0(X_i))].\n\\]", "evidence": "I checked the unmatched parenthesis on the page. The expected-reward identity with the parenthesis removed is also proved in the Lean statements of \\fid{P006-T25}.", "problem": "The parenthesis after $n$ is a misprint; the expected reward is $n$ times the displayed expectation.", "what_survives": "Proposition 5 and its proof are unchanged. The displayed policy is one of several maximizers, since the agent is indifferent at $u=1$; Section 2 defines $\\pi^w$ as a member of the set of maximizers, and the printed regret $\\alpha-\\beta/2$ is the regret of the displayed policy."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N006", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of five lines of text. The text reads: Figure 3 is the equivalent of Figure 1 for the Horse Colic dataset. For the Horse Colic dataset, the regret drops to close to zero if the principal knows only a few features. This is likely due to the fact that there is little variation in other feature values in the dataset conditioned packed cell volume and pulse, exacerbated by the fact that the dataset is so small. The value of gamma marg for these two features combined is small compared to the individual gamma marg values in Figure 3 (left), at 0.035. A red outline surrounds the value 0.035.", "caption": "Appendix E.5, paragraph below Figure 2 (PDF p. 27). The outline marks the reported value 0.035.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "For the fit recorded in the released notebook, the joint value of $\\gamma_{\\mathrm{marg}}$ for packed cell volume and pulse in the Horse Colic data is $0.028$ and not the printed $0.035$.", "id": "P006-N009", "location": "Appendix E.5, paragraph below Figure 2, last sentence (PDF p. 27)", "pdf_page": 27, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "The value of $\\gamma_{\\mathrm{marg}}$ for these two features combined is small compared to the individual $\\gamma_{\\mathrm{marg}}$ values in Figure 3 (left), at \\hl{0.035}.", "corrected": "The value of $\\gamma_{\\mathrm{marg}}$ for these two features combined is small compared to the individual $\\gamma_{\\mathrm{marg}}$ values in Figure 3 (left), at \\hl{0.028}.\n\nThe release can make this value explicit with one more cell, which evaluates\n\\[\n\\texttt{mean(abs(mu\\_0s\\_marginal - mu\\_0s))}\n\\]\nfor the pair of features \\texttt{x\\_obs = ['X\\_18', 'X\\_4']}.", "evidence": "The evidence is a derivation of exact bounds. The notebook prints the coefficients of the fit to eight decimals and does not print the intercept. The mean of the fitted untreated outcome over the 232 adult horses, which the notebook prints as $0.542912$, confines the intercept to the interval $(3.466, 3.472)$. Carrying the rounding intervals of the coefficients and this interval for the intercept through the calculation in rational arithmetic, I find that the joint $\\gamma_{\\mathrm{marg}}$ lies in $[0.028006, 0.028055]$ for every fit consistent with the printed output. The two features define 211 groups, and 177 of the 255 horses are alone in their group.", "problem": "The sentence reports the empirical $\\gamma_{\\mathrm{marg}}$ of Appendix~E.4 when the principal observes packed cell volume and pulse together. This is the average over the 255 horses of $|\\mu_0(X_i) - \\mu_0(X_i, U_i)|$, where $\\mu_0(X_i)$ is the mean of the fitted untreated outcome over the horses with the same two values. For the fit recorded in the released notebook this average rounds to $0.028$ to three decimals, and $0.035$ is not its value at the printed precision. The notebook computes $\\gamma_{\\mathrm{marg}}$ for single features and regret for cumulative sets of features. It contains no calculation of the joint value, and the origin of the number $0.035$ is not established.", "what_survives": "The comparison that the sentence makes stands: $0.028$ is small beside the values for the single features, $0.393$ for packed cell volume and $0.417$ for pulse. No calculation uses the printed number. The tables, Figure~3 and the regret after these two features, about $0.0008$ in the notebook, are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N009", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of six lines of text. A red box marks the words: the bound in Assumption 4 implies that Assumption 2 is satisfied with gamma marg less than or equal to gamma max.", "caption": "Appendix D.3, PDF p. 23: the implication from Assumption 4 to Assumption 2.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N011.png"}], "formal_release": null, "formal_statements": [{"id": "P006-T17", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T17"}], "headline": "The statement that Assumption 4 implies Assumption 2 with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$ holds when the private type has a conditional distribution given the covariates, as it does when it takes finitely many values or values in a Euclidean space; on general measurable spaces the implication can fail.", "historical_evidence_type": "lean_proof", "id": "P006-N011", "location": "Appendix D.3, PDF p. 23, paragraph after Assumption 4; Assumption 2, PDF p. 6", "pdf_page": 23, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P006-T17"], "text": {"as_printed": "Appendix D.3, PDF p.~23: ``Furthermore, \\hl{the bound in Assumption 4 implies that Assumption 2 is satisfied with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$}.''\n\nAssumption 4: ``Define the maximum difference in the untreated potential outcome for a given $x,u$ to be $\\Delta(x,u)=\\max_{\\tilde u\\in\\mathcal U}|\\mu_0(x,u)-\\mu_0(x,\\tilde u)|$. The average difference is bounded as: $E[\\Delta(X,U)]\\le\\gamma_{\\max}$.''\n\nAssumption 2, PDF p.~6: ``The effect of $U_i$ on the conditional untreated potential outcome is bounded as $E[|\\mu_0(X_i)-\\mu_0(X_i,U_i)|]\\le\\gamma_{\\mathrm{marg}}$.'' ``Note that $\\mu_0(X_i)=E[\\mu_0(X_i,U_i)\\mid X_i]$.''", "corrected": "Appendix D.3: ``Furthermore, \\hl{when $U$ has a conditional distribution given $X$, so that $\\mu_0(x)$ is the average of $\\mu_0(x,u)$ over it,} the bound in Assumption 4 implies that Assumption 2 is satisfied with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$.''", "intended_meaning": "If the private type $U$ has a conditional distribution given $X$, then for almost every $x$, $\\mu_0(x)$ is the average of $\\mu_0(x,v)$ over that distribution, so $|\\mu_0(x)-\\mu_0(x,u)|\\le\\Delta(x,u)$ for every $u$. With the usual integrability, taking expectations gives $E[|\\mu_0(X)-\\mu_0(X,U)|]\\le E[\\Delta(X,U)]\\le\\gamma_{\\max}$, so Assumption 2 holds with a constant $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$. A conditional distribution exists when $U$ takes finitely or countably many values and, more generally, when it takes values in a standard Borel space such as a Euclidean space, which covers the settings the paper describes. The paper does not state this condition. On a general measurable space, where $\\mu_0(X)$ is defined only as a conditional expectation, the implication can fail: in an example whose type space admits no such conditional distribution, $E[|\\mu_0(X)-\\mu_0(X,U)|]=3/4$ while $E[\\Delta(X,U)]=1/2$. The example concerns only this comparison. It does not bear on models in which $U$ has a conditional distribution given $X$, and the finite-type calculation of \\fid{P006-E014} is not affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N011", "work": {"arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/"}, "work_id": "P006"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first paragraph of the Introduction and the first two lines of the next paragraph. Red boxes mark the words 'the glmnet function in the R package of the same name' at the end of one line and '(Friedman, Hastie, and Tibshirani 2010)' at the start of the next line.", "caption": "Introduction, p. 149 (PDF p. 1), first paragraph. The marked words are the third software example and its citation.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E001.png"}], "formal_release": null, "formal_statements": [], "headline": "The glmnet function did not omit rows with missing data by default when the article appeared, and it should be removed from the list of software examples.", "id": "P007-E001", "location": "Introduction, p. 149 (PDF p. 1), first paragraph, second sentence", "pdf_page": 1, "printed_page": "149", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``It constitutes the default behavior for standard data analyses in popular software packages: for example, rows with any missing data are by default omitted by the \\textit{lm} function in R (R Core Team 2020), the \\textit{regress} command in Stata (Stata.com 2020), and \\hl{the \\textit{glmnet} function in the R package of the same name (Friedman, Hastie, and Tibshirani 2010)}.''", "corrected": "``It constitutes the default behavior for standard data analyses in popular software packages: for example, rows with any missing data are by default omitted by the \\textit{lm} function in R (R Core Team 2020) \\hl{and} the \\textit{regress} command in Stata (Stata.com 2020).''\n\nThe citation of Friedman, Hastie, and Tibshirani (2010) is removed with the example. It is the only citation of that reference in the article, and the entry in the reference list is removed as well.", "evidence": "The evidence is a check of the package source as released. I read the files \\texttt{R/glmnet.R} and \\texttt{R/elnet.R} of glmnet 4.1-3, released on CRAN on 2 November 2021 (\\url{https://raw.githubusercontent.com/cran/glmnet/4.1-3/R/glmnet.R}), and the source of releases 4.1, 4.1-1, 4.1-2 and 1.1-4, the version named in Friedman, Hastie, and Tibshirani (2010). None of them removes incomplete rows by default. I also ran the R code of 4.1-3 on eight rows, with the compiled fitting routine replaced by a function that records its arguments. With one missing predictor value, all eight rows reach the fitting routine. With one missing response value, the call stops with the error ``missing value where TRUE/FALSE needed''. Seven rows reach the fitting routine only when the incomplete row is removed beforehand. I did not run the compiled package of 2021, and I have not established what its fitting routine did with a missing predictor value. I did not check the \\textit{lm} and \\textit{regress} examples against the releases of that date.", "problem": "The sentence says that three functions omit incomplete rows by default, and the third does not. In glmnet 4.1-3, the release current when the article was published online on 2 March 2022, the function \\textit{glmnet} contains no step that removes rows. For a Gaussian response, it keeps the number of rows of its input and passes the matrix of predictors, with any missing values, to the fitting routine. A missing value in the response stops the function with an error. Rows are omitted only if the user removes them, from the predictors and the response together, before the call.", "what_survives": "No result, table or figure depends on the example, and the replication code does not use glmnet. The other two examples and the point that the sentence makes about default behavior are not affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E001", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph beginning 'We also assume that there is some idiosyncratic missingness in each variable', followed by Assumption 2. Red boxes mark the words 'one variable is always observed whenever another variable is observed', which run over two lines. Assumption 2 below bounds Pr(M_i1 = 0) and Pr(M_ij = 0 given that the bold vector M_i(j-1) equals 0) by q_*.", "caption": "Section 2, p. 150 (PDF p. 2): the paragraph before Assumption 2 and the assumption. The marked words describe what the assumption rules out.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E002.png"}], "formal_release": null, "formal_statements": [], "headline": "Assumption 2 rules out a variable that is always observed whenever all the variables before it in the ordering are observed; it does not rule out the reverse, as the sentence before the assumption can suggest.", "id": "P007-E002", "location": "Section 2, p. 150 (PDF p. 2), paragraph before Assumption 2, last sentence", "pdf_page": 2, "printed_page": "150", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Namely, we will assume that there is a factorization of the data such that all conditional probabilities that an observation is missing are bounded away from zero, given whenever prior variables are observed. Substantively, this assumption rules out the case where, for example, \\hl{one variable is always observed whenever another variable is observed}.''", "corrected": "``Substantively, this assumption rules out the case where, for example, \\hl{a variable is always observed whenever all preceding variables in the ordering are observed, provided that this event has positive probability}.''", "intended_meaning": "Assumption 2 bounds by $q_*<1$ the probability that variable $j$ is observed given that variables $1,\\dots,j-1$ are all observed, whenever that event has positive probability. The sentence describes this bound, and its example is to be read in the direction of the ordering: a variable cannot be always observed whenever all the variables before it are observed. Read in this way, the sentence is correct.\n\nThe assumption allows the reverse. For example, let $k=2$, and let $(M_{i1},M_{i2})$ take the values $(1,1)$, $(0,1)$ and $(0,0)$ with probabilities $1/2$, $1/4$ and $1/4$, where $M_{ij}=0$ means that the entry is observed. Then $\\Pr(M_{i1}=0)=1/2$ and $\\Pr(M_{i2}=0\\mid M_{i1}=0)=1/2$, and Assumption 2 holds with $q_*=1/2$. However, $\\Pr(M_{i1}=0\\mid M_{i2}=0)=1$: the first variable is always observed whenever the second variable is observed. No result uses the sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E002", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the proof of Lemma 3 under the heading 'Appendix: Proofs'. Seven red outlines mark: the words 'which is equivalent to say that all variables are fully missing' (two outlines, over a line break); the expression 'q i j less than or equal to q star, for all j in 1, 2, up to k'; and four occurrences of the comma-separated list 'q i 1, q i 2, up to q i k'.", "caption": "Proof of Lemma 3, p. 154 (PDF p. 6). Marked: the sentence on the case $q_* = 0$, the bound $q_{ij} \\leq q_*$ asserted for all $j$, and the four comma lists printed for the product $q_{i1} q_{i2} \\cdots q_{ik}$.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E003.png"}, {"alt": "Crop of three lines of the proof of Proposition 5, beginning 'Since q star in (0,1) and k = f(n)'. Three red outlines mark a greater-than-or-equal sign with a subscript star after 'k = f(n)', a second such sign after the n-th power of one minus q star to the f(n), and the expression one minus q star to the power 'f(n' with no closing parenthesis, raised to the power n.", "caption": "Proof of Proposition 5, p. 154 (PDF p. 6). Marked: the two signs $\\geq$ printed with a subscript star, and the expression $(1 - q_*^{f(n})^n$, in which the exponent $f(n)$ lacks its closing parenthesis.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E003-b.png"}, {"alt": "Crop of Lemma 8 and its proof in the supplement. Three red outlines mark the words 'Since this entails that all groups of variables are completely missing,' and, over a line break, the expression 'q i j less than or equal to q star, for all j in 1, 2, up to g'.", "caption": "Supplementary Material B, p. 4: Lemma 8 and its proof. Marked: the sentence on the case $q_* = 0$ and the bound $q_{ij} \\leq q_*$ asserted for all $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E003-c.png"}, {"alt": "Crop of two lines of the proof of Proposition 10, beginning 'Since q star in (0,1) and k = l(n)'. Four red outlines mark the letter k, two greater-than-or-equal signs that carry a subscript star, and the expression one minus q star to the power 'l(n' with no closing parenthesis, raised to the power n.", "caption": "Supplementary Material B, foot of p. 4: proof of Proposition 10. Marked: $k$ printed for $g$, the two signs $\\geq$ with a subscript star, and the expression $(1 - q_*^{l(n})^n$ with the unclosed exponent.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E003-d.png"}, {"alt": "Crop of two paragraphs of Supplementary Material B that define g, the number of variable groups, and the indicator for the j-th group in the i-th row. In the sentence 'we use M i j to represent the random vector collecting the missingness indicators up to variable j', the word 'variable' is outlined in red.", "caption": "Supplementary Material B, p. 3: definitions for groups of variables. The marked word “variable” should be “group”: in this section $j$ indexes groups.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E003-e.png"}], "formal_release": null, "formal_statements": [{"id": "P007-T3", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T3"}, {"id": "P007-T4", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T4"}, {"id": "P007-T5", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T5"}, {"id": "P007-T6", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T6"}], "headline": "The proofs of Lemmas 3 and 8 wrongly read $q_* = 0$ as every variable being missing and omit the case of a conditioning event of probability zero, and several symbols are misprinted; the two lemmas and Propositions 5 and 10 hold as stated.", "id": "P007-E003", "location": "Appendix, proofs of Lemma 3 and Proposition 5, p. 154 (PDF p. 6); Supplementary Material B, p. 3 (definition of $\\mathbf{M}_{ij}$) and p. 4 (proofs of Lemma 8 and Proposition 10)", "pdf_page": 6, "printed_page": "154", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P007-T3", "P007-T4", "P007-T5", "P007-T6"], "text": {"as_printed": "\\begin{itemize}\n\\item Proof of Lemma 3, p.~154: ``First suppose $q_* = 0$, \\hl{which is equivalent to say that all variables are fully missing}. Then $p_{all} = 1 = (1 - q_*^k)^n$. Now suppose $q_* \\in (0,1)$. By the independence assumption, $p_{all} = \\Pi_{i=1}^{n}(1 - \\Pr(\\mathbf{M}_{ik} = 0))$. Denote $q_{ij} = \\Pr(M_{ij} = 0 | \\mathbf{M}_{i(j-1)} = 0)$ if $j > 1$, else $q_{ij} = \\Pr(M_{ij} = 0)$. By Assumption 2, $\\hl{q_{ij} \\leq q_*}$, for all $j \\in \\{1, 2, \\ldots, k\\}$. By the chain rule of conditional probability, $\\Pr(\\mathbf{M}_{ik} = 0) = \\hl{q_{i1}, q_{i2}, \\ldots, q_{ik}}$. This means that the probability of a single observation containing at least one missing entry is $(1 - \\hl{q_{i1}, q_{i2}, \\ldots, q_{ik}})$. Since $q_* \\geq q_{ij}$ for all $j \\in \\{1, 2, \\ldots, k\\}$, $q_*^k \\geq \\hl{q_{i1}, q_{i2}, \\ldots, q_{ik}}$. Thus $(1 - q_*^k) \\leq (1 - \\hl{q_{i1}, q_{i2}, \\ldots, q_{ik}})$.''\n\\item Proof of Proposition 5, p.~154: ``Since $q_* \\in (0,1)$ and $k = f(n) \\hl{\\geq_*} 1$, $-q_*^{f(n)} > -1$ and $1 - q_*^{f(n)} \\leq 1$. By Bernoulli's Inequality, since $n \\in \\mathbb{N}$, $(1 - q_*^{f(n)})^n \\hl{\\geq_*} 1 + n(-q_*^{f(n)}) = 1 - nq_*^{f(n)}$. Thus $1 - nq_*^{f(n)} \\leq \\hl{(1 - q_*^{f(n})^n} \\leq 1$ in the common domain $n \\in \\mathbb{N}$.''\n\\item Supplement, p.~3: ``we use $\\mathbf{M}_{ij}$ to represent the random vector collecting the missingness indicators up to \\hl{variable} $j$, $(M_{i1}, M_{i2}, \\ldots, M_{ij})$.''\n\\item Supplement, proof of Lemma 8, p.~4: ``Suppose $q_* = 0$. \\hl{Since this entails that all groups of variables are completely missing}, $p_{all} = 1 = (1 - q_*^g)^n$. For the second case suppose $q_* \\in (0,1)$. By the group independence assumption, $p_{all} = \\Pi_{i=1}^{n}(1 - \\Pr(\\mathbf{M}_{ig} = 0))$. Denote $q_{ij} = \\Pr(M_{ij} = 0 | \\mathbf{M}_{i(j-1)} = 0)$ if $j > 1$, else $q_{ij} = \\Pr(M_{ij} = 0)$. By Assumption 7, $\\hl{q_{ij} \\leq q_*}$, for all $j \\in \\{1, 2, \\ldots, g\\}$.''\n\\item Supplement, proof of Proposition 10, p.~4: ``Since $q_* \\in (0,1)$ and $\\hl{k} = l(n) \\hl{\\geq_*} 1$, $-q_*^{l(n)} > -1$ and $1 - q_*^{l(n)} \\leq 1$. By Bernoulli's Inequality, since $n \\in \\mathbb{N}$, $(1 - q_*^{l(n)})^n \\hl{\\geq_*} 1 + n(-q_*^{l(n)}) = 1 - nq_*^{l(n)}$. Thus $1 - nq_*^{l(n)} \\leq \\hl{(1 - q_*^{l(n})^n} \\leq 1$ in the common'' (the sentence runs on to p.~5).\n\\end{itemize}", "corrected": "The statements of Lemma 3, Lemma 8, Proposition 5 and Proposition 10 stand as printed. The proofs are corrected as follows. For the supplement, read groups for variables, $g$ for $k$, $l(n)$ for $f(n)$, and Assumptions 6, 7 and 9 for Assumptions 1, 2 and 4.\n\n\\emph{Proof of Lemma 3.} Assume that the rows are mutually independent (Assumption 1) and that there exists $q_* \\in [0,1)$ such that, for all $i$, $\\Pr(M_{i1} = 0) \\leq q_*$ and $\\Pr(M_{ij} = 0 | \\mathbf{M}_{i(j-1)} = 0) \\leq q_*$ for all $j \\in \\{2, \\ldots, k\\}$ such that $\\Pr(\\mathbf{M}_{i(j-1)} = 0) > 0$ (Assumption 2). Fix a row $i$. Let $s_{i0} = 1$ and, for $j \\in \\{1, \\ldots, k\\}$, let $s_{ij} = \\Pr(\\mathbf{M}_{ij} = 0)$ be the probability that the first $j$ entries of row $i$ are observed. Then\n\\[\ns_{ij} \\leq q_* \\, s_{i(j-1)} \\quad \\text{for all } j \\in \\{1, \\ldots, k\\}.\n\\]\nFor $j = 1$ this is the first part of Assumption 2. For $j \\geq 2$ with $\\hl{s_{i(j-1)} = 0}$, the event $\\{\\mathbf{M}_{ij} = 0\\}$ is contained in $\\{\\mathbf{M}_{i(j-1)} = 0\\}$, so $s_{ij} = 0$. For $j \\geq 2$ with $s_{i(j-1)} > 0$, the second part of Assumption 2 gives $s_{ij} / s_{i(j-1)} = \\Pr(M_{ij} = 0 | \\mathbf{M}_{i(j-1)} = 0) \\leq q_*$. By induction on $j$, this implies that $s_{ik} \\leq q_*^k$. Since the rows are independent,\n\\[\np_{all} = \\prod_{i=1}^{n} (1 - s_{ik}) \\geq (1 - q_*^k)^n.\n\\]\nIf $q_* = 0$, then $s_{i1} = 0$ for every $i$: \\hl{the first variable is missing with probability one in every row}, and $p_{all} = 1$. Later variables need not be missing.\n\n\\emph{Proof of Proposition 5.} If $\\hl{q_* = 0}$, Lemma 3 gives $p_{all} = 1$ for every $n$ with $f(n) \\geq 1$. Suppose that $q_* \\in (0,1)$. The limit $\\lim_{n \\to \\infty} n q_*^{f(n)} = 0$ is obtained as printed. The paragraph that follows should read: ``Since $q_* \\in (0,1)$ and $k = f(n) \\hl{\\geq} 1$, $-q_*^{f(n)} > -1$ and $1 - q_*^{f(n)} \\leq 1$. By Bernoulli's Inequality, since $n \\in \\mathbb{N}$, $(1 - q_*^{f(n)})^n \\hl{\\geq} 1 + n(-q_*^{f(n)}) = 1 - nq_*^{f(n)}$. Thus $1 - nq_*^{f(n)} \\leq \\hl{(1 - q_*^{f(n)})^n} \\leq 1$ in the common domain $n \\in \\mathbb{N}$.'' The rest of the proof is as printed.\n\n\\emph{Other changes.} Each of the four comma lists in the printed proof of Lemma 3 is the product $\\hl{q_{i1} q_{i2} \\cdots q_{ik}}$. The proof of Proposition 10 should read $\\hl{g} = l(n) \\geq 1$, with the same three repairs as in the proof of Proposition 5. On p.~3 of the supplement, read ``up to \\hl{group} $j$''. In the proof of Lemma 8, ``the group independence assumption'' is Assumption 6, which is independence across rows.", "evidence": "The evidence consists of exact counterexamples to the two statements in the printed proofs, and a derivation of the lemmas and propositions under the stated assumptions, given under ``Corrected''.\n\nLet $n = 1$ and $k = 2$, and let the row satisfy $(M_{11}, M_{12}) = (1, 0)$ with probability one. Assumption 1 holds, since there is only one row. With $q_* = 0$, Assumption 2 holds: I have $\\Pr(M_{11} = 0) = 0 \\leq q_*$, and no condition is imposed for $j = 2$ because $\\Pr(M_{11} = 0) = 0$. Since the second variable is always observed, the variables are not all fully missing, while $p_{all} = 1 = (1 - q_*^2)^1$. With two groups of one variable each, the same example applies to the proof of Lemma 8.\n\nFor the same row and $q_* = 1/2$, Assumption 2 holds again, but $q_{12} = \\Pr(M_{12} = 0 | M_{11} = 0)$ is a ratio $0/0$. The lemma's conclusion holds, since $p_{all} = 1 \\geq 3/4$. The event of probability zero need not involve the first variable alone. Let $k = 3$ and $q_* = 1/2$, and let a row equal $(1,0,0)$ or $(0,1,0)$ with probability $1/2$ each. Then $\\Pr(M_{i1} = 0) = 1/2$ and $\\Pr(M_{i1} = M_{i2} = 0) = 0$, and $q_{i3}$ is undefined. For two independent rows with this distribution, $p_{all} = 1 \\geq 49/64$.\n\nA row equal to $(1,1)$ with probability one satisfies Assumption 2 with $q_* = 1/2$ as well as with $q_* = 0$. Complete missingness does not force $q_* = 0$. For the index in the proof of Proposition 10, two groups of two variables each give $g = 2$ and $k = 4$. I read the misprints from the printed pages. The supplement prints the product in the proof of Lemma 8 correctly, as $q_{i1} q_{i2} \\cdots q_{ig}$.", "problem": "The printed proofs have two defects of argument and several misprints. None of them makes a stated result false.\n\n\\emph{The case $q_* = 0$.} Both proofs read $q_* = 0$ as saying that all variables (in the supplement, all groups) are fully missing. Assumption 2 with $q_* = 0$ says only that the first variable is missing with probability one in every row. The conditions on later variables are then vacuous, because each is imposed only when the earlier variables are all observed with positive probability, and later variables may be observed. The conclusion $p_{all} = 1$ is still correct, since every row has a missing first entry. The printed equivalence also fails in the other direction. Since $q_*$ is an upper bound and need not be the smallest one, data in which every variable is always missing satisfy Assumption 2 with $q_* = 1/2$ as well.\n\n\\emph{Conditioning on an event of probability zero.} In the second case the proofs define $q_{ij}$, for $j > 1$, as the probability that entry $j$ is observed given that the first $j - 1$ entries are observed, and assert that $q_{ij} \\leq q_*$ for all $j$. Assumption 2 (in the supplement, Assumption 7) bounds this conditional probability only when the conditioning event has positive probability. When that event has probability zero, $q_{ij}$ is a ratio $0/0$ and the proofs state no convention for it. Neither the bound nor the product formula is then available as written. The split into $q_* = 0$ and $q_* \\in (0,1)$ does not cover this, since it can happen in either case.\n\n\\emph{Misprints.} In the proof of Lemma 3 the product $q_{i1} q_{i2} \\cdots q_{ik}$ is printed four times as a list separated by commas. In the proofs of Propositions 5 and 10 the sign $\\geq$ twice carries a subscript star, and in the sandwich inequality the exponent $f(n)$ (in the supplement, $l(n)$) lacks its closing parenthesis. The proof of Proposition 10 writes $k = l(n)$ where the number of groups, $g = l(n)$, is meant; $k$ is the number of variables, and $k \\geq g$. On p.~3 of the supplement, ``up to variable $j$'' should be ``up to group $j$''. The proofs of the two propositions also begin the logarithm argument with $q_* \\in (0,1)$ without saying that the case $q_* = 0$ follows from the lemma.", "what_survives": "Lemma 3, Lemma 8, Proposition 5 and Proposition 10 hold as stated, under the stated assumptions; no assumption is added and no rate changes. No number changes: the numerical illustrations of Supplementary Material A use the bound of Lemma 3 directly, and the simulations of Section 3 count complete rows directly. What changes is the wording of the case $q_* = 0$, one step in the proofs of the two lemmas, and the misprinted symbols."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E003", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the third paragraph of Section 3.1. Three terms are outlined in red: 'out' in 'how out theoretical assumptions', 'i.i.d.' in 'Assumption 1 asserts i.i.d. missingness across rows', and 'i.i.d.' in 'Our theoretical assumption of i.i.d. missingness across rows'.", "caption": "Section 3.1, third paragraph, p. 152 (PDF p. 4). The misprint “out” and the two places where Assumption 1 is called “i.i.d.” are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P007-T11", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T11"}], "headline": "Assumption 1 requires mutually independent rows, not i.i.d. rows, and “out theoretical assumptions” should read “our theoretical assumptions”.", "id": "P007-E004", "location": "Section 3.1, third paragraph, p. 152 (PDF p. 4), first, second and last sentences", "pdf_page": 4, "printed_page": "152", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P007-T11"], "text": {"as_printed": "``We briefly discuss how \\hl{out} theoretical assumptions align with this setting. Assumption 1 asserts \\hl{i.i.d.} missingness across rows, which is unlikely to be met in this setting. [\\ldots] Our theoretical assumption of \\hl{i.i.d.} missingness across rows can therefore be seen as optimistic when faced with real-world data and, all else equal, the probability that all data will be lost may be higher than theory might dictate.''", "corrected": "``We briefly discuss how \\hl{our} theoretical assumptions align with this setting. Assumption 1 asserts \\hl{mutually independent} missingness across rows, which is unlikely to be met in this setting. [\\ldots] Our theoretical assumption of \\hl{mutually independent} missingness across rows can therefore be seen as optimistic when faced with real-world data and, all else equal, the probability that all data will be lost may be higher than theory might dictate.''", "evidence": "The evidence is a check of the page against Assumption 1 as printed, and an example. Two independent rows with one variable each, observed with probabilities $1/4$ and $1/2$, satisfy Assumptions 1 and 2 with $q_* = 1/2$ and are not identically distributed. Lemma 3 holds for them, since $p_{all} = (3/4)(1/2) = 3/8 \\geq 1/4$.", "problem": "Assumption 1 (p.~150) states that the rows are mutually independent. It does not ask that they be identically distributed, and the proof of Lemma 3 works with row-specific probabilities. The paragraph twice describes the assumption as ``i.i.d.'', which adds a condition that the paper neither assumes nor uses. The first sentence also prints ``out'' for ``our''.", "what_survives": "Nothing depends on the label. Assumption 1 as stated, Lemma 3, Proposition 5 and the paragraph's point about clustered rows are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E004", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of two paragraphs. In the first, a red box marks the words 'smallest possible' in 'we will get the smallest possible k for each p_all that k =', followed by the floor of log(1 - p_all^(1/n)) over log(q_*). In the second, red boxes mark 'k = 33' in the example with n = 10,000 and q_* = 0.75 and 'k = 952' in the example with n = 10,000 and q_* = 0.99.", "caption": "Supporting Information A, supplement p. 2: the two paragraphs that describe Figure 4. The marked words call the plotted value the smallest possible $k$, and the marked values are given as yielding a 50% probability.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E005.png"}, {"alt": "Crop of the subcaptions '(a) p_all at most 0.5' and '(b) p_all at most 0.99' and of the caption 'Figure 4. Largest k such that p_all is at most 0.5, 0.99 plotted against n and q_*'. Red boxes mark the three occurrences of p_all, none of which is underlined.", "caption": "Supplement p. 2: the subcaptions and the caption of Figure 4. The caption says “Largest $k$”; the three marked symbols are printed without the underline that denotes the lower bound.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E005-caption.png"}], "formal_release": null, "formal_statements": [{"id": "P007-T7", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T7"}, {"id": "P007-T8", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T8"}], "headline": "The formula plotted in Figure 4 gives the largest $k$ at which the lower bound $(1-q_*^k)^n$ does not exceed the target, not the smallest $k$ that reaches it; at $k=33$ and $k=952$ the bound is 47.1% and 49.7%, and it first reaches 50% at $k=34$ and $k=953$.", "id": "P007-E005", "location": "Supporting Information A, supplement p. 2: the two paragraphs before Figure 4, and the caption and subcaptions of Figure 4", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P007-T7", "P007-T8"], "text": {"as_printed": "``Figure 4 illustrates, for a given $n,q_*$, how large $k$ can be while still ensuring that $\\hl{p_{all}}\\leq 0.5,0.99$. We compute this using the result from Lemma 3, $\\underline{p_{all}}=(1-q_*^k)^n$. Since $(1-q_*^k)^n$ is strictly increasing in $k$, solving for equality $\\hl{p_{all}}=(1-q_*^k)^n$ we will get the \\hl{smallest possible} $k$ for each $\\underline{p_{all}}$ that\n\\[\nk=\\left\\lfloor\\frac{\\log(1-\\underline{p_{all}}^{1/n})}{\\log(q_*)}\\right\\rfloor.\n\\]\n\\ldots\n\nWith more missingness, $q_*=0.75,0.90$, even relatively small $k$ can yield missingness of $\\underline{p_{all}}$. For example, even with $n=10,000$ and $q_*=0.75$ we need only $\\hl{k=33}$ to have a 50\\% probability that all rows will be missing. However, when missingness is very low, $k$ needs to be very large to cause all data to be missing. For example, with $n=10,000$ and $q_*=0.99$, we need $\\hl{k=952}$ to have a 50\\% probability that all rows will be missing.''\n\nSubcaptions and caption of Figure 4: ``(a) $\\hl{p_{all}}\\leq 0.5$'', ``(b) $\\hl{p_{all}}\\leq 0.99$'', ``Figure 4. Largest $k$ such that $\\hl{p_{all}}\\leq 0.5,0.99$ plotted against $n$ and $q_*$.''", "corrected": "``Figure 4 shows, for each $n$ and $q_*$, the \\hl{largest} nonnegative integer $k$ for which the sharp lower bound $\\underline{p_{all}}=(1-q_*^k)^n$ does not exceed $a$, with $a=0.5$ or $0.99$. Solving $(1-q_*^r)^n=a$ gives\n\\[\nr=\\frac{\\log(1-a^{1/n})}{\\log(q_*)},\n\\]\nand the plotted value is $k=\\lfloor r\\rfloor$. This calculation concerns the lowest possible probability that listwise deletion removes all rows under Assumptions 1 and 2, not an upper bound on the actual probability. \\ldots\n\n\\ldots\\ For $n=10,000$, the lower bound is \\hl{approximately 47.08\\%} at $q_*=0.75$, $k=33$, and \\hl{approximately 49.69\\%} at $q_*=0.99$, $k=952$. The smallest integers at which this lower bound is at least 50\\% are \\hl{34 and 953}, respectively.''\n\nSubcaptions and caption of Figure 4: ``(a) $\\hl{\\underline{p_{all}}}\\leq 0.5$'', ``(b) $\\hl{\\underline{p_{all}}}\\leq 0.99$'', ``Figure 4. Largest $k$ such that $\\hl{\\underline{p_{all}}}\\leq 0.5,0.99$ plotted against $n$ and $q_*$.'' In the titles of both panels, ``probability all data missing'' is replaced by ``lower bound on probability all data missing''.\n\nThe statement behind the figure is the following. Let $n\\ge1$, $0<q_*<1$ and $0<a<1$, and let $r$ be as in the display. The largest integer $k\\ge0$ such that $(1-q_*^k)^n\\le a$ is $\\lfloor r\\rfloor$, and the smallest integer $k\\ge0$ such that $(1-q_*^k)^n\\ge a$ is $\\lceil r\\rceil$.", "evidence": "The evidence is a derivation and a calculation in exact rational arithmetic. Let $n\\ge1$, $0<q_*<1$ and $0<a<1$, and set $r=\\log(1-a^{1/n})/\\log q_*$. For real $k\\ge0$, I have\n\\[\n\\begin{aligned}\nB(k)\\le a&\\iff 1-q_*^k\\le a^{1/n}\\iff q_*^k\\ge 1-a^{1/n}\\\\\n&\\iff k\\log q_*\\ge\\log(1-a^{1/n})\\iff k\\le r,\n\\end{aligned}\n\\]\nwhere the last step divides by $\\log q_*<0$. Reversing the inequalities, I obtain that $B(k)\\ge a$ if and only if $k\\ge r$. The largest integer $k$ such that $B(k)\\le a$ is $\\lfloor r\\rfloor$, and the smallest integer $k$ such that $B(k)\\ge a$ is $\\lceil r\\rceil$. For $n=10{,}000$ and $a=1/2$, the values of $B$ on both sides of $r$ are as follows.\n\n\\begin{audittable}{lrr}{The lower bound $B(k)=(1-q_*^k)^n$ at $n=10{,}000$, computed in exact rational arithmetic and rounded to five places.}\n$q_*$ & $k$ & $B(k)$ \\\\\n0.75 & 33 & 0.47075 \\\\\n0.75 & 34 & 0.56833 \\\\\n0.99 & 952 & 0.49689 \\\\\n0.99 & 953 & 0.50037 \\\\\n\\end{audittable}\n\nThe bound $B(k)$ is attained when all entries are independent and each is observed with probability $q_*$. In that case the probability that listwise deletion removes all rows is 47.1\\% at $k=33$ and 49.7\\% at $k=952$, below the 50\\% of the text. The statements printed without the underline fail as well. Let $n=10{,}000$, $q_*=0.75$ and $k=33$, and let all entries be independent, the first variable observed with probability $3/4$ and each of the other 32 variables with probability $1/4$. Assumptions 1 and 2 hold, $B(33)<0.5$, and $p_{all}=(1-3\\cdot4^{-33})^{10000}>0.99$. All 59,406 values plotted in Figure 4 satisfy $B(k)<a<B(k+1)$.", "problem": "Let $B(k)=(1-q_*^k)^n$ denote the lower bound of Lemma 3 on the probability $p_{all}$ that listwise deletion removes all rows; the supplement writes it $\\underline{p_{all}}$. For a target $a$, the floor in the display is the largest integer $k$ such that $B(k)\\le a$. The titles of the two panels (the first reads ``Largest k such that probability all data missing is $\\leq$ 0.50'') and the caption say so, and the code computes this value. The text calls it the smallest possible $k$, and the two examples present $k=33$ and $k=952$ as the numbers of variables needed for a 50\\% probability that all rows are missing. At these values the lower bound is below 50\\%: for $n=10{,}000$, I find $B(33)=0.4708$ at $q_*=0.75$ and $B(952)=0.4969$ at $q_*=0.99$. The smallest integers at which the lower bound is at least 50\\% are 34 and 953.\n\nThe first sentence, the subcaptions and the caption also print $p_{all}$ where the lower bound $\\underline{p_{all}}$ is meant. A lower bound on $p_{all}$ cannot ensure that $p_{all}\\le a$, and Figure 4 shows the largest $k$ such that $\\underline{p_{all}}\\le a$.", "what_survives": "The plotted values in Figure 4 and the code that produces them are correct when read as the largest $k$ at which the lower bound does not exceed the target, and no plotted value changes. Lemma 3, Proposition 5 and the two applications do not use this inversion. The corrected text is a statement about the lower bound of Lemma 3, and it contains no upper bound on $p_{all}$. The contrast between high and low rates of missingness stands."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E005", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of three lines beginning 'Our final numerical illustration considers an upper bound on the expected proportion of observations that are missing, 1 - q_*^k, which does not depend on n'. A red box marks the words 'upper bound'.", "caption": "Supporting Information A, supplement p. 2, last paragraph. The marked words should read “lower bound”.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E006.png"}, {"alt": "Crop of Figure 5. The title reads 'Lower bound on expected proportion of data missing'. Three curves, labelled q_* = 0.75, 0.90 and 0.99, show the lower bound on the expected proportion missing, from 0 to 1, against the number of variables, from 0 to 150; the curve for 0.99 ends near 0.78. In the caption below, 'Figure 5. Lower bound of expected proportion of all rows missing (p_all underlined) plotted against values of n,k and q_*', red boxes mark the underlined symbol p_all in parentheses and the letters 'n,k'.", "caption": "Supplement p. 3: Figure 5 with its title and caption. The title says “Lower bound on expected proportion of data missing”; the marked symbol denotes the bound of Figure 3, and the marked parameter list incorrectly includes $n$.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E006-figure5.png"}], "formal_release": null, "formal_statements": [{"id": "P007-T9", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T9"}, {"id": "P007-T10", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T10"}], "headline": "The quantity $1-q_*^k$ of Figure 5 is a lower bound, not an upper bound, on the expected proportion of rows removed, and the caption of Figure 5 labels it with the symbol $\\underline{p_{all}}$ and the dependence on $n$ of a different quantity.", "id": "P007-E006", "location": "Supporting Information A, supplement p. 2, last paragraph, first sentence; supplement p. 3, caption of Figure 5", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P007-T10", "P007-T9"], "text": {"as_printed": "Supplement p.~2: ``Our final numerical illustration considers an \\hl{upper bound} on the expected proportion of observations that are missing, $1-q_*^k$, which does not depend on $n$. Figure 5 plots the expected proportion of data missing versus the number of variables $k$.''\n\nSupplement p.~3: ``Figure 5. Lower bound of expected proportion of all rows missing ($\\hl{\\underline{p_{all}}}$) plotted against values of $\\hl{n},k$ and $q_*$.''", "corrected": "Supplement p.~2: ``Our final numerical illustration considers a \\hl{lower bound} on the expected proportion of rows removed by listwise deletion, $b_k=1-q_*^k$, which does not depend on $n$. Figure 5 plots this lower bound versus the number of variables $k$.''\n\nSupplement p.~3: ``Figure 5. Lower bound $\\hl{b_k=1-q_*^k}$ on the expected proportion of rows removed by listwise deletion, plotted against $\\hl{k}$ for $q_*\\in\\{0.75,0.90,0.99\\}$.''\n\nThe statement behind the figure is the following. Under Assumption 2, for all $n\\ge1$ and $k\\ge1$, the expected proportion of rows removed by listwise deletion is at least $1-q_*^k$. Equality holds when all entries are independent and each is observed with probability $q_*$. The lower bound $(1-q_*^k)^n$ of Lemma 3 on the probability that all rows are removed is strictly smaller than $1-q_*^k$ when $n>1$ and $\\hl{0<q_*<1}$.", "evidence": "The evidence is a derivation and an exact counterexample. Let $F_n$ denote the proportion of the $n$ rows that are removed. By Assumption 2 and the chain rule, each row is complete with probability at most $q_*^k$; a row for which some conditioning event of the assumption has probability zero is complete with probability zero. By linearity of expectation, I obtain $\\mathbb{E}[F_n]\\ge1-q_*^k$, and Assumption 1 is not used. For the counterexample to the upper bound, let $n=k=2$ and $q_*=3/4$, and let the four entries be independent, the first variable observed with probability $3/4$ and the second with probability $1/2$ in each row. Assumptions 1 and 2 hold, and each row is complete with probability $3/8$. I find $\\mathbb{E}[F_2]=5/8$, which exceeds $1-q_*^2=7/16$. In the same example the lower bound on the probability that both rows are removed is $(7/16)^2=49/256$, which differs from $7/16$. The code computes the plotted curve as $1-q_*^k$ for $k=1,\\dots,150$ and the three values of $q_*$, and all 1,350 values of its grid agree with this formula. At $q_*=0.99$ and $k=150$ the curve equals $0.7785$, while $(1-q_*^k)^n=1.3\\times10^{-11}$ for $n=100$.", "problem": "Under Assumption 2, each row is complete with probability at most $q_*^k$, and the expected proportion of rows removed by listwise deletion is at least $1-q_*^k$. The quantity is a lower bound, as the title of Figure 5 says, and the first sentence of the paragraph calls it an upper bound. The caption attaches to the curve the symbol $\\underline{p_{all}}$, which the supplement defines as $(1-q_*^k)^n$, the lower bound on the probability that all rows are removed, and it lists $n$ among the quantities that vary. The plotted quantity is $1-q_*^k$, which does not depend on $n$, and the figure varies $k$ and $q_*$ only. The two quantities agree when $n=1$, as the text after the paragraph notes.", "what_survives": "The curves of Figure 5 are correct and do not change. The correction replaces one word and the caption, and no assumption, proof or calculation in the applications depends on either."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E006", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the heading '3 Application' and the six lines below it. A red box marks the words '(Teorell et al. 2021) standard' in the phrase 'the January 2021 QoG (Teorell et al. 2021) standard cross-sectional dataset'.", "caption": "Section 3, p. 151 (PDF p. 3), first paragraph. The marked words name and cite the Standard dataset.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E009.png"}, {"alt": "Crop of three reference entries, the last partly shown. In the entry for Teorell, Sundström, Holmberg, Rothstein, Pachon and Dalli, 2021, a red box marks the word 'Standard' in the title 'The Quality of Government Standard Dataset, Version Jan21'.", "caption": "Reference list, p. 155 (PDF p. 7), entry for Teorell et al. The marked word is in the title of the dataset.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E009-reference.png"}], "formal_release": null, "formal_statements": [], "headline": "The Quality of Government data analysed in Section 3 are the January 2021 Basic cross-sectional dataset, not the Standard dataset that the article names and cites; the reported numbers stand for the Basic file.", "id": "P007-E009", "location": "Section 3, p. 151 (PDF p. 3), first paragraph; reference list, p. 155 (PDF p. 7), entry for Teorell et al.; replication files, readme.txt and Simcode_Qog.R", "pdf_page": 3, "printed_page": "151", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 3, p.~151: ``\\ldots\\ we considered two prominent datasets in use in the fields of comparative politics and international relations: the January 2021 QoG (\\hl{Teorell \\textit{et al.} 2021}) \\hl{standard} cross-sectional dataset, and the State Failure dataset \\ldots''\n\nReference list, p.~155: Teorell, J., A. Sundström, S. Holmberg, B. Rothstein, N. A. Pachon, and C. M. Dalli, 2021. ``The Quality of Government \\hl{Standard} Dataset, Version Jan21.'' University of Gothenburg, The Quality of Government Institute.\n\nReplication files, \\texttt{readme.txt}: ``\\texttt{qog\\_bas\\_cs\\_jan21.dta} available at: \\href{https://www.qogdata.pol.gu.se/data/qog_std_cs_jan21.dta}{https://www.qogdata.pol.gu.se/data/qog\\_\\hl{std}\\_cs\\_jan21.dta} (last accessed Jun 25, 2021)'', followed by the citation of ``The Quality of Government \\hl{Standard} Dataset, version Jan21'' with \\texttt{doi:10.18157/qog\\hl{std}jan21}.", "corrected": "Section 3, p.~151: ``\\ldots\\ the January 2021 QoG \\hl{Basic} cross-sectional dataset, and the State Failure dataset \\ldots''\n\nThe citation ``Teorell \\textit{et al.} 2021'' in Section 3 and in the Introduction (p.~149), and the entry in the reference list, are replaced by the reference for the Quality of Government Basic Dataset, Version Jan21. The file is \\texttt{qog\\_bas\\_cs\\_jan21.dta}, with version field \\texttt{QoGBasCSJan21}. I have not confirmed the author list and the DOI of that reference against the documentation of the Quality of Government Institute, and I do not give them here.\n\nIn the replication files, the download address and the citation in \\texttt{readme.txt} and in the comment of \\texttt{Simcode\\_Qog.R} are replaced by those of the Basic file. The Quality of Government column of Table 1 and Figure 1 keep their values, and their source is the Basic dataset.", "evidence": "The evidence is a recalculation from the replication files. The code drops six identifier columns (\\texttt{ccode}, \\texttt{cname}, \\texttt{ccodealp}, \\texttt{ccodecow}, \\texttt{ccodewb}, \\texttt{version}), which leaves 194 rows and 351 columns with 24,424 of 68,094 cells missing. The column with the most missing values has 176 of them, and six columns have none. These counts give the five numerical entries of the Quality of Government column of Table 1. Let $r_i$ denote the number of observed columns in row $i$. When $k$ of the 351 columns are drawn uniformly without replacement, the expected proportion of complete rows is\n\\[\n\\frac{1}{194}\\sum_{i=1}^{194}\\binom{r_i}{k}\\Big/\\binom{351}{k},\n\\]\nwhich is approximately $0.4364$ at $k=2$ and $0.00913$ at $k=17$. I ran the simulation code on the Basic file with the seed of the replication files (213) and 25,000 draws at each of the 100 values of $k$. The run reproduces all 100 estimates of expected proportions and all 100 probability estimates of the saved log at its printed precision. I did not have the Standard file, and I did not compute its dimensions, its missingness or its simulation curves.\n\n\\begin{audittable}{lll}{The Quality of Government numbers of the article, as printed and as recomputed from the Basic file. The first five rows are the numerical entries of the column of Table 1 (p.~152), and the last four are the thresholds of Section 3.2 (p.~153). The expected proportions are calculated exactly and rounded here, and the two probabilities are simulation estimates from 25,000 draws.}\nQuantity & As printed & From the Basic file \\\\\nNumber of observations & 194 & 194 \\\\\nNumber of variables & 351 & 351 \\\\\nProportion of missing values, average & 35.9\\% & 35.87\\% (24,424 of 68,094) \\\\\nProportion of missing values, maximum & 90.7\\% & 90.72\\% (176 of 194) \\\\\nNumber of variables fully observed & 6 & 6 \\\\\nExpected proportion of rows removed, 2 variables & more than 50\\% & 56.36\\% \\\\\nExpected proportion of rows removed, 17 variables & more than 99\\% & 99.09\\% \\\\\nProbability that all rows are removed, 14 variables & greater than 0.5 & 0.5076 \\\\\nProbability that all rows are removed, 52 variables & more than 0.99 & 0.9902 \\\\\n\\end{audittable}", "problem": "Section 3 describes the first dataset as the January 2021 QoG standard cross-sectional dataset, and the entry in the reference list is for the Standard dataset. The file that the replication code reads is \\texttt{qog\\_bas\\_cs\\_jan21.dta}. The data label stored in this file is ``Quality of Government Basic dataset 2021 - Cross-Section'', and its field \\texttt{version} equals \\texttt{QoGBasCSJan21} in all 194 rows. The Quality of Government column of Table 1, Figure 1 and the thresholds 2, 17, 14 and 52 of Section 3.2 are computed from this Basic file. The article attributes these results to a dataset that the replication files do not analyse. The readme and a comment in the script pair the name of the Basic file with the download address and the citation of the Standard file. The citation and download address identify a different edition. The simulation draws variables at random from the 351 variables of the Basic file, and its expected proportions and probabilities refer to that set of variables. The replication files contain no result for the Standard dataset.\n\n\\begin{audittable}{ll}{How each source identifies the Quality of Government data.}\nSource & Identification \\\\\nSection 3, p.~151 & ``standard cross-sectional dataset'' \\\\\nReference list, p.~155 & ``The Quality of Government Standard Dataset, Version Jan21'' \\\\\nReadme, download address & \\texttt{qog\\_std\\_cs\\_jan21.dta} \\\\\nReadme, citation & Standard Dataset, \\texttt{doi:10.18157/qogstdjan21} \\\\\nFile read by the code & \\texttt{qog\\_bas\\_cs\\_jan21.dta} \\\\\nData label stored in the file & ``Quality of Government Basic dataset 2021 - Cross-Section'' \\\\\nField \\texttt{version}, all 194 rows & \\texttt{QoGBasCSJan21} \\\\\n\\end{audittable}", "what_survives": "The Quality of Government column of Table 1 and the simulation estimates in Figure 1 and Section 3.2 stand for the Basic file. Section 2, the Appendix, the supplement and the State Failure application do not use this file. The correction changes the dataset, and with it the set of variables, that the Quality of Government application describes."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E009", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of Table 1, summary statistics for the QoG and State Failure datasets. In the State Failure column, the entry 66.8 per cent in the row 'Proportion of missing values (avg.)' is outlined in red. The other entries in that column are Country-years, 8,580, 1,205, 99.99 per cent and 79.", "caption": "Table 1, p. 152 (PDF p. 4). The marked entry, 66.8%, is an average over 1,224 variables; the rest of the column describes the 1,205 variables that remain after 19 were removed.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E010.png"}, {"alt": "Crop of the third paragraph of Section 3.2 and the start of the fourth. In the sentence 'With more than three variables, the loss is more than 99 per cent.' the words 'more than three' are outlined in red.", "caption": "Section 3.2, p. 153 (PDF p. 5), third paragraph. The marked words place the 99% threshold at four variables; in the simulation it is first passed at six.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E010-b.png"}, {"alt": "Line chart with a logarithmic vertical axis. Seven blue points give the estimated share of rows retained with one to seven variables: 33.6, 13.5, 6.06, 2.82, 1.37, 0.66 and 0.32 per cent. A dashed horizontal line marks 1 per cent of rows retained, that is 99 per cent lost. A shaded orange region below that line, beginning between three and four variables, marks the printed statement. The points for four and five variables lie above the line, and the points for six and seven lie below it. A black diamond at four variables marks the exact lower bound, 1.27 per cent.", "caption": "State Failure data: estimated proportion of rows retained after listwise deletion, by number of randomly selected variables, from the archived simulation log (logarithmic scale). The shaded region is the printed statement, a loss above 99% with more than three variables. The estimates for four and five variables lie outside it, and the exact lower bound at four variables (diamond) is 1.27%.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E010-loss-threshold.png"}], "formal_release": null, "formal_statements": [{"id": "P007-T1", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T1"}, {"id": "P007-T2", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T2"}, {"id": "P007-T12", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T12"}, {"id": "P007-T13", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T13"}], "headline": "For the State Failure data, the average proportion of missing values is 66.3%, not 66.8%, and in the simulation the estimated loss of rows exceeds 99% with six or more variables, not with more than three.", "historical_evidence_type": "lean_proof", "id": "P007-E010", "location": "Table 1, p. 152 (PDF p. 4), State Failure column, row “Proportion of missing values (avg.)”; Section 3.2, p. 153 (PDF p. 5), third paragraph, last sentence", "pdf_page": 4, "printed_page": "152", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P007-T1", "P007-T12", "P007-T13", "P007-T2"], "text": {"as_printed": "Table 1, State Failure column: Number of observations 8,580; Number of variables 1,205; Proportion of missing values (avg.) \\hl{66.8\\%}; Proportion of missing values (max) 99.99\\%; Number of variables fully observed 79.\n\nSection 3.2: ``The researcher can expect to lose more than 50\\% of the data if the only variable included is randomly selected. With \\hl{more than three} variables, the loss is more than 99\\%.''", "corrected": "Table 1, State Failure column: Proportion of missing values (avg.) \\hl{66.3\\%}.\n\nSection 3.2: ``The researcher can expect to lose more than 50\\% of the data if the only variable included is randomly selected. \\hl{In the simulation, the estimated} loss exceeds 99\\% with \\hl{six or more} variables.''\n\nIn the replication script, recompute the average after the wholly missing columns are removed, with \\texttt{prop.missing <- mean(colMeans(isnamatrix))} after line 44.", "evidence": "I checked the replication script and its saved log, recalculated the average, and derived an exact bound.\n\n\\emph{The average.} The script computes the average at line 29, removes the wholly missing variables at lines 39--40, and builds the table column at line 55 from the reduced data and the earlier average. The log prints the average as 0.668390827582004, taken over $8{,}580 \\times 1{,}224 = 10{,}501{,}920$ cells. Allowing for rounding, the only whole number of missing cells consistent with that value is 7,019,387. The 19 removed variables hold $19 \\times 8{,}580 = 163{,}020$ missing cells and no observed ones. The average over the remaining variables is\n\\[\n\\frac{7{,}019{,}387 - 163{,}020}{8{,}580 \\times 1{,}205} = \\frac{6{,}856{,}367}{10{,}338{,}900} = 0.66316\\ldots,\n\\]\nwhich rounds to 66.3\\%. I infer these counts from the log. My recalculation from the data file gives the same average, $0.66316\\ldots$, under the code's definition of a missing cell, as described under ``What was checked''.\n\n\\emph{The threshold.} Let $r_i$ be the number of observed cells in row $i$ among the 1,205 variables, so that $\\sum_i r_i = 10{,}338{,}900 - 6{,}856{,}367 = 3{,}482{,}533$. When four variables are drawn uniformly without replacement, row $i$ survives for $\\binom{r_i}{4}$ of the $\\binom{1{,}205}{4}$ equally likely draws. Since the function $r \\mapsto \\binom{r}{4}$ is convex on the integers, the sum $\\sum_i \\binom{r_i}{4}$ is smallest when the $r_i$ are as equal as possible, which means 405 in 947 rows and 406 in 7,633 rows. I obtain\n\\[\n\\begin{aligned}\n\\mathbb{E}[\\text{proportion of rows retained}] &= \\frac{\\sum_i \\binom{r_i}{4}}{8{,}580 \\binom{1{,}205}{4}} \\geq \\frac{947 \\binom{405}{4} + 7{,}633 \\binom{406}{4}}{8{,}580 \\binom{1{,}205}{4}} \\\\\n&= \\frac{24{,}513{,}560{,}217}{1{,}923{,}069{,}889{,}510} = 0.01274\\ldots.\n\\end{aligned}\n\\]\nThis implies that the expected loss with four variables is at most 98.73\\%, however the missing cells are arranged. The bound uses only the dimensions of the data and the number of observed cells.\n\nThe estimates of the archived simulation, with 25,000 draws for each number of variables, are in the table below. A Chernoff bound for the mean of 25,000 independent draws in $[0,1]$ applies to the two estimates nearest the threshold. If the expected proportion retained with five variables were at most 1\\%, an estimate as large as the one in the log would have probability at most $2.8 \\times 10^{-7}$. If it were at least 1\\% with six variables, an estimate as small as the one in the log would have probability at most $9.6 \\times 10^{-8}$.\n\n\\begin{audittable}{rrr}{State Failure data: estimates in the archived simulation log, by number of randomly selected variables.}\nVariables & Proportion of rows retained & Loss of rows \\\\\n1 & 0.3361604 & 66.38\\% \\\\\n2 & 0.1345654 & 86.54\\% \\\\\n3 & 0.06058382 & 93.94\\% \\\\\n4 & 0.02820753 & 97.18\\% \\\\\n5 & 0.01365003 & 98.63\\% \\\\\n6 & 0.006640252 & 99.34\\% \\\\\n7 & 0.003187953 & 99.68\\% \\\\\n\\end{audittable}", "problem": "Table 1 describes the State Failure data after the 19 variables with 100\\% missingness were removed: 8,580 country-years and 1,205 variables. The replication script computes the average proportion of missing values before it removes those 19 variables and does not compute it again. The printed 66.8\\% is an average over 1,224 variables, while the number of variables and the maximum in the same column refer to the 1,205 that remain. For those 1,205 variables the average is 66.3\\%.\n\nThe sentence in Section 3.2 places the 99\\% threshold too early. In the simulation that the article reports, the estimated loss of rows is 97.2\\% with four variables and 98.6\\% with five, and it first exceeds 99\\% with six. The gap at four variables is far larger than the simulation error. An exact bound shows that, for data of these dimensions with this many observed cells, the expected loss with four variables cannot reach 99\\%.", "what_survives": "Figure 2 and the simulation estimates are unchanged: the simulation uses the 1,205 variables that remain and never reads the average. The other State Failure entries of Table 1 do not change as a consequence. The QoG results, Lemma 3, Proposition 5 and the numerical illustrations do not depend on either number. The simulation still supports the conclusion that a few variables are enough for rapid loss of rows. One table entry and one sentence change. The script's part in the table entry is the subject of \\fid{P007-E014}."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E010", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three entries in the reference list. In the entry for Berk (1983), the words 'A. B. A. Peter, H. Rossi, and J. D. Wright,' after 'edited by' are outlined in red.", "caption": "References, p. 155 (PDF p. 7), entry for Berk (1983). The marked editor names are garbled.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E013.png"}], "formal_release": null, "formal_statements": [], "headline": "The editors of the Handbook of Survey Research are P. H. Rossi, J. D. Wright and A. B. Anderson.", "id": "P007-E013", "location": "References, p. 155 (PDF p. 7), entry for Berk (1983)", "pdf_page": 7, "printed_page": "155", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Berk, R. 1983. ``Applications of the General Linear Model to Survey Data.'' In \\emph{Handbook of Survey Research}, edited by \\hl{A. B. A. Peter, H. Rossi, and J. D. Wright}, pp.~495--546. Quantitative Studies in Social Relations. New York: Academic Press.", "corrected": "Berk, R. 1983. ``Applications of the General Linear Model to Survey Data.'' In \\emph{Handbook of Survey Research}, edited by \\hl{P. H. Rossi, J. D. Wright, and A. B. Anderson}, pp.~495--546. Quantitative Studies in Social Relations. New York: Academic Press.", "evidence": "The evidence is a check of a public source. The \\href{https://www.sciencedirect.com/book/edited-volume/9780125982269/handbook-of-survey-research}{publisher's catalogue entry} for the 1983 Academic Press edition (ISBN 9780125982269) lists the editors as Peter H. Rossi, James D. Wright and Andy B. Anderson, in that order, and the title page reproduced in a public preview of the book has the same names in the same order. The catalogue entry lists Berk's chapter as chapter 13, pp.~495--546.", "problem": "The entry names the editors of the volume as ``A. B. A. Peter, H. Rossi, and J. D. Wright''. The editors of the 1983 edition are Peter H. Rossi, James D. Wright and Andy B. Anderson.", "what_survives": "The chapter title, pages, series and publisher in the entry are as printed, and no result depends on it. I did not read the chapter itself."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E013", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Table 1 and the sentence that follows it. In the State Failure column, the entry 66.8 per cent in the row 'Proportion of missing values (avg.)' is outlined in red. The sentence below the table ends 'we removed 19 variables that exhibited 100 per cent missingness.'", "caption": "Table 1 and the sentence below it, p. 152 (PDF p. 4). The script computes the marked average before the removal of the 19 variables that the sentence describes.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-E014.png"}], "formal_release": null, "formal_statements": [{"id": "P007-T12", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T12"}, {"id": "P007-T13", "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-T13"}], "headline": "The script behind the State Failure column of Table 1 averages missingness before it drops the 19 wholly missing variables, and it counts an empty text entry imported from Stata as observed, a convention that the article does not state.", "id": "P007-E014", "location": "Table 1, p. 152 (PDF p. 4), State Failure column; replication script Simcode_Statefailure.R, lines 13, 28–29, 39–47 and 55", "pdf_page": 4, "printed_page": "152", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P007-T12", "P007-T13"], "text": {"as_printed": "Table 1, State Failure column: Number of observations 8,580; Number of variables 1,205; Proportion of missing values (avg.) \\hl{66.8\\%}; Proportion of missing values (max) 99.99\\%; Number of variables fully observed 79. Below the table: ``and in the case of the State Failure data, to apply the principle of charity, we removed 19 variables that exhibited 100\\% missingness.''\n\nReplication script \\texttt{Simcode\\_Statefailure.R}:\n\\begin{itemize}\n\\item line 13: \\texttt{statefailure <- read\\_dta(\"sort11v3.dta\")}\n\\item line 28: \\texttt{isnamatrix <- is.na(statefailurepure)}\n\\item line 29: \\texttt{prop.missing <- mean(colMeans(isnamatrix))}\n\\item line 44: \\texttt{isnamatrix <- is.na(statefailure\\_final)}\n\\end{itemize}", "corrected": "Table 1, State Failure column: Proportion of missing values (avg.) \\hl{66.3\\%}.\n\nIn the script, add \\texttt{prop.missing <- mean(colMeans(isnamatrix))} after line 44.\n\nAdd to the description of the data: ``A cell is counted as missing when \\texttt{is.na} is true after import with \\texttt{read\\_dta}; an empty text entry is counted as observed.'' An analysis that counts empty text entries as missing should be reported separately, with its own definition of a missing cell and its own set of variables, after the data and their documentation have been inspected.", "evidence": "The evidence is a derivation for the average and a test of the import on a small constructed file.\n\nLet $q_{\\text{before}}$ and $q_{\\text{after}}$ be the average proportions of missing cells over the 1,224 and the 1,205 variables. Since the 19 removed variables are wholly missing and all variables have the same 8,580 rows, I have\n\\[\n1{,}224 \\, q_{\\text{before}} = 19 + 1{,}205 \\, q_{\\text{after}}.\n\\]\nA value that rounds to 66.8\\% satisfies $q_{\\text{before}} < 0.6685$. Then the identity implies that\n\\[\nq_{\\text{after}} < \\frac{1{,}224 \\times 0.6685 - 19}{1{,}205} = \\frac{199{,}811}{301{,}250} < 0.6675,\n\\]\nand the average over the 1,205 variables cannot round to 66.8\\%. Substituting the value in the saved log, $q_{\\text{before}} = 0.668390827582004$, into the identity, I obtain $q_{\\text{after}} = 0.66316\\ldots$, which rounds to 66.3\\%. The log implies 6,856,367 missing cells out of 10,338,900 retained cells. The pre-removal average exceeds the retained average by 0.52 percentage points.\n\nFor the import, a Stata file with four rows was written and read back with \\texttt{read\\_dta} (haven 2.5.5; the authors report version 2.4.0). Its text variable held an empty string, ``0'', ``1'' and a missing value, and its numeric variable had no missing value. The text values came back as an empty string, ``0'', ``1'' and an empty string. Under \\texttt{is.na}, none of the 8 cells is missing and all 4 rows are complete. When empty strings are counted as missing, 2 cells are missing and 2 rows are complete. The test shows how the convention works. It does not show that the State Failure file has empty text entries.", "problem": "Two things are at issue. The first is the average in Table 1, corrected in \\fid{P007-E010}. The script computes it at line 29, over 1,224 variables, and line 55 reports it for the 1,205 variables that remain after the 19 wholly missing ones are dropped.\n\nThe second is how the script decides that a cell is missing. It applies \\texttt{is.na} to data read from a Stata file with \\texttt{read\\_dta}. An empty text entry in a Stata file arrives in R as an empty string, which \\texttt{is.na} does not count as missing. An empty text entry is then counted as an observed value. The article does not state this convention. I have not established whether the State Failure file contains empty text entries or how counting them as missing would change the results. For a fixed set of variables, counting them as missing adds missing cells and removes none. It could only show more rows lost.", "what_survives": "Recomputing the average changes no simulation result, since the simulation uses the 1,205 variables that remain and never reads the average. Figure 2 is unchanged. Under the script's own definition of a missing cell, the count of 79 fully observed variables is unaffected. The results of Section 2, the numerical illustrations and the QoG calculations do not depend on this script. If the State Failure file does hold empty text entries, the count of fully observed variables and the curves of Figure 2 could change under a definition that counts them as missing; no such change is established here."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-E014", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the abstract. A red box marks the clause 'for large n, listwise deletion will drop all rows with probability 1' in the second sentence.", "caption": "Abstract, p. 149 (PDF p. 1). The marked clause states the main result.", "url": "https://pmaronow.github.io/llm-audit/works/P007/exhibits/P007-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "In the abstract, “for large $n$, listwise deletion will drop all rows with probability 1” means that the probability of dropping all rows tends to 1 as $n$ grows, which is the statement of Proposition 5.", "id": "P007-N001", "location": "Abstract, p. 149 (PDF p. 1), second sentence", "pdf_page": 1, "printed_page": "149", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``We show that when (i) all data have some idiosyncratic missingness and (ii) the number of variables grows superlogarithmically in $n$, then, \\hl{for large $n$, listwise deletion will drop all rows with probability 1}.''", "corrected": "``We show that when (i) all data have some idiosyncratic missingness and (ii) the number of variables grows superlogarithmically in $n$, then \\hl{the probability that listwise deletion drops all rows tends to 1 as $n$ grows}.''", "intended_meaning": "The clause summarizes Proposition 5 (p.~151), which states that $\\lim_{n\\to\\infty}p_{all}=1$ under Assumptions 1, 2 and 4, where $p_{all}$ is the probability that listwise deletion removes all rows. This statement is correct. The clause is not a guarantee at a finite $n$: if all entries are independent and each is observed with probability $q\\in(0,1)$, then $p_{all}=(1-q^k)^n<1$ for every $n$ and $k$. For instance, $p_{all}=9/16$ when $q=1/2$ and $n=k=2$.\n\nA stronger reading also holds, although the article does not state it. Suppose that the datasets for all $n$ are defined on one probability space and that the bound $q_*$ of Assumption 2 does not depend on $n$. By Assumption 2, each row is complete with probability at most $q_*^{f(n)}$, and the probability that some row of the dataset with $n$ rows is complete is at most $nq_*^{f(n)}$. Since $f(n)/\\log n\\to\\infty$, I have $nq_*^{f(n)}\\le n^{-2}$ for all large $n$. These bounds are summable, and the first Borel--Cantelli lemma implies that, with probability 1, listwise deletion removes all rows for all sufficiently large $n$. The corresponding sentence of the Introduction says ``asymptotically'' and needs no change."}, "url": "https://pmaronow.github.io/llm-audit/works/P007/#P007-N001", "work": {"arxiv_id": "2101.11470", "doi": "10.1017/pan.2022.5", "title": "Listwise Deletion in High Dimensions", "url": "https://pmaronow.github.io/llm-audit/works/P007/"}, "work_id": "P007"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last line of the Figure 5 caption and four lines of text. Red boxes mark the two lines 'on an error rate of delta = 0.05, we would reject the sharp null hypothesis that there is no cumulative effect on the distance interval of [0.1, 0.5].'", "caption": "Section 5, p. 150 (PDF p. 13), first sentence of text below Figure 5. The marked sentence, which begins on p. 149, rejects the sharp null at $\\delta=0.05$.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E002.png"}, {"alt": "Crop of nine lines of text and one line of code. Red boxes mark the words 'marginally significant' in the sentence 'The results suggest that the AMR estimates are marginally significant when d is small', and the last two lines, 'The 0.025 and .975 percentiles of the sharp null permutation distribution of the cumulative effect are minus 7.97 and 7.97, respectively, while the actual cumulative effect is 12.38. Based'.", "caption": "Section 5, p. 149 (PDF p. 12), text after the output table. The marked words call the estimates marginally significant, and the marked lines compare the cumulative statistic with the two percentiles.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E002-b.png"}, {"alt": "Crop of seven lines of text. Red boxes mark the sentence 'At each distance d, the sharp null hypothesis of no effect is rejected if the AMR estimate is smaller than the delta over 2 percentile or larger than the 1 minus delta over 2 percentile.', which runs over three lines.", "caption": "Section 3, p. 146 (PDF p. 9). The marked sentence is the rejection rule: reject when the estimate lies outside the two percentiles.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E002-c.png"}, {"alt": "Crop of eight lines of text. Red boxes mark the sentence 'Such tests of the sharp null are approximately in exact finite samples, where the quality of the approximation depends the extent to the which the simulated permutations cover the full set of possible permutations.', which runs over four lines.", "caption": "Section 2.3, p. 144 (PDF p. 7). The marked sentence attributes the error of the test to the number of simulated permutations.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E002-d.png"}, {"alt": "Crop of one footnote line, 'The number doesn't make any difference here as there are in total 6 assignments.' A red box marks the words 'in total 6 assignments'.", "caption": "Footnote 5, p. 148 (PDF p. 11). The marked words give the number of assignments in the permutation distribution of the toy example.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E002-e.png"}, {"alt": "Two horizontal strips under the title 'The observed sum is the largest of six equally likely values: exact two-sided p = 2/6. The interpolated percentiles lie inside the two extreme values, so the rule rejects.' The upper strip runs from minus 9.6 to 13.6 and shows six points at about minus 8.00, minus 7.82, minus 4.44, 4.44, 7.82 and 8.00; dashed vertical lines at minus 7.97 and 7.97 are labelled .025 percentile and .975 percentile, and the regions beyond them are shaded and labelled 'rejected by the rule'. A blue diamond labelled 'observed 8.00' sits just right of the upper dashed line, and an orange square labelled 'as printed 12.38' sits far to the right. The lower strip, titled 'Upper tail, enlarged', runs from 7.79 to 8.012 and shows a black point labelled 'second largest 7.8211', a dashed line labelled '.975 percentile 7.9746 = 0.125 times second largest + 0.875 times largest', and the blue diamond labelled 'observed (largest) 7.9965', with an arrow of length 0.0219 between the line and the diamond. The horizontal axis is the sum of the AMR estimates over d = 0.1 to 0.5, in units of the outcome.", "caption": "The six possible values of the cumulative statistic on $[0.1,0.5]$ in the toy example, with the interpolated percentiles (dashed), the observed value (blue diamond) and the value printed on p. 149 (orange square). The lower panel enlarges the upper tail. The observed value is the largest of the six and lies $0.0219$ above the upper percentile.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E002-sixvalues.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T3", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T3"}, {"id": "P008-T4", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T4"}, {"id": "P008-T22", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T22"}], "headline": "With six equally likely assignments the toy example cannot reject the sharp null at $\\delta=0.05$: the exact two-sided p-value of the cumulative statistic is $2/6$, and the rule of p. 146, applied with interpolated percentiles, can reject a true sharp null with probability $1/3$.", "id": "P008-E002", "location": "Section 5, first sentence of p. 150 (PDF p. 13) and the text after the output table on p. 149 (PDF p. 12); the rejection rule of Section 3, p. 146 (PDF p. 9); Section 2.3, p. 144 (PDF p. 7); footnote 5, p. 148 (PDF p. 11)", "pdf_page": 13, "printed_page": "150", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [{"id": "P004-E005", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E005", "work_id": "P004"}], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P008-T22", "P008-T3", "P008-T4"], "text": {"as_printed": "p.~144: ``For a two-way test of the sharp null hypothesis for a specific AMR value with an error rate of $\\delta$, one can evaluate the estimated AMR relative to the $\\delta/2$ and $1-\\delta/2$ quantiles of the null permutation distribution. \\ldots\\ Such tests of the sharp null are \\hl{approximately in exact finite samples, where the quality of the approximation depends the extent to the which the simulated permutations cover the full set of possible permutations}.''\n\np.~146: ``At each distance $d$, the sharp null hypothesis of no effect \\hl{is rejected if the AMR estimate is smaller than the} $\\hl{\\delta/2}$ \\hl{percentile or larger than the} $\\hl{1-\\delta/2}$ \\hl{percentile}.''\n\np.~148, footnote 5: ``The number doesn't make any difference here as there are in total 6 assignments.''\n\np.~149: ``The results suggest that the AMR estimates are \\hl{marginally significant} when $d$ is small, where the estimates are slightly larger than the .975 percentile of the sharp null permutation distribution and the red line on the bottom is close to be positive.''\n\npp.~149--150: ``The 0.025 and .975 percentiles of the sharp null permutation distribution of the cumulative effect are $-7.97$ and $7.97$, respectively, while the actual cumulative effect is \\hl{12.38}. Based [p.~150] on an error rate of $\\delta = 0.05$, \\hl{we would reject the sharp null hypothesis} that there is no cumulative effect on the distance interval of $[0.1, 0.5]$.''", "corrected": "p.~146, the rule. Let $\\Omega$ be the set of assignments used for the permutation distribution, each with the same probability under the design (in the toy example, the six assignments with two treated nodes), and let $S(z)$ be the statistic computed from the observed outcomes with assignment $z$ (the AMR estimate at one distance, or a linear functional such as the cumulative sum). Define the exact two-sided p-value\n\\[\np=\\frac{\\#\\{z\\in\\Omega:\\ \\lvert S(z)\\rvert\\ge\\lvert S(z_{\\mathrm{obs}})\\rvert\\}}{\\lvert\\Omega\\rvert},\n\\]\nwhich counts the observed assignment and all ties. ``At each distance $d$, the sharp null hypothesis of no effect is rejected if \\hl{the exact two-sided p-value of the AMR estimate is at most} $\\hl{\\delta}$.'' Under the sharp null the probability of rejection is then at most $\\delta$. Indeed, the outcomes are fixed under the sharp null, and the assignments with $p\\le\\delta$ are contained in the set of assignments at least as extreme as the least extreme of them, which has at most $\\delta\\lvert\\Omega\\rvert$ elements. Critical values calibrated conservatively for the discrete distribution are an alternative to the p-value. If the assignments have unequal probabilities, \\hl{the entire fraction is replaced by the sum of the probabilities of the assignments counted}. If $\\Omega$ is too large to enumerate and $B$ assignments $Z_1,\\dots,Z_B$ are drawn independently from the design, independently of the observed assignment, the Monte Carlo p-value is\n\\[\np=\\frac{1+\\#\\{b:\\ \\lvert S(Z_b)\\rvert\\ge\\lvert S(z_{\\mathrm{obs}})\\rvert\\}}{B+1}.\n\\]\n\np.~144: ``Such tests of the sharp null \\hl{have level $\\delta$ in finite samples when the decision is based on the exact two-sided p-value. Interpolated $\\delta/2$ and $1-\\delta/2$ quantiles need not give a test of level $\\delta$, even when every permutation is enumerated}.''\n\np.~149: ``The AMR estimates \\hl{at $d=0.4,\\dots,0.7$ are slightly larger than the interpolated .975 percentile of the sharp null permutation distribution; each has exact two-sided p-value $2/6$}.''\n\npp.~149--150: ``The 0.025 and .975 percentiles of the sharp null permutation distribution of the cumulative effect are $-7.97$ and $7.97$, respectively, while the actual cumulative effect is \\hl{8.00, the largest of the six possible values. Its exact two-sided p-value is $2/6\\approx0.33$, so the sharp null hypothesis is not rejected at $\\delta=0.05$. With six equally likely assignments, no nonrandomized test can reject at this level}.''", "evidence": "The evidence is an exact counterexample to the rule, a derivation of the bound on the p-value, and a recalculation of the toy example.\n\n\\emph{Counterexample.} Let four nodes have circle averages $(0,1,2,3)$ at each distance $d=0.1,\\dots,0.5$ and under every assignment, and let two of them be treated. Such outcomes are realized by a spatial field in which successive nodes on a line are 10 units apart and the outcome equals $i-1$ within distance 1 of node $i$ and $0$ elsewhere, whatever the assignment. This field satisfies the sharp null. For the six assignments, I find the following.\n\\begin{audittable}{crcc}{Sharp null true, circle averages $(0,1,2,3)$: the difference in means $T$, its exact two-sided p-value, and the decision of the rule of p.~146 with interpolated percentiles.}\nAssignment & $T$ & Exact two-sided p-value & Rule rejects \\\\\n1100 & $-2$ & $1/3$ & yes \\\\\n1010 & $-1$ & $2/3$ & no \\\\\n1001 & $0$ & $1$ & no \\\\\n0110 & $0$ & $1$ & no \\\\\n0101 & $1$ & $2/3$ & no \\\\\n0011 & $2$ & $1/3$ & yes \\\\\n\\end{audittable}\nThe interpolated percentiles at levels $0.025$ and $0.975$ are $-15/8$ and $15/8$. The rule rejects for two of the six assignments, and its probability of rejection under the sharp null is $1/3$. For the sum over the five distances $0.1,\\dots,0.5$ every value is multiplied by five, the percentiles are $-75/8$ and $75/8$, and the same two assignments are rejected.\n\n\\emph{Bound on the p-value.} Under the sharp null the circle averages $m_i$ do not depend on the assignment. For an assignment $z$ with two treated nodes, the estimator at one distance is\n\\[\nT(z)=\\frac12\\sum_{i=1}^{4}z_im_i-\\frac12\\sum_{i=1}^{4}(1-z_i)m_i=\\frac12\\sum_{i=1}^{4}(2z_i-1)m_i ,\n\\]\nand for the complementary assignment I have $T(1-z)=-T(z)$. The same identity holds for a statistic $S$ that is a linear combination of the estimators at several distances. The exact two-sided p-value $\\#\\{w:\\lvert S(w)\\rvert\\ge\\lvert S(z)\\rvert\\}/6$ counts both $z$ and $1-z$, so it is at least $2/6$.\n\n\\emph{Recalculation.} I generated the toy data as in the package vignette (seed 123567); the observed assignment treats nodes 1 and 2. The data reproduce the thirty printed entries of the columns \\texttt{AMR\\_est}, \\texttt{Per.CI.l} and \\texttt{Per.CI.u}. The six values of the cumulative statistic on $[0.1,0.5]$ are $\\pm7.99649$, $\\pm7.82109$ and $\\pm4.44342$, and the observed value is $7.99649$. The interpolated percentiles are $\\pm7.97457$, in agreement with the printed $-7.97$ and $7.97$, and two of the six values are at least as extreme as the observed one. With the outcomes held fixed at the untreated raster, the rule rejects for two of the six assignments at each of the twenty distances. Both copies of the code in the archived repository of the package compute the percentiles with the R function \\texttt{quantile} at its default setting. I did not have access to the release cited in the paper.", "problem": "The toy example has four nodes, two of them treated. The permutation distribution of footnote 5 consists of six assignments that are equally likely. It is the Bernoulli design of Section 4, conditional on two treated nodes. On such a set no nonrandomized test can reject at $\\delta=0.05$, since a rejection region that contains a single assignment already has probability $1/6$. For a difference in means with two nodes in each group more is true. The complementary assignment reverses the sign of the statistic, so at least two of the six values are as extreme as the observed one, and the exact two-sided p-value is at least $2/6$.\n\nThe printed rejection arises because the percentiles are interpolated. For six values, the default R quantile at level $0.975$ equals $0.125$ times the second largest value plus $0.875$ times the largest, which is below the largest value whenever the two differ. The rule of p.~146 rejects when the estimate is larger than this percentile or smaller than its counterpart at level $0.025$. It therefore rejects whenever the observed assignment gives the largest or the smallest of six distinct values. Under the sharp null this event has probability $2/6$, not $0.05$, and enumerating every permutation does not change it. The sentence on p.~144 attributes the error of the test to the number of simulated permutations; for this rule the error remains when all permutations are used.\n\nIn the toy example the observed cumulative statistic on $[0.1,0.5]$ is $8.00$ (the printed $12.38$ is corrected in \\fid{P008-E003}). It is the largest of the six values and exceeds the interpolated upper percentile by about $0.02$. Its exact two-sided p-value is $2/6\\approx0.33$, and the sharp null is not rejected at $\\delta=0.05$. The sharp null is likewise not rejected at single distances. The estimates at $d=0.4,\\dots,0.7$ lie above the blue curve of Figure 5, and each has exact two-sided p-value $2/6$; the estimates at $d=0.1,0.2,0.3$ lie below it. As a statement about the randomization test, the words ``marginally significant'' are not supported at the 5\\% level at any distance.", "what_survives": "The definitions of the spatial potential outcomes and of the AMR, the toy point estimates and the curves of Figure 5 are unaffected. The cumulative statistic can still be used in a randomization test with a calibrated rule. The spatial-HAC intervals and the asymptotic claims are separate and are not addressed here. I find no additional error in the phrase ``when $d$ is small'': it can refer to $d=0.4,\\dots,0.7$, where the estimates do exceed the upper percentile. The hypothesis tested is the sharp null, which Section 2.3 distinguishes from the weak null.\n\nThe toy rejection is withdrawn, and the rule of p.~146 and the sentence on p.~144 are replaced. What is shown is a failure of calibration for a design with six possible assignments. The application of Section 6 uses the same rule, and its p-value has to be recomputed on its own assignment distribution; whether its conclusion changes is not established here."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E002", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of three lines: the code line 'testResult <- SpatialEffectTest(AMRest, c(0.1, 0.5))' and the sentence 'The 0.025 and .975 percentiles of the sharp null permutation distribution of the cumulative effect are minus 7.97 and 7.97, respectively, while the actual cumulative effect is 12.38.' A red box marks the number 12.38.", "caption": "Section 5, last lines of p. 149 (PDF p. 12). The marked number is reported as the cumulative statistic for the call above it, whose range is $[0.1,0.5]$.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E003.png"}, {"alt": "Crop of the header and first six rows of the printed output table, with columns dVec, AMR_est, Conley.CI.l, Conley.CI.u, Per.CI.l and Per.CI.u. A red box marks the AMR_est entries of rows 1 to 5: 1.491, 1.550, 1.550, 1.691 and 1.714.", "caption": "Output table, p. 149 (PDF p. 12), first six rows. The five marked AMR estimates, at $d=0.1,\\dots,0.5$, sum to $7.996$.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T11", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T11"}], "headline": "The observed cumulative statistic on $[0.1,0.5]$ is $8.00$, the sum of the five printed AMR estimates; the printed $12.38$ matches the sum of the estimates through $d=0.8$.", "historical_evidence_type": "lean_proof", "id": "P008-E003", "location": "Section 5, last sentence of p. 149 (PDF p. 12), after the call to SpatialEffectTest(); output table on the same page", "pdf_page": 12, "printed_page": "149", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T11"], "text": {"as_printed": "\\texttt{testResult <- SpatialEffectTest(AMRest, c(0.1, 0.5))}\n\n``The 0.025 and .975 percentiles of the sharp null permutation distribution of the cumulative effect are $-7.97$ and $7.97$, respectively, while the actual cumulative effect is \\hl{12.38}.''\n\n\\begin{audittable}{lrrrrrrrrrr}{Output table on the same page, columns \\texttt{dVec} and \\texttt{AMR\\_est}.}\n\\texttt{dVec} & $0.1$ & $0.2$ & $0.3$ & $0.4$ & $0.5$ & $0.6$ & $0.7$ & $0.8$ & $0.9$ & $1.0$ \\\\\n\\texttt{AMR\\_est} & $1.491$ & $1.550$ & $1.550$ & $1.691$ & $1.714$ & $1.700$ & $1.678$ & $1.002$ & $-0.816$ & $-0.791$ \\\\\n\\end{audittable}", "corrected": "``The 0.025 and .975 percentiles of the sharp null permutation distribution of the cumulative effect are $-7.97$ and $7.97$, respectively, while the actual cumulative effect is \\hl{8.00}.''\n\nThe unrounded sum is $7.99649$. It exceeds the upper interpolated percentile, $7.97457$, by $0.02193$. With six equally likely assignments the exact two-sided p-value of this sum is $1/3$, so the example does not establish a rejection of the sharp null at the 5\\% level.", "evidence": "The evidence is a derivation from the printed table and a recalculation of the toy example. From the printed estimates, I have\n\\[\n1.491+1.550+1.550+1.691+1.714=7.996 .\n\\]\nEach printed estimate is within $0.0005$ of its unrounded value, so the unrounded sum lies in $[7.9935,\\,7.9985]$. A number printed as $12.38$ lies in $[12.375,\\,12.385]$, and the two intervals are disjoint. Adding the next three estimates, I obtain $7.996+1.700+1.678+1.002=12.376$, which rounds to $12.38$. A sum multiplied by the step $0.1$ of the grid would be $0.7996$.\n\nFor the recalculation, I generated the toy data as in the package vignette (seed 123567) and enumerated the six assignments with two treated nodes. These data reproduce the ten printed estimates and their permutation bounds, and give the following.\n\\begin{audittable}{lrr}{The cumulative statistic and its interpolated percentiles on two ranges, recalculated.}\nRange & Observed sum & .025 and .975 percentiles \\\\\n$[0.1,\\,0.5]$ & $7.99649$ & $-7.97457$, $7.97457$ \\\\\n$[0.1,\\,0.8]$ & $12.37571$ & $-12.30669$, $12.30669$ \\\\\n\\end{audittable}\nThe \\texttt{SpatialEffectTest()} function in the archived repository of the package returns the same values. The release of the package cited in the paper and the script that produced the number are not available. The agreement with the sum through $d=0.8$ is a numerical fact and does not show how $12.38$ arose.", "problem": "Section 3 (p.~146) says that \\texttt{SpatialEffectTest()} ``generates the null permutation distribution of the sum of the AMR estimates within the specified range''. The ``actual cumulative effect'' for the range $[0.1,0.5]$ is therefore the sum of the estimates at $d=0.1,\\dots,0.5$. The five printed estimates sum to $7.996$, not to $12.38$. Note that the percentiles $-7.97$ and $7.97$ do belong to $[0.1,0.5]$. The sentence compares the percentiles of one range with a number that matches the observed sum on a wider range. It suggests a margin of about $4.4$ above the upper percentile, and the margin is $0.02$.", "what_survives": "The ten AMR estimates, their permutation bounds and the percentiles $-7.97$ and $7.97$ stand, and no later result is computed from this number. The corrected sum still exceeds the upper percentile, by $0.022$. The conclusion drawn from that comparison on p.~150, a rejection at $\\delta=0.05$, fails for a separate reason given in \\fid{P008-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E003", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of text and three lines of code. Red boxes mark '5/111 approximately equal to 0.045' in the sentence 'with a step size of 5/111 approximately 0.045', and 'by=5/111' in the code line 'dVec = seq(from=0, to=20/111, by=5/111) AMRest <- Spatial'.", "caption": "Section 6, p. 153 (PDF p. 16). The marked step size in the text and the marked argument in the code both give a step of 5 km.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E004.png"}, {"alt": "Crop of the two upper panels of Figure 8, 'Hajek Estimator with Its CIs' on the left and 'Smoothed Estimator with Its CIs' on the right, each with AMR estimates from minus 0.04 to 0.04 against distance from 0 to 20 kilometres. In the left panel a jagged black line with many short segments runs from about minus 0.01 at 0 km to about 0 at 20 km, between dotted red and blue lines. A tall red box marks the vertical band between 1 and 2.5 km in the left panel.", "caption": "Figure 8, upper panels, p. 154 (PDF p. 17). The marked band is the test range of 1 to 2.5 km in the panel of the Hajek estimator. The black curve has a point every 0.5 km, and four of its points lie in the band.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T12", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T12"}, {"id": "P008-T13", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T13"}], "headline": "The step of the distance grid is $0.5/111\\approx0.0045$, that is, 0.5 km, and not $5/111\\approx0.045$; with the printed step no grid point lies in the test range of 1 to 2.5 km.", "id": "P008-E004", "location": "Section 6, first paragraph and the two printed calls, p. 153 (PDF p. 16); Figure 8, p. 154 (PDF p. 17)", "pdf_page": 16, "printed_page": "153", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T12", "P008-T13"], "text": {"as_printed": "``Since 1 decimal degree is approximately 111 km, the distance values we select are between 0 and $20/111 \\approx 0.18$, with a step size of $\\hl{5/111 \\approx 0.045}$.''\n\n\\texttt{dVec = seq(from=0, to=20/111, \\hl{by=5/111}) AMRest <- Spatial}\n\n\\texttt{Effect(ras = ras, ras\\_Z = ras\\_Z, treatment = ’’treatment’’,}\n\n\\texttt{dVec = dVec, smooth= 1, cutoff = 10000,}\n\n\\texttt{dist.metric = ’’Geodesic’’)}\n\n\\texttt{testResult <- SpatialEffectTest(AMRest, test.range = c(1/111,}\n\n\\texttt{2.5/111))}\n\n``The estimated cumulative effect between 1 km and 2.5 km is $-0.0321$ (i.e., a 3.21 percentage point reduction).''", "corrected": "``Since 1 decimal degree is approximately 111 km, the distance values we select are between 0 and $20/111 \\approx 0.18$, with a step size of $\\hl{0.5/111 \\approx 0.0045}$.''\n\n\\texttt{dVec = seq(from=0, to=20/111, \\hl{by=0.5/111})}\n\nThe grid then has 41 points, at $0,0.5,\\dots,20$ km, and the four points at 1, 1.5, 2 and 2.5 km lie in the test range. The object \\texttt{AMRest} has to be computed on this grid before the test is run. A full replication of the test also needs the transformed inputs of the application and the release of the package used for the paper, which are not available.", "evidence": "The evidence is a derivation and a check of Figure 8. The printed grid is $d_k=5k/111$ for $k=0,\\dots,4$. The condition $d_k\\in[1/111,\\,2.5/111]$ is equivalent to $1\\le 5k\\le 5/2$, which fails for $k=0$ and for every $k\\ge 1$. The \\texttt{SpatialEffectTest()} function in the archived repository of the package, run with the printed grid on an example with four nodes and all six assignments enumerated, returns an observed sum of $0$ and both percentiles equal to $0$.\n\nIn the PDF, the black line of the Hajek panel of Figure 8 is a polyline with 41 vertices. Their horizontal positions agree with $0,0.5,\\dots,20$ km to within $0.0002$ km. The values at the four vertices in the test range, read from the axes of the figure, are as follows.\n\\begin{audittable}{rr}{The four plotted AMR estimates of Figure 8 in the range of 1 to 2.5 km.}\nDistance (km) & AMR estimate read from the plot \\\\\n$1.0$ & $-0.00968$ \\\\\n$1.5$ & $-0.00927$ \\\\\n$2.0$ & $-0.00493$ \\\\\n$2.5$ & $-0.00821$ \\\\\nSum & $-0.03209$ \\\\\n\\end{audittable}\nThe sum rounds to the reported $-0.0321$. These values are read from a published graphic and are not a recalculation from the data of the application, which are not available.", "problem": "The printed grid \\texttt{seq(from=0, to=20/111, by=5/111)} has five points, at 0, 5, 10, 15 and 20 km. The test range is $[1/111,\\,2.5/111]$, that is, 1 to 2.5 km, and none of the five points lies in it. By Section 3 (p.~146), \\texttt{SpatialEffectTest()} sums the AMR estimates at the grid points within the range. With the printed grid the observed sum and every permuted sum are empty sums, equal to zero, and the reported statistic $-0.0321$ and percentiles $-0.0304$ and $0.0336$ cannot come from the printed commands.\n\nFigure 8 shows the grid that was used. Its unsmoothed curve has 41 points at a spacing of 0.5 km, and the four estimates at 1, 1.5, 2 and 2.5 km sum to $-0.0321$. The step $5/111$ is a misprint for $0.5/111$, in the text and in the code.", "what_survives": "No reported number or conclusion changes: Figure 8 and the reported cumulative statistic agree with a grid of step 0.5 km. The correction concerns the grid only. The percentiles $-0.0304$ and $0.0336$ and the statement $p<0.05$ could not be recomputed and are not certified by it. The units of the cumulative statistic are the subject of \\fid{P008-N016}, and the layout of the printed call that of \\fid{P008-E009}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E004", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the display and the lines around it. The text reads: (AMR) for distance d, which is simply the average d-radius effect over the intervention nodes. The display reads: tau of d and alpha equals one over N times the sum over i from 1 to N of tau i of d and alpha, followed with no sign between them by one over N times the sum over i from 1 to N of the bracket mu-bar i of 1, d, alpha minus mu-bar i of 0, d, alpha. The next line reads: The interpretation of the AMR for distance d is the average effect of switching a node i in calligraphic N. Red outlines surround the second fraction one over N, the two symbols mu-bar i, and the words i in calligraphic N.", "caption": "Section 2.2, p. 143 (PDF p. 6), display defining the AMR. The outline at the second fraction marks the place of the missing equals sign; the other outlines mark the two bars and the undefined index set.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T14", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T14"}], "headline": "The display of the AMR lacks the equals sign between its two averages and carries bars on $\\mu_i$ that are not defined: the AMR is $\\tau(d;\\alpha)=N^{-1}\\sum_{i=1}^N\\tau_i(d;\\alpha)=N^{-1}\\sum_{i=1}^N[\\mu_i(1;d,\\alpha)-\\mu_i(0;d,\\alpha)]$.", "id": "P008-E005", "location": "Section 2.2, p. 143 (PDF p. 6), display defining $\\tau(d;\\alpha)$ and the sentence after it", "pdf_page": 6, "printed_page": "143", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T14"], "text": {"as_printed": "\\[\n\\tau(d;\\alpha)=\\frac{1}{N}\\sum_{i=1}^{N}\\tau_i(d;\\alpha)\\,\\hl{\\phantom{=}}\\,\\frac{1}{N}\\sum_{i=1}^{N}\\left[\\hl{\\bar{\\mu}_i}(1;d,\\alpha)-\\hl{\\bar{\\mu}_i}(0;d,\\alpha)\\right].\n\\]\nThe interpretation of the AMR for distance $d$ is the average effect of switching a node $\\hl{i\\in\\mathcal{N}}$ to treatment on points at distance $d$ from that node, marginalized over possible realizations of treatment statuses in other intervention nodes.", "corrected": "Let $\\mu_i(z;d,\\alpha)=\\mathrm{E}_{\\mathbf{Z}_{-i}}[\\mu_i(\\mathbf{Y}(z,\\mathbf{Z}_{-i});d)]$ be the potential circle average of radius $d$ around node $i$ under the design $\\alpha$, and let $\\tau_i(d;\\alpha)=\\mu_i(1;d,\\alpha)-\\mu_i(0;d,\\alpha)$, as on p.~143. The AMR for distance $d$ is\n\\[\n\\tau(d;\\alpha)=\\frac{1}{N}\\sum_{i=1}^{N}\\tau_i(d;\\alpha)\\hl{=}\\frac{1}{N}\\sum_{i=1}^{N}\\left[\\hl{\\mu_i}(1;d,\\alpha)-\\hl{\\mu_i}(0;d,\\alpha)\\right].\n\\]\nThe interpretation of the AMR for distance $d$ is the average effect of switching a node $\\hl{i\\in\\{1,\\ldots,N\\}}$ to treatment on points at distance $d$ from that node, marginalized over possible realizations of treatment statuses in other intervention nodes.", "evidence": "I derived the identity from the definitions that precede the display. On p.~143 the paper sets $\\tau_i(d;\\alpha)=\\mu_i(1;d,\\alpha)-\\mu_i(0;d,\\alpha)$. Substituting this into the first average, I obtain\n\\[\n\\frac1N\\sum_{i=1}^N\\tau_i(d;\\alpha)=\\frac1N\\sum_{i=1}^N\\left[\\mu_i(1;d,\\alpha)-\\mu_i(0;d,\\alpha)\\right],\n\\]\nand the two averages are equal. The missing sign is an equals sign. The product is a different quantity. For $N=2$ with $(\\mu_1(1),\\mu_2(1))=(3,7)$ and $(\\mu_1(0),\\mu_2(0))=(0,2)$, each average equals $4$ and their product is $16$. The estimator on p.~144 is written with $\\mu_i$ without a bar.", "problem": "The two averages stand side by side with no sign between them, and as printed the display reads as their product. The second average is written with $\\bar{\\mu}_i$, although the page defines only $\\mu_i(z;d,\\alpha)$. The sentence after the display uses a set $\\mathcal{N}$ that is not defined. By p.~141 it can only be the set $\\{1,\\ldots,N\\}$ of indices of the intervention nodes.", "what_survives": "The AMR as defined in the sentences before the display is unchanged, and so is the Hajek estimator on p.~144. Only the display and the index set after it change."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E005", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text. The text reads: As discussed in the previous section, the package provides users with two inference methods, a Fisher-style permutation test (per.se) and an asymptotic variance estimator based on Conley (1999) (conley.se). Users can change the error rate delta defining the level of statistical significance (conley.se). The permutation test can be adjusted to account for blocking or clustering in experimental design. For the asymptotic variance estimator. A red outline surrounds the second (conley.se), the one after the words statistical significance.", "caption": "Section 3, p. 145 (PDF p. 8), last paragraph. The outline marks the second (conley.se), which should read (alpha).", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E006.png"}], "formal_release": null, "formal_statements": [], "headline": "The argument that sets the error rate $\\delta$ is alpha, not conley.se; the argument conley.se switches the spatial-HAC output on and off.", "id": "P008-E006", "location": "Section 3, p. 145 (PDF p. 8), last paragraph, second sentence", "pdf_page": 8, "printed_page": "145", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "As discussed in the previous section, the package provides users with two inference methods, a Fisher-style permutation test (\\texttt{per.se}) and an asymptotic variance estimator based on Conley (1999) (\\texttt{conley.se}). Users can change the error rate $\\delta$ defining the level of statistical significance (\\hl{\\texttt{conley.se}}).", "corrected": "Users can change the error rate $\\delta$ defining the level of statistical significance (\\hl{\\texttt{alpha}}).", "evidence": "I checked the page against p.~146 and against the argument lists and manuals of both archived versions of the package. The condition \\texttt{conley.se == 1} in the source is false at the value 0.05, and the block that computes the unsmoothed spatial-HAC standard errors is then skipped. The release that the paper cites (version 0.1) is not archived, and I did not examine it.", "problem": "Since the names in parentheses in Section 3 are arguments of \\texttt{SpatialEffect()}, the sentence says that \\texttt{conley.se} sets the level $\\delta$. The next page says that \\texttt{Conley.SE} and \\texttt{Conley.CI} are returned ``if \\texttt{conley.se} equals 1'', and in both archived versions of the package the level is the argument \\texttt{alpha}, with default 0.05. In those versions, setting \\texttt{conley.se = 0.05} skips the unsmoothed spatial-HAC output.", "what_survives": "Users can change the level, as the sentence says, and the first \\texttt{(conley.se)} in the paragraph is correct. The two printed calls (pp.~148 and 153) set neither argument, and no reported number is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E006", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text. The text reads: SpatialEffect package offers is to define the circle average as points that are within a k-coordinate-wide bandwidth around points that are d coordinate units from the intervention node; this latter specification defines a disc or circular band (or donut) of intervention nodes that are between d minus k and d plus k coordinate units from the intervention node. The quantity mu i of Y of z and d is the average of points selected with the specified distance metric. Then. A red outline surrounds the words of intervention nodes.", "caption": "Section 2.2, p. 143 (PDF p. 6), paragraph after the circle-average display. The outline marks the phrase that names the wrong set.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T15", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T15"}], "headline": "The circular band consists of outcome points, not of intervention nodes.", "id": "P008-E007", "location": "Section 2.2, p. 143 (PDF p. 6), paragraph after the circle-average display, sentence beginning “Another option that the SpatialEffect package offers”", "pdf_page": 6, "printed_page": "143", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T15"], "text": {"as_printed": "Another option that the \\texttt{SpatialEffect} package offers is to define the circle average as points that are within a $k$-coordinate-wide bandwidth around points that are $d$ coordinate units from the intervention node; this latter specification defines a disc or circular band (or ``donut'') \\hl{of intervention nodes} that are between $d-k$ and $d+k$ coordinate units from the intervention node.", "corrected": "\\ldots\\ this latter specification defines a disc or circular band (or ``donut'') \\hl{of outcome points} that are between $d-k$ and $d+k$ coordinate units from the intervention node.", "evidence": "I checked the page against the definitions on p.~141 and the circle-average display on p.~143, which integrates $Y_x(\\mathbf{z})$ over the points $x$ such that $d_i(x)=d$, for one intervention node $i$.", "problem": "Since the band is the set over which outcomes are averaged, it consists of points $x$ at which outcomes are defined. Section 2.1 indexes these points separately from the intervention nodes $i$, and the words ``of intervention nodes'' name the wrong set.", "what_survives": "The formulas and the computations use outcome points, and nothing depends on the phrase. The width of the band is the subject of \\fid{P008-N001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E007", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of eight lines of text from the description of the output of the SpatialEffect command. Red boxes mark the word 'percentiles' after 'the delta over 2 and 1 minus delta over 2', and the word 'percentile' twice in the next sentence, 'smaller than the delta over 2 percentile or larger than the 1 minus delta over 2 percentile'.", "caption": "Section 3, p. 146 (PDF p. 9), the description of Per.CI. The three marked words name the levels $\\delta/2$ and $1-\\delta/2$ as percentiles.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E008.png"}, {"alt": "Crop of six lines of text. Red boxes mark '0.025 and .975' at the end of one line and 'percentiles' at the start of the next, in the sentence 'The blue dotted line shows the 0.025 and .975 percentiles of the sharp null permutation distribution'.", "caption": "Section 5, p. 149 (PDF p. 12), the description of Figure 5. The marked words give the levels $0.025$ and $.975$ and call them percentiles.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The numbers $\\delta/2$ and $1-\\delta/2$, and $0.025$ and $0.975$, are quantile levels on the probability scale; as percentiles they are the 2.5th and the 97.5th for $\\delta=0.05$.", "id": "P008-E008", "location": "Section 3, description of Per.CI, p. 146 (PDF p. 9); Section 5 and Figure 5 caption, pp. 149–150 (PDF pp. 12–13); Section 6 and Figure 8 caption, pp. 153–154 (PDF pp. 16–17)", "pdf_page": 9, "printed_page": "146", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~146: ``For a specified error rate of $\\delta$ (e.g., $\\delta = 0.05$), it is the $\\delta/2$ and $1-\\delta/2$ \\hl{percentiles} of the sharp null distribution generated by permutation. At each distance $d$, the sharp null hypothesis of no effect is rejected if the AMR estimate is smaller than the $\\delta/2$ \\hl{percentile} or larger than the $1-\\delta/2$ \\hl{percentile}.''\n\np.~149: ``The blue dotted line shows the 0.025 and .975 \\hl{percentiles} of the sharp null permutation distribution.'' On the same page: ``slightly larger than the .975 \\hl{percentile} of the sharp null permutation distribution'' and ``The 0.025 and .975 \\hl{percentiles} of the sharp null permutation distribution of the cumulative effect are $-7.97$ and $7.97$''.\n\np.~150, Figure 5 caption: ``the 0.025 and 0.975 \\hl{percentiles} of the sharp null permutation distribution.''\n\np.~153: ``the 0.025 and .975 \\hl{percentiles} of the sharp null permutation distribution are $-0.0304$ and $0.0336$''.\n\np.~154, Figure 8 caption: ``the 0.025 and .975 \\hl{percentiles} of the sharp null permutation distribution.''", "corrected": "Replace ``percentiles'' by ``quantiles'' at each of the places quoted: ``the $\\delta/2$ and $1-\\delta/2$ \\hl{quantiles}'' and ``the $\\delta/2$ \\hl{quantile} or \\ldots\\ the $1-\\delta/2$ \\hl{quantile}'' on p.~146, ``the .975 \\hl{quantile}'' on p.~149, ``the 0.025 and .975 \\hl{quantiles}'' on pp.~149, 153 and 154, and ``the 0.025 and 0.975 \\hl{quantiles}'' on p.~150. An equivalent repair keeps the word and multiplies the levels by 100: ``the \\hl{2.5th and 97.5th} percentiles''.\n\nThe decision rule in the second sentence on p.~146 is corrected separately in \\fid{P008-E002}.", "intended_meaning": "The passages give quantile levels on the probability scale and call them percentiles. A percentile rank is 100 times the level. Read literally, the ranks .025 and .975 are 100 times smaller than the ranks 2.5 and 97.5 that are meant. The intended meaning is clear from the context. The error rate is $\\delta=0.05$, and Section 2.3 (p.~144) names the same cutoffs correctly as ``the $\\delta/2$ and $1-\\delta/2$ quantiles of the null permutation distribution''. The legend of Figure 8 reads ``Quantiles of sharp null distribution'', and both copies of the code in the archived repository of the package compute the cutoffs with the R function \\texttt{quantile} at the levels \\texttt{alpha/2} and \\texttt{1-alpha/2}. For $\\delta=0.05$ these are the 2.5th and 97.5th percentiles.\n\nThis note concerns the label only. No reported cutoff was recomputed for it, and it does not address whether a test based on these cutoffs has level $\\delta$, which is the subject of \\fid{P008-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E008", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The four printed lines of R code on page 153, with two lines of text above and below. Red boxes mark four items: the assignment to AMRest that ends the first code line with the word Spatial, the word Effect that starts the second line, and the strings treatment and Geodesic, each set between pairs of apostrophes.", "caption": "Section 6, displayed call, p. 153 (PDF p. 16). The marks show the second assignment run onto the first line, the function name split across the line end, and the two strings set in typographic apostrophes.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "As printed, the displayed R commands do not parse: two assignments are run together, the name SpatialEffect is split across a line, and the strings are set in typographic quotation marks.", "id": "P008-E009", "location": "Section 6, displayed SpatialEffect() call, p. 153 (PDF p. 16)", "pdf_page": 16, "printed_page": "153", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{dVec = seq(from=0, to=20/111, by=5/111) \\hl{AMRest <- Spatial}}\n\n\\texttt{\\hl{Effect}(ras = ras, ras\\_Z = ras\\_Z, treatment = \\hl{’’treatment’’},}\n\n\\texttt{dVec = dVec, smooth= 1, cutoff = 10000,}\n\n\\texttt{dist.metric = \\hl{’’Geodesic’’})}", "corrected": "\\texttt{dVec = seq(from=0, to=20/111, by=5/111)}\n\n\\texttt{AMRest <- \\hl{SpatialEffect}(ras = ras, ras\\_Z = ras\\_Z, treatment = \\hl{\"treatment\"},}\n\n\\texttt{dVec = dVec, smooth= 1, cutoff = 10000,}\n\n\\texttt{dist.metric = \\hl{\"Geodesic\"})}", "evidence": "The evidence is a check of the page and a parse of the printed commands in R, which stops with an ``unexpected symbol'' error at \\texttt{AMRest}. The corrected commands parse as the two intended expressions. No estimation was run.", "problem": "The assignment to \\texttt{dVec} and the assignment to \\texttt{AMRest} are printed on one line with no separator, the function name is broken into \\texttt{Spatial} and \\texttt{Effect} at the line end, and the strings are delimited by pairs of typographic apostrophes. R stops at \\texttt{AMRest}. With the two assignments separated and ordinary quotes restored, R still reads \\texttt{AMRest <- Spatial} followed by a separate call to \\texttt{Effect()}.", "what_survives": "The intended commands and the surrounding explanation are unaffected; this is a repair of syntax only. It does not address the grid of distances in the same call, which is the subject of \\fid{P008-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E009", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of text. The text reads: Our analysis focuses on a quantity that we call the average marginalized response (AMR). The AMR measures how, on average, outcomes that at a given distance from an intervention site are affected by activating treatment at that site, taking into account ambient effects emanating from other intervention sites. From a policy perspective, the. A red outline surrounds the words that at.", "caption": "Section 1, p. 140 (PDF p. 3), paragraph beginning “Our analysis focuses”. The outline marks the place of the missing verb.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E010.png"}, {"alt": "Crop of five lines of text. The marked sentence reads: Such tests of the sharp null are approximately in exact finite samples, where the quality of the approximation depends the extent to the which the simulated permutations cover the full set of possible permutations. Red outlines surround the words approximately in exact and the words depends the extent to the which.", "caption": "Section 2.3, p. 144 (PDF p. 7), end of the description of the permutation test. The outlines mark the two garbled phrases.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E010-b.png"}, {"alt": "Crop of three lines of text. The text reads: We demonstrate the utility of the SpatialEffect package by evaluating a payments for ecosystems services (PES) conservation program in Uganda. PES programs are a form of environmental governance that financially renumerate landowners for environmental. A red outline surrounds the word renumerate.", "caption": "Section 6, p. 150 (PDF p. 13), last lines of the page. The outline marks the misspelled word.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E010-c.png"}, {"alt": "Crop of three lines of text. The text reads: number of treated villages within 5 km. The key reason for why we find more compelling evidence is that our analysis is not restricted to forest loss within the village boundaries per se, which was that case for Jayachandran et al. (2017). Our analysis extends to forest areas. A red outline surrounds the words that case.", "caption": "Section 6, p. 153 (PDF p. 16), last lines of the page. The outline marks the wrong word.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E010-d.png"}], "formal_release": null, "formal_statements": [], "headline": "Four sentences contain slips of copy: read “outcomes that are at a given distance” (p. 140), “approximately exact in finite samples” and “depends on the extent to which” (p. 144), “remunerate” (p. 150) and “which was the case” (p. 153).", "id": "P008-E010", "location": "Section 1, p. 140 (PDF p. 3); Section 2.3, p. 144 (PDF p. 7); Section 6, p. 150 (PDF p. 13) and p. 153 (PDF p. 16)", "pdf_page": 3, "printed_page": "140", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~140.} The AMR measures how, on average, outcomes \\hl{that at} a given distance from an intervention site are affected by activating treatment at that site, taking into account ambient effects emanating from other intervention sites.\n\n\\emph{p.~144.} Such tests of the sharp null are \\hl{approximately in exact} finite samples, where the quality of the approximation \\hl{depends the extent to the which} the simulated permutations cover the full set of possible permutations.\n\n\\emph{p.~150.} PES programs are a form of environmental governance that financially \\hl{renumerate} landowners for environmental performance, \\ldots\n\n\\emph{p.~153.} The key reason for why we find more compelling evidence is that our analysis is not restricted to forest loss within the village boundaries per se, which was \\hl{that case} for Jayachandran et al. (2017).", "corrected": "\\emph{p.~140.} The AMR measures how, on average, outcomes that \\hl{are} at a given distance from an intervention site are affected by activating treatment at that site, \\ldots\n\n\\emph{p.~144.} Such tests of the sharp null are \\hl{approximately exact in} finite samples, where the quality of the approximation \\hl{depends on the extent to which} the simulated permutations cover the full set of possible permutations.\n\n\\emph{p.~150.} PES programs are a form of environmental governance that financially \\hl{remunerate} landowners for environmental performance, \\ldots\n\n\\emph{p.~153.} \\ldots\\ our analysis is not restricted to forest loss within the village boundaries per se, which was \\hl{the case} for Jayachandran et al. (2017).", "evidence": "I checked the four pages.", "problem": "A verb is missing on p.~140. On p.~144 two words are out of order, a preposition is missing and an article is misplaced. On p.~150 and p.~153 one word is wrong. The context fixes the intended meaning in each case.", "what_survives": "Nothing depends on these phrases. The correction on p.~144 repairs the wording only. Whether the test described there is exact in finite samples is the subject of \\fid{P008-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E010", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Three entries of the reference list on page 155. Red boxes mark two strings of names: Dean E, Cyrus S, Stephanie Z in the entry for Aronow and coauthors (2020), and Erin O, Sills in the entry for Arriagada and coauthors (2012).", "caption": "References, p. 155 (PDF p. 18). The marked strings give three coauthors by given name and split Erin O. Sills into two authors; the entry above prints “Samii C” correctly. The entry for Hudgens and Halloran on p. 156 and the citation on p. 140 are not shown.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E011.png"}], "formal_release": null, "formal_statements": [], "headline": "Author names are garbled in three reference entries and in one citation in the text: given names are taken as surnames, and one author is split into two.", "id": "P008-E011", "location": "References, pp. 155–156 (PDF pp. 18–19); in-text citation, p. 140 (PDF p. 3)", "pdf_page": 18, "printed_page": "155–156", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Aronow PM, \\hl{Dean E, Cyrus S, Stephanie Z} (2020) Spillover effects in experimental data. arXiv:2001.05444\n\nArriagada RA, Ferraro PJ, \\hl{Erin O, Sills}, Pattanayak SK, Cordero-Sancho S (2012) Do payments for environmental services affect forest cover? A farm-level evaluation from costa rica. Land Econ 88(2):382--399.\n\nHudgens MG, \\hl{Elizabeth Halloran M} (2008) Toward causal inference with interference. J Am Stat Assoc 103(482):832--842\n\n[p.~140] Hudgens and \\hl{Elizabeth} Halloran (2008)", "corrected": "Aronow PM, \\hl{Eckles D, Samii C, Zonszein S} (2020) Spillover effects in experimental data. arXiv:2001.05444\n\nArriagada RA, Ferraro PJ, \\hl{Sills EO}, Pattanayak SK, Cordero-Sancho S (2012) Do payments for environmental services affect forest cover? A farm-level evaluation from costa rica. Land Econ 88(2):382--399.\n\nHudgens MG, \\hl{Halloran ME} (2008) Toward causal inference with interference. J Am Stat Assoc 103(482):832--842\n\n[p.~140] Hudgens and \\hl{Halloran} (2008)", "evidence": "The evidence is a check of the pages against public author records: the author list of the cited arXiv preprint, the publication record of the \\emph{Land Economics} article and the curriculum vitae of E. O. Sills, and the PubMed record of the article of Hudgens and Halloran together with the curriculum vitae of M. E. Halloran. The journal versions of record of the last two articles were not read.", "problem": "The first entry gives the coauthors Dean Eckles, Cyrus Samii and Stephanie Zonszein by given name and an initial of the surname. The second splits the single author Erin O. Sills into ``Erin O'' and ``Sills''. The third, and the citation on p.~140, treat ``Elizabeth'' as part of the surname of M. Elizabeth Halloran.", "what_survives": "The three works remain identifiable from their titles, years and identifiers, and no result of the paper is touched."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E011", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of code and one line of text. The code is 'dVec = seq(from=0, to=20/111, by=5/111) AMRest <- Spatial', 'Effect(ras = ras, ras_Z = ras_Z, treatment = ''treatment'',', 'dVec = dVec, smooth= 1, cutoff = 10000,' and 'dist.metric = ''Geodesic'')'. Red boxes mark 'ras_Z = ras_Z, treatment = ''treatment'',' and the string ''Geodesic'', each printed between pairs of apostrophes.", "caption": "Section 6, p. 153 (PDF p. 16), the printed call for the application. The first marked part supplies the polygons and the name of the treatment attribute, and the call has no Zdata. The second marked part shows the paired apostrophes printed in place of quotation marks.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E012.png"}, {"alt": "Crop of five lines of text about the option numpts. A red box marks the sentence 'We sample roughly the same number of points for circles with different radius.'", "caption": "Section 3, p. 145 (PDF p. 8). The marked sentence describes the default number of sampling points on a circle.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E012-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The printed polygon call passes no Zdata, and the later archived copy of the package then stops with “No treatment specified”; the printed quotation marks are not valid R; and the default number of sampling points on a circle grows with the radius.", "id": "P008-E012", "location": "Section 6, the printed SpatialEffect() call, p. 153 (PDF p. 16); the toy call, p. 148 (PDF p. 11); Section 3, the sentence on numpts, p. 145 (PDF p. 8)", "pdf_page": 16, "printed_page": "153", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~153:\n\n\\texttt{dVec = seq(from=0, to=20/111, by=5/111) AMRest <- Spatial}\n\n\\texttt{Effect(ras = ras, \\hl{ras\\_Z = ras\\_Z, treatment = ’’treatment’’},}\n\n\\texttt{dVec = dVec, smooth= 1, cutoff = 10000,}\n\n\\texttt{dist.metric = \\hl{’’Geodesic’’})}\n\np.~148: \\texttt{x\\_coord\\_Z = \\hl{’’x’’}, y\\_coord\\_Z = \\hl{’’y’’}, treatment = \\hl{’’Z’’}}\n\np.~145: ``\\texttt{numpts} is the number of sampling points on a circle. If no value is submitted, the number will be decided by the raster object's resolution. \\hl{We sample roughly the same number of points for circles with different radius.}''", "corrected": "For the later archived copy, the call needs a table of treatments whose rows are in the order of the polygons:\n\n\\texttt{polygon\\_treatments <- sf::st\\_drop\\_geometry(sf::st\\_as\\_sf(ras\\_Z))}\n\n\\texttt{AMRest <- SpatialEffect(ras = ras, ras\\_Z = ras\\_Z, \\hl{Zdata = polygon\\_treatments},}\n\n\\texttt{treatment = \\hl{\"treatment\"}, dVec = dVec, smooth = 1, cutoff = 10000,}\n\n\\texttt{dist.metric = \\hl{\"Geodesic\"})}\n\nHere \\texttt{dVec} is the grid corrected in \\fid{P008-E004}, and the table has to contain the attribute named by \\texttt{treatment}. To keep the interface that Section 3 describes, the function should build this table from the attributes of the polygons before it tests the treatment input and sets \\texttt{nz} and \\texttt{Zup}; deleting the test is not enough. The behaviour of the release cited in the paper is not established.\n\np.~148: \\texttt{x\\_coord\\_Z = \\hl{\"x\"}, y\\_coord\\_Z = \\hl{\"y\"}, treatment = \\hl{\"Z\"}}\n\np.~145: ``\\hl{For point interventions, unless \\texttt{numpts} is set, the number of sampling points increases approximately in proportion to the radius of the circle, at a fixed resolution of the raster.}''", "evidence": "The evidence is a run of the code and an exact computation. I called the \\texttt{SpatialEffect()} function of the later copy, unchanged, with the printed arguments and ordinary quotation marks, on two square polygons with treatment attributes 0 and 1 and a 4-by-4 raster. It stops with ``No treatment specified''. A copy of the function with the test deleted stops with ``object 'nz' not found''. The R parser rejects each of the five printed strings (\\texttt{’’x’’}, \\texttt{’’y’’}, \\texttt{’’Z’’}, \\texttt{’’treatment’’} and \\texttt{’’Geodesic’’}) and accepts them with double quotation marks. Computed in exact arithmetic from the formula above, the counts are 7, 10, 13 and 450, and they agree with the counts used by both copies. The manual page of the package also describes the default as proportional to the radius.", "problem": "Section 3 (p.~145) says that for polygon interventions the treatment status comes from the attributes of the polygon object \\texttt{ras\\_Z}. The call on p.~153 accordingly passes \\texttt{ras\\_Z} and \\texttt{treatment}, and no \\texttt{Zdata}. The paper cites version 0.1 of the package (Samii and Wang 2020), which is not available. The archived repository of the package holds two copies of the code, an older one and a later one, and only the later one accepts polygons. Its \\texttt{SpatialEffect()} first tests whether \\texttt{Zdata} is \\texttt{NULL}, which is the default, and stops with ``No treatment specified'' before it reads \\texttt{ras\\_Z}. Deleting the test does not repair the function. The number of nodes and the treatment vector, \\texttt{nz} and \\texttt{Zup}, are set only from \\texttt{Zdata}, and without them the function stops with ``object 'nz' not found''.\n\nThere are two smaller points. First, both printed calls enclose strings in pairs of apostrophes, which R does not parse; ordinary quotation marks are meant. Second, for point interventions the default number of sampling points on a circle of radius $d$, for a raster whose smallest cell dimension is $r$, with the default $\\texttt{evalpts}=1$, is in both archived copies\n\\[\n\\Bigl\\lceil \\pi\\,\\bigl\\lceil 1+d\\sqrt{2}/r\\bigr\\rceil \\Bigr\\rceil ,\n\\]\nwhich grows in proportion to the radius. On the toy grid $d=0.1,0.2,\\dots,2$ ($r=1$), it is 7 for $0.1\\le d\\le 0.7$, 10 for $0.8\\le d\\le 1.4$ and 13 for $1.5\\le d\\le 2$, and it would be 450 at $d=100$. Whether 7 to 13 is ``roughly the same number'' is a matter of wording, but as a description of the default the sentence on p.~145 is not accurate.\n\nIt is not known whether the call ran as printed with the release used for the paper. No reported number is affected.", "what_survives": "The definitions, the estimators and every reported number are unaffected. The toy call supplies \\texttt{Zdata} and is not stopped by this test once the quotation marks are corrected; its argument \\texttt{smooth = 1} is the subject of \\fid{P008-N007}. I did not make a complete run of the corrected polygon call on geographic data."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E012", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text. The marked sentence reads: For the asymptotic variance estimator, users need to specify a distance threshold (cutoff) beyond which interference dependence does not exist. It is followed by: This can be based on substantive judgments about the likely spatial extent of interference, and can be varied as part of a sensitivity analysis. Red outlines surround the words distance threshold (cutoff) beyond which interference dependence does not exist.", "caption": "Section 3, p. 145 (PDF p. 8). The outlines mark the description of cutoff as a distance threshold.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E015.png"}, {"alt": "Crop of a printed table with ten rows and the columns dVec, AMR\\_est, Conley.CI.l, Conley.CI.u, Per.CI.l and Per.CI.u. Row 6 reads: 0.6, 1.700, -0.038, 3.438, -1.673, 1.673. Red outlines surround the column headings Conley.CI.l and Conley.CI.u and the two values -0.038 and 3.438 in row 6.", "caption": "Output of summary(), p. 149 (PDF p. 12). The outlines mark the two columns of the spatial-HAC interval and the row for $d=0.6$, where the rule of the later version would give an interval of width zero.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E015-b.png"}, {"alt": "Dot plot with the distance between two nodes on the horizontal axis, from 0 to 6.6 units of the coordinates, and one row for each of the six pairs of nodes. Three orange squares at distance 2 are the pairs of nodes 1 and 2, 2 and 3, and 3 and 4. Two blue circles at distance 4 are the pairs 1 and 3, and 2 and 4. One blue circle at distance 6 is the pair 1 and 4. A dashed vertical line at 0.4 is labelled the cutoff set by the user. A solid vertical line at 1.4 is labelled bandwidth used when the radius 1/5 is entry 1 of the grid: 0.4 + 1 = 1.4, no pair of nodes enters, the variance is 1. A solid vertical line at 2.4 is labelled bandwidth used when it is entry 2: 0.4 + 2 = 2.4, three pairs enter, the variance is 1/2. The region between 1.4 and 2.4 is shaded and contains the three orange squares. The legend says that orange squares are pairs that enter the variance only under the bandwidth 2.4 and blue circles are pairs that enter under neither bandwidth.", "caption": "The counterexample. The six pairs of nodes are 2, 4 or 6 units apart. When the radius $1/5$ is the first entry of the grid, the later version uses the bandwidth $1.4$, no pair enters and the variance is $1$. When it is the second entry, the bandwidth is $2.4$, the three pairs of neighbors enter and the variance is $1/2$. The cutoff set by the user is $0.4$ in both cases.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-E015-bandwidth.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T9", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T9"}, {"id": "P008-T10", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T10"}], "headline": "In the later archived version of the package, the bandwidth of the spatial-HAC variance of the unsmoothed estimator is cutoff plus the position of the radius in dVec, and not the distance threshold that the paper describes; the bandwidth should be cutoff itself, or a stated function of the radius in the units of the distances.", "id": "P008-E015", "location": "Section 3, p. 145 (PDF p. 8), sentence on cutoff; file R/SpatialEffect.R of the later archived version of the package, lines 199–219 (the code is not printed in the article)", "pdf_page": 8, "printed_page": "145", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P008-T10", "P008-T9"], "text": {"as_printed": "\\emph{p.~145.} For the asymptotic variance estimator, users need to specify a distance threshold (\\texttt{cutoff}) beyond which interference dependence does not exist.\n\n\\emph{Later archived version of the package, \\texttt{R/SpatialEffect.R}, lines 199--201 and 215--219.}\n\n\\texttt{c <- cutoff}\n\n\\texttt{for (d in 1:length(dVec))\\{}\n\n\\texttt{\\ dUp <- dVec[d]}\n\n\\ldots\n\n\\texttt{\\ if (cutoff > 0)\\{}\n\n\\texttt{\\ \\ c <- cutoff\\hl{ + d}}\n\n\\texttt{\\ \\}}\n\n\\texttt{\\ Conley\\_result <- ConleySE(as.vector(res\\_d), as.matrix(W\\_meat), as.matrix(dist),}\n\n\\texttt{\\ \\ \\ as.double(c), as.integer(k))}", "corrected": "The sentence on p.~145 needs no change. If \\texttt{cutoff} is the final node-pair dependence threshold, I delete lines 215--217 in the later version of the package, so that the bandwidth passed to the variance routine is that threshold:\n\n\\texttt{c <- cutoff}\n\n\\texttt{for (d in 1:length(dVec))\\{}\n\n\\texttt{\\ dUp <- dVec[d]}\n\n\\ldots\n\n\\del{\\texttt{\\ if (cutoff > 0)\\{ c <- cutoff + d \\}}}\n\n\\texttt{\\ Conley\\_result <- ConleySE(as.vector(res\\_d), as.matrix(W\\_meat), as.matrix(dist),}\n\n\\texttt{\\ \\ \\ as.double(c), as.integer(k))}\n\nWith this change the variance in the counterexample is $1$ on both grids. If \\texttt{cutoff} is instead meant as a range of interference that has to be enlarged for circle averages of radius $d$, then the bandwidth must be a stated function of the radius \\texttt{dUp}, in the units of the distance matrix, and never of the index \\texttt{d}. Which enlargement is right is not established here. Replacing \\texttt{d} by \\texttt{dUp} removes the dependence on the grid, but it does not by itself establish that the resulting intervals are valid.", "evidence": "I checked an exact counterexample in rational arithmetic and the output printed on p.~149. The unchanged R function in the later version, supplied with the counterexample's circle averages and distances and an R implementation of the uniform-kernel variance formula, returns the same bandwidths and variances.\n\n\\emph{Counterexample.} Let four nodes lie at $(0,0)$, $(2,0)$, $(4,0)$ and $(6,0)$, with treatments $Z=(0,0,1,1)$, and let the outcome equal $0$, $2$, $4$ and $6$ on the discs of radius $3/10$ around them. The circle averages are $\\mu=(0,2,4,6)$ at each of the radii $1/10$, $1/5$ and $3/10$. I take Euclidean distance, the uniform kernel, \\texttt{edf = FALSE} and \\texttt{cutoff} $=2/5$. The regression of $\\mu$ on an intercept and $Z$ has coefficients $(1,4)$ and residuals $r=(-1,1,-1,1)$, and the estimate is $4$. The weights of the treatment coefficient are $a=(-1/2,-1/2,1/2,1/2)$. With the scores $g_i=a_ir_i$, that is, $g=(1/2,-1/2,-1/2,1/2)$, the variance with bandwidth $c$ is\n\\[\nV(c)=\\sum_{i,j}g_ig_j\\,\\mathbf{1}\\{D_{ij}\\le c\\},\n\\]\nwhere $D_{ij}$ is the distance between nodes $i$ and $j$. If $c<2$, only the terms with $i=j$ remain, and $V(c)=\\sum_i g_i^2=1$. If $2\\le c<4$, the three pairs of neighbors enter as well, and $V(c)=1+2(g_1g_2+g_2g_3+g_3g_4)=1+2(-1/4+1/4-1/4)=1/2$.\n\\begin{audittable}{lccccc}{The same radius $1/5$ on two grids of radii, under the rule of the later version and under a bandwidth equal to \\texttt{cutoff}.}\nGrid of radii & Place of $1/5$ & Bandwidth & Estimate & Variance & Standard error \\\\\n$(1/5,\\ 3/10)$ & 1 & $2/5+1=7/5$ & $4$ & $1$ & $1$ \\\\\n$(1/10,\\ 1/5,\\ 3/10)$ & 2 & $2/5+2=12/5$ & $4$ & $1/2$ & $0.7071$ \\\\\nEither grid, bandwidth \\texttt{cutoff} & 1 or 2 & $2/5$ & $4$ & $1$ & $1$ \\\\\n\\end{audittable}\n\n\\emph{The printed output.} Since the four nodes of the toy example lie in the square $[0,4]^2$ (Figure 2), two of them are at most $\\sqrt{32}$ apart. The printed call has \\texttt{cutoff = .4} and radii $0.1,0.2,\\ldots,2$, and it leaves the kernel at its default, the uniform kernel. At the sixth radius the rule of the later version gives the bandwidth $0.4+6=6.4>\\sqrt{32}$, and the kernel includes every pair of nodes. Then the variance is $(\\sum_i g_i)^2$, which is zero for any data, since the residuals of a regression on an intercept and the treatment sum to zero in each arm. The interval at $d=0.6$ would have width zero. The printed interval is $[-0.038,\\,3.438]$, of width $3.476$, and I conclude that this rule did not produce the printed table.", "problem": "The paper describes \\texttt{cutoff} as a distance: pairs of nodes farther apart than \\texttt{cutoff} contribute no covariance terms to the variance estimate. The later version of the package is the one distributed with the replication files of Wang, Samii, Chang and Aronow (2024). In its loop over the radii, the spatial-HAC standard error of the unsmoothed estimator is computed with the bandwidth \\texttt{cutoff + d} whenever \\texttt{cutoff} is positive. Here \\texttt{d} is the index $1,2,3,\\ldots$ of the loop, and the radius itself is \\texttt{dUp}. The bandwidth for a radius therefore depends on the place of that radius in the grid of radii \\texttt{dVec}, rather than on the radius itself.\n\nTwo consequences follow. The standard error of the estimate at a given radius can change when another radius is inserted before it in \\texttt{dVec}, although the estimate and the data are the same. Further, the bandwidth grows by one unit of the distance matrix with each entry of the grid, whatever the unit is. In the units of the toy example it exceeds every distance between two nodes at the sixth radius, $0.6$, and the variance there is exactly zero under the default uniform kernel and \\texttt{edf = FALSE}. When spatial-HAC intervals are requested with a positive \\texttt{cutoff}, the function passes this index-based bandwidth without warning.", "what_survives": "The sentence on p.~145 is correct as a description of the method. The definition of the AMR, the unsmoothed point estimates and permutation tests do not use this line, and the line is skipped when \\texttt{cutoff = 0}. I attribute no published interval to the defect: the release that the paper cites is not archived, the earlier archived version of the package passes \\texttt{cutoff} unchanged, and the printed toy intervals are incompatible with the rule. The later version's requested spatial-HAC standard errors and intervals of the unsmoothed estimator depend on this bandwidth. With \\texttt{edf = TRUE}, so do those of the smoothed estimator and its confidence bands, which reuse the degrees of freedom computed in the same loop."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-E015", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of text. The marked sentence reads: Another option that the SpatialEffect package offers is to define the circle average as points that are within a k-coordinate-wide bandwidth around points that are d coordinate units from the intervention node; this latter specification defines a disc or circular band (or donut) of intervention nodes that are between d minus k and d plus k coordinate units from the intervention node. Red outlines surround the words k-coordinate-wide bandwidth and the words between d minus k and d plus k.", "caption": "Section 2.2, p. 143 (PDF p. 6). The outlines mark the stated width of the band and its endpoints, which are $2k$ apart.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N001.png"}, {"alt": "Crop of five lines of text. The marked sentence reads: Users can choose a set of points that form the edge of a circle of radius d, that form a circular band (donut) centered at radius d, or that form a sector centered at d (disk), by altering the option cType. Red outlines surround the words centered at radius d and the words sector centered at d (disk).", "caption": "Section 3, p. 145 (PDF p. 8), sentence on the option cType. The outlines mark the descriptions of the donut and of the disk.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T16", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T16"}, {"id": "P008-T17", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T17"}], "headline": "For $d\\ge k$, a band between $d-k$ and $d+k$ has width $2k$, not $k$, and the “disk” is the full disc of radius $d$, not a sector; in the later archived version of the package the “donut” for polygon interventions is the band between consecutive radii of dVec and is not centered at $d$.", "id": "P008-N001", "location": "Section 2.2, p. 143 (PDF p. 6), sentence beginning “Another option that the SpatialEffect package offers”; Section 3, p. 145 (PDF p. 8), sentence on the option cType", "pdf_page": 6, "printed_page": "143", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T16", "P008-T17"], "text": {"as_printed": "\\emph{p.~143.} Another option that the \\texttt{SpatialEffect} package offers is to define the circle average as points that are within a \\hl{$k$-coordinate-wide} bandwidth around points that are $d$ coordinate units from the intervention node; this latter specification defines a disc or circular band (or ``donut'') of intervention nodes that are between $d-k$ and $d+k$ coordinate units from the intervention node.\n\n\\emph{p.~145.} Users can choose a set of points that form the edge of a circle of radius $d$, that form a circular band (``donut'') \\hl{centered at radius $d$}, or that form a \\hl{sector centered at $d$} (``disk''), by altering the option \\texttt{cType}.", "corrected": "\\emph{p.~143.} Another option that the \\texttt{SpatialEffect} package offers is to define the circle average as points that are within \\hl{distance $k$ of the circle of radius $d$ around the intervention node}; this latter specification defines a disc or circular band (or ``donut'') of outcome points that are between $\\hl{\\max(0,d-k)}$ and $d+k$ coordinate units from the intervention node\\hl{, so that $k$ is the half-width of the band when $d\\ge k$}.\n\n\\emph{p.~145.} Users can choose a set of points that form the edge of a circle of radius $d$, that form a circular band (``donut'') centered at radius $d$, or that form \\hl{the full disc of radius $d$ around the intervention node} (``disk''), by altering the option \\texttt{cType}. \\hl{In the later archived version of the package, the donut for polygon interventions is not centered at $d$: at the $j$th radius $d_j$ of \\texttt{dVec}, $j>1$, it averages the outcome cells between the buffers of radii $d_{j-1}$ and $d_j$ around the polygon, and at the first radius it averages the full buffer.}\n\nThe words ``of outcome points'' on p.~143 follow \\fid{P008-E007}.", "evidence": "I derived the width of the band and ran an exact example through the function \\texttt{SpatialEffect()} of the later version.\n\n\\emph{Width.} Let $r$ be the distance of a point $x$ from the node. By the reverse triangle inequality, every point of the circle of radius $d$ is at distance at least $|r-d|$ from $x$, and a radially aligned point of the circle attains this bound. This implies that the points within distance $k$ of the circle are the points such that $\\max(0,d-k)\\le r\\le d+k$. For $d=4$ and $k=1$ the band is $[3,5]$, of width $2$.\n\n\\emph{Example.} Let the treated polygon be a small square, let \\texttt{dVec} be $(2,4)$, and let the only outcome cells near the square lie at distances $0$, $19/8$, $27/8$ and $35/8$ from its boundary, with outcomes $4$, $2$, $0$ and $8$. Let all outcomes near the control polygon be $0$. The buffer of radius $2$ contains one cell, with sum $4$, and the buffer of radius $4$ contains three cells, with sum $6$.\n\\begin{audittable}{lcc}{The example at the radius $4$: the average of the outcome cells of the treated polygon over three regions, with the cells listed by their distance from the boundary of the polygon. The control averages are $0$, so each average is also the estimate of the AMR.}\nRegion & Cells, by distance & Average \\\\\nBetween the buffers of radii $2$ and $4$ (``donut'') & $19/8$, $27/8$ & $(6-4)/(3-1)=1$ \\\\\nBand $[3,5]$, of the same width, centered at $4$ & $27/8$, $35/8$ & $(0+8)/2=4$ \\\\\nFull buffer of radius $4$ (``disk'') & $0$, $19/8$, $27/8$ & $6/3=2$ \\\\\n\\end{audittable}\nThe function returns $1$ for the donut and $2$ for the disk at the radius $4$, in agreement with the first and third rows, and not the value $4$ of the centered band.", "problem": "The two sentences describe the band and the disc inexactly, and the description differs from what the available code does.\n\nFirst, the points within distance $k$ of the circle of radius $d$ are the points between $\\max(0,d-k)$ and $d+k$ from the node. The endpoints on p.~143 are correct when $d\\ge k$, but this band is $2k$ wide. Then $k$ is its half-width and not its width.\n\nSecond, $d$ is a radius and not a place, and a sector would need an angle, which neither the text nor the package specifies. The ``disk'' is the full disc of radius $d$ around the node. For a polygon it is the polygon together with all points within distance $d$ of it.\n\nThird, the only archived version of the package that has the option \\texttt{cType} is the one distributed with the replication files of Wang, Samii, Chang and Aronow (2024), which is later than the release that the paper cites (version 0.1, 2020). This later version has no argument $k$. For polygon interventions, its ``donut'' at the $j$th radius $d_j$ of \\texttt{dVec}, $j>1$, averages the outcome cells that lie between the buffers of radii $d_{j-1}$ and $d_j$ around the polygon, and at the first radius it averages the full buffer. Such a band ends at $d_j$ and is not centered there. Someone who follows the text would place a donut estimate half a band away from the region it averages. The release that the paper cites is not archived, and I could not determine whether it behaved in the same way.", "what_survives": "The corrections of wording change no reported number. The definition of the AMR on exact circles, the estimator and the toy example do not use these options. The bottom panel of Figure 8 uses the donut option, and where its points are drawn is the subject of \\fid{P008-N017}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N001", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of text. The marked sentence reads: For the asymptotic variance estimator, users need to specify a distance threshold (cutoff) beyond which interference dependence does not exist. It is followed by: This can be based on substantive judgments about the likely spatial extent of interference, and can be varied as part of a sensitivity analysis. Red outlines surround the words distance threshold (cutoff) beyond which interference dependence does not exist.", "caption": "Section 3, p. 145 (PDF p. 8). The outlines mark the rule for the choice of cutoff.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N002.png"}, {"alt": "Crop of three lines of text and four lines of R code. The text reads: point for the spatial HAC variance is set to be 0.4 and the number of permutations is set to be 1000. We use the SpatialEffect() function to estimate the AMR as follows. The code reads: dVec = seq(from=.1,to=2, by=.1); AMRest <- SpatialEffect(ras = ras\\_sim, Zdata = Zdata, x\\_coord\\_Z = ''x'', y\\_coord\\_Z = ''y'', treatment = ''Z'', dVec = dVec, cutoff = .4, nPerms = 1000, smooth = 1). Red outlines surround the words point for the spatial HAC variance is set to be 0.4 and the argument cutoff = .4.", "caption": "Section 5, p. 148 (PDF p. 11), end of the sentence that begins on p. 147, and the call below it. The outlines mark the cutoff 0.4 in the text and in the call.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N002-b.png"}, {"alt": "Crop of a printed table with ten rows and the columns dVec, AMR\\_est, Conley.CI.l, Conley.CI.u, Per.CI.l and Per.CI.u, followed by three lines of text. The Conley columns read, for d from 0.1 to 1.0: -0.277 and 3.259; -0.242 and 3.343; -0.242 and 3.343; -0.050 and 3.431; -0.020 and 3.448; -0.038 and 3.438; -0.066 and 3.422; -0.067 and 2.071; -2.424 and 0.791; -2.295 and 0.714. The text below reads: The results are also presented in Fig. 5. The black solid line in the graph is the AMR estimate for each of the distance values d. The red dotted line is the 95\\% confidence interval based on the spatial-HAC standard errors. Red outlines surround the column headings Conley.CI.l and Conley.CI.u, the two columns of values, and the sentence about the red dotted line.", "caption": "Output of summary(), p. 149 (PDF p. 12), and the sentence below it. The outlines mark the two columns of the spatial-HAC interval, which equal the HC0 interval in every row, and the description of the red line of Figure 5.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N002-c.png"}, {"alt": "Figure with two panels. The left panel is a dot plot of the distance between two nodes, in coordinate units from 0 to 4, with one row for each pair: nodes 3 and 4 at 1.43, nodes 1 and 3 at 1.48, nodes 1 and 4 at 2.83, nodes 1 and 2 at 3.32, nodes 2 and 3 at 3.39, and nodes 2 and 4 at 3.40. A vertical line at 0.4 is labelled cutoff 0.4, and the region to its left is shaded and contains no dot. The right panel plots the variance of the linearized estimator, in squared units of the outcome, against the radius d of the circle average from 0.1 to 2.0. A blue line with circles, variances and covariances of the nodes, starts at about 1.49 at d = 0.1, falls to about 0.39 at d = 1.2 and ends at about 0.52 at d = 2.0. An orange line with squares, variances only (all that the cutoff keeps), lies below it at every radius, from about 1.27 at d = 0.1 to about 0.35 at d = 2.0. The band between the two lines is shaded.", "caption": "The toy example. Left: the six distances between the four nodes, all above the cutoff 0.4, so that no pair enters the variance. Right: the variance of the linearized estimator under the Bernoulli design, with the covariances between nodes and without them, at the 20 radii; the part left out is positive throughout. The curves are variances of the linearization, not finite-sample variances, and they say nothing about coverage.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N002-cutoff.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T5", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T5"}, {"id": "P008-T6", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T6"}, {"id": "P008-T10", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T10"}, {"id": "P008-T23", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T23"}], "headline": "The intervals printed as spatial-HAC intervals in the toy example contain no covariance between nodes and equal the heteroskedasticity-robust (HC0) intervals, because the cutoff 0.4 is smaller than every distance between the four nodes; they do not follow the rule of Section 3 and do not adjust for interference between nodes.", "id": "P008-N002", "location": "Section 3, p. 145 (PDF p. 8), the rule for cutoff; Section 5, pp. 147–148 (PDF pp. 10–11), the cutoff of the toy example; p. 149 (PDF p. 12), the columns Conley.CI.l and Conley.CI.u and the description of the red line of Figure 5", "pdf_page": 8, "printed_page": "145", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T10", "P008-T23", "P008-T5", "P008-T6"], "text": {"as_printed": "\\emph{p.~145.} For the asymptotic variance estimator, users need to specify a distance threshold (\\texttt{cutoff}) beyond which interference dependence does not exist. This can be based on substantive judgments about the likely spatial extent of interference, and can be varied as part of a sensitivity analysis.\n\n\\emph{pp.~147--148.} The cutoff point for the spatial HAC variance is set to be \\hl{0.4} and the number of permutations is set to be 1000.\n\n\\emph{p.~149.} The red dotted line is the 95\\% confidence interval based on the \\hl{spatial-HAC} standard errors.", "corrected": "\\emph{p.~145.} For the asymptotic variance estimator, users need to specify a distance threshold (\\texttt{cutoff}) beyond which interference dependence does not exist \\hl{between the contributions of two intervention nodes to the estimate at the radius being analyzed; this distance can exceed the distance over which a node affects outcomes}.\n\n\\emph{pp.~147--148.} The cutoff point for the spatial HAC variance is set to be 0.4 and the number of permutations is set to be 1000. \\hl{With cutoff 0.4, no distinct pair of intervention nodes enters the variance estimator. The limits displayed as Conley.CI.l and Conley.CI.u equal HC0 limits and do not adjust for interference between nodes; they illustrate nominal intervals and not calibrated inference in this four-node example.}\n\n\\emph{p.~149.} The red dotted line is the 95\\% confidence interval based on the \\hl{HC0 standard errors, to which the spatial-HAC standard errors reduce at this cutoff}.\n\nNo other value of the cutoff is proposed for the example. With four nodes, a uniform kernel that includes every pair gives a variance estimate of zero, and a calibrated interval for this example would need a separately justified procedure.", "evidence": "The evidence is a recalculation of the toy example from the vignette distributed with the package, and an exact counterexample for the general point.\n\n\\emph{Recalculation.} Starting from the seed and the recipe of the vignette, I obtain the ten printed estimates and the twenty printed endpoints of the intervals, to three decimals. The smallest distance between two nodes is $1.4295>0.4$. With the uniform kernel, the spatial-HAC variance of the estimate is $\\sum_{i,j}q_iq_je_ie_j\\mathbf{1}\\{D_{ij}\\le 0.4\\}$, where $q=(1/2,1/2,-1/2,-1/2)$, $e_i$ is the residual of the circle average of node $i$ from the mean of its arm, and $D_{ij}$ is the distance between nodes $i$ and $j$. Only the terms with $i=j$ remain, and the variance is the HC0 variance $\\sum_i q_i^2e_i^2$.\n\\begin{audittable}{rrrrr}{The output printed on p.~149 beside the HC0 interval recomputed from the vignette, that is, the estimate plus and minus the $0.975$ normal quantile times the HC0 standard error.}\n$d$ & AMR\\_est, printed & Conley.CI, printed & HC0 standard error & HC0 interval \\\\\n$0.1$ & $1.491$ & $[-0.277,\\ 3.259]$ & $0.9020$ & $[-0.2772,\\ 3.2586]$ \\\\\n$0.2$ & $1.550$ & $[-0.242,\\ 3.343]$ & $0.9145$ & $[-0.2420,\\ 3.3429]$ \\\\\n$0.3$ & $1.550$ & $[-0.242,\\ 3.343]$ & $0.9145$ & $[-0.2420,\\ 3.3429]$ \\\\\n$0.4$ & $1.691$ & $[-0.050,\\ 3.431]$ & $0.8880$ & $[-0.0497,\\ 3.4313]$ \\\\\n$0.5$ & $1.714$ & $[-0.020,\\ 3.448]$ & $0.8846$ & $[-0.0197,\\ 3.4480]$ \\\\\n$0.6$ & $1.700$ & $[-0.038,\\ 3.438]$ & $0.8867$ & $[-0.0382,\\ 3.4376]$ \\\\\n$0.7$ & $1.678$ & $[-0.066,\\ 3.422]$ & $0.8900$ & $[-0.0664,\\ 3.4224]$ \\\\\n$0.8$ & $1.002$ & $[-0.067,\\ 2.071]$ & $0.5454$ & $[-0.0675,\\ 2.0705]$ \\\\\n$0.9$ & $-0.816$ & $[-2.424,\\ 0.791]$ & $0.8202$ & $[-2.4241,\\ 0.7912]$ \\\\\n$1.0$ & $-0.791$ & $[-2.295,\\ 0.714]$ & $0.7676$ & $[-2.2955,\\ 0.7136]$ \\\\\n\\end{audittable}\n\n\\emph{The covariances left out.} Since the effects are additive in the toy example, the circle average of node $i$ is $\\mu_i=b_i+\\sum_k a_{ik}Z_k$, where $a_{ik}$ is the average effect of node $k$ on the circle of node $i$. Let $m_1$ and $m_0$ be the averages over the nodes of the marginal circle means under treatment and under control, and let $\\phi_i=2Z_i(\\mu_i-m_1)-2(1-Z_i)(\\mu_i-m_0)$ be the contribution of node $i$ to the linearized Hajek estimator. Under independent Bernoulli($1/2$) assignment, a direct computation gives\n\\[\n\\operatorname{Cov}(\\phi_i,\\phi_j)=a_{ij}a_{ji}\\qquad(i\\ne j).\n\\]\nAt $d=0.1$ the twelve coefficients $a_{ij}$ with $i\\ne j$ are negative, which implies that the six covariances are positive. For nodes 3 and 4, which are $1.43$ apart, I have $a_{34}=-0.9235$ and $a_{43}=-1.1507$, and the covariance is $1.0627$. The variance of the mean of the four $\\phi_i$ at $d=0.1$ is $1.4946$ with the covariances and $1.2680$ without them. The part left out is positive at each of the 20 radii, between $0.013$ and $0.227$. Note that these are variances of the linearization under the design, and not the finite-sample variance of the estimator with four nodes. I make no claim about the coverage of the printed intervals.\n\n\\emph{Counterexample for the general point.} A cutoff equal to the range over which a node affects outcomes can be too small. Let two nodes lie at $(0,0)$ and $(2,0)$, let the outcome points be $(-1,0)$, $(1,0)$ and $(3,0)$, and let a treated node raise by $1$ the outcome of every point within distance $1$ of it. Under independent Bernoulli($1/2$) assignment and for $d=1$, the circle averages are $\\mu_1=Z_1+Z_2/2$ and $\\mu_2=Z_2+Z_1/2$, and the AMR is $1$. With $S_k=2Z_k-1$, I obtain $\\phi_1=\\phi_2=S_1S_2/2$, so that $\\operatorname{Var}(\\phi_i)=\\operatorname{Cov}(\\phi_1,\\phi_2)=1/4$. The variance of $(\\phi_1+\\phi_2)/2$ is $1/4$, and the terms with $i=j$ contribute $1/8$. A cutoff of $1$ excludes the pair, which is $2$ apart, and leaves out half of the variance. The same ratio holds for any number of such pairs placed far from one another.", "problem": "Section 3 asks for a cutoff beyond which interference dependence does not exist. In the toy example the four nodes are between $1.43$ and $3.40$ coordinate units apart, and the simulated effect of a node does not vanish over these distances: at distance $2$ it is $-0.58$. The cutoff 0.4 is below the smallest distance between two nodes, so no pair of distinct nodes enters the variance estimator. The estimator then reduces to the heteroskedasticity-robust (HC0) variance of a difference in means, and all ten printed intervals equal the HC0 intervals to the three decimals shown.\n\nThe example presents these intervals as spatial-HAC intervals, in a tutorial on inference under interference, but they contain no adjustment for dependence between nodes. That dependence is present in the reconstruction below. Under the Bernoulli design of the example, the contributions of two distinct nodes to the estimator have positive covariance for every pair of nodes at $d=0.1$, and the sum of these covariances is positive at each of the 20 radii. The rule on p.~145 is correct as written, and the defect lies in the example. A user who chooses \\texttt{cutoff} as the example does leaves these covariances out.", "what_survives": "The rule on p.~145 is correct, and this cutoff correction leaves the estimates, the permutation bounds and the printed numbers unchanged. The toy intervals are nominal HC0 intervals, and they can no longer be read as adjusted for interference. I have not established an error in the application's intervals; I could not obtain the package release and the exact transformed inputs used for them. Nor have I established an error in the asymptotic result that the paper attributes to Wang et al.\\ (2022). The application bases its statement of significance on a permutation test, which does not use the cutoff."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N002", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Figure 2 with its two-line caption. The left panel is a grey 4-by-4 raster titled 'Null raster' with four numbered black nodes. The right panel is a curve that starts near 1.6 at distance 0.1, falls to about minus 0.55 near distance 1.2 and rises to about minus 0.2 at distance 2. Red boxes mark the vertical axis label 'Effect on Yx' and the caption sentence 'The plot on the right presents the non-monotonic effect function.'", "caption": "Figure 2 and its caption, p. 147 (PDF p. 10). The marked axis label and the marked caption sentence present the curve as the effect on the outcome; its values are one third of the effect used in the toy example.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N004.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T18", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T18"}], "headline": "The right panel of Figure 2 plots a kernel $f(d)$, and in the toy example a treated node changes the outcome at distance $d$ by $3f(d)$, three times the plotted curve.", "id": "P008-N004", "location": "Figure 2, right panel and last sentence of the caption, p. 147 (PDF p. 10)", "pdf_page": 10, "printed_page": "147", "reaches": [{"effect": "cannot_determine", "effect_text": "could not be determined", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "same_error_restated", "effect_text": "a corresponding statement is printed there", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T18"], "text": {"as_printed": "``Figure 2. The plot on the left displays the 4-by-4 raster that records the outcome when treatment is absent ($Y_x(0)$). Black points are the intervention nodes. The plot on the right presents the non-monotonic \\hl{effect function}.''\n\nRight panel: the vertical axis is labelled ``\\hl{Effect on Yx}'' and the horizontal axis ``Distance from intervention node''. The curve falls from about $1.63$ at distance $0.1$ to about $-0.55$ near distance $1.2$, and rises to about $-0.19$ at distance $2$.", "corrected": "``The plot on the right presents the non-monotonic \\hl{kernel $f(d)$; in the toy example, each treated node contributes $3f(d)$ to the outcome in a raster cell whose centroid is at distance $d$ from that node}.''\n\nThe vertical axis of the right panel is labelled ``\\hl{Kernel $f(d)$}'', where\n\\[\nf(d)=2e^{-2d}-\\frac{128}{3}\\,d^{4}e^{-4d}.\n\\]\nAlternatively, the panel keeps the label ``Effect on Yx'' and plots $3f(d)$, with the values on the vertical axis multiplied by three.", "evidence": "I checked the two figures in the PDF, recalculated the toy example and derived the bound below.\n\nSixteen points of the curve of Figure 2, read from the drawing in the PDF, agree with $f$ to within $0.00004$. At $d=0.1$ the plotted value is $1.6346$, while $f(0.1)=1.6346$ and $3f(0.1)=4.9038$.\n\nI generated the toy data as in the package vignette, with seed 123567 and the outcome in raster cell $c$ equal to $Y_{0c}+a\\sum_{j}z_jf(r_{jc})$ with $a=3$, where $r_{jc}$ is the distance from the centre of the cell to node $j$. These data reproduce the ten AMR estimates of the printed table and its twenty permutation bounds to the third decimal. Their true AMR under the sixteen Bernoulli assignments agrees with the twenty plotted points of Figure 4 to within $0.0001$. At $d=1.4$, Figure 4 shows $-1.2578$; the true AMR is $-1.2579$ with $a=3$ and $-0.4193$ with $a=1$. With $a=1$ the first AMR estimate would be $0.655$, and the table prints $1.491$.\n\nAveraging does not account for the factor. For outcomes of the form above, the difference between the outcomes with node $i$ treated and with node $i$ under control is $af(r_{ic})$, whatever the assignment of the other nodes. Then the AMR at distance $d$ is $\\sum_{i,c}w_{ic}(d)\\,af(r_{ic})/4$, where the weights $w_{ic}(d)\\ge 0$ sum to one over $c$ for each node $i$. Let $h(r)=(128/3)r^{4}e^{-4r}$. Since $h'(r)=(512/3)\\,r^{3}(1-r)e^{-4r}$, the maximum of $h$ on $r\\ge 0$ is $h(1)=128/(3e^{4})$, and $e^{4}>\\sum_{k=0}^{6}4^{k}/k!=437/9$ implies that $h(1)<384/437$. Since $f(r)=2e^{-2r}-h(r)$ and $2e^{-2r}>0$, I obtain\n\\[\nf(r)>-\\frac{384}{437}>-1\\qquad\\text{for all } r\\ge 0 .\n\\]\nWith $a=1$ the AMR would be above $-384/437$ at every distance, and Figure 4 reaches $-1.26$.\n\nThe script that drew the figures of the paper is not available. The factor $3$ is identified by the agreement of the recalculation with the published table and with Figure 4.", "problem": "Section 4 describes the function as ``a non-monotonic effect emanating from each intervention node, which is presented on the right plot of Fig.~2'', and the axis of the panel reads ``Effect on Yx''. The panel is read as the change in the outcome that one treated node produces at a given distance. Its scale is too small by a factor of three. The plotted curve agrees with the kernel\n\\[\nf(d)=2e^{-2d}-\\frac{128}{3}\\,d^{4}e^{-4d},\n\\]\nthe difference of two gamma densities. The true AMR of Figure 4 and the output table on p.~149 agree with an effect of $3f(d)$ from each treated node. The paper does not state the multiplier.\n\nThe mismatch shows in a comparison of two figures. The true AMR in Figure 4 falls to about $-1.26$, below the minimum of the plotted function, about $-0.55$. When the effects of the nodes add, as they do in the toy data of the package vignette, the AMR is a weighted average of values of the single-node effect and cannot lie below its minimum.", "what_survives": "The shape of the curve and the purpose of the illustration are unaffected. No estimate changes: Figures 3 to 5, the printed estimates and intervals, the definitions of Section 2 and the application of Section 6 do not depend on the scale of Figure 2. Only the correspondence between the magnitudes in Figure 2 and those in Figures 3 and 4 changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N004", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the four panels Z13 to Z16 of Figure 3 and the two-line caption. Each panel is a grey 4-by-4 raster with four numbered round nodes, some black and some white. A red box surrounds panel Z16, in which all four nodes are white. Red boxes also mark the caption words 'Black' and 'nodes are treated and white ones are under control.'", "caption": "Figure 3, bottom row of panels and caption, p. 148 (PDF p. 11). The marked sentence says that black nodes are treated. In the marked panel Z16 all four nodes are white, and its raster is not the untreated raster of Figure 2.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N005.png"}, {"alt": "Crop of the four panels Z1 to Z4 of Figure 3. A red box surrounds panel Z1, in which all four numbered nodes are black. In each of the panels Z2, Z3 and Z4 one node is white and three are black.", "caption": "Figure 3, top row of panels, p. 148 (PDF p. 11). In the marked panel Z1 all four nodes are black, and its raster is the untreated raster shown in the left panel of Figure 2.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N005-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In Figure 3 the white nodes are treated and the black nodes are under control; the caption exchanges the two colours.", "id": "P008-N005", "location": "Figure 3 caption, second sentence, p. 148 (PDF p. 11)", "pdf_page": 11, "printed_page": "148", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Figure 3. The plots show the outcome's values in the raster under each of the sixteen possible assignments. \\hl{Black} nodes are treated and \\hl{white} ones are under control.''", "corrected": "``Figure 3. The plots show the outcome's values in the raster under each of the sixteen possible assignments. \\hl{White} nodes are treated and \\hl{black} ones are under control.''", "evidence": "I checked the page and measured grey levels (0 for black, 255 for white) at the centres of the sixteen cells of each panel in a rendering of the PDF. Panel Z1 differs from the untreated raster of Figure 2 by at most 2 grey levels. Panel Z16 differs from it by more than 5 grey levels in every cell and by as much as 105. With white read as 1, the marker patterns of the sixteen panels are the sixteen binary vectors, from $(0,0,0,0)$ in Z1 to $(1,1,1,1)$ in Z16. The package vignette draws each marker with a grey level equal to the treatment indicator, so that a treated node ($Z_i=1$) is white.", "problem": "The second sentence of the caption is the key to the node markers in the sixteen panels, and it is reversed. In panel Z1 all four markers are black, and the panel shows the same raster as the left panel of Figure 2, which records the outcome ``when treatment is absent''. Black markers are nodes under control. Panel Z16, with four white markers, shows a different raster. With the printed key, every panel is read with treatment and control exchanged, and a contrast computed by hand from the panels, as in the recipe on p.~147, changes sign.", "what_survives": "The coding $Z_i=1$ for a treated node, the rasters shown in Figure 3, the estimands and the estimators are unaffected. No number in the paper depends on the caption."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N005", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the printed output table with ten rows and the columns dVec, AMR_est, Conley.CI.l, Conley.CI.u, Per.CI.l and Per.CI.u. One red box marks the two column names Conley.CI.l and Conley.CI.u. Two more red boxes mark the ten values under each, from minus 0.277 to minus 2.295 and from 3.259 to 0.714.", "caption": "Output table, p. 149 (PDF p. 12). The marked columns are the lower and upper endpoints of the spatial-HAC intervals, which the older archived copy of the package does not return.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "The older archived copy of the SpatialEffect package does not reproduce the twenty endpoints of the spatial-HAC intervals in the toy table; the printed endpoints are matched by a regression of the four circle averages on an intercept and the treatment indicator, with the spatial-HAC variance computed across nodes.", "id": "P008-N006", "location": "Output table, columns Conley.CI.l and Conley.CI.u, p. 149 (PDF p. 12); red lines of Figure 5, p. 150 (PDF p. 13)", "pdf_page": 12, "printed_page": "149", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{rrrrrr}{Output table of the toy example, p.~149.}\n\\texttt{dVec} & \\texttt{AMR\\_est} & \\texttt{\\hl{Conley.CI.l}} & \\texttt{\\hl{Conley.CI.u}} & \\texttt{Per.CI.l} & \\texttt{Per.CI.u} \\\\\n$0.1$ & $1.491$ & $-0.277$ & $3.259$ & $-1.566$ & $1.566$ \\\\\n$0.2$ & $1.550$ & $-0.242$ & $3.343$ & $-1.626$ & $1.626$ \\\\\n$0.3$ & $1.550$ & $-0.242$ & $3.343$ & $-1.626$ & $1.626$ \\\\\n$0.4$ & $1.691$ & $-0.050$ & $3.431$ & $-1.666$ & $1.666$ \\\\\n$0.5$ & $1.714$ & $-0.020$ & $3.448$ & $-1.684$ & $1.684$ \\\\\n$0.6$ & $1.700$ & $-0.038$ & $3.438$ & $-1.673$ & $1.673$ \\\\\n$0.7$ & $1.678$ & $-0.066$ & $3.422$ & $-1.657$ & $1.657$ \\\\\n$0.8$ & $1.002$ & $-0.067$ & $2.071$ & $-1.005$ & $1.005$ \\\\\n$0.9$ & $-0.816$ & $-2.424$ & $0.791$ & $-1.328$ & $1.328$ \\\\\n$1.0$ & $-0.791$ & $-2.295$ & $0.714$ & $-1.307$ & $1.307$ \\\\\n\\end{audittable}", "corrected": "The printed table needs no change. A statement of the calculation that reproduces its spatial-HAC columns is as follows.\n\nFor each distance $d$, regress the four circle averages $\\mu_i(d)$ on an intercept and $Z_i$. Let $X$ be the $4\\times 2$ matrix with rows $(1,Z_i)$, let $e$ be the vector of residuals, and let $K_{ij}=\\mathbf{1}\\{\\lVert x(i)-x(j)\\rVert<\\texttt{cutoff}\\}$ for the node coordinates $x(i)$, with $\\texttt{cutoff}=0.4$. The covariance is\n\\[\n(X^{\\top}X)^{-1}X^{\\top}\\{K\\circ(ee^{\\top})\\}X(X^{\\top}X)^{-1},\n\\]\nwith no degrees-of-freedom correction. Let $V(d)$ be its diagonal entry for $Z_i$. The interval is $\\widehat{\\tau}(d)\\pm\\Phi^{-1}(0.975)\\sqrt{V(d)}$, where $\\widehat{\\tau}(d)$ is the AMR estimate. In the older archived copy, \\hl{this calculation replaces the covariance computed from the sampled raster outcomes}, and the argument \\texttt{smooth} is omitted.", "evidence": "The evidence is a recalculation from the code. I generated the toy data as in the package vignette (seed 123567), with observed assignment $Z=(1,1,0,0)$, and ran the older copy without the \\texttt{smooth} argument and with the name that the vignette gives to the treatment column.\n\nThe printed intervals are recovered as follows. For a distance $d$, let $\\mu_i$ be the circle average of node $i$, and consider the regression of $\\mu_1,\\dots,\\mu_4$ on the rows $X_i=(1,Z_i)$. The coefficient of $Z_i$ is the AMR estimate $\\widehat{\\tau}(d)=(\\mu_1+\\mu_2-\\mu_3-\\mu_4)/2$. With $B=(X^{\\top}X)^{-1}$, residuals $e$, and the uniform kernel $K_{ij}=\\mathbf{1}\\{\\lVert x(i)-x(j)\\rVert<0.4\\}$ on the node coordinates, the spatial-HAC covariance without a degrees-of-freedom correction is $BX^{\\top}\\{K\\circ(ee^{\\top})\\}XB$. Since the smallest distance between two nodes is $1.43$, the matrix $K$ is the identity, and the variance of $\\widehat{\\tau}(d)$ is\n\\[\nV(d)=\\frac{(\\mu_1-\\mu_2)^2+(\\mu_3-\\mu_4)^2}{8}.\n\\]\nThe interval is $\\widehat{\\tau}(d)\\pm\\Phi^{-1}(0.975)\\sqrt{V(d)}$. At $d=0.1$ the circle averages are\n\\[\n(\\mu_1,\\mu_2,\\mu_3,\\mu_4)=(-0.5941,\\ 1.8588,\\ -1.2093,\\ -0.5076),\n\\]\nand I find $\\widehat{\\tau}=1.4907$ and the interval $[-0.2772,\\,3.2586]$. Together with the AMR and permutation calculations, this reproduces all 50 printed cells of the table to the third decimal.\n\\begin{audittable}{rccc}{Spatial-HAC intervals of the toy example: as printed, from the older archived copy, and from the calculation on the four nodes.}\n$d$ & Printed & Older copy & Calculation on nodes \\\\\n$0.1$ & $[-0.277,\\ 3.259]$ & $[0.127,\\ 2.855]$ & $[-0.277,\\ 3.259]$ \\\\\n$0.2$ & $[-0.242,\\ 3.343]$ & $[0.296,\\ 2.805]$ & $[-0.242,\\ 3.343]$ \\\\\n$0.3$ & $[-0.242,\\ 3.343]$ & $[0.524,\\ 2.577]$ & $[-0.242,\\ 3.343]$ \\\\\n$0.4$ & $[-0.050,\\ 3.431]$ & $[0.768,\\ 2.614]$ & $[-0.050,\\ 3.431]$ \\\\\n$0.5$ & $[-0.020,\\ 3.448]$ & $[0.846,\\ 2.582]$ & $[-0.020,\\ 3.448]$ \\\\\n$0.6$ & $[-0.038,\\ 3.438]$ & $[0.799,\\ 2.601]$ & $[-0.038,\\ 3.438]$ \\\\\n$0.7$ & $[-0.066,\\ 3.422]$ & $[0.680,\\ 2.676]$ & $[-0.066,\\ 3.422]$ \\\\\n$0.8$ & $[-0.067,\\ 2.071]$ & $[-0.527,\\ 2.530]$ & $[-0.067,\\ 2.071]$ \\\\\n$0.9$ & $[-2.424,\\ 0.791]$ & $[-3.185,\\ 1.553]$ & $[-2.424,\\ 0.791]$ \\\\\n$1.0$ & $[-2.295,\\ 0.714]$ & $[-2.907,\\ 1.325]$ & $[-2.295,\\ 0.714]$ \\\\\n\\end{audittable}\nThe two formulas for the covariance are different, and an exact example shows it. Let every raster outcome equal $1$, with the sampled cells of the toy example at $d=0.1$. Then the circle averages all equal $1$ and the variance on the nodes is $0$, while the formula of the older copy gives $31/49$.", "problem": "The paper cites version 0.1 of the package (Samii and Wang 2020), which is not available. The archived repository of the package holds two copies of the code: an older copy at its top level and a later copy in the folder of a 2024 replication. Run on the toy data, the older copy returns the ten AMR estimates and the twenty permutation bounds of the printed table, and none of its twenty spatial-HAC endpoints. At $d=0.1$ it gives $[0.127,\\,2.855]$, where the table prints $[-0.277,\\,3.259]$. The largest difference in an endpoint is $0.866$, at $d=0.5$. The difference bears on the test of the weak null: the lower endpoints from the older copy are positive at the first seven distances, and the printed ones are negative.\n\nThe cause is the covariance. The older copy computes it from a regression on the sampled raster outcomes, 499 rows with the coordinates of the raster cells. A regression on the four circle averages, with the coordinates of the nodes, reproduces the printed intervals. The older copy also cannot run the printed call as it stands: its \\texttt{SpatialEffect()} has no \\texttt{smooth} argument and no \\texttt{ras\\_Z} argument, and the copy has no \\texttt{SpatialEffectTest()}.\n\nNothing printed is shown to be wrong. The finding is that the table cannot be reproduced from this archived copy.", "what_survives": "The printed table stands in all 50 cells. The definitions and the asymptotic discussion of Section 2.3 are not affected. The printed cumulative test does not use the spatial-HAC covariance. Which code produced the table is not established. This check does not concern the later copy, whose spatial-HAC calculation is the subject of \\fid{P008-E015}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N006", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the four-line caption of Figure 5. Red boxes mark the sentence 'The black dashed line is the smoothed AMR estimates', which ends on the last line before '(Color figure online)'.", "caption": "Figure 5 caption, p. 150 (PDF p. 13). The marked sentence identifies the dashed curve as the smoothed AMR estimates.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N007.png"}, {"alt": "Crop of four lines of code and two lines of text. The code is 'dVec = seq(from=.1,to=2, by=.1)' followed by the call 'AMRest <- SpatialEffect(...)' over three lines. A red box marks the last argument, 'smooth = 1'.", "caption": "Section 5, p. 148 (PDF p. 11), the printed call for the toy example. The marked argument requests the smoothed estimates.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Neither archived copy of the SpatialEffect package runs the printed toy call with smooth = 1, so the dashed curve of Figure 5 cannot be regenerated from that call.", "id": "P008-N007", "location": "Figure 5, black dashed curve and caption, p. 150 (PDF p. 13); the call with smooth = 1, p. 148 (PDF p. 11)", "pdf_page": 13, "printed_page": "150", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~148:\n\n\\texttt{dVec = seq(from=.1,to=2, by=.1)}\n\n\\texttt{AMRest <- SpatialEffect(ras = ras\\_sim, Zdata = Zdata, x\\_coord\\_Z = ’’x’’,}\n\n\\texttt{y\\_coord\\_Z = ’’y’’, treatment = ’’Z’’, dVec = dVec,}\n\n\\texttt{cutoff = .4, nPerms = 1000, \\hl{smooth = 1})}\n\np.~150, Figure 5 caption: ``The black dashed line is the \\hl{smoothed AMR estimates} (Color figure online).''", "corrected": "The caption needs no change. To obtain smoothed estimates from the later archived copy on the grid $0.1,0.2,\\dots,2.0$, \\hl{both bandwidths have to be supplied}, through the arguments \\texttt{bw} and \\texttt{bw\\_debias}, with $\\hl{\\texttt{bw}>0.1}$ for the linear fit and $\\hl{\\texttt{bw\\_debias}>0.2}$ for the quadratic bias correction. Each fit needs at least two and three distinct positive-weight distances, respectively, in each treatment group, including at the ends of the grid. Supplying \\texttt{bw} alone still starts the cross-validation. In the code, the cross-validation should discard a bandwidth for which $X^{\\top}WX$ is not of full rank before the inversion.\n\nThe bandwidths used for Figure 5 are not known. To reproduce the dashed curve one needs the release used for the paper and its smoothing settings.", "evidence": "The evidence is an exact computation and a run of the code. Let $t=d-0.1$ and let the design have the columns $(1,\\ Z,\\ t,\\ Zt,\\ t^{2},\\ (Zt)^{2})$, as in the later copy. On the rows with positive weight, I have $t\\in\\{0,\\,1/10\\}$ and hence $t^{2}=t/10$. Further, $(Zt)^{2}=Zt^{2}$ because $Z$ is binary. The vectors\n\\[\nv_1=(0,\\,0,\\,-1/10,\\,0,\\,1,\\,0)^{\\top},\\qquad v_2=(0,\\,0,\\,0,\\,-1/10,\\,0,\\,1)^{\\top}\n\\]\ntherefore satisfy $Xv_1=Xv_2=0$ for the design matrix $X$ restricted to those rows. For the diagonal matrix $W$ of their weights, this implies that $X^{\\top}WXv_1=X^{\\top}WXv_2=0$, and the $6\\times 6$ matrix $X^{\\top}WX$ has rank at most four. In exact arithmetic, I find rank four and determinant zero for the treatment patterns that occur in the cross-validation folds. The four-column matrix of the linear fit is invertible.\n\nI ran the smoothing functions of the later copy unchanged on an example with four nodes and the distances $0.1,0.2,\\dots,2.0$, and supplied the variables that the cross-validation function reads from its environment. The run stops with ``system is computationally singular: reciprocal condition number = 4.71786e-22''. With both bandwidths set to $0.4$, the smoother returns finite estimates at all twenty distances for this example, with the bias correction and without smoothed standard errors. I did not make a complete run of the printed call with these bandwidths on the toy data.", "problem": "The printed call requests smoothed estimates, and Figure 5 shows them as the dashed curve. The paper cites version 0.1 of the package (Samii and Wang 2020), which is not available. The archived repository of the package holds two copies of the code: an older copy at its top level and a later copy in the folder of a 2024 replication. Neither runs the printed call.\n\nIn the older copy, \\texttt{SpatialEffect()} has no argument \\texttt{smooth}, and R stops with ``unused argument (smooth = 1)''. The later copy accepts the argument and, with its default settings, chooses the two smoothing bandwidths by cross-validation over a grid whose first value is $(2-0.1)/10=0.19$. At the first evaluation distance, $0.1$, the triangular weights with this bandwidth are positive only at the distances $0.1$ and $0.2$. The bias-corrected fit is a regression on six columns that include a quadratic term, and two distinct distances do not determine a quadratic. The matrix to be inverted has rank at most four, whatever the outcomes, and R stops with ``system is computationally singular''. A call that stops in this step returns nothing, including the unsmoothed estimates it has already computed.\n\nIt is not shown that the published dashed curve is wrong, or that the cited release failed.", "what_survives": "The definitions, the unsmoothed estimator, the solid curve of Figure 5, the output table and its intervals are not affected. The printed test \\texttt{SpatialEffectTest(AMRest, c(0.1, 0.5))} uses the unsmoothed estimates and does not need the dashed curve. No published number is shown to be wrong."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N007", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Five lines of text from page 153. A red box marks the words: the area within the boundary of each village. They end the sentence that begins: Therefore, the starting point at 0 corresponds to.", "caption": "Section 6, first paragraph, p. 153 (PDF p. 16). The marked phrase describes the point at distance 0 as the area inside each village; the edge estimates at 0 use the cells on the village boundary.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N009.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T19", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T19"}, {"id": "P008-T20", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T20"}], "headline": "The value at distance 0 in the edge-based panels of Figure 8 summarizes raster cells on the village boundary, not the area inside the village.", "id": "P008-N009", "location": "Section 6, first paragraph, p. 153 (PDF p. 16); Figure 8, top panels, p. 154 (PDF p. 17)", "pdf_page": 16, "printed_page": "153", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P008-T19", "P008-T20"], "text": {"as_printed": "We consider the experiment as an example of polygon interventions. Therefore, the starting point at 0 corresponds to \\hl{the area within the boundary of each village}.", "corrected": "For the edge-based estimates, the value at distance 0 averages outcomes over \\hl{raster cells on the village boundary, rather than over the full village polygon}. The smoothed curve is fitted to these boundary-based summaries across distances. In the bottom panel, the first value is the contrast over the area within the village polygons.", "evidence": "The evidence is an exact counterexample and a recalculation. In the 2024 version of the package source, the default circle type, the edge, keeps only the boundary cells of the rasterized buffer, including at distance 0. I consider a raster of 200 cells with two polygons of $5 \\times 5$ cells, one treated and one control, and I set the outcome to 1 on the nine central cells of the treated polygon and to 0 elsewhere. The edge estimate at distance 0 uses the 16 boundary cells of each polygon and equals $0/16 - 0/16 = 0$, while the contrast over the polygon areas is $9/25 - 0/25 = 9/25$.\n\nOn the unaggregated forest-loss raster and the 121 village polygons in the replication files of the later companion paper, the boundary contrast is $-0.010516$ and the contrast inside the villages is $-0.013466$. The boundary value is about $21.9\\%$ smaller in magnitude. These two values agree visually with the left end of the Hajek curve and with the first step of the bottom panel of Figure 8. The inputs and the package release used for the published figure are not archived, and these files are not shown to be the ones behind it. No such percentage is established for the smoothed curve.", "problem": "The sentence says that the estimate at distance 0 summarizes outcomes over the whole village polygon. The two top panels of Figure 8 show edge estimates: at each distance $d$ they average the raster cells at distance $d$ from the village boundary. At $d = 0$ these are the cells on the inner boundary of the rasterized polygon, and the cells in its interior are left out. The smoothed curve is fitted to these boundary summaries across distances. Its value at 0 is not a contrast over the village area either.", "what_survives": "The estimator, the plotted estimates, the confidence intervals and the band estimates of the bottom panel are unchanged; only the description of the point at 0 changes. The test over 1--2.5 km uses only the distances in that window and does not involve this point."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N009", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Ten lines of text from page 153. Red boxes mark the words: is thus significant with p less than 0.05. The words run from the end of one line to the start of the next, in the sentence about the cumulative effect between 1 km and 2.5 km.", "caption": "Section 6, paragraph after the test call, p. 153 (PDF p. 16). The marked words assert significance at the 5% level using the reported sharp-null percentiles.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N012.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T1", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T1"}, {"id": "P008-T2", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T2"}, {"id": "P008-T4", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T4"}, {"id": "P008-T21", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T21"}], "headline": "The statement “significant with $p < 0.05$” is not established under the design of the experiment: in the inspected implementations, the printed call permutes treatment freely over the 121 villages, but treatment was assigned by seven subcounty lotteries between lists of villages balanced on covariates.", "id": "P008-N012", "location": "Section 6, displayed SpatialEffect() call and the paragraph after the SpatialEffectTest() call, p. 153 (PDF p. 16); see also p. 151 (PDF p. 14) and p. 145 (PDF p. 8)", "pdf_page": 16, "printed_page": "153", "reaches": [{"effect": "result_needs_same_repair", "effect_text": "a result or code path there needs the same repair", "explanation": null, "package": null, "recorded_as": "P004-N007", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "result_needs_same_repair", "effect_text": "a result or code path there needs the same repair", "explanation": null, "package": "SpatialEffect", "recorded_as": "P004-N007", "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P004-N007", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-N007", "work_id": "P004"}], "severity": "Major", "subject": "paper", "target_ids": ["P008-T1", "P008-T2", "P008-T21", "P008-T4"], "text": {"as_printed": "\\texttt{dVec = seq(from=0, to=20/111, by=5/111) AMRest <- Spatial}\n\n\\texttt{Effect(ras = ras, ras\\_Z = ras\\_Z, treatment = ’’treatment’’,}\n\n\\texttt{dVec = dVec, smooth= 1, cutoff = 10000,}\n\n\\texttt{dist.metric = ’’Geodesic’’)}\n\n\\texttt{testResult <- SpatialEffectTest(AMRest, test.range = c(1/111,}\n\n\\texttt{2.5/111))}\n\nFor the cumulative effect between 1 km and 2.5 km, the 0.025 and .975 percentiles of the sharp null permutation distribution are $-0.0304$ and $0.0336$ [\\ldots], respectively. The estimated cumulative effect between 1 km and 2.5 km is $-0.0321$ [\\ldots]. The cumulative effect between 1 km and 2.5 km is \\hl{thus significant with $p < 0.05$}.\n\n[p.~151] The study was carried out in 121 villages, 60 of which were randomly assigned to treatment (Fig.~6).", "corrected": "The cumulative effect between 1 km and 2.5 km is \\hl{below the reported lower sharp-null critical value, $-0.0304$}. \\hl{In the inspected implementations, the displayed call permutes treatment freely over the 121 villages. The experiment assigned treatment by a lottery in each of seven subcounties between two lists of villages balanced on covariates (Jayachandran et al. 2017, Supplementary Materials, section 2.2). The unrestricted comparison is therefore a benchmark and does not establish significance at the 5\\% level under the design of the experiment.}\n\nTo state significance under the design, the sharp-null distribution must be generated from the documented subcounty lotteries between balanced lists, or from a conditional distribution that is justified for that design, and the percentiles and the $p$-value must be recomputed. Passing the subcounty as \\texttt{blockvar} does not by itself impose the balance restriction. A claim that the spatial-HAC intervals are valid under this design requires its own justification.", "evidence": "I base this correction on the published protocol of the source study, the displayed call and an exact derivation. The protocol is quoted from the \\href{https://seemajayachandran.com/supplementary_material_pes.pdf}{copy of the Supplementary Materials} posted by the first author of the source study (Materials and Methods, section 2.2, p.~3). It refers to ``each of the 7 subcounties'' and says that, in advance of each lottery, the sample villages in the subcounty were divided ``into two sets, balanced on covariates''. The printed call shows no blocking or clustering argument. In both inspected implementations, these arguments default to \\texttt{NULL} and are passed to the permutation routine of the \\texttt{ri} package, which then draws uniformly among the assignments with the observed number of treated units. I could not reconstruct the published computation, since I do not have the cited package release or the original application code.\n\nNeither the size nor the direction of the change can be inferred without recomputing the test. Let four units be given, two of them treated, and let the statistic be the difference in means at the assignment $(1,0,1,0)$. With fixed outcomes $(0,1,0,1)$, the two-sided tail probability is $1/3$ under free permutation and $1/2$ when one unit is treated in each of the pairs $\\{1,2\\}$ and $\\{3,4\\}$. With outcomes $(0,1/3,2/3,1)$, it is $2/3$ and $1/2$. No corrected $p$-value for the application is established, and a test under the documented design could still reject at the 5\\% level.", "problem": "The paper takes the distribution of the treatment assignment as known and derives its inference from that distribution (p.~142). It also says that the permutation test can be adjusted for blocking or clustering in the design (p.~145). The printed call for the application passes neither \\texttt{blockvar} nor \\texttt{clustvar}. With these defaults, the inspected implementations draw the permutations uniformly from the assignments that treat 60 of the 121 villages.\n\nThe experiment was not randomized in this way. According to the Supplementary Materials of the source study (Jayachandran et al., 2017, section 2.2), treatment was assigned by a public lottery in each of seven subcounties. Before each lottery the sample villages of the subcounty were divided into two lists. The sizes of the two lists differ by at most one, and the lists are balanced on four village covariates (the number of forest owners sampled, distance to a road, income per capita and landholding size), with a threshold of 0.25 on each standardized mean difference. In small subcounties all partitions were examined, and in larger ones 1000 were sampled. One eligible partition was selected at random before each public draw, and the lottery chose between its two lists. This is neither an unrestricted choice of 60 villages nor an independent assignment of each village. Conditioning on the total of 60 treated villages does not remove the subcounty and balance restrictions.\n\n\\begin{audittable}{lll}{The assignment in the source experiment and the printed call's inspected defaults.}\n & Source experiment & Printed call \\\\\nAssignment groups & seven subcounties & all 121 villages together \\\\\nWhat is drawn & one of two lists of villages & 60 of the 121 villages \\\\\nBalance on covariates & required of the two lists & none \\\\\n\\end{audittable}\n\nThe unrestricted default does not generate the randomization distribution of the experiment. The reported rejection rests on a margin of $0.0017$, between $-0.0321$ and $-0.0304$, and does not establish significance at the 5\\% level under the design. This is the only formal test in the application. The restrictions also rule out independent assignment of the 121 villages, so a justification of the spatial-HAC intervals that relies on Bernoulli assignment needs a further argument for this design.", "what_survives": "The comparison $-0.0321 < -0.0304$ is arithmetically correct, and the circle averages and the estimated curve of Figure 8 do not depend on how the permutations are generated. The sharp-null method, the option of the package for blocked designs and the toy example are not affected. What changes is the conclusion. The significance statement, and with it the inferential support for a bandwagon effect ``up to 2.5 km'', is not established by the displayed call's inspected defaults until the test is recomputed under the documented design. A failure of the spatial-HAC intervals is not shown."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N012", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of text from page 153. Red boxes mark two parentheticals. The first reads: that is, a 3.04 percentage point reduction to a 3.36 percentage point increase in the cumulative probability of forest loss. The second reads: i.e., a 3.21 percentage point reduction.", "caption": "Section 6, paragraph after the test call, p. 153 (PDF p. 16). The two marked parentheticals read the summed statistic and its null percentiles as percentage-point changes in a probability of forest loss.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N016.png"}, {"alt": "Bar chart of four AMR estimates at 1, 1.5, 2 and 2.5 km, in percentage points: about -0.97, -0.93, -0.49 and -0.82. A solid blue horizontal line marks their mean at -0.80. A dashed orange horizontal line marks their sum multiplied by 100 at -3.21, far below every bar; this sum is not a probability change.", "caption": "The four estimates in the window from 1 to 2.5 km, read from the black curve in the top-left panel of Figure 8 (p. 154; PDF p. 17), with their mean and their sum, each multiplied by 100. The sum is about $-3.21$ on this scale and is not a probability change; the mean is about $-0.80$ percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N016-sum-and-mean.png"}], "formal_release": null, "formal_statements": [{"id": "P008-T7", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T7"}, {"id": "P008-T8", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T8"}, {"id": "P008-T24", "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-T24"}], "headline": "The printed $-0.0321$, $-0.0304$ and $0.0336$ are unweighted sums of AMR estimates at several distances, not changes of 3.21, 3.04 and 3.36 percentage points in a probability of forest loss.", "id": "P008-N016", "location": "Section 6, paragraph after the SpatialEffectTest() call, p. 153 (PDF p. 16), both parentheticals", "pdf_page": 16, "printed_page": "153", "reaches": [{"effect": "same_error_restated", "effect_text": "a corresponding statement is printed there", "explanation": null, "package": null, "recorded_as": "P004-E008", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P004/", "where": null, "work_id": "P004"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "SpatialEffect", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P008"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P004-E008", "url": "https://pmaronow.github.io/llm-audit/works/P004/#P004-E008", "work_id": "P004"}], "severity": "Moderate", "subject": "paper", "target_ids": ["P008-T24", "P008-T7", "P008-T8"], "text": {"as_printed": "For the cumulative effect between 1 km and 2.5 km, the 0.025 and .975 percentiles of the sharp null permutation distribution are $-0.0304$ and $0.0336$ \\hl{(that is, a 3.04 percentage point reduction to a 3.36 percentage point increase in the cumulative probability of forest loss)}, respectively. The estimated cumulative effect between 1 km and 2.5 km is $-0.0321$ \\hl{(i.e., a 3.21 percentage point reduction)}.", "corrected": "For the \\hl{sum of the AMR estimates at the distances evaluated} between 1 km and 2.5 km, \\hl{each with weight one}, the 0.025 and .975 percentiles of the sharp null permutation distribution are $-0.0304$ and $0.0336$, respectively. The \\hl{observed sum} is $-0.0321$, \\hl{which lies below the lower percentile}. \\hl{These numbers are values of a summed AMR statistic and are not changes in a probability of forest loss.}\n\nIf the grid consists of the four distances 1, 1.5, 2 and 2.5 km shown in Figure 8, the \\hl{equally weighted mean} of the four estimates, using the printed sum, is $-0.0321/4 = -0.008025$. This \\hl{estimates a reduction} of about \\hl{0.8 percentage points} for a probability of forest loss that \\hl{gives equal weight to each distance and each village and uses the paper's circle averages}. On the same scale the percentiles are $-0.0304/4 = -0.0076$ and $0.0336/4 = 0.0084$. The text should state the grid of distances that was used.", "evidence": "My evidence is a derivation and an exact counterexample: for loss probabilities $Q_b$ defined by drawing a village uniformly, setting its assignment to $b$, drawing a distance with probabilities $w_j$, and using the paper's circle-average measure, with other assignments averaged over the design, I obtain\n\\[\nQ_1 - Q_0 = \\sum_{j=1}^m w_j \\tau(d_j), \\qquad w_j \\ge 0, \\quad \\sum_{j=1}^m w_j = 1.\n\\]\nFor the counterexample, I take four village centres at least 20 km apart and 40000 points on each circle at distances 1, 1.5, 2 and 2.5 km, with independent assignments of probability $1/2$ and binary outcomes defined on all 640000 circle points and all four centres for every assignment.\nAt the assignment $(1,1,0,0)$, each AMR and its Hajek estimate equal $-0.008025$, giving sum $-0.0321$, while the constructed pooled loss probabilities are $0.0175$ under control and $0.009475$ under treatment, a change of $-0.8025$ percentage points.\nReading the sum as this probability change would give $0.0175 - 0.0321 = -0.0146 < 0$ under treatment; a companion binary model on the same spatial field gives sum $-4$ and probability change $-1$, ruling out a general conversion of the sum into a probability change.", "problem": "The cumulative statistic is the sum of the AMR estimates at the distances in the window, each with weight one (the summation is described on p.~146). Each AMR $\\tau(d)$ is a difference of two circle-average probabilities of forest loss at distance $d$, averaged over villages and over the other villages' assignments, with the selected village treated and not treated. A sum of $m$ such differences need not be a difference of probabilities. For a probability defined by mixing these circle averages over distances, the change is a weighted average $\\sum_j w_j \\tau(d_j)$ with nonnegative weights that sum to one. The weights of the printed statistic sum to $m$. The two parentheticals read the sum and its null percentiles as percentage-point changes in ``the cumulative probability of forest loss'', and the unit weights of the sum do not define such a probability. Figure 8 shows four estimates in the window, at 1, 1.5, 2 and 2.5 km. For this four-distance grid, dividing the printed sum $-0.0321$ by four gives a mean estimate of about $-0.0080$, an estimated reduction of 0.8 percentage points. The value 3.21 is the only effect size quoted for the application.\n\n\\begin{audittable}{lrr}{The printed values, and the same values as an equally weighted mean over four distances.}\nQuantity & As printed (sum) & Divided by four (mean) \\\\\n0.025 percentile of the null distribution & $-0.0304$ & $-0.0076$ \\\\\nObserved statistic & $-0.0321$ & $-0.008025$ \\\\\n0.975 percentile of the null distribution & $0.0336$ & $0.0084$ \\\\\n\\end{audittable}", "what_survives": "The AMR, the Hajek estimator and the use of a fixed sum of AMR estimates as the statistic of a sharp-null test are unaffected by this interpretation error. Dividing the statistic and every permutation value by the same positive number preserves ranks and the printed comparison $-0.0321 < -0.0304$, provided that the statistic and the percentiles use the same grid; the original application percentiles and exact $p$-value have not been independently reproduced. The summed AMR statistic does not represent a 3.21-percentage-point reduction in a probability of forest loss; the grid in the printed code has no distance inside the window, which is the subject of \\fid{P008-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N016", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The bottom panel of Figure 8, titled 2km Donut Estimates with CIs: a black step trace of AMR estimates rising from about -0.013 at the left to about zero at 20 km, between two red dotted confidence-interval traces. Red boxes mark the first horizontal step, drawn over 0 to 2 km, and the short vertical stroke at 20 km where the trace ends.", "caption": "Figure 8, bottom panel, p. 154 (PDF p. 17). The marked step over 0–2 km is the contrast inside the villages, and every later step is the estimate for the band before it; the estimate for 18–20 km appears only as the marked end of the stroke at 20 km.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N017.png"}, {"alt": "Step plot of AMR estimates against distance from the village boundary, 0 to 20 km, with a shaded region at the left for the inside of the village. A dashed orange trace reproduces the published steps. A solid blue trace places the ten exterior-band heights over their own bands and the village-interior height over the shaded region. Ten open circles show recalculated contrasts at the band midpoints, and an eleventh shows the village-interior contrast in the shaded region. The circles closely follow the solid blue trace.", "caption": "The heights of the bottom panel of Figure 8 as drawn (dashed) and over their corresponding regions (solid), with the contrasts recalculated from the replication files of the later companion paper (circles). The recalculated contrasts closely follow the solid trace; the published value for 18–20 km has no horizontal step.", "url": "https://pmaronow.github.io/llm-audit/works/P008/exhibits/P008-N017-band-placement.png"}], "formal_release": null, "formal_statements": [], "headline": "The horizontal steps in the bottom panel of Figure 8 place the 2 km band estimates and their confidence intervals over the next band out, and the contrast inside the villages over 0–2 km.", "id": "P008-N017", "location": "Figure 8, bottom panel (“2km Donut Estimates with CIs”), p. 154 (PDF p. 17); description on p. 153 (PDF p. 16)", "pdf_page": 17, "printed_page": "154", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "[Description of Figure 8, bottom panel, ``2km Donut Estimates with CIs'': a step trace with ten horizontal steps. The step \\hl{over 0--2 km} has height about $-0.0135$, the step \\hl{over 2--4 km} about $-0.0085$, the step \\hl{over 4--6 km} about $-0.0059$, the step \\hl{over 6--8 km} about $-0.0024$, the step \\hl{over 8--10 km} about $-0.0010$, and so on to \\hl{18--20 km}. The last value, about $0.0004$, appears \\hl{only as the end of the vertical stroke at 20 km}.]\n\n[p.~153] The bottom graph shows 2km ``donut'' estimates: we estimate the AMR using outcome raster outcomes within 2km bands, rather than values precisely at the edge of the circle. The results show a similar pattern as the two graphs above.", "corrected": "[Figure 8, bottom panel, redrawn: for $d = 2, 4, \\dots, 20$ km, the estimate for the band $(d-2, d]$ and its confidence interval are drawn \\hl{over $(d-2, d]$}. The estimate at $d = 0$ is \\hl{shown separately, as the contrast inside the villages}.]\n\nCaption, added: \\hl{Each step of the bottom panel is the estimate for the 2 km band of distances that it spans. The value at $d = 0$ is shown separately as the contrast inside the village polygons.}", "evidence": "I compared the published figure with band contrasts recalculated from the forest-loss raster and the 121 village polygons (60 treated, 61 control) in the replication files of the later companion paper, using nested buffers around each village. Matching the eleven heights read from the panel's vector paths to the recalculated contrasts with the same index $d$, I find a root-mean-square discrepancy of $0.000052$, compared with $0.002307$ for matching the ten horizontal steps to the bands they span, a factor of 44 larger. The vector paths show that each horizontal step runs to the right from $d$ to $d + 2$, for the estimates and for both confidence-interval traces. The replication files are those of a later paper and have not been shown to be the inputs of the published figure; the table shows the small differences that remain.\n\n\\begin{audittable}{llrr}{Each outcome region, the interval over which the bottom panel of Figure 8 draws its estimate, the height read from the published panel, and the contrast recalculated from the replication files.}\nBand & Drawn over & Height in Figure 8 & Recalculated \\\\\nInside the village & 0--2 km & $-0.013466$ & $-0.013466$ \\\\\n$(0,2]$ km & 2--4 km & $-0.008476$ & $-0.008468$ \\\\\n$(2,4]$ km & 4--6 km & $-0.005939$ & $-0.005851$ \\\\\n$(4,6]$ km & 6--8 km & $-0.002395$ & $-0.002439$ \\\\\n$(6,8]$ km & 8--10 km & $-0.000958$ & $-0.000866$ \\\\\n$(8,10]$ km & 10--12 km & $-0.000212$ & $-0.000246$ \\\\\n$(10,12]$ km & 12--14 km & $0.000355$ & $0.000317$ \\\\\n$(12,14]$ km & 14--16 km & $0.001280$ & $0.001313$ \\\\\n$(14,16]$ km & 16--18 km & $0.000231$ & $0.000179$ \\\\\n$(16,18]$ km & 18--20 km & $0.001673$ & $0.001658$ \\\\\n$(18,20]$ km & end of the stroke at 20 km & $0.000408$ & $0.000338$ \\\\\n\\end{audittable}", "problem": "The panel is titled as estimates for 2 km bands, and the text says that the AMR is estimated from outcomes within 2 km bands. The step drawn over 2--4 km therefore reads as the estimate for points between 2 and 4 km from a village boundary. It is the estimate for the band from 0 to 2 km. For $d = 2, 4, \\dots, 18$ km, the value is indexed by the outer edge $d$ of its band $(d-2, d]$ and is then drawn to the right of $d$, over $[d, d+2]$. The value drawn over 0--2 km is the contrast inside the villages, and the estimate for the band from 18 to 20 km has no horizontal step. Both confidence-interval traces are shifted in the same way. The text does not state this convention, and Section 3 describes a band as centred at $d$ (p.~145). The misplaced spans make the village-interior contrast appear just outside the villages and put each exterior-band step 2 km farther out.", "what_survives": "The heights of the estimates and of the confidence intervals can be kept: for $d = 2, 4, \\dots, 20$ km, $d$ is the outer edge of the band, while $d = 0$ denotes the village interior. The two top panels of Figure 8, the test over 1--2.5 km, the method and the toy example do not depend on this drawing convention; this correction does not by itself change the separate spillover-distance conclusion. The existing horizontal steps for the exterior bands move in by 2 km; the final band gains a horizontal step, and the village-interior contrast is shown separately."}, "url": "https://pmaronow.github.io/llm-audit/works/P008/#P008-N017", "work": {"arxiv_id": "2106.15081", "doi": "10.1007/s13253-022-00517-y", "title": "Inference in Spatial Experiments with Interference using the SpatialEffect Package", "url": "https://pmaronow.github.io/llm-audit/works/P008/"}, "work_id": "P008"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first paragraph of Section 6.3, Tail bounds, and the start of Proposition 11. The paragraph reads: Experimenters often use variance estimates to construct bounds on the tails of the sampling distribution of the point estimator, which in turn may be used to construct confidence intervals and hypothesis tests. A common approach is to combine a conservative variance estimator with a normal approximation of the sampling distribution, motivated by a central limit theorem. Such approximations may be reasonable when the interference is very sparse. For example, Theorem 2.7 in Chen and Shao (2004) can be applied to the HT point estimator under the Bernoulli design if d MAX = O(1), showing that the sampling distribution is approximately normal in large samples. This condition is, however, stronger than Assumption 2. The following proposition demonstrates that normal approximations will generally not be accurate under the conditions considered in this paper. A red outline surrounds the sentence that begins For example, Theorem 2.7 in Chen and Shao (2004). Below the paragraph, Proposition 11 begins: Chebyshev's inequality is asymptotically sharp with respect to the sampling distribution of the HT estimator for every sequence of Bernoulli designs under.", "caption": "Section 6.3, p. 693 (PDF p. 21). The outline marks the sentence that derives a normal approximation from $d_{\\mathrm{MAX}} = \\mathcal{O}(1)$ alone.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T7", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T7"}, {"id": "P009-T8", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T8"}], "headline": "The normal approximation for the HT estimator under a Bernoulli design does not follow from $d_{\\mathrm{MAX}} = \\mathcal{O}(1)$ alone; it holds under the further condition $\\liminf_{n \\to \\infty} n\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) > 0$.", "id": "P009-E001", "location": "Section 6.3, p. 693 (PDF p. 21), first paragraph, sentence beginning “For example, Theorem 2.7 in Chen and Shao (2004)”", "pdf_page": 21, "printed_page": "693", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T7", "P009-T8"], "text": {"as_printed": "Such approximations may be reasonable when the interference is very sparse. For example, Theorem~2.7 in Chen and Shao (2004) can be applied to the HT point estimator under the Bernoulli design \\hl{if $d_{\\mathrm{MAX}} = \\mathcal{O}(1)$}, showing that the sampling distribution is approximately normal in large samples. This condition is, however, stronger than Assumption~2.", "corrected": "For example, under Assumption~1, a Bernoulli design and $d_{\\mathrm{MAX}} = \\mathcal{O}(1)$, the HT point estimator centred at $\\tau_{\\mathrm{EATE}}$ and divided by its true standard deviation is asymptotically standard normal, \\hl{provided that $\\liminf_{n \\to \\infty} n\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) > 0$}:\n\\[\n\\frac{\\hat\\tau_{\\mathrm{HT}} - \\tau_{\\mathrm{EATE}}}{\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})^{1/2}} \\overset{d}{\\longrightarrow} \\mathcal{N}(0, 1).\n\\]\nThis condition on $d_{\\mathrm{MAX}}$ is, however, stronger than Assumption~2.", "evidence": "The evidence is an exact counterexample and a derivation. For each $n \\geq 1$, let the assignments be independent with probability $1/2$, and let $y_1(\\mathbf{z}) = z_1$ and $y_i(\\mathbf{z}) = 0$ for $i > 1$. All outcomes are bounded by one, Assumption~1 holds with $k = 2$, and $d_{\\mathrm{MAX}} = d_{\\mathrm{AVG}} = 1$. Assumption~2 holds as well. In this example $\\tau_{\\mathrm{EATE}} = 1/n$, $\\hat\\tau_{\\mathrm{HT}} = 2Z_1/n$ and $\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) = 1/n^2$, and I obtain\n\\[\n\\frac{\\hat\\tau_{\\mathrm{HT}} - \\tau_{\\mathrm{EATE}}}{\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})^{1/2}} = 2Z_1 - 1 \\qquad \\text{for every } n.\n\\]\nThe standardized estimator takes the values $-1$ and $1$ with probability $1/2$ each. Its distribution function jumps by $1/2$ at $-1$. This implies that its Kolmogorov distance from any continuous distribution function is at least $1/4$, and from the standard normal it is $1/2 - \\Phi(-1) \\approx 0.341$. Here $n\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) = 1/n$. Replacing the outcomes of the units $i > 1$ by $y_i(\\mathbf{z}) = z_i - 1/2$, I obtain $\\tau_{\\mathrm{EATE}} = 1$ and the same two-point law. The failure does not come from an estimand that tends to zero. Both examples agree with an enumeration of all assignments in exact arithmetic for $n = 1, 2, 3, 4, 8, 10$.\n\nFor the corrected statement, suppose that the design is Bernoulli, that $d_{\\mathrm{MAX}} \\leq D$ for all $n$, and that Assumption~1 holds. The outcome of unit $i$ depends on at most $D$ treatments, and each configuration of these has probability at least $k^{-D}$ by Assumption~1A. Since $\\mathrm{E}|Y_i|^2 \\leq k^2$ by Assumption~1B, every potential outcome satisfies $|y_i(\\mathbf{z})| \\leq k^{1 + D/2}$. I write $n\\hat\\tau_{\\mathrm{HT}} = \\sum_i T_i$. The centred summands $T_i - \\mathrm{E}[T_i]$ are uniformly bounded, and their sum is $n(\\hat\\tau_{\\mathrm{HT}} - \\tau_{\\mathrm{EATE}})$. If two sets of units have no pair with $d_{ij} = 1$ between them, then their summands depend on disjoint sets of independent treatments. If $n\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) \\geq c > 0$ for all large $n$, the variance of the sum is at least $cn$. Only $\\mathcal{O}(n)$ connected $r$-tuples of units contribute to the $r$th cumulant of the sum, so for each $r \\geq 3$ the $r$th cumulant of the standardized sum is $\\mathcal{O}(n^{1 - r/2})$ and tends to zero. Its first two cumulants are $0$ and $1$, and convergence to the standard normal law follows by the method of moments.", "problem": "A bound on $d_{\\mathrm{MAX}}$ limits the dependence between the summands of $\\hat\\tau_{\\mathrm{HT}}$, and it says nothing about their variance. Assumption~1 bounds moments from above only. The sequence of samples is arbitrary, and the variance of $\\hat\\tau_{\\mathrm{HT}}$ can be carried by a bounded number of units. Then $n\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})$ tends to zero, and the estimator, centred and divided by its standard deviation, need not be close to normal. This standardized sense is the one that the passage needs, since it concerns tail bounds, confidence intervals and tests. Note that the cited theorem is itself stated for centred summands whose sum has variance one.", "what_survives": "No proposition, simulation or conclusion of the paper uses this remark. Proposition~11 has its own construction in Supplement~A, and the consistency results of Section~5 hold in both examples, where the estimation error is $1/n$ in absolute value. The preceding sentence, that a normal approximation may be reasonable under very sparse interference, stands. The added condition is the non-degeneracy that a central limit theorem ordinarily carries. It is sufficient and not necessary. The corrected statement divides by the true standard deviation, not by an estimate of it."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E001", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the foot of p. 689. The first line reads: estimator under a Bernoulli design. A displayed formula defines the variance estimator Var-hat BER of tau-hat HT as one over n squared times the sum over i of Z i Y i squared over p i squared, plus one over n squared times the sum over i of (1 minus Z i) Y i squared over (1 minus p i) squared. The text below reads: The estimator is conservative under no interference, meaning that its expectation is greater than the true variance. On a normalized scale, the bias does not diminish asymptotically, so inferences based on the estimator will be conservative also in large samples. In particular, with a Bernoulli design under no interference. Equation (1) follows: n times the bracket Var-hat BER of tau-hat HT minus Var of tau-hat HT converges in probability to tau MSQ, which is at least 0. A red outline surrounds the sentence that begins On a normalized scale.", "caption": "Section 6.1, p. 689 (PDF p. 17). The outline marks the sentence that passes from the bias of the variance estimator to conservative inferences. Equation (1) follows it.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T1", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T1"}, {"id": "P009-T2", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T2"}, {"id": "P009-T48", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T48"}], "headline": "Equation (1) and Definition 10 control the variance estimator on an additive scale, and this property alone does not make inferences based on the estimator conservative in large samples.", "id": "P009-E002", "location": "Section 6.1, p. 689 (PDF p. 17), second sentence after the display that defines $\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})$; see also Definition 10, p. 691 (PDF p. 19)", "pdf_page": 17, "printed_page": "689", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T1", "P009-T2", "P009-T48"], "text": {"as_printed": "The estimator is conservative under no interference, meaning that its expectation is \\hl{greater than} the true variance. On a normalized scale, \\hl{the bias does not diminish asymptotically, so inferences based on the estimator will be conservative also in large samples}. In particular, with a Bernoulli design under no interference,\n\\[\n\\text{(1)}\\qquad n\\bigl[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}}) - \\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\bigr] \\overset{p}{\\longrightarrow} \\tau_{\\mathrm{MSQ}} \\geq 0,\n\\]", "corrected": "The estimator is conservative under no interference, meaning that its expectation is \\hl{at least as large as} the true variance. With a Bernoulli design under no interference,\n\\[\n\\text{(1)}\\qquad n\\bigl[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}}) - \\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\bigr] \\overset{p}{\\longrightarrow} \\tau_{\\mathrm{MSQ}} \\geq 0.\n\\]\n\\hl{Equation~(1) gives a nonnegative one-sided error for the variance estimator on the scale $n$; the scaled bias $\\tau_{\\mathrm{MSQ}}$ may vanish. This variance property alone does not establish that plug-in confidence intervals are conservative.}\n\nA coverage statement needs control of the estimator relative to the variance. Let the design be Bernoulli, so that $\\hat\\tau_{\\mathrm{HT}}$ is unbiased for $\\tau_{\\mathrm{EATE}}$, let $\\hat V \\geq 0$ be a variance estimator, and suppose that $\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) > 0$ and that\n\\[\n\\Pr\\bigl\\{\\hat V < (1 - \\eta)\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\bigr\\} \\to 0 \\qquad \\text{for every } 0 < \\eta < 1.\n\\]\nThen, for every $\\alpha \\in (0, 1)$,\n\\[\n\\liminf_{n \\to \\infty} \\Pr\\bigl\\{|\\hat\\tau_{\\mathrm{HT}} - \\tau_{\\mathrm{EATE}}| \\leq (\\hat V/\\alpha)^{1/2}\\bigr\\} \\geq 1 - \\alpha.\n\\]\nThe displayed condition follows from Definition~10 if $\\liminf_{n \\to \\infty}(n/d_{\\mathrm{AVG}})\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) > 0$. Definition~10 alone guarantees only a scaled additive variance property.", "evidence": "The evidence is an exact counterexample and a derivation. For each $n \\geq 1$, let $Z_1, \\dots, Z_n$ be independent with $\\Pr(Z_i = 1) = 1/2$, and let $y_1(\\mathbf{z}) = z_1$ and $y_i(\\mathbf{z}) = 0$ for $i \\geq 2$. There is no interference, all outcomes lie in $[0, 1]$, and Assumption~1 holds with $k = 2$. Further, $d_{\\mathrm{AVG}} = d_{\\mathrm{MAX}} = \\lambda_{\\max} = 1$. Assumption~2 holds, as do the conditions of Proposition~8 and of part~A of Propositions~9 and~10. Substituting the outcomes into the definitions of $\\tau_{\\mathrm{EATE}}$ and $\\hat\\tau_{\\mathrm{HT}}$ and into the display on p.~689, I obtain\n\\[\n\\tau_{\\mathrm{EATE}} = \\frac{1}{n}, \\qquad \\hat\\tau_{\\mathrm{HT}} = \\frac{2Z_1}{n}, \\qquad \\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) = \\frac{1}{n^2}, \\qquad \\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}}) = \\frac{4Z_1}{n^2}.\n\\]\nThe expectation of the variance estimator is $2/n^2$, which exceeds the variance. Further, $\\tau_{\\mathrm{MSQ}} = 1/n$, and $n[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}}) - \\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})]$ takes the values $-1/n$ and $3/n$. Equation~(1) and Definition~10 therefore hold, and the three inflated estimators of Section~6.2 coincide with $\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}$. The interval $\\hat\\tau_{\\mathrm{HT}} \\pm (\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})/\\alpha)^{1/2}$ with $\\alpha = 1/20$ is the plug-in version of the 95\\% Chebyshev interval that uses the true variance.\n\\begin{audittable}{lllll}{The two outcomes of the experiment, each of probability $1/2$, and the interval with $\\alpha = 1/20$.}\nEvent & $\\hat\\tau_{\\mathrm{HT}}$ & $\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})$ & Interval & Contains $\\tau_{\\mathrm{EATE}} = 1/n$ \\\\\n$Z_1 = 0$ & $0$ & $0$ & $[0, 0]$ & no \\\\\n$Z_1 = 1$ & $2/n$ & $4/n^2$ & $2/n \\pm \\sqrt{80}/n$ & yes \\\\\n\\end{audittable}\nThe interval contains $\\tau_{\\mathrm{EATE}}$ with probability exactly $1/2$ for every $n$, and the same holds for every $\\alpha \\in (0, 1)$. The interval with the true variance in place of the estimate has half-width $\\sqrt{20}/n$ and always contains $\\tau_{\\mathrm{EATE}}$. In this example $(n/d_{\\mathrm{AVG}})\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}}) = 1/n$ tends to zero, and the ratio of the estimate to the variance is $0$ or $4$. Enumerating all assignments for $n = 1, \\dots, 8$ in exact arithmetic, I find the same values.\n\nFor the corrected statement, let $T$ be unbiased for $\\theta$ with variance $V > 0$, let $\\hat V \\geq 0$, and fix $\\eta \\in (0, 1)$. On the event $\\hat V \\geq (1 - \\eta)V$, the interval $T \\pm (\\hat V/\\alpha)^{1/2}$ misses $\\theta$ only if $|T - \\theta| > ((1 - \\eta)V/\\alpha)^{1/2}$. By Chebyshev's inequality, I have\n\\[\n\\Pr\\bigl\\{|T - \\theta| > (\\hat V/\\alpha)^{1/2}\\bigr\\} \\leq \\Pr\\bigl\\{\\hat V < (1 - \\eta)V\\bigr\\} + \\frac{\\alpha}{1 - \\eta}.\n\\]\nIf the first term on the right tends to zero for every $\\eta$, then letting $n \\to \\infty$ and afterwards $\\eta \\to 0$ shows that the limit inferior of the coverage is at least $1 - \\alpha$. If $(n/d_{\\mathrm{AVG}})V \\geq c > 0$ for all large $n$, then the event $\\hat V < (1 - \\eta)V$ is contained in the event $n d_{\\mathrm{AVG}}^{-1}(\\hat V - V) \\leq -\\eta c$, and Definition~10 implies that the probability of the latter tends to zero.", "problem": "Equation~(1) and Definition~10 both concern the difference between a variance estimator $\\hat V$ and $\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})$ after scaling by $n$ or by $n d_{\\mathrm{AVG}}^{-1}$. They say that this scaled difference is not negative in the limit. For a plug-in Chebyshev interval, it is sufficient that $\\hat V$ is unlikely to be much smaller than $\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})$ relative to that variance. The additive property does not provide this when the normalized variance $(n/d_{\\mathrm{AVG}})\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})$ tends to zero. The scaled difference can then vanish while the ratio $\\hat V/\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})$ is zero with a fixed positive probability. The expectation can equal the true variance when all individual treatment effects are zero. The claim that the scaled bias does not diminish is also too strong. The limit $\\tau_{\\mathrm{MSQ}}$ in (1) depends on $n$ and may tend to zero, and then the bias on the normalized scale does diminish. The paper prints no interval formula and no coverage theorem. What is affected is this sentence, and any reading of ``asymptotically conservative'' in Section~6.2 as a guarantee of coverage by itself.", "what_survives": "Equation~(1), Definition~10 and Propositions~8--10 stand as printed. They are statements about the variance estimator, and the example satisfies them. Part~B of Propositions~9 and~10 assumes that $\\tau_{\\mathrm{MSQ}}$ is bounded away from zero. In that case, the proofs in Supplement~A show that the inflated estimator exceeds the variance with probability tending to one, and this is enough for the coverage statement below. The example does not satisfy part~B, and the paragraph after Proposition~9 describes behaviour of this kind. The consistency results of Section~5, Proposition~11 and the simulations are not affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E002", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the six-line paragraph that opens the page. Red boxes mark the words 'are dependent' in 'the corresponding outcomes are dependent' and in 'two units' outcomes are dependent when they are interference dependent according to Definition 5'.", "caption": "Section 5.2.1, p. 684 (PDF p. 12), first paragraph. The two marked phrases state dependence as a fact.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E003.png"}], "formal_release": null, "formal_statements": [], "headline": "Two units that are interference dependent in the sense of Definition 5 may have dependent outcomes under a Bernoulli design, but need not; the proofs use only the converse, that units with $d_{ij}=0$ have independent outcomes.", "id": "P009-E003", "location": "Section 5.2.1, p. 684 (PDF p. 12), first paragraph", "pdf_page": 12, "printed_page": "684", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``The outcomes of any two units are independent under a Bernoulli design when the no-interference assumption holds. This is not the case when units interfere. A single treatment may then affect two or more units, and the corresponding outcomes \\hl{are dependent}. That is, two units' outcomes \\hl{are dependent} when they are interference dependent according to Definition 5.''", "corrected": "``A single treatment may then affect two or more units, and the corresponding outcomes \\hl{may be dependent}. That is, two units' outcomes \\hl{may be dependent} when they are interference dependent according to Definition 5.''", "intended_meaning": "Definition 5 sets $d_{ij}=1$ when the treatment of some unit can affect both unit $i$ and unit $j$, and the paragraph uses the definition to mark where dependence between outcomes can occur. Under a Bernoulli design, if $d_{ij}=0$, then the outcomes of units $i$ and $j$ are functions of disjoint sets of independent treatments, and they are independent. This is the direction that the variance bound behind Proposition 2 uses (Lemma A9 of Supplement A), and it is correct. The printed sentences state the other direction, which does not hold in general. For example, let $Z_1$, $Z_2$ and $Z_3$ be independent with probability $1/2$, and let $y_1(\\mathbf{z})=z_3$, $y_3(\\mathbf{z})=z_3$ and $y_2(\\mathbf{z})=z_2+z_3-2z_2z_3$, which is the sum of $z_2$ and $z_3$ modulo $2$. Since the treatment of unit 3 can change both $Y_1$ and $Y_2$, I have $d_{12}=1$. Each of the four values of $(Y_1,Y_2)$ has probability $1/4$, and the two outcomes are independent."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E003", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of text. They read: Compared to the setting with no interference, the limit of the variance estimator contains two additional terms. The term B 1 captures the consequences of direct interference between the units. That is, it captures whether unit i interferes with unit j directly, in which case xi-breve j i is not equal to 0. If there is no interference, there are no spillover effects, so xi i j of 1 = xi i j of 0 = 0, and B 1 is zero. The term B 2 captures the consequences of indirect interference between units, namely when a third unit interferes with both i and j. Red outlines surround the words in which case at the end of the third line and the relation xi-breve j i is not equal to 0 at the start of the fourth.", "caption": "Section 6.1, p. 691 (PDF p. 19). The outlines mark the words “in which case” and the relation $\\breve\\xi_{ji} \\neq 0$ that continues them on the next line.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T13", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T13"}, {"id": "P009-T14", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T14"}], "headline": "Direct interference from unit $i$ to unit $j$ does not imply $\\breve\\xi_{ji} \\neq 0$, because the spillover effects under particular assignments can cancel in expectation.", "id": "P009-E004", "location": "Section 6.1, p. 691 (PDF p. 19), paragraph beginning “Compared to the setting with no interference”, third sentence", "pdf_page": 19, "printed_page": "691", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T13", "P009-T14"], "text": {"as_printed": "The term $B_1$ captures the consequences of direct interference between the units. That is, it captures whether unit $i$ interferes with unit $j$ directly, \\hl{in which case $\\breve\\xi_{ji} \\neq 0$}. If there is no interference, there are no spillover effects, so $\\xi_{ij}(1) = \\xi_{ij}(0) = 0$, and $B_1$ is zero.", "corrected": "The term $B_1$ captures the contributions of expected direct spillover effects. Direct interference from unit $i$ to unit $j$ \\hl{can give} $\\breve\\xi_{ji} \\neq 0$, \\hl{although the spillover effects under particular assignments may average to zero}. If there is no interference, there are no spillover effects, so $\\xi_{ij}(1) = \\xi_{ij}(0) = 0$, and $B_1$ is zero.", "evidence": "The evidence is an exact counterexample. Suppose that three units have independent assignments of probability $1/2$, and let $y_2(\\mathbf{z}) = z_1(2z_3 - 1)$ and $y_1 = y_3 = 0$. Since changing $z_1$ changes $y_2$ by $2z_3 - 1$, which is $1$ or $-1$, unit~1 interferes with unit~2. For either value of $z_2$, I have $\\xi_{21}(z_2) = \\mathrm{E}[2Z_3 - 1] = 0$, and hence $\\breve\\xi_{21} = 0$. All outcomes are bounded by one, and Assumption~1 holds with $k = 2$. Enumerating the eight assignments in exact arithmetic, I obtain the same values.", "problem": "Unit $i$ interferes with unit $j$ if changing the treatment of $i$ changes the outcome of $j$ under at least one assignment (p.~680). The coefficient $\\breve\\xi_{ji}$ is an average of such changes over the design (p.~690). Changes that are nonzero under every assignment can have opposite signs and average to zero. Interference from $i$ to $j$ therefore does not imply that $\\breve\\xi_{ji}$ is nonzero. The converse statement in the next sentence, that without interference $B_1$ is zero, is correct.", "what_survives": "The definitions of $\\xi_{ij}(z)$ and $\\breve\\xi_{ij}$, Proposition~7 and the later results on variance estimation are unaffected: the proposition uses the averaged coefficients as defined. Only the explanatory sentence changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E004", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first nine lines of the page. A red box marks the words 'approaches zero for all pairs of units with d i j equals 0', which end the sentence 'In particular, complete randomization under bounded interference constitutes a setting where dependence is weak'.", "caption": "Section 5.3, p. 688 (PDF p. 16), first paragraph. The marked words state that the coefficient approaches zero for all pairs of units with $d_{ij}=0$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T9", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T9"}, {"id": "P009-T10", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T10"}], "headline": "Under complete randomization with bounded interference, $\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)$ approaches zero for all but a vanishing fraction of the pairs of units with $d_{ij}=0$, not for all of them; a fixed pair can keep the value $1/4$.", "id": "P009-E005", "location": "Section 5.3, p. 688 (PDF p. 16), first paragraph, sentence beginning “In particular, complete randomization under bounded interference”", "pdf_page": 16, "printed_page": "688", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T10", "P009-T9"], "text": {"as_printed": "``In particular, complete randomization under bounded interference constitutes a setting where dependence is weak: $\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)$ approaches zero \\hl{for all pairs of units} with $d_{ij}=0$.''\n\nLater in the same paragraph: ``Complete randomization under the conditions of Proposition 3 combines the two settings: $\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)$ might be nonnegligible asymptotically for some pairs with $d_{ij}=0$, but such pairs are rare. For all other pairs with $d_{ij}=0$, the pair-level mixing coefficient quickly approaches zero.''", "corrected": "``In particular, complete randomization under bounded interference constitutes a setting where dependence is weak \\hl{for most pairs}: $\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)$ approaches zero \\hl{uniformly for all but an asymptotically vanishing fraction of the pairs of units} with $d_{ij}=0$\\hl{; the exceptional pairs can retain nonvanishing dependence}.''\n\nIn precise form: suppose that the design is complete randomization and that $d_{\\mathrm{AVG}}\\le D$ for all large $n$, and let $t_n=\\lfloor n^{1/4}\\rfloor$. Then, \\hl{for all sufficiently large $n$}, all pairs of units with $d_{ij}=0$, except a share of at most $2Dn/\\{t_n(n-D)\\}$ of them, satisfy\n\\[\n\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)\\le\\frac{3(2t_n)(2t_n-1)}{2(n-2t_n+1)} .\n\\]", "evidence": "The evidence is an exact counterexample for an infinite family of sample sizes and a derivation of the corrected statement.\n\nFor the counterexample, let $r\\ge1$, $n=4r+2$ and $h=n/2$, so that $h$ is odd, and let the design be complete randomization with $h$ treated units. Let the units be divided into two blocks $A$ and $B$ of $h$ units each such that $1\\in A$ and $2\\in B$, and let\n\\[\ny_1(\\mathbf{z})=\\frac{1}{h}\\sum_{\\ell\\in A}z_\\ell,\\qquad y_2(\\mathbf{z})=\\frac{1}{h}\\sum_{\\ell\\in B}z_\\ell,\\qquad y_j(\\mathbf{z})=z_j\\quad(3\\le j\\le n).\n\\]\nThe potential outcomes lie in $[0,1]$. Unit 1 is affected by the treatments in $A$, unit 2 by those in $B$, and every other unit by its own treatment only. Then $d_{12}=0$, $d_{\\mathrm{AVG}}=3-4/n$ and $C_1=2-2/n$, and no unit interferes with more than two units, itself included. Let $T_A$ and $T_B=h-T_A$ be the numbers of treated units in the two blocks. The event $\\{T_A\\ge(h+1)/2\\}$ is determined by $\\widetilde{\\mathbf{Z}}_1$, the event $\\{T_B\\le(h-1)/2\\}$ is determined by $\\widetilde{\\mathbf{Z}}_2$, and the two events coincide. Since the map $\\mathbf{z}\\mapsto\\mathbf{1}-\\mathbf{z}$ preserves the design and $h$ is odd, their common probability is $1/2$. By the definition of the mixing coefficient, I have\n\\[\n\\alpha(\\widetilde{\\mathbf{Z}}_1,\\widetilde{\\mathbf{Z}}_2)\\ge\\Bigl|\\frac12-\\frac12\\cdot\\frac12\\Bigr|=\\frac14 ,\n\\]\nwhich is the maximum value of the coefficient (p.~687). An enumeration of all assignments for $n=6,10,14,18$ in exact arithmetic agrees with these values.\n\nFor the corrected statement, suppose that $d_{\\mathrm{AVG}}\\le D$ for all large $n$, and let $a_i=\\sum_{\\ell=1}^nI_{\\ell i}$ be the number of units that interfere with unit $i$. Then $\\sum_{i=1}^na_i=nC_1\\le nd_{\\mathrm{AVG}}\\le nD$. Let $t_n=\\lfloor n^{1/4}\\rfloor$. At most $nD/t_n$ units have $a_i>t_n$, and at most $2Dn^2/t_n$ ordered pairs contain such a unit. Since there are at least $n^2-nD$ pairs such that $d_{ij}=0$, the excluded share tends to zero. For every other pair such that $d_{ij}=0$, the two sets of interfering units are disjoint and have $K\\le2t_n$ elements together. A coupling of $K$ draws without replacement with $K$ independent draws shows that the two laws of these $K$ treatments differ by at most $\\varepsilon_K=K(K-1)/\\{2(n-K+1)\\}$ on every event. Under the independent law, events on the two sets are independent, and I have\n\\[\n\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)\\le3\\varepsilon_K\\le\\frac{3(2t_n)(2t_n-1)}{2(n-2t_n+1)}=\\mathcal{O}(n^{-1/2})\n\\]\nuniformly over these pairs.", "problem": "Corollary 1 on p.~685 defines bounded interference as $d_{\\mathrm{AVG}}=\\mathcal{O}(1)$, which bounds an average. Such a bound allows a few units that are each affected by a large share of the sample, and under complete randomization the numbers of treated units in two large disjoint sets remain dependent. The sentence claims too much: the mixing coefficient need not approach zero for every pair of units with $d_{ij}=0$. It does approach zero, uniformly, outside a set of pairs whose share tends to zero. The paper describes this situation later in the same paragraph, for the conditions of Proposition 3, and the same description applies under bounded interference. The error is in this explanatory sentence only.", "what_survives": "Proposition 3, Corollary 1 and Proposition 5 do not use the sentence. The proof of Proposition 3 treats separately the pairs whose sets of interfering units are large, Corollary 1 follows from Proposition 3 and $C_1\\le d_{\\mathrm{AVG}}$, and Proposition 5 assumes the averaged conditions of Assumptions 5 and 6. In the counterexample, both estimators are within $6/n$ of EATE on every assignment, so the example does not contradict root-$n$ consistency. The correction weakens only the explanatory statement about pairs. The statement for all pairs would hold under a uniform bound on the number of units that interfere with each unit, which is an additional assumption; a bound on the number of units that each unit interferes with is not enough, as the counterexample shows."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E005", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of text, Definition 9 with its display of alpha EXT and alpha INT, and the line after the display. Red boxes mark the words 'maximum' at the end of one line and 'values of q and s' at the start of the next.", "caption": "Definition 9, p. 687 (PDF p. 15). The marked words ask for the maximum values of $q$ and $s$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T26", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T26"}, {"id": "P009-T45", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T45"}, {"id": "P009-T46", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T46"}], "headline": "In Definition 9, the “maximum values of $q$ and $s$” are taken with the constant $k$ of Assumption 1 held fixed, and they are infinite when the corresponding outcomes are bounded by that same $k$, in which case the exponents $(q-2)/q$ and $(s-1)/s$ are read as $1$.", "id": "P009-E006", "location": "Definition 9, p. 687 (PDF p. 15); Supplement A, proofs of Lemmas A20, A22 and A23, pp. 21 and 23", "pdf_page": 15, "printed_page": "687", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P009-T26", "P009-T45", "P009-T46"], "text": {"as_printed": "``Definition 9 (External and internal average mixing coefficients). For the \\hl{maximum values of $q$ and $s$} such that Assumptions 1B and 1C hold, let\n\\[\n\\alpha_{\\mathrm{EXT}}=\\frac{1}{n}\\sum_{i=1}^n\\sum_{j=1}^n(1-d_{ij})\\bigl[\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)\\bigr]^{\\frac{q-2}{q}}\\quad\\text{and}\\quad\\alpha_{\\mathrm{INT}}=\\sum_{i=1}^n\\bigl[\\alpha(Z_i,\\widetilde{\\mathbf{Z}}_{-i})\\bigr]^{\\frac{s-1}{s}},\n\\]\nwhere $0^0$ is defined as zero to accommodate the cases $q=2$ and $s=1$.''", "corrected": "``Definition 9 (External and internal average mixing coefficients). \\hl{Let $k$ be a constant for which Assumption 1 holds, and let $q$ and $s$ be the largest values, possibly infinite, such that Assumptions 1B and 1C hold with this $k$.} Let\n\\[\n\\alpha_{\\mathrm{EXT}}=\\frac{1}{n}\\sum_{i=1}^n\\sum_{j=1}^n(1-d_{ij})\\bigl[\\alpha(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j)\\bigr]^{\\frac{q-2}{q}}\\quad\\text{and}\\quad\\alpha_{\\mathrm{INT}}=\\sum_{i=1}^n\\bigl[\\alpha(Z_i,\\widetilde{\\mathbf{Z}}_{-i})\\bigr]^{\\frac{s-1}{s}},\n\\]\nwhere $0^0$ is defined as zero to accommodate the cases $q=2$ and $s=1$\\hl{, and where $(q-2)/q$ and $(s-1)/s$ are read as $1$ when $q$ or $s$ is infinite}.''\n\nThe same convention applies where Supplement A sets $s$ or $q$ to its maximum value, in the proofs of Lemmas A20, A22 and A23. An alternative is to fix any admissible $q\\ge2$ and $s\\ge1$ with a common finite $k$, to define the coefficients for those exponents, and to state Assumptions 5 and 6 and the rates with the same choices.", "intended_meaning": "Assumption 1 asks for constants $k<\\infty$, $q\\ge2$ and $s\\ge1$ such that the moment bounds of Assumptions 1B and 1C hold for all units in the sequence of samples, and Definition 9 then takes the maximum values of $q$ and $s$. If this is read as a request for the largest finite exponents, the definition selects nothing when the potential outcomes are bounded: if every potential outcome equals $1$ and $k=2$, then every finite $q\\ge2$ and $s\\ge1$ is admissible. The discussion after Assumption 1 on p.~682 covers this case: bounded potential outcomes correspond ``to the case where Assumption 1 holds as $q\\to\\infty$ and $s\\to\\infty$''. The intended reading keeps the constant $k$ of Assumption 1 fixed, takes the largest admissible exponents, and allows the value infinity, with $(q-2)/q$ and $(s-1)/s$ read as their limit $1$.\n\nWith this reading the definition is complete. For a fixed $k$, a finite supremum of the admissible exponents is attained, because at each sample size the outcomes take finitely many values and each moment is continuous in the exponent. If the admissible exponents are unbounded, then the outcomes are bounded by $k$, and the bounds of Lemmas A20 and A22 hold with exponent $1$ in the limit. A finite supremum that is not attained can occur only if the constant $k$ is allowed to change with the exponent, and Assumption 1 does not state that. The rates of Propositions 5 and 6 are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E006", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Remark 2. A red box marks the formula 'alpha EXT tends to n over 4' in the sentence 'The external mixing coefficient would not be zero in this case; in fact, alpha EXT tends to n over 4'.", "caption": "Remark 2, p. 688 (PDF p. 16). The marked statement gives the external mixing coefficient as tending to $n/4$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T26", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T26"}], "headline": "In Remark 2, the external mixing coefficient of the design that assigns the same treatment to all units is $(n-1)p(1-p)$, where $p$ is the probability of treatment; it is asymptotically $n/4$ only when $p=1/2$.", "id": "P009-E007", "location": "Remark 2, p. 688 (PDF p. 16), third and fourth sentences", "pdf_page": 16, "printed_page": "688", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T26"], "text": {"as_printed": "``Consider a design that assigns the same treatment to all units: $Z_1=Z_2=\\cdots=Z_n$. The external mixing coefficient would not be zero in this case; in fact, $\\hl{\\alpha_{\\mathrm{EXT}}\\to n/4}$.''", "corrected": "``Consider a design that assigns the same treatment to all units\\hl{, with probability $p$ of treatment}: $Z_1=Z_2=\\cdots=Z_n$. The external mixing coefficient would not be zero in this case; in fact, $\\hl{\\alpha_{\\mathrm{EXT}}=(n-1)p(1-p)}$\\hl{, which is of order $n$; for $p=1/2$, $\\alpha_{\\mathrm{EXT}}\\sim n/4$}.''\n\nThe statement holds when no units interfere and Assumption 1 holds, with $\\alpha_{\\mathrm{EXT}}$ as in Definition 9.", "evidence": "The evidence is a derivation from Definition 9. If no units interfere, then $d_{ij}=0$ for $i\\ne j$, and $\\widetilde{\\mathbf{Z}}_i$ generates the same events as $Z_i$. Let $Z_i=T$ for all $i$, where $\\Pr(T=1)=p$. The only events are the empty event, $\\{T=0\\}$, $\\{T=1\\}$ and the whole space. Enumerating the pairs of events, I find $\\alpha(Z_i,Z_j)=p-p^2=p(1-p)$ for $i\\ne j$. With no interference, each unit has two potential outcomes, and Assumptions 1A and 1B imply that $k^{-1}|y_i(z)|^2\\le\\mathbb{E}[|Y_i|^2]\\le k^2$. The outcomes are bounded by $k^{3/2}$, and with this constant in place of $k$, Assumption 1B holds for every $q$. The maximum $q$ of Definition 9 is then infinite, and the exponent $(q-2)/q$ is $1$ (see \\fid{P009-E006}). Summing over the $n(n-1)$ ordered pairs and dividing by $n$, I obtain $\\alpha_{\\mathrm{EXT}}=(n-1)p(1-p)$. For $p=1/3$ this is $2(n-1)/9$, and for $p=1/2$ it is $(n-1)/4$.", "problem": "The remark does not state the probability $p$ with which the common treatment is assigned, and the constant $1/4$ is correct only for $p=1/2$. In general the coefficient is $(n-1)p(1-p)$. The arrow to a quantity that depends on $n$ is also informal; asymptotic equivalence is meant. The point of the remark is unaffected. Since Assumption 1A keeps $p$ away from $0$ and $1$, the coefficient $\\alpha_{\\mathrm{EXT}}$ is of order $n$ and Assumption 5 fails.", "what_survives": "The order $n$ of the coefficient, the failure of Assumption 5 in this example for every $p$ that Assumption 1A allows, and Proposition 5 are unaffected. Only the constant changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E007", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of two paragraphs. The first reads: While any of the three terms of the limit in Proposition 7 can dominate the others asymptotically, B 2 will generally be the one we need to worry about. To see this, observe that tau MSQ is negligible on a normalized scale as long as the average interference dependence is not bounded asymptotically: d AVG tends to infinity. The amount of direct interference dependence is given by C 1, so B 1 = O(d AVG to the minus 1 times C 1). The amount of indirect dependence is given by d AVG, so B 2 = O(1). Hence, B 2 is the dominating term whenever d AVG dominates C 1, which, as we noted in Section 5.2, is the case whenever the interference is not too tightly clustered. A red outline surrounds the words Hence, B 2 is the dominating term whenever d AVG dominates C 1. The second paragraph begins: The key insight here is that when there is interference, these two additional terms are generally nonzero, and their sum may be both positive and negative.", "caption": "Section 6.1, p. 691 (PDF p. 19). The outline marks the clause that draws dominance of $B_2$ from the two upper bounds stated just before it.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T11", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T11"}, {"id": "P009-T12", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T12"}], "headline": "When $d_{\\mathrm{AVG}}$ dominates $C_1$, the term $B_2$ may dominate but need not: the orders $B_1 = \\mathcal{O}(d_{\\mathrm{AVG}}^{-1}C_1)$ and $B_2 = \\mathcal{O}(1)$ are upper bounds, and $B_2$ can be zero.", "id": "P009-E008", "location": "Section 6.1, p. 691 (PDF p. 19), paragraph beginning “While any of the three terms”, last sentence", "pdf_page": 19, "printed_page": "691", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T11", "P009-T12"], "text": {"as_printed": "The amount of direct interference dependence is given by $C_1$, so $B_1 = \\mathcal{O}(d_{\\mathrm{AVG}}^{-1}C_1)$. The amount of indirect dependence is given by $d_{\\mathrm{AVG}}$, so $B_2 = \\mathcal{O}(1)$. Hence, $B_2$ \\hl{is the dominating term whenever} $d_{\\mathrm{AVG}}$ dominates $C_1$, which, as we noted in Section~5.2, is the case whenever the interference is not too tightly clustered.", "corrected": "The amount of direct interference dependence is given by $C_1$, so $B_1 = \\mathcal{O}(d_{\\mathrm{AVG}}^{-1}C_1)$. The amount of indirect dependence is given by $d_{\\mathrm{AVG}}$, so $B_2 = \\mathcal{O}(1)$. Hence, when $d_{\\mathrm{AVG}}/C_1$ tends to infinity, $\\tau_{\\mathrm{MSQ}}/d_{\\mathrm{AVG}}$ and $B_1$ vanish on this scale, whereas $B_2$ is $\\mathcal{O}(1)$ and \\hl{may dominate}.\n\nThe remark that $d_{\\mathrm{AVG}}$ dominates $C_1$ whenever the interference is not too tightly clustered stays as printed. If guaranteed dominance is wanted, a lower bound on the magnitude of $B_2$ must be assumed: it suffices that $|B_2| \\geq c > 0$ for all large $n$, or more generally that $C_1/(d_{\\mathrm{AVG}}|B_2|) \\to 0$.", "evidence": "The evidence is an exact counterexample. For $n = m^2$ with $m \\geq 2$, I split the units into $m$ blocks of $m$ units and label the units of a block $0, 1, \\dots, m - 1$. All assignments are independent with probability $1/2$, and within each block\n\\[\ny_0(\\mathbf{z}) = z_0 + z_1, \\qquad y_i(\\mathbf{z}) = z_i + z_0 \\quad (1 \\leq i \\leq m - 1).\n\\]\nAll outcomes lie in $[0, 2]$, and Assumption~1 holds with $k = 2$ and $q = 4$. Unit~0 of a block interferes with every unit of the block, unit~1 with itself and unit~0, and each other unit only with itself. Counting these, I find $C_1 = 2$. Since every pair of units in a block is affected by the treatment of unit~0, I have $d_{\\mathrm{AVG}} = m = n^{1/2}$. In particular, $d_{\\mathrm{AVG}}/C_1$ tends to infinity and Assumption~2 holds.\n\nEvery effect of a unit's own treatment equals one, and $\\tau_{\\mathrm{MSQ}} = 1$. The spillover effects do not depend on the unit's own treatment, which implies that $\\xi_{ij}(1) - \\xi_{ij}(0) = 0$. The product $\\breve\\xi_{ij}\\breve\\xi_{ji}$ is nonzero only for the two ordered pairs formed by units~0 and~1 of a block, where it equals one. Summing over the blocks, I find $B_1 = 2m/(n d_{\\mathrm{AVG}}) = 2/m^2$. Each outcome is a sum of treatments, $y_i(\\mathbf{z}) = \\sum_k A_{ik}z_k$ with $A_{ik} \\in \\{0, 1\\}$. For $i \\neq j$, I have\n\\[\n\\operatorname{Cov}(Y_i, Y_j \\mid Z_i = a, Z_j = b) = \\frac{1}{4}\\sum_{k \\notin \\{i, j\\}} A_{ik}A_{jk},\n\\]\nwhich does not depend on $a$ or $b$. The alternating sum over $a$ and $b$ in the definition of $B_2$ is therefore zero for every pair, and $B_2 = 0$. Note that the covariances themselves are not all zero.\n\\begin{audittable}{rrrrrr}{The terms of the limit in Proposition~7 in the example, for $n = m^2$ and $m = 2, \\dots, 8$.}\n$n$ & $C_1$ & $d_{\\mathrm{AVG}}$ & $\\tau_{\\mathrm{MSQ}}/d_{\\mathrm{AVG}}$ & $B_1$ & $B_2$ \\\\\n$4$ & $2$ & $2$ & $1/2$ & $1/2$ & $0$ \\\\\n$9$ & $2$ & $3$ & $1/3$ & $2/9$ & $0$ \\\\\n$16$ & $2$ & $4$ & $1/4$ & $1/8$ & $0$ \\\\\n$25$ & $2$ & $5$ & $1/5$ & $2/25$ & $0$ \\\\\n$36$ & $2$ & $6$ & $1/6$ & $1/18$ & $0$ \\\\\n$49$ & $2$ & $7$ & $1/7$ & $2/49$ & $0$ \\\\\n$64$ & $2$ & $8$ & $1/8$ & $1/32$ & $0$ \\\\\n\\end{audittable}\nFor $m \\geq 3$ the largest of the three terms is $\\tau_{\\mathrm{MSQ}}/d_{\\mathrm{AVG}} = n^{-1/2}$, and $B_2$ is zero for every $m$. I enumerated in exact arithmetic all assignments of a block of up to eight units, and of the whole sample for $n = 9$ and for $n = 10$ (with one isolated unit). The results agree with these values and with the identity of Proposition~7 for the expected bias. A sample of any size $n$ can be built in the same way from blocks of $\\lfloor n^{1/2} \\rfloor$ units and isolated units.", "problem": "The two orders in the passage are upper bounds. They show that $\\tau_{\\mathrm{MSQ}}/d_{\\mathrm{AVG}}$ and $B_1$ become small when $d_{\\mathrm{AVG}}/C_1$ tends to infinity, and they show nothing about how large $B_2$ is. The term $B_2$ is a signed sum of conditional covariances, and its contributions can cancel. Without a lower bound on $|B_2|$ the conclusion that $B_2$ is the dominating term does not follow. It fails in the example below, where $B_2 = 0$ while the other two terms of the limit in Proposition~7 are positive.", "what_survives": "Proposition~7 and the two upper bounds are correct as printed, and no later result uses the comparison: Propositions~8--10 rest on variance bounds and concentration. The warning that the conventional variance estimator may be anticonservative under interference stands. What changes is that $B_2$ is the term that may be of order one, not the term that must be largest."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E008", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first display of the proof. The first line writes the variance of mu-hat_z as one over n squared times the double sum of covariances divided by Pr(Z_i = z) Pr(Z_j = z). A red box marks the second line, labelled Assumption 1A: at most k squared over n squared times the double sum of the covariances, without absolute values.", "caption": "Supplement A, p. 8: the opening of the proof of Lemma A9. The marked line replaces the weights $1/[\\Pr(Z_i=z)\\Pr(Z_j=z)]$ by $k^2$ in a sum of signed covariances.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E009.png"}, {"alt": "Crop showing Lemma A22 and the line 'Proof. Assumption 1A gives'. A red box marks the right side of the display: at most k squared over n squared times the double sum of the covariances of 1[Z_i = z] Y_i and 1[Z_j = z] Y_j.", "caption": "Supplement A, p. 22: Lemma A22 and the first line of its proof, with the same step marked.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E009-A22.png"}, {"alt": "Crop of a three-line display for the variance of n-hat_z over n. A red box marks the right side of the first line: at most k squared over n squared times the double sum of the covariances of 1[Z_i = z] and 1[Z_j = z]. A second red box marks the start of the third line: k squared over n squared times the sum over i and over j not equal to i of alpha(Z_i, Z_j).", "caption": "Supplement A, p. 23: the first display of the proof of Lemma A23. The first marked term is the same step for the indicators; the second marked term is the sum over $j\\ne i$, which leaves out the diagonal terms.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E009-A23.png"}, {"alt": "Crop of one display from the proof of Lemma A25. A red box marks its right side: at most k to the fourth times the double sum of the covariances of 1[Z_i = z] Y_i squared and 1[Z_j = z] Y_j squared.", "caption": "Supplement A, p. 25: the step in the proof of Lemma A25, for squared outcomes and with $k^4$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E009-A25.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T3", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T3"}, {"id": "P009-T4", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T4"}, {"id": "P009-T16", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T16"}, {"id": "P009-T29", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T29"}, {"id": "P009-T41", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T41"}, {"id": "P009-T46", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T46"}, {"id": "P009-T47", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T47"}], "headline": "Four proofs in Supplement A bound a weighted sum of signed covariances by replacing each weight with its maximum, which is not a valid inequality; with absolute covariances, and with the diagonal terms kept in Lemma A23, the same final bounds follow.", "id": "P009-E009", "location": "Supplement A, proof of Lemma A9, p. 8, first display, step labelled “Assumption 1A”; the same step in the proofs of Lemma A22 (p. 22), Lemma A23 (p. 23) and Lemma A25 (p. 25)", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T16", "P009-T29", "P009-T3", "P009-T4", "P009-T41", "P009-T46", "P009-T47"], "text": {"as_printed": "Proof of Lemma A9, p.~8; the second line is labelled ``Assumption 1A'':\n\\[\n\\begin{aligned}\n\\operatorname{Var}(\\hat\\mu_z)&=\\operatorname{Var}\\bigg(\\frac1n\\sum_{i=1}^n\\frac{\\mathbf{1}[Z_i=z]Y_i}{\\Pr(Z_i=z)}\\bigg)=\\frac1{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\frac{\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i,\\mathbf{1}[Z_j=z]Y_j\\big)}{\\Pr(Z_i=z)\\Pr(Z_j=z)}\\\\\n&\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i,\\mathbf{1}[Z_j=z]Y_j\\big)}.\n\\end{aligned}\n\\]\nProof of Lemma A22, p.~22: ``Assumption 1A gives:''\n\\[\n\\operatorname{Var}(\\hat\\mu_z)=\\operatorname{Var}\\bigg(\\frac1n\\sum_{i=1}^n\\frac{\\mathbf{1}[Z_i=z]Y_i}{\\Pr(Z_i=z)}\\bigg)\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i,\\mathbf{1}[Z_j=z]Y_j\\big)}\n\\]\nProof of Lemma A23, p.~23: ``First, consider the variance of $\\hat n_z/n$. Assumption 1A gives:''\n\\[\n\\begin{aligned}\n\\operatorname{Var}(\\hat n_z/n)&=\\operatorname{Var}\\bigg(\\frac1n\\sum_{i=1}^n\\frac{\\mathbf{1}[Z_i=z]}{\\Pr(Z_i=z)}\\bigg)\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z],\\mathbf{1}[Z_j=z]\\big)}\\\\\n&=\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\Big[\\Pr(Z_i=z,Z_j=z)-\\Pr(Z_i=z)\\Pr(Z_j=z)\\Big]\\\\\n&\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{\\hl{j\\ne i}}\\alpha\\big(Z_i,Z_j\\big)\\le\\frac{k^2d_{\\mathrm{AVG}}}{n}+\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n(1-d_{ij})\\alpha\\big(Z_i,Z_j\\big),\n\\end{aligned}\n\\]\nProof of Lemma A25, p.~25: ``Given Assumption 1A, for $z\\in\\{0,1\\}$:''\n\\[\n\\operatorname{Var}\\bigg(\\sum_{i=1}^n\\frac{\\mathbf{1}[Z_i=z]Y_i^2}{[\\Pr(Z_i=z)]^2}\\bigg)\\le k^4\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i^2,\\mathbf{1}[Z_j=z]Y_j^2\\big)}.\n\\]", "corrected": "The first inequality of each proof is replaced as follows, and the later steps are read with absolute covariances.\n\nLemma A9 (Bernoulli design, Assumption 1):\n\\[\n\\begin{aligned}\n\\operatorname{Var}(\\hat\\mu_z)&\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\big|\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i,\\mathbf{1}[Z_j=z]Y_j\\big)\\big|}\\\\\n&=\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^nd_{ij}\\big|\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i,\\mathbf{1}[Z_j=z]Y_j\\big)\\big|\\le k^4n^{-1}d_{\\mathrm{AVG}}.\n\\end{aligned}\n\\]\nLemma A22 (arbitrary designs, Assumption 1, with $q$ the maximum value that satisfies Assumption 1B):\n\\[\n\\begin{aligned}\n\\operatorname{Var}(\\hat\\mu_z)&\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\big|\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i,\\mathbf{1}[Z_j=z]Y_j\\big)\\big|}\\\\\n&\\le k^4n^{-1}d_{\\mathrm{AVG}}+\\frac{10k^4}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n(1-d_{ij})\\,\\alpha\\big(\\widetilde{\\mathbf{Z}}_i,\\widetilde{\\mathbf{Z}}_j\\big)^{\\frac{q-2}{q}}=k^4n^{-1}d_{\\mathrm{AVG}}+10k^4n^{-1}\\alpha_{\\mathrm{EXT}}.\n\\end{aligned}\n\\]\nLemma A23 (arbitrary designs, Assumption 1):\n\\[\n\\begin{aligned}\n\\operatorname{Var}(\\hat n_z/n)&\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\big|\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z],\\mathbf{1}[Z_j=z]\\big)\\big|}\\\\\n&\\le\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{\\hl{j=1}}^n\\alpha\\big(Z_i,Z_j\\big)\\le\\frac{k^2d_{\\mathrm{AVG}}}{n}+\\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n(1-d_{ij})\\alpha\\big(Z_i,Z_j\\big).\n\\end{aligned}\n\\]\nThe diagonal terms have $d_{ii}=1$ and are covered by the term $k^2d_{\\mathrm{AVG}}/n$, since $\\alpha(Z_i,Z_j)\\le1/4$.\n\nLemma A25 (Bernoulli design, Assumption 1 with $q\\ge4$), for $z\\in\\{0,1\\}$:\n\\[\n\\begin{aligned}\n\\operatorname{Var}\\bigg(\\sum_{i=1}^n\\frac{\\mathbf{1}[Z_i=z]Y_i^2}{[\\Pr(Z_i=z)]^2}\\bigg)&\\le k^4\\sum_{i=1}^n\\sum_{j=1}^n\\hl{\\big|\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i^2,\\mathbf{1}[Z_j=z]Y_j^2\\big)\\big|}\\\\\n&=k^4\\sum_{i=1}^n\\sum_{j=1}^nd_{ij}\\big|\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i^2,\\mathbf{1}[Z_j=z]Y_j^2\\big)\\big|\\le k^8nd_{\\mathrm{AVG}}.\n\\end{aligned}\n\\]", "evidence": "The evidence is three exact counterexamples with two units and a derivation of the repaired bounds. In each example $z=1$, $k=4$, the treatment probabilities of the two units are $1/4$ and $3/4$, and every potential outcome is at most one in absolute value.\n\\begin{itemize}\n\\item Lemmas A9 and A22. Let $Z_1$ and $Z_2$ be independent, and let $y_1(\\mathbf{z})=z_2$ and $y_2(\\mathbf{z})=-z_1$. Then $\\mathbf{1}[Z_1=1]Y_1=Z_1Z_2=-\\mathbf{1}[Z_2=1]Y_2$. The four covariances sum to zero, and $\\hat\\mu_1=(4/3)Z_1Z_2$ has variance $13/48$.\n\\item Lemma A23. Let $\\Pr(\\mathbf{Z}=(1,0))=1/4$ and $\\Pr(\\mathbf{Z}=(0,1))=3/4$, and let all potential outcomes equal one. Then $Z_1+Z_2=1$, the covariances of the indicators sum to zero, and $\\hat n_1/n$ takes the values $2$ and $2/3$ and has variance $1/3$.\n\\item Lemma A25. Let $Z_1$ and $Z_2$ be independent, and let $y_1(\\mathbf{z})=1$ and $y_2(\\mathbf{z})=1-z_1$. The covariances sum to $39/256$. The printed bound is then $39$, and the variance on the left is $367/9$.\n\\end{itemize}\n\\begin{audittable}{lrrr}{The left side of each printed inequality in the two-unit examples, the printed bound, and the bound with absolute covariances.}\nProof & Variance & Printed bound & Bound with absolute covariances \\\\\nLemmas A9 and A22 & $13/48$ & $0$ & $39/16$ \\\\\nLemma A23 & $1/3$ & $0$ & $3$ \\\\\nLemma A25 & $367/9$ & $39$ & $183$ \\\\\n\\end{audittable}\nThe failure is not special to $n=2$. With $m$ independent copies of an example and $n=2m$, the three variances are $13/(48m)$, $1/(3m)$ and $367m/9$, the printed bounds are $0$, $0$ and $39m$, and $d_{\\mathrm{AVG}}\\le2$ and $\\alpha_{\\mathrm{EXT}}\\le1$.\n\nThe repair rests on the following inequality. For random variables $X_1,\\dots,X_n$ with finite variances and constants $0<w_i\\le K$, I have\n\\[\n\\operatorname{Var}\\Big(\\sum_{i=1}^nw_iX_i\\Big)=\\sum_{i=1}^n\\sum_{j=1}^nw_iw_j\\operatorname{Cov}(X_i,X_j)\\le K^2\\sum_{i=1}^n\\sum_{j=1}^n\\big|\\operatorname{Cov}(X_i,X_j)\\big|.\n\\]\nThe later steps of the four proofs also hold with absolute covariances. By the Cauchy--Schwarz argument in the proof of Lemma A3, I have $|\\operatorname{Cov}(\\mathbf{1}[Z_i=z]Y_i^a,\\mathbf{1}[Z_j=z]Y_j^a)|\\le k^{2a}$ for $q\\ge2a$. Under a Bernoulli design the covariance is zero when $d_{ij}=0$. Lemma A21 is stated for the absolute covariance. Finally, $|\\operatorname{Cov}(\\mathbf{1}[Z_i=z],\\mathbf{1}[Z_j=z])|\\le\\alpha(Z_i,Z_j)\\le1/4$ for every pair, including $i=j$.", "problem": "By Assumption 1A, each weight $1/[\\Pr(Z_i=z)\\Pr(Z_j=z)]$ lies between $1$ and $k^2$. When the treatment probabilities differ across units the weights differ across pairs, and the covariances can be negative. Replacing every weight by $k^2$ then does not give an upper bound: in the unweighted sum, a negative covariance that carries a small weight can cancel a positive covariance that carries a large one. The proof of Lemma A23 applies the same step to the indicators $\\mathbf{1}[Z_i=z]$, and the proof of Lemma A25 applies it to the squared outcomes, with weights at most $k^4$. The third line in the proof of Lemma A23 also sums over $j\\ne i$, which leaves out the variances of the indicators; that index is the subject of \\fid{P009-E010}. The step is valid in the proofs of Lemmas A15 and A18. There every unit has the same treatment probability, and the weight is a common factor of a sum that is itself a variance.", "what_survives": "The final bounds of the four lemmas hold under the same assumptions and with the same constants: $k^4n^{-1}d_{\\mathrm{AVG}}$ in Lemma A9, $k^4n^{-1}d_{\\mathrm{AVG}}+10k^4n^{-1}\\alpha_{\\mathrm{EXT}}$ in Lemma A22, $k^2n^{-1}d_{\\mathrm{AVG}}+k^2n^{-1}\\alpha_{\\mathrm{EXT}}$ for $\\operatorname{Var}(\\hat n_z/n)$ in Lemma A23, and $4k^8n^{-1}d_{\\mathrm{AVG}}$ for the variance of the normalized variance estimator in Lemma A25. The results that rest on these lemmas are unchanged: Propositions 2, 5, 6 and 7, Corollaries A3--A5 and, through them, Propositions 8--10. Lemmas A15 and A18 need no change. Only the printed derivations change."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E009", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the opening of a proof. The first line reads: Proof. First, consider the variance of n-hat z over n. Assumption 1A gives. A display of three lines follows. First line: Var of n-hat z over n equals the variance of one over n times the sum over i of the indicator of Z i = z divided by Pr(Z i = z), which is at most k squared over n squared times the double sum over i and j from 1 to n of the covariance of the indicators of Z i = z and Z j = z. Second line: equals k squared over n squared times the double sum over i and j from 1 to n of Pr(Z i = z, Z j = z) minus Pr(Z i = z) Pr(Z j = z). Third line: is at most k squared over n squared times the sum over i from 1 to n and the sum over j not equal to i of alpha(Z i, Z j), which is at most k squared d AVG over n plus k squared over n squared times the double sum over i and j from 1 to n of (1 minus d i j) alpha(Z i, Z j). A red outline surrounds the summation sign with the index j not equal to i. The text below reads: where the second to last inequality follows from the definition of the alpha-mixing coefficient. The last inequality follows from 0 at most alpha(Z i, Z j) at most 0.25. Note that I i i = 1 by definition. It then follows that the sub-sigma-algebra generated by Z-tilde i is finer than the algebra generated by Z i, and we have alpha(Z i, Z j) at most alpha(Z-tilde i, Z-tilde j). The crop ends with the first line of the next display.", "caption": "Supplement A, p. 23, first display of the proof of Lemma A23. The outline marks the sum over $j \\neq i$ in the third line; the sums before and after it run over all $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T4", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T4"}, {"id": "P009-T15", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T15"}, {"id": "P009-T16", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T16"}, {"id": "P009-T46", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T46"}], "headline": "The middle sum in the first display of the proof of Lemma A23 should run over all $j$ and not over $j \\neq i$: the terms with $j = i$ are variances of assignment indicators and are positive.", "id": "P009-E010", "location": "Supplement A, p. 23 (PDF p. 23 of the supplement), proof of Lemma A23, first display, third line", "pdf_page": 23, "printed_page": "23", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T15", "P009-T16", "P009-T4", "P009-T46"], "text": {"as_printed": "\\[\n\\begin{aligned}\n&= \\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n \\Bigl[\\Pr(Z_i = z, Z_j = z) - \\Pr(Z_i = z)\\Pr(Z_j = z)\\Bigr] \\\\\n&\\leq \\frac{k^2}{n^2}\\sum_{i=1}^n \\hl{\\sum_{j \\neq i}} \\alpha(Z_i, Z_j) \\leq \\frac{k^2 d_{\\mathrm{AVG}}}{n} + \\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n (1 - d_{ij})\\alpha(Z_i, Z_j),\n\\end{aligned}\n\\]", "corrected": "\\[\n\\begin{aligned}\n&= \\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n \\Bigl[\\Pr(Z_i = z, Z_j = z) - \\Pr(Z_i = z)\\Pr(Z_j = z)\\Bigr] \\\\\n&\\leq \\frac{k^2}{n^2}\\sum_{i=1}^n \\hl{\\sum_{j=1}^n} \\alpha(Z_i, Z_j) \\leq \\frac{k^2 d_{\\mathrm{AVG}}}{n} + \\frac{k^2}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n (1 - d_{ij})\\alpha(Z_i, Z_j),\n\\end{aligned}\n\\]", "evidence": "The evidence is an exact counterexample and a derivation. For each $n$, suppose that $n$ units have independent assignments of probability $1/2$ and outcomes $y_i(\\mathbf{z}) = z_i$, and let $k = 2$ and $z = 1$. This sequence satisfies Assumptions~1, 2 and~5. Note that $\\alpha(Z_i, Z_j) = 0$ for $j \\neq i$ and $\\alpha(Z_i, Z_i) = 1/4$. The first line of the passage as printed above equals $1/n$, which is also the value of $\\operatorname{Var}(\\hat n_z/n)$, and the printed middle expression is $0$. For $n = 4$ the printed inequality reads $1/4 \\leq 0$. With the sum over all $j$ the middle expression is $1/4$, and the last expression is $1$.\n\nFor the corrected display, the definition of the alpha-mixing coefficient implies that $\\Pr(Z_i = z, Z_j = z) - \\Pr(Z_i = z)\\Pr(Z_j = z) \\leq \\alpha(Z_i, Z_j)$ for every pair, including $j = i$. Summing over all pairs, I obtain the corrected middle expression. Since $I_{ii} = 1$, Definition~5 gives $d_{ii} = 1$. Using $d_{ij} + (1 - d_{ij}) = 1$ and $0 \\leq \\alpha(Z_i, Z_j) \\leq 1/4$, I have\n\\[\n\\sum_{i=1}^n\\sum_{j=1}^n \\alpha(Z_i, Z_j) \\leq \\frac{1}{4}\\sum_{i=1}^n\\sum_{j=1}^n d_{ij} + \\sum_{i=1}^n\\sum_{j=1}^n (1 - d_{ij})\\alpha(Z_i, Z_j) = \\frac{n}{4}d_{\\mathrm{AVG}} + \\sum_{i=1}^n\\sum_{j=1}^n (1 - d_{ij})\\alpha(Z_i, Z_j).\n\\]\nMultiplying by $k^2/n^2$ gives a bound that is no larger than the printed last expression.", "problem": "For $j = i$ the bracketed term of the display equals $\\Pr(Z_i = z)(1 - \\Pr(Z_i = z))$, the variance of the indicator of $Z_i = z$, which is positive under Assumption~1A. The sum of $\\alpha(Z_i, Z_j)$ over $j \\neq i$ does not bound these terms: under independent assignments it is zero, while the left side is positive. The sum over all $j$ is what the neighbouring expressions use, and the term $k^2 d_{\\mathrm{AVG}}/n$ of the last expression already covers the pairs $j = i$, because $d_{ii} = 1$. The first inequality of the same display, which passes from the weighted variance to a signed sum of covariances, is the subject of \\fid{P009-E009}.", "what_survives": "The last expression of the display, the bound $\\operatorname{Var}(\\hat n_z/n) \\leq k^2 n^{-1}(d_{\\mathrm{AVG}} + \\alpha_{\\mathrm{EXT}})$ that follows it, the rate of Lemma~A23 and the rates of the Hájek estimator in Propositions~5 and~6 are unchanged. No assumption is added."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E010", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Lemma A6: if mu-hat_z minus mu-breve_z is o_p(1) and n-hat_z over n minus 1 is o_p(1), then (n over n-hat_z) mu-hat_z minus mu-breve_z equals O_p of a sum. A red box marks the sum (mu-hat_z minus mu-breve_z) + (n-hat_z over n minus 1), which has no absolute values.", "caption": "Supplement A, p. 4: the statement of Lemma A6. The marked argument of $\\mathcal{O}_{\\mathrm{p}}$ is a signed sum of the two errors.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E011.png"}, {"alt": "Crop of the last display of the proof and the sentence after it. The display writes the error of the ratio as (mu-hat_z minus mu-breve_z) minus (n-hat_z over n minus 1) mu-breve_z plus r_1 minus r_2. In the sentence, a red box marks the argument of o_p in the definition of r_1, the sum (mu-hat_z minus mu-breve_z) + (n-hat_z over n minus 1).", "caption": "Supplement A, p. 5: the end of the proof of Lemma A6. The marked scale of the remainder $r_1$ is the same signed sum.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E011-proof.png"}], "formal_release": null, "formal_statements": [], "headline": "Lemma A6 writes its bound as $\\mathcal{O}_{\\mathrm{p}}$ of the signed sum of two random errors; the bound that the proof gives, and that the later lemmas use, has the sum of their absolute values.", "id": "P009-E011", "location": "Supplement A, Lemma A6, p. 4 (statement) and p. 5 (end of the proof)", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Lemma A6, p.~4: ``If $\\hat\\mu_z-\\breve\\mu_z=o_{\\mathrm{p}}(1)$ and $\\hat n_z/n-1=o_{\\mathrm{p}}(1)$, then:''\n\\[\n\\frac{n}{\\hat n_z}\\hat\\mu_z-\\breve\\mu_z=\\mathcal{O}_{\\mathrm{p}}\\big(\\hl{(\\hat\\mu_z-\\breve\\mu_z)+(\\hat n_z/n-1)}\\big).\n\\]\nEnd of the proof, p.~5:\n\\[\n\\frac{n}{\\hat n_z}\\hat\\mu_z-\\breve\\mu_z=(\\hat\\mu_z-\\breve\\mu_z)-(\\hat n_z/n-1)\\breve\\mu_z+r_1-r_2,\n\\]\n``where $r_1=o_{\\mathrm{p}}\\big(\\hl{(\\hat\\mu_z-\\breve\\mu_z)+(\\hat n_z/n-1)}\\big)$ and $r_2=o_{\\mathrm{p}}\\big(\\hl{(\\hat n_z/n-1)(\\breve\\mu_z+1)}\\big)$ since $\\hat\\mu_z-\\breve\\mu_z$ converges in probability to zero and $\\hat n_z/n$ converges to one. The bound $|\\breve\\mu_z|\\le k^2$ completes the proof.''", "corrected": "Lemma A6: ``If $\\hat\\mu_z-\\breve\\mu_z=o_{\\mathrm{p}}(1)$ and $\\hat n_z/n-1=o_{\\mathrm{p}}(1)$, then:''\n\\[\n\\frac{n}{\\hat n_z}\\hat\\mu_z-\\breve\\mu_z=\\mathcal{O}_{\\mathrm{p}}\\big(\\hl{|\\hat\\mu_z-\\breve\\mu_z|+|\\hat n_z/n-1|}\\big).\n\\]\nHere $D_n=\\mathcal{O}_{\\mathrm{p}}(T_n)$ for nonnegative random variables $T_n$ means that for every $\\varepsilon>0$ there exist finite $M$ and $N$ such that $\\Pr(|D_n|>MT_n)\\le\\varepsilon$ for all $n\\ge N$. The lemma holds under Assumption 1, which enters through the bound $|\\breve\\mu_z|\\le k^2$. In the proof, $r_1=o_{\\mathrm{p}}\\big(\\hl{|\\hat\\mu_z-\\breve\\mu_z|+|\\hat n_z/n-1|}\\big)$ and $r_2=o_{\\mathrm{p}}\\big(\\hl{|\\hat n_z/n-1|(|\\breve\\mu_z|+1)}\\big)$.\n\nThe form used in Lemmas A10 and A23 follows: if in addition $\\hat\\mu_z-\\breve\\mu_z=\\mathcal{O}_{\\mathrm{p}}(a_n)$ and $\\hat n_z/n-1=\\mathcal{O}_{\\mathrm{p}}(b_n)$ for positive deterministic sequences $(a_n)$ and $(b_n)$, then $(n/\\hat n_z)\\hat\\mu_z-\\breve\\mu_z=\\mathcal{O}_{\\mathrm{p}}(a_n+b_n)$.", "intended_meaning": "The lemma transfers the rates of the two errors to the ratio, and the display is shorthand for a bound by their magnitudes. Let $A=\\hat\\mu_z-\\breve\\mu_z$, $B=\\hat n_z/n-1$ and $D=(n/\\hat n_z)\\hat\\mu_z-\\breve\\mu_z$. By the identity on p.~4 of the supplement, I have\n\\[\nD=\\frac{A-\\breve\\mu_zB}{1+B}.\n\\]\nSince $B=o_{\\mathrm{p}}(1)$, the event $\\{|B|\\le1/2\\}$ has probability tending to one. On this event, using the bound $|\\breve\\mu_z|\\le k^2$, I obtain $|D|\\le2(|A|+k^2|B|)\\le2\\max(1,k^2)(|A|+|B|)$, which implies that $D=\\mathcal{O}_{\\mathrm{p}}(|A|+|B|)$. Lemmas A10 and A23 apply the lemma with deterministic rates for $A$ and $B$, and this bound gives the rates stated there.\n\nThe absolute values are needed, because the signed sum can vanish when $D$ does not. For example, let $n=3r$ with $r$ odd, and let independent blocks of three units receive the assignments $(1,0,0)$ or $(0,1,1)$ with probability $1/2$ each. Let the outcomes be fixed at $(1,0,0)$ in every block. In this design there is no interference, and every treatment probability is $1/2$. Let $z=1$, and let $K$ denote the number of blocks that receive $(0,1,1)$. Then $A=(r-2K)/(3r)=-B$, and $A+B=0$ for every assignment, while $D=2(r-2K)/(3(r+K))\\ne0$ and $\\mathrm{E}[A^2]=\\mathrm{E}[B^2]=1/(9r)$. In the same example I find $r_1=B^2/(1+B)>0$, and the scale printed for $r_1$ needs absolute values as well."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E011", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the sentence 'We have T i at most reflected-c i at most min of m and n minus m, which, together with complete randomization, implies that T i is hyper-geometric according to' and the two displayed probability laws. Six red boxes mark the reflected letter c with subscript i in the lower entries of the binomial coefficients: two in the law of T i and four in the law of T i given Z i equals z.", "caption": "Supplement A, proof of Lemma A12, p. 10. The six marked symbols are the numbers of draws in the two hypergeometric laws; each is the count $\\check c_i$ of Definition A3.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E012.png"}, {"alt": "Crop of Definition A3. A red box marks the formula 'reflected-c i equals the sum over j from 1 to n of I j i', which is followed by the words 'be the number of units interfering with unit i'.", "caption": "Supplement A, Definition A3, p. 9. The marked sum runs over all $j$ from 1 to $n$, so it counts unit $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E012-b.png"}, {"alt": "Crop of the first three lines of the proof of Lemma A12. Red boxes mark the formula 'T i equals the sum over j not equal to i of I j i Z j' and the words 'excluding i itself'.", "caption": "Supplement A, start of the proof of Lemma A12, p. 9. The marked definition of $T_i$ sums over $j\\ne i$, and the marked words say that unit $i$ is excluded.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E012-c.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T17", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T17"}, {"id": "P009-T18", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T18"}], "headline": "In the proof of Lemma A12, the variable $T_i$ is hypergeometric with $\\check c_i-1$ draws, not $\\check c_i$, because the count $\\check c_i$ of Definition A3 includes unit $i$ and $T_i$ does not; the inequality stated in the lemma holds unchanged.", "id": "P009-E012", "location": "Supplement A, proof of Lemma A12, p. 10 (PDF p. 10 of Supplement A), displays after “hyper-geometric according to:” and the ratio identities that follow; Definition A3 and the definition of $T_i$, p. 9", "pdf_page": 10, "printed_page": "10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T17", "P009-T18"], "text": {"as_printed": "Supplement A prints the count of Definition A3 as a reflected letter $c$ with subscript $i$. It is written $\\check c_i$ here.\n\nDefinition A3, p.~9: ``Let $\\check c_i=\\sum_{j=1}^n I_{ji}$ be the number of units interfering with unit $i$.''\n\nProof of Lemma A12, p.~9: ``let $T_i=\\sum_{j\\ne i}I_{ji}Z_j$ be the element sum of $\\widetilde{\\mathbf{Z}}_{-i}$ (i.e., the number of treated units interfering with $i$, excluding $i$ itself).''\n\nProof of Lemma A12, p.~10: ``We have $T_i\\le\\check c_i\\le\\min(m,n-m)$, which, together with complete randomization, implies that $T_i$ is hyper-geometric according to:\n\\[\n\\Pr\\bigl(T_i=t\\bigr)=\\binom{m}{t}\\binom{n-m}{\\hl{\\check c_i}-t}\\binom{n}{\\hl{\\check c_i}}^{-1},\n\\]\n\\[\n\\begin{aligned}\n\\Pr\\bigl(T_i=t\\mid Z_i=z\\bigr)={}&z\\binom{m-1}{t}\\binom{n-m}{\\hl{\\check c_i}-t}\\binom{n-1}{\\hl{\\check c_i}}^{-1}\\\\\n&+(1-z)\\binom{m}{t}\\binom{n-m-1}{\\hl{\\check c_i}-t}\\binom{n-1}{\\hl{\\check c_i}}^{-1}.\\text{''}\n\\end{aligned}\n\\]\nThe ratio identities that follow end with\n\\[\n\\frac{\\Pr\\bigl(T_i=t\\mid Z_i=z\\bigr)}{\\Pr\\bigl(T_i=t\\bigr)}-1=\\frac{\\hl{\\check c_i}}{\\Pr(Z_i=z)(n-\\hl{\\check c_i})}\\biggl[\\Pr(Z_i=z)-z\\frac{t}{\\hl{\\check c_i}}-(1-z)\\frac{\\hl{\\check c_i}-t}{\\hl{\\check c_i}}\\biggr],\n\\]\nand the proof concludes: ``The expression within brackets is strictly between $-1$ and $1$, so it follows:\n\\[\n\\biggl|\\frac{\\Pr\\bigl(T=t\\mid Z_i=z\\bigr)}{\\Pr\\bigl(T=t\\bigr)}-1\\biggr|\\le\\frac{\\check c_i}{\\Pr(Z_i=z)(n-\\check c_i)}\\le\\frac{\\check c_i}{\\Pr(Z_i=z)\\min(m,n-m)},\\text{''}\n\\]", "corrected": "Let $\\hl{r_i=\\check c_i-1}$ be the number of units other than $i$ that interfere with unit $i$. \\hl{The probability ratios below are for integers $0\\le t\\le r_i$, the support of $T_i$.} Since $T_i\\le r_i<\\min(m,n-m)$, complete randomization implies that $T_i$ is hypergeometric according to:\n\\[\n\\Pr\\bigl(T_i=t\\bigr)=\\binom{m}{t}\\binom{n-m}{\\hl{r_i}-t}\\binom{n}{\\hl{r_i}}^{-1},\n\\]\n\\[\n\\begin{aligned}\n\\Pr\\bigl(T_i=t\\mid Z_i=z\\bigr)={}&z\\binom{m-1}{t}\\binom{n-m}{\\hl{r_i}-t}\\binom{n-1}{\\hl{r_i}}^{-1}\\\\\n&+(1-z)\\binom{m}{t}\\binom{n-m-1}{\\hl{r_i}-t}\\binom{n-1}{\\hl{r_i}}^{-1}.\n\\end{aligned}\n\\]\nIf $r_i=0$, then $T_i=0$ and the ratio of the two probabilities is one. If $r_i>0$, the identities of the printed proof hold with $r_i$ in place of $\\check c_i$ and end with\n\\[\n\\frac{\\Pr\\bigl(T_i=t\\mid Z_i=z\\bigr)}{\\Pr\\bigl(T_i=t\\bigr)}-1=\\frac{\\hl{r_i}}{\\Pr(Z_i=z)(n-\\hl{r_i})}\\biggl[\\Pr(Z_i=z)-z\\frac{t}{\\hl{r_i}}-(1-z)\\frac{\\hl{r_i}-t}{\\hl{r_i}}\\biggr].\n\\]\nThe expression within brackets is strictly between $-1$ and $1$, so in both cases\n\\[\n\\begin{aligned}\n\\biggl|\\frac{\\Pr\\bigl(T_i=t\\mid Z_i=z\\bigr)}{\\Pr\\bigl(T_i=t\\bigr)}-1\\biggr|&\\le\\frac{\\hl{r_i}}{\\Pr(Z_i=z)(n-\\hl{r_i})}\\\\\n&\\le\\frac{\\check c_i}{\\Pr(Z_i=z)(n-\\check c_i)}\\le\\frac{\\check c_i}{\\Pr(Z_i=z)\\min(m,n-m)},\n\\end{aligned}\n\\]\nwhere $r_i\\le\\check c_i\\le\\min(m,n-m)\\le\\max(m,n-m)$ gives the last two inequalities. Definition A3 and the statement of Lemma A12 are unchanged.", "evidence": "The evidence is an exact counterexample to the printed laws and a derivation of the corrected ones. Let $n=4$ and $m=2$, and let the potential outcomes be $y_1(\\mathbf{z})=z_1+z_2$ and $y_j(\\mathbf{z})=z_j$ for $j=2,3,4$. Then $I_{11}=I_{21}=1$, $\\check c_1=2=\\min(m,n-m)$ and $T_1=Z_2$. Enumerating the six assignments, I find the laws in the following table.\n\n\\begin{audittable}{rrrrrrr}{Laws of $T_1$ in the example with $n=4$, $m=2$ and $\\check c_1=2$: enumeration of the six assignments, and the printed formulas.}\n$t$ & $\\Pr(T_1=t)$ & printed & $\\Pr(T_1=t\\mid Z_1=1)$ & printed & $\\Pr(T_1=t\\mid Z_1=0)$ & printed \\\\\n$0$ & $1/2$ & $1/6$ & $2/3$ & $1/3$ & $1/3$ & $0$ \\\\\n$1$ & $1/2$ & $2/3$ & $1/3$ & $2/3$ & $2/3$ & $2/3$ \\\\\n$2$ & $0$ & $1/6$ & $0$ & $0$ & $0$ & $1/3$ \\\\\n\\end{audittable}\n\nFor the corrected laws, let $S=\\{j\\ne i:I_{ji}=1\\}$ and $r_i=|S|=\\check c_i-1$, so that $T_i=\\sum_{j\\in S}Z_j$. By the premise $\\check c_i\\le\\min(m,n-m)$ of the lemma, I have $r_i<\\min(m,n-m)$. Counting the assignments with $t$ treated units in $S$, I find\n\\[\n\\Pr(T_i=t)=\\binom{r_i}{t}\\binom{n-r_i}{m-t}\\binom{n}{m}^{-1}=\\binom{m}{t}\\binom{n-m}{r_i-t}\\binom{n}{r_i}^{-1}.\n\\]\nConditional on $Z_i=z$, the other $n-1$ units are completely randomized with $m-z$ treated units, and $i\\notin S$. The same count with $n-1$ and $m-z$ in place of $n$ and $m$ gives\n\\[\n\\Pr(T_i=t\\mid Z_i=z)=\\binom{m-z}{t}\\binom{n-1-m+z}{r_i-t}\\binom{n-1}{r_i}^{-1}.\n\\]\nThese are the printed laws with $r_i$ in place of $\\check c_i$. If $r_i=0$, then $T_i=0$ and the ratio of the two probabilities is one. Suppose $r_i>0$, and let $p_z=\\Pr(Z_i=z)=\\{zm+(1-z)(n-m)\\}/n$ and $u_z=zt+(1-z)(r_i-t)$. Cancelling factorials as in the printed proof, I obtain\n\\[\n\\frac{\\Pr(T_i=t\\mid Z_i=z)}{\\Pr(T_i=t)}-1=\\frac{n}{n-r_i}\\biggl[\\frac{r_i}{n}-z\\frac{t}{m}-(1-z)\\frac{r_i-t}{n-m}\\biggr]=\\frac{r_i}{p_z(n-r_i)}\\biggl[p_z-\\frac{u_z}{r_i}\\biggr].\n\\]\nSince $0\\le u_z\\le r_i$ and $0<p_z<1$, the last bracket lies strictly between $-1$ and $1$. The inequality $r_i\\le\\check c_i$ implies that $r_i/(n-r_i)\\le\\check c_i/(n-\\check c_i)$, and $n-\\check c_i\\ge\\max(m,n-m)\\ge\\min(m,n-m)$. Combining these inequalities, I obtain the chain of inequalities in the corrected passage below. Note that the reduction from the vector $\\widetilde{\\mathbf{Z}}_{-i}$ to its sum $T_i$ at the start of the printed proof is correct, since each pattern on $S$ with $t$ treated units has the same probability with and without the conditioning on $Z_i$.\n\nA check in rational arithmetic agrees with the derivation: the corrected laws match an enumeration of the assignments for every admissible $(n,m,\\check c_i)$ such that $2\\le n\\le10$, and the chain of inequalities holds in all $61{,}600$ cases $(n,m,\\check c_i,z,t)$ such that $2\\le n\\le40$. On that grid the left side is at most $19/42$ of the bound stated in the lemma.", "problem": "Definition A3 counts every unit that interferes with unit $i$, and by the convention $I_{ii}=1$ of the article (p.~680), unit $i$ is one of them. The variable $T_i$ counts the treated units among the others, and there are $\\check c_i-1$ of those. Under complete randomization, $T_i$ is the number of treated units in a fixed set of $\\check c_i-1$ units. Both printed laws, and the ratio identities derived from them, use one draw too many.\n\nThe error is confined to the proof. Using the correct number of draws in the same steps, I obtain a bound that is no larger than the one stated in Lemma A12. The lemma holds as stated.", "what_survives": "Lemma A12 holds as stated, with the same assumptions and constant. Lemma A14, which applies the inequality of Lemma A12 on p.~14 of Supplement A, Proposition 3, the part of Corollary 1 on complete randomization, and the reference to Lemma A12 in Remark 1 on p.~688 use only that inequality and need no change. Only the two probability laws and the ratio identities inside the proof are corrected."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E012", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop showing Lemma A7, e_AVG is at most C_2 squared, and the first two lines of its proof. A red box marks the equality sign at the end of the first line, in 'Definition 7 gives e_ij ='. The next line continues with the sum over l of (1 - d_ij) I_li I_rho(l)j, followed by 'is at most' and the sum over l of I_li I_rho(l)j.", "caption": "Supplement A, foot of p. 5: Lemma A7 and the first sentence of its proof. The marked sign should be $\\le$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E013.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T19", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T19"}, {"id": "P009-T20", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T20"}, {"id": "P009-T44", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T44"}], "headline": "In the proof of Lemma A7 the first relation is printed as an equality and should be an inequality, since $e_{ij}$ is an indicator and the sum can exceed one.", "id": "P009-E013", "location": "Supplement A, proof of Lemma A7, foot of p. 5, first relation", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T19", "P009-T20", "P009-T44"], "text": {"as_printed": "``Definition 7 gives $e_{ij}\\hl{=}\\sum_{\\ell=1}^n(1-d_{ij})I_{\\ell i}I_{\\rho(\\ell)j}\\le\\sum_{\\ell=1}^nI_{\\ell i}I_{\\rho(\\ell)j}$.''", "corrected": "``Definition 7 gives $e_{ij}\\hl{\\le}\\sum_{\\ell=1}^n(1-d_{ij})I_{\\ell i}I_{\\rho(\\ell)j}\\le\\sum_{\\ell=1}^nI_{\\ell i}I_{\\rho(\\ell)j}$.''\n\nThe rest of the proof is as printed. With $c_\\ell=\\sum_{i=1}^nI_{\\ell i}$, it gives, for every interference structure and every pairing $\\rho$,\n\\[\ne_{\\mathrm{AVG}}\\le\\frac1n\\sum_{i=1}^n\\sum_{j=1}^n\\sum_{\\ell=1}^nI_{\\ell i}I_{\\rho(\\ell)j}=\\frac1n\\sum_{\\ell=1}^nc_\\ell c_{\\rho(\\ell)}\\le\\frac1{2n}\\sum_{\\ell=1}^n\\big(c_\\ell^2+c_{\\rho(\\ell)}^2\\big)=\\frac1n\\sum_{\\ell=1}^nc_\\ell^2=C_2^2.\n\\]", "evidence": "The evidence is an exact counterexample and a derivation. Let six units be paired as $(1,2)$, $(3,4)$, $(5,6)$, let every unit interfere with itself, and let $I_{31}=I_{51}=I_{42}=I_{62}=1$ be the only other interference. The treatments that affect unit $1$ are those of units $1,3,5$, and the treatments that affect unit $2$ are those of units $2,4,6$. Since no treatment affects both units, I have $d_{12}=0$. By Definition 7, $e_{12}=1$, while the printed sum equals $3$, with terms from $\\ell=1,3,5$. With $\\le$ in place of $=$ the chain is valid for every interference structure and every pairing: an indicator that some term of a sum of zeros and ones equals one is at most the sum.", "problem": "Definition 7 (p.~685) sets $e_{ij}=1$ if $(1-d_{ij})I_{\\ell i}I_{\\rho(\\ell)j}=1$ for some unit $\\ell$, and $e_{ij}=0$ otherwise. The sum on the right counts the units $\\ell$ with this property, and several units can have it. The indicator is at most the sum and is not equal to it in general.", "what_survives": "Lemma A7 and the results for the paired design that use it are unchanged. The proof uses only the upper bound $e_{ij}\\le\\sum_{\\ell}I_{\\ell i}I_{\\rho(\\ell)j}$, and the display on p.~6 is correct as printed."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E013", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop showing Proposition 1 and the start of its proof, which defines lambda as the limit superior of d_AVG over n. A red box marks the words 'd_AVG is at least lambda n happens infinitely often', which run over two lines. Below, the display defines the potential outcomes y_i(z) for i up to the ceiling of lambda to the power 0.5 times n.", "caption": "Supplement A, p. 6: Proposition 1 and the first paragraph of its proof. The marked clause does not follow from the definition of the limit superior.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E014.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T5", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T5"}, {"id": "P009-T6", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T6"}], "headline": "The proof of Proposition 1 asserts that $d_{\\mathrm{AVG}}\\ge\\lambda n$ infinitely often when $\\lambda$ is the limit superior of $n^{-1}d_{\\mathrm{AVG}}$, which can fail; the subsequence should be selected with a constant $c$ such that $0<c<\\lambda$.", "id": "P009-E014", "location": "Supplement A, Section 4, proof of Proposition 1, p. 6, first paragraph; the step is cited again in the last paragraph of the proof, p. 7", "pdf_page": 6, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T5", "P009-T6"], "text": {"as_printed": "``Proof. Let $\\lambda=\\limsup_{n\\to\\infty}n^{-1}d_{\\mathrm{AVG}}$. Assumption 2 is equivalent to $\\lambda=0$, so if it does not hold, we know that $\\lambda$ is strictly greater than zero. The definition of $\\lambda$ implies that $d_{\\mathrm{AVG}}\\ge\\hl{\\lambda}n$ happens infinitely often. In these cases, set the potential outcomes to: \\ldots''\n\nLast paragraph of the proof, p.~7: ``\\ldots\\ with probability one whenever $d_{\\mathrm{AVG}}\\ge\\hl{\\lambda}n$, which, as we already noted, happens infinitely often.''", "corrected": "``Proof. Let $\\lambda=\\limsup_{n\\to\\infty}n^{-1}d_{\\mathrm{AVG}}$. Assumption 2 is equivalent to $\\lambda=0$, so if it does not hold, we know that $\\lambda$ is strictly greater than zero. $\\hl{\\text{Fix a constant }c\\text{ such that }0<c<\\lambda.}$ The definition of $\\lambda$ implies that $d_{\\mathrm{AVG}}\\ge\\hl{c}n$ happens infinitely often. In these cases, set the potential outcomes to:\n\\[\ny_i(\\mathbf{z})=\\begin{cases}n\\lceil\\lambda^{0.5}n\\rceil^{-1}z_1\\big[z_i\\Pr(Z_i=1)-(1-z_i)\\Pr(Z_i=0)\\big]&\\text{if }i\\le\\lceil\\lambda^{0.5}n\\rceil,\\\\0&\\text{otherwise,}\\end{cases}\n\\]\nwhere $\\lceil\\cdot\\rceil$ is the ceiling function.''\n\nIn the last paragraph of the proof, p.~7: ``\\ldots\\ with probability one whenever $d_{\\mathrm{AVG}}\\ge\\hl{c}n$, which, as we already noted, happens infinitely often.''\n\nThe constant $\\lambda$ stays in the construction, so that the limit superior of $n^{-1}d_{\\mathrm{AVG}}$ for the constructed outcomes is still $\\lambda$. For the sample sizes with $d_{\\mathrm{AVG}}<cn$, all potential outcomes may be set to zero.", "evidence": "The evidence is an exact counterexample to the step. For each $n$, let $h_n=\\max\\{1,\\lfloor n/2\\rfloor-1\\}$, and let $y_i(\\mathbf{z})=z_1$ for $i\\le h_n$ and $y_i(\\mathbf{z})=0$ otherwise. These potential outcomes lie in $[0,1]$, and Assumption 1 holds with $k=3$, for instance under complete randomization with $\\lfloor n/2\\rfloor$ treated units when $n\\ge2$. The treatment of unit $1$ affects units $1,\\dots,h_n$, and every unit interferes with itself, so $d_{ij}=1$ if and only if $i=j$ or $i,j\\le h_n$. I obtain\n\\[\nn^{-1}d_{\\mathrm{AVG}}=\\frac{h_n^2+n-h_n}{n^2}.\n\\]\nFor $n=2m$ with $m\\ge3$ this equals $1/4-(m-2)/(4m^2)$, and for $n=2m+1$ with $m\\ge2$ it equals $1/4-(8m-11)/(4(2m+1)^2)$. These two expressions imply that $\\lambda=1/4$ and $n^{-1}d_{\\mathrm{AVG}}<\\lambda$ for every $n\\ge5$. The event $d_{\\mathrm{AVG}}\\ge\\lambda n$ occurs only for $n\\le4$.\n\nThe repair is valid for the following reason. If $0<c<\\lambda$ and $n^{-1}d_{\\mathrm{AVG}}\\ge c$ held for finitely many $n$ only, then the limit superior would be at most $c$, which is less than $\\lambda$.", "problem": "A sequence can stay strictly below its limit superior from some index on, and it then reaches or exceeds the limit superior only finitely often. The sample sizes with $d_{\\mathrm{AVG}}\\ge\\lambda n$ are those for which the proof constructs the potential outcomes and shows that $|\\hat\\tau_{\\mathrm{HT}}-\\tau_{\\mathrm{EATE}}|\\ge k^{-1}$. If there are finitely many of them, the argument as printed says nothing about convergence.", "what_survives": "Proposition 1 and its use on p.~683 are unchanged, and the repair adds no assumption. The construction is kept with $\\lambda$ as printed: the constructed outcomes satisfy $d_{\\mathrm{AVG}}\\ge\\lambda n$, are bounded by $\\lambda^{-0.5}$, and give $\\tau_{\\mathrm{EATE}}=\\Pr(Z_1=1)$ and $\\hat\\tau_{\\mathrm{HT}}=Z_1$. The proof of Proposition 11 cites only the count of $d_{\\mathrm{AVG}}$ from this proof and is not affected. I did not assess the remark in the proof that the construction can be modified to replicate a specified sequence of $d_{\\mathrm{AVG}}$ exactly."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E014", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Definition 6 with the Horvitz-Thompson and Hajek estimators and the two lines of text after it. Red boxes mark the two divisors in the Hajek estimator: the sum over i of Z i over p i, and the sum over i of 1 minus Z i over 1 minus p i.", "caption": "Definition 6, p. 682 (PDF p. 10). The two marked denominators of the Hájek estimator are zero when the corresponding arm is empty.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E015.png"}, {"alt": "Crop of three lines of text and one display from the proof of Lemma A6. A red box marks the words 'we can disregard the event' in the sentence 'The second premise ensures that we can disregard the event n hat z equals 0, which gives'.", "caption": "Supplement A, proof of Lemma A6, p. 4. The marked words set aside the event $\\hat n_z=0$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E015-b.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T40", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T40"}, {"id": "P009-T41", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T41"}, {"id": "P009-T43", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T43"}], "headline": "Definition 6 leaves the Hájek estimator undefined when a treatment arm is empty. A finite convention, such as zero, leaves probability limits and stochastic rates unchanged when the empty-arm probability tends to zero; Proposition 1 requires an explicit convention without that guarantee.", "id": "P009-E015", "location": "Definition 6, p. 682 (PDF p. 10); Supplement A, proof of Lemma A6, p. 4 (PDF p. 4 of Supplement A)", "pdf_page": 10, "printed_page": "682", "reaches": [], "related_entries": [], "revisions": [{"classification_after": {"class": "clarity_note", "severity": null}, "classification_before": {"class": "clarity_note", "severity": null}, "note": "The headline and reading were revised; the classification is unchanged (clarity note, with no severity). How this came to light: the Lean work on Proposition 1 showed that individual overlap does not make empty treatment arms asymptotically negligible: a common fair treatment assignment leaves an arm empty with probability one. Why it changed: the earlier text said that any finite completion leaves every asymptotic result unchanged. The revised text limits that assurance to vanishing empty-arm probability and states that Proposition 1 is established with the entire Hájek contrast set to zero when either arm is empty. The corrected definition is unchanged. P. M. Aronow delegated this decision to the model session that runs the audit, which adopted the amendment recommended by the two checks; this note was written by that session."}], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P009-T40", "P009-T41", "P009-T43"], "text": {"as_printed": "Definition 6, p.~682:\n\\[\n\\hat\\tau_{\\mathrm{H\\acute{A}}}=\\Biggl(\\sum_{i=1}^n\\frac{Z_iY_i}{p_i}\\hl{\\Bigg/\\sum_{i=1}^n\\frac{Z_i}{p_i}}\\Biggr)-\\Biggl(\\sum_{i=1}^n\\frac{(1-Z_i)Y_i}{1-p_i}\\hl{\\Bigg/\\sum_{i=1}^n\\frac{1-Z_i}{1-p_i}}\\Biggr),\n\\]\n``where $p_i=\\Pr(Z_i=1)$ is the marginal treatment probability for unit $i$.''\n\nSupplement A, proof of Lemma A6, p.~4: ``The second premise ensures that we can disregard the event $\\hat n_z=0$, which gives:''", "corrected": "Definition 6, with the Hájek estimator as printed, ``where $p_i=\\Pr(Z_i=1)$ is the marginal treatment probability for unit $i$\\hl{, and where $\\hat\\tau_{\\mathrm{H\\acute{A}}}=0$ if $\\sum_{i=1}^nZ_i=0$ or $\\sum_{i=1}^n(1-Z_i)=0$}.''", "intended_meaning": "When both arms are nonempty, Definition 6 is the usual difference of weighted arm means. When no unit is treated, or no unit is in control, one of the two ratios is $0/0$, and the paper states no value for it. Under a Bernoulli design with $p_i=1/2$, for example, this event has probability $2^{1-n}$. For results that establish convergence of both normalized arm counts to one, the empty-arm event is asymptotically negligible. Under the second premise of Lemma A6, $\\hat n_z/n-1=o_p(1)$, the probability of the event $\\hat n_z=0$ is at most $\\Pr(|\\hat n_z/n-1|>1/2)$, which tends to zero. Under this arm-count convergence, suppose that two versions of the estimator agree when both arms are nonempty. Then they differ only on an event whose probability tends to zero, and they have the same limits in probability and the same rates. A convention does not by itself give a guarantee at a finite sample size, such as a coverage guarantee.\n\nProposition 1 has no such arm-count hypothesis: assigning all units the same fair Bernoulli treatment gives an empty arm with probability one. Setting the entire contrast to zero on that event makes its Hájek assertion precise, and the outcome construction proves that completed assertion."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E015", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the top of p. 37, with the running head Supplement A: Proofs and the page number 37. The text reads: The Bernoulli design and Assumption 1 imply that the support of z prime consists of 2 to the b n points each with a probability of at least k to the minus b n. It follows that: the limit as n tends to infinity of Pr(f star n of z SUB,n = 1) = alpha over 2, and the limit as n tends to infinity of Pr(f star n of z SUB,n = minus 1) = alpha over 2. Red outlines surround the symbol z prime in the first line and the words at least k to the minus b n in the second.", "caption": "Supplement A, p. 37, top of the page, in the proof of Proposition 11. The outlines mark the undefined symbol $\\mathbf{z}'$ and the lower bound “at least $k^{-b_n}$”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E016.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T27", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T27"}], "headline": "The two limits follow from an upper bound, $(1 - 1/k)^{b_n} \\to 0$, on the probability of each support point of $\\mathbf{z}_{\\mathrm{SUB},n}$ and not from the lower bound $k^{-b_n}$, and $\\mathbf{z}'$ should read $\\mathbf{z}_{\\mathrm{SUB},n}$.", "id": "P009-E016", "location": "Supplement A, p. 37, proof of Proposition 11, first paragraph of the page", "pdf_page": 37, "printed_page": "37", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T27"], "text": {"as_printed": "The Bernoulli design and Assumption~1 imply that the support of $\\hl{\\mathbf{z}'}$ consists of $2^{b_n}$ points each with a probability of \\hl{at least} $\\hl{k^{-b_n}}$. It follows that:\n\\[\n\\lim_{n \\to \\infty}\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = 1\\bigr) = \\alpha/2 \\qquad \\text{and} \\qquad \\lim_{n \\to \\infty}\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = -1\\bigr) = \\alpha/2.\n\\]", "corrected": "The Bernoulli design and Assumption~1 imply that the support of $\\hl{\\mathbf{z}_{\\mathrm{SUB},n}}$ consists of $2^{b_n}$ points, each with a probability of \\hl{at most} $\\hl{(1 - 1/k)^{b_n}}$, \\hl{which tends to zero. Disjoint sets of support points can then approximate the two target masses $\\alpha/2$, and the minimizing $f_n^*$ satisfies}\n\\[\n\\lim_{n \\to \\infty}\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = 1\\bigr) = \\alpha/2 \\qquad \\text{and} \\qquad \\lim_{n \\to \\infty}\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = -1\\bigr) = \\alpha/2.\n\\]", "evidence": "The evidence is a derivation. By Assumption~1A, I have $k^{-1} \\leq \\Pr(Z_i = 1) \\leq 1 - k^{-1}$ for every unit. Since the assignments are independent under a Bernoulli design, every support point $\\mathbf{a}$ of $\\mathbf{z}_{\\mathrm{SUB},n}$ satisfies\n\\[\nk^{-b_n} \\leq \\Pr(\\mathbf{z}_{\\mathrm{SUB},n} = \\mathbf{a}) \\leq \\delta_n, \\qquad \\delta_n = (1 - 1/k)^{b_n},\n\\]\nand $\\delta_n \\to 0$ because $b_n \\to \\infty$. The lower bound alone is not enough. Suppose that a distribution on $2^b$ points assigns one point the probability $1 - (2^b - 1)4^{-b}$ and every other point the probability $4^{-b}$. Then every point has probability at least $4^{-b}$, which is the printed lower bound with $k = 4$. The largest probability tends to one, and no set of points has probability near $\\alpha/2$ for $\\alpha \\in (0, 1)$. This distribution is not a Bernoulli design, and it serves only to test the implication. With the upper bound, I order the support points and take the longest initial segments whose total probabilities are at most $\\alpha/2$ and at most $\\alpha$. Each total falls short of its target by less than $\\delta_n$. Let $f$ equal $1$ on the first segment, $-1$ on the rest of the second, and $0$ elsewhere. With this choice, $\\Pr(f = 1)$ and $\\Pr(f = -1)$ are each within $\\delta_n$ of $\\alpha/2$. The total error of the minimizer $f_n^*$ is no larger than the total error of this function, which is at most $2\\delta_n$, and this implies both limits.", "problem": "The function $f_n^*$ is chosen on p.~36 to bring $\\Pr(f(\\mathbf{z}_{\\mathrm{SUB},n}) = 1)$ and $\\Pr(f(\\mathbf{z}_{\\mathrm{SUB},n}) = -1)$ as close to $\\alpha/2$ as possible. To ensure that sets of support points can approach these two masses, it is enough that the largest point probability tends to zero. The printed lower bound on the probability of each point does not ensure that: a distribution on $2^{b}$ points can satisfy such a lower bound while one point carries almost all of the mass. The bound that is needed is the upper bound $(1 - 1/k)^{b_n}$, which follows from the same assumptions. The symbol $\\mathbf{z}'$ is also not defined, and the vector $\\mathbf{z}_{\\mathrm{SUB},n}$ introduced on p.~36 is meant.", "what_survives": "Both limits hold, and with them the convergence in distribution on which Proposition~11 rests. No assumption is added. The comparison that uses these limits later in the proof is corrected in \\fid{P009-E017}."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E016", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of part of a proof. A first display states that the limit as n tends to infinity of the bracket Pr(f star n of z SUB,n = 1) minus Pr(f star n of z SUB,n = minus 1) equals 0. The text reads: Hence, for any fixed s in the real numbers without minus 1, 0 and 1, there exists an n prime such that the smallest distance from s to any point in the set of minus 1, 0 and 1 for all n at least n prime is less than: followed by the displayed difference Pr(f star n of z SUB,n = 1) minus Pr(f star n of z SUB,n = minus 1). Red outlines surround the words is less than and the displayed difference. The text continues: For the s under consideration and all such n at least n prime, and a last display states that Pr(n to the 0.5 d AVG to the minus 0.5 times (tau-hat HT minus E of tau-hat HT) is at most s) = Pr(f star n of z SUB,n is at most s).", "caption": "Supplement A, p. 38, proof of Proposition 11. The outlines mark the words “is less than” and the difference of probabilities, printed without an absolute value, to which the distance is compared.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E017.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T28", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T28"}], "headline": "The comparison is reversed and lacks an absolute value: for all large $n$ the distance from $s$ to $\\{-1, 0, 1\\}$ is greater than $|\\Pr(f_n^* = 1) - \\Pr(f_n^* = -1)|$.", "id": "P009-E017", "location": "Supplement A, p. 38, proof of Proposition 11, sentence beginning “Hence, for any fixed $s$”", "pdf_page": 38, "printed_page": "38", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T28"], "text": {"as_printed": "Hence, for any fixed $s \\in \\mathbb{R} \\setminus \\{-1, 0, 1\\}$, there exists an $n'$ such that the smallest distance from $s$ to any point in $\\{-1, 0, 1\\}$ for all $n \\geq n'$ \\hl{is less than}:\n\\[\n\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = 1\\bigr) - \\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = -1\\bigr).\n\\]", "corrected": "Hence, for any fixed $s \\in \\mathbb{R} \\setminus \\{-1, 0, 1\\}$, there exists an $n'$ such that, for all $n \\geq n'$, the smallest distance from $s$ to any point in $\\{-1, 0, 1\\}$ \\hl{is greater than}:\n\\[\n\\hl{\\Bigl|}\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = 1\\bigr) - \\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n}) = -1\\bigr)\\hl{\\Bigr|}.\n\\]", "evidence": "The evidence is a derivation. Let $F_n = f_n^*(\\mathbf{z}_{\\mathrm{SUB},n})$, which takes values in $\\{-1, 0, 1\\}$, and let $\\mu_n = \\mathrm{E}[F_n] = \\Pr(F_n = 1) - \\Pr(F_n = -1)$. By the display headed ``Taken together'' on p.~38, the normalized estimator equals $F_n - \\mu_n$. Suppose that $|\\mu_n| < \\delta_s$. Then no point of $\\{-1, 0, 1\\}$ lies between $s$ and $s + \\mu_n$, so for each $t \\in \\{-1, 0, 1\\}$ the inequalities $t \\leq s$ and $t \\leq s + \\mu_n$ are equivalent, and I have\n\\[\n\\Pr(F_n - \\mu_n \\leq s) = \\Pr(F_n \\leq s + \\mu_n) = \\Pr(F_n \\leq s),\n\\]\nwhich is the next display of the proof. Since $\\mu_n \\to 0$, the condition $|\\mu_n| < \\delta_s$ holds for all large $n$. The limits of $\\Pr(F_n = 1)$ and $\\Pr(F_n = -1)$ used here are those justified in \\fid{P009-E016}.", "problem": "For a fixed $s$ outside $\\{-1, 0, 1\\}$ the distance $\\delta_s$ from $s$ to that set is positive and does not depend on $n$. The displayed difference, $\\mu_n$ say, tends to zero by the display that precedes the sentence. A fixed positive number cannot be less than $\\mu_n$ for all large $n$, and $\\mu_n$ may be negative. The next display of the proof needs the opposite comparison with an absolute value, $|\\mu_n| < \\delta_s$, which does hold for all large $n$. A further misprint in the last display of the proof, $x$ for $s$, is the subject of \\fid{P009-E020}.", "what_survives": "The equality of distribution functions in the next display holds for all large $n$ under the corrected condition, and the convergence in distribution claimed in the proof is preserved. Nothing else in the proof changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E017", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop showing Lemma A3, whose statement bounds the covariance of 1[Z_i = z] Y_i^a and 1[Z_j = z] Y_j^a by k^(2a), and the first three lines of the proof. In the first line of the proof, red boxes mark the term Y_i^a in the second argument of the covariance and the term Y_i^a in the second variance. The second line has the expectations of |Y_i|^(2a) and |Y_j|^(2a).", "caption": "Supplement A, p. 3: Lemma A3 and the first lines of its proof. The two marked terms carry the index $i$ where the statement of the lemma and the second line of the proof have $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E018.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T29", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T29"}], "headline": "In the first line of the proof of Lemma A3 the outcome of unit $j$ is printed twice as $Y_i^a$; both should read $Y_j^a$.", "id": "P009-E018", "location": "Supplement A, proof of Lemma A3, p. 3 (PDF p. 3 of the supplement), first line of the display", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T29"], "text": {"as_printed": "``Proof. The Cauchy--Schwarz inequality gives:''\n\\[\n\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i^a,\\mathbf{1}[Z_j=z]\\hl{Y_i^a}\\big)\\le\\Big[\\operatorname{Var}\\big(\\mathbf{1}[Z_i=z]Y_i^a\\big)\\operatorname{Var}\\big(\\mathbf{1}[Z_j=z]\\hl{Y_i^a}\\big)\\Big]^{0.5}\n\\]", "corrected": "In the proof of Lemma A3 (Assumption 1B with $q\\ge2a$), for all $i,j\\in\\mathbf{U}$ and $z\\in\\{0,1\\}$:\n\\[\n\\operatorname{Cov}\\big(\\mathbf{1}[Z_i=z]Y_i^a,\\mathbf{1}[Z_j=z]\\hl{Y_j^a}\\big)\\le\\Big[\\operatorname{Var}\\big(\\mathbf{1}[Z_i=z]Y_i^a\\big)\\operatorname{Var}\\big(\\mathbf{1}[Z_j=z]\\hl{Y_j^a}\\big)\\Big]^{0.5}\n\\]\nThe remaining lines of the display are as printed.", "evidence": "The evidence is a check of the page and a derivation. Let $X_i=\\mathbf{1}[Z_i=z]Y_i^a$ and $X_j=\\mathbf{1}[Z_j=z]Y_j^a$. For $r\\in\\{i,j\\}$, I have $\\operatorname{Var}(X_r)\\le\\mathrm{E}[X_r^2]\\le\\mathrm{E}[|Y_r|^{2a}]$. Using the Cauchy--Schwarz inequality, I obtain $\\operatorname{Cov}(X_i,X_j)\\le[\\mathrm{E}[|Y_i|^{2a}]\\,\\mathrm{E}[|Y_j|^{2a}]]^{0.5}$, which is the second line of the proof. For $q\\ge2a$, Jensen's inequality and Assumption 1B imply that $\\mathrm{E}[|Y_r|^{2a}]\\le\\mathrm{E}[|Y_r|^q]^{2a/q}\\le k^{2a}$, and the bound $k^{2a}$ of the lemma follows.", "problem": "Lemma A3 bounds the covariance of $\\mathbf{1}[Z_i=z]Y_i^a$ and $\\mathbf{1}[Z_j=z]Y_j^a$, and the second line of its proof has $\\mathrm{E}[|Y_j|^{2a}]$. The first line carries $Y_i^a$ in the second argument of the covariance and in the second variance. As printed, the line is a valid Cauchy--Schwarz inequality, but for a covariance other than the one in the lemma, and the second line does not follow from it.", "what_survives": "Lemma A3 holds as stated, and its proof is unchanged from the second line on."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E018", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of part of a proof. The text reads: Consider the first case in the proposition using Lemma A27. By construction, tau MSQ is at least 0 and b n times d AVG to the minus 1 equals d AVG to the minus 1 times d MAX, which is at least 1, so: a displayed inequality states that the limit inferior of b n d AVG to the minus 1 times the bracket a n minus n b n to the minus 1 Var of tau-hat HT is at least the limit inferior of tau MSQ, which is at least 0. The text continues: and the first premise of the lemma is satisfied without invoking additional assumptions. The second premise is satisfied by Corollary A4 because Var-hat AVG of tau-hat HT = b n times Var-hat BER of tau-hat HT and b n = d MAX = o(n to the 0.5 times d AVG to the 0.5) by assumption. A red outline surrounds Var-hat AVG of tau-hat HT. The crop ends: Consider the second case in the proposition using Lemma A28. By assumption: the limit inferior of tau MSQ is at least c, which is greater than 0, so.", "caption": "Supplement A, p. 34, proof of Proposition 9. The outline marks the estimator printed with the subscript AVG, in a sentence where $b_n = d_{\\mathrm{MAX}}$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E019.png"}, {"alt": "Crop of the last lines of p. 35. A displayed inequality states that the limit inferior of b n d AVG to the minus 1 times the bracket a n minus n b n to the minus 1 Var of tau-hat HT is at least the limit inferior of tau MSQ, which is at least 0. The text reads: and the first premise of the lemma is satisfied without invoking additional assumptions. The second premise is satisfied by Corollary A4 because Var-hat AVG of tau-hat HT = b n times Var-hat BER of tau-hat HT and b n = lambda max = o(n to the 0.5 times d AVG to the 0.5) by assumption. A red outline surrounds Var-hat AVG of tau-hat HT.", "caption": "Supplement A, p. 35, proof of Proposition 10. The outline marks the same estimator, again with the subscript AVG, in a sentence where $b_n = \\lambda_{\\max}$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E019-b.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T30", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T30"}, {"id": "P009-T49", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T49"}, {"id": "P009-T50", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T50"}], "headline": "The subscript AVG should read MAX in the proof of Proposition 9 and SR in the proof of Proposition 10.", "id": "P009-E019", "location": "Supplement A, p. 34, proof of Proposition 9, and p. 35, proof of Proposition 10: in each, the sentence that applies Corollary A4", "pdf_page": 34, "printed_page": "34", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T30", "P009-T49", "P009-T50"], "text": {"as_printed": "Page~34: The second premise is satisfied by Corollary~A4 because $\\widehat{\\operatorname{Var}}_{\\hl{\\mathrm{AVG}}}(\\hat\\tau_{\\mathrm{HT}}) = b_n\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})$ and $b_n = d_{\\mathrm{MAX}} = o(n^{0.5}d_{\\mathrm{AVG}}^{0.5})$ by assumption.\n\nPage~35: The second premise is satisfied by Corollary~A4 because $\\widehat{\\operatorname{Var}}_{\\hl{\\mathrm{AVG}}}(\\hat\\tau_{\\mathrm{HT}}) = b_n\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})$ and $b_n = \\lambda_{\\max} = o(n^{0.5}d_{\\mathrm{AVG}}^{0.5})$ by assumption.", "corrected": "Page~34: The second premise is satisfied by Corollary~A4 because $\\widehat{\\operatorname{Var}}_{\\hl{\\mathrm{MAX}}}(\\hat\\tau_{\\mathrm{HT}}) = b_n\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})$ and $b_n = d_{\\mathrm{MAX}} = o(n^{0.5}d_{\\mathrm{AVG}}^{0.5})$ by assumption.\n\nPage~35: The second premise is satisfied by Corollary~A4 because $\\widehat{\\operatorname{Var}}_{\\hl{\\mathrm{SR}}}(\\hat\\tau_{\\mathrm{HT}}) = b_n\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})$ and $b_n = \\lambda_{\\max} = o(n^{0.5}d_{\\mathrm{AVG}}^{0.5})$ by assumption.", "evidence": "The evidence is a check of the two pages against the definitions on pp.~691--693 of the article and against the opening of each proof, where $b_n$ is chosen.", "problem": "The article defines $\\widehat{\\operatorname{Var}}_{\\mathrm{AVG}} = d_{\\mathrm{AVG}}\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}$, $\\widehat{\\operatorname{Var}}_{\\mathrm{MAX}} = d_{\\mathrm{MAX}}\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}$ and $\\widehat{\\operatorname{Var}}_{\\mathrm{SR}} = \\lambda_{\\max}\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}$ (pp.~691--693). The two proofs set $b_n = d_{\\mathrm{MAX}}$ and $b_n = \\lambda_{\\max}$, so the estimator that equals $b_n\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}$ is $\\widehat{\\operatorname{Var}}_{\\mathrm{MAX}}$ in the first and $\\widehat{\\operatorname{Var}}_{\\mathrm{SR}}$ in the second.", "what_survives": "Propositions~9 and~10 and their proofs are unaffected. The multiplier $b_n$ and every bound are the intended ones, and only the label of the estimator changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E019", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of text. Red boxes mark the words 'is to' at the end of one line and 'captures' at the start of the next, in the sentence 'The purpose of Definition 8 is to captures how common interference is between paired units'.", "caption": "Article, p. 686 (PDF p. 14). The marked words read “is to captures”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020.png"}, {"alt": "Crop of four lines of text. A red box marks the words 'structure Paired' in 'the pairs reflect some social structure Paired units may, for example, live in the same household'.", "caption": "Article, p. 686 (PDF p. 14). No full stop separates the marked words “structure” and “Paired”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020-b.png"}, {"alt": "Crop of five lines of text from the concluding section. A red box marks the word 'inference' in 'even in the presence of moderate inference with unknown form'.", "caption": "Article, p. 697 (PDF p. 25). The marked word is “inference”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020-c.png"}, {"alt": "Crop of the statement of Lemma A5 and the first line of its proof. Red boxes mark the word 'govern' in 'is govern by a sequence a n' and in 'the rate of convergence in probability is govern by a n'.", "caption": "Supplement A, Lemma A5, p. 4. The marked word “govern” appears twice.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020-d.png"}, {"alt": "Crop of several displayed equations with connecting text. Red boxes mark the word 'substract' in two sentences that read 'Add and substract'.", "caption": "Supplement A, proof of Proposition 7, p. 28. The marked word “substract” appears twice.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020-e.png"}, {"alt": "Crop of two displayed equations with one line of text between them. A red box marks the word 'substract' in 'Add and substract'.", "caption": "Supplement A, proof of Proposition 7, p. 29. The marked word is “substract”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020-f.png"}, {"alt": "Crop of the first four lines of the proof of Proposition 11. A red box marks the word 'convergence' in 'the normalized sampling distribution of the HT estimator convergence in distribution to the random variable in Definition A4'.", "caption": "Supplement A, proof of Proposition 11, p. 36. The marked word is “convergence”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020-g.png"}, {"alt": "Crop of the last display of the proof of Proposition 11, which gives the limit of a probability evaluated at s as a function with four cases. Four red boxes mark the letter x in the conditions 'if x less than minus 1', 'if x less than 0', 'if x less than 1' and 'if x at least 1'.", "caption": "Supplement A, end of the proof of Proposition 11, p. 38. The argument on the left of the display is $s$; the four marked conditions are written in $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E020-h.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T31", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T31"}], "headline": "The article and Supplement A should read “to capture”, “social structure. Paired units”, “moderate interference”, “is governed”, “subtract” and “converges”, and the four cases of the last display in the proof of Proposition 11 should be written in $s$, not $x$.", "id": "P009-E020", "location": "Article, p. 686 (PDF p. 14) and p. 697 (PDF p. 25); Supplement A, pp. 4, 28, 29, 36 and 38 (the same PDF pages of Supplement A)", "pdf_page": 14, "printed_page": "686", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T31"], "text": {"as_printed": "\\begin{itemize}\n\\item Article, p.~686: ``The purpose of Definition 8 is to \\hl{captures} how common interference is between paired units.''\n\\item Article, p.~686: ``It is, for example, common that the pairs reflect some social \\hl{structure Paired} units may, for example, live in the same household.''\n\\item Article, p.~697: ``experiments often are informative of EATE even in the presence of moderate \\hl{inference} with unknown form.''\n\\item Supplement A, Lemma A5, p.~4: ``If the rate of convergence in the $L_2$-norm for a sequence of random variables is \\hl{govern} by a sequence $(a_n)$, then the rate of convergence in probability is \\hl{govern} by $(a_n)$.''\n\\item Supplement A, proof of Proposition 7, p.~28 (twice) and p.~29 (once): ``Add and \\hl{substract}:''\n\\item Supplement A, proof of Proposition 11, p.~36: ``the normalized sampling distribution of the HT estimator \\hl{convergence} in distribution to the random variable in Definition A4''\n\\item Supplement A, proof of Proposition 11, p.~38:\n\\[\n\\lim_{n\\to\\infty}\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n})\\le s\\bigr)=\\Pr\\bigl(X(\\alpha)\\le s\\bigr)=\\begin{cases}0&\\text{if }\\hl{x}<-1,\\\\ \\alpha/2&\\text{if }\\hl{x}<0,\\\\ 1-\\alpha/2&\\text{if }\\hl{x}<1,\\\\ 1&\\text{if }\\hl{x}\\ge1.\\end{cases}\n\\]\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Article, p.~686: ``The purpose of Definition 8 is to \\hl{capture} how common interference is between paired units.''\n\\item Article, p.~686: ``It is, for example, common that the pairs reflect some social \\hl{structure. Paired} units may, for example, live in the same household.''\n\\item Article, p.~697: ``even in the presence of moderate \\hl{interference} with unknown form.''\n\\item Supplement A, Lemma A5: ``is \\hl{governed} by a sequence $(a_n)$, then the rate of convergence in probability is \\hl{governed} by $(a_n)$.''\n\\item Supplement A, pp.~28 and 29: ``Add and \\hl{subtract}:''\n\\item Supplement A, p.~36: ``the HT estimator \\hl{converges} in distribution to the random variable in Definition A4''\n\\item Supplement A, p.~38:\n\\[\n\\lim_{n\\to\\infty}\\Pr\\bigl(f_n^*(\\mathbf{z}_{\\mathrm{SUB},n})\\le s\\bigr)=\\Pr\\bigl(X(\\alpha)\\le s\\bigr)=\\begin{cases}0&\\text{if }\\hl{s}<-1,\\\\ \\alpha/2&\\text{if }\\hl{s}<0,\\\\ 1-\\alpha/2&\\text{if }\\hl{s}<1,\\\\ 1&\\text{if }\\hl{s}\\ge1.\\end{cases}\n\\]\nFor a display that is valid at every $s$, the four ranges are $s<-1$, $-1\\le s<0$, $0\\le s<1$ and $s\\ge1$.\n\\end{itemize}", "evidence": "I checked each listed misprint on the pages of the article and Supplement A.", "problem": "These are misprints, and in each case the sentence shows the intended wording. In the display on p.~38 of Supplement A, the left side is a distribution function evaluated at $s$, and the four cases are written in $x$. The paragraph before the display restricts $s$ to $\\mathbb{R}\\setminus\\{-1,0,1\\}$, and for those $s$ the four cases, read in order, are the distribution function of the random variable $X(\\alpha)$ of Definition A4.", "what_survives": "No mathematical statement changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E020", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph that introduces the interference set G_i as a subset of U without i and defines bal(G_i) as the sum over j in G_i of (2 Z_j - 1). In the last sentence, a red box marks 'I_ij = 1 if and only if i in G_j' and a second red box marks 'l in G_i intersected with G_j'.", "caption": "Supplement B, p. 2: the definition of $\\operatorname{bal}(G_i)$ and the sentence that describes $I_{ij}$ and $d_{ij}$. The marked conditions refer to the sets $G_j$, which exclude unit $j$ itself.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E021.png"}, {"alt": "Crop of the paragraph after Figure B4. A red box marks 'I_ij = 1 if and only if i = 1 and j at most a_n'. The sentence continues 'and thus d_ij = 1 whenever i, j at most a_n', and the next sentence gives C_1 = 1 + (floor of a_n minus 1) over n and d_AVG = 1 + floor of a_n times (floor of a_n minus 1) over n.", "caption": "Supplement B, p. 7, first sentence after Figure B4. The marked equivalence omits $I_{ii}=1$; the values of $C_1$ and $d_{\\mathrm{AVG}}$ in the next sentence include it.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E021-p7.png"}], "formal_release": null, "formal_statements": [], "headline": "Two descriptions of the interference indicators in Supplement B leave out each unit’s own treatment; the indicators are those of the sets $G_i\\cup\\{i\\}$, which is the convention that the calculations in the supplement use.", "id": "P009-E021", "location": "Supplement B, Section 1, p. 2, last sentence of the paragraph that defines $\\operatorname{bal}(G_i)$; Section 1.3, p. 7, first sentence after Figure B4", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~2: ``For each unit, an \\emph{interference set} $G_i$ is generated as a subset of $\\mathbf{U}\\setminus i$. \\ldots\\ The potential outcomes are such that $I_{ij}=1$ if and only if $\\hl{i\\in G_j}$, and, consequently, $d_{ij}=1$ if and only if a unit $\\ell$ exists such that $\\hl{\\ell\\in G_i\\cap G_j}$.''\n\np.~7: ``We have $I_{ij}=1$ if and only if $\\hl{i=1\\text{ and }j\\le a_n}$, and thus $d_{ij}=1$ whenever $i,j\\le a_n$.''", "corrected": "p.~2: ``The potential outcomes are such that $I_{ij}=1$ if and only if $\\hl{i\\in G_j\\cup\\{j\\}}$, and, consequently, $d_{ij}=1$ if and only if $\\hl{(G_i\\cup\\{i\\})\\cap(G_j\\cup\\{j\\})\\ne\\emptyset}$.''\n\np.~7: ``We have $I_{ij}=1$ if and only if $\\hl{i=j\\text{, or }i=1\\text{ and }j\\le a_n}$, and thus $d_{ij}=1$ whenever $i,j\\le a_n$.''\n\nThe sets $G_i$ and the function $\\operatorname{bal}(G_i)$ are as printed.", "intended_meaning": "The sentences describe the interference between different units. The outcome of unit $j$ also depends on its own treatment $z_j$, and the article sets $I_{jj}=1$ for every unit (p.~680). The treatments that affect unit $j$ are those of the units in $G_j\\cup\\{j\\}$, and $d_{ij}=1$ when two such sets intersect. Read literally, the sentence on p.~2 gives $I_{jj}=0$, and it misses pairs that are linked through one of the two units: for $n=2$, $G_1=\\{2\\}$ and $G_2=\\{1\\}$, the intersection $G_1\\cap G_2$ is empty, although $d_{12}=1$. The calculations that follow use the convention of the article: $C_1=d_{\\mathrm{AVG}}\\approx a_n$ on p.~3, the values of $C_1$ and $d_{\\mathrm{AVG}}$ on p.~7, and Section 2, which recalls on p.~9 that $I_{ij}=1$ when $i=j$. No reported quantity changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E021", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the paragraph that considers changing the assignment of a single unit i assigned to control under the Bernoulli design. A red box marks 'bal(G_j) = 1', the value said to follow from bal(G_j) = 0, and a second red box marks 'bal(G_j) = 0', the value said to follow from bal(G_j) = -1.", "caption": "Supplement B, p. 6, paragraph beginning “To understand this behavior”. The marked values are those of $\\operatorname{bal}(G_j)$ after unit $i$ moves from control to treatment; they should be $2$ and $1$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E022.png"}, {"alt": "Crop of the paragraph on complete and paired randomization. Red boxes mark four phrases: 'add one' (to bal(G_j) for the units in script G_i), 'remove one' (from bal(G_j) for units in script G_l), 'one added' and 'one removed'.", "caption": "Supplement B, p. 6, paragraph beginning “Now consider the same situation”. Each marked phrase states a change of one where the change is two.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E022-b.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T32", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T32"}], "headline": "The explanation on p. 6 says that changing one unit’s assignment moves $\\operatorname{bal}(G_j)$ by one; it moves it by two.", "id": "P009-E022", "location": "Supplement B, Section 1.2, p. 6, paragraphs beginning “To understand this behavior” and “Now consider the same situation”", "pdf_page": 6, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T32"], "text": {"as_printed": "``Of the units in $\\mathcal{G}_i$, some fraction will have $\\operatorname{bal}(G_j)=0$. When we change $i$'s assignment, those units will change to $\\operatorname{bal}(G_j)=\\hl{1}$, and thus change the values of their outcomes. Similarly, some other fraction of $\\mathcal{G}_i$ will have $\\operatorname{bal}(G_j)=-1$, and their values will change to $\\operatorname{bal}(G_j)=\\hl{0}$.''\n\nNext paragraph: ``\\ldots\\ when we change $i$'s assignment from control to treatment and add \\hl{one} to $\\operatorname{bal}(G_j)$ for the units in $\\mathcal{G}_i$, a counteracting effect happens because we simultaneously change $\\ell$'s assignment from treatment to control and remove \\hl{one} from $\\operatorname{bal}(G_j)$ for units in $\\mathcal{G}_\\ell$. The number of units that get \\hl{one} added to $\\operatorname{bal}(G_j)$ will, with high probability and relative to the sample size, be close to the number of units that get \\hl{one} removed \\ldots''", "corrected": "``Of the units in $\\mathcal{G}_i$, some fraction will have $\\operatorname{bal}(G_j)=0$. When we change $i$'s assignment, those units will change to $\\operatorname{bal}(G_j)=\\hl{2}$, and thus change the values of their outcomes. Similarly, some other fraction of $\\mathcal{G}_i$ will have $\\operatorname{bal}(G_j)=-1$, and their values will change to $\\operatorname{bal}(G_j)=\\hl{1}$.''\n\nNext paragraph: ``\\ldots\\ when we change $i$'s assignment from control to treatment and add \\hl{two} to $\\operatorname{bal}(G_j)$ for the units in $\\mathcal{G}_i$, a counteracting effect happens because we simultaneously change $\\ell$'s assignment from treatment to control and remove \\hl{two} from $\\operatorname{bal}(G_j)$ for units in $\\mathcal{G}_\\ell$. The number of units that get \\hl{two} added to $\\operatorname{bal}(G_j)$ will, with high probability and relative to the sample size, be close to the number of units that get \\hl{two} removed \\ldots''", "evidence": "The evidence is a derivation from the definition of the balance. The coefficient of $z_j$ in the outcome of unit $j$ is $0$, $1$ or $2$ according as the balance is negative, zero or positive. An increase of two changes this coefficient exactly when the balance starts at $-2$, $-1$ or $0$, and a decrease of two changes it exactly when the balance starts at $0$, $1$ or $2$. The simulation code in Supplement C computes the balance from the terms $2Z_l-1$, and the simulations do not contain the slip.", "problem": "The balance is $\\operatorname{bal}(G_j)=\\sum_{l\\in G_j}(2Z_l-1)$ (p.~2). When unit $i\\in G_j$ moves from control to treatment, its term changes from $-1$ to $+1$, and the balance increases by two. The values after the change are $2$ and $1$, not $1$ and $0$, and the second paragraph should add and remove two.", "what_survives": "The qualitative explanation is unchanged: the sign of the balance changes for some units, and moving a second unit in the opposite direction counteracts the first change. The set $\\{0,1\\}$ in the second paragraph can stay, since the sentence speaks of some fraction of the units; the starting values from which a decrease changes the outcome are $0$, $1$ and $2$. I did not check any statement about the size of these fractions."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E022", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of five lines of text between two displays. A red box marks the words 'xi i j of 0 equals 1' in the sentence 'Hence, the units are unaffected by spillover effects when they are not treated, so xi i j of 0 equals 1 for all i and j'.", "caption": "Supplement B, Section 1.4, p. 8. The marked value of $\\xi_{ij}(0)$ is printed as 1.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E023.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T33", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T33"}], "headline": "In the illustration of Section 1.4 of Supplement B, the spillover effect on an untreated unit is $\\xi_{ij}(0)=0$, not $1$.", "id": "P009-E023", "location": "Supplement B, Section 1.4, p. 8 (PDF p. 8 of Supplement B), paragraph beginning “Recall that”", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T33"], "text": {"as_printed": "``Recall that $y_i(0;\\mathbf{z}_{-i})=X_i+\\varepsilon_i$ for all units. Hence, the units are unaffected by spillover effects when they are not treated, so $\\xi_{ij}(0)=\\hl{1}$ for all $i$ and $j$.''", "corrected": "``Hence, the units are unaffected by spillover effects when they are not treated, so $\\xi_{ij}(0)=\\hl{0}$ for all \\hl{distinct} $i$ and $j$.''", "evidence": "The evidence is a derivation from the definition and a check of the page. In the illustration, the two potential outcomes whose difference defines $\\xi_{ij}(0)$ both equal $X_i+\\varepsilon_i$, and their difference is identically zero. The display for $B_1$ that follows the sentence is the one obtained with $\\xi_{ij}(0)=0$: each of its $\\lfloor a_n\\rfloor-1$ nonzero summands equals $2\\breve{Y}_1\\cdot 2=2+4X_1+4\\varepsilon_1$.", "problem": "The sentence states the reason for the value zero and prints the value one. The article defines $\\xi_{ij}(z)$ on p.~690, for distinct units $i$ and $j$, as the spillover effect from unit $j$ to unit $i$ when the treatment of unit $i$ is held at $z$. If an untreated unit is unaffected by the treatments of the other units, then this effect is zero.", "what_survives": "This correction leaves the printed expression for $B_1$ unchanged: it already uses the value zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E023", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop from the display of B 2 as a double sum over i and j from 2 to floor of a n, through the display of B 1. A red box marks the numerator, floor of a n times floor of a n minus 1, of the fraction printed for B 2 after the words 'It follows that'.", "caption": "Supplement B, Section 1.4, p. 8. The double sum for $B_2$ starts at 2 in both indices; the marked numerator of the closed form counts $\\lfloor a_n\\rfloor(\\lfloor a_n\\rfloor-1)$ pairs.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E024.png"}, {"alt": "Crop of the display that states the limit in probability of the normalized difference between the variance estimator and the variance. A red box marks the last term of the numerator on the right side, floor of a n times floor of a n minus 1.", "caption": "Supplement B, Section 1.4, p. 8, display after “Taken together, we can conclude that”. The marked term of the numerator carries the same count.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E024-b.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T21", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T21"}, {"id": "P009-T22", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T22"}], "headline": "In the illustration of Section 1.4 of Supplement B, with $b=\\lfloor a_n\\rfloor$, the term $B_2$ equals $(b-1)(b-2)/\\{n+b(b-1)\\}$, not $b(b-1)/\\{n+b(b-1)\\}$, and the right side of the displayed limit becomes $\\{n-(b-1)^2\\}/\\{n+b(b-1)\\}$; the three limiting regimes are unchanged.", "id": "P009-E024", "location": "Supplement B, Section 1.4, p. 8 (PDF p. 8 of Supplement B), display after “It follows that” and display after “Taken together, we can conclude that”", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T21", "P009-T22"], "text": {"as_printed": "``Taken together, we have\n\\[\nB_2=\\frac{1}{nd_{\\mathrm{AVG}}}\\sum_{i=2}^{\\lfloor a_n\\rfloor}\\sum_{j=2}^{\\lfloor a_n\\rfloor}\\mathbf{1}[i\\ne j]\\operatorname{Cov}\\bigl(Y_i,Y_j\\mid Z_i=1,Z_j=1\\bigr)\n\\]\nConditional on $Z_i=1$, we have $Y_i=2Z_1+X_i+\\varepsilon_i$ for all $2\\le i\\le a_n$. Hence, for all $2\\le i\\ne j\\le a_n$,\n\\[\n\\operatorname{Cov}\\bigl(Y_i,Y_j\\mid Z_i=1,Z_j=1\\bigr)=4\\operatorname{Var}(Z_1)=1.\n\\]\nIt follows that\n\\[\nB_2=\\frac{\\hl{\\lfloor a_n\\rfloor\\bigl(\\lfloor a_n\\rfloor-1\\bigr)}}{n+\\lfloor a_n\\rfloor\\bigl(\\lfloor a_n\\rfloor-1\\bigr)}.\n\\]\n\\ldots\\ Taken together, we can conclude that\n\\[\nnd_{\\mathrm{AVG}}^{-1}\\Bigl[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})-\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\Bigr]\\xrightarrow{p}\\frac{n-\\bigl(\\lfloor a_n\\rfloor-1\\bigr)-\\hl{\\lfloor a_n\\rfloor\\bigl(\\lfloor a_n\\rfloor-1\\bigr)}}{n+\\lfloor a_n\\rfloor(\\lfloor a_n\\rfloor-1)}.\\text{''}\n\\]", "corrected": "Let $b=\\lfloor a_n\\rfloor$, and let the assignments be independent with probability $1/2$, as in the supplement's calculation. The units affected by the treatment of unit 1 are $2,\\ldots,b$, which gives $(b-1)(b-2)$ ordered pairs of distinct units. It follows that\n\\[\nB_2=\\frac{\\hl{(b-1)(b-2)}}{n+b(b-1)}.\n\\]\nUnder $X_1+\\varepsilon_1=-1/4$, $B_1=(b-1)/\\{n+b(b-1)\\}$ as printed, and\n\\[\n\\frac{\\tau_{\\mathrm{MSQ}}}{d_{\\mathrm{AVG}}}-B_1-B_2=\\frac{n-(b-1)-\\hl{(b-1)(b-2)}}{n+b(b-1)}=\\frac{n-(b-1)^2}{n+b(b-1)}.\n\\]\nFor every $n$ and every $1\\le b\\le n$, this is the exact value of $nd_{\\mathrm{AVG}}^{-1}\\{\\mathbb{E}[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})]-\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\}$. Taken together,\n\\[\nnd_{\\mathrm{AVG}}^{-1}\\Bigl[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})-\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\Bigr]-\\frac{n-\\hl{(b-1)^2}}{n+b(b-1)}\\xrightarrow{p}0.\n\\]\nThe printed right side differs from this one by $2(b-1)/\\{n+b(b-1)\\}\\le n^{-1/2}$, so the printed display is also true in this sense, and the three regimes are as printed.", "evidence": "The evidence is a count, a direct derivation and an exact enumeration. Let $m=b-1$ and $D=nd_{\\mathrm{AVG}}=n+b(b-1)$. The count gives $B_2=m(m-1)/D$. Under $X_1+\\varepsilon_1=-1/4$, the supplement's values $B_1=m/D$ and $\\tau_{\\mathrm{MSQ}}=1$ are correct, and the right side in Proposition 7 is\n\\[\n\\frac{\\tau_{\\mathrm{MSQ}}}{d_{\\mathrm{AVG}}}-B_1-B_2=\\frac{n-m-m(m-1)}{D}=\\frac{n-m^2}{D}.\n\\]\nA direct computation of $\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})$ and of $\\mathbb{E}[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})]$ under independent assignments with probability $1/2$, which does not use Proposition 7, gives\n\\[\n\\frac{n}{d_{\\mathrm{AVG}}}\\Bigl\\{\\mathbb{E}\\bigl[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})\\bigr]-\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\Bigr\\}=\\frac{n-2m\\{1+2(X_1+\\varepsilon_1)\\}-m(m-1)}{D},\n\\]\nwhich equals $(n-m^2)/D$ when $X_1+\\varepsilon_1=-1/4$.\n\nFor $n=6$, $b=4$, $X_1+\\varepsilon_1=-1/4$ and $X_i+\\varepsilon_i=1$ for $i\\ge2$, enumerating the $64$ assignments in rational arithmetic, I find $D=18$, $B_1=1/6$, $B_2=1/3$, $\\mathbb{E}[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})]=229/144$ and $\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})=241/144$. The normalized expected difference is $-1/6$. According to the printed formulas, $B_2=2/3$ and the right side is $-1/2$. The same enumeration for every $1\\le b\\le n\\le 8$ ($36$ cases) agrees with the corrected formulas. Four of these cases are shown below.\n\n\\begin{audittable}{rrrrrr}{Printed and exact values in the illustration, with $X_1+\\varepsilon_1=-1/4$ and $X_i+\\varepsilon_i=1$ for $i\\ge2$.}\n$n$ & $b$ & $B_2$, printed & $B_2$, exact & right side of the limit, printed & normalized expected difference, exact \\\\\n$5$ & $3$ & $6/11$ & $2/11$ & $-3/11$ & $1/11$ \\\\\n$6$ & $4$ & $2/3$ & $1/3$ & $-1/2$ & $-1/6$ \\\\\n$8$ & $5$ & $5/7$ & $3/7$ & $-4/7$ & $-2/7$ \\\\\n$8$ & $8$ & $7/8$ & $21/32$ & $-55/64$ & $-41/64$ \\\\\n\\end{audittable}\n\nThe corrected right side exceeds the printed one by $\\delta_n=2m/D$. The identity $D^2-4nm^2=(n-m^2)^2+2nm+2m^3+m^2$ implies that $D^2\\ge4nm^2$, and I conclude that $0\\le\\delta_n\\le n^{-1/2}$ for all $n$ and all $1\\le b\\le n$.", "problem": "Let $b=\\lfloor a_n\\rfloor$. The double sum runs over the ordered pairs $(i,j)$ such that $2\\le i\\ne j\\le b$. There are $b-1$ such indices and $(b-1)(b-2)$ such ordered pairs, and each covariance equals one. The printed numerator $b(b-1)$ is the number of ordered pairs of distinct indices among $1,\\ldots,b$, which includes unit 1, although the supplement notes a few lines earlier that the covariance is zero whenever $i=1$ or $j=1$. The printed value of $B_2$ is too large by $2(b-1)/\\{n+b(b-1)\\}$, and the same excess enters the last term of the numerator in the displayed limit.\n\nThe excess is at most $n^{-1/2}$. The displayed limit keeps $n$ on its right side and is read as the statement that the difference of the two sides tends to zero in probability. In that sense it remains true, and the three regimes discussed after it are unchanged. What is wrong is the exact value of $B_2$ and, with it, the exact value of the normalized expected difference $nd_{\\mathrm{AVG}}^{-1}\\{\\mathbb{E}[\\widehat{\\operatorname{Var}}_{\\mathrm{BER}}(\\hat\\tau_{\\mathrm{HT}})]-\\operatorname{Var}(\\hat\\tau_{\\mathrm{HT}})\\}$ at a finite sample size.", "what_survives": "The three limits are unchanged: one when $a_n=o(n^{0.5})$, minus one when $n=o(a_n^2)$, and $(1-c^2)/(1+c^2)$ when $a_n/n^{0.5}$ tends to $c>0$. The displayed limit also remains true in the sense above, since its right side differs from the corrected one by at most $n^{-1/2}$. Proposition 7 defines $B_2$ by the correct sum, and its proof does not use this illustration. The simulation code excludes unit 1 from the units that unit 1 affects and does not use the closed form for $B_2$. The reported bias, standard errors and root mean square errors do not depend on the count. At a finite sample size the sign of the normalized expected difference can differ from the sign of the printed right side, as in the first row of the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E024", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the title 'Table B1' and its two-line caption. A red box marks the first word, 'Bias', of 'Bias of the HT and HA estimators with respect to EATE in the simulation study when the interference is contained within groups of units as described in Section 1.1'.", "caption": "Supplement B, caption of Table B1, p. 14. The marked first word of the caption is “Bias”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E025.png"}, {"alt": "Crop of the lower part of Table B1. Red boxes mark the row labelled n at the end of Panel B, with entries 0.997, 1.000, 1.000, 1.000 for each estimator, and the row labelled n at the end of Panel C, with entries 0.999, 1.000, 1.000, 1.000 for each estimator. Below the table is the note 'Each cell presents the result from 50,000 draws from the data generating process'.", "caption": "Supplement B, Table B1, p. 14: the last rows of Panel B (complete randomization), all of Panel C (paired randomization) and the note. The two marked rows, for $a_n=n$, have entries from 0.997 to 1.000.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E025-b.png"}, {"alt": "Crop of the title 'Table B3' and its two-line caption. A red box marks the first word, 'Bias'.", "caption": "Supplement B, caption of Table B3, p. 16. The marked first word is “Bias”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E025-c.png"}, {"alt": "Crop of the title 'Table B5' and its two-line caption. A red box marks the first word, 'Bias'.", "caption": "Supplement B, caption of Table B5, p. 18. The marked first word is “Bias”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E025-d.png"}, {"alt": "Crop of the title 'Table B7' and its two-line caption. A red box marks the first word, 'Bias'.", "caption": "Supplement B, caption of Table B7, p. 20. The marked first word is “Bias”.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E025-e.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T25", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T25"}], "headline": "Tables B1, B3, B5 and B7 of Supplement B report the absolute value of the Monte Carlo bias, and their captions should begin “Absolute bias”; every printed entry is unchanged.", "id": "P009-E025", "location": "Supplement B, captions of Tables B1, B3, B5 and B7, pp. 14, 16, 18 and 20 (the same PDF pages of Supplement B); replication files, make_stats_df.R", "pdf_page": 14, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T25"], "text": {"as_printed": "Table B1, p.~14: ``\\hl{Bias} of the HT and HÁ estimators with respect to EATE in the simulation study when the interference is contained within groups of units as described in Section 1.1.''\n\nTable B3, p.~16: ``\\hl{Bias} of the HT and HÁ estimators with respect to EATE when the interference structure is randomly generated using the unweighted balance function as described in Section 1.2.''\n\nTable B5, p.~18: ``\\hl{Bias} of the HT and HÁ estimators with respect to EATE when the interference structure is randomly generated using the weighted balance function as described in the end of Section 1.2.''\n\nTable B7, p.~20: ``\\hl{Bias} of the HT and HÁ estimators with respect to EATE when only one unit is interfering with other units as described in Section 1.3.''\n\nEach table has the note ``Note: Each cell presents the result from 50,000 draws from the data generating process.'' In Tables B1 and B3, for both estimators and for $n=10^2,10^3,10^4,10^5$, the row $a_n=n$ reads $0.997$, $1.000$, $1.000$, $1.000$ under complete randomization and $0.999$, $1.000$, $1.000$, $1.000$ under paired randomization.\n\nReplication files, \\texttt{make\\_stats\\_df.R}:\n\n\\texttt{get\\_bias <- function(res\\_mat) \\{ abs(rowMeans(res\\_mat - 1)) \\}}", "corrected": "Tables B1, B3, B5 and B7: ``\\hl{Absolute bias} of the HT and HÁ estimators with respect to EATE \\ldots'', with the rest of each caption as printed. A common note may add: ``Absolute bias is $|R^{-1}\\sum_{r=1}^R(\\hat\\tau_r-1)|$, where $R=50{,}000$.'' The entries are unchanged.", "evidence": "The evidence is a check of the code and of the pages, a recalculation from the cached results, and an exact calculation. Regenerating the eight tables of bias and standard errors from the cached results in the replication files reproduces all $1{,}104$ printed cells at the printed precision. For the exact calculation, let $a_n=n$ with $n$ even. Every assignment under complete or paired randomization has $n/2$ treated units, every observed outcome equals its baseline $X_i+\\varepsilon_i$, and both estimators equal $(2/n)\\sum_{i=1}^n(2Z_i-1)(X_i+\\varepsilon_i)$, whose expectation is $0$. Since EATE is $1$, the bias is $-1$ for every even $n$. The cached results hold the magnitudes only, and the signs of the Monte Carlo biases of the original runs were not recovered.", "problem": "The function that produces the entries takes the absolute value of the mean estimation error over the $R=50{,}000$ draws, $|R^{-1}\\sum_{r=1}^R(\\hat\\tau_r-1)|$. This is the size of the Monte Carlo bias without its sign, and the captions call it ``Bias''. The difference matters most in the rows $a_n=n$ under complete and paired randomization, where the entries near $1$ correspond to a bias of $-1$. The printed magnitudes are correct, and the tables of nonnegative entries are naturally read as sizes of bias, which is the intended meaning.", "what_survives": "All printed magnitudes, the tables of standard errors (B2, B4, B6 and B8), the root mean square errors and the figures are unaffected, and no conclusion of the paper uses the sign of these entries. The rows $a_n=n$ lie outside the restricted-interference assumption, so a bias of $-1$ there does not contradict any proposition."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E025", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph beginning 'The primary focus in the simulations is the root mean square error (RMSE) of the Hajek estimator'. A red box marks its last sentence: 'The bias is negligible relative to the variance for the settings and sample sizes we consider here.'", "caption": "Supplement B, p. 3, the paragraph after Figure B1. The marked sentence covers every setting in the figures and tables that the paragraph introduces.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E026.png"}, {"alt": "Crop of Panel B of Table B1, with the bias of the Horvitz-Thompson and Hajek estimators for n = 100, 1000, 10000 and 100000 and six sequences a_n. In the rows for a_n = 1, 25, 8 n^0.25, 2.5 n^0.5 and 0.25 n the entries are at most 0.041. A red box marks the last row, a_n = n, with entries 0.997, 1.000, 1.000, 1.000 for each estimator.", "caption": "Supplement B, p. 14: Panel B (complete randomization) of Table B1. In the marked row $a_n=n$ the bias of both estimators is $0.997$ or $1.000$, and the estimand is $\\tau_{\\mathrm{EATE}}=1$. Panel C and Table B3 have the same pattern.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-E026-tableB1.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T23", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T23"}, {"id": "P009-T24", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T24"}], "headline": "The summary that the bias is negligible relative to the variance does not hold under complete interference ($a_n=n$) with complete or paired randomization, where both estimators have expectation $0$ and EATE is $1$; the sentence needs this exception.", "id": "P009-E026", "location": "Supplement B, p. 3, paragraph after Figure B1, last sentence; the rows concerned are $a_n=n$ in Panels B and C of Tables B1–B4 (pp. 14–17)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T23", "P009-T24"], "text": {"as_printed": "``The bias is negligible relative to the variance \\hl{for the settings and sample sizes we consider here}.''\n\nTables B1 and B3, row $a_n=n$, for $n=10^2,10^3,10^4,10^5$ and for both estimators: bias $0.997$, $1.000$, $1.000$, $1.000$ in Panel B (complete randomization) and $0.999$, $1.000$, $1.000$, $1.000$ in Panel C (paired randomization). Tables B2 and B4, same row: standard error $0.440$, $0.139$, $0.044$, $0.014$ in Panel B and $0.404$, $0.127$, $0.040$, $0.013$ in Panel C.", "corrected": "``The bias is negligible relative to the variance for the settings and sample sizes we consider here\\hl{, except under complete interference with the unweighted balance function and complete or paired randomization. In that exception, both estimators have expectation zero while EATE is one, so their signed bias is $-1$.}''\n\nA sentence may also be added to the paragraph on p.~6 that begins ``The extreme case'': ``Despite the unchanged standard error, in this extreme case both estimators are centered at zero rather than at EATE.''", "evidence": "The evidence is a derivation from the data generating process on p.~2 of the supplement, together with the printed tables. Under complete interference $G_i=\\mathbf{U}\\setminus\\{i\\}$, and under either design exactly $n/2$ units are treated. Substituting into the definition of the balance, I obtain\n\\[\n\\operatorname{bal}(G_i)=\\sum_{j\\ne i}(2Z_j-1)=1-2Z_i.\n\\]\nA treated unit has balance $-1$ and outcome $X_i+\\varepsilon_i$. A control unit has balance $+1$ and outcome $2\\cdot0+X_i+\\varepsilon_i$. Every observed outcome is $X_i+\\varepsilon_i$, and both estimators equal $(2/n)\\sum_i(2Z_i-1)(X_i+\\varepsilon_i)$, which has expectation zero because $\\Pr(Z_i=1)=1/2$ given the covariates. With the other assignments held fixed, the effect of unit $i$'s own treatment is $2(1-Z_i)$, which implies that $\\tau_{\\mathrm{ATE}}(\\mathbf{Z})=1$ and $\\tau_{\\mathrm{EATE}}=1$. The bias is exactly $-1$. The variance of the estimators is $58/(3n)$ under complete randomization and $49/(3n)+36/[n(n+1)(n+2)]$ under paired randomization. The root mean square error with respect to EATE then tends to one.\n\\begin{audittable}{llrrr}{Row $a_n=n$: the printed bias (Tables B1 and B3) and standard error (Tables B2 and B4), and the standard deviation from the exact variance.}\nDesign & $n$ & Printed bias & Printed standard error & Exact standard deviation \\\\\nComplete & $10^2$ & $0.997$ & $0.440$ & $0.4397$ \\\\\nComplete & $10^3$ & $1.000$ & $0.139$ & $0.1390$ \\\\\nComplete & $10^4$ & $1.000$ & $0.044$ & $0.0440$ \\\\\nComplete & $10^5$ & $1.000$ & $0.014$ & $0.0139$ \\\\\nPaired & $10^2$ & $0.999$ & $0.404$ & $0.4042$ \\\\\nPaired & $10^3$ & $1.000$ & $0.127$ & $0.1278$ \\\\\nPaired & $10^4$ & $1.000$ & $0.040$ & $0.0404$ \\\\\nPaired & $10^5$ & $1.000$ & $0.013$ & $0.0128$ \\\\\n\\end{audittable}\nIn every other row of Tables B1--B8 the rounded bias is less than a tenth of the rounded standard error; the largest ratio is $0.099$.", "problem": "The paragraph introduces the figures and the tables at the end of the supplement and summarizes ``these results''. Its last sentence covers the rows $a_n=n$ of the tables as well. In these rows, under complete and paired randomization, the tabulated bias is about one, which is the size of the estimand itself. From the rounded entries, it is more than twice the standard error at $n=10^2$ and about $71$ and $77$ times the standard error at $n=10^5$. The two estimators concentrate at $0$ and not at $\\tau_{\\mathrm{EATE}}=1$. The low standard errors in these rows do not indicate accurate estimation. The tables print the bias in absolute value; its sign is the subject of \\fid{P009-E025}.", "what_survives": "The tabulated bias and standard errors in these rows need no correction, and Figures B2 and B3 do not show $a_n=n$. The statement on p.~6 that the outcomes and the standard error are unchanged in this case is also correct, although it does not mention the bias. No result of the article is contradicted. Here $d_{\\mathrm{AVG}}=n$ and Assumption 2 fails, and the article itself has a related example on p.~684. Only the scope of the summary sentence changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-E026", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the paragraph beginning 'Previous investigations under unknown interference'. A red box marks the words 'compilers and noncompilers' at the end of the sentence on Sobel (2006).", "caption": "Section 2, p. 675 (PDF p. 3), the sentence on Sobel (2006). The marked words are the misprint.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "“Compilers and noncompilers” should read “compliers and noncompliers”.", "id": "P009-N001", "location": "Section 2, p. 675 (PDF p. 3), paragraph beginning “Previous investigations”, sentence on Sobel (2006)", "pdf_page": 3, "printed_page": "675", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Sobel (2006) derives the expectation of an instrumental variables estimator used in housing mobility experiments under unknown interference, and he shows that it is a mixture of primary and spillover effects for \\hl{compilers} and \\hl{noncompilers}.''", "corrected": "``\\ldots\\ and he shows that it is a mixture of primary and spillover effects for \\hl{compliers} and \\hl{noncompliers}.''", "evidence": "I checked the spelling only and did not compare the sentence with Sobel (2006).", "problem": "The sentence describes an instrumental variables estimator, whose two compliance groups are the compliers and the noncompliers. Both words are misspelled.", "what_survives": "Nothing else is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-N001", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first six lines of Remark 1. Red boxes mark the words 'would suggest slower rates of convergence than given by Proposition 3', which run over two lines, in the sentence that begins 'The current proposition applied to that design'.", "caption": "Remark 1, p. 688 (PDF p. 16). The marked words compare the rates of Proposition 5 and Proposition 3 under complete randomization.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "Proposition 5 does not recover Proposition 3 in full generality and can give slower rates under complete randomization, but the comparison depends on the interference structure and on the outcome moments; with bounded outcomes Proposition 5 can give the smaller bound.", "id": "P009-N002", "location": "Remark 1, p. 688 (PDF p. 16), third sentence", "pdf_page": 16, "printed_page": "688", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Remark 1. The convergence results for Bernoulli and paired randomization presented in the previous subsections can be proven as consequences of Proposition 5. This is not the case for complete randomization. The current proposition applied to that design \\hl{would suggest slower rates of convergence than given by Proposition 3}. This highlights that Proposition 5 provides worst-case rates for all designs that satisfy the stated conditions.''", "corrected": "``The current proposition \\hl{does not recover Proposition 3 in its full generality and can suggest slower rates under complete randomization; the comparison depends on the interference structure and outcome moments}.''", "intended_meaning": "The remark compares two general guarantees. Its point is that Proposition 3, the result for complete randomization, cannot be deduced from Proposition 5 under the weak moment conditions that Section 5.2 needs ($q=2$ and $s=1$ suffice there, as p.~682 says). This is correct: a block construction in which only second moments are bounded satisfies the conditions of Proposition 3 and violates Assumption 5.\n\nThe third sentence is not an ordering of the two bounds that holds for every interference structure. Let the units be divided into blocks of size $a$, with $n/2$ treated units under complete randomization, and let the potential outcomes be the bounded functions $y_i(\\mathbf{z})=z_i+a^{-1}\\sum_{j\\in B(i)}z_j$, where $B(i)$ is the block of unit $i$, so that $d_{\\mathrm{AVG}}=C_1=a$. For this family, exact formulas for the laws of the block counts imply that\n\\[\n\\alpha_{\\mathrm{EXT}}\\le\\frac{a}{4}\\qquad\\text{and}\\qquad\\frac{\\alpha_{\\mathrm{INT}}}{n}\\le\\frac{(a-1)^{1/2}}{4(n-a+1)} .\n\\]\nFor even $t$, if $n=t^4$ and $a=t$, then both propositions apply, the bound of Proposition 3 is of order $n^{-1/4}$, and the bound of Proposition 5 is of order $n^{-3/8}$. If $n=t^2$ and $a=t$, then Proposition 5 gives consistency with a bound of order $n^{-1/4}$, and the condition $C_1=o(n^{0.5})$ of Proposition 3 fails. These are comparisons of upper bounds, and neither proposition is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-N002", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the last two lines of a displayed chain and the sentence after it. The first line reads: equals one over n times the sum over i of I sub rho(i) i times the sum over z from 0 to 1 of the absolute value of E of y i of (z; Z-tilde minus i) given Z i = z, minus E of y i of (z; Z-tilde minus i). The second line reads: is at most 2 k squared over n times the sum over i of I sub rho(i) i, which equals 2 k squared n to the minus 1 R SUM. A red outline surrounds the symbols is at most 2 k squared over n. The sentence below reads: where the second to last inequality follows from Lemma A4, and a second red outline surrounds it. The crop ends with the statement of Lemma A18.", "caption": "Supplement A, p. 17, end of the proof of Lemma A17. The outlines mark the constant $2k^2$ in the last line of the display and the sentence that attributes the step to Lemma A4.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-N003.png"}], "formal_release": null, "formal_statements": [{"id": "P009-T34", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T34"}, {"id": "P009-T44", "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-T44"}], "headline": "The constant $2k^2$ in the last step of the proof of Lemma A17 does not follow from Lemma A4, which gives $4k^2$ for the two terms; the bound holds by a direct argument under paired randomization.", "id": "P009-N003", "location": "Supplement A, p. 17 (PDF p. 17 of the supplement), proof of Lemma A17, last line of the final display and the sentence after it", "pdf_page": 17, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P009-T34", "P009-T44"], "text": {"as_printed": "\\[\n\\begin{aligned}\n&= \\frac{1}{n}\\sum_{i=1}^n I_{\\rho(i)i}\\sum_{z=0}^1 \\Bigl|\\mathrm{E}[y_i(z;\\widetilde{\\mathbf{Z}}_{-i}) \\mid Z_i = z] - \\mathrm{E}[y_i(z;\\widetilde{\\mathbf{Z}}_{-i})]\\Bigr| \\\\\n&\\leq \\frac{2k^2}{n}\\sum_{i=1}^n I_{\\rho(i)i} = 2k^2 n^{-1} R_{\\mathrm{SUM}}.\n\\end{aligned}\n\\]\nwhere the \\hl{second to last inequality follows from Lemma~A4}.", "corrected": "\\[\n\\begin{aligned}\n&= \\frac{1}{n}\\sum_{i=1}^n I_{\\rho(i)i}\\sum_{z=0}^1 \\Bigl|\\mathrm{E}[y_i(z;\\widetilde{\\mathbf{Z}}_{-i}) \\mid Z_i = z] - \\mathrm{E}[y_i(z;\\widetilde{\\mathbf{Z}}_{-i})]\\Bigr| \\\\\n&\\leq \\frac{2k^2}{n}\\sum_{i=1}^n I_{\\rho(i)i} = 2k^2 n^{-1} R_{\\mathrm{SUM}},\n\\end{aligned}\n\\]\nwhere the \\hl{last inequality holds because paired randomization has $\\Pr(Z_i = z) = 1/2$: each absolute difference is then at most $\\mathrm{E}|y_i(z;\\widetilde{\\mathbf{Z}}_{-i})| \\leq k$ by Assumption~1C, and the inner sum is at most $2k \\leq 2k^2$}.", "evidence": "The evidence is a derivation. Fix $i$ and $z$, and let $g_z = y_i(z;\\widetilde{\\mathbf{Z}}_{-i})$. Since $\\Pr(Z_i = z) = 1/2$ under paired randomization, $\\mathrm{E}[g_z]$ is the average of $\\mathrm{E}[g_z \\mid Z_i = z]$ and $\\mathrm{E}[g_z \\mid Z_i = 1 - z]$, and I have\n\\[\n\\begin{aligned}\n\\bigl|\\mathrm{E}[g_z \\mid Z_i = z] - \\mathrm{E}[g_z]\\bigr| &= \\frac{1}{2}\\bigl|\\mathrm{E}[g_z \\mid Z_i = z] - \\mathrm{E}[g_z \\mid Z_i = 1 - z]\\bigr| \\\\\n&\\leq \\frac{1}{2}\\bigl(\\mathrm{E}[|g_z| \\mid Z_i = z] + \\mathrm{E}[|g_z| \\mid Z_i = 1 - z]\\bigr) = \\mathrm{E}|g_z| \\leq k.\n\\end{aligned}\n\\]\nIn the last inequality, I used Assumption~1C, with Jensen's inequality since $s \\geq 1$. Summing over $z \\in \\{0, 1\\}$, I find that the inner sum is at most $2k$, and $2k \\leq 2k^2$ because Assumption~1A forces $k \\geq 2$. The terms with $I_{\\rho(i)i} = 0$ vanish, as the printed proof shows.", "problem": "Lemma~A4 (Supplement~A, p.~3) bounds $|\\mathrm{E}[y_i(z;\\mathbf{Z}_{-i}) \\mid Z_i = z] - \\mathrm{E}[y_i(z;\\mathbf{Z}_{-i})]|$ by $2k^2$ for a single value of $z$. The inner sum in the display has two terms, $z = 0$ and $z = 1$. Applying the lemma to each term, I obtain $4k^2$ and not the printed $2k^2$. The parallel step in the proof of Lemma~A14 (Supplement~A, p.~14) applies Lemma~A4 in the same way and carries the factor $4k^2$. The step in question is also the last inequality of the display, not the second to last. The printed bound is nevertheless correct, because under paired randomization each of the two differences is at most $k$.", "what_survives": "Lemma~A17 holds as stated, with its constant, and Proposition~4 uses only the order $n^{-1}R_{\\mathrm{SUM}}$. Only the justification of the last step changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-N003", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the paragraph beginning 'The primary focus in the simulations is the root mean square error (RMSE) of the Hajek estimator'. A red box marks the phrase 'especially for the Bernoulli design' in the sentence that compares the HT and Hajek estimators.", "caption": "Supplement B, p. 3, the paragraph after Figure B1. The marked phrase suggests a smaller difference under the other two designs, where there is none.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "“Especially for the Bernoulli design” understates the comparison: the two estimators are identical under complete and paired randomization and differ only under the Bernoulli design.", "id": "P009-N005", "location": "Supplement B, p. 3, paragraph after Figure B1, third sentence", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``To summarize these results, the HT estimator performs worse than the HÁ estimator, \\hl{especially for the Bernoulli design}, but the qualitative conclusions are the same.''", "corrected": "``To summarize these results, the HT estimator performs worse than the HÁ estimator \\hl{under the Bernoulli design; the estimators coincide under complete and paired randomization, and} the qualitative conclusions are the same.''", "intended_meaning": "The sentence says that the Horvitz--Thompson estimator is less precise and that the difference is concentrated in the Bernoulli design. Under complete and paired randomization every unit has the same treatment probability and the group sizes are fixed. The two estimators then coincide (Lemmas A16 and A19 of Supplement A). The tables agree: the columns of the two estimators are identical in Panels B and C of Tables B1--B8. Figures B5--B7 divide the root mean square error of each estimator by that estimator's own value under the Bernoulli design with $n=100$ and no interference. The curves in their Panels B and C lie lower than in Figures B2--B4, although the errors themselves are equal."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-N005", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop from the display of B 2 through the sentence 'The insights from this illustration extend to other values, but this assumption eases the derivations'. A red box marks the equation 'X 1 plus epsilon 1 equals minus one quarter'.", "caption": "Supplement B, Section 1.4, p. 8. The marked assumption sets $X_1+\\varepsilon_1$ to $-1/4$.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-N008.png"}, {"alt": "Crop of four lines of text below the definition of the potential outcomes. Red boxes mark the intervals '0 to 3' in 'an observed covariate that is uniformly distributed on the interval 0 to 3' and '0 to 7' in 'an error term that is uniformly distributed on the interval 0 to 7'.", "caption": "Supplement B, Section 1, p. 2. The marked intervals are the supports of $X_i$ and $\\varepsilon_i$ in the simulation.", "url": "https://pmaronow.github.io/llm-audit/works/P009/exhibits/P009-N008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The value $X_1+\\varepsilon_1=-1/4$ assumed in the illustration lies outside the range $[0,10]$ that the simulation’s data generating process allows; it is a separate simplifying choice for unit 1, and the limiting regimes hold for every value in that range.", "id": "P009-N008", "location": "Supplement B, Section 1.4, p. 8 (PDF p. 8 of Supplement B), sentence beginning “To ease the exposition”; Section 1, p. 2", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``To ease the exposition in this illustration, we will assume that the potential outcome for unit 1 is such that $\\hl{X_1+\\varepsilon_1=-1/4}$, in which case\n\\[\nB_1=\\frac{\\bigl(\\lfloor a_n\\rfloor-1\\bigr)}{n+\\lfloor a_n\\rfloor\\bigl(\\lfloor a_n\\rfloor-1\\bigr)}.\n\\]\nThe insights from this illustration extend to other values, but this assumption eases the derivations.''", "corrected": "``To ease the exposition in this illustration, we will assume that the potential outcome for unit 1 is such that $X_1+\\varepsilon_1=-1/4$ \\hl{(for this illustration only, this value replaces the baseline of unit 1 in the data generating process of Section 1, where $X_1+\\varepsilon_1$ lies in $[0,10]$)}, in which case\n\\[\nB_1=\\frac{\\bigl(\\lfloor a_n\\rfloor-1\\bigr)}{n+\\lfloor a_n\\rfloor\\bigl(\\lfloor a_n\\rfloor-1\\bigr)}.\n\\]\nThe insights from this illustration extend to \\hl{every value in $[0,10]$: for these values $0\\le B_1\\le21n^{-1/2}$, and the three limiting regimes are the same}.''", "intended_meaning": "The assumption is a normalization: it makes the coefficient $2+4X_1+4\\varepsilon_1$ in the general expression for $B_1$ equal to one. Supplement B, p.~2, specifies that $X_i$ is uniform on $[0,3]$ and $\\varepsilon_i$ is uniform on $[0,7]$. The value $-1/4$ therefore cannot occur in the simulation, and the text does not say that the illustration replaces the baseline of unit 1. Read as a separate fixed baseline for unit 1, the assumption is permissible, since the article takes the potential outcomes as fixed. The statement that the insights extend to other values is also correct. Let $c=X_1+\\varepsilon_1\\in[0,10]$ and $t=\\lfloor a_n\\rfloor-1$. Then $2+4c\\le42$ and $n+t(t+1)\\ge2n^{1/2}t$, and I have\n\\[\n0\\le B_1=\\frac{(2+4c)\\,t}{n+t(t+1)}\\le\\frac{21}{n^{1/2}}\n\\]\nuniformly in $a_n$ and $c$. The term $B_1$ vanishes in each of the three regimes for every such $c$, and the limits are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P009/#P009-N008", "work": {"arxiv_id": "1711.06399", "doi": "10.1214/20-AOS1973", "title": "Average treatment effects in the presence of unknown interference", "url": "https://pmaronow.github.io/llm-audit/works/P009/"}, "work_id": "P009"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 1. After the hypotheses, the display gives a lower bound E[Y | Y less than or equal to the quantile of Y given Z = z, S = 1 evaluated at one minus the sum over the other arms of Pr[S = 0 | Z = z'] divided by Pr[S = 1 | Z = z], Z = z], then the target E[Y(z) | S(1) = S(2) = ... = S(K) = 1], then an upper bound of the same form with Y greater than or equal to the quantile evaluated at the sum. Red outlines surround the two occurrences of Z = z at the end of the outer expectations. The last two lines say that these bounds hold when the sum is below 1, else the bounds are minus infinity and infinity.", "caption": "Proposition 1, p. 575 (PDF p. 4). The outlines mark the conditioning events of the two outer expectations, which contain $Z=z$ and not $S=1$; the quantile inside each expectation is that of $Y$ given $Z=z$ and $S=1$.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E001.png"}, {"alt": "Crop of the last eight lines of the proof of Proposition 1. The final bound reads: E[Y(1) | U = 1, S(1) = 1] is at most E[Y | Y greater than or equal to the quantile of Y given Z = 1, S = 1 evaluated at the sum over k from 2 to K of Pr[S = 0 | Z = k] over Pr[S = 1 | Z = k], Z = 1] when that sum is below 1, else the upper bound is infinite. A red outline surrounds Z = 1 at the end of the expectation.", "caption": "End of the proof of Proposition 1, Appendix A, p. 580 (PDF p. 9). The outline marks the conditioning event of the final expectation, which again lacks $S=1$.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T3", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T3"}, {"id": "P010-T4", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T4"}, {"id": "P010-T15", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T15"}], "headline": "The two outer expectations in the bounds of Proposition 1 must condition on $S=1$ as well as on $Z=z$; as printed they average over subjects who failed the check, and the inequalities can fail.", "id": "P010-E001", "location": "Proposition 1, displayed bounds, p. 575 (PDF p. 4); final display of its proof, Appendix A, p. 580 (PDF p. 9)", "pdf_page": 4, "printed_page": "575", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T15", "P010-T3", "P010-T4"], "text": {"as_printed": "\\textbf{Proposition 1.} Suppose that $\\Pr[S(1)=S(2)=\\cdots=S(K)=1]>0$ and that $Y$ is continuous with unbounded support. Let $Q_{Y\\mid Z=z,S=1}(.)$ denote the conditional quantile function of $Y$ given $Z=z$ and $S=1$. Then, sharp bounds for $\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]$ are given by\n\\[\n\\begin{aligned}\n&\\mathrm{E}\\left[Y\\mid Y\\le Q_{Y\\mid Z=z,S=1}\\left(1-\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}\\right),\\ \\hl{Z=z}\\right]\\\\\n&\\qquad\\le \\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]\\le\\\\\n&\\mathrm{E}\\left[Y\\mid Y\\ge Q_{Y\\mid Z=z,S=1}\\left(\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}\\right),\\ \\hl{Z=z}\\right]\n\\end{aligned}\n\\]\nwhen $\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}<1$, else these bounds are given by $-\\infty\\le\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]\\le\\infty$.", "corrected": "\\textbf{Proposition 1.} Suppose that Assumptions 1 to 3 hold, that $\\Pr[S(1)=S(2)=\\cdots=S(K)=1]>0$ and that $Y$ is continuous with unbounded support. Let $Q_{Y\\mid Z=z,S=1}(.)$ denote the conditional quantile function of $Y$ given $Z=z$ and $S=1$. Then, when\n\\[\n\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}<1,\n\\]\nsharp bounds for $\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]$ are given by\n\\[\n\\begin{aligned}\n&\\mathrm{E}\\left[Y\\mid Y\\le Q_{Y\\mid Z=z,S=1}\\left(1-\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}\\right),\\ Z=z,\\ \\hl{S=1}\\right]\\\\\n&\\qquad\\le \\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]\\le\\\\\n&\\mathrm{E}\\left[Y\\mid Y\\ge Q_{Y\\mid Z=z,S=1}\\left(\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}\\right),\\ Z=z,\\ \\hl{S=1}\\right].\n\\end{aligned}\n\\]\nThe means are taken to be finite, as under the regularity conditions of Section 2.1 (see \\fid{P010-E003}). In the notation of the derivation above, the two bounds are $\\frac{p_z}{L}\\int_0^{L/p_z}Q_z(u)\\,du$ and $\\frac{p_z}{L}\\int_{1-L/p_z}^1Q_z(u)\\,du$. The final display of the proof on p.~580 takes the same correction: its expectation conditions on the tail event, on $Z=1$ and on $\\hl{S=1}$. That display also needs the denominator correction to $\\hl{\\Pr[S=1\\mid Z=1]}$ described in \\fid{P010-E005}. For the case in which the sum is at least 1, see \\fid{P010-E002}.", "evidence": "The evidence is an exact counterexample to the printed display and a derivation of the corrected one.\n\n\\emph{Counterexample.} Let $K=2$, and let $Z$ be independent of the potential outcomes with $\\Pr[Z=1]=\\Pr[Z=2]=1/2$. Let $X$ be exponential with rate 1, and let $V$ have density $e^{-2|v|}$ on the real line. There are three groups of subjects. With probability $1/4$, $S(1)=S(2)=1$ and $Y(1)=-X$. With probability $1/4$, $S(1)=1$, $S(2)=0$ and $Y(1)=X$. With probability $1/2$, $S(1)=0$, $S(2)=1$ and $Y(1)=V$. In every group $Y(2)$ has density $e^{-|y|}/2$. Then $Y$ is continuous with support the real line and finite moments. The passing rates are $\\Pr[S=1\\mid Z=1]=1/2$ and $\\Pr[S=1\\mid Z=2]=3/4$. For $z=1$ the sum in the display, which I call the trimming ratio of the arm, is $(1/4)/(1/2)=1/2<1$. The passers in arm 1 have density $e^{-|y|}/2$, whose median is 0, and both quantiles in the display equal 0. A direct computation gives\n\\[\n\\mathrm{E}[Y\\mid Y\\le 0,\\ Z=1]=\\frac{-3/8}{1/2}=-\\frac34,\\qquad \\mathrm{E}[Y\\mid Y\\le 0,\\ Z=1,\\ S=1]=\\frac{-1/4}{1/4}=-1 .\n\\]\nThe target is $\\mathrm{E}[Y(1)\\mid S(1)=S(2)=1]=\\mathrm{E}[-X]=-1$. The printed lower bound asserts $-3/4\\le -1$, which is false. Exchanging $X$ and $-X$ between the first two groups leaves the distribution of $(Y,S,Z)$ unchanged and makes the target equal to 1, while the printed upper bound is $3/4$. The printed interval is $[-3/4,3/4]$ in both cases. With $S=1$ added the interval is $[-1,1]$, and the two examples attain its endpoints.\n\n\\emph{Derivation.} I write $A=\\{S(1)=\\cdots=S(K)=1\\}$, $p_k=\\Pr[S=1\\mid Z=k]$, $L=1-\\sum_{k=1}^K(1-p_k)$ and $Q_z=Q_{Y\\mid Z=z,S=1}$. The trimming ratio of arm $z$ equals $1-L/p_z$, and it is below 1 exactly when $L>0$. By the union bound, I have $\\Pr[A]\\ge L$. Since $A\\subseteq\\{S(z)=1\\}$, the subjects in $A$ are a fraction $r\\ge L/p_z$ of the passers in arm $z$. Among all subsets of these passers of fraction $r$, the mean outcome is at least $r^{-1}\\int_0^rQ_z(u)\\,du$ and at most $r^{-1}\\int_{1-r}^1Q_z(u)\\,du$. The first average is nondecreasing in $r$ and the second is nonincreasing, and I obtain\n\\[\n\\frac{p_z}{L}\\int_0^{L/p_z}Q_z(u)\\,du\\ \\le\\ \\mathrm{E}[Y(z)\\mid A]\\ \\le\\ \\frac{p_z}{L}\\int_{1-L/p_z}^1Q_z(u)\\,du .\n\\]\nFor a continuous outcome these two averages are the conditional means of the corrected display. The bounds are sharp. Let $A$ be the lowest, or the highest, fraction $L/p_z$ of the passers in arm $z$, and divide the remaining passers of that arm into disjoint sets of probabilities $1-p_k$, $k\\ne z$, on which $S(k)=0$ and the other indicators equal 1. Outside the passers of arm $z$, set $S(z)=0$ and the other indicators equal to 1, and draw the remaining potential outcomes from their observed distributions given passing or failing. This law has the observed distribution of $(Y,S)$ in every arm and attains the endpoint.", "problem": "The target is a mean over a subset of the subjects who pass in arm $z$, and the quantile function $Q_{Y\\mid Z=z,S=1}$ is that of those passers. The outer expectations, as printed, condition only on $Z=z$ and on the tail event. They average over every subject of arm $z$ whose outcome lies in the tail, including subjects who failed the check, and such a mean need not bound the target. The sentence after the proposition describes trimming ``the empirical distributions of subjects who pass the manipulation check'', and the replication code computes the bounds from passers only. The statement intended has $S=1$ in both conditioning events. The same omission occurs in the final display of the proof on p.~580, where the expectation conditions on the tail event and on $Z=1$ alone.", "what_survives": "The bounds for a trimming ratio below 1 survive in the form intended, and they are sharp. Adding $S=1$ changes no reported number, since the deposited functions already restrict the bound calculations to passers. This omission does not affect Lemma 1, Corollaries 1 to 5 or the comparisons of point estimates in Section 3. The branch in which the ratio is at least 1 is the subject of \\fid{P010-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E001", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 1. The first sentence reads: Suppose that Pr[S(1) = S(2) = ... = S(K) = 1] > 0 and that Y is continuous with unbounded support. After the displayed trimming bounds, the last two lines read: when the sum of Pr[S = 0 | Z = z'] over Pr[S = 1 | Z = z] is less than 1, else these bounds are given by minus infinity less than or equal to E[Y(z) | S(1) = S(2) = ... = S(K) = 1] less than or equal to infinity. Red outlines surround the words unbounded support, the symbol for minus infinity and the symbol for infinity.", "caption": "Proposition 1, p. 575 (PDF p. 4). The outlines mark the hypothesis on the support of $Y$ and the two infinite endpoints of the final clause.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E002.png"}, {"alt": "Crop of the last eight lines of the proof of Proposition 1. Red outlines surround the words else the upper bound is infinite at the end of the second line, and the same words, split across two lines, after the final display of the proof.", "caption": "End of the proof of Proposition 1, Appendix A, p. 580 (PDF p. 9). The outlines mark the two clauses that give an infinite upper bound when the ratio is at least 1.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T5", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T5"}, {"id": "P010-T6", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T6"}, {"id": "P010-T14", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T14"}, {"id": "P010-T15", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T15"}], "headline": "When the sum in the last clause of Proposition 1 is at least 1, the sharp bounds are the endpoints of the support of $Y$ among the subjects who pass in arm $z$; they are infinite only where that support is unbounded, which unbounded support of $Y$ does not imply.", "id": "P010-E002", "location": "Proposition 1, hypothesis and final clause, p. 575 (PDF p. 4); the two clauses “else the upper bound is infinite” in its proof, Appendix A, p. 580 (PDF p. 9)", "pdf_page": 4, "printed_page": "575", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T14", "P010-T15", "P010-T5", "P010-T6"], "text": {"as_printed": "\\textbf{Proposition 1.} Suppose that $\\Pr[S(1)=S(2)=\\cdots=S(K)=1]>0$ and that $Y$ is continuous with \\hl{unbounded support}. \\ldots\\ Then, sharp bounds for $\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]$ are given by \\ldots\\ when $\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}<1$, else these bounds are given by $\\hl{-\\infty}\\le\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]\\le\\hl{\\infty}$.\n\nProof, p.~580, twice: ``\\ldots\\ else the upper bound is \\hl{infinite}.''", "corrected": "\\textbf{Proposition 1, second case.} Suppose that Assumptions 1 to 3 hold, that $\\Pr[S(1)=S(2)=\\cdots=S(K)=1]>0$ and that $Y$ is continuous. \\hl{Restrict attention to compatible laws for which the target mean exists and is finite.} Let\n\\[\n\\hl{\\ell_z=\\operatorname{ess\\,inf}(Y\\mid Z=z,\\ S=1)},\\qquad \\hl{u_z=\\operatorname{ess\\,sup}(Y\\mid Z=z,\\ S=1)},\n\\]\nwhich may be infinite. When $\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\Pr[S=0\\mid Z=z']/\\Pr[S=1\\mid Z=z]\\ge 1$, sharp bounds for $\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]$ are given by\n\\[\n\\hl{\\ell_z}\\le\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]\\le\\hl{u_z}.\n\\]\nHere sharp means that every such law has its mean in this interval and that compatible finite means approach each finite endpoint or are unbounded in the direction of an infinite endpoint; a finite endpoint is not attained. The lower bound is $-\\infty$ only when the support of $Y$ given $Z=z$ and $S=1$ is unbounded below, and the upper bound is $\\infty$ only when it is unbounded above. In the proof on p.~580, ``else the upper bound is infinite'' becomes ``else the upper bound is \\hl{the essential supremum} of $Y$ given $Z=1$ and $S=1$''. The first case, in which the sum is below 1, is as corrected in \\fid{P010-E001}.", "evidence": "The evidence is an exact counterexample to the printed branch and a derivation of the corrected one.\n\n\\emph{Counterexample.} Let $K=2$, and let $Z$ be independent of the potential outcomes with $\\Pr[Z=1]=\\Pr[Z=2]=1/2$. Let $B$ be a Bernoulli variable with $\\Pr[B=1]=1/3$, and set $S(1)=S(2)=B$. If $B=1$, let $Y(1)=Y(2)$ have density $b(x)=30x^2(1-x)^2$ on $(0,1)$. If $B=0$, let $Y(1)=Y(2)$ have density $5/[2(1+|v|)^6]$ on the real line. Then the passing rate is $1/3$ in each arm, the trimming ratio is $(2/3)/(1/3)=2\\ge 1$, and $\\Pr[S(1)=S(2)=1]=1/3>0$. The observed $Y$ has a continuous density that is positive on the whole real line, with $\\mathrm{E}[Y]=1/6$, $\\mathrm{E}[Y^2]=13/63$ and $\\mathrm{E}|Y|^3=19/84$. All the assumptions of the proposition hold, and the printed bounds are $-\\infty$ and $\\infty$.\n\nConsider any law of the potential outcomes with this observed distribution. The subjects who would pass under both conditions pass in arm $z$, and the passers in arm $z$ have outcomes in $(0,1)$. This implies that\n\\[\n0<\\mathrm{E}[Y(z)\\mid S(1)=S(2)=1]<1 .\n\\]\nBoth endpoints are approached. Fix $t\\in(0,1)$ and let $F(t)=10t^3-15t^4+6t^5$ be the distribution function of $b$. Give the pairs $(S(1),S(2))=(1,1)$, $(1,0)$, $(0,1)$ and $(0,0)$ the probabilities $a$, $1/3-a$, $1/3-a$ and $1/3+a$, where $a=F(t)/3$. Let the outcomes in the first group follow $b$ conditional on $(0,t)$, let passing outcomes in the second and third groups follow $b$ conditional on $[t,1)$, and let failing outcomes follow the second density. This law has the same distribution of $(Y,S,Z)$, and the mean in its first group is below $t$. Using $(1-t,1)$ in place of $(0,t)$, I obtain a mean above $1-t$. For $t=1/3$ the two means are $73/306$ and $233/306$. The sharp bounds are therefore 0 and 1, and neither is attained.\n\n\\emph{Derivation.} In general, let the trimming ratio of arm $z$ be at least 1, and let $\\ell_z$ and $u_z$ be the essential infimum and supremum of $Y$ given $Z=z$ and $S=1$. Since every subject who would pass under all conditions passes in that arm, I have $\\ell_z\\le\\mathrm{E}[Y(z)\\mid S(1)=\\cdots=S(K)=1]\\le u_z$ whenever the mean is defined. Let $E$ be a bounded set of positive probability under the distribution of the passers in arm $z$. Then there exists a law with the same observed distribution such that the probability of passing under all conditions is positive, and such that the outcomes of the subjects who would pass under all conditions follow that distribution conditional on $E$. Letting $E$ approach either end of the support, I obtain means arbitrarily close to $\\ell_z$ and to $u_z$, or beyond any given value when the endpoint is infinite.", "problem": "The variable $Y$ is the observed response of all subjects. Its support can be the whole real line while the outcomes of the subjects who pass in arm $z$ are bounded, because the subjects who fail contribute to the support of $Y$. Every subject who would pass under all conditions passes in arm $z$. The target mean therefore lies between the endpoints of the support of $Y$ given $Z=z$ and $S=1$. When that support is bounded, the target is confined to a finite interval. The printed interval from $-\\infty$ to $\\infty$ still contains the target, but it is not sharp, and sharpness is what the proposition claims. The error is confined to the branch in which the sum in the display, which I call the trimming ratio of arm $z$, is at least 1.", "what_survives": "The branch with a trimming ratio below 1 and the estimand are unaffected, and the bounds are sharp in the corrected form. The printed interval from $-\\infty$ to $\\infty$ remains a valid bound in this branch. The statement that the bounds may have infinite width holds when the outcomes of the passers are unbounded in both directions. No number in the paper comes from this branch. The ratio is below 1 in all 12 simulation designs of Table 1, where its largest value is $2/3$, and in all three arms of the application, weighted and unweighted. The application has binary outcomes."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E002", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines. After the end of Proposition 1, the text reads: Assuming that Pr[S(1) = S(2) = ... = S(K) = 1] > 0 ensures that E[Y(z) | S(1) = S(2) = ... = S(K) = 1] exists and continuity of Y ensures that the quantile function is well defined. As with Lee (2009)'s bounds, even when Y is discrete, then bounds can be constructed simply by trimming. A red outline surrounds the words from Assuming that to exists, across a line break.", "caption": "The sentence after Proposition 1, p. 575 (PDF p. 4), below the last two lines of the proposition. The outline marks the clause that derives existence of the mean from positivity.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E003.png"}], "formal_release": null, "formal_statements": [], "headline": "A positive probability of passing under all conditions makes the conditioning event non-null; a finite conditional mean also needs integrability, for which the finite third moments mentioned in Section 2.1 suffice.", "id": "P010-E003", "location": "Sentence after Proposition 1, p. 575 (PDF p. 4)", "pdf_page": 4, "printed_page": "575", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Assuming that $\\Pr[S(1)=S(2)=\\cdots=S(K)=1]>0$ \\hl{ensures that} $\\mathrm{E}[Y(z)\\mid S(1)=S(2)=\\cdots=S(K)=1]$ \\hl{exists} and continuity of $Y$ ensures that the quantile function is well defined.", "corrected": "Assuming that $\\Pr[A]>0$, where $A=\\{S(1)=S(2)=\\cdots=S(K)=1\\}$, \\hl{makes the conditioning event non-null; under the integrability condition $\\mathrm{E}|Y(z)|<\\infty$, the conditional mean $\\mathrm{E}[Y(z)\\mid A]$ exists and is finite. The finite third moments mentioned in Section 2.1 are sufficient for this condition.} Continuity of $Y$ ensures that the quantile function is well defined.", "intended_meaning": "I write $A=\\{S(1)=S(2)=\\cdots=S(K)=1\\}$. The sentence means that $A$ has positive probability, and conditioning on $A$ is then legitimate. The mean $\\mathrm{E}[Y(z)\\mid A]$ exists and is finite when $\\mathrm{E}|Y(z)|<\\infty$, since $\\mathrm{E}[|Y(z)|\\mid A]\\le\\mathrm{E}|Y(z)|/\\Pr[A]$. Section 2.1 mentions suitable regularity conditions, finite third moments among them, and finite third moments imply that $Y(z)$ is integrable. Read with those conditions, the statement is correct, and the application has binary outcomes. Positivity alone does not give existence. If $Z$ is randomized, $S(1)=S(2)=1$ and $Y(1)=Y(2)$ is a standard Cauchy variable, then $\\Pr[A]=1$, the outcome $Y$ is continuous with unbounded support, and its mean does not exist."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E003", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Four lines from the proof of Proposition 1 on page 580, beginning Thus if. In the second line, red boxes mark the less-than-or-equal sign after Y(1) and the characters 1 minus at the start of the argument of the quantile function Q.", "caption": "Appendix A, p. 580: the first upper bound in the proof of Proposition 1. The relation $\\le$ and the quantile argument $1-\\cdots$ of the lower-tail event are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T13", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T13"}, {"id": "P010-T14", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T14"}], "headline": "The first upper bound in the proof is written with the lower tail $Y(1)\\le Q(1-r)$; it should have the upper tail $Y(1)\\ge Q(r)$, with the expectation conditioned on $S(1)=1$.", "id": "P010-E004", "location": "Appendix A, proof of Proposition 1, p. 580 (PDF p. 9), sentence after the display for $p_{U^0}$", "pdf_page": 9, "printed_page": "580", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T13", "P010-T14"], "text": {"as_printed": "Thus if $\\frac{\\sum_{k=2}^{K}\\Pr[S(k)=0]}{\\Pr[S(1)=1]}<1$, a sharp upper bound is given by $\\mathrm{E}[Y(1)\\mid U=1,S(1)=1]\\le\\mathrm{E}[Y(1)\\mid Y(1)\\mathrel{\\hl{\\le}}Q_{Y(1)\\mid S(1)=1}(\\hl{1-}\\sum_{k=2}^{K}\\frac{\\Pr[S(k)=0]}{\\Pr[S(1)=1]})]$, else the upper bound is infinite.\n\n[Earlier in the same proof:] $\\mathrm{E}[Y(1)\\mid U=1,S(1)=1]$ is maximized when $U=1$ for all $Y(1)\\ge\\hl{Q_{Y(1)}}(p_{U^0})$.", "corrected": "Let $U=1$ if $S(2)=\\cdots=S(K)=1$ and $U=0$ otherwise, let $Y(1)$ be continuous, let $Q_{Y(1)\\mid S(1)=1}$ denote the quantile function of $Y(1)$ given $S(1)=1$, and set $r=\\sum_{k=2}^{K}\\Pr[S(k)=0]/\\Pr[S(1)=1]$. If $r<1$, a sharp upper bound is given by\n\\[\n\\mathrm{E}[Y(1)\\mid U=1,S(1)=1]\\le\\mathrm{E}\\big[Y(1)\\mid Y(1)\\mathrel{\\hl{\\ge}}Q_{Y(1)\\mid S(1)=1}(\\hl{r}),\\ \\hl{S(1)=1}\\big].\n\\]\nIn the earlier sentence, $\\mathrm{E}[Y(1)\\mid U=1,S(1)=1]$ is maximized when $U=1$ for all $Y(1)\\ge\\hl{Q_{Y(1)\\mid S(1)=1}}(p_{U^0})$. The case $r\\ge1$ is left as printed here; \\fid{P010-E002} concerns it.", "evidence": "The evidence is an exact counterexample to the printed inequality and a derivation of the corrected one. For the counterexample, let $K=2$ and $S(1)=S(2)=B$, where $B$ is Bernoulli with $\\Pr[B=1]=0.8$, and let $Y(1)$ be standard normal and independent of $B$, with randomized assignment. Then $U=B$, the left side is $\\mathrm{E}[Y(1)\\mid B=1]=0$, and $r=0.2/0.8=0.25$. The right side is the mean of a standard normal variable below its $0.75$ quantile, which is $-\\varphi(\\Phi^{-1}(0.75))/0.75\\approx-0.4237$. The display then asserts $0\\le-0.4237$. The corrected right side is $+0.4237$.\n\nFor the derivation, I write $A=\\{S(1)=1\\}$. The event $\\{U=0\\}\\cap A$ is contained in $\\bigcup_{k\\ge2}\\{S(k)=0\\}$. By the union bound, I have $\\Pr[U=0\\mid A]\\le\\min(1,r)$. If $r<1$, the subjects with $U=1$ then make up a share of at least $1-r$ of $A$. Since $Y(1)$ is continuous given $A$, the mean of $Y(1)$ over a subset of $A$ of conditional probability at least $1-r$ is largest on the upper tail $\\{Y(1)\\ge Q_{Y(1)\\mid A}(r)\\}$. This yields the corrected bound, and the lower tail $\\{Y(1)\\le Q_{Y(1)\\mid A}(1-r)\\}$ gives the lower bound.", "problem": "The proof bounds $\\mathrm{E}[Y(1)\\mid U=1,S(1)=1]$ from above, where $U=1$ for the subjects who would also pass under treatments $2,\\ldots,K$. The sentence quoted second says that this mean is largest when those subjects occupy the upper tail of $Y(1)$ among the subjects with $S(1)=1$. The displayed bound conditions on the lower tail instead: with $r=\\sum_{k=2}^{K}\\Pr[S(k)=0]/\\Pr[S(1)=1]$, it uses the event $Y(1)\\le Q_{Y(1)\\mid S(1)=1}(1-r)$, which belongs to the lower bound. The inequality is false as printed. Further, the expectation on the right is not conditioned on $S(1)=1$, and the quantile in the earlier sentence lacks the subscript $Y(1)\\mid S(1)=1$. The next display of the proof and the statement of Proposition 1 have the upper tail.", "what_survives": "The union bound, the upper-tail trimming argument and the upper bound stated in Proposition 1 are unaffected; the error is a misprint in one display of the proof. The display that follows has a separate slip in its denominator (\\fid{P010-E005}), and the display of the Proposition itself omits the conditioning on $S=1$ (\\fid{P010-E001})."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E004", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the last seven lines of the proof of Proposition 1. The first two lines read: Thus if the sum over k from 2 to K of Pr[S(k) = 0] divided by Pr[S(1) = 1] is less than 1, a sharp upper bound is given by E[Y(1) given U = 1, S(1) = 1] at most E[Y(1) given Y(1) at most Q of 1 minus the sum of Pr[S(k) = 0] over Pr[S(1) = 1]], else the upper bound is infinite. The next lines read: By random assignment and SUTVA, the conditional distribution of Y(1) given S(1) = 1 is equivalent to the conditional distribution of Y given S = 1, Z = 1, and the marginal distributions of S(k) are each equivalent to S given Z = k. Thus a sharp upper bound is given by E[Y(1) given U = 1, S(1) = 1] at most E[Y given Y at least Q of the sum over k from 2 to K of Pr[S = 0 given Z = k] over Pr[S = 1 given Z = k], Z = 1] when the sum over k from 2 to K of Pr[S = 0 given Z = k] over Pr[S = 1 given Z = k] is less than 1, else the upper bound is infinite. The bounds are invariant to indexing of treatments Z, thus yielding the general upper bound in Proposition 1. Analogous calculations yield lower bounds. Red outlines surround the two denominators Pr[S = 1 given Z = k].", "caption": "Appendix A, end of the proof of Proposition 1, p. 580 (PDF p. 9). The outlines mark the two denominators $\\Pr[S=1\\mid Z=k]$; the display two lines above has the denominator $\\Pr[S(1)=1]$.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T15", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T15"}], "headline": "In the last display of the proof of Proposition 1, both ratios should have the denominator $\\Pr[S=1\\mid Z=1]$, the passing probability of the arm whose mean is bounded, and not $\\Pr[S=1\\mid Z=k]$.", "id": "P010-E005", "location": "Appendix A, proof of Proposition 1, last paragraph, p. 580 (PDF p. 9): the argument of the quantile function and the condition after “when” in the final upper bound", "pdf_page": 9, "printed_page": "580", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T15"], "text": {"as_printed": "Thus a sharp upper bound is given by\n\\[\n\\mathrm{E}[Y(1)\\mid U=1,S(1)=1]\\le\\mathrm{E}\\Bigl[Y\\Bigm| Y\\ge Q_{Y\\mid Z=1,S=1}\\Bigl(\\sum_{k=2}^{K}\\frac{\\Pr[S=0\\mid Z=k]}{\\Pr[S=1\\mid Z=\\hl{k}]}\\Bigr),Z=1\\Bigr]\n\\]\nwhen $\\sum_{k=2}^{K}\\frac{\\Pr[S=0\\mid Z=k]}{\\Pr[S=1\\mid Z=\\hl{k}]}<1$, else the upper bound is infinite.", "corrected": "By random assignment and SUTVA, the conditional distribution of $Y(1)\\mid S(1)=1$ is equivalent to the conditional distribution of $Y\\mid S=1,Z=1$, and the marginal distributions of $S(k)$ are each equivalent to $S\\mid Z=k$. Let\n\\[\nr_1=\\sum_{k=2}^{K}\\frac{\\Pr[S=0\\mid Z=k]}{\\Pr[S=1\\mid Z=\\hl{1}]}.\n\\]\nSuppose that Assumptions~1 and~2 hold, that $\\Pr[S(1)=S(2)=\\cdots=S(K)=1]>0$ and that $Y$ is continuous with unbounded support. Then a sharp upper bound is given by\n\\[\n\\mathrm{E}[Y(1)\\mid U=1,S(1)=1]\\le\\mathrm{E}\\bigl[Y\\bigm| Y\\ge Q_{Y\\mid Z=1,S=1}(r_1),Z=1,\\hl{S=1}\\bigr]\n\\]\nwhen $r_1<1$.", "evidence": "The evidence is a derivation and two examples with $K=2$. Let $p_k=\\Pr[S=1\\mid Z=k]$, $A=\\{S(1)=1\\}$ and $d=\\sum_{k=2}^{K}\\Pr[S(k)=0]$. Since $\\Pr[A\\cap\\{U=0\\}]\\le\\min\\{p_1,d\\}$, I have $p_{U^0}\\le\\min\\{1,d/p_1\\}$. Suppose that $r_1=d/p_1<1$. Then at least the fraction $1-r_1$ of the subjects in $A$ have $U=1$. For a continuous outcome, the largest mean over a subset of $A$ of relative mass at least $1-r_1$ is the mean of $Y(1)$ above its $r_1$ quantile in $A$. Random assignment and SUTVA imply that $d=\\sum_{k=2}^{K}(1-p_k)$, and hence $r_1=\\sum_{k=2}^{K}(1-p_k)/p_1$.\n\nIn the first example, $p_1=3/5$ and $p_2=4/5$. The correct ratio is $(1/5)/(3/5)=1/3$, and the printed ratio is $(1/5)/(4/5)=1/4$. Let $Y(1)$ be standard normal among the subjects with $S(1)=1$, let $S(2)=1$ for exactly the upper two thirds of these outcomes, and let $S(2)=1$ for every subject with $S(1)=0$. Then $\\Pr[S(2)=1]=2/5+2/5=4/5$, and the mean to be bounded is $\\mathrm{E}[Y(1)\\mid Y(1)\\ge\\Phi^{-1}(1/3)]=0.5454$. The printed formula, with $S=1$ added to the conditioning event, gives $\\mathrm{E}[Y\\mid Y\\ge\\Phi^{-1}(1/4)]=0.4237$, which is below this mean. In the second example, $p_1=9/10$ and $p_2=1/2$. The correct ratio is $5/9$, and the bound is finite. The printed ratio is $1$, and the printed display declares the bound infinite.", "problem": "The display restates the preceding one in terms of observed quantities. The ratio in the preceding display is $\\sum_{k=2}^{K}\\Pr[S(k)=0]/\\Pr[S(1)=1]$. By random assignment and SUTVA, I have $\\Pr[S(k)=0]=\\Pr[S=0\\mid Z=k]$ and $\\Pr[S(1)=1]=\\Pr[S=1\\mid Z=1]$. The denominator is therefore $\\Pr[S=1\\mid Z=1]$ for every $k$, in agreement with the statement of Proposition~1, where it is $\\Pr[S=1\\mid Z=z]$. As printed, the ratio is the sum of the failure odds of the other arms. This changes the fraction that is trimmed and the condition under which the bound is finite, and the printed expression need not be an upper bound. The conditional mean on the right also lacks the event $S=1$. That omission is the subject of \\fid{P010-E001}.", "what_survives": "The preceding display, the statement of Proposition~1 and the function \\texttt{returnbounds()} of the replication files all divide by the passing probability of the arm that is bounded. The correction changes this display only. The case $r_1\\ge1$ of the bound is the subject of \\fid{P010-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E005", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of a paragraph and two displays from the proof of Proposition 1. The text reads: We again do not observe the joint distribution (U, S(1)). By sigma-additivity, a sharp upper bound is obtained for Pr[U = 0] is obtained when the regions where S(2), S(3), ..., S(K) each equal zero are disjoint, with. The first display bounds Pr[U = 0] above by the sum over k from 2 to K of Pr[S(k) = 0] when that sum is less than 1, and by 1 when the sum is at least 1. The text continues: Thus, among all possible joint distributions (U, S(1)), Pr[U = 0 given S(1) = 1] = p U 0 is maximized when. The second display sets p U 0 equal to the sum of Pr[S(k) = 0] divided by Pr[S(1) = 1] when that ratio is less than 1, and equal to 1 when the ratio is at least 1. Red outlines surround the second occurrence of the words is obtained, the words are disjoint, the words is maximized, and the value 1 in the second case of the second display.", "caption": "Appendix A, proof of Proposition 1, p. 580 (PDF p. 9). The outlines mark the repeated words “is obtained”, the words “are disjoint”, the words “is maximized” and the value 1 in the second case of the display for $p_{U^0}$.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E006.png"}], "formal_release": null, "formal_statements": [], "headline": "The bounds on $\\Pr[U=0]$ and on $p_{U^0}$ are correct as displayed, but the failure events can be disjoint only when their probabilities sum to at most one, and the value $p_{U^0}=1$ is a supremum that is not attained under the assumption of Proposition 1.", "id": "P010-E006", "location": "Appendix A, proof of Proposition 1, third paragraph and its two displays, p. 580 (PDF p. 9)", "pdf_page": 9, "printed_page": "580", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "By $\\sigma$-additivity, a sharp upper bound is obtained for $\\Pr[U=0]$ \\hl{is obtained} when the regions where $S(2),S(3),\\ldots,S(K)$ each equal zero \\hl{are disjoint}, with\n\\[\n\\Pr[U=0]\\le\\begin{cases}\\displaystyle\\sum_{k=2}^{K}\\Pr[S(k)=0] & :\\ \\sum_{k=2}^{K}\\Pr[S(k)=0]<1\\\\[1ex] 1 & :\\ \\displaystyle\\sum_{k=2}^{K}\\Pr[S(k)=0]\\ge 1.\\end{cases}\n\\]\nThus, among all possible joint distributions $(U,S(1))$, $\\Pr[U=0\\mid S(1)=1]=p_{U^0}$ \\hl{is maximized} when\n\\[\np_{U^0}=\\begin{cases}\\dfrac{\\sum_{k=2}^{K}\\Pr[S(k)=0]}{\\Pr[S(1)=1]} & :\\ \\dfrac{\\sum_{k=2}^{K}\\Pr[S(k)=0]}{\\Pr[S(1)=1]}<1\\\\[2ex] {}\\hl{1} & :\\ \\dfrac{\\sum_{k=2}^{K}\\Pr[S(k)=0]}{\\Pr[S(1)=1]}\\ge 1.\\end{cases}\n\\]", "corrected": "\\hl{The union bound gives $\\Pr[U=0]\\le\\min\\bigl\\{1,\\sum_{k=2}^{K}\\Pr[S(k)=0]\\bigr\\}$. Disjoint failure events attain the uncapped bound when their probabilities sum to less than one; at or above one, overlapping events can approach the capped endpoint under positive all-pass mass.} Thus, among all possible joint distributions $(U,S(1))$ \\hl{with the given passing rates and positive all-pass mass}, \\hl{the supremum of} $\\Pr[U=0\\mid S(1)=1]=p_{U^0}$ \\hl{is}\n\\[\n\\hl{\\sup p_{U^0}=\\min\\Bigl\\{1,\\frac{\\sum_{k=2}^{K}\\Pr[S(k)=0]}{\\Pr[S(1)=1]}\\Bigr\\}.}\n\\]\n\\hl{When $\\sum_{k=2}^{K}\\Pr[S(k)=0]\\ge\\Pr[S(1)=1]$, the endpoint one is approached and not attained, since $\\Pr[S(1)=\\cdots=S(K)=1]>0$.}", "intended_meaning": "Let $d=\\sum_{k=2}^{K}\\Pr[S(k)=0]$ and $p_1=\\Pr[S(1)=1]$. The two displays state that $\\Pr[U=0]\\le\\min\\{1,d\\}$ and that the share $p_{U^0}$ of subjects with $U=0$ among those with $S(1)=1$ is at most $\\min\\{1,d/p_1\\}$. Both bounds hold, since the event $\\{S(1)=1,U=0\\}$ has probability at most $\\min\\{p_1,d\\}$, and the trimming fraction of Proposition~1 rests on the second. The wording needs two qualifications. First, the failure events can be disjoint only if $d\\le1$. If $d>1$, they must overlap. For example, two events of probability $7/10$ share probability at least $2/5$. The case distinction in the first display already caps the bound at one. Second, Proposition~1 assumes that $q=\\Pr[S(1)=\\cdots=S(K)=1]>0$, and then $p_{U^0}=1-q/p_1<1$. This implies that the value one in the second display is a supremum and not a maximum. For example, suppose that $K=2$, with $\\Pr[S(1)=1]=3/5$ and $\\Pr[S(2)=0]=3/5$, and that the probabilities $\\varepsilon$, $3/5-\\varepsilon$, $2/5-\\varepsilon$ and $\\varepsilon$ are assigned to $(S(1),S(2))=(1,1)$, $(1,0)$, $(0,1)$ and $(0,0)$. Then $p_{U^0}=1-5\\varepsilon/3$, which equals $5/6$, $59/60$ and $599/600$ for $\\varepsilon=1/10$, $1/100$ and $1/1000$. The value $p_{U^0}=1$ means that no subject with $S(1)=1$ passes under every treatment; the failure events need not cover the whole population. The words ``is obtained'' are printed twice."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E006", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Footnotes 6 and 7 at the foot of page 578. Red boxes mark the clause R sub i equals 1 if an observation i is in the replication study, in the first line of footnote 7, and the fraction p sub i over 1 minus p sub i at its end.", "caption": "Footnote 7, p. 578. The definition of $R_i$ and the weight are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T11", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T11"}, {"id": "P010-T12", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T12"}], "headline": "With $R_i=1$ for the replication study, the weight $p_i/(1-p_i)$ is the reciprocal of the weight that carries the replication sample to the covariate profile of the original study; the indicator should equal one for the original study.", "id": "P010-E007", "location": "Footnote 7, p. 578 (PDF p. 7)", "pdf_page": 7, "printed_page": "578", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T11", "P010-T12"], "text": {"as_printed": "Let $R_i = 1$ if an observation $i$ is in the \\hl{replication study}, else let $R_i = 0$ if an observation was in the \\hl{original data}. We performed a logistic regression of $R_i$ on the following covariates $\\mathbf{X}_i$: Education, Party, Religion, Political Interest, Income, Gender, News Interest, Voter Registration, Birth Year, Region, Race, and Ideology (with mean-imputation for missingness). Using the output of this logistic regression, we computed a predicted value $p_i = \\Pr[R_i = 1 \\mid \\mathbf{X}_i]$ for each observation $i$. To reweight the replication study to the original study's covariate profile, we weighted each observation in the replication sample by $\\frac{p_i}{1-p_i}$.", "corrected": "Let $R_i = 1$ if an observation $i$ is in the \\hl{original study}, else let $R_i = 0$ if an observation was in the \\hl{replication study}. We performed a logistic regression of $R_i$ on the following covariates $\\mathbf{X}_i$: Education, Party, Religion, Political Interest, Income, Gender, News Interest, Voter Registration, Birth Year, Region, Race, and Ideology (with mean-imputation for missingness). Using the output of this logistic regression, we computed a predicted value $p_i = \\Pr[R_i = 1 \\mid \\mathbf{X}_i]$ for each observation $i$. To reweight the replication study to the original study's covariate profile, we weighted each observation in the replication sample by $\\frac{p_i}{1-p_i}$.\n\nEquivalently, define $R_i=1$ for the replication study and $R_i=0$ for the original study, set $p_i=\\Pr[R_i=1\\mid\\mathbf{X}_i]$ with this coding, and weight each observation in the replication sample by $\\hl{\\frac{1-p_i}{p_i}}$.", "evidence": "The evidence is a derivation and an exact counterexample. Let $f_R$ and $f_O$ denote the covariate distributions of the replication and original samples, and let $\\pi=\\Pr[R=1]$ in the pooled data. Suppose that $R=1$ for the replication study, and set $p(x)=\\Pr[R=1\\mid X=x]$. By Bayes' rule, I have\n\\[\n\\frac{p(x)}{1-p(x)}=\\frac{\\pi}{1-\\pi}\\,\\frac{f_R(x)}{f_O(x)} .\n\\]\nThis implies that the weight that turns $f_R$ into $f_O$ is proportional to $(1-p(x))/p(x)$; the constant $\\pi/(1-\\pi)$ cancels when the weights are normalized.\n\nFor the counterexample, suppose that the pooled data consist of eight subjects with one binary covariate, with $X=(0,0,0,1)$ in the original sample and $X=(0,1,1,1)$ in the replication sample. The logistic regression is saturated, and its fitted values are $p(0)=1/4$ and $p(1)=3/4$. With the printed weights, $1/3$ at $X=0$ and $3$ at $X=1$, the weighted replication sample has $\\Pr(X=1)=27/28$. With the reciprocal weights, I obtain $\\Pr(X=1)=1/4$, which is the value in the original sample.\n\nIn the replication code the indicator is one for the original rows, the fitted probability is that of original membership, and each replication row receives the odds of that probability.", "problem": "The footnote sets $R_i=1$ for the replication study and then weights the replication observations by $p_i/(1-p_i)$, where $p_i=\\Pr[R_i=1\\mid\\mathbf{X}_i]$. With that coding, $p_i/(1-p_i)$ is proportional to the ratio of the replication covariate density to the original one. The stated target, the covariate profile of the original study, requires the reciprocal. The replication code defines the indicator the other way, as one for the original PSV rows and zero for the replication rows, and it weights the replication rows by the odds of original membership. The computed weights therefore have the stated direction, and only the coding in the footnote is reversed.", "what_survives": "No number changes. The correction is to the wording of the footnote, and the replication code needs no change. The weights enter Panels B--E of Figure 1, the weighted Tables 2--4 and the weighted analyses of Appendix B, and an implementation of the footnote as printed would produce different weights for these. The results of Section 2, the proofs, the simulations and the unweighted analyses do not use the weights. The weighted estimates themselves could not be recomputed, because the PSV data are withheld from the replication files."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E007", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of the summary of results. Item (iv) ends: is not sufficient to justify dropping subjects, because the types of subjects that comprise those groups may differ between treatments. Item (v) reads: If a check precedes treatment (e.g., a pretreatment attention check), or if its result is otherwise unrelated to treatment assignment, dropping subjects who fail such a check does not lead to bias in estimation of outcomes for those who would pass the manipulation check under all treatment conditions. A red outline surrounds the words if its result is otherwise unrelated to treatment assignment.", "caption": "Section 2.4, item (v), p. 576 (PDF p. 5). The outline marks the clause that states the condition loosely; item (iv), above it, says that equal proportions of failures are not sufficient.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E008.png"}, {"alt": "Crop of five lines. The sentence reads: There is a condition under which dropping subjects who fail a manipulation check recovers this quantity, namely that in which the treatment assigned to a subject is statistically unrelated to whether or not that subject passes or fails the manipulation. Below it, Corollary 1 begins: If Pr[S(1) = S(2) = ... = S(K)] = 1. A red outline surrounds the words statistically unrelated to.", "caption": "Section 2.1, p. 574 (PDF p. 3): the sentence that introduces Corollary 1, with the corollary below it. The outline marks “statistically unrelated to”; the corollary states the condition as $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "“Unrelated to treatment assignment” means that no subject’s result on the check is changed by the assignment, $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$; equal passing rates across arms are not enough.", "id": "P010-E008", "location": "Section 2.4, item (v), p. 576 (PDF p. 5); Section 2.1, sentence before Corollary 1, p. 574 (PDF p. 3)", "pdf_page": 5, "printed_page": "576", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 2.4, p.~576: (v) If a check precedes treatment (e.g., a pretreatment attention check), or \\hl{if its result is otherwise unrelated to treatment assignment}, dropping subjects who fail such a check does not lead to bias in estimation of outcomes for those who would pass the manipulation check under all treatment conditions.\n\nSection 2.1, p.~574: There is a condition under which dropping subjects who fail a manipulation check recovers this quantity, namely that in which the treatment assigned to a subject is \\hl{statistically unrelated to} whether or not that subject passes or fails the manipulation.", "corrected": "Section 2.4: (v) If a check precedes treatment (e.g., a pretreatment attention check), or \\hl{if each subject's passing status is otherwise unaffected by treatment assignment}, dropping subjects who fail such a check does not lead to bias in estimation of outcomes for those who would pass the manipulation check under all treatment conditions.\n\nSection 2.1: There is a condition under which dropping subjects who fail a manipulation check recovers this quantity, namely \\hl{that each subject's passing status is unchanged across treatment assignments}: $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$.", "intended_meaning": "Both sentences refer to the condition of Corollary 1, $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$: each subject's result on the check is the same under every assignment. The paragraph after Corollary 1 says that ``the treatment cannot affect whether or not a subject passes the manipulation check'', and with this condition both sentences are correct. Read literally, ``statistically unrelated'' is independence of $S$ and $Z$, which is a weaker condition. Corollary 3 states that independence is not sufficient, and item (iv) of the same summary says so in words. In the example that establishes Corollary 3, the passing rate is $2/3$ in both arms, the effect is zero for every subject, and the difference in means among the passers is $1/2$ (for its sign, see \\fid{P010-E010})."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E008", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The heading Appendix A. Proofs and the first two lines of the proof of Corollary 1 at the foot of page 579. A red box marks the words thus ensuring, followed by the statement that the vector Y(1), Y(2), up to Y(K), and S is independent of Z.", "caption": "Appendix A, foot of p. 579: the opening of the proof of Corollary 1, with the inference marked. The display that it introduces is at the top of p. 580.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T7", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T7"}, {"id": "P010-T8", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T8"}], "headline": "The proof infers that the whole vector $(Y(1),\\ldots,Y(K),S)$ is independent of $Z$, which Assumption 2 does not imply; the corollary follows from the independence of $(Y(a),S)$ and $Z$ for each $a$ separately.", "id": "P010-E009", "location": "Appendix A, proof of Corollary 1, pp. 579–580 (PDF pp. 8–9)", "pdf_page": 8, "printed_page": "579–580", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T7", "P010-T8"], "text": {"as_printed": "\\emph{Proof of Corollary 1.} $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$ implies $S=S(1)=S(2)=\\cdots=S(K)$, \\hl{thus ensuring} $\\hl{(Y(1),Y(2),\\ldots,Y(K),S)\\perp\\!\\!\\!\\perp Z}$. Joint independence implies that\n\\[\n\\begin{aligned}\n\\mathrm{E}[Y(z)\\mid S=1,Z=z]-\\mathrm{E}[Y(z')\\mid S=1,Z=z']&=\\mathrm{E}[Y(z)\\mid S(1)=1,S(2)=1,\\ldots,S(K)=1]\\\\\n&\\quad-\\mathrm{E}[Y(z')\\mid S=1].\n\\end{aligned}\n\\]", "corrected": "\\emph{Proof of Corollary 1.} Suppose that Assumptions 1--3 hold and that $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$. By Assumption 1, $S=S(1)=S(2)=\\cdots=S(K)$ almost surely. \\hl{Thus Assumption 2 gives} $\\hl{(Y(a),S)\\perp\\!\\!\\!\\perp Z}$ \\hl{for each} $a\\in\\{1,\\ldots,K\\}$. For each such $a$,\n\\[\n\\begin{aligned}\n\\mathrm{E}[Y\\mid S=1,Z=a]&=\\mathrm{E}[Y(a)\\mid S=1,Z=a]=\\mathrm{E}[Y(a)\\mid S=1]\\\\\n&=\\mathrm{E}[Y(a)\\mid S(1)=\\cdots=S(K)=1].\n\\end{aligned}\n\\]\nSubtracting the identities for $a=z$ and $a=z'$ and applying linearity gives the corollary.", "evidence": "The evidence is an exact counterexample to the inference and a derivation of the corollary. For the counterexample, let $K=2$ and suppose that the following four states have probability $1/4$ each:\n\\[\n(Z,Y(1),Y(2))\\in\\{(1,0,0),\\ (1,1,1),\\ (2,0,1),\\ (2,1,0)\\},\\qquad S(1)=S(2)=1 .\n\\]\nFor $a=1$ and for $a=2$, the pair $(Y(a),S(a))$ takes the values $(0,1)$ and $(1,1)$ with probability $1/2$ each, given either value of $Z$. Then Assumptions 1--3 and the premise hold. However, $\\Pr[Y(1)=Y(2)\\mid Z=1]=1$ and $\\Pr[Y(1)=Y(2)\\mid Z=2]=0$. This implies that the vector $(Y(1),Y(2),S)$ is not independent of $Z$. Note that both sides of Corollary 1 equal zero in this example; it contradicts the proof step and not the corollary.\n\nFor the derivation, fix $a\\in\\{1,\\ldots,K\\}$. By Assumption 1 and the premise, I have $S=S(a)$ almost surely. Then Assumption 2 implies that $(Y(a),S)\\perp\\!\\!\\!\\perp Z$. In particular, $S\\perp\\!\\!\\!\\perp Z$, and Assumption 3 then gives $\\Pr[S=1,Z=t]>0$ for every $t$. Then\n\\[\n\\begin{aligned}\n\\mathrm{E}[Y\\mid S=1,Z=a]&=\\mathrm{E}[Y(a)\\mid S=1,Z=a]=\\mathrm{E}[Y(a)\\mid S=1]\\\\\n&=\\mathrm{E}[Y(a)\\mid S(1)=\\cdots=S(K)=1].\n\\end{aligned}\n\\]\nIn the first equality, I used Assumption 1. In the second, I used the independence of $(Y(a),S)$ and $Z$, and the third holds because the two conditioning events agree almost surely. Subtracting this identity for $a=z'$ from the one for $a=z$, I obtain the corollary.", "problem": "Assumption 2 states, for each $z$ separately, that $(Y(z),S(z))$ is independent of $Z$. It does not constrain the joint distribution of potential outcomes under different treatments. Under the premise $\\Pr[S(1)=\\cdots=S(K)]=1$, each $S(z)$ may be replaced by $S$, and I obtain $(Y(a),S)\\perp\\!\\!\\!\\perp Z$ for each $a$. The premise does not imply that the vector $(Y(1),\\ldots,Y(K),S)$ is independent of $Z$. The corollary does not need the stronger statement, and its conclusion is correct. Under random assignment of the treatment the stronger independence also holds. The gap lies between the proof and Assumption 2 as stated. The last term of the display, $\\mathrm{E}[Y(z')\\mid S=1]$, is correct as printed, because under the premise the events $\\{S=1\\}$ and $\\{S(1)=\\cdots=S(K)=1\\}$ agree almost surely.", "what_survives": "Corollary 1 holds under Assumptions 1--3 as stated, and the repair adds no assumption. Corollary 2, the bounds, the simulations and the application do not use the independence of the whole vector."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E009", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the proof of Corollary 3. It reads: Proof of Corollary 3. We prove the claim via a simple counterexample. Suppose Supp(Z) = {1, 2} and f(Y(1), Y(2), S(1), S(2)) equals 1/3 for Y(1) = 1, Y(2) = 1, S(1) = 0, S(2) = 1; 1/3 for Y(1) = 0, Y(2) = 0, S(1) = 1, S(2) = 0; 1/3 for Y(1) = 0, Y(2) = 0, S(1) = 1, S(2) = 1; and 0 otherwise. Note that E[S(2) minus S(1)] = 0 and E[tau given S(1) = S(2) = 1] = E[tau given S(1) = 1, S(2) = 0] = E[tau given S(1) = 0, S(2) = 1] = 0. E[Y given S = 1, Z = 2] minus E[Y given S = 1, Z = 1] = 0 minus 1/2 = minus 1/2. A red outline surrounds the final arithmetic, 0 minus 1/2 = minus 1/2.", "caption": "Appendix A, proof of Corollary 3, p. 580 (PDF p. 9). The outline marks the arithmetic that closes the proof; the distribution above it gives the two means.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T16", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T16"}, {"id": "P010-T21", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T21"}], "headline": "In the counterexample that proves Corollary 3, the contrast among the subjects who pass is $1/2-0=1/2$, not $0-1/2=-1/2$.", "id": "P010-E010", "location": "Appendix A, proof of Corollary 3, last line, p. 580 (PDF p. 9)", "pdf_page": 9, "printed_page": "580", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T16", "P010-T21"], "text": {"as_printed": "Note that $\\mathrm{E}[S(2)-S(1)]=0$ and $\\mathrm{E}[\\tau\\mid S(1)=S(2)=1]=\\mathrm{E}[\\tau\\mid S(1)=1,S(2)=0]=\\mathrm{E}[\\tau\\mid S(1)=0,S(2)=1]=0$. $\\mathrm{E}[Y\\mid S=1,Z=2]-\\mathrm{E}[Y\\mid S=1,Z=1]=\\hl{0-1/2=-1/2}$.", "corrected": "Note that $\\mathrm{E}[S(2)-S(1)]=0$ and, \\hl{with $\\tau=Y(2)-Y(1)$}, $\\mathrm{E}[\\tau\\mid S(1)=S(2)=1]=\\mathrm{E}[\\tau\\mid S(1)=1,S(2)=0]=\\mathrm{E}[\\tau\\mid S(1)=0,S(2)=1]=0$. $\\mathrm{E}[Y\\mid S=1,Z=2]-\\mathrm{E}[Y\\mid S=1,Z=1]=\\hl{1/2-0=1/2}$.", "evidence": "The evidence is a recalculation from the printed distribution, which puts probability $1/3$ on each of three types of subject.\n\\begin{audittable}{lcccccc}{The three types of the counterexample, each of probability $1/3$.}\nType & $Y(1)$ & $Y(2)$ & $S(1)$ & $S(2)$ & Passes under $Z=1$ & Passes under $Z=2$ \\\\\n1 & 1 & 1 & 0 & 1 & no & yes \\\\\n2 & 0 & 0 & 1 & 0 & yes & no \\\\\n3 & 0 & 0 & 1 & 1 & yes & yes \\\\\n\\end{audittable}\nUnder $Z=2$ the subjects who pass are of types 1 and 3, with outcomes $1$ and $0$, and I have $\\mathrm{E}[Y\\mid S=1,Z=2]=1/2$. Under $Z=1$ they are of types 2 and 3, both with outcome $0$, and $\\mathrm{E}[Y\\mid S=1,Z=1]=0$. Note that the passing rate is $2/3$ in each arm, and that $Y(2)-Y(1)=0$ for every type.", "problem": "The two conditional means are interchanged. Under the distribution displayed in the proof, the subjects who pass under treatment~2 have mean outcome $1/2$, and the subjects who pass under treatment~1 have mean outcome $0$. The contrast of treatment~2 with treatment~1 is $+1/2$. The symbol $\\tau$ is also used without a definition. It stands for the effect $Y(2)-Y(1)$ of a subject.", "what_survives": "Corollary~3 and its proof stand. The example has the same passing rate in both arms, a zero effect for every subject and a contrast among the subjects who pass that is not zero. The proof uses nothing else. Only the order of the two means and the sign change."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E010", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last paragraph of Appendix C. It reads: The width of the bounds also depends on multiple factors. As the variability of potential outcomes increases (characterized by sigma, and to a lesser extent lambda), the width of the bounds increases, as evidenced by comparing, e.g., row 1 vs. 2 vs. 3. The width of the bounds also depends on failure rates; again compare rows 1 to 4 to 5 to 8 to 9 to 12. The bounds do not depend on any unobservable features of the joint distributions of potential outcomes and responses to the manipulation check. To wit, the width of the bounds does not change as rho(S(1), S(2)) is varied; compare, e.g., row 1 to row 4. Red outlines surround the last sentence, which begins To wit, on its two lines.", "caption": "Appendix C, last paragraph, p. 581 (PDF p. 10). The outline marks the sentence that claims the width does not change with the correlation.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E011.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T9", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T9"}, {"id": "P010-T10", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T10"}], "headline": "The width of the bounds does change with $\\rho(S(1),S(2))$ in the simulation design: rows 1 and 4 of Table 1 have population widths 1.7986 and 1.7680, because the correlation changes the distribution of outcomes among the subjects who pass.", "id": "P010-E011", "location": "Appendix C, last paragraph, last two sentences, p. 581 (PDF p. 10); Table 1, rows 1 and 4, p. 582 (PDF p. 11)", "pdf_page": 10, "printed_page": "581", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T10", "P010-T9"], "text": {"as_printed": "The bounds do not depend on any unobservable features of the joint distributions of potential outcomes and responses to the manipulation check. \\hl{To wit, the width of the bounds does not change as $\\rho(S(1),S(2))$ is varied; compare, e.g., row 1 to row 4.}", "corrected": "The bounds depend only on the passing rates and the outcome distributions among passers in each treatment arm. \\hl{Holding these observables fixed, the bounds are invariant to unobserved features of the joint distribution. Changing the correlation in the simulation comparison of rows 1 and 4 also changes the observable outcome distributions among passers, so the widths change slightly.}", "evidence": "The evidence is an exact calculation in the paper's own design. Rows 1 and 4 have $\\lambda=\\sigma=1$ and $\\Pr[S(1)=1]=\\Pr[S(2)=1]=p=4/5$, with $\\rho=1/4$ and $\\rho=1/2$. For two indicators with mean $p$, I have $a=\\Pr[S(1)=S(2)=1]=p^2+\\rho p(1-p)$, and the other three joint probabilities follow from the margins. Among the subjects with $S(1)=1$, the outcome is $\\varepsilon$ if $S(2)=1$ and $\\varepsilon-1$ if $S(2)=0$, where $\\varepsilon$ is standard normal. Its distribution function is\n\\[\nF_w(x)=(1-w)\\Phi(x)+w\\Phi(x+1),\\qquad w=\\Pr[S(2)=0\\mid S(1)=1].\n\\]\nBy random assignment, $F_w$ is the distribution of $Y$ given $S=1$ and $Z=1$. Its mean is $-w$ and its variance is $1+w(1-w)$. The distribution among the subjects who pass under treatment~2 is the reflection of $F_w$ about zero. In both rows the bounds retain the fraction $(2p-1)/p=3/4$ of each distribution. Let $q_t=F_w^{-1}(t)$ and\n\\[\nM_w(x)=\\int_{-\\infty}^{x}y\\,dF_w(y)=-(1-w)\\phi(x)-w\\Phi(x+1)-w\\phi(x+1).\n\\]\nThe bounds for the mean under treatment~1 are\n\\[\nL_w=\\frac{M_w(q_{3/4})}{3/4},\\qquad U_w=\\frac{-w-M_w(q_{1/4})}{3/4},\n\\]\nand those for the mean under treatment~2 are $-U_w$ and $-L_w$. The bounds on the difference in means are $[-2U_w,-2L_w]$, with width $W(w)=2(U_w-L_w)$.\n\\begin{audittable}{lcccccccc}{Rows 1 and 4 of Table~1: the probability $a$ of passing under both treatments, the mixture weight $w=(p-a)/p$, and the population bounds.}\nRow & $\\rho$ & $a$ & $w$ & $q_{1/4}$ & $q_{3/4}$ & $L_w$ & $U_w$ & $W(w)$ \\\\\n1 & $1/4$ & $17/25$ & $3/20$ & $-0.8566$ & $0.5714$ & $-0.5956$ & $0.3036$ & $1.7986$ \\\\\n4 & $1/2$ & $18/25$ & $1/10$ & $-0.7953$ & $0.6074$ & $-0.5388$ & $0.3452$ & $1.7680$ \\\\\n\\end{audittable}\nEnclosing every quantity between rational numbers, I find that\n\\[\n\\begin{aligned}\n1.798577298898&\\le W\\le1.798577299027\\quad\\text{in row 1},\\\\\n1.767983385194&\\le W\\le1.767983385319\\quad\\text{in row 4}.\n\\end{aligned}\n\\]\nThe two widths differ by more than $0.0305$, which is 1.7 percent, and the calculation involves no simulation.", "problem": "The first sentence is correct. The bounds are functions of the passing rates and of the distributions of outcomes among the subjects who pass in each arm, and these are observable. The second sentence does not follow from it. Rows 1 and 4 of Table~1 have the same $\\lambda$, $\\sigma$ and passing rates and differ in the correlation $\\rho(S(1),S(2))$. In the simulation design the outcome of a subject is $\\lambda[S(2)-S(1)]$ plus normal noise. Among the subjects who pass under treatment~1, the distribution of the outcome is a mixture of two normal distributions. The weight of the mixture is $\\Pr[S(2)=0\\mid S(1)=1]$, which depends on the correlation. Changing the correlation changes an observable distribution, and the bounds change with it. Note that Table~1 itself prints different widths in the two rows, 1.753 and 1.723.", "what_survives": "The statement that the bounds depend only on observable quantities stands. No other result, and no summary of the simulations in the main text, uses the claim of invariance. The printed widths of rows 1 and 4 are themselves affected by the error described in \\fid{P010-E016}, and the population widths above do not depend on it."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E011", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The heading 4 Conclusion and four lines of body text from page 579. A red box marks the words Corollary 3 in the second line.", "caption": "Section 4, p. 579: the first sentence of the Conclusion, with the cross-reference marked.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E012.png"}], "formal_release": null, "formal_statements": [], "headline": "The result on pretreatment attention checks is Corollary 1, not Corollary 3.", "id": "P010-E012", "location": "Section 4, first sentence, p. 579 (PDF p. 8)", "pdf_page": 8, "printed_page": "579", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We reiterate that our critiques do not apply to research designs that use pretreatment attention checks to screen subjects, as established by Corollary \\hl{3}.", "corrected": "We reiterate that our critiques do not apply to research designs that use pretreatment attention checks to screen subjects, as established by Corollary \\hl{1}.", "evidence": "A check of the page: the sentence prints ``Corollary 3'', and the two corollaries on pp.~574--575 say what is described above.", "problem": "Corollary 1 (p.~574) and the paragraph after it show that discarding subjects is unproblematic when the treatment cannot affect the check, as with a check given before treatment. Corollary 3 (p.~575) is the negative result that independence of $S$ and $Z$ does not justify dropping subjects.", "what_survives": "The recommendation is correct, and only the cross-reference changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E012", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of Table 5. The caption reads: Table 5. Unweighted covariate distributions among subjects who failed the manipulation check. The columns are Mean and (SE) for Treatment 1, Treatment 2 and Treatment 3. The rows shown are Education, Democrat, Republican, Independent, Other party, Political interest (5-point scale), Income, Age (continuous), Age (1st quartile) to Age (4th quartile), and Female. The row for Age (continuous) reads 40.960, 5.781, 52.658, 3.535, 39.028, 3.592, and the row for Female reads 0.249, 0.114, 0.688, 0.069, 0.637, 0.102. Red outlines surround the word Unweighted in the caption and the whole block of numbers.", "caption": "Table 5, p. 586 (PDF p. 15), caption and first thirteen rows. The outlines mark the word “Unweighted” and the entries, which are those of Table 2; the remaining twelve rows are also identical to Table 2.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E013.png"}, {"alt": "Crop of the top of Table 2. The caption reads: Table 2. Weighted covariate distributions among subjects who failed the manipulation check. The columns and the thirteen rows shown, Education to Female, are the same as in the crop of Table 5, with the same numbers: the row for Age (continuous) reads 40.960, 5.781, 52.658, 3.535, 39.028, 3.592, and the row for Female reads 0.249, 0.114, 0.688, 0.069, 0.637, 0.102. A red outline surrounds the word Weighted in the caption.", "caption": "Table 2, p. 582 (PDF p. 11), caption and first thirteen rows, for comparison. The outline marks the word “Weighted”. The entries are the same as those of Table 5.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E013-b.png"}, {"alt": "Three panels of a dot plot, for Treatment 1 (N = 35), Treatment 2 (N = 182) and Treatment 3 (N = 40), with a shared list of eighteen covariates on the vertical axis: Democrat, Republican, Independent, Other party, the four age quartiles, Female, Registered to vote, White, Black, Other race, Northeast, Midwest, South, West and Elite. The horizontal axis is the share of the subjects who failed the manipulation check, from 0 to 100 per cent. Each covariate has an open orange ring for the value printed in Table 5 and a filled blue dot for the unweighted mean, joined by a grey bar. The two differ for nearly every covariate. For Female the rings are at 25, 69 and 64 per cent and the dots at 43, 52 and 50 per cent. For the first age quartile the rings are at 47, 23 and 61 per cent and the dots at 86, 67 and 85 per cent. Only for the third age quartile in Treatment 1 do the ring and the dot coincide exactly, at zero.", "caption": "The eighteen covariates of Table 5 that are shares, by treatment: the value printed in Table 5 (open rings) and the unweighted mean (filled dots). The unweighted means are recomputed from the public data, and for the four age quartiles they are recovered from Tables 6 and 7.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E013-table5.png"}], "formal_release": null, "formal_statements": [], "headline": "Table 5 is captioned as the unweighted covariate distributions of the subjects who failed the manipulation check, but its entries are the weighted ones of Table 2; 74 of its 75 means differ from the unweighted means.", "id": "P010-E013", "location": "Appendix E, Table 5, all entries, p. 586 (PDF p. 15); compare Table 2, p. 582 (PDF p. 11); replication files, abp_replication.R, line 1261", "pdf_page": 15, "printed_page": "586", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table~5, p.~586.} Table 5. Unweighted covariate distributions among subjects who failed the manipulation check.\n\\begin{audittable}{lrrrrrr}{Table~5 as printed, with separate Mean and (SE) columns. The highlighted means differ from the unweighted means.}\n & Treatment 1 & & Treatment 2 & & Treatment 3 & \\\\\n & Mean & (SE) & Mean & (SE) & Mean & (SE) \\\\\nEducation & \\hl{2.931} & 0.178 & \\hl{3.236} & 0.213 & \\hl{2.801} & 0.168 \\\\\nDemocrat & \\hl{0.336} & 0.135 & \\hl{0.545} & 0.086 & \\hl{0.390} & 0.114 \\\\\nRepublican & \\hl{0.424} & 0.186 & \\hl{0.355} & 0.082 & \\hl{0.237} & 0.088 \\\\\nIndependent & \\hl{0.172} & 0.103 & \\hl{0.072} & 0.027 & \\hl{0.168} & 0.092 \\\\\nOther party & \\hl{0.042} & 0.031 & \\hl{0.015} & 0.008 & \\hl{0.032} & 0.023 \\\\\nPolitical interest (5-point scale) & \\hl{3.542} & 0.195 & \\hl{3.782} & 0.114 & \\hl{3.090} & 0.277 \\\\\nIncome & \\hl{7.558} & 0.692 & \\hl{6.253} & 0.491 & \\hl{7.192} & 0.616 \\\\\nAge (continuous) & \\hl{40.960} & 5.781 & \\hl{52.658} & 3.535 & \\hl{39.028} & 3.592 \\\\\nAge (1st quartile) & \\hl{0.470} & 0.170 & \\hl{0.231} & 0.049 & \\hl{0.611} & 0.126 \\\\\nAge (2nd quartile) & \\hl{0.242} & 0.127 & \\hl{0.171} & 0.044 & \\hl{0.107} & 0.080 \\\\\nAge (3rd quartile) & 0.000 & 0.000 & \\hl{0.214} & 0.063 & \\hl{0.180} & 0.107 \\\\\nAge (4th quartile) & \\hl{0.287} & 0.203 & \\hl{0.384} & 0.104 & \\hl{0.102} & 0.090 \\\\\nFemale & \\hl{0.249} & 0.114 & \\hl{0.688} & 0.069 & \\hl{0.637} & 0.102 \\\\\nIdeology (5-point scale) & \\hl{3.116} & 0.318 & \\hl{3.019} & 0.145 & \\hl{2.874} & 0.202 \\\\\nNews interest (4-point scale) & \\hl{1.959} & 0.290 & \\hl{1.636} & 0.113 & \\hl{2.458} & 0.265 \\\\\nRegistered to vote & \\hl{0.746} & 0.119 & \\hl{0.919} & 0.041 & \\hl{0.832} & 0.092 \\\\\nWhite & \\hl{0.630} & 0.141 & \\hl{0.867} & 0.035 & \\hl{0.733} & 0.095 \\\\\nBlack & \\hl{0.292} & 0.125 & \\hl{0.053} & 0.020 & \\hl{0.036} & 0.027 \\\\\nOther race & \\hl{0.078} & 0.048 & \\hl{0.080} & 0.025 & \\hl{0.231} & 0.093 \\\\\nReligious importance & \\hl{4.680} & 0.457 & \\hl{4.623} & 0.257 & \\hl{3.774} & 0.391 \\\\\nNortheast & \\hl{0.034} & 0.023 & \\hl{0.121} & 0.037 & \\hl{0.217} & 0.104 \\\\\nMidwest & \\hl{0.056} & 0.035 & \\hl{0.404} & 0.099 & \\hl{0.183} & 0.093 \\\\\nSouth & \\hl{0.406} & 0.149 & \\hl{0.326} & 0.074 & \\hl{0.511} & 0.111 \\\\\nWest & \\hl{0.504} & 0.166 & \\hl{0.149} & 0.041 & \\hl{0.089} & 0.042 \\\\\nElite & \\hl{0.089} & 0.073 & \\hl{0.076} & 0.046 & \\hl{0.059} & 0.037 \\\\\nObservations & $N=35$ & & $N=182$ & & $N=40$ & \\\\\n\\end{audittable}\n\\emph{Replication files, \\texttt{abp\\_replication.R}, line 1261 (not printed in the article).}\n\n$\\texttt{failed.tab <- format(}\\hl{\\texttt{w.failed.tab}}\\texttt{, digits = 3, nsmall = 3)}$", "corrected": "\\emph{Replication files, line 1261.}\n\n$\\texttt{failed.tab <- format(}\\hl{\\texttt{failed.tab}}\\texttt{, digits = 3, nsmall = 3)}$\n\n\\emph{Table~5.} Unweighted covariate distributions among subjects who failed the manipulation check.\n\\begin{audittable}{lrrr}{Table~5 with unweighted means. The standard errors in parentheses are the exact standard deviations of the bootstrap distribution of each mean; the estimates from 10,000 bootstrap samples that the replication code produces may differ from them slightly.}\nCovariate & Treatment 1 & Treatment 2 & Treatment 3 \\\\\nEducation & \\hl{3.457} (\\hl{0.199}) & \\hl{4.033} (\\hl{0.095}) & \\hl{3.275} (\\hl{0.190}) \\\\\nDemocrat & \\hl{0.486} (\\hl{0.084}) & \\hl{0.549} (\\hl{0.037}) & \\hl{0.350} (\\hl{0.075}) \\\\\nRepublican & \\hl{0.171} (\\hl{0.064}) & \\hl{0.247} (\\hl{0.032}) & \\hl{0.200} (\\hl{0.063}) \\\\\nIndependent & \\hl{0.200} (\\hl{0.068}) & \\hl{0.137} (\\hl{0.026}) & \\hl{0.175} (\\hl{0.060}) \\\\\nOther party & \\hl{0.086} (\\hl{0.047}) & \\hl{0.038} (\\hl{0.014}) & \\hl{0.075} (\\hl{0.042}) \\\\\nPolitical interest (5-point scale) & \\hl{3.200} (\\hl{0.139}) & \\hl{3.313} (\\hl{0.071}) & \\hl{3.108} (\\hl{0.173}) \\\\\nIncome & \\hl{7.236} (\\hl{0.594}) & \\hl{6.435} (\\hl{0.252}) & \\hl{6.808} (\\hl{0.482}) \\\\\nAge (continuous) & \\hl{28.571} (\\hl{1.577}) & \\hl{35.890} (\\hl{0.927}) & \\hl{30.750} (\\hl{1.703}) \\\\\nAge (1st quartile) & \\hl{0.857} (\\hl{0.059}) & \\hl{0.670} (\\hl{0.035}) & \\hl{0.850} (\\hl{0.056}) \\\\\nAge (2nd quartile) & \\hl{0.114} (\\hl{0.054}) & \\hl{0.159} (\\hl{0.027}) & \\hl{0.050} (\\hl{0.034}) \\\\\nAge (3rd quartile) & 0.000 (0.000) & \\hl{0.099} (\\hl{0.022}) & \\hl{0.075} (\\hl{0.042}) \\\\\nAge (4th quartile) & \\hl{0.029} (\\hl{0.028}) & \\hl{0.071} (\\hl{0.019}) & \\hl{0.025} (\\hl{0.025}) \\\\\nFemale & \\hl{0.429} (\\hl{0.084}) & \\hl{0.522} (\\hl{0.037}) & \\hl{0.500} (\\hl{0.079}) \\\\\nIdeology (5-point scale) & \\hl{2.646} (\\hl{0.161}) & \\hl{2.674} (\\hl{0.076}) & \\hl{2.780} (\\hl{0.167}) \\\\\nNews interest (4-point scale) & \\hl{2.400} (\\hl{0.135}) & \\hl{2.100} (\\hl{0.070}) & \\hl{2.334} (\\hl{0.138}) \\\\\nRegistered to vote & \\hl{0.714} (\\hl{0.076}) & \\hl{0.894} (\\hl{0.023}) & \\hl{0.843} (\\hl{0.056}) \\\\\nWhite & \\hl{0.571} (\\hl{0.084}) & \\hl{0.780} (\\hl{0.031}) & \\hl{0.700} (\\hl{0.072}) \\\\\nBlack & \\hl{0.286} (\\hl{0.076}) & \\hl{0.077} (0.020) & \\hl{0.050} (\\hl{0.034}) \\\\\nOther race & \\hl{0.143} (\\hl{0.059}) & \\hl{0.143} (\\hl{0.026}) & \\hl{0.250} (\\hl{0.068}) \\\\\nReligious importance & \\hl{3.171} (\\hl{0.327}) & \\hl{3.214} (\\hl{0.147}) & \\hl{3.075} (\\hl{0.286}) \\\\\nNortheast & \\hl{0.143} (\\hl{0.059}) & \\hl{0.231} (\\hl{0.031}) & \\hl{0.180} (\\hl{0.060}) \\\\\nMidwest & \\hl{0.171} (\\hl{0.064}) & \\hl{0.198} (\\hl{0.030}) & \\hl{0.181} (\\hl{0.060}) \\\\\nSouth & \\hl{0.400} (\\hl{0.083}) & \\hl{0.390} (\\hl{0.036}) & \\hl{0.484} (\\hl{0.078}) \\\\\nWest & \\hl{0.286} (\\hl{0.076}) & \\hl{0.181} (\\hl{0.029}) & \\hl{0.156} (\\hl{0.056}) \\\\\nElite & \\hl{0.057} (\\hl{0.039}) & \\hl{0.044} (\\hl{0.015}) & \\hl{0.100} (\\hl{0.047}) \\\\\nObservations & $N=35$ & $N=182$ & $N=40$ \\\\\n\\end{audittable}", "evidence": "The evidence is a recalculation from the public data in exact arithmetic, with two checks of the printed tables against one another. Applying to the public data the recoding of the replication script, with its replacement of missing covariates by the mean over the replication sample, I obtain the 63 means of the 21 covariates that need no confidential input. Among the subjects who failed, $15$ of $35$, $95$ of $182$ and $20$ of $40$ are women. The printed shares cannot be unweighted shares. A count $k$ with $k/35$, $k/182$ or $k/40$ equal to 0.249, 0.688 or 0.637 after rounding would have to lie in $[8.6975,8.7325]$, $[125.125,125.307]$ or $[25.46,25.50]$, and none of these intervals contains an integer.\n\nThe twelve means of the age quartiles depend on quartiles of the sample of Press, Sagan and Valentino, which is not public. They can be recovered from Tables~6 and~7. Since every sample there has fewer than 1,000 subjects, each share printed to three decimals fixes one integer count. The count among the subjects who failed is the count among all subjects minus the count among those who passed. The counts are $(30,4,0,1)$, $(122,29,18,13)$ and $(34,2,3,1)$ for the three treatments. Classifying the ages in the public data at the boundaries 39, 50 and 59 reproduces every one of these counts.\n\nFor any covariate, $N_a\\bar x_a=N_p\\bar x_p+N_f\\bar x_f$, where the subscripts denote all subjects, those who passed and those who failed. With the printed means of age in Treatment~2, the right side divided by $N_a$ is $(356\\times34.963+182\\times52.658)/538=40.949$, where Table~7 prints $35.277$. The gap is $5.67$, and rounding allows at most $0.001$. With the recomputed means, all 75 identities hold exactly, and all 150 means of Tables~6 and~7 are reproduced to the printed precision. The one mean of Table~5 that is unchanged is the zero for the third age quartile in Treatment~1.\n\nThe standard errors of the paper are bootstrap estimates from 10,000 samples. The standard deviation of the mean of a bootstrap sample has the closed form $\\bigl(\\sum_i(x_i-\\bar x)^2\\bigr)^{1/2}/n$. For the share of women in Treatment~1 it is $\\bigl((3/7)(4/7)/35\\bigr)^{1/2}=0.084$, where Table~5 prints $0.114$.", "problem": "Appendix~E announces the unweighted covariate profiles of the subjects, by treatment and by performance on the manipulation check, as the counterpart of the weighted profiles of Appendix~D. Table~5 does not contain them. Its 150 numerical entries and its three numbers of observations are those of Table~2, which gives the weighted profiles of the subjects who failed. The cause is one line of the replication code. The script computes the unweighted table \\texttt{failed.tab} in lines 1246--1258, and in line 1261, where it formats the table, it overwrites it with the formatted weighted table \\texttt{w.failed.tab}. The two files \\texttt{tab\\_2.txt} and \\texttt{tab\\_5.txt} saved with the replication files are identical byte for byte.\n\nThe unweighted profiles of the subjects who failed are not printed anywhere in the paper, and they differ from the printed values in 74 of the 75 means. The share of women among these subjects is 42.9, 52.2 and 50.0 percent in the three treatments, where Table~5 prints 24.9, 68.8 and 63.7 percent. Their mean age is 28.6, 35.9 and 30.8 years, where Table~5 prints 41.0, 52.7 and 39.0. Table~5 therefore gives the wrong unweighted composition of the subjects who would be dropped.", "what_survives": "Table~2 and the comparison in Section~3 that cites it, with 63.7 and 24.9 percent of women among the subjects who failed under Treatments~3 and~1, are explicitly weighted and are not affected by this error. I could not recompute the weighted values, since the weights require data that are not public. Tables~6 and~7 are produced from other objects of the code, and their means are reproduced. No figure, estimate, bound or theoretical result of the paper uses Table~5."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E013", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top row of Figure 1, three column charts with a vertical axis from 0 to 120 percent. Panel A, Original, has pairs of columns for 90/90 (N=157), 90/70 (N=127) and 90/45 (N=162). Panel B, Replication (Dropping Subjects; Weighted), has pairs of columns for 90/90 (N=509), 90/70 (N=356) and 90/45 (N=493). Panel C, Replication (Full Sample; Weighted), has pairs for 90/90 (N=544), 90/70 (N=538) and 90/45 (N=533). Each column carries a short vertical interval bar. Red outlines surround the six columns and their interval bars in Panel B and in Panel C.", "caption": "Figure 1, Panels A to C, p. 577 (PDF p. 6). The outlines mark the columns of Panels B and C, which show the weighted estimates of the replication, with their 95% bars.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E014.png"}, {"alt": "Crop of the six-line caption of Figure 1, which describes Panels A to E. A red outline surrounds the last sentence, across a line break: Vertical bars represent 95\\% confidence intervals on point estimates calculated using the bootstrap.", "caption": "Caption of Figure 1, p. 577 (PDF p. 6). The outline marks the sentence that describes the vertical bars.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E014-b.png"}, {"alt": "Listing of R code with line numbers. Lines 433 to 434 carry the comment Produce unweighted, censored approval SE estimates and assign boot.dat column 1. Lines 443 to 444 do the same for unweighted, uncensored estimates and column 7. Lines 453 to 455 carry the comment Produce weighted, censored approval SE estimates and assign column 13 with weighted.mean. Lines 469 to 471 do the same for weighted, uncensored estimates and column 19. Lines 615 to 620 draw Panel B, titled Replication (Dropping Subjects; Weighted), with interval bars at w.cens.means plus or minus 1.96 times boot.ses[1:6]. Lines 629 to 634 draw Panel C, titled Replication (Full Sample; Weighted), with interval bars at w.means plus or minus 1.96 times boot.ses[7:12]. Red outlines surround the four occurrences of boot.ses[1:6] and the four occurrences of boot.ses[7:12].", "caption": "Replication code, not a page of the article: abp_replication.R. Lines 433–471 (excerpts) show that bootstrap columns 1 and 7 hold unweighted means and columns 13 and 19 weighted means; lines 615–620 and 629–634 draw Panels B and C. The outlines mark the indices of the standard errors, which point to the unweighted columns.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E014-code.png"}, {"alt": "Chart with two panels that share a horizontal axis from 0 to 80 percent. The title reads: Figure 1, Panels B and C: the published 95\\% bars are about half as wide as the bars that the weighted standard errors give. The left panel is Panel B, subjects who failed the check dropped; the right panel is Panel C, full sample. Each panel has six rows: 90/90 approve, 90/90 prefer, 90/70 approve, 90/70 prefer, 90/45 approve, 90/45 prefer. In each row a thick orange bar with a black dot shows the estimate and the bar as published, and below it a thin blue bar with end marks shows the approximate replacement. In every row the blue bar is about twice as long as the orange bar. For example, in Panel B the row 90/70 prefer has the published bar from 48.3 to 58.7 and the replacement from 42.9 to 64.1. The legend states that the saved weighted standard error is 1.87 to 2.39 times as large as the one behind the published bar.", "caption": "The twelve weighted estimates of Figure 1, Panels B and C, with the 95% bar as published (thick, with the estimate as a dot) and the approximate bar from the saved weighted standard error (thin, with end marks). The published values are read from the coordinates of the published figure; the replacements are the saved weighted mean plus or minus 1.96 saved weighted standard errors, and they are approximate for the reasons given in the text.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E014-intervals.png"}], "formal_release": null, "formal_statements": [], "headline": "The 95% bars on the weighted estimates in Figure 1, Panels B and C, are built from the bootstrap standard errors of the unweighted means; the saved weighted standard errors are about twice as large, and the bars should be about twice as wide.", "id": "P010-E014", "location": "Figure 1, Panels B and C, and its caption, p. 577 (PDF p. 6); replication code abp_replication.R, lines 617–621 and 631–635", "pdf_page": 6, "printed_page": "577", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Caption of Figure 1, last sentence: ``Vertical bars represent \\hl{95\\% confidence intervals on point estimates} calculated using the bootstrap.''\n\nReplication code that draws the bars (not printed in the article). Panel B, lines 617--618:\n\n\\texttt{segments(x0 = fig1.b, x1 = fig1.b, y0 = (w.cens.means - 1.96 * boot.ses[\\hl{1:6}]) * 100, y1 = (w.cens.means + 1.96 * boot.ses[\\hl{1:6}]) * 100)}\n\nPanel C, lines 631--632:\n\n\\texttt{segments(x0 = fig1.c, x1 = fig1.c, y0 = (w.means - 1.96 * boot.ses[\\hl{7:12}]) * 100, y1 = (w.means + 1.96 * boot.ses[\\hl{7:12}]) * 100)}\n\nThe calls to \\texttt{arrows()} in lines 619--621 and 633--635 use the same indices.", "corrected": "The caption stands. The bars in Panels B and C are to be redrawn from the weighted bootstrap columns. In the replication code, both calls for Panel B take\n\n\\texttt{boot.ses[\\hl{13:18}]}\n\nand both calls for Panel C take\n\n\\texttt{boot.ses[\\hl{19:24}]}.\n\nThe fourth column of the table above gives approximate replacements, which are 1.87 to 2.39 times as wide as the published bars. The exact bars require a new bootstrap run with the original fitted weights.", "evidence": "The evidence has four parts. First, I read the code that fills the 24 bootstrap columns and the indices in the plotting calls, and I ran those calls to confirm which columns they select. Second, I measured the published figure: the twelve bars in Panels B and C of Figure 1 are as wide as those of Figure 2, to within 0.0096 percentage points. The third part is an exact example in which the two standard errors differ. For two observations with outcomes 0 and 1 and weights 1 and 9, the bootstrap variance of the unweighted mean is $1/8$ and that of the weighted mean is $33/200$. The fourth part is the output files \\texttt{tab\\_3.txt} and \\texttt{tab\\_4.txt} in the replication files, which report weighted means and weighted bootstrap standard errors for the two outcomes. The table compares the published bars with the bars these standard errors give.\n\n\\begin{audittable}{lrrrr}{Figure 1, Panels B and C: weighted estimate, 95\\% bar as published and approximate bar from the saved weighted standard error, in percent. The last column is the saved weighted standard error divided by the standard error behind the published bar.}\nPanel, arm, outcome & Estimate & Bar as published & Bar from weighted SE & Ratio of SEs \\\\\n\\hline\nB, 90/90, approve & 48.5 & 44.2 to 52.8 & 39.5 to 57.5 & 2.11 \\\\\nB, 90/90, prefer & 10.5 & 7.7 to 13.3 & 5.0 to 16.0 & 1.95 \\\\\nB, 90/70, approve & 60.0 & 54.7 to 65.2 & 49.8 to 70.2 & 1.94 \\\\\nB, 90/70, prefer & 53.5 & 48.3 to 58.7 & 42.9 to 64.1 & 2.02 \\\\\nB, 90/45, approve & 64.2 & 60.1 to 68.4 & 54.4 to 74.0 & 2.35 \\\\\nB, 90/45, prefer & 63.1 & 58.9 to 67.3 & 53.1 to 73.1 & 2.39 \\\\\nC, 90/90, approve & 47.2 & 43.1 to 51.3 & 38.4 to 56.0 & 2.15 \\\\\nC, 90/90, prefer & 10.7 & 7.9 to 13.6 & 5.4 to 16.0 & 1.87 \\\\\nC, 90/70, approve & 66.4 & 62.2 to 70.7 & 58.4 to 74.4 & 1.90 \\\\\nC, 90/70, prefer & 61.2 & 56.9 to 65.4 & 52.8 to 69.6 & 1.99 \\\\\nC, 90/45, approve & 62.8 & 58.8 to 66.9 & 53.2 to 72.4 & 2.39 \\\\\nC, 90/45, prefer & 61.4 & 57.3 to 65.5 & 52.0 to 70.8 & 2.31 \\\\\n\\end{audittable}\n\nThe bars in the fourth column are approximate. The saved standard errors are rounded to three decimals and come from bootstrap runs separate from the one that produced the figure. The exact bars require a new run with the fitted weights, and the weights depend on the data of Press, Sagan and Valentino (2013), which are not public.", "problem": "Panels B and C show the replication's weighted estimates, which Section 3 presents as its primary results. The bootstrap in the replication code fills 24 columns. Columns 1 to 12 hold unweighted means, 1 to 6 for the subjects who passed the check and 7 to 12 for all subjects. Columns 13 to 24 hold the weighted means in the same order. The calls that draw Panels B and C centre each bar at a weighted estimate and take its standard error from columns 1 to 6 or 7 to 12. The bars show the uncertainty of the unweighted means. The weighted columns are computed and never drawn. In the published figure the twelve bars have the same widths as the corresponding bars of the unweighted Figure 2, to within 0.01 percentage points. A weighted mean and an unweighted mean of the same observations have different variances in general. Here the weighted mean is the more variable. The weighted standard errors that the authors' own output reports for the same twelve estimates are 1.87 to 2.39 times the standard errors behind the published bars. Someone who judges the precision of the weighted comparison from Figure 1 sees bars about half as wide as the caption describes. In Panel B, for example, the published bars for preference under 90--70 and 90--45 do not overlap (48.3 to 58.7 and 58.9 to 67.3), and the bars from the weighted standard errors do (42.9 to 64.1 and 53.1 to 73.1). Whether two bars overlap is not a test of the difference, and I draw no conclusion about that difference here.", "what_survives": "The weighted point estimates survive, and with them the changes on including the subjects who failed. Under 90--70, approval rises from 60.0\\% to 66.4\\% and preference from 53.5\\% to 61.2\\%. In the full sample the preference estimates under 90--70 and 90--45 are 61.2\\% and 61.4\\%. With the approximate bars, these two preference estimates for the full sample remain above 50\\% at their lower ends. Panel A, the separately bootstrapped bounds of Panels D and E, the unweighted Figure 2 and the results of Section 2 are unaffected by this indexing error. The weighted comparison in Section 3, p.~578, is less precise than the figure shows. The finding does not show that a conclusion of that section reverses."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E014", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the heading Appendix C. Simulations and its first paragraph. It reads: We assume a treatment Z with Supp(Z) = {1, 2} and Pr(Z = 1) = 1/2. We generated potential outcomes Y(1) = Y(2) = lambda [S(2) minus S(1)] + N(0, sigma), and vary lambda, sigma, and the joint distribution of (S(1), S(2)). Note that in the simulation, we have assumed that there is no effect of the treatment whatsoever, and the results would be invariant to the introduction of any constant treatment effect. Lambda represents the divergence in potential outcomes between those who would pass and those who would fail the manipulation check and sigma represents the unexplained variability of potential outcomes. To put our results in asymptopia, we assume N = 1,000, and perform 100,000 simulations. Red outlines surround N = 1,000 and 100,000 in the last sentence.", "caption": "Appendix C, first paragraph, p. 581 (PDF p. 10). The outlines mark the two counts, $N=1{,}000$ and 100,000.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E015.png"}, {"alt": "Crop of the caption of Table 1 and the first two rows of the table. The caption reads: Table 1. Simulations demonstrating the effects of dropping. Simulations performed with N = 1,000 and 100,000 simulations; bound widths are presented as averages over all simulations. The column heads are lambda, Pr[S(1) = 1], Pr[S(2) = 1], rho(S(1), S(2)), sigma, Bias and Bound Width, and the first two rows are 1, 1, 0.8, 0.8, 0.25, 1, 0.300, 1.753 and 2, 10, 0.8, 0.8, 0.25, 1, 3.005, 5.070. Red outlines surround N = 1,000 at the end of the first line of the caption and 100,000 at the start of the second.", "caption": "Caption of Table 1, p. 582 (PDF p. 11), with the head of the table. The outlines mark the same two counts.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E015-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The simulations of Table 1 used $N=100{,}000$ subjects and 1,000 simulations for each parameter specification; Appendix C and the caption of Table 1 interchange the two counts.", "id": "P010-E015", "location": "Appendix C, first paragraph, last sentence, p. 581 (PDF p. 10); caption of Table 1, p. 582 (PDF p. 11)", "pdf_page": 10, "printed_page": "581", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Appendix~C, p.~581.} To put our results in asymptopia, we assume $N=\\hl{1{,}000}$, and perform \\hl{100,000} simulations.\n\n\\emph{Caption of Table~1, p.~582.} Simulations performed with $N=\\hl{1{,}000}$ and \\hl{100,000} simulations; bound widths are presented as averages over all simulations.", "corrected": "\\emph{Appendix~C.} To put our results in asymptopia, we assume $N=\\hl{100{,}000}$, and perform \\hl{1,000} simulations \\hl{for each parameter specification}.\n\n\\emph{Caption of Table~1.} Simulations performed with $N=\\hl{100{,}000}$ and \\hl{1,000} simulations \\hl{for each parameter specification}; \\ldots", "evidence": "The evidence is a check of the replication files. The simulation function is defined by the line \\texttt{do\\_sims <- function (x, n = 100000, niter = 1000)}. In each pass of its loop \\texttt{for (i in 1:niter)} it draws \\texttt{n} assignments, \\texttt{n} pairs of responses to the manipulation check and \\texttt{n} outcome errors. Line 797 of \\texttt{abp\\_replication.R} calls it for the twelve rows without setting either argument. Executing that call up to its first random draw, I recorded $n=100{,}000$ and a loop of length 1,000 for all twelve rows. The log saved with the files prints the progress counter $100,200,\\ldots,1000$ twelve times and no larger value. The saved table \\texttt{tab\\_1.txt} agrees with the published Table~1 in all 84 numerical cells. I ran no simulation at either pair of counts.", "problem": "The two counts are interchanged. The code in the replication files generates $100{,}000$ subjects in each simulated data set and 1,000 data sets for each of the twelve rows of Table~1. The table saved with the files is the published one. The counts that were run cannot be recovered from the paper. The two designs generate the same number of subject records in total, but they are different experiments. Table~1 illustrates the estimators at $N=100{,}000$, and it gives no evidence about their behaviour at $N=1{,}000$.", "what_survives": "Every entry of Table~1 and every comparison that the paper draws from it is unchanged by this correction, since the table was computed with the counts stated here. The results of Section~2 and Appendix~A and the application of Section~3 do not use the simulation counts. Two other corrections concern the entries of Table~1, \\fid{P010-E016} and \\fid{P010-N014}. The alternative to correcting the two sentences is to run the simulations with the printed counts and to replace the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E015", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 1 with its caption. The caption reads: Table 1. Simulations demonstrating the effects of dropping. Simulations performed with N = 1,000 and 100,000 simulations; bound widths are presented as averages over all simulations. The table has twelve numbered rows and the columns lambda, Pr[S(1) = 1], Pr[S(2) = 1], rho(S(1), S(2)), sigma, Bias and Bound Width. The Bound Width entries in rows 1 to 12 are 1.753, 5.070, 16.543, 1.723, 1.613, 3.403, 15.247, 1.569, 4.692, 19.712, 42.635 and 4.589. Rows 1 to 4 have passing rates 0.8 and 0.8, rows 5 to 8 have 0.6 and 0.8, and rows 9 to 12 have 0.6 and 0.6. Red outlines surround the words averages over all simulations in the caption and the whole Bound Width column with its heading.", "caption": "Table 1, p. 582 (PDF p. 11). The outlines mark the Bound Width column and the words “averages over all simulations” in the caption.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E016.png"}, {"alt": "Dot plot with one line for each of the twelve rows of Table 1, in three groups by passing rates: 0.8 and 0.8, 0.6 and 0.8, 0.6 and 0.6. The horizontal axis is the width of the bounds on the difference in means, in units of the simulated outcome, on a logarithmic scale with labeled ticks from 1.5 to 50. Each row has an open orange ring for the printed entry and a filled blue dot for the population width. In rows 1 to 4 and 9 to 12 the ring and the dot nearly coincide. In rows 5 to 8 the ring lies far to the left of the dot, joined to it by a grey bar, with the labels printed 45 percent below, printed 67 percent below, printed 43 percent below and printed 45 percent below. The rings of rows 5 and 8, near 1.6, lie to the left of those of rows 1 and 4, near 1.75, while the dots of rows 5 and 8, near 2.9, lie to the right.", "caption": "The Bound Width entries of Table 1 as printed (open rings) and the population widths of the bounds of Proposition 1 for the same designs (filled dots), on a logarithmic scale. In rows 5 to 8 the printed entry is 43 to 67 percent below the population width; in the other rows the two differ by 2 to 3 percent.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E016-widths.png"}], "formal_release": null, "formal_statements": [], "headline": "The Bound Width column of Table 1 was computed by a function that gives the empirical distribution of one arm the length of the other arm’s sample and drops the simulations in which the result is missing; rows 5 to 8 are 43 to 67 percent below the population widths of the proposed bounds, and the other rows 2 to 3 percent below.", "id": "P010-E016", "location": "Table 1, column “Bound Width” (all twelve rows) and caption, p. 582 (PDF p. 11); the statements about the width of the bounds in Section 2.2, p. 576 (PDF p. 5), and Appendix C, p. 581 (PDF p. 10); replication files, abp_replication_functions.R, lines 138 and 182", "pdf_page": 11, "printed_page": "582", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table~1, p.~582, caption and last column.} Table 1. Simulations demonstrating the effects of dropping. Simulations performed with $N=1{,}000$ and 100,000 simulations; bound widths are presented as \\hl{averages over all simulations}.\n\nBound Width, rows 1 to 12: \\hl{1.753, 5.070, 16.543, 1.723, 1.613, 3.403, 15.247, 1.569, 4.692, 19.712, 42.635, 4.589}.\n\n\\emph{Replication files, \\texttt{abp\\_replication\\_functions.R}, lines 137, 138 and 182 (not printed in the article).}\n\n$\\texttt{OutS0.CDF <- rep(1/nrow(OutS0),nrow(OutS0))}$\n\n$\\texttt{OutS1.CDF <- rep(1/nrow(OutS1),nrow(}\\hl{\\texttt{OutS0}}\\texttt{))}$\n\n$\\texttt{return(c(mean(biased), mean(trimH - trimL, }\\hl{\\texttt{na.rm = TRUE}}\\texttt{)))}$", "corrected": "\\emph{Replication files, line 138.}\n\n$\\texttt{OutS1.CDF <- rep(1/nrow(OutS1),nrow(}\\hl{\\texttt{OutS1}}\\texttt{))}$\n\n\\emph{Line 182.} Take the average of the widths \\hl{without \\texttt{na.rm = TRUE}}. Report missing numerical results rather than dropping them, and record mathematically infinite widths as infinite.\n\n\\emph{Table~1.} Recompute the twelve entries of the Bound Width column with the repaired function, and state in the caption the sample size and the number of simulations of that run (see \\fid{P010-E015}). The population widths for the twelve rows are 1.799, 5.228, 16.959, 1.768, 2.915, 10.297, 26.889, 2.859, 4.806, 20.089, 43.678 and 4.698.", "evidence": "The evidence consists of two exact examples evaluated with the function as deposited, a derivation of the population widths, and a new simulation in which every data set is passed through the function with and without the repair.\n\nIn the first example, the outcomes of the subjects who pass are $(1,2,3,4)$ in arm 0 and $(10,11,12)$ in arm 1, and one subject fails in each arm. The fraction to trim in arm 1 is $(1/5)/(3/4)=4/15$. The function builds the vector $(1/3,2/3,1,4/3)$. All four entries exceed $4/15$, the upper-tail selection is $(10,11,12,\\texttt{NA})$, and its mean is \\texttt{NA}. With a vector of length three, the selection is $(10,11,12)$ and the mean is $11$. In the second example, the outcomes are $(1,2,3)$ and $(10,11,12,13)$, again with one failure in each arm, and the fraction to trim in arm 1 is $(1/4)/(4/5)=5/16$. The vector $(1/4,1/2,3/4)$ is recycled over four outcomes. The lower-tail selection becomes $(10,11,13)$, with mean $34/3$, and the upper-tail selection $(11,12)$, with mean $23/2$. After the length repair, the selections are $(10,11)$ and $(11,12,13)$, with means $21/2$ and $12$. The spread of arm 1 is $1/6$ in place of $3/2$, and the function's width for the difference in means is $2/3$ in place of $2$. These comparisons retain the function's whole-observation cutoff convention.\n\nFor the population widths, let $p_0$ and $p_1$ be the two passing rates, and let\n\\[\nd=p_0p_1+\\rho\\sqrt{p_0(1-p_0)p_1(1-p_1)}\n\\]\nbe the probability of passing under both treatments. Among the subjects who pass, the outcomes follow the mixtures\n\\[\nF_0=\\frac{d}{p_0}N(0,\\sigma^2)+\\Bigl(1-\\frac{d}{p_0}\\Bigr)N(-\\lambda,\\sigma^2),\\qquad\nF_1=\\frac{d}{p_1}N(0,\\sigma^2)+\\Bigl(1-\\frac{d}{p_1}\\Bigr)N(\\lambda,\\sigma^2).\n\\]\nThe bounds retain the fraction $r_j=(p_0+p_1-1)/p_j$ of $F_j$. Let $Q_j$ be the quantile function of $F_j$. The width of the bounds on the difference in means is\n\\[\nW=\\sum_{j=0}^{1}\\frac{1}{r_j}\\Bigl[\\int_{1-r_j}^{1}Q_j(u)\\,du-\\int_{0}^{r_j}Q_j(u)\\,du\\Bigr].\n\\]\nNote that these widths are large-sample values. They are not the averages that a corrected run at a finite sample size would give, and the data sets of the original run are not available.\n\nIn the new simulation, I drew 100 data sets with $n=100{,}000$ for each row from the exact joint distribution of $(S(1),S(2))$. I passed each through the function as deposited and through the function with line 138 repaired. The function as deposited returned a width for 39 to 53 of the 100 data sets in the rows with equal passing rates and for all 100 in rows 5 to 8. Its averages lie close to the printed entries. With the repair, every data set gave a finite width, and the averages lie close to the population widths. Among data sets with widths from both functions, the average increase is between 8.9 and 713 Monte Carlo standard errors in the twelve rows.\n\\begin{audittable}{rlrrrrrr}{The Bound Width column of Table~1. ``Deposited'' and ``Repaired'' are averages over 100 new data sets with $n=100{,}000$ for each row, computed by the function as deposited and with line 138 repaired; ``Kept'' is the number of these data sets for which the function as deposited returned a width. ``Population'' is the large-sample width of the bounds of Proposition~1 for the design of the row, and ``Shortfall'' is the amount by which the printed entry lies below it.}\nRow & Rates & Printed & Deposited & Kept & Repaired & Population & Shortfall \\\\\n1 & 0.8, 0.8 & 1.753 & 1.752 & 50 & 1.798 & 1.799 & 2.5\\% \\\\\n2 & 0.8, 0.8 & 5.070 & 5.079 & 48 & 5.232 & 5.228 & 3.0\\% \\\\\n3 & 0.8, 0.8 & 16.543 & 16.605 & 52 & 16.965 & 16.959 & 2.5\\% \\\\\n4 & 0.8, 0.8 & 1.723 & 1.716 & 46 & 1.768 & 1.768 & 2.5\\% \\\\\n5 & 0.6, 0.8 & 1.613 & 1.616 & 100 & 2.916 & 2.915 & 44.7\\% \\\\\n6 & 0.6, 0.8 & 3.403 & 3.441 & 100 & 10.300 & 10.297 & 67.0\\% \\\\\n7 & 0.6, 0.8 & 15.247 & 15.271 & 100 & 26.929 & 26.889 & 43.3\\% \\\\\n8 & 0.6, 0.8 & 1.569 & 1.569 & 100 & 2.859 & 2.859 & 45.1\\% \\\\\n9 & 0.6, 0.6 & 4.692 & 4.686 & 51 & 4.806 & 4.806 & 2.4\\% \\\\\n10 & 0.6, 0.6 & 19.712 & 19.772 & 39 & 20.084 & 20.089 & 1.9\\% \\\\\n11 & 0.6, 0.6 & 42.635 & 42.570 & 53 & 43.650 & 43.678 & 2.4\\% \\\\\n12 & 0.6, 0.6 & 4.589 & 4.588 & 41 & 4.709 & 4.698 & 2.3\\% \\\\\n\\end{audittable}", "problem": "In each simulated data set, the function \\texttt{returnbounds\\_sims()} sorts the outcomes of the subjects who pass in each arm and attaches to them the values of their empirical distribution function. It uses these values to select the tails to be trimmed. Let $m_0$ and $m_1$ be the numbers of subjects who pass in the code's arms 0 and 1, which are the paper's treatments 1 and 2. For arm 1, the function builds a vector of $m_0$ entries equal to $1/m_1$, where $m_1$ entries are needed. It uses the cumulative sums $1/m_1,2/m_1,\\ldots,m_0/m_1$ to select among the $m_1$ outcomes. Suppose that $m_0>m_1$ and the fraction to trim is less than one. Then the entries after the $m_1$-th exceed one, and R selects an outcome that does not exist. The upper endpoint and the width for that data set are missing. Line 182 averages the widths with \\texttt{na.rm = TRUE}. The reported number is then an average over the remaining data sets only, and not over all simulations as the caption says. Suppose instead that $m_0<m_1$. Then the selection vector is too short, and R recycles it. This can select outcomes outside the intended tails: the largest outcomes of the arm can be left out of its upper-tail mean and enter its lower-tail mean.\n\nAll twelve entries of the column are affected. In rows 5 to 8, the passing rates are 0.6 and 0.8, and the ratio $m_0/m_1$ is typically close to $3/4$ in large samples. The vector then covers only the lowest three quarters of the outcomes of arm 1. The printed widths 1.613, 3.403, 15.247 and 1.569 are close to the large-sample limit of this computation, which is 1.614, 3.427, 15.247 and 1.568. They are 43 to 67 percent below the population widths of the bounds of Proposition~1, which are 2.915, 10.297, 26.889 and 2.859. In the other eight rows the passing rates are equal. About half of the data sets have $m_0>m_1$ and are dropped; among the rest, those with $m_0<m_1$ use a recycled vector. The printed widths are 1.9 to 3.0 percent below the population widths.\n\nAs printed, the widths in rows 5 to 8 are smaller than the widths in rows 1 to 4, although the failure rate is higher. This contradicts the statements of Section~2.2 and Appendix~C that the width of the bounds increases with the failure rates, for which Appendix~C cites rows 1--4, 5--8 and 9--12. The population widths increase from each of these blocks to the next. The statements hold in these designs, but the printed column does not support them.", "what_survives": "The Bias column of Table~1 is computed in other lines of the code and is not affected by this error. The bounds of the application in Section~3 are computed by a different function, \\texttt{returnbounds()}, which builds the distribution of each arm from the weights of that arm. The theoretical results and the paper's warning about dropping subjects do not use the simulation function. The population widths agree with what the paper says about the width in these designs: it grows with the failure rates, and with $\\sigma$ and $\\lambda$ in the comparisons that the paper draws. All twelve entries of the column have to be recomputed. The repair needs no new assumption."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E016", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of the paragraph that follows Proposition 1. It says that bounds can be constructed by trimming the observations associated with the upper or lower proportions, given by the sum of Pr[S = 0 | Z = z'] over Pr[S = 1 | Z = z], of the empirical distributions of subjects who pass the manipulation check. A red outline surrounds the sentence: With weighted data, this entails using the weighted empirical distribution function.", "caption": "Paragraph after Proposition 1, p. 575 (PDF p. 4): the passage that the deposited functions implement. The outline marks the sentence on weighted data; the passage itself is correct.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E017.png"}, {"alt": "Listing of ten lines of R code with line numbers. Lines 103 to 105 copy the normalized weights of the passers of three arms into vectors named OutS0.CDF, OutS1.CDF and OutS2.CDF. Lines 107 to 109 accumulate them in loops over 2:length of each vector. Lines 111 and 112 compute Out0U and Out0L as weighted means of the outcomes selected by OutS0.CDF>trim0 and by OutS0.CDF<(1-trim0). Red outlines surround 2:length(OutS0.CDF) in line 107 and the two comparisons, each of which occurs twice in its line.", "caption": "Replication code, not a page of the article: returnbounds() in abp_replication_functions.R, lines 103–112. The outlines mark the loop that fails for a single passer and the strict comparisons that select whole observations.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-E017-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The deposited functions that compute the bounds keep or drop whole observations at the trimming cut-off; the fraction of the passers' distribution that they retain is in general not the fraction that Proposition 1 specifies, and the bounds they return are not the sharp ones.", "id": "P010-E017", "location": "Paragraph after Proposition 1, p. 575 (PDF p. 4); replication code, returnbounds() and returnbounds_sims() (abp_replication_functions.R, lines 76–149)", "pdf_page": 4, "printed_page": "575", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Article, p.~575: ``As with Lee (2009)'s bounds, even when $Y$ is discrete, then bounds can be constructed simply by trimming the observations associated with the upper or lower $\\sum_{z'\\in\\{1,\\ldots,K\\}:z'\\ne z}\\frac{\\Pr[S=0\\mid Z=z']}{\\Pr[S=1\\mid Z=z]}$th proportions of the empirical distributions of subjects who pass the manipulation check under treatment and control. With weighted data, this entails using the weighted empirical distribution function.''\n\nReplication code, \\texttt{returnbounds()}, lines 111--112 (not printed in the article):\n\n\\texttt{Out0U <- weighted.mean(OutS0\\$Out[\\hl{OutS0.CDF>trim0}], OutS0\\$Weight[\\hl{OutS0.CDF>trim0}])}\n\n\\texttt{Out0L <- weighted.mean(OutS0\\$Out[\\hl{OutS0.CDF<(1-trim0)}], OutS0\\$Weight[\\hl{OutS0.CDF<(1-trim0)}])}", "corrected": "The article's sentences stand. The computation that they describe is as follows. In arm $z$, let the passers have outcomes $y_1\\le\\cdots\\le y_n$ and weights $w_1,\\ldots,w_n$ that sum to 1, and let $C_i=w_1+\\cdots+w_i$, with $C_0=0$. Let $t_z<1$ be the trimming ratio of the arm and $k_z=1-t_z$. The lower and upper bounds are\n\\[\nL_z=\\frac{1}{k_z}\\sum_{i=1}^n y_i\\,\\hl{\\min\\{w_i,\\ \\max(0,\\ k_z-C_{i-1})\\}},\\qquad\nU_z=\\frac{1}{k_z}\\sum_{i=1}^n y_i\\,\\hl{\\min\\{w_i,\\ \\max(0,\\ C_i-t_z)\\}}.\n\\]\nFor a binary outcome with mean $\\mu_z$ among the passers, these are $L_z=\\max\\{0,(\\mu_z-t_z)/(1-t_z)\\}$ and $U_z=\\min\\{1,\\mu_z/(1-t_z)\\}$. For $t_z=0$ both bounds equal the passers' mean outcome. For $t_z\\ge 1$, and a positive probability of passing under all conditions, the bounds are the smallest and the largest outcome among the passers of the arm (see \\fid{P010-E002}). The cumulative weights should be computed in a way that also works for a single passer.", "evidence": "The evidence is an exact counterexample, which I also ran through the deposited function, and a recalculation of the application's unweighted bounds.\n\nLet each of three arms contain six observations: five passers with outcomes $0,0,1,1,1$ and weight 1 each, and one subject who failed, with weight $4/5$. Each passing rate is $25/29$, the trimming ratio is $t_z=8/25$, the fraction to retain is $17/25$, and the passers have mean outcome $3/5$. The sharp bounds are\n\\[\n\\left[\\frac{3/5-8/25}{17/25},\\ \\frac{3/5}{17/25}\\right]=\\left[\\frac{7}{17},\\ \\frac{15}{17}\\right].\n\\]\nThe passers' cumulative weights are $1/5,2/5,3/5,4/5,1$. For the upper bound the function keeps the four passers whose cumulative weight exceeds $8/25$, of mass $4/5$, and for the lower bound the three whose cumulative weight is below $17/25$, of mass $3/5$. It returns $[1/3,3/4]$. There exists a population with the same observed distribution in every arm such that the subjects who would pass under all conditions have mean outcome $15/17$. In it these subjects have probability $17/29$, and those among them with outcome 1 have probability $15/29$. The returned value $3/4$ is therefore not an upper bound, and it falls short by $9/68$.\n\nIn the application, I recomputed the 12 unweighted bounds of Figure 2, Panels D and E, with exact fractions. The largest change is $0.23468$ percentage points. It occurs in the upper bound for approval under 90--70, which is $66.667\\%$ from the deposited function and $66.901\\%$ with exact fractions. Each bound stays on the same side of $50\\%$, and the order of the bounds within each outcome is unchanged.", "problem": "The article describes the correct computation. Let $t_z$ denote the trimming ratio of arm $z$, that is, the sum in the quoted sentence. The bounds are the means of the lower and of the upper fraction $1-t_z$ of the passers' weighted empirical distribution. In general the cut-off falls inside an observation, and only a part of its weight is kept. The deposited functions select whole observations instead. They compare the cumulative weight of each observation with the cut-off, with strict inequalities. The mass that they retain is in general not $1-t_z$. The upper bound that is returned can lie below a value of the target that the data permit, and the lower bound need not be sharp. There are two further defects. The loops over \\texttt{2:length(CDF)} stop with an error when an arm has a single passer. When the cut-off coincides with a cumulative weight, the strict comparisons depend on rounding in the accumulated sums. Panels D and E of Figures 1 and 2 and their bootstrap standard errors use \\texttt{returnbounds()}. The Bound Width column of Table 1 uses \\texttt{returnbounds\\_sims()}.", "what_survives": "This implementation error does not refute the trimming argument of Proposition 1. Correcting the two functions does not change the point estimates of Panels A to C in both figures, the covariate tables or the Bias column of Table 1. The effect on the weighted bounds of Figure 1 is not known, since the weights cannot be rebuilt without the data of Press, Sagan and Valentino (2013), which are not public. I did not quantify the effect on the bootstrap intervals of the bound panels or on the widths in Table 1. A separate defect of that column is the subject of \\fid{P010-E016}."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-E017", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of the first paragraph of the introduction. The sentence reads: Articles including this practice have been published in top journals in multiple disciplines over recent years, including the American Political Science Review, the Journal of Personality and Social Psychology, Psychological Science, and the Journal of Marketing. A red outline surrounds the words Journal of Marketing, which are followed by the marker of footnote 2.", "caption": "Section 1, first paragraph, p. 572 (PDF p. 1). The outline marks the fourth journal named in the sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "The supplementary bibliography contains no article from the Journal of Marketing; its two marketing entries are from the Journal of Marketing Research.", "id": "P010-N001", "location": "Section 1, first paragraph, p. 572 (PDF p. 1)", "pdf_page": 1, "printed_page": "572", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Articles including this practice have been published in top journals in multiple disciplines over recent years, including the \\emph{American Political Science Review}, the \\emph{Journal of Personality and Social Psychology}, \\emph{Psychological Science}, and the \\emph{\\hl{Journal of Marketing}}.", "corrected": "\\ldots\\ \\emph{Psychological Science}, and the \\emph{Journal of Marketing \\hl{Research}}.\n\nIf a qualifying article in the \\emph{Journal of Marketing} was meant, that journal name can remain and the article should be added to the bibliography.", "evidence": "The evidence is a count of the journal titles in the supplementary bibliography. The count shows that the sentence does not match the bibliography. It does not show that the \\emph{Journal of Marketing} has never published such a study, and I did not search that journal.", "problem": "The sentence follows the count of 59 articles and 36 dissertations, and it reads as a list of examples from the bibliography in the supplementary information. That bibliography has 151 entries, and none is from the \\emph{Journal of Marketing}. Two of the 95 entries coded as dropping subjects after a posttreatment check are from the \\emph{Journal of Marketing Research}, which is a different journal. The other three journals named in the sentence have 1, 7 and 1 such entries.", "what_survives": "Nothing depends on the sentence. The counts of 59 articles and 36 dissertations agree with the bibliography."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N001", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines. The text reads: Finally, we require that at least some subjects pass the manipulation check in both treatment and control. Assumption 3 (Nonzero Passing Rates). For all z in the set 1 to K, Pr[S | Z = z] > 0. Note that Assumption 3 is verifiable from the experimental data. Red outlines surround the words both treatment and control, across a line break, and the expression Pr[S | Z = z], in which S is not followed by = 1.", "caption": "Assumption 3 and the sentence that introduces it, p. 574 (PDF p. 3). The outlines mark the probability printed without an event for $S$ and the words “both treatment and control”.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N002.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T17", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T17"}], "headline": "Assumption 3 omits the event for $S$: the condition is $\\Pr[S=1\\mid Z=z]>0$ for every arm $z$.", "id": "P010-N002", "location": "Assumption 3 and the sentence before it, p. 574 (PDF p. 3)", "pdf_page": 3, "printed_page": "574", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T17"], "text": {"as_printed": "Finally, we require that at least some subjects pass the manipulation check in \\hl{both treatment and control}.\n\n\\textbf{Assumption 3 (Nonzero Passing Rates).} For all $z\\in\\{1,\\ldots,K\\}$, $\\Pr[\\hl{S}\\mid Z=z]>0$.", "corrected": "Finally, we require that at least some subjects pass the manipulation check in \\hl{every treatment arm}.\n\n\\textbf{Assumption 3 (Nonzero Passing Rates).} For all $z\\in\\{1,\\ldots,K\\}$, $\\Pr[S\\hl{{}=1}\\mid Z=z]>0$.", "evidence": "The evidence is a check of the page. Proposition 1 on p.~575 divides by $\\Pr[S=1\\mid Z=z]$ and conditions on $S=1$.", "problem": "The variable $S$ equals 1 for a subject who passes the check and 0 otherwise, and a probability needs an event. The name of the assumption and the sentence before it give the event $S=1$. That sentence speaks of treatment and control, while the setting has $K$ arms; the display quantifies over all of them.", "what_survives": "All results survive. They use the condition as corrected."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N002", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines. Corollary 1 reads: If Pr[S(1) = S(2) = ... = S(K)] = 1, then E[Y | S = 1, Z = z] - E[Y | S = 1, Z = z'] = E[Y(z) - Y(z')] | S(1) = S(2) = ... = S(K) = 1]. A red outline surrounds the closing square bracket that follows Y(z') and precedes the conditioning bar.", "caption": "Corollary 1 and the sentence before it, p. 574 (PDF p. 3). The outline marks the stray bracket.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N003.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T18", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T18"}], "headline": "The display of Corollary 1 has a stray closing bracket after $Y(z')$; the conditioning event belongs inside the expectation.", "id": "P010-N003", "location": "Corollary 1, p. 574 (PDF p. 3)", "pdf_page": 3, "printed_page": "574", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P010-T18"], "text": {"as_printed": "\\textbf{Corollary 1.} If $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$, then $\\mathrm{E}[Y\\mid S=1,Z=z]-\\mathrm{E}[Y\\mid S=1,Z=z']=\\mathrm{E}[Y(z)-Y(z')\\hl{]}\\mid S(1)=S(2)=\\cdots=S(K)=1]$.", "corrected": "\\textbf{Corollary 1.} If $\\Pr[S(1)=S(2)=\\cdots=S(K)]=1$, then $\\mathrm{E}[Y\\mid S=1,Z=z]-\\mathrm{E}[Y\\mid S=1,Z=z']=\\mathrm{E}[Y(z)-Y(z')\\hl{\\mid}S(1)=S(2)=\\cdots=S(K)=1]$.", "evidence": "The evidence is a check of the page. Corollary 3 on p.~575 prints the same expression with the conditioning event inside the expectation.", "problem": "As printed, the expectation closes before the conditioning bar, and the last bracket has no partner. The expression intended is the target defined earlier on the same page, the average effect among subjects who would pass under all conditions.", "what_survives": "The corollary and its proof are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N003", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Six lines of body text from page 578. Red boxes mark the words other (representative) at the end of one line and samples (Hauser and Schwarz 2016) at the start of the next.", "caption": "Section 3, p. 578: the sentence on the attentiveness of Mechanical Turk subjects. The comparison group and its citation are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "The cited article compares Mechanical Turk participants with undergraduates; the comparison with “representative” samples, and with the sample of PSV, is an extrapolation.", "id": "P010-N005", "location": "Section 3, p. 578 (PDF p. 7), sentence beginning “Note that there is a good body of evidence”", "pdf_page": 7, "printed_page": "578", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Note that there is a good body of evidence to suggest that subjects on Mechanical Turk tend to be more attentive than \\hl{other (representative) samples (Hauser and Schwarz 2016)}, and thus it is possible that our passing rates are higher than those of PSV.", "corrected": "Note that there is evidence that subjects on Mechanical Turk tend to be more attentive than \\hl{college subject-pool participants (Hauser and Schwarz 2016), and that article cites earlier comparisons with adult internet panels}, and thus it is possible, \\hl{although this evidence does not concern the sample of PSV}, that our passing rates are higher than those of PSV.", "intended_meaning": "The sentence offers a possibility, that the passing rates in the replication exceed those of PSV, and the next sentence draws its consequences only conditionally (``If so''). The three studies of Hauser and Schwarz (2016) compare Mechanical Turk participants with undergraduates (samples of 396 and 85, 185 and 245, and 149 and 90). Comparisons with adult internet panels appear only in the earlier work that the article cites, and nothing in it concerns the sample or the manipulation check of PSV. The remark is hedged, is not shown to be false, and no result of the paper depends on it."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N005", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Seven lines of body text from the top of page 579. Red boxes mark the words conditions have overlapping regions, in both outcome variables, then largely uninformative, then very little is revealed.", "caption": "Section 3, top of p. 579: the end of the sentence that begins at the foot of p. 578 with “We observe that the bounds for all treatment”. The overlap claim and the two generalizations are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N011.png"}, {"alt": "Two bar charts from page 577, titled Replication (Lower Bounds; Weighted) and Replication (Upper Bounds; Weighted), each with a dark bar for Approve Nuclear Use and a light bar for Prefer Nuclear Use in the conditions 90/90, 90/70 and 90/45. Red boxes mark the light bars for 90/70 and 90/45 in the lower-bound panel, which reach about 42 and 26 percent, and the light bar for 90/90 in the upper-bound panel, which reaches about 20 percent.", "caption": "Figure 1, Panels D and E, p. 577: the estimated lower bounds (Panel D) and upper bounds (Panel E). The three “Prefer Nuclear Use” bars used in the comparison are marked: the upper bound for 90–90 (about 19.7%) and the lower bounds for 90–70 (about 41.5%) and 90–45 (about 26.5%).", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N011-figure1.png"}, {"alt": "Four panels of horizontal intervals on a scale from 0 to 100 percent, with rows for Figure 1 (weighted) and Figure 2 (unweighted) and columns for Prefer nuclear use and Approve nuclear use. In the preference panels the interval for 90-90, from 0 to 19.7 in Figure 1 and from 0 to 21.4 in Figure 2, ends before the intervals for 90-70 (41.5 to 66.8, and 31.0 to 57.4) and 90-45 (26.5 to 100, and 40.3 to 100) begin; a grey band marks the gap. In the approval panels the three intervals overlap: 1.1 to 91.1, 49.4 to 74.7 and 29.6 to 100 in Figure 1, and 0 to 67.4, 40.2 to 66.7 and 39.2 to 100 in Figure 2.", "caption": "The estimated bounds of Figure 1 (weighted; bar heights read from the published figure) and of Figure 2 (unweighted; recomputed from the replication data), drawn as intervals. For preference the interval for 90–90 lies below both others; for approval all three overlap.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N011-bounds.png"}], "formal_release": null, "formal_statements": [{"id": "P010-T1", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T1"}, {"id": "P010-T2", "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-T2"}], "headline": "The estimated bounds do not overlap across all treatment conditions for both outcomes: for “prefer nuclear use” the bounds for 90–90 lie wholly below those for 90–70 and for 90–45, in both figures.", "id": "P010-N011", "location": "Section 3, last paragraph, pp. 578–579 (PDF pp. 7–8); Figure 1, Panels D–E, p. 577 (PDF p. 6); Figure 2, Panels D–E, p. 585 (PDF p. 14)", "pdf_page": 7, "printed_page": "578–579", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P010-T1", "P010-T2"], "text": {"as_printed": "We observe that the bounds for \\hl{all treatment conditions have overlapping regions, in both outcome variables}, making it impossible to fully differentiate results across the treatments. The bounds are \\hl{largely uninformative}, suggesting the fundamental uncertainty induced by attempting to estimate effects among the subpopulation of subjects who would always pass the manipulation check. If these effects are indeed the inferential target of the researcher, \\hl{very little is revealed} by the experimental data, and dropping subjects may introduce serious bias of unknown sign and magnitude (as any values within the range of the bounds are compatible with the experimental data).", "corrected": "\\hl{The estimated approval bounds overlap across all three treatments. For preference, the estimated upper bound for 90--90 is below the estimated lower bounds for both 90--70 and 90--45, while the bounds for 90--70 and 90--45 overlap.} Thus the estimated preference bounds distinguish equal effectiveness from a nuclear advantage, although the bounds remain wide and do not fully order all three treatments or determine precise effect magnitudes. Sampling uncertainty is distinct from the width of these bounds.\n\nThe two sentences that follow in the paper, on bounds that are ``largely uninformative'' and data that reveal ``very little'', should be qualified to acknowledge these two comparisons.", "evidence": "The evidence is a recalculation from the replication data and a check of the published figures. For Figure 2, recomputing the bounds in exact arithmetic, with the trimming rule of the replication code, I find the preference intervals $[0, 61/285]$, $[87/281, 27/47]$ and $[56/139, 1]$. They agree with the output of the replication code's own function and with the plotted bars. The lower bounds on the two contrasts with 90--90 are\n\\[\n\\frac{87}{281}-\\frac{61}{285}=\\frac{7654}{80085}\n\\qquad\\text{and}\\qquad\n\\frac{56}{139}-\\frac{61}{285}=\\frac{7481}{39615},\n\\]\nthat is, 9.56 and 18.88 percentage points. With exact fractional trimming at the cut-off (see \\fid{P010-E017}) the two gaps are 9.50 and 18.91 points. For Figure 1 the weighted estimates cannot be recomputed, because the PSV data are withheld from the replication files. The heights of its bars were read from the drawing instructions of the published figure, and the corresponding gaps are about 21.8 and 6.8 points.\n\n\\begin{audittable}{llll}{Estimated bounds on the mean outcome among subjects who would pass under all three treatments, in percent. Figure 1: read from the published figure. Figure 2: recomputed from the replication data.}\nFigure and outcome & 90--90 & 90--70 & 90--45 \\\\\nFigure 1 (weighted), prefer & $[0,\\ 19.7]$ & $[41.5,\\ 66.8]$ & $[26.5,\\ 100]$ \\\\\nFigure 1 (weighted), approve & $[1.1,\\ 91.1]$ & $[49.4,\\ 74.7]$ & $[29.6,\\ 100]$ \\\\\nFigure 2 (unweighted), prefer & $[0,\\ 21.4]$ & $[31.0,\\ 57.4]$ & $[40.3,\\ 100]$ \\\\\nFigure 2 (unweighted), approve & $[0,\\ 67.4]$ & $[40.2,\\ 66.7]$ & $[39.2,\\ 100]$ \\\\\n\\end{audittable}", "problem": "Panels D and E of Figures 1 and 2 show, for each condition, the estimated lower and upper bounds on the mean outcome among subjects who would pass the manipulation check under all three treatments. The sentence says that these intervals overlap across all conditions for both outcomes. For approval they do. For preference they do not. In the weighted Figure 1 the interval for 90--90 is about $[0, 19.7]$ percent, against $[41.5, 66.8]$ for 90--70 and $[26.5, 100]$ for 90--45. In the unweighted Figure 2 the three intervals are $[0, 21.4]$, $[31.0, 57.4]$ and $[40.3, 100]$. In both figures the upper bound for 90--90 is below the lower bounds for the other two conditions. The claim then fails whether ``all'' is read as a common intersection or as pairwise overlap. The estimated bounds separate the condition of equal effectiveness from the two conditions with a nuclear advantage, and only the intervals for 90--70 and 90--45 overlap each other. The two sentences that follow, that the bounds are ``largely uninformative'' and that ``very little is revealed'', are too strong for the preference outcome. They are judgments without a fixed threshold, and I do not say that they are false: the bounds are wide, and they do not order 90--70 against 90--45.", "what_survives": "The approval bounds overlap across all three conditions, the preference bounds for 90--70 and 90--45 overlap each other, and all the bounds are wide. The statement that the treatments cannot be fully differentiated stands. The general warning about dropping subjects, the identification results and the comparison of Panels B and C do not depend on this sentence. What changes is the claim that the bounds reveal very little: for preference, the estimated bounds place 90--90 below both other conditions. This is a statement about the estimated bounds only, and it does not settle the question of sampling uncertainty: in Figure 1 the 95\\% bars on the upper bound for 90--90 overlap the bars on the two lower bounds."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N011", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the second paragraph of Appendix B. It reads: First, because the original experiment was performed prior to the onset of the Syrian civil war, we sought to assess whether the results were invariant to shifts in time and context (i.e., whether the results might differ in our replication, given the political changes that have occurred in Syria). We thus randomized whether treatment frames presented the scenario in Syria or Lebanon, which was used as an analog to pre-civil-war Syria; treatments were assigned through a 2 by 5 factorial design. We found no statistically or substantively significant difference between Syria and Lebanon treatment frames, demonstrating that the results presented in PSV are robust to these temporal and contextual changes. Red outlines surround the last sentence, which begins We found no statistically, on its three lines.", "caption": "Appendix B, second paragraph, p. 581 (PDF p. 10). The outline marks the sentence that reports no difference between the Syria and Lebanon frames.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N012.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication code computes the Syria–Lebanon point differences and their $p$-values with the failure flags of the last bootstrap sample and with standard errors for a different contrast; the four $p$-values are never printed or used, and the reported absence of a difference agrees with the saved intervals.", "id": "P010-N012", "location": "Appendix B, second paragraph, last sentence, p. 581 (PDF p. 10); replication files, abp_replication.R, lines 880–893 and 1158–1171 (the code is not printed in the article)", "pdf_page": 10, "printed_page": "581", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Appendix~B, p.~581.} We found no statistically or substantively significant difference between Syria and Lebanon treatment frames, demonstrating that the results presented in PSV are robust to these temporal and contextual changes.\n\n\\emph{Replication files, \\texttt{abp\\_replication.R}, lines 880--885 (not printed in the article; indentation reduced).}\n\n$\\texttt{w.sl.approv.diff <-}$\n\n$\\texttt{~~with(weighted.dat[weighted.dat\\$rep.treat \\%in\\% c(\"1L\", \"2L\", \"3L\") \\&}$\n\n$\\texttt{~~~~~~}\\hl{\\texttt{samp\\$failed == 1}}\\texttt{,], weighted.mean(approv, w = weights)) -}$\n\n$\\texttt{~~with(weighted.dat[weighted.dat\\$rep.treat \\%in\\% c(\"1S\", \"2S\", \"3S\") \\&}$\n\n$\\texttt{~~~~~~weighted.dat\\$failed == 1,], weighted.mean(approv, w = weights))}$\n\n$\\texttt{w.sl.approv.p <- 2 * (1 - pnorm(abs(w.sl.approv.diff / }\\hl{\\texttt{w.sl.se[7]}}\\texttt{)))}$", "corrected": "\\emph{Replication files, lines 880--885, with the same change in lines 888--893.} The weighted point difference is taken over all subjects, as in the bootstrap that supplies its standard error:\n\n$\\texttt{w.sl.approv.diff <-}$\n\n$\\texttt{~~with(weighted.dat[weighted.dat\\$rep.treat \\%in\\% c(\"1L\", \"2L\", \"3L\"),],}$\n\n$\\texttt{~~~~~~~weighted.mean(approv, w = weights)) -}$\n\n$\\texttt{~~with(weighted.dat[weighted.dat\\$rep.treat \\%in\\% c(\"1S\", \"2S\", \"3S\"),],}$\n\n$\\texttt{~~~~~~~weighted.mean(approv, w = weights))}$\n\n\\emph{Lines 1158--1171.} In the two unweighted differences, the data frame \\texttt{samp} in the condition \\texttt{samp\\$failed == 1} becomes \\hl{\\texttt{weighted.dat}}, and the block is labelled as a comparison among the subjects who failed the manipulation check. The sentence in Appendix~B needs no change.", "evidence": "The evidence is an exact counterexample, a direct evaluation of the expressions in the replication files, and a recalculation from the public data. The counterexample has 24 rows, four in each of the six cells 1L to 3S. I ran the two bootstrap loops twice with 10,000 specified samples, and the two runs differ only in the order of the rows of the last sample. All bootstrap statistics and standard errors agree between the runs. The weighted difference for approval is $0$ in one run and $1/3$ in the other, with $p$-values $1$ and $4.9\\times10^{-8}$. The unweighted difference is $0$ and $1/2$, with $p$-values $1$ and $0.046$. In the same example the weighted difference over all subjects is $7/10$, and the weighted difference among the subjects who failed is $0$.\n\nFor the 1,615 subjects of treatments 1 to 3 in the public data, without weights, approval is $429/807$ under Lebanon and $420/808$ under Syria, a difference of 1.2 percentage points ($p=0.64$). Preference is $354/807$ and $329/808$, a difference of 3.1 points ($p=0.20$). The log saved with the replication files prints the weighted bootstrap intervals for the two pooled contrasts over all subjects, $[-0.111,0.094]$ for approval and $[-0.041,0.185]$ for preference. All eight weighted intervals that it prints include zero. The weighted point estimates could not be recomputed, since the data of Press, Sagan and Valentino that define the weights are not public.", "problem": "The data frame \\texttt{samp} is the last sample drawn in the bootstrap loop that precedes these lines. R combines the two logical vectors position by position, and the condition \\texttt{samp\\$failed == 1} selects the Lebanon rows of the original data by the failure flags of unrelated rows. The point difference and its $p$-value then depend on the order of the last bootstrap sample. The same expression occurs in the weighted difference for preference (lines 888--893) and in the two unweighted differences (lines 1158--1171). There is a second mismatch in the weighted lines. The standard errors \\texttt{w.sl.se[7]} and \\texttt{w.sl.se[8]} come from a bootstrap of the contrast over all subjects, while the Syria side of the numerator is restricted to the subjects who failed the manipulation check. Replacing \\texttt{samp} by \\texttt{weighted.dat} alone would not remove this mismatch. The unweighted bootstrap restricts both sides to the subjects who failed, which is consistent but is a different comparison from one over all subjects. The four $p$-values are assigned and never used again, and the saved log does not print them. The replication files do not show which numbers the sentence in Appendix~B rests on.", "what_survives": "The sentence in Appendix~B is not shown to be false. The weighted bootstrap intervals in the saved log are computed before the faulty lines, and they include zero. The recalculation over all subjects without weights does not reject equality either, although it does not establish that the two frames are equivalent. No table, figure or estimate of the paper uses the four $p$-values."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N012", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the third paragraph of Appendix B. It reads: Second, we analyzed whether the PSV study's use of posttreatment covariates introduced bias. We added another treatment (rendering our augmented replication a 2 by 2 by 5 factorial design) that randomized whether subjects answered these questions before or after treatment. This analysis failed to reveal any statistically or substantively significant results. Red outlines surround the last sentence, which begins This analysis at the end of the third line and continues on the fourth.", "caption": "Appendix B, third paragraph, p. 581 (PDF p. 10). The outline marks the sentence that reports no significant results for the placement of the covariate questions.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N013.png"}], "formal_release": null, "formal_statements": [], "headline": "The statement that the analysis of the placement of the covariate questions revealed no significant results holds for the primary, weighted analysis; the unweighted analysis in the replication files gives a nominally significant difference for one of the two outcomes.", "id": "P010-N013", "location": "Appendix B, third paragraph, last sentence, p. 581 (PDF p. 10)", "pdf_page": 10, "printed_page": "581", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Second, we analyzed whether the PSV study's use of posttreatment covariates introduced bias. We added another treatment (rendering our augmented replication a $2\\times2\\times5$ factorial design) that randomized whether subjects answered these questions before or after treatment. \\hl{This analysis} failed to reveal any statistically or substantively significant results.", "corrected": "\\hl{The primary weighted analysis} failed to reveal any statistically or substantively significant results.", "intended_meaning": "Section~3 (p.~578) names the analysis with inverse probability weights as the primary one, and the sentence is naturally read as a report of that analysis. In that sense it is correct. The log saved with the replication files gives the weighted bootstrap 95\\% intervals $[-0.134,0.066]$ for approval and $[-0.155,0.062]$ for preference, and both include zero. The same log contains the unweighted version of the analysis. Its interval for preference, $[-0.099,-0.003]$, excludes zero, and its interval for approval, $[-0.095,0.004]$, includes it. In the public data, $335$ of the $841$ subjects who answered the covariate questions before treatment preferred the nuclear strike, against $348$ of the $774$ who answered them afterwards. The difference is $-5.1$ percentage points, with nominal $p=0.037$ from the normal approximation and $p=0.039$ from Fisher's exact test. No adjustment for multiple comparisons is applied, and the paper sets no threshold for substantive significance. The unweighted result does not establish an effect in the population, a bias from conditioning on posttreatment covariates or a reversal of the conclusions of PSV. A further sentence could report it: in the unweighted analysis the estimated difference in preference is $-5.1$ percentage points, with nominal bootstrap 95\\% interval $[-9.9,-0.3]$."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N013", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Table 1 with its caption. The table has twelve numbered rows and the columns lambda, Pr[S(1) = 1], Pr[S(2) = 1], rho(S(1), S(2)), sigma, Bias and Bound Width. The Bias entries in rows 1 to 12 are 0.300, 3.005, 0.298, 0.200, 0.456, 4.561, 0.459, 0.315, 0.598, 5.973, 0.598 and 0.398. The column rho(S(1), S(2)) reads 0.25 in every row except rows 4, 8 and 12, where it reads 0.50. Red outlines surround the Bias entries 3.005 in row 2, 4.561 in row 6, 0.598 in row 9, 5.973 in row 10 and 0.398 in row 12.", "caption": "Table 1, p. 582 (PDF p. 11). The outlines mark the five Bias entries that differ from the exact values of the stated designs, in rows 2, 6, 9, 10 and 12.", "url": "https://pmaronow.github.io/llm-audit/works/P010/exhibits/P010-N014.png"}], "formal_release": null, "formal_statements": [], "headline": "The pairs $(S(1),S(2))$ of the simulations were drawn with a sampler that, in version 0.9-19, attains the stated correlation only approximately; if this version was used, this can account for the five Bias entries of Table 1 that differ from the exact values of the stated designs by up to 0.5 percent.", "id": "P010-N014", "location": "Table 1, column “Bias”, rows 2, 6, 9, 10 and 12, p. 582 (PDF p. 11); replication files, function do_sims(), line 165 of abp_replication_functions.R", "pdf_page": 11, "printed_page": "582", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table~1, p.~582.} Column $\\rho(S(1),S(2))$, rows 1 to 12: 0.25, 0.25, 0.25, 0.50, 0.25, 0.25, 0.25, 0.50, 0.25, 0.25, 0.25, 0.50. Column Bias, rows 1 to 12: 0.300, \\hl{3.005}, 0.298, 0.200, 0.456, \\hl{4.561}, 0.459, 0.315, \\hl{0.598}, \\hl{5.973}, 0.598, \\hl{0.398}.\n\n\\emph{Replication files, \\texttt{abp\\_replication\\_functions.R}, line 165 (not printed in the article).}\n\n$\\texttt{tv <- }\\hl{\\texttt{rmvbin}}\\texttt{(n, c(PrS0, PrS1), bincorr = (1 - rho) * diag(2) + rho)}$", "corrected": "\\emph{Appendix~C, to be added to the description of the simulations.} \\hl{The pairs $(S(1),S(2))$ were drawn with the function \\texttt{rmvbin} of the R package \\texttt{bindata}. Version 0.9-19 of the package attains the requested correlation only approximately. If this version was used, the correlations attained are 0.249 and 0.498 at passing rates 0.8 and 0.8, 0.252 and 0.498 at passing rates 0.6 and 0.8, and 0.253 and 0.503 at passing rates 0.6 and 0.6, where Table~1 states 0.25 and 0.50.} The version of the package that was used is to be stated there.\n\nAlternatively, the pairs are drawn from the exact joint probabilities $1-p-q+t$, $p-t$, $q-t$ and $t$ of $(S(1),S(2))=(0,0)$, $(1,0)$, $(0,1)$ and $(1,1)$, and Table~1 is recomputed and reported with Monte Carlo standard errors.", "evidence": "The evidence is a derivation of the exact bias, a reading of the source of the package in the version published on CRAN in November 2012, and a numerical integration of the distribution that this version samples. Let $p$ and $q$ be the two passing rates and $t=pq+\\rho\\sqrt{p(1-p)q(1-q)}$ the probability of passing under both treatments. Among the subjects who pass under treatment~2 the mean outcome is $\\lambda(q-t)/q$, and among those who pass under treatment~1 it is $-\\lambda(p-t)/p$. Subtracting, I obtain the expected difference in means after dropping,\n\\[\n\\lambda\\Bigl(2-t\\Bigl(\\frac1p+\\frac1q\\Bigr)\\Bigr),\n\\]\nwhich equals $2\\lambda(1-p)(1-\\rho)$ when $p=q$. Replacing $t$ by the joint probability that the sampler attains, I obtain the expectation under the sampler.\n\\begin{audittable}{rrrrr}{The five Bias entries of Table~1 that differ from the exact value of the stated design. The Monte Carlo standard error is a first-order value for $10^8$ simulated subjects per row.}\nRow & Printed & Exact, stated design & Expected under the sampler & Monte Carlo standard error \\\\\n2 & 3.005 & 3.0000 & 3.0050 & 0.0008 \\\\\n6 & 4.561 & 4.5711 & 4.5595 & 0.0010 \\\\\n9 & 0.598 & 0.6000 & 0.5975 & 0.0003 \\\\\n10 & 5.973 & 6.0000 & 5.9754 & 0.0012 \\\\\n12 & 0.398 & 0.4000 & 0.3979 & 0.0003 \\\\\n\\end{audittable}\nThe printed entry of row 10 is 22 standard errors from the exact value and about two from the expectation under the sampler. A new simulation of row 10, with pairs drawn from the exact joint distribution and the bounds computed by the function as deposited, gave an average bias of 6.0013, with standard error 0.0012. The Bound Width entry of row 10, 19.712, is also below the average width 19.803 of that simulation, whose standard error is 0.014. The sampler is not shown to account for this gap. That entry is in any case subject to \\fid{P010-E016}.", "problem": "The Bias column reports the average, over the simulations, of the difference in means after dropping the subjects who fail. For the designs stated in the table, this average has an expectation in closed form. The entries of rows 2, 6, 9, 10 and 12 lie between 5 and 22 Monte Carlo standard errors from it. The gaps are small, at most 0.5 percent, but they are many times the simulation error. The cause that the sources support is the sampler. The code draws the pairs $(S(1),S(2))$ with the function \\texttt{rmvbin} of the R package \\texttt{bindata}. In version 0.9-19 of the package, the requested joint probability is converted into the correlation of a pair of normal variables by linear interpolation in a stored table of simulated values. The normal pair is then thresholded. The binary correlation that results differs from the requested one. At passing rates 0.6 and 0.6 and requested correlation 0.25 it is 0.2531, and the probability of passing under both treatments is 0.42074 where the design requires 0.42. If this version was used, the designs that were simulated are not exactly the designs in the head of the table. The replication files name the package without its version, and the data sets of the original run are not available. The attribution rests on the assumption that this version was used.", "what_survives": "Every comparison that the paper draws from the Bias column is unchanged when the entries are replaced by the exact values. Examples are 0.3 against 3 in rows 1 and 2, 0.4571 against 4.5711 in rows 5 and 6, and 0.6 against 6 in rows 9 and 10. The mechanism of the bias, the warning about dropping subjects and the application do not depend on these decimals. The correction needs no new assumption."}, "url": "https://pmaronow.github.io/llm-audit/works/P010/#P010-N014", "work": {"arxiv_id": null, "doi": "10.1017/pan.2019.5", "title": "A Note on Dropping Experimental Subjects who Fail a Manipulation Check", "url": "https://pmaronow.github.io/llm-audit/works/P010/"}, "work_id": "P010"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the opening of Section 3.1. It shows the definition of the plug-in sample variance and display (3.1) for the estimator V1-hat of A and Z. Red boxes mark the sentence 'The corresponding oracle estimator V1-hat(A^O; Z) is consistent.', the line 'Proposition 1. Under Assumption 2, for any epsilon > 0,' and the display stating that the limit of Pr(|n V1-hat(A^O; Z) - n var(beta-hat)| > epsilon) is 0.", "caption": "Section 3.1, p. 2242 (PDF p. 5): the estimator (3.1), the sentence that calls the oracle estimator consistent, and Proposition 1. The marked statements assume Assumption 2 only.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E001.png"}, {"alt": "Crop of the first four lines of page 2243. A red box marks the sentence 'Asymptotic unbiasedness follows directly from linearity of expectations and var(beta-hat) = O(n^-1).', which runs over two lines.", "caption": "Proof of Proposition 1, p. 2243 (PDF p. 6), first paragraph. The marked sentence is the step that omits the products of the deviations of the unit means.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E001-b.png"}, {"alt": "Crop of three paragraphs. A red box marks the statement 'Corollary 1. Given Assumption 2, the limit as n tends to infinity of Pr(n var(beta-hat) - n V1-hat(A^m; Z) > epsilon) = 0 for any epsilon > 0.' The paragraph before it says that as n grows large the variance estimate V1-hat(A^m, Z) is conservative.", "caption": "Corollary 1, p. 2245 (PDF p. 8), with the paragraphs before and after it.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E001-c.png"}, {"alt": "Crop of Assumption 3 (Homoskedasticity) and the four lines after it. A red box marks the sentence 'Under homoskedasticity, the general estimator V1-hat(A^m, Z) developed in Section 3.1 provides a conservative variance estimate.'", "caption": "Assumption 3 and the first sentence after it, p. 2246 (PDF p. 9). The marked sentence claims that the general estimator is conservative under homoskedasticity.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E001-d.png"}, {"alt": "Crop of a three-line paragraph and Corollaries 3 and 4. A red box marks 'Corollary 3. Given Assumption 2, confidence intervals formed as beta-hat plus or minus z times the square root of V1-hat(A^m) have at least 100(1 - alpha) percent coverage for beta in large n.' Corollary 4, for V2-hat and V2-hat-prime under Assumptions 2 and 3, is not marked.", "caption": "Corollaries 3 and 4, p. 2248 (PDF p. 11). Corollary 3, marked, is the coverage claim for the general estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E001-e.png"}, {"alt": "Two line charts with the same horizontal axis, the value a at 0, 1, the square root of 3 and 2.5. Left panel, mean of n times the variance estimate: orange circles for the published estimator at 4.00, 2.99, 1.00 and minus 2.26, lying on the curve 4 minus a squared; a dashed line at the true value 4; blue squares for the degree-corrected estimator near 6; green triangles for the positive-part estimator near 4. Right panel, coverage of the nominal 95 percent interval: the published estimator at 0.951, 0.904, 0.671 and 0.000; the two corrected estimators between 0.94 and 0.99, close to the dashed nominal line at 0.95.", "caption": "Simulation of the cycle example with a known graph, $n=2000$, 4000 replications for each value of $a$. Left: the mean of $n\\widehat V$ for the published estimator (3.1) follows its limit $4-a^2$ and leaves the true value 4, while the degree-corrected estimator $\\widehat V_1^{+}$ stays near 6 and the positive-part estimator $\\widehat V_{+}$ near 4. Right: the coverage of the nominal 95% interval built on the published estimator falls from 0.951 at $a=0$ to 0.904, 0.671 and 0.000; the two corrected estimators cover at 0.94 or more.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E001-unequal-means.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T1", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T1"}, {"id": "P011-T2", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T2"}, {"id": "P011-T3", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T3"}, {"id": "P011-T4", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T4"}, {"id": "P011-T5", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T5"}, {"id": "P011-T6", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T6"}, {"id": "P011-T7", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T7"}, {"id": "P011-T8", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T8"}, {"id": "P011-T9", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T9"}, {"id": "P011-T10", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T10"}, {"id": "P011-T16", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T16"}, {"id": "P011-T29", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T29"}, {"id": "P011-T31", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T31"}, {"id": "P011-T32", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T32"}, {"id": "P011-T33", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T33"}, {"id": "P011-T41", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T41"}], "headline": "Under Assumption 2 the oracle estimator satisfies $n\\widehat V_1(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)=n^{-1}\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta+o_p(1)$ with $\\delta_i=\\mathbb{E}[\\theta_iY_i]-\\beta$, and the term on the right can be negative; Proposition 1, Corollaries 1 and 3 and the sentence after Assumption 3 therefore hold only under a further condition on the unit means, on the graph, or on the known edges and remaining degree capacity.", "id": "P011-E001", "location": "Section 3.1, p. 2242 (PDF p. 5): equation (3.1), the sentence after it and Proposition 1. Proof of Proposition 1, p. 2243 (PDF p. 6). Corollary 1, p. 2245 (PDF p. 8). First sentence after Assumption 3, p. 2246 (PDF p. 9). Corollary 3, p. 2248 (PDF p. 11). The abstract, p. 2238, states the same claims.", "pdf_page": 5, "printed_page": "2242", "reaches": [{"effect": "result_needs_same_repair", "effect_text": "a result or code path there needs the same repair", "explanation": null, "package": "depinf", "recorded_as": "P011-N006", "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P011-N006", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-N006", "work_id": "P011"}], "severity": "Critical", "subject": "paper", "target_ids": ["P011-T1", "P011-T10", "P011-T16", "P011-T2", "P011-T29", "P011-T3", "P011-T31", "P011-T32", "P011-T33", "P011-T4", "P011-T41", "P011-T5", "P011-T6", "P011-T7", "P011-T8", "P011-T9"], "text": {"as_printed": "P.~2242: ``The corresponding oracle estimator $\\widehat V_1(A^{\\mathcal O};Z)$ is \\hl{consistent}. Proposition 1. \\hl{Under Assumption 2}, for any $\\epsilon>0$, $\\lim_{n\\to\\infty}\\Pr(|n\\widehat V_1(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)|>\\epsilon)=0$.''\n\nP.~2243, in the proof: ``\\hl{Asymptotic unbiasedness follows directly} from linearity of expectations and $\\operatorname{var}(\\hat\\beta)=O(n^{-1})$.''\n\nP.~2245: ``Corollary 1. \\hl{Given Assumption 2}, $\\lim_{n\\to\\infty}\\Pr(n\\operatorname{var}(\\hat\\beta)-n\\widehat V_1(A^m;Z)>\\epsilon)=0$ for any $\\epsilon>0$.''\n\nP.~2246: ``\\hl{Under homoskedasticity}, the general estimator $\\widehat V_1(A^m,Z)$ developed in Section 3.1 provides a conservative variance estimate.''\n\nP.~2248: ``Corollary 3. \\hl{Given Assumption 2}, confidence intervals formed as $\\hat\\beta\\pm z_{1-\\alpha/2}\\sqrt{\\widehat V_1(A^m)}$ have at least $100(1-\\alpha)\\%$ coverage for $\\beta$ in large $n$.''", "corrected": "Throughout, $X_i=\\theta_iY_i$, $\\delta_i=\\mathbb{E}[X_i]-\\beta$, $e_i=X_i-\\hat\\beta$, and $d_i'=\\min\\{d_i,n-1\\}$.\n\n\\textbf{Proposition 1.} Under Assumption 2, for any $\\epsilon>0$,\n\\[\n\\lim_{n\\to\\infty}\\Pr\\Bigl(\\bigl|n\\widehat V_1(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)-\\hl{\\tfrac1n\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta}\\bigr|>\\epsilon\\Bigr)=0.\n\\]\nHence the conclusion of Proposition 1 as printed holds \\hl{if and only if $n^{-1}\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta\\to0$}. It holds in particular if $\\mathbb{E}[\\theta_iY_i]=\\beta$ for every $i\\in\\mathcal V_S$. In the proof, the sentence on asymptotic unbiasedness becomes $\\mathbb{E}[n\\widehat V_1(A^{\\mathcal O};Z)]=n\\operatorname{var}(\\hat\\beta)+n^{-1}\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta+O(n^{-1})$.\n\n\\textbf{Corollary 1: a broader sufficient condition.} Retain Assumptions 1--2 and the original point estimator. Let $A_R$ be the known-edge adjacency matrix. Choose fixed valid integer caps\n\\[\n\\deg_{A^{\\mathcal O}}(i)\\le h_i\\le\\min\\{d_i,n-1\\},\n\\qquad \\max_i h_i\\le D<\\infty,\n\\]\nand define\n\\[\nr_i=\\sum_j(A_R)_{ij},\\qquad s_i=h_i-r_i,\\qquad\nK_R=I+A_R+\\operatorname{diag}(s_i).\n\\]\nThe original maximization may retain larger supplied caps. Let $Q\\succeq0$ be fixed with $\\|Q\\|_{\\mathrm{op}}=O(1)$, and write the variance inputs as\n\\[\nv=X-\\mathbb{E}[X]+\\delta+a,\\qquad\n\\|\\delta\\|^2=O(n),\\qquad\n\\frac{\\|a\\|^2}{n}\\to_p0,\\qquad\n\\frac{\\max_i v_i^2}{n}\\to_p0,\n\\]\nwhere $\\delta$ is deterministic. Here $\\delta$ is the centering error for these inputs; for the published inputs $v=e=X-\\hat\\beta\\mathbf1$, it is the original $\\mathbb{E}[X]-\\beta\\mathbf1$. All the other input conditions then follow from Assumption 2. Define\n\\[\n\\widehat V(v,Q)=\\frac1{n^2}\n\\left\\{\\max_{A\\in\\mathcal A(Z)}v^{\\mathsf T}(I+A)v+v^{\\mathsf T}Qv\\right\\}.\n\\]\nIf condition (C) holds,\n\\[\n\\hl{\\liminf_{n\\to\\infty}\\frac{\\delta^{\\mathsf T}(K_R+Q)\\delta}{n}\\ge0},\n\\]\nthen, for every $\\epsilon>0$,\n\\[\n\\lim_{n\\to\\infty}\\Pr\\{n\\operatorname{var}(\\hat\\beta)-n\\widehat V(v,Q)>\\epsilon\\}=0.\n\\]\nFor $v=e$ and $Q=0$ this is the original maximized estimator. Condition (C) concerns only the relevant mean directions: if $\\delta\\in\\operatorname{col}(U)$ and $U^{\\mathsf T}U=I$, it suffices that\n\\[\n\\lambda_{\\min}\\{U^{\\mathsf T}(K_R+Q)U\\}\\ge-\\varepsilon_n,\n\\qquad \\varepsilon_n\\to0.\n\\]\nThe negative eigenvalues of the full matrix need not vanish.\n\nFor $Q=0$, each of the following ensures (C). The first two use published centering, for which the mean deviations are uniformly bounded.\n\\begin{enumerate}\n\\item There are $o(n)$ known edges. This includes degree-only constraints, with no known edges.\n\\item Deleting $o(n)$ known edges leaves vertex-disjoint clique components. In particular, it suffices that only $o(n)$ known edges belong to components that are not cliques. The retained clique edges may number $O(n)$ and their caps need not be saturated. Exact disjoint cliques are a special case; overlapping cliques alone do not suffice.\n\\item For every vertex, $h_i\\ge2r_i-1$. Then\n\\[\nK_R=A_R+\\operatorname{diag}(r_i)\n+\\operatorname{diag}(1+h_i-2r_i)\\succeq0.\n\\]\n\\item The mean deviations lie in a specified space on which $K_R$ is positive semidefinite, or its smallest restricted eigenvalue tends to zero from below, as in the spectral condition above. Common transformed means, $\\mathbb{E}[X_i]=\\beta$, give $\\delta=0$.\n\\end{enumerate}\nThere is also the direct sufficient condition $\\liminf n^{-1}\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta\\ge0$, from the corrected Proposition 1. A true graph consisting of disjoint cliques satisfies this condition for every mean vector. Neither this condition nor (C) requires the mean term to tend to zero, so neither establishes the published oracle's consistency.\n\n\\textbf{Two fixed corrections after maximization.} For any mean vector satisfying the input conditions, (C) holds with\n\\[\nQ=(-K_R)_+,\n\\]\nthe negative spectral part of $K_R$ with its signs reversed, since $K_R+Q\\succeq0$. Alternatively, let $F$ be a fixed set of known edges whose deletion leaves vertex-disjoint cliques, and take its Laplacian $Q=L_F$. The added term is\n\\[\n\\frac1{n^2}v^{\\mathsf T}L_Fv\n=\\frac1{n^2}\\sum_{\\{i,j\\}\\in F}(v_i-v_j)^2.\n\\]\nThere is no requirement that $|F|=o(n)$: if $A_{R\\setminus F}$ is the remaining adjacency matrix,\n\\[\nK_R+L_F=I+A_{R\\setminus F}\n+\\operatorname{diag}\\{s_i+\\deg_F(i)\\}\\succeq0.\n\\]\nBoth choices have bounded operator norm under the stated caps. Each is added after maximization, so it preserves the maximizing graph; either may widen the variance bound.\n\nThe sufficient condition follows from a compatible-graph construction and a variance calculation. Proper-color the known graph, sort the inputs within each color and remaining-capacity class, and join consecutive groups of $s_i+1$ vertices into cliques. The new edges respect all caps and do not duplicate known edges. The loss relative to $v^{\\mathsf T}K_Rv$ is at most a constant depending on $D$ times $\\max_i v_i^2$, which gives\n\\[\nn\\widehat V(v,Q)\\ge\\frac1n v^{\\mathsf T}(K_R+Q)v-o_p(1).\n\\]\nPut $u=X-\\mathbb{E}[X]$, $\\Sigma=\\operatorname{cov}(X)$ and $B=I+A^{\\mathcal O}$. If $A_U=A^{\\mathcal O}-A_R$ has degrees $t_i$, then\n\\[\nK_R-B=L_U+\\operatorname{diag}(s_i-t_i)\\succeq0,\n\\qquad\n\\frac{\\operatorname{tr}(B\\Sigma)}n=n\\operatorname{var}(\\hat\\beta).\n\\]\nBounded-degree dependence gives $n^{-1}u^{\\mathsf T}K_Ru=\\operatorname{tr}(K_R\\Sigma)/n+o_p(1)$ and $\\|\\Sigma\\|_{\\mathrm{op}}=O(1)$. The deterministic mean cross term is $o_p(1)$ after division by $n$, and the input remainder $a$ changes the quadratic form by $o_p(1)$ on that scale. The possibly dense correction needs no local concentration argument: $u^{\\mathsf T}Qu\\ge0$. Hence\n\\[\nn\\widehat V(v,Q)\\ge n\\operatorname{var}(\\hat\\beta)\n+\\frac1n\\delta^{\\mathsf T}(K_R+Q)\\delta+o_p(1),\n\\]\nwhich proves conservativeness under (C). The first two graph routes follow by bounding the adverse mean cross products by a constant times the number of exceptional known edges. The covariance and mean terms are separated in the same manner as in \\href{https://arxiv.org/abs/2603.11497}{Yap's Theorem 2}; the graph construction and probability argument above are needed for this estimator.\n\n\\textbf{Corollary 3.} Given Assumptions 1--2 and the input conditions and condition (C) above, the intervals\n\\[\n\\hat\\beta\\pm z_{1-\\alpha/2}\\sqrt{\\widehat V(v,Q)}\n\\]\nhave at least $100(1-\\alpha)\\%$ coverage for $\\beta$ in large $n$. The interval is taken to be empty when its variance estimate is negative; this event has probability tending to zero. The conclusion follows from Lemma 2 and the conservativeness bound, since $n\\operatorname{var}(\\hat\\beta)$ tends to a positive constant. It does not require consistency of the maximum or of the original oracle. The same coverage conclusion holds for the original maximized estimator under the direct mean-term condition stated above.\n\n\\textbf{Section 3.2, sentence after Assumption 3.} ``\\hl{Under condition (C) with published centering and $Q=0$}, the general estimator $\\widehat V_1(A^m,Z)$ developed in Section 3.1 provides a conservative variance estimate.'' The direct mean-term condition above is another sufficient qualification. Assumption 3 alone does not imply either one.\n\n\\textbf{Estimators that are conservative for any unit means.} For a compatible $A$ with degrees $d_i^A=\\sum_jA_{ij}$, define\n\\[\n\\widehat V_1^{+}(A;Z)=\\widehat V_1(A;Z)+\\frac1{n^2}\\sum_{i\\in\\mathcal V_S}\\Bigl(1-\\frac1{d_i'}\\Bigr)d_i^A\\,e_i^2,\n\\qquad\n\\widehat V_Y'(Z)=\\frac1{n^2}\\sum_{i\\in\\mathcal V_S}(1+d_i')\\,e_i^2,\n\\]\nwhere the summand of the first sum is zero when $d_i'=0$. Let $A^{+}$ maximize $\\widehat V_1^{+}(A;Z)$ over $\\mathcal A(Z)$. Given Assumption 2, for any unit means and any $\\epsilon>0$,\n\\[\n\\lim_{n\\to\\infty}\\Pr\\bigl(n\\operatorname{var}(\\hat\\beta)-n\\widehat V>\\epsilon\\bigr)=0\n\\]\nfor $\\widehat V$ equal to $\\widehat V_1^{+}(A^{\\mathcal O};Z)$, $\\widehat V_1^{+}(A^{+};Z)$, $\\widehat V_Y'(Z)$, or the smaller of the last two. When the graph is known, the same holds for $\\widehat V_{+}(Z)=n^{-2}e^{\\mathsf T}(I+A^{\\mathcal O})_{+}e$, where $(I+A^{\\mathcal O})_{+}$ is obtained from $I+A^{\\mathcal O}$ by setting its negative eigenvalues to zero. These estimators are nonnegative, and by Proposition 3 their Wald intervals have at least $100(1-\\alpha)\\%$ coverage in large $n$.\n\nThese statements hold for the following reason. Let $C$ be the covariance matrix of $(X_1,\\ldots,X_n)$, and let $M$ be a deterministic symmetric matrix whose operator norm is bounded in $n$. The same derivation gives\n\\[\n\\frac1n e^{\\mathsf T}Me-n\\operatorname{var}(\\hat\\beta)=\\frac1n\\Bigl[\\operatorname{tr}\\bigl((M-I-A^{\\mathcal O})C\\bigr)+\\delta^{\\mathsf T}M\\delta\\Bigr]+o_p(1).\n\\]\nBoth terms in the bracket are nonnegative when $M$ and $M-I-A^{\\mathcal O}$ are positive semidefinite. For $\\widehat V_1^{+}(A^{\\mathcal O};Z)$, the matrix $M$ is the sum of $A^{\\mathcal O}+\\operatorname{diag}(d_i^{A^{\\mathcal O}})$, whose quadratic form is $\\sum_{i<j}A^{\\mathcal O}_{ij}(x_i+x_j)^2$, and of the nonnegative diagonal matrix $\\operatorname{diag}(1-d_i^{A^{\\mathcal O}}/d_i')$, where the ratio is zero when $d_i'=0$. The matrix $M-I-A^{\\mathcal O}$ is then a nonnegative diagonal matrix. For $\\widehat V_Y'$, since $d_i'\\ge d_i^{A^{\\mathcal O}}$, I have $\\widehat V_Y'\\ge n^{-2}e^{\\mathsf T}(I+\\operatorname{diag}(d_i^{A^{\\mathcal O}}))e$ in every sample, and for $M=I+\\operatorname{diag}(d_i^{A^{\\mathcal O}})$ the matrix $M-I-A^{\\mathcal O}$ is the Laplacian of the graph, which is positive semidefinite. The maximum over $\\mathcal A(Z)$ is at least the oracle value in every sample, as in the proof of Corollary 1. The display must not be applied to a matrix that is chosen from the data. The no-known-edge case also follows directly from $\\widehat V_Y'$: if no edge is known and the bounds $d_i'$ are at most $D$, then $0\\le n\\widehat V_Y'-n\\widehat V_1(A^m;Z)\\le B^2n^{-1}\\sum_{d=1}^{D}(3d^2+2d)$ in every sample, with $B=\\max_i|e_i|\\le2c_1$. If the bounds $d_i'$ are not bounded in $n$, the same argument applies after each bound is replaced by the smaller of $d_i'$ and the least integer above $c_2$, which can only lower the maximum.\n\nThe estimator $\\widehat V_1^{+}$ is affine in $A$. Program (3.6) therefore applies with the weights $\\hat v_{ij}=e_ie_j+(1-1/d_i')e_i^2/2+(1-1/d_j')e_j^2/2$, with \\hl{$1-1/d_i'$ interpreted as zero when $d_i'=0$}, and $n^2\\widehat V_1^{+}(A^{+};Z)=\\sum_ie_i^2+2w^{*}$, where $w^{*}$ is the optimal value. These estimators are conservative and in general not consistent. In the cycle example with $a=\\sqrt3$ and true value 4, the limits of $n\\widehat V_1^{+}$, $n\\widehat V_Y'$ and $n\\widehat V_{+}$ are 6, 15 and 4.\n\n\\textbf{Regression residuals.} Retain the original point estimator. For nonrandom regressors $f_i$, fit $Y_i$ by $\\theta_i^2$-weighted least squares and use\n\\[\nv_i=\\theta_i(Y_i-f_i^{\\mathsf T}\\hat\\gamma)\n\\]\nonly in the variance estimator. Let $W$ have rows $\\theta_i f_i^{\\mathsf T}$ and suppose $\\operatorname{rank}(W)=o(n)$; a fixed number of regressors suffices. The conditions are met if $\\mathbb{E}[Y_i]=f_i^{\\mathsf T}\\gamma$, or, more generally, if\n\\[\n\\inf_\\gamma\\frac1n\\sum_i\\theta_i^2\n\\{\\mathbb{E}[Y_i]-f_i^{\\mathsf T}\\gamma\\}^2\\longrightarrow0.\n\\]\nTo see the prediction rate, let $P$ be the orthogonal projection onto $\\operatorname{col}(W)$ and $\\mu=\\mathbb{E}[X]$. The transformed fitted values are $PX$, including when coefficients are not unique or some weights are zero. Since $\\|\\Sigma\\|_{\\mathrm{op}}=O(1)$,\n\\[\n\\begin{aligned}\n\\frac1n\\mathbb{E}\\|PX-\\mu\\|^2\n&=\\frac{\\operatorname{tr}(P\\Sigma)+\\|(I-P)\\mu\\|^2}{n}\\\\\n&\\le\\frac{\\|\\Sigma\\|_{\\mathrm{op}}\\operatorname{rank}(W)}n\n+\\inf_\\gamma\\frac1n\\sum_i\\theta_i^2\n\\{\\mathbb{E}[Y_i]-f_i^{\\mathsf T}\\gamma\\}^2\n\\longrightarrow0.\n\\end{aligned}\n\\]\nThus $v=u+\\delta+a$ with $\\delta=(I-P)\\mu$, $a=-Pu$, $\\|\\delta\\|^2/n\\to0$ and $\\|a\\|^2/n\\to_p0$. These bounds also imply $\\max_i v_i^2/n\\to_p0$ and (C). Under exact specification $\\delta=0$. No separate coefficient-convergence rate or uniformly bounded-regressor assumption is required.\n\nMore generally, any predictions $\\hat m_i$ satisfying\n\\[\n\\frac1n\\sum_i\\theta_i^2\\{\\hat m_i-\\mathbb{E}[Y_i]\\}^2\\to_p0\n\\]\ncan replace the weighted least-squares fit. In all these regression cases, the \\hl{uncorrected true-graph residual oracle} is consistent:\n\\[\n\\frac1n v^{\\mathsf T}(I+A^{\\mathcal O})v\n-n\\operatorname{var}(\\hat\\beta)\\to_p0.\n\\]\nIndeed, $\\|v-u\\|^2/n\\to_p0$ and $\\|I+A^{\\mathcal O}\\|_{\\mathrm{op}}=O(1)$, so replacing $u$ by $v$ changes the scaled quadratic form by $o_p(1)$. The maximum over compatible graphs is conservative and need not be consistent. Adding $Q$ can likewise introduce further conservativeness. Correct specification alone does not justify arbitrary unweighted fitting under the paper's assumptions.\n\n\\textbf{Abstract.} ``The oracle variance estimator has an asymptotic mean term. \\hl{It is consistent, at the scale of $n$ times the variance, if and only if that term vanishes, and it can be anticonservative otherwise. The maximized estimator has asymptotically conservative coverage under the additional sufficient conditions stated above; fixed positive semidefinite corrections and suitable regression residuals provide further repairs.}'' The coverage guarantees of Corollary 4 are unaffected.", "evidence": "I use a derivation of the displayed identity and exact counterexamples to the printed statements. For the identity, let $W_i=X_i-\\beta$, $D=\\hat\\beta-\\beta$ and $B=I+A^{\\mathcal O}$. Then\n\\[\nn\\widehat V_1(A^{\\mathcal O};Z)=\\frac1n W^{\\mathsf T}BW-2D\\,\\frac{\\mathbf{1}^{\\mathsf T}BW}{n}+D^2\\,\\frac{\\mathbf{1}^{\\mathsf T}B\\mathbf{1}}{n}.\n\\]\nSince $\\operatorname{cov}(X_i,X_j)=0$ unless $i=j$ or $A^{\\mathcal O}_{ij}=1$, I have $\\mathbb{E}[n^{-1}W^{\\mathsf T}BW]=n\\operatorname{var}(\\hat\\beta)+n^{-1}\\delta^{\\mathsf T}B\\delta$. Each variable $W_i\\sum_jB_{ij}W_j$ is bounded by $(1+c_2)c_1^2$ and depends only on the outcomes of $i$ and its neighbours. Two such variables are independent when their vertices are at graph distance greater than three. Counting the pairs that may be dependent, I obtain $\\operatorname{var}(n^{-1}W^{\\mathsf T}BW)\\le(1+c_2)^5c_1^4/n$. Further, $\\mathbb{E}[D^2]\\le(1+c_2)c_1^2/n$, and the last two terms of the display are at most $2(1+c_2)c_1|D|+(1+c_2)D^2$ in absolute value. Chebyshev's inequality then implies that the identity holds.\n\nThe first example consists of independent blocks of three units $a$, $b$, $c$ on a path. Let $U$ and $V$ be independent fair signs, let $\\theta_i=1$, and set\n\\[\n(Y_a,Y_b,Y_c)=\\Bigl(1+U,\\ -2+\\tfrac35U+\\tfrac45V,\\ 1+V\\Bigr).\n\\]\nSince the outcomes of $a$ and $c$ are independent, the path is a dependency graph. Every variance equals 1, the unit means are $(1,-2,1)$, and $\\beta=0$. All edges are known and the degrees are $(1,2,1)$, which implies that $A^{\\mathcal O}$ is the only compatible matrix. Assumptions 1, 2 and 3 hold. By an exact enumeration, I have $n\\operatorname{var}(\\hat\\beta)=29/15$ and $\\mathbb{E}[n\\widehat V_1(A^m;Z)]=19/15-217/(45n)$, and $n\\widehat V_1(A^m;Z)$ tends to $19/15$ in probability. For every $\\epsilon<2/3$, the probability in Corollary 1 tends to one. The coverage of the nominal 95\\% interval tends to $2\\Phi(z_{0.975}\\sqrt{19/29})-1=0.887$. This example is a correctly specified regression with covariate $(1,-2,1)$, intercept 0 and slope 1, in which $\\hat\\beta$ is the fitted intercept. With the unit means doubled, $n\\widehat V_1(A^m;Z)$ tends to $-11/15$, and the estimate is negative with probability tending to one.\n\nThe second example consists of independent blocks of four units on a cycle. Each edge carries an independent fair sign, $Y_i$ is $as_i$ plus the two signs on the edges at $i$, with $s=(1,-1,1,-1)$, and $\\theta_i=1$. The covariance matrix of a block is $2I+A$, every edge is known and every degree is 2. Then $n\\operatorname{var}(\\hat\\beta)=4$ and $n\\widehat V_1(A^{\\mathcal O};Z)$ tends to $4-a^2$. For $a=3/2$ the limit is $7/4$, the exact mean is $7/4-12/n$, and the coverage of the nominal 95\\% interval tends to about $0.805$.\n\n\\begin{audittable}{llll}{Exact examples that satisfy Assumptions 1, 2 and 3. Every edge is known and every degree is attained, so that $A^m=A^{\\mathcal O}$. The last column is the limiting coverage of the nominal 95\\% interval; in the second row the estimate is negative with probability tending to one.}\nExample & $n\\operatorname{var}(\\hat\\beta)$ & Limit of $n\\widehat V_1(A^m;Z)$ & Coverage \\\\\nPaths of three, unit means $(1,-2,1)$ & $29/15$ & $19/15$ & $0.887$ \\\\\nPaths of three, unit means $(2,-4,2)$ & $29/15$ & $-11/15$ & -- \\\\\nCycles of four, unit means $\\pm3/2$ & $4$ & $7/4$ & $0.805$ \\\\\n\\end{audittable}\n\nSimulations of the second example, which I call example A, agree with these limits. The figure shows them for four values of $a$. The next table adds example B, a difference in means on the same cycles. The units of a cycle alternate between treatment and control, $Y_i$ is $\\mu$ plus the two edge signs plus an independent unit sign, and $\\theta_i=\\pm2$. In this example, $\\beta=0$, $n\\operatorname{var}(\\hat\\beta)=4$ and the unit means are $\\pm2\\mu$. The estimators $\\widehat V_1^{+}$, $\\widehat V_Y'$ and $\\widehat V_{+}$ are defined in the corrected statements below. Residual centering replaces $e_i$ by $\\theta_i$ times the deviation of $Y_i$ from the mean of its arm.\n\n\\begin{audittable}{lrrrr}{Known graph, $n=2000$, 4000 replications. Example A with $a=\\sqrt3$ and example B with $\\mu=1$; the true value $n\\operatorname{var}(\\hat\\beta)$ is 4 in both. Mean of $n\\widehat V$ and coverage of the nominal 95\\% interval; a nonpositive variance estimate counts as a miss. Monte Carlo standard errors of the coverages are at most $0.008$.}\nEstimator & A: $n\\widehat V$ & A: coverage & B: $n\\widehat V$ & B: coverage \\\\\n$\\widehat V_1(A^{\\mathcal O})$, as published & 1.00 & 0.671 & 0.00 & 0.222 \\\\\n$\\widehat V_1^{+}(A^{\\mathcal O})$ & 5.99 & 0.984 & 16.00 & 1.000 \\\\\n$\\widehat V_Y'$ & 15.00 & 1.000 & 47.99 & 1.000 \\\\\n$\\widehat V_{+}$ & 4.00 & 0.954 & 9.00 & 0.997 \\\\\nResidual centering & -- & -- & 4.01 & 0.950 \\\\\n\\end{audittable}\n\nThe last table concerns the unknown graph. The maximization over compatible graphs is solved by integer programming, with $n=120$, degree bounds $d_i=2$ and the two examples of the previous table. With no known edge the published estimator is conservative. It fails when the known edges fix most of the graph.\n\n\\begin{audittable}{llrrrr}{Unknown graph, $n=120$, $d_i=2$, 1000 replications, true value 4; examples A ($a=\\sqrt3$) and B ($\\mu=1$). The published maximal estimator $\\widehat V_1(A^m)$ beside $\\widehat V_{\\min}$, the smaller of $\\widehat V_1^{+}(A^{+})$ and $\\widehat V_Y'$. The 108 edges were drawn at random once. Coverage is computed as in the previous table. Monte Carlo standard errors of the coverages are at most $0.016$.}\nExample & Known edges & $n\\widehat V_1(A^m)$ & Coverage & $n\\widehat V_{\\min}$ & Coverage \\\\\nA & none & 14.90 & 1.000 & 14.90 & 1.000 \\\\\nA & 27 of the 30 cycles & 2.20 & 0.826 & 7.16 & 0.994 \\\\\nA & 108 of the 120 edges & 2.25 & 0.835 & 7.21 & 0.994 \\\\\nA & all & 0.91 & 0.587 & 5.87 & 0.977 \\\\\nB & none & 47.76 & 1.000 & 47.77 & 1.000 \\\\\nB & 27 of the 30 cycles & 4.22 & 0.842 & 20.14 & 1.000 \\\\\nB & 108 of the 120 edges & 4.32 & 0.885 & 20.24 & 1.000 \\\\\nB & all & $-0.15$ & 0.333 & 15.77 & 1.000 \\\\\n\\end{audittable}\n\nThe simulations illustrate the limits and are not part of the argument. The term $n^{-1}\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta$ can also be zero or positive, and unequal unit means do not always make the published estimator anticonservative.", "problem": "Definition 3 requires that the average of the unit means $\\mathbb{E}[\\theta_iY_i]$ equal $\\beta$. It does not require that each unit mean equal $\\beta$. In the leading example of the paper, an OLS coefficient with $\\mathbb{E}[Y_i]=x_i^{\\mathsf T}\\beta$, the unit means can differ. The estimator (3.1) centres every $\\theta_iY_i$ at the single number $\\hat\\beta$. Let $X_i=\\theta_iY_i$ and $\\delta_i=\\mathbb{E}[X_i]-\\beta$. Then $\\mathbb{E}[(X_i-\\beta)(X_j-\\beta)]=\\operatorname{cov}(X_i,X_j)+\\delta_i\\delta_j$, and the products $\\delta_i\\delta_j$ need not average out. Under Assumption 2,\n\\[\nn\\widehat V_1(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)=\\frac1n\\,\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta+o_p(1).\n\\]\nThe step of the proof that asserts asymptotic unbiasedness omits the term on the right. The same omission affects $\\hat\\sigma_\\theta^2$, whose asymptotic approximation includes $n^{-1}\\sum_i\\delta_i^2$.\n\nNote that the term on the right is deterministic. It need not tend to zero, and Proposition 1 fails. Since the matrix $I+A^{\\mathcal O}$ is positive semidefinite only when the graph $\\mathcal G_S$ is a disjoint union of cliques, the term can be negative. The oracle estimator then underestimates the variance. If the known edges and the degrees determine the graph, then $A^m=A^{\\mathcal O}$, and the maximal compatible estimator underestimates the variance as well. Corollaries 1 and 3 fail in that case. Assumption 3 equalizes the variances of the $\\theta_iY_i$ and not their means. The examples below are homoskedastic, which shows that the sentence on p.~2246 fails too. The abstract states the consistency of the oracle estimator and conservative coverage without the condition.\n\nThe condition $n^{-1}\\delta^{\\mathsf T}(I+A^{\\mathcal O})\\delta\\to0$ is necessary and sufficient for Proposition 1 under Assumption 2. It restricts the unit means, and a correctly specified OLS model does not imply it. No estimator is consistent, at the scale of $n$ times the variance, for every bounded configuration of unit means. Indeed, let $a,b>0$ and $X_i=as_i+b\\epsilon_i$ with independent fair signs $\\epsilon_i$. For each fixed infinite sequence of signs $s_i$, I have $n\\operatorname{var}(\\hat\\beta)=b^2$. If the $s_i$ are themselves independent fair signs, then $n\\operatorname{var}(\\hat\\beta)=a^2+b^2$. An estimator that is consistent under every fixed infinite sequence tends to $b^2$ in the second model as well, where the target is $a^2+b^2$. Proposition 1 holds exactly when the term displayed above tends to zero. Conservativeness of the maximized estimator requires less: an additional condition on the known edges, remaining degree capacity and relevant mean directions supplies further sufficient cases without changing the estimator. This condition does not imply consistency of the original oracle. Fixed positive semidefinite corrections or suitable regression residuals provide other repairs; the resulting maximal bounds are conservative and need not be consistent.", "what_survives": "The following are unaffected.\n\\begin{itemize}\n\\item The definitions, and the inequality $\\widehat V(A^m;Z)\\ge\\widehat V(A^{\\mathcal O};Z)$ in every sample.\n\\item The programs (3.5), (3.6) and (3.8) as optimization problems.\n\\item The bound of order $1/n$ on the variance of the estimator in the proof of Proposition 1, after the corrections to display (3.3) in \\fid{P011-E003}.\n\\item The central limit theorem of Lemma 2.\n\\item Lemma 3 and Proposition 3, which are conditional on a property of the variance estimator.\n\\end{itemize}\n\nThe conclusions of Proposition 2, Lemma 1 and Corollaries 2 and 4 for $\\widehat V_2$ and $\\widehat V_2'$ hold with unequal unit means, by a direct argument, although the printed proof of Proposition 2 does not (see \\fid{P011-E004}).\n\nThe general guarantees need qualification when unit means differ and edges are known: Proposition 1, Corollaries 1 and 3, the sentence on p.~2246, and the corresponding claims of the abstract, the introduction and the discussion. The corrected conservativeness condition below retains the original maximized estimator in additional graph and mean configurations, including sparse known edges and known graphs close to disjoint cliques. It does not remove the need for an additional condition, and homoskedasticity alone does not supply one. The heteroskedastic branch of the package computes the same estimator, and its vignette applies it to OLS coefficients with known recruitment edges (see \\fid{P011-N006}). The article reports no simulation or application results, so no empirical number changes. I have not established that any interval in the vignette undercovers."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E001", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": {"class": "error", "severity": "Moderate"}, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of eleven lines of text. A red box marks the words 'is NP-hard' in the sentence 'This program (3.5) is NP-hard, but it admits a polynomial time approximation scheme (PTAS).' The lines above say that in general the program is a multidimensional knapsack problem.", "caption": "P. 2245 (PDF p. 8), the paragraphs after program (3.6). The marked words classify program (3.5) as NP-hard.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T13", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T13"}, {"id": "P011-T14", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T14"}, {"id": "P011-T30", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T30"}], "headline": "Program (3.5) is a maximum-weight degree-constrained subgraph problem, which is solved exactly in polynomial time by reduction to weighted matching; hence it is NP-hard only if $\\mathrm{P}=\\mathrm{NP}$.", "id": "P011-E002", "location": "P. 2245 (PDF p. 8), the two paragraphs after program (3.6)", "pdf_page": 8, "printed_page": "2245", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [{"classification_after": {"class": "error", "severity": "Minor"}, "classification_before": {"class": "error", "severity": "Moderate"}, "note": "The classification was lowered from a Moderate to a Minor error, and the wording was revised. How this came to light: review of the remaining Lean complexity statement showed that P011-E002 already covers the words “is NP-hard” on p. 2245. The exact polynomial-time route described in the finding makes NP-hardness of the nontrivial threshold decision problem equivalent to $\\mathrm{P}=\\mathrm{NP}$, which is open. Why it changed: the paper asserted this unresolved classification with unwarranted confidence, and little depends on it. The knapsack encoding, optimization formulations and their optima remain valid, as do the use of general integer-programming solvers and certified relaxation bounds; no statistical result uses NP-hardness. The reduction and the qualification about fractional LP optima are retained. The earlier Moderate rating is preserved as history. This revision was made at P. M. Aronow's decision and was written by the model session that runs the audit."}], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P011-T13", "P011-T14", "P011-T30"], "text": {"as_printed": "``The resulting program has at most $n(n-1)/2$ decision variables and in general it is a multidimensional knapsack problem [12]. See chapter 9 of Kellerer et al 13 for an overview of the multidimensional knapsack problem.\n\nThis program (3.5) \\hl{is NP-hard}, but it admits a polynomial time approximation scheme (PTAS). Typical PTAS depend heavily on the size of the problem and their running time can be very high (see, e.g., section 9.4.2 of Kellerer et al 12).''", "corrected": "``The resulting program has at most $n(n-1)/2$ decision variables. It can be encoded as a special multidimensional knapsack instance. Its degree-incidence structure makes it a maximum-weight simple $b$-matching problem. After fixing the mandatory edges and subtracting their incidences from the degree bounds, it \\hl{admits an exact polynomial-time solution} by reduction to ordinary maximum-weight matching; hence it is NP-hard only if $\\mathrm{P}=\\mathrm{NP}$, which is open. Program (3.8) is its unit-weight case. General integer-programming solvers remain an implementation option; the linear relaxation with degree constraints alone can have fractional optima, and it still provides an upper bound when it is solved to optimality or comes with a certified bound.''", "evidence": "I give an explicit reduction. Let $b_i$ be $\\min\\{d_i,n-1\\}$ minus the number of known edges at $i$, let the optional edges have integer weights $W_e$ after scaling, and let $M=1+\\max(\\{0\\}\\cup\\{|W_e|:e\\text{ optional}\\})$. For each vertex $i$, I create $b_i$ slot vertices. For each optional edge $e=\\{i,j\\}$, I create two vertices joined by an edge of weight $4M$, and I join them to every slot of $i$ and to every slot of $j$, respectively, by edges of weight $2M+W_e$. In a maximum-weight matching, the two vertices of each $e$ are matched either to each other or to slots at both ends. Together with the known edges, the optional edges whose vertices are matched to slots form an optimal solution of (3.6). The weight of the matching is $4Mq+2\\sum W_e$, summed over those optional edges, where $q$ is the number of optional edges. The new graph has at most $2n(n-1)$ vertices. I checked the reduction against exhaustive edge-set enumeration on 13,824 three-vertex inputs and on 96 rational inputs with four or five vertices.\n\nThe same check shows that the linear relaxation with degree constraints alone need not have an integral optimum.\n\n\\begin{audittable}{lrr}{Exact optima of the integer programs and of their linear relaxations in two examples with no known edge and every $d_i=1$: six units with $\\theta_iY_i-\\hat\\beta=(1,1,1,-1,-1,-1)$, and three units.}\nQuantity & Integer program & Linear relaxation \\\\\nObjective of (3.6), six units & 2 & 3 \\\\\nObjective of (3.5), six units & 4 & 6 \\\\\n$\\widehat V_1$ of (3.1) at the optimum, six units & $5/18$ & $1/3$ \\\\\nObjective of (3.8), three units & 2 & 3 \\\\\n\\end{audittable}\n\nThe knapsack encoding remains valid. Where approximation ratios apply, an exact polynomial algorithm also supplies an approximation scheme. I have not tested the paper's remark on the running time of Gurobi.", "problem": "Programs (3.5) and (3.6) choose a set of edges on $n$ vertices so as to maximize a sum of edge weights. Each vertex has an upper bound on its degree, and the known edges must be included. Once the known edges are fixed and their incidences are subtracted from the degree bounds, this is a maximum-weight simple $b$-matching problem. It reduces to ordinary maximum-weight matching on a graph of polynomial size, and matching is solved exactly in polynomial time. Program (3.8) is the case of unit weights. Program (3.5) is solvable in polynomial time; hence it is NP-hard only if $\\mathrm{P}=\\mathrm{NP}$. For its nontrivial threshold decision problem, NP-hardness is equivalent to $\\mathrm{P}=\\mathrm{NP}$, which is open. The error is the confident assertion of this unresolved classification. The program can be written as a multidimensional knapsack instance, in which each item uses one unit of exactly two resources. The hardness of the general knapsack problem does not transfer to this special case. The computational guidance should acknowledge the exact polynomial algorithm. General integer-programming solvers and certified relaxations remain valid options; no failure of their reported running times is established by this correction.", "what_survives": "This computational correction leaves the formulations (3.5), (3.6) and (3.8), their optima, the maximal compatible estimators, the inequality between the maximal and the oracle estimators, and the upper bound from an optimal linear relaxation unchanged. No statistical result uses NP-hardness. General integer-programming solvers remain a valid way to solve the program. The comparisons of exact solvers with relaxations on p.~2248 and in Appendix B should mention the polynomial algorithm."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E002", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the whole of display (3.3), thirteen lines. Red boxes mark the factor '(Y_j - beta-hat)' in the second line, the product 'theta_i theta_j theta_k theta_l' followed by an opening bracket in the line that begins '+ E[beta-hat^4]', and the term '- theta_k theta_l E[Y_i beta-hat] E[Y_k Y_l]' at the end of the next line.", "caption": "Display (3.3), p. 2243 (PDF p. 6). The three marked places are the factor without $\\theta_j$, the prefactor printed before the opening bracket, and the term without $\\theta_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T17", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T17"}, {"id": "P011-T18", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T18"}], "headline": "In display (3.3) the weights $\\theta_j$ and $\\theta_i$ are each missing once, and the opening bracket of the subtracted product of expectations is misplaced.", "id": "P011-E003", "location": "Display (3.3), p. 2243 (PDF p. 6): the second line, the line that begins “$+\\,\\mathrm{E}[\\hat\\beta^4]$”, and the line after it", "pdf_page": 6, "printed_page": "2243", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P011-T17", "P011-T18"], "text": {"as_printed": "\\[\n\\begin{aligned}\n\\xi_{ijkl}&=\\operatorname{cov}\\bigl((\\theta_iY_i-\\hat\\beta)(\\theta_jY_j-\\hat\\beta),(\\theta_kY_k-\\hat\\beta)(\\theta_lY_l-\\hat\\beta)\\bigr)\\\\\n&=\\mathrm{E}\\bigl[(\\theta_iY_i-\\hat\\beta)\\hl{(Y_j-\\hat\\beta)}(\\theta_kY_k-\\hat\\beta)(\\theta_lY_l-\\hat\\beta)\\bigr]\\\\\n&\\quad-\\mathrm{E}\\bigl[(\\theta_iY_i-\\hat\\beta)(\\theta_jY_j-\\hat\\beta)\\bigr]\\mathrm{E}\\bigl[(\\theta_kY_k-\\hat\\beta)(\\theta_lY_l-\\hat\\beta)\\bigr]\\\\\n&=\\cdots+\\mathrm{E}[\\hat\\beta^4]-\\theta_i\\theta_j\\theta_k\\theta_l\\hl{\\bigl[}\\mathrm{E}[Y_iY_j]\\mathrm{E}[Y_kY_l]-\\theta_i\\theta_j\\theta_k\\mathrm{E}[Y_iY_j]\\mathrm{E}[Y_k\\hat\\beta]\\\\\n&\\quad-\\theta_i\\theta_j\\theta_l\\mathrm{E}[Y_iY_j]\\mathrm{E}[Y_l\\hat\\beta]+\\theta_i\\theta_j\\mathrm{E}[Y_iY_j]\\mathrm{E}[\\hat\\beta^2]-\\hl{\\theta_k\\theta_l}\\mathrm{E}[Y_i\\hat\\beta]\\mathrm{E}[Y_kY_l]+\\cdots+\\mathrm{E}[\\hat\\beta^2]\\mathrm{E}[\\hat\\beta^2]\\bigr]\n\\end{aligned}\n\\]", "corrected": "The identity in weighted variables requires the displayed weighted moments to exist. The raw-outcome expansion below is read where its individual mixed moments exist. Assumption 2 bounds $X_i=\\theta_iY_i$; it does not by itself ensure raw moments for coordinates with $\\theta_i=0$.\nLet $X_i=\\theta_iY_i$. Then\n\\[\n\\begin{aligned}\n\\xi_{ijkl}&=\\mathrm{E}\\bigl[(X_i-\\hat\\beta)\\hl{(X_j-\\hat\\beta)}(X_k-\\hat\\beta)(X_l-\\hat\\beta)\\bigr]-Q_{ij}Q_{kl},\\\\\nQ_{ij}&=\\theta_i\\theta_j\\mathrm{E}[Y_iY_j]-\\theta_i\\mathrm{E}[Y_i\\hat\\beta]-\\theta_j\\mathrm{E}[Y_j\\hat\\beta]+\\mathrm{E}[\\hat\\beta^2].\n\\end{aligned}\n\\]\nThe sixteen terms of the first expectation are those printed, and the sixteen terms of $Q_{ij}Q_{kl}$ are those printed inside the bracket, with three changes to the printed form.\n\\begin{enumerate}\n\\item The second factor of the first expectation is $(\\hl{\\theta_j}Y_j-\\hat\\beta)$.\n\\item The subtracted block begins $-\\hl{\\bigl[}\\theta_i\\theta_j\\theta_k\\theta_l\\mathrm{E}[Y_iY_j]\\mathrm{E}[Y_kY_l]-\\cdots$.\n\\item The fifth term of the block is $-\\hl{\\theta_i}\\theta_k\\theta_l\\mathrm{E}[Y_i\\hat\\beta]\\mathrm{E}[Y_kY_l]$.\n\\end{enumerate}", "evidence": "The corrected display follows from the identity $\\operatorname{cov}(U,V)=\\mathrm{E}[UV]-\\mathrm{E}[U]\\mathrm{E}[V]$ and a termwise expansion. For a numerical check, let $Y=(1,1,1,1)$ and $\\theta=(2,3,4,5)$ be constants. In this case $\\hat\\beta=7/2$ and the covariance is 0. The unexpanded expression as printed equals $9/4$, the expansion as printed equals $93009/16$, and the corrected display equals 0.", "problem": "The definition printed above the display is $\\xi_{ijkl}=\\operatorname{cov}\\bigl((\\theta_iY_i-\\hat\\beta)(\\theta_jY_j-\\hat\\beta),(\\theta_kY_k-\\hat\\beta)(\\theta_lY_l-\\hat\\beta)\\bigr)$. The first expectation is taken of a product whose second factor is $\\theta_jY_j-\\hat\\beta$. In the expansion of the subtracted product of two expectations, the first of the sixteen terms is $\\theta_i\\theta_j\\theta_k\\theta_l\\mathrm{E}[Y_iY_j]\\mathrm{E}[Y_kY_l]$, and the bracket must open before it. As printed, the product $\\theta_i\\theta_j\\theta_k\\theta_l$ multiplies all sixteen terms. The term in $\\mathrm{E}[Y_i\\hat\\beta]\\mathrm{E}[Y_kY_l]$ carries $\\theta_i\\theta_k\\theta_l$.", "what_survives": "The sixteen terms of the first expectation are correct as printed. The covariance terms of display (3.4), which the proof uses, agree with the corrected expansion. The other error in the proof of Proposition 1 is separate (see \\fid{P011-E001})."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E003", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the proof of Proposition 2, followed by the paragraph that introduces program (3.8). Red boxes mark the words 'allowing us to invoke Slutsky's Theorem and Chebyshev's Inequality to show' and the limit that follows them: the limit, as n tends to infinity, of the probability that the absolute value of n V 2 star of A O, Z minus n var of beta hat is less than epsilon, set equal to 0.", "caption": "Proof of Proposition 2, p. 2246 (PDF p. 9). The marked sentence asserts that $n\\widehat V_2^*(A^{\\mathcal O};Z)$ consistently estimates $n\\operatorname{var}(\\hat\\beta)$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E004.png"}, {"alt": "Two line charts against the sample size n, on a logarithmic scale from 2 to 128. In the left chart the expected value of n V 2 star rises from 1.50 at n = 2 to 1.992 at n = 128 and approaches a dashed line at 2, labelled actual limit; a solid line at 1 is labelled as the limit asserted in the printed proof, the scaled variance. In the right chart the probability that n V 2 star misses the scaled variance by more than one half rises from 0.5 to 0.997, and the probability that n V 2 falls short of the scaled variance by more than one half falls from 0.25 to below 10 to the power minus 18.", "caption": "Exact values in the counterexample, for $n=2,4,8,16,32,64,128$. Left: the expected value of $n\\widehat V_2^*(A^{\\mathcal O};Z)$, against the limit $n\\operatorname{var}(\\hat\\beta)=1$ that the printed proof asserts. Right: the probability that $n\\widehat V_2^*(A^{\\mathcal O};Z)$ differs from $n\\operatorname{var}(\\hat\\beta)$ by more than $1/2$, and the probability of the event of Proposition 2 with $\\epsilon=1/2$. The graph of the example has no edges, so the auxiliary estimator and $\\widehat V_2(A^{\\mathcal O};Z)$ coincide.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E004-auxiliary-limit.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T11", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T11"}, {"id": "P011-T12", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T12"}, {"id": "P011-T19", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T19"}, {"id": "P011-T34", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T34"}, {"id": "P011-T36", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T36"}], "headline": "The proof of Proposition 2 rests on the consistency of an auxiliary estimator, which can fail when the means $E[\\theta_iY_i]$ differ across units; Proposition 2 itself holds under Assumptions 2 and 3 by a direct one-sided argument.", "id": "P011-E004", "location": "Proof of Proposition 2, p. 2246 (PDF p. 9)", "pdf_page": 9, "printed_page": "2246", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P011-T11", "P011-T12", "P011-T19", "P011-T34", "P011-T36"], "text": {"as_printed": "\\emph{Proof.} We first define an alternative oracle estimator which presumes knowledge of the correlations $\\rho_i$,\n\\[\n\\widehat V_2^*(A^{\\mathcal O};Z)=\\frac{\\hat\\sigma_\\theta^2}{n}\\left[1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij}\\rho_i\\right].\n\\]\nMultiplying by $n$, $n\\widehat V_2^*(A^{\\mathcal O};Z)=\\hat\\sigma_\\theta^2\\left[1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij}\\rho_i\\right]$. As in the proof of Proposition 1, $\\hat\\sigma_\\theta^2$ converges in mean square. By Assumption 2, $1\\le1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij}\\le1+c_2$, allowing us to invoke Slutsky's Theorem and Chebyshev's Inequality to show $\\hl{\\lim_{n\\to\\infty}\\Pr(|n\\widehat V_2^*(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)|<\\epsilon)=0}$. The Cauchy-Schwarz Inequality (i.e., all $\\rho_i\\le1$) implies $\\widehat V_2^*(A^{\\mathcal O};Z)\\le\\widehat V_2(A^{\\mathcal O};Z)$ across all sample realizations. The result follows directly.", "corrected": "The statement is unchanged: given Assumptions 2 and 3, $\\lim_{n\\to\\infty}\\Pr(n\\operatorname{var}(\\hat\\beta)-n\\widehat V_2(A^{\\mathcal O};Z)>\\epsilon)=0$ for any $\\epsilon>0$. I replace the proof by the following.\n\n\\emph{Proof.} Let $X_i=\\theta_iY_i$ and $\\delta_i=E[X_i]-\\beta$, and let $\\sigma_n^2=\\operatorname{var}(X_i)$ be the common variance of Assumption 3 at sample size $n$. Define\n\\[\n\\begin{aligned}\nM_n&=\\frac1n\\sum_{i\\in\\mathcal V_S}\\delta_i^2\\ge0, &\\qquad g_n&=1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij},\\\\\nU_n&=\\frac1n\\sum_{i\\in\\mathcal V_S}(X_i-\\beta)^2, &\\qquad R_n&=\\hat\\sigma_\\theta^2-\\sigma_n^2-M_n .\n\\end{aligned}\n\\]\nBy Assumption 2, $|X_i-\\beta|\\le c_1$ and $1\\le g_n\\le1+c_2$. Since $E[(X_i-\\beta)^2]=\\sigma_n^2+\\delta_i^2$, it follows that $E[U_n]=\\sigma_n^2+M_n$. For $i\\ne j$ with $A^{\\mathcal O}_{ij}=0$, the variables $X_i$ and $X_j$ are independent, so that $\\operatorname{cov}((X_i-\\beta)^2,(X_j-\\beta)^2)=0$ and $\\operatorname{cov}(X_i,X_j)=0$. At most $n(1+c_2)$ ordered pairs $(i,j)$ have $i=j$ or $A^{\\mathcal O}_{ij}=1$, and for such a pair the two covariances are at most $c_1^4$ and $c_1^2$ in absolute value. Hence\n\\[\n\\operatorname{var}(U_n)\\le\\frac{(1+c_2)c_1^4}{n},\\qquad\nE[(\\hat\\beta-\\beta)^2]=\\operatorname{var}(\\hat\\beta)\\le\\frac{(1+c_2)c_1^2}{n},\n\\]\nwhere the equality uses $E[\\hat\\beta]=\\beta$. Since $|\\hat\\beta-\\beta|\\le c_1$, this gives $E[(\\hat\\beta-\\beta)^4]\\le(1+c_2)c_1^4/n$. The identity $\\hat\\sigma_\\theta^2=U_n-(\\hat\\beta-\\beta)^2$ gives $R_n=(U_n-E[U_n])-(\\hat\\beta-\\beta)^2$, and therefore\n\\[\nE[R_n^2]\\le2\\operatorname{var}(U_n)+2E[(\\hat\\beta-\\beta)^4]\\le\\frac{4(1+c_2)c_1^4}{n}.\n\\]\nBy the Cauchy--Schwarz inequality and Assumption 3, $\\operatorname{cov}(X_i,X_j)\\le\\sigma_n^2$ for all $i$ and $j$, and $\\operatorname{cov}(X_i,X_j)=0$ when $i\\ne j$ and $A^{\\mathcal O}_{ij}=0$. Hence\n\\[\nn\\operatorname{var}(\\hat\\beta)=\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}\\operatorname{cov}(X_i,X_j)\\le\\sigma_n^2g_n .\n\\]\nBy (3.7), $n\\widehat V_2(A^{\\mathcal O};Z)=g_n\\hat\\sigma_\\theta^2=g_n(\\sigma_n^2+M_n+R_n)$. Since $M_n\\ge0$,\n\\[\n\\hl{n\\operatorname{var}(\\hat\\beta)-n\\widehat V_2(A^{\\mathcal O};Z)\\le-g_n(M_n+R_n)\\le(1+c_2)|R_n|} .\n\\]\nBy Markov's inequality, for every $\\epsilon>0$,\n\\[\n\\Pr\\bigl(n\\operatorname{var}(\\hat\\beta)-n\\widehat V_2(A^{\\mathcal O};Z)>\\epsilon\\bigr)\\le\\frac{(1+c_2)^2E[R_n^2]}{\\epsilon^2}\\le\\frac{4(1+c_2)^3c_1^4}{n\\epsilon^2}\\to0 .\n\\]\nThe argument uses no auxiliary estimator and no correlations. It needs no condition on the means beyond $E[\\hat\\beta]=\\beta$, and no limit for $\\sigma_n^2$ or $M_n$.", "evidence": "I use an exact counterexample. Let $\\varepsilon_1,\\varepsilon_2,\\ldots$ be independent random variables such that $\\Pr(\\varepsilon_i=1)=\\Pr(\\varepsilon_i=-1)=1/2$, and let $Y_i=\\mu_i+\\varepsilon_i$ with $\\mu_i=2$ for odd $i$ and $\\mu_i=0$ for even $i$. At sample size $n$ let $\\theta_i=n/(2\\lceil n/2\\rceil)$ for every $i$, let $\\beta=1$, and let the graph have no edges, with every vertex observed. Then $E[\\hat\\beta]=1$, $|\\theta_iY_i-\\beta|\\le2$, every degree is zero, $|\\theta_i|\\le1$, $\\operatorname{var}(\\theta_iY_i)=\\theta_i^2$ is the same for all $i$, and $n\\operatorname{var}(\\hat\\beta)=\\theta_1^2\\to1$. Assumptions 2 and 3 hold with $c_1=2$, $c_2=1$, $c_3=1$ and $c_4=2$. Since $A^{\\mathcal O}=0$, I have $n\\widehat V_2^*(A^{\\mathcal O};Z)=n\\widehat V_2(A^{\\mathcal O};Z)=\\hat\\sigma_\\theta^2$, whatever the correlations are taken to be. For even $n$ the weights equal $1$ and $M_n=1$, and a direct computation gives\n\\[\nE\\bigl[(\\hat\\sigma_\\theta^2-2)^2\\bigr]=\\frac4n+\\frac3{n^2}-\\frac2{n^3}.\n\\]\nAlong even $n$, this shows that $n\\widehat V_2^*(A^{\\mathcal O};Z)\\to2$ in mean square, while $n\\operatorname{var}(\\hat\\beta)=1$. Markov's inequality then implies that\n\\[\n\\Pr\\bigl(|n\\widehat V_2^*(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)|>1/2\\bigr)\\ge1-4\\Bigl(\\frac4n+\\frac3{n^2}-\\frac2{n^3}\\Bigr)\\to1 .\n\\]\nFor odd $n$, I find $M_n\\to1$ and the same two limits by the same computation. An enumeration of all outcomes in exact arithmetic agrees with these formulas for the sample sizes in the table.\n\\begin{audittable}{rllll}{Exact values in the counterexample (the last two columns are rounded).}\n$n$ & $E[n\\widehat V_2^*(A^{\\mathcal O};Z)]$ & $n\\operatorname{var}(\\hat\\beta)$ & $\\Pr(|n\\widehat V_2^*-n\\operatorname{var}(\\hat\\beta)|>1/2)$ & $\\Pr(n\\operatorname{var}(\\hat\\beta)-n\\widehat V_2>1/2)$ \\\\\n\\hline\n$2$ & $3/2$ & $1$ & $0.5000$ & $0.2500$ \\\\\n$8$ & $15/8$ & $1$ & $0.6641$ & $0.0352$ \\\\\n$32$ & $63/32$ & $1$ & $0.8887$ & $6.3\\times10^{-6}$ \\\\\n$128$ & $255/128$ & $1$ & $0.9970$ & $2.6\\times10^{-19}$ \\\\\n\\end{audittable}\nNote that the last column is the probability of the event of Proposition 2, which tends to $0$ as the proposition asserts. Unequal means do not always destroy the consistency. If $\\mu_1=2$, $\\mu_2=0$ and $\\mu_i=1$ for $i\\ge3$, with $\\theta_i=1$ for $n\\ge2$, then $M_n=2/n$ and $E[(n\\widehat V_2^*(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta))^2]=11/n^2-2/n^3\\to0$. The replacement proof below is a derivation, and each of its steps is given there.", "problem": "The proof introduces an auxiliary estimator $\\widehat V_2^*(A^{\\mathcal O};Z)$ and asserts that $n\\widehat V_2^*(A^{\\mathcal O};Z)$ is consistent for $n\\operatorname{var}(\\hat\\beta)$. It then passes to $\\widehat V_2(A^{\\mathcal O};Z)$ by the inequality $\\widehat V_2^*(A^{\\mathcal O};Z)\\le\\widehat V_2(A^{\\mathcal O};Z)$. The consistency is false in general under Assumptions 2 and 3. Definition 3 requires only that the average of the means $E[\\theta_iY_i]$ equal $\\beta$, and Assumption 3 equates the variances of the $\\theta_iY_i$, not their means. Let $\\delta_i=E[\\theta_iY_i]-\\beta$ and $M_n=n^{-1}\\sum_{i\\in\\mathcal V_S}\\delta_i^2$. Since the sample variance $\\hat\\sigma_\\theta^2$ is centered at the single value $\\hat\\beta$, it tracks the common variance of the $\\theta_iY_i$ plus $M_n$, and not the common variance alone. In the example below $M_n\\to1$, and $n\\widehat V_2^*(A^{\\mathcal O};Z)$ tends to $2$ while $n\\operatorname{var}(\\hat\\beta)$ tends to $1$. In this example, the sentence that $\\hat\\sigma_\\theta^2$ ``converges in mean square'' is correct; the error lies in the limit that the next sentence assigns to the estimator. The cause is the one described in \\fid{P011-E001}. The marked display also carries a reversed inequality sign (\\fid{P011-E005}), and the proof writes the correlations with a single index (\\fid{P011-E006}). After both are corrected, the consistency still fails.", "what_survives": "Proposition 2 holds as stated, with no added assumption, and Lemma 1 and Corollaries 2 and 4 keep their conclusions. The estimators (3.7) and (3.9), program (3.8) and the software are unaffected, since the auxiliary estimator appears only in this proof. This correction does not restore Proposition 1 or Corollaries 1 and 3, which are the subject of \\fid{P011-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E004", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the last five lines of the proof of Proposition 2. A red box marks the symbols 'less than epsilon' inside the probability, in the limit that is set equal to 0.", "caption": "Proof of Proposition 2, p. 2246 (PDF p. 9). The marked inequality in the event is printed as “less than $\\epsilon$”.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T19", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T19"}, {"id": "P011-T20", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T20"}, {"id": "P011-T34", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T34"}], "headline": "The event in the limit should read $>\\epsilon$, not $<\\epsilon$.", "id": "P011-E005", "location": "Proof of Proposition 2, p. 2246 (PDF p. 9), the limit after “Chebyshev’s Inequality to show”", "pdf_page": 9, "printed_page": "2246", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P011-T19", "P011-T20", "P011-T34"], "text": {"as_printed": "\\ldots\\ Chebyshev's Inequality to show $\\lim_{n\\to\\infty}\\Pr(|n\\widehat V_2^*(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)|\\hl{<}\\epsilon)=0$.", "corrected": "\\ldots\\ Chebyshev's Inequality to show $\\lim_{n\\to\\infty}\\Pr(|n\\widehat V_2^*(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)|\\hl{>}\\epsilon)=0$.\n\nThis is the statement that the sentence intends, for any $\\epsilon>0$ and under Assumptions 2 and 3. It holds in the example above, where the means $E[\\theta_iY_i]$ are equal, and it does not hold in general (\\fid{P011-E004}).", "evidence": "I checked the page and made a short calculation. Proposition 1 on p.~2242 and Lemma 3 on p.~2247 state consistency with $>\\epsilon$. To see that the printed limit fails, let the $Y_i$ be independent with $\\Pr(Y_i=1)=\\Pr(Y_i=-1)=1/2$, let $\\theta_i=1$ and $\\beta=0$, and let the graph have no edges. Then $n\\operatorname{var}(\\hat\\beta)=1$ and $n\\widehat V_2^*(A^{\\mathcal O};Z)=\\hat\\sigma_\\theta^2=1-\\hat\\beta^2$. The absolute error equals $\\hat\\beta^2$, and $E[\\hat\\beta^2]=1/n$. By Markov's inequality, I have\n\\[\n\\Pr\\bigl(|n\\widehat V_2^*(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)|<\\epsilon\\bigr)\\ge1-\\frac1{n\\epsilon}\\to1\n\\]\nfor every fixed $\\epsilon>0$.", "problem": "The sentence asserts convergence in probability of the auxiliary estimator, for which the probability that the error exceeds $\\epsilon$ must tend to zero. The inequality sign is reversed. As printed, the display says that the probability of an error smaller than $\\epsilon$ tends to zero.", "what_survives": "Only the sign changes. The corrected display is the consistency of the auxiliary estimator, which is the step that \\fid{P011-E004} shows to be false in general. The replacement proof given there does not use it."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E005", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the whole proof of Proposition 2. Four red boxes mark the symbol rho with subscript i: after the words 'of the correlations', at the end of the double sum in the displayed definition of V 2 star, at the end of the double sum in the line that begins 'Multiplying by n', and in the words 'all rho i less than or equal to 1'.", "caption": "Proof of Proposition 2, p. 2246 (PDF p. 9). The four marked symbols are the correlations, each printed with the single index $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T11", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T11"}, {"id": "P011-T19", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T19"}, {"id": "P011-T21", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T21"}, {"id": "P011-T34", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T34"}], "headline": "The correlations in the proof of Proposition 2 belong to pairs of units, $\\rho_{ij}$; they are printed four times with a single index, $\\rho_i$, which the paper does not define.", "id": "P011-E006", "location": "Proof of Proposition 2, p. 2246 (PDF p. 9): the sentence that defines the auxiliary estimator, its two displays, and “all $\\rho_i\\le1$”", "pdf_page": 9, "printed_page": "2246", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P011-T11", "P011-T19", "P011-T21", "P011-T34"], "text": {"as_printed": "We first define an alternative oracle estimator which presumes knowledge of the correlations $\\hl{\\rho_i}$,\n\\[\n\\widehat V_2^*(A^{\\mathcal O};Z)=\\frac{\\hat\\sigma_\\theta^2}{n}\\left[1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij}\\hl{\\rho_i}\\right].\n\\]\nMultiplying by $n$, $n\\widehat V_2^*(A^{\\mathcal O};Z)=\\hat\\sigma_\\theta^2\\left[1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij}\\hl{\\rho_i}\\right]$. \\ldots\\ The Cauchy-Schwarz Inequality (i.e., all $\\hl{\\rho_i}\\le1$) implies $\\widehat V_2^*(A^{\\mathcal O};Z)\\le\\widehat V_2(A^{\\mathcal O};Z)$ across all sample realizations.", "corrected": "Let $X_i=\\theta_iY_i$, and let $\\sigma_\\theta^2=\\operatorname{var}(X_i)$ be the common variance of Assumption 3. For $i,j\\in\\mathcal V_S$ define $\\hl{\\rho_{ij}}=\\operatorname{cov}(X_i,X_j)/\\sigma_\\theta^2$ if $\\sigma_\\theta^2>0$, and $\\rho_{ij}=0$ if $\\sigma_\\theta^2=0$. The auxiliary estimator is\n\\[\n\\widehat V_2^*(A^{\\mathcal O};Z)=\\frac{\\hat\\sigma_\\theta^2}{n}\\left[1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij}\\hl{\\rho_{ij}}\\right],\n\\]\nwith $\\hl{\\rho_{ij}}$ also in the display that follows ``Multiplying by $n$'' and in ``all $\\hl{\\rho_{ij}}\\le1$''. Under Assumption 2, $n\\operatorname{var}(\\hat\\beta)\\to c_3>0$, so $\\sigma_\\theta^2>0$ for all large $n$.", "evidence": "I checked the page and derived the variance identity. Let $X_i=\\theta_iY_i$, let $\\sigma_\\theta^2>0$ be the common variance, and let $\\rho_{ij}=\\operatorname{cov}(X_i,X_j)/\\sigma_\\theta^2$. Since $X_i$ and $X_j$ are independent when $i\\ne j$ and $A^{\\mathcal O}_{ij}=0$, I have\n\\[\nn\\operatorname{var}(\\hat\\beta)=\\frac1n\\Bigl[\\sum_{i\\in\\mathcal V_S}\\operatorname{var}(X_i)+\\sum_{i\\ne j}\\operatorname{cov}(X_i,X_j)\\Bigr]\n=\\sigma_\\theta^2\\Bigl[1+\\frac1n\\sum_{i\\in\\mathcal V_S}\\sum_{j\\in\\mathcal V_S}A^{\\mathcal O}_{ij}\\rho_{ij}\\Bigr].\n\\]\nThis is the identity on which the auxiliary estimator is built. An example shows that no single index can stand for the pair. Let $U$ and $V$ be independent, each equal to $1$ or $-1$ with probability $1/2$, and let $X_1=U$, $X_2=3U/5+4V/5$ and $X_3=V$ on the path with edges $\\{1,2\\}$ and $\\{2,3\\}$. The three variances equal $1$, the two correlations at vertex 2 are $\\rho_{12}=3/5$ and $\\rho_{23}=4/5$, and both sides of the identity equal $29/15$. An average of the $\\rho_{ij}$ over the neighbors $j$ of $i$ would give the same double sum, but the paper defines no such quantity.", "problem": "The symbol $\\rho_i$ is not defined in the paper. The proof needs the correlation between $\\theta_iY_i$ and $\\theta_jY_j$, which depends on the pair $(i,j)$: two neighbors of one vertex can have different correlations with it. The bound $\\operatorname{cov}(\\theta_iY_i,\\theta_jY_j)\\le\\sigma_\\theta^2A^{\\mathcal O}_{ij}$ for distinct units, stated before (3.7), and the double sum over $i$ and $j$ show that the correlation of the pair is meant and that the index $j$ was dropped.", "what_survives": "The inequality $\\widehat V_2^*(A^{\\mathcal O};Z)\\le\\widehat V_2(A^{\\mathcal O};Z)$ is unchanged, since $\\rho_{ij}\\le1$, $A^{\\mathcal O}_{ij}\\ge0$ and $\\hat\\sigma_\\theta^2\\ge0$. The corrected index does not establish the consistency that the proof asserts for the auxiliary estimator (\\fid{P011-E004})."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E006", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last six lines of the page. Red boxes mark 'a_ij = 1' at the end of one line and 'if v-hat_ij is not equal to 0, and 0 otherwise,' at the start of the next.", "caption": "Last paragraph of p. 2244 (PDF p. 7). The marked clause defines the decision variable by a condition on the data.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T22", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T22"}, {"id": "P011-T23", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T23"}, {"id": "P011-T27", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T27"}], "headline": "The decision variable $a_{ij}$ of program (3.6) is the indicator that the edge $\\{i,j\\}$ is selected, and not the indicator that $\\hat v_{ij}\\neq0$.", "id": "P011-E007", "location": "Last paragraph of p. 2244 (PDF p. 7), the sentence that defines the decision variables of program (3.6)", "pdf_page": 7, "printed_page": "2244", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P011-T22", "P011-T23", "P011-T27"], "text": {"as_printed": "``Let $\\hat v_{ij}$ be the $ij$th element of the sample covariance matrix with $i=1,...,n$ and $j=1,...,n$. Since the sample covariance matrix is symmetric, we can focus on its upper triangular part and use the decision variable $a_{ij}=1$ if $\\hl{\\hat v_{ij}\\neq0}$, and 0 otherwise, for each $i<j$.''", "corrected": "``Let $\\hl{\\hat v_{ij}=(\\theta_iY_i-\\hat\\beta)(\\theta_jY_j-\\hat\\beta)}$ for $i=1,...,n$ and $j=1,...,n$. Since this matrix is symmetric, we can focus on its upper triangular part and use the decision variable $a_{ij}=1$ if \\hl{the edge $\\{i,j\\}$ is selected}, and 0 otherwise, for each $i<j$.''", "evidence": "I base this correction on an exact example. It has three units with $\\theta_i=1$, centred outcomes $(-2,1/2,3/2)$, degree bounds $(1,1,1)$ and no known edge. The products $\\hat v_{12}$, $\\hat v_{13}$ and $\\hat v_{23}$ are $-1$, $-3$ and $3/4$. All are nonzero, and under the printed rule every vertex has degree 2. A maximizer of (3.6) selects the edge $\\{2,3\\}$ only. The package builds its objective from the outer product of the centred values and imposes the constraints on the degrees and on the known edges separately.", "problem": "As printed, the data fix every $a_{ij}$, and nothing is left to optimize. The values so fixed can violate the degree constraints of (3.6), and they can leave out a known edge whose coefficient is zero. Program (3.6), which is printed next, treats the $a_{ij}$ as free binary variables and $\\hat v_{ij}$ as their objective coefficients. The sentence also leaves the matrix with entries $\\hat v_{ij}$ undefined. By (3.5), its entries are the cross-products $(\\theta_iY_i-\\hat\\beta)(\\theta_jY_j-\\hat\\beta)$.", "what_survives": "Program (3.6) and the package use the intended variable, and nothing depends on the sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E007", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of six lines of the second paragraph of the introduction. A red box marks the word 'non-independence' in the phrase 'they imply marginal non-independence relations in a set of variables'. The preceding sentence reads 'Vertices represent individual units and edges represent the possibility of probabilistic dependence.'", "caption": "Introduction, p. 2239 (PDF p. 2), second paragraph. The marked word should read “independence”.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E008.png"}], "formal_release": null, "formal_statements": [], "headline": "A dependency graph implies marginal independence relations; an edge permits dependence and does not require it.", "id": "P011-E008", "location": "Introduction, p. 2239 (PDF p. 2), second paragraph", "pdf_page": 2, "printed_page": "2239", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Dependency graphs are useful because they imply marginal \\hl{non-independence} relations in a set of variables, and the class of joint distributions compatible with any dependency graph is flexible.''", "corrected": "``Dependency graphs are useful because they imply marginal \\hl{independence} relations in a set of variables, and the class of joint distributions compatible with any dependency graph is flexible.'' Absent edges between disjoint sets of vertices imply independence of the corresponding outcomes; present edges permit dependence and do not require it.", "intended_meaning": "The preceding sentence says that edges represent the possibility of probabilistic dependence. Definition 1 (p.~2240) is a one-way statement: if no edge joins two disjoint sets of vertices, then the outcomes of the two sets are independent. The graph therefore implies independence relations, and an edge implies nothing. For example, two independent Bernoulli variables joined by an edge satisfy Definition 1. The paper uses Definition 1 throughout, and no result depends on this sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E008", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of nine lines of Appendix B. A red box marks the word 'depinf' in the sentence 'In both depgraph and depinf, we give the option to find an exact solution to (3.5) and (3.8)'. The sentence before it reads 'depinf includes two basic functions: depgraph, ... and depvar for calculating the variance estimates (3.1) and (3.7).'", "caption": "Appendix B, p. 2250 (PDF p. 13). The marked name is that of the package; the preceding sentence names the two functions depgraph and depvar.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "“In both depgraph and depinf” should read “In both depgraph and depvar”: depinf is the package, and its two functions are depgraph and depvar.", "id": "P011-E009", "location": "Appendix B, p. 2250 (PDF p. 13), sentence beginning “In both depgraph and depinf”", "pdf_page": 13, "printed_page": "2250", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "In both \\texttt{depgraph} and \\hl{\\texttt{depinf}}, we give the option to find an exact solution to (3.5) and (3.8) via integer programming, or an approximate solution to the relaxations of (3.5) and (3.8) via linear programming.", "corrected": "``In both \\texttt{depgraph} and \\hl{\\texttt{depvar}}, we give the option to find an exact solution to (3.5) and (3.8) via integer programming, or an approximate solution to the relaxations of (3.5) and (3.8) via linear programming.''", "evidence": "I checked the page and the package source. Both functions take the argument \\texttt{approximate}, which selects the integer program or its relaxation, and \\texttt{depvar} passes it to \\texttt{depgraph}. The package defines no function named \\texttt{depinf}.", "problem": "The preceding sentence names the two functions of the package, \\texttt{depgraph} and \\texttt{depvar}. The second name in this sentence is the name of the package.", "what_survives": "Nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E009", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing in monospaced type with line numbers. A red box marks line 30, 'rows_GR = (row_count+1):(row_count+length(cols_GR))', and a second red box marks line 33, 'row_count = max(rows_GR)'. Lines 73 and 82 below them build the vectors bvec and sense with length(rows_GR) further entries.", "caption": "Package depinf, file R/depgraph.r, lines 27–34, 72–74 and 81–83, typeset from the file with line numbers added. The two marked lines build the row indices of the known-edge constraints.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E011.png"}], "formal_release": null, "formal_statements": [], "headline": "The function depgraph stops with an error when the matrix GR of known edges is supplied and contains no edge; in that case the row indices of the known-edge constraints must form an empty sequence.", "id": "P011-E011", "location": "R package depinf, named in Appendix B, p. 2250 (PDF p. 13): R/depgraph.r, lines 27–34, with lines 73 and 82. The code is not printed in the article.", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Lines 30 and 33 of \\texttt{R/depgraph.r}, inside the block \\texttt{if (!is.null(GR))}:\n\n\\texttt{rows\\_GR = \\hl{(row\\_count+1):(row\\_count+length(cols\\_GR))}}\n\n\\texttt{row\\_count = \\hl{max(rows\\_GR)}}", "corrected": "\\texttt{rows\\_GR = \\hl{row\\_count + seq\\_along(cols\\_GR)}}\n\n\\texttt{row\\_count = \\hl{row\\_count + length(cols\\_GR)}}", "evidence": "I base this correction on an exact example, run with the package and the GLPK solver. Let \\texttt{v} be the $3\\times3$ matrix of ones, let $d=(1,1,1)$, and let \\texttt{GR} be the $3\\times3$ matrix of zeros. Then the vector \\texttt{rows\\_GR} is \\texttt{4:3}. The constraint matrix has three rows, and the right-hand side and the directions have length five. The solver interface stops with the message \\texttt{Arguments 'mat', 'dir', and/or 'rhs' not conformable.} The same failure occurs with the scalar \\texttt{d = 1} (one row against three entries), for the integer program and for the linear program, and in both branches of \\texttt{depvar} that call \\texttt{depgraph}. With the two corrected lines, a matrix of zeros and \\texttt{GR = NULL} produce the same program and the same answer in these tests, and the tested calls with known edges are unchanged.", "problem": "The article allows a recruitment graph with no known edge: on p.~2248 the constraint $A\\succeq A_R$ becomes $A\\succeq 0$ and is superfluous. A matrix \\texttt{GR} of zeros encodes this case, and it should produce the same program as \\texttt{GR = NULL}. In the code, \\texttt{cols\\_GR} is then empty, and in R the expression \\texttt{(row\\_count+1):(row\\_count+0)} is a descending sequence of length two and not an empty sequence. The right-hand side and the vector of constraint directions (lines 73 and 82) receive two entries that correspond to no row of the constraint matrix, and the solver rejects the program.", "what_survives": "No wrong number is produced, because the call stops. The equivalent call \\texttt{GR = NULL} works. The calls of the vignette pass \\texttt{NULL} or a matrix with edges (97 known edges in the example data and 468 in the application), and they do not take this path. The programs, estimators and proofs of the article do not depend on this code. The CPLEX and Gurobi branches receive the same program, and they were not run."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E011", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of equation (3.9), which defines V 2 prime of Z as sigma hat theta squared over n, times 1 plus the average of the truncated degrees d i prime, followed by Lemma 1. Red boxes mark the sentence 'The estimator (3.9) does not depend on any particular compatible adjacency matrix.'", "caption": "Equation (3.9), p. 2247 (PDF p. 10): the closed form that the help page promises for this call. The marked sentence is correct; the defect is in the package, whose output for this call does depend on an adjacency matrix.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E012.png"}, {"alt": "Listing of line 24 of the file depvar.Rd in typewriter type, on five rows. Red boxes mark the last sentence: 'Finally, if code case = homoskedastic and code GR is code NULL, (9) is calculated.'", "caption": "Help page of depvar, source file man/depvar.Rd, line 24, typeset from the package with the line number added and the long line continued on four further rows. The marked sentence says that (9) is calculated when GR is NULL.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E012-b.png"}, {"alt": "Listing of lines 39 to 53 of the file depvar.r. Red boxes mark line 42, 'out = depgraph(v, d, GR, solver, approximate)', and line 52, 'V_hat = V2(A_max, x)'. No line of the listing treats GR = NULL separately.", "caption": "R/depvar.r, lines 39–53, typeset from the package with line numbers added. The marked lines call depgraph for every homoskedastic case and evaluate V2 at the graph it returns.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E012-c.png"}, {"alt": "Listing of lines 114 to 119 of the file replication.Rmd. A red box marks the comment '# V2 prime' on line 114. Lines 115 to 118 assign d1 to d4 as the square root of the V_hat returned by depvar with arguments NULL and case = \"homoskedastic\", and line 119 collects them as col7.", "caption": "vignettes/replication.Rmd, lines 114–119, typeset from the package with line numbers added. The marked comment labels the four calls below it as $\\widehat V_2'$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E012-d.png"}], "formal_release": null, "formal_statements": [], "headline": "For case = \"homoskedastic\" with GR = NULL, the function depvar returns the estimator (3.7) at an optimized graph, not the closed form (3.9) that its help page states, and the vignette labels this output $\\widehat V_2'$.", "id": "P011-E012", "location": "Package depinf: man/depvar.Rd, line 24; R/depvar.r, lines 39–53; vignettes/replication.Rmd, lines 114–119. In the article: equation (3.9), p. 2247 (PDF p. 10); Appendix B, p. 2250 (PDF p. 13)", "pdf_page": 10, "printed_page": "2247", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Help page of \\texttt{depvar} (\\texttt{man/depvar.Rd}, line 24, with the code markup of the source shown as typewriter text): ``If \\texttt{case = homoskedastic} and \\texttt{GR} is not \\texttt{NULL}, then (7) in section 3.2 of the same paper is calculated. Finally, if \\texttt{case = homoskedastic} and \\texttt{GR} is \\texttt{NULL}, \\hl{(9) is calculated}.''\n\n\\texttt{R/depvar.r}, lines 39 and 42, and lines 51 and 52:\n\n\\texttt{else if (case==\"homoskedastic\") \\{}\n\n\\texttt{\\hl{out = depgraph(v, d, GR, solver, approximate)}}\n\n\\texttt{else if (case==\"homoskedastic\") \\{}\n\n\\texttt{\\hl{V\\_hat = V2(A\\_max, x)}}\n\n\\texttt{vignettes/replication.Rmd}, lines 114 and 115; lines 116 to 118 repeat the call for coefficients 2 to 4, and line 119 collects the four values as \\texttt{col7}:\n\n\\texttt{\\# \\hl{V2'}}\n\n\\texttt{d1 = sqrt(depvar(Y, Theta[1, ]*n, d, NULL, case = \"homoskedastic\")\\$V\\_hat)}", "corrected": "To honor the help page's choice of estimator while retaining the package's variance divisor $n-1$ (\\fid{P011-E015}), the code changes. For \\texttt{case = \"homoskedastic\"} and \\texttt{GR = NULL}, before any graph program is solved, set \\texttt{x = y*theta} and \\texttt{n = length(x)} and return, for a vector \\texttt{d},\n\\hl{\\texttt{var(x)/n * (1 + sum(pmin(d, n-1))/n)}},\nwith \\texttt{A\\_max = NULL}, \\texttt{obj\\_val = NA\\_real\\_} and \\texttt{time = 0}. For a scalar \\texttt{d}, which the help page defines as the total number of nonzero off-diagonal entries of $A$, return \\texttt{var(x)/n * (1 + min(d, n*(n-1))/n)}. The optimized estimator is kept when \\texttt{GR} is not \\texttt{NULL}. The four values of \\texttt{col7} in the vignette then become $0.43917668$, $0.19200914$, $0.01474585$ and $0.27175162$.\n\nA smaller correction keeps the code and every number, and changes the documentation. Help page: ``Finally, if \\texttt{case = homoskedastic} and \\texttt{GR} is \\texttt{NULL}, \\hl{(7) is calculated at the solution of (8) with no known dependencies (by default, of its linear relaxation)}.'' Vignette, line 114: \\texttt{\\# \\hl{V2, no known dependencies}}. The closed form (3.9) is then obtained directly, as in the sensitivity analysis of the vignette.", "evidence": "I used an exact example and a recalculation. Let $y=(0,1,2)$, $\\theta=(1,1,1)$ and $d=(2,0,0)$, with \\texttt{GR = NULL}. The degree bounds of vertices 2 and 3 exclude every edge, and the zero matrix is the only feasible point of (3.8) and of its relaxation. The package's sample variance is $1$. The returned value is then $1/3$, while the closed form is $(1/3)(1+2/3)=5/9$. The unmodified function returns $1/3$ with \\texttt{approximate = 1} and with \\texttt{approximate = 0}. With the paper's divisor $n$ in place of the package's divisor $n-1$ (\\fid{P011-E015}), the two values are $2/9$ and $10/27$. On the vignette's data, with $n=522$, the truncated degrees sum to $8561$ and the relaxed program without known edges attains $8560$. The resulting standard errors agree with the printed \\texttt{col7} and not with the closed form, which the vignette's sensitivity table computes directly.\n\\begin{audittable}{lrrr}{Standard errors for the vignette's four coefficients: its column col7, and the two estimators recalculated using the package's variance divisor $n-1$.}\nCoefficient & \\texttt{col7} as printed & (3.7) at the optimized graph & Closed form (3.9) \\\\\n\\hline\nintercept & 0.43915251 & 0.43915251 & 0.43917668 \\\\\nhcv & 0.19199857 & 0.19199857 & 0.19200914 \\\\\nage & 0.01474504 & 0.01474504 & 0.01474585 \\\\\nsex & 0.27173666 & 0.27173666 & 0.27175162 \\\\\n\\end{audittable}", "problem": "The help page promises the closed form (3.9), which depends on the degrees only. The code does not compute it in this call. In the comparisons below, I use the package's variance divisor $n-1$ for both formulas; its difference from the paper's divisor $n$ is addressed in \\fid{P011-E015}. For \\texttt{case = \"homoskedastic\"} it always calls \\texttt{depgraph}, which solves program (3.8), by default its linear relaxation, with no known edges when \\texttt{GR} is \\texttt{NULL}. It then returns (3.7) at that solution. Since each row sum of a feasible matrix is at most $\\min\\{d_i,n-1\\}$, the returned value is at most (3.9). When the sample variance is positive, it is strictly smaller unless every row attains its bound. The vignette labels the output of this call \\texttt{V2'}. The four standard errors in its column \\texttt{col7} are values of (3.7) under the label of (3.9). Appendix B of the article describes \\texttt{depvar} correctly, as ``calculating the variance estimates (3.1) and (3.7)''.", "what_survives": "The two underlying formulas are bounds of the paper, and (3.7) at the optimized graph is no larger than (3.9) when the same variance divisor is used. No loss of conservativeness is shown, and no number in the article is affected. Equations (3.7) and (3.9), Lemma 1 and Appendix B are unchanged. What changes is the description of this call in the help page and the label of \\texttt{col7} in the vignette, or else the value that the call returns."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E012", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing in monospaced type with line numbers. Red boxes mark 'n = length(d)' on line 26, 'V_hat = V2p(d, x)' on line 55, and 'A_max = A_max, obj_val = obj_val, time = time' inside the return statement on line 58.", "caption": "Package depinf, file R/depvar.r, lines 23–29 and 48–58, typeset from the file with line numbers added. The branch on line 55 assigns V_hat only, and line 58 returns three further objects.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E013.png"}], "formal_release": null, "formal_statements": [], "headline": "The option case = \"homoskedastic, worst\" of depvar, which is the only route to the closed form (3.9), stops with an error because three of the fields that it returns are never assigned.", "id": "P011-E013", "location": "R package depinf, named in Appendix B, p. 2250 (PDF p. 13): R/depvar.r, lines 23–29 and 54–58. The code is not printed in the article, and the option is not listed in the help page man/depvar.Rd.", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Lines 54--58 of \\texttt{R/depvar.r}:\n\n\\texttt{else if (case==\"homoskedastic, worst\") \\{}\n\n\\texttt{V\\_hat = V2p(d, x)}\n\n\\texttt{\\}}\n\n\\texttt{return(list(V\\_hat = V\\_hat, \\hl{A\\_max = A\\_max, obj\\_val = obj\\_val, time = time}))}\n\nLine 26, in the helper \\texttt{V2p}: \\texttt{\\hl{n = length(d)}}", "corrected": "On the branch \\texttt{case==\"homoskedastic, worst\"}, require \\hl{\\texttt{length(d) == length(x)}} and assign the three fields:\n\n\\texttt{V\\_hat = V2p(d, x)}\n\n\\texttt{\\hl{A\\_max = NULL; obj\\_val = NA\\_real\\_; time = 0}}\n\nIn the helper \\texttt{V2p}, take the sample size from the observations: \\texttt{\\hl{n = length(x)}}. The option and the placeholder values should be documented in the help page. If a scalar \\texttt{d} is also accepted with its documented meaning, a bound $D$ on the total number of nonzero off-diagonal entries, use $\\min\\{D,n(n-1)\\}$ as \\hl{an upper bound on the sum of the degrees} in the closed form. A common bound for every vertex is a different input, and it must be expanded to a vector of length $n$.", "evidence": "I base this correction on a run of the package code and an exact enumeration. With outcomes $(0,1,2)$, $\\theta=(1,1,1)$, no known edge, and $d=(2,0,0)$ or $d=(2,2,2)$, the call stops at the return statement, and no solver is involved. The helper computes $5/9$ and $1$ for these two inputs. If only the return statement is repaired, then for the scalar $d=0$ the call returns $1$, where the value with no edge is $1/3$. Replacing \\texttt{length(d)} by \\texttt{length(x)} does not handle a scalar either. The help page defines a scalar \\texttt{d} as a bound on the total number of nonzero off-diagonal entries. With this meaning, $d=6$ allows the complete graph on three vertices, for which the formula of the helper equals $1$, and the helper with \\texttt{length(x)} returns $5/9$. An enumeration of the eight graphs on three vertices confirms these values.", "problem": "The branch assigns \\texttt{V\\_hat} only. The return statement on line 58 also evaluates \\texttt{A\\_max}, \\texttt{obj\\_val} and \\texttt{time}, which are assigned only in the two branches that call \\texttt{depgraph}. In a clean R session the call stops with \\texttt{object 'A\\_max' not found}. A second defect lies on the same path. The helper \\texttt{V2p} takes the sample size from the length of \\texttt{d}. This is the number of observations when \\texttt{d} is a vector of degrees, which is the case of the article. For a scalar \\texttt{d} it is 1.", "what_survives": "Equation (3.9), Lemma 1 and Corollaries 2 and 4 are not affected, because the defect is in the software only. No wrong number is returned, because the call stops. Nothing in the article, the help pages or the vignette calls this option. The vignette obtains its values labelled $\\widehat V_2'$ from the documented case \\texttt{\"homoskedastic\"} (see \\fid{P011-E012}), and its sensitivity analysis computes the closed form directly."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E013", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of Appendix B. A red box marks the words 'By default, depgraph uses GLPK,'. The line before it reads 'In order to solve (3.5) and (3.8), depgraph can use three different optimization solvers: CPLEX, GLPK and Gurobi.'", "caption": "Appendix B, p. 2250 (PDF p. 13). The marked sentence names GLPK as the default solver; the defect is in the GLPK branch of depgraph.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E014.png"}, {"alt": "Listing of lines 164 to 183 of the file depgraph.r. Red boxes mark line 165, 'out = Rglpk_solve_LP(cvec, Amat, dir, bvec, bounds = bound, types = var_type, max = TRUE)'; lines 173 to 175, which read out$solution; lines 178 and 179, which read out$solution; and line 182, 'obj_val = out$optimum'. No line of the listing mentions out$status.", "caption": "R/depgraph.r, lines 164–183, typeset from the package with line numbers added. The marked lines are the call to the solver and every later use of its output; none tests the status of the solve.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E014-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The function depgraph uses the solver’s output without checking that the solve succeeded: after a failed GLPK solve it returns a graph and an objective value as if they were optimal.", "id": "P011-E014", "location": "Appendix B, p. 2250 (PDF p. 13). Package depinf: R/depgraph.r, lines 156–183 (GLPK), 116–155 (Gurobi) and 92–115 (CPLEX)", "pdf_page": 13, "printed_page": "2250", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Appendix B, p.~2250: ``In order to solve (3.5) and (3.8), \\texttt{depgraph} can use three different optimization solvers: CPLEX, GLPK and Gurobi. By default, \\texttt{depgraph} uses GLPK, \\ldots''\n\n\\texttt{R/depgraph.r}, lines 165, 173 to 175, 178, 179 and 182 (line 165 is broken here to fit the page):\n\\[\n\\begin{aligned}\n&\\texttt{out = Rglpk\\_solve\\_LP(cvec, Amat, dir, bvec, bounds = bound,}\\\\\n&\\qquad\\texttt{types = var\\_type, max = TRUE)}\\\\\n&\\texttt{group\\_1 = i\\_ind[}\\hl{\\texttt{out\\$solution}}\\texttt{!=0]}\\\\\n&\\texttt{group\\_2 = j\\_ind[}\\hl{\\texttt{out\\$solution}}\\texttt{!=0]}\\\\\n&\\texttt{approx\\_sol = }\\hl{\\texttt{out\\$solution}}\\\\\n&\\texttt{group\\_1 = i\\_ind[}\\hl{\\texttt{out\\$solution}}\\texttt{==1]}\\\\\n&\\texttt{group\\_2 = j\\_ind[}\\hl{\\texttt{out\\$solution}}\\texttt{==1]}\\\\\n&\\texttt{obj\\_val = }\\hl{\\texttt{out\\$optimum}}\n\\end{aligned}\n\\]", "corrected": "In the GLPK branch, immediately after line 165 and before \\texttt{out\\$solution} or \\texttt{out\\$optimum} is used, insert the following statement (broken here over three lines).\n\\[\n\\begin{aligned}\n&\\hl{\\texttt{if (!identical(out\\$status, 0L))}}\\\\\n&\\qquad\\hl{\\texttt{stop(sprintf(\"GLPK did not find an optimal solution (status \\%s)\",}}\\\\\n&\\qquad\\qquad\\hl{\\texttt{out\\$status))}}\n\\end{aligned}\n\\]\nUnder the default convention of \\texttt{Rglpk\\_solve\\_LP}, status $0$ means that an optimal solution was found. With this statement in place, both infeasible calls above stop with the message ``GLPK did not find an optimal solution (status 1)'', and a feasible control with $d=(1,1,0)$ and the same known edge returns status $0$ and the same graph and objective as before. This correction does not cover the CPLEX and Gurobi branches, each of which needs its own test of the termination status. If an interrupted maximization is to be used for inference, the variance bound for (3.1) should be computed from a certified upper bound $U$ on the objective of (3.6), as $(n\\hat\\sigma_\\theta^2+2U)/n^2$, and not from the incumbent.", "evidence": "I used the saved output of a GLPK run of the package and exact arithmetic. Let $n=3$ and $d=(0,0,0)$, let \\texttt{GR} contain the single edge $\\{1,2\\}$, and let every objective coefficient equal $1$. Since the constraints $a_{12}=1$ and $a_{12}+a_{13}\\le0$ contradict each other, the integer program and its relaxation are infeasible. GLPK (Rglpk 0.6.5.1, GLPK 5.0) returns status $1$, the solution $(0,0,0)$ and the objective $0$ in both modes. The unmodified \\texttt{depgraph} then returns the zero matrix as \\texttt{A\\_max} and \\texttt{obj\\_val = 0}, although the known edge requires $A_{12}=1$. With $y=(1,1,-2)$ and $\\theta=(1,1,1)$, \\texttt{depvar} returns \\texttt{V\\_hat = 1} in each of the four combinations of case and mode. These inputs lie outside the paper's framework, in which $A^{\\mathcal O}$ is compatible with the data and the feasible set is not empty.\n\nFor the Gurobi branch the evidence is a reading of the code and an exact example of what an incumbent can do. For $n=4$, $y=(1,1,-1,-1)$, $\\theta_i=1$ and $d=(1,1,1,1)$, the zero graph is feasible with objective $0$ and the maximum is $2$. Equation (3.1) gives $1/4$ at the zero graph and $1/2$ at the maximizer. Neither Gurobi nor CPLEX was run. No failed or interrupted solve was observed on data that satisfy the paper's assumptions.", "problem": "After the call to the solver, \\texttt{depgraph} reads the solution vector and the objective value and returns them as the maximizing graph \\texttt{A\\_max} and the optimum \\texttt{obj\\_val}. None of its three branches tests the status that the solver reports. When a solve fails, the result is returned as an ordinary one, with no error and no warning, and \\texttt{depvar} computes a variance estimate from it. The Gurobi branch sets a time limit of 12 hours and reads the returned solution and objective in the same way. If the limit were reached, a returned incumbent would be only a lower bound on the maximum, and it need not dominate the oracle estimator.", "what_survives": "The results of the paper concern the maximal compatible estimator or an upper bound on it, and they are not affected. Solves that succeed, the bound from a solved relaxation and the closed form (3.9) are unaffected. No number in the article is involved, and no value of the vignette is shown to come from a failed solve. The vignette raises each degree to at least the number of known edges at the vertex, so the contradictory input above cannot arise there. What changes is that a failed solve must stop with an error."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E014", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first three lines of Section 3.1 and display (3.1). A red box marks the factor n to the power minus 1 in 'plug-in sample variance sigma-hat-theta squared = n^-1 times the sum over i in V_S of (theta_i Y_i - beta-hat) squared'.", "caption": "Section 3.1, p. 2242 (PDF p. 5). The marked factor is the divisor $n$ in the definition of the plug-in sample variance, which enters (3.1).", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E015.png"}, {"alt": "Listing in monospaced type with line numbers, showing the helper functions V1, V2 and V2p. Red boxes mark the call 'var(x)' at the start of the expression assigned to V1_hat on line 11, of the expression 'var(x)/n * (1 + sum(A)/n)' on line 19, and of the expression 'var(x)/n * (1 + sum(pmin(d, n-1))/n)' on line 27.", "caption": "Package depinf, file R/depvar.r, lines 7–29, typeset from the file with line numbers added. Each of the three helpers calls var(x), which divides by $n-1$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E015-b.png"}, {"alt": "Crop of the first six lines of Appendix B. A red box marks the clause 'depvar for calculating the variance estimates (3.1) and (3.7).'", "caption": "Appendix B, p. 2250 (PDF p. 13). The marked clause says that depvar calculates the variance estimates (3.1) and (3.7).", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-E015-c.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T25", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T25"}, {"id": "P011-T26", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T26"}, {"id": "P011-T35", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T35"}], "headline": "The package’s variance helpers use divisor $n-1$ where the paper defines the sample variance with divisor $n$, so their values can exceed (3.1), (3.7) and (3.9).", "id": "P011-E015", "location": "Section 3.1, p. 2242 (PDF p. 5), the definition of $\\hat\\sigma_\\theta^2$ before (3.1); Appendix B, p. 2250 (PDF p. 13); R package depinf, R/depvar.r, lines 11, 19 and 27", "pdf_page": 5, "printed_page": "2242", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P011-T25", "P011-T26", "P011-T35"], "text": {"as_printed": "P.~2242: ``Define the plug-in sample variance $\\hat\\sigma_\\theta^2=n^{-1}\\sum_{i\\in\\mathcal V_S}(\\theta_iY_i-\\hat\\beta)^2$''. Appendix B, p.~2250: ``\\texttt{depvar} for calculating the variance estimates (3.1) and (3.7)''.\n\n\\texttt{R/depvar.r}, line 11: \\texttt{V1\\_hat = as.numeric( \\hl{var(x)}/n + (t(x-mx) \\%*\\% A \\%*\\% (x-mx))/(}$\\texttt{n\\textasciicircum 2}$\\texttt{) )}\n\nLine 19: \\texttt{V2\\_hat = as.numeric( \\hl{var(x)}/n * (1 + sum(A)/n) )}\n\nLine 27: \\texttt{V2\\_hat = as.numeric( \\hl{var(x)}/n * (1 + sum(pmin(d, n-1))/n) )}", "corrected": "In the helpers \\texttt{V1}, \\texttt{V2} and \\texttt{V2p} of \\texttt{R/depvar.r}, replace each \\texttt{var(x)} by $\\hl{\\texttt{mean((x-mean(x))\\textasciicircum 2)}}$. At a fixed $A$, the value of $\\widehat V_1$ then decreases by $S/[n^2(n-1)]$, and the values of $\\widehat V_2$ and $\\widehat V_2'$ are multiplied by $(n-1)/n$.\nMake the same replacement in the standalone closed-form calculation in \\texttt{inst/doc/replication.R}, line 126.", "evidence": "I base this correction on this derivation and an exact example, run with the package. For $x=(0,1,2)$, $\\theta=(1,1,1)$, $d=(0,0,0)$ and $A=0$, the value of (3.1) is $2/9$ and the package returns $1/3$.", "problem": "For complete data with $n>1$ and a degree vector of length $n$, the function \\texttt{var} of R divides by $n-1$. Let $x_i=\\theta_iY_i$ and $S=\\sum_i(x_i-\\bar x)^2$. At a fixed $A$, the value of $\\widehat V_1$ in the package exceeds (3.1) by $S/[n^2(n-1)]$, and the cross-product term is the same. The values of $\\widehat V_2$ and $\\widehat V_2'$ in the package equal (3.7) and (3.9) multiplied by $n/(n-1)$.", "what_survives": "The difference is upward. Under Assumption 2 the $\\widehat V_1$ difference tends to zero after multiplication by $n$; for $\\widehat V_2$ and $\\widehat V_2'$, the factor $n/(n-1)$ tends to one and preserves any otherwise valid asymptotic conservativeness guarantee. The maximizing graphs are the same, since $\\hat\\sigma_\\theta^2$ does not depend on $A$. No statement of the paper changes, and the article contains no numerical simulation or application table. I have not recomputed the standard errors of the vignette. The change of divisor does not repair \\fid{P011-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E015", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Assumption 2 (Asymptotic scaling), eight lines. A red box marks 'Z = (G_R, Y, d)' in the second line, 'graphs G and observed data Z = (G_R, Y, d), where G_R = (V_S, E_R)'.", "caption": "Assumption 2, p. 2242 (PDF p. 5). The marked list of the observed data omits $\\theta$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N001.png"}, {"alt": "Crop of display (3.5), 'maximize (Theta Y - beta-hat)' A (Theta Y - beta-hat) subject to A1 precedes-or-equals d, A succeeds-or-equals A_R', and the two lines after it. Red boxes mark the hyphenated word 'less-than', in quotation marks, which is split over two lines.", "caption": "Program (3.5) and the sentence after it, p. 2244 (PDF p. 7). The marked word describes the relation $\\preceq$ of the constraint $A\\mathbf{1}\\preceq d$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N001-b.png"}, {"alt": "Crop of the line 'Assumption 3 (Homoskedasticity). var(theta_i Y_i) = var(theta_j Y_j), for all i, j in V.' and the two lines after it. A red box marks the calligraphic V at the end of the assumption.", "caption": "Assumption 3, p. 2246 (PDF p. 9). The marked set is the full vertex set $\\mathcal V$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N001-c.png"}, {"alt": "Crop of three lines. A red box marks 'E_S', with an upright E, in 'of edges that can be in E_S. For i in V_S, let d-prime_i = min{d_i, n - 1} be the degree of'.", "caption": "Last paragraph of p. 2246 (PDF p. 9). The marked symbol is the edge set, printed with a roman E.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N001-d.png"}, {"alt": "Crop of the first seven lines of the Discussion. Red boxes mark 'Z =' at the end of one line and '(Y, d, G_R),' at the start of the next, in 'Assumption 1, which states that we observe Z = (Y, d, G_R), can be weakened'.", "caption": "Discussion, p. 2248 (PDF p. 11), first paragraph. The marked restatement of Assumption 1 omits $\\theta$.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N001-e.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T24", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T24"}, {"id": "P011-T36", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T36"}], "headline": "Five slips of notation: $\\theta$ is missing from two restatements of the observed data $Z$, “less-than” should read “less-than-or-equal-to”, Assumption 3 should quantify over $\\mathcal V_S$, and the edge set $\\mathcal E_S$ is printed once with a roman E.", "id": "P011-N001", "location": "Assumption 2, p. 2242 (PDF p. 5); the sentence after (3.5), p. 2244 (PDF p. 7); Assumption 3 and the last paragraph of p. 2246 (PDF p. 9); Discussion, p. 2248 (PDF p. 11)", "pdf_page": 5, "printed_page": "2242", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P011-T24", "P011-T36"], "text": {"as_printed": "\\begin{enumerate}\n\\item P.~2242, Assumption 2: ``observed data $\\hl{Z=(\\mathcal G_R,Y,d)}$''.\n\\item P.~2248, Discussion: ``Assumption 1, which states that we observe $\\hl{Z=(Y,d,\\mathcal G_R)}$''.\n\\item P.~2244: ``$\\preceq$ denotes the element-wise ``\\hl{less-than}'' relation''.\n\\item P.~2246, Assumption 3: ``$\\operatorname{var}(\\theta_iY_i)=\\operatorname{var}(\\theta_jY_j),\\ \\forall i,j\\in\\hl{\\mathcal V}$''.\n\\item P.~2246, last paragraph: ``the maximum number of edges that can be in $\\hl{\\mathrm{E}_S}$''.\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item Assumption 2: ``observed data $Z=(\\hl{Y,d,\\theta,\\mathcal G_R})$''.\n\\item Discussion: ``Assumption 1, which states that we observe $Z=(\\hl{Y,d,\\theta,\\mathcal G_R})$''.\n\\item ``$\\preceq$ denotes the element-wise ``\\hl{less-than-or-equal-to}'' relation''.\n\\item Assumption 3: ``$\\operatorname{var}(\\theta_iY_i)=\\operatorname{var}(\\theta_jY_j),\\ \\forall i,j\\in\\hl{\\mathcal V_S}$''.\n\\item ``the maximum number of edges that can be in $\\hl{\\mathcal E_S}$''.\n\\end{enumerate}", "evidence": "I base these corrections on a check of the pages against the definitions of Section 2. Four further points of notation were examined and are not defects: the two definitions of $\\Theta$ (pp.~2241 and 2244) have separate scopes, the upper limit $n$ on the sums of p.~2241 is redundant, the scalar $\\hat\\beta$ in $\\Theta Y-\\hat\\beta$ is subtracted from each entry, and $A^m$ in Lemma 1 is the maximizer for $\\widehat V_2$.", "problem": "By Assumption 1 and the sentence after it (p.~2240), I have $Z=(Y,d,\\theta,\\mathcal G_R)$, and the coefficients $\\theta$ are observed. The constraints $A\\mathbf{1}\\preceq d$ and $A\\succeq A_R$ are not strict, since Definition 4 requires $\\sum_jA_{ij}\\le d_i$. Section 2 defines $\\theta_1,\\ldots,\\theta_n$ for the observed vertices only, and $N$ may exceed $n$, so that $\\theta_iY_i$ is undefined for $i\\in\\mathcal V\\setminus\\mathcal V_S$. Every sum in (3.7), (3.8) and (3.9) and in the proof of Lemma 1 runs over $\\mathcal V_S$. Definition 2 writes the edge set of $\\mathcal G_S$ as $\\mathcal E_S$, and the roman E denotes expectation elsewhere.", "what_survives": "No result changes. Each item is corrected from the definitions nearby."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-N001", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of eight lines of text. A red box marks the number 12 at the start of a line, which completes '(see, e.g., section 9.4.2 of Kellerer et al 12)'. The second line of the crop ends 'See chapter 9 of Kellerer et al 13 for'.", "caption": "P. 2245 (PDF p. 8). The marked reference number follows “section 9.4.2 of Kellerer et al” at the end of the line above; chapter 9 of the same book is cited as 13 four lines earlier.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N002.png"}, {"alt": "Crop of three lines. A red box marks 'et al' in 'has been proven by, e.g., Theorem 2.7 of Chen et al [6].'", "caption": "P. 2247 (PDF p. 10), after Lemma 2. The marked words follow the name of the first of two authors.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N002-b.png"}, {"alt": "Crop of reference entries [5] and [6]. A red box marks 'et al' in '[6] Chen LH, Shao QM, et al (2004) Normal approximation under local dependence. The Annals of Probability 32(3):1985-2028'.", "caption": "References, p. 2251 (PDF p. 14), entry [6].", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N002-c.png"}, {"alt": "Crop of reference entries [18] and [19]. A red box marks 'et al' in '[19] Ogburn EL, VanderWeele TJ, et al (2017) Vaccines, contagion, and social networks. The Annals of Applied Statistics 11(2):919-948'.", "caption": "References, p. 2251 (PDF p. 14), entry [19].", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N002-d.png"}], "formal_release": null, "formal_statements": [], "headline": "Section 9.4.2 of Kellerer, Pferschy and Pisinger belongs to reference [13] and not to [12], and “et al” is spurious for the two-author works [6] and [19].", "id": "P011-N002", "location": "P. 2245 (PDF p. 8), “section 9.4.2 of Kellerer et al 12”; p. 2247 (PDF p. 10), “Theorem 2.7 of Chen et al [6]”; references [6] and [19], p. 2251 (PDF p. 14)", "pdf_page": 8, "printed_page": "2245", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{enumerate}\n\\item P.~2245: ``(see, e.g., section 9.4.2 of Kellerer et al \\hl{12})''.\n\\item P.~2247: ``Theorem 2.7 of Chen \\hl{et al} [6]''.\n\\item P.~2251: ``[6] Chen LH, Shao QM, \\hl{et al} (2004) Normal approximation under local dependence. The Annals of Probability 32(3):1985--2028''.\n\\item P.~2251: ``[19] Ogburn EL, VanderWeele TJ, \\hl{et al} (2017) Vaccines, contagion, and social networks. The Annals of Applied Statistics 11(2):919--948''.\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item ``(see, e.g., section 9.4.2 of Kellerer et al \\hl{13})''.\n\\item ``Theorem 2.7 of Chen \\hl{and Shao} [6]''.\n\\item ``[6] Chen LH, Shao QM (2004) Normal approximation under local dependence. The Annals of Probability 32(3):1985--2028''.\n\\item ``[19] Ogburn EL, VanderWeele TJ (2017) Vaccines, contagion, and social networks. The Annals of Applied Statistics 11(2):919--948''.\n\\end{enumerate}\n\nIf reference [12] is kept for its other use on p.~2245, its entry should identify it as Appendix A of \\emph{Knapsack Problems}, pp.~483--493.", "evidence": "I base these corrections on a check of the pages against the published sources. These are the table of contents of the first edition of the book (2004), the byline of the article of Chen and Shao, and the bibliographic record of the article of Ogburn and VanderWeele. Theorem 2.7 of Chen and Shao is a normal-approximation bound for sums of random variables indexed by the vertices of a dependency graph. This is the result that the paper cites for Lemma 2.", "problem": "Reference [12], ``Introduction to NP-Completeness of knapsack problems'', is Appendix A (pp.~483--493) of the book \\emph{Knapsack Problems}, which is reference [13]. Chapter 9 of the book treats multidimensional knapsack problems, and its section 9.4.2 treats polynomial time approximation schemes. The paper cites chapter 9 as [13] four lines earlier. Chen and Shao (2004) and Ogburn and VanderWeele (2017) each have two authors.", "what_survives": "Every cited work is identifiable, and the use of Theorem 2.7 for Lemma 2 is unaffected. Only the citations change."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-N002", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the Acknowledgement section. Red boxes mark the sentence 'We are grateful to Robert Heimer for helpful comments and for providing the SATHCAP data, funded by NIH/NIDA grant U01 DA017387', which runs over three lines.", "caption": "Acknowledgement, p. 2249 (PDF p. 12). The marked sentence is the only mention of the SATHCAP data in the published article, and it is correct.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N006.png"}, {"alt": "Listing of lines 13 to 15 of the file replication.Rmd. Line 13 is the title of the vignette. A red box marks line 15: 'In this vignette, we replicate the main analyses in Section 5 of the paper.'", "caption": "vignettes/replication.Rmd, lines 13–15, typeset from the package with line numbers added. The marked sentence refers to a Section 5 with analyses; Section 5 of the published article is the Discussion.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N006-b.png"}, {"alt": "Listing of lines 100 to 105 of the file replication.Rmd. Line 100 is the comment '# V1, some known dependencies'. A red box marks lines 101 to 104, which assign b1 to b4 as the square root of the V_hat returned by depvar with arguments Y, a row of Theta times n, d, GR, case = \"heteroskedastic\", solver = \"gurobi\" and approximate = 0. Line 105 collects them as col5.", "caption": "vignettes/replication.Rmd, lines 100–105, typeset from the package with line numbers added. The four marked calls compute $\\widehat V_1$ from the raw outcomes Y, the least-squares weights and the known edges GR.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N006-c.png"}, {"alt": "Listing of line 3 of the file README.md, on four rows. Red boxes mark the sentence 'The primary function, depvar, produces conservative variance estimates for generic linear unbiased estimators'.", "caption": "README.md, line 3, typeset from the package with the line number added and the long line continued on three further rows. The marked sentence describes the output of depvar as conservative for generic linear unbiased estimators.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N006-d.png"}], "formal_release": null, "formal_statements": [{"id": "P011-T15", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T15"}, {"id": "P011-T16", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T16"}, {"id": "P011-T35", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-T35"}], "headline": "The package vignette applies the general estimator $\\widehat V_1$, with raw outcomes and known recruitment edges, to four least-squares coefficients, and the package describes the result as conservative; the paper’s assumptions alone do not justify that guarantee for these calls, and the analyses in “Section 5 of the paper” that the vignette says it replicates are not in the published article.", "id": "P011-N006", "location": "Acknowledgement, p. 2249 (PDF p. 12); Section 5, p. 2248 (PDF p. 11); equation (3.1) and Proposition 1, p. 2242 (PDF p. 5). Package depinf: vignettes/replication.Rmd, lines 15 and 100–105; README.md, line 3", "pdf_page": 12, "printed_page": "2249", "reaches": [{"effect": "result_needs_same_repair", "effect_text": "a result or code path there needs the same repair", "explanation": null, "package": "depinf", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P011"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P011-E001", "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-E001", "work_id": "P011"}], "severity": "Moderate", "subject": "paper", "target_ids": ["P011-T15", "P011-T16", "P011-T35"], "text": {"as_printed": "Acknowledgement, p.~2249: ``We are grateful to Robert Heimer for helpful comments and for providing the SATHCAP data, funded by NIH/NIDA grant U01 DA017387.''\n\n\\texttt{vignettes/replication.Rmd}, line 15: ``In this vignette, we replicate the main analyses in \\hl{Section 5 of the paper}.''\n\n\\texttt{vignettes/replication.Rmd}, lines 100 and 101 (line 101 is broken here to fit the page); lines 102 to 104 repeat the call for coefficients 2 to 4, and line 105 collects the four values as \\texttt{col5}:\n\\[\n\\begin{aligned}\n&\\texttt{\\# V1, some known dependencies}\\\\\n&\\texttt{b1 = sqrt(depvar(}\\hl{\\texttt{Y}}\\texttt{, Theta[1, ]*n, d, GR, case = \"heteroskedastic\",}\\\\\n&\\qquad\\texttt{solver = \"gurobi\", approximate = 0)\\$V\\_hat)}\n\\end{aligned}\n\\]\n\n\\texttt{README.md}, line 3: ``The primary function, depvar, produces \\hl{conservative} variance estimates for \\hl{generic} linear unbiased estimators''", "corrected": "Vignette, line 15: ``\\hl{This vignette illustrates the method with the SATHCAP data. The analysis is not included in the published article.}''\n\nREADME, line 3: ``The primary function, depvar, produces variance estimates for linear unbiased estimators. \\hl{The general estimator is conservative under an additional condition on the means or known graph, with the input restrictions stated below; the homoskedastic estimators require Assumptions 2 and 3}.''\n\nThe condition is the following. Let $h_i=E[\\theta_iY_i]-\\beta$ for $i\\in\\mathcal V_S$. If Assumption 2 holds and\n\\[\n\\hl{\\frac1n\\,h'(I+A^{\\mathcal O})h\\to0},\n\\]\nthen $n\\widehat V_1(A^{\\mathcal O};Z)-n\\operatorname{var}(\\hat\\beta)\\to0$ in probability. A sufficient condition is $E[\\theta_iY_i]=\\beta$ for every observed unit. If only $\\liminf_{n\\to\\infty}n^{-1}h'(I+A^{\\mathcal O})h\\ge0$ holds, then the oracle estimator, and with it the maximal compatible estimator, is asymptotically conservative. Unbiasedness of $\\hat\\beta$ does not imply either condition, and least-squares coefficients do not satisfy them in general. For the maximized estimator alone, condition (C) of \\fid{P011-E001} is another sufficient route. It uses the known-edge graph and remaining degree capacity and includes cases with $o(n)$ known edges, a known graph made into disjoint cliques by $o(n)$ deletions, enough unused capacity, or a compatible mean space. It does not imply consistency of the original raw-outcome oracle. The fixed positive semidefinite corrections in that finding provide further alternatives after maximization.\n\nFor least-squares coefficients there is an alternative that keeps the estimand. Replace the weighted outcomes in (3.1) and (3.5) by the weighted residuals $q_i=\\theta_i(Y_i-x_i'\\hat b)$, where $\\hat b$ is the least-squares fit, and maximize $n^{-2}q'(I+A)q$ over the compatible graphs. A derivation shows that the oracle version is consistent if, in addition to Assumption 2, the linear mean model is correct, the number of covariates is fixed, the \\hl{unweighted outcomes,} rows of $X$ and weights are uniformly bounded, and the smallest eigenvalue of $X'X/n$ is bounded away from zero. With the package's $n-1$ variance denominator, the relaxed standard errors on the vignette's data become $0.2728$, $0.1461$, $0.0093$ and $0.1521$ for the intercept, hcv, age and sex, against $0.3258$, $0.1595$, $0.0106$ and $0.2057$ from the raw outcomes. These numbers illustrate the alternative. They do not show that its conditions hold for these data.\n\nA further route fits the means for the variance calculation by $\\theta_i^2$-weighted least squares and uses $q_i=\\theta_i(Y_i-x_i^{\\mathsf T}\\hat b_{\\mathrm w})$. The point estimator remains the original least-squares coefficient. \\hl{Under Assumption 2,} if $W$ has rows $\\theta_i x_i^{\\mathsf T}$, $\\operatorname{rank}(W)=o(n)$, and the mean model is correct or\n\\[\n\\inf_b\\frac1n\\sum_i\\theta_i^2\n\\{\\mathbb{E}[Y_i]-x_i^{\\mathsf T}b\\}^2\\to0,\n\\]\nthen the average squared transformed prediction error tends to zero \\hl{in probability}. As derived in \\fid{P011-E001}, this gives consistency of the uncorrected true-graph residual oracle and conservativeness of its maximum. No separate root-$n$ coefficient rate or bounded unweighted outcomes are needed for this weighted route. The stronger assumptions of the ordinary least-squares route above remain sufficient. Neither route has been shown to hold for the vignette's data. The illustrated standard errors use the ordinary fit above.", "evidence": "I base this correction on an exact counterexample, a derivation and a recalculation.\n\nFor the expansion, let $u_i=\\theta_iY_i-E[\\theta_iY_i]$, $\\delta=\\hat\\beta-\\beta$ and $B=I+A^{\\mathcal O}$, so that $n\\widehat V_1(A^{\\mathcal O};Z)=n^{-1}(u+h-\\delta\\mathbf{1})'B(u+h-\\delta\\mathbf{1})$. Since the weighted outcomes and the degrees are bounded, I have $\\operatorname{var}(\\delta)=O(n^{-1})$, and replacing $\\delta$ by $0$ changes the expression by $o_p(1)$. Next, $n^{-1}E[u'Bu]=n\\operatorname{var}(\\hat\\beta)$, because outcomes that are not adjacent are independent. The cross term has mean zero, $E[u'Bh]=0$. The terms $n^{-1}u'Bu$ and $n^{-1}u'Bh$ have variance $O(n^{-1})$. By Chebyshev's inequality, I obtain the expansion.\n\nFor the counterexample, let $n=4k$ and take $k$ independent blocks. Each block is a star with a center and three leaves. All three edges are known and the degrees are $(3,1,1,1)$, which leaves the true graph as the only compatible one. Within a block let the covariate be $x=(3,-1,-1,-1)$, let the outcomes $L_1,L_2,L_3$ of the leaves be independent with $\\Pr(L_j=1)=1/4$ and $\\Pr(L_j=0)=3/4$, and let the outcome of the center equal $1$ with conditional probability $3/4+(1/12)\\sum_{j=1}^3(L_j-1/4)$ and $0$ otherwise. Then $E[Y_i]=3/8+x_i/8$, that is, the linear model is correct. The least-squares intercept is the sample mean, with $\\theta_i=1$ and $\\beta=3/8$. Assumption 2 holds with $c_1=1$, $c_2=3$, $c_3=27/128$ and $c_4=2$. In each block $h=(3/8,-1/8,-1/8,-1/8)$ and $h'(I+A^{\\mathcal O})h=-3/32$, that is, $-3/128$ per observation. A computation with the 16 outcomes of a block in exact arithmetic gives $n\\operatorname{var}(\\hat\\beta)=27/128$ and $n\\widehat V_1\\to3/16$ in probability, for the oracle estimator, the maximal compatible estimator and the relaxed estimator alike. For $\\epsilon=3/256$, the probability in Corollary 1 tends to $1$ and not to $0$. The limiting coverage of the nominal $100(1-\\alpha)\\%$ interval is $2\\Phi(z_{1-\\alpha/2}\\sqrt{8/9})-1$, which is below $1-\\alpha$. A second design shows that the bias can be positive.\n\\begin{audittable}{lrrr}{Exact limits for the least-squares intercept in the two block designs.}\nCovariate in a block & $n\\operatorname{var}(\\hat\\beta)$ & Limit of $n\\widehat V_1$ & Ratio \\\\\n\\hline\n$x=(3,-1,-1,-1)$ & $27/128$ & $3/16$ & $8/9$ \\\\\n$x=(0,1,-1,0)$ & $47/192$ & $53/192$ & $53/47$ \\\\\n\\end{audittable}\nFor the recalculation, I used the package's data and functions, with the linear relaxation solved by GLPK and the 468 known edges imposed. The standard error of the hcv coefficient from $\\widehat V_1$ changes as follows when a constant is added to every outcome, while the estimate and the residuals do not change.\n\\begin{audittable}{rrr}{The hcv coefficient in the vignette's regression, with a constant added to every outcome.}\nConstant added & Estimate & Standard error from $\\widehat V_1$ \\\\\n\\hline\n$0$ & $0.3809$ & $0.1595$ \\\\\n$1$ & $0.3809$ & $0.5025$ \\\\\n$10$ & $0.3809$ & $3.8700$ \\\\\n\\end{audittable}\nThe vignette's table was computed with Gurobi in integer mode, which was not available. Its printed value for this coefficient is $0.15951663$. Sensitivity to a shift is a symptom and not a refutation of coverage. The recalculation does not show whether the vignette's standard errors are too large or too small, since that depends on the unknown means and on the unknown dependency graph.", "problem": "The published article has no empirical section. Its Section 5 is the Discussion, and the SATHCAP data appear only in the Acknowledgement, which is correct as it stands. The analysis of these data is in the vignette of the package that Appendix B names as the implementation of the method. The package is dated 20 January 2017 and cites the 2017 preprint. The words ``Section 5 of the paper'' may refer to an earlier version, which I could not obtain.\n\nThe vignette regresses HIV status on three covariates for $n=522$ respondents. It computes standard errors for the four coefficients from $\\widehat V_1$ with the 468 known recruitment edges, and it passes the raw outcomes $Y_i$ with the weights $\\theta_i=n[(X'X)^{-1}X']_{ji}$. Known edges and unequal transformed means are allowed here, so the paper's unqualified guarantee for $\\widehat V_1$ is not available (\\fid{P011-E001}). For a regression coefficient the means $E[\\theta_iY_i]$ can differ across units. Let $h_i=E[\\theta_iY_i]-\\beta$. Under Assumption 2 the oracle estimator satisfies\n\\[\nn\\widehat V_1(A^{\\mathcal O};Z)=n\\operatorname{var}(\\hat\\beta)+\\frac1n\\,h'(I+A^{\\mathcal O})h+o_p(1),\n\\]\nand the quadratic form can have either sign. If the added term has a negative limit and the known edges force a unique graph, the maximal compatible estimator is too small as well. The standard errors that the vignette computes in this way therefore carry no guarantee from the paper's assumptions alone, and the README's unrestricted description does not hold for $\\widehat V_1$. The broader condition (C) in \\fid{P011-E001} supplies additional ways to justify the original maximized estimator, but its graph, capacity or mean restrictions have not been established for this application. One symptom is that these standard errors change when a constant is added to every outcome, although the slope estimates and the residuals do not.", "what_survives": "The Acknowledgement is correct, and no number in the published article is involved. The least-squares estimates and their unbiasedness under the linear mean model, the definition of compatible graphs, the domination of the oracle estimator by the maximal compatible one, and program (3.5) with its relaxation are unaffected. The homoskedastic columns of the vignette rest on Assumption 3 and Corollary 2. This counterexample does not touch them, and they were not examined here. What changes is the status of the standard errors that the vignette computes from $\\widehat V_1$ with the known edges, namely its column \\texttt{col5} and the $V_1$ rows of its sensitivity table, and the description of \\texttt{depvar} in the README."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-N006", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading 'Appendix A: Proofs', the first two lines of the first proof and the first line of display (A.1). A red box marks the heading 'Proof of 1.'", "caption": "Appendix A, p. 2249 (PDF p. 12). The marked heading gives the number of the result without its name.", "url": "https://pmaronow.github.io/llm-audit/works/P011/exhibits/P011-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The heading “Proof of 1.” should read “Proof of Corollary 1.”", "id": "P011-N007", "location": "Appendix A, p. 2249 (PDF p. 12), first proof heading", "pdf_page": 12, "printed_page": "2249", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\hl{\\emph{Proof of 1.}} Across all sample realizations, $\\widehat V_1(A^m;Z)\\ge\\widehat V_1(A^{\\mathcal O};Z)$. Then by Proposition 1,", "corrected": "\\hl{\\emph{Proof of Corollary 1.}} Across all sample realizations, $\\widehat V_1(A^m;Z)\\ge\\widehat V_1(A^{\\mathcal O};Z)$. Then by Proposition 1,", "evidence": "I compared the page with Corollary 1 on p.~2245 and with Proposition 1 and its proof on pp.~2242--2244.", "problem": "The heading omits the name of the result. The proof is that of Corollary 1. On p.~2245 the corollary is followed by the words ``Proof is given in the Appendix'', the conclusion of display (A.1) is the statement of Corollary 1, and Proposition 1 has its proof in the main text.", "what_survives": "Nothing else changes. Whether the argument of (A.1) succeeds depends on Proposition 1, which is the subject of \\fid{P011-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P011/#P011-N007", "work": {"arxiv_id": "1602.00359", "doi": "10.1214/18-ejs1448", "title": "Confidence Intervals for Linear Unbiased Estimators under Constrained Dependence", "url": "https://pmaronow.github.io/llm-audit/works/P011/"}, "work_id": "P011"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of lines 12 to 23 of a numbered listing. Red boxes mark line 13, if j in V_R then; line 21, end if; and line 22, Add {j, u} to E*_SU, which is indented at the level of line 13 and comes after the end if.", "caption": "Web Appendix 2, first proposal mechanism, lines 12–23, Web Appendix p. 3 (supplement PDF p. 3). Marked: the conditional opened at line 13, its end at line 21, and line 22, which follows the end.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E001.png"}, {"alt": "Screenshot of the end of the first listing and the two-line second listing. Red boxes mark line 27, Remove any isolated vertices from V*_SU, and line 1 of the second listing, Choose u in the set of unsampled vertices.", "caption": "Web Appendix 2, line 27 and the second proposal mechanism, Web Appendix p. 3 (supplement PDF p. 3). Marked: the removal of “any isolated vertices” and the choice of an unsampled vertex.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E001-b.png"}, {"alt": "Three small graph drawings. Left: sampled vertices 1 and 2 joined by an arrow, and unsampled vertex 3 joined to vertex 1. Center: vertices 1 and 2 as before, a new unsampled vertex 4 joined to vertex 1, and a dashed orange edge between the unsampled vertices 3 and 4; vertex 3 has no edge to a sampled vertex. Right: vertices 1, 2 and 4 only, with vertex 4 joined to vertex 1.", "caption": "The counterexample. Left: a compatible state. Center: the output of the listing as printed for $i=1$, $j=3$, with the edge $\\{3,4\\}$ between two unsampled vertices. Right: the output of the same choices with line 22 moved inside the conditional of line 13.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E001-counterexample.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T1", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T1"}, {"id": "P012-T2", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T2"}, {"id": "P012-T19", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T19"}, {"id": "P012-T24", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T24"}], "headline": "Line 22 of the edge proposal lies outside the conditional opened at line 13, so the listing as printed can return a graph outside the compatible set; moving line 22 inside that conditional, with two boundary rules, repairs it.", "id": "P012-E001", "location": "Web Appendix 2, first proposal mechanism, lines 13–22 and 27, and second proposal mechanism, Web Appendix p. 3 (supplement PDF p. 3); the listing begins on Web Appendix p. 2", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P012-T1", "P012-T19", "P012-T2", "P012-T24"], "text": {"as_printed": "\\begin{audittable}{rl}{Web Appendix 2, first proposal mechanism, line 2 on Web Appendix p.~2, lines 3--4, 12--23 and 27 on p.~3, as printed (indentation as in the listing).}\nLine & Statement \\\\\n2: & Randomly choose $i\\in V_R$ and $j\\in V_{SU}$ with $i\\neq j$. \\\\\n3: & \\textbf{if} $\\{i,j\\}\\in E_{SU}$ and $\\{i,j\\}\\notin E_R$ \\textbf{then} \\\\\n4: & ~~~Remove $\\{i,j\\}$ from $E^*_{SU}$ \\\\\n12: & ~~~Add $\\{i,u\\}$ to $E^*_{SU}$ \\\\\n13: & ~~~\\textbf{if} $j\\in V_R$ \\textbf{then} \\\\\n14: & ~~~~~~$B\\sim\\operatorname{Bernoulli}(1/2)$ \\\\\n15: & ~~~~~~\\textbf{if} $B<0.5$ and $U_{-j}\\neq\\emptyset$ \\textbf{then} \\\\\n16: & ~~~~~~~~~Randomly choose $u\\in U_{-j}$ \\\\\n17: & ~~~~~~\\textbf{else} \\\\\n18: & ~~~~~~~~~Add a new vertex $u$ to $V^*_{SU}$ \\\\\n19: & ~~~~~~~~~Randomly choose a trait $Z^*_u\\in\\{0,1\\}$ \\\\\n20: & ~~~~~~\\textbf{end if} \\\\\n21: & ~~~\\textbf{end if} \\\\\n22: & ~~~\\hl{Add $\\{j,u\\}$ to $E^*_{SU}$} \\\\\n23: & \\textbf{else if} $\\{i,j\\}\\notin E_{SU}$ and $\\exists u_1,u_2\\in U:\\{i,u_1\\}\\in E_{SU}$ and $\\{j,u_2\\}\\in E_{SU}$ \\textbf{then} \\\\\n27: & Remove \\hl{any isolated vertices} from $V^*_{SU}$ \\\\\n\\end{audittable}\nSecond proposal mechanism: 1: \\hl{Choose $u\\in\\{u\\in V_{SU}:u\\notin V_R\\}$}. 2: Set $Z^*_u=1-Z_u$.", "corrected": "\\begin{audittable}{rl}{Web Appendix 2, first proposal mechanism, lines 13--22 and 27, corrected.}\nLine & Statement \\\\\n13: & ~~~\\textbf{if} $j\\in V_R$ \\textbf{then} \\\\\n14: & ~~~~~~$B\\sim\\operatorname{Bernoulli}(1/2)$ \\\\\n15: & ~~~~~~\\textbf{if} $B<0.5$ and $U_{-j}\\neq\\emptyset$ \\textbf{then} \\\\\n16: & ~~~~~~~~~Randomly choose $u\\in U_{-j}$ \\\\\n17: & ~~~~~~\\textbf{else} \\\\\n18: & ~~~~~~~~~Add a new vertex $u$ to $V^*_{SU}$ \\\\\n19: & ~~~~~~~~~Randomly choose a trait $Z^*_u\\in\\{0,1\\}$ \\\\\n20: & ~~~~~~\\textbf{end if} \\\\\n21: & ~~~~~~\\hl{Add $\\{j,u\\}$ to $E^*_{SU}$} \\\\\n22: & ~~~\\hl{\\textbf{end if}} \\\\\n27: & Remove any isolated \\hl{unsampled} vertices from $V^*_{SU}$ \\\\\n\\end{audittable}\nSecond proposal mechanism: 1: \\hl{If there is no unsampled vertex, return the current state. Otherwise} choose $u\\in\\{u\\in V_{SU}:u\\notin V_R\\}$. 2: Set $Z^*_u=1-Z_u$.", "evidence": "I checked an exact counterexample and a derivation for the corrected listing. Let $V_R=\\{1,2\\}$ with vertex 1 recruiting vertex 2, $\\mathbf d_R=(2,1)$ and $\\mathbf Z_R=(0,1)$. The state with $V_{SU}=\\{1,2,3\\}$, $E_{SU}=\\{\\{1,2\\},\\{1,3\\}\\}$ and $Z_3=0$ is compatible. Let $i=1$ and $j=3$. Line 4 removes $\\{1,3\\}$. Since the set $U_{-1}$ is empty, lines 9 and 10 create a new vertex $u=4$, say with $Z_4=1$, and line 12 adds $\\{1,4\\}$. The condition of line 13 is false. Line 22 adds $\\{3,4\\}$, and line 27 removes nothing. The output has the edges $\\{1,2\\}$, $\\{1,4\\}$ and $\\{3,4\\}$: the edge $\\{3,4\\}$ joins two unsampled vertices, and vertex 3 has no sampled neighbor. With line 22 moved inside the conditional, the same choices produce the vertices $\\{1,2,4\\}$ and the edges $\\{1,2\\}$ and $\\{1,4\\}$, a compatible state. A second example, with three sampled vertices, fails in the same way for every outcome of the coin and of the new trait.\n\nFor the corrected listing every execution returns a compatible pair. If an edge $\\{i,j\\}$ with $j$ unsampled is removed, $i$ loses one edge and gains $\\{i,u\\}$, and $j$ is deleted if it becomes isolated. If $j$ is sampled, each of $i$ and $j$ loses one edge and gains one edge to an unsampled vertex that was not its neighbor. In lines 23 to 25 each of $i$ and $j$ loses one edge to an unsampled vertex and gains $\\{i,j\\}$. In every case each remaining edge has a sampled endpoint. An unsampled vertex then has no sampled neighbor exactly when it is isolated, and line 27 removes it. The trait flip changes the trait of one unsampled vertex only. The sets $U_{-i}$ and $U_{-j}$ of the current state need no update between lines 12 and 16, because the replacement at $i$ does not change which existing unsampled vertices are adjacent to $j$.", "problem": "In the listing the conditional opened at line 13, ``if $j\\in V_R$'', is closed at line 21, and line 22, which adds the edge $\\{j,u\\}$, comes after it. Line 22 runs also when $j$ is an unsampled vertex. In that case $u$ is still the unsampled vertex attached to $i$ at line 12, and line 22 joins two unsampled vertices. The graph $G_{SU}$ has no such edges by definition. The step can also leave $j$ with no sampled neighbor, against condition 3 of Definition 3, and line 27 does not remove $j$, because $j$ is not isolated. The listing is the paper's only specification of the kernel $K$ of Web Appendix 1, which is defined on pairs of compatible states. Proposition 1 therefore does not cover the listing as printed, and bounds computed by an implementation that follows it literally need not be the bounds of Definition 4.\n\nTwo boundary cases also need a rule. Line 27 removes ``any isolated vertices'', which would delete a sampled seed of reported degree zero. The second proposal mechanism has no vertex to choose when the current state has no unsampled vertex. The materials available to me do not include the code used for the Hartford bounds, and I do not know whether it followed the listing.", "what_survives": "The definitions of $h$, $p$, the compatible set and the identification intervals do not depend on the listing, and neither do Results 1--3 or the full-network values of Table 1. I have not established a change to any reported number. Proposition 1 requires a compatible proposal and its other hypotheses (see \\fid{P012-N008}). The corrected listing keeps every proposal inside the compatible set; this alone does not establish convergence, and it does not show what the code behind the intervals of Table 1 did."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-E001", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of the sentence that defines p_1 and p_2 and of display (A2). A red box marks the first equals sign, between p_1 minus p_2 and the bracket that contains the terms of the single recruit j.", "caption": "Appendix, proof of Result 3, display (A2), p. 160 (PDF p. 8), right column. The marked equality holds only when $i$ has a single recruit.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T12", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T12"}, {"id": "P012-T13", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T13"}], "headline": "Display (A2) keeps one recruitment by $i$; when $i$ recruited two or more subjects, $p_1-p_2$ is a sum over all of them, with the same sign, and Result 3 is unchanged.", "id": "P012-E002", "location": "Appendix, proof of Result 3, display (A2), p. 160 (PDF p. 8), right column; Result 3 is stated on p. 155 (PDF p. 3)", "pdf_page": 8, "printed_page": "160", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P012-T12", "P012-T13"], "text": {"as_printed": "\\[\n\\begin{aligned}\np_1-p_2&\\;\\hl{=}\\;\\frac{1}{n-|M|}\\left[\\left(1\\{Z_i=Z_j\\}-\\frac{K^1}{|S_i(t_j)|}\\right)-\\left(1\\{Z_i=Z_j\\}-\\frac{K^1-1}{|S_i(t_j)|}\\right)\\right]\\\\\n&=\\frac{-1}{(n-|M|)|S_i(t_j)|}\\ne0.\\qquad\\text{(A2)}\n\\end{aligned}\n\\]", "corrected": "Suppose, as in the printed proof, that $Z_i=1$, that $u$ is an unsampled vertex adjacent to $i$ and to no other vertex of $V_R$, and that $Z_u=1$ in $\\mathbf Z^1_{SU}$ and $Z_u=0$ in $\\mathbf Z^2_{SU}$. Then\n\\[\np_1-p_2=\\hl{-\\frac{1}{n-|M|}\\sum_{\\ell\\in V_R\\setminus M:\\,r_\\ell=i}\\frac{1}{|S_i(t_\\ell)|}}<0.\\qquad\\text{(A2)}\n\\]\nThe sum is not empty, because $i$ recruited $j$, and each $|S_i(t_\\ell)|$ is a positive integer. Therefore preferential recruitment is not point identified.", "evidence": "The evidence is a derivation and an exact counterexample. Under the printed assumption $Z_i=1$, the graph and the recruitment times are the same in the two pairs. Each set $S_{r_\\ell}(t_\\ell)$ and each indicator $1\\{Z_\\ell=Z_{r_\\ell}\\}$ is then the same in both. If $\\ell$ was not recruited by $i$, then $u\\notin S_{r_\\ell}(t_\\ell)$, because $u$ has no sampled neighbor other than $i$, and the summand of $\\ell$ does not change. If $\\ell$ was recruited by $i$, the number of vertices of $S_i(t_\\ell)$ with the trait of $i$ is smaller by one in the second pair. Subtracting the two averages term by term, I obtain the corrected display below. There exists a compatible pair of the kind the proof needs: it has the recruitment edges and, for each $v\\in V_R$, $d_v-d_v^r$ new unsampled vertices adjacent to $v$ only.\n\nFor the counterexample let $V_R=\\{1,2,3\\}$, let the seed 1 recruit 2 and then 3, let $\\mathbf d_R=(3,1,1)$ and $Z_1=Z_2=Z_3=1$, and let $u=4$ be the one unsampled neighbor of vertex 1. Then $S_1(t_2)=\\{2,3,4\\}$ and $S_1(t_3)=\\{3,4\\}$. With $Z_4=1$, $p_1=0$. With $Z_4=0$,\n\\[\np_2=\\frac12\\Bigl[\\Bigl(1-\\frac23\\Bigr)+\\Bigl(1-\\frac12\\Bigr)\\Bigr]=\\frac5{12},\n\\]\nand I find $p_1-p_2=-5/12$. Display (A2) gives $-1/6$ with $j=2$ and $-1/4$ with $j=3$.", "problem": "The proof defines $p_1$ and $p_2$ as the whole functional of equation (2) and changes the trait of an unsampled neighbor $u$ of $i$ from 1 to 0. Since the vertex $u$ is never recruited, I have $u\\in S_i(t_\\ell)$ for every $\\ell$ that $i$ recruited, and the change of $Z_u$ alters every one of those summands of (2). Display (A2) keeps the summand of the single recruit $j$. Its first equality is therefore false when $i$ has two or more recruits, which Result 3 allows (``at least 1 other vertex''). The omitted terms have the sign of the term kept, and $p_1-p_2$ is still nonzero. The printed minus sign is correct.", "what_survives": "Result 3 holds with its printed hypotheses, and display (A2) is exact when $i$ has exactly one recruit. Equation (2), Figure 1 and the application use the full functional $p$, and no later formula uses the magnitude in (A2). Nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-E002", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of the proof of Result 1. Red boxes mark the words: be any compatible subgraph, which run over a line break.", "caption": "Appendix, proof of Result 1, p. 160 (PDF p. 8), right column. The marked words allow any compatible starting graph that contains $\\{i,u\\}$ and $\\{j,u\\}$.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E003.png"}, {"alt": "Screenshot of the proof of Result 2 up to the words The difference is. Red boxes mark the words: be any compatible subgraph, and the sentence: Clearly (G^2_SU, Z^2_SU) is in C(G_R, d_R, Z_R).", "caption": "Appendix, proof of Result 2, p. 160 (PDF p. 8), right column. The marked words allow any compatible starting pair and assert that the pair obtained by the swap is compatible.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T8", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T8"}, {"id": "P012-T9", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T9"}, {"id": "P012-T10", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T10"}, {"id": "P012-T11", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T11"}], "headline": "The proofs of Results 1 and 2 start from “any” compatible graph with the named edges, for which the described edge moves can fail; both proofs are repaired by starting from a graph built with new unsampled vertices, and the two results stand as printed.", "id": "P012-E003", "location": "Appendix, proofs of Results 1 and 2, p. 160 (PDF p. 8), right column; the results are stated on p. 155 (PDF p. 3)", "pdf_page": 8, "printed_page": "160", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P012-T10", "P012-T11", "P012-T8", "P012-T9"], "text": {"as_printed": "\\emph{Proof of Result 1:} Let $G^1_{SU}=(V^1_{SU},E^1_{SU})$ be \\hl{any compatible subgraph} with $\\{i,u\\}\\in E^1_{SU}$, $\\{j,u\\}\\in E^1_{SU}$, where $u\\notin V_R$ is an unsampled vertex. Let $G^2_{SU}=(V^2_{SU},E^2_{SU})$ be identical to $G^1_{SU}$ except that $\\{i,j\\}\\in E^2_{SU}$, so neither $i$ nor $j$ is connected to $u$.\n\n\\emph{Proof of Result 2:} Let $(G^1_{SU},\\mathbf Z^1_{SU})$ be \\hl{any compatible subgraph and trait set} with the property that $\\{i,u_1\\}\\in E^1_{SU}$ and $\\{j,u_2\\}\\in E^1_{SU}$, where $u_1$ and $u_2$ are unsampled vertices with $Z^1_{u_1}=0$ and $Z^1_{u_2}=1$. Let $(G^2_{SU},\\mathbf Z^2_{SU})$ be identical to $(G^1_{SU},\\mathbf Z^1_{SU})$ except that the edges connecting $i$ and $j$ to $u_1$ and $u_2$ are swapped: $\\{i,u_1\\}\\notin E^2_{SU}$, $\\{j,u_2\\}\\notin E^2_{SU}$, and $\\{i,u_2\\}\\in E^2_{SU}$ and $\\{j,u_1\\}\\in E^2_{SU}$. \\hl{Clearly} $(G^2_{SU},\\mathbf Z^2_{SU})\\in\\mathrm{C}(G_R,\\mathbf d_R,\\mathbf Z_R)$.", "corrected": "Results 1 and 2 are unchanged. In both proofs write $s_v=d_v-d_v^r$ for $v\\in V_R$.\n\n\\emph{Proof of Result 1:} \\hl{Construct $G^1_{SU}$ as follows.} Take the recruitment edges as undirected edges. Add one new unsampled vertex $u$ with the edges $\\{i,u\\}$ and $\\{j,u\\}$. Attach $s_i-1$ further new unsampled vertices to $i$, $s_j-1$ to $j$, and $s_v$ to every other $v\\in V_R$, each adjacent to that one sampled vertex only. Then $G^1_{SU}$ is compatible and \\hl{$\\{i,j\\}\\notin E^1_{SU}$}. Let $G^2_{SU}$ be obtained from $G^1_{SU}$ by removing $\\{i,u\\}$ and $\\{j,u\\}$, adding $\\{i,j\\}$, and deleting $u$. The proof continues as printed.\n\n\\emph{Proof of Result 2:} \\hl{Construct $(G^1_{SU},\\mathbf Z^1_{SU})$ as follows.} Take the recruitment edges as undirected edges and attach $s_v$ new unsampled vertices to each $v\\in V_R$, each adjacent to $v$ only. Let $u_1$ be one of those attached to $i$ and $u_2$ one of those attached to $j$, with $Z^1_{u_1}=0$ and $Z^1_{u_2}=1$; the traits of the other unsampled vertices are arbitrary. Then \\hl{$\\{i,u_2\\}\\notin E^1_{SU}$ and $\\{j,u_1\\}\\notin E^1_{SU}$}. Let $(G^2_{SU},\\mathbf Z^2_{SU})$ be identical to $(G^1_{SU},\\mathbf Z^1_{SU})$ except that $\\{i,u_1\\}$ and $\\{j,u_2\\}$ are replaced by $\\{i,u_2\\}$ and $\\{j,u_1\\}$. The swap preserves every degree, so $(G^2_{SU},\\mathbf Z^2_{SU})\\in\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$, and the proof continues as printed.", "evidence": "The evidence is two exact counterexamples to the printed steps and a derivation of the corrected ones. Let $V_R=\\{1,2,3\\}$ with recruitments $1\\to2\\to3$, reported degrees $\\mathbf d_R=(3,2,3)$ and traits $\\mathbf Z_R=(0,0,1)$, and let $i=1$, $j=3$. I write an edge $\\{k,l\\}$ as $kl$.\n\\begin{itemize}\n\\item Result 1: with $u=4$, the graph with edges $12$, $23$, $13$, $14$, $34$ is compatible and contains $14$ and $34$. The printed step removes $14$ and $34$, adds $13$, which is already present, and deletes vertex 4. The result has edges $12$, $23$, $13$ and sampled degrees $(2,2,2)$ in place of $(3,2,3)$.\n\\item Result 2: with $u_1=4$, $u_2=5$, $Z_4=0$ and $Z_5=1$, the graph with edges $12$, $23$, $14$, $35$, $15$, $34$ is compatible and contains $14$ and $35$. The swap removes $14$ and $35$ and adds $15$ and $34$, which are already present. The result has edges $12$, $23$, $15$, $34$ and sampled degrees $(2,2,2)$.\n\\end{itemize}\nFor the corrected steps let $s_v=d_v-d_v^r\\ge0$. In the starting graph of the corrected proof of Result 1, the vertex $i$ has degree $d_i^r+1+(s_i-1)=d_i$, likewise for $j$, and every other sampled $v$ has degree $d_v^r+s_v=d_v$. Each unsampled vertex has a sampled neighbor, and no two unsampled vertices are adjacent. The graph is then compatible. Its edges among sampled vertices are the recruitment edges, and since $(i,j)\\notin E_R$ with $i<j$, the edge $\\{i,j\\}$ is not one of them. Replacing $\\{i,u\\}$ and $\\{j,u\\}$ by $\\{i,j\\}$ and deleting $u$ leaves every sampled degree unchanged. In the corrected proof of Result 2 the vertices $u_1$ and $u_2$ are each adjacent to one sampled vertex only. Then $\\{i,u_2\\}$ and $\\{j,u_1\\}$ are absent, and the swap preserves every degree. Suppose that $Z_i=0$ and $Z_j=1$, as in the printed proof, and let $m=|E_{SU}|$ and $Q=\\sum_vd_vZ_v$. The common denominator of $h_1$ and $h_2$ is $D=Q(2m-Q)/(2m)$, and $D>0$ because $Q\\ge d_j>0$ and $2m-Q\\ge d_i>0$. The numerators differ by 2, and I obtain $h_1-h_2=2/D>0$, as (A1) states. On the instance $V_R=\\{1,2\\}$ with vertex 1 recruiting vertex 2, $\\mathbf d_R=(2,2)$ and $\\mathbf Z_R=(0,1)$, the two graphs have $h_1=1/3$ and $h_2=-1$, and $2/D=4/3$.", "problem": "Each proof takes as its starting graph any compatible graph that contains the named edges, changes some edges, and asserts that the result is compatible. This can fail. In the proof of Result 1 the edge $\\{i,j\\}$ may already belong to $E^1_{SU}$, since its absence from $E_R$ does not exclude it. Removing $\\{i,u\\}$ and $\\{j,u\\}$ then lowers the degrees of $i$ and $j$ by one, and the later claim that $\\{i,j\\}\\notin G^1_S$ is false. In the proof of Result 2 the edges $\\{i,u_2\\}$ and $\\{j,u_1\\}$ may already belong to $E^1_{SU}$. The swap then lowers the degrees of $i$ and $j$ by one, the pair $(G^2_{SU},\\mathbf Z^2_{SU})$ is not compatible, and display (A1), which uses one denominator for $h_1$ and $h_2$, does not apply.", "what_survives": "Results 1 and 2 hold exactly as printed, with the same hypotheses, and display (A1) is correct for the corrected swap. Result 2 needs no condition that excludes a recruitment edge between $i$ and $j$: in the last instance above $i$ recruits $j$. The correction restricts the choice of a witness in two existence proofs and adds no assumption on the data or the network. Definitions 1--4, Result 3, Figure 1 and the application do not depend on these steps."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-E003", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of Definition 1 with equation (1) for h as a ratio of two sums. A red box marks the denominator, the sum of (d_i delta_ij minus d_i d_j over 2|E_SU|) Z_i Z_j.", "caption": "Definition 1 and equation (1), p. 155 (PDF p. 3), left column. The marked denominator is zero when all vertices of positive degree have the same trait.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E006.png"}, {"alt": "Screenshot of equation (2) for p. A red box marks the leading factor 1 over (n minus |M|).", "caption": "Equation (2), p. 155 (PDF p. 3), top of the right column. The marked factor $1/(n-|M|)$ is undefined for a sample that consists of seeds only.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E006-b.png"}, {"alt": "Screenshot of Definition 4. A red box marks the words: every (G_SU, Z_SU) in C(G_R, d_R, Z_R).", "caption": "Definition 4, p. 155 (PDF p. 3), right column. The marked words take $h$ and $p$ at every compatible pair.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E006-c.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T14", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T14"}, {"id": "P012-T15", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T15"}, {"id": "P012-T24", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T24"}], "headline": "Equations (1) and (2) are undefined at some compatible pairs, so Definition 4 needs the conditions under which $h$ and $p$ are defined on the whole compatible set.", "id": "P012-E006", "location": "Definitions 1, 2 and 4, equations (1) and (2), p. 155 (PDF p. 3); the same functionals are used in Web Appendix 1, p. 1", "pdf_page": 3, "printed_page": "155", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P012-T14", "P012-T15", "P012-T24"], "text": {"as_printed": "\\emph{Definition 1:} Homophily is the correlation between trait values of vertices connected by an edge,\n\\[\nh(G_{SU},\\mathbf Z_{SU})=\\frac{\\sum_{i,j\\in V_{SU}}(A_{ij}-d_id_j/2|E_{SU}|)Z_iZ_j}{\\hl{\\sum_{i,j\\in V_{SU}}(d_i\\delta_{ij}-d_id_j/2|E_{SU}|)Z_iZ_j}},\\qquad(1)\n\\]\n\\emph{Definition 2:} Preferential recruitment is the average deviation from proportional recruitment, given knowledge of $G_{SU}$, $G_R$, $\\mathbf t_R$, and $\\mathbf Z_{SU}$,\n\\[\np(G_{SU},G_R,\\mathbf t_R,\\mathbf Z_{SU})=\\hl{\\frac{1}{n-|M|}}\\sum_{j\\notin M}\\left(1\\{Z_j=Z_{r_j}\\}-\\frac{|\\{k\\in S_{r_j}(t_j):Z_{r_j}=Z_k\\}|}{|S_{r_j}(t_j)|}\\right),\\qquad(2)\n\\]\nwhere $M$ is the set of seeds.\n\n\\emph{Definition 4:} Identification region: The identification regions for $h$ and $p$ are given by the smallest intervals that contain $h(G_{SU},\\mathbf Z_{SU})$ and $p(G_{SU},G_R,\\mathbf t_R,\\mathbf Z_{SU})$ for \\hl{every} $(G_{SU},\\mathbf Z_{SU})\\in\\mathrm{C}(G_R,\\mathbf d_R,\\mathbf Z_R)$.", "corrected": "\\emph{Definition 1} applies to pairs $(G_{SU},\\mathbf Z_{SU})$ with \\hl{$|E_{SU}|>0$ and $D_0D_1>0$}, where $D_z=\\sum_{i\\in V_{SU}:Z_i=z}d_i$; at any other pair $h$ is undefined. \\emph{Definition 2} applies when \\hl{$n>|M|$}; otherwise $p$ is undefined.\n\n\\emph{Definition 4:} Identification region. \\hl{Suppose that $\\sum_{i\\in V_R:Z_i=z}d_i>0$ for $z=0$ and for $z=1$.} Then $h$ is defined at every pair in $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$, and the identification region for $h$ is the smallest interval that contains $h(G_{SU},\\mathbf Z_{SU})$ for every $(G_{SU},\\mathbf Z_{SU})\\in\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$. \\hl{Suppose that $n>|M|$.} Then $p$ is defined at every pair in $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$, and the identification region for $p$ is the smallest interval that contains $p(G_{SU},G_R,\\mathbf t_R,\\mathbf Z_{SU})$ for every such pair. If the condition for $h$ or for $p$ fails, the identification region of that functional is undefined.", "evidence": "The evidence is a derivation and three exact examples. The identities $Z_i^2=Z_i$ and $\\sum_id_i=2m$ imply that\n\\[\n\\sum_{i,j\\in V_{SU}}\\Bigl(d_i\\delta_{ij}-\\frac{d_id_j}{2m}\\Bigr)Z_iZ_j=q-\\frac{q^2}{2m}=\\frac{q(2m-q)}{2m}.\n\\]\nI write $D_1=q$ and $D_0=2m-q$ for the total degrees of the vertices with trait $1$ and trait $0$; the denominator is then $D_0D_1/(2m)$.\n\\begin{enumerate}\n\\item Let vertex 1 be a seed that recruits vertex 2, with $\\mathbf d_R=(1,1)$ and $\\mathbf Z_R=(0,0)$. The only compatible pair is the single edge $\\{1,2\\}$. Here $m=1$ and $q=0$, and (1) is $0/0$, while $p=0$.\n\\item Let vertices 1 and 2 both be seeds, with $\\mathbf d_R=(1,1)$ and $\\mathbf Z_R=(0,1)$. With the edge $\\{1,2\\}$ the pair is compatible and $h=-1$, and (2) is an empty sum divided by $n-|M|=0$ at every compatible pair.\n\\item Let vertex 1 recruit vertex 2, with $\\mathbf d_R=(1,4)$ and $\\mathbf Z_R=(1,1)$. Vertex 2 has three unsampled neighbors, and the eight assignments of their traits are the eight compatible pairs. Seven of them have $h\\in\\{-3/5,-1/3,-1/7\\}$; at the eighth, with all traits equal to 1, $h$ is undefined.\n\\end{enumerate}\nThe condition in the corrected Definition 4 is the exact one for $h$. If, for each trait, the reported degrees of the sampled vertices with that trait have a positive sum, then $D_0>0$ and $D_1>0$ at every compatible pair, because compatible graphs preserve the reported degrees. If all sampled vertices of positive degree share one trait, giving that trait to every unsampled vertex yields a compatible pair at which (1) is $0/0$, because every edge of $G_{SU}$ has a sampled endpoint.", "problem": "Definition 4 takes $h$ and $p$ at every compatible pair, but compatibility does not make the two formulas defined. Let $m=|E_{SU}|$ and $q=\\sum_{i\\in V_{SU}}d_iZ_i$. For $m>0$ the denominator of (1) equals $q(2m-q)/(2m)$. When all vertices of positive degree have the same trait ($q=0$ or $q=2m$), this denominator and the numerator both vanish, and (1) reads $0/0$. For $m=0$ the terms $d_id_j/2|E_{SU}|$ are themselves undefined. The factor $1/(n-|M|)$ in (2) is undefined when the sample consists of seeds only. The paper states no condition that excludes these cases.", "what_survives": "No result or number changes. The hypotheses of Result 2 imply that there exist sampled vertices of both traits with positive degree, and the hypothesis of Result 3 implies that at least one recruitment took place. Then $h$ and $p$ are defined on the whole compatible set where those results apply. Figure 1 meets both conditions, and the application has $527-6=521$ recruitments. What changes is the statement of Definition 4, the optimization of Web Appendix 1 over the compatible set, and the sentence on p.~158 that three assumptions suffice for the bounds: each needs the two conditions below."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-E006", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Screenshot of three lines of Web Appendix 1. A red box marks the word non-decreasing in the sentence: Let T_t be a positive non-decreasing sequence indexed by t, with limit zero.", "caption": "Web Appendix 1, Web Appendix p. 1 (supplement PDF p. 1). The marked word should read “non-increasing”.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E007.png"}, {"alt": "Screenshot of display (2), a chain of four infinite sums ending in a comparison with the sum of 1 over t. Red boxes mark the lower limit t=1 under each of the four summation signs.", "caption": "Proof of Proposition 1, display (2), Web Appendix p. 2 (supplement PDF p. 2). The four marked sums start at $t=1$, where $T_t=(\\epsilon\\log t)^{-1}$ is undefined.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-E007-b.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T16", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T16"}, {"id": "P012-T24", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T24"}], "headline": "The cooling schedule is called “non-decreasing” where “non-increasing” is meant, and the schedule and the sums of display (2) start at $t=1$, where $\\log t=0$.", "id": "P012-E007", "location": "Web Appendix 1, Web Appendix p. 1 (supplement PDF p. 1), sentence after the definition of the kernel $K$; Proposition 1 and display (2), Web Appendix p. 2", "pdf_page": 1, "printed_page": "1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P012-T16", "P012-T24"], "text": {"as_printed": "Let $T_t$ be a positive \\hl{non-decreasing} sequence indexed by $t$, with $\\lim_{t\\to\\infty}T_t=0$. (Web Appendix p.~1.)\n\nProposition 1. Let the cooling schedule be given by $T_t=\\bigl(\\epsilon\\log(t)\\bigr)^{-1}$ where $\\epsilon>0$ is a constant. (Web Appendix p.~2.)\n\\[\n\\sum_{\\hl{\\scriptstyle t=1}}^{\\infty}\\exp[-D^*/T_t]=\\sum_{\\hl{\\scriptstyle t=1}}^{\\infty}\\exp[-D^*\\epsilon\\log(t)]=\\sum_{\\hl{\\scriptstyle t=1}}^{\\infty}\\frac{1}{t^{D^*\\epsilon}}\\ge\\sum_{\\hl{\\scriptstyle t=1}}^{\\infty}\\frac1t=\\infty\\qquad(2)\n\\]", "corrected": "Let $T_t$ be a positive \\hl{non-increasing} sequence indexed by $t$, with $\\lim_{t\\to\\infty}T_t=0$.\n\nProposition 1. Let the cooling schedule be given by \\hl{a finite $T_1\\ge T_2$ and} $T_t=\\bigl(\\epsilon\\log(t)\\bigr)^{-1}$ \\hl{for $t\\ge2$}, where $\\epsilon>0$ is a constant.\n\\[\n\\sum_{\\hl{\\scriptstyle t=2}}^{\\infty}\\exp[-D^*/T_t]=\\sum_{\\hl{\\scriptstyle t=2}}^{\\infty}\\exp[-D^*\\epsilon\\log(t)]=\\sum_{\\hl{\\scriptstyle t=2}}^{\\infty}\\frac{1}{t^{D^*\\epsilon}}\\ge\\sum_{\\hl{\\scriptstyle t=2}}^{\\infty}\\frac1t=\\infty\\qquad(2)\n\\]", "evidence": "The evidence is a derivation. For $t>1$, differentiation gives\n\\[\n\\frac{d}{dt}\\,\\frac{1}{\\epsilon\\log t}=-\\frac{1}{\\epsilon\\,t\\,(\\log t)^2}<0.\n\\]\nFor $t\\ge2$, the bound $D^*\\epsilon\\le1$ implies that $t^{-D^*\\epsilon}\\ge t^{-1}$, and $\\sum_{t\\ge2}1/t=\\infty$. The comparison in display (2) is unchanged when the first term is dropped.", "problem": "A positive non-decreasing sequence satisfies $T_t\\ge T_1>0$ and cannot tend to zero; the schedule of Proposition 1 is decreasing for $t>1$, and ``non-increasing'' is meant. The schedule is also undefined at the first index, since $\\log1=0$, although the sums of display (2) start at $t=1$.", "what_survives": "The bound $D^*\\epsilon\\le1$ and the comparison with the harmonic series stand. This correction concerns the wording and the first index only; the other hypotheses of the convergence theorem are the subject of \\fid{P012-N008}, and the proposal kernel that of \\fid{P012-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-E007", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of four lines of the introduction. A red box marks the words: the estimators used by these programs are not documented.", "caption": "Introduction, p. 154 (PDF p. 2), right column. The marked clause says that the estimators of RDSAT and RDS Analyst are not documented.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "The homophily estimators of RDS Analyst were publicly documented before the paper was accepted; the statement that the estimators of both programs “are not documented” is wrong for that program.", "id": "P012-N001", "location": "Introduction, p. 154 (PDF p. 2), right column, sentence beginning “Two software tools”; reference 39 is on p. 160 (PDF p. 8)", "pdf_page": 2, "printed_page": "154", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Two software tools for analysis of RDS data produce estimates of homophily or preferential recruitment---RDSAT (38) and RDS Analyst (39)---but \\hl{the estimators used by these programs are not documented}.", "corrected": "Two software tools for analysis of RDS data produce estimates of homophily or preferential recruitment---RDSAT (38) and RDS Analyst (39)\\hl{; the RDS Analyst estimators are described in the help page of the function \\texttt{homophily.estimates} of the RDS package (39)}.", "evidence": "A check of public sources: the help file of \\texttt{homophily.estimates} in RDS 0.5 (\\url{https://github.com/cran/RDS/blob/0.5/man/homophily.estimates.Rd}), the function's source in the same release, the help pages of RDS Analyst for recruitment homophily and population homophily (last modified in August and September 2013), and the source of the RDS Analyst interface, in which the two dialogs for homophily call \\texttt{homophily.estimates}.", "problem": "The sentence says of both programs that their estimators are not documented. For RDS Analyst this is not the case. Reference 39 is the RDS package, version 0.5, released on 28 November 2013. Its help page for the function \\texttt{homophily.estimates} says that the function ``computes an estimate of the population homophily and the recruitment homophily based on a categorical variable'' and defines both ratios, and RDS Analyst obtains its homophily output by calling this function. The paper was accepted on 9 March 2017. I could not obtain the RDSAT 7.1 manual, and the half of the sentence that concerns RDSAT is not settled either way. The citation of reference 39 for RDS Analyst is apt and is not an error.", "what_survives": "No definition, result or number of the paper depends on this sentence. That the estimators are documented does not mean that they identify $h$ or $p$ as the paper defines them; the paper's argument on that point is untouched."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-N001", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Screenshot of Table 1 with columns Trait, h, I_h, p and I_p and rows Sex, Crack cocaine use and Homelessness. Red boxes mark the three lower limits of I_h: -0.7027, -0.6918 and -0.7179.", "caption": "Table 1, p. 158 (PDF p. 6). The three lower limits of $I_h$ are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N002.png"}, {"alt": "Two-panel chart. Upper panel: three horizontal blue bars for sex, crack cocaine use and homelessness span the printed intervals from about -0.70 to about 0.83 on an axis from -1 to 1, with a dot at each reported point value of h; an orange dashed vertical line at -0.7373 lies to the left of all three bars. Lower panel: the same lower ends enlarged, with the printed limits -0.7027, -0.6918 and -0.7179 to the right of the dashed line and orange arrows labeled at least 0.0345, 0.0454 and 0.0193 spanning the gaps.", "caption": "The printed intervals $I_h$ of Table 1 on the whole range of $h$ (upper panel) and their lower ends enlarged (lower panel). The dots show the point values after excluding vertices with missing traits and their incident edges. The dashed line is the bound $-1462/1983$; for nomination-network degrees, the true lower limit of each interval lies at or to the left of it.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N002-lower-limits.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T3", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T3"}, {"id": "P012-T4", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T4"}, {"id": "P012-T5", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T5"}, {"id": "P012-T20", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T20"}], "headline": "The three lower limits of $I_h$ in Table 1 cannot be the lower limits of the identification intervals of Definition 4 for the network described on p. 156: a compatible graph has $h\\le-1462/1983\\approx-0.7373$, which is below all three.", "historical_evidence_type": "lean_proof", "id": "P012-N002", "location": "Table 1, p. 158 (PDF p. 6), column $I_h$, the three lower limits; the network counts are on p. 156 (PDF p. 4)", "pdf_page": 6, "printed_page": "158", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P012-T20", "P012-T3", "P012-T4", "P012-T5"], "text": {"as_printed": "\\begin{audittable}{lrl}{Table 1, homophily columns, as printed.}\nTrait & $h$ & $I_h$ \\\\\nSex & $0.2638$ & $\\hl{-0.7027}$, $0.8292$ \\\\\nCrack cocaine use & $0.2711$ & $\\hl{-0.6918}$, $0.8231$ \\\\\nHomelessness & $-0.2154$ & $\\hl{-0.7179}$, $0.8242$ \\\\\n\\end{audittable}\nThe RDS-net study included $|V_{SU}|=2{,}626$ people, of whom $n=|V_R|=527$ were either seeds or recruited subjects, and 2,099 were nominated but never-recruited subjects. There are $|E_{SU}|=3{,}307$ edges in $G_{SU}$, of which $|E_S|=1{,}180$ link recruited subjects to recruited subjects, and 2,127 link recruited subjects to unrecruited subjects.", "corrected": "\\begin{audittable}{lrl}{Table 1, homophily columns, corrected as far as the published information allows for nomination-network degrees.}\nTrait & $h$ & $I_h$ \\\\\nSex & $0.2638$ & $\\hl{\\text{lower limit at most } -1462/1983}$, $0.8292$ \\\\\nCrack cocaine use & $0.2711$ & $\\hl{\\text{lower limit at most } -1462/1983}$, $0.8231$ \\\\\nHomelessness & $-0.2154$ & $\\hl{\\text{lower limit at most } -1462/1983}$, $0.8242$ \\\\\n\\end{audittable}\nTo retain the stated nomination-network interpretation, the three lower limits are to be recomputed from \\hl{that network's sampled degrees} and the traits of the recruited subjects. For those degrees, the value $-1462/1983\\approx-0.7373$ is an upper bound on each true lower limit and is not itself the limit. The degree vector $\\mathbf d_R$ used for the bounds, and its relation to the nomination network, should be stated. If a degree vector other than that of the nomination network was used, the intervals \\hl{must be checked for that input}, and the nomination network is not a member of the corresponding compatible set. The exact minimum has a closed form, given in \\fid{P012-N007}.", "evidence": "The evidence is a derivation, with its algebra checked by exact arithmetic and by a Lean proof. For a graph with $a$ edges joining two vertices of trait 1, $b$ edges joining two vertices of trait 0 and $c$ edges joining unlike vertices, equation (1) reduces to\n\\[\nh=\\phi(a,b,c)=\\frac{4ab-c^2}{(2a+c)(2b+c)}.\n\\]\nIn the graph constructed above let $a$, $b$ and $c_R$ count the recruitment edges of the three kinds; then $a+b+c_R=r=521$. All $S$ added edges join unlike vertices, and I have $h=\\phi(a,b,c)$ with $c=c_R+S$. I set $T=r+S=3{,}966$. A direct computation gives $(2a+c)(2b+c)=T^2-(a-b)^2>0$ and\n\\[\n\\frac{r-S}{T}\\,(2a+c)(2b+c)-(4ab-c^2)=2T\\,c_R+\\frac{2S}{T}\\,(a-b)^2\\ge0,\n\\]\nand dividing by this positive product I obtain $h\\le(r-S)/T=-1462/1983$. The bound uses only the degree total $4{,}487$ and the fact that each $d_i$ is at least the recruitment degree of $i$; it does not depend on the unknown traits, individual degrees or recruitment graph. The constructed graph has $3{,}445$ unsampled vertices, more than the $2{,}099$ of the nomination network. Definition 3 allows this, since it does not fix the number of unsampled vertices, and Web Appendix 2 itself creates new ones.\n\nEvaluating $\\phi(a,b,c_R+S)$ exactly at all $136{,}503$ triples of nonnegative integers with $a+b+c_R=521$, I find the largest value $-178409/241987\\approx-0.73727$, at $(a,b,c_R)=(260,261,0)$ and $(261,260,0)$. A Lean proof covers the inequality above for nonnegative rational $a$, $b$, $c_R$ with sum $521$ and its comparison with the three printed limits; it also covers the graph construction, its compatibility and the reduction of equation (1).\n\\begin{audittable}{lrrr}{The printed lower limits of $I_h$ against the bound $-1462/1983$ (exact differences).}\nTrait & Printed lower limit & Printed limit minus bound & Decimal (rounded) \\\\\nSex & $-0.7027$ & $685459/19830000$ & $0.034567$ \\\\\nCrack cocaine use & $-0.6918$ & $450803/9915000$ & $0.045467$ \\\\\nHomelessness & $-0.7179$ & $384043/19830000$ & $0.019367$ \\\\\n\\end{audittable}", "problem": "Definition 4 makes $I_h$ the smallest interval that contains $h$ at every compatible pair. The paper identifies the nomination network with $G_{SU}$ and defines $d_i$ as the number of edges at $i$. The counts on p.~156 then fix the total of the reported degrees: $2(1{,}180)+2{,}127=4{,}487$. The $527-6=521$ recruitment edges use $1{,}042$ of these edge-ends and leave $S=3{,}445$. I attach each of the $S$ spare edge-ends to a new unsampled vertex of its own, and give that vertex the trait opposite to the trait of its sampled neighbor. The resulting graph satisfies the four conditions of Definition 3, and whatever the traits of the recruited subjects its homophily is at most $(521-3{,}445)/(521+3{,}445)=-1462/1983\\approx-0.7373$. Each printed lower limit exceeds this value, by at least $0.0345$ (sex), $0.0454$ (crack cocaine use) and $0.0193$ (homelessness), which rounding to four decimals cannot explain. Either the search that produced the limits stopped short of the minimum, or the degrees given to it were not those of the nomination network. In the second case the nomination network violates condition 4 of Definition 3 and is not in the compatible set, and the comparison of the point values with the intervals is not the one the paper describes. The published materials do not supply the Hartford data or analysis code, so I cannot say which case holds or give the correct limits.", "what_survives": "The point values of $h$ and $p$, the upper limits of $I_h$ and the intervals $I_p$ are not shown wrong by this finding. What the paper reads from the table holds with any wider intervals: each point value lies inside its interval and every interval covers zero. Definitions 1--4 and Results 1--3 do not depend on the table. For nomination-network degrees, the three lower limits of $I_h$ and the widths of those intervals change, and the remark on p.~158 that the regions ``were considerably smaller than the outcome space $[-1,1]$'' has to be reassessed once the limits are recomputed; this finding does not show it false."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-N002", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of the end of the proof of Result 1. Red boxes mark the words: under the conditions of definition 3, which run over a line break.", "caption": "Appendix, end of the proof of Result 1, p. 160 (PDF p. 8), right column. The marked words cite Definition 3.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N005.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T17", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T17"}], "headline": "The last step of the proof of Result 1 cites Definition 3 where it needs the compatibility of recruitment-induced subgraphs defined at the start of the same proof.", "id": "P012-N005", "location": "Appendix, proof of Result 1, p. 160 (PDF p. 8), right column, sentence beginning “Clearly both”", "pdf_page": 8, "printed_page": "160", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P012-T17"], "text": {"as_printed": "Clearly both $G^1_S$ and $G^2_S$ are compatible with the observed data \\hl{under the conditions of definition 3}, but $\\{i,j\\}\\notin G^1_S$ and $\\{i,j\\}\\in G^2_S$.", "corrected": "Clearly both $G^1_S$ and $G^2_S$ are compatible with the observed data \\hl{in the sense defined at the start of this proof}, but $\\{i,j\\}\\notin G^1_S$ and $\\{i,j\\}\\in G^2_S$.", "evidence": "A check of the page and a small example. Let vertices 1 and 2 be seeds with $d_1=d_2=1$ and no recruitment, and let the augmented graph have the edges $\\{1,u\\}$ and $\\{2,u\\}$. Removing $u$ leaves degrees $(0,0)$, which fail condition 4 of Definition 3 and satisfy the opening definition.", "problem": "Definition 3 concerns augmented graphs, and its condition 4 requires each sampled vertex to have exactly its reported degree. Removing the unsampled vertices lowers those degrees, and $G^1_S$ and $G^2_S$ need not satisfy Definition 3. They satisfy the definition given at the opening of the proof, which asks that the degree of $i$ in $G_S$ be at most $d_i$, and this is what the step needs.", "what_survives": "The step is valid with the right reference, and Result 1 is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-N005", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of the proof of Result 2 up to the words The difference is. Four red boxes mark the argument G_R inside h(G^1_SU, G_R, Z^1_SU) and h(G^2_SU, G_R, Z^2_SU), each of which appears twice.", "caption": "Appendix, proof of Result 2, p. 160 (PDF p. 8), right column. The four marked occurrences of $G_R$ are redundant arguments of $h$.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N006.png"}, {"alt": "Screenshot of the sentence beginning To ease notation. A red box marks the bold Z in h(G_SU, Z), which has no subscript.", "caption": "p. 155 (PDF p. 3), right column, below equation (2). The marked $\\mathbf Z$ lacks the subscript $SU$.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N006-b.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T18", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T18"}], "headline": "In the proof of Result 2 the function $h$ is written four times with a redundant argument $G_R$, and on p. 155 once with an unsubscripted $\\mathbf Z$; the function $h(G_{SU},\\mathbf Z_{SU})$ of equation (1) is meant.", "id": "P012-N006", "location": "Appendix, proof of Result 2, p. 160 (PDF p. 8), right column, four occurrences; and p. 155 (PDF p. 3), sentence beginning “To ease notation”", "pdf_page": 8, "printed_page": "160", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P012-T18"], "text": {"as_printed": "[\\ldots] such that $h(G^1_{SU},\\hl{G_R,}\\,\\mathbf Z^1_{SU})\\ne h(G^2_{SU},\\hl{G_R,}\\,\\mathbf Z^2_{SU})$. [\\ldots] Let $h_1=h(G^1_{SU},\\hl{G_R,}\\,\\mathbf Z^1_{SU})$ and $h_2=h(G^2_{SU},\\hl{G_R,}\\,\\mathbf Z^2_{SU})$ be the calculated values of homophily. (p.~160.)\n\nTo ease notation, we will often use $h$ and $p$ to refer to $h(G_{SU},\\hl{\\mathbf Z})$ and $p(G_{SU},G_R,\\mathbf t_R,\\mathbf Z_{SU})$ respectively. (p.~155.)", "corrected": "[\\ldots] such that $\\hl{h(G^1_{SU},\\mathbf Z^1_{SU})}\\ne\\hl{h(G^2_{SU},\\mathbf Z^2_{SU})}$. [\\ldots] Let $h_1=\\hl{h(G^1_{SU},\\mathbf Z^1_{SU})}$ and $h_2=\\hl{h(G^2_{SU},\\mathbf Z^2_{SU})}$ be the calculated values of homophily.\n\nTo ease notation, we will often use $h$ and $p$ to refer to $h(G_{SU},\\hl{\\mathbf Z_{SU}})$ and $p(G_{SU},G_R,\\mathbf t_R,\\mathbf Z_{SU})$ respectively.", "evidence": "A check of the two pages against Definition 1, equation (1) and Definition 4. Display (A1) uses adjacency, degrees and traits only, in agreement with equation (1).", "problem": "Definition 1 gives $h$ two arguments, the augmented graph and its traits, and $h$ does not depend on the recruitment graph. The symbol $\\mathbf Z$ without a subscript is not defined in the paper. Both are slips of notation.", "what_survives": "The calculation in (A1) and Result 2 are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-N006", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Screenshot of the heading Web Appendix 1 and its first three lines. A red box marks the words: there are no general closed-form expressions for the extrema of h and p.", "caption": "Web Appendix 1, first sentence, Web Appendix p. 1 (supplement PDF p. 1). The marked words deny that the extrema of $h$ and $p$ have closed forms.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N007.png"}], "formal_release": null, "formal_statements": [{"id": "P012-T6", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T6"}, {"id": "P012-T7", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T7"}, {"id": "P012-T21", "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-T21"}], "headline": "The extrema of $h$ and $p$ over the compatible set have closed forms that are computed in one pass over the data; the opening sentence of Web Appendix 1 says that there are none.", "id": "P012-N007", "location": "Web Appendix 1, first sentence, Web Appendix p. 1 (supplement PDF p. 1)", "pdf_page": 1, "printed_page": "1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P012-T21", "P012-T6", "P012-T7"], "text": {"as_printed": "Unfortunately \\hl{there are no general closed-form expressions} for the extrema of $h$ and $p$ on $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$.", "corrected": "\\hl{The extrema of $h$ and $p$ on $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$ have closed-form expressions.} For \\hl{observed data with a nonempty compatible set}, let $s_i=d_i-d_i^r$ for $i\\in V_R$, let $S_z=\\sum_{i\\in V_R:Z_i=z}s_i$ for $z=0,1$ and $S=S_0+S_1$, and let $a$, $b$ and $c_R$ be the numbers of recruitment edges that join two vertices of trait 1, two vertices of trait 0, and unlike vertices. For a nonseed $j$ let $b_j$ be the number of vertices recruited by $r_j$ at or after time $t_j$, and $a_j$ the number of those with trait $Z_{r_j}$. Suppose that $\\sum_{i\\in V_R:Z_i=z}d_i>0$ for $z=0$ and for $z=1$, and that $n>|M|$. Let $\\phi(x,y,c)=(4xy-c^2)/((2x+c)(2y+c))$. Then\n\\[\n\\begin{aligned}\n\\min_{\\mathcal C}h&=\\phi(a,b,c_R+S),\\\\\n\\max_{\\mathcal C}h&=\\phi(a+S_1,b+S_0,c_R),\\\\\n\\min_{\\mathcal C}p&=\\frac{1}{n-|M|}\\sum_{j\\notin M}\\Bigl[\\mathbf 1\\{Z_j=Z_{r_j}\\}-\\frac{a_j+s_{r_j}}{b_j+s_{r_j}}\\Bigr],\\\\\n\\max_{\\mathcal C}p&=\\frac{1}{n-|M|}\\sum_{j\\notin M}\\Bigl[\\mathbf 1\\{Z_j=Z_{r_j}\\}-\\frac{a_j}{b_j+s_{r_j}}\\Bigr].\n\\end{aligned}\n\\]\nEach extremum is attained at a compatible pair in which every spare edge-end of a sampled vertex has its own unsampled neighbor: of the opposite trait for the minimum of $h$ and the maximum of $p$, of the same trait for the maximum of $h$ and the minimum of $p$. These four endpoints are computed directly from the data; simulated annealing remains available for other objectives or for additional constraints on the graph.", "evidence": "The evidence is a complete derivation, with exact enumeration as a check. The notation is that of the corrected statement below.\n\n\\emph{Homophily.} Let $x$, $y$ and $c$ be the numbers of edges of a compatible graph that join two vertices of trait 1, two vertices of trait 0, and unlike vertices. Equation (1) reduces to\n\\[\nh=\\phi(x,y,c)=\\frac{4xy-c^2}{(2x+c)(2y+c)}=1-\\frac{c}{2x+c}-\\frac{c}{2y+c}.\n\\]\nEvery edge outside the recruitment graph uses at least one spare edge-end of a sampled vertex, of trait 1 if the edge joins two vertices of trait 1 and of trait 0 if it joins two vertices of trait 0. Then $a\\le x\\le a+S_1$, $b\\le y\\le b+S_0$ and $c_R\\le c\\le c_R+S$. The function $\\phi$ is nondecreasing in $x$ and in $y$ and nonincreasing in $c$: for $X\\ge x$ and $C\\ge c$,\n\\[\n\\begin{aligned}\n\\phi(X,y,c)-\\phi(x,y,c)&=\\frac{2c(X-x)}{(2X+c)(2x+c)}\\ge0,\\\\\n\\phi(x,y,c)-\\phi(x,y,C)&=\\frac{2x(C-c)}{(2x+c)(2x+C)}+\\frac{2y(C-c)}{(2y+c)(2y+C)}\\ge0,\n\\end{aligned}\n\\]\nand the first identity holds with $x$ and $y$ exchanged. Under the two conditions on the reported degrees every denominator that occurs is positive. I conclude that $\\phi(a,b,c_R+S)\\le h\\le\\phi(a+S_1,b+S_0,c_R)$. Both bounds are attained. To see this, I keep the recruitment edges and attach to each sampled vertex $i$ exactly $s_i$ new unsampled neighbors of degree one; the graph is compatible. If each new vertex has the trait opposite to that of its neighbor, the counts are $(a,b,c_R+S)$; if each has the trait of its neighbor, they are $(a+S_1,b+S_0,c_R)$.\n\n\\emph{Preferential recruitment.} Fix a nonseed $j$ with recruiter $r=r_j$. The $b_j$ vertices recruited by $r$ at or after $t_j$ are neighbors of $r$ that are susceptible just before $t_j$, and they are the only susceptible neighbors of $r$ fixed by the recruitment graph; $a_j$ of them have the trait of $r$. Any other susceptible neighbor of $r$ uses a spare edge-end of $r$. Let $y_j\\le s_r$ be the number of these and $q_j\\le y_j$ the number with the trait of $r$. The fraction subtracted in the summand of $j$ in equation (2) is $(a_j+q_j)/(b_j+y_j)$, and by cross-multiplication I find\n\\[\n\\frac{a_j}{b_j+s_r}\\le\\frac{a_j+q_j}{b_j+y_j}\\le\\frac{a_j+s_r}{b_j+s_r}.\n\\]\nIn the graph with one new unsampled neighbor for every spare edge-end, these neighbors are never recruited, and $y_j=s_{r_j}$ for every $j$ at once. With equal traits, $q_j=s_{r_j}$ for every $j$, which is the minimum of $p$; with opposite traits, $q_j=0$ for every $j$, which is the maximum.\n\n\\emph{Checks.} For the example of Figure 1, I have $a=3$, $b=0$, $c_R=1$, $S_1=7$, $S_0=2$ and $S=9$, and from the formulas I obtain the four values in the table. They equal the extrema over all $26{,}182$ compatible pairs of that example, counted up to the labeling of the unsampled vertices and found by exhaustive enumeration with exact arithmetic. The formulas also agree with exhaustive enumeration on $5{,}008$ further small data sets, with $86{,}988$ compatible pairs in all.\n\\begin{audittable}{llrr}{Figure 1 of the paper: the closed forms against the printed extrema.}\nEndpoint & Closed form & Exact value & Printed in Figure 1 \\\\\nMinimum of $h$ & $\\phi(3,0,10)$ & $-5/8$ & $-0.625$ \\\\\nMaximum of $h$ & $\\phi(10,2,1)$ & $79/105$ & $0.752$ \\\\\nMinimum of $p$ & $\\tfrac14\\bigl(\\tfrac15+0-\\tfrac34+0\\bigr)$ & $-11/80$ & $-0.138$ \\\\\nMaximum of $p$ & $\\tfrac14\\bigl(\\tfrac45+\\tfrac13+0+\\tfrac12\\bigr)$ & $49/120$ & $0.408$ \\\\\n\\end{audittable}", "problem": "The sentence is the premise of the computational procedure of Web Appendixes 1 and 2, that is, of the simulated-annealing search and of Proposition 1. It is false under the conditions stated below. All four endpoints of the identification intervals are explicit functions of quantities read from the observed data: the recruitment edges counted by the traits of their ends, the spare degrees $d_i-d_i^r$, and the order of recruitment. Each spare edge-end of a sampled vertex can be given its own unsampled neighbor, and the extreme values of $h$ and of $p$ are attained when the traits of these neighbors are all opposite to, or all equal to, the trait of the sampled vertex. A search is unnecessary for these four endpoints. For them, the appendix search, its convergence argument and the search for extremizing compatible graphs can all be replaced by the formulas and the two constructions with separate unsampled neighbors. A finite run of a search can return an interval that is too narrow. The formulas give the endpoints exactly. This finding does not by itself show any reported number to be wrong.", "what_survives": "This correction leaves Definitions 1--4, the identification regions themselves and Results 1--3 unchanged; the four exact extrema round to the values printed in Figure 1. The formulas also give short proofs of the nonidentification conclusions in Results 2 and 3. The central motivation survives: incomplete network observation can leave $h$ and $p$ undetermined even when their sharp bounds are easy to compute. The full-network values in Table 1 do not use the search. Simulated annealing may still serve for another objective $J(h,p)$ or for a compatible set with further constraints, and whether Proposition 1 is true is not at issue here. The premise of Web Appendixes 1 and 2 changes: the four endpoints should be computed from the formulas. The intervals of Table 1 can be recomputed in this way once the data are available; if the printed endpoints came from compatible graphs, the exact intervals are no narrower and still cover zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-N007", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of Proposition 1 and the first paragraph of its proof. Red boxes mark epsilon greater than 0 in the statement, 0 less than epsilon less than 1 in the proof, and the sentence that begins: Theorem 1 of Hajek (3) states that.", "caption": "Proposition 1 and the start of its proof, Web Appendix p. 2 (supplement PDF p. 2). Marked: the range of $\\epsilon$ in the statement, the narrower range in the proof, and the restatement of Hajek’s theorem.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "Proposition 1 holds for the intended proposal, but its proof quotes Hajek’s theorem without the theorem’s standing hypotheses, restricts $\\epsilon$ to $(0,1)$ without need, and cites for connectivity an argument made for a smaller state space.", "id": "P012-N008", "location": "Web Appendix 1, Proposition 1 and its proof, Web Appendix p. 2 (supplement PDF p. 2); the sentence on connectivity is in Web Appendix 2, Web Appendix p. 3", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Proposition 1. Let the cooling schedule be given by $T_t=\\bigl(\\epsilon\\log(t)\\bigr)^{-1}$ where \\hl{$\\epsilon>0$} is a constant. Then $\\lim_{t\\to\\infty}\\Pr\\bigl((G_{SU},\\mathbf Z_{SU})_t\\in\\mathcal M\\bigr)=1$.\n\nProof. Let $J(h,p)=1/(1+\\epsilon+h)$ for \\hl{$0<\\epsilon<1$} and let $\\mathcal M$ be the set of $(G_{SU},\\mathbf Z_{SU})$ that achieve the global maximum of $J$ on $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$. [\\ldots] \\hl{Theorem 1 of Hajek (3) states that} if $T_t\\to0$ and $\\sum_{t=1}^{\\infty}\\exp[-D^*/T_t]$ diverges, then the sequence $(G_{SU},\\mathbf Z_{SU})_t$ converges in probability to an element of $\\mathcal M$.\n\nThe space $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$ is connected via proposals of this type (see 6, for explanation). (Web Appendix p.~3.)", "corrected": "Proposition 1. \\hl{Assume that $h$ and $p$ are defined throughout $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$} (see \\fid{P012-E006}). \\hl{Let $K$ be the repaired proposal kernel of Web Appendix 2 on this finite set} (see \\fid{P012-E001}). \\hl{It is fixed, irreducible and weakly reversible in the sense of Hajek (3).} Let the cooling schedule be given by \\hl{a finite $T_1\\ge T_2$ and} $T_t=\\bigl(\\epsilon\\log(t)\\bigr)^{-1}$ \\hl{for $t\\ge2$}, where $\\epsilon>0$ is a constant. Then $\\lim_{t\\to\\infty}\\Pr\\bigl((G_{SU},\\mathbf Z_{SU})_t\\in\\mathcal M\\bigr)=1$.\n\nProof. Let $J(h,p)=1/(1+\\epsilon+h)$ for \\hl{$\\epsilon>0$} and let $\\mathcal M$ be the set of $(G_{SU},\\mathbf Z_{SU})$ that achieve the global maximum of $J$ on $\\mathcal C(G_R,\\mathbf d_R,\\mathbf Z_R)$. [\\ldots] Theorem 1 of Hajek (3) states that\\hl{, for a fixed proposal that is irreducible and weakly reversible and for positive nonincreasing temperatures} $T_t\\to0$, if $\\sum_{t=2}^{\\infty}\\exp[-D^*/T_t]$ diverges, then \\hl{the probability that $(G_{SU},\\mathbf Z_{SU})_t$ belongs to $\\mathcal M$ tends to one}.", "intended_meaning": "Proposition 1 applies Theorem 1 of Hajek (1988) to the annealing chain on the finite set $\\mathcal C$ that the proposals of Web Appendix 2 define, with the proposal repairs in \\fid{P012-E001} and with $h$ and $p$ defined throughout $\\mathcal C$ (see \\fid{P012-E006}). A check of the published article shows that the theorem assumes a fixed proposal that is irreducible and weakly reversible, and positive temperatures that are nonincreasing with limit zero. Under these hypotheses the probability that the chain is in the set of global optima tends to one exactly when $\\sum_k\\exp(-d^*/T_k)=+\\infty$. The summary in the proof omits the hypotheses.\n\nThe intended chain satisfies them. From any compatible pair a sequence of proposals leads to the pair in which the recruitment edges are the only edges among sampled vertices and every spare edge-end has its own unsampled neighbor of trait 0. Each proposal can be undone by a path of proposals. These two facts imply that the chain is irreducible. The path can be chosen so that $J$ stays at or above its smaller value at the two ends, which is weak reversibility. The schedule $(\\epsilon\\log t)^{-1}$ is decreasing for $t\\ge2$. The bound $D^*\\le1/\\epsilon$ and the comparison with $\\sum_t1/t$ hold for every $\\epsilon>0$, and the restriction to $0<\\epsilon<1$ in the proof is unnecessary. The argument cited from reference 6, which I could check in the preprint only, concerns graphs on the sampled vertices with upper bounds on the degrees; this is a smaller space than the pairs $(G_{SU},\\mathbf Z_{SU})$ with unsampled vertices and traits, and the argument above covers the larger one. The convergence is to the set $\\mathcal M$ and not to one maximizing pair. The undefined $T_1$ is the subject of \\fid{P012-E007}."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-N008", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of the heading References and entries 1 and 2. Red boxes mark the author list Kirkpatrick C. D., Vecchi M. P. and the word simmulated in entry 1.", "caption": "Web Appendix, References, entry [1], Web Appendix p. 3 (supplement PDF p. 3). The author list and the misspelled word are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P012/exhibits/P012-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "Reference [1] of the Web Appendix omits the second author, gives the first author the wrong initials and misspells “simulated”.", "id": "P012-N009", "location": "Web Appendix, References, entry [1], Web Appendix p. 3 (supplement PDF p. 3)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "[1] \\hl{Kirkpatrick C. D., Vecchi M. P.}. Optimization by \\hl{simmulated} annealing \\emph{Science}. 1983;220:671--680.", "corrected": "[1] \\hl{Kirkpatrick S, Gelatt CD Jr, Vecchi MP}. Optimization by \\hl{simulated} annealing. \\emph{Science}. 1983;220(4598):671--680.", "evidence": "A check of the published article: its first page carries the title ``Optimization by Simulated Annealing'' and the byline ``S. Kirkpatrick, C. D. Gelatt, Jr., M. P. Vecchi''.", "problem": "The article is by S.~Kirkpatrick, C.~D.~Gelatt Jr. and M.~P.~Vecchi. The entry omits Gelatt, attaches his initials to Kirkpatrick, and misspells the title.", "what_survives": "The journal, volume and pages identify the article, and nothing in the Web Appendix depends on the entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P012/#P012-N009", "work": {"arxiv_id": "1511.05397", "doi": "10.1093/aje/kwx208", "title": "Identification of homophily and preferential recruitment in respondent-driven sampling", "url": "https://pmaronow.github.io/llm-audit/works/P012/"}, "work_id": "P012"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 6.2. A red box marks the words 'Given Conditions 3, 5, and 6'. The proposition continues: Wald-type intervals constructed as tau-hat-HT(d_k, d_l) plus or minus z_(1 minus alpha/2) times the square root of the estimated variance will tend to cover tau_HT(d_k, d_l) at least 100(1 minus alpha) percent of the time for large N.", "caption": "Proposition 6.2, p. 1926 (PDF p. 15). The marked hypotheses do not suffice for the conclusion.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E001.png"}, {"alt": "Crop of the middle of the proof of Proposition 6.2. Red boxes mark the sentence 'Note that all a_ij(D_i, D_j) are bounded above by some finite constant by Condition 3'. It is followed by a display bounding the variance of N times the estimated variance by N to the power minus 2 times a fourfold sum of covariances of the terms h_ij a_ij(D_i, D_j) and h_kl a_kl(D_k, D_l).", "caption": "Appendix A.7, p. 1943 (PDF p. 32). The marked sentence takes the terms $a_{ij}(D_i,D_j)$ of the variance estimator to be bounded by Condition 3, which bounds only the marginal exposure probabilities.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E001-b.png"}, {"alt": "Chart with the number of units N on a logarithmic horizontal axis and the coverage of the nominal 95 percent interval on the vertical axis, with ticks from 0.90 to 0.95. Three blue points show midpoints of exact coverage brackets in the counterexample: 0.9350 at N = 100, 0.9232 at N = 400 and 0.9063 at N = 2,000. A solid black horizontal line at 0.95 is labelled as the nominal level that Proposition 6.2 states as a lower bound for large N. A dashed blue horizontal line at 0.9104 is labelled as the limit of the coverage in the counterexample as N grows. All three points lie below the solid line.", "caption": "The counterexample for P013-E001: coverage of the nominal 95% interval of Proposition 6.2 for $N=100$, $400$ and $2{,}000$ units ($n=50$, $200$ and $1{,}000$ pairs, critical value $1.96$), with the nominal level (solid line) and the limit $0.9104$ of the coverage (dashed line). Each point is the midpoint of an exact bracket of width below $0.0002$. The bounds use binomial probabilities and need not approach the limit monotonically.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E001-coverage.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T1", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T1"}, {"id": "P013-T2", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T2"}, {"id": "P013-T18", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T18"}], "headline": "Proposition 6.2 needs a further condition, that the positive joint exposure probabilities that enter the variance estimator of equation (11) are bounded away from zero; Conditions 3, 5 and 6 alone do not imply the stated coverage.", "id": "P013-E001", "location": "Proposition 6.2, p. 1926 (PDF p. 15); Condition 3, p. 1925 (PDF p. 14); Appendix A.7, p. 1943 (PDF p. 32)", "pdf_page": 15, "printed_page": "1926", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "result_needs_same_repair", "effect_text": "a result or code path there needs the same repair", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P013-T1", "P013-T18", "P013-T2"], "text": {"as_printed": "Condition 3 (p.~1925): ``Potential outcomes and exposure probabilities are bounded so that, for all values $i$ and $d_k$, $|y_i(d_k)|\\le c_1<\\infty$ and $|1/\\pi_i(d_k)|\\le c_2<\\infty$.''\n\nProposition 6.2 (p.~1926): ``\\hl{Given Conditions 3, 5, and 6}, Wald-type intervals constructed as\n\\[\n\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)\\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}\n\\]\nwill tend to cover $\\tau_{\\mathrm{HT}}(d_k,d_l)$ at least $100(1-\\alpha)\\%$ of the time for large $N$.''\n\nAppendix A.7 (p.~1943): ``Denote $a_{ij}(D_i,D_j)$ as the sum of the elements in $\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$ that incorporate observations $i$ and $j$. Note that all $a_{ij}(D_i,D_j)$ are bounded above by some finite constant \\hl{by Condition 3}:\n\\[\n\\begin{aligned}\n\\operatorname{Var}\\bigl[N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]\\bigr]\n&\\le N^{-2}\\operatorname{Var}\\Bigl[\\sum_{i=1}^{N}\\sum_{j=1}^{N}h_{ij}a_{ij}(D_i,D_j)\\Bigr]\\\\\n&=N^{-2}\\sum_{i=1}^{N}\\sum_{j=1}^{N}\\sum_{k=1}^{N}\\sum_{l=1}^{N}\\operatorname{Cov}\\bigl[h_{ij}a_{ij}(D_i,D_j),h_{kl}a_{kl}(D_k,D_l)\\bigr].\n\\end{aligned}\n\\]\n\\ldots\\ Thus, $\\operatorname{Var}[N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]]=O(N^{-2}\\times N)=O(N^{-1})$.''", "corrected": "Proposition 6.2. Let Conditions 3 and 6 hold, and let Condition 5 hold with $h_{ij}$ the indicator of a dependency graph, as corrected in \\fid{P013-E002}. \\hl{Suppose in addition that there exists a constant} $\\hl{b>0}$\\hl{, not depending on} $\\hl{N}$\\hl{, that bounds from below every positive joint exposure probability entering equations (7) and (10)}: for all $N$ and all $i\\ne j$, if $\\pi_{ij}(d_k)>0$ then $\\pi_{ij}(d_k)\\ge b$, and likewise for $\\pi_{ij}(d_l)$; and for all $N$ and all $i,j$, if $\\pi_{ij}(d_k,d_l)>0$ then $\\pi_{ij}(d_k,d_l)\\ge b$. Joint probabilities equal to zero remain permitted, with the corrections of Section 5. Write $\\widehat{V}_N=\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$ and $\\hl{\\widehat{V}_N^+=\\max\\{0,\\widehat{V}_N\\}}$ to define the interval when the variance estimate is negative. Then the Wald-type intervals constructed as\n\\[\n\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)\\pm z_{1-\\alpha/2}\\sqrt{\\hl{\\widehat{V}_N^+}}\n\\]\nsatisfy\n\\[\n\\hl{\\liminf_{N\\to\\infty}}\\,P\\Bigl\\{\\bigl|\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)-\\tau(d_k,d_l)\\bigr|\\le z_{1-\\alpha/2}\\sqrt{\\hl{\\widehat{V}_N^+}}\\Bigr\\}\\ge1-\\alpha .\n\\]\nThe target is written $\\tau(d_k,d_l)$, as in \\fid{P013-E012}.\n\nThe added condition may be replaced by the weaker one that, for every $\\delta>0$,\n\\[\nP\\bigl\\{N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]<N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]-\\delta\\bigr\\}\\to0 .\n\\]\nThis holds if $N(\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]-\\mathrm{E}[\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]])\\to0$ in probability, and also if $Nb_N\\to\\infty$, where $b_N$ is the smallest positive joint exposure probability entering (7) and (10) at size $N$.\n\nAppendix A.7: ``Note that all $a_{ij}(D_i,D_j)$ are bounded above by some finite constant by Condition 3 \\hl{and the lower bound on the positive joint exposure probabilities}.''", "evidence": "The evidence is an exact counterexample to the printed statement and a derivation of the corrected one.\n\nFor $N=2n$, let the units form $n$ pairs that are randomized independently. Within a pair, the assignments $(0,0)$ and $(1,1)$ each have probability $\\epsilon_n=1/(4n^3)$, and $(1,0)$ and $(0,1)$ each have probability $q_n=1/2-\\epsilon_n$. There is no interference: $D_i=Z_i$, with $d_k=1$ and $d_l=0$. Both potential outcomes equal $1$ for the first unit of each pair and $-1$ for the second, and the average effect is $0$. The outcomes are bounded and every marginal exposure probability is $1/2$, and Condition 3 holds with $c_1=1$ and $c_2=2$. Since the pairs are independent, Condition 5 holds with $m=2$, for a genuine dependency graph. Further, I have $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}]=8(1-2\\epsilon_n)\\to8$, and Condition 6 holds. All joint exposure probabilities of distinct units are positive, and those of equal exposures within a pair equal $\\epsilon_n\\to0$.\n\nLet $\\widehat{V}_N$ denote the estimator of equation (11). Substituting into (7), (10) and (11), I obtain $N^2\\widehat{V}_N=\\sum_{b=1}^{n}B_b$, where $B_b=2/\\epsilon_n$ if the two units of pair $b$ receive the same assignment and $B_b=16-2/q_n$ otherwise. Then $\\mathrm{E}[B_b]=16(1-2\\epsilon_n)$, and $\\widehat{V}_N$ is exactly unbiased. The probability that some pair receives equal assignments is at most $2n\\epsilon_n=1/(2n^2)\\to0$, and on the complementary event I have $N\\widehat{V}_N=8-1/q_n\\to6$. I conclude that $N\\widehat{V}_N\\to6$ in probability, while $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}]\\to8$. Further, I find that\n\\[\n\\operatorname{Var}[N\\widehat{V}_N]=\\frac{2(1-4\\epsilon_n)^4}{n\\epsilon_n(1-2\\epsilon_n)}\\sim8n^2,\n\\]\ncontradicting the concentration bound in Appendix A.7. Since $\\sqrt{N}\\,\\widehat{\\tau}_{\\mathrm{HT}}$ is asymptotically normal with mean $0$ and variance $8$, the coverage of the nominal $100(1-\\alpha)\\%$ interval tends to\n\\[\n2\\Phi\\bigl(z_{1-\\alpha/2}\\sqrt{3/4}\\bigr)-1<1-\\alpha,\n\\]\nwhich is $0.9104$ for $\\alpha=0.05$. A variant of the example with $\\epsilon=N^{-2}$ has the same limits.\n\n\\begin{audittable}{lll}{The counterexample at $N=6$ ($\\epsilon_3=1/108$), by exact enumeration of all 64 assignments, and in the limit.}\nQuantity & $N=6$ & Limit \\\\\n$N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}]$ & $212/27\\approx7.85$ & $8$ \\\\\n$\\mathrm{E}[N\\widehat{V}_N]$ & $212/27\\approx7.85$ & $8$ \\\\\n$N\\widehat{V}_N$ when every pair is split & $316/53\\approx5.96$ & $6$ \\\\\nProbability that every pair is split & $148877/157464\\approx0.945$ & $1$ \\\\\n\\end{audittable}\n\nFor the corrected statement, suppose in addition that every positive joint exposure probability that enters (7) and (10) is at least $b>0$, and let Condition 5 hold for a symmetric dependency graph with row sums at most $m$. I add every self-edge, writing $\\widetilde h_{ij}=1$ when $i=j$ or $h_{ij}=1$. This preserves the dependency condition and bounds every row sum by $m+1$. I write $N^2\\widehat{V}_N=\\sum_{i,j}B_{ij}(D_i,D_j)$, where $B_{ij}$ collects the terms that involve units $i$ and $j$. A term with positive joint probability is bounded in absolute value by $(1+1/b)c_1^2c_2^2$, and the corrections for zero joint probabilities divide only by marginal probabilities. Then there exists a constant $C$, which does not depend on the population size, such that $|B_{ij}|\\le C$ for every pair of units. If $\\widetilde h_{ij}=0$, then the exposures of $i$ and $j$ are independent, the joint probabilities are products of marginal probabilities, and $B_{ij}=0$. Then at most $N(m+1)$ of the $B_{ij}$ are nonzero. Two of them can be correlated only if a unit of one is adjacent under $\\widetilde h$ to a unit of the other, which leaves at most $4N(m+1)^3$ nonzero covariances, each at most $4C^2$ in absolute value. I obtain\n\\[\n\\operatorname{Var}[N\\widehat{V}_N]\\le N^{-2}\\,(4N(m+1)^3)(4C^2)=16C^2(m+1)^3/N\\to0.\n\\]\nLet $c_N=N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]\\to c>0$. Proposition 5.6 implies that $\\mathrm{E}[N\\widehat{V}_N]\\ge c_N$. By Chebyshev's inequality, I have $P\\{N\\widehat{V}_N<c_N-\\delta\\}\\to0$ for every $\\delta>0$. The statistic $T_N=\\sqrt{N}(\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)-\\tau(d_k,d_l))/\\sqrt{c_N}$ is asymptotically standard normal by the central limit theorem that Appendix A.7 invokes, which involves no joint probability. For $0<\\delta<c$, on the event $N\\widehat{V}_N\\ge c_N-\\delta$ the interval covers whenever $|T_N|\\le z_{1-\\alpha/2}\\sqrt{1-\\delta/c_N}$. Then the lower limit of the coverage is at least $2\\Phi(z_{1-\\alpha/2}\\sqrt{1-\\delta/c})-1$. Letting $\\delta\\downarrow0$, I obtain the lower bound $1-\\alpha$. The argument does not require that $\\mathrm{E}[N\\widehat{V}_N]$ converge.\n\nThe added condition holds automatically for the following designs described in the paper. Consider first Bernoulli assignment at a fixed rate $p$ with first-order interference on a graph of degree at most $\\Delta$, a fixed-rate version of the illustration given after Condition 5. A joint exposure event of two units is determined by at most $2(\\Delta+1)$ independent assignments, and every positive joint probability is at least $\\min(p,1-p)^{2(\\Delta+1)}$. In the design of Section 9, a fixed fraction of each school is treated uniformly and schools are randomized independently, and the bound follows from Condition 3. In the block-randomized design of Section 10, the blocks are randomized uniformly at fixed quotas and a school has at most 64 eligible students. Condition 3 then implies that every positive joint probability is at least $c_2^{-2}2^{-129}$. For arbitrary known designs the condition is a further restriction. The counterexample above, which has independent groups of two units, does not satisfy it.", "problem": "Proposition 6.2 states that, under Conditions 3, 5 and 6, Wald intervals built from the variance estimator of equation (11) are asymptotically conservative. Condition 3 bounds the potential outcomes and the inverse marginal exposure probabilities $1/\\pi_i(d_k)$. Through equations (7) and (10), the variance estimator also divides by the positive joint exposure probabilities $\\pi_{ij}(d_k)$, $\\pi_{ij}(d_l)$ and $\\pi_{ij}(d_k,d_l)$, and the three conditions do not keep these away from zero. The step of Appendix A.7 that takes every $a_{ij}(D_i,D_j)$ to be bounded ``by Condition 3'' is therefore not justified. The bound $O(N^{-1})$ on the variance of the scaled variance estimator rests on that step, and it can fail. When it fails, the variance estimator can be unbiased and still converge in probability to a value below the true scaled variance, because the estimate is large on rare assignments. The interval then covers less often than its nominal level in the limit. This is so even when Condition 5 is read as a dependency graph of bounded degree, that is, after the correction of \\fid{P013-E002}.", "what_survives": "The estimand, the Horvitz--Thompson point estimator and the variance estimator of equation (11) with its corrections for zero joint probabilities are unchanged. The results on unbiasedness in finite samples, the exact variance and covariance formulas and Proposition 5.6 are unaffected; in the counterexample Proposition 5.6 holds with equality. The point estimator is consistent and asymptotically normal under a dependency graph of bounded degree, and these properties do not involve joint probabilities. With the added condition the coverage conclusion holds, and the condition is automatic for the Bernoulli illustration with a fixed rate and for the designs of Sections 9 and 10 wherever Conditions 3 and 5 hold for them. No entry of Table 1 or Table 2 changes, and no reported interval is shown to undercover. What is lost is the claim for arbitrary known designs under Conditions 3, 5 and 6 alone; the refinements of Section 7, which use equation (11), need the same control in addition to their own assumptions. The example has limiting coverage $0.9104$, and it is not established that coverage can tend to zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E001", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph that defines the binary dependency indicator h_ij, followed by Condition 5 (Local dependence). Red boxes mark the four 'not an element of' signs in clauses (i) j not in Gamma_2, (ii) i not in Gamma_2, (iii) i not in Gamma_1 and (iv) j not in Gamma_1.", "caption": "Section 6.2, definition of $h_{ij}$ and Condition 5, p. 1925 (PDF p. 14). The four marked signs are printed as $\\notin$ and should be $\\in$.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T2", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T2"}, {"id": "P013-T5", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T5"}, {"id": "P013-T6", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T6"}], "headline": "In the definition of the dependency indicator $h_{ij}$, the four signs $\\notin$ should read $\\in$.", "id": "P013-E002", "location": "Section 6.2, definition of $h_{ij}$ before Condition 5, p. 1925 (PDF p. 14)", "pdf_page": 14, "printed_page": "1925", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P013-T2", "P013-T5", "P013-T6"], "text": {"as_printed": "``Define a binary dependency indicator $h_{ij}$ over all pairs $(i,j)\\in\\{1,2,\\ldots,N\\}\\times\\{1,2,\\ldots,N\\}$, where $h_{ij}$ satisfies the following: for any pair of disjoint sets $\\Gamma_1$ and $\\Gamma_2\\subseteq\\{1,\\ldots,N\\}$ such that there exists no pair $(i,j)$ with $h_{ij}=1$ and either (i) $i\\in\\Gamma_1$ and $j\\hl{\\notin}\\Gamma_2$, (ii) $j\\in\\Gamma_1$ and $i\\hl{\\notin}\\Gamma_2$, (iii) $i\\hl{\\notin}\\Gamma_1$ and $j\\in\\Gamma_2$, or (iv) $j\\hl{\\notin}\\Gamma_1$ and $i\\in\\Gamma_2$, $\\{D_i,i\\in\\Gamma_1\\}$ and $\\{D_i,i\\in\\Gamma_2\\}$ are independent.''", "corrected": "Define a binary dependency indicator $h_{ij}$ over all pairs $(i,j)\\in\\{1,2,\\ldots,N\\}\\times\\{1,2,\\ldots,N\\}$, where $h_{ij}$ satisfies the following: for any pair of disjoint sets $\\Gamma_1$ and $\\Gamma_2\\subseteq\\{1,\\ldots,N\\}$ such that there exists no pair $(i,j)$ with $h_{ij}=1$ and either (i) $i\\in\\Gamma_1$ and $j\\hl{\\in}\\Gamma_2$, (ii) $j\\in\\Gamma_1$ and $i\\hl{\\in}\\Gamma_2$, (iii) $i\\hl{\\in}\\Gamma_1$ and $j\\in\\Gamma_2$, or (iv) $j\\hl{\\in}\\Gamma_1$ and $i\\in\\Gamma_2$, $\\{D_i,i\\in\\Gamma_1\\}$ and $\\{D_i,i\\in\\Gamma_2\\}$ are independent.\n\nFor symmetric $h$ the definition reads as follows. For every pair of disjoint sets $\\Gamma_1$ and $\\Gamma_2\\subseteq\\{1,\\ldots,N\\}$, if $h_{ij}=0$ for all $i\\in\\Gamma_1$ and $j\\in\\Gamma_2$, then the random vectors $(D_i)_{i\\in\\Gamma_1}$ and $(D_j)_{j\\in\\Gamma_2}$ are independent. Condition 5 is unchanged: there exists a finite constant $m$ such that, for all $U_N$ and all $i\\in\\{1,\\ldots,N\\}$, $\\sum_{j=1}^{N}h_{ij}\\le m$. Independence of each pair of nonadjacent units would not suffice; the vectors of exposures of the two sets must be independent.", "evidence": "The evidence is two exact counterexamples to the printed definition. First, let every unit receive the same fair Bernoulli exposure, $D_i=X$. With $h$ the identity matrix, the printed definition is satisfied and Condition 5 holds with $m=1$. But every pair of units is dependent, which implies that $\\sum_{i=1}^{N}\\sum_{j=1}^{N}g_{ij}=N^2$, and Condition 4 fails. Second, for $N=36r^2$ let the number $S$ of treated units take the values $18r^2-9r$, $18r^2$ and $18r^2+9r$ with probabilities $1/18$, $8/9$ and $1/18$, let the treated units be chosen uniformly given $S$, and let $D_i=Z_i$, $y_i(1)=1$ and $y_i(0)=0$. Distinct units are pairwise independent, and Conditions 1--4 hold. Condition 5 holds as printed, with $h$ the adjacency matrix of disjoint triangles and $m=2$. Condition 6 holds, since $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}]=1$ for every $r$. The estimator standardized by its true standard deviation takes the values $-3$, $0$ and $3$ with probabilities $1/18$, $8/9$ and $1/18$ for every $r$, which is not a normal limit. The interval of Proposition 6.2 with critical value $2$ has coverage exactly $8/9$ for every $r\\ge5$, which is below its nominal level. Neither example satisfies the corrected definition.\n\nApplying the corrected definition to two singletons, I find that any two dependent units are adjacent. Since the row sums are at most $m$, I obtain $\\sum_{i}\\sum_{j}g_{ij}\\le N(m+1)=O(N)$, which is the statement on p.~1926 that Condition 5 subsumes Condition 4.", "problem": "As printed, the two sets of exposures must be independent only when no edge of $h$ lies inside either set or leaves either set for the outside; edges between the two sets are allowed. This is not the condition that no edge joins the sets, and the requirement can be vacuous. If $h$ is the identity matrix, or the adjacency matrix of disjoint triangles, the premise holds only when both sets are empty. Every joint distribution of the exposures then satisfies the definition, with row sums $1$ or $2$. The printed definition therefore supports neither the statement on p.~1926 that Condition 5 subsumes Condition 4 nor the central limit theorem used for Proposition 6.2. The sentence before the definition, which announces ``joint independence of observations rather than pairwise independence'', and the description on p.~1926 of a dependency graph of limited degree show what is meant.", "what_survives": "The unbiasedness results, the exact variance formulas, Proposition 5.6 and Proposition 6.1, which uses Conditions 3 and 4, do not involve $h_{ij}$. With the four signs corrected the definition is the dependency-graph condition that p.~1926 describes and that Appendix A.7 uses, and no reported number depends on it. The correction does not by itself settle Proposition 6.2, which is the subject of \\fid{P013-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E002", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 4.1 and the paragraph after it. A red box marks the right-hand side of the displayed inequality, at most (1 minus pi_0(d_k)) to the power R + 1. A second red box marks the rule R greater than or equal to log(b) divided by log(1 minus pi_0(d_k)), minus 1, in the sentence that follows.", "caption": "Proposition 4.1 and the rule for the number of replicates, p. 1920 (PDF p. 9). The marked bound and the marked rule hold when all $y_i(d_k)$ have the same sign and $y^T(d_k)\\ne0$, restrictions the proposition does not state.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E003.png"}, {"alt": "Crop of the end of the proof of Proposition 4.1. After the words 'And so', a display equates the absolute bias with the absolute value of the sum over i of y_i(d_k) times (1 minus pi_i(d_k)) to the power R + 1. A red box marks the next line, at most the absolute value of y^T(d_k) times (1 minus pi_0(d_k)) to the power R + 1.", "caption": "Appendix A.3, last display, p. 1938 (PDF p. 27). The marked inequality is the step that can fail for outcomes of mixed sign.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T3", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T3"}, {"id": "P013-T4", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T4"}, {"id": "P013-T12", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T12"}], "headline": "The bound of Proposition 4.1 holds for potential outcomes of one sign with nonzero total; for general nonzero totals, multiplying its right-hand side by $\\sum_i|y_i(d_k)|/|y^T(d_k)|$ gives a valid bound, and the rule for the number of replicates changes in the same way.", "id": "P013-E003", "location": "Proposition 4.1 and the paragraph after it, p. 1920 (PDF p. 9); last inequality of Appendix A.3, p. 1938 (PDF p. 27)", "pdf_page": 9, "printed_page": "1920", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P013-T12", "P013-T3", "P013-T4"], "text": {"as_printed": "Proposition 4.1 (p.~1920): ``The relative bias for $\\widehat{y^T_{\\mathrm{HT},R}}(d_k)$ is bounded as\n\\[\n\\left|\\frac{\\mathrm{E}[\\widehat{y^T_{\\mathrm{HT},R}}(d_k)]-y^T(d_k)}{y^T(d_k)}\\right|\\le\\hl{(1-\\pi_0(d_k))^{R+1}},\n\\]\nwhere $\\pi_0(d_k)=\\min_i\\{\\pi_i(d_k)\\}$.\n\nFor a relative bias target of $b$ and given some approximation of $\\pi_0(d_k)$, the bound implies selecting a number of replicates $R\\ge\\log(\\hl{b})/\\log(1-\\pi_0(d_k))-1$. Thus, for $b=0.005$ and $\\pi_0(d_k)=0.0005$, this would imply at least 10,593 replicates.''\n\nAppendix A.3, last display (p.~1938):\n\\[\n\\left|\\mathrm{E}[\\widehat{y^T_{\\mathrm{HT},R}}(d_k)]-y^T(d_k)\\right|=\\left|\\sum_{i=1}^{N}y_i(d_k)(1-\\pi_i(d_k))^{R+1}\\right|\\le\\hl{|y^T(d_k)|}(1-\\pi_0(d_k))^{R+1}.\n\\]", "corrected": "Proposition 4.1. Let the $R$ replicates be drawn independently from the randomization plan and independently of the experiment, let $\\pi_0(d_k)=\\min_i\\{\\pi_i(d_k)\\}$ with $0<\\pi_i(d_k)<1$ for every $i$, and write $T=y^T(d_k)$ and $\\hl{L=\\sum_{i=1}^{N}|y_i(d_k)|}$. Then\n\\[\n\\left|\\mathrm{E}[\\widehat{y^T_{\\mathrm{HT},R}}(d_k)]-T\\right|\\le\\hl{L}\\,(1-\\pi_0(d_k))^{R+1}.\n\\]\nIf $T\\ne0$, the relative bias for $\\widehat{y^T_{\\mathrm{HT},R}}(d_k)$ is bounded as\n\\[\n\\left|\\frac{\\mathrm{E}[\\widehat{y^T_{\\mathrm{HT},R}}(d_k)]-T}{T}\\right|\\le\\hl{\\kappa}\\,(1-\\pi_0(d_k))^{R+1},\\qquad\\hl{\\kappa=L/|T|}.\n\\]\nIf all $y_i(d_k)$ have the same sign and $T\\ne0$, then $\\kappa=1$ and the bound is $(1-\\pi_0(d_k))^{R+1}$, as printed.\n\nFor $\\hl{T\\ne0}$ and a relative bias target of $\\hl{b>0}$, it suffices to select a \\hl{nonnegative integer} number of replicates $R\\ge\\log(\\hl{b/\\kappa})/\\log(1-\\pi_0(d_k))-1$, using $\\pi_0(d_k)$ \\hl{or a positive lower bound for it}, and $\\kappa$ or an upper bound for it. For outcomes of one sign this is the printed rule, and for $b=0.005$ and $\\pi_0(d_k)=0.0005$ it gives at least 10,593 replicates.\n\nAppendix A.3, last display:\n\\[\n\\left|\\mathrm{E}[\\widehat{y^T_{\\mathrm{HT},R}}(d_k)]-y^T(d_k)\\right|=\\left|\\sum_{i=1}^{N}y_i(d_k)(1-\\pi_i(d_k))^{R+1}\\right|\\le\\hl{\\sum_{i=1}^{N}|y_i(d_k)|}\\,(1-\\pi_0(d_k))^{R+1}.\n\\]", "evidence": "The evidence is an exact counterexample and a derivation of the corrected bound. Let $N=2$, with no interference, and let the two units be exposed to $d_k$ independently with probabilities $\\pi_1(d_k)=1/3$ and $\\pi_2(d_k)=2/3$. Let $y_1(d_k)=2$ and $y_2(d_k)=-1$, and let $R=1$ replicate be drawn independently of the experiment. Then $y^T(d_k)=1$. Summing over the 16 pairs of an actual and a simulated assignment, I obtain $\\mathrm{E}[\\widehat{y^T_{\\mathrm{HT},R}}(d_k)]=2/9$. The relative bias is $7/9$, and the printed bound is $(1-1/3)^2=4/9$. For the target $b=1/2$ the printed rule selects $R=1$, and the relative bias $7/9$ exceeds the target. With outcomes $(1,-1)$ in the same design, the total is $0$ and the expected estimate is $-1/3$.\n\n\\begin{audittable}{lll}{The two-unit example with $R=1$ and exposure probabilities $(1/3,2/3)$, by exact enumeration.}\nQuantity & Outcomes $(2,-1)$ & Outcomes $(1,-1)$ \\\\\nTotal $y^T(d_k)$ & $1$ & $0$ \\\\\nExpected estimate & $2/9$ & $-1/3$ \\\\\nAbsolute bias & $7/9$ & $1/3$ \\\\\nPrinted bound on the absolute bias & $4/9$ & $0$ \\\\\nCorrected bound on the absolute bias & $4/3$ & $8/9$ \\\\\n\\end{audittable}\n\nFor the corrected bound, I write $q_i=(1-\\pi_i(d_k))^{R+1}$ and $q_0=(1-\\pi_0(d_k))^{R+1}$, so that $0\\le q_i\\le q_0$. By the identity of Appendix A.3 and the triangle inequality, I have\n\\[\n\\left|\\mathrm{E}[\\widehat{y^T_{\\mathrm{HT},R}}(d_k)]-y^T(d_k)\\right|=\\Bigl|\\sum_{i=1}^{N}y_i(d_k)q_i\\Bigr|\\le\\sum_{i=1}^{N}|y_i(d_k)|q_i\\le q_0\\sum_{i=1}^{N}|y_i(d_k)|.\n\\]\nIf all $y_i(d_k)$ have the same sign and $y^T(d_k)\\ne0$, then $\\sum_i|y_i(d_k)|=|y^T(d_k)|$ and the printed bound follows. In the example, the corrected rule with $b=1/2$ gives $R=4$, for which the corrected bound is $32/81$ and the relative bias is $7/27$.", "problem": "Proposition 4.1 bounds the relative bias of the Horvitz--Thompson total computed with simulated exposure probabilities by $(1-\\pi_0(d_k))^{R+1}$, and it places no restriction on the signs of the potential outcomes. With independent replication, the expectation identities of Appendix A.3 are correct: the bias equals $-\\sum_i y_i(d_k)(1-\\pi_i(d_k))^{R+1}$. The last step of the proof bounds the absolute value of this weighted sum by $|y^T(d_k)|(1-\\pi_0(d_k))^{R+1}$. That step can fail when the outcomes have mixed signs and the exposure probabilities differ across units, because positive and negative outcomes can cancel in the total $y^T(d_k)$ without cancelling in the weighted sum. When $y^T(d_k)=0$ the relative bias is not defined. The context does not exclude this case, since the paper considers outcomes centred on zero. For mixed signs, the rule $R\\ge\\log(b)/\\log(1-\\pi_0(d_k))-1$ does not in general guarantee a relative bias of at most $b$, and no number of replicates chosen from $\\pi_0(d_k)$ and $b$ alone can guarantee it for all signed outcomes with nonzero total.", "what_survives": "The expectation identities of Appendix A.3 hold under independent replication, and the bias of the approximate total still tends to zero as $R$ grows in a fixed finite population. The unbiasedness of the Horvitz--Thompson estimator with exact probabilities (Lemma 4.1), Proposition 4.2 and the later variance and asymptotic results do not use Proposition 4.1. The arithmetic of the printed example is right: the threshold is $10{,}592.99$, and 10,593 is the smallest admissible integer. The outcomes of the simulation (counts multiplied by positive factors) and of the application (a binary outcome) are nonnegative, and the printed bound applies when the corresponding potential-outcome total is nonzero. It need not apply to totals of signed residuals. What changes is the scope of the proposition and of the rule for $R$."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E003", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of text and the first row of expression (12). A red box marks the condition 'Cov of g of x i, xi i of d k, and Z i, equals 0', which the sentence calls a sufficient condition for unbiasedness.", "caption": "Section 7.1, p. 1927 (PDF p. 16), the sentence before expression (12). The marked condition involves the unit’s own assignment $Z_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T16", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T16"}], "headline": "Expression (12) is unbiased when the predictions $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))$ are fixed before the randomization, which is the case that Section 7.1 describes; for predictions that vary with the assignment, a sufficient condition is zero covariance with the exposure indicator $\\mathbf{I}(D_i=d_k)$, and zero covariance with $Z_i$ is not enough.", "id": "P013-E004", "location": "Section 7.1, p. 1927 (PDF p. 16), sentence before expression (12)", "pdf_page": 16, "printed_page": "1927", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P013-T16"], "text": {"as_printed": "We have some predefined function $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))\\to\\mathbb{R}$, where $\\boldsymbol{\\xi}_i$ is a parameter vector. Ideally $g(\\cdot)$ is calibrated on auxiliary data to produce values that approximate $y_i(d_k)$. We assume $\\operatorname{Cov}[g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k)),\\hl{Z_i}]=0$ as a sufficient condition for unbiasedness. Define\n\\[\n(12)\\qquad \\widehat{y^T_G}(d_k)=\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)}-\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)\\frac{g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))}{\\pi_i(d_k)}+\\sum_{i=1}^N g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k)),\n\\]\nwhich is unbiased for $y^T(d_k)$ by\n\\[\n\\mathrm{E}\\left[-\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)\\frac{g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))}{\\pi_i(d_k)}+\\sum_{i=1}^N g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))\\right]=0.\n\\]", "corrected": "Let Conditions 1 and 2 hold and let $\\pi_i(d_k)>0$ for every $i$. If the predictions $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))$ are \\hl{fixed}, that is, if they do not vary with the assignment $\\mathbf{Z}$, then expression (12) is unbiased for $y^T(d_k)$. If the predictions are random, then\n\\[\n\\mathrm{E}\\bigl[\\widehat{y^T_G}(d_k)\\bigr]-y^T(d_k)=-\\sum_{i=1}^N\\frac{\\operatorname{Cov}[g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k)),\\mathbf{I}(D_i=d_k)]}{\\pi_i(d_k)},\n\\]\nand a sufficient condition for unbiasedness is\n\\[\n\\operatorname{Cov}[g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k)),\\hl{\\mathbf{I}(D_i=d_k)}]=0\\quad\\text{for every } i .\n\\]\nZero covariance with $Z_i$ alone does not suffice. Neither condition is necessary, since the weighted covariances in the sum may cancel.", "intended_meaning": "Section 7.1 describes a difference estimator whose predictions are settled before the experiment. The function $g$ is ``predefined'' and ideally ``calibrated on auxiliary data'', the variance paragraph on the same page requires that $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))$ ``is fixed'', and the regression fitted to the experimental data is treated separately, with the warning that ``unbiasedness is compromised''. For fixed predictions $g_i$, expression (12) is unbiased under the standing assumptions of the paper, with no further condition. Indeed, since $\\mathrm{E}[\\mathbf{I}(D_i=d_k)]=\\pi_i(d_k)$, I have\n\\[\n\\mathrm{E}\\left[g_i-\\mathbf{I}(D_i=d_k)\\frac{g_i}{\\pi_i(d_k)}\\right]=g_i-g_i\\,\\frac{\\pi_i(d_k)}{\\pi_i(d_k)}=0 .\n\\]\nUnder this reading the printed covariance condition is redundant, and the display after (12) is correct.\n\nRead literally, as a condition that admits predictions depending on the assignment, the sentence is too weak. For random predictions $G_i$, a direct computation gives\n\\[\n\\mathrm{E}\\bigl[\\widehat{y^T_G}(d_k)\\bigr]-y^T(d_k)=-\\sum_{i=1}^N\\frac{\\operatorname{Cov}[G_i,\\mathbf{I}(D_i=d_k)]}{\\pi_i(d_k)} .\n\\]\nUnder interference the exposure $D_i=f(\\mathbf{Z},\\theta_i)$ depends on the assignments of other units, and zero covariance with $Z_i$ does not imply that the right side vanishes. An exact example shows this. Let $N=2$, let $Z_1$ and $Z_2$ be independent and uniform on $\\{1,2\\}$, and let $D_1=Z_2$ and $D_2=Z_1$. Let $d_k=2$, with potential outcomes $y_1(2)=3$, $y_2(2)=5$ and $y_1(1)=y_2(1)=0$, and let $G_1=Z_2-1$ and $G_2=Z_1-1$. Then $\\operatorname{Cov}[G_i,Z_i]=0$ for both units, while $\\operatorname{Cov}[G_i,\\mathbf{I}(D_i=2)]=1/4$. Expression (12) takes the values $0$, $5$, $9$ and $14$ with probability $1/4$ each, and its expectation is $7$. The target total is $8$. With the fixed predictions $G_1=G_2=1$ in the same design, expression (12) takes the values $2$, $6$, $10$ and $14$, with expectation $8$.\n\nThis example establishes no bias in any reported estimate; the fitted regression estimator is treated separately in expression (14). The simulation of Section 9 uses the Horvitz--Thompson, Hajek, weighted least squares and ordinary least squares estimators and the simple difference in sample means, and the application of Section 10 uses the first three."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E004", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the first paragraph of Section 7.1, from 'Chapter 6]. Covariance adjustment of this variety' to 'perhaps obtained from analysis of auxiliary datasets.' Red boxes mark the words 'the difference estimator addresses the problem of location noninvariance that afflicts Horvitz-Thompson-type estimators', which run over two lines.", "caption": "Section 7.1, first paragraph, p. 1927 (PDF p. 16). The marked sentence says, without a condition, that the difference estimator addresses location noninvariance.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E005.png"}, {"alt": "Crop of expression (13) and of the four lines of text after it. Expression (13) writes the estimator y-hat-T-G of d_k as the sum over i of I(D_i = d_k) times epsilon_i(d_k) divided by pi_i(d_k), plus the sum over i of g(x_i, xi_i(d_k)). A red box marks the words 'is location invariant' in the sentence 'Middleton and Aronow (2011) and Aronow and Middleton (2013) demonstrate that y-hat-T-G of d_k is location invariant'. The next sentence says that variance estimation proceeds as in Section 5 as long as g(x_i, xi_i(d_k)) is fixed.", "caption": "Expression (13) and the sentences after it, p. 1927 (PDF p. 16). The marked words state the property for $\\widehat{y^T_G}(d_k)$ without a condition on the predictions.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E005-b.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T21", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T21"}, {"id": "P013-T22", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T22"}], "headline": "The difference estimator of expressions (12) and (13) is location equivariant if a constant added to every outcome is also added to every prediction; Section 7.1 states the property without this condition, and it fails in general if the predictions are held numerically fixed, because the estimated total then changes by the constant times the realized sum of the weights $\\mathbf{I}(D_i=d_k)/\\pi_i(d_k)$, which need not equal the constant times $N$.", "id": "P013-E005", "location": "Section 7.1, p. 1927 (PDF p. 16): first paragraph, and the sentence after expression (13)", "pdf_page": 16, "printed_page": "1927", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P037/", "where": null, "work_id": "P037"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P013-T21", "P013-T22"], "text": {"as_printed": "``In addition, the difference estimator \\hl{addresses the problem of location noninvariance} that afflicts Horvitz--Thompson-type estimators [Fuller (2009), pages 9--10].''\n\n``We have some predefined function $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))\\to\\mathbb{R}$, where $\\boldsymbol{\\xi}_i$ is a parameter vector. Ideally $g(\\cdot)$ is calibrated on auxiliary data to produce values that approximate $y_i(d_k)$. \\ldots\\ Define\n\\[\n(12)\\qquad \\widehat{y^T_G}(d_k)=\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)}-\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)\\frac{g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))}{\\pi_i(d_k)}+\\sum_{i=1}^N g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k)),\n\\]\n\\ldots''\n\n``Middleton and Aronow (2011) and Aronow and Middleton (2013) demonstrate that $\\widehat{y^T_G}(d_k)$ \\hl{is location invariant}. Variance estimation proceeds as in Section 5 using $\\varepsilon_i(d_k)$ in place of $y_i(d_k)$ as long as $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))$ is fixed.''", "corrected": "First paragraph: ``In addition, the difference estimator addresses the problem of location noninvariance that afflicts Horvitz--Thompson-type estimators [Fuller (2009), pages 9--10]\\hl{, provided that the predictions are shifted by the same constant as the outcomes}.''\n\nAfter expression (13): ``Middleton and Aronow (2011) and Aronow and Middleton (2013) demonstrate that $\\widehat{y^T_G}(d_k)$ is location \\hl{equivariant when adding a constant $c$ to all outcomes also adds $c$ to all predictions}.''\n\nIn full, the corrected statement is as follows. Let Conditions 1 and 2 hold, let $\\pi_i(d_k)>0$ and $\\pi_i(d_l)>0$ for every $i$, and let the predictions $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))$ and $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_l))$ not vary with the assignment. Suppose that a constant $c$ is added to every potential outcome and \\hl{that $c$ is also added to every prediction}. Then, on every assignment, $\\widehat{y^T_G}(d_k)$ increases by $cN$, the estimated mean $\\widehat{y^T_G}(d_k)/N$ increases by $c$, and the estimated contrast $[\\widehat{y^T_G}(d_k)-\\widehat{y^T_G}(d_l)]/N$ does not change. If instead the outcomes are shifted and the predictions are held numerically fixed, then $\\widehat{y^T_G}(d_k)$ increases by\n\\[\nc\\sum_{i=1}^N\\frac{\\mathbf{I}(D_i=d_k)}{\\pi_i(d_k)},\n\\]\nwhich is not $cN$ in general.\n\nThe condition is sufficient and not necessary. If the prediction of unit $i$ changes by $h_i$, the estimated total increases by $cN$ on an assignment exactly when $\\sum_{i=1}^N(1-w_i)(h_i-c)=0$, where $w_i=\\mathbf{I}(D_i=d_k)/\\pi_i(d_k)$. For predictions held numerically fixed and $c\\ne0$, this condition reads $\\sum_{i=1}^N w_i=N$, and the estimated contrast between $d_k$ and $d_l$ is unchanged exactly when the two realized sums of the weights are equal. For example, if a fixed number $m_k$ of the $N$ units receive exposure $d_k$ and $\\pi_i(d_k)=m_k/N$ for every $i$, the weights sum to $N$ on every assignment.", "evidence": "The evidence is a derivation of the exact change of the estimator under a shift of the outcomes, an exact counterexample with the predictions held numerically fixed, and a check of the cited works.\n\nLet $g_i=g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))$, $w_i=\\mathbf{I}(D_i=d_k)/\\pi_i(d_k)$ and $W_k=\\sum_{i=1}^N w_i$. By Condition 2, I have\n\\[\n\\widehat{y^T_G}(d_k)=\\sum_{i=1}^N w_i\\,\\bigl(y_i(d_k)-g_i\\bigr)+\\sum_{i=1}^N g_i .\n\\]\nSuppose that every potential outcome increases by $c$ and that each prediction $g_i$ changes by $h_i$, while the design, the exposures and the covariates stay the same. Substituting into the display, I obtain that the estimated total changes by\n\\[\n\\sum_{i=1}^N w_i\\,(c-h_i)+\\sum_{i=1}^N h_i=cW_k+\\sum_{i=1}^N(1-w_i)h_i .\n\\]\nSince $\\sum_{i=1}^N(1-w_i)=N-W_k$, this change differs from $cN$ by\n\\[\n\\sum_{i=1}^N(1-w_i)(h_i-c).\n\\]\nIf $h_i=c$ for every $i$, each term vanishes. The residuals $y_i(d_k)-g_i$ are unchanged, the estimated total increases by $cN$ on every assignment, and the estimated contrast $[\\widehat{y^T_G}(d_k)-\\widehat{y^T_G}(d_l)]/N$ between two exposures does not change. If $h_i=0$ for every $i$, the estimated total increases by $cW_k$, the error is $c\\,(W_k-N)$, and the estimated contrast changes by $c\\,(W_k-W_l)/N$. Since $\\mathrm{E}[w_i]=1$, the error has expectation zero, but for $c\\ne0$ it vanishes on a given assignment only if $W_k=N$.\n\nFor the counterexample, let $N=3$ with no interference. One unit is selected, with probabilities $1/4$, $1/4$ and $1/2$ for the three units. For the selected unit $D_i=d_k$, and for the other two $D_i=d_l$. Then $\\pi_i(d_k)$ equals $1/4$, $1/4$, $1/2$ and $\\pi_i(d_l)$ equals $3/4$, $3/4$, $1/2$, and Conditions 1 and 2 hold. Let the potential outcomes be $(2,3,5)$ under both exposures, and let the predictions be the fixed numbers $(2,3,5)$ for both exposures. The predictions do not vary with the assignment, their covariance with $Z_i$ is zero, and they equal the outcomes. Both estimated totals equal $10$ on every assignment, and the estimated contrast is $0$. I now add $c=1$ to every potential outcome and keep the predictions. Both true totals become $13$, and the true contrast $\\tau(d_k,d_l)$ remains $0$. Every residual equals $1$, and the estimated totals are $10+W_k$ and $10+W_l$.\n\n\\begin{audittable}{crrrrrr}{The three-unit example after $1$ is added to every outcome, with the predictions held numerically fixed at $(2,3,5)$: the realized sums of the weights, the two estimated totals and the estimated contrast. Both true totals equal $13$ and the true contrast equals $0$.}\nSelected unit & Probability & $W_k$ & $W_l$ & $\\widehat{y^T_G}(d_k)$ & $\\widehat{y^T_G}(d_l)$ & Estimated contrast \\\\\n1 & $1/4$ & $4$ & $10/3$ & $14$ & $40/3$ & $2/9$ \\\\\n2 & $1/4$ & $4$ & $10/3$ & $14$ & $40/3$ & $2/9$ \\\\\n3 & $1/2$ & $2$ & $8/3$ & $12$ & $38/3$ & $-2/9$ \\\\\n\\end{audittable}\n\nNeither estimated total increases by $cN=3$ on any assignment, and the estimated contrast is never $0$. Both estimated totals have expectation $13$ and remain unbiased. The variance of the estimated contrast rises from $0$ to $4/81$. If $1$ is also added to every prediction, both estimated totals equal $13$ and the estimated contrast equals $0$ on every assignment. Conditions 1--6 do not restore the property. Independent copies of this design, with every outcome and every prediction equal to $1$, form a sequence that satisfies them with $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]=4/27$, and a shift of the outcomes alone still changes the estimated contrast.\n\nWhere I could consult the cited works, they state the property with the condition. Aronow and Middleton (2013) recode the outcomes as $Y_i^*=b_0+b_1Y_i$ (p.~139) and, in their Appendix C (p.~151), require that the ``predictive function will also change by the identical transformation''. Their equation (41) is $f(X_i;\\theta_i)^*=b_0+b_1f(X_i;\\theta_i)$. The published version of the 2011 paper (Middleton and Aronow, 2015, Section 5.2) has the same requirement: ``the same transformation must be applied to $k$'', with $k^*=b_0+b_1k$. The 2011 conference paper itself was not obtained.", "problem": "Section 7.1 says twice that the difference estimator does not depend on the origin of the outcome scale in the way that Horvitz--Thompson estimators do. The first statement is in its first paragraph. The second follows expression (13) and attributes the property to two cited works. Neither states a condition. For an estimated total the property is equivariance. If a constant $c$ is added to every potential outcome, the estimated total should increase by $cN$, and the estimated contrast between two exposures should not change. Whether expression (12) has this property depends on what happens to the predictions $g(\\mathbf{x}_i,\\boldsymbol{\\xi}_i(d_k))$ when the outcomes are shifted, and the section does not say. It calls the predictions ``predefined'' and ``fixed'', and these words refer to the randomization, not to the origin of the outcome scale. If the predictions are shifted by $c$ with the outcomes, the property holds. If they are held numerically fixed, which the printed assumptions allow, it fails in general. The estimated total then increases by $cW_k$, where $W_k=\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)/\\pi_i(d_k)$ is the realized sum of the weights. The error $c\\,(W_k-N)$ varies with the assignment. Aronow and Middleton (2013) show the property for predictions that are recoded by the same transformation as the outcomes. The sentence after expression (13) reports this conditional result without its condition.", "what_survives": "Expressions (12) and (13), the estimands and the exposure mappings are unchanged. For fixed predictions, expression (12) is unbiased, the variance calculation with $\\varepsilon_i(d_k)$ in place of $y_i(d_k)$ stands, and the adjustment can still reduce the variance. No later result of the paper uses the property. The regression estimators, the Hajek estimator and the difference in sample means that the paper reports have the property without an assumption on predefined predictions. The weighted least squares fit of expression (14) with an exposure intercept shifts its fitted values by $c$ when the fit is identified. The Hajek ratio of expression (15) and the sample means shift by $c$ when their exposure groups are not empty. In the ordinary least squares regression on exposure indicators and degree, the contrasts between exposures are unchanged when they are estimable. The simulation of Section 9 and the application of Section 10 use these estimators and the Horvitz--Thompson estimator, for which the paper does not claim the property. They do not use expression (12) with predefined predictions, and no entry of Table 1 or Table 2 is shown to be wrong. Under Conditions 3 and 4, $W_k/N$ tends to $1$ in probability. Under a shift of the outcomes alone, the change $c\\,(W_k-W_l)/N$ of an estimated contrast then tends to zero in probability, but it can remain of the order of the sampling error. What is lost is the unconditional statement that predefined predictions remove the dependence on the origin of the outcome scale."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E005", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Condition 6 (Nonzero limiting variance) and the paragraph after it. Red boxes mark the word 'Convergence' and the words 'is generally ensured by Conditions 3 and 5' in the sentence 'Convergence of N Var[tau-hat-HT(d_k, d_l)] to a non-negative constant is generally ensured by Conditions 3 and 5, sufficient for root-n consistency'.", "caption": "Condition 6 and the paragraph after it, p. 1926 (PDF p. 15). The marked words say that Conditions 3 and 5 ensure convergence, where they ensure boundedness.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E006.png"}], "formal_release": null, "formal_statements": [], "headline": "Conditions 3 and 5 make $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$ bounded, not convergent; Condition 6 assumes both that the limit exists and that it is positive.", "id": "P013-E006", "location": "Condition 6 and the paragraph after it, p. 1926 (PDF p. 15)", "pdf_page": 15, "printed_page": "1926", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Condition 6 (Nonzero limiting variance.). $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]\\xrightarrow{p}c$, where $c>0$.\n\n\\hl{Convergence} of $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$ to a non-negative constant \\hl{is generally ensured by Conditions 3 and 5}, sufficient for root-$n$ consistency of $\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)$. Condition 6 is a mild regularity condition that ensures that this constant is positive, and rules out degenerate cases (e.g., all outcomes are zero).''", "corrected": "Conditions 3 and 5 imply \\hl{that} $\\hl{N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]=O(1)}$, sufficient for root-$N$ consistency of $\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)$. Condition 6 \\hl{additionally assumes that this sequence converges to a positive constant}, and rules out degenerate cases (e.g., all outcomes are zero).", "intended_meaning": "The paragraph explains why a positive limiting variance is assumed. With Condition 5 read as a dependency graph of bounded degree, Conditions 3 and 5 leave at most $(m+1)N$ nonzero covariance terms, each bounded, so that $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]\\le (m+1)(2c_1c_2)^2$. The scaled variance is bounded, which suffices for the root-$N$ rate, but it need not converge. For example, with $y_i(d_k)=1$, $y_i(d_l)=0$ and independent assignments with exposure probability $p_N=1/2$ for even $N$ and $1/4$ for odd $N$, I have $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]=(1-p_N)/p_N$, which equals $1$ and $3$ along the two subsequences. Condition 6 itself assumes the limit and its positivity, and Proposition 6.2 lists Condition 6 among its hypotheses. No assumption is missing. The arrow $\\xrightarrow{p}$ in Condition 6 is applied to a deterministic sequence and denotes ordinary convergence."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E006", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the six lines that end the proof of Proposition 6.2. The text says that under Conditions 3, 5, and 6, by Chen and Shao (2004), Theorem 2.7, the estimator minus its target, divided by the square root of its variance, is asymptotically N(0, 1), while the square root of the ratio of the variance to the estimated variance converges in probability to a quantity in (0, 1]. It continues: By Slutsky's theorem, t is asymptotically normal and Wald-type confidence intervals constructed as the estimator plus or minus z times the square root of the estimated variance will tend to cover the target at least 100(1 - alpha) percent of the time as N tends to infinity. Red outlines surround the words converges in probability to a quantity in (0, 1] and the words t is asymptotically normal.", "caption": "Appendix A.7, last paragraph, p. 1943 (PDF p. 32). The outlines mark the claim that the variance ratio converges in probability to a quantity in $(0,1]$ and the claim that $t$ is asymptotically normal.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E007.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T7", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T7"}, {"id": "P013-T8", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T8"}], "headline": "Even when the conservative variance estimator concentrates around its expectation, it need not converge, and the variance ratio and the studentized statistic $t$ need not have limits; coverage of at least $1 - \\alpha$ in the limit inferior follows from a one-sided bound once concentration and the true-variance central limit theorem hold.", "id": "P013-E007", "location": "Appendix A.7, proof of Proposition 6.2, first and last paragraphs, p. 1943 (PDF p. 32)", "pdf_page": 32, "printed_page": "1943", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P013-T7", "P013-T8"], "text": {"as_printed": "\\emph{First paragraph of A.7.} By Proposition 5.6, $\\mathrm{E}[N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]]\\ge N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$. Thus, by Chebyshev's inequality, $\\operatorname{Var}[N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]]\\xrightarrow{p}0$ is sufficient to establish \\hl{convergence} of $N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$ \\hl{to a value} greater than or equal to $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$, which is itself nonzero by Condition 6.\n\n\\emph{Last paragraph of A.7.} Define\n\\[\nt=\\frac{\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)-\\tau_{\\mathrm{HT}}(d_k,d_l)}{\\sqrt{\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}}=\\frac{\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)-\\tau_{\\mathrm{HT}}(d_k,d_l)}{\\sqrt{\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}}\\left(\\frac{\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}{\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}\\right)^{1/2}.\n\\]\nUnder Conditions 3, 5, and 6, then, by Chen and Shao [(2004), Theorem 2.7], $\\frac{\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)-\\tau_{\\mathrm{HT}}(d_k,d_l)}{\\sqrt{\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}}$ is asymptotically $\\mathrm{N}(0,1)$, while $(\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]/\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)])^{1/2}$ \\hl{converges in probability to a quantity in} $(0,1]$. By Slutsky's theorem, $t$ \\hl{is asymptotically normal} and Wald-type confidence intervals constructed as $\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)\\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}$ will tend to cover $\\tau_{\\mathrm{HT}}(d_k,d_l)$ at least $100(1-\\alpha)\\%$ of the time as $N\\to\\infty$.", "corrected": "\\emph{First paragraph of A.7.} By Proposition 5.6, $\\mathrm{E}[N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]]\\ge N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$. Thus, by Chebyshev's inequality, $\\operatorname{Var}[N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]]\\xrightarrow{p}0$ is sufficient to establish \\hl{concentration} of $N\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$ \\hl{around its expectation, which is} greater than or equal to $N\\operatorname{Var}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]$, which is itself nonzero by Condition 6. \\hl{The expectation need not converge, and no limit of the variance estimator is used below.}\n\n\\emph{Last paragraph of A.7.} Let $\\widehat\\tau_N = \\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)$ and $\\tau_N = \\tau_{\\mathrm{HT}}(d_k,d_l)$, and set $V_N = \\operatorname{Var}[\\widehat\\tau_N]$, $\\widehat V_N = \\widehat{\\operatorname{Var}}[\\widehat\\tau_N]$, $v_N = N V_N$, $W_N = N \\widehat V_N$ and $Z_N = (\\widehat\\tau_N - \\tau_N)/\\sqrt{V_N}$. Under Conditions 3, 5, and 6, then, by Chen and Shao [(2004), Theorem 2.7], $Z_N$ is asymptotically $\\mathrm{N}(0,1)$. By Condition 6, $v_N \\to c > 0$; by Proposition 5.6, $\\mathrm{E}[W_N] \\ge v_N$; \\hl{for this step, assume the concentration premise} $\\operatorname{Var}(W_N) \\to 0$. For each $\\epsilon \\in (0,1)$, Chebyshev's inequality gives\n\\[\n\\hl{\\Pr\\bigl( W_N < (1-\\epsilon) v_N \\bigr) \\le \\frac{\\operatorname{Var}(W_N)}{\\epsilon^2 v_N^2} \\longrightarrow 0 ,}\n\\]\nI \\hl{count a negative variance estimate as noncoverage}; the bound shows that its probability tends to zero. On the complementary event $\\sqrt{\\widehat V_N/V_N} \\ge \\sqrt{1-\\epsilon}$. With $z = z_{1-\\alpha/2}$, it follows that\n\\[\n\\hl{\\liminf_{N \\to \\infty} \\Pr\\bigl( |\\widehat\\tau_N - \\tau_N| \\le z \\sqrt{\\widehat V_N} \\bigr) \\ge 2 \\Phi\\bigl( z \\sqrt{1-\\epsilon} \\bigr) - 1 .}\n\\]\nLetting $\\epsilon \\downarrow 0$, Wald-type confidence intervals constructed as $\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)\\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}$ cover $\\tau_{\\mathrm{HT}}(d_k,d_l)$ with probability at least $1-\\alpha$ \\hl{in the limit inferior} as $N\\to\\infty$. \\hl{No limit is asserted for the variance ratio or for the statistic} $t$.", "evidence": "The evidence is an exact counterexample to the two printed claims and a derivation of the corrected bound.\n\n\\emph{Counterexample.} Set $n_0 = 0$ and $n_r = 2^{2^r}$ for $r \\ge 1$. The units are the positive integers, the population of size $N$ is $\\{1, \\ldots, N\\}$, and block $r$ consists of the units $n_{r-1}+1, \\ldots, n_r$. There are two exposures, $d_k = 1$ and $d_l = 0$. In a block with $r$ odd, every unit has potential outcomes $(y_i(1), y_i(0)) = (1,1)$. In a block with $r$ even, the units alternate between $(2,0)$ and $(0,2)$, beginning with $(2,0)$. The assignments $Z_i$ are independent with $\\Pr(Z_i = 1) = 1/2$, and $D_i = Z_i$. Conditions 1 and 2 hold. Since the outcomes are at most $2$ and every exposure probability is $1/2$, Condition 3 holds. With $h_{ij} = \\mathbf{1}\\{i = j\\}$ and $m = 1$, Condition 5 holds, and every joint exposure probability of two distinct units equals $1/4$.\n\nLet $\\widehat\\tau_N$ and $\\tau_N$ be the estimator and the estimand in the population of size $N$, and set $R_i = 2 Z_i - 1$. Since $y_i(1) + y_i(0) = 2$ for every unit, I have\n\\[\n\\widehat\\tau_N - \\tau_N = \\frac{2}{N} \\sum_{i=1}^{N} R_i , \\qquad N V_N = 4\n\\]\nfor every $N$, so Condition 6 holds with $c = 4$, and the standardized estimator is asymptotically standard normal. Substituting the exposure probabilities into (7), (10) and (11), I obtain $\\widehat V_N = 4 N^{-2} \\sum_{i \\le N} Y_i^2$. Let $b_N$ be the number of units $i \\le N$ in blocks with $r$ even, and set $f_N = b_N/N$. A unit in an odd block has $Y_i^2 = 1$, and a unit in an even block has $Y_i^2 = 2 + 2 s_i R_i$ with $s_i = 1$ or $s_i = -1$. Taking the expectation and the variance of the sum, I find\n\\[\n\\mathbb{E}[N \\widehat V_N] = 4 + 4 f_N , \\qquad \\operatorname{Var}(N \\widehat V_N) = \\frac{64 \\, b_N}{N^2} \\le \\frac{64}{N} .\n\\]\nThe variance tends to zero, as the printed argument requires. The expectation does not converge: at the end of block $r$, I have $f_{n_r} \\le n_{r-1}/n_r$ for $r$ odd and $1 - f_{n_r} \\le n_{r-1}/n_r$ for $r$ even, and $n_{r-1}/n_r = 2^{-2^{r-1}}$ for $r \\ge 2$, which tends to zero.\n\\begin{audittable}{rrrll}{The example at the ends of the first four blocks. The scaled variance $N V_N$ equals $4$ in every row.}\n$r$ & $N = n_r$ & $b_N$ & $\\mathbb{E}[N \\widehat V_N]$ & $\\operatorname{Var}(N \\widehat V_N)$ \\\\\n$1$ & $4$ & $0$ & $4$ & $0$ \\\\\n$2$ & $16$ & $12$ & $7$ & $3$ \\\\\n$3$ & $256$ & $12$ & $67/16 \\approx 4.19$ & $3/256$ \\\\\n$4$ & $65536$ & $65292$ & $32707/4096 \\approx 7.99$ & $16323/16777216$ \\\\\n\\end{audittable}\nAlong $N = n_r$ with $r$ odd, $N \\widehat V_N$ tends to $4$ in probability, the ratio $(V_N/\\widehat V_N)^{1/2}$ tends to $1$, and $t$ converges in distribution to $\\mathcal{N}(0,1)$. Along $N = n_r$ with $r$ even, $N \\widehat V_N$ tends to $8$ in probability, the ratio tends to $1/\\sqrt{2}$, and $t$ converges in distribution to $\\mathcal{N}(0,1/2)$. Neither the ratio nor $t$ has a limit along the full sequence. Enumerating all $2^N$ assignments for each $N \\le 10$ in exact arithmetic, I obtain the same values from the printed expressions (7), (10) and (11).\n\n\\emph{Corrected bound.} Set $v_N = N V_N$, $W_N = N \\widehat V_N$ and $Z_N = (\\widehat\\tau_N - \\tau_N)/\\sqrt{V_N}$. Suppose that $v_N \\to c > 0$, that $Z_N$ converges in distribution to $\\mathcal{N}(0,1)$, that $\\mathbb{E}[W_N] \\ge v_N$, and that $\\operatorname{Var}(W_N) \\to 0$. I count a negative variance estimate as noncoverage in the probabilities below. Fix $\\epsilon \\in (0,1)$. If $W_N < (1-\\epsilon) v_N$, then $\\mathbb{E}[W_N] - W_N > \\epsilon v_N$. By Chebyshev's inequality, this implies that\n\\[\n\\Pr\\bigl( W_N < (1-\\epsilon) v_N \\bigr) \\le \\frac{\\operatorname{Var}(W_N)}{\\epsilon^2 v_N^2} \\longrightarrow 0 .\n\\]\nOn the complementary event, $\\widehat V_N$ is positive and $\\sqrt{\\widehat V_N / V_N} \\ge \\sqrt{1-\\epsilon}$. With $z = z_{1-\\alpha/2}$, I have\n\\[\n\\Pr\\bigl( |\\widehat\\tau_N - \\tau_N| \\le z \\sqrt{\\widehat V_N} \\bigr) \\ge \\Pr\\bigl( |Z_N| \\le z \\sqrt{1-\\epsilon} \\bigr) - \\Pr\\bigl( W_N < (1-\\epsilon) v_N \\bigr) .\n\\]\nTaking the limit inferior and using the convergence of $Z_N$, I obtain\n\\[\n\\liminf_{N \\to \\infty} \\Pr\\bigl( |\\widehat\\tau_N - \\tau_N| \\le z \\sqrt{\\widehat V_N} \\bigr) \\ge 2 \\Phi\\bigl( z \\sqrt{1-\\epsilon} \\bigr) - 1 .\n\\]\nSince $\\epsilon$ is arbitrary, the limit inferior of the coverage is at least $2\\Phi(z) - 1 = 1 - \\alpha$. Note that the argument uses no limit for $W_N$ and no independence between $Z_N$ and $W_N$.", "problem": "Let $V_N$ denote the variance of the HT estimator of the effect, and let $\\widehat V_N$ denote the variance estimator (11). The last paragraph of A.7 uses three premises: the estimator standardized by $\\sqrt{V_N}$ is asymptotically standard normal; $N V_N$ tends to $c > 0$ (Condition~6); and $N \\widehat V_N$ has expectation at least $N V_N$ (Proposition~5.6) and variance tending to zero. By Chebyshev's inequality, the third premise implies that $N \\widehat V_N$ is close to its own expectation with probability tending to one. It does not imply convergence ``to a value''. The expectation of $N \\widehat V_N$ exceeds $N V_N$ by the bias of the conservative estimator. No condition controls how this bias varies with $N$, and the expectation may fail to converge. The opening inference of A.7 is then not valid. The two claims of the last paragraph that rest on it can fail under Conditions 3, 5 and 6: that the ratio $(V_N/\\widehat V_N)^{1/2}$ converges in probability to a quantity in $(0,1]$, and that $t$ is asymptotically normal by Slutsky's theorem.\n\nThe conclusion of the paragraph does not need these limits. Coverage of at least $100(1-\\alpha)\\%$ for large $N$ is a lower bound, and a one-sided bound on $N \\widehat V_N$ is enough to obtain it. This finding takes the three premises as given. Whether the variance of $N \\widehat V_N$ tends to zero under Conditions 3, 5 and 6 is a separate question, treated in \\fid{P013-E001}.", "what_survives": "The HT estimator, the variance estimator (11), unbiasedness, Proposition~5.6, Proposition~6.1 and the central limit theorem for the estimator standardized by its true variance are unaffected. The coverage conclusion of Proposition~6.2 holds as a lower bound on the limit inferior of the coverage, under the three premises of the last paragraph of A.7 and with no new assumption. The proof no longer gives a single limiting law for $t$ or a single limit for the coverage. In the example, the coverage tends to $1 - \\alpha$ along one subsequence and to $2\\Phi(z_{1-\\alpha/2}\\sqrt{2}) - 1$ along the other. To keep the printed limit, one would add the assumption that $\\mathbb{E}[N \\widehat V_N]$ converges to a finite $s \\ge c$; then the ratio tends to $\\sqrt{c/s}$ and $t$ converges in distribution to $\\mathcal{N}(0, c/s)$. No number in Table~1 or Table~2 changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E007", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the lower part of Table 2 with its notes. Four Hajek rows are followed by four WLS rows, and each row gives an estimand, an estimate, a standard error and a 95 percent interval. The WLS rows read: tau(d001, d000), 0.056, 0.066, (-0.072, 0.186); tau(d011, d000), 0.156, 0.037, (0.083, 0.229); tau(d101, d000), 0.295, 0.124, (0.050, 0.536); tau(d111, d000), 0.306, 0.049, (0.212, 0.404). Red outlines surround the estimate 0.295 and the interval (0.050, 0.536) in the third WLS row, and the estimate 0.306 and the interval (0.212, 0.404) in the fourth. The last table note reads: S.E. = Estimated standard error; CI = Normal approximation confidence interval.", "caption": "Table 2, p. 1936 (PDF p. 25): the Hajek and WLS rows and the table notes. The columns are estimator, estimand, estimate, S.E. and 95% CI. The outlines mark the estimate and the interval in the WLS rows for $\\tau(d_{101},d_{000})$ and $\\tau(d_{111},d_{000})$.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T10", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T10"}], "headline": "In the WLS rows of Table 2 for $\\tau(d_{101},d_{000})$ and $\\tau(d_{111},d_{000})$, the printed estimate and the printed 95% interval cannot both be correct: a normal-approximation interval is centred at its estimate, and the two differ by more than rounding allows.", "historical_evidence_type": "lean_proof", "id": "P013-E008", "location": "Table 2, WLS rows for $\\tau(d_{101},d_{000})$ and $\\tau(d_{111},d_{000})$, p. 1936 (PDF p. 25)", "pdf_page": 25, "printed_page": "1936", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P013-T10"], "text": {"as_printed": "\\begin{audittable}{llrrr}{Table~2, WLS rows.}\nEstimator & Estimand & Estimate & S.E. & 95\\% CI \\\\\nWLS & $\\tau(d_{001},d_{000})$ & 0.056 & 0.066 & $(-0.072, 0.186)$ \\\\\n & $\\tau(d_{011},d_{000})$ & 0.156 & 0.037 & $(0.083, 0.229)$ \\\\\n & $\\tau(d_{101},d_{000})$ & \\hl{0.295} & 0.124 & $\\hl{(0.050, 0.536)}$ \\\\\n & $\\tau(d_{111},d_{000})$ & \\hl{0.306} & 0.049 & $\\hl{(0.212, 0.404)}$ \\\\\n\\end{audittable}\n\\emph{Table note.} S.E. = Estimated standard error; CI = Normal approximation confidence interval.", "corrected": "Table~2, WLS rows for $\\tau(d_{101},d_{000})$ and $\\tau(d_{111},d_{000})$: in each row at least one of the estimate and the two interval endpoints is misprinted. The printed estimates and endpoints cannot arise from normal-approximation intervals centred at the same estimates, even allowing for rounding to three decimals. The two rows should be checked against the original unrounded output and the wrong cells corrected. The replacement entries cannot be determined from the published table or from the public data.\n\nThe table below gives two ways of making each row consistent, with $z = 1.96$. In the first, the interval and standard error are kept at their printed precision and the estimate changes. In the second, the printed estimate and standard error are taken as exact and the interval changes. They show the size of the change, and neither is a recovered value.\n\\begin{audittable}{llll}{Two ways of restoring consistency in each row (estimate, 95\\% CI).}\nRow & As printed & Interval and S.E. kept & Estimate and S.E. kept \\\\\n$\\tau(d_{101},d_{000})$ & $0.295$, $(0.050, 0.536)$ & $\\hl{0.293}$, $(0.050, 0.536)$ & $0.295$, $\\hl{(0.052, 0.538)}$ \\\\\n$\\tau(d_{111},d_{000})$ & $0.306$, $(0.212, 0.404)$ & $\\hl{0.308}$, $(0.212, 0.404)$ & $0.306$, $\\hl{(0.210, 0.402)}$ \\\\\n\\end{audittable}", "evidence": "The evidence is a derivation in exact arithmetic and a recalculation on the public data. Let $L = e - zs$ and $U = e + zs$ be the unrounded endpoints. Then $(L + U)/2 = e$. A number printed to three decimals as $p$ lies in $[p - 0.0005, p + 0.0005]$ under any rule for ties. Then $e$ lies within $0.0005$ of the printed estimate, and $(L + U)/2$ lies within $0.0005$ of the midpoint of the printed endpoints.\n\\begin{audittable}{lrlrlr}{The ranges that rounding allows in the two WLS rows.}\nRow & Estimate & Range for $e$ & Interval midpoint & Range for $(L+U)/2$ & Gap \\\\\n$\\tau(d_{101},d_{000})$ & $0.295$ & $[0.2945, 0.2955]$ & $0.293$ & $[0.2925, 0.2935]$ & $0.001$ \\\\\n$\\tau(d_{111},d_{000})$ & $0.306$ & $[0.3055, 0.3065]$ & $0.308$ & $[0.3075, 0.3085]$ & $0.001$ \\\\\n\\end{audittable}\nIn both rows the two ranges are disjoint. Note that the argument uses neither the standard error nor the critical value. The same comparison in the other ten rows of Table~2 finds nine that are compatible and one, the WLS row for $\\tau(d_{001},d_{000})$, that is compatible only at a boundary. Its printed cells $0.056$ and $(-0.072, 0.186)$ force $e = 0.0565$, $L = -0.0725$ and $U = 0.1855$, and these values print as shown when ties are rounded to even. A third misprint is not established.\n\nThe recalculation uses the public data release of the field experiment (ICPSR study 37070, version 2). With an adapter for the public data, it simulates the exposure probabilities with the seed and the 10,000 draws of the script released with the data and with the sampler that R used before version 3.6. It keeps the eligible students whose five estimated probabilities exceed $0.01$. It fits least squares weighted by inverse exposure probabilities, with an intercept for each exposure and one slope on network degree. This reconstruction has $N=2{,}050$ students and reproduces all twelve point estimates of Table~2 to three decimals. Its WLS estimates for the two rows are approximately $0.294544$ and $0.3063$, which round to the printed $0.295$ and $0.306$ and not to the midpoints $0.293$ and $0.308$. None of its standard errors rounds to a printed one. The cut-off and the form of the regression are choices of the reconstruction. None of the released scripts computes Table~2, and the authors' own files and unrounded output are not public; ICPSR warns that some variables in the public release were changed. The recalculation supports the two printed estimates. It does not identify the misprinted cells, and it does not recover the endpoints.", "problem": "The note to Table~2 says that the intervals are normal-approximation intervals. Such an interval has endpoints $e - zs$ and $e + zs$, where $e$ is the estimate, $s$ its standard error and $z$ the critical value, and its midpoint is $e$ for every $s$ and $z$. Each printed number is rounded to three decimals, so a printed estimate and the midpoint of a printed interval can differ by at most $0.001$. In the WLS row for $\\tau(d_{101},d_{000})$ the midpoint of $(0.050, 0.536)$ is $0.293$ and the estimate is $0.295$. In the row for $\\tau(d_{111},d_{000})$ the midpoint of $(0.212, 0.404)$ is $0.308$ and the estimate is $0.306$. Both differences are $0.002$. I conclude that in each row at least one of the estimate and the two endpoints is misprinted, and that no change to the standard error removes the inconsistency. The table does not show which cell is wrong.", "what_survives": "The eight HT and Hajek rows of Table~2 and the WLS row for $\\tau(d_{011},d_{000})$ are arithmetically compatible with their intervals after rounding, and the WLS row for $\\tau(d_{001},d_{000})$ is not shown to be wrong. I find no evidence that these inconsistencies reverse the qualitative conclusions in Section~10. Under either illustrative repair below, the direct effects stay near 30 percentage points and the lower endpoints stay positive. The printed HT and Hajek intervals for the direct and indirect contrasts are positive on their own. These inconsistencies have no demonstrated consequence for the theory, the simulation of Table~1 or the exposure mapping. At least one cell changes in each of the two rows, and its value requires the original unrounded output."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E008", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the closing sentences of Section 4. Red boxes mark the covariance Cov[y-hat-T-HT(d_k), y-hat-T-HT(d_l)] and, on the next line, the words 'non-negative bias' in the sentence 'we can always identify estimators for Var and Cov that are guaranteed to have non-negative bias'.", "caption": "Section 4, last paragraph, p. 1921 (PDF p. 10). The marked phrase “non-negative bias” is applied to the marked covariance as well as to the variance.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "The sentence gives the covariance estimator the same sign of bias as the variance estimators; the covariance estimators of Section 5 have non-positive bias, which is what a conservative variance for the contrast requires.", "id": "P013-E009", "location": "Section 4, last paragraph, p. 1921 (PDF p. 10)", "pdf_page": 10, "printed_page": "1921", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Nonetheless, we can always identify estimators for $\\operatorname{Var}[\\widehat{y^T_{\\mathrm{HT}}}(d_k)]$ and $\\hl{\\operatorname{Cov}[\\widehat{y^T_{\\mathrm{HT}}}(d_k),\\widehat{y^T_{\\mathrm{HT}}}(d_l)]}$ that are guaranteed to have \\hl{non-negative bias}. Thus, we can always identify a conservative approximation to the exact variances.''", "corrected": "Nonetheless, estimators for $\\operatorname{Var}[\\widehat{y^T_{\\mathrm{HT}}}(d_k)]$ with non-negative bias and estimators for $\\operatorname{Cov}[\\widehat{y^T_{\\mathrm{HT}}}(d_k),\\widehat{y^T_{\\mathrm{HT}}}(d_l)]$ with \\hl{non-positive bias} can always be identified. Together they give a conservative approximation to \\hl{the variance of the estimated contrast}.", "intended_meaning": "The sentence introduces the conservative estimator of the variance of a contrast. In equation (5) the covariance enters with coefficient $-2$. For the estimate of that variance to be conservative, it suffices for the variance estimators to be biased upward and the covariance estimator downward. Section 5 supplies such estimators. Propositions 5.3 and 5.5 state that $\\mathrm{E}[\\widehat{\\operatorname{Cov}}]\\le\\operatorname{Cov}$, and when all joint probabilities of distinct units are positive, the bias of estimator (8) is $-(1/2)\\sum_i(y_i(d_k)-y_i(d_l))^2\\le0$. For three units with independent fair assignments, $y_i(d_k)=1$ and $y_i(d_l)=0$, the covariance is $0$ and estimator (8) has expectation $-3/2$. Equations (5), (8), (10) and (11) and Propositions 5.3, 5.5 and 5.6 are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E009", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the heading Appendix A: Proofs and the four lines below it. The text reads: A.1. Proof of Proposition 3.1. The replication procedure is equivalent to drawing a random sample without replacement from Omega, with probabilities of selection equal to those which are defined in the randomization plan. The result follows from the strong law of large numbers. A red outline surrounds the words without replacement.", "caption": "Appendix A.1, p. 1938 (PDF p. 27): the proof of Proposition 3.1 in full. The outline marks “without replacement”.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T20", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T20"}], "headline": "The replicate assignments are independent draws from the randomization plan, that is, draws with replacement from $\\Omega$; the proof says “without replacement”.", "id": "P013-E010", "location": "Appendix A.1, proof of Proposition 3.1, first sentence, p. 1938 (PDF p. 27)", "pdf_page": 27, "printed_page": "1938", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P013-T20"], "text": {"as_printed": "A.1. Proof of Proposition 3.1. The replication procedure is equivalent to drawing a random sample \\hl{without replacement} from $\\Omega$, with probabilities of selection equal to those which are defined in the randomization plan. The result follows from the strong law of large numbers.", "corrected": "A.1. Proof of Proposition 3.1. The replication procedure is equivalent to \\hl{independent draws with replacement} from $\\Omega$, \\hl{with each assignment vector drawn according to} the randomization plan. The result follows from the strong law of large numbers.", "intended_meaning": "Section~3 (p.~1917) describes the procedure as producing ``$R$ random replicate $\\mathbf{z}$'s based on the randomization plan'', and Proposition~3.1 lets $R$ tend to infinity. The replicates are independent draws from the design, which are draws with replacement from $\\Omega$. This reading is correct. For independent replicates, the strong law implies that each of the finitely many matrix entries converges, and the additive smoothing does not affect the limit. The binomial expectation used in Appendix~A.3 holds for independent replicates as well. Read literally, the printed phrase cannot hold as $R \\to \\infty$, since $\\Omega$ is finite and a sample without replacement has at most $|\\Omega|$ draws. The sampler of the package that rejects repeated assignments is a separate implementation issue."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E010", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of one displayed formula and five lines of text. The display is N to the power minus 2 times a quadruple sum over i, j, k and l of the covariance of h sub ij times a sub ij (D sub i, D sub j) and h sub kl times a sub kl (D sub k, D sub l). The text reads: Note that this covariance is not equal to 0 if and only if h sub ij = 1 and h sub kl = 1, and either h sub ik = 1, h sub il = 1, h sub jk = 1, or h sub jl = 1. By Condition 5, given m much less than N, each of these four conditions is satisfied by fewer than N m cubed of the elements of the quadruple summation, and the number of elements in their union is at most 4 N m cubed. A red outline surrounds the words if and only if.", "caption": "Appendix A.7, p. 1943 (PDF p. 32): the last line of the display that expands the variance into a quadruple sum of covariances, and the sentence after it. The outline marks “if and only if”.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E011.png"}], "formal_release": null, "formal_statements": [], "headline": "The condition on the indicators $h$ is necessary for a nonzero covariance and is not equivalent to it; the count that follows uses only the necessary direction, and “if and only if” should read “only if”.", "id": "P013-E011", "location": "Appendix A.7, sentence after the quadruple covariance sum, p. 1943 (PDF p. 32)", "pdf_page": 32, "printed_page": "1943", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Note that $\\operatorname{Cov}[h_{ij}a_{ij}(D_i,D_j),h_{kl}a_{kl}(D_k,D_l)]\\ne0$ \\hl{if and only if} $h_{ij}=1$ and $h_{kl}=1$, and either $h_{ik}=1$, $h_{il}=1$, $h_{jk}=1$, or $h_{jl}=1$. By Condition 5, given $m\\ll N$, each of these four conditions is satisfied by fewer than $Nm^3$ of the elements of the quadruple summation, and the number of elements in their union is at most $4Nm^3$.", "corrected": "Note that $\\operatorname{Cov}[h_{ij}a_{ij}(D_i,D_j),h_{kl}a_{kl}(D_k,D_l)]\\ne0$ \\hl{only if} $h_{ij}=1$ and $h_{kl}=1$, and either $h_{ik}=1$, $h_{il}=1$, $h_{jk}=1$, or $h_{jl}=1$.", "intended_meaning": "The indicators $h_{ij}$ are the edges of the dependency graph described after Condition~5 (p.~1926). A covariance term can be nonzero only if both pairs are edges and some edge joins the two pairs. Without such an edge, the exposures of the two pairs are independent and the covariance is zero. This reading is correct, and the next sentence uses only this direction, since it bounds the number of terms that can be nonzero. The converse is false, since an edge permits dependence and does not force a nonzero covariance. With independent exposures of probability $1/2$, outcomes $y_i(1) = 1$ and $y_i(0) = 0$, and $h_{ij} = 1$ within blocks of four units, the terms $a_{12}$ and $a_{34}$ are identically zero although $h_{12} = h_{34} = h_{13} = 1$, and Condition~6 holds. The note concerns this sentence only; the other steps of A.7 are treated in \\fid{P013-E001} and \\fid{P013-E007}."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E011", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Proposition 6.2 and the first lines of Section 7. A red box marks the symbol tau with subscript HT, without a hat, in the words 'will tend to cover tau HT of d k, d l at least 100 times 1 minus alpha percent of the time for large N'.", "caption": "Proposition 6.2, p. 1926 (PDF p. 15). The marked coverage target carries the subscript HT of the estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E012.png"}, {"alt": "Crop of the last five lines of Appendix A.6. A red box marks the symbol mu with subscript HT, without a hat, in the difference that is said to converge to zero in probability. Earlier in the same sentence the words 'Consistency of mu hat HT for mu of d k' appear, not marked.", "caption": "Appendix A.6, p. 1942 (PDF p. 31), last lines of the proof of Proposition 6.1. The marked centering term carries the subscript HT; the same sentence names the target $\\mu(d_k)$ correctly.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E012-b.png"}, {"alt": "Crop of the definition of the statistic t and the paragraph after it. Four red boxes mark the symbol tau with subscript HT, without a hat: in the numerators of the two fractions that define t, in the numerator of the small fraction said to be asymptotically normal, and in the words 'will tend to cover tau HT of d k, d l'.", "caption": "Appendix A.7, p. 1943 (PDF p. 32), end of the proof of Proposition 6.2. The four marked terms are the targets subtracted in the statistic $t$ and in the standardized ratio, and the target covered by the intervals.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E012-c.png"}, {"alt": "Crop of the sentence 'Following Fattorini (2006), the following proposition provides guidance on choosing R in terms of a bound on the relative bias for estimating y T of d k', followed by the first line of Proposition 4.1. A red box marks the symbol y T of d k, which is printed with a wide hat.", "caption": "Sentence before Proposition 4.1, p. 1920 (PDF p. 9). The marked target total carries a hat.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E012-d.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T9", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T9"}, {"id": "P013-T18", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T18"}], "headline": "The targets in Proposition 6.2 and Appendices A.6 and A.7 are the population quantities $\\mu(d_k)$ and $\\tau(d_k,d_l)$, which carry no subscript HT, and the relative bias before Proposition 4.1 is for estimating $y^T(d_k)$, which carries no hat.", "id": "P013-E012", "location": "Proposition 6.2, p. 1926 (PDF p. 15); Appendix A.6, p. 1942 (PDF p. 31); Appendix A.7, p. 1943 (PDF p. 32); sentence before Proposition 4.1, p. 1920 (PDF p. 9)", "pdf_page": 15, "printed_page": "1926", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P013-T18", "P013-T9"], "text": {"as_printed": "Proposition 6.2, p.~1926: ``will tend to cover $\\tau_{\\hl{\\mathrm{HT}}}(d_k,d_l)$ at least $100(1-\\alpha)\\%$ of the time for large $N$.''\n\nAppendix A.6, p.~1942: ``as this implies that $\\widehat{\\mu_{\\mathrm{HT}}}(d_k)-\\mu_{\\hl{\\mathrm{HT}}}(d_k)\\xrightarrow{p}0$.''\n\nAppendix A.7, p.~1943: ``Define\n\\[\nt=\\frac{\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)-\\tau_{\\hl{\\mathrm{HT}}}(d_k,d_l)}{\\sqrt{\\widehat{\\operatorname{Var}}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}}\n=\\frac{\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)-\\tau_{\\hl{\\mathrm{HT}}}(d_k,d_l)}{\\sqrt{\\operatorname{Var}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}}\n\\left(\\frac{\\operatorname{Var}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}{\\widehat{\\operatorname{Var}}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}\\right)^{1/2}.\n\\]\nUnder Conditions 3, 5, and 6, then, by Chen and Shao [(2004), Theorem 2.7],\n\\[\n\\frac{\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)-\\tau_{\\hl{\\mathrm{HT}}}(d_k,d_l)}{\\sqrt{\\operatorname{Var}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}}\n\\]\nis asymptotically $\\mathrm{N}(0,1)$ \\ldots\\ will tend to cover $\\tau_{\\hl{\\mathrm{HT}}}(d_k,d_l)$ at least $100(1-\\alpha)\\%$ of the time as $N\\to\\infty$.''\n\nSentence before Proposition 4.1, p.~1920: ``a bound on the relative bias for estimating $\\hl{\\widehat{y^T}}(d_k)$.''", "corrected": "Proposition 6.2: ``will tend to cover $\\hl{\\tau}(d_k,d_l)$ at least $100(1-\\alpha)\\%$ of the time for large $N$.''\n\nAppendix A.6: ``as this implies that $\\widehat{\\mu_{\\mathrm{HT}}}(d_k)-\\hl{\\mu}(d_k)\\xrightarrow{p}0$.''\n\nAppendix A.7:\n\\[\nt=\\frac{\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)-\\hl{\\tau}(d_k,d_l)}{\\sqrt{\\widehat{\\operatorname{Var}}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}}\n=\\frac{\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)-\\hl{\\tau}(d_k,d_l)}{\\sqrt{\\operatorname{Var}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}}\n\\left(\\frac{\\operatorname{Var}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}{\\widehat{\\operatorname{Var}}[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)]}\\right)^{1/2},\n\\]\nwith $\\hl{\\tau}(d_k,d_l)$ also in the numerator of the standardized ratio that follows and in ``will tend to cover $\\hl{\\tau}(d_k,d_l)$ at least $100(1-\\alpha)\\%$ of the time as $N\\to\\infty$.''\n\nSentence before Proposition 4.1: ``a bound on the relative bias for estimating $\\hl{y^T}(d_k)$.''\n\nThe hats and the subscript HT remain on the estimators.", "evidence": "This is a check of the pages. The definitions are on pp.~1919--1920. Proposition 6.1 on p.~1925 states consistency with the target $\\tau(d_k,d_l)$, and the sentence of Appendix A.6 that contains the misprint begins ``Consistency of $\\widehat{\\mu_{\\mathrm{HT}}}(d_k)$ for $\\mu(d_k)$''. The hat on p.~1920 was confirmed on an enlargement of the page.", "problem": "Section 4 defines the population mean $\\mu(d_k)=N^{-1}\\sum_{i=1}^N y_i(d_k)$, the total $y^T(d_k)=\\sum_{i=1}^N y_i(d_k)$ and the average causal effect $\\tau(d_k,d_l)=N^{-1}\\sum_{i=1}^N y_i(d_k)-N^{-1}\\sum_{i=1}^N y_i(d_l)$. The subscript HT and the hat belong to their Horvitz--Thompson estimators. The symbols $\\tau_{\\mathrm{HT}}(d_k,d_l)$ and $\\mu_{\\mathrm{HT}}(d_k)$ without hats are not defined in the paper. The first is printed five times where $\\tau(d_k,d_l)$ is meant (once in Proposition 6.2 and four times in Appendix A.7), and the second once where $\\mu(d_k)$ is meant (at the end of Appendix A.6). In the sentence before Proposition 4.1 the target total carries a hat. These are misprints in the names of the targets.", "what_survives": "The estimands, the estimators and the intended statements of consistency, standardization and coverage are unchanged. Only the notation for the targets changes. This correction does not address the substance of Proposition 6.2 or of its proof, which are the subject of \\fid{P013-E001} and \\fid{P013-E007}."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E012", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the first lines of the notes under Table 1. A red box marks the words 'weighted by exposure probabilities' in the line that begins 'WLS = Least squares'.", "caption": "Notes to Table 1, p. 1933 (PDF p. 22). The marked words describe the weights of the WLS estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E013.png"}, {"alt": "Crop of the notes under Table 2. A red box marks the words 'weighted by exposure probabilities' in the line that begins 'WLS = Least squares'.", "caption": "Notes to Table 2, p. 1936 (PDF p. 25). The same words are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E013-b.png"}, {"alt": "Crop of the display that defines the estimated parameter vector as the minimizer of a weighted sum of squared residuals, with three lines of text above and two below. A red box marks the weight, the fraction 1 over pi i of d k.", "caption": "Section 7.1, p. 1928 (PDF p. 17), the weighted least squares criterion. The marked weight is the inverse of the exposure probability.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E013-c.png"}], "formal_release": null, "formal_statements": [], "headline": "In the notes to Tables 1 and 2, “weighted by exposure probabilities” means weighted by the inverse exposure probabilities $1/\\pi_i(d_k)$ of the least squares criterion in Section 7.1.", "id": "P013-E013", "location": "Notes to Table 1, p. 1933 (PDF p. 22), and to Table 2, p. 1936 (PDF p. 25); the weights are defined in Section 7.1, p. 1928 (PDF p. 17)", "pdf_page": 22, "printed_page": "1933", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Notes to Table 1 and to Table 2: ``WLS = Least squares \\hl{weighted by exposure probabilities} with covariate adjustment for network degree and linearized variance estimator.''", "corrected": "In both notes: ``WLS = Least squares weighted by \\hl{inverse} exposure probabilities with covariate adjustment for network degree and linearized variance estimator.''", "intended_meaning": "Section 7.1 defines the weighted least squares fit on p.~1928 by the criterion\n\\[\n\\widehat{\\boldsymbol{\\xi}}(d_k)=\\arg\\min_{\\boldsymbol{\\xi}(d_k)}\\sum_{i:D_i=d_k}\\frac{1}{\\pi_i(d_k)}\\bigl[Y_i-g(\\mathbf{x}_i,\\boldsymbol{\\xi}(d_k))\\bigr]^2 ,\n\\]\nand says that ``weighting by $1/\\pi_i(d_k)$ ensures that the regression proceeds on a sample representative of the population of potential outcomes''. The table notes name the source of the weights and omit the word ``inverse''. The wording of the notes does not show which weights the computations used.\n\nOne of the scripts released with the data of the field experiment weights a regression by inverse exposure probabilities, but that regression belongs to a different analysis. A recalculation on the public data of the field experiment (ICPSR study 37070, version 2) supports the reading for Table 2. The recalculation simulates the exposure probabilities with the seed and the 10,000 draws of the script released with the data and with the sampler that R used before version 3.6. It keeps the eligible students whose five estimated probabilities exceed $0.01$, and it fits one slope on network degree for all exposures. This reconstruction has $N=2{,}050$ students and reproduces all twelve point estimates of Table 2 to three decimals. With inverse weights, the WLS contrasts for $d_{001}$, $d_{011}$, $d_{101}$ and $d_{111}$ against $d_{000}$ are $0.0562$, $0.1563$, $0.2945$ and $0.3063$, which round to the printed $0.056$, $0.156$, $0.295$ and $0.306$. With the probabilities themselves as weights and every other input fixed, the contrasts are $0.0771$, $0.1571$, $0.2524$ and $0.3022$, and none rounds to its printed value. Adding the residual term of expression (14) to the fit with probability weights, I obtain $0.057$, $0.155$, $0.295$ and $0.307$, of which one equals its printed value.\n\nThe $0.01$ cut-off is inferred from a different analysis, whereas the paper specifies nonzero probabilities; the common slope is also a choice of the reconstruction. The script that produced Table 2 and the authors' own files are not public, and ICPSR's notes on the release say that some variables were changed. The recalculation corroborates the inverse weights for Table 2 and does not certify them, and it says nothing about Table 1. Nothing else depends on the wording."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E013", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text. The passage reads: Three features emerge from examination of the exposure mapping. First, the exposure mapping reflects four different sources of exposure to the program: being in a school with the program (School), having a peer who was a participation student (Indirect), and being a participating student (Direct). The next sentence begins: Second, only students selected as ``eligible have a nonzero possibility of being in all exposure con-; the crop ends in the middle of the word conditions. Red outlines surround the word four in the second line and the word participation in the third line.", "caption": "Section 10, first paragraph below Figure 2, p. 1935 (PDF p. 24). The outlines mark “four”, where three sources are listed, and “participation”, which should read “participating”.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E014.png"}], "formal_release": null, "formal_statements": [], "headline": "The sentence lists three sources of exposure to the program, not four, and “participation student” should read “participating student”.", "id": "P013-E014", "location": "Section 10, first paragraph below Figure 2, p. 1935 (PDF p. 24)", "pdf_page": 24, "printed_page": "1935", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "First, the exposure mapping reflects \\hl{four} different sources of exposure to the program: being in a school with the program (School), having a peer who was a \\hl{participation} student (Indirect), and being a participating student (Direct).", "corrected": "First, the exposure mapping reflects \\hl{three} different sources of exposure to the program: being in a school with the program (School), having a peer who was a \\hl{participating} student (Indirect), and being a participating student (Direct).", "evidence": "The evidence is a check of the pages: the sentence on p.~1935 and the display of the exposure mapping on p.~1934.", "problem": "The sentence announces four sources and lists three: School, Indirect and Direct. The exposure mapping on p.~1934 is built from the three corresponding indicators, which generate five exposure conditions, four of them other than no exposure; ``participation student'' is a slip for ``participating student''.", "what_survives": "The exposure mapping, the five exposure conditions and every estimate of Section~10 are unaffected; two words change."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E014", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 233 to 243 in monospaced type with line numbers. A red box marks line 234, 'meanse is assigned function(x) mean(max(0,x) to the power .5)'. The lines below it call meanse on six vectors of variance estimates.", "caption": "Package interference, file misc/ReplicationAoAS.R, lines 233–243, typeset from the source file with line numbers added. The marked line defines the function; the same definition is repeated at line 381.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E016-b.png"}, {"alt": "Crop of the title and column headings of Table 1 and its three rows for the HT estimator. A red box marks the column heading 'Mean S.E.'. The entries of that column in the three rows shown are 0.05, 0.19 and 0.28.", "caption": "Table 1, p. 1933 (PDF p. 22), heading and first rows. The marked column is the counterpart in the paper of the script’s MeanSE output; no entry of the column has been shown wrong.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E016.png"}], "formal_release": null, "formal_statements": [], "headline": "The function labelled “Mean SE” in the replication script distributed with the interference package returns the largest standard error across replications, not their mean.", "id": "P013-E016", "location": "interference package, misc/ReplicationAoAS.R, lines 234 and 381; counterpart in the paper: Table 1, column “Mean S.E.”, p. 1933 (PDF p. 22)", "pdf_page": 22, "printed_page": "1933", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{misc/ReplicationAoAS.R}, lines 233--234, and again lines 380--381:\n\n\\texttt{\\#\\# Mean SE}\n\n$\\texttt{meanse <- function(x) mean(}\\hl{\\texttt{max}}\\texttt{(0,x)\\textasciicircum.5)}$", "corrected": "At both lines, with the maximum taken element by element:\n\n\\texttt{meanse <- function(x) mean(sqrt(\\hl{pmax}(0,x)))}\n\nFor variance estimates $v_1,\\dots,v_B$ this returns\n\\[\n\\frac{1}{B}\\sum_{b=1}^B\\sqrt{\\max(0,v_b)} ,\n\\]\nand negative variance estimates are then set to zero separately in each replication, a rule that should be stated. For nonnegative variance estimates \\texttt{mean(sqrt(x))} gives the same value.", "evidence": "This finding rests on an exact counterexample and on running the code. For two replications with variance estimates $(1,4)$, the function returns $\\sqrt{4}=2$, and the mean standard error is $(1+2)/2=3/2$. For $(1,4,9,100)$ the values are $10$ and $4$. Both definitions in the script (lines 234 and 381) were run with their six calls, for three Horvitz--Thompson and three Hajek contrasts, on the input $(1,4)$: each returned $2$ in all six cells, and with \\texttt{pmax} in place of \\texttt{max} each returned $1.5$.", "problem": "The script calls this function on a vector of variance estimates, one for each simulated replication, and writes the result to the column \\texttt{MeanSE} of its output table. In R, \\texttt{max(0,x)} is a single number, the largest of $0$ and the elements of \\texttt{x}. The function then takes the square root of that number and averages the one value it has. For a nonempty vector of finite variance estimates $v_1,\\dots,v_B$ it returns $\\max_b\\sqrt{\\max(0,v_b)}$, the largest standard error across replications after negative variance estimates are set to zero. The heading ``Mean S.E.'' of Table 1 and the script's own label call for the mean, $B^{-1}\\sum_{b=1}^B\\sqrt{\\max(0,v_b)}$. The value returned is never smaller than this mean, and equals it only when all the standard errors are equal.", "what_survives": "The bias, standard deviation, root mean square error and coverage columns of the script do not call this function, and the estimation functions of the package do not use it. The estimators and proofs of the paper are unaffected. No entry of the Mean S.E. column of the published Table 1 is shown to be wrong or shown to be right. The script is a later implementation, its inputs are not available, and it is not established that it produced the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E016", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. In line 71 of the replication script a red box marks the argument hop=1 in the call to estimates(). In lines 103 and 106 red boxes mark the expressions all_estimates$var_tau_ht_const_eff and all_estimates$var_tau_ht_max. Below, lines 165 to 174 of R/estimates.R show the arguments of estimates(): obs_exposure, obs_outcome, obs_prob_exposure, n_var_permutations, effect_estimators, variance_estimators, control_condition and treated_conditions. There is no argument named hop.", "caption": "Source code of the package, not a page of the article: misc/ReplicationAoAS.R, lines 71 and 103–107, above the signature of estimates() in R/estimates.R, lines 165–174. The marked argument hop=1 is not among the arguments of the function, and the two marked fields are not returned under its default settings.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E017.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication script shipped with the package calls estimates() with an argument hop that the function does not accept, which stops the script at line 71; the call should drop hop, name the control exposure and request the variance outputs that the script stores.", "id": "P013-E017", "location": "R package interference (commit c3fc7b3): misc/ReplicationAoAS.R, lines 71 and 103–107; R/estimates.R, lines 165–174. The code is not printed in the article.", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "result_fails", "effect_text": "a result or code path there relies on it and fails with it", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{misc/ReplicationAoAS.R}, line 71 (trailing comment omitted):\n\n\\texttt{all\\_estimates <- estimates(exposure, obs\\_outcome, obs\\_prob\\_exposure, n\\_var\\_permutations = 50, \\hl{hop=1})}\n\nLines 103 and 106 (trailing comments omitted):\n\n\\texttt{var\\_estimate\\_ht\\_const\\_eff <- as.list(\\hl{all\\_estimates\\$var\\_tau\\_ht\\_const\\_eff})}\n\n\\texttt{var\\_estimate\\_ht\\_max <- as.list(\\hl{all\\_estimates\\$var\\_tau\\_ht\\_max})}\n\n\\texttt{R/estimates.R}, lines 165--174 (the signature, joined on one line):\n\n\\texttt{estimates <- function(obs\\_exposure, obs\\_outcome, obs\\_prob\\_exposure, n\\_var\\_permutations = 10, effect\\_estimators = c('hajek', 'horvitz-thompson'), variance\\_estimators = c('hajek', 'horvitz-thompson'), control\\_condition=NULL, treated\\_conditions=NULL)}", "corrected": "\\texttt{all\\_estimates <- estimates(exposure, obs\\_outcome, obs\\_prob\\_exposure, n\\_var\\_permutations = 50, \\hl{control\\_condition = 'no', variance\\_estimators = c('hajek', 'horvitz-thompson', 'max\\_ht\\_const')})}\n\nThe choice \\texttt{'max\\_ht\\_const'} also requests the constant-effect variance estimator. Alternatively, remove \\texttt{hop} and delete lines 97--107, which extract and store the derived variances that the later table statistics do not use.", "evidence": "The evidence is an exact check of the code on a small example. When I run line 71 as shipped against the function in the same package, I obtain the error quoted above. On a network of six units whose exposure probabilities are known exactly, the call without \\texttt{hop} returns variance vectors of length zero, and the script fails when it writes them to a CSV file (R 4.4.2, \\texttt{data.table} 1.18.2.1). The corrected call below passes the same steps. I did not run the whole script, because the Add Health files that it reads are not available.", "problem": "The function \\texttt{estimates()} in the same package has no argument \\texttt{hop}. R rejects the call on line 71 with the error ``unused argument (hop = 1)'' before any estimator is computed, whatever the data. It is not enough to remove \\texttt{hop}. With the default \\texttt{control\\_condition=NULL}, the outputs \\texttt{var\\_tau\\_ht} and \\texttt{var\\_tau\\_h} are empty. The default \\texttt{variance\\_estimators} does not produce \\texttt{var\\_tau\\_ht\\_const\\_eff} or \\texttt{var\\_tau\\_ht\\_max}, which lines 103--107 store, and the script then fails when it writes its results to a CSV file.", "what_survives": "The finding concerns only whether a later script runs. It bears on no statement of the article, and no entry of Table 1 is shown to be wrong. The variance fields that the script fails to store are not used in its later table statistics, which recompute the variances of the effects from the variances and covariances of the outcome totals. The check does not establish which version of the code produced the published Table 1."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E017", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. Lines 109 to 112 of R/estimates.R say that yT_ht contains the Horvitz-Thompson estimator of the total as derived in Equation 1 of Aronow and Samii (2017). Lines 216 to 218 compute yT_ht with rowSums. A red box marks line 219, yT_ht[rowSums(!is.na(obs_outcome_by_exposure)) == 0] <- NA. In the replication script, line 131 is a comment saying that the aggregation removes rows for which the estimator yT is NA, and in line 136 red boxes mark the four subsetting terms [!is.na(yT_ht_dir_ind1)].", "caption": "Source code of the package, not a page of the article: R/estimates.R, lines 109–112 and 216–219, and misc/ReplicationAoAS.R, lines 131 and 136. The marked line 219 replaces the total by NA when no unit has the exposure, and the marked terms of line 136 remove the schools with a missing total from both sums of each ratio.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E018-code.png"}, {"alt": "Crop of the paragraph beginning 'The number of units in the population, N, is fixed' and of equation (1). Red boxes mark the words 'The number of units in the population, N, is fixed' and the right-hand side of equation (1), the sum over i from 1 to N of I(D_i = d_k) Y_i divided by pi_i(d_k).", "caption": "Equation (1) and the paragraph before it, p. 1919 (PDF p. 8): the estimator that the package documents yT_ht to be. The marked sum equals zero when no unit has exposure $d_k$, and the marked words fix the population. The passage is correct.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E018.png"}], "formal_release": null, "formal_statements": [], "headline": "When no unit realizes an exposure, the package returns a missing value for the Horvitz–Thompson total that equation (1) defines as zero, and the first aggregation of the replication script then drops those schools; the total should stay zero and the aggregation should run over the fixed population.", "id": "P013-E018", "location": "R package interference (commit c3fc7b3): R/estimates.R, lines 216–219, documented on lines 109–112; misc/ReplicationAoAS.R, lines 131 and 136. Counterpart in the article: equation (1), p. 1919 (PDF p. 8).", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P044-E006", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P045-E004", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P044-E006", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E006", "work_id": "P044"}, {"id": "P045-E004", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E004", "work_id": "P045"}], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Equation (1), p.~1919:\n\\[\n\\widehat{y^T_{\\mathrm{HT}}}(d_k)=\\sum_{i=1}^{N}\\mathbf{I}(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)}.\n\\]\n\n\\texttt{R/estimates.R}, lines 216--219:\n\n\\texttt{yT\\_ht <- rowSums(obs\\_outcome\\_by\\_exposure / obs\\_prob\\_exposure\\_individual\\_kk, na.rm = T)}\n\n\\texttt{\\hl{yT\\_ht[rowSums(!is.na(obs\\_outcome\\_by\\_exposure)) == 0] <- NA}}\n\n\\texttt{misc/ReplicationAoAS.R}, line 136 (beginning):\n\n\\texttt{tau11\\_ht <- estimates\\_all\\_schools[, sum(yT\\_ht\\_dir\\_ind1[\\hl{!is.na(yT\\_ht\\_dir\\_ind1)}])/\\ldots}", "corrected": "\\texttt{R/estimates.R}, lines 216--219, \\hl{with line 219 deleted}:\n\n\\texttt{yT\\_ht <- rowSums(obs\\_outcome\\_by\\_exposure / obs\\_prob\\_exposure\\_individual\\_kk, na.rm = T)}\n\nIn the replication script, aggregate the Horvitz--Thompson contrasts, and the effects they are compared with, over all schools with the fixed population in the denominator, and do not delete schools according to the exposures realized. Keep the Horvitz--Thompson variance and covariance expressions as defined when an exposure is empty, and treat an empty Hajek denominator separately.", "evidence": "The evidence is an exact counterexample, which I confirmed by running the unchanged functions and aggregation lines on every assignment. The example has two schools and 12 units, the four exposures of Section 9, and a strictly positive probability of every exposure for every unit. The average effect over the fixed population is $4/3$, and the estimator built from equation (1) has expectation $4/3$. The estimator that deletes schools as the script does has expectation $14/9$, which is a bias of $2/9$. The population to which it refers has average effect $4/3$ or $1$, depending on the assignment.", "problem": "The documentation of \\texttt{estimates()} identifies its output \\texttt{yT\\_ht} with equation (1). Equation (1) is a sum over the units with $D_i=d_k$. It equals zero when no unit has that exposure, and the unbiasedness stated after it counts such assignments with the value zero. Line 219 replaces that zero by \\texttt{NA}, and the missing value passes to the contrasts. The first aggregation of the replication script, from line 136, then removes the affected schools. The denominator of the estimate, and the population to which it refers, then change from one assignment to the next.", "what_survives": "Equations (1) and (3), Lemma 4.1 and Proposition 4.2 are correct as printed, and line 219 changes no point estimate when every exposure is realized. The package uses a missing value for an empty Hajek denominator; the article adopts the convention $0/0=0$ on p.~1922. The available files do not identify the code that produced Table 1, and no entry of Table 1 or Table 2 is shown to be wrong."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E018", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of Table 1, titled Results from school friends' network simulated experiment. The column headings are Estimator, Estimand, Bias, S.D., RMSE, Mean S.E., 95 percent CI Coverage and 90 percent CI Coverage. Red outlines surround the two headings 95 percent CI Coverage and 90 percent CI Coverage. Below the headings are the three HT rows. For tau(d01, d00) the entries are 0.00, 0.04, 0.04, 0.05, 0.960 and 0.924. For tau(d10, d00) they are 0.00, 0.10, 0.10, 0.19, 0.986 and 0.970. For tau(d11, d00) they are 0.00, 0.13, 0.13, 0.28, 0.990 and 0.970.", "caption": "Table 1, p. 1933 (PDF p. 22): the title, the column headings and the three HT rows. The outlines mark the two coverage columns, which the coverage function of the script is written to reproduce.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-E019.png"}], "formal_release": null, "formal_statements": [], "headline": "The coverage function of the replication script takes the square root of the absolute value of each variance estimate and leaves missing comparisons out of the denominator, which departs from the interval of Proposition 6.2 when a variance estimate is negative or a comparison is missing; no entry of the published Table 1 is shown to be affected.", "id": "P013-E019", "location": "Replication script misc/ReplicationAoAS.R of the R package interference, lines 247, 261, 394 and 408; the paper’s counterparts are Table 1, columns “95% CI Coverage” and “90% CI Coverage”, p. 1933 (PDF p. 22), and the interval of Proposition 6.2, p. 1926 (PDF p. 15)", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Script, lines 247, 261, 394 and 408 (one line of code, broken here after the argument list).}\n\\[\n\\begin{aligned}\n&\\texttt{coverage <- function(est,var,tau)}\\\\\n&\\qquad\\texttt{mean(abs(est-tau) < alphanorm*}\\hl{\\texttt{abs(var)}}\\texttt{\\textasciicircum.5,}\\hl{\\texttt{na.rm=TRUE}}\\texttt{)}\n\\end{aligned}\n\\]\n\n\\emph{Proposition~6.2, p.~1926.} Given Conditions 3, 5, and 6, Wald-type intervals constructed as\n\\[\n\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l) \\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]}\n\\]\nwill tend to cover $\\tau_{\\mathrm{HT}}(d_k,d_l)$ at least $100(1-\\alpha)\\%$ of the time for large $N$.\n\n\\emph{Table~1, note, p.~1933.} CI = Normal approximation confidence interval.", "corrected": "To report the coverage of the interval of Proposition~6.2 as printed, the function should \\hl{flag} each replication whose variance estimate is negative or not finite, in place of taking its absolute value, and flag \\hl{any other undefined coverage comparison}. The script should then report the number of replications, the number evaluated, the number with a negative variance estimate, and the number missing for another reason. A rate over the evaluated replications is a conditional rate and should be labelled as one; a rate over all replications needs a stated rule for those in which the interval is undefined. Whichever variance rule is used, state the endpoint convention: use \\hl{\\texttt{<=} for a closed interval}, or specify an open interval if \\texttt{<} is retained.\n\nAlternatively, the absolute value may be kept, if the script states that its interval is\n\\[\n\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l) \\pm z_{1-\\alpha/2}\\sqrt{\\hl{\\bigl|\\widehat{\\operatorname{Var}}[\\widehat{\\tau}_{\\mathrm{HT}}(d_k,d_l)]\\bigr|}} ,\n\\]\nreports how often the absolute value changes the estimate, and reports the denominator. Removing \\texttt{abs()} and keeping \\texttt{na.rm=TRUE} is not sufficient, since negative estimates would then be dropped without notice. Truncation at zero is a third possible rule; I do not select one here.", "evidence": "The evidence is an exact counterexample and an evaluation of the script's own definitions. The example has four units and two exposures with $D_i = Z_i$, where $1$ stands for $d_k$ and $0$ for $d_l$. The pair $(Z_1, Z_2)$ equals $(0,0)$ or $(1,1)$ with probability $1/100$ each, and $(0,1)$ or $(1,0)$ with probability $49/100$ each; $Z_3$ and $Z_4$ are independent fair coins, independent of $(Z_1, Z_2)$. The potential outcomes are $(y_1(d_l), \\ldots, y_4(d_l)) = (1, 1, 0, 0)$ and $(y_1(d_k), \\ldots, y_4(d_k)) = (3/2, 3/2, 0, 0)$, and the average effect is $\\tau = 1/4$. Every exposure probability is $1/2$, and every joint exposure probability of two distinct units is positive. Computing $\\widehat\\tau$ and the variance estimate (11) for each of the 16 assignments, I find three cases.\n\\begin{audittable}{lrrr}{The estimate and the variance estimate (11) in the four-unit example, by the assignment of the first two units.}\n$(Z_1, Z_2)$ & Probability & $\\widehat\\tau$ & $\\widehat V$ \\\\\n$(1,1)$ & $1/100$ & $3/2$ & $-207/8$ \\\\\n$(0,0)$ & $1/100$ & $-1$ & $-23/2$ \\\\\n$(0,1)$ or $(1,0)$ & $49/50$ & $1/4$ & $349/784$ \\\\\n\\end{audittable}\nIn the first two cases the squared error is $25/16$. With $z = 2$, the squared radius $z^2 |\\widehat V|$ used by the function is $207/2$ and $46$. Both cases count as covered, and the coverage over the design is $1$. If the variance estimate is truncated at zero instead, both cases fail and the coverage is $49/50$. The example agrees with Proposition~5.6, since $\\mathbb{E}[\\widehat V] = 1/16$ and $\\operatorname{Var}(\\widehat\\tau) = 1/32$.\n\nOn the assignment $(1,1,0,0)$, the estimation functions of the package return $\\widehat\\tau = 1.5$ and $\\widehat V = -25.875$, and the square root of $\\widehat V$ is not a number. Each of the four copies of the function counts this assignment as covered at both critical values of the script. For the second departure, I take the inputs $\\widehat\\tau = (1/2, 1/2, 1/2)$, $\\tau = 0$ and $\\widehat V = (1, -1, \\mathrm{NA})$. The comparisons are TRUE, TRUE and NA, and the function returns $1$ from two of the three replications; counting the missing one as a failure gives $2/3$. The four-unit example has no interference, a special case that the setting of the paper allows. It is not the simulation design of Section~9.", "problem": "The script says in its first line that it replicates Table~1, and it labels the output of this function as 95\\% and 90\\% coverage. Let $\\widehat V$ be the variance estimate in one replication of the simulated experiment, $\\widehat\\tau$ its estimate and $z$ the critical value. The function returns the share of replications such that $|\\widehat\\tau - \\tau| < z\\sqrt{|\\widehat V|}$, and it takes this share over the replications whose comparison is not missing. Two departures are established here. First, the variance estimator (11) can be negative, and the printed interval is then undefined; the function replaces $\\widehat V$ by $|\\widehat V|$ and evaluates the replication. Second, a replication whose comparison is missing is removed from the numerator and from the denominator. The function then returns a rate among the evaluated replications and not a rate among all replications. The script reports neither how often these cases occur nor the denominator.\n\nThe strict \\texttt{<} also excludes exact endpoint matches, even when the variance estimate is nonnegative and the comparison is defined. The printed $\\pm$ formula leaves the endpoint convention unstated, so I do not treat this alone as a further confirmed error.\n\nTaking the absolute value is not in itself invalid. Since $|\\widehat V| \\ge \\widehat V$, it defines a different variance estimator, which is nonnegative and remains conservative in expectation. The defect is that the function is presented as the coverage of the printed interval, while it computes the coverage of a modified interval among the evaluated replications. Its effect on a reported number is not established. The network and outcome files that the script reads are not available. No negative variance estimate or missing comparison is shown to have occurred in the published simulation. Nothing identifies this script as the code that produced the published Table~1: the script computes the HT and Hajek rows only, and the table reports five estimators.", "what_survives": "The function feeds only the two coverage columns of the two tables that the script writes. The point estimates, the bias, the standard deviation and the root mean square error of the script do not pass through it. Its ``Mean SE'' function is a separate matter, recorded as \\fid{P013-E016}. No entry of the published Table~1 is shown to be wrong, and no replacement value is available. The estimands, the estimators, Proposition~5.6 and the asymptotic results of the paper are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E019", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop showing expression (8) on three lines: the first line is a double sum of indicator products divided by pi_ij(d_k, d_l) times Y_i over pi_i(d_k) times Y_j over pi_j(d_l); the second line is the factor pi_ij(d_k, d_l) minus pi_i(d_k) pi_j(d_l); the third line subtracts a sum over i of two terms in Y_i squared. Below are Proposition 5.3 and the sentence beginning 'A proof for Proposition 5.3 follows from noting that the term on the second line in expression (8)', in which red boxes mark the words 'second line'.", "caption": "Expression (8), Proposition 5.3 and the sentence after it, p. 1923 (PDF p. 12). Expression (8) is set on three lines; the marked words point to its second line, and the term meant is on its third.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "“The term on the second line in expression (8)” should read “the term on the last line in expression (8)”.", "id": "P013-N001", "location": "Sentence after Proposition 5.3, p. 1923 (PDF p. 12)", "pdf_page": 12, "printed_page": "1923", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``A proof for Proposition 5.3 follows from noting that the term on the \\hl{second} line in expression (8) has expected value less than or equal to the quantity in the last line of expression (6), again via Young's inequality; see Aronow and Samii [(2013), Proposition 2, reproduced in the Appendix below] for greater detail.''", "corrected": "``A proof for Proposition 5.3 follows from noting that the term on the \\hl{last} line in expression (8) has expected value less than or equal to the quantity in the last line of expression (6), again via Young's inequality.''", "evidence": "The evidence is a check of the page and a short derivation. The last line of (8) has expectation $-(1/2)\\sum_i[y_i(d_k)^2+y_i(d_l)^2]$, and the last line of (6) is $-\\sum_i y_i(d_k)y_i(d_l)$. Their difference is $-(1/2)\\sum_i[y_i(d_k)-y_i(d_l)]^2\\le0$, which is the inequality that the sentence states.", "problem": "Expression (8) is set on three lines. Its second line is the factor $\\times[\\pi_{ij}(d_k,d_l)-\\pi_i(d_k)\\pi_j(d_l)]$ of the double sum. The term that the sentence means is the sum on the third and last line,\n\\[\n-\\sum_{i\\in U}\\left[\\frac{\\mathbf{I}(D_i=d_k)Y_i^2}{2\\pi_i(d_k)}+\\frac{\\mathbf{I}(D_i=d_l)Y_i^2}{2\\pi_i(d_l)}\\right].\n\\]", "what_survives": "Proposition 5.3 and its proof are unaffected. Only the reference to the line changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N001", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Corollary 8.1 and the line after it. A red box marks the weight w with subscript i, z prime in the second inner sum, which runs over assignments z prime with f of z prime and theta i equal to d l.", "caption": "Corollary 8.1, p. 1930 (PDF p. 19). The marked weight belongs to the sum over assignments that induce exposure $d_l$.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N003.png"}, {"alt": "Crop of expression (16) and the three lines after it. A red box marks the words 'w i, z equals p z divided by pi i of d k' after the word 'where'.", "caption": "Proposition 8.1, p. 1929 (PDF p. 18), expression (16) and the definition of the weights. The marked definition has $\\pi_i(d_k)$ in the denominator.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T17", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T17"}], "headline": "In Corollary 8.1 the weight in the second sum is $w_{i,\\mathbf{z}'}=p_{\\mathbf{z}'}/\\pi_i(d_l)$: each weight divides by the probability of the exposure that its assignment induces.", "id": "P013-N003", "location": "Corollary 8.1, p. 1930 (PDF p. 19); definition of the weights after expression (16), p. 1929 (PDF p. 18)", "pdf_page": 19, "printed_page": "1930", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P013-T17"], "text": {"as_printed": "Proposition 8.1, p.~1929: ``where $w_{i,\\mathbf{z}}=p_{\\mathbf{z}}/\\hl{\\pi_i(d_k)}$.''\n\nCorollary 8.1, p.~1930: Under Condition 7,\n\\[\n\\mathrm{E}\\bigl[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)\\bigr]=\\frac{1}{N}\\sum_{i=1}^N\\left[\\sum_{\\mathbf{z}:f(\\mathbf{z},\\theta_i)=d_k}w_{i,\\mathbf{z}}\\,y_i^r(\\mathbf{z})-\\sum_{\\mathbf{z}':f(\\mathbf{z}',\\theta_i)=d_l}\\hl{w_{i,\\mathbf{z}'}}\\,y_i^r(\\mathbf{z}')\\right].\n\\]", "corrected": "Proposition 8.1: ``where $w_{i,\\mathbf{z}}=p_{\\mathbf{z}}/\\hl{\\pi_i(f(\\mathbf{z},\\theta_i))}$'', for assignments $\\mathbf{z}$ whose induced exposure $f(\\mathbf{z},\\theta_i)$ has positive probability for unit $i$. With this definition, Corollary 8.1 states that\n\\[\n\\mathrm{E}\\bigl[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_l)\\bigr]=\\frac{1}{N}\\sum_{i=1}^N\\left[\\sum_{\\mathbf{z}:f(\\mathbf{z},\\theta_i)=d_k}\\frac{p_{\\mathbf{z}}}{\\pi_i(d_k)}\\,y_i^r(\\mathbf{z})-\\sum_{\\mathbf{z}':f(\\mathbf{z}',\\theta_i)=d_l}\\frac{p_{\\mathbf{z}'}}{\\hl{\\pi_i(d_l)}}\\,y_i^r(\\mathbf{z}')\\right].\n\\]", "intended_meaning": "Proposition 8.1 concerns one exposure $d_k$, and its weight $p_{\\mathbf{z}}/\\pi_i(d_k)$ is the probability of the assignment $\\mathbf{z}$ given that unit $i$ receives exposure $d_k$. Corollary 8.1 applies the proposition once to $d_k$ and once to $d_l$, and the weight in its second sum is $p_{\\mathbf{z}'}/\\pi_i(d_l)$. This exposure-specific reading is correct. By Condition 7, for any exposure $d$ such that $\\pi_i(d)>0$, I have\n\\[\n\\mathrm{E}\\left[\\mathbf{I}(D_i=d)\\frac{Y_i}{\\pi_i(d)}\\right]=\\sum_{\\mathbf{z}:f(\\mathbf{z},\\theta_i)=d}\\frac{p_{\\mathbf{z}}}{\\pi_i(d)}\\,y_i^r(\\mathbf{z}) .\n\\]\nSubtracting this identity for $d_l$ from the identity for $d_k$, summing over $i$ and dividing by $N$, I obtain the corollary. If the printed definition is kept literally in the second sum, the formula is false in general. For one unit and four assignments with probabilities $1/8$, $1/8$, $3/8$, $3/8$ and outcomes $0$, $1$, $2$, $3$, of which the first two induce $d_k$ and the last two $d_l$, the bracket equals $1/2-5/2=-2$ with the weights as intended and $1/2-15/2=-7$ with the denominator $\\pi_i(d_k)$ in both sums. Proposition 8.1, the corollary as intended and the derivation in Appendix A.8 are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N003", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the last eight lines of the paragraph that begins 'This experiment is repeated independently', with the first two lines of the next paragraph. One red box marks the number 0.00067 in the sentence about the student with degree of 39. Two more red boxes mark the words 'there were no cases with probabilities less than 0.01 for either the indirect- or no-exposure conditions', which run over two lines.", "caption": "Section 9, p. 1931 (PDF p. 20). The two marked statements cannot both describe exposure probabilities of the design.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N004.png"}, {"alt": "Crop of six lines of the first paragraph of Section 9. Red boxes mark the words 'randomly assigned without replacement and with uniform probability to 1/10 of individuals in a school network', which run over two lines.", "caption": "Section 9, p. 1930 (PDF p. 19). The marked words state the design of the simulation.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T11", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T11"}], "headline": "The value 0.00067 for the student of degree 39 and the statement that no student has a no-exposure probability below 0.01 cannot both be exposure probabilities of the stated design: at least one of them is inaccurate, or both are estimates from a finite number of simulated assignments.", "historical_evidence_type": "lean_proof", "id": "P013-N004", "location": "Section 9, p. 1931 (PDF p. 20), paragraph beginning “This experiment is repeated independently”; the design is stated on p. 1930 (PDF p. 19)", "pdf_page": 20, "printed_page": "1931", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P013-T11"], "text": {"as_printed": "p.~1930: ``We simulate experiments in which a treatment, $\\mathbf{Z}$, is randomly assigned without replacement and with uniform probability to 1/10 of individuals in a school network.''\n\np.~1931: ``To give an idea of the range of exposure probabilities, for the student with degree of 39, the probability of isolated direct exposure was \\hl{0.00067}. In Figure 3 of the Appendix, we display the cumulative distribution functions for the four exposure probabilities. About 3\\% of students have an exposure probability of less than 0.01 for the direct + indirect exposure condition, 0.5\\% for isolated direct exposure, and then there were \\hl{no cases with probabilities less than 0.01 for either the indirect- or no-exposure conditions}.''", "corrected": "For a student with 39 friends in a school of $N$ students, of whom $n\\ge 1$ are treated uniformly at random without replacement with $N\\le 10n+5$, the probabilities of the design satisfy\n\\[\n\\pi_i(d_{00})=\\frac{N-39-n}{n}\\,\\pi_i(d_{10}),\\qquad\\text{and hence}\\qquad \\pi_i(d_{00})<9\\,\\pi_i(d_{10})\\quad\\text{when } \\pi_i(d_{10})>0 .\n\\]\nThe passage can be corrected in one of two ways.\n\nIf the probabilities were estimated by simulation: ``for the student with degree of 39, the \\hl{estimated} probability of isolated direct exposure was 0.00067 \\ldots\\ there were no cases with \\hl{estimated} probabilities less than 0.01 for either the indirect- or no-exposure conditions'', together with the estimator used, the number $R$ of simulated assignments and the precision of the estimates, and with Figure 3 labelled as showing estimated probabilities. A statement about estimated probabilities does not show that no probability of the design is below 0.01.\n\nIf probabilities of the design are meant, the two binomial expressions above are to be evaluated for the student's school, and at least one of the two statements replaced. I cannot give the replacement values, because the size of the school and the number treated are not in the available materials.", "evidence": "This finding rests on a derivation by counting. Let the student's school have $N$ students, of whom $n$ are treated, every set of $n$ being equally likely, and let $p$ and $q$ be the student's probabilities of $d_{10}$ and $d_{00}$. There are $N-40$ students who are neither the student nor one of the 39 friends, and\n\\[\np=\\frac{\\binom{N-40}{n-1}}{\\binom{N}{n}},\\qquad q=\\frac{\\binom{N-40}{n}}{\\binom{N}{n}},\\qquad \\frac{q}{p}=\\frac{N-39-n}{n}\\quad\\text{when }p>0 .\n\\]\nIf $n=N/10$, the ratio is $9-39/n<9$. If $n$ is $N/10$ rounded to the nearest integer, then $N\\le 10n+5$ and the ratio is at most $9-34/n<9$; the ratio is also below 9 if the number treated is rounded down or up. If 0.00067 is a rounded value, then $p\\le 0.000675$, and I obtain $q\\le 9p\\le 0.006075<0.01$. Conversely, $q\\ge 0.01$ implies that $p\\ge 1/900>0.000675$. In an exhaustive count for $N=44$ and $n=4$ (135,751 assignments), I find $p=4/135751$ and $q=1/135751$, in agreement with the formula. The counts, the identity $q=((N-39-n)/n)\\,p$ and the bounds have Lean proofs for the uniform design, with $1\\le n\\le N-39$ and, for the bounds, $N\\le10n+5$ (\\fid{P013-T11}).\n\nEstimates behave differently. With $R=1500$ and no simulated assignment placing the student in $d_{10}$, the estimate of $p$ is $1/1501=0.000666\\ldots$, which rounds to 0.00067, and there exist 1500 admissible assignments in a school with $N=300$ and $n=30$ for which the estimates are $1/1501$ and $21/1501>0.01$. The replication script distributed with the \\texttt{interference} package uses 1500 assignments and this estimator. It is not established that this script produced the published numbers; it describes 139 networks, and the paper 144 school classes.", "problem": "In the exposure mapping of Section 9, a student is in isolated direct exposure $d_{10}$ when the student is treated and no friend is, and in no exposure $d_{00}$ when neither the student nor any friend is treated. For a student with 39 friends these two probabilities are tied together under the stated design: the second is less than nine times the first. If the probability of $d_{10}$ is 0.00067, then the probability of $d_{00}$ for the same student is below 0.01, and the paragraph says that there were no such cases. The paragraph does not say how the probabilities were computed. Section 4 (pp.~1919--1920) describes estimates of the form $(X_i+1)/(R+1)$ from $R$ simulated assignments, and two such estimates can agree with both printed statements. Which reading applies to the published numbers is not established.", "what_survives": "The theoretical results are stated for given exposure probabilities and do not depend on these two descriptive statements, and no later result uses the value 0.00067. The entries of Table 1 were not recomputed, and this finding neither shows them wrong nor confirms them. Whether the published simulation used estimated probabilities remains unresolved. The summaries of 3\\% and 0.5\\% are not in question individually, and the application of Section 10 uses another design and exposure mapping."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N004", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first three items of the bulleted list of estimators. A red box marks the words 'The Hajek ratio estimator given in expression (15)' in the second item.", "caption": "Section 9, p. 1932 (PDF p. 21), list of the estimators of the simulation. The marked item names expression (15) as the Hajek estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N006.png"}, {"alt": "Crop of expression (15) with two lines of text above and two below. A red box marks the formula: mu hat H of d k equals the sum over i from 1 to N of the indicator of D i equal to d k times Y i over pi i of d k, divided by the sum over i from 1 to N of the indicator times 1 over pi i of d k.", "caption": "Expression (15), p. 1928 (PDF p. 17). The marked ratio has sums over the whole population in its numerator and its denominator.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N006-b.png"}, {"alt": "Listing of lines 225 to 235 in monospaced type with line numbers. One red box marks lines 226 to 228, which assign to mu_h the Horvitz-Thompson total divided by the row sums of exposure indicators over exposure probabilities. A second red box marks line 231, 'yT_h is assigned mu_h times N'.", "caption": "Package interference, file R/estimates.R, lines 225–235, typeset from the source file with line numbers added. The marked lines compute a school’s Hajek ratio and multiply it by the size of the school.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N006-c.png"}, {"alt": "Listing of lines 143 to 152 in monospaced type with line numbers. One red box marks line 143, which assigns to tau01_h the sum of yT_h_ind1 over the sum of N minus the sum of yT_h_no over the sum of N. A second red box marks lines 150 to 152, which assign weighted means of the school-level Hajek contrasts with weights N.", "caption": "Package interference, file misc/ReplicationAoAS.R, lines 143–152, typeset from the source file with line numbers added. The marked lines combine the school-level Hajek quantities across schools.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N006-d.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication script distributed with the interference package averages the Hajek ratios of the schools with weights proportional to school size, and this differs in general from the single pooled ratio of expression (15), which Section 9 names.", "id": "P013-N006", "location": "Section 9, list of estimators, p. 1932 (PDF p. 21); expression (15), p. 1928 (PDF p. 17); interference package, misc/ReplicationAoAS.R lines 141–143 and 150–152, R/estimates.R lines 226–231", "pdf_page": 21, "printed_page": "1932", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 9, p.~1932: ``The Hajek ratio estimator \\hl{given in expression (15)}, with the associated linearized variance estimator;''\n\nExpression (15), p.~1928:\n\\[\n\\widehat{\\mu_H}(d_k)=\\frac{\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)}}{\\sum_{i=1}^N\\mathbf{I}(D_i=d_k)\\frac{1}{\\pi_i(d_k)}} .\n\\]\n\n\\texttt{R/estimates.R}, lines 226--228 (line breaks joined) and 231, run once for each school with \\texttt{N} the size of the school:\n\n\\texttt{mu\\_h <- yT\\_ht / rowSums(t(obs\\_exposure) / obs\\_prob\\_exposure\\_individual\\_kk, na.rm = T)}\n\n\\texttt{yT\\_h <- mu\\_h * N}\n\n\\texttt{misc/ReplicationAoAS.R}, line 143:\n\n\\texttt{tau01\\_h <- estimates\\_all\\_schools[, sum(yT\\_h\\_ind1)/sum(N)-sum(yT\\_h\\_no)/sum(N), by=tr]}", "corrected": "Either of the following removes the discrepancy. Expression (15) needs no change.\n\nTo compute expression (15) for the simulation, pool the schools before taking the ratio:\n\\[\n\\widehat{\\mu_H}(d_k)=\\frac{\\hl{\\sum_s A_{sk}}}{\\hl{\\sum_s B_{sk}}},\\qquad A_{sk}=\\sum_{i\\in s}\\mathbf{I}(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)},\\qquad B_{sk}=\\sum_{i\\in s}\\mathbf{I}(D_i=d_k)\\frac{1}{\\pi_i(d_k)},\n\\]\nwith the residuals of the linearized variance estimator taken about this pooled mean. A school in which no student receives exposure $d_k$ contributes zero to both sums, and $\\sum_s B_{sk}$ must be positive.\n\nAlternatively, describe the estimator that the script computes: ``The Hajek ratio estimator given in expression (15), \\hl{applied within each school and averaged over schools with weights proportional to the number of students}, with the associated linearized variance estimator''. In symbols, this is $\\sum_s (N_s/N)\\,A_{sk}/B_{sk}$ when every $B_{sk}>0$. The script marks an empty school exposure as missing and either propagates or omits it in its different aggregations.\n\nA correction to the published Table 1 cannot be given without the original computation.", "evidence": "This finding rests on an exact counterexample and on running the package code on it. There are two schools of 30 students: school A is a cycle and school B is a path, each on the vertices $1,\\dots,30$ in order. In each school three students are treated uniformly at random, as in the design of Section 9, and the treated set is $\\{2,3,10\\}$ in both. Outcomes under no exposure are 1 in A and 2 in B, and they are multiplied by $2$, $3/2$ and $5/4$ under $d_{11}$, $d_{10}$ and $d_{01}$, as in Section 9. All exposure probabilities are positive, and all four exposure groups are nonempty in both schools. In exact arithmetic, I find the following contrasts for the two computations.\n\\begin{audittable}{lccc}{Estimated contrasts in the two-school example: the script's aggregation and pooled expression (15).}\nContrast & Script & Pooled (15) & Pooled minus script \\\\\n$\\tau(d_{11},d_{00})$ & $3/2$ & $1929/1285$ & $3/2570$ \\\\\n$\\tau(d_{10},d_{00})$ & $3/4$ & $3861/5140$ & $3/2570$ \\\\\n$\\tau(d_{01},d_{00})$ & $3/8$ & $5669/12850$ & $3401/51400$ \\\\\n\\end{audittable}\nFor the indirect effect, the value from the script is $0.375$ and the pooled ratio is about $0.4412$. The package's school-level function and lines 141--143 and 150--152 of the script, run on this example, all return the values in the column headed Script.", "problem": "Section 9 names expression (15) as the Hajek estimator of the simulation, and expression (15) is a ratio of two sums over the whole population, here the students of all schools. The script computes another quantity. It calls the estimation function once for each school, obtains the school's Hajek ratio multiplied by the size of the school, sums these products over schools and divides by the total number of students. For school $s$ of size $N_s$, let $A_{sk}=\\sum_{i\\in s}\\mathbf{I}(D_i=d_k)Y_i/\\pi_i(d_k)$ and $B_{sk}=\\sum_{i\\in s}\\mathbf{I}(D_i=d_k)/\\pi_i(d_k)$, and let $N=\\sum_s N_s$. When every school has a positive denominator $B_{sk}$, the script computes\n\\[\n\\sum_s\\frac{N_s}{N}\\,\\frac{A_{sk}}{B_{sk}},\\qquad\\text{and expression (15) is}\\qquad \\frac{\\sum_s A_{sk}}{\\sum_s B_{sk}}=\\sum_s\\frac{B_{sk}}{\\sum_t B_{tk}}\\,\\frac{A_{sk}}{B_{sk}} .\n\\]\nThe first weights the school ratios by $N_s/N$, and the second by the realized $B_{sk}/\\sum_t B_{tk}$. Although $\\mathrm{E}[B_{sk}]=N_s$, the realized $B_{sk}$ can differ from $N_s$, and the two estimates can differ. Both are ratio estimators of the same mean $\\mu(d_k)$, and the paper does not state a formula for combining schools. The discrepancy is between the script and the pooled ratio; expression (15) is not in error.", "what_survives": "Expression (15) is correct as printed, and an average of the Hajek ratios of the schools, weighted by school size, is a coherent estimator of the same mean. The exposure mappings, the theoretical results and the other estimators are unaffected. No entry of Table 1 is shown to be wrong: the original simulation code and inputs are not available, and which aggregation produced the published table is unresolved. The application of Section 10 and Table 2 are not affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N006", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of nine reference entries, from Angrist and Krueger (1999) to Bowers, Fredrickson and Panagopolous (2013). Red outlines surround four items. The first is the editor's name Ahsenfelter in the entry for Angrist and Krueger. The second is the page range 135 to 144 in the entry for Aronow and Middleton (2013), Journal of Causal Inference 1. The third is the page range 546 to 577 in the entry for Aronow, Samii and Assenova (2015), Polit. Anal. 23. The fourth is the name Panagopolous in the entry for Bowers, Fredrickson and Panagopolous (2013).", "caption": "References, p. 1945 (PDF p. 34), from Angrist and Krueger (1999) to Bowers et al. (2013). The outlines mark four of the six slips: “Ahsenfelter”, the page ranges 135–144 and 546–577, and “Panagopolous”. The entries for Paluck (2011) and Neyman (1990) are on pp. 1946 and 1947.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N011.png"}], "formal_release": null, "formal_statements": [], "headline": "Five reference entries have a misspelled name, a wrong page range or a wrong journal title, one of the misspellings recurs in the text, and the entry for Neyman (1990) is out of alphabetical order.", "id": "P013-N011", "location": "References, pp. 1945–1947 (PDF pp. 34–36); Section 2, p. 1914 (PDF p. 3)", "pdf_page": 34, "printed_page": "1945–1947", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item Angrist and Krueger (1999), p.~1945: In \\emph{Handbook of Labor Economics} (O. C. \\hl{Ahsenfelter} and D. Card, eds.) 3.\n\\item Aronow and Middleton (2013), p.~1945: \\emph{Journal of Causal Inference} 1 135--\\hl{144}.\n\\item Aronow, Samii and Assenova (2015), p.~1945: \\emph{Polit. Anal.} 23 \\hl{546}--577.\n\\item Bowers et al.\\ (2013), p.~1945: BOWERS, J., FREDRICKSON, M. M. and \\hl{PANAGOPOLOUS}, C. (2013). The text on p.~1914 reads: Bowers, Fredrickson and \\hl{Panagopolous} (2013) apply exact tests to evaluate parameters in models of spillover processes.\n\\item Paluck (2011), p.~1946: \\emph{\\hl{J. Exp. Psychol.}} 47 350--358.\n\\item Neyman (1990), p.~1947, author headings in printed order: SOBEL, M. E. (2006). \\hl{NEYMAN, J. S. (1990).} TCHETGEN TCHETGEN, E. J. and VANDERWEELE, T. J. (2012).\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Angrist and Krueger (1999): In \\emph{Handbook of Labor Economics} (O. C. \\hl{Ashenfelter} and D. Card, eds.) 3.\n\\item Aronow and Middleton (2013): \\emph{Journal of Causal Inference} 1 135--\\hl{154}.\n\\item Aronow, Samii and Assenova (2015): \\emph{Polit. Anal.} 23 \\hl{564}--577.\n\\item Bowers et al.\\ (2013): BOWERS, J., FREDRICKSON, M. M. and \\hl{PANAGOPOULOS}, C. (2013). On p.~1914: Bowers, Fredrickson and \\hl{Panagopoulos} (2013) apply exact tests to evaluate parameters in models of spillover processes.\n\\item Paluck (2011): \\emph{\\hl{J. Exp. Soc. Psychol.}} 47 350--358.\n\\item Neyman (1990): the entry is placed \\hl{after Middleton and Aronow (2011) and before Ogburn and VanderWeele (2014)}.\n\\end{itemize}", "evidence": "The evidence is a check of the pages for the order of the list, and a check of the publishers' records for the other five items. The records are those of Elsevier for volume 3A of the \\emph{Handbook of Labor Economics} and for Paluck (2011), of Cambridge University Press for the two articles in \\emph{Political Analysis}, and of De Gruyter for the \\emph{Journal of Causal Inference}.", "problem": "The records of the publishers give Ashenfelter and Panagopoulos for the two names, pp.~135--154 for Aronow and Middleton (2013), pp.~564--577 for Aronow, Samii and Assenova (2015), and the \\emph{Journal of Experimental Social Psychology} for Paluck (2011). The reference list is otherwise alphabetical, and the entry for Neyman (1990) belongs on p.~1946, between Middleton and Aronow (2011) and Ogburn and VanderWeele (2014).", "what_survives": "Each cited work remains identifiable from its authors and title, and no result or estimate depends on these details."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N011", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. In lines 169 to 177 of R/make_tr_vec_bernoulli.R, a red box marks the condition R > 2 ^ N of an if statement that stops the function with the message 'R must be smaller than 2 ^ N, the number of possible treatment assignements'. In lines 179 to 187, red boxes mark line 181, a while loop whose condition is that the new vector duplicates one of the vectors already sampled, and line 183, which draws a new vector with rbinom. Below, lines 83 and 88 of R/make_exposure_prob.R divide matrix products of exposure indicators by R + 1 and by R.", "caption": "Source code of the package, not a page of the article: the check on the number of replicates (lines 169–177) and the sampling loop (lines 179–187) of make_tr_vec_bernoulli_indiv, and the two frequency formulas of R/make_exposure_prob.R (lines 83 and 88). The marked lines 181 and 183 redraw an assignment vector until it differs from every vector already kept, and the marked condition on line 169 limits $R$ to the number of distinct vectors.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N012-code.png"}, {"alt": "Crop of the paragraph about approximating exposure probabilities by simulation. A red box marks the words 'produce R random replicate z's based on the randomization plan'. The paragraph ends with the estimators (I-hat-k I-hat-k-prime plus the identity matrix) divided by (R + 1), and I-hat-k I-hat-l-prime divided by R.", "caption": "Section 3, p. 1917 (PDF p. 6): the original formulas used by the later package. The marked words describe replicates drawn from the randomization plan; the two estimators at the end of the paragraph weight the replicates equally. The formulas are correct for independent repetitions of the whole design; Appendix A.1’s conflicting without-replacement wording is addressed separately in the clarity note P013-E010.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N012.png"}], "formal_release": null, "formal_statements": [], "headline": "The sampler for Bernoulli designs in the later package rejects every assignment vector that repeats an earlier one, and the exposure probabilities are then computed with equal weights on the vectors kept, which distorts them (for $N=100$, $p=0.01$ and $R=1000$ the estimated probabilities of treatment average at least $0.01898$, against $0.01$ under the design); the sampler should keep every independent draw.", "id": "P013-N012", "location": "R package interference (commit c3fc7b3): R/make_tr_vec_bernoulli.R, function make_tr_vec_bernoulli_indiv, lines 169–187; R/make_exposure_prob.R, lines 83 and 88. The package DESCRIPTION credits Stephanie Zonszein, Cyrus Samii and P. M. Aronow; the corresponding formulas appear in Section 3, p. 1917 (PDF p. 6).", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P044-N014", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P044-N014", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N014", "work_id": "P044"}], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{R/make\\_tr\\_vec\\_bernoulli.R}, lines 169--177 (written on two lines):\n\n$\\texttt{if (}\\hl{\\texttt{R > 2 \\textasciicircum{} N}}\\texttt{) \\{ stop(paste(\"R must be smaller than\", 2 \\textasciicircum{} N,}$\n\n$\\texttt{\", the number of possible treatment assignements\")) \\}}$\n\nLines 179--187 (the inner loop joined on one line):\n\n\\texttt{for (i in 1:R) \\{}\n\n\\texttt{vec <- rbinom(N, 1, p)}\n\n\\texttt{\\hl{while (any(duplicated(rbind(vec, tr\\_vec\\_sampled[1:i - 1,]))))} \\{ \\hl{vec <- rbinom(N, 1, p)} \\}}\n\n\\texttt{tr\\_vec\\_sampled[i,] <- vec}\n\n\\texttt{\\}}\n\n\\texttt{R/make\\_exposure\\_prob.R}, lines 83 and 88:\n\n\\texttt{(I\\_exposure[[i]] \\%*\\% t(I\\_exposure[[i]]) + diag(N)) / (R + 1)}\n\n\\texttt{(I\\_exposure[[i]] \\%*\\% t(I\\_exposure[[j]])) / R}\n\nSection 3, p.~1917: ``One may nonetheless approximate the $\\boldsymbol{\\pi}_i$'s and joint probabilities with arbitrary precision through simulation, that is, produce $R$ random replicate $\\mathbf{z}$'s based on the randomization plan.''", "corrected": "\\texttt{R/make\\_tr\\_vec\\_bernoulli.R}, lines 169--187, \\hl{with the check of lines 169--177 and the inner loop of lines 181--185 deleted}:\n\n\\texttt{for (i in 1:R) \\{}\n\n\\texttt{vec <- rbinom(N, 1, p)}\n\n\\texttt{tr\\_vec\\_sampled[i,] <- vec}\n\n\\texttt{\\}}\n\nThe sampler then keeps every independent draw, repeated vectors included, and $R$ is no longer limited by $2^N$. The unit-level sampler's documentation should say that repeated vectors are kept and remove its stated bound on $R$. The two formulas of \\texttt{R/make\\_exposure\\_prob.R} stay as they are. For any other design, each replicate should be an independent draw of the whole randomization, with all of its blocks and clusters, and the replicates should be drawn independently of the assignment observed in the experiment. If all distinct assignment vectors are enumerated instead, each must be weighted by its probability under the design, as in $\\mathbf{I}_k\\mathbf{P}\\mathbf{I}_l'$. These weights alone do not correct a random subset of the vectors. Appendix A.1 itself says ``without replacement''; that wording is addressed separately in the clarity note \\fid{P013-E010} and does not establish use of this package for the publication.", "evidence": "The evidence is an exact calculation for a design of ordinary size, a run of the unchanged code on that design, and an exact counterexample with two units.\n\nLet $N=100$ and $p=1/100$, and let the exposure of a unit be its own treatment. Among $R$ distinct assignment vectors, at most one has no treated unit and at most $100$ have exactly one treated unit. Every further vector has at least two. For $R=1000$ the vectors kept then contain at least $100+2\\cdot899=1898$ treated entries. I call the fraction of the vectors kept in which a unit is treated its raw frequency of treatment. Averaged over the units, the raw frequency of treatment is at least $1898/(100\\cdot1000)=0.01898$. This bound holds for every sample that the sampler can return, whatever the seed. The same count gives the other rows of the table.\n\n\\begin{audittable}{rrrr}{Bernoulli design with $N=100$ and $p=0.01$: lower bounds that hold for every sample of $R$ distinct assignment vectors. The probability of treatment under the design is $0.01$. Values that are not exact are rounded down.}\n$R$ & Treated entries & Average raw frequency of treatment & Relative excess over $0.01$ \\\\\n$1{,}000$ & $1{,}898$ & $0.018980$ & $89.80\\%$ \\\\\n$1{,}500$ & $2{,}898$ & $0.019320$ & $93.20\\%$ \\\\\n$10{,}000$ & $24{,}847$ & $0.024847$ & $148.47\\%$ \\\\\n$10{,}593$ & $26{,}626$ & $0.025135$ & $151.35\\%$ \\\\\n$100{,}000$ & $294{,}847$ & $0.029484$ & $194.84\\%$ \\\\\n\\end{audittable}\n\nSince the sampler treats the units symmetrically, the expected raw frequency of each unit obeys the same bounds. The diagonal entries that the package returns add one to each count and to $R$, and for $R=1000$ their average is at least $1998/100100\\approx0.01996$. The raw joint frequency of treatment of a pair of units is the fraction of the vectors kept in which both units are treated. Counting the ordered pairs of treated units in the same way, I find that its average over the ordered pairs of units is at least $1798/(1000\\cdot100\\cdot99)\\approx0.0001816$ for $R=1000$, against $p^2=0.0001$. The value $R=1500$ is the number of replicates in the replication script of the package, and $R=10{,}593$ is the number in the example after Proposition 4.1. The rows apply these numbers to the Bernoulli design. They do not recalculate any result of the article. For small $R$ the bound is not informative: for $R=100$ it is $0.0099$. The distortion is present there as well. For $R=2$ the expected raw frequency of treatment is $0.011057$ to six decimals, which is $10.57\\%$ above $0.01$.\n\nI ran the unchanged sampler with $N=100$, $p=0.01$, $R=1000$ and seed $101$. The $1000$ vectors kept are distinct and contain $2240$ treated entries. The average raw frequency of treatment is $0.0224$, and every unit is treated in at least $12$ of the vectors. On a graph of $50$ disjoint edges, the first-order exposure mapping of the package, which has the four exposures of Section 9, gives the following frequencies.\n\n\\begin{audittable}{lrr}{One run of the unchanged sampler ($N=100$, $p=0.01$, $R=1000$, seed 101) on $50$ disjoint edges: probabilities of the four exposures under the design and their raw frequencies in the run, averaged over the units.}\nExposure & Probability under the design & Average raw frequency \\\\\n$d_{11}$ (direct and indirect exposure) & $0.0001$ & $0.00032$ \\\\\n$d_{10}$ (isolated direct exposure) & $0.0099$ & $0.02208$ \\\\\n$d_{01}$ (indirect exposure) & $0.0099$ & $0.02208$ \\\\\n$d_{00}$ (no exposure) & $0.9801$ & $0.95552$ \\\\\n\\end{audittable}\n\nLet every outcome under treatment equal $1$, so that the total under treatment is $100$. Suppose that this total is estimated by the Horvitz--Thompson estimator with the probabilities that the package returns for this run. Given the vectors kept, the expectation of the estimator over the Bernoulli design is $44.31$, with an empty sum counted as zero. With independent draws, the identity of Appendix A.3 implies that the expectation over the replicates and the design is $100[1-(99/100)^{1001}]$.\n\nFor an exact counterexample with two units, let $N=2$ and $R=4$, and let each unit be treated with probability $p=1/5$. The sampler then keeps each of the four assignment vectors once. Each unit is treated in half of them, although its probability of treatment under the design is $1/5$. When the exposure of a unit is its own treatment, the package returns $3/5$ and $1/5$ for the marginal and joint probabilities of treatment. Their values under the design are $1/5$ and $1/25$. The distortion does not require that the sampler exhaust the assignment vectors: for $R=2$ the expected raw frequency of treatment is $383/1260$. At the observed assignment $(1,0)$ with both outcomes equal to $1$, \\texttt{estimates()} returns a Horvitz--Thompson total of $5/3$ for the treated exposure with variance estimate $10/9$, and the probabilities under the design give $5$ and $20$. I confirmed these values by running the unchanged code.\n\nThe size of the distortion depends on the design. For $p=1/2$ all assignment vectors are equally likely, and the raw frequencies are unbiased. If the first $R$ independent draws are distinct, the sampler returns exactly these draws. For $N=1000$, $p=0.01$ and $R=1500$, the probability that two of them coincide is at most $0.0024$, and the expectations of the raw frequencies differ from those under independent draws by at most the same amount.", "problem": "I could not establish that this implementation was used for computations in the 2017 article. The loop on lines 181--185 redraws any assignment vector that repeats an earlier one. The $R$ vectors kept are then distinct, and they are not independent draws from the Bernoulli design. The function \\texttt{make\\_exposure\\_prob} estimates the exposure probabilities by the unweighted frequencies over these vectors, as in the estimators $(\\widehat{\\mathbf{I}}_k\\widehat{\\mathbf{I}}_k'+\\iota_N)/(R+1)$ and $\\widehat{\\mathbf{I}}_k\\widehat{\\mathbf{I}}_l'/R$ of Section 3. Unweighted frequencies estimate the probabilities under the design when the draws are independent, or when all assignment vectors are equally likely. Under a Bernoulli design with $p\\ne1/2$ the assignment vectors are not equally likely, and the sampler keeps each of them at most once. A vector whose probability exceeds $1/R$ is therefore under-represented in every sample, and less likely vectors take its place. This happens in populations of ordinary size. For $N=100$ and $p=0.01$ there are $2^{100}$ assignment vectors, but the vector with no treated unit has probability $(99/100)^{100}\\approx0.366$. Two independent draws coincide with probability about $0.135$. In this design the less likely vectors have more treated units, and the estimated probabilities of treatment are too large for the numbers of replicates examined in the evidence. The estimated probabilities enter the Horvitz--Thompson estimates and their variance estimates as weights. The check on lines 169--177 follows from the same choice. The distinct vectors are exhausted at $R=2^N$, and the limit $R\\to\\infty$ of Proposition 3.1 is not available to this sampler. The article names Bernoulli designs on a graph of bounded degree as a setting for its asymptotic results (p.~1926). Conditions 1--6 do not exclude the examples below, since a sequence of Bernoulli designs with $p$ fixed at $0.01$ and bounded outcomes can satisfy them, with or without first-order interference on disjoint edges.", "what_survives": "Section 3 and Proposition 3.1 are unaffected when the replicates are drawn independently from the randomization plan, and the two estimators of Section 3 are implemented as printed. The designs, the exposure mappings, the estimands and the results that take the exposure probabilities as known are unchanged. The simulation of Section 9 treats a fixed number of units chosen uniformly; the later package's Table 1 replication script uses the permutation sampler, not this function, and does not establish which program generated the published table. That sampler also rejects repeats by default, but this distortion does not arise when equally likely vectors are sampled without replacement. No entry of Table 1 or Table 2 is shown to be affected by this error. I could not establish when the loop was introduced, who introduced it, or a causal connection between the code and Appendix A.1. I could not identify the code used to compute the exposure probabilities for the application of Section 10 or determine how its replicates were drawn. What is lost is the assurance that the probabilities computed from this sampler are those of the Bernoulli design. In sparse designs such as the one above they are not, and the Horvitz--Thompson estimates and variance estimates that use them inherit the error."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N012", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. Lines 44 to 48 document control_condition as the name of the single condition considered the pure control condition. Line 172 declares control_condition=NULL. In line 314 red boxes mark yT_ht['no'] and != 'no' in the definition of tau_ht. In line 318 red boxes mark mu_h['no'] and != 'no' in the definition of tau_h.", "caption": "Source code of the package, not a page of the article: R/estimates.R, the documentation of control_condition (lines 44–48), its declaration (line 172) and the point contrasts (lines 313–320). The marked terms refer to the exposure named 'no' and not to control_condition.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N013.png"}], "formal_release": null, "formal_statements": [], "headline": "The function estimates() subtracts the exposure named 'no' whatever control_condition is supplied; the two point contrasts should use the supplied control condition.", "id": "P013-N013", "location": "R package interference (commit c3fc7b3): R/estimates.R, lines 313–318; the argument control_condition is declared on line 172 and documented on lines 44–48. The code is not printed in the article; the contrast it computes is equation (3), p. 1920 (PDF p. 9).", "pdf_page": null, "printed_page": null, "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P044-E008", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": "interference", "recorded_as": null, "to": "package", "url": null, "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P044-E008", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E008", "work_id": "P044"}], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{R/estimates.R}, lines 314 and 318:\n\n\\texttt{tau\\_ht <- (1 / N) * (yT\\_ht - yT\\_ht[\\hl{'no'}])[names(yT\\_ht) != \\hl{'no'}]}\n\n\\texttt{tau\\_h <- (mu\\_h - mu\\_h[\\hl{'no'}])[names(mu\\_h) != \\hl{'no'}]}", "corrected": "\\texttt{tau\\_ht <- (1 / N) * (yT\\_ht - yT\\_ht[\\hl{control\\_condition}])[names(yT\\_ht) != \\hl{control\\_condition}]}\n\n\\texttt{tau\\_h <- (mu\\_h - mu\\_h[\\hl{control\\_condition}])[names(mu\\_h) != \\hl{control\\_condition}]}\n\nIf \\texttt{control\\_condition} is \\texttt{NULL}, set it to \\texttt{'no'} before these lines, which keeps the earlier point-estimate behaviour. The same literal in \\texttt{tau\\_dsm} should be replaced in the same way.", "evidence": "The evidence is an exact counterexample, run on the unchanged function. Let two units have independent fair binary assignments, and suppose that the observed assignment is $(1,0)$, that the exposures are named \\texttt{treated} and \\texttt{baseline}, that the outcomes are $(3,1)$ and that \\texttt{control\\_condition = 'baseline'}. The Horvitz--Thompson totals are $6$ and $2$. Both contrasts equal $2$, since $(6-2)/2=2$ and $3-1=2$. The function returns \\texttt{NA} for both, together with the variance estimates $10$ (Horvitz--Thompson) and $0$ (Hajek) for \\texttt{treated}. In a second design with three units and a third exposure named \\texttt{'no'}, the Horvitz--Thompson contrasts $2/3$ and $16/3$ against \\texttt{baseline} are returned as $-14/3$ and $-16/3$, and the Hajek contrasts $2$ and $4$ are returned as $-2$ and $-4$.", "problem": "The function documents \\texttt{control\\_condition} as the name of the pure control exposure and uses it when it computes variances (lines 279--306). The Horvitz--Thompson and Hajek point contrasts on lines 314 and 318 instead subtract, and exclude, the exposure literally named \\texttt{'no'}. With any other control label the contrasts are missing, or, if some other exposure is named \\texttt{'no'}, they are taken against the wrong exposure; the point estimates then do not correspond to the variances returned with them.", "what_survives": "Equation (3) and the Hajek estimator of equation (15) are unaffected, and so is every use of the function with the control exposure named \\texttt{'no'}, which is the label of the package's own exposure mappings. The simulation and the application of the article compare each exposure with no exposure ($d_{00}$ and $d_{000}$), and no published result is shown to change. I read the same literals in \\texttt{R/estimates\\_batch.R} but did not run that function."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N013", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the first paragraph of Section 10. The passage reads: asking students to nominate up to 10 students in their school that they had chosen to spend time with, face to face or online, in the last two weeks. These nominations were used to construct an undirected adjacency graph so that students were considered peers if either student nominated one another. A red box marks the words 'in the last two weeks.'", "caption": "Section 10, first paragraph, p. 1934 (PDF p. 23). The marked words give the nomination question a recall period of two weeks; the questionnaire says “the last few weeks”.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N015.png"}], "formal_release": null, "formal_statements": [], "headline": "“In the last two weeks” should read “in the last few weeks”, which is the wording of the nomination question in the questionnaire released with the data.", "id": "P013-N015", "location": "Section 10, first paragraph, p. 1934 (PDF p. 23)", "pdf_page": 23, "printed_page": "1934", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``At the beginning of the school year, the research team measured students' social networks, asking students to nominate up to 10 students in their school that they had chosen to spend time with, face to face or online, in the last \\hl{two} weeks.''", "corrected": "``At the beginning of the school year, the research team measured students' social networks, asking students to nominate up to 10 students in their school that they had chosen to spend time with, face to face or online, in the last \\hl{few} weeks.''", "evidence": "I checked the published page against the questionnaire and the codebook. Both forms of the questionnaire print the question with the words ``the last few weeks''. The codebook labels the ten nomination variables of the first-wave survey with the same words. The check concerns the written question. It does not establish which period the students had in mind, and the sources record no spoken instructions.", "problem": "The sentence describes the question that measured the networks at the beginning of the school year, and it gives the question a recall period of two weeks. The questionnaire released with the data of the field experiment (ICPSR study 37070) reads ``In the last few weeks I decided to spend time with these students at my school: (in school, out of school, or online)'', followed by ten numbered lines. The written question names no number of weeks.", "what_survives": "The network of Section 10 is built from the recorded nominations: the released script forms an undirected graph from the ten nomination variables. With the nominations held fixed, no exposure, estimate or conclusion of Section 10 depends on the word. One word changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N015", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of source code with line numbers. Line 57 assigns to ind.perms the result of genperms(Zraw, blockvar = withinblock, maxiter = niter). In line 58, which assigns to sch.perms, a red box marks the call genperms(schtreat, clustvar = schid, maxiter = niter). Below, lines 87 to 90 of the second script are a loop over i from 1 to 14 that draws a random order of the labels A, A, B, B and assigns it to the schools of block i.", "caption": "Source code of the field experiment’s scripts, not a page of the article: lines 57–58 of 1. Sanitize and generate perms, probs.R and lines 87–90 of 7. Blocking, randomization, and summary statistics.R. The marked call draws the school lottery with the schools as clusters and with no block variable. Lines 87–90 are the lottery that assigned the schools, two of the four schools in each of 14 blocks.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N016-code.png"}, {"alt": "Crop of five lines of the first paragraph of Section 10. The passage reads: The experiment involved two levels of randomization. First, 28 of 56 schools were randomly selected to host the anti-conflict program, via block randomization. Within all schools, a group of between 40 to 64 students were nonrandomly selected as eligible to participate in the program. A red box marks the words 'via block randomization'.", "caption": "Section 10, first paragraph, p. 1934 (PDF p. 23). The marked words describe the school lottery. They are correct, and the marked call of the script does not implement them.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N016.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T25", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T25"}, {"id": "P013-T26", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T26"}], "headline": "The script that simulates the assignments of the field experiment draws the 28 host schools from all 56 schools and ignores the 14 blocks of four in which the schools were randomized; the call should pass the school blocks, and the sentence of the article, which says “via block randomization”, is correct.", "id": "P013-N016", "location": "Section 10, first paragraph, p. 1934 (PDF p. 23). Code: scripts released with the data of the field experiment (ICPSR study 37070, version 2), 1. Sanitize and generate perms, probs.R, line 58, compared with 7. Blocking, randomization, and summary statistics.R, lines 73–74 and 87–90. The code is not printed in the article.", "pdf_page": 23, "printed_page": "1934", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P037-N031", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P037/", "where": null, "work_id": "P037"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P037-N031", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N031", "work_id": "P037"}], "severity": "Minor", "subject": "paper", "target_ids": ["P013-T25", "P013-T26"], "text": {"as_printed": "Section 10, p.~1934: ``The experiment involved two levels of randomization. First, 28 of 56 schools were randomly selected to host the anti-conflict program, via block randomization.''\n\n\\texttt{1. Sanitize and generate perms, probs.R}, lines 57--58 (trailing comments omitted):\n\n\\texttt{ind.perms <- genperms(Zraw, blockvar = withinblock, maxiter = niter)}\n\n\\texttt{sch.perms <- \\hl{genperms(schtreat, clustvar = schid, maxiter = niter)}}\n\n\\texttt{7. Blocking, randomization, and summary statistics.R}, lines 87--90:\n\n\\texttt{for(i in 1:14) \\{}\n\n\\texttt{Zblock <- sample(c(\"A\",\"A\",\"B\",\"B\"))}\n\n\\texttt{treatment.blind[blockvar == i] <- Zblock}\n\n\\texttt{\\}}", "corrected": "\\texttt{1. Sanitize and generate perms, probs.R}, line 58 (written on two lines):\n\n\\texttt{sch.perms <- genperms(schtreat, \\hl{blockvar = schrb,} clustvar = schid,}\n\n\\texttt{maxiter = niter)}\n\nHere \\texttt{schrb} is the school-block variable of the data, and each school lies in one block. The sentence of Section 10 stays as printed.", "evidence": "The evidence is a reading of the two scripts, an evaluation of the function that the script calls, and an exact calculation. With the arguments of line 58 and the schools of the public data, the design routine of \\texttt{ri} (version 0.9) returns one block of 56 schools with 28 hosts. With the block variable added, it returns 14 blocks of four schools with two hosts each. I ran \\texttt{genperms} to exhaustion on eight synthetic schools in two blocks of four. The call without the block variable enumerates 70 assignments, of which 34 do not have two hosts in each block. The call with the block variable enumerates 36 assignments, and all of them do.\n\nFor the 56 schools, the blocked design has $\\binom{4}{2}^{14}=6^{14}$ equally likely assignments, and the lottery of the script has $\\binom{56}{28}$. Counting the assignments in which two given schools both host, I obtain\n\\[\n\\frac{6^{13}}{6^{14}}=\\frac{1}{6},\\qquad \\frac{3^2\\cdot6^{12}}{6^{14}}=\\frac{1}{4},\\qquad \\frac{\\binom{54}{26}}{\\binom{56}{28}}=\\frac{28\\cdot27}{56\\cdot55}=\\frac{27}{110}\n\\]\nfor two schools of one block under the blocked design, for two schools of different blocks under the blocked design, and for any two schools under the script.\n\nAn exact example shows the consequence for a variance estimate. I take the 56 schools as the units and hosting as the exposure. Let the outcome under hosting be $1$ at two schools and $0$ at the others, and let every outcome without hosting be $0$. Let $p$ be the probability under the design that both schools host. The Horvitz--Thompson estimator of the total under hosting has variance $8p$. Suppose that the estimator of expression (7) is computed with a joint probability $q$ in place of $p$. A direct computation shows that its expectation under the design is $2+8p(q-1/4)/q$, which equals $8p$ for $q=p$ and $2-4p/27$ for $q=27/110$. For two schools of one block, the expectation is $160/81$ and the variance is $4/3$. For two schools of different blocks, the expectation is $53/27$ and the variance is $2$. With the joint probability of the script, the expected variance estimate is too large by $52/81$ in the first case and too small by $1/27$ in the second.", "problem": "The sentence of the article is correct. The script that assigned the schools forms 14 blocks and draws two hosts among the four schools of each block (lines 87--90). The public data show 14 blocks of four schools with two hosts in each. The defect is in the script that simulates the assignments of this experiment and computes the exposure probabilities from them. Its student lottery on line 57 passes the student blocks. Its school lottery on line 58 passes the schools as clusters and passes no block variable. The function \\texttt{genperms} of the R package \\texttt{ri} then treats the 56 schools as one block and draws 28 of them uniformly. Note that each school hosts the program with probability $1/2$ under both designs. The probability that two given schools both host is $27/110$ under the script. Under the blocked design it is $1/6$ for two schools of one block and $1/4$ for two schools of different blocks. The simulated draws then give joint exposure probabilities for students of different schools that are not those of the design. These joint probabilities enter the variance estimators of expressions (7), (10) and (11).", "what_survives": "The sentence of the article needs no change. Under the exposure mapping of Section 10, the exposure of a student depends on the lottery of the student's own school and on the student lottery within that school. This implies that the exact probabilities of the five exposures are the same under the two school designs. Point estimates computed with exact probabilities on a fixed sample therefore do not depend on the defect. Simulated probabilities from a finite number of draws, and a sample chosen from them, need not agree between the two designs. None of the eleven scripts released with the data computes Table 2. It is not established that the variance estimates of Table 2 used these draws, and no number of Table 2 is shown to be wrong on this account. In a separate reconstruction on the public data (ICPSR 37070 V2), an inferred screen requiring all five exposure probabilities to exceed $0.01$ and the legacy R sampler give 2,050 students and match all twelve printed point estimates to three decimals. On that sample, the Horvitz--Thompson standard errors for the $d_{011}$ and $d_{111}$ contrasts against $d_{000}$ are $0.0571$ and $0.0801$ with exact blocked probabilities, and $0.0569$ and $0.0785$ with the script's 10,000 unblocked draws. ICPSR reports that some released variables were changed; the original sample and variance code are unavailable, so these are conditional reconstruction values. Neither pair reproduces the printed $0.029$ and $0.020$. The results of Sections 3 to 8 and the simulation of Section 9 are unaffected. What changes is the script. After the correction, the exposure matrices must be generated again, and every calculation that resamples from them must be repeated."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N016", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of seven lines of text after Table 2. The passage reads: The outcome of interest, y i in the set 0, 1, is a binary indicator for whether the subject wore a program wristband. The effect estimates characterize, for eligible students, the average increase in the probability of wearing a wristband relative to the average in the no exposure condition. (The average for eligible students in the no exposure condition was essentially zero, at 0.000.) Red boxes mark the words 'whether the subject wore a program wristband' and, across a line break, the words 'the probability of wearing a wristband'.", "caption": "Section 10, first paragraph after Table 2, p. 1936 (PDF p. 25). The marked words define the outcome as whether the subject wore a program wristband, and the estimand as the probability of wearing a wristband. The passage gives no rule for a student whose report is absent.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N020.png"}, {"alt": "Dot plot with twelve positions on the horizontal axis: the exposures d001, d011, d101 and d111, each compared with no exposure, for the HT, Hajek and WLS estimators. The vertical axis is the point estimate, with ticks from 0 to 0.4. At each position a black horizontal bar shows the value printed in Table 2, and a blue circle near the bar shows the estimate with students with no recorded general-item answer kept with their released outcome 0, N = 2,050. An orange square, higher, shows the estimate without the 151 students with no recorded second-wave survey, N = 1,899, and a green triangle, higher still, shows the estimate without the 342 students with no recorded general-item answer, N = 1,708. For example, for the HT estimate of d101 the printed value is 0.305, and the circle, square and triangle round to 0.305, 0.329 and 0.366. The title reads: Table 2 is reproduced on the public data only when students with no recorded answer are kept and scored as not wearing the wristband.", "caption": "The twelve point estimates of Table 2, recalculated on the public data under three treatments of the students with no recorded answer to the general wristband item. The horizontal bars are the printed values. Among these three policies, only keeping these students with their released outcome zero ($N=2{,}050$) reproduces all twelve estimates after rounding. Without the 151 students with no recorded second-wave survey ($N=1{,}899$), or without all 342 students with no recorded general-item answer ($N=1{,}708$), none does. The values are in the second table in the Evidence section above.", "url": "https://pmaronow.github.io/llm-audit/works/P013/exhibits/P013-N020-outcome-coding.png"}], "formal_release": null, "formal_statements": [{"id": "P013-T23", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T23"}, {"id": "P013-T24", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T24"}, {"id": "P013-T27", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-T27"}], "headline": "In a reconstruction on the public data, all twelve point estimates of Table 2 are reproduced with 342 of 2,050 students who have no recorded general wristband answer kept and scored zero; the article should state whether it uses this recorded orange-wristband outcome and disclose its missing-answer rule.", "id": "P013-N020", "location": "Section 10, first paragraph after Table 2, p. 1936 (PDF p. 25); Table 2, all rows", "pdf_page": 25, "printed_page": "1936", "reaches": [{"effect": "cannot_determine", "effect_text": "could not be determined", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P037/", "where": null, "work_id": "P037"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P013-T23", "P013-T24", "P013-T27"], "text": {"as_printed": "``The outcome of interest, $y_i\\in\\{0,1\\}$, is a binary indicator for \\hl{whether the subject wore a program wristband}. The effect estimates characterize, for eligible students, the average increase in \\hl{the probability of wearing a wristband} relative to the average in the no exposure condition.''", "corrected": "``The outcome of interest, $y_i\\in\\{0,1\\}$, is a binary indicator for \\hl{whether the subject reported wearing an orange program wristband; a student with no recorded answer to the general wristband item, including a student with no recorded second-wave survey, is kept in the sample with $y_i=0$}. \\hl{Under the exposure-mapping assumptions for this recorded outcome,} the effect estimates characterize, for eligible students, the average increase in \\hl{the probability of such a recorded report} relative to the average in the no exposure condition.''\n\nIn the public-data reconstruction with the strict $0.01$ Monte Carlo screen described above, the students with no recorded general-item answer are 342 of 2,050, and 151 of them have no recorded second-wave survey. This wording is the correction if the authors' own outcome file codes these students as the public file does, which should be checked against that file. Under the outcome model above, retaining the wearing contrast requires $L(d_k)=L(d_{000})$ for each contrast; otherwise it requires information or assumptions identifying the unrecorded outcomes and a new analysis with its standard errors.", "evidence": "The evidence is a recalculation on the public data, a derivation and an exact example. The recalculation uses the released code's unblocked school-assignment generator, seed 123 and 10,000 draws, with the sampler that R used before version 3.6; the article describes school assignment as blocked. It keeps the eligible students whose five unsmoothed estimated exposure probabilities exceed $0.01$, and it fits the WLS rows with one slope on network degree. This reconstruction has $N=2{,}050$ students and, with the orange-wristband indicator as released, reproduces all twelve point estimates of Table 2 to three decimals. None of the eleven scripts released with the data computes Table 2, and the reconstruction is not the authors' calculation. Recall that the article limits the analysis to students with nonzero probabilities and does not state the cut-off of $0.01$. Whether the published analysis used it is not determined.\n\n\\begin{audittable}{lrrrr}{Public data, reconstructed sample: the students with no recorded answer to the general wristband item, by observed exposure, with their share of $N$ and the number among them with no recorded second-wave survey. All 342 have the value 0 on the orange-wristband indicator.}\nObserved exposure & $N$ & No answer & Share & No survey \\\\\n$d_{000}$ (no exposure) & 1,034 & 198 & 19.1\\% & 78 \\\\\n$d_{001}$ (school only) & 157 & 25 & 15.9\\% & 11 \\\\\n$d_{011}$ (indirect) & 348 & 51 & 14.7\\% & 26 \\\\\n$d_{101}$ (isolated direct) & 199 & 25 & 12.6\\% & 13 \\\\\n$d_{111}$ (direct and indirect) & 312 & 43 & 13.8\\% & 23 \\\\\nAll & 2,050 & 342 & 16.7\\% & 151 \\\\\n\\end{audittable}\n\n\\begin{audittable}{llrrrr}{Point estimates of Table 2 as printed and as recalculated on the public data under three treatments of the students with no recorded general-item answer: kept and scored 0 ($N=2{,}050$), without the 151 students with no recorded second-wave survey ($N=1{,}899$), and without all 342 ($N=1{,}708$).}\nEstimator & Contrast & Printed & Kept as 0 & Without 151 & Without 342 \\\\\nHT & $\\tau(d_{001},d_{000})$ & 0.057 & 0.057 & 0.062 & 0.069 \\\\\n & $\\tau(d_{011},d_{000})$ & 0.154 & 0.154 & 0.166 & 0.185 \\\\\n & $\\tau(d_{101},d_{000})$ & 0.305 & 0.305 & 0.329 & 0.366 \\\\\n & $\\tau(d_{111},d_{000})$ & 0.299 & 0.299 & 0.323 & 0.359 \\\\\nHajek & $\\tau(d_{001},d_{000})$ & 0.058 & 0.058 & 0.062 & 0.069 \\\\\n & $\\tau(d_{011},d_{000})$ & 0.154 & 0.154 & 0.167 & 0.181 \\\\\n & $\\tau(d_{101},d_{000})$ & 0.292 & 0.292 & 0.310 & 0.332 \\\\\n & $\\tau(d_{111},d_{000})$ & 0.307 & 0.307 & 0.330 & 0.355 \\\\\nWLS & $\\tau(d_{001},d_{000})$ & 0.056 & 0.056 & 0.060 & 0.068 \\\\\n & $\\tau(d_{011},d_{000})$ & 0.156 & 0.156 & 0.169 & 0.184 \\\\\n & $\\tau(d_{101},d_{000})$ & 0.295 & 0.295 & 0.312 & 0.336 \\\\\n & $\\tau(d_{111},d_{000})$ & 0.306 & 0.306 & 0.329 & 0.353 \\\\\n\\end{audittable}\n\nEvery student removed in the last two columns has outcome $0$. With the exposure probabilities held fixed, the numerators of the Horvitz--Thompson means are then unchanged, and only the divisor $N$ changes. Each HT contrast with these students kept equals $1708/2050=854/1025$ of its value without the 342 students, and $1899/2050$ of its value without the 151. These ratios describe a change of divisor. They do not measure a bias.\n\nTo isolate the effect of absent reports, let $R_i(d)$ indicate that usable wearing information for student $i$ is recorded under exposure $d$, let $y_i(d)$ be the wearing outcome, and suppose the analyzed outcome is $g_i(d)=R_i(d)\\,y_i(d)$. This model assumes that recorded reports equal wearing. Suppose also that $y_i$ and $R_i$ depend on the assignment only through the exposure and that the exposure probabilities used in the estimator are exact and positive. By Lemma 4.1 applied to $g$, I have\n\\[\n\\mathrm{E}\\bigl[\\widehat{\\mu_{\\mathrm{HT}}}(d)\\bigr]=\\frac{1}{N}\\sum_{i=1}^{N}g_i(d)=\\mu(d)-L(d),\\qquad L(d)=\\frac{1}{N}\\sum_{i=1}^{N}\\bigl(1-R_i(d)\\bigr)\\,y_i(d),\n\\]\nwhere the estimator is computed from the recorded outcomes and $\\mu(d)$ is the mean of the wearing outcomes $y_i(d)$. Subtracting the same identity for $d_{000}$, I obtain\n\\[\n\\mathrm{E}\\bigl[\\widehat{\\tau_{\\mathrm{HT}}}(d_k,d_{000})\\bigr]-\\tau(d_k,d_{000})=L(d_{000})-L(d_k).\n\\]\nThis identity implies that the estimator with absent reports counted as zero is unbiased for the effect on wearing if and only if $L(d_k)=L(d_{000})$. It suffices that under both exposures no student whose report would be absent would wear the wristband. Randomization, bounded outcomes and positive exposure probabilities do not imply this condition.\n\nThe exact example has two units with independent fair assignments and no interference. The first unit always reports, and its wearing outcome is $0$ without treatment and $1$ with it. The second unit never reports. Its wearing outcomes without and with treatment may be $(0,0)$, $(0,1)$ or $(1,0)$, and the recorded data are the same in the three cases under all four assignments. The effect on wearing is $1/2$, $1$ or $0$. The estimator with the absent report counted as zero has expectation $1/2$ in each case, and its bias is $0$, $-1/2$ or $1/2$. The estimator on the reporting unit alone has expectation $1$. The recorded data therefore do not determine the sign of the bias.\n\nTwo limits apply. In the whole public file, 4,108 of the 24,471 records have no recorded answer to the general item, and four of them have the value $1$ on the orange-wristband indicator. An absent answer to the general item does not prove that no information on colour was recorded. The released scripts do not show how the zeros arose: the cleaning script sets codes above 4 to missing and otherwise leaves the indicator as it is.", "problem": "The article describes the outcome as whether the student wore a program wristband, measured by self-report (p.~1934), and it gives no rule for a student whose report is absent. On the public data release of the field experiment (ICPSR study 37070, version 2), the variable that reproduces Table 2 is an indicator for a recorded orange-wristband report. It is $0$ or $1$ for every student. The general wristband item of the second-wave survey, ``I wore a Roots wristband this past week'', has no recorded answer for 342 of the 2,050 students of the reconstructed sample. Of these 342 students, 151 have no recorded second-wave survey, and 191 have a survey and no answer to the item. All 342 have the value $0$ on the indicator. Among the three policies compared here, the twelve point estimates of Table 2 are reproduced to three decimals only when these students are kept in the sample with outcome $0$. Without the 342 students the sample has 1,708 students, and without the 151 alone it has 1,899. In both cases none of the twelve estimates is reproduced. The article does not distinguish an observed negative answer from a zero recorded without an answer. For one student in six in this reconstruction, the zero stands where no answer to the general wristband item is recorded. Under the exposure assumptions for this recorded outcome, the reconstructed point estimates concern effects on a recorded report of wearing, with an absent report counted as zero. In the outcome model below, effects on wearing require a further condition that the article does not state: the students whose reports would be absent must contribute equally to the mean of wearing under the two exposures compared. The sizes that the text quotes, about 30 percentage points for direct exposure and 15 to 16 for indirect exposure, depend on the outcome definition. ICPSR notes that some released variables were changed. The authors' own outcome file is not public, and I could not check whether it codes these students as the public file does.", "what_survives": "All twelve point estimates of Table 2 are reproduced on the public data under this coding, and so is the mean of zero in the no-exposure condition (0 of 1,034 students). The broad order of the effects is the same under each of the three treatments and for each estimator: school-only exposure, then indirect exposure, then the two direct exposures. This outcome-coding issue does not affect the estimators or the results of Sections 3 to 8, and the simulation of Section 9 has no absent outcomes. What changes is the definition of the outcome, and with it the estimand of Table 2 and the meaning of the sizes quoted on pp.~1936--1937. No bias in the effects on wearing is identified. This finding concerns point estimates only; a different treatment of these students would need its own standard errors."}, "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N020", "work": {"arxiv_id": "1305.6156", "doi": "10.1214/16-aoas1005", "title": "Estimating Average Causal Effects Under General Interference, with Application to a Social Network Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P013/"}, "work_id": "P013"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Two lines of text introduce the individual average potential outcome under treatment z, followed by a display. A red box marks the fraction that defines Y-bar i of z. Its numerator is the sum over z in Supp(Z) of y i of z times the indicator that z i equals z, and its denominator is the sum over z in Supp(Z) of that indicator. Two lines below the display begin 'Note that this estimand is regime specific'.", "caption": "Section 2, first display, p. 131 (PDF p. 4). The marked fraction defines $\\overline{Y}_i(z)$ as an average over the assignment vectors in $\\operatorname{Supp}(\\mathbf{Z})$. No probability appears in either sum.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E001.png"}, {"alt": "Three lines of text say that under local randomization the difference-in-means estimator is unbiased for the average direct effect for subjects near the threshold. A red box marks the display 'E of tau-hat equals one over N0 times, in square brackets, the sum over i with Xi in W0 of Y-bar i of 1 minus the sum over i with Xi in W0 of Y-bar i of 0'. A second red box marks the sentence 'where the result trivially follows from linearity of expectations.' Three lines of the next paragraph, on inference, follow.", "caption": "Section 2, p. 131 (PDF p. 4). The marked display equates $\\mathrm{E}[\\hat\\tau]$ with the target built from the printed $\\overline{Y}_i(z)$, and the marked sentence attributes the identity to linearity of expectations.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E001-b.png"}, {"alt": "Bar chart with three groups, one for each assignment vector with z1 equal to 0. The groups are (0, 0, 1, 0), where the outcome of subject 1 is 0; (0, 0, 1, 1), where it is 1; and (0, 1, 0, 0), where it is 0. In each group a hatched blue bar shows the weight in the printed average, which is 1/3 in all three groups, and an orange bar shows the probability given Z1 equal to 0, which is 1/4, 1/4 and 1/2. The legend says that the printed weights give Y-bar 1 of 0 equal to 1/3 and that the probabilities give the conditional expectation 1/4. The title reads 'The printed average weights assignment vectors equally; their probabilities under the design differ'.", "caption": "The first example, subject 1 under control. The printed $\\overline{Y}_1(0)$ assigns weight $1/3$ to each of the three assignment vectors with $z_1=0$ and equals $1/3$. Given $Z_1=0$, these vectors have probabilities $1/4$, $1/4$ and $1/2$ under the design, and the expected outcome is $1/4$. All values are exact.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E001-weights.png"}], "formal_release": null, "formal_statements": [{"id": "P014-T1", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T1"}, {"id": "P014-T2", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T2"}, {"id": "P014-T3", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T3"}, {"id": "P014-T4", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T4"}, {"id": "P014-T7", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T7"}, {"id": "P014-T9", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T9"}, {"id": "P014-T10", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T10"}], "headline": "The difference-in-means estimator is unbiased for the probability-weighted average direct effect, in which $\\overline{Y}_i(z)=\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=z]$, and not in general for the target printed on p. 131, which weights the assignment vectors in $\\operatorname{Supp}(\\mathbf{Z})$ equally.", "id": "P014-E001", "location": "Section 2, p. 131 (PDF p. 4): the first display, which defines $\\overline{Y}_i(z)$; the display that defines $\\tau$; and the display for $\\mathrm{E}[\\hat\\tau]$ with the sentence “where the result trivially follows from linearity of expectations”. The corresponding claim is in the Introduction, p. 129 (PDF p. 2)", "pdf_page": 4, "printed_page": "131", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P014-T1", "P014-T10", "P014-T2", "P014-T3", "P014-T4", "P014-T7", "P014-T9"], "text": {"as_printed": "Denote the individual average potential outcome under treatment $z$ for all subjects $i:X_i\\in W_0$,\n\\[\n\\overline{Y}_i(z)=\\frac{\\sum_{\\mathbf{z}\\in\\operatorname{Supp}(\\mathbf{Z})}\\hl{y_i(\\mathbf{z})\\mathrm{I}(\\mathbf{z}_i=z)}}{\\sum_{\\mathbf{z}\\in\\operatorname{Supp}(\\mathbf{Z})}\\hl{\\mathrm{I}(\\mathbf{z}_i=z)}}.\n\\]\n\n\\ldots\n\nThen our primary target is the average direct effect (Hudgens and Halloran, 2008) of treatment for subjects who are close to the threshold:\n\\[\n\\tau=N_0^{-1}\\sum_{i:X_i\\in W_0}\\overline{Y}_i(1)-\\overline{Y}_i(0).\n\\]\n\n\\ldots\n\nFormally, the difference-in-means estimator for subjects local to the threshold,\n\\[\n\\hat\\tau=\\frac{\\sum_{i:X_i\\in W_0}Y_iZ_i}{\\sum_{i:X_i\\in W_0}Z_i}-\\frac{\\sum_{i:X_i\\in W_0}Y_i(1-Z_i)}{\\sum_{i:X_i\\in W_0}(1-Z_i)}.\n\\]\nThen, under the assumption of local randomization, the difference-in-means estimator as applied to subjects local to the threshold is unbiased for the average direct effect for subjects near the threshold:\n\\[\n\\mathrm{E}[\\hat\\tau]=\\frac{1}{N_0}\\Biggl[\\sum_{i:X_i\\in W_0}\\hl{\\overline{Y}_i(1)}-\\sum_{i:X_i\\in W_0}\\hl{\\overline{Y}_i(0)}\\Biggr],\n\\]\nwhere the result \\hl{trivially} follows from linearity of expectations.", "corrected": "Let $S=\\{i:X_i\\in W_0\\}$ be the fixed set of subjects in the window and $n=N_0$. Assume local randomization: the $n$ distinct, fixed values of the running variable in the window, of which $m$ are nonnegative with $0<m<n$, are arranged among the subjects in $S$ uniformly at random, so that $\\sum_{i\\in S}Z_i=m$ with probability one and $\\Pr(Z_i=1)=m/n$ for every $i\\in S$. The allocation outside the window is not restricted. The potential outcomes $y_i(\\mathbf{z})$ are fixed and may depend on the whole assignment vector. Denote the individual average potential outcome under treatment $z$ for all subjects $i:X_i\\in W_0$,\n\\[\n\\overline{Y}_i(z)=\\frac{\\sum_{\\mathbf{z}\\in\\operatorname{Supp}(\\mathbf{Z})}y_i(\\mathbf{z})\\mathrm{I}(\\mathbf{z}_i=z)\\,\\hl{\\Pr(\\mathbf{Z}=\\mathbf{z})}}{\\sum_{\\mathbf{z}\\in\\operatorname{Supp}(\\mathbf{Z})}\\mathrm{I}(\\mathbf{z}_i=z)\\,\\hl{\\Pr(\\mathbf{Z}=\\mathbf{z})}}\\hl{=\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=z]}.\n\\]\nThen our primary target is the average direct effect (Hudgens and Halloran, 2008) of treatment for subjects who are close to the threshold:\n\\[\n\\tau=N_0^{-1}\\sum_{i:X_i\\in W_0}\\bigl[\\overline{Y}_i(1)-\\overline{Y}_i(0)\\bigr].\n\\]\nThe difference-in-means estimator $\\hat\\tau$ is as printed. Then, under the assumption of local randomization, the difference-in-means estimator as applied to subjects local to the threshold is unbiased for the average direct effect for subjects near the threshold:\n\\[\n\\mathrm{E}[\\hat\\tau]=\\frac{1}{N_0}\\Biggl[\\sum_{i:X_i\\in W_0}\\overline{Y}_i(1)-\\sum_{i:X_i\\in W_0}\\overline{Y}_i(0)\\Biggr],\n\\]\nwhere the result follows from \\hl{the fixed numbers of treated and control subjects in the window, their common probability of treatment, and} linearity of expectations. The brackets in the display for $\\tau$ are those of \\fid{P014-N004}. The treated-minus-control contrast uses the reverse of Hudgens and Halloran's sign convention (\\fid{P014-N002}).\n\nAlternatively, the definition of $\\overline{Y}_i(z)$ printed on p.~131 can be kept under the additional assumption that $\\mathbf{Z}$ is uniformly distributed on $\\operatorname{Supp}(\\mathbf{Z})$, which holds, for instance, when the assignments of the subjects outside the window are fixed. The two definitions then agree.", "evidence": "The evidence is two exact counterexamples to the printed identity, and a derivation of the corrected one.\n\n\\emph{The first example.} Let $N=4$, $b=2$ and $W_0=[-2,2]$. Let $(X_1,X_2,X_3)$ be a uniformly random arrangement of the values $-2$, $-1$ and $1$, and set $X_4=3$ if $(X_1,X_2,X_3)=(-2,-1,1)$ and $X_4=-3$ otherwise. The vector $\\mathbf{X}$ has six support points, each of probability $1/6=1/N_0!$. Subjects 1, 2 and 3 are always in the window and subject 4 never is, the values in the window are distinct and the same in every case, and exactly one subject in the window is treated. Every condition printed on p.~130 holds, with the no-ties condition read for distinct subjects (\\fid{P014-E003}), and in addition all six arrangements are equiprobable. Each subject in the window is treated with probability $1/3$. For all sixteen assignment vectors $\\mathbf{z}$, set $y_1(\\mathbf{z})=y_2(\\mathbf{z})=y_3(\\mathbf{z})=z_4$ and $y_4(\\mathbf{z})=0$.\n\\begin{audittable}{crrrr}{The first example: the four distinct assignment vectors $\\mathbf{z}=(z_1,z_2,z_3,z_4)$. The second column counts the arrangements of the running variable that produce $\\mathbf{z}$; each arrangement has probability $1/6$. The fourth column is the outcome of subjects 1, 2 and 3.}\n$\\mathbf{z}$ & Arrangements & $\\Pr(\\mathbf{Z}=\\mathbf{z})$ & $y_i(\\mathbf{z})=z_4$ & $\\hat\\tau$ \\\\\n$(0,0,1,0)$ & 1 & $1/6$ & 0 & 0 \\\\\n$(0,0,1,1)$ & 1 & $1/6$ & 1 & 0 \\\\\n$(0,1,0,0)$ & 2 & $1/3$ & 0 & 0 \\\\\n$(1,0,0,0)$ & 2 & $1/3$ & 0 & 0 \\\\\n\\end{audittable}\nThe three subjects in the window always have the same outcome, so that $\\hat\\tau=0$ for every assignment and $\\mathrm{E}[\\hat\\tau]=0$. The printed $\\overline{Y}_i(z)$ counts each row of the table once. For subject 1 under control, the rows with $z_1=0$ are the first three, with outcomes $0$, $1$ and $0$, and the printed average is $1/3$. Given $Z_1=0$, these rows have probabilities $1/4$, $1/4$ and $1/2$, and the conditional expectation is $1/4$.\n\\begin{audittable}{crrrr}{The first example: the individual average potential outcomes under the printed definition and under the probability-weighted definition.}\nSubject $i$ & Printed $\\overline{Y}_i(1)$ & Printed $\\overline{Y}_i(0)$ & $\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=1]$ & $\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=0]$ \\\\\n1 & 0 & $1/3$ & 0 & $1/4$ \\\\\n2 & 0 & $1/3$ & 0 & $1/4$ \\\\\n3 & $1/2$ & 0 & $1/2$ & 0 \\\\\n\\end{audittable}\nFrom the second and third columns, I obtain the printed target, and from the last two the probability-weighted target:\n\\[\n\\tau=\\frac13\\Bigl(-\\frac13-\\frac13+\\frac12\\Bigr)=-\\frac{1}{18},\n\\qquad\n\\frac13\\Bigl(-\\frac14-\\frac14+\\frac12\\Bigr)=0=\\mathrm{E}[\\hat\\tau].\n\\]\nThe printed identity fails by $1/18$.\n\n\\emph{The second example.} Let $N=4$, $b=3$ and $W_0=[-3,3]$. Let $(X_1,X_2,X_3)$ be a uniformly random arrangement of $-1$, $1$ and $2$, so that two subjects in the window are treated, and set $X_4=4$ if $(X_1,X_2,X_3)=(-1,1,2)$ and $X_4=-4$ otherwise. Set $y_2(\\mathbf{z})=12z_2z_4$ and all other potential outcomes equal to zero. The same conditions hold. The assignment vectors $(0,1,1,0)$ and $(0,1,1,1)$ have probability $1/6$ each, and $(1,0,1,0)$ and $(1,1,0,0)$ have probability $1/3$ each. The estimator equals $6$ at $(0,1,1,1)$ and $0$ elsewhere, so that $\\mathrm{E}[\\hat\\tau]=1$. The printed $\\overline{Y}_2(1)$ is $(0+12+0)/3=4$, and I find $\\tau=4/3$. The conditional expectation $\\mathrm{E}[y_2(\\mathbf{Z})\\mid Z_2=1]$ is $3$, and the probability-weighted target is $1$.\n\\begin{audittable}{lrrr}{The two exact examples. In each, the six arrangements of the running variable in the window are equiprobable and the conditions printed on p.~130 hold, with the no-ties condition read for distinct subjects.}\nExample & Printed $\\tau$ & $\\mathrm{E}[\\hat\\tau]$ & Probability-weighted target \\\\\nFirst & $-1/18$ & 0 & 0 \\\\\nSecond & $4/3$ & 1 & 1 \\\\\n\\end{audittable}\n\n\\emph{The corrected identity.} Let $S$ be the set of subjects in the window and $n=N_0$. Suppose that $\\sum_{i\\in S}Z_i=m$ with probability one, where $0<m<n$, and that $\\Pr(Z_i=1)=m/n$ for every $i\\in S$. Both hold when the $n$ distinct values in the window, of which $m$ are nonnegative, are arranged uniformly at random: a subject receives each value in $(n-1)!$ of the $n!$ arrangements. I write $p(\\mathbf{z})=\\Pr(\\mathbf{Z}=\\mathbf{z})$ and\n\\[\n\\overline{Y}_i(a)=\\frac{\\sum_{\\mathbf{z}}p(\\mathbf{z})\\,y_i(\\mathbf{z})\\,\\mathrm{I}(z_i=a)}{\\sum_{\\mathbf{z}}p(\\mathbf{z})\\,\\mathrm{I}(z_i=a)}=\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=a],\\qquad a\\in\\{0,1\\}.\n\\]\nSince the numbers of treated and control subjects in the window are fixed, I may write\n\\[\n\\hat\\tau=\\frac1m\\sum_{i\\in S}y_i(\\mathbf{Z})Z_i-\\frac{1}{n-m}\\sum_{i\\in S}y_i(\\mathbf{Z})(1-Z_i).\n\\]\nFor each $i\\in S$, by the definition of $\\overline{Y}_i(a)$ and the identity $\\Pr(Z_i=1)=m/n$, I have\n\\[\n\\mathrm{E}[y_i(\\mathbf{Z})Z_i]=\\sum_{\\mathbf{z}}p(\\mathbf{z})\\,y_i(\\mathbf{z})\\,\\mathrm{I}(z_i=1)=\\frac{m}{n}\\,\\overline{Y}_i(1),\n\\qquad\n\\mathrm{E}[y_i(\\mathbf{Z})(1-Z_i)]=\\frac{n-m}{n}\\,\\overline{Y}_i(0).\n\\]\nSubstituting these into the expectation of the previous display, I obtain\n\\[\n\\mathrm{E}[\\hat\\tau]=\\frac1n\\sum_{i\\in S}\\bigl\\{\\overline{Y}_i(1)-\\overline{Y}_i(0)\\bigr\\}.\n\\]\nNote that this derivation uses no assumption on the allocation outside the window. In both examples the difference between $\\mathrm{E}[\\hat\\tau]$ and the probability-weighted target vanishes identically as a linear form in the 48 potential outcomes of the three subjects in the window, and the difference between $\\mathrm{E}[\\hat\\tau]$ and the printed target does not.\n\n\\emph{When the printed target is correct.} If $\\mathbf{Z}$ is uniform on $\\operatorname{Supp}(\\mathbf{Z})$, then the two definitions of $\\overline{Y}_i(z)$ agree. This is the case when the assignments outside the window are fixed, since each assignment vector is then produced by exactly $m!(n-m)!$ arrangements. With $X_4$ fixed in either example, the printed identity holds for all potential outcomes. The two definitions also agree when there is no interference, since $y_i(\\mathbf{z})$ then depends on $z_i$ alone.", "problem": "The first display defines $\\overline{Y}_i(z)$ as an average of $y_i(\\mathbf{z})$ over the distinct assignment vectors in $\\operatorname{Supp}(\\mathbf{Z})$ whose $i$th component is $z$. Each such vector counts once, whatever its probability. The expectation of the difference-in-means estimator weights each assignment vector by its probability. Under local randomization, by linearity of expectation, I have\n\\[\n\\mathrm{E}[\\hat\\tau]=\\frac{1}{N_0}\\sum_{i:X_i\\in W_0}\\bigl\\{\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=1]-\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=0]\\bigr\\}.\n\\]\nThis equals the printed target for all potential outcomes only if, for each subject $i$ in the window and each $z$, the conditional distribution of $\\mathbf{Z}$ given $Z_i=z$ is uniform on its support. Local randomization does not imply this. Under it the arrangements of the running variable in the window are equiprobable, so that the subjects in the window have equal probabilities of treatment. However, the allocation outside the window is ``left completely unspecified'', and the number of arrangements that produce a particular assignment vector of the whole population can vary from one vector to another. The distinct assignment vectors can then have unequal probabilities, so the identity $\\mathrm{E}[\\hat\\tau]=\\tau$ need not hold for the printed $\\tau$ and cannot be justified by linearity of expectations alone.\n\nThe prose around the displays describes the probability-weighted quantity: the target is ``the average outcome we would expect to see in treated subjects (across all subjects within the window and across all randomizations)'' minus the same for control subjects. The display can disagree with the prose when assignment vectors have unequal probabilities, and the text does not settle whether the probabilities were omitted from the display or an assumption of uniform assignment was intended. To keep the allocation outside the window unrestricted, the correction inserts $\\Pr(\\mathbf{Z}=\\mathbf{z})$ in both sums of the first display. The estimator is unchanged and the estimand changes.", "what_survives": "The difference-in-means estimator, the potential-outcome model with arbitrary interference and the local randomization design are unchanged. The estimator is exactly unbiased for the probability-weighted target with no restriction on the allocation outside the window. This implies that the statement of the Introduction, that it is unbiased for a regime-specific causal effect, holds for that target. The printed target and identity hold as printed when $\\mathbf{Z}$ is uniform on its support, and when there is no interference. The definition of the estimand changes; in the first example its value changes from $-1/18$ to $0$. The paper reports no simulations, application or numerical results that depend on the definition. The paragraph on inference on pp.~131--132 refers to the same target, and its sufficient conditions were not examined for this correction."}, "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-E001", "work": {"arxiv_id": null, "doi": "10.1353/obs.2017.0001", "title": "The Regression Discontinuity Design under Interference: A Local Randomization-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P014/"}, "work_id": "P014"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Nine lines of text from the final paragraph of Section 2. A red box marks the clause 'Pr(X = x) = 1/N0!, for all x in Supp(X)', with X and x in bold. The line continues 'where N0 is the sum of the indicators that Xi is in W0: implying that all permutations of the running variable values (and therefore treatment values) within the window W0 are equiprobable'.", "caption": "Section 2, final paragraph, p. 130 (PDF p. 3). The marked clause assigns probability $1/N_0!$ to each support point of the full vector $\\mathbf{X}$. The words after the colon draw the conclusion that the permutations within the window are equiprobable.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P014-T2", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T2"}, {"id": "P014-T3", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T3"}, {"id": "P014-T5", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T5"}, {"id": "P014-T6", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T6"}, {"id": "P014-T7", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T7"}], "headline": "The probability condition in the local randomization assumption should be placed on the running variables of the subjects in the window, $\\Pr(\\mathbf{X}_S=a)=1/N_0!$ for every arrangement $a$ of their values, and not on the full vector $\\mathbf{X}$.", "id": "P014-E002", "location": "Section 2, final paragraph, p. 130 (PDF p. 3), sentence beginning “Our local randomization assumption requires that”; related sentence on p. 131 (PDF p. 4), beginning “While the treatment allocation scheme”", "pdf_page": 3, "printed_page": "130", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P014-T2", "P014-T3", "P014-T5", "P014-T6", "P014-T7"], "text": {"as_printed": "Our local randomization assumption requires that $F_{\\mathbf{x},W_0}(t)=F_0(t)$, $\\forall\\mathbf{x}\\in\\operatorname{Supp}(\\mathbf{X})$ and $\\forall t$; $\\Pr(X_i\\in W_0)\\in\\{0,1\\}$, $\\forall i\\in\\{1,\\ldots,N\\}$, $\\Pr(\\hl{\\mathbf{X}=\\mathbf{x}})=\\frac{1}{N_0!}$, $\\forall\\hl{\\mathbf{x}\\in\\operatorname{Supp}(\\mathbf{X})}$, where $N_0=\\sum_{i=1}^{N}\\mathrm{I}(X_i\\in W_0)$: implying that all permutations of the running variable values (and therefore treatment values) within the window $W_0$ are equiprobable. Further assume that $0<\\sum_{i:X_i\\in W_0}Z_i<N_0$, so that at least one subject within the window is treated, and at least one subject within the window is in the control group.", "corrected": "Let $S=\\{i:\\Pr(X_i\\in W_0)=1\\}$ be the set of subjects in the window, and let $n=N_0$ be its size. Our local randomization assumption requires that $\\Pr(X_i\\in W_0)\\in\\{0,1\\}$, $\\forall i\\in\\{1,\\ldots,N\\}$; that $F_{\\mathbf{x},W_0}(t)=F_0(t)$, $\\forall\\mathbf{x}\\in\\operatorname{Supp}(\\mathbf{X})$ and $\\forall t$, where $F_0$ is the empirical distribution function of $n$ distinct values $v_1,\\ldots,v_n$; and that\n\\[\n\\Pr(\\hl{\\mathbf{X}_S=a})=\\frac{1}{n!}\\quad\\text{for every arrangement } \\hl{a} \\text{ of } v_1,\\ldots,v_n \\text{ among the subjects in } S,\n\\]\nwhere $\\mathbf{X}_S=(X_i)_{i\\in S}$. That is, all permutations of the running variable values (and therefore treatment values) within the window $W_0$ are equiprobable. The distribution of the running variables outside the window, given $\\mathbf{X}_S$, is restricted only by $\\Pr(X_i\\in W_0)=0$ for $i\\notin S$, and it need not be independent of $\\mathbf{X}_S$. Further assume that the number $m$ of nonnegative values among $v_1,\\ldots,v_n$ satisfies $0<m<n$, so that at least one subject within the window is treated, and at least one subject within the window is in the control group. Then $\\sum_{i\\in S}Z_i=m$ with probability one, and $\\Pr(Z_i=1)=m/n$ for every $i\\in S$.", "evidence": "The evidence is an exact counterexample to the implication, together with a counting argument for the corrected condition.\n\n\\emph{The counterexample.} Let $N=4$, $b=2$ and $W_0=[-2,2]$, and assign probability $1/6$ to each of the six vectors\n\\[\n(1,-1,-2,3),\\quad (-1,1,-2,3),\\quad (-1,-2,1,3),\\quad (-1,-2,1,4),\\quad (-1,-2,1,5),\\quad (-1,-2,1,6).\n\\]\nSubjects 1, 2 and 3 are in the window with probability one and subject 4 is never in it, so that $N_0=3$. In each vector the values in the window are $-2$, $-1$ and $1$. They are distinct, and the empirical distribution function $F_{\\mathbf{x},W_0}$ is the same for all six. Each support point has probability $1/6=1/N_0!$, and exactly one subject in the window is treated. Every printed clause holds. However, the arrangement $(-1,-2,1)$ has probability $2/3$, the arrangements $(1,-1,-2)$ and $(-1,1,-2)$ have probability $1/6$ each, and the other three have probability zero. Subjects 1, 2 and 3 are treated with probabilities $1/6$, $1/6$ and $2/3$. Let $y_1(\\mathbf{z})=z_1$, and let all other potential outcomes be zero, so that there is no interference. Then $\\tau=1/3$, while $\\hat\\tau=Z_1$ and $\\mathrm{E}[\\hat\\tau]=1/6$.\n\n\\emph{The counting argument.} Let $S$ be the set of subjects in the window, write $n=N_0$, and let $\\mathbf{X}_S=(X_i)_{i\\in S}$. For an arrangement $a$, let $k(a)$ be the number of support points $\\mathbf{x}$ of $\\mathbf{X}$ such that $\\mathbf{x}_S=a$. Under the printed clauses, I have $\\Pr(\\mathbf{X}_S=a)=k(a)/n!$, and nothing determines $k(a)$. If in addition $\\Pr(\\mathbf{X}_S=a)=1/n!$ for every $a$, then $k(a)=1$ for every $a$, so that the values outside the window are a function of $\\mathbf{X}_S$. For instance, let the arrangement be uniform over the six arrangements of $-2$, $-1$ and $1$, and let $X_4$ be independent of it, equal to $-3$ with probability $1/3$ and to $3$ with probability $2/3$. Then $\\mathbf{X}$ has twelve support points, with probabilities $1/18$ and $1/9$, and none has probability $1/6$. Under the corrected condition, let $m$ be the number of nonnegative values in the window. A subject $i\\in S$ receives a nonnegative value in $m(n-1)!$ of the $n!$ arrangements, which implies that $\\Pr(Z_i=1)=m/n$.", "problem": "Under the marked clause, every support point of the full vector $\\mathbf{X}=(X_1,\\ldots,X_N)$ has probability $1/N_0!$. The sentence then says that its clauses imply that all permutations of the running variable values within the window are equiprobable. They do not. I define an arrangement to be an assignment of the $N_0$ values in the window to the $N_0$ subjects in the window. Two support points of $\\mathbf{X}$ can have the same arrangement and differ outside the window. The clauses fix the number of support points at $N_0!$, but they do not fix how many support points have a given arrangement. Some arrangements can have probability zero and others probability above $1/N_0!$, and the subjects in the window can then have unequal probabilities of treatment.\n\nThe clause is also too strong once the conclusion drawn from it is granted. If the $N_0!$ arrangements are equiprobable and every support point of $\\mathbf{X}$ has probability $1/N_0!$, then the running variables outside the window are a function of the arrangement. An allocation outside the window that remains random given the arrangement is excluded, although p.~131 says that this allocation is ``left completely unspecified''. The words after the colon state the intended assumption, which is a uniformly random arrangement, and the correction states that assumption directly.", "what_survives": "The potential-outcome model, the threshold rule that sets $Z_i=1$ when $X_i\\ge 0$, the fixed set of subjects in the window and the difference-in-means estimator are unchanged. The corrected condition is the assumption that the prose of the sentence already states, and it changes neither the estimator nor the intended estimand. Under it every subject in the window is treated with probability $m/n$, and without interference the unbiasedness argument on p.~131 holds as printed. This correction does not repair the definition of the target under interference, which is the subject of \\fid{P014-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-E002", "work": {"arxiv_id": null, "doi": "10.1353/obs.2017.0001", "title": "The Regression Discontinuity Design under Interference: A Local Randomization-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P014/"}, "work_id": "P014"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Three lines of text. A red box marks the formula 'Pr(Xi = Xj) = 0, for all i, j : Xi, Xj in W0'. It is followed by the words 'so that no two observed age values within the window are exactly identical'.", "caption": "Section 2, final paragraph, p. 130 (PDF p. 3). The marked condition is quantified over all $i,j$ such that $X_i,X_j\\in W_0$, with no restriction to $i\\neq j$.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P014-T1", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T1"}, {"id": "P014-T4", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T4"}, {"id": "P014-T5", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T5"}, {"id": "P014-T7", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T7"}, {"id": "P014-T9", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T9"}], "headline": "The no-ties condition should be restricted to distinct subjects: $\\Pr(X_i=X_j)=0$ for all $i\\neq j$ such that $X_i,X_j\\in W_0$.", "id": "P014-E003", "location": "Section 2, final paragraph, p. 130 (PDF p. 3), sentence beginning “To simplify our definition of local randomization”", "pdf_page": 3, "printed_page": "130", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P014-T1", "P014-T4", "P014-T5", "P014-T7", "P014-T9"], "text": {"as_printed": "To simplify our definition of local randomization, without loss of generality, assume that $\\Pr(X_i=X_j)=0,\\ \\hl{\\forall i,j}:X_i,X_j\\in W_0$, so that no two observed age values within the window are exactly identical.", "corrected": "To simplify our definition of local randomization, without loss of generality, assume that $\\Pr(X_i=X_j)=0,\\ \\forall i,j:\\hl{i\\neq j},\\ X_i,X_j\\in W_0$, so that no two observed age values within the window are exactly identical.", "evidence": "The evidence is a check of the page and a one-line derivation. No restriction $i\\neq j$ is printed. The window contains at least one subject, since the same paragraph assumes $0<\\sum_{i:X_i\\in W_0}Z_i<N_0$. For such a subject $i$, the printed condition with $j=i$ requires $\\Pr(X_i=X_i)=0$, while the event $\\{X_i=X_i\\}$ has probability one.", "problem": "The quantifier ranges over all pairs $i,j$ of subjects in the window, including $j=i$. Since $\\Pr(X_i=X_i)=1$, the condition cannot hold as printed. The words that follow it, ``no two observed age values'', state the intended condition, which concerns distinct subjects.", "what_survives": "Adding $i\\neq j$ expresses the intended no-ties condition without changing the intended local randomization design. I did not examine whether ``without loss of generality'' is justified for every model of the running variable."}, "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-E003", "work": {"arxiv_id": null, "doi": "10.1353/obs.2017.0001", "title": "The Regression Discontinuity Design under Interference: A Local Randomization-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P014/"}, "work_id": "P014"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Two entries of the reference list. In the first, Cattaneo, Frandsen and Titiunik (2015), a red box marks the closing quotation mark after the word Senate at the end of the title. In the second, Cattaneo, Titiunik and Vasquez-Bare (2016), a red box marks the surname Vasquez-Bare, spelled with an s.", "caption": "References, p. 132 (PDF p. 5). The marks show the closing quotation mark after the title of the Cattaneo, Frandsen and Titiunik entry, which has no opening mark, and the spelling “Vasquez-Bare”.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E004.png"}, {"alt": "Four entries of the reference list: Lee (2008), Liu and Hudgens (2014), Neyman (1923) and Rosenbaum (2007). A red box marks the closing quotation mark after 'elections.' in the Lee entry. Two red boxes mark the third and fourth lines of the Neyman entry, which read 'erzy Splawa, D. M. Dabrowska, and T. P. Speed [1923.] 1990. On the application of probability theory to agricultural experiments: Essay on principles, section 9.' A red box marks the words 'American American' in the journal name of the Rosenbaum entry.", "caption": "References, p. 133 (PDF p. 6). The marks show the unmatched closing quotation mark in the Lee entry; the two lines of the Neyman entry that hold the fragment “erzy Splawa”, the date “[1923.] 1990” and the repeated title; and the repeated word “American” in the Rosenbaum entry.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E004-b.png"}, {"alt": "The three lines of the entry Thistlethwaite and Campbell (1960). A red box marks the closing quotation mark after the word experiment at the end of the title.", "caption": "References, p. 133 (PDF p. 6). The mark shows the unmatched closing quotation mark in the Thistlethwaite and Campbell entry.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-E004-c.png"}], "formal_release": null, "formal_statements": [], "headline": "Six entries of the reference list contain misprints: a misspelled surname, a repeated word, three closing quotation marks without an opening mark, and a Neyman entry that prints its title twice and sets a fragment of the author’s name among the translators.", "id": "P014-E004", "location": "References, pp. 132–133 (PDF pp. 5–6): the entries Cattaneo, Frandsen and Titiunik (2015) and Cattaneo, Titiunik and Vasquez-Bare (2016) on p. 132; Lee (2008), Neyman (1923), Rosenbaum (2007), and Thistlethwaite and Campbell (1960) on p. 133", "pdf_page": 5, "printed_page": "132–133", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item Cattaneo, M.D., B.R. Frandsen, and R. Titiunik (2015). Randomization inference in the regression discontinuity design: an application to party advantages in the U.S. Senate.\\hl{''} \\emph{Journal of Causal Inference}. 3(1): 1 --24.\n\\item Cattaneo, M.D., R. Titiunik, and G. \\hl{Vasquez-Bare} (2016). Inference in regression discontinuity designs under local randomization. \\emph{The Stata Journal}. 16(2): 331 -- 367.\n\\item Lee, D. (2008) Randomized experiments from non-random selection in U.S. House elections.\\hl{''} \\emph{Journal of Econometrics}. 142: 675 -- 697.\n\\item Neyman, J.S. (1923). On the application of probability theory to agricultural experiments: Essay on principles, section 9. Translated and reprinted by \\hl{erzy Splawa,} D. M. Dabrowska, and T. P. Speed [1923.] 1990. \\hl{On the application of probability theory to agricultural experiments: Essay on principles, section 9.} \\emph{Statistical Science} \\textbf{5}: 465--480.\n\\item Rosenbaum, P.R (2007). Interference between units in randomized experiments. \\emph{Journal of the American \\hl{American} Statistical Association}. 102(477): 191 -- 200.\n\\item Thistlethwaite, D. and D. Campbell. (1960). Regression-discontinuity analysis: an alternative to the ex post facto experiment.\\hl{''} \\emph{Journal of Educational Psychology}. 51: 309 -- 317.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Cattaneo, M.D., B.R. Frandsen, and R. Titiunik (2015). Randomization inference in the regression discontinuity design: an application to party advantages in the U.S. Senate. \\emph{Journal of Causal Inference}. 3(1): 1 --24.\n\\item Cattaneo, M.D., R. Titiunik, and G. \\hl{Vazquez-Bare} (2016). Inference in regression discontinuity designs under local randomization. \\emph{The Stata Journal}. 16(2): 331 -- 367.\n\\item Lee, D. (2008) Randomized experiments from non-random selection in U.S. House elections. \\emph{Journal of Econometrics}. 142: 675 -- 697.\n\\item \\hl{Splawa-Neyman, J. (1923 [1990]).} On the application of probability theory to agricultural experiments. Essay on principles. Section 9. \\hl{Translated and edited by D. M. Dabrowska and T. P. Speed.} \\emph{Statistical Science} 5(4): 465--472 (with discussion, 465--480).\n\\item Rosenbaum, P.R (2007). Interference between units in randomized experiments. \\emph{Journal of the American Statistical Association}. 102(477): 191 -- 200.\n\\item Thistlethwaite, D. and D. Campbell. (1960). Regression-discontinuity analysis: an alternative to the ex post facto experiment. \\emph{Journal of Educational Psychology}. 51: 309 -- 317.\n\\end{itemize}", "evidence": "The evidence is a check of the pages against public sources. The article page of \\emph{The Stata Journal} names the third author as Gonzalo Vazquez-Bare. The title page of the 1990 translation in \\emph{Statistical Science} names the author as Jerzy Splawa-Neyman, carries the credit ``Translated and edited by D. M. Dabrowska and T. P. Speed'', and has the masthead ``1990, Vol. 5, No. 4, 465-480''. The contents of that issue list the comment by D. B. Rubin separately at pp.~472--480.", "problem": "The surname in the 2016 entry is spelled ``Vasquez-Bare'', while the text on pp.~129 and 132 has ``Vazquez-Bare'', which is the spelling of the journal's byline. The Rosenbaum entry prints ``American'' twice. Three titles end with a closing quotation mark that has no opening mark. The Neyman entry runs two citations of one item together. Its title is printed twice, the date appears as ``[1923.] 1990'', and ``erzy Splawa'', a fragment of the author's name Jerzy Splawa-Neyman, stands among the translators. The page range 465--480 of that entry is not an error: it is the range on the masthead of the translation, which covers the translation (465--472) and the discussion that follows it.", "what_survives": "Every cited work remains identifiable, and no mathematical content or result of the paper is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-E004", "work": {"arxiv_id": null, "doi": "10.1353/obs.2017.0001", "title": "The Regression Discontinuity Design under Interference: A Local Randomization-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P014/"}, "work_id": "P014"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of text followed by a display. A red box marks the words 'average direct effect (Hudgens and Halloran, 2008)' in the sentence 'Then our primary target is the average direct effect (Hudgens and Halloran, 2008) of treatment for subjects who are close to the threshold'. The display reads 'tau equals N0 to the minus 1 times the sum over i with Xi in W0 of Y-bar i of 1 minus Y-bar i of 0'.", "caption": "Section 2, p. 131 (PDF p. 4). The marked words identify the target with the average direct effect of Hudgens and Halloran (2008). The display below them is treated minus control.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-N002.png"}, {"alt": "Four lines of text from the end of the Introduction. Red boxes mark the words 'This causal effect is equivalent to Hudgens and Halloran (2008)'s average direct effect', which run over two lines. The sentence continues 'for the subpopulation of subjects within the window near the threshold'.", "caption": "Introduction, p. 129 (PDF p. 2). The marked sentence says that the causal effect is equivalent to the average direct effect of Hudgens and Halloran (2008).", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-N002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P014-T7", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T7"}, {"id": "P014-T9", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T9"}, {"id": "P014-T10", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T10"}], "headline": "The target is the average direct effect of Hudgens and Halloran (2008) for the subjects in the window once $\\overline{Y}_i(z)$ is the conditional expectation $\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=z]$, and then with the opposite sign: their direct effect is control minus treatment.", "id": "P014-N002", "location": "Introduction, p. 129 (PDF p. 2), sentence beginning “This causal effect is equivalent to”; Section 2, p. 131 (PDF p. 4), sentence beginning “Then our primary target is the average direct effect”", "pdf_page": 2, "printed_page": "129", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P014-T10", "P014-T7", "P014-T9"], "text": {"as_printed": "\\emph{Introduction (p.~129).} This causal effect is \\hl{equivalent to} Hudgens and Halloran (2008)'s average direct effect for the subpopulation of subjects within the window near the threshold.\n\n\\emph{Section 2 (p.~131).} Then our primary target is \\hl{the average direct effect (Hudgens and Halloran, 2008)} of treatment for subjects who are close to the threshold:\n\\[\n\\tau=N_0^{-1}\\sum_{i:X_i\\in W_0}\\overline{Y}_i(1)-\\overline{Y}_i(0).\n\\]", "corrected": "\\emph{Introduction (p.~129).} With the \\hl{probability-weighted averages} of \\fid{P014-E001}, this causal effect is, \\hl{up to the sign convention,} Hudgens and Halloran (2008)'s average direct effect for the subpopulation of subjects within the window near the threshold.\n\n\\emph{Section 2 (p.~131).} With $\\overline{Y}_i(z)=\\mathrm{E}[y_i(\\mathbf{Z})\\mid Z_i=z]$ as in \\fid{P014-E001}, our primary target is the average direct effect (Hudgens and Halloran, 2008) of treatment for subjects who are close to the threshold, \\hl{written as treated minus control, which is the reverse of their sign convention}:\n\\[\n\\tau=N_0^{-1}\\sum_{i:X_i\\in W_0}\\bigl[\\overline{Y}_i(1)-\\overline{Y}_i(0)\\bigr].\n\\]", "evidence": "The evidence is a reading of the cited paper and an exact example. In the public author manuscript of Hudgens and Halloran (2008), the definition in Section 3.1 multiplies each potential outcome by $\\Pr_\\psi(\\mathbf{Z}_{i(j)}=\\omega\\mid Z_{ij}=z)$ and is followed by the statement that the ``individual average potential outcome is the conditional expectation''. In Section 3.2, equation (3) defines the individual direct causal effect as $\\overline{Y}_{ij}(0;\\psi)-\\overline{Y}_{ij}(1;\\psi)$. I did not check the page and equation numbers of the version of record.\n\nFor the example, let $N=4$, $b=1$ and $W_0=[-1,1]$. Let $(X_1,X_2,X_3)$ be a uniformly random arrangement of $-1$, $-1/2$ and $1/2$, and set $X_4=2$ if $(X_1,X_2,X_3)=(-1,-1/2,1/2)$ and $X_4=-2$ otherwise. Set $y_1(\\mathbf{z})=z_4$ and all other potential outcomes equal to zero. The conditions printed on p.~130 hold and the six arrangements are equiprobable. The assignment vectors $(0,0,1,0)$ and $(0,0,1,1)$ have probability $1/6$ each, and $(0,1,0,0)$ and $(1,0,0,0)$ have probability $1/3$ each. The printed $\\overline{Y}_1(0)$ is $1/3$, while $\\mathrm{E}[y_1(\\mathbf{Z})\\mid Z_1=0]=1/4$, and all other averages are zero. I obtain\n\\[\n\\tau=-\\frac19,\\qquad \\frac13\\bigl\\{\\mathrm{E}[y_1(\\mathbf{Z})\\mid Z_1=1]-\\mathrm{E}[y_1(\\mathbf{Z})\\mid Z_1=0]\\bigr\\}=-\\frac{1}{12}=\\mathrm{E}[\\hat\\tau],\n\\]\nand the average direct effect in the convention of Hudgens and Halloran is $1/12$. The printed $\\tau$ equals neither $1/12$ nor $-1/12$. To isolate the sign, fix the assignment of subject 4 at control and set $y_i(\\mathbf{z})=z_i$ for the subjects in the window. Then the two definitions of $\\overline{Y}_i(z)$ agree, $\\tau=\\mathrm{E}[\\hat\\tau]=1$, and the average direct effect in their convention is $-1$.", "problem": "Hudgens and Halloran (2008) define the individual average potential outcome as a conditional expectation. The outcomes of an individual are averaged over the assignments of the other individuals, each weighted by its conditional probability given the individual's own treatment. They define the individual direct causal effect as the average under control minus the average under treatment. The target $\\tau$ differs from the average of that effect over the subjects in the window in two ways. First, the printed $\\overline{Y}_i(z)$ weights the distinct assignment vectors equally and is not their conditional expectation in general; this is the error corrected in \\fid{P014-E001}. Second, $\\tau$ is treated minus control. Once the probabilities are inserted, $\\tau$ is therefore the negative of their average direct effect for the subjects in the window. The sign is a convention. The paper uses treated minus control in both $\\tau$ and $\\hat\\tau$, and no result on estimation depends on it.", "what_survives": "The estimator, the design and, with the definition corrected in \\fid{P014-E001}, the unbiasedness of the estimator are unaffected. The reference to Hudgens and Halloran (2008) remains the right one for the probability-weighted target. The identification with their average direct effect needs a remark on the sign."}, "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-N002", "work": {"arxiv_id": null, "doi": "10.1353/obs.2017.0001", "title": "The Regression Discontinuity Design under Interference: A Local Randomization-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P014/"}, "work_id": "P014"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Three lines of text followed by a display and two more lines. The display reads 'tau equals N0 to the minus 1 times the sum over i with Xi in W0 of Y-bar i of 1 minus Y-bar i of 0'. A red box marks the two terms after the summation sign, 'Y-bar i of 1 minus Y-bar i of 0', which are not enclosed in brackets.", "caption": "Section 2, p. 131 (PDF p. 4), the display that defines $\\tau$. The marked summand $\\overline{Y}_i(1)-\\overline{Y}_i(0)$ has no brackets.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-N004.png"}], "formal_release": null, "formal_statements": [{"id": "P014-T4", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T4"}, {"id": "P014-T8", "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-T8"}], "headline": "The summand in the definition of $\\tau$ should be bracketed: $\\tau=N_0^{-1}\\sum_{i:X_i\\in W_0}[\\overline{Y}_i(1)-\\overline{Y}_i(0)]$.", "id": "P014-N004", "location": "Section 2, p. 131 (PDF p. 4), the display that defines $\\tau$", "pdf_page": 4, "printed_page": "131", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P014-T4", "P014-T8"], "text": {"as_printed": "\\[\n\\tau=N_0^{-1}\\sum_{i:X_i\\in W_0}\\hl{\\overline{Y}_i(1)-\\overline{Y}_i(0)}.\n\\]", "corrected": "\\[\n\\tau=N_0^{-1}\\sum_{i:X_i\\in W_0}\\hl{\\bigl[}\\overline{Y}_i(1)-\\overline{Y}_i(0)\\hl{\\bigr]}.\n\\]", "evidence": "The evidence is a check of the page. The display that defines $\\tau$ has neither brackets nor a second summation sign. The later display reads $\\mathrm{E}[\\hat\\tau]=N_0^{-1}[\\sum_{i:X_i\\in W_0}\\overline{Y}_i(1)-\\sum_{i:X_i\\in W_0}\\overline{Y}_i(0)]$, and by linearity of the finite sum its right side equals the bracketed form below.", "problem": "The two terms after the summation sign are not bracketed. Read literally, the sum and the factor $N_0^{-1}$ apply to $\\overline{Y}_i(1)$ alone, and the index $i$ in $\\overline{Y}_i(0)$ is free. The prose that follows, and the display for $\\mathrm{E}[\\hat\\tau]$ on the same page, show that both terms belong to the summand.", "what_survives": "The intended target and its agreement with the display for $\\mathrm{E}[\\hat\\tau]$ are unaffected, and nothing depends on the missing brackets. The definition of $\\overline{Y}_i(z)$ inside the sum is corrected separately in \\fid{P014-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-N004", "work": {"arxiv_id": null, "doi": "10.1353/obs.2017.0001", "title": "The Regression Discontinuity Design under Interference: A Local Randomization-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P014/"}, "work_id": "P014"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Two entries of the reference list, Rosenbaum (2007) and Sobel (2006). A red box marks the page range 1398-1408 at the end of the Sobel entry.", "caption": "References, p. 133 (PDF p. 6). The marked page range of the Sobel (2006) entry ends at 1408.", "url": "https://pmaronow.github.io/llm-audit/works/P014/exhibits/P014-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "The page range of Sobel (2006) is 1398–1407, not 1398–1408.", "id": "P014-N006", "location": "References, p. 133 (PDF p. 6), the entry Sobel (2006)", "pdf_page": 6, "printed_page": "133", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Sobel, M.E. (2006). What do randomized studies of housing mobility demonstrate?: Causal inference in the face of interference. \\emph{Journal of the American Statistical Association} 101 (476): 1398--\\hl{1408}.", "corrected": "Sobel, M.E. (2006). What do randomized studies of housing mobility demonstrate?: Causal inference in the face of interference. \\emph{Journal of the American Statistical Association} 101 (476): 1398--\\hl{1407}.", "evidence": "The evidence is a check against the publisher. Its table of contents for the \\emph{Journal of the American Statistical Association} 101(476) lists Sobel's article at pages 1398--1407 and the next article, ``Exceedance Control of the False Discovery Proportion'', at pages 1408--1417. The publisher's page for the article has the same range.", "problem": "The range is one page too long. Page 1408 of that issue is the first page of the next article.", "what_survives": "The cited work remains identifiable, and nothing in the paper depends on the last page."}, "url": "https://pmaronow.github.io/llm-audit/works/P014/#P014-N006", "work": {"arxiv_id": null, "doi": "10.1353/obs.2017.0001", "title": "The Regression Discontinuity Design under Interference: A Local Randomization-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P014/"}, "work_id": "P014"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last fourteen lines of the left column. Red boxes mark the words 'the sum of the weights' in the definition; the words '12 contribute over half (51 percent)', which run over two lines; and five names in the list of the top twelve countries: Niger, Madagascar, Poland, Lesotho and Belarus.", "caption": "P. 258 (PDF p. 9), left column. The first marked phrase defines the contribution by sums. The marked share and the five marked countries come from the index of country means.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E001.png"}, {"alt": "Crop of Figure 1, two world maps side by side with the note beneath them. The left map, 'Nominal Sample', shows the countries of the sample in black. A red box marks the right map, 'Effective Sample', in which a few countries are dark, among them Argentina, Niger and Pakistan, and most are white or pale gray.", "caption": "Figure 1, p. 258 (PDF p. 9). The marked right panel, “Effective Sample”, is shaded by the index of country means.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E001-b.png"}, {"alt": "Dot plot in three groups, with the share of the weight in percent on the horizontal axis. For each country an open orange diamond gives the printed index and a filled blue circle gives the share by sums, joined by a line. First group, named on p. 258 and among the twelve largest sums: Uruguay, Philippines, Pakistan, Hungary, Argentina, Peru, Zimbabwe; Uruguay's circle is at 8.9 and its diamond at 6.4. Second group, named on p. 258 but not among the twelve largest sums: Lesotho, Madagascar, Poland, Niger, Belarus; Niger's diamond is at 4.7 and its circle at 1.2. Third group, among the twelve largest sums but not named: Chile, Fiji, Zambia, Ecuador, Nepal, with circles between 3.2 and 4.0 and diamonds between 2.3 and 2.8. A note says that the twelve named countries hold 50.9 percent of the printed index and 47.6 percent of the summed weight, and that the twelve countries with the largest sums hold 56.3 percent.", "caption": "The 17 countries that are among the twelve largest under either index, with the number of observed country-years in parentheses. Diamonds give the index behind the printed list (country mean, normalized); circles give the share that the text defines (country sum over the total). Drawn from the exact recalculation.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E001-shares.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T23", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T23"}], "headline": "The country contributions reported for the Jensen illustration were computed from each country’s mean weight and not from the sum that the text describes; with sums, the twelve largest contributors hold 56% of the weight, five of the twelve named countries are replaced, and 34 countries are needed to reach 90%.", "id": "P015-E001", "location": "Section “Effects of Regime Type on FDI”, p. 258 (PDF p. 9): left column, from “To measure the extent” to the list of twelve countries; right column, first paragraph; Figure 1, right panel", "pdf_page": 9, "printed_page": "258", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P015-T23"], "text": {"as_printed": "To measure the extent to which a given country contributes to the effect estimate, we compute the analogue to the left-hand side of Expression (9), taking the sum of the weights for that country (summing over all time periods) and dividing it by the sum of all the weights in the sample. The right panel in Figure 1 shows the results. Darker shading means a larger contribution by that country, and white shading means a small or negligible contribution. From the sample of 114 countries, 12 contribute over half (\\hl{51\\%}) of the weight used to construct the estimate of the effect of regime type on FDI, and \\hl{32} contribute 90\\% of the weight. The top 12 contributing countries, in descending order of their weights, are Uruguay, Hungary, \\hl{Niger}, Philippines, Argentina, \\hl{Madagascar}, Pakistan, Zimbabwe, \\hl{Poland}, Peru, \\hl{Lesotho}, and \\hl{Belarus}.\n\nImportantly, the lowest-contributing 14 countries combine to contribute less than 0.05\\% of the weight used to construct the effect estimate. These lowest-contributing countries, in ascending order of their weights, are Central African Republic, Albania, Germany, Russia, Haiti, Benin, South Africa, Yemen, Honduras, \\hl{DR Congo}, Czech Republic, Lithuania, \\hl{China}, and Latvia.", "corrected": "To measure the extent to which a given country contributes to the effect estimate, we compute the analogue to the left-hand side of Expression (9), taking the sum of the weights for that country (summing over all time periods) and dividing it by the sum of all the weights in the sample. The right panel in Figure 1 shows the results. Darker shading means a larger contribution by that country, and white shading means a small or negligible contribution. From the sample of 114 countries, 12 contribute over half (\\hl{56\\%}) of the weight used to construct the estimate of the effect of regime type on FDI, and \\hl{34} contribute \\hl{at least} 90\\% of the weight. The top 12 contributing countries, in descending order of their weights, are Uruguay, Philippines, Pakistan, Hungary, Argentina, Peru, Zimbabwe, \\hl{Chile}, \\hl{Fiji}, \\hl{Zambia}, \\hl{Ecuador}, and \\hl{Nepal}.\n\nImportantly, the lowest-contributing 14 countries combine to contribute less than 0.05\\% of the weight used to construct the effect estimate. These lowest-contributing countries, in ascending order of their weights, are Central African Republic and Albania (both with weight zero), Russia, Germany, Haiti, Honduras, Benin, South Africa, Yemen, \\hl{Republic of the Congo}, Czech Republic, Latvia, Lithuania, and \\hl{Namibia}.\n\nIn both lists the order of the countries that remain also changes. The right panel of Figure~1 is to be redrawn with shading proportional to the country sums, and the regional summary and the remarks on case selection are to be rewritten from the corrected lists. The exact shares are 56.3347\\% for the twelve largest contributors, 90.3453\\% for the largest 34 (the largest 33 hold 89.8429\\%), and 0.0107\\% for the lowest 14.", "evidence": "I checked the exact recalculation from the authors' data. The regression of regime type on the stated controls was solved in rational arithmetic for all 1,630 country-years and 114 countries, and the residuals satisfy all 123 normal equations exactly. The index of country means reproduces the printed twelve in the printed order, with a share of 50.91\\%, and it reproduces the printed list of the 14 lowest once its tenth name is corrected (see \\fid{P015-E002}). The shares by country sums are in the last column of the table. Under sums, 10 countries hold 49.7\\% and 11 hold 53.1\\%. The printed 90\\% rounds the mean-based share of 89.71\\%; it does not assert a minimum of 32 countries to reach at least 90\\%.\n\n\\begin{audittable}{lrrr}{Shares of the weight in the Jensen illustration, p.~258: as printed, and recalculated from the authors' data. ``Means'' is each country's mean weight divided by the sum of the 114 country means; ``sums'' is each country's summed weight divided by the total, as the text defines the contribution.}\nQuantity & As printed & Means & Sums \\\\\nTwelve largest countries & 51\\% & 50.91\\% & 56.33\\% \\\\\nTwelve countries named in the text & 51\\% & 50.91\\% & 47.58\\% \\\\\nLargest 32 countries & 90\\% & 89.71\\% & 89.25\\% \\\\\nLargest 33 countries & -- & 90.25\\% & 89.84\\% \\\\\nLargest 34 countries & -- & 90.72\\% & 90.35\\% \\\\\nLowest 14 countries & $<0.05\\%$ & 0.0458\\% & 0.0107\\% \\\\\n\\end{audittable}", "problem": "The page defines the contribution of a country as the sum of its weights over all its years, divided by the sum of all the weights in the sample. This is the left side of Expression~(9) with $Z_i$ the indicator of the country. The numbers that follow were computed differently: from the mean weight of each country, divided by the sum of the 114 country means. The replication script takes the country mean (\\texttt{tapply(w, jensen.cc\\$country, mean)}, line 33) and shades the right panel of Figure~1 with it. Let $p_c$ be the share of country $c$ by sums and $n_c$ its number of observed years. Then the index of country means is\n\\[\nq_c=\\frac{p_c/n_c}{\\sum_d p_d/n_d},\n\\]\nand $q_c=p_c$ for every country only if all countries with positive weight are observed for the same number of years. In this panel $n_c$ ranges from 1 to 22, and the index of country means raises the countries that are observed in few years. Niger, with 4 years, is third on the printed list and 27th by its sum. Under the definition that the page gives, the printed statements change as follows. The twelve named countries hold 47.6\\% of the weight, which is less than half. Five of them (Niger, Madagascar, Poland, Lesotho and Belarus) are not among the twelve largest contributors, and the order of the other seven changes. The largest 32 countries hold 89.2\\% of the weight, and 34 are needed to pass 90\\%. The list of the 14 lowest contributors changes in order and in one member: Namibia replaces China (its tenth name is a separate error, \\fid{P015-E002}). The regional summary and the advice on case selection in the right column were written from the printed list and the map.", "what_survives": "The finding that few countries carry the estimate survives: 12 of 114 countries hold 56.3\\% of the weight, 11 pass one half, and 34 pass 90\\%. The 14 lowest contributors hold 0.011\\%, inside the printed bound of 0.05\\%, and Russia, China and Germany each hold less than 0.005\\%. The aggregation by country takes place after the regression is fitted, so the OLS coefficient and the weights $\\hat w_i$ do not change. The left panel of Figure~1, the definition of the weights and the application to Gerber and Huber (2010) are not touched by this error. What changes is the right panel of Figure~1, the two lists, the shares, and the regional summary. Niger, Madagascar, Poland, Lesotho and Belarus leave the twelve, and Chile, Fiji, Zambia, Ecuador and Nepal enter. If the printed numbers were kept, the index would have to be described as a country's average weight per observed year, which is a different quantity from the contribution that Expression~(9) defines."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E001", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of nine lines at the top of the right column. A red box marks the words 'DR Congo' in the list 'Central African Republic, Albania, Germany, Russia, Haiti, Benin, South Africa, Yemen, Honduras, DR Congo, Czech Republic, Lithuania, China, and Latvia'. The first line of the crop says that these countries contribute less than 0.05 percent of the weight.", "caption": "P. 258 (PDF p. 9), right column, list of the 14 lowest-contributing countries. The marked name is the tenth entry.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E002.png"}], "formal_release": null, "formal_statements": [], "headline": "The tenth country in the list of the 14 lowest contributors is the Republic of the Congo, not DR Congo.", "id": "P015-E002", "location": "Section “Effects of Regime Type on FDI”, p. 258 (PDF p. 9), right column, list of the 14 lowest-contributing countries, tenth entry", "pdf_page": 9, "printed_page": "258", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "These lowest-contributing countries, in ascending order of their weights, are Central African Republic, Albania, Germany, Russia, Haiti, Benin, South Africa, Yemen, Honduras, \\hl{DR Congo}, Czech Republic, Lithuania, China, and Latvia.", "corrected": "\\hl{In the original ranking by country means}, these lowest-contributing countries, in ascending order of their weights, are Central African Republic, Albania, Germany, Russia, Haiti, Benin, South Africa, Yemen, Honduras, \\hl{Republic of the Congo}, Czech Republic, Lithuania, China, and Latvia.", "evidence": "The evidence is an exact recalculation from the authors' data, cross-checked against the authors' regression in R. With the Republic of the Congo in tenth place, the list is the list of the 14 lowest under the index of country means, which produced the printed figures, and these countries hold 0.0458\\% of it; the first two entries are tied at zero. The 14 names as printed hold 0.2225\\% by means and 0.2173\\% by sums. The authors' file that links the country names of the data to the names of the map also keeps the two countries apart.", "problem": "``DR Congo'' denotes the Democratic Republic of the Congo. In the data it is a separate country with 18 observed years, and it is not among the 14 lowest contributors: counted from the bottom it ranks 63rd by country means and 64th by country sums. The country in tenth place is the Republic of the Congo, with 4 observed years, under either way of aggregating the weights. With DR Congo in the list, the 14 named countries hold about 0.22\\% of the weight, which is above the bound of 0.05\\% stated in the preceding sentence.", "what_survives": "Only the name changes; no weight, coefficient or map does. The bound of 0.05\\% holds for the correct 14 countries. This correction repairs the list as it was computed. The list by country sums, which is the quantity the text describes, differs further, and \\fid{P015-E001} gives it."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E002", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of eight lines of the left column. One red box marks the number 0.020 in the words 'corresponds to a 0.020 increase in net FDI inflows'. A second red box marks '(p less than 0.001)' at the end of the same sentence.", "caption": "P. 258 (PDF p. 9), left column, first paragraph below Figure 1. The marked coefficient and significance level are those attributed to Model 10 of Jensen (2003).", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E003.png"}], "formal_release": null, "formal_statements": [], "headline": "Jensen (2003), Table 4, Model 10, reports a coefficient of 0.021 that is significant at the .01 level; the paper quotes it as 0.020 with $p<0.001$.", "id": "P015-E003", "location": "Section “Effects of Regime Type on FDI”, p. 258 (PDF p. 9), left column, first paragraph below Figure 1", "pdf_page": 9, "printed_page": "258", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "The resulting estimate implies that a one-unit increase in the Polity score corresponds to a \\hl{0.020} increase in net FDI inflows as a percentage of gross domestic product ($p < \\hl{0.001}$).", "corrected": "The resulting estimate implies that a one-unit increase in the Polity score corresponds to a \\hl{0.021} increase in net FDI inflows as a percentage of gross domestic product ($p < \\hl{0.01}$).", "evidence": "This is a check of the cited source: Jensen (2003), \\emph{International Organization} 57(3), p.~604, Table~4, Model~10, in the published version; p.~602 of the same article says that the standard errors are panel-corrected. The replication files agree on the coefficient: fitting OLS with the stated controls on all 1,630 country-years, I find 0.0211. The conventional standard error of that fit gives a two-sided $p$ of 0.043, which follows a different convention from Jensen's and is not what the sentence quotes.", "problem": "The sentence summarizes Model~10 in Table~4 of Jensen (2003), which the preceding sentences name. In that table the coefficient on democracy is 0.021 with three stars, the test statistic beneath it is 2.606, and the legend reads ``***$p<.01$''. The coefficient is therefore misquoted in the third decimal, and the significance level is overstated: the source supports $p<.01$, and under a normal reference a statistic of 2.606 has a two-sided $p$ of about 0.009, which is above 0.001.", "what_survives": "The association remains positive and significant at the .01 level in the source. No weight, figure or result of the paper uses these two numbers. Whether the ``$p<.05$'' quoted on p.~259 for the estimate of Gerber and Huber (2010) matches their table could not be settled, because their Table~4A was not available to me; \\fid{P015-E004} reports what the replication data show."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E003", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of the left column. A red box marks '20 percent', printed with a percent sign, in the words 'is associated with a 0.077 increase in log change in projected vacation spending (p less than .05), implying about a 20 percent increase'.", "caption": "P. 259 (PDF p. 10), left column, end of the paragraph beginning “We focus on regression Model (4)”. The marked percentage is the reading of the coefficient 0.077.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E004.png"}, {"alt": "Crop of five lines of the left column. A red box marks the words 'the natural log of the change' in the sentence 'We focus on regression Model (4) in Table 4A of their article, which uses a behavioral indicator of economic assessments: the natural log of the change in respondents' predicted vacation spending over the pre- and post-election waves of the survey'.", "caption": "P. 259 (PDF p. 10), left column, beginning of the same paragraph. The marked words describe the outcome.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T16", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T16"}], "headline": "A coefficient of 0.077 on a natural-log outcome corresponds to an increase of about 8%, not 20%, and the outcome is the change in the natural log of one plus predicted vacation spending, not the natural log of the change.", "id": "P015-E004", "location": "Section “Effects of Partisanship on Economic Assessments”, p. 259 (PDF p. 10), left column, paragraph beginning “We focus on regression Model (4)”", "pdf_page": 10, "printed_page": "259", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P015-T16"], "text": {"as_printed": "We focus on regression Model (4) in Table 4A of their article, which uses a behavioral indicator of economic assessments: \\hl{the natural log of the change} in respondents' predicted vacation spending over the pre- and post-election waves of the survey. [\\ldots] Gerber and Huber(2010) estimate that a unit change in a respondent's party identification scale in the direction of ``strong Democrat'' is associated with a 0.077 increase in log change in projected vacation spending ($p < .05$), implying about a \\hl{20\\%} increase.", "corrected": "We focus on regression Model (4) in Table 4A of their article, which uses a behavioral indicator of economic assessments: \\hl{the change in the natural log of one plus} respondents' predicted vacation spending over the pre- and post-election waves of the survey. [\\ldots] Gerber and Huber(2010) estimate that a unit change in a respondent's party identification scale in the direction of ``strong Democrat'' is associated with a 0.077 increase in log change in projected vacation spending ($p < .05$), implying about \\hl{an 8\\%} increase \\hl{in the exponentiated fitted outcome}.\n\nHere the outcome is $Y_i=\\log(1+S_{i,\\mathrm{post}})-\\log(1+S_{i,\\mathrm{pre}})$, and the exponentiated fitted outcome is the fitted geometric growth factor of $(1+S_{\\mathrm{post}})/(1+S_{\\mathrm{pre}})$. The 8\\% is not a percentage effect on arithmetic mean spending. Standard errors clustered by state (50 clusters, with a finite-sample adjustment and a $t_{49}$ reference distribution) give $p=0.0274<.05$.", "evidence": "The evidence is a derivation and a recalculation from the replication files. Let $\\hat m(d,x)$ be the fitted linear predictor at party identification $d$ and controls $x$, and let $\\hat\\beta$ be the coefficient on party identification. Then $\\hat m(d+1,x)-\\hat m(d,x)=\\hat\\beta$, which implies that\n\\[\n\\frac{\\exp\\{\\hat m(d+1,x)\\}}{\\exp\\{\\hat m(d,x)\\}}=e^{\\hat\\beta},\n\\]\nso the relative increase in the exponentiated fitted outcome is $e^{\\hat\\beta}-1$. For $x=77/1000$, bounding $e^x$ below by the partial sum $P_7=\\sum_{k=0}^{7}x^k/k!$ and above by $P_7+(x^8/8!)/(1-x/9)$ and comparing exact fractions, I obtain\n\\[\n1.0800420763<e^{0.077}<1.0800420765,\\qquad 8.00420763<100\\,(e^{0.077}-1)<8.00420765 .\n\\]\nThe replication data give a coefficient of 0.07683 on 1,469 observations, hence 7.99\\% from the unrounded value. In those data the outcome variable is exactly the difference of the two log variables that are stored with it, and each of these equals $\\log(1+S)$ to the precision of storage.\n\nNote that the 8\\% describes the fitted geometric growth factor of $(1+S_{\\mathrm{post}})/(1+S_{\\mathrm{pre}})$ and not arithmetic mean spending. Consider two data sets of four observations, each with $S_{\\mathrm{pre}}=1$ throughout and two observations at each value of a binary $D$. Post-election spending is $(1,1)$ at $D=0$ in both data sets; it is $(1,1)$ at $D=1$ in the first and $(0,3)$ at $D=1$ in the second. The coefficient on $D$ is zero in both, since $[\\log(1/2)+\\log 2]/2=0$, while mean post-election spending at $D=1$ exceeds that at $D=0$ by 0\\% in the first and by 50\\% in the second.", "problem": "Two descriptions in the paragraph are wrong. First, a one-point move on the party identification scale raises the fitted outcome by 0.077 natural-log points. Since $e^{0.077}-1=0.080$, the implied increase is about 8\\%. An increase of 20\\% would need a coefficient of $\\log 1.2=0.182$, or a move of about 2.4 points on a scale that runs from $-2$ to $2$. Second, the outcome of the fitted model is the change in the natural log of one plus predicted vacation spending,\n\\[\nY_i=\\log(1+S_{i,\\mathrm{post}})-\\log(1+S_{i,\\mathrm{pre}}),\n\\]\nwhere $S_{i,\\mathrm{pre}}$ and $S_{i,\\mathrm{post}}$ are the predicted spending of respondent $i$ in the two waves. The natural log of the change, $\\log(S_{i,\\mathrm{post}}-S_{i,\\mathrm{pre}})$, is undefined for the 676 respondents whose predicted spending did not change and for the 420 whose predicted spending fell. The words ``log change'' later in the paragraph, and Table~1, describe the variable correctly.", "what_survives": "The coefficient 0.077, the sample of 1,469 and every entry of Table~1 are unaffected: all 76 numerical cells of the table reproduce at the printed precision. The weights and the comparison of the nominal and effective samples do not use the 20\\%. The printed $p<.05$ is reproduced with standard errors clustered by state (50 clusters, a $t_{49}$ reference, $p=0.0274$). With the ordinary standard errors of OLS, I find $p=0.076$, and I have not been able to check which convention the table of Gerber and Huber (2010) uses."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E004", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The first eleven lines of the right column of page 257. Red boxes mark two sentences. The first reads: In effect, the estimator computes a series of quantities along the lines of Expression (7) and aggregates them on the basis of their precision. The second reads: The effective sample is created from a combination of the precision weights and the within-group multiple regression weights, followed by the mark of footnote 17. Between them, unmarked, is the sentence: The precision weighting does not undo the sample distortions that the basic multiple regression weights introduce within each group.", "caption": "Page 257 (PDF p. 8), top of the right column; the paragraph begins at the foot of the left column. The two marked sentences describe the estimator as an aggregation of the within-group quantities of Expression (7) by their precision.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T9", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T9"}, {"id": "P015-T10", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T10"}], "headline": "The random-coefficient estimator weights the full vectors of group-specific OLS coefficients by matrices, and its treatment coordinate is in general not a precision-weighted average of the within-group quantities of Expression (7); that description is exact only under a condition on the matrix weights, and is otherwise an approximation for large samples within groups.", "id": "P015-E005", "location": "“Random Coefficient Models”, p. 257 (PDF p. 8), from the last three lines of the left column through the first paragraph of the right column; Expression (7) is on p. 254 (PDF p. 5)", "pdf_page": 8, "printed_page": "257", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P015-T10", "P015-T9"], "text": {"as_printed": "Under the standard working assumption that the $\\beta_g$s are independent draws from a distribution centered on $\\bar\\beta$, the estimator is a precision-weighted average of the group-specific OLS estimates (Hsiao and Pesaran 2008). Each of the group-specific OLS estimates is simply the value within each group of the quantity given by Expression (7). \\hl{In effect, the estimator computes a series of quantities along the lines of Expression (7) and aggregates them on the basis of their precision.} The precision weighting does not undo the sample distortions that the basic multiple regression weights introduce within each group. \\hl{The effective sample is created from a combination of the precision weights and the within-group multiple regression weights.}$^{17}$", "corrected": "Under the standard working assumption that the group coefficient vectors $\\theta_g=(\\alpha_g,\\beta_g,\\gamma_g')'$ are independent draws from a distribution with mean $\\mu$, whose treatment coordinate is $\\bar\\beta$, and covariance matrix $S$, independently of the regressors and the errors, the estimator is a precision-weighted average of the \\hl{vectors of} group-specific OLS estimates (Hsiao and Pesaran 2008). \\hl{Let $\\widehat\\theta_g$ be the vector of OLS estimates in group $g$, and let $P_g$ be the inverse of its covariance matrix $S+\\sigma_g^2(Z_g'Z_g)^{-1}$, where $Z_g$ is the matrix of regressors of group $g$, with $Z_g'Z_g$ invertible, and $\\sigma_g^2$ is the error variance of group $g$, with mean-zero errors uncorrelated within and across groups given the regressors. The estimator is}\n\\[\n\\hl{\\widehat\\mu=\\sum_gA_g\\widehat\\theta_g,\\qquad A_g=\\Bigl(\\sum_hP_h\\Bigr)^{-1}P_g,}\n\\]\n\\hl{and its treatment coordinate estimates $\\bar\\beta$.} The treatment coordinate of each $\\widehat\\theta_g$ \\hl{converges in probability to the within-group quantity in Expression (7), under its assumptions and regularity conditions}. \\hl{The treatment coordinate of $\\widehat\\mu$ in general also includes the other coordinates of the $\\widehat\\theta_g$, and Expression (7) alone does not characterize it. It is a combination $\\sum_ga_g\\widehat\\beta_g$ of the group-specific treatment estimates with scalar weights $a_g$, for all values of the estimates, if and only if the treatment row of every $A_g$ has zero entries off the treatment coordinate. When the samples within groups grow so that $(Z_g'Z_g)^{-1}$ tends to zero, with $S$ positive definite, fixed error variances and the number of groups $G$ fixed, $A_g$ tends to $G^{-1}I$ and those entries vanish.} The precision weighting does not undo the sample distortions that the basic multiple regression weights introduce within each group. \\hl{Under that condition, or approximately in that limit,} the effective sample is created from a combination of the precision weights and the within-group multiple regression weights.", "evidence": "The evidence is a derivation of the estimator and an exact counterexample. Let $\\theta_g=(\\alpha_g,\\beta_g,\\gamma_g')'$ be the coefficient vector of group $g$, let $Z_g$ be its matrix of regressors, with $Z_g'Z_g$ invertible, and let $\\widehat\\theta_g$ be its OLS estimate. Suppose that the $\\theta_g$ are independent with mean $\\mu$ and covariance matrix $S$, independently of the regressors and the errors, and that the errors are uncorrelated within and across groups, with mean zero and variance $\\sigma_g^2$ in group $g$ given the regressors. Then $\\widehat\\theta_g$ has the covariance matrix $\\Omega_g=S+\\sigma_g^2(Z_g'Z_g)^{-1}$ given the regressors. Let $P_g=\\Omega_g^{-1}$. The generalized least squares estimator minimizes $\\sum_g(\\widehat\\theta_g-m)'P_g(\\widehat\\theta_g-m)$ over $m$, and I obtain\n\\[\n\\widehat\\mu=\\sum_gA_g\\widehat\\theta_g,\\qquad A_g=\\Bigl(\\sum_hP_h\\Bigr)^{-1}P_g,\\qquad \\sum_gA_g=I.\n\\]\nGeneralized least squares on the original observations gives the same estimate. Let $e_\\beta$ denote the coordinate vector of the treatment. The treatment coordinate of $\\widehat\\mu$ is the pooled treatment estimate. For given scalars $a_g$, it equals $\\sum_ga_g\\widehat\\beta_g$ for all values of the $\\widehat\\theta_g$ if and only if $e_\\beta'A_g=a_ge_\\beta'$ for every $g$.\n\nThe counterexample has two groups of three observations, an intercept and a binary treatment, no covariates, $S=I$ and $\\sigma_1^2=\\sigma_2^2=1$.\n\\begin{audittable}{lllrrrr}{The counterexample. For these designs the pooled treatment estimate is $(5/61)\\widehat\\alpha_1-(5/61)\\widehat\\alpha_2+(32/61)\\widehat\\beta_1+(29/61)\\widehat\\beta_2$.}\nGroup & Treatments & Outcomes & $\\widehat\\alpha_g$ & $\\widehat\\beta_g$ & Weight on $\\widehat\\alpha_g$ & Weight on $\\widehat\\beta_g$ \\\\\n1 & $(0,1,1)$ & $(0,0,0)$ & $0$ & $0$ & $5/61$ & $32/61$ \\\\\n2 & $(0,0,1)$ & $(2,2,2)$ & $2$ & $0$ & $-5/61$ & $29/61$ \\\\\n\\end{audittable}\nThe treatment estimates of both groups are zero, and every combination of them with scalar weights is zero. The pooled estimate of the mean coefficient vector is $(68/61,-10/61)'$, and the pooled treatment estimate is $-10/61$. The sample has positive probability under a correctly specified model in which the treatment is assigned at random and the treatment coefficients are independent draws centered on $\\bar\\beta=0$; under one such model an event producing this sample has probability $19683/134217728$. If each row of the two designs is repeated $r$ times, the pooled treatment estimate is $(-4r-6)/(16r^2+36r+9)$, which is nonzero for every $r$ and tends to zero as $r$ grows.", "problem": "The target $\\bar\\beta$ is a scalar, the average over groups of the treatment coefficient in the Swamy model (10), and Expression (7) is the limit of another scalar, the OLS coefficient on the treatment. The passage therefore says that the estimator pools the quantities of Expression (7), one for each group, with scalar precision weights, and that the effective sample combines those weights with the regression weights within each group. The generalized least squares estimator does not have this form in general. It weights the full vector of OLS coefficients of each group, which holds the intercept, the treatment coefficient and the coefficients on the covariates, by the inverse of the covariance matrix of that vector. The treatment coordinate of the pooled estimate then includes the intercepts and the other coefficients of the groups, with weights that need not vanish, and Expression (7) alone does not characterize it. The words ``precision-weighted average'' are correct when the weights are matrices that act on the coefficient vectors. What fails is the reduction to the scalar quantities of Expression (7).", "what_survives": "Expression (7), Theorem A.1, the comparison with the leverage statistic in Appendix B and the weights computed for the two applications concern OLS and do not use this passage. The distinction in footnote 17 between an average over groups and an average over units is unaffected, and the counterexample does not contradict the consistency of the estimator under a correctly specified model with exogenous treatment: with known covariance matrices its conditional mean is $\\mu$, because $\\sum_gA_g=I$. The warning that random coefficient models do not by themselves return a representative average of unit-level effects is not refuted. What changes is the account of the effective sample for these models on p.~257, which the introduction previews on p.~252. The effective sample is the stated combination of precision weights and within-group regression weights only under the condition given below, or approximately when the samples within groups are large. No later formula, table or figure uses that account. The check treats the covariance matrices as known and does not address versions of the estimator that estimate them."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E005", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text at the foot of the left column of page 261. The passage reads: We can draw a connection between the positivity assumption and the results that we obtained above for multiple regression. The positivity assumption implies that Var[D_i given X_i = x] is greater than 0, for all values of x. Thus, when a multiple regression places (either approximately or exactly) no weight on a set of units with a given covariate profile, this result implies that positivity is violated and. Red outlines surround the word approximately and the words positivity is violated.", "caption": "p. 261 (PDF p. 12), end of the left column: the start of the paragraph. The outlines mark “approximately” and “positivity is violated”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E006.png"}, {"alt": "Crop of twenty-three lines of text at the top of the right column of page 261, continuing the paragraph. It includes the sentences: In the Jensen (2003) study, many substantively important countries, including Russia and China, cannot be included in estimates of the causal effects of regime type on FDI. No consistent, extrapolation-free estimator of E[tau_i(d, d')] is available for any (d, d') if our interest is in the average over the entirety of the 114 countries in the nominal sample. Later: Of course, the effective sample that multiple regression generates is one for which positivity holds by construction. Red outlines surround the words No consistent, extrapolation-free estimator and the words positivity holds by construction.", "caption": "p. 261 (PDF p. 12), top of the right column: the rest of the paragraph. The outlines mark “No consistent, extrapolation-free estimator” and “positivity holds by construction.”", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E006-b.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T1", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T1"}, {"id": "P015-T2", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T2"}, {"id": "P015-T3", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T3"}, {"id": "P015-T4", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T4"}, {"id": "P015-T17", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T17"}], "headline": "A regression weight that is small but positive does not show that positivity fails or that a representative effect cannot be estimated, and for a treatment with more than two values the effective sample need not satisfy positivity for a chosen pair $(d,d')$; an exactly zero population conditional variance does rule positivity out.", "id": "P015-E006", "location": "Section “Estimating Representative Causal Effects”, p. 261 (PDF p. 12): last paragraph of the left column, continued in the right column, from “The positivity assumption implies” to “holds by construction”; the positivity display is in the left column of the same page", "pdf_page": 12, "printed_page": "261", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P015-T1", "P015-T17", "P015-T2", "P015-T3", "P015-T4"], "text": {"as_printed": "\\emph{The definition, p.~261, left column.} Formally, the positivity assumption is as follows:\n\\[\n\\Pr[D_i = d \\mid X_i = x] > 0, \\qquad \\Pr[D_i = d' \\mid X_i = x] > 0\n\\]\nfor all values of $x$ in the target population represented by the nominal sample.\n\n\\emph{The passage, p.~261.} We can draw a connection between the positivity assumption and the results that we obtained above for multiple regression. The positivity assumption implies that $\\operatorname{Var}[D_i \\mid X_i = x] > 0$, for all values of $x$. Thus, when a multiple regression places (either \\hl{approximately} or \\hl{exactly}) no weight on a set of units with a given covariate profile, this result implies that \\hl{positivity is violated} and representative causal effects cannot be estimated for the target population without assumption-based extrapolation or interpolation. Examination of the effective sample provides insight about what can be learned from any given study. In the Jensen (2003) study, many substantively important countries---including Russia and China---cannot be included in estimates of the causal effects of regime type on FDI. \\hl{No consistent, extrapolation-free estimator} of $\\mathrm{E}[\\tau_i(d,d')]$ is available for any $(d,d')$ if our interest is in the average over the entirety of the 114 countries in the nominal sample. This is not due to any fault on Jensen's part; it is an implication of the variation that nature has provided. However, in other studies, such as Gerber and Huber (2010), positivity appears to hold, and thus estimation of representative average causal effects is straightforward. Of course, the effective sample that multiple regression generates is one for which \\hl{positivity holds by construction}. In conjunction with the linearity assumptions embodied by Expressions (1) and (3) (assumptions that are with little loss of generality when $D_i$ is binary), regression facilitates estimation in a manner that reflects the regions of the data where causal inference is possible.", "corrected": "\\emph{p.~261, from ``The positivity assumption implies'' to ``holds by construction''.} Positivity for two distinct treatment values $d$ and $d'$ implies that $\\operatorname{Var}[D_i \\mid X_i = x] > 0$, for all values of $x$ in the target population. Thus, when the conditional variance of the treatment at a covariate profile is \\hl{exactly zero in the population}, positivity is violated at that profile for every pair $(d,d')$ with $d \\neq d'$, and \\hl{representative effects for that profile require} assumption-based extrapolation or interpolation. \\hl{A small positive conditional variance, or a small estimated regression weight, does not establish a violation of positivity.} \\hl{A small positive population conditional variance} shows that the profile supplies little identifying variation, and estimates for it may be imprecise.\n\nIn the Jensen (2003) study, the weights show that many substantively important countries---including Russia and China---\\hl{contribute very little to this regression estimate}. The statement that no consistent, extrapolation-free estimator of $\\mathrm{E}[\\tau_i(d,d')]$ is available for the 114 countries of the nominal sample \\hl{requires in addition a violation of positivity in the population} for the chosen pair, and the weights alone do not establish one.\n\nIn Gerber and Huber (2010), \\hl{the weighting diagnostics alone do not establish positivity for a chosen pair}.\n\nThe effective \\hl{population defined by the population weights $w_i$} is one in which \\hl{the conditional variance of the treatment is positive} at every represented covariate profile. For a binary treatment this is positivity for the pair $(0,1)$. For a treatment with more than two values, positivity for a chosen pair $(d,d')$ requires in addition that \\hl{both treatment values have positive conditional probability at each of these profiles in the effective population}.\n\nThe opening sentence, the sentence beginning ``Examination of the effective sample'', and the final sentence on the linearity assumptions are unchanged.", "evidence": "The evidence is three exact examples, each computed in rational arithmetic. By the effective population I mean the nominal population reweighted in proportion to $w_i$.\n\n\\emph{Example A: a small weight with positivity.} Let $X_i$ equal $0$ or $1$ with probability $1/2$ each, let $D_i$ be binary with $\\Pr[D_i = 1 \\mid X_i = 0] = \\varepsilon = 10^{-6}$ and $\\Pr[D_i = 1 \\mid X_i = 1] = 1/2$, and let $Y_i(d) = 2X_i + (1 + X_i)d$. Since the potential outcomes are functions of $X_i$, unconfoundedness holds. The effects are linear in $d$ with $\\tau_i = 1 + X_i$, and $\\mathrm{E}[D_i \\mid X_i] = \\varepsilon + (1/2 - \\varepsilon)X_i$ is linear in $X_i$. Both treatment values have positive probability at both profiles.\n\\begin{audittable}{rrrrrr}{Example A: the two covariate profiles.}\n$X_i$ & $\\Pr[D_i = 1 \\mid X_i]$ & $\\operatorname{Var}[D_i \\mid X_i]$ & Nominal share & Effective share & $\\tau_i$ \\\\\n$0$ & $1/10^{6}$ & $999999/10^{12}$ & $1/2$ & $999999/250000999999$ & $1$ \\\\\n$1$ & $1/2$ & $1/4$ & $1/2$ & the remainder & $2$ \\\\\n\\end{audittable}\nThe effective share of the profile $X_i = 0$ is about $4.0 \\times 10^{-6}$. The average effect in the nominal population is $3/2$. The population OLS coefficient on $D_i$ is $500000999999/250000999999$, which is the weighted average of Theorem~A.1. Let $e(x) = \\Pr[D_i = 1 \\mid X_i = x]$ and\n\\[\nZ_i = \\frac{D_i Y_i}{e(X_i)} - \\frac{(1 - D_i) Y_i}{1 - e(X_i)} .\n\\]\nComputing its first two moments, I find $\\mathrm{E}[Z_i] = 3/2$ and $\\operatorname{Var}(Z_i) = 2000071/4$. The mean of $n$ independent copies of $Z_i$ is then unbiased for $3/2$ with variance $2000071/(4n)$, and Chebyshev's inequality implies that it is consistent. Note that this estimator uses only outcomes that are observed under each treatment at each profile. The same construction has positivity for every $\\varepsilon$ in $(0, 1/2)$. The effective share of the profile $X_i = 0$ is $\\varepsilon(1-\\varepsilon)/[\\varepsilon(1-\\varepsilon) + 1/4]$, which tends to $0$ as $\\varepsilon \\to 0$. Hence no threshold for ``approximately no weight'' implies a violation of positivity.\n\n\\emph{Example B: positive weights, and a chosen pair without support.} Let $(X_i, D_i)$ take the values $(0,0)$, $(0,1)$, $(1,0)$ and $(1,2)$ with probability $1/4$ each. Then $\\mathrm{E}[D_i \\mid X_i] = 1/2 + X_i/2$ is linear in $X_i$.\n\\begin{audittable}{rrrrr}{Example B: the four cells, each with nominal probability $1/4$.}\n$X_i$ & $D_i$ & $\\mathrm{E}[D_i \\mid X_i]$ & $w_i$ & Effective probability \\\\\n$0$ & $0$ & $1/2$ & $1/4$ & $1/10$ \\\\\n$0$ & $1$ & $1/2$ & $1/4$ & $1/10$ \\\\\n$1$ & $0$ & $1$ & $1$ & $2/5$ \\\\\n$1$ & $2$ & $1$ & $1$ & $2/5$ \\\\\n\\end{audittable}\nEvery weight is positive and the profile $X_i = 1$ has effective share $4/5$, yet $\\Pr[D_i = 1 \\mid X_i = 1] = 0$ in the nominal and in the effective population. Positivity for the pair $(0,1)$ fails. I compare two sets of potential outcomes. In the first, $Y_i(d) = d$ for every unit. The second differs from the first only in that $Y_i(1) = 3$ when $X_i = 1$. Both satisfy unconfoundedness. Since $D_i = 1$ never occurs at $X_i = 1$, the observed $(X_i, D_i, Y_i)$ has the same distribution under both. The average of $\\tau_i(0,1)$ over the nominal population is $1$ in the first and $2$ in the second; over the effective population it is $1$ and $13/5$. I conclude that no estimator based on the observed data is consistent in both cases.\n\n\\emph{Example C: an estimated weight of zero.} Suppose that $X_i$ and $D_i$ are independent with $\\Pr[X_i = 1] = \\Pr[D_i = 1] = 1/2$, so that positivity holds and $\\operatorname{Var}[D_i \\mid X_i = x] = 1/4$ at both profiles. The ordered sample $(X_i, D_i) = (0,0), (0,0), (1,0), (1,1)$ has probability $1/256$. In this sample the regression of $D_i$ on a constant and $X_i$ has coefficients $(0, 1/2)$ and residuals $(0, 0, -1/2, 1/2)$. The two units with $X_i = 0$ receive an estimated weight of exactly zero.", "problem": "The paragraph draws three conclusions from the regression weights. Recall that $w_i = (D_i - \\mathrm{E}[D_i \\mid X_i])^2$, so that $\\mathrm{E}[w_i \\mid X_i = x] = \\operatorname{Var}[D_i \\mid X_i = x]$ by the definition of the conditional variance. One direction is correct. Positivity for two distinct values $d$ and $d'$ at $x$ implies that $\\operatorname{Var}[D_i \\mid X_i = x] > 0$. Hence positivity fails for every pair at any covariate profile such that this variance is zero in the population. The three conclusions of the paragraph go further, and none of them follows.\n\nFirst, the paragraph treats a weight of approximately zero as a violation of positivity. A conditional variance that is small and positive is compatible with positivity at every profile, and with a consistent estimator of the representative effect that uses only observed outcomes. A small positive population conditional variance shows that the profile supplies little variation in the treatment and that estimates for it may be imprecise. Second, the paragraph says that positivity holds in the effective sample by construction. At the population level, the effective sample assigns positive weight to the profiles at which the treatment varies. For a binary treatment this is positivity for the pair $(0,1)$. For a treatment with more than two values it means only that $D_i$ takes at least two values at $x$, and these need not be the chosen $d$ and $d'$. The positivity display on the same page is stated for the chosen pair. Both applications have such treatments: a regime score with 21 values in Jensen (2003), and party identification on a five-point scale in Gerber and Huber (2010). Third, the weights used in an application are estimated: they are squared residuals from a fitted regression. An estimated weight that is zero or small does not establish that the conditional variance is zero in the population.\n\nThe statement about Jensen (2003), that no consistent, extrapolation-free estimator exists for any $(d,d')$ over the 114 countries, therefore does not follow from the small weights of countries such as Russia and China. Whether the statement is true is a separate question, which I do not settle. In the replication data Russia has two observations, both with regime score 14, and China has seven, all with score 3. These observations are compatible with a violation of positivity, and they do not determine the conditional distribution of the regime score in the population.", "what_survives": "Theorem~A.1, the weights of Expressions (7) and (8), the effective-sample summaries of Expression (9), the simulations and the weighting diagnostics for both applications are unaffected. So is the conclusion that OLS estimates an average effect weighted by the conditional variance of the treatment, which can differ from the average effect in the nominal population. It remains true that zero conditional variance in the population rules out positivity, and that positive conditional variance is positivity for a binary treatment. The paragraph can no longer conclude from near-zero weights that positivity is violated, or that positivity holds in the effective sample when the treatment has more than two values. The statement about Jensen (2003) becomes conditional on a violation of positivity that is shown by other means. No reported number changes. The form of the positivity display for a continuous treatment is a separate point, \\fid{P015-N011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E006", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the heading Appendix C: Maximum Likelihood Estimation for Nonlinear Models and the fourteen lines of text below it, at the foot of the right column of page 264. Red outlines surround the sentence This holds for generalized linear models such as probit, Poisson regression, ordered logit or probit, multinomial logit or probit, and duration models, and the words To apply the result below for these other models, with which the page ends.", "caption": "Appendix C, first paragraph, p. 264 (PDF p. 15), right column. The outlines mark the sentence that lists the other models and the start of the instruction, which continues on p. 265.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E008.png"}, {"alt": "Crop of the top of the left column of page 265. The first two lines read: one substitutes the relevant predicted mean or inverse link function into the estimating equation. Red outlines surround these two lines. Below them are the logistic regression model, Pr[Y_i = 1 given D_i, X_i] = inverse logit of (alpha_ell + beta_ell D_i + X_i gamma_ell), the definition of W_i, and the estimating equation: the sum over i of W_i times (Y_i minus the inverse logit of W_i theta-hat) equals the zero vector of size 1 by (K+2).", "caption": "Appendix C, p. 265 (PDF p. 16), top of the left column: the end of the instruction, the logistic model and its estimating equation. The outlines mark “one substitutes the relevant predicted mean or inverse link function into the estimating equation.”", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E008-b.png"}, {"alt": "Crop of the section heading Maximum Likelihood Estimation and Generalized Linear Models and the five lines below it. Red outlines surround the first sentence: The results for linear regression carry over directly to nonlinear regression models fit via maximum likelihood. The text continues: This includes the class of generalized linear models, such as logit, probit, multinomial logit or probit, Poisson regression, and parametric or semiparametric (e.g., Cox); the sentence continues on the next page.", "caption": "Main text, p. 256 (PDF p. 7), right column, start of the section “Maximum Likelihood Estimation and Generalized Linear Models”. The outlines mark its first sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E008-c.png"}, {"alt": "Crop of seven lines of text from the left column of page 257. The passage ends: (See Appendix C for the mathematical details. The logic extends to maximum likelihood estimates of any generalized linear model.) Red outlines surround the sentence The logic extends to maximum likelihood estimates of any generalized linear model.", "caption": "Main text, p. 257 (PDF p. 8), left column, end of the paragraph on logistic regression. The outlines mark “The logic extends to maximum likelihood estimates of any generalized linear model.”", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E008-d.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T5", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T5"}, {"id": "P015-T6", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T6"}, {"id": "P015-T7", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T7"}, {"id": "P015-T8", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T8"}, {"id": "P015-T18", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T18"}], "headline": "Substituting another inverse link into the logistic estimating equation of Appendix C does not yield the maximum-likelihood equations of probit and the other listed models, and the argument that carries the OLS weights over to nonlinear models holds as printed only for models whose likelihood score is a constant multiple of the equation so obtained, such as logistic regression and Poisson regression with the log link.", "id": "P015-E008", "location": "Appendix C, first paragraph, p. 264 (PDF p. 15), continued at the top of p. 265 (PDF p. 16); the related statements of the main text are in the section “Maximum Likelihood Estimation and Generalized Linear Models”, pp. 256–257 (PDF pp. 7–8)", "pdf_page": 15, "printed_page": "264", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P015-T18", "P015-T5", "P015-T6", "P015-T7", "P015-T8"], "text": {"as_printed": "\\emph{Appendix~C, pp.~264--265.} While our illustration is for logistic regression, the results hold for any nonlinear model that can be expressed in terms of estimating equations in the manner that follows (Liang and Zeger 1986). \\hl{This holds for generalized linear models such as probit, Poisson regression, ordered logit or probit, multinomial logit or probit, and duration models} (McCullagh and Nelder 1999). To apply the result below for these other models, \\hl{one substitutes the relevant predicted mean or inverse link function into the estimating equation}.\n\n\\emph{Appendix~C, p.~265.} The ML estimator for $\\hat\\theta$ solves the following estimating equation (Liang and Zeger 1986):\n\\[\n\\sum_{i=1}^{n} \\psi(Y_i, W_i, \\hat\\theta) = \\sum_{i=1}^{n} \\bigl( W_i\\{Y_i - \\operatorname{logit}^{-1}(W_i\\hat\\theta)\\} \\bigr) = \\mathbf{0}_{1\\times(K+2)},\n\\]\nwhere $K$ is the number of variables in $X_i$.\n\n\\emph{Main text, pp.~256--257.} The results for linear regression \\hl{carry over directly} to nonlinear regression models fit via maximum likelihood. This includes the class of generalized linear models, such as logit, probit, multinomial logit or probit, Poisson regression, and parametric or semiparametric (e.g., Cox) duration models (Long 1997; McCullagh and Nelder 1999). [\\ldots] (See Appendix C for the mathematical details. The logic extends to maximum likelihood estimates of \\hl{any generalized linear model}.)", "corrected": "\\emph{Appendix~C, first paragraph.} The illustration is for logistic regression. The same argument applies to any model \\hl{whose maximum-likelihood score is a common nonzero multiple of the estimating equation displayed below}, with the inverse link of the model in place of the inverse logit. A scalar generalized linear model \\hl{with its canonical link} and a common dispersion is of this kind; examples are logistic regression and Poisson regression with the log link, and for such a model one substitutes the inverse link function into the estimating equation. Other links and likelihoods, including probit, ordered and multinomial models and duration models, \\hl{require their own score equations and their own approximation arguments}. For a binary probit model the score is\n\\[\n\\sum_{i=1}^{n} W_i \\, \\hl{\\frac{\\phi(W_i\\theta)}{\\Phi(W_i\\theta)\\{1 - \\Phi(W_i\\theta)\\}}} \\, \\{Y_i - \\Phi(W_i\\theta)\\} ,\n\\]\nwhere $\\phi$ and $\\Phi$ are the standard normal density and distribution function. A first-order expansion of this score must also differentiate the multiplier $\\phi/[\\Phi(1-\\Phi)]$. It reduces to the OLS equations only if the resulting term cancels, or under a further local approximation that is stated and justified. The logistic calculation alone does not give the OLS weights for the other listed models.\n\n\\emph{Main text, p.~256.} The results for linear regression carry over, \\hl{as a first-order approximation}, to logistic regression and to other nonlinear regression models fit via maximum likelihood \\hl{whose score has the form of the estimating equation of Appendix~C}, such as Poisson regression with the log link.\n\n\\emph{Main text, p.~257.} (See Appendix C for the mathematical details. The logic extends to maximum likelihood estimates of \\hl{scalar} generalized linear models \\hl{with a canonical link and a common dispersion}; other models, including probit, \\hl{need a separate argument}.)\n\nThe sentence on p.~252 that the results ``also carry through to generalized linear models (logit, probit, etc.)'' and the last sentence of Appendix~C are limited in the same way.", "evidence": "The evidence is a derivation of the score and exact counterexamples.\n\n\\emph{The score.} For a binary outcome with $\\Pr[Y_i = 1 \\mid W_i] = \\mu(\\eta_i)$, the log likelihood of one observation is $Y_i \\log \\mu(\\eta_i) + (1 - Y_i)\\log\\{1 - \\mu(\\eta_i)\\}$. Differentiating in $\\eta_i$, I have\n\\[\n\\mu'(\\eta_i)\\left\\{ \\frac{Y_i}{\\mu(\\eta_i)} - \\frac{1 - Y_i}{1 - \\mu(\\eta_i)} \\right\\} = h(\\eta_i)\\{Y_i - \\mu(\\eta_i)\\}, \\qquad h = \\frac{\\mu'}{\\mu(1-\\mu)} .\n\\]\nFor $\\mu = \\operatorname{logit}^{-1}$, $\\mu' = \\mu(1-\\mu)$ and $h \\equiv 1$.\n\n\\emph{Two parameter values that solve the substituted equation and are not ML estimates.} In both examples $W_i = (1 \\;\\; D_i \\;\\; X_i)$ with $D_i$ binary, and the cells are the same for $D_i = 0$ and for $D_i = 1$. In the first example $\\mu(\\eta) = \\eta/(1+\\eta)$ on $\\eta>0$ and $\\theta = (1, 0, 1)'$, and the cells $X_i = 0, 1, 2$ have $12$ observations each, with $7$, $6$ and $10$ successes. In the second example the model is probit and $\\theta = (0, 0, a)'$ with $a = \\Phi^{-1}(3/4)$, and the cells $X_i = -1, 0, 1$ have $8$ observations each, with $3$, $2$ and $7$ successes.\n\\begin{audittable}{llll}{The substituted equation and the likelihood score at the same parameter value.}\nModel & $\\theta$ & $\\sum_i W_i'\\{Y_i - \\mu(W_i\\theta)\\}$ & Likelihood score \\\\\n$\\mu(\\eta) = \\eta/(1+\\eta)$ & $(1, 0, 1)'$ & $(0, 0, 0)'$ & $(2/3,\\; 1/3,\\; -2/3)'$ \\\\\nProbit & $(0, 0, a)'$ & $(0, 0, 0)'$ & $(4s,\\; 2s,\\; 0)'$ with $s = h(a) - h(0)$ \\\\\n\\end{audittable}\nIn the probit example the treatment coordinate $2s$ of the score exceeds $13/625$, by a bound obtained in rational arithmetic, and its value is about $0.198$. A point at which the score is not zero is not an ML estimate.\n\n\\emph{The first-order term.} Fix $p$ in $(0,1)$ and let $\\eta_0 = \\mu^{-1}(p)$, $c = \\mu'(\\eta_0)$, $h_0 = h(\\eta_0)$ and $h_1 = h'(\\eta_0)$. Writing $\\eta_i = \\eta_0 + \\delta_i$ and expanding both factors, I obtain\n\\[\nh(\\eta_i)\\{Y_i - \\mu(\\eta_i)\\} = h_0 (Y_i - p) - h_0 c\\, \\delta_i + h_1 (Y_i - p)\\, \\delta_i + O(\\delta_i^2) .\n\\]\nThe substitution keeps the first two terms and drops the third, which is of first order in $\\delta_i$. Note that $h_1 = 0$ for the logit.\n\n\\emph{Its effect on the coefficient.} Let $X_i$ take the values $-1$, $0$ and $1$ with probability $1/3$ each, let $D_i$ be binary with $\\Pr[D_i = 1 \\mid X_i] = 1/2 + X_i/4$, and let $\\mathrm{E}[Y_i \\mid D_i, X_i] = q_0(X_i) + \\varepsilon (D_i - 1/2 - X_i/4)$, where $q_0(-1) = q_0(1) = p + 1/16$ and $q_0(0) = p - 1/8$. The mean of $Y_i$ is $p$. Potential outcomes with this conditional mean can be chosen such that unconfoundedness holds; the treatment is binary, and $\\mathrm{E}[D_i \\mid X_i]$ is linear in $X_i$. The OLS coefficient on $D_i$ is exactly $\\varepsilon$. If the substitution were valid, the ML coefficient on $D_i$ would then be $\\varepsilon/c$ to first order. Differentiating the equation that sets the likelihood score to zero, I find that the derivative of the ML coefficient at $\\varepsilon = 0$ is $1/(c + k/80)$, where $k = h_1/h_0$.\n\\begin{audittable}{llll}{Derivative at $\\varepsilon = 0$ of the ML coefficient on $D_i$: the value implied by the substitution and the value from the likelihood score.}\nModel & $p$ & Substitution: $1/c$ & Likelihood score: $1/(c + k/80)$ \\\\\n$\\mu(\\eta) = \\eta^2$, $0<\\eta<1$ & $1/4$ & $1$ & $120/119$ \\\\\nProbit & $\\Phi(-1)$ & in $[4.132731354122,\\, 4.132731354123]$ & in $[4.184073577247,\\, 4.184073577248]$ \\\\\n\\end{audittable}\nHence the ML coefficient differs at first order in $\\varepsilon$ from the value implied by the substitution. With the same $p$ and with $q_0$ constant, the derivative equals $1/c$. For these models the first-order scale is therefore not a function of the mean of $Y_i$ alone. If $\\varepsilon$ is replaced by $\\varepsilon\\tau(X_i)$ in this example, the relative weights on the effects $\\tau(X_i)$ in the derivative still follow $\\operatorname{Var}[D_i \\mid X_i]$. The example shows that the scaling fails, and it does not show different relative weights.", "problem": "Appendix~C derives its result for logistic regression in two steps. The ML estimate solves the estimating equation displayed above. Replacing $\\operatorname{logit}^{-1}$ by its first-order Taylor polynomial about $\\operatorname{logit}(P_Y)$ then turns this equation into the OLS normal equations, up to the scale factor $c = P_Y(1-P_Y)$. The appendix says that the result holds for probit, Poisson, ordered, multinomial and duration models as well, and that it follows by substituting the predicted mean or inverse link of the model into the estimating equation.\n\nThe first step fails in general. For a binary outcome with mean function $\\mu$ and linear predictor $\\eta_i = W_i\\theta$, the likelihood score is $\\sum_i W_i\\, h(\\eta_i)\\{Y_i - \\mu(\\eta_i)\\}$ with $h = \\mu'/[\\mu(1-\\mu)]$. For the logit, $h$ is identically $1$ and the printed equation is the likelihood score. For probit, $h(\\eta) = \\phi(\\eta)/[\\Phi(\\eta)\\{1-\\Phi(\\eta)\\}]$ varies with $\\eta$ and cannot be taken out of the sum. The substituted equation is then not the probit score, and a root need not be the probit ML estimate. The scores of ordered, multinomial and duration models have other forms again. For example, the Cox partial-likelihood score involves the risk sets, and it cannot be obtained by substituting a scalar inverse link.\n\nThe second step changes as well. A first-order expansion of the probit score must expand $h$ along with $\\Phi$, which leaves a first-order term that involves the outcome and has no counterpart in the OLS equations. This term vanishes for probit when the expansion point is $P_Y = 1/2$. It can cancel in particular designs, and it can be of higher order under further local assumptions, none of which the appendix states. The appendix therefore establishes the first-order OLS weighting for logistic regression, and the same argument covers the models whose score is a constant multiple of the substituted equation, such as Poisson regression with the log link. For probit and the other listed models the statement is not established by this argument. I do not show that a local version with suitable qualifications fails for those models.", "what_survives": "Theorem~A.1 and Corollary~A.1, the formulas for the effective sample under OLS, the comparison with the leverage in Appendix~B and the argument about external validity do not use this extension. Both applications compute their weights from least-squares regressions of the treatment on the controls, and none of their reported numbers changes on this account. The logistic score and its first-order reduction to the OLS equations stand, apart from the misprint \\fid{P015-E010} and the notational point \\fid{P015-E011}. The extension narrows to the models whose score is a constant multiple of the substituted equation. For probit, ordered, multinomial and duration models the same weights are not established by Appendix~C and would need a separate argument. The erratum of 2017 does not address this passage."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E008", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of ten lines of the left column. Two red boxes mark the words 'approximation of arbitrary precision', which run over two lines, in the sentence 'We can obtain an approximation of arbitrary precision for the weights by using a Taylor expansion of the inverse link function'.", "caption": "P. 257 (PDF p. 8), left column. The marked words promise an approximation of arbitrary precision; the next sentence fixes the center of the expansion.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T11", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T11"}, {"id": "P015-T12", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T12"}], "headline": "One Taylor expansion of the inverse logit about the logit of the mean outcome does not give an approximation of arbitrary precision for the weights, because the series has a finite radius of convergence and the fitted linear predictors of a logistic regression can lie outside it; what remains is the first-order, local approximation that the paper derives and uses.", "id": "P015-E009", "location": "Section “Maximum Likelihood Estimation and Generalized Linear Models”, p. 257 (PDF p. 8), left column, sentence beginning “We can obtain an approximation”", "pdf_page": 8, "printed_page": "257", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P015-T11", "P015-T12"], "text": {"as_printed": "We can obtain \\hl{an approximation of arbitrary precision} for the weights by using a Taylor expansion of the inverse link function, $\\mathrm{logit}^{-1}(.)$. Suppose we center such an expansion on the logit of the mean of $Y_i$.", "corrected": "\\hl{A first-order} Taylor expansion of the inverse link function, $\\mathrm{logit}^{-1}(.)$, \\hl{provides a local approximation} for the weights. Suppose we center \\hl{this} expansion on the logit of the mean of $Y_i$; \\hl{its accuracy depends on how close the fitted linear predictors are to that center.}\n\nA statement about expansions of increasing order needs the following hypotheses. Let $f(a)=1/[1+\\exp(-a)]$, let $0<P_Y<1$, let $c=\\mathrm{logit}(P_Y)$, and let $T_N$ be the Taylor polynomial of $f$ of degree $N$ at $c$. Suppose that every relevant linear predictor $a$ satisfies $|a-c|\\le r$ for some $r<\\sqrt{c^2+\\pi^2}$. Then for any $\\rho$ with $r<\\rho<\\sqrt{c^2+\\pi^2}$, with $M_\\rho=\\max_{|z-c|=\\rho}|f(z)|$,\n\\[\n\\sup_{|a-c|\\le r}\\,\\bigl|f(a)-T_N(a)\\bigr|\\le M_\\rho\\,\\frac{(r/\\rho)^{N+1}}{1-r/\\rho}\\longrightarrow 0\\quad\\text{as } N\\to\\infty .\n\\]\nThis controls the error in the fitted mean function only. A guarantee for the fitted coefficients or for the weights would need, in addition, a stability argument for the estimating equations, which the paper does not give.", "evidence": "The evidence is an exact counterexample together with a derivation that covers every degree. Consider a logistic regression on an intercept and a binary treatment with 200 observations: 100 have $D_i=0$, of which 1 has $Y_i=1$, and 100 have $D_i=1$, of which 99 have $Y_i=1$. Because both outcomes occur in both cells, the maximum likelihood estimate is unique and finite. It equals $\\hat\\alpha_\\ell=-\\log 99$ and $\\hat\\beta_\\ell=2\\log 99$, with fitted probabilities $1/100$ and $99/100$ and fitted linear predictors $-\\log 99$ and $\\log 99$. The mean of $Y_i$ is $1/2$. Then the center of the expansion is $\\mathrm{logit}(1/2)=0$ and the radius of convergence is $\\pi$. Since $e<11/4$ and $(11/4)^9=2357947691/262144<9801=99^2$, I have $\\log 99>9/2>4>\\pi$. Both fitted linear predictors lie outside the disk of convergence.\n\nFurther, the error is bounded below at every degree. The Taylor series of $f$ at $0$ is\n\\[\nf(a)=\\frac12+\\sum_{m\\ge 1}(-1)^{m-1}b_m a^{2m-1},\\qquad b_m=2\\sum_{k\\ge 0}\\bigl((2k+1)\\pi\\bigr)^{-2m}>0,\n\\]\nwith $b_1=1/4$ and $b_2=1/48$. By the Cauchy--Schwarz inequality, $b_m^2\\le b_{m-1}b_{m+1}$ for $m\\ge 2$, which implies that $b_{m+1}/b_m\\ge b_2/b_1=1/12$ for all $m\\ge 1$. For $a\\ge 9/2$ the terms $t_m=b_ma^{2m-1}$ therefore satisfy $t_{m+1}/t_m\\ge 27/16$. Pairing consecutive terms, I find that at such $a$ every Taylor polynomial with an odd number of nonconstant terms is at least $1/2+a/4\\ge 13/8$, and every one with an even number is at most $1/2+a/4-a^3/48\\le -35/128$. The fitted probability at $a=\\log 99$ is $99/100$, and the constant polynomial equals $1/2$. Hence every Taylor polynomial $T_N$ of $f$ at $0$, of every degree $N\\ge 0$, satisfies\n\\[\n\\bigl|T_N(\\log 99)-\\tfrac{99}{100}\\bigr|\\ge \\tfrac{49}{100},\n\\]\nand by the symmetry $f(-a)=1-f(a)$ the same bound holds at $-\\log 99$. Repeating the data at each value of a binary control $X_i\\in\\{0,1\\}$, I obtain 400 observations, a control coefficient of zero, and the same failure.", "problem": "The sentence says that one Taylor expansion of $f(a)=1/[1+\\exp(-a)]$, centered at the logit of the mean of $Y_i$, can be carried to any accuracy by taking more terms. That holds only at linear predictors inside the radius of convergence of the series. Since $f$ has poles at the odd multiples of $i\\pi$, its Taylor series about a real center $c$ has radius $\\sqrt{c^2+\\pi^2}$, which equals $\\pi$ when $c=0$. A logistic regression can have a design of full rank and a maximum likelihood estimate that is unique and finite, and still have fitted linear predictors that lie farther than this from the center. At such predictors the Taylor polynomials do not approach $f$ as the degree grows. The argument that follows the sentence, and Appendix~C, use the first-order term only.", "what_survives": "The first-order step of Appendix~C is a formal linearization and does not require the series to converge; how accurate it is for given data depends on how far the fitted linear predictors are from the center, which I did not quantify. The OLS weighting result and its derivation in Appendix~A, the comparison with leverage in Appendix~B and Figure~B1, and the two applications, which are fitted by OLS, do not use the sentence. What is lost is the unrestricted promise of arbitrary precision for the nonlinear extension on pp.~256--257, to which the introduction refers on p.~252. The counterexample concerns the single expansion that the sentence specifies; it says nothing about approximations with other centers or by other methods."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E009", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of a short passage from the left column of page 265. The text reads: We may approximate the inverse logit function by taking a first-order Taylor expansion about logit(P_Y) = log[P_Y/(1 - P_Y)], where P_Y = Pr[Y_i = 1]. The display reads: inverse logit of a is approximately P_Y plus the fraction exp[logit(P_Y)] over [1 + logit(P_Y)] squared, times [a - logit(P_y)], which equals b + ca. Red outlines surround the denominator [1 + logit(P_Y)] squared and the symbol P with a lowercase subscript y in the last bracket. Below: where b = P_Y - P_Y(1 - P_Y) logit(P_Y) and the scale factor c = P_Y(1 - P_Y).", "caption": "Appendix C, p. 265 (PDF p. 16), left column: the Taylor expansion of $\\operatorname{logit}^{-1}(a)$ and the definitions of $b$ and $c$. The outlines mark the denominator $[1 + \\operatorname{logit}(P_Y)]^2$ and the symbol $P_y$.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T13", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T13"}], "headline": "In the displayed Taylor expansion the denominator lacks an exponential and should read $[1+\\exp\\{\\operatorname{logit}(P_Y)\\}]^2$, and $P_y$ in the last bracket should read $P_Y$; the constants $b$ and $c$ printed below the display are correct.", "id": "P015-E010", "location": "Appendix C, display of the first-order Taylor expansion of $\\operatorname{logit}^{-1}(a)$, p. 265 (PDF p. 16), left column", "pdf_page": 16, "printed_page": "265", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P015-T13"], "text": {"as_printed": "We may approximate the $\\operatorname{logit}^{-1}(.)$ function by taking a first-order Taylor expansion about $\\operatorname{logit}(P_Y) = \\log[P_Y/(1-P_Y)]$, where $P_Y = \\Pr[Y_i = 1]$:\n\\[\n\\begin{aligned}\n\\operatorname{logit}^{-1}(a) &\\approx P_Y + \\frac{\\exp[\\operatorname{logit}(P_Y)]}{[1 + \\hl{\\operatorname{logit}(P_Y)}]^2}\\,[a - \\operatorname{logit}(\\hl{P_y})]\\\\\n&= b + ca,\n\\end{aligned}\n\\]\nwhere $b = P_Y - P_Y(1-P_Y)\\operatorname{logit}(P_Y)$ and the scale factor $c = P_Y(1-P_Y)$.", "corrected": "With $\\operatorname{logit}^{-1}(a) = [1 + \\exp(-a)]^{-1}$ and $P_Y = \\Pr[Y_i = 1]$ in $(0,1)$, the first-order Taylor expansion about $\\operatorname{logit}(P_Y) = \\log[P_Y/(1-P_Y)]$ is\n\\[\n\\begin{aligned}\n\\operatorname{logit}^{-1}(a) &\\approx P_Y + \\frac{\\exp[\\operatorname{logit}(P_Y)]}{[1 + \\hl{\\exp\\{\\operatorname{logit}(P_Y)\\}}]^2}\\,[a - \\operatorname{logit}(\\hl{P_Y})]\\\\\n&= b + ca,\n\\end{aligned}\n\\]\nwhere $b = P_Y - P_Y(1-P_Y)\\operatorname{logit}(P_Y)$ and the scale factor $c = P_Y(1-P_Y)$.", "evidence": "The evidence is a derivation and a check of the page. Let $g(a) = \\operatorname{logit}^{-1}(a) = [1 + \\exp(-a)]^{-1}$. Differentiating, I have\n\\[\ng'(a) = \\frac{\\exp(-a)}{[1 + \\exp(-a)]^2} = \\frac{\\exp(a)}{[1 + \\exp(a)]^2} .\n\\]\nLet $L = \\operatorname{logit}(P_Y)$, so that $\\exp(L) = P_Y/(1-P_Y)$ and $1 + \\exp(L) = 1/(1-P_Y)$. Substituting these into the derivative, I obtain\n\\[\ng'(L) = \\frac{P_Y}{1-P_Y}\\,(1-P_Y)^2 = P_Y(1-P_Y) = c .\n\\]\nSince $g(L) = P_Y$, the expansion is $g(a) \\approx P_Y + c\\,(a - L) = (P_Y - cL) + ca$, which is $b + ca$ with $b$ as printed. At $P_Y = 1/2$, $L = 0$: the printed fraction is $\\exp(0)/[1+0]^2 = 1$, and $c = 1/4$.", "problem": "The coefficient of $a - \\operatorname{logit}(P_Y)$ in the expansion is the derivative of $\\operatorname{logit}^{-1}$ at $\\operatorname{logit}(P_Y)$. The printed fraction is not this derivative, because its denominator has $\\operatorname{logit}(P_Y)$ where $\\exp[\\operatorname{logit}(P_Y)]$ belongs. At $P_Y = 1/2$ the printed fraction equals $1$, and the derivative equals $1/4$. The last bracket prints $P_y$ for $P_Y$. The constants $b$ and $c$ that follow the display are those of the correct expansion, and both slips are confined to the display.", "what_survives": "The definitions of $b$ and $c$ and the calculation that follows them are unaffected; the notation of that calculation is the subject of \\fid{P015-E011}. Two symbols change in one display."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E010", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the lower part of the left column of page 265. It begins: Using the linearized inverse logit function, we may define approximate estimating equations. The display reads: the sum over i of psi(Y_i, W_i, theta-hat) is approximately the sum over i of psi-tilde(Y_i, W_i, theta-hat), which equals the sum over i of [W_i(Y_i - b - c W_i theta-hat)], which equals the zero vector of size 1 by (K+2). Red outlines surround the summand W_i(Y_i - b - c W_i theta-hat) and the final equality with the zero vector. Below: Without loss of generality, we may reframe the problem by defining an alternative coefficient vector theta-tilde = ((b + c alpha-hat), c beta-hat, c gamma-hat)', so that the sum of psi-tilde equals the sum of [W_i(Y_i - W_i theta-tilde)] equals zero, which are precisely the estimating equations for the OLS estimator.", "caption": "Appendix C, p. 265 (PDF p. 16), left column: the approximate estimating equations and the reparametrization. The outlines mark the linearized summand evaluated at $\\hat\\theta$ and the “$= \\mathbf{0}_{1\\times(K+2)}$” that follows it.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E011.png"}], "formal_release": null, "formal_statements": [], "headline": "The symbol $\\hat\\theta$ is defined as the root of the exact logistic score and is then used for the root of the linearized score; the two roots differ, and the equality with the OLS estimating equations holds for the root of the linearized score.", "id": "P015-E011", "location": "Appendix C, the display introduced by “we may define approximate estimating equations” and the display that follows it, p. 265 (PDF p. 16), left column", "pdf_page": 16, "printed_page": "265", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The ML estimator for $\\hat\\theta$ solves the following estimating equation (Liang and Zeger 1986):\n\\[\n\\sum_{i=1}^{n} \\psi(Y_i, W_i, \\hat\\theta) = \\sum_{i=1}^{n} \\bigl( W_i\\{Y_i - \\operatorname{logit}^{-1}(W_i\\hat\\theta)\\} \\bigr) = \\mathbf{0}_{1\\times(K+2)},\n\\]\n[\\ldots] Using the linearized $\\operatorname{logit}^{-1}(.)$ function, we may define approximate estimating equations:\n\\[\n\\begin{aligned}\n\\sum_{i=1}^{n} \\psi(Y_i, W_i, \\hat\\theta) &\\approx \\sum_{i=1}^{n} \\tilde\\psi(Y_i, W_i, \\hat\\theta)\\\\\n&= \\sum_{i=1}^{n} \\bigl[ W_i(Y_i - b - c W_i \\hl{\\hat\\theta}) \\bigr]\\\\\n&{}\\hl{= \\mathbf{0}_{1\\times(K+2)}}.\n\\end{aligned}\n\\]\nWithout loss of generality, we may reframe the problem by defining an alternative coefficient vector $\\tilde\\theta = ((b + c\\hat\\alpha_\\ell) \\;\\; c\\hat\\beta_\\ell \\;\\; c\\hat\\gamma_\\ell)'$, so that\n\\[\n\\sum_{i=1}^{n} \\tilde\\psi(Y_i, W_i, \\hat\\theta) = \\sum_{i=1}^{n} \\bigl[ W_i(Y_i - W_i\\tilde\\theta) \\bigr] = \\mathbf{0}_{1\\times(K+2)},\n\\]\nwhich are precisely the estimating equations for the OLS estimator.", "corrected": "Let $\\hl{\\hat\\theta_{\\mathrm{lin}}}$ solve the linearized estimating equations\n\\[\n\\sum_{i=1}^{n} \\bigl[ W_i(Y_i - b - c W_i \\hl{\\hat\\theta_{\\mathrm{lin}}}) \\bigr] = \\mathbf{0}_{1\\times(K+2)} ,\n\\]\nand define $\\tilde\\theta = b e_1 + c\\,\\hat\\theta_{\\mathrm{lin}}$, where $e_1 = (1 \\;\\; 0 \\;\\; \\cdots \\;\\; 0)'$. Then $\\sum_{i=1}^{n} [ W_i(Y_i - W_i\\tilde\\theta) ] = \\mathbf{0}_{1\\times(K+2)}$, which are the estimating equations for the OLS estimator. The vector $\\hat\\theta_{\\mathrm{lin}}$ is the first-order surrogate for the ML root $\\hat\\theta$, so that $\\hat\\beta_{\\mathrm{OLS}} \\approx c\\,\\hat\\beta_\\ell$. If the linearized score is evaluated at $\\hat\\theta$ itself, the last equality becomes approximate unless the Taylor remainder is included.", "intended_meaning": "I read the passage as defining a linearized score and solving it. If $\\hat\\theta_{\\mathrm{lin}}$ denotes the root of the linearized score and $e_1 = (1, 0, \\ldots, 0)'$, then $b e_1 + c\\,\\hat\\theta_{\\mathrm{lin}}$ is exactly the OLS estimate, and $\\hat\\theta_{\\mathrm{lin}}$ serves as a first-order surrogate for the ML root. This reading is correct, and the passage itself speaks of ``approximate estimating equations'' and of a ``first-order approximation''. Read literally, $\\hat\\theta$ keeps its definition as the ML root, and the ML root does not make the linearized score zero. For example, suppose that 2 of 10 controls and 8 of 10 treated units are successes, and that the regressors are a constant and $D_i$. Then the ML root is $(-\\log 4,\\, 2\\log 4)$ and the linearized score at this root is $(0,\\, 3 - \\log 32)$. The root of the linearized score is $(-6/5,\\, 12/5)$. The OLS slope is $3/5$, and $c$ times the ML slope is $(\\log 4)/2$, about $0.693$."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E011", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the foot of the left column of page 264. It begins: Now we consider the leverage. The leverage for unit i is h_ii = W_i (W'W)^(-1) W_i', where W_i and W are defined as in Appendix A. After normalizing by n, straightforward mathematical calculations obtain. Four displayed lines follow: E[n h_ii given D_i = 0, X_i = 0] = 7.43; E[n h_ii given D_i = 1, X_i = 0] = 4.21; E[n h_ii given D_i = 0, X_i = 1] = 1.71; E[n h_ii given D_i = 1, X_i = 1] = 4.21. Then: Applying the law of iterated expectations, E[n h_ii given X_i = 0] = 4.86, E[n h_ii given X_i = 1] = 2.38. Red outlines surround the left-hand sides of the six displayed equations, that is, the six conditional expectations of n h_ii.", "caption": "Appendix B, p. 264 (PDF p. 15), left column: the definition of the leverage and the six displayed values. The outlines mark the six conditional expectations of $n \\cdot h_{ii}$.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E012.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T20", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T20"}], "headline": "The six values printed as conditional expectations of $n \\cdot h_{ii}$ are population values of the scaled leverage, or exact values for a fixed design with the stated proportions; they are not expectations of the leverage in a random sample of size $n$, and all six values are correct.", "id": "P015-E012", "location": "Appendix B, the displays that follow “Now we consider the leverage”, p. 264 (PDF p. 15), left column", "pdf_page": 15, "printed_page": "264", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P015-T20"], "text": {"as_printed": "The leverage for unit $i$ is\n\\[\nh_{ii} = W_i(\\mathbf{W}'\\mathbf{W})^{-1}W_i',\n\\]\nwhere $W_i$ and $\\mathbf{W}$ are defined as in Appendix A. After normalizing by $n$, straightforward mathematical calculations obtain\n\\[\n\\begin{aligned}\n{}\\hl{\\mathrm{E}[n \\cdot h_{ii} \\mid D_i = 0, X_i = 0]} &= 7.43,\\\\\n{}\\hl{\\mathrm{E}[n \\cdot h_{ii} \\mid D_i = 1, X_i = 0]} &= 4.21,\\\\\n{}\\hl{\\mathrm{E}[n \\cdot h_{ii} \\mid D_i = 0, X_i = 1]} &= 1.71,\\\\\n{}\\hl{\\mathrm{E}[n \\cdot h_{ii} \\mid D_i = 1, X_i = 1]} &= 4.21.\n\\end{aligned}\n\\]\nApplying the law of iterated expectations,\n\\[\n\\begin{aligned}\n{}\\hl{\\mathrm{E}[n \\cdot h_{ii} \\mid X_i = 0]} &= 4.86,\\\\\n{}\\hl{\\mathrm{E}[n \\cdot h_{ii} \\mid X_i = 1]} &= 2.38,\n\\end{aligned}\n\\]\nso that units with $X_i = 1$ have, on average, 51\\% \\emph{less} leverage than units with $X_i = 0$ have.", "corrected": "Let $Q = \\mathrm{E}[W_i'W_i]$ and define the population scaled leverage $\\hl{\\ell_i = W_i Q^{-1} W_i'}$, the limit of $n \\cdot h_{ii}$. Straightforward calculations give, to two decimal places,\n\\[\n\\begin{aligned}\n\\mathrm{E}[\\hl{\\ell_i} \\mid D_i = 0, X_i = 0] &= 7.43, & \\mathrm{E}[\\hl{\\ell_i} \\mid D_i = 1, X_i = 0] &= 4.21,\\\\\n\\mathrm{E}[\\hl{\\ell_i} \\mid D_i = 0, X_i = 1] &= 1.71, & \\mathrm{E}[\\hl{\\ell_i} \\mid D_i = 1, X_i = 1] &= 4.21,\n\\end{aligned}\n\\]\nand $\\mathrm{E}[\\hl{\\ell_i} \\mid X_i = 0] = 4.86$, $\\mathrm{E}[\\hl{\\ell_i} \\mid X_i = 1] = 2.38$. These are population values of the scaled leverage, or exact values of $n \\cdot h_{ii}$ for a fixed design with the stated proportions, and not exact expectations for a random design of finite size. The population values and the comparisons of 22\\% and 51\\% remain correct.", "intended_meaning": "The example compares the scaled leverage of the four cells of $(D_i, X_i)$ in the population whose joint distribution is printed on p.~263. With $Q = \\mathrm{E}[W_i'W_i]$, the population scaled leverage is $\\ell_i = W_i Q^{-1} W_i'$. I obtain the exact values $52/7$, $59/14$, $12/7$ and $59/14$ in the four cells, with means $34/7$ at $X_i = 0$ and $50/21$ at $X_i = 1$, and these round to the printed numbers. A fixed design of $n = 20$ units with cell counts $1$, $4$, $11$ and $4$ has the same values of $n \\cdot h_{ii}$ exactly. Since the printed values do not depend on $n$, this reading is clear. Read literally, the notation refers to the leverage in a random sample of size $n$, and the values are not its expectations. For example, for $n = 3$ every sample of full rank has $n \\cdot h_{ii} = 3$ in every cell. The comparisons of 22\\% and 51\\% are correct."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E012", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the display 'Pr[Y 1 = 1] = inverse logit of alpha plus beta D i plus X i gamma', with three lines of text above and three below. A red box marks the symbol Y with subscript 1 on the left side.", "caption": "Display of the logistic regression model, p. 257 (PDF p. 8), left column. The marked outcome carries the subscript $1$.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E013.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T14", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T14"}], "headline": "In the logistic regression model the outcome carries the subscript $i$, not $1$, and the probability is conditional on $D_i$ and $X_i$.", "id": "P015-E013", "location": "Section “Maximum Likelihood Estimation and Generalized Linear Models”, p. 257 (PDF p. 8), left column, display of the logistic regression model", "pdf_page": 8, "printed_page": "257", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P015-T14"], "text": {"as_printed": "\\[\n\\Pr[Y_{\\hl{1}} = 1] = \\mathrm{logit}^{-1}(\\alpha_\\ell + \\beta_\\ell D_i + X_i\\gamma_\\ell),\n\\]", "corrected": "\\[\n\\Pr[Y_{\\hl{i}} = 1 \\hl{\\mid D_i, X_i}] = \\mathrm{logit}^{-1}(\\alpha_\\ell + \\beta_\\ell D_i + X_i\\gamma_\\ell),\n\\]", "evidence": "This is a check of the page. At 300 dpi the subscript is the numeral $1$ and no conditioning is printed. Appendix~C, p.~265, prints $\\Pr[Y_i = 1 \\mid D_i, X_i]$ with the same right side.", "problem": "The left side names the outcome of unit $1$ and conditions on nothing, while the right side depends on the treatment and the covariates of unit $i$. One unconditional probability for unit $1$ cannot in general equal the different values that the right side takes as $i$ varies. The intended model is the conditional model for unit $i$, which Appendix~C prints on p.~265 with the same right side.", "what_survives": "This correction changes no fitted model or numerical result."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E013", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of a four-line reference entry: Angrist, Joshua D., and Alan B. Krueger. 1999. Empirical Strategies in Labor Economics. In Handbook of Labor Economics (Vol. 3), ed. Orley C. Ahsenfelter and David Card. Amsterdam: North Holland, 1277-366. A red outline surrounds the name Ahsenfelter.", "caption": "References, p. 265 (PDF p. 16): the entry Angrist and Krueger (1999). The outline marks “Ahsenfelter”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E014.png"}, {"alt": "Crop of a five-line reference entry: Barron, Reuben M., and David A. Kenny. 1986. The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations. Journal of Personality and Social Psychology 51(6): 1173-82. A red outline surrounds the name Barron at the start of the entry.", "caption": "References, p. 265 (PDF p. 16): the entry Barron and Kenny (1986). The outline marks “Barron”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E014-b.png"}, {"alt": "Crop of footnote 12, six lines: Background characteristics that modify the effect of a treatment are known in the methodology literature as moderators (Barron and Kenny 1986). In other words, they are covariates for which there are nonzero interaction effects with the causal factor. VanderWeele and Robins (2007) discuss challenges for causal inference when studying moderators (which they term effect modifiers). A red outline surrounds the name Barron at the end of the second line.", "caption": "Footnote 12, p. 255 (PDF p. 6). The outline marks “Barron”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E014-c.png"}, {"alt": "Crop of a three-line reference entry: Campbell, Donald T. 1957. Factors Relevant to the Validity of Experiments in Social Settings. Psychology Bulletin 45(4): 297-312. Red outlines surround the word Psychology and the number 45.", "caption": "References, p. 265 (PDF p. 16): the entry Campbell (1957). The outlines mark “Psychology” and the volume number “45”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E014-d.png"}, {"alt": "Crop of a three-line reference entry: Van der Laan, Mark, and Sherri Rose. 2011. Targeted Learning: Causal Inference for Obervational and Experimental Data. New York: Springer. A red outline surrounds the misspelt word Obervational.", "caption": "References, p. 267 (PDF p. 18): the entry Van der Laan and Rose (2011). The outline marks “Obervational”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-E014-e.png"}], "formal_release": null, "formal_statements": [], "headline": "Four reference entries contain misprints: “Ahsenfelter” should read “Ashenfelter”, “Barron” should read “Baron” (also in footnote 12), Campbell (1957) appeared in Psychological Bulletin 54(4), and “Obervational” should read “Observational”.", "id": "P015-E014", "location": "References, p. 265 (PDF p. 16): the entries Angrist and Krueger (1999), Barron and Kenny (1986) and Campbell (1957); footnote 12, p. 255 (PDF p. 6); References, p. 267 (PDF p. 18): the entry Van der Laan and Rose (2011)", "pdf_page": 16, "printed_page": "265", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item p.~265: Angrist, Joshua D., and Alan B. Krueger. 1999. ``Empirical Strategies in Labor Economics.'' In \\emph{Handbook of Labor Economics} (Vol. 3), ed. Orley C. \\hl{Ahsenfelter} and David Card. Amsterdam: North Holland, 1277--366.\n\\item p.~265: \\hl{Barron}, Reuben M., and David A. Kenny. 1986. ``The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations.'' \\emph{Journal of Personality and Social Psychology} 51(6): 1173--82.\n\\item p.~255, footnote~12: Background characteristics that modify the effect of a treatment are known in the methodology literature as ``moderators'' (\\hl{Barron} and Kenny 1986).\n\\item p.~265: Campbell, Donald T. 1957. ``Factors Relevant to the Validity of Experiments in Social Settings.'' \\emph{\\hl{Psychology} Bulletin} \\hl{45}(4): 297--312.\n\\item p.~267: Van der Laan, Mark, and Sherri Rose. 2011. \\emph{Targeted Learning: Causal Inference for \\hl{Obervational} and Experimental Data.} New York: Springer.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Angrist, Joshua D., and Alan B. Krueger. 1999. ``Empirical Strategies in Labor Economics.'' In \\emph{Handbook of Labor Economics} (Vol. 3), ed. Orley C. \\hl{Ashenfelter} and David Card. Amsterdam: North Holland, 1277--366.\n\\item \\hl{Baron}, Reuben M., and David A. Kenny. 1986. ``The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations.'' \\emph{Journal of Personality and Social Psychology} 51(6): 1173--82.\n\\item Footnote~12: \\ldots known in the methodology literature as ``moderators'' (\\hl{Baron} and Kenny 1986).\n\\item Campbell, Donald T. 1957. ``Factors Relevant to the Validity of Experiments in Social Settings.'' \\emph{\\hl{Psychological} Bulletin} \\hl{54}(4): 297--312.\n\\item Van der Laan, Mark, and Sherri Rose. 2011. \\emph{Targeted Learning: Causal Inference for \\hl{Observational} and Experimental Data.} New York: Springer.\n\\end{itemize}", "evidence": "I checked the pages against the publishers' pages and the entries of the National Library of Medicine. The PubMed entry for Campbell (1957) reads ``1957 Jul;54(4):297-312'' (\\href{https://pubmed.ncbi.nlm.nih.gov/13465924/}{PubMed 13465924}), and the NLM Catalog has the title of the journal as ``Psychological bulletin''. The PubMed entry for the article of 1986 lists the authors as R M Baron and D A Kenny (\\href{https://pubmed.ncbi.nlm.nih.gov/3806354/}{PubMed 3806354}). Elsevier lists the editors of volume~3 of the \\emph{Handbook of Labor Economics} as Orley Ashenfelter and David Card (\\href{https://shop.elsevier.com/books/handbook-of-labor-economics/ashenfelter/978-0-444-82289-5}{ISBN 978-0-444-82289-5}). On Springer's page the title is \\emph{Targeted Learning: Causal Inference for Observational and Experimental Data} (\\href{https://link.springer.com/book/10.1007/978-1-4419-9782-1}{doi:10.1007/978-1-4419-9782-1}), and the paper's own entry for Petersen et al.\\ (2011) on p.~266 prints the same title with ``Observational''.", "problem": "Two names, a journal title with its volume number, and a word of a book title are misprinted in the reference list, and one of the names recurs in footnote~12. Each cited work can still be located.", "what_survives": "Nothing in the argument, the formulas or the numbers depends on these entries. Five passages change: four reference entries and footnote~12."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-E014", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Nine lines of the left column of page 253. A red box marks the words The only way, which open the sentence: The only way that our assumptions could be more restrictive would be to assume either that causal effects are constant for all units or that all sources of effect heterogeneity have been modeled.", "caption": "Page 253 (PDF p. 4), left column. The marked words open the sentence that names two stronger assumptions.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The sentence that begins “The only way” names two stronger assumptions under which regression recovers the average effect; they are examples, and not the only such assumptions.", "id": "P015-N003", "location": "p. 253 (PDF p. 4), left column, first full paragraph", "pdf_page": 4, "printed_page": "253", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\hl{The only way} that our assumptions could be more restrictive would be to assume either that causal effects are constant for all units or that all sources of effect heterogeneity have been modeled.", "corrected": "\\hl{Two ways to make} our assumptions more restrictive would be to assume either that causal effects are constant for all units or that all sources of effect heterogeneity have been modeled.", "intended_meaning": "The paragraph argues that the assumptions of the paper are as favourable to regression as possible, and the sentence names two stronger assumptions under which the unequal weights no longer matter. Read in this way the sentence is correct, and no result uses it. Taken literally, ``the only way'' says too much. Under Conditions (1)--(3), Expressions (7) and (8) imply that the regression coefficient converges to $\\mathrm{E}[\\tau_i]+\\operatorname{Cov}(v(X_i),t(X_i))/\\mathrm{E}[v(X_i)]$, where $v(X_i)=\\operatorname{Var}[D_i\\mid X_i]$ and $t(X_i)=\\mathrm{E}[\\tau_i\\mid X_i]$. The limit is the average effect whenever this covariance is zero, for instance when the conditional variance of the treatment is constant, although the effects vary and are not modeled. In an exact example with $\\operatorname{Var}[D_i\\mid X_i]=2/3$ for every value of $X_i$, the regression coefficient and the average effect both equal $7/5$."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-N003", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Seven lines of the right column of page 254, followed by Expression (7). A red box marks the words Conditions (1) and (3) in the sentence: Applying results from partial regression (Greene 2008, 27 to 29) along with Conditions (1) and (3), multiple regression generates a weighted average of causal effects of the form. Expression (7) below states that beta hat converges in probability to the ratio of E of w i tau i to E of w i, where w i is the square of D i minus E of D i given X i.", "caption": "Page 254 (PDF p. 5), right column: the sentence that introduces Expression (7). The marked cross-reference omits Condition (2).", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N004.png"}, {"alt": "Seven lines of the left column of page 256. A red box marks the word density in the phrase: consistent estimation of each of the covariate cumulative density functions in the effective sample.", "caption": "Page 256 (PDF p. 7), left column. The marked word should be “distribution”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N004-b.png"}, {"alt": "The sentence The OLS estimates may be written as, followed by a display: a column vector with entries alpha hat, beta hat, gamma hat 1, vertical dots and gamma hat k equals the inverse of W prime W times W prime Y. A red box marks the last entry, gamma hat with a lower-case subscript k.", "caption": "Page 262 (PDF p. 13), right column, proof of Theorem A.1. The marked entry carries the subscript $k$; the number of covariates is $K$.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N004-c.png"}, {"alt": "A four-line display from the proof of Theorem A.1, which shows that E of D i Y 0 i minus E of V i omega Y 0 i equals zero. A red box marks its equation number, (15). The next sentence reads: We now consider the second term in the right-hand side of Expression (13), where.", "caption": "Page 263 (PDF p. 14), left column, proof of Theorem A.1. The marked number follows display (13), which is higher on the same page; no display is numbered (14).", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N004-d.png"}], "formal_release": null, "formal_statements": [{"id": "P015-T15", "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-T15"}], "headline": "Four small slips: the derivation of Expression (7) uses Conditions (1)–(3), not only Conditions (1) and (3); the last entry of the OLS coefficient vector on p. 262 is $\\hat\\gamma_K$; the display numbered (15) on p. 263 should be numbered (14); and the covariate functions on p. 256 are cumulative distribution functions.", "id": "P015-N004", "location": "p. 254 (PDF p. 5), sentence before Expression (7); p. 256 (PDF p. 7), left column; p. 262 (PDF p. 13), display after “The OLS estimates may be written as”; p. 263 (PDF p. 14), display numbered (15)", "pdf_page": 5, "printed_page": "254", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P015-T15"], "text": {"as_printed": "Page 254: Applying results from partial regression (Greene 2008, 27--29) along with \\hl{Conditions (1) and (3)}, multiple regression generates a weighted average of causal effects of the form\n\\[\n\\hat\\beta\\xrightarrow{p}\\frac{\\mathrm{E}[w_i\\tau_i]}{\\mathrm{E}[w_i]},\\ \\text{where } w_i=(D_i-\\mathrm{E}[D_i\\mid X_i])^2.\\qquad(7)\n\\]\n\nPage 256: Similar calculations---redefining $Z_i$ appropriately---permit consistent estimation of each of the covariate cumulative \\hl{density} functions in the effective sample.$^{14}$\n\nPage 262: The OLS estimates may be written as\n\\[\n\\begin{pmatrix}\\hat\\alpha\\\\ \\hat\\beta\\\\ \\hat\\gamma_1\\\\ \\vdots\\\\ \\hat\\gamma_{\\hl{k}}\\end{pmatrix}=(\\mathbf{W}'\\mathbf{W})^{-1}\\mathbf{W}'Y.\n\\]\n\nPage 263:\n\\[\n\\begin{aligned}\n\\mathrm{E}[D_iY_{0i}]-\\mathrm{E}[(V_i\\omega)Y_{0i}]&=\\mathrm{E}_X[\\mathrm{E}[D_iY_{0i}\\mid X_i]]-\\mathrm{E}_X[(V_i\\omega)\\mathrm{E}[Y_{0i}\\mid X_i]]\\\\\n&=\\mathrm{E}_X[\\mathrm{E}[D_i\\mid X_i]\\mathrm{E}[Y_{0i}\\mid X_i]]-\\mathrm{E}_X[(V_i\\omega)\\mathrm{E}[Y_{0i}\\mid X_i]]=0.\\qquad\\hl{(15)}\n\\end{aligned}\n\\]", "corrected": "Page 254: Applying results from partial regression (Greene 2008, 27--29) along with \\hl{Conditions (1)--(3)}, multiple regression generates a weighted average of causal effects of the form\n\\[\n\\hat\\beta\\xrightarrow{p}\\frac{\\mathrm{E}[w_i\\tau_i]}{\\mathrm{E}[w_i]},\\ \\text{where } w_i=(D_i-\\mathrm{E}[D_i\\mid X_i])^2.\\qquad(7)\n\\]\nHere Condition (1) is $Y_i(d)=Y_i(0)+\\tau_i\\cdot d$, Condition (2) is $(Y_i(0),\\tau_i)\\perp\\!\\!\\!\\perp D_i\\mid X_i$, and Condition (3) is $\\mathrm{E}[D_i\\mid X_i]=V_i\\omega$ with $V_i=(1\\ \\ X_i)$. Theorem A.1 states the result under these three conditions and the regularity condition that the OLS solutions converge to a limit.\n\nPage 256: Similar calculations---redefining $Z_i$ appropriately---permit consistent estimation of each of the covariate cumulative \\hl{distribution} functions in the effective sample.$^{14}$\n\nPage 262: The OLS estimates may be written as\n\\[\n\\begin{pmatrix}\\hat\\alpha\\\\ \\hat\\beta\\\\ \\hat\\gamma_1\\\\ \\vdots\\\\ \\hat\\gamma_{\\hl{K}}\\end{pmatrix}=(\\mathbf{W}'\\mathbf{W})^{-1}\\mathbf{W}'Y.\n\\]\n\nPage 263: the display that ends with ``$=0.$'' is numbered \\hl{(14)}.", "evidence": "The evidence is a check of the pages and, for the cross-reference, a derivation. In the proof on p.~263, the term of Expression (13) that involves the baseline outcome has the limit $(\\mathrm{E}[D_iY_{0i}]-\\mathrm{E}[(V_i\\omega)Y_{0i}])/\\mathrm{E}[w_i]$. By Condition (2) and then Condition (3), I have\n\\[\n\\mathrm{E}[D_iY_{0i}\\mid X_i]=\\mathrm{E}[D_i\\mid X_i]\\,\\mathrm{E}[Y_{0i}\\mid X_i]=(V_i\\omega)\\,\\mathrm{E}[Y_{0i}\\mid X_i],\n\\]\nand the numerator of this limit is zero. For the second term of Expression (13), the same two conditions imply that\n\\[\n\\mathrm{E}[(D_i-V_i\\omega)\\tau_i\\mid X_i]=\\mathrm{E}[D_i-V_i\\omega\\mid X_i]\\,\\mathrm{E}[\\tau_i\\mid X_i]=0,\n\\]\nand this term vanishes in the limit. Conditions (1) and (3) alone do not give either factorization. The proof in the paper names unconfoundedness at both steps.", "problem": "The sentence before Expression (7) cites Conditions (1) and (3) as the conditions that the result needs, but the derivation also uses Condition (2), unconfoundedness, which the proof of Theorem A.1 on p.~263 invokes twice. The three other slips are typographical. In the OLS coefficient vector on p.~262 the last entry carries the subscript $k$, while the number of covariates is $K$ everywhere else; the numbered displays of Appendix A run (12), (13), (15), with no (14); and p.~256 has ``cumulative density functions'' for cumulative distribution functions.", "what_survives": "Nothing else changes. Conditions (1)--(3) are stated on p.~253 and Theorem A.1 lists unconfoundedness among its hypotheses. The correction adds no assumption, and Expression (7) stands as printed. Two neighbouring points of notation are not errors. The symbol $\\gamma$ denotes the intercept in Model (5) and the coefficient vector on $X_i$ in Model (6), and each use is defined where it appears. The symbol $Y_{0i}$ in Theorem A.1 denotes the baseline outcome $Y_i(0)$ of the main text."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-N004", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of seven lines of the left column. A red box marks the words 'FDI variable' in the sentence 'We first regress the FDI variable on the various control variables (including the country and decade fixed effects) and then extract the residuals'.", "caption": "P. 258 (PDF p. 9), left column, paragraph beginning “Of course”. The marked words name the outcome where the treatment is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "The weights come from regressing the regime type variable, not the FDI variable, on the control variables; the published erratum of 2017 already makes this correction.", "id": "P015-N008", "location": "Section “Effects of Regime Type on FDI”, p. 258 (PDF p. 9), left column, paragraph beginning “Of course”", "pdf_page": 9, "printed_page": "258", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We first regress the \\hl{FDI} variable on the various control variables (including the country and decade fixed effects) and then extract the residuals.", "corrected": "We first regress the \\hl{regime type} variable on the various control variables (including the country and decade fixed effects) and then extract the residuals.", "evidence": "This is a check of the page against Expression~(7), against the replication script, which regresses \\texttt{regime} on the controls and squares the residuals, and against the erratum (\\emph{American Journal of Political Science} 61(2), April 2017, p.~504), which prints the replacement sentence.", "problem": "FDI is the outcome of the regression. The multiple regression weights are the squared residuals of the treatment, which here is regime type: Expression~(7) on p.~254 defines $w_i=(D_i-\\mathrm{E}[D_i\\mid X_i])^2$, and the sentences before this one speak of the unexplained variation in regime type.", "what_survives": "Since the weights were computed from the regime type variable, no number changes. The correction does not bear on how the weights are then aggregated by country, which is the subject of \\fid{P015-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-N008", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the top of page 260 with the title of Table 1 and its column headings. The title reads: Table 1, Summary Statistics for the Nominal and Effective Samples Relevant to Model (4) in Table 4A of Gerber and Huber Gerber and Huber (2010). A red outline surrounds the second occurrence of the words Gerber and Huber. Below the title are the headings Nominal Sample and Effective Sample, each with the columns Mean and S.D., and the heading Variable.", "caption": "Table 1, p. 260 (PDF p. 11): the title and the column headings. The outline marks the second “Gerber and Huber”.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N010.png"}], "formal_release": null, "formal_statements": [], "headline": "The title of Table 1 prints “Gerber and Huber” twice; it should end “of Gerber and Huber (2010)”, as the published erratum of 2017 states.", "id": "P015-N010", "location": "Table 1, title, p. 260 (PDF p. 11)", "pdf_page": 11, "printed_page": "260", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "TABLE 1 Summary Statistics for the Nominal and Effective Samples Relevant to Model (4) in Table 4A of Gerber and Huber \\hl{Gerber and Huber} (2010)", "corrected": "TABLE 1 Summary Statistics for the Nominal and Effective Samples Relevant to Model (4) in Table 4A \\hl{of Gerber and Huber (2010)}", "evidence": "I checked the page and the published erratum (\\emph{American Journal of Political Science} 61(2), April 2017, p.~504), which notes an error in the title for Table~1 and says that it should simply read ``\\ldots of Gerber and Huber (2010).''", "problem": "The names of the authors of the study are printed twice in succession. The table concerns Model (4) in Table 4A of one study, Gerber and Huber (2010).", "what_survives": "No entry of Table~1 changes, and the erratum corrects none. Nothing else in the paper depends on the title."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-N010", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first eight lines of text and one displayed line at the top of the left column of page 261. The text before the display ends: Formally, the positivity assumption is as follows. A displayed line follows, reading Pr[D_i = d given X_i = x] greater than 0, Pr[D_i = d-prime given X_i = x] greater than 0, and a red outline surrounds this display. Below it: for all values of x in the target population represented by the nominal sample.", "caption": "p. 261 (PDF p. 12), top of the left column. The outline marks the display of the positivity assumption.", "url": "https://pmaronow.github.io/llm-audit/works/P015/exhibits/P015-N011.png"}], "formal_release": null, "formal_statements": [], "headline": "The positivity display uses point probabilities for a discrete treatment; for a continuous treatment, both chosen values must lie in the support of the conditional treatment distribution.", "id": "P015-N011", "location": "Section “Estimating Representative Causal Effects”, display of the positivity assumption, p. 261 (PDF p. 12), left column", "pdf_page": 12, "printed_page": "261", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Formally, the positivity assumption is as follows:\n\\[\n\\hl{\\Pr[D_i = d \\mid X_i = x]} > 0, \\qquad \\hl{\\Pr[D_i = d' \\mid X_i = x]} > 0\n\\]\nfor all values of $x$ in the target population represented by the nominal sample.", "corrected": "For a discrete treatment, $\\Pr[D_i = d \\mid X_i = x] > 0$ and $\\Pr[D_i = d' \\mid X_i = x] > 0$ for all values of $x$ in the target population represented by the nominal sample. \\hl{For a continuous treatment, require both $d$ and $d'$ to be in the support of the conditional treatment distribution: for every $\\varepsilon>0$, $\\Pr[|D_i-d|<\\varepsilon\\mid X_i=x]>0$ and $\\Pr[|D_i-d'|<\\varepsilon\\mid X_i=x]>0$ for all such $x$.} A conditional density version continuous and positive at both target values is sufficient for this support condition. Pointwise positivity of an unspecified density version is insufficient. Identification at the two points also requires suitable regularity of the response in the treatment value.", "intended_meaning": "I read the condition as requiring both chosen treatment values to be available at every covariate profile of the target population, as the display states for a discrete treatment. The paper also allows a continuous treatment (pp.~252--253), and then every point probability is zero even when both values are available. For example, if $D_i$ is uniform on $(0,1)$ and independent of $X_i$, then $\\Pr[D_i = d \\mid X_i = x] = 0$ for every $d\\in(0,1)$, while the usual conditional density equals $1$.\n\nFor a continuous treatment, availability means that every open neighborhood of each chosen value has positive conditional probability at the relevant covariate profile. This is a condition on the conditional treatment law. Pointwise positivity of an unspecified density version is insufficient: changing a density at isolated points leaves the law unchanged. In the uniform example, a density can be set to $1$ at the isolated value $2$ without changing the law, although $2$ remains outside the support. A conditional density version continuous and positive at both target values is sufficient for the neighborhood condition. Identification at the two points also requires suitable regularity of the response in the treatment value. The discrete-treatment discussion and the main linear regression theorem are unaffected by this notation."}, "url": "https://pmaronow.github.io/llm-audit/works/P015/#P015-N011", "work": {"arxiv_id": null, "doi": "10.1111/ajps.12185", "title": "Does Regression Produce Representative Estimates of Causal Effects?", "url": "https://pmaronow.github.io/llm-audit/works/P015/"}, "work_id": "P015"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Eight lines of text from page 396 that give steps (2), (3) and (4) of the testing procedure. A red box marks the expression T(Y_i, Z_i) in the words 'comparing the observed test statistic, T(Y_i, Z_i) against the distribution of that test statistic'.", "caption": "Steps (2) to (4) of the testing procedure, p. 396. The mark shows the observed test statistic of step (4), printed with the raw outcome $Y_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T7", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T7"}, {"id": "P016-T8", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T8"}], "headline": "In step (4) the observed test statistic is $\\mathcal{T}(\\tilde y_{i,Z_i=0},Z_i)$, computed from the outcome adjusted under the hypothesis, and not $\\mathcal{T}(Y_i,Z_i)$.", "id": "P016-E001", "location": "Section 1, second paragraph, step (4), p. 396 (PDF p. 2)", "pdf_page": 2, "printed_page": "396", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P016-T7", "P016-T8"], "text": {"as_printed": "(2) using this result to adjust the observed outcome to represent what would be implied by the hypothesis for a given $\\tau_0$ such that $\\tilde y_{i,Z_i=0}=Y_i-Z_i\\tau_0$; and (3) under the hypothesis, $\\tilde y_{i,Z_i=0}$ should have no systematic relationship with treatment assignment, so we can summarize this relationship with a test statistic, $\\mathcal{T}(\\tilde y_{i,Z_i=0},Z_i)$. A distribution of values for this test statistic arises from repetitions of the treatment assignment process (new draws of $\\mathbf{z}$ from all of the ways that such treatment assignment vectors could have been produced); and finally (4) a $p$ value arises by comparing the observed test statistic, $\\hl{\\mathcal{T}(Y_i,Z_i)}$ against the distribution of that test statistic that characterizes the hypothesis.", "corrected": "(4) a $p$ value arises by comparing the observed test statistic, $\\hl{\\mathcal{T}(\\tilde y_{i,Z_i=0},Z_i)}$, evaluated at the realized assignment, against the distribution of that test statistic that characterizes the hypothesis.\n\nIn full: let $Y$ be the vector of observed outcomes, $Z$ the observed assignment and $u_i=Y_i-\\tau_0Z_i$ the outcome adjusted under the hypothesized additive effect $\\tau_0$. Compare the observed $\\mathcal{T}(u,Z)$ with the distribution of $\\mathcal{T}(u,Z')$, where $Z'$ follows the experimental assignment law and $u$ is held fixed.", "evidence": "The evidence is a derivation, an exact counterexample to the rule as printed, and a check of the replication code.\n\nLet $Z$ be the realized assignment and $Y$ the vector of observed outcomes. Suppose that the hypothesis $y_{i,Z_i=1}=y_{i,Z_i=0}+\\tau_0$ holds, with no interference. Inserting it into the identity $Y_i=Z_iy_{i,Z_i=1}+(1-Z_i)y_{i,Z_i=0}$, I obtain $Y_i=y_{i,Z_i=0}+\\tau_0Z_i$. This implies that $u_i=Y_i-\\tau_0Z_i$ equals $y_{i,Z_i=0}$, which does not depend on the assignment. Under the hypothesis, the statistic at the realized assignment is $\\mathcal{T}(u,Z)$, and its randomization distribution is that of $\\mathcal{T}(u,Z')$, where $Z'$ follows the assignment law of the experiment and $u$ is held fixed. For the difference in means, I have $\\mathcal{T}(Y,Z)=\\mathcal{T}(u,Z)+\\tau_0$.\n\nFor the counterexample, I take four units, two of them treated, with the six assignments equally likely, $u=(0,1,2,3)$, true effect $\\tau_0=10$, and the difference in means as the statistic. The six values of $\\mathcal{T}(u,Z')$ are $-2,-1,0,0,1,2$. At $Z=(0,0,1,1)$ the observed outcomes are $Y=(0,1,12,13)$, the raw statistic is $12$, and the adjusted statistic is $2$. The two-sided $p$ value is the proportion of the six values of $\\mathcal{T}(u,Z')$ that are at least as large in absolute value as the observed statistic. It is $0$ for the raw statistic and $1/3$ for the adjusted statistic. Under each of the six assignments the raw statistic lies between $8$ and $12$, and its $p$ value is $0$. The rule as printed rejects the true hypothesis $\\tau_0=10$ with probability one.\n\nThe randomization routine of the replication code adjusts the outcome once, evaluates the observed statistic on the adjusted outcome, and compares it with the statistic evaluated on the same adjusted outcome under the sampled assignments.", "problem": "Steps (2) and (3) define the adjusted outcome $\\tilde y_{i,Z_i=0}=Y_i-Z_i\\tau_0$ and build the randomization distribution from it. Step (4) then writes the observed statistic with the raw outcome $Y_i$. In general $\\mathcal{T}(Y,Z)$ differs from $\\mathcal{T}(\\tilde y,Z)$; the two coincide when $\\tau_0=0$ or when the statistic is unchanged by the adjustment. As printed, the observed statistic is then compared with the distribution of a different quantity. Steps (2) and (3) show which statistic is meant, and the replication code computes it.", "what_survives": "The procedure of steps (1) to (3) and the inspected randomization routine are unchanged. I found no reported result that depends on the raw statistic. Only the symbol in step (4) changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E001", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Footnotes 2, 3 and 4 of page 396. In footnote 3, a red box marks the bracketed expression F_1(y_i,0) - F_0(y_i,0) that follows 'max over i = 1,...,n' in the definition of the KS test statistic.", "caption": "Footnotes 2 to 4, p. 396. The mark shows the difference of the two ECDFs in the KS statistic of footnote 3, printed in square brackets without an absolute value.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E002.png"}, {"alt": "Four lines of text followed by displayed equation (1), T(y_0, z)_SSR defined as the sum over i of (y-tilde_i,Zi=0 minus y-tilde-bar_i,Zi=0) squared. Red boxes mark the words 'sums the difference' in the text and the overbarred term y-tilde-bar_i,Zi=0 in the equation.", "caption": "The sentence that introduces equation (1), and equation (1), p. 397. The marks show the words “sums the difference” and the mean that is subtracted in the summand.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E002-b.png"}, {"alt": "Four lines of text followed by displayed equation (2), T(y_0, z)_SSR defined as the sum over i of (y-tilde_i,Zi=0 minus X beta-hat) squared. Red boxes mark the words 'Z_i, z-transpose S, and 1-transpose S into a matrix X,' in the text and the term X beta-hat in the equation.", "caption": "The sentence that introduces equation (2), and equation (2), p. 399. The marks show the list of columns of $\\mathbf{X}$, which has no constant, and the fitted vector $\\mathbf{X}\\hat{\\boldsymbol{\\beta}}$, printed without a row index.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E002-c.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T5", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T5"}, {"id": "P016-T9", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T9"}, {"id": "P016-T10", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T10"}], "headline": "The three displayed definitions of the test statistics need the absolute value in the KS statistic, the mean of the unit’s own assignment group in equation (1), and the row index and the intercept in equation (2).", "id": "P016-E002", "location": "Footnote 3, p. 396 (PDF p. 2); equation (1) and the sentence before it, p. 397 (PDF p. 3); equation (2) and the sentence before it, p. 399 (PDF p. 5)", "pdf_page": 2, "printed_page": "396", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P016-T10", "P016-T5", "P016-T9"], "text": {"as_printed": "Footnote 3, p.~396: If the empirical cumulative distribution function (ECDF) of the treated units is $F_1$ and the ECDF of the control units is $F_0$, then the KS test statistic is $\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{KS}}=\\max_{i=1,\\ldots,n}\\hl{[F_1(y_{i,0})-F_0(y_{i,0})]}$, where $F(x)=(1/n)\\sum_{i=1}^{n}I(x_i\\le x)$ records the proportion of the distribution of $x$ at or below $x_i$ (Hollander 1999, section 5.4).\n\np.~397: The simplest version of the sum-of-squared-residual (SSR) test statistic merely \\hl{sums the difference} between the mean of the outcome implied by the hypothesis and individual outcomes, thereby including the correlation between treated and control outcomes as a part of the distribution of the statistic:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(\\tilde y_{i,Z_i=0}-\\hl{\\overline{\\tilde y}_{i,Z_i=0}})^2 \\qquad (1)\n\\]\n\np.~399: If we collected \\hl{$Z_i$, $\\mathbf{z}^{T}\\mathbf{S}$, and $\\mathbf{1}^{T}\\mathbf{S}$} into a matrix $\\mathbf{X}$, and fit the $\\tilde y_{i,Z_i=0}$ as a linear function of $\\mathbf{X}$ with fitted coefficients $\\hat{\\boldsymbol{\\beta}}$ then we could define the test statistic as\n\\[\n\\mathcal{T}(y_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(\\tilde y_{i,Z_i=0}-\\hl{\\mathbf{X}\\hat{\\boldsymbol{\\beta}}})^2. \\qquad (2)\n\\]", "corrected": "Write $u_i=\\tilde y_{i,Z_i=0}$. For $g\\in\\{0,1\\}$, let $n_g=\\#\\{i:z_i=g\\}>0$, let $\\bar u_g=n_g^{-1}\\sum_{i:z_i=g}u_i$, and let $F_g(t)=n_g^{-1}\\sum_{i:z_i=g}I(u_i\\le t)$.\n\nFootnote 3:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{KS}}=\\max_{i=1,\\ldots,n}\\hl{|}F_1(u_i)-F_0(u_i)\\hl{|}=\\sup_t|F_1(t)-F_0(t)|.\n\\]\n\nEquation (1): The simplest version of the sum-of-squared-residual (SSR) test statistic sums the \\hl{squared differences} between individual outcomes and \\hl{the mean of the outcome in the unit's own assignment group}:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(u_i-\\hl{\\bar u_{z_i}})^2. \\qquad (1)\n\\]\n\nEquation (2): Let $\\mathbf{X}(\\mathbf{z})$ be the matrix with rows $x_i^{T}=(\\hl{1},z_i,(\\mathbf{z}^{T}\\mathbf{S})_i,(\\mathbf{1}^{T}\\mathbf{S})_i)$, and let $\\hat{\\boldsymbol{\\beta}}$ be a least squares coefficient vector. Then\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\min_b\\|u-\\mathbf{X}(\\mathbf{z})b\\|_2^2=\\sum_i\\{u_i-\\hl{(\\mathbf{X}(\\mathbf{z})\\hat{\\boldsymbol{\\beta}})_i}\\}^2. \\qquad (2)\n\\]", "evidence": "The evidence is exact arithmetic on small examples, a derivation, and a check of the functions in the replication code on the same examples.\n\nFor the KS statistic, let the adjusted outcomes be $u=(0,1,2,3)$, with $z=(0,0,1,1)$. At the points $0,1,2,3$ the differences $F_1-F_0$ are $-1/2,-1,-1/2,0$. The display as printed gives $0$, and the two-sided statistic is $1$.\n\nFor equation (1), an intercept and $z$ fit one constant $c_g$ to each assignment group $g$. For any constant $c$, I have $\\sum_{i:z_i=g}(u_i-c)^2=\\sum_{i:z_i=g}(u_i-\\bar u_g)^2+n_g(\\bar u_g-c)^2$, where $\\bar u_g$ is the mean of the $n_g$ units of group $g$. The fitted constants are therefore the group means, and the residual sum of squares is $\\sum_i(u_i-\\bar u_{z_i})^2$. With $u=(0,1,2,3)$ and two of the four units treated, the total sum of squares is $5$ under all six assignments, and the residual sum of squares takes the values $1,4,5,5,4,1$. At the assignment that treats units 1 and 2, the lower-tail $p$ value is $1/3$ for the residual sum of squares and $1$ for the total sum of squares.\n\nFor equation (2), take six units with the single edge $\\{1,2\\}$, $z=(1,0,1,0,1,0)$ and $u=(1,1,1,1,1,1)$. The residual sum of squares is $0$ with the intercept and $2$ without it.\n\nOn these inputs the functions of the replication code return $1$ for the KS statistic, $1,4,5,5,4,1$ for the simple SSR statistic and $0$ for the network SSR statistic. These are the values of the corrected definitions.", "problem": "Read literally, the KS display differs from the statistic used, equation (1) leaves its mean undefined, and equation (2) omits the component index of its fitted vector.\n\\begin{enumerate}\n\\item Footnote 3 has no absolute value, and the display is the one-sided statistic $\\max_i\\{F_1-F_0\\}$. The statistic described, which is sensitive to any difference between the two distributions, and the statistic coded are the two-sided $\\max_i|F_1-F_0|$.\n\\item In equation (1) the overbar carries the unit index $i$ and is not defined. The statistic meant is the residual sum of squares from the regression of the adjusted outcome on an intercept and $z$, in which the mean subtracted from unit $i$ is the mean of its own assignment group. If the overbar is read as the mean over all units, the display is the total sum of squares, which does not depend on $\\mathbf{z}$. The sentence before the display says ``sums the difference'' where the squared difference is meant.\n\\item In equation (2) the summand subtracts the whole vector $\\mathbf{X}\\hat{\\boldsymbol{\\beta}}$ from the scalar $\\tilde y_{i,Z_i=0}$, since the row index is missing. The list of columns of $\\mathbf{X}$ has no constant. The regression in the replication code has an intercept.\n\\end{enumerate}\nThe surrounding text on least squares regression identifies the statistics that are meant.", "what_survives": "The inspected functions of the replication code compute the corrected definitions. These notation repairs do not change their outputs, and I found no error in the comparisons of Figs.~2 to 4 caused by these defects. Note that the signed maximum of footnote 3 is a valid one-sided statistic; it is not the statistic used. The power recalculations are described below under what I checked."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E002", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Seven lines of text from page 397, beginning 'example,', followed by displayed equation (1). Red boxes mark the words 'thereby ignoring information about the joint distribution' and the words 'thereby including the correlation between treated and control outcomes as a part of the distribution of the statistic'.", "caption": "The contrast before equation (1), p. 397; the crop begins after “For”. The marks show the statement that the KS test ignores information about the joint distribution, and the statement that the simplest SSR statistic includes the correlation between treated and control outcomes.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T1", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T1"}, {"id": "P016-T2", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T2"}, {"id": "P016-T5", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T5"}], "headline": "The simple SSR statistic of equation (1) uses no correlation between treated and control outcomes that the KS statistic ignores: both are functions of the two group distributions and their sizes, and with fixed adjusted outcomes and group sizes the SSR test is the test based on the absolute difference in group means.", "id": "P016-E004", "location": "Section 2, second paragraph, the two sentences before equation (1), p. 397 (PDF p. 3)", "pdf_page": 3, "printed_page": "397", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P016-T1", "P016-T2", "P016-T5"], "text": {"as_printed": "For example, the KS test uses the maximum difference in the empirical cumulative distributions of each treatment group calculated without regard for the relationship between the treated and control distributions, \\hl{thereby ignoring information about the joint distribution} that could increase the precision of the test. The simplest version of the sum-of-squared-residual (SSR) test statistic merely sums the difference between the mean of the outcome implied by the hypothesis and individual outcomes, \\hl{thereby including the correlation between treated and control outcomes as a part of the distribution of the statistic}:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(\\tilde y_{i,Z_i=0}-\\bar{\\tilde y}_{i,Z_i=0})^2 \\qquad (1)\n\\]", "corrected": "For example, the KS test uses the maximum difference in the empirical cumulative distributions of each treatment group \\del{calculated without regard for the relationship between the treated and control distributions, thereby ignoring information about the joint distribution that could increase the precision of the test}. The simplest version of the sum-of-squared-residual (SSR) test statistic \\hl{is the residual sum of squares from a regression of the adjusted outcome on an intercept and treatment. It is determined by the first two empirical moments and the size of each treatment group. With fixed adjusted outcomes and group sizes, smaller SSR values correspond exactly to larger absolute differences in group means. Its sensitivity differs from that of the KS statistic, but it does not acquire pairing information absent from the two group distributions. Adding fixed-network regressors can use information about the alignment of outcomes with the network. The randomization distribution of either statistic also depends on the assignment design.}", "evidence": "I use a derivation and exact finite examples.\n\nLet $a_i=\\tilde y_{i,Z_i=0}$ be the adjusted outcomes for one hypothesis, held fixed. Let $z\\in\\{0,1\\}^n$ have $n_1>0$ ones and $n_0>0$ zeros, with $n=n_0+n_1$. I write $m_g$ for the mean of $a_i$ over the units with $z_i=g$, $m$ for the mean over all units, and $T(a)=\\sum_i(a_i-m)^2$. For any group $g$ and any constant $c$, I have\n\\[\n\\sum_{i:z_i=g}(a_i-c)^2=\\sum_{i:z_i=g}(a_i-m_g)^2+n_g(m_g-c)^2,\n\\]\nsince the cross term vanishes. The least squares fit on an intercept and $z$ therefore has fitted values $m_0$ and $m_1$, and its residual sum of squares is $R(a,z)=\\sum_g\\sum_{i:z_i=g}(a_i-m_g)^2$. Taking $c=m$ and summing over the two groups, I obtain $T(a)=R(a,z)+n_0(m_0-m)^2+n_1(m_1-m)^2$. Since $m_0-m=-n_1(m_1-m_0)/n$ and $m_1-m=n_0(m_1-m_0)/n$, I conclude that\n\\[\nR(a,z)=T(a)-\\frac{n_0n_1}{n}(m_1-m_0)^2.\n\\]\nThe term $T(a)$ and the factor $n_0n_1/n$ do not depend on the assignment. For any two assignments $z$ and $w$ with these group sizes, I have $R(a,w)\\le R(a,z)$ if and only if $|m_1(w)-m_0(w)|\\ge|m_1(z)-m_0(z)|$, ties included. This implies that, for every assignment law $P_Z$ on such assignments,\n\\[\nP_Z\\{R(a,Z)\\le R(a,z_{\\mathrm{obs}})\\}=P_Z\\{|m_1(Z)-m_0(Z)|\\ge|m_1(z_{\\mathrm{obs}})-m_0(z_{\\mathrm{obs}})|\\}.\n\\]\nFurther, I have $R=n_0(q_0-m_0^2)+n_1(q_1-m_1^2)$, where $q_g$ is the mean of $a_i^2$ in group $g$. The two group distributions and their sizes therefore determine $R$.\n\nThe table shows the example $a=(0,1,2,3)$ with two of the four units treated.\n\\begin{audittable}{lrrrrrr}{The six assignments of two treated units among four, with $a=(0,1,2,3)$ and each assignment of probability $1/6$. Here $\\delta=m_1-m_0$ and $n_0n_1/n=1$, so the SSR equals $5-\\delta^2$. The lower-tail $p$ value of the SSR equals the upper-tail $p$ value of $|\\delta|$ at every assignment. All values are exact.}\nTreated units & $\\delta$ & $T(a)$ & SSR & KS & $p_{\\mathrm{SSR}}=p_{|\\delta|}$ & $p_{\\mathrm{KS}}$ \\\\\n1, 2 & $-2$ & 5 & 1 & 1 & $1/3$ & $1/3$ \\\\\n1, 3 & $-1$ & 5 & 4 & $1/2$ & $2/3$ & 1 \\\\\n1, 4 & 0 & 5 & 5 & $1/2$ & 1 & 1 \\\\\n2, 3 & 0 & 5 & 5 & $1/2$ & 1 & 1 \\\\\n2, 4 & 1 & 5 & 4 & $1/2$ & $2/3$ & 1 \\\\\n3, 4 & 2 & 5 & 1 & 1 & $1/3$ & $1/3$ \\\\\n\\end{audittable}\nThree further exact computations bear on the two printed statements. First, for treated values $(0,2)$ and control values $(1,3)$, the SSR is $4$ and the KS statistic is $1/2$ whichever control value is paired with which treated value, although the empirical covariance of the pairs, with divisor $2$, is $1$ under one pairing and $-1$ under the other. Second, both randomization distributions depend on the design: at the assignment that treats units 1 and 2, both $p$ values are $1/3$ under complete randomization and $1/2$ when one unit is chosen uniformly and independently for treatment in each of the blocks $\\{1,3\\}$ and $\\{2,4\\}$. Third, the SSR is not a function of the KS value: when units 1 and 3 are treated, the KS statistic is $1/2$ and the SSR is $4$, and when units 1 and 4 are treated, the KS statistic is $1/2$ and the SSR is $5$.", "problem": "The two sentences give a reason to prefer the simple SSR statistic to the KS statistic: the KS statistic is said to ignore information about the joint distribution of treated and control outcomes, and the SSR statistic is said to include their correlation. This reason does not hold. The statistic meant by equation (1), and computed by the replication code, is the residual sum of squares from the regression of the adjusted outcome on an intercept and the treatment indicator (see \\fid{P016-E002}). It is determined by the group sizes and the first two moments of the adjusted outcome within each treatment group. Like the KS statistic, it is a function of the two group distributions and their sizes, and no pairing of treated with control units and no covariance between them enters either statistic. With the adjusted outcomes and the group sizes fixed, a smaller SSR corresponds exactly to a larger absolute difference in group means. The SSR test is then the test based on the absolute difference in means. A narrower statement is true: the randomization distribution of the SSR statistic reflects the dependence between the two groups that the assignment design induces. The same holds for the randomization distribution of the KS statistic, and the statement does not distinguish the two. The two statistics differ in power because they summarize the two group distributions differently.", "what_survives": "The simple SSR statistic and the validity of its exact randomization test are unaffected by this correction, and no simulation, figure or part of the application depends on this explanation. The SSR and KS statistics remain different summaries with different power (Fig.~2). The statistics with network regressors do use information beyond the two group distributions: in an exact four-unit example, two outcome vectors with the same group distributions have SSR $0$ and SSR $4$ once degree is added to the regression. I correct only the two explanatory sentences here. I have not recomputed the reported power values."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E004", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Eight lines of text from the top of page 397. Red boxes mark the words 'with this property (\"effect increasing\" test statistics)' and the words 'any effect increasing test statistic ought to be a valid test'.", "caption": "The end of the quotation and the two sentences that follow it, p. 397. The first mark shows the mistaken identification of the quoted property with “effect increasing”. The second marks a sufficient validity statement, whose conclusion remains correct.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T8", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T8"}, {"id": "P016-T11", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T11"}, {"id": "P016-T12", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T12"}, {"id": "P016-T14", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T14"}], "headline": "A statistic that is small when the two distributions are similar and large when they diverge need not be “effect increasing”, and a randomization test of a sharp null hypothesis is valid for any test statistic, with or without that property.", "id": "P016-E005", "location": "Section 1, last paragraph, the two sentences after the quotation, p. 397 (PDF p. 3); the quotation begins on p. 396", "pdf_page": 3, "printed_page": "397", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P016-T11", "P016-T12", "P016-T14", "P016-T8"], "text": {"as_printed": "``Select a test statistic [$\\mathcal{T}$] that will be small when the treated and control distributions in the adjusted data \\ldots\\ are similar, and large when the distributions diverge.'' Rosenbaum (2002, Proposition 4 and 5, section 2.9) presents results proving that test statistics \\hl{with this property (``effect increasing'' test statistics)} produce an unbiased test of the hypothesis of no effects or positive effects when the positive effects involve one parameter. Such results mean that a test of the sharp null of no effects based on a known randomization using \\emph{any} effect increasing test statistic ought to be a valid test (in that the test should produce $p$ values less than $\\alpha$ no more than $100\\alpha\\%$ of the time when the null is true) even though different test statistics may imply different power against false hypotheses.", "corrected": "\\hl{A statistic is effect increasing if increasing treated responses and decreasing control responses, at fixed assignment, cannot decrease it. The quoted advice about distributional divergence does not imply this directional condition. Under a sharp null hypothesis, calibrating any specified test statistic under the known assignment law gives a valid test; unbiasedness against alternatives is a separate question.}", "evidence": "The evidence is an exact counterexample and a derivation.\n\nFix the assignment $z=(1,1,0,0)$ and change $y=(0,1,2,3)$ to $y'=(2,3,2,3)$. Both treated responses rise by $2$ and the control responses are unchanged. The two-sided KS statistic is $1$ for $y$, where every treated value lies below every control value, and $0$ for $y'$, where both groups have the values $2$ and $3$. Since the statistic decreases, it is not effect increasing. The KS function of the replication code returns the same two values, and the signed maximum printed in footnote 3 also falls from $1$ to $0$.\n\nFor validity, let $\\Omega$ be the finite set of assignments, with known probabilities $q(z)>0$. Under the sharp null hypothesis of no effects, the outcome vector $y^0$ is the same under every assignment. Any statistic then has fixed scores $s(z)=\\mathcal{T}(y^0,z)$. Define the $p$ value $p(z)=\\sum_{w\\in\\Omega:s(w)\\ge s(z)}q(w)$. Let $v_1>\\cdots>v_k$ be the distinct scores, let $m_j$ be the probability that $s(Z)=v_j$, and let $M_j=m_1+\\cdots+m_j$. An assignment with score $v_j$ has $p(z)=M_j$, and $M_j$ is strictly increasing in $j$. Fix $\\alpha\\in[0,1]$. If $M_1>\\alpha$, the event $\\{p(Z)\\le\\alpha\\}$ is empty. Otherwise, let $r$ be the largest index such that $M_r\\le\\alpha$. Then the event consists of the assignments with scores $v_1,\\ldots,v_r$, and its probability is $M_r\\le\\alpha$. I conclude that $\\Pr\\{p(Z)\\le\\alpha\\}\\le\\alpha$ for every statistic. The argument does not use the effect increasing property.\n\nThe cited propositions and the quotation were not compared with the two works cited.", "problem": "The passage identifies the property in the quotation, being small for similar distributions and large for divergent ones, with the effect increasing property. The two are different. A statistic is effect increasing if, at a fixed assignment, raising treated responses and lowering control responses cannot decrease it. This condition is directional. An unsigned measure of the difference between two distributions need not satisfy it, and the two-sided KS statistic does not. The next sentence gives a correct sufficient condition for validity, but validity holds for every test statistic calibrated under the known assignment law, and whether a test is unbiased against alternatives is a separate question.", "what_survives": "The conclusion that the randomization test of the sharp null hypothesis is valid stands, for a more general reason. Nothing later in the note uses the effect increasing property: the statistics, the simulations and the application are unaffected, and the note does not claim that its SSR statistics are effect increasing. Only these explanatory sentences change."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E005", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines from page 400: the sentence 'With distance matrix Gamma, Coppock evaluated the model:', the displayed model y_0 = y_z - beta_1 z - beta_2 g(Gamma z), and the four lines that follow. Red boxes mark the words 'distance matrix', the term g(Gamma z) in the display, and the words 'normalizes the sums of distances to treated neighbors to have unit variance.'.", "caption": "The application model, Section 3, p. 400. The marks show the words “distance matrix”, the term $g(\\Gamma\\mathbf{z})$ of the model, and the description of $g$ as a normalization of sums of distances to unit variance.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E006.png"}, {"alt": "Five lines of text from page 400. A red box marks the words 'less than .05' in the sentence 'The proportion of p values from that test less than .05 is plotted in Fig. 3'.", "caption": "Section 2.2, p. 400. The mark shows the rejection rule of the simulations as printed, “less than .05”.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E006-b.png"}, {"alt": "Scatter plot. Horizontal axis: raw exposure (Sz)_i, the sum of similarities to treated legislators, from 0 to 25. Vertical axis: exposure in standard deviations across the 70 legislators, from -5 to 2. A dashed grey line rising from about -4.8 at raw exposure 0 to about 1 at raw exposure 24 shows the raw sums centred and scaled to unit sample variance, labelled as described on page 400. Orange circles (35 treated legislators) and blue squares (35 control legislators) show the exposure in the replication files, labelled as coded. The points form a cloud between raw exposures 14 and 24 and between -1.9 and 1.6 on the vertical axis, with no visible linear relation to the dashed line; at raw exposure 0 and vertical value near 0, a blue square covers an orange circle at the same position. The title reads 'The exposure used for Fig. 4 is not the raw sum normalized to unit variance: the correlation between the two is 0.066'.", "caption": "The exposure used in the application against the raw exposure, for the 70 legislators of the application table in the replication files. If p. 400’s normalization is read as centring and scaling the raw similarity sum $(Sz)_i$ to unit sample variance, the points would lie on the dashed line. The correlation between the two standardized exposures is $0.0665$. Two legislators have raw exposure $0$ and identical standardized exposure; their symbols overlap.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E006-exposure.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T3", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T3"}, {"id": "P016-T4", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T4"}, {"id": "P016-T6", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T6"}], "headline": "The model behind Fig. 4 uses a similarity exposure from which the unit’s mean exposure among saved assignments with the same own treatment status is subtracted before standardization, not normalized sums of distances; the simulations use a two-sided $p$ value and reject when it is at most $.05$, and the application uses one-sided $p$ values.", "id": "P016-E006", "location": "Section 3, the model that begins “With distance matrix $\\Gamma$”, p. 400 (PDF p. 6); Section 2.2, “less than .05”, p. 400; also p. 398 and the caption of Fig. 3, p. 401", "pdf_page": 6, "printed_page": "400", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P016-T3", "P016-T4", "P016-T6"], "text": {"as_printed": "With \\hl{distance matrix} $\\Gamma$, Coppock evaluated the model:\n\\[\n\\mathbf{y}_0=\\mathbf{y}_{\\mathbf{z}}-\\beta_1\\mathbf{z}-\\beta_2\\hl{g(\\Gamma\\mathbf{z})},\n\\]\nwhere $g$ is a function that \\hl{normalizes the sums of distances to treated neighbors to have unit variance}. This model has a direct effect ($\\beta_1$) and an indirect effect ($\\beta_2$) that is linear in the distances to treated neighbors. Coppock evaluated this spillover model using the SSR + Design statistic as presented in this article.\n\nSection 2.2, same page: The proportion of $p$ values from that test \\hl{less than .05} is plotted in Fig. 3: darker values show fewer rejections, lighter values record more rejections.", "corrected": "With \\hl{similarity matrix $S$}, where $S_{ij}=1-|s_i-s_j|/2$ for the W-NOMINATE scores of legislators $i\\ne j$ and the diagonal and undefined entries are zero, the model evaluated is\n\\[\n\\mathbf{y}_0=\\mathbf{y}_{\\mathbf{z}}-\\beta_1\\mathbf{z}-\\beta_2\\hl{e(\\mathbf{z})},\\qquad \\hl{e_i(z)=\\frac{h_i(z)-\\bar h(z)}{s_h(z)}},\\qquad \\hl{h_i(z)=(Sz)_i-z_i\\mu_{i1}-(1-z_i)\\mu_{i0}}.\n\\]\nHere $\\mu_{ia}$ is the mean of $(Sz)_i$ over those of the 10,000 saved assignments of the matched design in which $z_i=a$, and $\\bar h(z)$ and $s_h(z)>0$ are the mean and the sample standard deviation of $h_i(z)$ across the 70 legislators. The indirect effect $\\beta_2$ multiplies this \\hl{centred, standardized similarity exposure}, and Fig.~4 and the rectangle on p.~402 refer to this model. For each hypothesis the adjusted outcome $u=Y-\\beta_1Z-\\beta_2e(Z)$ is held fixed. For each reference assignment $z$ the exposure $e(z)$ is recomputed, and the SSR statistic is the residual sum of squares from the regression of $u$ on \\hl{an intercept, $z$ and $e(z)$, with no separate degree column}, \\hl{using the 67 legislators with observed outcomes}. The application uses 500 reference assignments drawn from the saved assignments. Its $p$ value is the proportion of reference statistics that are at most the observed statistic for SSR, and at least the observed statistic for KS. For each axis, the SSR bounds are taken at \\hl{the second coordinate of each adjacent grid pair where the largest $p$ value along the other axis changes from at most $.05$ to greater than $.05$, or conversely}.\n\nI describe the procedure used for the published figures as follows. Section 2.2: The proportion of $p$ values from that test \\hl{less than or equal to} .05 is plotted in Fig. 3, \\hl{where the $p$ value is twice the smaller of the proportions of sampled statistics that are at most, and at least, the observed statistic, capped at $1$}. The same descriptive change applies to ``lower than $\\alpha=0.05$'' on p.~398 and to the caption of Fig.~3. This describes the original rule used for Figs.~2 and 3. Figures for the repaired procedure in \\fid{P016-E008} require recomputation.", "evidence": "I base this correction on the replication code, exact arithmetic on the application table of the replication files, and exact small examples.\n\nLet $r_i=(Sz)_i$ be the raw exposure and $h_i$ the exposure after the subtraction. If the subtraction could be absorbed by an intercept, the direct effect and a rescaling of the indirect effect, then there would exist constants $a$, $b$ and $c$ such that $h_i=a+bz_i+cr_i$ for all $i$, and the four columns $(1,z,r,h)$ would have rank at most three. For the first four rows of the application table, the determinant of these four columns, computed in exact rational arithmetic from the decimals of the table, is $-1.4030\\ldots\\ne0$. No such constants exist. Across the 70 legislators the correlation between the standardized raw exposure and the standardized exposure used is $0.0665$, and the largest difference between the two is $4.76$ standard deviations.\n\nThe same holds in an example in which all assignments are enumerated. Let six units have scores $(-1,-3/4,-1/2,0,1/4,1)$ and form the pairs $(1,2)$, $(3,4)$ and $(5,6)$, with one treated unit in each pair, a design with eight assignments. At $z=(1,0,1,0,1,0)$ the raw exposures $r$ and the exposures $h$ after the subtraction are\n\\[\nr=(9/8,\\ 9/4,\\ 11/8,\\ 17/8,\\ 1,\\ 7/8),\\qquad h=(5/16,\\ 5/16,\\ 1/8,\\ 1/8,\\ -3/16,\\ -3/16).\n\\]\nUnits 1 and 2 have equal exposure after the subtraction and unequal raw exposure, while units 1 and 3 differ in both. This implies that $h$ is not of the form $a+cr$, and the rank of $(1,z,r,h)$ is four. For the outcome $y=(0,1,0,1,1,0)$ the residual sum of squares is $4/3$ from the regression on $(1,z,h)$ and $655/708$ from the regression on $(1,z,r)$.\n\nFor the rejection rule, take the reference values $1,2,\\ldots,1000$. At the observed value $25$ the lower tail proportion is $1/40$ and the two-sided $p$ value of the simulation code is exactly $.05$. The code counts a rejection, and the printed rule does not. At the observed value $40$ the lower tail proportion is $.04$ and the two-sided $p$ value is $.08$; at $980$ they are $.98$ and $.042$. Neither the one-sided rule nor the two-sided rule rejects whenever the other does.\n\\begin{audittable}{lll}{How $p$ values are formed and used, in the text and in the replication code. $L$ and $U$ are the proportions of the sampled statistics that are at most, and at least, the observed statistic.}\n & Text & Replication code \\\\\nSimulations, $p$ value & not stated & $\\min\\{1,2\\min(L,U)\\}$ \\\\\nSimulations, rejection & $p<.05$ & $p\\le.05$ \\\\\nApplication, $p$ value & not stated & $L$ for SSR, $U$ for KS, 500 draws \\\\\nApplication, bounds of the SSR region & not stated & transitions in whether the largest $p$ along the other axis is $>.05$ \\\\\n\\end{audittable}\nThe numerical comparison uses the application table that is among the replication files. The data file that the figure script loads is not among them, and I did not recompute Fig.~4.", "problem": "The description of the application does not specify the model to which Fig.~4 and the rectangle on p.~402 refer. In the replication code the matrix is a similarity matrix $S$, with $S_{ij}=1-|s_i-s_j|/2$ for the W-NOMINATE scores $s_i$ and $s_j$, so that ideologically closer legislators have larger weights; the diagonal and undefined entries are zero. The exposure of legislator $i$ is not $(Sz)_i$ normalized to unit variance. The code first subtracts from $(Sz)_i$ its mean over those of 10,000 saved assignments in which legislator $i$ has the same treatment status, and only then centres and scales across legislators. This subtraction is not a recentring or a rescaling. The exposure used is not an affine function of the raw exposure, even when an intercept and a change in the direct effect are allowed, and in the application table of the replication files the two standardized exposures have correlation $0.07$. The indirect effect $\\beta_2$ of Fig.~4 therefore multiplies a different variable from the one that the text describes, and its sign and its unit can be read only with the coded definition. The regression in the SSR statistic has an intercept, own assignment and this exposure, with no separate degree column, and the phrase ``SSR + Design statistic as presented in this article'' does not convey this.\n\nThe text also does not say how the $p$ values are formed. The simulations behind Figs.~2 and 3 use a two-sided $p$ value, twice the smaller of the two tail proportions, capped at $1$, and they count a rejection when $p\\le.05$. The text and the caption of Fig.~3 say ``less than .05''. The application uses one-sided $p$ values from 500 draws, the lower tail for SSR and the upper tail for KS. The two rules define different tests, and the note does not distinguish them.", "what_survives": "This correction changes the description of the model and testing rules used by the replication code. The note's comparison of a bounded SSR region and an unbounded KS region in Fig.~4 concerns the coded model. I have not established a numerical error in Fig.~4 or in the rectangle, and the absence of a degree column is not shown to make the test invalid. Figs.~2 and 3 are results for the two-sided rule with rejection at $p\\le.05$; they cannot be relabelled as results for another rule without recomputation, and my recalculated null rejection rates are compatible with the reported rates of about 4\\% and 2\\% within Monte Carlo error; they do not establish a level guarantee. The meaning of $\\beta_2$ changes: it is the coefficient of a centred, standardized similarity exposure, not of normalized sums of distances."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E006", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Seven lines of text from page 400. Red boxes mark the words 'less than .05' and the sentence part 'All of these test statistics are valid - they reject the true null of tau = .5, beta = 2 no more than 5 percent of the time at alpha = .05 -'.", "caption": "Section 2.2, first paragraph, p. 400. The marks show the rejection rule as printed, “less than .05”, and the statement that all six tests reject the true null no more than 5% of the time at $\\alpha=.05$. The statement is correct for the rule as printed; the simulation code applies a different rule.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T13", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T13"}, {"id": "P016-T14", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T14"}, {"id": "P016-T21", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T21"}], "headline": "The printed rule, which rejects when $p<.05$, is valid at level $.05$; the simulation code rejects when $p\\le.05$ with tail proportions that do not count the observed statistic, and that rule can have size slightly above $.05$.", "id": "P016-E008", "location": "Section 2.2, first two paragraphs, p. 400 (PDF p. 6), with the validity statement on p. 397 (PDF p. 3); the rejection rule is in the simulation scripts of the replication files", "pdf_page": 6, "printed_page": "400", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P016-T13", "P016-T14", "P016-T21"], "text": {"as_printed": "The proportion of $p$ values from that test less than .05 is plotted in Fig. 3: darker values show fewer rejections, lighter values record more rejections. All of these test statistics are valid---they reject the true null of $\\tau=.5,\\ \\beta=2$ no more than 5\\% of the time at $\\alpha=.05$---and the plots are darkest in the area where the two lines showing the true parameters intersect.\n\nSimulation scripts of the replication files: \\texttt{m <- m \\hl{<=} ALPHA}, with \\texttt{ALPHA <- 0.05}.", "corrected": "The validity statement on p.~397 stands. For a repaired simulation procedure with the existing tail estimator, \\hl{1,000 or 10,000 independent reference draws and a fixed statistic}, count a rejection when $p\\,\\hl{<}\\,.05$. The published Figs.~2 and 3 describe the original $p\\le.05$ rule (\\fid{P016-E006}); they have not been recomputed for the repaired procedure. Figures for the repaired procedure require recomputation.\n\nFor a $p$ value that is valid at every threshold, let $L$ and $U$ be the numbers of the $B$ reference statistics that are at most, and at least, the observed statistic, and set\n\\[\np_L=\\frac{\\hl{1+}L}{B\\hl{+1}},\\qquad p_U=\\frac{\\hl{1+}U}{B\\hl{+1}},\\qquad p=\\min\\{1,2\\min(p_L,p_U)\\}.\n\\]\nIf the observed statistic and the $B$ reference statistics are exchangeable under the null hypothesis, as they are for a fixed statistic and independent draws from the assignment law, then $\\Pr(p\\le\\alpha)\\le\\alpha$ for every $\\alpha\\in[0,1]$. Sampling from a fixed saved set of assignments, as in the application, requires a separate justification of exchangeability.", "evidence": "The evidence is an exact counterexample, computed by enumeration and rational arithmetic, and a derivation.\n\nFor the counterexample, I take sixteen units, eight of them treated, with all $\\binom{16}{8}=12870$ assignments equally likely, the sharp null hypothesis of no effect with outcomes $2^i$ for $i=0,\\ldots,15$, and the simple SSR statistic. The statistic takes 6435 distinct values over these assignments, each shared by an assignment and its complement. The observed assignment and the $B$ reference assignments are drawn independently from the design.\n\\begin{audittable}{lrr}{Exact null rejection probability at nominal level $.05$ in the sixteen-unit example, rounded to six decimals.}\nReference draws $B$ & Coded rule, $p\\le.05$ & Printed rule, $p<.05$ \\\\\n$1000$ & $0.051793$ & $0.049795$ \\\\\n$10{,}000$ & $0.050040$ & $0.049840$ \\\\\n\\end{audittable}\nThe probability that the coded $p$ value equals $0$ is $0.0018$ for $B=1000$.\n\nFor the derivation, let $T_0$ be the observed statistic and $T_1,\\ldots,T_B$ the reference statistics, and suppose that these $B+1$ statistics are exchangeable under the null hypothesis. Set $L=\\#\\{b:T_b\\le T_0\\}$ and $U=\\#\\{b:T_b\\ge T_0\\}$. With ties broken at random, the rank of $T_0$ among the $B+1$ statistics is uniform on $\\{1,\\ldots,B+1\\}$, and this rank is at most $L+1$. I therefore have $\\Pr(L\\le c)\\le(c+1)/(B+1)$ for every integer $c\\ge0$, and the same bound for $U$. The event $2\\min(L,U)/B<\\alpha$ requires that $L\\le c$ or $U\\le c$ with $c=\\lceil\\alpha B/2\\rceil-1$, and by the union bound its probability is at most $2(c+1)/(B+1)$. At $\\alpha=.05$ and $B=1000$, I have $c=24$, and the bound is $50/1001<.05$. The coded rule allows $c=\\lfloor\\alpha B/2\\rfloor=25$, for which the same bound is $52/1001>.05$; the counterexample shows that the size can exceed $.05$. Next, set $p_L=(1+L)/(B+1)$ and $p_U=(1+U)/(B+1)$. For every $a\\in[0,1]$, I have $\\Pr(p_L\\le a)=\\Pr(L+1\\le\\lfloor a(B+1)\\rfloor)\\le\\lfloor a(B+1)\\rfloor/(B+1)\\le a$, and the same holds for $p_U$. For $\\alpha<1$ the event $\\min\\{1,2\\min(p_L,p_U)\\}\\le\\alpha$ is the union of $\\{p_L\\le\\alpha/2\\}$ and $\\{p_U\\le\\alpha/2\\}$. Its probability is at most $\\alpha$.", "problem": "The validity statements on pp.~397 and 400 are correct for the rule as printed. With a fixed statistic and $B$ fresh reference draws, the rule that rejects when the $p$ value is less than $.05$ has null rejection probability at most $50/1001$ for $B=1000$ and at most $500/10001$ for $B=10{,}000$. The simulation code applies a different rule. Each tail is the proportion of the $B$ reference statistics that are at most, or at least, the observed statistic, and the observed statistic is not counted among them. The $p$ value is twice the smaller tail, capped at $1$, and a rejection is counted when $p\\le.05$. This rule is not a test of level $.05$ in general: its size can exceed $.05$ slightly, and its $p$ value can equal $0$.", "what_survives": "The validity statement for the exact randomization test on p.~397 and the printed rule at level $.05$ stand, as do the statistics and the framework. No reported rejection rate, power ranking or figure is shown to change. Note that the counterexample is a constructed design of sixteen units and not the simulation of 256 units in the note, and the excess over $.05$ is a fraction of a percentage point. The $p$ values of the application are computed from a fixed saved set of assignments; they need a separate argument and were not examined here."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E008", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Four lines of text from page 400. A red box marks the words 'we sampled 1000 permutations. We executed that test 10,000 times'.", "caption": "Section 2.2, first paragraph, p. 400. The mark shows the description of the simulation: 1000 sampled permutations for each test, and 10,000 tests for each pair of parameters.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E009.png"}, {"alt": "Four lines of text from page 400. A red box marks the words 'of the 10,000 simulations' in 'the SSR-based tests rejected the truth roughly 4 percent of the 10,000 simulations but the KS tests rejected the truth roughly 2 percent of the time'.", "caption": "Section 2.2, second paragraph, p. 400. The mark shows the number of simulations to which the rejection rates of about 4% and 2% are referred.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The text describes the simulation behind Fig. 3 as 10,000 experiments of 1000 sampled permutations each, and the replication code defaults to 1000 experiments of 10,000 sampled assignments each; which setting produced the published figure is not known.", "id": "P016-E009", "location": "Section 2.2, first and second paragraphs, p. 400 (PDF p. 6)", "pdf_page": 6, "printed_page": "400", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "For each test statistic, we tested the hypothesis $\\tau=\\tau_0,\\ \\beta=\\tau_0$ by using a simulated permutation test: \\hl{we sampled 1000 permutations. We executed that test 10,000 times} for each pair of parameters.\n\nAgain, all of the tests are valid in the sense of rejecting the truth no more than 5\\% of the time when $\\alpha=.05$ although all of these tests are conservative: the SSR-based tests rejected the truth roughly 4\\% \\hl{of the 10,000 simulations} but the KS tests rejected the truth roughly 2\\% of the time.", "corrected": "The deposited driver for Fig.~3 defaults to \\hl{1,000 simulated experiments with 10,000 sampled assignments for each permutation distribution}. This allocation differs from the 10,000 experiments with 1,000 sampled permutations described on p.~400. The counts used to produce the published figure cannot be recovered from the deposited files, because saved results and run logs are not supplied; the published counts should be reconciled against the original invocation and results.", "evidence": "The evidence is a recalculation from the replication code and a check of the page. Running the count expressions, the sampler and the test routine of both released versions of the code, I obtain a matrix of 1000 assignments of the 256 units, and each test receives 10,000 samples. The standard errors are $\\sqrt{p(1-p)/N}$ at $p=1/25$, which is $0.006197$ for $N=1000$ and $0.001960$ for $N=10{,}000$. I obtained these counts from the code settings and calls, without simulating rejection rates.", "problem": "The text and the replication code disagree about how the simulation behind Fig.~3 is divided. In the code, when no setting is supplied, the number of simulated experiments is 1000 and each test samples 10,000 assignments, and the driver files of the archive supply no other setting. The environment variable that the code provides changes only the number of experiments; with 10,000 experiments each test still samples 10,000 assignments, which again differs from the text. Since saved results and run logs are not among the replication files, the setting used for the published figure cannot be recovered. The difference bears on the precision of the reported rates: at a rejection rate of $0.04$ the Monte Carlo standard error is $0.0062$ with 1000 experiments and $0.0020$ with 10,000.", "what_survives": "No rejection rate, power comparison or conclusion is shown to be wrong. The discrepancy establishes that the text and the code disagree, and it does not establish which setting was run. Fig.~2 is unaffected, since its script sets 1000 sampled assignments itself, and so are the statistics, the framework and the application."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E009", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Five lines of text from page 402. A red box, in two parts across a line break, marks the words 'outside the rectangle with corners (-0.6, 0.2) and (0.3, -0.4).'.", "caption": "Section 3, last sentence, p. 402. The mark shows the rectangle that summarizes the SSR region. The defect is in the endpoint summaries that the application script saves; the printed rectangle is not shown to come from them.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E010.png"}], "formal_release": null, "formal_statements": [], "headline": "The application script saves, as the upper endpoint of an interior accepted range, the first rejected grid point and not the last accepted one; the $p$ values, Fig. 4 and the printed rectangle are not shown to depend on this.", "id": "P016-E010", "location": "Section 3, last sentence, p. 402 (PDF p. 8); the endpoint helper find_breaks is in the application script of the replication files, lines 12–18, 77–78 and 83–84", "pdf_page": 8, "printed_page": "402", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "it would be very unlikely that the data were generated from the model with parameters outside the rectangle with corners $(-0.6, 0.2)$ and $(0.3, -0.4)$.\n\nHelper \\texttt{find\\_breaks} of \\texttt{CoppockJEPS\\_figure2code.R}, the application script of the replication files:\n\n\\texttt{breaks[i+1] <- x[i]!=x[i+1]}\n\n\\texttt{return(which(breaks))}\n\nIn the main part of the script, and likewise for the indirect effect:\n\n\\texttt{direct\\_breaks <- find\\_breaks(apply(pmat.ssr, MARGIN=1, FUN=max) >.05)}\n\n\\texttt{directs.95.2 <- directs[\\hl{direct\\_breaks}]}", "corrected": "For each parameter, the endpoints of the accepted range are the extremes of the grid coordinates whose projected $p$ value exceeds $.05$, that is \\texttt{range(grid[accepted])} in place of the transition positions, and an empty accepted set is reported as empty. When a projection is accepted at positions $r,\\ldots,s$, its endpoints are the grid coordinates $g_r$ and $\\hl{g_s}$, not $g_r$ and $g_{s+1}$. If the accepted set is not connected, the product of the two ranges is a bounding rectangle and not the accepted set itself. This corrects the endpoint extraction only; it does not alter the matrices of $p$ values or establish corrected values for the rounded rectangle on p.~402.", "evidence": "I checked the derivation and an exact counterexample using the original code.\n\nLet $a_i$ indicate that the largest $p$ value in row $i$ of the matrix exceeds $.05$, for $i=1,\\ldots,n$. The helper sets entry $i+1$ of its output to the truth value of $a_i\\ne a_{i+1}$ for $i=1,\\ldots,n-1$, and it returns the positions of the true entries. If the accepted positions are exactly $r,\\ldots,s$ with $1<r\\le s<n$, then the changes occur at $i=r-1$ and at $i=s$, and the helper returns $(r,s+1)$. The script then saves the grid values $(g_r,g_{s+1})$. The extremes of the accepted grid values are $(g_r,g_s)$. On the grid of the script, $g_i=(2i-51)/49$, which implies that $g_{s+1}-g_s=2/49$.\n\nFor the counterexample, let the $50\\times50$ matrix have $p_{ij}=1/10$ for $20\\le i\\le30$ and $22\\le j\\le27$, and $p_{ij}=1/50$ elsewhere. The accepted direct range is positions 20 to 30, that is $[-11/49,9/49]$, and the script saves positions 20 and 31, that is $[-11/49,11/49]$. The accepted indirect range is positions 22 to 27, that is $[-1/7,3/49]$, and the script saves positions 22 and 28, that is $[-1/7,5/49]$. The helper returns $r$ and $s+1$ for each of the 1176 interior runs of the grid. I did not recompute the application's $p$-value matrices to establish this endpoint error.", "problem": "The helper \\texttt{find\\_breaks} returns the positions that follow each change of the accepted or rejected indicator along a grid axis, and the script uses those positions as the endpoints of the accepted range. For a run of accepted grid points at positions $r,\\ldots,s$ strictly inside the grid, the helper returns $r$ and $s+1$. The lower endpoint is right. The upper endpoint is the first rejected grid point, one step above the last accepted one; the step is $2/49\\approx0.04$ on the grid of 50 points over $[-1,1]$. The defect is in the endpoint summaries that the script saves. The rectangle on p.~402 is rounded text in the manuscript source, and it is not shown to come from these endpoints.", "what_survives": "The two matrices of $p$ values, the shading and the axes of Fig.~4, and the contrast between a bounded SSR region and a KS region that reaches the edges of the grid are computed before this step and do not use it. The correction changes only the saved endpoint summaries: for an interior run, the upper endpoint falls by $2/49$. It does not establish corrected values for the rounded rectangle on p.~402."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E010", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Eleven lines of text from the last paragraph of page 398. Red boxes mark the words 'control group distribution.' in the sentence 'the hypotheses only have different implications for the control group distribution' and the word 'effected' in 'not very effected by the model of effects'.", "caption": "Last paragraph of p. 398. The marks show “control group distribution”, where the treated group is meant, and the misprint “effected”.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011.png"}, {"alt": "Footnotes 2, 3 and 4 of page 396. In the last line of footnote 3, red boxes mark the symbol x_i in 'at or below x_i' and the citation '(Hollander 1999, section 5.4).'.", "caption": "Footnote 3, p. 396. The marks show $x_i$, printed for the threshold $x$, and the citation of the 1999 book under one author.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011-b.png"}, {"alt": "Four lines of text from page 400. A red box marks 'beta = tau_0' in the words 'we tested the hypothesis tau = tau_0, beta = tau_0 by using a simulated permutation test'.", "caption": "Section 2.2, p. 400. The mark shows the second part of the joint hypothesis, printed as $\\beta=\\tau_0$.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011-c.png"}, {"alt": "Three lines that end Section 2.2, the heading '3 Application: Legislative Information Spillovers', and three lines of Section 3. Red boxes mark the words 'appear be' in 'the KS tests appear be less powerful in general' and the words 'State Senate,' in 'in the New Mexico State Senate, Butler and Nickerson randomly assigned legislators'.", "caption": "End of Section 2.2 and beginning of Section 3, p. 400. The marks show the misprint “appear be” and the chamber printed as “State Senate”.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011-d.png"}, {"alt": "Five lines of the reference list on page 402. In the entry that begins 'Brady, Henry E. 2008.', a red box marks the name 'Collier David,' in the list of editors 'Janet M. Box-Steffensmeier, Henry E. Brady and Collier David'.", "caption": "Reference list, p. 402, the entry Brady (2008). The mark shows the editor’s name printed as “Collier David”.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011-e.png"}, {"alt": "Five lines of the reference list on page 403. A red box marks 'Hollander, Myles. 1999.' at the start of the entry 'Hollander, Myles. 1999. Nonparametric statistical methods, 2nd ed. Hoboken, NJ: Wiley-Interscience.'.", "caption": "Reference list, p. 403. The mark shows the entry for the 1999 second edition of Nonparametric statistical methods, printed under one author.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011-f.png"}, {"alt": "Five lines of text from page 399. A red box marks the word 'controlgroups.' in 'a shift in the location of the distributions of the treated versus the controlgroups.'.", "caption": "Second paragraph of p. 399. The mark shows the misprint “controlgroups”.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011-g.png"}, {"alt": "Six lines of the abstract on page 395. A red box, in two parts across a line break, marks the citation '(Hollander 1999, section 5.4)' after the words 'the simple Kolmogorov-Smirnov test statistic'.", "caption": "Abstract, p. 395. The mark shows the citation of the 1999 book under one author.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E011-h.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T15", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T15"}, {"id": "P016-T16", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T16"}], "headline": "Six local corrections: the hypotheses of Fig. 2 differ in what they imply for the treated group, the ECDF threshold in footnote 3 is $x$, the joint hypothesis on p. 400 is $\\tau=\\tau_0$, $\\beta=\\beta_0$, the chamber is the New Mexico House of Representatives, two reference entries misstate names, and three words are misprinted.", "id": "P016-E011", "location": "Last paragraph of p. 398 (PDF p. 4); also footnote 3, p. 396; p. 399; p. 400; the reference list, pp. 402–403; and the abstract, p. 395", "pdf_page": 4, "printed_page": "398", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P016-T15", "P016-T16"], "text": {"as_printed": "\\begin{enumerate}\n\\item p.~398: Here, since we have no interference, the hypotheses only have different implications for the \\hl{control group} distribution.\n\\item Footnote 3, p.~396: where $F(x)=(1/n)\\sum_{i=1}^{n}I(x_i\\le x)$ records the proportion of the distribution of $x$ at or below $\\hl{x_i}$ (Hollander 1999, section 5.4).\n\\item p.~400: For each test statistic, we tested the hypothesis $\\tau=\\tau_0,\\ \\hl{\\beta=\\tau_0}$ by using a simulated permutation test\n\\item p.~400: Leading up to a key vote on a budget bill, SB24, in the New Mexico \\hl{State Senate}, Butler and Nickerson randomly assigned legislators to see constituent survey results.\n\\item p.~402: Brady, Henry E. 2008. Causation and explanation in social science. In: \\emph{Oxford handbook of political methodology}, eds. Janet M. Box-Steffensmeier, Henry E. Brady and \\hl{Collier David}, 217--70. p.~403: \\hl{Hollander, Myles. 1999.} \\emph{Nonparametric statistical methods}, 2nd ed. Hoboken, NJ: Wiley-Interscience. Abstract, p.~395, and footnote 3: (\\hl{Hollander 1999}, section 5.4)\n\\item p.~398: not very \\hl{effected} by the model of effects. p.~399: the treated versus the \\hl{controlgroups}. p.~400: the KS tests \\hl{appear be} less powerful in general\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item p.~398: Here, since we have no interference, the hypotheses only have different implications for the \\hl{treated group} distribution.\n\\item Footnote 3, p.~396: records the proportion of the distribution of $x$ at or below $\\hl{x}$ (\\hl{Hollander and Wolfe} 1999, section 5.4).\n\\item p.~400: we tested the hypothesis $\\tau=\\tau_0,\\ \\hl{\\beta=\\beta_0}$\n\\item p.~400: in the New Mexico \\hl{House of Representatives}\n\\item p.~402: eds. Janet M. Box-Steffensmeier, Henry E. Brady and \\hl{David Collier}. p.~403: \\hl{Hollander, Myles, and Douglas A. Wolfe. 1999.} \\emph{Nonparametric statistical methods}, 2nd ed. Abstract, p.~395: (\\hl{Hollander and Wolfe} 1999, section 5.4)\n\\item p.~398: not very \\hl{affected} by the model of effects. p.~399: the treated versus the \\hl{control groups}. p.~400: the KS tests \\hl{appear to be} less powerful in general\n\\end{enumerate}", "evidence": "The evidence is a check of the page for each item, exact small examples, the replication files, and a library catalogue record. With $Y=(9,11,10,12)$ and $Z=(0,0,1,1)$, the adjusted outcomes are $(9,11,19/2,23/2)$ under the additive hypothesis $\\tau_0=1/2$ and $(9,11,5,6)$ under the multiplicative hypothesis $\\tau_0=2$: the control values are the same and the treated values differ. The simulation code tests every pair on a grid of 22 values of $\\beta$ and 21 values of $\\tau$, and the grid contains the truth $(2,0.5)$. The roll-call table and the application table of the replication files each have 70 rows, one for the representative of each of House districts 1 to 70. The catalogue record of the 1999 second edition (WorldCat, OCLC 301603607) lists Myles Hollander and Douglas A. Wolfe as authors.", "problem": "\\begin{enumerate}\n\\item Without interference, the adjustment changes only the outcomes of the units assigned to treatment, as p.~397 says and Fig.~2 shows. The two hypotheses therefore differ in what they imply for the treated group, and the control group is the same under both.\n\\item The ECDF $F(x)$ records the proportion of the sample at or below the threshold $x$, not at or below a sample point $x_i$.\n\\item The two parameters vary separately, and the truth is $\\tau=.5$, $\\beta=2$. The family $\\tau=\\tau_0$, $\\beta=\\tau_0$ does not contain the truth.\n\\item The legislators who were randomized, and whose votes are analysed, are the 70 members of the New Mexico House of Representatives.\n\\item The 1999 second edition of \\emph{Nonparametric Statistical Methods} has two authors, Myles Hollander and Douglas A. Wolfe. In the Brady entry the name of the editor David Collier is printed in reverse order.\n\\item Three words are misprinted.\n\\end{enumerate}", "what_survives": "None of these corrections changes an implemented formula, a computation or a reported number. The adjustment, the ECDF formula, equation (3) and the replication code already have the correct content, and the application uses the House data."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E011", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Five lines of text from page 397. A red box marks the subscript Z_i=1 of the first y-tilde in the words 'Comparing the mean of y-tilde_i,Zi=1 given Z_i = 1 to the mean of y-tilde_i,Zi=0 given Z_i = 0'.", "caption": "Section 2, first paragraph, p. 397. The mark shows the subscript $Z_i=1$ of the first adjusted outcome in the comparison of means.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-N002.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T17", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T17"}], "headline": "The first mean in the comparison is the mean of $\\tilde y_{i,Z_i=0}$ among the treated units, and its subscript is $Z_i=0$, not $Z_i=1$.", "id": "P016-N002", "location": "Section 2, first paragraph, p. 397 (PDF p. 3)", "pdf_page": 3, "printed_page": "397", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P016-T17"], "text": {"as_printed": "For example, the implication of the constant, additive effects model, $\\tilde y_{i,Z_i=0}=Y_i-Z_i\\tau$, only changes the distribution of outcomes for units in the treated condition. Comparing the mean of $\\tilde y_{i,\\hl{Z_i=1}}|Z_i=1$ to the mean of $\\tilde y_{i,Z_i=0}|Z_i=0$ makes intuitive sense in this case", "corrected": "For example, the implication of the constant, additive effects model, $\\tilde y_{i,Z_i=0}=Y_i-Z_i\\tau$, only changes the distribution of outcomes for units in the treated condition. Comparing the mean of $\\tilde y_{i,\\hl{Z_i=0}}|Z_i=1$ to the mean of $\\tilde y_{i,Z_i=0}|Z_i=0$ makes intuitive sense in this case", "evidence": "The evidence is a derivation with a numerical illustration, and a check of the page. Let $u_i=Y_i-Z_i\\tau$. The comparison meant is $\\operatorname{mean}(u\\mid Z=1)-\\operatorname{mean}(u\\mid Z=0)$. If $\\tilde y_{i,Z_i=1}$ is read as the treated potential outcome implied by the hypothesis, then it equals $u_i+\\tau$, and the printed comparison equals $\\operatorname{mean}(u\\mid Z=1)-\\operatorname{mean}(u\\mid Z=0)+\\tau$, which retains the hypothesized effect. With $u=(0,1,2,3)$, $\\tau=10$ and $Z=(0,0,1,1)$, the corrected comparison is $2$ and the comparison under this literal reading is $12$. The replication code uses the same adjusted outcome for both groups.", "problem": "The comparison meant is between the mean of the adjusted outcome $\\tilde y_{i,Z_i=0}$ in the treated group and its mean in the control group. The first term carries the subscript $Z_i=1$. The quantity $\\tilde y_{i,Z_i=1}$ is not defined in the note: the preceding sentence and p.~396 define only $\\tilde y_{i,Z_i=0}$. The conditioning event $Z_i=1$ identifies the treated group correctly.", "what_survives": "The comparison meant, the randomization procedure and the inspected replication code are unchanged. I found no reported result that depends on this subscript."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-N002", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Eleven lines of text from the last paragraph of page 398. Red boxes mark the words 'which produced very similar implications' and the parenthesis '(which makes some sense here because we are choosing parameters at which both models imply more or less the same patterns in outcomes).'.", "caption": "Last paragraph of p. 398. The marks show the qualitative comparison of the two hypotheses, which does not establish equal expected mean shifts; the defect is in the calibration code.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-N003.png"}, {"alt": "Figure 2 of the note: two panels of boxplots, titled 'Normal Outcome' and 'Jittered Geometric Outcome', each with columns y0, Obs., Add. Model and Mult. Model, above the caption. Below the model columns are printed the parameters and powers: for the Normal outcome, tau_0 = 0.34 with pow_SSR = 0.7 and pow_KS = 0.54, and tau_0 = 1.043 with pow_SSR = 0.86 and pow_KS = 0.74; for the Jittered Geometric outcome, tau_0 = 0.34 with pow_SSR = 0.7 and pow_KS = 1, and tau_0 = 1.043 with pow_SSR = 0.85 and pow_KS = 1. Red boxes mark the four labels 'tau_0 = 0.34' and 'tau_0 = 1.043'.", "caption": "Fig. 2 and its caption, p. 398. The marks show the hypothesized parameters printed below the “Add. Model” and “Mult. Model” boxplots of each panel: $\\tau_0=0.34$ and $\\tau_0=1.043$.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-N003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P016-T18", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T18"}, {"id": "P016-T19", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T19"}], "headline": "The additive and multiplicative hypotheses of Fig. 2 are not matched in strength: the replication code chooses the multiplicative parameter by a criterion that can match a single treated unit, and $\\tau_0=1.043$ implies an expected shift in the treated mean about 20% larger than $\\tau_0=0.34$ does.", "id": "P016-N003", "location": "Last paragraph of p. 398 (PDF p. 4), with the labels of Fig. 2 on the same page; the calibration is in the script ksvsssr-setup.R of the replication files, lines 52–61", "pdf_page": 4, "printed_page": "398", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P016-T18", "P016-T19"], "text": {"as_printed": "In this case, we chose the two parameters for the additive and multiplicative models which produced very similar implications (as shown by the similarity in the ``Add. Model'' and ``Mult. Model'' boxplots in both panels of the figure).\n\nFor these particular parameters, we see the performance of the SSR test statistic as better than the KS test statistic with Normal outcomes, but not very effected by the model of effects (which makes some sense here because we are choosing parameters at which both models imply more or less the same patterns in outcomes).\n\nFig.~2, Normal Outcome. Add. Model: $\\tau_0=0.34$, $\\mathrm{pow}_{\\mathrm{SSR}}=0.7$, $\\mathrm{pow}_{\\mathrm{KS}}=0.54$. Mult. Model: $\\tau_0=1.043$, $\\mathrm{pow}_{\\mathrm{SSR}}=0.86$, $\\mathrm{pow}_{\\mathrm{KS}}=0.74$.\n\nCalibration in \\texttt{ksvsssr-setup.R}, line 55: \\texttt{\\hl{median}(abs(mulres-addres))}.", "corrected": "In this case, the two parameters for the additive and multiplicative models were chosen by a criterion that can match the outcomes implied by the two models for one treated unit. \\hl{The two hypotheses do not match the expected shift in the treated mean, which is about $0.34$ under the additive hypothesis and about $0.41$ under the multiplicative hypothesis.}\n\n\\hl{SSR has greater power than KS for the Normal outcome under both hypotheses ($0.70$ versus $0.54$ and $0.86$ versus $0.74$), and less power for the Jittered Geometric outcome. These comparisons preserve the ordering of the two statistics, but the hypotheses do not match the expected adjusted mean shifts.}\n\nFor a comparison at matched expected mean shift, the multiplicative parameter is $a=\\bar y/(\\bar y-t)$, which is about $1.0353$ for $t=0.3434$ and $\\bar y=10.0692$, and the powers under that hypothesis must be computed anew. A calibration criterion should summarize the differences among the treated units only, not the median over the 128 unchanged control units and the 128 treated units.", "evidence": "The evidence is a derivation, an exact example, and a recalculation with the replication code.\n\nFor a no-effect outcome $y_i$ and an assignment $z$, the additive model with parameter $t$ generates $y_i+z_it$ and the multiplicative model with parameter $a$ generates $y_i[1+z_i(a-1)]$. The two agree for a control unit, and for a treated unit they differ by $|(a-1)y_i-t|$. With 128 control units and 128 treated units, the median of the 256 absolute differences is the average of the 128th and the 129th smallest, and the criterion equals\n\\[\nf(a)=\\frac12\\min_{i:z_i=1}|(a-1)y_i-t|.\n\\]\nFor every treated unit $j$ such that $y_j\\ne0$, I have $f(a_j)=0$ at the value $a_j=1+t/y_j$, whatever the differences for the other 127 treated units. The criterion can therefore match one treated unit without matching the group means.\n\nFor the exact example, let the control outcomes be 64 values equal to $9$ and 64 equal to $11$, and let the treated outcomes be one $8$, 63 nines, 63 elevens and one $12$, so that both groups have mean $10$. With $t=34/99$ and $a=1+t/8=413/396$ the criterion is exactly zero. The multiplicative adjustment then lowers the treated mean by $10(1-1/a)=170/413$, which is $495/413\\approx1.20$ times $t$.\n\nFor the recalculation, the data generation and the optimizer call of the replication code were run with the random number convention that R used in 2016. I find $t=0.343434$ at the grid entry used for Fig.~2 and $a=1.042615$, which round to the printed $0.34$ and $1.043$. The mean of the Normal outcome is $\\bar y=10.069155$. Under complete randomization the expected treated mean is $\\bar y$. The additive adjustment lowers the treated mean by $t$ under every assignment, and the multiplicative adjustment lowers it by $(1-1/a)\\bar y$ in expectation.\n\\begin{audittable}{lrrr}{Expected fall in the adjusted treated mean of the Normal outcome under the two hypotheses of Fig.~2, from the recalculation.}\nMultiplicative parameter $a$ & Multiplicative, $(1-1/a)\\bar y$ & Additive, $t$ & Excess \\\\\n$1.043$, as printed & $0.4151$ & $0.3434$ & $20.9\\%$ \\\\\n$1.042615$, recalculated & $0.4116$ & $0.3434$ & $19.8\\%$ \\\\\n\\end{audittable}\nI did not simulate power at matched expected mean shifts. I have not shown that the larger shift accounts for the whole difference in power between the two columns, since the multiplicative adjustment also rescales the spread of the treated outcomes by $1/a$.", "problem": "The text says that the two hypotheses of Fig.~2 were chosen to have very similar implications, and it uses this to explain why the comparison of SSR with KS is little affected by the model of effects. Read as a statement about the ordering of the two statistics, the text is correct: SSR has more power than KS for the Normal outcome and less for the skewed outcome under both hypotheses. The two hypotheses are not of equal strength, however, and the difference between the ``Add.'' and ``Mult.'' columns cannot be read as the effect of the model alone. The replication code chooses the multiplicative parameter by minimizing the median, over all 256 units, of the absolute difference between the outcomes that the two models generate under one assignment. Since the 128 control units contribute exact zeros, this median is half the smallest difference among the treated units, and it vanishes as soon as one treated unit matches. The parameter chosen, $1.0426$ (printed as $1.043$), lowers the expected adjusted treated mean of the Normal outcome by $0.412$. The additive hypothesis lowers it by $0.343$, and the multiplicative shift is about 20\\% larger.", "what_survives": "The printed powers stand as reports of the simulations that were run, and the ordering of SSR and KS within each column of Fig.~2 stands. Fig.~1, the network simulation of Fig.~3, which uses a separate grid, and the application do not use this calibration. A reading of the two columns as hypotheses of equal strength does not stand, and neither does the inference that the model of effects matters little at matched strength."}, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-N003", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/"}, "work_id": "P016"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Definition 2 with its display of three cases, the two sentences after it, Proposition 1 and the paragraph that follows. Red boxes mark the second and third cases of the display, 'E_G[tau] + c : theta-hat - E_G[tau] > c' and 'E_G[tau] - c : theta-hat - E_G[tau] < -c'; the words 'across all F_n' in Proposition 1; the words 'a weighted mean can obtain any value in the interval defined by the infimum and supremum of its distribution's support'; and the words 'across all realizations'.", "caption": "Definition 2, Proposition 1 and the paragraph that follows it, PDF p. 2. The marks show the two cases of Definition 2 in which the target equals $\\mathrm{E}_G[\\tau]+c$ or $\\mathrm{E}_G[\\tau]-c$, the words “across all $F_n$” in Proposition 1, the fact given as its proof, and the words “across all realizations”.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E001.png"}, {"alt": "Chart with the exponent k of the weighting tau to the power k on the horizontal axis, at the five equally spaced labels 0, 1, 2, 10 and 100 (spacing is not to scale), and the value on the scale of the effect on the vertical axis, labelled from 0 to 1. Five black dots show the weighted means 1/2, 2/3, 3/4, 11/12 and 101/102, which rise toward 1. A dashed blue horizontal line at 1 is labelled as the printed target on the event with radius c equal to 1/2; the last dot lies just below it. A solid orange horizontal line at 3/4 is labelled as the target on the event with radius d equal to 1/4; the dot for k equal to 2 lies on it. The title reads 'Weighted means of tau, uniform on (0, 1), approach the printed target 1 and do not reach it'.", "caption": "The counterexample to Proposition 1, with $\\tau$ uniform on $(0,1)$ and $c=1/2$. On the event that every observation is an untreated unit with outcome $-1$, the printed target is $1$ (dashed line). The weightings $w=\\tau^k$ with $k=0,1,2,10,100$ have weighted means $1/2$, $2/3$, $3/4$, $11/12$ and $101/102$; by the derivation in the text, no admissible weighting has weighted mean $1$. With radius $d=1/4$ the target on the same event is $3/4$ (solid line), which the weighting $\\tau^2$ attains. All values are exact.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E001-endpoint.png"}], "formal_release": null, "formal_statements": [{"id": "P017-T1", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T1"}, {"id": "P017-T2", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T2"}, {"id": "P017-T3", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T3"}, {"id": "P017-T4", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T4"}, {"id": "P017-T9", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T9"}], "headline": "Proposition 1 fails when the target of Definition 2 is an endpoint of the support of $\\tau$ that has probability zero; with a fixed clipping radius $0<d<c$ in place of $c$, the target is a nonnegative weighted average of causal effects for every empirical distribution.", "historical_evidence_type": "lean_proof", "id": "P017-E001", "location": "Definition 2 and Proposition 1, PDF p. 2 (the pages are not numbered), with the proof sentence and the explanation that follow Proposition 1", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P017-T1", "P017-T2", "P017-T3", "P017-T4", "P017-T9"], "text": {"as_printed": "Definition 2: Let the target parameter\n\\[\n\\theta_{F_n}=\\begin{cases}\\hat\\theta & :\\ |\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le \\hl{c}\\\\ \\mathrm{E}_G[\\tau]+\\hl{c} & :\\ \\hat\\theta-\\mathrm{E}_G[\\tau]>\\hl{c}\\\\ \\mathrm{E}_G[\\tau]-\\hl{c} & :\\ \\hat\\theta-\\mathrm{E}_G[\\tau]<-\\hl{c}\\end{cases},\n\\]\nwhere, as in Assumption 1, $c=\\min(\\sup(\\operatorname{Supp}_G[\\tau])-\\mathrm{E}_G[\\tau],\\ \\mathrm{E}_G[\\tau]-\\inf(\\operatorname{Supp}_G[\\tau]))$.\n\nThe target parameter adapts naturally to the closest value in an interval surrounding $\\mathrm{E}_G[\\tau]$, where the width of the interval is defined by the support of $\\tau$. We formalize how each $\\theta_{F_n}$ is a local average treatment effect.\n\nProposition 1: There exists a nonnegative weighting associated with each empirical distribution $F_n$, $w_{F_n}$, such that \\hl{across all $F_n$}, $\\theta_{F_n}=\\frac{\\mathrm{E}_G[w_{F_n}\\tau]}{\\mathrm{E}_G[w_{F_n}]}$.\n\nA proof of Proposition 1 follows directly from the fact that \\hl{a weighted mean can obtain any value in the interval defined by the infimum and supremum of its distribution's support}. Proposition 1 asserts that \\hl{across all realizations}, the target parameter $\\theta_{F_n}$ corresponds to an average causal effect for at least one subpopulation.", "corrected": "Definition 2: Let $c$ be as in Assumption 1, and fix a finite radius $d$ such that $0<d<c$. Let the target parameter\n\\[\n\\theta_{F_n}=\\begin{cases}\\hat\\theta & :\\ |\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le \\hl{d}\\\\ \\mathrm{E}_G[\\tau]+\\hl{d} & :\\ \\hat\\theta-\\mathrm{E}_G[\\tau]>\\hl{d}\\\\ \\mathrm{E}_G[\\tau]-\\hl{d} & :\\ \\hat\\theta-\\mathrm{E}_G[\\tau]<-\\hl{d}\\end{cases}.\n\\]\n\nProposition 1: Suppose that $\\tau$ has finite expectation and that Assumption 1 holds. There exists a nonnegative weighting associated with each empirical distribution $F_n$, $w_{F_n}$, with $0<\\mathrm{E}_G[w_{F_n}]<\\infty$ and $\\mathrm{E}_G[|w_{F_n}\\tau|]<\\infty$, such that across all $F_n$, $\\theta_{F_n}=\\frac{\\mathrm{E}_G[w_{F_n}\\tau]}{\\mathrm{E}_G[w_{F_n}]}$.\n\nA proof of Proposition 1 follows from the fact that a weighted mean with nonnegative weights can obtain \\hl{every value strictly between} the infimum and supremum of its distribution's support, \\hl{and obtains a finite endpoint of the support if and only if that endpoint has positive probability}. Since $d<c$, every value in $[\\mathrm{E}_G[\\tau]-d,\\ \\mathrm{E}_G[\\tau]+d]$ lies strictly between the infimum and the supremum of the support of $\\tau$. Proposition 1 asserts that across all realizations, the target parameter $\\theta_{F_n}$ corresponds to an average causal effect for at least one subpopulation.\n\nThe later conditions on the target use $d$ in place of $c$; in particular, the hypothesis of Corollary 1 becomes $d\\ge\\max(\\sup(\\operatorname{Supp}[\\hat\\theta])-\\mathrm{E}_G[\\tau],\\ \\mathrm{E}_G[\\tau]-\\inf(\\operatorname{Supp}[\\hat\\theta]))$. If the support of $\\tau$ is unbounded in both directions, then $c=+\\infty$, and the printed target $\\theta_{F_n}=\\hat\\theta$ may be kept.\n\nAlternatively, Definition 2 may be kept with the radius $c$ and Proposition 1 weakened: the weighting exists for each $F_n$ such that $\\theta_{F_n}$ lies strictly between the infimum and the supremum of the support of $\\tau$, or equals one of them and that value has positive probability under $G$. Such a weighting then exists with probability tending to one \\hl{under the paper's consistency and heterogeneity assumptions}, but \\hl{need not exist} across all realizations.", "evidence": "The evidence is a derivation and an exact counterexample. The obstruction at an endpoint also has a Lean proof.\n\n\\emph{The values that a weighted mean attains.} I call a weighting admissible if it is a measurable function $w\\ge0$ of the full data such that $0<\\mathrm{E}_G[w]<\\infty$ and $\\mathrm{E}_G[|w\\tau|]<\\infty$. Let $t$ be a real number. If $p_0=\\Pr_G(\\tau=t)>0$, then the weighting $w=\\mathbf{1}\\{\\tau=t\\}/p_0$ satisfies $\\mathrm{E}_G[w]=1$ and $\\mathrm{E}_G[w\\tau]=t$. Suppose next that $p_-=\\Pr_G(\\tau<t)>0$ and $p_+=\\Pr_G(\\tau>t)>0$, and set\n\\[\nm_-=\\frac{\\mathrm{E}_G[\\tau\\mathbf{1}\\{\\tau<t\\}]}{p_-},\\qquad m_+=\\frac{\\mathrm{E}_G[\\tau\\mathbf{1}\\{\\tau>t\\}]}{p_+},\\qquad a=\\frac{m_+-t}{m_+-m_-}.\n\\]\nThen $m_-<t<m_+$ and $0<a<1$. The weighting\n\\[\nw=\\frac{a}{p_-}\\mathbf{1}\\{\\tau<t\\}+\\frac{1-a}{p_+}\\mathbf{1}\\{\\tau>t\\}\n\\]\nis bounded and nonnegative. Computing the two expectations, I find $\\mathrm{E}_G[w]=a+(1-a)=1$ and $\\mathrm{E}_G[w\\tau]=am_-+(1-a)m_+=t$. Conversely, suppose that $\\Pr_G(\\tau=t)=0$ and $\\Pr_G(\\tau>t)=0$, so that $\\tau<t$ almost surely. If an admissible weighting $w$ had weighted mean $t$, then $\\mathrm{E}_G[w(t-\\tau)]=t\\,\\mathrm{E}_G[w]-\\mathrm{E}_G[w\\tau]=0$. Since $w(t-\\tau)\\ge0$, this implies that $w(t-\\tau)=0$ almost surely. Since $t-\\tau>0$ almost surely, I have $w=0$ almost surely, which contradicts $\\mathrm{E}_G[w]>0$. The case $\\Pr_G(\\tau<t)=0$ is the same, with $w(\\tau-t)$ in place of $w(t-\\tau)$. I conclude that an admissible weighting with weighted mean $t$ exists if and only if\n\\[\n\\Pr_G(\\tau=t)>0,\\quad\\text{or}\\quad \\Pr_G(\\tau<t)>0\\ \\text{ and }\\ \\Pr_G(\\tau>t)>0.\n\\]\nEvery $t$ strictly between the infimum and the supremum of the support of $\\tau$ satisfies the second condition. A finite endpoint of the support satisfies neither condition unless it has positive probability.\n\n\\emph{The counterexample.} Let $\\tau$ be uniform on $(0,1)$, let $B$ equal $-1$ or $1$ with probability $1/2$ each, and let $T$ be a fair treatment assignment, with $\\tau$, $B$ and $T$ independent. Let $Y(0)=B$, $Y(1)=B+\\tau$ and $Y=TY(1)+(1-T)Y(0)$. Then $\\mathrm{E}_G[\\tau]=1/2$, the support of $\\tau$ is $[0,1]$ and $c=1/2$, so that Definition 2 clips $\\hat\\theta$ to $[0,1]$. For $n$ independent units, I take the inverse-probability estimator $\\hat\\theta=n^{-1}\\sum_{i=1}^nX_i$ with $X_i=2T_iY_i-2(1-T_i)Y_i$. The summands are bounded, with mean $1/2$ and variance $53/12$. By the central limit theorem, I have $\\sqrt n(\\hat\\theta-1/2)\\Rightarrow N(0,53/12)$. On the event that $(T_i,Y_i)=(0,-1)$ for all $i$, every $X_i$ equals $2$, and therefore $\\hat\\theta=2$ and $\\theta_{F_n}=1$. Since $\\tau<1$ almost surely, no admissible weighting has weighted mean $1$, by the characterization above.\n\\begin{audittable}{lr}{The counterexample to Proposition 1, with $\\tau$ uniform on $(0,1)$. All values are exact.}\nQuantity & Value \\\\\n$\\mathrm{E}_G[\\tau]$ and $c$ & $1/2$ and $1/2$ \\\\\nMean and variance of $X_i$ & $1/2$ and $53/12$ \\\\\nProbability that $(T_i,Y_i)=(0,-1)$ for every $i$ & $4^{-n}$ \\\\\nEstimate $\\hat\\theta$ on this event & $2$ \\\\\nPrinted target $\\theta_{F_n}$ on this event & $1$ \\\\\nTarget on this event with radius $d=1/4$ & $3/4$ \\\\\n\\end{audittable}\nThe Lean proof covers the following statement: for an integrable weighting $w\\ge0$ with positive expectation, on a population in which $\\tau<1$ almost surely and $w\\tau$ is integrable, the weighted mean is less than $1$. It also covers the uniform distribution on $(0,1)$, for which the value obtained by clipping the estimate $2$ to $[0,1]$ has no such representation. The Lean statements also specify the Gaussian limit, the probability $4^{-n}$ of this event at every $n\\ge1$, and the identity expressing the estimator as an integral against the empirical distribution of the observed pairs. A second example has a failure event of probability $16^{-n}$: its effect is discrete, with values that accumulate at $-1$ and $1$ and do not reach them.", "problem": "Proposition 1 asserts that, for every empirical distribution $F_n$, the target $\\theta_{F_n}$ of Definition 2 equals $\\mathrm{E}_G[w\\tau]/\\mathrm{E}_G[w]$ for some nonnegative weighting $w$. Definition 2 clips $\\hat\\theta$ to the closed interval $[\\mathrm{E}_G[\\tau]-c,\\ \\mathrm{E}_G[\\tau]+c]$. When $c$ is finite, at least one end of this interval is the infimum or the supremum of the support of $\\tau$, and the target equals that value whenever $\\hat\\theta$ lies at or beyond it. A weighted mean with nonnegative weights attains every value strictly between the infimum and the supremum of the support, but it attains a finite endpoint only if the endpoint has positive probability. The fact given as the proof of Proposition 1 includes the endpoints without this condition. If, for example, $\\tau$ is bounded and has a continuous distribution, the endpoint has probability zero. On the realizations where $\\hat\\theta$ lies at or beyond it, the target is a value that no weighting attains, and it is not the average causal effect of any subpopulation. These realizations have probability tending to zero as $n\\to\\infty$, but in the example below they have positive probability at every sample size. The abstract, the introduction and the discussion describe the target as a local average causal effect, and that description depends on Proposition 1.", "what_survives": "The conclusions of Propositions 2, 3 and 4 and the equality $\\Pr(\\hat\\theta=\\theta_{F_n})=1$ of Corollary 1 hold for the target as printed, since they can be derived without Proposition 1. Proposition 1 holds as printed when $\\tau$ has finite support or support unbounded in both directions, and for the printed target the weighted representation exists with probability tending to one. The statement that the printed target is a local average causal effect in every realization does not survive; in particular, under the hypothesis of Corollary 1 the common value of $\\hat\\theta$ and $\\theta_{F_n}$ need not be a weighted average of causal effects. With the radius $d$ the target differs from the printed one on the realizations with $|\\hat\\theta-\\mathrm{E}_G[\\tau]|>d$. No new assumption on $G$ is needed, the conclusions of Propositions 2, 3 and 4 hold with $d$ in place of $c$, and the claim that $\\hat\\theta$ is superefficient for a data-adaptive local average causal effect stands."}, "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-E001", "work": {"arxiv_id": "1601.01413", "doi": "10.1515/jci-2016-0007", "title": "Data-Adaptive Causal Effects and Superefficiency", "url": "https://pmaronow.github.io/llm-audit/works/P017/"}, "work_id": "P017"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The five lines of the proof of Proposition 3. Red boxes mark the words '(theta-tilde minus theta_Fn) is o_p(a_n) for any positive sequence (a_n)'; the expression 'Pr(theta-tilde minus theta_Fn not equal to 0) = 2 Phi(-c root n / sigma)', which runs over a line break; and the term 'o_p(a_n)' in the last line, 'Thus theta-hat minus theta_Fn = o_p(a_n) + o_p(n to the -1/2) = o_p(n to the -1/2), yielding the result'.", "caption": "Proof of Proposition 3, PDF p. 2. The marks show the claim that $(\\tilde\\theta-\\theta_{F_n})$ is $o_p(a_n)$ for any positive sequence, the equality $\\Pr(\\tilde\\theta-\\theta_{F_n}\\ne0)=2\\Phi(-c\\sqrt n/\\sigma)$, and the term $o_p(a_n)$ in the last line, where the claim is used.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E002.png"}, {"alt": "Proposition 4 and the whole of its proof, including a display of three lines. Red boxes mark the words 'Within c of E_G[tau], the squared deviation is zero' and the first line of the display, 'E_G[(theta-tilde minus theta_Fn) squared] is at most 2 times the integral from c to infinity of x squared times the normal density with variance sigma squared over n', which is printed without a differential.", "caption": "Proposition 4 and its proof, PDF p. 3. The marks show the sentence “Within $c$ of $\\mathrm{E}_G[\\tau]$, the squared deviation is zero” and the first line of the display, which bounds $\\mathrm{E}_G[(\\tilde\\theta-\\theta_{F_n})^2]$ by a Gaussian tail integral.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E002-b.png"}, {"alt": "Chart with two rows on a logarithmic horizontal axis labelled from 10 to the -7 to 1. In the row labelled 'Probability Pr(theta-tilde minus theta_Fn not equal to 0)', an open blue diamond at 5.7 times 10 to the -7 is joined by a thin line to a filled orange circle at 1. In the row labelled 'Mean square E[(theta-tilde minus theta_Fn) squared]', an open blue diamond at 6.2 times 10 to the -7 is joined to a filled orange circle at 1.6 times 10 to the -3. The legend says that the diamonds are the values as printed, 2 Phi(-c root n / sigma) and the tail integral, and that the circles are the actual values in the example. The title reads 'In the example at n = 25, the printed probability and the printed bound are far below the actual values'.", "caption": "The counterexample at $n=25$, with $\\mathrm{E}_G[\\tau]=0$, $c=\\sigma=1$ and $\\hat\\theta=\\tilde\\theta+1/n$, on a logarithmic scale. The printed probability $2\\Phi(-5)\\approx5.7\\times10^{-7}$ stands against the actual value $1$, and the printed tail integral $\\approx6.2\\times10^{-7}$ against the actual mean square $\\approx1.6\\times10^{-3}$. The mean square values are from numerical integration. They are consistent with the exact bounds: the tail integral is at most $15/15625$, and the mean square is at least $21/15625$.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E002-printed-vs-actual.png"}], "formal_release": null, "formal_statements": [{"id": "P017-T5", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T5"}, {"id": "P017-T6", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T6"}, {"id": "P017-T7", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T7"}, {"id": "P017-T8", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T8"}, {"id": "P017-T11", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T11"}, {"id": "P017-T14", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T14"}, {"id": "P017-T15", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T15"}], "headline": "The proofs of Propositions 3 and 4 clip the normal part $\\tilde\\theta$ where Definition 2 clips $\\hat\\theta$; the probability equality, the rate claimed for $\\tilde\\theta-\\theta_{F_n}$ and the displayed bound on the mean square are false in general, and both propositions hold as stated by a direct clipping argument.", "id": "P017-E002", "location": "Proof of Proposition 3, PDF p. 2, the claim that $(\\tilde\\theta-\\theta_{F_n})$ is $o_p(a_n)$ and the equality that ends in $2\\Phi(-c\\sqrt n/\\sigma)$; proof of Proposition 4, PDF p. 3, the sentence “Within $c$ of $\\mathrm{E}_G[\\tau]$, the squared deviation is zero” and the first line of the display", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P017-T11", "P017-T14", "P017-T15", "P017-T5", "P017-T6", "P017-T7", "P017-T8"], "text": {"as_printed": "\\emph{Proof of Proposition 3, PDF p.~2.} Proof: Decompose $\\hat\\theta$ into $\\tilde\\theta=N(\\mathrm{E}_G[\\tau],\\sigma^2/n)$ and $u=o_p(n^{-1/2})$, so that $\\hat\\theta=\\tilde\\theta+u$. Since \\hl{$(\\tilde\\theta-\\theta_{F_n})$ is $o_p(a_n)$ for any positive sequence $(a_n)$}, the rate of convergence of $\\hat\\theta$ is at worst governed by the bound ensured by $u$'s $o_p(n^{-1/2})$ convergence. To prove the claim, note that for any positive $\\varepsilon$, $\\Pr\\left(\\frac{|\\tilde\\theta-\\theta_{F_n}|}{a_n}\\ge\\varepsilon\\right)\\le\\Pr\\left(\\tilde\\theta-\\theta_{F_n}\\ne0\\right)\\hl{=2\\Phi(-c\\sqrt n/\\sigma)}$, where $\\Phi(.)$ denotes the standard Normal CDF. Since $\\lim_{n\\to\\infty}2\\Phi(-c\\sqrt n/\\sigma)=0$, \\hl{$(\\tilde\\theta-\\theta_{F_n})$ is $o_p(a_n)$}. Thus $\\hat\\theta-\\theta_{F_n}=\\hl{o_p(a_n)}+o_p(n^{-1/2})=o_p(n^{-1/2})$, yielding the result.\n\n\\emph{Proof of Proposition 4, PDF p.~3, from its second sentence.} To obtain the rate of convergence of the mean square error of $\\tilde\\theta$, we integrate over its squared deviation from the target parameter. \\hl{Within $c$ of $\\mathrm{E}_G[\\tau]$, the squared deviation is zero}, thus we need only integrate over the squared deviation over the tails of the normal distribution. To ease calculations, we obtain an upper bound by integrating over the squared deviation from $\\mathrm{E}_G[\\tau]$, rather than from $\\theta_{F_n}$:\n\\[\n\\begin{aligned}\n\\mathrm{E}_G[(\\tilde\\theta-\\theta_{F_n})^2]&\\hl{\\le}2\\int_c^\\infty x^2\\frac{\\sqrt n}{\\sigma\\sqrt{2\\pi}}e^{-\\frac{x^2n}{2\\sigma^2}}\\\\\n&=c\\sigma\\sqrt{\\frac{2}{\\pi}}\\frac{e^{-\\frac{c^2n}{2\\sigma^2}}}{\\sqrt n}+2\\sigma^2\\frac{\\Phi\\left(\\frac{-c\\sqrt n}{\\sigma}\\right)}{n}\\\\\n&=o(n^{-1}).\n\\end{aligned}\n\\]", "corrected": "Let $\\mu=\\mathrm{E}_G[\\tau]$. For finite $c$, let $C(x)=\\min(\\mu+c,\\max(\\mu-c,x))$ and $r(x)=x-C(x)$, so that $\\theta_{F_n}=C(\\hat\\theta)$ and $\\hat\\theta-\\theta_{F_n}=r(\\hat\\theta)$. If $c=+\\infty$, then $\\theta_{F_n}=\\hat\\theta$ and both conclusions are immediate.\n\n\\emph{Proof of Proposition 3.} Since $\\hat\\theta$ is root-$n$ consistent, $\\Pr(|\\hat\\theta-\\mu|>c)\\to0$. The residual $r(\\hat\\theta)$ is zero whenever $|\\hat\\theta-\\mu|\\le c$. Hence for any positive sequence $(a_n)$ and any positive $\\varepsilon$,\n\\[\n\\Pr\\left(\\frac{|\\hl{\\hat\\theta}-\\theta_{F_n}|}{a_n}>\\varepsilon\\right)\\le\\hl{\\Pr\\left(|\\hat\\theta-\\mu|>c\\right)}\\longrightarrow0.\n\\]\nThus $\\hat\\theta-\\theta_{F_n}$ is $o_p(a_n)$ for any positive sequence $(a_n)$, and the choice $a_n=n^{-1/2}$ yields the result.\n\n\\emph{Proof of Proposition 4.} Let $\\hat\\theta=\\tilde\\theta+\\varepsilon/\\sqrt n$ with $\\tilde\\theta=N(\\mu,\\sigma^2/n)$. Since $r$ is 1-Lipschitz, $|r(\\hat\\theta)|\\le|r(\\tilde\\theta)|+|\\varepsilon|/\\sqrt n$, and the inequality $(A+B)^2\\le2A^2+2B^2$ gives\n\\[\n\\mathrm{E}[(\\hat\\theta-\\theta_{F_n})^2]\\le2\\,\\mathrm{E}[\\hl{r(\\tilde\\theta)^2}]+\\hl{\\frac{2\\,\\mathrm{E}[\\varepsilon^2]}{n}}.\n\\]\nWithin $c$ of $\\mu$, the residual $r(\\tilde\\theta)$ is zero, and elsewhere $r(\\tilde\\theta)^2\\le(\\tilde\\theta-\\mu)^2$. Hence only the tails of the normal distribution contribute:\n\\[\n\\begin{aligned}\n\\mathrm{E}[\\hl{r(\\tilde\\theta)^2}]&\\le2\\int_c^\\infty x^2\\frac{\\sqrt n}{\\sigma\\sqrt{2\\pi}}e^{-\\frac{x^2n}{2\\sigma^2}}\\,dx\\\\\n&=c\\sigma\\sqrt{\\frac{2}{\\pi}}\\frac{e^{-\\frac{c^2n}{2\\sigma^2}}}{\\sqrt n}+2\\sigma^2\\frac{\\Phi\\left(\\frac{-c\\sqrt n}{\\sigma}\\right)}{n}\\\\\n&=o(n^{-1}).\n\\end{aligned}\n\\]\nThe same expectation is at most $\\mathrm{E}[(\\tilde\\theta-\\mu)^4]/c^2=3\\sigma^4/(c^2n^2)$. Since $\\mathrm{E}[\\varepsilon^2]=o(n^{-1/2})$, it follows that\n\\[\n\\mathrm{E}[(\\hat\\theta-\\theta_{F_n})^2]\\le\\frac{6\\sigma^4}{c^2n^2}+\\frac{2\\,\\mathrm{E}[\\varepsilon^2]}{n}=o(n^{-1}).\n\\]", "evidence": "I give an exact counterexample to the three steps and derive both propositions in the corrected proofs below.\n\nLet the effects equal $-1$ or $1$ with probability $1/2$ each, so that $\\mathrm{E}_G[\\tau]=0$ and $c=1$, and let $X_1,\\ldots,X_n$ be independent $N(0,1)$ observations. Set $\\tilde\\theta=\\bar X_n$ and $\\hat\\theta=\\tilde\\theta+1/n$. Then $\\sigma=1$ and $u=1/n=o_p(n^{-1/2})$. Further, $\\sqrt n\\,\\hat\\theta=N(0,1)+\\varepsilon$ with $\\varepsilon=n^{-1/2}$ and $\\mathrm{E}[\\varepsilon^2]=1/n=o(n^{-1/2})$, and the assumptions of both propositions hold. The difference $\\tilde\\theta-\\theta_{F_n}$ equals $-1/n$ when $|\\hat\\theta|\\le1$, and outside this event it vanishes only if $\\tilde\\theta=1$, which has probability zero. I therefore have $\\Pr(\\tilde\\theta-\\theta_{F_n}\\ne0)=1$ for all $n$, while the printed value is $2\\Phi(-\\sqrt n)=\\Pr(|\\tilde\\theta|>1)\\le1/n$. For $n\\ge2$, the event $|\\tilde\\theta|\\le1/2$ has probability at least $1-4/n$, and on it $\\tilde\\theta-\\theta_{F_n}=-1/n$. It follows that $n(\\tilde\\theta-\\theta_{F_n})\\to-1$ in probability, and $\\tilde\\theta-\\theta_{F_n}$ is not $o_p(a_n)$ for $a_n=1/n$. At $n=25$, using the same event, I obtain\n\\[\n\\mathrm{E}[(\\tilde\\theta-\\theta_{F_n})^2]\\ge\\frac{1}{625}\\cdot\\frac{21}{25}=\\frac{21}{15625},\n\\]\nwhile the printed tail integral equals $\\mathrm{E}[\\tilde\\theta^2\\mathbf{1}\\{|\\tilde\\theta|>1\\}]$, which is at most $\\mathrm{E}[\\tilde\\theta^6]=15/15625$. The printed inequality fails by at least $6/15625$.\n\\begin{audittable}{lll}{The printed calculations in the counterexample, at $n=25$ in the first three rows. The mean square values are from numerical integration; decimal values are rounded.}\nQuantity & As printed & In the example \\\\\n$\\Pr(\\tilde\\theta-\\theta_{F_n}\\ne0)$ & $2\\Phi(-5)\\approx5.7\\times10^{-7}$ & $1$ \\\\\n$\\mathrm{E}[(\\tilde\\theta-\\theta_{F_n})^2]$, decimal value & at most $6.2\\times10^{-7}$ & $1.6\\times10^{-3}$ \\\\\n$\\mathrm{E}[(\\tilde\\theta-\\theta_{F_n})^2]$, exact bound & at most $15/15625$ & at least $21/15625$ \\\\\nLimit in probability of $n(\\tilde\\theta-\\theta_{F_n})$ & $0$ & $-1$ \\\\\n\\end{audittable}\nThe corrected proofs use three facts. First, Definition 1 implies that $\\sqrt n(\\hat\\theta-\\mathrm{E}_G[\\tau])=O_p(1)$, and hence that $\\hat\\theta$ is consistent. Second, the function $r(x)=x-C(x)$, where $C$ is the clip to the interval, is continuous and piecewise linear with slopes $1$, $0$ and $1$, so that $|r(y)-r(x)|\\le|y-x|$ for all $x$ and $y$. Third, a variable $W\\sim N(0,\\sigma^2/n)$ has $\\mathrm{E}[W^4]=3\\sigma^4/n^2$. Note that the argument for Proposition 4 does not use independence of $\\tilde\\theta$ and $\\varepsilon$.", "problem": "Definition 2 defines $\\theta_{F_n}$ as the clip of the estimator $\\hat\\theta=\\tilde\\theta+u$ to the interval $[\\mathrm{E}_G[\\tau]-c,\\ \\mathrm{E}_G[\\tau]+c]$. Both proofs compute with the normal part $\\tilde\\theta$ as though $\\theta_{F_n}$ were the clip of $\\tilde\\theta$. On the event $|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c$ the target equals $\\hat\\theta$, which implies that $\\tilde\\theta-\\theta_{F_n}=-u$, and $u$ is not zero in general. In the proof of Proposition 3, the probability $\\Pr(\\tilde\\theta-\\theta_{F_n}\\ne0)$ need not equal $2\\Phi(-c\\sqrt n/\\sigma)$, and $\\tilde\\theta-\\theta_{F_n}$ need not be $o_p(a_n)$ for every positive sequence $(a_n)$. In the proof of Proposition 4, the squared deviation need not vanish when $|\\tilde\\theta-\\mathrm{E}_G[\\tau]|\\le c$; when $|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c$, it equals $u^2$. The Gaussian tail integral does not bound $\\mathrm{E}[(\\tilde\\theta-\\theta_{F_n})^2]$ in general. The three steps do hold for some remainders, for instance a remainder that is zero when $\\tilde\\theta$ is in the interval and moves $\\tilde\\theta$ away from the interval otherwise. They do not follow from the assumptions of the two propositions. Both propositions are true as stated, but the printed proofs do not establish them.", "what_survives": "Propositions 3 and 4 hold as stated under their printed assumptions, with no further assumption, and the superefficiency and mean square rates stated in the abstract, the introduction and the discussion hold for the Definition 2 target. Proposition 3 holds for any consistent estimator of $\\mathrm{E}_G[\\tau]$, and then $\\hat\\theta-\\theta_{F_n}=o_p(a_n)$ for any positive sequence $(a_n)$. The closed-form evaluation of the Gaussian tail integral is correct, and the intermediate conclusion $\\mathrm{E}[(\\tilde\\theta-\\theta_{F_n})^2]=o(n^{-1})$ is true, although the displayed bound does not establish it. Definition 2, Proposition 2 and Corollary 1 do not use these steps. The corrected proofs also apply to the target of \\fid{P017-E001}, with a fixed finite radius $0<d<c$ in place of $c$."}, "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-E002", "work": {"arxiv_id": "1601.01413", "doi": "10.1515/jci-2016-0007", "title": "Data-Adaptive Causal Effects and Superefficiency", "url": "https://pmaronow.github.io/llm-audit/works/P017/"}, "work_id": "P017"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The last two lines of the proof of Proposition 3, followed by a paragraph of five lines that begins 'In short, Proposition 3 demonstrates that the probability that theta-hat falls inside the support of the effect distribution converges to one quickly'. A red box marks the words 'inside the support of the effect distribution', which run over a line break.", "caption": "End of the proof of Proposition 3 and the paragraph that follows it, PDF p. 2. The marked words name the support of the effect distribution as the region in which the estimation error is zero.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P017-T13", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T13"}], "headline": "The event on which the estimation error is zero is that $\\hat\\theta$ lies in the interval $[\\mathrm{E}_G[\\tau]-c,\\ \\mathrm{E}_G[\\tau]+c]$ of Definition 2, which is not in general the support of the effect distribution.", "id": "P017-E003", "location": "Paragraph after the proof of Proposition 3, PDF p. 2, first sentence (“In short, Proposition 3 demonstrates …”)", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P017-T13"], "text": {"as_printed": "In short, Proposition 3 demonstrates that the probability that $\\hat\\theta$ falls \\hl{inside the support of the effect distribution} converges to one quickly; conditional on this event, then estimation error is zero (as the target parameter takes on the value as the estimator with probability one).", "corrected": "In short, Proposition 3 demonstrates that the probability that $\\hat\\theta$ falls \\hl{inside the interval $[\\mathrm{E}_G[\\tau]-c,\\ \\mathrm{E}_G[\\tau]+c]$} converges to one \\del{quickly}; on this event, estimation error is zero (as the target parameter takes on the same value as the estimator with probability one).\n\nThe deletion of ``quickly'' is explained in \\fid{P017-N001}.", "intended_meaning": "By Definition 2, $\\hat\\theta=\\theta_{F_n}$ exactly when $|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c$. This is the event that the sentence intends: the sentence after the proof of Corollary 1 names the interval $[\\mathrm{E}_G[\\tau]-c,\\ \\mathrm{E}_G[\\tau]+c]$. The support of $\\tau$ is a different set in general. If $\\tau$ is exponential with mean $1$, then the support is $[0,\\infty)$ and the interval is $[0,2]$, and an estimate $\\hat\\theta=3$ lies in the support with error $1$. If $\\tau$ equals $-1$ or $1$ with probability $1/2$ each, then the interval is $[-1,1]$, and an estimate $\\hat\\theta=0.03$ has error zero and lies outside the support $\\{-1,1\\}$. With the interval in place of the support, the sentence names the correct event, and since $\\hat\\theta$ is consistent and $c>0$, the probability of this event tends to one. No result depends on the sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-E003", "work": {"arxiv_id": "1601.01413", "doi": "10.1515/jci-2016-0007", "title": "Data-Adaptive Causal Effects and Superefficiency", "url": "https://pmaronow.github.io/llm-audit/works/P017/"}, "work_id": "P017"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Corollary 1, which ends 'Then Pr(theta-hat = theta_Fn) = 1', followed by three lines that begin 'A proof of Corollary 1 follows by noting that Pr(|theta-hat minus E_G[tau]| is at most c) = 0, and applying Definition 2'. A red box marks '= 0'.", "caption": "Corollary 1 and the first lines of the paragraph that follows it, PDF p. 2. The mark shows the value $0$ printed for $\\Pr(|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c)$.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E004.png"}, {"alt": "Four lines of the proof of Proposition 4, followed by a display of three lines. A red box marks the integral on the right of the first line of the display, '2 times the integral from c to infinity of x squared times root n over sigma root 2 pi times e to the power minus x squared n over 2 sigma squared'. No dx follows the integrand.", "caption": "Proof of Proposition 4, PDF p. 3: the four lines before the display, and the display. The marked integral is printed without a differential.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P017-T10", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T10"}, {"id": "P017-T11", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T11"}], "headline": "In the proof of Corollary 1, the probability $\\Pr(|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c)$ is $1$ and not $0$; in the proof of Proposition 4, the Gaussian integral lacks its differential $dx$.", "id": "P017-E004", "location": "Sentence after Corollary 1, PDF p. 2 (“A proof of Corollary 1 follows by noting that …”); first line of the display in the proof of Proposition 4, PDF p. 3", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P017-T10", "P017-T11"], "text": {"as_printed": "\\emph{After Corollary 1, PDF p.~2.} A proof of Corollary 1 follows by noting that $\\Pr(|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c)=\\hl{0}$, and applying Definition 2.\n\n\\emph{Proof of Proposition 4, PDF p.~3, first line of the display. No differential is printed.}\n\\[\n\\mathrm{E}_G[(\\tilde\\theta-\\theta_{F_n})^2]\\le2\\int_c^\\infty x^2\\frac{\\sqrt n}{\\sigma\\sqrt{2\\pi}}e^{-\\frac{x^2n}{2\\sigma^2}}\n\\]", "corrected": "A proof of Corollary 1 follows by noting that $\\Pr(|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c)=\\hl{1}$, and applying Definition 2.\n\n\\emph{Proof of Proposition 4, the integral in the first line of the display.}\n\\[\n2\\int_c^\\infty x^2\\frac{\\sqrt n}{\\sigma\\sqrt{2\\pi}}e^{-\\frac{x^2n}{2\\sigma^2}}\\hl{\\,dx}\n\\]", "evidence": "The evidence is a check of the page and a derivation. Let $\\mu=\\mathrm{E}_G[\\tau]$. The hypothesis $c\\ge\\max(\\sup(\\operatorname{Supp}[\\hat\\theta])-\\mu,\\ \\mu-\\inf(\\operatorname{Supp}[\\hat\\theta]))$ implies that $\\inf(\\operatorname{Supp}[\\hat\\theta])\\ge\\mu-c$ and $\\sup(\\operatorname{Supp}[\\hat\\theta])\\le\\mu+c$. Then $|\\hat\\theta-\\mu|\\le c$ almost surely, and by Definition 2, I have $\\Pr(\\hat\\theta=\\theta_{F_n})=1$. For the integral, set $s=\\sigma/\\sqrt n$ and $a=c/s$, and let $\\phi$ be the standard normal density. Substituting $x=sz$ and integrating by parts, I obtain\n\\[\n2\\int_c^\\infty x^2\\frac{\\sqrt n}{\\sigma\\sqrt{2\\pi}}e^{-\\frac{x^2n}{2\\sigma^2}}\\,dx=2s^2\\int_a^\\infty z^2\\phi(z)\\,dz=2s^2\\bigl(a\\phi(a)+\\Phi(-a)\\bigr),\n\\]\nwhich is the second line of the printed display. Numerical integration agrees with the printed closed form at three choices of $c$, $\\sigma$ and $n$.", "problem": "The hypothesis of Corollary 1 places the support of $\\hat\\theta$ inside the interval $[\\mathrm{E}_G[\\tau]-c,\\ \\mathrm{E}_G[\\tau]+c]$, so that the event $|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c$ has probability one. The proof sentence assigns it probability zero, which contradicts the corollary and the sentence that follows the proof. In the proof of Proposition 4, the integral is printed without $dx$. Both are misprints. In the same display the expectation carries a subscript $G$ that the statement of Proposition 4 and the sentence after the display omit. This is a difference of notation and not an error, since the sampling expectation is the one induced by $G$.", "what_survives": "Corollary 1 and the closed-form evaluation of the integral are correct as printed, and no result changes. The inequality in which the integral appears is a separate matter, treated in \\fid{P017-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-E004", "work": {"arxiv_id": "1601.01413", "doi": "10.1515/jci-2016-0007", "title": "Data-Adaptive Causal Effects and Superefficiency", "url": "https://pmaronow.github.io/llm-audit/works/P017/"}, "work_id": "P017"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Three lines of text that begin 'In short, Proposition 3 demonstrates that the probability that theta-hat falls inside the support of the effect distribution converges to one quickly; conditional on this event, then estimation error is zero'. A red box marks the word 'quickly'.", "caption": "Paragraph after the proof of Proposition 3, PDF p. 2, first three lines. The marked word attributes a fast rate to the probability that the estimation error is zero.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-N001.png"}], "formal_release": null, "formal_statements": [{"id": "P017-T12", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T12"}, {"id": "P017-T13", "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-T13"}], "headline": "Under the assumptions of Proposition 3, the probability that $\\hat\\theta$ equals its target tends to one, but at no particular rate; the word “quickly” should be deleted.", "id": "P017-N001", "location": "Paragraph after the proof of Proposition 3, PDF p. 2, first sentence, the word “quickly”", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P017-T12", "P017-T13"], "text": {"as_printed": "In short, Proposition 3 demonstrates that the probability that $\\hat\\theta$ falls inside the support of the effect distribution converges to one \\hl{quickly}; conditional on this event, then estimation error is zero (as the target parameter takes on the value as the estimator with probability one).", "corrected": "In short, Proposition 3 demonstrates that the probability that $\\hat\\theta$ falls \\hl{inside the interval $[\\mathrm{E}_G[\\tau]-c,\\ \\mathrm{E}_G[\\tau]+c]$} converges to one \\del{quickly}; on this event, estimation error is zero (as the target parameter takes on the same value as the estimator with probability one).\n\nIn precise terms: if Assumption 1 holds and $\\hat\\theta$ is a root-$n$ consistent and asymptotically normal estimator of $\\mathrm{E}_G[\\tau]$, then $\\Pr(|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c)\\to1$, and on this event $\\hat\\theta=\\theta_{F_n}$. No rate of convergence of this probability follows from these assumptions. The interval wording is explained in \\fid{P017-E003}.", "evidence": "The evidence is an exact counterexample. Let $(p_n)$ be any sequence in $(0,1]$ that tends to zero. Let the effect $\\tau$ equal $-1$ or $1$ with probability $1/4$ each and be uniform on $[-1,1]$ otherwise, so that $\\mathrm{E}_G[\\tau]=0$, the support of $\\tau$ is $[-1,1]$ and $c=1$. Let the data contain independent observations $X_i\\sim N(0,1)$ and $U_i$ uniform on $[0,1)$. Let $B_n$ be the indicator of the event that the fractional part of $\\sum_{i=1}^nU_i$ is less than $p_n$. I have $\\Pr(B_n=1)=p_n$, and $B_n$ is independent of $\\bar X_n$. Let\n\\[\nA_n=\\max\\{-1/2,\\min(\\bar X_n,1/2)\\},\\qquad\\hat\\theta=A_n+10B_n.\n\\]\nThe estimator $\\hat\\theta$ is a symmetric function of the sample. I have $\\sqrt n\\,\\hat\\theta=\\sqrt n\\,\\bar X_n+\\varepsilon_n$, where $\\sqrt n\\,\\bar X_n\\sim N(0,1)$ and $\\Pr(\\varepsilon_n\\ne0)\\le4/n+p_n\\to0$, and Definition 1 holds with $\\sigma^2=1$. If $B_n=0$, then $\\hat\\theta=A_n\\in[-1/2,1/2]$ and $\\theta_{F_n}=\\hat\\theta$. If $B_n=1$, then $\\hat\\theta\\ge19/2$, $\\theta_{F_n}=1$ and the error is at least $17/2$. I conclude that\n\\[\n\\Pr(\\hat\\theta\\ne\\theta_{F_n})=\\Pr(|\\hat\\theta|>1)=p_n.\n\\]\nFor $p_n=1/(1+\\lfloor\\log_2n\\rfloor)$, this probability is $1/10$ at $n=1000$ and $1/40$ at $n=10^{12}$, and it is not $O(n^{-\\alpha})$ for any $\\alpha>0$. The conclusion of Proposition 3 holds in the example, since $\\Pr(\\sqrt n|\\hat\\theta-\\theta_{F_n}|>\\delta)\\le p_n$ for all $\\delta>0$.", "problem": "The sentence follows a proof in which the expression $2\\Phi(-c\\sqrt n/\\sigma)$ appears, and in that context ``quickly'' says that the probability of zero estimation error tends to one at a fast rate. No rate follows from the assumptions of Proposition 3. Definition 1 requires only that $\\sqrt n(\\hat\\theta-\\mathrm{E}_G[\\tau])=N(0,\\sigma^2)+o_p(1)$, and the $o_p(1)$ term may be large on an event whose probability tends to zero arbitrarily slowly. For a decomposition $\\hat\\theta=\\tilde\\theta+u$ with $\\tilde\\theta=N(\\mathrm{E}_G[\\tau],\\sigma^2/n)$, by the union bound, I have\n\\[\n\\Pr\\left(|\\hat\\theta-\\mathrm{E}_G[\\tau]|>c\\right)\\le2\\Phi\\left(-\\frac{c\\sqrt n}{2\\sigma}\\right)+\\Pr\\left(|u|>c/2\\right),\n\\]\nand the condition $u=o_p(n^{-1/2})$ does not control the rate of the second term. The sentence prints no specific rate. If ``quickly'' is read as meaning only that the probability tends to one, then the sentence asserts no rate and needs no correction.", "what_survives": "Proposition 3 is unaffected: its conclusion uses only that the probability of the event $|\\hat\\theta-\\mathrm{E}_G[\\tau]|\\le c$ tends to one. No later result uses a rate for this probability. Proposition 4 makes its own stronger assumption on the remainder, and its conclusion holds under that assumption; the example above does not satisfy it."}, "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-N001", "work": {"arxiv_id": "1601.01413", "doi": "10.1515/jci-2016-0007", "title": "Data-Adaptive Causal Effects and Superefficiency", "url": "https://pmaronow.github.io/llm-audit/works/P017/"}, "work_id": "P017"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Two entries of the reference list. Entry 9 reads 'Balzer LB, Petersen ML, van der Laan MJ. Targeted estimation and inference for the sample average treatment effect Bepress, 2015 Berkeley, CA.' Entry 10 reads 'van der Laan MJ, Hubbard AE, Pajouh SK. Statistical inference for data adaptive target parameters. Princeton, NJ: Bepress, 2013.' A red box marks 'Princeton, NJ'.", "caption": "References 9 and 10, PDF p. 4. The mark shows the place of publication printed for reference 10. Reference 9, also from Bepress, has Berkeley, CA.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-N003.png"}, {"alt": "The first entry of the reference list, 'Crump RK, Hotz VJ, Imbens GW, Mitnik OA. Dealing with limited overlap in estimation of average treatment effects. Biometrika 2009.', and the first line of the second entry. A red box marks 'Biometrika 2009.'", "caption": "Reference 1, PDF p. 3. The marked words are all that the entry prints after the title: the journal and the year, with no volume, issue or pages.", "url": "https://pmaronow.github.io/llm-audit/works/P017/exhibits/P017-N003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Reference 10 is Working Paper 314 of the U.C. Berkeley Division of Biostatistics Working Paper Series, published in Berkeley, CA, and not in Princeton, NJ; reference 1 lacks its volume, issue and pages, which are 96(1):187–199.", "id": "P017-N003", "location": "References 1 (PDF p. 3) and 10 (PDF p. 4)", "pdf_page": 3, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "1. Crump RK, Hotz VJ, Imbens GW, Mitnik OA. Dealing with limited overlap in estimation of average treatment effects. \\hl{Biometrika 2009.}\n\n10. van der Laan MJ, Hubbard AE, Pajouh SK. Statistical inference for data adaptive target parameters. \\hl{Princeton, NJ}: Bepress, 2013.", "corrected": "1. Crump RK, Hotz VJ, Imbens GW, Mitnik OA. Dealing with limited overlap in estimation of average treatment effects. Biometrika 2009\\hl{;96(1):187--199}.\n\n10. van der Laan MJ, Hubbard AE, Pajouh SK. Statistical inference for data adaptive target parameters. \\hl{U.C. Berkeley Division of Biostatistics Working Paper Series, Working Paper 314. Berkeley, CA}: Bepress, 2013.", "evidence": "The evidence is a check of the page against public bibliographic sources. The page of the series for the working paper (\\url{https://biostats.bepress.com/ucbbiostat/paper314/}) reads ``U.C. Berkeley Division of Biostatistics Working Paper Series. Working Paper 314'', with the date June 2013. A later article by Balzer, Petersen and van der Laan cites the same report as ``Berkeley: Division of Biostatistics, University of California; 2013''. The publisher's page for the article of Crump, Hotz, Imbens and Mitnik has Biometrika 96(1):187--199, 2009. The words quoted in Section 3 appear on p.~188 in the indexed text of a copy in the journal's format that one of the authors has posted. I did not obtain the working paper itself or the publisher's full text of the article. The sources do not show at what stage the place of publication was misprinted.", "problem": "Reference 10 names Princeton, NJ as the place of publication. The item is Working Paper 314 of the U.C. Berkeley Division of Biostatistics Working Paper Series, dated June 2013, and its place of publication is Berkeley, CA. Reference 1 has the journal and the year only, although Section 3 cites its p.~188. Both entries still identify the works, and no result depends on them.", "what_survives": "The discussion and the results are unaffected. The year 2009, the citation of p.~188 and the quotation in Section 3 are correct as printed."}, "url": "https://pmaronow.github.io/llm-audit/works/P017/#P017-N003", "work": {"arxiv_id": "1601.01413", "doi": "10.1515/jci-2016-0007", "title": "Data-Adaptive Causal Effects and Superefficiency", "url": "https://pmaronow.github.io/llm-audit/works/P017/"}, "work_id": "P017"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Lemma 1 with equations (1) and (2). Equation (2) sets the variance of the sum of X_d epsilon_d equal to a sum over d of two terms labelled A and B. Red boxes mark 'Var[epsilon_d | X]' inside term A and 'Cov[epsilon_d, epsilon_d-prime | X]' inside term B. The letter E of term B is printed in the summation subscript.", "caption": "Lemma 1, p. 566 (PDF p. 3). The marks are on the conditional variance in term $A$ and on the conditional covariance in term $B$ of equation (2).", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E001.png"}, {"alt": "Crop of equation (6), a chain of three equalities for the variance of the sum of X_d epsilon_d. The first line is a double sum of Cov[X_d epsilon_d, X_d-prime epsilon_d-prime]. A red box marks the summand of the second line, the expectation of X_d X_d-prime-transpose times Cov[epsilon_d, epsilon_d-prime | X]. The third line repeats the right side of equation (2), with E in front of the braces in both terms.", "caption": "Appendix A, equation (6), p. 573 (PDF p. 10). The mark is on the second line, where the covariances of the scores are replaced by expectations of conditional covariances of the residuals.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E001-b.png"}, {"alt": "Two line charts side by side, one for the intercept entry and one for the slope entry of V. The horizontal axis is the number of units N, from 3 to 200 on a logarithmic scale. In the left chart a blue curve falls from 16.88 at N = 3 to 11.68 at N = 200, toward a grey horizontal line at 11.63. In the right chart a blue curve falls from 4.02 at N = 3 to 2.83 at N = 200, toward a grey horizontal line at 2.82. In both charts an orange dashed line lies at zero for every N, the value given by equation (2) as printed.", "caption": "The intercept and slope entries of $V$ in equation (1) in the first example, for $N$ from 3 to 200 units. The blue curves use the exact variance of the score sum, with dots at $N=3,4,5$, where it was also obtained by direct integration, and the grey lines are the limits $3664/315$ and $296/105$. The orange dashed lines are the value that results when equation (2) is used as printed, zero at every $N$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E001-variance.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T1", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T1"}, {"id": "P018-T2", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T2"}, {"id": "P018-T21", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T21"}, {"id": "P018-T26", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T26"}, {"id": "P018-T27", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T27"}], "headline": "Equation (2) expresses the variance of the score sum through conditional variances and covariances of the residuals, which can omit $\\operatorname{Var}(\\mathrm{E}[\\sum_dX_d\\epsilon_d\\mid\\mathbf{X}])$ when the conditional mean is not linear; the terms should be the unconditional moments $\\mathrm{E}\\{X_dX_d'\\epsilon_d^2\\}$ and $\\mathrm{E}\\{X_dX_{d'}'\\epsilon_d\\epsilon_{d'}\\}$.", "id": "P018-E001", "location": "Lemma 1, equation (2), p. 566 (PDF p. 3); Appendix A, equation (6), second and third lines, p. 573 (PDF p. 10). The same form recurs in $\\Sigma$ of Appendix B, p. 573, in $V_r$, p. 568 (PDF p. 5), and in $V_w$, p. 576 (PDF p. 13)", "pdf_page": 3, "printed_page": "566", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P018-T1", "P018-T2", "P018-T21", "P018-T26", "P018-T27"], "text": {"as_printed": "p.~566: ``Lemma 1. Suppose data and $\\hat\\beta$ as defined above. As $N\\to\\infty$, the asymptotic distribution of $\\sqrt{N}(\\hat\\beta-\\beta)$ has mean zero and variance\n\\[\nV=\\frac{N}{D^2}\\,\\mathrm{E}[X_dX_d']^{-1}\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big]\\mathrm{E}[X_dX_d']^{-1},\\qquad(1)\n\\]\nwith\n\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big]=\\sum_{d=1}^{D}\\Big(\\underbrace{\\mathrm{E}\\{X_dX_d'\\hl{\\operatorname{Var}[\\epsilon_d|\\mathbf{X}]}\\}}_{A}+\\underbrace{\\sum_{d'\\in\\mathcal{S}(d)\\,\\mathrm{E}}\\{X_dX_{d'}'\\hl{\\operatorname{Cov}[\\epsilon_d,\\epsilon_{d'}|\\mathbf{X}]}\\}}_{B}\\Big),\\qquad(2)\n\\]\nwhere $\\mathcal{S}(d)=\\{d'\\ne d\\}\\cap\\{d':s(d)\\cap s(d')\\ne\\emptyset\\}$, the set of dyads other than $d$ that share a member from $d$.''\n\np.~573, equation (6), first two lines:\n\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big]=\\sum_{d=1}^{D}\\sum_{d'=1}^{D}\\operatorname{Cov}[X_d\\epsilon_d,X_{d'}\\epsilon_{d'}]=\\sum_{d=1}^{D}\\sum_{d'=1}^{D}\\mathrm{E}\\{X_dX_{d'}'\\hl{\\operatorname{Cov}[\\epsilon_d,\\epsilon_{d'}|\\mathbf{X}]}\\}\n\\]", "corrected": "``Lemma 1. Suppose data and $\\hat\\beta$ as defined above: the units $i=1,\\dots,N$ are a random (i.i.d.) sample; the data are the $D=N(N-1)/2$ dyads $d(i,j)$; $Y_d$ and the $k$-vector $X_d$, which includes the constant, are determined by the characteristics of the two units of $d$ and have support bounded in a finite interval; the usual rank conditions on $\\mathbf{X}$ hold; $\\beta$ is the coefficient of the population least-squares fit, with residual $\\epsilon_d=Y_d-X_d'\\beta$, so that $\\mathrm{E}[X_d\\epsilon_d]=0$ while $\\mathrm{E}[\\epsilon_d\\mid\\mathbf{X}]$ need not be zero; and $\\hat\\beta=(\\mathbf{X}'\\mathbf{X})^{-1}\\mathbf{X}'Y$. As $N\\to\\infty$, the asymptotic distribution of $\\sqrt{N}(\\hat\\beta-\\beta)$ has mean zero and variance\n\\[\nV=\\frac{N}{D^2}\\,\\mathrm{E}[X_dX_d']^{-1}\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big]\\mathrm{E}[X_dX_d']^{-1},\\qquad(1)\n\\]\nwith\n\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big]=\\sum_{d=1}^{D}\\Big(\\underbrace{\\mathrm{E}\\{X_dX_d'\\hl{\\epsilon_d^2}\\}}_{A}+\\underbrace{\\sum_{d'\\in\\mathcal{S}(d)}\\mathrm{E}\\{X_dX_{d'}'\\hl{\\epsilon_d\\epsilon_{d'}}\\}}_{B}\\Big),\\qquad(2)\n\\]\nwhere $\\mathcal{S}(d)=\\{d'\\ne d\\}\\cap\\{d':s(d)\\cap s(d')\\ne\\emptyset\\}$, the set of dyads other than $d$ that share a member from $d$.''\n\nIn equation (6), the second line becomes $\\sum_{d}\\sum_{d'}\\mathrm{E}\\{X_dX_{d'}'\\hl{\\epsilon_d\\epsilon_{d'}}\\}$ and the third line becomes the right side of (2) as corrected. Equivalently, the second line of (6) may be kept as printed if $\\operatorname{Var}(\\mathrm{E}[\\sum_dX_d\\epsilon_d\\mid\\mathbf{X}])$ is added to it. In $\\Sigma$ of Appendix B, in $V_r$ and in $V_w$, the conditional variances and covariances are replaced in the same way by the second moments of the scores, which are $X_d'\\epsilon_d$ for repeated observations (with $X_d$ a $T(d)\\times k$ regressor matrix and $\\epsilon_d$ a $T(d)$-vector of residuals) and $w_dX_d\\epsilon_d$ for weighted observations.", "evidence": "The evidence is a derivation and an exact counterexample. I write $h_d=X_d\\epsilon_d$ for the score of dyad $d$ and $m_d=\\mathrm{E}[\\epsilon_d\\mid\\mathbf{X}]$. By the definition of $\\beta$, I have $\\mathrm{E}[h_d]=0$. Then\n\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}h_d\\Big]=\\sum_{d=1}^{D}\\sum_{d'=1}^{D}\\mathrm{E}[h_dh_{d'}'].\n\\]\nIf $d$ and $d'$ share no unit, then $h_d$ and $h_{d'}$ are functions of disjoint sets of sampled units. They are independent, and $\\mathrm{E}[h_dh_{d'}']=\\mathrm{E}[h_d]\\mathrm{E}[h_{d'}]'=0$. The remaining pairs are $d'=d$ and $d'\\in\\mathcal{S}(d)$, and I obtain equation (2) as corrected below. To compare with the printed form, note that $X_d$ is a function of $\\mathbf{X}$, so that $\\mathrm{E}[h_d\\mid\\mathbf{X}]=X_dm_d$. By the law of total covariance, I have\n\\[\n\\operatorname{Cov}[h_d,h_{d'}]=\\mathrm{E}\\{X_dX_{d'}'\\operatorname{Cov}[\\epsilon_d,\\epsilon_{d'}\\mid\\mathbf{X}]\\}+\\mathrm{E}\\{X_dX_{d'}'m_dm_{d'}\\}.\n\\]\nSumming over all pairs $(d,d')$, I obtain\n\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}h_d\\Big]=\\sum_{d=1}^{D}\\sum_{d'=1}^{D}\\mathrm{E}\\{X_dX_{d'}'\\operatorname{Cov}[\\epsilon_d,\\epsilon_{d'}\\mid\\mathbf{X}]\\}+M_N,\\qquad M_N=\\operatorname{Var}\\Big(\\sum_{d=1}^{D}X_dm_d\\Big).\n\\]\nThe first term on the right is the second line of (6). The matrix $M_N$ is positive semidefinite, and it vanishes when $m_d=0$ for every $d$.\n\nFor the counterexample, fix $N\\ge3$, let $Z_1,\\dots,Z_N$ be independent and uniform on $[-1,1]$, and for $i<j$ set $X_{ij}=(1,\\,2+Z_i+Z_j)'$ and $Y_{ij}=(Z_i+Z_j)^2$. The data are bounded, the design matrix has rank two with probability one, and\n\\[\nQ=\\mathrm{E}[X_{ij}X_{ij}']=\\begin{pmatrix}1&2\\\\2&14/3\\end{pmatrix},\\qquad\\beta=\\begin{pmatrix}2/3\\\\0\\end{pmatrix},\\qquad\\epsilon_{ij}=(Z_i+Z_j)^2-\\frac{2}{3}.\n\\]\nSince $\\epsilon_{ij}$ is a function of $X_{ij}$, every conditional variance and covariance in (2) is zero, and the right side of (2) is the zero matrix for every $N$. Exact integration gives\n\\[\nH=\\mathrm{E}[h_{12}h_{12}']=\\begin{pmatrix}28/45&56/45\\\\56/45&3448/945\\end{pmatrix},\\qquad\\Gamma=\\mathrm{E}[h_{12}h_{13}']=\\begin{pmatrix}4/45&8/45\\\\8/45&632/945\\end{pmatrix}.\n\\]\nThere are $D$ pairs with $d'=d$ and $N(N-1)(N-2)$ ordered pairs of distinct dyads that share a unit, and I have\n\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}h_d\\Big]=DH+N(N-1)(N-2)\\Gamma.\n\\]\nThe table lists this matrix for $N=3,4,5$, where it was also computed by integrating the square of the score sum directly. As $N\\to\\infty$, the matrix $(N/D^2)\\operatorname{Var}[\\sum_dh_d]$ converges to $4\\Gamma$. Then equation (1) implies that\n\\[\nV\\to4Q^{-1}\\Gamma Q^{-1}=\\begin{pmatrix}3664/315&-592/105\\\\-592/105&296/105\\end{pmatrix}.\n\\]\nThe limit has determinant $4736/4725$ and is positive definite, while the right side of (2) as printed is the zero matrix. A derivation shows that, in this example, $N\\hat V$ converges in probability to the same limit, where $\\hat V$ is the estimator (3). A second example was checked separately, with $Z_i$ uniform on $[0,1]$, $X_{ij}=(1,\\,Z_i+Z_j)'$ and the same $Y_{ij}$. There $\\beta=(-5/6,2)'$, the right side of (2) is again zero, and the limit of $V$ is the positive definite matrix with rows $(229/315,\\,-74/105)$ and $(-74/105,\\,74/105)$.\n\\begin{audittable}{rrrrr}{The variance of the score sum $\\sum_dX_d\\epsilon_d$ in the first example: exact entries, and the value that equation (2) gives as printed.}\n$N$ & entry $(1,1)$ & entry $(1,2)$ & entry $(2,2)$ & every entry by (2) as printed \\\\\n3 & $12/5$ & $24/5$ & $4712/315$ & 0 \\\\\n4 & $88/15$ & $176/15$ & $1328/35$ & 0 \\\\\n5 & $104/9$ & $208/9$ & $14480/189$ & 0 \\\\\n\\end{audittable}\nThe examples show that the omitted term can be of the order of the variance itself. They do not show that it is nonzero for every nonlinear conditional mean.", "problem": "The left side of (2) is the variance of the score sum $\\sum_dX_d\\epsilon_d$ over the random sample of units. The right side is built from variances and covariances of the residuals that are conditional on the whole design matrix $\\mathbf{X}$. The two sides agree when $\\mathrm{E}[\\epsilon_d\\mid\\mathbf{X}]=0$, which the paper does not assume. It defines $\\beta$ as the coefficient of the population least-squares fit, ``allowing for possible non-linearity in the true relationship between $\\mathrm{E}[Y_{d(i,j)}|X_{d(i,j)}]$ and $X_{d(i,j)}$'' (p.~566), and footnote 2 (p.~565) says that ``the derivation is robust to violations of the assumption that the regression specification is correct for the conditional expectation''. This definition implies that $\\mathrm{E}[X_d\\epsilon_d]=0$, but does not imply that the conditional mean of $\\epsilon_d$ is zero. In that setting the step from the first to the second line of (6) drops the matrix $M_N=\\operatorname{Var}(\\mathrm{E}[\\sum_dX_d\\epsilon_d\\mid\\mathbf{X}])$, the variance of the conditional mean of the score sum. This matrix is positive semidefinite and need not vanish. In the example below it is of order $N^3$, the order of the variance itself. There the right side of (2) is zero, and the limiting variance is positive definite. The third line of (6) and equation (2) also discard the conditional covariances of dyads that share no unit. The independence of such dyads does not by itself justify this step, which plays no role in the example.\n\nThe matrix $\\Sigma$ in Appendix B (p.~573) and the limits $V_r$ of Corollary 1 (p.~568) and $V_w$ of Corollary 2 (p.~576) are defined in the same conditional form, and they are wrong for the same reason. The estimator (3) is built from products of fitted residuals and does not use (2). In (2) the letter E of term $B$ is printed in the summation subscript; this misprint is the subject of \\fid{P018-N001}.", "what_survives": "Equation (1), the definition of $\\beta$ and the OLS estimator are unaffected, and so is Proposition 2, which is an algebraic identity. The estimator (3) and its repeated and weighted versions are products of fitted residuals and do not use (2). No reported number from the simulations or the applications is shown to be wrong. The correction adds no assumption and changes neither the estimand nor the estimator. What changes is the formula (2), the second and third lines of (6), the definition of $\\Sigma$ in Appendix B and the limits $V_r$ and $V_w$. The proof in Appendix B is the subject of \\fid{P018-E006}, and the displays of $V_r$ and $V_w$ have further misprints, treated in \\fid{P018-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E001", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 1: the estimator of equation (3), the sentence 'Under the conditions of Lemma 1 and if X_d and e_d have bounded support with nonzero second moments and cross-moments, then as N goes to infinity', the display 'N V-hat minus V converges in probability to 0', and five lines of the next paragraph. Red boxes mark 'with nonzero second moments and cross-moments' and 'Standard error estimates for beta-hat are obtained from the square roots of the diagonals of V-hat.'", "caption": "Proposition 1 and the paragraph after it, p. 567 (PDF p. 4). The marks are on the condition “with nonzero second moments and cross-moments” and on the sentence that prescribes the standard errors.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E002.png"}, {"alt": "Crop of six lines from the introduction on page 565. A red box marks 'valid confidence intervals, while our procedure does.' in the sentence 'Second, Erikson, Pinto, and Rader's (2014) solution of non-parametric randomization inference does not provide a procedure for obtaining valid confidence intervals, while our procedure does.'", "caption": "Section 1, p. 565 (PDF p. 2). The mark is on the claim of valid confidence intervals.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T7", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T7"}, {"id": "P018-T8", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T8"}], "headline": "The stated conditions allow a coefficient whose limiting variance is zero, for which the variance estimate can be negative; the claim of valid confidence intervals and tests needs a positive limiting variance for each tested contrast and a central limit theorem.", "id": "P018-E002", "location": "Proposition 1 and the two sentences after it, p. 567 (PDF p. 4); the conditions before Lemma 1, p. 566 (PDF p. 3); Section 1, p. 565 (PDF p. 2); Section 5, p. 571 (PDF p. 8)", "pdf_page": 4, "printed_page": "567", "reaches": [{"effect": "result_needs_same_repair", "effect_text": "a result or code path there needs the same repair", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T7", "P018-T8"], "text": {"as_printed": "p.~567: ``Under the conditions of Lemma 1 and if $X_{\\mathrm{d}}$ and $e_d$ have bounded support \\hl{with nonzero second moments and cross-moments}, then as $N\\to\\infty$,\n\\[\nN\\hat V-V\\overset{p}{\\to}0.\n\\]\nThe proposition indicates that $\\hat V$ provides a consistent estimator to characterize the true asymptotic sampling distribution of the regression coefficients, $\\hat\\beta$. \\hl{Standard error estimates for $\\hat\\beta$ are obtained from the square roots of the diagonals of $\\hat V$.}''\n\np.~565: ``Second, Erikson, Pinto, and Rader's (2014) solution of non-parametric randomization inference does not provide a procedure for obtaining \\hl{valid confidence intervals, while our procedure does}.''", "corrected": "Proposition 1 stands. Add after it: ``Let $Q=\\mathrm{E}[X_dX_d']$, let $\\Omega=\\mathrm{E}[X_{d(i,j)}X_{d(i,k)}'\\epsilon_{d(i,j)}\\epsilon_{d(i,k)}]$ for distinct units $i$, $j$, $k$, and let $V_\\infty=4Q^{-1}\\Omega Q^{-1}$. Under the conditions of Lemma 1 (a random (i.i.d.) sample of units, $X_d$ and $Y_d$ with support bounded in a finite interval, and the usual rank conditions on $\\mathbf{X}$), $\\sqrt{N}(\\hat\\beta-\\beta)$ converges in distribution to a normal vector with mean zero and covariance matrix $V_\\infty$, which may be singular. \\hl{Let $c$ be a fixed vector such that $c'V_\\infty c>0$.} Then, under the conditions of Proposition 1,\n\\[\n\\frac{c'(\\hat\\beta-\\beta)}{\\sqrt{c'\\hat Vc}}\\overset{d}{\\to}N(0,1)\\qquad\\text{as }N\\to\\infty.\n\\]\nIn particular, the standard error of $\\hat\\beta_k$, the square root of the $k$th diagonal entry of $\\hat V$, supports a normal confidence interval or test \\hl{when the $k$th diagonal entry of $V_\\infty$ is positive}.''\n\np.~565: ``\\ldots does not provide a procedure for obtaining valid confidence intervals, while our procedure does \\hl{for coefficients and contrasts whose limiting variance is positive}.''", "evidence": "The evidence is an exact counterexample and a derivation. For each unit $i$ let $Z_i$ and $W_i$ be independent signs, each equal to $\\pm1$ with probability $1/2$, and for $i<j$ set $X_{ij}=(1,\\,1+Z_iZ_j)'$ and $Y_{ij}=W_i+W_j$. The data and the fitted residuals are bounded, and, for $N\\ge3$, the design matrix has full rank whenever both signs of $Z$ occur. The population coefficient is $\\beta=(0,0)'$, and $\\epsilon_{ij}=Y_{ij}$. I have $\\mathrm{E}[X_{ij}X_{ij}'\\epsilon_{ij}^2]=2Q$, where\n\\[\nQ=\\mathrm{E}[X_{ij}X_{ij}']=\\begin{pmatrix}1&1\\\\1&2\\end{pmatrix},\\qquad\\Omega=\\mathrm{E}[X_{ij}X_{ik}'\\epsilon_{ij}\\epsilon_{ik}]=\\begin{pmatrix}1&1\\\\1&1\\end{pmatrix},\n\\]\nand every entry of these matrices is nonzero. By equation (1), I have\n\\[\nV=\\frac{N}{D^2}\\,Q^{-1}\\{2DQ+N(N-1)(N-2)\\Omega\\}Q^{-1}\\to4Q^{-1}\\Omega Q^{-1}=\\begin{pmatrix}4&0\\\\0&0\\end{pmatrix},\n\\]\nand the slope entry of $V$ is $4/(N-1)$. I take eight units with $(Z_i,W_i)$ equal to $(1,1)$, $(1,-1)$, $(-1,1)$, $(-1,-1)$, each twice. Then $\\hat\\beta=0$, and the estimator (3) is\n\\[\n\\hat V=\\begin{pmatrix}3/8&-1/48\\\\-1/48&-13/288\\end{pmatrix}.\n\\]\nThe estimated variance of the slope is $-13/288$, and its standard error is not defined. This is not an accident of one sample. Let $\\hat\\beta_1$ denote the slope estimate and $\\hat V_{22}$ its estimated variance. As $N\\to\\infty$, the pair $(N\\hat\\beta_1,N^2\\hat V_{22})$ converges in distribution to $(2AB,\\,4(A^2+B^2-1))$, where $A$ and $B$ are independent standard normal variables. The probability that $\\hat V_{22}$ is negative therefore converges to $\\Pr(A^2+B^2<1)=1-e^{-1/2}$, about $0.393$. In the same example $N\\hat V$ converges in probability to $\\operatorname{diag}(4,0)$, which agrees with Proposition 1.\n\nFor the repair, let $U_i$ denote the characteristics of unit $i$, write $s(U_i,U_j)=X_{d(i,j)}\\epsilon_{d(i,j)}$, and set $\\eta(u)=\\mathrm{E}[s(u,U_j)]$ and $r(u,v)=s(u,v)-\\eta(u)-\\eta(v)$. Then $\\mathrm{E}[\\eta(U_i)]=0$ and $\\mathrm{E}[r(U_i,U_j)\\mid U_i]=0$, and\n\\[\n\\frac{1}{D}\\sum_{i<j}s(U_i,U_j)=\\frac{2}{N}\\sum_{i=1}^{N}\\eta(U_i)+\\frac{1}{D}\\sum_{i<j}r(U_i,U_j).\n\\]\nDistinct terms of the last sum are uncorrelated, and its covariance matrix is $\\mathrm{E}[rr']/D$, of order $N^{-2}$. The vectors $\\eta(U_i)$ are i.i.d., bounded and of mean zero, with covariance matrix $\\Omega=\\mathrm{E}[X_{d(i,j)}X_{d(i,k)}'\\epsilon_{d(i,j)}\\epsilon_{d(i,k)}]$, and $D^{-1}\\sum_dX_dX_d'$ converges in probability to $Q=\\mathrm{E}[X_dX_d']$. By the central limit theorem for i.i.d.\\ vectors and Slutsky's theorem, I obtain that $\\sqrt{N}(\\hat\\beta-\\beta)$ converges in distribution to a normal vector with mean zero and covariance matrix $V_\\infty=4Q^{-1}\\Omega Q^{-1}$, which is also the limit of $V$. If $c'V_\\infty c>0$, then Proposition 1 implies that $Nc'\\hat Vc$ converges in probability to $c'V_\\infty c$, and I conclude by Slutsky's theorem that the ratio has a standard normal limit.", "problem": "Lemma 1 states the mean and the variance of the limit of $\\sqrt{N}(\\hat\\beta-\\beta)$, and by Proposition 1 the difference $N\\hat V-V$ converges to zero in probability. Neither statement excludes a coefficient, or a linear combination $c'\\beta$ of coefficients, whose limiting variance is zero. For such a contrast the proposition says only that $Nc'\\hat Vc$ converges to zero. It does not say that $c'\\hat Vc$ is positive, or that the ratio of $c'(\\hat\\beta-\\beta)$ to $\\sqrt{c'\\hat Vc}$ is approximately standard normal. The rank conditions on $\\mathbf{X}$ and the condition of ``nonzero second moments and cross-moments'' do not exclude this case: in the example below every entry of the matrices of second moments and cross-moments is nonzero, and the limiting variance of the slope is zero. The paper also states no central limit theorem for the score sum, and a confidence interval based on the normal distribution needs one. The conditions as stated are not sufficient for the claim of valid confidence intervals on p.~565, for the standard errors prescribed after Proposition 1, or for the test in Section 5.", "what_survives": "Lemma 1 as a statement of mean and variance and the conclusion $N\\hat V-V\\overset{p}{\\to}0$ of Proposition 1 are not contradicted; both hold in the example. Proposition 2, the computation of the estimator and every reported coefficient and standard error are unaffected, and the limiting variance in the simulation design of Section 4 is positive definite. What changes is the scope of the claims about confidence intervals and tests, which hold for contrasts whose limiting variance is positive. I have not checked this condition for the two applications. The corresponding condition is needed for repeated observations and for generalized linear models."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E002", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": {"class": "error", "severity": "Minor"}, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the heading Appendix D: Weighted Observations, the paragraph that follows, the display of the weighted least squares estimator, and the first two lines of Corollary 2. Red boxes mark the words with weights finite and fixed; and the words assume the same conditions as in Proposition 1.", "caption": "Appendix D, p. 575 (PDF p. 12). The marked phrases are the only conditions on the weights: they are finite and fixed, and the conditions of Proposition 1 are assumed.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E003.png"}, {"alt": "Crop of four lines from the paragraph before Corollary 1. A red box marks the words fixed and finite in the clause where the T of d values are fixed and finite.", "caption": "P. 567 (PDF p. 4), setting for repeated observations. The marked words bound each $T(d)$ separately and state no bound that is uniform over dyads.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T9", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T9"}, {"id": "P018-T10", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T10"}, {"id": "P018-T28", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T28"}], "headline": "Corollary 2 needs the weights to be bounded after a common rescaling and the rescaled weighted design to have a positive definite limit; weights that are only finite and fixed do not suffice. Corollary 1 also needs uniformly bounded $T(d)$, a positive definite limiting design matrix and a condition on products of dyad-score means when record kinds follow different laws.", "id": "P018-E003", "location": "Appendix D and Corollary 2, p. 575 (PDF p. 12); also the repeated-observations setting before Corollary 1, p. 567 (PDF p. 4)", "pdf_page": 12, "printed_page": "575", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [{"classification_after": {"class": "error", "severity": "Minor"}, "classification_before": {"class": "error", "severity": "Minor"}, "note": "The repeated-data correction was expanded while the classification was still Minor. When record kinds follow different laws, the stacked normal equations center the total score but need not center each dyad’s score. The correction therefore adds a condition requiring the normalized sum of products of score means over overlapping dyads to tend to zero. The estimator and the score-covariance target are unchanged. P. M. Aronow delegated this decision to the model session that runs the audit, which adopted the amendment recommended by the two checks; this note was written by that session."}, {"classification_after": {"class": "error", "severity": "Moderate"}, "classification_before": {"class": "error", "severity": "Minor"}, "note": "Later, the model session running the audit raised the classification from Minor to Moderate, after P. M. Aronow suggested Moderate and left the final decision to that session. The additional condition on score means is substantive when different record kinds follow different laws, a reading supported by p. 567; it holds automatically when every record follows one common law. The correction, estimator, cross-sectional result and weighted corollary are unchanged. This note was written by that session."}], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P018-T10", "P018-T28", "P018-T9"], "text": {"as_printed": "P.~575: Assume weighted directed dyad observations \\hl{with weights finite and fixed}; denote the weight for dyad $d$ as $w_d$. [\\ldots] Corollary 2. For the weighted dyads case, \\hl{assume the same conditions as in Proposition 1} and consider the following variance estimator:\n\\[\n\\hat V_w=(\\mathbf X'\\mathbf W\\mathbf X)^{-1}\\sum_{d=1}^{D}\\Big(X_dX_d'w_d^2e_d^2+\\sum_{d'\\in\\mathcal S(d)}X_dX_{d'}'w_dw_{d'}e_de_{d'}\\Big)(\\mathbf X'\\mathbf W\\mathbf X)^{-1}.\\qquad(7)\n\\]\nThen, as $N\\to\\infty$, $N\\operatorname{Var}[\\hat\\beta_w-\\beta_w]-V_w\\overset{p}{\\to}0$, and $N\\hat V_w-V_w\\overset{p}{\\to}0$,\n\nP.~567: Formally, suppose that for each dyad $d$ observations are indexed by $t=1,\\dots,T(d)$, where the $T(d)$ values are \\hl{fixed and finite}.", "corrected": "P.~575: Assume weighted directed dyad observations with weights fixed \\hl{and such that, for some constants $a_N>0$ and $C<\\infty$, $0<w_d/a_N\\le C$ for all dyads $d$ and all $N$}; denote the weight for dyad $d$ as $w_d$. [\\ldots] Corollary 2. For the weighted dyads case, assume the same conditions as in Proposition 1, \\hl{and assume that the matrices $D^{-1}\\sum_{d=1}^{D}\\mathrm E[(w_d/a_N)X_dX_d']$ converge to a positive definite limit}, and consider the variance estimator (7). Then, as $N\\to\\infty$, the conclusions are those printed, with the target $V_w$ as corrected in \\fid{P018-E004} and the first conclusion requiring uniform integrability of $N\\|\\hat\\beta_w-\\beta_w\\|^2$ (\\fid{P018-N004}).\n\nP.~567: Formally, suppose that for each dyad $d$ observations are indexed by $t=1,\\dots,T(d)$, where the $T(d)$ values are fixed and \\hl{bounded uniformly in $d$ and $N$}.\n\nFor the repeated-data consistency conclusion, \\hl{for each record kind $t$, one common measurable function of the two unit characteristics determines $(X_{dt},Y_{dt})$, independently of $N$ and $d$ wherever that record exists}. Write $L_N=\\sum_dT_N(d)$, $q_{d,N}=X_d'\\epsilon_d$, $m_{d,N}=\\mathrm E[q_{d,N}]$ and $H_{r,N}=L_N^{-1}\\sum_d\\mathrm E[X_d'X_d]$, where $\\epsilon_d$ is the vector of population residuals from the stacked projection. In addition to the uniform repetition bound, \\hl{assume that $H_{r,N}\\to H_r$ with $H_r$ positive definite and that}\n\\[\n\\hl{\\frac{N}{L_N^2}\\sum_{(d,f):\\,s(d)\\cap s(f)\\ne\\emptyset}m_{d,N}m_{f,N}'\\to0.}\n\\]\nThe overlap sum is ordered and includes $d=f$. Then $N\\hat V_r-V_r\\to0$ in probability for the corrected target\n\\[\nV_r=\\frac{N}{L_N^2}H_{r,N}^{-1}\\operatorname{Var}\\Bigl[\\sum_dq_{d,N}\\Bigr]H_{r,N}^{-1}.\n\\]\nThe stacked population normal equations give $\\sum_dm_{d,N}=0$ for all sufficiently large $N$; they do not imply this overlap mean-product condition.\n\nUnder the weight conditions above each rescaled score $(w_d/a_N)X_d\\epsilon_d$ is bounded, and the counts of dependent pairs in the corrected proof of Proposition 1 (\\fid{P018-E006}) give the same rates for the weighted sums. For the weights $w_{ij}=2^{-i-j}$ of the example, the first condition forces $a_N\\ge1/(8C)$. Then $D^{-1}\\sum_dw_d/a_N\\le8C\\,\\mathbf X'\\mathbf W\\mathbf X/D\\to0$, and the second condition fails.", "evidence": "The evidence is an exact counterexample to Corollary 2 as printed. Let $A_1,A_2,\\dots$ be independent with $P(A_i=1)=P(A_i=-1)=1/2$, and for $i<j\\le N$ let $X_{ij}=1$, $Y_{ij}=A_i+A_j$ and $w_{ij}=2^{-i-j}$. The weights are positive, fixed and at most $1/8$. All data are bounded, $\\mathbf X'\\mathbf X/D=1$ and $\\beta_w=0$, while $\\mathbf X'\\mathbf W\\mathbf X/D\\to0$. The estimator is $\\hat\\beta_w=\\sum_ic_{i,N}A_i$ with $c_{1,N}=3/\\{4(1-2^{-N})\\}\\ge3/4$. Since the signs are independent with variance one, I have\n\\[\n\\operatorname{Var}(\\hat\\beta_w)=\\sum_{i=1}^{N}c_{i,N}^2\\ge\\frac9{16}\\quad\\text{for all }N\\ge2,\\qquad\\operatorname{Var}(\\hat\\beta_w)\\to\\frac{36}{35}.\n\\]\nAt $N=8$ the variance equals $56098/53975$, which is about $1.039$. On the event $A_1=\\dots=A_8=1$, which has probability $1/256$, I have $|\\hat V_w|\\le1/16$ for every $N\\ge8$. Combining the two bounds, I obtain\n\\[\nP\\Big(\\big|N\\hat V_w-N\\operatorname{Var}(\\hat\\beta_w)\\big|\\ge\\frac N2\\Big)\\ge\\frac1{256}\\quad\\text{for every }N\\ge8.\n\\]\nThe two limits of Corollary 2 would imply that this probability tends to zero. With $T(i,j)=2^{i+j}$ identical observations per dyad and no weights, the estimator and the sandwich of Corollary 1 have the same joint distribution as in the weighted example, after the order of the units is reversed. A uniform bound on the $T(d)$ excludes this second example.", "problem": "The conditions of Proposition 1 concern the unweighted design: bounded support and the rank of $\\mathbf X$. They do not control the weighted design $\\mathbf X'\\mathbf W\\mathbf X$. Weights that are positive, fixed and uniformly bounded can place almost all of the weight on a few units. Then $\\hat\\beta_w$ does not converge, and the two limits stated in Corollary 2 cannot both hold for any target.\n\nFor repeated observations, ``fixed and finite'' read dyad by dyad allows the $T(d)$ to grow across dyads, with the same effect. The discussion on p.~568 indicates that a bound uniform over dyads is meant, and the correction states it. Even with a uniform repetition bound, different record kinds, such as years or directions, may follow different laws. The stacked population normal equations center the total score, but need not give each dyad a mean-zero block score. The estimator sums score products over overlapping dyads, so these products can include a nonvanishing contribution from the dyad-score means that is absent from the covariance target. The correction specifies one common data function per record kind, convergence of the normalized expected design matrix to a positive definite limit, and convergence to zero of the normalized sum of these mean products.", "what_survives": "The estimator (7), the estimand and the algebraic identity of Proposition 2 are unchanged, and the example does not concern the cross-sectional result without weights: the unweighted estimator in the example has variance $4/N$. The simulations use $T=2$ for every dyad and the application has at most 90 years per dyad, which is compatible with bounded $T(d)$. The added conditions restrict the weights that the corollary covers, and conditions of the same form are needed for the extension to generalized linear models. The conditions are sufficient ones; I do not claim that they are the weakest."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E003", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of p. 568, with the running head 568, Peter M. Aronow et al. The word where is followed by a display: V r equals N over the square of the sum over d from 1 to D of T of d, times the inverse of E of X d times X d transposed, times E of a large bracket, times the inverse of E of X d times X d transposed. The bracket holds E of X d transposed times Cov of epsilon d given X times X d, plus the sum over d-prime in S of d of E of X d transposed times Cov of epsilon d and epsilon d-prime given X times X d-prime. Three red outlines surround the product X d times X d transposed in the left factor, the letter E in front of the large bracket, and the product X d times X d transposed in the right factor. The first lines of the next paragraph follow.", "caption": "Top of p. 568 (PDF p. 5): the display that defines $V_r$. The outlines mark the product $X_d X_d'$ in both outer factors and the outer expectation E of the middle factor.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E004.png"}, {"alt": "Crop of one display introduced by the word where: V w equals N over D squared, times the inverse of E of w d times X d transposed times X d, times E of a large bracket, times the inverse of E of w d times X d times X d transposed. The bracket holds E of X d times X d transposed times w d squared times Var of epsilon d given X, plus the sum over d-prime in S of d of E of X d times X d-prime transposed times w d w d-prime times Cov of epsilon d and epsilon d-prime given X. Two red outlines surround the product X d transposed times X d in the left factor and the letter E in front of the large bracket.", "caption": "Top of p. 576 (PDF p. 13): the display that defines $V_w$. The outlines mark the scalar product $X_d' X_d$ in the left factor and the outer expectation E.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T10", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T10"}, {"id": "P018-T11", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T11"}, {"id": "P018-T12", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T12"}, {"id": "P018-T28", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T28"}], "headline": "The limits $V_r$ and $V_w$ need a sum over dyads in the middle factor, where an expectation is printed, and normalised $k \\times k$ Gram matrices in the outer factors.", "id": "P018-E004", "location": "Display that defines $V_r$ after Corollary 1, p. 568 (PDF p. 5, top); display that defines $V_w$ after Corollary 2, p. 576 (PDF p. 13, top)", "pdf_page": 5, "printed_page": "568", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T10", "P018-T11", "P018-T12", "P018-T28"], "text": {"as_printed": "\\[\n\\begin{aligned}\nV_r = {}& \\frac{N}{\\bigl[\\sum_{d=1}^{D} T(d)\\bigr]^{2}}\\, \\operatorname{E}[\\hl{X_d X_d'}]^{-1}\\, \\hl{\\operatorname{E}} \\Bigl( \\operatorname{E}[X_d' \\operatorname{Cov}[\\epsilon_d | \\mathbf{X}] X_d] \\\\\n& + \\sum_{d' \\in \\mathcal{S}(d)} \\operatorname{E}[X_d' \\operatorname{Cov}[\\epsilon_d, \\epsilon_{d'} | \\mathbf{X}] X_{d'}] \\Bigr) \\operatorname{E}[\\hl{X_d X_d'}]^{-1}.\n\\end{aligned}\n\\]\n\\[\n\\begin{aligned}\nV_w = {}& \\frac{N}{D^{2}}\\, \\operatorname{E}[w_d \\hl{X_d' X_d}]^{-1}\\, \\hl{\\operatorname{E}} \\Bigl( \\operatorname{E}\\bigl[X_d X_d' w_d^2 \\operatorname{Var}[\\epsilon_d | \\mathbf{X}]\\bigr] \\\\\n& + \\sum_{d' \\in \\mathcal{S}(d)} \\operatorname{E}\\bigl[X_d X_{d'}' w_d w_{d'} \\operatorname{Cov}[\\epsilon_d, \\epsilon_{d'} | \\mathbf{X}]\\bigr] \\Bigr) \\operatorname{E}[w_d X_d X_d']^{-1}.\n\\end{aligned}\n\\]", "corrected": "For the repeated dyads case, let $X_d$ be the $T(d) \\times k$ matrix with rows $X_{dt}'$, and let $\\epsilon_d$ be the vector of the $T(d)$ population residuals of dyad $d$. Write $L = \\sum_{d=1}^{D} T(d)$, and define\n\\[\nH_r = \\frac{1}{L} \\sum_{d=1}^{D} \\operatorname{E}[X_d' X_d], \\qquad \\Omega_r = \\operatorname{Var}\\Bigl[\\sum_{d=1}^{D} X_d' \\epsilon_d\\Bigr].\n\\]\nThen\n\\[\nV_r = \\frac{N}{L^{2}}\\, \\hl{H_r^{-1}\\, \\Omega_r\\, H_r^{-1}}.\n\\]\nFor the weighted dyads case, let $X_d$ be the $k$-vector of regressors of dyad $d$, and define\n\\[\nH_w = \\frac{1}{D} \\sum_{d=1}^{D} \\operatorname{E}[w_d X_d X_d'], \\qquad \\Omega_w = \\operatorname{Var}\\Bigl[\\sum_{d=1}^{D} w_d X_d \\epsilon_d\\Bigr].\n\\]\nThen\n\\[\nV_w = \\frac{N}{D^{2}}\\, \\hl{H_w^{-1}\\, \\Omega_w\\, H_w^{-1}}.\n\\]", "evidence": "The evidence is an exact counterexample with a derivation. Let $U_1, \\dots, U_N$ be independent signs, each equal to $\\pm 1$ with probability $1/2$, and for each dyad set $X_{ij} = 1$ and $Y_{ij} = U_i + U_j$, with $T(d) = 1$ and $w_d = 1$. The population coefficient is $\\beta = 0$, and $\\epsilon_{ij} = U_i + U_j$. Each unit belongs to $N - 1$ dyads, so $\\widehat\\beta = (2/N) \\sum_i U_i$, and\n\\[\nN \\operatorname{Var}[\\widehat\\beta - \\beta] = 4.\n\\]\nA dyad has residual variance $2$ and covariance $1$ with each of the $2(N-2)$ dyads that share a member with it. The printed bracket equals $2 + 2(N - 2) = 2N - 2$ for every $d$, and\n\\[\nV_r^{\\text{printed}} = V_w^{\\text{printed}} = \\frac{N(2N - 2)}{D^2} = \\frac{4}{D}.\n\\]\nWith the sum over dyads restored, the middle factor is $D(2N - 2) = N(N-1)^2$, and the limit is $(N/D^2)\\, N(N-1)^2 = 4$.\n\\begin{audittable}{rrrrr}{The example $X_{ij} = 1$, $Y_{ij} = U_i + U_j$: exact values for four sample sizes.}\n$N$ & $D$ & $N\\operatorname{Var}[\\widehat\\beta]$ & $V_r$ and $V_w$ as printed & $V_r$ and $V_w$ as corrected \\\\\n$3$ & $3$ & $4$ & $4/3$ & $4$ \\\\\n$4$ & $6$ & $4$ & $2/3$ & $4$ \\\\\n$5$ & $10$ & $4$ & $2/5$ & $4$ \\\\\n$10$ & $45$ & $4$ & $4/45$ & $4$ \\\\\n\\end{audittable}\nNote that the estimators carry the sum over dyads. In this example $N\\widehat V_r = N\\widehat V_w$ converges to $4$ in probability. For the outer factors, suppose that each observation of the example is repeated twice, so that $T(d) = 2$ and $X_d = (1, 1)'$. Then\n\\[\n\\operatorname{E}[X_d X_d'] = \\begin{pmatrix} 1 & 1 \\\\ 1 & 1 \\end{pmatrix}\n\\]\nis singular, while $\\operatorname{E}[X_d' X_d] = 2$. Restoring the sum and writing $X_d' X_d$, with no other change, I find the value $1$. The normalised matrix $H_r$ below gives $4$.\n\nThe form of the corrected displays comes from the identity\n\\[\n\\sqrt{N}(\\widehat\\beta_r - \\beta_r) = \\Bigl(\\frac{\\mathbf{X}'\\mathbf{X}}{L}\\Bigr)^{-1} \\frac{\\sqrt{N}}{L} \\sum_{d=1}^{D} X_d' \\epsilon_d,\n\\]\nwhere $L = \\sum_d T(d)$. Replacing $\\mathbf{X}'\\mathbf{X}/L$ by its population version $H_r$, I obtain a vector with covariance matrix $(N/L^2) H_r^{-1} \\Omega_r H_r^{-1}$. The weighted case is the same, with $\\mathbf{X}'\\mathbf{W}\\mathbf{X}/D$, $D$ and $\\sum_d w_d X_d \\epsilon_d$.", "problem": "Both displays print an outer expectation of the bracket for a single dyad $d$, where equations~(1) and~(2) have a sum over $d$. In the example below, both printed limits equal $4/D$ and tend to zero, while $N\\operatorname{Var}[\\widehat\\beta]$ equals $4$. Both conclusions of each corollary then fail with the printed definitions. The outer factors do not conform either. In the repeated dyads case, $X_d$ is the $T(d) \\times k$ matrix with rows $X_{dt}'$, and $X_d X_d'$ is a $T(d) \\times T(d)$ matrix, whose size varies with $d$. In the weighted case, the left factor $\\operatorname{E}[w_d X_d' X_d]^{-1}$ is the inverse of a scalar, and the right factor is the inverse of a $k \\times k$ matrix. For $V_r$ it is not enough to restore the sum and to write $X_d' X_d$. The prefactor counts observations, through $\\sum_d T(d)$, and the Gram matrix must be normalised by the same count.", "what_survives": "The estimators $\\widehat V_r$ on p.~567 and $\\widehat V_w$ in equation~(7), equation~(1), Proposition~2 and Appendix~C do not use these displays. The replication scripts compute the sandwich through the decomposition of Proposition~2, in the repeated-observations simulation and in both applications, and no reported number changes. The corrected displays define the population covariance targets at sample size $N$; their limits, when they exist, are the asymptotic covariance matrices. They are not a proof of the two corollaries. The middle factor is written below as the variance of the total score, as in equation~(1). The conditional form of the printed middle factor is the subject of \\fid{P018-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E004", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the logistic estimating equation, the definition of p d, the display of the estimator V hat l, and the definitions of M and r d. A red box marks the second regressor in the cross-dyad term, X prime with subscript d, where the subscript d prime belongs.", "caption": "Appendix E, p. 576 (PDF p. 13), display of $\\hat V_l$. The marked second regressor of the cross term has the subscript $d$; the residual beside it has the subscript $d'$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T13", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T13"}, {"id": "P018-T14", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T14"}], "headline": "In the plug-in variance estimator for logistic regression of Appendix E, the cross-dyad term is $X_dX_{d'}'r_dr_{d'}$, which pairs the regressors of both dyads, and not $X_dX_d'r_dr_{d'}$.", "id": "P018-E005", "location": "Appendix E, p. 576 (PDF p. 13), display of the plug-in variance estimator $\\hat V_l$, cross-dyad term", "pdf_page": 13, "printed_page": "576", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T13", "P018-T14"], "text": {"as_printed": "The plug-in variance estimator for logistic regression coefficients is given by\n\\[\n\\hat V_l=(\\mathbf X'\\mathbf M\\mathbf X)^{-1}\\sum_{d=1}^{D}\\Big(X_dX_d'r_d^2+\\sum_{d'\\in\\mathcal S(d)}X_dX_{\\hl{d}}'r_dr_{d'}\\Big)(\\mathbf X'\\mathbf M\\mathbf X)^{-1},\n\\]\nwhere $\\mathbf M=\\operatorname{diag}(p_d(1-p_d))_{D\\times D}$ and $r_d=Y_d-p_d$.", "corrected": "The plug-in variance estimator for logistic regression coefficients is given by\n\\[\n\\hat V_l=(\\mathbf X'\\mathbf M\\mathbf X)^{-1}\\sum_{d=1}^{D}\\Big(X_dX_d'r_d^2+\\sum_{d'\\in\\mathcal S(d)}X_dX_{\\hl{d'}}'r_dr_{d'}\\Big)(\\mathbf X'\\mathbf M\\mathbf X)^{-1},\n\\]\nwhere $\\mathbf M=\\operatorname{diag}(p_d(1-p_d))_{D\\times D}$, $r_d=Y_d-p_d$, $p_d=\\operatorname{expit}(X_d'\\hat\\beta)$, and $\\mathcal S(d)$ is the set of dyads other than $d$ that share a member with $d$.", "evidence": "The evidence is an exact example. Let the six dyads of four units be ordered as $\\{1,2\\}$, $\\{1,3\\}$, $\\{1,4\\}$, $\\{2,3\\}$, $\\{2,4\\}$, $\\{3,4\\}$, and let $X_d=(1,x_d)'$ with $x=(0,2,5,5,1,1)$ and $Y=(1,1,1,0,0,0)$. The score vanishes at $\\hat\\beta=(0,0)'$. Since the design has rank two, the log-likelihood is strictly concave, and this point is the maximum likelihood estimate. At this estimate, I have $p_d=1/2$, $r_d=\\pm1/2$ and\n\\[\n\\mathbf X'\\mathbf M\\mathbf X=\\begin{pmatrix}3/2&7/2\\\\7/2&14\\end{pmatrix}.\n\\]\nWith the cross term $X_dX_{d'}'r_dr_{d'}$ and with the printed cross term, I obtain respectively\n\\[\n\\begin{pmatrix}38/25&-64/175\\\\-64/175&192/1225\\end{pmatrix}\\qquad\\text{and}\\qquad\\begin{pmatrix}8/5&-2/5\\\\-2/5&6/35\\end{pmatrix}.\n\\]\nThe first matrix agrees with the computation from sums of scores by unit, less the dyad terms, which is the form that the authors' code uses. A recalculation with that code from the replication data reproduces the nine coefficients of the pooled logistic regression in Table 1 and their dyadic standard errors to three decimals.", "problem": "The estimating function printed just above the display is the score $X_d(Y_d-p_d)=X_dr_d$. The sandwich sums the products of the scores of dyads that share a unit, and the product of the scores of $d$ and $d'$ is $X_dX_{d'}'r_dr_{d'}$, as in equations (3) and (7). The printed cross term repeats $X_d$ where $X_{d'}$ belongs. A computation that follows the display literally gives a different matrix.", "what_survives": "The authors' code forms products of scores and does not contain the misprint. Table 1 and the conclusion about the coefficient for international organizations are unchanged. The formulas for linear regression do not use this display. The other notation of Appendix E is corrected in \\fid{P018-N010}."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E005", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the heading Appendix B: Proof of Proposition 1 and the first three lines of the proof. A red box marks condition (i), which reads E of V hat minus V tends to zero.", "caption": "Appendix B, p. 573 (PDF p. 10), first lines of the proof. The marked condition (i) has no factor $N$, while the target and condition (ii) concern $N\\hat V$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E006.png"}, {"alt": "Crop of two displays: the definition of Sigma as a sum over dyads of expected conditional variances and covariances of the population residuals, and the expectation of Sigma hat written with the fitted residuals e. A red box marks the arrow and the symbol Sigma at the end of the second display.", "caption": "Appendix B, p. 573 (PDF p. 10), last display of the page. The marked limit compares the two sums without normalization.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E006-b.png"}, {"alt": "Crop of the first four lines of page 574. One red box marks the clause boundedness of X d implies uniform boundedness of the inverse of E of X d X d prime. A second red box marks E of V hat minus V tends to zero.", "caption": "Appendix B, p. 574 (PDF p. 11), first lines. The marked clause infers a bound on the inverse from bounded regressors, and the marked limit again has no factor $N$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E006-c.png"}, {"alt": "Crop of part (ii) of the proof: three lines of text, the two-line display defining Omega as N squared D to the minus four times the variance of a double sum, and a paragraph on quadruples of dyads. Red boxes mark the words so we can treat this as O of one, the product e d e d prime inside the variance, the words positive values, and the words six distinct ways.", "caption": "Appendix B, p. 574 (PDF p. 11), part (ii) of the proof. Marked: the inverse Gram matrix treated as $O(1)$ inside a variance, the fitted residuals $e_de_{d'}$ in $\\Omega$, the words “positive values”, and “six distinct ways”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E006-d.png"}, {"alt": "Line chart with the number of units N from 4 to 1024 on a logarithmic horizontal axis and the exact mean of the difference from V equals 4 on the vertical axis, from minus 4 to 0. A dashed orange curve, labelled as printed, E of V hat minus V, starts at minus 3.83 and settles on the reference line at minus 4. A solid blue curve, labelled with the factor N, E of N V hat minus V, rises from minus 3.33 at N equals 4 to about minus 0.02 at N equals 1024, approaching the reference line at 0. Squares and circles mark the enumerated values at N equals 4, 6 and 8.", "caption": "The two mean conditions in the example with $X_{ij}=1$ and $Y_{ij}=U_i+U_j$, for $N=4$ to $1024$. The mean printed in condition (i) of p. 573 tends to $-4$; with the factor $N$ it tends to $0$. The points at $N=4,6,8$ are exact enumerations.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E006-mean-condition.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T3", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T3"}, {"id": "P018-T4", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T4"}, {"id": "P018-T27", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T27"}], "headline": "Three steps of the proof of Proposition 1 in Appendix B fail, and the conclusion $N\\hat V-V\\overset{p}{\\to}0$ follows instead from an argument in probability that uses the population scores, a bound for the substitution of fitted residuals, and convergence of the inverse Gram matrix.", "id": "P018-E006", "location": "Appendix B, proof of Proposition 1, pp. 573–574 (PDF pp. 10–11)", "pdf_page": 10, "printed_page": "573–574", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P018-T27", "P018-T3", "P018-T4"], "text": {"as_printed": "P.~573: \\emph{Proof.} By Chebyshev's inequality, two conditions are sufficient for $N\\hat V-V\\overset{p}{\\to}0$ as $N\\to\\infty$: (i) $\\hl{\\mathrm E[\\hat V-V]\\to0}$ and (ii) $\\operatorname{Var}[N\\hat V-V]=\\operatorname{Var}[N\\hat V]\\to0$ (Lehmann 1999, Theorem 2.1.1). [\\ldots] We have\n\\[\n\\mathrm E[\\hat\\Sigma]=\\sum_{d=1}^{D}\\Big(\\mathrm E\\{X_dX_d'\\mathrm E[e_d^2\\mid\\mathbf X]\\}+\\sum_{d'\\in\\mathcal S(d)}\\mathrm E\\{X_dX_{d'}'\\mathrm E[e_de_{d'}\\mid\\mathbf X]\\}\\Big)\\hl{\\to\\Sigma}\n\\]\nP.~574: by consistency of $\\hat\\beta$, an implication of Lemma 1. Lemma 1 also established that $(\\mathbf X'\\mathbf X/D)^{-1}\\overset{p}{\\to}\\mathrm E[X_dX_d']^{-1}$, while \\hl{boundedness of $X_d$ implies uniform boundedness of $\\mathrm E[X_dX_d']^{-1}$}. Taking these elements together, uniform continuity implies $\\hl{\\mathrm E[\\hat V-V]\\to0}$ (cf. White 1980a, Theorem 1). [\\ldots] We have established that $(\\mathbf X'\\mathbf X/D)^{-1}$ converges in probability to a finite limit, $E[X_dX_d']^{-1}$, \\hl{so we can treat this as $O(1)$}. Given bounded support, $\\operatorname{Var}[N\\hat V]$ is of the same order as\n\\[\n\\Omega=N^2D^{-4}\\operatorname{Var}\\Big[\\sum_d\\sum_{d'}I_{d,d'}X_dX_{d'}'\\hl{e_de_{d'}}\\Big]\n\\]\n[\\ldots] The covariance terms take \\hl{positive values} when $I_{d,d'}=I_{d'',d'''}=1$, which occurs at rate $O(N^{-2})$ (i.e., when we have pairs of terms that are actually summed into $\\hat V$), and we have dependence across $(d,d')$ and $(d'',d''')$. Then, \\hl{the six distinct ways} that such dependence can occur and the associated proportion of quadruples for which these occur for a sample of size $N$ are as follows: [\\ldots]", "corrected": "\\emph{Proof of Proposition 1.} Assume, as on p.~566, that the units are an i.i.d.\\ sample from a fixed population, that $(X_d,Y_d)$ is a function of the characteristics of the two units of dyad $d$, that $\\|X_d\\|\\le C_X$ and $Y_d$ is bounded, and that $Q=\\mathrm E[X_dX_d']$ is nonsingular. Let $\\beta$ be the population least-squares coefficient and $\\epsilon_d=Y_d-X_d'\\beta$, so that $\\mathrm E[X_d\\epsilon_d]=0$ and $|\\epsilon_d|\\le C_e$ for a constant $C_e$. Let $Q_N=\\mathbf X'\\mathbf X/D$ and, for residuals $r=(r_d)$, let\n\\[\nM_N(r)=\\hl{\\frac{N}{D^2}}\\sum_{(d,d'):\\,s(d)\\cap s(d')\\ne\\emptyset}X_dX_{d'}'r_dr_{d'},\\qquad T_N=\\mathrm E[M_N(\\epsilon)]=\\frac{N}{D^2}\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big].\n\\]\nThe sum runs over ordered pairs of dyads and includes $d'=d$. The second expression for $T_N$ holds because the scores $X_d\\epsilon_d$ have mean zero and the scores of two dyads with no common unit are independent. The target of equation (1) is $V=Q^{-1}T_NQ^{-1}$, and equation (3) reads $N\\hat V=Q_N^{-1}M_N(e)Q_N^{-1}$. The proof has four steps, each \\hl{in probability}.\n\n\\emph{Step 1.} Each dyad shares a unit with exactly $2N-3$ dyads, itself included. The summands of $Q_N$ are bounded and $\\mathrm E[Q_N]=Q$, so each entry of $Q_N$ has variance at most a constant times $D(2N-3)/D^2=O(N^{-1})$. Chebyshev's inequality gives $Q_N\\to Q$ in probability, and $Q_N^{-1}\\to Q^{-1}$ in probability because $Q$ is nonsingular. The same count gives $D^{-1}\\sum_dX_d\\epsilon_d=O_p(N^{-1/2})$. Hence $\\delta_N=\\hat\\beta-\\beta=Q_N^{-1}D^{-1}\\sum_dX_d\\epsilon_d=O_p(N^{-1/2})$.\n\n\\emph{Step 2.} The sum in $M_N$ has $D(2N-3)$ terms, so $\\|T_N\\|\\le4C_X^2C_e^2$. Each term involves two or three units, and it is independent of every term that has no unit in common with it. At most $3(N-1)(3N-5)$ terms have a unit in common with a given term. At most $D(2N-3)\\cdot3(N-1)(3N-5)=O(N^5)$ pairs of terms then have a covariance that is not zero, and each covariance is at most $2C_X^4C_e^4$ in absolute value. For each entry $(a,b)$,\n\\[\n\\operatorname{Var}\\big([M_N(\\hl{\\epsilon})]_{ab}\\big)\\le\\frac{N^2}{D^4}\\,2C_X^4C_e^4\\,D(2N-3)\\,3(N-1)(3N-5)=O(N^{-1}),\n\\]\nand Chebyshev's inequality gives $M_N(\\epsilon)-T_N\\to0$ in probability.\n\n\\emph{Step 3.} Since $e_d=\\epsilon_d-X_d'\\delta_N$, every pair of dyads satisfies $|e_de_{d'}-\\epsilon_d\\epsilon_{d'}|\\le2C_eC_X\\|\\delta_N\\|+C_X^2\\|\\delta_N\\|^2$. Hence\n\\[\n\\hl{\\|M_N(e)-M_N(\\epsilon)\\|\\le4C_X^2\\big(2C_eC_X\\|\\delta_N\\|+C_X^2\\|\\delta_N\\|^2\\big)}=o_p(1).\n\\]\n\n\\emph{Step 4.} By Steps 2 and 3, $M_N(e)-T_N=o_p(1)$. The identity\n\\[\nN\\hat V-V=Q_N^{-1}\\big[M_N(e)-T_N\\big]Q_N^{-1}+(Q_N^{-1}-Q^{-1})T_NQ_N^{-1}+Q^{-1}T_N(Q_N^{-1}-Q^{-1})\n\\]\nand Step 1 imply that $N\\hat V-V\\overset{p}{\\to}0$, since $T_N$ is bounded and $Q_N^{-1}=O_p(1)$. No mean or variance of $Q_N^{-1}$ or of $N\\hat V$ is used. If $Q_N$ is singular on some samples, $\\hat V$ may be defined arbitrarily there, because the probability of these samples tends to zero.", "evidence": "The evidence is an exact counterexample to the printed steps and a derivation of the proposition by another route, given in the corrected proof below. Let $U_1,\\dots,U_N$ be independent with $P(U_i=1)=P(U_i=-1)=1/2$, and for each of the $D=N(N-1)/2$ dyads let $X_{ij}=1$ and $Y_{ij}=U_i+U_j$. In this example $\\beta=0$ and $\\mathbf X'\\mathbf X=D$, all variables are bounded, and $V=4$. The fitted residuals are $e_{ij}=U_i+U_j-2\\bar U$, where $\\bar U$ is the mean of the $U_i$. Computing the moments directly, I obtain\n\\[\n\\Sigma=N(N-1)^2,\\qquad\\mathrm E[\\hat\\Sigma]=(N-1)(N-2)(N-3),\\qquad\\mathrm E[\\hat\\Sigma]-\\Sigma=(-4N+6)(N-1)\\to-\\infty,\n\\]\n\\[\n\\mathrm E[N\\hat V]=\\frac{4(N-2)(N-3)}{N(N-1)}\\to4,\\qquad\\mathrm E[\\hat V-V]\\to-4,\\qquad\\operatorname{Var}[N\\hat V]=\\frac{32(N-2)^2(N-3)^2}{N^3(N-1)^3}\\to0.\n\\]\nBy the first and last limits of the second display and Chebyshev's inequality, I have $N\\hat V\\to4=V$ in probability, as Proposition 1 states. Both printed limits of means are false in this example. For $N\\ge6$, I find\n\\[\n\\operatorname{Cov}(e_{12}e_{13},\\,e_{45}e_{46})=\\frac{16(N+6)}{N^3},\n\\]\nwhich equals $7/16$ at $N=8$, although the unit sets $\\{1,2,3\\}$ and $\\{4,5,6\\}$ are disjoint. The same covariance for the population residuals is zero. At $N=8$, I also have $\\operatorname{Cov}(e_{12}e_{13},\\,e_{14}e_{15})=-1/16$. Enumeration of all $2^N$ sign patterns in exact arithmetic for $N=4,6,8$ agrees with these formulas.\n\\begin{audittable}{lrrr}{Exact values in the example, for which $V=4$.}\nQuantity & $N=4$ & $N=6$ & $N=8$ \\\\ \\hline\n$\\Sigma$ & $36$ & $150$ & $392$ \\\\\n$\\mathrm E[\\hat\\Sigma]$ & $6$ & $60$ & $210$ \\\\\n$\\mathrm E[\\hat V-V]$, as printed in (i) & $-23/6$ & $-56/15$ & $-209/56$ \\\\\n$\\mathrm E[N\\hat V-V]$ & $-10/3$ & $-12/5$ & $-13/7$ \\\\\n$\\operatorname{Var}[N\\hat V]$ & $2/27$ & $64/375$ & $225/1372$ \\\\\n$\\operatorname{Cov}(e_{12}e_{13},\\,e_{45}e_{46})$ & -- & $8/9$ & $7/16$ \\\\\n\\end{audittable}\nFor the third step, let $T_1,\\dots,T_N$ be independent with density $t^{-2}$ on $[1,\\infty)$, and let $Z_i=e^{-T_i}$ and $X_{ij}=(1,Z_i+Z_j)'$. The regressors are bounded, the sample design has full rank almost surely for $N\\ge3$, and the population Gram matrix is nonsingular. For each $k\\ge1$, every $T_i$ is at least $k$ with probability $k^{-N}$, and then the norm of $(\\mathbf X'\\mathbf X/D)^{-1}$ is at least $e^{2k}/4$. This implies that the expected norm of this inverse is infinite for every $N\\ge3$, while the inverse converges in probability to its population value. The example shows that there is no automatic bound on the moments of the inverse. It does not show that $N\\hat V$ has infinite moments.", "problem": "Proposition 1 states that $N\\hat V-V\\to0$ in probability, where $\\hat V$ is the estimator (3). The printed proof attempts to bound the mean and the variance of $N\\hat V$ and applies Chebyshev's inequality. Three of its steps fail.\n\nThe first is the scaling. With condition (ii), Chebyshev's inequality requires $\\mathrm E[N\\hat V-V]\\to0$. Condition (i) is printed without the factor $N$, and p.~574 repeats it in that form. The printed pair of conditions is not sufficient, and in the example below the printed mean tends to $-4$ and not to $0$. The display $\\mathrm E[\\hat\\Sigma]\\to\\Sigma$ compares two unnormalized sums, and consistency of $\\hat\\beta$ does not make their difference vanish.\n\nThe second concerns the residuals. The $e_d$ in $\\hat\\Sigma$ and $\\Omega$ are the fitted residuals $e_d=Y_d-X_d'\\hat\\beta$, which p.~566 distinguishes from the population residuals $\\epsilon_d$. Since $\\hat\\beta$ depends on every unit, the products $e_de_{d'}$ and $e_{d''}e_{d'''}$ can be correlated even when the two pairs of dyads have no unit in common. Each of the six cases on p.~574 requires a common unit, and the six cases therefore do not exhaust the covariances that are not zero. These covariances can also be negative, and the count has to bound absolute values.\n\nThe third concerns the inverse Gram matrix. The proof treats the random matrix $(\\mathbf X'\\mathbf X/D)^{-1}$ as a constant of order one inside the mean and the variance of $N\\hat V$. Convergence in probability implies that this matrix is bounded in probability, but not that its moments are bounded. Boundedness of $X_d$ bounds $\\mathrm E[X_dX_d']$ and not its inverse. The inverse is finite because of the rank condition of p.~566.", "what_survives": "The estimator (3) and the finite-sample identity of Proposition 2 are unchanged. Proposition 1 holds for $V$ as defined in equation (1), under the paper's assumptions and with no new one; the conditional form of $\\Sigma$ printed in the proof is corrected in \\fid{P018-E001}. Corollaries 1 and 2 appeal to Proposition 1 and require analogous probability arguments. The same fitted-score issue must be addressed for generalized linear models. These defects establish no change to the reported simulation or application numbers; I have not recalculated them here. The moments of $N\\hat V$ and of $\\hat\\beta$ are the subject of \\fid{P018-N004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E006", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of two displays that follow the word and. The first defines Sigma-hat C,i as the sum over j not equal to i and over k not equal to i of X d(i,j) transposed times e d(i,j) times e d(i,k) transposed times X d(i,k), plus a sum indexed by j, k not equal to i of the sum over t from 1 to T of d(j,k) of X d(j,k)t times X d(j,k)t transposed times e d(j,k)t squared. The second defines Sigma-hat D and Sigma-hat 0. A red outline surrounds the index j, k not equal to i under the third summation sign. The first line of the next paragraph follows.", "caption": "Proposition 2, p. 568 (PDF p. 5): the definition of $\\widehat\\Sigma_{C,i}$. The outline marks the index $j, k \\ne i$ of the sum over the dyads that do not contain unit $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E007.png"}, {"alt": "Crop of one display introduced by the word Define: D i equals the sum over j not equal to i of X d(i,j) transposed times e d(i,j) times e d(i,j) transposed times X d(i,j), and E i equals a sum indexed by j, k not equal to i of the sum over t from 1 to T of d(j,k) of X d(j,k)t times X d(j,k)t transposed times e d(j,k)t squared. A red outline surrounds the index j, k not equal to i under the first summation sign of E i. The next line reads: Then, working with Sigma-hat C,i, Sigma-hat D, and Sigma-hat 0 as defined in the proposition.", "caption": "Appendix C, p. 575 (PDF p. 12): the definition of $E_i$. The outline marks the same index $j, k \\ne i$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E007-b.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T15", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T15"}, {"id": "P018-T16", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T16"}], "headline": "The sum over the dyads that do not contain unit $i$ runs over $j < k$ with $j, k \\ne i$, so that each such dyad is counted once.", "id": "P018-E007", "location": "Proposition 2, definition of $\\widehat\\Sigma_{C,i}$, p. 568 (PDF p. 5); the same index in $E_i$, Appendix C, p. 575 (PDF p. 12)", "pdf_page": 5, "printed_page": "568", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T15", "P018-T16"], "text": {"as_printed": "\\[\n\\widehat\\Sigma_{C,i} = \\sum_{j \\ne i} \\sum_{k \\ne i} X_{d(i,j)}' e_{d(i,j)} e_{d(i,k)}' X_{d(i,k)} + \\sum_{\\hl{j,k \\ne i}} \\sum_{t=1}^{T(d(j,k))} X_{d(j,k)t} X_{d(j,k)t}' e_{d(j,k)t}^{2},\n\\]\nand, in Appendix~C,\n\\[\nE_i = \\sum_{\\hl{j,k \\ne i}} \\sum_{t=1}^{T(d(j,k))} X_{d(j,k)t} X_{d(j,k)t}' e_{d(j,k)t}^{2}.\n\\]", "corrected": "\\[\n\\widehat\\Sigma_{C,i} = \\sum_{j \\ne i} \\sum_{k \\ne i} X_{d(i,j)}' e_{d(i,j)} e_{d(i,k)}' X_{d(i,k)} + \\sum_{\\substack{\\hl{j < k} \\\\ j, k \\ne i}} \\sum_{t=1}^{T(d(j,k))} X_{d(j,k)t} X_{d(j,k)t}' e_{d(j,k)t}^{2},\n\\]\nand, in Appendix~C,\n\\[\nE_i = \\sum_{\\substack{\\hl{j < k} \\\\ j, k \\ne i}} \\sum_{t=1}^{T(d(j,k))} X_{d(j,k)t} X_{d(j,k)t}' e_{d(j,k)t}^{2}.\n\\]", "evidence": "The evidence is a derivation by counting, with an exact example. A dyad avoids exactly $N - 2$ units. If each dyad without unit $i$ enters $E_i$ once, then $\\sum_i E_i = (N-2)\\widehat\\Sigma_0$, and $\\sum_i \\widehat\\Sigma_{C,i} - \\widehat\\Sigma_D - (N-2)\\widehat\\Sigma_0$ equals the middle factor of $\\widehat V_r$. If each enters twice, then $\\sum_i E_i = 2(N-2)\\widehat\\Sigma_0$. For $N = 4$, one observation per dyad and scalar scores $X_d e_d = 1, 2, \\dots, 6$, I have $\\widehat\\Sigma_0 = 91$, and the middle factor of $\\widehat V_r$, computed directly from its definition, is $385$. Counting each dyad once, I find $\\sum_i E_i = 182$, and the decomposition returns $385$. Counting each dyad twice gives $\\sum_i E_i = 364$, and the decomposition returns $567$. The replication script assigns each observation without unit $i$ to its own cluster, which counts each dyad once.", "problem": "The index $j, k \\ne i$ carries neither $j < k$ nor $j \\ne k$. The first double sum of the same display runs over ordered pairs. Read in the same way, the second sum includes the dyads $d(j,j)$, which the paper excludes, and counts every other dyad twice, since $d(j,k) = d(k,j)$. The identity $(N-2)\\widehat\\Sigma_0 = \\sum_i E_i$ of Appendix~C then fails by a factor of $2$. The sentence below Proposition~2 and the count in Appendix~C show that each dyad without unit $i$ is meant to enter once.", "what_survives": "Proposition~2 and the identities of Appendix~C hold with the index as corrected. The replication code and every reported number are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E007", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of text. They read: with normal data) corresponds precisely to the assumptions of a two-way normal random effects model, and so it is no surprise that we find this model to be more efficient than OLS and to produce consistent standard errors. However, as discussed above, such models are sensitive to misspecification. For example, in fitting a linear approximation model to mildly non-linear data, the dyadic cluster robust standard errors continue to be consistent for the coefficients of the linear approximation. The degree of misspecification in this case is not extreme (as illustrated in the. Red outlines surround the words corresponds precisely and the words consistent standard errors.", "caption": "Text below Figure 1, p. 570 (PDF p. 7), the continuation of the paragraph that begins on p. 569. The outlines mark “corresponds precisely” and “consistent standard errors.”", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E008.png"}, {"alt": "Crop of eight lines of text. They read: Figure 2 uses the same data generating process as was described in the main text, and presents the same results for OLS with either the dyadic cluster robust standard error estimator or the heteroskedastic robust estimator. In addition, we have added random effects estimates. Given that the data generating process conforms exactly to the assumptions of a two-way random effects model, we obtain the straightforward result that the random effects estimates are both more efficient (the blue diamonds tend to be lower than the black ones) and the random effects standard errors are consistent. Figure 3 illustrates the robustness of the dyadic cluster robust estimator relative to a random. Red outlines surround the words conforms exactly and the words the random effects standard errors are consistent.", "caption": "Supporting Information, p. 2, paragraph under Figure 2. The outlines mark “conforms exactly” and “the random effects standard errors are consistent.”", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E008-b.png"}, {"alt": "Crop of five lines of text. They read: A cost of the robust approach is efficiency. Our simulations show that a two-way (for cross-sectional data) random effects model can be considerably more efficient and provide reliable inference when the conditional mean is correctly specified. Our robust estimator could provide the basis of a test for misspecification (White 1980a, 1981; King and Roberts 2015). If there is little evidence of misspecification, the random effects estimator would be a reasonable choice given its efficiency. A red outline in two parts surrounds the words provide reliable inference when the conditional mean is correctly specified.", "caption": "Conclusion, p. 572 (PDF p. 9). The outline marks the claim of reliable inference when the conditional mean is correctly specified.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E008-c.png"}, {"alt": "The published plot titled s.e. beta-hat 0 (cross section), with vertical axis Standard error labelled 0.0, 0.4 and 0.8 and horizontal axis N at 20, 50, 100 and 150. At each N it shows a black box and a black diamond, a blue box and a blue diamond, and a red box near the bottom. Added to the plot at each N is a short orange horizontal mark at the height 2 over the square root of N: 0.447, 0.283, 0.200 and 0.163. At each of the four sample sizes the blue box lies below the orange mark, and the blue diamond lies at about the height of the mark or slightly above it. A heading above the plot reads: The published random-effects standard errors for the intercept lie below the exact lower bound 2 over the square root of N. A note below says that the orange marks were added and that the rest is the panel as published.", "caption": "Figure 2 of the Supporting Information, left panel (p. 2), as published, with the exact lower bound $2/\\sqrt{N}$ for the standard deviation of the random effects intercept estimator added in orange. At each sample size the blue box of random effects standard errors lies below the bound.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E008-intercept-bound.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T5", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T5"}, {"id": "P018-T6", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T6"}, {"id": "P018-T25", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T25"}], "headline": "The random effects model fitted in the simulations assigns each unit two independent effects, one for each position in the dyad; it is misspecified even in the normal linear design, and the simulations do not show that its standard errors are consistent or that inference from it is reliable.", "id": "P018-E008", "location": "Paragraph that begins on p. 569 and continues below Figure 1 on p. 570 (PDF pp. 6–7); Supporting Information, p. 2 (text under Figure 2) and p. 4; Conclusion, p. 572 (PDF p. 9)", "pdf_page": 6, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P018-T25", "P018-T5", "P018-T6"], "text": {"as_printed": "Main text, pp.~569--570: ``Specifically, we compare OLS with dyadic cluster robust standard errors to a two-way random effects model that incorporates normal random effects for each of the $N$ units (Gelman and Hill 2007). The data-generating process captured above (i.e., expression 4 with normal data) \\hl{corresponds precisely} to the assumptions of a two-way normal random effects model, and so it is no surprise that we find this model to be more efficient than OLS and to produce \\hl{consistent standard errors}.''\n\nSupporting Information, p.~2: ``Given that the data generating process \\hl{conforms exactly} to the assumptions of a two-way random effects model, we obtain the straightforward result that the random effects estimates are both more efficient (the blue diamonds tend to be lower than the black ones) and \\hl{the random effects standard errors are consistent}.''\n\nConclusion, p.~572: ``Our simulations show that a two-way (for cross-sectional data) random effects model can be considerably more efficient and \\hl{provide reliable inference when the conditional mean is correctly specified}.''", "corrected": "Main text, pp.~569--570: ``Specifically, we compare OLS with dyadic cluster robust standard errors to a two-way random effects model \\hl{with independent normal random effects for the first-listed and the second-listed unit of each dyad. The data-generating process captured above (i.e., expression 4 with normal data) has a single effect for each unit, the same in either position, and the fitted model is therefore misspecified in its covariance even with normal data. In these simulations its estimates vary less than the OLS estimates, but its standard errors for the intercept are understated, and the comparison does not demonstrate reliable random effects inference.}''\n\nSupporting Information, pp.~2 and~4: the same change, with one addition. \\hl{The nonlinear design adds misspecification of the conditional mean to a covariance that is already misspecified, and the discrepancy in Figure~3 cannot be attributed to the mean alone.}\n\nConclusion, p.~572: ``Our simulations show that a two-way (for cross-sectional data) random effects model can be considerably more efficient. \\hl{They do not show that it provides reliable inference, because the model fitted was not correctly specified.}''\n\nA correctly specified comparison requires a single random effect per unit, entering each dyad through both of its members, and new simulations.", "evidence": "The evidence is an exact counterexample with a derivation. For $N = 3$, I order the dyads as $\\{1,2\\}$, $\\{1,3\\}$, $\\{2,3\\}$. Let $\\tau_1$ and $\\tau_2$ be the variances of the two fitted effects and $\\sigma^2$ the residual variance. The covariance matrix of the simulated errors and that of the fitted model are\n\\[\n\\begin{pmatrix} 3 & 1 & 1 \\\\ 1 & 3 & 1 \\\\ 1 & 1 & 3 \\end{pmatrix} \\qquad \\text{and} \\qquad \\begin{pmatrix} \\tau_1 + \\tau_2 + \\sigma^2 & \\tau_1 & 0 \\\\ \\tau_1 & \\tau_1 + \\tau_2 + \\sigma^2 & \\tau_2 \\\\ 0 & \\tau_2 & \\tau_1 + \\tau_2 + \\sigma^2 \\end{pmatrix}.\n\\]\nThe corner entries differ for every choice of $\\tau_1$, $\\tau_2$ and $\\sigma^2$, and these three dyads occur in every sample. The design matrix that \\texttt{lme4} builds for the two terms also implies this zero covariance.\n\nA wrong covariance alone does not show that the standard error of a coefficient is wrong. In the following example it is. Let the unit covariates be $(0, 1, 3)$, so that the dyad regressor is $(1, 3, 2)$, and fix $\\tau_1 = \\tau_2 = \\sigma^2 = 1$. The generalised least squares estimator under the fitted covariance reports the intercept variance $71/16$. Its variance under the simulated covariance is $379/64$. The ratio of the two is $284/379$, and the ratio of the standard errors is $0.866$. This computation holds the variance components fixed and is not a statement about the fitted model with estimated components.\n\nThe variance of the intercept estimator of the fitted model itself has a lower bound. Let $\\bar\\alpha$ denote the mean of $\\alpha_1, \\dots, \\alpha_N$. Every error contains the term $2\\bar\\alpha$, since\n\\[\n\\alpha_i + \\alpha_j + \\nu_{d(i,j)t} = 2\\bar\\alpha + (\\alpha_i - \\bar\\alpha) + (\\alpha_j - \\bar\\alpha) + \\nu_{d(i,j)t},\n\\]\nand $\\bar\\alpha$ is independent of the other three terms and of the regressors. Adding a constant $c$ to every outcome leaves the restricted likelihood of the variance components unchanged and adds $c$ to the estimated intercept. The intercept estimate is therefore the sum of $2\\bar\\alpha$ and an independent variable, and I have\n\\[\n\\operatorname{Var}[\\widehat\\beta_0] \\ge 4 \\operatorname{Var}[\\bar\\alpha] = \\frac{4}{N}.\n\\]\n\\begin{audittable}{rrr}{Exact lower bounds for the variance and the standard deviation of the random effects intercept estimator in the simulation design.}\n$N$ & Variance, $4/N$ & Standard deviation, $2/\\sqrt{N}$ \\\\\n$20$ & $1/5$ & $0.447$ \\\\\n$50$ & $2/25$ & $0.283$ \\\\\n$100$ & $1/25$ & $0.200$ \\\\\n$150$ & $2/75$ & $0.163$ \\\\\n\\end{audittable}\nIn the left panel of Figure~2 of the Supporting Information, the boxes of the standard errors of the random effects model lie below these bounds at all four sample sizes. This is a reading of the published figure. The original simulation output is not in the replication files, and no numerical ratio was extracted. The evidence does not establish the limit of the fitted standard errors as $N$ grows, a particular percentage shortfall, or any effect on the standard errors of the slope.", "problem": "In equation~(4) the error of the dyad $\\{i, j\\}$ is $\\alpha_i + \\alpha_j + \\nu_{d(i,j)t}$. Each unit has one effect $\\alpha_i$, which enters every dyad that contains the unit. The replication scripts list the lower-numbered unit of each dyad first. They fit the random effects model with the terms \\texttt{(1|dyad1) + (1|dyad2)}, one for the unit listed first and one for the unit listed second. The two terms are independent random vectors. A unit has one effect when it is listed first and another, independent effect when it is listed second. The dyads $\\{1, 2\\}$ and $\\{2, 3\\}$ share unit~2 in different positions. Their errors have covariance $1$ in the simulated data and covariance $0$ in the fitted model, for all values of its variance components. The data-generating process does not correspond to the fitted model, whose covariance is wrong at every sample size, even in the normal linear design. The same holds in the design with mild misspecification of the mean (Figure~3 of the Supporting Information). It also holds in the fit with repeated observations, where the added dyad effect does not restore the missing covariance. Two statements of the paper are not supported for the model as fitted. The first is that its standard errors are consistent and its inference reliable when the conditional mean is correct. In Figure~2 of the Supporting Information, the boxes of random effects standard errors for the intercept lie below the plotted standard deviation of the estimates at all four sample sizes. The second is that the failure of these standard errors in Figure~3 comes from the misspecified mean. That comparison changes the mean of a model whose covariance was already misspecified.", "what_survives": "The dyadic cluster robust estimator, its theory, the OLS estimates, Figure~1 and the two applications do not use the random effects fit and are not affected. The published figures may still show correctly that the estimates of the random effects model vary less than the OLS estimates. A random effects model with one effect per unit is correctly specified for equation~(4); its performance, and the efficiency comparison, would have to be simulated anew. The simulations do not support the statements that the random effects model has consistent standard errors and provides reliable inference. The failure in Figure~3 of the Supporting Information cannot be attributed to the mean alone."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E008", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Figure 4 of the Supporting Information: a scatter plot of Y against the absolute difference of X_i and X_j, which runs from 0 to about 3.4, with a red straight line and a blue curve that bends upward and ends near height 7.2. Below the plot is the caption 'Figure 4: Illustrative scatter plot and linear approximation for the misspecification case. The blue line is the true conditional mean, and the red line is the linear approximation.' Red boxes mark 'The blue line is the true conditional mean,'.", "caption": "Supporting Information, Figure 4 and its caption, p. 3. The marks are on the part of the caption that calls the blue line the true conditional mean.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E009.png"}, {"alt": "Crop of seven lines of text from page 4 of the Supporting Information. A red box marks 'and the true conditional mean (in blue).' in the sentence 'Figure 4 shows the scatter plot of Y against the absolute difference of X_i and X_j, the linear approximation (in red) for one of the simulation replicates (N = 50), and the true conditional mean (in blue).'", "caption": "Supporting Information, p. 4. The mark is on “and the true conditional mean (in blue)”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The blue curve in Figure 4 of the Supporting Information is a quadratic least-squares fit to the plotted sample, not the true conditional mean $x+0.25x^2$.", "id": "P018-E009", "location": "Supporting Information, Figure 4 and its caption, p. 3 (supplement PDF p. 3); the sentence beginning “Figure 4 shows”, p. 4 (supplement PDF p. 4)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Supporting Information, p.~3: ``Figure 4: Illustrative scatter plot and linear approximation for the misspecification case. \\hl{The blue line is the true conditional mean}, and the red line is the linear approximation.''\n\nSupporting Information, p.~4: ``Figure 4 shows the scatter plot of $Y_{d(i,j)t}$ against $|X_i-X_j|$, the linear approximation (in red) for one of the simulation replicates ($N=50$), \\hl{and the true conditional mean (in blue)}.''", "corrected": "Caption: ``\\ldots \\hl{The blue line is a quadratic least-squares fit to the plotted sample}, and the red line is the linear \\hl{least-squares fit}.''\n\np.~4: ``\\ldots the linear approximation (in red) for one of the simulation replicates ($N=50$), \\hl{and a quadratic least-squares fit to that sample (in blue)}.''\n\nAlternatively, the label can stay if the figure is drawn again with the curve $x+0.25x^2$ in blue.", "evidence": "The evidence is a check of the script and an exact comparison with the published figure. Line 437 of the graph script draws the blue curve from the values \\texttt{predict(lm(dY.ord$\\sim$dX.ord+I(dX.ord$\\text{\\textasciicircum}$2)))}. In the published PDF, the coordinates of the blue curve and of the axis ticks place its right end at $x\\approx3.363$ and height $7.247$, where $x+0.25x^2\\approx6.191$. The gap of about $1.06$ stays above 1 when the rounding of the coordinates is allowed for.", "problem": "Let $x=|X_i-X_j|$. In the design of the Supporting Information, $\\beta_0=0$, $\\beta_1=1$ and $\\beta_2=0.25$, and the shocks have mean zero and are independent of the covariates, so that the conditional mean of $Y$ given $x$ is $x+0.25x^2$. The plotting script in the replication files draws the blue curve as the fitted values of a least-squares regression of the simulated outcomes on $x$ and $x^2$ with an intercept. Its three coefficients are estimates from the one sample plotted, and with probability one they differ from $(0,1,0.25)$.", "what_survives": "The figure is an illustration, and the fitted curve enters no number. The design, the simulation summaries in Figure 3 of the Supporting Information and the conclusions drawn from them are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E009", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Table 1 without its title: nine rows of coefficients and standard errors in six columns, the first two headed Original GEE estimates and the last four Simple logistic regression estimates. In the note below the table, red boxes mark the numbers 39,988 and 1,178.", "caption": "Table 1, p. 572 (PDF p. 9). The marked counts in the note stand under all six columns; columns (3)–(6) are computed from 39,996 observations in 1,186 dyads.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E010.png"}, {"alt": "Crop of four lines from the second paragraph of Section 5. Red boxes mark the words 1178 dyads and the number 39,988.", "caption": "P. 571 (PDF p. 8), second paragraph of Section 5. The marked counts are the same as in the note to Table 1.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E010-b.png"}, {"alt": "Crop of three lines from the first paragraph of the speed-dating section of the Supporting Information. A red box marks the words 278 women.", "caption": "Supporting Information, p. 5, first paragraph. The marked count of women is 278; the sample that reproduces Table 2 has 268.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E010-c.png"}, {"alt": "Crop of Table 2 of the Supporting Information with its title: three rows, Ambition, Attractiveness and Intelligence, and four columns of coefficients and standard errors. In the note below the table, a red box marks the words 278 subjects.", "caption": "Supporting Information, p. 5, Table 2. The marked note repeats the count of 278 subjects.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E010-d.png"}], "formal_release": null, "formal_statements": [], "headline": "The pooled logistic columns of Table 1 use 39,996 observations in 1,186 dyads, not 39,988 and 1,178. A processed copy of the speed-dating data reproduces Table 2 of the Supporting Information with 268 women, against the printed 278; this count remains unconfirmed against the authors' raw file.", "id": "P018-E010", "location": "Note to Table 1, p. 572 (PDF p. 9); Section 5, second paragraph, p. 571 (PDF p. 8); Supporting Information, p. 5, first paragraph and note to Table 2", "pdf_page": 9, "printed_page": "572", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~571: They use annual data on 146 states in the international system paired into \\hl{1178} dyads (out of 10,585 possible) and observed for as few as one and as many as 90 years between 1885 and 1991, for a total of \\hl{39,988} observations.\n\nTable 1, p.~572: \\emph{Note}: $N=\\hl{39{,}988}$ dyad--year observations for \\hl{1,178} dyads and 146 units.\n\nSupporting Information, p.~5: These data include \\hl{278} women paired into 3,457 female-male dyads. [\\ldots] $N=3{,}457$ dyad observations for \\hl{278} subjects.", "corrected": "Note to Table 1: columns (3)--(6) use $N=\\hl{39{,}996}$ dyad--year observations for \\hl{1,186} dyads and 146 units, with 1 to 90 observations per dyad. If the GEE of columns (1)--(2) excludes the eight dyads with a single observation, its sample has $N=39{,}988$ observations for 1,178 dyads and \\hl{145} units, with \\hl{2} to 90 observations per dyad; this is to be confirmed from the estimation sample.\n\nSupporting Information, p.~5: These data include \\hl{268} women paired into 3,457 female-male dyads. [\\ldots] $N=3{,}457$ dyad observations for \\hl{268} subjects. This count is to be confirmed against the authors' raw data file before the text is changed.", "evidence": "I recalculated the sample counts from the replication data; they are exact.\n\\begin{audittable}{lrrrr}{The two samples in the replication data for Table 1, and the printed description.}\nSample & Observations & Dyads & States & Observations per dyad \\\\ \\hline\nAll complete dyad-years & 39,996 & 1,186 & 146 & 1 to 90 \\\\\nWithout the eight single-observation dyads & 39,988 & 1,178 & 145 & 2 to 90 \\\\\nAs printed (p.~571 and note to Table 1) & 39,988 & 1,178 & 146 & 1 to 90 \\\\\n\\end{audittable}\nThe count of 268 women comes from a processed copy of the speed-dating data that is distributed with an R package by other authors. With the fixed effects and weights of the authors' script, this copy reproduces all 12 entries of Table 2 to four decimals. I did not use the authors' raw data file, and I have not confirmed the count of 268 against that file.\n\nI have not established which sample the GEE of columns (1)--(2) used. Since an AR(1) working correlation is defined for a dyad with one observation, the model alone does not require that these dyads be dropped. My GEE calculations on the smaller sample approximate columns (1)--(2), but do not reproduce every entry to the printed rounding or establish that this was the GEE sample. The original Stata run was not repeated.", "problem": "The note stands under the whole of Table 1 and gives no separate counts for columns (3)--(6). The replication data hold 39,996 complete dyad-years in 1,186 dyads among 146 states, with 1 to 90 observations per dyad. The pooled logistic regression on all of them reproduces every coefficient in column (3) and every dyadic standard error in column (6). The printed counts 39,988 and 1,178 are what remains when the eight dyads with a single observation are removed. That smaller sample has 145 states and 2 to 90 observations per dyad, and several of its coefficients do not round to the printed ones: for lower dependence it gives $-67.733$ against the printed $-67.668$. Neither sample has all four properties stated on p.~571.\n\nIn the Supporting Information, the sample of 3,457 dyads that reproduces Table 2 has 268 women and 277 men.", "what_survives": "The nine coefficients of the pooled logistic regression in Table 1 and their dyadic standard errors are reproduced unchanged, and the test statistic for international organizations is $1.31$, below the threshold $1.64$. All 12 entries of Table 2 of the Supporting Information are reproduced. No theorem, formula or simulation depends on these counts. Only the descriptions of the samples change."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E010", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines from the list of predictors in Section 5. A red box marks the phrase an indicator of whether both states are members of a common international organization.", "caption": "P. 571 (PDF p. 8), first paragraph of Section 5. The marked phrase describes the international-organizations predictor as an indicator; the regressions use a count.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E011.png"}], "formal_release": null, "formal_statements": [], "headline": "The international-organizations predictor is the number of international governmental organizations to which both states belong, and not an indicator of a common membership.", "id": "P018-E011", "location": "Section 5, first paragraph, p. 571 (PDF p. 8), list of predictors", "pdf_page": 8, "printed_page": "571", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "the ratio of the value of trade and the larger GDP of the two states, \\hl{an indicator of whether both states are members of a common international organization}, an indicator of whether the states are geographically noncontiguous, the log of the geographic (Euclidean) distance between the two states' capitals,", "corrected": "the ratio of the value of trade and the larger GDP of the two states, \\hl{the number of international governmental organizations to which both states belong}, an indicator of whether the states are geographically noncontiguous, the log of the geographic (Euclidean) distance between the two states' capitals,", "evidence": "The evidence is an observation in the replication data and a recalculation. For the dyad with state codes 2 and 20 in 1920 the variable equals 10, and an indicator of a common membership would equal 1. With the count as the regressor, the pooled logistic regression reproduces the nine coefficients of Table 1 and their dyadic standard errors to three decimals.", "problem": "The regressions in the replication files use the variable \\texttt{smigoabi} as it stands, without reducing it to two values. Its label in the data is ``jnt IGO members, interpolated'', and it takes every integer value from 0 to 130. The coefficient $-0.011$ in Table 1 is therefore the change in the log-odds of a dispute for one more shared organization. It is not the difference between dyads with and without a common membership.", "what_survives": "The estimate $-0.01117$ and its dyadic standard error $0.00850$ are unchanged. Since their ratio is $1.31$ in absolute value, the coefficient still fails the threshold $1.64$. Only the description of the predictor changes, and with it the unit of the coefficient. The same sentence describes the distance variable as the distance between the two capitals, while its label in the data is ``ln distance, min port-to-port''. The replication files do not show how the distances were constructed, and I propose no correction there."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E011", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of five lines from the second paragraph of Section 5. Red boxes mark the words sometimes an order of, at the end of one line, and magnitude larger, at the start of the next line.", "caption": "P. 571 (PDF p. 8), second paragraph of Section 5. The marked words state a tenfold increase; the ratios in Table 1 are between 2.3 and 4.4.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E012.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T24", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T24"}], "headline": "In Table 1 the dyadic cluster-robust standard errors are about 2.3 to 4.4 times as large as those that ignore all clustering, and not an order of magnitude larger.", "historical_evidence_type": "lean_proof", "id": "P018-E012", "location": "Section 5, second paragraph, p. 571 (PDF p. 8), sentence beginning “Accounting for the dyadic and repeated-observation clustering”", "pdf_page": 8, "printed_page": "571", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T24"], "text": {"as_printed": "Accounting for the dyadic and repeated-observation clustering results in standard error estimates that are \\hl{sometimes an order of magnitude larger than} what we obtain when we ignore all clustering and also considerably larger than what one would estimate were one to account for repeated dyads clustering but ignore the inter-dyadic clustering.", "corrected": "Accounting for the dyadic and repeated-observation clustering results in standard error estimates that are \\hl{about 2.3 to 4.4 times as large as the displayed estimates that ignore all clustering} and also considerably larger than what one would estimate were one to account for repeated dyads clustering but ignore the inter-dyadic clustering.", "evidence": "The evidence is exact arithmetic on the nine rows of Table 1 and a Lean proof that standard errors which round to the printed entries have a ratio below ten in every row. The largest ratio that the rounding allows is $0.0085/0.0015=17/3$, for international organizations. A recalculation from the replication data gives ratios from $2.31$ to $4.68$.", "problem": "The standard errors that ignore all clustering are in column (4) of Table 1, and those that account for dyadic clustering are in column (6). Dividing column (6) by column (4), I find ratios from $24.749/10.734=2.31$ for lower dependence to $0.344/0.078=4.41$ for only minor powers. No row has a ratio near ten. Six of the nine ratios of variances exceed ten, but the sentence and the table concern standard errors.", "what_survives": "Table 1 is unchanged. Every dyadic standard error exceeds the two other standard errors in its row. The coefficient for international organizations still fails the threshold $1.64$: its ratio to the printed standard error is $0.011/0.008=1.375$, and at most $23/15$ when I allow for rounding. Only the stated size of the increase is weakened."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E012", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of seven lines from the first paragraphs of page 565. Red boxes mark the phrases 'sizes that are far too small' and 'true distribution of the parameters of interest.' in the sentence 'Failure to account for dyadic clustering may result in significance tests or confidence intervals with sizes that are far too small relative to the true distribution of the parameters of interest.'", "caption": "Section 1, first paragraph, p. 565 (PDF p. 2). The marks are on “sizes that are far too small” and on “true distribution of the parameters of interest”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E013.png"}, {"alt": "Crop of six lines from Section 5 on page 571. A red box marks 't > 1.64' in the sentence 'But when one takes into account dyadic clustering, the coefficient for international organizations would fail to pass a conventional significance test (e.g., t > 1.64).'", "caption": "Section 5, second paragraph, p. 571 (PDF p. 8). The mark is on “$t>1.64$”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E013-b.png"}], "formal_release": null, "formal_statements": [], "headline": "“Sizes that are far too small” means tests that reject too often and confidence intervals that are too narrow, and “$t>1.64$” means that the absolute value of the $t$-ratio is compared with 1.64.", "id": "P018-E013", "location": "Section 1, first paragraph, p. 565 (PDF p. 2); Section 5, second paragraph, p. 571 (PDF p. 8)", "pdf_page": 2, "printed_page": "565", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~565: ``Failure to account for dyadic clustering may result in significance tests or confidence intervals with \\hl{sizes that are far too small} relative to the \\hl{true distribution of the parameters of interest}.''\n\np.~571: ``But when one takes into account dyadic clustering, the coefficient for international organizations would fail to pass a conventional significance test (e.g., \\hl{$t>1.64$}).''", "corrected": "p.~565: ``Failure to account for dyadic clustering may result in \\hl{significance tests that overreject and confidence intervals that are too narrow} relative to the \\hl{sampling distribution of the coefficient estimates}.''\n\np.~571: ``\\ldots would fail to pass a conventional significance test (e.g., \\hl{$|t|>1.64$ for an approximately 10\\% two-sided normal test}).''", "intended_meaning": "The sentence on p.~565 says that the uncertainty of the estimates is understated when dyadic clustering is ignored. Standard errors that are too small make a test reject a true null hypothesis too often, so that its size is too large, and they make confidence intervals too narrow. The distribution meant is the sampling distribution of the coefficient estimates, and the Conclusion (p.~572) describes the same failure as ``seriously anti-conservative''. On p.~571 the coefficient is negative, and the comparison intended is between the absolute value of the $t$-ratio and 1.64. Table 1 reports $-0.011$ and $0.008$ for international organizations, and their ratio is $-11/8$. Allowing for the rounding of both entries, the absolute ratio lies between $21/17$ and $23/15$, below 1.64, and the conclusion about this coefficient stands."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E013", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the foot of p. 568. The last line of the text reads: compute the dyadic cluster robust estimator using readily available robust standard error commands, with footnote mark 4. Footnote 4 reads: Code for implementation in R is available from the replication archive posted to the Dataverse page references in the acknowledgments note. A red outline surrounds the word references.", "caption": "Footnote 4, p. 568 (PDF p. 5). The outline marks the word “references”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E014.png"}, {"alt": "Crop of seven lines of text. They read: Figure 2 and 3 show additional simulation results where we make comparisons to mixed effects models, specifically, to a two-way random effects model that incorporates random effects for each of N units in the data. All random effects models were fit using the lmer package in R. In both sets of graphs, black box plots show the distribution of standard error estimates for OLS coefficients using the proposed dyadic cluster robust estimator. Gray box plots show standard error estimates for OLS coefficients from a heteroskedasticity robust estimator. Blue box plots show the distribution of standard error estimators from a two-way random effects model with a random. Red outlines surround the words lmer package and the words Gray box plots.", "caption": "Supporting Information, p. 2, first paragraph. The outlines mark “lmer package” and “Gray box plots”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E014-b.png"}, {"alt": "Crop of a plot titled s.e. beta-hat 0 (cross section), with vertical axis Standard error and labelled ticks at 0.0, 0.4 and 0.8 and horizontal axis positions 20, 50, 100 and 150. The legend lists a black diamond, True OLS s.e.; a blue diamond, True RE s.e.; a black box, Dyadic-cluster robust s.e. est.; a red box, Naive cluster robust s.e. est.; and a blue box, RE s.e. est. At each of the four positions the plot shows a black box, a blue box and, lowest, a red box, with black and blue diamonds marking the true standard errors. A red outline surrounds the legend text Naive cluster robust s.e. est.", "caption": "Supporting Information, p. 2, Figure 2, left panel. The outline marks the legend entry “Naive cluster robust s.e. est.”, whose boxes are drawn in red.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E014-c.png"}, {"alt": "Crop of five lines of text. They read: spatial error lag approach. The appropriate random effects model would be a three-way random effects model to account for each state in a dyad as well as the dyad as a whole over time. We attempted to fit three-way random effects logit models using the lmer package in R and the melogit function in Stata. We posted the jobs to a university cluster with 24 Intel Xeon cores and 48 gigabytes of RAM per node. In neither case did the random effects models converge. In fact. A red outline surrounds the words lmer package.", "caption": "p. 571 (PDF p. 8). The outline marks “lmer package”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E014-d.png"}], "formal_release": null, "formal_statements": [], "headline": "The comparison boxes in the figures of the Supporting Information are red and show the HC2 estimator, lmer is a function of the lme4 package, and footnote 4 should read “referenced”.", "id": "P018-E014", "location": "Footnote 4, p. 568 (PDF p. 5); p. 571 (PDF p. 8); Supporting Information pp. 2 and 4, and the legends of its Figures 2 and 3", "pdf_page": 5, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item Footnote~4, p.~568: ``Code for implementation in R is available from the replication archive posted to the Dataverse page \\hl{references} in the acknowledgments note.''\n\\item p.~571: ``We attempted to fit three-way random effects logit models using the \\hl{\\texttt{lmer} package} in R and the \\texttt{melogit} function in Stata.''\n\\item Supporting Information, p.~2: ``All random effects models were fit using the \\hl{\\texttt{lmer} package} in R. \\ldots\\ \\hl{Gray} box plots show standard error estimates for OLS coefficients from a heteroskedasticity robust estimator.''\n\\item Supporting Information, Figures~2 and~3, legend entry of the red boxes: ``\\hl{Naive cluster robust} s.e. est.''\n\\item Supporting Information, p.~4: ``But the random effects standard error estimator (as computed in the \\hl{\\texttt{lmer} package} for R) is inconsistent and anti-conservative: the blue box plots converge to a limit that is substantially below the true standard error (the blue diamonds).''\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Footnote~4: ``\\ldots\\ posted to the Dataverse page \\hl{referenced} in the acknowledgments note.''\n\\item p.~571 and Supporting Information, pp.~2 and~4: ``the \\hl{\\texttt{lmer} function in the \\texttt{lme4} package}''.\n\\item Supporting Information, p.~2: ``\\hl{Red} box plots show standard error estimates for OLS coefficients from \\hl{the HC2} heteroskedasticity robust estimator.''\n\\item Supporting Information, Figures~2 and~3, legend: ``\\hl{HC2 heteroskedasticity-robust} s.e. est.''\n\\item Supporting Information, p.~4, for clarity: ``the random effects standard error estimates remain substantially below the true standard errors (the blue diamonds), in relative terms, as $N$ increases.''\n\\end{itemize}", "evidence": "I checked the pages and the replication scripts. The scripts compute the comparison standard errors with the HC2 option and draw them in red under the legend ``Naive cluster robust s.e. est.''", "problem": "In Figures~2 and~3 of the Supporting Information, the boxes of the heteroskedasticity robust estimator are drawn in red, and the text calls them gray. The replication scripts compute these standard errors with the HC2 estimator, and the legends name it ``Naive cluster robust''. In a cross-section each dyad has one observation, and the legend can be read as clustering on single dyads. It is not the name that the text uses. The fitting function \\texttt{lmer} belongs to the \\texttt{lme4} package. Footnote~4 prints ``references'' for ``referenced''.", "what_survives": "No simulation draw, plotted value or result changes. I have not established that a different estimator was computed, and I find no error of substance in the last sentence of p.~4 of the Supporting Information, for which I give a plainer wording below."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E014", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of two reference entries. Red boxes mark the given name Nathanial in the Beck entry, the word ecometrics in its title, and the run-together name BujaAndreas at the start of the next entry.", "caption": "References, p. 576 (PDF p. 13), entries Beck et al. (2006) and Buja et al. (2014). Marked: “Nathanial”, “ecometrics” and “BujaAndreas”. The second and third are misprints; the first awaits a check.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E015.png"}, {"alt": "Crop of the Hubbard reference entry with one line before and after it. A red box marks the surname Fliescher of the third author.", "caption": "References, p. 577 (PDF p. 14), entry Hubbard et al. (2010). The marked surname “Fliescher” awaits a check against the cited work.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E015-b.png"}, {"alt": "Crop of the last entries under White with one line before and after. A red box marks the title Asymptotic theory of econometricians in the 1984 entry.", "caption": "References, p. 577 (PDF p. 14), entry White (1984). The marked title “Asymptotic theory of econometricians” awaits a check against the cited work.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-E015-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The reference list has two misprints, “BujaAndreas” for “Buja, Andreas” and “ecometrics” for “econometrics”; three other printed forms await a check against the cited works.", "id": "P018-E015", "location": "References, pp. 576–577 (PDF pp. 13–14): entries Beck et al. (2006), Buja et al. (2014), Hubbard et al. (2010) and White (1984)", "pdf_page": 13, "printed_page": "576–577", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~576: Beck, Nathanial, Kristian Skrede Gleditsch, and Kyle Beardsley. 2006. Space is more than geography: Using spatial \\hl{ecometrics} in the study of political economy. [\\ldots] \\hl{BujaAndreas}, Richard Berk, Lawrence Brown, Edward George, Emil Pitkin, Mikhail Traskin, Linda Zhao, and Kai Zhang. 2014.\n\nP.~577: Hubbard, Alan E., Jennifer Ahern, Nancy L. Fliescher, Mark Van Der Laan, [\\ldots] ---------. 1984. \\textit{Asymptotic theory of econometricians}. New York: Academic Press.", "corrected": "P.~576: Beck, Nathanial, Kristian Skrede Gleditsch, and Kyle Beardsley. 2006. Space is more than geography: Using spatial \\hl{econometrics} in the study of political economy. [\\ldots] \\hl{Buja, Andreas}, Richard Berk, Lawrence Brown, Edward George, Emil Pitkin, Mikhail Traskin, Linda Zhao, and Kai Zhang. 2014.\n\nThe forms ``Nathanial'', ``Fliescher'' and ``Asymptotic theory of econometricians'' are to be replaced by ``Nathaniel'', ``Fleischer'' and ``Asymptotic theory for econometricians'' only after a check against the cited works.", "evidence": "I checked each complete entry on pp.~576--577 against the rest of the list.", "problem": "Every other entry separates the first author's surname and given name by a comma and a space, and the Buja entry runs them together. The word ``ecometrics'' in the title of Beck et al.\\ stands for ``econometrics''. Three further forms are printed as transcribed: the given name ``Nathanial'', the surname ``Fliescher'' and the title ``Asymptotic theory of econometricians''. The readings ``Nathaniel'', ``Fleischer'' and ``Asymptotic theory for econometricians'' are plausible, but they have not been checked against the cited works.", "what_survives": "The four cited works remain identifiable, and no result is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-E015", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of equation (2) with the word 'with' above it and the definition of the set S(d) below it. In the second term, labelled B, the summation sign has the subscript 'd-prime in S(d)' followed by an upright E, which a red box marks. No E stands before the opening brace of the summand.", "caption": "Equation (2), p. 566 (PDF p. 3). The mark is on the letter E, printed at the end of the summation subscript of term $B$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N001.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T21", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T21"}], "headline": "In equation (2) the expectation operator of term $B$ is set inside the summation subscript; it belongs in front of the braces, as in equation (6).", "id": "P018-N001", "location": "Lemma 1, equation (2), term $B$, p. 566 (PDF p. 3)", "pdf_page": 3, "printed_page": "566", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T21"], "text": {"as_printed": "\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big]=\\sum_{d=1}^{D}\\Big(\\underbrace{\\mathrm{E}\\{X_dX_d'\\operatorname{Var}[\\epsilon_d|\\mathbf{X}]\\}}_{A}+\\underbrace{\\sum_{d'\\in\\mathcal{S}(d)\\,\\hl{\\scriptstyle\\mathrm{E}}}\\{X_dX_{d'}'\\operatorname{Cov}[\\epsilon_d,\\epsilon_{d'}|\\mathbf{X}]\\}}_{B}\\Big),\\qquad(2)\n\\]", "corrected": "\\[\n\\operatorname{Var}\\Big[\\sum_{d=1}^{D}X_d\\epsilon_d\\Big]=\\sum_{d=1}^{D}\\Big(\\underbrace{\\mathrm{E}\\{X_dX_d'\\operatorname{Var}[\\epsilon_d|\\mathbf{X}]\\}}_{A}+\\underbrace{\\sum_{d'\\in\\mathcal{S}(d)}\\hl{\\mathrm{E}}\\{X_dX_{d'}'\\operatorname{Cov}[\\epsilon_d,\\epsilon_{d'}|\\mathbf{X}]\\}}_{B}\\Big),\\qquad(2)\n\\]\nThis display corrects the position of E only. With the correction of \\fid{P018-E001}, the conditional variance and covariance are also replaced by $\\epsilon_d^2$ and $\\epsilon_d\\epsilon_{d'}$.", "evidence": "The evidence is a check of the page. Equation (6) in Appendix A (p.~573) prints the same term with E in front of the braces.", "problem": "The letter E is printed on the line of the summation subscript, after $d'\\in\\mathcal{S}(d)$, and no expectation stands before the braces. Term $B$ is the sum, over the dyads in $\\mathcal{S}(d)$, of the expectations of the cross terms.", "what_survives": "Nothing depends on the misprint. The content of equation (2) is a separate matter, treated in \\fid{P018-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N001", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Proposition 1 with equation (3), the sentence 'Under the conditions of Lemma 1 and if X_d and e_d have bounded support with nonzero second moments and cross-moments, then as N goes to infinity', and the display 'N V-hat minus V converges in probability to 0'. A red box marks the first X_d, whose subscript d is upright while the other subscripts are italic.", "caption": "Proposition 1, p. 567 (PDF p. 4). The mark is on the first $X_d$ of the sentence, whose subscript is upright.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N002.png"}, {"alt": "Crop of the start of the proof of Lemma 1: a display for root-N times beta-hat minus beta, the sentence 'By continuity of the inverse, the finite support of X_d and e_d, X full rank, and the fact that the sum of dependent terms in this sum is O(D to the power 3/2) (Lehmann 1999, Equation (2.8.4)), we have that', and a second display. A red box marks 'finite support'.", "caption": "Appendix A, proof of Lemma 1, p. 573 (PDF p. 10). The mark is on “finite support”.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In Appendix A, “the finite support of $X_d$ and $e_d$” means the bounded support assumed in Proposition 1, and the first $X_d$ of the proposition has an upright subscript.", "id": "P018-N002", "location": "Proposition 1, p. 567 (PDF p. 4); Appendix A, proof of Lemma 1, p. 573 (PDF p. 10)", "pdf_page": 4, "printed_page": "567", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~567: ``Under the conditions of Lemma 1 and if $X_{\\hl{\\mathrm{d}}}$ and $e_d$ have bounded support with nonzero second moments and cross-moments, then as $N\\to\\infty$''\n\np.~573: ``By continuity of the inverse, the \\hl{finite support} of $X_d$ and $e_d$, $\\mathbf{X}$ full rank, and the fact that the sum of dependent terms in this sum is $O(D^{3/2})$ (Lehmann 1999, Equation (2.8.4)), we have that''", "corrected": "p.~567: ``Under the conditions of Lemma 1 and if $X_{\\hl{d}}$ and $e_d$ have bounded support with nonzero second moments and cross-moments, then as $N\\to\\infty$''\n\np.~573: ``By continuity of the inverse, the \\hl{bounded support} of $X_d$ and $e_d$, $\\mathbf{X}$ full rank, \\ldots''", "intended_meaning": "Proposition 1 assumes that the regressors $X_d$ and the fitted residuals $e_d=Y_d-X_d'\\hat\\beta$ have bounded support. A fitted residual is a random variable, and a condition on its support is meaningful. Bounded $X_d$ and $Y_d$ bound the population residual $\\epsilon_d$, but they do not by themselves bound $e_d$ uniformly in $N$. The corrected proof in \\fid{P018-E006} establishes the conclusion of Proposition 1 without this condition. Appendix A recalls this hypothesis, and ``finite support'' there refers to support bounded in a finite interval, as on p.~566, and not to a distribution with finitely many values. The upright $\\mathrm{d}$ is the index $d$. Nothing depends on either point."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N002", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of ten lines of text and one display from p. 567. The first lines read: Under the conditions of Lemma 1 and if X d and e d have bounded support with nonzero second moments and cross-moments, then as N tends to infinity, N V-hat minus V converges in probability to 0. The paragraph below reads: The proposition indicates that V-hat provides a consistent estimator to characterize the true asymptotic sampling distribution of the regression coefficients, beta-hat. Standard error estimates for beta-hat are obtained from the square roots of the diagonals of V-hat. The assumption of bounded support for X d and e d merely rules out situations that are unlikely to arise in real-world data anyway. The first three lines of the next paragraph, on repeated dyad observations, follow. A red outline in two parts surrounds the sentence that begins Standard error estimates.", "caption": "Text after Proposition 1, p. 567 (PDF p. 4). The outline marks the sentence that tells how standard errors are obtained from $\\widehat V$.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N003.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T17", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T17"}, {"id": "P018-T18", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T18"}], "headline": "The variance estimator $\\widehat V$ of equation (3) need not be positive semi-definite in a finite sample, and a standard error is the square root of a diagonal entry only when that entry is nonnegative.", "id": "P018-N003", "location": "Sentence after Proposition 1 and equation (3), p. 567 (PDF p. 4)", "pdf_page": 4, "printed_page": "567", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T17", "P018-T18"], "text": {"as_printed": "Standard error estimates for $\\widehat\\beta$ are obtained from \\hl{the square roots of the diagonals of $\\widehat V$}.", "corrected": "Standard error estimates for $\\widehat\\beta$ are obtained from the square roots of the diagonals of $\\widehat V$, \\hl{for each diagonal entry that is nonnegative. The estimator $\\widehat V$ need not be positive semi-definite in a finite sample, and a negative diagonal entry has no real-valued standard error. Summaries of simulations should report the number of draws with an undefined standard error and how those draws were treated.}\n\nIf a value is wanted in every sample, the convention $\\widetilde V_{jj} = \\max(\\widehat V_{jj}, 0)$ may be stated separately. This convention changes the estimator in finite samples. Since $|\\max(a, 0) - \\max(b, 0)| \\le |a - b|$ and $V_{jj} \\ge 0$, it keeps the convergence of $N\\widetilde V_{jj} - V_{jj}$ to zero in probability. It does not guarantee coverage or a positive semi-definite matrix.", "evidence": "The evidence is an exact counterexample in rational arithmetic. Let $N = 4$, with one observation per dyad and $X_d' = (1, z_d)$, and consider the following data.\n\\begin{audittable}{lrrr}{An example with $N = 4$: the regressor $z_d$, the outcome $Y_d$ and the OLS residual $e_d$ of each dyad.}\nDyad & $z_d$ & $Y_d$ & $e_d$ \\\\\n$\\{1,2\\}$ & $1$ & $4$ & $3/5$ \\\\\n$\\{1,3\\}$ & $2$ & $3$ & $-4/5$ \\\\\n$\\{1,4\\}$ & $3$ & $5$ & $4/5$ \\\\\n$\\{2,3\\}$ & $1$ & $3$ & $-2/5$ \\\\\n$\\{2,4\\}$ & $2$ & $3$ & $-4/5$ \\\\\n$\\{3,4\\}$ & $1$ & $4$ & $3/5$ \\\\\n\\end{audittable}\nBy a direct computation, I have\n\\[\n\\mathbf{X}'\\mathbf{X} = \\begin{pmatrix} 6 & 10 \\\\ 10 & 20 \\end{pmatrix}, \\qquad \\widehat\\beta = \\begin{pmatrix} 3 \\\\ 2/5 \\end{pmatrix}, \\qquad \\mathbf{X}'e = \\begin{pmatrix} 0 \\\\ 0 \\end{pmatrix},\n\\]\nand the residuals in the last column of the table. Summing $X_d X_{d'}' e_d e_{d'}$ over the pairs of dyads that share a member, as in equation~(3), I obtain\n\\[\nS'WS = \\begin{pmatrix} -34/25 & -2 \\\\ -2 & -98/25 \\end{pmatrix}, \\qquad \\widehat V = \\begin{pmatrix} -17/50 & 21/125 \\\\ 21/125 & -58/625 \\end{pmatrix}.\n\\]\nBoth diagonal entries of $\\widehat V$ are negative. The variance function in the replication files returns the same slope entry, $-0.0928$.\n\nThe conditions of Proposition~1 do not exclude these data. Let the characteristics $(T_i, A_i)$ of the units be independent, with $T_i$ uniform on $\\{0, 1, 2, 3\\}$ and $A_i$ uniform on $\\{-1, 1\\}$. Set $z_{ij} = |T_i - T_j|$ and $Y_{ij} = z_{ij} + H_{T_i T_j} + A_i + A_j$, where the rows and columns of\n\\[\nH = \\begin{pmatrix} 0 & 1 & -1 & 0 \\\\ 1 & 0 & 0 & -1 \\\\ -1 & 0 & 0 & 1 \\\\ 0 & -1 & 1 & 0 \\end{pmatrix}\n\\]\nare indexed by $0, 1, 2, 3$. The data are bounded, with $z_{ij} \\in [0, 3]$ and $Y_{ij} \\in [-2, 5]$. The matrix $\\operatorname{E}[X_d X_d']$ has determinant $15/16$, and the covariance matrix of the limiting distribution of $\\sqrt{N}(\\widehat\\beta - \\beta)$ is positive definite, with determinant $67072/50625$. The event $T = (0, 1, 2, 3)$, $A = (1, 1, 1, 1)$ has probability $1/4096$ and produces the table above. The event $T = (3, 2, 1, 0)$, $A = (1, 1, 1, 1)$ produces the same table, and the probability of the table is $1/2048$.\n\nA second example lies in the simulation design of Section~4. Let $N = 20$. Let the unit covariates be $X_i = i$ for $i \\le 4$ and $X_i = 5/2$ for $i \\ge 5$, and let $\\alpha_i = 0$ for all $i$. Let $\\nu_{d(1,2)} = \\nu_{d(3,4)} = 1$ and $\\nu_{d(1,3)} = \\nu_{d(2,4)} = -1$, with $\\nu_d = 0$ for the other dyads. Then $\\mathbf{X}'\\mathbf{X}$ has determinant $13524$, and the slope entry of $\\widehat V$ is\n\\[\n\\widehat V_{22} = -\\frac{31032915283}{43557147269307} \\approx -0.000712.\n\\]\nThe estimator is continuous near this point and the normal density is positive there. A negative slope entry therefore has positive probability in the published design with $N = 20$. The example does not show how often this occurs.", "problem": "The sentence carries no restriction, and the paper does not say that a diagonal entry of $\\widehat V$ can be negative. Let $S$ be the $D \\times k$ matrix whose row $d$ is $e_d X_d'$, and let $W$ be the $D \\times D$ matrix such that $W_{dd'} = 1$ if the dyads $d$ and $d'$ share a member (including $d' = d$) and $W_{dd'} = 0$ otherwise. Then equation~(3) reads\n\\[\n\\widehat V = (\\mathbf{X}'\\mathbf{X})^{-1} S' W S\\, (\\mathbf{X}'\\mathbf{X})^{-1}.\n\\]\nThe matrix $W$ is not positive semi-definite. For $N = 4$ its eigenvalues are $5$, $1$, $1$, $1$, $-1$, $-1$. A diagonal entry of $\\widehat V$ can be negative, and the corresponding standard error is then not a real number. In the replication files, the simulation scripts compute \\texttt{sqrt(diag(Vhat))}, which returns \\texttt{NaN} with a warning in such a draw. A box plot of the saved standard errors then leaves the draw out without notice.", "what_survives": "This finding does not affect equation~(3), Proposition~1 or Proposition~2. Suppose that $N\\widehat V_{jj}$ converges in probability to a limit $V_{\\infty,jj} > 0$. Then\n\\[\n\\Pr\\bigl(\\widehat V_{jj} < 0\\bigr) \\le \\Pr\\bigl(|N\\widehat V_{jj} - V_{\\infty,jj}| \\ge V_{\\infty,jj}\\bigr) \\to 0,\n\\]\nand the probability of a negative entry vanishes as $N$ grows. No value in a published figure or table is shown to be wrong. The replication files do not contain the simulation output, and I cannot determine whether any draw behind Figure~1, or behind Figures~2 and~3 of the Supporting Information, had an undefined standard error."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N003", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Corollary 1. The statement reads: For the repeated dyads case, assume the same conditions as in Proposition 1 and consider the following variance estimator, followed by a display that defines V-hat r as a sandwich with a sum over dyads d and over the dyads d-prime in S of d. Then, as N tends to infinity, N Var of beta-hat r minus beta r, minus V r, converges in probability to 0, and N V-hat r minus V r converges in probability to 0. A red outline surrounds the first of the two displays, N Var of beta-hat r minus beta r, minus V r.", "caption": "Corollary 1, p. 567 (PDF p. 4). The outline marks the first conclusion, which concerns the variance of $\\widehat\\beta_r$ in a finite sample.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N004.png"}, {"alt": "Crop of Corollary 2. The statement reads: For the weighted dyads case, assume the same conditions as in Proposition 1 and consider the following variance estimator, followed by display (7), which defines V-hat w as a weighted sandwich. Then, as N tends to infinity, N Var of beta-hat w minus beta w, minus V w, converges in probability to 0, and N V-hat w minus V w converges in probability to 0. A red outline surrounds the first of the two displays, N Var of beta-hat w minus beta w, minus V w.", "caption": "Corollary 2, Appendix D, p. 575 (PDF p. 12). The outline marks the first conclusion, which has the same form as in Corollary 1.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T19", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T19"}, {"id": "P018-T20", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T20"}], "headline": "The first conclusion of Corollaries 1 and 2 should describe $V_r$ and $V_w$ as covariance matrices of limiting distributions, because the variance of the estimator in a finite sample need not exist under the stated conditions.", "id": "P018-N004", "location": "Corollary 1, first display after “Then, as $N \\to \\infty$”, p. 567 (PDF p. 4); the same display of Corollary 2, p. 575 (PDF p. 12)", "pdf_page": 4, "printed_page": "567", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T19", "P018-T20"], "text": {"as_printed": "Corollary~1: \\ldots\\ Then, as $N \\to \\infty$,\n\\[\n\\hl{N\\operatorname{Var}[\\widehat\\beta_r - \\beta_r]} - V_r \\overset{p}{\\to} 0,\n\\]\nand\n\\[\nN\\widehat V_r - V_r \\overset{p}{\\to} 0,\n\\]\nCorollary~2: \\ldots\\ Then, as $N \\to \\infty$,\n\\[\n\\hl{N\\operatorname{Var}[\\widehat\\beta_w - \\beta_w]} - V_w \\overset{p}{\\to} 0,\n\\]\nand\n\\[\nN\\widehat V_w - V_w \\overset{p}{\\to} 0,\n\\]", "corrected": "Apply the \\hl{repetition and weight regularity conditions} in \\fid{P018-E003}, and \\hl{suppose that the corrected targets} in \\fid{P018-E004} \\hl{converge to $V_{r,\\infty}$ and $V_{w,\\infty}$, respectively}.\n\nCorollary~1. For the repeated dyads case, assume the same conditions as in Proposition~1 and consider the following variance estimator:\n\\[\n\\widehat V_r = (\\mathbf{X}'\\mathbf{X})^{-1} \\sum_{d=1}^{D} \\Bigl( X_d' e_d e_d' X_d + \\sum_{d' \\in \\mathcal{S}(d)} X_d' e_d e_{d'}' X_{d'} \\Bigr) (\\mathbf{X}'\\mathbf{X})^{-1}.\n\\]\nThen, as $N \\to \\infty$, \\hl{the limiting distribution of $\\sqrt{N}(\\widehat\\beta_r - \\beta_r)$ has mean zero and covariance matrix $V_{r,\\infty}$}, and\n\\[\nN\\widehat V_r - V_r \\overset{p}{\\to} 0.\n\\]\n\\hl{If, in addition, $N\\lVert\\widehat\\beta_r - \\beta_r\\rVert^2$ is uniformly integrable, then $N\\operatorname{Var}[\\widehat\\beta_r - \\beta_r] - V_r \\to 0$.}\n\nCorollary~2. For the weighted dyads case, assume the same conditions as in Proposition~1 and consider the following variance estimator:\n\\[\n\\widehat V_w = (\\mathbf{X}'\\mathbf{W}\\mathbf{X})^{-1} \\sum_{d=1}^{D} \\Bigl( X_d X_d' w_d^2 e_d^2 + \\sum_{d' \\in \\mathcal{S}(d)} X_d X_{d'}' w_d w_{d'} e_d e_{d'} \\Bigr) (\\mathbf{X}'\\mathbf{W}\\mathbf{X})^{-1}.\n\\]\nThen, as $N \\to \\infty$, \\hl{the limiting distribution of $\\sqrt{N}(\\widehat\\beta_w - \\beta_w)$ has mean zero and covariance matrix $V_{w,\\infty}$}, and\n\\[\nN\\widehat V_w - V_w \\overset{p}{\\to} 0.\n\\]\n\\hl{If, in addition, $N\\lVert\\widehat\\beta_w - \\beta_w\\rVert^2$ is uniformly integrable, then $N\\operatorname{Var}[\\widehat\\beta_w - \\beta_w] - V_w \\to 0$.}\n\nHere $V_r$ and $V_w$ are the matrices given in \\fid{P018-E004}.", "evidence": "The evidence is an exact counterexample with a derivation. Let the units be independent. For each unit $i$, let $M_i$ take the value $m \\ge 1$ with probability $2^{-m}$, let $C_i$ be uniform on $\\{1, \\dots, m\\}$ given $M_i = m$, and let $A_i$ and $B_i$ be independent signs, each equal to $\\pm 1$ with probability $1/2$. Define $\\delta_m = 1/(m^2 2^{m^2 + 2})$. For a dyad $\\{i, j\\}$ such that $M_i = M_j = m$ and $C_i \\ne C_j$, set $s_{ij} = \\delta_m$ and $a_{ij} = A_i + A_j$. For every other dyad, set $s_{ij} = 1$ and $a_{ij} = 0$. Each dyad has two observations, so that $T(d) = 2$, given by\n\\[\nX_{ij,\\pm} = (1, 1 \\pm s_{ij})', \\qquad Y_{ij,\\pm} = B_i + B_j \\pm a_{ij}.\n\\]\nNote that the regressors lie in $[0, 2]$ and the outcomes in $[-4, 4]$. The sample residuals are at most $8$ in absolute value for all $N$ and all samples. Every sample design has full rank, and the population matrix $\\operatorname{E}[X X']$ has determinant $q = \\operatorname{E}[s_{12}^2] \\ge 2/3$. The population coefficient is $\\beta_r = 0$, and the OLS slope is\n\\[\n\\widehat b = \\frac{\\sum_{i<j} s_{ij} a_{ij}}{\\sum_{i<j} s_{ij}^2}.\n\\]\nFix $N \\ge 3$ and $m \\ge N(N-1)$, and let $E_m$ be the event that $M_i = m$ for all $i$ and that the $C_i$ are distinct. On $E_m$ every $s_{ij}$ equals $\\delta_m$, and $\\widehat b = 2\\bar A/\\delta_m$, where $\\bar A$ is the mean of the $A_i$. This implies that $\\operatorname{E}[\\widehat b^2 \\mid E_m] = 4/(N\\delta_m^2)$. Since $\\Pr(E_m) \\ge 2^{-mN-1}$, I obtain\n\\[\n\\operatorname{E}\\bigl[\\widehat b^2 \\mathbf{1}_{E_m}\\bigr] \\ge \\frac{32}{N}\\, m^4\\, 2^{2m^2 - mN} \\ge 2^{m^2}.\n\\]\nIn the last inequality, I used that $m \\ge N$. The right side is unbounded in $m$. I conclude that $\\operatorname{E}[\\widehat b^2] = \\infty$ for all $N \\ge 3$, and $\\operatorname{Var}[\\widehat\\beta_r - \\beta_r]$ is not finite. By a parallel computation, the slope entry of $\\widehat V_r$ has infinite expectation for all $N \\ge 4$. The two observations of a dyad may be read as its two directions with unit weights, and the model is then an example for Corollary~2.\n\nIn this model $\\sqrt{N}(\\widehat\\beta_r - \\beta_r)$ converges in distribution to a normal vector with a finite, positive definite covariance matrix, and $N\\widehat V_r$ converges to that matrix in probability. The second conclusion of each corollary is not contradicted. For the last sentence of the corrected statement, recall that convergence in distribution together with uniform integrability of $N\\lVert\\widehat\\beta - \\beta\\rVert^2$ implies convergence of the first and second moments.", "problem": "The first display of each corollary is a statement about the variance of the estimator in a sample of $N$ units. Under the conditions of the corollaries this variance need not be finite. The estimator contains the inverse of $\\mathbf{X}'\\mathbf{X}$, and bounded data with a design of full rank do not bound the moments of that inverse. Convergence in distribution of $\\sqrt{N}(\\widehat\\beta - \\beta)$ and convergence in probability of $N\\widehat V$ do not imply that second moments converge. The limiting distribution can have a finite covariance matrix while the estimator has no finite variance at any $N$. The proof in Appendix~B argues through the mean and the variance of $N\\widehat V$, and these need not be finite either; see \\fid{P018-E006}.", "what_survives": "The second conclusion of each corollary, the estimators $\\widehat V_r$ and $\\widehat V_w$, Proposition~2 and every reported number are not affected by this finding. Inference in the paper rests on the limiting distribution and on the convergence of $N\\widehat V$ in probability. The statement about the variance of the estimator at finite $N$ is lost unless a moment condition is added. The example does not satisfy such a condition, and I have not established one for the applications."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N004", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of fifteen lines of text from p. 568: the paragraph that begins 'Some remarks are in order when it comes to the repeated observations case' and the first two lines of the next paragraph. A red outline in two parts surrounds 'such as serial correlation of fixed order or, more generally, strong mixing over time' in the sentence 'Under fixed N and growing T(d) further assumptions would be required for consistency, such as serial correlation of fixed order or, more generally, strong mixing over time.' A second red outline surrounds '(Hansen 2007; Stock and Watson 2008)' at the end of the sentence 'With fixed N and growing T(d), however, additional assumptions are needed for consistency (Hansen 2007; Stock and Watson 2008).'", "caption": "First paragraph after the display that defines $V_r$, p. 568 (PDF p. 5). The marks are on the assumptions named in the fourth sentence and on the two citations that end the paragraph.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "With $N$ fixed and $T(d)$ growing, serial correlation of fixed order or strong mixing over time can support consistency of $\\widehat\\beta_r$, but such assumptions do not in general make the variance estimator $\\widehat V_r$ consistent at the scale needed for inference, and Hansen (2007) and Stock and Watson (2008) do not show that they do.", "id": "P018-N005", "location": "First paragraph after the display that defines $V_r$, fourth and last sentences, p. 568 (PDF p. 5)", "pdf_page": 5, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``First, our analysis of the repeated observations case takes the $T(d)$ values to be fixed and finite and demonstrates consistency as $N$ grows. [\\ldots] Under fixed $N$ and growing $T(d)$ further assumptions would be required for consistency, \\hl{such as serial correlation of fixed order or, more generally, strong mixing over time}. [\\ldots] With fixed $N$ and growing $T(d)$, however, additional assumptions are needed for consistency \\hl{(Hansen 2007; Stock and Watson 2008)}.''", "corrected": "The fourth sentence becomes: ``Under fixed $N$ and growing $T(d)$\\hl{, the consistency result above does not apply. Assumptions} such as serial correlation of fixed order or, more generally, strong mixing over time \\hl{can support consistency of $\\widehat\\beta_r$ under suitable conditions, but they do not in general make $\\widehat V_r$ consistent at the scale needed for inference}.''\n\nThe last sentence becomes: ``With fixed $N$ and growing $T(d)$, however, \\hl{inference with a fixed number of units, or a consistent alternative variance estimator, requires a separate analysis. Hansen (2007) and Stock and Watson (2008) give such analyses for panel data, and their results do not establish the corresponding statements for dyadic clustering}.''", "evidence": "The evidence is a check of the two cited papers and an exact counterexample. \\href{https://personal.utdallas.edu/~d.sul/Econo2/Hansen-2007-JoE-PanelHAC.pdf}{Hansen (2007)} studies the clustered covariance estimator $\\widehat W$, which contains the products of scores for all pairs of periods within a unit. His Theorem~4 assumes strong mixing, moment conditions, exogeneity and independence across units, and it gives a limit in distribution for $\\widehat W$ as $T\\to\\infty$ with $n$ fixed. Remark~3.7 (p.~607) reads: ``Theorem 4 verifies that $\\widehat W$ is not consistent but does have a limiting distribution as $T\\to\\infty$ with $n$ fixed.'' Corollary~4.1 assumes in addition that the matrices $Q_i$ and $W_i$ of his notation are the same for all units. Under the null hypothesis, it gives the limit $\\sqrt{n/(n-1)}\\,t_{n-1}$ for the $t$-statistic and the limit $\\{nq/(n-q)\\}F_{q,n-q}$ for the Wald statistic with $q<n$ restrictions. \\href{https://www.princeton.edu/~mwatson/papers/Stock_Watson_Ecta_2008.pdf}{Stock and Watson (2008)} show that the usual heteroskedasticity-robust estimator can be inconsistent in a regression with fixed effects when $T>2$ is fixed and $n$ grows. Their bias-adjusted estimator, equation (6), is consistent under their Assumptions 1--5 when $n$ or $T$ grows, ``which includes the cases of $n$ fixed or $T$ fixed'' (equation (7), p.~157). Remark~10 (p.~159) says that the clustered covariance estimator ``averages only across entities''. Remark~11 discusses inference with $n$ fixed and $T$ growing, and Remark~12 extends the bias-adjusted estimator to serial correlation of finite order. I read Hansen (2007) in a typeset copy that carries the page numbers of the journal, and Stock and Watson (2008) in the published version.\n\nFor the counterexample, let $N=4$ and observe each of the six dyads in the periods $t=1,\\dots,T$. The regressor is the constant $1$, and $Y_{ij,t}=a_{i,t}+a_{j,t}$, where the $a_{i,t}$ are independent and equal to $1$ or $-1$ with probability $1/2$ each. Then $\\beta_r=0$, the data are bounded, and dyads that share no unit are independent. The errors are independent over time, so that their serial correlation has order zero and they are strongly mixing. Let $A_i=\\sum_ta_{i,t}$, $\\bar A=(A_1+A_2+A_3+A_4)/4$ and $B_i=A_i-\\bar A$. Each unit belongs to three dyads, and I have\n\\[\n\\widehat\\beta_r=\\frac{A_1+A_2+A_3+A_4}{2T},\\qquad\\operatorname{Var}[\\widehat\\beta_r]=\\frac{1}{T}.\n\\]\nThen $\\widehat\\beta_r$ is consistent as $T\\to\\infty$. The fitted residuals of the dyad $\\{i,j\\}$ sum over $t$ to $B_i+B_j$. Substituting these sums into the estimator of Corollary~1, I obtain\n\\[\n\\widehat V_r=\\frac{1}{18T^2}\\sum_{i=1}^{4}B_i^2,\\qquad R_T=\\frac{\\widehat V_r}{\\operatorname{Var}[\\widehat\\beta_r]}=T\\widehat V_r=\\frac{1}{18T}\\sum_{i=1}^{4}B_i^2.\n\\]\nSince $\\mathrm{E}[\\sum_iB_i^2]=3T$, I have $\\mathrm{E}[R_T]=1/6$ for every $T$. By a direct computation, I find that $\\operatorname{Var}[R_T]=(4T-3)/(216T)$. Since $R_T\\ge0$, Markov's inequality implies that $\\Pr(R_T\\le1/2)\\ge2/3$ for every $T$, and $R_T$ does not converge to $1$ in probability. As $T\\to\\infty$, the ratio $R_T$ converges in distribution to $\\chi^2_3/18$. Exact enumeration for $T=1,2,3,4,8$ agrees with these values. The same computation applies to the slope in a regression with dyad fixed effects, for even $T$ and the regressor $x_t=(-1)^t$. In the example, $\\widehat V_r-\\operatorname{Var}[\\widehat\\beta_r]$ converges to zero in probability, because both terms do, although their ratio does not converge to one.\n\nA second example supports the literal statement that further assumptions are needed. Suppose that each $a_{i,t}$ is replaced by a sign $a_i$ that is drawn once and kept in every period. Then $\\widehat\\beta_r=(a_1+a_2+a_3+a_4)/2$ for every $T$, and $\\widehat\\beta_r$ is not consistent.", "problem": "The paragraph follows Corollary~1, which states that $N\\widehat V_r-V_r$ converges to zero in probability as $N\\to\\infty$ with the $T(d)$ fixed and finite. It introduces no other estimator. In this context I read the two marked sentences as saying that serial correlation of fixed order or strong mixing over time would make $\\widehat V_r$ consistent at the scale needed for inference when $N$ is fixed and $T(d)$ grows. This is not so. Under suitable conditions, such assumptions support consistency of $\\widehat\\beta_r$. Corollary~1 multiplies the variance of $\\widehat\\beta_r$ and its estimator by $N$. In the balanced example below, where $T(d)=T$, the relevant estimator is $T\\widehat V_r$, and the named assumptions do not make it consistent. In the example below the errors are independent over time, so that both named assumptions hold. There the ratio of $\\widehat V_r$ to the variance of $\\widehat\\beta_r$ has mean $1/6$ for every number of periods, and its limit is random.\n\nThe two cited papers do not support this reading. Hansen (2007) shows that, for a fixed number of units and a growing number of periods, the clustered covariance estimator is not consistent under strong mixing. It converges in distribution to a random limit; under further homogeneity conditions, inference uses scaled $t$ and $F$ reference distributions. Stock and Watson (2008) treat that case with a different estimator, which is adjusted for bias. They note that the clustered covariance estimator averages only across units. Both papers concern panel data, and neither establishes a result for dyadic clustering, where the clusters overlap. Read literally, the two sentences say only that further assumptions are needed, which is true. The paragraph does not specify whether consistency refers to $\\widehat\\beta_r$, $\\widehat V_r$, or a different variance estimator.", "what_survives": "No result of the paper depends on the two sentences. Corollary~1 concerns $N\\to\\infty$ with the $T(d)$ fixed and finite, and this finding does not bear on it. No subsequent theorem, simulation or application invokes a fixed-$N$, growing-$T(d)$ consistency result. Section~4 varies $N$ with one or two observations per dyad, and Section~5 invokes no result for fixed $N$ and growing $T(d)$. What changes is the suggestion that assumptions on the dependence over time would make $\\widehat V_r$ consistent at the scale needed for inference when $N$ is fixed. I have not shown that no added assumption can do so, and neither cited paper establishes a result for dyadic clustering."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N005", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of equation (4) and six lines of text. Equation (4) reads: Y d(i,j)t equals beta 0 plus beta 1 times the absolute value of X i minus X j, plus alpha i plus alpha j plus nu d(i,j)t, where an underbrace labels the last three terms epsilon d(i,j)t. The text reads: where Y d(i,j)t is the tth observed outcome for the dyad that includes units i and j, X i, X j, alpha i, alpha j, and u d(i,j)t are independent draws from standard normal distributions, and the compound error, epsilon d(i,j)t equals alpha i plus alpha j plus nu d(i,j)t, is unobserved. A red outline surrounds the symbol u d(i,j)t at the start of the second line of text.", "caption": "Section 4, p. 569 (PDF p. 6): equation (4) and the sentence after it. The outline marks $u_{d(i,j)t}$ in the list of independent draws.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N006.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T22", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T22"}], "headline": "The dyad-level shock in the list of independent draws is $\\nu_{d(i,j)t}$, not $u_{d(i,j)t}$.", "id": "P018-N006", "location": "Section 4, sentence after equation (4), p. 569 (PDF p. 6)", "pdf_page": 6, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T22"], "text": {"as_printed": "\\ldots\\ $X_i$, $X_j$, $\\alpha_i$, $\\alpha_j$, and $\\hl{u_{d(i,j)t}}$ are independent draws from standard normal distributions, and the compound error, $\\epsilon_{d(i,j)t} = \\alpha_i + \\alpha_j + \\nu_{d(i,j)t}$, is unobserved.", "corrected": "\\ldots\\ $X_i$, $X_j$, $\\alpha_i$, $\\alpha_j$, and $\\hl{\\nu_{d(i,j)t}}$ are independent draws from standard normal distributions, and the compound error, $\\epsilon_{d(i,j)t} = \\alpha_i + \\alpha_j + \\nu_{d(i,j)t}$, is unobserved.", "evidence": "I checked the printed page.", "problem": "Equation~(4) and the compound error in the same sentence write the shock as $\\nu_{d(i,j)t}$. No $u_{d(i,j)t}$ is defined.", "what_survives": "The simulation design and all results are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N006", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the paragraph after Figure 1. Red boxes mark the words (as illustrated in the, at the end of one line, and Appendix), at the start of the next line.", "caption": "P. 570 (PDF p. 7), paragraph after Figure 1. The marked parenthesis points to the Appendix, while the illustration is Figure 4 of the Supporting Information.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "The illustration of the misspecified simulation design is Figure 4 of the Supporting Information, and not in the Appendix.", "id": "P018-N008", "location": "Section 4, p. 570 (PDF p. 7), paragraph after Figure 1", "pdf_page": 7, "printed_page": "570", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "The degree of misspecification in this case is not extreme (as illustrated in \\hl{the Appendix}) and resembles the kind of approximations that are common in applied work (Buja et al. 2014).", "corrected": "The degree of misspecification in this case is not extreme (as illustrated in \\hl{Figure 4 of the supplementary information}) and resembles the kind of approximations that are common in applied work (Buja et al. 2014).", "evidence": "I checked the pages: p.~570, Appendices A--E, and pp.~3--4 of the Supporting Information, where Figure 4 has the caption ``Illustrative scatter plot and linear approximation for the misspecification case''.", "problem": "Appendices A--E (pp.~573--576) contain the proofs and the extensions to weighted observations and generalized linear models. They contain no illustration of the misspecified design. The scatter plot with its linear approximation is Figure 4 on p.~3 of the Supporting Information, and p.~4 there discusses it. On p.~565 the article distinguishes its appendix from this supplement, which it calls the supplementary information.", "what_survives": "Only the cross-reference changes. No result depends on it."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N008", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the first paragraph of Appendix E, the display defining the matrices A and B, the logistic estimating equation, and the line that defines p d. One red box marks the argument D with subscript i in the second factor of B. A second red box marks expit of X d beta hat, printed with a bold X and no prime.", "caption": "Appendix E, p. 576 (PDF p. 13). Marked: the argument $D_i$ of the second score in $\\mathbf B$, and the logistic index printed without a transpose.", "url": "https://pmaronow.github.io/llm-audit/works/P018/exhibits/P018-N010.png"}], "formal_release": null, "formal_statements": [{"id": "P018-T23", "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-T23"}], "headline": "In Appendix E the second score in $\\mathbf B$ takes the same data argument $D$ as the first, and the logistic index is the scalar $X_d'\\hat\\beta$.", "id": "P018-N010", "location": "Appendix E, p. 576 (PDF p. 13): display defining $\\mathbf A$ and $\\mathbf B$, and the definition of $p_d$", "pdf_page": 13, "printed_page": "576", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P032/", "where": null, "work_id": "P032"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P018-T23"], "text": {"as_printed": "Given an estimating equation, $\\psi(D;\\theta)$, on data $D$, and with parameters $\\theta$ and parameter estimates $\\hat\\theta$, the sandwich approximation for the variance is $\\mathbf A^{-1}\\mathbf B\\{\\mathbf A^{-1}\\}'$, where\n\\[\n\\mathbf A=\\mathrm E\\Big[-\\frac{\\partial}{\\partial\\theta'}\\psi(D;\\hat\\theta)\\Big]\\ \\text{and}\\ \\ \\mathbf B=\\mathrm E\\,\\psi(D;\\hat\\theta)\\psi(D_{\\hl{i}};\\hat\\theta)'.\n\\]\n[\\ldots] where $p_d=\\operatorname{expit}(\\hl{\\mathbf X_d\\hat\\beta})$, the predicted probability for dyad $d$.", "corrected": "Given an estimating equation, $\\psi(D;\\theta)$, on data $D$, and with parameters $\\theta$ and parameter estimates $\\hat\\theta$, the sandwich approximation for the variance is $\\mathbf A^{-1}\\mathbf B\\{\\mathbf A^{-1}\\}'$, where\n\\[\n\\mathbf A=\\mathrm E\\Big[-\\frac{\\partial}{\\partial\\theta'}\\psi(D;\\hat\\theta)\\Big]\\ \\text{and}\\ \\ \\mathbf B=\\mathrm E\\big[\\psi(D;\\hat\\theta)\\psi(\\hl{D};\\hat\\theta)'\\big].\n\\]\n[\\ldots] where $p_d=\\operatorname{expit}(\\hl{X_d'\\hat\\beta})$, the predicted probability for dyad $d$.", "evidence": "I checked the page and the dimensions. Let $s=\\psi(D;\\hat\\theta)$, a vector with the dimension of $\\theta$. Then $\\mathrm E[ss']$ is a square matrix of the dimension of $\\mathbf A$, as the product $\\mathbf A^{-1}\\mathbf B\\{\\mathbf A^{-1}\\}'$ requires. For $k>1$ the product of two $k\\times1$ vectors is not defined, and $X_d'\\hat\\beta$ is the scalar index of the logistic model.", "problem": "No $D_i$ is defined. The paragraph defines the data $D$, the parameter and the estimating function $\\psi(D;\\theta)$, and $\\mathbf B$ is the second moment of the score that defines $\\mathbf A$. In the logistic index, $X_d$ and $\\hat\\beta$ are both column vectors of length $k$, so their product is not defined for $k>1$. The article writes the regression as $Y_d=X_d'\\beta+\\epsilon_d$ on p.~566, and the scalar $X_d'\\hat\\beta$ is meant. The symbol is also set in bold here, while the regressor of a dyad is $X_d$ elsewhere.", "what_survives": "The intended score moment and the predicted probability are unchanged. The paragraph says that $D$ denotes the data here, and this second use of the symbol, beside $D$ for the number of dyads, is not an error. The cross term of the estimator $\\hat V_l$ that follows is corrected in \\fid{P018-E005}."}, "url": "https://pmaronow.github.io/llm-audit/works/P018/#P018-N010", "work": {"arxiv_id": "1312.3398", "doi": "10.1093/pan/mpv018", "title": "Cluster-Robust Variance Estimation for Dyadic Data", "url": "https://pmaronow.github.io/llm-audit/works/P018/"}, "work_id": "P018"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first paragraph of Section 4.1 and three lines of the next. A red box marks 'V_i + 1' in the sentence 'Any valid test of distributional equality between V_i (under Z_i = 0, Y_i = 1) and V_i + 1 (under Z_i = 1, Y_i = 1) will permit rejection of the null.' The sentence before it states Pr[V_i = t | Z_i = 0, Y_i = 1] = Pr[V_i = (t + 1) | Z_i = 1, Y_i = 1].", "caption": "Section 4.1, first paragraph, p. 50 (PDF p. 8). The marked term shifts the list response of the treated arm up by one, and the identity two lines above it requires a shift down.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T5", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T5"}, {"id": "P019-T6", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T6"}], "headline": "The sentence on distributional tests shifts the list response of the treated arm the wrong way: the response to compare with $V_i$ in the control arm is $V_i-1$ in the treated arm, not $V_i+1$.", "id": "P019-E001", "location": "Section 4.1, first paragraph, p. 50 (PDF p. 8)", "pdf_page": 8, "printed_page": "50", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P019-T5", "P019-T6"], "text": {"as_printed": "``Under these assumptions, for all $t$, $\\Pr[V_i=t\\mid Z_i=0,Y_i=1]=\\Pr[V_i=(t+1)\\mid Z_i=1,Y_i=1]$, thus tests of distributional equality are appropriate. Any valid test of distributional equality between $V_i$ (under $Z_i=0$, $Y_i=1$) and $\\hl{V_i+1}$ (under $Z_i=1$, $Y_i=1$) will permit rejection of the null.''", "corrected": "``Any valid test of distributional equality between $V_i$ (under $Z_i=0$, $Y_i=1$) and $\\hl{V_i-1}$ (under $Z_i=1$, $Y_i=1$) will permit rejection of the null.''\n\nIn full: suppose that Assumptions 1--3 hold and that the events $\\{Z_i=z,Y_i=1\\}$, $z\\in\\{0,1\\}$, have positive probability, as Assumption 4 requires. Then for all $t$,\n\\[\n\\Pr[V_i=t\\mid Z_i=0,Y_i=1]=\\Pr[V_i-1=t\\mid Z_i=1,Y_i=1].\n\\]\nEquivalently, one may compare $V_i+1$ in the control arm with $V_i$ in the treated arm. The scalar test of $\\beta=1$ remains valid.", "evidence": "The evidence is a derivation and an exact counterexample to the printed comparison. Suppose that Assumptions 1--3 hold and that $Y_i=1$. Then $X_i=1$ by Assumption 1, and by Assumption 2 I have $V_i=W_i$ in the control arm and $V_i=W_i+1$ in the treated arm. By Assumption 3, the conditional distribution of $W_i$ given $Y_i=1$ is the same in the two arms. I conclude that $V_i$ in the control arm and $V_i-1$ in the treated arm have the same distribution. For the counterexample, let $X_i=Y_i$ be a Bernoulli variable with mean $1/2$, let $W_i$ and $Z_i$ be Bernoulli variables with mean $1/2$ such that $X_i$, $W_i$ and $Z_i$ are independent, and set $V_i=W_i+X_iZ_i$. All four assumptions hold. Among respondents with $Y_i=1$, the response $V_i$ in the control arm takes the values $0$ and $1$ with probability $1/2$ each. In the treated arm, $V_i+1$ takes the values $2$ and $3$ with probability $1/2$ each, and $V_i-1$ takes the values $0$ and $1$ with probability $1/2$ each.", "problem": "The identity in the first sentence says that, among respondents who answer ``Yes'', the list response in the treated arm is distributed as the list response in the control arm plus one. The distributions that are equal under the null are therefore those of $V_i$ in the control arm and $V_i-1$ in the treated arm, or equivalently those of $V_i+1$ in the control arm and $V_i$ in the treated arm. The second sentence compares $V_i$ in the control arm with $V_i+1$ in the treated arm, that is, $W_i$ with $W_i+2$. These two distributions need not be equal when all four assumptions hold, and a test built on the sentence as printed can systematically reject a valid model.", "what_survives": "The probability identity, the null $\\beta=1$, the test of Proposition 3 and the reported placebo tests are unaffected. The replication code tests the difference in means against 1, as the paper describes. Only a distributional test implemented from the sentence as printed would use the wrong comparison."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E001", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Proposition 1 with equation (3) and the four lines that follow. A red box marks 'Assumption 4 (Non-Degenerate Distributions),' in 'Given Assumption 4 (Non-Degenerate Distributions), the asymptotic variance of mu-hat is characterized by'. The paragraph after the equation begins 'The proof is given in Appendix B. Under Assumptions 1-4, mu-hat is root-n consistent and asymptotically normal'.", "caption": "Proposition 1, p. 49 (PDF p. 7). The marked hypothesis names Assumption 4 alone, and the sentence after the proposition says “Under Assumptions 1–4”.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T18", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T18"}], "headline": "Proposition 1 names only Assumption 4, but its variance formula (3) also uses Assumptions 1–3; the hypothesis should read “Given Assumptions 1–4”.", "id": "P019-E002", "location": "Proposition 1, p. 49 (PDF p. 7)", "pdf_page": 7, "printed_page": "49", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P019-T18"], "text": {"as_printed": "``Proposition 1. Given \\hl{Assumption 4 (Non-Degenerate Distributions)}, the asymptotic variance of $\\hat\\mu$ is characterized by''\n\\[\n\\begin{aligned}\n\\operatorname*{plim}_{n\\to\\infty} n\\operatorname{Var}[\\hat\\mu]={}&\\frac{(1-\\mu)^2}{1-\\mu p}\\,\\mu p+(1-\\mu p)\\\\\n&\\times\\left[\\frac{\\operatorname{Var}[V_i\\mid Z_i=1,Y_i=0]}{\\gamma}+\\frac{\\operatorname{Var}[V_i\\mid Z_i=0,Y_i=0]}{1-\\gamma}\\right].\\qquad(3)\n\\end{aligned}\n\\]", "corrected": "``Proposition 1. Given \\hl{Assumptions 1--4}, the asymptotic variance of $\\hat\\mu$ is characterized by''\n\\[\n\\operatorname*{plim}_{n\\to\\infty} n\\operatorname{Var}[\\hat\\mu]=\\frac{(1-\\mu)^2}{1-\\mu p}\\,\\mu p+(1-\\mu p)\\left[\\frac{\\operatorname{Var}[V_i\\mid Z_i=1,Y_i=0]}{\\gamma}+\\frac{\\operatorname{Var}[V_i\\mid Z_i=0,Y_i=0]}{1-\\gamma}\\right].\\qquad(3)\n\\]", "intended_meaning": "The proposition is stated within the model of Section 2, in which Assumptions 1--3 are maintained, and Assumption 4 is the condition added for inference. The sentence after the proposition, Corollary 1 and Proposition 2 all say ``Assumptions 1--4'', and under these four assumptions formula (3) is correct. Read as a self-contained statement, the proposition omits hypotheses that (3) needs. To see where they enter, I write $q=\\mathrm{E}[Y_i]$, $m_z=\\mathrm{E}[V_i\\mid Z_i=z,Y_i=0]$, $d=m_1-m_0$ and $s_z=\\operatorname{Var}[V_i\\mid Z_i=z,Y_i=0]$. For independent observations with $Z_i$ independent of $Y_i$, the delta method gives the asymptotic variance\n\\[\nq(1-q)(1-d)^2+(1-q)\\left[\\frac{s_1}{\\gamma}+\\frac{s_0}{1-\\gamma}\\right].\n\\]\nBy Assumption 1, I have $q=\\mu p$, and Assumptions 1--3 imply that $d=\\mathrm{E}[X_i\\mid Y_i=0]=(\\mu-\\mu p)/(1-\\mu p)$, so that $1-d=(1-\\mu)/(1-\\mu p)$. Substituting these values, I obtain (3). Note that the formula can fail without Assumption 1. Let $X_i$, $Y_i$, $W_i$ and $Z_i$ be independent Bernoulli variables with mean $1/2$, and set $V_i=W_i+X_iZ_i$. Then Assumptions 2--4 hold and Assumption 1 does not. In this example $\\mu=p=\\gamma=1/2$, $s_1=1/2$ and $s_0=1/4$, the asymptotic variance of $\\hat\\mu$ is $13/16$, and the right side of (3) is $29/24$. The proof of the proposition in Appendix B is the subject of \\fid{P019-E003}."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E002", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of display (6) in the proof of Lemma 2, with the two lines of text before it and the line after it. A red box marks the sum over i from 1 to k of the probability that the sum of Z-j over j from 1 to k equals i. The display sets the conditional expectation of V-bar-1,0 given 1 minus Y-bar equal to k over n first to E of V-1 given Z-1 equal to 1 and Y-1 equal to 0 times this sum, and then to that expectation alone.", "caption": "Proof of Lemma 2, display (6), p. 62 (PDF p. 20). The marked sum is set equal to 1 in the next line; it equals $1-(1-\\gamma)^{k}$, because the case of no treated subject among the $k$ is left out.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003.png"}, {"alt": "Crop of the proof of Proposition 1. Its first line reads 'Proof. The proof proceeds by working with linearized variances:' and is followed by display (8), which writes the variance of mu-hat as the variance of the product of 1 minus Y-bar and the quantity V-bar-1,0 minus V-bar-0,0 minus 1. Red boxes mark the words 'are uncorrelated,' in the sentence 'By Lemma 2, V-bar-1,0 and V-bar-0,0 are uncorrelated, so the variance of the product in (8) decomposes as follows:' and the first equality of (9), which writes the variance of mu-hat as a sum of three products of means and variances of 1 minus Y-bar and of V-bar-1,0 minus V-bar-0,0. The next equality in the crop ends with plus O of n to the minus 2.", "caption": "Proof of Proposition 1, from its first line through the first two equalities of display (9), p. 63 (PDF p. 21). The proof begins by saying that it works with linearized variances. The marked equality is the variance formula for a product of independent variables; it is printed without a remainder term, and the next equality carries $+O(n^{-2})$. The marked words omit “with $1-\\bar Y$”.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003-b.png"}, {"alt": "Crop of the last line of display (9) and the sentence after it. A red box marks the equation n gamma equals the sum of Z-i over i from 1 to n, in the sentence 'where gamma is the probability of receiving treatment, so n gamma equals the sum of Z-i. Multiplying by n yields the desired result.'", "caption": "Proof of Proposition 1, sentence after display (9), p. 63 (PDF p. 21). The marked equation sets $n\\gamma$, an expected count, equal to the realized number of treated subjects.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003-c.png"}, {"alt": "Crop of the definitions of the sample means in Section 3 and of Lemma 2. Red boxes mark the two ratios that define V-bar-1,0 and V-bar-0,0, each a sum of list responses divided by the number of subjects in the subgroup, and the statement 'Lemma 2. V-bar-1,0 and V-bar-0,0 are uncorrelated with 1 minus Y-bar.'", "caption": "Section 3, p. 48 (PDF p. 6). The two marked ratios are $0/0$ when their subgroup is empty, and the marked lemma is stated without a condition.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003-d.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T3", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T3"}, {"id": "P019-T4", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T4"}, {"id": "P019-T18", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T18"}], "headline": "The proofs of Lemma 2 and Proposition 1 in Appendix B are worded as exact arguments for a finite sample, but they are meant to leading order and on samples in which both subgroups of subjects who answer “No” are non-empty, as the announced linearization indicates; read in this way they are correct, and formula (3) is unchanged.", "id": "P019-E003", "location": "Appendix B.1, proof of Lemma 2, displays (5)–(7), pp. 62–63 (PDF pp. 20–21); proof of Proposition 1, display (9) and the sentences before and after it, p. 63 (PDF p. 21); definitions of the subgroup means and Lemma 2, Section 3, p. 48 (PDF p. 6)", "pdf_page": 20, "printed_page": "62–63", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P019-T18", "P019-T3", "P019-T4"], "text": {"as_printed": "Section 3, p.~48:\n\\[\n\\bar V_{1,0}=\\frac{\\sum_{i=1}^{n}V_iZ_i(1-Y_i)}{\\sum_{i=1}^{n}Z_i(1-Y_i)}\n\\qquad\\text{and}\\qquad\n\\bar V_{0,0}=\\frac{\\sum_{i=1}^{n}V_i(1-Z_i)(1-Y_i)}{\\sum_{i=1}^{n}(1-Z_i)(1-Y_i)}.\n\\]\n``Lemma 2. $\\bar V_{1,0}$ and $\\bar V_{0,0}$ \\hl{are uncorrelated with} $1-\\bar Y$.''\n\nProof of Lemma 2, p.~62, display (6):\n\\[\n\\mathrm{E}\\Bigl[\\bar V_{1,0}\\Bigm|1-\\bar Y=\\frac kn\\Bigr]\n=\\mathrm{E}[V_1\\mid Z_1=1,Y_1=0]\\,\\hl{\\sum_{i=1}^{k}\\Pr\\Bigl(\\sum_{j=1}^{k}Z_j=i\\Bigr)}\n=\\mathrm{E}[V_1\\mid Z_1=1,Y_1=0].\n\\]\nDisplay (7), p.~63, has the same step for $\\bar V_{0,0}$, with $\\Pr\\bigl(\\sum_{j=1}^{k}(1-Z_j)=i\\bigr)$.\n\nProof of Proposition 1, p.~63. The proof begins ``The proof proceeds by working with linearized variances:'', followed by display (8). It continues: ``By Lemma 2, $\\bar V_{1,0}$ and $\\bar V_{0,0}$ \\hl{are uncorrelated}, so the variance of the product in (8) decomposes as follows:''\n\\[\n\\begin{aligned}\n\\operatorname{Var}[\\hat\\mu]\\ \\hl{=}\\ {}&(\\mathrm{E}[1-\\bar Y])^{2}\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}]\n+\\operatorname{Var}[1-\\bar Y]\\,(\\mathrm{E}[\\bar V_{1,0}-\\bar V_{0,0}-1])^{2}\\\\\n&+\\operatorname{Var}[1-\\bar Y]\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}]\n\\end{aligned}\n\\]\nThis is the first equality of (9); the later lines of (9) carry $+O(n^{-2})$.\n\nAfter (9), p.~63: ``where $\\gamma$ is the probability of receiving treatment, so $\\hl{n\\gamma=\\sum_{i=1}^{n}Z_i}$. Multiplying by $n$ yields the desired result.''", "corrected": "\\emph{Convention.} When a subgroup is empty, its mean is set to a fixed bounded value, for example 0, and the variance estimator is set to a fixed bounded value wherever its formula is undefined. This is a definition and not a new assumption on the population.\n\n\\emph{Lemma 2.} Suppose that $\\hl{n\\ge2}$, that the subjects are independent and identically distributed and that Assumptions 1--4 hold. Let $N_{z,0}=\\sum_{i=1}^{n}\\mathbf 1\\{Z_i=z,\\,Y_i=0\\}$ for $z\\in\\{0,1\\}$ and $D=\\{N_{1,0}>0,\\ N_{0,0}>0\\}$. Then, on $D$,\n\\[\n\\mathrm{E}[\\bar V_{z,0}\\mid Y_1,Z_1,\\dots,Y_n,Z_n]=\\mathrm{E}[V_i\\mid Z_i=z,Y_i=0]\\qquad(z=0,1),\n\\]\nand, \\hl{conditionally on $D$}, $\\bar V_{1,0}$ and $\\bar V_{0,0}$ are uncorrelated with $1-\\bar Y$. In the proof, condition on all of $(Y_1,Z_1,\\dots,Y_n,Z_n)$ in place of the argument of displays (4)--(7).\n\n\\emph{Proof of Proposition 1.} ``By Lemma 2, $\\bar V_{1,0}$ and $\\bar V_{0,0}$ are uncorrelated \\hl{with $1-\\bar Y$ given $D$}. \\hl{Using conditional total variance and a first-order expansion}, the variance of the product in (8) decomposes as follows:''\n\\[\n\\begin{aligned}\n\\operatorname{Var}[\\hat\\mu\\mid D]={}&(\\mathrm{E}[1-\\bar Y\\mid D])^{2}\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}\\mid D]\n+\\operatorname{Var}[1-\\bar Y\\mid D]\\,(\\mathrm{E}[\\bar V_{1,0}-\\bar V_{0,0}-1\\mid D])^{2}\\\\\n&+\\operatorname{Var}[1-\\bar Y\\mid D]\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}\\mid D]\\ \\hl{{}+O(n^{-2})}.\n\\end{aligned}\n\\]\nThe remaining lines of (9) hold to their stated order. With the convention above, $n\\operatorname{Var}[\\hat\\mu]$ converges to the right-hand side of (3), whichever bounded values are chosen.\n\nAfter (9): ``where $\\gamma$ is the probability of receiving treatment, so $n\\gamma=\\hl{\\mathrm{E}}\\bigl[\\sum_{i=1}^{n}Z_i\\bigr]$. Multiplying by $n$ yields the desired result.''", "intended_meaning": "The proof of Proposition 1 begins with the words ``The proof proceeds by working with linearized variances'', the later lines of (9) carry $+O(n^{-2})$, and the proposition states a limit. The argument of Appendix B is therefore meant to leading order in $n$. The subgroup means are ratios, which are defined only when both subgroups of subjects with $Y_i=0$ are non-empty, and the argument concerns such samples. On this reading it is correct, although its steps are worded as exact statements for a finite sample.\n\nLet $N_{z,0}$ be the number of subjects with $Z_i=z$ and $Y_i=0$, and let $D=\\{N_{1,0}>0,\\ N_{0,0}>0\\}$. I write $m_z$ and $s_z$ for the mean and the variance of $V_i$ given $Z_i=z$ and $Y_i=0$. Under independent sampling, the complement of $D$ has positive probability for every $n$. The sums in (6) and (7) leave out the case of an empty subgroup, and they equal $1-(1-\\gamma)^{k}$ and $1-\\gamma^{k}$, not 1. Given all the direct answers and assignments, on $D$ the mean $\\bar V_{z,0}$ has conditional expectation $m_z$ and conditional variance $s_z/N_{z,0}$. Since $1-\\bar Y$ is determined by the direct answers, this implies that $\\bar V_{1,0}$ and $\\bar V_{0,0}$ are uncorrelated with $1-\\bar Y$ conditionally on $D$. This is the lemma as intended.\n\nThe first equality of (9) is the formula for the variance of a product of two independent variables, and Lemma 2 does not imply it as an exact identity. Let $h=s_1/N_{1,0}+s_0/N_{0,0}$, which on $D$ is the conditional variance of $\\bar V_{1,0}-\\bar V_{0,0}$ given all the direct answers and assignments. By the law of total variance, I find that $\\operatorname{Var}[\\hat\\mu\\mid D]$ minus the right-hand side of the first equality, with its moments taken conditionally on $D$, equals\n\\[\n\\operatorname{Cov}\\bigl((1-\\bar Y)^{2},h\\bigm|D\\bigr)=-\\frac{2\\mu p}{n^{2}}\\Bigl(\\frac{s_1}{\\gamma}+\\frac{s_0}{1-\\gamma}\\Bigr)+o(n^{-2}).\n\\]\nThis term has the order of the remainder in the later lines of (9), and the limit in (3) does not depend on it. The sentence before (9) cites Lemma 2 for the statement that the two subgroup means are uncorrelated. The lemma says that each is uncorrelated with $1-\\bar Y$, and the words ``with $1-\\bar Y$'' are to be supplied.\n\nSince $\\gamma=\\Pr(Z_i=1)$ is a population quantity, $n\\gamma$ is the expected number of treated subjects, which is what the calculation to leading order uses. The realized number is $\\sum_iZ_i=n\\hat\\gamma$, with $\\hat\\gamma$ as defined on p.~49. In the first list experiment of Study A, 245 of the 500 subjects were treated, while $n\\gamma=250$.\n\nFormula (3), the asymptotic normality of $\\hat\\mu$, Corollary 1 and Proposition 2 are unaffected. Formula (3) also follows from the influence function of $\\hat\\mu$, without Lemma 2. In the original main analysis every subgroup defined by $(Z_i,Y_i)$ has at least 21 subjects. The hypothesis of Proposition 1 is the subject of \\fid{P019-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E003", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of Appendix C. Red boxes mark the sentence 'The treatment list response for Design Affected subjects was generated as a ceiling effect: the control list plus one, except for those with a 4 on the control list; those units' treatment list response remained equal to 4.', which ends with the footnote mark 1. The crop begins partway through the sentence specifying a binomial distribution with a success probability of 0.4 and four trials. The lines above the marked sentence say that treatment list responses of units meeting the assumptions were set equal to the control list response plus one.", "caption": "Online supplement, Appendix C, second paragraph, p. 2. The marked sentence describes the simulated design effect as the control-list response plus one, capped at 4. The superscript refers to footnote 1.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004.png"}, {"alt": "Crop of four lines. Red boxes mark the words 'or change their responses to the non-sensitive items on the treatment list, the placebo test achieves 80 percent power.' in the sentence that begins 'When approximately 20 percent of a sample of 800 confessors falsely confess, lie on the treatment list,'. The page prints percent signs.", "caption": "Online supplement, first lines of p. 3. The marked clause extends the statement of 80% power to the design effect.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004-b.png"}, {"alt": "Figure A1, 'Power of Placebo Test I to Detect Violations of Assumptions 1-3', with four shaded panels that show power against the total number of admitters, from 100 to 1000, and a proportion from 0 to 1. A red box marks the title of the lower-left panel, 'Violated assumption: no design effects', whose horizontal axis is 'Proportion of Design Affected'. In that panel the red band for power near 0.8 runs from a proportion of about 0.6 at 100 admitters to about 0.2 at 800 to 1000 admitters, much as in the two upper panels for false confessions and for liars.", "caption": "Online supplement, Figure A1, p. 3. The marked panel is the one for design effects. Its red band, which marks power near 0.80, lies near a proportion of 0.2 at 800 admitters.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004-c.png"}, {"alt": "Bar chart. The horizontal axis is the share of subjects designated design-affected, at 0, 20 and 100 percent. The vertical axis is power, from 0 to 1. Blue bars, for the rule in the replication code, have heights 0.050, 0.780 and 1.000. Orange bars, for the rule described in Appendix C, have heights 0.050, 0.051 and 0.067. A horizontal line at 0.80 marks the power stated in the supplement for 20 percent of 800, and a line at 0.05 marks the level of the test.", "caption": "Power of Placebo Test I at the 5% level with 800 admitters, by the share of subjects designated design-affected, under the coded rule (blue) and the described rule (orange). Each bar is the share of rejections in 20,000 simulated experiments, with Monte Carlo standard error at most 0.003. The line at 0.80 is the power that the supplement states for 20% of 800.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004-power.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T1", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T1"}, {"id": "P019-T2", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T2"}, {"id": "P019-T16", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T16"}], "headline": "The simulated design effect is not the “ceiling effect” that Appendix C describes: the code adds the sensitive item after the capped increment, and against the rule as described Placebo Test I has power of about 0.05, not 0.80, with 800 admitters and 20% design-affected.", "id": "P019-E004", "location": "Online supplement, Appendix C, second paragraph and footnote 1, p. 2; the sentence on 80% power and Figure A1, lower-left panel, p. 3 (supplement PDF pp. 2–3). Replication files: ACCG_list_source.R, lines 82–86 and 103–106. The main text points to these results in Section 4.1, pp. 50–51 (PDF pp. 8–9)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P019-T1", "P019-T16", "P019-T2"], "text": {"as_printed": "Supplement p.~2: ``Control list responses were drawn from a binomial distribution with a success probability of 0.4 and four trials. For those units who do meet the assumptions of No False Confessions, No Liars, or No Design Effects, treatment list responses were set equal to the control list response plus one. The treatment list responses for Liars and False Confessors were set equal to their control list responses. The treatment list response for Design Affected subjects was generated as a ``ceiling effect'': \\hl{the control list plus one, except for those with a ``4'' on the control list; those units' treatment list response remained equal to 4}.$^{1}$'' Footnote 1, attached to this sentence: ``This is one of many possible design effects; the power of our test to detect any particular design effect depends on the manner in which it changes subjects' treatment list responses.''\n\nSupplement p.~3: ``When approximately 20\\% of a sample of 800 confessors falsely confess, lie on the treatment list, \\hl{or change their responses to the non-sensitive items on the treatment list}, the placebo test achieves 80\\% power.'' The lower-left panel of Figure A1 has the title ``Violated assumption: no design effects''. Its horizontal axis is the proportion of ``Design Affected'', and its vertical axis is the ``Total Number of Admitters''.\n\nIn the file \\texttt{ACCG\\_list\\_source.R} of the replication files, lines 83--85 are the body of the function \\texttt{design\\_affector}, and lines 103--106 are in the function \\texttt{list\\_simulator}:\n\\begin{itemize}\n\\item \\texttt{x\\_star <- x}\n\\item \\texttt{x\\_star[x!=4] <- x[x!=4] + 1}\n\\item \\texttt{return(x\\_star)}\n\\item \\texttt{V0 <- rbinom(N, size=4, prob=baseline\\_binomial\\_prob)}\n\\item \\texttt{V1 <- V0+ Ystar*(1-listliar)}\n\\item \\texttt{V1[designaffected==1] <- design\\_affector(V0[designaffected==1]) +}\n\\item \\texttt{\\hl{Ystar[designaffected==1]*(1-listliar[designaffected==1])}}\n\\end{itemize}", "corrected": "Supplement p.~2: ``The treatment list response for Design Affected subjects was generated \\hl{by raising the response to the non-sensitive items by one, except for those with a ``4'' on the control list, and then adding the sensitive item. For subjects who engage in the behavior and do not lie on the list, the treatment list response is therefore the control list response plus two, or 5 for those with a ``4'' on the control list.}''\n\nSupplement p.~3: ``When approximately 20\\% of a sample of 800 confessors falsely confess, lie on the treatment list, or \\hl{report one more non-sensitive item on the treatment list unless they already report all four, as described above}, the placebo test achieves \\hl{about} 80\\% power.''\n\nIf the ceiling rule of the printed text is kept instead, the code should set the treatment-list response of a design-affected admitter to $\\min(W+1,4)$ with no further addition, the lower-left panel should be computed again, and the text should say that against this design effect the test has power of about 0.05 with 800 admitters and 20\\% design-affected.", "evidence": "The evidence is an exact comparison of the two rules and a simulation of both with the authors' test function. Running the function \\texttt{design\\_affector} unchanged and adding the sensitive item, I obtain the following responses.\n\\begin{audittable}{rrrr}{Treatment-list response of an admitter in the simulation for the lower-left panel, by control-list response $W$.}\n$W$ & Not design-affected & Design-affected, described rule & Design-affected, coded rule \\\\\n\\hline\n0 & 1 & 1 & 2 \\\\\n1 & 2 & 2 & 3 \\\\\n2 & 3 & 3 & 4 \\\\\n3 & 4 & 4 & 5 \\\\\n4 & 5 & 4 & 5 \\\\\n\\end{audittable}\nThe control-list response $W$ is binomial with four trials and success probability $2/5$, and the values $0,\\dots,4$ have probabilities $81/625$, $216/625$, $216/625$, $96/625$ and $16/625$. Let $A$ indicate a design-affected subject, and let $\\rho$ be the share of such subjects, who are drawn at random independently of $W$ and of the treatment. The treatment-list response of an admitter is $W+1-A\\,\\mathbf{1}\\{W=4\\}$ under the described rule and $W+1+A\\,\\mathbf{1}\\{W<4\\}$ under the coded rule. Taking expectations, I obtain\n\\[\n\\beta_{\\text{described}}=1-\\frac{16}{625}\\,\\rho,\\qquad \\beta_{\\text{coded}}=1+\\frac{609}{625}\\,\\rho.\n\\]\nAt $\\rho=1/5$ these values are $3109/3125=0.99488$ and $3734/3125=1.19488$.\n\nThe simulation follows the script for the panel: the stated number of admitters, 200 withholders and 200 respondents who do not engage in the behavior, with the specified share of all subjects designated design-affected. Both rules are applied to the same draws, the test is the authors' \\texttt{placeboI\\_test}, and the null is rejected when $p<0.05$.\n\\begin{audittable}{rrrr}{Power of Placebo Test I at the 5\\% level, from 20,000 simulated experiments in each row, with Monte Carlo standard errors in parentheses.}\nAdmitters & Share design-affected & Described rule & Coded rule \\\\\n\\hline\n800 & 0 & 0.05015 (0.00154) & 0.05015 (0.00154) \\\\\n800 & 0.20 & 0.05070 (0.00155) & 0.77965 (0.00293) \\\\\n800 & 1 & 0.06710 (0.00177) & 1.00000 \\\\\n1000 & 1 & 0.06950 (0.00180) & 1.00000 \\\\\n\\end{audittable}\nIn the last two rows the coded rule led to rejection in all 20,000 experiments. In a second simulation, written independently and run with another seed, the power is 0.05145 and 0.77780 at 800 admitters and a share of 0.20. The full grid of Figure A1 was not computed again, and the values of its cells were not recovered. The simulation does not show that the power under the described rule stays near 0.05 in arbitrarily large samples, since $\\beta_{\\text{described}}$ differs from 1 whenever $\\rho>0$.", "problem": "Appendix C says that the treatment-list response of a design-affected subject is the control-list response plus one, capped at four. I call this the described rule. I write $W$ for the control-list response, and I call a respondent who answers ``Yes'' to the direct question an admitter, as Figure A1 does. In the simulation for the lower-left panel of Figure A1, no admitter confesses falsely or lies on the list. For a design-affected admitter, the response under the described rule is $\\min(W+1,4)$, that is, $1,2,3,4,4$ for $W=0,\\dots,4$. The replication code does something else. The function \\texttt{design\\_affector} raises the count of non-sensitive items by one unless it is already four, and the simulator then adds the sensitive item. I call this the coded rule. Under the coded rule the response is therefore $\\min(W+1,4)+1$, that is, $2,3,4,5,5$. The two rules differ at every value of $W$.\n\nThe two rules lead to different power. Placebo Test I tests whether the difference $\\beta$ between the mean responses of the two arms among admitters equals 1. Under the described rule a design-affected admitter differs from an unaffected one only when $W=4$, an event of probability $0.4^4=0.0256$. Then $\\beta$ falls below 1 by $0.0256$ times the affected share. Under the coded rule $\\beta$ rises above 1 by $0.9744$ times that share. The lower-left panel of Figure A1 and the last clause of the sentence on 80\\% power report high power. The coded rule has it and the described rule does not. With 800 admitters and 20\\% design-affected, the power is about 0.78 under the coded rule and about 0.05, the level of the test, under the described rule. Footnote 1 says that power depends on the form of the design effect. The statement of 80\\% power is then read as a statement about the rule just described. The evidence cannot decide which of the two the authors intended, the description or the code.", "what_survives": "The defect is confined to three places: the description of the design effect in Appendix C, the reading of the lower-left panel of Figure A1 as that design effect, and the last clause of the sentence on 80\\% power. The example in the main text, with about 80 percent power when 20 percent of 800 are false confessors, concerns false confessions and is not affected. The test formula of Proposition 3, the identification and efficiency results, the other panels of Figure A1, Placebo Test II, and the estimates and placebo p-values of the application do not depend on the simulated design effect. The high-power conclusion does not hold for the printed ceiling rule. Either the statement of 80\\% power is attached to the positive design effect that was simulated, or the ceiling rule is kept and the panel shows power near the level of the test at the point that the text cites."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E004", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last paragraph of p. 50. Red boxes mark the sentence 'We also explore the power of Placebo Test I using a series of Monte Carlo simulations.' The paragraph continues with the factors that are varied, the example of about 30 percent power, and the beginning of the example with 800 subjects.", "caption": "Section 4.1, last paragraph, p. 50 (PDF p. 8). The marked sentence introduces the Monte Carlo power study of Placebo Test I. The faults are in the script deposited for that study and not on this page.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E005.png"}], "formal_release": null, "formal_statements": [], "headline": "The deposited script for Figure A1, the power study of Placebo Test I, cannot run as written, and the script for Tables A2–A8 calls a package that it does not load; no published number is shown to be wrong.", "id": "P019-E005", "location": "Replication files (Harvard Dataverse, doi:10.7910/DVN/29984, version 1.0): the script of source functions, the script for the power study of Placebo Test I and the script for the replication experiment, at the lines quoted below. In the paper: Section 4.1, last paragraph, p. 50 (PDF p. 8), and Figure A1 of the online supplement, p. 3", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "The file \\texttt{readme.txt} says: ``This replication archive contains the data and programs necessary to replicate the figures and tables presented in the main text and the appendix.'' Its entry for \\texttt{ACCG\\_list\\_placebo\\_I\\_power.R} has the three lines ``This file conducts the power analysis of Placebo Test I.'', ``Inputs: ACCG\\_list\\_source'' and ``Outputs: Figure A1''.\n\nIn \\texttt{ACCG\\_list\\_source.R}, line 68 is the last line of the function \\texttt{placeboI\\_test}, and line 112 is in the function \\texttt{list\\_simulator}:\n\\begin{itemize}\n\\item \\texttt{return(c(est=est, se=se, p\\_value=p\\_value, n=n))}\n\\item \\texttt{reject\\_null <- as.numeric(placeboI\\_test(V=V, Z=Z, Y=Y)\\hl{\\$p\\_value}<0.05)}\n\\end{itemize}\nLine 88 begins the definition of \\texttt{list\\_simulator}. The first four arguments of this function are named \\texttt{N\\_trueadmitter}, \\texttt{N\\_falseconfessor}, \\texttt{N\\_withholder} and \\texttt{N\\_innocent}. The other three are named \\texttt{pr\\_listliar}, \\texttt{pr\\_designaffected} and \\texttt{baseline\\_binomial\\_prob}.\n\nIn \\texttt{ACCG\\_list\\_placebo\\_I\\_power.R}, line 41 calls \\hl{\\texttt{listsimulator}} with the named arguments \\hl{\\texttt{N.trueadmitter}}, \\hl{\\texttt{N.falseconfessor}}, \\hl{\\texttt{N.withholder}}, \\hl{\\texttt{N.innocent}} and \\texttt{baseline\\_binomial\\_prob}. Lines 83, 124 and 166 have the same form. Line 83 adds the argument \\hl{\\texttt{pr.listliar}}, and line 124 adds the argument \\hl{\\texttt{pr.designaffected}}.\n\nIn \\texttt{ACCG\\_list\\_replication\\_experiment.R}, line 6, \\texttt{library(lmtest)}, is the only line that loads a package, and the function \\texttt{xtable} is called at lines 45, 58, 73, 88, 100, 113 and 142.", "corrected": "In \\texttt{ACCG\\_list\\_placebo\\_I\\_power.R}, lines 41, 83, 124 and 166 should call \\hl{\\texttt{list\\_simulator}} with the argument names \\hl{\\texttt{N\\_trueadmitter}}, \\hl{\\texttt{N\\_falseconfessor}}, \\hl{\\texttt{N\\_withholder}}, \\hl{\\texttt{N\\_innocent}}, and, where used, \\hl{\\texttt{pr\\_listliar}} and \\hl{\\texttt{pr\\_designaffected}}. Line 112 of \\texttt{ACCG\\_list\\_source.R} should read as follows.\n\\begin{itemize}\n\\item \\texttt{reject\\_null <- as.numeric(placeboI\\_test(V=V, Z=Z, Y=Y)\\hl{[[\"p\\_value\"]]}<0.05)}\n\\end{itemize}\nThe function \\texttt{placeboI\\_test} should keep its numeric vector as return value, since the code for the tables binds these vectors as rows. The script \\texttt{ACCG\\_list\\_replication\\_experiment.R} should load the package with \\hl{\\texttt{library(xtable)}}. The user must also set the working directory and arrange the folders for programs and data, as the comments in the scripts ask. To make new parallel simulations repeatable, the script should fix the seed and initialize the random streams of the workers, for example with \\texttt{clusterSetRNGStream}; a call to \\texttt{set.seed} alone does not reach the workers. A new seed cannot recover the published draws.", "evidence": "The evidence is a set of executable checks on the unchanged files, each in a new R session. The four calls to the simulator report \\texttt{could not find function \"listsimulator\"}. With that name pointed at \\texttt{list\\_simulator}, they report unused arguments. A direct call of \\texttt{list\\_simulator} reports \\texttt{\\$ operator is invalid for atomic vectors}. A small example makes the last failure exact. For $V=(0,2,1,3)$, $Z=(0,0,1,1)$ and $Y=(1,1,1,1)$, the function \\texttt{placeboI\\_test} returns a numeric vector with estimate $1$, standard error $\\sqrt2$, p-value $1$ and $n=4$, and \\texttt{\\$p\\_value} fails on it. With the directories set, the script for Tables A2--A8 stops at line 45 with \\texttt{could not find function \"xtable\"}. With the directories set and \\texttt{xtable} made available, and with the estimator code unchanged, I find that all 360 numeric entries of Tables 3--8 and of Tables A1--A8 of the supplement agree with the publication at the printed precision.", "problem": "The readme says that the archive contains the programs needed to replicate the figures and tables. The two power scripts and the two analysis scripts each begin with a call that sets the working directory, and a comment asks the user to set it. The readme describes a layout with folders for programs and data. This is ordinary setup. After the setup, the script for Figure A1 still cannot run. Its four calls to the simulator use the name \\texttt{listsimulator}, which is not defined, and argument names with dots. The function supplied is \\texttt{list\\_simulator}, and its argument names have underscores. The supplied function itself stops on every call that reaches line 112, since \\texttt{placeboI\\_test} returns a named numeric vector and \\texttt{\\$p\\_value} cannot be extracted from such a vector. The script for Tables A2--A8 stops at its first call to \\texttt{xtable}, a package that it never loads. The script for Figure A1 also sets no seed, and the Monte Carlo draws of the published figure cannot be repeated.", "what_survives": "These faults do not show that any theorem, estimator formula, estimate or power value is wrong. The functions that produce the empirical tables do not call the simulator, and the 360 table entries compared here agree. Two points are not established: which version of the code drew the published Figure A1, and whether the full grid of that figure is recovered once the script runs. The full grid was not computed again."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E005", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The two upper panels of Figure A2, 'Power of Placebo Test II to Detect Violations of Treatment Independence'. A red box marks the title 'Power of Placebo Test II, N= 100' of the left panel. The right panel is titled 'N= 200'. Each panel shades power from white, for 0, to black, for 1, against Pr(Y=1|Z=1) on the horizontal axis and Pr(Y=1|Z=0) on the vertical axis, with a pale band along the diagonal and a red band for power near 0.8 on either side of it.", "caption": "Online supplement, Figure A2, upper panels, p. 4. The marked panel, for $N=100$, is computed without the opposite rejection tail, and the panel beside it, for $N=200$, includes that tail. The omitted tail is largest on the nondegenerate diagonal, where the first panel plots 0.025 and the second 0.05.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T15", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T15"}], "headline": "The panel of Figure A2 for $N=100$ leaves out the rejection tail opposite to the true difference, which the other three panels count; it understates the two-sided power by at most 0.025.", "id": "P019-E006", "location": "Online supplement, Appendix D, p. 3, and Figure A2, first panel, p. 4 (supplement PDF pp. 3–4). Replication files: ACCG_list_placebo_II_power.R, line 31", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P019-T15"], "text": {"as_printed": "Supplement p.~3: ``The power of Placebo Test II to detect violations of Treatment Independence is equivalent to the power of test of a difference in proportions. Following the implementation in R (see accompanying text: Dalgaard 2008, p. 159), the power of the test for $N = 100$, $200$, $500$, and $1000$ is shown in Figure A2 below.'' Figure A2, first panel: ``Power of Placebo Test II, $\\hl{N=100}$''.\n\nReplication files, \\texttt{ACCG\\_list\\_placebo\\_II\\_power.R}. Line 31, for $N=100$, and line 62, for $N=200$, compute the power with the following two calls:\n\\begin{itemize}\n\\item \\texttt{\\hl{power.prop.test(n=50, p1=pr.yes.treat, p2=pr.yes.control)}}\n\\item \\texttt{power.prop.test(n=100, p1=pr.yes.treat, p2=pr.yes.control, strict=TRUE)}\n\\end{itemize}\nLines 93 and 124, for $N=500$ and $N=1000$, have the call of line 62 with \\texttt{n=250} and \\texttt{n=500}.", "corrected": "Line 31 should compute the power with the following call, and the first panel should be drawn from it.\n\\begin{itemize}\n\\item \\texttt{power.prop.test(n=50, p1=pr.yes.treat, p2=pr.yes.control\\hl{, strict=TRUE})}\n\\end{itemize}\nThe description of the figure should read: ``\\hl{Figure A2 shows normal-approximation planning power for a two-sided test of a difference in proportions at the 5\\% level, with equal treatment and control group sizes and both rejection tails included in every panel.}''", "evidence": "The evidence is a derivation of the omitted term, its exact value at one plotted point, and an evaluation of the original calls at every plotted point. Let each group have size $n$, let $p_1$ and $p_2$ be the two proportions, and set $d=|p_1-p_2|$, $S=p_1(1-p_1)+p_2(1-p_2)$, $V=(p_1+p_2)(1-(p_1+p_2)/2)$ and $c=\\Phi^{-1}(0.975)$. Suppose that $S>0$. In the normal approximation of the function, the probability of rejection in the direction of the true difference is $A$ and the probability of rejection in the opposite direction is $B$, where\n\\[\nA=\\Phi\\left(\\frac{\\sqrt n\\,d-c\\sqrt V}{\\sqrt S}\\right),\\qquad B=\\Phi\\left(\\frac{-\\sqrt n\\,d-c\\sqrt V}{\\sqrt S}\\right).\n\\]\nLine 31 plots $A$, and the other three panels plot $A+B$. Since $V-S=(p_1-p_2)^2/2\\ge0$, I have $(\\sqrt n\\,d+c\\sqrt V)/\\sqrt S\\ge c$, which implies that\n\\[\n0<B\\le1-\\Phi(c)=0.025,\n\\]\nwith equality when $p_1=p_2$. At $n=50$ and $p_1=p_2=1/2$ the plotted value is exactly $1/40$, and the value with both tails is $1/20$.\n\\begin{audittable}{rrrrr}{Normal-approximation power for $N=100$ ($n=50$ in each group) at three points: as plotted by line 31, and with both rejection tails.}\n$p_1$ & $p_2$ & As plotted & Both tails & Omitted \\\\\n\\hline\n0.50 & 0.50 & 0.0250 & 0.0500 & 0.0250 \\\\\n0.40 & 0.41 & 0.0316 & 0.0512 & 0.0196 \\\\\n0.40 & 0.50 & 0.1686 & 0.1700 & 0.0014 \\\\\n\\end{audittable}\nAmong the 10,201 points of the first panel, the largest change at a point whose plotted power is at least 0.74 is $1.3\\times10^{-6}$. No point enters or leaves the red band that marks power near 0.80. The two corner points at which both proportions are 0 or both are 1 have $S=0$ and are undefined in every panel.", "problem": "Proposition 4 defines the p-value of Placebo Test II as two-sided. Figure A2 presents the power of the test at four sample sizes, computed with the R function \\texttt{power.prop.test} for two groups of size $N/2$. By default this function counts, in a two-sided calculation, only the rejections in the direction of the true difference. The option \\texttt{strict=TRUE} adds the rejections in the opposite direction. The calls for $N=200$, $500$ and $1000$ set the option, and the call for $N=100$ does not. The first panel therefore shows a different quantity from the other three. The omitted probability is positive when $S>0$, with $S$ defined below, and it is at most 0.025, the value that it takes when the two proportions are equal and lie strictly between 0 and 1. On that diagonal the first panel plots 0.025 where the other panels plot 0.05.", "what_survives": "The other three panels, the red band near 0.80 on the plotted grid, and the definition of the test in Proposition 4 are unaffected. No empirical p-value uses this function: the replication code computes the p-values of Placebo Test II directly as $2\\Phi(-|\\hat\\delta|/\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\delta]})$ in a separate function. The correction stays within the normal approximation and does not make the figure an exact calculation of power in finite samples."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E006", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the CNN row of Table A1, Survey Questions and Response Options. Red boxes mark the words 'read or watch' in the list question, 'select how many you read or watch in the course of an ordinary month', and the words 'do you watch CNN?' in the direct question. The list items are The New York Times, CNN in bold, The Huffington Post, Fox News and Politico; the response options to the direct question are Yes and No.", "caption": "Table A1, p. 61 (PDF p. 19), row “CNN (L5 & D5)”. The list question asks about reading or watching; the direct question asks about watching only.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E007.png"}, {"alt": "Crop of a paragraph of Section 5.2. A red box, which runs over two lines, marks the words 'providing confidence that the list experiments and the direct questions are measuring the same quantities' in the sentence that introduces Figure 1.", "caption": "Section 5.2, p. 54 (PDF p. 12). The marked remark takes agreement of the estimates as a sign that the two kinds of question measure the same quantities.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E007-b.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T13", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T13"}, {"id": "P019-T14", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T14"}], "headline": "Because the list question for CNN asks which organizations respondents “read or watch” and the direct question asks only whether they “watch CNN”, a direct “No” need not be withholding, and the list-based estimates for CNN may measure the broader behaviour; the CNN comparison needs this qualification.", "id": "P019-E007", "location": "Appendix A.1, Table A1, p. 61 (PDF p. 19), row “CNN (L5 & D5)”; Section 5.2, p. 54 (PDF p. 12)", "pdf_page": 19, "printed_page": "61", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P019-T13", "P019-T14"], "text": {"as_printed": "Table A1, row ``CNN (L5 \\& D5)''. List question: ``Here is a list of four (five) news organizations. Please read all four (five) and select how many you \\hl{read or watch} in the course of an ordinary month. We do not want to know which ones, just how many.'' Items: \\textit{The New York Times}; \\textbf{CNN}; The Huffington Post; Fox News; Politico. Direct question: ``In the course of an ordinary month, do you \\hl{watch} CNN?'' Response options: Yes; No.\n\nSection 5.2, p.~54: ``Figure 1 presents these results graphically: the estimates generally agree (providing confidence that the list experiments and the direct questions are measuring the same quantities), and the 95 percent confidence intervals around the combined estimate are always tighter than those around the standard estimate.''", "corrected": "Add to the discussion of the CNN results in Section 5.2: ``For CNN, the list question asks which organizations respondents \\hl{read or watch}, whereas the direct question asks only whether they \\hl{watch} CNN. A person who reads CNN but does not watch it can therefore answer both questions truthfully, be counted by the list and answer `No' to the direct question; that discrepancy need not be withholding about watching. Under the ordinary reading of the list question, \\hl{if the identifying assumptions hold for reading or watching CNN}, the standard list and combined estimates target that broader behaviour.''\n\nThe questions stay as printed in Table A1, since they are the questions that were asked. Future surveys should use the same wording for the sensitive item in the list question and in the direct question.", "evidence": "The evidence is a check of the page and an exact example. Suppose that one-quarter of the population watches CNN and says so, one-quarter reads CNN without watching it, and one-half does neither. Everyone answers both questions truthfully, the baseline count is 1 or 2 with equal probability, and assignment is by a fair coin independent of everything else. Then the mean direct answer is $1/4$, which is the share that watches CNN, while the standard list difference and the combined expression (1) both equal $1/2$, the share that reads or watches. With $X_i$ taken as reading or watching, the numerical conditions of Assumptions 1--4 hold, and so do the null hypotheses of both placebo tests ($\\beta=1$ and $\\delta=0$). If those who only read CNN are replaced by watchers who answer ``No'', then the joint distribution of assignment, direct answer and list count is unchanged, and the responses cannot distinguish the two cases. If those who only read CNN are 1 percent of the population, then the mean direct answer is $1/4$ and the two list-based quantities are $13/50$. Near agreement of the estimates does not show that the questions measure the same behaviour. The example shows what the wording allows. It does not show that such respondents were in the sample, and no reported CNN estimate is shown to be wrong.", "problem": "The paper's framework has one behaviour $X_i$ for each topic: the list counts it for treated respondents, and the direct question asks about it. For CNN the two questions do not ask about the same behaviour. Under the ordinary reading of ``read or watch'', a respondent counts CNN in the list if she reads it or watches it, and she answers ``Yes'' to the direct question only if she watches it. A respondent who reads CNN and does not watch it answers both questions truthfully when she counts CNN in the list and answers ``No''. Therefore, for CNN, a gap between the list-based estimates and the direct estimate need not reflect withholding, and the standard list and combined estimates may measure reading or watching CNN, which is a broader behaviour than the one in the direct question. The remark in Section 5.2 is tentative, but for CNN the wording of the questions does not support it. The paper does not show how respondents understood the list question, and some may have taken ``read or watch'' to mean reading newspapers and watching broadcasters.", "what_survives": "The identification and efficiency results, which assume a single behaviour $X_i$, are unaffected, and so are the other four topics. The CNN entries of Tables 3 and 6 and of Figures 1 and 2, the placebo tests and the replicated effect of the treatment list on direct answers about CNN stand as statements about the responses. Their interpretation for CNN changes in the tables and figures, in the replication, and in supplement Appendix E, which describes the standard list estimate as an estimate of ``those watching CNN''. The paper already reports that CNN fails Placebo Test I in Study A ($p=0.034$) and Placebo Test II in Study B ($p=0.003$), and it calls the combined estimate for CNN in Study B unreliable."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E007", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the proof of Lemma 1. A red box marks 'W_i + mu - W_i' in equation (2), which reads E[V_i | Z_i = 1] - E[V_i | Z_i = 0] = W_i + mu - W_i = mu. A second red box marks 'Assumption 2.' at the end of the sentence 'The result follows since E[X_i | Y_i = 1] = 1 by Assumption 2.'", "caption": "Proof of Lemma 1, p. 48 (PDF p. 6). The first mark is on the middle expression of equation (2), where $W_i$ stands for $\\mathrm{E}[W_i]$. The second is on the citation of Assumption 2 for a consequence of Monotonicity.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008.png"}, {"alt": "Crop of Corollary 1: 'If Assumptions 1-4 hold, then confidence intervals constructed as mu-hat plus or minus z times the square root of the estimated variance will have mu 100(1 - alpha) percent coverage for mu for large n.' A red box marks the first mu, which stands directly before '100(1 - alpha)', and the percent sign follows.", "caption": "Corollary 1, p. 49 (PDF p. 7). The marked $\\mu$ before the coverage percentage does not belong to the sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008-b.png"}, {"alt": "Crop of the start of the proof of Lemma 2, with the sentence 'Suppose that exactly k direct responses are zero', display (4), whose last line conditions on 'Y_j = 0 for j = 1,...,k', and the sentence 'where we have reordered the indices so that Y_1 = ... = Y_k = 1.' A red box marks 'Y_1 = ... = Y_k = 1.'", "caption": "Appendix B, proof of Lemma 2, p. 62 (PDF p. 20). Display (4) conditions on $Y_j=0$ for $j=1,\\dots,k$, and the marked phrase gives the same responses the value 1.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008-c.png"}, {"alt": "Crop of two lines of the proof of Proposition 2, 'By Assumptions 1 and 2 (Monotonicity, No Liars, and No Design Effects), E[V_i | Z_i = 1, Y_i = 1] = E[V_i | Z_i = 0, Y_i = 1] + 1. Then,', followed by the start of a display. A red box marks 'Assumptions 1 and 2'.", "caption": "Appendix B, proof of Proposition 2, p. 64 (PDF p. 22). The marked citation omits Assumption 3.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008-d.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T11", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T11"}, {"id": "P019-T19", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T19"}], "headline": "Five local slips in the derivations: equation (2) should contain $\\mathrm{E}[W_i]$ in place of $W_i$, Monotonicity is Assumption 1 and not Assumption 2, Corollary 1 has a stray $\\mu$, the reordered responses in the proof of Lemma 2 are zeros, and one step in the proof of Proposition 2 also uses Assumption 3.", "id": "P019-E008", "location": "Proof of Lemma 1, equation (2) and last sentence, p. 48 (PDF p. 6); Corollary 1, p. 49 (PDF p. 7); Appendix B, proof of Lemma 2, sentence after display (4), p. 62 (PDF p. 20); Appendix B, proof of Proposition 2, p. 64 (PDF p. 22)", "pdf_page": 6, "printed_page": "48", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P019-T11", "P019-T19"], "text": {"as_printed": "\\begin{enumerate}\n\\item p.~48, proof of Lemma 1: ``By Assumptions 2 and 3,''\n\\[\n\\mathrm{E}[V_i\\mid Z_i=1]-\\mathrm{E}[V_i\\mid Z_i=0]=\\hl{W_i}+\\mu-\\hl{W_i}=\\mu.\\qquad(2)\n\\]\n\\item p.~48, end of the same proof: ``The result follows since $\\mathrm{E}[X_i\\mid Y_i=1]=1$ by \\hl{Assumption 2}.''\n\\item p.~49: ``Corollary 1. If Assumptions 1--4 hold, then confidence intervals constructed as $\\hat\\mu\\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\mu]}$ will have $\\hl{\\mu}$ $100(1-\\alpha)\\%$ coverage for $\\mu$ for large $n$.''\n\\item p.~62, proof of Lemma 2, after display (4): ``where we have reordered the indices so that $Y_1=\\cdots=Y_k=\\hl{1}$.''\n\\item p.~64, proof of Proposition 2: ``By \\hl{Assumptions 1 and 2} (Monotonicity, No Liars, and No Design Effects), $\\mathrm{E}[V_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[V_i\\mid Z_i=0,Y_i=1]+1$.''\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item p.~48: ``By Assumptions 2 and 3,''\n\\[\n\\mathrm{E}[V_i\\mid Z_i=1]-\\mathrm{E}[V_i\\mid Z_i=0]=\\hl{\\mathrm{E}[W_i]}+\\mu-\\hl{\\mathrm{E}[W_i]}=\\mu.\\qquad(2)\n\\]\n\\item p.~48: ``The result follows since $\\mathrm{E}[X_i\\mid Y_i=1]=1$ by \\hl{Assumption 1}.''\n\\item p.~49: ``Corollary 1. If Assumptions 1--4 hold, then confidence intervals constructed as $\\hat\\mu\\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\mu]}$ will have $100(1-\\alpha)\\%$ coverage for $\\mu$ for large $n$.'' The $\\mu$ before the percentage is deleted.\n\\item p.~62: ``where we have reordered the indices so that $Y_1=\\cdots=Y_k=\\hl{0}$.''\n\\item p.~64: ``By \\hl{Assumptions 1--3} (Monotonicity, No Liars, No Design Effects, \\hl{and Treatment Independence}), $\\mathrm{E}[V_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[V_i\\mid Z_i=0,Y_i=1]+1$.''\n\\end{enumerate}", "evidence": "The evidence is a check of the pages against the definitions of the assumptions on pp.~46--47, and a derivation of the identities in (a) and (e). For (a), recall that Assumption 2 states that $V_i=W_i+X_iZ_i$. By Assumption 3, the pair $(W_i,X_i)$ is independent of $Z_i$, and I have\n\\[\n\\mathrm{E}[V_i\\mid Z_i=1]=\\mathrm{E}[W_i]+\\mathrm{E}[X_i]=\\mathrm{E}[W_i]+\\mu,\\qquad \\mathrm{E}[V_i\\mid Z_i=0]=\\mathrm{E}[W_i].\n\\]\nFor (e), suppose that $Y_i=1$. Then Assumption 1 implies that $X_i=1$, and by Assumption 2 I have $V_i=W_i+Z_i$. By Assumption 3, the conditional distribution of $W_i$ given $Y_i=1$ is the same in the two arms, so that $\\mathrm{E}[W_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[W_i\\mid Z_i=0,Y_i=1]$. I conclude that $\\mathrm{E}[V_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[V_i\\mid Z_i=0,Y_i=1]+1$.", "problem": "(a) The middle expression of equation (2) is a difference of two expectations. It should contain the mean $\\mathrm{E}[W_i]$ of the baseline response and not the random variable $W_i$. The two terms cancel in either form, and the value $\\mu$ is unaffected. (b) The equality $\\mathrm{E}[X_i\\mid Y_i=1]=1$ says that no respondent confesses falsely, which is Monotonicity (Assumption 1). Assumption 2 is No Liars and No Design Effects, and the sentence after the proof calls Assumption 1 Monotonicity. (c) The $\\mu$ before the coverage percentage does not belong to the sentence. (d) Display (4) conditions on $Y_j=0$ for $j=1,\\dots,k$, and the paragraph begins ``Suppose that exactly $k$ direct responses are zero''. The reordered responses are therefore the zeros. (e) The step equates the baseline means of the two arms among respondents with $Y_i=1$, which uses Treatment Independence (Assumption 3) as well as Assumptions 1 and 2. All three are hypotheses of Proposition 2.", "what_survives": "Nothing depends on these slips. The identity (1) of Lemma 1, the coverage statement of Corollary 1 and the inequality of Proposition 2 are unchanged, and so are the estimator and the reported analyses. Other steps of the proofs in Appendix B are the subject of \\fid{P019-E003}."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E008", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Table 5, Study A (Directs First): Placebo Test II, with columns for the estimate, SE, p-value and n, and five rows from Nuclear power to CNN. A red box marks the header of the estimate column, which is printed as beta-hat. The estimates below it are 0.066, -0.000, 0.016, -0.030 and -0.056, each with n equal to 500.", "caption": "Table 5, p. 56 (PDF p. 14). The marked header names the statistic of Placebo Test I; the entries below it are the estimates $\\hat\\delta$ of Placebo Test II.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009.png"}, {"alt": "Crop of supplement Tables A3 and A4. Both captions read 'Study A (Direct First): Placebo Test I'. Red boxes mark the words 'Placebo Test I' in the caption of Table A4 and the beta-hat header of its estimate column. Table A3 has estimates near 1 and n between 31 and 327; Table A4 has estimates near 0 and n equal to 506 in every row.", "caption": "Supplement, p. 6. Table A4 repeats the caption “Placebo Test I” of Table A3 above it, and its estimate column is headed $\\hat\\beta$; its entries, all with $n=506$, are the differences of Placebo Test II.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009-c.png"}, {"alt": "Crop of a paragraph of Section 5.3. A red box marks the words 'direction question' in the sentence 'In Study B, subjects were exposed to either a treatment or a control list before answering the direction question.' The next sentence refers to answers to the direct questions.", "caption": "Section 5.3, p. 57 (PDF p. 15). The marked words should read “direct question”.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009-b.png"}, {"alt": "Crop of the first paragraph of supplement Appendix D. A red box marks the citation 'Dalgaard 2008, p. 159' in the sentence 'Following the implementation in R (see accompanying text: Dalgaard 2008, p. 159), the power of the test for N = 100, 200, 500, and 1000 is shown in Figure A2 below.'", "caption": "Supplement Appendix D, p. 3. The marked citation has no entry in the reference list of the article, and the supplement has no reference list.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009-d.png"}], "formal_release": null, "formal_statements": [], "headline": "The four tables of Placebo Test II head their estimate column $\\hat\\beta$ where $\\hat\\delta$ is meant, supplement Table A4 is captioned Placebo Test I where it reports Placebo Test II, “direction question” should read “direct question”, and the citation of Dalgaard (2008) has no reference entry; every number in these tables agrees with a recalculation from the replication files.", "id": "P019-E009", "location": "Table 5, p. 56 (PDF p. 14), and Table 8, p. 58 (PDF p. 16), estimate column; Section 5.3, p. 57 (PDF p. 15); supplement Tables A4 and A7, pp. 6–7, and Appendix D, p. 3; analysis scripts in the replication files", "pdf_page": 14, "printed_page": "56", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item Table 5, p.~56: ``Table 5. Study A (Directs First): Placebo Test II'', with the estimate column headed $\\hl{\\hat\\beta}$. Table 8, p.~58, and supplement Tables A4 and A7 carry the same header.\n\\item Supplement, p.~6: ``Table A4: Study A (Direct First): \\hl{Placebo Test I}''. The table above it is ``Table A3: Study A (Direct First): Placebo Test I''.\n\\item Section 5.3, p.~57: ``In Study B, subjects were exposed to either a treatment or a control list before answering the \\hl{direction} question.''\n\\item Supplement Appendix D, p.~3: ``Following the implementation in R (see accompanying text: \\hl{Dalgaard 2008, p. 159}), the power of the test for $N = 100$, $200$, $500$, and $1000$ is shown in Figure A2 below.''\n\\item Replication files: the scripts for the main and the replication experiment label the p-value column of Placebo Test I ``Pr($\\beta\\hl{=0}$)'', typed as a formula in LaTeX notation, in four places.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Tables 5, 8, A4 and A7: head the estimate column $\\hl{\\hat\\delta}$.\n\\item ``Table A4: Study A (Direct First): \\hl{Placebo Test II}''.\n\\item ``In Study B, subjects were exposed to either a treatment or a control list before answering the \\hl{direct} question.''\n\\item Add a complete reference entry for Dalgaard (2008), after checking which work is meant and the page cited.\n\\item In the four places in the scripts: \\texttt{colnames(placebo\\_I\\_table) <- c(\"Estimate\", \"SE\", \"p-value (H0: beta = 1)\", \"N\")}.\n\\end{itemize}", "evidence": "The evidence is a check of the pages and a recalculation from the replication files. All 80 entries of the four tables (estimates, standard errors, p-values and sample sizes) agree with the calculation of Placebo Test II at the printed precision, and each of the 20 estimates differs from the corresponding $\\hat\\beta$. For nuclear power in Study A, for example, $\\hat\\delta=276/4165=0.066$, the first entry of Table 5, while $\\hat\\beta=28321/26871=1.054$, the first entry of Table 4. I could not determine from the article, the supplement or the replication files which work the Dalgaard citation refers to, or check its page number.", "problem": "Sections 4.1 and 4.2 define two statistics. Placebo Test I uses $\\hat\\beta$, the difference in mean list counts between treatment and control among those who answer ``Yes'' to the direct question, with null value $\\beta=1$. Placebo Test II uses $\\hat\\delta$, the difference between treatment and control in the proportion who answer ``Yes'', with null value $\\delta=0$. Tables 5, 8, A4 and A7 report Placebo Test II and head the estimate column with the symbol of Placebo Test I. Table A4 also carries the caption of Table A3. The word ``direction'' is a slip for ``direct''. The citation of Dalgaard (2008) has no entry in the reference list on pp.~65--66, and the supplement has no reference list. In the scripts the label of the p-value column of Placebo Test I names the null $\\beta=0$, while the function computes the two-sided p-value for $\\beta=1$, as the paper states. Further, the label is written with a single backslash before \\texttt{beta}, which R reads together with the letter b as a backspace character.", "what_survives": "Every estimate, standard error, p-value and sample size in the placebo tables stands, and so do the conclusions drawn from them, including the significant results of Placebo Test II for nuclear power and CNN in Study B. The corrections change headers, one caption, one word, one reference and a label in the code."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E009", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Figure 1, a bar chart of the direct, standard list and combined estimates of prevalence for five topics, with interval whiskers, on a vertical axis from 0.0 to 1.0. A red box marks the foot of the standard list and combined bars for Spanish-speaking, where the two whiskers run into the zero axis without the horizontal cap that ends every other whisker.", "caption": "Figure 1, p. 55 (PDF p. 13). The marked whiskers, for the standard list and combined estimates of Spanish speaking, end at the zero axis with no lower cap; the computed lower limits are $-0.121$ and $-0.102$.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E010.png"}, {"alt": "Crop of Figure 2, a bar chart of the direct, standard list and combined estimates of prevalence for five topics in Study B, with interval whiskers, on a vertical axis from 0.0 to 1.0. Two red boxes mark the feet of the standard list and combined bars for Spanish-speaking and for Muslim Teachers, where four whiskers run into the zero axis without a lower cap.", "caption": "Figure 2, p. 57 (PDF p. 15). The four marked whiskers, for the standard list and combined estimates of Spanish speaking and of Muslim teachers, end at the zero axis with no lower cap.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E010-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Six lower confidence limits in Figures 1 and 2 are negative, and the figures cut them off at the zero axis without saying so; the intervals are not truncated in the calculation and follow from the estimates and standard errors of Tables 3 and 6.", "id": "P019-E010", "location": "Figure 1, p. 55 (PDF p. 13), and Figure 2, p. 57 (PDF p. 15): the lower ends of the interval whiskers at the zero axis", "pdf_page": 13, "printed_page": "55", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Figure 1. Study A (Directs First): Three Estimates of Prevalence.''\n\n``Figure 2. Study B (Lists First): Three Estimates of Prevalence.''\n\nBoth figures have a vertical axis labelled ``Prevalence'' with ticks from 0.0 to 1.0. The whiskers of the standard list and combined estimates for Spanish speaking in Figure 1, and for Spanish speaking and Muslim teachers in Figure 2, \\hl{run down to the zero axis and have no lower cap}.", "corrected": "``Figure 1. Study A (Directs First): Three Estimates of Prevalence. \\hl{Lower Wald confidence limits below zero are clipped at the plotting boundary.}''\n\n``Figure 2. Study B (Lists First): Three Estimates of Prevalence. \\hl{Lower Wald confidence limits below zero are clipped at the plotting boundary.}''\n\nAlternatively, the vertical axis of both figures can be extended below zero so that the complete intervals are shown.", "intended_meaning": "The figures show each estimate with a 95 percent interval, the estimate plus or minus 1.96 standard errors, and six of these intervals have a negative lower limit. The plotting code in the replication files draws the full interval on an axis limited to the range from 0 to 1, and the plotting region cuts the whisker off at zero. The missing lower cap is the only sign of this, and the captions and the text do not mention it. Nothing printed is false, and the intervals are not truncated in the calculation. Recalculating them from the replication files, I find the limits below.\n\\begin{audittable}{llrrrr}{The six intervals whose lower limit lies below the zero axis.}\nStudy and topic & Estimator & Estimate & SE & Lower limit & Upper limit \\\\\nA, Spanish speaking & Standard list & 0.035 & 0.079 & $-0.1207$ & 0.190 \\\\\nA, Spanish speaking & Combined & 0.042 & 0.074 & $-0.1017$ & 0.187 \\\\\nB, Spanish speaking & Standard list & 0.144 & 0.083 & $-0.0192$ & 0.307 \\\\\nB, Spanish speaking & Combined & 0.149 & 0.076 & $-0.0006$ & 0.298 \\\\\nB, Muslim teachers & Standard list & 0.107 & 0.083 & $-0.0554$ & 0.268 \\\\\nB, Muslim teachers & Combined & 0.123 & 0.074 & $-0.0218$ & 0.268 \\\\\n\\end{audittable}\nSince the estimates and standard errors agree with Tables 3 and 6, the limits can be recovered from those tables to within rounding. The fourth limit is so close to zero that the rounded entries of Table 6 give a value just above zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E010", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of a paragraph of Section 5.1. A red box, which runs over two lines, marks the words '(with the exception of the diagonal)' in the sentence 'Consistent with our randomization procedure, each cell in the table (with the exception of the diagonal) contains approximately one-quarter of the subjects.' The sentence before it says that all randomizations used Bernoulli assignment with probability 0.5.", "caption": "Section 5.1, p. 53 (PDF p. 11), first paragraph below Table 2. The marked words are the only exception that the sentence makes to its statement about the cells of Table 1.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E011.png"}, {"alt": "Crop of Table 1, Number of Subjects in Each Treatment Condition, a triangular table that crosses Study A or B and the treatment and control conditions of Lists 1 to 5. Twelve red boxes mark the zeros printed in the six two-by-two blocks that cross a factor with itself, for example 500, 0 over 0, 514 for Study A or B and 517, 0 over 0, 497 for List 1. Cells crossing distinct factors hold counts between 219 and 278.", "caption": "Table 1, p. 52 (PDF p. 10). The twelve marked zeros lie beside the diagonal, in the blocks that cross a factor with itself, and the printed exception does not cover them.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E011-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The statement that each cell of Table 1 outside the diagonal holds about one-quarter of the subjects is meant for the cells that cross two different randomized factors; the cells that pair the two levels of one factor are zero by construction.", "id": "P019-E011", "location": "Section 5.1, p. 53 (PDF p. 11), first paragraph below Table 2; the sentence describes Table 1, p. 52 (PDF p. 10)", "pdf_page": 11, "printed_page": "53", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``All randomizations used Bernoulli random assignment with equal probability 0.5. Consistent with our randomization procedure, each cell in the table \\hl{(with the exception of the diagonal)} contains approximately one-quarter of the subjects.''\n\nTable 1, p.~52, prints the block for Study A or B as $500,\\ 0$ over $0,\\ 514$, and the block for List 1 as $517,\\ 0$ over $0,\\ 497$.", "corrected": "``Consistent with our randomization procedure, each cell \\hl{crossing levels of distinct randomized factors} contains approximately one-quarter of the subjects\\hl{; mutually exclusive levels of the same factor have structural zeros}.''", "intended_meaning": "Table 1 crosses six randomized factors: the study (A or B) and the treatment or control condition of each of the five lists. A cell that crosses levels of two different factors holds about one-quarter of the $n=1{,}014$ subjects; these counts range from 219 to 278. This is the intended statement, and it is correct. A block that crosses a factor with itself holds the two marginal counts on its diagonal and two zeros beside it, since no subject is in both levels of one factor. The printed exception does not cover these twelve zeros, unless ``the diagonal'' is read as the six diagonal blocks. All 84 counts of Table 1 agree with a recalculation from the replication files."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E011", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Proposition 3. A red box marks 'Assumptions 1-4' in 'Under the null hypothesis that Assumptions 1-4 (Monotonicity, No Liars, No Design Effects, and Treatment Independence) hold, beta = 1.' The next sentence begins 'For large n, if Assumption 4 (Non-Degenerate Distributions) holds', and is followed by the p-value formula, 2 Phi of minus the absolute value of beta-hat minus 1 over the square root of the estimated variance of beta-hat.", "caption": "Proposition 3, p. 50 (PDF p. 8). The marked range is followed by the names of Assumptions 1–3, and Assumption 4 is invoked separately in the next sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N001.png"}, {"alt": "Crop of the paragraph that begins 'The results of Placebo Test I for Study B are presented in table 7.' Two red boxes mark 'As-' at the end of one line and 'sumptions 1-4' at the start of the next, in the words 'should be equal to 1, under Assumptions 1-4 (Monotonicity, No Liars, No Design Effects, and Treatment Independence).'", "caption": "Section 5.3, paragraph on Table 7, p. 56 (PDF p. 14). The same range, broken across two lines.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T12", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T12"}, {"id": "P019-T21", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T21"}], "headline": "“Assumptions 1–4” is printed twice where the four names in parentheses are those of Assumptions 1–3.", "id": "P019-N001", "location": "Proposition 3, p. 50 (PDF p. 8); Section 5.3, paragraph on Table 7, p. 56 (PDF p. 14)", "pdf_page": 8, "printed_page": "50", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P019-T12", "P019-T21"], "text": {"as_printed": "p.~50: ``Proposition 3. Under the null hypothesis that \\hl{Assumptions 1--4} (Monotonicity, No Liars, No Design Effects, and Treatment Independence) hold, $\\beta=1$. For large $n$, if Assumption 4 (Non-Degenerate Distributions) holds, then a two-sided p-value is given by\n\\[\n2\\Phi\\left(\\frac{-|\\hat\\beta-1|}{\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\beta]}}\\right),\n\\]\nwhere $\\Phi(\\cdot)$ is the normal CDF.''\n\np.~56: ``Among the subgroup of respondents who answer ``Yes'' to the direct question, the list experiment difference-in-means estimate $\\beta$ should be equal to 1, under \\hl{Assumptions 1--4} (Monotonicity, No Liars, No Design Effects, and Treatment Independence).''", "corrected": "p.~50: ``Proposition 3. Under the null hypothesis that \\hl{Assumptions 1--3} (Monotonicity, No Liars, No Design Effects, and Treatment Independence) hold, \\hl{provided $\\Pr[Z_i=z,Y_i=1]>0$ for $z=0,1$}, $\\beta=1$. For large $n$, if Assumption 4 (Non-Degenerate Distributions) holds, then a two-sided p-value is given by\n\\[\n2\\Phi\\left(\\frac{-|\\hat\\beta-1|}{\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\beta]}}\\right),\n\\]\nwhere $\\Phi(\\cdot)$ is the normal CDF.''\n\np.~56: ``\\ldots\\ should be equal to 1, under \\hl{Assumptions 1--3} (Monotonicity, No Liars, No Design Effects, and Treatment Independence).''", "evidence": "The evidence is a check of the two passages against the definitions of the assumptions on pp.~46--47 and against p.~2 of the supplement, and a derivation of $\\beta=1$ from Assumptions 1--3. Among respondents with $Y_i=1$, Assumption 1 implies that $X_i=1$, Assumption 2 then gives $V_i=W_i+Z_i$, and by Assumption 3 the mean of $W_i$ is the same in the two arms.", "problem": "Monotonicity is Assumption 1, No Liars and No Design Effects together are Assumption 2, and Treatment Independence is Assumption 3. Assumption 4 is Non-Degenerate Distributions, which the second sentence of Proposition 3 invokes separately. The range that matches the four names is 1--3. The first paragraph of Section 4.1 lists the same four names, and Appendix C of the supplement calls the test ``a joint test of Assumptions 1-3''. The printed implication is not false, since $\\beta=1$ also holds under the stronger hypothesis. The misprint is in the label of the null hypothesis.", "what_survives": "Nothing depends on the range. The identity $\\beta=1$ and the p-value for large samples are unchanged. Assumption 4 is still needed, as the proposition says, so that $\\beta$ is defined and the normal approximation applies."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-N001", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of a paragraph of Section 5.3. A red box, which runs over two lines, marks the clause 'a joint test via Fisher's method is insignificant as well'. The preceding sentence says that the list experiment difference-in-means estimate beta should be equal to 1 under Assumptions 1 to 4.", "caption": "Section 5.3, p. 56 (PDF p. 14). The marked clause reports the joint test; the sentence before it names the assumptions, each of which concerns one list and its direct question.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N003.png"}, {"alt": "Crop of Table 7, Study B (Lists First): Placebo Test I, with columns for the estimate beta-hat, SE, p-value and n, and five rows from Nuclear power to CNN. A red box marks the p-value column: 0.294, 0.339, 0.309, 0.293 and 0.256.", "caption": "Table 7, p. 57 (PDF p. 15). The marked column holds the five p-values that Fisher’s method combines.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T9", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T9"}, {"id": "P019-T10", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T10"}], "headline": "The usual chi-square calibration of Fisher’s joint test of the five placebo tests of Study B assumes that the five placebo statistics are asymptotically independent; Assumptions 1–4, which are stated list by list, do not imply this, and the premise should be stated. The joint null is not rejected by either Fisher’s method with this premise or a joint Wald test that allows dependence.", "id": "P019-N003", "location": "Section 5.3, p. 56 (PDF p. 14), paragraph beginning “The results of Placebo Test I for Study B”; Table 7, p. 57 (PDF p. 15)", "pdf_page": 14, "printed_page": "56", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P019-T10", "P019-T9"], "text": {"as_printed": "``Among the subgroup of respondents who answer ``Yes'' to the direct question, the list experiment difference-in-means estimate $\\beta$ should be equal to 1, under Assumptions 1--4 (Monotonicity, No Liars, No Design Effects, and Treatment Independence). None of the values of $\\beta$ are statistically significantly different from 1 using the placebo test, and \\hl{a joint test via Fisher's method is insignificant as well}.''\n\nTable 7, p.~57, gives the five p-values 0.294, 0.339, 0.309, 0.293 and 0.256.", "corrected": "``None of the values of $\\beta$ are statistically significantly different from 1 using the placebo test. \\hl{Assuming that the five placebo statistics are asymptotically independent}, Fisher's method gives a joint statistic of \\hl{12.143 on 10 degrees of freedom} ($\\hl{p=0.276}$) and does not reject the common null.''\n\nThe statement behind this sentence, with its hypotheses, is the following. Suppose that the respondents are independent and identically distributed, that Assumptions 1--4 hold for each of the five lists, that the assignments $Z_1,\\dots,Z_5$ are mutually independent, and that $(Z_1,\\dots,Z_5)$ is independent of $(W_j,X_j,Y_j)_{j=1}^{5}$. Then $T=-2\\sum_{j=1}^{5}\\log p_j$ converges in distribution to $\\chi^2_{10}$.\n\nWithout the added premise, the sentence can report the joint Wald test instead: ``\\ldots and a joint Wald test of $\\beta_1=\\dots=\\beta_5=1$ that allows covariance between the five statistics does not reject ($W=5.147$ on 5 degrees of freedom, $p=0.398$).'' This test requires independent and identically distributed respondents and a nonsingular limiting covariance matrix.", "evidence": "The evidence is a derivation, an exact counterexample and a recalculation.\n\n\\emph{Derivation.} I index the lists by $j$ and write $q_j=\\Pr(Y_j=1)$, $\\gamma_j=\\Pr(Z_j=1)$ and $m_j=E[W_j\\mid Y_j=1]$. Under Assumptions 1--4 for list $j$, I have $\\beta_j=1$ and\n\\[\n\\sqrt n\\,(\\hat\\beta_j-1)=n^{-1/2}\\sum_{i=1}^{n}\\psi_{ij}+o_p(1),\\qquad\n\\psi_j=\\frac{Y_j}{q_j}\\cdot\\frac{Z_j-\\gamma_j}{\\gamma_j(1-\\gamma_j)}\\cdot(W_j-m_j).\n\\]\nThe five statistics are jointly asymptotically normal with covariances $E[\\psi_j\\psi_k]$. Suppose that $Z_1,\\dots,Z_5$ are mutually independent and that $(Z_1,\\dots,Z_5)$ is independent of $L=(W_j,X_j,Y_j)_{j=1}^{5}$. For $j\\neq k$, the conditional expectation $E[\\psi_j\\psi_k\\mid L]$ contains the factor $E[Z_j-\\gamma_j]\\,E[Z_k-\\gamma_k]=0$. Then the limiting covariance matrix is diagonal, the five p-values are asymptotically independent and uniform, and $T$ converges in distribution to $\\chi^2_{10}$.\n\n\\emph{Counterexample.} For each respondent, let $D\\in\\{0,1\\}$, $U\\in\\{-1,1\\}$ and $Z_1,\\dots,Z_5\\in\\{0,1\\}$ be independent, each taking its two values with equal probability. Let $A_j=2Z_j-1$ and define, with empty products equal to 1,\n\\[\nX_j=D,\\qquad W_j=\\tfrac32+\\tfrac12\\,U\\prod_{k<j}A_k,\\qquad Y_j=D\\,\\mathbf 1\\Bigl\\{\\prod_{k>j}A_k=1\\Bigr\\},\\qquad V_j=W_j+DZ_j .\n\\]\nThe baseline response to list $j$ depends on the assignments to the earlier lists only, and the direct answers, which follow all the lists, depend on the assignments to the later lists; the order of Study B permits both. Every list satisfies Assumptions 1--4 with respect to its own assignment, and $\\beta_j=1$ for every $j$. Enumerating the 128 equally likely outcomes in exact arithmetic, I find that the covariance matrix of $(\\psi_1,\\dots,\\psi_5)$ is\n\\[\n\\begin{pmatrix}4&2&2&2&2\\\\2&4&2&2&2\\\\2&2&4&2&2\\\\2&2&2&4&2\\\\2&2&2&2&2\\end{pmatrix}.\n\\]\nThe standardized statistics have correlation $1/2$ among the first four lists and $1/\\sqrt2$ between each of these and the fifth, while each individual p-value remains valid. In this limit the variance of $T$ exceeds 20, and a $\\chi^2_{10}$ variable has variance exactly 20. Hence $T$ does not converge to $\\chi^2_{10}$. The example is artificial. It shows that the premise does not follow from the stated assumptions, and it does not show that such dependence is present in the data.\n\n\\emph{Recalculation.} From the replication files, I recover the entries of Table 7. The two joint tests are as follows.\n\\begin{audittable}{lrrrl}{Joint tests of $\\beta_1=\\dots=\\beta_5=1$ in Study B, from the replication files.}\nTest & Statistic & Degrees of freedom & p-value & Requires \\\\\nFisher's method & 12.143 & 10 & 0.276 & asymptotically independent statistics \\\\\nJoint Wald test & 5.147 & 5 & 0.398 & nonsingular limiting covariance \\\\\n\\end{audittable}\nThe Wald statistic is $W=(\\hat\\beta-\\mathbf 1)^{\\top}\\widehat C^{-1}(\\hat\\beta-\\mathbf 1)$, where $\\widehat C$ is the covariance matrix of $\\hat\\beta$ estimated from the respondents' influence values, so that covariance between lists is allowed. The estimated correlations between lists lie between $-0.034$ and $0.069$.", "problem": "Fisher's method combines the five p-values $p_1,\\dots,p_5$ of Table 7 into $T=-2\\sum_{j=1}^{5}\\log p_j$ and refers $T$ to the chi-square distribution with 10 degrees of freedom. This reference distribution is correct when the five p-values are independent, which here requires the five placebo statistics $\\hat\\beta_j$ to be asymptotically independent. The paper does not state this premise, and Assumptions 1--4 do not supply it. Each assumption concerns one list experiment and its direct question. In particular, Treatment Independence makes $(W_i,X_i,Y_i)$ for a topic independent of the assignment $Z_i$ to the list on that topic, and it says nothing about the assignments to the other four lists. In Study B each respondent answered all five lists and then the five direct questions. Therefore the assignment to one list may affect the baseline response to a later list, or a later direct answer, without violating any of the assumptions as stated; Section 5.3 itself raises the possibility of such priming. Note that the calibration does not fail merely because the same respondents enter all five tests. If the five assignments are mutually independent and jointly independent of the full vector $(W_j,X_j,Y_j)_{j=1}^{5}$, then the five statistics are asymptotically independent. The paper states no such joint condition.", "what_survives": "The entries of Table 7, the five individual tests, the value of Fisher's statistic and its nonrejection all stand, and a joint test that allows dependence between the lists does not reject either. No other result of the paper uses this step. The joint p-value from Fisher's method rests on a premise beyond Assumptions 1--4, and the paper's own discussion of priming and order effects shows that the design does not guarantee this premise."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-N003", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the heading 'Appendix C: Placebo Test I Power Analysis' and the first seven lines below it. A red box marks 'it must either be because' in the sentence 'If this estimate is different from 1.0, it must either be because a) some of those who answer Yes are falsely confessing (thereby violating monotonicity) or b) the standard list experiment assumptions of No Liars and No Design Effects are not met.' The first sentence of the paragraph calls the test 'a joint test of Assumptions 1-3 (Monotonicity, No Liars, No Design Effects, and Treatment Independence)'.", "caption": "Online supplement, Appendix C, first paragraph, p. 2. The marked words present the two causes that follow as the only ones.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N006.png"}], "formal_release": null, "formal_statements": [{"id": "P019-T7", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T7"}, {"id": "P019-T8", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T8"}], "headline": "A list difference other than 1 among respondents who answer “Yes” can also come from a failure of Treatment Independence, which the supplement’s list of two causes leaves out.", "id": "P019-N006", "location": "Online supplement, Appendix C, first paragraph, third sentence, p. 2 (supplement PDF p. 2). The related passage in the main text is Section 4.1 and Proposition 3, p. 50 (PDF p. 8)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P019-T7", "P019-T8"], "text": {"as_printed": "``The placebo test proposed in Section 4 is a joint test of Assumptions 1-3 (Monotonicity, No Liars, No Design Effects, and Treatment Independence). In brief, the test considers whether the conventional list experimental estimate appears to be significantly different from 1.0 among the subset of subjects who answer ``Yes'' to the direct question. If this estimate is different from 1.0, \\hl{it must either be because} a) some of those who answer ``Yes'' are falsely confessing (thereby violating monotonicity) or b) the standard list experiment assumptions of No Liars and No Design Effects are not met.''", "corrected": "``If \\hl{the population difference-in-means among subjects who answer ``Yes''} is different from 1.0, \\hl{at least one of Assumptions 1-3 must fail}: some subjects falsely confess, No Liars or No Design Effects fails, \\hl{or Treatment Independence fails. A statistically significant sample departure is evidence against this joint null.}''\n\nFor a population difference, the two-cause explanation is also correct if it begins ``Provided Assumption 3 (Treatment Independence) holds''. That version covers less, and the paper reports likely violations of Assumption 3 in Study B.", "evidence": "The evidence is a derivation and an exact counterexample. Under Assumptions 1 and 2, every respondent with $Y_i=1$ has $X_i=1$ and $V_i=W_i+Z_i$, and I have\n\\[\n\\beta=1+\\mathrm{E}[W_i\\mid Z_i=1,Y_i=1]-\\mathrm{E}[W_i\\mid Z_i=0,Y_i=1].\n\\]\nAssumption 3 implies that the last two terms are equal, and without it they need not be. For the counterexample, let $B$, $C$, $X$ and $Z$ be independent Bernoulli variables with mean $1/2$, and set\n\\[\nW=B+C,\\qquad Y=X\\,\\mathbf{1}\\{C=Z\\},\\qquad V=W+XZ.\n\\]\nSince $Y\\le X$, Assumption 1 holds, and the three classes of respondents $(X,Y)=(0,0)$, $(1,0)$ and $(1,1)$ have probabilities $1/2$, $1/4$ and $1/4$. Assumption 2 holds by the definition of $V$. The treatment $Z$ is independent of $(B,C,X)$, but the direct answer depends on $Z$, and Assumption 3 fails: the event $(W,X,Y)=(0,1,0)$ has probability $1/16$ and never occurs together with $Z=0$. The list response $V$ has the following moments in the four cells.\n\\begin{audittable}{rrrrr}{The counterexample: probability of each cell of $(Z,Y)$, and the mean and variance of the list response $V$ in the cell.}\n$Z$ & $Y$ & Probability & $\\mathrm{E}[V\\mid Z,Y]$ & $\\operatorname{Var}[V\\mid Z,Y]$ \\\\\n\\hline\n0 & 0 & $3/8$ & $7/6$ & $17/36$ \\\\\n0 & 1 & $1/8$ & $1/2$ & $1/4$ \\\\\n1 & 0 & $3/8$ & $7/6$ & $17/36$ \\\\\n1 & 1 & $1/8$ & $5/2$ & $1/4$ \\\\\n\\end{audittable}\nAssumption 4 holds, since all four variances are positive, $\\operatorname{Var}[Y]=3/16$ and $\\operatorname{Var}[Z]=1/4$. From the two rows with $Y=1$, I obtain $\\beta=5/2-1/2=2$. The share of ``Yes'' answers is $1/4$ in each arm, so that $\\delta=0$. Equal response rates in the two arms therefore do not imply Assumption 3.", "problem": "The sentence presents two causes as the only ones: false confessions, which violate Assumption 1, or a failure of No Liars or No Design Effects, which together are Assumption 2. The joint null of the test also contains Treatment Independence (Assumption 3), as the first sentence of the paragraph and Proposition 3 say. A failure of Assumption 3 alone can make the population difference $\\beta$ differ from 1. This happens when the answer to the direct question depends on the list treatment. Then the respondents who answer ``Yes'' in the two arms differ in their baseline list responses, although no one confesses falsely, no one lies on the list, and the responses to the non-sensitive items do not change. Note that an effect of the treatment on the direct answer is not a design effect in the sense of Assumption 2; the paper places it under Assumption 3 (p.~47). Here I read ``estimate'' as the population quantity $\\beta$, so that sampling variation is not at issue.", "what_survives": "Section 4.1 and Proposition 3 include Treatment Independence in the null, and the implication $\\beta=1$ under the stated assumptions is unaffected. In the simulations of Appendix C, the direct answers are fixed before the treatment is assigned at random. Assumption 3 holds there, and the omitted cause does not arise. The power values were not computed again for this finding. The application already separates Study A, in which the direct questions come first, from Study B, for which the paper discusses failures of Treatment Independence. What changes is the diagnosis. A rejection by Placebo Test I cannot in general be attributed only to false confessions or to the list responses."}, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-N006", "work": {"arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/"}, "work_id": "P019"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the Introduction paragraph on efficiency. A red box marks the words 'are guaranteed to be relatively inefficient' in the sentence 'Estimators that are asymptotically biased are guaranteed to be relatively inefficient for a sufficiently large sample size'. The lines above read 'researchers often consider efficiency (typically mean square error) to be of paramount importance'.", "caption": "Introduction, p. 40 (PDF p. 2). The marked words state the guarantee; two sentences earlier the paragraph takes efficiency to be “typically mean square error”.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E001.png"}, {"alt": "Crop of the first paragraph of Section 3. A red box marks the words 'hence, asymptotically inefficient' at the end of the sentence 'Third, the derivations will help us to identify conditions under which common estimators are not consistent and, hence, asymptotically inefficient.'", "caption": "Section 3, first paragraph, p. 42 (PDF p. 4). The marked words draw the same inference from inconsistency.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T1", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T1"}, {"id": "P020-T2", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T2"}], "headline": "Persistent bias does not guarantee that an estimator eventually has the larger mean squared error; the guarantee holds against a comparator whose mean squared error tends to zero.", "id": "P020-E001", "location": "Introduction, paragraph beginning “Bias is not the only statistical property”, p. 40 (PDF p. 2); Section 3, first paragraph, p. 42 (PDF p. 4)", "pdf_page": 2, "printed_page": "40", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P020-T1", "P020-T2"], "text": {"as_printed": "p.~40: ``Estimators that are asymptotically biased \\hl{are guaranteed to be relatively inefficient} for a sufficiently large sample size, and we provide an empirical example where bias is critical in undermining the relative efficiency of common estimators.''\n\np.~42: ``Third, the derivations will help us to identify conditions under which common estimators are not consistent and, \\hl{hence, asymptotically inefficient}.''", "corrected": "p.~40: ``\\hl{Persistent asymptotic bias contributes a nonvanishing squared-bias term to mean squared error and can undermine relative efficiency. It guarantees eventual mean-squared-error inferiority to a specified comparator when that comparator's mean squared error tends to zero}.'' The clause on the empirical example is unchanged.\n\np.~42: ``\\ldots\\ conditions under which common estimators are not consistent; \\hl{whether they are relatively inefficient depends on comparative mean squared errors}.''\n\nIn full: let $T_h$ and $U_h$ be estimators of $\\Delta_h$ along a sequence of finite populations indexed by $h$, each with finite second moment, such that the bias of $T_h$ converges to a limit $b\\neq0$ and $\\operatorname{MSE}(U_h)\\to0$ as $h\\to\\infty$. Then $\\operatorname{MSE}(U_h)<\\operatorname{MSE}(T_h)$ for all sufficiently large $h$.", "evidence": "The evidence is an exact counterexample, computed by enumerating every assignment in rational arithmetic. Let $M=3$, $m_t=1$ and $m_c=2$, and let the clusters have sizes $1$, $2$ and $4$. The control outcomes of the individuals are $(0)$, $(1,0)$ and $(1,0,0,0)$ in the three clusters, and every treatment outcome is $0$. Then $N=7$ and $\\Delta=-2/7$. Each cluster is the treated one with probability $1/3$. Enumerating the three assignments, I obtain the following values of the estimators.\n\\begin{audittable}{lrrr}{The three-cluster example: the estimate under each of the three equally likely assignments, and the bias and mean squared error of each estimator. The average treatment effect is $-2/7$.}\nTreated cluster & Difference in means & Horvitz--Thompson & Des Raj, $k=2/21$ \\\\\n\\hline\n1 & $-1/3$ & $-3/7$ & $-17/49$ \\\\\n2 & $-1/5$ & $-3/14$ & $-19/98$ \\\\\n3 & $-1/3$ & $-3/14$ & $-31/98$ \\\\\n\\hline\nBias & $-1/315$ & $0$ & $0$ \\\\\nMean squared error & $131/33075\\approx0.00396$ & $1/98\\approx0.01020$ & $3/686\\approx0.00437$ \\\\\n\\end{audittable}\nFor a fixed $k$ the Des Raj estimator of Section 5.1 is unbiased, and I find that its mean squared error is $3/686+(9/14)(k-2/21)^2$. This implies that no fixed $k$ does better than $3/686$. The choice $k=0$ is the Horvitz--Thompson estimator. Now suppose that each individual is copied $h$ times within its cluster, as in the construction on p.~47. The cluster sizes and totals are multiplied by $h$, and $N=7h$. Every entry of the table is then the same for every $h$. I conclude that the difference in means has bias $-1/315$ at every sample size. It is inconsistent, since its error is at least $1/21$ in absolute value under every assignment. Its mean squared error is nevertheless smaller than that of the Horvitz--Thompson estimator and of the Des Raj estimator with any fixed $k$.\n\nThe corrected statement holds because mean squared error is the sum of the variance and the squared bias. Suppose that the bias $b_h$ of $T_h$ tends to $b\\neq0$. Then $\\operatorname{MSE}(T_h)\\ge b_h^2>b^2/4$ for all large $h$. If also $\\operatorname{MSE}(U_h)\\to0$, then $\\operatorname{MSE}(U_h)<b^2/4$ for all large $h$.", "problem": "The paragraph on p.~40 measures efficiency by mean squared error. The marked sentence says that an estimator whose bias does not vanish must, in a large enough sample, have a larger mean squared error than the estimator with which it is compared, and the sentence on p.~42 draws the same conclusion from inconsistency. One sufficient condition is that the mean squared error of that comparator tend to zero, which neither sentence assumes. If asymptotic efficiency is defined so as to require consistency, the phrase on p.~42 holds by definition, but that reading gives no ranking by mean squared error and does not support the sentence on p.~40. The asymptotic regimes of the paper include the one in which the number of clusters is fixed and the clusters grow (p.~40 and Section 3.3, p.~47). In that regime the variance of an unbiased estimator need not vanish, and a biased estimator can keep the smaller mean squared error at every sample size.", "what_survives": "I found no use of the guarantee in the derivations checked. The bias formula and the replication arguments of Section 3 are unaffected, and so are the results on the Horvitz--Thompson estimator in Section 4. The same holds for the results of Section 5 on the Des Raj estimator: that it is unbiased, and the explicit comparison of its variance with that of the Horvitz--Thompson estimator. The rankings by root mean squared error in Sections 6.4 and 6.5 are computed for the application and do not rest on these sentences. The motivation survives in the weaker form of the next sentence of the paper: bias cannot always be ignored when efficiency is the concern. The counterexample does not show that biased estimators are better in general."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E001", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of equation (8), which gives the bias as minus M over N times the difference of two covariance terms divided by m_t and m_c, and of the lines after it. A red box marks the words 'is biased.' at the end of the sentence 'if the size of the cluster is correlated with the potential outcomes in the cluster, the difference-in-means estimator is biased.' The next paragraph begins 'In some special cases, there will be no bias'.", "caption": "Equation (8) and the sentences after it, p. 45 (PDF p. 7). The marked words state as certain a bias that equation (8) makes possible.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E002.png"}, {"alt": "Crop of the first lines of the page. A red box marks the words 'the bias arises whenever' in the sentence 'As our derivation will show, the bias arises whenever outcomes are related to cluster size.'", "caption": "First lines of p. 44 (PDF p. 6). The marked words announce the same condition before the derivation.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T11", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T11"}], "headline": "Read literally, two sentences say that the difference in means is biased whenever cluster size is related to outcomes; the meaning is that such a relation can bias it, and by equation (8) the terms of the two arms can cancel.", "id": "P020-E002", "location": "Section 3.2, the sentence after equation (8) and the paragraph that follows, p. 45 (PDF p. 7); first sentence of p. 44 (PDF p. 6)", "pdf_page": 7, "printed_page": "45", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P020-T11"], "text": {"as_printed": "p.~44: ``As our derivation will show, \\hl{the bias arises whenever} outcomes are related to cluster size.''\n\np.~45: ``Inspection of this term reveals that, if the size of the cluster is correlated with the potential outcomes in the cluster, the difference-in-means estimator \\hl{is biased}. Moreover, the presence of the terms $1/m_t$ and $1/m_c$ shows that the magnitude (and even the direction) of the bias can depend on the relative number of clusters allocated to treatment and control.\n\nIn some special cases, there will be no bias, such as when the cluster size does not vary or when there is no covariance between cluster size and outcomes.''", "corrected": "p.~44: ``\\ldots\\ \\hl{bias can arise when} outcomes are related to cluster size.''\n\np.~45: ``Inspection of this term reveals that, if the size of the cluster is correlated with the potential outcomes in the cluster, the difference-in-means estimator \\hl{can be biased}. \\hl{The terms of the two arms in equation (8) can cancel, and the estimator is unbiased exactly when they are equal.} Moreover, the presence of the terms $1/m_t$ and $1/m_c$ shows that the magnitude (and even the direction) of the bias can depend on the relative number of clusters allocated to treatment and control.\n\nIn some special cases, there will be no bias, such as when the cluster size does not vary or when \\hl{both covariances in equation (8) are zero}.''", "intended_meaning": "The passage warns that the difference in means is not unbiased in general when cluster size is related to outcomes. The warning is correct, and equation (8) and the paragraph on special cases state the exact condition. I write $R_1$ and $R_0$ for the mean outcomes of the treated and the control individuals under a given assignment, and $n_t$ and $n_c$ for the numbers of treated and control individuals. Then equation (8) reads\n\\[\n\\mathrm{E}[\\widehat{\\Delta_S}]-\\Delta=-\\frac{M}{N}\\left[\\frac{\\operatorname{Cov}(R_1,n_t)}{m_t}-\\frac{\\operatorname{Cov}(R_0,n_c)}{m_c}\\right],\n\\]\nwhich implies that the bias is zero exactly when the two terms in the bracket are equal. The words ``whenever'' and ``is biased'' claim more than this. For example, let $M=2$ and $m_t=m_c=1$, and suppose that the first cluster has one individual with $Y_0=Y_1=0$ and the second has two individuals with $Y_0=Y_1=1$. The outcomes are perfectly correlated with cluster size, and both covariances equal $1/4$. The estimator takes the values $-1$ and $1$ with probability $1/2$ each, and its expectation is $0=\\Delta$. In a second example the arms have unequal numbers of clusters ($M=3$, $m_t=1$, sizes $1$, $2$, $3$), and the effects are positive and vary with cluster size. There $\\Delta=44/45$, each arm mean has bias $-7/90$, and the difference in means is unbiased. The balanced design under the sharp null in Section 6.4, for which the paper reports no bias, is another case of cancellation.\n\nNote that equation (8) is correct as printed, including its sign, and that the estimator can be biased: with sizes $(1,2)$, control means $(0,1)$ and treatment means $(0,2)$ in the two clusters, its bias is $-1/6$. In the clause ``no covariance between cluster size and outcomes'', the covariances are the two in equation (8). No estimator, derivation or reported number depends on the stronger wording."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E002", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the logistic regression model P(Y_ij = 1) = 1 - (1 + exp(alpha + tau t_j + beta n_j + phi n_j t_j + sum of gamma_f Gamma_f)) to the power -1 and of the paragraph below it. Red boxes mark 'n_j is the cluster size,' and the sentence 'We estimate tau as 0.4, beta as 1.4 and phi as -0.9.'", "caption": "The logistic model of Section 6.2, p. 61 (PDF p. 23). The marked passages define $n_j$ as the cluster size and report the coefficients; with $n_j$ a count of individuals, $0.4-0.9\\,n_j$ is negative for every cluster.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E003.png"}, {"alt": "Crop of the last three lines of Section 6.2. A red box marks '= 0.007,' in 'Marginalizing over the imputation process, the ATE is in expectation the sum over i of (p_ti - p_ci) divided by N = 0.007, or 0.7 percentage points.'", "caption": "Last sentence of Section 6.2, p. 61 (PDF p. 23). The marked value is the positive average effect that the model is reported to give.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T3", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T3"}, {"id": "P020-T4", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T4"}], "headline": "With $n_j$ the number of individuals in a cluster, the printed coefficients $\\tau=0.4$ and $\\phi=-0.9$ make the modelled effect negative for every individual, against the reported average effect of $+0.007$; the predictor’s units and coefficients should be reconciled.", "historical_evidence_type": "lean_proof", "id": "P020-E003", "location": "Section 6.2, p. 61 (PDF p. 23): the logistic regression model, the sentence that reports the coefficients, and the last sentence of the section", "pdf_page": 23, "printed_page": "61", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P020-T3", "P020-T4"], "text": {"as_printed": "\\[\nP(Y_{ij}=1)=1-\\left(1+\\exp\\left(\\alpha+\\tau t_j+\\beta \\hl{n_j}+\\phi \\hl{n_j}t_j+\\sum_{f=1}^{F-1}\\gamma_f\\Gamma_f\\right)\\right)^{-1}\n\\]\n``where $t_j$ is a treatment indicator for cluster $j$, $\\hl{n_j}$ \\hl{is the cluster size}, $F$ is the number of blocks (in this case, 40), $\\Gamma_f$ is an indicator variable indicating whether cluster $j$ is in block $f$. The terms $\\alpha$, $\\tau$, $\\beta$, $\\phi$ and $\\gamma_f$ are coefficients estimated from the data using maximum likelihood methods. Note the coefficient $\\phi$ is responsible for the heterogeneous treatment effects. We estimate $\\tau$ as \\hl{0.4}, $\\beta$ as 1.4 and $\\phi$ as $\\hl{-0.9}$.''\n\n``Marginalizing over the imputation process, the ATE is in expectation\n\\[\n\\frac{\\sum_i(p_{ti}-p_{ci})}{N}\\hl{=0.007},\n\\]\nor 0.7 percentage points.''", "corrected": "The passage should state the units of the cluster-size predictor and give the coefficients in those units. If the predictor was cluster size in thousands of individuals, the passage reads:\n\\[\nP(Y_{ij}=1)=1-\\left(1+\\exp\\left(\\alpha+\\tau t_j+\\beta \\hl{s_j}+\\phi \\hl{s_j}t_j+\\sum_{f=1}^{F-1}\\gamma_f\\Gamma_f\\right)\\right)^{-1}\n\\]\n``where $t_j$ is a treatment indicator for cluster $j$, $\\hl{s_j=n_j/1000}$ \\hl{is the cluster size in thousands of individuals}, $F$ is the number of blocks (in this case, 40), $\\Gamma_f$ is an indicator variable indicating whether cluster $j$ is in block $f$.'' The coefficients are then as printed: $\\tau=0.4$, $\\beta=1.4$ and $\\phi=-0.9$.\n\nEquivalently, the passage can keep $n_j$ and report $\\beta$ as $0.0014$ and $\\phi$ as $-0.0009$. With this scale the difference $0.4-0.0009\\,n_j$ is positive for clusters of fewer than $4000/9\\approx444.4$ individuals and negative for larger ones, and the contradiction in sign disappears. The scale in thousands is an inference; the specification that was used has not been recovered. The reported value $0.007$ should be checked on the sample of 80 clusters.", "evidence": "The evidence is the derivation above, together with a Lean 4 proof of the same statement for the model as printed, which covers every finite nonempty population. Rounding does not change the sign: coefficients that round to $0.4$ and $-0.9$ are at most $0.45$ and $-0.85$, so that $\\tau+\\phi n_j\\le-0.40$ when $n_j\\ge1$. As a check of the scale only, I fitted the same model to all 85 clusters of the public file with cluster size in thousands. The fit gives $\\tau=0.43$, $\\beta=1.43$ and $\\phi=-0.99$, with an average of $p_{ti}-p_{ci}$ equal to $0.010$. This is not the fit to 80 clusters that the paper reports, and it does not reproduce $0.007$.", "problem": "The paper defines $n_j$ as the number of individuals in cluster $j$ (p.~44), and Section 6.2 does not rescale it. For an individual in cluster $j$, the difference between the linear predictors under treatment and under control is $\\tau+\\phi n_j=0.4-0.9\\,n_j$, which is at most $-0.5$ when $n_j\\ge1$. Since the printed probability is an increasing function of the linear predictor, I have $p_{ti}<p_{ci}$ for every individual. The average of $p_{ti}-p_{ci}$ is then negative for all values of $\\alpha$, $\\beta$ and $\\gamma_f$, and it cannot equal $0.007$. In the public file of the source study the clusters hold between 30 and 990 voters. A plausible explanation is that cluster size entered the fit in other units, such as thousands of voters, but the paper does not say so, and the scale cannot be recovered from the text.", "what_survives": "The results on the estimators and their variances in Sections 2--5 and the appendices do not use the fitted coefficients. The scenario under the sharp null in Section 6.4 and Figures 2 and 3 uses a separate imputation and is unaffected. Section 6.5 and Figures 4 and 5 use the scenario with heterogeneous effects. No result there is shown to change, but the scenario cannot be reproduced from the model as printed. The units that were used and the coefficients of the fit to 80 clusters are not recovered."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E003", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the end of Appendix B: equation (30), the sentence 'So the expected value of the estimator will be', equation (31) and the paragraph that follows. A red box marks the second argument of the second covariance in equation (31), n bar with subscript c j. The first covariance in the same equation has the same argument, unmarked.", "caption": "Appendix B, p. 71 (PDF p. 33), equations (30) and (31). The marked term in (31) repeats the control-arm mean $\\overline{n_{cj}}$ of the first covariance.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T5", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T5"}, {"id": "P020-T6", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T6"}], "headline": "In equation (31) the second covariance is with the mean cluster size in the treatment arm, $\\overline{n_{tj}}$, as in the same formula in Section 5.3 (p. 57).", "id": "P020-E005", "location": "Appendix B, equation (31), p. 71 (PDF p. 33)", "pdf_page": 33, "printed_page": "71", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P020-T5", "P020-T6"], "text": {"as_printed": "\\[\n\\mathrm{E}\\left[\\frac{\\widehat{Y^T_{1,R1}}-\\widehat{Y^T_{0,R1}}}{N}\\right]=\\Delta+\\frac{M}{N}\\left(\\operatorname{Cov}(\\hat k,\\overline{n_{cj}})-\\operatorname{Cov}(\\hat k,\\overline{n_{\\hl{c}j}})\\right).\\qquad(31)\n\\]", "corrected": "Let $M$ clusters of sizes $n_j$, with $N=\\sum_j n_j$, be completely randomized, $m_t$ to treatment and $m_c=M-m_t$ to control. Let $\\hat k$ be an estimator of $k$ computed from the data of the same experiment, so that it is constant within a randomization and can vary across randomizations. Let $\\overline{n_{tj}}$ and $\\overline{n_{cj}}$ be the mean sizes of the clusters in treatment and in control in a given randomization. Then the Des Raj difference estimator of equations (20) and (21), with $\\hat k$ in place of $k$, satisfies\n\\[\n\\mathrm{E}\\left[\\frac{\\widehat{Y^T_{1,R1}}-\\widehat{Y^T_{0,R1}}}{N}\\right]=\\Delta+\\frac{M}{N}\\left(\\operatorname{Cov}(\\hat k,\\overline{n_{cj}})-\\operatorname{Cov}(\\hat k,\\overline{n_{\\hl{t}j}})\\right).\\qquad(31)\n\\]", "evidence": "The evidence is a derivation and an exact example. Let $\\overline{n_{tj}}$ and $\\overline{n_{cj}}$ be the mean sizes of the clusters assigned to treatment and to control in a given randomization, so that $\\mathrm{E}[\\overline{n_{tj}}]=\\mathrm{E}[\\overline{n_{cj}}]=N/M$. The estimator $\\hat k$ is constant within a randomization. By equation (21) with $\\hat k$ in place of $k$, I have\n\\[\n\\widehat{Y^T_{1,R1}}=\\frac{M}{m_t}\\sum_{j\\in J_1}Y_j^T-M\\hat k\\Bigl(\\overline{n_{tj}}-\\frac{N}{M}\\Bigr).\n\\]\nTaking expectations, I obtain\n\\[\n\\mathrm{E}\\bigl[\\widehat{Y^T_{1,R1}}\\bigr]=Y_1^T-M\\bigl(\\mathrm{E}[\\hat k\\,\\overline{n_{tj}}]-\\mathrm{E}[\\hat k]\\,\\mathrm{E}[\\overline{n_{tj}}]\\bigr)=Y_1^T-M\\operatorname{Cov}(\\hat k,\\overline{n_{tj}}),\n\\]\nwhich is equation (29). By the same argument for the control arm, I have equation (30), $\\mathrm{E}\\bigl[\\widehat{Y^T_{0,R1}}\\bigr]=Y_0^T-M\\operatorname{Cov}(\\hat k,\\overline{n_{cj}})$. Subtracting (30) from (29) and dividing by $N$, I obtain the corrected (31), since $\\Delta=(Y_1^T-Y_0^T)/N$.\n\nFor the example, take $M=3$ clusters of sizes $(1,2,3)$, so that $N=6$, with one cluster treated. Let the cluster totals be $(0,0,0)$ under control and $(0,0,3)$ under treatment, so that $\\Delta=1/2$, and let $\\hat k$ be the least-squares slope of the observed cluster totals on $n_j$, with an intercept. Over the three assignments, $\\hat k$ takes the values $(0,0,3/2)$ and the estimator takes the values $(0,0,3/8)$. Its expectation is $1/8$, and its bias is $-3/8$. Here $\\operatorname{Cov}(\\hat k,\\overline{n_{cj}})=-1/4$ and $\\operatorname{Cov}(\\hat k,\\overline{n_{tj}})=1/2$, and the bias term of the corrected formula is $(3/6)(-1/4-1/2)=-3/8$. The printed formula gives $0$.", "problem": "As printed, the two covariances in (31) are the same, and the term that the next sentence calls the bias is identically zero. Subtracting equation (30) from equation (29), I obtain the treatment-arm mean $\\overline{n_{tj}}$ in the second covariance, and Section 5.3 prints the formula in that form on p.~57. The second subscript $c$ is a misprint.", "what_survives": "Equations (29) and (30), their derivation and the bias formula of Section 5.3 are correct as printed. The unbiasedness of the Des Raj estimator for a coefficient $k$ fixed before the analysis is not affected. Only the last display of Appendix B changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E005", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the last four lines of the page. A red box marks '2 sigma squared of n_j, must be positive,' in the sentence 'Since the second derivative with respect to k, 2 sigma squared of n_j, must be positive, we may set the first derivative equal to zero and solve for k, so that'.", "caption": "Section 5.3, last lines of p. 56 (PDF p. 18). The marked words assert that $2\\sigma^2(n_j)$ is positive, which holds when the cluster sizes are not all equal.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E007.png"}, {"alt": "Crop of the paragraph beginning 'The chosen value of k will reduce the variability of the Des Raj estimator'. Red boxes mark 'will have better precision than' and, on the next line, 'the HT estimator unless' in the sentence 'In other words, the Des Raj estimator will have better precision than the HT estimator unless the researcher picks a k with the wrong sign or chooses a k that is more than twice the magnitude of k_optim*.' The sentence before gives the intervals 0 < k < 2 k_optim* and 0 > k > 2 k_optim*.", "caption": "Section 5.3, p. 57 (PDF p. 19). The strict intervals are printed in the sentence before; the marked summary omits the boundary cases.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E007-b.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T14", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T14"}], "headline": "The formulas for the optimal $k$ presuppose that the cluster sizes are not all equal, $\\sigma^2(n_j)>0$, and the informal summary on p. 57 leaves out the cases $k=0$ and $k=2k_{\\mathit{optim}*}$, in which the two variances are equal.", "id": "P020-E007", "location": "Section 5.3: last paragraph of p. 56 (PDF p. 18), leading to equation (26); paragraph beginning “The chosen value of $k$”, p. 57 (PDF p. 19)", "pdf_page": 18, "printed_page": "56", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P020-T14"], "text": {"as_printed": "pp.~56--57: ``Since the second derivative with respect to $k$, $2\\sigma^2(n_j)$, \\hl{must be positive}, we may set the first derivative equal to zero and solve for $k$, so that'' and then equation (26),\n\\[\nk_{\\mathit{optim}_c}=\\frac{\\sigma(n_j,Y^T_{0j})}{\\sigma^2(n_j)}.\n\\]\n\np.~57: ``The chosen value of $k$ will reduce the variability of the Des Raj estimator, $\\widehat{\\Delta_{R1}}$, relative to the HT estimator when, for $k_{\\mathit{optim}*}>0$, $0<k<2k_{\\mathit{optim}*}$ and, for $k_{\\mathit{optim}*}<0$, $0>k>2k_{\\mathit{optim}*}$. In other words, the Des Raj estimator \\hl{will have better precision than the HT estimator unless} the researcher picks a $k$ with the wrong sign or chooses a $k$ that is more than twice the magnitude of $k_{\\mathit{optim}*}$.''", "corrected": "pp.~56--57: ``\\hl{Assume that the cluster sizes are not all equal, so that $\\sigma^2(n_j)>0$.} Since the second derivative with respect to $k$, $2\\sigma^2(n_j)$, \\hl{is then positive}, \\ldots\\ \\hl{If all clusters have the same size, the adjustment vanishes and every $k$ gives the HT estimator.}''\n\np.~57: ``In other words, the Des Raj estimator will have better precision than the HT estimator \\hl{when $k$ lies strictly between $0$ and $2k_{\\mathit{optim}*}$. The two estimators have the same precision at $k=0$ and at $k=2k_{\\mathit{optim}*}$, and when $k_{\\mathit{optim}*}=0$ no nonzero $k$ improves on the HT estimator.}''", "intended_meaning": "Section 5 concerns clusters of unequal size, and for these the formulas are correct. I write $\\mathrm{V}(k)$ for the variance of the Des Raj estimator with a fixed $k$, so that $\\mathrm{V}(0)$ is the variance of the Horvitz--Thompson estimator. Let $c=(M-m_c)/m_c$, $t=(M-m_t)/m_t$ and $v=(M-1)N^2/M^2$, as in Appendix C. By the expansion in Appendix C, I have\n\\[\nv\\,[\\mathrm{V}(k)-\\mathrm{V}(0)]=(c+t+2)\\,\\sigma^2(n_j)\\,k\\,(k-2k_{\\mathit{optim}*}),\\qquad k_{\\mathit{optim}*}=\\frac{m_t}{M}k_{\\mathit{optim}_c}+\\frac{m_c}{M}k_{\\mathit{optim}_t}.\n\\]\nWhen $\\sigma^2(n_j)>0$ the variance has a unique minimum at $k_{\\mathit{optim}*}$, and the Des Raj estimator has the smaller variance exactly when $k(k-2k_{\\mathit{optim}*})<0$. These are the strict intervals that the paper prints before the marked sentence. The two variances are equal at $k=0$ and at $k=2k_{\\mathit{optim}*}$, and when $k_{\\mathit{optim}*}=0$ no nonzero $k$ improves on the Horvitz--Thompson estimator. For example, let $M=4$ and $m_t=2$, with sizes $(1,2,3,4)$, control totals $(3,3,5,9)$ and a constant effect of $1$ on the totals. Then $k_{\\mathit{optim}*}=2$, and the variance is $1.28$ at $k=0$, $0.2133$ at $k=2$ and $1.28$ at $k=4$.\n\nWhen all clusters have the same size, $n_j-N/M=0$ for every $j$. The adjustment then vanishes, the Des Raj estimator coincides with the Horvitz--Thompson estimator for every $k$, and the quotients in equation (26) and in Appendix C are $0/0$. The Des Raj estimator remains unbiased in that case. Nothing in the paper depends on the two omissions."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E007", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the paragraph beginning 'Recent theoretical results suggest'. Red boxes mark the words 'will have a higher value' and, on the next line, 'than that of the true variance' in the sentence 'These results indicate V-hat-N of Delta-hat-HT will have a higher value than that of the true variance if treatment effects are in fact constant at the cluster scale.' The sentence before says the estimator 'will be conservative relative to the true variance if effects are constant (at the cluster scale) or if the number of clusters is balanced'.", "caption": "Section 4.2, p. 52 (PDF p. 14). The marked words describe the estimate itself as higher than the true variance; the property that holds is the one in the sentence before, that the estimator is conservative.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T16", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T16"}], "headline": "“Will have a higher value than that of the true variance” is meant in expectation: under the stated conditions the sharp-null variance estimator has nonnegative bias, and the estimate from one assignment can fall below the true variance.", "id": "P020-E009", "location": "Section 4.2, paragraph beginning “Recent theoretical results suggest”, p. 52 (PDF p. 14)", "pdf_page": 14, "printed_page": "52", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P020-T16"], "text": {"as_printed": "``In general, $\\hat{\\mathrm{V}}^N(\\widehat{\\Delta_{HT}})$ will be conservative relative to the true variance if effects are constant (at the cluster scale) or if the number of clusters is balanced, in a result that directly follows from theorem 3 of Ding (2014) and Samii and Aronow (2012) (by way of the relationship between pooled and combined variance). These results indicate $\\hat{\\mathrm{V}}^N(\\widehat{\\Delta_{HT}})$ \\hl{will have a higher value than that of the true variance} if treatment effects are in fact constant at the cluster scale.''", "corrected": "``\\hl{Under constant cluster-total treatment effects or balanced cluster counts,} $\\hl{\\mathrm{E}[\\hat{\\mathrm{V}}^N(\\widehat{\\Delta_{HT}})]\\ge\\mathrm{V}(\\widehat{\\Delta_{HT}})}$; \\hl{the realized estimate need not exceed the true variance}.''", "intended_meaning": "On p.~51 the paper calls a variance estimator conservative when its bias is nonnegative. The marked sentence restates that property for the sharp-null variance estimator: its expectation over the assignments is at least the true variance. The property holds. Let $d_j=Y^T_{1j}-Y^T_{0j}$ denote the effect on the total of cluster $j$, with mean $\\overline{d}$ and variance $\\sigma^2(d)$ over the $M$ clusters. Using equation (17) and the closed form of the estimator on p.~52, I obtain\n\\[\n\\mathrm{E}[\\hat{\\mathrm{V}}^N(\\widehat{\\Delta_{HT}})]-\\mathrm{V}(\\widehat{\\Delta_{HT}})=\\frac{M^2}{N^2(M-1)}\\,\\tau^2\\ge0\n\\]\nwhen $d_j=\\tau$ for every cluster, and\n\\[\n\\mathrm{E}[\\hat{\\mathrm{V}}^N(\\widehat{\\Delta_{HT}})]-\\mathrm{V}(\\widehat{\\Delta_{HT}})=\\frac{M^2}{N^2(M-1)}\\left[\\overline{d}^{\\,2}+\\frac{M-2}{M-1}\\,\\sigma^2(d)\\right]\\ge0\n\\]\nwhen $m_t=m_c$. Note that the inequality is weak: equality holds under the sharp null. The estimate from a single assignment can be smaller than the true variance. For example, let $M=N=4$ and $m_t=m_c=2$, with control totals $(0,0,2,2)$ and a constant effect of $2$. Then the true variance is $4/3$, and the estimator has mean $8/3$ over the six assignments. When the first two clusters are treated, each of the four observed totals equals $2$ and the estimate is $0$. Here ``constant at the cluster scale'' means a constant effect on the cluster totals. Nothing in the paper depends on the wording."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E009", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the paragraph that sets up growth in cluster size. Red boxes mark 'M units' in 'the original population of M units is exactly copied (h-1) times' and 'equation 9' in 'is defined as the difference-in-means estimator as in equation 9, but now the inner summation takes place across all h n_j units in each cluster'.", "caption": "Section 3.3, p. 47 (PDF p. 9), items 1 and 2. The marked words call the $M$ clusters “units” and cite equation 9, a probability limit, for the definition of the estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010.png"}, {"alt": "Crop of the display for the variance estimator V-hat of Delta-hat-HT, whose first sum contains Y_j^T minus Y-bar_cj^T, and of the sentence below it. A red box marks the symbol Y-bar_mj^T in 'where Y-bar_mj^T equals the sum over j in J_0 of Y_j^T divided by m_c'.", "caption": "The variance estimator and its “where” clause, p. 51 (PDF p. 13), item 3. The display uses the control mean with subscript $cj$; the marked symbol in the definition has subscript $mj$.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p51.png"}, {"alt": "Crop of the lines that introduce equation (20). A red box marks 'Y_0j^T and Y_0j^T' in 'we first derive our estimates of the study population totals, Y_0j^T and Y_0j^T by differencing off some of the variability'. Equation (20) follows.", "caption": "Sentence before equation (20), p. 54 (PDF p. 16), item 4. The marked symbols name the control total of cluster $j$ twice where the two population totals are meant.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p54.png"}, {"alt": "Crop of the first paragraph of Section 5.2 and equation (24). A red box marks 'Y_0ij^T and Y_1ij^T' in 'the same transformation must be applied to k as to Y_0ij^T and Y_1ij^T'. Two lines above, the text reads 'When Y_0ij and Y_1ij are linearly transformed'.", "caption": "Section 5.2, first paragraph, p. 55 (PDF p. 17), item 5. The marked individual outcomes carry the superscript $T$ of a total; the same outcomes are printed without it two lines above.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p55.png"}, {"alt": "Crop of the first three lines of equation (25). In the second line a red box marks U_1j^T inside the sum over j in J_0 of (b_0 N/M + b_1 U_1j^T). The third line reads M over N times the bracket of one over m_t times b_1 times the sum over J_1 of U_1j^T minus one over m_c times b_1 times the sum over J_0 of U_0j^T.", "caption": "Equation (25), first three lines, p. 56 (PDF p. 18), item 6. In the second line the marked term in the sum over the control clusters $J_0$ is $U^T_{1j}$; the third line has $U^T_{0j}$ in the same sum.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p56.png"}, {"alt": "Crop of the first lines of the page. A red box marks 'for units' in 'n-bar_cj is the mean value of n_j for units in the control condition in a given randomization'. The line above reads 'n-bar_tj is the mean value of n_j for clusters in the treatment condition'.", "caption": "First lines of p. 58 (PDF p. 20), item 9. The marked words take the mean of $n_j$ over “units” in the control condition; the parallel clause for the treatment condition says “clusters”.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p58a.png"}, {"alt": "Crop of the first paragraph of Section 5.4. Red boxes mark X_aij^T in 'A covariates for each individual i in cluster j, denoted by X_aij^T' and Y_j in 'a regression of Y_j on cluster size and cluster-level covariates'. The same paragraph defines the cluster total X_aj^T as the sum over i of X_aij.", "caption": "Section 5.4, first paragraph, p. 58 (PDF p. 20), items 7 and 8. The marked individual covariate carries the superscript $T$, and the marked outcome of the regression lacks it.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p58b.png"}, {"alt": "Crop of the top of the page. A red box marks 'equation 20' in 'In this scenario, the expected value of equation 20 yields'. The display below begins with the expectation of Y-hat_1,R1^T, written with M over m_t and a sum over j in J_1.", "caption": "Appendix B, sentence before equation (29), p. 71 (PDF p. 33), item 10. The marked reference is to equation 20, the estimator of the control total; the display that follows is the expectation of the estimator of the treatment total, equation 21.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p71a.png"}, {"alt": "Crop of equation (30) and the sentence after it. A red box marks 'for units' in 'where n-bar_cj is the mean value of n_j for units in the control condition in a given randomization'.", "caption": "Appendix B, sentence after equation (30), p. 71 (PDF p. 33), item 9. The marked words repeat “for units” from p. 58.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p71b.png"}, {"alt": "Crop of the top of the page and equation (33). A red box marks the first of two identical terms sigma squared of U_j0^T in 'Now note that the terms sigma squared of U_j0^T, sigma squared of U_j0^T, and sigma of U_j0^T, U_j1^T in equation 32 can be written as follows'. Equation (33) below expands sigma squared of U_j1^T.", "caption": "Appendix C, sentence before equation (33), p. 72 (PDF p. 34), item 11. The list names $\\sigma^2(U^T_{j0})$ twice; the first, marked, should be $\\sigma^2(U^T_{j1})$, the term expanded in equation (33).", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p72a.png"}, {"alt": "Crop of three lines of a display. The first line reads sigma of U_j0^T, U_j1^T equals E[U_j0^T U_j1^T] minus U-bar_0^T U-bar_1^T. In the second line a red box marks 'E[Y_j0^T - k delta_j](Y_j1^T - k delta_j)', where the square bracket closes after the first factor. The third line reads E[Y_j0^T Y_j1^T - Y_j0^T k delta_j - Y_j1^T k delta_j + k squared delta_j squared] minus Y-bar_0^T Y-bar_1^T.", "caption": "Appendix C, first three lines of the display that ends in equation (35), p. 72 (PDF p. 34), item 12. In the marked expression the bracket of the expectation closes before the second factor; the next line expands the product inside the expectation.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E010-p72b.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T7", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T7"}, {"id": "P020-T8", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T8"}, {"id": "P020-T9", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T9"}, {"id": "P020-T15", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T15"}], "headline": "Local misprints in labels, indices, cross-references and one bracket: among them, the control sum in the second line of equation (25) should contain $U^T_{0j}$, and the expectation in the second line of the display that ends in equation (35) should enclose both factors.", "id": "P020-E010", "location": "Section 3.3, p. 47 (PDF p. 9); p. 51 (PDF p. 13); p. 54 (PDF p. 16); p. 55 (PDF p. 17); equation (25), p. 56 (PDF p. 18); p. 58 (PDF p. 20); Appendix B, p. 71 (PDF p. 33); Appendix C, p. 72 (PDF p. 34)", "pdf_page": 9, "printed_page": "47", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P020-T15", "P020-T7", "P020-T8", "P020-T9"], "text": {"as_printed": "\\begin{enumerate}\n\\item p.~47: ``(1) the original population of \\hl{$M$ units} is exactly copied $(h-1)$ times, but this time the $h$ copies of a cluster are considered part of one supercluster''\n\\item p.~47: ``(3) $\\widehat{\\Delta_S}$ is defined as the difference-in-means estimator as in \\hl{equation 9}, but now the inner summation takes place across all $hn_j$ units in each cluster''\n\\item p.~51: ``where $\\hl{\\overline{Y^T_{mj}}}=\\sum_{j\\in J_0}Y^T_j/m_c$, the mean value of $Y^T_j$ over all $j\\in J_0$''\n\\item p.~54: ``we first derive our estimates of the study population totals, $\\hl{Y^T_{0j}}$ and $\\hl{Y^T_{0j}}$''\n\\item p.~55: ``the same transformation must be applied to $k$ as to $\\hl{Y^T_{0ij}}$ and $\\hl{Y^T_{1ij}}$''\n\\item p.~56, equation (25), second line:\n\\[\n=\\frac{M}{N}\\left[\\frac{1}{m_t}\\sum_{j\\in J_1}(b_0\\cdot N/M+b_1\\cdot U^T_{1j})-\\frac{1}{m_c}\\sum_{j\\in J_0}(b_0\\cdot N/M+b_1\\cdot \\hl{U^T_{1j}})\\right]\n\\]\n\\item p.~58: ``$A$ covariates for each individual $i$ in cluster $j$, denoted by $\\hl{X^T_{aij}}$''\n\\item p.~58: ``with a regression of $\\hl{Y_j}$ on cluster size and cluster-level covariates''\n\\item pp.~58 and~71: ``$\\overline{n_{cj}}$ is the mean value of $n_j$ \\hl{for units} in the control condition in a given randomization''\n\\item p.~71: ``In this scenario, the expected value of \\hl{equation 20} yields'', followed by the expectation of $\\widehat{Y^T_{1,R1}}$\n\\item p.~72: ``Now note that the terms $\\hl{\\sigma^2(U^T_{j0})}$, $\\sigma^2(U^T_{j0})$, and $\\sigma(U^T_{j0},U^T_{j1})$ in equation 32 can be written as follows''\n\\item p.~72, second line of the display that ends in equation (35):\n\\[\n=\\mathrm{E}[Y^T_{j0}-k\\delta_j\\hl{]}(Y^T_{j1}-k\\delta_j)-\\overline{Y^T_0}\\,\\overline{Y^T_1}\n\\]\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item p.~47: ``(1) the original population of \\hl{$M$ clusters} is exactly copied $(h-1)$ times''\n\\item p.~47: ``(3) $\\widehat{\\Delta_S}$ is defined as the difference-in-means estimator as in \\hl{equation 6}''\n\\item p.~51: ``where $\\hl{\\overline{Y^T_{cj}}}=\\sum_{j\\in J_0}Y^T_j/m_c$''\n\\item p.~54: ``the study population totals, $\\hl{Y^T_0}$ and $\\hl{Y^T_1}$''\n\\item p.~55: ``the same transformation must be applied to $k$ as to $\\hl{Y_{0ij}}$ and $\\hl{Y_{1ij}}$''\n\\item p.~56, equation (25):\n\\[\n\\begin{aligned}\n\\widehat{\\Delta^*_{R1}}&=\\frac{M}{N}\\left[\\frac{1}{m_t}\\sum_{j\\in J_1}U^{T*}_j-\\frac{1}{m_c}\\sum_{j\\in J_0}U^{T*}_j\\right]\\\\\n&=\\frac{M}{N}\\left[\\frac{1}{m_t}\\sum_{j\\in J_1}(b_0\\cdot N/M+b_1\\cdot U^T_{1j})-\\frac{1}{m_c}\\sum_{j\\in J_0}(b_0\\cdot N/M+b_1\\cdot \\hl{U^T_{0j}})\\right]\\\\\n&=\\frac{M}{N}\\left[\\frac{1}{m_t}b_1\\sum_{j\\in J_1}U^T_{1j}-\\frac{1}{m_c}b_1\\sum_{j\\in J_0}U^T_{0j}\\right]\\\\\n&=b_1\\cdot\\widehat{\\Delta_{R1}}.\n\\end{aligned}\n\\]\n\\item p.~58: ``denoted by $\\hl{X_{aij}}$''\n\\item p.~58: ``with a regression of $\\hl{Y^T_j}$ on cluster size and cluster-level covariates''\n\\item pp.~58 and~71: ``$\\overline{n_{cj}}$ is the mean value of $n_j$ \\hl{for clusters} in the control condition in a given randomization''\n\\item p.~71: ``In this scenario, the expected value of \\hl{equation 21} yields''\n\\item p.~72: ``Now note that the terms $\\hl{\\sigma^2(U^T_{j1})}$, $\\sigma^2(U^T_{j0})$, and $\\sigma(U^T_{j0},U^T_{j1})$ in equation 32 can be written as follows''\n\\item p.~72, with $\\delta_j=n_j-N/M$ and $\\mathrm{E}$ the average over the $M$ clusters:\n\\[\n\\begin{aligned}\n\\sigma(U^T_{j0},U^T_{j1})&=\\mathrm{E}[U^T_{j0}U^T_{j1}]-\\overline{U^T_0}\\,\\overline{U^T_1}\\\\\n&=\\mathrm{E}[\\hl{(}Y^T_{j0}-k\\delta_j\\hl{)}(Y^T_{j1}-k\\delta_j)\\hl{]}-\\overline{Y^T_0}\\,\\overline{Y^T_1}\\\\\n&=\\sigma(Y^T_{j0},Y^T_{j1})-k\\sigma(Y^T_{j0},n_j)-k\\sigma(Y^T_{j1},n_j)+k^2\\sigma^2(n_j).\n\\end{aligned}\n\\]\n\\end{enumerate}", "evidence": "The evidence is a check of the pages against the definitions on pp.~44, 46, 48 and 54, with a derivation for the two displayed lines. In equation (25), with $U^T_{0j}$ in the control sum, the constants $b_0\\cdot N/M$ cancel between the two arm means, and $b_1$ times the original estimator remains. For example, let the cluster sizes be $(1,2)$, the control totals $(0,0)$ and the treatment totals $(1,2)$, with $k=0$, $b_0=0$ and $b_1=1$, and let the first cluster be treated. Then the estimate is $2/3$, and the second line as printed evaluates to $-2/3$. In the display that ends in equation (35), $\\mathrm{E}$ is the average over the $M$ clusters and $\\delta_j=n_j-N/M$ has average zero. Expanding $\\mathrm{E}[(Y^T_{j0}-k\\delta_j)(Y^T_{j1}-k\\delta_j)]-\\overline{Y^T_0}\\,\\overline{Y^T_1}$, I obtain $\\sigma(Y^T_{j0},Y^T_{j1})-k\\sigma(Y^T_{j0},n_j)-k\\sigma(Y^T_{j1},n_j)+k^2\\sigma^2(n_j)$, which is equation (35). With $k=0$ and both vectors of totals equal to $(0,2)$, the covariance is $1$. The line as printed instead equals $\\mathrm{E}[Y^T_{j0}]\\,Y^T_{j1}-1=Y^T_{j1}-1$, which is $-1$ for one cluster and $1$ for the other.", "problem": "Each item is a slip in a label, an index, a cross-reference or a bracket, and the neighbouring lines show the intended form. In items 6 and 12 a displayed line is false as printed.\n\\begin{enumerate}\n\\item $M$ is the number of clusters (p.~44), and the parallel construction on p.~46 says ``$M$ clusters''.\n\\item Equation 9 is a probability limit. The estimator with an inner summation is defined in equation 6.\n\\item The display above the clause writes the control mean as $\\overline{Y^T_{cj}}$.\n\\item Equations (20) and (21) estimate the population totals $Y^T_0$ and $Y^T_1$. The symbol $Y^T_{0j}$ is the control total of cluster $j$ (p.~48).\n\\item The superscript $T$ marks a total (p.~48). Individual outcomes do not carry it.\n\\item The sentence before the display defines $U^{T*}_{0j}=b_0\\cdot N/M+b_1\\cdot U^T_{0j}$ for the control clusters, and the third line of the display has $U^T_{0j}$ in the sum over $J_0$.\n\\item The individual covariate is $X_{aij}$. The same paragraph defines the total $X^T_{aj}=\\sum_{i=1}^{n_j}X_{aij}$.\n\\item The regression is of the cluster total $Y^T_j$, as on p.~54.\n\\item The mean $\\overline{n_{cj}}$ is taken over the clusters in the control condition. The clause for $\\overline{n_{tj}}$ says ``for clusters in the treatment condition''.\n\\item Equation 20 defines the estimator of the control total. The display that follows is the expectation of the estimator of the treatment total, equation 21.\n\\item Equation 32 has one variance for each arm, and the expansions that follow, ``respectively'', are (33) for $\\sigma^2(U^T_{j1})$, (34) for $\\sigma^2(U^T_{j0})$ and (35) for the covariance.\n\\item The expectation is of the product of the two factors. The next line of the display expands that product inside the expectation.\n\\end{enumerate}\nNote that Appendix C writes the indices in the order $j0$, $j1$, and the main text writes $0j$, $1j$. This is a relabelling and needs no correction.", "what_survives": "No result changes. Equation (10), the variance estimator on p.~51, the estimators (20) and (21), the invariance result (25), the bias formulas (29) and (30) and the optimal $k$ of Appendix C hold once the slips are corrected. The last lines of the displays (25) and (35) are correct as printed."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E010", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the three design-based panels of Figure 2 and its caption. One red box marks the word 'upper-right' in the last sentence of the caption. A second red box marks the labels Bias = 0.0, SD = 9.2, RMSE = 9.2, which sit in the upper left corner of the Horvitz-Thompson panel. The other two panels carry their labels in the upper left as well.", "caption": "Figure 2, p. 64 (PDF p. 26), lower row of panels and caption, turned upright. The caption places the labels in the upper right of each plot (the word is marked). They are in the upper left, as the marked labels of the first panel show.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E011.png"}, {"alt": "Crop of the three design-based panels of Figure 4 and its caption. One red box marks the word 'upper-right' in the caption. A second red box marks the labels Bias = 0.0, SD = 1.5, RMSE = 1.5 in the upper left corner of the panel 'Raj difference (n j)'.", "caption": "Figure 4, p. 67 (PDF p. 29), lower row of panels and caption, turned upright. The same sentence appears (the word “upper-right” is marked), and the labels are again in the upper left, as marked in the second panel.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E011-b.png"}, {"alt": "Crop of the three design-based panels of Figure 5 and its caption. A red box marks the last sentence of the caption, 'Distributions for the SEs under the sharp null hypothesis of no treatment effect were too narrow to display.' Each of the three panels above it shows a density curve with a solid and a dotted vertical line.", "caption": "Figure 5, p. 69 (PDF p. 31), lower row of panels and caption, turned upright. The marked sentence says that distributions under the sharp null were too narrow to display; Figure 5 shows the scenario with heterogeneous effects, and every panel has its distribution.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-E011-c.png"}], "formal_release": null, "formal_statements": [], "headline": "In Figures 2 and 4 the labels are in the upper left of each panel, not the upper right, and the last sentence of the caption of Figure 5 belongs to Figure 3.", "id": "P020-E011", "location": "Captions of Figure 2, p. 64 (PDF p. 26), Figure 4, p. 67 (PDF p. 29), and Figure 5, p. 69 (PDF p. 31)", "pdf_page": 26, "printed_page": "64", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Captions of Figures 2 and 4: ``Bias and SE estimates in the \\hl{upper-right} of each plot are computed from each empirical distribution.''\n\nCaption of Figure 5, last sentence: ``\\hl{Distributions for the SEs under the sharp null hypothesis of no treatment effect were too narrow to display.}''", "corrected": "Captions of Figures 2 and 4: ``\\hl{Bias, SD, and RMSE summaries} in the \\hl{upper-left} of each plot are computed from each empirical distribution.''\n\nCaption of Figure 5: the caption ends with ``Bias and SD estimates are computed from each empirical distribution.'' \\del{Distributions for the SEs under the sharp null hypothesis of no treatment effect were too narrow to display.}", "evidence": "The evidence is a check of the pages: Figures 2 to 5 were inspected upright, panel by panel.", "problem": "Every panel of Figures 2 and 4 carries its labels (Bias, SD and RMSE) in the upper left. Figure 5 shows the scenario with heterogeneous effects and displays all seven distributions of standard errors. Its last sentence repeats a remark from the caption of Figure 3. That figure concerns the sharp-null scenario, and the distributions for its three design-based estimators are omitted.", "what_survives": "The panels and their numbers are unaffected. The sentence about the sharp null is true of Figure 3, whose caption already contains it. Naming the three labels in the corrected caption is optional, since ``SE'' is acceptable usage for the standard deviation of a point estimator."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-E011", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the opening lines of the page. Red boxes mark 'Y_0 i = Y_1i = Y_i', in which the i of the first symbol sits on the baseline, and the two identities 'sigma squared of Y_0j = sigma squared of Y_1j = sigma squared of Y_j' and 'sigma of Y_0j, Y_1j = sigma squared of Y_j', none of which carries a superscript T.", "caption": "First lines of p. 52 (PDF p. 14). The marked symbols are the broken subscript in $Y_{0i}$ and the two facts, printed without the superscript $T$ of a cluster total.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N001.png"}, {"alt": "Crop of the paragraph. A red box marks 'n_c or n_t' in 'it tends to be more stable than V-hat of Delta-hat-HT, particularly when either n_c or n_t is small, because it combines the variance of the treatment and control groups'.", "caption": "Paragraph beginning “The primary benefit of using”, p. 52 (PDF p. 14). The marked symbols are the numbers of individuals in the two arms; the numbers of clusters $m_c$ and $m_t$ are meant.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P020-T10", "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-T10"}], "headline": "On p. 52 the two facts under the sharp null should be written for the cluster totals $Y^T_{0j}$, $Y^T_{1j}$ and $Y^T_j$, the subscript of $Y_{0i}$ is broken, and “$n_c$ or $n_t$” should read “$m_c$ or $m_t$”.", "id": "P020-N001", "location": "Section 4.2, first two lines of p. 52 (PDF p. 14) and the paragraph beginning “The primary benefit of using”", "pdf_page": 14, "printed_page": "52", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P020-T10"], "text": {"as_printed": "``identical: $\\hl{Y_0i}=Y_{1i}=Y_i$. When the sharp null hypothesis of no effect holds, we know two important facts: $\\hl{\\sigma^2(Y_{0j})=\\sigma^2(Y_{1j})=\\sigma^2(Y_j)}$ and $\\hl{\\sigma(Y_{0j},Y_{1j})=\\sigma^2(Y_j)}$.''\n\n``The primary benefit of using $\\hat{\\mathrm{V}}^N(\\widehat{\\Delta_{HT}})$ is that it tends to be more stable than $\\hat{\\mathrm{V}}(\\widehat{\\Delta_{HT}})$, particularly when either $\\hl{n_c}$ or $\\hl{n_t}$ is small, because it combines the variance of the treatment and control groups.''", "corrected": "``identical: $\\hl{Y_{0i}}=Y_{1i}=Y_i$. \\ldots\\ two important facts: $\\sigma^2(\\hl{Y^T_{0j}})=\\sigma^2(\\hl{Y^T_{1j}})=\\sigma^2(\\hl{Y^T_j})$ and $\\sigma(\\hl{Y^T_{0j}},\\hl{Y^T_{1j}})=\\sigma^2(\\hl{Y^T_j})$.''\n\n``The primary benefit of using $\\hat{\\mathrm{V}}^N(\\widehat{\\Delta_{HT}})$ is that it tends to be more stable than $\\hat{\\mathrm{V}}(\\widehat{\\Delta_{HT}})$, particularly when either $\\hl{m_c}$ or $\\hl{m_t}$ is small, because it combines the variance of the treatment and control groups.''", "evidence": "The evidence is a check of the page against the definitions on pp.~44, 48, 50 and 51. The display that follows the two facts on p.~52 writes $\\sigma^2(Y^T_j)$.", "problem": "The variances that enter equation (17), and the display that follows these lines on p.~52, are variances of cluster totals: $Y^T_{0j}$ and $Y^T_{1j}$ are defined on p.~48 and their variances on p.~50. Without the superscript the symbols $Y_{0j}$, $Y_{1j}$ and $Y_j$ are not defined. In $Y_0i$ the index $i$ is printed on the baseline. The symbols $n_c$ and $n_t$ denote the numbers of individuals in the control and treatment arms (p.~44), which are random under the assignment of clusters. The estimator $\\hat{\\mathrm{V}}(\\widehat{\\Delta_{HT}})$ has the denominators $m_c(m_c-1)$ and $m_t(m_t-1)$ and is undefined when $m_c$ or $m_t$ is $1$ (p.~51). Its stability therefore depends on the number of clusters in each arm.", "what_survives": "Equation (17), the display for $\\mathrm{V}^N(\\widehat{\\Delta_{HT}})$ and the remark on stability are correct with these readings. Under the sharp null the identities hold for the cluster totals as they do for the individual outcomes. Nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-N001", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the last paragraph of Section 6.1. A red box marks the word 'three' in 'We use randomization inference to examine the behavior of our estimators and compare them with the behavior of three commonly used estimators.'", "caption": "Section 6.1, last paragraph, p. 60 (PDF p. 22). The marked word gives the number of comparison estimators as three; Section 6.3 defines four.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N003.png"}, {"alt": "Crop of the fixed-effects regression with covariates and the sentence below it. A red box marks 'precinct-level' in 'where beta_2 is the coefficient on precinct-level voter turnout in 2000, beta_3 is the coefficient on age'.", "caption": "Section 6.3, p. 62 (PDF p. 24). The marked word places the turnout covariate at the level of the precinct; p. 59 defines it as the turnout rate in the cable district.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "“Three commonly used estimators” should read “four”, and the turnout covariate is measured for the cable district and is not “precinct-level”.", "id": "P020-N003", "location": "Section 6.1, last paragraph, p. 60 (PDF p. 22); Section 6.3, p. 62 (PDF p. 24)", "pdf_page": 22, "printed_page": "60", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~60: ``We use randomization inference to examine the behavior of our estimators and compare them with the behavior of \\hl{three} commonly used estimators.''\n\np.~62: ``where $\\beta_2$ is the coefficient on \\hl{precinct-level} voter turnout in 2000, $\\beta_3$ is the coefficient on age, and the model is fitted with ordinary least squares.''", "corrected": "p.~60: ``\\ldots\\ compare them with the behavior of \\hl{four} commonly used estimators.''\n\np.~62: ``where $\\beta_2$ is the coefficient on \\hl{the cable-district turnout rate} in 2000, $\\beta_3$ is the coefficient on age, and the model is fitted with ordinary least squares.''", "evidence": "I checked pp.~59--63 and the panels of Figures 2--5.", "problem": "Section 6.3 opens with ``We will consider four regression-based estimators as well as the three design-based estimators proposed in this paper'' (p.~61). It defines four regression estimators, and each of Figures 2--5 has four regression panels. The turnout covariate is defined on p.~59 as the ``turnout rate in the cable district in the 2000 election''. The clusters are cable districts, and the turnout covariate is defined at that level.", "what_survives": "The estimators that are compared are those of Section 6.3 and the figures, and no number changes. The two covariates are indexed $X_2$, $X_3$ in the regressions (p.~62) and $X^T_1$, $X^T_2$ in the Des Raj model (p.~63). Each model defines its own indices, and this is not an error."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-N003", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of a paragraph of Section 6.3. Red boxes mark the sentence 'For both fixed effects models, Huber-White robust cluster standard errors are estimated', which runs over two lines. The sentences before it discuss bias from regression adjustment, and the sentence after it says that these standard errors may be unreliable in finite samples.", "caption": "Section 6.3, p. 62 (PDF p. 24). The marked sentence is the only statement of a standard-error procedure for the least-squares regressions, and it covers the two fixed-effects models only.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "Section 6.3 states a standard-error procedure for every estimator except the regression without covariates of equation (27); the procedure used for that regression needs to be stated, and ordinary least-squares standard errors cannot explain the figure on whole-cluster subsets of the public data.", "id": "P020-N005", "location": "Section 6.3, p. 62 (PDF p. 24), paragraph beginning “As Freedman (2008a) notes”; equation (27), p. 61 (PDF p. 23); Figure 3, p. 65 (PDF p. 27)", "pdf_page": 24, "printed_page": "62", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 6.3, p.~61, after equation (27): ``Our estimate of $\\beta_1$ follows from fitting the model with ordinary least squares.''\n\nSection 6.3, p.~62: ``For \\hl{both fixed effects models}, Huber-White ``robust'' cluster standard errors are estimated.''\n\nSection 6.4, pp.~63 and 66: ``In the case of the regression with no covariates (difference-in-means) the standard errors are biased upwards due to the failure of this model to account for blocking.''", "corrected": "``For both fixed effects models, Huber-White ``robust'' cluster standard errors are estimated. \\hl{For the regression without covariates, the standard errors are computed by [the procedure that was used].}''\n\nThe added sentence cannot be completed from the article or from the replication files of the source study. For each robust standard error, the description should also give the clustering variable, the covariance formula, the finite-sample correction and the treatment of the blocks.", "evidence": "The evidence is a check of the pages and a recalculation from the public data of the source study. Figure 3 reports a bias of 0.53 percentage points for this panel, measured from the standard deviation of the estimator, which Figure 2 reports as 2.4. Then the mean of the plotted standard errors is about 2.93 points. On whole-cluster subsets of the public data, the standard error from ordinary least squares cannot be that large. For a binary outcome regressed on a constant and a treatment indicator, the residual sum of squares is at most $N/4$, so that the residual variance satisfies $s^2\\le N/(4(N-2))$. With $n_t$ and $n_c$ individuals in the two arms, the squared standard error of $\\hat\\beta_1$ is $s^2(1/n_t+1/n_c)$. Each arm consists of 40 whole clusters, and in the public data the 40 smallest of the 85 clusters contain $L=5{,}654$ individuals together, so that each arm has at least $L$ individuals. I have\n\\[\ns^2\\Bigl(\\frac{1}{n_t}+\\frac{1}{n_c}\\Bigr)\\le\\frac{N}{4(N-2)}\\Bigl(\\frac{1}{L}+\\frac{1}{N-L}\\Bigr).\n\\]\nWith $N=22{,}733$ the square root of the right side gives a bound of about $0.767$ percentage points, for every assignment under these assumptions. This rules out ordinary least-squares standard errors for that sample construction. It does not identify the procedure or establish that the authors used the public data unchanged.", "problem": "Section 6.3 specifies the standard errors of the two fixed-effects regressions (cluster-robust), of the random-effects regression (produced by \\texttt{lmer()}) and of the three design-based estimators (the square root of the estimated variance under the sharp null). For the first regression, equation (27), it says only that the coefficient is fitted by ordinary least squares. Figures 3 and 5 nevertheless show a distribution of standard errors for this regression, and Section 6.4 interprets its upward bias. The paper does not say which standard error was computed, and the procedure cannot be recovered from the article or from the scripts of the source study, which serve a different analysis.", "what_survives": "The point estimators as displayed and the labels printed in the figures stand. No standard-error implementation and no design-based formula is shown to be wrong. The distributions of the regression standard errors and the comparisons of their biases cannot be reproduced exactly."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-N005", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the four regression panels of Figure 4: Reg. (Diff-in-means), Reg. w/FE, Reg. w/FE, cov., and Reg. w/FE, cov, RE. Each panel shows a density curve over an axis from minus 10 to 10 percentage points, a solid vertical line at the mean and a dotted vertical line just right of zero. Red boxes mark the labels Bias = -0.2, Bias = 0.9, Bias = 0.9 and Bias = 2.0. Below them, unmarked, are SD = 2.3, 1.5, 1.5, 1.3 and RMSE = 2.3, 1.8, 1.8, 2.4.", "caption": "Figure 4, p. 67 (PDF p. 29), upper row of panels, turned upright. The four marked bias labels agree, to one decimal place, with the solid line minus the dotted line, and the dotted lines stand near 1.0 on the axis, not at 0.7.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N008.png"}, {"alt": "Crop of eight lines of Section 6.5. One red box marks '(0.7 percentage points)' at the end of the sentence 'Dotted vertical lines indicate the true treatment effect (0.7 percentage points).' A second red box marks 'is only slightly biased downward' in the sentence about the regression without covariates.", "caption": "Section 6.5, p. 66 (PDF p. 28), first two paragraphs. The marked phrases are the true effect that the dotted lines are said to indicate and the description of the difference-in-means that rests on the printed bias of $-0.2$.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N008-b.png"}, {"alt": "Chart with one row for each of the seven panels of Figure 4 and a horizontal axis from 0 to 3.3 percentage points. A blue vertical line marks 0.7, the true effect stated in the text. Orange diamonds mark the dotted lines of the four regression panels, all at 1.0. Black circles mark the solid lines: 0.75 for the difference-in-means, 1.90 and 1.92 for the two fixed-effects regressions, 3.02 for the random-effects regression, 0.82 for Horvitz-Thompson and 0.77 and 0.75 for the two Des Raj estimators. Two columns of numbers at the right give the printed bias (minus 0.2, 0.9, 0.9, 2.0, 0.0, 0.0, 0.0) and the mean minus 0.7 (0.05, 1.20, 1.22, 2.32, 0.12, 0.07, 0.05).", "caption": "The lines of Figure 4 as read from the published PDF, for all seven panels. In the four regression panels the dotted line (diamond) stands at 1.0 and not at the stated true effect of 0.7 (vertical line). The columns at the right give the printed bias and the plotted mean minus 0.7, in percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P020/exhibits/P020-N008-reference.png"}], "formal_release": null, "formal_statements": [], "headline": "In the four regression panels of Figure 4, bias is measured from a dotted line at 1.0 percentage point and not from the stated true effect of 0.7; measured from 0.7, the biases of the fixed-effects and random-effects regressions are about 1.2 and 2.3 points instead of 0.9 and 2.0, and the figure does not establish downward bias of the difference-in-means.", "id": "P020-N008", "location": "Figure 4, p. 67 (PDF p. 29), the four regression panels; Section 6.5, p. 66 (PDF p. 28), first two paragraphs; Section 6.2, p. 61 (PDF p. 23), last sentence", "pdf_page": 29, "printed_page": "67", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Section 6.2, p.~61, last sentence: ``Marginalizing over the imputation process, the ATE is in expectation $\\frac{\\sum_i(p_{ti}-p_{ci})}{N}=0.007$, or 0.7 percentage points.''\n\nSection 6.5, p.~66, first paragraph: ``Figure 4 displays results under treatment effect heterogeneity. Solid vertical lines indicate the mean of the sampling distributions. Dotted vertical lines indicate the true treatment effect (0.7 percentage points).''\n\nSection 6.5, p.~66, second paragraph: ``Interestingly, the regression without covariates (difference-in-means) is only slightly biased \\hl{downward}.''\n\nFigure 4, p.~67, labels of the four regression panels, in percentage points. In each of these panels the dotted line is drawn near 1.0 on the horizontal axis.\n\n\\begin{audittable}{lrrr}{Figure 4 as printed: labels of the regression panels.}\nPanel & Bias & SD & RMSE \\\\\nReg. (Diff-in-means) & $\\hl{-0.2}$ & $2.3$ & $2.3$ \\\\\nReg. w/FE & $\\hl{0.9}$ & $1.5$ & $\\hl{1.8}$ \\\\\nReg. w/FE, cov. & $\\hl{0.9}$ & $1.5$ & $\\hl{1.8}$ \\\\\nReg. w/FE, cov, RE & $\\hl{2.0}$ & $1.3$ & $\\hl{2.4}$ \\\\\n\\end{audittable}", "corrected": "In Figure 4, the dotted lines \\hl{should be drawn at the common, unrounded true effect of the scenario}, and \\hl{all empirical bias and RMSE labels should be computed from it}. If that effect is 0.7 percentage points, as stated on pp.~61 and 66, then the published means and standard deviations give the following approximate labels.\n\n\\begin{audittable}{lrrr}{Figure 4 with bias measured from 0.7: approximate labels of the regression panels, in percentage points.}\nPanel & Bias & SD & RMSE \\\\\nReg. (Diff-in-means) & $\\hl{+0.05}$ & $2.3$ & $2.30$ \\\\\nReg. w/FE & $\\hl{1.20}$ & $1.5$ & $\\hl{1.92}$ \\\\\nReg. w/FE, cov. & $\\hl{1.22}$ & $1.5$ & $\\hl{1.93}$ \\\\\nReg. w/FE, cov, RE & $\\hl{2.32}$ & $1.3$ & $\\hl{2.66}$ \\\\\n\\end{audittable}\n\nSection 6.5, second paragraph: ``Interestingly, the regression without covariates (difference-in-means) \\hl{has a mean close to the true effect}.''\n\nExact labels require the unrounded true effect and the original draws. Changing the text to 1.0 has not been checked against the imputation model, and it would leave the labels ``Bias = 0.0'' of the three design-based panels in conflict with their plotted means of 0.82, 0.77 and 0.75.", "evidence": "The evidence is a recalculation from the published PDF, in which Figure 4 is stored as vector artwork. The coordinates of the axis ticks, the solid lines and the dotted lines were read from the page and converted to the scale of the axis in exact rational arithmetic. The remaining ticks fit this calibration to within 0.0015 points, and the incompatibility with the stated effect persists with a tolerance of 0.005. A second, independent reading of the coordinates agrees with the first to within 0.002 points.\n\n\\begin{audittable}{lrrrrrr}{Figure 4, regression panels: positions of the solid line (the mean) and of the dotted line, read from the published PDF, and the bias and RMSE that follow if the true effect is 0.7. All entries are in percentage points.}\nPanel & Solid & Dotted & Printed bias & Solid minus 0.7 & Printed RMSE & RMSE at 0.7 \\\\\nReg. (Diff-in-means) & $0.753$ & $1.001$ & $-0.2$ & $+0.053$ & $2.3$ & $2.25$--$2.35$ \\\\\nReg. w/FE & $1.904$ & $1.001$ & $0.9$ & $+1.204$ & $1.8$ & $1.88$--$1.96$ \\\\\nReg. w/FE, cov. & $1.915$ & $0.999$ & $0.9$ & $+1.215$ & $1.8$ & $1.89$--$1.97$ \\\\\nReg. w/FE, cov, RE & $3.024$ & $0.999$ & $2.0$ & $+2.324$ & $2.4$ & $2.64$--$2.69$ \\\\\n\\end{audittable}\n\nThe solid line minus the dotted line is $-0.248$, $0.903$, $0.916$ and $2.025$ in the four panels, and these values round to the printed biases. Suppose that a printed bias $b$ is rounded to one decimal and that the solid line is at $m$. Then a target compatible with the label lies in $[m-b-0.05,\\,m-b+0.05]$. For the four panels these intervals are $[0.903,1.003]$, $[0.954,1.054]$, $[0.965,1.065]$ and $[0.974,1.074]$, and each of them lies above $0.75$. For the RMSE, I use the identity\n\\[\n\\mathrm{E}\\bigl[(T-\\theta)^2\\bigr]=\\operatorname{Var}(T)+\\bigl(\\mathrm{E}[T]-\\theta\\bigr)^2\n\\]\napplied to the empirical distribution of the 5,000 estimates, with $\\theta=0.7$, the plotted mean and a standard deviation within $0.05$ of the printed one. If the standard deviation uses divisor 4,999, its square is multiplied by $4{,}999/5{,}000$ to give the empirical variance. The last column of the table shows the resulting approximate range. The printed $2.3$ of the difference-in-means panel lies in its range, and the printed $1.8$, $1.8$ and $2.4$ lie outside theirs. The last column gives guides rather than exact replacement labels: the plotted positions and the printed standard deviations are rounded, and the original draws are not available.", "problem": "Section 6.2 states that the true effect in the scenario with heterogeneous effects is 0.7 percentage points, and Section 6.5 says that the dotted lines of Figure 4 mark it. In the published figure, the dotted lines stand at 1.00 on the axis in the four regression panels, and each printed bias equals the position of the solid line minus the position of the dotted line, to the printed decimal. The three design-based panels show no separate dotted line. The text and the figure cannot both be right, and rounding does not reconcile them: a value printed as 0.007 lies between 0.65 and 0.75 percentage points, and every dotted line lies between 0.99 and 1.01. If the true effect is 0.7, as the text states twice, then the biases of the four regressions are about 0.3 points higher than printed: about $+0.05$ for the difference-in-means, $+1.20$ and $+1.22$ for the two fixed-effects regressions, and $+2.32$ for the random-effects regression. The RMSE labels of the fixed-effects and random-effects panels ($1.8$, $1.8$ and $2.4$) are then too small. From the published means and standard deviations, I obtain about $1.9$, $1.9$ and a value between $2.6$ and $2.7$. The sentence that the difference-in-means is ``only slightly biased downward'' rests on the printed $-0.2$. Measured from 0.7, the plotted mean of that estimator is higher by about 0.05, with a Monte Carlo standard error of about 0.03, so the figure does not establish downward bias.\n\nThe label ``Bias = 0.0'' of the Horvitz--Thompson panel also needs to be recomputed. The mean drawn in that panel is 0.82, which exceeds every target between 0.65 and 0.75 by more than 0.05. If the target is exactly 0.7, the same holds for the Des Raj panel with $n_j$, whose mean is 0.77. These are disagreements between labels and plotted means in a simulation of 5,000 draws, and they do not imply that either design-based estimator is biased. The cause of the mismatch is not established, since the original simulation code and draws are not available.", "what_survives": "The inconsistency concerns the reporting of Figure 4 and the downward-bias sentence of Section 6.5. With the published means and standard deviations and a target near 0.7, the biases of the fixed-effects and random-effects regressions remain positive, the random-effects regression still has the smallest standard deviation, and the Horvitz--Thompson estimator remains imprecise. The grouping by RMSE is unchanged: the smallest values belong to the Des Raj estimators, and then come the fixed-effects regressions, the difference-in-means, the random-effects regression and the Horvitz--Thompson estimator. On these numbers, the statement that the Des Raj estimators are the most precise in terms of RMSE stands. The arguments on bias and consistency in Section 3 and the estimators and unbiasedness results of Sections 4 and 5 do not use Figure 4. Figures 2 and 3 come from the separate calculation under the sharp null. Figure 5 concerns standard errors and does not depend on the placement of Figure 4's reference lines. What changes is the size of the regression biases, the RMSE labels of three panels, and the statement that the difference-in-means shows a downward bias in this example; the empirical bias labels of the design-based panels also require checking. The theoretical result that the difference-in-means can be biased is not affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P020/#P020-N008", "work": {"arxiv_id": null, "doi": "10.1515/spp-2013-0002", "title": "Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P020/"}, "work_id": "P020"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the heading 2. Results, the first paragraph of the section, and Conditions 1 and 2. A red outline surrounds the words Without loss of generality at the start of the last sentence of the paragraph. The sentence continues: suppose that Y has bounded support and that D has support in the set 1 to K.", "caption": "Section 2, first paragraph, p. 101 (PDF p. 2). The outline marks the words “Without loss of generality”, which introduce the bounds on the outcome and on the degree.", "url": "https://pmaronow.github.io/llm-audit/works/P021/exhibits/P021-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P021-T2", "url": "https://pmaronow.github.io/llm-audit/works/P021/#P021-T2"}], "headline": "The bounds on the outcome and on the degree are assumptions that hold for every population in the sequence, and they are not made without loss of generality.", "id": "P021-E001", "location": "Section 2, first paragraph, last sentence, p. 101 (PDF p. 2)", "pdf_page": 2, "printed_page": "101", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P021-T2"], "text": {"as_printed": "Formally, consider a sequence of populations and samples converging weakly to a joint limit distribution on the outcome, (reported) degree, and sample, denoted $(Y, D, S)$. Let the $\\mathrm{E}[\\cdot]$ and $\\Pr[\\cdot]$ operators refer to features of this limiting distribution. In RDS, we observe the empirical joint distribution of the outcome $Y$ and degree $D$ conditional on the sampling indicator $S = 1$. \\hl{Without loss of generality}, suppose that $Y$ has bounded support and that $D$ has support in the set $\\{1, \\ldots, K\\}$.", "corrected": "Formally, consider a sequence of populations and samples \\hl{whose empirical joint laws converge weakly, in probability if they are random,} to a joint limit distribution on the outcome, (reported) degree, and sample, denoted $(Y, D, S)$. Let the $\\mathrm{E}[\\cdot]$ and $\\Pr[\\cdot]$ operators refer to features of this limiting distribution. In RDS, we observe the empirical joint distribution of the outcome $Y$ and degree $D$ conditional on the sampling indicator $S = 1$. \\hl{For simplicity, assume that there exist constants $B<\\infty$ and $K<\\infty$ such that every population in the sequence has outcomes in $[-B,B]$ and degrees in $\\{1, \\ldots, K\\}$.}", "intended_meaning": "The two bounds are meant as standing assumptions, made for simplicity. Every population in the sequence has outcomes in a common interval $[-B,B]$ and degrees in $\\{1,\\ldots,K\\}$. On this reading, Proposition 1 and Corollary 1 are correct. For Corollary 1, suppose that $\\Pr[S=1\\mid D=k]=ak$ with $a>0$, and let $A_N$ and $H_N$ denote the averages of $sy/d$ and $s/d$ over the $N$ units of a population, where $s=1$ for a sampled unit and $s=0$ otherwise. The estimator (1) equals $A_N/H_N$. Since the functions $sy/d$ and $s/d$ are bounded and continuous on $[-B,B]\\times\\{1,\\ldots,K\\}\\times\\{0,1\\}$, the weak convergence assumed in the first sentence of Section 2 implies that\n\\[\nH_N \\to \\mathrm{E}[S/D] = a, \\qquad A_N \\to \\mathrm{E}[SY/D] = a\\,\\mathrm{E}[Y],\n\\]\nwhere I used Condition 1 in the last equality. I conclude that $A_N/H_N \\to \\mathrm{E}[Y]$.\n\nThe words ``without loss of generality'' claim more than this, and two exact examples explain the outcome restriction. First, let $D=1$, let $\\Pr[Y=j]=2/3^{j+1}$ for every integer $j\\ge 0$, and let $S$ be independent of $Y$ with $\\Pr[S=1]=2/5$. Then Conditions 1 and 2 hold with $f(1)=1$ and $c=2/5$, and $\\mathrm{E}[Y]=1/2$. The outcome is unbounded, and clipping $Y$ at an integer $M\\ge 1$ changes the mean to $(1-3^{-M})/2$. Second, a bound on the limiting $Y$ alone is not enough for Corollary 1. For each $m\\ge 1$, I consider a population of $N=3m$ units of degree $1$ in which $m$ units are sampled, one sampled unit has the outcome $2m$, and every other unit has the outcome $0$. The empirical joint laws converge weakly to the law with mass $2/3$ at $(Y,D,S)=(0,1,0)$ and mass $1/3$ at $(0,1,1)$. This limit satisfies Conditions 1 and 2 with $f(1)=1$ and $c=1/3$, and it has $\\mathrm{E}[Y]=0$. However, $A_N=2/3$ and $H_N=1/3$, and the estimator (1) equals $2$ for every $m$.\n\nThe common bound on the degree is stronger than Corollary 1 needs. When $\\Pr[S=1\\mid D=k]=ak$, the positivity in Condition 1 excludes degree $0$, and the inequality $ak\\le 1$ bounds the degrees of the limit. A condition under which $A_N$ and $H_N$ converge to $\\mathrm{E}[SY/D]$ and $\\mathrm{E}[S/D]$ can replace the common bound on the outcome."}, "url": "https://pmaronow.github.io/llm-audit/works/P021/#P021-E001", "work": {"arxiv_id": "1504.03574", "doi": "10.1016/j.spl.2015.07.003", "title": "Nonparametric Identification for Respondent-Driven Sampling", "url": "https://pmaronow.github.io/llm-audit/works/P021/"}, "work_id": "P021"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the reference list, from the entry for Berchenko, Rosenblatt and Frost to the first line of the entry for McCreesh and coauthors. Red outlines surround the digits 13043505 at the end of the Berchenko entry and the digits 14014718 at the end of the entry for Lunagomez and Airoldi. The Crawford entry reads http://arxiv.org/pdf/1406.0721.", "caption": "References, p. 102 (PDF p. 3). The outlines mark the two arXiv identifiers printed without the dot. The entry for Crawford (2014), three lines below the first, prints its identifier with the dot.", "url": "https://pmaronow.github.io/llm-audit/works/P021/exhibits/P021-E002.png"}], "formal_release": null, "formal_statements": [], "headline": "The two arXiv identifiers should read arXiv:1304.3505 and arXiv:1401.4718; each is printed without the dot.", "id": "P021-E002", "location": "References, p. 102 (PDF p. 3): the entries for Berchenko, Rosenblatt and Frost (2013) and for Lunagomez and Airoldi (2014)", "pdf_page": 3, "printed_page": "102", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Berchenko, Y., Rosenblatt, J., Frost, S.D., 2013, Modeling and analysing respondent driven sampling as a counting process. arXiv preprint arXiv:\\hl{13043505}.\n\nLunagomez, S., Airoldi, E.M., 2014, Valid inference from non-ignorable network sampling designs. arXiv preprint arXiv:\\hl{14014718}.", "corrected": "Berchenko, Y., Rosenblatt, J., Frost, S.D., 2013, Modeling and analysing respondent driven sampling as a counting process. arXiv preprint arXiv:\\hl{1304.3505}.\n\nLunagomez, S., Airoldi, E.M., 2014, Valid inference from non-ignorable network sampling designs. arXiv preprint arXiv:\\hl{1401.4718}.\n\nThe links should point to \\url{https://arxiv.org/abs/1304.3505} and \\url{https://arxiv.org/abs/1401.4718}.", "evidence": "I checked the page: both identifiers appear without the dot on p.~102, while the entry for Crawford (2014) on the same page prints 1406.0721 with the dot. In the PDF, the two links point to addresses that end in \\texttt{13043505} and \\texttt{14014718}. The first versions of arXiv:1304.3505 and arXiv:1401.4718 carry the cited titles and authors.", "problem": "An arXiv identifier of 2013 or 2014 has a dot between the four digits of the year and month and the serial number. Both entries print the eight digits without the dot, and the links in the PDF point to the same strings.", "what_survives": "No result, proof or estimator depends on these entries. Only the two identifiers and their links change."}, "url": "https://pmaronow.github.io/llm-audit/works/P021/#P021-E002", "work": {"arxiv_id": "1504.03574", "doi": "10.1016/j.spl.2015.07.003", "title": "Nonparametric Identification for Respondent-Driven Sampling", "url": "https://pmaronow.github.io/llm-audit/works/P021/"}, "work_id": "P021"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of nine lines of the reference list, from the entry for Johnston and coauthors to the entry for Salganik (2012). Red outlines surround the volume and issue 12 (1) before the pages 131 to 141 in the Johnston entry, and the volume and issue 83 (1) before the pages 98 to 112 in the entry for Salganik (2006).", "caption": "References, p. 102 (PDF p. 3). The outlines mark the volume and issue, “12 (1)” and “83 (1)”, in the entries for Johnston et al. (2008) and for Salganik (2006).", "url": "https://pmaronow.github.io/llm-audit/works/P021/exhibits/P021-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "Two reference entries give a supplement as the ordinary issue 1: the issues are AIDS Behav. 12 (Suppl 1) and J. Urban Health 83 (Suppl 1).", "id": "P021-N001", "location": "References, p. 102 (PDF p. 3): the entries for Johnston et al. (2008) and for Salganik (2006)", "pdf_page": 3, "printed_page": "102", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Johnston, L.G., Malekinejad, M., Kendall, C., Iuppa, I.M., Rutherford, G.W., 2008. Implementation challenges to using respondent-driven sampling methodology for HIV biological and behavioral surveillance: field experiences in international settings. AIDS Behav. 12 \\hl{(1)}, 131--141.\n\nSalganik, M.J., 2006. Variance estimation, design effects, and sample size calculations for respondent-driven sampling. J. Urban Health 83 \\hl{(1)}, 98--112.", "corrected": "Johnston, L.G., Malekinejad, M., Kendall, C., Iuppa, I.M., Rutherford, G.W., 2008. Implementation challenges to using respondent-driven sampling methodology for HIV biological and behavioral surveillance: field experiences in international settings. AIDS Behav. 12 \\hl{(Suppl 1)}, 131--141.\n\nSalganik, M.J., 2006. Variance estimation, design effects, and sample size calculations for respondent-driven sampling. J. Urban Health 83 \\hl{(Suppl 1)}, 98--112.", "evidence": "I compared the entries with the publisher's pages for the two articles. The page for the article of Johnston et al.\\ gives ``AIDS Behav 12 (Suppl 1), 131--141 (2008)'' (\\url{https://link.springer.com/article/10.1007/s10461-008-9413-1}). The page for the article of Salganik gives ``J Urban Health 83 (Suppl 1), 98--112 (2006)'' (\\url{https://link.springer.com/article/10.1007/s11524-006-9106-x}). The National Library of Medicine indexes the same articles as 12(4 Suppl):S131--S141 and 83(6 Suppl):i98--i112.", "problem": "Both articles appeared in supplements to their volumes, and the two entries print the supplement as the ordinary issue ``(1)''. The volumes and the page ranges are correct, and both articles can still be found from them. The two arXiv identifiers in the same list are the subject of \\fid{P021-E002}.", "what_survives": "No result depends on these entries. A comparison with the article of Gile et al.\\ (2015) supports the attributions to it. The phrase quoted with the locator ``pg.~3'' is on the third page of their article, p.~243, and Table 1 keeps every substantive entry of their Table 1 on p.~244. The other entries of the reference list agree with the publishers' or indexers' records in volume, issue, year and first page. The entries for Broadhead et al.\\ (1998) and McCreesh et al.\\ (2012) give only the first page of the ranges 42--57 and 138--147."}, "url": "https://pmaronow.github.io/llm-audit/works/P021/#P021-N001", "work": {"arxiv_id": "1504.03574", "doi": "10.1016/j.spl.2015.07.003", "title": "Nonparametric Identification for Respondent-Driven Sampling", "url": "https://pmaronow.github.io/llm-audit/works/P021/"}, "work_id": "P021"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Condition 1, Condition 2, Proposition 1 with equation (2), and the first line of the proof. Red outlines surround three items: the equation Pr[S = 1 | D = k] = c f(k) in Condition 2, and the summation signs with limits k = 1 and K in the numerator and in the denominator of equation (2). Condition 1, which is not marked, begins: For all k such that Pr[D = k] > 0.", "caption": "Conditions 1 and 2 and Proposition 1, p. 101 (PDF p. 2). The outlines mark the equation of Condition 2, which carries no restriction on $k$, and the two sums in (2), which run from $k=1$ to $K$.", "url": "https://pmaronow.github.io/llm-audit/works/P021/exhibits/P021-N003.png"}, {"alt": "Crop of the Discussion. Red outlines surround the words neither Condition 1 nor Condition 2 has directly testable implications, at the end of one line and the start of the next. The sentence begins: Without knowledge of the characteristics of the unsampled subjects.", "caption": "Discussion, second paragraph, p. 102 (PDF p. 3). The outlines mark the statement that neither condition has directly testable implications; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P021/exhibits/P021-N003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P021-T1", "url": "https://pmaronow.github.io/llm-audit/works/P021/#P021-T1"}], "headline": "Condition 2 and the sums in (2) and (4) are meant for the degrees $k$ such that $\\Pr[D=k]>0$, the restriction that Condition 1 states.", "id": "P021-N003", "location": "Condition 2, Proposition 1 with equation (2), and the proof with equations (3) and (4), p. 101 (PDF p. 2); Discussion, second paragraph, p. 102 (PDF p. 3)", "pdf_page": 2, "printed_page": "101", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P021-T1"], "text": {"as_printed": "\\emph{Section 2, p.~101.} Condition 2 (Knowledge of the Conditional Probability of Sampling). $\\Pr[S = 1 \\mid D = \\hl{k}] = cf(k)$, where $f(\\cdot)$ is known and $c$ is an unknown scale parameter.\n\nProposition 1. Given Conditions 1 and 2, the population mean is identified, with\n\\[\n\\mathrm{E}[Y] = \\frac{\\hl{\\sum_{k=1}^{K}} \\mathrm{E}[Y \\mid S = 1, D = k]\\,\\frac{\\Pr[D = k \\mid S = 1]}{f(k)}}{\\hl{\\sum_{k=1}^{K}} \\frac{\\Pr[D = k \\mid S = 1]}{f(k)}}. \\qquad (2)\n\\]\n\n\\emph{Discussion, p.~102.} Without knowledge of the characteristics of the unsampled subjects, \\hl{neither Condition 1 nor Condition 2 has directly testable implications}, and thus the value of any diagnostics must depend on further assumptions about the generative process.", "corrected": "\\emph{Section 2, p.~101.} \\hl{Let $A=\\{k:\\Pr[D=k]>0\\}$.} Condition 2 (Knowledge of the Conditional Probability of Sampling). \\hl{For all $k\\in A$,} $\\Pr[S = 1 \\mid D = k] = cf(k)$, where $f(\\cdot)$ is known and $c$ is an unknown scale parameter.\n\nProposition 1. Given Conditions 1 and 2, the population mean is identified, with\n\\[\n\\mathrm{E}[Y] = \\frac{\\hl{\\sum_{k\\in A}} \\mathrm{E}[Y \\mid S = 1, D = k]\\,\\frac{\\Pr[D = k \\mid S = 1]}{f(k)}}{\\hl{\\sum_{k\\in A}} \\frac{\\Pr[D = k \\mid S = 1]}{f(k)}}. \\qquad (2)\n\\]\nThe proof's ``for each degree $k$'' and equation (3) refer to $k\\in A$, and both sums in each fraction of (4) are over $k\\in A$. Under Condition 1, $A=\\{k:\\Pr[D=k\\mid S=1]>0\\}$, so the observed distribution determines $A$.\n\n\\emph{Discussion, p.~102.} Without knowledge of the characteristics of the unsampled subjects, \\hl{and apart from compatibility of the observed degree support with the known sampling-probability profile,} neither Condition 1 nor Condition 2 has directly testable implications, and thus the value of any diagnostics must depend on further assumptions about the generative process.", "intended_meaning": "Condition 1 is stated for the degrees $k$ such that $\\Pr[D=k]>0$. Condition 2, the proof's ``for each degree $k$'', equation (3) and the sums in (2) and (4) are meant for the same degrees, and I write $A=\\{k:\\Pr[D=k]>0\\}$ for this set. Read literally, the sums also include every $k\\le K$ such that $\\Pr[D=k]=0$. For such $k$ the conditional means are not defined, and the weight $\\Pr[D=k\\mid S=1]/f(k)$ is $0/0$ when $f(k)=0$. For example, let $K=3$, $\\Pr[D=1]=\\Pr[D=3]=1/2$, $f(1)=1$, $f(2)=0$, $f(3)=3$ and $c=1/4$, with $Y=0$ when $D=1$ and $Y=1$ when $D=3$. Then $\\Pr[S=1]=1/2$, the observed degree probabilities are $(1/4,0,3/4)$, and the term $k=2$ of (2) is $0/0$. With the sums taken over $A=\\{1,3\\}$, I obtain $(1/4)/(1/2)=1/2$, which is $\\mathrm{E}[Y]$.\n\nOn $A$ the proof is correct and needs no further assumption. By Condition 1, $\\Pr[S=1\\mid D=k]>0$ for all $k\\in A$. It follows that $\\Pr[S=1]>0$, that $A=\\{k:\\Pr[D=k\\mid S=1]>0\\}$, and that $cf(k)>0$ for all $k\\in A$. By Bayes' rule and Condition 2, I have\n\\[\n\\frac{\\Pr[D=k\\mid S=1]}{f(k)} = \\frac{c}{\\Pr[S=1]}\\,\\Pr[D=k] \\qquad \\text{for all } k\\in A.\n\\]\nSumming over $A$, I find that the denominator of (2) equals $c/\\Pr[S=1]$ and, by Condition 1, that the numerator equals $(c/\\Pr[S=1])\\,\\mathrm{E}[Y]$. Their ratio is $\\mathrm{E}[Y]$.\n\nThe sentence of the Discussion needs a qualification of the same kind. If the known function $f$ vanishes at some $k$, then Conditions 1 and 2 imply that $\\Pr[D=k\\mid S=1]=0$, which can be checked in the sample. For an $f$ that is strictly positive on the observed degrees, the two conditions place no further restriction on the observed law, and the sentence holds as printed. In Corollary 1, $f(k)\\propto k$ has no zero on $\\{1,\\ldots,K\\}$."}, "url": "https://pmaronow.github.io/llm-audit/works/P021/#P021-N003", "work": {"arxiv_id": "1504.03574", "doi": "10.1016/j.spl.2015.07.003", "title": "Nonparametric Identification for Respondent-Driven Sampling", "url": "https://pmaronow.github.io/llm-audit/works/P021/"}, "work_id": "P021"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Five lines of text from the last paragraph of Section 7. A red box marks the clause 'total variance is greater than the conditional variance (2)'.", "caption": "Section 7, last paragraph, p. 859 (PDF p. 10). The marked clause compares the total variance with the conditional variance (2).", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-A001.png"}], "formal_release": null, "formal_statements": [], "headline": "With stochastic potential outcomes, the total variance is at least the expected conditional variance (2), and it is strictly greater only when the population average effect varies.", "id": "P022-A001", "location": "Section 7, last paragraph, p. 859 (PDF p. 10), sentence beginning “When the potential outcomes are stochastic”", "pdf_page": 10, "printed_page": "859", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "When the potential outcomes are stochastic, \\hl{the total variance is greater than the conditional variance (2)} because of the additional stochasticity.", "corrected": "When the potential outcomes are stochastic \\hl{and independent of the assignment, and the relevant variances exist, the total variance equals the expected conditional variance (2) plus the variance of the population average effect; it is at least the expected conditional variance, and it is strictly greater when the population average effect varies}.", "intended_meaning": "The sentence appeals to the law of total variance. Suppose the potential outcomes $Y$ are random and independent of the assignment, and write $V_N(Y)$ and $\\tau_N(Y)$ for the conditional variance (2) and the population average effect given $Y$. Since $\\mathrm{E}_X(\\hat\\tau_N\\mid Y)=\\tau_N(Y)$, I have\n\\[\n\\operatorname{Var}(\\hat\\tau_N) = \\mathrm{E}_Y\\{V_N(Y)\\} + \\operatorname{Var}_Y\\{\\tau_N(Y)\\}\n\\]\nwhenever these variances exist. The total variance is at least the expected conditional variance, and the inequality is strict exactly when $\\operatorname{Var}_Y\\{\\tau_N(Y)\\}>0$. This is the comparison the sentence intends, and it is correct. Read literally, the sentence omits the expectation and the case of equality. For example, let $N=n=4$, $m=2$ and $y_1=y_0=(Z,0,0,0)$, where $Z$ takes the values $0$ and $1$ with probability $1/2$ each. Then $\\tau_N=0$ and $V_N=Z^2/4$, the total variance equals $\\mathrm{E}(V_N)=1/8$, and it is below the conditional variance $1/4$ when $Z=1$. Nothing in the paper depends on the sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-A001", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The opening of Lemma 2. A red box marks the words 'Suppose conditions 1-3 of Proposition 1 hold. Then'. The display below it states that the supremum over y of the distance between G and G N tends to 0, and likewise for F and F N.", "caption": "Lemma 2, p. 860 (PDF p. 11): the hypothesis and the first assertion. The marked line states the hypothesis, conditions 1–3 of Proposition 1.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E001.png"}, {"alt": "Four lines from the proof of Lemma 2. A red box marks the sentence 'By conditions 2 and 3, this is satisfied for all N sufficiently large', which follows the requirement that the distance between G and G N at the points s i minus 1, and between their left limits at s i, be below eta 1 over 2.", "caption": "Proof of Lemma 2, p. 861 (PDF p. 12). The marked sentence obtains the convergence of the left limits $G_N(s_i-)$ from conditions 2 and 3.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E001-b.png"}, {"alt": "Conditions 2, 3 and 4 of Proposition 1. A red box marks condition 3: 'G N of y tends to G of y at any discontinuity point of G, and F N of y tends to F of y at any discontinuity point of F'.", "caption": "Proposition 1, p. 855 (PDF p. 6). The marked condition 3 concerns the values $G_N(y)$ and $F_N(y)$ at the discontinuity points of $G$ and $F$, and not the left limits.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E001-c.png"}, {"alt": "The first three lines of Remark 1. A red box marks the sentence 'Condition 3 is used to establish the functional convergence of (G N, F N) to (G, F)'.", "caption": "Remark 1, p. 855 (PDF p. 6). The marked sentence says what condition 3 is used for.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E001-d.png"}, {"alt": "Line chart with population size N from 4 to 100 on a logarithmic horizontal axis and the largest distribution-function error on the vertical axis, with tick labels from 0 to 3/4. Treatment outcomes (blue circles) and control outcomes (orange squares) both take the values 1/2, 3/5 and 4/7 at N equal to 4, 5 and 7. At N equal to 6, the treatment error is 2/3 and the control error is 1/2. Both series equal 1/2 at N equal to 8, 12, 16, 32 and 100. A dashed line at 1/2 is labelled 'counterexample: at least 1/2 for every N at least 4', and a dotted line at 0 is labelled 'Lemma 2 as printed: tends to 0'.", "caption": "The counterexample to Lemma 2. The largest error of the population distribution functions, $\\sup_y|G(y)-G_N(y)|$ and $\\sup_y|F(y)-F_N(y)|$, is shown at nine population sizes from 4 to 100 (exact values). It is never below $1/2$, where Lemma 2 as printed asserts convergence to 0.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E001-cdf-error.png"}], "formal_release": null, "formal_statements": [{"id": "P022-T1", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T1"}, {"id": "P022-T2", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T2"}, {"id": "P022-T3", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T3"}, {"id": "P022-T15", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T15"}, {"id": "P022-T18", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T18"}], "headline": "Lemma 2 does not hold under conditions 1–3 of Proposition 1: it needs in addition that the left limits $G_N(x-)$ and $F_N(x-)$ converge to $G(x-)$ and $F(x-)$ at the discontinuity points of $G$ and $F$, while Propositions 1 and 2 keep their statements through proofs that do not use the lemma.", "id": "P022-E001", "location": "Lemma 2, p. 860 (PDF p. 11), and its proof, p. 861 (PDF p. 12); condition 3 of Proposition 1 and Remark 1, p. 855 (PDF p. 6); the lemma is used on pp. 856, 862 and 867–869", "pdf_page": 11, "printed_page": "860", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P022-T1", "P022-T15", "P022-T18", "P022-T2", "P022-T3"], "text": {"as_printed": "\\emph{Condition 3 of Proposition 1 (p.~855).} $G_N(y)\\to G(y)$ at any discontinuity point of $G$, and $F_N(y)\\to F(y)$ at any discontinuity point of $F$;\n\n\\emph{Remark 1 (p.~855).} Condition 3 is used to establish the functional convergence of $(G_N,F_N)$ to $(G,F)$.\n\n\\emph{Lemma 2 (p.~860).} Suppose \\hl{conditions 1--3 of Proposition 1} hold. Then\n\\[\n\\sup_y|G(y)-G_N(y)|\\to 0 \\quad\\text{and}\\quad \\sup_y|F(y)-F_N(y)|\\to 0 .\n\\]\nIn addition, given $\\eta_1,\\eta_0>0$, there exist two positive integers $K_1(\\eta_1)$ and $K_0(\\eta_0)$ such that\n\\[\n\\begin{aligned}\n\\limsup_N\\Bigl\\{N\\,\\mathrm{P}\\Bigl(\\sup_y|G(y)-\\hat G_N(y)|\\ge\\eta_1\\Bigr)\\Bigr\\} &\\le \\frac{(1-\\theta\\rho)K_1(\\eta_1)}{\\theta\\rho\\,\\eta_1^2},\\\\\n\\limsup_N\\Bigl\\{N\\,\\mathrm{P}\\Bigl(\\sup_y|F(y)-\\hat F_N(y)|\\ge\\eta_0\\Bigr)\\Bigr\\} &\\le \\frac{\\{1-\\theta(1-\\rho)\\}K_0(\\eta_0)}{\\theta(1-\\rho)\\,\\eta_0^2}.\n\\end{aligned}\n\\]\nThe integers are nonincreasing in $\\eta$, and depend also on the limiting distribution $H$ of $(y_1,y_0)$.\n\n\\emph{Proof of Lemma 2 (p.~861).} hence $\\sup_y|G(y)-G_N(y)|<\\eta_1$ if $|G(s_{i-1})-G_N(s_{i-1})|<\\eta_1/2$ and $|G(s_i-)-G_N(s_i-)|<\\eta_1/2$ for all $i$. \\hl{By conditions 2 and 3, this is satisfied for all $N$ sufficiently large.}", "corrected": "\\emph{Lemma 2.} Suppose conditions 1--3 of Proposition 1 hold, that is, $(m/N,n/N)\\to(\\theta\\rho,\\theta)$ for $\\theta\\in(0,1]$ and $\\rho\\in(0,1)$; $H_N$ converges weakly to a limit distribution $H$ with marginals $G(y)=H(y,\\infty)$ and $F(y)=H(\\infty,y)$; and $G_N(y)\\to G(y)$ at any discontinuity point of $G$, and $F_N(y)\\to F(y)$ at any discontinuity point of $F$. Suppose in addition that $\\hl{G_N(y-)\\to G(y-)}$ at any discontinuity point of $G$, and $\\hl{F_N(y-)\\to F(y-)}$ at any discontinuity point of $F$. Then\n\\[\n\\sup_y|G(y)-G_N(y)|\\to 0 \\quad\\text{and}\\quad \\sup_y|F(y)-F_N(y)|\\to 0 .\n\\]\nIn addition, given $\\eta_1,\\eta_0>0$, there exist two positive integers $K_1(\\eta_1)$ and $K_0(\\eta_0)$ such that\n\\[\n\\begin{aligned}\n\\limsup_N\\Bigl\\{N\\,\\mathrm{P}\\Bigl(\\sup_y|G(y)-\\hat G_N(y)|\\ge\\eta_1\\Bigr)\\Bigr\\} &\\le \\frac{(1-\\theta\\rho)K_1(\\eta_1)}{\\theta\\rho\\,\\eta_1^2},\\\\\n\\limsup_N\\Bigl\\{N\\,\\mathrm{P}\\Bigl(\\sup_y|F(y)-\\hat F_N(y)|\\ge\\eta_0\\Bigr)\\Bigr\\} &\\le \\frac{\\{1-\\theta(1-\\rho)\\}K_0(\\eta_0)}{\\theta(1-\\rho)\\,\\eta_0^2}.\n\\end{aligned}\n\\]\nThe integers are nonincreasing in $\\eta$, and $K_1(\\eta)=K_0(\\eta)=2\\lceil 4/\\min(\\eta,1)\\rceil$ is an admissible choice.\n\n\\emph{Remark 1.} Condition 3, \\hl{together with the convergence of the left limits} $G_N(y-)\\to G(y-)$ and $F_N(y-)\\to F(y-)$ at the same points, is used to establish the functional convergence of $(G_N,F_N)$ to $(G,F)$. When the units of $U_N$ are independent and identically distributed samples from a superpopulation, both conditions hold with probability one, and both are satisfied if $G$ and $F$ are continuous.\n\n\\emph{Proposition 1.} The statement is unchanged. The proof outline and stages (i) and (ii) of the proof rest on the uniform convergence of $(\\hat G_N,\\hat F_N)$ to $(G,F)$. They are replaced by the weak convergence of $\\hat G_N$ and $\\hat F_N$ to $G$ and $F$ in probability and, by condition 4, the convergence of their quantile functions in $L^2(0,1)$ in probability. No condition at the discontinuity points is needed.\n\n\\emph{Proposition 2.} The statement is unchanged. The steps of the proof that apply Lemma 2 are replaced by the following bound. Suppose $|y_{1i}|\\le C$ and $|y_{0i}|\\le C$ for all $i$, and let $c>0$ be such that $m\\ge cN$ and $n-m\\ge cN$ for all large $N$, as condition 1 allows. Then for $B\\in\\{H,L\\}$, every $\\varepsilon>0$, and every such $N$ with $N\\ge\\max\\{2/c,\\ 16C^2/(c^2\\varepsilon)\\}$,\n\\[\n\\mathrm{P}\\bigl\\{N|\\hat V_N^B-V_N^B|\\ge\\varepsilon\\bigr\\} \\le \\frac{16\\,(16/c+8)^2\\,C^4}{cN\\varepsilon^2}.\n\\]", "evidence": "The evidence is an exact counterexample to the lemma, together with derivations of the corrected statements.\n\n\\emph{The counterexample.} For $i=1,2,\\ldots$, let $(a_i,b_i)$ run through the cycle $(0,0),(0,1),(1,0),(1,1)$, and define\n\\[\ny_{1i} = a_i - \\frac{1}{i+2}, \\qquad y_{0i} = b_i - \\frac{1}{i+2}.\n\\]\nLet $U_N$ consist of the first $N$ units, for $N\\ge 4$, and set $n=N$ and $m=\\lfloor N/2\\rfloor$. The populations are nested and the outcomes are fixed. Condition 1 holds with $\\theta=1$ and $\\rho=1/2$. Condition 2 holds: $H_N$ converges weakly to the distribution $H$ with mass $1/4$ at each of the points $(0,0)$, $(0,1)$, $(1,0)$ and $(1,1)$. Both marginals of $H$ have mass $1/2$ at $0$ and at $1$, so that $G(x)=F(x)=1/2$ for $0\\le x<1$. Condition 3 holds as well. Since $y_{1i}\\le 0$ exactly when $a_i=0$, I have $G_N(0)\\to 1/2 = G(0)$, and since every outcome is below $1$, I have $G_N(1)=1=G(1)$; the same holds for $F_N$. All outcomes lie in $(-1,1)$, which implies condition 4, the boundedness assumed in Proposition 2 and the moment condition of Corollary 1. The distribution $H$ is not concentrated on a line.\n\nSet $t_N = 1 - 1/\\{2(N+2)\\}$. For $i\\le N$, I have $y_{1i}\\le 1-1/(N+2) < t_N$ and $y_{0i}\\le 1-1/(N+2) < t_N$, and $t_N<1$. Then $G_N(t_N)=F_N(t_N)=1$, and $\\hat G_N(t_N)=\\hat F_N(t_N)=1$ for every assignment, while $G(t_N)=F(t_N)=1/2$. This implies that, for every $N\\ge 4$ and every assignment,\n\\[\n\\sup_y|G(y)-G_N(y)|\\ge \\frac12, \\qquad \\sup_y|G(y)-\\hat G_N(y)|\\ge\\frac12,\n\\]\nand likewise for $F$, $F_N$ and $\\hat F_N$. The first assertion of the lemma fails. For $\\eta_1=\\eta_0=1/4$, the two probabilities in the second assertion equal $1$, and $N$ times each of them tends to infinity, while the right-hand sides equal $16K_1(1/4)$ and $16K_0(1/4)$. There exist no integers $K_1(1/4)$ and $K_0(1/4)$ for which the second assertion holds. Note that the left limits that the proof requires do not converge: $G_N(0-)\\to 1/2$ while $G(0-)=0$, and $G_N(1-)=1$ while $G(1-)=1/2$. The table lists exact values at three population sizes, obtained by enumerating every assignment.\n\\begin{audittable}{rrrrr}{The counterexample at three population sizes, with $\\eta_1=1/4$ and $m=N/2$. The population error is $\\sup_y|G(y)-G_N(y)|$. The smallest sample error is the minimum of $\\sup_y|G(y)-\\hat G_N(y)|$ over all assignments, and the probability is that of the event $\\sup_y|G(y)-\\hat G_N(y)|\\ge 1/4$ over all assignments. Every entry is exact, and the same values hold for $F$, $F_N$ and $\\hat F_N$.}\n$N$ & Assignments & Population error & Smallest sample error & Probability \\\\\n4 & 6 & $1/2$ & $1/2$ & 1 \\\\\n8 & 70 & $1/2$ & $1/2$ & 1 \\\\\n12 & 924 & $1/2$ & $1/2$ & 1 \\\\\n\\end{audittable}\n\n\\emph{The added condition suffices.} I write $P$ for $G$ or $F$, $P_N$ for the corresponding distribution function in the population, and $\\hat P_N$ for the one in the sample, with $k$ sampled units and $\\alpha=\\lim k/N>0$. By weak convergence, I have $P_N(x)\\to P(x)$ and $P_N(x-)\\to P(x-)$ at every continuity point $x$ of $P$. At the discontinuity points, both limits hold by condition 3 and the added condition. Fix $\\eta$ such that $0<\\eta\\le 1$, and set $R=\\lceil 4/\\eta\\rceil$. The points $\\pm\\infty$ and the distinct quantiles of $P$ at the levels $j/R$, for $1\\le j\\le R-1$, form a partition with at most $R$ intervals such that $P(t-)-P(s)\\le 1/R$ for adjacent points $s<t$. Since distribution functions are monotone, for every distribution function $A$, I have\n\\[\n\\sup_x|P(x)-A(x)| \\le \\frac1R + \\max\\bigl\\{|P(s)-A(s)|,\\ |P(t-)-A(t-)|\\bigr\\},\n\\]\nwhere the maximum is over the adjacent points $s<t$ of the partition. Taking $A=P_N$, I obtain the uniform convergence. For $A=\\hat P_N$, each of the at most $2R$ values $\\hat P_N(s)$ and $\\hat P_N(t-)$ has mean $P_N(s)$ or $P_N(t-)$ and variance at most $(N-k)/\\{4(N-1)k\\}$. Using Chebyshev's inequality and the union bound, I obtain\n\\[\n\\limsup_N N\\,\\mathrm{P}\\Bigl(\\sup_x|P(x)-\\hat P_N(x)|\\ge\\eta\\Bigr) \\le \\frac{2R(1-\\alpha)}{\\alpha\\eta^2}.\n\\]\nThis is the printed bound with $K(\\eta)=2\\lceil 4/\\eta\\rceil$. For $\\eta>1$ the event is empty.\n\n\\emph{Proposition 1 without the lemma.} Let $c>0$ be such that $m\\ge cN$ and $n-m\\ge cN$ for all large $N$. For fixed $x$, the value $\\hat G_N(x)$ has mean $G_N(x)$ and variance at most $1/(4cN)$, so that $\\hat G_N(x)-G_N(x)\\to 0$ in probability. By condition 2, $\\hat G_N$ converges weakly to $G$ in probability, and the same holds for $\\hat F_N$ and $F$. For every sample and every $T>0$, I have\n\\[\n\\int x^2\\,\\mathbf{1}\\{|x|>T\\}\\,d\\hat G_N(x) \\le \\frac{N}{m}\\int x^2\\,\\mathbf{1}\\{|x|>T\\}\\,dG_N(x),\n\\]\nand by condition 4 the sampled distributions are uniformly square-integrable. Weak convergence and uniform square-integrability imply that the quantile functions converge in $L^2(0,1)$, in probability for the samples. The means, the second moments and the two integrals of products of quantile functions in (7)--(9) are continuous in the quantile functions with respect to that norm. I conclude that $NV_N^H$ and $NV_N^L$ have the limits stated in Proposition 1, and that $N(\\hat V_N^B-V_N^B)\\to 0$ in probability for $B\\in\\{H,L\\}$. Condition 3 is not used.\n\n\\emph{Proposition 2 without the lemma.} Suppose $|y_{1i}|\\le C$ and $|y_{0i}|\\le C$, and write $P_{1N}=G_N$, $P_{0N}=F_N$, $k_1=m$ and $k_0=n-m$. For $a\\in\\{0,1\\}$, define\n\\[\nD_a=\\int_{-C}^{C}|\\hat P_{aN}(x)-P_{aN}(x)|\\,dx .\n\\]\nBy the Cauchy--Schwarz inequality and the bound on the variance above, I have $\\mathrm{E}\\,D_a^2\\le C^2/(cN)$, so that $\\mathrm{E}(D_1+D_0)^2\\le 4C^2/(cN)$. The quantile functions $q_A$ and $q_P$ of two distributions on $[-C,C]$ with distribution functions $A$ and $P$ satisfy $\\int_0^1|q_A(u)-q_P(u)|\\,du=\\int_{-C}^{C}|A(x)-P(x)|\\,dx$. It follows that the error of the sample mean in arm $a$ is at most $D_a$, the error of the unbiased variance estimator is at most $8CD_a+C^2/(k_a-1)$, and the error of each estimated covariance bound in (8) is at most $2C(D_1+D_0)$. Substituting these bounds into (9), I obtain, for $N\\ge 2/c$ and $B\\in\\{H,L\\}$,\n\\[\nN|\\hat V_N^B-V_N^B| \\le \\Bigl(\\frac{16}{c}+8\\Bigr)C(D_1+D_0)+\\frac{8C^2}{c^2N}.\n\\]\nMarkov's inequality then implies the bound stated in the correction below.", "problem": "Lemma 2 makes two assertions about the distribution functions $G_N$ and $F_N$ of the potential outcomes in the population $U_N$ and their sample analogues $\\hat G_N$ and $\\hat F_N$. The first is that $G_N$ and $F_N$ converge uniformly to their limits $G$ and $F$. The second is that the probability of a uniform error of at least $\\eta$ between a sample distribution function and its limit is at most of order $1/N$. Neither follows from conditions 1--3. Weak convergence (condition 2) implies that $G_N(y)\\to G(y)$ at the points where $G$ is continuous, and condition 3 adds the points where $G$ jumps. Uniform convergence also requires the left limits to converge at the jumps, that is, $G_N(y-)\\to G(y-)$, and no condition supplies this. The proof on p.~861 needs $|G(s_i-)-G_N(s_i-)|<\\eta_1/2$ for large $N$ and obtains it ``by conditions 2 and 3''. At a jump of $G$ this step can fail. If the outcomes of a population approach a jump point of $G$ from below without reaching it, then conditions 1--4 can hold while the uniform error stays at $1/2$ or above for every $N$ and every assignment.\n\nThe lemma underlies Remark 1, the proof outline on p.~856, stages (i) and (ii) in the proof of Proposition 1 on p.~862, and the bounds for the last terms in the proof of Proposition 2 on pp.~867--869. Those steps are not justified by the stated conditions when $G$ or $F$ has a jump. The conclusions of Propositions 1 and 2 are not affected, since both hold under their printed hypotheses by arguments that do not pass through uniform convergence. To repair the lemma, one regularity condition must be added. That condition holds when $G$ and $F$ are continuous, and it holds with probability one when the units are independent and identically distributed draws from a superpopulation.", "what_survives": "Lemma 1 and the sharpness of the interval $[V_N^L,V_N^H]$, the bounds and estimators (7)--(9), and the ordering of $\\hat V_N^H$ and $\\hat V_N^L$ relative to Neyman's bounds do not depend on Lemma 2. Propositions 1 and 2 keep their hypotheses and conclusions. With them stand the consistency of $\\hat V_N^H$ used in Corollary 1 and the argument for asymptotically conservative Wald intervals. This error does not change the computations in the application, the simulations or Appendix C. Three things change: Lemma 2 gains a hypothesis, Remark 1 is reworded, and the proof outline and the steps in the proofs of Propositions 1 and 2 that rest on the lemma are replaced."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-E001", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The final portion of the R function sharp.var. A red box marks the three lines 'p_i <- unique(sort(c(seq(0,n-m,1)/(n-m),seq(0,m,1)/m))) -', '.Machine$double.eps^.5' and 'p_i[1] <- .Machine$double.eps^.5'. A second red box marks '(p_i-p_i_minus)' inside the return expression.", "caption": "Appendix B, R code, p. 869 (PDF p. 20). The upper mark is the three lines that shift the partition points by $\\sqrt{\\texttt{.Machine\\$double.eps}}=2^{-26}$; the lower mark is the weights formed from the shifted points.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E002.png"}, {"alt": "Equation (8): the estimated upper and lower covariance bounds as sums over i from 1 to P. Red boxes mark the weight '(p i minus p i minus 1)' in each of the two sums.", "caption": "Equation (8), p. 854 (PDF p. 5). The marked weights are the widths $p_i-p_{i-1}$ of the cells of the partition, which sum to 1.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P022-T8", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T8"}, {"id": "P022-T9", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T9"}], "headline": "The printed function shifts the partition points by $2^{-26}$ and uses the shifted points as integration weights, which sum to $1-2^{-25}$, and its output is not the estimate of equations (8)–(9); the widths $p_i-p_{i-1}$ must be kept, with the quantiles selected at the midpoints of the cells.", "id": "P022-E002", "location": "Appendix B, R code of sharp.var, p. 869 (PDF p. 20): the lines that set p_i and the weights in the return expression; equation (8), p. 854 (PDF p. 5)", "pdf_page": 20, "printed_page": "869", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P022-T8", "P022-T9"], "text": {"as_printed": "\\emph{Appendix B, p.~869, ninth to seventeenth lines of the function.}\n\n$\\texttt{p\\_i <- unique(sort(c(seq(0,n-m,1)/(n-m),seq(0,m,1)/m)))~}\\hl{\\texttt{-}}$\n\n$\\hl{\\texttt{.Machine\\$double.eps\\textasciicircum.5}}$\n\n$\\hl{\\texttt{p\\_i[1] <- .Machine\\$double.eps\\textasciicircum.5}}$\n\n$\\texttt{yti <- yt[ceiling(p\\_i*m)]}$\n\n$\\texttt{yci <- yc[ceiling(p\\_i*(n-m))]}$\n\n$\\texttt{p\\_i\\_minus <- c(NA,p\\_i[1: (length(p\\_i)-1)])}$\n\n$\\texttt{return(((N-m)/m * FPvar(yt,N) + (N-(n-m))/(n-m) * FPvar(yc,N)}$\n\n$\\texttt{~~+ 2*sum((}\\hl{\\texttt{(p\\_i-p\\_i\\_minus)}}\\texttt{*yti*yci)[2:length(p\\_i)])}$\n\n$\\texttt{~~- 2*mean(yt)*mean(yc))/(N-1))}$\n\n\\emph{Equation (8), p.~854.}\n\\[\n\\begin{aligned}\n\\hat\\sigma_N^H(y_1,y_0)&=\\sum_{i=1}^P(p_i-p_{i-1})\\,y_{1[i]}y_{0[i]}-\\hat\\mu_N(y_1)\\hat\\mu_N(y_0),\\\\\n\\hat\\sigma_N^L(y_1,y_0)&=\\sum_{i=1}^P(p_i-p_{i-1})\\,y_{1[i]}y_{0[P+1-i]}-\\hat\\mu_N(y_1)\\hat\\mu_N(y_0),\n\\end{aligned}\n\\]", "corrected": "Keep the partition $0=p_0<p_1<\\cdots<p_P=1$ of equation (8) and its widths, and select the order statistics at the midpoints $u_r=(p_{r-1}+p_r)/2$. Let $k=n-m$, let $\\bar y_t$ and $\\bar y_c$ be the sample means, let $Q_t$ be the quantile function of the treatment sample, and let $Q_c^H$ and $Q_c^L$ be the quantile functions of the control sample in ascending and in descending order. Then\n\\[\n\\hat\\sigma_N^{B}(y_1,y_0)=\\sum_{r=1}^P(\\hl{p_r-p_{r-1}})\\,\\bigl(Q_t(u_r)-\\bar y_t\\bigr)\\bigl(Q_c^{B}(u_r)-\\bar y_c\\bigr), \\qquad B\\in\\{H,L\\},\n\\]\nand this covariance is substituted into (9). The same value is obtained from the weights $w_{ij}=[\\min(ik,jm)-\\max\\{(i-1)k,(j-1)m\\}]_+/(mk)$ applied to the centered products of the $i$th sorted treatment outcome and the $j$th sorted control outcome, with the control order reversed for the lower bound. For finite $N$, in R:\n\n$\\texttt{sharp.var <- function(yt,yc,N=length(c(yt,yc)),upper=TRUE) \\{}$\n\n$\\texttt{~~m <- length(yt)}$\n\n$\\texttt{~~k <- length(yc)}$\n\n$\\texttt{~~stopifnot(m >= 2, k >= 2, N >= m+k, all(is.finite(c(yt,yc))))}$\n\n$\\texttt{~~FPvar <- function(x,N) (N-1)/(N*(length(x)-1)) *}$\n\n$\\texttt{~~~~sum((x - mean(x))\\textasciicircum 2)}$\n\n$\\texttt{~~vt <- FPvar(yt,N)}$\n\n$\\texttt{~~vc <- FPvar(yc,N)}$\n\n$\\texttt{~~yt <- sort(yt) - mean(yt)}$\n\n$\\texttt{~~yc <- sort(yc,decreasing=!upper) - mean(yc)}$\n\n$\\texttt{~~p <- unique(sort(c(seq(0,k,1)/k,seq(0,m,1)/m)))}$\n\n$\\texttt{~~lo <- p[-length(p)]}$\n\n$\\texttt{~~hi <- p[-1]}$\n\n$\\texttt{~~mid <- (lo+hi)/2}$\n\n$\\texttt{~~cov <- sum((hi-lo)*yt[ceiling(mid*m)]*yc[ceiling(mid*k)])}$\n\n$\\texttt{~~((N-m)/m * vt + (N-k)/k * vc + 2*cov)/(N-1)}$\n\n$\\texttt{\\}}$", "evidence": "I checked an exact counterexample in rational arithmetic and reproduced it in R. Let $k=n-m$, and suppose every cell of the partition is wider than $2e$, so that the shifted points select the same order statistics as (8). Let $a_1$ be the smallest outcome under treatment, and let $b_*$ be the smallest outcome under control for the upper bound and the largest for the lower bound. Since only the first weight changes, the output of the printed function differs from the estimate (9) by exactly\n\\[\n-\\frac{4e\\,a_1b_*}{N-1}, \\qquad e=2^{-26}.\n\\]\nWith all outcomes equal to $1$, $N=n=4$ and $m=2$, the randomization variance and both estimates (9) are $0$, and the printed function returns $-4e/3=-1/50331648$ for both bounds; R returns $-1.9868\\times10^{-8}$. The table lists this case and three others.\n\\begin{audittable}{lrlrr}{The estimate of equation (9) and the output of the printed function, in exact arithmetic with $e=2^{-26}$. The line break in the definition of \\texttt{FPvar} is joined.}\nOutcomes in the two arms & $N$ & Bound & Equation (9) & Printed function \\\\\n$(1,1)$ in both & 4 & upper & $0$ & $-1/50331648\\approx-1.99\\times10^{-8}$ \\\\\n$(1,1)$ in both & 4 & lower & $0$ & $-1/50331648\\approx-1.99\\times10^{-8}$ \\\\\n$(0,1)$ in both & 4 & upper & $5/12$ & $5/12$ \\\\\n$(0,1)$ in both & 4 & lower & $1/12$ & $1/12$ \\\\\n$(10^6,10^6+1)$ in both & 4 & upper & $5/12$ & $-244135505/12288\\approx-19867.80$ \\\\\n$(10^6,10^6+1)$ in both & 4 & lower & $1/12$ & $-15624950089/786432\\approx-19868.15$ \\\\\ntreatment $(-2,1,4)$, control $(1,3)$ & 8 & upper & $179/56$ & $179/56+1/58720256$ \\\\\ntreatment $(-2,1,4)$, control $(1,3)$ & 8 & lower & $115/56$ & $115/56+3/58720256$ \\\\\n\\end{audittable}\nThe third and fourth rows agree because $a_1=0$. The fifth and sixth rows are the third and fourth with $10^6$ added to every outcome, which leaves (9) unchanged. For the application, the cells of the partition have width at least $1/(781\\cdot780)>2e$, and by the displayed formula the discrepancy is at most $4e\\cdot150^2/1560=375/436207616\\approx8.6\\times10^{-7}$ USD$^2$. Since the data are not available, no entry of Table 1 or Table 2 was recomputed, and none is shown to change.", "problem": "Equation (8) weights each product of order statistics by the width $p_i-p_{i-1}$ of a cell of the partition $P_{m,n}$ of $[0,1]$, and the widths sum to $1$. The function subtracts $e=\\sqrt{\\texttt{.Machine\\$double.eps}}=2^{-26}$ from every partition point, resets the first point to $e$, and uses the shifted points both to select the order statistics and as weights. When every cell is wider than $2e$, the shifted points select the order statistics of (8). The first weight, however, becomes $p_1-p_0-2e$, and the weights sum to $1-2e=1-2^{-25}$. A fraction $2e$ of the first product is lost from the sum. The output is then not invariant to adding a constant to every outcome, as the estimate (9) is, and it can be negative. The discrepancy is about $6\\times10^{-8}$ times the product of two extreme outcomes, divided by $N-1$. For the application, with $N=1561$ and outcomes between \\$0 and \\$150, it is at most $8.6\\times10^{-7}$ USD$^2$, against a reported estimate of 0.186. It vanishes for the upper bound when the smallest outcome in either arm is zero.", "what_survives": "The definitions (7)--(9), the sharpness of the bounds and the asymptotic results concern the estimators as defined in the text and are unaffected. Table 3 was computed by numerical integration of Beta quantile functions and does not use this function. No number in Table 1 or Table 2 is shown to be wrong."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-E002", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The paragraph that introduces the R function sharp.var and the first eight lines of its code. A red box marks the words 'N (the total number of units in the population)'. A second red box marks the expression '(N-1)/(N*(length(x)-1))' in the line that defines FPvar.", "caption": "Appendix B, p. 869 (PDF p. 20). The upper mark is the description of the argument N; the lower mark is the expression in FPvar that evaluates Inf/Inf when N=Inf.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P022-T10", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T10"}], "headline": "Called with N=Inf, the printed function returns NaN; the infinite-population estimate $s_T^2/m+s_C^2/(n-m)$ requires a branch of its own.", "id": "P022-E003", "location": "Appendix B, description of the argument N and R code of sharp.var, p. 869 (PDF p. 20); the statements about $N=\\infty$ on pp. 850 and 852 are correct", "pdf_page": 20, "printed_page": "869", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P022-T10"], "text": {"as_printed": "\\emph{Appendix B, p.~869.} The other inputs are \\texttt{yt} (the observed outcomes under treatment), \\texttt{yc} (the observed outcomes under control) and \\hl{\\texttt{N} (the total number of units in the population)}.\n\n$\\texttt{FPvar <- function(x,N)~}\\hl{\\texttt{(N-1)/(N*(length(x)-1))}}$\n\n\\emph{Introduction, p.~850.} Our estimator is also applicable to all possible cases of $N$ and $n$ ($n=N<\\infty$, $n<N<\\infty$, and $N=\\infty$), thus providing a unified treatment of the problem.", "corrected": "Join the two lines that define \\texttt{FPvar} (\\fid{P022-E004}). After the two lines that define \\texttt{m} and \\texttt{n}, insert\n\n$\\hl{\\texttt{if (is.infinite(N) \\&\\& N > 0) return(stats::var(yt)/m + stats::var(yc)/(n-m))}}$\n\nWith this line the function returns $s_T^2/m+s_C^2/(n-m)$ for \\texttt{N=Inf} in both modes, and every call with finite \\texttt{N} is unchanged. The arm sizes $m\\ge 2$ and $n-m\\ge 2$ assumed on p.~851 make both sample variances well defined. The description may then read: \\texttt{N} (the total number of units in the population\\hl{, or \\texttt{Inf} for an infinite population}).", "evidence": "I checked an exact example and derived the limit. For treatment and control outcomes $(0,1)$ and $(0,1)$, the printed function, with the line break in \\texttt{FPvar} joined, returns \\texttt{NaN} for \\texttt{N=Inf} with \\texttt{upper=TRUE} and with \\texttt{upper=FALSE}. For finite $N$ it returns $(N-2)/(2N)+1/\\{2(N-1)\\}$ for the upper bound and $(N-2)/(2N)-1/\\{2(N-1)\\}$ for the lower bound, that is, $5/12$ and $1/12$ at $N=4$ and $41/90$ and $31/90$ at $N=10$, and both expressions tend to $1/2$. In general, let $s_T^2$ and $s_C^2$ be the sample variances with denominators $m-1$ and $k-1$, where $k=n-m$. Since $\\texttt{FPvar(yt,N)}=(N-1)s_T^2/N$, the output for finite $N$ is\n\\[\n\\Bigl(\\frac1m-\\frac1N\\Bigr)s_T^2+\\Bigl(\\frac1k-\\frac1N\\Bigr)s_C^2+\\frac{2\\hat c}{N-1},\n\\]\nwhere $\\hat c$ is the covariance term, which does not depend on $N$. Letting $N\\to\\infty$, I obtain $s_T^2/m+s_C^2/k$ for both bounds.", "problem": "The paper says that the estimator covers an infinite population, and that as $N\\to\\infty$ with $m$ and $n$ fixed it becomes the usual variance estimate for a difference of independent means. Both statements are correct. Appendix B presents \\texttt{sharp.var} as the implementation, with \\texttt{N} the size of the population, and the value of an infinite \\texttt{N} in R is \\texttt{Inf}. With \\texttt{N=Inf}, the expression in \\texttt{FPvar} evaluates \\texttt{Inf/Inf}, which is \\texttt{NaN}, and the return expression propagates \\texttt{NaN}. The function returns \\texttt{NaN} for both bounds. The paper documents no call with \\texttt{N=Inf}. This is a gap in the code and not a false statement.", "what_survives": "The limit stated on pp.~850 and 852, the formulas for finite $N$ and the asymptotic results are unaffected. The application has $N=n=1561$, and the simulations use the same finite population. No reported number involves a call with \\texttt{N=Inf}."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-E003", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The first eight lines of the R function sharp.var. A red box marks the two lines 'FPvar <- function(x,N) (N-1)/(N*(length(x)-1))' and '* sum((x - mean(x))^2)'.", "caption": "Appendix B, R code, p. 869 (PDF p. 20). The two marked lines define FPvar; the second begins with the multiplication sign.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E004.png"}], "formal_release": null, "formal_statements": [], "headline": "The multiplication sign that opens the second line of the definition of FPvar belongs at the end of the first line; as printed, the code does not parse.", "id": "P022-E004", "location": "Appendix B, R code of sharp.var, fourth and fifth printed lines, p. 869 (PDF p. 20)", "pdf_page": 20, "printed_page": "869", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Appendix B, p.~869, fourth and fifth lines of the function.}\n\n$\\texttt{FPvar <- function(x,N) (N-1)/(N*(length(x)-1))}$\n\n$\\texttt{~~}\\hl{\\texttt{*}}\\texttt{~sum((x - mean(x))\\textasciicircum 2)}$", "corrected": "\\emph{Fourth and fifth lines of the function.}\n\n$\\texttt{FPvar <- function(x,N) (N-1)/(N*(length(x)-1))~}\\hl{\\texttt{*}}$\n\n$\\texttt{~~sum((x - mean(x))\\textasciicircum 2)}$", "evidence": "I checked the code by running it. The block copied with its printed line breaks fails to parse at the place stated. With the operator moved to the end of the preceding line and no other change, the function parses and runs.", "problem": "R ends a statement at a line break when the expression read so far is complete. The first of the two lines is a complete function definition. The second then begins a new statement with a binary operator, and the parser stops with the message \\texttt{unexpected '*'} at line 5, column 5 of the function.", "what_survives": "The formula intended for \\texttt{FPvar} and the estimator are unchanged by the syntax repair. The reported numbers were not recomputed. Two further defects of the same function are \\fid{P022-E002} and \\fid{P022-E003}."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-E004", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Equations (11) and (12) with the sentence that introduces them. A red box marks the second equals sign in equation (11), between the term with sigma squared N and the fraction N minus m over 4 (N minus 1) m. In equation (12) the corresponding sign is 'less than or equal to'.", "caption": "Proof of Lemma 2, equations (11) and (12), p. 861 (PDF p. 12). The marked sign in (11) is printed as an equality; the corresponding sign in (12) is an inequality.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P022-T4", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T4"}, {"id": "P022-T5", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T5"}], "headline": "The last relation in equation (11) is an upper bound, not an equality.", "id": "P022-E005", "location": "Proof of Lemma 2, equation (11), p. 861 (PDF p. 12)", "pdf_page": 12, "printed_page": "861", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P022-T4", "P022-T5"], "text": {"as_printed": "Noting that $\\sigma_N^2(I\\{y_1\\le y\\})=G_N(y)\\{1-G_N(y)\\}\\le 1/4$ and similarly $\\sigma_N^2(I\\{y_0\\le y\\})\\le 1/4$, we have for all $y$,\n\\[\n\\operatorname{Var}_X\\hat G_N(y)=\\frac{N-m}{(N-1)m}\\,\\sigma_N^2\\bigl(I\\{y_1\\le y\\}\\bigr)\\ \\hl{=}\\ \\frac{N-m}{4(N-1)m}, \\qquad (11)\n\\]", "corrected": "Let $G_N$ be the distribution function of $y_1$ in the population $U_N$, and let $\\hat G_N$ be the distribution function of the observed outcomes of the $m$ units assigned to treatment. For all $y$,\n\\[\n\\operatorname{Var}_X\\hat G_N(y)=\\frac{N-m}{(N-1)m}\\,G_N(y)\\{1-G_N(y)\\}\\ \\hl{\\le}\\ \\frac{N-m}{4(N-1)m}. \\qquad (11)\n\\]", "evidence": "I used an exact counterexample. Let $N=4$ and $m=2$, and take $y$ below every potential outcome under treatment. Then $\\hat G_N(y)=0$ in each of the six assignments, so that $\\operatorname{Var}_X\\hat G_N(y)=0$, while the right-hand side of (11) is $(4-2)/(4\\cdot 3\\cdot 2)=1/12$.", "problem": "The variance $\\sigma_N^2(I\\{y_1\\le y\\})=G_N(y)\\{1-G_N(y)\\}$ is at most $1/4$, and it equals $1/4$ only when $G_N(y)=1/2$. The last relation in (11) is therefore an inequality. The sentence before the display and the companion display (12) state the bound correctly.", "what_survives": "The proof substitutes (11) into (10), and that step uses only the upper bound. The bound that follows is unchanged. The separate problem with Lemma 2 is \\fid{P022-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-E005", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Three lines of text and the first line of a display from the proof of Proposition 2. Red boxes mark '1 minus epsilon 1' twice: in the sentence 'To bound the probability that the last term exceeds 1 minus epsilon 1', and at the right end of the display, where the probability concerns an integral being less than 1 minus epsilon 1.", "caption": "Proof of Proposition 2, p. 867 (PDF p. 18). The marked tolerance $1-\\varepsilon_1$ is assigned to the last term, in the sentence and in the first line of the display that follows.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E006.png"}, {"alt": "The sentence 'The bounds (17)-(23) imply that' and the display below it. A red box marks the symbol V with subscript N, without a superscript, inside the absolute value 'V hat H N minus V N'.", "caption": "Proof of Proposition 2, display after equation (23), p. 868 (PDF p. 19). The marked symbol is printed $V_N$, without the superscript $H$.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-E006-b.png"}], "formal_release": null, "formal_statements": [{"id": "P022-T6", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T6"}, {"id": "P022-T7", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T7"}], "headline": "In the proof of the first bound of Proposition 2, the last term must receive a positive share of the tolerance $\\varepsilon_1$, such as $\\varepsilon_1/2$, in place of $1-\\varepsilon_1$, and the concluding display concerns $V_N^H$, not $V_N$.", "id": "P022-E006", "location": "Proof of Proposition 2, bound for the last term, p. 867 (PDF p. 18), through equation (23) and the display after it, p. 868 (PDF p. 19)", "pdf_page": 18, "printed_page": "867", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P022-T6", "P022-T7"], "text": {"as_printed": "\\emph{p.~865.} Let $\\nu_1,\\ldots,\\nu_8>0$ be a tuple whose sum is $\\hl{\\varepsilon_1}$.\n\n\\emph{p.~867.} To bound the probability that the last term exceeds $\\hl{1-\\varepsilon_1}$, first note that $|u-u'|<\\eta$ and $|v-v'|<\\eta$ implies $|\\min(u,v)-\\min(u',v')|<\\eta$. This gives the third inequality below:\n\\[\n\\begin{aligned}\n&\\mathrm{P}\\Bigl[\\frac{2N}{N-1}\\Bigl|\\int_{[-C,C]^2}\\bigl\\{\\hat H_N^H(y_1,y_0)-H_N^H(y_1,y_0)\\bigr\\}\\,dy_1\\,dy_0\\Bigr|<\\hl{1-\\varepsilon_1}\\Bigr]\\\\\n&\\qquad\\ge \\mathrm{P}\\Bigl\\{\\sup_{y_1,y_0}\\bigl|\\hat H_N^H(y_1,y_0)-H_N^H(y_1,y_0)\\bigr|<\\frac{(N-1)(\\hl{1-\\varepsilon_1})}{8NC^2}\\Bigr\\}\n\\end{aligned}\n\\]\n\n\\emph{p.~868, equation (23) and the text after it.}\n\\[\n\\limsup_N N\\,\\mathrm{P}\\Bigl[\\frac{2N}{N-1}\\Bigl|\\int_{[-C,C]^2}\\bigl\\{\\hat H_N^H(y_1,y_0)-H_N^H(y_1,y_0)\\bigr\\}\\,dy_1\\,dy_0\\Bigr|\\ge\\hl{1-\\varepsilon_1}\\Bigr]\\le C^4\\kappa_1(\\varepsilon_1).\n\\]\nThe bounds (17)--(23) imply that\n\\[\n\\limsup_N N\\,\\mathrm{P}\\bigl(N|\\hat V_N^H-\\hl{V_N}|\\ge\\varepsilon\\bigr)\\le C^4\\Bigl(\\sum_{i=1}^8\\frac{c_i}{\\nu_i^2}+\\kappa_1(\\varepsilon_1)\\Bigr).\n\\]\nBy minimizing the right-hand side over $\\nu_1,\\ldots,\\nu_8>0$ subject to the constraint $\\nu_1+\\cdots+\\nu_8=\\hl{\\varepsilon_1}$, the sum in the parenthesis can be absorbed into $\\kappa_1(\\varepsilon_1)$, yielding the desired convergence rate for $N\\hat V_N^H$.", "corrected": "\\emph{p.~865, for the bound on $\\hat V_N^H$.} Let $\\nu_1,\\ldots,\\nu_8>0$ be a tuple whose sum is $\\hl{\\varepsilon_1/2}$.\n\n\\emph{pp.~867--868.} To bound the probability that the last term exceeds $\\hl{\\varepsilon_1/2}$, replace $1-\\varepsilon_1$ by $\\varepsilon_1/2$ in every line of the two displays that follow, with $0<\\xi<\\varepsilon_1/(16C^2)$ \\hl{for $C>0$ (the case $C=0$ is trivial)}. Equation (23) becomes\n\\[\n\\limsup_N N\\,\\mathrm{P}\\Bigl[\\frac{2N}{N-1}\\Bigl|\\int_{[-C,C]^2}\\bigl\\{\\hat H_N^H(y_1,y_0)-H_N^H(y_1,y_0)\\bigr\\}\\,dy_1\\,dy_0\\Bigr|\\ge\\hl{\\varepsilon_1/2}\\Bigr]\\le C^4\\kappa_1(\\varepsilon_1).\n\\]\nThe bounds (17)--(23) imply that\n\\[\n\\limsup_N N\\,\\mathrm{P}\\bigl(N|\\hat V_N^H-\\hl{V_N^H}|\\ge\\varepsilon\\bigr)\\le C^4\\Bigl(\\sum_{i=1}^8\\frac{c_i}{\\nu_i^2}+\\kappa_1(\\varepsilon_1)\\Bigr).\n\\]\nBy minimizing the right-hand side over $\\nu_1,\\ldots,\\nu_8>0$ subject to the constraint $\\nu_1+\\cdots+\\nu_8=\\hl{\\varepsilon_1/2}$, the sum in the parenthesis can be absorbed into $\\kappa_1(\\varepsilon_1)$.\n\nIn the argument for $\\hat V_N^L$ the tuple keeps the sum $\\varepsilon_1$, as printed, and the last term keeps the tolerance $\\varepsilon_2+\\varepsilon_3$.", "evidence": "The evidence is a derivation with two exact examples. For the tolerance, take $\\varepsilon=3$ and $\\varepsilon_1=\\varepsilon_2=\\varepsilon_3=1$. The event in (23) is then that a nonnegative quantity is at least $0$. It has probability $1$, and the left-hand side of (23) is infinite. For the target, let $N=n=4r$ and $m=N/2$, with $r$ units at each of the points $(0,0)$, $(0,1)$, $(1,0)$ and $(1,1)$. Both marginal variances are $1/4$, the covariance is $0$ and its sharp upper bound is $1/4$, so that $NV_N=N/\\{2(N-1)\\}$ and $NV_N^H=N/(N-1)$. Since $N\\hat V_N^H\\to 1$ in probability, $N|\\hat V_N^H-V_N|$ tends to $1/2$ and not to $0$.\n\nThe repair rests on the inequality that the proof already uses. Since the square $[-C,C]^2$ has area $4C^2$, I have\n\\[\n\\frac{2N}{N-1}\\Bigl|\\int_{[-C,C]^2}\\bigl\\{\\hat H_N^H(y_1,y_0)-H_N^H(y_1,y_0)\\bigr\\}\\,dy_1\\,dy_0\\Bigr|\\le\\frac{8NC^2}{N-1}\\sup_{y_1,y_0}\\bigl|\\hat H_N^H(y_1,y_0)-H_N^H(y_1,y_0)\\bigr| .\n\\]\nA tolerance $x>0$ for the last term therefore corresponds to the threshold $(N-1)x/(8NC^2)$ for the supremum. With $x=\\varepsilon_1/2$ and $\\nu_1+\\cdots+\\nu_8=\\varepsilon_1/2$, the union bound controls the event $\\{N|\\hat V_N^H-V_N^H|\\ge\\varepsilon_1\\}$, which contains the event at $\\varepsilon$ because $\\varepsilon_1<\\varepsilon$.", "problem": "Proposition 2 bounds $\\mathrm{P}\\{N|\\hat V_N^H-V_N^H|\\ge\\varepsilon\\}$ for an arbitrary $\\varepsilon>0$, with $\\varepsilon_1+\\varepsilon_2+\\varepsilon_3=\\varepsilon$. The proof splits $N(\\hat V_N^H-V_N^H)$ into seven terms. The first six receive the tolerances $\\nu_1,\\ldots,\\nu_8$, whose sum is $\\varepsilon_1$, and the last receives $1-\\varepsilon_1$. The tolerances add to $1$ and not to $\\varepsilon$. As printed, the argument bounds the event $\\{N|\\hat V_N^H-V_N^H|\\ge 1\\}$. When $\\varepsilon_1\\ge 1$ the tolerance $1-\\varepsilon_1$ is not positive, and (23) cannot hold with a finite $\\kappa_1(\\varepsilon_1)$. The display after (23) has a second slip: it names the true variance $V_N$ where the sharp upper bound $V_N^H$ is meant. Read literally, it says that $N\\hat V_N^H$ concentrates around $NV_N$, which is false when the bound is not attained.", "what_survives": "The statement of Proposition 2 and its $O(1/N)$ rate are unaffected. The decomposition into seven terms and the bounds (17)--(22) stand, and the argument for $\\hat V_N^L$ keeps its own tolerance $\\varepsilon_2+\\varepsilon_3$. The use of Lemma 2 in the same passage is a separate matter, treated in \\fid{P022-E001}, where the rate is obtained from a bound that does not use the lemma."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-E006", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Nine lines of text from Section 6. A red box marks the figure 7 percent in the sentence which says that the upper bound estimator yields a 7 percent reduction in the nominal variance. The sentences that follow say that over 100 and over 75 additional subjects would be needed.", "caption": "Section 6, second paragraph, p. 857 (PDF p. 8). The marked figure is the stated reduction in the nominal variance.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-N003.png"}, {"alt": "Table 1 with five rows. The variance estimates are 0.199, 0.196, 0.003, 0.186 and 0.098. Red boxes mark the confidence intervals (minus 1.555, 0.192) in the row Conventional and (minus 1.528, 0.165) in the row Sharp upper bound.", "caption": "Table 1, p. 857 (PDF p. 8). The marked cells are the 95% confidence intervals for the conventional estimator and for the sharp upper bound. They determine the ratio of the two variances more closely than the rounded variances do.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-N003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P022-T11", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T11"}], "headline": "Table 1 puts the reduction in the nominal variance from $\\hat V_N^a$ to $\\hat V_N^H$ at about 6%, not 7%.", "historical_evidence_type": "lean_proof", "id": "P022-N003", "location": "Section 6, second paragraph, p. 857 (PDF p. 8), sentence beginning “Compared to the conventional variance estimator”; Table 1, p. 857", "pdf_page": 8, "printed_page": "857", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P022-T11"], "text": {"as_printed": "Compared to the conventional variance estimator $\\hat V_N^a$, we find that our upper bound estimator yields a \\hl{7\\%} reduction in the nominal variance.", "corrected": "Compared to the conventional variance estimator $\\hat V_N^a$, we find that our upper bound estimator yields a \\hl{6\\%} reduction in the nominal variance.", "evidence": "The evidence is a recalculation from Table 1 in exact rational arithmetic. Let $a$ and $h$ be the half-widths of the intervals for $\\hat V_N^a$ and $\\hat V_N^H$. From the printed endpoints, I have $a = 0.8735$ and $h = 0.8465$, each to within $0.0005$, so that $a\\le 0.8740$ and $h\\ge 0.8460$. Since $\\hat V_N^H/\\hat V_N^a = (h/a)^2$, I obtain\n\\[\n\\frac{\\hat V_N^H}{\\hat V_N^a} \\ge \\Bigl(\\frac{0.8460}{0.8740}\\Bigr)^2 = \\frac{178929}{190969},\n\\qquad\nD \\le \\frac{1204000}{190969} \\approx 6.305 < 6.5 .\n\\]\nUsing the variance column, I have $\\hat V_N^H/\\hat V_N^a \\le 0.1865/0.1985 = 373/397$, which gives $D\\ge 2400/397\\approx 6.045$. The inequality $D<6.5$, under the hypotheses $a>0$, $a\\le 0.874$ and $h\\ge 0.846$, also has a Lean proof. Since the data of the application are not available, the unrounded value of $D$ cannot be recovered.", "problem": "The reduction is $D = 100\\,(1-\\hat V_N^H/\\hat V_N^a)$ percent. From the variances printed in Table 1, I find $1-0.186/0.199 \\approx 6.53\\%$, which rounds to 7\\%. However, each of them is rounded to three decimals, and their rounding can change the whole-percent summary. The 95\\% confidence intervals in the same table determine the ratio more closely. Both are Wald intervals with the same critical value. The ratio of the two variances is then the square of the ratio of the two half-widths. Allowing for the rounding of every endpoint, the intervals place $D$ between 5.867 and 6.305. With the variance column as well, $D$ lies between 6.045 and 6.305, and every value in that range rounds to 6\\%.", "what_survives": "Table 1 is unchanged, and the two statements that follow the percentage hold. By the same bounds and nominal inverse-sample-size scaling, matching $\\hat V_N^H$ would take between 100.4 and 105.1 additional subjects with $\\hat V_N^a$, and between 75.3 and 78.5 with $\\hat V_N^{b+}$, in agreement with ``over 100'' and ``over 75''. No formula, proof or other number in the paper uses the percentage."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-N003", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Seven lines of text from Section 6.1. A red box marks the parenthesis 'minus 0.007 USD squared' after the words 'nonpositive bias'. Later in the paragraph the sharp estimator is said to have bias minus 0.010 USD squared.", "caption": "Section 6.1, paragraph on the Sharp Null, p. 858 (PDF p. 9). The marked parenthesis gives the bias of Neyman’s upper bound estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-N005.png"}, {"alt": "Table 2 with three blocks of three rows. In the Sharp Null block, red boxes mark 'True Var.: 0.199' and, in the row Neyman upper bound, the mean variance estimate 0.193 and the mean 95 percent CI width 1.724. The coverage in that row is 94.1 percent.", "caption": "Table 2, p. 858 (PDF p. 9). The marked entries are the true variance under the Sharp Null and, in the row for Neyman’s upper bound, the mean variance estimate and the mean width of the 95% confidence intervals.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-N005-b.png"}], "formal_release": null, "formal_statements": [{"id": "P022-T12", "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-T12"}], "headline": "The bias of $-0.007$ USD$^2$ quoted for $\\hat V_N^{b+}$ under the Sharp Null cannot be reconciled with Table 2; the statement that the bias is nonpositive stands without the figure.", "historical_evidence_type": "lean_proof", "id": "P022-N005", "location": "Section 6.1, paragraph on the Sharp Null, p. 858 (PDF p. 9); Table 2, Sharp Null block, row “Neyman upper bound”, p. 858", "pdf_page": 9, "printed_page": "858", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P022-T12"], "text": {"as_printed": "Due to the nonlinearity of the square root function, the Cauchy--Schwarz inequality implies that $\\hat V_N^{b+}$ has nonpositive bias ($\\hl{-0.007}$ USD$^2$).\n\n\\emph{Table 2, Sharp Null (True Var.: 0.199), row ``Neyman upper bound ($\\hat V_N^{b+}$)''.} Mean var.\\ estimate 0.193; Mean 95\\% CI width 1.724; Coverage for $\\tau_N$ 94.1\\%.", "corrected": "Due to the nonlinearity of the square root function, the Cauchy--Schwarz inequality implies that $\\hat V_N^{b+}$ has nonpositive bias \\del{($-0.007$ USD$^2$)}.\n\nThe difference of the two means printed in Table 2 is $0.193-0.199=-0.006$ USD$^2$. A figure to three decimals should be taken from the simulation output.", "evidence": "The evidence is an exact calculation from the printed numbers. Let $v_j$ be the value of $\\hat V_N^{b+}$ in assignment $j$, let $w_j = c\\sqrt{v_j}$ be the width of its interval, where $c = 2z_{0.975}$, and let $\\bar v$ and $\\bar w$ be the means of $v_j$ and $w_j$ over the simulated assignments. By the Cauchy--Schwarz inequality, I have $\\bar w^2\\le c^2\\bar v$. Let $V$ be the true variance. Since each printed number is rounded to three decimals, I have $V\\le 0.1995$, $\\bar w\\ge 1.7235$ and $\\bar v - V\\le -0.0065$, so that $\\bar v\\le 0.193$. Setting $c = 3.92$, I obtain\n\\[\n\\bar v \\ge \\frac{1.7235^2}{3.92^2} = \\frac{11881809}{61465600} \\approx 0.193308 > 0.193,\n\\qquad\n\\bar v - V \\ge -0.006192 .\n\\]\nThe contradiction does not depend on the value $1.96$. The row of Table 1 for $\\hat V_N^{b+}$, with variance 0.196 and interval $(-1.548, 0.185)$, gives $c^2\\le 1.734^2/0.1955 = 44217/2875$, which implies that $\\bar v\\ge 30364623/157216000\\approx 0.193140 > 0.193$. The inequality $\\bar w^2\\le c^2\\bar v$ and the contradiction obtained from the bound on $c^2$ have a Lean proof.\n\nWith $z=1.96$, the table summaries bound the simulated bias between $-0.006192$ and $-0.005$, so it rounds to $-0.006$ or to $-0.005$; they do not determine the corrected figure. If the figure $-0.007$ is an exact bias computed separately from the simulations of Table 2, the contradiction is not strict. In that case 25 million independent uniform assignments would produce the discrepancy with probability below $10^{-10}$.", "problem": "The bias of a variance estimator is its expectation less the true variance, and the paragraph discusses the 25 million simulated assignments that Table 2 summarizes. For those assignments the table reports a true variance of 0.199, a mean variance estimate of 0.193 for $\\hat V_N^{b+}$, and a mean width of 1.724 for its 95\\% intervals. A bias that rounds to $-0.007$ and a true variance that rounds to 0.199 require a mean variance estimate of at most 0.193. The mean width requires more. Each interval is a Wald interval of full width $2z\\sqrt{v}$, where $v$ is the variance estimate, and the square of a mean of square roots is at most the mean. With $z=1.96$, a mean width of at least 1.7235 forces a mean variance estimate of at least 0.193308. The four printed numbers cannot all be rounded summaries of the same simulations with the stated intervals. At least one of them is off in the third decimal, or the intervals were not constructed as stated. The simulation output is needed to say which.", "what_survives": "The statement that $\\hat V_N^{b+}$ has nonpositive bias under the Sharp Null is correct, and $\\hat V_N^a$ is unbiased there. The ordering of the three estimators, the coverage figures and the bias of $-0.010$ USD$^2$ quoted for $\\hat V_N^H$ are not contradicted by this calculation. None of them was recomputed, since the data are not available. No result of the paper uses the figure."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-N005", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Text from Section 6.1. A red box marks the symbol tau with subscript N, printed without a hat, in the phrase 'the sampling distribution of tau N is not perfectly normal'. The variance estimators in the neighbouring lines carry hats.", "caption": "Section 6.1, paragraph on the Sharp Null, p. 858 (PDF p. 9). The marked symbol is printed without a hat.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "The sampling distribution in the parenthesis is that of the estimator $\\hat\\tau_N$; the hat is missing.", "id": "P022-N006", "location": "Section 6.1, paragraph on the Sharp Null, p. 858 (PDF p. 9), parenthesis after “95.2% and 94.1%, respectively”", "pdf_page": 9, "printed_page": "858", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "(the former is not exactly 95\\% because the sampling distribution of $\\hl{\\tau_N}$ is not perfectly normal)", "corrected": "(the former is not exactly 95\\% because the sampling distribution of $\\hl{\\hat\\tau_N}$ is not perfectly normal)", "evidence": "I checked the page at high magnification. The $\\tau$ carries no hat, while the variance estimators in the neighbouring lines do.", "problem": "The quantity $\\tau_N$ is the average treatment effect in the population. It is fixed under the design and has no sampling distribution. The parenthesis refers to the difference-in-means estimator $\\hat\\tau_N$ defined in (1).", "what_survives": "The coverage figures and the explanation offered for them are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-N006", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "The bound for the first term in the proof of Proposition 2: a four-line display followed by four lines of text. A red box in the first line of the display marks the fraction N (m minus 1) over m (N minus 1). A second red box in the text marks the symbol beta with subscript N.", "caption": "Proof of Proposition 2, bound for the first term, p. 865 (PDF p. 16). The lower mark is the first use of $\\beta_N$, and the upper mark is the coefficient in the display that it stands for. The symbol recurs once in the next line.", "url": "https://pmaronow.github.io/llm-audit/works/P022/exhibits/P022-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The symbol $\\beta_N$ stands for the coefficient $N(m-1)/\\{m(N-1)\\}$ in the display above it; the paper does not define it.", "id": "P022-N007", "location": "Proof of Proposition 2, bound for the first term, p. 865 (PDF p. 16), sentence beginning “where the first inequality follows from”", "pdf_page": 16, "printed_page": "865", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "where the first inequality follows from $|\\mathrm{E}_{\\hat G_N}(y_1^2)-\\hl{\\beta_N}\\mathrm{E}_{G_N}(y_1^2)|$ ${}\\le|\\mathrm{E}_{\\hat G_N}(y_1^2)-\\mathrm{E}_{G_N}(y_1^2)|$ ${}+|(1-\\hl{\\beta_N})|\\mathrm{E}_{G_N}(y_1^2)$, and the second inequality from Chebyshev's inequality and the fact that $\\mathrm{E}_X\\mathrm{E}_{\\hat G_N}(y_1^p)=\\mathrm{E}_{G_N}(y^p)$.", "corrected": "where $\\hl{\\beta_N=N(m-1)/\\{m(N-1)\\}}$, the first inequality follows from $|\\mathrm{E}_{\\hat G_N}(y_1^2)-\\beta_N\\mathrm{E}_{G_N}(y_1^2)|\\le|\\mathrm{E}_{\\hat G_N}(y_1^2)-\\mathrm{E}_{G_N}(y_1^2)|+|(1-\\beta_N)|\\mathrm{E}_{G_N}(y_1^2)$, and the second inequality from Chebyshev's inequality and the fact that $\\mathrm{E}_X\\mathrm{E}_{\\hat G_N}(y_1^p)=\\mathrm{E}_{G_N}(y^p)$.", "intended_meaning": "The symbol $\\beta_N$ occurs only in this sentence. It abbreviates the coefficient of $\\mathrm{E}_{G_N}(y_1^2)$ in the first line of the display above, that is, $\\beta_N=N(m-1)/\\{m(N-1)\\}$. With this value, $1-\\beta_N=(N-m)/\\{m(N-1)\\}$, which accounts for the term $(N-m)C^2/\\{m(N-1)\\}$ in the second line of the display, and the first inequality holds. In the last identity, $y$ is the generic argument of the distribution $G_N$, so that $\\mathrm{E}_{G_N}(y^p)$ and $\\mathrm{E}_{G_N}(y_1^p)$ denote the same moment and the identity is correct as printed."}, "url": "https://pmaronow.github.io/llm-audit/works/P022/#P022-N007", "work": {"arxiv_id": "1405.6555", "doi": "10.1214/13-aos1200", "title": "Sharp bounds on the variance in randomized experiments", "url": "https://pmaronow.github.io/llm-audit/works/P022/"}, "work_id": "P022"}
{"before_review": {"class": "error", "severity": "Major"}, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of seven lines of the paragraph continued from page 2 onto page 3. Red boxes mark the phrases: are mathematically equivalent to studies of close elections; rely solely on data from close elections.", "caption": "Section 1, p. 3 (PDF p. 3), paragraph continued from p. 2. The two marked phrases say that all flexibly specified OLS studies are mathematically equivalent to studies of close elections, and that the estimates rely solely on data from close elections. Only the exclusive-outcome-use assertion is rejected here; the equivalence results are not refuted.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E001.png"}, {"alt": "Screenshot of four lines at the end of Section 3.2. A red box marks the phrase: will also only use information from close elections.", "caption": "Section 3.2, p. 7 (PDF p. 7), last paragraph. The marked phrase draws the same conclusion for any well-specified OLS estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E001-b.png"}, {"alt": "Screenshot of the first six lines of Section 5. A red box marks the words: therefore equally applicable.", "caption": "Section 5, p. 8 (PDF p. 8), first two sentences. The first sentence says only that OLS may rely solely on close elections, which is correct; the second broadly transfers concerns about bias and external validity to full-sample regressions. Shared-cutoff concerns can survive clarification of the claim about exclusive outcome use.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E001-c.png"}, {"alt": "Two panels of vertical stems, one pair per election, ordered by vote margin. Blue circles (polynomial regression) sit away from zero at every election, including the most distant ones; in the left panel they range from about minus 0.72 to 0.72 and in the right panel from about minus 1.8 to 1.8. Orange squares (RD estimate and coarsened regression) are at minus 1 and plus 1 for the two elections in the shaded close-election window and exactly at zero for every other election.", "caption": "Weights of the outcomes in the two exact examples. Left: linear control, six elections, window $(-1/200,1/200]$. Right: degree-7 control, sixteen elections, window $(-1/5,1/5]$. Circles are the weights in the coefficient of the polynomial regression, equation (2), which are nonzero at every election. Squares are the weights in the RD estimate, which equals the coefficient of the coarsened regression of Proposition 1; they are $\\pm1$ at the two elections inside the window (shaded) and zero elsewhere.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E001-weights.png"}, {"alt": "Line chart with a logarithmic horizontal axis, the absolute vote margin m from 0.02 to 100 percentage points, and a vertical axis, the share of the absolute weight on elections with margin at most m, from 0 to 100 per cent. A solid blue line (linear control) and a dashed orange line (degree-7 control) stay below 5 per cent up to m equal to 0.5, where circles mark 1.7 and 4.0 per cent. They rise to about 30 and 53 per cent at 10 points and to about 83 and 87 per cent at 50 points, and step up to 100 per cent at 100 points. A green dash-dotted line at 100 per cent, starting from a square at m equal to 0.5, is the coarsened regression. A dotted vertical line at 0.5 is labelled: window of rows (d) and (e).", "caption": "The 8,610 complete House-election observations in this calculation, using the 2014 release of Caughey and Sekhon’s (2011) replication data: the share of the absolute weight $\\sum_i|w_i|$ that falls on elections with absolute margin at most $m$ percentage points. Solid line: linear specification of row (b) of Table 1. Dashed line: degree-7 specification of row (c). At $m=0.5$, the window of rows (d) and (e), the two shares are 1.7% and 4.0% (circles). Dash-dotted line and square: the coarsened specification of row (d), whose weight lies entirely on the 85 elections within 0.5 points. The step at $m=100$ is the 922 elections with a margin of exactly 100 points.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E001-house-weights.png"}], "formal_release": null, "formal_statements": [{"id": "P023-T1", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T1"}, {"id": "P023-T2", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T2"}, {"id": "P023-T3", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T3"}, {"id": "P023-T10", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T10"}], "headline": "Full-sample OLS with the coarsened control of Proposition 1 relies solely on outcomes from close elections; a polynomial regression can estimate the same jump at the threshold while also using distant outcomes.", "id": "P023-E001", "location": "Section 1, p. 3 (PDF p. 3), paragraph continued from p. 2; repeated in the last paragraph of Section 3.2, p. 7, and in the second sentence of Section 5, p. 8", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P023-T1", "P023-T10", "P023-T2", "P023-T3"], "text": {"as_printed": "[p.~3] ``In fact, we demonstrate that all flexibly specified empirical studies of the party incumbency advantage using ordinary least squares (OLS) regression are mathematically equivalent to studies of close elections. The mechanics of linear regression with a covariate control for vote share are such that the resulting estimates \\hl{rely solely on data from close elections}. It follows that any concerns about the study of close elections are equally applicable to regression analyses including the entire dataset of all elections.''\n\n[p.~7] ``Since the RD estimator only uses information from close elections to estimate causal effects, it is also generally the case that the OLS estimator, when well-specified, \\hl{will also only use information from close elections} to estimate causal effects.''\n\n[p.~8] ``We have demonstrated that OLS estimation of the party incumbency advantage may produce estimates that rely solely on data from close elections. The limitations of studying close elections to estimate causal effects -- both in terms of bias and of external validity -- are therefore equally applicable to regression analyses that include the entire dataset.''", "corrected": "The sentences on p.~3 should read: ``In fact, \\hl{under Conditions 1--3 of Proposition 1,} the OLS estimate from all elections equals the \\hl{degree-0, rectangular-kernel} RD estimate and relies solely on \\hl{outcomes} from close elections. When equation (2) exactly specifies the conditional mean, \\hl{under conditions giving shared consistency, the polynomial OLS and shrinking-bandwidth RD estimates converge to the same cutoff jump and their difference converges to zero in probability}. Outcomes of elections outside the close-election window \\hl{can also enter the polynomial OLS estimate}.''\n\nThe sentence on p.~7 should read: ``The OLS estimator \\hl{with the coarsened control of Proposition 1} uses \\hl{outcomes} from close elections only. An exactly specified polynomial regression \\hl{can also use distant outcomes}. \\hl{Under shared consistency, its difference from a shrinking-bandwidth RD estimate converges to zero in probability}; this does not imply equal observation weights, sampling variance or finite-sample bias.''\n\nThe second sentence of Section 5, p.~8, should read: ``The limitations of studying close elections apply in full \\hl{to the coarsened regression}. For a polynomial regression, \\hl{concerns about causal interpretation and scope can be shared when they arise from the common cutoff target and the same identifying assumptions}; equal targets alone do not imply equal bias.''", "evidence": "The evidence is a derivation, two exact counterexamples and a recalculation of the weights on the available House replication data. Let $r$ be the residual of the vector $\\mathit{DemWin}$ after least-squares projection on the constant and the powers $\\mathit{DemVot},\\ldots,\\mathit{DemVot}^J$. For an identified coefficient in equation (2), I obtain from the normal equations\n\\[\n\\hat\\beta_{OLS2}=\\sum_i w_i\\,\\mathit{DV}_{i(t+1)},\\qquad w_i=\\frac{r_i}{\\sum_j r_j^2},\n\\]\nwhich implies that raising one outcome by $\\delta$ changes the estimate by $w_i\\delta$.\n\nFor the linear control ($J=1$), take six elections with margins $-3/4$, $-1/4$, $-1/400$, $1/400$, $1/4$, $3/4$ and outcomes $\\mathit{DV}=1/2+\\mathit{DemWin}/10+\\mathit{DemVot}/20$. Equation (2) then holds exactly with $\\beta=1/10$. In the order of the margins, the weights are\n\\[\nw=\\frac{1}{139202}\\,(20299,\\,-59901,\\,-99600,\\,99600,\\,59901,\\,-20299).\n\\]\nWith $k=1/200$, only the elections at $\\pm1/400$ lie in the window. For the degree-7 control, take sixteen elections with margins $\\pm1/8,\\pm2/8,\\ldots,\\pm1$ and outcomes $1/2+\\mathit{DemWin}/10+\\mathit{DemVot}/20+\\mathit{DemVot}^7/100$. Equation (2) with $J=7$ holds exactly, again with $\\beta=1/10$, and the sixteen weights, in increasing order of margin, are\n\\[\n\\begin{aligned}\nw=\\frac{1}{1607}\\,(&{-}175,\\,725,\\,-685,\\,-643,\\,554,\\,1289,\\,239,\\,-2911,\\\\\n&2911,\\,-239,\\,-1289,\\,-554,\\,643,\\,685,\\,-725,\\,175),\n\\end{aligned}\n\\]\nnone of them zero. With $k=1/5$, only the elections at $\\pm1/8$ lie in the window. The table shows what happens when the outcome of the election with the largest margin ($3/4$ in the first example, $1$ in the second) is raised by $1/100$.\n\n\\begin{audittable}{lrrr}{Estimates before and after one distant outcome is raised by $1/100$.}\nEstimate & Before & After & Change \\\\\n\\hline\nPolynomial regression, $J=1$ & $1/10$ & $1371721/13920200$ & $-20299/13920200$ \\\\\nRD, coarsened regression, $k=1/200$ & $401/4000$ & $401/4000$ & $0$ \\\\\nPolynomial regression, $J=7$ & $1/10$ & $3249/32140$ & $7/6428$ \\\\\nRD, coarsened regression, $k=1/5$ & $11796481/104857600$ & $11796481/104857600$ & $0$ \\\\\n\\end{audittable}\n\nIn each example the two outcome vectors are realizations of one correctly specified model. Take errors independent of one another and of the margins, with probabilities $1/4$, $1/2$ and $1/4$ at $-1/100$, $0$ and $1/100$. Conditional on the stated margins, the baseline and perturbed error vectors have probabilities $1/64$ and $1/128$ in the six-election example, and $1/65536$ and $1/131072$ in the sixteen-election example. These probabilities establish that the outcomes are possible under the same conditional-mean model; they do not measure the probability of an OLS--RD discrepancy exceeding a fixed positive amount. The finite designs themselves do not test a shrinking-bandwidth asymptotic claim.\n\nThe targets do agree. For margins uniform on $[-1,1]$ and the same two conditional means, the population coefficient of equation (2) is $1/10$, and the difference in means within bandwidth $k$ is $1/10+k/20$ for $J=1$ and $1/10+k/20+k^7/400$ for $J=7$, which tends to $1/10$ as $k\\to0$. With independent observations, the bounded mean-zero errors just specified and, for example, $k_N=N^{-1/3}$, both estimators are consistent for $1/10$. The finite grids are used for conditional sensitivity calculations; no positive probability of drawing those exact grids from a continuous distribution is asserted.\n\nMore generally, whenever both estimators converge in probability to the same $\\beta$, for each fixed $\\epsilon>0$ the triangle inequality gives\n\\[\n\\begin{aligned}\nP\\bigl(|\\hat\\beta_{OLS2}-\\hat\\tau_{RD}|>\\epsilon\\bigr)\n&\\le P\\bigl(|\\hat\\beta_{OLS2}-\\beta|>\\epsilon/2\\bigr)\\\\\n&\\quad+P\\bigl(|\\hat\\tau_{RD}-\\beta|>\\epsilon/2\\bigr)\\longrightarrow0.\n\\end{aligned}\n\\]\nThis requires no independence between the estimators. It does not assert equality at finite sample sizes, a probability for a particular sample size, or equivalence after division by a shrinking standard error.\n\nTheir variance contributions can still differ. With homoskedastic errors, elections whose margins have absolute value at least $1/2$ contribute $1/8$ of the leading error variance of the coefficient of equation (2) for $J=1$ and $572158921/10276044800$ (about 5.6\\%) for $J=7$. The leading contribution from the errors to the variance is $16\\sigma^2/N$ for $J=1$ and $(65536/1225)\\sigma^2/N$ for $J=7$, against $800\\sigma^2/N$ for the difference in means with $k=1/200$. The polynomial contributions vanish at order $1/N$, so their positive distant shares do not imply a persistent discrepancy between the estimates.\n\nOn the available House replication data, the two polynomial specifications of Table 1 also put most of their weight on distant elections. In the 2014 release of the replication data of Caughey and Sekhon (2011), 8,610 elections from 1942--2008 have the outcome (\\texttt{DPctNxt}), the margin (\\texttt{DifDPct}) and the winner (\\texttt{DemWin}) recorded, and a recalculation in exact arithmetic gives the weights $w_i$ for the specifications of rows (b) and (c) on these elections. The available codebook defines this margin as the leading Democrat's percentage minus the leading non-Democrat's, rather than a two-party renormalization. The 85 elections within 0.5 points of the threshold form the window of rows (d) and (e). They carry 1.7\\% of $\\sum_i|w_i|$ for the linear control and 4.0\\% for the degree-7 control. Each of the other 8,525 elections has a nonzero weight in both regressions, and the elections decided by more than 10 points carry 70\\% and 47\\% of $\\sum_i|w_i|$. In the coarsened specification of row (d), the 85 elections carry all of the weight. The three coefficients are nonetheless close: 11.15, 9.95 and 10.54 points. With the sample restricted to the 8,594 elections flagged \\texttt{Use=1} for Caughey and Sekhon's (2011) RD analysis, the two shares are 1.8\\% and 4.0\\%. These are shares of absolute outcome weights, not probabilities of an OLS--RD discrepancy or causal population weights. These calculations do not establish the authors' exact sample or code, the population fit of either polynomial, or a difference in causal bias or external validity.", "problem": "The paper distinguishes two results. Proposition 1 concerns the regression whose controls are indicators for bins of width $2k$ of the margin $\\mathit{DemVot}_{it}$, the ``coarsened control'' of Table 1; I call it the coarsened regression. Its OLS coefficient equals the difference in means within the window $(-k,k]$, and every outcome outside the window has weight zero in it. Proposition 2 concerns regressions with a polynomial in the margin, such as the linear and degree-7 controls of Table 1. Appendix B compares the polynomial and shrinking-window RD estimates through the cutoff jump $\\beta=f(0)-\\lim_{x\\uparrow0}g(x)$, rather than through their observation weights.\n\nRead as a whole, the paper separates exact coarsened equivalence from approximate polynomial equivalence in large samples. The phrase ``mathematically equivalent'' on p.~3 should be read in that context. Nevertheless, the explanation in terms of the mechanics of regression, followed by ``rely solely on data'' on p.~3 and ``only use information'' on p.~7, naturally asserts exclusive dependence on close-election outcomes. That assertion is false for the permitted finite polynomial controls. Its failure does not demonstrate failure of either equivalence result. The identified coefficient of a polynomial regression of degree $J$ is a weighted sum $\\sum_i w_i\\,\\mathit{DV}_i$ over all elections. The weight $w_i$ is proportional to the residual of $\\mathit{DemWin}$ on the polynomial terms, and this residual is nonzero at all but at most $2J$ distinct margins. Raising the outcome of a distant election with nonzero weight then changes the estimate, even when the polynomial model is exactly correct, while the RD estimate and the coefficient of the coarsened regression do not move.\n\nTwo estimators with the same limit need not have the same weights, the same sampling variance or the same bias at a fixed bandwidth. Under shared consistency their difference nevertheless converges to zero in probability; nonzero distant weights do not refute that approximation. For the coarsened regression, equality of the estimators transfers the properties of the specified RD estimate directly. For polynomial controls, Appendix B already supplies the common-target argument, and Section 2 describes assumptions needed for its causal interpretation. Different outcome weights do not remove concerns that arise from that common target and those assumptions. They establish neither different causal bias nor broader external validity in the application.", "what_survives": "Proposition 1 and Section 5's statement that OLS ``may'' rely solely on close-election outcomes are unaffected. When the conditional mean has exactly the form of equation (2), its coefficient and a consistent shrinking-bandwidth RD estimator estimate the same jump; under shared consistency their difference converges to zero in probability. The paper's distinction between exact and approximate equivalence and Appendix B's common-target argument are compatible with the weight calculations. Those calculations do not adjudicate the general conditions for polynomial approximation; see also \\fid{P023-E002}.\n\nThis clarification does not change any entry of Table 1 (for the entries, see \\fid{P023-E003} and \\fid{P023-E004}), and the advice to examine regression assumptions stands. The exclusive-data explanation requires qualification for polynomial controls. Concerns about causal identification and external validity can remain shared when they arise from the common cutoff target and the same identifying assumptions; the weight calculation establishes neither different causal bias nor broader causal scope in the application."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-E001", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Screenshot of a displayed approximation of the conditional expectation and the four lines after it. A red box marks the sentence: E[u given DemVot] is approximately 0 and therefore E of the OLS estimator is approximately beta.", "caption": "Appendix B, p. 10 (PDF p. 10). The marked sentence passes from a small approximation error of the conditional mean to a small error of the coefficient.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E002.png"}, {"alt": "Screenshot of the Remark to Proposition 2 and two lines of the next paragraph. Red boxes mark the words: model fit cannot be meaningfully improved by including additional terms.", "caption": "Remark to Proposition 2, p. 7 (PDF p. 7). The marked words state the test of fit.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E002-b.png"}, {"alt": "Line chart with a logarithmic horizontal axis, the upper bound on the gain in fit from further terms, from about 0.00001 to 0.8 squared percentage points, and a vertical axis, the OLS coefficient on DemWin, from 0 to 10 percentage points. A solid blue line (linear control) and a dashed orange line (degree-7 control) lie at about 10 for small values of the bound and fall to about 8 and 4.5 at the right edge. A grey dashed line at 0 is labelled as the jump at the threshold and the limit of the RD estimate. An open circle at a bound of 1/1200 marks coefficients of 9.998 and 9.993.", "caption": "The population coefficient on $\\mathit{DemWin}$ in the linear and the degree-7 regression (2), for margins uniform on $[-1,1]$, no jump at the threshold and smooth part $h_\\eta(x)=(1/20)\\operatorname{clip}(x/\\eta,-1,1)$, as $\\eta$ runs from $10^{-6}$ to $10^{-1}$. The horizontal axis is the bound $10^4a^2\\eta/3$, in squared percentage points, on the reduction in mean squared error that any further terms could give. The coefficient tends to 10 percentage points as the bound tends to zero, while the jump, to which the RD estimate converges, is 0 (dashed line). The circle marks the example of the text, $\\eta=1/10000$.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E002-fit-and-coefficient.png"}], "formal_release": null, "formal_statements": [{"id": "P023-T4", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T4"}, {"id": "P023-T5", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T5"}, {"id": "P023-T6", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T6"}, {"id": "P023-T7", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T7"}, {"id": "P023-T8", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T8"}, {"id": "P023-T12", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T12"}], "headline": "A small error in the polynomial approximation of the conditional mean does not by itself make the coefficient on $\\mathit{DemWin}$ close to the jump $\\beta$: Proposition 2 holds for a sufficiently large degree under a density condition and a Lipschitz condition, and the test of fit in its Remark does not certify a given degree.", "id": "P023-E002", "location": "Appendix B, p. 10 (PDF p. 10), the sentence beginning “$E[u\\mid \\mathit{DemVot}]\\approx0$ and therefore”; Proposition 2 and its Remark, p. 7 (PDF p. 7)", "pdf_page": 10, "printed_page": "10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P023-T12", "P023-T4", "P023-T5", "P023-T6", "P023-T7", "P023-T8"], "text": {"as_printed": "[p.~7] ``Proposition 2. Under weak regularity conditions, if $N\\to\\infty$, $k\\to0$, $Nk\\to\\infty$, and $J$ is ``sufficiently large,'' the OLS estimator of $\\beta$ in (2), $\\hat\\beta_{OLS2}\\approx\\hat\\tau_{RD}$. Remark. \\ldots\\ Put simply, in order for the result to hold, we require that \\hl{the model fit cannot be meaningfully improved by including additional terms} in the polynomial expansion of $\\mathit{DemVot}_{it}$.''\n\n[p.~10] ``Our assumptions about $g()$ and $f()$ imply that $h()$ is continuous on $[-1,1]$ and $h(0)=0$. By the Weierstrass Approximation Theorem, for any $\\epsilon>0$, there exist some $J^*$ and $\\gamma_1^*,\\ldots,\\gamma_{J^*}^*$ such that $|h(x)-\\sum_{j=1}^{J^*}\\gamma_j^*x^j|<\\epsilon$ for all $x\\in[-1,1]$. We assume that $J$ in (2) is sufficiently large to ensure that there exist $\\gamma_1,\\ldots,\\gamma_J$ such that $|h(x)-\\sum_{j=1}^{J}\\gamma_jx^j|\\approx0$ for all $x\\in[-1,1]$. Then\n\\[\nE[\\mathit{DV}\\mid \\mathit{DemVot}]\\approx\\alpha+\\beta I[\\mathit{DemVot}\\ge0]+\\sum_{j=1}^{J}\\gamma_j\\mathit{DemVot}^j,\n\\]\n$E[u\\mid \\mathit{DemVot}]\\approx0$ \\hl{and therefore} $E[\\hat\\beta_{OLS2}]\\approx\\beta$. If the sample is drawn i.i.d. from a population $U$ with $\\operatorname{Supp}(\\mathit{DemVot})\\supseteq[-k,k]$, where $k>0$, then for large enough $N$, we have $\\hat\\beta_{OLS2}\\approx\\beta$.''", "corrected": "Proposition 2 should read as follows. Let the observations $(X,Y)=(\\mathit{DemVot}_{it},\\mathit{DV}_{i(t+1)})$ be i.i.d. with $E[Y^2]<\\infty$, and let $D=I[X\\ge0]$. Suppose that the law of $X$ is supported on $[-1,1]$, \\hl{has no atom at $0$, and satisfies $P(X\\in A)\\ge c\\,|A|$ for every Borel set $A\\subseteq[-\\delta,\\delta]$}, for some $c>0$ and $\\delta>0$, where $|A|$ is the Lebesgue measure of $A$; that $E[Y\\mid X=x]=\\alpha+\\beta I[x\\ge0]+h(x)$ with $h(0)=0$ and \\hl{$|h(x)-h(y)|\\le L|x-y|$ for all $x,y\\in[-1,1]$}; and that $\\operatorname{Var}(Y\\mid X)$ is bounded on a neighbourhood of $0$. Let $b_J$ be the coefficient on $D$ in the population least-squares regression of $Y$ on $D,1,X,\\ldots,X^J$. Then for every $J\\ge1$,\n\\[\n\\hl{|b_J-\\beta|\\le\\frac{8KL}{\\sqrt{c\\,\\delta\\,(J+1)}},\\qquad K=\\frac{3(1+\\pi^4)}{4\\pi}.}\n\\]\nFor each fixed $J$, $\\hat\\beta_{OLS2}\\to b_J$ in probability as $N\\to\\infty$, and $\\hat\\tau_{RD}\\to\\beta$ in probability as $N\\to\\infty$, $k\\to0$ and $Nk\\to\\infty$. Hence for every $\\epsilon>0$ there exists $J_0$ such that, for each fixed $J\\ge J_0$, the difference $\\hat\\beta_{OLS2}-\\hat\\tau_{RD}$ converges in probability to $b_J-\\beta$, and \\hl{$|b_J-\\beta|<\\epsilon$}. If $J=J_N$ grows with $N$, the difference converges in probability to zero provided $J_N\\to\\infty$ and $(J_N+2)^2/(N\\lambda_{J_N}^3)\\to0$, where $\\lambda_J$ is the smallest eigenvalue of the second-moment matrix of $(1,D,X,\\ldots,X^J)$.\n\nThe Remark should read: if $h$ is a polynomial of degree at most $J$, then $b_J=\\beta$. If $\\sup_x|h(x)-p(x)|\\le\\epsilon$ for some polynomial $p$ of degree at most $J$, then \\hl{$|b_J-\\beta|\\le\\kappa_J\\epsilon$}, where $\\kappa_J=E|r_J|/E[r_J^2]$ and $r_J$ is the residual of $D$ on $1,X,\\ldots,X^J$; for margins uniform on $[-1,1]$, $\\kappa_1=10/3$ and $\\kappa_7\\approx4.764$. That the fit cannot be meaningfully improved by further terms \\hl{does not imply} that $b_J$ is close to $\\beta$.\n\nIn Appendix B, the sentence should read: ``$E[u\\mid \\mathit{DemVot}]\\approx0$, \\hl{and the coefficient $b_J$ satisfies $b_J-\\beta=E[r_J(h-p_J)]/E[r_J^2]$, which is small when $\\|h-p_J\\|_2$ is small relative to $\\|r_J\\|_2$}.''", "evidence": "The evidence is a derivation and exact counterexamples. Let $X=\\mathit{DemVot}$, $D=I[X\\ge0]$ and $Y=\\mathit{DV}$, with $E[Y\\mid X]=\\alpha+\\beta D+h(X)$ as in Appendix B. Let $r_J$ be the residual of $D$ after projection in $L^2$ on $1,X,\\ldots,X^J$, and set $v_J=E[r_J^2]$, which I assume positive. Then $E[r_J\\,p(X)]=0$ for every polynomial $p$ of degree at most $J$, and $E[r_JD]=v_J$. Let $b_J$ be the coefficient on $D$ in the population regression (2). Subtracting from the normal equation for $D$ the combination of the other normal equations that defines the projection, I obtain $E[r_J(Y-b_JD)]=0$. Then\n\\[\nb_J-\\beta=\\frac{E[r_J\\,h(X)]}{v_J}=\\frac{E[r_J\\,(h(X)-p(X))]}{v_J}.\n\\]\nBy the triangle inequality and the Cauchy--Schwarz inequality, I have\n\\[\n|b_J-\\beta|\\le\\kappa_J\\,\\|h-p\\|_\\infty,\\qquad \\kappa_J=\\frac{E|r_J|}{v_J},\\qquad\\text{and}\\qquad |b_J-\\beta|\\le\\frac{\\|h-p\\|_2}{\\sqrt{v_J}}.\n\\]\nSince $E|r_J|\\ge E[r_J(2D-1)]=2v_J$, the factor $\\kappa_J$ is at least 2. For fixed $J$ and an i.i.d. sample with finite second moments, $\\hat\\beta_{OLS2}$ converges in probability to $b_J$, and $b_J-\\beta$ is then the difference between the two estimators in large samples.\n\nFor margins uniform on $[-1,1]$, I find by a direct computation that $v_J=c_n^2/4$ with $c_n=\\binom{2n}{n}4^{-n}$ and $n=\\lfloor(J+1)/2\\rfloor$, so that $v_J$ is of order $1/J$, and that $\\kappa_J=(4/\\pi)\\log J+O(1)$. The table lists these constants for the two degrees of Table 1, together with the following example. Let $h_\\eta(x)=a\\,\\operatorname{clip}(x/\\eta,-1,1)$ with $a=1/20$ and $\\eta=1/10000$, and let $Y=1/2+h_\\eta(X)+U$, where $U=\\pm1/10$ with equal probability, independently of $X$. Then $h_\\eta$ is Lipschitz, $h_\\eta(0)=0$, the jump is $\\beta=0$, and the RD estimate converges to 0. The predictor $1/2+a(2D-1)$ lies in the span of the regressors of (2), and its mean squared error exceeds the noise variance $1/100$ by $a^2\\eta/3=1/12{,}000{,}000$. No further terms can lower the mean squared error by more than this amount.\n\n\\begin{audittable}{lrrrrr}{Uniform margins on $[-1,1]$: the constants, and the coefficient $b_J$ in percentage points for the example with $\\beta=0$, $a=1/20$ and $\\eta=1/10000$, with the largest reduction in mean squared error that further terms could give.}\nDegree & $v_J$ & $\\kappa_J$ & $1/\\sqrt{v_J}$ & $b_J$ (points) & Gain in fit at most \\\\\n\\hline\n$J=1$ & $1/16$ & $10/3$ & $4$ & $9.998$ & $1/12{,}000{,}000$ \\\\\n$J=7$ & $1225/65536$ & $4.7642$ & $256/35$ & $9.993$ & $1/12{,}000{,}000$ \\\\\n\\end{audittable}\n\nIn exact terms, $b_1=99980001/1000000000$. As $\\eta\\to0$ with $J$ fixed, the bound $a^2\\eta/3$ tends to zero and $b_J$ tends to $2a=1/10$. A second example concentrates the margins: all but a fraction $10^{-12}$ of the mass lies in two windows of half-width $1/2000$ around $\\pm1/2$, the density is continuous and positive on $[-1,1]$, and $h$ is Lipschitz with $\\sup|h|=1/2000$ and vanishes near 0. Then $\\beta=0$, further terms can lower the mean squared error by at most $1/4{,}000{,}000$, and yet $b_1=-499999999999503/500000999999500$. With the fraction $10^{-96}$, the degree-7 coefficient lies between $-1001/1000$ and $-999/1000$.\n\nContinuity alone fails even for one fixed design. Since $\\kappa_J$ is unbounded for uniform margins, a uniformly convergent series of polynomials defines a continuous $h$ with $h(0)=0$ and $\\sup|h|\\le1/12$ such that $b_{J_m}\\ge m/4$ along a sequence of degrees $J_m\\to\\infty$, while $\\beta=0$.\n\nThe corrected statement below rests on two estimates. Comparing with uniform margins on $[-\\delta,\\delta]$, I have $v_J\\ge c\\,\\delta/(4(J+1))$ under its density condition. Under its Lipschitz condition, convolution with a Jackson kernel yields a polynomial $p_J$ of degree at most $J$ with $p_J(0)=0$ and $\\|h-p_J\\|_\\infty\\le4KL/(J+1)$, where $K=3(1+\\pi^4)/(4\\pi)$. Inserting these into the second bound above, I obtain the rate of the corrected statement. The limit of $\\hat\\tau_{RD}$ follows from $|E[h(X)\\mid0<X\\le k]-E[h(X)\\mid-k<X<0]|\\le2Lk$ and from the growth of the number of observations in the window on each side of the threshold, which has expectation at least $Nck$ on each side.", "problem": "Appendix B writes the conditional mean of the outcome as $\\alpha+\\beta D+h(X)$, with $X=\\mathit{DemVot}$, $D=I[X\\ge0]$ and $h$ continuous, approximates $h$ uniformly by a polynomial, and concludes that the coefficient on $D$ is close to the jump $\\beta$. The step marked above leaves out a factor. The error of the coefficient is the approximation error weighted by the part of $D$ that the polynomial terms do not explain, divided by the variance of that part. It is bounded by $\\kappa_J$ times the uniform approximation error, where the factor $\\kappa_J\\ge2$ depends on the degree and on the distribution of the margin. For uniform margins it grows without bound in $J$. The continuity assumptions on $f$ and $g$ do not imply Proposition 2. Indeed, for uniform margins there exists a continuous $h$ with $h(0)=0$ such that the coefficient does not converge to $\\beta$ as the degree grows.\n\nThe test of fit in the Remark, which the surrounding text and Appendix B use as a sufficient condition, measures a different quantity from the approximation error of $h$, and it does not certify a given degree. With uniform margins and no jump, the linear and the degree-7 regression can have a coefficient about 10 percentage points from $\\beta$ while no further terms could lower the mean squared error by more than $1/12{,}000{,}000$. Rows (b) and (c) of Table 1 use these two degrees. Whether they are adequate for the data of the application cannot be determined from the paper.", "what_survives": "Proposition 1 and Appendix A do not use this step. If $h$ is exactly a polynomial of degree at most $J$, then $b_J=\\beta$. Proposition 2 holds as a statement about a sufficiently large degree under the conditions of the corrected statement, which are of the kind its ``weak regularity conditions'' ordinarily cover. No entry of Table 1 is shown to be wrong, and the similarity of the printed estimates stands as a description. What changes is the proof in Appendix B, the assumptions of Proposition 2, and the Remark. Rows (b) and (c) of Table 1 are instances of Proposition 2 only under a quantitative condition on the degrees 1 and 7, which the paper does not check."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-E002", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Screenshot of the lower part of Table 1: row (d) Coarsened control 10.54 pp, the heading Regression discontinuity (close elections), rows (e) to (h), and the note giving the bandwidths 0.5 and 1.0 percentage points. Red boxes mark the entries 10.57 pp of row (f) and 12.38 pp of row (h).", "caption": "Table 1, p. 8 (PDF p. 8), rows (d) to (h) and the note on bandwidths. The marked entries are the two RD estimates with a triangular kernel.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E003.png"}, {"alt": "Screenshot of the paragraph that describes the four RD estimators, followed by the first two lines of the next paragraph. Red boxes mark the words: tor (f) replaces the rectangular kernel in tau-hat RD with a triangular kernel; and the sentence: Estimator (h) uses both a triangular kernel and a Degree 1 polynomial.", "caption": "Section 4, p. 8 (PDF p. 8), the paragraph below Table 1. The marked sentences describe estimators (f) and (h).", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "On the replication data of Caughey and Sekhon (2011), the estimators with a triangular kernel that are described for rows (f) and (h) of Table 1 give 13.08 and 15.30 percentage points, not the printed 10.57 and 12.38.", "id": "P023-E003", "location": "Table 1, rows (f) and (h), p. 8 (PDF p. 8), and the paragraph below the table", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lr}{Table 1, p.~8: the rows for the RD estimators.}\n & Estimated effect \\\\\nRegression discontinuity (close elections) & \\\\\n(e) Degree 0, Rectangular kernel & 10.54 pp \\\\\n(f) Degree 0, Triangular kernel & \\hl{10.57 pp} \\\\\n(g) Degree 1, Rectangular kernel & 13.15 pp \\\\\n(h) Degree 1, Triangular kernel & \\hl{12.38 pp} \\\\\n\\end{audittable}\n\\emph{Table note.} ``Bandwidth $k=0.5$ percentage points for Degree 0 RD estimates and coarsened control OLS, $k=1.0$ percentage points for Degree 1 RD estimates.''\n\n[p.~8] ``Estimator (f) replaces the rectangular kernel in $\\hat\\tau_{RD}$ with a triangular kernel, so that observations closer to the discontinuity receive greater weight. \\ldots\\ Estimator (h) uses both a triangular kernel and a Degree 1 polynomial.''", "corrected": "\\begin{audittable}{lr}{Table 1, rows (f) and (h), recalculated.}\n & Estimated effect \\\\\n(f) Degree 0, Triangular kernel & \\hl{13.08 pp} \\\\\n(h) Degree 1, Triangular kernel & \\hl{15.30 pp} \\\\\n\\end{audittable}\n\nThese are the values for the release of 10 April 2014 of the data, the 8,610 elections with the outcome, the margin and the winner recorded, and the weights $\\max(1-|x|/k,0)$, with a constant on each side for $k=0.5$ and a line on each side for $k=1.0$. Section 4 should state the release, the sample and the weights, and the comparison below the table should be drawn from the corrected entries. If the four outcomes of \\fid{P023-E004} are also replaced by their linked next-election percentages, rows (f) and (h) read 10.36 and 10.91.", "evidence": "The evidence is a recalculation from the data in exact rational arithmetic on the stored values; a second, independent computation agrees with it to $10^{-10}$ points. The data are the release of 10 April 2014, and the sample is the 8,610 elections with the outcome, the margin and the winner recorded. Let $x_i$ be the margin of election $i$ in points and $y_i$ its outcome. Row (f) is the difference between the two sides of the threshold in the weighted mean $\\sum_i w_iy_i/\\sum_i w_i$, with $w_i=1-|x_i|/0.5$ over the elections with $|x_i|<0.5$. The weighted means are 40.99 for the Democratic losses and 54.08 for the Democratic wins. Row (h) is the difference at $x=0$ between two weighted least-squares lines, one on each side, with $w_i=1-|x_i|$ over the elections with $|x_i|<1$. The two intercepts are 38.86 and 54.16.\n\n\\begin{audittable}{llrrr}{The four RD rows of Table 1, in percentage points, as printed and as recalculated.}\nRow & Elections (losses + wins) & Printed & Recalculated & Difference \\\\\n\\hline\n(e) Degree 0, rectangular, $k=0.5$ & 85 (42 + 43) & 10.54 & 10.54 & 0.00 \\\\\n(f) Degree 0, triangular, $k=0.5$ & 85 (42 + 43) & 10.57 & 13.08 & 2.51 \\\\\n(g) Degree 1, rectangular, $k=1.0$ & 168 (86 + 82) & 13.15 & 13.15 & 0.00 \\\\\n(h) Degree 1, triangular, $k=1.0$ & 168 (86 + 82) & 12.38 & 15.30 & 2.92 \\\\\n\\end{audittable}\n\nNo margin equals $0$, $\\pm0.5$ or $\\pm1.0$ points, so the treatment of ties and of the endpoints of a window cannot account for the differences. If the sample is restricted to the elections that Caughey and Sekhon (2011) flag for their RD analysis, the same elections lie in both windows, and the four rows do not change.\n\nThe four outcomes of \\fid{P023-E004} are one hundredth of their linked next-election percentages at storage precision. All four elections lie within 0.3 points of the threshold, and two of them within 0.06 points, where the triangular weights are largest. With those linked percentages substituted, I obtain 10.36 and 10.91 points for rows (f) and (h). Neither computation returns the printed 10.57 and 12.38; the accuracy of the links and percentages remains unverified.", "problem": "Row (f) is described as the difference in means $\\hat\\tau_{RD}$ of row (e) with a triangular kernel in place of the rectangular one, and row (h) as the local linear estimate of row (g) with the same change. I read these descriptions in the standard way. An election with margin $x$ has weight $\\max(1-|x|/k,0)$, with $k=0.5$ points and a constant fitted on each side of the threshold in row (f), and with $k=1.0$ point and a line fitted on each side in row (h). On the replication data of Caughey and Sekhon (2011), I find 13.08 and 15.30 points for these two estimators. From the same data and the same elections, I obtain rows (e) and (g) as printed, 10.54 and 13.15, and a coarsened regression equal to row (e), as in row (d). The two printed entries are thus lower by 2.51 and 2.92 points than the values that the described estimators take on the data that reproduce the neighbouring rows.\n\nIn the printed table the estimates with the triangular kernel are close to those with the rectangular kernel: the differences are $+0.03$ and $-0.77$ points. Recalculated, the triangular estimates exceed the rectangular ones by 2.54 and 2.16 points, and, with the other entries as printed, the estimates other than the naive one range from 9.97 to 15.30 points, where the printed ones range from 9.97 to 13.15. I do not know the cause of the two differences. A different release of the data, a choice of weights, bandwidth or sample that the paper does not describe, and a slip in computation or reporting are all consistent with the evidence; the paper names no release, and its analysis code was not available.", "what_survives": "Rows (e) and (g) are reproduced as printed, and the coarsened regression of row (d) equals row (e) exactly, as Proposition 1 requires. The propositions and their proofs do not use Table 1. Every estimate other than the naive one remains positive. What changes is two entries of the table and the comparison drawn from them: the sentence below the table that all estimates other than the naive one are ``substantively similar'' then covers a range from 9.97 to 15.30 points. The paper gives no threshold for this phrase, and I draw no conclusion on whether it still applies."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-E003", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Screenshot of the five lines of the first paragraph of Section 4. Red boxes mark the words: the replication data provided by Caughey and Sekhon (2011).", "caption": "Section 4, p. 7 (PDF p. 7), first paragraph. The marked words identify the source study’s replication data; the outcome is the Democratic vote share in the next election.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E004.png"}, {"alt": "Screenshot of the body of Table 1: rows (a) to (d) under the heading Ordinary least squares (full data), rows (e) to (h) under the heading Regression discontinuity (close elections), and the note on bandwidths. Red boxes mark the entries 10.54 pp of row (d), 10.54 pp of row (e) and 13.15 pp of row (g).", "caption": "Table 1, p. 8 (PDF p. 8), the eight rows and the note on bandwidths. The marked entries of rows (d), (e) and (g) agree, after rounding, with calculations from the 10 April 2014 release before changing the four outcomes.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E004-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Four next-election outcomes in the 10 April 2014 release of the Caughey and Sekhon (2011) data have a 100-fold scale inconsistency with their linked percentages; replacing only those outcomes in the calculation below lowers rows (d), (e) and (g) from 10.54, 10.54 and 13.15 to 9.32, 9.32 and 10.40 percentage points.", "id": "P023-E004", "location": "Section 4, first paragraph, p. 7 (PDF p. 7), and Table 1, p. 8 (PDF p. 8)", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "[p.~7] ``We use the replication data provided by Caughey and Sekhon (2011), restricting our estimates to all elections with available data. Our outcome measure, $\\mathit{DV}_{i(t+1)}$ is Democratic vote share in election $t+1$ in region $i$.''\n\n\\begin{audittable}{lr}{Table 1, p.~8: rows (d), (e) and (g).}\n & Estimated effect \\\\\n(d) Coarsened control & 10.54 pp \\\\\n(e) Degree 0, Rectangular kernel & 10.54 pp \\\\\n(g) Degree 1, Rectangular kernel & 13.15 pp \\\\\n\\end{audittable}", "corrected": "[p.~7] After ``restricting our estimates to all elections with available data'', Section 4 should \\hl{identify the data release} and explain how the outcomes at observations 3123, 7243, 7687 and 9374 are treated if that release contains these scale inconsistencies.\n\n\\begin{audittable}{lr}{Sensitivity calculation for Table 1, rows (d), (e) and (g), on the 8,610 complete elections, replacing only the four outcomes by their linked percentages.}\n & Estimated effect \\\\\n(d) Coarsened control & \\hl{9.32 pp} \\\\\n(e) Degree 0, Rectangular kernel & \\hl{9.32 pp} \\\\\n(g) Degree 1, Rectangular kernel & \\hl{10.40 pp} \\\\\n\\end{audittable}\n\nThe other estimates change as in the second table of the evidence. These are separate sensitivity results for the stated sample and estimators. Replacing the printed entries additionally requires verification of the intended same-district links, linked percentages, and the authors' historical data, sample, transformations and estimator implementation.", "evidence": "I recalculated the estimates from the data (the release of 10 April 2014, with 16,968 observations, of which 8,980 have an outcome and a linked percentage). The four observations listed in the first table below are the only ones with a positive outcome at which 100 times the outcome equals the linked percentage, to the precision of the stored numbers. Their recorded outcomes also equal, to storage precision, the Democratic shares at the linked observations: $83265/217115$, $70920/230597$, $121121/286860$ and $104806/171713$.\n\n\\begin{audittable}{llrrr}{The four elections of 2000 with inconsistent outcome scales.}\nObservation & District & Margin (points) & Recorded outcome & Linked percentage \\\\\n\\hline\n3123 & Florida 22 & $-0.28$ & 0.3835 & 38.35 \\\\\n7243 & Michigan 8 & $-0.05$ & 0.3075 & 30.75 \\\\\n7687 & Minnesota 2 & $-0.05$ & 0.4222 & 42.22 \\\\\n9374 & New Jersey 12 & $0.22$ & 0.6104 & 61.04 \\\\\n\\end{audittable}\n\nThe eight Table 1 specifications were translated using \\texttt{DifDPct} as the margin and a common sample of 8,610 elections with the outcome, margin and winner recorded. Each was computed twice with the stored numbers treated as exact binary fractions: once with the outcomes as recorded, and once with the four outcomes replaced by the linked percentages. The second table reports both computations in percentage points; the changes are computed before rounding.\n\n\\begin{audittable}{lrrrr}{Table 1 recalculated, with the outcomes as recorded and with the four outcomes replaced by the linked percentages.}\nRow & Printed & As recorded & Four replaced & Change \\\\\n\\hline\n(a) Naive (no control) & 34.40 & 34.38 & 34.37 & $-0.02$ \\\\\n(b) Linear control & 11.15 & 11.15 & 11.13 & $-0.02$ \\\\\n(c) Polynomial (Degree 7) control & 9.97 & 9.95 & 9.87 & $-0.08$ \\\\\n(d) Coarsened control & 10.54 & 10.54 & 9.32 & $-1.22$ \\\\\n(e) Degree 0, Rectangular kernel & 10.54 & 10.54 & 9.32 & $-1.22$ \\\\\n(f) Degree 0, Triangular kernel & 10.57 & 13.08 & 10.36 & $-2.73$ \\\\\n(g) Degree 1, Rectangular kernel & 13.15 & 13.15 & 10.40 & $-2.75$ \\\\\n(h) Degree 1, Triangular kernel & 12.38 & 15.30 & 10.91 & $-4.40$ \\\\\n\\end{audittable}\n\nWith the sample restricted to the 8,594 elections that Caughey and Sekhon (2011) flag for their RD analysis, rows (a) to (c) are 34.40, 11.14 and 9.97 with the outcomes as recorded and 34.38, 11.12 and 9.89 with the four replaced, and rows (d) to (h) are as in the table. Neither sample reproduces all eight printed entries. Row (d) equals row (e) exactly in every computation. For rows (f) and (h), whose printed entries neither computation returns, see \\fid{P023-E003}.", "problem": "In the replication data of Caughey and Sekhon (2011) the outcome is the variable \\texttt{DPctNxt}, which the codebook defines as the ``Democratic percentage of the vote in the next election'', and the variable \\texttt{NxtElcOb} gives the observation number of that election. For four elections of 2000 the recorded outcome is one hundredth of the Democratic percentage at the linked observation, to the precision of the stored numbers. The linked observation is the election of 2002 with the same state and district number. I call this percentage the linked percentage. No other positive outcome shows this ratio.\n\nThe four elections were decided by margins of $-0.28$, $-0.05$, $-0.05$ and $0.22$ points. They are therefore among the 85 elections in the 0.5-point window of rows (d) to (f) and among the 168 elections in the 1.0-point window of rows (g) and (h). In the smaller window, the three proportions recorded for Democratic losses lower the mean outcome of the losing side by 2.62 points, and the proportion recorded for the Democratic win lowers the mean of the winning side by 1.41 points. The difference in means of rows (d) and (e) is then higher by 1.22 points than with the linked percentages. With the four outcomes replaced by the linked percentages and nothing else changed, rows (d) and (e) fall from 10.54 to 9.32 points and row (g) falls from 13.15 to 10.40. The printed entries of these three rows agree to the second decimal with the values computed from the four outcomes as recorded.\n\nTwo points are not established. The first is that the linked percentages are the right outcomes for the four elections. The codebook codes the outcome as missing when the seat was redistricted between the two elections, and the linked elections of 2002 are not among the elections that Caughey and Sekhon (2011) flag for their RD analysis and carry no link back to a previous election. The data alone do not show that the two elections are in the same district. The second is that the published analysis read these four values. The paper names no release of the data and its analysis code was not available. Agreement of the printed rows (d), (e) and (g) with calculations from the release of 10 April 2014 does not establish historical use.", "what_survives": "The equality of rows (d) and (e) holds for every vector of outcomes and is unaffected, and the propositions do not use the table. Rows (a) to (c) move by less than 0.1 point. In this calculation, with the four outcomes replaced, rows (b) to (h) lie between 9.32 and 11.13 points, and their numerical agreement stands. What changes is the size of the close-election estimates: rows (d), (e) and (g) fall by 1.22, 1.22 and 2.75 points. The four values are in the data as Caughey and Sekhon released them, and nothing shows that the authors of the paper introduced them."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-E004", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Screenshot of five lines of the first paragraph of the introduction. A red box marks the figure 0.6\\%, together with the opening parenthesis before it.", "caption": "Section 1, p. 1 (PDF p. 1), first paragraph. The marked figure is the share of House elections of 1944–2006 decided by less than 0.5 percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E005.png"}], "formal_release": null, "formal_statements": [], "headline": "In the data of Caughey and Sekhon (2011), 103 of the 13,923 House elections of 1944–2006 have a recorded absolute Democratic margin below 0.5 percentage points, which rounds to 0.7%, not 0.6%.", "id": "P023-E005", "location": "Section 1, first paragraph, p. 1 (PDF p. 1)", "pdf_page": 1, "printed_page": "1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``In the case of the US House of Representatives, close elections are quite rare (\\hl{0.6\\%} of elections between 1944 and 2006 were determined by $<0.5$ percentage points), and characterizing a causal effect that is only applicable to a tiny, unrepresentative sample of elections may seem undesirable on substantive grounds.''", "corrected": "``\\ldots\\ \\hl{recorded} close elections are quite rare (\\hl{0.7\\%} of elections between 1944 and 2006 \\hl{had a recorded absolute Democratic margin of} $<0.5$ percentage points), \\ldots'' The count behind the figure is 103 of 13,923 elections in the data of Caughey and Sekhon (2011), release of 10 April 2014, with the margin missing for 587. This fraction is a lower bound for the share decided by less than 0.5 points between winner and runner-up.", "evidence": "I recalculated the share from the data (the release of 10 April 2014; the paper names no release). The margin is missing for 587 of the 13,923 elections, and the missing margins cannot restore the printed figure. By the codebook's definition of the margin, each of the 103 elections was decided by less than 0.5 points between the winner and the runner-up. The share of all elections is then at least $103/13{,}923$. Among the elections with a recorded margin the share is $103/13{,}336\\approx0.77\\%$. The ratio $85/13{,}923\\approx0.61\\%$ rounds to the printed figure, but its numerator counts only the close elections that Caughey and Sekhon (2011) flag for their RD analysis and omits 18 others, while its denominator counts all elections. The paper does not say how its figure was computed.", "problem": "The paper gives no numerator, denominator or counting rule for this figure. The replication data of Caughey and Sekhon (2011), which Section 4 uses for the application, contain 13,923 observations for 1944--2006, one for each seat and election. In 103 of them the recorded margin between the leading Democrat and the leading non-Democrat is below 0.5 points in absolute value, that is, in about 0.74\\% of them. A share that rounds to 0.6\\% would require between 77 and 90 such elections.", "what_survives": "Each of the recorded close-election shares above is below 1\\%. The figure enters no proposition, no proof and no entry of Table 1. The data do not determine the exact share of all elections decided by less than 0.5 points, since the margin between winner and runner-up is not recorded for every election; this share rounds to 0.7\\% if at most one further close election lies outside the 103."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-E005", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Screenshot of Condition 1 and the Remark that follows it. Red boxes mark the two strict inequalities of the condition, DemVot between 0 and k and DemVot between minus k and 0, and the word necessary in the last sentence of the Remark.", "caption": "Condition 1 and its Remark, p. 5 (PDF p. 5). The two strict inequalities of the condition and the word “necessary” are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E006.png"}, {"alt": "Screenshot of the two definitions of the demeaned variables in Appendix A. Red boxes mark the factor, the inverse of the number of observations in bin j, in each definition.", "caption": "Appendix A, p. 9 (PDF p. 9), definitions of the demeaned variables. The marked factor $|\\tilde U_j|^{-1}$ is undefined for an empty bin.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E006-b.png"}, {"alt": "Screenshot of four lines at the end of Section 2. Red boxes mark the expressions 50 percent plus epsilon and 50 percent minus epsilon.", "caption": "Section 2, p. 4 (PDF p. 4), last sentence. The marked vote shares put the threshold at 50%.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E006-c.png"}], "formal_release": null, "formal_statements": [{"id": "P023-T9", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T9"}], "headline": "Condition 1 is sufficient, not necessary, for the displayed RD estimate; the bin means of Appendix A are taken over nonempty bins; and the 50% threshold on p. 4 is that of a two-party election.", "id": "P023-E006", "location": "Remark to Condition 1, p. 5 (PDF p. 5); Appendix A, p. 9, definitions of the demeaned variables; Section 2, p. 4, last sentence", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P023-T9"], "text": {"as_printed": "[p.~5] ``Condition 1. There exists at least one value of $(i,t)$ s.t. $0<\\mathit{DemVot}_{it}<k$ and there also exists at least one value of $(i',t')$ s.t. $-k<\\mathit{DemVot}_{i't'}<0$ \\ldots\\ Remark. Condition 1 is a mild regularity condition that requires that there exist at least one observation on each side of the discontinuity in the bandwidth. This condition is \\hl{necessary} for the existence of an RD estimate.''\n\n[p.~9] ``where $\\ddot{\\mathit{DemWin}}_{it}\\equiv \\mathit{DemWin}_{it}-\\sum_{j=0}^{J}\\big[X_{ijt}\\,\\hl{|\\tilde U_j|^{-1}}\\sum_{i't'\\in\\tilde U_j}\\mathit{DemWin}_{i't'}\\big]$ \\ldots''\n\n[p.~4] ``\\ldots\\ the difference between the predicted outcome at a vote share of \\hl{$50\\%+\\epsilon$} and the predicted outcome at \\hl{$50\\%-\\epsilon$} (for $\\epsilon\\to0$) can be interpreted as an estimate of the causal effect of the party incumbency advantage.''", "corrected": "[p.~5] ``This condition is \\hl{sufficient} for the existence of the RD estimate displayed below. \\hl{That estimate exists if and only if $U_b$ contains at least one observation with $\\mathit{DemWin}_{it}=1$ and at least one with $\\mathit{DemWin}_{it}=0$.}''\n\n[p.~9] The sums over $j$ in the two definitions run over the \\hl{nonempty} bins only, and the indicator of an empty bin is left out of the regression.\n\n[p.~4] ``\\ldots\\ the difference, \\hl{in a two-party election,} between the predicted outcome at a vote share of $50\\%+\\epsilon$ and the predicted outcome at $50\\%-\\epsilon$ \\ldots''", "intended_meaning": "The RD estimate displayed on p.~6 is a difference in means over $U_b=\\{it:-k<\\mathit{DemVot}_{it}\\le k\\}$. It exists as soon as $U_b$ contains one Democratic win and one Democratic loss, so the strict inequalities of Condition 1 are sufficient for it and not necessary. For example, with $k=1/2$, margins $-3/4,-1/4,1/2,3/4$ and outcomes $1/10,1/5,4/5,9/10$, Condition 1 fails and the RD and coarsened OLS estimates both equal $3/5$. Conditions 1--3 allow a bin to be empty, and $|\\tilde U_j|^{-1}$ is then undefined. I read the intended convention as dropping empty bins; the coefficient on $\\mathit{DemWin}$ is then unique, and Proposition 1 holds. The 50\\% threshold is that of a two-party election, which the paper supposes from p.~5 on. With three parties a plurality can be below 50\\% (shares of 40\\%, 35\\% and 25\\%, say)."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-E006", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of the three lines of footnote 3. A red box marks the word Caughley in the citation Caughley and Sekhon (2011).", "caption": "Footnote 3, p. 3 (PDF p. 3). The marked surname is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E008.png"}, {"alt": "Screenshot of the three lines of footnote 4. A red box marks the word Caughley in the citation Caughley and Sekhon (2011).", "caption": "Footnote 4, p. 4 (PDF p. 4). The same misspelling.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-E008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In footnotes 3 and 4, “Caughley” should read “Caughey”.", "id": "P023-E008", "location": "Footnote 3, p. 3 (PDF p. 3), and footnote 4, p. 4 (PDF p. 4)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Footnote 3: ``\\ldots\\ Gelman and King (1990), Lee (2008), \\hl{Caughley} and Sekhon (2011) and Erikson and Titiunik (2012).'' Footnote 4: ``\\ldots\\ Lee (2008), Lee and Lemieux (2010) and \\hl{Caughley} and Sekhon (2011) for greater technical details on the RD design \\ldots''", "corrected": "Footnote 3: ``\\ldots\\ Gelman and King (1990), Lee (2008), \\hl{Caughey} and Sekhon (2011) and Erikson and Titiunik (2012).'' Footnote 4: ``\\ldots\\ Lee (2008), Lee and Lemieux (2010) and \\hl{Caughey} and Sekhon (2011) \\ldots''", "evidence": "I checked the pages: the reference list on p.~11 and the text on pp.~6, 7 and~9 have ``Caughey''.", "problem": "The surname of the first author of the cited study is misspelled in both footnotes.", "what_survives": "Nothing depends on the spelling, and the reference list identifies the work."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-E008", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Screenshot of the three lines of the Remark to Condition 3. A red box marks the words: k percentage points.", "caption": "Remark to Condition 3, p. 5 (PDF p. 5). The marked words give the bandwidth as $k$ percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-N001.png"}, {"alt": "Screenshot of four lines of text above Condition 1. Red boxes mark the sentence: Let k (the bandwidth) be a number between 0 and 1.", "caption": "Section 3.1, p. 5 (PDF p. 5), the definition of the bandwidth. The marked sentence puts $k$ between 0 and 1.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-N001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P023-T11", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T11"}], "headline": "Since the theory measures the bandwidth $k$ as a proportion, “$k$ percentage points” in the Remark to Condition 3 means $100k$ percentage points, and the 0.5 points of the application is $k=0.005$.", "id": "P023-N001", "location": "Section 3.1, p. 5 (PDF p. 5), definition of $k$ and Remark to Condition 3; Section 4, p. 7; note to Table 1, p. 8", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P023-T11"], "text": {"as_printed": "[p.~5] ``Let $k$ (the bandwidth) be a number between 0 and 1.'' \\ldots\\ ``Remark. Condition 3 is logically equivalent to producing an estimate by taking a difference in means within a bandwidth of \\hl{$k$ percentage points} of 0 on the forcing variable, $\\mathit{DemVot}_{it}$.''\n\n[p.~7] ``\\ldots\\ with \\hl{$k=0.5$ percentage points} (pp) \\ldots''\n\n[p.~8, note to Table 1] ``Bandwidth \\hl{$k=0.5$ percentage points} for Degree 0 RD estimates and coarsened control OLS, \\hl{$k=1.0$ percentage points} for Degree 1 RD estimates.''", "corrected": "[p.~5] ``\\ldots\\ within a bandwidth of \\hl{$k$ (that is, $100k$ percentage points)} of 0 on the forcing variable \\ldots''\n\n[p.~7] ``\\ldots\\ with \\hl{$k=0.005$ (0.5 percentage points)} \\ldots''\n\n[p.~8, note to Table 1] ``Bandwidth \\hl{$k=0.005$ (0.5 percentage points)} for Degree 0 RD estimates and coarsened control OLS, \\hl{$k=0.01$ (1.0 percentage point)} for Degree 1 RD estimates.''", "intended_meaning": "In Section 3.1 and Appendix B the margin $\\mathit{DemVot}_{it}$ lies in $[-1,1]$, and the formula $J=2\\lceil(1-k)/(2k)\\rceil$ of Condition 2 takes $k$ on the same scale. The bandwidths of the application are stated in percentage points. I convert 0.5 points to $k=0.005$, for which $J=200$ in Condition 2, that is, 201 bins one percentage point wide, and 1.0 point to $k=0.01$. Read literally, $k=0.5$ would give $J=2$ and a window of 50 points on each side of the threshold. No result changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-N001", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of the first paragraph of Section 4 and the two lines after it. A red box marks the words: all elections with available data.", "caption": "Section 4, p. 7 (PDF p. 7), first paragraph. The marked words state the availability rule for the sample.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "Section 4 should name the release of the data, the variables, the sample rule and the sample sizes behind Table 1; “all elections with available data” does not determine them.", "id": "P023-N002", "location": "Section 4, first paragraph, p. 7 (PDF p. 7), and Table 1, p. 8", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``We use the replication data provided by Caughey and Sekhon (2011), restricting our estimates to \\hl{all elections with available data}. Our outcome measure, $\\mathit{DV}_{i(t+1)}$ is Democratic vote share in election $t+1$ in region $i$.''", "corrected": "After the first sentence of Section 4, the text should state \\hl{the release of the data file, the variables used for the outcome, the margin and the win indicator, whether the sample is restricted by the flag \\texttt{Use} of the source study, how missing values are treated in each regression, and the number of elections behind each row of Table 1}. I cannot determine the authors' choices from the paper or from the codebook. The separate numerical and data issues are addressed in \\fid{P023-E003} and \\fid{P023-E004}.", "evidence": "I checked pp.~7--8 against the codebook of the data (the copy posted by the first author of the source study, dated 22 December 2025), which defines \\texttt{Use} on its p.~1 and \\texttt{DifDPct} on its p.~2. Independent recalculations from the corrected release of 10 April 2014 used 8,610 elections complete on \\texttt{DPctNxt}, \\texttt{DifDPct} and \\texttt{DemWin}, or 8,594 when also requiring \\texttt{Use}=1. Neither sample reproduces every row to the second decimal. The small differences in rows (a) and (c) do not change the comparison that the table serves.", "problem": "The codebook of the replication data of Caughey and Sekhon (2011) defines a flag, \\texttt{Use}, for a narrower ``RD incumbency advantage sample''. Its margin variable, \\texttt{DifDPct}, is the percentage of the leading Democrat minus that of the leading non-Democrat, while the paper defines $\\mathit{DemVot}_{it}$ as the difference between Democratic and Republican vote share. Section 4 does not say which release of the data was used, which variables stand for the margin and the outcome, whether the sample was restricted by the flag, or how missing values were treated. Table 1 reports no sample sizes and no standard errors. The table cannot be reproduced exactly from this description. This is a gap in the description, separate from the numerical and data issues in \\fid{P023-E003} and \\fid{P023-E004}.", "what_survives": "The broad comparison between the naive estimator and the others survives, with the numerical qualifications in \\fid{P023-E003} and \\fid{P023-E004}. The two definitions of the margin agree under the two-party supposition of p.~5. The propositions do not use the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-N002", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Screenshot of three lines of the proof of Proposition 1. Red boxes mark the subscript, i in U sub b, under each of the two summation signs in the formula for the OLS estimator.", "caption": "Appendix A, p. 9 (PDF p. 9), the display after “Therefore,”. The index of each of the two sums is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P023/exhibits/P023-N004.png"}], "formal_release": null, "formal_statements": [{"id": "P023-T9", "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-T9"}], "headline": "In the display after “Therefore,” in Appendix A, both sums should run over $it\\in U_b$, not $i\\in U_b$.", "id": "P023-N004", "location": "Appendix A, p. 9 (PDF p. 9), the display after “Therefore,”", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P023-T9"], "text": {"as_printed": "``Therefore, $\\hat\\beta_{OLS}=\\big(\\sum_{\\hl{i}\\in U_b}\\ddot{\\mathit{DemWin}}_{it}^{2}\\big)^{-1}\\big(\\sum_{\\hl{i}\\in U_b}\\ddot{\\mathit{DemWin}}_{it}\\,\\ddot{\\mathit{DV}}_{i(t+1)}\\big)$, the least squares estimator as applied only to observations in the interval covering the discontinuity (noting that $U_b=\\tilde U_{J/2}$).''", "corrected": "``Therefore, $\\hat\\beta_{OLS}=\\big(\\sum_{\\hl{it}\\in U_b}\\ddot{\\mathit{DemWin}}_{it}^{2}\\big)^{-1}\\big(\\sum_{\\hl{it}\\in U_b}\\ddot{\\mathit{DemWin}}_{it}\\,\\ddot{\\mathit{DV}}_{i(t+1)}\\big)$, the least squares estimator as applied only to observations in the interval covering the discontinuity (noting that $U_b=\\tilde U_{J/2}$).''", "evidence": "I checked the page: $U_b=\\{it:-k<\\mathit{DemVot}_{it}\\le k\\}$ is defined on p.~6, and the display on p.~6 and the next display on p.~9 sum over $it\\in U_b$.", "problem": "The set $U_b$ consists of pairs $it$, and the summands depend on $t$, so a sum over $i$ alone is not defined.", "what_survives": "The proof of Proposition 1 is unaffected; the intended sums are over the pairs in the central bin."}, "url": "https://pmaronow.github.io/llm-audit/works/P023/#P023-N004", "work": {"arxiv_id": null, "doi": "10.1515/spp-2014-0003", "title": "A Note on Close Elections and Regression Analysis of the Party Incumbency Advantage", "url": "https://pmaronow.github.io/llm-audit/works/P023/"}, "work_id": "P023"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of eight lines of a paragraph on page 174. Red boxes mark the sentence 'In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time', which runs over two lines. The lines above it introduce a within-subjects design that randomizes the timing of the treatment and say that this approach relies on fairly strong assumptions. The text after it begins 'A stronger design would be to deploy a series of different messages'.", "caption": "Section “Single-Medium Multi-Outcome Crossover Design”, p. 174 (PDF p. 8), first paragraph. The marked sentence states the condition directly after the paragraph has introduced the design that randomizes the timing of the treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-A002.png"}], "formal_release": null, "formal_statements": [{"id": "P024-T1", "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-T1"}, {"id": "P024-T2", "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-T2"}], "headline": "A design that randomizes the timing of the treatment does not require the treatment to be the sole systematic factor that changes over time; I narrow the sentence to a warning about unadjusted before-after comparisons.", "id": "P024-A002", "location": "Section “Single-Medium Multi-Outcome Crossover Design”, p. 174 (PDF p. 8), first paragraph, sentence beginning “In order to attribute a change in outcomes to the treatment”", "pdf_page": 8, "printed_page": "174", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P024-T1", "P024-T2"], "text": {"as_printed": "``This constraint obviously rules out between-subjects designs. One alternative is to deploy a within-subjects design that randomizes the timing of the treatment. This approach can be informative, but it relies on fairly strong assumptions (see Gerber \\& Green, 2012, chapter 8). In order to attribute a change in outcomes to the treatment, \\hl{one must assume that the treatment is the sole systematic factor that changes over time}. A stronger design would be to deploy a series of different messages in this media market and gauges distinct outcomes that measure responses to each message.''", "corrected": "The sentence ``In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time.'' is replaced by the following three sentences.\n\n``\\hl{An unadjusted before-after change can reflect other factors that vary over time.} \\hl{With a known randomization mechanism, randomized timing can support design-based effect estimation and sharp-null randomization inference despite fixed calendar trends.} \\hl{Effect estimation must specify the estimand, ensure the relevant assignment probabilities are positive, and address treatment history, anticipation, and carryover in its estimand or assumptions.}''\n\nThe second of these sentences rests on the following two statements. Let the onset $S$ of a permanent treatment be drawn by a known randomization mechanism, and let $Z_t=\\mathbf{1}\\{t\\ge S\\}$. Suppose that the potential outcomes are fixed and that the outcome in period $t$ depends on $S$ only through $Z_t$, with values $Y_t(1)$ and $Y_t(0)$. Set $p_t=\\mathbb{P}(Z_t=1)$, and let $\\mathcal{T}$ be a nonempty finite set of periods such that $0<p_t<1$ for all $t\\in\\mathcal{T}$. Then\n\\[\n\\mathbb{E}\\biggl[\\frac{1}{|\\mathcal{T}|}\\sum_{t\\in\\mathcal{T}}\\Bigl\\{\\frac{Z_tY_t}{p_t}-\\frac{(1-Z_t)Y_t}{1-p_t}\\Bigr\\}\\biggr]\n=\\frac{1}{|\\mathcal{T}|}\\sum_{t\\in\\mathcal{T}}\\bigl[Y_t(1)-Y_t(0)\\bigr],\n\\]\nwith no condition on how $Y_t(0)$ varies with $t$. Next, suppose that $S$ is uniform on a finite set of $m$ onsets and that the outcome path is the same for every onset, which is the sharp null hypothesis. For a statistic $D$, set $p(s)=\\#\\{u:D(u)\\ge D(s)\\}/m$. Then $\\mathbb{P}(p(S)\\le\\alpha)\\le\\alpha$ for every $\\alpha\\in[0,1]$, whatever the outcome path. This test needs no separate assumption about trends, anticipation or carryover, because the sharp null hypothesis fixes the outcomes under every onset.", "evidence": "The evidence is an exact counterexample, computed in rational arithmetic. Consider one outlet, one outcome observed in the periods $t=1,2,3,4$, and a treatment that is permanent once it starts. The onset $S$ is uniform on $\\{2,3,4\\}$, and $Z_t=\\mathbf{1}\\{t\\ge S\\}$ indicates treatment in period $t$. The potential outcomes are\n\\[\nY_t(S)=t^2+2\\cdot\\mathbf{1}\\{t\\ge S\\},\\qquad t=1,2,3,4.\n\\]\nThe outcome depends on the onset only through $Z_t$, and I write $Y_t(0)=t^2$ and $Y_t(1)=t^2+2$. The effect of the treatment is $2$ in every period, and the untreated outcomes $1,4,9,16$ follow a fixed trend. The treatment is therefore not the sole systematic factor that changes over time. Set $p_t=\\mathbb{P}(Z_t=1)$, so that $(p_1,p_2,p_3,p_4)=(0,1/3,2/3,1)$. Treatment and control both have positive probability in periods 2 and 3 only. I define\n\\[\n\\tau=\\frac{1}{2}\\sum_{t=2}^{3}\\bigl[Y_t(1)-Y_t(0)\\bigr]=2,\\qquad\n\\widehat{\\tau}=\\frac{1}{2}\\sum_{t=2}^{3}\\Bigl\\{\\frac{Z_tY_t}{p_t}-\\frac{(1-Z_t)Y_t}{1-p_t}\\Bigr\\},\n\\]\nwhere $Y_t$ is the observed outcome. Since $Z_tY_t=Z_tY_t(1)$, $(1-Z_t)Y_t=(1-Z_t)Y_t(0)$ and $\\mathbb{E}[Z_t]=p_t$, I have\n\\[\n\\mathbb{E}\\Bigl[\\frac{Z_tY_t}{p_t}\\Bigr]=Y_t(1),\\qquad\n\\mathbb{E}\\Bigl[\\frac{(1-Z_t)Y_t}{1-p_t}\\Bigr]=Y_t(0),\n\\]\nand hence $\\mathbb{E}[\\widehat{\\tau}]=\\tau$. Note that no step uses the form of $Y_t(0)$. For the three onsets, I obtain $\\widehat{\\tau}=69/4$, $21/4$ and $-33/2$, and the mean of these values is $2$.\n\nFor the unadjusted comparison, let $D(s)$ be the mean outcome over the periods $t\\ge s$ minus the mean outcome over the periods $t<s$. Under the sharp null hypothesis of no effect, $Y_t=t^2$ for every onset, and a direct computation gives $D(2)=26/3$, $D(3)=10$ and $D(4)=34/3$. The expectation of $D(S)$ is $10$, although the effect is zero. With the effect of $2$ restored, the expectation is $12$. Randomizing the onset does not make the unadjusted difference unbiased, and this is the content of the narrower warning.\n\nUnder the sharp null hypothesis, the outcome path is the same for every onset, and the distribution of $D(S)$ is therefore known. The upper-tail $p$ value $p(s)=\\#\\{u:D(u)\\ge D(s)\\}/3$ takes the values $1$, $2/3$ and $1/3$, and $\\mathbb{P}(p(S)\\le k/3)=k/3$ for $k=0,1,2,3$. The test is valid despite the trend. The same argument applies to any outcome path and to $m$ equally likely onsets. If $k>0$ onsets have $p$ values at most $\\alpha$, then the smallest statistic among them has at least $k$ onsets at or above it, and its $p$ value is at least $k/m$. This implies that $k/m\\le\\alpha$, and the probability of rejection is $k/m$. If $k=0$, the probability of rejection is zero.\n\n\\begin{audittable}{lrrr}{The counterexample, by onset. The weighted estimate is computed with a true effect of 2. The unadjusted difference and its $p$ value are computed with a true effect of 0.}\nOnset $S$ & Weighted estimate $\\widehat{\\tau}$ & Unadjusted difference $D$ & $p$ value of $D$ \\\\\n$2$ & $69/4$ & $26/3$ & $1$ \\\\\n$3$ & $21/4$ & $10$ & $2/3$ \\\\\n$4$ & $-33/2$ & $34/3$ & $1/3$ \\\\\nMean over onsets & $2$ & $10$ & \\\\\n\\end{audittable}\n\nAn independent recalculation agrees with these values. In a second example, with six periods, onsets uniform on $\\{2,\\dots,6\\}$ and untreated outcomes $10t+3t^2$, I obtain the estimates $2505/16$, $1665/16$, $695/16$, $-785/16$ and $-245$, whose mean is again the true effect $2$. The four-period example has limits. The estimator $\\widehat{\\tau}$ is unbiased and imprecise, with variance $1561/8$. The example uses the fact that the outcome in a period depends on the onset only through whether the treatment has begun, and it estimates the effect only at the periods in which both treatment and control have positive probability. It does not show that the multi-message design has no advantage in precision or robustness.", "problem": "The paragraph introduces ``a within-subjects design that randomizes the timing of the treatment'', says that this approach relies on fairly strong assumptions, and then states the condition. Read in this order, the sentence says that the design with randomized timing can attribute a change in outcomes to the treatment only if nothing else systematic changes over time. This is not required. Suppose that the timing is drawn by a known randomization mechanism. Then the probability of treatment in each period is known. If the outcome in a period depends on the timing only through whether the treatment has begun, then an estimator that weights the outcomes by these probabilities is unbiased for the average effect over the periods in which both treatment and control have positive probability, whatever the calendar trend. Further, under the sharp null hypothesis of no effect, the outcome path is the same for every possible timing. A randomization test of this hypothesis is then valid whatever the trend. An unadjusted before-after difference can nevertheless reflect factors other than treatment. That warning is sound, but the paragraph does not limit the sentence to such comparisons. The next sentence calls the multi-message design ``stronger''. The ground for this comparison needs the same qualification, since the single-message design with randomized timing does not rest on an assumption of no trend. The word ``gauges'' in the last quoted sentence is a misprint; see \\fid{P024-E015}.", "what_survives": "The warning about unadjusted before-after comparisons stands. The radio example with six organizations, the television example with nine segments and the randomization tests of the sharp null hypothesis that are proposed for them remain viable. Such a test is valid when it uses the actual randomization mechanism and the stated sharp null hypothesis, and it does not require the absence of calendar trends. The effect estimators and the assumptions about treatment history in those two studies were not examined. There are two changes. The necessity claim becomes a warning, and the reason for calling the multi-message design stronger can no longer be that randomized timing requires the absence of a trend. No number, table or later result in the article is derived from the sentence, and the other design sections do not depend on it."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-A002", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the end of the paragraph on the experiment of Bond et al. A red box marks the words 'approximately 0.7 percentage points.' in the sentence 'Public voting records revealed a significantly higher voting rate among the treatment group, with an average treatment effect of approximately 0.7 percentage points.' The next sentence ends 'had no apparent effect.'", "caption": "Section “Between-Subjects Design”, p. 171 (PDF p. 5). The marked words give the size of the effect that the article attributes to Bond et al. (2012).", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E001.png"}, {"alt": "Crop of six lines of the cited article by Bond et al. A red box marks the figure 0.39 percent in the sentence 'who received the social message were 0.39 percent (s.e.m., 0.17 percent; t-test, P = 0.02) more likely to vote than users who received no message at all.' The following sentence says that the difference in voting between those who received the social message and those who received the informational message was 0.39 percent (s.e.m., 0.17 percent; t-test, P = 0.02).", "caption": "The cited article, Bond et al. (2012), Nature 489, p. 296, left column. The marked estimate is the difference in validated turnout between users who received the social message and users who received no message.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E001-b.png"}, {"alt": "Dot plot with a horizontal axis from 0 to 0.8 percentage points of validated turnout and three rows. In the row 'Social message minus no message' a blue circle at 0.39 is the estimate reported by Bond et al. and an orange diamond at 0.7 is the value printed in the article; an arrow between them is labelled 'printed value is 0.31 higher'. In the row 'Social message minus informational message' a blue circle is at 0.39. In the row 'Informational message minus no message' a blue circle is at 0.00. The title reads 'The article prints 0.7 percentage points; Bond et al. (2012) report 0.39'.", "caption": "The value printed on p. 171 beside the three differences in validated turnout that Bond et al. (2012) report, in percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E001-turnout-effect.png"}], "formal_release": null, "formal_statements": [], "headline": "The effect on turnout attributed to Bond et al. (2012) should read approximately 0.4 percentage points, not approximately 0.7: the cited article reports 0.39.", "id": "P024-E001", "location": "Section “Between-Subjects Design”, second paragraph on p. 171 (PDF p. 5), sentence beginning “Public voting records revealed”", "pdf_page": 5, "printed_page": "171", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``In order to estimate the average causal effect of an application that both invited Facebook users to indicate whether they had voted and showed which of their friends had voted, Bond et al.\\ simply compared those assigned to receive the application to those who were assigned not to receive it. Public voting records revealed a significantly higher voting rate among the treatment group, with an average treatment effect of approximately \\hl{0.7} percentage points. A different application that invited randomly selected Facebook users to indicate whether they had voted but did not indicate whether their friends had voted had no apparent effect.''", "corrected": "``Public voting records revealed a significantly higher voting rate among the treatment group, with an average treatment effect of approximately \\hl{0.4} percentage points.''\n\nThe estimate reported by Bond et al.\\ (2012, p.~296) is 0.39 percentage points, with standard error 0.17 percentage points and $P=0.02$, for users who received the social message compared with users who received no message.", "evidence": "I checked the cited article and compared the two values exactly. On p.~296 of Bond et al.\\ (2012), the paragraph on validated voting states that users ``who received the social message were 0.39\\% (s.e.m., 0.17\\%; $t$-test, $P=0.02$) more likely to vote than users who received no message at all'', and that the difference between the social message and the informational message was also 0.39\\%. Figure~1 of that article and p.~8 of its supplementary information agree. The table lists the three reported comparisons.\n\\begin{audittable}{lrrr}{Differences in validated turnout reported by Bond et al.\\ (2012), in percentage points, and the value printed in the article.}\nComparison & Estimate & Standard error & $P$ \\\\\nSocial message against no message & 0.39 & 0.17 & 0.02 \\\\\nSocial message against informational message & 0.39 & 0.17 & 0.02 \\\\\nInformational message against no message & 0.00 & 0.28 & 0.98 \\\\\nAs printed on p.~171 of the article & 0.7 & & \\\\\n\\end{audittable}\nI found no overall direct estimate of 0.7 percentage points in the article of Bond et al.\\ or in its supplement. The numeral 0.7 occurs in Table~S6 of the supplement (p.~20), where it is an interaction threshold and not an average effect on turnout. A value that rounds to 0.39 lies in $[0.385,0.395)$, and a value that rounds to 0.7 lies in $[0.65,0.75)$; the two intervals are $0.255$ apart.", "problem": "The sentence gives the size of the direct effect that Bond et al.\\ (2012) estimated. This effect is the difference in turnout, taken from public voting records, between users assigned to the application that showed which friends had voted and users not assigned to it. Bond et al.\\ call that application the social message, and they call the application that did not show the votes of friends the informational message. They report the difference between users who received the social message and users who received no message as 0.39 percentage points, with standard error 0.17 percentage points and $P=0.02$. They report the same estimate when the comparison group is the users who received the informational message. The printed value is larger by 0.31 percentage points, a factor of $70/39\\approx 1.8$. It is not a rounding of the reported estimate, which rounds to 0.4. A different comparison group does not account for it, since both comparisons give 0.39. The estimates of Bond et al.\\ for the friends of treated users concern other comparisons, and the article discusses them separately, in the next paragraph and in Note~2. It is not established where the value 0.7 came from.", "what_survives": "The qualitative statements of the paragraph stand. Turnout was higher among users who received the social message, and the informational message had no apparent effect. Only the size of the effect changes. The value 0.7 occurs once in the article, and I found no later numerical use of it."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E001", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the paragraph that begins 'Clustered assignment poses two statistical complications'. Red boxes mark the words 'difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of clusters becomes very large,' which run over two lines. The sentence continues 'but bias may be appreciable in studies with only a few dozen clusters of varying size.'", "caption": "Section “Between-Subjects Clustered Design”, second paragraph, p. 171 (PDF p. 5). The marked clause states consistency with no condition other than a large number of clusters.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P024-T4", "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-T4"}, {"id": "P024-T5", "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-T5"}], "headline": "The difference-in-means estimator is consistent as the number of clusters grows only under further conditions, the main one being that no cluster keeps a nonvanishing share of all subjects.", "id": "P024-E002", "location": "Section “Between-Subjects Clustered Design”, second paragraph, p. 171 (PDF p. 5), sentence beginning “The difference-in-means estimator is consistent”", "pdf_page": 5, "printed_page": "171", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P024-T4", "P024-T5"], "text": {"as_printed": "``Clustered assignment poses two statistical complications. The first is that when clusters contain different numbers of subjects, the standard estimation approach---comparing treatment and control group averages---no longer gives unbiased estimates of the average treatment effect among all subjects (Middleton, 2008, Middleton \\& Aronow, 2011). The difference-in-means estimator is consistent, which is to say that \\hl{it converges to the correct answer as the number of clusters becomes very large}, but bias may be appreciable in studies with only a few dozen clusters of varying size.''", "corrected": "``\\hl{With uniformly bounded potential outcomes and a common cluster treatment probability bounded away from zero and one,} the difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of \\hl{independently assigned} clusters becomes very large, \\hl{provided that no cluster accounts for a nonvanishing share of all subjects}, but bias may be appreciable in studies with only a few dozen clusters of varying size.''\n\nA full statement, which also covers complete randomization of the clusters, is as follows. For each $K$, consider a finite population of $N$ subjects in $K$ clusters of sizes $n_1,\\dots,n_K$, with fixed potential outcomes and no interference between subjects. Suppose that the following four conditions hold.\n\\begin{enumerate}\n\\item Every potential outcome is at most $B$ in absolute value, where $B$ does not depend on $K$.\n\\item Either the clusters are assigned to treatment independently with a common probability $p_K$, or $m_K$ of the $K$ clusters are treated, all such allocations being equally likely, and $p_K=m_K/K$.\n\\item There exists $\\eta>0$, not depending on $K$, such that $\\eta\\le p_K\\le 1-\\eta$.\n\\item $\\max_j n_j/N\\to 0$ as $K\\to\\infty$, or equivalently $\\sum_j(n_j/N)^2\\to 0$.\n\\end{enumerate}\nLet $\\widehat\\tau_K$ denote the difference between the mean observed outcome of the treated subjects and that of the control subjects, and let $\\tau_K$ denote the average treatment effect among the $N$ subjects. Then $\\mathbb{P}(|\\widehat\\tau_K-\\tau_K|>\\varepsilon)\\to 0$ as $K\\to\\infty$, for every $\\varepsilon>0$. The four conditions are sufficient, and I do not claim that they are necessary. A growing number of clusters alone is not sufficient when one cluster keeps a nonvanishing share of the subjects.", "evidence": "The evidence is an exact counterexample to the sentence as printed and a derivation of the corrected statement.\n\n\\emph{Counterexample.} Fix an integer $K\\ge 2$. The population has one cluster of $K^2$ subjects and $K-1$ clusters of one subject each. Every subject in the large cluster has potential outcomes $Y(0)=1/2$ and $Y(1)=3/4$, and each of the other subjects has $Y(0)=1/4$ and $Y(1)=1/2$. Every subject has treatment effect $1/4$, so that the average treatment effect is $\\tau=1/4$ for every $K$. All outcomes lie strictly between $0$ and $1$, all are observed, and there is no interference. The share of the subjects in the large cluster is $K^2/(K^2+K-1)$, which tends to $1$.\n\nLet $\\widehat\\tau$ denote the difference in means, and let $q$ denote the number of one-subject clusters in the same arm as the large cluster. Suppose that both arms are nonempty, so that $0\\le q\\le K-2$ and the other arm holds only one-subject clusters. If the large cluster is treated, then the mean outcome of the treated arm is $((3/4)K^2+q/2)/(K^2+q)$ and that of the control arm is $1/4$. If the large cluster is in control, then the two means are $1/2$ and $((1/2)K^2+q/4)/(K^2+q)$. Subtracting, I obtain\n\\[\n\\begin{aligned}\n\\widehat\\tau&=\\frac14+\\frac14\\,\\frac{K^2}{K^2+q}&&\\text{if the large cluster is treated,}\\\\\n\\widehat\\tau&=\\frac14-\\frac14\\,\\frac{K^2}{K^2+q}&&\\text{if it is in control.}\n\\end{aligned}\n\\]\nIn both cases\n\\[\n|\\widehat\\tau-\\tau|\\ge\\frac14\\,\\frac{K^2}{K^2+K-2}>\\frac18 .\n\\]\nSuppose first that the $K$ clusters are assigned independently, each to treatment with probability $1/2$. Then both arms are nonempty with probability $1-2^{1-K}$, and I obtain $\\mathbb{P}(|\\widehat\\tau-\\tau|>1/8)\\ge 1-2^{1-K}\\to 1$. Suppose next that $K$ is even and that exactly $K/2$ clusters are treated, all such allocations being equally likely. Then $q=K/2-1$, and $\\widehat\\tau=1/4\\pm r_K/4$ with probability $1/2$ each, where $r_K=K^2/(K^2+K/2-1)$. In this design the estimator is unbiased for every $K$, its variance $r_K^2/16$ tends to $1/16$, and it is not consistent. For $K=6$ the twenty allocations give the estimate $37/76$ ten times and $1/76$ ten times. The table lists exact values for $K\\le 12$, found by enumerating every allocation in rational arithmetic.\n\\begin{audittable}{rrrr}{The counterexample for $K\\le 12$. Second column: the variance of $\\widehat\\tau$ when $K/2$ clusters are treated; in this design $\\widehat\\tau$ has mean $1/4$ and differs from $\\tau=1/4$ by more than $1/8$ in every allocation. Third and fourth columns: independent assignment with probability $1/2$; the smallest error is taken over the allocations with both arms nonempty.}\n$K$ & Variance of $\\widehat\\tau$ & $\\mathbb{P}$(both arms nonempty) & Smallest $|\\widehat\\tau-\\tau|$ \\\\\n2 & $1/16$ & $1/2$ & $1/4$ \\\\\n4 & $16/289$ & $7/8$ & $2/9$ \\\\\n6 & $81/1444$ & $31/32$ & $9/40$ \\\\\n8 & $256/4489$ & $127/128$ & $8/35$ \\\\\n10 & $625/10816$ & $511/512$ & $25/108$ \\\\\n12 & $1296/22201$ & $2047/2048$ & $18/77$ \\\\\nLimit & $1/16$ & $1$ & $1/4$ \\\\\n\\end{audittable}\n\n\\emph{Derivation of the corrected statement.} For each $K$, let the population consist of $K$ clusters of sizes $n_1,\\dots,n_K$, and set $N=\\sum_j n_j$ and $w_j=n_j/N$. Let $\\bar Y_j(z)$ denote the mean potential outcome of cluster $j$ under condition $z\\in\\{0,1\\}$, and set $\\mu_z=\\sum_j w_j\\bar Y_j(z)$, so that the average treatment effect is $\\tau_K=\\mu_1-\\mu_0$. Let $Z_j$ denote the treatment indicator of cluster $j$, and define\n\\[\nA=\\sum_j Z_jw_j\\bar Y_j(1),\\qquad C=\\sum_j(1-Z_j)w_j\\bar Y_j(0),\\qquad D=\\sum_j Z_jw_j .\n\\]\nThen the difference in means is $\\widehat\\tau_K=A/D-C/(1-D)$. Suppose that there exist constants $B$ and $\\eta>0$, not depending on $K$, such that every potential outcome is at most $B$ in absolute value and each cluster is treated with the same probability $p_K$, where $\\eta\\le p_K\\le 1-\\eta$. Suppose also that $\\max_j w_j\\to 0$.\n\nSince the weights $w_j$ are nonnegative and sum to one, I have\n\\[\n\\Bigl(\\max_j w_j\\Bigr)^2\\le\\sum_j w_j^2\\le\\max_j w_j ,\n\\]\nso that $\\max_j w_j\\to 0$ if and only if $\\sum_j w_j^2\\to 0$. The expectations of $A$, $C$ and $D$ are $p_K\\mu_1$, $(1-p_K)\\mu_0$ and $p_K$. Let $x_1,\\dots,x_K$ be fixed numbers. If the clusters are assigned independently, then $\\operatorname{Var}\\bigl(\\sum_j Z_jx_j\\bigr)=p_K(1-p_K)\\sum_j x_j^2$. If instead $m_K=p_KK$ clusters are treated and all such allocations are equally likely, then $\\operatorname{Cov}(Z_j,Z_l)=-p_K(1-p_K)/(K-1)$ for $j\\ne l$, and expanding the variance gives\n\\[\n\\operatorname{Var}\\Bigl(\\sum_j Z_jx_j\\Bigr)=p_K(1-p_K)\\,\\frac{K}{K-1}\\Bigl[\\sum_j x_j^2-\\frac1K\\Bigl(\\sum_j x_j\\Bigr)^2\\Bigr]\\le 2\\,p_K(1-p_K)\\sum_j x_j^2 .\n\\]\nIn the last inequality, I used $K/(K-1)\\le 2$ and dropped a nonnegative term. Since $p_K(1-p_K)\\le 1/4$ and $|\\bar Y_j(z)|\\le B$, in both designs the variances of $A$ and $C$ are at most $(B^2/2)\\sum_j w_j^2$, and the variance of $D$ is at most $(1/2)\\sum_j w_j^2$. These bounds tend to zero, and Chebyshev's inequality implies that $A-p_K\\mu_1$, $C-(1-p_K)\\mu_0$ and $D-p_K$ converge to zero in probability. On the event $|D-p_K|<\\eta/2$, both $D$ and $1-D$ are at least $\\eta/2$. Using $|\\mu_z|\\le B$, I find\n\\[\n\\begin{aligned}\n\\Bigl|\\frac{A}{D}-\\mu_1\\Bigr|&\\le\\frac{2}{\\eta}\\bigl(|A-p_K\\mu_1|+B\\,|D-p_K|\\bigr),\\\\\n\\Bigl|\\frac{C}{1-D}-\\mu_0\\Bigr|&\\le\\frac{2}{\\eta}\\bigl(|C-(1-p_K)\\mu_0|+B\\,|D-p_K|\\bigr)\n\\end{aligned}\n\\]\non this event. I conclude that $\\widehat\\tau_K-\\tau_K\\to 0$ in probability. An allocation with an empty arm lies in the event $|D-p_K|\\ge\\eta/2$, whose probability tends to zero. The conclusion therefore does not depend on how the estimator is defined on such allocations. Note that in the counterexample $\\max_j w_j=K^2/(K^2+K-1)\\to 1$, while the other hypotheses hold.", "problem": "The sentence says that the difference between the mean outcomes of treated and control subjects converges to the average treatment effect among all subjects as the number of clusters grows. A large number of clusters is not sufficient for this. If one cluster keeps a nonvanishing share of all subjects, then the assignment of that cluster can continue to move the estimate, however many other clusters are added. The article does not say how the clusters grow, and it states no condition that excludes such a cluster. Under the usual regularity conditions for a law of large numbers the sentence is correct, and the correction supplies them.", "what_survives": "The rest of the paragraph stands. The difference in means can be biased when cluster sizes vary, the bias can matter with a few dozen clusters, and blocking on cluster size is a remedy. The discussion of standard errors and of randomization inference, and the distinction between individual-level and cluster-level effects, are unaffected. No later statement of the article is derived from the consistency sentence, and no estimate of a cited study changes. The check does not show that any cited application has a cluster of this kind."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E002", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the paragraph that begins 'Some recent examples of between-subjects clustered designs'. Red boxes mark the words 'assigned 85 cable TV systems of varying sizes to receive Rock The Vote ads', which run over two lines, in the sentence that begins 'Green and Vavreck (2008)' and ends 'encouraging voter participation.'", "caption": "Section “Between-Subjects Clustered Design”, p. 172 (PDF p. 6). The marked words attach the count 85 to the systems assigned to receive ads.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E003.png"}], "formal_release": null, "formal_statements": [], "headline": "The 85 cable TV systems are all the systems that were randomized, to the ads or to control; 85 is not the number of systems that received ads.", "id": "P024-E003", "location": "Section “Between-Subjects Clustered Design”, paragraph beginning “Some recent examples of between-subjects clustered designs”, second sentence, p. 172 (PDF p. 6)", "pdf_page": 6, "printed_page": "172", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Green and Vavreck (2008) \\hl{assigned 85 cable TV systems} of varying sizes \\hl{to receive} Rock The Vote ads encouraging voter participation.''", "corrected": "``Green and Vavreck (2008) \\hl{randomly} assigned 85 cable TV systems of varying sizes to receive Rock The Vote ads encouraging voter participation \\hl{or to serve as controls}.''", "intended_meaning": "The sentence describes an experiment that randomized exposure to the ads over a pool of 85 cable systems, some assigned to treatment and the others to control. The preceding page defines this design. The same paragraph describes another study as assigning clusters of Facebook users ``to receive'' advertisements, and it then refers to ``both the treatment and control groups'' of that study. Under this reading the sentence is correct. Read literally, as a count of the systems assigned to the ads, it would be false. In Green and Vavreck (2008), and in the documentation of the data set \\texttt{RockTheVote} in the R package \\texttt{pscl}, 85 is the size of the whole pool, controls included. The two sources differ on the division of the pool. The documentation gives 42 treated and 43 control systems. Green and Vavreck (2008) describe the allocation as one treated system in each of 40 strata, which implies 40 treated and 45 control systems. I did not resolve this difference, and the corrected wording does not depend on it. The count 85 occurs once in the article, and no calculation uses it."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E003", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines at the end of the first paragraph of the section. A red box marks the words 'clusters are assigned independently,' in the sentence 'In the between-subjects clustered design, clusters are assigned independently, but individuals within clusters are not.' The sentence before it reads 'In effect, all individuals living within the same geographic cluster are assigned together to either treatment or control conditions.'", "caption": "Section “Between-Subjects Clustered Design”, first paragraph, p. 171 (PDF p. 5). The marked words describe the assignment of clusters.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E004.png"}], "formal_release": null, "formal_statements": [], "headline": "“Clusters are assigned independently” means that the clusters, and not the individuals, are the units of random assignment; the assignments of different clusters need not be stochastically independent.", "id": "P024-E004", "location": "Section “Between-Subjects Clustered Design”, last sentence of the first paragraph, p. 171 (PDF p. 5)", "pdf_page": 5, "printed_page": "171", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``In effect, all individuals living within the same geographic cluster are assigned together to either treatment or control conditions. In the between-subjects clustered design, clusters are assigned \\hl{independently}, but individuals within clusters are not.''", "corrected": "``In the between-subjects clustered design, \\hl{random assignment occurs at the cluster level, and the individuals within each cluster are assigned jointly, according to the cluster-level randomization scheme}.''", "intended_meaning": "The sentence contrasts the clusters, which are the units of assignment, with the individuals in a cluster, who share one assignment. This is correct, and no later statement in the article uses independence between the assignments of different clusters. In the technical sense of the word the description would be too narrow, because stochastic independence across clusters is not a defining property of cluster randomization. Suppose that two of four clusters are treated and that the six allocations are equally likely, as in the illustration with four media markets later on the same page. Let $Z_1$ and $Z_2$ denote the treatment indicators of two clusters. Then $\\mathbb{P}(Z_1=1)=\\mathbb{P}(Z_2=1)=1/2$ and $\\mathbb{P}(Z_1=1,Z_2=1)=1/6$, which is not $1/4$, and $\\operatorname{Cov}(Z_1,Z_2)=-1/12$. Note that independent assignment of each cluster with a fixed probability remains a valid special case."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E004", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the first paragraph of the article. A red box marks the citation 'Arceneaux (2009)' in the sentence 'A vast research literature has accumulated, but as Arceneaux (2009) notes in his recent review, this literature is hampered by methodological shortcomings.'", "caption": "Introduction, first paragraph, p. 168 (PDF p. 2). The marked citation gives the year 2009.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E005.png"}, {"alt": "Crop of seven lines of the reference list. A red box marks 'Arceneaux, K. (2010).' at the start of the entry 'The benefits of experimental methods for the study of campaign effects. Political Communication, 27, 199-215.' The next entry is by Arceneaux and Nickerson, is dated 2009 and has the title 'Who is mobilized to vote? A re-analysis of 11 field experiments.'", "caption": "Reference list, p. 177 (PDF p. 11). The marked entry is the only work listed under Arceneaux alone, and it is dated 2010. The entry below it, dated 2009, has a second author.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E005-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The citation “Arceneaux (2009)” on p. 168 should read “Arceneaux (2010)”, the year of the review in the reference list.", "id": "P024-E005", "location": "Introduction, first paragraph, second sentence, p. 168 (PDF p. 2); reference list, p. 177 (PDF p. 11)", "pdf_page": 2, "printed_page": "168", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~168: ``A vast research literature has accumulated, but as Arceneaux (\\hl{2009}) notes in his recent review, this literature is hampered by methodological shortcomings.''\n\nReference list, p.~177: ``Arceneaux, K. (2010). The benefits of experimental methods for the study of campaign effects. \\emph{Political Communication}, \\emph{27}, 199--215.''", "corrected": "p.~168: ``\\ldots as Arceneaux (\\hl{2010}) notes in his recent review \\ldots''", "evidence": "This is a check of the page. I read p.~168 and the reference list on p.~177, and I looked at every occurrence of the name in the article. I did not compare the year of the review with a source outside the article. The article alone does not show whether a review of 2009 was meant and its entry omitted.", "problem": "The year in the text matches no entry in the reference list. The only work listed under Arceneaux alone is the review of 2010. The only Arceneaux entry dated 2009 is Arceneaux and Nickerson (2009), which is a different work and is cited once, on p.~172.", "what_survives": "Everything else is unaffected. No statement depends on the year."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E005", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the three-line entry for Broockman and Green (in press) and the two-line entry for Chong and Druckman (2010) below it. A red box marks 's1109' in the DOI, which is printed as 10.1007/s1109-013-9239-z.", "caption": "References, p. 178 (PDF p. 12), first entry on the page. The marked journal code in the DOI is missing a digit.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E006.png"}], "formal_release": null, "formal_statements": [], "headline": "The DOI in the entry for Broockman and Green (in press) is missing a digit and should read 10.1007/s11109-013-9239-z.", "id": "P024-E006", "location": "References, p. 178 (PDF p. 12), entry for Broockman and Green (in press): the DOI", "pdf_page": 12, "printed_page": "178", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Broockman, D. E., \\& Green, D. P. (in press). Do online advertisements increase political candidates’ name recognition or favorability? Evidence from randomized field experiments. \\emph{Political Behavior}. doi: 10.1007/\\hl{s1109}-013-9239-z", "corrected": "Broockman, D. E., \\& Green, D. P. (in press). Do online advertisements increase political candidates’ name recognition or favorability? Evidence from randomized field experiments. \\emph{Political Behavior}. doi: 10.1007/\\hl{s11109}-013-9239-z", "evidence": "The evidence is a check of the page against the publisher's record. The record of Springer for the article gives the DOI as 10.1007/s11109-013-9239-z.", "problem": "The journal code in the DOI is printed as s1109, and one digit 1 is missing from it.", "what_survives": "The article remains identifiable from its authors, title and journal, and no statement about the study in the text depends on the DOI."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E006", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three lines of the reference list: the two-line entry for Paluck (2009) and the first line of the entry for Paluck (2010). A red box marks the page range '594-600' at the end of the first entry, after Journal of Personality and Social Psychology, 96.", "caption": "References, p. 179 (PDF p. 13), entry for Paluck (2009). The marked page range is wrong.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E007.png"}], "formal_release": null, "formal_statements": [], "headline": "The page range of Paluck (2009) is 574–587, not 594–600.", "id": "P024-E007", "location": "References, p. 179 (PDF p. 13), entry for Paluck (2009): the page range", "pdf_page": 13, "printed_page": "179", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Paluck, E. L. (2009). Reducing intergroup prejudice and conflict using the media: A field experiment in Rwanda. \\emph{Journal of Personality and Social Psychology}, \\emph{96}, \\hl{594--600}.", "corrected": "Paluck, E. L. (2009). Reducing intergroup prejudice and conflict using the media: A field experiment in Rwanda. \\emph{Journal of Personality and Social Psychology}, \\emph{96}, \\hl{574--587}.", "evidence": "The evidence is a check of the page against the article itself. The line at the foot of its first page gives volume 96, number 3, pp.~574--587, and the publication record of Princeton University agrees.", "problem": "The article occupies pp.~574--587 of volume 96 of the journal, in issue 3, so both ends of the printed range are wrong.", "what_survives": "The article remains identifiable from its author, title, journal and volume, and no statement about the study in the text depends on the page range."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E007", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of four lines of the reference list: the last line of the entry for Paluck (2010), the two-line entry for Paluck and Green (2009), and the first line of the entry for Paluck, Lagunes, Green, Vavreck and Peer (2013). Red boxes mark the subtitle 'A field experiment on a mass media intervention in Rwanda', which runs over two lines, and the volume number '104' after American Political Science Review.", "caption": "References, p. 179 (PDF p. 13), entry for Paluck and Green (2009). The marked subtitle and the marked volume number differ from the published article.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E008.png"}], "formal_release": null, "formal_statements": [], "headline": "The entry for Paluck and Green (2009) gives volume 104 for an article in volume 103, and a subtitle that is not the published one.", "id": "P024-E008", "location": "References, p. 179 (PDF p. 13), entry for Paluck and Green (2009): the subtitle and the volume", "pdf_page": 13, "printed_page": "179", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Paluck, E. L., \\& Green, D. P. (2009). Deference, dissent, and dispute resolution: \\hl{A field experiment on a mass media intervention in Rwanda}. \\emph{American Political Science Review}, \\emph{\\hl{104}}, 622--644.", "corrected": "Paluck, E. L., \\& Green, D. P. (2009). Deference, dissent, and dispute resolution: \\hl{An experimental intervention using mass media to change norms and behavior in Rwanda}. \\emph{American Political Science Review}, \\emph{\\hl{103}}, 622--644.", "evidence": "The evidence is a check of the page against the record of Cambridge University Press for the article, which gives this subtitle, volume 103 and issue 4. Other entries in the reference list support this volume correction. Its three other entries from this journal, for 2005, 2010 and 2011, have volumes 99, 104 and 105. Since the journal publishes one volume per year, this implies that a 2009 article is in volume 103.", "problem": "The article appeared in volume 103 of the \\emph{American Political Science Review}, in issue 4, and its subtitle is ``An experimental intervention using mass media to change norms and behavior in Rwanda''. The year and the pages are correct as printed.", "what_survives": "The article remains identifiable from its authors, year, main title and pages, and no statement about the study in the text depends on the subtitle or the volume."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E008", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of the reference list: the last line of the entry for Paluck (2009) and the three-line entry for Paluck (2010). Red boxes mark the word 'contacts', the hyphenated word 'perspective-taking', which is broken over two lines, and the word 'the' before 'Eastern Democratic Republic of Congo'.", "caption": "References, p. 179 (PDF p. 13), entry for Paluck (2010). The three marked items differ from the published title.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "The title in the entry for Paluck (2010) departs from the published title in three places: “contact” is singular, “perspective taking” has no hyphen, and no “the” precedes “Eastern”.", "id": "P024-E009", "location": "References, p. 179 (PDF p. 13), entry for Paluck (2010): the title", "pdf_page": 13, "printed_page": "179", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Paluck, E. L. (2010). Is it better not to talk? Group polarization, extended \\hl{contacts}, and \\hl{perspective-taking} in \\hl{the} Eastern Democratic Republic of Congo. \\emph{Personality and Social Psychology Bulletin}, \\emph{36}, 1170--1185.", "corrected": "Paluck, E. L. (2010). Is it better not to talk? Group polarization, extended \\hl{contact}, and \\hl{perspective taking} \\hl{in Eastern} Democratic Republic of Congo. \\emph{Personality and Social Psychology Bulletin}, \\emph{36}, 1170--1185.", "evidence": "The evidence is a check of the page against the record of SAGE for the article.", "problem": "The published title is ``Is it better not to talk? Group polarization, extended contact, and perspective taking in Eastern Democratic Republic of Congo''. The three differences do not prevent identification of the article.", "what_survives": "The year, volume and pages identify the article, and the description of the study on p.~169 does not quote the title."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E009", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the first paragraph of the Conclusion on page 176. A red box marks the word 'Capella' in the citation 'Kang, Capella, and Fishbein, 2009'.", "caption": "Conclusion, p. 176 (PDF p. 10), first paragraph. The marked surname in the citation of Kang et al. (2009) is printed with one “p”.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E010.png"}, {"alt": "Crop of the first eight lines of the reference list on page 179, with the entries for Kang, Imai and Lee. Red boxes mark the word 'Capella' in the entry beginning 'Kang, Y., Capella, J. N.' and in the entry beginning 'Lee, S., Capella, J. N.'.", "caption": "Reference list, p. 179 (PDF p. 13), first entries. The marked surname is printed with one “p” in the entries for Kang et al. (2009) and Lee et al. (2011).", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E010-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The surname Cappella is misspelled “Capella” in one citation and two reference entries.", "id": "P024-E010", "location": "Conclusion, p. 176 (PDF p. 10), first paragraph; reference list, p. 179 (PDF p. 13), entries Kang et al. (2009) and Lee et al. (2011)", "pdf_page": 10, "printed_page": "176", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Conclusion, p.~176: ``the effects of media campaigns designed to discourage the use of drugs (Fishbein et al., 2002; Kang, \\hl{Capella}, \\& Fishbein, 2009; Werb et al., 2011)''\n\nReference list, p.~179: ``Kang, Y., \\hl{Capella}, J. N., \\& Fishbein, M. (2009). The effect of marijuana scenes in anti-marijuana public service announcements on adolescents' evaluation of ad effectiveness. \\emph{Health Communication}, \\emph{24}, 483--493.''\n\nReference list, p.~179: ``Lee, S., \\hl{Capella}, J. N., Lerman, C., \\& Strasser, A. A. (2011). Smoking cues, argument strength, and perceived effectiveness of anti-smoking PSAs. \\emph{Nicotine and Tobacco Research}, \\emph{13}, 282--290.''", "corrected": "Conclusion, p.~176: ``(Fishbein et al., 2002; Kang, \\hl{Cappella}, \\& Fishbein, 2009; Werb et al., 2011)''\n\nReference list, p.~179: ``Kang, Y., \\hl{Cappella}, J. N., \\& Fishbein, M. (2009).'' The rest of the entry is unchanged.\n\nReference list, p.~179: ``Lee, S., \\hl{Cappella}, J. N., Lerman, C., \\& Strasser, A. A. (2011).'' The rest of the entry is unchanged.", "evidence": "The evidence is a check of the pages and of the author lists of the two cited articles. In PubMed, both articles list the second author as ``Joseph N Cappella'', and the publisher's author list for the article of Kang et al.\\ (doi:10.1080/10410230903104269) agrees.", "problem": "The second author of both cited articles is Joseph N. Cappella, whose surname has two p's.", "what_survives": "The titles, years, journals and page ranges identify the intended articles, and nothing else is affected. Only these three places were checked; the other names in the article were not."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E010", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the reference list: the last line of the entry for Sinclair, McConnell and Green (2012), the one-line entry for Sparks (2012), and the three-line entry for Werb and five coauthors (2011). A red box marks the surname 'Keer' of the fourth author in the first line of the last entry.", "caption": "References, p. 180 (PDF p. 14), entry for Werb et al. (2011). The marked surname is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E011.png"}], "formal_release": null, "formal_statements": [], "headline": "The fourth author of Werb et al. (2011) is Kerr, not Keer.", "id": "P024-E011", "location": "References, p. 180 (PDF p. 14), entry for Werb et al. (2011): the fourth author", "pdf_page": 14, "printed_page": "180", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Werb, D., Mills, E. J., DeBeck, K., \\hl{Keer}, T., Montaner, J. S. G., \\& Wood, E. (2011). The effectiveness of anti-illicit-drug public service announcements: A systematic review and meta-analysis. \\emph{Journal of Epidemiology and Community Health}, \\emph{65}, 834--840.", "corrected": "Werb, D., Mills, E. J., DeBeck, K., \\hl{Kerr}, T., Montaner, J. S. G., \\& Wood, E. (2011). The effectiveness of anti-illicit-drug public service announcements: A systematic review and meta-analysis. \\emph{Journal of Epidemiology and Community Health}, \\emph{65}, 834--840.", "evidence": "The evidence is a check of the page against the PubMed record of the article, which lists the fourth author as Thomas Kerr. The author list of the publisher, BMJ, agrees.", "problem": "The surname of the fourth author, Thomas Kerr, is misspelled.", "what_survives": "The article remains identifiable from its first author, title and journal. The text cites it on p.~176 as Werb et al.\\ and does not name the fourth author."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E011", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the three-line entry for Santa and Cochran (2008). A red box marks the word 'view' in the phrase 'message type and view characteristics' on the second line.", "caption": "References, p. 179 (PDF p. 13), last entry on the page. The marked word should be “viewer”.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E012.png"}], "formal_release": null, "formal_statements": [], "headline": "The title in the entry for Santa and Cochran (2008) should read “viewer characteristics”, not “view characteristics”.", "id": "P024-E012", "location": "References, p. 179 (PDF p. 13), entry for Santa and Cochran (2008): the title", "pdf_page": 13, "printed_page": "179", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Santa, A. F., \\& Cochran, B. N. (2008). Does the impact of anti-drinking and driving public service announcements differ based on message type and \\hl{view} characteristics? \\emph{Journal of Drug Education}, \\emph{38}, 109--129.", "corrected": "Santa, A. F., \\& Cochran, B. N. (2008). Does the impact of anti-drinking and driving public service announcements differ based on message type and \\hl{viewer} characteristics? \\emph{Journal of Drug Education}, \\emph{38}, 109--129.", "evidence": "The evidence is a check of the page against the PubMed record of the article. The publication record of the University of Montana agrees.", "problem": "The published title has ``viewer characteristics'', and the entry drops the last two letters of ``viewer''.", "what_survives": "The article remains identifiable, and the citation of it on p.~176 does not quote the title."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E012", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the reference list on page 177. A red box marks the words 'Media communication and research methods' at the start of the italic title in the entry 'Berger, A. (2011)'. The entry ends '(2nd ed.). Thousand Oaks, CA: Sage.'", "caption": "Reference list, p. 177 (PDF p. 11), entry for Berger (2011). In the marked title the conjunction stands after “communication”.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E013.png"}], "formal_release": null, "formal_statements": [], "headline": "The title of Berger (2011) is “Media and communication research methods”, not “Media communication and research methods”.", "id": "P024-E013", "location": "Reference list, p. 177 (PDF p. 11), entry Berger (2011)", "pdf_page": 11, "printed_page": "177", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Berger, A. (2011). \\emph{Media} \\hl{\\emph{communication and}} \\emph{research methods: An introduction to qualitative and quantitative approaches} (2nd ed.). Thousand Oaks, CA: Sage.''", "corrected": "``Berger, A. (2011). \\emph{Media} \\hl{\\emph{and communication}} \\emph{research methods: An introduction to qualitative and quantitative approaches} (2nd ed.). Thousand Oaks, CA: Sage.''", "evidence": "The evidence is a check of the page and of the library catalogue entry for the cited edition (WorldCat, OCLC 804747412). The title field of that entry reads ``Media and communication research methods : an introduction to qualitative and quantitative approaches'', and its edition field reads ``2nd ed''.", "problem": "The conjunction is misplaced. The title of the book is \\emph{Media and communication research methods: An introduction to qualitative and quantitative approaches}.", "what_survives": "The edition, the year and the publisher stand as printed, and nothing else is affected. The contents of the book were not examined."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E013", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three lines of the reference list: the last line of the entry for Rokeach (1971), the one-line entry for Rosenbaum (2002), and the first line of the entry for Santa and Cochran (2008). A red box marks the edition statement in parentheses after the title Observational studies; it reads 2d, then a backslash, then a full stop.", "caption": "References, p. 179 (PDF p. 13), entry for Rosenbaum (2002). The marked edition statement contains a backslash in place of “ed”.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E014.png"}], "formal_release": null, "formal_statements": [], "headline": "The edition statement in the entry for Rosenbaum (2002) is garbled and should read “(2nd ed.)”.", "id": "P024-E014", "location": "References, p. 179 (PDF p. 13), entry for Rosenbaum (2002): the edition statement", "pdf_page": 13, "printed_page": "179", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Rosenbaum, P. R. (2002). \\emph{Observational studies} \\hl{(2d \\textbackslash.)}. New York, NY: Springer.", "corrected": "Rosenbaum, P. R. (2002). \\emph{Observational studies} \\hl{(2nd ed.)}. New York, NY: Springer.", "evidence": "The evidence is a check of the page. The backslash is printed on the page and is also present in the text of the PDF.", "problem": "The parenthesis contains a backslash where the abbreviation ``ed'' belongs. The meaning, a second edition, is clear, and the same list has the usual form ``(2nd ed.)'' in the entry for Jensen (2011) on p.~178.", "what_survives": "Everything else is unaffected, since the entry still identifies the book."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E014", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of a paragraph on page 174. A red box marks the word 'gauges' in the line 'to deploy a series of different messages in this media market and gauges distinct outcomes'.", "caption": "Section “Single-Medium Multi-Outcome Crossover Design”, p. 174 (PDF p. 8), first paragraph. The marked verb is coordinated with “to deploy” and needs the base form.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E015.png"}], "formal_release": null, "formal_statements": [], "headline": "“and gauges” should read “and gauge”.", "id": "P024-E015", "location": "Section “Single-Medium Multi-Outcome Crossover Design”, p. 174 (PDF p. 8), first paragraph, sentence beginning “A stronger design would be”", "pdf_page": 8, "printed_page": "174", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``A stronger design would be to deploy a series of different messages in this media market and \\hl{gauges} distinct outcomes that measure responses to each message.''", "corrected": "``A stronger design would be to deploy a series of different messages in this media market and \\hl{gauge} distinct outcomes that measure responses to each message.''", "evidence": "The evidence is a check of the page.", "problem": "The infinitive introduced by ``to'' governs both coordinated verbs, ``deploy'' and ``gauge''. The second verb therefore needs the base form.", "what_survives": "The meaning of the sentence is clear, and nothing else is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E015", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the reference list on page 177. A red box marks 'Nickerson. D. W.' in the entry 'Arceneaux, K., and Nickerson. D. W. (2009). Who is mobilized to vote?'. The entries above and below it, for Arceneaux (2010) and for Aronow, Calfano and Green (2013), have a comma after each surname.", "caption": "Reference list, p. 177 (PDF p. 11). In the marked name a full stop stands where the list has a comma between surname and initials.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E016.png"}, {"alt": "Crop of five lines of the reference list on page 178. A red box marks 'Murdock G.' in the entry 'Deacon, D., Murdock G., Pickering, M., and Golding, P. (2007)'. The other names in the entry have a comma after the surname.", "caption": "Reference list, p. 178 (PDF p. 12). The marked name lacks the comma between surname and initial.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E016-b.png"}, {"alt": "Crop of seven lines of the reference list on page 178. The entries appear in the order Fisher, Franz, Fishbein, Gaines. A red box marks 'Fishbein, M.' at the start of the third entry.", "caption": "Reference list, p. 178 (PDF p. 12). The marked entry, Fishbein et al. (2002), is printed after Fisher (1935) and Franz et al. (2007).", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E016-c.png"}, {"alt": "Crop of the first eight lines of the reference list on page 179. The entries appear in the order Kang, Imai, Lee. A red box marks 'Imai, K.' at the start of the second entry.", "caption": "Reference list, top of p. 179 (PDF p. 13). The marked entry, Imai, King and Nall (2009), is printed after Kang et al. (2009); the entry for Jensen (2011) closes p. 178.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-E016-d.png"}], "formal_release": null, "formal_statements": [], "headline": "In the reference list, two author names have wrong punctuation (“Nickerson. D. W.” and “Murdock G.”), and two entries are out of alphabetical order: Fishbein et al. (2002) follows Fisher and Franz, and Imai, King and Nall (2009) follows Jensen and Kang.", "id": "P024-E016", "location": "Reference list, pp. 177–179 (PDF pp. 11–13): entries Arceneaux and Nickerson (2009), Deacon et al. (2007), Fishbein et al. (2002), and Imai, King and Nall (2009)", "pdf_page": 11, "printed_page": "177–179", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~177: ``Arceneaux, K., \\& \\hl{Nickerson.} D. W. (2009). Who is mobilized to vote? A re-analysis of 11 field experiments. \\emph{American Journal of Political Science}, \\emph{53}, 1--16.''\n\nP.~178: ``Deacon, D., \\hl{Murdock G.}, Pickering, M., \\& Golding, P. (2007). \\emph{Researching communications: A practical guide to methods in media and cultural analysis}. New York, NY: Bloomsbury USA.''\n\nP.~178, order of entries: Fisher, R. A. (1935); Franz, M. M., Freedman, P. B., Goldstein, K. M., \\& Ridout, T. N. (2007); \\hl{Fishbein}, M., Hall-Jamieson, K., Zimmer, E., von Haeften, I., \\& Nabi, R. (2002); Gaines, B. J., Kuklinski, J. H., \\& Quirk, P. J. (2007).\n\nPp.~178--179, order of entries: Jensen, K. B. (Ed.). (2011), the last entry of p.~178; Kang, Y., Capella, J. N., \\& Fishbein, M. (2009); \\hl{Imai}, K., King, G., \\& Nall, C. (2009); Lee, S., Capella, J. N., Lerman, C., \\& Strasser, A. A. (2011).", "corrected": "P.~177: ``Arceneaux, K., \\& \\hl{Nickerson,} D. W. (2009).'' The rest of the entry is unchanged.\n\nP.~178: ``Deacon, D., \\hl{Murdock, G.}, Pickering, M., \\& Golding, P. (2007).'' The rest of the entry is unchanged.\n\nOrder of entries: the entry for \\hl{Fishbein} et al.\\ (2002) is placed immediately before Fisher (1935), so that the order is Farquhar et al.\\ (1985), Fishbein et al.\\ (2002), Fisher (1935), Franz et al.\\ (2007), Gaines et al.\\ (2007). The entry for \\hl{Imai}, King and Nall (2009) is placed immediately before Jensen (2011), so that the order is Hudgens and Halloran (2008), Imai, King and Nall (2009), Jensen (2011), Kang et al.\\ (2009), Lee et al.\\ (2011).", "evidence": "The evidence is a check of the pages. In the printed order of the 64 entries of the list, entries 20 to 22 are Fisher, Franz and Fishbein, and entries 35 to 37 are Jensen, Kang and Imai.", "problem": "Elsewhere in the list a comma separates each surname from the initials, and the entries follow the alphabetical order of the first authors. ``Fishbein'' precedes ``Fisher'' and ``Franz'', and ``Imai'' precedes ``Jensen'' and ``Kang''.", "what_survives": "The four entries identify the intended works, and nothing else is affected. The other bibliographic details of these entries were not checked. The spelling ``Capella'' in two of the quoted entries is corrected in \\fid{P024-E010}."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-E016", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the paragraph that begins 'Some recent examples of field experiments'. A red box marks the citation 'Green and Vavreck (2007)' in the sentence 'Green and Vavreck (2007) randomly assigned cable TV markets to receive Rock The Vote ads designed to encourage young people to vote in the 2004 presidential election.'", "caption": "Introduction, p. 169 (PDF p. 3). The marked citation gives the year 2007.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N001.png"}, {"alt": "Crop of five lines of the reference list. A red box marks the authors and year, Green, D. P., and Vavreck, L. (2008), at the start of the entry 'Analysis of cluster-randomized experiments: A comparison of alternative estimation approaches. Political Analysis, 16, 138-152.' The entries above and below are Gerber and Green (2012) and Gueron (2002).", "caption": "Reference list, p. 178 (PDF p. 12). The marked entry is the only work by Green and Vavreck in the list, and it is dated 2008.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The citation “Green and Vavreck (2007)” on p. 169 should read “Green and Vavreck (2008)”, as on pp. 172 and 173 and in the reference list.", "id": "P024-N001", "location": "Introduction, paragraph beginning “Some recent examples of field experiments”, second sentence, p. 169 (PDF p. 3); reference list, p. 178 (PDF p. 12)", "pdf_page": 3, "printed_page": "169", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~169: ``Green and Vavreck (\\hl{2007}) randomly assigned cable TV markets to receive Rock The Vote ads designed to encourage young people to vote in the 2004 presidential election.''\n\np.~172: ``Green and Vavreck (2008) assigned 85 cable TV systems of varying sizes to receive Rock The Vote ads encouraging voter participation.''\n\nReference list, p.~178: ``Green, D. P., \\& Vavreck, L. (2008). Analysis of cluster-randomized experiments: A comparison of alternative estimation approaches. \\emph{Political Analysis}, \\emph{16}, 138--152.''", "corrected": "p.~169: ``Green and Vavreck (\\hl{2008}) randomly assigned cable TV markets to receive Rock The Vote ads \\ldots''", "evidence": "This is a check of the page. I read pp.~169, 172, 173 and 178, and I searched the article for every occurrence of ``Vavreck''. I did not consult a source outside the article, so an earlier report of the experiment dated 2007 is not excluded.", "problem": "The citation of 2007 has no entry in the reference list. The same Rock The Vote experiment is described under Green and Vavreck (2008) on p.~172 and cited as Green \\& Vavreck (2008) on p.~173, and the only entry for these authors is dated 2008.", "what_survives": "Everything else is unaffected. No statement depends on the year."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-N001", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines near the bottom of the page. Red boxes mark the words 'consider the contrast between an experiment in which two of four media markets, each comprising 1 million inhabitants, are assigned either to treatment or control.', which run over three lines. The next sentence begins 'If the four media markets tend to have different outcomes'.", "caption": "Section “Between-Subjects Clustered Design”, last paragraph of p. 171 (PDF p. 5). The marked sentence opens a contrast and names only one of the two experiments.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N002.png"}, {"alt": "Crop of the running head '172 Donald P. Green et al.' and the first four lines of the page. Red boxes mark the words 'On the other hand, a study with 4 million individually assigned subjects', which run over two lines; the sentence continues 'would have negligible sampling variability.'", "caption": "Top of p. 172 (PDF p. 6). The marked words supply the second half of the contrast, two sentences later.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The four-market illustration means an experiment that assigns four media markets, two to treatment and two to control, in contrast with a study that assigns 4 million subjects individually.", "id": "P024-N002", "location": "Section “Between-Subjects Clustered Design”, last paragraph of p. 171 (PDF p. 5), third sentence, continuing on p. 172 (PDF p. 6)", "pdf_page": 5, "printed_page": "171", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``To get an intuitive sense of this issue, consider \\hl{the contrast between} an experiment in which \\hl{two of four media markets}, each comprising 1 million inhabitants, \\hl{are assigned either to treatment or control}. If the four media markets tend to have different outcomes, this design will generate highly variable results because two high- or low-scoring media markets have a good chance of being assigned to the same condition. \\hl{On the other hand}, a study with 4 million individually assigned subjects would have negligible sampling variability.''", "corrected": "``To get an intuitive sense of this issue, consider an experiment in which \\hl{four media markets}, each comprising 1 million inhabitants, are assigned \\hl{two to treatment and two to control}. If the four media markets tend to have different outcomes, this design will generate highly variable results because two high- or low-scoring media markets have a good chance of being assigned to the same condition. \\hl{By contrast}, a study with 4 million individually assigned subjects would have negligible sampling variability.''", "intended_meaning": "The passage compares two designs for the same 4 million people. The first is an experiment that assigns four media markets of 1 million inhabitants each, two to treatment and two to control. The second is a study that assigns the 4 million subjects individually. The first sentence does not say this well. The word ``between'' introduces one experiment, and the other arrives two sentences later. The phrase ``two of four media markets \\ldots\\ are assigned either to treatment or control'' leaves the division into two treated and two control markets implicit. The meaning is recoverable. The first paragraph of the section defines the assignment of clusters to treatment or control. On p.~172 the article calls the illustration ``the four-market experiment'' and names the study with which it is compared. The point of the illustration is correct, and the warning about standard errors does not change."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-N002", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the first paragraph of the page. A red box marks the words 'hierarchical modeling', followed by the citation of Arceneaux and Nickerson, 2009, in parentheses. The list of options begins 'the estimation of robust cluster standard errors (Wooldridge, 2003), random effects estimation'.", "caption": "First paragraph of p. 172 (PDF p. 6). The marked citation is offered for hierarchical modeling.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N003.png"}, {"alt": "Crop of six lines of the reference list. Red boxes mark the two lines of the entry by Arceneaux and Nickerson, dated 2009: 'Who is mobilized to vote? A re-analysis of 11 field experiments. American Journal of Political Science, 53, 1-16.' The entry above it is Arceneaux, K. (2010), and the entry below it is Aronow, Calfano and Green (2013).", "caption": "Reference list, p. 177 (PDF p. 11). The marked entry is the only work by Arceneaux and Nickerson in the list.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The reference for hierarchical modeling should be Arceneaux and Nickerson (2009), “Modeling certainty with clustered data: A comparison of methods”, Political Analysis 17, 177–190; the listed article by the same authors fits no hierarchical model.", "id": "P024-N003", "location": "First paragraph of p. 172 (PDF p. 6), sentence beginning “Options include”; reference list, p. 177 (PDF p. 11)", "pdf_page": 6, "printed_page": "172", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~172: ``Options include the estimation of robust cluster standard errors (Wooldridge, 2003), random effects estimation (Green \\& Vavreck, 2008), hierarchical modeling (\\hl{Arceneaux \\& Nickerson, 2009}), or design-based estimates (Hansen \\& Bowers, 2009; Middleton \\& Aronow, 2011; Gerber \\& Green, 2012).''\n\nReference list, p.~177: ``Arceneaux, K., \\& Nickerson. D. W. (2009). \\hl{Who is mobilized to vote? A re-analysis of 11 field experiments.} \\emph{American Journal of Political Science}, \\emph{53}, 1--16.''", "corrected": "The citation on p.~172 stands, and the entry in the reference list is replaced. If the listed article is also kept, the two works of 2009 need distinguishing letters.\n\nReference list: ``Arceneaux, K., \\& Nickerson\\hl{,} D. W. (2009). \\hl{Modeling certainty with clustered data: A comparison of methods. \\emph{Political Analysis}, \\emph{17}, 177--190.}''", "evidence": "I checked the page, the listed article and the abstract of the second article. In the listed article, the Method section (p.~5) ``presents the probit model'', and note~13 (p.~8) says that ``bootstrapping occurs not only among individuals within an experiment but also between elections included in the analysis''. The abstract of the second article names the methods that it compares: ``clustered robust standard errors (SEs), random effects, hierarchical linear model, and aggregated ordinary least squares''.", "problem": "The citation resolves to the only entry for these authors, the article in the \\emph{American Journal of Political Science}. The listed article estimates probit and bivariate probit models with individual covariates and the salience of the election. It treats dependence by a bootstrap across individuals and elections, and it does not fit or illustrate a hierarchical model. The same authors published a second article in 2009, in \\emph{Political Analysis}, which compares hierarchical linear models with other methods for clustered data. The second article supports the statement and is not in the reference list. That it is the one intended is an inference from its subject, its authors and its year. The full stop after ``Nickerson'' in the entry is as printed (see \\fid{P024-E016}).", "what_survives": "The list of ways to allow for clustering stands, hierarchical modeling included, and so does the warning that these estimates are unreliable with few clusters. Only the reference changes. The citation occurs once in the article."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-N003", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading 'References' and the first three lines of the list. A red box marks the author and year that open the first entry, 'Abrajano, M. (2010).'; the title of the entry is Campaigning to the new American electorate: Advertising to Latino voters.", "caption": "References, p. 177 (PDF p. 11), first entry of the list. The marked entry, Abrajano (2010), is not cited in the article.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N007.png"}, {"alt": "Crop of five lines of the reference list: the entry for Deacon, Murdock, Pickering and Golding (2007) and the entry for Dong and Maynard (2013). A red box marks the authors and year that open the second entry, Dong, N., and Maynard, R. (2013); the title of the entry begins PowerUp!: A tool for calculating minimum detectable effect sizes.", "caption": "References, p. 178 (PDF p. 12). The marked entry, Dong and Maynard (2013), is not cited in the article.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N007-b.png"}, {"alt": "Crop of three lines of the reference list: the entry for Fisher (1935) and the two-line entry for Franz et al. A red box marks the authors and year that open the second entry, Franz, M. M., Freedman, P. B., Goldstein, K. M., and Ridout, T. N. (2007); the title of the entry is Campaign advertising and American democracy.", "caption": "References, p. 178 (PDF p. 12). The marked entry, Franz, Freedman, Goldstein and Ridout (2007), is not cited in the article.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N007-c.png"}, {"alt": "Crop of three lines of the reference list: the last line of the entry for Hansen and Bowers (2009) and the two-line entry for Hillygus and Shields. A red box marks the authors and year that open the entry, Hillygus, D. S., and Shields, T. G. (2009); the title of the entry is The persuadable voter: Wedge issues in presidential campaigns.", "caption": "References, p. 178 (PDF p. 12). The marked entry, Hillygus and Shields (2009), is not cited in the article.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N007-d.png"}, {"alt": "Crop of three lines of the reference list: the last line of the entry for Mutz and Reeves (2005) and the two-line entry for Nickerson. A red box marks the author and year that open the entry, 'Nickerson, D. (2005).'; the title of the entry is Scalable protocols offer efficient design for field experiments.", "caption": "References, p. 179 (PDF p. 13). The marked entry, Nickerson (2005), is not cited in the article.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N007-e.png"}, {"alt": "Crop of three lines of the reference list: the last line of the entry for the Pew Center for Civic Journalism (1999) and the two-line entry for Prior. A red box marks the author and year that open the entry, 'Prior, M. (2009).'; the title of the entry is Improving media effects research through better measurement of news exposure.", "caption": "References, p. 179 (PDF p. 13). The marked entry, Prior (2009), is not cited in the article.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N007-f.png"}], "formal_release": null, "formal_statements": [], "headline": "Six of the 64 entries in the reference list are not cited anywhere in the abstract, the text or the notes.", "id": "P024-N007", "location": "References, pp. 177–179 (PDF pp. 11–13): the entries for Abrajano (2010), Dong and Maynard (2013), Franz et al. (2007), Hillygus and Shields (2009), Nickerson (2005) and Prior (2009)", "pdf_page": 11, "printed_page": "177–179", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item p.~177: \\hl{Abrajano, M. (2010).} \\emph{Campaigning to the new American electorate: Advertising to Latino voters}. Palo Alto, CA: Stanford University Press.\n\\item p.~178: \\hl{Dong, N., \\& Maynard, R. (2013).} \\emph{PowerUp!}: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. \\emph{Journal of Research on Educational Effectiveness}, \\emph{6}, 24--67.\n\\item p.~178: \\hl{Franz, M. M., Freedman, P. B., Goldstein, K. M., \\& Ridout, T. N. (2007).} \\emph{Campaign advertising and American democracy}. Philadelphia, PA: Temple University Press.\n\\item p.~178: \\hl{Hillygus, D. S., \\& Shields, T. G. (2009).} \\emph{The persuadable voter: Wedge issues in presidential campaigns}. Princeton, NJ: Princeton University Press.\n\\item p.~179: \\hl{Nickerson, D. (2005).} Scalable protocols offer efficient design for field experiments. \\emph{Political Analysis}, \\emph{13}, 233--252.\n\\item p.~179: \\hl{Prior, M. (2009).} Improving media effects research through better measurement of news exposure. \\emph{Journal of Politics}, \\emph{71}, 893--908.\n\\end{itemize}", "corrected": "The six entries are \\hl{removed} from the reference list. An entry that was meant to be cited is instead kept, and a citation to it is added at the passage that it supports.", "evidence": "The evidence is a check of the pages. Searching the abstract, the text and Notes 1--5 (pp.~168--177) for the surname of each first author, I found no occurrence of Abrajano, Dong, Franz, Hillygus or Prior as a name, and only the occurrence of Nickerson quoted above. The same pages do not mention any of the six works by its title alone.", "problem": "The reference list contains six works that the article never cites. The only occurrence of the name Nickerson before the reference list is the citation ``Arceneaux \\& Nickerson, 2009'' on p.~172, which belongs to a different entry. The article does not show whether the six entries remain from an earlier draft or were meant as background reading.", "what_survives": "Everything else is unaffected, since no statement of the article depends on these entries."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-N007", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three lines of the reference list: the last line of the entry for Horvitz and Thompson (1952) and the two-line entry for Hovland, Lumsdaine and Sheffield (1949). Red boxes mark the words 'in' and 'communications' in the italic title, which is printed as Experiments in mass communications.", "caption": "References, p. 178 (PDF p. 12), entry for Hovland, Lumsdaine and Sheffield (1949). The two marked words of the title differ from the published title.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N008.png"}, {"alt": "Crop of three lines of the reference list: the word 'University.' that ends the preceding entry and the two-line entry for Panagopoulos and Green (2008). A red box marks 'Field experiments:' at the start of the title, which continues Testing the impact of radio advertisements on electoral competition.", "caption": "References, p. 179 (PDF p. 13), entry for Panagopoulos and Green (2008). The marked words end in a colon that the published title does not have.", "url": "https://pmaronow.github.io/llm-audit/works/P024/exhibits/P024-N008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Two reference titles depart from the published ones: the 1949 book is titled Experiments on mass communication, and the title of Panagopoulos and Green (2008) has no colon after “Field experiments”.", "id": "P024-N008", "location": "References, p. 178 (PDF p. 12), entry for Hovland, Lumsdaine and Sheffield (1949), and p. 179 (PDF p. 13), entry for Panagopoulos and Green (2008): the titles", "pdf_page": 12, "printed_page": "178", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item p.~178: Hovland, C., Lumsdaine, A. A., \\& Sheffield, F. (1949). \\emph{Experiments \\hl{in} mass \\hl{communications}}. Princeton, NJ: Princeton University Press.\n\\item p.~179: Panagopoulos, C., \\& Green, D. P. (2008). Field experiments\\hl{:} Testing the impact of radio advertisements on electoral competition. \\emph{American Journal of Political Science}, \\emph{52}, 156--168.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Hovland, C., Lumsdaine, A. A., \\& Sheffield, F. (1949). \\emph{Experiments \\hl{on} mass \\hl{communication}}. Princeton, NJ: Princeton University Press.\n\\item Panagopoulos, C., \\& Green, D. P. (2008). Field \\hl{experiments testing} the impact of radio advertisements on electoral competition. \\emph{American Journal of Political Science}, \\emph{52}, 156--168.\n\\end{itemize}", "evidence": "The evidence is a check of the pages against two records. The first is the catalogue record of the University of Canterbury library for the edition of the book published by Princeton University Press in 1949, and the second is the record of Wiley for the article.", "problem": "The book of Hovland, Lumsdaine and Sheffield is titled \\emph{Experiments on mass communication}, with ``on'' and the singular. The published title of Panagopoulos and Green (2008) is ``Field experiments testing the impact of radio advertisements on electoral competition'', and the colon in the entry changes only the punctuation.", "what_survives": "Both works remain identifiable, and the citations of them in the text do not quote their titles."}, "url": "https://pmaronow.github.io/llm-audit/works/P024/#P024-N008", "work": {"arxiv_id": null, "doi": "10.1080/10584609.2013.828142", "title": "Field Experimental Designs for the Study of Media Effects", "url": "https://pmaronow.github.io/llm-audit/works/P024/"}, "work_id": "P024"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of the second paragraph of Section 4, from the interval estimates for each population to the foot of the page. Red outlines surround the words overlap only when gamma is greater than 3 in the fourth line and overlap when gamma is greater than 1.25 in the fifth line.", "caption": "Section 4, second paragraph, p. 238 (PDF p. 4). The outlines mark the clauses “overlap only when $\\gamma>3$” and “overlap when $\\gamma>1{\\cdot}25$”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-A001.png"}], "formal_release": null, "formal_statements": [], "headline": "The values $3$ and $1.25$ are rows of Table 1; the two pairs of intervals begin to overlap at about $\\gamma=3.30$ and $\\gamma=1.43$.", "id": "P025-A001", "location": "Section 4, second paragraph, fourth sentence, p. 238 (PDF p. 4)", "pdf_page": 4, "printed_page": "238", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The estimates for the shooting gallery and needle exchange populations \\hl{overlap only when $\\gamma > 3$}; the estimates for the street needle and needle exchange populations \\hl{overlap when $\\gamma > 1{\\cdot}25$}.", "corrected": "\\hl{Among the values of $\\gamma$ shown in Table 1,} the estimates for the shooting gallery and needle exchange populations overlap only when $\\gamma > 3$; the estimates for the street needle and needle exchange populations overlap when $\\gamma > 1{\\cdot}25$. \\hl{For continuous $\\gamma$, the intervals overlap when $\\gamma$ is at least about $3{\\cdot}30$ and $1{\\cdot}43$, respectively.}", "intended_meaning": "I read the sentence as a summary of Table 1, whose rows are $\\gamma=1$, $1.25$, $1.5$, $2$, $2.5$, $3$ and $4$. On these rows both clauses are correct. The shooting gallery and needle exchange intervals are disjoint at $\\gamma=3$ and overlap at $\\gamma=4$, and the street and needle exchange intervals are disjoint at $\\gamma=1.25$ and overlap at $\\gamma=1.5$. For a continuous $\\gamma$, the first clause is a correct necessary condition, and the second clause would fail between $1.25$ and about $1.43$. At $\\gamma=1.3$, the lower bound for the street sample is $270/439$ and the upper bound for the needle exchange sample is $6539/11509$, which is smaller. For sample proportions $p_A>p_B$, the two intervals overlap exactly when $\\gamma^2\\geqslant p_A(1-p_B)/\\{p_B(1-p_A)\\}$. With the printed proportions $0.917$, $0.675$ and $0.503$, the thresholds are $\\sqrt{455749/41749}\\approx 3.304$ and $\\sqrt{13419/6539}\\approx 1.433$. The entries of the table are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-A001", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the second half of Proposition 1, from An optimal solution to (3) to the displayed formula for the lower bound. A red outline surrounds the greater-than-or-equal sign that stands before y sub k plus 1 in the third line.", "caption": "Proposition 1, p. 236 (PDF p. 2). The outline marks the inequality sign in the rule that defines $k^{\\mathrm L}$.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E001.png"}, {"alt": "Crop of the line and k minus is the smallest nonnegative integer k satisfying, followed by a displayed ratio of sums. A red outline surrounds the greater-than-or-equal sign between the ratio and y sub (k plus 1).", "caption": "Proposition 2, p. 238 (PDF p. 4), directly below Table 1. The outline marks the inequality sign in the rule that defines $k_-$.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P025-T1", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T1"}, {"id": "P025-T2", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T2"}, {"id": "P025-T3", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T3"}, {"id": "P025-T4", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T4"}, {"id": "P025-T7", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T7"}, {"id": "P025-T9", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T9"}, {"id": "P025-T19", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T19"}], "headline": "In the rules that define the lower cut points $k^{\\mathrm L}$ and $k_-$, the inequality $\\geqslant$ should read $\\leqslant$.", "id": "P025-E001", "location": "Proposition 1, the rule that defines $k^{\\mathrm L}$, p. 236 (PDF p. 2); Proposition 2, the rule that defines $k_-$, p. 238 (PDF p. 4)", "pdf_page": 2, "printed_page": "236", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P025-T1", "P025-T19", "P025-T2", "P025-T3", "P025-T4", "P025-T7", "P025-T9"], "text": {"as_printed": "\\emph{Proposition 1, p.~236.} An optimal solution to (3) is $w^{\\mathrm L} = (1/\\alpha, \\ldots, 1/\\alpha, 1/\\beta, \\ldots, 1/\\beta)$, where the first $k^{\\mathrm L}$ entries equal $1/\\alpha$ and the remaining $n - k^{\\mathrm L}$ entries equal $1/\\beta$, with $k^{\\mathrm L}$ being the smallest nonnegative integer $k$ that satisfies $\\{\\gamma \\sum_{i=1}^{k} y_i + \\sum_{i=k+1}^{n} y_i\\}/\\{\\gamma k + (n - k)\\} \\hl{\\geqslant} y_{k+1}$. The lower bound on $\\hat\\mu$ is therefore\n\\[\n\\hat\\mu^{\\mathrm L} = \\frac{\\gamma \\sum_{i=1}^{k^{\\mathrm L}} y_i + \\sum_{i=k^{\\mathrm L}+1}^{n} y_i}{\\gamma k^{\\mathrm L} + (n - k^{\\mathrm L})}.\n\\]\n\n\\emph{Proposition 2, p.~238.} and $k_-$ is the smallest nonnegative integer $k$ satisfying\n\\[\n\\frac{\\gamma \\sum_{j=1}^{k} \\Pi_j y_{(j)} + \\sum_{j=k+1}^{|Y|} \\Pi_j y_{(j)}}{\\gamma \\sum_{j=1}^{k} \\Pi_j + \\sum_{j=k+1}^{|Y|} \\Pi_j} \\hl{\\geqslant} y_{(k+1)}.\n\\]", "corrected": "\\emph{Proposition 1, p.~236.} Let $\\gamma=\\beta/\\alpha\\geqslant 1$ and $y_1\\leqslant\\cdots\\leqslant y_n$. An optimal solution to (3) is $w^{\\mathrm L} = (1/\\alpha, \\ldots, 1/\\alpha, 1/\\beta, \\ldots, 1/\\beta)$, where the first $k^{\\mathrm L}$ entries equal $1/\\alpha$ and the remaining $n - k^{\\mathrm L}$ entries equal $1/\\beta$, with $k^{\\mathrm L}$ being the smallest \\hl{integer $k$ with $0\\leqslant k<n$} that satisfies $\\{\\gamma \\sum_{i=1}^{k} y_i + \\sum_{i=k+1}^{n} y_i\\}/\\{\\gamma k + (n - k)\\} \\hl{\\leqslant} y_{k+1}$. The lower bound on $\\hat\\mu$ is therefore\n\\[\n\\hat\\mu^{\\mathrm L} = \\frac{\\gamma \\sum_{i=1}^{k^{\\mathrm L}} y_i + \\sum_{i=k^{\\mathrm L}+1}^{n} y_i}{\\gamma k^{\\mathrm L} + (n - k^{\\mathrm L})}.\n\\]\n\n\\emph{Proposition 2, p.~238.} Let $y_{(1)}<\\cdots<y_{(|Y|)}$ and $\\Pi_j>0$ be as in Section 3.2. Then $k_-$ is the smallest \\hl{integer $k$ with $0\\leqslant k<|Y|$} satisfying\n\\[\n\\frac{\\gamma \\sum_{j=1}^{k} \\Pi_j y_{(j)} + \\sum_{j=k+1}^{|Y|} \\Pi_j y_{(j)}}{\\gamma \\sum_{j=1}^{k} \\Pi_j + \\sum_{j=k+1}^{|Y|} \\Pi_j} \\hl{\\leqslant} y_{(k+1)}.\n\\]\nIn each rule the last index, $k=n-1$ or $k=|Y|-1$, satisfies the inequality, and the cut point exists.", "evidence": "The evidence is an exact counterexample for each proposition and a derivation of the corrected rule.\n\n\\emph{Counterexample.} Let $y=(0,1)$, $\\alpha=1/2$ and $\\beta=1$, so that $\\gamma=2$. I write $(u,v)=(\\beta w_1,\\beta w_2)\\in[1,2]^2$ for the scaled weights. The ratio in (3) is $v/(u+v)$. The printed rule accepts $k=0$, since $1/2\\geqslant 0$, and returns $w^{\\mathrm L}=(1,1)$ and $\\hat\\mu^{\\mathrm L}=1/2$. For all $(u,v)\\in[1,2]^2$, I have\n\\[\n\\frac{v}{u+v}-\\frac13=\\frac{(2-u)+2(v-1)}{3(u+v)}\\geqslant 0,\n\\]\nwith equality at $(u,v)=(2,1)$. The minimum of (3) is $1/3$, and the printed lower bound exceeds it by $1/6$. For Proposition 2, I consider a population of two units with outcomes $(0,1)$ and selection probabilities $(1/2,1)$. Then $(\\Pi_1,\\Pi_2)=(1/2,1)$, the ratio in (7) is $v/(u/2+v)$, and\n\\[\n\\frac{v}{u/2+v}-\\frac12=\\frac{(2-u)+2(v-1)}{2(u+2v)}\\geqslant 0 .\n\\]\nThe minimum of (7) is $1/2$, which is the population mean, and the ratio at the printed $W_-=(1,1)$ is $2/3$. The table lists the values at the four vertices.\n\n\\begin{audittable}{lcc}{Value of the ratio at the vertices of the box of scaled weights $[1,2]^2$, for outcomes $(0,1)$, $\\alpha=1/2$ and $\\beta=1$.}\nScaled weights $(u,v)$ & Proposition 1, ratio in (3) & Proposition 2 with $\\Pi=(1/2,1)$, ratio in (7) \\\\\n$(1,1)$, printed rule, $k=0$ & $1/2$ & $2/3$ \\\\\n$(1,2)$ & $2/3$ & $4/5$ \\\\\n$(2,1)$, corrected rule, $k=1$ & $1/3$ & $1/2$ \\\\\n$(2,2)$ & $1/2$ & $2/3$ \\\\\n\\end{audittable}\n\n\\emph{Derivation of the corrected rule.} Both propositions are instances of one problem. Let $x_1\\leqslant\\cdots\\leqslant x_m$, let $a_j>0$, and let $\\gamma\\geqslant 1$. Proposition 1 has $m=n$, $x_j=y_j$ and $a_j=1$, and Proposition 2 has $m=|Y|$, $x_j=y_{(j)}$ and $a_j=\\Pi_j$. After the weights are multiplied by $\\beta$, the lower problem is to minimize\n\\[\nR(q)=\\frac{\\sum_{j=1}^{m}a_jq_jx_j}{\\sum_{j=1}^{m}a_jq_j}\\qquad\\text{over } 1\\leqslant q_j\\leqslant\\gamma .\n\\]\nFor $0\\leqslant k\\leqslant m$, I define\n\\[\nD_k=\\gamma\\sum_{j=1}^{k}a_j+\\sum_{j=k+1}^{m}a_j,\\qquad\nB_k=\\gamma\\sum_{j=1}^{k}a_jx_j+\\sum_{j=k+1}^{m}a_jx_j,\\qquad F_k=\\frac{B_k}{D_k},\n\\]\nso that $F_k$ is the left side of the inequality in each rule. By these definitions, I have\n\\[\nF_{k+1}-F_k=\\frac{(\\gamma-1)\\,a_{k+1}\\,(x_{k+1}-F_k)}{D_{k+1}}\\qquad\\text{for } 0\\leqslant k<m .\n\\]\nFor $\\gamma>1$, the value $F_k$ decreases at the step $k$ exactly when $F_k>x_{k+1}$. Let $k_*$ be the smallest $k\\in\\{0,\\ldots,m-1\\}$ such that $F_k\\leqslant x_{k+1}$. Such a $k$ exists, since $F_{m-1}$ is a weighted average of outcomes that are at most $x_m$. I set $\\lambda=F_{k_*}$, and $q^*_j=\\gamma$ for $j\\leqslant k_*$ and $q^*_j=1$ for $j>k_*$. If $k_*>0$, then $F_{k_*-1}>x_{k_*}$ because $k_*$ is the smallest such index, and\n\\[\nF_{k_*}-x_{k_*}=\\frac{D_{k_*-1}\\,(F_{k_*-1}-x_{k_*})}{D_{k_*}}>0 .\n\\]\nIt follows that $x_j<\\lambda$ for all $j\\leqslant k_*$, and $x_j\\geqslant x_{k_*+1}\\geqslant\\lambda$ for all $j>k_*$. Let $q$ be any feasible vector. For $j\\leqslant k_*$, I have $q_j-q^*_j\\leqslant 0$ and $x_j-\\lambda<0$, and for $j>k_*$, I have $q_j-q^*_j\\geqslant 0$ and $x_j-\\lambda\\geqslant 0$. Since $\\sum_ja_jq^*_j(x_j-\\lambda)=0$, I obtain\n\\[\nR(q)-\\lambda=\\frac{\\sum_{j}a_j(q_j-q^*_j)(x_j-\\lambda)}{\\sum_{j}a_jq_j}\\geqslant 0 .\n\\]\nThe vector $q^*$ minimizes $R$ over the whole box, including when outcomes are tied and when $\\gamma=1$.", "problem": "The outcomes are in increasing order. At $k=0$ the left side of the printed inequality is the sample mean in Proposition 1, and the $\\Pi$-weighted mean of the distinct outcomes in Proposition 2. Since neither is smaller than the smallest outcome, $k=0$ satisfies the printed inequality in every case, and both printed rules return the cut point $0$. The vector $w^{\\mathrm L}$, or $W_-$, then has every entry equal to $1/\\beta$. Evaluating these vectors in (3) and (7) gives the unweighted sample mean and the $\\Pi$-weighted mean, respectively. When $\\gamma>1$ and the outcomes are not all equal, this vector does not minimize (3) or (7), since a larger weight on the smallest outcome lowers the ratio. The left endpoint of (5), computed with the printed $k^{\\mathrm L}$, then leaves out values that $\\hat\\mu$ can take. The paper does not display the correct rule elsewhere. The Appendix says that the solution to (3) ``can be derived in a similar manner'', and the supplementary proof of Proposition 2 treats $\\mu_+$ only. When $\\gamma=1$ or all outcomes are equal, the ratio has the same value at every feasible weight vector, and the printed rules return a minimizer.", "what_survives": "The correction needs no new assumption. The definitions (2), (3), (6) and (7), the rules for $k^{\\mathrm H}$ and $k_+$, the upper bounds, and the closed form for dichotomous outcomes are unaffected by this sign correction. All 42 entries of Table 1 are reproduced from the reported percentages by the closed form, and this sign correction does not change the table. The estimated bounds still converge to the limits defined by (6) and (7). The rules for $k^{\\mathrm L}$ and $k_-$ change, and with them the left endpoint of (5) and computations of the lower bound that use either printed rule. The summation limits in the displayed formula for $\\mu_-$ are corrected separately in \\fid{P025-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-E001", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first line of Proposition 2 and the two displayed formulas for mu plus and mu minus. Red outlines surround the two lower summation limits j equals k minus, minus 1, in the numerator and in the denominator of the formula for mu minus, on the right.", "caption": "Proposition 2, p. 237 (PDF p. 3). The outlines mark the lower limits $j=k_--1$ of the second sums in the numerator and the denominator of the formula for $\\mu_-$.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P025-T2", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T2"}, {"id": "P025-T4", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T4"}, {"id": "P025-T9", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T9"}, {"id": "P025-T10", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T10"}], "headline": "In the displayed formula for $\\mu_-$, both tail sums should start at $j=k_-+1$, not at $j=k_--1$.", "id": "P025-E002", "location": "Proposition 2, displayed formula for $\\mu_-$, p. 237 (PDF p. 3)", "pdf_page": 3, "printed_page": "237", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P025-T10", "P025-T2", "P025-T4", "P025-T9"], "text": {"as_printed": "\\[\n\\mu_- = \\frac{\\gamma \\sum_{j=1}^{k_-} \\Pi_j y_{(j)} + \\sum_{j=\\hl{\\scriptstyle k_- -1}}^{|Y|} \\Pi_j y_{(j)}}{\\gamma \\sum_{j=1}^{k_-} \\Pi_j + \\sum_{j=\\hl{\\scriptstyle k_- -1}}^{|Y|} \\Pi_j},\n\\]", "corrected": "With $k_-$ and $W_-$ as in Proposition 2, the value of (7) at $W_-$ is\n\\[\n\\mu_- = \\frac{\\gamma \\sum_{j=1}^{k_-} \\Pi_j y_{(j)} + \\sum_{j=\\hl{\\scriptstyle k_- +1}}^{|Y|} \\Pi_j y_{(j)}}{\\gamma \\sum_{j=1}^{k_-} \\Pi_j + \\sum_{j=\\hl{\\scriptstyle k_- +1}}^{|Y|} \\Pi_j},\n\\]\nwhere a sum over an empty range is zero.", "evidence": "The evidence is a substitution and an exact example. For $0\\leqslant k\\leqslant|Y|$, let $W^{(k)}$ have its first $k$ entries equal to $1/\\alpha$ and the others equal to $1/\\beta$. Substituting $W^{(k)}$ into the ratio in (7) and multiplying the numerator and the denominator by $\\beta$, I obtain\n\\[\n\\frac{\\gamma\\sum_{j=1}^{k}\\Pi_jy_{(j)}+\\sum_{j=k+1}^{|Y|}\\Pi_jy_{(j)}}{\\gamma\\sum_{j=1}^{k}\\Pi_j+\\sum_{j=k+1}^{|Y|}\\Pi_j},\n\\]\nwhich is the corrected display at $k=k_-$. For the example, let $\\alpha=1/2$, $\\beta=1$, and let the population have outcomes $(0,0,1,1,2,3)$ and selection probabilities $(1/2,1/2,1/2,1/2,1,1)$. The distinct outcomes are $(0,1,2,3)$ and $\\Pi=(1,1,1,1)$. With the weights scaled by $\\beta$, the ratio in (7) is $F(W)=(W_2+2W_3+3W_4)/(W_1+W_2+W_3+W_4)$ on $[1,2]^4$, and\n\\[\nF(W)-\\frac76=\\frac{7(2-W_1)+(2-W_2)+5(W_3-1)+11(W_4-1)}{6(W_1+W_2+W_3+W_4)}\\geqslant 0 ,\n\\]\nwith equality only at $W=(2,2,1,1)$. The minimum of (7) is $7/6$, and it is attained at the cut point $k_-=2$. At $k_-=2$ the printed display equals $\\{2(0+1)+(0+1+2+3)\\}/\\{2(1+1)+(1+1+1+1)\\}=1$, and the corrected display equals $\\{2(0+1)+(2+3)\\}/\\{2(1+1)+(1+1)\\}=7/6$.", "problem": "Proposition 2 states that the first $k_-$ entries of $W_-$ equal $1/\\alpha$ and the remaining $|Y|-k_-$ entries equal $1/\\beta$. The value of (7) at $W_-$ therefore has the classes $j\\leqslant k_-$ with the factor $\\gamma$ and the classes $j\\geqslant k_-+1$ without it. As printed, the second sums start at $k_--1$. For $k_-\\geqslant 2$, the classes $k_--1$ and $k_-$ are counted in both sums, and for $k_-=0$ or $k_-=1$, the lower limit is an index with no class. The displays for $\\mu_+$, for the cut points and in Assumption 2 start their second sums at $k+1$.", "what_survives": "The definitions (6) and (7), the description of $W_-$, the bounds of Section 3.1 and Table 1 are unaffected. The supplementary material works with the full weighted sums and does not use the printed limits. The rule that selects $k_-$ is corrected in \\fid{P025-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-E002", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last three lines of page 238, which contain the sentence on when the intervals overlap. A red outline surrounds the words This means that for the shooting gallery needles to possibly show the same, which begin the last sentence of Section 4 and run to the end of the page.", "caption": "Section 4, start of the last sentence, foot of p. 238 (PDF p. 4). The outline marks the words “This means that for the shooting gallery needles to possibly show the same”, which follow the two statements about overlap.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E003.png"}, {"alt": "Crop of the first three lines of page 239, which complete the last sentence of Section 4. Red outlines surround the words must have been more than at the end of the first line and three times as likely at the start of the second, and the words with the converse holding for at the end of the second line and exchange needles at the start of the third.", "caption": "Section 4, end of the last sentence, p. 239 (PDF p. 5). The outlines mark the words “must have been more than three times as likely” and “with the converse holding for exchange needles”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E003-b.png"}, {"alt": "A plot with the shooting gallery ratio, infected against uninfected, on the horizontal axis and the needle exchange ratio, uninfected against infected, on the vertical axis, both from 1 to 6. A blue curve of constant product 1056/97, about 10.9, falls from the upper left to the lower right. A shaded square with a dashed orange border covers the region where both ratios exceed 3 and is labelled as printed: both ratios must exceed 3. The curve enters the shaded region through its left edge, passes through the open point near (3.30, 3.30), labelled the smallest common cap, and exits through the bottom edge. Two filled points on the curve lie outside the region: design B at gallery ratio 11/4 = 2.75 with both prevalences 4/5, and design A at exchange ratio 288/97, about 2.97, with both prevalences 3/4. The title reads Equal reweighted prevalence fixes the product of the two selection ratios; neither ratio need exceed three.", "caption": "The pairs of selection ratios for which the shooting gallery and needle exchange samples have equal reweighted prevalence, computed from the count-derived proportions $44/48$ and $291/579$. The curve is the set of pairs whose product is $1056/97$. The shaded region is what the printed sentence asserts. The filled points are designs A and B of the table, and the open point is the pair at which both ratios equal $\\sqrt{1056/97}\\approx 3.30$.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E003-selection-ratios.png"}], "formal_release": null, "formal_statements": [{"id": "P025-T5", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T5"}, {"id": "P025-T6", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T6"}], "headline": "For the shooting gallery and needle exchange intervals to overlap, the common bound $\\gamma$ must be at least about $3.30$; neither individual selection ratio needs to exceed three.", "id": "P025-E003", "location": "Section 4, last sentence, pp. 238–239 (PDF pp. 4–5)", "pdf_page": 4, "printed_page": "238–239", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P025-T5", "P025-T6"], "text": {"as_printed": "This means that for the shooting gallery needles to possibly show the same HIV prevalence rate as the exchange needles, infected shooting gallery needles \\hl{must have been more than three times as likely} to be tested than uninfected shooting gallery needles, \\hl{with the converse holding for exchange needles}.", "corrected": "\\hl{For the shooting gallery and needle exchange interval estimates to overlap, the common bound on the ratios of selection probabilities must be at least about $3{\\cdot}30$. This does not require both populations' actual directional selection ratios to exceed three. With the printed proportions $0{\\cdot}917$ and $0{\\cdot}503$, equal reweighted estimates require that the product of the two effective ratios, for infected against uninfected shooting gallery needles and for uninfected against infected exchange needles, equal $455749/41749$, which is about $10{\\cdot}9$.}", "evidence": "The evidence is a derivation and two exact designs.\n\n\\emph{Derivation.} Let a sample have an infected fraction $p$, and write $\\pi_1$ and $\\pi_0$ for the effective probabilities obtained by averaging inverse selection probabilities within each sampled infection group and taking reciprocals. By (1), the Hajek estimate $q$ of the prevalence satisfies $q/(1-q)=\\{p/(1-p)\\}(\\pi_0/\\pi_1)$. I write $r_G=\\pi_{G,1}/\\pi_{G,0}$ for the shooting gallery and $r_E=\\pi_{E,0}/\\pi_{E,1}$ for the needle exchange. The two estimates are equal exactly when\n\\[\nr_G\\,r_E=\\frac{p_G(1-p_E)}{(1-p_G)\\,p_E}.\n\\]\nWith the printed proportions $0.917$ and $0.503$, the right side is $455749/41749\\approx 10.92$, and with the count-derived proportions $44/48$ and $291/579$, it is $1056/97\\approx 10.89$. By the closed form on p.~237, the two intervals overlap exactly when $\\gamma^2$ is at least this number. The condition is $\\gamma\\geqslant\\sqrt{455749/41749}\\approx 3.304$ with the printed proportions and $\\gamma\\geqslant\\sqrt{1056/97}\\approx 3.299$ with the count-derived proportions. At the smallest such $\\gamma$ both ratios equal $\\gamma$, and at a larger bound, such as the row $\\gamma=4$ of Table 1, they can differ.\n\n\\emph{Two designs.} In each design below, a fixed number of needles is drawn uniformly from the infected and from the uninfected needles of each population. The samples have the sizes $48$ and $579$ reported in Section 4 and positive counts $44$ and $291$ implied by rounding its percentages to the nearest tenth of a percent. The two populations have the same prevalence, the two Hajek estimates equal this prevalence on every draw, and all selection probabilities lie in $[1/4,1]$, so that $\\gamma=4$. In design A the ratio for the needle exchange is below three, and in design B the ratio for the shooting gallery is below three. The conclusion is the same for two further choices of the selection probabilities, with the proportions $0.917$ and $0.503$ taken as exact. There the ratios $(r_G,r_E)$ are $(917/249,\\ 1491/503)$ at prevalence $3/4$, with $1491/503<3$, and $(917/332,\\ 1988/503)$ at prevalence $4/5$, with $917/332<3$.\n\n\\begin{audittable}{llllll}{Two exact designs with equal prevalence in which one selection ratio is below three. Pairs are (infected, uninfected). The ratio is $r_G$ for the shooting gallery and $r_E$ for the needle exchange.}\nDesign & Population & Tested & Selection probabilities & Ratio & Prevalence \\\\\nA, gallery & $(48,16)$ & $(44,4)$ & $(11/12,\\ 1/4)$ & $11/3\\approx 3.67$ & $3/4$ \\\\\nA, exchange & $(864,288)$ & $(291,288)$ & $(97/288,\\ 1)$ & $288/97\\approx 2.97$ & $3/4$ \\\\\nB, gallery & $(48,12)$ & $(44,4)$ & $(11/12,\\ 1/3)$ & $11/4=2.75$ & $4/5$ \\\\\nB, exchange & $(1156,289)$ & $(291,288)$ & $(291/1156,\\ 288/289)$ & $384/97\\approx 3.96$ & $4/5$ \\\\\n\\end{audittable}", "problem": "The sentence makes two claims about the selection of the samples. It says that infected shooting gallery needles were more than three times as likely to be tested as uninfected ones, and that uninfected exchange needles were more than three times as likely to be tested as infected ones. Table 1 does not show this. The parameter $\\gamma$ is a common bound on how far the selection probabilities may differ within a population. On p.~236 the paper says that $\\gamma=5$ ``indicates that some units may be up to five times more likely to be sampled than others''. The table shows that the two intervals overlap only when this bound is larger than three. It says nothing about the ratio that each sample attains. Equal reweighted estimates fix the product of the two effective ratios, which must equal the odds ratio of the two sample proportions, about $10.9$. The product can be split unevenly, and one ratio can be below three when the other is larger. The paper does not assume that the two ratios are equal.", "what_survives": "Table 1 is unchanged, and so is the statement that the shooting gallery and needle exchange intervals overlap only when $\\gamma>3$, with the first overlap in the table at $\\gamma=4$. The comparison of the street and needle exchange samples and the method of sensitivity analysis are unaffected. I replace the inference in the last sentence about the two selection processes by a weaker statement about the common bound, without adding an assumption."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-E003", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of page 240: the word Thus, a displayed formula in which k superscript H equals a minimum over k of one condition, which equals a minimum over k of a second condition, and the sentence The solution to (3) can be derived in a similar manner. A red outline surrounds the equality sign between the two minima.", "caption": "Appendix, end of the proof of Proposition 1, p. 240 (PDF p. 6). The outline marks the equality between the two minima that define $k^{\\mathrm H}$.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E004.png"}, {"alt": "Crop of three lines of the Appendix proof, ending Then the difference of two ratios is less than or equal to 0 implies that, followed by a displayed inequality with y sub k plus 1 on the right. Red outlines surround the less-than-or-equal sign at the end of the second line and the words 0 implies that at the start of the third.", "caption": "Appendix, proof of Proposition 1, p. 239 (PDF p. 5). The outlines mark the words “$\\leqslant 0$ implies that”, which precede the displayed inequality.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E004-b.png"}, {"alt": "Crop of four lines of the supplementary material: Then the condition mu(1/beta, 1/beta) greater than mu(1/alpha, 1/beta) is equivalent to, the displayed inequality (B1), and the condition mu(1/alpha, 1/alpha) greater than mu(1/alpha, 1/beta) is equivalent to, and the displayed inequality (B2). Red outlines surround the words is equivalent to in the first and third lines.", "caption": "Supplementary material, proof of Proposition 2, p. 2. The outlines mark the two occurrences of “is equivalent to” that introduce (B1) and (B2).", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E004-c.png"}], "formal_release": null, "formal_statements": [{"id": "P025-T11", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T11"}, {"id": "P025-T12", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T12"}], "headline": "The Appendix argument for $k^{\\mathrm H}$, and the steps (B1) and (B2) of the supplementary material, hold for $\\gamma>1$; the case $\\gamma=1$ needs a separate sentence.", "id": "P025-E004", "location": "Appendix, proof of Proposition 1, pp. 239–240 (PDF pp. 5–6); supplementary material, proof of Proposition 2, (B1) and (B2), p. 2", "pdf_page": 5, "printed_page": "239–240", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P025-T11", "P025-T12"], "text": {"as_printed": "\\emph{Appendix, p.~239.} Let $w^k = (1/\\beta, \\ldots, 1/\\beta, 1/\\alpha, \\ldots, 1/\\alpha)$ where the first $k$ entries are $1/\\beta$ and the last $n - k$ are $1/\\alpha$. Then $(w^{k+1\\prime} y)/(w^{k+1\\prime} 1_n) - (w^{k\\prime} y)/(w^{k\\prime} 1_n) \\leqslant 0$ \\hl{implies that}\n\\[\n\\frac{\\sum_{i=1}^{k} y_i + \\gamma \\sum_{i=k+1}^{n} y_i}{k + \\gamma (n - k)} \\leqslant y_{k+1}.\n\\]\n\n\\emph{Appendix, p.~240.} Thus\n\\[\nk^{\\mathrm H} = \\min\\left\\{ k \\geqslant 0 : \\frac{w^{k\\prime} y}{w^{k\\prime} 1_n} \\geqslant \\frac{w^{k+1\\prime} y}{w^{k+1\\prime} 1_n} \\right\\} \\hl{=} \\min\\left\\{ k \\geqslant 0 : \\frac{\\sum_{i=1}^{k} y_i + \\gamma \\sum_{i=k+1}^{n} y_i}{k + \\gamma (n - k)} \\leqslant y_{k+1} \\right\\}.\n\\]\n\n\\emph{Supplementary material, p.~2.} Then the condition $\\mu(1/\\beta, 1/\\beta) > \\mu(1/\\alpha, 1/\\beta)$ \\hl{is equivalent to}\n\\[\nc_2 y_{(k)} - c_1 + \\beta^{-1} \\Pi_{k+1} (y_{(k)} - y_{(k+1)}) < 0 \\qquad \\text{(B1)}\n\\]\nand the condition $\\mu(1/\\alpha, 1/\\alpha) > \\mu(1/\\alpha, 1/\\beta)$ \\hl{is equivalent to}\n\\[\nc_1 - c_2 y_{(k+1)} + \\alpha^{-1} \\Pi_k (y_{(k)} - y_{(k+1)}) < 0. \\qquad \\text{(B2)}\n\\]", "corrected": "\\emph{Appendix, proof of Proposition 1.} \\hl{If $\\gamma=1$, then $\\alpha=\\beta$ and the only feasible weight vector is $(1/\\alpha,\\ldots,1/\\alpha)$. The bounds in (2) and (3) then both equal the sample mean, the cut point $k^{\\mathrm H}$ of Proposition 1 selects this vector, and (4) holds. For the rest of the proof, assume that $\\gamma>1$.} Let $w^k = (1/\\beta, \\ldots, 1/\\beta, 1/\\alpha, \\ldots, 1/\\alpha)$ where the first $k$ entries are $1/\\beta$ and the last $n - k$ are $1/\\alpha$. Then\\hl{, for $0\\leqslant k<n$,} $(w^{k+1\\prime} y)/(w^{k+1\\prime} 1_n) - (w^{k\\prime} y)/(w^{k\\prime} 1_n) \\leqslant 0$ \\hl{if and only if}\n\\[\n\\frac{\\sum_{i=1}^{k} y_i + \\gamma \\sum_{i=k+1}^{n} y_i}{k + \\gamma (n - k)} \\leqslant y_{k+1}.\n\\]\nThe display on p.~240 then holds as printed, with $k$ ranging over $0\\leqslant k<n$.\n\n\\emph{Supplementary material, proof of Proposition 2.} \\hl{If $\\alpha=\\beta$, then every feasible $W_j$ equals $1/\\alpha$, and (6) and (7) both equal $\\mu$. For the rest of the proof, assume that $\\alpha<\\beta$.} Then the condition $\\mu(1/\\beta, 1/\\beta) > \\mu(1/\\alpha, 1/\\beta)$ is equivalent to (B1), and the condition $\\mu(1/\\alpha, 1/\\alpha) > \\mu(1/\\alpha, 1/\\beta)$ is equivalent to (B2).", "evidence": "The evidence is an exact counterexample to the four printed steps and the identities that show where $\\gamma>1$ is used. Let $n=N=2$, $y=(0,1)$ and $\\alpha=\\beta=1$, so that $\\gamma=1$ and $w^0=w^1=w^2=(1,1)$. Then $F_0=F_1=F_2=1/2$. At $k=0$, I have $F_1-F_0=0\\leqslant 0$, and the conclusion $F_0=1/2\\leqslant y_1=0$ is false. For $k\\in\\{0,1\\}$, the first minimum on p.~240 is $0$ and the second is $1$. In the supplementary material, the two classes have $y_{(1)}=0$, $y_{(2)}=1$ and $\\Pi_1=\\Pi_2=1$. With $k=1$, I have $c_1=c_2=0$, the left sides of (B1) and (B2) both equal $-1$, and $\\mu(1/\\beta,1/\\beta)=\\mu(1/\\alpha,1/\\beta)=\\mu(1/\\alpha,1/\\alpha)=1/2$. For the general case, let $a=1/\\beta$, $b=1/\\alpha$, $p=\\Pi_k$, $q=\\Pi_{k+1}$, and let $E_1$ and $E_2$ be the left sides of (B1) and (B2). A direct computation gives\n\\[\n\\mu(a,a)-\\mu(b,a)=\\frac{(a-b)\\,p\\,E_1}{(c_2+ap+aq)(c_2+bp+aq)},\\qquad\n\\mu(b,b)-\\mu(b,a)=\\frac{(a-b)\\,q\\,E_2}{(c_2+bp+bq)(c_2+bp+aq)} .\n\\]\nBoth denominators are positive. Since $a-b<0$ for $\\alpha<\\beta$, the two equivalences hold in this case. For $\\alpha=\\beta$, both differences vanish.", "problem": "Proposition 1 allows $\\gamma=1$, and Table 1 includes it. Let $F_k=(w^{k\\prime}y)/(w^{k\\prime}1_n)$ and $D_k=k+\\gamma(n-k)$. Then\n\\[\nF_{k+1}-F_k=\\frac{(1-\\gamma)\\,(y_{k+1}-F_k)}{D_{k+1}}\\qquad\\text{for } 0\\leqslant k<n .\n\\]\nFor $\\gamma>1$, this identity implies that $F_{k+1}-F_k\\leqslant 0$ if and only if $F_k\\leqslant y_{k+1}$, and the two minima on p.~240 agree. For $\\gamma=1$, the vectors $w^k$ coincide and the difference is zero for every $k$. The printed implication and the equality of the two minima can then fail. The steps (B1) and (B2) have the same gap. Each difference of objective values equals $1/\\beta-1/\\alpha$ times a positive factor times the left side of (B1) or (B2), and for $\\alpha=\\beta$ the difference is zero whatever the sign of that left side. The results are not affected. For $\\gamma=1$ the only feasible weight vector has every entry equal to $1/\\alpha$, both bounds in (2) and (3) equal the sample mean, and both (6) and (7) equal $\\mu$.", "what_survives": "The upper bound (4) and its optimal weights hold for all $\\gamma\\geqslant 1$. Lemma 1, Propositions 2 and 3, the closed form for dichotomous outcomes and Table 1 are unaffected by this boundary case; the rows $\\gamma=1$ are the sample percentages. The identity for the sum of the left sides of (B1) and (B2) in the supplementary material is correct. One case is added to each proof, and no assumption or result changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-E004", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop that begins By Proposition 2, shows Assumption 2 with its two displayed inequalities, and ends with Proposition 3. Red outlines surround four items: the words necessary to guarantee convergence in the second line, the words For any k greater than or equal to 0 at the start of Assumption 2, and in Proposition 3 the symbols mu sub h superscript H and mu sub h superscript L, neither of which has a hat.", "caption": "Section 3.2, from the paragraph before Assumption 2 to Proposition 3, p. 238 (PDF p. 4). The outlines mark the words “necessary to guarantee convergence”, the words “For any $k\\geqslant 0$”, and the symbols $\\mu_h^{\\mathrm H}$ and $\\mu_h^{\\mathrm L}$, which are printed without hats.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P025-T13", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T13"}, {"id": "P025-T14", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T14"}, {"id": "P025-T21", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T21"}], "headline": "Assumption 2 is sufficient for the proof of Proposition 3 and is not necessary for the estimated bounds to converge; Proposition 3 should carry hats on the estimators, and Assumption 2 is meant for $0\\leqslant k<|Y|$.", "id": "P025-E005", "location": "Section 3.2, the sentence before Assumption 2, Assumption 2, and Proposition 3, p. 238 (PDF p. 4)", "pdf_page": 4, "printed_page": "238", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P025-T13", "P025-T14", "P025-T21"], "text": {"as_printed": "A mild technical condition on $k_+$ and $k_-$ is \\hl{necessary to guarantee convergence}.\n\nAssumption 2. \\hl{For any $k \\geqslant 0$},\n\\[\ny_{(k+1)} \\neq \\frac{\\sum_{j=1}^{k} \\Pi_j y_{(j)} + \\gamma \\sum_{j=k+1}^{|Y|} \\Pi_j y_{(j)}}{\\sum_{j=1}^{k} \\Pi_j + \\gamma \\sum_{j=k+1}^{|Y|} \\Pi_j}, \\qquad\ny_{(k+1)} \\neq \\frac{\\gamma \\sum_{j=1}^{k} \\Pi_j y_{(j)} + \\sum_{j=k+1}^{|Y|} \\Pi_j y_{(j)}}{\\gamma \\sum_{j=1}^{k} \\Pi_j + \\sum_{j=k+1}^{|Y|} \\Pi_j}.\n\\]\nPROPOSITION 3. If Assumptions 1 and 2 hold, then as $h \\to \\infty$, $\\hl{\\mu_h^{\\mathrm H}} \\to \\mu_+$ and $\\hl{\\mu_h^{\\mathrm L}} \\to \\mu_-$ almost everywhere.", "corrected": "A mild technical condition on $k_+$ and $k_-$ is \\hl{sufficient for the proof of convergence}.\n\nAssumption 2. \\hl{For every integer $k$ with $0 \\leqslant k < |Y|$},\n\\[\ny_{(k+1)} \\neq \\frac{\\sum_{j=1}^{k} \\Pi_j y_{(j)} + \\gamma \\sum_{j=k+1}^{|Y|} \\Pi_j y_{(j)}}{\\sum_{j=1}^{k} \\Pi_j + \\gamma \\sum_{j=k+1}^{|Y|} \\Pi_j}, \\qquad\ny_{(k+1)} \\neq \\frac{\\gamma \\sum_{j=1}^{k} \\Pi_j y_{(j)} + \\sum_{j=k+1}^{|Y|} \\Pi_j y_{(j)}}{\\gamma \\sum_{j=1}^{k} \\Pi_j + \\sum_{j=k+1}^{|Y|} \\Pi_j}.\n\\]\nPROPOSITION 3. If Assumptions 1 and 2 hold, then as $h \\to \\infty$, $\\hl{\\hat\\mu_h^{\\mathrm H}} \\to \\mu_+$ and $\\hl{\\hat\\mu_h^{\\mathrm L}} \\to \\mu_-$ almost everywhere.\n\n\\hl{The conclusion also holds without Assumption 2: if Assumption 1 holds, then as $h\\to\\infty$, $\\hat\\mu_h^{\\mathrm H}\\to\\mu_+$ and $\\hat\\mu_h^{\\mathrm L}\\to\\mu_-$ almost everywhere.}", "evidence": "The evidence is an exact example in which Assumption 2 fails and the bounds converge, a derivation that Assumption 1 suffices, and a check of the page for the hats and the range of $k$.\n\n\\emph{Example.} Let the population have outcomes $(0,0,2,2,3,3)$. Each replication selects one unit from each pair of units with equal outcomes, each with probability $1/2$, and the three choices are independent. Then $\\pi_i=1/2$ for all $i$, and I take $\\alpha=1/4$ and $\\beta=1/2$, so that $\\gamma=2$. The distinct outcomes are $(0,2,3)$ and $\\Pi=(1,1,1)$. After $h$ replications the sample contains exactly $h$ units at each outcome. With class weights $(w_0,w_2,w_3)\\in[2,4]^3$, the ratio is $f=(2w_2+3w_3)/(w_0+w_2+w_3)$ for the sample and in the limit, and\n\\[\n\\begin{aligned}\n4(2w_2+3w_3)-5(w_0+w_2+w_3)&=3(w_2-2)+7(w_3-2)+5(4-w_0)\\geqslant 0,\\\\\n2(w_0+w_2+w_3)-(2w_2+3w_3)&=2(w_0-2)+(4-w_3)\\geqslant 0 .\n\\end{aligned}\n\\]\nBy these two inequalities, I have $5/4\\leqslant f\\leqslant 2$, with equality at $(4,2,2)$ and at $(2,4,4)$. I conclude that $[\\hat\\mu_h^{\\mathrm L},\\hat\\mu_h^{\\mathrm H}]=[\\mu_-,\\mu_+]=[5/4,2]$ for every $h$ and every sample path. Assumption 2 fails at $k=1$, since the first ratio there equals $\\{1\\cdot 0+2(2+3)\\}/\\{1+2(1+1)\\}=2=y_{(2)}$.\n\n\\emph{Derivation.} Let $P_{h,j}$ be the number of sampled units of outcome $y_{(j)}$ after $h$ replications, divided by $h$. By Assumption 1, the counts of class $j$ in the successive replications are independent and identically distributed, they are bounded, and their mean is $\\Pi_j$. The strong law of large numbers implies that $P_{h,j}\\to\\Pi_j$ almost surely for every $j$. For a vector $p$ with nonnegative entries and positive sum, I define\n\\[\nf(W,p)=\\frac{\\sum_jW_jp_jy_{(j)}}{\\sum_jW_jp_j},\\qquad W\\in B=[1/\\beta,1/\\alpha]^{|Y|}.\n\\]\nIn (2) and (3) the weights of the units in one class can be replaced by their average, which lies in $[1/\\beta,1/\\alpha]$. Whenever the sample is nonempty, $\\hat\\mu_h^{\\mathrm H}=\\max_{W\\in B}f(W,P_h)$ and $\\hat\\mu_h^{\\mathrm L}=\\min_{W\\in B}f(W,P_h)$, and by (6) and (7), $\\mu_+=\\max_{W\\in B}f(W,\\Pi)$ and $\\mu_-=\\min_{W\\in B}f(W,\\Pi)$. Let $R=y_{(|Y|)}-y_{(1)}$ and $S=\\sum_j\\Pi_j$. With $m=f(W,p)$, I have\n\\[\nm-f(W,\\Pi)=\\frac{\\sum_jW_j(p_j-\\Pi_j)(y_{(j)}-m)}{\\sum_jW_j\\Pi_j},\n\\]\nand therefore\n\\[\n\\bigl|f(W,p)-f(W,\\Pi)\\bigr|\\leqslant\\frac{\\gamma R}{S}\\sum_j|p_j-\\Pi_j|\\qquad\\text{for all } W\\in B .\n\\]\nIn the last inequality, I used that $|y_{(j)}-m|\\leqslant R$ and $1/\\beta\\leqslant W_j\\leqslant 1/\\alpha$. The maximum and the minimum over $B$ change by at most the same amount. With $p=P_h$, I obtain $\\hat\\mu_h^{\\mathrm H}\\to\\mu_+$ and $\\hat\\mu_h^{\\mathrm L}\\to\\mu_-$ almost surely, including when the ratios of Assumption 2 tie.", "problem": "There are three points. First, the estimated bounds $\\hat\\mu_h^{\\mathrm H}$ and $\\hat\\mu_h^{\\mathrm L}$ converge almost surely to $\\mu_+$ and $\\mu_-$ under Assumption 1 alone. The supplementary material uses Assumption 2 to make the estimated cut point equal to $k_+$ for all large $h$, which the convergence of the bounds does not require. Proposition 3 is correct as a statement of sufficient conditions, with a hypothesis that it does not need. Second, Proposition 3 prints $\\mu_h^{\\mathrm H}$ and $\\mu_h^{\\mathrm L}$ without hats. Assumption 1 defines the estimators $\\hat\\mu_h^{\\mathrm H}$ and $\\hat\\mu_h^{\\mathrm L}$, and no quantity without a hat is defined. Third, $y_{(k+1)}$ is defined only for $0\\leqslant k<|Y|$, and ``for any $k\\geqslant 0$'' includes indices at which Assumption 2 has no meaning.", "what_survives": "The conclusion of Proposition 3 holds, under Assumption 1 alone. The optimization problems (2), (3), (6) and (7), Table 1 and the application are unaffected. The sentence before Assumption 2 and the notation of Proposition 3 and Assumption 2 change."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-E005", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the reference list of the article, from Brewer (1979) to Rosenbaum (2002). Red outlines surround six items: the names C.-E. Wretman in the entry for Cassel, Sarndal and Wretman; the words The Foundations at the end of one line and of Survey Sampling at the start of the next in the entry for Hajek; the pages 2 to 8 in the entry for Kaplan; and the words Statistical Methods in Epidemiology, Ed. E. Halloran, and pp. 1 to 92 on the next line, in the entry for Robins, Rotnitzky and Scharfstein.", "caption": "References, p. 240 (PDF p. 6). The outlines mark “C.-E. Wretman”, where the separator is missing, the volume title “The Foundations of Survey Sampling”, the pages “2–8”, and the clause “Statistical Methods in Epidemiology, Ed. E. Halloran, pp. 1–92”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-E007.png"}], "formal_release": null, "formal_statements": [], "headline": "Four entries of the reference list need a correction: the separator before Wretman, the title of the volume that contains the comment of Hajek, the pages of Kaplan (1991), and the title, editors and pages of the volume that contains Robins et al.", "id": "P025-E007", "location": "References, entries for Cassel et al. (1976), Hajek (1971), Kaplan (1991) and Robins et al. (1999), p. 240 (PDF p. 6)", "pdf_page": 6, "printed_page": "240", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "CASSEL, C. M., SÄRNDAL, \\hl{C.-E. WRETMAN}, J. H. (1976). Some results on generalized difference estimation and generalized regression estimation for finite populations. \\emph{Biometrika} \\textbf{63}, 615--20.\n\nHAJEK, J. (1971). Comment on ``An essay on the logical foundations of survey sampling, part one.''. In \\hl{\\emph{The Foundations of Survey Sampling}}, Ed. V. P. Godambe \\& D. A. Sprott, p. 246. Toronto: Holt, Rinehart, and Winston.\n\nKAPLAN, E. H. (1991). Evaluating needle-exchange programs via syringe tracking and testing (STT). \\emph{AIDS Public Pol. J.} \\textbf{6}, \\hl{2--8}.\n\nROBINS, J. M., ROTNITZKY, A. \\& SCHARFSTEIN, D. O. (1999). Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In \\hl{\\emph{Statistical Methods in Epidemiology}, Ed. E. Halloran, pp. 1--92}. New York: Springer.", "corrected": "CASSEL, C. M., SÄRNDAL, C.-E. \\hl{\\&} WRETMAN, J. H. (1976). Some results on generalized difference estimation and generalized regression estimation for finite populations. \\emph{Biometrika} \\textbf{63}, 615--20.\n\nHAJEK, J. (1971). Comment on ``An essay on the logical foundations of survey sampling, part one.''. In \\hl{\\emph{Foundations of Statistical Inference}}, Ed. V. P. Godambe \\& D. A. Sprott, p. 246. Toronto: Holt, Rinehart, and Winston.\n\nKAPLAN, E. H. (1991). Evaluating needle-exchange programs via syringe tracking and testing (STT). \\emph{AIDS Public Pol. J.} \\textbf{6}, \\hl{109--15}.\n\nROBINS, J. M., ROTNITZKY, A. \\& SCHARFSTEIN, D. O. (1999). Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In \\hl{\\emph{Statistical Models in Epidemiology, the Environment, and Clinical Trials}, Eds. M. E. Halloran \\& D. Berry, pp. 1--94}. New York: Springer.", "evidence": "I compared the printed entries with public bibliographic records. The publisher's record for the volume of Halloran and Berry gives its title, its editors and the pages $1$--$94$ of the chapter (\\url{https://link.springer.com/book/10.1007/978-1-4612-1284-3}). E.~H. Kaplan's list of his papers gives \\emph{AIDS and Public Policy Journal} 6:109--115, 1991 (\\url{https://faculty.som.yale.edu/edkaplan/papers/}). Two library catalogues give the title \\emph{Foundations of statistical inference: a symposium} for the volume of 1971 edited by Godambe and Sprott (\\url{https://ci.nii.ac.jp/ncid/BA8882938X}). No record that locates the comment of Hajek within that volume was found, and the page $246$ is left as printed. The publisher released the volume of Halloran and Berry in 1999 and dates its copyright 2000, and the year 1999 is not counted as an error.", "problem": "The entry for Cassel et al.\\ lacks the separator before the third author. The volume edited by Godambe and Sprott is \\emph{Foundations of Statistical Inference}. The article of Kaplan is on pp.~109--115 of its volume. The chapter of Robins et al.\\ is on pp.~1--94 of \\emph{Statistical Models in Epidemiology, the Environment, and Clinical Trials}, edited by M.~E. Halloran and D.~Berry. Each cited work can still be identified, and no result or number depends on these entries.", "what_survives": "The mathematics, the calculations and the citations in the text are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-E007", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of the second paragraph of the Introduction, beginning Our approach serves two primary purposes for applied practitioners. Red outlines surround the words sharp bounds on the population mean, which run from the end of the second line to the start of the third.", "caption": "Section 1, second paragraph, p. 235 (PDF p. 1). The outlines mark the words “sharp bounds on the population mean”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "At a finite sample size the bounds are sharp bounds on the Hajek estimator, and they bound the population mean only in the limit of Section 3.2.", "id": "P025-N001", "location": "Section 1, second paragraph, second sentence, p. 235 (PDF p. 1)", "pdf_page": 1, "printed_page": "235", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "First, when there is a theoretical basis for bounding the probabilities of sample selection, the proposed method yields \\hl{sharp bounds on the population mean}.", "corrected": "First, when there is a theoretical basis for bounding the probabilities of sample selection, the proposed method yields sharp bounds on \\hl{the Hajek estimator of} the population mean.", "intended_meaning": "I read the sentence as referring to the interval estimate of Section 3.1, which is the set of values that the Hajek estimator $\\hat\\mu$ can take when every $\\pi_i$ lies in $[\\alpha,\\beta]$. The Summary (p.~235) and Section 3.1 (p.~236) state the claim in this form, and it is correct. At a finite sample size the interval need not contain the population mean $\\mu$. For the population $(0,10)$ with one unit selected uniformly and $\\alpha=\\beta=1/2$, the interval is $[0,0]$ or $[10,10]$, and $\\mu=5$. The limiting statement about $\\mu$ is the subject of \\fid{P025-N002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-N001", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop that begins However, W sub j is actually only identified up to the interval, continues through the displayed equations (6) and (7) for mu plus and mu minus, and ends with the first line of Proposition 2. Red outlines surround the words so in the limit, mu can be identified up to the interval bounded by, on two lines, and the words on mu are sharp in Proposition 2.", "caption": "Section 3.2, from the sentence before (6) to the first line of Proposition 2, p. 237 (PDF p. 3). The outlines mark the words “so in the limit, $\\mu$ can be identified up to the interval bounded by” and “on $\\mu$ are sharp”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N002.png"}, {"alt": "Crop of the heading 3.2 Asymptotic sharpness of estimated bounds and the sentence below it. A red outline surrounds the words asymptotically sharp bounds on mu itself.", "caption": "Section 3.2, heading and first sentence, p. 237 (PDF p. 3). The outline marks the words “asymptotically sharp bounds on $\\mu$ itself”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P025-T15", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T15"}, {"id": "P025-T16", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T16"}], "headline": "The bounds $\\mu_-$ and $\\mu_+$ are sharp for the population mean relative to the limiting distribution of outcomes among sampled units, with the population size unrestricted, and the statement should name this information.", "id": "P025-N002", "location": "Section 3.2, first sentence, the sentence before (6), and the first sentence of Proposition 2, p. 237 (PDF p. 3)", "pdf_page": 3, "printed_page": "237", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P025-T15", "P025-T16"], "text": {"as_printed": "The proposed interval estimator yields \\hl{asymptotically sharp bounds on $\\mu$ itself} under the scaling proposed by Brewer (1979) and articulated in Assumption 1.\n\nHowever, $W_j$ is actually only identified up to the interval $\\beta^{-1} \\leqslant W_j \\leqslant \\alpha^{-1}$; \\hl{so in the limit, $\\mu$ can be identified up to the interval bounded by}\n\\[\n\\mu_+ = \\max_W \\left\\{ \\frac{\\sum_{j=1}^{|Y|} W_j \\Pi_j y_{(j)}}{\\sum_{j=1}^{|Y|} W_j \\Pi_j} : 1/\\beta \\leqslant W_1, \\ldots, W_{|Y|} \\leqslant 1/\\alpha \\right\\} \\qquad (6)\n\\]\nand\n\\[\n\\mu_- = \\min_W \\left\\{ \\frac{\\sum_{j=1}^{|Y|} W_j \\Pi_j y_{(j)}}{\\sum_{j=1}^{|Y|} W_j \\Pi_j} : 1/\\beta \\leqslant W_1, \\ldots, W_{|Y|} \\leqslant 1/\\alpha \\right\\}. \\qquad (7)\n\\]\nPROPOSITION 2. In the limit $h \\to \\infty$, the bounds $\\mu_+$ and $\\mu_-$ \\hl{on $\\mu$ are sharp}. Specifically,", "corrected": "The proposed interval estimator yields asymptotically sharp bounds on $\\mu$ itself under the scaling proposed by Brewer (1979) and articulated in Assumption 1\\hl{, relative to the limiting distribution of the outcomes among sampled units}.\n\nHowever, $W_j$ is actually only identified up to the interval $\\beta^{-1} \\leqslant W_j \\leqslant \\alpha^{-1}$; so in the limit, \\hl{given only the shares $q_j=\\Pi_j/\\sum_{k=1}^{|Y|}\\Pi_k$ and with the population size $N$ unrestricted,} $\\mu$ can be identified up to the interval bounded by $\\mu_+$ in (6) and $\\mu_-$ in (7).\n\nPROPOSITION 2. \\hl{Let $0<\\alpha\\leqslant\\beta\\leqslant 1$, and let $q_j=\\Pi_j/\\sum_{k=1}^{|Y|}\\Pi_k$ for $j=1,\\ldots,|Y|$ be the limiting shares of the outcomes among sampled units. Suppose that only $q$ is retained from the limit $h\\to\\infty$ and that the size $N$ of the original population is unrestricted. Then} the bounds $\\mu_+$ and $\\mu_-$ on $\\mu$ are sharp\\hl{, in the following sense. Every finite population and sampling design with $\\alpha\\leqslant\\pi_i\\leqslant\\beta$ for all $i$ and with shares $q$ has $\\mu_-\\leqslant\\mu\\leqslant\\mu_+$, and $\\mu_-$ and $\\mu_+$ are the infimum and the supremum of the means of such populations.} Specifically, $\\mu_+$ and $\\mu_-$ are given by the two displayed formulas, with the corrections of \\fid{P025-E001} and \\fid{P025-E002}. \\hl{If $N$ is known, or if the totals $\\Pi_j$ are retained together with fixed numerical values of $\\alpha$ and $\\beta$, then $[\\mu_-,\\mu_+]$ still contains $\\mu$, and it can be strictly wider than the set of compatible means.}", "evidence": "The evidence is two exact counterexamples to the unqualified claim and a derivation of the qualified one.\n\n\\emph{Totals retained, $N$ unknown.} Let the population have outcomes $(0,1,1)$, and let each replication select one of the three samples of two units with probability $1/3$. Every $\\pi_i$ equals $2/3$, and I take $\\alpha=1/2$ and $\\beta=3/4$. The totals for the outcomes $0$ and $1$ are $2/3$ and $4/3$. The sizes $m_0$ and $m_1$ of the two classes satisfy $8/9\\leqslant m_0\\leqslant 4/3$ and $16/9\\leqslant m_1\\leqslant 8/3$, which implies that $(m_0,m_1)=(1,2)$ and $\\mu=2/3$. Equations (6) and (7) give $[\\mu_-,\\mu_+]=[4/7,3/4]$. The gaps $2/3-4/7=2/21$ and $3/4-2/3=1/12$ do not depend on $h$.\n\n\\emph{$N$ known.} Let $\\Pi=(50,100)$ for the outcomes $0$ and $1$, $\\alpha=1/10$, $\\beta=1/5$ and $N=1000$. The class sizes satisfy $250\\leqslant m_0\\leqslant 500$, $500\\leqslant m_1\\leqslant 1000$ and $m_0+m_1=1000$. The compatible means are $m_1/1000$ with $500\\leqslant m_1\\leqslant 750$, and each is attained by a design that selects $50$ zeros and $100$ ones uniformly. The sharp interval is $[1/2,3/4]$, and the interval defined by (6) and (7) is $[1/2,4/5]$.\n\n\\emph{Shares only, $N$ unrestricted.} Let $r_j=|\\Upsilon_j|/N$ be the population proportions, so that $\\mu=\\sum_jr_jy_{(j)}$ and $r_j=W_jq_j/\\sum_kW_kq_k$. As $W$ ranges over the box, $r$ ranges over the set\n\\[\n\\mathcal C_\\gamma(q)=\\Bigl\\{r:\\ r_j\\geqslant 0,\\ \\sum_jr_j=1,\\ r_jq_k\\leqslant\\gamma\\,r_kq_j \\text{ for all } j,k\\Bigr\\},\n\\]\nand $\\mu_-$ and $\\mu_+$ are the minimum and the maximum of $\\sum_jr_jy_{(j)}$ over $\\mathcal C_\\gamma(q)$. Since every compatible population has $r\\in\\mathcal C_\\gamma(q)$, it satisfies $\\mu_-\\leqslant\\mu\\leqslant\\mu_+$. Conversely, let $r\\in\\mathcal C_\\gamma(q)$ have rational coordinates, and let $d$ be an integer such that every $dr_j$ is an integer. I consider a population with $dr_j$ units of outcome $y_{(j)}$, each selected with probability $tq_j/r_j$, where $t$ satisfies\n\\[\n\\alpha\\max_j\\frac{r_j}{q_j}\\leqslant t\\leqslant\\beta\\min_j\\frac{r_j}{q_j}.\n\\]\nSuch a $t$ exists because $r\\in\\mathcal C_\\gamma(q)$. These probabilities lie in $[\\alpha,\\beta]$, the shares of the outcomes among sampled units are exactly $q$, and the population mean is $\\sum_jr_jy_{(j)}$. For $\\gamma>1$, every inequality that defines $\\mathcal C_\\gamma(q)$ is strict at $r=q$, which implies that the rational points are dense in $\\mathcal C_\\gamma(q)$. I conclude that the closure of the set of compatible population means is $[\\mu_-,\\mu_+]$. For $\\gamma=1$, the set $\\mathcal C_\\gamma(q)$ is the point $q$. In the first example, the shares are $q=(1/3,2/3)$, and both bounds are attained when only $q$ is retained. A population of three zeros, each selected with probability $1/2$, and four ones, each selected with probability $3/4$, has the mean $4/7$. A population of one zero, selected with probability $3/4$, and three ones, each selected with probability $1/2$, has the mean $3/4$.", "problem": "Proposition 2 and the first sentence of Section 3.2 call the interval $[\\mu_-,\\mu_+]$ sharp for $\\mu$ and do not say relative to which information. The supplementary material shows that $\\mu_-$ and $\\mu_+$ are the minimum and the maximum of the ratio over the box $1/\\beta\\leqslant W_j\\leqslant 1/\\alpha$. It does not show that populations compatible with what is observed in the limit have means that reach or approach these values, and whether they do depends on what is observed. The identification display on p.~237 names only the limiting shares $q_j=\\Pi_j/\\sum_k\\Pi_k$ of the outcomes among sampled units. Relative to these shares, with the population size $N$ unrestricted, the claim holds when sharp means that $\\mu_-$ and $\\mu_+$ are the infimum and the supremum of the compatible population means. The claim fails for richer information. The class sizes $|\\Upsilon_j|=W_j\\Pi_j$ are positive integers. If the totals $\\Pi_j$ themselves are retained, with fixed numerical $\\alpha$ and $\\beta$, these integers lie in $[\\Pi_j/\\beta,\\Pi_j/\\alpha]$, and if $N$ is known they sum to $N$. The interval can then be strictly narrowed. The same qualification applies to the conclusion drawn from Proposition 2 on p.~238 and to the sentence of Section 1 discussed in \\fid{P025-N001}. In every case $[\\mu_-,\\mu_+]$ contains $\\mu$.", "what_survives": "The formulas (6) and (7), the form of the optimal weights, the bounds on the Hajek estimator, the convergence of the estimated bounds to $\\mu_-$ and $\\mu_+$, Table 1 and the sensitivity analysis do not change. The interval $[\\mu_-,\\mu_+]$ is a valid bound on $\\mu$ whatever else is known. The stronger claim does not hold. When $N$ or the totals per replication are also known, the interval can be narrowed. Sharpness also does not mean that every point of the interval is the mean of a finite population."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-N002", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 1 with its caption, Estimated bounds for prevalence rates, in percent, of HIV proviral DNA in syringes collected as part of the New Haven syringe tracking and testing programme. The table has three panels, Shooting gallery n = 48, Street n = 160 and Needle exchange n = 579, each with columns gamma, lower bound and upper bound and seven rows for gamma from 1 to 4. Red outlines surround six entries: in the shooting gallery panel the lower bounds 84.7 at gamma 2 and 73.4 at gamma 4, and in the needle exchange panel the upper bound 55.9 at gamma 1.25, the lower bound 40.3 and the upper bound 60.3 at gamma 1.5, and the upper bound 71.7 at gamma 2.5.", "caption": "Table 1, p. 238 (PDF p. 4). The outlines mark the six entries that are $0.1$ too high: $84{\\cdot}7$ and $73{\\cdot}4$ for the shooting gallery, and $55{\\cdot}9$, $40{\\cdot}3$, $60{\\cdot}3$ and $71{\\cdot}7$ for the needle exchange.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N003.png"}], "formal_release": null, "formal_statements": [{"id": "P025-T17", "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-T17"}], "headline": "Six entries of Table 1 are $0.1$ percentage point above the bounds computed from the counts that the sample sizes and percentages imply.", "id": "P025-N003", "location": "Table 1, p. 238 (PDF p. 4): shooting gallery, $\\hat\\mu^{\\mathrm L}$ at $\\gamma=2$ and $\\gamma=4$; needle exchange, $\\hat\\mu^{\\mathrm H}$ at $\\gamma=1{\\cdot}25$, both bounds at $\\gamma=1{\\cdot}5$, and $\\hat\\mu^{\\mathrm H}$ at $\\gamma=2{\\cdot}5$", "pdf_page": 4, "printed_page": "238", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P025-T17"], "text": {"as_printed": "Table 1, estimated bounds (\\%). Shooting gallery, $n = 48$: at $\\gamma = 2$, $\\hat\\mu^{\\mathrm L} = \\hl{84{\\cdot}7}$; at $\\gamma = 4$, $\\hat\\mu^{\\mathrm L} = \\hl{73{\\cdot}4}$. Needle exchange, $n = 579$: at $\\gamma = 1{\\cdot}25$, $\\hat\\mu^{\\mathrm H} = \\hl{55{\\cdot}9}$; at $\\gamma = 1{\\cdot}5$, $\\hat\\mu^{\\mathrm L} = \\hl{40{\\cdot}3}$ and $\\hat\\mu^{\\mathrm H} = \\hl{60{\\cdot}3}$; at $\\gamma = 2{\\cdot}5$, $\\hat\\mu^{\\mathrm H} = \\hl{71{\\cdot}7}$.", "corrected": "Table 1, estimated bounds (\\%), computed from $44$ of $48$, $108$ of $160$ and $291$ of $579$ positive tests and rounded at the last step. Shooting gallery, $n = 48$: at $\\gamma = 2$, $\\hat\\mu^{\\mathrm L} = \\hl{84{\\cdot}6}$; at $\\gamma = 4$, $\\hat\\mu^{\\mathrm L} = \\hl{73{\\cdot}3}$. Needle exchange, $n = 579$: at $\\gamma = 1{\\cdot}25$, $\\hat\\mu^{\\mathrm H} = \\hl{55{\\cdot}8}$; at $\\gamma = 1{\\cdot}5$, $\\hat\\mu^{\\mathrm L} = \\hl{40{\\cdot}2}$ and $\\hat\\mu^{\\mathrm H} = \\hl{60{\\cdot}2}$; at $\\gamma = 2{\\cdot}5$, $\\hat\\mu^{\\mathrm H} = \\hl{71{\\cdot}6}$. The other 36 entries are unchanged.", "evidence": "I recalculated all 42 entries in exact rational arithmetic from the closed form $[p/\\{\\gamma(1-p)+p\\},\\ \\gamma p/\\{(1-p)+\\gamma p\\}]$. The counts are inferred from the sample sizes and percentages printed in Section 4 and not from the data of Kaplan (1991). The other 36 entries agree with the counts.\n\n\\begin{audittable}{llrrr}{The six entries of Table 1 that change, in percent.}\nSample & Entry & Exact value from the counts & Rounded & Printed \\\\\nShooting gallery & $\\hat\\mu^{\\mathrm L}$ at $\\gamma=2$ & $1100/13=84.615\\ldots$ & $84.6$ & $84.7$ \\\\\nShooting gallery & $\\hat\\mu^{\\mathrm L}$ at $\\gamma=4$ & $220/3=73.333\\ldots$ & $73.3$ & $73.4$ \\\\\nNeedle exchange & $\\hat\\mu^{\\mathrm H}$ at $\\gamma=1.25$ & $48500/869=55.811\\ldots$ & $55.8$ & $55.9$ \\\\\nNeedle exchange & $\\hat\\mu^{\\mathrm L}$ at $\\gamma=1.5$ & $9700/241=40.248\\ldots$ & $40.2$ & $40.3$ \\\\\nNeedle exchange & $\\hat\\mu^{\\mathrm H}$ at $\\gamma=1.5$ & $9700/161=60.248\\ldots$ & $60.2$ & $60.3$ \\\\\nNeedle exchange & $\\hat\\mu^{\\mathrm H}$ at $\\gamma=2.5$ & $48500/677=71.639\\ldots$ & $71.6$ & $71.7$ \\\\\n\\end{audittable}", "problem": "Section 4 reports $48$, $160$ and $579$ tested needles, of which $91.7\\%$, $67.5\\%$ and $50.3\\%$ tested positive. Each percentage is compatible with exactly one count of positive tests, namely $44$ of $48$, $108$ of $160$ and $291$ of $579$. The closed form on p.~237 defines $p$ as the sample proportion. With $p=44/48$ and $p=291/579$, six bounds round to a value $0.1$ below the printed one. All 42 printed entries are reproduced when the rounded values $0.917$, $0.675$ and $0.503$ are used as $p$. The six entries are therefore bounds for rounded inputs and not, at the precision shown, bounds for the samples described.", "what_survives": "No qualitative conclusion of Section 4 changes. For each pair of samples and each value of $\\gamma$ in the table, the two intervals overlap after the correction exactly when they overlap in the printed table. The closed form and the propositions do not depend on the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-N003", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last three lines of page 238. A red outline surrounds the words possibly show the same at the end of the last line, where the sentence breaks across the page.", "caption": "Section 4, start of the last sentence, foot of p. 238 (PDF p. 4). The outline marks the words “possibly show the same”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N006.png"}, {"alt": "Crop of the first three lines of page 239, which complete the last sentence of Section 4. Red outlines surround the words HIV prevalence rate at the start of the first line and the words must have been later in the same line.", "caption": "Section 4, end of the last sentence, p. 239 (PDF p. 5). The outlines mark the words “HIV prevalence rate” and “must have been”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The comparison in Section 4 is between the estimated bounds for the samples in hand, and it does not allow for sampling error.", "id": "P025-N006", "location": "Section 4, last sentence, pp. 238–239 (PDF pp. 4–5)", "pdf_page": 4, "printed_page": "238–239", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "This means that for the shooting gallery needles to \\hl{possibly show the same HIV prevalence rate} as the exchange needles, infected shooting gallery needles \\hl{must have been} more than three times as likely to be tested than uninfected shooting gallery needles, with the converse holding for exchange needles.", "corrected": "For the shooting gallery and needle exchange \\hl{interval estimates} to overlap, the common bound on the ratios of selection probabilities must be at least about $3{\\cdot}30$. \\hl{This comparison is conditional on the observed samples and does not account for sampling uncertainty.}", "intended_meaning": "I read the paragraph as comparing the estimated bounds of Table 1 with the observed samples held fixed, and ``the same HIV prevalence rate'' refers to the estimates. On this reading the comparison of the intervals is correct. The paper attaches no confidence level to Table 1, and the shooting gallery sample has $48$ needles. Read as a condition for equal true prevalences, the sentence would not follow. Let $n$ be the size of a sample and $k$ its number of positive tests. Suppose that the population consists of $32$ blocks of $n$ needles, that $16$ blocks contain $k$ positives and $16$ contain $n-k$, and that one block is selected uniformly and tested in full. Every needle is tested with probability $1/32$, and the true prevalence is $1/2$. With independent selections in the three populations, the reported counts $44$ of $48$, $108$ of $160$ and $291$ of $579$ occur together with probability $1/8$. The claim about each selection ratio is corrected in \\fid{P025-E003}."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-N006", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the last three lines of the proof of Proposition 3 in the supplementary material. A red outline surrounds the words along each sample path omega in the second line, after the statement that the estimated upper bound tends to mu plus.", "caption": "Supplementary material, end of the proof of Proposition 3, p. 2. The outline marks the words “along each sample path $\\omega$”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N007.png"}, {"alt": "Crop of the heading Proof of Proposition 3 and the two lines below it. A red outline surrounds the opening words Fix a sample realization omega in Omega = {0, 1} to the power of the natural numbers.", "caption": "Supplementary material, start of the proof of Proposition 3, p. 2. The outline marks the words “Fix a sample realization $\\omega\\in\\Omega=\\{0,1\\}^{\\mathbb N}$”.", "url": "https://pmaronow.github.io/llm-audit/works/P025/exhibits/P025-N007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In the proof of Proposition 3, the estimated bound converges along almost every sample path, as the proposition states, and not along each one.", "id": "P025-N007", "location": "Supplementary material, proof of Proposition 3, first and last sentences, p. 2", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Fix a sample realization $\\hl{\\omega \\in \\Omega} = \\{0, 1\\}^{\\mathbb N}$ so that $I_1(\\omega), I_2(\\omega), \\ldots$ is a deterministic binary sequence. [\\ldots] By the strong law of large numbers $P_{h,j} \\to \\Pi_j$ as $h \\to \\infty$. [\\ldots] Then (B4) implies that $\\hat\\mu_h^{H} \\to \\mu_+$ \\hl{along each sample path $\\omega$}.", "corrected": "Fix a sample realization $\\omega \\in \\hl{\\Omega_0}$, \\hl{where $\\Omega_0 = \\{\\omega \\in \\Omega : P_{h,j}(\\omega) \\to \\Pi_j \\text{ for every } j\\}$ has probability one by the strong law of large numbers}, so that $I_1(\\omega), I_2(\\omega), \\ldots$ is a deterministic binary sequence. [\\ldots] Then (B4) implies that $\\hat\\mu_h^{H} \\to \\mu_+$ along \\hl{almost every} sample path $\\omega$.", "intended_meaning": "I apply the argument only to the event of probability one on which the strong law of large numbers gives $P_{h,j}\\to\\Pi_j$ for every $j$. Its conclusion is the convergence almost everywhere that Proposition 3 states. On a path outside the event the convergence can fail. For the population $(0,1)$ with one unit selected uniformly in each replication, $\\alpha=1/4$ and $\\beta=3/4$, Assumption 2 holds and $\\mu_+=3/4$, and on the path that selects the unit with outcome $0$ in every replication the upper bound equals $0$ for all $h$. This path has probability zero. The sample space $\\{0,1\\}^{\\mathbb N}$ consists of infinite sequences and needs no change."}, "url": "https://pmaronow.github.io/llm-audit/works/P025/#P025-N007", "work": {"arxiv_id": null, "doi": "10.1093/biomet/ass064", "title": "Interval Estimation of Population Means under Unknown but Bounded Probabilities of Sample Selection", "url": "https://pmaronow.github.io/llm-audit/works/P025/"}, "work_id": "P025"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the first four lines of the page. A red box marks the inequality 'Var of Y-hat-1-T-star is less than Var of Y-hat-1-T', which follows the words 'will be correlated across randomizations, thus yielding'.", "caption": "Top of p. 141 (PDF p. 7), the end of the sentence that begins on p. 140. The marked inequality is stated as a consequence of correlation alone.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-A001.png"}, {"alt": "Crop of five lines of the paragraph beginning 'Consider two ways that one might derive theta-i'. A red box marks the words 'may increase the variance of the estimate,'.", "caption": "Section 5.1, paragraph after equation [19], p. 142 (PDF p. 8). The marked words are the paper’s own warning that a poor choice of $\\theta_i$ can raise the variance.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-A001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The intuition on pp. 140–141 passes from correlation to a strict reduction in variance; with predictions fixed across randomizations, the variance falls exactly when $2\\operatorname{Cov}(H,F)>\\operatorname{Var}(F)$ for the outcome total $H$ and the prediction total $F$.", "id": "P026-A001", "location": "Section 5, sentence after equations [13] and [14], pp. 140–141 (PDF pp. 6–7); compare the warning after equation [19], p. 142 (PDF p. 8)", "pdf_page": 6, "printed_page": "140–141", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "pp.~140--141: ``In inspecting eq. [13], one intuition is that if $f(\\mathbf{X}_i,\\theta_i)$ predicts the value of $Y_{1i}$, then across units in the study population $f(\\mathbf{X}_i,\\theta_i)$ and $Y_{1i}$ will be correlated. By implication then, $\\sum_{i=1}^{N}T_i\\frac{Y_{1i}}{\\pi_{1i}}$ and $\\sum_{i=1}^{N}T_i\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}}$ will be correlated across randomizations, thus yielding $\\widehat{Y_1^{T*}}$ such that $\\hl{\\operatorname{Var}\\left(\\widehat{Y_1^{T*}}\\right)<\\operatorname{Var}\\left(\\widehat{Y_1^{T}}\\right)}$. The same intuition holds for eq. [14], so that both estimators will typically have precision gains.''\n\np.~142: ``The choice of $\\theta_i$ may be suboptimal and, if chosen poorly, may increase the variance of the estimate, but, so long as the analyst does not use the data at hand in forming a judgment, there will be no consequence for bias.''", "corrected": "``\\ldots will be correlated across randomizations. \\hl{Let} $H$ \\hl{and} $F$ \\hl{denote these two totals. When the predictions are fixed across randomizations, the adjusted total has smaller variance than the unadjusted total exactly when} $\\hl{2\\operatorname{Cov}(H,F)>\\operatorname{Var}(F)}$. \\hl{Correlation alone does not ensure this, and a poor prediction can increase the variance (see p.~142).} The same holds for eq. [14], and both estimators will typically have precision gains when the predictions are good.''", "intended_meaning": "The passage states an intuition and not a theorem: a prediction that tracks the outcome usually lowers the variance. The words ``one intuition'' and ``typically'' qualify it, and p.~142 warns that a poorly chosen $\\theta_i$ may increase the variance. Read in this way the passage is correct. Read literally, the step from ``correlated'' to a strict inequality leaves out a condition. Let\n\\[\nH=\\sum_{i=1}^{N}T_i\\frac{Y_{1i}}{\\pi_{1i}},\\qquad F=\\sum_{i=1}^{N}T_i\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}},\n\\]\nand suppose that the predictions are fixed across randomizations, so that $\\sum_{i=1}^{N}f(\\mathbf{X}_i,\\theta_i)$ is a constant. By equation [13], I have $\\widehat{Y_1^{T*}}=H-F+\\sum_{i=1}^{N}f(\\mathbf{X}_i,\\theta_i)$, and therefore\n\\[\n\\operatorname{Var}\\left(\\widehat{Y_1^{T*}}\\right)-\\operatorname{Var}\\left(\\widehat{Y_1^{T}}\\right)=\\operatorname{Var}(F)-2\\operatorname{Cov}(H,F).\n\\]\nThe variance falls exactly when $2\\operatorname{Cov}(H,F)>\\operatorname{Var}(F)$, and a positive correlation does not ensure this. Suppose that the population has two units, with $Y_{11}=1$ and $Y_{12}=2$, that exactly one of them is treated, each with probability $1/2$, and that the predictions are fixed at $f=3Y_{1i}$. Then $H$ takes the values $2$ and $4$, $F$ takes the values $6$ and $12$, and the correlation of $H$ and $F$ is $1$. A direct computation gives $\\operatorname{Var}(H)=1$, $\\operatorname{Var}(F)=9$ and $\\operatorname{Cov}(H,F)=3$, and the adjusted total takes the values $5$ and $1$, with variance $4$. If the predictions are random, the same identity holds with $F-\\sum_{i=1}^{N}f(\\mathbf{X}_i,\\theta_i)$ in place of $F$. The definitions of the estimators, the results on unbiasedness and the warning on p.~142 are unaffected, and no reported number depends on this sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-A001", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the paragraph beginning 'Following the basic logic of Williams'. Red boxes mark the words 'any', 'or all of the elements of the set of Y-j such that T-i is independent of T-j', and 'then T-i is independent of theta-i'.", "caption": "Section 5.1, second paragraph, p. 142 (PDF p. 8). The marked words define the admissible outcomes by independence of $T_i$ from each $T_j$ separately and conclude that $T_i$ is independent of $\\theta_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E001.png"}, {"alt": "Crop of the four lines of footnote 5. Red boxes mark the words 'without loss of generality,', the expression 'f of X-i and g of Z and Y-j', which has a single outcome Y-j as its argument, and the step 'T-i independent of T-j implies T-i independent of f-prime of T-j'.", "caption": "Footnote 5, p. 142 (PDF p. 8). The argument is written for a single outcome $Y_j$ and is said to lose no generality.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P026-T4", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T4"}, {"id": "P026-T5", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T5"}, {"id": "P026-T18", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T18"}], "headline": "A sufficient condition for fitting $\\theta_i$ from other units' outcomes while preserving unbiasedness is independence of $T_i$ from the whole vector of assignments used in the fit; independence from each assignment separately does not suffice.", "id": "P026-E001", "location": "Section 5.1, second paragraph, and footnote 5, p. 142 (PDF p. 8)", "pdf_page": 8, "printed_page": "142", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P026-T18", "P026-T4", "P026-T5"], "text": {"as_printed": "``If $\\theta_i$ is a function of \\hl{any or all} of the elements of the set $\\hl{\\{Y_j:T_i\\perp\\!\\!\\!\\perp T_j\\}}$ and no other random variables, then $T_i\\perp\\!\\!\\!\\perp\\theta_i$. It follows that $T_i\\perp\\!\\!\\!\\perp f(\\mathbf{X}_i,\\theta_i)$ and therefore $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$.${}^{5}$''\n\nFootnote 5: ``More formally and \\hl{without loss of generality}, let $f(\\mathbf{X}_i,\\theta_i)=f(\\mathbf{X}_i,g(\\mathbf{Z},\\hl{Y_j}))=f(\\mathbf{X}_i,g(\\mathbf{Z},\\hl{h(Y_{0j},Y_{1j};T_j)})))$, where $\\mathbf{Z}$ is a matrix of pretreatment covariates (that may or may not coincide with $\\mathbf{X}_j$), $g(.)$ is an arbitrary function (e.g., the least squares fit), and $h(.)$ is the function implied by eq. [1]. Since only $T_j$ is a random variable, the random variable $f(\\mathbf{X}_i,\\theta_i)$ equals some function $f'(T_j)$. $T_i\\perp\\!\\!\\!\\perp T_j$ implies $T_i\\perp\\!\\!\\!\\perp f'(T_j)$ (equivalently $T_i\\perp\\!\\!\\!\\perp f(\\mathbf{X}_i,\\theta_i)$) which, in turn, implies $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$.''", "corrected": "``If, for a set $J_i$ of units fixed before the randomization, \\hl{the assignment} $T_i$ \\hl{is independent of the vector} $\\hl{(T_j)_{j\\in J_i}}$, and $\\theta_i$ is a function of the elements of the set $\\hl{\\{Y_j:j\\in J_i\\}}$, of fixed pretreatment quantities and of no other random variables, then $T_i\\perp\\!\\!\\!\\perp\\theta_i$. It follows that $T_i\\perp\\!\\!\\!\\perp f(\\mathbf{X}_i,\\theta_i)$ and therefore $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$.''\n\nFootnote 5: ``More formally, let\n\\[\nf(\\mathbf{X}_i,\\theta_i)=f\\left(\\mathbf{X}_i,g\\left(\\mathbf{Z},\\hl{(Y_j)_{j\\in J_i}}\\right)\\right)=f\\left(\\mathbf{X}_i,g\\left(\\mathbf{Z},\\hl{(h(Y_{0j},Y_{1j};T_j))_{j\\in J_i}}\\right)\\right),\n\\]\nwhere $\\mathbf{Z}$ is a matrix of pretreatment covariates, $g(.)$ is an arbitrary function (e.g., the least squares fit), and $h(.)$ is the function implied by eq. [1]. Since only the $T_j$ are random variables, the random variable $f(\\mathbf{X}_i,\\theta_i)$ equals some function $\\hl{f'((T_j)_{j\\in J_i})}$. \\hl{Independence of} $T_i$ \\hl{and the vector} $(T_j)_{j\\in J_i}$ implies $T_i\\perp\\!\\!\\!\\perp f'((T_j)_{j\\in J_i})$ (equivalently $T_i\\perp\\!\\!\\!\\perp f(\\mathbf{X}_i,\\theta_i)$) which, in turn, implies $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$.''\n\nThe premise can be replaced by a direct check that $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$ for all $i$. It also suffices that $T_i$ be independent of the vector of observed outcomes $(Y_j)_{j\\in J_i}$.", "evidence": "This correction rests on an exact counterexample, computed in rational arithmetic, and on a derivation of the corrected condition.\n\n\\emph{Counterexample.} Let $N=6$, and let the randomization $(T_1,\\ldots,T_6)$ take the four values $010101$, $011010$, $100110$, $101001$ with probability $1/4$ each. Equivalently, $T=(A,1-A,B,1-B,C,1-C)$, where $A$ and $B$ are independent fair bits and $C=A+B-2AB$. Every unit has $\\pi_{1i}=\\pi_{0i}=1/2$. For two units in different pairs among $\\{1,2\\}$, $\\{3,4\\}$, $\\{5,6\\}$, each of the four joint outcomes of their assignments has probability $1/4$, and the two assignments are independent. The printed set then contains $Y_3,Y_4,Y_5,Y_6$ for units 1 and 2, $Y_1,Y_2,Y_5,Y_6$ for units 3 and 4, and $Y_1,Y_2,Y_3,Y_4$ for units 5 and 6. The assignment $T_5$ is not independent of the pair $(T_1,T_3)$: $\\Pr(T_5=0,T_1=0,T_3=0)=1/4$, while $\\Pr(T_5=0)\\Pr(T_1=0,T_3=0)=1/8$.\n\nLet $Y_{0i}=0$, $Y_{1i}=1$ and $X_i=1$ for every unit, and let $f(X,\\theta)=X\\theta$. Then $Y_i=T_i$ and $\\Delta=1$. Let $Q(u,v)=u+v-2uv$, and fix the rules\n\\[\n\\begin{aligned}\n\\theta_1&=Q(Y_3,Y_5),&\\qquad\\theta_3&=Q(Y_1,Y_5),&\\qquad\\theta_5&=Q(Y_1,Y_3),\\\\\n\\theta_2&=1-Q(Y_3,Y_5),&\\qquad\\theta_4&=1-Q(Y_1,Y_5),&\\qquad\\theta_6&=1-Q(Y_1,Y_3).\n\\end{aligned}\n\\]\nEach rule uses two outcomes of the printed set for its unit and no other random variable. Since $Q(B,C)=A$, $Q(A,C)=B$ and $Q(A,B)=C$, I have $f(X_i,\\theta_i)=\\theta_i=T_i$ for every unit in every randomization. Hence $\\operatorname{Cov}(T_i,f(X_i,\\theta_i))=1/2-1/4=1/4$ for all $i$. Substituting $\\pi_{1i}=\\pi_{0i}=1/2$ and $f(X_i,\\theta_i)=T_i$ into [13] and [14], I obtain\n\\[\n\\widehat{Y_1^{T*}}=\\sum_{i=1}^{6}\\left(2T_i-2T_i^2+T_i\\right)=3,\\qquad\\widehat{Y_0^{T*}}=\\sum_{i=1}^{6}\\left(-2(1-T_i)T_i+T_i\\right)=3,\n\\]\nso that $\\widehat{\\Delta}_G=(3-3)/6=0$ in every randomization, and the bias is $-1$. By equation [19], the bias is again $-(1/6)\\cdot 6\\cdot(1/4)\\cdot(2+2)=-1$. The Horvitz--Thompson estimate equals $1$ in every randomization.\n\\begin{audittable}{llrrrr}{The four randomizations of the design with six units, each with probability $1/4$. The true totals are $Y_1^T=6$ and $Y_0^T=0$, and $\\Delta=1$.}\nRandomization & Predictions $f(X_i,\\theta_i)$, $i=1,\\ldots,6$ & $\\widehat{Y_1^{T*}}$ & $\\widehat{Y_0^{T*}}$ & $\\widehat{\\Delta}_G$ & $\\widehat{\\Delta}_{HT}$ \\\\\n$010101$ & $(0,1,0,1,0,1)$ & $3$ & $3$ & $0$ & $1$ \\\\\n$011010$ & $(0,1,1,0,1,0)$ & $3$ & $3$ & $0$ & $1$ \\\\\n$100110$ & $(1,0,0,1,1,0)$ & $3$ & $3$ & $0$ & $1$ \\\\\n$101001$ & $(1,0,1,0,0,1)$ & $3$ & $3$ & $0$ & $1$ \\\\\n\\end{audittable}\n\n\\emph{Derivation of the corrected condition.} Fix a set $J_i$ of units, and let $V_i=(T_j)_{j\\in J_i}$. Suppose that $T_i$ is independent of $V_i$ and that $\\theta_i$ depends only on $(Y_j)_{j\\in J_i}$ and on fixed quantities. Since the potential outcomes are fixed, equation [1] implies that $f(\\mathbf{X}_i,\\theta_i)=H_i(V_i)$ for a fixed function $H_i$. For $a\\in\\{0,1\\}$ and any set $D$ of values,\n\\[\n\\begin{aligned}\n\\Pr(T_i=a,\\,H_i(V_i)\\in D)&=\\sum_{v:H_i(v)\\in D}\\Pr(T_i=a,\\,V_i=v)\\\\\n&=\\Pr(T_i=a)\\sum_{v:H_i(v)\\in D}\\Pr(V_i=v)\\\\\n&=\\Pr(T_i=a)\\Pr(H_i(V_i)\\in D).\n\\end{aligned}\n\\]\nThe sums are finite. Hence $T_i$ and $f(\\mathbf{X}_i,\\theta_i)$ are independent, and their covariance is zero. Equation [19] then implies that $\\widehat{\\Delta}_G$ is unbiased.", "problem": "The set $\\{Y_j:T_i\\perp\\!\\!\\!\\perp T_j\\}$ is defined by the independence of $T_i$ from each $T_j$ separately. A parameter $\\theta_i$ fitted from several of these outcomes is a function of several assignments jointly, and $T_i$ can be independent of each of them and still depend on the vector they form. The conclusion $T_i\\perp\\!\\!\\!\\perp\\theta_i$ does not follow from the stated premise, and neither does $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$. Footnote 5 treats a fit that uses one outcome $Y_j$, in which case $\\theta_i$ is a function of $T_j$ alone, and for that case its argument is correct. The reduction to that case is not without loss of generality when the fit uses several outcomes.\n\nWhen the premise holds only pairwise, the generalized difference estimator can be biased. In the design with six units described below, every randomization has three treated units and three controls, every outcome used in a fit belongs to the printed set, the estimate is $0$ in every randomization, and the average treatment effect is $1$.", "what_survives": "The Horvitz--Thompson estimator, the definitions [13]--[16], the bias identity [19], and the sufficient condition $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$ with its proof in Appendix B are unaffected. The two fitting examples in Section 5.1 satisfy the corrected condition. These are designs in which units are assigned independently and the fit for unit $i$ leaves out unit $i$, and designs in which blocks are randomized independently and the fit leaves out the block of unit $i$, as in Section 7. Under the sharp null the observed outcomes are fixed, and so is any fit from them. A prediction that is additive in the outcomes with fixed coefficients, $f(\\mathbf{X}_i,\\theta_i)=a_i+\\sum_jb_{ij}Y_j$, has zero covariance with $T_i$ under the printed pairwise rule, because $\\operatorname{Cov}(T_i,Y_j)=(Y_{1j}-Y_{0j})\\operatorname{Cov}(T_i,T_j)=0$. What changes is the general rule for an arbitrary fit: the stronger premise gives a valid guarantee of unbiasedness and excludes designs such as the one above."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-E001", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop showing equation 29, the three lines of text that follow it, equation 30 and the line after it. A red box marks the clause 'then pi-prime-1k0l is greater than zero for all l not equal to k', with footnote mark 6, in the sentence that begins 'Since, for all pairs k, l such that k is not equal to l, the probability that T-prime-k differs from T-prime-l is positive'.", "caption": "Section 6.3, p. 145 (PDF p. 11), from equation [29] to equation [30]. The marked clause infers $\\pi'_{1k0l}>0$ for all $l\\ne k$ from the fact that distinct cluster indicators differ with positive probability.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E002.png"}, {"alt": "Crop of the three lines that introduce the covariance estimator, equation 34, and the two lines after it, which refer to Appendix E for the bound on the expectation of the estimator and say that the estimator is unbiased under the sharp null. A red box marks the denominator pi-prime-1k0l of the first fraction in equation 34.", "caption": "Equation [34], p. 146 (PDF p. 12). The marked joint probability is a denominator, and the estimator is undefined for a pair $k\\ne l$ with $\\pi'_{1k0l}=0$.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E002-d.png"}, {"alt": "Crop of the three rows on the right-hand side of equation 35 and the four lines of text below it. Red boxes mark the term 'minus two times T-prime-k times one minus T-prime-l, over pi-prime-1k0l plus epsilon-1k0l, and so on' in the second row; the sentence in parentheses 'epsilon-akbl is included to avoid division by zero'; the words 'conservative has been established'; and the words 'the estimator is exactly unbiased'.", "caption": "Equation [35] and the sentences after it, p. 147 (PDF p. 13). The marked cross term is set to zero by $\\varepsilon_{1k0l}$ when $\\pi'_{1k0l}=0$, and no term bounds the product it leaves out; the marked sentences state that the estimator is conservative and, under the sharp null, exactly unbiased.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E002-b.png"}, {"alt": "Crop of five lines of the paragraph on variance estimation in the review of the literature. A red box marks the words 'conservative for any experimental design,', with 'any' in italics.", "caption": "Fourth paragraph of p. 137 (PDF p. 3). The marked words state the scope claimed for the variance estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E002-c.png"}], "formal_release": null, "formal_statements": [{"id": "P026-T1", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T1"}, {"id": "P026-T2", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T2"}, {"id": "P026-T3", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T3"}, {"id": "P026-T8", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T8"}, {"id": "P026-T17", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T17"}, {"id": "P026-T24", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T24"}], "headline": "Two distinct clusters need not have $\\pi'_{1k0l}>0$; when $\\pi'_{1k0l}=0$ for some $k\\ne l$, the covariance estimator [34] is undefined and the variance estimator [35] can be biased downward, also under the sharp null. Adding one term to [35] makes it conservative for every design.", "id": "P026-E002", "location": "Section 6.3, sentence between equations [29] and [30], p. 145 (PDF p. 11); equation [34], p. 146 (PDF p. 12); equation [35], the sentences after it and footnote 9, p. 147 (PDF p. 13); Appendix E, p. 152 (PDF p. 18); p. 137 (PDF p. 3)", "pdf_page": 11, "printed_page": "145", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P026-T1", "P026-T17", "P026-T2", "P026-T24", "P026-T3", "P026-T8"], "text": {"as_printed": "p.~145: ``Since, for all pairs $k,l$ such that $k\\ne l$, $\\Pr(T'_k\\ne T'_l)>0$, \\hl{then} $\\hl{\\pi'_{1k0l}>0}$ \\hl{for all} $\\hl{l\\ne k}$.${}^{6}$ Therefore, the covariance expression may be reduced further.''\n\np.~146, equation [34]:\n\\[\n\\begin{aligned}\n\\widehat{\\operatorname{Cov}}_C\\left(\\widehat{Y_1^T},\\widehat{Y_0^T}\\right)={}&\\sum_{k=1}^{M}\\sum_{l\\ne k}\\frac{T'_k(1-T'_l)}{\\hl{\\pi'_{1k0l}}}(\\pi'_{1k0l}-\\pi'_{1k}\\pi'_{0l})\\frac{Y'_kY'_l}{\\pi'_{1k}\\pi'_{0l}}\\\\\n&-\\sum_{k=1}^{M}T'_k\\frac{(Y'_k)^2}{2\\pi'_{1k}}-\\sum_{k=1}^{M}(1-T'_k)\\frac{(Y'_k)^2}{2\\pi'_{0k}}.\n\\end{aligned}\n\\]\n\np.~147, equation [35]:\n\\[\n\\begin{aligned}\n&\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_{HT})=\\frac{1}{N^2}\\sum_{k=1}^{M}\\Bigg[T'_k\\left(\\frac{Y'_k}{\\pi'_{1k}}\\right)^2+(1-T'_k)\\left(\\frac{Y'_k}{\\pi'_{0k}}\\right)^2\\\\\n&\\quad+\\sum_{l\\ne k}\\Bigg(\\frac{T'_kT'_l}{\\pi'_{1k1l}+\\varepsilon_{1k1l}}(\\pi'_{1k1l}-\\pi'_{1k}\\pi'_{1l})\\frac{Y'_kY'_l}{\\pi'_{1k}\\pi'_{1l}}+\\frac{(1-T'_k)(1-T'_l)}{\\pi'_{0k0l}+\\varepsilon_{0k0l}}(\\pi'_{0k0l}-\\pi'_{0k}\\pi'_{0l})\\frac{Y'_kY'_l}{\\pi'_{0k}\\pi'_{0l}}\\\\\n&\\qquad\\qquad-2\\frac{T'_k(1-T'_l)}{\\pi'_{1k0l}+\\varepsilon_{1k0l}}(\\pi'_{1k0l}-\\pi'_{1k}\\pi'_{0l})\\frac{Y'_kY'_l}{\\pi'_{1k}\\pi'_{0l}}\\Bigg)\\\\\n&\\quad+\\sum_{\\forall l:\\pi'_{1k1l}=0}\\left(T'_k\\frac{(Y'_k)^2}{2\\pi'_{1k}}+T'_l\\frac{(Y'_l)^2}{2\\pi'_{1l}}\\right)+\\sum_{\\forall l:\\pi'_{0k0l}=0}\\left((1-T'_k)\\frac{(Y'_k)^2}{2\\pi'_{0k}}+(1-T'_l)\\frac{(Y'_l)^2}{2\\pi'_{0l}}\\right)\\Bigg],\n\\end{aligned}\n\\]\n``where $\\varepsilon_{akbl}=1[\\pi'_{akbl}=0]$. ($\\varepsilon_{akbl}$ is included to avoid division by zero.) The fact that $\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_{HT})$ \\hl{is conservative has been established}. But also note that when, for all $k,l$, $\\pi'_{1k1l}>0$ and $\\pi'_{0k0l}>0$, and there is no treatment effect whatsoever, \\hl{the estimator is exactly unbiased}.''\n\np.~137: ``Our article extends this prior work by proposing a new Horvitz--Thompson-based variance estimator that is \\hl{conservative for \\emph{any} experimental design}, though additional regularity conditions would be required for use in constructing confidence intervals and hypothesis tests.''", "corrected": "p.~145: ``For all pairs $k,l$ such that $k\\ne l$, $\\Pr(T'_k\\ne T'_l)=\\pi'_{1k0l}+\\pi'_{0k1l}>0$, \\hl{and at least one of} $\\pi'_{1k0l}$ \\hl{and} $\\pi'_{0k1l}$ \\hl{is positive}. \\hl{A pair} $k\\ne l$ \\hl{with} $\\pi'_{1k0l}=0$ \\hl{contributes} $-Y'_{1k}Y'_{0l}$ \\hl{in both forms of eq. [29]}. Therefore, the covariance expression may be reduced further.'' Equation [30] stands as printed.\n\nEquation [34], for every design with $\\pi'_{1k}>0$ and $\\pi'_{0k}>0$ for all $k$:\n\\[\n\\begin{aligned}\n\\widehat{\\operatorname{Cov}}_C\\left(\\widehat{Y_1^T},\\widehat{Y_0^T}\\right)={}&\\sum_{k=1}^{M}\\sum_{\\hl{\\scriptstyle l\\ne k:\\pi'_{1k0l}>0}}\\frac{T'_k(1-T'_l)}{\\pi'_{1k0l}}(\\pi'_{1k0l}-\\pi'_{1k}\\pi'_{0l})\\frac{Y'_kY'_l}{\\pi'_{1k}\\pi'_{0l}}\\\\\n&-\\sum_{k=1}^{M}T'_k\\frac{(Y'_k)^2}{2\\pi'_{1k}}-\\sum_{k=1}^{M}(1-T'_k)\\frac{(Y'_k)^2}{2\\pi'_{0k}}\\\\\n&{}\\hl{-\\frac12\\sum_{k=1}^{M}\\sum_{l\\ne k:\\pi'_{1k0l}=0}\\left(T'_k\\frac{(Y'_k)^2}{\\pi'_{1k}}+(1-T'_l)\\frac{(Y'_l)^2}{\\pi'_{0l}}\\right)}.\n\\end{aligned}\n\\]\nIts expectation is at most $\\operatorname{Cov}(\\widehat{Y_1^T},\\widehat{Y_0^T})$, and the covariance minus its expectation is\n\\[\n\\frac12\\sum_{k=1}^{M}(Y'_{1k}-Y'_{0k})^2+\\frac12\\sum_{k=1}^{M}\\sum_{l\\ne k:\\pi'_{1k0l}=0}(Y'_{1k}-Y'_{0l})^2.\n\\]\nIn Appendix E the bound [43] is replaced accordingly: for each pair $k\\ne l$ with $\\pi'_{1k0l}=0$, Young's inequality replaces $-Y'_{1k}Y'_{0l}$ in the lower bound by $\\hl{-\\left((Y'_{1k})^2+(Y'_{0l})^2\\right)/2}$. The corrected [34] estimates this bound without bias.\n\nEquation [35], with $\\varepsilon_{akbl}=1[\\pi'_{akbl}=0]$:\n\\[\n\\begin{aligned}\n&\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_{HT})=\\frac{1}{N^2}\\sum_{k=1}^{M}\\Bigg[T'_k\\left(\\frac{Y'_k}{\\pi'_{1k}}\\right)^2+(1-T'_k)\\left(\\frac{Y'_k}{\\pi'_{0k}}\\right)^2\\\\\n&\\quad+\\sum_{l\\ne k}\\Bigg(\\frac{T'_kT'_l}{\\pi'_{1k1l}+\\varepsilon_{1k1l}}(\\pi'_{1k1l}-\\pi'_{1k}\\pi'_{1l})\\frac{Y'_k}{\\pi'_{1k}}\\frac{Y'_l}{\\pi'_{1l}}+\\frac{(1-T'_k)(1-T'_l)}{\\pi'_{0k0l}+\\varepsilon_{0k0l}}(\\pi'_{0k0l}-\\pi'_{0k}\\pi'_{0l})\\frac{Y'_k}{\\pi'_{0k}}\\frac{Y'_l}{\\pi'_{0l}}\\\\\n&\\qquad\\qquad-2\\frac{T'_k(1-T'_l)}{\\pi'_{1k0l}+\\varepsilon_{1k0l}}(\\pi'_{1k0l}-\\pi'_{1k}\\pi'_{0l})\\frac{Y'_kY'_l}{\\pi'_{1k}\\pi'_{0l}}\\Bigg)\\\\\n&\\quad+\\sum_{\\forall l:\\pi'_{1k1l}=0}\\left(T'_k\\frac{(Y'_k)^2}{2\\pi'_{1k}}+T'_l\\frac{(Y'_l)^2}{2\\pi'_{1l}}\\right)+\\sum_{\\forall l:\\pi'_{0k0l}=0}\\left((1-T'_k)\\frac{(Y'_k)^2}{2\\pi'_{0k}}+(1-T'_l)\\frac{(Y'_l)^2}{2\\pi'_{0l}}\\right)\\\\\n&\\quad\\hl{+\\sum_{l\\ne k:\\pi'_{1k0l}=0}\\left(T'_k\\frac{(Y'_k)^2}{\\pi'_{1k}}+(1-T'_l)\\frac{(Y'_l)^2}{\\pi'_{0l}}\\right)}\\Bigg].\n\\end{aligned}\n\\]\nHere $N$ is the number of units, $T'_k$ is the assignment of cluster $k$, $Y'_k$ is its observed outcome total, and $\\pi'_{1k}$, $\\pi'_{0k}$, $\\pi'_{1k1l}$, $\\pi'_{0k0l}$, $\\pi'_{1k0l}$ are the marginal and joint assignment probabilities of the clusters.\n\nStatement after [35]: for every design with fixed potential outcomes and $\\pi'_{1k}>0$, $\\pi'_{0k}>0$ for all $k$, the corrected estimator is conservative, $\\mathrm{E}[\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_{HT})]\\ge\\operatorname{Var}(\\widehat{\\Delta}_{HT})$. When, in addition, $\\pi'_{1k1l}>0$ and $\\pi'_{0k0l}>0$ for all $k,l$ and there is no treatment effect whatsoever,\n\\[\n\\mathrm{E}\\left[\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_{HT})\\right]-\\operatorname{Var}(\\widehat{\\Delta}_{HT})=\\hl{\\frac{1}{N^2}\\sum_{k=1}^{M}\\sum_{l\\ne k:\\pi'_{1k0l}=0}(Y'_k-Y'_l)^2},\n\\]\nand the estimator is exactly unbiased if and only if $\\hl{Y'_k=Y'_l}$ for every pair $k\\ne l$ with $\\hl{\\pi'_{1k0l}=0}$. This holds in particular when $\\pi'_{1k0l}>0$ for all $k\\ne l$. The same statements hold for $\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_G)$ with fixed $f$ (footnote 9), with the cluster totals of $U_i=Y_i-f(\\mathbf{X}_i,\\theta_i)$ in place of $Y'_k$.\n\nAlternatively, keep [34] and [35] as printed and add the assumption that $\\pi'_{1k0l}>0$ for all $k\\ne l$. The covariance and HT variance guarantees after [34] and [35], Appendix E's expectation argument for [34], and footnote 9's sharp-null claim for fixed $f$ are then restored, and ``conservative for any experimental design'' on p.~137 becomes ``conservative for any experimental design with $\\pi'_{1k0l}>0$ for all $k\\ne l$''.", "evidence": "This correction rests on an exact counterexample, computed in rational arithmetic, and on a derivation of the bias identity and of the corrected estimator.\n\n\\emph{Counterexample.} Let $N=M=6$, with each unit its own cluster, and let all twelve potential outcomes equal $1$, so that the sharp null holds. Let the randomization be drawn uniformly from the fourteen randomizations such that exactly three units are treated and $T'_1\\le T'_2$. The marginal probabilities of treatment are $(2/7,5/7,1/2,1/2,1/2,1/2)$. Every joint probability $\\pi'_{1k1l}$ and $\\pi'_{0k0l}$ is positive, the smallest being $1/14$, and every two indicators differ with probability at least $3/7$. The probability $\\Pr(T'_1=1,T'_2=0)$ is zero, and it is the only $\\pi'_{1k0l}$ with $k\\ne l$ that vanishes. All hypotheses of the printed statements hold. Enumerating the fourteen randomizations, I find\n\\[\n\\operatorname{Var}(\\widehat{\\Delta}_{HT})=\\frac{9}{700},\\qquad\\mathrm{E}\\left[\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_{HT})\\right]=-\\frac{269}{6300},\n\\]\na bias of $-1/18=-2\\cdot 1\\cdot 1/6^2$, in agreement with the identity above. The covariance of the two estimated totals is $27/175$, and [34] with its undefined term omitted has expectation $202/175$, which exceeds the covariance by $1$. With the added term the expectation of the variance estimator is $9/700$.\n\\begin{audittable}{lrrrrrr}{The design with six units under the sharp null with all outcomes equal to $1$. The statistics are constant within each value of $(T'_1,T'_2)$. The true variance is $9/700$.}\n$(T'_1,T'_2)$ & Randomizations & Probability & $\\widehat{\\Delta}_{HT}$ & [35] as printed & Added term & [35] corrected \\\\\n$(0,0)$ & $4$ & $2/7$ & $-3/20$ & $-619/3600$ & $7/72$ & $-269/3600$ \\\\\n$(0,1)$ & $6$ & $3/7$ & $0$ & $7/54$ & $0$ & $7/54$ \\\\\n$(1,1)$ & $4$ & $2/7$ & $3/20$ & $-619/3600$ & $7/72$ & $-269/3600$ \\\\\nExpectation & $14$ & $1$ & $0$ & $-269/6300$ & $1/18$ & $9/700$ \\\\\n\\end{audittable}\nThe corrected estimator is not unbiased under the sharp null in general. In the same design with both potential outcomes equal to $(1,2,1,1,1,1)$, the variance is $2759/12600$, the expectation of [35] as printed is $151/1400$, and the expectation of the corrected estimator is $3109/12600$. Its bias is $1/36=(1-2)^2/6^2$. Under complete randomization of three of six units every $\\pi'_{1k0l}$ with $k\\ne l$ equals $3/10$, and with sharp-null outcomes $(1,2,3,-1,0,4)$ the variance and the expectation of [35] both equal $7/3$.\n\n\\emph{Derivation.} I take expectations in $N^2$ times [35] term by term and compare with $N^2\\operatorname{Var}(\\widehat{\\Delta}_{HT})$, whose off-diagonal part is the sum over ordered pairs $k\\ne l$ of\n\\[\n\\left(\\frac{\\pi'_{1k1l}}{\\pi'_{1k}\\pi'_{1l}}-1\\right)x_kx_l+\\left(\\frac{\\pi'_{0k0l}}{\\pi'_{0k}\\pi'_{0l}}-1\\right)z_kz_l-2\\left(\\frac{\\pi'_{1k0l}}{\\pi'_{1k}\\pi'_{0l}}-1\\right)x_kz_l.\n\\]\nThe diagonal terms of [35] have expectation $x_k^2/\\pi'_{1k}+z_k^2/\\pi'_{0k}$, which exceeds the diagonal part of the variance by $(x_k-z_k)^2$. A product term with a positive joint probability is unbiased for the corresponding term of the variance. For $(k,l)\\in Z_{11}$ the variance has the term $-x_kx_l$, while [35] has a product term equal to zero and a bounding term with expectation $(x_k^2+x_l^2)/2$. The difference is $(x_k+x_l)^2/2$, and the same holds for $Z_{00}$ with $z$ in place of $x$. For $(k,l)\\in Z_{10}$ the variance has the term $+2x_kz_l$ and [35] has zero. Summing these contributions, I obtain the identity stated above. The added term has expectation $N^{-2}\\sum_{Z_{10}}(x_k^2+z_l^2)$, and $x_k^2+z_l^2-2x_kz_l=(x_k-z_l)^2$. For the corrected estimator the last term of the identity becomes $+\\sum_{Z_{10}}(x_k-z_l)^2$, and every term is nonnegative. Under the sharp null, with $x_k=z_k=Y'_k$ and $Z_{11}$, $Z_{00}$ empty, the bias reduces to $N^{-2}\\sum_{Z_{10}}(Y'_k-Y'_l)^2$.", "problem": "For binary indicators, $\\Pr(T'_k\\ne T'_l)=\\pi'_{1k0l}+\\pi'_{0k1l}$. Since clusters $k$ and $l$ are distinct, at least one of the two probabilities on the right is positive, and not necessarily both. If cluster $k$ can be treated only when cluster $l$ is treated, then $\\pi'_{1k0l}=0$ although $k\\ne l$. The paper assumes that the marginal probabilities are positive. It does not assume that they are equal for the two clusters, in which case both probabilities would be positive.\n\nThree statements rest on the deduction. First, the estimator [34] divides by $\\pi'_{1k0l}$ and is undefined for such a pair. Second, equation [35] sets the corresponding cross term to zero through $\\varepsilon_{1k0l}$ and adds nothing in its place. The variance of $\\widehat{\\Delta}_{HT}$ contains the term $2Y'_{1k}Y'_{0l}/N^2$ for that pair, and [35] neither estimates nor bounds it, so that [35] is not conservative in general. Third, under the sharp null with $\\pi'_{1k1l}>0$ and $\\pi'_{0k0l}>0$ for all $k,l$, [35] is not unbiased in general. Appendix E is affected in the same way, since the bound [43] contains the products $Y'_{1k}Y'_{0l}$ for such pairs and these cannot be estimated by weighting with inverse probabilities. The same holds for footnote 9, already for $f\\equiv 0$, and for the statements of the abstract, of p.~137 and of the discussion that the variance estimation is conservative.\n\nThe bias of [35] is as follows. Let $x_k=Y'_{1k}$ and $z_k=Y'_{0k}$, and let $Z_{11}$, $Z_{00}$ and $Z_{10}$ be the sets of ordered pairs $(k,l)$ with $k\\ne l$ such that $\\pi'_{1k1l}=0$, $\\pi'_{0k0l}=0$ and $\\pi'_{1k0l}=0$, respectively. Then\n\\[\nN^2\\left\\{\\mathrm{E}\\left[\\widehat{\\operatorname{Var}}_C(\\widehat{\\Delta}_{HT})\\right]-\\operatorname{Var}(\\widehat{\\Delta}_{HT})\\right\\}=\\sum_{k=1}^{M}(x_k-z_k)^2+\\frac12\\sum_{Z_{11}}(x_k+x_l)^2+\\frac12\\sum_{Z_{00}}(z_k+z_l)^2-2\\sum_{Z_{10}}x_kz_l.\n\\]\nThe first three terms are nonnegative. The last can be negative when the outcomes are positive, and it can outweigh the others. In the design with six units described below the sharp null holds, the variance is $9/700$, and the expectation of [35] is $-269/6300$.\n\nThere are two ways to correct this, and each has a cost. The first is to add a term to the estimator, given below. The corrected estimator is conservative for every design with positive marginal probabilities, but under the sharp null with positive same-arm joint probabilities it is unbiased only when the cluster totals agree across every pair with $\\pi'_{1k0l}=0$. The second is to keep [34] and [35] and to assume that $\\pi'_{1k0l}>0$ for all $k\\ne l$. The printed statements then hold, for a smaller class of designs that excludes the one below.", "what_survives": "The population variance and covariance identities [20]--[30] are unaffected by this error. In particular [30] is correct, because a pair $k\\ne l$ with $\\pi'_{1k0l}=0$ contributes $-Y'_{1k}Y'_{0l}$ to either form of the covariance. Only the sentence that justifies it changes. The Young bound [43] and the estimators [32] and [33] of the variances of the two estimated totals are unaffected, and so are the point estimators and their unbiasedness. In every design with $\\pi'_{1k0l}>0$ for all $k\\ne l$, the covariance and HT variance guarantees of [34] and [35] hold. The example of Section 7 is such a design: its within-block probabilities $\\pi'_{1k0l}$ are $1/3$ and $4/15$ and its cross-block probabilities are $1/3$ and $1/6$, and the added term is zero there. Three things change. The estimator [35] gains a term in designs where one cluster can be treated only if another is. The statement that the variance estimator is conservative for any design holds for the corrected estimator and not for [35] as printed. Exact unbiasedness under the sharp null requires a further condition."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-E002", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 2, which has eight columns of estimator properties: IPW, FE, FE (Cov.), RE (Cov.), HT, G (Prior), G (Linear) and G (Logit), and ten rows. Red boxes mark the whole columns FE (Cov.), G (Linear) and G (Logit), whose SE entries are 0.191, 0.162 and 0.170.", "caption": "Table 2, p. 149 (PDF p. 15). The three marked columns do not follow from the population of Table 1 and the estimators of Section 7.1; within the two G columns the entries equal to 0.000 stand.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E003.png"}, {"alt": "Crop of Table 1, which lists for each of 16 units its block, cluster, two potential outcomes, the covariate X-i and the probability of treatment. A red box marks the column X-i, with the values 4, 0, 4, 1, 4, 2, 4, 1, 2, 5, 4, 1, 4, 2, 2, 3.", "caption": "Table 1, p. 147 (PDF p. 13). The marked column is the covariate from which the three columns of Table 2 were recomputed.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E003-b.png"}, {"alt": "Dot plot with one row for each of seven estimators and a horizontal axis from 0 to .46 for the standard error of the ATE estimator. For IPW, FE, HT and G (Prior) the open circle for the printed value and the filled diamond for the recomputed value nearly coincide, at about .28, .28, .43 and .30. For FE (Cov.) the printed value is .191 and the recomputed value .279; for G (Linear), .162 and .246; for G (Logit), .170 and .247.", "caption": "The SE row of Table 2 as printed and as recomputed from Table 1 over the 90 randomizations, for the seven estimators recomputed. The four columns that do not use $X$ agree with the table; FE (Cov.), G (Linear) and G (Logit) do not. RE (Cov.) is not shown because its recalculation is not settled.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E003-table2-se.png"}], "formal_release": null, "formal_statements": [{"id": "P026-T6", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T6"}, {"id": "P026-T7", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T7"}, {"id": "P026-T14", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T14"}, {"id": "P026-T19", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T19"}], "headline": "Three columns of Table 2 do not follow from the population printed in Table 1: for that population the standard errors of FE (Cov.), G (Linear) and G (Logit) are .279, .246 and .247, not .191, .162 and .170, and the gain in precision from covariate adjustment is smaller than the table shows.", "id": "P026-E003", "location": "Table 2, columns FE (Cov.), G (Linear) and G (Logit), p. 149 (PDF p. 15); compare Table 1, column $X_i$, p. 147 (PDF p. 13), and Section 7.1, p. 148 (PDF p. 14)", "pdf_page": 15, "printed_page": "149", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P026-T14", "P026-T19", "P026-T6", "P026-T7"], "text": {"as_printed": "Table 1 gives $X_i=(4,0,4,1,4,2,4,1,2,5,4,1,4,2,2,3)$ for units 1 to 16.\n\\begin{audittable}{lrrr}{Table 2, p.~149: three of the eight columns as printed.}\n & FE (Cov.) & G (Linear) & G (Logit) \\\\\n$\\Delta$ & 0.000 & 0.000 & 0.000 \\\\\n$\\mathrm{E}[\\widehat{\\Delta}]$ & $\\hl{-0.012}$ & 0.000 & 0.000 \\\\\nBias & $\\hl{-0.012}$ & 0.000 & 0.000 \\\\\nSE & \\hl{0.191} & \\hl{0.162} & \\hl{0.170} \\\\\nRMSE & \\hl{0.191} & \\hl{0.162} & \\hl{0.170} \\\\\nVar & \\hl{0.036} & \\hl{0.026} & \\hl{0.029} \\\\\n$\\mathrm{E}[\\widehat{\\operatorname{Var}}]$ & \\hl{0.031} & \\hl{0.026} & \\hl{0.029} \\\\\nBias & $\\hl{-0.005}$ & 0.000 & 0.000 \\\\\nSE & \\hl{0.011} & \\hl{0.007} & \\hl{0.007} \\\\\nRMSE & \\hl{0.012} & \\hl{0.007} & \\hl{0.007} \\\\\n\\end{audittable}", "corrected": "For the population printed in Table 1, the three columns of Table 2 read as follows.\n\\begin{audittable}{lrrr}{Table 2, three columns recomputed from Table 1.}\n & FE (Cov.) & G (Linear) & G (Logit) \\\\\n$\\Delta$ & 0.000 & 0.000 & 0.000 \\\\\n$\\mathrm{E}[\\widehat{\\Delta}]$ & $\\hl{-0.014}$ & 0.000 & 0.000 \\\\\nBias & $\\hl{-0.014}$ & 0.000 & 0.000 \\\\\nSE & \\hl{0.279} & \\hl{0.246} & \\hl{0.247} \\\\\nRMSE & \\hl{0.279} & \\hl{0.246} & \\hl{0.247} \\\\\nVar & \\hl{0.078} & \\hl{0.060} & \\hl{0.061} \\\\\n$\\mathrm{E}[\\widehat{\\operatorname{Var}}]$ & \\hl{0.072} & \\hl{0.060} & \\hl{0.061} \\\\\nBias & $\\hl{-0.006}$ & 0.000 & 0.000 \\\\\nSE & \\hl{0.019} & \\hl{0.012} & \\hl{0.013} \\\\\nRMSE & \\hl{0.020} & \\hl{0.012} & \\hl{0.013} \\\\\n\\end{audittable}\nThe \\hl{FE (Cov.)} variance-estimator rows here use the cluster-robust estimator with the factor $G(N-1)/((G-1)(N-K))$, for $G=10$ clusters, $N=16$ units and $K$ regressors, \\hl{including the intercept}, and this convention should be stated. In Section 7.2 the comparison becomes: for this population, G (Linear) and G (Logit) are unbiased and have lower RMSE than IPW, FE and FE (Cov.). The RE (Cov.) column, and the statement that the variance estimators of all regression-based estimators are negatively biased, should be reconciled with the original inputs and settings. If Table 1 and not Table 2 carries the misprint, the correction belongs in the column $X_i$ of Table 1, and it requires the original inputs.", "evidence": "This correction rests on a recalculation over all 90 randomizations. For G (Linear) the recalculation is exact. Least squares on the opposite block gives the predictions $1/4+X_i/8$ for the units of block 1 and $70/93+X_i/31$ for the units of block 2. The estimator has mean $0$ and variance $38307/634880\\approx .060337$, and the variance estimator [35] has mean $38307/634880$ and standard error $.012370$. For G (Logit) both logistic fits have unique finite maximum-likelihood estimates, and the moments are computed to high precision. The mean and standard error of FE (Cov.) do not depend on the variance-estimator convention. I computed its variance-estimator rows using the cluster-robust estimator with the finite-sample factor $(10/9)\\cdot 15/(16-K)$, where $K$ is the number of regressors, the intercept included. The paper does not state this factor. With it, the recalculation reproduces the IPW and FE columns. A second, independent calculation agrees with these values.\n\\begin{audittable}{lrrrrrr}{Table 2 as printed and as recomputed from the population of Table 1, for the three columns that differ. Each column of the table is followed by its recomputed values.}\n & FE (Cov.) & recomputed & G (Linear) & recomputed & G (Logit) & recomputed \\\\\n$\\Delta$ & .000 & .000 & .000 & .000 & .000 & .000 \\\\\n$\\mathrm{E}[\\widehat{\\Delta}]$ & $-.012$ & $-.014$ & .000 & .000 & .000 & .000 \\\\\nBias & $-.012$ & $-.014$ & .000 & .000 & .000 & .000 \\\\\nSE & .191 & .279 & .162 & .246 & .170 & .247 \\\\\nRMSE & .191 & .279 & .162 & .246 & .170 & .247 \\\\\nVar & .036 & .078 & .026 & .060 & .029 & .061 \\\\\n$\\mathrm{E}[\\widehat{\\operatorname{Var}}]$ & .031 & .072 & .026 & .060 & .029 & .061 \\\\\nBias & $-.005$ & $-.006$ & .000 & .000 & .000 & .000 \\\\\nSE & .011 & .019 & .007 & .012 & .007 & .013 \\\\\nRMSE & .012 & .020 & .007 & .012 & .007 & .013 \\\\\n\\end{audittable}", "problem": "Table 2 reports, for the population of Table 1 and the estimators of Section 7.1, the moments of each estimator over the 90 possible randomizations. When they are recomputed from the printed population, three of the four columns that use the covariate $X$ differ from the table. For G (Linear) the standard error is $.246$ and the variance $.060$, where the table has $.162$ and $.026$. For G (Logit) the standard error is $.247$, and for FE (Cov.) it is $.279$. The variances differ by a factor of about two, which rounding cannot explain. All 40 entries of the four columns that do not use $X$ (IPW, FE, HT and G (Prior)) agree with the table to the third decimal.\n\nThe example overstates the gain from covariate adjustment. Relative to IPW, the standard error of G (Linear) is lower by about 11\\%, and not by the 41\\% that the printed $.162$ and $.276$ imply.\n\nI have not established which of the two tables is misprinted. The commands and inputs behind Table 2 are not among the replication files. If $X$ is changed to $2,3,3,3$ for units 2, 4, 8 and 12, where Table 1 has $0,1,1,1$, the printed columns are nearly reproduced, but one entry is not (the RMSE of the FE (Cov.) variance estimator is $.01255$, which rounds to $.013$ against the printed $.012$), and I have found no evidence that these values were used. The RE (Cov.) column is not settled either. The version of \\texttt{lme4} that the paper cites (0.999375-37) could not be run, and version 1.1-37 does not reproduce the column. With REML the estimator has mean $-.008$ and standard error $.303$, and its variance estimator has bias $+.002$. With maximum likelihood the variance estimator has negative bias. Whether ``all of the regression-based estimators, including variance estimators, are negatively biased'' (Section 7.2) holds for the variance estimator of the random-effects fit remains open.", "what_survives": "The four columns that do not use $X$ stand as printed. For the printed population the estimators G (Linear) and G (Logit) remain unbiased, and so do their variance estimators. They still have lower RMSE than IPW, FE and FE (Cov.), by a smaller margin: $.246$ and $.247$ against $.277$, $.283$ and $.279$. They also have lower RMSE than the random-effects fits obtained with version 1.1-37 of \\texttt{lme4}. The comparison with the RE (Cov.) column as printed is not settled. No formula, proof or theoretical result uses the entries of Table 2."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-E003", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the sentence 'Cov-C of the two estimated totals may be estimated without bias', the two-row display that follows it, and three lines of the proof. One red box encloses the first row of the display, whose left side reads 'Var-hat-C of Y-hat-1-T' and whose two terms contain T-prime-k and the product T-prime-k T-prime-l with pi-prime-1k1l. A second red box marks the identity 'E of T-prime-k times one minus T-prime-l equals pi-prime-1k0l' at the start of the proof.", "caption": "Appendix E, p. 152 (PDF p. 18). The first row of the display, marked, has the label and the two leading terms of the variance estimator [32]; the identity marked below it, on which the proof relies, concerns the factor $T'_k(1-T'_l)$, which the display does not contain.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P026-T8", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T8"}, {"id": "P026-T9", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T9"}], "headline": "The display in Appendix E that should state the estimator of the covariance bound carries the label and the first two terms of the variance estimator for the treatment total; it should be equation [34].", "id": "P026-E004", "location": "Appendix E, unnumbered display after “$\\operatorname{Cov}_C(\\widehat{Y_1^T},\\widehat{Y_0^T})$ may be estimated without bias:”, p. 152 (PDF p. 18)", "pdf_page": 18, "printed_page": "152", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P026-T8", "P026-T9"], "text": {"as_printed": "``$\\operatorname{Cov}_C\\left(\\widehat{Y_1^T},\\widehat{Y_0^T}\\right)$ may be estimated without bias:''\n\\[\n\\begin{aligned}\n{}\\hl{\\widehat{\\operatorname{Var}}_C(\\widehat{Y_1^T})}={}&{}\\hl{\\sum_{k=1}^{M}T'_k(1-\\pi'_{1k})\\left(\\frac{Y'_k}{\\pi'_{1k}}\\right)^2}+\\hl{\\sum_{k=1}^{M}\\sum_{l\\ne k:\\pi'_{1k1l}>0}\\frac{T'_kT'_l}{\\pi'_{1k1l}}(\\pi'_{1k1l}-\\pi'_{1k}\\pi'_{1l})\\frac{Y'_k}{\\pi'_{1k}}\\frac{Y'_l}{\\pi'_{1l}}}\\\\\n&-\\sum_{k=1}^{M}T'_k\\frac{(Y'_k)^2}{2\\pi'_{1k}}-\\sum_{k=1}^{M}(1-T'_k)\\frac{(Y'_k)^2}{2\\pi'_{0k}},\n\\end{aligned}\n\\]\n``by $\\mathrm{E}[T'_k(1-T'_l)]=\\pi'_{1k0l}$, $\\mathrm{E}[T'_k]=\\pi'_{1k}$, $\\mathrm{E}[1-T'_k]=\\pi'_{0k}$ and eq. [1].''", "corrected": "Suppose that $\\pi'_{1k0l}>0$ for all $k\\ne l$. Then ``$\\operatorname{Cov}_C\\left(\\widehat{Y_1^T},\\widehat{Y_0^T}\\right)$ may be estimated without bias:''\n\\[\n\\begin{aligned}\n{}\\hl{\\widehat{\\operatorname{Cov}}_C\\left(\\widehat{Y_1^T},\\widehat{Y_0^T}\\right)}={}&{}\\hl{\\sum_{k=1}^{M}\\sum_{l\\ne k}\\frac{T'_k(1-T'_l)}{\\pi'_{1k0l}}(\\pi'_{1k0l}-\\pi'_{1k}\\pi'_{0l})\\frac{Y'_kY'_l}{\\pi'_{1k}\\pi'_{0l}}}\\\\\n&-\\sum_{k=1}^{M}T'_k\\frac{(Y'_k)^2}{2\\pi'_{1k}}-\\sum_{k=1}^{M}(1-T'_k)\\frac{(Y'_k)^2}{2\\pi'_{0k}},\n\\end{aligned}\n\\]\n``by $\\mathrm{E}[T'_k(1-T'_l)]=\\pi'_{1k0l}$, $\\mathrm{E}[T'_k]=\\pi'_{1k}$, $\\mathrm{E}[1-T'_k]=\\pi'_{0k}$ and eq. [1].'' This is equation [34]. The rest of the proof stands: $\\mathrm{E}[\\widehat{\\operatorname{Cov}}_C(\\widehat{Y_1^T},\\widehat{Y_0^T})]=\\operatorname{Cov}_C(\\widehat{Y_1^T},\\widehat{Y_0^T})\\le\\operatorname{Cov}(\\widehat{Y_1^T},\\widehat{Y_0^T})$, with equality when $Y'_{0k}=Y'_{1k}$ for all $k$.", "evidence": "This correction rests on an exact counterexample to the display as printed and on a derivation for the corrected display.\n\n\\emph{Counterexample.} Let $N=M=4$, with each unit its own cluster, and let two of the four units be treated, each of the six subsets having probability $1/6$. Let $Y'_{1k}=Y'_{0k}$ take the values $1,2,4,7$. Then $\\pi'_{1k}=\\pi'_{0k}=1/2$, and for $k\\ne l$ the joint probabilities are $\\pi'_{1k1l}=\\pi'_{0k0l}=1/6$ and $\\pi'_{1k0l}=1/3$, all positive. If $u$ and $v$ are the two treated outcomes, the printed right side equals $2(u-v)^2-70$, and equation [34] equals the product of the treated and the control outcome sums minus $70$. The printed right side has expectation $-42$. The covariance of the two estimated totals, the bound [43] and the expectation of [34] all equal $-28$.\n\\begin{audittable}{lrrrr}{The six randomizations of the design with four units, each with probability $1/6$, under the sharp null with outcomes $1,2,4,7$.}\nTreated units & $\\widehat{Y_1^T}$ & $\\widehat{Y_0^T}$ & Display as printed & Equation [34] \\\\\n$\\{1,2\\}$ & $6$ & $22$ & $-68$ & $-37$ \\\\\n$\\{1,3\\}$ & $10$ & $18$ & $-52$ & $-25$ \\\\\n$\\{1,4\\}$ & $16$ & $12$ & $2$ & $-22$ \\\\\n$\\{2,3\\}$ & $12$ & $16$ & $-62$ & $-22$ \\\\\n$\\{2,4\\}$ & $18$ & $10$ & $-20$ & $-25$ \\\\\n$\\{3,4\\}$ & $22$ & $6$ & $-52$ & $-37$ \\\\\nExpectation & $14$ & $14$ & $-42$ & $-28$ \\\\\n\\end{audittable}\n\n\\emph{Derivation.} Suppose that $\\pi'_{1k0l}>0$ for all $k\\ne l$. Since $T'_k(1-T'_l)Y'_kY'_l=T'_k(1-T'_l)Y'_{1k}Y'_{0l}$, the cross term of [34] has expectation $(\\pi'_{1k0l}-\\pi'_{1k}\\pi'_{0l})Y'_{1k}Y'_{0l}/(\\pi'_{1k}\\pi'_{0l})$. The two remaining sums have expectations $\\sum_k(Y'_{1k})^2/2$ and $\\sum_k(Y'_{0k})^2/2$. Summing, I obtain $\\mathrm{E}[\\widehat{\\operatorname{Cov}}_C]=\\operatorname{Cov}_C$, the right side of [43]. Equations [30] and [43] imply that\n\\[\n\\operatorname{Cov}\\left(\\widehat{Y_1^T},\\widehat{Y_0^T}\\right)-\\operatorname{Cov}_C\\left(\\widehat{Y_1^T},\\widehat{Y_0^T}\\right)=\\frac12\\sum_{k=1}^{M}(Y'_{1k}-Y'_{0k})^2\\ge 0,\n\\]\nwith equality under the sharp null.", "problem": "The sentence before the display and the proof after it concern the covariance bound $\\operatorname{Cov}_C$ of equation [43]. The left side of the display is the label of the variance estimator for the treatment total, and its first row has the two leading terms of that estimator, equation [32]. Only the last two sums belong to the covariance estimator. The expectation of the expression as printed is not $\\operatorname{Cov}_C$, and the identity $\\mathrm{E}[T'_k(1-T'_l)]=\\pi'_{1k0l}$ invoked after the display concerns a factor that the display does not contain. The estimator meant is equation [34] of the main text.", "what_survives": "Equation [34] is printed correctly in the main text, and equation [35] is built from it and not from this display, so that the misprint does not reach the variance estimator. The identity [30] and the bound [43] are unaffected. In the example of Section 7 the variance of $\\widehat{\\Delta}_{HT}$ and the expectation of [35] both equal $353/1920$, which rounds to the $0.184$ of Table 2. The corrected display has the stated expectation when $\\pi'_{1k0l}>0$ for all $k\\ne l$. The case in which some $\\pi'_{1k0l}$ vanishes is the subject of \\fid{P026-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-E004", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the two-row display that rewrites the Horvitz-Thompson estimator of the treatment total, with the two lines of text above it. A red box marks the fraction 'one over pi-i' in the first sum; the second sum has 'one over pi-1i'.", "caption": "Section 6.3, p. 144 (PDF p. 10). In the first sum the marked denominator is $\\pi_i$, a symbol that the paper does not define; the next sum has $\\pi_{1i}$.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E005.png"}, {"alt": "Crop of footnote 8 with the last line of footnote 7 above it. A red box marks the reference number '[40]' after the name Imai.", "caption": "Footnote 8, p. 145 (PDF p. 11). The marked reference number should be 39.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E005-b.png"}, {"alt": "Crop of the display for the conservative variance estimator in Appendix D, with the line above it and the four lines of text below it. Red boxes mark the symbols T-k and T-l in the last row, printed without primes; the first row has T-prime-k and T-prime-l.", "caption": "Appendix D, p. 152 (PDF p. 18). In the last row the two marked indicators lack the prime that the cluster indicators carry in the first row.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E005-c.png"}, {"alt": "Crop of the three rows of equation 24. A red box marks a full stop at the end of the second row, after the fraction Y-0j over pi-0j; the third row continues with minus two times a double sum and ends with the closing parenthesis.", "caption": "Equation [24], p. 143 (PDF p. 9). The marked full stop ends the second row although the expression continues on the third.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E005-d.png"}], "formal_release": null, "formal_statements": [{"id": "P026-T10", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T10"}, {"id": "P026-T11", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T11"}, {"id": "P026-T12", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T12"}], "headline": "Four misprints: $\\pi_i$ should read $\\pi_{1i}$ in the cluster-total display, “Imai [40]” should read “Imai [39]” in footnote 8, $T_k$ and $T_l$ should read $T'_k$ and $T'_l$ in the last row of the Appendix D display, and the full stop inside equation [24] should be deleted.", "id": "P026-E005", "location": "Section 6.3, display that rewrites $\\widehat{Y^T_{1,HT}}$, p. 144 (PDF p. 10); footnote 8, p. 145 (PDF p. 11); Appendix D, last row of the estimator display, p. 152 (PDF p. 18); equation [24], second row, p. 143 (PDF p. 9)", "pdf_page": 10, "printed_page": "144", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P026-T10", "P026-T11", "P026-T12"], "text": {"as_printed": "(a) p.~144:\n\\[\n\\widehat{Y^T_{1,HT}}=\\sum_{i=1}^{N}\\frac{1}{\\hl{\\pi_i}}T_iY_i=\\sum_{k=1}^{M}\\sum_{i\\in K_k}\\frac{1}{\\pi_{1i}}T_iY_i=\\sum_{k=1}^{M}\\sum_{i\\in K_k}\\frac{1}{\\pi'_{1k}}T'_kY_i=\\sum_{k=1}^{M}\\frac{1}{\\pi'_{1k}}T'_k\\sum_{i\\in K_k}Y_i=\\sum_{k=1}^{M}\\frac{1}{\\pi'_{1k}}T'_kY'_k,\n\\]\n\n(b) p.~145, footnote 8: ``The variance estimators derived in this paper do not reduce to those proposed by Neyman [49], Imai \\hl{[40]} or Middleton and Aronow [5], due to differences in how the covariance term is approximated.''\n\n(c) p.~152, Appendix D, last row of the display for $\\widehat{\\operatorname{Var}}_C(\\widehat{Y_1^T})$:\n\\[\n+\\sum_{k=1}^{M}\\sum_{\\forall l:\\pi'_{1k1l}=0}\\left(\\hl{T_k}\\frac{(Y'_k)^2}{2\\pi'_{1k}}+\\hl{T_l}\\frac{(Y'_l)^2}{2\\pi'_{1l}}\\right),\n\\]\n\n(d) p.~143, second row of equation [24]:\n\\[\n+\\sum_{i=1}^{N}\\pi_{0i}(1-\\pi_{0i})\\left(\\frac{Y_{0i}}{\\pi_{0i}}\\right)^2+\\sum_{i=1}^{N}\\sum_{j\\ne i}(\\pi_{0i0j}-\\pi_{0i}\\pi_{0j})\\frac{Y_{0i}Y_{0j}}{\\pi_{0i}\\pi_{0j}}\\hl{.}\n\\]\nThe third row continues the expression inside the same parenthesis.", "corrected": "(a)\n\\[\n\\widehat{Y^T_{1,HT}}=\\sum_{i=1}^{N}\\frac{1}{\\hl{\\pi_{1i}}}T_iY_i=\\sum_{k=1}^{M}\\sum_{i\\in K_k}\\frac{1}{\\pi_{1i}}T_iY_i=\\cdots=\\sum_{k=1}^{M}\\frac{1}{\\pi'_{1k}}T'_kY'_k.\n\\]\n\n(b) ``\\ldots those proposed by Neyman [49], Imai \\hl{[39]} or Middleton and Aronow [5] \\ldots''\n\n(c)\n\\[\n+\\sum_{k=1}^{M}\\sum_{\\forall l:\\pi'_{1k1l}=0}\\left(\\hl{T'_k}\\frac{(Y'_k)^2}{2\\pi'_{1k}}+\\hl{T'_l}\\frac{(Y'_l)^2}{2\\pi'_{1l}}\\right),\n\\]\n\n(d) The second row of [24] ends without a full stop:\n\\[\n+\\sum_{i=1}^{N}\\pi_{0i}(1-\\pi_{0i})\\left(\\frac{Y_{0i}}{\\pi_{0i}}\\right)^2+\\sum_{i=1}^{N}\\sum_{j\\ne i}(\\pi_{0i0j}-\\pi_{0i}\\pi_{0j})\\frac{Y_{0i}}{\\pi_{0i}}\\frac{Y_{0j}}{\\pi_{0j}}\n\\]", "evidence": "This correction rests on a check of the pages against the adjacent formulas, the definitions on pp.~138 and 144, equations [6], [26] and [32], and the reference list on pp.~153--154.", "problem": "(a) The paper defines $\\pi_{1i}$ and $\\pi_{0i}$ and no $\\pi_i$, and the next expression in the chain and equation [6] have $\\pi_{1i}$. (b) Reference 40 is Hansen and Bowers. Imai's paper on variance identification is reference 39, and p.~137 cites it correctly. (c) The cluster indicators are $T'_k$. The first row of the same display and the corresponding term of equation [32] carry the primes, and the argument that follows uses $\\mathrm{E}[T'_k]=\\pi'_{1k}$. (d) The parenthesis opened in the first row of [24] closes at the end of the third row.", "what_survives": "Everything else is unaffected. The cluster-total identity, the bound of Appendix D and equation [24] are correct after these repairs."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-E005", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the paragraph on Lin and on Miratrix, Sekhon and Yu. A red box marks the word 'than' in 'a special case of the regression-based estimator than Lin proposes'.", "caption": "Paragraph on Lin, p. 137 (PDF p. 3). The marked word should read “that”.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E006.png"}, {"alt": "Crop of the four lines that end Section 5 before the heading of Section 5.1. A red box marks the word 'auxillary' in 'derived from an auxillary or prior source'.", "caption": "Last sentence before Section 5.1, p. 141 (PDF p. 7). The marked word is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E006-b.png"}, {"alt": "Crop of the heading '6 Sampling variance' and the two lines below it. A red box marks the words 'In the section,' at the start of the first line.", "caption": "First sentence of Section 6, p. 142 (PDF p. 8). The marked words should read “In this section”.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E006-c.png"}, {"alt": "Crop of the reference list from entry 16 to entry 21. Red boxes mark '17, 1995' in the Godambe entry, number 17, and '663-84' in the Horvitz and Thompson entry, number 20.", "caption": "References 16 to 21, p. 153 (PDF p. 19). In reference 17 the marked volume number is repeated and the marked year should be 1955; in reference 20 the marked page range should end at 685.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E006-d.png"}, {"alt": "Crop of reference entries 36 and 37. A red box marks the word 'post-stratication' in the title of the entry for Miratrix, Sekhon and Yu.", "caption": "References 36 and 37, p. 153 (PDF p. 19). The marked word in the title is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-E006-e.png"}], "formal_release": null, "formal_statements": [], "headline": "Three words in the text and three entries of the reference list are misprinted: “than” should read “that”, “auxillary” should read “auxiliary”, “In the section” should read “In this section”, Godambe’s article is dated 1955 and its volume number is printed twice, Horvitz and Thompson’s article ends on p. 685, and “post-stratication” should read “post-stratification”.", "id": "P026-E006", "location": "p. 137 (PDF p. 3), paragraph on Lin [9]; p. 141 (PDF p. 7), last sentence before Section 5.1; p. 142 (PDF p. 8), first sentence of Section 6; references 17, 20 and 37, p. 153 (PDF p. 19)", "pdf_page": 3, "printed_page": "137", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item (a) p.~137: ``(and, in fact, the post-stratification-based estimator may be shown to be a special case of the regression-based estimator \\hl{than} Lin proposes).''\n\\item (b) p.~141: ``The simplest way for this assumption to hold is for $\\theta_i$ to be derived from an \\hl{auxillary} or prior source, but we examine this selection process further in Section 5.1.''\n\\item (c) p.~142: ``In \\hl{the} section, we will provide expressions for the sampling variance of the HT estimator and the generalized difference estimator.''\n\\item (d) p.~153: ``17. Godambe VP. A unified theory of sampling from finite populations. J R Stat Soc Ser B Methodol \\hl{17, 1995};17(2):269--278.''\n\\item (e) p.~153: ``20. Horvitz DG, Thompson DJ. A generalization of sampling without replacement from a finite universe. J Am Stat Assoc 1952;47:663--\\hl{84}.''\n\\item (f) p.~153: ``37. Miratrix L, Sekhon J, Yu B. Adjusting treatment effect estimates by \\hl{post-stratication} in randomized experiments. J R Stat Soc Ser B Methodol, 2013;75(2):369--396.''\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item (a) ``\\ldots a special case of the regression-based estimator \\hl{that} Lin proposes).''\n\\item (b) ``\\ldots derived from an \\hl{auxiliary} or prior source \\ldots''\n\\item (c) ``In \\hl{this} section, \\ldots''\n\\item (d) ``17. Godambe VP. A unified theory of sampling from finite populations. J R Stat Soc Ser B Methodol \\hl{1955};17(2):269--278.''\n\\item (e) ``\\ldots J Am Stat Assoc 1952;47:663--\\hl{85}.''\n\\item (f) ``\\ldots Adjusting treatment effect estimates by \\hl{post-stratification} in randomized experiments \\ldots''\n\\end{itemize}", "evidence": "This correction rests on a check of the pages and on the publication records of two cited articles. The table of contents of the journal issue lists Godambe's article as Volume 17, Issue 2, July 1955, pp.~269--278. The JSTOR record of Horvitz and Thompson's article has Vol.~47, No.~260 (Dec., 1952), pp.~663--685. The paper spells ``auxiliary'' and ``post-stratification'' correctly on pp.~142 and 137.", "problem": "These are six misprints. In (d) the year should be 1955, and the volume number 17 appears twice. In (e) the article ends on p.~685. The meaning is clear in each case, and each cited work can still be identified.", "what_survives": "Everything else is unaffected, since no result depends on these words or entries."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-E006", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of equation 17 with the two lines of text above it and the words 'and, likewise,' below it. A red box encloses the expression that begins 'E, open square bracket' and runs through the third sum; no closing bracket appears inside the box or after it.", "caption": "Equation [17], p. 141 (PDF p. 7). The marked expectation opens a bracket that is not closed before the equals sign.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-N001.png"}], "formal_release": null, "formal_statements": [{"id": "P026-T13", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T13"}], "headline": "In equation [17] the bracket of the second expectation is not closed; it closes after the third sum.", "id": "P026-N001", "location": "Section 5.1, equation [17], p. 141 (PDF p. 7)", "pdf_page": 7, "printed_page": "141", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P026-T13"], "text": {"as_printed": "\\[\n\\mathrm{E}\\left[\\widehat{Y_1^{T*}}\\right]=\\mathrm{E}\\hl{\\Bigg[}\\sum_{i=1}^{N}T_i\\frac{Y_{1i}}{\\pi_{1i}}-\\sum_{i=1}^{N}T_i\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}}+\\sum_{i=1}^{N}f(\\mathbf{X}_i,\\theta_i)=Y_1^T-\\sum_{i=1}^{N}\\operatorname{Cov}\\left(T_i,\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}}\\right),\\qquad\\text{[17]}\n\\]", "corrected": "\\[\n\\mathrm{E}\\left[\\widehat{Y_1^{T*}}\\right]=\\mathrm{E}\\Bigg[\\sum_{i=1}^{N}T_i\\frac{Y_{1i}}{\\pi_{1i}}-\\sum_{i=1}^{N}T_i\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}}+\\sum_{i=1}^{N}f(\\mathbf{X}_i,\\theta_i)\\hl{\\Bigg]}=Y_1^T-\\sum_{i=1}^{N}\\operatorname{Cov}\\left(T_i,\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}}\\right).\n\\]", "evidence": "This correction rests on a check of the page and on a short derivation. Since $\\mathrm{E}[T_i]=\\pi_{1i}$, I have\n\\[\n\\mathrm{E}\\left[T_i\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}}\\right]=\\operatorname{Cov}\\left(T_i,\\frac{f(\\mathbf{X}_i,\\theta_i)}{\\pi_{1i}}\\right)+\\mathrm{E}\\left[f(\\mathbf{X}_i,\\theta_i)\\right],\n\\]\nwhether or not $f(\\mathbf{X}_i,\\theta_i)$ is random. The first sum has expectation $Y_1^T$, and the terms $\\mathrm{E}[f(\\mathbf{X}_i,\\theta_i)]$ cancel against the expectation of the third sum. Summing over the units, I obtain the right side of [17].", "problem": "The bracket that opens after the second $\\mathrm{E}$ has no closing bracket, and as printed the expectation appears to extend over the equals sign. The identity is correct when the bracket closes after the third sum.", "what_survives": "Everything else is unaffected. The sign and the denominator of the covariance term are correct as printed, and equations [18] and [19] do not change."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-N001", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the five lines that precede equation 31. A red box marks the words 'the same treatment condition.' at the end of the sentence 'This assumption is equivalent to assuming that all pairs of units have nonzero probability of being assigned to the same treatment condition'.", "caption": "Section 6.4, first paragraph on p. 146 (PDF p. 12). The marked words paraphrase the assumption stated in symbols in the sentence before.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "“The same treatment condition” means each condition: every pair must have positive probability of being jointly treated and positive probability of being jointly in control.", "id": "P026-N002", "location": "Section 6.4, first paragraph on p. 146 (PDF p. 12), second sentence, before equation [31]", "pdf_page": 12, "printed_page": "146", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``First, let us derive an unbiased estimator of $\\operatorname{Var}(\\widehat{Y_1^T})$ under the assumption that, for all pairs $k,l$, $\\pi'_{1k1l}$ and $\\pi'_{0k0l}>0$. This assumption is equivalent to assuming that all pairs of units have nonzero probability of being assigned to \\hl{the same treatment condition}.''", "corrected": "``This assumption is equivalent to assuming that every pair of units has positive probability of being \\hl{jointly assigned to treatment and positive probability of being jointly assigned to control}.''", "intended_meaning": "The assumption is the one printed in symbols: $\\pi'_{1k1l}>0$ and $\\pi'_{0k0l}>0$ for all pairs. The second sentence restates it loosely. The example that follows it, of pair-randomized studies in which the two units of a pair are never jointly treated, shows that joint assignment to each condition is meant, and equation [31] is unbiased under that assumption. Read literally, the second sentence says only that $\\Pr(T'_k=T'_l)=\\pi'_{1k1l}+\\pi'_{0k0l}>0$, which is weaker. With three clusters of one unit each, exactly one of which is treated, each with probability $1/3$, every pair of distinct clusters has $\\pi'_{0k0l}=1/3$ and $\\pi'_{1k1l}=0$. The pair is then assigned to the same condition with probability $1/3$, and the assumption fails. That the sentence speaks of pairs of units and not of clusters is not an error. Two units in different clusters have the joint probabilities of their clusters, and two units of one cluster are jointly treated and jointly in control with the positive marginal probabilities of that cluster. No formula or result changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-N002", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing in monospaced type of parts of the help source for the function estate: the description, the usage line, the item for the argument X and the item for the argument HT, with line numbers. Red boxes mark the words 'unbiased Horvitz-Thompson estimation' in the description and the words 'when HT=TRUE, invokes the Horvitz-Thompson (difference-in-totals) estimator' in the item for HT.", "caption": "Package ri, file man/estate.Rd, lines 7–12, 20–22 and 29–31, typeset from the source file with line numbers added; a row of dots stands for the lines left out. The marked words call Horvitz–Thompson estimation unbiased and describe HT=TRUE without separating the case in which X is supplied.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-N004.png"}, {"alt": "Listing in monospaced type of lines 28 to 36 of the function estate, with line numbers: the branch commented 'HT' and the branch commented 'Raj-Within'. A red box marks line 33, 'Ypre2 is assigned the fitted values of lm of Yd on X'.", "caption": "Package ri, file R/estate.R, lines 28–36, typeset from the source file with line numbers added. With HT == TRUE and X supplied, the marked line regresses Y (or Y-Ypre when Ypre is supplied) on X, using all units.", "url": "https://pmaronow.github.io/llm-audit/works/P026/exhibits/P026-N004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P026-T15", "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-T15"}], "headline": "The help page of estate should say that with HT=TRUE and covariates X supplied the function fits a regression using all units, and that the resulting estimator is not unbiased in general.", "id": "P026-N004", "location": "R package ri (version 0.9), help page of estate (man/estate.Rd, description and argument HT) and R/estate.R, lines 31–36; no passage of the article", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P026-T15"], "text": {"as_printed": "Help page of \\texttt{estate}, description: ``Function for estimating the average treatment effect (ATE). Permits regression adjustment for covariates, difference estimation (with a pretreatment measure of the outcome variable), inverse probability weighting, and \\hl{unbiased Horvitz-Thompson estimation}.''\n\nHelp page, argument \\texttt{HT}: ``when \\texttt{HT=TRUE}, \\hl{invokes the Horvitz-Thompson (difference-in-totals) estimator}. When \\texttt{HT=FALSE}, invokes the inverse-probability-weighted regression estimator''\n\nFile \\texttt{R/estate.R}, lines 31 to 36: when \\texttt{HT == TRUE} and \\texttt{X} is supplied, \\texttt{Ypre2} receives the fitted values from regressing \\texttt{Yd} on \\texttt{X} with \\texttt{lm}. The code sets \\texttt{Yd2} to \\texttt{Yd - Ypre2} and computes \\texttt{ate} as the weighted difference of the treatment and control residual totals divided by the number of units. The source-code exhibit below shows these lines verbatim.", "corrected": "Description: ``\\ldots inverse probability weighting, and Horvitz-Thompson estimation, \\hl{which is unbiased when X is NULL, the assignment probabilities are correct, and any Ypre adjustment is fixed before randomization}.''\n\nArgument \\texttt{HT}: ``\\hl{When HT=TRUE and X=NULL, computes the Horvitz-Thompson difference of totals divided by N, applied to Y (or Y-Ypre). With the correct assignment probabilities and fixed pretreatment Ypre, this is unbiased for the ATE. When HT=TRUE and X is supplied, first fits a least squares regression of Y (or Y-Ypre when Ypre is supplied) on X using all units and applies the same weighted contrast to its residuals. This within-sample covariate adjustment is not generally unbiased for the ATE. To use externally computed adjustments satisfying the covariance condition of the article, pass their prediction vector as Ypre and leave X NULL.} When \\texttt{HT=FALSE}, invokes the inverse-probability-weighted regression estimator.''\n\nThe code does not change.", "evidence": "This correction rests on an exact counterexample, computed in rational arithmetic and reproduced by running the package's code. Suppose that the population has three units, with $X=(0,1,2)$, $Y_{0i}=0$ and $Y_{1i}=1$, so that the average treatment effect is $1$, and that exactly one unit is treated, each with probability $1/3$. With \\texttt{prob} equal to $(1/3,1/3,1/3)$, \\texttt{HT=TRUE} and \\texttt{X} supplied, the function returns $1/4$, $1$ and $1/4$ for the three randomizations. Its expectation is $1/2$ and its bias is $-1/2$. The covariances of $T_i$ with the fitted values are $(1/6,0,1/6)$, and by equation [19] the bias is again $-(1/3)(3+3/2)(1/6+0+1/6)=-1/2$. Without \\texttt{X} the function returns $1$ for every randomization. With \\texttt{Ypre} fixed at $(0,1,2)$ and \\texttt{X=NULL} it returns $5/2$, $1$ and $-1/2$, with expectation $1$.", "problem": "With \\texttt{HT=TRUE} and \\texttt{X} supplied, \\texttt{estate} regresses \\texttt{Y} (or \\texttt{Y-Ypre} when \\texttt{Ypre} is supplied) on \\texttt{X}, using all units, including unit $i$, and applies the Horvitz--Thompson contrast to the residuals. This is the generalized difference estimator with $\\theta$ fitted from the data at hand. Section 5.1 of the article says that unbiasedness is then not guaranteed, and equation [19] gives the bias. The help page calls Horvitz--Thompson estimation unbiased and describes \\texttt{HT=TRUE} as invoking the Horvitz--Thompson estimator, and it does not separate the branch that uses \\texttt{X}. A user who combines the two arguments is not told that unbiasedness is no longer guaranteed. The code defines a valid estimator, which adjusts by residuals fitted within the sample. What is missing is the qualification in the help page.", "what_survives": "No result of the article depends on this branch. Section 7.1 fits its predictions on the opposite block, and its example imposes the sharp null. The article's results on the Horvitz--Thompson estimator and on the generalized difference estimator under $\\operatorname{Cov}(T_i,f(\\mathbf{X}_i,\\theta_i))=0$ are not contradicted, since the bias found is the one that equation [19] predicts. With \\texttt{X=NULL}, correct probabilities and a fixed \\texttt{Ypre}, the function is unbiased. Predictions that satisfy the covariance condition can be computed outside the function and passed as \\texttt{Ypre}. Not every call with \\texttt{X} is biased: under the sharp null the fitted values are fixed. The functions \\texttt{gendist} and \\texttt{estlate} pass \\texttt{X} to the same branch."}, "url": "https://pmaronow.github.io/llm-audit/works/P026/#P026-N004", "work": {"arxiv_id": null, "doi": "10.1515/jci-2012-0009", "title": "A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P026/"}, "work_id": "P026"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of three lines of text and the displayed formula for rho ICC. Red boxes mark the words 'the intra-class correlation over strata', which run from the second line into the third, and the term '1/2' in the denominator 'alpha squared plus 1/2' of the displayed fraction.", "caption": "Supplement, top of p. A2 (supplement PDF p. 2). The marked words call the displayed ratio the intra-class correlation; with the marked $1/2$ it is a variance ratio, and the correlation has 1 in that place.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-A001.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T16", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T16"}, {"id": "P027-T17", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T17"}], "headline": "Under independent outcome errors, the ordinary same-stratum pair correlation is $\\alpha^2/(\\alpha^2+1)$; the printed $\\alpha^2/(\\alpha^2+1/2)$ is a variance ratio built from the divisor-1 variance $2\\alpha^2$ of the two stratum means.", "id": "P027-A001", "location": "Supplement, Section B, first lines and first display of p. A2 (supplement PDF p. 2); the $\\rho_{ICC}$ labels of Table A1, p. A5, and Figure A1, p. A6", "pdf_page": 2, "printed_page": "A2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P027-T16", "P027-T17"], "text": {"as_printed": "``where $\\alpha_s$ is a stratum-specific constant and $\\epsilon_{ks} \\sim \\mathrm{N}(0, 1)$. For the two strata, 1 and 2, we keep things simple by setting $\\alpha_1 = \\alpha = -\\alpha_2$, in which case \\hl{the intra-class correlation over strata} is given by,'' followed by the display\n\\[\n\\rho_{ICC} = \\frac{\\alpha^2}{\\alpha^2 + \\hl{1/2}}.\n\\]", "corrected": "``For the two strata, 1 and 2, we keep things simple by setting $\\alpha_1 = \\alpha = -\\alpha_2$, and the simulations are indexed by \\hl{the variance ratio}\n\\[\n\\rho_{ICC} = \\hl{\\frac{S_\\alpha^2}{S_\\alpha^2+1}} = \\frac{\\alpha^2}{\\alpha^2 + 1/2},\\qquad S_\\alpha^2=\\sum_{s=1}^{2}\\frac{(\\alpha_s-\\bar\\alpha)^2}{2-1}=2\\alpha^2,\n\\]\nwhere $\\bar\\alpha=0$ is the mean of the two stratum constants. Table A1 and Figure A1 use this parameter. Under independent outcome errors, the correlation of two distinct units sharing \\hl{a uniformly chosen stratum} is instead \\hl{$\\alpha^2/(\\alpha^2+1)$}, which equals $\\rho_{ICC}/(2-\\rho_{ICC})$.''", "evidence": "The evidence is a derivation with an exact instance. Let $S=\\pm 1$ with probability $1/2$ each mark the common stratum, and let $Y=\\alpha S+\\epsilon$ and $Y'=\\alpha S+\\epsilon'$, where $\\epsilon$ and $\\epsilon'$ are independent $\\mathrm{N}(0,1)$ variables that are independent of $S$. Then I have $\\mathbb{E}Y=\\mathbb{E}Y'=0$, $\\operatorname{Var}(Y)=\\operatorname{Var}(Y')=\\alpha^2+1$ and $\\operatorname{Cov}(Y,Y')=\\alpha^2$, which implies that\n\\[\n\\operatorname{Corr}(Y,Y')=\\frac{\\alpha^2}{\\alpha^2+1}.\n\\]\nAt $\\alpha=1/2$ this is $1/5$, against the printed $1/3$. The printed ratio equals $S_\\alpha^2/(S_\\alpha^2+1)$, where $S_\\alpha^2=\\sum_{s=1}^{2}(\\alpha_s-\\bar\\alpha)^2/(2-1)=2\\alpha^2$ is the variance of the two stratum means with divisor 1. Solving the printed formula for $\\alpha^2$ and substituting, I obtain $\\operatorname{Corr}(Y,Y')=\\rho_{ICC}/(2-\\rho_{ICC})$, an increasing function of $\\rho_{ICC}$. For the labels 0.33, 0.50, 0.67, 0.82, 0.90 and 0.99 of Table A1, taken as exact, the correlations are 0.198, 0.333, 0.504, 0.695, 0.818 and 0.980.", "problem": "The supplement does not say which convention it uses, and under the ordinary one the value is different. For two distinct units of the same stratum, where the stratum is either of the two with probability $1/2$ and the errors are independent with unit variance, the correlation of the outcomes is $\\alpha^2/(\\alpha^2+1)$. The printed ratio exceeds this by $\\alpha^2/\\bigl((\\alpha^2+1)(2\\alpha^2+1)\\bigr)$, which vanishes only at $\\alpha=0$. The $\\rho_{ICC}$ values that label Table A1 and Figure A1 are not ordinary correlations: the label 0.50 corresponds to a correlation of $1/3$. The printed ratio is a coherent parameter of another kind, and the printed formula still identifies every scenario.", "what_survives": "No simulated value changes. Since the relabelling is monotone, the order of the scenarios, every estimator value in Table A1 and Figure A1 and every comparison within a scenario stand, as does the remark that all approximations perform badly when the parameter is close to 1. A threshold stated in terms of $\\rho_{ICC}$ must be converted if the ordinary correlation is meant. The main text and the simulations of Section C do not use this parameter."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-A001", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of five lines of the supplement. A red box marks the words 'the empirical variance of mu-hat HT' in the sentence that begins 'For each rho ICC and M value, we used'.", "caption": "Supplement, Section B, p. A2 (supplement PDF p. 2). The marked words describe the “True” column without saying over which draws the variance is taken.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-A002.png"}, {"alt": "Crop of six lines of the introduction of the supplement. Red boxes mark the words 'computed exactly using expressions from Propositions 1 and 2 in the main text.', which run from the third line into the fourth.", "caption": "Supplement, Section A, p. A1 (supplement PDF p. 1). The marked words say that the expected values of the paper’s two estimators were computed exactly.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-A002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "“True” in Tables A1 to A3 targets the sampling variance conditional on each generated population, averaged over the populations; “the empirical variance of $\\hat\\mu_{HT}$” does not say this, and the number of populations behind Tables A2 and A3 is not given.", "id": "P027-A002", "location": "Supplement, Section B, p. A2 (supplement PDF p. 2), paragraph beginning “Table A1 and Figure A1 display”; compare Section A, p. A1, and Section C, pp. A3–A4", "pdf_page": 2, "printed_page": "A2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Supplement, p.~A2: ``We simulated 1,000 sets of outcomes, $(y_1, y_2)$ over varying values of $M$ and $\\rho_{ICC}$. To each set, we applied the single-unit-per-stratum design.'' Later in the same section: ``For each $\\rho_{ICC}$ and $M$ value, we used \\hl{the empirical variance of $\\hat\\mu_{HT}$} as an estimate of the true variance (Var or ``True''), and means of the computed $(1/N^2)\\widehat{\\operatorname{Var}}(\\hat t)$ \\ldots values as estimates of the expected values of these estimators.''\n\nSupplement, p.~A1: ``In the simulations, expected values of $\\widehat{\\operatorname{Var}}(\\hat t)$ and $\\widehat{\\operatorname{Var}}_C(\\hat t)$ were computed exactly using expressions from Propositions 1 and 2 in the main text.''", "corrected": "``For each $\\rho_{ICC}$ and $M$ value, ``True'' \\hl{targets} the sampling variance of $\\hat\\mu_{HT}$ \\hl{conditional on a generated finite population, averaged over the generated populations}; it is not the pooled variance obtained by drawing a fresh population and one sample from it at every repetition.'' The supplement should also state how each conditional variance was calculated or estimated and how many populations were generated for Tables A2 and A3. I could not establish these details without the simulation code, which was unavailable to me.", "intended_meaning": "The simulations concern the sampling variance for fixed outcomes. I therefore read ``True'' as targeting the variance of $\\hat\\mu_{HT}$ over samples for one generated population, averaged over the generated populations. Table A1 agrees with this reading and not with a variance pooled over fresh populations. With two strata of size $M$ and independent unit-variance errors the expected conditional variance is $(M-1)/(2M)$, which is 0.25 at $M=2$ as printed. The pooled variance is $1/2$. On this reading the exact computation within each population described on p.~A1 and the averages over populations described on p.~A2 are compatible. The supplement does not say how the conditional variance was obtained for each population. Pages A3--A4 do not state the number of populations behind Tables A2 and A3; the ``1000 simulated sample draws'' mentioned there concern the with-replacement estimator. No table entry is shown to be affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-A002", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the supplement. Red boxes mark the words 'estimator dominates' on the third line and 'for smaller stratum sizes and the two perform similarly for larger stratum sizes.' on the fourth line.", "caption": "Supplement, Section B, last paragraph, p. A2 (supplement PDF p. 2). The marked words compare the corrected estimator with the collapsed-stratum estimator by stratum size alone.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T14", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T14"}, {"id": "P027-T15", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T15"}], "headline": "The corrected estimator has smaller averaged bias than the collapsed-stratum estimator exactly when $M<2(1+\\alpha^2)$, and for large strata the two are close only when $\\rho_{ICC}$ is near 1.", "id": "P027-E001", "location": "Supplement, Section B, last paragraph, p. A2 (supplement PDF p. 2); compare Table A1, p. A5", "pdf_page": 2, "printed_page": "A2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P027-T14", "P027-T15"], "text": {"as_printed": "``The $(1/N^2)\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ estimator \\hl{dominates} $\\widehat V_{CS}(\\hat\\mu_{HT})$ \\hl{for smaller stratum sizes and the two perform similarly for larger stratum sizes}.''", "corrected": "``Averaging over the sampling design and the generated populations, with independent unit-variance errors about $\\alpha$ and $-\\alpha$, the $(1/N^2)\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ estimator has smaller bias than $\\widehat V_{CS}(\\hat\\mu_{HT})$ \\hl{exactly when $M<2(1+\\alpha^2)$}; the two tie when $M=2(1+\\alpha^2)$, and it has larger bias otherwise. This is a comparison of biases, not of mean squared errors. For fixed $\\alpha$ the difference of the expected estimates tends to $1/2$ as $M$ grows and \\hl{their ratio tends to $2-\\rho_{ICC}$}, so large strata alone do not make the two similarly accurate; the expected estimates are relatively close when $\\rho_{ICC}$ is near 1.''", "evidence": "The evidence is a derivation with exact instances. Let $x$ and $z$ be the outcomes of the units drawn from strata 1 and 2. Since $\\pi=1/M$ and $N=2M$, the printed estimators reduce to\n\\[\n\\frac{1}{N^2}\\widehat{\\operatorname{Var}}_{C2}(\\hat t)=\\frac{M-1}{2M}\\left(x^2+z^2\\right),\\qquad \\widehat V_{CS}(\\hat\\mu_{HT})=\\frac14\\left(x-z\\right)^2 .\n\\]\nWith independent unit-variance errors about $\\alpha$ and $-\\alpha$, I have $\\mathbb{E}x^2=\\mathbb{E}z^2=1+\\alpha^2$ and $\\mathbb{E}xz=-\\alpha^2$, and the sampling variance of $\\hat\\mu_{HT}$, averaged over populations, is $(M-1)/(2M)$. Subtracting this variance from the expected estimates, I obtain the averaged biases\n\\[\nB_{C2}=\\frac{M-1}{2M}\\left(1+2\\alpha^2\\right),\\qquad B_{CS}=\\alpha^2+\\frac{1}{2M},\\qquad B_{C2}-B_{CS}=\\frac12-\\frac{1+\\alpha^2}{M}.\n\\]\nBoth biases are nonnegative. The ratio of the two expected estimates is $(1-1/M)(2-\\rho_{ICC})$.\n\n\\begin{audittable}{rrrrrrr}{Exact averaged values under the stated model.}\n$M$ & $\\alpha$ & True variance & Mean of $(1/N^2)\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ & Mean of $\\widehat V_{CS}$ & $B_{C2}$ & $B_{CS}$ \\\\\n2 & 0 & $1/4$ & $1/2$ & $1/2$ & $1/4$ & $1/4$ \\\\\n5 & 0 & $2/5$ & $4/5$ & $1/2$ & $2/5$ & $1/10$ \\\\\n5 & 2 & $2/5$ & 4 & $9/2$ & $18/5$ & $41/10$ \\\\\n500 & 0 & $499/1000$ & $499/500$ & $1/2$ & $499/1000$ & $1/1000$ \\\\\n\\end{audittable}\n\nEnumerating the 25 samples of a fixed population with $M=5$, I find the same reversal: the biases are $3/10$ and 0 at $\\alpha=0$, and $7/2$ and 4 at $\\alpha=2$.", "problem": "The comparison depends on the stratum means as well as on the stratum size, and the sentence mentions only the size. Averaged over samples and generated populations, the corrected estimator has the smaller bias only when $M<2(1+\\alpha^2)$. At $M=5$ the ranking reverses between $\\alpha=0$ and $\\alpha=2$. For large $M$ the two expected values do not approach each other: their difference tends to $1/2$, which is the limit of the true variance, and their ratio tends to $2-\\rho_{ICC}$. At $M=500$ and $\\alpha=0$ the relative biases are 1 and $1/499$. Table A1 shows the same pattern: its printed nominal bias is larger for $\\widehat{\\operatorname{Var}}_{C2}$ in 23 of the 35 rows, equal in 1 and smaller in 11.", "what_survives": "Table A1, Figure A1, the estimator definitions and the conservativeness of $\\widehat{\\operatorname{Var}}_{C2}$ stand. The first clause holds for the smallest stratum size shown, $M=2$, with a tie at $\\alpha=0$. The main text's weaker statement on p.~240, that these estimators are less biased than the collapsed-stratum estimator ``in a range of scenarios'', also stands. The comparison here is of biases, and nothing is established about mean squared error."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E001", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of two lines of text, a displayed formula and two further lines. A red box marks the display 'pi k equals x k over the sum of x l from l = 1 to N, times m'.", "caption": "Supplement, Section C, p. A3 (supplement PDF p. 3). The marked display gives the first order inclusion probability with no bound at 1.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E002.png"}, {"alt": "Crop of four lines of the supplement. Red boxes mark the words 'negative binomial distribution with mean 6, variance 16, and offset of 1 to avoid zero values.', which run from the second line into the third.", "caption": "Supplement, Section C, p. A3 (supplement PDF p. 3). The marked words give the law of the sizes, which is unbounded.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T12", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T12"}, {"id": "P027-T13", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T13"}], "headline": "The formula $\\pi_k=mx_k/\\sum_lx_l$ can exceed 1 under the stated negative binomial sizes; the recipe needs a rule for such units, for example $\\pi_k=\\min(1,cx_k)$ with $\\sum_k\\pi_k=m$.", "id": "P027-E002", "location": "Supplement, Section C, p. A3 (supplement PDF p. 3): the size variable and the display for $\\pi_k$", "pdf_page": 3, "printed_page": "A3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P027-T12", "P027-T13"], "text": {"as_printed": "``For the simulations, a size variable, $x = (x_1, ..., x_N)$, is produced as a vector of $N$ random draws from a \\hl{negative binomial distribution with mean 6, variance 16, and offset of 1} to avoid zero values.'' Later on the same page: ``Systematic PPS sampling was implemented such that each unit's first order inclusion probability is given by,\n\\[\n\\hl{\\pi_k = \\frac{x_k}{\\sum_{l=1}^{N} x_l}\\, m,}\n\\]\nwhere $m$ is the size of the systematic sample, and joint inclusion probabilities are computed as per Madow (1949). For the simulations, sampling and computation of inclusion probabilities were carried out using functions provided by Tillé and Matei (2011).''", "corrected": "One possible repair is:\n``Systematic PPS sampling can use first order inclusion probabilities given by,\n\\[\n\\pi_k = \\hl{\\min\\left(1,\\,c\\,x_k\\right),\\qquad c \\text{ chosen so that } \\sum_{k=1}^{N}\\pi_k=m,}\n\\]\nwhere $m$ is the size of the systematic sample\\hl{; a systematic sample is drawn with these probabilities in the given order of the population,} and joint inclusion probabilities are \\hl{those of that design}.'' Alternatively the printed display is kept and the size vector is redrawn until $m\\max_kx_k\\le\\sum_{l=1}^{N}x_l$, conditioning the size-generation law on this inequality.\n\nWhen no unit is oversized, $c=m/\\sum_lx_l$ and the display reduces to the printed one. On the example above, $c=1/7$ gives $(\\pi_1,\\dots,\\pi_8)=(1,1/7,\\dots,1/7)$, which is realized by the seven samples $\\{1,j\\}$, $j=2,\\dots,8$, each with probability $1/7$. Capping the printed values at 1 without redistributing gives a total of $51/37$, not 2. The actual implementation is needed to establish whether either rule matches the reported runs.", "evidence": "The evidence is an exact counterexample and an exact calculation. For $N=8$, $m=2$ and $x=(30,1,\\dots,1)$ I have $\\sum_lx_l=37$ and $(\\pi_1,\\dots,\\pi_8)=(60/37,2/37,\\dots,2/37)$. The entries sum to 2, but $\\pi_1>1$. Note that $\\mathbb{E}(I_1)\\le 1$ for any sampling design. A negative binomial law with mean 6 and variance 16 before the offset has success probability $3/8$ and size parameter $18/5$, and it assigns positive probability to every nonnegative integer. The shifted law therefore gives the vector above positive probability. With independent draws under this interpretation, the probability that some $\\pi_k$ exceeds 1 lies between 0.000168504 and 0.000168505. Each of the six pairs $(N,m)$ of Table A2 admits such a vector. Since the simulation code was unavailable to me, I do not know whether such a case arose or how it was handled.", "problem": "Since the sizes are unbounded, $m\\max_kx_k$ can exceed $\\sum_lx_l$. The display then gives a value above 1, which is not the expectation of an inclusion indicator. In the scenario $N=8$, $m=2$ the size vector $(30,1,1,1,1,1,1,1)$ has $\\pi_1=60/37$. Under the stated law of the sizes some $\\pi_k$ exceeds 1 with probability about 0.00017 in that scenario. The supplement states no rule for such units, and the simulation cannot be reproduced from its description when they occur.", "what_survives": "No entry of Table A2 and no comparison is shown to be wrong. The results of the main text start from valid inclusion probabilities and are untouched. Section B is also untouched, as is the periodic design of Table A3, where $\\max_k\\pi_k=10m/(6N)\\le 5/12$."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E002", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of the supplement. A red box marks the words 'y is proportional to x' in the sentence that begins 'Because y is proportional to x with x greater than or equal to 1, Proposition 1 in the main text indicates'.", "caption": "Supplement, Section C, p. A3 (supplement PDF p. 3). The marked premise is that $y$ is proportional to $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E003.png"}, {"alt": "Crop of two lines of text, a displayed formula and two further lines. A red box marks the display 'y k equals beta x k plus epsilon k'. The line below it says that epsilon k is Gaussian noise with mean zero.", "caption": "Supplement, Section C, p. A3 (supplement PDF p. 3). The marked display is the outcome model, in which Gaussian noise is added to $\\beta x_k$.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T10", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T10"}, {"id": "P027-T11", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T11"}], "headline": "The simulated outcomes $y_k=\\beta x_k+\\epsilon_k$ are not proportional to $x$ and can change sign; Proposition 1 then implies that the uncorrected estimator is conservative only on average over the generated outcomes, not for each generated population.", "id": "P027-E003", "location": "Supplement, Section C, p. A3 (supplement PDF p. 3), paragraph beginning “Because $y$ is proportional to $x$”", "pdf_page": 3, "printed_page": "A3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P027-T10", "P027-T11"], "text": {"as_printed": "``Because \\hl{$y$ is proportional to $x$} with $x \\geq 1$, Proposition 1 in the main text indicates that the Horvitz-Thompson variance estimator for $\\hat\\mu_{HT}$, $(1/N^2)\\widehat{\\operatorname{Var}}(\\hat t)$, provides a conservative approximation of the sampling variance of $\\hat\\mu_{HT}$.''\n\nThe outcome model is displayed earlier on the same page: ``$y_k = \\beta x_k + \\epsilon_k$,'' where ``$\\epsilon_k$ is Gaussian noise with mean zero and standard deviation $\\sigma_y\\sqrt{1-\\rho^2}$'' and ``we fix $\\sigma_y = 30$''.", "corrected": "``For a size vector $x$ giving \\hl{valid inclusion probabilities}, \\hl{assuming independent mean-zero errors conditional on $x$}, the model has \\hl{$\\mathrm{E}_\\epsilon[y_k\\mid x]=\\beta x_k$} with $x_k\\geq 1$. Proposition 1 in the main text therefore indicates that the Horvitz-Thompson variance estimator for $\\hat\\mu_{HT}$, $(1/N^2)\\widehat{\\operatorname{Var}}(\\hat t)$, has nonnegative bias \\hl{after averaging over the generated outcomes}:\n\\[\n\\mathrm{E}_\\epsilon[A\\mid x]=\\beta^2\\sum_{k\\ne l:\\,\\pi_{kl}=0}x_kx_l\\geq 0.\n\\]\nIt \\hl{does not guarantee} a conservative approximation for each generated finite population, because positive sizes and Gaussian errors do not ensure outcomes of one sign.''", "evidence": "The evidence is an exact counterexample and a derivation. I set $N=8$ and $m=2$, with sizes and outcomes\n\\[\nx=(1,3,1,3,1,3,1,3),\\qquad y=(1,-1,1,-1,1,-1,1,-1).\n\\]\nThen $\\pi_k=x_k/8$, and systematic PPS sampling with a uniform start on $[0,8)$ has four possible samples.\n\n\\begin{audittable}{llrrr}{The eight-unit example, enumerated in full.}\nStart & Sample & Probability & $\\hat t$ & $\\widehat{\\operatorname{Var}}(\\hat t)$ \\\\\n$[0,1)$ & $\\{1,5\\}$ & $1/8$ & 16 & 224 \\\\\n$[1,4)$ & $\\{2,6\\}$ & $3/8$ & $-16/3$ & $160/9$ \\\\\n$[4,5)$ & $\\{3,7\\}$ & $1/8$ & 16 & 224 \\\\\n$[5,8)$ & $\\{4,8\\}$ & $3/8$ & $-16/3$ & $160/9$ \\\\\n\\end{audittable}\n\nFrom the table, I find $\\operatorname{Var}(\\hat t)=256/3$ and $\\mathbb{E}\\,\\widehat{\\operatorname{Var}}(\\hat t)=208/3$, which implies that $A=-16$. For the mean, the sampling variance is $4/3$ and the expected estimate is $13/12$, a bias of $-1/4$. Every outcome vector within $1/8$ of this $y$ in each coordinate has $A\\le -13/4$, and this set has positive probability under independent Gaussian noise for every listed $\\rho<1$.\n\nFor the average, fix a feasible size vector $x$ and suppose that the errors are independent with mean zero. For $k\\ne l$ I have $\\mathbb{E}_\\epsilon[y_ky_l\\mid x]=\\beta^2x_kx_l$. Summing over the pairs that are never selected together, I obtain\n\\[\n\\mathbb{E}_\\epsilon[A\\mid x]=\\beta^2\\sum_{k\\ne l:\\,\\pi_{kl}=0}x_kx_l\\ge 0 .\n\\]", "problem": "By Proposition 1, the bias is nonnegative when all outcomes have the same sign. The outcomes of this simulation are $\\beta x_k$ plus Gaussian noise with standard deviation $30\\sqrt{1-\\rho^2}$. They are not proportional to $x$ and need not share a sign, and the bias term $A$ can be negative for a generated population. A weaker guarantee does hold: with independent errors the bias is nonnegative after averaging over the generated outcomes.", "what_survives": "Proposition 1, the same-sign condition as the main text states it on p.~234, and the conservativeness of the corrected estimator stand; the bias of $\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ is $(1/2)\\sum_{k\\ne l:\\,\\pi_{kl}=0}(y_k+y_l)^2\\ge 0$. The positive averaged biases of Table A2 agree with the averaged statement and are not in question. Only the guarantee for each generated population of this simulation is lost, and the label ``Conservative (HT)'' of Figure A2 is to be read as a statement about the average over outcomes."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E003", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": {"class": "error", "severity": "Moderate"}, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last four lines of the text of the supplement. Red boxes mark the words 'almost perfectly.' on the second line, 'canceling each other out,' at the end of the third line and 'in which case A tends to 0.' on the fourth line.", "caption": "Supplement, Section C, last paragraph before the References, p. A4 (supplement PDF p. 4). The marked words are the accuracy claim and the explanation given for it, both of which need qualifying.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E004.png"}, {"alt": "The title reads 'Table A3: printed relative biases and the derived ratio of outcome averages'. Two panels, one for sampling fraction one quarter and one for one twentieth, plot relative bias from minus 0.175 to 0.03 against rho from 0 to 1. Blue open markers joined by lines are the printed values for three designs in each panel. In the left panel they lie between minus 0.14 and minus 0.09, near an orange dashed line at minus 5 over 36, about minus 0.139. In the right panel they lie between minus 0.03 and minus 0.01, near an orange dashed line at minus 5 over 180, about minus 0.028. A grey line at zero is labelled as a reference line for zero relative bias. No printed value is at or above zero.", "caption": "Relative bias of the uncorrected estimator in the 60 rows of supplement Table A3, as printed there to two decimals, with the ratio $-5/(9q)$ derived for the stated design (dashed) and zero as a reference line. The derived ratio is a ratio of averages over outcomes and is not a recomputation of the table.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E004-ht-relative-bias-r2.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T1", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T1"}, {"id": "P027-T2", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T2"}], "headline": "In the periodic design the elements of the bias term cancel only in part: $A=-\\sum_gT_g^2$, minus the sum of the squared outcome totals of the $q=N/m$ possible samples, so the uncorrected estimator is biased downward unless every such total is zero; under the stated outcome model the ratio of bias averaged over outcomes to variance averaged over outcomes is $-5/(9q)$.", "id": "P027-E004", "location": "Supplement, Section C, last paragraph before the References, p. A4 (supplement PDF p. 4); compare Table A3, p. A8", "pdf_page": 4, "printed_page": "A4", "reaches": [], "related_entries": [], "revisions": [{"classification_after": {"class": "clarity_note", "severity": null}, "classification_before": {"class": "error", "severity": "Moderate"}, "note": "Reclassified from a Moderate error to a clarity note. The derivation shows partial cancellation in $A$, but does not establish that the paragraph’s qualitative description is false. The paragraph gives no numerical threshold for “almost perfectly” or scale or limit for “$A$ tends to 0”, and Table A3 prints the negative bias. The derivation, exact example and figure are retained."}], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P027-T1", "P027-T2"], "text": {"as_printed": "``Interestingly, $\\widehat{\\operatorname{Var}}(\\hat t)$ tends to match the true value of the variance \\hl{almost perfectly}. This occurs because of the special nature of the data generating process, which results in the elements in the bias term, $A$, \\hl{canceling each other out, in which case $A$ tends to 0}.''", "corrected": "A clarified version of the two sentences is:\n\n``Interestingly, $\\widehat{\\operatorname{Var}}(\\hat t)$ tends to be \\hl{close to} the true value of the variance, \\hl{especially at the smaller sampling fraction}. This occurs because of the special nature of the data generating process, which results in the elements in the bias term, $A$, \\hl{partly canceling each other out}. \\hl{The cancellation is not complete}: if $T_g$ is the demeaned outcome total of the $g$th of the $q=N/m$ possible samples, then\n\\[\n\\hl{A=-\\sum_{g=1}^{q}T_g^2\\le 0,}\n\\]\n\\hl{so that $\\widehat{\\operatorname{Var}}(\\hat t)$ is biased downward unless every $T_g$ is zero}. Under the outcome model used here, the ratio of the bias averaged over outcomes to the variance averaged over outcomes is \\hl{$-5/(9q)$}, about $-3\\%$ when $m/N=1/20$ and $-14\\%$ when $m/N=1/4$.''", "intended_meaning": "The paragraph reports the simulation results of Table A3 and Figure A3 and offers a qualitative reason for them. It gives ``almost perfectly'' no numerical threshold, and it gives ``$A$ tends to 0'' no scale and no limiting operation. On its natural reading it describes approximate agreement explained by approximate cancellation, and on that reading it is not false. The cancellation is partial, however, and what it leaves is a systematic downward bias, which the paragraph does not mention; Table A3 itself prints that bias.\n\nIn the six designs of Table A3 the possible systematic samples are $q=N/m$ disjoint groups of units, with $q=4$ or $q=20$. With demeaned outcomes the products that make up $A$ can have both signs and offset one another, but their sum equals minus the sum of the squared group totals. It is zero only when every group total is zero, and demeaning makes the totals sum to zero without forcing each of them to be zero. The uncorrected estimator therefore has nonpositive bias in this design. Under the stated outcome model the ratio of the averaged bias to the averaged variance is $-5/(9q)$, about $-14\\%$ at $m/N=1/4$ and $-3\\%$ at $m/N=1/20$. Table A3 prints relative biases between $-0.03$ and $-0.01$ for $q=20$ and between $-0.14$ and $-0.09$ for $q=4$. The agreement is therefore close at the smaller sampling fraction, while at the larger one the printed shortfall is 9 to 14 per cent of the true variance. Taken literally, as a statement about the unscaled term, ``$A$ tends to 0'' is misleading: for $\\rho>0$ and a fixed sampling fraction, $A/N^2$ does not vanish as $N$ grows. At a fixed sample size, by contrast, the ratio $-5/(9q)$ tends to zero as $N$ grows. The paragraph specifies neither limit.\n\nThe evidence is a derivation and an exact example. Let $z_k=\\tilde y_k$ be the demeaned outcomes. Since the sizes alternate between 2 and 10 and $q$ is even, a systematic PPS draw selects one of the groups $G_g=\\{g,g+q,\\dots,g+(m-1)q\\}$, $g=1,\\dots,q$, with probability $p_g=1/(3q)$ for odd $g$ and $p_g=5/(3q)$ for even $g$. Two distinct units have $\\pi_{kl}=0$ exactly when they lie in different groups. Let $T_g=\\sum_{k\\in G_g}z_k$, and note that $\\sum_gT_g=0$ because the outcomes are demeaned. By the definition of $A$ on p.~234 of the main text, I have\n\\[\nA=\\sum_{g}\\sum_{h\\ne g}T_gT_h=\\Bigl(\\sum_gT_g\\Bigr)^2-\\sum_gT_g^2=-\\sum_gT_g^2\\le 0,\n\\]\nand the bias of $(1/N^2)\\widehat{\\operatorname{Var}}(\\hat t)$ is $A/N^2$.\n\nFor an exact instance, let $N=8$, $m=2$, with sizes $(2,10,2,10,2,10,2,10)$, demeaned outcomes $(-1,1,-1,1,-1,1,-1,1)$ and a uniform start on $[0,24)$.\n\n\\begin{audittable}{llrrrrr}{The eight-unit periodic example, enumerated in full. The true variance of $\\hat\\mu_{HT}$ is $9/5$.}\nStart & Sample & Probability & $T_g$ & $\\hat\\mu_{HT}$ & $(1/N^2)\\widehat{\\operatorname{Var}}(\\hat t)$ & $(1/N^2)\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ \\\\\n$[0,2)$ & $\\{1,5\\}$ & $1/12$ & $-2$ & $-3$ & $33/4$ & $21/2$ \\\\\n$[2,12)$ & $\\{2,6\\}$ & $5/12$ & 2 & $3/5$ & $21/100$ & $33/50$ \\\\\n$[12,14)$ & $\\{3,7\\}$ & $1/12$ & $-2$ & $-3$ & $33/4$ & $21/2$ \\\\\n$[14,24)$ & $\\{4,8\\}$ & $5/12$ & 2 & $3/5$ & $21/100$ & $33/50$ \\\\\nExpectation & & & & 0 & $31/20$ & $23/10$ \\\\\n\\end{audittable}\n\nHere $A=-16$, the bias is $A/N^2=-1/4$, and the relative bias is $-5/36$. The 48 products $z_kz_l$ over ordered pairs of units in different groups are each $\\pm1$: 16 are positive and 32 negative, so that the absolute value of their sum is a third of the sum of their absolute values. With the same alternating outcomes, the bias is $-1/4$ in each of the three designs with $q=4$. In each of the three with $q=20$ it is $-1/20$, a relative bias of $-1/36$, and the absolute value of the sum of the products is a nineteenth of the sum of their absolute values.\n\nUnder independent Gaussian residuals with a common variance, $\\mathbb{E}_y(T_g^2)$ takes the same value $D$ for every group. Then I have $\\mathbb{E}_y(A)=-qD$ and $\\mathbb{E}_y\\operatorname{Var}(\\hat t)=D\\sum_g1/p_g=9q^2D/5$, and their ratio is $-5/(9q)$. This is a ratio of averages over outcomes. It is not generally the relative bias of an individual population: when its variance is positive, that bias lies between $-5/(3q)$ and $-1/(3q)$. It does not reconstruct the entries of Table A3. With the sample standard deviation of $x$ in $\\beta$, the quantity $A/N^2$ converges in probability to $-900\\rho^2/q$ as $N$ grows with $q$ and $\\rho>0$ fixed. Table A3 prints a negative bias for the uncorrected estimator in all 60 rows.\n\nProposition 1, the corrected estimator and its conservativeness, and the main text's comparison with the with-replacement estimator under periodicity are unaffected. In the exact example the corrected estimator has expectation $23/10$ against a true variance of $9/5$, and the with-replacement estimator is identically zero. The earlier sentence on p.~A4, that $\\widehat{\\operatorname{Var}}(\\hat t)$ ``may have either positive or negative bias'', is correct as a general statement, although in this design the bias is never positive. This note does not revise the entries of Table A3; \\fid{P027-N004} concerns four of its rows, and \\fid{P027-E007} the label of the uncorrected series."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E004", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the display 'y k equals beta x k plus epsilon k' and the six lines below it. Red boxes mark the words 'sigma x and sigma y are the standard deviation of the N values in x and y respectively, where E x of sigma x = 4 and we fix sigma y = 30', which run over two lines.", "caption": "Supplement, Section C, p. A3 (supplement PDF p. 3). The marked words define $\\sigma_x$ and $\\sigma_y$ as standard deviations of the realized values, set $\\mathrm{E}_x(\\sigma_x)=4$ and fix $\\sigma_y=30$.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E005.png"}], "formal_release": null, "formal_statements": [], "headline": "The values 4, 30 and $\\rho$ are nominal settings of the simulation; the standard deviations and, when defined, the correlation of the $N$ realized values vary from one generated population to the next.", "id": "P027-E005", "location": "Supplement, Section C, p. A3 (supplement PDF p. 3), the definitions after the display $y_k=\\beta x_k+\\epsilon_k$", "pdf_page": 3, "printed_page": "A3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``for $k = 1, ..., N$, where $\\beta = \\rho(\\sigma_y/\\sigma_x)$, $\\epsilon_k$ is Gaussian noise with mean zero and standard deviation $\\sigma_y\\sqrt{1-\\rho^2}$, and \\hl{$\\sigma_x$ and $\\sigma_y$ are the standard deviation of the $N$ values in $x$ and $y$ respectively, where $\\mathrm{E}_x(\\sigma_x) = 4$ and we fix $\\sigma_y = 30$} (again, to mimic data distributed as in Wolter (2007, Ch. 8)). Thus, $x$ and $y$ have a linear relationship through the origin and \\hl{their correlation is given by $\\rho$}.''", "corrected": "``for $k = 1, ..., N$, where $\\beta = \\rho(\\sigma_y/\\sigma_x)$, $\\epsilon_k$ is Gaussian noise with mean zero and standard deviation $\\sigma_y\\sqrt{1-\\rho^2}$, $\\sigma_y = 30$ is a \\hl{nominal} outcome scale and $\\rho$ is a \\hl{nominal} correlation. The size distribution has standard deviation 4. The standard deviations and, \\hl{when defined}, the correlation of the $N$ realized values \\hl{vary across generated populations}.'' The supplement should also state which standard deviation of the sizes enters $\\beta$, the realized one or the value 4, and how a realized standard deviation of zero is handled. The simulation code or an explicit description is needed to settle this.", "intended_meaning": "The paragraph sets the nominal calibration of the simulation: sizes drawn from a law with standard deviation 4, an outcome scale of 30 and a correlation parameter $\\rho$. Read literally as statements about the $N$ realized values, they do not hold. The expected standard deviation of a finite sample from a law with variance 16 is below 4 by Jensen's inequality, and with Gaussian noise the realized standard deviation of $y$ and, when defined, the realized correlation of $x$ and $y$ are random. The nominal reading is coherent, and no entry of Tables A2 and A3 is shown to depend on the difference. The supplement does not say whether $\\beta$ uses the realized standard deviation of the sizes or the value 4, nor what is done when the realized standard deviation is zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E005", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the displayed inequality of Corollary 2 and the one-line proof beneath it. A red box marks the words 'Proposition 1' in the sentence 'Proposition 1 therefore holds.'", "caption": "Proof of Corollary 2, p. 237 (PDF p. 9). The marked reference should be to Proposition 2.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E006.png"}, {"alt": "Crop of five lines of the first paragraph of the introduction. A red box marks the word 'remains' in 'Optimal methods for variance estimation in these cases remains an open problem.'", "caption": "Section 1, first paragraph, p. 231 (PDF p. 3). The marked verb should be “remain”.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E006-b.png"}, {"alt": "Crop of the end of page A1 of the supplement, with two displayed formulas. A red box marks the case '0 if s = t' in the display for the joint inclusion probability, and a second red box marks the symbol epsilon with subscript 'is' in the display for the outcome.", "caption": "Supplement, Section B, foot of p. A1 (supplement PDF p. 1). The marked case lacks the restriction to distinct units, and the marked residual should carry the index $k$.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E006-c.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T18", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T18"}, {"id": "P027-T23", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T23"}], "headline": "Four misprints: the proof of Corollary 2 should cite Proposition 2, “remains” should read “remain”, the display for $\\pi_{ks,lt}$ in the supplement applies to distinct units only, and $\\epsilon_{is}$ should read $\\epsilon_{ks}$.", "id": "P027-E006", "location": "Proof of Corollary 2, p. 237 (PDF p. 9); Section 1, first paragraph, p. 231 (PDF p. 3); supplement, Section B, pp. A1–A2 (supplement PDF pp. 1–2)", "pdf_page": 9, "printed_page": "237", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P027-T18", "P027-T23"], "text": {"as_printed": "\\begin{itemize}\n\\item (a) p.~237: ``Proof. For all pairs $k, l$ such that $\\pi_{kl} = 0$, $1/a_{kl} + 1/b_{kl} = 1$. \\hl{Proposition 1} therefore holds.''\n\\item (b) p.~231: ``Optimal methods for variance estimation in these cases \\hl{remains} an open problem.''\n\\item (c) Supplement, p.~A1: ``and for units $k$ and $l$ in strata 1 or 2, we have'' followed by the display\n\\[\n\\pi_{ks,lt} = \\begin{cases} \\hl{0} & \\hl{\\text{if } s = t} \\\\ \\frac{1}{M^2} & \\text{if } s \\neq t \\end{cases}.\n\\]\n\\item (d) Supplement, pp.~A1--A2: ``In stratum $s$, outcomes $y_s = (y_{1s}, ..., y_{Ms})$ are drawn as, $y_{ks} = \\alpha_s + \\hl{\\epsilon_{is}}$, where $\\alpha_s$ is a stratum-specific constant and $\\epsilon_{ks} \\sim \\mathrm{N}(0, 1)$.''\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item (a) ``Proof. For all pairs $k, l$ such that $\\pi_{kl} = 0$, $1/a_{kl} + 1/b_{kl} = 1$. \\hl{Proposition 2 therefore applies.}''\n\\item (b) ``Optimal methods for variance estimation in these cases \\hl{remain} an open problem.''\n\\item (c) ``and for \\hl{distinct} units \\hl{$(k,s)\\neq(l,t)$} in strata 1 or 2, we have'' followed by the same display, with $\\pi_{ks,ks}=1/M$ for a unit paired with itself.\n\\item (d) ``$y_{ks} = \\alpha_s + \\hl{\\epsilon_{ks}}$''\n\\end{itemize}", "evidence": "This correction rests on a check of the pages. Proposition 1 (p.~234) is the bias identity, and Proposition 2 (p.~235) is the inequality $\\mathbb{E}[\\widehat{\\operatorname{Var}}_C(\\hat t)]\\ge\\operatorname{Var}(\\hat t)$ under $1/a_{kl}+1/b_{kl}=1$. Since the main text sets $\\pi_{kk}=\\pi_k$ on p.~233, I have $\\pi_{ks,ks}=\\pi_{ks}=1/M$. Page A2 continues with $\\epsilon_{ks}$.", "problem": "I identify four local slips. In (a) the inequality of Corollary 2 follows from Proposition 2, whose hypothesis the proof checks. In (c) the first case, taken literally for a unit paired with itself, gives $\\pi_{ks,ks}=0$ in place of $1/M$. In (d) the index $i$ is not defined.", "what_survives": "These slips do not affect Corollary 2 or the simulation design."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E006", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the caption of Figure A3 and its first two panels, for N = 8, m = 2 and for N = 40, m = 2, each plotting E(V) against rho. A red box marks the legend entry 'Conservative (HT)' in the first panel. In both panels the short-dashed curve of open circles that this entry names lies below the solid curve of filled circles labelled 'True'.", "caption": "Supplement, Figure A3, p. A10 (supplement PDF p. 10), caption and first row of panels. The marked legend entry calls the uncorrected series conservative; its curve, the short-dashed line of open circles, lies below the solid “True” curve.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E007.png"}, {"alt": "Crop of four lines of the supplement. Red boxes mark the label 'Conservative (HT)', which is broken across the second and third lines as 'Con-' and 'servative (HT)'.", "caption": "Supplement, Section C, p. A4 (supplement PDF p. 4). The marked words give the same label in the text.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In the periodic simulation the uncorrected Horvitz–Thompson series is not conservative; it should be labelled “Uncorrected HT”, not “Conservative (HT)”.", "id": "P027-E007", "location": "Supplement, Figure A3, legend of the first panel, p. A10 (supplement PDF p. 10); Section C, p. A4 (supplement PDF p. 4), paragraph beginning “Table A3 and Figure A3 display”", "pdf_page": 10, "printed_page": "A10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Supplement, p.~A10, legend of Figure A3: ``\\hl{Conservative} (HT)''.\n\nSupplement, p.~A4: ``Each graph plots the true variance (Var or ``True''), expected value of $(1/N^2)\\widehat{\\operatorname{Var}}(\\hat t)$ ($\\widehat{\\operatorname{Var}}_{HT}$ or ``\\hl{Conservative} (HT)''), expected value of $(1/N^2)\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ ($\\widehat{\\operatorname{Var}}_{C2}$ or ``Conservative ($A^*$)''), and expected value of $\\widehat{\\operatorname{Var}}_{WR}(\\hat\\mu_{HT})$ (``With-repl.'') over values of $\\rho$ and for varying sample sizes ($m$) and sampling proportions ($m/N$).''", "corrected": "Legend of Figure A3: ``\\hl{Uncorrected HT}''.\n\nSupplement, p.~A4: ``\\ldots expected value of $(1/N^2)\\widehat{\\operatorname{Var}}(\\hat t)$ ($\\widehat{\\operatorname{Var}}_{HT}$ or ``\\hl{Uncorrected HT}''), \\ldots''", "evidence": "This is a recalculation from Table A3. The printed mean of $\\widehat{\\operatorname{Var}}_{HT}$ is below the printed true variance in each of the 60 rows, and each printed nominal bias is negative. In row 10, for example, $1227.24-1397.05=-169.81$.", "problem": "The label calls the uncorrected estimator conservative, but in this simulation its mean is below the true variance in every row of Table A3. The curve is identified by its formula, and the same page says that the estimator may have bias of either sign. Only the descriptor is wrong. Figure A2 uses the same descriptor.", "what_survives": "The estimator formulas, the tabulated values and the comparisons are unaffected. The sign of the bias in this design is derived in \\fid{P027-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E007", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the caption of Figure A2 and its first two panels, for N = 8, m = 2 and for N = 40, m = 2, each plotting E(V) from 0 to 2000 against rho. Red boxes mark the upper right corner of each panel, where the dashed curve of open circles labelled 'Conservative (HT)' rises through the top edge of the frame.", "caption": "Supplement, Figure A2, p. A9 (supplement PDF p. 9), caption and first row of panels. The marked corners are where the dashed Horvitz–Thompson curve leaves the frame; the markers at $\\rho=0.99$ for $N=8$ and at $\\rho=0.95$ and $0.99$ for $N=40$ are not drawn. The same happens at $\\rho=0.99$ in the panels for $N=600$ and $N=1200$, which are not shown here.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E008.png"}], "formal_release": null, "formal_statements": [], "headline": "Five Horvitz–Thompson markers at $\\rho=0.95$ and $\\rho=0.99$ lie above the plotting range of Figure A2 and are not drawn; Table A2 gives their values.", "id": "P027-E008", "location": "Supplement, Figure A2, p. A9 (supplement PDF p. 9); the values are in Table A2, p. A7", "pdf_page": 9, "printed_page": "A9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Figure A2: Expected value of estimators of sampling variance of $\\hat\\mu_{HT}$ under PPS systematic sampling for different values of $\\rho$, sample sizes ($m$), and sampling proportions ($m/N$): random size case.'' The vertical axes of the six panels are labelled from 0 to 2000, and the plotting area ends a little above 2000. The dashed curve labelled ``Conservative (HT)'' \\hl{leaves the top of the frame} in the panels for $(N,m)=(8,2)$, $(40,2)$, $(600,30)$ and $(1200,60)$, with no marker or note for the points that fall outside.", "corrected": "The vertical range of every panel is extended to include 2667.59, or a note is added under the figure: ``The Horvitz-Thompson values 2236.50 ($N=8$, $\\rho=0.99$), 2473.52 and 2667.59 ($N=40$, $\\rho=0.95$ and $0.99$), 2383.53 ($N=600$, $\\rho=0.99$) and 2350.79 ($N=1200$, $\\rho=0.99$) \\hl{lie above the plotting range}; see Table A2.''", "intended_meaning": "Nothing printed is false. The curves visibly run out of the frame, and Table A2 lists every value. The five markers that are not drawn have the values 2236.50 ($N=8$, $m=2$, $\\rho=0.99$), 2473.52 and 2667.59 ($N=40$, $m=2$, $\\rho=0.95$ and $0.99$), 2383.53 ($N=600$, $m=30$, $\\rho=0.99$) and 2350.79 ($N=1200$, $m=60$, $\\rho=0.99$). The markers at $\\rho=0.95$ for $N=600$ (2173.08) and $N=1200$ (2171.46) exceed 2000 and remain visible."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E008", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading 'References' and the two-line entry for Hansen, Hurwitz and Madow (1953). A red box marks the words 'Methods and Theory.' at the start of the second line of the entry.", "caption": "First entry of the References, p. 240 (PDF p. 12). The marked subtitle should read “Methods and Applications”.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "Volume I of Hansen, Hurwitz and Madow (1953) is subtitled “Methods and Applications”, not “Methods and Theory”.", "id": "P027-E009", "location": "References, entry for Hansen, Hurwitz and Madow (1953), p. 240 (PDF p. 12)", "pdf_page": 12, "printed_page": "240", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Hansen, M.H., Hurwitz, W.N. and Madow, W.G. (1953). \\emph{Sample Survey Methods and Theory: Volume 1 \\hl{Methods and Theory}}. New York: John Wiley \\& Sons, Inc.''", "corrected": "``Hansen, M.H., Hurwitz, W.N. and Madow, W.G. (1953). \\emph{Sample Survey Methods and Theory: Volume \\hl{I: Methods and Applications}}. New York: John Wiley \\& Sons, Inc.''", "evidence": "I checked the subtitle against public bibliographic sources. The Helka library catalogue entry for the 1953 New York Wiley edition (\\url{https://www.finna.fi/Record/helka.998190593506253}) reads ``Sample survey methods and theory. Volume 1, Methods and applications'', and the publisher's listing of the 1993 Wiley Classics Library reprint has the same subtitle. I did not consult the book itself.", "problem": "The reference prints the wrong subtitle for Volume I. The book is still identified by its authors, title, volume, year and publisher.", "what_survives": "Everything else is unaffected, since nothing depends on this entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-E009", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": {"class": "error", "severity": "Moderate"}, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the four-line paragraph that begins 'If any units k, l are in clusters'. Red boxes mark the words 'Combining units will tend to reduce the bias of the variance estimator', which run from the second line into the third.", "caption": "Section 3.1, last paragraph, p. 237 (PDF p. 9). The marked words state that combining units will tend to reduce the bias of the variance estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-N002.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T3", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T3"}, {"id": "P027-T4", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T4"}], "headline": "Totalling units that are always selected together need not reduce the bias of the corrected variance estimator: with $a_{kl}=b_{kl}=2$ it can raise the bias when the clusters differ in size, and it cannot raise the bias when every two clusters with zero joint inclusion probability have the same size.", "id": "P027-N002", "location": "Section 3.1, paragraph after the proof of Corollary 1, p. 237 (PDF p. 9)", "pdf_page": 9, "printed_page": "237", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P027-T3", "P027-T4"], "text": {"as_printed": "``If any units $k, l$ are in clusters (\\emph{i.e.}, $\\Pr(I_k \\neq I_l) = 0$), these units should be totaled into one larger unit before estimation. Combining units \\hl{will tend to reduce the bias} of the variance estimator because only pairs of cluster-level totals will be included in $A^*$, as opposed to all constituent pairs.''", "corrected": "An optional clarification after the original recommendation is:\n\n``For $a_{kl}=b_{kl}=2$, the prediction that totalling will tend to reduce bias is \\hl{a heuristic, not a guarantee}. Totalling can increase the bias when cluster sizes differ. If every two clusters with zero joint inclusion probability have \\hl{equal sizes}, the bias does not increase.''", "intended_meaning": "The paragraph gives a heuristic reason to expect smaller bias. The words ``will tend to'' allow exceptions; they do not assert that totalling reduces bias for every admissible design and fixed outcome vector. The number of pairs does not settle the direction of the change, because totalling also replaces sums of squared outcomes by squared totals, and these can be larger. For the choice $a_{kl}=b_{kl}=2$ of Section 3.2, totalling leaves the Horvitz--Thompson total, its variance, the uncorrected variance estimator and the term $A$ unchanged. The whole change in the bias of $\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ then comes from $A^*$, and this change can have either sign. In a population of three units with outcomes $(1,1,1)$, in which units 1 and 2 are always selected together, the bias rises from 8 to 9 after totalling. When all outcomes equal the same nonzero constant, totalling raises the bias whenever two clusters with zero joint inclusion probability differ in size. Designs that draw one cluster per stratum are of this kind when clusters within a stratum differ in size. The paragraph specifies neither a class of populations nor a distribution defining this tendency. The counterexample establishes a limitation of the advice, not falsity of its hedged prediction.\n\nThe evidence is a derivation and an exact counterexample. I partition the population into groups $G_g$ of units that are always selected together, and I write $n_g$ for their sizes, $Y_g=\\sum_{k\\in G_g}y_k$ for their totals and $S_g=\\sum_{k\\in G_g}y_k^2$ for their sums of squares. Let $\\mathcal{E}$ be the set of unordered pairs $\\{g,h\\}$ of groups with zero joint inclusion probability. For $a_{kl}=b_{kl}=2$, summing the terms of $A^*$ over each pair of groups gives\n\\[\nA^*_{\\text{units}}=\\sum_{\\{g,h\\}\\in\\mathcal{E}}\\left(n_hS_g+n_gS_h\\right),\\qquad A^*_{\\text{totalled}}=\\sum_{\\{g,h\\}\\in\\mathcal{E}}\\left(Y_g^2+Y_h^2\\right).\n\\]\nSince $A$ is the same in both cases, the bias $A+A^*$ before totalling minus the bias after totalling is\n\\[\n\\sum_{\\{g,h\\}\\in\\mathcal{E}}\\left(n_hS_g+n_gS_h-Y_g^2-Y_h^2\\right).\n\\]\nIf $y_k=v$ for all $k$, then $S_g=n_gv^2$ and $Y_g=n_gv$, and this difference equals $-v^2\\sum_{\\{g,h\\}\\in\\mathcal{E}}(n_g-n_h)^2\\le 0$. If $n_g=n_h$ for every pair in $\\mathcal{E}$, each summand is nonnegative, because $n_gS_g\\ge Y_g^2$.\n\nIn the three-unit example the possible samples are $\\{1,2\\}$ and $\\{3\\}$, each with probability $1/2$, and $y=(1,1,1)$. The true variance of $\\hat t$ is 1.\n\n\\begin{audittable}{llrrrrr}{The three-unit example, enumerated in full. The bias of the corrected estimator is $9-1=8$ on the original units and $10-1=9$ after units 1 and 2 are totalled.}\nSample & Probability & $\\hat t$ & $\\widehat A^*$, units & $\\widehat A^*$, totalled & $\\widehat{\\operatorname{Var}}_{C2}$, units & $\\widehat{\\operatorname{Var}}_{C2}$, totalled \\\\\n$\\{1,2\\}$ & $1/2$ & 4 & 4 & 8 & 12 & 16 \\\\\n$\\{3\\}$ & $1/2$ & 2 & 4 & 2 & 6 & 4 \\\\\nExpectation & & 3 & 4 & 5 & 9 & 10 \\\\\n\\end{audittable}\n\nFurther exact cases show both directions. In each, exactly one cluster is selected, with probability $1/2$ unless stated, and a bar separates the clusters.\n\n\\begin{audittable}{lrrr}{Bias of $\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ before and after totalling, by exact enumeration.}\nOutcomes by cluster & $\\operatorname{Var}(\\hat t)$ & Bias, units & Bias, totalled \\\\\n$(1,1\\mid 1)$ & 1 & 8 & 9 \\\\\n$(1,1\\mid 0)$ & 4 & 2 & 4 \\\\\n$(1,1\\mid -1/4)$ & $81/16$ & $9/8$ & $49/16$ \\\\\n$(1,1,1,1\\mid 1)$ & 9 & 16 & 25 \\\\\n$(3,1,2,5,4\\mid 2)$, probabilities $1/3$ and $2/3$ & 392 & 135 & 289 \\\\\n$(1,-1\\mid 2,-2)$ & 0 & 20 & 0 \\\\\n\\end{audittable}\n\nPropositions 1 and 2 and Corollaries 1 to 3 are unaffected. The estimator $\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ is conservative with and without totalling, and no proof uses this paragraph. The practical advice is heuristic; the example supplies an important exception without refuting the claimed tendency. The phrase ``will tend to'' is not refuted as a statement about typical populations, since the paper specifies no distribution of outcomes or cluster sizes, and totalling does reduce the bias in some cases. Since the groups in the periodic designs of the supplement have equal sizes, totalling cannot raise the bias there (see \\fid{P027-N005})."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-N002", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of rows 29 to 60 of a table of numbers with fifteen columns. Red boxes mark, in rows 30, 40, 50 and 60, the two adjacent cells that hold the mean of the uncorrected estimator and the mean of the corrected estimator: 1412.15 and 2029.28, 1564.34 and 2396.58, 1430.59 and 2032.07, 1562.38 and 2395.86.", "caption": "Supplement Table A3, p. A8 (supplement PDF p. 8), rows 29 to 60. The columns are the row number, $N$, $m$, $\\rho$, $\\bar y$, the true variance, the means of $\\widehat{\\operatorname{Var}}_{HT}$, $\\widehat{\\operatorname{Var}}_{C2}$ and $\\widehat{\\operatorname{Var}}_{WR}$, their relative biases and their nominal biases. The marked cells are the means of $\\widehat{\\operatorname{Var}}_{HT}$ and $\\widehat{\\operatorname{Var}}_{C2}$ in rows 30, 40, 50 and 60.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-N004.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T9", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T9"}, {"id": "P027-T19", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T19"}], "headline": "In the four rows of Table A3 with $\\rho=0.99$ and $N\\ge 120$, the printed means of $\\widehat{\\operatorname{Var}}_{HT}$ and $\\widehat{\\operatorname{Var}}_{C2}$ are overwhelmingly inconsistent with the exact-expectation simulation that Section A describes; the expectations under the stated model suggest that the $\\widehat{\\operatorname{Var}}_{HT}$ means are 1 to 5 per cent too high.", "id": "P027-N004", "location": "Supplement Table A3, p. A8 (supplement PDF p. 8), rows 30, 40, 50 and 60, columns $\\widehat{\\operatorname{Var}}_{HT}$ and $\\widehat{\\operatorname{Var}}_{C2}$ under “Mean from sims.”; compare Section A, p. A1", "pdf_page": 8, "printed_page": "A8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P027-T19", "P027-T9"], "text": {"as_printed": "Supplement, p.~A1: ``In the simulations, expected values of $\\widehat{\\operatorname{Var}}(\\hat t)$ and $\\widehat{\\operatorname{Var}}_C(\\hat t)$ were computed exactly using expressions from Propositions 1 and 2 in the main text.''\n\nSupplement, Table A3 (``Simulation results for PPS systematic sampling: periodic size case''), four of its rows:\n\n\\begin{audittable}{rrrrrrrrr}{Rows 30, 40, 50 and 60 of Table A3 as printed: the true variance, and the mean and the relative bias of $\\widehat{\\operatorname{Var}}_{HT}$ and of $\\widehat{\\operatorname{Var}}_{C2}$.}\nRow & $N$ & $m$ & $\\rho$ & True Var & Mean, HT & Mean, C2 & Rel. bias, HT & Rel. bias, C2 \\\\\n30 & 120 & 30 & 0.99 & 1578.87 & \\hl{1412.15} & \\hl{2029.28} & $-0.11$ & 0.29 \\\\\n40 & 600 & 30 & 0.99 & 1587.01 & \\hl{1564.34} & \\hl{2396.58} & $-0.01$ & 0.51 \\\\\n50 & 240 & 60 & 0.99 & 1581.13 & \\hl{1430.59} & \\hl{2032.07} & $-0.10$ & 0.29 \\\\\n60 & 1200 & 60 & 0.99 & 1586.32 & \\hl{1562.38} & \\hl{2395.86} & $-0.02$ & 0.51 \\\\\n\\end{audittable}", "corrected": "A note under Table A3: ``In rows 30, 40, 50 and 60 the means of $\\widehat{\\operatorname{Var}}_{HT}$ and $\\widehat{\\operatorname{Var}}_{C2}$ are \\hl{statistically incompatible with averages of exact design expectations} for the design and outcome model of Section C with \\hl{independent Gaussian errors}; the lower-tail probability for each row is below $10^{-56}$. Under that design and model the ratio of the expected bias of $\\widehat{\\operatorname{Var}}_{HT}$ to the expected true variance is \\hl{$-5/36$} when $N/m=4$ and \\hl{$-1/36$} when $N/m=20$.''\n\nFor the stated design, these entries and the matching points of Figure A3 are to be regenerated from the original populations with the exact probabilities $p_r=1/(3q)$ and $5/(3q)$ of the $q$ possible samples, and the number of generated populations is to be stated. If the simulation was run differently, with another ordering of the sizes, another noise model, or averages of simulated estimates in place of exact expectations, then the description in Sections A and C needs the correction instead.", "evidence": "The evidence is a derivation with explicit probability bounds; no simulation was run. Let $q=N/m$, and let $z$ be the demeaned outcomes, $Y_r$ their totals over the $q$ possible samples, $p_r$ the probabilities of these samples and $Q=\\sum_iz_i^2$. On the scale of the mean, the true variance, the expectation $H$ of the uncorrected estimator and the expectation $C$ of the unit-level corrected estimator are\n\\[\nV=\\frac{1}{N^2}\\sum_r\\frac{Y_r^2}{p_r},\\qquad H=V-\\frac{1}{N^2}\\sum_rY_r^2,\\qquad C=H+\\frac{(N-m)Q}{N^2}.\n\\]\nI set $R=5(q-1)/(9q-5)$ and $T=C-(1+R)H$. Decomposing the demeaned vectors into four orthogonal subspaces diagonalizes $T$, and its coefficient on the direction $x-6$ is zero. With $z=\\beta(x-6)+\\epsilon-\\bar\\epsilon$ and independent $\\mathrm{N}(0,\\sigma^2)$ errors, $\\sigma^2=900(1-\\rho^2)$, the signal and its cross terms with the noise drop out, and I obtain\n\\[\nT=a\\left[X_W+c\\left(X_G-X_L\\right)\\right],\\qquad a=\\frac{(N-m)\\sigma^2}{N^2},\\quad c=\\frac{6q}{9q-5},\n\\]\nwhere $X_W$, $X_G$ and $X_L$ are independent chi-squared variables with $N-q$, $q/2-1$ and $q/2-1$ degrees of freedom. This implies that $\\mathbb{E}T=a(N-q)>0$ and $T\\ge -acX_L$. A Markov bound on $\\exp(X_L/4)$ then bounds the probability of the printed values, allowing an error of 0.005 in each printed mean and any number $K\\ge 1$ of generated populations.\n\n\\begin{audittable}{rrrrrrr}{The four rows: the contrast $T$ formed from the printed means, the upper bound on its probability, and the expectations $\\mathbb{E}H$ and $\\mathbb{E}C$ under the stated model beside the printed means of $\\widehat{\\operatorname{Var}}_{HT}$ and $\\widehat{\\operatorname{Var}}_{C2}$.}\nRow & $T$ & Probability bound & $\\mathbb{E}H$, model & Mean, HT & $\\mathbb{E}C$, model & Mean, C2 \\\\\n30 & $-66.17$ & $10^{-56}$ & 1356.54 & 1412.15 & 2025.91 & 2029.28 \\\\\n40 & $-16.97$ & $10^{-64}$ & 1542.08 & 1564.34 & 2395.65 & 2396.58 \\\\\n50 & $-90.74$ & $10^{-157}$ & 1361.89 & 1430.59 & 2034.08 & 2032.07 \\\\\n60 & $-14.67$ & $10^{-111}$ & 1542.87 & 1562.38 & 2397.15 & 2395.86 \\\\\n\\end{audittable}\n\nThe model expectations use the sample standard deviation of $x$ in $\\beta$; they are not replacements for the unknown realized averages. The bound does not use the ``True'' or with-replacement columns. It is a statement of probability and not of impossibility, since Gaussian noise is unbounded. For the other rows examined the same bound is uninformative (rows 20, 21 and 41) or weak (row 10, at most $2^{-4}$).", "problem": "Section A says that the expected values of the two estimators were computed exactly for each generated population. In the periodic design one combination of the two expectations does not depend on the signal $\\beta x$ at all: it is a known quadratic form in the Gaussian noise, with positive mean. In rows 30, 40, 50 and 60 the combination formed from the printed means equals $-66.17$, $-16.97$, $-90.74$ and $-14.67$. With independent Gaussian errors at the stated scale, values at least this negative have probability below $10^{-56}$, whatever the number of generated populations and after allowing for rounding. I conclude that at least one of the two columns, or the description of how they were computed, is wrong in these rows. It is not settled which. The expectations under the stated model are close to the printed $\\widehat{\\operatorname{Var}}_{C2}$ means and 1 to 5 per cent below the printed $\\widehat{\\operatorname{Var}}_{HT}$ means. On that account the table understates the negative bias of the uncorrected estimator in these rows: it prints relative biases of $-0.11$, $-0.01$, $-0.10$ and $-0.02$, against about $-0.14$ when $N/m=4$ and $-0.03$ when $N/m=20$.", "what_survives": "Proposition 1 and the conservativeness of the corrected estimator stand; the check uses them. The qualitative comparison in the affected rows of Table A3 survives: the uncorrected estimator is slightly below the true variance, the corrected estimator is above it, and the with-replacement estimator is far below it. Tables A1 and A2 are not touched by this check. No corrected entries are supplied, because the generated populations and the code are not available. The explanation that these entries appear to support is qualified in \\fid{P027-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-N004", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of the main text. Red boxes mark the words 'these units should be totaled into one larger unit before estimation.', which run from the end of the second line into the third.", "caption": "Main text, Section 3.1, p. 237 (PDF p. 9). The marked words are the instruction to total units that are always selected together.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-N005.png"}, {"alt": "Crop of four lines of the supplement. Red boxes mark the words 'is generally quite conservative.', which follow the symbol for the corrected variance estimator.", "caption": "Supplement, Section C, p. A4 (supplement PDF p. 4). The marked words describe the $\\widehat{\\operatorname{Var}}_{C2}$ results of the periodic simulation.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-N005-b.png"}, {"alt": "Crop of rows 20 to 52 of a table of numbers with fifteen columns. Red boxes mark, in rows 21, 31, 41 and 51, the mean of the corrected estimator (653.92, 891.43, 622.28, 861.11) and its relative bias (14.37, 14.82, 25.89, 29.13).", "caption": "Supplement Table A3, p. A8 (supplement PDF p. 8), rows 20 to 52. The columns are the row number, $N$, $m$, $\\rho$, $\\bar y$, the true variance, the means of $\\widehat{\\operatorname{Var}}_{HT}$, $\\widehat{\\operatorname{Var}}_{C2}$ and $\\widehat{\\operatorname{Var}}_{WR}$, their relative biases and their nominal biases. The marked cells are the mean and the relative bias of $\\widehat{\\operatorname{Var}}_{C2}$ in rows 21, 31, 41 and 51.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-N005-c.png"}, {"alt": "Two panels, one for sampling fraction one quarter and one for one twentieth, plot relative bias on a logarithmic axis with labelled ticks from 0.1 to 30 against rho from 0 to 1. Blue open markers joined by lines are the printed values for three designs in each panel. For the designs with m equal to 30 or 60 they fall from about 14 to 29 at rho 0.05 to about 0.3 or 0.5 at rho 0.99. A grey band marks the range possible after totalling, one sixth to five sixths in the left panel and three tenths to three halves in the right panel, and an orange dashed line marks the ratio of outcome-averaged bias to outcome-averaged variance, five eighteenths on the left and one half on the right. The printed values for the designs with m equal to 2 lie inside the band.", "caption": "Relative bias of $\\widehat{\\operatorname{Var}}_{C2}$ in the 60 rows of supplement Table A3, formed as (printed mean $-$ printed true variance)/printed true variance, on a logarithmic scale. The shaded band is the range that the estimator can reach after always-co-selected units are totalled; the dashed line is the ratio of outcome-averaged bias to outcome-averaged variance derived for the totalled estimator under the stated outcome model. The printed values for the designs with $m=30$ and $m=60$ lie far above the band at low $\\rho$.", "url": "https://pmaronow.github.io/llm-audit/works/P027/exhibits/P027-N005-c2-relative-bias.png"}], "formal_release": null, "formal_statements": [{"id": "P027-T5", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T5"}, {"id": "P027-T6", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T6"}, {"id": "P027-T9", "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-T9"}], "headline": "The $\\widehat{\\operatorname{Var}}_{C2}$ results of the periodic simulation (Table A3 and Figure A3) are for the estimator applied to the original units, without the totalling of always-co-selected units that p. 237 of the main text recommends, and the supplement should say so.", "id": "P027-N005", "location": "Main text, Section 3.1, p. 237 (PDF p. 9), paragraph beginning “If any units”; supplement, Section C, p. A4 (supplement PDF p. 4); supplement Table A3, p. A8 (supplement PDF p. 8), columns $\\widehat{\\operatorname{Var}}_{C2}$", "pdf_page": 9, "printed_page": "237", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P027-T5", "P027-T6", "P027-T9"], "text": {"as_printed": "Main text, p.~237: ``If any units $k, l$ are in clusters (\\emph{i.e.}, $\\Pr(I_k \\neq I_l) = 0$), these units should be totaled into one larger unit before estimation.''\n\nSupplement, p.~A4: ``As expected, $\\widehat{\\operatorname{Var}}_{WR}(\\hat\\mu_{HT})$ tends to greatly understate the sampling variance of $\\hat\\mu_{HT}$, while $\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ is generally quite conservative.''\n\nSupplement, Table A3 (``Simulation results for PPS systematic sampling: periodic size case.''), four of its rows:\n\n\\begin{audittable}{rrrrrrrr}{Rows 21, 31, 41 and 51 of Table A3 as printed: the true variance, the means of $\\widehat{\\operatorname{Var}}_{HT}$ and $\\widehat{\\operatorname{Var}}_{C2}$, and the relative bias of $\\widehat{\\operatorname{Var}}_{C2}$.}\nRow & $N$ & $m$ & $\\rho$ & True Var & Mean, HT & Mean, C2 & Rel. bias, C2 \\\\\n21 & 120 & 30 & 0.05 & 42.55 & 36.78 & \\hl{653.92} & \\hl{14.37} \\\\\n31 & 600 & 30 & 0.05 & 56.35 & 54.90 & \\hl{891.43} & \\hl{14.82} \\\\\n41 & 240 & 60 & 0.05 & 23.14 & 20.39 & \\hl{622.28} & \\hl{25.89} \\\\\n51 & 1200 & 60 & 0.05 & 28.58 & 27.82 & \\hl{861.11} & \\hl{29.13} \\\\\n\\end{audittable}", "corrected": "Supplement, p.~A4: ``As expected, $\\widehat{\\operatorname{Var}}_{WR}(\\hat\\mu_{HT})$ tends to greatly understate the sampling variance of $\\hat\\mu_{HT}$, while $\\widehat{\\operatorname{Var}}_{C2}(\\hat t)$ is generally quite conservative. The $\\widehat{\\operatorname{Var}}_{C2}$ results here \\hl{illustrate the estimator applied to the original units, rather than the estimator after combining always-co-selected units} as recommended on p.~237 of the main text.'' The same note belongs under Table A3 and Figure A3.\n\nFor the totalled estimator in these designs, with $q=N/m$ equal to 4 or 20 and demeaned outcomes, the expectation on the scale of the mean is $\\bigl(V+(q-2)S\\bigr)/N^2$, and its relative bias, \\hl{when $V>0$}, lies between $(q-2)/(3q)$ and $5(q-2)/(3q)$.", "evidence": "The evidence is a derivation, a bound that holds for every population, and an exact example. Let $z_k=\\tilde y_k$ be the demeaned outcomes. Since the sizes alternate between 2 and 10 and $q$ is even, the sample is one of the groups $G_c=\\{c,c+q,\\dots,c+(m-1)q\\}$, $c=1,\\dots,q$, which are selected with probability $p_c=1/(3q)$ for odd $c$ and $p_c=5/(3q)$ for even $c$. I define $Y_c=\\sum_{k\\in G_c}z_k$, $S=\\sum_cY_c^2$ and $Q=\\sum_kz_k^2$. On the scale of the total, I have $V=\\operatorname{Var}(\\hat t)=\\sum_cY_c^2/p_c$, the uncorrected estimator has expectation $H=V-S$, and\n\\[\n\\mathbb{E}\\,\\widehat{\\operatorname{Var}}_{C2}(\\hat t)=V-S+m(q-1)Q\\ \\text{ on the original units},\\qquad \\mathbb{E}\\,\\widehat{\\operatorname{Var}}_{C2}(\\hat t)=V+(q-2)S\\ \\text{ after totalling}.\n\\]\nThe first exceeds the second by $(q-1)(mQ-S)\\ge 0$. Since $3qS/5\\le V\\le 3qS$, for $V>0$ the relative bias $(q-2)S/V$ of the totalled estimator satisfies\n\\[\n\\frac{q-2}{3q}\\le\\frac{(q-2)S}{V}\\le\\frac{5(q-2)}{3q}.\n\\]\nAllowing an error of 0.005 in each printed entry, the printed $\\widehat{\\operatorname{Var}}_{C2}$ mean lies above the upper bound in 26 of the 60 rows, all with $m\\ge 30$ and $\\rho\\le 0.6$. Under the stated outcome model with fixed $\\beta$ and independent mean-zero errors of equal variance, the ratio of the outcome-averaged bias of the totalled estimator to the outcome-averaged variance is $(5/9)(1-2/q)$, that is, $5/18$ for $q=4$ and $1/2$ for $q=20$.\n\nAn exact example for $N=120$, $m=30$ takes $z_k=x_k-6+2\\lfloor k/4\\rfloor-29$ for $k=0,\\dots,119$, with $x_k=2$ for even $k$ and $x_k=10$ for odd $k$. On the scale of the mean the true variance is $144/5=28.8$, the unit-level estimator has expectation $5231/20=261.55$, and the totalled estimator has expectation $184/5=36.8$.\n\nSince $S=V-H$, the totalled estimator has expectation $V+(q-2)(V-H)$. If the printed true variance and the printed mean of $\\widehat{\\operatorname{Var}}_{HT}$ are averages of $V$ and $H$ over the same populations, then the mean of the totalled estimator follows from those two columns.\n\n\\begin{audittable}{rrrrrr}{The four rows quoted above: the printed relative bias of $\\widehat{\\operatorname{Var}}_{C2}$, the range possible for the totalled estimator, and the mean and relative bias of the totalled estimator implied by the printed true variance and the printed mean of $\\widehat{\\operatorname{Var}}_{HT}$.}\nRow & $q$ & Printed rel. bias & Range after totalling & Implied mean & Implied rel. bias \\\\\n21 & 4 & 14.37 & $1/6$ to $5/6$ & 54.09 & 0.27 \\\\\n31 & 20 & 14.82 & $3/10$ to $3/2$ & 82.45 & 0.46 \\\\\n41 & 4 & 25.89 & $1/6$ to $5/6$ & 28.64 & 0.24 \\\\\n51 & 20 & 29.13 & $3/10$ to $3/2$ & 42.26 & 0.48 \\\\\n\\end{audittable}\n\nOn these implied values the totalled estimator is closer to the true variance than the original-unit with-replacement estimator in 41 of the 60 rows; the printed column is closer in 20. I do not have the simulation code or the generated populations, so I cannot confirm that the printed entries are correctly computed values of the unit-level estimator.", "problem": "In each of the six designs of Table A3 the systematic sample is one of $q=N/m$ disjoint groups of $m$ units, and all units of a group are always selected together. This is the case in which p.~237 says that the units ``should be totaled into one larger unit before estimation.'' The $\\widehat{\\operatorname{Var}}_{C2}$ entries of Table A3 cannot be values of the estimator after such totalling. For any demeaned population with positive design variance the relative bias of the totalled estimator lies between $1/6$ and $5/6$ when $q=4$ and between $3/10$ and $3/2$ when $q=20$. The table prints 14.37, 14.82, 25.89 and 29.13 in the rows shown above. I therefore read these entries as results for the estimator applied to the $N$ original units; the supplement does not disclose this departure from the totalling recommendation. Taken as the performance of the procedure that the main text recommends, Table A3 and Figure A3 overstate its bias at low $\\rho$ by one to two orders of magnitude. The unit-level estimator is conservative, and ``generally quite conservative'' correctly describes the displayed values.", "what_survives": "The Horvitz--Thompson estimator, its variance and the uncorrected variance estimator are the same with and without totalling, and $\\widehat{\\operatorname{Var}}_{C2}$ is conservative in both forms. I retain the original-unit with-replacement estimator as the comparator. The claim on p.~240 of the main text, that the estimators perform favourably under substantial periodicity, is not weakened: in these designs totalling cannot raise the bias, and at $\\rho=0.99$ the printed relative biases (0.28 to 0.51) are close to the outcome-averaged ratios $5/18$ and $1/2$ derived for the totalled estimator. Only the reading of Table A3, Figure A3 and the sentence on p.~A4 changes: they describe the unit-level estimator, not the procedure recommended on p.~237. That recommendation is itself qualified in \\fid{P027-N002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P027/#P027-N005", "work": {"arxiv_id": null, "doi": null, "title": "Conservative Variance Estimation for Sampling Designs with Zero Pairwise Inclusion Probabilities", "url": "https://pmaronow.github.io/llm-audit/works/P027/"}, "work_id": "P027"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the proof of Proposition 1 from equation (2) through equation (3) to the three displayed inequalities. Red outlines surround two items in the sentence after equation (3). The first is the final less-than-or-equal-to 1 in the constraint 0 is at most EY sub 1 minus P sub c EY sub c (1), which is at most 1. The second is the symbol EY sub c (1) in the words: Then substituting the implied value of EY sub c (1) from Eq. (3). The third displayed inequality below begins (EY sub 1 + P sub c minus 1) / P sub c.", "caption": "Proof of Proposition 1, p. 678 (PDF p. 2), from equation (2) to the three displayed inequalities. The outlines mark the upper limit $1$ of the third constraint and the symbol $EY_c(1)$ in “the implied value of $EY_c(1)$”.", "url": "https://pmaronow.github.io/llm-audit/works/P028/exhibits/P028-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P028-T4", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T4"}, {"id": "P028-T5", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T5"}, {"id": "P028-T8", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T8"}], "headline": "In the proof of Proposition 1, the third constraint after equation (3) has upper limit $1 - P_c$ and not $1$, and the mean substituted out of equation (3) is $EY_c(0)$ and not $EY_c(1)$; the three displayed inequalities and the proposition are unchanged.", "id": "P028-E001", "location": "Proof of Proposition 1, the sentence between equation (3) and the three displayed inequalities, p. 678 (PDF p. 2)", "pdf_page": 2, "printed_page": "678", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P028-T4", "P028-T5", "P028-T8"], "text": {"as_printed": "Since $EY_n = [EY_1 - P_c EY_c(1)]/(1 - P_c)$, we may reframe the problem as\n\\[\nP_c EY_c(0) + EY_1 - P_c EY_c(1) = EY_0, \\qquad (3)\n\\]\nsuch that $0 \\le EY_c(0) \\le 1$, $0 \\le EY_c(1) \\le 1$, and $0 \\le EY_1 - P_c EY_c(1) \\le \\hl{1}$. Then substituting the implied value of $\\hl{EY_c(1)}$ from Eq.~(3) into the system of inequalities and rearranging terms, we obtain\n\\[\n\\begin{aligned}\n&0 \\le EY_c(1) \\le 1,\\\\\n&(EY_1 - EY_0)/P_c \\le EY_c(1) \\le 1 + (EY_1 - EY_0)/P_c,\\\\\n&(EY_1 + P_c - 1)/P_c \\le EY_c(1) \\le EY_1/P_c,\n\\end{aligned}\n\\]\nwhich yields the interval for $EY_c(1)$ asserted by the proposition. The interval for $EY_c(0)$ follows via substitution from the equality in Eq.~(3).", "corrected": "Since $EY_n = [EY_1 - P_c EY_c(1)]/(1 - P_c)$, we may reframe the problem as\n\\[\nP_c EY_c(0) + EY_1 - P_c EY_c(1) = EY_0, \\qquad (3)\n\\]\nsuch that $0 \\le EY_c(0) \\le 1$, $0 \\le EY_c(1) \\le 1$, and $0 \\le EY_1 - P_c EY_c(1) \\le \\hl{1 - P_c}$. Then substituting the implied value of $\\hl{EY_c(0)}$ from Eq.~(3) into the system of inequalities and rearranging terms, we obtain\n\\[\n\\begin{aligned}\n&0 \\le EY_c(1) \\le 1,\\\\\n&(EY_1 - EY_0)/P_c \\le EY_c(1) \\le 1 + (EY_1 - EY_0)/P_c,\\\\\n&(EY_1 + P_c - 1)/P_c \\le EY_c(1) \\le EY_1/P_c,\n\\end{aligned}\n\\]\nwhich yields the interval for $EY_c(1)$ asserted by the proposition. The interval for $EY_c(0)$ follows via substitution from the equality in Eq.~(3).", "evidence": "I give a derivation, with an exact example which shows that the printed system is not equivalent to the original one. Suppose that $0 < P_c < 1$. By (2), I have $EY_n = [EY_1 - P_c EY_c(1)]/(1 - P_c)$, and therefore\n\\[\n0 \\le EY_n \\le 1 \\iff 0 \\le EY_1 - P_c EY_c(1) \\le 1 - P_c \\iff \\frac{EY_1 + P_c - 1}{P_c} \\le EY_c(1) \\le \\frac{EY_1}{P_c}.\n\\]\nBy (3), I have $EY_c(0) = EY_c(1) - (EY_1 - EY_0)/P_c$, which implies that\n\\[\n0 \\le EY_c(0) \\le 1 \\iff \\frac{EY_1 - EY_0}{P_c} \\le EY_c(1) \\le 1 + \\frac{EY_1 - EY_0}{P_c}.\n\\]\nTogether with $0 \\le EY_c(1) \\le 1$, these are the three displayed inequalities. Since $(EY_1 + P_c - 1)/P_c = 1 - (1 - EY_1)/P_c$, their intersection is the interval for $EY_c(1)$ in Proposition 1.\n\nFor the example, let $P_c = 1/2$, $EY_0 = EY_1 = 3/4$ and $EY_c(0) = EY_c(1) = 0$. Equation (3) and the three printed constraints hold, since $EY_1 - P_c EY_c(1) = 3/4 \\le 1$. But the implied noncomplier mean is $EY_n = (3/4)/(1/2) = 3/2 > 1$. For these values of $P_c$, $EY_0$ and $EY_1$, I obtain the interval $[0, 1]$ for $EY_c(1)$ from the printed system, and the interval $[1/2, 1]$ of Proposition 1 from the corrected system.", "problem": "The third constraint restates $0 \\le EY_n \\le 1$ after the substitution $EY_n = [EY_1 - P_c EY_c(1)]/(1 - P_c)$. Multiplying by $1 - P_c$, I obtain the upper limit $1 - P_c$, and the sentence prints $1$. For $0 < P_c < 1$ the printed constraint is weaker than the one it replaces. It implies only the lower bound $(EY_1 - 1)/P_c$ for $EY_c(1)$, and not the lower bound $(EY_1 + P_c - 1)/P_c$ in the third line of the display. The next sentence names the wrong mean. Since the system that follows is written in $EY_c(1)$, the mean eliminated through equation (3) is $EY_c(0)$. The display itself uses the intended constraint and the intended substitution.", "what_survives": "Proposition 1, its sharpness, the three displayed inequalities and the LATE identity are unaffected. The bounds of the application are also unaffected. From the reported $0.45$, $0.51$ and $0.21$ the intervals are $[0, 5/7]$ and $[2/7, 1]$, printed as $[0, 0.71]$ and $[0.29, 1]$. The correction changes one limit and one symbol in the proof and has no cost. The case $P_c = 1$ is treated in \\fid{P028-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-E001", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2012.11.012", "title": "Sharp Bounds for Complier Average Potential Outcomes in Experiments with Noncompliance and Incomplete Reporting", "url": "https://pmaronow.github.io/llm-audit/works/P028/"}, "work_id": "P028"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first six lines of the proof of Proposition 1, with equations (1) and (2). Red outlines surround two items. The first is the definition EY sub n is identically E of Y given D(1) = D(0), in the first line of the proof. The second is the divisor, slash open parenthesis 1 minus P sub c close parenthesis, in the sentence: Since EY sub n = [EY sub 1 minus P sub c EY sub c (1)] / (1 minus P sub c), we may reframe the problem as.", "caption": "Proof of Proposition 1, p. 678 (PDF p. 2). The outlines mark the definition of the noncomplier mean $EY_n$ and the division by $1 - P_c$.", "url": "https://pmaronow.github.io/llm-audit/works/P028/exhibits/P028-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P028-T6", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T6"}, {"id": "P028-T7", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T7"}, {"id": "P028-T8", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T8"}, {"id": "P028-T9", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T9"}], "headline": "Since the proof of Proposition 1 defines a mean among noncompliers and divides by $1 - P_c$, it needs a separate case for full compliance, $P_c = 1$, which the assumptions allow; the printed intervals are correct in that case.", "id": "P028-E002", "location": "Proof of Proposition 1, first sentence and the sentence after equation (2), p. 678 (PDF p. 2); assumptions (iii) and (iv), Section 2, p. 678", "pdf_page": 2, "printed_page": "678", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P028-T6", "P028-T7", "P028-T8", "P028-T9"], "text": {"as_printed": "\\emph{Section 2, p.~678.} (iii) non-zero first-stage effect: $E[D_i(1) - D_i(0)] \\neq 0$; (iv) monotonicity: $\\forall i$, $D_i(1) \\geq D_i(0)$.\n\n\\emph{Proof of Proposition 1, p.~678.} Define $\\hl{EY_n \\equiv \\mathrm{E}[Y \\mid D(1) = D(0)]}$, $EY_c(0) \\equiv \\mathrm{E}[Y(0) \\mid D(1) > D(0)]$ and $EY_c(1) \\equiv \\mathrm{E}[Y(1) \\mid D(1) > D(0)]$. By the law of total probability,\n\\[\n\\begin{aligned}\nP_c EY_c(0) + (1 - P_c) EY_n &= EY_0, \\qquad (1)\\\\\nP_c EY_c(1) + (1 - P_c) EY_n &= EY_1, \\qquad (2)\n\\end{aligned}\n\\]\ngiven the constraints that $0 \\le EY_c(0) \\le 1$, $0 \\le EY_c(1) \\le 1$, and $0 \\le EY_n \\le 1$. Since $EY_n = [EY_1 - P_c EY_c(1)]\\hl{/(1 - P_c)}$, we may reframe the problem as", "corrected": "Insert at the start of the proof of Proposition 1:\n\n\\hl{If $P_c = 1$, a unit is a complier with probability one, so $EY_c(0) = EY_0$ and $EY_c(1) = EY_1$; the displayed intervals reduce to these singletons. Henceforth suppose $0 < P_c < 1$.}\n\nThe proof can also be written with no mean over the noncompliers, and then no separate case is needed. Assume (i)--(iv) of Section 2 and $Y \\in [0,1]$, and assume that both assignment arms have positive probability. Let $C$ denote the complier event $D(1) > D(0)$, and set $p = P_c = \\Pr(C) > 0$, $e_z = EY_z = \\mathrm{E}[Y \\mid Z = z]$ and $c_d = \\mathrm{E}[Y(d) \\mid C]$. Define the noncomplier contribution\n\\[\nm = \\mathrm{E}[Y \\cdot D \\mid Z = 0] + \\mathrm{E}[Y \\cdot (1 - D) \\mid Z = 1],\n\\]\nwhere $Y$ and $D$ are the observed outcome and treatment and the dot denotes multiplication. Then\n\\[\n0 \\le m \\le 1 - p, \\qquad e_z = p\\,c_z + m \\quad \\text{for } z \\in \\{0, 1\\}.\n\\]\nFor $p < 1$, $m = (1 - p) EY_n$. For $p = 1$, $m = 0$ and $c_z = e_z$, with no mean over an empty group and no division by $1 - p$.\n\nCorollary 1 is correct as printed when the four cell means exist. To state it in a form that allows empty treatment-by-assignment cells, replace its two weighted cell means by the two product moments in $m$, and use $c_0 = (e_0 - m)/p$ and $c_1 = (e_1 - m)/p$. Each product moment equals the printed probability times the cell mean whenever that cell mean exists, and it is zero when the cell is empty.", "evidence": "I give an exact counterexample to the step of the proof, together with a derivation that covers $P_c = 1$. For the counterexample, let $Z$ be $0$ or $1$ with probability $1/2$ each, and suppose that $D(0) = 0$, $D(1) = 1$, $Y(0) = 0$, $Y(1) = 1$ and $Y(d, z) = Y(d)$ for every unit. Assumptions (i)--(iv) hold and $Y \\in [0,1]$. I have $P_c = 1$, $EY_0 = 0$, $EY_1 = 1$, $EY_c(0) = 0$ and $EY_c(1) = 1$. However, $\\Pr[D(1) = D(0)] = 0$, and $[EY_1 - P_c EY_c(1)]/(1 - P_c) = 0/0$.\n\nFor the derivation, write $p = P_c$, $e_z = EY_z$ and $c_d = EY_c(d)$, and let $A$ and $N$ denote the events $D(0) = D(1) = 1$ (always-takers) and $D(0) = D(1) = 0$ (never-takers). By monotonicity, $\\Pr(A) + \\Pr(N) = 1 - p$. Define $m = \\mathrm{E}[\\mathbf{1}_A Y(1)] + \\mathrm{E}[\\mathbf{1}_N Y(0)]$. Since $Y \\in [0,1]$, I have $0 \\le m \\le 1 - p$. In the arm $Z = 0$ the observed outcome is $Y(0)$ for compliers and never-takers and $Y(1)$ for always-takers, and in the arm $Z = 1$ it is $Y(1)$ for compliers and always-takers and $Y(0)$ for never-takers. By random assignment and exclusion, it follows that\n\\[\ne_0 = p\\,c_0 + m, \\qquad e_1 = p\\,c_1 + m .\n\\]\nThese equations use no mean over the noncompliers. If $p < 1$, then $m = (1 - p) EY_n$ and they are (1) and (2). If $p = 1$, then $m = 0$, and I have $c_0 = e_0$ and $c_1 = e_1$. Setting $P_c = 1$ in the printed intervals, I obtain\n\\[\n\\begin{aligned}\n\\max\\{0,\\, e_0 - e_1,\\, e_0\\} &= e_0 = \\min\\{1,\\, 1 - e_1 + e_0,\\, e_0\\},\\\\\n\\max\\{0,\\, e_1 - e_0,\\, e_1\\} &= e_1 = \\min\\{1,\\, 1 + e_1 - e_0,\\, e_1\\},\n\\end{aligned}\n\\]\nbecause $0 \\le e_0 \\le 1$ and $0 \\le e_1 \\le 1$. The two intervals are the points $e_0$ and $e_1$. In the arm $Z = 0$ the product $Y \\cdot D$ equals $Y(1)$ on $A$ and $0$ elsewhere, and in the arm $Z = 1$ the product $Y \\cdot (1 - D)$ equals $Y(0)$ on $N$ and $0$ elsewhere. This implies the expression for $m$ in terms of observed variables that is used below. A check in exact arithmetic agrees. The printed intervals are single points in 81 models with full compliance, and the formulas with $m$ hold in 810 finite models, some with empty cells.", "problem": "Assumptions (iii) and (iv) imply that $0 < P_c \\le 1$, and no assumption of Section 2 excludes $P_c = 1$. When $P_c = 1$ the event $D(1) = D(0)$ has probability zero. The mean $EY_n$ is then not defined, and the quotient $[EY_1 - P_c EY_c(1)]/(1 - P_c)$ is $0/0$. The proof treats no separate case and does not cover full compliance. The omission is in the proof and not in the result. At $P_c = 1$ the two printed intervals reduce to the points $EY_0$ and $EY_1$, which are the complier average potential outcomes in that case. The ratio in the proof of Corollary 1 has the same denominator, $\\Pr[D = 1 \\mid Z = 0] + \\Pr[D = 0 \\mid Z = 1] = 1 - P_c$. Corollary 1 assumes that the four cell means $\\mathrm{E}[Y \\mid D = d, Z = z]$ are known. This assumption requires that $P_c < 1$, and the corollary is correct as printed.", "what_survives": "The intervals of Proposition 1 are correct as printed, for $P_c = 1$ as well as for $P_c < 1$, and they remain sharp. The LATE identity divides by $P_c > 0$ and is unaffected, and Corollary 1 holds under its premise that the four cell means are known. The application has two-sided noncompliance, with $P_c = 0.21$, and no printed number changes. The repair adds a case to the proof and no assumption."}, "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-E002", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2012.11.012", "title": "Sharp Bounds for Complier Average Potential Outcomes in Experiments with Noncompliance and Incomplete Reporting", "url": "https://pmaronow.github.io/llm-audit/works/P028/"}, "work_id": "P028"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the four entries of the reference list. A red outline surrounds the numeral 4 in the third entry, which ends: Review of Economic Studies 4, 555 to 574. The first entry ends: Journal of the American Statistical Association 91, 444 to 455.", "caption": "References, p. 679 (PDF p. 3). The outline marks the numeral 4 in the Imbens and Rubin (1997) entry, in the position where the first entry gives the volume 91.", "url": "https://pmaronow.github.io/llm-audit/works/P028/exhibits/P028-E003.png"}], "formal_release": null, "formal_statements": [], "headline": "The Imbens and Rubin (1997) entry prints the issue number, 4, where the volume belongs; the article is in Review of Economic Studies, volume 64, issue 4, pp. 555–574.", "id": "P028-E003", "location": "References, third entry (Imbens and Rubin, 1997), p. 679 (PDF p. 3)", "pdf_page": 3, "printed_page": "679", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Imbens, G.W., Rubin, D.B., 1997. Estimating outcome distributions for compliers in instrumental variables models. Review of Economic Studies \\hl{4}, 555--574.", "corrected": "Imbens, G.W., Rubin, D.B., 1997. Estimating outcome distributions for compliers in instrumental variables models. Review of Economic Studies \\hl{64}, 555--574.", "evidence": "I checked the journal's table of contents. Oxford Academic lists the article under ``The Review of Economic Studies, Volume 64, Issue 4, October 1997, Pages 555--574''.", "problem": "The numeral 4 stands where the other journal entry of the list has its volume (``Journal of the American Statistical Association 91, 444--455''). The article is in volume 64, and 4 is the number of the issue. The entry omits the volume.", "what_survives": "The cited work is identified by its authors, title, year and pages, and no result of the paper depends on the entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-E003", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2012.11.012", "title": "Sharp Bounds for Complier Average Potential Outcomes in Experiments with Noncompliance and Incomplete Reporting", "url": "https://pmaronow.github.io/llm-audit/works/P028/"}, "work_id": "P028"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of Section 2, from the sentence that defines the treatment assigned to unit i through the four assumptions (i) to (iv) to the first two lines of the next paragraph. A red outline surrounds the words: given (i), labels are exchangeable. The sentence reads: First, given (i), labels are exchangeable and so we drop notational dependence on i.", "caption": "Section 2, first paragraph, p. 678 (PDF p. 2). The outline marks the words “given (i), labels are exchangeable”, the reason given for dropping the unit index $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P028/exhibits/P028-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "The unit index is dropped as a change to generic-unit population notation, in which $D$ and $Y$ are the realized treatment and outcome; random assignment does not make the unit labels exchangeable, and no further assumption is needed.", "id": "P028-N002", "location": "Section 2, first paragraph, p. 678 (PDF p. 2)", "pdf_page": 2, "printed_page": "678", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "We assume (i) random assignment: $[Y_i(d, z), D_i(z) : d, z \\in \\{0, 1\\}] \\perp\\!\\!\\!\\perp Z_i$; (ii) exclusion restriction: $Y_i(d, z)$ does not depend on $z$, so we may write $Y_i(d, z) = Y_i(d)$; (iii) non-zero first-stage effect: $E[D_i(1) - D_i(0)] \\neq 0$; (iv) monotonicity: $\\forall i$, $D_i(1) \\geq D_i(0)$. We make two notational simplifications. First, \\hl{given (i), labels are exchangeable and so} we drop notational dependence on $i$. Second, we define $P_c \\equiv \\Pr[D(1) > D(0)]$, $EY_1 \\equiv \\mathrm{E}[Y \\mid Z = 1]$, and $EY_0 \\equiv \\mathrm{E}[Y \\mid Z = 0]$.", "corrected": "We make two notational simplifications. First, we drop notational dependence on $i$ \\hl{and use generic-unit population notation, writing $D = D(Z)$ and $Y = Y(D)$ for the realized treatment and outcome}. Second, we define $P_c \\equiv \\Pr[D(1) > D(0)]$, $EY_1 \\equiv \\mathrm{E}[Y \\mid Z = 1]$, and $EY_0 \\equiv \\mathrm{E}[Y \\mid Z = 0]$.", "intended_meaning": "The sentence changes notation. From this point the paper writes $Y(d)$, $D(z)$ and $Z$ for a generic unit of the population, and $D = D(Z)$ and $Y = Y(D)$ denote the realized treatment and the realized outcome. The paper does not display these two identities, but their usual meaning is clear from the definitions before the sentence, from the discussion of population quantities after it, and from the application. On this reading the LATE identity, Proposition 1 and Corollary 1 hold as stated.\n\nRead literally, the reason given for dropping the index does not hold. Assumption (i) says that the assignment of each unit is independent of the potential outcomes and potential treatments of that unit, and this does not imply that the unit labels are exchangeable. For example, consider two units such that $D_i(0) = 0$ and $D_i(1) = 1$, with $Y_1(0) = Y_1(1) = 0$ and $Y_2(0) = Y_2(1) = 1$, each assigned by an independent fair coin. Since the potential outcomes of each unit are constant, (i) holds. However, the outcome of the first unit is always $0$, and the outcome of the second is always $1$.\n\nThe same example shows that the index can be dropped without assuming that the units are identically distributed. Let the generic unit be one of the two units, chosen with probability $1/2$ each and independently of the assignments. By a direct computation, I find that $\\mathrm{E}[Y \\mid Z = 0] = \\mathrm{E}[Y \\mid Z = 1] = 1/2$. The population quantities of Section 2 are defined in this example, although the two units have different outcome distributions."}, "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-N002", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2012.11.012", "title": "Sharp Bounds for Complier Average Potential Outcomes in Experiments with Noncompliance and Incomplete Reporting", "url": "https://pmaronow.github.io/llm-audit/works/P028/"}, "work_id": "P028"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the end of the first paragraph of Section 4, which gives the bounds [0, 0.71] and [0.29, 1], and of the whole second paragraph. A red outline surrounds the words: we are able to fill in the necessary missing information. The paragraph continues with the four quantities 0.163, 0.634, 0.506 and 0.442 and with the sentence: Application of Corollary 1 yields E[Y(0) | D(1) > D(0)] = 0.43 and E[Y(1) | D(1) > D(0)] = 0.72.", "caption": "Section 4, second paragraph, p. 679 (PDF p. 3), with the last three lines of the first paragraph. The outline marks the words “we are able to fill in the necessary missing information”.", "url": "https://pmaronow.github.io/llm-audit/works/P028/exhibits/P028-N005.png"}, {"alt": "Chart with a horizontal axis from 0 to 1, labelled complier average potential outcome, the proportion whose view of at least one candidate changed. Title: 71 missing outcomes leave the complier means in a range, not at a point. There are two rows. In the row for the control complier mean, a grey line runs from 0 to 0.71, the bounds printed in Section 4, and below it a blue bar runs from 0.14 to 0.75, the range over all completions of the 71 missing outcomes. An orange diamond on the bar marks the printed estimate 0.43, and black ticks just to its right mark the estimates under three conventional treatments, 0.436 to 0.439. In the row for the treated complier mean, the grey line runs from 0.29 to 1 and the blue bar from 0.66 to 0.75, with the diamond at the printed estimate 0.72 and the ticks at 0.712 to 0.721.", "caption": "The two complier means in the example of Section 4 when the 71 missing outcomes are unrestricted. For each mean the blue bar is the range over all completions of the missing binary outcomes, with viewing as coded in the replication data; the diamond is the point estimate printed in the paper; the black ticks are the estimates under three conventional treatments of the missing outcomes; and the grey line is the interval printed in the first paragraph of Section 4, shown for comparison. The ranges are not confidence intervals.", "url": "https://pmaronow.github.io/llm-audit/works/P028/exhibits/P028-N005-missing-outcomes.png"}], "formal_release": null, "formal_statements": [{"id": "P028-T1", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T1"}, {"id": "P028-T2", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T2"}, {"id": "P028-T3", "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-T3"}], "headline": "The point estimates $0.43$ and $0.72$ require an assumption about the 71 respondents whose outcome is missing, and the paper states none; under the assumptions of Section 2 alone, the replication data do not point identify the complier average potential outcomes.", "id": "P028-N005", "location": "Section 4, second paragraph, p. 679 (PDF p. 3)", "pdf_page": 3, "printed_page": "679", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P028-T1", "P028-T2", "P028-T3"], "text": {"as_printed": "Point identification is possible when additional information about the joint distribution of $Y, D, Z$ is known. Since the replication data set is publicly available for the study$^{2}$, \\hl{we are able to fill in the necessary missing information}: $\\Pr[D = 1 \\mid Z = 0] = 0.163$, $\\Pr[D = 0 \\mid Z = 1] = 0.634$, $\\mathrm{E}[Y \\mid D = 1, Z = 0] = 0.506$ and $\\mathrm{E}[Y \\mid D = 0, Z = 1] = 0.442$. Application of Corollary 1 yields $\\mathrm{E}[Y(0) \\mid D(1) > D(0)] = 0.43$ and $\\mathrm{E}[Y(1) \\mid D(1) > D(0)] = 0.72$.", "corrected": "Point identification is possible when additional information about the joint distribution of $Y, D, Z$ is known. Since the replication data set is publicly available for the study$^{2}$, we are able to \\hl{estimate the necessary missing quantities}: $\\Pr[D = 1 \\mid Z = 0] = 0.163$, $\\Pr[D = 0 \\mid Z = 1] = 0.634$, $\\mathrm{E}[Y \\mid D = 1, Z = 0] = 0.506$ and $\\mathrm{E}[Y \\mid D = 0, Z = 1] = 0.442$. \\hl{The outcome is missing for 71 of the 1000 respondents, 37 of the 495 assigned to control and 34 of the 505 assigned to encouragement. Assume in addition that, within each assignment group, whether the outcome is recorded is independent of $(Y, D)$, so that means among the respondents whose outcome is recorded estimate the corresponding population means. Under this assumption, the reported values can be interpreted as point estimates for the study population:} $\\mathrm{E}[Y(0) \\mid D(1) > D(0)] = 0.43$ and $\\mathrm{E}[Y(1) \\mid D(1) > D(0)] = 0.72$. \\hl{Without an assumption about the missing outcomes, these data do not point identify the complier average potential outcomes.}\n\nThe added assumption is sufficient for a calculation that takes the outcome means from the respondents whose outcome is recorded and the take-up rates from all respondents. It does not follow from random assignment and does not specify the exact calculation of the printed values; I have not established it for this study. A complete statement would also give the coding of the outcome and of viewing and the respondents who enter each quantity; the paper's own analysis script and original denominators are unavailable. If no assumption about nonresponse is made, the two point estimates should be replaced by the ranges over the missing outcomes: with viewing as coded in the replication data, $0.14$ to $0.75$ for the control complier mean and $0.66$ to $0.75$ for the treated complier mean.", "evidence": "The evidence is an exact counterexample. There exist two models that satisfy every assumption of Section 2, in which the observed data have the same distribution and the complier means differ. The observed data are the assignment $Z$, the viewing variable $D$ as coded in the replication data, whether the outcome is recorded, and the outcome when it is recorded. The table lists the counts.\n\\begin{audittable}{llrrrrr}{Respondents in the replication data, by assignment $Z$ and viewing $D$. The outcome $Y$ is coded as in the source study.}\n$Z$ & $D$ & Respondents & Outcome recorded & $Y = 1$ & $Y = 0$ & Outcome missing \\\\\n0 & 0 & 415 & 379 & 167 & 212 & 36 \\\\\n0 & 1 & 80 & 79 & 40 & 39 & 1 \\\\\n1 & 0 & 320 & 294 & 130 & 164 & 26 \\\\\n1 & 1 & 185 & 177 & 110 & 67 & 8 \\\\\n\\end{audittable}\nLet $x_{zd}$ denote the number of missing outcomes in the cell $(Z, D) = (z, d)$ that equal $1$, so that $0 \\le x_{00} \\le 36$, $0 \\le x_{01} \\le 1$, $0 \\le x_{10} \\le 26$ and $0 \\le x_{11} \\le 8$. For the counterexample I treat the proportions within each arm, completed in this way, as a population distribution. Under monotonicity the cell $(Z, D) = (0, 1)$ consists of always-takers and the cell $(1, 0)$ of never-takers. Their probabilities are $p_a = 80/495$ and $p_n = 320/505$, and compliers have probability $p_c = 1 - p_a - p_n = 2047/9999$. Let $\\mu_d = \\mathrm{E}[Y(d) \\mid D(1) > D(0)]$. By random assignment and exclusion, I have\n\\[\n\\frac{167 + x_{00}}{495} = \\frac{130 + x_{10}}{505} + p_c\\,\\mu_0, \\qquad \\frac{110 + x_{11}}{505} = \\frac{40 + x_{01}}{495} + p_c\\,\\mu_1 .\n\\]\nThese are the equations of Corollary 1, written with joint probabilities. Solving them, I obtain\n\\[\n\\mu_0 = \\frac{(167 + x_{00})/495 - (130 + x_{10})/505}{2047/9999}, \\qquad \\mu_1 = \\frac{(110 + x_{11})/505 - (40 + x_{01})/495}{2047/9999}.\n\\]\nChoosing $(x_{00}, x_{01}, x_{10}, x_{11}) = (0, 1, 26, 0)$, I obtain\n\\[\n(\\mu_0, \\mu_1) = (1423/10235,\\ 6749/10235) \\approx (0.139,\\ 0.659),\n\\]\nand the choice $(36, 0, 0, 8)$ gives\n\\[\n(\\mu_0, \\mu_1) = (7633/10235,\\ 7642/10235) \\approx (0.746,\\ 0.747).\n\\]\nSince $\\mu_0$ increases in $x_{00}$ and decreases in $x_{10}$, and $\\mu_1$ increases in $x_{11}$ and decreases in $x_{01}$, these are the extreme values over all completions of the missing outcomes.\n\nFor each of the two choices there exists an explicit population model. It has always-takers, never-takers and compliers with probabilities $p_a$, $p_n$ and $p_c$, binary potential outcomes, and, for each assignment, an indicator that the outcome is recorded whose probability depends on the type of the unit and on its outcome. In each model the assignment is independent of all potential variables, exclusion and monotonicity hold, the first stage is $2047/9999$, and the observed data have the distribution in the table. A check in exact arithmetic verifies these properties for the two choices and for the other 14 choices such that each $x_{zd}$ is $0$ or its maximum.\n\nThe ranges are extrema over the missing binary outcomes, with viewing held as coded. They are not confidence intervals, and they do not show that $0.43$ or $0.72$ is biased. Under three conventional treatments of the missing outcomes, the estimates are close to the printed ones. With outcome means among respondents whose outcome is recorded and take-up rates among all respondents, I obtain $(0.439, 0.721)$ from Corollary 1. With every quantity computed among the 929 respondents whose outcome is recorded, I find $(0.436, 0.719)$. With the cell means among recorded outcomes weighted by the cell probabilities among all respondents, the estimates are $(0.436, 0.712)$.", "problem": "The source study codes the outcome as $1$ if the respondent's opinion of either mayoral candidate changed, as $0$ if both opinions were unchanged, and as missing otherwise. With this coding the outcome is missing for 71 of the 1000 respondents in the replication data: 37 of the 495 assigned to control and 34 of the 505 assigned to encouragement, at different rates in the four cells defined by assignment and viewing. The paper does not mention these respondents. It does not say which respondents enter each quantity, and it states no assumption about nonresponse. Section 2 assumes random assignment, exclusion, a non-zero first stage, monotonicity and bounded outcomes, and none of these assumptions concerns which outcomes are recorded. Corollary 1 identifies the two complier average potential outcomes, which I call the complier means, from the joint distribution of $(Y, D, Z)$ in the study population. The replication data determine the distribution of $Y$ only among respondents whose outcome is recorded, and a mean among those respondents equals the corresponding population mean only under a further assumption. The words ``we are able to fill in the necessary missing information'' claim more than the data supply. The values $0.43$ and $0.72$ are presented as point estimates under the assumptions of Section 2, and they also depend on an assumption about nonresponse that the paper does not state and that random assignment does not imply.\n\nThe correction does not say that either number is wrong. Under conventional treatments of the missing outcomes the estimates are about $0.44$ and $0.72$, and no bias is shown. The repair has a cost. To keep $0.43$ and $0.72$ as point estimates for the study population, one must add an assumption about nonresponse that the application may not satisfy. Without such an assumption, the point estimates must be replaced by a range over the missing outcomes, which is a weaker conclusion.", "what_survives": "Corollary 1 is unaffected, since it identifies the complier means when the population moments in its statement are known. Proposition 1 and the reporting recommendation of Section 5 do not depend on the example, and no printed number is shown to be wrong. What changes is the status of $0.43$ and $0.72$, and of the statement that the replication data tighten inference to point estimates. Both hold only under an added assumption about nonresponse. The CACE of 29 percentage points and the bounds of the first paragraph use outcome means among respondents whose outcome is recorded. They need the same care if they are read as statements about the whole study population."}, "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-N005", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2012.11.012", "title": "Sharp Bounds for Complier Average Potential Outcomes in Experiments with Noncompliance and Incomplete Reporting", "url": "https://pmaronow.github.io/llm-audit/works/P028/"}, "work_id": "P028"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Section 4 from its second line to the end of its second paragraph. Red outlines surround five items. In the first paragraph: P sub c = 0.21, in the sentence The authors report estimates of EY sub 0 = 0.45, EY sub 1 = 0.51 and P sub c = 0.21. In the second paragraph: the values 0.163 and 0.634 of the two take-up rates, and the values 0.43 and 0.72 in the sentence Application of Corollary 1 yields E[Y(0) | D(1) > D(0)] = 0.43 and E[Y(1) | D(1) > D(0)] = 0.72.", "caption": "Section 4, p. 679 (PDF p. 3). The outlines mark the first stage $P_c = 0.21$ in the first paragraph, and the take-up rates $0.163$ and $0.634$ and the point estimates $0.43$ and $0.72$ in the second.", "url": "https://pmaronow.github.io/llm-audit/works/P028/exhibits/P028-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The seven figures printed in Section 4 are not one consistently rounded set, and the paper does not state the calculation behind $0.43$ and $0.72$; both values are reproduced when the reported outcome means are combined with unrounded take-up rates and cell means from the replication data.", "id": "P028-N007", "location": "Section 4, first and second paragraphs, p. 679 (PDF p. 3)", "pdf_page": 3, "printed_page": "679", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The authors report estimates of $EY_0 = 0.45$, $EY_1 = 0.51$ and $\\hl{P_c = 0.21}$. While the CACE can be estimated as 29 percentage points, the authors do not provide sufficient information for point estimation of complier average potential outcomes. [\\ldots] Since the replication data set is publicly available for the study$^{2}$, we are able to fill in the necessary missing information: $\\Pr[D = 1 \\mid Z = 0] = \\hl{0.163}$, $\\Pr[D = 0 \\mid Z = 1] = \\hl{0.634}$, $\\mathrm{E}[Y \\mid D = 1, Z = 0] = 0.506$ and $\\mathrm{E}[Y \\mid D = 0, Z = 1] = 0.442$. Application of Corollary 1 yields $\\mathrm{E}[Y(0) \\mid D(1) > D(0)] = \\hl{0.43}$ and $\\mathrm{E}[Y(1) \\mid D(1) > D(0)] = \\hl{0.72}$. Our inferences are considerably tightened by knowledge of additional features of the distribution that were not reported.", "corrected": "Application of Corollary 1 yields $\\mathrm{E}[Y(0) \\mid D(1) > D(0)] = 0.43$ and $\\mathrm{E}[Y(1) \\mid D(1) > D(0)] = 0.72$. \\hl{These point estimates can be reproduced by retaining the reported outcome means $0.45$ and $0.51$ and using the unrounded take-up rates and cell outcome means from the replication data. With uncertain viewing responses coded as missing, this calculation uses the first stage $P_c = 1 - 80/492 - 320/505 \\approx 0.204$, rather than the $0.21$ reported by the source study.}\n\nThe added sentences describe one calculation that reproduces the printed values. If the original calculation was different, they should describe that calculation.", "intended_meaning": "The first paragraph of Section 4 uses the three figures reported by the source study, and the second paragraph reports point estimates that were computed after consulting the replication data. All of these figures are rounded estimates, and on this reading the two paragraphs need not use the same first stage. The paper does not say which first stage and which precision enter the point estimates, and the printed figures alone do not reproduce them.\n\nUnder monotonicity, $P_c = 1 - \\Pr[D = 1 \\mid Z = 0] - \\Pr[D = 0 \\mid Z = 1]$. Rates that round to $0.163$ and $0.634$ imply that $0.202 < P_c \\le 0.204$, while a first stage that rounds to $0.21$ satisfies $0.205 \\le P_c < 0.215$. Hence the seven printed figures are not one consistently rounded set. Substituting them into Corollary 1, I obtain\n\\[\n\\frac{0.45 - 0.163 \\cdot 0.506 - 0.634 \\cdot 0.442}{0.21} = \\frac{0.087294}{0.21} \\approx 0.4157, \\qquad 0.4157 + \\frac{0.51 - 0.45}{0.21} \\approx 0.7014,\n\\]\nwhich round to $0.42$ and $0.70$. With $P_c = 0.203$, the value implied by the two printed rates, the same substitution gives $0.4300$ and $0.7256$, which round to $0.43$ and $0.73$.\n\nOne calculation returns both printed values. I keep the reported outcome means $0.45$ and $0.51$, and I take from the replication data the unrounded take-up rates $80/492$ and $320/505$, the unrounded cell means $40/79$ and $130/294$, and the first stage $P_c = 1 - 80/492 - 320/505 = 2531/12423 \\approx 0.2037$. The rate $80/492$ leaves out three control respondents whose answer to the viewing question was ``not sure'' or a refusal. Using Corollary 1, I find\n\\[\n\\begin{aligned}\n\\frac{0.45 - \\frac{80}{492} \\cdot \\frac{40}{79} - \\frac{320}{505} \\cdot \\frac{130}{294}}{2531/12423} &\\approx 0.4294,\\\\\n\\frac{0.51 - \\frac{80}{492} \\cdot \\frac{40}{79} - \\frac{320}{505} \\cdot \\frac{130}{294}}{2531/12423} &\\approx 0.7239,\n\\end{aligned}\n\\]\nwhich round to $0.43$ and $0.72$. No analysis script for the application is available. This calculation is a reconstruction and may differ from the one originally made.\n\nWith every input computed from the replication data at full precision, the estimates round to $0.44$ and $0.72$ under each of three conventions for the sample and for the coding of viewing: $(0.4393, 0.7207)$, $(0.4390, 0.7217)$ and $(0.4359, 0.7192)$. No printed estimate is shown to be wrong. The printed estimates lie inside the printed bounds $[0, 0.71]$ and $[0.29, 1]$, and this note does not affect the statement that the additional information tightens the inferences. The assumption on which the point estimates rest is the subject of \\fid{P028-N005}."}, "url": "https://pmaronow.github.io/llm-audit/works/P028/#P028-N007", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2012.11.012", "title": "Sharp Bounds for Complier Average Potential Outcomes in Experiments with Noncompliance and Incomplete Reporting", "url": "https://pmaronow.github.io/llm-audit/works/P028/"}, "work_id": "P028"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five displayed lines. The first reads X1 sub i, X2 sub i, X3 sub i distributed as U(-1, 1). The second reads a, b, c, d, e, f, g, h distributed as U(-2, 2) and has a red outline. The third reads AC sub i = I[2 + a + b X1 sub i + c X2 sub i + d X3 sub i + N(0, 3)], with a red outline around N(0, 3). The fourth reads A sub i = I[-2 + e + f X1 sub i + g X2 sub i + h X3 sub i + N(0, 3)] AC sub i, with a red outline around N(0, 3). The fifth reads C sub i = AC sub i - A sub i.", "caption": "Supplement, p. A1 (supplement PDF p. 1), first lines of the baseline design. The outlines mark the two terms $\\mathbf{N}(0,3)$, whose scale is not stated, and the line for the coefficients, which does not say when they are drawn.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-A001.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T19", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T19"}], "headline": "In the simulation design, $\\mathbf{N}(0,3)$ is a normal draw with standard deviation $3$, and the coefficients $a,\\dots,h$ are drawn once for each simulated dataset; the supplement states neither convention.", "id": "P029-A001", "location": "Supplement, p. A1 (supplement PDF p. 1), “Baseline Data Generating Process”: the line for $a,\\dots,h$ and the two terms $\\mathbf{N}(0,3)$", "pdf_page": 1, "printed_page": "A1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T19"], "text": {"as_printed": "\\[\n\\begin{aligned}\n& X1_i, X2_i, X3_i \\sim \\mathbf{U}(-1,1)\\\\\n& \\hl{a, b, c, d, e, f, g, h \\sim \\mathbf{U}(-2,2)}\\\\\n& AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\hl{\\mathbf{N}(0,3)}]\\\\\n& A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\hl{\\mathbf{N}(0,3)}]AC_i\n\\end{aligned}\n\\]", "corrected": "\\[\n\\begin{aligned}\n& X1_i, X2_i, X3_i \\sim \\mathbf{U}(-1,1)\\\\\n& a, b, c, d, e, f, g, h \\sim \\mathbf{U}(-2,2)\\\\\n& AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\mathbf{N}(0,3)]\\\\\n& A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\mathbf{N}(0,3)]AC_i\n\\end{aligned}\n\\]\n\\hl{Here $\\mathbf{N}(0,3)$ denotes a normal draw with mean zero and standard deviation $3$. The scalar coefficients $a,\\dots,h$ are drawn once for each simulated dataset and are shared by all of its observations.}\n\nAn alternative for the first sentence is to write $\\mathbf{N}(0,3^2)$ and to state that the second argument is the variance.", "evidence": "The evidence is a check of the page against the replication files. In \\texttt{Simulations Alpha 6-20.R}, the function \\texttt{dgp} generates one simulated dataset. It draws each disturbance with \\texttt{rnorm(N,0,3)}, where the third argument of \\texttt{rnorm} is the standard deviation. Each coefficient is drawn with \\texttt{runif(1,-2,2)}, once for each simulated dataset, and the eight numbers are passed to \\texttt{dgp}.", "problem": "The supplement does not say whether the second argument of $\\mathbf{N}(0,3)$ is a variance or a standard deviation. A common convention in mathematics takes it as the variance. The simulation code draws the disturbances with standard deviation $3$. The choice changes the compliance rates of the design: at zero covariates and $a = 0$, the probability that $AC_i = 1$ is $\\Phi(2/3) = 0.7475$ with standard deviation $3$ and $\\Phi(2/\\sqrt{3}) = 0.8759$ with variance $3$. The supplement also does not say when the coefficients $a,\\dots,h$ are drawn. They carry no index $i$, which shows that all units share them. In the code they are drawn once for each simulated dataset. Someone who implements the design from the supplement alone has to guess both conventions. No reported number is shown to be wrong.", "what_survives": "The simulation design in the code and all reported simulation results are not shown to be affected. The notation of the supplement already indicates that the coefficients are common to all units. The correction adds two sentences to the description of the design."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-A001", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of two lines. The text reads: A1 and A2. We subclassify each of these plots to DGPs with low (mean(P sub C) < 0.33), medium (0.33 is at most mean(P sub C) < 0.50) and high (0.50 is at most mean(P sub C)) compliance. Red outlines surround the three occurrences of mean(P sub C).", "caption": "Supplement, p. A3 (supplement PDF p. 3), the two lines above Figure A1. The outlines mark the three occurrences of the label $\\operatorname{mean}(P_C)$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-A002.png"}], "formal_release": null, "formal_statements": [], "headline": "The compliance groups of Figures A1 and A2 are formed from the realized fraction of compliers in each simulated dataset, $\\bar C = N^{-1}\\sum_i C_i$, which the supplement labels $\\operatorname{mean}(P_C)$.", "id": "P029-A002", "location": "Supplement, p. A3 (supplement PDF p. 3), sentence above Figure A1", "pdf_page": 3, "printed_page": "A3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "We subclassify each of these plots to DGPs with low ($\\hl{\\operatorname{mean}(P_C)} < 0.33$), medium ($0.33 \\le \\hl{\\operatorname{mean}(P_C)} < 0.50$) and high ($0.50 \\le \\hl{\\operatorname{mean}(P_C)}$) compliance.", "corrected": "We subclassify each of these plots to DGPs with low ($\\hl{\\bar C} < 0.33$), medium ($0.33 \\le \\hl{\\bar C} < 0.50$) and high ($0.50 \\le \\hl{\\bar C}$) compliance\\hl{, where $\\bar C = N^{-1}\\sum_i C_i$ is the realized fraction of compliers in each simulated dataset}.", "intended_meaning": "The sentence sorts the simulated datasets by their overall compliance, with the cut-offs $0.33$ and $0.50$, and the code does the same. In the article, $P_{Ci}$ is the probability that unit $i$ is a complier given its covariates (p.~496), so the label suggests an average of probabilities. The code stores, for each dataset, the mean of the indicators $C_i = AC_i - A_i$ that unit $i$ is a complier, and the display script sorts on this mean. The realized fraction and the mean probability have the same conditional expectation, but they differ in a finite dataset. In one simulated dataset of $100$ units, with all covariates and coefficients equal to zero, the realized fraction is $0.53$ and the probability is $\\Phi(2/3)(1-\\Phi(-2/3)) = 0.5588$. The thresholds and the meaning of low, medium and high compliance are as printed, and no figure changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-A002", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of equation (2) and the eight lines below it. Equation (2) reads D sub i = gamma 0 + gamma 1 Z sub i + delta 1 Q sub 1i + delta 2 Q sub 2i + ... + delta K Q sub Ki + e sub i. The text explains two-stage least squares and ends: Given the model in equations (1) and (2), the SEM IV framework relies on two main assumptions for consistency. First, the correlation between Z sub i and u sub i must approach zero as n grows. Second, the covariance between Z sub i and D sub i must converge to a nonzero quantity as n grows. Red outlines surround the two sentences that begin First and Second.", "caption": "Section 2.1, p. 494 (PDF p. 3). The outlines mark the two conditions for consistency of 2SLS. Equation (2), at the top of the crop, contains the covariates $Q_{1i},\\dots,Q_{Ki}$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E001.png"}, {"alt": "Crop of four lines. The first sentence reads: We are able to consistently estimate beta 1 using ordinary least squares (OLS) regression when D sub i is asymptotically uncorrelated with the error term (one over n times the sum of D sub i u sub i converges in probability to 0). The next sentence says that this requirement is violated when D sub i is endogenous. The crop ends partway through the sentence about finding an instrument, after Y sub i. A red outline surrounds the words asymptotically uncorrelated with the error term and the formula in parentheses.", "caption": "Section 2.1, p. 494 (PDF p. 3), first lines of the page. The outline marks the condition for consistency of OLS.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The consistency conditions of Section 2.1 are stated for $D_i$ and $Z_i$ themselves, although equations (1) and (2) contain covariates; the conditions meant are the usual ones for $D_i$ and $Z_i$ after the covariates are partialled out.", "id": "P029-E001", "location": "Section 2.1, p. 494 (PDF p. 3), first sentence of the page and the two sentences after “two main assumptions for consistency”; equation (1) is on p. 493 (PDF p. 2)", "pdf_page": 3, "printed_page": "494", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "We are able to consistently estimate $\\beta_1$ using ordinary least squares (OLS) regression when $D_i$ is asymptotically uncorrelated with the error term ($\\hl{\\frac{1}{n}\\Sigma D_i u_i \\to_p 0}$). This requirement is violated, however, when $D_i$ is endogenous. [\\ldots] Given the model in equations (1) and (2), the SEM IV framework relies on two main assumptions for consistency. First, the \\hl{correlation between $Z_i$ and $u_i$} must approach zero as $n$ grows. Second, the \\hl{covariance between $Z_i$ and $D_i$} must converge to a nonzero quantity as $n$ grows.", "corrected": "Suppose that \\hl{the observations are independent and identically distributed}, that the second moments are finite and that the second-moment matrix of the intercept and the covariates is nonsingular. Let $d_i$ and $z_i$ be the residuals from the linear projections of $D_i$ and $Z_i$ on an intercept and $Q_{1i},\\dots,Q_{Ki}$. OLS is consistent for $\\beta_1$ when \\hl{$E(d_iu_i) = 0$ and $E(d_i^2) > 0$}. 2SLS with the single instrument $Z_i$ is consistent for $\\beta_1$ when \\hl{$E(z_iu_i) = 0$ and $E(z_iD_i) \\neq 0$}. For other sampling schemes, require stable probability limits for the sample moments and a nonsingular limiting controls matrix, with the same zero-moment and nonzero-denominator conditions using sample projection residuals. Orthogonality of $u_i$ to every regressor, or to every instrument, together with the usual rank condition, is a stronger sufficient condition.", "intended_meaning": "The passage is a short account of OLS and 2SLS with one endogenous treatment. It takes for granted that the covariates $Q_{1i},\\dots,Q_{Ki}$ are valid controls, which serve as their own instruments, and that the instrument is related to the treatment when the covariates are held fixed. With these conventions the account is correct, and the phrase ``two main assumptions'' does not claim to be a complete list.\n\nThe printed conditions do not suffice if they are read literally, as conditions on $D_i$ and $Z_i$ alone. Let $d_i$ and $z_i$ be the residuals from the linear projections of $D_i$ and $Z_i$ on an intercept and the covariates. For independent, identically distributed sampling with finite second moments, and a nonsingular second-moment matrix of the intercept and the covariates, the two estimators have the limits\n\\[\n\\operatorname{plim}\\widehat\\beta_{1}^{\\mathrm{OLS}} = \\beta_1 + \\frac{E(d_iu_i)}{E(d_i^2)}, \\qquad \\operatorname{plim}\\widehat\\beta_{1}^{\\mathrm{IV}} = \\beta_1 + \\frac{E(z_iu_i)}{E(z_iD_i)},\n\\]\nwhen the denominators are not zero. An example shows the difference. Let $Z$ equal $0$ or $1$ with probability $1/2$ each, let $U$ and $V$ equal $-1$ or $1$ with probability $1/2$ each, and let $Z$, $U$ and $V$ be independent. Set $D = Z$, $Q = D + U/2 + V/4$, $u = U$ and $Y = 2 + D + U$. Equations (1) and (2) hold with $\\beta_1 = 1$ and $\\lambda_1 = 0$. I have $E(Du) = 0$, $\\operatorname{Cov}(Z,u) = 0$ and $\\operatorname{Cov}(Z,D) = 1/4$, and the moment matrices of the regressors and of the instruments are nonsingular, with determinant $5/64$. The covariate is correlated with the error, since $E(Qu) = 1/2$. A direct computation gives $E(zu) = -2/9$ and $E(zD) = 5/36$. Substituting these values into the two limits, I find that both OLS and 2SLS converge to $1 + (-2/9)/(5/36) = -3/5$. In a second example the covariate is exogenous and $\\operatorname{Cov}(Z,D) = 1/16$, but $E(zD) = 0$, and $\\beta_1$ is not identified.\n\nNo later proof or calculation in the paper is shown to rely on the literal reading. From Section~2.2 on, the instrument is randomized and independent of the pretreatment covariates, and the argument for the LATE and for ICSW does not use the conditions of Section~2.1."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E001", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of eight lines of text. A red box marks the words 'these assumptions are considerably weaker than (and subsumed by) those of the SEM framework.' The lines above state the fourth assumption, that there are no defiers, and the fifth, that Z is randomly assigned.", "caption": "Section 2.2, p. 495 (PDF p. 4), end of the paragraph that lists the five IV assumptions. The marked sentence compares them with the assumptions of the SEM framework.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T11", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T11"}, {"id": "P029-T12", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T12"}], "headline": "The five potential-outcome assumptions are not weaker than, or subsumed by, the SEM assumptions of Section 2.1: neither set of assumptions implies the other.", "id": "P029-E002", "location": "Section 2.2, p. 495 (PDF p. 4), last sentence of the paragraph that lists the five IV assumptions", "pdf_page": 4, "printed_page": "495", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T11", "P029-T12"], "text": {"as_printed": "Fourth, we assume that the population contains no defiers, i.e., $\\Pr(D_0 > D_1) = 0$. Fifth, we assume that $Z_i$ is randomly assigned and thus independent of all potential outcomes. As no parametric assumptions are necessary, these assumptions are \\hl{considerably weaker than (and subsumed by)} those of the SEM framework.", "corrected": "Fourth, we assume that the population contains no defiers, i.e., $\\Pr(D_0 > D_1) = 0$. Fifth, we assume that $Z_i$ is randomly assigned and thus independent of all potential outcomes. \\hl{These assumptions permit heterogeneous treatment effects but impose random assignment and absence of defiers, neither of which follows from the SEM conditions described above. The two sets of causal assumptions are therefore distinct and non-nested.}", "evidence": "The evidence is three exact counterexamples. Each is a finite population with binary $Z$ and $D$, no covariates and independently sampled units, in which $D=D_Z$ and $Y=Y_D$.\n\\begin{itemize}\n\\item \\emph{The SEM conditions hold and the population contains defiers.} A unit is a complier with probability $3/4$ and a defier with probability $1/4$, and $Z\\sim\\mathrm{Bernoulli}(1/4)$ is independent of the type. I set $u=3/4$ for a defier and $u=-1/4$ for a complier, $Y_0=u$ and $Y_1=u+1$. Then $Y=D+u$ with $\\beta_1=1$, and $D=1/4+Z/2+e$ with $\\mathrm{E}(e\\mid Z)=0$. I have $\\operatorname{Var}(Z)=\\operatorname{Var}(u)=3/16$, $\\operatorname{Cov}(Z,u)=0$ and $\\operatorname{Cov}(Z,D)=3/32$. The instrument is independent of all potential outcomes and the exclusion restriction holds, but $\\Pr(D_0>D_1)=1/4$.\n\\item \\emph{The SEM conditions hold and $Z$ is not independent of the potential outcomes.} Let $U$ be uniform on $\\{-2,-1,1,2\\}$, and set $Z=\\mathbf{1}\\{|U|=2\\}$, $D_0=\\mathbf{1}\\{U>0\\}$, $D_1=1$, $Y_0=U$ and $Y_1=U+1$. Then $Y=D+U$, and $D=1/2+Z/2+e$ with $\\mathrm{E}(e\\mid Z)=0$. I have $\\mathrm{E}(U\\mid Z)=0$, $\\operatorname{Cov}(Z,U)=0$ and $\\operatorname{Cov}(Z,D)=1/8$, and there are no defiers. But $\\Pr(|Y_0|=2\\mid Z=1)=1$ and $\\Pr(|Y_0|=2\\mid Z=0)=0$.\n\\item \\emph{The five assumptions hold and the effect is not constant.} Let $H$ be uniform on $\\{1,2\\}$, and let $Z\\sim\\mathrm{Bernoulli}(1/2)$ be independent of $H$. Every unit is a complier, $Y_0=0$ and $Y_1=H$. The effect is $1$ for half of the units and $2$ for the other half, while a causal reading of equation (1) requires that $Y_1-Y_0=\\beta_1$ for every unit.\n\\end{itemize}\nNote that in the first two populations the treatment is endogenous ($\\operatorname{Cov}(D,u)$ is $3/32$ and $1/4$), and the population IV ratio equals the common effect $1$. These populations are not counterexamples to IV estimation under a constant effect.", "problem": "The sentence asserts that the SEM conditions of Section 2.1 imply the five assumptions just listed. For the model in equations (1) and (2), with a common effect $\\beta_1$, Section 2.1 requires that the correlation between $Z_i$ and $u_i$ tend to zero and that the covariance between $Z_i$ and $D_i$ tend to a nonzero limit. These conditions do not exclude defiers, and a zero correlation does not make $Z_i$ independent of the potential outcomes. In the other direction, the five assumptions let the effect $Y_1-Y_0$ vary across units, and the SEM formulation does not. Neither set of assumptions implies the other. The point that the passage serves is correct: the potential-outcome formulation permits heterogeneous treatment effects.", "what_survives": "Equation (3) is stated under the five assumptions themselves, and Section 3.4 invokes the IV assumptions directly for ICSW. Neither uses the comparison, and no simulation or application result depends on it. Only the comparison changes. The claim that one set of assumptions contains the other is withdrawn."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E002", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text. The first lines display the two equalities of the typical ignorability assumption, for Y1 and for Y0 given D = 1 and D = 0. Two red boxes mark the words 'Assumption 1 is less restrictive than the typical ignorability assumption', which are followed by 'as there is no need to account for baseline selection bias'.", "caption": "Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1. The marked words rank Assumption 1 below the ignorability assumption displayed in the lines above.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T2", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T2"}, {"id": "P029-T20", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T20"}], "headline": "Assumption 1 is not less restrictive than the ignorability assumption displayed beside it: under two-sided noncompliance neither assumption implies the other.", "id": "P029-E003", "location": "Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1", "pdf_page": 8, "printed_page": "499", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P029-T2", "P029-T20"], "text": {"as_printed": "Assumption 1 entails that, for all $\\mathbf{x} \\in \\mathrm{Supp}(\\mathbf{X})$, $\\mathrm{E}(Y_1 - Y_0|\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_1 - Y_0|D_1 > D_0, \\mathbf{X}=\\mathbf{x})$: [\\ldots]\n\nHowever, Assumption 1 may be distinguished from the typical ignorability assumption (Rubin 1978), which requires that both $\\mathrm{E}(Y_1|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_1|D=0,\\mathbf{X}=\\mathbf{x})$ and $\\mathrm{E}(Y_0|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_0|D=0,\\mathbf{X}=\\mathbf{x})$. \\hl{Assumption 1 is less restrictive than the typical ignorability assumption} as there is no need to account for baseline selection bias, only the determinants of treatment effect heterogeneity.", "corrected": "However, Assumption 1 may be distinguished from the typical ignorability assumption (Rubin 1978), which requires that both $\\mathrm{E}(Y_1|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_1|D=0,\\mathbf{X}=\\mathbf{x})$ and $\\mathrm{E}(Y_0|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_0|D=0,\\mathbf{X}=\\mathbf{x})$. \\hl{Assumption 1 and the typical ignorability assumption displayed here impose different restrictions and are not generally ordered. Assumption 1 permits baseline selection bias, provided that the conditional average treatment effect for compliers equals that for the full population.}\n\n\\hl{Under one-sided noncompliance ($D_0=0$), random assignment and positive probabilities of both assignment arms and compliance at each covariate profile, the typical ignorability assumption implies Assumption 1.}", "evidence": "I use two exact counterexamples with a single covariate profile. In both, a unit is a complier ($C$), an always-taker ($A$) or a never-taker ($N$) with probabilities $1/2$, $1/4$ and $1/4$, and $Z\\sim\\mathrm{Bernoulli}(1/2)$ is independent of the type. The five IV assumptions hold, the difference in treatment rates between the arms is $1/2$, and Assumption 3 holds with $\\epsilon=1/4$. Half of the treated units and half of the untreated units are compliers.\n\\begin{audittable}{lrrr}{Potential outcomes $(Y_0,Y_1)$ by type in the two populations.}\nType & Probability & Population 1 & Population 2 \\\\\n\\hline\nComplier & $1/2$ & $(0,1)$ & $(0,1)$ \\\\\nAlways-taker & $1/4$ & $(0,0)$ & $(1,2)$ \\\\\nNever-taker & $1/4$ & $(0,0)$ & $(0,1)$ \\\\\n\\end{audittable}\n\\begin{audittable}{lrr}{The two assumptions in the two populations.}\nQuantity & Population 1 & Population 2 \\\\\n\\hline\n$\\mathrm{E}(Y_0|D=1)$ and $\\mathrm{E}(Y_0|D=0)$ & $0$ and $0$ & $1/2$ and $0$ \\\\\n$\\mathrm{E}(Y_1|D=1)$ and $\\mathrm{E}(Y_1|D=0)$ & $1/2$ and $1/2$ & $3/2$ and $1$ \\\\\nIgnorability, as displayed & holds & fails \\\\\n$\\mathrm{E}(Y_1-Y_0)$ and $\\mathrm{E}(Y_1-Y_0|D_1>D_0)$ & $1/2$ and $1$ & $1$ and $1$ \\\\\nAssumption 1 & fails & holds \\\\\nLimit of ICSW minus the ATE & $1/2$ & $0$ \\\\\n\\end{audittable}\nNote that in population 1 the pair $(Y_0,Y_1)$ is independent of $D$, which is stronger than the displayed equalities. In general, with $\\mu_{td}$ the mean of $Y_d$ for type $t$ and $\\tau_t=\\mu_{t1}-\\mu_{t0}$, ignorability holds in these populations if and only if $\\mu_{Ad}=\\mu_{Nd}$ for $d=0,1$, and Assumption 1 holds if and only if $\\tau_C=(\\tau_A+\\tau_N)/2$.\n\nThe implication holds in one case. Suppose that noncompliance is one-sided, $D_0=0$ for every unit, with random assignment and positive probabilities of both assignment arms and compliance at each covariate profile $\\mathbf{x}$. If ignorability holds, let $m_d$ be the common value of $\\mathrm{E}(Y_d|D,\\mathbf{X}=\\mathbf{x})$, so that $\\mathrm{E}(Y_d|\\mathbf{X}=\\mathbf{x})=m_d$. The treated units are the compliers assigned to treatment, and I have $m_d=\\mathrm{E}(Y_d|D_1>D_0,Z=1,\\mathbf{X}=\\mathbf{x})=\\mathrm{E}(Y_d|D_1>D_0,\\mathbf{X}=\\mathbf{x})$. Subtracting the identity for $d=0$ from the identity for $d=1$, I obtain Assumption 1.", "problem": "The sentence ranks two conditions at a fixed covariate profile. The ignorability assumption equates the mean of each potential outcome between the units that received treatment and the units that did not. Assumption 1 equates the mean effect among compliers with the mean effect among all units. To say that Assumption 1 is less restrictive is to say that ignorability implies it. The reason given, that Assumption 1 need not account for baseline selection, shows only that Assumption 1 can hold when ignorability fails. Under two-sided noncompliance the implication fails as well. The treated group mixes compliers with always-takers, and the untreated group mixes compliers with never-takers. In the two populations below, ignorability compares always-takers with never-takers and places no restriction on complier means. Assumption 1 compares compliers with everyone else. The method covers two-sided noncompliance (p.~496), and the Albertson and Lawrence application has it. Someone choosing between ICSW and a method that assumes selection on observables is told that the ICSW assumption is the weaker of the two. It is a different assumption.", "what_survives": "Assumption 1 and the consistency argument, which invokes Assumption 1 between equations (10) and (11), are unaffected. The observation that Assumption 1 permits baseline selection and the example of the medical trial also stand. This correction changes no estimate. The ranking is withdrawn. In an application with two-sided noncompliance, the reading of the ICSW estimate as an ATE rests on Assumption 1 itself, and ignorability of the treatment received does not supply it. Whether Assumption 1 holds in either application is not established here."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E003", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of text. Red boxes mark the sentence 'More generally, even if treatment effects are heterogeneous, but still statistically independent of compliance, Assumption 1 would necessarily be satisfied, and both 2SLS and ICSW would be consistent for the ATE.'", "caption": "Section 3.4, p. 499 (PDF p. 8). The marked sentence states a sufficient condition for Assumption 1 and for the consistency of both estimators.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E004.png"}, {"alt": "Two rows of panels with a common horizontal axis, the probability limit of the estimator, marked at 3/8, 7/16, 1/2, 9/16 and 5/8; a vertical line stands at the ATE of 1/2. In population A, effects independent of compliance but not given X, the 2SLS point (blue circle) lies on the line at 1/2 and the ICSW point (orange diamond) lies to the left at 7/16. In population B, effects independent of compliance given X but not marginally, the 2SLS point lies to the right at 5/8 and the ICSW point lies on the line at 1/2.", "caption": "Probability limits of 2SLS and ICSW in the two exact populations, against the ATE of $1/2$ (vertical line). The marginal and conditional readings of the sentence on p. 499 each fail for one estimator. In population A the ICSW limit is $7/16$, and in population B the 2SLS limit is $5/8$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E004-limits.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T3", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T3"}, {"id": "P029-T4", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T4"}, {"id": "P029-T21", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T21"}], "headline": "Independence of the treatment effects from compliance does not imply Assumption 1: the sentence requires that the mean effect among compliers equal the population mean effect both unconditionally, for 2SLS, and given every covariate profile, for Assumption 1 and ICSW.", "id": "P029-E004", "location": "Section 3.4, p. 499 (PDF p. 8), last sentence of the paragraph beginning “We can describe sufficient conditions for and consequences of Assumption 1”", "pdf_page": 8, "printed_page": "499", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P029-T21", "P029-T3", "P029-T4"], "text": {"as_printed": "More generally, even if treatment effects are heterogeneous, but still \\hl{statistically independent of compliance, Assumption 1 would necessarily be satisfied}, and both 2SLS and ICSW would be consistent for the ATE.", "corrected": "More generally, \\hl{with heterogeneous treatment effects, if the mean treatment effect among compliers equals the population mean both unconditionally and conditional on every covariate profile $\\mathbf{X}$ used in the compliance score}, Assumption 1 \\hl{holds} and, \\hl{under the other stated assumptions and regularity conditions}, both 2SLS and ICSW \\hl{are} consistent for the ATE.\n\nIn symbols, the two conditions are $\\mathrm{E}(Y_1-Y_0|D_1>D_0)=\\mathrm{E}(Y_1-Y_0)$ and $\\mathrm{E}(Y_1-Y_0|D_1>D_0,\\mathbf{X}=\\mathbf{x})=\\mathrm{E}(Y_1-Y_0|\\mathbf{X}=\\mathbf{x})$ for every $\\mathbf{x}\\hl{\\in\\operatorname{Supp}(\\mathbf{X})}$. The first suffices for 2SLS, and the second, which is Assumption 1, suffices for ICSW.", "evidence": "The evidence is two exact counterexamples and the identities that explain them. I write $\\tau=Y_1-Y_0$, $f(\\mathbf{x})=\\Pr(\\mathbf{X}=\\mathbf{x})$, $p(\\mathbf{x})=\\Pr(D_1>D_0|\\mathbf{X}=\\mathbf{x})$ and $\\mu(\\mathbf{x})=\\mathrm{E}(\\tau|D_1>D_0,\\mathbf{X}=\\mathbf{x})$. Under the IV assumptions, with a discrete covariate and correct scores, the limits of 2SLS and of ICSW are\n\\[\n\\theta_{\\mathrm{IV}}=\\frac{\\sum_{\\mathbf{x}}f(\\mathbf{x})p(\\mathbf{x})\\mu(\\mathbf{x})}{\\Pr(D_1>D_0)},\\qquad\n\\theta_{\\mathrm{ICSW}}=\\sum_{\\mathbf{x}}f(\\mathbf{x})\\mu(\\mathbf{x}),\\qquad\n\\theta_{\\mathrm{IV}}-\\theta_{\\mathrm{ICSW}}=\\frac{\\operatorname{Cov}\\{p(\\mathbf{X}),\\mu(\\mathbf{X})\\}}{\\Pr(D_1>D_0)} .\n\\]\nAssumption 1 implies that $\\theta_{\\mathrm{ICSW}}$ equals the ATE, $\\sum_{\\mathbf{x}}f(\\mathbf{x})\\mathrm{E}(\\tau|\\mathbf{X}=\\mathbf{x})$.\n\nBoth populations have a binary covariate $X$, a complier indicator $C$, an effect $\\tau\\in\\{0,1\\}$, outcomes $Y_0=0$ and $Y_1=\\tau$, and $Z\\sim\\mathrm{Bernoulli}(1/2)$ independent of everything else. A unit that is not a complier is an always-taker or a never-taker with probability $1/2$ each, independently. In population A, $C$ and $\\tau$ are independent $\\mathrm{Bernoulli}(1/2)$ variables, and $\\Pr(X=1|C,\\tau)$ is $3/4$ if $C=\\tau=1$ and $1/4$ otherwise. In population B, $X\\sim\\mathrm{Bernoulli}(1/2)$ and, given $X$, the variables $C$ and $\\tau$ are independent with $\\Pr(C=1|X)=\\Pr(\\tau=1|X)$ equal to $1/4$ at $X=0$ and $3/4$ at $X=1$.\n\\begin{audittable}{lrrrr}{The two populations by covariate value.}\nPopulation and stratum & $f(x)$ & $p(x)$ & $\\mathrm{E}(\\tau|X=x)$ & $\\mu(x)$ \\\\\n\\hline\nA, $X=0$ & $5/8$ & $2/5$ & $2/5$ & $1/4$ \\\\\nA, $X=1$ & $3/8$ & $2/3$ & $2/3$ & $3/4$ \\\\\nB, $X=0$ & $1/2$ & $1/4$ & $1/4$ & $1/4$ \\\\\nB, $X=1$ & $1/2$ & $3/4$ & $3/4$ & $3/4$ \\\\\n\\end{audittable}\n\\begin{audittable}{lrr}{The sentence of p.~499 in the two populations.}\nQuantity & Population A & Population B \\\\\n\\hline\n$\\tau$ independent of compliance type & yes & no \\\\\n$\\tau$ independent of compliance type given $X$ & no & yes \\\\\nAssumption 1 & fails & holds \\\\\nATE & $1/2$ & $1/2$ \\\\\nLimit of 2SLS (the LATE) & $1/2$ & $5/8$ \\\\\nLimit of ICSW & $7/16$ & $1/2$ \\\\\n\\end{audittable}\nIn population A, I have $\\theta_{\\mathrm{ICSW}}=(5/8)(1/4)+(3/8)(3/4)=7/16$. In population B, I have $\\theta_{\\mathrm{IV}}=(5/16)/(1/2)=5/8$. Assumption 3 holds with $\\epsilon=1/3$ in A and $\\epsilon=1/8$ in B, and the saturated compliance model of equations (5)--(7) fits both populations, so that the scores are correct and bounded. Solving the IV moment equations in exact arithmetic, with and without $X$ as a regressor, I find the same four limits.", "problem": "Assumption 1 is a condition given the covariates: the mean effect among compliers with profile $\\mathbf{x}$ equals the mean effect among all units with profile $\\mathbf{x}$. Independence of the effects from compliance, without qualification, is independence in the whole population. It implies that the LATE equals the ATE, and 2SLS is then consistent for the ATE. It does not give the conditional equality, because effects and compliance that are independent overall can be dependent among the units that share a covariate value. ICSW with correct scores then converges to the average over $\\mathbf{x}$ of the complier effects, which is not the ATE in general. If the independence is read as conditional on the covariates, Assumption 1 holds and ICSW is consistent, but 2SLS converges to the LATE, which then differs from the ATE in general. Neither of these two readings makes the whole sentence true. The sentence suggests that ICSW cannot hurt when 2SLS already recovers the ATE. The first population below shows that weighting on covariates can introduce a bias.", "what_survives": "The consistency result in the appendix, which applies Assumption 1 between equations (10) and (11), is unaffected by this finding, and so are the preceding sentences on a constant effect, the estimator and every reported number. The sufficient condition is withdrawn. The repair adds a restriction on the effects among compliers, which the applications may not satisfy. Whether they satisfy it is not established here."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E004", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Assumption 2 with its five lines of text. The text reads: Consistency of P-hat C i for P C i. Assumption 2 amounts to requiring proper specification of the ML estimator. Application of a flexible parametrization of the covariates in X, as is possible with large samples, will necessarily yield consistency, so long as the rate of growth in the number of estimated parameters is sufficiently slow in n. In the case of a saturated model and F(.) as the normal CDF, P-hat C i will be unbiased for the conditional probability of compliance. A red outline surrounds the word unbiased in the last line.", "caption": "Assumption 2, p. 500 (PDF p. 9). The outline marks “unbiased” in the last sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T13", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T13"}, {"id": "P029-T14", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T14"}], "headline": "The maximum-likelihood compliance score of a saturated model is consistent but not unbiased: within a covariate stratum it is the difference between the treatment-receipt rates of the two instrument arms, truncated at zero, and under two-sided noncompliance its expectation exceeds the compliance score in every finite sample with both arms nonempty.", "id": "P029-E005", "location": "Assumption 2, last sentence, p. 500 (PDF p. 9); the estimator is defined in Section 3.1, equations (5)–(7), pp. 496–497 (PDF pp. 5–6)", "pdf_page": 9, "printed_page": "500", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T13", "P029-T14"], "text": {"as_printed": "In the case of a saturated model and $F(.)$ as the normal CDF, $\\hat{P}_{Ci}$ will be \\hl{unbiased} for the conditional probability of compliance.", "corrected": "In the case of a saturated model, with a fixed finite set of covariate strata and the number of observations in both instrument arms of every stratum tending to infinity, $\\hat{P}_{Ci}$, allowing boundary fitted probabilities, will be \\hl{consistent} for the conditional probability of compliance. For a unit in stratum $x$ with \\hl{both arms nonempty}, it equals $\\max(\\bar D_{1x} - \\bar D_{0x}, 0)$, the difference between the treatment-receipt rates of the two instrument arms in that stratum truncated below at zero, and it \\hl{need not be unbiased} in finite samples. Under one-sided noncompliance it equals the receipt rate $\\bar D_{1x}$ of the arm $Z = 1$ and is unbiased when that arm is nonempty. These statements hold for any $F$ that maps onto $(0,1)$.", "evidence": "The evidence is a derivation of the estimator and an exact counterexample. For the derivation, I fix a stratum with $n_1 > 0$ units in the arm $Z = 1$ and $n_0 > 0$ units in the arm $Z = 0$, of which $k_1$ and $k_0$ receive treatment. I write $p = P_{A,C}$ and $q = P_{A,C} P_{A|A,C}$ for the two receipt probabilities of the model. Up to a constant, the likelihood (7) of the stratum is\n\\[\nL(p, q) = p^{k_1} (1-p)^{n_1 - k_1} \\, q^{k_0} (1-q)^{n_0 - k_0}, \\qquad 0 \\le q \\le p \\le 1 ,\n\\]\nand the fitted score is $p - q$ at the maximum. If $k_0/n_0 \\le k_1/n_1$, then the unconstrained maximizer $(k_1/n_1, k_0/n_0)$ satisfies the constraint. If $k_0/n_0 > k_1/n_1$, then $k_0 > 0$ and $k_1 < n_1$, so $L$ vanishes on the edges $q = 0$ and $p = 1$. Further, $L$ has no stationary point such that $0 < q < p < 1$, since such a point would be $(k_1/n_1, k_0/n_0)$. The maximum then lies on the diagonal $q = p = t$, where $L(t,t) = t^{k_1 + k_0}(1-t)^{n_1 + n_0 - k_1 - k_0}$ is largest at $t = (k_1 + k_0)/(n_1 + n_0)$. Combining the two cases, I obtain the formula above. A probit model with finite coefficients has $0 < q < p < 1$, and a sample whose maximizer lies on the boundary has a supremum that is not attained. The formula takes the maximum over the closure.\n\nFor the counterexample, I take a single stratum in which a unit is an always-taker with probability $1/4$, a complier with probability $1/2$ and a never-taker with probability $1/4$, with no defiers. One of two independently sampled units is assigned to each arm at random. The receipt probabilities are $p_1 = 3/4$ in the arm $Z = 1$ and $p_0 = 1/4$ in the arm $Z = 0$, and the compliance score is $1/2$. The model is correctly specified, with $P_{A,C} = 3/4$ and $P_{A|A,C} = 1/3$.\n\\begin{audittable}{llll}{The four possible samples with one unit in each arm. Here $d_1$ and $d_0$ are the receipts in the arms $Z = 1$ and $Z = 0$.}\nReceipts $(d_1, d_0)$ & Probability & Fitted $(p, q)$ & Fitted score \\\\\n$(0, 0)$ & $3/16$ & $(0, 0)$ & $0$ \\\\\n$(0, 1)$ & $1/16$ & $(1/2, 1/2)$ & $0$ \\\\\n$(1, 0)$ & $9/16$ & $(1, 0)$ & $1$ \\\\\n$(1, 1)$ & $3/16$ & $(1, 1)$ & $0$ \\\\\n\\end{audittable}\nThe expectation of the fitted score is $9/16$, and the bias is $1/16$. The untruncated difference $d_1 - d_0$ has expectation $3/4 - 1/4 = 1/2$. The estimator replaces its value $-1$ in the second row by $0$.\n\nThe bias is not special to one unit per arm. Let $T = \\bar D_{1x} - \\bar D_{0x}$, and let $c$ be the compliance score. Since $\\max(T, 0) = T + \\max(-T, 0)$ and $\\mathbb{E}[T] = c$, I have\n\\[\n\\mathbb{E}\\bigl[\\hat P_{Ci}\\bigr] - c = \\mathbb{E}\\bigl[\\max(-T, 0)\\bigr] \\ge 0 ,\n\\]\nand the inequality is strict for all arm sizes whenever $p_0 > 0$ and $p_1 < 1$. With five units in each arm, $p_1 = 1/2$ and $p_0 = 3/10$, I enumerate the 36 pairs of counts and find the expectation $15659/64000 \\approx 0.245$, against a compliance score of $1/5$.\n\nConsistency holds. For $c \\ge 0$ and every real $t$, I have $|\\max(t, 0) - c| \\le |t - c|$. Chebyshev's inequality then implies that, for every $\\delta > 0$,\n\\[\n\\Pr\\bigl(|\\hat P_{Ci} - c| > \\delta\\bigr) \\le \\frac{1}{\\delta^2}\\left(\\frac{p_1(1-p_1)}{n_1} + \\frac{p_0(1-p_0)}{n_0}\\right) ,\n\\]\nand the right side tends to zero as $n_1$ and $n_0$ tend to infinity. Under one-sided noncompliance, $p_0 = 0$, so $k_0 = 0$ and $\\hat P_{Ci} = \\bar D_{1x}$, which is unbiased when the arm $Z = 1$ is not empty.", "problem": "The estimator $\\hat P_{Ci}$ is the maximum-likelihood estimator of Section~3.1. By equations (5)--(7), the model sets the probability of receiving treatment to $P_{A,C,i}$ for a unit with $Z_i = 1$ and to $P_{A,C,i} P_{A|A,C,i}$ for a unit with $Z_i = 0$. The compliance score is the difference of the two. Since the second probability is at most the first, the model cannot fit a covariate stratum in which the arm $Z = 0$ receives treatment at a higher rate than the arm $Z = 1$. In a saturated model with both arms nonempty in stratum $x$, let $\\bar D_{1x}$ and $\\bar D_{0x}$ be the treatment-receipt rates in stratum $x$, that is, the shares of units that receive treatment in the arm $Z = 1$ and in the arm $Z = 0$. If $\\bar D_{0x} \\le \\bar D_{1x}$, then the fitted probabilities are these two rates, and the fitted score is $\\bar D_{1x} - \\bar D_{0x}$. If $\\bar D_{0x} > \\bar D_{1x}$, then both fitted probabilities equal the pooled rate, and the fitted score is zero. Hence\n\\[\n\\hat P_{Ci} = \\max\\bigl(\\bar D_{1x} - \\bar D_{0x},\\, 0\\bigr) \\qquad \\text{for every unit } i \\text{ in stratum } x .\n\\]\nThe difference $\\bar D_{1x} - \\bar D_{0x}$ is unbiased for the compliance score, and the truncation replaces its negative values by zero. The expectation of $\\hat P_{Ci}$ is therefore larger than the compliance score whenever a negative difference has positive probability. This is the case in every finite sample when the stratum contains always-takers and never-takers. The printed sentence is true under one-sided noncompliance, where the arm $Z = 0$ never receives treatment and the truncation cannot bind. Note that the normal CDF plays no role: the formula holds for any $F$ that maps onto $(0,1)$.", "what_survives": "Assumption~2 requires that $\\hat P_{Ci}$ be consistent, not that it be unbiased, and the consistency argument of the Appendix uses the assumption in that form. The estimands, the ICSW estimator and its identification argument do not depend on the sentence. The two applications and the simulations fit parametric models and not saturated ones, and none of their calculations uses the sentence. I did not recompute them for this finding. The phrase ``strata means under saturation'' on p.~496 needs the same qualification, since the two fitted probabilities are the separate arm means only when $\\bar D_{0x} \\le \\bar D_{1x}$. What changes is the claim that the score is unbiased, which becomes a claim that it is consistent."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E005", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of footnote 6, seven lines. A red box marks the words 'because c tends to 0 as n tends to infinity' in the sentence 'Asymptotically, assuming that alpha remains fixed, the Winsorization has no impact on estimation'. Earlier in the footnote c is defined as the 1 over n to the alpha quantile of the estimated compliance scores.", "caption": "Footnote 6, p. 498 (PDF p. 7). The marked clause gives the reason for the asymptotic irrelevance of Winsorization.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E006.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T15", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T15"}, {"id": "P029-T16", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T16"}], "headline": "The Winsorization cut-off $c$ need not tend to zero; the fraction of scores changed tends to zero, and under suitable regularity conditions Winsorization leaves the probability limit unchanged.", "id": "P029-E006", "location": "Footnote 6, p. 498 (PDF p. 7), sentence beginning “Asymptotically, assuming that $\\alpha$ remains fixed”", "pdf_page": 7, "printed_page": "498", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T15", "P029-T16"], "text": {"as_printed": "Define $c$ as the $1/n^{\\alpha}$th quantile of $\\hat{P}_{Ci}$, where $\\alpha$ is a positive calibrating constant. We replace all $\\hat{P}_{Ci}<c$ with $c$. [\\ldots] Asymptotically, assuming that $\\alpha$ remains fixed, the Winsorization has no impact on estimation, \\hl{because $c\\to0$ as $n\\to\\infty$}.", "corrected": "In place of the sentence beginning ``Asymptotically'': \\hl{For fixed $\\alpha>0$, the fraction of observations whose scores are changed tends to zero. Under Assumption 3 and regularity conditions ensuring bounded estimated inverse scores with probability tending to one and integrable outcomes (for example, uniform score consistency and $\\mathrm{E}|Y|<\\infty$), Winsorization does not change the probability limit of the displayed ICSW estimator. The cutoff $c$ need not tend to zero. This consistency argument does not establish first-order distributional equivalence.}\n\nThe rest of the footnote is unchanged.", "evidence": "The evidence is an exact counterexample to the limit and a derivation of the corrected statement.\n\n\\emph{Counterexample.} Let the covariate be constant. A unit is an always-taker with probability $1/4$, a complier with probability $1/2$ and a never-taker with probability $1/4$, the assignment $Z\\sim\\mathrm{Bernoulli}(1/2)$ is independent of the type, and $Y_0=0$, $Y_1=1$. The IV assumptions hold, Assumption 1 holds because the effect is $1$ for every unit, and Assumption 3 holds with $\\epsilon=1/4$ because the compliance score is $1/2$. With the exact scores, every quantile of the scores is $1/2$, and $c=1/2$ for every $n$ and every $\\alpha$. With the ML estimator of equations (5)--(7), the fitted score is the same for all units and eventually equals the difference between the proportions treated in the two arms, once those proportions are interior and correctly ordered. By the strong law, $c$ converges almost surely to $3/4-1/4=1/2$. No score is below $c$, the weights are unchanged, and the ICSW ratio is $1$, the ATE, whenever its empirical first stage is nonzero. Note that the example contradicts the reason given and not the conclusion.\n\n\\emph{Derivation.} Let $p_n=n^{-\\alpha}$, let $c_n$ be the cut-off, and let $m_n$ be the number of scores strictly below $c_n$. For the quantile rule that the code uses, I have $m_n/n\\le p_n+1/n$. Suppose that $\\max_{i\\le n}|\\hat{P}_{Ci}-P_{Ci}|\\to0$ in probability and that $\\mathrm{E}|Y|<\\infty$. By Assumption 3 and this convergence, every estimated score exceeds $\\epsilon/2$ with probability tending to one. On that event the weights $w_i=1/\\hat{P}_{Ci}$ and the Winsorized weights $w_i^*=1/\\max(\\hat{P}_{Ci},c_n)$ satisfy $0\\le w_i-w_i^*\\le2/\\epsilon$, and they differ for at most $m_n$ units. For any variables with $|V_i|\\le K$, I obtain\n\\[\n\\Bigl|\\frac1n\\sum_{i=1}^n(w_i-w_i^*)V_i\\Bigr|\\le\\frac{2K}{\\epsilon}\\Bigl(p_n+\\frac1n\\Bigr)\\to0 .\n\\]\nThis bound covers the weighted sums of $\\mathbf{1}\\{Z_i=z\\}$ and $\\mathbf{1}\\{Z_i=z\\}D_i$. For the weighted sums of the outcome, truncating $Y_i$ at a level $M$ and using $\\mathrm{E}|Y|<\\infty$, I find that the change tends to zero in probability. Since the weighted difference in treatment rates has the positive limit $1/\\mathrm{E}\\{1/P_C(\\mathbf{X})\\}$, I conclude that the Winsorized and original estimators have the same probability limit. The bound is of order $n^{-\\alpha}$, which for $\\alpha=0.275$ is not $o(n^{-1/2})$. The argument does not show that the two estimators have the same first-order distribution, and it does not show that they differ.", "problem": "The cut-off $c$ is a quantile of the estimated compliance scores, taken at the level $n^{-\\alpha}$. As $n$ grows the level tends to zero, but the quantile need not, since Assumption 3 keeps the true scores above some $\\epsilon>0$. The sentence's conclusion holds for a different reason. The fraction of scores that Winsorization changes is at most $n^{-\\alpha}+1/n$, and the weights that it changes stay bounded under conditions of the kind that the article already invokes.", "what_survives": "The Winsorization rule, the code, which computes a quantile, and every estimate are unchanged. The appendix does not use the limit $c\\to0$, and consistency holds under the conditions stated below. No application estimate and no simulation result is shown to be wrong. Only the reason given in the footnote changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E006", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Assumption 3 with its eight lines of text. The assumption reads: For all x in Supp(X), Pr(D1 greater than D0 given X = x) lies in the interval from epsilon, excluded, to 1, included, where epsilon is greater than 0. The text continues: Note that IV estimation already requires the assumption of nonzero compliance in the full population, and we generalize this assumption to every covariate profile. This assumption has both theoretical and practical implications. If Assumption 3 fails, then the ATE cannot generally be estimated using the variation induced by Z: the instrument simply does not cause any variation in the treatment received by some types of units. In practice, if a unit has zero probability of compliance, the weighting procedure will produce infinite weights asymptotically, leading to an undefined estimate. Conditional on the validity of Assumptions 1 and 2, Assumption 3 is therefore verifiable in large samples. Red outlines surround the sentence from If Assumption 3 fails to some types of units, which runs over three lines.", "caption": "Assumption 3 and its explanation, p. 500 (PDF p. 9). The outlines mark the sentence that describes the failure of the assumption.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E007.png"}], "formal_release": null, "formal_statements": [], "headline": "Assumption 3 can fail although every covariate profile has compliers; the explanation that follows it describes only the case of a profile with no compliers.", "id": "P029-E007", "location": "Assumption 3 and the explanation that follows it, p. 500 (PDF p. 9)", "pdf_page": 9, "printed_page": "500", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "For all $\\mathbf{x} \\in \\operatorname{Supp}(\\mathbf{X})$, $\\Pr(D_1 > D_0 \\mid \\mathbf{X} = \\mathbf{x}) \\in (\\epsilon, 1]$, where $\\epsilon > 0$. Note that IV estimation already requires the assumption of nonzero compliance in the full population, and we generalize this assumption to every covariate profile. This assumption has both theoretical and practical implications. \\hl{If Assumption 3 fails}, then the ATE cannot generally be estimated using the variation induced by $Z$: \\hl{the instrument simply does not cause any variation in the treatment received by some types of units}. In practice, if a unit has zero probability of compliance, the weighting procedure will produce infinite weights asymptotically, leading to an undefined estimate.", "corrected": "\\hl{If compliance is zero for a set of covariate profiles with positive population probability}, the ATE cannot generally be \\hl{identified} using the variation induced by $Z$. \\hl{Assumption 3 is stronger than this pointwise overlap requirement: a uniformly positive lower bound also rules out arbitrarily large inverse compliance weights.}", "intended_meaning": "The intended point is that a covariate profile without compliers cannot be reweighted: the instrument moves no unit's treatment there, and the inverse of the compliance score is infinite. This is correct, and ``cannot generally'' does not claim an impossibility for every law that violates the assumption. Read literally, however, the sentence equates the failure of Assumption~3 with the existence of such a profile. The assumption asks for more, namely a lower bound $\\epsilon$ that holds for all profiles at once, and the failure of a uniform bound need not imply a profile with zero compliance.\n\nFor an example, let $\\Pr(X = k) = 2^{-k}$ for $k = 1, 2, \\ldots$, and let the compliance score at $X = k$ be $1/k$. Every profile has compliers, and no $\\epsilon > 0$ bounds the scores from below. Suppose further that there are no always-takers or defiers, that $Z$ is assigned independently as a fair coin, that the outcomes are bounded and that the treatment effect is $1$ for every unit. The weights $w = k$ formed from the true scores satisfy $\\mathbb{E}[w] = 2$ and $\\mathbb{E}[w^2] = 6$, and the weighted IV ratio is consistent for the ATE, which is $1$. The example concerns the true scores and says nothing about estimated scores. Assumption~3 as stated, and every result that uses it, is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E007", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of footnote 3. A red box marks the plus sign between '(F(theta A,C x sub i) Z sub i) to the power D sub i' and '(1 minus F(theta A,C x sub i) Z sub i) to the power 1 minus D sub i' in the displayed likelihood. The next sentence calls it the standard likelihood function for binomial regression.", "caption": "Footnote 3, p. 497 (PDF p. 6). The marked plus sign joins the two Bernoulli factors of the one-sided likelihood.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T17", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T17"}], "headline": "In the one-sided likelihood of footnote 3 the two Bernoulli factors are multiplied, not added.", "id": "P029-E008", "location": "Footnote 3, p. 497 (PDF p. 6), displayed likelihood", "pdf_page": 6, "printed_page": "497", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T17"], "text": {"as_printed": "\\[\nL(\\theta_{\\mathbf{C}}|D,Z) = \\Pi_{i=1}^{n}\\bigl[(F(\\hl{\\theta_{\\mathbf{A,C}}}\\mathbf{x_i})Z_i)^{D_i} \\;\\hl{+}\\; (1 - F(\\hl{\\theta_{\\mathbf{A,C}}}\\mathbf{x_i})Z_i)^{1-D_i}\\bigr].\n\\]\nThis likelihood function is a familiar one: it is the standard likelihood function for binomial regression, if only applied to units in the treatment condition.", "corrected": "\\[\nL(\\theta_{\\mathbf{C}}|D,Z) = \\Pi_{i=1}^{n}\\bigl[(F(\\hl{\\theta_{\\mathbf{C}}}\\mathbf{x_i})Z_i)^{D_i} \\;\\hl{\\times}\\; (1 - F(\\hl{\\theta_{\\mathbf{C}}}\\mathbf{x_i})Z_i)^{1-D_i}\\bigr].\n\\]", "evidence": "The evidence is a derivation. Setting $P_{A|A,C,i}=0$ in equation (7), I obtain the likelihood for unit $i$,\n\\[\n(P_{A,C,i}Z_i)^{D_i}(1-P_{A,C,i}Z_i)^{1-D_i}.\n\\] For an assigned unit with $F(\\cdot)=3/4$, the printed factor is $5/4$ when $D_i=0$ and $7/4$ when $D_i=1$, in place of $1/4$ and $3/4$. The one-sided likelihood in the replication code is the product.", "problem": "With a plus sign every factor of the product is at least $1$, and the display is not a likelihood. The reduction of equation (7) and the likelihood for binomial regression, which the next sentence names, both have a product. The parameter is also written $\\theta_{\\mathbf{C}}$ on the left and $\\theta_{\\mathbf{A,C}}$ on the right. With no always-takers these index the same probability, and one subscript is meant.", "what_survives": "Equation (7), the definition of the ML estimate, the sentence on binomial regression and every computed result are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E008", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of equation (6) and the five lines after it. A red box marks 'P sub C,i = Pr(D sub 1i greater than D sub 0i)' in the sentence 'Therefore, we may define the compliance score as'.", "caption": "Section 3.1, p. 496 (PDF p. 5), after equation (6). The marked definition of the compliance score has no conditioning on the covariates.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009.png"}, {"alt": "Crop of a two-line display and the three lines of text after it. A red box marks the left side 'Pr(D = 1 given X = x sub i)'. The right side is 'Pr(D1 greater than D0 given X = x sub i) times Z sub i plus Pr(D0 = 1 given X = x sub i)'.", "caption": "Section 3.1, p. 497 (PDF p. 6), first display. The marked left side conditions on the covariates only, while the right side depends on $Z_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-b.png"}, {"alt": "Crop of equation (9) for the weighted ITT with one line of text above and one below. A red box marks the vertical bar between 'D1 greater than D0' and 'X = x' inside the expectation of Y1 minus Y0 in the numerator.", "caption": "Appendix, p. 504 (PDF p. 13), equation (9). The marked bar is the second conditioning bar in the expectation.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-c.png"}, {"alt": "Crop of one line of text, ending 'a consistent estimator of the weighted ratio will be', and of the ratio estimator written as four weighted sums over four weighted sums. A red box marks the weight 'w sub Ci' with a double hat in the second sum of the denominator; every other weight has one hat.", "caption": "Appendix, p. 504 (PDF p. 13), display after equation (14). The marked weight carries two hats.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-d.png"}, {"alt": "Crop of six displayed lines defining X1, X2, X3, the coefficients a to h, AC sub i, A sub i, C sub i and D sub i. Red boxes mark the closing bracket of the indicator in the line for AC sub i, the closing bracket of the indicator in the line for A sub i, and the first factor 'D sub i' on the right of 'D sub i = D sub i Z sub i + A sub i'.", "caption": "Supplement, p. A1 (PDF p. 1), baseline data generating process. The marked brackets close indicators that have no test $>0$, and the marked $D_i$ stands where $C_i$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-f.png"}, {"alt": "Crop of the heading 'Heterogeneous Treatment Effects, Model is Incorrect', the line 'Same as the baseline DGP, except:' and the two lines defining AC sub i and A sub i with a Logistic(0,3) term. Red boxes mark the closing bracket of each indicator.", "caption": "Supplement, p. A2 (PDF p. 2), design “Heterogeneous Treatment Effects, Model is Incorrect”. The marked brackets close indicators that have no test $>0$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-g.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T18", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T18"}, {"id": "P029-T19", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T19"}, {"id": "P029-T26", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T26"}], "headline": "Six misprints of notation: the compliance score and the receipt probability each lack a conditioning variable, equation (9) has a second conditioning bar, one weight has two hats, and the simulation recipe prints $D_i$ for $C_i$ and omits the test $>0$ in its indicators.", "id": "P029-E009", "location": "Section 3.1, pp. 496–497 (PDF pp. 5–6); Appendix, p. 504 (PDF p. 13), equation (9) and the display after equation (14); Supplement, pp. A1–A2 (PDF pp. 1–2)", "pdf_page": 5, "printed_page": "496–497", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T18", "P029-T19", "P029-T26"], "text": {"as_printed": "\\begin{enumerate}\n\\item p.~496: ``Therefore, we may define the compliance score as $P_{C,i} = \\hl{\\Pr(D_{1i} > D_{0i})}$.''\n\\item p.~497: $\\hl{\\Pr(D=1|\\mathbf{X}=\\mathbf{x}_i)} = \\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x}_i)Z_i + \\Pr(D_0=1|\\mathbf{X}=\\mathbf{x}_i) = P_{A,C,i}(1-P_{A|A,C,i})Z_i + P_{A,C,i}P_{A|A,C,i}$.\n\\item p.~504, equation (9):\n\\[\n\\mathrm{ITT}^w = \\frac{1}{\\overline{w_c}}\\int_\\Omega \\frac{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})\\,\\mathrm{E}(Y_1 - Y_0|D_1 > D_0\\,\\hl{|}\\,\\mathbf{X}=\\mathbf{x})}{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})}f(\\mathbf{x})d\\mathbf{x}.\n\\]\n\\item p.~504, display after equation (14), first ratio of the denominator:\n\\[\n\\Bigl(\\sum_{i=1}^{n}\\hat{w}_{Ci}Z_iD_i\\Bigr)\\Big/\\Bigl(\\sum_{i=1}^{n}\\hl{\\hat{\\hat{w}}_{Ci}}Z_i\\Bigr).\n\\]\n\\item Supplement, p.~A1: $AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\mathbf{N}(0,3)\\hl{]}$, $A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\mathbf{N}(0,3)\\hl{]}AC_i$, $C_i = AC_i - A_i$, $D_i = \\hl{D_i}Z_i + A_i$.\n\\item Supplement, p.~A2: $AC_i = \\mathbf{I}[2 + a + (1 + bX1_i)(1 + cX2_i) + dX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{]}$, $A_i = \\mathbf{I}[-2 + e + (1 + fX1_i)(1 + gX2_i) + hX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{]}AC_i$.\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item p.~496: $P_{C,i} = \\Pr(D_{1i} > D_{0i}\\hl{\\,|\\,\\mathbf{X}=\\mathbf{x}_i})$.\n\\item p.~497: $\\Pr(\\hl{D_i}=1|\\mathbf{X}=\\mathbf{x}_i\\hl{,Z_i}) = \\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x}_i)Z_i + \\Pr(D_0=1|\\mathbf{X}=\\mathbf{x}_i) = P_{A,C,i}(1-P_{A|A,C,i})Z_i + P_{A,C,i}P_{A|A,C,i}$.\n\\item p.~504, equation (9):\n\\[\n\\mathrm{ITT}^w = \\frac{1}{\\overline{w_c}}\\int_\\Omega \\frac{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})\\,\\mathrm{E}(Y_1 - Y_0|D_1 > D_0\\hl{,}\\,\\mathbf{X}=\\mathbf{x})}{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})}f(\\mathbf{x})d\\mathbf{x}.\n\\]\n\\item p.~504, display after equation (14), first ratio of the denominator:\n\\[\n\\Bigl(\\sum_{i=1}^{n}\\hat{w}_{Ci}Z_iD_i\\Bigr)\\Big/\\Bigl(\\sum_{i=1}^{n}\\hl{\\hat{w}_{Ci}}Z_i\\Bigr).\n\\]\n\\item Supplement, p.~A1: $AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\mathbf{N}(0,3)\\hl{>0}]$, $A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\mathbf{N}(0,3)\\hl{>0}]AC_i$, $C_i = AC_i - A_i$, $D_i = \\hl{C_i}Z_i + A_i$.\n\\item Supplement, p.~A2: $AC_i = \\mathbf{I}[2 + a + (1 + bX1_i)(1 + cX2_i) + dX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{>0}]$, $A_i = \\mathbf{I}[-2 + e + (1 + fX1_i)(1 + gX2_i) + hX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{>0}]AC_i$.\n\\end{enumerate}\nThe supplement may also state that assignment is balanced, with $n/2$ units in each arm, as in the code, and the step before equation (11) may be attributed to the law of total expectation. Neither omission is an error.", "evidence": "The evidence is a check of the pages against the neighbouring formulas and the simulation code, with two short calculations. With no always-takers, compliance probability $1/2$ and $Z\\sim\\mathrm{Bernoulli}(1/2)$, I have $\\Pr(D=1|\\mathbf{X})=1/4$, while the printed right side is $0$ for $Z_i=0$ and $1/2$ for $Z_i=1$. For an always-taker assigned to treatment, the identity printed on p.~A1 reads $D_i=D_i+1$. All six simulation scripts in the replication files test whether the latent index is positive and set \\texttt{D <- C*Z + A}.", "problem": "Each item is a slip of notation, and the neighbouring formulas and the simulation code give the intended meaning. The compliance score is conditional on the covariates, since p.~496 first defines $P_{Ci}=\\Pr(D_1>D_0|\\mathbf{X}=\\mathbf{x}_i)$ and p.~497 uses that definition again. In the first display on p.~497 the right side depends on $Z_i$, so that the left side must also condition on $Z_i$. In equation (9) the second bar stands for a comma, as in equations (8) and (10). The weight with two hats is $\\hat{w}_{Ci}$, as in Section 3.3 and in the display at the top of p.~505. In the supplement each indicator is the indicator that the latent index is positive, and the treatment received is $C_iZ_i+A_i$. As printed, $D_i$ is defined in terms of itself.", "what_survives": "The definitions used elsewhere, the argument in the appendix and the simulations are unaffected. Two further points about these passages are not errors. The receipt probability lies strictly in $(0,1)$ under the probit specification of equations (5) and (6). The step before equation (11) is attributed to the law of total probability. It is a correct step, by the law of total expectation."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E009", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of three rows of Table 1, each with a coefficient and an interval in two columns. Age missing: 12.7 with interval (-16.5, 44.1), and 17.7 with interval (-15.9, 50.3). Democrat: 2.6 with interval (0.6, 4.7), and 2.6 with interval (0.7, 04.8). Republican: 3.8 with interval (1.3, 6.2), and 3.2 with interval (0.8, 5.8). A red outline surrounds the upper limit 04.8 of the Democrat interval in the second column.", "caption": "Table 1, rows “Age missing”, “Democrat” and “Republican”, p. 501 (PDF p. 10). The two columns of numbers are LATE (2SLS), (1), and ATE (ICSW), (2). The outline marks the upper limit “04.8” in column (2).", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E010.png"}, {"alt": "Crop of the supplement section headed (3) Bootstrap Confidence Interval Coverage. The text reads: In Table A6, we present percent-sign coverage with the bootstrap for n in 500, 2500, 5000 assuming that the model is correct and alpha = 0.275. Due to computational limitations, we use only 500 bootstrap iterations and 500 simulations for each n. Below it, Table A6, Bootstrap Coverage, has the columns n, 90 percent Interval Coverage and 95 percent Interval Coverage, with the rows 500, 0.94, 0.98; 2500, 0.91, 0.94; 5000, 0.92, 0.95. A red outline surrounds the percent sign and the word coverage in the first line.", "caption": "Supplement, p. A9 (supplement PDF p. 9): the sentence that introduces Table A6, and the table. The outline marks “% coverage”; the entries of the table are proportions.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E010-A6.png"}], "formal_release": null, "formal_statements": [], "headline": "Three labels do not match what they label: the Democrat interval in column (2) of Table 1 ends at 4.8, printed “04.8”; Table A6 reports coverage as proportions and not as percentages; and the simulation scripts carry the comment “ATE is always 1”, while the average effect in the simulated population is 5.", "id": "P029-E010", "location": "Table 1, row “Democrat”, column (2), p. 501 (PDF p. 10); Section 4.2, p. 502 (PDF p. 11), with supplement Table A8, p. A12 (supplement PDF p. 12); supplement p. A9 (supplement PDF p. 9), the sentence that introduces Table A6; the simulation scripts of the replication files", "pdf_page": 10, "printed_page": "501", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table~1, row ``Democrat'', p.~501.} Column (1): 2.6 $(0.6, 4.7)$. Column (2): 2.6 $(0.7, \\hl{0}4.8)$.\n\n\\emph{Section~4.2, p.~502.} The compliance score most strongly correlates with political interest, newspaper reading, and education (see the supplementary appendix for a full correlation matrix).\n\n\\emph{Supplement, p.~A9.} In Table A6, we present \\hl{\\% coverage} with the bootstrap for $n \\in 500, 2500, 5000$ assuming that the model is correct and $\\alpha = 0.275$.\n\\begin{audittable}{rrr}{Supplement Table~A6: Bootstrap Coverage.}\n$n$ & 90\\% Interval Coverage & 95\\% Interval Coverage \\\\\n500 & 0.94 & 0.98 \\\\\n2500 & 0.91 & 0.94 \\\\\n5000 & 0.92 & 0.95 \\\\\n\\end{audittable}\n\\emph{Simulation scripts.} \\texttt{\\# ATE is always \\hl{1}}", "corrected": "\\emph{Table~1, row ``Democrat'', column (2).} 2.6 $(0.7, \\hl{4.8})$.\n\n\\emph{Supplement, p.~A9.} In Table A6, we present \\hl{coverage proportions} with the bootstrap for $n \\in 500, 2500, 5000$ assuming that the model is correct and $\\alpha = 0.275$.\n\n\\emph{Simulation scripts.} \\texttt{\\# Superpopulation ATE is \\hl{5}; heterogeneous sample-average effects vary, while the homogeneous sample effect is 5.}\n\n\\emph{Optional clarification of Section~4.2.} The \\hl{Winsorized} compliance score correlates most strongly with political interest and newspaper reading, \\hl{followed by party identification and education} (see the supplementary appendix for a full correlation matrix). Alternatively, the printed sentence may be kept and labelled as referring to the score before Winsorization.", "evidence": "The evidence is a check of the pages and two recalculations. Integrating each of the two effect functions over the three covariates, I obtain 5. From the replication data, the correlations of the score before Winsorization are 0.71 with political interest, 0.70 with newspaper reading, 0.40 with education and 0.32 with party ID. After Winsorization with $\\alpha = 0.275$ they are 0.65, 0.62, 0.36 and 0.37, which are the entries of Table~A8. The comment is at line 44 of \\texttt{Simulations Alpha 6-20.R} and of the three \\texttt{Simulations Misspecification} scripts named \\texttt{Right}, \\texttt{Wrong Model} and \\texttt{Only X1}, and at line 45 of \\texttt{Simulations Misspecification Noise 6-20.R}.", "problem": "In Table~1 the upper limit of the Democrat interval in column (2) is printed with a leading zero. The supplement announces ``\\% coverage'', and the entries of Table~A6 are proportions: 0.94 stands for 94\\%. The five simulation scripts carry the comment that the ATE is always 1. They generate the treatment effect $5(1 + X_1 + X_2 + X_3)$ in the baseline design and $5(1 + X_1 + X_2 + X_1 X_2 + X_3)$ in the design with an interaction, with independent covariates that are uniform on $(-1, 1)$. The average effect in the population is then 5, and the average effect in a simulated sample varies from sample to sample. In the homogeneous design the effect is 5 for every unit.\n\nThe sentence of Section~4.2 is not an error; the optional clarification below makes explicit which score it describes. The sentence names political interest, newspaper reading and education as the strongest correlates of the compliance score, while Table~A8 reports a correlation of 0.37 for party ID and of 0.36 for education. Table~A8 is computed with the Winsorized score. For the score before Winsorization, the three strongest correlates are those of the sentence. The sentence does not say which score it describes.", "what_survives": "No estimate, interval or fit changes. The label of Table~A6 and the comment do not enter a computation. The limit 4.8 is the printed value without its leading zero; I obtain 4.7515 in the full bootstrap, which rounds to 4.8."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E010", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of a table with the caption Table A3: Noise (Homogeneous). The columns are n, OLS (Z), OLS (D), 2SLS and ICSW. The rows read: 500: 2.50, 2.64, 0.53, 0.58. 1000: 2.50, 2.62, 0.37, 0.39. 2500: 2.49, 2.61, 0.23, 0.23. 5000: 2.49, 2.64, 0.16, 0.16. 10000: 2.47, 2.61, 0.11, 0.11. 25000: 2.48, 2.63, 0.07, 0.07. One red outline surrounds all 24 entries of the four estimator columns.", "caption": "Supplement, Table A3, p. A7 (supplement PDF p. 7). The outline marks the 24 RMSE entries; their replication formulas subtract the heterogeneous sample-average effect $B$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011.png"}, {"alt": "Crop of three lines. A bold heading reads Homogeneous Treatment Effects, Conditioning on Random Noise. The next line reads Same as the previous DGP, except: and the last line reads tau sub i = 5, for all i. A red outline surrounds the last line.", "caption": "Supplement, p. A2 (supplement PDF p. 2). The outline marks the line that sets the treatment effect of every unit to $5$ in the homogeneous design.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011-b.png"}, {"alt": "Listing of R code with line numbers, in two parts. In the first part, line 22 sets tau1 to 5*(1+X1+X2+X3), line 24 sets Y2 to Y0 + 5*(D), line 26 returns list(D,Y1,tau1,C,Y2), line 83 sets output[1] to mean(dgpout[[3]]), and lines 91 to 94 store in output[9] to output[12] the four estimates computed from Y2. Lines 125 and 127 compute t500 and t1000 as the square root of the column means of (x - simoutput[1,isN]) squared over rows 9:12; a red outline surrounds simoutput[1,isN] on each of the two lines. In the second part, line 60 of the display script has the same statement for t500 with the same red outline, line 72 calls xtable on the six results, and line 74 plots them with the title Noise (Homogeneous).", "caption": "Source code of the replication files, not a page of the article or of the supplement: Simulations Misspecification Noise 6-20.R, lines 22–26, 83, 91–94 and 124–127, and Sim-Display Misspec.R, lines 59–60 and 72–74. The outlines mark the benchmark simoutput[1,isN] in the RMSE statements for rows 9 to 12.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011-c.png"}, {"alt": "Line chart. The horizontal axis is the sample size n on a logarithmic scale, with ticks at 500, 1,000, 2,500, 5,000, 10,000 and 25,000. The vertical axis is labelled RMSE or standard deviation (units of the outcome) and runs from 0 to about 0.65. The title reads: Table A3 measures error about a benchmark that is itself random, where the effect in the homogeneous design is exactly 5. An orange line with square markers gives the ICSW RMSE printed in Table A3: 0.58, 0.39, 0.23, 0.16, 0.11 and 0.07. A blue line with round markers gives the printed 2SLS RMSE: 0.53, 0.37, 0.23, 0.16, 0.11 and 0.07. A dashed green line with triangular markers gives the standard deviation of the benchmark used by the code, 5 divided by the square root of n: 0.224, 0.158, 0.100, 0.071, 0.050 and 0.032. A note below the chart reads: Corrected RMSEs are not shown: they need the simulation output or a new run.", "caption": "The RMSEs of 2SLS and ICSW printed in Table A3 and the standard deviation $5/\\sqrt{n}$ of the benchmark that the code subtracts from each estimate, by sample size. The benchmark has mean $5$, the true effect, and its standard deviation falls from $0.224$ at $n = 500$ to $0.032$ at $n = 25{,}000$. Corrected RMSEs are not shown, because they are not established.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011-benchmark.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T5", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T5"}, {"id": "P029-T6", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T6"}, {"id": "P029-T22", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T22"}], "headline": "The replication code for Table A3 and the “Noise (Homogeneous)” panel of Figure A3 computes RMSEs about the sample-average effect of the companion heterogeneous design, a random quantity with mean $5$ and variance $25/n$; the effect in the homogeneous design is exactly $5$, and the RMSEs should be computed about $5$.", "id": "P029-E011", "location": "Supplement: Table A3, p. A7 (supplement PDF p. 7); Figure A3, panel “Noise (Homogeneous)”, p. A6 (supplement PDF p. 6); definition of the design, p. A2 (supplement PDF p. 2). Replication files: Simulations Misspecification Noise 6-20.R, lines 125–135, and Sim-Display Misspec.R, lines 60–70", "pdf_page": 7, "printed_page": "A7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T22", "P029-T5", "P029-T6"], "text": {"as_printed": "\\emph{Supplement, p.~A2.} \\textbf{Homogeneous Treatment Effects, Conditioning on Random Noise}\n\nSame as the previous DGP, except: $\\tau_i = 5, \\forall i$.\n\n\\emph{Supplement, p.~A7.}\n\\begin{audittable}{rrrrr}{Table A3: Noise (Homogeneous)}\n$n$ & OLS ($Z$) & OLS ($D$) & 2SLS & ICSW \\\\\n500 & 2.50 & 2.64 & 0.53 & 0.58 \\\\\n1000 & 2.50 & 2.62 & 0.37 & 0.39 \\\\\n2500 & 2.49 & 2.61 & 0.23 & 0.23 \\\\\n5000 & 2.49 & 2.64 & 0.16 & 0.16 \\\\\n10000 & 2.47 & 2.61 & 0.11 & 0.11 \\\\\n25000 & 2.48 & 2.63 & 0.07 & 0.07 \\\\\n\\end{audittable}\n\n\\emph{Replication files, \\texttt{Simulations Misspecification Noise 6-20.R}, lines 22, 24, 26, 83 and 125. Line 125 is broken here after \\texttt{function(x)}. Lines 127, 129, 131, 133 and 135 repeat it for the other sample sizes, and lines 60 to 70 of \\texttt{Sim-Display Misspec.R} repeat all six.}\n\\[\n\\begin{aligned}\n&\\texttt{tau1<- 5*(1+X1+X2+X3)}\\\\\n&\\texttt{Y2 <- Y0 + 5*(D)}\\\\\n&\\texttt{return(list(D,Y1,tau1,C,Y2))}\\\\\n&\\texttt{output[1] <- mean(dgpout[[3]])}\\\\\n&\\texttt{t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)}\\\\\n&\\qquad\\texttt{(x - }\\hl{\\texttt{simoutput[1,isN]}}\\texttt{)\\textasciicircum 2))\\textasciicircum .5,mean(simoutput[7,isN]))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files.} In the six RMSE statements for rows 9 to 12 of \\texttt{Simulations Misspecification Noise 6-20.R} (lines 125, 127, 129, 131, 133 and 135) and in the six corresponding statements of \\texttt{Sim-Display Misspec.R} (lines 60, 62, 64, 66, 68 and 70), the benchmark is the constant $5$:\n\\[\n\\begin{aligned}\n&\\texttt{t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)}\\\\\n&\\qquad\\texttt{(x - }\\hl{\\texttt{5}}\\texttt{)\\textasciicircum 2))\\textasciicircum .5,mean(simoutput[7,isN]))}\n\\end{aligned}\n\\]\nRow~1 and the statements for rows 2 to 5, which score the heterogeneous design, stay as they are.\n\n\\emph{Supplement.} For the design with $\\tau_i = 5$ for all $i$, the RMSE of estimator $j$ at sample size $n$ is\n\\[\n\\mathrm{RMSE}_j = \\Bigl(\\frac{1}{R}\\sum_{r=1}^{R}(t_{jr} - \\hl{5})^2\\Bigr)^{1/2},\n\\]\nwhere $t_{jr}$ is the estimate in the simulated dataset $r$ and $R = 2{,}500$ is the number of simulated datasets. Regenerate Table~A3 and the ``Noise (Homogeneous)'' panel of Figure~A3 with this formula, from the saved output of the simulation or from a new run. The corrected entries are not established here.", "evidence": "The evidence is an exact counterexample and a derivation, together with a reading of the two scripts. Let $t_{jr}$ be the estimate of estimator $j$ in the simulated dataset $r$, let $B_r$ be the benchmark of that dataset, and let $R$ be the number of simulated datasets for a given $n$. Both scripts return\n\\[\n\\Bigl(\\frac{1}{R}\\sum_{r=1}^{R}(t_{jr} - B_r)^2\\Bigr)^{1/2} \\quad\\text{where the design requires}\\quad \\Bigl(\\frac{1}{R}\\sum_{r=1}^{R}(t_{jr} - 5)^2\\Bigr)^{1/2}.\n\\]\n\nFor the counterexample, I take $n = 500$ units in $50$ identical blocks of ten. Set $a = \\dots = h = 0$, let the latent covariates be $(X1_i, X2_i, X3_i) = (1/5, 0, 0)$ for every unit, and let all three disturbances be zero. In each block the observed covariates take the five values $(0,0,0)$, $(1/2,0,0)$, $(0,1/2,0)$, $(0,0,1/2)$ and $(-1/2,-1/2,-1/2)$, each once with $Z_i = 0$ and once with $Z_i = 1$. Then $AC_i = 1$ and $A_i = 0$ for every unit, so that $D_i = Z_i$, $Y0_i = 5$ and $Y2_i = 5 + 5Z_i$. The regressors and the instruments coincide and have full rank, and the 2SLS coefficient on $D$ is exactly $5$. Since $\\texttt{tau1}_i = 5(1 + 1/5) = 6$ for every unit, I have $B = 6$. The code assigns the squared error $(5-6)^2 = 1$ to an estimate that equals the true effect. A second input differs only in the disturbance of the outcome, which equals $Z_i$. For this input, I have $Y2_i = 5 + 6Z_i$, the 2SLS estimate is $6$, and again $B = 6$. The code assigns the squared error $0$ to an estimate that is off by $1$.\n\\begin{audittable}{lrrrr}{The two exact inputs with $n = 500$: the 2SLS estimate $t$, the benchmark $B$ that the code subtracts, and the two squared errors.}\nInput & $t$ & $B$ & $(t-B)^2$, as coded & $(t-5)^2$, correct \\\\\nFirst & 5 & 6 & 1 & 0 \\\\\nSecond & 6 & 6 & 0 & 1 \\\\\n\\end{audittable}\nThe original functions of the replication files, run on the first input, return $5$ for all four homogeneous fits, and the RMSE expressions of both scripts return $1$ for 2SLS; with the benchmark replaced by $5$ they return $0$. Since the thresholds hold with strict inequality and the designs are nonsingular in both inputs, the same discrepancy occurs on a set of inputs of positive probability.\n\nI compute the variance of the benchmark from the design. A variable that is uniform on $(-1,1)$ has mean $0$ and variance $1/3$, and $B - 5$ is $5/n$ times a sum of $3n$ independent such variables. I obtain\n\\[\nE(B) = 5, \\qquad \\operatorname{Var}(B) = \\frac{25}{n^2}\\cdot 3n\\cdot\\frac{1}{3} = \\frac{25}{n}.\n\\]\nFor any estimate $t$, I have\n\\[\n(t - B)^2 - (t - 5)^2 = (B - 5)^2 - 2(t - 5)(B - 5).\n\\]\nIf $t$ has a finite second moment, taking expectations gives $E[(t-B)^2] - E[(t-5)^2] = 25/n - 2\\operatorname{Cov}(t, B)$. The two inputs above show that the difference can have either sign in a single dataset. The size of the change in each entry of Table~A3, and whether the entry rises or falls, is therefore not established.", "problem": "In the homogeneous design every unit has the treatment effect $5$. This implies that the sample-average effect, the ATE and the LATE are all equal to $5$, and that the RMSE of an estimator is its root mean squared error about $5$. The simulation script generates the homogeneous outcome $Y2 = Y0 + 5D$ together with the outcome of the heterogeneous design, and it stores the four estimates from $Y2$ in rows 9 to 12 of its output. It scores these rows against row~1. Row~1 holds the mean of $\\texttt{tau1}_i = 5(1 + X1_i + X2_i + X3_i)$, the sample-average effect of the heterogeneous design. This benchmark is\n\\[\nB = 5 + \\frac{5}{n}\\sum_{i=1}^{n}(X1_i + X2_i + X3_i),\n\\]\na random quantity with mean $5$ and variance $25/n$. Both reporting scripts therefore compute a root mean squared distance from $B$ for every estimator and sample size in Table~A3 and the corresponding panel of Figure~A3, instead of the required RMSE about $5$. The corrected values cannot be computed from the printed ones, because the difference depends on how each estimate varies with $B$ across the simulated datasets. They require the saved output of the simulation, which is not among the replication files, or a new run.", "what_survives": "The definition of the homogeneous design and the fitted estimates are not affected, and 2SLS and ICSW both estimate the constant effect $5$ in this design. The correction changes the scoring of one supplementary design: Table~A3 and one panel of Figure~A3 have to be regenerated. The identification and consistency arguments, the two applications, the other simulation designs, the study of the Winsorization constant and the bootstrap study do not depend on it. Whether the corrected RMSEs of 2SLS and ICSW remain close to each other is not established."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E011", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of Table 2, titled 2SLS and ICSW estimates for Albertson and Lawrence (2009). Under Knowledge are the columns LATE (2SLS), (1), and ATE (ICSW), (2); under Opinion are the columns LATE (2SLS), (3), and ATE (ICSW), (4). Row Watching debate: 0.28 with interval (-0.04, 0.59); 0.40 with interval (0.01, 1.26); -0.07 with interval (-0.25, 0.13); -0.05 with interval (-0.31, 0.24). Row Intercept: 1.81 with interval (1.33, 2.26); 1.90 with interval (1.34, 2.50); 1.01 with interval (0.70, 1.30); 0.99 with interval (0.60, 1.29). A red outline surrounds column (4), from its heading ATE (ICSW) to the interval (0.60, 1.29).", "caption": "Table 2, title, column headings and first two rows, p. 503 (PDF p. 12). The outline marks column (4), the opinion ICSW model, whose upper limits line 195 of the script is written to assemble. The printed limits are finite.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E012.png"}, {"alt": "Image of R code in a monospaced face, under the heading albertsonlawrence.R, lines 193-195 (replication files). Line 193, over three rows: mat1 <- round(cbind(apply(ests,2,quantile,c(0.025))[1:10],apply(ests,2,quantile,c(0.025))[11:20],apply(ests,2,quantile,c(0.025))[21:30],apply(ests,2,quantile,c(0.025))[31:40]),digits=2). Line 194 is empty. Line 195, over three rows: mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],apply(ests,2,quantile,c(0.975))[11:20],apply(ests,2,quantile,c(0.975))[21:30],apply(ests,2,quantile,c(0.975))[310]),digits=2). A red outline surrounds the subscript [310] near the end of line 195.", "caption": "Replication script albertsonlawrence.R, lines 193–195, typeset from the replication files (a script has no published page). The outline marks the subscript [310] in line 195; line 193 has [31:40] in the same place.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E012-code.png"}], "formal_release": null, "formal_statements": [], "headline": "In the line of the replication script that assembles the upper bootstrap limits for Table 2, the last subscript [310] should read [31:40]; as deposited, the line returns missing values for the ten upper limits of the opinion ICSW model, and no printed limit of Table 2 is shown to be wrong.", "id": "P029-E012", "location": "Replication script albertsonlawrence.R, line 195; the paper’s counterpart is Table 2, column (4), upper confidence limits, p. 503 (PDF p. 12)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Script, lines 193 and 195 (each is one line of code, broken here after each argument).}\n\\[\n\\begin{aligned}\n&\\texttt{mat1 <- round(cbind(apply(ests,2,quantile,c(0.025))[1:10],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.025))[11:20],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.025))[21:30],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.025))[31:40]),digits=2)}\\\\\n&\\texttt{mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[11:20],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[21:30],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))}\\hl{\\texttt{[310]}}\\texttt{),digits=2)}\n\\end{aligned}\n\\]\n\n\\emph{Table~2, column (4), Opinion, ATE (ICSW), p.~503.} Watching debate: $-0.05$ $(-0.31, 0.24)$.", "corrected": "\\emph{Script, line 195 (one line of code).}\n\\[\n\\begin{aligned}\n&\\texttt{mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[11:20],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[21:30],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))}\\hl{\\texttt{[31:40]}}\\texttt{),digits=2)}\n\\end{aligned}\n\\]", "evidence": "The evidence is an exact counterexample. I ran the two lines as deposited on a matrix with two rows and 40 columns such that the entries in column $j$ are $j$ and $j + 40$. The quantiles of column $j$ at $0.025$ and $0.975$ are $j + 1$ and $j + 39$. Line 193 returns the four intended columns. Line 195 returns $40, \\ldots, 49$, then $50, \\ldots, 59$, then $60, \\ldots, 69$, and ten missing values in the fourth column. With \\texttt{[31:40]} it returns $70, \\ldots, 79$ there and leaves the other columns unchanged. Neither \\texttt{mat1} nor \\texttt{mat2} is used elsewhere in the replication scripts, and the script saves its workspace at line 155, before both lines.", "problem": "The bootstrap matrix \\texttt{ests} of the script has 40 columns: ten coefficients for each of four models, in the order knowledge 2SLS, knowledge ICSW, opinion 2SLS and opinion ICSW. Line 193 assembles the lower limits from the columns 1 to 10, 11 to 20, 21 to 30 and 31 to 40. Line 195 does the same for the upper limits, except that its last subscript is \\texttt{[310]}. The vector of 40 quantiles has no element 310. R returns a single missing value there, and \\texttt{cbind} repeats it ten times. The fourth column of \\texttt{mat2}, which should hold the upper limits of the opinion ICSW model in column (4) of Table~2, is then missing in all ten rows.", "what_survives": "The typo comes after the point-estimation and bootstrap calculations of the script. The point estimates, the bootstrap draws, the quantiles of the differences in coefficients, the matrix of lower limits and the first three columns of upper limits do not pass through it. Table~2 prints ten finite upper limits in column (4), and none is shown to be wrong. How the table was assembled from the output is not known. Bootstrap draws with missing coefficients are a separate question, which the repair does not address, and I did not establish that any occurred."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E012", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of R code with line numbers. Lines 39 and 40 read return( and -sum(weights*log((p.D)^D * (1 - p.D)^(1 - D)))). After a gap, line 45 begins the one-sided function with if (onesided == TRUE) complier.neg.logLik.fnc <- function (par), lines 46 to 48 set theta, p.C and p.D <- p.C * Z, and lines 49 and 50 read return( and -sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights))). A red outline surrounds the characters * weights on line 50.", "caption": "Source code of the replication files, not a page of the article: icsw.R, lines 39–41 and 45–52. The outline marks the weights inside the logarithm on line 50; on line 40 the weights multiply the logarithm.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E014.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T8", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T8"}], "headline": "In the one-sided branch of the function complier.score, the observation weights are inside the logarithm, so positive weights leave the likelihood minimizers unchanged; the weights should multiply the logarithm, as in the two-sided branch.", "id": "P029-E014", "location": "Replication files, icsw.R, function complier.score, branch for one-sided noncompliance, lines 45–52 (objective on line 50); the objective for two-sided noncompliance is on lines 39–40", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P029-T8"], "text": {"as_printed": "\\emph{Replication files, \\texttt{icsw.R}, lines 39--40 (two-sided noncompliance) and lines 49--50 (one-sided noncompliance).}\n\\[\n\\begin{aligned}\n&\\texttt{return(}\\\\\n&\\quad\\texttt{-sum(weights*log((p.D)\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D))))}\\\\\n&\\texttt{return(}\\\\\n&\\quad\\texttt{-sum(log((p.D)\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D)}\\hl{\\texttt{ * weights}}\\texttt{)))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files, \\texttt{icsw.R}, lines 49--50.}\n\\[\n\\begin{aligned}\n&\\texttt{return(}\\\\\n&\\quad\\texttt{-sum(}\\hl{\\texttt{weights[weights > 0]*}}\\texttt{log(((p.D)\\textasciicircum D *}\\\\\n&\\qquad\\texttt{(1 - p.D)\\textasciicircum (1 - D))}\\hl{\\texttt{[weights > 0]}}\\texttt{)))}\n\\end{aligned}\n\\]\nIn full, for weights $w_i \\ge 0$ the one-sided branch minimizes over $\\theta$\n\\[\n-\\sum_{i:\\,w_i > 0} w_i \\log L_i(\\theta), \\qquad L_i(\\theta) = p_i(\\theta)^{D_i}\\bigl(1 - p_i(\\theta)\\bigr)^{1 - D_i}, \\qquad p_i(\\theta) = Z_i\\,F(x_i^{\\top}\\theta),\n\\]\nwhere $x_i$ consists of an intercept and the standardized covariates of unit $i$, and $F$ is the distribution function passed as \\texttt{link}, by default $\\Phi$. Observations with weight zero are left out before the logarithm is taken. The logarithm is evaluated as $\\log p_i$ when $D_i = 1$ and as $\\log(1 - p_i)$ when $D_i = 0$, which avoids the product $0\\cdot\\log 0$ for units with $Z_i = 0$.", "evidence": "The evidence is an exact counterexample and a run of the original function. The example has six observations with one covariate $W_i$.\n\\begin{audittable}{rrrrr}{Six observations with one-sided noncompliance and positive weights.}\n$i$ & $W_i$ & $Z_i$ & $D_i$ & $w_i$ \\\\\n1 & $-1$ & 1 & 0 & 1 \\\\\n2 & $-1$ & 1 & 1 & 3 \\\\\n3 & $1$ & 1 & 0 & 3 \\\\\n4 & $1$ & 1 & 1 & 1 \\\\\n5 & $-1$ & 0 & 0 & 5 \\\\\n6 & $1$ & 0 & 0 & 7 \\\\\n\\end{audittable}\nWith an intercept and one slope, the fitted probabilities $p_-$ and $p_+$ of compliance at $W_i = -1$ and $W_i = 1$ range freely over $(0,1)^2$, and the two control observations have $L_i = 1$. For $a, b > 0$, the function $-a\\log p - b\\log(1-p)$ has its unique minimum at $p = a/(a+b)$. The function on line 50 is $-\\log\\bigl(p_-(1-p_-)\\,p_+(1-p_+)\\bigr)$ plus a constant, and its minimum is at $(p_-, p_+) = (1/2, 1/2)$. The weighted negative log likelihood is $-\\log\\bigl(p_-^3(1-p_-)\\,p_+(1-p_+)^3\\bigr)$, and its minimum is at $(3/4, 1/4)$. The weighted likelihood $\\prod_i L_i^{w_i}$ equals $1/256$ at the first point and $729/65536$ at the second.\n\nThe original function was also run on eight observations: at each of $W_i = -1$ and $W_i = 1$, three units with $Z_i = 1$ and outcomes $D_i = 0, 0, 1$, with weights $1, 1, 6$ and $3, 3, 2$, and one unit with $Z_i = 0$. It returns the score $1/3$ for every unit, with unit weights and with these weights. The weighted fit is $3/4$ at $W_i = -1$ and $1/4$ at $W_i = 1$; the corrected objective and a weighted probit regression both return it. With a weight equal to zero, the original objective is infinite at the starting value.", "problem": "Let $w_i$ be the weight of observation $i$ and let $L_i(\\theta)$ be its Bernoulli likelihood contribution. Line 50 computes\n\\[\n-\\sum_i \\log\\bigl(w_i L_i(\\theta)\\bigr) = -\\sum_i \\log L_i(\\theta) - \\sum_i \\log w_i .\n\\]\nThe last sum does not depend on $\\theta$. This implies that, for positive weights, the minimizers of this function are those of the unweighted negative log likelihood. A zero weight makes the function infinite. The function meant is the weighted negative log likelihood $-\\sum_i w_i \\log L_i(\\theta)$, which line 40 computes for two-sided noncompliance. No script of the replication files calls \\texttt{complier.score} with \\texttt{onesided=TRUE}, and no reported number is affected. Unequal weights therefore do not weight the fit as intended.", "what_survives": "The canvassing application fits its scores by a weighted probit regression on the units with $Z_i = 1$ and does not use this branch. The debate application and the simulations use the two-sided branch, whose objective weights the logarithm correctly. All 15 calls to \\texttt{complier.score} in the replication files leave \\texttt{onesided} at its default \\texttt{FALSE}. No table, simulation or result of the paper changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E014", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of R code with line numbers. Line 15 declares complier.score with the arguments D, Z, W, weights = NULL, link = pnorm, genoud = TRUE, num.iter and onesided=FALSE. Lines 32 and 33 set p.AC and p.A to link(rowSums(...)) and line 47 sets p.C to link(rowSums(...)). Line 86 sets CA.score to pnorm(rowSums(...)), line 87 sets A.score to pnorm(rowSums(...)) times CA.score, line 88 sets C.score to CA.score - A.score, and line 98 sets C.score to pnorm(rowSums(...)) in the one-sided branch. Red outlines surround the word pnorm on lines 86, 87 and 98.", "caption": "Source code of the replication files, not a page of the article: icsw.R, lines 15, 32–33, 47, 86–89 and 98–99. The outlines mark pnorm in the three statements that compute the returned scores; the likelihood on lines 32, 33 and 47 uses link.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E015.png"}], "formal_release": null, "formal_statements": [], "headline": "The function complier.score fits the likelihood with the distribution function passed as link and then computes the returned scores with the normal distribution function; the two agree for the default probit, which every analysis of the paper uses.", "id": "P029-E015", "location": "Replication files, icsw.R, function complier.score, lines 86–87 and 98; the argument link is declared on line 15 and used in the likelihood on lines 32–33 and 47", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Replication files, \\texttt{icsw.R}, lines 32--33 and 47 (likelihood) and lines 86, 87 and 98 (returned scores). The last three lines are broken here after the multiplication sign.}\n\\[\n\\begin{aligned}\n&\\texttt{p.AC <- link(rowSums(t(t(W1) * theta)))}\\\\\n&\\texttt{p.A <- link(rowSums(t(t(W1) * theta.A)))}\\\\\n&\\texttt{p.C <- link(rowSums(t(t(W1) * theta)))}\\\\\n&\\texttt{CA.score <- (}\\hl{\\texttt{pnorm}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par)/2)]))))}\\\\\n&\\texttt{A.score <- (}\\hl{\\texttt{pnorm}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[(length(opt.log\\$par)/2+1):(length(opt.log\\$par))])))) * CA.score}\\\\\n&\\texttt{C.score <- (}\\hl{\\texttt{pnorm}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par))]))))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files, \\texttt{icsw.R}, lines 86, 87 and 98.}\n\\[\n\\begin{aligned}\n&\\texttt{CA.score <- (}\\hl{\\texttt{link}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par)/2)]))))}\\\\\n&\\texttt{A.score <- (}\\hl{\\texttt{link}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[(length(opt.log\\$par)/2+1):(length(opt.log\\$par))])))) * CA.score}\\\\\n&\\texttt{C.score <- (}\\hl{\\texttt{link}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par))]))))}\n\\end{aligned}\n\\]\nIn full, with fitted coefficient vectors $\\widehat\\theta$ and $\\widehat\\theta_A$ and the distribution function $F$ passed as \\texttt{link}, the two-sided branch returns the always-taker score $F(x_i^{\\top}\\widehat\\theta)\\,F(x_i^{\\top}\\widehat\\theta_A)$ and the complier score $F(x_i^{\\top}\\widehat\\theta)\\bigl(1 - F(x_i^{\\top}\\widehat\\theta_A)\\bigr)$, and the one-sided branch returns the complier score $F(x_i^{\\top}\\widehat\\theta)$. Calls with the default probit are unchanged.", "evidence": "The evidence is an exact counterexample and a run of the original function. Let $F(t) = 1/(1 + 3e^{-t})$, which is a distribution function with $F(0) = 1/4$, and recall that $\\Phi(0) = 1/2$. The data have $64$ observations with one covariate $W_i = \\pm 1$ and $16$ observations in each of the four cells of $(W_i, Z_i)$. In each cell with $Z_i = 0$ one unit is treated, and in each cell with $Z_i = 1$ four units are treated. At the zero coefficient vector, the model assigns probability $q = F(0) = 1/4$ to being a complier or an always-taker and probability $r = F(0) = 1/4$ to being an always-taker given either type. The fitted probabilities of treatment are then $qr = 1/16$ when $Z_i = 0$ and $q = 1/4$ when $Z_i = 1$. Since these equal the observed proportions in all four cells, the zero vector maximizes the likelihood, and it is the unique maximizer because $F$ is strictly increasing. The fitted model has always-taker score $qr = 1/16$ and complier score $q(1-r) = 3/16$. The function returns $\\Phi(0)\\Phi(0) = 1/4$ and $\\Phi(0)(1 - \\Phi(0)) = 1/4$. For the one-sided branch, I use the same data with no treated unit among those with $Z_i = 0$. The fitted complier score is $F(0) = 1/4$, and the function returns $\\Phi(0) = 1/2$. A run of the original function on both datasets reproduces the returned values.", "problem": "The article defines the fitted compliance score with the same distribution function $F$ as the likelihood (p.~497), and it takes $F = \\Phi$ for its own calculations. In the function, a distribution function supplied through \\texttt{link} enters the likelihood, but the returned scores apply \\texttt{pnorm} to the fitted coefficients. With the default \\texttt{link = pnorm} the two coincide for every coefficient vector. With another distribution function, the returned scores can differ from the scores of the fitted model.", "what_survives": "All 15 calls to \\texttt{complier.score} in the replication files omit \\texttt{link} and use the probit, for which the returned scores are those of the fitted model. The canvassing application obtains its scores from a probit regression. The article's formula for the score with a general $F$ is correct as printed. No reported number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E015", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last six lines of Section 3.3. Red boxes mark the sentence 'To characterize uncertainty, we recommend bootstrapping the entire process---from computing the compliance score to weighted IV estimation (Abadie 2002).', which ends with the marker of footnote 7.", "caption": "Section 3.3, p. 498 (PDF p. 7), last paragraph. The marked sentence recommends bootstrapping the whole procedure, the compliance score included. The recommendation is correct; the error is in the package that implements it.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E016.png"}, {"alt": "Listing of eighteen lines of R code. A red box marks the call 'fit <- glm(D ~ W, df, subset = (Z == 1), family = binomial(link=\"probit\"))' on lines 43 to 47, which has no weights argument. A second red box marks '* weights' inside the logarithm in line 91, 'return(-sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights)))'.", "caption": "Package icsw 1.0.0, R/compliance.score.R, lines 43–48 and 81–92, typeset from the source file with line numbers added. The marked call fits the one-sided probit without weights; the marked factor multiplies the likelihood contribution by the weights inside the logarithm.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E016-code.png"}, {"alt": "Two panels, for the strata W = 0 and W = 1, with the bootstrap replicate on the horizontal axis (original fit, then 1 to 4) and the fitted compliance score from 0 to 1 on the vertical axis. Blue circles, the published code, lie at 0.25 in every replicate in the left panel and at 0.75 in the right panel. Orange diamonds, a probit refitted with the replicate weights, have the same score as the circles at the original fit and then move: about 0.33, 0.12, 0.22 and 0.19 on the left, and about 0.99, 0.88, 0.67 and 0.63 on the right.", "caption": "Fitted compliance scores in the original fit and in four bootstrap replicates of the one-sided code of icsw 1.0.0, for the 16-observation example. The published code returns $0.25$ and $0.75$ every time; a probit that uses the replicate weights returns scores that change with the weights. Four replicates show the path through the code and do not estimate a sampling distribution.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E016-bootstrap-scores.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T7", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T7"}, {"id": "P029-T8", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T8"}, {"id": "P029-T23", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T23"}], "headline": "In the R package icsw (version 1.0.0) the observation weights must enter the fit of the compliance score: as published, the one-sided probit omits them and the two-sided objective puts them inside the logarithm, so that the bootstrap does not refit the scores to its weights.", "id": "P029-E016", "location": "Section 3.3, p. 498 (PDF p. 7), last paragraph; R package icsw, version 1.0.0: R/compliance.score.R, lines 43–47 and 91, R/icsw.tsls.R, lines 28–37, and the help page of icsw.tsls", "pdf_page": 7, "printed_page": "498", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P029-T23", "P029-T7", "P029-T8"], "text": {"as_printed": "Article, p.~498: To characterize uncertainty, we recommend bootstrapping the entire process---from computing the compliance score to weighted IV estimation (Abadie 2002).$^{7}$\n\nPackage, \\texttt{R/compliance.score.R}, lines 43--47 (line breaks removed; the call has no \\texttt{weights} argument):\n\n$\\texttt{fit <- }\\hl{\\texttt{glm(D \\textasciitilde{} W, df, subset = (Z == 1), family = binomial(link=\"probit\"))}}$\n\nLine 91:\n\n$\\texttt{return(-sum(log((p.D)\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D)}\\hl{\\texttt{ * weights}}\\texttt{)))}$\n\n\\texttt{R/icsw.tsls.R}, lines 29--32 (line breaks removed; the call continues):\n\n\\texttt{weights.r <- rexp(n.obs, 1) \\# Bayesian bootstrap}\n\n\\texttt{coefs.r <- icsw.tsls.fit(D, X, Y, Z, W, weights = weights * weights.r,}\n\nHelp page of \\texttt{icsw.tsls}: Estimate average treatment effects using two-stage least squares with a binary instrument and binary treatment and weighting with inverse complier scores (probabilities of compliance). Optionally, bootstrap the \\hl{entire} estimation process for the purpose of hypothesis testing and constructing confidence intervals.", "corrected": "In \\texttt{R/compliance.score.R}, lines 43--47:\n\n$\\texttt{fit <- glm(D \\textasciitilde{} W, df, subset = (Z == 1),}\\hl{\\texttt{ weights = weights,}}$\n\n$\\texttt{family = binomial(link=\"probit\"))}$\n\nLine 91:\n\n$\\texttt{return(-sum(}\\hl{\\texttt{weights * }}\\texttt{log(p.D\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D))))}$\n\nLine 89 keeps \\texttt{p.D} in $[10^{-5},1-10^{-5}]$, so that every logarithm is finite and a weight of zero removes the observation from the sum.", "evidence": "The evidence is a derivation for the two-sided objective, an exact counterexample for the one-sided branch, and a run of the package's code. Let $L_i(\\theta)=p_i(\\theta)^{D_i}\\{1-p_i(\\theta)\\}^{1-D_i}$, where the code keeps $p_i(\\theta)$ in $[10^{-5},1-10^{-5}]$. For positive weights $w_i$, the objective on line 91 is\n\\[\n-\\sum_i\\log\\{L_i(\\theta)w_i\\}=-\\sum_i\\log L_i(\\theta)-\\sum_i\\log w_i ,\n\\]\nand the last sum does not depend on $\\theta$. The weighted negative log-likelihood is $-\\sum_i w_i\\log L_i(\\theta)$. If some $w_i=0$, then $\\log\\{L_i(\\theta)w_i\\}=-\\infty$ for every $\\theta$.\n\nThe counterexample has 16 observations, four in each cell of a binary covariate $W$ and the assignment $Z$. No unit assigned to control is treated. Among the assigned units, one of four is treated at $W=0$ and three of four at $W=1$. Each treated unit has weight $9$ and every other unit has weight $1$. A probit with an intercept and $W$ is saturated, so that the fitted score at each value of $W$ is the proportion treated among the assigned units, weighted when the fit uses the weights. The outcome is $Y=D(1+W)$, and, with an intercept and no other covariates in the IV fit, the weighted IV estimate reduces to $\\sum_{Z_i=1}v_iY_i/\\sum_{Z_i=1}v_iD_i$ with $v_i$ the weight divided by the score.\n\\begin{audittable}{lrr}{Exact one-sided example with 16 observations and weights 9 and 1.}\nQuantity & Package 1.0.0 & Weights in the probit \\\\\n\\hline\nCompliance score, $W=0$ & $1/4$ & $9/(9+3)=3/4$ \\\\\nCompliance score, $W=1$ & $3/4$ & $27/(27+1)=27/28$ \\\\\nICSW coefficient on $D$ & $(36+72)/(36+36)=3/2$ & $(12+56)/(12+28)=17/10$ \\\\\n\\end{audittable}\nRunning the unmodified package code on these data, I find scores $(0.25,0.75)$ with and without the weights and a coefficient of $1.5$. With \\texttt{weights = weights} added to the call, the coefficient is $1.7$. The first step also matters for the bootstrap. Starting from unit weights, I change the weight of one untreated assigned unit at $W=0$ from $1$ to $t$. The package's coefficient stays at $3/2$, while refitting the score gives $(t+11)/(t+7)$, with derivative $-1/16$ at $t=1$. In a trace of four bootstrap replicates of the one-sided code on the same $D$, $Z$ and $W$, the scores are identical in the original fit and in every replicate. Each bootstrap coefficient equals the second stage computed with the scores held fixed.\n\nThree points are not established. The size and the direction of the error in the standard errors are unknown. Leaving out the first step removes a variance and a covariance, and the sign of their sum is not determined. In the two-sided branch the weights do enter a regression for starting values, and a numerical optimizer may return different approximate fits from different starts, although the objective has the same minimizers. The default optimizer for the two-sided branch (\\texttt{rgenoud}) was not run, and I do not state what the example on the help page returns.", "problem": "The article recommends that the bootstrap repeat the whole procedure, the fit of the compliance score included, and that recommendation is correct. The package says on its help page that it bootstraps ``the entire estimation process'', and it does so by reweighting. Each replicate draws exponential weights, multiplies them into the observation weights and calls the fitting routine again. The compliance model is not fitted with those weights. In the one-sided branch the probit is fitted by \\texttt{glm} without a \\texttt{weights} argument, and the fitted scores are the same in every replicate. In the two-sided branch the weights multiply the likelihood contribution inside the logarithm. This adds a term to the objective that does not depend on the parameters. The set of minimizers is then the same for every vector of positive weights. There are three consequences. The package's bootstrap standard errors leave out the variation of the first step, contrary to the article and to the help page. Sampling weights or missing-data weights that the user supplies are not used to weight the compliance-model log likelihood, and this can change the estimate itself. A weight of zero makes the two-sided objective infinite at every parameter value. The example on the help page sets the weight to zero for the 9 of its 507 observations with a missing outcome.", "what_survives": "This package defect does not implicate the article's mathematical results or reported estimates. The article's own scripts do not use this package. The one-sided application fits a weighted probit directly and refits it on resampled units. The two-sided application and the bootstrap simulations resample rows and use the helper in the replication files, whose two-sided objective has the weights outside the logarithm. The one-sided branch of that helper has the same misplaced weights, but the article's one-sided application does not use it. With unit weights and no bootstrap, the package's compliance models use ordinary unweighted objectives. Package results with nonconstant weights, and all bootstrap standard errors from the package, are to be recomputed after the repair."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E016", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Listing of R code with line numbers, in two parts. In the first part, lines 111 to 114 call minqa::bobyqa with starting, complier.neg.logLik.fnc and control = list(maxit = num.iter, trace = 0), and line 115 reads if (opt.log$convergence > 0) warning(\"Nelder-Mead did not converge!\"). Red outlines surround maxit = num.iter, trace = 0 on line 113, and opt.log$convergence and Nelder-Mead on line 115. The second part shows line 81 of icsw.R, which calls bobyqa with control=list(iprint=0,maxfun=num.iter), lower=-5, upper=5, without marks.", "caption": "Source code, not a page of the article: package icsw, R/compliance.score.R, lines 110–116, and for comparison line 81 of icsw.R in the replication files. The outlines mark the two control names, the field that the optimizer does not return, and the name of another optimizer in the warning.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E017.png"}], "formal_release": null, "formal_statements": [], "headline": "With genoud = FALSE, the package function compliance.score stops with an error under two-sided noncompliance: it passes control names that the BOBYQA optimizer does not use, and it tests a field that the optimizer does not return.", "id": "P029-E017", "location": "R package icsw, version 1.0.0, file R/compliance.score.R, lines 110–116 (the branch genoud = FALSE of the fit for two-sided noncompliance)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Package \\texttt{icsw}, \\texttt{R/compliance.score.R}, lines 111--115.}\n\\[\n\\begin{aligned}\n&\\texttt{opt.log <- minqa::bobyqa(}\\\\\n&\\quad\\texttt{starting, complier.neg.logLik.fnc,}\\\\\n&\\quad\\texttt{control = list(}\\hl{\\texttt{maxit}}\\texttt{ = num.iter, }\\hl{\\texttt{trace}}\\texttt{ = 0)}\\\\\n&\\quad\\texttt{)}\\\\\n&\\texttt{if (opt.log\\$}\\hl{\\texttt{convergence}}\\texttt{ > 0) warning(\"}\\hl{\\texttt{Nelder-Mead}}\\texttt{ did not converge!\")}\n\\end{aligned}\n\\]", "corrected": "\\emph{Package \\texttt{icsw}, \\texttt{R/compliance.score.R}, lines 111--115.}\n\\[\n\\begin{aligned}\n&\\texttt{opt.log <- minqa::bobyqa(}\\\\\n&\\quad\\texttt{starting, complier.neg.logLik.fnc,}\\\\\n&\\quad\\texttt{control = list(}\\hl{\\texttt{maxfun}}\\texttt{ = num.iter, }\\hl{\\texttt{iprint}}\\texttt{ = 0)}\\\\\n&\\quad\\texttt{)}\\\\\n&\\texttt{if (opt.log\\$}\\hl{\\texttt{ierr}}\\texttt{ > 0) warning(\"}\\hl{\\texttt{BOBYQA}}\\texttt{ did not converge!\")}\n\\end{aligned}\n\\]", "evidence": "The evidence is a derivation from the interface of the optimizer and a run of the package source, under R~4.4.2 with \\texttt{minqa}~1.2.8. The example has 32 units with two-sided noncompliance, eight in each cell of a binary covariate and the assignment, with treatment rates $1/4$ and $3/4$. The unmodified function issues the warning ``unused control arguments ignored'' and stops with ``argument is of length zero''. The function \\texttt{icsw.tsls.fit} with \\texttt{genoud = FALSE} stops with the same error. With the corrections below, the function returns 32 finite scores and \\texttt{ierr} equal to $0$. With \\texttt{num.iter = 1} it returns \\texttt{ierr} equal to $1$ and the corrected warning. The function \\texttt{complier.score} in \\texttt{icsw.R} of the replication files, which the analyses of the paper use, calls \\texttt{bobyqa} with \\texttt{maxfun} and \\texttt{iprint}, has no such test, and returns scores on the same example.", "problem": "The function \\texttt{bobyqa} of the package \\texttt{minqa} takes the controls \\texttt{maxfun} and \\texttt{iprint}, and it returns a list with the components \\texttt{par}, \\texttt{fval}, \\texttt{feval}, \\texttt{ierr} and \\texttt{msg}. The names \\texttt{maxit} and \\texttt{trace} are ignored with a warning, and a value of \\texttt{num.iter} chosen by the user has no effect. The returned list has no component \\texttt{convergence}. The expression \\texttt{opt.log\\$convergence} is then \\texttt{NULL}, the comparison with $0$ has length zero, and \\texttt{if} stops with the error ``argument is of length zero''. No scores are returned. The warning names Nelder--Mead, and the optimizer is BOBYQA. The default \\texttt{genoud = TRUE} and the fit for one-sided noncompliance do not pass through these lines.", "what_survives": "The defect is in a branch of the package that is not the default, and it stops the computation without returning a number. The scripts for the simulations and the applications of the paper use the function in the replication files and not the package. No definition, result or reported number of the paper is affected. The correction restores the branch; it does not show that BOBYQA finds the global maximum of the likelihood."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E017", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of R code with line numbers. Line 8 reads low <- simoutput[6,] < 0.33 and line 10 reads RMSE <- apply(simoutput[7:nrow(simoutput),low],1,function(x) mean((x - simoutput[1,low])^2)^.5). Lines 15 and 17 repeat the pair for med, defined as (simoutput[6,] < 0.50) and (0.33 <= simoutput[6,]), and lines 22 and 24 repeat it for high, defined as simoutput[6,] >= 0.50. Red outlines surround simoutput[7:nrow(simoutput),low] on line 10, simoutput[7:nrow(simoutput),med] on line 17 and simoutput[7:nrow(simoutput),high] on line 24.", "caption": "Source code of the replication files, not a page of the supplement: Sim-Display Alpha.R, lines 8–10, 15–17 and 22–24. The outlines mark the three extractions that lose the matrix shape when a group has one dataset.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E019.png"}], "formal_release": null, "formal_statements": [], "headline": "The script that draws Figures A1 and A2 stops with an error when a compliance group contains exactly one simulated dataset, because the extraction of the group loses its matrix shape; adding drop=FALSE repairs it.", "id": "P029-E019", "location": "Replication files, Sim-Display Alpha.R, lines 10, 17 and 24; the script draws Figures A1 and A2 of the supplement (supplement PDF pp. 3–4)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Replication files, \\texttt{Sim-Display Alpha.R}, lines 8 and 10. Line 10 is broken here after \\texttt{function(x)}. Lines 17 and 24 repeat it with \\texttt{med} and \\texttt{high} in place of \\texttt{low}.}\n\\[\n\\begin{aligned}\n&\\texttt{low <- simoutput[6,] < 0.33}\\\\\n&\\texttt{RMSE <- apply(}\\hl{\\texttt{simoutput[7:nrow(simoutput),low]}}\\texttt{,1,function(x)}\\\\\n&\\qquad\\texttt{mean((x - simoutput[1,low])\\textasciicircum 2)\\textasciicircum .5)}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files, \\texttt{Sim-Display Alpha.R}, line 10, and likewise lines 17 and 24 with \\texttt{med} and \\texttt{high}.}\n\\[\n\\begin{aligned}\n&\\texttt{RMSE <- apply(simoutput[7:nrow(simoutput),low}\\hl{\\texttt{,drop=FALSE}}\\texttt{],1,function(x)}\\\\\n&\\qquad\\texttt{mean((x - simoutput[1,low])\\textasciicircum 2)\\textasciicircum .5)}\n\\end{aligned}\n\\]\nThe script may also skip an empty group and report its RMSE as undefined.", "evidence": "The evidence is an exact counterexample, obtained by evaluating the statements of the script. On a matrix with nine rows and three columns whose compliance row is $(0.2, 0.4, 0.6)$, each group contains one dataset, and all three statements stop with the error above. With \\texttt{drop=FALSE} added, each statement returns an RMSE of $1$ in each of its three rows. On a matrix with two datasets in each group, the original and the repaired statements return the same values.", "problem": "When \\texttt{low} selects exactly one column, R returns a vector in place of a matrix with one column, and \\texttt{apply} stops with the error ``dim(X) must have a positive length''. The same holds for \\texttt{med} and \\texttt{high}. No group with a single dataset is shown to occur in the published runs of 2,500 simulated datasets. With two or more datasets in a group the statements work, and the RMSE formulas are right.", "what_survives": "The RMSE formulas and all calculations for groups with two or more datasets are unaffected, and the published figures are not shown to be affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E019", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the introduction. The text reads: Several influential papers, including Deaton (2009) and Heckman and Urzua (2010), have expressed concern about the interpretation of the LATE. Both papers argue that the LATE is not usually the parameter of interest, as we are unlikely to learn much about the process at work by estimating the LATE (Deaton 2009, 430) and the LATE is often very difficult to interpret as an answer to an interesting economic question (Heckman and Urzua 2010, 35). The crop ends inside the next sentence. One red outline surrounds the word process, and another surrounds the citation (Deaton 2009, 430).", "caption": "Section 1, p. 492 (PDF p. 1). The outlines mark the word “process” in the quotation and the citation with the page number 430.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020.png"}, {"alt": "Crop of thirteen lines of the reference list, from the entry for Angrist and Fernandez-Val (2013) to the entry for Deaton (2009). In the first entry the editors are printed as Manuel Arellano Daron Acemoglu, and Eddie Deke, with the pages 401-36. Red outlines surround the words Arellano Daron and the words Deke, 401-36. The entries for Angrist and Pischke, Arceneaux and Nickerson (2009 and 2010) and Aronow and Carnegie follow without marks. In the last entry, Deaton, Angus. 2009. Instruments of development, Proceedings of the British Academy, 2008 Lectures, a red outline surrounds 162:123-60.", "caption": "Reference list, p. 505 (PDF p. 14). The outlines mark the two editors' names that run together, the misspelled name with the page range of the chapter, and the page range of the edition of Deaton’s essay that the list gives.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020-b.png"}, {"alt": "Crop of the two lines of one reference entry. The text reads: Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til Strmer, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. American Journal of Epidemiology 173(7):761-7. A red outline surrounds the words Til Strmer.", "caption": "Reference list, p. 505 (PDF p. 14). The outline marks the misspelled name in the entry for Funk et al.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020-c.png"}, {"alt": "Crop of sixteen lines at the top of the last page of the reference list, from the entry for Joffe and Brensinger (2003) to the entry for Verba, Schlozman and Brady (1995). Four red outlines surround: 22(1) in Statistics in Medicine 22(1):1285-303; Robert C. in Putnam, Robert C. 2000; the word renewal in Bowling alone: The collapse and renewal of American Community; and the title Voluntarism in American Politics. The entries for Joffe, Ten Have and Brensinger, for Rosenstone and Hansen, for Roy, Hogan and Marcus and the two entries for Rubin carry no marks.", "caption": "Reference list, p. 506 (PDF p. 15). The outlines mark the issue number in the entry for Joffe and Brensinger, the middle initial and the word “renewal” in the entry for Putnam, and the shortened title in the entry for Verba, Schlozman and Brady.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020-d.png"}], "formal_release": null, "formal_statements": [], "headline": "The quotation from Deaton reads “processes” in the original and is on p. 130 of the edition listed, and five entries of the reference list have a wrong name, title, issue or page range.", "id": "P029-E020", "location": "Section 1, p. 492 (PDF p. 1), quotation from Deaton; reference list, pp. 505–506 (PDF pp. 14–15), entries for Angrist and Fernandez-Val, Deaton, Funk et al., Joffe and Brensinger, Putnam, and Verba, Schlozman and Brady", "pdf_page": 1, "printed_page": "492", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Section~1, p.~492.} Both papers argue that the LATE is not usually the parameter of interest, as ``we are unlikely to learn much about the \\hl{process} at work'' by estimating the LATE (Deaton 2009, \\hl{430}) [\\ldots]\n\n\\emph{Reference list, pp.~505--506.}\n\\begin{itemize}\n\\item Angrist, Joshua D., and Ivan Fernandez-Val. 2013. ExtrapoLATE-ing: External validity and overidentification in the LATE framework. In \\emph{Advances in economics and econometrics: Tenth world congress}, Vol.~3, eds. Manuel \\hl{Arellano Daron} Acemoglu, and Eddie \\hl{Deke}, 401--\\hl{36}. Cambridge: Cambridge University Press.\n\\item Deaton, Angus. 2009. Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development. \\emph{Proceedings of the British Academy, 2008 Lectures} 162:123--60.\n\\item Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til \\hl{Strmer}, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. \\emph{American Journal of Epidemiology} 173(7):761--7.\n\\item Joffe, Marshall M., and Colleen Brensinger. 2003. Weighting in instrumental variables and G-estimation. \\emph{Statistics in Medicine} 22(\\hl{1}):1285--303.\n\\item Putnam, Robert \\hl{C.} 2000. \\emph{Bowling alone: The collapse and \\hl{renewal} of American Community}. New York: Simon and Schuster.\n\\item Verba, Sidney, Kay Lehman Schlozman, and Henry E. Brady. 1995. \\emph{\\hl{Voluntarism in American Politics}}. Cambridge, MA: Harvard University Press.\n\\end{itemize}", "corrected": "\\emph{Section~1, p.~492.} Both papers argue that the LATE is not usually the parameter of interest, as ``we are unlikely to learn much about the \\hl{processes} at work'' by estimating the LATE (Deaton 2009, \\hl{130}) [\\ldots]\n\nAn alternative is to cite and list the 2010 version of the essay, \\emph{Journal of Economic Literature} 48(2):424--55, and to keep the page 430.\n\n\\emph{Reference list.}\n\\begin{itemize}\n\\item Angrist, Joshua D., and Ivan Fernandez-Val. 2013. ExtrapoLATE-ing: External validity and overidentification in the LATE framework. In \\emph{Advances in economics and econometrics: Tenth world congress}, Vol.~3, eds. Manuel \\hl{Arellano, Daron} Acemoglu, and Eddie \\hl{Dekel}, 401--\\hl{34}. Cambridge: Cambridge University Press.\n\\item Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til \\hl{Stürmer}, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. \\emph{American Journal of Epidemiology} 173(7):761--7.\n\\item Joffe, Marshall M., and Colleen Brensinger. 2003. Weighting in instrumental variables and G-estimation. \\emph{Statistics in Medicine} 22(\\hl{8}):1285--303.\n\\item Putnam, Robert \\hl{D.} 2000. \\emph{Bowling alone: The collapse and \\hl{revival} of American Community}. New York: Simon and Schuster.\n\\item Verba, Sidney, Kay Lehman Schlozman, and Henry E. Brady. 1995. \\emph{\\hl{Voice and Equality: Civic} Voluntarism in American Politics}. Cambridge, MA: Harvard University Press.\n\\end{itemize}", "evidence": "The evidence is a check of the page against the cited works and the records of their publishers. In both editions of Deaton's essay the sentence reads ``We are unlikely to learn much about the processes at work''. It is on p.~430 of the 2010 version (\\emph{Journal of Economic Literature} 48(2), 424--455) and on p.~130 of the 2009 version in the \\emph{Proceedings of the British Academy}. The publisher's record of the chapter by Angrist and Fernandez-Val names the editors Daron Acemoglu, Manuel Arellano and Eddie Dekel, with the pages 401--434. In the publisher's record of the article by Joffe and Brensinger, the issue is Volume~22, Issue~8. Bibliographic records give the names Til Stürmer and Robert D. Putnam, and the titles \\emph{Bowling Alone: The Collapse and Revival of American Community} and \\emph{Voice and Equality: Civic Voluntarism in American Politics}.\n\nOne point is not settled. The same page of the paper cites ``Imbens (2009, 403)'', and the reference list has an NBER working paper for this citation. Page 403 lies in the page range 399--423 of the version of that comment in the \\emph{Journal of Economic Literature} in 2010. I could not confirm the page of the quoted words in either edition.", "problem": "The quotation changes a word of the original, and its page number does not belong to the edition in the reference list, whose pages are 123 to 160. Page 430 is the page of the passage in the version of the essay that appeared in the \\emph{Journal of Economic Literature} in 2010. In the reference list, two editors' names run together and a third is misspelled, one chapter ends at p.~434 and not at p.~436, an author's name has lost a letter, an issue number and a middle initial are wrong, a subtitle has ``renewal'' for ``revival'', and one title has lost its first words. Each cited work can still be identified.", "what_survives": "The argument of the paper does not depend on any of these details, and the sense of the quotation is unchanged. No correction is proposed for the citation of Imbens until the edition is checked."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E020", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of three rows of Table 1, each with a coefficient and an interval in two columns. Age: 0.2 with interval (0.2, 0.2), and 0.2 with interval (0.1, 0.2). Age missing: 12.7 with interval (-16.5, 44.1), and 17.7 with interval (-15.9, 50.3). Democrat: 2.6 with interval (0.6, 4.7), and 2.6 with interval (0.7, 04.8). Red outlines surround the label Age missing and the two coefficients 12.7 and 17.7.", "caption": "Table 1, rows “Age”, “Age missing” and “Democrat”, p. 501 (PDF p. 10). The two columns of numbers are LATE (2SLS), (1), and ATE (ICSW), (2). The outlines mark the label “Age missing” and its two coefficients, which the script as deposited does not reproduce.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E021.png"}, {"alt": "Image of six lines of R code in a monospaced face, under the heading greengerbernickerson.R, lines 33-36 and 45 (replication files). Line 33: age[age>101] <- mean(age[age <= 101],na.rm=TRUE). Line 34: age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE). Line 35: ageM <- rep(0,N). Line 36: ageM[age==mean(age[age <= 101],na.rm=TRUE)] <- 1. Line 45, after an ellipsis: covs <- cbind(makeind(data.frame(factor(city))),famsize,white,black,voted00,voted99a,primary,age,ageM,dem,rep,ind)[,2:17]. A red outline surrounds the subscript of line 36, from the opening bracket before age==mean to the closing bracket.", "caption": "Replication script greengerbernickerson.R, lines 33–36 and 45, typeset from the replication files (a script has no published page). The outline marks the comparison in line 36.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E021-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication script rebuilds the age-missing indicator by testing whether an imputed age equals a recomputed mean, which on a current system flags 1 of the 2,488 records with a missing or impossible age and gives “Age missing” coefficients near 97 where Table 1 prints 12.7 and 17.7; the printed point estimates are reproduced when the indicator is set before the imputation.", "id": "P029-E021", "location": "Replication script greengerbernickerson.R, lines 33–36 and 45; the paper’s counterparts are Table 1, row “Age missing”, p. 501 (PDF p. 10), and supplement Table A7, column “Age M.” and row “Age Miss.”, p. A11 (supplement PDF p. 11)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Script, lines 33--36 and 45 (line 45 is one line of code, broken here).}\n\\[\n\\begin{aligned}\n&\\texttt{age[age>101] <- mean(age[age <= 101],na.rm=TRUE)}\\\\\n&\\texttt{age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)}\\\\\n&\\texttt{ageM <- rep(0,N)}\\\\\n&\\texttt{ageM[}\\hl{\\texttt{age==mean(age[age <= 101],na.rm=TRUE)}}\\texttt{] <- 1}\\\\\n&\\texttt{covs <- cbind(makeind(data.frame(factor(city))),famsize,white,black,voted00,}\\\\\n&\\qquad\\texttt{voted99a,primary,age,ageM,dem,rep,ind)[,2:17]}\n\\end{aligned}\n\\]\n\n\\emph{Table~1, p.~501.}\n\\begin{audittable}{lll}{Table~1, row ``Age missing'', with the bootstrap 95\\% confidence intervals.}\n & LATE (2SLS), (1) & ATE (ICSW), (2) \\\\\nAge missing & 12.7 $(-16.5, 44.1)$ & 17.7 $(-15.9, 50.3)$ \\\\\n\\end{audittable}\n\\emph{Supplement Table~A7, p.~A11.} Comp. Score, Age M.: $-0.55$.", "corrected": "In \\texttt{greengerbernickerson.R}, the indicator should be set before age is imputed and kept in \\texttt{covs} (line 45), and lines 35--36 should be deleted:\n\\[\n\\begin{aligned}\n&\\hl{\\texttt{ageM <- as.numeric(is.na(age) | age > 101)}}\\\\\n&\\texttt{age[age>101] <- mean(age[age <= 101],na.rm=TRUE)}\\\\\n&\\texttt{age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)}\n\\end{aligned}\n\\]\nOn R 4.4.2 for arm64, the comparison in the script as deposited flags 1 of the 2,488 records with a missing or impossible age. The indicator set before the imputation flags all of them, and it reproduces the printed point estimates of Table~1 and the corresponding entries of Table~A7.", "evidence": "The evidence is a recalculation from the replication data, in which I compare the exact rational representations of the three floating-point means. The data hold 18,933 records, of which 2,487 have a missing age and one has an age above 101. The 16,445 valid ages have mean $742242/16445$. I evaluated the script's expressions for the compliance score and for the two estimators twice, once with the indicator of lines 35--36 and once with the indicator set to 1 for the 2,488 records before the imputation.\n\\begin{audittable}{lrrr}{Results from the replication data under the two constructions of the age-missing indicator. Coefficients are in percentage points.}\nQuantity & Script as deposited & Indicator set before imputation & Printed \\\\\nRecords flagged & 1 & 2,488 & \\\\\n2SLS, Age missing & 97.35 & 12.65 & 12.7 \\\\\nICSW, Age missing & 96.60 & 17.71 & 17.7 \\\\\n2SLS, Campaign contact & 5.64 & 5.61 & 5.6 \\\\\nICSW, Campaign contact & 6.71 & 6.67 & 6.7 \\\\\nCorrelation of score with indicator & $-0.01$ & $-0.55$ & $-0.55$ \\\\\n\\end{audittable}\nWith the indicator set before the imputation, I find that all 24 point estimates of Table~1 agree with the printed values to one decimal. The 17 entries of the column ``Age M.'' and the 12 entries of the row ``Age Miss.'' of Table~A7 agree to two decimals. With the script as deposited, the ICSW coefficient on ``Voted in 1999 general election'' also changes, from 27.2 to 27.3.\n\nFor the bootstrap, I took one draw that the resampling of line 94 can produce, in which the flagged record is replaced by another record of its stratum. With the script as deposited, the coefficient on the indicator is missing in both models and both quantile calls stop with an error. With the indicator set before the imputation, the same draw runs. I did not recompute the 2,500-draw confidence intervals, and I do not know how or on what system the published run was made.", "problem": "The note to Table~1 lists a dummy variable for missing data on age among the covariates. The script first replaces the ages above 101, and then the missing ages, by the mean of the valid ages (lines 33 and 34). Only afterwards does it set the indicator to 1 for every record whose age equals that mean, which line 36 computes a third time. In exact arithmetic the three means agree, since adding copies of the mean to a sample does not change its mean. In floating-point arithmetic they need not agree to the last bit, and line 36 tests exact equality.\n\nOn R 4.4.2 for arm64 under macOS, the first and third means are equal and the second is smaller by $3/2^{47}$. The one age above 101 (the value 2001) was replaced by the first mean and passes the test. The 2,487 missing ages were replaced by the second mean and fail it. The indicator is then 1 for one record, where it should be 1 for 2,488. With this indicator the script returns ``Age missing'' coefficients of 97.3 and 96.6 percentage points, against the printed 12.7 and 17.7. The correlation between the compliance score and the indicator is $-0.01$, against the $-0.55$ of Table~A7. The bootstrap of the script can also stop. A draw that omits the one flagged record leaves a column of zeros, the coefficient on that column is missing, and the quantile calls of the script do not remove missing values.\n\nThe test is unreliable in exact arithmetic too, in the other direction. For the ages $20$, $30$, $40$ and one missing value, the mean is $30$ at every stage, and the test flags the observed age $30$ along with the imputed one. I checked one system, and the number of flagged records may differ on another.", "what_survives": "Table~1 and Table~A7 are not shown to be wrong: the indicator set before the imputation reproduces every printed point estimate of Table~1 and the entries of Table~A7 that involve the indicator. Under both constructions the effects of contact round to 5.6 and 6.7 percentage points, and the ATE is 19\\% larger than the LATE. In this script the compliance-score fit, the adjusted coefficients, the correlation matrix, the bootstrap and the permutation diagnostic all use the indicator. The confidence intervals and the diagnostic were not recomputed. The theory, the simulations and the application to Albertson and Lawrence (2009) do not use this script."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E021", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the four lines of footnote 1. The text reads: Other examples include: Hidalgo et al. (2010) show that the effects of economic shocks on conflict are larger in regions with high levels of preexisting inequality; Altonji and Dunn (1996) demonstrate that the returns to education are larger for students in high-quality schools; Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are larger for high-propensity voters. A red outline surrounds the last clause, from Arceneaux and Nickerson (2009) to the end.", "caption": "Footnote 1, p. 494 (PDF p. 3). The outline marks the clause on Arceneaux and Nickerson (2009).", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "Arceneaux and Nickerson (2009) find larger effects of canvassing for high-propensity voters in low-salience elections only; in high-salience elections they find the opposite.", "id": "P029-N001", "location": "Footnote 1, p. 494 (PDF p. 3), last clause", "pdf_page": 3, "printed_page": "494", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Other examples include: Hidalgo et al. (2010) show that the effects of economic shocks on conflict are larger in regions with high levels of preexisting inequality; Altonji and Dunn (1996) demonstrate that the returns to education are larger for students in high-quality schools; Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are \\hl{larger for high-propensity voters}.", "corrected": "[\\ldots] Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are larger for high-propensity voters \\hl{in low-salience elections}.\n\nThe footnote may add that the pattern is reversed in high-salience elections.", "evidence": "The evidence is a check of the footnote against the cited article (\\emph{American Journal of Political Science} 53(1):1--16). In its discussion of Figure~4, the article has the phrase ``boost turnout among high-propensity voters but not low-propensity voters in low-salience elections'' on p.~10, and it continues on p.~11: ``The opposite takes place in high-salience elections.'' Its Table~3 (p.~12) reports effects of 3.3\\%, 11.7\\% and 12.3\\% for low-, medium- and high-propensity voters in low-salience elections, and of 12.4\\%, 11.6\\% and 3.2\\% in high-salience elections.", "problem": "The clause reads as a general statement about the direction of the heterogeneity. In the cited article the direction depends on the salience of the election. The larger effect for high-propensity voters is a finding of that article for low-salience elections, and the footnote omits that the pattern reverses in high-salience elections.", "what_survives": "The footnote lists examples of heterogeneous treatment effects, and the cited article is such an example. Nothing in the paper depends on the direction of the heterogeneity. The corrected clause is narrower than the printed one."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N001", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text. The text reads: We begin by estimating compliance scores for the sample. As recommended above, we use all observed pretreatment covariates, which include the covariates listed in Table 1 and dummy variables for the six cities in the sample. Although the set of available covariates may or may not satisfy the ignorability assumption, we will still recover an estimate for a population that has a covariate profile that matches that of the full population. We see that compliers are more likely to have voted previously, be members of a major party, and identify as white (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates). We now apply ICSW; the crop ends in the middle of this sentence. A red outline surrounds the words be members of a major party, and identify as white.", "caption": "Section 4.1, last paragraph of p. 501 (PDF p. 10). The outline marks the description of the compliers by party and race.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N004.png"}, {"alt": "Crop of the top of a correlation table with the column headings C. Score, Fam. Size, White, Black, Voted 00, Voted 99, Voted Pri., Age, Age M., Dem, Rep and Ind. Row Comp. Score: 1.00, 0.37, 0.61, 0.32, 0.40, 0.43, 0.25, 0.23, -0.55, 0.38, 0.31, -0.47. Row Columbus: -0.55, -0.17, -0.20, -0.07, -0.15, -0.05, -0.19, -0.00, 1.00, -0.11, -0.03, 0.79. Row Detroit: 0.05, 0.03, -0.30, -0.11, -0.00, -0.24, 0.23, 0.16, -0.23, -0.28, -0.19, -0.18. Row Minneapolis: -0.45, -0.24, -0.21, -0.08, -0.06, -0.14, -0.01, -0.14, -0.16, -0.20, -0.14, -0.13. Row Raleigh: 0.75, 0.26, 0.89, 0.33, 0.13, 0.17, -0.00, 0.02, -0.22, 0.46, 0.43, -0.18. Row St. Paul: 0.07, 0.15, -0.18, -0.07, 0.16, 0.30, 0.06, -0.08, -0.14, -0.17, -0.12, -0.11. Red outlines surround three entries of the row Comp. Score: 0.61 under White, 0.38 under Dem and 0.31 under Rep.", "caption": "Supplement Table A7, p. A11 (supplement PDF p. 11): the column headings and the first six rows. The table is printed sideways and is turned upright here. The outlines mark the correlations of the compliance score with White, Dem and Rep. In the row “Raleigh”, the entry under “C. Score” is 0.75 and the entry under “White” is 0.89.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N004-A7.png"}, {"alt": "Figure with two panels of horizontal bars, titled: The White indicator is 1 only in Raleigh, the city with the highest scores; within Raleigh, voters recorded as White have the lowest mean score. Upper panel, mean estimated compliance score. Five grey bars are cities where race is not recorded and the White indicator is 0 for every voter: Columbus (n = 2,478) 0.147, Minneapolis (n = 2,827) 0.185, Bridgeport (n = 1,806) 0.279, Detroit (n = 4,954) 0.309, St. Paul (n = 2,208) 0.320. One orange bar is Raleigh voters recorded as White, indicator 1 (n = 3,884): 0.432. Two blue bars are Raleigh voters not recorded as White, indicator 0: recorded Black (n = 646) 0.487, and other or unknown (n = 130) 0.529. Lower panel, correlation of the estimated compliance score with the White indicator: a solid dark bar for all six cities, the published value of Table A7, at plus 0.61, and a hatched bar for the correlation within Raleigh from this check, at minus 0.41.", "caption": "Mean estimated compliance score by city and, within Raleigh, by recorded race (upper panel), and the correlation between the score and the White indicator over the six cities, as in Table A7, and within Raleigh (lower panel). The values are from the recalculation with the replication data.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N004-scores-by-group.png"}], "formal_release": null, "formal_statements": [{"id": "P029-T9", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T9"}, {"id": "P029-T10", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T10"}], "headline": "The data do not support the statement that compliers in the canvassing study are more likely to be members of a major party and to identify as white: the race and party indicators are zero wherever the label is not recorded, race is recorded only in Raleigh, and within Raleigh the voters recorded as White have lower estimated compliance scores than the others.", "id": "P029-N004", "location": "Section 4.1, last paragraph of p. 501 (PDF p. 10), sentence beginning “We see that compliers”; supplement Table A7, p. A11 (supplement PDF p. 11)", "pdf_page": 10, "printed_page": "501", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P029-T10", "P029-T9"], "text": {"as_printed": "We see that compliers are more likely to have voted previously, \\hl{be members of a major party, and identify as white} (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates).\n\n\\begin{audittable}{lrrrrrrrrr}{Supplement Table~A7, rows ``Comp. Score'' and ``Raleigh'', nine of the twelve columns.}\n & C. Score & White & Black & Voted 00 & Voted 99 & Voted Pri. & Dem & Rep & Ind \\\\\nComp. Score & 1.00 & 0.61 & 0.32 & 0.40 & 0.43 & 0.25 & 0.38 & 0.31 & $-0.47$ \\\\\nRaleigh & 0.75 & 0.89 & 0.33 & 0.13 & 0.17 & $-0.00$ & 0.46 & 0.43 & $-0.18$ \\\\\n\\end{audittable}", "corrected": "\\emph{Section~4.1, p.~501.} Estimated compliance scores are positively associated with prior voting. The pooled correlations with \\hl{recorded} White, \\hl{Dem and Rep} indicators \\hl{also reflect differences between cities in contact rates and covariate availability}. Within Raleigh, the only city supplying race labels, voters recorded as White have \\hl{lower} mean estimated compliance scores than the remaining records (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates).\n\n\\emph{Note to add to supplement Table~A7.} Blank race and party labels are coded zero in the respective indicators; race is unavailable outside Raleigh and party is unavailable throughout Detroit, Minneapolis, and St.~Paul, with additional blank party records in Columbus. Age, primary-vote history, and 1999-vote history are unavailable in particular cities and are imputed. The correlations in the table do not change.", "evidence": "The evidence is a recalculation from the replication data and an exact counterexample to the inference from recorded labels. The recalculation fits the compliance model of the script. It reproduces the correlations of the compliance score in Table~A7 and the twelve point estimates in each column of Table~1 at the printed precision, with the age-missing indicator set before the imputation (see \\fid{P029-E021}). The first table lists the blank labels and the mean score in each city.\n\\begin{audittable}{lrrrr}{Voters, blank race and party labels, and mean estimated compliance score, by city.}\nCity & Voters & Race blank & Party blank & Mean score \\\\\nBridgeport & 1,806 & 1,806 & 0 & 0.279 \\\\\nColumbus & 2,478 & 2,478 & 489 & 0.147 \\\\\nDetroit & 4,954 & 4,954 & 4,954 & 0.309 \\\\\nMinneapolis & 2,827 & 2,827 & 2,827 & 0.185 \\\\\nRaleigh & 4,660 & 0 & 0 & 0.442 \\\\\nSt.~Paul & 2,208 & 2,208 & 2,208 & 0.320 \\\\\n\\end{audittable}\nIn Raleigh, 21 of the 4,660 race labels are unknown or unspecified. The covariance between the score $s$ and an indicator $A$ splits exactly into a part within cities and a part between city means,\n\\[\n\\operatorname{Cov}(s, A) = \\mathbb{E}\\bigl[(s - s_G)(A - A_G)\\bigr] + \\mathbb{E}\\bigl[(s_G - \\bar s)(A_G - \\bar A)\\bigr] ,\n\\]\nwhere $s_G$ and $A_G$ are the means in the voter's city and $\\bar s$ and $\\bar A$ are the overall means.\n\\begin{audittable}{lrrr}{The pooled covariance of the estimated compliance score with each indicator, and its two parts.}\nIndicator & Pooled & Between cities & Within cities \\\\\nWhite & $0.0269$ & $0.0290$ & $-0.0021$ \\\\\nDem & $0.0158$ & $0.0145$ & $0.0013$ \\\\\nRep & $0.0099$ & $0.0093$ & $0.0007$ \\\\\nInd & $-0.0142$ & $-0.0133$ & $-0.0009$ \\\\\n\\end{audittable}\nFor White the part within cities is negative, and all of it comes from Raleigh.\n\nFor the counterexample, I complete the unobserved quantities in two ways that agree with every recorded label and every observed variable. In both, a voter assigned to canvassing is a complier if contacted, and a control voter, whose status is not observed, is not a complier. I assign the 21 unknown or unspecified Raleigh race labels as not White in both completions. There are 2,551 compliers. In the first completion every voter with a blank race label is not White. Then 3,884 voters are White, 602 of them are compliers, and the compliance rate of the White voters exceeds that of the others by $744791/29225158 \\approx 0.025$. In the second completion every voter with a blank race label is White. Then 18,157 voters are White, 2,391 of them are compliers, and the difference is $-131213/1761229 \\approx -0.075$. With the estimated scores held fixed, the correlation between the score and race is $0.61$ in the first completion and $-0.37$ in the second. For party, assigning every blank label as outside the two major parties gives a positive association; assigning blank labels as major-party only for noncompliers gives a negative association, with recorded LIB affiliations outside the two major parties in both cases. Both completions agree with the data. This implies that the data do not determine the sign, and they do not show which completion is closer to the truth.", "problem": "The sentence reads a description of the compliers off the correlations in Table~A7 between the estimated compliance score and the covariates: 0.61 for White, 0.38 for Dem and 0.31 for Rep. These numbers are right. They do not measure race or party membership in the six cities. The replication script sets each indicator to 1 when the label of the voter matches and to 0 otherwise, and a blank label counts as 0. Race is blank for all 14,273 voters outside Raleigh. The White indicator is then 1 only for the 3,884 Raleigh voters with the label W, and Raleigh has the highest mean estimated score of the six cities, 0.44 against 0.15 to 0.32. The pooled correlation mostly separates Raleigh from the other cities. Table~A7 itself shows that the Raleigh indicator has correlation 0.89 with White and 0.75 with the score, and that the Black indicator is also positively correlated with the score (0.32).\n\nWithin Raleigh, the one city where race is recorded, the correlation between the score and the White indicator is $-0.41$. The mean estimated score is 0.432 for the voters recorded as White, 0.487 for those recorded as Black and 0.529 for the 130 voters with another or an unknown label. Among the Raleigh voters assigned to canvassing, 602 of the 1,370 recorded as White were contacted (43.9\\%), against 160 of the 315 others (50.8\\%).\n\nParty is blank for every voter in Detroit, Minneapolis and St.~Paul and for 489 voters in Columbus. More than 90\\% of the pooled covariance between the score and each party indicator is covariance between city means. For party the sign does not reverse within Raleigh, where the correlation of the score with the combined Dem/Rep indicator (which includes the code LIB in Dem) is 0.18, against 0.53 in the pooled data.\n\nI conclude that the correlations cited do not support the description of the compliers by race and party. I do not conclude that the description is false. The association between race or party and compliance over the six cities depends on the labels that were not recorded, and these data cannot determine it. The part of the sentence about previous voting is supported.", "what_survives": "The correlations of Table~A7 are numerically right. The statement that compliers are more likely to have voted previously stands: the score is positively correlated with turnout in 2000 and in 1999 in the pooled data and within cities. The sentence is not an input to any calculation. A change of wording leaves the point estimates of Table~1, the comparison of 5.6 with 6.7 percentage points and the bootstrap as they are. I did not recompute the intervals. The method, the simulations and the application to Albertson and Lawrence (2009) do not depend on the sentence. What changes is the description of the compliers, which must be limited to labels that were recorded."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N004", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of footnote 6, seven lines. One red box marks 'but Winsorizing may sometimes increase finite sample bias.' A second red box, on the next line, marks 'because c tends to 0 as n tends to infinity.'", "caption": "Footnote 6, p. 498 (PDF p. 7). The upper mark is the article’s caution that Winsorizing can add finite-sample bias; the lower mark is the false limit.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N006.png"}, {"alt": "Crop of the head of Table 2, '2SLS and ICSW estimates for Albertson and Lawrence (2009)', with columns Knowledge LATE (2SLS), Knowledge ATE (ICSW), Opinion LATE (2SLS) and Opinion ATE (ICSW). In the row 'Watching debate' the entries are 0.28 (-0.04, 0.59), 0.40 (0.01, 1.26), -0.07 (-0.25, 0.13) and -0.05 (-0.31, 0.24). A red box marks 0.40 (0.01, 1.26).", "caption": "Table 2, p. 503 (PDF p. 12), first rows. The marked entry is the ICSW estimate of the effect of watching the debate on knowledge, with its bootstrap interval.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N006-table2.png"}], "formal_release": null, "formal_statements": [], "headline": "The knowledge estimate of Table 2 depends on the Winsorization constant, and I reproduce the printed point estimates; the error is the statement of footnote 6 that $c\\to0$, the same error that P029-E006 corrects.", "id": "P029-N006", "location": "Footnote 6, p. 498 (PDF p. 7); Table 2, row “Watching debate”, column (2), and the paragraph that discusses it, pp. 502–503 (PDF pp. 11–12)", "pdf_page": 7, "printed_page": "498", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Footnote 6: Winsorizing is performed because very low probabilities can introduce instability in the estimates resulting from IPW (Elliott 2009), but Winsorizing may sometimes increase finite sample bias. Asymptotically, assuming that $\\alpha$ remains fixed, the Winsorization has no impact on estimation, \\hl{because $c\\to0$ as $n\\to\\infty$}.\n\nTable 2, row ``Watching debate'': $0.28$ $(-0.04, 0.59)$ in column (1), Knowledge, LATE (2SLS); $0.40$ $(0.01, 1.26)$ in column (2), Knowledge, ATE (ICSW).\n\nText: Although Albertson and Lawrence (2009) find that compliers are 0.28 points more informed after viewing the program, we find that viewers in the entire subject population are 0.40 points more informed after viewing. We therefore estimate that the ATE is approximately 45\\% larger than the LATE.", "corrected": "Footnote 6, in place of the sentence beginning ``Asymptotically'': \\hl{For fixed $\\alpha > 0$, the quantile level $1/n^{\\alpha}$ tends to zero, so the fraction of scores changed vanishes. Under Assumption 3 and regularity conditions ensuring bounded estimated inverse scores with probability tending to one, integrable outcomes and a nonzero limiting weighted first stage, Winsorization does not change the probability limit of the displayed ICSW estimator.}\n\n\\fid{P029-E006} gives the same correction with the regularity conditions written out. Table 2 and the comparison of $45\\%$ are unchanged.", "evidence": "The evidence is a recalculation from the replication data and an exact counterexample to the limit. The recalculation fits the scores once and varies $\\alpha$. It reproduces all 40 point estimates of Table 2 to two decimals. The excess of the ICSW estimate over the 2SLS estimate is $44.9\\%$ with unrounded coefficients. The bootstrap intervals were not recomputed.\n\\begin{audittable}{rrrrr}{Knowledge estimate for the debate application as $\\alpha$ varies ($n=507$; scores fitted once).}\n$\\alpha$ & Cut-off $c$ & Scores raised & Estimate & Excess over 2SLS \\\\\n\\hline\n$0$ & $0.717$ & $506$ & $0.276$ & $0.0\\%$ \\\\\n$0.275$ & $0.290$ & $92$ & $0.400$ & $44.9\\%$ \\\\\n$0.5$ & $0.162$ & $23$ & $0.505$ & $83.2\\%$ \\\\\n$1$ & $0.065$ & $1$ & $0.612$ & $122.0\\%$ \\\\\n$\\infty$ & $0.063$ & $0$ & $0.612$ & $121.9\\%$ \\\\\n\\end{audittable}\nFor the limit, I consider one covariate profile and one-sided noncompliance with $D_0=0$ and $D_1=C$, where $\\Pr(C=1)=1/2$, outcomes $Y_0=0$ and $Y_1=1$, and $Z\\sim\\mathrm{Bernoulli}(1/2)$ independent of $C$. The article's assumptions hold, with $\\epsilon=1/4$ in Assumption 3. Every true score is $1/2$, and $c=1/2$ for every $n$ with the exact scores. The fitted score is the proportion treated among the assigned units, which converges almost surely to $1/2$, and $c$ converges to $1/2$ with it. No score is changed, and the ICSW ratio equals the ATE of $1$ whenever its empirical first stage is nonzero.", "problem": "The point raised is that the knowledge estimate depends on the constant $\\alpha$ and that the article does not report this dependence. With $n=507$ the rule in footnote 6 raises the 92 lowest scores, $18.1\\%$ of the sample, to $0.290$. The estimate is $0.40$ at $\\alpha=0.275$ and $0.61$ without Winsorization, and it equals the 2SLS estimate at $\\alpha=0$. This dependence is not an error in Table 2. The estimates are computed by the rule that the article states, and the footnote says that Winsorizing can add finite-sample bias. The article does not claim that the estimate is free of the choice of $\\alpha$. The error is in the same footnote: the cut-off $c$ does not tend to zero. This is the statement corrected in \\fid{P029-E006}.", "what_survives": "The Winsorization rule and its implementation, the 40 point estimates of Table 2 and the comparison of $45\\%$ stand. The conclusion that Winsorization does not change the limit of the estimator holds under positivity and regularity conditions. The true ATE, the size of the bias from Winsorization and the best $\\alpha$ for these data are not established."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N006", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of footnote 7, five lines. Red boxes mark the sentence 'In our own simulation studies, assuming alpha = 0.275, we find that the bootstrap has conservative coverage for ICSW in small samples, and proper coverage in larger samples.'", "caption": "Footnote 7, p. 498 (PDF p. 7). The marked sentence summarizes the bootstrap coverage found in the supplement without naming the quantity covered.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N009.png"}, {"alt": "Crop of the section heading and six lines of text. A red box marks 'We present tables of RMSE vs' at the end of the first line, which continues 'n in 500, 1000, 2500, 5000, 10000, 25000 under the 5 different models, including misspecification.'", "caption": "Supplement, p. A5 (PDF p. 5), section “(2) Properties of ICSW with and without Misspecification”. The marked words introduce the RMSE tables; the target of the RMSE is not stated.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N009-A5.png"}, {"alt": "Crop of the section heading, three lines of text and Table A6. A red box marks 'In Table A6, we present percent coverage with the bootstrap'. Table A6 has rows n = 500, 2500 and 5000, with 90 percent interval coverage 0.94, 0.91 and 0.92 and 95 percent interval coverage 0.98, 0.94 and 0.95.", "caption": "Supplement, p. A9 (PDF p. 9), section “(3) Bootstrap Confidence Interval Coverage”, with Table A6. The marked words introduce the coverage; the quantity covered is not stated.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N009-A9.png"}], "formal_release": null, "formal_statements": [], "headline": "The heterogeneous-effect RMSEs and bootstrap coverage in the supplement are measured against the sample average effect of each simulated data set, $n^{-1}\\sum_i\\tau_i$, and not against the population ATE of 5; the supplement and footnote 7 should say so.", "id": "P029-N009", "location": "Supplement, pp. A5 and A9 (PDF pp. 5 and 9), with Tables A1, A2 and A4–A6 and Figures A1–A3; article, footnote 7, p. 498 (PDF p. 7)", "pdf_page": 5, "printed_page": "A5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Supplement, p.~A5:\n\nWe assume $\\alpha = 0.275$, as recommended in the text. We present tables of \\hl{RMSE} vs $n \\in 500, 1000, 2500, 5000, 10000, 25000$ under the 5 different models, including misspecification.\n\nSupplement, p.~A9:\n\nIn Table A6, we present \\hl{\\% coverage} with the bootstrap for $n \\in 500, 2500, 5000$ assuming that the model is correct and $\\alpha = 0.275$.\n\nArticle, footnote 7:\n\nIn our own simulation studies, assuming $\\alpha = 0.275$, we find that the bootstrap has conservative \\hl{coverage} for ICSW in small samples, and proper \\hl{coverage} in larger samples.\n\nSimulation scripts in the replication files: \\texttt{output[1] <- mean(dgpout[[3]])}", "corrected": "Supplement, added to p.~A5 and p.~A9: \\hl{In the heterogeneous-effect simulations, RMSE is computed relative to the realized sample average treatment effect, $S_n=n^{-1}\\sum_i\\tau_i$. Table A6 likewise reports bootstrap interval coverage of $S_n$. These summaries do not report RMSE or coverage for the population ATE, $\\mathrm{E}\\,\\tau_i=5$.}\n\nArticle, footnote 7: In our own simulation studies, assuming $\\alpha = 0.275$, we find that the bootstrap has conservative coverage \\hl{of the realized sample average treatment effect} for ICSW in small samples, and proper coverage in larger samples.", "evidence": "The evidence is a derivation and a check of the scoring lines in the simulation scripts. For $X$ uniform on $(-1,1)$, $\\mathrm{E}X=0$ and $\\mathrm{E}X^2=1/3$. By independence of the three covariates, I have $\\mathrm{E}\\tau_i=5$ and $\\operatorname{Var}(\\tau_i)=25(1/3+1/3+1/3)=25$, and $\\operatorname{Var}(S_n)=25/n$. In the design ``Model is Incorrect'', $\\tau_i=5(1+X1_i+X2_i+X1_iX2_i+X3_i)$, and $\\operatorname{Var}(S_n)=250/(9n)$. Let $R_S$ and $R_\\theta$ denote the RMSE of an estimator $\\hat\\theta$ around $S_n$ and around $5$. Then\n\\[\nR_\\theta^2-R_S^2=\\operatorname{Var}(S_n)+2\\,\\mathrm{E}\\{(\\hat\\theta-S_n)(S_n-5)\\},\n\\]\nand the cross term need not vanish. Adding $25/n$ to a printed squared RMSE is not a valid conversion. In the scripts, the first stored quantity is \\texttt{mean(dgpout[[3]])}, the RMSE lines subtract it from each estimate, and the coverage lines in \\texttt{sims-BS.R} test whether it lies between the percentile endpoints. Applied to test inputs, the scripts' scoring lines and the same lines with $5$ as the target give different RMSEs and different coverage.\n\nThe saved simulation output is not in the replication files, and the tables were not recomputed against $5$. No printed entry, no ranking of the estimators and no statement about coverage is shown to change.", "problem": "The article defines the ATE as the population expectation $\\mathrm{E}(Y_1-Y_0)$. In the baseline simulation design with heterogeneous effects, $\\tau_i=5(1+X1_i+X2_i+X3_i)$ with covariates uniform on $(-1,1)$, and the population ATE is $5$. The heterogeneous-effect scripts store, for each simulated data set, the sample average $S_n=n^{-1}\\sum_i\\tau_i$. They compute the RMSE of each estimator around $S_n$, and they compute coverage as the frequency with which the bootstrap percentile interval contains $S_n$. The sample average $S_n$ is random. Scoring an estimator against it is legitimate, but the supplement does not say that it does so, and the definition in the article points to the population ATE. The RMSE around $S_n$ and the RMSE around $5$ are different quantities, and so are the two coverage probabilities.", "what_survives": "The heterogeneous-effect scoring formulas are valid for the sample average effect. Since $S_n$ converges to $5$, an estimator is consistent for one target if and only if it is consistent for the other. The identification argument, the estimator and the applications are unaffected. What changes is the reading of Tables A1, A2, A4 and A5, the heterogeneous panels of Figure A3, Figures A1 and A2, Table A6, and footnotes 6 and 7, which cite them. This evidence concerns the sample average effect. This finding does not cover Table A3, the design with a constant effect."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N009", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the sentence 'Thus the ML estimate of theta A,C and theta A given A,C' the display that follows and one line after it. A red box marks the display, in which the pair of estimates equals the argmax over both coefficient vectors of the likelihood L given D and Z.", "caption": "Section 3.1, p. 497 (PDF p. 6). The marked display defines the ML estimate as the maximizer of the likelihood over all coefficient vectors. The display is correct as printed.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N010.png"}, {"alt": "Listing of seven lines of R code from the function complier.score, three of them continued on a second row. Red boxes mark '1e4' in the default 'num.iter = ifelse(genoud, 200, 1e4)' on line 15, and 'maxfun=num.iter' and 'lower=-5,upper=5' in the call to bobyqa on line 81. Lines 86 to 89 compute CA.score, A.score and C.score from the returned parameters and return them.", "caption": "Replication files, icsw.R, lines 15, 80–81 and 86–89, typeset from the source file with line numbers added; long lines are continued after an arrow. The marked terms set the cap of 10,000 evaluations and the bounds $\\pm5$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N010-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication code for the two-sided compliance score confines each standardized coefficient to $[-5,5]$, stops after 10,000 evaluations and uses the result without a convergence check, so it can return a fit that is not the ML estimate defined on p. 497.", "id": "P029-N010", "location": "Section 3.1, p. 497 (PDF p. 6), display that defines the ML estimate; replication files, icsw.R, function complier.score, lines 15, 81 and 86–89, called at lines 37 and 92 of albertsonlawrence.R", "pdf_page": 6, "printed_page": "497", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Article, p.~497: Thus the ML estimate of $\\{\\theta_{\\mathbf{A,C}}, \\theta_{\\mathbf{A|A,C}}\\}$,\n\\[\n\\{\\hat\\theta_{\\mathbf{A,C}}, \\hat\\theta_{\\mathbf{A|A,C}}\\} = \\operatorname*{argmax}_{\\theta_{\\mathbf{A,C}},\\,\\theta_{\\mathbf{A|A,C}}}\\bigl[L(\\theta_{\\mathbf{A,C}}, \\theta_{\\mathbf{A|A,C}}|D,Z)\\bigr].\n\\]\n\nReplication files, \\texttt{icsw.R}, line 15 (one line in the file, shown here in two pieces):\n\n\\texttt{complier.score <- function(D, Z, W, weights = NULL, link = pnorm,}\n\n\\texttt{genoud = TRUE, num.iter = ifelse(genoud, 200, \\hl{1e4}),onesided=FALSE) \\{}\n\nLine 81 (also in two pieces):\n\n\\texttt{opt.log <- bobyqa(starting, complier.neg.logLik.fnc,control=list(iprint=0,}\n\n\\texttt{\\hl{maxfun=num.iter}),\\hl{lower=-5,upper=5})}\n\nLine 86, the first of the four lines that compute and return the scores (in two pieces):\n\n\\texttt{CA.score <- (pnorm(rowSums(t(t(W1) *}\n\n\\texttt{opt.log\\$par[1:(length(opt.log\\$par)/2)]))))}", "corrected": "The definition on p.~497 is unchanged. In \\texttt{icsw.R} and its callers, I would test the optimizer status before using the scores, and continue or report failed fits. I would remove or expand coefficient bounds that exclude better fits. A status of $0$ does not by itself establish a global maximum; I would describe the scores as numerical approximations to the ML scores unless their accuracy is established.", "evidence": "The evidence is an exact counterexample, a run of the code on it, and a run of the code on four bootstrap draws of the application.\n\nThe counterexample has 96 observations and two covariates, $W_1=U$ and $W_2=U+V/50$ with $U,V\\in\\{-1,1\\}$. Each of the eight cells $(U,V,Z)$ holds twelve observations. For either value of $U$, the number of observations with $D=1$ is $1$ for $(V,Z)=(-1,0)$, $2$ for $(-1,1)$, $5$ for $(1,0)$ and $10$ for $(1,1)$. Let $t$ be such that $\\Phi(t)=5/6$. The model of equations (5)--(7) holds with coefficients $\\theta_{\\mathbf{A,C}}=(0,-50t,50t)$ and $\\theta_{\\mathbf{A|A,C}}=(0,0,0)$ on the covariates $(1,W_1,W_2)$. Since these coefficients fit the proportion treated in every cell, they attain the maximum of the likelihood. In the standardized coordinates that the code uses, the third coefficient exceeds $100/3$. Let $M$ be the maximum of the likelihood and $B$ an upper bound for the likelihood over the box, namely\n\\[\nM=\\Bigl(\\frac56\\Bigr)^{40}\\Bigl(\\frac16\\Bigr)^{8}B_0,\\qquad B=\\Bigl(\\frac{11}{20}\\Bigr)^{40}\\Bigl(\\frac{9}{20}\\Bigr)^{8}B_0,\\qquad B_0=\\Bigl(\\frac{5}{12}\\Bigr)^{10}\\Bigl(\\frac{7}{12}\\Bigr)^{14}\\Bigl(\\frac{1}{12}\\Bigr)^{2}\\Bigl(\\frac{11}{12}\\Bigr)^{22}.\n\\]\nThe bound $B$ holds for the printed likelihood and for the likelihood with the probabilities clipped as in the code. Comparing the two in exact rational arithmetic, I find that $M>5000B$. Run on these data, the unchanged helper returns status $0$ after 253 evaluations, with two coefficients at the bound $5$ and negative log-likelihood $56.378$. The value at the maximizer is $44.812$.\n\nThe run on the application uses the script's seed and R 4.4.2 with \\texttt{minqa} 1.2.8. The fit on the full data ends with status $0$ after 1,096 evaluations. Of the first four bootstrap draws, the fourth stops at the cap (status $1$, 10,000 evaluations), and its scores are finite and are used. Continuing that search for 37,259 more evaluations lowers the negative log-likelihood from $197.12769$ to $197.12729$. A separate recalculation of the full bootstrap reported 39 capped fits among the 2,500 draws. I have not confirmed that count.", "problem": "The article defines the estimate as the maximizer of the likelihood over all coefficient vectors. The helper \\texttt{complier.score}, called with \\texttt{genoud=FALSE}, standardizes the covariates and searches with BOBYQA inside the box $[-5,5]$ for every coefficient. It stops after at most 10,000 evaluations of the objective and computes the scores from the last iterate without testing the status code that the optimizer returns. The box can exclude the unrestricted maximizer, and stopping at the cap does not establish convergence. The script nevertheless uses the scores without checking the returned status. The Albertson and Lawrence script calls this branch for the point estimate (line 37) and in every bootstrap replicate (line 92), so that the ICSW columns of Table 2 depend on these fits.", "what_survives": "The display on p.~497 is correct and stands as printed. No value in Table 2 is shown to change. A recalculation of the bootstrap that kept the capped fits, as the authors' code does, agreed with the printed intervals within Monte Carlo error. The intervals have not been recomputed with converged fits, and the software versions used for the article are not known. The 2SLS columns do not use the scores, and the consistency result is conditional on consistent scores. The finding concerns only the branch \\texttt{genoud=FALSE} of the two-sided fit in \\texttt{icsw.R}."}, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N010", "work": {"arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/"}, "work_id": "P029"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The displayed Equation (2), 'p = Pr(t_0 <= t) = |{t_0 in T_0 : t_0 <= t}| / |Omega-prime|', followed by a paragraph of seven lines. A red box marks 'p <= 1 - alpha/2' in the sentence 'fail to reject H_0 if alpha/2 <= p <= 1 - alpha/2, reject otherwise'. A second red box marks the word 'exact' in the sentence 'This procedure is an exact, distribution-free test for detecting interference between units.'", "caption": "Equation (2) and the paragraph after it, p. 7 (PDF p. 5). The upper condition of the two-tailed rule, $p\\le 1-\\alpha/2$, and the word “exact” are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E001.png"}, {"alt": "A chart with seven rows and a horizontal axis from 0 to 1 labelled 'Exact probability of rejecting when there is no interference', with a vertical line at the nominal level 0.05. Each row has an orange circle for the two-tailed rule as printed and a blue square for the corrected rule, and the exact values are listed at the right. Five units with the statistic equal to 1 in both assignments: printed 1, corrected 0. Forty-five units with the statistic equal to -1 or 1: printed 1, corrected 1/42. Reference distribution of the article's Table 1: printed 1/2, corrected 0. Six equally likely values with no ties: printed 1/6, corrected 0. Forty values: printed 1/40, corrected 1/20. Fifty values: printed 3/50, corrected 1/25. Five thousand values: printed 249/5000, corrected 1/20.", "caption": "Exact probability of rejection when there is no interference, at a nominal level of $0.05$, for the two-tailed rule as printed on p. 7 and for the corrected rule. The first three rows are the three examples above; the last four are reference distributions with $n$ equally likely distinct values. The printed rule rejects with probability $1$, $1$, $1/2$ and $1/6$ in the first four rows and with probability $3/50$ for $n=50$; it is below $0.05$ for $n=40$ and $n=5{,}000$. The corrected rule never exceeds $0.05$.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E001-null-rejection.png"}], "formal_release": null, "formal_statements": [{"id": "P030-T1", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T1"}, {"id": "P030-T2", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T2"}, {"id": "P030-T3", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T3"}, {"id": "P030-T4", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T4"}], "headline": "The two-tailed rule printed after Equation (2) can reject with probability greater than $\\alpha$ under the null hypothesis when the reference distribution is discrete; the upper side of the rule must use the inclusive upper-tail probability $\\Pr(t_0\\ge t)$.", "id": "P030-E001", "location": "The two-tailed rule after Equation (2), p. 7 (PDF p. 5)", "pdf_page": 5, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P030-T1", "P030-T2", "P030-T3", "P030-T4"], "text": {"as_printed": "\\[\np=\\Pr(t_0\\le t)=|\\{t_0\\in T_0:t_0\\le t\\}|/|\\Omega'|. \\qquad (2)\n\\]\n\\ldots\\ In these cases, a two-tailed significance test may be implemented using $p$: fail to reject $H_0$ if $\\alpha/2\\le\\hl{p\\le 1-\\alpha/2}$, reject otherwise. This procedure is an \\hl{exact}, distribution-free test for detecting interference between units.", "corrected": "Let $p_L=\\Pr(t_0\\le t)$, as in Equation (2), and let $\\hl{p_U=\\Pr(t_0\\ge t)}$. Both are computed under the conditional distribution of $Z$ on $\\Omega'$, and every assignment is counted, so that tied values of the statistic are retained. In these cases, a two-tailed significance test may be implemented using $\\hl{p_2=\\min\\{1,2\\min(p_L,p_U)\\}}$: reject $H_0$ if $\\hl{p_2\\le\\alpha}$, fail to reject otherwise. Suppose that the outcomes of the fixed units do not depend on $Z_{\\mathit{variant}}$, which is the hypothesis of no interference for these units, and that the probabilities of the assignments in $\\Omega'$ are those of the design. Then this test rejects with conditional probability \\hl{at most $\\alpha$}, for every finite reference distribution. The procedure is exact in this sense: its level is at most $\\alpha$ in finite samples, and it need not equal $\\alpha$. The rule that rejects if $p_L<\\alpha/2$ or $p_U<\\alpha/2$ keeps the strict endpoints of the printed rule and has the same property.", "evidence": "The evidence is three exact counterexamples and a derivation of the corrected rule. All probabilities are exact, and $\\alpha=1/20$ throughout.\n\nFirst, I consider five units on a line at the positions $0,4,8,-1,-2$. The units at $0,4,8$ form the fixed subset; they are never treated, and their outcomes are $0,1,2$ under every assignment. Exactly one of the units at $-1$ and $-2$ is treated, each with probability $1/2$, and both have outcome $0$ under every assignment. No outcome depends on any treatment, so $H_0$ holds. The distances from the fixed units to the treated unit are $(1,5,9)$ or $(2,6,10)$, and $\\rho=1$ under both assignments. The example therefore satisfies Equation (1) even as it is printed (see \\fid{P030-E002}). Then $p=1>39/40$ under both assignments, and the printed rule rejects with probability 1.\n\nSecond, I keep the fixed units and place 42 other units at $-1,-2,\\ldots,-41$ and $9$, each with outcome $0$ under every assignment. One of them is treated, each with probability $1/42$. The statistic equals $1$ when the treated unit is at a negative position and $-1$ when it is at $9$. For $\\rho=-1$ I have $p=1/42<1/40$, and for $\\rho=1$ I have $p=1$. The printed rule rejects both values, again with probability 1, although the statistic is not constant.\n\nThird, the reference distribution of Table 1 arises without interference when the ten outcomes $(1,2,2,2,2,3,2,3,2,1)$ are the same under every assignment. (The article's example generates these outcomes with interference; the population used here is a separate one.) The table below lists the six equally likely assignments. The printed rule rejects the three with $p=1$, which have total probability $1/2$.\n\n\\begin{audittable}{crrrll}{The reference distribution of Table 1 at $\\alpha=0.05$, with $p_L=\\Pr(t_0\\le t)$ and $p_U=\\Pr(t_0\\ge t)$. The printed rule rejects when $p_L<0.025$ or $p_L>0.975$; the corrected rule rejects when $2\\min(p_L,p_U)\\le 0.05$.}\nTreated unit & $\\rho_0$ & $p_L$ & $p_U$ & Printed rule & Corrected rule \\\\\n2 & $-0.105$ & $1$ & $1/2$ & \\hl{reject} & do not reject \\\\\n3 & $-0.105$ & $1$ & $1/2$ & \\hl{reject} & do not reject \\\\\n4 & $-0.105$ & $1$ & $1/2$ & \\hl{reject} & do not reject \\\\\n5 & $-0.833$ & $1/2$ & $2/3$ & do not reject & do not reject \\\\\n6 & $-0.949$ & $1/3$ & $1$ & do not reject & do not reject \\\\\n7 & $-0.949$ & $1/3$ & $1$ & do not reject & do not reject \\\\\n\\end{audittable}\n\nFor the corrected rule, let the reference distribution have the distinct values $x_1<\\cdots<x_m$ with probabilities $q_1,\\ldots,q_m$, and set $C_0=0$ and $C_j=q_1+\\cdots+q_j$. Under $H_0$ the observed statistic $t$ is a draw from this distribution. If $t=x_j$, then $p_L=C_j$ and $p_U=1-C_{j-1}$. Fix $u\\in[0,1]$. Since the $C_j$ increase, the event $p_L\\le u$ consists of the values $x_1,\\ldots,x_k$, where $k$ is the largest index with $C_k\\le u$, and its probability is $C_k\\le u$; the event is empty if there is no such index. In the same way, the event $p_U\\le u$ has probability at most $u$. Taking $u=\\alpha/2$ with $0<\\alpha<1$, I obtain\n\\[\n\\Pr\\bigl(2\\min(p_L,p_U)\\le\\alpha\\bigr)\\le\\Pr(p_L\\le\\alpha/2)+\\Pr(p_U\\le\\alpha/2)\\le\\alpha .\n\\]\nIn the first inequality, I used the union bound. In the same notation, I have $p_U=1-p_L+\\Pr(t_0=t)$. The printed rule compares $1-p_L$ with $\\alpha/2$ on the upper side, where the corrected rule compares $p_U$.", "problem": "The rule uses only the lower-tail probability $p=\\Pr(t_0\\le t)$ of Equation (2), which includes the observed value. On the lower side the rule rejects when $p<\\alpha/2$, and under $H_0$ this event has probability at most $\\alpha/2$. On the upper side it rejects when $p>1-\\alpha/2$, that is, when $\\Pr(t_0>t)<\\alpha/2$. This is a condition on the strict upper tail, which leaves out the probability that the reference distribution gives to the observed value. When the reference distribution is discrete, this probability can be large. In particular, the largest value of the statistic has $p=1$ and is always rejected, however probable it is. The rule can then reject with probability greater than $\\alpha$ when there is no interference, and the statement that the two-tailed procedure is exact does not hold. Excluding ties does not remove the problem. With $n$ equally likely distinct values, the rule rejects with probability $(2\\lceil n\\alpha/2\\rceil-1)/n$, which is $1/6$ for $n=6$ and $\\alpha=0.05$. Not every discrete distribution leads to a rejection probability above the nominal level: for $n=5{,}000$ the same expression equals $249/5000$. The one-tailed $p$ value of Equation (2) is not affected.", "what_survives": "The conditional randomization construction and the one-tailed $p$ value of Equation (2) are valid under the same assumptions, and Table 1 with its $p=0.333$ is unchanged. The application reports a lower-tail $p<0.001$ on p.~13, which the correction does not change; taking that value as given, the corrected two-tailed $p$ value is below $0.002$. The correction needs no new assumption. Its cost is that some two-tailed rejections are withdrawn. With strict endpoints, every rejection by the corrected rule is also a rejection by the printed rule, and in the first example no rejection remains. The article reports two-tailed rejection rates for its simulations (83\\% on p.~11, and Figure 2). The simulation code is not available, and how these rates change under the corrected rule has not been established."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E001", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Four lines of text ending 'With the t for a given realization of Z denoted t_Z, it follows that we may define the salient null hypothesis', then the displayed Equation (1), 'H_0 : t_Z = t_Z-prime, for all Z, Z-prime in Omega-prime', then two further lines of text. A red box marks 't_Z = t_Z-prime' in the display.", "caption": "Equation (1) with the sentence that introduces it, p. 7 (PDF p. 5). The equality $t_Z=t_{Z'}$ is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E002.png"}, {"alt": "Table 1, titled 'Conditional Randomizations and Associated Test Statistics Under H_0', with columns Z, Y_fixed, D_nearest and rho_0 and six rows. Y_fixed is {1, 3, 2, 1} in every row. A red box marks the rho_0 column, whose entries are -0.105, -0.105, -0.105, -0.833, -0.949 and -0.949.", "caption": "Table 1, p. 9 (PDF p. 7), whose title says that its statistics are computed under $H_0$. The column of $\\rho_0$ is marked: it takes three different values over the six assignments in $\\Omega'$.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E002-table1.png"}], "formal_release": null, "formal_statements": [{"id": "P030-T5", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T5"}, {"id": "P030-T6", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T6"}], "headline": "Equation (1) states the null hypothesis as equality of the test statistic over the assignments in $\\Omega'$; the hypothesis that the procedure tests is equality of the fixed units' outcomes, under which the statistic still varies.", "id": "P030-E002", "location": "Equation (1) and the sentence that introduces it, p. 7 (PDF p. 5)", "pdf_page": 5, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P030-T5", "P030-T6"], "text": {"as_printed": "With the $t$ for a given realization of $Z$ denoted $t_Z$, \\hl{it follows that} we may define the salient null hypothesis\n\\[\nH_0:\\hl{t_Z=t_{Z'}},\\ \\forall Z,Z'\\in\\Omega'. \\qquad (1)\n\\]", "corrected": "With the $t$ for a given realization of $Z$ denoted $t_Z$, \\hl{the salient null hypothesis is}\n\\[\nH_0:\\hl{Y_{i,Z}=Y_{i,Z'}}\\ \\hl{\\text{for every } i \\text{ with } S_i=1},\\ \\forall Z,Z'\\in\\Omega'. \\qquad (1)\n\\]\nThis hypothesis follows from $H_{0''}$, because every assignment in $\\Omega'$ leaves the treatment of the fixed units unchanged. Under $H_0$, the observed $Y_{\\mathit{fixed}}$ is held constant and $t_Z$ is recomputed for each $Z\\in\\Omega'$. The values $t_Z$ need not be equal.", "evidence": "The evidence is a counterexample computed in exact arithmetic. I consider ten units at the positions $1,\\ldots,10$, of which one is treated, each with probability $1/10$. Let the outcome of unit $i$ be $b_i$ under every assignment, where $b=(1,2,2,2,2,3,2,3,2,1)$. No outcome depends on any treatment, so there is no interference. Fix the units $1,8,9,10$ and suppose that unit 7 is treated. Then $\\Omega'$ consists of the six assignments that treat one of the units $2,\\ldots,7$, each with conditional probability $1/6$, and $Y_{\\mathit{fixed}}=(1,3,2,1)$ under all six. The Spearman correlation between $Y_{\\mathit{fixed}}$ and the distance to the treated unit is $-1/\\sqrt{90}$ when the treated unit is 2, 3 or 4, it is $-5/6$ when the treated unit is 5, and it is $-3/\\sqrt{10}$ when the treated unit is 6 or 7. These are the values $-0.105$, $-0.833$ and $-0.949$ of Table 1. Since $1/90\\ne 25/36$, I have $t_Z\\ne t_{Z'}$ for two assignments in $\\Omega'$, although the outcomes do not depend on the assignment. Counting the two assignments with $t_Z\\le -3/\\sqrt{10}$, I obtain $p=2/6$, which is the article's $0.333$.", "problem": "The hypothesis of no interference, $H_{0''}$ on p.~6, concerns outcomes: the outcome of a unit does not change when its own treatment does not. Since every assignment in $\\Omega'$ leaves the treatment of the fixed units unchanged, $H_{0''}$ implies that $Y_{\\mathit{fixed}}$ is the same for all $Z\\in\\Omega'$. It does not imply that the statistic is the same. The statistic $t_Z$ depends on $Z_{\\mathit{variant}}$ as well as on $Y_{\\mathit{fixed}}$, for example through $D_{\\mathit{nearest}}$, and it can vary over $\\Omega'$ whether or not there is interference; the two sentences before the display say as much. Equation (1) does not follow from them, and it is not the hypothesis that the procedure tests. If Equation (1) held as printed, the reference distribution would be a point mass at the observed value and the $p$ value of Equation (2) would always equal 1. The article itself uses the hypothesis on outcomes. Table 1 lists three different values of the statistic under $H_0$, and p.~10 says that ``in each randomization, we are assuming $H_0$, so $Y_{\\mathit{fixed}}$ remains constant across randomizations''.", "what_survives": "The correction adds no assumption and changes no number. The hypothesis $H_{0''}$ and the conditioning on $Z_{\\mathit{fixed}}$ are not affected, and Table 1 and $p=0.333$ are already computed under the hypothesis on outcomes. The simulations and the application are described with a reference distribution that varies over the assignments, which agrees with the corrected hypothesis; their numbers were not recomputed for this correction."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E002", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Five lines of text. In the third line a red box marks the denominator '(var(r_D_nearest) var(r_Y_fixed))' of the formula 'rho = cov(r_D_nearest, r_Y_fixed) / (var(r_D_nearest) var(r_Y_fixed))'. In the fourth line a second red box marks the word 'monotone' in 'whenever D_nearest has a monotone relationship with Y_fixed'.", "caption": "Practical Considerations, first paragraph, p. 8 (PDF p. 6). The denominator of the formula for $\\rho$ and the word “monotone” are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P030-T8", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T8"}, {"id": "P030-T9", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T9"}], "headline": "The formula for the rank correlation lacks the square root in its denominator, and $\\rho$ equals $1$ or $-1$ under a strictly monotone relationship, not under every monotone one.", "id": "P030-E004", "location": "Practical Considerations, first paragraph, p. 8 (PDF p. 6): the formula for $\\rho$ and the sentence after it", "pdf_page": 6, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P030-T8", "P030-T9"], "text": {"as_printed": "Whenever values are tied, the average rank is used. $\\rho$ then is the Pearson correlation coefficient of the ranks of these variables: $\\rho=\\operatorname{cov}(r_{D_{\\mathit{nearest}}},r_{Y_{\\mathit{fixed}}})/\\hl{(\\operatorname{var}(r_{D_{\\mathit{nearest}}})\\operatorname{var}(r_{Y_{\\mathit{fixed}}}))}$. $\\rho$ has a perfect correlation coefficient (of 1 or -1) whenever $D_{\\mathit{nearest}}$ has a \\hl{monotone} relationship with $Y_{\\mathit{fixed}}$.", "corrected": "Whenever values are tied, the average rank is used. $\\rho$ then is the Pearson correlation coefficient of the ranks of these variables,\n\\[\n\\rho=\\frac{\\operatorname{cov}(r_{D_{\\mathit{nearest}}},r_{Y_{\\mathit{fixed}}})}{\\hl{\\sqrt{\\operatorname{var}(r_{D_{\\mathit{nearest}}})\\operatorname{var}(r_{Y_{\\mathit{fixed}}})}}},\n\\]\nwhich is defined when neither vector of ranks is constant. $|\\rho|=1$ exactly when the centered vectors of ranks are proportional. This holds whenever $D_{\\mathit{nearest}}$ has a \\hl{strictly} monotone relationship with $Y_{\\mathit{fixed}}$, under which ties in one variable correspond to ties in the other. A relationship that is monotone but not strictly monotone need not give $|\\rho|=1$.", "evidence": "The evidence is a recalculation from Table 1 and a short derivation. In the last row of Table 1, the ranks of $D_{\\mathit{nearest}}=(6,1,2,3)$ are $(4,1,2,3)$ and the ranks of $Y_{\\mathit{fixed}}=(1,3,2,1)$ are $(3/2,4,3,3/2)$. With divisor $n$, their covariance is $-9/8$ and their variances are $5/4$ and $9/8$. The printed expression gives $(-9/8)/(45/32)=-4/5$, and $-3/5$ with divisor $n-1$. The Pearson correlation is $(-9/8)/\\sqrt{45/32}=-3/\\sqrt{10}$, which is the value $-0.949$ in the table. All six rows of Table 1 agree with the formula that has the square root.\n\nFor the second sentence, let $D=(1,2,3)$ and $Y=(0,0,1)$, so that $Y$ is a nondecreasing function of $D$. The average ranks are $(1,2,3)$ and $(3/2,3/2,3)$, and $\\rho=\\sqrt{3}/2$. The case of equality in the Cauchy--Schwarz inequality implies that $|\\rho|=1$ exactly when the centered vectors of ranks are proportional.", "problem": "The sentence names the Pearson correlation of the ranks, which divides the covariance by the square root of the product of the variances. The displayed expression divides by the product itself, and it is not a correlation. The next sentence is too broad when tied values are given average ranks. A relationship that is monotone but not strictly monotone can produce ties in one variable and not in the other, and then $|\\rho|<1$.", "what_survives": "Table 1, $p=0.333$ and the test itself use the Pearson correlation of the ranks and do not change. I found no reported number from the simulations or the application that is shown to depend on the printed expression."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E004", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Five lines of text. In the first line a red box marks the word 'exponential' in 'The third model is an exponential model'. In the second line a second red box marks the formula 'sum over j of ((1 - delta_ij) Z_j / ((1.5 d_ij)^2))' that defines Theta_i. The next sentence reads 'In the exponential model, the effect asymptotically approaches infinity as d_ij approaches zero.'", "caption": "Simulation Studies, p. 10 (PDF p. 8): the definition of the third distance-decay model. The word “exponential” and the sum that defines $\\Theta_i$ are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E006.png"}, {"alt": "The lower row of Figure 2. At the left, a panel titled 'Exponential Model', marked with a red box, plots 'Raw Value' against 'Distance' from 0 to 4: a curve that leaves the top of the panel near distance 0.7 and falls towards zero. At the right, a panel titled 'Simulated Indirect + Spatial DGP' plots 'Successful Detection Rate' against N from 100 to 800 for three significance levels.", "caption": "Figure 2, p. 12 (PDF p. 10), lower panels. The title “Exponential Model” is marked; the curve below it is $1/(1.5d)^2$.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E006-fig2.png"}], "formal_release": null, "formal_statements": [{"id": "P030-T10", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T10"}], "headline": "The third simulated model is an inverse-square decay, not an exponential one, and its sums should exclude the term $j=i$.", "id": "P030-E006", "location": "Simulation Studies, the third distance-decay model, p. 10 (PDF p. 8); the name recurs on p. 11 (PDF p. 9) and in Figure 2, p. 12 (PDF p. 10)", "pdf_page": 8, "printed_page": "10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P030-T10"], "text": {"as_printed": "The third model is an \\hl{exponential} model, for which $\\Theta_i=\\hl{{\\textstyle\\sum_j}}\\bigl((1-\\delta_{ij})Z_j/((1.5d_{ij})^2)\\bigr)$ and $\\Psi_i=\\hl{{\\textstyle\\sum_j}}\\bigl((1-\\delta_{ij})/((1.5d_{ij})^2)\\bigr)$. In the \\hl{exponential} model, the effect asymptotically approaches $\\infty$ as $d_{ij}$ approaches zero. In all cases, both $\\Theta$ and $\\Psi$ are mean-normalized to one.", "corrected": "The third model is an \\hl{inverse-square} model, for which\n\\[\n\\Theta_i=\\hl{\\sum_{j\\ne i}}Z_j/(1.5d_{ij})^2 \\quad\\text{and}\\quad \\Psi_i=\\hl{\\sum_{j\\ne i}}1/(1.5d_{ij})^2 .\n\\]\nIn the \\hl{inverse-square} model, the effect asymptotically approaches $\\infty$ as $d_{ij}$ approaches zero. In all cases, both $\\Theta$ and $\\Psi$ are mean-normalized to one.\n\nThe same name replaces ``exponential'' on p.~11 (``DGPs with the exponential model'') and in Figure 2 (``Exponential Model'').", "evidence": "The evidence is a derivation from the printed formula and a check of the pages. Let $k(d)=4/(9d^2)$. Then $k(2d)/k(d)=1/4$ for every $d>0$, and the derivative of $\\log k(d)$ is $-2/d$, which is not constant as it is for an exponential decay. The curve in the lower-left panel of Figure 2 agrees with $k(2/3)=1$, $k(1)=4/9$ and $k(2)=1/9$. The page prints no restriction to $j\\ne i$.", "problem": "For $d>0$ the decay function is $1/(1.5d)^2=4/(9d^2)$, a power of the distance. It falls by a factor of 4 whenever the distance doubles, and it diverges as $d\\to 0$, as the article notes. An exponential decay $\\exp(-cd)$ is bounded at $d=0$ and falls by a constant factor for each unit of distance. The model is called exponential on pp.~10 and 11 and in the title of the lower-left panel of Figure 2. As described, the simulations do not include an exponential decay. Separately, the sums run over all $j$. For $j=i$ the summand is $(1-\\delta_{ii})/(1.5d_{ii})^2=0/0$, which the factor $1-\\delta_{ij}$ does not define; the sums meant are over $j\\ne i$.", "what_survives": "The formula and the plotted curve agree on the decay function, and no reported detection rate is shown to change. The simulation code is not available. The correction concerns the name and the notation, and it is not a check of the implementation. The simulations cover Gaussian, linear and inverse-square decay; they do not show how the method behaves under exponential decay."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E006", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The foot of Figure 1, with the axis label Uniform Distribution Quantiles under each of its two panels, then the caption: Figure 1. Quantile-quantile plots of p values from simulations with the Gaussian distance decay DGP (N = 200) against the uniform distribution. A red box marks '(N = 200)'. Two lines of the next paragraph are visible below.", "caption": "Caption of Figure 1, p. 11 (PDF p. 9): the sample size “($N = 200$)” is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E007.png"}, {"alt": "Five lines of text beginning 'First, we illustrate a set of results for a simulation study using the Gaussian model, where N = 400, 200 units are randomly assigned to treatment, and 200 units are assigned to control'. A red box marks 'N = 400,'. The last line gives N_fixed = 100.", "caption": "Last paragraph of p. 10 (PDF p. 8): the text introduces the same simulation with “$N = 400$”, which is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E007-body.png"}], "formal_release": null, "formal_statements": [], "headline": "The text states $N = 400$ for the simulation shown in Figure 1 and the caption states $N = 200$; one of the two numbers is wrong.", "id": "P030-E007", "location": "Figure 1 caption, p. 11 (PDF p. 9); the simulation it belongs to is described at the foot of p. 10 (PDF p. 8)", "pdf_page": 9, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Page 10, text: ``First, we illustrate a set of results for a simulation study using the Gaussian model, where\n$N = \\hl{400}$, 200 units are randomly assigned to treatment, and 200 units are assigned to control. For all\nsimulations, we choose the fixed subset as a random sample of control units; following the rule of thumb,\n$N_{\\mathit{fixed}} = 100$.''\n\nPage 11, caption: ``Figure 1. Quantile-quantile plots of $p$ values from simulations with the Gaussian distance decay\nDGP ($N = \\hl{200}$) against the uniform distribution''", "corrected": "The two descriptions are to be reconciled with the original simulation output. If the design in the text produced\nthe figure, the caption reads ``\\ldots with the Gaussian distance decay DGP ($N = \\hl{400}$) against the uniform\ndistribution''. If a simulation with $N = 200$ produced it, the text needs the counts of that simulation instead:\nunder the same rule of thumb, 100 treated units, 100 control units and $N_{\\mathit{fixed}} = 50$. Without that\noutput, neither number is relabelled.", "evidence": "A check of the two pages. The simulation code and its output are not available, and the two pages do not settle which\nsample size produced the plot.", "problem": "In the paper $N$ is the total number of units (p.~5). The text introduces Figure 1 with $N = 400$, of which 200 units are treated and 200 are controls, and the caption of the same figure states $N = 200$.", "what_survives": "Only the sample size attributed to Figure 1 is in doubt. The plot, the 83\\% rejection rate and the\nKolmogorov--Smirnov $p = 0.661$ are not shown to be wrong."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E007", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of text. Red boxes mark 'N = 344,056,' at the end of one line and 'N_fixed = 114,629, N_control = 191,228, and N_treatment = 38,198.' on the next line.", "caption": "Application, p. 13 (PDF p. 11): the four counts in the sentence that begins “To summarize” are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P030-T11", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T11"}], "headline": "The three group counts sum to 344,055, one fewer than the printed $N = 344{,}056$.", "id": "P030-E008", "location": "Application, p. 13 (PDF p. 11), the sentence that begins “To summarize”", "pdf_page": 11, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P030-T11"], "text": {"as_printed": "To summarize, $N = \\hl{344{,}056}$, $N_{\\mathit{fixed}} = \\hl{114{,}629}$, $N_{\\mathit{control}} = \\hl{191{,}228}$, and\n$N_{\\mathit{treatment}} = \\hl{38{,}198}$.", "corrected": "The counts are to be reconciled with the analysis sample. If the total, the control count and the treatment count\nare kept, the sentence reads: ``To summarize, $N = 344{,}056$, $N_{\\mathit{fixed}} = \\hl{114{,}630}$,\n$N_{\\mathit{control}} = 191{,}228$, and $N_{\\mathit{treatment}} = 38{,}198$.'' The sum is also restored\nif any one of the other three entries is changed by one unit: $N = 344{,}055$, $N_{\\mathit{control}} = 191{,}229$ or\n$N_{\\mathit{treatment}} = 38{,}199$.", "evidence": "An exact calculation with the printed counts. The log of the source study records 344,084 voters before the stratum\nwas omitted: 114,640 in the three fixed conditions, 191,243 in control and 38,201 in the neighbors condition. Then the printed $N$ implies that 28 voters were omitted, and the printed group counts imply 29. I did not examine the voter-level data, and the omitted stratum and the incorrect counts are not identified.", "problem": "The fixed, control and treatment groups are disjoint and together make up the analysis sample, so their counts sum\nto $N$. A direct computation gives $114{,}629 + 191{,}228 + 38{,}198 = 344{,}055$. The counts are inconsistent\nby one; the arithmetic does not identify which entries are wrong.", "what_survives": "The method and the simulations do not use these counts. The discrepancy alone does not show that the estimate of\nthe direct effect, the rank correlation, Figure 3 or the $p$ value rests on a wrong sample. I did not recompute\nthose, since the distances and the analysis code are not available."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E008", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Figure 3: a histogram with a smooth curve over it, centred near 0.015 on a horizontal axis labelled Rank Correlation that runs from 0.00 to 0.03, with a dashed vertical line near 0.006. The vertical axis is labelled Density. Below, the caption reads: Figure 3. p distribution associated with H0 for the experimental evaluation. A red box marks the letter p.", "caption": "Figure 3 and its caption, p. 14 (PDF p. 12): the symbol $p$ in the caption is marked. The horizontal axis reads “Rank Correlation”.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "The caption of Figure 3 prints $p$ where $\\rho_0$ is meant: the figure shows the distribution of the rank correlation under $H_0$.", "id": "P030-E009", "location": "Figure 3 caption, p. 14 (PDF p. 12)", "pdf_page": 12, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Figure 3. $\\hl{p}$ distribution associated with $H_0$ for the experimental evaluation", "corrected": "Figure 3. $\\hl{\\rho_0}$ distribution associated with $H_0$ for the experimental evaluation", "evidence": "A check of the page: the caption, the axis label and the sentence on p.~13.", "problem": "The figure is a histogram of the rank correlation under the null hypothesis. Its horizontal axis is labelled\n``Rank Correlation'', and p.~13 introduces it as ``$T_0$, the approximate distribution of $\\rho_0$ associated with\n$H_0$''. The symbol in the caption is the italic $p$ that the caption of Figure 1 uses for $p$ values.", "what_survives": "Only the symbol in the caption changes. The histogram and the results drawn from it are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E009", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "The top row of Figure 2: a small plot of the Gaussian kernel against distance, and a plot of successful detection rate against N from 100 to 800 with three rising curves. The top curve has diamonds; the middle and lowest curves both have open circles. Red boxes mark the four circles of the lowest curve at N = 100, 200, 300 and 400, and the legend line that shows an open triangle for alpha = 0.01.", "caption": "Figure 2, Gaussian model row, p. 12 (PDF p. 10): the markers of the lowest curve at $N = 100$ to $400$ and the legend entry for $\\alpha = 0.01$ are marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-E010.png"}], "formal_release": null, "formal_statements": [], "headline": "In the Gaussian panel of Figure 2 the $\\alpha = 0.01$ curve is drawn with circles, the marker that the legend assigns to $\\alpha = 0.05$; it should carry triangles.", "id": "P030-E010", "location": "Figure 2, Gaussian model panel (top right) and its legend, p. 12 (PDF p. 10)", "pdf_page": 10, "printed_page": "12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "A figure, described. The legend in the Gaussian panel assigns a diamond to $\\alpha = 0.20$, a circle to\n$\\alpha = 0.05$ and a triangle to $\\alpha = 0.01$. In that panel the top curve has diamonds, the middle curve has\ncircles and the lowest curve has \\hl{circles}. In the linear and exponential panels the lowest curve has triangles.", "corrected": "Draw the Gaussian $\\alpha = 0.01$ curve with \\hl{open triangles}, as the legend has it, and keep open circles for\n$\\alpha = 0.05$.", "evidence": "A check of the page. Enlarged, the markers of the lowest Gaussian curve at $N = 100$ to $400$, where the curves are separated, are circles.", "problem": "Two curves in the Gaussian panel carry the same marker, and the legend does not identify the lowest one. Its position does. Since the test rejects less often at a smaller $\\alpha$, the lowest curve is the $\\alpha = 0.01$ series.", "what_survives": "Only the marker is wrong. No plotted value is shown to be wrong, and nothing else in the paper depends on it."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-E010", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Eleven lines of text from the Method section. Red boxes mark the expression 'Z_i = Z_i-prime', with the prime on the subscript i, in the second line (the definition of Omega-prime) and in the eighth line (the statement of the null hypothesis H_0''). Two lines below the second box the text reads 'Because Z_fixed = Z-prime_fixed, for all Z, Z-prime in Omega-prime', with the prime on Z.", "caption": "Method, second paragraph, p. 6 (PDF p. 4). The two conditions $Z_i=Z_{i'}$ are marked; two lines below the second mark, the same condition is printed correctly for the fixed subset, as $Z_{\\mathit{fixed}}=Z'_{\\mathit{fixed}}$.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-N001.png"}], "formal_release": null, "formal_statements": [{"id": "P030-T7", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T7"}, {"id": "P030-T12", "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-T12"}], "headline": "In the definition of $\\Omega'$ and in $H_{0''}$ the prime is printed on the unit index, $Z_i=Z_{i'}$, where $Z_i=Z'_i$ is meant.", "id": "P030-N001", "location": "Method, second paragraph, p. 6 (PDF p. 4): the definition of $\\Omega'$ and the statement of $H_{0''}$", "pdf_page": 4, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P030-T12", "P030-T7"], "text": {"as_printed": "Let $\\Omega'$ be all possible randomizations conditional on observing $Z_{\\mathit{fixed}}$. That is, $Z_{\\mathit{fixed}}$ is invariant for all $Z$ in $\\Omega'$; $Z_i=\\hl{Z_{i'}}$, $\\forall Z,Z'\\in\\Omega':S_i=1$. \\ldots\\ However, we are concerned with a different sharp null hypothesis: $H_{0''}:Y_{i,Z}=Y_{i,Z'},\\ \\forall i,Z,Z'\\in\\Omega:Z_i=\\hl{Z_{i'}}$.", "corrected": "Let $\\Omega'$ be all possible randomizations conditional on observing $Z_{\\mathit{fixed}}$. That is, $Z_{\\mathit{fixed}}$ is invariant for all $Z$ in $\\Omega'$; $Z_i=\\hl{Z'_i}$, $\\forall Z,Z'\\in\\Omega':S_i=1$. \\ldots\\ $H_{0''}:Y_{i,Z}=Y_{i,Z'},\\ \\forall i,Z,Z'\\in\\Omega:Z_i=\\hl{Z'_i}$.", "evidence": "The evidence is a check of the page. The units are indexed by $i\\in\\{1,2,\\ldots,N\\}$ on p.~4, and no index $i'$ is introduced. Two lines below the statement of $H_{0''}$, the same condition for the fixed subset is printed with the prime on the vector: $Z_{\\mathit{fixed}}=Z'_{\\mathit{fixed}}$, $\\forall Z,Z'\\in\\Omega'$.", "problem": "As printed, each condition compares the treatment of unit $i$ with the treatment of a unit $i'$ in the same vector $Z$, and no unit $i'$ is defined. The condition meant is that unit $i$ has the same treatment under the two assignments $Z$ and $Z'$.", "what_survives": "Nothing depends on the misprint. The article uses $\\Omega'$ and $H_{0''}$ in their intended form in everything that follows."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-N001", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Four lines of text ending 'and assuming that all Z in Omega-prime occur with equal probability,', followed by the displayed Equation (2). A red box marks the numerator '|{t_0 in T_0 : t_0 <= t}|'.", "caption": "Equation (2) with the sentences that introduce it, p. 7 (PDF p. 5). The numerator is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "The numerator of Equation (2) is the number of assignments in $\\Omega'$ whose statistic is at most the observed one; assignments with equal statistics are each counted.", "id": "P030-N002", "location": "Equation (2), the numerator, p. 7 (PDF p. 5)", "pdf_page": 5, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\[\np=\\Pr(t_0\\le t)=\\hl{|\\{t_0\\in T_0:t_0\\le t\\}|}/|\\Omega'|. \\qquad (2)\n\\]", "corrected": "\\[\np=\\Pr(t_0\\le t)=\\hl{|\\{Z\\in\\Omega':t_0(Z)\\le t\\}|}/|\\Omega'|, \\qquad (2)\n\\]\nwhere $t_0(Z)$ denotes the statistic computed under $H_0$ for the assignment $Z$.", "intended_meaning": "The numerator is written as the size of a set of values of the statistic, and a set in the usual sense does not repeat equal values. The count meant is over assignments. The paragraph before the display builds $T_0$ from one $t_0$ for each $Z\\in\\Omega'$ and calls it a distribution, and the worked example counts ``2 of the 6 conditional randomizations'' on p.~10, although the six assignments of Table 1 have only three distinct values of $\\rho_0$. Read in this way, the display is the lower-tail probability, and I obtain the article's $p=2/6=0.333$. Counting distinct values would give $1/6$."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-N002", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of text. In the third line a red box marks an exclamation mark followed by the existential quantifier, in the sentence 'And for each Z in Omega-prime, ! exists rho_0, the rank correlation under H_0.'", "caption": "A Simple Example, second paragraph, p. 9 (PDF p. 7). The symbol $!\\exists$ is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The symbol “$!\\exists\\rho_0$” means that each assignment in $\\Omega'$ has exactly one value $\\rho_0$; the usual symbol is $\\exists!$.", "id": "P030-N003", "location": "A Simple Example, second paragraph, p. 9 (PDF p. 7)", "pdf_page": 7, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "And for each $Z\\in\\Omega'$, $\\hl{!\\exists}\\rho_0$, the rank correlation under $H_0$.", "corrected": "And for each $Z\\in\\Omega'$, \\hl{there exists a unique} $\\rho_0$, the rank correlation under $H_0$.", "intended_meaning": "The exclamation mark is printed before the existential quantifier. This is not the usual order for the symbol of unique existence, and the mark could be taken for a negation. The article defines no such convention, and Table 1 lists one $\\rho_0$ for each of the six assignments. The sentence means that each $Z\\in\\Omega'$ determines exactly one rank correlation under $H_0$, and nothing false is printed."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-N003", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "simulation", "exhibits": [{"alt": "Seven lines of text. Red boxes mark the words 'in over 95 percent of simulations' at the end of one line and 'once N reaches approximately 5,000.)' at the start of the next, inside the sentence '(Although not shown in the graph, the method detects interference with alpha = 0.05 for DGPs with the exponential model in over 95 percent of simulations once N reaches approximately 5,000.)'.", "caption": "Simulation Studies, p. 11 (PDF p. 9): the parenthetical on detection at large $N$, with “in over 95% of simulations once $N$ reaches approximately 5,000.)” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "With the two-tailed test of the surrounding text, a recalculation detects interference under the third model in 93.0% of simulations at $N = 5{,}000$ and 92.25% at $N = 6{,}000$, not in over 95%.", "id": "P030-N005", "location": "Simulation Studies, p. 11 (PDF p. 9), the parenthetical that follows the description of Figure 2", "pdf_page": 9, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "(Although not shown in the graph, the method detects interference with $\\alpha = 0.05$ for DGPs with the exponential\nmodel in \\hl{over 95\\% of simulations once $N$ reaches approximately 5,000}.)", "corrected": "(Although not shown in the graph, for DGPs with the \\hl{inverse-square} model and $\\alpha = 0.05$, a recalculation with\n5,000 conditional randomizations per simulation detects interference with the two-tailed test in\n\\hl{93.0\\% of 600 simulations at $N = 5{,}000$ and 92.25\\% of 400 simulations at $N = 6{,}000$} (standard errors\n1.04 and 1.34 percentage points). With the one-tailed $p$ value of Equation (2), the rates are 94.83\\% and 94.5\\%\n(standard errors 0.90 and 1.14). At $N = 8{,}000$, in 300 simulations, the rates are 97.00\\% two-tailed and\n97.33\\% one-tailed (standard errors 0.98 and 0.93).)\n\nThese rates are for the design of pp.~10--11: the kernel $1/(1.5 d_{ij})^2$ summed over $j \\neq i$, $\\Theta$ and\n$\\Psi$ each normalized to mean one, half of the units treated, and a random half of the controls fixed. The words\n``over 95\\% of simulations once $N$ reaches approximately 5,000'' are not kept. If the sentence is restated for\nthe one-tailed test, they are still not supported, since the one-tailed rates at $N = 5{,}000$ and $6{,}000$ are not shown to\nexceed 95\\%.", "evidence": "A recalculation by simulation of the design printed on pp.~10--11. Each unit has coordinates drawn uniformly on\n$[0, \\sqrt{N}]^2$ and noise $\\epsilon_i \\sim N(0,1)$. The outcome is\n\\[\nY_i = Z_i + \\Theta_i + \\Psi_i + \\epsilon_i, \\qquad\n\\Theta_i = \\sum_{j \\neq i} \\frac{Z_j}{(1.5 d_{ij})^2}, \\qquad\n\\Psi_i = \\sum_{j \\neq i} \\frac{1}{(1.5 d_{ij})^2},\n\\]\nwhere $\\Theta$ and $\\Psi$ are each divided by their mean over the $N$ units. Half of the units are treated, and a\nrandom half of the controls is the fixed subset. The statistic is the rank correlation between\n$Y_{\\mathit{fixed}}$ and the distance to the nearest treated unit, and each simulation uses 5,000\nconditional randomizations, the number that the paper reports for Figure 1. The one-tailed test rejects when the $p$ value of Equation (2) is\nbelow 0.05. The two-tailed test rejects when it is below 0.025 or above 0.975.\n\n\\begin{audittable}{rrrrrr}{Third model, $\\alpha = 0.05$: rejections among the simulations, with the rate and its Monte Carlo standard error in percentage points.}\n$N$ & Simulations & Two-tailed & Rate (s.e.) & One-tailed & Rate (s.e.) \\\\\n5,000 & 600 & 558 & 93.00\\% (1.04) & 569 & 94.83\\% (0.90) \\\\\n6,000 & 400 & 369 & 92.25\\% (1.34) & 378 & 94.50\\% (1.14) \\\\\n8,000 & 300 & 291 & 97.00\\% (0.98) & 292 & 97.33\\% (0.93) \\\\\n\\end{audittable}\n\nAt $N = 5{,}000$ and $N = 6{,}000$ the exact 95\\% intervals for the two-tailed rate are $[0.907, 0.949]$ and\n$[0.892, 0.947]$, and both end below 0.95. Testing the two counts together, 927 rejections in 1,000 simulations, with an exact binomial test against a rate of at least 0.95 at both values of $N$, I find $p = 0.001$. The one-tailed\nintervals, $[0.927, 0.965]$ and $[0.918, 0.965]$, contain 0.95. I checked the code against an exhaustive\nenumeration of the 84 conditional assignments of a small example.\n\nThe original simulation code and output are not available. The rates above are estimates from this recalculation,\nwith the Monte Carlo error stated, and three values of $N$ do not establish the rate at every larger $N$.", "problem": "The sentence concerns the third model, which the paper calls exponential and whose kernel is the inverse square\n$1/(1.5 d_{ij})^2$ (see \\fid{P030-E006}). It does not say whether its test is one-tailed or two-tailed, and\n``approximately 5,000'' has no stated tolerance. Since the paragraph before it reports a rejection rate ``when using a two-tailed test'', the natural reading is the two-tailed rule of p.~7 at $\\alpha = 0.05$.\n\nUnder that reading the stated rate is too high. In a recalculation of the design as printed, the two-tailed test\ndetects interference in 93.0\\% of simulations at $N = 5{,}000$ and in 92.25\\% at $N = 6{,}000$. With the\none-tailed $p$ value of Equation (2), the rates are 94.83\\% and 94.5\\%, which neither bear out ``over 95\\%'' nor\ncontradict it. At $N = 8{,}000$ the estimate of the two-tailed rate is 97.00\\%.", "what_survives": "The method, the worked example, the Gaussian result and the plotted curves of Figure 2 do not depend on this\nsentence. The application reports $p < 0.001$, which rejects under either tail rule. Only the sentence itself\nchanges: near $N = 5{,}000$ the detection rate of the two-tailed test under this model is about 93\\%, and the one-tailed rate\nis not shown to exceed 95\\%."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-N005", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "simulation", "exhibits": [{"alt": "The middle row of Figure 2: a plot of the linear kernel falling from 2 at distance 0 to 0 at distance 2, and a plot of successful detection rate against N from 100 to 800. A red box encloses the three rising curves. The middle curve, with circles, runs from about 0.15 at N = 100 through about 0.53 at N = 400 to about 0.83 at N = 800.", "caption": "Figure 2, linear model row, p. 12 (PDF p. 10): the detection-rate panel is marked. By the legend of the Gaussian panel above it, diamonds, circles and triangles are $\\alpha = 0.20$, $0.05$ and $0.01$.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-N006.png"}, {"alt": "Seven lines of text that define the second and third models. Red boxes mark the factor [2 - d_ij > 0](2 - d_ij) in the formula for Theta_i and the same factor in the formula for Psi_i, which breaks across two lines. The last sentence reads: In all cases, both Theta and Psi are mean-normalized to one.", "caption": "Definition of the linear model, p. 10 (PDF p. 8): the weight $[2 - d_{ij} > 0](2 - d_{ij})$ is marked in $\\Theta_i$ and in $\\Psi_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-N006-kernel.png"}, {"alt": "Three panels of detection rate against N on a log scale, titled Gaussian model, Linear model and Third model (inverse square). In the first and third panels the published and recomputed curves stay close. In the linear panel the published curve rises from 0.15 to 0.83 and the recomputed curve from 0.10 to 0.49; at N = 400 the two are labelled 0.53 and 0.27.", "caption": "Detection rates at $\\alpha = 0.05$ read from the published Figure 2 (blue, solid) and recomputed from the models as printed (orange, dashed; the band is the Wilson 95% interval for simulation uncertainty), at $N = 100$, 200, 400 and 800. The two are close for the Gaussian model and the third model and differ substantially for the linear model at these values of $N$.", "url": "https://pmaronow.github.io/llm-audit/works/P030/exhibits/P030-N006-linear-panel.png"}], "formal_release": null, "formal_statements": [], "headline": "The linear panel of Figure 2 shows higher detection rates than the linear model printed on p. 10 produces: at $\\alpha = 0.05$, about 0.53 against 0.27 at $N = 400$, and 0.83 against 0.49 at $N = 800$.", "id": "P030-N006", "location": "Figure 2, linear model panel (middle right), p. 12 (PDF p. 10); the linear model is defined on p. 10 (PDF p. 8)", "pdf_page": 10, "printed_page": "12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Page 10: ``The second model is a linear model, for which\n$\\Theta_i = \\sum_j \\big( (1 - \\delta_{ij}) Z_j [2 - d_{ij} > 0](2 - d_{ij}) \\big)$ and\n$\\Psi_i = \\sum_j \\big( (1 - \\delta_{ij}) [2 - d_{ij} > 0](2 - d_{ij}) \\big)$, where $[\\cdot]$ is the Iverson\nbracket. \\ldots In all cases, both $\\Theta$ and $\\Psi$ are mean-normalized to one.''\n\nPage 12, Figure 2, linear model panel. The markers have no numerical labels; the values are my readings of the plotted\nmarkers, to within 0.01 for circles. At $\\alpha = 0.05$ the plotted detection rate is about \\hl{0.15} at\n$N = 100$, \\hl{0.30} at $N = 200$, \\hl{0.53} at $N = 400$ and \\hl{0.83} at $N = 800$.", "corrected": "If the linear model printed on p.~10 is the intended one, I would redraw the linear panel of Figure 2 from it and publish the simulation specification and code. Take\n$\\Theta_i = \\sum_j \\big( (1 - \\delta_{ij}) Z_j [2 - d_{ij} > 0](2 - d_{ij}) \\big)$ and\n$\\Psi_i = \\sum_j \\big( (1 - \\delta_{ij}) [2 - d_{ij} > 0](2 - d_{ij}) \\big)$, each normalized to mean one over all $N$ units.\nUse $Y_i = Z_i + \\Theta_i + \\Psi_i + e_i$, with independent $e_i \\sim \\mathcal{N}(0,1)$, independent uniform locations on a square of area $N$, exactly $N/2$ uniformly chosen treated units and $N/4$ uniformly chosen control units fixed.\nIn each of 5,000 conditional randomizations, hold the fixed units' outcomes and treatment status constant and uniformly assign $N/2$ treatments among the remaining $3N/4$ units, using the rank-correlation statistic and two-tailed rule described above. For this design my estimated two-tailed detection rate at $\\alpha = 0.05$\nis about \\hl{0.10} at $N = 100$, \\hl{0.14} at $N = 200$, \\hl{0.27} at $N = 400$ and \\hl{0.49} at $N = 800$\n(Monte Carlo standard errors 0.009, 0.010, 0.013 and 0.014). The table above gives the rates at $\\alpha = 0.20$ and $0.01$. I\ndid not compute the rates at $N = 300$, 500, 600 and 700.\n\nIf the plotted curves came from a different model, its specification should replace the formulas for $\\Theta_i$ and $\\Psi_i$ on\np.~10 and the kernel drawn at the left of the panel should match it. The evidence does not identify it, and I give no replacement\nformula.", "evidence": "A recalculation by simulation of the model as printed. At each of $N = 100$, 200, 400 and 800, I ran 1,200 independent\nsimulations, each with 5,000 conditional randomizations and the two-tailed rule of p.~7. The design follows the\ntext: units placed uniformly on a square of area $N$, $N/2$ units treated, a random half of the controls fixed,\nstandard normal noise, and the rank correlation between the outcomes of the fixed units and the distance to the\nnearest treated unit as the statistic. The table sets the rates read from the published figure beside the\nrecomputed rates.\n\n\\begin{audittable}{rrrrr}{Linear model: detection rate read from Figure 2 and the rate recomputed from the model as printed (1,200 simulations at each $N$, two-tailed test).}\n$N$ & $\\alpha$ & Figure 2 (read) & Recomputed & Wilson 95\\% interval \\\\\n100 & 0.20 & 0.389 & 0.282 & 0.257 to 0.308 \\\\\n100 & 0.05 & 0.154 & 0.098 & 0.083 to 0.116 \\\\\n100 & 0.01 & 0.054 & 0.030 & 0.022 to 0.041 \\\\\n200 & 0.20 & 0.550 & 0.364 & 0.337 to 0.392 \\\\\n200 & 0.05 & 0.295 & 0.141 & 0.122 to 0.162 \\\\\n200 & 0.01 & 0.128 & 0.052 & 0.041 to 0.066 \\\\\n400 & 0.20 & 0.779 & 0.524 & 0.496 to 0.552 \\\\\n400 & 0.05 & 0.530 & 0.266 & 0.242 to 0.292 \\\\\n400 & 0.01 & 0.302 & 0.104 & 0.088 to 0.123 \\\\\n800 & 0.20 & 0.953 & 0.722 & 0.696 to 0.746 \\\\\n800 & 0.05 & 0.826 & 0.488 & 0.460 to 0.517 \\\\\n800 & 0.01 & 0.624 & 0.265 & 0.241 to 0.291 \\\\\n\\end{audittable}\n\nThe readings carry a tolerance of 0.01 for diamonds and circles and 0.02 for triangles. At $\\alpha = 0.05$ the\nfigure's value lies above the 95\\% interval by more than that tolerance at every checked $N$, and at $N = 400$ and 800\nthis holds at all three values of $\\alpha$. With 20,000 conditional randomizations at $N = 400$, in 400\nsimulations, I find 0.275 at $\\alpha = 0.05$ (standard error 0.022). The number of randomizations does not explain the gap. I checked the code against a direct calculation on all 495 conditional assignments of an example\nwith 16 units. This is a statistical comparison and not an exact calculation of power. It does not say whether the\nfigure or the formula departs from what was simulated.", "problem": "Figure 2 reports, for each of the three models, how often the test detects interference as $N$ grows.\nFor the linear model the plotted rates lie above the rates that the model produces as it is printed on p.~10:\nweights $2 - d_{ij}$ out to distance 2, the unit's own term left out, and $\\Theta$ and $\\Psi$ each normalized to\nmean one. At $\\alpha = 0.05$ the figure shows about 0.53 at $N = 400$ and 0.83 at $N = 800$. For the printed model, I obtain 0.27 and 0.49. At the four checked values of $N$, the same recalculation comes within 0.024 of every reading for the Gaussian panel and the panel of the third model. The large gap is specific to the linear panel at those values.\nEither the plotted linear curves or the printed linear model is not what was simulated, and the paper's simulation\ncode is not available to say which. A cutoff of $\\sqrt{2}$ in place of 2 comes closer but overshoots the figure\n(0.60 at $N = 400$ and 0.88 at $N = 800$). It does not identify the simulated model either. Taken from this\npanel, the power of the test under linear decay is overstated for the model as printed.", "what_survives": "The test and its construction do not use these simulated rates. The Gaussian simulation of Figure 1, the other two\npanels of Figure 2 and the voter-turnout application do not use the linear model. The recomputed linear rates\nstill rise across the four checked values of $N$, supporting the qualitative trend of increasing detection.\nFor the printed linear model, at $\\alpha = 0.05$ and $N = 400$ the estimated rate is 0.27 where the figure\nshows 0.53. My recalculations stop at $N = 800$ and do not test the remark that the rate appears to converge to\n100\\%."}, "url": "https://pmaronow.github.io/llm-audit/works/P030/#P030-N006", "work": {"arxiv_id": null, "doi": "10.1177/0049124112437535", "title": "A General Method for Detecting Interference Between Units in Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P030/"}, "work_id": "P030"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the abstract on the first page. A red box marks the words 'equivalent to' in the sentence saying that the Neyman and constant-effects estimators are equivalent to a variant of the White estimator and the homoskedastic OLS estimator, respectively.", "caption": "Abstract, p. 365 (PDF p. 1). The marked words state the equivalence of the constant-effects and OLS estimators without the finite-sample factor.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-A001.png"}, {"alt": "Crop of the end of the second paragraph of the introduction. A red box marks the words 'We also show that all four of these estimators are equivalent under a balanced design'. The preceding sentence, not marked, contains the words 'subject to a finite-sample scaling coefficient'.", "caption": "Introduction, second paragraph, p. 365 (PDF p. 1). The marked sentence follows the one that says “subject to a finite-sample scaling coefficient”.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-A001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The abstract and the balanced-design sentence of the introduction call the estimators equivalent without repeating the finite-sample factor $(N-2)/(N-1)$ of Theorems 2 and 3.", "id": "P031-A001", "location": "Abstract and Introduction, second paragraph, p. 365 (PDF p. 1)", "pdf_page": 1, "printed_page": "365", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Abstract.} This paper demonstrates that the randomization-based ``Neyman'' and constant-effects estimators for the variance of estimated average treatment effects are \\hl{equivalent to} a variant of the White ``heteroskedasticity-robust'' estimator and the homoskedastic ordinary least squares (OLS) estimator, respectively.\n\n\\emph{Introduction.} We also show that \\hl{all four of these estimators are equivalent} under a balanced design, suggesting that, in many cases, the choice of variance estimator is irrelevant.", "corrected": "\\emph{Abstract.} This paper demonstrates that the randomization-based ``Neyman'' and constant-effects estimators for the variance of estimated average treatment effects are equivalent to a variant of the White ``heteroskedasticity-robust'' estimator and, \\hl{up to the finite-sample factor} $\\hl{(N-2)/(N-1)}$, the homoskedastic ordinary least squares (OLS) estimator, respectively. That is,\n\\[\n\\hat V_{CE}(\\hat\\beta)=\\frac{N-2}{N-1}\\,\\hat V_{OLS}(\\hat\\beta),\n\\]\nwhere $N$ is the number of units.\n\n\\emph{Introduction.} \\hl{After the constant-effects estimator is multiplied by} $\\hl{(N-1)/(N-2)}$, all four of these estimators \\hl{coincide} under a balanced design.", "intended_meaning": "The two sentences mean equivalence up to the finite-sample scaling. The sentence before the second one says ``subject to a finite-sample scaling coefficient'', and Theorems 2 and 3 and the Discussion state the factor. Theorem 2 states that $\\hat V_{CE}(\\hat\\beta)=\\frac{N-2}{N-1}\\hat V_{OLS}(\\hat\\beta)$, and Theorem 3 states that $\\frac{N-1}{N-2}\\hat V_{CE}(\\hat\\beta)=\\hat V_{OLS}(\\hat\\beta)=\\hat V_N(\\hat\\beta)=\\hat V_{HC2}(\\hat\\beta)$ when $N=2M$. Both hold as printed. The raw estimates are not all equal. For $N=4$, $M=2$, $X=(1,1,0,0)$ and $Y=(0,2,0,4)$, I find in exact arithmetic that $\\hat V_{CE}(\\hat\\beta)=10/3$ and $\\hat V_{OLS}(\\hat\\beta)=\\hat V_N(\\hat\\beta)=\\hat V_{HC2}(\\hat\\beta)=5$. No theorem or formula changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-A001", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2011.10.024", "title": "On Equivalencies Between Design-Based and Regression-Based Variance Estimators for Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P031/"}, "work_id": "P031"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph that follows the displayed bias of the Neyman estimator. Red boxes mark the sentence beginning 'As demonstrated in Freedman (2008a, Theorem 1), asymptotic normality of beta-hat implies that large-sample confidence intervals' and ending 'approximately correct or conservative coverage'.", "caption": "Section 3, p. 367 (PDF p. 3), below the display of the bias of the Neyman estimator. The marked sentence infers coverage from asymptotic normality without stating further conditions.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E001.png"}, {"alt": "Crop of the two lines after the proof of Corollary 1. Red boxes mark the sentences 'Above, we noted that large-sample confidence intervals based on V-hat-N had either approximately correct or conservative coverage. Clearly, this extends to V-hat-HC2', followed by the marker of footnote 2.", "caption": "Section 3, p. 368 (PDF p. 4), after the proof of Corollary 1. The marked sentences extend the coverage statement to the HC2 estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E001-b.png"}, {"alt": "Crop of footnote 2. Red boxes mark its second sentence, 'The other estimators are all asymptotically equivalent to HC2 and, thus, large-sample confidence intervals based on any of the heteroskedasticity-robust estimators have either approximately correct or conservative coverage'.", "caption": "Footnote 2, p. 368 (PDF p. 4). The marked sentence extends the statement to the other heteroskedasticity-robust estimators.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E001-c.png"}], "formal_release": null, "formal_statements": [{"id": "P031-T5", "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-T5"}], "headline": "The statements about the large-sample coverage of the Neyman and HC2 intervals, and their extension to HR, HC1 and HC3 in footnote 2, hold in the asymptotic regime of the cited results, and the conditions of that regime should be stated.", "id": "P031-E001", "location": "Section 3, p. 367 (PDF p. 3), sentence after the bias display; p. 368 (PDF p. 4), sentence after Corollary 1 and footnote 2", "pdf_page": 3, "printed_page": "367", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P031-T5"], "text": {"as_printed": "p.~367: ``The Neyman estimator is therefore conservative: its bias is always nonnegative. As demonstrated in Freedman (2008a, Theorem 1), \\hl{asymptotic normality of} $\\hat\\beta$ \\hl{implies} that large-sample confidence intervals, using $\\hat V_N(\\hat\\beta)$ in a normal approximation will have either approximately correct or conservative coverage.''\n\np.~368: ``Above, we noted that large-sample confidence intervals based on $\\hat V_N(\\hat\\beta)$ had either approximately correct or conservative coverage. \\hl{Clearly, this extends} to $\\hat V_{HC2}(\\hat\\beta)$.''\n\np.~368, footnote 2: ``Implications for the other heteroskedasticity-robust estimators (HR, HC1, and HC3) studied in MacKinnon and White (1985) and Angrist and Pischke (2009, 299--308) are straightforward. The other estimators \\hl{are all asymptotically equivalent to HC2} and, thus, large-sample confidence intervals based on any of the heteroskedasticity-robust estimators have either approximately correct or conservative coverage.''", "corrected": "\\hl{Under conditions ensuring that} $(\\hat\\beta-\\beta)/\\sqrt{V(\\hat\\beta)}$ \\hl{converges in distribution to a standard normal variable and that} $\\hl{\\hat V_N(\\hat\\beta)/V(\\hat\\beta)\\to_p c\\in[1,\\infty)}$, normal confidence intervals based on the Neyman estimator or on HC2 have asymptotically correct or conservative coverage. \\hl{If both the treatment arm and the control arm grow}, HR, HC1 and HC3 are first-order equivalent to HC2 and inherit this conclusion under the same conditions. For the coverage result, it does not suffice that the variance estimator has nonnegative expected bias.", "intended_meaning": "The sentences are meant in the large-sample regime of the results they cite: both arms grow, with $M/N\\to p_0\\in(0,1)$, and the potential outcomes have convergent second moments and bounded fourth moments. Freedman (2008a) states Theorem 1 under conditions of this kind, which I read in the IMS reprint of that article, and in that regime the sentences are correct. Let $v_1$ and $v_0$ denote the limiting variances of the potential outcomes and $v_\\tau$ the limiting variance of the unit-level treatment effects, and suppose that $v=v_1/p_0+v_0/(1-p_0)-v_\\tau>0$. Then $N\\,V(\\hat\\beta)\\to v$ and $N\\hat V_N(\\hat\\beta)\\to_p v_1/p_0+v_0/(1-p_0)\\ge v$, and Slutsky's theorem implies that normal intervals have asymptotically correct or conservative coverage. Since the leverages $1/M$ and $1/(N-M)$ tend to zero, HR, HC1 and HC3 agree with HC2 to first order. This deduction is mine. Theorem 1 of Freedman (2008a) is a central limit theorem and does not itself state the coverage result.\n\nThe paper does not state these conditions, and they are needed, since Section 2 assumes only $1<M<N-1$. In a sequence of finite populations with $M=2$, which lies outside the cited regime, the standardized $\\hat\\beta$ has a standard normal limit, the nominal 95\\% interval has limiting coverage of about 69.97\\%, and HR and HC3 equal $1/2$ and $2$ times HC2. Theorems 1--3 and Corollary 1 are finite-sample statements and do not use these sentences."}, "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-E001", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2011.10.024", "title": "On Equivalencies Between Design-Based and Regression-Based Variance Estimators for Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P031/"}, "work_id": "P031"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop showing the statement of Theorem 2 and the first sentence of its proof. A red box marks the term sigma(y0, y1), printed without tildes, at the end of the chain sigma-squared(y0-tilde) = sigma-squared(y1-tilde) = sigma(y0, y1).", "caption": "Proof of Theorem 2, first sentence, p. 369 (PDF p. 5). The marked covariance is that of the original potential outcomes; the proof needs the covariance of the hypothesized outcomes.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E002.png"}, {"alt": "Crop of the three-line display that ends the proof of Theorem 2 and of the paragraph after it. Red boxes mark the summand e-hat squared, printed without the subscript i, in the first and second lines of the display, and the symbol V CE(beta-hat), printed without a hat, in the sentence ending 'carries over to V CE(beta-hat)'.", "caption": "Proof of Theorem 2, final display, and the sentence after the proof, p. 369 (PDF p. 5). The two marked summands lack the index $i$, and the marked estimator lacks its hat.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E002-b.png"}, {"alt": "Crop of the proof of Theorem 3. Red boxes mark V CE(beta-hat), printed without a hat, on the left of the first line of the display, and the two symbols V-hat(y1) and V-hat(y0) in the numerators of the second line.", "caption": "Proof of Theorem 3, p. 370 (PDF p. 6). The marked estimator lacks its hat, and the two marked symbols $\\hat V(y_1)$, $\\hat V(y_0)$ are defined nowhere in the paper.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E002-c.png"}], "formal_release": null, "formal_statements": [{"id": "P031-T7", "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-T7"}, {"id": "P031-T8", "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-T8"}, {"id": "P031-T9", "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-T9"}], "headline": "In the proofs of Theorems 2 and 3, the covariance $\\sigma(y_0,y_1)$ should read $\\sigma(\\tilde y_0,\\tilde y_1)$, two residual sums lack the index $i$, the constant-effects estimator twice lacks its hat, and $\\hat V(y_1)$, $\\hat V(y_0)$ should read $\\hat\\sigma^2(y_1)$, $\\hat\\sigma^2(y_0)$; both theorems hold as stated.", "id": "P031-E002", "location": "Proof of Theorem 2 and the sentence after it, p. 369 (PDF p. 5); proof of Theorem 3, p. 370 (PDF p. 6)", "pdf_page": 5, "printed_page": "369", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P031-T7", "P031-T8", "P031-T9"], "text": {"as_printed": "(a) Proof of Theorem 2, first sentence (p.~369): ``Proof. Since $\\tilde y_0=\\tilde y_1-\\hat\\beta$, $\\sigma^2(\\tilde y_0)=\\sigma^2(\\tilde y_1)=\\sigma(\\hl{y_0},\\hl{y_1})$. Thus, substituting this into expression (5), randomization yields,''\n\\[\n\\hat V_{CE}(\\hat\\beta)=\\frac{N}{N-1}\\,\\frac{N}{M(N-M)}\\,\\sigma^2(\\tilde y_0).\n\\]\n\n(b) Proof of Theorem 2, final display (p.~369):\n\\[\n\\begin{aligned}\n\\frac{N-2}{N-1}\\hat V_{OLS}(\\hat\\beta)\n&=\\frac{N-2}{N-1}\\,\\frac{\\sum_{i=1}^{N}\\hl{\\hat e^{2}}}{N-2}\\,\\frac{N}{M(N-M)}\\\\\n&=\\frac{N}{N-1}\\,\\frac{\\sum_{i=1}^{N}\\hl{\\hat e^{2}}}{N}\\,\\frac{N}{M(N-M)}\\\\\n&=\\frac{N}{N-1}\\,\\frac{N}{M(N-M)}\\,\\sigma^2(\\tilde y_0)=\\hat V_{CE}(\\hat\\beta).\n\\end{aligned}\n\\]\n\n(c) Sentence after the proof (p.~369): ``As a result, the fact noted above that when $N\\ne 2M$, $\\hat V_{OLS}(\\hat\\beta)$ may be either too large or too small relative to $V(\\hat\\beta)$ carries over to $\\hl{V}_{CE}(\\hat\\beta)$.''\n\n(d) Proof of Theorem 3 (p.~370): ``With $N=2M$ and by Theorems 1 and 2,''\n\\[\n\\begin{aligned}\n\\frac{N-1}{N-2}\\hl{V}_{CE}(\\hat\\beta)\n&=\\hat V_{OLS}(\\hat\\beta)\n=\\frac{\\sum_{i=1}^{N}\\hat e_i^{2}}{2(M-1)}\\,\\frac{2M}{M^{2}}\n=\\frac{\\dfrac{\\sum_{i=1}^{M}\\hat e_i^{2}}{M-1}}{M}+\\frac{\\dfrac{\\sum_{i=M+1}^{N}\\hat e_i^{2}}{M-1}}{M}\\\\\n&=\\frac{\\hl{\\hat V}(y_1)}{M}+\\frac{\\hl{\\hat V}(y_0)}{M}=\\hat V_N(\\hat\\beta)=\\hat V_{HC2}(\\hat\\beta).\n\\end{aligned}\n\\]", "corrected": "(a) ``Since $\\tilde y_0=\\tilde y_1-\\hat\\beta$, $\\sigma^2(\\tilde y_0)=\\sigma^2(\\tilde y_1)=\\sigma(\\hl{\\tilde y_0},\\hl{\\tilde y_1})$.''\n\n(b)\n\\[\n\\begin{aligned}\n\\frac{N-2}{N-1}\\hat V_{OLS}(\\hat\\beta)\n&=\\frac{N-2}{N-1}\\,\\frac{\\sum_{i=1}^{N}\\hl{\\hat e_i^{2}}}{N-2}\\,\\frac{N}{M(N-M)}\\\\\n&=\\frac{N}{N-1}\\,\\frac{\\sum_{i=1}^{N}\\hl{\\hat e_i^{2}}}{N}\\,\\frac{N}{M(N-M)}\\\\\n&=\\frac{N}{N-1}\\,\\frac{N}{M(N-M)}\\,\\sigma^2(\\tilde y_0)=\\hat V_{CE}(\\hat\\beta).\n\\end{aligned}\n\\]\n\n(c) ``\\ldots carries over to $\\hl{\\hat V}_{CE}(\\hat\\beta)$.''\n\n(d)\n\\[\n\\begin{aligned}\n\\frac{N-1}{N-2}\\hl{\\hat V}_{CE}(\\hat\\beta)\n&=\\hat V_{OLS}(\\hat\\beta)\n=\\frac{\\sum_{i=1}^{N}\\hat e_i^{2}}{2(M-1)}\\,\\frac{2M}{M^{2}}\n=\\frac{\\dfrac{\\sum_{i=1}^{M}\\hat e_i^{2}}{M-1}}{M}+\\frac{\\dfrac{\\sum_{i=M+1}^{N}\\hat e_i^{2}}{M-1}}{M}\\\\\n&=\\frac{\\hl{\\hat\\sigma^2}(y_1)}{M}+\\frac{\\hl{\\hat\\sigma^2}(y_0)}{M}=\\hat V_N(\\hat\\beta)=\\hat V_{HC2}(\\hat\\beta).\n\\end{aligned}\n\\]\n\nThe corrected identities hold for arbitrary observed outcomes. Let $\\bar Y_T$ and $\\bar Y_C$ denote the means of the observed outcomes in the treatment and control groups, and set $S=\\sum_{i=1}^{N}\\hat e_i^{2}$. By (4), I have $\\hat\\alpha=\\bar Y_C$ and $\\hat\\beta=\\bar Y_T-\\bar Y_C$. Then $\\hat e_i=Y_i-\\bar Y_T$ for $i\\le M$ and $\\hat e_i=Y_i-\\bar Y_C$ for $i>M$. By the definitions of the hypothesized outcomes on p.~368, I have\n\\[\n\\tilde y_{0i}=Y_i-X_i\\hat\\beta=\\hat e_i+\\bar Y_C,\\qquad \\tilde y_{1i}=\\tilde y_{0i}+\\hat\\beta=\\hat e_i+\\bar Y_T.\n\\]\nThe residuals sum to zero within each group. The population means of $\\tilde y_0$ and $\\tilde y_1$ are therefore $\\bar Y_C$ and $\\bar Y_T$, and both centered vectors equal $(\\hat e_i)_{i=1}^{N}$. Using the definitions of the population variance and covariance on p.~366, I obtain\n\\[\n\\sigma^2(\\tilde y_0)=\\sigma^2(\\tilde y_1)=\\sigma(\\tilde y_0,\\tilde y_1)=\\frac{S}{N}.\n\\]\nThis is the corrected sentence (a), and it shows that the indexed sum in (b) equals $N\\sigma^2(\\tilde y_0)$. Substituting it into (5), I find\n\\[\n\\hat V_{CE}(\\hat\\beta)=\\frac{N}{N-1}\\left(\\frac{S}{NM}+\\frac{S}{N(N-M)}\\right)+\\frac{1}{N-1}\\left(\\frac{2S}{N}-\\frac{S}{N}-\\frac{S}{N}\\right)=\\frac{N}{N-1}\\,\\frac{S}{M(N-M)}.\n\\]\nSince the $[2,2]$ element of $(Z'Z)^{-1}$ is $N/(M(N-M))$, I have $\\hat V_{OLS}(\\hat\\beta)=\\frac{S}{N-2}\\,\\frac{N}{M(N-M)}$. Multiplying by $(N-2)/(N-1)$ gives $\\hat V_{CE}(\\hat\\beta)$, which is Theorem 2.\n\nFor Theorem 3, suppose that $N=2M$, so that $N-M=M$ and $N-2=2(M-1)$. By Theorem 2 and the formula for $\\hat V_{OLS}(\\hat\\beta)$, I have\n\\[\n\\frac{N-1}{N-2}\\hat V_{CE}(\\hat\\beta)=\\hat V_{OLS}(\\hat\\beta)=\\frac{S}{2(M-1)}\\,\\frac{2M}{M^{2}}=\\frac{1}{M}\\,\\frac{\\sum_{i=1}^{M}\\hat e_i^{2}}{M-1}+\\frac{1}{M}\\,\\frac{\\sum_{i=M+1}^{N}\\hat e_i^{2}}{M-1}.\n\\]\nBy the proof of Theorem 1, the two quotients on the right are $\\hat\\sigma^2(y_1)$ and $\\hat\\sigma^2(y_0)$. The right side is therefore $\\hat V_N(\\hat\\beta)$, and Theorem 1 implies that it equals $\\hat V_{HC2}(\\hat\\beta)$.", "evidence": "This correction rests on a check of the pages, an exact counterexample to the equality in (a) as printed, and a derivation of the corrected identities, given below. For the counterexample, let $N=4$, $M=2$ and $y_0=y_1=(0,2,5,9)$, and suppose that units 1 and 2 are treated. Then $Y=(0,2,5,9)$ and $\\hat\\beta=1-7=-6$, and the hypothesized outcomes are $\\tilde y_0=(6,8,5,9)$ and $\\tilde y_1=(0,2,-1,3)$. A direct computation gives $\\sigma^2(\\tilde y_0)=\\sigma^2(\\tilde y_1)=\\sigma(\\tilde y_0,\\tilde y_1)=5/2$ and $\\sigma(y_0,y_1)=23/2$. In the same example $\\hat V_{CE}(\\hat\\beta)=10/3$ and $\\hat V_{OLS}(\\hat\\beta)=\\hat V_N(\\hat\\beta)=\\hat V_{HC2}(\\hat\\beta)=5$, as Theorems 2 and 3 require.", "problem": "There are four kinds of misprint, and one of them makes an equality false as printed.\n\n(a) Since the hypothesized potential outcomes satisfy $\\tilde y_{1i}=\\tilde y_{0i}+\\hat\\beta$ for every unit, their two variances equal their covariance $\\sigma(\\tilde y_0,\\tilde y_1)$. This is the covariance that appears in expression (5). The covariance $\\sigma(y_0,y_1)$ of the original potential outcomes is a different quantity, and the equality with it does not hold in general, even when the true effects are constant.\n\n(b) The summand $\\hat e^{2}$ has no index under a sum over $i$, although the residuals are defined as $\\hat e_i$ on p.~367 and no scalar $\\hat e$ is defined.\n\n(c) The symbol $V_{CE}(\\hat\\beta)$ lacks the hat of the estimator $\\hat V_{CE}(\\hat\\beta)$ defined in (5), and in the paper an unhatted $V$ denotes the true variance of equation (3).\n\n(d) The symbols $\\hat V(y_1)$ and $\\hat V(y_0)$ are not defined, and the quantities meant are the sample variances $\\hat\\sigma^2(y_1)$ and $\\hat\\sigma^2(y_0)$ of p.~366.", "what_survives": "Theorems 2 and 3 hold as stated for all observed outcomes when $1<M<N-1$, and no theorem statement changes. Theorem 1 and Corollary 1 do not involve these symbols."}, "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-E002", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2011.10.024", "title": "On Equivalencies Between Design-Based and Regression-Based Variance Estimators for Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P031/"}, "work_id": "P031"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of footnotes 3 and 4 at the bottom of the page. A red box marks the phrase 'the limiting values of the control group mean and average treatment effect' in the second line of footnote 3.", "caption": "Footnote 3, p. 369 (PDF p. 5). The marked phrase names two means as the quantities on which the relation depends.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P031-T11", "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-T11"}], "headline": "The asymptotic relation between $\\hat V_{OLS}(\\hat\\beta)$ and $V(\\hat\\beta)$ depends on the treatment fraction and on the limiting variances and covariance of the potential outcomes, not on the control group mean and the average treatment effect.", "id": "P031-E003", "location": "Footnote 3, p. 369 (PDF p. 5)", "pdf_page": 5, "printed_page": "369", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P031-T11"], "text": {"as_printed": "``Freedman (2008b, p. 190) gives the exact asymptotic relation between $\\hat V_{OLS}(\\hat\\beta)$ and $V(\\hat\\beta)$, which depends on the size of the treatment versus control group and \\hl{the limiting values of the control group mean and average treatment effect}.''", "corrected": "``Freedman (2008b, p. 190) gives the exact asymptotic relation between $\\hat V_{OLS}(\\hat\\beta)$ and $V(\\hat\\beta)$, which depends on the size of the treatment versus control group and \\hl{the limiting variances and covariance of the treatment and control potential outcomes}.''\n\nThe relation is as follows. Suppose that, along a sequence of finite populations, $M/N\\to p\\in(0,1)$, $\\sigma^2(y_1)\\to v_1$, $\\sigma^2(y_0)\\to v_0$ and $\\sigma(y_1,y_0)\\to c_{10}$. Then\n\\[\n\\lim N\\,V(\\hat\\beta)=\\frac{1-p}{p}\\,v_1+\\frac{p}{1-p}\\,v_0+2c_{10},\n\\qquad\n\\lim N\\,E\\bigl[\\hat V_{OLS}(\\hat\\beta)\\bigr]=\\frac{v_1}{1-p}+\\frac{v_0}{p}.\n\\]\nIf the sample second moments also converge, which holds when the potential outcomes are uniformly bounded, then $N\\hat V_{OLS}(\\hat\\beta)$ converges in probability to the second limit. The second limit minus the first is\n\\[\n\\frac{(2p-1)(v_1-v_0)}{p(1-p)}+v_1+v_0-2c_{10},\n\\]\nwhere $v_1+v_0-2c_{10}$ is the limiting variance of the unit-level treatment effects. No mean of the potential outcomes appears.\n\nI first show that neither variance depends on the two means. Let $\\bar Y_T$ and $\\bar Y_C$ denote the means of the observed outcomes in the treatment and control groups, and set $S=\\sum_{i=1}^{N}\\hat e_i^{2}$. By (4), the residual $\\hat e_i$ equals $Y_i-\\bar Y_T$ for a treated unit and $Y_i-\\bar Y_C$ for a control unit, and\n\\[\n\\hat V_{OLS}(\\hat\\beta)=\\frac{S}{N-2}\\left(\\frac{1}{M}+\\frac{1}{N-M}\\right).\n\\]\nFix constants $c$ and $d$, and replace $y_{0i}$ by $y_{0i}+c$ and $y_{1i}$ by $y_{1i}+c+d$ for every $i$. For a given assignment, $Y_i$ becomes $Y_i+c+dX_i$. Then $\\bar Y_C$ increases by $c$ and $\\bar Y_T$ by $c+d$. Every residual is therefore unchanged, and I conclude that $S$ and $\\hat V_{OLS}(\\hat\\beta)$ are unchanged for every assignment. Further, $\\hat\\beta$ becomes $\\hat\\beta+d$, so $V(\\hat\\beta)$ is unchanged. The control group mean $\\bar y_0$ has moved by $c$ and the average treatment effect $\\beta$ by $d$.\n\nNext, I compute the two limits. Collecting the coefficients of $\\sigma^2(y_1)$ and $\\sigma^2(y_0)$ in (3), I obtain\n\\[\nV(\\hat\\beta)=\\frac{1}{N-1}\\left(\\frac{N-M}{M}\\,\\sigma^2(y_1)+\\frac{M}{N-M}\\,\\sigma^2(y_0)+2\\sigma(y_1,y_0)\\right).\n\\]\nLet $S_T$ and $S_C$ denote the sums of $\\hat e_i^{2}$ over the treated and the control units, so that $S=S_T+S_C$, and let $\\bar y_{1,T}$ denote the mean of $y_1$ over the treated units. I have $S_T=\\sum_{i:X_i=1}(y_{1i}-\\bar y_1)^2-M(\\bar y_{1,T}-\\bar y_1)^2$ and $\\operatorname{Var}(\\bar y_{1,T})=\\frac{N-M}{M(N-1)}\\,\\sigma^2(y_1)$. Taking expectations over the assignments, and arguing in the same way for the control group, I find\n\\[\nE[S_T]=\\frac{N(M-1)}{N-1}\\,\\sigma^2(y_1),\\qquad E[S_C]=\\frac{N(N-M-1)}{N-1}\\,\\sigma^2(y_0).\n\\]\nInserting these into the formula for $\\hat V_{OLS}(\\hat\\beta)$, I obtain\n\\[\nE\\bigl[\\hat V_{OLS}(\\hat\\beta)\\bigr]=\\frac{N}{(N-1)(N-2)}\\Bigl[(M-1)\\,\\sigma^2(y_1)+(N-M-1)\\,\\sigma^2(y_0)\\Bigr]\\left(\\frac{1}{M}+\\frac{1}{N-M}\\right).\n\\]\nMultiplying the expressions for $V(\\hat\\beta)$ and $E[\\hat V_{OLS}(\\hat\\beta)]$ by $N$ and letting $N\\to\\infty$ with $M/N\\to p$ gives the two limits stated above.\n\nFor the sequence in the table, $\\sigma^2(y_0)=1$, $\\sigma^2(y_1)=a^2$ and $\\sigma(y_1,y_0)=a$ for every $k$, and $p=1/4$. The limits are $\\lim N\\,V(\\hat\\beta)=3a^2+2a+1/3$ and $\\operatorname{plim}N\\hat V_{OLS}(\\hat\\beta)=4a^2/3+4$. In this sequence the convergence in probability can be seen directly. With $r$ the mean of $z$ over the treated units, I have $N\\hat V_{OLS}(\\hat\\beta)=\\frac{N}{N-2}\\bigl[4a^2/3+4-(4a^2/3+4/9)\\,r^2\\bigr]$ and $E[r^2]=3/(N-1)$, and Markov's inequality implies that $r^2\\to 0$ in probability.", "evidence": "This correction rests on a derivation, given below, and on an exact counterexample. For each integer $k\\ge 2$ let $N=4k$ and $M=k$, let $z_i=-1$ for $i\\le 2k$ and $z_i=1$ for $i>2k$, and set $y_{0i}=z_i$ and $y_{1i}=az_i+2$ for a fixed $a>0$. The outcomes are bounded, and for every $k$ and every $a$ the control group mean is $0$ and the average treatment effect is $2$. The limiting ratio of $\\hat V_{OLS}(\\hat\\beta)$ to $V(\\hat\\beta)$ is $1$, $4/7$ or $52/25$ for $a=1$, $2$ or $1/2$.\n\\begin{audittable}{llllll}{Limits for the sequence above with treatment fraction $1/4$. The two means are the same in every row.}\n$a$ & Control group mean & Average treatment effect & $\\operatorname{plim} N\\hat V_{OLS}(\\hat\\beta)$ & $\\lim N\\,V(\\hat\\beta)$ & Ratio \\\\\n$1$ & $0$ & $2$ & $16/3$ & $16/3$ & $1$ \\\\\n$2$ & $0$ & $2$ & $28/3$ & $49/3$ & $4/7$ \\\\\n$1/2$ & $0$ & $2$ & $13/3$ & $25/12$ & $52/25$ \\\\\n\\end{audittable}\nExact enumeration of all assignments for $N=8$ and $N=12$ agrees with the finite-sample formulas below. In each of the 744 cases enumerated (every assignment, for the three values of $a$), the shift with $c=7$ and $d=-5$ leaves $\\hat V_{OLS}(\\hat\\beta)$ unchanged. I did not read p.~190 of Freedman (2008b), and I do not assess the attribution to that page here.", "problem": "The footnote names two first moments, the control group mean and the average treatment effect, as the features of the outcomes that determine how $\\hat V_{OLS}(\\hat\\beta)$ compares with $V(\\hat\\beta)$ in large samples. Neither variance depends on them. If a constant $c$ is added to every $y_{0i}$ and $c+d$ to every $y_{1i}$, then the control group mean changes by $c$ and the average treatment effect by $d$, while $\\hat V_{OLS}(\\hat\\beta)$, for every assignment, and $V(\\hat\\beta)$ do not change. The comparison depends on the treatment fraction and on second moments: the limiting variances of the two potential outcomes and their limiting covariance. The sentence that the footnote supports, that $\\hat V_{OLS}(\\hat\\beta)$ may be too large or too small when $N\\ne 2M$, is correct.", "what_survives": "Theorem 2 and its scaling are untouched. The example confirms the warning that the OLS estimator can overstate or understate the variance, and the warning carries over to the constant-effects estimator because $(N-2)/(N-1)\\to 1$. No estimator, theorem or number in the paper is derived from the footnote."}, "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-E003", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2011.10.024", "title": "On Equivalencies Between Design-Based and Regression-Based Variance Estimators for Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P031/"}, "work_id": "P031"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of footnote 1 at the bottom of the page, with the line of text it is attached to. A red box marks the word 'Schwartz' in 'Cauchy-Schwartz inequality'.", "caption": "Footnote 1, p. 366 (PDF p. 2). The marked name is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E004.png"}, {"alt": "Crop of the reference list from the Cochran entry to the Rosenbaum 2009 entry. Red boxes mark 'ed..' in the Cochran (1977) entry, 'ed..' in the Freedman, Pisani and Purves (1998) entry, the word 'heteroskedastiticy' in the MacKinnon and White (1985) entry, and 'ed..' in the Rosenbaum (2002b) entry.", "caption": "References, p. 370 (PDF p. 6). The marked word in the MacKinnon and White entry is misspelled, and each of the three marked abbreviations “ed.” is followed by a second full stop.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-E004-b.png"}], "formal_release": null, "formal_statements": [], "headline": "“Cauchy–Schwartz” should read “Cauchy–Schwarz”, “heteroskedastiticy” should read “heteroskedasticity”, and “ed..” should read “ed.” in three reference entries.", "id": "P031-E004", "location": "Footnote 1, p. 366 (PDF p. 2); References, p. 370 (PDF p. 6)", "pdf_page": 2, "printed_page": "366", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item Footnote 1: ``This is due to $\\sigma^2(y_0)+\\sigma^2(y_1)-2\\sqrt{\\sigma^2(y_0)}\\sqrt{\\sigma^2(y_1)}=(\\sqrt{\\sigma^2(y_0)}-\\sqrt{\\sigma^2(y_1)})^2\\geq 0$, and $2\\sqrt{\\sigma^2(y_0)}\\sqrt{\\sigma^2(y_1)}\\geq 2\\sigma(y_0,y_1)$ by the Cauchy--\\hl{Schwartz} inequality.''\n\\item ``MacKinnon, J.G., White, H., 1985. Some \\hl{heteroskedastiticy}-consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics 29, 305--325.''\n\\item ``Cochran, W.G., 1977. Sampling Techniques, third \\hl{ed..} John Wiley, New York.''\n\\item ``Freedman, D., Pisani, R., Purves, R., 1998. Statistics, third \\hl{ed..} W.W. Norton, New York.''\n\\item ``Rosenbaum, P.R., 2002b. Observational Studies, second \\hl{ed..} Springer, New York.''\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Footnote 1: ``\\ldots by the Cauchy--\\hl{Schwarz} inequality.''\n\\item ``MacKinnon, J.G., White, H., 1985. Some \\hl{heteroskedasticity}-consistent covariance matrix estimators \\ldots''\n\\item ``Sampling Techniques, third \\hl{ed.} John Wiley, New York.''\n\\item ``Statistics, third \\hl{ed.} W.W. Norton, New York.''\n\\item ``Observational Studies, second \\hl{ed.} Springer, New York.''\n\\end{itemize}", "evidence": "This is a check of the page. The inequality in footnote 1 is the Cauchy--Schwarz bound on a covariance and holds as printed, and the correct spelling ``heteroskedasticity-consistent'' appears in the entry for White (1980) on the same page.", "problem": "The name of the inequality and one word of a reference title are misspelled, and in three entries ``ed.'' is followed by a second full stop. The meaning is clear in each case.", "what_survives": "Everything else is unaffected, since no statement of the paper depends on these words."}, "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-E004", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2011.10.024", "title": "On Equivalencies Between Design-Based and Regression-Based Variance Estimators for Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P031/"}, "work_id": "P031"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 1 to 9 of the function omni.ate in monospaced type with line numbers. A red box marks line 6, 'prob is assigned genprob(perms)', and line 7, 'ate is assigned estate(Y,Z,prob=prob)'.", "caption": "Package ri, file R/omni.ate.R, lines 1–9, typeset from the source file with line numbers added. The marked lines estimate the assignment probabilities from the permutation matrix and use them as weights in the point estimate.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-N002.png"}, {"alt": "Listing of lines 19 to 22 of the function omni.ate in monospaced type with line numbers. Red boxes mark line 20, 'distA is assigned gendist(YsA,perms) inside suppressWarnings', and line 22, 'se is assigned Pse(distA)'.", "caption": "Package ri, file R/omni.ate.R, lines 19–22, typeset from the source file with line numbers added. The marked lines compute the standard error as the dispersion of the generated distribution.", "url": "https://pmaronow.github.io/llm-audit/works/P031/exhibits/P031-N002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "With the default approximate permutation matrix under complete randomization, omni.ate weights units by sampled assignment frequencies, so its point estimate and standard error can differ from the difference in means of equation (2) and the square root of expression (5).", "id": "P031-N002", "location": "R package ri (version 0.9), function omni.ate, file R/omni.ate.R, lines 6–7 and 20–22; no passage of the article", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "File \\texttt{R/omni.ate.R}, lines 6 and 7:\n\n\\texttt{prob <- \\hl{genprob(perms)}}\n\n\\texttt{ate <- estate(Y,Z,\\hl{prob=prob})}\n\nLines 20 and 22:\n\n\\texttt{suppressWarnings(distA <- gendist(YsA,perms))}\n\n\\texttt{se <- Pse(distA)}", "corrected": "Under complete randomization the probability of treatment is known to equal $\\hl{M/N}$ for every unit, and \\texttt{genprobexact(Z)} returns it. If \\texttt{omni.ate} passes these probabilities to \\texttt{estate} and to both calls of \\texttt{gendist}, its point estimate is the difference in means $\\hat\\beta$ of equation (2). The standard error computed from a sampled distribution remains approximate. The standard error of the paper is the \\hl{closed form}\n\\[\n\\sqrt{\\frac{N}{N-1}\\,\\frac{N}{M(N-M)}\\,\\sigma^2(\\tilde y_0)},\\qquad \\tilde y_{0i}=Y_i-X_i\\hat\\beta .\n\\]\nProbabilities supplied by the user for other designs should be kept.", "evidence": "This correction rests on an exact counterexample and on runs of the unmodified package. For $N=16$, $M=8$, $Y=(1,0,\\ldots,0)$ and units 1--8 treated, the difference in means of equation (2) is $1/8$ and the variance of expression (5) is $7/480$. One admissible sample of 10,000 assignments makes the package's estimate exactly $1/5$, and two runs with fixed seeds return about $0.1272$ and $0.1263$. With all 12,870 assignments enumerated, the package returns $1/8$ and, to floating-point accuracy, a squared standard error of $7/480$.", "problem": "Under complete randomization, when a design has more than 10,000 possible assignments and \\texttt{maxiter} is left at its default, \\texttt{genperms} announces an approximate method and returns 10,000 random assignments. The function \\texttt{omni.ate} then estimates each unit's probability of treatment from those draws and uses the estimates as inverse-probability weights, although under complete randomization the probability is known to be $M/N$. The reported \\texttt{ate} can then differ from the difference in means of equation (2) and can change from call to call for the same data. The reported \\texttt{se} is the dispersion of the sampled distribution, not the square root of expression (5).", "what_survives": "The equations of the paper and Theorem 2 are untouched, and no number in the article comes from the package. Under complete randomization, with all assignments enumerated, the package agrees with equation (2) and expression (5)."}, "url": "https://pmaronow.github.io/llm-audit/works/P031/#P031-N002", "work": {"arxiv_id": null, "doi": "10.1016/j.spl.2011.10.024", "title": "On Equivalencies Between Design-Based and Regression-Based Variance Estimators for Randomized Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P031/"}, "work_id": "P031"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "A paragraph of ten lines from the Introduction. A red box in the fourth line surrounds the words 'DCRSEs will still underestimate uncertainty'. A second red box at the end of the seventh line surrounds 'a lower bound', in the sentence 'The reanalysis we present therefore offers a lower bound on the severity of the consequences of inadequate clustering practices in previous research'.", "caption": "Introduction, paragraph beginning “Note, however”, p. 187 (PDF p. 2). The marks are on the statement that DCRSEs “will still underestimate uncertainty” and on “a lower bound”.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E001.png"}, {"alt": "Two groups of lines from one paragraph, separated by a dashed grey rule. In the first group a red box surrounds 'still improve' in the sentence 'DCRSEs are not sufficient, although still improve over the naive approach or clustering by repeated dyad approach, when there are interdependencies throughout the entire system'. In the second group a red box surrounds 'we expect the variance of least squares estimates to be larger', which follows 'Aronow et al. (2018) developed conservative estimators for the variance of least squares estimators in such cases where there is further dependence, and in such cases'.", "caption": "Section 3, paragraph beginning “Note that the above approach”, p. 191 (PDF p. 6): its first four lines and its last six, with the four lines between them left out at the dashed rule. The marks are on “still improve” and on the expectation of a larger variance.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E001-p191.png"}, {"alt": "The six lines of the first paragraph of the Conclusion. A red box in the fifth line surrounds 'a lower bound', in the phrase 'thereby revealing a lower bound on the severity of the consequences of inadequate clustering practices in previous quantitative dyadic research'.", "caption": "Conclusion, first paragraph, p. 196 (PDF p. 11). The mark is on “a lower bound”.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E001-p196.png"}, {"alt": "A line chart. The horizontal axis is the number of independent blocks of dyads, m, from 1 to 100 on a logarithmic scale. The vertical axis is the expected DCR variance estimate divided by the true variance, from 0.5 to 1.8. A horizontal line at 1 is labelled 'ratio 1: the estimate equals the true variance on average', and the region below it is shaded and labelled 'below 1: uncertainty is underestimated, the direction stated on p. 187'. A blue curve with circles, labelled 'Example 4: six dyads among four members', starts at 2/3 for m = 1, crosses 1 before m = 2, where it is 13/12, passes 17/12 at m = 10 and approaches a dashed line labelled 'limit 3/2'. An orange curve with squares, labelled 'Example 5: dyads (A,B), (A,C), (D,E), (D,F)', starts at 7/6 for m = 1, is 17/12 at m = 2 and approaches a dashed line labelled 'limit 5/3'. The title reads 'Negative dependence between non-incident dyads: the DCR variance estimate settles above the true variance in both examples'.", "caption": "Expected value of the DCR variance estimator, without a small-sample correction, divided by the true variance of the estimated intercept, for $m$ independent blocks of Example 4 (circles, curve $3/2-5/(6m)$) and of Example 5 (squares, curve $(10m-3)/(6m)$). The markers are the values computed exactly for $m=1,2,10$ and for $m=1,2$, and the dashed lines are the limits $3/2$ and $5/3$. A ratio above 1 means that the DCR variance estimate is too large on average; both limits contradict the unconditional underestimation claim on p. 187.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E001-dcr-ratio.png"}], "formal_release": null, "formal_statements": [{"id": "P032-T1", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T1"}, {"id": "P032-T2", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T2"}, {"id": "P032-T5", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T5"}], "headline": "Under dependence between non-incident dyads, DCRSEs can overstate uncertainty as well as understate it, and the reanalysis is a lower bound on the consequences of inadequate clustering only under a sign condition that the article does not state.", "id": "P032-E001", "location": "Introduction, paragraph beginning “Note, however”, p. 187 (PDF p. 2); Section 3, paragraph beginning “Note that the above approach”, p. 191 (PDF p. 6); Conclusion, first paragraph, p. 196 (PDF p. 11)", "pdf_page": 2, "printed_page": "187", "reaches": [{"effect": "same_error_restated", "effect_text": "a corresponding statement is printed there", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P018/", "where": null, "work_id": "P018"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P032-T1", "P032-T2", "P032-T5"], "text": {"as_printed": "\\emph{Page 187.} When there is dependence between non-incident dyads (i.e., dyads that do not share a common member), DCRSEs \\hl{will still underestimate} uncertainty, just as only clustering on repeated dyads \\hl{underestimates} uncertainty when there is dependence across incident dyads. DCRSEs should therefore not be considered a replacement for approaches to clustering that (appropriately) account for greater amounts of dependence in the data. The reanalysis we present therefore offers \\hl{a lower bound} on the severity of the consequences of inadequate clustering practices in previous research, or, in other words, reveals the extent to which dyadic clustering \\emph{alone} is jeopardizing the reliability of research.\n\n\\emph{Page 191.} DCRSEs are \\emph{not} sufficient---although \\hl{still improve} over the naive approach or clustering by repeated dyad approach---when there are interdependencies throughout the entire system (i.e., across non-incident dyads). \\ldots\\ Aronow \\emph{et al.} (2018) developed conservative estimators for the variance of least squares estimators in such cases where there is further dependence---and in such cases \\hl{we expect the variance of least squares estimates to be larger} than those that only take into account dyadic clustering.\n\n\\emph{Page 196.} We investigated the consequences of failing to account for dyadic clustering in previous empirical research by reanalyzing all quantitative dyadic analyses in \\emph{International Organization} published in a 6-year window, thereby revealing \\hl{a lower bound} on the severity of the consequences of inadequate clustering practices in previous quantitative dyadic research.", "corrected": "\\emph{Page 187.} When there is dependence between non-incident dyads (i.e., dyads that do not share a common member), DCRSEs \\hl{may} still underestimate uncertainty, just as only clustering on repeated dyads \\hl{may underestimate} uncertainty when there is dependence across incident dyads. \\hl{Non-incident dependence can also lower the sampling variance of a coefficient relative to the population dyadic sandwich value, so DCRSEs can overestimate uncertainty.} DCRSEs should therefore not be considered a replacement for approaches to clustering that (appropriately) account for greater amounts of dependence in the data. The reanalysis \\hl{compares DCRSEs with the standard errors from the original variance estimation strategies and documents the resulting changes in standard errors and statistical significance}.\n\n\\emph{The condition.} Let $X$ be a fixed $n\\times p$ design matrix of full column rank, let $Y=X\\beta+u$ with $\\mathbb{E}[u]=0$ and $\\mathbb{E}[uu^T]=\\Omega$, and let $\\widehat\\beta$ be the least squares estimator. Fix $c\\in\\mathbb{R}^p$ and set $h=X(X^TX)^{-1}c$. Let $d_r$ be the dyad of observation $r$, and let $\\Omega_D$ have entries $(\\Omega_D)_{rs}=\\Omega_{rs}$ if $d_r\\cap d_s\\neq\\varnothing$ and $(\\Omega_D)_{rs}=0$ otherwise. Then\n\\[\n\\operatorname{Var}(c^T\\widehat\\beta)-h^T\\Omega_Dh=\\sum_{(r,s):\\,d_r\\cap d_s=\\varnothing}h_rh_s\\Omega_{rs},\n\\]\nwhere the sum is over ordered pairs of observations. \\hl{If this sum is nonnegative}, then $h^T\\Omega_Dh\\le\\operatorname{Var}(c^T\\widehat\\beta)$. If it is positive, the inequality is strict. If it is negative, then $h^T\\Omega_Dh>\\operatorname{Var}(c^T\\widehat\\beta)$. The statement compares the variance of $c^T\\widehat\\beta$ with the population dyadic sandwich value. It does not order the estimates computed from residuals in every sample.\n\n\\emph{Page 191.} DCRSEs are \\emph{not} sufficient when there are interdependencies throughout the entire system (i.e., across non-incident dyads)\\hl{, and an improvement over the naive approach or clustering by repeated dyad approach is not guaranteed}. \\ldots\\ Aronow \\emph{et al.} (2018) developed conservative estimators for the variance of least squares estimators in such cases where there is further dependence---and in such cases the variance of least squares estimates \\hl{is larger than the population dyadic sandwich value when the sum above is positive for the coefficient, and smaller when it is negative}.\n\n\\emph{Page 196.} \\ldots\\ by reanalyzing all quantitative dyadic analyses in \\emph{International Organization} published in a 6-year window, thereby \\hl{documenting how standard errors and statistical significance change when DCRSEs are used. Without a justified sign for each coefficient's omitted weighted covariance sum, the reanalysis is not a lower bound on the consequences of all inadequately modeled dependence}.", "evidence": "The evidence is a derivation and six exact examples. Let $X$ be a fixed $n\\times p$ matrix of full column rank, and let $Y=X\\beta+u$ with $\\mathbb{E}[u]=0$ and $\\mathbb{E}[uu^T]=\\Omega$. Fix $c\\in\\mathbb{R}^p$ and set $h=X(X^TX)^{-1}c$. Since $\\widehat\\beta-\\beta=(X^TX)^{-1}X^Tu$, I have $c^T\\widehat\\beta-c^T\\beta=h^Tu$, and the true variance is $V=\\operatorname{Var}(c^T\\widehat\\beta)=h^T\\Omega h$. Let $d_r$ denote the dyad of observation $r$, and let $\\Omega_D$ impose the restriction of Table 4, with entries $(\\Omega_D)_{rs}=\\Omega_{rs}$ if $d_r\\cap d_s\\neq\\varnothing$ and $(\\Omega_D)_{rs}=0$ otherwise. The population dyadic sandwich value is $V_D=h^T\\Omega_Dh$. Subtracting, I obtain\n\\[\nV-V_D=h^T(\\Omega-\\Omega_D)h=\\sum_{(r,s):\\,d_r\\cap d_s=\\varnothing}h_rh_s\\Omega_{rs},\n\\]\nwhere the sum is over ordered pairs of observations. I denote this sum by $\\Delta$. This identity implies that $V_D\\le V$ if and only if $\\Delta\\ge0$. Note that positive definiteness of $\\Omega$ does not restrict the sign of $\\Delta$.\n\nExamples 1 to 3 have two observations, on two dyads with no common member. In Examples 1 and 3 the coefficient is the intercept, with $h=(1/2,1/2)^T$. In Example 2 it is the slope on the single regressor $x=(-1,1)^T$, with $h=(-1/2,1/2)^T$. Example 4 has one observation on each of the six dyads among four members, and Examples 5 and 6 have one observation on each of the dyads $(A,B)$, $(A,C)$, $(D,E)$ and $(D,F)$. In Examples 4 to 6 the coefficient is the intercept. Every error has variance 1. In Examples 1 to 5 the matrix $\\Omega$ is positive definite, and in Example 6 it is the covariance matrix of an explicit distribution.\n\n\\begin{audittable}{llrrr}{The true variance $V$, the variance $V_D$ with the covariances of non-incident dyads set to zero, and $\\Delta=V-V_D$.}\nExample & Covariances between observations & $V$ & $V_D$ & $\\Delta$ \\\\\n1 & non-incident $-1/2$ & $1/4$ & $1/2$ & $-1/4$ \\\\\n2 & non-incident $1/2$ & $1/4$ & $1/2$ & $-1/4$ \\\\\n3 & non-incident $1/2$ & $3/4$ & $1/2$ & $1/4$ \\\\\n4 & incident $1/8$, non-incident $-1/2$ & $1/6$ & $1/4$ & $-1/12$ \\\\\n5 & incident $1/4$, non-incident $-1/4$ & $3/16$ & $5/16$ & $-1/8$ \\\\\n6 & incident $1/4$, non-incident $1/4$ & $7/16$ & $5/16$ & $1/8$ \\\\\n\\end{audittable}\n\nExamples 1, 2, 4 and 5 have $V<V_D$. Example 2 has a positive covariance between the two errors, and Examples 4 and 5 have positive dependence between incident dyads. Examples 3 and 6 show that the other direction also occurs.\n\nThe fitted estimator also overstates variance in the limit. In Example 4 the errors can be built from independent signs and are bounded. For $m$ independent copies of this block on disjoint members, I have $n=6m$ and $V=1/n$, and the DCR variance estimator, computed from residuals without a small-sample correction, has expectation $3/(2n)-5/n^2$. The ratio of this expectation to $V$ is $3/2-5/n$, which equals $2/3$ for $m=1$, $13/12$ for $m=2$ and $17/12$ for $m=10$. By the law of large numbers, the ratio of the estimator to $V$ converges to $3/2$. For $m$ independent copies of Example 5, I have $V=3/(16m)$, and the estimator has expectation $(10m-3)/(32m^2)$, which exceeds $V$ for every $m\\ge1$. The ratio of the estimator to $V$ converges in probability to $5/3$. In the limit, DCRSEs exceed the standard deviation of the estimate by the factors $\\sqrt{3/2}$ and $\\sqrt{5/3}$. In Example 4 the variance with every covariance between distinct observations set to zero, as under the naive approach, equals $V$. In Example 5 it is $1/4$, which lies between $V=3/16$ and $V_D=5/16$.", "problem": "The passage on p.~187 states a direction: when non-incident dyads are dependent, DCRSEs underestimate uncertainty. From this direction the article infers that its reanalysis is a lower bound on the consequences of inadequate clustering, and the Conclusion repeats the inference. The direction does not hold in general. The DCR variance estimator treats as zero the covariances between observations whose dyads share no member. In the variance of an estimated coefficient, each omitted covariance is weighted by the product of the weights of its two observations, and the sum of these weighted terms can be positive, zero or negative. When the sum is negative, the true variance of the estimated coefficient is smaller than the population dyadic sandwich value, and DCRSEs can overstate uncertainty. The article states no condition on the sign of the sum. The condition in Section 2.1, that $a$ and $c$ are of the same sign, belongs to the toy example. A positive covariance between the errors of non-incident dyads is also not enough, since the weights of two observations in a slope can have opposite signs. Without a sign condition, the comparison of DCRSEs with the original standard errors measures the effect of dyadic clustering and bounds the effect of further dependence in neither direction. The expectation of a larger variance on p.~191 requires a positive omitted weighted covariance sum; the aside there that DCRSEs still improve over the naive approach is not guaranteed.", "what_survives": "The toy example of Section 2.1 with its condition on the signs of $a$ and $c$, the sandwich identity, the restriction of Table 4 and the decomposition on p.~191 are unaffected. The ratios of DCRSEs to the original standard errors, the comparisons of $p$-values and the software do not use the direction of the omitted dependence, and this finding changes none of them. I have not independently recalculated those reported values. The sentence in the third paragraph of the Conclusion, that the problem ``may be even more severe'', is correct as a statement of possibility (Examples 3 and 6). What changes is the interpretation. The reanalysis is not shown to be a lower bound. A population variance lower bound requires $\\Delta\\ge0$ for each coefficient, which has not been examined for the 22 studies. The result of Aronow, Crawford and Zubizarreta (2018) cited on p.~191 is not in question."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E001", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the paragraph that begins For individual studies, showing six complete lines and part of the next. A red outline marks the sentence: Three studies see half or more of their KEVs become insignificant at the 5 percent level upon reanalysis.", "caption": "p. 196 (PDF p. 11), Section 4.1, paragraph beginning “For individual studies”. The marked sentence gives the count as three.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E002.png"}, {"alt": "Crop of Table A.2 with one row per study. Red outlines mark the study names Colgan (2014), Efrat and Newman (2016), Fang, Johnson, and Leeds (2014) and Weisiger (2016), and their entries 0.889, 0.5, 1 and 0.607 in the column Sig. to Insig. Two further red outlines mark the word defering in the rows for Efrat and Newman (2016) and Pratt (2018).", "caption": "Supplementary Appendix, p. A3 (PDF p. 3), Table A.2, columns “Study” to “Sig.$\\to$Insig.”. The four studies with an entry of $0.5$ or more are marked, with their entries $0.889$, $0.5$, $1$ and $0.607$, and so is the word “defering” in two rows.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E002-tableA2.png"}], "formal_release": null, "formal_statements": [{"id": "P032-T10", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T10"}], "headline": "Four studies, not three, lose significance in half or more of their KEVs, and “deferring” is misspelled twice in Supplementary Table A.2.", "historical_evidence_type": "lean_proof", "id": "P032-E002", "location": "Section 4.1, p. 196 (PDF p. 11), paragraph beginning “For individual studies”; Supplementary Table A.2, Supplementary Appendix p. A3 (PDF p. 3), entries for Efrat and Newman (2016) and Pratt (2018)", "pdf_page": 11, "printed_page": "196", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P032-T10"], "text": {"as_printed": "\\emph{p.~196.} \\hl{Three} studies see half or more of their KEVs become insignificant at the 5\\% level upon reanalysis.\n\n\\emph{Table A.2, column ``Primary Relationship Tested''.} Efrat and Newman (2016): Likelihood of \\hl{defering} to another country's laws. Pratt (2018): Likelihood of \\hl{defering} to another IO.", "corrected": "\\emph{p.~196.} \\hl{Four} studies see half or more of their KEVs become insignificant at the 5\\% level upon reanalysis.\n\n\\emph{Table A.2.} Efrat and Newman (2016): Likelihood of \\hl{deferring} to another country's laws. Pratt (2018): Likelihood of \\hl{deferring} to another IO.\n\nAlternatively, the sentence on p.~196 keeps ``Three'' and reads ``more than half'' in place of ``half or more''.", "evidence": "The evidence is a recount from the column Sig.$\\to$Insig.\\ of Table A.2, where every other study is at $0.375$ or below. The four counts agree with the saved model outputs in the replication files. The spelling is a check of the page.", "problem": "Supplementary Table A.2 lists four studies in which the share of KEVs that go from significant to insignificant is one half or more: Colgan (2014) with $8/9$, Efrat and Newman (2016) with $2/4$, Fang, Johnson, and Leeds (2014) with $1/1$ and Weisiger (2016) with $17/28$. Three of them are strictly above one half.", "what_survives": "The numerical entries of Table A.2 and every aggregate are unchanged. The neighbouring counts in the paragraph, $18$ of $22$ studies with an average SER above one and $7$ of $22$ that lose no significance, agree with the table."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E002", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of text. In the second line a red box surrounds the link 'section 3', in the sentence 'With the same six observations, section 3 demonstrates the additional clustering permitted.' The fourth line begins the next paragraph with 'In Table 3, we can see that'.", "caption": "First paragraph below Table 4, p. 190 (PDF p. 5), with the first line of the next paragraph. The mark is on “section 3”.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E003.png"}], "formal_release": null, "formal_statements": [], "headline": "The cross-reference “section 3” should read “Table 3”.", "id": "P032-E003", "location": "First paragraph below Table 4, second sentence, p. 190 (PDF p. 5)", "pdf_page": 5, "printed_page": "190", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "With the same six observations, \\hl{section 3} demonstrates the additional clustering permitted.", "corrected": "With the same six observations, \\hl{Table 3} demonstrates the additional clustering permitted.", "evidence": "I checked the page. The next paragraph begins ``In Table 3'', and the parallel sentence on p.~189 reads ``Table 2 demonstrates''.", "problem": "The passage points to a section instead of the covariance table used in its comparison. The sentence stands in Section 3 and introduces the covariance structure assumed under clustering by repeated dyad, which Table 3 displays.", "what_survives": "The covariance comparison is unaffected by the wrong cross-reference."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E003", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of page 193, from 'To quantify the impact of neglecting dyadic clustering'. Two red boxes mark the words 'the standard error produced using the original variance estimation strategy', which run over two lines and end with the footnote mark 13.", "caption": "Section 4.1, p. 193 (PDF p. 8). The marked words define the denominator of the SER. For seven KEVs the command exports a standard error from the default variance of the estimation command, while the original studies used a robust one. The definition is correct as printed.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E004.png"}], "formal_release": null, "formal_statements": [], "headline": "In the Stata command, the branch that should fit the comparison model with a robust variance when rse is requested tests crse a second time and can never run; for seven KEVs in three models of Kinne (2018) and Weisiger (2016), the comparison standard error, its $p$-value and the denominator of the SER then come from the default variance of the estimation command and not from the robust variance of the original study, and line 321 should test rse.", "id": "P032-E004", "location": "Section 4.1, p. 193 (PDF p. 8), definition of the standard error ratio. Stata package dcr (commit 734ee6a) and its identical copy in the replication files: dcr.ado, lines 310–334, and the help entry for rse; calls on lines 352, 381 and 1199 of the replication script 1_reanalysis_stata.do. The code is not printed in the article.", "pdf_page": 8, "printed_page": "193", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~193: ``We compute a standard error ratio (SER) for all KEVs, which is the DCRSE divided by the standard error produced using the original variance estimation strategy.${}^{13}$''\n\nStata help: ``\\texttt{rse} specifies that a robust standard error (RSE) should be computed in addition to a DCRSE, e.g., for the purpose of calculating the ratio of DCRSE to RSE. This RSE is only output to the Excel spreadsheet created by \\texttt{xlsxpath}.''\n\n\\texttt{dcr.ado}, line 310 and lines 320--334. Lines 311--319 hold the three clustered fits of the first branch, omitted here.\n\n\\texttt{if (\"`crse'\"!=\"\") \\{}\n\n\\texttt{\\}}\n\n\\texttt{else if (\"\\hl{`crse'}\"!=\"\") \\{}\n\n\\texttt{if \"`syntax'\" == \"vce\" \\{}\n\n\\texttt{qui `prefix' `command\\_line' if `touse' `weight', `options' vce(robust)}\n\n\\texttt{\\}}\n\n\\texttt{if \"`syntax'\" == \"clusterOnly\" \\{}\n\n\\texttt{qui `prefix' `command\\_line' if `touse' `weight', `options'}\n\n\\texttt{\\}}\n\n\\texttt{if \"`syntax'\" == \"robustCluster\" \\{}\n\n\\texttt{qui `prefix' `command\\_line' if `touse' `weight', `options' robust}\n\n\\texttt{\\}}\n\n\\texttt{\\}}\n\n\\texttt{else \\{}\n\n\\texttt{qui `prefix' `command\\_line' if `touse' `weight', `options'}\n\n\\texttt{\\}}", "corrected": "\\texttt{dcr.ado}, line 321, with the three fits on lines 322--330 kept:\n\n\\texttt{else if (\"\\hl{`rse'}\"!=\"\") \\{}\n\nAfter this change, I would refit the comparison models of Kinne (2018), Figure 8, Model 1 and of Weisiger (2016), Table 3, Model 1 and Table 4, Model 1 in the primary and in the small-sample scripts, and regenerate the comparison standard errors, the $p$-values, the SERs and the tables and figures that use them. I give no corrected values, since they require these fits.", "evidence": "I checked the code and computed two exact examples. In \\texttt{dcr.ado} the word \\texttt{rse} occurs only on line 16, where the option is declared, and on line 36, where it is removed from the options passed to the estimation command. Lines 310 and 321 carry the same test.\n\nIn the example, $y=(0,0,1,3)$ is regressed on an intercept and $x=(-1,0,0,1)$, the four dyads share no member, and the command is called with \\texttt{rse} and \\texttt{dofundo(reglike)}. Then $\\widehat\\beta=(1,\\,3/2)$, the residuals are $(1/2,-1,0,1/2)$, and the three variance matrices are diagonal.\n\n\\begin{audittable}{lrr}{The exact example: the slope, as the command exports it and as \\texttt{rse} requests it.}\nQuantity & As exported & With the robust comparison \\\\\nComparison variance of the slope & $3/8$ (default) & $1/4$ (robust) \\\\\nDCR variance of the slope & $1/8$ & $1/8$ \\\\\nSquared SER & $1/3$ & $1/2$ \\\\\nSquared $z$ statistic of the comparison fit & $6$ & $9$ \\\\\n\\end{audittable}\n\nA second exact example is a logit model on the fifteen dyads among six members. Its slope has default variance $47/84$, robust variance $2045/5488$ and DCR variance $195/392$. The squared SER is $585/658<1$ as exported and $546/409>1$ with the robust comparison. In this example the defect moves the ratio from one side of one to the other.\n\nI computed both examples from the formulas of the command and of the estimation commands; I did not run Stata. The seven KEVs are listed below with the values that the command exported. They are not corrected values. I did not fit the three models again.\n\n\\begin{audittable}{llrrr}{The seven KEVs whose comparison standard error comes from the default variance: saved values of the primary reanalysis.}\nModel & KEV & Comparison SE & DCRSE & SER \\\\\nKinne (2018), Figure 8, Model 1 & \\texttt{l\\_DCA2paths} & 0.0682 & 0.0960 & 1.407 \\\\\nKinne (2018), Figure 8, Model 1 & \\texttt{l\\_DCAdegree} & 0.0664 & 0.1505 & 2.267 \\\\\nWeisiger (2016), Table 3, Model 1 & \\texttt{lndeadpop} & 0.0906 & 0.7135 & 7.872 \\\\\nWeisiger (2016), Table 3, Model 1 & \\texttt{pcturnsum} & 0.7262 & 1.4858 & 2.046 \\\\\nWeisiger (2016), Table 4, Model 1 & \\texttt{lndeadpop} & 0.0913 & 0.6853 & 7.505 \\\\\nWeisiger (2016), Table 4, Model 1 & \\texttt{pcncsum} & 0.9749 & 1.1480 & 1.178 \\\\\nWeisiger (2016), Table 4, Model 1 & \\texttt{pcnewcsum} & 1.1052 & 1.5973 & 1.445 \\\\\n\\end{audittable}", "problem": "Beside the DCRSE, the command exports a comparison standard error from a second fit of the model, and the ratio of the two is the SER of the reanalysis. Three branches choose the second fit. The first runs when \\texttt{crse} is given and clusters on the named variable. The second is meant to run when \\texttt{rse} is given and to use the robust variance of the estimation command. Its condition repeats that of the first. The branch is reached only when \\texttt{crse} is empty and entered only when \\texttt{crse} is not empty, which cannot happen. No other line tests \\texttt{rse}. A call with \\texttt{rse} and without \\texttt{crse} then falls to the third branch, which fits the model without adding a variance option. For the reanalysis calls, the comparison standard error, its $p$-value and the SER are then computed from the default variance of the estimation command. The reanalysis has nine such calls. Six are \\texttt{stcrreg} models, whose default variance is already robust, and for them the defect need not change anything. Three are models whose default variance is not robust and whose original fit asked for a robust one: the \\texttt{logit} model of Kinne (2018), Figure 8, Model 1, and the \\texttt{stcox} models of Weisiger (2016), Table 3, Model 1 and Table 4, Model 1. Their seven KEVs, two of the six of Kinne and five of the 28 of Weisiger, are compared with a standard error that does not come from the original variance estimation strategy.", "what_survives": "The definition of the SER on p.~193 is correct. The DCR variance is computed before this block and is unchanged by this repair, as are the coefficients, the DCRSEs and the $p$-values based on them. The R package, the calls with \\texttt{crse} and the calls with neither option are not affected. The comparison standard errors, their $p$-values and the SERs of the seven KEVs need to be recomputed; I have not measured their numerical changes. They enter the rows Kinne (2018) and Weisiger (2016) of Supplementary Table A.2, the 2016, 2018, Security and All rows of Table 5, Figures 1--3 and, through the small-sample scripts, Supplementary Table A.5. Under the inverse study frequency weighting the seven KEVs carry $(2/6+5/28)/22$, about 2\\%, of the total weight; without an upper bound on their corrected ratios, this gives no upper bound on the corrected mean SER. No rounded figure of the article, the 1.74, the 0.22 and the 67\\% among them, is shown to change. The DCRSEs of the Weisiger models are the subject of \\fid{P032-N004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E004", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of footnote 3 on page 191, four lines of small type, below the last line of footnote 2. A red box marks the words 'negative parameter estimate variances' in the first line. A second red box marks the words 'force positive semi-definiteness', with force in quotation marks, in the third line.", "caption": "Footnote 3, p. 191 (PDF p. 6). The first marked phrase is the case that the footnote addresses, a negative variance of a parameter estimate. The second is the result that the help text of the software promises whenever the option is set.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P032-T6", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T6"}, {"id": "P032-T7", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T7"}], "headline": "The option of the R and Stata packages that is documented to force the variance matrix to be positive semidefinite replaces negative eigenvalues by zero only when a diagonal entry of the matrix is negative, and it can return an indefinite matrix; the eigenvalue step should run whenever the option is set.", "id": "P032-E005", "location": "Footnote 3, p. 191 (PDF p. 6), and the software option that it describes. R package dcr (commit 06a80ad): R/dyad_funcs.R, lines 70 and 155, and the help entry for posdef. Stata package dcr (commit 734ee6a): dcr.ado, lines 246–282, and the help entry for psd. The code is not printed in the article.", "pdf_page": 6, "printed_page": "191", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P032-T6", "P032-T7"], "text": {"as_printed": "Footnote 3, p.~191:\n\nTo account for rare instances where negative parameter estimate variances arise due to non-positive semi-definite variance-covariance matrices, an ``eigendecomposition of the estimated variance matrix \\ldots\\ [that] converts any negative eigenvalue(s) to zero'' and then reassembles the matrix to ``force'' positive semi-definiteness is performed, in line with suggestions from Cameron et al. (2011) and Cameron and Miller (2014).\n\nR help, argument \\texttt{posdef} of the functions \\texttt{dcr} and \\texttt{dcr\\_sandwich}: ``A logical value indicating whether or not DCR variance-covariance matrix is forced to be positive semi-definite.''\n\n\\texttt{R/dyad\\_funcs.R}, lines 70 and 155:\n\n\\texttt{if (\\hl{sum(diag(V.hat) < 0) >= 1 \\&} posdef == TRUE) \\{}\n\nStata help: ``\\texttt{psd} forces the final variance-covariance matrix to be positive semi-definite.''\n\n\\texttt{dcr.ado}, line 268, inside the block that runs when \\texttt{psd} is set. Lines 252--258 set the scalar \\texttt{Indnegvar} to 1 only if a diagonal entry of the matrix is negative.\n\n\\texttt{\\hl{if (Indnegvar==1) \\{}}", "corrected": "When the option is set, the packages eigendecompose the symmetric estimated variance matrix \\hl{whatever the signs of its diagonal entries}, replace each negative eigenvalue by zero and reassemble the matrix. For $V=Q\\operatorname{diag}(\\lambda_1,\\dots,\\lambda_k)Q^\\top$ with $Q$ orthogonal, the option returns\n\\[\nV_+=Q\\operatorname{diag}\\bigl(\\max\\{\\lambda_1,0\\},\\dots,\\max\\{\\lambda_k,0\\}\\bigr)Q^\\top,\n\\]\nwhich is positive semidefinite.\n\n\\texttt{R/dyad\\_funcs.R}, lines 70 and 73, and likewise lines 155 and 158:\n\n\\texttt{if (posdef == TRUE) \\{}\n\n\\texttt{V.hat <- decomp\\$vectors \\%*\\% diag(pos\\_eigens\\hl{, nrow = length(pos\\_eigens)}) \\%*\\% t(decomp\\$vectors)}\n\nThe second argument of \\texttt{diag} keeps the formula right for a model with one parameter, where R reads a single number as a dimension.\n\n\\texttt{dcr.ado}: the test on line 268 is removed, so that lines 269--281 run whenever \\texttt{psd} is set.\n\nIn the example above, the corrected option returns approximately\n\\[\n\\begin{pmatrix}0.12609&-0.02150\\\\-0.02150&0.00367\\end{pmatrix}.\n\\]", "evidence": "The evidence is an exact counterexample, computed in rational arithmetic and reproduced by running both R functions.\n\nI regress $y$ on an intercept and $x$ by least squares on the ten dyads among five members, taken in the order $(1,2)$, $(1,3)$, $(2,3)$, $(1,4)$, $(2,4)$, $(3,4)$, $(1,5)$, $(2,5)$, $(3,5)$, $(4,5)$, with\n\\[\nx=(3,0,2,1,-1,1,-3,-2,3,-2),\\qquad y=(3,-2,-2,1,1,0,0,2,1,0).\n\\]\nThen\n\\[\nX^\\top X=\\begin{pmatrix}10&2\\\\2&42\\end{pmatrix},\\qquad \\widehat\\beta=(5/13,\\,1/13).\n\\]\nWith $N=5$, the decomposition of p.~191 gives, in the order (intercept, slope),\n\\[\n\\widehat V_r=\\frac{1}{3655808}\\begin{pmatrix}460765&-79681\\\\-79681&7157\\end{pmatrix}.\n\\]\nSince both diagonal entries are positive, neither package applies the eigenvalue step. The determinant of $\\widehat V_r$ is $-70529/308915776$, which implies that one eigenvalue is negative. For $c=(1,6)$, I have\n\\[\nc^\\top\\widehat V_r\\,c=\\frac{460765-12\\cdot 79681+36\\cdot 7157}{3655808}=-\\frac{237755}{3655808}<0,\n\\]\nand this is the estimated variance of the intercept plus six times the slope. Both functions of the R package, installed without modification, return this matrix when \\texttt{posdef = TRUE}, with eigenvalues $0.12975$ and $-0.00176$. For the Stata command, the same matrix follows from the source for the call \\texttt{dcr regress y x, dm1() dm2() dofundo(reglike) psd}, up to the order of the coefficients. Stata was not run.\n\nThe corrected option returns a positive semidefinite matrix. In the notation of the corrected statement below, with $q_j$ the columns of $Q$, I have $c^\\top V_+c=\\sum_j\\max\\{\\lambda_j,0\\}\\,(q_j^\\top c)^2\\ge 0$ for every vector $c$.", "problem": "A symmetric matrix whose diagonal entries are all positive can have a negative eigenvalue. Both packages run the eigenvalue step of footnote 3 only if some diagonal entry of the DCR variance matrix is negative. If the matrix is indefinite and its diagonal is positive, the step is skipped. The matrix that the option returns then assigns a negative variance to some linear combination of the coefficients. The help texts of both packages state without this restriction that the option forces the matrix to be positive semidefinite. Footnote 3 speaks of negative parameter variances, and the code agrees with it. The defect is in the guarantee of the software option.", "what_survives": "Footnote 3 needs no change. The decomposition and the coefficient estimates are unaffected. The corrected option can change individual DCRSEs when it now leaves an indefinite matrix unchanged. A matrix that is already positive semidefinite is unchanged. Where the option already applies the eigenvalue step, the spectral formula is unchanged. The reanalysis scripts set this option. I did not examine whether any of the 174 models has an indefinite matrix with a positive diagonal, and no published number is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E005", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last five lines of the Conclusion and the start of the Acknowledgements. A red outline marks the words: These open-source packages implement DCRSEs for all the models in the reanalysis sample (and more).", "caption": "p. 196 (PDF p. 11), last paragraph of the Conclusion. The marked clause presents the R and Stata packages as implementations of DCRSEs for the models of the reanalysis; the finding concerns the R package.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E006.png"}, {"alt": "Six four-by-four grids in two rows and three columns. Each cell holds a weight, plus one, zero or minus one, for a pair of observations. Top row, the estimator: in the first two columns, two blocks of plus ones for the dyads that share member A and the dyads that share member D, variance one half; in the third column, plus ones on the diagonal only, variance one quarter. Bottom row, the released code: in the first column the two off-diagonal weights for the dyads A,B and A,C are zero, variance three eighths; in the second column the plus ones link observations 1 and 3 and observations 2 and 4, variance zero; in the third column the off-diagonal weights between the first two dyads, whose labels collide, are minus one, variance one eighth. The cells that differ from the estimator are outlined.", "caption": "Three exact examples with four observations, responses $y=(1,1,-1,-1)$ and an intercept-only fit. The top row shows the weights on pairs of observations that the estimator defines, and the bottom row the weights that the released routines use on that input. The routines return the variances $3/8$, $0$ and $1/8$ where the estimator gives $1/2$, $1/2$ and $1/4$.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E006-pairs.png"}], "formal_release": null, "formal_statements": [], "headline": "On a subset or reordered data frame, or with member labels that contain underscores, the R routines dcr() and dcr_sandwich() can return a wrong covariance matrix without a warning, because they read row names as row positions and join the two member labels of a dyad with an underscore.", "id": "P032-E006", "location": "Section 5 (Conclusion), p. 196 (PDF p. 11), sentence beginning “These open-source packages”; R package dcr, file R/dyad_funcs.R, lines 34–35 and 111–112, and file R/helper_funcs.R, line 38", "pdf_page": 11, "printed_page": "196", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~196.} These open-source packages implement DCRSEs for all the models in the reanalysis sample (and more), and mirror syntax familiar to users of R and Stata.\n\n\\emph{\\texttt{R/dyad\\_funcs.R}, lines 34--35, in \\texttt{dcr()}; repeated at lines 111--112, in \\texttt{dcr\\_sandwich()}.}\n\n\\texttt{touse\\_obs <- \\hl{as.integer(rownames(model.frame(model)))}}\n\n\\texttt{touse\\_data <- data[touse\\_obs,]}\n\n\\emph{\\texttt{R/helper\\_funcs.R}, line 38, in \\texttt{create\\_dyadid()}.}\n\n\\texttt{dyad\\_id <- unlist(lapply(sorteddyads, function(x) as.factor(\\hl{paste0(x, collapse = \"\\_\")})), use.names = F)}", "corrected": "\\emph{\\texttt{R/dyad\\_funcs.R}, in \\texttt{dcr()} and in \\texttt{dcr\\_sandwich()}.}\n\n\\texttt{touse\\_obs <- \\hl{match(rownames(model.frame(model)), rownames(data))}}\n\n\\texttt{\\hl{stopifnot(!anyNA(touse\\_obs), !anyDuplicated(touse\\_obs))}}\n\n\\texttt{touse\\_data <- data[touse\\_obs,]}\n\n\\emph{\\texttt{R/helper\\_funcs.R}, in \\texttt{create\\_dyadid()}, in place of \\texttt{paste0(x, collapse = \"\\_\")}.}\n\n\\texttt{\\hl{paste0(nchar(enc2utf8(x), type = \"bytes\"), \":\", enc2utf8(x), collapse = \"\")}}\n\nThe first change matches each fitted row to the row of the data with the same name and stops if a fitted row has no match or two fitted rows match the same data row. The matching must also be used to align every clustering vector to the model's observation order, preserving each covariance routine's handling of omitted rows. Where row names cannot recover this correspondence, the routines should require an explicit observation identifier. The second change puts both labels in UTF-8 and writes each label's byte length before it, so that (A\\_B, C) becomes \\texttt{3:A\\_B1:C} and (A, B\\_C) becomes \\texttt{1:A3:B\\_C}; grouping on the two sorted columns jointly serves equally well. The same change applies to the copies of \\texttt{create\\_dyadid()} in the two R replication scripts. Neither change alters the estimator.", "evidence": "The evidence is three exact examples, computed in rational arithmetic and reproduced by running the unmodified package. Each has four observations, responses $y=(1,1,-1,-1)$ and a least-squares fit with an intercept only. The fitted intercept is $0$, the residuals equal $y$, and every variance below is a sum of products of residuals divided by $16$.\n\\begin{audittable}{llllrr}{Variance of the intercept estimate in three exact examples.}\nDyads & Row names & Routine & Members counted & Returned & Correct \\\\\nA--B, A--C, D--E, D--F & 3, 4, 5, 6 & \\texttt{dcr()} & 3 of 6 & $3/8$ & $1/2$ \\\\\nA--B, A--C, D--E, D--F & 1, 3, 2, 4 & \\texttt{dcr\\_sandwich()} & 6 of 6 & $0$ & $1/2$ \\\\\nA\\_B--C, A--B\\_C, D--E, F--G & 1, 2, 3, 4 & both & 8 of 8 & $1/8$ & $1/4$ \\\\\n\\end{audittable}\nSince the dyads are distinct in the first two examples, $\\widehat V_D=\\widehat V_0=4/16=1/4$. The clusters of members A and D each give $(2^2+1+1)/16=3/8$, and the other four members give $1/4$ each. By the decomposition, I have\n\\[\n\\widehat V_r=\\frac{7}{4}-\\frac{1}{4}-4\\cdot\\frac{1}{4}=\\frac{1}{2}.\n\\]\nWith row names 3 to 6, \\texttt{dcr()} reads rows 3 and 4 of the four-row frame and two rows that do not exist. It finds the members D, E and F only, and returns\n\\[\n\\frac{7}{8}-\\frac{1}{4}-(3-2)\\cdot\\frac{1}{4}=\\frac{3}{8}.\n\\]\nWith row names 1, 3, 2, 4, \\texttt{dcr\\_sandwich()} assigns member A to observations 1 and 3 and member D to observations 2 and 4. The residuals in each of these clusters sum to zero, so the clusters of A and D give $2/16=1/8$ each, and the routine returns\n\\[\n2\\cdot\\frac{1}{8}+4\\cdot\\frac{1}{4}-\\frac{1}{4}-4\\cdot\\frac{1}{4}=0.\n\\]\nIn the third example the eight members are distinct. Then no two observations share a member, and I find $\\widehat V_r=\\widehat V_0=1/4$. Since the first two dyads are both labelled A\\_B\\_C, the routines compute $\\widehat V_D=\\{(1+1)^2+1+1\\}/16=3/8$ in place of $1/4$ and return\n\\[\n8\\cdot\\frac{1}{4}-\\frac{3}{8}-6\\cdot\\frac{1}{4}=\\frac{1}{8}.\n\\]\nRow names $101$ to $104$, or names that are not numbers, make both routines stop with an error. With the two changes given below, both routines return $1/2$, $1/2$ and $1/4$ on these inputs.", "problem": "The R package is the released implementation of the estimator on p.~191,\n\\[\n\\widehat V_r=\\sum_{i=1}^{N}\\widehat V_{c,i}-\\widehat V_D-(N-2)\\widehat V_0 .\n\\]\nBoth routines convert the row names of the fitted model frame to integers and use them as row positions in the data. Row names label rows; subsetting or reordering a data frame can preserve names that no longer equal row positions. This can make \\texttt{dcr()} build its list of dyad members from the wrong rows, so that $N$ and the sum over members can be wrong; \\texttt{dcr\\_sandwich()} can also attach cluster labels to the wrong observations. Separately, \\texttt{create\\_dyadid()} labels a dyad by sorting its two member labels and joining them with an underscore. The pairs (A\\_B, C) and (A, B\\_C) both receive the label A\\_B\\_C. Two dyads that share no member are then treated as one repeated dyad in $\\widehat V_D$. In each case the routine returns a covariance matrix and gives no warning. The inputs are ordinary. The replication script itself passes a subset of the data for models 1 and 2 of Pratt (2018), where the result happens to be unaffected, and nothing in the documentation of the package excludes underscores in member labels.", "what_survives": "This finding does not concern the estimator, its decomposition on p.~191, the Stata command or the simulations. No number reported in the paper is shown to change. For fits 1 to 5 of Pratt (2018), the covariance matrix without the optional small-sample and positive-semidefinite adjustments agrees with an independent calculation to within $10^{-8}$, all $16$ members (fits 1 and 2) and all $25$ members (fits 3 to 5) are counted, and no distinct dyads share a label. The three Arel-Bundock (2017) fits and the sixth Pratt fit could not be checked."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E006", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of page 192. A red box marks the words '691 KEVs' in the sentence 'The final analytic sample consisted of 691 KEVs across 174 models from 22 studies', which carries the footnote mark 8.", "caption": "Section 4, p. 192 (PDF p. 7). The marked count is that of the sample which the merge script produced; two KEVs that the Stata script designates are missing from it.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E008.png"}, {"alt": "The title, the header and the first six rows of Table A.2, Primary Results, Study Level. In the third row, Bermeo and Leblang (2015), IPE, Foreign aid, a red box marks the entry 29 in the column KEVs. The rest of the row reads 0.655, 0.207, 0, 0.138 and Avg. SER 2.2.", "caption": "Supplementary Table A.2, p. A3 (PDF p. 3 of the supplement, shown upright), first rows. The marked entry is the number of KEVs of Bermeo and Leblang (2015), which is 31 with the two omitted KEVs.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "A selector in the merge script joins two parameter names into one string and so drops two KEVs of Bermeo and Leblang (2015): with them the sample has 693 KEVs, not 691, and that study has 31, not 29, while every printed ratio and proportion of Table 5 keeps its two-decimal value.", "id": "P032-E008", "location": "Section 4, p. 192 (PDF p. 7), sentence beginning “The final analytic sample”; Supplementary Table A.2, row Bermeo and Leblang (2015), p. A3 (PDF p. 3 of the supplement); column KEVs of Table 5, p. 193 (PDF p. 8). Replication files: code/3_reanalysis_merge.R, line 128, and the same line of code/dof_corrections/3_reanalysis_merge_dof.R. The code is not printed in the article.", "pdf_page": 7, "printed_page": "192", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~192: ``The final analytic sample consisted of \\hl{691} KEVs across 174 models from 22 studies.''\n\n\\texttt{3\\_reanalysis\\_merge.R}, line 128:\n\n\\texttt{analysis\\_extract\\_stata(here(\"results\", \"reanalysis\\_stata\", \"bermeo2015/t4m7.xlsx\"),  kev = c(\\hl{\"listock\\_lag MigrationPolicy\\_lag\"}, \"policy\\_stock\\_lag\")),}\n\nSupplementary Table A.2, p.~A3, row Bermeo and Leblang (2015), columns KEVs, Sig.$\\to$Sig., Sig.$\\to$Insig., Insig.$\\to$Sig., Insig.$\\to$Insig., Avg.~SER: \\hl{29}, \\hl{0.655}, \\hl{0.207}, 0, \\hl{0.138}, \\hl{2.2}. Table 5, p.~193, column KEVs: \\hl{122} (2015), \\hl{270} (IPE), \\hl{691} (All).", "corrected": "P.~192: ``The final analytic sample consisted of \\hl{693} KEVs across 174 models from 22 studies.''\n\nLine 128 of both merge scripts, with the path unchanged:\n\n\\texttt{kev = c(\\hl{\"listock\\_lag\", \"MigrationPolicy\\_lag\"}, \"policy\\_stock\\_lag\")),}\n\nSupplementary Table A.2, row Bermeo and Leblang (2015): \\hl{31}, \\hl{0.677}, \\hl{0.194}, 0, \\hl{0.129}, \\hl{2.205}. Last row of the table: \\hl{0.456}, \\hl{0.223}, 0.046, \\hl{0.275}, 1.743. Table 5 and Supplementary Table A.5, column KEVs: \\hl{124} (2015), \\hl{272} (IPE), \\hl{693} (All).\n\nThese values are conditional on the saved estimates. The tables and figures are to be regenerated from the corrected merge.", "evidence": "The evidence is a recalculation from the saved estimates of the replication files. I ran the authors' extraction function with the printed selector and with the three names separated. I find 29 rows for the study with the first and 31 with the second, in 11 models both times. The two restored rows have SER 2.218 (\\texttt{listock\\_lag}) and 2.331 (\\texttt{MigrationPolicy\\_lag}), and both are significant at the 5\\% level before and after. The counts follow by addition: $667+6+18=691$ becomes $669+6+18=693$, where 6 and 18 are the KEVs of the two studies analyzed in R, and the counts for 2015 and for IPE become 124 and 272. For the study, the script averages ratios rounded to three decimals, and $63.811/29\\approx 2.200$ becomes $68.360/31\\approx 2.205$. The overall average SER lies in $[1.74304,\\,1.74314]$ before and in $[1.74325,\\,1.74335]$ after, and it rounds to 1.743 both times. These are intervals because the means of the two R studies are known to three decimals.", "problem": "The Stata script names three KEVs for Table 4, Model 7 of Bermeo and Leblang (2015). Its line 100 reads \\texttt{* Table 4, Model 7; KEV: listock\\_lag MigrationPolicy\\_lag policy\\_stock\\_lag}. The saved results of that model have one row for each. The selector compares whole strings, and the string \\texttt{\"listock\\_lag MigrationPolicy\\_lag\"} equals no parameter name. Only \\texttt{policy\\_stock\\_lag} is kept, and two designated KEVs are left out of the sample. The merge script for the small-sample correction has the same line. The sentence on p.~192 counts correctly the sample that the script produced. The omission is in the sample.", "what_survives": "The 174 models and the 22 studies are unchanged, as are all fitted coefficients and variances. Every printed SER and proportion of Table 5 keeps its two-decimal value, including 1.74, 0.22 and 0.05, and the overall average SER remains 1.743. What changes is the number of KEVs in Table 5 and in Supplementary Table A.5, the study row and three entries of the last row in Supplementary Table A.2, and Figures 1--3, which incorporate the two additional KEV estimates."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E008", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The whole of Table A.3, Reanalysis for Models without (C)RSEs, with three rows: Kinne (2018), Copelovitch and Putnam (2014), and Fang, Johnson, and Leeds (2014). A red box marks the three entries of the last column, Avg. SER: 1.03, 0.90 and 1.69. Below the table is the note 'repeated dyad CRSEs computed as comparison to DCRSEs'.", "caption": "Supplementary Table A.3, p. A4 (PDF p. 4 of the supplement). The marked column holds ratios of the DCRSE to the nonrobust standard error; the note below the table names the repeated-dyad clustered standard error as the comparison.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E009.png"}, {"alt": "Crop of footnote 9 on page 192, between the last line of footnote 8 and footnote 10. Two red boxes mark the words 'repeated dyad cluster robust standard errors are added in', which run over two lines.", "caption": "Footnote 9, p. 192 (PDF p. 7), which cites the table. The marked words state the comparison that the table is meant to report.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E009-b.png"}, {"alt": "A dot chart with three rows and a horizontal axis for the average SER with tick marks from 0.8 to 1.8. Kinne (2018), 4 KEVs: a blue circle labeled 1.03, as printed, joined by a line to an orange square labeled 0.94, recomputed. Copelovitch and Putnam (2014), 45 KEVs: a blue circle labeled 0.90 and an orange square labeled 0.92. Fang, Johnson, and Leeds (2014), 1 KEV: a blue circle labeled 1.69 joined to an orange square labeled 1.22. A dashed vertical line at 1 is labeled DCRSE equals the comparison SE.", "caption": "Means of three-decimal SERs for each row of Supplementary Table A.3, using the nonrobust comparison (circles) and the repeated-dyad clustered comparison (squares); labels are rounded to two decimals. The dashed line marks a ratio of one. For Kinne (2018) the recomputed ratio is below one; for Fang, Johnson, and Leeds (2014) it rounds to 1.22 in place of 1.69.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E009-avg-ser.png"}], "formal_release": null, "formal_statements": [], "headline": "The three average SERs of Supplementary Table A.3 divide the DCRSE by the nonrobust standard error of the primary analysis and not by the repeated-dyad clustered standard error that the table note and footnote 9 name; with the stated comparison they are 0.94, 0.92 and 1.22 in place of 1.03, 0.90 and 1.69.", "id": "P032-E009", "location": "Supplementary Table A.3, column Avg. SER, p. A4 (PDF p. 4 of the supplement), cited in footnote 9, p. 192 (PDF p. 7). Replication script: 3_reanalysis_merge.R, block “CRSE REDO”, lines 707–739. The code is not printed in the article.", "pdf_page": 4, "printed_page": "A4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Supplementary Table A.3, p.~A4:\n\n\\begin{audittable}{lrrrrrr}{Table A.3: Reanalysis for Models without (C)RSEs}\nStudy & KEVs & Sig.$\\to$Sig. & Sig.$\\to$Insig. & Insig.$\\to$Sig. & Insig.$\\to$Insig. & Avg.~SER \\\\\nKinne (2018) & 4 & 0.75 & 0.00 & 0.00 & 0.25 & \\hl{1.03} \\\\\nCopelovitch and Putnam (2014) & 45 & 0.42 & 0.11 & 0.11 & 0.36 & \\hl{0.90} \\\\\nFang, Johnson, and Leeds (2014) & 1 & 0.00 & 1.00 & 0.00 & 0.00 & \\hl{1.69} \\\\\n\\end{audittable}\n\n``Note: repeated dyad CRSEs computed as comparison to DCRSEs.''\n\nFootnote 9, p.~192: ``See Supplementary Table A.3 for a summary of the results of DCR estimation for these models when repeated dyad cluster robust standard errors are added in.''\n\n\\texttt{3\\_reanalysis\\_merge.R}, line 710, the first of 24 calls of this form on lines 710--738:\n\n\\texttt{analysis\\_extract\\_stata(here(\"results\", \\hl{\"reanalysis\\_stata\"}, \"copelovitch2014/t1m2.xlsx\"), kev = c(\"uncertainty\")),}", "corrected": "Supplementary Table A.3, with the note unchanged:\n\n\\begin{audittable}{lrrrrrr}{Table A.3: Reanalysis for Models without (C)RSEs}\nStudy & KEVs & Sig.$\\to$Sig. & Sig.$\\to$Insig. & Insig.$\\to$Sig. & Insig.$\\to$Insig. & Avg.~SER \\\\\nKinne (2018) & 4 & 0.75 & 0.00 & 0.00 & 0.25 & \\hl{0.94} \\\\\nCopelovitch and Putnam (2014) & 45 & 0.42 & 0.11 & 0.11 & 0.36 & \\hl{0.92} \\\\\nFang, Johnson, and Leeds (2014) & 1 & 0.00 & 1.00 & 0.00 & 0.00 & \\hl{1.22} \\\\\n\\end{audittable}\n\n\\texttt{3\\_reanalysis\\_merge.R}, the 24 calls on lines 710--738, of which the first is\n\n\\texttt{analysis\\_extract\\_stata(here(\"results\", \\hl{\"reanalysis\\_stata\\_crse\"}, \"copelovitch2014/t1m2.xlsx\"), kev = c(\"uncertainty\")),}\n\nThese are comparisons of saved estimates and not new fits.", "evidence": "The evidence is a recalculation from the saved result files of the replication package. No model was fitted again. I applied the rules of the merge script to the 50 KEVs that the block selects, once in the 24 files that it reads and once in the 24 files that the Stata script writes for the clustered comparison. The rules round each ratio to three decimals, average the rounded ratios, round the mean to three decimals and display it to two decimals. The coefficients and DCRSEs agree up to numerical rounding; the comparison standard errors and their p-values change.\n\n\\begin{audittable}{lrrrr}{Average SER of Supplementary Table A.3 under the two comparison standard errors. ``Files read'' are the result files with the nonrobust comparison, which the block reads; ``clustered files'' are those written for the repeated-dyad clustered comparison. Means are of ratios rounded to three decimals.}\nStudy & KEVs & Printed & Files read & Clustered files \\\\\nKinne (2018) & 4 & 1.03 & $1033/1000=1.033$ & $3741/4000=0.93525$ \\\\\nCopelovitch and Putnam (2014) & 45 & 0.90 & $40417/45000\\approx0.89816$ & $41351/45000\\approx0.91891$ \\\\\nFang, Johnson, and Leeds (2014) & 1 & 1.69 & $1691/1000=1.691$ & $611/500=1.222$ \\\\\n\\end{audittable}\n\nThe files that the block reads reproduce the printed column exactly. For each of the 50 KEVs the significance category at the 5\\% level is the same in both sets of files, so the four proportions of each row are unchanged. As an instance, the one KEV of Fang, Johnson, and Leeds has DCRSE $0.2308$, nonrobust standard error $0.1365$ and repeated-dyad clustered standard error $0.1889$, to four decimals; the two ratios round to $1.691$ and $1.222$.", "problem": "Twenty-four models in three studies used neither robust nor clustered standard errors. The Stata script runs the DCR command on each of them twice. The first run uses the comparison standard error of the original study, which is the nonrobust one, and the second uses a standard error clustered on repeated dyads. The script writes the two sets of results to two folders. The block of the merge script that builds Table A.3 reads the first folder for all 24 models. The column Avg.~SER then repeats ratios of the primary analysis. The entries for Copelovitch and Putnam and for Fang, Johnson, and Leeds are rounded versions of those in their rows in Supplementary Table A.2, 0.898 and 1.691. The table does not report the comparison that its note and footnote 9 describe. The difference changes how the table is read. For Kinne (2018) the average ratio is below one and not above it. For Fang, Johnson, and Leeds (2014) the DCRSE is about 22\\% larger than the repeated-dyad clustered standard error, in place of the roughly 69\\% implied by the printed ratio.", "what_survives": "The numbers of KEVs and the four significance proportions of Table A.3 are unchanged, and the table note and footnote 9 stand once the column is corrected. Table 5, with the average SER of 1.74 and the significance-loss proportion of 0.22, Figures 1--3 and Supplementary Tables A.1, A.2 and A.4 are built from other objects of the script and do not change under this correction. The merge script for the small-sample correction makes the same choice of folder in a block whose table is not printed."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E009", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the caption of Figure 3 and its note, below the horizontal axis label Original p-value. A red outline marks the last sentence of the note: The area of each plotted data point is proportional to its inverse study frequency weight.", "caption": "p. 195 (PDF p. 10), caption and note of Figure 3. The marked sentence says that the area of a point is proportional to its weight.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E010.png"}], "formal_release": null, "formal_statements": [], "headline": "In Figure 3 the size of a point increases with its inverse study frequency weight, but its area is not proportional to that weight.", "id": "P032-E010", "location": "Note to Figure 3, p. 195 (PDF p. 10); Section 4.1, p. 196 (PDF p. 11), last sentence", "pdf_page": 10, "printed_page": "195", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~195, note to Figure 3.} The \\hl{area} of each plotted data point \\hl{is proportional to} its inverse study frequency weight.\n\n\\emph{p.~196.} Figure 3 visualizes how the precisions of individual KEV estimates change with the application of DCRSEs across all reanalyzed studies, with the \\hl{area} of each plotted data point \\hl{being proportional to} its inverse study frequency weight.", "corrected": "\\emph{p.~195, note to Figure 3.} The \\hl{size} of each plotted data point \\hl{increases with} its inverse study frequency weight.\n\n\\emph{p.~196.} Figure 3 visualizes how the precisions of individual KEV estimates change with the application of DCRSEs across all reanalyzed studies, with \\hl{point size increasing with} inverse study frequency weight.\n\nTo keep the printed wording instead, both panels would be drawn again with an area scale that starts at zero and with no stroke on the points (\\texttt{scale\\_size\\_area(max\\_size = 6)} and \\texttt{stroke = 0}).", "evidence": "The evidence is a count of pixels in the image embedded in the published PDF. The point for Horowitz and Stam (2014), variable \\texttt{combat}, has weight $1/3$ and covers $512$ fully black and $605$ nonwhite pixels. The point for Fang, Johnson, and Leeds (2014) has weight $1$ and covers $1131$ and $1263$. However the gray edge pixels are counted, the ratio of the two areas is at most $1263/512\\approx 2.47$, which is less than the ratio $3$ of the weights. In each legend the circles labelled $1.00$ and $0.25$ have an area ratio below $4$: at most $635/214$ in the top panel and $1271/430$ in the bottom panel.", "problem": "The plotting code maps the weight to the size of the point without setting a scale that makes point area proportional to weight. In the published figure a point with a larger weight is drawn larger, but the areas are not in proportion to the weights.", "what_survives": "The plotted $p$-values, the weighted averages and proportions in Table 5 and Supplementary Table A.2, and Figures 1 and 2 do not use the point-size scale and are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E010", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last row of Table A.5 and the note below it. A red outline marks the sentence that begins: For the purposes of producing a true apples-to-apples comparison, original standard error estimates of reanalyzed studies are similarly treated with a small-sample correction. The sentence ends: degrees-of-freedom equals N sub c minus 1.", "caption": "Supplementary Appendix, p. A5 (PDF p. 5), note to Table A.5. The marked sentence says that the original standard errors receive the factor $\\sqrt{N_c/(N_c-1)}$ and a $t$ reference distribution with $N_c-1$ degrees of freedom, where $N_c$ counts the clusters in the analytic sample.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E011.png"}], "formal_release": null, "formal_statements": [], "headline": "In the small-sample reanalysis behind Supplementary Table A.5, the code does not apply the stated correction to the original standard errors of the three Arel-Bundock (2017) models, and it takes the degrees of freedom for four Pratt (2018) models from clusters counted before observations are dropped.", "id": "P032-E011", "location": "Note to Supplementary Table A.5, Supplementary Appendix p. A5 (PDF p. 5), sentence beginning “For the purposes of producing a true apples-to-apples comparison”; cited in footnote 12, p. 193 (PDF p. 8); replication file code/dof_corrections/2_reanalysis_r_dof.R, lines 87–101 and 151–179", "pdf_page": 5, "printed_page": "A5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table A.5, note.} For the purposes of producing a true apples-to-apples comparison, original standard error estimates of reanalyzed studies are \\hl{similarly treated} with a small-sample correction that is equivalent to multiplying the final estimates of standard error by $\\sqrt{N_c/(N_c-1)}$, where $N_c$ is the number of unique clusters \\hl{in the analytic sample}; the test statistic is likewise compared to a t-distribution with degrees-of-freedom $=N_c-1$.\n\n\\emph{Replication file, lines 88 and 90 (Arel-Bundock, model 1; models 2 and 3 are alike).}\n\n\\texttt{original\\_se = \\hl{sqrt(diag(vcovHC(m1\\_arelbundock2017, type = \"HC0\")))},}\n\n\\texttt{df\\_ori = nobs(m1\\_arelbundock2017) - 1,}\n\n\\emph{Replication file, line 154 (Pratt, model 1; models 2 to 6 are alike).}\n\n\\texttt{df\\_ori = \\hl{length(unique(spec1\\_data\\_pratt2018\\$Dir.DyadID))} - 1,}", "corrected": "In \\texttt{2\\_reanalysis\\_r\\_dof.R}, for each of the three Arel-Bundock (2017) models, the original standard error is \\hl{$\\sqrt{n/(n-1)}$ times} the HC0 standard error, where $n$ is \\texttt{nobs(model)}. For each Pratt (2018) model, the original degrees of freedom are $N_c-1$, where $N_c$ is the number of distinct values of \\texttt{Dir.DyadID} \\hl{in the source-data rows matched by row name to \\texttt{model.frame(model)}}. This gives $208$, $208$, $289$, $328$ and $327$ degrees of freedom for models 1 to 5. Table A.5 is then recomputed.\n\nThe note to Table A.5 should also state which finite-sample factors each original standard error carries. After the two changes these are the HC0 standard error times $\\sqrt{n/(n-1)}$ for Arel-Bundock (2017); the \\texttt{sandwich::vcovCL} defaults for Pratt (2018), with no further factor; and, for the Stata studies, the covariance returned by the original command times $N_c/(N_c-1)$.", "evidence": "The evidence is an exact example run through the script's own function and a refit of the Pratt models. Let $x=(0,1,2,3)$ and $y=(0,1,1,3)$, and consider the least-squares fit with an intercept. The coefficients are $(-1/10,\\,9/10)$ and the residuals are $(1/10,\\,1/5,\\,-7/10,\\,2/5)$. A direct computation gives\n\\[\nV_{\\mathrm{HC0}}=\\begin{pmatrix}113/5000&-21/2500\\\\-21/2500&103/5000\\end{pmatrix},\\qquad\n\\frac{4}{3}\\,V_{\\mathrm{HC0}}=\\begin{pmatrix}113/3750&-7/625\\\\-7/625&103/3750\\end{pmatrix}.\n\\]\nThe function \\texttt{reg\\_table} of the script returns the standard errors $0.1503$ and $0.1435$ from the first matrix, while the note with $N_c=n=4$ requires $0.1736$ and $0.1657$ from the second. In general the squared ratio of the two is $(n-1)/n$. For the first model of Arel-Bundock (2017), with $n=3971$ (Supplementary Table A.1), the missing factor is $\\sqrt{3971/3970}\\approx 1.000126$.\n\nFor Pratt (2018), refitting models 1 to 5 on the data in the replication files and counting the clusters among the fitted rows, I obtain the following.\n\\begin{audittable}{rrrrrr}{Pratt (2018), models 1--5: clusters and original degrees of freedom.}\nModel & Rows in fit & Clusters, all rows & Clusters, fit & df in script & df by the note \\\\\n1 & 1177 & 238 & 209 & 237 & 208 \\\\\n2 & 1177 & 238 & 209 & 237 & 208 \\\\\n3 & 2025 & 329 & 290 & 328 & 289 \\\\\n4 & 3718 & 329 & 329 & 328 & 328 \\\\\n5 & 3343 & 329 & 328 & 328 & 327 \\\\\n\\end{audittable}\nThe $p$-value of the KEV in model 1 is $0.07748$ with $237$ degrees of freedom and $0.07766$ with $208$. No KEV in these five models crosses the 5\\% level when the degrees of freedom are corrected. Model 6 was not refitted.", "problem": "The note describes two steps for the original standard errors, and the R script for this reanalysis departs from each. For the three Arel-Bundock (2017) models the original standard error is the HC0 standard error, and no factor is applied. Under HC0 each observation is its own cluster. This implies that $N_c$ is the number $n$ of observations in the fit, and the note calls for the factor $\\sqrt{n/(n-1)}$. For the Pratt (2018) models the degrees of freedom are one less than the number of clusters in the data frame passed to the fit. The fit drops rows with missing values, and the clusters that remain are fewer in models 1, 2, 3 and 5.\n\nTwo other points qualify the comparison and are not counted as errors. The original standard errors for Pratt (2018) come from \\texttt{sandwich::vcovCL}, whose defaults already supply $G/(G-1)$ for $G$ clusters, and $(n-1)/(n-k)$ in the least-squares fits. For the Stata studies the command multiplies the refitted original covariance by $N_c/(N_c-1)$, as the note says. For Shelef (2016) that covariance already carries $\\{G/(G-1)\\}\\{(n-1)/(n-k)\\}$, and the cluster factor enters twice.", "what_survives": "This finding does not contradict the DCRSEs, the primary analysis in Table 5, or the statement of footnote 12 that the small-sample reanalysis shows no substantive differences. The two corrections bear on Table A.5 alone. There the original standard errors of the first model of Arel-Bundock (2017) grow by a factor of about $1.0001$, and the original $p$-values of the KEVs in models 1 to 5 of Pratt (2018) move by less than $0.0002$. A corrected Table A.5 has not been computed."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E011", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three entries of the reference list. A red outline marks the year 2017 in the entry: Erikson, R. S., P. M. Pinto, and K. T. Rader. 2017. Dyadic Analysis in International Relations: A Cautionary Tale. Political Analysis 22 (4): 457 to 463.", "caption": "p. 197 (PDF p. 12), reference list. The marked year in the entry for Erikson, Pinto, and Rader is 2017; the entry places the article in volume 22, issue 4.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-E012.png"}], "formal_release": null, "formal_statements": [], "headline": "The article by Erikson, Pinto, and Rader was published in 2014, not 2017.", "id": "P032-E012", "location": "References, p. 197 (PDF p. 12), entry for Erikson, Pinto, and Rader; citations on pp. 186–187 (PDF pp. 1–2)", "pdf_page": 12, "printed_page": "197", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~197.} Erikson, R. S., P. M. Pinto, and K. T. Rader. \\hl{2017}. ``Dyadic Analysis in International Relations: A Cautionary Tale.'' \\emph{Political Analysis} 22 (4): 457--463.\n\n\\emph{pp.~186--187.} ``Erikson, Pinto, and Rader (\\hl{2017})''; ``Erikson et al. (\\hl{2017})''; ``Erikson et al. \\hl{2017}''.", "corrected": "\\emph{p.~197.} Erikson, R. S., P. M. Pinto, and K. T. Rader. \\hl{2014}. ``Dyadic Analysis in International Relations: A Cautionary Tale.'' \\emph{Political Analysis} 22 (4): 457--463.\n\n\\emph{pp.~186--187.} ``Erikson, Pinto, and Rader (\\hl{2014})''; ``Erikson et al. (\\hl{2014})''; ``Erikson et al. \\hl{2014}''.", "evidence": "The evidence is a check of the publisher's page for the article, \\url{https://doi.org/10.1093/pan/mpt051}, which gives Volume 22, Issue 4, Autumn 2014, pp.~457--463. In the same reference list, \\emph{Political Analysis} 23 (4) is dated 2015 and 24 (4) is dated 2016.", "problem": "Volume 22, issue 4 of \\emph{Political Analysis} is the Autumn 2014 issue. The publisher's page for the article also carries the line ``Published online by Cambridge University Press: 04 January 2017'', and that date is not the year of the journal issue.", "what_survives": "The article is identified correctly by its authors, title, volume, issue and pages; only the year changes, in the reference and in its three citations."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-E012", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "A table titled 'Table 4. Dyadic clustering.' with header cell 'ijt / i-prime j-prime t-prime' and six columns and six rows labelled 111, 112, 211, 212, 221, 222. A red box surrounds the six column labels and a second red box surrounds the six row labels. The cells contain expectations of products of residuals, such as E[u_111 u_112], except that the four cells pairing rows 111 and 112 with columns 221 and 222, and the four cells pairing rows 221 and 222 with columns 111 and 112, contain 0.", "caption": "Table 4, p. 190 (PDF p. 5). The column labels and the row labels are marked; Tables 2 and 3 on the same page carry the same labels.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N003.png"}], "formal_release": null, "formal_statements": [{"id": "P032-T11", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T11"}], "headline": "The six observations of Tables 2–4 stand for three dyads, each observed twice, of which the middle one shares a member with each of the other two; with the indices of p. 189, the labels 111, 112, 221 and 222 pair a country with itself.", "id": "P032-N003", "location": "Tables 2, 3 and 4, row and column labels, p. 190 (PDF p. 5)", "pdf_page": 5, "printed_page": "190", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P032-T11"], "text": {"as_printed": "In each of Tables 2--4 the header is $ijt\\,/\\,i'j't'$, and the rows and columns are labelled \\hl{111, 112, 211, 212, 221, 222}. Table 4, for example, has $\\mathrm{E}[u_{111}u_{112}]$ in the cell $(111,112)$, $\\mathrm{E}[u_{221}u_{222}]$ in the cell $(221,222)$, and $0$ in the cells that pair 111 or 112 with 221 or 222.", "corrected": "In Tables 2--4, replace the row and column labels, and the matching subscripts of $u$, as follows: 111 by \\hl{121}, 112 by \\hl{122}, 211 by \\hl{131}, 212 by \\hl{132}, 221 by \\hl{341} and 222 by \\hl{342}. The observations are then the dyads $(1,2)$, $(1,3)$ and $(3,4)$, each in two periods, and every zero and nonzero entry of the three tables is unchanged.", "intended_meaning": "Page 189 defines $Y_{ijt}$ as the level of commerce between countries $i$ and $j$ in period $t$. With these indices, the printed labels are the dyads $(1,1)$, $(2,1)$ and $(2,2)$ in periods 1 and 2, and two of the three pair a country with itself. The tables are a schematic of three pairs of countries, each observed twice, in which the middle pair shares a member with each outer pair and the outer pairs share none. The zero pattern of each table is correct for this picture, and also for the labels as printed. The labels matter only if the illustration is carried to the decomposition on p.~191. For a dyad that pairs a member with itself, the decomposition gives weight 0 to the covariance between its two observations, where Table 4 has a nonzero entry. With distinct members, the decomposition agrees with Table 4 in all 36 cells."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N003", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of page 191: two lines of text, the displayed equation V hat r equals the sum over i from 1 to N of V hat c,i, minus V hat D, minus (N minus 2) V hat 0, and the paragraph that defines the four matrices. A red box marks the term (N minus 2) V hat 0 in the display. Two more red boxes mark the words 'V hat 0 is the estimated heteroskedasticity-consistent variance-covariance matrix', which run over two lines.", "caption": "The decomposition on p. 191 (PDF p. 6). The marked term is the piece that the command takes from a fit with vce(robust); the marked words define it as the heteroskedasticity-consistent matrix, in which every observation is its own cluster. The display is correct as printed.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N004.png"}, {"alt": "The whole of Table A.2, Primary Results, Study Level: 22 rows, one per study, with columns Study, Subfield, Secondary Subfield, Primary Relationship Tested, KEVs, four significance proportions and Avg. SER, and a last row Average-of-Averages. A red box marks the row Efrat and Newman (2016), IOs/International Law, 4 KEVs, Avg. SER 4.158. A second red box marks the row Weisiger (2016), Security, 28 KEVs, Avg. SER 3.268.", "caption": "Supplementary Table A.2, p. A3 (PDF p. 3 of the supplement, shown upright). The marked rows are the two studies whose models are fitted with stcox and stcrreg. Their average SERs, 4.158 and 3.268, are the largest and the third largest of the table. The values are not shown to be wrong; they were not computed with the observation-level matrix.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P032-T3", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T3"}, {"id": "P032-T4", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T4"}, {"id": "P032-T11", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T11"}], "headline": "For the 13 survival models behind the Efrat and Newman (2016) and Weisiger (2016) rows of Supplementary Table A.2, the Stata command takes $\\widehat V_0$ from a fit with vce(robust), whose variance on data declared with stset and an id() variable is clustered on that variable and is not the observation-level matrix that the decomposition requires; the piece should be clustered on an observation index and the 13 models fitted again.", "id": "P032-N004", "location": "Display for $\\widehat V_r$ and definition of $\\widehat V_0$, p. 191 (PDF p. 6); Supplementary Table A.2, rows Efrat and Newman (2016) and Weisiger (2016), p. A3 (PDF p. 3 of the supplement). Replication files: dcr.ado, lines 163–190, and 1_reanalysis_stata.do, lines 319–334 and 347–418. The code is not printed in the article.", "pdf_page": 6, "printed_page": "191", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P032-T11", "P032-T3", "P032-T4"], "text": {"as_printed": "P.~191:\n\\[\n\\widehat V_r=\\sum_{i=1}^{N}\\widehat V_{c,i}-\\widehat V_D-(N-2)\\widehat V_0,\n\\]\nwhere \\ldots\\ $\\widehat V_0$ is the estimated heteroskedasticity-consistent variance--covariance matrix.\n\n\\texttt{dcr.ado}, lines 164, 178 and 190, in the block headed ``Calculate heteroskedasticity robust vcov'':\n\n\\texttt{qui `prefix' `command\\_line' if `touse' `weight', `options' \\hl{vce(robust)}}\n\n\\texttt{matrix vhet = e(V)}\n\n\\texttt{matrix vcov = vcov - ((N\\_ud - 2)*unadjust*vhet)}\n\n\\texttt{1\\_reanalysis\\_stata.do}, line 319 (Efrat and Newman) and line 347 (Weisiger). Lines 355, 368, 376, 384 and 397 declare the same id as line 347, and line 408 declares \\texttt{id(idvar)}.\n\n\\texttt{stset T1, failure(acceptance==1) id(dyad)}\n\n\\texttt{stset warday, id(warnum) f(endday)}\n\nSupplementary Table A.2, p.~A3, columns KEVs, Sig.$\\to$Sig., Sig.$\\to$Insig., Insig.$\\to$Sig., Insig.$\\to$Insig., Avg.~SER. Efrat and Newman (2016): 4, 0.5, 0.5, 0, 0, 4.158. Weisiger (2016): 28, 0.143, 0.607, 0, 0.25, 3.268.", "corrected": "The decomposition, with the conditions under which it is an identity. Let each observation $r$ of the estimation sample have an unordered dyad $d_r$ of two distinct members, let $N$ be the number of distinct members, and let $z_r$ be the contribution of observation $r$ to the estimate, that is, its score contribution multiplied by the common bread matrix. Let all pieces come from one fit on one sample, without finite-sample factors:\n\\[\n\\widehat V_0=\\sum_r z_rz_r^\\top,\\qquad\n\\widehat V_D=\\sum_{r,s:\\,d_r=d_s}z_rz_s^\\top,\\qquad\n\\widehat V_{c,i}=\\sum_{r,s:\\,i\\in d_r\\cap d_s}z_rz_s^\\top+\\sum_{r:\\,i\\notin d_r}z_rz_r^\\top .\n\\]\nThen\n\\[\n\\sum_{i=1}^{N}\\widehat V_{c,i}-\\widehat V_D-(N-2)\\widehat V_0=\\sum_{r,s:\\,d_r\\cap d_s\\ne\\varnothing}z_rz_s^\\top=\\widehat V_r .\n\\]\nIn particular, $\\widehat V_0$ is \\hl{clustered on single observations and not on the id of the survival-time data}.\n\nIn \\texttt{dcr.ado}, for a command run on data with an \\texttt{stset} id, the piece \\texttt{vhet} is fitted with \\hl{an explicit cluster variable that takes a different value on every observation of the estimation sample}, in place of \\texttt{vce(robust)}. The \\texttt{stset} id is kept, since \\texttt{stcrreg} uses it for its censoring weights. Every piece is fitted with \\texttt{noadjust} and \\texttt{\\hl{dofundo(none)}}, or the factor that the command applied to a piece is removed with the number of clusters which that fit reports, so that all pieces carry the same scaling. This describes the repair; it is not a patch that has been run in Stata.\n\nThe rows Efrat and Newman (2016) and Weisiger (2016) of Supplementary Table A.2, and the entries of Table 5, Supplementary Table A.5 and Figures 1--3 that use them, are to be recomputed after the 13 models are fitted again. I give no corrected values.", "evidence": "The evidence has three parts: a check of the code, the documentation of the two Stata commands, and an exact example.\n\nIn \\texttt{dcr.ado}, lines 108--119 give all observations that contain a member the cluster value $-99$ and give every other observation its own row number. Lines 126 and 202 fit the repeated-dyad piece and the member-specific pieces with explicit cluster variables. Line 164 fits the piece called \\texttt{vhet} with \\texttt{vce(robust)}, and line 190 subtracts $(N-2)$ times it.\n\nThe Stata Survival Analysis Reference Manual, Release 15 (StataCorp 2017), in the entry [ST] stcox, technical note under ``Robust estimate of variance'' (p.~17), says of \\texttt{vce(robust)} on data with an id that ``stcox knew to specify vce(cluster id)'', and it lets an explicit cluster variable override the id. The entry [ST] stcrreg, last paragraph of ``Methods and formulas'' (p.~24), states that the clustered version of the variance is used when a cluster variable is given ``or an ID variable has been stset.'' The description of \\texttt{noadjust} in [ST] stcox (p.~4) gives the default multiplier as $n/(n-1)$ or $g/(g-1)$. I read these entries, and the current ones, which agree with them. I did not read Release 17, which the replication files name, and Stata was not run.\n\nThe example has six subjects, one for each dyad among four members, with ten observations, one covariate $x$ and no weights. Status 1 is the event of interest, and status 2 is a competing event, treated as censoring.\n\n\\begin{audittable}{rrcrrrr}{An exact survival example: ten observations, six ids, four dyad members.}\nRow & Id & Dyad & Interval & $x$ & Status & Score at $\\widehat\\beta=0$ \\\\\n1 & 1 & $\\{1,2\\}$ & $(0,1]$ & $1$ & 1 & $2/3$ \\\\\n2 & 2 & $\\{3,4\\}$ & $(0,1]$ & $-1$ & 1 & $-2/3$ \\\\\n3 & 3 & $\\{1,3\\}$ & $(0,1]$ & $1$ & 0 & $-1/3$ \\\\\n4 & 3 & $\\{1,3\\}$ & $(1,2]$ & $3$ & 1 & $3/2$ \\\\\n5 & 4 & $\\{2,4\\}$ & $(0,1]$ & $-1$ & 0 & $1/3$ \\\\\n6 & 4 & $\\{2,4\\}$ & $(1,2]$ & $-3$ & 1 & $-3/2$ \\\\\n7 & 5 & $\\{1,4\\}$ & $(0,1]$ & $1$ & 0 & $-1/3$ \\\\\n8 & 5 & $\\{1,4\\}$ & $(1,3]$ & $1$ & 2 & $-1/2$ \\\\\n9 & 6 & $\\{2,3\\}$ & $(0,1]$ & $-1$ & 0 & $1/3$ \\\\\n10 & 6 & $\\{2,3\\}$ & $(1,3]$ & $-1$ & 2 & $1/2$ \\\\\n\\end{audittable}\n\nWith Breslow ties the Cox partial likelihood is maximized at $\\widehat\\beta=0$, the information is $12$, and the scores of the ten observations are those of the last column. Multiplying the sums of squared scores by $1/144$, I obtain the pieces below.\n\n\\begin{audittable}{lr}{The pieces of the decomposition in the example ($N=4$), and what the decomposition returns.}\nQuantity & Value \\\\\n$\\widehat V_0$, every observation its own cluster & $19/432$ \\\\\n$\\widehat V_{\\mathrm{id}}$, clustered on the id (equal to $\\widehat V_D$ here) & $15/432$ \\\\\n$\\sum_i\\widehat V_{c,i}$ & $68/432$ \\\\\nDecomposition as printed: $\\sum_i\\widehat V_{c,i}-\\widehat V_D-2\\widehat V_0$ & $15/432=5/144$ \\\\\nWith $\\widehat V_{\\mathrm{id}}$ in place of $\\widehat V_0$ & $23/432$ \\\\\nWith $\\alpha\\widehat V_{\\mathrm{id}}$ in place of $\\widehat V_0$, $\\alpha=(9/10)(6/5)=27/25$ & $103/2160$ \\\\\n\\end{audittable}\n\nThe printed decomposition returns $5/144$, which is also the sum of the products of all pairs of observations that share a member. With the variance matrix clustered on the id, it returns a variance $23/15$ times as large, and with the factor of the Weisiger calls a variance $103/75$ times as large. All three values are positive, so the eigenvalue step changes none of them.\n\nThe identity in the corrected statement holds by counting. The product $z_rz_s^\\top$ of two observations enters $\\sum_i\\widehat V_{c,i}$ once for each member $i$ for which the two observations lie in one cluster. If $r=s$, this happens $N$ times, and subtracting $\\widehat V_D$ and $(N-2)\\widehat V_0$ leaves $N-1-(N-2)=1$. If $r\\ne s$ and the two observations have the same dyad, it happens twice, and subtracting $\\widehat V_D$ leaves $1$. If the dyads differ and share one member, it happens once and nothing is subtracted. If the dyads share no member, the product does not appear.", "problem": "The decomposition is an identity when all of its pieces are sums over the contributions of the same observations, where an observation is one row of the data. In $\\widehat V_{c,i}$ all observations whose dyad contains member $i$ form one cluster and each other observation is its own cluster, and in $\\widehat V_0$ every observation is its own cluster. The Stata command builds the member-specific pieces in this way. It obtains $\\widehat V_0$ by fitting the model again with \\texttt{vce(robust)}, and it subtracts $(N-2)$ times the result. For \\texttt{stcox} and \\texttt{stcrreg}, on data declared with \\texttt{stset} and an \\texttt{id()} variable, the variance that Stata returns for \\texttt{vce(robust)} is clustered on the id. The contributions of all observations with one id are summed before they are squared. The reanalysis declares such an id for the three \\texttt{stcox} models of Efrat and Newman (2016) and for the ten models of Weisiger (2016), two \\texttt{stcox} and eight \\texttt{stcrreg}. These 13 models hold 32 KEVs. For them the command returns, before the eigenvalue step of footnote 3 and when every piece carries the same scaling,\n\\[\n\\sum_{i=1}^{N}\\widehat V_{c,i}-\\widehat V_D-(N-2)\\widehat V_{\\mathrm{id}}=\\widehat V_r+(N-2)\\bigl(\\widehat V_0-\\widehat V_{\\mathrm{id}}\\bigr),\n\\]\nwhere $\\widehat V_{\\mathrm{id}}$ is the variance matrix clustered on the id. The Weisiger calls set \\texttt{dofundo(asylike)}, which multiplies the piece by $(n-1)/n$, with $n$ the number of observations, while the factor that Stata documents for a clustered variance is $g/(g-1)$, with $g$ the number of ids. The last term is then $(N-2)\\bigl(\\widehat V_0-\\alpha\\widehat V_{\\mathrm{id}}\\bigr)$ with $\\alpha=((n-1)/n)(g/(g-1))$. The difference from $\\widehat V_r$ has no fixed sign, and any difference between the two matrices is multiplied by $N-2$. I did not fit the 13 models again, and I do not know the size or the sign of the difference in the two studies.", "what_survives": "The printed decomposition is correct as an identity, and the 20 studies that are not fitted with these survival commands are not touched by this point. What changes is the standing of the two rows of Supplementary Table A.2. They are not shown to be DCRSEs as p.~191 defines them. The two rows enter the 2016, IOs/International Law, Security and All rows of Table 5, the upper end of the range ``0.90 to 4.16'' on p.~196, Figures 1--3 and, through the small-sample scripts, Supplementary Table A.5. No corrected value has been computed, and none of these numbers is shown to change. The following averages show how much rests on the two rows and are not corrections. The mean of the Avg.~SER column of Table A.2 over the other 20 studies is 1.546, against 1.743 over all 22. The mean for IOs/International Law without Efrat and Newman is 1.701, and the mean for Security without Weisiger is 1.338."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N004", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the top of page 191: two lines of text, the displayed equation for V hat r, and the paragraph that defines its terms. Two small red boxes mark the letter N, once as the upper limit of the sum and once inside the bracket (N minus 2). Two more red boxes mark the words 'V hat c,i is the estimated dyad-member-i-specific clustered variance-covariance matrix', which run over two lines.", "caption": "The decomposition on p. 191 (PDF p. 6). The marked symbol $N$ is not defined on the page, and the marked definition of $\\widehat V_{c,i}$ does not say how the remaining observations are clustered.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N005.png"}], "formal_release": null, "formal_statements": [{"id": "P032-T11", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T11"}], "headline": "The decomposition does not say what $N$ is, nor how $\\widehat V_{c,i}$ treats the observations whose dyad does not contain member $i$: $N$ is the number of unique dyad members, and each such observation is its own cluster.", "id": "P032-N005", "location": "Display for $\\widehat V_r$ and the sentence that defines its terms, p. 191 (PDF p. 6)", "pdf_page": 6, "printed_page": "191", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P032-T11"], "text": {"as_printed": "\\[\n\\widehat V_r=\\sum_{i=1}^{\\hl{\\scriptstyle N}}\\widehat V_{c,i}-\\widehat V_D-(\\hl{N}-2)\\widehat V_0,\n\\]\nwhere $\\widehat V_r$ is the estimated DCR variance--covariance matrix for longitudinal data; $\\widehat V_{c,i}$ is the estimated \\hl{dyad-member-$i$-specific clustered} variance--covariance matrix; $\\widehat V_D$ is the estimated repeated-dyad clustered variance--covariance matrix; and $\\widehat V_0$ is the estimated heteroskedasticity-consistent variance--covariance matrix.", "corrected": "After the display and its definitions, add: \\hl{Here $N$ is the number of unique dyad members. In $\\widehat V_{c,i}$, all observations whose dyad contains member $i$ form one cluster and each remaining observation forms its own singleton cluster. The components use the same unadjusted sandwich convention, and each dyad contains two distinct members.}", "intended_meaning": "$N$ is the number of unique dyad members, as the note to Supplementary Table A.5 says. In $\\widehat V_{c,i}$ all observations whose dyad contains member $i$ form one cluster, which footnote 2 indicates, and every other observation is a cluster by itself. The pieces carry no finite-sample factors, and each dyad has two distinct members. With these definitions the display is an exact identity. In $\\sum_i\\widehat V_{c,i}$ the square of an observation appears $N$ times, the product of two observations of the same dyad twice, and the product of two observations whose dyads share one member once. Subtracting $\\widehat V_D$ and $(N-2)\\widehat V_0$ leaves each of these exactly once. The R package uses these definitions."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N005", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of page 192. A red box marks the words 'Three studies' at the start of the sentence 'Three studies employ at least one model that has fixed effects for both members of the dyad under consideration', which carries the footnote mark 10.", "caption": "Section 4, p. 192 (PDF p. 7). The marked count includes Shelef (2016), whose reanalyzed models have no fixed effect for either dyad member.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N006.png"}, {"alt": "The whole of Table A.4, Reanalysis for Models with Fixed Effects for both Dyad Members, with three rows. A red box marks the second row: Shelef (2016), 9 KEVs, 0.89, 0.00, 0.00, 0.11, Avg. SER 1.07. The other rows are Carter and Poast (2020) and Bermeo and Leblang (2015).", "caption": "Supplementary Table A.4, p. A4 (PDF p. 4 of the supplement). The marked row is computed from models with region-by-decade indicators only.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N006-b.png"}, {"alt": "Crop of footnote 10 on page 192, between the last line of footnote 9 and the first line of footnote 11. A red box marks the words 'three papers' at the end of the sentence 'Supplementary Table A.4 shows the results of applying DCRSEs to the KEVs in these three papers'.", "caption": "Footnote 10, p. 192 (PDF p. 7), which repeats the count.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N006-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The nine reanalyzed models of Shelef (2016) have region-by-decade indicators and no fixed effect for either dyad member; two studies, not three, have a model with fixed effects for both members of the dyad, and the Shelef row does not belong in Supplementary Table A.4.", "id": "P032-N006", "location": "Section 4, p. 192 (PDF p. 7), sentence beginning “Three studies employ”, and footnote 10; Supplementary Table A.4, row Shelef (2016), p. A4 (PDF p. 4 of the supplement)", "pdf_page": 7, "printed_page": "192", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "P.~192: ``\\hl{Three} studies employ at least one model that has fixed effects for both members of the dyad under consideration.''\n\nFootnote 10, p.~192: ``Supplementary Table A.4 shows the results of applying DCRSEs to the KEVs in these \\hl{three} papers.''\n\nSupplementary Table A.4, p.~A4:\n\n\\begin{audittable}{lrrrrrr}{Table A.4: Reanalysis for Models with Fixed Effects for both Dyad Members}\nStudy & KEVs & Sig.$\\to$Sig. & Sig.$\\to$Insig. & Insig.$\\to$Sig. & Insig.$\\to$Insig. & Avg.~SER \\\\\nCarter and Poast (2020) & 8 & 0.00 & 0.38 & 0.50 & 0.12 & 1.10 \\\\\n\\hl{Shelef (2016)} & 9 & 0.89 & 0.00 & 0.00 & 0.11 & 1.07 \\\\\nBermeo and Leblang (2015) & 1 & 1.00 & 0.00 & 0.00 & 0.00 & 2.13 \\\\\n\\end{audittable}", "corrected": "P.~192: ``\\hl{Two} studies employ at least one \\hl{reanalyzed} model that has fixed effects for both members of the dyad under consideration.''\n\nFootnote 10: ``Supplementary Table A.4 shows the results of applying DCRSEs to the KEVs in these \\hl{two} papers.''\n\nSupplementary Table A.4 keeps its title and the rows Carter and Poast (2020) and Bermeo and Leblang (2015). The row Shelef (2016) is removed, and \\texttt{shelef2016\\_FE} is removed from \\texttt{FE\\_papers} in \\texttt{3\\_reanalysis\\_merge.R} (lines 824--828).", "evidence": "The evidence is an exact check on the data of the replication files. Observations 3 and 7 of the Shelef data file are in the estimation sample of all nine models and have the same level, 7, of \\texttt{regionfe\\_region\\_decade}. Their members are $(255,315)$ and $(315,260)$, with the losing state first. Let $w$ be the vector with $+1$ on observation 3, $-1$ on observation 7 and $0$ elsewhere. Then $w$ is orthogonal to every indicator of \\texttt{regionfe\\_region\\_decade}. Its inner product with the indicator of losing state 255 is $1$, and the same holds for the indicator of gaining state 315 and for the indicator that state 255 is either member. This implies that none of these three indicators is a linear combination of the region-by-decade indicators. The other regressors cannot supply the member effects either. In the three groups of models (Tables 3, 4 and 5 of the study) the design matrix has rank at most 34, 31 and 33. The indicators of the losing and of the gaining states have ranks 114 and 114, 105 and 104, and 83 and 83.", "problem": "Each of the nine Shelef regressions behind this row has one set of indicators, \\texttt{i.regionfe\\_region\\_decade}, and no indicators for the state that loses territory (\\texttt{loseccode}) or for the state that gains it (\\texttt{gainccode}). The first of them reads\n\n\\texttt{reg midpost homelandclaim priormid coethnicity oil\\_gas capital\\_distance\\_log dyaddem allydummy cincrat i.regionfe\\_region\\_decade if colony2==0, cluster(dyadid4)}\n\nThe options \\texttt{dm1(loseccode)} and \\texttt{dm2(gainccode)} of the DCR command name the dyad members for the variance and add no regressors. Region-by-decade indicators do not contain the member effects. These models are not models with fixed effects for both members of the dyad, and the subgroup that p.~192 and footnote 10 describe has two studies.", "what_survives": "The numbers of the Shelef row are those of the nine models as fitted. I did not recompute them, and no error in them is shown. Shelef (2016) stays in the primary analysis of 22 studies, and Table 5 and Supplementary Tables A.1 and A.2 do not change. What changes is the subgroup with fixed effects for both members. It has two studies and 9 KEVs ($8+1$), not three studies and 18. The article draws no conclusion from this subgroup."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N006", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of footnotes 12, 13 and 14 at the foot of page 193. A red outline marks the second sentence of footnote 13: The SER can therefore be interpreted as the inflation (or deflation) of the original standard error introduced by inter-dyad dependencies.", "caption": "p. 193 (PDF p. 8), footnotes 12–14. The marked sentence of footnote 13 attributes the whole ratio to inter-dyad dependencies.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "A SER different from one need not come from inter-dyad dependence alone: the DCRSE receives no small-sample correction, while the original standard error keeps the finite-sample factors of its software.", "id": "P032-N007", "location": "Section 4, pp. 192–193 (PDF pp. 7–8), sentence beginning “Also, for the purposes of our primary reanalysis”; Section 4.1, p. 193 (PDF p. 8), definition of the SER and footnote 13", "pdf_page": 7, "printed_page": "192–193", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{pp.~192--193.} Also, for the purposes of our primary reanalysis, there are no small-sample corrections made to \\hl{our standard error estimates}. Finite-sample corrections would inflate our standard error estimates to account for increased sampling variability, and thus might paint an overly pessimistic picture of the original literature.\n\n\\emph{p.~193.} We compute a standard error ratio (SER) for all KEVs, which is the DCRSE divided by the standard error produced using the original variance estimation strategy.\n\n\\emph{p.~193, footnote 13.} For example, if the original model clustered standard errors on repeated dyads, the SER is a ratio of DCR standard error to repeated dyad clustered standard error. The SER can therefore be interpreted as the inflation (or deflation) of the original standard error \\hl{introduced by inter-dyad dependencies}.", "corrected": "\\emph{pp.~192--193.} Also, for the purposes of the primary reanalysis, no final small-sample correction is made to \\hl{the DCR standard error estimates; the original standard errors keep the finite-sample adjustments returned by the replication software}.\n\n\\emph{p.~193, footnote 13.} For example, if the original model clustered standard errors on repeated dyads, the SER is a ratio of DCR standard error to repeated dyad clustered standard error. The SER therefore measures the inflation (or deflation) of the original standard error under DCR re-estimation; \\hl{it reflects the additional inter-dyad covariance terms and any difference in finite-sample adjustment between the two standard errors}.", "intended_meaning": "The phrase ``our standard error estimates'' refers to the DCRSEs, which receive no final small-sample correction in the primary reanalysis. The SER is computed as defined, and the comparison with the original standard errors is the one the paper intends. The denominator is the standard error that the replication software returns, and it keeps that software's finite-sample factors. In the nine least-squares models of Shelef (2016), the clustered variance of the original fit carries the factor $q=\\{G/(G-1)\\}\\{(n-1)/(n-k)\\}$ for $n$ observations, $G$ clusters and design rank $k$, while the DCR variance is constructed from components with their finite-sample factors removed. If $R$ denotes the published ratio, then the ratio with no factor on either side is $\\sqrt{q}\\,R$, and the published ratio is the smaller of the two since $q>1$. For Shelef (2016) the mean of $R$ is $1.071$, as in Supplementary Table A.2, and the mean of $\\sqrt{q}\\,R$ is $1.149$. An exact example shows that the factor alone can move the ratio. Suppose that each of the six dyads among four members has one observation, that the fit has an intercept only, and that the outcomes are $(1,0,0,0,0,-1)$ on the dyads $(1,2),(1,3),(1,4),(2,3),(2,4),(3,4)$. The two nonzero residuals lie on disjoint dyads. Then both variances without factors equal $1/18$ and the original variance with $q=6/5$ is $1/15$, and I obtain $\\sqrt{5/6}\\approx 0.913$ for the SER. No revised average over the 22 studies has been computed, and the reported ratios stand as defined."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N007", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the first four lines of the Data Availability Statement. A red outline marks the words: command for Stata (version 15 or higher). The next line is the command git clone git@github.com:jscarlson/stata-dcr.git.", "caption": "p. 196 (PDF p. 11), Data Availability Statement. The marked words give Stata 15 as the minimum version for the dcr command.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N010.png"}], "formal_release": null, "formal_statements": [], "headline": "The stated minimum of Stata 15 does not hold for the conditional-logit path of the dcr command, because the supplied nclogit program requires Stata 16.1 or higher.", "id": "P032-N010", "location": "Data Availability Statement, p. 196 (PDF p. 11); the same sentence in the Supplementary Appendix, “Statistical Programming Suite”, p. A1 (PDF p. 1)", "pdf_page": 11, "printed_page": "196", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~196 and Supplementary Appendix, p.~A1.} To access the source code for the \\texttt{dcr} command for Stata (\\hl{version 15 or higher}), clone its GitHub repository:\n\n\\emph{\\texttt{nclogit.ado}, lines 1--2.}\n\n\\texttt{program nclogit, eclass}\n\n\\texttt{version 16.1}", "corrected": "\\emph{p.~196 and Supplementary Appendix, p.~A1.} To access the source code for the \\texttt{dcr} command for Stata (\\hl{the \\texttt{dcr} wrapper declares version 15; the supplied \\texttt{nclogit} conditional-logit path requires version 16.1 or higher}), clone its GitHub repository:", "evidence": "The evidence is a reading of the source of \\texttt{dcr.ado} and of \\texttt{nclogit.ado}, whose two released copies are identical, together with the rule for Stata's \\texttt{version} command. Stata was not run.", "problem": "The program \\texttt{nclogit}, supplied with \\texttt{dcr} for conditional-logit models, begins with the statement \\texttt{version 16.1}, which is executed before any argument is read. Stata 15 and 15.1 cannot run a program that requests a later version. Every call of \\texttt{dcr nclogit} stops at that statement, although \\texttt{dcr.ado} itself declares version 15. The replication uses this path for three models, one of Weisiger and Yarhi-Milo (2015) and two of Kinne (2018).", "what_survives": "Results produced with Stata/MP 17.0, the version named in the replication README, are unaffected, and no estimate, table or figure changes. The other estimator paths of \\texttt{dcr} are not shown to fail on Stata 15."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N010", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The title 'Table A.1: Dyadic Data Summary Statistics, Study Level' and the header row 'Study, Observations, Data Start Year, Data End Year, Number of Unique Dyads', then a dashed grey rule, then seven rows from Pratt (2018) to Weisiger and Yarhi-Milo (2015) and the note below the table. The row 'Schneider and Tobin (2020)' reads 4760, 1976, 2009, 756, and a red box surrounds 4760.", "caption": "Supplementary Table A.1, supplement p. A2: the title and header, the last seven rows and the note, with the fifteen rows between them left out at the dashed rule. The mark is on the number of observations for Schneider and Tobin (2020).", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N011.png"}], "formal_release": null, "formal_statements": [], "headline": "The number of observations for Schneider and Tobin (2020) in Supplementary Table A.1 is 952, not 4760, and the 756 dyads of that row are directed donor–recipient pairs.", "id": "P032-N011", "location": "Supplementary Table A.1, row “Schneider and Tobin (2020)”, supplement p. A2 (PDF p. 2 of the Supplementary Appendix)", "pdf_page": 2, "printed_page": "A2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Table A.1: Dyadic Data Summary Statistics, Study Level. The columns are Study, Observations, Data Start Year, Data End Year and Number of Unique Dyads, and the row reads:\n\nSchneider and Tobin (2020), \\hl{4760}, 1976, 2009, 756.\n\nNote: summary statistics for each study computed using data from a representative (usually the first) reanalyzed model, as analytic samples and data often vary by model.", "corrected": "Schneider and Tobin (2020), \\hl{952}, 1976, 2009, 756.\n\n\\hl{For Schneider and Tobin (2020), 756 counts directed donor--recipient pairs. Ignoring direction, there are 741 pairs, six of which pair a country with itself.}", "evidence": "The evidence is a recalculation from the replication files. The data file has 4624 rows, of which 952 satisfy the condition \\texttt{g7==1} of the three models. Each of these is a distinct donor--recipient--year, and the years run from 1976 to 2009. The 96 saved $p$-values that use a $t$ distribution (two columns for each of 16 coefficients in three models) are reproduced with $n=952$ and 936 degrees of freedom to within $2.8\\times10^{-16}$, and not with $n=4760$. The 952 observations contain 756 distinct ordered pairs and 741 unordered pairs: 15 pairs of countries occur in both orders, and six of the 741 pair a country with itself. The code that filled the summary workbook is not in the replication files, and how the entry 4760 was produced cannot be traced.", "problem": "The note to the table says that the statistics come from a representative reanalyzed model. The three reanalyzed models of this study are fitted to 952 donor--recipient--year observations. The printed 4760 equals $5\\times952$, and the code in the replication files requests five imputations. The entry has the size of five completed copies of the sample and does not count the observations once. The entry 756 is the number of directed donor--recipient pairs among the 952 observations. With direction ignored, as in the clustering of the reanalysis (p.~192), there are 741 pairs, and the table does not say which convention it uses.", "what_survives": "The correction changes this row of Table A.1 and the corresponding cells of the summary workbook in the replication files. The years 1976 and 2009 are correct, and 756 is correct as a count of directed pairs. The estimates, standard errors, standard error ratios and significance comparisons are computed from the model outputs and do not use these two columns."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N011", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "A listing of Stata code with line numbers. Line 1375 of 1_reanalysis_stata.do is a call 'dcr logit bilateral_bailout ...' that wraps over six rows; red outlines surround 'dofundo(asylike)' and 'prefix(mi estimate, post:)' in its options. Below, lines 125 to 127 of dcr.ado run the prefixed command with vce(cluster did), line 145 reads 'matrix vrep = e(V)', lines 153 to 155 set 'scalar unadjust = ((N_did - 1)/N_did)' when dofundo is asylike, and line 157 reads 'matrix vcov = -1*unadjust*vrep', with a red outline around 'unadjust*vrep'.", "caption": "Source code of the replication files, not a page of the article: 1_reanalysis_stata.do, line 1375, and stata-dcr/dcr.ado, lines 125–127, 145 and 153–157. The outlines mark the option that removes the finite-sample correction, the multiple-imputation prefix, and the product of the factor with the matrix returned under that prefix.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N012.png"}, {"alt": "The title 'Table A.2: Primary Results, Study Level' and the header row 'Study, Subfield, Secondary Subfield, Primary Relationship Tested, KEVs, Sig. to Sig., Sig. to Insig., Insig. to Sig., Insig. to Insig., Avg. SER', then a dashed grey rule, then the rows from Pratt (2018) to Weisiger and Yarhi-Milo (2015) and the row 'Average-of-Averages (= ISFW)'. A red box surrounds the row 'Schneider and Tobin (2020), IPE, International finance, Domestic spillovers and international bailout, 18, 0.333, 0, 0.056, 0.611, 0.999'.", "caption": "Supplementary Table A.2, supplement p. A3: the title and header, the last seven study rows and the row of averages, with the fifteen rows between them left out at the dashed rule. The marked row holds the results that pass through this code. No entry of the row is shown to be wrong.", "url": "https://pmaronow.github.io/llm-audit/works/P032/exhibits/P032-N012-tableA2.png"}], "formal_release": null, "formal_statements": [{"id": "P032-T8", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T8"}, {"id": "P032-T9", "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-T9"}], "headline": "For the one study fitted to multiply imputed data, the code removes the finite-sample corrections from covariance matrices that are already pooled over the imputations, which also rescales the between-imputation component; the result need not equal the DCR covariance computed within each imputation and then pooled.", "id": "P032-N012", "location": "Replication files: 1_reanalysis_stata.do, lines 1375, 1380 and 1385, and stata-dcr/dcr.ado, lines 125–238. The results enter Supplementary Table A.2, row “Schneider and Tobin (2020)”, supplement p. A3 (PDF p. 3 of the Supplementary Appendix). The article prints no statement on multiple imputation.", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P032-T8", "P032-T9"], "text": {"as_printed": "Line 1375 of \\texttt{1\\_reanalysis\\_stata.do}, with the regressors and the output path left out:\n\n\\texttt{dcr logit bilateral\\_bailout} \\ldots\\ \\texttt{if g7==1, dm1(cc) dm2(dc)} \\texttt{crse(donor\\_country) psd} \\hl{\\texttt{dofundo(asylike)}} \\texttt{syntax(vce) xlsxpath(}\\ldots\\texttt{)} \\hl{\\texttt{prefix(mi estimate, post:)}}\n\nLines 145, 154 and 157 of \\texttt{stata-dcr/dcr.ado}, for the repeated-dyad component:\n\n\\texttt{matrix vrep = e(V)}\n\n\\texttt{scalar unadjust = ((N\\_did - 1)/N\\_did)}\n\n\\texttt{matrix vcov = -1*}\\hl{\\texttt{unadjust*vrep}}\n\nThe 18 key explanatory variables (KEVs) of the three models give the row of Supplementary Table A.2: Schneider and Tobin (2020), 18, 0.333, 0, 0.056, 0.611, 0.999.", "corrected": "For a model fitted to $M$ imputed data sets, the DCR covariance is computed as follows.\n\n\\begin{enumerate}\n\\item In each imputation $m$, compute each component covariance and remove its finite-sample correction with the factor of that component, taken from the sample and the clusters of that imputation. If \\hl{no same-country dyad appears in more than one observation}, form\n\\[\n\\widehat V_r^{(m)}=\\sum_{i=1}^{K}\\widehat V_{c,i}^{(m)}-\\widehat V_D^{(m)}-(K-2)\\widehat V_0^{(m)}.\n\\]\nIf \\hl{a same-country dyad is observed more than once}, compute $\\widehat V_r^{(m)}$ by \\hl{the direct sandwich sum over observation pairs whose dyads share a member}, including an observation paired with itself.\n\\item Pool by Rubin's rules, with the between-imputation term entering \\hl{once and with coefficient 1}:\n\\[\n\\widehat V_r=\\frac{1}{M}\\sum_{m=1}^{M}\\widehat V_r^{(m)}+\\hl{\\Bigl(1+\\frac{1}{M}\\Bigr)B}.\n\\]\n\\item Apply the eigenvalue adjustment of footnote 3, where it is used, at a stated point of this sequence.\n\\end{enumerate}\n\nWhen \\hl{no same-country dyad appears in more than one observation}, the sample, the clusters and the factors are the same in every imputation and no eigenvalue adjustment is active, the result equals the matrix that the code now computes plus \\hl{$(1-c)(1+1/M)B$}, with $c$ as in the evidence above. The option \\texttt{dofundo(none)} is not a repair, since it leaves the finite-sample corrections in the within-imputation components. The three models and the summaries that use them are then to be recomputed.", "evidence": "The evidence is a derivation from the code and an exact counterexample. No Stata run was made, and the three models were not refitted. Let $K$ be the number of dyad members (the $N$ of p.~191), $n$ the number of observations and $M$ the number of imputations. The decomposition has one component for each member $i$, the repeated-dyad component $D$ and the heteroskedasticity-consistent component $0$, with signed weights $s_i=1$, $s_D=-1$ and $s_0=-(K-2)$, so that $\\sum_js_j=1$. For component $j$, let $\\bar U_j$ be the average over the imputations of the within-imputation covariance without finite-sample correction, and let $a_j$ be the factor by which the command multiplies. The corrected within-imputation covariance is the uncorrected one divided by $a_j$. Suppose that no same-country dyad appears in more than one observation, that the sample, the clusters and the factors are the same in every imputation and that no eigenvalue adjustment is made. Since the fitted coefficients do not depend on the clustering, every component has the same $Q=(1+1/M)B$. The matrix returned for component $j$ is $T_j=\\bar U_j/a_j+Q$, and the command computes\n\\[\nT_{\\mathrm{code}}=\\sum_js_ja_jT_j=\\sum_js_j\\bar U_j+cQ,\\qquad c=\\sum_js_ja_j.\n\\]\nThe DCR covariance computed within each imputation and then pooled is $T_*=\\sum_js_j\\bar U_j+Q$. Subtracting, I find $T_{\\mathrm{code}}-T_*=(c-1)(1+1/M)B$. With the option \\texttt{dofundo(asylike)}, the factors are $a_i=1-1/G_i$, $a_D=1-1/G_D$ and $a_0=1-1/n$, where $G_i$ and $G_D$ are the numbers of clusters in the component for member $i$ and in the repeated-dyad component. Substituting these factors into the definition of $c$, I obtain\n\\[\nc-1=-\\sum_{i=1}^{K}\\frac{1}{G_i}+\\frac{1}{G_D}+\\frac{K-2}{n},\n\\]\nwhich is not zero in general.\n\nThe decomposition includes each observation paired with itself; it omits only pairs of different observations on a common same-country dyad. In the source data, I count each of the six same-country dyads just once among the 952 observations with \\texttt{g7==1}, so none contributes such an omission.\n\nFor the counterexample, I consider 24 observations, four on each of the six dyads among four members, an intercept-only logit and five imputations. Three imputations complete the data to 12 successes and two to 18. Then $K=4$, $n=24$, $G_D=6$ and $G_i=13$ for every member, and $c=4(12/13)-5/6-2(23/24)=49/52$. The fitted intercepts are $0$ and $\\log3$, and I have $B=3(\\log3)^2/10$ and $Q=9(\\log3)^2/25$. The DCR variances within the two kinds of imputation are $2/9$ and $8/81$, with average $14/81$. A direct computation gives $T_*=14/81+(9/25)(\\log3)^2$ and $T_{\\mathrm{code}}=14/81+(441/1300)(\\log3)^2$, and the difference is $-(27/1300)(\\log3)^2<0$. All the variances are positive, and the eigenvalue adjustment is not active.", "problem": "Each of the three calls runs every component of the decomposition on p.~191 under the prefix \\texttt{mi estimate, post:}. Under the standard behaviour of this prefix, the matrix \\texttt{e(V)} returned for a component is the covariance pooled over the five imputations by Rubin's rules: the average of the within-imputation covariances plus $(1+1/M)B$, where $B$ is the between-imputation covariance of the coefficients and $M=5$. The command then multiplies the returned matrix by the factor that removes the finite-sample correction of that component. This factor belongs to the within-imputation covariance only. Applied to the pooled matrix, it also multiplies $(1+1/M)B$. The components carry different factors, and the between-imputation term enters the result with a coefficient $c$ that differs from 1 in general. The DCR covariance computed within each imputation and then pooled has coefficient 1. The decomposition on p.~191 is correct for one complete data set if no same-country dyad appears in more than one observation.", "what_survives": "The decomposition on p.~191 is unaffected. The reanalyses of the other 21 studies do not use this multiple-imputation prefix. The logit coefficients of the three models are unaffected, since the covariance estimator does not change them. The DCR covariance of the three models depends on this code, and with it the DCRSEs, the standard error ratios and the DCR $p$-values of the 18 KEVs in the row of Supplementary Table A.2. These enter the 2020, IPE and overall rows of Table 5 and Figures 1--3, and the same calls are repeated in the small-sample reanalysis behind Supplementary Table A.5. No value in these tables is shown to be wrong. The size and direction of the difference for the three models have not been computed, because the replication files do not keep the coefficient and covariance matrices of each imputation."}, "url": "https://pmaronow.github.io/llm-audit/works/P032/#P032-N012", "work": {"arxiv_id": "2109.03774", "doi": "10.1017/pan.2023.26", "title": "Dyadic Clustering in International Relations", "url": "https://pmaronow.github.io/llm-audit/works/P032/"}, "work_id": "P032"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the section heading 'Why is it hard to estimate RD effects?' and the first nine lines of its opening paragraph. A red box marks the words 'RD design' in the first sentence, which begins 'The causal effect studied in the RD design is the difference'.", "caption": "Section “Why is it hard to estimate RD effects?”, first paragraph, p. 3 (PDF p. 3), left column. The marked words name the design without restricting it to the sharp case.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-A001.png"}], "formal_release": null, "formal_statements": [], "headline": "The opening identification argument is the argument for a sharp RD design, and the paragraph does not say “sharp”.", "id": "P033-A001", "location": "Section “Why is it hard to estimate RD effects?”, first paragraph, p. 3 (PDF p. 3), left column", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The causal effect studied in the \\hl{RD design} is the difference between the expected treated and control potential outcomes at a cutoff where treatment assignment changes in a discontinuous fashion. If the conditional expectation function of the outcome were known, this effect can be calculated without error under an assumption that the conditional expectation functions of the potential outcomes are continuous at the cut point. In other words, the RD effect is \\emph{identified} under a continuity assumption.", "corrected": "The causal effect studied in \\hl{a sharp} RD design is the difference between the expected treated and control potential outcomes at a cutoff where treatment assignment changes in a discontinuous fashion.\n\nA sentence on fuzzy designs may be added: \\hl{A fuzzy RD analysis uses the jump in the outcome divided by the jump in treatment receipt. For binary treatment receipt, under the additional continuity, exclusion, monotonicity and nonzero-first-stage assumptions, this ratio identifies an effect for compliers at the cutoff.}", "intended_meaning": "The paragraph states the identification argument for a sharp design, in which the running variable determines treatment. Let $X$ be the running variable and $c$ the cutoff, with the support of $X$ approaching $c$ from both sides. Set $D=\\mathbf{1}\\{X\\ge c\\}$ and $Y=DY(1)+(1-D)Y(0)$, and suppose that $\\mu_d(x)=\\mathbb{E}[Y(d)\\mid X=x]$ is continuous at $c$ for $d=0,1$. Then\n\\[\n\\lim_{x\\downarrow c}\\mathbb{E}[Y\\mid X=x]-\\lim_{x\\uparrow c}\\mathbb{E}[Y\\mid X=x]=\\mu_1(c)-\\mu_0(c)=\\mathbb{E}[Y(1)-Y(0)\\mid X=c],\n\\]\nwhich is the effect named in the first sentence. Read as this introductory case, the paragraph is correct.\n\nThe argument does not extend to a fuzzy design, in which only the probability of receiving treatment jumps at the cutoff. For example, suppose that half of the units are compliers with effect $2$ and half are never-takers. The two observed regression functions are the same whether the effect for the never-takers is $0$ or $10$, while the effect over all units is $1$ in the first case and $6$ in the second. The sample includes fuzzy designs, which are listed in Table~S1 of the supplement, and the replication code treats them separately from sharp designs. I found no formula or result of the paper that applies the argument outside the sharp case."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-A001", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Scatter plot with horizontal axis 'Original P-Value' and vertical axis 'RDHonest P-Value', both from 0 to 1, with a red diagonal line and dashed lines at 0.05 on each axis; most points lie near the lower left corner. Red boxes mark the words 'RDHonest P-Value' in the title, the vertical-axis label 'RDHonest P-Value', and the words 'p-values derived using RDHonest' in the caption 'Figure S9: Original p-values vs. p-values derived using RDHonest'.", "caption": "Supplement, Figure S9, p. 18. The marked words, in the title, the vertical-axis label and the caption, call the plotted quantity a $p$-value of RDHonest.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E001.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T12", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T12"}, {"id": "P033-T13", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T13"}], "headline": "The vertical axis of Figure S9 shows a normal-tail transform of the pseudo $t$-statistic built from one 95% bias-aware interval, which reproduces the 5% decision of that interval but is not in general a valid $p$-value at other levels.", "id": "P033-E001", "location": "Supplement, Figure S9, p. 18: title, vertical-axis label and caption", "pdf_page": 18, "printed_page": "18", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P033-T12", "P033-T13"], "text": {"as_printed": "Title: ``Original versus \\hl{RDHonest P-Value}''. Subtitle: ``(Studies with Replication Data Reanalyzed with RDHonest)''. Horizontal axis: ``Original P-Value''. Vertical axis: ``\\hl{RDHonest P-Value}''.\n\nCaption: ``Figure S9: Original $p$-values vs. \\hl{$p$-values derived using \\texttt{RDHonest}}''.", "corrected": "Title: ``Original \\hl{p-values versus RDHonest pseudo p-values}''. Vertical axis: ``RDHonest \\hl{pseudo p-value}''.\n\nCaption: ``Figure S9: Original $p$-values versus \\hl{normal-tail transforms of RDHonest pseudo $t$-statistics. The vertical quantity is $p_*=2\\Phi(-z_{0.975}|\\widehat\\theta|/h)$, where $h$ is the reported 95\\% bias-aware confidence-interval half-length. The rule $p_*<0.05$ is exactly equivalent to this interval excluding zero. These transforms are not generally valid $p$-values at other significance levels; any 5\\% validity inherits the interval's coverage assumptions.}''\n\nThe plotted values are unchanged. A $p$-value that is valid at every level can be obtained by inverting a valid, nested family of bias-aware intervals, one for each level, under its assumptions on curvature and variance. In the Gaussian model $T\\sim N(\\theta+b,s^2)$ with known $s>0$ and known bound $|b|\\le B$, the value $\\Phi\\bigl((B-|T|)/s\\bigr)+\\Phi\\bigl((-B-|T|)/s\\bigr)$ is a valid $p$-value for $\\theta=0$. I do not give replacement values for the figure.", "evidence": "This correction rests on a derivation, an exact counterexample and a comparison with the replication files. I checked the Figure~S9 labels in the released figure and the caption in the supplement's extracted text; I did not have the typeset supplement page. Since $\\Phi$ is strictly increasing, I have for every $h>0$ that\n\\[\np_*<0.05\\iff 2\\Phi\\bigl(-z|\\widehat\\theta|/h\\bigr)<2\\Phi(-z)\\iff|\\widehat\\theta|>h\\iff 0\\notin[\\widehat\\theta-h,\\widehat\\theta+h].\n\\]\nFor the counterexample, let $T\\sim N(\\theta+b,1)$, where the bias $b$ satisfies $|b|\\le B$ for a known bound $B=1$. Let $H$ be the unique positive number such that\n\\[\n\\Phi(H-B)-\\Phi(-H-B)=\\frac{19}{20}.\n\\]\nThe interval $[T-H,T+H]$ covers $\\theta$ with probability at least $19/20$ for every $\\theta$ and every $|b|\\le B$, with equality at $b=\\pm B$. Applied to this interval, the transform is $p_*(T)=2\\Phi(-z|T|/H)$. Fix $\\theta=0$, $b=1$ and $\\alpha=1/5$, and set $q=\\Phi^{-1}(9/10)$ and $C=Hq/z$. Then $p_*\\le 1/5$ exactly when $|T|\\ge C$, and since $T\\sim N(1,1)$, I have\n\\[\n\\Pr(p_*\\le 1/5)=\\Phi(1-C)+\\Phi(-1-C).\n\\]\nIn exact rational arithmetic, I find that $2.64614554821530<H<2.64614554821540$ and that\n\\[\n0.23579196727091<\\Pr(p_*\\le 1/5)<0.23579196727101.\n\\]\nThe transform therefore rejects a true null about 23.6\\% of the time at the nominal 20\\% level, although the worst-case coverage of the interval is exactly 95\\%. With $B=b=1/2$ the same probability exceeds $0.20414905632765$. At $\\alpha=1/20$ the rejection probability is exactly $1/20$.\n\nThe 46 values plotted in Figure~S9 agree with this transform of the stored pseudo $t$-statistics to within $1.3\\times 10^{-15}$. For each of them, $p_*<0.05$ agrees with the 95\\% interval excluding zero, which 20 of the 46 intervals do. I have not established the rejection rates of the transform for the 46 estimates themselves. When there is no bias, so that $B=0$ and $H=z$, the transform is the ordinary two-sided normal $p$-value at every level.", "problem": "The quantity on the vertical axis is not a $p$-value returned by the RDHonest procedure. Let $\\widehat\\theta$ be the RDHonest estimate, let $h$ be the half-length of its 95\\% bias-aware confidence interval, and write $z=\\Phi^{-1}(0.975)$. The replication code computes the pseudo standard error $h/z$ and the pseudo $t$-statistic $z|\\widehat\\theta|/h$, and it plots\n\\[\np_*=2\\Phi\\bigl(-z|\\widehat\\theta|/h\\bigr).\n\\]\nThe main text (p.~8) and Section~S9 call the $t$-statistics ``pseudo'', but the title, the axis label and the caption of Figure~S9 call $p_*$ a $p$-value without qualification. A $p$-value $P$ for a null hypothesis must satisfy $\\Pr(P\\le\\alpha)\\le\\alpha$ under the null for every level $\\alpha$. The rule $p_*<0.05$ rejects exactly when the interval excludes zero. This implies that the rule has level at most 5\\% when the interval has 95\\% coverage. At other levels $p_*$ need not satisfy the requirement, as the example below shows.", "what_survives": "No plotted number, interval or 5\\% decision changes. The pseudo $t$-statistics and pseudo standard errors of Figures~S5 to~S8 and Table~S6 are as the paper describes them. The rdrobust results, the caliper tests and the power tables use other quantities, and Figure~S9 is the only place where these values are used. The reading of the full vertical scale of Figure~S9, and of its diagonal, as a comparison of $p$-values does change."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E001", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of seven lines of the right column of page 3. A red box marks the word 'variance' in the sentence 'Conventional methods used to gauge this accuracy rely on an assumption that the bias of the estimator is negligible compared to its variance, as would be the case in, for example, a randomized experiment.'", "caption": "Section “Why is it hard to estimate RD effects?”, p. 3 (PDF p. 3), right column. The marked word gives the scale against which the bias is compared.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E002.png"}], "formal_release": null, "formal_statements": [], "headline": "Conventional inference requires the bias of the estimator to be negligible compared to its standard deviation, not its variance.", "id": "P033-E002", "location": "Section “Why is it hard to estimate RD effects?”, p. 3 (PDF p. 3), right column, paragraph beginning “The use of bandwidth selection methods to maximize accuracy”", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Conventional methods used to gauge this accuracy rely on an assumption that the bias of the estimator is negligible compared to its \\hl{variance}, as would be the case in, for example, a randomized experiment.", "corrected": "Conventional methods used to gauge this accuracy rely on an assumption that the bias of the estimator is negligible compared to its \\hl{standard deviation}, as would be the case in, for example, a randomized experiment.\n\nAn equivalent wording is that the \\hl{squared} bias of the estimator is negligible compared to its variance.", "intended_meaning": "The sentence means that the standardized bias is negligible, that is, that the bias is small relative to the standard deviation of the estimator. This is the correct condition. I write the estimator as $T_n=\\theta+b_n+s_nZ_n$, where $b_n$ is its bias, $s_n$ is its standard deviation, and $Z_n$ converges in distribution to $N(0,1)$. With a consistent standard error, conventional inference centered at $\\theta$ relies on\n\\[\n\\frac{T_n-\\theta}{s_n}=Z_n+\\frac{b_n}{s_n}\n\\]\nhaving a standard normal limit, which holds when $|b_n|/s_n\\to 0$, or equivalently $b_n^2/s_n^2\\to 0$.\n\nThe ratio of the bias to the variance is a different quantity, and it depends on the units of the outcome, since multiplying the outcome by $a$ multiplies the bias by $a$ and the variance by $a^2$. For independent $X_i\\sim N(\\theta,1)$ and $T_n=\\bar X_n+n^{-3/4}$, I have $b_n/s_n=n^{-1/4}\\to 0$ and $b_n/s_n^2=n^{1/4}\\to\\infty$. The conventional 95\\% interval then has coverage $\\Phi(z_{0.975}-n^{-1/4})-\\Phi(-z_{0.975}-n^{-1/4})$, which equals $0.938467$ at $n=100$ and $0.948854$ at $n=10{,}000$, and tends to $0.95$. No estimator, code or result of the paper uses the printed wording."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E002", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the paragraph that introduces the funnel plot. Red outlines mark the word 'sample' at the end of one line, the word 'variance' at the start of the next, and the number '1.96' in the sentence about the slope of the significance boundary.", "caption": "Results, p. 7 (PDF p. 7), right column. The marked terms are “sample variance”, broken across two lines, and the slope “1.96”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T10", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T10"}, {"id": "P033-T11", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T11"}, {"id": "P033-T18", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T18"}], "headline": "Figure 4 is standardized by the sample standard deviation of the outcome among control units, and its significance boundary has slope $1/z_{.975}\\approx0.51$, not $1.96$.", "historical_evidence_type": "lean_proof", "id": "P033-E003", "location": "Results, p. 7 (PDF p. 7), right column, paragraph introducing Figure 4", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P033-T10", "P033-T11", "P033-T18"], "text": {"as_printed": "We use the \\hl{sample variance} of the outcome among control units for the standardization. The boundary for statistical significance at the 5\\% level is a line emanating from the origin with a slope of \\hl{1.96}, here drawn in red, where estimates to the right of this line attain statistical significance at the conventional level.", "corrected": "We use the \\hl{sample standard deviation} of the outcome among control units for the standardization. The boundary for statistical significance at the 5\\% level is a line emanating from the origin with a slope of \\hl{$1/z_{.975}\\approx1/1.96$}, here drawn in red, where estimates to the right of this line attain statistical significance at the conventional level.\n\nHere the horizontal axis is the absolute standardized estimate $|\\widehat\\tau|/s_0$ and the vertical axis is its standardized standard error $\\widehat{\\mathrm{SE}}/s_0$, as in the published figure.", "evidence": "The evidence is a derivation, an exact counterexample to the printed slope, and a check of the code and of one study in the replication files. Let $s_0>0$ be the sample standard deviation of the outcome among control units, $\\widehat\\tau$ an estimate and $\\widehat{\\mathrm{SE}}>0$ its standard error. The figure plots\n\\[\nx=|\\widehat\\tau|/s_0,\\qquad y=\\widehat{\\mathrm{SE}}/s_0,\n\\]\nso that $x/y=|\\widehat\\tau|/\\widehat{\\mathrm{SE}}=|t|$. Let $z=z_{.975}$. Since $y>0$ and $z>0$, I have $|t|\\ge z$ if and only if $y\\le x/z$. The boundary is the line $y=x/z$ through the origin, with slope $1/z\\approx0.5102$, and the significant estimates lie to its right.\n\nFor the counterexample, let $z=1.96=49/25$. The point $(x,y)=(1/2,1/2)$ has $|t|=1<1.96$. At height $y=1/2$ the line of slope $49/25$ passes through $x=25/98<1/2$. The point lies to the right of that line and is not significant.\n\nThe plotting script of the replication files draws the line with slope \\texttt{1/qnorm(0.975)} and divides both coordinates by \\texttt{y0\\_sd}. The analysis script computes this divisor with \\texttt{sd()} over the observations below the cutoff. For Boas and Hidalgo (2011), these are 568 observations with 82 ones and 486 zeros. Their sample variance is $123/994\\approx0.124$, and the saved divisor agrees with $\\sqrt{123/994}\\approx0.3518$. I checked this one study only.", "problem": "The paragraph misstates two features of Figure~4. The figure and the code that draws it are right. First, the estimates and their standard errors are divided by the sample standard deviation of the outcome among observations below the cutoff, not by the sample variance. Second, the figure has the standardized estimate on the horizontal axis and the standardized standard error on the vertical axis. On these axes the line $|t|=1.96$ has slope $1/1.96$, the reciprocal of the printed value. A line of slope $1.96$, with the rule that estimates to its right are significant, would classify estimates with $|t|<1.96$ as significant.", "what_survives": "Figure~4, its red line, the estimates, the standard errors and every classification as significant are unchanged. The power analysis uses its own standard deviation, that of control units within the default bandwidth (p.~9), which this correction does not touch. Only the two terms in the description change."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E003", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Table 2. A red outline marks the percent sign that precedes the words 'Effect size' in the heading of the first column.", "caption": "Table 2, p. 9 (PDF p. 9). The marked character is the percent sign in the heading of the first column.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E005.png"}, {"alt": "Crop of Table 3 with its two panels, 'Studies using election data' and 'Studies not using election data'. In each panel a red outline marks the percent sign that precedes the words 'Effect size'.", "caption": "Table 3, p. 10 (PDF p. 10), both panels. The marked characters are the percent signs in the two headings of the first column.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E005-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The first column of Tables 2 and 3 gives effect sizes in units of Cohen’s $d$, and the percent sign in its heading does not belong there.", "id": "P033-E005", "location": "Table 2, p. 9 (PDF p. 9), and both panels of Table 3, p. 10 (PDF p. 10): heading of the first column", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "First-column heading of Table~2 and of both panels of Table~3: ``\\hl{\\%} Effect size'', above the rows 0.1, 0.2, 0.5 and 0.8.", "corrected": "``Effect size \\hl{(Cohen's $d$)}'', without the percent sign, in Table~2 and in both panels of Table~3. Deleting the percent sign alone is also correct.", "evidence": "The evidence is a check of the pages and of the replication files. The table files there carry the heading ``Effect size''. The analysis script multiplies each value of $d$ by the standard deviation of the outcome and does not convert it to a percentage.", "problem": "The rows of the first column are the four values of Cohen's $d$ defined on p.~9, in units of the standard deviation of the outcome. They are not percentages. The percent sign belongs only to the columns, which are the power thresholds 60, 80 and 95 under the heading ``Power (\\%)''. No entry of the tables needs rescaling.", "what_survives": "All entries of the tables, the power calculations and the comparisons drawn from them are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E005", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the paragraph that begins 'The results of the reanalysis are largely unchanged'. A red outline marks the words 'Section S8' in its second sentence.", "caption": "p. 8 (PDF p. 8), left column. The marked reference is “Section S8”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E006.png"}, {"alt": "Crop of a supplement page with the heading 'S3 Yearly number of published RD studies', Table S2 with total 45, Table S3 with total 46, and a four-line note. Red outlines mark 'Table' at the end of one line with 'A3' at the start of the next, then 'Table A3', then 'Table A2'.", "caption": "Online supplement, p. 8: Tables S2 and S3 and the note below them. The marked labels are “Table A3” (twice) and “Table A2”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The rdhonest results are in Section S9 of the supplement, not Section S8, and the note below Table S3 calls Tables S3 and S2 “Table A3” and “Table A2”.", "id": "P033-E006", "location": "p. 8 (PDF p. 8), left column, paragraph on the rdhonest procedure; online supplement, p. 8, note below Table S3", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Main text, p.~8: ``\\hl{Section S8} in the supplement reports the results using this alternative approach.''\n\nSupplement, p.~8: ``Therefore, we count this article twice in \\hl{Table A3}, once for each bandwidth selection, and so \\hl{Table A3} includes one more unit (46) than \\hl{Table A2} (45).''", "corrected": "Main text, p.~8: ``\\hl{Section S9} in the supplement reports the results using this alternative approach.''\n\nSupplement, p.~8: ``Therefore, we count this article twice in \\hl{Table S3}, once for each bandwidth selection, and so \\hl{Table S3} includes one more unit (46) than \\hl{Table S2} (45).''", "evidence": "The evidence is a check of the page for the main text. For the supplement I had the text of the official file and the page image shown above, not the PDF itself. The headings of Sections S8 and S9 and the titles and totals of Tables S2 and S3 are as stated. The table files of the replication files carry the same two tables.", "problem": "Two cross-references point to the wrong place. Section~S8 of the supplement is ``All studies with replication data''. The results of the rdhonest reanalysis are in Section~S9, ``Re-analysis using rdhonest package''. The reference to Section~S8 on p.~7, for the reanalysis of all 36 studies, is right. The supplement has no Table A2 or A3. The note stands below Tables S2 and S3 and explains their totals, 45 and 46.", "what_survives": "Correcting these labels changes no number or result."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E006", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of the left column of page 4. A red box marks the word 'worse-case' in 'with the worse-case bias consistent with the observed data'. The next sentence begins 'The worst-case is taken with respect to a stipulated class of functions'.", "caption": "Paragraph beginning “The third approach”, p. 4 (PDF p. 4), left column. The marked word should read “worst-case”, as in the next sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008.png"}, {"alt": "Crop of five lines of the left column of page 11. A red box marks the words 'it as possible,' in the sentence 'However, insofar as it as possible, having a full record of analyses that were intended would be helpful for the accumulation of knowledge.'", "caption": "Paragraph on preregistration, p. 11 (PDF p. 11), left column. The marked words should read “it is possible”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008-b.png"}, {"alt": "Crop of the heading 'S6 Reproduction of originally reported estimates' and the two paragraphs under it. Red boxes mark the words 'This is section' at the start of the first paragraph and the words 'the manually feed' in the entry for de Benedictis-Kessner and Warshaw (2016).", "caption": "Supplement, Section S6, p. 12. The first marked phrase should read “In this section” and the second “then manually feed”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008-c.png"}, {"alt": "Crop of the heading 'S9 Re-analysis using rdhonest package' and the paragraph under it. A red box marks the words 'under under' at the start of the third line, before 'second order Hölder class'.", "caption": "Supplement, Section S9, p. 16, first paragraph. The marked word is printed twice.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008-d.png"}], "formal_release": null, "formal_statements": [], "headline": "“worse-case” should read “worst-case”, “insofar as it as possible” should read “insofar as it is possible”, “This is section” should read “In this section”, “the manually feed” should read “then manually feed”, and one “under” in “under under” should be deleted.", "id": "P033-E008", "location": "p. 4 (PDF p. 4), left column; p. 11 (PDF p. 11), left column; Supplement, Section S6, p. 12, and Section S9, p. 16", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item p.~4: ``The third approach is to adjust the inference procedure with the \\hl{worse-case} bias consistent with the observed data.''\n\\item p.~11: ``However, insofar as it \\hl{as} possible, having a full record of analyses that were intended would be helpful for the accumulation of knowledge.''\n\\item Supplement, p.~12: ``\\hl{This is section} we document all failures to reproduce the authors' original estimates during our replication.''\n\\item Supplement, p.~12: ``When reproducing their estimates we select the bandwidth through the \\texttt{rdbwselect\\_2014} for backward compatibility purposes, and \\hl{the} manually feed this bandwidth into the \\texttt{rdrobust} package.''\n\\item Supplement, p.~16: ``\\ldots to estimate a bound on the second derivative for inference under \\hl{under} second order Hölder class \\ldots''\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item p.~4: ``The third approach is to adjust the inference procedure with the \\hl{worst-case} bias consistent with the observed data.''\n\\item p.~11: ``However, insofar as it \\hl{is} possible, having a full record of analyses that were intended would be helpful for the accumulation of knowledge.''\n\\item Supplement, p.~12: ``\\hl{In this section} we document all failures to reproduce the authors' original estimates during our replication.''\n\\item Supplement, p.~12: ``\\ldots and \\hl{then} manually feed this bandwidth into the \\texttt{rdrobust} package.''\n\\item Supplement, p.~16: ``\\ldots for inference \\hl{under} second order Hölder class \\ldots''\n\\end{itemize}", "evidence": "I checked the pages for the two phrases of the main text. The sentence after the first one prints ``The worst-case is taken with respect to a stipulated class of functions'', and p.~8 prints ``worst-case bias adjustment''. I compared the three supplement phrases in its extracted text with the marked images of those passages accompanying this entry; they agree. I did not have the typeset supplement pages for comparison.", "problem": "Each of the five phrases has a wrong, missing or repeated word. The meaning is clear in every case.", "what_survives": "Everything else is unaffected, since no statement of the paper depends on these words."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E008", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of the right column of page 4. Red boxes mark the words 'between 2009 and' at the end of one line and '2018.' at the start of the next, in the sentence 'We collected all studies using an RD design published in the American Political Science Review, American Journal of Political Science, and Journal of Politics between 2009 and 2018.'", "caption": "Section “The state of RD studies in political science”, first sentence, p. 4 (PDF p. 4), right column. The marked words give the sample period, without saying how an article is dated.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E009.png"}, {"alt": "Crop of one entry of the reference list of the supplement: 'Schickler, E., Pearson, K., and Feinstein, B. D. (2010). Congressional parties and civil rights politics from 1933 to 1972. The Journal of Politics, 72(3), 672-689.' A red box marks the year '(2010)'.", "caption": "Supplement, reference list, p. 23. The supplement itself cites the study with the marked year 2010.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Tables S2 and S3 count the study of Schickler, Pearson and Feinstein in 2009, but it was published in 2010, and the 2009 and 2010 entries of four rows are each off by one.", "id": "P033-E009", "location": "Supplement, Section S3, Tables S2 and S3, p. 8; the sample period is stated on p. 4 (PDF p. 4) of the main text, right column", "pdf_page": null, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Main text, p.~4: ``We collected all studies using an RD design published in the \\emph{American Political Science Review}, \\emph{American Journal of Political Science}, and \\emph{Journal of Politics} between 2009 and 2018.''\n\n\\begin{audittable}{llrrr}{Table~S2 (Yearly Published RD Articles) and Table~S3 (Bandwidth Selection in Sample of RD Articles), Supplement p.~8: the columns 2009, 2010 and Total, as printed. The columns 2011 to 2018 are not affected.}\nTable & Row & 2009 & 2010 & Total \\\\\nS2 & APSR & 1 & 0 & 19 \\\\\nS2 & AJPS & 0 & 0 & 10 \\\\\nS2 & JOP & \\hl{1} & \\hl{0} & 16 \\\\\nS2 & All & \\hl{2} & \\hl{0} & 45 \\\\\nS3 & Automated (e.g. CCT and IK) & 1 & 0 & 22 \\\\\nS3 & Non-automated & \\hl{1} & \\hl{0} & 20 \\\\\nS3 & Global Polynomial & 0 & 0 & 4 \\\\\nS3 & All & \\hl{2} & \\hl{0} & 46 \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{llrrr}{Tables~S2 and~S3: the columns 2009, 2010 and Total, corrected. The columns 2011 to 2018 are as printed.}\nTable & Row & 2009 & 2010 & Total \\\\\nS2 & APSR & 1 & 0 & 19 \\\\\nS2 & AJPS & 0 & 0 & 10 \\\\\nS2 & JOP & \\hl{0} & \\hl{1} & 16 \\\\\nS2 & All & \\hl{1} & \\hl{1} & 45 \\\\\nS3 & Automated (e.g. CCT and IK) & 1 & 0 & 22 \\\\\nS3 & Non-automated & \\hl{0} & \\hl{1} & 20 \\\\\nS3 & Global Polynomial & 0 & 0 & 4 \\\\\nS3 & All & \\hl{1} & \\hl{1} & 46 \\\\\n\\end{audittable}\n\nIf Cavaillé and Marshall is retained in 2018, Section~S3 should state the dating rule: \\hl{Articles are counted in the year of their first online publication. Under this rule Cavaillé and Marshall (2019) is counted in 2018.}", "evidence": "This correction rests on a recalculation from the replication files and on the publisher's dates. When the table code is run unchanged, it reproduces Tables~S2 and~S3. When only the year of this study is changed to 2010, the eight highlighted entries change and no others, and Tables~2, 3 and~S6, which the same script produces, do not change. The entries of Tables~S2 and~S3 are taken from the extracted text of the supplement and from the tables in the replication files, which agree; I did not have the typeset supplement pages.", "problem": "The two tables count Schickler, Pearson and Feinstein in 2009. Table~S1 and the reference list of the supplement cite the study as 2010, in \\emph{The Journal of Politics} 72(3), 672--689. According to the publisher, the article appeared online on 12 July 2010 and in the print issue of July 2010, and no date in 2009 is listed. The study therefore belongs in 2010 whether an article is dated by its first online publication or by its issue. In the replication code the year is read from the internal label of the article, \\texttt{schickler\\_etal\\_2009}. The study is in the JOP row of Table~S2, the Non-automated row of Table~S3 and both All rows. In each of these four rows the 2009 entry is one too high and the 2010 entry is one too low.\n\nA second study is counted in a year other than its cited year. Cavaillé and Marshall is cited as 2019, in \\emph{American Political Science Review} 113(1), and is counted among the seven APSR articles of 2018. It appeared online on 15 November 2018 and in the issue of February 2019. Its count in 2018, and its place in the period 2009--2018, are correct if articles are dated by first online publication. The paper does not state this rule.", "what_survives": "All row totals, the sample of 45 studies, and every estimate and weight are unchanged, since no analysis uses the year of publication. Only the eight annual counts change."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E009", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the paragraph that describes the power analysis. A red outline marks the words 'robust standard errors as implemented by rdrobust', which run over two lines.", "caption": "Power analysis, p. 9 (PDF p. 9), left column. The marked words name the standard error that the power calculation is said to use.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E010.png"}, {"alt": "Crop of Table 2, the proportion of analyses achieving 60, 80 and 95 percent power by effect size. Red outlines mark the cells 0.65 (effect size 0.5, power 60), 0.78 (effect size 0.8, power 60) and 0.65 (effect size 0.8, power 95).", "caption": "Table 2, p. 9 (PDF p. 9). The marked cells are the three that change.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E010-b.png"}, {"alt": "Crop of the first panel of Table 3, 'Studies using election data'. Red outlines mark the cells 0.59 (effect size 0.5, power 60), 0.78 (effect size 0.8, power 60) and 0.59 (effect size 0.8, power 95).", "caption": "Table 3, p. 10 (PDF p. 10), panel “Studies using election data”. The marked cells are the three that change.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E010-c.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T9", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T9"}, {"id": "P033-T17", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T17"}], "headline": "For four clustered estimates, the saved powers match a calculation with a standard error smaller than the robust standard error of rdrobust; with the robust standard error, six cells of Tables 2 and 3 fall by $0.02$ to $0.04$.", "id": "P033-E010", "location": "Power analysis, p. 9 (PDF p. 9), left column; Table 2, p. 9; Table 3, p. 10 (PDF p. 10), panel “Studies using election data”", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P033-T17", "P033-T9"], "text": {"as_printed": "p.~9: ``The power analyses presume that $p$-values will be constructed with the bias adjustment and \\hl{robust standard errors as implemented by rdrobust}.''\n\n\\begin{audittable}{lrrr}{Table 2, p.~9, rows 0.5 and 0.8, as printed.}\n\\% Effect size & 60 & 80 & 95 \\\\\n0.5 & \\hl{0.65} & 0.56 & 0.48 \\\\\n0.8 & \\hl{0.78} & 0.70 & \\hl{0.65} \\\\\n\\end{audittable}\n\n\\begin{audittable}{lrrr}{Table 3, p.~10, studies using election data, rows 0.5 and 0.8, as printed.}\n\\% Effect size & 60 & 80 & 95 \\\\\n0.5 & \\hl{0.59} & 0.46 & 0.40 \\\\\n0.8 & \\hl{0.78} & 0.66 & \\hl{0.59} \\\\\n\\end{audittable}", "corrected": "The sentence of p.~9 stands as the description of the calculation. For the four estimates, the power at effect size $d$ is\n\\[\n\\Phi\\bigl(d\\,\\sigma_0/s-z_{.975}\\bigr)+\\Phi\\bigl(-d\\,\\sigma_0/s-z_{.975}\\bigr),\n\\]\nwhere $s$ is the robust standard error of the \\texttt{rdrobust} fit and $\\sigma_0$ is the standard deviation of the outcome among control units within the default bandwidth. The bandwidths, the effect sizes and the study weights are as in the paper, and all other saved powers are held fixed. This is a partial correction: the seven further clustered estimates are not recomputed.\n\n\\begin{audittable}{lrrr}{Table 2, rows 0.5 and 0.8, corrected for the four estimates.}\nEffect size & 60 & 80 & 95 \\\\\n0.5 & \\hl{0.62} & 0.56 & 0.48 \\\\\n0.8 & \\hl{0.76} & 0.70 & \\hl{0.62} \\\\\n\\end{audittable}\n\n\\begin{audittable}{lrrr}{Table 3, studies using election data, rows 0.5 and 0.8, corrected for the four estimates.}\nEffect size & 60 & 80 & 95 \\\\\n0.5 & \\hl{0.55} & 0.46 & 0.40 \\\\\n0.8 & \\hl{0.74} & 0.66 & \\hl{0.55} \\\\\n\\end{audittable}", "evidence": "The evidence is a recalculation from the replication files. Under the normal approximation of p.~9, the test statistic has distribution $N(\\tau/s,1)$ when the effect is $\\tau$. With $a=|\\tau|/s$ and $z=z_{.975}$, the power of the two-sided 5\\% test is\n\\[\nf(a)=\\Phi(a-z)+\\Phi(-a-z),\\qquad \\tau=d\\,\\sigma_0,\n\\]\nwhere $\\sigma_0$ is the standard deviation of the outcome among control units within the bandwidth. For $a>0$, I have $f'(a)=\\varphi(a-z)-\\varphi(a+z)>0$, which implies that the power is strictly larger for a smaller standard error. I had the data of 45 of the 64 estimates. Fitting \\texttt{rdrobust} again for these 45 reproduces the saved coefficients and standard errors to a relative error of $4.3\\times10^{-15}$. The 180 saved powers of these estimates equal $f(|\\tau|/(s\\sqrt{\\rho}))$ to within $7\\times10^{-16}$, where $\\rho=1$ without clusters and $\\rho=G/(G_L+G_R)$ with clusters.\n\n\\begin{audittable}{lrrrr}{The four estimates: cluster counts and the ratio of the squared standard error used for power to $s^2$.}\nEstimate & $G$ & $G_L$ & $G_R$ & $G/(G_L+G_R)$ \\\\\nGerber and Hopkins (2011) & 57 & 23 & 46 & $19/23$ \\\\\nHall and Thompson (2018) & 185 & 98 & 101 & $185/199$ \\\\\nLopes da Fonseca (2017) & 278 & 202 & 226 & $139/214$ \\\\\nSances (2017) & 204 & 133 & 130 & $204/263$ \\\\\n\\end{audittable}\n\n\\begin{audittable}{lrrr}{The three threshold crossings: saved power and the power $f(\\tau/s)$ with the robust standard error.}\nEstimate and effect size & Saved & With $s$ & Threshold crossed \\\\\nLopes da Fonseca (2017), $d=0.5$ & $0.774$ & $0.589$ & $0.60$ \\\\\nLopes da Fonseca (2017), $d=0.8$ & $0.991$ & $0.938$ & $0.95$ \\\\\nSances (2017), $d=0.8$ & $0.663$ & $0.554$ & $0.60$ \\\\\n\\end{audittable}\n\nThe powers of Gerber and Hopkins and of Hall and Thompson are also too large, by at most $0.04$, and cross no threshold. The largest reduction among the four estimates is $0.185$. As exact fractions, the Table~2 cells change from $31/48$ to $89/144$, from $113/144$ to $109/144$ and from $31/48$ to $89/144$, and the Table~3 cells from $61/104$ to $57/104$, from $81/104$ to $77/104$ and from $61/104$ to $57/104$. All 36 printed cells are reproduced from the saved powers before the change.\n\nTwo limits apply. The source of the \\texttt{rdpower} package was not available, and I do not say which of its lines produce the factor. The data files of seven further clustered estimates, from Bohlken (2018), Gulzar and Pasquale (2017) and Larreguy et al.\\ (2016), belong to the replication files, but I did not have them. Their powers were not recomputed, and further cells may change. Note that the estimate of Broockman and Ryan (2016) also has clusters on both sides ($G=380$, $G_L=379$, $G_R=380$), but its powers round to $1$ with either standard error. Its standard error is the subject of \\fid{P033-N011}.", "problem": "The paper says that the power calculation uses the robust standard error of \\texttt{rdrobust}. For four sharp clustered estimates, from Gerber and Hopkins (2011), Hall and Thompson (2018), Lopes da Fonseca (2017) and Sances (2017), the saved power does not match this description. Let $s$ be the robust standard error of the \\texttt{rdrobust} fit. Let $G$ be the number of distinct clusters, and let $G_L$ and $G_R$ be the numbers of clusters with observations to the left and to the right of the cutoff. All counts are taken after incomplete observations are removed. Each saved power equals the power under the normal approximation with the standard error\n\\[\ns\\,\\sqrt{G/(G_L+G_R)}\n\\]\nin place of $s$. If there exists a cluster with observations on both sides of the cutoff, I have $G<G_L+G_R$. The standard error used for power is then smaller than $s$, and the power is too large. Two of the four estimates cross a threshold of the tables. Each has weight $1$, and each crossing lowers a cell of Table~2 by $1/36$ and a cell of the election panel of Table~3 by $1/26$.", "what_survives": "Coefficients, standard errors, $p$-values, $t$-statistics, the caliper tests and the funnel plots do not use the saved powers. The other 30 cells are unchanged by this correction, including the 80\\% column, so that the percentages quoted in the text (22\\%, 56\\% and 70\\%) stand. The correction lowers power and reinforces the conclusion that the studies are underpowered. It restores the standard error that the paper describes. It does not show that this standard error is right for every clustered design."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E010", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of the right column of page 4. A red box marks the words 'among all 45 studies in our sample.' in the sentence 'Figure 1 presents the distribution of t-statistics for the published findings among all 45 studies in our sample.'", "caption": "Paragraph beginning “Figure 1 presents the distribution”, p. 4 (PDF p. 4), right column. The marked words give the sample that the figure is said to show.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N002.png"}, {"alt": "Crop of Figure 1 with its caption: two histograms of t-statistics, panel (a) 'Main reported result' and panel (b) 'All primary results', each with a vertical line at 1.96, a horizontal axis with ticks at 0.98, 1.96, 2.94 and 3.92, and a vertical axis from 0 to 2. Red boxes mark the vertical-axis label 'Density' of panel (a) and the tick label '3.92' at the right end of its horizontal axis.", "caption": "Figure 1, p. 5 (PDF p. 5). The marked labels of panel (a) are the vertical axis, “Density”, and the last tick of the horizontal axis, $3.92$. The panel shows 37 of the 45 main results.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The histograms of Figures 1, 2, 3 and 5 leave out every $t$-statistic above $3.92$ and scale each density to the weight that remains, and each panel of Figure 4 leaves out 3 or 4 of the 39 estimates; the text and the captions do not say so.", "id": "P033-N002", "location": "p. 4 (PDF p. 4), right column, paragraph beginning “Figure 1 presents the distribution”; p. 7 (PDF p. 7); Figures 1, 2, 3, 4 and 5 (pp. 5, 6, 8 and 10)", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~4: ``Figure~1 presents the distribution of $t$-statistics for the published findings \\hl{among all 45 studies in our sample}. The first panel presents the distribution of the main reported result of each article, meaning that each article contributes one $t$-statistic.''\n\np.~7: ``Figure~3 presents the distribution of $t$-statistics \\hl{among the 39 estimates} from the 25 studies included in the main replication exercise.''\n\np.~7: ``Figure~4 presents such a funnel plot \\hl{for the 39 estimates} in the replication study.''\n\nVertical axis of panels (a) of Figures~1, 2 and~3: ``\\hl{Density}''.", "corrected": "The following note should accompany Figures~1, 2, 3 and~5. \\hl{Each histogram shows the $t$-statistics in the interval from $0$ to $2\\Phi^{-1}(0.975)$, approximately $[0,3.92]$. The density is normalized to the weight in that interval, separately in each panel. The panels leave out 8 of 45, 12 of 80, 4 of 42, 8 of 38, 9 of 39, 10 of 39, 10 of 39, 7 of 47 and 5 of 33 statistics in Figures~1(a), 1(b), 2(a), 2(b), 3(a), 3(b), 3(c), 5(a) and 5(b), which is 17.8\\%, 16.1\\%, 13.6\\%, 18.5\\%, 21.0\\%, 26.0\\%, 29.0\\%, 13.7\\% and 21.4\\% of the weight.}\n\nThe following note should accompany Figure~4. \\hl{The panels show the estimates within the axis limits. Panels (a), (b) and (c) leave out 3, 3 and 4 of the 39 estimates.}\n\nIf the heights are instead to describe the full sample, each printed bar is multiplied by $V/W$, the reciprocal of the factor in the table above, and the weight outside the interval is reported.", "evidence": "This correction rests on a recalculation from the replication files and on a measurement of the published figure. The table lists what each panel leaves out, computed from the stored $t$-statistics and the weights of the paper.\n\n\\begin{audittable}{lrrrr}{Histogram panels: statistics that enter the code, statistics above the axis limit, share of weight left out, and the factor $W/V$ by which the printed bars exceed a density of the full sample.}\nPanel & Statistics & Left out & Share of weight left out & $W/V$ \\\\\n1(a) & 45 & 8 & $8/45=17.8\\%$ & $45/37=1.216$ \\\\\n1(b) & 80 & 12 & $29/180=16.1\\%$ & $180/151=1.192$ \\\\\n2(a) & 42 & 4 & $3/22=13.6\\%$ & $22/19=1.158$ \\\\\n2(b) & 38 & 8 & $17/92=18.5\\%$ & $92/75=1.227$ \\\\\n3(a) & 39 & 9 & $21/100=21.0\\%$ & $100/79=1.266$ \\\\\n3(b) & 39 & 10 & $13/50=26.0\\%$ & $50/37=1.351$ \\\\\n3(c) & 39 & 10 & $29/100=29.0\\%$ & $100/71=1.408$ \\\\\n5(a) & 47 & 7 & $17/124=13.7\\%$ & $124/107=1.159$ \\\\\n5(b) & 33 & 5 & $3/14=21.4\\%$ & $14/11=1.273$ \\\\\n\\end{audittable}\n\nThe bars of Figure~1(a) in the published PDF represent 37 results in total. A bar that holds one result has measured height $0.2482$. This agrees with $1/(37w)=0.2482$ and not with $1/(45w)=0.2041$. The figures in the replication files contain 36, 36 and 35 points for the three panels of Figure~4.", "problem": "The code for the histograms sets the horizontal axis to the interval from $0$ to $2\\Phi^{-1}(0.975)$, which is approximately $[0,3.92]$. A $t$-statistic above this limit enters no bin, and the density of each panel is then normalized to the weight that remains in the interval. Figure~1(a) shows 37 of the 45 main results, and each of its bars is higher by the factor $45/37$ than it would be in a density of all 45. Let $W$ be the total weight of the sample named in the text, let $V$ be the weight in the interval, and let $B_j$ be the weight in bin $j$, where each bin has width $w=\\Phi^{-1}(0.975)/18$. The printed height of bin $j$ is $B_j/(wV)$, and the height in a density of the full sample is $B_j/(wW)$. The printed heights are those of the distribution conditional on $t\\le 2\\Phi^{-1}(0.975)$.\n\nThe share of weight left out differs across panels. In Figure~3 it is 21\\%, 26\\% and 29\\% in panels (a), (b) and (c). The three panels that the text compares are conditioned on different sets. The funnel plots of Figure~4 have axis limits $0\\le x\\le 2$ and $0\\le y\\le 0.8$, and panels (a), (b) and (c) leave out 3, 3 and 4 of the 39 estimates. The main text, the captions and the text of the supplement state neither the omissions nor the normalization. The range of each axis is visible in the figures, and the counts 45, 80 and 39 given in the text are the correct sizes of the samples that enter the code.", "what_survives": "Within a panel, the order of the bars, the position of the spike near $1.96$ and the ratios between bars are unchanged, since every bar of the panel is rescaled by the same constant. Table~1 and Tables~S4 and~S5 use their designated main-estimate samples without these plotting limits, and their windows end below the axis limit. Tables~2 and~3 use all 64 analyses. In the full data the weighted share of $t$-statistics at or above $1.96$ is $0.77$, $0.63$ and $0.49$ for the samples of Figure~3(a), (b) and (c), so this decline in the share of significant results holds without the axis limit. The absolute scale of the densities and comparisons of bar heights between panels do change."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N002", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of the right column of page 4. A red box marks the words 'reported p-values,' in the sentence 'For the small number of the studies where neither t-statistics nor standard errors were reported, we reconstruct t-statistics using reported p-values, confidence intervals and point estimates, or using replication code and data provided by the authors.'", "caption": "End of the paragraph on Figure 1, p. 4 (PDF p. 4), right column. The marked words name a source that the replication code does not use to reconstruct a $t$-statistic.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N003.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T14", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T14"}, {"id": "P033-T15", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T15"}], "headline": "The replication code reconstructs no $t$-statistic from a reported $p$-value, and it computes a standard error from a confidence interval as the full width divided by twice the critical value, not from the upper limit and the point estimate as Section S4 says.", "id": "P033-N003", "location": "p. 4 (PDF p. 4), right column, last sentence of the paragraph beginning “Figure 1 presents the distribution”; Supplement, Section S4, pp. 9–10", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P033-T14", "P033-T15"], "text": {"as_printed": "Main text, p.~4: ``For the small number of the studies where neither $t$-statistics nor standard errors were reported, we reconstruct $t$-statistics using \\hl{reported $p$-values,} confidence intervals and point estimates, or using replication code and data provided by the authors.''\n\nSupplement, Section~S4, entry for Ariga (2015): ``The standard error estimate was imputed by (a) \\hl{subtracting the point estimate value from the upper value} of the reported 90\\% confidence interval, and then (b) dividing that value by the critical value for a 90\\% confidence interval (1.64).''\n\nSupplement, Section~S4, entries for Holbein (2016) and for Kalšnja and Titiunik (2017), as named in the extracted text: ``The standard error was not explicitly listed and so we imputed by using \\hl{the upper bound of a 95\\% CI and the point estimate} along with the critical value of 1.96.''", "corrected": "Main text, p.~4: ``For the small number of the studies where neither $t$-statistics nor standard errors were reported, we reconstruct $t$-statistics using \\del{reported $p$-values,} confidence intervals and point estimates, or using replication code and data provided by the authors.''\n\nSupplement, Section~S4, entry for Ariga (2015): the standard error is \\hl{the width $U-L$ of the reported 90\\% confidence interval $[L,U]$ divided by $2\\Phi^{-1}(0.95)$}.\n\nSupplement, Section~S4, entries for Holbein (2016) and for Klašnja and Titiunik (2017): the standard error is \\hl{the width $U-L$ of the reported 95\\% confidence interval $[L,U]$ divided by $2\\Phi^{-1}(0.975)$}.\n\nIn each case the $t$-statistic is the absolute point estimate divided by this standard error. Reported $p$-values are recorded separately, and a missing $p$-value is derived from the $t$-statistic by the normal approximation.", "evidence": "This correction rests on a derivation, an exact example and a recalculation from the replication files. I checked the Supplement Section~S4 quotations against its extracted text; I did not have the typeset supplement pages. Set $s_W=(U-L)/(2q)$ and $s_U=(U-b)/q$. Then\n\\[\n2q\\,(s_W-s_U)=(U-L)-2(U-b)=2b-U-L,\n\\]\nwhich implies that $s_W=s_U$ if and only if $b=(U+L)/2$. For Klašnja and Titiunik the recorded values are $b=-0.15$ and $[L,U]=[-0.213,-0.101]$. Then $U-L=0.112$ and $U-b=0.049$, and I obtain $s_W/s_U=8/7$ for every $q>0$. At the printed critical value $q=1.96$, I have $s_W=1/35$ and $s_U=1/40$, and the absolute $t$-statistics are $21/4=5.25$ and $6$. The replication code, which uses $q=\\Phi^{-1}(0.975)$, gives $5.2499$.\n\nOf the 81 rows of the raw metadata, 65 report a standard error, 1 reports a $t$-statistic, 2 use a 90\\% interval, 10 use a 95\\% interval and 3 use replication data. After every reported $p$-value is erased from the raw metadata, the cleaning code returns the same 81 standard errors and $t$-statistics.", "problem": "The description of how missing $t$-statistics were reconstructed differs from the replication code in two respects. First, the code fills a missing standard error from a reported $t$-statistic, from a 90\\% or 95\\% confidence interval, or from the replication data, and then computes every missing $t$-statistic as the absolute estimate divided by the standard error. No $t$-statistic is recovered from a reported $p$-value. Second, for a reported interval $[L,U]$ and point estimate $b$, the code computes the standard error as $(U-L)/(2q)$, where $q$ is $\\Phi^{-1}(0.95)$ for a 90\\% interval and $\\Phi^{-1}(0.975)$ for a 95\\% interval. Section~S4 instead describes the rule $(U-b)/q$ for Ariga, for Holbein, and for Klašnja and Titiunik. The two rules agree only when the interval is centered on the estimate. Of the six estimates to which Section~S4 applies this rule, four have a recorded interval that is not exactly centered. For three of them the two standard errors differ by less than 1\\%, and for Klašnja and Titiunik they differ by the factor $8/7$.", "what_survives": "Correcting the description changes no numerical result, and the cleaning code reproduces the cleaned data exactly. The reanalysis, its caliper tests and the power calculations do not use the imputed standard errors. If the rule of Section~S4 were applied instead, with its printed critical values, Tables~1 and~S4 would keep their counts and $p$-values. One estimate of Holbein (2016) would move by one histogram bin in Figures~1(b), 2(b) and~5(b), and six entries of Table~S6 would change."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N003", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of nine lines of the left column of page 5. Red boxes mark the words 'null hypothesis of no real pathologies' and, two lines below, 'equal success probability,' in the sentence 'We calculate the probability of observing an imbalance at least this extreme under the null hypothesis of no real pathologies in the RD literature by using a binomial distribution with equal success probability'.", "caption": "Paragraph beginning “Table 1 presents the results from the caliper tests”, p. 5 (PDF p. 5), left column. The first marked phrase is the null hypothesis that is named, and the second is the probability that the calculation uses.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N004.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T7", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T7"}, {"id": "P033-T8", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T8"}], "headline": "The $p$-values of Table 1 are binomial tails under equal probability mass on the two sides of $1.96$ within each window, and this condition does not follow from the absence of selection on significance alone.", "id": "P033-N004", "location": "p. 5 (PDF p. 5), left column, paragraph beginning “Table 1 presents the results from the caliper tests”, and Table 1", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P033-T7", "P033-T8"], "text": {"as_printed": "We calculate the probability of observing an imbalance at least this extreme under the \\hl{null hypothesis of no real pathologies} in the RD literature by using a binomial distribution with \\hl{equal success probability}, capturing that we would expect a study to be equally likely to be just above and below the threshold under the null. [\\ldots] The $p$-values of the caliper tests range between 3\\% and 7\\%, depending on the window size, indicating that it is unlikely that we would observe this large of an imbalance in the $t$-statistics if there were no real pathologies in the RD literature.", "corrected": "The first of the two sentences quoted above is replaced by the following. ``\\hl{Conditional on the number of studies in each caliper, we calculate one-sided binomial tail probabilities under a working null of equal probability mass above and below the threshold. Interpreting these probabilities as evidence of publication bias requires an additional justification that the unselected distribution has approximately equal mass on the two sides, or a sensitivity analysis allowing unequal mass. Publication independence alone does not imply the working null.}'' The second sentence, on the range of 3\\% to 7\\%, is then a statement about this working null.\n\nIn full, the statement is as follows. Let $c=1.96$ and let $\\hl{0<h<c}$ be the half-width of a window. Suppose that the studies are independent and that, in the absence of selection, the absolute $t$-statistic of each study has density $f$. Let $A_h$ and $B_h$ be the integrals of $f$ over $[c,c+h]$ and $[c-h,c]$, with $\\hl{A_h+B_h>0}$. Conditional on $n$ studies in the window $[c-h,c+h]$, the number $K$ above $c$ is binomial with parameters $n$ and $p_h=A_h/(A_h+B_h)$, and\n\\[\n\\Pr(K\\ge k\\mid n)=\\sum_{j=k}^{n}\\binom{n}{j}p_h^{\\,j}(1-p_h)^{n-j}.\n\\]\nThe entries of Table~1 are the values of this sum at $p_h=1/2$, which is the case \\hl{$A_h=B_h$}. If $f$ is nonincreasing on the window, then $p_h\\le 1/2$ and the one-sided test is conservative. If $f\\ge m>0$ and $f$ is $L$-Lipschitz on the window, then $|p_h-1/2|\\le Lh/(4m)$. Table~1 itself is unchanged.", "evidence": "This correction rests on exact counterexamples. Suppose that the studies are independent, that every study is published, and that the absolute $t$-statistic has density $f(x)=x/8$ on $[0,4]$. This density is smooth and positive on every window and has no jump at $1.96$. With $c=49/25$, a direct computation gives $A_h=(2ch+h^2)/16$ and $B_h=(2ch-h^2)/16$, so that $p_h=1/2+h/(4c)$. For $h=c/10$, $3c/20$ and $c/5$, I obtain $p_h=21/40$, $43/80$ and $11/20$. Given $n$ studies in the window, the number above $c$ is binomial with parameters $n$ and $p_h$. The table compares the upper tails at the counts of Table~1.\n\n\\begin{audittable}{lrrrr}{One-sided binomial tails at the counts of Table~1: with probability $1/2$, as printed, and with the probability $p_h$ of the density $f(x)=x/8$.}\nWindow & Below & Above & Tail with $1/2$ & Tail with $p_h$ \\\\\n10\\% & 3 & 9 & $299/4096=0.073$ & $0.100$ \\\\\n15\\% & 4 & 11 & $1941/32768=0.059$ & $0.102$ \\\\\n20\\% & 5 & 14 & $2083/65536=0.032$ & $0.078$ \\\\\n\\end{audittable}\n\nWith 19 studies in the 20\\% window, the test at the 5\\% level rejects exactly when at least 14 lie above $c$, and under this density it rejects with probability $0.0777$. In a second example each study has an unbiased normal estimate with standard error $1$ and true effect $7/2$, and every study is published. In the 20\\% window, $p_h$ then lies between $0.644747758104$ and $0.644747758105$, and the tail at 14 of 19 lies between $0.280413461677$ and $0.280413461678$. If instead every true effect is zero, then $p_h$ is approximately $0.3189$ and the one-sided test is conservative. I note that these examples are not estimates of $p_h$ for the 45 studies of the paper. For those studies I have established neither the size of the error nor its direction.", "problem": "The counts of Table~1 and its binomial tail probabilities are correct. The calculation takes each study in a window to lie above $1.96$ with probability $1/2$. Let $f$ be the density of the absolute $t$-statistic when publication does not depend on significance, set $c=1.96$, and let $h$ be the half-width of the window. Define\n\\[\nA_h=\\int_c^{c+h}f(x)\\,dx,\\qquad B_h=\\int_{c-h}^{c}f(x)\\,dx,\\qquad p_h=\\frac{A_h}{A_h+B_h}.\n\\]\nA study in the window lies above $c$ with probability $p_h$, and $p_h=1/2$ exactly when $A_h=B_h$. The null hypothesis that the 5\\% level plays no part in publication does not imply this equality at the half-widths $0.196$, $0.294$ and $0.392$ that the paper uses. If $f$ is differentiable at $c$ and $f(c)>0$, then\n\\[\np_h-\\frac12=\\frac{h\\,f'(c)}{4f(c)}+o(h),\n\\]\nwhich shows that the two counts are approximately equal as $h$ tends to zero, as the preceding paragraph of the paper says. No bound follows at a fixed $h$. The passage reads the tails of 3\\% to 7\\% as the probability of the imbalance when there is no selection, and that reading requires the further condition $A_h=B_h$, at least approximately. Equal mass on the two sides is the usual working assumption of the caliper test, which the paper takes from Gerber and Malhotra (2008a, 2008b).", "what_survives": "The counts, the imbalance of about three to one and the histograms are unchanged, and so are the entries of Table~1 when they are read as tails under equal mass on the two sides. The reanalysis and the power calculations do not use these probabilities. The reading of the 3\\% to 7\\% as the probability of the imbalance in the absence of selection does change. It now requires equal mass on the two sides, and the same holds for Tables~S4 and~S5 and for the discussion of bandwidth groups on p.~6."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N004", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the paragraph on panel 3(c). A red outline marks the words 'indicating that these studies have systematically overestimated the accuracy of their estimates.' at the end of the paragraph.", "caption": "Results, p. 7 (PDF p. 7), right column. The marked clause draws the inference from the shift in panel 3(c) to the accuracy of the original studies.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006.png"}, {"alt": "Crop of the last five lines of the abstract. A red outline marks the sentence 'However, researchers tend to use inappropriate methods for inference, rendering standard errors artificially small.'", "caption": "Abstract, p. 1 (PDF p. 1). The marked sentence states the diagnosis of the original studies.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006-b.png"}, {"alt": "Crop of the first paragraph of the concluding remarks. A red outline marks the sentence 'More than half of the studies in our sample do not properly estimate the accuracy of their estimates, leading to the associated hypothesis tests and confidence intervals being misleading.'", "caption": "Concluding remarks, p. 10 (PDF p. 10), right column. The marked sentence gives the share of studies.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006-c.png"}, {"alt": "Scatter plot on a logarithmic vertical axis. The 39 estimates are ordered by the ratio of the robust reanalysis standard error to the original standard error. Twelve open orange squares lie below the horizontal line at ratio 1: eleven between about 0.68 and 0.97, and one, labelled Broockman and Ryan, at 0.08. Twenty-seven blue circles lie above the line, rising from just above 1 to about 18.", "caption": "The comparison behind the three sentences, from the results saved in the replication files: for each of the 39 estimates of the main reanalysis, the robust standard error of the reanalysis divided by the original standard error. The lowest point is the estimate of Broockman and Ryan (2016), whose reanalysis standard error is corrected in P033-N011.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006-se-ratios.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T1", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T1"}, {"id": "P033-T2", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T2"}, {"id": "P033-T16", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T16"}], "headline": "The reanalysis shows that standardized standard errors become larger and absolute $t$-statistics smaller on average; it does not establish that the original studies used inappropriate inference, or that more than half of them did.", "id": "P033-N006", "location": "Results, p. 7 (PDF p. 7), right column, paragraph on panel 3(c); abstract, p. 1 (PDF p. 1); introduction, p. 2 (PDF p. 2); Concluding remarks, p. 10 (PDF p. 10)", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": ["P033-T1", "P033-T16", "P033-T2"], "text": {"as_printed": "Abstract, p.~1: ``However, researchers tend to use \\hl{inappropriate methods for inference, rendering standard errors artificially small}.''\n\nResults, p.~7, on panel 3(c): ``Here, we see a clear shift to the left, \\hl{indicating that these studies have systematically overestimated the accuracy of their estimates}.''\n\nConcluding remarks, p.~10: ``\\hl{More than half of the studies in our sample do not properly estimate the accuracy of their estimates}, leading to the associated hypothesis tests and confidence intervals being misleading.''", "corrected": "Abstract, p.~1: ``However, \\hl{conventional RD inference that neglects nonnegligible smoothing bias can be misleading under the usual smoothness assumptions. A standardized reanalysis often produces larger estimated standard errors and smaller absolute test statistics than the original analyses. This pattern is consistent with understated uncertainty in the original findings, but it does not by itself establish that the original inference was inappropriate}.''\n\nResults, p.~7: ``Here, we see a clear shift to the left. \\hl{This pattern is consistent with understated uncertainty in the original studies, but changes in the estimator, the bandwidth and the precision of the bias estimate also affect it; the comparison alone does not establish that the original analyses were miscalibrated}.''\n\nConcluding remarks, p.~10: ``\\hl{In our standardized reanalysis of 25 studies, the robust standard error exceeds the original standard error for 27 of 39 estimates, and the absolute test statistic is smaller for 28 of 39. Standardized standard errors are larger and absolute test statistics smaller on average. This pattern is consistent with understated uncertainty in the original studies, but the comparison alone does not establish inappropriate original inference or the fraction of studies with misleading hypothesis tests and confidence intervals}.''\n\nThe sentence of the introduction that draws the same inference (p.~2, ``This indicates that the body of published RD studies tends to systematically overestimate the accuracy of their findings'') takes the same qualification: ``This \\hl{is consistent with a tendency of} the body of published RD studies to systematically overestimate the accuracy of their findings\\hl{, but does not establish it}.''", "evidence": "The evidence is a derivation with exact examples, a comparison under bounded curvature and undersmoothing, a recalculation of the counts from the results saved in the replication files, and a check of the pages.\n\nFor the example, let the running variable $X$ be uniform on $(-5,5)$ with a sharp cutoff at $0$, and let $Y_i(0)=2+X_i+\\varepsilon_i$ and $Y_i(1)=6+X_i+\\varepsilon_i$, where the errors $\\varepsilon_i$ are independent $N(0,1)$, independent of the scores, and their variance is known. The RD effect is $4$. I condition on eight observations with scores $\\pm1,\\pm2,\\pm3,\\pm4$ and compare three estimators. Each is the difference between a fitted intercept on the right and a fitted intercept on the left. The estimator $G$ fits a line to the four points on each side. The estimator $L$ is local linear with the triangular kernel and bandwidth $4$. The estimator $R$ subtracts from $L$ an estimate of its bias from a quadratic fit with the same kernel and bandwidth. With the points on one side ordered by their distance $1,2,3,4$ from the cutoff, the intercept weights of $G$, $L$ and $R$ are\n\\[\ng=(1,\\,1/2,\\,0,\\,-1/2),\\qquad l=(3/2,\\,0,\\,-1/2,\\,0),\\qquad r=(3,\\,-3,\\,1,\\,0).\n\\]\nEach vector of weights sums to $1$ and has first moment $0$, which implies that each estimator is unbiased for $4$. Since the errors are independent with variance $1$, I have\n\\[\n\\operatorname{Var}(G)=2\\sum_i g_i^2=3,\\qquad \\operatorname{Var}(L)=2\\sum_i l_i^2=5,\\qquad \\operatorname{Var}(R)=2\\sum_i r_i^2=38.\n\\]\nEach estimator is a linear combination of independent normal errors. I conclude that $(G-4)/\\sqrt3$, $(L-4)/\\sqrt5$ and $(R-4)/\\sqrt{38}$ are standard normal. For each of the three, the interval that extends $z_{.975}$ standard errors on either side of the estimate covers the effect with probability exactly $0.95$ at these specified affine conditional means and conditional on the eight scores. The standard error rises from $1.73$ to $2.24$ to $6.16$, with pointwise coverage unchanged. For the outcomes with errors $(1/10,-1/10,1/10,-1/10)$ on each side, all three estimates equal $4$, and the squared $t$-statistics are $16/3$, $16/5$ and $8/19$. Since $16/3>(1.96)^2=2401/625>16/5>8/19$, the first estimate is significant at $1.96$ and the two replacements are not.\n\n\\begin{audittable}{lrrrr}{The example: three estimators of an RD effect equal to 4, each with 95\\% coverage at the specified affine means; no coverage guarantee over a smoothness class is asserted.}\nEstimator & Variance & Standard error & Coverage & Squared $t$ \\\\\n$G$: linear fit on each side & $3$ & $1.73$ & $0.95$ & $16/3=5.33$ \\\\\n$L$: local linear, bandwidth $4$ & $5$ & $2.24$ & $0.95$ & $16/5=3.20$ \\\\\n$R$: $L$ with bias correction & $38$ & $6.16$ & $0.95$ & $8/19=0.42$ \\\\\nThreshold $(1.96)^2$ & & & & $3.84$ \\\\\n\\end{audittable}\n\nFor the opposite case, I define $\\psi(x)=(x^2-1)(x^2-4)(x^2-9)(x^2-16)/576$. This polynomial vanishes at the eight scores, and $\\psi(0)=1$. Let the mean of $Y(1)$ be $6+x+20\\psi(x)$ instead. The effect is now $24$, and the distribution of the eight outcomes is unchanged, so that the three estimates, their standard errors and the comparison between them are unchanged. The interval $G\\pm1.96\\sqrt3$ covers $24$ only if $|G-4|>16$. By Chebyshev's inequality, this event has probability at most $3/256$. The comparison is the same in the two cases. Coverage is exactly $95\\%$ at the first, affine pair of means and at most $1.2\\%$ at the second pair. These pointwise calculations show different actual coverage, not a coverage guarantee over a common smoothness class. Even a small quadratic perturbation, $\\mathbb{E}[Y(1)\\mid X=x]=6+x+ax^2$ with $a>0$, biases $G$ by $-5a$. At $a=1/10$ its conditional coverage is about $0.9404$, although the curvature is only $1/5$. Exact coverage at the affine means therefore cannot rebut the paper's smoothness-based criticism.\n\nA further example concerns covariates. Suppose that observed covariates $Z_i\\sim N(0,3)$, independent across observations and of the scores and errors, enter both potential outcomes with known coefficient $1$. Then the estimator $G$ applied to $Y-Z$ has variance $3$, and applied to $Y$ it has variance $12$. Both intervals have exact coverage at the stipulated covariate and affine data-generating process, and omitting the adjustment doubles the standard error. This does not establish validity for an original study under its maintained assumptions. The main call to \\texttt{rdrobust} in the replication files passes no covariates, and the original estimate of Gerber and Hopkins (2011) is coded there as adjusted for covariates.\n\nThere is also a comparison with valid inference over a smoothness class. Take scores $x_{+,j}=j/n$ and $x_{-,j}=-j/n$, $j=1,\\ldots,n$, independent $N(0,1)$ errors of known variance, and twice differentiable conditional means with $|m_d''(x)|\\le M$ for a fixed finite $M$. Put $m=\\lfloor n^{3/4}\\rfloor$ and $h=(m+1)/n$. Let $A$ be the local linear difference at the cutoff, using the first $m$ observations on each side with triangular weights $1-j/(m+1)$. Let $B$ subtract its estimated quadratic Taylor bias using the same bandwidth and kernel. The weighted normal equations make $B$ the difference of the local quadratic intercepts.\n\nFor $u=j/(m+1)$, the limiting intercept weight functions are\n\\[\nK_1(u)=(6-12u)(1-u),\\qquad K_2(u)=(12-60u+60u^2)(1-u).\n\\]\nTheir squared integrals are $24/5$ and $72/7$. Both fits reproduce constants and lines and have bounded sums of absolute intercept weights, so Taylor's theorem bounds each effect estimator's bias by a constant times $Mh^2$. Their variances are asymptotic to $2(24/5)/m$ and $2(72/7)/m$. Thus the absolute bias divided by its standard deviation is $O(h^2\\sqrt m)=O(n^{-1/8})$, uniformly over the stated mean class. The centered noise is exactly normal, so both conventional intervals have coverage tending uniformly to $0.95$. Yet $\\operatorname{Var}(B)/\\operatorname{Var}(A)\\to15/7$ and the ratio of standard errors tends to $\\sqrt{15/7}\\approx1.464$; for a fixed nonzero effect, the ratio of absolute test statistics for $B$ and $A$ tends in probability to $\\sqrt{7/15}<1$. This establishes the paper's undersmoothing exception without exact linearity. It does not reproduce any sampled study or the automatic bandwidths of \\texttt{rdrobust}.\n\n\\begin{audittable}{lrl}{Counts available in the paper, the supplement and the replication files. ``More than half'' of the sample requires 23 of 45 studies with inappropriate inference.}\nItem & Count & Basis \\\\\nStudies coded with the CCT bandwidth & 12 of 45 & a choice of bandwidth \\\\\nStudies coded with an IK or cross-validated bandwidth & 10 of 45 & a choice of bandwidth \\\\\nStudies coded with a non-automated bandwidth & 23 of 45 & a choice of bandwidth \\\\\nStudies in the main reanalysis & 25 of 45 & data available, not coded CCT \\\\\nEstimates with a larger robust standard error & 27 of 39 & estimates, not studies \\\\\nEstimates with a smaller absolute $t$-statistic & 28 of 39 & estimates, not studies \\\\\nStudies with a larger robust standard error & 16 of 25 & one main estimate per study \\\\\nEstimates above $1.96$: original, reanalysis & 30, 18 of 39 & two estimators compared \\\\\n\\end{audittable}\n\nThe 10 studies of the second row are 7 with an IK bandwidth and 3 with a cross-validated one, and the 23 studies of the third row include 4 with a global polynomial specification. Weighting each study equally, the mean standardized standard error rises from $0.188$ to $0.265$ and the mean absolute $t$-statistic falls from $5.03$ to $3.91$. The count of 16 studies holds for the main estimate of each study and for the mean ratio within a study, with different sets of 16. With the conventional standard error at the same bandwidth, the counts are 21 of 39 estimates and 12 of 25 studies.\n\nThe examples do not represent any study in the sample or the data-driven defaults of \\texttt{rdrobust}. I did not examine the original articles or their code. The pointwise affine calculations do not validate the original methods under the paper's smoothness assumptions, and the undersmoothing derivation does not show that the sampled studies used suitable bandwidths. The replacement comparison does not identify which original procedures failed their claimed coverage conditions or establish the asserted majority.", "problem": "The three sentences state a diagnosis of the original studies: their methods of inference were inappropriate, their standard errors were too small, and this holds for more than half of the 45 studies in the sample. The paper supports the diagnosis with a comparison. It estimates 39 effects from 25 studies again with the default settings of \\texttt{rdrobust}, and it compares the new standard errors and $t$-statistics with the published ones. The robust standard error of the reanalysis exceeds the original standard error for 27 of the 39 estimates, the absolute $t$-statistic is smaller for 28, and the number of estimates above $1.96$ falls from 30 to 18.\n\nThis comparison is between two estimators. To assess whether an original interval was appropriate for the RD argument, I need to assess its bias and sampling distribution, reported standard error and critical value under the maintained smoothness assumptions and any additional justified restrictions. Exact coverage at linear conditional means does not establish validity under those broader assumptions. The reanalysis changes the estimator and the bandwidth, adds a bias correction whose noise enters the standard error, and uses no covariates. These changes affect sampling variance. The finite example below has exact coverage at affine means and illustrates that variance comparison; it does not show validity across the smoothness class. Correct undersmoothing, which the paper explicitly accepts, can provide such validity without exact linearity, and a bias-corrected replacement may still have a larger standard error. The same comparison can also arise when the original interval almost never covers the effect. The comparison alone does not distinguish the two cases. This does not vindicate a method that lacks its claimed coverage guarantee under the maintained model; the paper need not establish actual curvature to criticize such a method. Its general warning about nonnegligible smoothing bias is sound. What still requires a connection to the original procedures is the empirical diagnosis and its stated prevalence.\n\nThe sentence on p.~10 needs a count. ``More than half'' of 45 studies means at least 23 studies with inappropriate inference. The paper does not classify the 45 studies by whether their inference is valid, and it does not say which count the sentence rests on. The relevant counts from the paper, the supplement and the replication files are listed in the second table below; none classifies inference validity. On pp.~3--4 the paper names three valid approaches: undersmoothing with conventional inference, bias correction with a standard error that accounts for it, and a worst-case adjustment for bias. If a study chose its bandwidth without an automated rule, or did not use \\texttt{rdrobust}, it does not follow that its inference is inappropriate. Note that exactly 23 studies are coded with a non-automated bandwidth. The paper does not present this count as the basis of the sentence, and by the criteria of pp.~3--4 it does not count inappropriate inference.\n\nThe diagnosis is not shown to be false. The wording of p.~7 is evidential (``indicating''), rather than a claim that every increase in a replacement standard error proves original undercoverage. The theory and reanalysis support concern about inference that neglects smoothing bias. The descriptions of the original methods and the replacement comparisons do not establish how many original studies failed the relevant bias-control conditions, particularly the asserted majority of 45. The more limited empirical conclusion is that the reanalysis is consistent with understated uncertainty.", "what_survives": "The descriptive comparison of the saved results stands: the counts 27 of 39 and 28 of 39, the fall from 30 to 18 estimates above $1.96$, and the larger standardized standard errors on average. The reading that this pattern is consistent with understated uncertainty survives. The general warning about nonnegligible smoothing bias on pp.~3--4 remains sound, as do the three valid routes to inference described there. The bunching of $t$-statistics, the caliper tests and the power analysis do not use the disputed prevalence claim. What changes is the empirical diagnosis in the abstract: it is qualified as a concern supported by the theory and the comparison, and the undocumented share ``more than half'' is withdrawn. The counts above use the saved values, including the erroneous Broockman and Ryan standard error addressed in \\fid{P033-N011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N006", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph that defines effect size by Cohen's d. A red outline marks the words 'the standard deviation of the outcome of control units'.", "caption": "Power analysis, p. 9 (PDF p. 9), left column. The marked words name the standard deviation used to standardize the effect.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010.png"}, {"alt": "Crop of Table 2. A red outline marks the cell 0.10 in the row for effect size 0.1 and the column for power 95.", "caption": "Table 2, p. 9 (PDF p. 9). The marked cell is the share of analyses with 95% power at effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010-b.png"}, {"alt": "Crop of the second panel of Table 3, 'Studies not using election data'. A red outline marks the cell 0.28 in the row for effect size 0.1 and the column for power 95.", "caption": "Table 3, p. 10 (PDF p. 10), panel “Studies not using election data”. The marked cell is the share with 95% power at effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010-c.png"}, {"alt": "Line chart. The horizontal axis is the standard deviation of the outcome on the scale of an individual probability of voting, from 0 to 0.74; the vertical axis is power at d = 0.1, from 0 to 1. The curve rises from 0.05 at zero to 0.897 at a standard deviation of 0.5, where a vertical line marks the largest population standard deviation of a binary outcome; the region beyond is shaded. Open circles on the curve mark standard deviations 0.343, 0.434 and 0.559, needed for 60, 80 and 95 percent power. A blue arrow at the top right marks the value as computed in the replication files: standard deviation 24.6 (voter counts per cell), power 1.", "caption": "Power at $d=0.1$ for Coppock and Green (2016) under the normal approximation of p. 9, as a function of the standard deviation that standardizes the effect, with the robust standard error held at its saved value (approximately $0.0155$). Open circles mark the standard deviations needed for 60%, 80% and 95% power.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010-power-vs-sd.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T5", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T5"}, {"id": "P033-T6", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T6"}, {"id": "P033-T17", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T17"}, {"id": "P033-T18", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T18"}], "headline": "For Coppock and Green (2016), the estimate is an effect on the probability of voting but is standardized by the standard deviation of counts of voters, so that its “$d=0.1$” is not Cohen’s $d$ for the individual voting contrast; under the paper’s normal approximation with the saved standard error, the study cannot reach 95% power at $d=0.1$.", "id": "P033-N010", "location": "Power analysis, p. 9 (PDF p. 9), left column; Table 2, p. 9, row 0.1; Table 3, p. 10 (PDF p. 10), panel “Studies not using election data”, row 0.1", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P033-T17", "P033-T18", "P033-T5", "P033-T6"], "text": {"as_printed": "p.~9: ``We measure effect size by Cohen's $d$, which is the treatment effect standardized by the standard deviation of the outcome (Cohen, 1988). We use \\hl{the standard deviation of the outcome of control units} within the default bandwidth in the rdrobust package for this standardization.''\n\nTable~2, p.~9, row 0.1 (power 60, 80, 95): 0.17, 0.15, \\hl{0.10}.\n\nTable~3, p.~10, studies not using election data, row 0.1: 0.40, 0.35, \\hl{0.28}.", "corrected": "p.~9: ``We measure effect size by Cohen's $d$, which is the treatment effect standardized by the standard deviation of the outcome (Cohen, 1988). We use the standard deviation of the outcome of control units within the default bandwidth in the rdrobust package for this standardization\\hl{, with the outcome and effect referring to the same treatment contrast. For Coppock and Green (2016), the intended contrast is an effect on an individual's probability of voting, so that the standard deviation needed is that of individual turnout among control units. The earlier code instead standardized an aggregate fuzzy-IV slope per prior voter by the standard deviation of outcome counts. That statistic is reproducible but does not give individual Cohen's $d$. Individual Cohen's $d$ cannot be computed from the replication data, and we report this study's standardized effect sizes as unavailable. Since the population standard deviation of a binary outcome is at most $1/2$, the study does not attain 95\\% power at $d=0.1$ under the normal approximation with the saved robust standard error}.''\n\nTable~2, row 0.1: 0.17, 0.15, \\hl{0.08}.\n\nTable~3, studies not using election data, row 0.1: 0.40, 0.35, \\hl{0.175}, which is about $0.18$.\n\nThe retained entries $0.17$ and $0.40$ are conditional on the study attaining 60\\% power at $d=0.1$; $0.15$ and $0.35$ are conditional on its attaining 80\\% power. Under the same approximation, the respective standard deviations of individual turnout required are approximately $0.343$ and $0.434$.", "evidence": "The evidence is an exact counterexample to interpreting the code-defined standardization as individual Cohen's $d$, a recalculation from the replication files, and a bound on power for that individual target.\n\nThe counterexample has two types of cell on each side of an eligibility cutoff. The two types are equally likely, and each cell has 100 individuals. On the eligible side, each cell has 50 voters in the earlier election, and the two types have 10 and 50 voters in the later one; five of the earlier voters in each cell are induced to vote later, so the later-voter counts without treatment would be 5 and 45. On the ineligible side, no one voted in the earlier election, and the two types have 10 and 40 voters in the later one. The fuzzy RD effect is $(30-25)/(50-0)=1/10$. Turnout without treatment is $1/4$ on each side, with variance $3/16$, and Cohen's $d$ satisfies $d^2=(1/10)^2/(3/16)=4/75$. The sample variance of the counts on the eligible side is $(10-30)^2+(50-30)^2=800$, and the standardization of the replication files gives $d^2=(1/10)^2/800=1/80000$. Doubling the population and the counts of every cell leaves the effect at $1/10$ and Cohen's $d^2$ at $4/75$. The variance of the counts becomes $3200$, and the second quantity becomes $1/320000$.\n\nFor the bound, a binary outcome with mean $p$ has variance $p-p^2\\le1/4$. Let $s$ be the saved robust standard error (approximately $0.0155$) and $z=z_{.975}$. Under the normal approximation, the power at $d=0.1$ with standard deviation $\\sigma$ is\n\\[\n\\Phi(\\mu-z)+\\Phi(-\\mu-z),\\qquad \\mu=0.1\\,\\sigma/s,\n\\]\nwhich increases in $\\mu\\ge0$. Substituting $\\sigma=1/2$, I obtain approximately $0.8973$. Solving for the standard deviations such that the power equals 60\\%, 80\\% and 95\\%, I find approximately $0.343$, $0.434$ and $0.559$. The last exceeds $1/2$. A sample standard deviation with divisor $N-1$ can exceed $1/2$ slightly. For a control group that contains all untreated later-election voters in the cells to the right of the cutoff within the bandwidth, at least 31,731 individuals, the bound is approximately $0.89727$.\n\nA fit of \\texttt{rdrobust} version 2.2 to the analysis file reproduces the saved coefficient and robust standard error to within $10^{-8}$, with bandwidth $105.66$ days; the original code used version 2.1.1. The local count standard deviation is $24.57954623$, so the code-defined target at $d=0.1$ is $2.45795462$. These reproduce the alternative actually computed, not individual Cohen's $d$. At the same bandwidths, doubling every count leaves the coefficient and its standard error unchanged. The study is one of the ten studies that do not use election data and has weight $1$. Changing its indicator for 95\\% power at $d=0.1$ from $1$ to $0$ changes the two cells as follows.\n\n\\begin{audittable}{lrr}{The two cells at effect size 0.1 and power 95\\%, with only this study changed.}\nCell & As printed & Corrected \\\\\nTable 2 & $5/48\\approx0.104$ & $11/144\\approx0.076$ \\\\\nTable 3, studies not using election data & $11/40=0.275$ & $7/40=0.175$ \\\\\n\\end{audittable}\n\nThe bound holds for the normal approximation with the saved standard error. It is not a statement about the exact power of the test in finite samples. I take the saved powers from the replication files; I have not independently executed \\texttt{rdpower}.", "problem": "The definition is right in general. For Coppock and Green, the code accurately computes an outcome-standardized aggregate fuzzy-IV coefficient per additional prior voter, a different comparison from Cohen's $d$ for individual turnout. In the analysis file for Coppock and Green (2016), each of the 140,659 rows is an aggregate: the outcome and the treatment are counts of voters, with maxima 294 and 552. The reanalysis is a fuzzy RD of one count on the other. Its estimate, $0.066$ with robust standard error $0.0155$, is a ratio of two discontinuities in counts. This is an aggregate outcome-count slope per prior voter; under the scientific interpretation in the paper's study metadata, the ratio identifies an effect on an individual's probability of voting. The code divides this effect by the standard deviation of the counts, $24.58$ within the bandwidth for the power analysis and $41.66$ below the cutoff for the funnel plots. The code's $d=0.1$ passes an alternative of $2.46$ to the fuzzy-RD slope calculation, and the saved power equals $1$ at all four effect sizes. That value is consistent with the code-defined target and saved standard error; the mismatch concerns its interpretation as individual Cohen's $d$. The divisor also depends on the size of the aggregates: at fixed bandwidths, doubling every count leaves the effect and its standard error unchanged and doubles the divisor.\n\nFor the intended individual voting contrast, the outcome is binary and its population standard deviation is at most $1/2$. An effect size of $d=0.1$ is then an effect of at most $0.05$, and under the normal approximation of the paper, with the saved robust standard error, the power at $d=0.1$ is at most about $0.8973$. Under this approximation, the study cannot attain 95\\% power at $d=0.1$, and one cell in each of Tables 2 and 3 changes. Whether the study attains 60\\% or 80\\% power at $d=0.1$ depends on the standard deviation of individual turnout, which the analysis file does not contain. The code also takes the observations to the left of the cutoff as the control units. In this file the treatment is concentrated on the left: within 200 days of the cutoff its mean is $100.7$ on the left and $0.42$ on the right. The left side is not a group of untreated individuals.", "what_survives": "The estimate itself, its standard error, its $t$-statistic and its $p$-value do not use the standardization, and the histograms of $t$-statistics and the caliper tests are unchanged. A common divisor cancels in the ratio of an estimate to its standard error, so that the study stays on the same side of the significance line in the funnel plots. What changes is its row of the power analysis and its position on the common scale of Figures 4, S4, S7 and S8. In panel (a) of Figure~4 it is plotted at about $(0.002,0.00005)$. The advantage of studies without election data at $d=0.1$ becomes smaller and keeps its sign. If this study also failed the 60\\% and 80\\% thresholds, the proportions would be $0.30$ and $0.25$, against $1/13\\approx0.08$ for election studies. Replacing the code-defined power of $1$ with power for the individual Cohen-$d$ target reinforces the conclusion that the studies are underpowered. The correction of \\fid{P033-N011} lowers the same cell of Table~2 for another study; each value here applies one correction alone."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N010", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the paragraph that begins 'Our primary reanalysis uses'. A red outline marks the words 'the default settings in the R package rdrobust', which run over two lines.", "caption": "Reanalysis method, p. 7 (PDF p. 7), left column. The marked words name the procedure used in the primary reanalysis.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-d.png"}, {"alt": "Crop of the paragraph that introduces Table 2. A red outline marks the figure 17 percent in the sentence 'For example, we see that only 17 percent of the analyses attain 60 percent power to detect a standardized effect of 0.1.'", "caption": "p. 9 (PDF p. 9), left column. The marked figure is the share of analyses with 60% power at effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011.png"}, {"alt": "Crop of Table 2. A red outline marks the three cells of the row for effect size 0.1: 0.17, 0.15 and 0.10.", "caption": "Table 2, p. 9 (PDF p. 9). The marked row is effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-b.png"}, {"alt": "Crop of the first panel of Table 3, 'Studies using election data'. A red outline marks the three cells of the row for effect size 0.1: 0.08, 0.08 and 0.04.", "caption": "Table 3, p. 10 (PDF p. 10), panel “Studies using election data”. The marked row is effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-c.png"}, {"alt": "Two dot plots side by side with logarithmic horizontal axes and three rows: Broockman and Ryan (2016), and estimates (b) and (c) of Schickler et al. Left panel, robust standard error: for Broockman and Ryan the default value is 0.0022 (blue circle), against 0.079 with plug-in residuals at the saved bandwidth (orange square) and 0.033 with bandwidths selected again (open orange diamond); for the two Schickler estimates the three values lie close together between 0.09 and 0.16. Right panel, absolute robust t-statistic with a vertical line at 1.96: for Broockman and Ryan 23.04 by default against 0.65 and 1.83; for Schickler (b) about 2.0 by default against about 1.5 and 1.6; for Schickler (c) about 3.1 to 3.7.", "caption": "Robust standard error and absolute robust $t$-statistic of three clustered reanalyses, under the default variance (as saved in the replication files) and with plug-in residuals, at the saved bandwidths and with bandwidths selected again.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-se-by-variance.png"}], "formal_release": null, "formal_statements": [{"id": "P033-T3", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T3"}, {"id": "P033-T4", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T4"}, {"id": "P033-T17", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T17"}], "headline": "With the default variance of rdrobust and clustering, a cell of at least four observations that share a value of the running variable inside one cluster adds nothing to the variance; the reanalysis standard error of Broockman and Ryan (2016) and its saved power of $1$ come from this cancellation, and correcting that study alone under the normal approximation turns the 17% of p. 9 into about 14%.", "id": "P033-N011", "location": "Table 2, p. 9 (PDF p. 9), row 0.1, and the sentence on 17% of the analyses; Table 3, p. 10 (PDF p. 10), panel “Studies using election data”, row 0.1; reanalysis method, p. 7 (PDF p. 7)", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P033-T17", "P033-T3", "P033-T4"], "text": {"as_printed": "p.~7: ``Our primary reanalysis uses \\hl{the default settings in the R package rdrobust} (Calonico et al., 2015).''\n\nTable~2, p.~9, row 0.1 (power 60, 80, 95): \\hl{0.17}, \\hl{0.15}, \\hl{0.10}.\n\np.~9: ``For example, we see that only \\hl{17\\%} of the analyses attain 60\\% power to detect a standardized effect of 0.1.''\n\nTable~3, p.~10, studies using election data, row 0.1: \\hl{0.08}, \\hl{0.08}, \\hl{0.04}.", "corrected": "p.~7: ``Our primary reanalysis uses the default settings in the R package rdrobust (Calonico et al., 2015)\\hl{, except for the variance in clustered designs in which observations in a cluster share values of the running variable. In such a design, the default nearest-neighbour residuals within a cell of four or more observations with the same value of the running variable sum to zero, and a cell of this kind inside one cluster contributes nothing to the cluster-robust variance. For these designs we replace those residuals, for example with regression plug-in residuals, retain the specified clusters and covariance when a cluster spans the cutoff, and propagate the change through bandwidth selection and power analysis}.''\n\nTable~2, row 0.1: \\hl{0.14}, \\hl{0.13} ($1/8$ exactly), \\hl{0.08}.\n\np.~9: ``For example, we see that only \\hl{14\\%} of the analyses attain 60\\% power to detect a standardized effect of 0.1.''\n\nTable~3, studies using election data, row 0.1: \\hl{0.04}, \\hl{0.04}, \\hl{0.00}.\n\nThese values correct the estimate of Broockman and Ryan (2016) alone, by the normal approximation with plug-in residuals, and hold every other saved power fixed. The cells at effect size 0.2 and power 95\\% become $0.15$ in Table~2 and $0.06$ in the election panel of Table~3. A full correction fits every clustered study again, including the three that were not checked here.", "evidence": "The evidence is a derivation, an exact counterexample, and a recalculation on the data of the replication files.\n\nLet $C$ be a cell of $m\\ge4$ observations with outcomes $y_i$ and sum $S$. The nearest-neighbour residual of observation $i$ is\n\\[\nr_i=\\sqrt{\\frac{m-1}{m}}\\Bigl(y_i-\\frac{S-y_i}{m-1}\\Bigr)=\\sqrt{\\frac{m}{m-1}}\\Bigl(y_i-\\frac{S}{m}\\Bigr),\n\\]\nwhich implies that $\\sum_{i\\in C}r_i=0$. The weight of an observation in the local polynomial fit depends only on its value of the running variable. The contribution of $C$ to the score of a cluster that contains it is this weight times $\\sum_{i\\in C}r_i$, which is zero. The same holds for the bias-corrected estimator.\n\nThe counterexample has ten independent clusters at $x=-5,\\dots,-1,1,\\dots,5$, with five observations each and cutoff $0$. The estimator is local linear with a quadratic fit for the bias, the triangular kernel and bandwidths $h=b=6$. Let $U_1,\\dots,U_{10}$ be independent and uniform on $\\{-1,1\\}$, and let every observation of cluster $g$ have the outcome $U_g$ under both treatments. The RD effect is $0$. The conventional estimator equals $\\sum_g a_gU_g$ with\n\\[\na=(1/5,\\,1/5,\\,0,\\,-2/5,\\,-1,\\,1,\\,2/5,\\,0,\\,-1/5,\\,-1/5),\n\\]\nand its variance is $\\sum_g a_g^2=62/25$. A direct computation gives the variance $10$ for the bias-corrected estimator. Every nearest-neighbour residual is $0$, and both estimated variances are $0$ in all $1024$ outcomes, while the two estimates are nonzero in $912$ and $968$ of them. If independent noise, uniform on $\\{-1,1\\}$, is added to every observation, then the covariance matrix of each cluster is positive definite. The true variances become $372/125$ and $12$, and the estimated variances stay $0$.\n\nMore clusters do not help. For $n\\ge3$, I place $n(n+1)$ clusters of five observations on each side at $x=\\pm j/(n(n+1))$ and set $h=b=1/n$, so that $n$ clusters on each side have positive weight. With outcome $U_g$ for every observation in cluster $g$, without the added observation noise, the variance of the conventional estimator is\n\\[\nV_n=\\frac{12(4n^2+5n-1)}{5n(n-1)(n+1)},\\qquad nV_n\\to48/5,\n\\]\nand the default estimate of it is $0$ for all $n$.\n\nOn the data of Broockman and Ryan, I find the same cancellation with exact rational arithmetic. In the union of the two bandwidth windows there are 43 cells on each side, each with at least four respondents, and all but one lie inside one district. The exception is the cell at $x=0.0013536513$ with 46 respondents from two districts, whose sums of deviations from the mean of the cell are $+19/46$ and $-19/46$. The sandwich formula applied to this cell alone reproduces both saved standard errors. A fit of \\texttt{rdrobust} with the default variance reproduces the saved coefficients and standard errors of the three estimates in the figure to within $4.8\\times10^{-16}$.\n\nFor the tables, I use the power under the normal approximation, $\\Phi(d\\,\\sigma_0/s-z_{.975})+\\Phi(-d\\,\\sigma_0/s-z_{.975})$, where $s$ is the robust standard error with plug-in residuals and $\\sigma_0$ the standard deviation of the outcome among control units within the bandwidth. At $d=0.1$ the power of the Broockman and Ryan estimate is $0.089$ at the saved bandwidths and $0.294$ with bandwidths selected again, below all three thresholds in both cases. The study has weight $1$ of 36, and of 26 among election studies.\n\n\\begin{audittable}{lrrr}{Cells that change when the Broockman and Ryan estimate alone is corrected, with every other saved power held fixed. The values are the same at the saved bandwidths and with bandwidths selected again.}\nCell (effect size, power) & As printed & Corrected & As a fraction \\\\\nTable 2 (0.1, 60\\%) & 0.17 & 0.139 & $5/36$ \\\\\nTable 2 (0.1, 80\\%) & 0.15 & 0.125 & $1/8$ \\\\\nTable 2 (0.1, 95\\%) & 0.10 & 0.076 & $11/144$ \\\\\nTable 2 (0.2, 95\\%) & 0.18 & 0.153 & $11/72$ \\\\\nTable 3, election data (0.1, 60\\%) & 0.08 & 0.038 & $1/26$ \\\\\nTable 3, election data (0.1, 80\\%) & 0.08 & 0.038 & $1/26$ \\\\\nTable 3, election data (0.1, 95\\%) & 0.04 & 0 & $0$ \\\\\nTable 3, election data (0.2, 95\\%) & 0.10 & 0.058 & $3/52$ \\\\\n\\end{audittable}\n\nAt the saved bandwidths, six further cells fall: the 60\\% and 80\\% cells at $d=0.2$ and the 95\\% cell at $d=0.5$, in Table~2 and in the election panel of Table~3. For Schickler et al., the robust standard error of estimate (b) at the saved bandwidths moves from $0.092$ to $0.122$ and its $t$-statistic from $2.00$ to $1.51$. The robust standard error of estimate (c) moves from $0.129$ to $0.153$.\n\nSeveral limits apply. The true sampling standard deviations are unknown, and the ratio of the two standard errors compares one estimator with another, not an estimator with the truth. Plug-in residuals remove the cancellation. I do not claim that the resulting intervals have correct coverage or that this is the only repair. Computing the conventional standard error by weighted least squares with errors clustered by district, I obtain $0.0630$ at the saved bandwidth. The \\texttt{rdpower} package was not available, and the powers above are the normal approximation. The paper used version 2.1.1 of \\texttt{rdrobust} and the recalculation used version 2.2, which reproduces the saved values.", "problem": "The reanalysis calls \\texttt{rdrobust} with the cluster variable of each study and the default variance estimator, which is built from nearest-neighbour residuals. The cancellation arises in the package's documented default combination of nearest-neighbour residuals and clustering, inherited through the paper's ordinary call; the paper did not independently write the cancelling variance formula. I call a set of observations with the same value of the running variable a cell. When a cell has $m\\ge4$ observations, the neighbours of each observation are the other $m-1$ members of the cell. The residual of an observation is then $\\sqrt{m/(m-1)}$ times the deviation of its outcome from the mean of the cell. For any values of the outcomes, these residuals sum to zero over the cell. The regression weights depend only on the running variable and are constant on the cell. A cell that lies inside one cluster then contributes zero to the score of that cluster. The uncertainty shared by the members of the cluster is left out of the variance. If every cell in the estimation and bias bandwidths has at least four observations and lies inside one cluster, the estimated variance is zero for the conventional and for the bias-corrected estimator.\n\nThe data of Broockman and Ryan (2016) nearly have this structure. The 29,413 survey respondents share 759 distinct values of the running variable, and the clusters are 380 districts. Inside the selected bandwidth there are 1,516 respondents in 36 cells, and 1,470 of them are in cells contained in one district. The only cell with a nonzero contribution is one of 46 respondents shared by two districts. The saved standard errors, $0.00211$ for the conventional estimate and $0.00222$ for the robust one, come from this cell alone. The reanalysis $t$-statistic of $23.0$ and the saved power of $1$ at every effect size follow from them. With regression plug-in residuals in place of the nearest-neighbour residuals and the same clusters, the robust standard error is $0.0790$ at the saved bandwidths, with $t=0.65$, and $0.0330$ when the bandwidths are selected again, with $t=1.83$.\n\nThe paper uses the default procedure as its reanalysis benchmark. Its statement that the cited methods represent progress is a general methodological judgment and is not refuted here. The application of this default as an uncertainty benchmark needs qualification for clustered designs in which observations in a cluster share values of the running variable. The paper should have checked whether the documented default was suitable for those designs. A larger number of clusters does not remove the problem. I established the cancellation for one study of 36. Two estimates of Schickler et al., with cells of two or three observations, are sensitive to the residuals to a lesser degree. Three other clustered studies, Bohlken (2018), Gulzar and Pasquale (2017) and Larreguy et al.\\ (2016), were not checked, since I did not have their data files.", "what_survives": "The statistics reported by the original studies, the selection of studies, the bunching of the original $t$-statistics and the caliper tests on those original statistics do not use the reanalysis variance. Reanalyses without this cancellation are unaffected by this finding, and so is the rdhonest reanalysis, which is restricted to studies without clustering. The Broockman and Ryan replacements lower the apparent precision and power of that reanalysis, reinforcing the conclusions on low power and on larger reanalysis standard errors. Survival of these broad conclusions does not validate the affected default as an uncertainty benchmark. What changes is the Broockman and Ryan reanalysis: its standard errors, its bandwidth, its $t$-statistics, its place in panels (b) and (c) of Figures 3 and 4, and its power. With it change the first rows of Table~2 and of the election panel of Table~3, the quoted 17\\%, and the summaries of the supplement that use the same row. \\fid{P033-E010} concerns a different defect in the same tables and changes other cells. Under the stated normal approximation, \\fid{P033-N010} lowers the same 95\\% cell of Table~2 for another study, and each value here applies one correction alone."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N011", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three entries of the reference list: Hahn, Todd and Klaauw (2001), Hall (2015), and Imbens and Kalyanaraman (2011). Red outlines mark 'Klaauw W' in the first entry and '(2011)' in the third.", "caption": "References, p. 12 (PDF p. 12), right column. The marked terms are the author name “Klaauw W” and the year “(2011)”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N013.png"}], "formal_release": null, "formal_statements": [], "headline": "The third author of Hahn, Todd and Van der Klaauw (2001) is printed as “Klaauw W”, and the entry for Imbens and Kalyanaraman pairs the year 2011 with volume 79(3), which is an issue of 2012.", "id": "P033-N013", "location": "References, p. 12 (PDF p. 12), entries for Hahn et al. (2001) and Imbens and Kalyanaraman (2011); both are cited on p. 3", "pdf_page": 12, "printed_page": "12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Hahn J, Todd P and \\hl{Klaauw W} (2001) Identification and estimation of treatment effects with a regression-discontinuity design. \\emph{Econometrica} 69(1): 201--209.\n\nImbens G and Kalyanaraman K \\hl{(2011)} Optimal bandwidth choice for the regression discontinuity estimator. \\emph{The Review of Economic Studies} 79(3): 933--959.", "corrected": "Hahn J, Todd P and \\hl{Van der Klaauw W} (2001) Identification and estimation of treatment effects with a regression-discontinuity design. \\emph{Econometrica} 69(1): 201--209.\n\nImbens G and Kalyanaraman K \\hl{(2012)} Optimal bandwidth choice for the regression discontinuity estimator. \\emph{The Review of Economic Studies} 79(3): 933--959.\n\nThe year 2011 can be kept if the entry says that it is the year of advance online publication. The citation on p.~3 follows the year chosen.", "evidence": "The evidence is a check of the page and of the publishers' pages for the two articles. The byline of the \\emph{Econometrica} article reads ``Wilbert Van der Klaauw''. The page of the second article reads ``Volume 79, Issue 3, July 2012, Pages 933--959'', with the publication date 24 November 2011.", "problem": "The surname of the third author is Van der Klaauw, and the entry drops part of it. Volume 79, issue 3 of \\emph{The Review of Economic Studies} is the issue of July 2012. The article appeared online in advance on 24 November 2011. The year 2011 is a date of the article, but it does not match the volume and issue that the entry cites. Both entries still identify the works.", "what_survives": "No argument or result of the paper depends on either detail."}, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N013", "work": {"arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/"}, "work_id": "P033"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of Table OA5 with its title: Study 1, Information about deepfakes has no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). The table has columns Favor, Vote and More Essential Oils in Medicine, and three rows. A red outline surrounds the label of the first row, Information Treatment Effect, whose entries are minus 0.22, minus 0.10 and minus 0.25. The row 95 percent Upper Bound has 0.25, 0.08 and 0.26, and the row 95 percent Lower Bound has minus 0.73, minus 0.31 and minus 0.75.", "caption": "Appendix, Table OA5, p. 13. The outline marks the row label “Information Treatment Effect”, which does not say that the effect is taken for the real video.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-A001.png"}, {"alt": "Crop of Table OA6 with its title: Study 1, Deepfakes have no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). The table has columns Favor, Vote and More Essential Oils in Medicine, and three rows. A red outline surrounds the label of the first row, Deepfake Treatment Effect, whose entries are 0.19, 0.05 and minus 0.14. The row 95 percent Upper Bound has 0.67, 0.24 and 0.39, and the row 95 percent Lower Bound has minus 0.32, minus 0.16 and minus 0.63.", "caption": "Appendix, Table OA6, p. 14. The outline marks the row label “Deepfake Treatment Effect”, which does not say that the effect is taken without the information treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-A001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P034-T3", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T3"}, {"id": "P034-T4", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T4"}], "headline": "Table OA5 reports the effect of the information treatment for the real video, and Table OA6 the effect of the deepfake without the information treatment; these are the main coefficients of models that contain the interaction of the two treatments, and not effects averaged over the other treatment.", "historical_evidence_type": "lean_proof", "id": "P034-A001", "location": "Appendix, Table OA5, p. 13, and Table OA6, p. 14 (titles, row labels and notes); cited from Section 3.1, p. 9 (PDF p. 9)", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P034-T3", "P034-T4"], "text": {"as_printed": "\\emph{Appendix, p.~13.} Table OA5: Study 1 -- Information about deepfakes has no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). \\hl{Information Treatment Effect}: $-0.22$ (Favor), $-0.10$ (Vote), $-0.25$ (More Essential Oils in Medicine?). 95\\% Upper Bound: $0.25$, $0.08$, $0.26$. 95\\% Lower Bound: $-0.73$, $-0.31$, $-0.75$.\n\n\\emph{Appendix, p.~14.} Table OA6: Study 1 -- Deepfakes have no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). \\hl{Deepfake Treatment Effect}: $0.19$ (Favor), $0.05$ (Vote), $-0.14$ (More Essential Oils in Medicine?). 95\\% Upper Bound: $0.67$, $0.24$, $0.39$. 95\\% Lower Bound: $-0.32$, $-0.16$, $-0.63$.", "corrected": "\\emph{Table OA5.} Title: Study 1 -- Information about deepfakes has no \\hl{statistically significant effect, for the real video,} on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Row label: Information Treatment Effect \\hl{for the real video (deepfake $= 0$)}.\n\n\\emph{Table OA6.} Title: Study 1 -- Deepfakes have no \\hl{statistically significant effect, without information about deepfakes,} on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Row label: Deepfake Treatment Effect \\hl{without information about deepfakes (information $= 0$)}.\n\n\\emph{Note to both tables.} \\hl{Each estimate is a main coefficient of a covariate-adjusted model that contains the information treatment, the video type and their interaction, and the intervals use the bounds of Section 2. Table OA5 compares information with no information for the real video. Table OA6 compares the deepfake with the real video without information.} Averaging over the other treatment would define different contrasts; the displayed coefficients need only be labelled explicitly.", "evidence": "The evidence is a derivation of the two contrasts and a recalculation from the Study 1 raw data. Let $w$ and $v$ be the indicators of the information treatment and of the deepfake video, and let $z$ be the vector of covariates. I write the fitted model as\n\\[\nm(w, v, z) = a + b w + c v + d w v + \\gamma^{\\mathsf T} z .\n\\]\nFor fixed $z$, I have\n\\[\nm(1, v, z) - m(0, v, z) = b + d v, \\qquad m(w, 1, z) - m(w, 0, z) = c + d w .\n\\]\nTable OA5 reports $b$, which is the first contrast at $v = 0$, and Table OA6 reports $c$, which is the second contrast at $w = 0$. Suppose that a fraction $q$ of the target population has $v = 1$. Then the first contrast, averaged over $v$, is $(1 - q) b + q (b + d) = b + q d$. Similarly, the second contrast averaged over $w$ is $c + p d$, where $p$ is the fraction with $w = 1$. The main coefficient $b$ equals the averaged effect only if $q d = 0$, and $c$ only if $p d = 0$. For example, if the four cell means for $(w, v) = (0,0), (1,0), (0,1), (1,1)$ are $0, 0, 0, 2$, then $b = 0$ and $d = 2$, and the average with $q = 1/2$ is $1$.\n\nThe recalculation reproduces all 18 printed entries from the raw data, in the sample of 1,252 participants that remains after the exclusion stated in the notes. The fitted interaction coefficients are $-0.153$ for favorability, $-0.065$ for vote and $-0.041$ for essential oils.\n\\begin{audittable}{llll}{The main coefficients printed in Tables OA5 and OA6, beside the averages $b + d/2$ and $c + d/2$ from the same models. Each entry is an estimate with its interval, and the intervals for the averages apply the paper's procedure for the bounds to those contrasts.}\nContrast & Outcome & Main coefficient & Average, equal weights \\\\\nInformation & Favor & $-0.219$ $[-0.732, 0.248]$ & $-0.295$ $[-0.670, 0.063]$ \\\\\nInformation & Vote & $-0.102$ $[-0.311, 0.079]$ & $-0.134$ $[-0.285, 0.006]$ \\\\\nInformation & Oils & $-0.251$ $[-0.752, 0.263]$ & $-0.271$ $[-0.652, 0.109]$ \\\\\nDeepfake & Favor & $0.188$ $[-0.322, 0.669]$ & $0.111$ $[-0.254, 0.479]$ \\\\\nDeepfake & Vote & $0.049$ $[-0.162, 0.235]$ & $0.017$ $[-0.133, 0.159]$ \\\\\nDeepfake & Oils & $-0.137$ $[-0.634, 0.388]$ & $-0.157$ $[-0.531, 0.230]$ \\\\\n\\end{audittable}\nEvery interval in the table contains zero. In the complete cases, the averaged effects of the information treatment have $p$-values of $0.004$, $0.004$ and $0.014$. These tests leave out the missing outcomes that the bounds are meant to cover, and they do not overturn the reading of ``no impact'' given above.", "problem": "The row labels read as the effects of the two treatments. Neither the titles nor the notes say at which level of the other treatment each effect is taken. The models behind the tables contain both treatments and their interaction, and each table reports a main coefficient. Table OA5 therefore reports the effect of the information treatment for participants who saw the real video, and Table OA6 reports the effect of the deepfake for participants who did not receive the information treatment. An effect averaged over the other treatment is a different contrast, since it adds a share of the interaction coefficient to the main coefficient. For favorability, the $-0.22$ of Table OA5 can be taken for the averaged effect, which is $-0.295$ with equal weights. The secondary hypotheses of the Study 1 pre-analysis plan compare information with no information, and the deepfake with the real video, without conditioning on the other treatment; the tables do not report those contrasts. All 18 printed entries are correct for the contrasts that were fitted. The meaning of ``no impact'' is clear from the parenthetical in each title and from Section 5 of the Appendix: no difference is statistically significant with the bounds.", "what_survives": "All 18 entries stand as main coefficients with their bounds, and so does the reading that no difference is statistically significant with the bounds. The main results on disbelief in the video, the Study 2 results and the pooled regressions for favorability and vote are separate calculations and do not use these entries. What changes is the labelling: each table reports the effect of one treatment at the reference level of the other, and neither reports the averaged contrasts of the pre-analysis plan."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-A001", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Table OA8 with its title: Table OA8: Study 2 - Republicans appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video. Two red boxes mark the words 'increases in favorability', split over two lines. The table has the columns Vote and Favor and three rows: Information*Republican, 0.06 and 0.24; 95 percent Upper Bound, 0.20 and 0.53; 95 percent Lower Bound, minus 0.09 and minus 0.06.", "caption": "Appendix p. 15 (PDF p. 15 of the Appendix), Table OA8. The marked words of the title report increases in favorability; the table prints only the interaction of the information with the Republican indicator.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E001.png"}, {"alt": "Crop of Table OA9 with its title: Table OA9: Study 2 - Participants who are pro-life appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video. A red box marks the words 'increases in favorability'. The table has the columns Vote and Favor and three rows: Information*Pro-Life, 0.02 and 0.13; 95 percent Upper Bound, 0.17 and 0.43; 95 percent Lower Bound, minus 0.13 and minus 0.17.", "caption": "Appendix p. 15 (PDF p. 15 of the Appendix), Table OA9. The marked words of the title report increases in favorability; the table prints only the interaction of the information with the pro-life indicator.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E001-b.png"}, {"alt": "Crop of seven lines of Section 5 of the Appendix. The text says that pro-life Republicans should be less likely to favor Romney if they believe that he was or is pro-choice, that favorability and vote intention levels in the Information condition should be higher than in the No Information condition, and that this pattern should be reversed for pro-choice Democrats. Two red boxes mark the words 'consistent with this theory in terms of estimate directions', split over two lines, in the sentence 'Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.'", "caption": "Appendix p. 19 (PDF p. 19 of the Appendix), Section 5, “Pooled”. The marked words say that the results agree with the theory in the direction of the estimates; the theory stated before them predicts higher favorability and vote intention with the information.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E001-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The titles of Tables OA8 and OA9 report “increases” among Republicans and among pro-life participants, but the estimated effects of the information within these two groups are slightly negative and not distinguishable from zero; the tables show differences between groups that are positive and not significant.", "id": "P034-E001", "location": "Appendix, titles of Tables OA8 and OA9, p. 15 (PDF p. 15 of the Appendix), repeated in the contents, p. 1; Section 5, “Pooled”, p. 19 (PDF p. 19 of the Appendix)", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Appendix p.~15: ``Table OA8: Study 2 -- Republicans appear to have directional (non-significant) \\hl{increases in favorability} when information about deepfakes is added to Romney video''. The table has the columns Vote and Favor. Information*Republican: 0.06, 0.24. 95\\% Upper Bound: 0.20, 0.53. 95\\% Lower Bound: $-0.09$, $-0.06$.\n\nAppendix p.~15: ``Table OA9: Study 2 -- Participants who are pro-life appear to have directional (non-significant) \\hl{increases in favorability} when information about deepfakes is added to Romney video''. Information*Pro-Life: 0.02, 0.13. 95\\% Upper Bound: 0.17, 0.43. 95\\% Lower Bound: $-0.13$, $-0.17$.\n\nAppendix p.~19: ``For instance, in Study 2, pro-life Republicans should be less likely to favor Romney if they believe that he was or is pro-choice. And since information about deepfakes increases disbelief that Romney was ever pro-choice, favorability and vote intention levels in the Information condition should be higher than in the No Information condition. This pattern should be reversed for pro-choice Democrats. Our results (see Section 3, Tables 6-7) are \\hl{consistent with this theory in terms of estimate directions} but are not statistically significant.''", "corrected": "Appendix pp.~1 and~15: ``Table OA8: Study 2 -- \\hl{Information effects on vote intention and favorability are estimated to be less negative among Republicans than Democrats; the differences are not statistically significant}''.\n\nAppendix pp.~1 and~15: ``Table OA9: Study 2 -- \\hl{Information effects on vote intention and favorability are estimated to be less negative among pro-life than pro-choice participants; the differences are not statistically significant}''.\n\nAppendix p.~19, in place of the last sentence quoted above: ``\\hl{The interaction point estimates of Tables OA8--OA9 are positive, consistent with the pre-specified comparative predictions, but the differences are not statistically significant. The effects within Republicans and within pro-life participants are estimated to be negative on both outcomes, with 95\\% intervals that include zero; these tables do not estimate a separate effect for participants who are both pro-life and Republican.}''\n\nThe sentences before it can stay as a statement of the theory. An effect within a group is the sum $b+d$, and it is to be reported with an interval that uses the covariance of the two coefficients, as in the table above. The cross-reference ``Section 3, Tables 6-7'' of the printed sentence is corrected in \\fid{P034-E015}.", "evidence": "The evidence is a recalculation from the public Study 2 data. The final cleaning code and the four regressions of the replication files were translated from Stata into Python; Stata was not run. The recalculation gives the analysis sample of 1,925 respondents, 966 without and 959 with information, and it reproduces the four printed interactions with their printed bounds. The regressions of Table OA8 exclude Independents and use 1,750 respondents. Each interval below is the estimate $\\pm 1.96$ robust standard errors, as in the Appendix. The standard error of $b+d$ uses the covariance of the two coefficients, that is, $\\operatorname{Var}(\\hat b+\\hat d)=\\operatorname{Var}(\\hat b)+2\\operatorname{Cov}(\\hat b,\\hat d)+\\operatorname{Var}(\\hat d)$.\n\\begin{audittable}{lrrrrrl}{The coefficient $b$ on the information, the interaction $d$ and the sum $b+d$ in the four regressions of Tables OA8 and OA9. The entries are rounded from the unrounded estimates, so a sum can differ in the last digit from the sum of the rounded coefficients.}\nTable, outcome & $N$ & $b$ & $d$ & $b+d$ & SE of $b+d$ & 95\\% interval for $b+d$ \\\\\n\\hline\nOA8, Vote & 1,750 & $-0.099$ & $0.055$ & $-0.043$ & $0.052$ & $[-0.146,\\,0.059]$ \\\\\nOA8, Favor & 1,750 & $-0.243$ & $0.238$ & $-0.005$ & $0.110$ & $[-0.221,\\,0.211]$ \\\\\nOA9, Vote & 1,925 & $-0.076$ & $0.021$ & $-0.055$ & $0.061$ & $[-0.175,\\,0.065]$ \\\\\nOA9, Favor & 1,925 & $-0.179$ & $0.130$ & $-0.049$ & $0.127$ & $[-0.298,\\,0.200]$ \\\\\n\\end{audittable}\nThe vote is coded $-1$, $0$, $1$ and favorability from $-3$ to $3$, and the entries are points on these scales. The signs do not depend on rounding: solving the normal equations of the four regressions in exact rational arithmetic, I find that $b<b+d<0$ and $d>0$ in each.", "problem": "Each table reports, for each outcome, one coefficient with its bounds: the interaction of the information about deepfakes with the group. In each regression, let $b$ be the coefficient on the information and let $d$ be the coefficient on the interaction. The fitted outcome is $a+bT+cG+dTG+x'\\theta$, where $T$ indicates the information, $G$ the group and $x$ the covariates. The estimated effect of the information is then $b$ in the comparison group, which is Democrats in Table OA8 and pro-choice participants in Table OA9, and $b+d$ in the group that the title names. The tables print $d$ and not $b$. A positive $d$ says that the estimated effect is larger in the named group than in the comparison group. It does not say that the effect in the named group is positive.\n\nIn the Study 2 data all four sums $b+d$ are negative. They are $-0.043$ for the vote and $-0.005$ for favorability among Republicans, and $-0.055$ and $-0.049$ among pro-life participants. Each 95\\% interval for a sum contains zero. The titles describe as increases the two favorability estimates that are slightly negative, and the data establish neither an increase nor a decrease within either group.\n\nRead as a statement about the levels that the theory predicts, the sentence of Section 5 is not supported either: the theory predicts higher favorability and vote intention with the information among pro-life Republicans. The estimates do agree with the prediction of both pre-analysis plans of Study 2, which is comparative. The plans expect the interaction of the treatment with a Republican indicator, and with a pro-life indicator, to be positive. All four interactions are positive, and none is statistically significant. Tables OA8 and OA9 come from separate regressions, and neither estimates a separate effect for respondents who are both pro-life and Republican.", "what_survives": "The four interactions, their printed bounds and their lack of statistical significance stand, and their signs agree with the predictions of both pre-analysis plans. The regressions are unchanged; only their description changes. The pooled estimates of Section 5 and the results of the main text come from other regressions and do not depend on these titles. What changes is the reading of the tables: the estimates do not show increases within either group, and the tables do not support the levels that the theory of Section 5 predicts. The check does not establish a decrease within either group."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E001", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of the Introduction. A red box marks the words 'were unable to discriminate' in the sentence 'We found that participants were unable to discriminate between real and deepfaked videos even when they were informed about the existence of deepfakes.'", "caption": "Introduction, p. 3 (PDF p. 3). The marked words state as an inability what the tests report as a difference that was not detected.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002.png"}, {"alt": "Crop of six lines of the abstract. Three red boxes mark the clause 'did not enhance participants' ability to successfully spot manipulated videos', which runs over two lines, in the sentence 'We conducted two online survey experiments, and found that informing participants about deepfakes did not enhance participants' ability to successfully spot manipulated videos but consistently induced them to believe the videos they watched were fake, even when they were real.'", "caption": "Abstract, p. 1 (PDF p. 1). The marked clause summarizes the test of the interaction on p. 8, which was not significant.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-e.png"}, {"alt": "Crop of the last four footnote lines of the page. A red box marks the words 'were not able to discern' in footnote 15: 'Participants were not able to discern between a real video and a deepfake version of the same video without the information treatment (see Section 3 of the Appendix for more details).'", "caption": "Footnote 15, p. 7 (PDF p. 7). The marked words make the same statement for participants who saw no warning.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-b.png"}, {"alt": "Crop of the first four lines of Section 3 of the Appendix. Two red boxes mark the words 'we show that participants were unable to do so.', split over two lines, after the sentence 'One key question is whether American voters can discern between deepfakes and real videos without being informed about deepfakes.'", "caption": "Appendix, Section 3, p. 6 (PDF p. 6 of the Appendix). The marked sentence says that Figure OA1 shows the inability.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-c.png"}, {"alt": "Crop of the title of Figure OA1 and the line above it. A red box marks the words 'were unable to discriminate' in the title 'Figure OA1: Voters were unable to discriminate between a real video and a deepfake (Study 1)'.", "caption": "Appendix p. 6, title of Figure OA1. The marked words repeat the statement in the title.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-d.png"}], "formal_release": null, "formal_statements": [], "headline": "“Unable to discriminate” and “did not enhance” mean that no difference and no improvement were detected; the tests reported do not establish that there was none.", "id": "P034-E002", "location": "Abstract, p. 1 (PDF p. 1); Introduction, p. 3 (PDF p. 3); footnote 15, p. 7 (PDF p. 7); Appendix, Section 3 and the title of Figure OA1, p. 6 (PDF p. 6 of the Appendix)", "pdf_page": 1, "printed_page": "1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Abstract, p.~1: ``We conducted two online survey experiments, and found that informing participants about deepfakes \\hl{did not enhance participants' ability to successfully spot manipulated videos} but consistently induced them to believe the videos they watched were fake, even when they were real.''\n\np.~3: ``We found that participants \\hl{were unable to discriminate} between real and deepfaked videos even when they were informed about the existence of deepfakes.''\n\np.~7, footnote 15: ``Participants \\hl{were not able to discern} between a real video and a deepfake version of the same video without the information treatment (see Section 3 of the Appendix for more details).''\n\nAppendix p.~6: ``One key question is whether American voters can discern between deepfakes and real videos without being informed about deepfakes. In Figure OA1, \\hl{we show that participants were unable to do so}.''\n\nAppendix p.~6, title: ``Figure OA1: Voters \\hl{were unable to discriminate} between a real video and a deepfake (Study 1)''", "corrected": "Abstract: ``We conducted two online survey experiments. \\hl{For the selected Study 1 videos, we did not detect an improvement in discrimination between the real and manipulated videos when participants were informed about deepfakes; the information} consistently induced them to believe the videos they watched were fake, even when they were real.''\n\np.~3: ``\\hl{For the selected Study 1 videos, we did not detect a statistically significant difference in participants' ratings of the real and deepfaked videos, with or without information about deepfakes.}''\n\np.~7, footnote 15: ``\\hl{Without the information treatment, we did not detect a statistically significant difference in participants' ratings of the real video and its deepfake version} (see Section 3 of the Appendix for more details).''\n\nAppendix p.~6: ``In Figure OA1, \\hl{we report no statistically significant difference between the real and deepfake conditions for the selected stimulus}.''\n\nAppendix p.~6, title: ``Figure OA1: \\hl{No statistically significant real/deepfake difference in participants' ratings} (Study 1, \\hl{without deepfake information})''\n\nThese are results of tests of a difference and not of an equivalence analysis. The statements of the paper about power and about generalization stay as they are.", "intended_meaning": "The passages summarize tests of differences in mean ratings for the two videos of Study 1. They mean that the study detected no significant difference between the ratings of the real video and of its deepfake version, with or without the warning, and no significant improvement due to the warning. This is correct, and p.~8 states it: the interaction of the two treatments ``is not statistically significant'', and its estimate has the opposite sign from an improvement. Recalculating from the Study 1 data, I obtain the following values. Without the warning, the adjusted difference (deepfake minus real) in the belief that the video is a deepfake is $0.062$, with $p=0.59$. Its complete-case 95\\% interval is $[-0.164,0.287]$, and its interval with Manski bounds is $[-0.384,0.469]$, as in Figure OA1. With the warning, the difference is $-0.130$, with $p=0.26$ and interval $[-0.478,0.215]$ with Manski bounds. The interaction is $-0.214$, with $p=0.19$ as in footnote 17 and complete-case interval $[-0.534,0.107]$.\n\nRead literally, ``we show that participants were unable'' says more, namely that the tests establish that participants could not tell the videos apart. A test that does not reject does not establish this, and the intervals above contain positive values. For example, suppose that every rating is $\\pm1$, and that the probability of $+1$ is $2/5$ for the real video and $3/5$ for the deepfake without the warning, and $3/10$ and $7/10$ with it. Then the mean difference is $2/5$ without the warning and $4/5$ with it. With ten participants in each of the four cells, the event that every cell has five ratings of each sign has positive probability (about $0.0004$). On this event every sample difference and the sample interaction are zero, with $p=1$. This example does not show that the study's participants could discriminate or that the warning helped them. No estimate, test or conclusion of the paper changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E002", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of the article. The text reads: The leftmost panel of Figure 2 on the next page illustrates that when the information about deepfakes was randomly added to a deepfake video, participants were 0.5 points more likely to believe the video they were watching was fabricated (p less than 0.001), followed by footnote mark 14 and the start of the next sentences. A red outline surrounds the parenthetical p less than 0.001.", "caption": "Section 3, p. 7 (PDF p. 7). The outline marks the $p$-value printed for the effect of 0.5 points in the deepfake-video arm of Study 1.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E003.png"}, {"alt": "Crop of the eight-line paragraph of the Appendix that begins: The major deviation is the use of Manski bounds for confidence intervals and statistical significance. A red outline surrounds the words for confidence intervals and statistical significance. The paragraph goes on to say that the distribution of outcomes that least favors the hypothesis is imposed on all missing outcomes, the survey instrument maximum in the control and the minimum in treatment, that a covariate-adjusted regression with robust standard errors is run, and that the lower bound of its 95 percent confidence interval is the lower bound of the reported interval, with the same done for a best-case distribution to obtain the upper bound.", "caption": "Appendix, Section 2, Study 1, p. 5. The outline marks the statement that the bounds are used for confidence intervals and statistical significance; the rest of the paragraph describes how missing outcomes are filled in.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The $p<0.001$ printed for the Study 1 effect in the deepfake-video arm is the complete-case value; in the worst-case regression that the Appendix prescribes for statistical significance, $p \\approx 0.003$, and the pooled $p$-values are computed without the bounds.", "id": "P034-E003", "location": "Section 3, paragraph beginning “The leftmost panel of Figure 2”, p. 7 (PDF p. 7); Appendix, Section 2, Study 1, p. 5; pooled estimates on pp. 8–9 and in the Appendix, Section 5, p. 19", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Section 3, p.~7.} The leftmost panel of Figure 2 on the next page illustrates that when the information about deepfakes was randomly added to a deepfake video, participants were 0.5 points more likely to believe the video they were watching was fabricated ($\\hl{p<0.001}$).\n\n\\emph{Appendix, Section 2, Study 1, p.~5.} The major deviation is the use of Manski bounds for confidence intervals \\hl{and statistical significance}. [\\ldots] We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment), run a covariate-adjusted OLS regression with robust standard errors over all outcomes, and use the lower bound of the resulting 95\\% confidence interval as the lower bound of our reported 95\\% CI.", "corrected": "\\emph{Section 3, p.~7.} [\\ldots] participants were 0.5 points more likely to believe the video they were watching was fabricated (\\hl{complete-case regression, $n = 685$:} $p<0.001$\\hl{; with missing outcomes set to the values that least favor the hypothesis, as in Section 2 of the Appendix, $n = 700$: estimate $0.35$, $p \\approx 0.003$; the 95\\% interval formed from the worst- and best-case regressions is $[0.12, 0.85]$}).\n\n\\emph{Section 3, pp.~8--9, and Appendix, Section 5, p.~19.} \\hl{The pooled estimates and their $p$-values come from robust regressions with a study fixed effect and shared covariates, without the Study 1 bounds; the pooled regression for intention to vote codes a missing vote as zero.}", "evidence": "The evidence is a recalculation from the Study 1 raw data, following the cleaning and analysis code in the replication files, together with an exact lower bound for the $p$-value of the worst-case regression. The deepfake-video arm has 349 control participants, of whom 6 have a missing outcome, and 351 treated participants, of whom 9 have a missing outcome. The worst-case regression keeps every observed outcome, sets the 6 missing control outcomes to $3$ and sets the 9 missing treated outcomes to $-3$; these are the ends of the scale. Each regression has an intercept, the treatment indicator and seven covariate columns, and its standard errors are the robust ones that Stata reports.\n\\begin{audittable}{lrrrrr}{The effect of the information treatment in the deepfake-video arm of Study 1 in the three regressions.}\nRegression & $n$ & Estimate & Robust SE & Degrees of freedom & Two-sided $p$ \\\\\nComplete case & 685 & 0.4988516 & 0.1163098 & 676 & 0.0000206 \\\\\nWorst case & 700 & 0.3549310 & 0.1193355 & 691 & 0.0030396 \\\\\nBest case & 700 & 0.6235811 & 0.1178226 & 691 & 0.0000002 \\\\\n\\end{audittable}\nFollowing the rule of the Appendix, with the multiplier $1.96$ that the program uses, I obtain the ends of the 95\\% interval:\n\\[\n0.3549310 - 1.96 \\times 0.1193355 \\approx 0.121033, \\qquad 0.6235811 + 1.96 \\times 0.1178226 \\approx 0.854513 .\n\\]\nThis interval agrees with the one drawn in the left panel of Figure 2, and it excludes zero.\n\nIn the worst-case regression, the $t$ statistic is $2.974$ on 691 degrees of freedom. The bound $p > 0.001$ does not depend on a numerical evaluation of the $t$ distribution. In exact rational arithmetic, the estimate $b$ and its robust variance $\\widehat{V}$ satisfy $b > 0$ and $b^2/\\widehat{V} < 9$ (the ratio is $8.846$), which implies that $0 < t < 3$. The density of the $t$ distribution with 691 degrees of freedom is\n\\[\nf(x) = \\frac{C}{\\pi \\sqrt{691}} \\Bigl( 1 + \\frac{x^2}{691} \\Bigr)^{-346}, \\qquad C = \\frac{4^{345} (345!)^2}{690!},\n\\]\nand it is decreasing for $x > 0$. Since $t < 3$, I have\n\\[\np = 2 \\int_t^\\infty f(x) \\, dx > 2 \\int_3^{33/10} f(x) \\, dx \\ge \\frac{3}{5} f\\Bigl( \\frac{33}{10} \\Bigr) > \\frac{3}{5} \\cdot \\frac{C}{(22/7) \\cdot 27} \\Bigl( 1 + \\frac{(33/10)^2}{691} \\Bigr)^{-346} .\n\\]\nIn the last inequality, I used that $\\pi < 22/7$ and $\\sqrt{691} < 27$. The right side is a rational number, approximately $0.0010409$, and it exceeds $1/1000$. A two-sided normal test gives $0.0029$ and a one-sided $t$ test gives $0.0015$; neither restores the printed threshold. One admissible completion of the missing outcomes suffices for this conclusion; I do not claim that $0.0030$ is the largest $p$-value over all completions.\n\nIn the real-video arm of Study 1, I find $p = 0.0000165$ in the worst-case regression, with the interval $[0.2877, 1.0999]$. The code shows that the pooled regressions omit the bounds. I reproduced the three pooled estimates of pp.~8--9 without bounds, as reported in Reproduction. I did not compute the pooled regressions with the Study 1 bounds.", "problem": "The Appendix states that, in Study 1, bounds are used for confidence intervals and for statistical significance. To form the lower end of an interval, each missing outcome is set to the value that least favors the hypothesis, and a covariate-adjusted regression with robust standard errors is fitted to all randomized participants. I call this the worst-case regression, after the ``worst-case bounds'' of p.~5 of the article, and I call the regression with the opposite values the best-case regression. The $p<0.001$ on p.~7 does not come from this procedure. It is the $p$-value of the complete-case regression, which drops the 15 participants in the deepfake-video arm whose outcome is missing. In the worst-case regression the two-sided $p$-value is $0.0030$. Under the method that the Appendix announces, the effect is therefore significant at the 1\\% level and not at the 0.1\\% level. The page does not say which regression the parenthetical reports. The program that computes the bounds returns a point estimate and the two ends of the interval, and no $p$-value.\n\nThe pooled estimates raise a second point. The estimates of 0.40 points, 0.13 points and 6.0 percentage points on pp.~8--9, and of 3.8 percentage points and 0.17 points in Section 5 of the Appendix, come from robust regressions with a study fixed effect and shared covariates. These regressions include the real-video arm of Study 1, where 17 control participants and 4 treated participants did not complete the survey, and they do not use the Study 1 bounds. Four of the five regressions drop participants whose outcome is missing. The fifth, for intention to vote, uses an indicator that the vote equals yes, which codes the 11 missing votes in the real-video arm of Study 1 as zero (see \\fid{P034-E011}). The paper does not promise bounds for the pooled analyses, and the note to Figure OA3 separates the Study 1 bounds from the pooled regression for the outcomes of that figure. The pooled $p$-values are not shown to be wrong, but the text does not say that they are computed without the bounds.", "what_survives": "The complete-case estimate of 0.5 points, the intervals in Figure 2 and the positive effect in the deepfake-video arm at the 5\\% level are unaffected, and the effect remains significant at the 1\\% level in the worst-case regression. In the real-video arm of Study 1, $p<0.001$ holds in the worst-case regression as well. The Study 2 analyses do not use the bounds. No pooled estimate or pooled $p$-value is shown to be wrong; whether the pooled results remain significant with bounds was not checked. What changes is the significance level on p.~7 under the method of the Appendix, and the description of the pooled analyses."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E003", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of Section 2 of the Appendix. Two red boxes mark the words 'onto all missing outcomes', split over two lines, in the sentence 'We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment)'.", "caption": "Appendix, Section 2, p. 5 (PDF p. 5 of the Appendix). The marked words say which values are replaced by the extremes: the missing outcomes.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E005.png"}, {"alt": "Rendering of fourteen numbered lines of Stata code under the headings 'ORIGINAL CODE: manski_ci.ado' and 'SOURCE LINES 14-27'. Red boxes mark the condition complete==0, written with Stata macro quotes, at the end of line 20, 'replace outcomenew=max if treat==0 and complete==0', and of line 21, 'replace outcomenew=min if treat==1 and complete==0'. Each of the two lines carries the comment 'replace missing outcomes with bounds'.", "caption": "Replication files, manski_ci.ado, lines 14–27, in a rendering of the released file. The two marked conditions select every respondent whose completion flag is zero, whether or not the outcome analyzed is missing.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E005-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The bounding program replaces every outcome of the 36 respondents who did not answer the last survey item, including answers that they gave, and ten intervals for secondary outcomes of Study 1 are wider than under the procedure described in the Appendix.", "id": "P034-E005", "location": "Appendix, Section 2, Study 1, p. 5 (PDF p. 5 of the Appendix), sentence beginning “We impose the distribution of outcomes”; replication files, manski_ci.ado, lines 19–21 and 36–38. The article refers to this section on p. 5 (PDF p. 5)", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Appendix p.~5: ``We impose the distribution of outcomes that least favors our hypothesis \\hl{onto all missing outcomes} (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment), run a covariate-adjusted OLS regression with robust standard errors over all outcomes, and use the lower bound of the resulting 95\\% confidence interval as the lower bound of our reported 95\\% CI. We do the same with a best-case distribution of missing outcomes to estimate the upper bound of our reported 95\\% CI.''\n\n\\texttt{manski\\_ci.ado}, lines 19--21 (the worst case):\n\n\\texttt{gen `outcomenew'=`outcome'}\n\n\\texttt{replace `outcomenew'=`max' if `treat'==0 \\& \\hl{`complete'==0}}\n\n\\texttt{replace `outcomenew'=`min' if `treat'==1 \\& \\hl{`complete'==0}}\n\nLines 36--38 (the best case) are the same with \\texttt{max} and \\texttt{min} exchanged.", "corrected": "Either the description or the program can be changed. To keep the printed numbers, the Appendix should say what the program does: ``\\hl{For every outcome, we treat all answers of a respondent who did not answer the final survey item as missing, including answers given to earlier questions.} We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes \\ldots''\n\nTo carry out the procedure as described, each of the four replacement conditions (lines 20, 21, 37 and 38 of \\texttt{manski\\_ci.ado}) should test the outcome itself. For example, line 20 becomes\n\n\\texttt{replace `outcomenew'=`max' if `treat'==0 \\& \\hl{missing(`outcome')}}\n\nThe Appendix can then read: ``\\hl{For each outcome, we retain every observed answer and replace only missing values} with the instrument maximum in control and minimum in treatment for the lower endpoint, reversing these assignments for the upper endpoint. We fit the same adjusted regressions with robust standard errors \\hl{and recompute the affected intervals}.''\n\nIn full, let $[L,U]$ be the range of the instrument. The two imputed outcomes are\n\\[\nY_i^{\\mathrm{low}}=\\begin{cases}Y_i&\\text{if }M_i=0,\\\\U&\\text{if }M_i=1\\text{ and }T_i=0,\\\\L&\\text{if }M_i=1\\text{ and }T_i=1,\\end{cases}\n\\qquad\nY_i^{\\mathrm{up}}=\\begin{cases}Y_i&\\text{if }M_i=0,\\\\L&\\text{if }M_i=1\\text{ and }T_i=0,\\\\U&\\text{if }M_i=1\\text{ and }T_i=1.\\end{cases}\n\\]\nThe reported interval is $[\\hat\\beta^{\\mathrm{low}}-1.96\\,\\widehat{\\mathrm{se}}^{\\mathrm{low}},\\ \\hat\\beta^{\\mathrm{up}}+1.96\\,\\widehat{\\mathrm{se}}^{\\mathrm{up}}]$, where $\\hat\\beta$ and $\\widehat{\\mathrm{se}}$ are the coefficient on $T_i$ and its robust standard error in the covariate-adjusted regression of the imputed outcome. With this rule the intervals of Table OA5 become, at the printed precision, $[-0.61,0.17]$ for favorability and $[-0.26,0.06]$ for the vote, and those of Table OA6 become $[-0.22,0.59]$ and $[-0.12,0.22]$. The intervals for essential oils are unchanged.", "evidence": "The evidence is one respondent in the data who shows the discrepancy exactly, and a recalculation from the Study 1 raw data. A respondent in an arm without the warning did not answer the last item and answered ``Extremely unfavorable'' ($-3$) to the question on favorability. In the fit for the lower endpoint, the program replaces this answer by $+3$. The rule of the Appendix keeps $-3$ in both fits.\n\nI rebuilt the Study 1 analysis file from the raw data with the released cleaning rules. I then computed each interval twice, once as the program does and once replacing only missing values. The first computation agrees with all 18 printed entries of Tables OA5 and OA6. Of the 17 intervals computed, ten change.\n\\begin{audittable}{lrr}{The ten intervals that change, as the program computes them and when only missing values are replaced. The Appendix prints the first four to two decimals and shows the other six in figures.}\nInterval & As coded & Only missing values replaced \\\\\n\\hline\nTable OA5, favorability & $[-0.732,\\,0.248]$ & $[-0.611,\\,0.167]$ \\\\\nTable OA5, vote & $[-0.311,\\,0.079]$ & $[-0.265,\\,0.058]$ \\\\\nTable OA6, favorability & $[-0.322,\\,0.669]$ & $[-0.217,\\,0.595]$ \\\\\nTable OA6, vote & $[-0.162,\\,0.235]$ & $[-0.119,\\,0.220]$ \\\\\nFigure OA1, unconvinced & $[-0.169,\\,0.187]$ & $[-0.156,\\,0.176]$ \\\\\nFigure OA1, don't know & $[-0.123,\\,0.085]$ & $[-0.117,\\,0.079]$ \\\\\nFigure OA2a, real video & $[0.014,\\,0.358]$ & $[0.033,\\,0.340]$ \\\\\nFigure OA2b, real video & $[-0.045,\\,0.157]$ & $[-0.036,\\,0.148]$ \\\\\nFigure OA3, trust in footage on TV & $[-0.094,\\,0.233]$ & $[-0.087,\\,0.230]$ \\\\\nFigure OA3, trust in footage online & $[-0.221,\\,0.103]$ & $[-0.216,\\,0.098]$ \\\\\n\\end{audittable}\nEach interval as coded contains the interval obtained when only missing values are replaced. The seven intervals that do not change are the two of Figure 2, the first column of Figure OA1, the panels for the deepfake video in Figures OA2a and OA2b, and the columns for essential oils in Tables OA5 and OA6. No point estimate changes, and each of the 17 intervals contains zero under one rule if and only if it does under the other.", "problem": "The Appendix describes the bounds outcome by outcome: the extreme values are imposed on the missing values of the outcome that is analyzed, and the observed answers are kept. The program does something else. I write $Y_i$ for the outcome analyzed, $T_i$ for the treatment indicator passed to the program, and $M_i$ for the indicator that $Y_i$ is missing. Let $C_i$ denote the completion flag, which in every call is the indicator that respondent $i$ answered the last item of the survey, the guess of the politician's party. The program replaces $Y_i$ by an extreme value when $C_i=0$, and not when $M_i=1$. A respondent who answered an earlier question and then left the survey has $C_i=0$ and $M_i=0$ for that question, and the answer is overwritten in at least one of the two fits. The two rules agree for an outcome only if $M_i=1$ exactly when $C_i=0$.\n\nIn Study 1, 36 of the 1,396 randomized respondents did not answer the last item. All 36 are missing on the main outcome, the belief that the video is a deepfake, and on the question about essential oils, and for these two outcomes the rules coincide. For the other outcomes they do not. Of the 36, 11 answered the questions on favorability and on the vote, 6 answered the two questions on whether the politician believes what is said, and 2 answered each of the two questions on trust.", "what_survives": "The main result is untouched: for the outcome of Figure 2 the two rules replace the same 36 values, and the fits are identical. All point estimates stand. The conclusions drawn from the ten intervals also stand, since none changes its position relative to zero. In particular, the interval for the real video in Figure OA2a stays above zero. The Study 2 regressions and the pooled regressions do not use the program. What changes is the width of ten intervals for secondary outcomes of Study 1. That the wider intervals contain the narrower ones in these data is not a statement about coverage, and no failure of coverage is established."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E005", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the six lines of footnote 20. The footnote says that the most conservative coding is reported, that as a robustness check the same analysis is conducted with a liberal interpretation of the same responses, and continues: Using this liberal interpretation, we recover a slightly larger treatment effect of 3.2 percentage points (p less than 0.001). A red outline surrounds the words 3.2 percentage points (p less than 0.001). The last sentence points to Section 8 of the Appendix for the full list of ambiguous responses.", "caption": "Footnote 20, p. 9 (PDF p. 9). The outline marks the estimate and $p$-value reported for the liberal coding.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E006.png"}, {"alt": "Listing of sixteen numbered lines of Stata code under the heading ORIGINAL CODE: 3-analysis_FINAL.do, SOURCE LINES 394-409. Line 394 renames FR_abortion_flag to FR_abortion_flag_conservative. Lines 395 to 398 preserve the data, set the global frmethod to liberal, run free_response_cleanup.do and restore. Line 399 reads merge 1:1 responseid using data/study2/study2_coded_free_responses_to_merge, gen(_merge_handcode_lib). Lines 401 to 407 generate FR_abortion_flag equal to 1 and replace it by 0 if any of the three final codes is curr_prochoice or past_prochoice. Line 409 regresses FR_abortion_flag on tr_warn and the macro covars for Study 2 participants with romney_party equal to 1. Nothing is outlined.", "caption": "Replication code, 3-analysis_FINAL.do, lines 394–409. Line 399 merges the liberal codes without replacing the conservative ones already in memory, and line 409 runs the regression with the macro covars.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E006-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication code for the robustness check of footnote 20 keeps the conservative codes and changes the covariate list, so it does not carry out the check that the footnote describes; a recalculation with the liberal codes and Study 2 covariates reproduces the printed 3.2 percentage points (see P034-E010).", "id": "P034-E006", "location": "Footnote 20, p. 9 (PDF p. 9); replication code, 3-analysis_FINAL.do, lines 393–413, with lines 227–228, 296, 317 and 366", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Footnote 20, p.~9.} We report the most conservative coding so that ambiguous responses like ``support women'' and ``fight for women's causes'' are coded as ``other.'' As a robustness check, we conduct the same analysis but with a liberal interpretation of the same responses (i.e., we code these types of ambiguous responses as ``pro-choice''). Using this liberal interpretation, we recover a slightly larger treatment effect of 3.2 percentage points ($p<0.001$).\n\n\\emph{\\texttt{3-analysis\\_FINAL.do}, lines 399 and 409.}\n\n\\texttt{\\hl{merge} 1:1 responseid using} [\\ldots]\\texttt{, gen(\\_merge\\_handcode\\_lib)}\n\n\\texttt{reg FR\\_abortion\\_flag tr\\_warn \\hl{\\$covars} if study==2 \\& romney\\_party==1, robust}", "corrected": "\\emph{\\texttt{3-analysis\\_FINAL.do}, lines 398--409.} The conservative code variables are dropped before the second merge, and the regression names the covariates of Study 2:\n\n\\texttt{restore}\n\n\\texttt{\\hl{drop final\\_code1 final\\_code2 final\\_code3}}\n\n\\texttt{merge 1:1 responseid using} [\\ldots]\\texttt{, gen(\\_merge\\_handcode\\_lib)}\n\nLines 401--407, which construct \\texttt{FR\\_abortion\\_flag} from the merged codes, remain between the merge and the regression:\n\n\\texttt{reg FR\\_abortion\\_flag tr\\_warn \\hl{c.age i.gender i.party i.region c.hhi} ///}\n\n\\texttt{\\hl{elder-pro\\_vax conlib imputed\\_conlib bigfive* pol\\_know\\_total polinterest} ///}\n\n\\texttt{if study==2 \\& romney\\_party==1, robust}\n\n\\emph{Footnote 20.} I reproduce the printed estimate and $p$-value in the translated calculation of \\fid{P034-E010}. The repaired Stata code should be run on the Study 2 analysis data to verify its output against that calculation.", "evidence": "The evidence is a trace of the code and an exact check on the file of hand-coded responses in the replication files, which has 7,671 responses from 2,557 participants. Applying the conservative rules to this file, I find 103 pro-choice responses from 101 participants; the liberal rules give 138 from 134. The two sets of codes differ for 36 responses from 36 participants, and the outcome would change for 33 participants; with the merge at line 399 it changes for none. For example, one participant names the facts ``MAN'', ``WHITE'' and ``FOR WOMEN'S RIGHTS''. The outcome variable \\texttt{FR\\_abortion\\_flag} equals $1$ when no response has a pro-choice code. With the conservative codes \\texttt{demog}, \\texttt{demog} and \\texttt{other} of this participant, it equals $1$. With the liberal codes \\texttt{demog}, \\texttt{demog} and \\texttt{curr\\_prochoice}, it equals $0$, and the block returns $1$. The counts above refer to the whole file and not to the regression sample. I checked the documented default merge rule without running Stata. In the recalculation reported in \\fid{P034-E010}, I generated the conservative and liberal outcomes separately and matched them to the Study 2 sample, avoiding the merge that retains the conservative codes. With the liberal outcome, the expanded Study 2 covariates and HC1 standard errors, I obtain a drop of 3.20 percentage points ($p=0.0006$) on $N=1{,}925$. The calculation translates the relevant Stata regression into Python and uses the supplied cleaned covariates; Stata was not run.", "problem": "Footnote 20 describes the analysis behind the estimate of 3.1 percentage points, repeated with one change: ambiguous responses are coded as pro-choice. The block of the replication code that produces the regression of the footnote does not do this, for two reasons. First, the conservative code variables \\texttt{final\\_code1}, \\texttt{final\\_code2} and \\texttt{final\\_code3} are merged into the analysis data at line 296 and are never dropped. The block writes the liberal codes to a file and merges that file at line 399 with a plain one-to-one merge. Under Stata's default rule for a merge, a matched observation keeps the values already in memory for every variable that both files contain. The liberal codes therefore never enter the data, and the new outcome equals the conservative outcome on every row. Second, the regression at line 409 takes its covariates from the global macro \\texttt{covars}. Line 366 has reset this macro to the six covariates of Study 1. The conservative regression at line 317 used the expanded list of Study 2 that lines 227--228 define. The block thus fits the conservative outcome with a different covariate list, and its output cannot show what the liberal coding does to the estimate.", "what_survives": "The conservative estimate of 3.1 percentage points at line 317, the closed-ended result of 14.1 percentage points and the main analyses of both studies are computed before this block and do not depend on it. I reproduce the 3.2 percentage points and $p<0.001$ of footnote 20 in the translated calculation of \\fid{P034-E010}. The deposited replication code still needs both repairs to carry out the stated robustness check."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E006", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of nine lines of the article, beginning: We also evaluated whether information about deepfakes can affect how participants internalize the content of the video. The marked sentence reads: We found that information about deepfakes did not significantly change how likely participants were to mention abortion (p equals 0.95), but there was a major shift in what participants perceived Romney's position on abortion to be. A red outline surrounds the words mention abortion (p equals 0.95). The following sentence says that the information about deepfakes caused a 3.1-percentage-point drop in the percentage of participants who associated Romney with a pro-choice position (p less than 0.001).", "caption": "Section 3, second paragraph of p. 9 (PDF p. 9). The outline marks the outcome described as mentioning abortion, with its $p$-value of 0.95; the next sentence gives the drop of 3.1 percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E007.png"}, {"alt": "Listing of fourteen numbered lines of Stata code under the heading ORIGINAL CODE: 3-analysis_FINAL.do, SOURCE LINES 302-315. Line 302 generates FR_abortion_flag equal to 1 and line 304 generates mention_abort equal to 0. A loop over i from 1 to 3 replaces FR_abortion_flag by 0 if final_code i equals curr_prochoice or past_prochoice, and replaces mention_abort by 1 if final_code i equals curr_prolife or past_prolife or FR_abortion_flag equals 0. Line 313 regresses mention_abort on tr_warn and the covariates for Study 2 participants with romney_party equal to 1, and the comment on lines 314 and 315 quotes the sentence of page 9 with p equals 0.95. Nothing is outlined.", "caption": "Replication code, 3-analysis_FINAL.do, lines 302–315. Lines 304 and 309–310 build mention_abort from the pro-choice and pro-life codes, and line 313 is the regression behind $p = 0.95$.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E007-code.png"}], "formal_release": null, "formal_statements": [], "headline": "Two hand-coded responses that support abortion choice are coded as pro-life, and the outcome described as mentioning abortion is an indicator of a pro-choice or pro-life code, which misses six participants who name abortion.", "id": "P034-E007", "location": "Section 3, second paragraph of p. 9 (PDF p. 9); Appendix, Table OA3, p. 11, and Section 8, p. 23; file of hand-coded responses in the replication files; 3-analysis_FINAL.do, lines 302–313", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Section 3, p.~9.} We found that information about deepfakes did not significantly change how likely participants were to \\hl{mention abortion} ($p=0.95$), but there was a major shift in what participants perceived Romney's position on abortion to be. The information about deepfakes caused a 3.1-percentage-point drop in the percentage of participants who associated Romney with a pro-choice position ($p<0.001$).\n\n\\emph{Hand-coded responses in the replication files.} ``HE WILL FIGHT FOR WOMEN'S RIGHTS TO CHOOSE'': \\hl{\\texttt{curr\\_prolife}}. ``HE’S PRO ABORTION'': \\hl{\\texttt{curr\\_prolife}}.", "corrected": "\\emph{Section 3, p.~9.} We found that information about deepfakes did not significantly change how likely participants were to \\hl{give a response coded as a current or past pro-choice or pro-life position} ($p=0.95$), but there was a major shift in what participants perceived Romney's position on abortion to be.\n\n\\emph{Hand-coded responses.} ``HE WILL FIGHT FOR WOMEN'S RIGHTS TO CHOOSE'': \\hl{\\texttt{curr\\_prochoice}}. ``HE’S PRO ABORTION'': \\hl{\\texttt{curr\\_prochoice}}. The response ``HE WILL LET WOMEN HAVE THE RIGHT TO CHOOSE FOR ABORTION'' carries the same reversed code and is to be recoded with them; the hand codes were not reviewed exhaustively. Table OA3 and the estimate of 3.1 percentage points are then to be recomputed on the analysis sample.", "evidence": "The evidence is an exact check on the file of hand-coded responses, which has 7,671 responses from 2,557 participants, and a derivation from the code. For $j = 1, 2, 3$, I set $P_j = 1$ if response $j$ has a current or past pro-choice code and $P_j = 0$ otherwise, and I define $L_j$ in the same way for the pro-life codes. Lines 302--311 of \\texttt{3-analysis\\_FINAL.do} start with \\texttt{FR\\_abortion\\_flag} equal to $1$ and \\texttt{mention\\_abort} equal to $0$. For each $j$ in turn, they set the first variable to $0$ if $P_j = 1$, and then set the second to $1$ if $L_j = 1$ or the first variable is $0$. I write $F$ and $M$ for the final values of the two variables. By induction on $j$, I obtain\n\\[\nF = \\prod_{j=1}^{3} (1 - P_j), \\qquad M = 1 - \\prod_{j=1}^{3} (1 - P_j - L_j) .\n\\]\nThe second equation shows that $M$ indicates a pro-choice or pro-life code, and not a mention of the topic. One participant's responses have the codes \\texttt{other}, \\texttt{dontknow} and \\texttt{dontknow}, and the first response is the word ``ABORTION''. Since $P_j = L_j = 0$ for every $j$, I have $M = 0$ for this participant. The six participants who name abortion and have $M = 0$ gave the responses ``ABORTION'' (four participants), ``MADE CLAIMS ABOUT ABORTION'' and ``MORMON CHURCH DOES NOT SUPPORT ABORTION''.\n\nIn the whole file, the two recodes raise the number of participants with a pro-choice code from 101 to 103 and leave $M$ unchanged. There are 150 participants with $M = 1$, and counting the six participants as well gives 156. The 53 rules of the conservative cleanup, applied to the file, reproduce all 7,671 codes saved in the replication files. The file covers 2,557 participants, and the analysis sample has 1,925. I did not establish analysis-sample membership for all of these participants or compute the corrected Table OA3 and regressions.", "problem": "The two analyses in this passage use hand codes for the three facts that each participant named. First, two responses that state support for abortion choice carry the code for a current pro-life position. With the correct code, each of the two participants counts as associating Romney with a pro-choice position, which bears on the estimate of 3.1 percentage points and on the pro-choice and pro-life rows of Table OA3. These two responses are not the only ones: the response ``HE WILL LET WOMEN HAVE THE RIGHT TO CHOOSE FOR ABORTION'' has the same reversed code. Second, the outcome behind $p = 0.95$ does not measure whether a participant mentions abortion. The variable \\texttt{mention\\_abort} equals $1$ exactly when at least one of the three responses has a current or past pro-choice code or a current or past pro-life code. A response that names abortion and receives another code does not count. The conservative cleanup printed on p.~23 of the Appendix assigns the code ``other'' to the bare response ``ABORTION''. Six participants in the file of hand-coded responses name abortion and have \\texttt{mention\\_abort} equal to $0$.", "what_survives": "The two recodes do not change \\texttt{mention\\_abort}, and describing this variable accurately changes no number. The result $p = 0.95$ stands as a statement about pro-choice and pro-life codes. The outcomes for disbelief in the video and the closed-ended result of 14.1 percentage points use other variables. The estimate of 3.1 percentage points, its $p$-value and the pro-choice and pro-life rows of Table OA3 are to be recomputed after the recodes. The recodes concern a few responses, and their effect on these numbers was not computed."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E007", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the Education and Partisanship blocks of Table OA1, with three columns of percentages. A red box marks the row 'Postgrad 17 percent, 25 percent, 22 percent'. The rows above read 'HS or less 18, 16, 17', 'Some College 37, 29, 32' and 'Bachelor's Degree 27, 29, 29'; the rows below give partisanship.", "caption": "Appendix p. 7 (PDF p. 7 of the Appendix), Table OA1, the rows for education and partisanship; the columns are Study 1, Study 2 and Pooled. The marked row is the postgraduate share, whose stored count for Study 1 includes five respondents with missing education.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E009.png"}], "formal_release": null, "formal_statements": [], "headline": "The stored count behind the Study 1 postgraduate share in Table OA1 includes five respondents whose education is missing (243 for 238); the printed 17% is unchanged, and the pooled 22% is not shown to be wrong.", "id": "P034-E009", "location": "Appendix, Table OA1 (Summary statistics), p. 7 (PDF p. 7 of the Appendix), row “Postgrad”; replication files, tables/sum_stats_table.xlsx", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lrrr}{Table OA1, Education block, last row, and the last row of the table.}\n & Study 1 & Study 2 & Pooled \\\\\n\\hline\nPostgrad & 17\\% & 25\\% & 22\\% \\\\\nTotal N & 1396 & 1925 & 3321 \\\\\n\\end{audittable}\nFor this row, the workbook of the replication files stores the shares $\\hl{0.1740687679}$ (Study 1), $0.2493506494$ (Study 2) and $\\hl{0.2177055104}$ (pooled).", "corrected": "The row should be defined and its denominator stated: ``Postgrad'' is the share of all participants with education code 7 or 8, and a missing response is not counted in the numerator. For Study 1 the count is \\hl{238} of 1,396, and not 243, and the entry stays at 17\\%. The entries for Study 2 and for the pooled sample should be recomputed from the Study 2 data with the same definition before the printed 25\\% and 22\\% are kept or changed.", "evidence": "The evidence is a recalculation from the Study 1 raw data and from the stored workbook. With the final rules for eligibility, attention checks, duplicates and randomization, the counts of the education codes 1 to 8 are 41, 217, 57, 300, 155, 383, 218 and 20, and 5 respondents have no code. These sum to 1,396, and $218+20=238$. Multiplying the stored shares by the totals 1396, 1925 and 3321, I obtain 243, 480 and 723.", "problem": "The stored share for Study 1 equals $243/1396$. In the Study 1 raw data, after the final exclusions, there are 1,396 randomized respondents, of whom 238 have a postgraduate education (codes 7 and 8 of the education variable) and 5 have no recorded education. The stored count is $238+5$: the five respondents with missing education are counted as postgraduates. The stored pooled share equals $723/3321$, and the stored share for Study 2 equals $480/1925$. Since $723=243+480$, the pooled count carries the same five respondents.\n\nThe printed entry for Study 1 does not change: $238/1396=17.05\\%$, and $238/1391=17.11\\%$ if the five are also removed from the denominator. Whether the pooled entry changes cannot be decided from the Study 1 data. Removing the five respondents, I obtain $718/3321=21.62\\%$, which still rounds to 22\\%. The pooled entry would round to 21\\% only if at least four of the 480 respondents counted for Study 2 also had missing education. I did not check this education count against the Study 2 data. The program for the supplementary materials calls a second program to write the workbook, and that second program is not among the replication files. The step that produced the count is therefore not identified.", "what_survives": "The printed 17\\% for Study 1 is unchanged, and the printed pooled 22\\% is not shown to be wrong. No treatment effect or conclusion of the paper is shown to depend on the postgraduate count."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E009", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the top of Table OA3, titled Study 2 - Hand-coded Romney facts across conditions, with the columns No Info about Deepfakes, Info about Deepfakes and Total. The first row, Don't Know / No Response Given, has the percentages 28.2, 27.0 and 27.6 percent and, below them, the counts N 818, 777 and 1,595. Red boxes mark the counts 777 and 1,595. The second row, Is Republican, has 16.5 percent in each column and the counts 477, 475 and 952.", "caption": "Appendix, Table OA3, p. 11 (PDF p. 11 of the Appendix): the head of the table and its first two rows. The marked counts, 777 and 1,595, include one substantive answer whose text was lost before hand-coding.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E010.png"}], "formal_release": null, "formal_statements": [], "headline": "Six free-response answers from five respondents of Study 2 were lost or cut short before they were hand-coded; one substantive answer in the analysis sample was coded as “don’t know”, and the count 777 in the first row of Table OA3 should be 776, with total 1,594.", "id": "P034-E010", "location": "Appendix, Table OA3, first row, p. 11 (PDF p. 11 of the Appendix); the paragraph on the open-ended question and footnote 20, p. 9 (PDF p. 9); replication files: the raw survey file of Study 2, the cleaned data, the file of hand-coded responses, and 2-study2_data_cleanup_FINAL.do, line 2", "pdf_page": 11, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Appendix p.~11, ``Table OA3: Study 2 - Hand-coded Romney facts across conditions'', first row, under the columns ``No Info about Deepfakes'', ``Info about Deepfakes'' and ``Total''.\n\nDon't Know / No Response Given: 28.2\\%, 27.0\\%, 27.6\\%. $N$: 818, \\hl{777}, \\hl{1,595}.\n\nFile of hand-coded responses in the replication files, third answer of one respondent in the arm with information about deepfakes: the text is \\hl{empty}, and the code is \\hl{\\texttt{dontknow}}.", "corrected": "Appendix p.~11, Table OA3, first row.\n\nDon't Know / No Response Given: 28.2\\%, 27.0\\%, 27.6\\%. $N$: 818, \\hl{776}, \\hl{1,594}.\n\nThe answer \\texttt{Canceled \"Big Bird\"} is counted in a substantive row. With the code \\texttt{other}, the row ``Other'' reads 8.9\\%, \\hl{8.4\\%}, 8.6\\% and $N$: 258, \\hl{241}, \\hl{499}.\n\nIn the replication files, the seven damaged responses in the cleaned data and the six corresponding entries in the file of hand-coded responses are to be restored from the raw survey file, matched by respondent and by the position of the answer. The six hand-coded entries are to be coded again from the full text. Three keep the code \\texttt{dontknow}, the sentence on health care keeps the code \\texttt{policy}, and the two substantive answers receive substantive non-abortion codes. The raw text should be kept beside any normalized text that is used for coding. The conservative and liberal hand-code files are then to be generated separately, followed by Table OA3 and the analyses of the open-ended question.", "evidence": "The evidence is a recalculation from the replication files. I compared the raw survey file of Study 2 with the cleaned data and with the file of hand-coded responses, matching the answers by respondent and by their position among the three. The cleaned data hold 8,973 answers from 2,991 respondents, and seven differ from the raw file. The file of hand-coded responses holds 7,671 answers from 2,557 respondents, and six differ. The seventh belongs to a respondent who fails the age check and was not hand-coded. Every other answer agrees with the raw file up to capitalization and spacing. The six answers are the following.\n\\begin{enumerate}\n\\item \\texttt{He asked to be called \"Ike\"}. The hand-coded entry is empty, with the code \\texttt{dontknow}. The respondent is outside the analysis sample.\n\\item \\texttt{'don't know\"}. The entry is empty, with the code \\texttt{dontknow}. The respondent is in the analysis sample, in the arm with information.\n\\item \\texttt{Canceled \"Big Bird\"}. The entry is empty, with the code \\texttt{dontknow}. The respondent is in the analysis sample, in the arm with information.\n\\item \\texttt{\" don't know \"}, written twice by one respondent, as the second and the third answer. Both entries are empty, with the code \\texttt{dontknow}. The respondent is in the analysis sample, in the arm without information.\n\\item A sentence on the health care plan that Romney supported in Massachusetts, which ends \\texttt{which was said to be the \"model\" for Obamacare}. The entry is ``FOR OBAMACARE'', with the code \\texttt{policy}. The respondent is in the analysis sample, in the arm with information.\n\\end{enumerate}\nWith the codes as they are in the replication files, I reproduce every count of Table OA3. The answer of item 3 is substantive, and the table changes as follows when it leaves the first row. Which substantive row receives it is a coding judgment, and the table gives two choices.\n\\begin{audittable}{lll}{The rows of Table OA3 that change when the answer of item 3 is coded as substantive. Each entry gives the counts for the columns ``No Info about Deepfakes'', ``Info about Deepfakes'' and ``Total''.}\nRow & Printed & After the answer is restored \\\\\n\\hline\nDon't Know / No Response Given & 818, 777, 1,595 & 818, 776, 1,594 \\\\\nOther, if the answer is coded \\texttt{other} & 258, 240, 498 & 258, 241, 499 \\\\\nStance on Other Policies, if it is coded \\texttt{policy} & 24, 25, 49 & 24, 26, 50 \\\\\n\\end{audittable}\nThe denominators 2,898, 2,877 and 5,775 do not change, and the percentages of the first row still round to 28.2\\%, 27.0\\% and 27.6\\%. With the code \\texttt{other}, the percentage of the row ``Other'' in the column with information becomes 8.4\\% (printed 8.3\\%).\n\nNone of the six answers has a pro-choice or pro-life code, and none receives one when its text is restored. This implies that the two outcomes that the analyses of p.~9 build from the hand codes are the same before and after for each of the 2,557 respondents. This holds under both the conservative coding and the liberal coding of footnote 20. I read the files with a strict parser of comma-separated text in Python and repeated the comparison in R. Stata was not run. The check does not identify what in the import of the raw file (\\texttt{insheet}, line 2 of \\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}) causes the loss.", "problem": "Each participant of Study 2 was asked to name three facts about Romney, and each answer was coded by hand. The codes are the basis of Table OA3 and of the analyses of the open-ended question on p.~9. Some answers did not reach the coders as they were written. Among the answers of the 2,991 respondents in the cleaned data, seven contain a double quotation mark in the raw survey file. In the cleaned data, each of the seven consists only of the text after its last quotation mark. Six of the seven are among the 7,671 answers that were hand-coded, and they come from five respondents. Five of these entries are empty, and the sixth reads ``FOR OBAMACARE'' in place of a full sentence.\n\nAll five empty entries have the code \\texttt{dontknow}. For three of them the code is right, since the respondent wrote ``don't know'' between quotation marks. The other two respondents gave substantive answers, \\texttt{He asked to be called \"Ike\"} and \\texttt{Canceled \"Big Bird\"}. The first of these two respondents did not identify Romney's party and is outside the analysis sample. The second is in the analysis sample, in the arm with information about deepfakes. The first row of Table OA3 counts this answer as ``Don't Know / No Response Given'', and its entries 777 and 1,595 are too large by one.", "what_survives": "Restoring these responses changes no reported effect. The result $p=0.95$ and the drop of 3.1 percentage points on p.~9, and the drop of 3.2 percentage points in footnote 20, use outcomes that are identical before and after the six answers are restored. In the recalculation these are $p=0.953$, 3.09 points ($p=0.0003$) and 3.20 points ($p=0.0006$). The denominators of Table OA3 and its rows for pro-choice and pro-life positions are not affected by this correction. Study 1, the closed-ended outcomes of Study 2 and the pooled analyses do not use the hand codes. What changes is one count and one total in the first row of Table OA3, the row that receives the restored answer, and the text of seven responses in the cleaned data, including six in the file of hand-coded responses. Other corrections to Table OA3 are given in \\fid{P034-E007} and \\fid{P034-N010}."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E010", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the first lines under the heading Pooled. The text reads: Information about deepfakes appears to have some impact on favorability and intention to vote for the politician featured in the video clip. Overall, information about deepfakes decreases intention to vote for the politician by 3.8 percentage points (p<0.05), with little difference in effect sizes across studies. Similarly, information about deepfakes decreases the politician's favorability by 0.17 points on a 7-point scale (from extremely unfavorable to extremely favorable) (p<0.05). The crop ends in the middle of the next sentence, which begins There are, however, strong theoretical reasons. A red outline surrounds the words 3.8 percentage points (p<0.05).", "caption": "Online Appendix, Section 5, “Pooled”, p. 19 (Appendix PDF p. 19). The outline marks the estimate that the binary vote regression produces.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E011.png"}, {"alt": "Listing of six lines of Stata code with line numbers 851 to 856. Line 851 is preserve, line 852 is blank, and line 853 is a keep if command with the condition study==1 and tr_d==0, or study==2 and romney_party==1. A red outline surrounds line 854, gen vote_bin=vote==1. Lines 855 and 856 regress vote_bin on tr_warn and the covariates pro_vax, acupuncture, conlib, imputed_conlib, i.party and hhi with robust standard errors, first pooled with i.study and then by study.", "caption": "Source code of the replication files, not a page of the article or of the Appendix: 4-Supplementary-Materials_FINAL.do, lines 851–856. The outline marks line 854, which sets the outcome to 0 when the vote question was not answered.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E011-code.png"}], "formal_release": null, "formal_statements": [], "headline": "The binary vote outcome behind the pooled estimate of 3.8 percentage points gives the value 0 to participants who did not answer the vote question; it should be defined only for participants who answered, which removes 11 participants from the Study 1 regression and moves its coefficient from about $-0.045$ to about $-0.052$.", "id": "P034-E011", "location": "Replication file 4-Supplementary-Materials_FINAL.do, line 854; its printed result is in the online Appendix, Section 5, “Pooled”, second sentence, p. 19 (Appendix PDF p. 19)", "pdf_page": 19, "printed_page": "19", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Online Appendix, p.~19.} Overall, information about deepfakes decreases intention to vote for the politician by 3.8 percentage points (p$<$0.05), with little difference in effect sizes across studies.\n\n\\emph{Replication file \\texttt{4-Supplementary-Materials\\_FINAL.do}, lines 853--856 (lines 855 and 856 are each one line of code, broken here before \\texttt{imputed\\_conlib}).}\n\\[\n\\begin{aligned}\n&\\texttt{keep if (study==1 \\& tr\\_d==0 ) | (study==2 \\& romney\\_party==1)}\\\\\n&\\texttt{gen vote\\_bin=}\\hl{\\texttt{vote==1}}\\\\\n&\\texttt{reg vote\\_bin tr\\_warn~ pro\\_vax acupuncture conlib}\\\\\n&\\qquad\\texttt{imputed\\_conlib i.party hhi i.study, robust}\\\\\n&\\texttt{bysort study:\\ reg vote\\_bin tr\\_warn~ pro\\_vax acupuncture conlib}\\\\\n&\\qquad\\texttt{imputed\\_conlib i.party hhi, robust}\n\\end{aligned}\n\\]", "corrected": "Line 854 should read\n\n\\texttt{gen vote\\_bin=\\hl{(vote==1) if vote<.}}\n\nThe condition \\texttt{if !missing(vote)} is equivalent. The outcome \\texttt{vote\\_bin} then equals $1$ for ``Yes'' and $0$ for ``No'' and ``Don't Know / Not Sure'', and it is missing when the question was not answered. The regressions of lines 855 and 856 then use the participants who answered. For Study~1, the regression of line 856 has $N = 685$ and a coefficient on the information of about $-0.052$ (robust standard error $0.033$), in place of $N = 696$ and about $-0.045$.\n\nThe sentence on p.~19 should report the coefficient of the pooled regression of line 855, computed with this outcome on the data of both studies, and should describe it as an effect among the participants who answered the vote question. The corrected line keeps unanswered questions missing but does not address attrition, and a statement about all randomized participants needs a separate argument.", "evidence": "The evidence is a reading of the code and an exact recalculation from the Study~1 raw data, in which I applied the exclusions and imputations of the cleaning code.\n\\begin{audittable}{lrrrrrr}{Answers to the vote question in the two real-video arms of Study~1.}\nArm & Yes & No & Don't know & Unanswered & All & Answered \\\\\nNo Information & 99 & 158 & 81 & 9 & 347 & 338 \\\\\nInformation about Deepfakes & 81 & 183 & 83 & 2 & 349 & 347 \\\\\n\\end{audittable}\nWithout covariates, the difference between the two arms in the share of ``Yes'' is\n\\[\n\\frac{81}{349}-\\frac{99}{347} = -\\frac{6444}{121103} \\approx -0.0532\n\\]\nwith the outcome as coded, and\n\\[\n\\frac{81}{347}-\\frac{99}{338} = -\\frac{6975}{117286} \\approx -0.0595\n\\]\namong the participants who answered. For the Study~1 regression of line 856, with the covariates of the code and robust (HC1) standard errors, I obtain the following.\n\\begin{audittable}{lrrrr}{The Study~1 regression of line 856 under the two definitions of the outcome.}\nOutcome & $N$ & Coefficient on the information & Robust s.e. & $p$ \\\\\nAs coded (unanswered set to 0) & 696 & $-0.04486$ & $0.03289$ & $0.173$ \\\\\nParticipants who answered & 685 & $-0.05245$ & $0.03326$ & $0.115$ \\\\\n\\end{audittable}\nThe coefficients are computed in exact rational arithmetic, and the standard errors and $p$-values in floating point. The calculation does not imitate the single-precision storage of Stata, and the last digits may differ from Stata's output. The pooled regression of line 855 and its Study~2 component were not recalculated.", "problem": "The vote question has three answers, coded $-1$ for ``No'', $0$ for ``Don't Know / Not Sure'' and $1$ for ``Yes'' (Appendix p.~15). The cleaning code leaves the variable \\texttt{vote} missing when the question was not answered. In Stata a missing value is not equal to $1$, which implies that the expression \\texttt{vote==1} evaluates to $0$ when \\texttt{vote} is missing. Line 854 then gives a participant who did not answer the same value as a participant who answered ``No'' or ``Don't Know / Not Sure''. A missing outcome would be dropped from a regression, and a value of $0$ is kept. The regressions of lines 855 and 856 estimate the effect of information about deepfakes on an outcome that equals $1$ when a ``Yes'' was recorded and $0$ otherwise, while the sentence on p.~19 reports a decrease in the intention to vote for the politician.\n\nThe case occurs in the data. In the two real-video arms of Study~1, 11 participants did not answer the vote question, 9 of 347 in the No Information arm and 2 of 349 in the Information about Deepfakes arm. All 11 enter the regression with the value $0$. If the outcome is defined only for participants who answered, the Study~1 regression has 685 observations in place of 696, and its coefficient on the information changes from about $-0.045$ to about $-0.052$. The corrected pooled estimate is not given here. I did not recalculate the pooled regression of line 855 or its Study~2 component. The estimate of 3.8 percentage points and its significance at the 5\\% level are neither reproduced nor replaced.", "what_survives": "The correction touches the binary vote regressions of lines 855 and 856 and the sentence on p.~19 that reports them. The favorability regressions that follow, the analyses of the three-level vote outcome with bounds in Tables~OA5 and OA6, and the results of the main text on disbelief, trust and policy knowledge do not use this outcome. Whether the pooled estimate stays near 3.8 percentage points and significant at the 5\\% level is not established; in Study~1 the coefficient moves away from zero."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E011", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the title of Figure OA2a and the headings of its three panels. The title reads: Figure OA2a: Study 2 - Impacts of information about deepfakes on disbelief (3-point scale): unconvinced that politician believes what is being said. A red outline surrounds the words Study 2. The panel headings read: Impact of Information Above Deepfake Video (Study 1), Impact of Information Above Real Video (Study 1) and Impact of Information Above Real Video (Study 2).", "caption": "Appendix, Figure OA2a, title and panel headings, p. 9. The outline marks “Study 2” in the title; the first two panels are headed Study 1.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012.png"}, {"alt": "Crop of the title of Figure OA2b and the headings of its three panels. The title reads: Figure OA2b: Study 2 - Impacts of information about deepfakes on disbelief (binary): don't know if politician truly believes what is being said. A red outline surrounds the words Study 2. The panel headings read: Impact of Information Above Deepfake Video (Study 1), Impact of Information Above Real Video (Study 1) and Impact of Information Above Real Video (Study 2).", "caption": "Appendix, Figure OA2b, title and panel headings, p. 10. The outline marks “Study 2” in the title; the first two panels are headed Study 1.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012-b.png"}, {"alt": "Crop of the eight-line italic note under Figure OA2b. Red outlines surround, over three lines, the words: how convinced are you that the politician believes what is being said? (3-point scale from 1 equals not at all convinced to minus 1 equals very convinced). The sentence that contains them begins: All three columns measure impact in terms of the same outcome. The rest of the note describes the rows of the figure and says that the two leftmost graphs of treatment effects use Manski bounds and the rightmost uses standard errors from an OLS regression.", "caption": "Appendix, note to Figure OA2b, p. 10. The outlines mark the definition of the three-point outcome, which belongs to Figure OA2a; it runs over three lines.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012-c.png"}, {"alt": "Crop of the six-line italic note under Figure OA3. The third line reads: range from 0 equals not at all confident to 4 equals completely confident, and a red outline surrounds the digit 4. The last sentence reads: The pooled estimate is estimated using a regression with shared covariates; it excludes all participants in the deepfake condition in Study 2, and a second red outline surrounds the final words Study 2.", "caption": "Appendix, note to Figure OA3, p. 18. The outlines mark the upper end of the trust scale, printed as 4, and the study named for the excluded deepfake condition, printed as Study 2.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012-d.png"}], "formal_release": null, "formal_statements": [], "headline": "Figures OA2a and OA2b show Studies 1 and 2, the note to Figure OA2b should define its binary outcome, the trust item ranges from 0 to 3, and the pooled estimate in Figure OA3 excludes the deepfake arm of Study 1.", "id": "P034-E012", "location": "Appendix: title of Figure OA2a, p. 9; title and note of Figure OA2b, p. 10; note to Figure OA3, p. 18; the two titles as repeated in the contents, p. 1; Study 1 pre-analysis plan, p. 4", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Figure OA2a, title, p.~9.} Figure OA2a: \\hl{Study 2} - Impacts of information about deepfakes on disbelief (3-point scale): unconvinced that politician believes what is being said\n\n\\emph{Figure OA2b, title and note, p.~10.} Figure OA2b: \\hl{Study 2} - Impacts of information about deepfakes on disbelief (binary): don't know if politician truly believes what is being said\n\n[\\ldots] All three columns measure impact in terms of the same outcome: \\hl{how convinced are you that the politician believes what is being said? (3-point scale from 1=''not at all convinced'' to -1=''very convinced'')}.\n\n\\emph{Figure OA3, note, p.~18.} Responses range from 0 = ``not at all confident'' to \\hl{4} = ``completely confident.'' [\\ldots] The pooled estimate is estimated using a regression with shared covariates; it excludes all participants in the deepfake condition in \\hl{Study 2}.\n\n\\emph{Study 1 pre-analysis plan, p.~4.} Trust is measured on a 4-point scale (ranging from ``0 = not at all confident'' to ``\\hl{4} = completely confident'').", "corrected": "\\emph{Figures OA2a and OA2b, titles (and the contents, p.~1).} Figure OA2a: \\hl{Studies 1 and 2} - Impacts of information about deepfakes on disbelief (3-point scale) [\\ldots] Figure OA2b: \\hl{Studies 1 and 2} - Impacts of information about deepfakes on disbelief (binary) [\\ldots]\n\n\\emph{Figure OA2b, note.} All three columns measure \\hl{an indicator equal to 1 for ``I don't know if the politician truly believes what is said in the video,'' and 0 for either alternative response}. The label of the vertical axis becomes \\hl{Don't know if politician truly believes (binary)}.\n\n\\emph{Figure OA3, note.} Responses range from 0 = ``not at all confident'' to \\hl{3} = ``completely confident.'' [\\ldots] it excludes all participants in the deepfake condition in \\hl{Study 1}.\n\n\\emph{Study 1 pre-analysis plan, p.~4.} Trust is measured on a 4-point scale (ranging from ``0 = not at all confident'' to ``\\hl{3} = completely confident'').", "evidence": "The evidence is a check of the pages against the survey instruments, the pre-analysis plans and the replication code. Both survey instruments list four options for the trust item, and both cleaning scripts code them 0 to 3. The code for Figure OA2b plots the indicator of the answer ``I don't know if the politician truly believes what is said in the video'', and the pooled regression for Figure OA3 selects the real-video participants of Study 1 and the Study 2 participants who knew Romney's party.", "problem": "These are misprints in titles and notes. In Figures OA2a and OA2b the first two panels are headed Study 1 and the third is headed Study 2. The note to Figure OA2b repeats the definition of the three-point outcome of Figure OA2a, but Figure OA2b plots a binary outcome, as its title says. Its axis label ``Dont Know if Real'' is also inexact, since the question asks whether the politician truly believes the statement. The trust item has four response options, coded 0 to 3. Study 2 has no deepfake condition, and the pooled estimate in Figure OA3 excludes the deepfake arm of Study 1, as footnote 19 of the article says.", "what_survives": "The plotted estimates, the coding of the outcomes and the exclusion used in the pooled regression are unaffected, and only the titles, notes and axis label change. The Study 2 estimates in these figures were not recalculated."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E012", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the lower edge of Figure 1, its caption and the first line of the next paragraph. A red box marks the word 'Schreenshot' in the caption 'Figure 1: Schreenshot of the video used in Study 1.'", "caption": "Article p. 4 (PDF p. 4), caption of Figure 1, item 2. The marked word is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015.png"}, {"alt": "Crop of five lines of the Introduction. A red box marks the words 'YouGov (according to 2021)' in the sentence 'A cursory search of the five most popular news websites in the U.S. YouGov (according to 2021) for the term deepfake on the news aggregator Google News returned 11,700 news articles discussing deepfakes'.", "caption": "Article p. 2 (PDF p. 2), second paragraph, item 1. In the marked words the name of the source stands outside the parenthesis and the year inside it.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-p2.png"}, {"alt": "Crop of five footnote lines. A red box marks the word 'reponded' in the sentence '27 percent thought the politician was a Democrat, 16 percent thought he was a Republican, and 27 percent reponded that he was Independent.'", "caption": "Article p. 4 (PDF p. 4), footnote 8, item 3. The marked word is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-p4.png"}, {"alt": "Crop of the top of the page. A red box marks the word 'SECITON' in the heading 'SECITON 3: Discernment of Deepfakes Without Deepfake Information'. Two lines of text follow.", "caption": "Appendix p. 6, heading of Section 3, item 4. The marked word is misspelled; the contents on p. 1 of the Appendix have the same spelling.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a06.png"}, {"alt": "Crop of a footnote: '1 See Swaim, B. \\& Nussbaum, J. (2016). The Perfect Presidential Stump Speech. FiveThirtyEight. November. Retrieved from' followed by a link. A red box marks the word 'November.'", "caption": "Appendix p. 3, footnote 1, item 5. The marked month is the date given for the FiveThirtyEight article.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a03.png"}, {"alt": "Crop of the reference entry 'Swaim, Barton, and Jeff Nussbaum. 2016. The Perfect Presidential Stump Speech (October 3, 2016).' followed by a link, and of the first line of the next entry. A red box marks '(October 3, 2016)'.", "caption": "Article p. 14 (PDF p. 14), reference list, item 5. The marked date is given for the same article.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-p14.png"}, {"alt": "Crop of the last two rows of Table OA2 (totals 369, 406, 370, 215 and 1,360) and the note under the table: 'Lucid asks for participants' political party before they are able to enroll in any surveys. ANOVA f-statistic=2.84 (p less than .0369).' A red box marks the parenthesis with the p-value.", "caption": "Appendix p. 8, note to Table OA2, item 6. The marked expression gives the rounded $p$-value as an upper bound.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a08.png"}, {"alt": "Crop of six rows of Table OA3 with their percentages and counts. Red boxes mark the word 'trusworthy' in the label 'Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.)' and the word 'bussinessman' in the label 'Demographic Fact (e.g., white, male, bussinessman)'.", "caption": "Appendix p. 11, Table OA3, item 7. The two marked words in the row labels are misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a11.png"}, {"alt": "Crop of the last row of Table OA5 (95 percent lower bound: -0.73, -0.31, -0.75) and the note under it. Red boxes mark the words 'This is defined as ALL participants EXCEPT', split over two lines, in the note 'As mentioned in the PAP, this analysis excludes participants who are likely supporters of alternative medicine. This is defined as ALL participants EXCEPT those who BOTH oppose government mandates for vaccination AND support government subsidies for acupuncture procedures.'", "caption": "Appendix p. 13, note to Table OA5, item 8; the note to Table OA6 on p. 14 is the same. After the marked words the note describes the participants who are kept, and the sentence before them speaks of those who are excluded.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a13.png"}, {"alt": "Crop of footnote 4 at the foot of the page, with the page number 18. A red box marks the words 'See Table OA5.' at the end of the footnote, after 'was used in Study 2 and produced very similar treatment effects.'", "caption": "Appendix p. 18, footnote 4, item 9. The marked reference is to Table OA5; the results for the alternative wording are in Table OA7.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a18.png"}, {"alt": "Crop of three lines. A red box marks the words 'Section 3, Tables 3-4,' in the sentence 'As shown in Section 3, Tables 3-4, we saw no statistically significant differences.'", "caption": "Appendix p. 19, Section 5, Study 1, item 10. The marked reference names Section 3 and Tables 3-4; the results are in Section 4, Tables OA5 and OA6.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a19a.png"}, {"alt": "Crop of five lines. A red box marks the words 'Section 3, Table OA9,' in the sentence 'As seen in Section 3, Table OA9, we find that participants surveyed before the election have higher levels (0.14 points more) of belief that the video they watched is deepfaked than participants surveyed after the election'.", "caption": "Appendix p. 19, Section 5, Study 2, item 11. The marked reference names Section 3 and Table OA9; the estimate of 0.14 points is in Section 4, Table OA11.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a19b.png"}, {"alt": "Crop of three lines. A red box marks the words 'Section 3, Tables 6-7)' in the sentence 'Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.'", "caption": "Appendix p. 19, Section 5, Pooled, item 12. The marked reference names Section 3 and Tables 6-7; the interactions are in Section 4, Tables OA8 and OA9.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a19c.png"}, {"alt": "Crop of the last eight lines of Section 6. Red boxes mark 'Table OA8)' in 'the interaction of the information treatment and Republican self-identification (see Table OA8)' and the first 'pro-choice' in 'it is not clear that all rank-and-file Republicans identify being pro-choice as a clearly Republican position; 31 percent of surveyed Republicans identified themselves as pro-choice in 2019.'", "caption": "Appendix p. 20, Section 6, items 13 and 14. The marked reference is to Table OA8, and the marked “pro-choice” is followed by the share of Republicans who call themselves pro-choice.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a20.png"}], "formal_release": null, "formal_statements": [], "headline": "Misprints and wrong cross-references: five references in the Appendix point to the wrong table or section, the last sentence of Appendix Section 6 says “pro-choice” where “pro-life” is meant, and the note to Table OA2 prints $p<.0369$ for $p=.0369$.", "id": "P034-E015", "location": "Article: p. 2 (PDF p. 2); Figure 1 caption and footnote 8, p. 4 (PDF p. 4); reference list, p. 14 (PDF p. 14). Appendix: contents, p. 1; footnote 1, p. 3; heading of Section 3, p. 6; note to Table OA2, p. 8; Table OA3, p. 11; notes to Tables OA5 and OA6, pp. 13–14; footnote 4, p. 18; Section 5, p. 19; Section 6, p. 20 (the same PDF pages of the Appendix)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{enumerate}\n\\item Article p.~2: ``A cursory search of the five most popular news websites in the U.S. \\hl{YouGov (according to 2021)} for the term \\ldots''\n\\item Article p.~4, caption: ``Figure 1: \\hl{Schreenshot} of the video used in Study 1.''\n\\item Article p.~4, footnote 8: ``27\\% \\hl{reponded} that he was Independent''\n\\item Appendix pp.~1 and~6, heading: ``\\hl{SECITON} 3: Discernment of Deepfakes Without Deepfake Information''\n\\item Appendix p.~3, footnote 1: ``Swaim, B. \\& Nussbaum, J. (2016). The Perfect Presidential Stump Speech. FiveThirtyEight. \\hl{November}.'' Article p.~14, reference list: Swaim, Barton, and Jeff Nussbaum. 2016. ``The Perfect Presidential Stump Speech'' (\\hl{October 3, 2016}).\n\\item Appendix p.~8, note to Table OA2: ``ANOVA f-statistic=2.84 ($p\\hl{<}.0369$).''\n\\item Appendix p.~11, Table OA3, row labels: ``Subjective Evaluation of Character (e.g., \\hl{trusworthy}, traitor, etc.)'' and ``Demographic Fact (e.g., white, male, \\hl{bussinessman})''\n\\item Appendix pp.~13 and~14, notes to Tables OA5 and OA6: ``As mentioned in the PAP, this analysis excludes participants who are likely supporters of alternative medicine. \\hl{This is defined as} ALL participants EXCEPT those who BOTH oppose government mandates for vaccination AND support government subsidies for acupuncture procedures.''\n\\item Appendix p.~18, footnote 4: ``\\ldots\\ was used in Study 2 and produced very similar treatment effects. See \\hl{Table OA5}.''\n\\item Appendix p.~19: ``As shown in \\hl{Section 3, Tables 3-4}, we saw no statistically significant differences.''\n\\item Appendix p.~19: ``As seen in \\hl{Section 3, Table OA9}, we find that participants surveyed before the election have higher levels (0.14 points more) of belief that the video they watched is deepfaked than participants surveyed after the election ($p<.05$).''\n\\item Appendix p.~19: ``Our results (see \\hl{Section 3, Tables 6-7}) are consistent with this theory in terms of estimate directions but are not statistically significant.''\n\\item Appendix p.~20: ``While we do include results illustrating the interaction of the information treatment and Republican self-identification (see \\hl{Table OA8}), we cannot directly address the question of motivated reasoning with this data for three reasons.''\n\\item Appendix p.~20: ``And finally, it is not clear that all rank-and-file Republicans identify being \\hl{pro-choice} as a clearly Republican position; 31\\% of surveyed Republicans identified themselves as pro-choice in 2019.''\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item ``the five most popular news websites in the U.S. \\hl{(according to YouGov 2021)} for the term \\ldots''\n\\item ``Figure 1: \\hl{Screenshot} of the video used in Study 1.''\n\\item ``27\\% \\hl{responded} that he was Independent''\n\\item ``\\hl{SECTION} 3: Discernment of Deepfakes Without Deepfake Information'', in the contents and in the heading\n\\item The two entries should give the same date. The reference list of the article gives October 3, 2016, and the Appendix should agree with it once the date has been checked against the article.\n\\item ``ANOVA F-statistic=2.84 ($p\\hl{=}.0369$).''\n\\item ``\\hl{trustworthy}'' and ``\\hl{businessman}''\n\\item ``\\hl{The retained analysis sample comprises all participants except} those who both oppose government mandates for vaccination and support government subsidies for acupuncture procedures.''\n\\item ``See \\hl{Table OA7}.''\n\\item ``As shown in \\hl{Section 4, Tables OA5--OA6}, we saw no statistically significant differences.''\n\\item ``As seen in \\hl{Section 4, Table OA11}, we find that participants surveyed before the election \\ldots''\n\\item ``Our results (see \\hl{Section 4, Tables OA8--OA9}) are consistent with this theory \\ldots''\n\\item ``(see \\hl{Table OA10})''\n\\item ``\\ldots\\ identify being \\hl{pro-life} as a clearly Republican position; 31\\% of surveyed Republicans identified themselves as pro-choice in 2019.''\n\\end{enumerate}", "evidence": "The evidence is a check of the pages against the contents of the Appendix, with a recalculation for item 6 and, for item 8, a reading of the program for the supplementary materials. For item 6, I computed the one-way analysis of variance from the counts printed in Table OA2. The participant's party is scored 1 (Republican), 2 (Independent) and 3 (Democrat), as in the replication files, and the four guesses are the groups. A direct computation gives $F(3,1356)=2.836964$ and $p=0.036942$. For item 8, the condition in that program excludes a participant exactly when the participant both supports subsidies for acupuncture and opposes vaccine mandates.", "problem": "Each item is a slip in spelling, in a label or in a cross-reference, and the surrounding text shows what was meant. In item 6 the printed expression is false as written.\n\\begin{enumerate}\n\\item The source of the ranking is YouGov (2021). The name and the year have changed places with the words ``according to''.\n\\item The word is ``Screenshot''.\n\\item The word is ``responded''.\n\\item The word is ``SECTION'', as in the headings of the other sections.\n\\item The two entries date the same article differently. I did not check the date against the article.\n\\item From the twelve counts of Table OA2, the statistic is $F(3,1356)=2.837$ with $p=0.03694$. This value rounds to $.0369$ and is larger than $.0369$, and the printed inequality is false. The note prints the rounded $p$-value as a bound where an equality is meant.\n\\item The words are ``trustworthy'' and ``businessman''.\n\\item ``This'' has no clear antecedent. The clause that follows describes the participants who are kept, and the sentence before it speaks of those who are excluded. The program for the supplementary materials keeps every participant except those who both support subsidies for acupuncture and oppose vaccine mandates, as the clause says. The note can be read as defining the excluded group by that clause, which would reverse the sample.\n\\item The results for the alternative wording are in Table OA7 (p.~14), whose title and note quote that wording. Table OA5 reports favorability, vote choice and support for essential oils in Study 1.\n\\item Section 3 of the Appendix contains only Figure OA1. The secondary results for Study 1 are in Section 4, Tables OA5 and OA6 (pp.~13--14).\n\\item The estimate for the election, $-0.14$, is in Section 4, Table OA11 (p.~16). Table OA9 reports the interaction with being pro-life.\n\\item The interactions with party and with the stance on abortion are in Section 4, Tables OA8 and OA9 (p.~15).\n\\item Section 6 concerns dismissing the video as a deepfake. The interaction of the warning with Republican identification for those outcomes is in Table OA10 (p.~16), and Table OA8 reports it for vote choice and favorability.\n\\item The clause after the semicolon says that 31\\% of Republicans call themselves pro-choice. That share is a reason to doubt that all Republicans regard the pro-life position as their party's position. It is no reason to doubt this of the pro-choice position, which the article (p.~6) calls ``an unusual stance for a Republican''.\n\\end{enumerate}", "what_survives": "No estimate, interval or conclusion changes. The $p$-value of Table OA2 remains below $.05$. The analyses of Tables OA5 and OA6 use the sample that the notes intend. The Appendix still says, correctly, that the data cannot settle the question of motivated reasoning."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E015", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of the plan. Two red boxes mark the words 'we conclude that missingness occurs at random', split over two lines, in the sentence 'If there is no differential attrition, we conclude that missingness occurs at random and exclude all affected observations from analysis.'", "caption": "Amended pre-analysis plan for Study 2, Section 8, second paragraph (PDF p. 5 of the plan). The marked words draw the conclusion from the absence of differential attrition.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017.png"}, {"alt": "Crop of the last four lines of the note to Figure OA2a. Two red boxes mark the words 'as there was no differential attrition.', split over two lines, in the sentence 'The rightmost treatment effect graph uses covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.'", "caption": "Appendix p. 9 (PDF p. 9 of the Appendix), note to Figure OA2a. The marked clause gives the absence of differential attrition as the reason for the complete-case interval of Study 2.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017-c.png"}, {"alt": "Crop of four lines of the note to Figure OA3. A red box marks the words 'as there was no differential attrition.' in the sentence 'The confidence intervals in Study 2 are estimated using covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.'", "caption": "Appendix p. 18 (PDF p. 18 of the Appendix), note to Figure OA3. The marked clause gives the same reason.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017-d.png"}, {"alt": "Crop of the last four lines of Section 2.2. A red box marks the words 'we found no evidence of differential attrition:' in the sentence 'In Study 2, we found no evidence of differential attrition: survey completion rates did not differ significantly across treatment arms.'", "caption": "Section 2.2 of the article, p. 6 (PDF p. 6). The marked sentence reports the result of the test. It is not in error; the inference from it is made in the plan and in the two notes of the Appendix.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017-b.png"}], "formal_release": null, "formal_statements": [{"id": "P034-T1", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T1"}, {"id": "P034-T2", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T2"}], "headline": "The absence of differential attrition does not show that outcomes are missing at random; the complete-case analysis of Study 2 needs a separate assumption about missingness, which the chi-squared test cannot supply.", "id": "P034-E017", "location": "Amended pre-analysis plan for Study 2, Section 8 (Missing Data), second paragraph (PDF p. 5 of the plan, whose pages are not numbered); notes to Figure OA2a, Appendix p. 9, and to Figure OA3, Appendix p. 18; related sentence in Section 2.2 of the article, p. 6 (PDF p. 6)", "pdf_page": 5, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P034-T1", "P034-T2"], "text": {"as_printed": "Plan, Section 8: ``Since there may be individuals who are randomized but do not finish the survey (and thus do not provide outcome responses), we will check for differential attrition using a chi-squared test (see line 13 in the Stata code). If there is no differential attrition, \\hl{we conclude that missingness occurs at random} and exclude all affected observations from analysis. If there is differential attrition (i.e., chi-squared p-value$<$.05), we will use inverse probability weights to weight the population of all subjects who passed the attention checks.''\n\nAppendix p.~9, note to Figure OA2a: ``The rightmost treatment effect graph uses covariate-adjusted standard errors from an OLS regression, \\hl{as there was no differential attrition}.''\n\nAppendix p.~18, note to Figure OA3: ``The confidence intervals in Study 2 are estimated using covariate-adjusted standard errors from an OLS regression, \\hl{as there was no differential attrition}.''", "corrected": "Plan, Section 8: ``\\hl{If the chi-squared test does not reject, we will report no detected difference in completion rates across treatment arms. This result does not establish random or ignorable outcome missingness. Complete-case covariate-adjusted OLS requires a separate identifying condition; for example, within the target population, completion under each treatment is independent of potential outcomes and pretreatment covariates, with positive completion probability. The test does not verify that condition. If a suitable identifying condition cannot be justified, we will report attrition bounds or sensitivity analyses.}''\n\nIn the notes to Figures OA2a and OA3, the clause ``as there was no differential attrition'' should be replaced in the same way. The intervals of Study 2 are complete-case intervals, no difference in completion rates was detected, and reading them as intervals for the randomized sample rests on the separate condition.\n\nIn full, one sufficient condition for the complete-case analysis is the following. Let $T\\in\\{0,1\\}$ be the treatment, let $Y(0)$ and $Y(1)$ be real potential outcomes with finite second moments, let $R(0),R(1)\\in\\{0,1\\}$ be the potential completion indicators, and let $X$ be a vector of pretreatment covariates with finite second moments. Set $Y=Y(T)$, $R=R(T)$ and $\\tau=\\mathbb{E}[Y(1)-Y(0)]$. Suppose that (i) $T$ is independent of $(Y(0),Y(1),R(0),R(1),X)$ and $0<P(T=1)<1$, and that (ii) for each $t\\in\\{0,1\\}$, $R(t)$ is independent of $(Y(0),Y(1),X)$ and $P(R(t)=1)>0$. Then $\\mathbb{E}[Y\\mid T=t,R=1]=\\mathbb{E}[Y(t)]$ for $t=0,1$, the law of $X$ given $\\{T=t,R=1\\}$ is the law of $X$, and every solution of the normal equations of the complete-case regression of $Y$ on $(1,T,X)$ has coefficient $\\tau$ on $T$. Condition (ii) does not follow from (i) and equal completion rates. In the first case of the table, (i) holds and both completion rates are $1/2$, while $\\tau=0$ and the coefficient on $T$ is $12/25$.", "evidence": "The evidence is an exact counterexample, computed in rational arithmetic. Let $0<b<a<1$. Let $U$ and $X$ be independent Bernoulli variables with parameter $1/2$, and set $Y(0)=Y(1)=U$, so that the effect is zero for every unit. Given $U$, let $R(0)$ and $R(1)$ be independent with\n\\[\n\\begin{aligned}\nP(R(0)=1\\mid U=0)&=a, & P(R(0)=1\\mid U=1)&=b,\\\\\nP(R(1)=1\\mid U=0)&=b, & P(R(1)=1\\mid U=1)&=a,\n\\end{aligned}\n\\]\nand let $T$ be a Bernoulli variable with parameter $1/2$, independent of all the others. Then the completion rate is $(a+b)/2$ in each arm. The complete-case mean of $Y$ is $b/(a+b)$ in the control arm and $a/(a+b)$ in the treated arm, and the difference is $\\delta=(a-b)/(a+b)>0$. Among the complete cases $X$ is independent of $(T,Y)$ and the two arms are equally likely. Solving the normal equations of the complete-case regression of $Y$ on $(1,T,X)$, I find that the coefficient on $T$ is also $\\delta$. Two choices of $a$ and $b$ are listed in the table. For each choice the table shows a sample that the process produces with positive probability, and in this sample the chi-squared statistic for completion by arm is exactly zero.\n\\begin{audittable}{lrr}{Two cases of the counterexample. The true effect is zero in both. The sample of each column is divided evenly between $X=0$ and $X=1$ in every cell.}\n & First case & Second case \\\\\n\\hline\n$a$, $b$ & $37/50$, $13/50$ & $499/500$, $489/500$ \\\\\nAttrition in each arm & $1/2$ & $3/250$ ($1.2\\%$) \\\\\nComplete-case mean of $Y$, control and treated & $13/50$, $37/50$ & $489/988$, $499/988$ \\\\\nComplete-case coefficient on $T$ & $12/25=0.48$ & $5/494\\approx0.0101$ \\\\\nSample size & $400$ & $4000$ \\\\\nMissing and observed in each arm & $100$, $100$ & $24$, $1976$ \\\\\nObserved $Y=0$ and $Y=1$, control & $74$, $26$ & $998$, $978$ \\\\\nObserved $Y=0$ and $Y=1$, treated & $26$, $74$ & $978$, $998$ \\\\\nPearson chi-squared statistic, $p$-value & $0$, $1$ & $0$, $1$ \\\\\n\\end{audittable}\nIn both cases the distribution of the observed data, that is, of $(T,X,R)$ and of $Y$ when $R=1$, is the same as under a second process in which outcomes are missing completely at random and the true effect is $\\delta$. This implies that no test computed from the observed data can tell the two processes apart.\n\nThe corrected statement below has two hypotheses: (i) the treatment is randomized, and (ii) completion under each treatment is independent of the potential outcomes and the covariates and has positive probability. It holds for the following reason. By (i), the event $\\{T=t,R=1\\}$ equals $\\{T=t,R(t)=1\\}$ and has probability $P(T=t)P(R(t)=1)>0$. By (i) and (ii), the law of $(Y(t),X)$ given this event is its unconditional law. This implies the first two conclusions. Let $p=P(T=1\\mid R=1)$, and let $(\\beta_0,\\beta_T,\\beta_X)$ solve the normal equations. Subtracting $p$ times the equation for the intercept from the equation for $T$, I obtain\n\\[\n\\mathbb{E}\\bigl[(T-p)(Y-\\beta_0-\\beta_TT-\\beta_X'X)\\mid R=1\\bigr]=0.\n\\]\nNote that $\\mathbb{E}[(T-p)X\\mid R=1]=0$, since $X$ has the same law in both arms of the complete cases, and that $\\mathbb{E}[(T-p)T\\mid R=1]=p(1-p)$. The display then reads $p(1-p)\\{\\mathbb{E}[Y\\mid T=1,R=1]-\\mathbb{E}[Y\\mid T=0,R=1]\\}=\\beta_T\\,p(1-p)$, and I conclude that $\\beta_T=\\tau$.", "problem": "The plan treats a chi-squared test that does not reject equal completion rates as showing that outcomes are missing at random. On that basis it analyzes only the respondents who completed the survey. The conclusion does not follow. I write $T$ for the randomized treatment, $Y(0)$ and $Y(1)$ for the potential outcomes, $R(0)$ and $R(1)$ for the indicators that the outcome would be observed under each assignment, and $X$ for the pretreatment covariates. The outcome $Y=Y(T)$ is observed when $R=R(T)$ equals $1$. Under randomization the test addresses the hypothesis\n\\[\nP(R=1\\mid T=0)=P(R=1\\mid T=1),\n\\]\nthat is, $\\mathbb{E}[R(0)]=\\mathbb{E}[R(1)]$. This hypothesis concerns how many outcomes are missing in each arm, and not which outcomes are missing. The complete-case difference in means converges to $\\tau_{\\mathrm{CC}}=\\mathbb{E}[Y(1)\\mid R(1)=1]-\\mathbb{E}[Y(0)\\mid R(0)=1]$. For the average effect $\\tau=\\mathbb{E}[Y(1)-Y(0)]$, I have\n\\[\n\\tau_{\\mathrm{CC}}-\\tau=\\bigl\\{\\mathbb{E}[Y(1)\\mid R(1)=1]-\\mathbb{E}[Y(1)]\\bigr\\}-\\bigl\\{\\mathbb{E}[Y(0)\\mid R(0)=1]-\\mathbb{E}[Y(0)]\\bigr\\}.\n\\]\nEqual completion rates do not make either bracket vanish. The implication fails even when the two rates are known to be equal, and the gap is therefore not one of statistical power.\n\nThe rule of the plan was applied. The analysis program of the replication files tests completion by arm and carries the comment ``attrition not differential, so as per PAP we drop missings''. It then drops the respondents with a missing outcome before the Study 2 regressions and the pooled regressions. The notes to Figures OA2a and OA3 say that the intervals of Study 2 are complete-case intervals because there was no differential attrition.", "what_survives": "The sentence of the article on p.~6 reports the result of a test, and it stands; I did not repeat the test for this finding. The randomized design, the arithmetic of the regressions and the comparisons among respondents who completed the survey are unaffected. The bounds used for Study 1 do not rely on this inference. No bias, wrong estimate, wrong $p$-value or failed interval in Study 2 is established. What changes is the justification. The Study 2 and pooled estimates describe the randomized sample only under an additional assumption about missingness, and the test cannot supply it. If the assumption is stated, it is a new hypothesis that the data may not satisfy. If bounds replace it, the conclusions may become weaker."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E017", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three entries of the reference list. The middle entry reads: Harwell, Drew. 2021. Remember the 'deepfake cheerleader mom'? Prosecutors now admit they can't prove fake-video claims (March 14, 2021). https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/. A red outline surrounds the word March. The entry above is H.R.6395, the National Defense Authorization Act for Fiscal Year 2021, and the entry below is Kuklinski et al. 2000.", "caption": "References, p. 13 (PDF p. 13): the Harwell entry between its two neighbours. The outline marks the month “March”; the link printed in the same entry carries 2021/05/14.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E018-b.png"}, {"alt": "Crop of three entries of the reference list. The middle entry reads: Ternovski, John, and Lilla Orr. 2022. Evidence of rising rates of inattentiveness on Lucid in 2020, https://osf.io/preprints/socarxiv/8sbe4/. A red outline surrounds the words Ternovski, John, and Lilla Orr. 2022. The entry above is Swire et al. 2017 and the entry below is Thalen 2021, which is dated May 17, 2021.", "caption": "References, p. 14 (PDF p. 14): the entry cited in footnote 6, between its two neighbours. The outline marks the authors and year, which are those of the 2022 journal article; the title and link that follow are those of the 2020 memo.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E018.png"}, {"alt": "Crop of footnote 6. It reads: Lucid is an increasingly popular alternative to Amazon Mechanical Turk for social science survey research. Many well-known findings have been replicated on Lucid, suggesting the platform is capable of providing high-quality data (Coppock and McClellan 2019; Peyton, Huber, and Coppock 2021). During the COVID-19 pandemic, Ternovski and Orr (2022) found that Lucid data can provide reliable data when researchers screen on attentiveness, which we do here. A red outline surrounds the words Ternovski and Orr (2022).", "caption": "Footnote 6, p. 3 (PDF p. 3). The outline marks the citation “Ternovski and Orr (2022)”, which the footnote gives as the source of its statement about screening on attentiveness.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E018-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The Harwell reference should be dated May 14, 2021, not March 14, and the reference “Ternovski and Orr 2022” joins the authors and year of a 2022 journal article to the title and link of a 2020 memo by four authors.", "id": "P034-E018", "location": "References, entry “Harwell, Drew. 2021.”, p. 13 (PDF p. 13), and entry “Ternovski, John, and Lilla Orr. 2022.”, p. 14 (PDF p. 14), which is cited in footnote 6, p. 3 (PDF p. 3)", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{References, p.~13.} Harwell, Drew. 2021. ``Remember the `deepfake cheerleader mom'? Prosecutors now admit they can't prove fake-video claims'' (\\hl{March} 14, 2021). \\url{https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/}.\n\n\\emph{References, p.~14.} \\hl{Ternovski, John, and Lilla Orr. 2022.} ``Evidence of rising rates of inattentiveness on Lucid in 2020,'' \\url{https://osf.io/preprints/socarxiv/8sbe4/}.\n\n\\emph{Footnote~6, p.~3.} During the COVID-19 pandemic, Ternovski and Orr (2022) found that Lucid data can provide reliable data when researchers screen on attentiveness, which we do here.", "corrected": "\\emph{References, p.~13.} Harwell, Drew. 2021. ``Remember the `deepfake cheerleader mom'? Prosecutors now admit they can't prove fake-video claims'' (\\hl{May} 14, 2021). \\url{https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/}.\n\n\\emph{References, p.~14.} The entry should cite one work in full. If the memo is the source of footnote~6, then footnote~6 cites Aronow et al.\\ (2020) and the entry is\n\n\\hl{Aronow, P. M., Joshua Kalla, Lilla Orr, and John Ternovski. 2020.} ``Evidence of rising rates of inattentiveness on Lucid in 2020,'' \\url{https://osf.io/preprints/socarxiv/8sbe4/}.\n\nIf the journal article is the source, then the entry is\n\nTernovski, John, and Lilla Orr. 2022. ``\\hl{A Note on Increases in Inattentive Online Survey-Takers Since 2020}.'' \\emph{Journal of Quantitative Description: Digital Media} 2. \\url{https://doi.org/10.51685/jqd.2022.002}.\n\nIn the second case, footnote~6 should describe screening on attentiveness as a way to reduce measurement error, to be used with continuing checks of data quality, and not as sufficient to validate the data.", "evidence": "The evidence is a check of the pages against the public pages of the cited sources. The page of the Harwell article at the publisher displays the date May 14, 2021 below the headline. The journal lists Ternovski and Orr as the authors of ``A Note on Increases in Inattentive Online Survey-Takers Since 2020'' (\\emph{Journal of Quantitative Description: Digital Media} 2, DOI 10.51685/jqd.2022.002, published on 7 February 2022). A library catalogue entry for the memo, with DOI 10.31235/osf.io/8sbe4, lists the four authors and the date 15 September 2020. The full text of the memo and the history of its versions on OSF were not obtained. These sources do not exclude that the two works are stages of one project or that the author list of the memo changed between versions.", "problem": "The reference list has two slips. In the Harwell entry, the date in parentheses is March 14, 2021, while the publisher dates the article May 14, 2021, as the link of the entry itself does (\\texttt{2021/05/14}).\n\nThe second entry joins the data of two publications. The title ``Evidence of rising rates of inattentiveness on Lucid in 2020'' and the SocArXiv identifier \\texttt{8sbe4} belong to a memo that is catalogued under four authors, Aronow, Kalla, Orr and Ternovski (2020). The pre-analysis plan of Study~2 cites the memo under those four names (p.~6, footnote~3). The authors and year of the entry, Ternovski and Orr (2022), belong to the journal article ``A Note on Increases in Inattentive Online Survey-Takers Since 2020'', which has a different title and a different DOI. The entry does not identify which of the two footnote~6 relies on, and the evidence does not determine which one the entry was meant to cite.", "what_survives": "No result of the paper depends on either entry. I did not establish whether the statement of footnote~6 about Lucid is correct. Both works favour screening on attentiveness. The journal article adds that attention checks ``do contain drawbacks and are not a panacea'' (pp.~25--26), and the abstract of the memo recommends stringent attention checks."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E018", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of the article under the heading on heterogeneous treatment effects. The marked sentence reads: We found some evidence that participants surveyed before the election displayed higher levels of distrust in political videos than those surveyed afterwards (p less than 0.05). A red outline surrounds the parenthetical p less than 0.05.", "caption": "Section 3.1, p. 9 (PDF p. 9). The outline marks the $p$-value reported for the difference between participants surveyed before and after the election.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019.png"}, {"alt": "Crop of Table OA11 with its title: Study 2, Some evidence of more distrust immediately before a major election. The table has columns Deepfake?, Unconvinced and Don't Know if Real. Red outlines surround the three entries of the first column: minus 0.14 for the Post Election Indicator, minus 0.01 for the 95 percent Upper Bound and minus 0.27 for the 95 percent Lower Bound. The other columns read minus 0.05, 0.02, minus 0.12 and minus 0.01, 0.04, minus 0.05.", "caption": "Appendix, Table OA11, p. 16. The outlines mark the three entries of the column “Deepfake?”, which differ from the table saved in the replication files.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019-b.png"}, {"alt": "Listing of four numbered lines of Stata code under the heading ORIGINAL CODE: 2-study2_data_cleanup_FINAL.do, SOURCE LINES 312-315. Line 313 is the comment: respondent's device: computer fl_25 ; mobile fl_87. Line 314 generates mobile equal to 0 if fl_25 is empty and fl_87 is not empty. Line 315 replaces mobile by 1 if fl_25 is not empty and fl_87 is empty. Nothing is outlined.", "caption": "Replication code, 2-study2_data_cleanup_FINAL.do, lines 312–315. The comment on line 313 gives the mapping of the two routes to devices, and lines 314–315 assign mobile the opposite values.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019-code-a.png"}, {"alt": "Listing of numbered excerpts of Stata code under the heading ORIGINAL CODE: 4-Supplementary-Materials_FINAL.do. Lines 3 to 7 include local figures=0 and local tables=0. Lines 190 to 193 open a block with the condition that figures equals 1. Lines 207 to 211 define the global macro covars as c.age i.gender i.party i.region c.hhi elder-pro_vax conlib imputed_conlib bigfive* pol_know_total polinterest and run a regression with it. Lines 597 to 604 set up the saved table trust_SM_table.xlsx and, on line 603, regress trust_tv_actually on tr_warn and the macro covars for Study 2 participants with romney_party equal to 1. Nothing is outlined.", "caption": "Replication code, 4-Supplementary-Materials_FINAL.do, excerpts. Lines 6–7 set both switches to zero, lines 207–208 define covars inside the block for figures that opens at line 190, and line 603 is a table regression that reads the macro.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019-code-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In the replication files, the device indicator of Study 2 is reversed, the supplementary script uses the covariate macro before defining it when only tables are requested, and the saved tables for Tables OA7, OA11 and OA12 differ from the printed ones; no printed estimate is shown to be wrong.", "id": "P034-E019", "location": "Section 3.1, p. 9 (PDF p. 9); Appendix, Tables OA7 (p. 14), OA11 (p. 16) and OA12 (p. 17), and the saved tables for them; replication code, lines 313–315 of 2-study2_data_cleanup_FINAL.do and lines 6–7, 593–766 and 840–841 of 4-Supplementary-Materials_FINAL.do", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{\\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}, lines 313--315.}\n\n\\texttt{/*respondent's device: computer fl\\_25 ; mobile fl\\_87*/}\n\n\\texttt{gen mobile=\\hl{0} if fl\\_25==\"\" \\& fl\\_87!=\"\"}\n\n\\texttt{replace mobile=\\hl{1} if fl\\_25!=\"\" \\& fl\\_87==\"\"}\n\n\\emph{\\texttt{4-Supplementary-Materials\\_FINAL.do}, lines 6, 7 and 594.}\n\n\\texttt{local figures=0}\n\n\\texttt{local tables=0}\n\n\\texttt{reg trust\\_tv\\_actually tr\\_warn \\hl{\\$covars} if romney\\_party==1 \\& study==2, robust}\n\n\\emph{Appendix, Table OA11, p.~16, column ``Deepfake?''.} Post Election Indicator: $-0.14$. 95\\% Upper Bound: $-0.01$. 95\\% Lower Bound: $-0.27$.\n\n\\emph{Section 3.1, p.~9.} We found some evidence that participants surveyed before the election displayed higher levels of distrust in political videos than those surveyed afterwards ($p<0.05$).", "corrected": "\\emph{\\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}, lines 314--315.}\n\n\\texttt{gen mobile=\\hl{1} if fl\\_25==\"\" \\& fl\\_87!=\"\"}\n\n\\texttt{replace mobile=\\hl{0} if fl\\_25!=\"\" \\& fl\\_87==\"\"}\n\n\\emph{\\texttt{4-Supplementary-Materials\\_FINAL.do}.} The definition of lines 840--841 is to be placed before line 593, outside the blocks for figures:\n\n\\texttt{\\hl{global covars c.age i.gender i.party i.region c.hhi elder-pro\\_vax conlib ///}}\n\n\\texttt{\\hl{imputed\\_conlib bigfive* pol\\_know\\_total polinterest}}\n\nI reproduce the printed tables in the translated calculation of \\fid{P034-N011}. After the native scripts are repaired, the saved workbooks for Tables OA7, OA11 and OA12 should be regenerated from the Study 2 data and replaced with verified outputs, with any remaining difference from the printed tables explained.", "evidence": "I checked the two device assignments, traced the supplementary script and compared the saved tables with the printed ones. I then recalculated Tables OA7, OA11 and OA12 from the Study 2 data in \\fid{P034-N011}, translating the relevant code and using the supplied cleaned covariates. Stata was not run. The recalculation reproduces the printed tables under the code sample; it does not establish how the discrepant saved workbooks were produced.\n\\begin{audittable}{llll}{The columns of Tables OA7, OA11 and OA12 in which the saved tables differ from the printed ones. Each entry is an estimate with its lower and upper bounds, and the saved values are rounded to three decimals.}\nTable & Column & Saved in the replication files & Printed \\\\\nOA7 & Trust TV & $-0.012$ $[-0.095, 0.070]$ & $0.01$ $[-0.07, 0.09]$ \\\\\nOA7 & Trust Online & $-0.032$ $[-0.117, 0.052]$ & $-0.04$ $[-0.12, 0.04]$ \\\\\nOA11 & Deepfake? & $-0.121$ $[-0.255, 0.012]$ & $-0.14$ $[-0.27, -0.01]$ \\\\\nOA11 & Don't Know if Real & $-0.003$ $[-0.047, 0.042]$ & $-0.01$ $[-0.05, 0.04]$ \\\\\nOA12 & Deepfake? & $0.132$ $[-0.135, 0.400]$ & $0.12$ $[-0.14, 0.39]$ \\\\\n\\end{audittable}\nIn the first column of Table OA11, the width of the saved interval implies a standard error of $0.0681$. The saved estimate is then $1.777$ standard errors from zero, which is less than $1.96$. The saved tables for Tables OA5, OA6, OA8, OA9 and OA10 agree with the printed tables in every entry.", "problem": "There are three defects in the replication files. First, the cleaning script of Study 2 assigns \\texttt{mobile} the values opposite to its own comment and to the cleaning script of Study 1: a participant with only the mobile route \\texttt{fl\\_87} filled gets $0$, and a participant with only the computer route \\texttt{fl\\_25} filled gets $1$. Second, the supplementary script has one switch for figures and one for tables, and every definition of the global macro \\texttt{covars} before line 840 lies inside a block that runs only when the switch for figures is on. With the switch for tables on and the switch for figures off, 16 regressions in the table blocks read the macro before it is defined, among them the regressions for Tables OA7, OA11 and OA12. The macro is then empty in a fresh Stata session, and it holds the six covariates of Study 1 if the main analysis script has run first, as the README instructs; in neither case do these regressions use the expanded covariate list of Study 2. Third, the saved tables for Tables OA7, OA11 and OA12 differ from the printed tables in 5 of 6, 4 of 9 and 3 of 9 entries. In the first column of Table OA11, the saved estimate is $-0.121$ with the interval $[-0.255, 0.012]$, which contains zero. The printed entry is $-0.14$ with the interval $[-0.27, -0.01]$, and the sentence on p.~9 reports $p<0.05$. The replication files do not show which specification or data produced the saved tables, or whether the undefined macro explains the difference.", "what_survives": "No printed estimate, interval or $p$-value is shown to be wrong. The device indicator enters no regression. In Tables OA7 and OA12, every interval contains zero in the saved version and in the printed one. The main analysis script and the supplementary script each define the covariates of Study 2 before their own regression for the comparison across the election, so the model behind the sentence on p.~9 is specified apart from the table blocks. In the recalculation of \\fid{P034-N011}, I reproduce all 24 printed entries of Tables OA7, OA11 and OA12 with the code sample; the election comparison has $p=0.038$, and $p=0.047$ with the plan sample, so the $p<0.05$ sentence stands. This does not establish how the discrepant saved tables were produced."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E019", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the Introduction. Two red boxes mark the citation '(Wittenberg et al. 2021', which is split over two lines, in the sentence 'recent randomized experiments on the impacts of deepfakes in American politics have found no evidence that people believe the content of the manipulated videos (Wittenberg et al. 2021; Vaccari and Chadwick 2020).'", "caption": "Introduction, first paragraph of p. 2 (PDF p. 2). The marked citation is given, together with Vaccari and Chadwick (2020), for experiments on deepfakes that found no evidence of belief in the manipulated videos.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N001.png"}, {"alt": "Crop of one entry of the reference list, which begins 'Wittenberg, Chloe, Ben M Tappin, Adam J Berinsky, and David G Rand. 2021.' A red box marks the title '(minimal) persuasive advantage of political video over text', which is followed by 'Proceedings of the National Academy of Sciences 118 (47)' and a DOI link.", "caption": "Reference list, p. 14 (PDF p. 14). The marked title of the cited article describes a comparison of political video with text.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Wittenberg et al. (2021) is not an experiment on deepfakes and does not support the statement that such experiments found no evidence that people believe manipulated videos.", "id": "P034-N001", "location": "Introduction, first paragraph of p. 2 (PDF p. 2), sentence beginning “However, recent randomized experiments”; reference entry on p. 14 (PDF p. 14)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~2: ``However, recent randomized experiments on the impacts of deepfakes in American politics have found no evidence that people believe the content of the manipulated videos (\\hl{Wittenberg et al. 2021}; Vaccari and Chadwick 2020). Barari, Lucas, and Munger (2021) found that deepfakes were persuasive, but that their effects were comparable to those of textual misinformation.''\n\np.~14, reference list: Wittenberg, Chloe, Ben M Tappin, Adam J Berinsky, and David G Rand. 2021. ``The (minimal) persuasive advantage of political video over text.'' \\emph{Proceedings of the National Academy of Sciences} 118 (47).", "corrected": "p.~2: ``However, \\hl{a recent randomized experiment on a deepfake of American political figures found that viewers were not significantly more likely to be deceived when the deepfake was shown without an educational reveal than with one, although they were more uncertain (Vaccari and Chadwick 2020)}. Barari, Lucas, and Munger (2021) found that deepfakes were persuasive, but that their effects were comparable to those of textual misinformation.''\n\nIf Wittenberg et al.\\ (2021) is cited, it should be cited separately, as a comparison of ordinary political video with text and not as an experiment on deepfakes. The comparison on p.~7 can stay qualitative; it is clearer if it names the comparator of that study, the same deepfake shown with the reveal.", "evidence": "The evidence is a check of the cited sources. I read Wittenberg et al.\\ (2021) in the publisher's typeset version posted by one of its authors (\\emph{PNAS} 118(47), seven pages), and Vaccari and Chadwick (2020) in the version of record (\\emph{Social Media + Society} 6(1), pp.~1--13). The passages quoted above are on p.~5 of the first and on p.~4 of the second. On p.~6 of the second, the analysis reads ``A smaller, though by no means negligible, group (16\\%) was deceived''.", "problem": "The sentence attributes to two studies the finding that randomized experiments on deepfakes found no evidence that people believe the manipulated videos. The first of the two, Wittenberg et al.\\ (2021), did not study deepfakes. Its experiments compared ordinary videos, on politics and on COVID-19, with transcripts and with controls. Its Discussion (p.~5) describes the subject as ``belief in unmanipulated content'' and says that ``future work should explore whether our results replicate in the context of misinformation, including deepfakes''. One of the two citations therefore does not support the sentence.\n\nThe second citation is an experiment on a deepfake, and its result is narrower than the sentence. Vaccari and Chadwick (2020) showed British respondents a deepfake of American political figures, with or without an educational reveal. They report that 16\\% of the respondents were deceived, that the difference in deception between the versions was not statistically significant, and that uncertainty was higher without the reveal. The design has no control group (p.~4: ``Our study does not include a control group'').", "what_survives": "The two experiments of the paper, their estimates and their bounds do not use the cited findings. What changes is the summary of earlier work in the motivation, which has to be narrowed. I did not establish that the phrase ``prior research has found that such videos have minimal effects'' in the abstract is wrong, or that the comparison with Vaccari and Chadwick on p.~7 is wrong. I could not obtain the 2021 preprint of Barari, Lucas and Munger, and the sentence about it was not checked against its text."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N001", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the Introduction. A red box marks the citation 'Pennycook, Bear, et al. 2020' in the sentence 'Official warnings about fake news similarly induce participants to disbelieve true headlines (Clayton et al. 2020; Pennycook, Bear, et al. 2020).'", "caption": "Introduction, third paragraph of p. 2 (PDF p. 2). The marked citation is one of the two given for the statement that warnings lead participants to disbelieve true headlines.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N002.png"}, {"alt": "Crop of the first six lines of Section 4. Red boxes mark the words 'consistently found' and the citation 'Pennycook, Bear, et al. 2020' in the sentence 'Our findings add to the growing body of literature on textual misinformation, which has consistently found that warnings about fake news may help readers reject misinformation but may have the undesirable effect of increasing Americans' disbelief in true news stories (e.g., Pennycook, Bear, et al. 2020; Clayton et al. 2020).'", "caption": "Section 4, first sentence, p. 9 (PDF p. 9). The marked words “consistently found” and the marked citation repeat the attribution.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N002-b.png"}, {"alt": "Crop of one entry of the reference list, which begins 'Pennycook, Gordon, Adam Bear, Evan T Collins, and David G Rand. 2020.' Two red boxes mark the words 'increases perceived accuracy of headlines without warnings', split over two lines, at the end of the title 'The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings.'", "caption": "Reference list, p. 13 (PDF p. 13). The marked words of the title of the cited article state an increase in the perceived accuracy of headlines without warnings.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N002-c.png"}], "formal_release": null, "formal_statements": [], "headline": "Pennycook, Bear, Collins and Rand (2020) does not report that warnings lower belief in true headlines; of the two works cited for that statement, only Clayton et al. (2020) supports it.", "id": "P034-N002", "location": "Introduction, third paragraph of p. 2 (PDF p. 2); Section 4, first sentence, p. 9 (PDF p. 9); reference entry on p. 13 (PDF p. 13)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~2: ``Official warnings about fake news similarly induce participants to disbelieve true headlines (Clayton et al.\\ 2020; \\hl{Pennycook, Bear, et al. 2020}).''\n\np.~9: ``Our findings add to the growing body of literature on textual misinformation, which \\hl{has consistently found} that warnings about fake news may help readers reject misinformation but may have the undesirable effect of increasing Americans' disbelief in true news stories (e.g., \\hl{Pennycook, Bear, et al. 2020}; Clayton et al.\\ 2020).''\n\np.~13, reference list: Pennycook, Gordon, Adam Bear, Evan T Collins, and David G Rand. 2020. ``The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings.'' \\emph{Management Science} 66 (11): 4944--57.", "corrected": "p.~2: ``Official warnings about fake news similarly induce participants to disbelieve true headlines (\\hl{Clayton et al. 2020}).''\n\np.~9: ``Our findings add to the growing body of literature on textual misinformation, which \\hl{has found that general warnings about fake news can reduce belief in both false and true headlines (e.g., Clayton et al. 2020)}.''\n\nIf Pennycook, Bear, et al.\\ (2020) is kept, it should be described separately: when warnings were attached to a subset of false headlines, the headlines without warnings were rated as more accurate.", "evidence": "The evidence is a check of the cited sources. I read the version of record of Pennycook et al.\\ (\\emph{Management Science} 66(11), pp.~4944--4957), including the methods and results of both of its studies. For Clayton et al.\\ (\\emph{Political Behavior} 42, pp.~1073--1095) I read the publisher's abstract only, and I did not obtain the full text.", "problem": "Both sentences cite Pennycook, Bear, Collins and Rand (2020) for the finding that warnings about fake news lower belief in true headlines. On belief, that article reports the opposite. In its Study 1, the proportion rating true headlines accurate was .542 when warnings were attached to a subset of the false headlines and .530 in the control (p.~4948): ``headlines that were not tagged in the warning treatment were rated as more accurate than those in the control''. The article does report one adverse result for true headlines, in its Study 2 (p.~4953): untagged true headlines were shared at a rate of 35.2\\% under warnings plus verification and 40.1\\% in the control. That outcome is sharing and not belief. The other citation supports the statement: the abstract of Clayton et al.\\ (2020) says that ``exposure to a general warning decreased belief in the accuracy of true headlines''. The phrase ``has consistently found'' on p.~9 thus rests on one of the two works that it cites.", "what_survives": "The analogy with textual misinformation stands on Clayton et al.\\ (2020), and no design, estimate or test in the paper depends on the citation. The check does not show that the wider literature disagrees on this point; I did not review it."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N002", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the top of Table OA3, titled Study 2 - Hand-coded Romney facts across conditions, with the columns No Info about Deepfakes, Info about Deepfakes and Total. The rows shown, with percentages in the three columns and counts below them, are Don't Know / No Response Given (28.2, 27.0, 27.6 percent; N 818, 777, 1,595), Is Republican (16.5, 16.5, 16.5 percent; N 477, 475, 952), Ran for President (9.7, 10.7, 10.2 percent; N 281, 309, 590) and Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.) (9.2, 9.3, 9.3 percent; N 266, 268, 534). A red outline surrounds the last percentage of the fourth row, 9.3 percent, in the Total column.", "caption": "Online Appendix, Table OA3, p. 11: the head of the table and its first four categories. The outline marks the Total percentage of “Subjective Evaluation of Character”, printed as 9.3% above the count 534.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N010.png"}, {"alt": "Crop of the bottom of Table OA3. The rows shown, with percentages in the three columns and counts below them, are Stance on Other Policies (0.8, 0.9, 0.9 percent; N 24, 25, 49), Is/Was Pro-Life (0.3, 1.4, 0.8 percent; N 9, 39, 48), Believe Video is Real (0.3, 0.5, 0.4 percent; N 9, 15, 24), Believe Video is Fake (0.2, 0.1, 0.2 percent; N 5, 4, 9) and Total (100, 100, 100 percent; N 2,898, 2,877, 5,775). A red outline surrounds the percentage 0.9 percent in the Total column of the first row.", "caption": "Online Appendix, Table OA3, p. 11: the last four categories and the foot of the table. The outline marks the Total percentage of “Stance on Other Policies”, printed as 0.9% above the count 49; the last row gives the column totals 2,898, 2,877 and 5,775.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N010-b.png"}], "formal_release": null, "formal_statements": [{"id": "P034-T5", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T5"}], "headline": "In the Total column of Table OA3, the percentage for “Subjective Evaluation of Character” should be 9.2%, not 9.3%, and the percentage for “Stance on Other Policies” should be 0.8%, not 0.9%.", "id": "P034-N010", "location": "Online Appendix, Table OA3, column “Total”, rows “Subjective Evaluation of Character” and “Stance on Other Policies”, p. 11 (Appendix PDF p. 11)", "pdf_page": 11, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P034-T5"], "text": {"as_printed": "\\begin{audittable}{lrrr}{Table OA3: Study 2 - Hand-coded Romney facts across conditions. The two rows at issue and the last row, each with its row of counts.}\n & No Info about Deepfakes & Info about Deepfakes & Total \\\\\nSubjective Evaluation of Character (e.g., trusworthy, traitor, etc.) & 9.2\\% & 9.3\\% & \\hl{9.3\\%} \\\\\nN & 266 & 268 & 534 \\\\\nStance on Other Policies & 0.8\\% & 0.9\\% & \\hl{0.9\\%} \\\\\nN & 24 & 25 & 49 \\\\\nTotal & 100\\% & 100\\% & 100\\% \\\\\nN & 2,898 & 2,877 & 5,775 \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lrrr}{Table OA3, the two rows as corrected.}\n & No Info about Deepfakes & Info about Deepfakes & Total \\\\\nSubjective Evaluation of Character (e.g., trusworthy, traitor, etc.) & 9.2\\% & 9.3\\% & \\hl{9.2\\%} \\\\\nN & 266 & 268 & 534 \\\\\nStance on Other Policies & 0.8\\% & 0.9\\% & \\hl{0.8\\%} \\\\\nN & 24 & 25 & 49 \\\\\n\\end{audittable}\nEach percentage in the Total column is the count in its row divided by 5,775, the count at the foot of the column, rounded to one decimal place.", "evidence": "The evidence is a recalculation from the printed counts in exact decimal arithmetic. I have\n\\[\n\\frac{534}{5775}\\times 100 = 9.246753\\ldots, \\qquad \\frac{49}{5775}\\times 100 = 0.848484\\ldots,\n\\]\nand at one decimal place the first value rounds to $9.2$ and the second to $0.8$. The other 37 percentages of the table agree with their counts, and the counts of each column sum to the printed totals 2,898, 2,877 and 5,775.", "problem": "Each percentage in the table is the count in its row divided by the count at the foot of its column, rounded to one decimal place. In the Total column the counts are 534 and 49. Dividing them by 5,775, I find 9.2\\% and 0.8\\%. The table prints 9.3\\% and 0.9\\%.", "what_survives": "Only these two cells change. The counts, the other percentages of Table~OA3 and every analysis of the paper are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N010", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the end of the passage on Study 1 and the passage on Study 2 in Section 2 of the Appendix. The last paragraph on Study 1 reads: We also had two minor deviations from the PAP. To be consistent with Study 2's PAP, we 1) dropped the one IP address that was randomized three times (there were no other IP duplicates that watched our video) and 2) used robust standard errors. These modifications have negligible impacts on the reported results. Under the heading Study 2, a red box marks the sentence 'There were no deviations from the PAP in Study 2.'", "caption": "Appendix, Section 2, p. 5 (PDF p. 5 of the Appendix). The marked sentence says that Study 2 had no deviations from its pre-analysis plan.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N011.png"}, {"alt": "Crop of Section 9, Exclusions, of the amended plan. After the sentence 'We exclude all participants who satisfy any one of the following conditions:' come four bullets: did not consent or meet the eligibility criteria; 'Have duplicate IP addresses and reached randomization.', marked with a red box; failed any one of the audio, visual, and attention checks; self-reported age was not within one year of their age according to Lucid. After a sentence on participants who could not identify Mitt Romney's political party, the text reads 'For SQ3_election1 and SQ4_election2, we exclude respondents who:' followed by two bullets, each marked with a red box: 'Take any portion of the survey between 8am and midnight November 3rd' and 'Or take any portion of the survey after 8am November 10th'.", "caption": "Amended pre-analysis plan for Study 2, Section 9, “Exclusions” (PDF p. 6 of the plan). The marked lines are the exclusion for duplicate IP addresses, which names respondents who reached randomization, and the two exclusions by time for the analyses before and after the election.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N011-b.png"}, {"alt": "Crop of Table OA11 with its title: Table OA11: Study 2 - Some evidence of more distrust immediately before a major election. The columns are Deepfake?, Unconvinced and Don't Know if Real. The row Post Election Indicator reads minus 0.14, minus 0.05, minus 0.01. The row 95 percent Upper Bound reads minus 0.01, 0.02, 0.04. The row 95 percent Lower Bound reads minus 0.27, minus 0.12, minus 0.05. Red boxes mark the first two entries of the column Deepfake?, minus 0.14 and minus 0.01.", "caption": "Appendix, Table OA11, p. 16 (PDF p. 16 of the Appendix). The table is computed on the 1,830 respondents who started the survey on November 1, 2, 8 or 9. With the exclusions of the amended plan, which leave 1,839 respondents, the two marked entries become $-0.13$ and $0.00$.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N011-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The Appendix says that Study 2 had no deviations from its pre-analysis plan, but the analyses before and after the election drop nine eligible respondents whom the amended plan keeps (1,830 analyzed instead of 1,839), and the cleaning code drops rows with duplicated IP addresses without requiring that they reached randomization, a rule that the plan does not state.", "id": "P034-N011", "location": "Appendix, Section 2, “Study 2”, p. 5 (PDF p. 5 of the Appendix); Tables OA11 and OA12, Appendix pp. 16–17; Section 3.1, p. 9 (PDF p. 9); amended pre-analysis plan for Study 2, Section 9 (PDF p. 6 of the plan, whose pages are not numbered); replication files, lines 334–336 of 3-analysis_FINAL.do and lines 23–30 and 195–196 of 2-study2_data_cleanup_FINAL.do", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Appendix p.~5, Section 2, under ``Study 2'': ``\\hl{There were no deviations from the PAP in Study 2.}''\n\nAmended pre-analysis plan for Study 2, Section 9: ``We exclude all participants who satisfy any one of the following conditions:'' The second of the four conditions is ``Have duplicate IP addresses and reached randomization.'' The section continues: ``For \\textit{SQ3\\_election1} and \\textit{SQ4\\_election2}, we exclude respondents who:'' with the two conditions ``Take any portion of the survey between 8am and midnight November 3$^{\\text{rd}}$'' and ``Or take any portion of the survey after 8am November 10$^{\\text{th}}$''.\n\n\\texttt{3-analysis\\_FINAL.do}, lines 334--336:\n\n\\texttt{reg deepfake i.tr\\_warn i.post\\_election\\_ind \\$covars if romney\\_party==1 ///}\n\n\\texttt{\\& study==2 \\& \\hl{(stata\\_start==mdy(11,1,2020) | stata\\_start==mdy(11,2,2020) ///}}\n\n\\texttt{\\hl{| stata\\_start==mdy(11,8,2020) | stata\\_start==mdy(11,9,2020))}, robust}\n\n\\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}, lines 23--25:\n\n\\texttt{duplicates tag ip, gen(dup)}\n\n\\texttt{tab dup}\n\n\\texttt{\\hl{drop if dup>0}}", "corrected": "Either the Appendix reports the two departures, or the code follows the plan.\n\nTo keep the printed tables, Appendix p.~5 can read: ``\\hl{Study 2 had two deviations from the PAP. First, the analyses for SQ3\\_election1 and SQ4\\_election2 (Tables OA11 and OA12) include only respondents who started the survey on November 1, 2, 8 or 9, 2020. This rule excludes nine respondents whom the exclusions of the PAP retain, so that these analyses use 1,830 respondents and not 1,839. Second, after removing previews and responses without respondent identifiers, we removed every remaining response whose IP address occurred more than once, whether or not it reached randomization, before imputing missing household income with the mean of the cleaned sample.}''\n\nTo follow the plan for these data, let $s_i$ and $e_i$ be the recorded start and end of the survey of respondent $i$, in 2020. The restriction to four start dates is replaced by the exclusion of respondent $i$ when\n\\[\n\\hl{\\bigl(s_i<\\text{November 4, 0:00 and } e_i\\ge\\text{November 3, 8:00}\\bigr)\\quad\\text{or}\\quad e_i>\\text{November 10, 8:00}}.\n\\]\nAt the precision shown, Tables OA11 and OA12 then take the values in the last column of the table above. In the cleaning, keep the existing population used to count duplicates, and identify randomized rows from the recorded treatment branches. Restrict \\texttt{drop if dup>0} \\hl{to those randomized rows}, and impute household income after that step. \\hl{Restrict the following \\texttt{isid rid} and \\texttt{isid ip} uniqueness checks to randomized rows as well.} The Appendix should also say among which rows a duplicate is counted.", "evidence": "The evidence is a recalculation from the public Study 2 data. The relevant cleaning and regression calculations were translated from Stata into Python, using the supplied cleaned data for other covariates; Stata was not run. The translated cleaning gives the 2,991 respondents of the cleaned data and the 1,925 of the analysis sample, 966 without and 959 with information. With the rules of the code, I reproduce all 24 printed entries of Tables OA7, OA11 and OA12. (The tables stored in the replication files differ from the printed ones; see \\fid{P034-E019}.) All 1,925 surveys have recorded start and end times; no endpoint lies on a boundary, and no survey spans the whole November 3 exclusion period. I then computed Tables OA11 and OA12 again with the exclusions of the plan.\n\\begin{audittable}{lll}{Tables OA11 and OA12 with the sample of the code and with the sample of the amended plan. Each entry is the coefficient with its interval of $\\pm 1.96$ robust standard errors, as in the Appendix.}\nTable and column & Code, $N=1{,}830$ & Plan, $N=1{,}839$ \\\\\n\\hline\nOA11, Deepfake? & $-0.140$ $[-0.272,\\,-0.008]$ & $-0.134$ $[-0.266,\\,-0.002]$ \\\\\nOA11, Unconvinced & $-0.047$ $[-0.116,\\,0.022]$ & $-0.047$ $[-0.116,\\,0.022]$ \\\\\nOA11, Don't Know if Real & $-0.009$ $[-0.053,\\,0.036]$ & $-0.005$ $[-0.050,\\,0.039]$ \\\\\nOA12, Deepfake? & $0.123$ $[-0.140,\\,0.386]$ & $0.133$ $[-0.129,\\,0.395]$ \\\\\nOA12, Unconvinced & $-0.059$ $[-0.195,\\,0.077]$ & $-0.057$ $[-0.193,\\,0.079]$ \\\\\nOA12, Don't Know if Real & $0.029$ $[-0.059,\\,0.116]$ & $0.027$ $[-0.061,\\,0.114]$ \\\\\n\\end{audittable}\nFor the first row, the $p$-value is $0.038$ with the sample of the code and $0.047$ with the sample of the plan. At the printed precision, six entries change with the sample of the plan. In Table OA11 the first column becomes $-0.13$ with upper bound $0.00$ (printed $-0.14$ and $-0.01$). In Table OA12 the first column becomes $0.13$ with lower bound $-0.13$ (printed $0.12$ and $-0.14$), the lower bound of the second column becomes $-0.19$ (printed $-0.20$), and the upper bound of the third becomes $0.11$ (printed $0.12$).\n\nFor the second departure, I repeated the cleaning and removed only the duplicated rows that reached randomization. The effect of the information on the main outcome of Study 2 is $0.263595$ with either mean of income, and no printed entry of Tables OA7, OA11 and OA12 changes.", "problem": "The code of Study 2 departs from the amended pre-analysis plan in two places, and the Appendix reports no deviation.\n\nThe first departure concerns the analyses before and after the election, which are Tables OA11 and OA12 and the sentence of Section 3.1 on p.~9. The plan excludes a respondent who took any portion of the survey between 8am and midnight on November 3, or after 8am on November 10. The code keeps a respondent only if the survey was started on November 1, 2, 8 or 9; the regressions behind the two tables, on lines 728--766 of \\texttt{4-Supplementary-Materials\\_FINAL.do}, carry the same restriction. Of the 1,925 respondents in the analysis sample, the code drops 95 and the plan drops 86. The nine respondents whom only the code drops took the survey outside the two excluded periods. Six started and finished on November 3 before 8am, one took it on the evening of November 4, and two started and finished on November 10 before 8am. The code keeps no respondent whom the plan excludes. The tables are computed on 1,830 respondents, and the plan specifies 1,839.\n\nThe second departure concerns duplicate IP addresses. The plan excludes participants who ``Have duplicate IP addresses and reached randomization.'' After removing previews and rows without respondent identifiers, the code removes every remaining row whose IP address occurs more than once, whether or not it reached randomization. At that step 176 rows have a duplicated address; 62 of them reached randomization and 114 did not. If duplicates are counted among these rows under both rules, then the two rules leave the same 1,925 respondents in the analysis sample. They do not leave the same cleaned file. Nine of the 114 rows pass every other step of the cleaning. The code replaces each missing value of household income by the mean over the cleaned file, and it does so before it selects the analysis sample. That mean is $387/37\\approx 10.459$ without the nine rows and $29877/2858\\approx 10.454$ with them, and 75 of the 1,925 respondents receive it. The plan does not say whether a duplicate is counted among all rows or only among those that reached randomization. On the second reading, 13 more respondents would enter the analysis sample.", "what_survives": "The sentence of Section 3.1 stands: the difference between respondents surveyed before and after the election has $p<0.05$ with either sample. Every interval of Table OA12 contains zero with either sample. The printed entries of Tables OA11 and OA12 are correct for the respondents who started the survey on November 1, 2, 8 or 9. The main effect of Study 2 remains $0.26$ with $p<0.001$ under either mean of income; I did not recompute every other coefficient or results under the alternative duplicate-counting rule. Study 1 does not pass through this code, and the other analyses of Study 2 and the pooled analyses do not use the restriction by date."}, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N011", "work": {"arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/"}, "work_id": "P034"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the end of the abstract. Under the heading Results, the text reads: We estimate that viewing a photograph of Michelle Obama's face relative to control conditions caused children to be 19 percent more likely to choose fruit over candy. Under the heading Conclusions, a red outline surrounds the first sentence: Michelle Obama's initiative to reduce childhood obesity has influenced children's dietary preferences. The remaining sentence of the Conclusions, on whether this influence extends beyond Halloween in New Haven, is not outlined.", "caption": "Abstract, Results and Conclusions, p. 2/6 (PDF p. 2). The outline marks the first sentence of the Conclusions, which attributes the estimated effect to the initiative.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E001.png"}, {"alt": "Crop of five lines from page 5. The sentence reads: In particular, this is a strongly Democratic neighborhood, and if children in Democratic households are either more aware of Michelle Obama's campaign or more favorably disposed to it, we would generate an overestimate of the treatment effect of her campaign on the general population of those in the United States. A red outline surrounds the words the treatment effect of her campaign.", "caption": "Results and conclusions, second paragraph, p. 5/6 (PDF p. 5). The outline marks the phrase that calls the estimate the treatment effect of the campaign.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E001-p5.png"}, {"alt": "Two dot plots side by side under the title: Two models give the same fruit choices on each side of the porch but different effects of the campaign. The left plot, headed Observable in the experiment, the two models agree, has six rows, one for each year and side. In each row a filled blue dot for the direct-display model sits inside an open orange ring for the campaign-mediated model at the same value: 20.3 percent and 25.3 percent for the comparison and Obama sides in 2012, 21.9 and 23.6 percent in 2014, 25.5 and 31.8 percent in 2015. The right plot, headed Not observable, the two models differ, has three rows, one for each year. The blue dot for the direct-display model is at zero in every row, and the orange ring for the campaign-mediated model is at 5.0 percentage points in 2012, 1.7 in 2014 and 6.3 in 2015.", "caption": "The counterexample for P035-E001, by year. Left: the probability of choosing fruit on each side of the porch, which is the same in the direct-display model (filled dot) and in the campaign-mediated model (open ring) and equals the proportion in the released data. Right: the mean effect of the campaign on fruit choice on the Obama side, which is zero in the direct-display model and 5.0, 1.7 and 6.3 percentage points in the campaign-mediated model.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E001-models.png"}], "formal_release": null, "formal_statements": [{"id": "P035-T1", "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-T1"}, {"id": "P035-T3", "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-T3"}], "headline": "The experiment estimates the effect of assignment to the Michelle Obama porch condition; it does not establish that her initiative to reduce childhood obesity caused the difference.", "id": "P035-E001", "location": "Abstract, Conclusions, first sentence, p. 2/6 (PDF p. 2), and the campaign-effect wording in Results and conclusions, p. 5/6 (PDF p. 5); read with the hypothesis and rationale in the Introduction, third paragraph, p. 2/6", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": ["P035-T1", "P035-T3"], "text": {"as_printed": "\\emph{Abstract, Results (p.~2/6).} We estimate that viewing a photograph of Michelle Obama's face relative to control conditions caused children to be 19\\% more likely to choose fruit over candy.\n\n\\emph{Abstract, Conclusions (p.~2/6).} \\hl{Michelle Obama's initiative to reduce childhood obesity has influenced children's dietary preferences.} Whether this influence extends beyond Halloween trick-or-treating in New Haven, CT on the porch of an economics professor requires further research.\n\n\\emph{Introduction, context (p.~2/6).} While the campaign gained publicity, an experimental evaluation of the effects of the entire campaign would not be possible due to the initiative being rolled out across the United States through mass media.\n\nHowever, we have experimentally assessed the microfoundations of Let's Move in order to evaluate whether the public health campaign altered dietary decision-making. When children are exposed to the image of a figurehead regularly associated with the public health campaign's message, the subconscious reminder may reinforce the message.\n\n\\emph{Results and conclusions (p.~5/6).} In particular, this is a strongly Democratic neighborhood, and if children in Democratic households are either more aware of Michelle Obama's campaign or more favorably disposed to it, we would generate an overestimate of \\hl{the treatment effect of her campaign} on the general population of those in the United States.", "corrected": "\\emph{Abstract, Conclusions.} \\hl{These trials estimate that assignment to the implemented Michelle Obama porch condition increased the probability of choosing raisins rather than candy relative to the pooled comparison conditions. Campaign reinforcement is a possible explanation; the experiment does not establish that the initiative or its health message caused the difference.} Whether \\hl{the estimated porch-condition effect} extends beyond Halloween trick-or-treating in New Haven, CT on the porch of an economics professor requires further research.\n\n\\emph{Results and conclusions, p.~5/6.} In particular, this is a strongly Democratic neighborhood, and if children in Democratic households are either more aware of Michelle Obama's campaign or more favorably disposed to it, we \\hl{could} generate an overestimate of \\hl{the effect of the implemented Michelle Obama porch condition} on the general population of those in the United States.", "evidence": "The evidence is an exact counterexample: two models of the experiment that agree on everything it can observe and disagree on the effect of the campaign. The mathematical notation is introduced for this check; the paper itself names the Obama and Comparison conditions without defining a mediator or campaign-effect parameter. Fix a year $y$, and let $p_{y0}$ and $p_{y1}$ be the proportions choosing fruit on the comparison side and on the Obama side in the released data. Let $L_y$ be the least common multiple of their denominators, and set $t_{y0} = L_y p_{y0}$ and $t_{y1} = L_y p_{y1}$, which are integers. Each child has a latent value $R$, uniform on $\\{1, \\dots, L_y\\}$ and independent of the assigned side $Z \\in \\{0,1\\}$, where $Z = 1$ denotes the Obama side. Let $A \\in \\{0,1\\}$ indicate whether the campaign is present, so that $A = 1$ throughout the study. This counterfactual holds the specified porch displays fixed when the campaign is absent. I write $1$ for fruit and $0$ for candy, and I define the potential outcomes in the direct-display model and in the campaign-mediated model by\n\\[\nY^{\\mathrm{D}}_y(z,a;r) = \\mathbf{1}\\{ r \\le t_{y0} + z (t_{y1} - t_{y0}) \\},\n\\qquad\nY^{\\mathrm{M}}_y(z,a;r) = \\mathbf{1}\\{ r \\le t_{y0} + a z (t_{y1} - t_{y0}) \\}.\n\\]\nSetting $a = 1$, I have $Y^{\\mathrm{D}}_y(z,1;r) = Y^{\\mathrm{M}}_y(z,1;r)$ for every $z$ and $r$. Since $A = 1$ throughout the study, the two models give each child the same outcome on either side, and they give the same distribution to every data set that the experiment can produce, at any sample size. In both models the probability of choosing fruit is $t_{y0}/L_y = p_{y0}$ on the comparison side and $t_{y1}/L_y = p_{y1}$ on the Obama side.\n\nThe two models differ in the effect of the campaign. In the direct-display model, $a$ does not enter the definition, which implies that $Y^{\\mathrm{D}}_y(z,1;r) - Y^{\\mathrm{D}}_y(z,0;r) = 0$ for every $z$ and $r$. In the campaign-mediated model, the difference $Y^{\\mathrm{M}}_y(1,1;r) - Y^{\\mathrm{M}}_y(1,0;r)$ equals $1$ if $t_{y0} < r \\le t_{y1}$ and $0$ otherwise. Taking the mean over the latent value, I obtain\n\\[\n\\mathbb{E}\\bigl[ Y^{\\mathrm{M}}_y(1,1;R) - Y^{\\mathrm{M}}_y(1,0;R) \\bigr] = \\frac{t_{y1} - t_{y0}}{L_y} = p_{y1} - p_{y0} > 0 .\n\\]\nThe right side is positive in each of the three years, as the table shows.\n\\begin{audittable}{lllll}{The two models in each year. Both models reproduce the proportions choosing fruit in the released data (second and third columns). The last two columns give the mean effect of the campaign on the Obama side in each model.}\nYear & Comparison side, $p_{y0}$ & Obama side, $p_{y1}$ & Direct-display model & Campaign-mediated model \\\\\n2012 & $14/69 \\approx 0.203$ & $24/95 \\approx 0.253$ & $0$ & $326/6555 \\approx 0.050$ \\\\\n2014 & $58/265 \\approx 0.219$ & $37/157 \\approx 0.236$ & $0$ & $699/41605 \\approx 0.017$ \\\\\n2015 & $113/444 \\approx 0.255$ & $61/192 \\approx 0.318$ & $0$ & $449/7104 \\approx 0.063$ \\\\\n\\end{audittable}\nA quantity that the experiment identifies takes the same value in any two models with the same observable distribution. The effect of the campaign is $0$ in the direct-display model and positive in the campaign-mediated model. I conclude that the experiment does not identify it. Both models assume random assignment, consistency and no interference, and in both the Michelle Obama porch condition has an effect. They can also share the stated public association between Michelle Obama and the campaign: in the direct-display model, that association does not affect the choice. Calibration to the observed proportions specifies an exact counterexample; it does not assert that the unknown population probabilities equal those sample proportions.\n\nThe construction also respects the disclosed comparison conditions. Applying it separately to each comparison protocol reproduces every integer allocation compatible with S1 Table and the pooled counts: Romney in 2012; Clinton and no photograph in 2014; and either compatible Clinton/no-photograph split in 2015. Averaging over protocols leaves the campaign effect on the Obama side equal to $p_{y1}-p_{y0}$ in the campaign-mediated model and zero in the direct-display model. The compatible splits are not recovered participant labels. Both models can include the disclosed 2014 prompting and the separately randomized 2015 prompt with no effect on choice. The paper does not require a nonzero prompt effect, so this completion is permitted. Pooling the prompt is not itself a defect, and these comparisons do not resolve the separate questions about queue assignment or interference.\n\nA recalculation from the released data, with the estimator in the replication code, reproduces the printed estimates. The correction changes none of them.\n\\begin{audittable}{lll}{The printed estimates beside the recalculation from the released data.}\nQuantity & Printed & Recalculated \\\\\nFruit choice, Obama side (95\\% interval) & 28.1\\% (23.8--32.4\\%) & 0.28066 (0.23751, 0.32381) \\\\\nFruit choice, comparison side (95\\% interval) & 23.5\\% (20.5--26.5\\%) & 0.23525 (0.20512, 0.26538) \\\\\nRelative increase & 19\\% & 0.19305 \\\\\nTwo-sided $p$ & 0.091 & 0.09078 \\\\\n\\end{audittable}", "problem": "The experiment assigned children to a side of the porch. The Obama side had a large photograph of Michelle Obama and a poster with the name ``Obama''; I refer to assignment to this side as the Michelle Obama porch condition. The comparison side had a photograph of Ann Romney (2012) or of Hillary Clinton (2014, 2015), or no photograph (2014, 2015), and posters with the name ``Romney'' or ``Clinton''. Exposure to the \\emph{Let's Move} campaign, Michelle Obama's initiative to reduce childhood obesity, was not varied, and the recorded variables include no measure of campaign exposure, recall or the proposed reminder pathway. Under the assumptions of the design (valid assignment, consistency and no interference), the contrast between the two sides estimates the effect of the Michelle Obama porch condition. The first sentence of the Conclusions attributes that effect to the campaign. The attribution requires further assumptions linking at least part of the contrast to the campaign. Recognition of the person, a political association, or another response to the name or the photograph could account for the difference without any campaign contribution. The Introduction explicitly motivates campaign reinforcement through the figurehead's association with the health message. It proposes this pathway without imposing or testing a restriction that excludes other pathways, or otherwise ensures that some of the contrast comes from the campaign.\n\nThe paper says that it cannot evaluate the entire nationwide campaign, and it limits its conclusion to Halloween in New Haven. Neither qualification supplies the missing restriction, since the attribution is not identified even for the children on this porch. The Introduction states a research aim and a proposed explanation: a subconscious reminder may reinforce the campaign's message. That passage need not be corrected as though it asserted a separately demonstrated mechanism. The categorical attribution is in the abstract's Conclusions; p.~5/6 continues it by calling the estimate ``the treatment effect of her campaign'' in its conditional discussion of extrapolation.\n\nThe correction weakens the conclusion and changes no number. To establish some campaign contribution, one could assume that noncampaign contributions are zero, or bound their combined contribution below a positive porch-condition effect. The paper supplies neither restriction. The correction does not say that the campaign had no influence, that the proposed explanation is unreasonable, or that there is no contextual evidence favoring it. These data do not settle the actual campaign contribution in either direction. I also do not count the wording of the Results sentence as an error. Since the photograph and the name poster were always assigned together, the design cannot separate their contributions. But ``viewing a photograph of Michelle Obama's face'' can reasonably be read as a label for the Michelle Obama porch condition.", "what_survives": "The yearly proportions on each side, the pooled contrast with inverse probability weights, the confidence intervals, the $p$-value and Fig.~1 are unaffected. The 19\\% relative increase stands as an estimate of the effect of the Michelle Obama porch condition, conditional on the assumptions about assignment, consistency and interference; the questions about assignment in 2014 and 2015 are separate from this correction, and neither the paper nor the released data settle them. What changes is the categorical campaign conclusion and the campaign-effect wording on p.~5/6. The Introduction's proposed reminder mechanism and stated research purpose can remain, together with the explicit description of the randomized intervention and the geographic, Halloween, repeat-participation and priming qualifications. No later mathematical result depends on the campaign attribution."}, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-E001", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/"}, "work_id": "P035"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading Introduction and the first five lines below it. The second sentence reads: Based on an analysis of the National Health and Nutrition Examination Survey, 1999 to 2014, researchers show an estimated 19 percent increase in the prevalence of Class 1 obesity (low risk obesity, BMI 30.0 to 34.9) in the United States from 1999 to 2014, with citation 4. Red outlines surround the two halves of the parenthetical, low risk obesity, BMI 30.0 to 34.9, at the end of one line and the start of the next.", "caption": "Introduction, first paragraph, p. 2/6 (PDF p. 2). The outline marks the parenthetical that describes Class 1 obesity by the BMI range 30.0 to 34.9; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E002.png"}], "formal_release": null, "formal_statements": [], "headline": "Reference [4] defines Class 1 obesity in children as BMI at or above the age- and sex-specific 95th percentile, including higher obesity classes, and not as BMI 30.0 to 34.9; the 19% increase is correct.", "id": "P035-E002", "location": "Introduction, first paragraph, second sentence, p. 2/6 (PDF p. 2)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Based on an analysis of the National Health and Nutrition Examination Survey, 1999--2014, researchers show an estimated 19\\% increase in the prevalence of Class 1 obesity (\\hl{low risk obesity, BMI 30.0 to 34.9}) in the United States from 1999 to 2014 [4].", "corrected": "Based on an analysis of the National Health and Nutrition Examination Survey, 1999--2014, researchers show an estimated 19\\% increase in the prevalence of Class 1 obesity (\\hl{BMI at or above the age- and sex-specific 95th percentile, including higher obesity classes}) in the United States from 1999 to 2014 [4].", "evidence": "The evidence is a check of the cited source, Skinner, Perrin and Skelton, \\emph{Obesity} 24(5):1116--1123 (2016), in the version of record. Its Methods define ``obesity as $\\ge$95th percentile'', and the first footnote of its Table~1 states that ``Each weight category is inclusive of the higher weight categories''. By its Table~2, the prevalence of this class was 14.6\\% in 1999--2000 and 17.4\\% in 2013--2014, which gives a relative increase of $(17.4 - 14.6)/14.6 \\approx 0.192$.", "problem": "The parenthetical describes Class 1 obesity by a fixed range of body mass index. Reference [4] concerns US children aged 2 to 19 and defines this class, which it writes as class I, as BMI at or above the 95th percentile for age and sex, with each weight category including the higher ones; fixed values of BMI appear in its definitions only as alternatives for classes II (35) and III (40).", "what_survives": "The 19\\% increase stands, and no quantity in the trial uses this prevalence or the BMI range; the 19\\% in the abstract is a different quantity, the relative increase in fruit choice."}, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-E002", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/"}, "work_id": "P035"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of nine lines of the Methods, beginning One side of the porch, denoted Obama, had a large photo of Michelle Obama in front of the children. Red outlines surround two passages, each running over a line break. The first is the parenthetical: See S1 Table for the disaggregation of comparison groups. The second is the sentence: We pool our analysis, irrespective of this additional source of variation.", "caption": "Methods, third paragraph, p. 3/6 (PDF p. 3). The outlines mark the pointer to S1 Table for the comparison groups and the sentence that pools the analysis over the 2015 priming assignment.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The paper does not say how the Clinton and no-photo conditions were allocated on the comparison side, and the released data record neither the comparison condition nor the 2015 priming assignment.", "id": "P035-N003", "location": "Methods, third paragraph, p. 3/6 (PDF p. 3); S1 Table", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The other side--denoted \\emph{Comparison}--had a photo of Ann Romney (2012), a photo of Hillary Clinton (2014, 2015), or no photo (2014, 2015). \\hl{(See S1 Table for the disaggregation of comparison groups.)} Other posters were taped to the wall that said either ``Obama'' or ``Romney''/``Clinton'' on the respective sides of the porch. In 2014, we primed the trick-or-treater by pointing to the photo and asking if they knew who that was; in 2015, trick-or-treaters were individually randomly assigned (using a pre-coded Excel spreadsheet) to receive or not receive the additional priming. \\hl{We pool our analysis, irrespective of this additional source of variation.}", "corrected": "(See S1 Table for the disaggregation of comparison groups. \\hl{The released data record each child's side of the porch but not the comparison condition or the 2015 priming assignment.})\n\nA complete statement would also say how and when the Clinton and no-photo conditions were allocated on the comparison side. The published sources do not contain this information.", "intended_meaning": "Children were allocated to two sides of the porch. In 2014 and 2015 the comparison side had two conditions, a photograph of Hillary Clinton or no photograph, and the analysis pools these conditions and the 2015 priming assignment. This reading agrees with the page, and every entry of S1 Table is compatible with the released data. The paper does not say how or when the two conditions were allocated, and the released data record neither the comparison condition nor the priming assignment. The Clinton and No photo rows of S1 Table cannot be regenerated from the released data: in 2015, fruit counts of 55 and 58 or of 56 and 57 in these two rows both match the printed means and standard errors. The pooled estimates do not use these labels."}, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-N003", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/"}, "work_id": "P035"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the last five lines of page 3. The text reads: For analysis, we use the difference-in-means estimates for each year. We had missing outcome data on one 12 year old, female participant in 2012; this participant is dropped from analysis. Our findings are substantively unchanged by imputing either a fruit or candy outcome for the participant. Red outlines surround the words this participant is dropped from analysis, at the end of one line and the start of the next.", "caption": "Methods, last paragraph, p. 3/6 (PDF p. 3). The outline marks the statement that the participant with the missing outcome is dropped from analysis; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-N004.png"}], "formal_release": null, "formal_statements": [], "headline": "The participant with the missing outcome is excluded from the outcome regressions, and her assignment still enters the estimated treatment probabilities.", "id": "P035-N004", "location": "Methods, last paragraph, p. 3/6 (PDF p. 3)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "(We had missing outcome data on one 12 year old, female participant in 2012; this participant is \\hl{dropped from analysis}. Our findings are substantively unchanged by imputing either a fruit or candy outcome for the participant.) To compute the pooled estimates, we aggregate across years using inverse probability of treatment weighting.", "corrected": "(We had missing outcome data on one 12 year old, female participant in 2012; this participant is \\hl{excluded from the outcome regressions, while treatment probabilities are estimated using all observed assignments}. Our findings are substantively unchanged by imputing either a fruit or candy outcome for the participant.)", "intended_meaning": "In this parenthesis, ``dropped from analysis'' means that the participant is omitted from the outcome means and the outcome regressions, which use 1222 rows; the replication code does this. Her assignment still enters the estimate of the treatment probabilities, which uses all 1223 rows and gives the 2012 probability $96/165$ in place of $95/164$. The weighting method, as the paper describes it, allows this. Deleting her before the weights are estimated changes the pooled difference from 0.045414 to 0.045371 and the $p$-value from 0.0908 to 0.0910, and no printed number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-N004", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/"}, "work_id": "P035"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of text. The first line reads: where beta sub 1 represents the peer assignment indicator, X. The passage continues: represents the vector of covariates, including country-level covariates and indicators for region and epsilon is a disturbance with E[epsilon | Peers, X] = 0. Note that, when covariates are not included, this estimator of beta sub 1 reduces to the well-known difference-in-means. A red outline surrounds the first beta sub 1.", "caption": "Section III, first paragraph below Table 2, p. 596 (PDF p. 3), left column. The outline marks $\\beta_1$, which the sentence calls the peer assignment indicator.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-E001.png"}, {"alt": "Crop of the heading III. Results, three lines of text and a displayed equation. The text reads: We conduct an ordinary least squares (OLS) regression of the outcomes on an indicator for assignment to peer treatment. The model we estimate is. The equation reads: Y = beta sub 0 + beta sub 1 Peers + X gamma + epsilon. A red outline surrounds the product beta sub 1 Peers.", "caption": "Section III, opening sentence and the regression model, foot of p. 595 (PDF p. 2), right column. The outline marks the product $\\beta_1 \\mathit{Peers}$: in the display, $\\beta_1$ multiplies the indicator $\\mathit{Peers}$. The display is correct as printed.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-E001-model.png"}], "formal_release": null, "formal_statements": [{"id": "P036-T1", "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-T1"}], "headline": "In the sentence that defines the terms of the regression model, the peer assignment indicator is $\\mathit{Peers}$, and $\\beta_1$ is its coefficient.", "id": "P036-E001", "location": "Section III: the regression model at the foot of p. 595 (PDF p. 2), and the sentence that defines its terms, first paragraph below Table 2, p. 596 (PDF p. 3)", "pdf_page": 2, "printed_page": "595", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P036-T1"], "text": {"as_printed": "\\emph{p.~595.} We conduct an ordinary least squares (OLS) regression of the outcomes on an indicator for assignment to peer treatment. The model we estimate is\n\\[\nY = \\beta_0 + \\beta_1 \\mathit{Peers} + \\boldsymbol{X}\\gamma + \\epsilon\n\\]\n\\emph{p.~596.} where $\\hl{\\beta_1}$ represents the peer assignment indicator, $\\boldsymbol{X}$ represents the vector of covariates, including country-level covariates and indicators for region and $\\varepsilon$ is a disturbance with $E[\\varepsilon \\mid \\mathit{Peers}, \\boldsymbol{X}] = 0$. Note that, when covariates are not included, this estimator of $\\beta_1$ reduces to the well-known difference-in-means.", "corrected": "We conduct an ordinary least squares (OLS) regression of the outcomes on an indicator for assignment to peer treatment. The model we estimate is\n\\[\nY = \\beta_0 + \\beta_1 \\mathit{Peers} + \\boldsymbol{X}\\gamma + \\epsilon\n\\]\nwhere \\hl{$\\mathit{Peers}$ is an indicator equal to one for assignment to the peer treatment and zero for assignment to the world-affairs treatment, $\\beta_1$ is its coefficient}, $\\boldsymbol{X}$ represents the vector of covariates, including country-level covariates and indicators for region and $\\varepsilon$ is a disturbance with $E[\\varepsilon \\mid \\mathit{Peers}, \\boldsymbol{X}] = 0$. Note that, when covariates are not included, this estimator of $\\beta_1$ reduces to the well-known difference-in-means.\n\nThe change of one symbol is also sufficient: ``where $\\hl{\\mathit{Peers}}$ represents the peer assignment indicator''.", "evidence": "I checked the page and recalculated the two unadjusted coefficients. On the page, the display and the later phrase ``the coefficient on the peer assignment indicator'' fix the roles of the two symbols. With these roles, the last sentence of the passage holds. In the regression of $Y$ on an intercept and $\\mathit{Peers}$, the two normal equations state that the residuals have mean zero in each treatment group. The fitted values are the two group means, and I have\n\\[\n\\hat\\beta_0 = \\bar Y_0, \\qquad \\hat\\beta_1 = \\bar Y_1 - \\bar Y_0 ,\n\\]\nwhere $\\bar Y_1$ and $\\bar Y_0$ are the mean outcomes of the missions assigned to the peer treatment and to the world affairs treatment. From the counts in Table 1, I obtain $\\hat\\beta_1 = 10/94 - 10/95 = 1/893 \\approx 0.0011$ for any response and $\\hat\\beta_1 = 10/94 - 7/95 = 146/4465 \\approx 0.0327$ for interest response. These values round to the entries $0.001$ and $0.033$ of Table 2, specifications 1 and 4.", "problem": "In the display, $\\beta_1$ multiplies $\\mathit{Peers}$. The regressor $\\mathit{Peers}$ is the indicator for assignment to the peer treatment, and $\\beta_1$ is its coefficient. The sentence below the display assigns the role of the indicator to $\\beta_1$. The paper uses the two symbols correctly elsewhere: the same paragraph speaks of ``this estimator of $\\beta_1$'', and the following paragraphs call the reported value ``the coefficient on the peer assignment indicator''. The intended meaning is clear from this context.", "what_survives": "The model, the estimates and standard errors of Table 2, and the sentence on the difference in means are unaffected. The correction changes only the wording of one definition."}, "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-E001", "work": {"arxiv_id": null, "doi": "10.1080/13504851.2016.1213359", "title": "Information and Access to United Nations Diplomatic Missions", "url": "https://pmaronow.github.io/llm-audit/works/P036/"}, "work_id": "P036"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Table 2. The title reads: Table 2. Ordinary least squares regressions: any response. A red outline surrounds the words any response at the end of the title. Below the title, the heading Any response spans columns (1), (2) and (3), and the heading Interest response spans columns (4), (5) and (6). The rows are Treatment effect and Control mean, each with standard errors in parentheses, Country characteristics and Region indicators. The first two lines of the note identify any response as the dependent variable for specifications 1, 2 and 3 and interest response for specifications 4, 5 and 6. The next sentence names heteroscedasticity-consistent standard errors and MacKinnon and White (1985), then is cut off after HC2.", "caption": "Table 2 with the first two lines of its note, p. 596 (PDF p. 3). The outline marks “any response” in the title; the heading over specifications 4, 5 and 6 is “Interest response”.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-E002.png"}], "formal_release": null, "formal_statements": [], "headline": "The title of Table 2 should name both outcomes of the table, any response and interest response.", "id": "P036-E002", "location": "Table 2, title, p. 596 (PDF p. 3)", "pdf_page": 3, "printed_page": "596", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Table 2. Ordinary least squares regressions: \\hl{any response}.", "corrected": "Table 2. Ordinary least squares regressions: any response \\hl{and interest response}.", "evidence": "I checked the page. The heading ``Any response'' spans specifications 1, 2 and 3, the heading ``Interest response'' spans specifications 4, 5 and 6, and the first two sentences of the table note assign the two dependent variables in the same way.", "problem": "The title names one of the two outcomes. Specifications 1, 2 and 3 are regressions of the any-response indicator, and specifications 4, 5 and 6 are regressions of the interest-response indicator.", "what_survives": "The column headings and the note identify the outcomes correctly and need no change. No entry of the table changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-E002", "work": {"arxiv_id": null, "doi": "10.1080/13504851.2016.1213359", "title": "Information and Access to United Nations Diplomatic Missions", "url": "https://pmaronow.github.io/llm-audit/works/P036/"}, "work_id": "P036"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the left column of page 594. The text reads: how these parties can help them achieve their goals. In the context of American politics, Kalla and Broockman (2015) ask whether donations to political campaigns secure preferential treatment from lawmakers. They find that policymakers made themselves available for meetings more often when they. A red outline surrounds the year (2015).", "caption": "Section I, “Information and access”, first paragraph, p. 594 (PDF p. 1), left column. The outline marks the year 2015 in the citation of Kalla and Broockman.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-E003.png"}, {"alt": "Crop of the reference list on page 597. After the last line of the preceding entry, the entry reads: Kalla, J. L., and D. E. Broockman. 2016. Campaign Contributions Facilitate Access to Congressional Officials: A Randomized Field Experiment. American Journal of Political Science 60 (3): 545 to 558. doi:10.1111/ajps.12180. A red outline surrounds the year 2016.", "caption": "References, p. 597 (PDF p. 4), right column: the only entry for Kalla and Broockman. The outline marks its year, 2016.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-E003-reference.png"}, {"alt": "Crop of the heading Funding and the four lines below it: Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. A red outline surrounds the word authors(s), which is printed with a second s in parentheses.", "caption": "Funding paragraph, p. 597 (PDF p. 4), left column. The outline marks the misprint “authors(s)”.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-E003-funding.png"}], "formal_release": null, "formal_statements": [], "headline": "The citation of Kalla and Broockman on p. 594 should carry the year 2016, as in the reference list, and the Funding paragraph should read “authors”.", "id": "P036-E003", "location": "Section I, “Information and access”, first paragraph, p. 594 (PDF p. 1); References and Funding paragraph, p. 597 (PDF p. 4)", "pdf_page": 1, "printed_page": "594", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~594.} In the context of American politics, Kalla and Broockman (\\hl{2015}) ask whether donations to political campaigns secure preferential treatment from lawmakers.\n\n\\emph{p.~597, References.} Kalla, J. L., and D. E. Broockman. 2016. Campaign Contributions Facilitate Access to Congressional Officials: A Randomized Field Experiment. \\emph{American Journal of Political Science} 60 (3): 545--558. doi:10.1111/ajps.12180.\n\n\\emph{p.~597, Funding.} Any opinion, findings, and conclusions or recommendations expressed in this material are those of the \\hl{authors(s)} and do not necessarily reflect the views of the National Science Foundation.", "corrected": "\\emph{p.~594.} In the context of American politics, Kalla and Broockman (\\hl{2016}) ask whether donations to political campaigns secure preferential treatment from lawmakers.\n\n\\emph{p.~597, Funding.} Any opinion, findings, and conclusions or recommendations expressed in this material are those of the \\hl{authors} and do not necessarily reflect the views of the National Science Foundation.", "evidence": "I checked the citation on p.~594, and the Funding paragraph and the complete reference list on p.~597. I did not check the date on which the cited article first appeared online. The two years disagree, but this does not show that 2015 is a false date.", "problem": "The text cites Kalla and Broockman with the year 2015, and the reference list has a single entry for these authors, dated 2016. The Funding paragraph prints ``authors(s)'' for ``authors''.", "what_survives": "The cited work is identified by its title and DOI, and nothing in the paper depends on either point."}, "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-E003", "work": {"arxiv_id": null, "doi": "10.1080/13504851.2016.1213359", "title": "Information and Access to United Nations Diplomatic Missions", "url": "https://pmaronow.github.io/llm-audit/works/P036/"}, "work_id": "P036"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of Table 2 below its title. The heading Any response spans columns (1) to (3), and the heading Interest response spans columns (4) to (6). The rows are Treatment effect and Control mean, each with standard errors in parentheses, then Country characteristics, with entries No, Yes, Yes, No, Yes, Yes, and Region indicators, with entries No, No, Yes, No, No, Yes. In the note below the table, a red outline surrounds the words: these estimators are generally conservative under a randomization model (Samii and Aronow 2012). The words before the outline read: ``Heteroscedasticity-consistent SEs are reported in parentheses, using MacKinnon and White (1985) HC2 estimators;''.", "caption": "Table 2 below its title, with the first four lines of its note, p. 596 (PDF p. 3). The outline marks the statement that the HC2 estimators are generally conservative under a randomization model, with its citation. The rows “Country characteristics” and “Region indicators” show that specifications 2, 3, 5 and 6 include covariates.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-N003.png"}], "formal_release": null, "formal_statements": [{"id": "P036-T2", "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-T2"}], "headline": "The statement in the note to Table 2 that the HC2 estimators are generally conservative is not false under its natural reading, but Samii and Aronow (2012), the paper cited for it, covers only specifications 1 and 4, which have no covariates.", "id": "P036-N003", "location": "Table 2, note, third sentence, p. 596 (PDF p. 3); the sentence applies to the standard errors of all six specifications", "pdf_page": 3, "printed_page": "596", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P036-T2"], "text": {"as_printed": "Heteroscedasticity-consistent SEs are reported in parentheses, using MacKinnon and White (1985) HC2 estimators; \\hl{these estimators are generally conservative under a randomization model} (Samii and Aronow 2012).", "corrected": "Heteroscedasticity-consistent SEs are reported in parentheses, using MacKinnon and White (1985) HC2 estimators; \\hl{for specifications 1 and 4, the variance estimator for the treatment effect is conservative in expectation under complete randomization} (Samii and Aronow 2012).\n\nA statement about specifications 2, 3, 5 and 6 would need a separate result, cited with its conditions.", "intended_meaning": "I read the word ``generally'' as a hedge: the note does not claim an exact bound at a fixed number of units for every regression, or for every assignment. For specifications 1 and 4, which have no covariates, the HC2 variance estimator for the treatment effect equals the Neyman variance estimator and is conservative in expectation under complete randomization. This is Theorem 1 and Corollary 1 of Samii and Aronow (2012). The cited paper considers the regression of the outcome on an intercept and a treatment indicator alone, and it has no result for a regression with covariates. For specifications 2, 3, 5 and 6, in which covariates enter additively, a separate derivation shows that the estimator is conservative to first order as the number of units grows, when the number of covariates is fixed and regularity conditions hold. This derivation gives no bound at a fixed number of units, and it is not established that its conditions hold in the present application. Read in this way, the statement is not false, and the citation supports it for specifications 1 and 4 only. No estimate, standard error or conclusion of the paper depends on the citation."}, "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-N003", "work": {"arxiv_id": null, "doi": "10.1080/13504851.2016.1213359", "title": "Information and Access to United Nations Diplomatic Missions", "url": "https://pmaronow.github.io/llm-audit/works/P036/"}, "work_id": "P036"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text: response rate, as compared to the world affairs treatment. As this is a simple difference-in-means, this is also reflected in the ‘Per cent’ column under the ‘Any response’ treatment in Table 1. Given the relatively large SE compared. A red outline surrounds the word treatment that follows ‘Any response’.", "caption": "Section III, third paragraph on p. 596 (PDF p. 3), left column. The outline marks “treatment” in the cross-reference to the “Any response” columns of Table 1.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-N005.png"}, {"alt": "Crop of four lines of text: also reflected in the ‘Per cent’ column under the ‘Interest response’ treatment in Table 1. This effect size is not different from zero at any conventional confidence level, and the effect size does not move. A red outline surrounds the word treatment that follows ‘Interest response’.", "caption": "Section III, top of the right column, p. 596 (PDF p. 3). The outline marks “treatment” in the cross-reference to the “Interest response” columns of Table 1.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-N005-interest.png"}], "formal_release": null, "formal_statements": [], "headline": "In two cross-references to Table 1, “treatment” means the outcome heading: “Any response” and “Interest response” are the two outcomes, and the treatments are the rows Peers and World affairs.", "id": "P036-N005", "location": "Section III, p. 596 (PDF p. 3): third paragraph on the page, left column, and the fourth paragraph where it continues at the top of the right column", "pdf_page": 3, "printed_page": "596", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "As this is a simple difference-in-means, this is also reflected in the ‘Per cent’ column under the ‘Any response’ \\hl{treatment} in Table 1. [\\ldots] This is also reflected in the ‘Per cent’ column under the ‘Interest response’ \\hl{treatment} in Table 1.", "corrected": "As this is a simple difference-in-means, this is also reflected in the ‘Per cent’ column under the ‘Any response’ \\hl{heading} in Table 1. [\\ldots] This is also reflected in the ‘Per cent’ column under the ‘Interest response’ \\hl{heading} in Table 1.", "intended_meaning": "The two phrases point to the ‘Per cent’ columns under the headings ‘Any response’ and ‘Interest response’ of Table 1. In that table the treatment groups are the rows, Peers and World affairs, and the two headings are the outcomes that Section II defines. The intended columns are clear, and what the two sentences say about them is correct: the differences between the ‘Per cent’ entries are the 0.1 and 3.3 percentage points of specifications 1 and 4."}, "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-N005", "work": {"arxiv_id": null, "doi": "10.1080/13504851.2016.1213359", "title": "Information and Access to United Nations Diplomatic Missions", "url": "https://pmaronow.github.io/llm-audit/works/P036/"}, "work_id": "P036"}
{"before_review": {"class": "not_confirmed", "severity": null}, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the Conclusion heading and its first three lines. A red outline surrounds nearly at the end of the first line. The next line reads 10 percent of subjects in both of our treatment arms, and; the following line begins the description of interest.", "caption": "Conclusion, p. 596 (PDF p. 3), right column. The outline marks “nearly” in the description of the response rates.", "url": "https://pmaronow.github.io/llm-audit/works/P036/exhibits/P036-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "Both response rates exceed 10%, but the text says “nearly” 10%", "id": "P036-N009", "location": "Conclusion, printed p. 596, first clause", "pdf_page": 3, "printed_page": "596", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "responses from \\hl{nearly} 10\\% of subjects in both of our treatment arms", "corrected": "responses from \\hl{about} 10\\% of subjects in both of our treatment arms", "evidence": "Table~1, printed p.~595, gives any-response proportions $10/94$ and $10/95$. Then $10/94 \\approx 10.64\\%$ and $10/95 \\approx 10.53\\%$. Both rates exceed 10\\%.", "problem": "On its natural reading, ``nearly 10\\%'' means close to but below 10\\%. Both arms have response rates above 10\\%. A direction-neutral reading of ``nearly'' exists, but on the natural reading the word is wrong here.", "what_survives": "This wording repair requires no change to the reported counts, percentages, treatment comparison or conclusion. Each rate is still roughly one in ten."}, "url": "https://pmaronow.github.io/llm-audit/works/P036/#P036-N009", "work": {"arxiv_id": null, "doi": "10.1080/13504851.2016.1213359", "title": "Information and Access to United Nations Diplomatic Missions", "url": "https://pmaronow.github.io/llm-audit/works/P036/"}, "work_id": "P036"}
{"before_review": {"class": "error", "severity": "Major"}, "class": "not_confirmed", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of nine lines of text. Red boxes mark the last three lines: 'any differences between the proportion of treated seeds who were referents and the proportion of seed-eligibles who were referents are attributable to randomization. As a result, our regression'.", "caption": "Methods and Materials, p. 568 (PDF p. 3), foot of the left column. The marked lines attribute the differences in composition to randomization and begin the conclusion with “As a result”. The sentence continues at the head of the right column.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001.png"}, {"alt": "Crop of seven lines of text. Red boxes mark the first two lines: 'strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group on school-wide conflict and other outcomes.'", "caption": "Methods and Materials, p. 568 (PDF p. 3), head of the right column. The marked lines complete the sentence, which claims an estimate of the causal effect of the composition of the seed group.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001-b.png"}, {"alt": "Crop of equation (2) and the six lines after it. The equation sets the outcome of school i in block j equal to a block effect, plus tau-zero times the treatment D, plus beta times the composition C of the seed group times D, plus gamma times the composition C-superscript-E of the seed-eligible group, plus an error term. Red boxes mark the end of the passage: 'As the expectation of C is proportional to C-superscript-E, controlling for C-superscript-E ensures that any random deviations in C from its expectation may be'.", "caption": "SI Appendix, p. 13, equation (2) and its interpretation. The passage expressly qualifies the interaction model’s causal interpretation by validity of the linear approximation, then gives an abbreviated randomization argument involving the eligible-pool control. The marked sentence ends on p. 14 with “interpreted causally”.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001-c.png"}, {"alt": "Plot of the outcome of a school against the composition C of its seed group. A dashed grey line at 0.3 stands for every control school. A blue horizontal line at 0.2 carries five circles at C = 0, 0.05, 0.1, 0.15 and 0.2, the largest at 0.1: treated schools with q = 1/10. An orange horizontal line at 0 carries squares at C from 0 to 0.6, the largest at 0.3: treated schools with q = 3/10. Black triangles mark the mean composition, q. Text gives the coefficients from the expected normal equations: minus 16/31 on treatment times C in the printed regression and 0 on treatment times (C minus q) put in its place; the effect of C in every school is 0.", "caption": "The constructed example in P037-E001; it contains no data of the study. Treated schools whose seed-eligible group has composition $q=1/10$ have outcome $1/5$ at every composition $C$ of the seed group, treated schools with $q=3/10$ have outcome $0$, and every control school has outcome $3/10$. The area of a marker is the probability of that value of $C$. The composition of the seed group has no effect on any school, but treatment effects differ with $q$, violating the sufficient common-response restriction. The expected normal equations give $-16/31$ on $DC$, and $0$ when $D(C-q)$ replaces $DC$. This illustrates insufficiency of the design alone, not a refutation of the qualified causal-response model.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E001-counterexample.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T1", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T1"}, {"id": "P037-T2", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T2"}, {"id": "P037-T14", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T14"}], "headline": "The composition coefficient is identified under the supplement’s qualified causal-response model; randomization and the eligible-pool control alone do not identify it.", "id": "P037-E001", "location": "Methods and Materials, p. 568 (PDF p. 3), paragraph beginning “The randomization of schools and seeds”, last two sentences; SI Appendix, p. 13, equation (2) and the sentences after it, repeated for equations (4) and (6) on pp. 15–16 and in the note to Table S7 (p. 42)", "pdf_page": 3, "printed_page": "568", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P037-T1", "P037-T14", "P037-T2"], "text": {"reason": "I did not establish an identification error under the supplement's qualified causal-response model. The counterexample establishes that randomization and an eligible-pool main effect alone are insufficient. It does not establish failure within the qualified model or show that the identifying restriction fails in the actual schools. The exact conditioning and residual restriction remain insufficiently explicit.\n\n\\emph{The printed qualification.} The main-text randomization argument is read together with the supplement's interaction model and the validity qualification attached to it:\n\nMain text, p.~568: Although there was heterogeneity across schools in the proportion of seed-eligible students who were social referents, any differences between the proportion of treated seeds who were referents and the proportion of seed-eligibles who were referents are attributable to randomization. As a result, our regression strategy allowed us to estimate the causal effect of the proportion of social referents in the seed group on school-wide conflict and other outcomes.\n\nSI p.~13:\n\\[\nY_{ij}=\\alpha_j+\\tau_0D_{ij}+\\beta C_{ij}D_{ij}+\\gamma C^E_{ij}+u_{ij}. \\qquad (2)\n\\]\nHere, assuming that the linear approximation for causal effect heterogeneity is valid, $\\tau_0$ is the average causal effect of treatment when the composition of the seed group is 0\\% referents and $\\beta$ represents the slope of the average causal effect with respect to group composition in percentage points (i.e., assuming that the slope is constant, the average causal effect of treatment at a given composition $c\\%$ is $\\tau_0+\\beta c$.) As $\\mathrm{E}\\,[C_{ij}]\\propto C^E_{ij}$, controlling for $C^E_{ij}$ ensures that any random deviations in $C_{ij}$ from $\\mathrm{E}\\,[C_{ij}]$ may be interpreted causally.\n\n\n\n\n\nThe qualification is repeated for equations (4) and (6) and in the note to Table S7. Read with the displayed equation, it more naturally proposes a conditional average causal response $\\tau_0+\\beta c$, with eligible composition in a baseline shared by treatment and control and an appropriate residual mean or orthogonality condition. This need not mean identical realized treatment effects in every school. The repeated descriptions of average effects also support a weaker marginal-curve reading, and the conditioning set and residual condition are not explicit. On balance I prefer the qualified conditional-model reading. My comment on this entry states that the linear model was meant to express a common baseline treatment effect while preserving efficiency with few schools. The counterexample below satisfies marginal average linearity but violates the stronger identifying response restriction. It shows what does not follow from randomization alone, rather than an error within that model. Neither the example nor the observed specification differences establishes whether the restriction holds in these schools.\n\n\\emph{What the design supplies.} Let $D$ indicate that a school is treated, let $C$ be the proportion of seeds who are social referents, and let $q=C^E$ be that proportion in the seed-eligible group. Let $\\mu$ be the mean of $C$ under the stratified seed draw, given the eligible group. The supplement writes $\\mathrm{E}[C_{ij}]\\propto C^E_{ij}$, and $\\mu=q$ when all strata are sampled at the same rate. The interaction decomposes as\n\\[\nDC=D\\mu+D(C-\\mu).\n\\]\nThe second component is a mean-zero seed innovation, conditional on school assignment. The first is randomized through school assignment and carries treatment variation associated with fixed eligible pools. A main effect for $q$ does not remove $D\\mu$. Additional moderation of the treatment intercept can therefore enter the printed composition coefficient. Under a common treatment contrast, however, both components have coefficient $\\beta$; the decomposition alone establishes no bias under that sufficient model.\n\nThe authors' code fits the displayed terms. Its syntax does not establish the causal meaning or empirical adequacy of the response approximation.\n\nIn \\texttt{1. Sanitize and generate perms, probs.R}, lines 141--146 compute $C$ and $q$ separately.\n\nIn \\texttt{4. School results-with-correction.R}, lines 203--209 regress on $DC$ with $q$ as a main effect, without $Dq$ or centering $C$.\n\n\n\nThe mathematical evidence is an exact design-alone counterexample in rational arithmetic and conditional identification arguments. I also examined the authors' scripts and ran them on the public release of the data (ICPSR 37070, version 2). In my calculations, compositions are measured as fractions; a slope per percentage point is one hundredth of the slope reported.\n\n\\emph{The example.} A block has four schools whose seed-eligible groups have compositions $q=(1/10,1/10,3/10,3/10)$. Each school has 40 seed-eligible students in four strata of ten. Each stratum contains one social referent in the first two schools and three in the last two. Two of the four schools are treated, each pair with probability $1/6$. In a treated school five students are drawn from each stratum, which gives 20 seeds. The potential outcomes of school $i$ are\n\\[\nY_i(0)=\\frac{3}{10},\\qquad Y_i(1,c)=\\frac{3}{10}-q_i \\quad\\text{for every composition } c \\text{ of the seed group.}\n\\]\nThe composition of the seed group has no effect on any school. The average effect of treatment is $-1/5$ at every $c$ feasible in all four schools, and it is linear in $c$ with slope $0$. The design-mean relation holds: $\\mathrm{E}[C_i\\mid D_i=1]=q_i$, and $\\operatorname{Var}(C_i\\mid D_i=1)=q_i(1-q_i)/36$. The marginal average causal curve is linear, but the sufficient conditional-response restriction does not hold: at the common feasible composition $c=0$, the treatment effects are $-1/10$ and $-3/10$ according to eligible-pool type. The exact check enumerates all 454 collapsed states of school assignment and seed composition.\n\nConsider the regression of $Y$ on block effects, $D$, $q$ and $DC$. After the block means are removed, the expected cross-products of the columns $(D,q,DC)$ per block, and their expected cross-products with $Y$, are\n\\[\nG=\\begin{pmatrix}1&0&1/5\\\\ 0&1/25&1/50\\\\ 1/5&1/50&151/2400\\end{pmatrix},\\qquad\nh=\\begin{pmatrix}-1/5\\\\ -1/50\\\\ -17/300\\end{pmatrix}.\n\\]\nSolving $G\\,(\\tau_0,\\gamma,\\beta)^{\\top}=h$, I obtain\n\\[\n(\\tau_0,\\gamma,\\beta)=\\Bigl(-\\frac{3}{31},\\,-\\frac{15}{62},\\,-\\frac{16}{31}\\Bigr).\n\\]\nThe value $-16/31$ is also the limit in probability of the least-squares coefficient on $DC$ when independent copies of the block are pooled. It is not claimed to be the expectation of that coefficient in a finite sample. With $\\tau_0=-1/5$ and treatment-intercept deviations $\\eta_i=1/5-q_i$, the projection identity below has numerator $-1/150$ and denominator $31/2400$. Their ratio is $-16/31$. The nonzero coefficient is the contribution of baseline treatment moderation, which the qualified identifying model excludes or restricts. It cannot refute identification under a common treatment contrast or the weaker zero-numerator condition.\n\n\\begin{audittable}{llrr}{Coefficient on the composition term in the example, from the expected normal equations with block effects. In the second and third outcomes the composition of the seed group has slope $1/5$.}\nOutcome & Regressors & Coefficient & True slope \\\\\n\\hline\n$3/10-Dq$ & $D$, $q$, $DC$ (as printed) & $-16/31$ & $0$ \\\\\n$3/10-Dq$ & $D$, $q$, $DC$, $Dq$ & $0$ & $0$ \\\\\n$3/10-Dq$ & $D$, $q$, $D(C-q)$ & $0$ & $0$ \\\\\n$3/10-Dq+DC/5$ & $D$, $q$, $DC$ (as printed) & $-49/155$ & $1/5$ \\\\\n$3/10-Dq+DC/5$ & $D$, $q$, $D(C-q)$ & $1/5$ & $1/5$ \\\\\n$3/10-D/5-q/2+DC/5$ & $D$, $q$, $DC$ (as printed) & $1/5$ & $1/5$ \\\\\n\\end{audittable}\n\nThe last row conforms to the displayed model, and the printed regression recovers its slope. An exactly linear untreated mean is unnecessary for the following expected-equations result: arbitrary fixed untreated school outcomes are allowed.\n\n\\emph{A sufficient identifying moment.} Fix the school population, eligible pools, analysis membership and pretreatment controls. Let $M$ remove the mean within each included block, and write $d=MD$, $z=M(DC)$ and $V=M[q,X]$ for a full-rank basis of the fixed nuisance controls. Suppose school-treatment probabilities are equal, $M\\mu$ lies in the span of $V$, and residual variation is positive. Let\n\\[\nR=z-\\rho d-Vg\n\\]\nbe the residual from projecting $z$ on $d$ and $V$, with coefficients chosen from design-expected cross-products. These are not assignment-specific fitted projection coefficients. The expected normal equations give $\\mathbb{E}[R]=0$, $\\mathbb{E}[R^{\\top}d]=0$, $\\mathbb{E}[R^{\\top}V]=0$ and $\\mathbb{E}[R^{\\top}z]=\\mathbb{E}[R^{\\top}R]>0$.\n\nWithin the common-slope class, write\n\\[\nY_i(0)=a_i,\\qquad Y_i(1,c)-Y_i(0)=\\tau_0+\\eta_i+\\beta c,\n\\]\nwhere $a_i$ and $\\eta_i$ are arbitrary fixed numbers. Since $\\mathbb{E}[R]=0$, arbitrary untreated levels contribute nothing to the expected residual cross-product. Eliminating the other regressors from the expected normal equations gives\n\\[\n\\beta^*-\\beta=\n\\frac{\\mathbb{E}[\\sum_i R_iD_i\\eta_i]}{\\mathbb{E}[\\sum_i R_i^2]}.\n\\]\nZero numerator identifies $\\beta$ under these conditions; a common school treatment intercept is a stronger sufficient case. School treatment effects need not be constant for the numerator to vanish. This is identification by expected equations and consistency under independent repetition of bounded blocks with limiting full-rank moments, not finite-sample design unbiasedness. Fixed membership and controls are assumptions of this argument, not conclusions about the actual analysis sample.\n\nA separate sufficient route is a full conditional causal-mean model with the corresponding residual mean condition. If the conditional expectation of $Y$ given the design matrix is that matrix times its coefficient vector, the expected normal equations identify those coefficients; with realized full rank, least squares is conditionally unbiased under that model. This route does not require the controls to span $\\mu$. It concerns a conditional average response, allowing residual heterogeneity, and is distinct from imposing a common realized treatment intercept in every school.\n\n\\emph{Optional alternatives.} To allow arbitrary fixed treatment intercepts, suppose that $Y_i(0)=a_i$ and $Y_i(1,c)=a_i+t_i+bc$, with common within-school slope $b$. Keep analysis membership fixed and nuisance regressors independent of the seed draw. Put $U_i=D_i(C_i-\\mu_i)$. Then\n\\[\nY_i=a_i+D_i(t_i+b\\mu_i)+bU_i.\n\\]\nConditional on school assignment, $U_i$ has mean zero. After fixed block demeaning it is orthogonal in expectation to the other terms and to seed-independent nuisance regressors, including block effects, $D$, $q$, $D\\mu$ and fixed covariates. Its expected normal equation identifies $b$ when residual innovation variation is positive. Thus replacing $DC$ by $D(C-\\mu)$, or retaining $DC$ and adding $D\\mu$, provides an alternative expected-equations identification strategy under these assumptions. Independent bounded-block repetition gives consistency, not finite-sample design unbiasedness. These alternatives are optional when the original causal-response model holds. They need not agree in finite samples when fitted with different nuisance spaces.\n\nIn the authors' design the seeds are drawn within strata, with a fixed number from each. If school $i$ has $N_{ih}$ seed-eligible students, $K_{ih}$ social referents and $n_{ih}$ seeds in stratum $h$, and $M_i=\\sum_h n_{ih}$, then\n\\[\n\\mu_i=\\frac{1}{M_i}\\sum_h n_{ih}\\frac{K_{ih}}{N_{ih}},\n\\]\n\\[\n\\operatorname{Var}(C_i\\mid D_i=1)=\\frac{1}{M_i^{2}}\\sum_h n_{ih}\\frac{K_{ih}}{N_{ih}}\\Bigl(1-\\frac{K_{ih}}{N_{ih}}\\Bigr)\\frac{N_{ih}-n_{ih}}{N_{ih}-1}.\n\\]\nA stratum with one student contributes nothing to the variance. The eligible-pool composition $q$ can replace $\\mu$ in these alternatives only when the two agree; equal sampling fractions suffice. With heterogeneous composition slopes, the estimand and its weights must be stated. These common-slope arguments do not establish a plain average of school-specific slopes. On the public data $\\mu_i=q_i$ in 55 of the 56 schools. In one control school the two differ by $0.0018$, because its strata are sampled at different rates.\n\n\\emph{Specification sensitivity on the public data.} I ran the authors' scripts for the school-level regressions on the public data, with an adapter that decodes and renames its variables as the scripts expect, and I varied only the composition specification. The covariates and the rule for including schools are the authors'. The regression for disciplinary reports has 49 schools.\n\n\\begin{audittable}{lrrr}{Sensitivity of the composition coefficient for disciplinary reports of peer conflict, on the public data (ICPSR 37070, version 2), 49 schools. The standard error is the robust (HC0) standard error of the authors' script, and the $P$ value is two-sided with a normal reference, as in the article. Table S7 prints $-0.85$ and $0.46$ for the first row.}\nRegressors & Coefficient & Standard error & $P$ \\\\\n\\hline\n$D$, $DC$, $q$ (as printed) & $-0.841$ & $0.463$ & $0.069$ \\\\\n$D$, $DC$, $Dq$, $q$ & $-0.479$ & $0.440$ & $0.277$ \\\\\n$D$, $D(C-\\mu)$, $q$ & $-0.393$ & $0.476$ & $0.409$ \\\\\n\\end{audittable}\n\nWith $D\\mu$ in place of $Dq$ the second row is unchanged, since $\\mu=q$ in every treated school. On the public data, the second row is $-0.0048$ reports per student per percentage point. As an illustration on the public data, using the authors' construction of the curve of Fig.~2D gives $0.095$ reports per student at a composition of 20\\% from the first row, a reduction of 57.3\\% from the fitted control mean, and $0.125$ and 43.7\\% from the second row with $D\\mu$. The article prints $0.08$ and 60\\% (see \\fid{P037-N010}).\n\nThese values have three limits. First, they are values on the public data. The authors' own input files are not public, and ICPSR states that some variables of the release were changed. On the public data five of the eight coefficients of Table S7 and seven of its eight robust standard errors agree with the printed values to two decimals; the coefficient $-0.841$ rounds to $-0.84$, not to the printed $-0.85$. Second, the three rows differ in what they assume. The differences between them show that the estimate depends on the specification. They do not show which specification is correct, and they do not show that the printed estimate is biased. If the slope differs between schools, the second and third rows generally target a weighted average of the school slopes, and not their plain average. Third, these $P$ values use the stated normal reference and are reported as part of that sensitivity calculation. These sensitivity calculations do not include the bootstrap. The comparison does not establish an incorrect published $P$ value or select a uniquely correct replacement coefficient; test calibration is considered separately in \\fid{P037-N006}.\n\n\\emph{What follows.} The composition-specific causal interpretations of equations (2), (4) and (6), Table S7, Fig.~2, and the projection of $0.08$ disciplinary events per student and a $60\\%$ reduction at $20\\%$ social referents remain conditional on the response approximation or another justified identification strategy. The example and specification differences do not establish failure within that model or empirical misspecification. The same condition reaches the Abstract's ``stronger'' statement and the Discussion's recommendation insofar as they rely on the school-level composition result; separate peer-exposure evidence also bears on the role of social referents.\n\nThe between-school randomization and separate overall-treatment regressions (script 4, lines 158 and 162) do not use this composition-identification step. The released script also uses predictions from the interaction regression for adjusted overall summaries (lines 253--279 and 286). Even if the composition interpretation failed, it would not follow that an averaged overall treatment contrast was wrong. The peer-exposure analyses use a different identification step and are not invalidated here.\n\nA clarification could make the existing qualification explicit: under the valid causal-response approximation described in the SI Appendix, the interaction coefficient estimates the response to seed composition. Randomization and the eligible-pool control alone do not guarantee that approximation; residual treatment heterogeneity must satisfy the corresponding identifying mean or moment condition. On the preferred reading this states the scope of an existing substantive assumption. Under the competing marginal-only reading it would be an additional restriction. These results require no change to the fitted specification, published numbers or policy conclusion. I did not establish an identification error under the qualified model, and its empirical adequacy remains unresolved."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E001", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of text followed by a displayed definition by cases. In the text, a red box marks the symbol z sub i in the sentence The crossproduct of z sub i and theta sub i, z prime theta sub i, is equal to the number of student i's peers that are assigned to be seeds. The display defines f(z, theta sub i) by five cases: d C (Control exposure), d S (School exposure), d NI (Non social referent seed exposure), d I (Social referent seed exposure) and d D (Direct treatment condition). A red box marks the condition of the third case, I((z circle gamma) prime theta sub i = 0) s sub i = 1.", "caption": "SI Appendix, p. 17 (PDF p. 17): the definitions that precede the exposure mapping, and the mapping. The marked condition for $d_{NI}$ has neither the factor $(1-z_i)$ nor a requirement that some peer is a seed. The marked $z_i$ in the text above stands where the vector $\\mathbf{z}$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T7", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T7"}, {"id": "P037-T8", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T8"}], "headline": "The condition printed for the nonreferent exposure $d_{NI}$ lacks the factors $(1-z_i)$ and $\\mathbf{I}(\\mathbf{z}'\\theta_i>0)$, so the five cases overlap; with both factors the cases partition the assignments, and the released code computes the corrected condition.", "id": "P037-E002", "location": "SI Appendix, p. 17 (PDF p. 17 of the SI Appendix), display after “The exposure mapping is:”, third case ($d_{NI}$), and the sentence beginning “The crossproduct of” above it; compare the main text, p. 568 (PDF p. 3)", "pdf_page": 17, "printed_page": "17", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T7", "P037-T8"], "text": {"as_printed": "``The crossproduct of \\hl{$z_i$} and $\\theta_i$, $\\mathbf{z}'\\theta_i$, is equal to the number of student $i$'s peers that are assigned to be seeds.''\n\nThe exposure mapping is:\n\\[\nf(\\mathbf{z},\\theta_i)=\\begin{cases}\nd_C\\ (\\text{Control exposure}): & (1-s_i)=1\\\\\nd_S\\ (\\text{School exposure}): & (1-z_i)\\mathbf{I}(\\mathbf{z}'\\theta_i=0)s_i=1\\\\\nd_{NI}\\ (\\text{Non social referent seed exposure}): & \\hl{\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i=0)s_i=1}\\\\\nd_I\\ (\\text{Social referent seed exposure}): & (1-z_i)\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i>0)s_i=1\\\\\nd_D\\ (\\text{Direct treatment condition}): & z_i=1\n\\end{cases}\n\\]", "corrected": "``The crossproduct of \\hl{$\\mathbf{z}$} and $\\theta_i$, $\\mathbf{z}'\\theta_i$, is equal to the number of student $i$'s peers that are assigned to be seeds.''\n\nLet $s_i$ indicate that the school of student $i$ is treated, let $z_i$ indicate that student $i$ is assigned to be a seed, let $\\theta_i$ be the row of student $i$ in the network adjacency matrix, and let $\\gamma$ be the vector of social referent indicators, as on p.~17. The exposure mapping is\n\\[\nf(\\mathbf{z},\\theta_i)=\\begin{cases}\nd_C\\ (\\text{Control exposure}): & (1-s_i)=1\\\\\nd_S\\ (\\text{School exposure}): & (1-z_i)\\mathbf{I}(\\mathbf{z}'\\theta_i=0)s_i=1\\\\\nd_{NI}\\ (\\text{Non social referent seed exposure}): & \\hl{(1-z_i)\\mathbf{I}(\\mathbf{z}'\\theta_i>0)}\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i=0)s_i=1\\\\\nd_I\\ (\\text{Social referent seed exposure}): & (1-z_i)\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i>0)s_i=1\\\\\nd_D\\ (\\text{Direct treatment condition}): & z_i=1\n\\end{cases}\n\\]\nFor every assignment that the design can generate, exactly one of the five conditions holds.", "evidence": "I base this finding on two exact counterexamples, a derivation, a reading of the released code and a recalculation on the public data. Let $t_i=\\mathbf{z}'\\theta_i$ denote the number of seed peers of student $i$, and let $r_i=(\\mathbf{z}\\circ\\gamma)'\\theta_i$ denote the number of seed peers who are social referents. For $(s_i,z_i,t_i,r_i)=(1,0,0,0)$ the printed conditions for $d_S$ and $d_{NI}$ both hold, and for $(1,1,0,0)$ the printed conditions for $d_{NI}$ and $d_D$ both hold. Both configurations can occur under the design, since a treated school can have all of its other seeds outside the peers of student $i$.\n\nWith the corrected condition, exactly one case holds for every assignment that the design can generate. A control school has no seeds. Then $s_i=0$ and $z_i=0$, and only $d_C$ holds. In a treated school, only $d_D$ holds when $z_i=1$. If $z_i=0$, then only $d_S$ holds when $t_i=0$, and when $t_i>0$ the case is $d_{NI}$ if $r_i=0$ and $d_I$ if $r_i>0$. Since $\\gamma$ and $\\theta_i$ have entries in $\\{0,1\\}$, I have $0\\le r_i\\le t_i$, and the cases are exhaustive and disjoint. Enumerating the 133 configurations $(s_i,z_i,t_i,r_i)$ with at most ten peers, I find exactly one corrected case in each.\n\nThe released code computes the corrected condition, with $t_i$ and $r_i$ counted in a within-school graph that includes nominations in either direction (see \\fid{P037-N032}). Let $b_i=\\mathbf{I}(t_i>0)$ and $q_i=\\mathbf{I}(r_i>0)$. The exposure script sets the indicator of exposure to some seed to $(1-z_i)b_i$ and the indicator of exposure to a social referent seed to $(1-z_i)q_i$ (lines 196 and 199 for the simulated assignments, lines 217 and 222 for the observed one), and the network script takes their difference as the nonreferent exposure (lines 203 and 233). Since $q_i\\le b_i$, I have $b_i-q_i=\\mathbf{I}(t_i>0)\\mathbf{I}(r_i=0)$, and the difference is\n\\[\n(1-z_i)(b_i-q_i)=(1-z_i)\\,\\mathbf{I}(\\mathbf{z}'\\theta_i>0)\\,\\mathbf{I}((\\mathbf{z}\\circ\\gamma)'\\theta_i=0),\n\\]\nwhich is zero in a control school, where no student is a seed. The code estimates the probability of nonreferent exposure by subtracting the two simulated frequencies (network script, line 188), giving the frequency of the corrected condition. Using the authors' scripts with an adapter for the public data release (ICPSR 37070 V2), I find that the code's indicator equals the corrected condition for all 24,191 retained students under the observed assignment and for all 241,910,000 pairs of a student and an assignment in 10,000 simulated assignments, for both definitions of a social referent (indegree and betweenness), and the two probability estimates differ by less than $10^{-15}$. On the public data the printed condition, read literally, would add 5,754 students with school exposure and 665 seeds to the nonreferent category under the observed assignment (indegree definition).", "problem": "As printed, the condition for $d_{NI}$ says only that the school is treated and that no social referent peer of student $i$ is a seed. It does not exclude a student who is a seed, and it does not require that at least one peer is a seed. A student in a treated school who is not a seed and has no seed peers satisfies the conditions for both $d_S$ and $d_{NI}$, and a seed none of whose seed peers is a social referent satisfies the conditions for both $d_{NI}$ and $d_D$. The five cases do not define a function. The main text (p.~568) states the category in words: students in treated schools ``for whom at least one peer is a seed but no seed is a social referent''. In the sentence above the display, the product $\\mathbf{z}'\\theta_i$ is called the crossproduct of $z_i$ and $\\theta_i$, where the vector $\\mathbf{z}$ is meant.", "what_survives": "The main text states the intended categories in words. The released code has both missing factors in the observed exposure, in the simulated exposures and in the estimated probability of nonreferent exposure, and correcting the display leaves these quantities unchanged. No reported estimate is corrected by this finding. I have not independently reproduced every published estimate, and cannot establish equality with the authors' original data: their own data file is not public, and ICPSR reports changes to some released variables."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E002", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Nine lines of text from page 18 of the SI Appendix. The first line holds the definition pi sub i of d sub k equals the sum over z in Omega of I(f(z, theta sub i) = d sub k) times Pr(Z = z). The fifth line holds the formula pi-hat sub i of d sub k equals I[ one over J times the sum from l = 1 to J of f(z superscript J, theta sub i) = d sub k ]. One red box marks the opening I[ of the indicator, and a second red box marks z superscript J inside the sum.", "caption": "SI Appendix, p. 18 (PDF p. 18), second paragraph: the exact definition of $\\pi_i(d_k)$ in the first line, and the simulated version in the fifth line. The first mark is on the indicator, which stands outside the average; the second is on $\\mathbf{z}^J$, the summand’s assignment, which does not depend on the index $\\ell$.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T9", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T9"}, {"id": "P037-T30", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T30"}], "headline": "In the formula for the simulated exposure probability, the indicator belongs inside the average and the summand should use the $\\ell$th simulated assignment, not the $J$th; the released code averages the draw-specific indicators.", "id": "P037-E003", "location": "SI Appendix, p. 18 (PDF p. 18 of the SI Appendix), second paragraph, formula after “according to the experimental design:”; the same formula on p. 14 (PDF p. 14)", "pdf_page": 18, "printed_page": "18", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T30", "P037-T9"], "text": {"as_printed": "``We can achieve an arbitrarily precise estimate of $\\pi_i$ via simulated random assignments of $\\mathbf{Z}$ according to the experimental design:\n\\[\n\\hat\\pi_i(d_k)=\\hl{\\mathbf{I}[}\\frac{1}{J}\\sum_{\\ell=1}^{J}f(\\hl{\\mathbf{z}^{J}},\\theta_i)=d_k],\n\\]\nwhere $J$ is the number of simulated random assignments, and \\hl{$z^J$} denotes the \\hl{$J$th} simulated random assignment vector.''", "corrected": "On pp.~14 and 18:\n\\[\n\\hat\\pi_i(d_k)=\\frac{1}{J}\\sum_{\\ell=1}^{J}\\hl{\\mathbf{I}[}f(\\hl{\\mathbf{z}^{\\ell}},\\theta_i)=d_k],\n\\]\nwhere $J$ is the number of simulated random assignments, \\hl{$\\mathbf{z}^{\\ell}$} denotes the \\hl{$\\ell$th} simulated random assignment vector, the assignments are drawn independently according to the experimental design, and $f$ is the exposure mapping as corrected in \\fid{P037-E002}. Then $\\hat\\pi_i(d_k)$ is unbiased for $\\pi_i(d_k)$, with variance $\\pi_i(d_k)(1-\\pi_i(d_k))/J$.", "evidence": "The evidence is a check of pp.~14 and 18, a numerical counterexample, a derivation, a reading of the released code and a recalculation on the public data. Suppose that two simulated assignments induce exposures coded 0 and 1, and that $d_k$ is the exposure coded 1. The proportion of assignments with exposure $d_k$ is $1/2$. The printed formula uses the last assignment in both terms and gives 1. With $\\mathbf{z}^{\\ell}$ in place of $\\mathbf{z}^{J}$ and the indicator left outside the average, I obtain $\\mathbf{I}[(0+1)/2=1]=0$.\n\nFor the corrected formula, let $\\mathbf{z}^{1},\\ldots,\\mathbf{z}^{J}$ be independent draws from the design, and set $X_\\ell=\\mathbf{I}[f(\\mathbf{z}^{\\ell},\\theta_i)=d_k]$. Each $X_\\ell$ has mean $\\pi_i(d_k)$ and variance $\\pi_i(d_k)(1-\\pi_i(d_k))$, which implies that $\\hat\\pi_i(d_k)$ has mean $\\pi_i(d_k)$ and variance $\\pi_i(d_k)(1-\\pi_i(d_k))/J$.\n\nThe released code performs this averaging. Its probability script forms a 0/1 indicator of each exposure for each simulated assignment (lines 60 and 170--206) and averages each indicator over the assignments (lines 208--214). A reconstruction with an adapter on the public data (ICPSR 37070 V2) executed these expressions for all 10,000 assignments and 24,191 retained students. Every indicator is 0 or 1, the five basic exposures partition every assignment, and each average equals the number of assignments with the exposure divided by 10,000. In this reconstruction, the simulated probabilities of control exposure lie between 0.4881 and 0.5125 and are fractional for every student, where the indicator of the last assignment would be 0 or 1.", "problem": "As printed, the right-hand side is the indicator that an average of exposure labels equals $d_k$. The labels are categories, and an average of categories is not defined. A single indicator takes only the values 0 and 1, so it is not the simulated frequency and cannot become arbitrarily precise for a probability strictly between 0 and 1 as $J$ grows. The summand carries the superscript $J$ and does not depend on the index $\\ell$: every term uses the last simulated assignment. The preceding definition states the quantity that is meant, the probability that an assignment induces exposure $d_k$. The same formula is printed on p.~14.", "what_survives": "The exact definition $\\pi_i(d_k)=\\sum_{\\mathbf{z}\\in\\Omega}\\mathbf{I}(f(\\mathbf{z},\\theta_i)=d_k)\\Pr(\\mathbf{Z}=\\mathbf{z})$, printed before the formula, is correct and fixes the meaning. The released code averages the indicators of the separate assignments, and correcting the two displays changes no output of that code. No reported estimate is shown to change. The authors' original data file and saved simulations are unavailable, ICPSR reports changes to released variables, and the reconstruction uses the current R runtime, so I cannot recover the historical probabilities exactly. This check does not cover other simulation choices or outcome analyses (see \\fid{P037-N031} for the school draws)."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E003", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Four lines of text from page 19 of the SI Appendix, beginning Consistency and asymptotic normality directly follow from the law of large numbers and a Lindeberg-type central limit theorem given a growing sequence of independent schools of finite size, as holds by construction given our between-school randomization. One red box marks the words independent schools, and a second red box marks the words as holds by construction.", "caption": "SI Appendix, p. 19 (PDF p. 19). The marked words state that the schools are independent and that this holds by construction; the design assigns two of the four schools in each block to treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T10", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T10"}, {"id": "P037-T11", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T11"}], "headline": "The schools are not independent by construction, because schools in the same randomization block of four are dependent; the independent units are the blocks, and the argument for consistency and asymptotic normality should be made for block totals under stated conditions.", "id": "P037-E004", "location": "SI Appendix, p. 19 (PDF p. 19 of the SI Appendix), paragraph beginning “Consistency and asymptotic normality”; the design is described on p. 5 and in the caption of Fig. S1 (PDF p. 24)", "pdf_page": 19, "printed_page": "19", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [{"id": "P045-N018", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N018", "work_id": "P045"}], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T10", "P037-T11"], "text": {"as_printed": "``Consistency and asymptotic normality directly follow from the law of large numbers and a Lindeberg-type central limit theorem given a growing sequence of \\hl{independent schools} of finite size, \\hl{as holds by construction} given our between-school randomization.''", "corrected": "``Consistency and asymptotic normality follow from the law of large numbers and a Lindeberg-type central limit theorem \\hl{for the totals of the four-school randomization blocks}, given a growing sequence of \\hl{independently randomized blocks} of finite size \\hl{and the conditions below. Schools within a block are dependent.}''\n\nA sufficient set of conditions, for the fixed population above, an exposure $d_k$ and in the notation of the evidence, is the following, with positive constants $m$, $c$, $\\eta$, $K$ and $v$ that do not depend on the number of blocks $B$. For a contrast, impose the exposure-specific conditions for both exposures.\n\\begin{enumerate}\n\\item The randomizations of different blocks are independent, including the draws of seeds within their schools. Each student's exposure depends only on the randomization of that student's block.\n\\item Each block has at most $m$ retained students, and $N_B\\ge cB$.\n\\item For every retained student, $\\pi_i(d_k)\\ge\\eta>0$, $|y_i(d_k)|\\le K$ and $|g_i|\\le K$.\n\\item The variance $s_B^2=\\sum_{b=1}^{B}\\operatorname{Var}(X_b)$ satisfies $s_B^2\\ge vB$ with $v>0$; for a contrast, the same holds for the block totals of the contrast.\n\\item The regression is linear in covariates $x_i$ of fixed dimension and bounded norm and is fitted by inverse-probability-weighted least squares on the retained students with exposure $d_k$. The sums over all retained students satisfy $N_B^{-1}\\sum_i x_ix_i^\\top\\to Q$ for a positive definite matrix $Q$ and $N_B^{-1}\\sum_i x_iy_i(d_k)\\to w$, and $\\beta_*=Q^{-1}w$.\n\\item The simulated probabilities satisfy $\\max_i|\\hat\\pi_i(d_k)-\\pi_i(d_k)|=o_p(B^{-1/2})$.\n\\end{enumerate}\nUnder conditions 1--4, the estimator with exact probabilities and fixed predictions is consistent and asymptotically normal as $B\\to\\infty$; conditions 5 and 6 extend the conclusion to the fitted regression and to the simulated probabilities. The conditions are sufficient and are not claimed to be necessary, and they are not checked here for a sequence that contains the 56 schools of the study. Condition 5 gives a sufficient linear case of the regression displayed on p.~19; the regression of the released code is the subject of \\fid{P037-N033}. Regrouping the proof does not change a variance calculation under constant effects that simulates the blocked randomization.\n\nTo make the released variance sampler follow the design, line 58 of \\texttt{1. Sanitize and generate perms, probs.R} should read:\n\n\\texttt{sch.perms <- genperms(schtreat, \\hl{blockvar = schrb}, clustvar = schid, maxiter = niter)}\n\nThe exposure draws must then be regenerated and passed through scripts 1--3 before recomputing the network standard errors, intervals and $p$-values. This code repair is separate from regrouping the asymptotic proof.", "evidence": "The evidence is an exact enumeration, a derivation and a test of the released school sampler on the public data. Let $T_A,T_B,T_C,T_D$ denote the treatment indicators of the four schools of a block. The six allocations of two treated schools are equally likely; three of them treat school $A$, three treat school $B$, and one treats both. I have $\\Pr\\{T_A=1\\}=\\Pr\\{T_B=1\\}=1/2$ and $\\Pr\\{T_A=1,T_B=1\\}=1/6$, where independence would require $1/4$, and $\\operatorname{Cov}(T_A,T_B)=-1/12$. The same covariance holds for every pair of schools in the block and for the control-exposure indicators $1-T_A$ and $1-T_B$. The dependence does not rest on equal probabilities for the six allocations: the sum $T_A+T_B+T_C+T_D=2$ has variance zero, while the sum of four independent indicators of mean $1/2$ has variance $1$. The released script for the experimental randomization permutes two labels of one arm and two labels of the other within each block of four (\\texttt{7. Blocking, randomization, and summary statistics.R}, lines 83--90), and on the public data each of the 14 blocks has four schools, two of them treated.\n\nThe simulation script, \\texttt{1. Sanitize and generate perms, probs.R}, line 58, calls \\texttt{genperms} without \\texttt{blockvar=schrb}, drawing 28 of 56 schools without the original blocks; exact counting gives covariance $-1/220$ for every distinct school pair. In a 10,000-draw test on the public data (ICPSR 37070 V2), every draw from this call violated at least one original block, while adding \\texttt{blockvar=schrb} gave no violations. Script 1 saves the first 1,000 exposure draws (lines 232--242), script 2 loads them (line 16), and script 3 uses them for the network standard errors and Wald bands (lines 237--264 and 325--328). The public data reproduce the published 24,191 pre-analysis records and the 56-school, 14-block allocation, but ICPSR warns that variables were changed and the authors' original analysis files are unavailable.\n\nFor the corrected statement, at each number of blocks $B$, fix the schools, within-school networks, covariates, potential outcomes and retained student sets. Fix an exposure $d_k$, and write $I_i=\\mathbf{1}\\{D_i=d_k\\}$ and $\\pi_i=\\pi_i(d_k)$. Let $S_b$ denote the retained students of block $b$, and let $N_B=\\sum_{b=1}^B|S_b|$. For fixed predictions $g_i$, define\n\\[\nh_i=g_i+\\frac{I_i}{\\pi_i}\\bigl(y_i(d_k)-g_i\\bigr),\\qquad X_b=\\sum_{i\\in S_b}h_i,\\qquad \\widetilde\\mu_B=\\frac{1}{N_B}\\sum_{b=1}^{B}X_b .\n\\]\nSince $\\mathbb{E}[I_i]=\\pi_i$, I have $\\mathbb{E}[h_i]=y_i(d_k)$, and $\\widetilde\\mu_B$ is unbiased for $\\mu_B=N_B^{-1}\\sum_i y_i(d_k)$. Each block total $X_b$ depends only on the randomization of block $b$, and condition 1 of the corrected statement implies that the $X_b$ are independent. Under conditions 2 and 3, I have $|h_i|\\le H$ with $H=K+2K/\\eta$, and $|X_b|\\le mH$. Using these bounds and the independence of the block totals, I obtain\n\\[\n\\operatorname{Var}(\\widetilde\\mu_B)=\\frac{1}{N_B^2}\\sum_{b=1}^{B}\\operatorname{Var}(X_b)\\le\\frac{m^2H^2}{c^2B}.\n\\]\nThis bound and Chebyshev's inequality imply consistency. Under condition 4, for every $\\epsilon>0$ I have $|X_b-\\mathbb{E}[X_b]|\\le 2mH<\\epsilon s_B$ once $B$ is large. Then every term of the Lindeberg sum vanishes, and $N_B(\\widetilde\\mu_B-\\mu_B)/s_B$ converges in distribution to a standard normal variable. For a contrast between two exposures, $h_i$ is replaced by the difference of its values for the two exposures.\n\nWith fitted coefficients $\\widehat\\beta$, and with $g_i=x_i^\\top\\beta_*$ in $\\widetilde\\mu_B$, the regression estimator with exact probabilities differs from $\\widetilde\\mu_B$ by\n\\[\n\\Bigl[\\frac{1}{N_B}\\sum_i\\Bigl(1-\\frac{I_i}{\\pi_i}\\Bigr)x_i\\Bigr]^\\top\\bigl(\\widehat\\beta-\\beta_*\\bigr).\n\\]\nUnder conditions 1--3 and 5, the bracket is $O_p(B^{-1/2})$ and $\\widehat\\beta-\\beta_*=o_p(1)$. I conclude that the difference is $o_p(B^{-1/2})$, which is negligible beside $s_B/N_B\\ge\\sqrt{v}/(m\\sqrt{B})$. Under condition 6, replacing $\\pi_i$ by the simulated $\\hat\\pi_i$ changes the estimator by $o_p(B^{-1/2})$ as well.", "problem": "The design randomizes schools within blocks of four: in each of the 14 blocks, two schools are assigned to treatment and two to control (SI Appendix, p.~5, and the caption of Fig.~S1). The treatment indicators of two schools in a block are dependent, and so are their control-exposure indicators $1-s_i$. The schools are not independent by construction, and the law of large numbers and the central limit theorem cannot be applied to them as the sentence does. The independent units are the blocks. The argument can be made for block totals, under regularity conditions that the sentence does not state.", "what_survives": "Regrouping the asymptotic proof alone leaves the exposure definitions, the estimand and the point estimates unchanged. The finding does not show that the estimator is inconsistent or that a reported standard error, interval or $p$-value is wrong. The normal approximation behind the Wald intervals and $p$-values of the social-influence analysis (SI Appendix, p.~20, and Fig.~3) needs the argument for blocks and its conditions. A variance simulation under constant effects that follows the allocation of two schools in each block of four keeps the dependence within blocks. The released school sampler instead omits the blocks; its numerical effect on the public data is measured in \\fid{P037-N031}. Regenerating the simulated probabilities may change the retained set and fitted estimates. This correction does not affect the school-climate regressions."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E004", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Table S6, Average school-level climate effects caused by anti-conflict groups, with columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict, and rows School level main effect, Heteroskedasticity Robust SE, Bayesian Bootstrap SE and Wild Bootstrap SE. A red box marks the row label Bayesian Bootstrap SE, whose entries are (0.01), (0.05), (0.01) and (0.03). In the note below the table, a red box that runs over a line break marks the words applying weights to each school drawn from the standard uniform distribution.", "caption": "SI Appendix, Table S6 and its note (PDF p. 41; the page is printed sideways and is shown upright). The marks are on the row label “Bayesian Bootstrap SE” and on the words of the note that describe the weights as drawn from the standard uniform distribution.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E005.png"}, {"alt": "Table S7, Average school-level climate effects caused by anti-conflict groups, interacted with proportion of social referent seeds in the group, with the same four columns as Table S6. Under School effect with 0 percent Social Referents and under Interaction: Proportion of Referents in Treatment group there are three rows of standard errors each. Red boxes mark the two row labels Bayesian Bootstrap SE, whose entries are (0.03), (0.11), (0.02), (0.05) and (0.22), (0.80), (0.13), (0.35). In the note below the table, a red box marks the words applying weights to each school drawn from the standard uniform distribution.", "caption": "SI Appendix, Table S7 and its note (PDF p. 42; the page is printed sideways and is shown upright). The marks are on the two rows labelled “Bayesian Bootstrap SE” and on the same description of the weights in the note.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E005-b.png"}, {"alt": "Dot plot with twelve rows in three groups: Table S6, school level main effect; Table S7, school effect with 0 percent social referents; Table S7, interaction with proportion of referents. Each group has the rows Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict. The horizontal axis is the ratio of the standard deviation with uniform weights to the standard deviation with Dirichlet weights, from 0.55 to 1.03. A blue vertical line at 1 is labelled Bayesian bootstrap (Dirichlet weights). The twelve orange dots lie between 0.683 and 0.730, each with a narrow orange bar extending two Monte Carlo standard errors on each side and its value printed beside it: 0.683, 0.711, 0.693, 0.720 in the first group; 0.710, 0.715, 0.712, 0.723 in the second; 0.720, 0.706, 0.716, 0.730 in the third.", "caption": "The ratio of the standard deviation of each refitted coefficient under uniform school weights to its standard deviation under Dirichlet$(1,\\ldots,1)$ weights, for the twelve entries of Tables S6 and S7, from 50,000 paired refits per regression on the public data (ICPSR 37070 V2). The vertical line at 1 is the Bayesian-bootstrap reference. Orange bars extend two Monte Carlo standard errors on each side of a dot; each standard error is at most 0.003.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E005-weights.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T4", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T4"}, {"id": "P037-T5", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T5"}, {"id": "P037-T16", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T16"}], "headline": "The twelve entries labelled “Bayesian Bootstrap SE” in Tables S6 and S7 were computed with an independent uniform weight for each school, which is not the Bayesian bootstrap; in a reconstruction on the public data, uniform-weight dispersions are 27% to 32% smaller than the corresponding dispersions under the Bayesian bootstrap. The rows should be relabelled as dispersions under uniform weights, or recomputed with Dirichlet$(1,\\ldots,1)$ weights.", "id": "P037-E005", "location": "SI Appendix, Table S6 (PDF p. 41 of the SI Appendix), note and row “Bayesian Bootstrap SE”; the same note and two rows of Table S7 (PDF p. 42)", "pdf_page": 41, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T16", "P037-T4", "P037-T5"], "text": {"as_printed": "Notes to Tables S6 and S7: ``\\hl{Bayesian bootstrap standard errors} generated using 5,000 replicates, \\hl{applying weights to each school drawn from the standard uniform distribution}.''\n\nRow label, once in Table S6 and twice in Table S7: ``\\hl{Bayesian Bootstrap SE}''.", "corrected": "The three rows are labelled ``\\hl{Uniform-weight perturbation SD}'', their entries stay as printed, and the notes to Tables S6 and S7 read: ``\\hl{Uniform-weight perturbation standard deviations} generated using 5,000 replicates, applying \\hl{independent $U(0,1)$ weights to the schools}.'' The entries are dispersions under uniform weights, and they are not Bayesian-bootstrap standard errors.\n\nIf Bayesian-bootstrap standard errors are wanted instead, the twelve entries are recomputed as follows. For replicate $r=1,\\ldots,R$, with $R=5{,}000$ for the revised notes below, draw independent standard exponential variables $E_1^{(r)},\\ldots,E_n^{(r)}$ for the $n$ schools of the regression, set $p_s^{(r)}=E_s^{(r)}/\\sum_t E_t^{(r)}$, and compute\n\\[\n\\widehat b^{(r)}=\\arg\\min_b\\sum_{s=1}^{n}p_s^{(r)}\\bigl(y_s-x_s^\\top b\\bigr)^2,\\qquad \\widehat{\\mathrm{SE}}=\\Bigl[\\frac{1}{R-1}\\sum_{r=1}^{R}\\bigl(\\widehat b^{(r)}_k-\\bar b_k\\bigr)^2\\Bigr]^{1/2},\n\\]\nwhere $k$ is the coefficient of interest and $\\bar b_k$ is the mean of the $\\widehat b^{(r)}_k$. The weights $p^{(r)}$ then have the Dirichlet$(1,\\ldots,1)$ law, and the notes read: ``Bayesian bootstrap standard errors generated using 5,000 replicates, applying weights to \\hl{the schools with the Dirichlet$(1,\\ldots,1)$ law, obtained by dividing independent standard exponential draws by their sum}.'' The column ``Dirichlet'' of the table above gives estimates of this dispersion on the public data with $R=50{,}000$, and their conditional Monte Carlo standard errors. The use of the method for the blocked design would have to be justified.", "evidence": "The evidence is an exact counterexample, a reading of the released code and a recalculation on the public data. Let $U_1,\\ldots,U_{56}$ be independent and uniform on $(0,1)$, and set $P_i=U_i/\\sum_j U_j$. Let $E_1,\\ldots,E_{56}$ be independent standard exponential variables, and set $B_i=E_i/\\sum_j E_j$, so that $(B_1,\\ldots,B_{56})$ has the Dirichlet$(1,\\ldots,1)$ law. Consider the event $p_1/(p_1+p_2)<1/4$. Since $P_1/(P_1+P_2)=U_1/(U_1+U_2)$, the event holds for $P$ exactly when $U_1<U_2/3$, and I have\n\\[\n\\Pr\\{U_1<U_2/3\\}=\\int_0^1\\frac{v}{3}\\,dv=\\frac16 .\n\\]\nFor $B$ the event is $E_1<E_2/3$, and a direct computation gives\n\\[\n\\Pr\\{E_1<E_2/3\\}=\\int_0^\\infty e^{-v}\\bigl(1-e^{-v/3}\\bigr)\\,dv=\\frac14 .\n\\]\nEqual laws assign equal probabilities to every event, and the two laws are therefore different. Only two of the weights enter the event, and the conclusion holds for any number of schools from two upwards, including the 49 schools of the peer-conflict regressions.\n\n\\begin{audittable}{lll}{Two laws for the weights $p_1,\\ldots,p_{56}$ of the 56 schools, normalized to sum to one. Uniform weights: independent $U(0,1)$ draws, as the notes describe and the released code computes. Bayesian bootstrap: Dirichlet$(1,\\ldots,1)$ weights, from independent standard exponential draws. The last row refers to a draw before normalization.}\nQuantity & Uniform weights & Bayesian bootstrap \\\\\nDensity in the first 55 coordinates & $1/\\bigl(56(\\max_i p_i)^{56}\\bigr)$ & $55!$ \\\\\n$\\Pr\\{p_1/(p_1+p_2)<1/4\\}$ & $1/6$ & $1/4$ \\\\\nVariance over squared mean of a draw & $1/3$ & $1$ \\\\\n\\end{audittable}\n\nNormalization does not remove the difference. For a design matrix $X$ of full column rank, positive weights $w$ with $W=\\operatorname{diag}(w)$, and a constant $c>0$, the weighted least-squares coefficients satisfy\n\\[\n\\widehat b(cw)=(cX^\\top WX)^{-1}cX^\\top Wy=\\widehat b(w).\n\\]\nA refit with uniform weights has the same coefficients whether or not the weights are divided by their sum.\n\nFor independent weights, to first order the covariance of the refitted coefficients is $\\sigma^2/\\mu^2$ times the unadjusted heteroskedasticity-robust covariance (HC0) of the least-squares coefficients, where $\\mu$ and $\\sigma^2$ denote the mean and variance of a weight. The factor is $1/3$ for $U(0,1)$ weights and $1$ for standard exponential weights. This is a statement about the linearized coefficients. It is not a correction for the sample at hand, and it does not justify multiplying the printed entries by $\\sqrt{3}$.\n\nI ran the two loops of the released code in a reconstruction on the public data (ICPSR 37070 V2), with adapted inputs and 5,000 replicates as in the code, and then refitted each regression 50,000 times with uniform weights and with Dirichlet weights built from the same draws ($U$ for the uniform weight and $-\\log U$ for the exponential one). The table reports the results, and the figure shows the ratios.\n\n\\begin{audittable}{llrrrr}{The twelve entries labelled ``Bayesian Bootstrap SE''. Printed: the entries of Tables S6 and S7. Uniform: standard deviation of the coefficient over 5,000 replicates of the released code's loop on the public data. Dirichlet: standard deviation over 50,000 refits with Dirichlet$(1,\\ldots,1)$ weights on the public data, with its conditional Monte Carlo standard error in parentheses. Ratio: uniform over Dirichlet in 50,000 paired refits. The values in the last three columns are computed from the public data; they do not establish exact values for the authors' own data file.}\nCoefficient & Outcome & Printed & Uniform & Dirichlet & Ratio \\\\\nS6, main effect & Talk & 0.01 & 0.0125 & 0.0182 (0.000059) & 0.683 \\\\\n & Norms & 0.05 & 0.0539 & 0.0761 (0.000247) & 0.711 \\\\\n & Wristband & 0.01 & 0.0081 & 0.0115 (0.000036) & 0.693 \\\\\n & Conflict & 0.03 & 0.0252 & 0.0349 (0.000110) & 0.720 \\\\\nS7, 0\\% referents & Talk & 0.03 & 0.0250 & 0.0352 (0.000115) & 0.710 \\\\\n & Norms & 0.11 & 0.1107 & 0.1515 (0.000495) & 0.715 \\\\\n & Wristband & 0.02 & 0.0160 & 0.0222 (0.000069) & 0.712 \\\\\n & Conflict & 0.05 & 0.0510 & 0.0692 (0.000220) & 0.723 \\\\\nS7, interaction & Talk & 0.22 & 0.2117 & 0.2986 (0.000957) & 0.720 \\\\\n & Norms & 0.80 & 0.8183 & 1.1336 (0.003750) & 0.706 \\\\\n & Wristband & 0.13 & 0.1370 & 0.1863 (0.000602) & 0.716 \\\\\n & Conflict & 0.35 & 0.3575 & 0.4821 (0.001569) & 0.730 \\\\\n\\end{audittable}\n\nThe twelve ratios lie between 0.683 and 0.730, so the standard deviations under uniform weights are 27.0\\% to 31.7\\% smaller. The public data reproduce the printed tables closely but not exactly. Nine of the twelve point estimates and eleven of the twelve robust standard errors round to the printed values, and each of the twelve standard deviations under uniform weights is compatible with the printed entry once rounding and two Monte Carlo standard errors are allowed (eight of them with rounding alone). The robust covariance of the code is HC0.", "problem": "The Bayesian bootstrap assigns random weights to the units; the weights sum to one and have the Dirichlet$(1,\\ldots,1)$ law. Dirichlet weights are obtained from independent draws of the standard exponential distribution, each divided by the sum of the draws, or, for $n$ schools, as the $n$ gaps formed by $n-1$ ordered independent $U(0,1)$ draws together with the endpoints $0$ and $1$. The notes to Tables S6 and S7 describe a different rule, a weight for each school drawn from the standard uniform distribution, and the released code computes that rule. In each of 5,000 replicates it draws one independent $U(0,1)$ weight for each school, passes the weights to the regression unchanged, and keeps the coefficient; the entry is the standard deviation of the 5,000 coefficients (\\texttt{4. School results-with-correction.R}, lines 165--175 for Table S6 and 212--226 for Table S7). The code has no ordering, no gaps and no logarithm. Uniform weights do not have the Dirichlet law, and they do not acquire it when they are divided by their sum. The notes describe the computation correctly, and the name of the rows is wrong. The four entries of Table S6 and the eight entries of Table S7 are dispersions under uniform weights, and they are not standard errors from a Bayesian bootstrap. In the comparison using 50,000 paired refits per regression on the public data, each of the twelve uniform-weight dispersions is smaller than its counterpart with Dirichlet weights, by 27\\% to 32\\%. Read as Bayesian-bootstrap standard errors, the rows understate the dispersion under that method on the public data, and they cannot serve as a check of the robust standard errors by that method.", "what_survives": "This finding does not require changing the point estimates of Tables S6 and S7, the rows of heteroskedasticity-robust standard errors or the rows of standard errors from the wild bootstrap. For school-level effects, the main text computes confidence intervals and $P$ values from robust standard errors (p.~568), and the intervals of Fig.~2 use heteroskedasticity-robust standard errors (caption, p.~569); neither uses the rows in question. The student-level Wald tests use randomization-based variance estimates (p.~569); those analyses and the headline estimates do not depend on these rows. What changes is the reading of the twelve entries, which cannot be cited as standard errors from a Bayesian bootstrap. The values with Dirichlet weights in the table are Monte Carlo estimates on the public data. They are not replacements for the printed entries, which would require the authors' own data file; that file is not public, and ICPSR states that some variables of the release were changed. The comparison is specific to these regressions. It establishes no general ordering of the two dispersions and no coverage property of either weighting under the blocked design."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E005", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of eight lines of text beginning 'The survey that all students took at the beginning of the year'. Red boxes mark the sentence 'Norms questions were on a scale of 0 to 5, with answers ranging from almost nobody to almost everybody.'", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked sentence gives one response scale for the descriptive and the prescriptive norm questions.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006.png"}, {"alt": "Crop of the heading 'Descriptive norms scale (asked at pretest and posttest)' and the first two items of the list. A red box marks the lines 'How often do you see students...' and '(0 = Never, 1 = 1-2 times / month, 2 = About 1 time /week, 3 = 2-3 times / week, 4 = Every day)'.", "caption": "SI Appendix, p. 8. The supplement’s own statement of the descriptive-norm question and of its scale from 0 (Never) to 4 (Every day), which the sentence on p. 567 does not describe.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-b.png"}, {"alt": "Crop of four lines. A red box marks '(Yes/No)' after 'Talking to friends item: This year, friends talked with me about how to change student behavior'. The next item, 'Orange wristband report item: I wore an orange Roots wristband this past week', is also followed by '(Yes/No)', which is not marked.", "caption": "SI Appendix, p. 10. The marked answer format is wrong for the talking item, which had four answers coded 0 to 3. The wristband item below it is also printed with a Yes/No answer.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-c.png"}, {"alt": "Crop of the heading 'Prescriptive norms scale', the stem 'How many students think it is...' and items 1 to 9 of the list. Red boxes mark item 3, 'Not funny to mess with or pick on others'; item 4, 'Not OK to say mean things to girl because of what she does with boys'; item 5, 'Not funny to post online or text mean things'; item 6, 'Not OK to use racial, ethnic jokes'; and item 8, 'OK to tell adults about conflict'.", "caption": "SI Appendix, p. 9, list of the prescriptive-norm items. The marked items (3–6 and 8) have the opposite orientation to the questionnaire statements.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-d.png"}, {"alt": "Crop of Table S3, 'Factor loadings for Prescriptive norm items', with the columns Survey Item, Wave 1 and Wave 2. Red boxes mark the consecutive row labels 'Item 7' and 'Item 9', between which there is no Item 8, and the last row label, 'Item 13', which has a Wave 2 loading of minus 0.38 and no Wave 1 entry.", "caption": "SI Appendix, Table S3, p. 38. The rows keep the item numbers of the questionnaire: there is no Item 8, and Item 13 corresponds to item 12 of the list on p. 9.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E006-e.png"}], "formal_release": null, "formal_statements": [], "headline": "The descriptive-norm items had a frequency scale from 0 to 4 and the talking item a scale from 0 to 3, not the single 0–5 scale and the Yes/No answer that are printed; the supplement’s lists of items also need a key to the item numbers of Tables S3 and S4.", "id": "P037-E006", "location": "Methods and Materials, p. 567 (PDF p. 2), right column, paragraph beginning “The survey that all students took”; SI Appendix, pp. 8–10 (survey wording) and pp. 38–39 (Tables S3 and S4)", "pdf_page": 2, "printed_page": "567", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Main text, p.~567: \\ldots\\ students' estimates of descriptive norms (\\hl{how many students at the school} participate in various forms of conflict), and prescriptive norms (how many students at the school think it is desirable to participate in various forms of conflict). \\hl{Norms questions were on a scale of 0--5}, with answers ranging from ``almost nobody'' to ``almost everybody.''\n\nSI p.~10: Talking to friends item: ``This year, friends talked with me about how to change student behavior'' \\hl{(Yes/No)}\n\nSI p.~9, prescriptive norms scale, ``How many students think it is\\ldots'', items 3--6 and 8 of the list:\n\\begin{itemize}\n\\item 3. \\hl{Not funny} to mess with or pick on others\n\\item 4. \\hl{Not OK} to say mean things to girl because of what she does with boys\n\\item 5. \\hl{Not funny} to post online or text mean things\n\\item 6. \\hl{Not OK} to use racial, ethnic jokes\n\\item 8. \\hl{OK} to tell adults about conflict\n\\end{itemize}\n\nTable S3 (SI p.~38) has the rows Item 1 to Item 7 and Item 9 to Item 13. Table S4 (SI p.~39) has the rows Item 1 to Item 12 and Item 14.", "corrected": "Main text, p.~567: Descriptive-norm questions asked \\hl{how often students saw the listed behaviors at school (0 = Never, 1 = 1--2 times/month, 2 = About 1 time/week, 3 = 2--3 times/week, 4 = Every day)}. Prescriptive-norm questions asked how many students at the school \\hl{endorsed the listed statements, with six answers coded 0--5 (Almost nobody, A few people, About 25\\%, About 50\\%, About 75\\%, Almost everyone)}.\n\nSI p.~10: Talking to friends item: ``This year, \\hl{my} friends talked with me about how to change \\hl{the way students at this school act around each other}'' \\hl{(0 = Not true, 1 = A little bit, 2 = Some, 3 = Very true)}\n\nSI p.~9: Items 3--6 and 8 of the list are summaries in the opposite orientation to the questionnaire wording. In the questionnaire, item 6 reads ``It's okay to use jokes to insult someone else's race or ethnicity'', and the item listed as 8 says that it is not okay to tell an adult at school about student conflicts. \\hl{The supplied scripts compute the index as the first principal component of the covariance matrix of the selected input variables, on complete cases, without reversing individual items. The outcome analyses use this component with its sign reversed as a whole. The loadings of Table S3 refer to these input variables, up to the overall sign of the component.} For item 8 of the list at the second wave, the heading in the codebook of the public release reads ``okay'', against the questionnaire of both waves and the investigators' codebook. I report this difference of labels and draw no conclusion from it about the data.\n\nTables S3 and S4: The item numbers are those of the questionnaire. Items 9--13 of Table S3 are items 8--12 of the list on SI p.~9, and Item 14 of Table S4 is item 13 of the list on SI p.~8. Prescriptive item 8 and descriptive item 13 of the questionnaire, on working hard at school, appear in neither the lists nor the tables, and the scripts leave them out of the indices.", "evidence": "I checked the pages against the questionnaire and the two codebooks of the public release (ICPSR 37070, version 2). Both waves of the questionnaire print the five frequency answers for the descriptive items and six answers, from ``Almost nobody'' to ``Almost everyone'', for the prescriptive items. The second wave prints the talking statement with the answers Not true, A little bit, Some and Very true. The investigators' codebook marks the prescriptive items 3--6 and 9 of the questionnaire for reverse coding and lists the supplement's wording as the intended labels of the reversed items. These notes explain the wording of the list. They do not show that the items were recoded.\n\nI then read the authors' analysis scripts and ran the parts that bear on these items on the public data with an adapter for the released file. The scripts select the input variables by their questionnaire numbers and do not reverse any single item (\\texttt{2. Cleaning.R}, lines 56--61). The index is the first principal component of the covariance matrix of the selected input variables, on complete cases (\\texttt{5. Factor loadings and description.R}), and the scripts for the outcome analyses reverse the sign of the whole prescriptive component. On the public data all 48 loadings of Tables S3 and S4 agree with the printed values to two decimals, after one choice of overall sign for each component. The talking item keeps its four scores. On the public data, the adjusted school-level effect on talking is $0.0874$ points of the 0--3 score (robust HC0 standard error $0.0176$) in 56 schools, and Table S6 prints $0.09$ and $0.02$.", "problem": "The main text describes both sets of norm items with one question and one response scale. The question and the scale are those of the prescriptive items. The descriptive items asked ``How often do you see students\\ldots'', with five answers coded from 0 (Never) to 4 (Every day), as SI p.~8 states. The talking item had four answers coded 0 to 3, as the axes of Figs.~2A and 3A say. This implies that the effect of 0.09 on talking in Table S6 is in points of that score. It is not a difference of nine percentage points in the proportion answering Yes.\n\nIn the supplement's list of prescriptive items, items 3--6 and 8 have the opposite orientation to the questionnaire wording, and the list does not say so. The two lists also renumber the items after two questions on schoolwork were left out, while Tables S3 and S4 keep the numbers of the questionnaire. Items 8--12 of the list on SI p.~9 are Items 9--13 of Table S3, and item 13 of the list on SI p.~8 is Item 14 of Table S4.", "what_survives": "These documentation corrections require no change to the reproduced loadings of Tables S3 and S4 or the talking effect and robust standard error in Table S6, which agree on the public data at the printed precision. The authors' own input files are not public and ICPSR states that some variables of the release were changed, so I cannot establish their original inputs or coding before the supplied scripts; I did not check other estimates here. I did not examine here whether the final norms score has the direction that the article states; the orientation of the index is discussed in \\fid{P037-N023}."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E006", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Table S1, Summary statistics for each school block and for overall sample: fourteen school blocks and a Total Sample row, each with a row of means and a row of dispersions, in sixteen columns, among them Home language, School size and percent White. A red box marks the column heading Home language. A second red box marks 0.40, the Total Sample mean of that column. In the note below the table, two red boxes mark the words speak only English (1) vs. other languages (0) across a line break.", "caption": "SI Appendix, Table S1 and the first lines of its note (PDF p. 36; the page is printed sideways and is shown upright). The marks are on the column heading “Home language”, on its Total Sample mean 0.40, and on the words of the note that define the column as the proportion who speak only English (1) vs. other languages (0).", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E007.png"}], "formal_release": null, "formal_statements": [], "headline": "The note to Table S1 reverses the coding of the home-language column: the entries are the share of students who speak a language other than English at home, alone or together with English, and not the share who speak only English.", "id": "P037-E007", "location": "SI Appendix, Table S1 (PDF p. 36 of the SI Appendix; the page carries no number), column “Home language” and its definition in the note", "pdf_page": 36, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Note to Table S1: ``home language'' represents the average proportion of students who reported that they speak \\hl{only English (1) vs. other languages (0)} at home in the pre-randomization student surveys;\n\nColumn ``Home language'', means for school blocks 1 to 14: 0.42, 0.35, 0.44, 0.20, 0.41, 0.23, 0.71, 0.35, 0.84, 0.11, 0.31, 0.28, 0.41, 0.49. Total Sample: 0.40, with 0.24 in the second row.", "corrected": "Note to Table S1: ``home language'' represents the average proportion of students who reported \\hl{speaking a language other than English at home, either alone or together with English (1), versus speaking only English (0)}, in the pre-randomization student surveys.\n\nThe entries of the column stay as printed. If the share who speak only English is wanted instead, the means are replaced by their complements, taken before rounding, and the standard deviations stay as printed. On the public data the complements for school blocks 1 to 14 are 0.58, 0.65, 0.56, 0.80, 0.59, 0.77, 0.29, 0.65, 0.16, 0.89, 0.69, 0.72, 0.59 and 0.51, and 0.60 for the Total Sample.", "evidence": "The evidence is a reading of the released code and a recalculation on the public data. I ran the relevant recoding and aggregation lines on the public data (ICPSR 37070 V2), with explicit translation of factor and missing-value codes. All 14 block means, all 14 block standard deviations and the two entries of the Total Sample row agree with the printed column at two decimals. The standard deviations are computed with divisor 4 within a block and 56 for the whole sample. On the public data, the mean over the 56 schools of the share who speak another language at home is 0.3968, the mean of the share who speak only English is 0.6032, and the standard deviation is 0.2383. In the public file, 14,342 students answer English only, 706 another language only and 7,671 English and another language, and 1,752 answers are missing. If Table S1's participants form a subset of the public release with unchanged language answers, the printed definition also conflicts with the public counts without reference to the code: by the Cauchy--Schwarz inequality, the block means and standard deviations of the column and of school size allow at most 9,273 English-only answers among the 22,732 survey participants of Table S1, and any 22,732 of the 24,471 released records contain at least 12,603.\n\nFor the regressions, let $x$ denote the language control and $Z$ the other columns of the design. Since the intercept and the block fixed effects span the constant, there exists a vector $c$, zero outside those columns, such that $Zc=\\mathbf{1}$, and I have $Z\\theta+x\\gamma=Z(\\theta+c\\gamma)+(1-x)(-\\gamma)$. This identity implies that replacing $x$ by $1-x$ on the same analysis rows leaves the column space, the fitted values, the residuals and the treatment coefficients unchanged. In the two adjusted regressions for peer conflict at the school level, on the public data (49 schools), the treatment and interaction coefficients change by less than $4\\times 10^{-16}$ and their robust standard errors by less than $1.2\\times 10^{-15}$.", "problem": "Read with the note, the column says that on average across schools 40\\% of students speak only English at home. The released code computes the column from the opposite indicator. The cleaning script codes a student as 0 for English only and as 1 for another language, alone or together with English (\\texttt{2. Cleaning.R}, lines 42--48), and the script that produces Table S1 averages this indicator within schools and then over the four schools of each block (\\texttt{7. Blocking, randomization, and summary statistics.R}, lines 112--125 and 159--195). The entries of the column are shares of students who speak another language at home, and the definition in the note is reversed.", "what_survives": "The numbers of the column stay as printed, and the correction of the note changes no computation. The adjusted regressions at the school level and the network regression use the same indicator, of another language at home, as an additive control (\\texttt{4. School results-with-correction.R}, lines 162 and 208; \\texttt{3. Network results.R}, line 216). If the control were replaced by its exact English-only complement on the same analysis rows, with unchanged regression weights, the treatment coefficients, the coefficients of the interaction with composition and their robust standard errors would not change; this follows from the identity above for the additive regressions, and I checked it numerically only for the two regressions for peer conflict. The row ``Home language'' of Fig.~S2 is computed from the three-category answer (0, 1, 2), not from this indicator, and the correction does not bear on it. The row ``Home language is English'' of Table S2 is coded separately, with English only as 1. The authors' own data file is not public, and ICPSR notes that some variables in the public release were changed; the agreement of all 30 printed entries does not show that the public file is identical to it."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E007", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Six lines of text from page 6 of the SI Appendix. Two red boxes mark the parenthesis (schools with less than 550 students) across a line break, after the words no more than 64 seed-eligible students. The next sentence reads: For schools with between 200 and 550, we scaled linearly between 40 and 64 seed-eligible students, rounding to the nearest 4.", "caption": "SI Appendix, p. 6 (PDF p. 6), first paragraph. The marked parenthesis attaches the cap of 64 seed-eligible students to schools with less than 550 students; the sentence after it scales the number from 40 at 200 students to 64 at 550.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T12", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T12"}], "headline": "The parenthesis attached to the cap of 64 seed-eligible students names the wrong range of school sizes: the upper branch of the rule applies at 550 students and holds for larger schools.", "id": "P037-E008", "location": "SI Appendix, “Study design”, p. 6 (PDF p. 6 of the SI Appendix), first paragraph, sentence beginning “such that we had no fewer than 40”; compare the main text, p. 567 (PDF p. 2)", "pdf_page": 6, "printed_page": "6", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T12"], "text": {"as_printed": "``such that we had no fewer than 40 seed-eligible students (schools with fewer than 200 students), and no more than 64 seed-eligible students (schools with \\hl{less than} 550 students). For schools with between 200 and 550, we scaled linearly between 40 and 64 seed-eligible students, rounding to the nearest 4.''", "corrected": "``\\ldots and no more than 64 seed-eligible students (schools with \\hl{at least} 550 students).''\n\nWith explicit endpoints, the published rule is: 40 seed-eligible students for a school of at most 200 students, 64 for a school of at least 550 students, and for sizes in between the linear interpolation between 40 and 64, rounded to the nearest multiple of four.", "evidence": "I checked the page, recalculated the printed rule and checked the public data (ICPSR 37070 V2). Under the printed rule, a school of 100 or of 200 students has 40 seed-eligible students, a school of 375 has 52, and a school of 550 or of 715 has 64.\n\nThe authors' analysis code does not compute the eligible group; it reads eligibility from the data. I ran its filtering and classification lines with an adapter on the public data, retaining 24,191 students in 56 schools and 728 seeds in the 28 treatment schools, matching the published counts. The public school-size variable INCSIZE reproduces all 14 block means of school size in Table S1. With that variable, each of the 14 schools of at least 550 students has 64 seed-eligible students, and each of the nine schools of at most 200 students has 40. Of the 33 schools in between, 14 have the number that the printed rule predicts and 19 do not.", "problem": "The sentence after the parenthesis scales the number of seed-eligible students linearly from 40 at 200 students to 64 at 550. The upper branch fixes the number at 64 for schools with at least 550 students; rounding can also give 64 below 550. Table S1 has a block of schools whose mean size is 715. The parenthesis should name the schools with at least 550 students.", "what_survives": "The rule is recoverable from the sentence after the parenthesis, and correcting the wording leaves the released analysis unchanged. The released analysis code reads eligibility as given and does not reconstruct its selection. The checks on the public data do not establish an implementation or randomization failure, but do not confirm the printed interpolation for every school of intermediate size. The selection routine is absent, and I could not identify the enrollment figures used at selection; ICPSR reports changes to some released variables. The main text's ``designed to represent 15\\% of the school population'' (p.~567) states the aim of the design and needs no change. The seed-eligible students are not 15\\% of each school, since 40 of 200 students is 20\\% and 64 of 550 is $11.6\\%$; on the public data, using INCSIZE as the denominator and averaging school shares across the 28 treatment schools, the printed rule gives $15.0\\%$ and the eligible groups in the data $14.8\\%$."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E008", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Fig. 3, four panels with its caption at the right. Each panel shows four dots with dotted vertical 95 percent confidence intervals, for the exposures 'No exposure from seed or school', 'Treatment school, no exposure to seed student', 'Exposed to non-referent seed student' and 'Exposed to social referent seed student'. Red boxes mark the rightmost dot of panel B, which lies slightly above 0.0, and the rightmost dot of panel D, which lies a little below 0.2.", "caption": "Fig. 3 and its caption, p. 570 (PDF p. 5). The two marked dots are the estimates for students exposed to a social referent seed in panel B (perceived social norm against conflict) and panel D (disciplinary records of peer conflict), where the mean gap from Table S5 is largest within each of those panels.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E009.png"}, {"alt": "Crop of Table S5, 'Causal social influence effects from seed students', with columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict and four exposure rows. Red boxes mark, in the row 'Exposed to social referent seed student', the Prescriptive norms entry 0.007 with standard error (0.113) and the Peer conflict entry 0.170 with standard error (0.037). A third red box marks the words 'Figure 4' in the last sentence of the note, 'A graphical version of this table is presented in Figure 4 in the main text of the paper.'", "caption": "SI Appendix, Table S5, p. 40 (supplement PDF p. 40), shown upright; the table is printed sideways on the page. The marked entries are the estimates and standard errors for students exposed to a social referent seed, for prescriptive norms and for peer conflict. The marked words of the note are its reference to the corresponding figure.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Fig. 3 and Table S5 give different values for the same exposure means and standard errors: 10 of the 16 means and 11 of the 16 standard errors differ by more than rounding and the drawing allow, and the authors' code, run on the public data, reproduces the figure and not the table.", "id": "P037-E009", "location": "Fig. 3, panels A–D, p. 570 (PDF p. 5); SI Appendix, Table S5, p. 40 (supplement PDF p. 40)", "pdf_page": 5, "printed_page": "570", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lrr}{Table S5 (SI p.~40), first row, two of its four columns; standard errors in parentheses.}\n & Prescriptive norms & Peer conflict \\\\\nExposed to social referent seed student & \\hl{0.007} & \\hl{0.170} \\\\\n & \\hl{(0.113)} & \\hl{(0.037)} \\\\\n\\end{audittable}\n\nTable S5, note: ``A graphical version of this table is presented in \\hl{Figure 4} in the main text of the paper.''\n\nFig.~3 (p.~570) plots the four exposure means for each outcome with 95\\% confidence intervals and prints no numbers.", "corrected": "Table S5 should be regenerated from the estimates that Fig.~3 plots. On the public data, the authors' table script gives:\n\n\\begin{audittable}{lrr}{Table S5 as printed, and as the authors' table script gives it on the public data (ICPSR 37070 V2): estimate (standard error). The outcomes are the four columns of the table, Friends talk about conflict (Talk), Prescriptive norms (Norms), Wristband and Peer conflict (Conflict); the exposures are its four rows. Highlighted values differ from the printed ones.}\nOutcome and exposure & As printed & On the public data \\\\\nTalk, social referent seed & 0.747 (0.032) & 0.747 (\\hl{0.033}) \\\\\nTalk, non-referent seed & 0.696 (0.051) & \\hl{0.704} (\\hl{0.050}) \\\\\nTalk, treatment school, no seed & 0.695 (0.084) & 0.695 (\\hl{0.080}) \\\\\nTalk, no seed or school & 0.624 (0.020) & \\hl{0.625} (0.020) \\\\\nNorms, social referent seed & 0.007 (0.113) & \\hl{0.030} (\\hl{0.106}) \\\\\nNorms, non-referent seed & $-0.468$ (0.163) & \\hl{$-0.456$} (\\hl{0.151}) \\\\\nNorms, treatment school, no seed & $-0.825$ (0.254) & \\hl{$-0.823$} (\\hl{0.258}) \\\\\nNorms, no seed or school & 0.141 (0.075) & \\hl{0.127} (\\hl{0.071}) \\\\\nWristband, social referent seed & 0.232 (0.012) & \\hl{0.229} (0.012) \\\\\nWristband, non-referent seed & 0.124 (0.016) & \\hl{0.123} (0.016) \\\\\nWristband, treatment school, no seed & 0.095 (0.025) & \\hl{0.096} (\\hl{0.026}) \\\\\nWristband, no seed or school & 0.002 (0.008) & \\hl{0.004} (0.008) \\\\\nConflict, social referent seed & 0.170 (0.037) & \\hl{0.183} (\\hl{0.035}) \\\\\nConflict, non-referent seed & 0.190 (0.052) & \\hl{0.185} (\\hl{0.049}) \\\\\nConflict, treatment school, no seed & 0.152 (0.072) & \\hl{0.148} (\\hl{0.077}) \\\\\nConflict, no seed or school & 0.200 (0.027) & \\hl{0.196} (\\hl{0.026}) \\\\\n\\end{audittable}\n\nThese values come from the public data and not from the authors' own files. \\hl{Before rounding,} they agree with all 32 values of Fig.~3 within the precision of the drawing. Their samples are 2,457, 2,273, 2,847 and 2,495 students for the four outcomes, and the note's ``subpopulation of 2,451 students'' does not describe them (\\fid{P037-N025}). If the authors' files give another final output, the figure and the table should both be regenerated from it and the difference explained. In the note, ``Figure 4'' should read ``\\hl{Figure 3}'' (\\fid{P037-E010}).", "evidence": "The evidence is a recalculation from the published PDF and a run of the authors' code on the public data.\n\nFrom the PDF, I read each dot and interval of Fig.~3 from the vector drawing and calibrated it on the grid lines of its panel; the standard error is the length of the interval divided by $2 \\times 1.96$. A value counts as a mismatch only if the figure, with $\\pm 0.1$ PDF point allowed for each coordinate, and the table, with rounding to three decimals allowed, cannot coincide. By this count, 10 of the 16 means and 11 of the 16 standard errors do not match.\n\nOn the public data (ICPSR 37070 V2), I ran the authors' code for the four outcomes of Fig.~3: 10,000 simulated assignments for the exposure probabilities, the first 1,000 of them for the standard errors, the seed of the code, the random sampler that R used before version 3.6, the sign convention for principal components that it used before version 3.5, and social referents defined by indegree. All 16 unrounded means and all 16 unrounded standard errors agree with Fig.~3 within the precision of the drawing; the largest differences are about $0.00002$ between a mean and its plotted interval's midpoint and $0.000004$ between a standard error and the value inferred from its interval's width. Rounded to three decimals as the table script rounds them, 14 of the 16 means and 12 of the 16 standard errors differ from the printed Table S5. The table below shows panel B and the referent row of panel D.\n\n\\begin{audittable}{lrrrrrr}{Means and standard errors read from Fig.~3, computed by the authors' code on the public data, and printed in Table S5 (S5).}\nPanel and exposure & Fig.~3 mean & Public data & S5 mean & Fig.~3 SE & Public data & S5 SE \\\\\nB, control school & $0.127$ & $0.127$ & $0.141$ & $0.071$ & $0.071$ & $0.075$ \\\\\nB, treatment school, no seed & $-0.823$ & $-0.823$ & $-0.825$ & $0.258$ & $0.258$ & $0.254$ \\\\\nB, nonreferent seed & $-0.456$ & $-0.456$ & $-0.468$ & $0.151$ & $0.151$ & $0.163$ \\\\\nB, social referent seed & $0.030$ & $0.030$ & $0.007$ & $0.106$ & $0.106$ & $0.113$ \\\\\nD, social referent seed & $0.183$ & $0.183$ & $0.170$ & $0.035$ & $0.035$ & $0.037$ \\\\\n\\end{audittable}\n\nThe public data are not the authors' own analysis files, which are not available, and ICPSR states that some variables in the release were changed. The run shows that the released code and the public data reproduce the figure. It does not show which output the authors took as final, or why the printed table differs.", "problem": "Table S5 and Fig.~3 are two displays of one set of estimates: the covariate-adjusted mean of each outcome under each of the four exposure conditions, with standard errors computed under constant effects. The note to the table calls the figure its graphical version, and the figure it means is Fig.~3 (\\fid{P037-E010}). In the authors' code the two displays have one source. The script that draws the figure plots each mean with an interval of $\\pm 1.96$ standard errors and saves the same means and standard errors, and the script that writes the table reads the saved values and rounds them, with no further estimation. The two displays should then agree to the rounding of the table. They do not. For students exposed to a social referent seed, the mean is about $0.030$ in Fig.~3B against $0.007$ in the table, and about $0.183$ in Fig.~3D against $0.170$. The interval in Fig.~3B corresponds to a standard error of about $0.106$, against $0.113$. At least one of the two displays does not show the output it is said to show, and the publication does not say which.", "what_survives": "In every panel the four point estimates have the same order in the figure and in the table, including the larger means under referent exposure in panels A to C, and no stated conclusion is shown to reverse. The largest gap, $0.007$ against $0.030$, is a fraction of the standard error of that mean, which is $0.11$. Since the marginal standard errors of the table do not determine the variance of a contrast, the mismatch does not show that a reported test changes. For the contrast between referent and nonreferent exposure on norms, the difference on the public data is $0.486$, with standard error $0.189$ and $P = 0.0101$, where the simulation error of $P$, conditional on the estimated exposure probabilities and fitted residuals, is about $0.0017$. The crossing of $0.01$ is within that error, and it does not establish that the printed $P < 0.01$ changes. The school-level results of Figs.~1 and 2 and Tables S6 and S7 are separate estimates."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E009", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of a paragraph of five lines. A red box marks 'Points 1 - 6' in the sentence 'Points 1 - 6 follow a rough chronological introduction of the program’s principles'.", "caption": "SI Appendix, p. 2. The marked range should read 1–5: the list that follows has five principles.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010.png"}, {"alt": "Crop of the note of six lines under Table S5. A red box marks 'Figure 4' in the last sentence, 'A graphical version of this table is presented in Figure 4 in the main text of the paper.'", "caption": "SI Appendix, note to Table S5, p. 40. The page is printed sideways and is shown here turned upright. The marked reference should be to Figure 3.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-b.png"}, {"alt": "Crop of four lines of text. A red box marks 'Table 1' in the sentence 'See Table 1 for variable descriptions.'", "caption": "SI Appendix, p. 25, text under Fig. S2. The marked reference should be to Table S1.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-c.png"}, {"alt": "Crop of references 30 and 31. A red box marks 'Hartwick H' in 'Barki H, Hartwick H (2004) Conceptualizing the construct of interpersonal conflict. Int J Conflict Manage 15(3):216-244.'", "caption": "Reference list, p. 571 (PDF p. 6), reference 31. The marked author is J. Hartwick.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-d.png"}, {"alt": "Crop of references 38 and 39. A red box marks '(2011)' in 'Marwick A, Boyd D (2011) It's just drama: Teen perspectives on conflict and aggression in a networked era. J Youth Stud 17(9):1187-1204.'", "caption": "Reference list, p. 571 (PDF p. 6), reference 39. The marked year should be 2014.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-e.png"}, {"alt": "Crop of references 47 to 49. A red box marks '47(1)' in 'Horvitz DG, Thompson DJ (1952) A generalization of sampling without replacement from a finite universe. J Am Stat Assoc 47(1):663-685.'", "caption": "Reference list, p. 571 (PDF p. 6), reference 48. The marked issue number should be 260.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-E010-f.png"}], "formal_release": null, "formal_statements": [], "headline": "Six slips of cross-reference and citation: the supplement lists five principles, not six; the graph of Table S5 is Fig. 3 of the article; the variable descriptions are in Table S1; and references 31, 39 and 48 each have one wrong detail.", "id": "P037-E010", "location": "SI Appendix, p. 2 (PDF p. 2 of the supplement), paragraph beginning “Each principle described below”; also SI p. 25 (text under Fig. S2), SI p. 40 (note to Table S5), and the reference list of the article, p. 571 (PDF p. 6), references 31, 39 and 48", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "SI p.~2: Points 1 - \\hl{6} follow a rough chronological introduction of the program’s principles\n\nSI p.~40, note to Table S5: A graphical version of this table is presented in Figure \\hl{4} in the main text of the paper.\n\nSI p.~25: See Table \\hl{1} for variable descriptions.\n\np.~571: 31. Barki H, Hartwick \\hl{H} (2004) Conceptualizing the construct of interpersonal conflict. \\emph{Int J Conflict Manage} 15(3):216--244.\n\np.~571: 39. Marwick A, Boyd D (\\hl{2011}) ‘It’s just drama’: Teen perspectives on conflict and aggression in a networked era. \\emph{J Youth Stud} 17(9):1187--1204.\n\np.~571: 48. Horvitz DG, Thompson DJ (1952) A generalization of sampling without replacement from a finite universe. \\emph{J Am Stat Assoc} 47(\\hl{1}):663--685.", "corrected": "SI p.~2: Points 1--\\hl{5} follow a rough chronological introduction of the program’s principles\n\nSI p.~40, note to Table S5: A graphical version of this table is presented in Figure \\hl{3} in the main text of the paper.\n\nSI p.~25: See Table \\hl{S1} for variable descriptions.\n\np.~571: 31. Barki H, Hartwick \\hl{J} (2004) Conceptualizing the construct of interpersonal conflict. \\emph{Int J Conflict Manage} 15(3):216--244.\n\np.~571: 39. Marwick A, Boyd D (\\hl{2014}) ‘It’s just drama’: Teen perspectives on conflict and aggression in a networked era. \\emph{J Youth Stud} 17(9):1187--1204.\n\np.~571: 48. Horvitz DG, Thompson DJ (1952) A generalization of sampling without replacement from a finite universe. \\emph{J Am Stat Assoc} 47(\\hl{260}):663--685.", "evidence": "I checked the pages and, for the three references, the publishers' bibliographic information. The page for reference 31 names Jon Hartwick as second author. The page for reference 39 reads \\emph{Journal of Youth Studies}, volume 17, 2014, issue 9, pages 1187--1204. The page for reference 48 reads \\emph{Journal of the American Statistical Association}, volume 47, 1952, issue 260, pages 663--685.\n\nFor the two cross-references of the supplement I also read the authors' analysis scripts and ran their graph and table assembly steps with distinct test values in place of estimates. All sixteen mean inputs and sixteen standard-error inputs pass to Table S5 with the exposure order reversed. The scripts for Fig.~S2 and for Table S1 give fifteen corresponding covariate labels. The targets are Fig.~3 and Table S1. This check follows the inputs and labels, and it reproduces no estimate.", "problem": "The supplement lists five guiding principles; the list ends with principle 5 on SI p.~4. The article has three figures, and the graph of Table S5 is Fig.~3. The fifteen covariates of Fig.~S2 are described in the note to Table S1 (SI p.~36). In the reference list, the second author of reference 31 is J.~Hartwick, reference 39 was published in 2014, and reference 48 is in issue 260 of volume 47.", "what_survives": "Each intended target can be identified, and no estimate or conclusion depends on these items. The reference on SI p.~13 to ``the interaction model in (4)'' is not counted as an error, since equation (4) on SI p.~15 repeats the interaction model (2). This finding says nothing about the correctness of the numerical results."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E010", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of seven lines of the abstract in bold type. A red box marks 30 percent in the sentence 'Compared with control schools, disciplinary reports of student conflict at treatment schools were reduced by 30 percent over 1 year.'", "caption": "Abstract, p. 566 (PDF p. 1). The marked figure is the reduction in disciplinary reports, which the correction of 2018 replaces by 25%.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N001.png"}, {"alt": "Crop of six lines of text. A red box marks 30 percent in the passage 'In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools, a 30 percent reduction in peer conflict reports.'", "caption": "Results, p. 569 (PDF p. 4), left column, first paragraph. The marked 30% is the quotient $0.06/0.20$ of the two rounded numbers in the same passage.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N001-b.png"}, {"alt": "Crop of seven lines of text. A red box marks 30 percent in 'our intervention reduced overall levels of disciplinary reports of peer conflict by an estimated 30 percent over 1 y.' The next sentence reads 'To put this in perspective, our estimates imply that the intervention reduced the total number of disciplinary events from 2,695 events to 2,012 events across the 11,938 students in treatment schools.'", "caption": "Discussion, p. 571 (PDF p. 6), left column, first paragraph. Three lines below the marked figure are the totals 2,695 and 2,012, which give a reduction of 25.3%.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N001-c.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T6", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T6"}, {"id": "P037-T17", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T17"}], "headline": "The reduction in disciplinary reports of peer conflict is 25%, not 30%; the authors' correction of 2018 makes this change in the four places where 30% is printed.", "id": "P037-N001", "location": "Abstract, p. 566 (PDF p. 1), line 11; also the Significance statement, p. 566, line 8; Results, p. 569 (PDF p. 4), left column, first paragraph; Discussion, p. 571 (PDF p. 6), left column, first paragraph", "pdf_page": 1, "printed_page": "566", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T17", "P037-T6"], "text": {"as_printed": "Abstract: Compared with control schools, disciplinary reports of student conflict at treatment schools were reduced by \\hl{30\\%} over 1 year.\n\nSignificance: \\ldots\\ our intervention reduced overall levels of conflict by an estimated \\hl{30\\%}.\n\nResults, p.~569: In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools, a \\hl{30\\%} reduction in peer conflict reports.\n\nDiscussion, p.~571: \\ldots\\ our intervention reduced overall levels of disciplinary reports of peer conflict by an estimated \\hl{30\\%} over 1 y. To put this in perspective, our estimates imply that the intervention reduced the total number of disciplinary events from 2,695 events to 2,012 events across the 11,938 students in treatment schools.", "corrected": "Abstract: Compared with control schools, disciplinary reports of student conflict at treatment schools were reduced by \\hl{25\\%} over 1 year.\n\nSignificance: \\ldots\\ our intervention reduced overall levels of conflict by an estimated \\hl{25\\%}.\n\nResults, p.~569: In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year \\hl{(the raw control-school mean in the 56-school calculation, rounded)}. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools \\hl{(a covariate-adjusted decrease, rounded)}, a \\hl{25\\%} reduction in peer conflict reports \\hl{when the mean predictions under control and under treatment are both taken, unrounded, from the covariate-adjusted interaction regression and averaged over the same 49 schools}.\n\nDiscussion, p.~571: \\ldots\\ our intervention reduced overall levels of disciplinary reports of peer conflict by an estimated \\hl{25\\%} over 1 y.\n\nThe replacement of 30 by 25 in the four places is the correction of 2018. The three phrases added to the Results are mine. They state the basis of each number, since $0.06/0.20$ is not $25\\%$. On the public data the two mean predictions are $0.2225$ and $0.1660$, and the reduction is $25.4\\%$.", "evidence": "The evidence is a recalculation from the published numbers, a check of the pages against the correction of 2018 (PNAS 115(15):E3598), and a calculation with the authors' analysis scripts on the public release of the data (ICPSR 37070, version 2).\n\n\\emph{The published numbers.} The Discussion estimates 2,695 disciplinary events without the intervention and 2,012 with it, for the same 11,938 students. The reduction is\n\\[\n\\frac{2695-2012}{2695}=\\frac{683}{2695}\\approx 25.3\\%.\n\\]\nThis lies between $24.5\\%$ and $25.5\\%$, and it rounds to $25\\%$. It equals $30\\%$ only if $10\\cdot 683=3\\cdot 2695$, that is, $6830=8085$. If each total was rounded to a whole number of events, the reduction lies between $1364/5389\\approx 25.31\\%$ and $152/599\\approx 25.38\\%$. The quotient $0.06/0.20$ equals $30\\%$ exactly, and the printed figure is not a slip of division.\n\n\\emph{The scripts and the public data.} The authors' script \\texttt{4. School results-with-correction.R} computes the percentages of the correction. Line 208 fits the covariate-adjusted regression with the interaction of treatment and seed-group composition. Lines 253--256 average its predictions under control and under treatment over the 49 schools of the analysis, and line 286 divides their difference by the average prediction under control. I ran these lines on the public data, with an adapter that maps its variables to the inputs the scripts expect. The average predictions are $0.222537$ under control and $0.165975$ under treatment, and\n\\[\n\\frac{0.222537-0.165975}{0.222537}\\approx 25.42\\%.\n\\]\nThe same calculation gives $28.68\\%$ for the quotient of the adjusted difference and the raw control mean $0.197241$ (line 270), and $22.87\\%$ for the quotient of the raw difference and the raw control mean (line 289). These round to the 25\\%, 29\\% and 23\\% of the correction. The raw means use all 56 schools, as discussed in \\fid{P037-N009}.\n\nThe regression of Table S6 is a different fit (line 162, without the interaction). On the public data its coefficient is $-0.057272$, and the difference of the average predictions of the interaction regression is $-0.056562$. Both round to the printed $-0.06$. On the public data, I reproduce the Discussion's totals from the regression of Table S6: its average predictions over the 49 schools, multiplied by 11,938, are $2695.4$ and $2011.7$, a reduction of $25.37\\%$. The three reductions $25.34\\%$ (printed totals), $25.37\\%$ and $25.42\\%$ are different quantities, and each rounds to $25\\%$.\n\n\\begin{audittable}{lrr}{The reduction in disciplinary reports on each basis. The second column is from the article and the correction of 2018: the first row is the printed $0.06/0.20$, the third row is the correction, and the last row is $(2695-2012)/2695$. The third column is computed with the authors' script on the public data (ICPSR 37070, version 2), not on the authors' own files; its last row uses the totals $2695.4$ and $2011.7$.}\nBasis & As published & On the public data \\\\\n\\hline\nAdjusted effect over raw control mean, both rounded & 30\\% & 30\\% \\\\\nThe same quotient, unrounded & 29\\% & 28.68\\% \\\\\nAdjusted effect over adjusted control mean & 25\\% & 25.42\\% \\\\\nRaw effect over raw control mean & 23\\% & 22.87\\% \\\\\nTotals in the Discussion & 25.3\\% & 25.37\\% \\\\\n\\end{audittable}\n\nOn the public data, the authors' rule for selecting records keeps 24,191 students in 56 schools, 49 schools enter the regressions of disciplinary reports, and the treatment schools have 11,938 students. The recomputed quantities agree with the corresponding published values at their printed precision. The authors' own input files are not public, and ICPSR states that some variables of the release were changed. The values above are values on the public data. They are not the unrounded values of the authors' files.", "problem": "The article reports the effect on disciplinary reports as a 30\\% reduction in four places. The figure is the quotient $0.06/0.20$ of the two numbers in the Results. According to the correction of 2018, the first is the covariate-adjusted estimate of the effect and the second is the unadjusted mean of the control schools, and both are rounded. The numerator and the denominator are on different bases. The correction states that the same quotient is 29\\% without rounding, that the reduction is 25\\% when the effect and the control mean are both covariate-adjusted, and that it is 23\\% when both are unadjusted. From the estimated totals in the Discussion, I also obtain 25\\% after rounding. A citation of the original figure overstates the reduction by five percentage points, one sixth of the printed 30\\%.", "what_survives": "This correction leaves the estimate of 0.06 fewer disciplinary reports per student (Table S6: $-0.06$, standard error $0.03$), the totals in the Discussion, the direction of the effect and the other analyses unchanged. The reduction is five percentage points smaller than printed. The correction requires no new causal assumption or estimator. The correction of 2018 also contains a sentence on statistical significance, which is the subject of \\fid{P037-N008}; the school-level tests are the subject of \\fid{P037-N006}."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N001", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of text. Red boxes mark the sentence 'Each school's entire school body took a survey at the same time on a given day (n = 24,191 students).'", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked sentence gives 24,191 as the number of students who took the survey.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N003.png"}, {"alt": "Crop of five lines of the note under Table S1. Red boxes mark the clause 'school size is the average school population as measured by the number of students who took the pre-randomization student surveys'.", "caption": "SI Appendix, note to Table S1, p. 36. The page is printed sideways and is shown here turned upright. The marked clause defines school size as the number of students who took the survey before randomization.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N003-b.png"}, {"alt": "Crop of the first five columns of Table S1: School Block, percent White (self-ID), Home language, Friends like house and School size. Red boxes mark the column heading 'School size' and the Total Sample mean 405.93. The School size means of blocks 1 to 14 are 368.25, 389.75, 226.50, 715.00, 122.00, 528.50, 335.50, 492.75, 320.25, 318.50, 512.75, 468.75, 481.25 and 403.25.", "caption": "SI Appendix, Table S1, p. 36, first five columns, turned upright. For each block of four schools the marked column gives the mean school size and, on the line below, its dispersion. The fourteen means sum to 5,683. The marked entry 405.93 is the mean over the 56 schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N003-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The article’s 24,191 is reproduced on the public data as the number of student records in the analysis, including students without a baseline survey response; it is not an established count of students who took the survey before randomization, and the school-size means of Table S1 imply a total of 22,732.", "id": "P037-N003", "location": "Methods and Materials, p. 567 (PDF p. 2), right column, paragraph beginning “We measured social connections”; compare SI Appendix, Table S1 and its note, p. 36", "pdf_page": 2, "printed_page": "567", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Main text, p.~567: Each school's \\hl{entire school body took a survey} at the same time on a given day \\hl{($n = 24{,}191$ students)}.\n\nSI p.~36, note to Table S1: ``school size'' is the average school population \\hl{as measured by the number of students who took the pre-randomization student surveys};", "corrected": "Main text, p.~567: \\hl{A school-wide baseline survey was administered on a given day at each school. The analysis sample contained 24,191 student records, including students without a baseline survey response.}\n\nOn the same page, ``The survey that all students took at the beginning of the year'' should read ``\\hl{The baseline survey}''.\n\nSI p.~36, note to Table S1: ``school size'' is \\hl{the variable \\texttt{incsize}, whose published block means imply a total of 22,732 over the 56 schools. The public release defines it as the school population included in the study, limited to the participating grades where applicable. Whether it also equals the number of students who took the survey cannot be established without the authors' files or a documented indicator of response.}", "evidence": "I recalculated the total from Table S1, read the authors' analysis scripts, and ran the parts that bear on these counts on the public data.\n\n\\emph{Table S1.} Each of the 14 blocks has four schools, and the implied number of survey takers is four times the sum of the block means in the column ``School size''. The means are 368.25, 389.75, 226.50, 715.00, 122.00, 528.50, 335.50, 492.75, 320.25, 318.50, 512.75, 468.75, 481.25 and 403.25. Their sum is $5683$, and\n\\[\n4\\cdot 5683=22732,\\qquad 24191-22732=1459 .\n\\]\nRounding does not account for the difference. Each printed mean is a multiple of $1/4$, and errors of at most $0.005$ in the 14 means change the total by at most $4\\cdot 14\\cdot 0.005=0.28$. The total agrees with the mean of the Total Sample row, since $22732/56\\approx 405.93$.\n\n\\emph{The scripts and the public data.} The script \\texttt{1. Sanitize and generate perms, probs.R} (lines 33--35) keeps the records with a known school block and a roster number other than 999. On the public data this rule keeps 24,191 records in 56 schools. The script \\texttt{7. Blocking, randomization, and summary statistics.R} takes the school size of Table S1 from the variable \\texttt{incsize}. Its block means on the public data equal the fourteen printed means, and its total is 22,732. Of the 24,191 records in the public data, 22,810 have a survey type (\\texttt{SURTYP}) recorded and 22,829 have at least one observed baseline field from \\texttt{GENDER} through \\texttt{CRUNT}, excluding \\texttt{AGE\\_NEW}, \\texttt{GENM} and \\texttt{GRM}; 1,362 have none under this criterion. The ratio $22732/24191\\approx 94.0\\%$ is a ratio of the two published counts. It is not an established response rate.\n\nIn a reconstruction on the public data, the network has all 24,191 records as vertices. The 1,362 records without an observed baseline answer under this criterion have no outgoing nomination links. Of these, 817 are nominated by other students, and 146 have a seed among their peers when a nomination in either direction counts (see \\fid{P037-N032}).", "problem": "The sentence says that every student took the survey and reports their number as 24,191. Applying the authors' sample-selection code to the public data gives 24,191 student records kept for the analysis. The rule that keeps them asks for a known school block and a roster number other than 999, and it does not ask for a baseline response. Table S1 defines school size as the number of students who took the survey before randomization, and its block means imply 22,732 such students, 1,459 fewer. SI p.~6 refers to ``students who failed to take the survey'', and SI p.~5 reports 24,191 as the total sample of the study. The count of the main text is not an established count of survey takers.\n\nThe original number of students who answered the baseline survey is not established. On the public release of the data (ICPSR 37070, version 2), the school-size variable of Table S1 sums to 22,732, and the codebook of the release defines it as the school population included in the study, not as a count of completed surveys. Two indicators of a baseline response in the public data count 22,810 and 22,829 records.", "what_survives": "No estimate is shown to change. The design of 56 schools in 14 blocks is not contradicted, and the Abstract's ``56 schools with 24,191 students'' need not count completed surveys. The public data agree with the published 24,191, with the 56 schools and with all fourteen school-size means of Table S1. The authors' own input files are not public, ICPSR reports changes to some variables, and the number of completed baseline surveys in the original files is not established. The code retains students without a baseline response in the network and includes their available outcomes in school-level means; network regressions have further inclusion criteria. The nominations that they would have made are not observed, and I did not estimate the effect of their absence on any result."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N003", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of text. A red box marks 'seed group' in 'controlling for the proportion of students in the seed group who were social referents'.", "caption": "Methods and Materials, p. 568 (PDF p. 3), left column. The marked words should read “seed-eligible group”.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "The control variable of the interaction regression is the proportion of social referents in the seed-eligible group, not in the seed group.", "id": "P037-N005", "location": "Methods and Materials, p. 568 (PDF p. 3), left column, paragraph beginning “The randomization of schools and seeds”", "pdf_page": 3, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\ldots\\ we use linear regression interacting school-level treatment with seed group composition and controlling for the proportion of students in the \\hl{seed group} who were social referents.", "corrected": "\\ldots\\ we use linear regression interacting school-level treatment with seed group composition and controlling for the proportion of students in the \\hl{seed-eligible group} who were social referents.", "evidence": "I checked the page against the supplement and against the authors' analysis scripts. Equations (2), (4) and (6) on SI pp.~13, 15 and 16 contain $\\beta C_{ij}D_{ij}$ and $\\gamma C^E_{ij}$, and the note to Table S7 (SI p.~42) says that a control is included for the expected value of the seed group composition.\n\nThe script \\texttt{1. Sanitize and generate perms, probs.R} (lines 143--146) computes two proportions of social referents for each school, one among students assigned to the seed group and one among seed-eligible students. The script \\texttt{4. School results-with-correction.R} (lines 140--141 and 203--209) interacts treatment with the first and uses the second as the control. Using an adapter with these scripts on the public data (ICPSR 37070, version 2), I confirmed the seed-eligible control in both regression specifications for the three direct outcomes; the two proportions differ in 43 of 56 schools.", "problem": "As printed, the sentence names the same variable twice: the proportion of social referents in the seed group is the seed group composition. The control in the regression is the proportion of social referents in the seed-eligible group, $C^E_{ij}$. The next sentence of the paragraph contrasts the two proportions.", "what_survives": "A word was dropped, and no estimate changes on account of it, since the released code uses the control that the supplement specifies. The authors' own input files are not public, ICPSR warns that some released variables were changed, and this check does not reproduce every estimate. On the public data, five of the six coefficients of Table S7 that I compared agree at the printed precision, and the interaction for peer conflict is $-0.84$ where the table prints $-0.85$. The regression's causal interpretation remains conditional on the response approximation stated in the SI Appendix. That model scope is separate from this omitted-word correction: restoring the control name neither verifies nor refutes the approximation."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N005", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of seven lines of text. Red boxes mark the sentence 'We computed confidence intervals and P values using robust SEs under a normal approximation.'", "caption": "Methods and Materials, p. 568 (PDF p. 3), right column. The marked sentence describes the school-level confidence intervals and $P$ values. It does not name the robust estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006.png"}, {"alt": "Crop of seven lines of text beginning 'We first turn to the question of school climate'. Red boxes mark 'significantly reduced' and '(P < 0.05, heteroskedasticity-robust Wald test)' in the sentence 'Figs. 1 and 2 reveal that the treatment significantly reduced average levels of disciplinary reports of peer conflict in treatment compared with control schools (P < 0.05, heteroskedasticity-robust Wald test).'", "caption": "Results, p. 569 (PDF p. 4), left column, first paragraph. The marked words report the main effect on disciplinary reports as significant at the 0.05 level.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-b.png"}, {"alt": "Crop of seven lines of the legend of Fig. 2. Red boxes mark '(P = 0.07, heteroskedasticity-robust Wald test)' in the sentence 'We find that a higher proportion of social referent seeds assigned to the anticonflict group increases the size of the overall school effect on disciplinary reports of peer conflict (P = 0.07, heteroskedasticity-robust Wald test).'", "caption": "Legend of Fig. 2, p. 569 (PDF p. 4). The marked words give the $P$ value of the interaction between treatment and the proportion of social referent seeds.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-c.png"}, {"alt": "Crop of Table S6, 'Average school-level climate effects caused by anti-conflict groups', with the columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict, and its note. The row School level main effect reads 0.09, minus 0.01, 0.16 and minus 0.06. Red boxes mark the entry (0.03) of the row Heteroskedasticity Robust SE and the entry (0.03) of the row Wild Bootstrap SE in the column Peer conflict. The note says that wild bootstrap standard errors were generated using 5,000 replicates, multiplying residuals by draws from the standard normal distribution.", "caption": "SI Appendix, Table S6, p. 41. The page is printed sideways and is shown here turned upright. The two marked standard errors of the peer-conflict column estimate the same conditional standard deviation: the wild bootstrap of the authors' script has conditional variance equal to the robust variance and adds no finite-sample adjustment. Its simulated standard error is subject to Monte Carlo variation.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-d.png"}, {"alt": "Dot plot with five rows and a horizontal axis from 0 to 18 percent; a dashed vertical line marks the nominal level of 5 percent. Each row has a blue circle for the main effect of treatment (Table S6) and an orange square for the interaction of treatment and referent share (Table S7). HC0 with a normal reference, the test of the article, under normal null errors: 13.7 and 16.0 percent. The same test under blocked re-randomization: 13.0 and 12.4 percent. HC1 with a t reference: 5.2 and 6.2 percent. HC2 with a t reference: 5.4 and 5.7 percent. HC3 with a normal reference: 2.0 and 1.9 percent. A note says 49 schools, 19 and 20 estimated coefficients, 100,000 draws of normal null errors and 30,000 draws of blocked re-randomization.", "caption": "Share of draws in which a true null hypothesis is rejected at the nominal 5% level, on the school-level design reconstructed from the public data (ICPSR 37070, version 2). The first two rows are the test of the article. The other rows are variants that are not established as calibrated for the hypotheses of the article. Values are properties of the two null experiments described under Evidence, not rejection probabilities for the authors' own files.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N006-rejection.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T18", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T18"}, {"id": "P037-T37", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T37"}], "headline": "The HC0 normal-reference school tests reject a true null more often than their nominal 5% level in specified null experiments on the public-data design; a calibrated replacement for the original hypotheses remains unestablished.", "id": "P037-N006", "location": "Methods and Materials, p. 568 (PDF p. 3), right column, sentence beginning “We computed confidence intervals”; Results, p. 569 (PDF p. 4), first paragraph; legend of Fig. 2, p. 569; SI Appendix, Tables S6 and S7, pp. 41–42", "pdf_page": 3, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": ["P037-T18", "P037-T37"], "text": {"as_printed": "Main text, p.~568: We computed confidence intervals and $P$ values using \\hl{robust SEs under a normal approximation}.\n\nResults, p.~569: Figs. 1 and 2 reveal that the treatment \\hl{significantly reduced} average levels of disciplinary reports of peer conflict in treatment compared with control schools \\hl{($P < 0.05$, heteroskedasticity-robust Wald test)}.\n\nLegend of Fig.~2, p.~569: We find that a higher proportion of social referent seeds assigned to the anticonflict group increases the size of the overall school effect on disciplinary reports of peer conflict \\hl{($P = 0.07$, heteroskedasticity-robust Wald test)}.\n\nTable S6 (SI p.~41), column ``Peer conflict'': School level main effect $-0.06$; Heteroskedasticity Robust SE $(0.03)$; Wild Bootstrap SE $(0.03)$. Table S7 (SI p.~42), column ``Peer conflict'', interaction: $-0.85$; Heteroskedasticity Robust SE $(0.46)$; Wild Bootstrap SE $(0.47)$.", "corrected": "Main text, p.~568: Confidence intervals and $P$ values use \\hl{HC0 standard errors, without a finite-sample adjustment}, under a normal approximation. \\hl{In null experiments on the public-data design with 49 schools and 19 or 20 estimated coefficients, this test rejects a true null hypothesis more often than its nominal level.}\n\nNotes to Tables S6 and S7: \\hl{The heteroskedasticity-robust standard errors are HC0. The wild bootstrap multiplies the unscaled residuals by standard normal draws. Given the data, its variance equals HC0, and it adds no finite-sample adjustment.}\n\nResults, p.~569: Figs. 1 and 2 \\hl{show lower} average levels of disciplinary reports of peer conflict in treatment compared with control schools. \\hl{On the public data (ICPSR 37070, version 2) the estimate is $-0.057$ reports per student, with HC0 standard error $0.035$ and two-sided $P=0.099$ under a normal reference; it is not significant at the 0.05 level.}\n\nLegend of Fig.~2, p.~569: A higher proportion of social referent seeds assigned to the anticonflict group is associated with a larger estimated overall school effect on disciplinary reports of peer conflict (\\hl{on the public data, coefficient $-0.84$, HC0 standard error $0.46$, $P=0.069$ under a normal reference}).\n\nBoth $P$ values come from a test that, at the nominal 5\\% level, rejects a true null hypothesis in 12 to 16\\% of draws in null experiments on the public-data design. As sensitivity calculations on the public data, HC1 with a normal reference gives $P=0.20$ and $0.16$, HC2 with a $t$ reference gives $0.21$ and $0.19$, and a re-randomization of the schools within blocks, under the hypothesis of no effect on any school, gives $0.20$ and $0.15$. None of these is established as a calibrated test of a zero average effect or of a zero interaction. The standard errors reported with the two estimates above remain HC0; neither the additional rounding digit nor the sensitivity calculations establish improved sampling-uncertainty estimation. The values on the authors' own files cannot be given here.", "evidence": "I checked the authors' script, derived the covariance identities below, calculated the estimates with the script on the public release of the data (ICPSR 37070, version 2), and examined two null experiments on the school-level design reconstructed from that release.\n\n\\emph{The bootstrap reproduces HC0.} Let $X$ be the matrix of regressors, of full column rank, and write $A=(X^{\\top}X)^{-1}X^{\\top}$, $H=XA$ and $e=(I-H)y$ for the residuals. The bootstrap of the script regresses $X\\widehat b+\\operatorname{diag}(e)v$ on $X$, where the entries of $v$ are independent with mean 0 and variance 1. Its coefficient is $b^{*}=\\widehat b+A\\operatorname{diag}(e)v$, and hence\n\\[\n\\operatorname{Var}(b^{*}\\mid X,y)=A\\operatorname{diag}(e^{2})A^{\\top}=V_{\\mathrm{HC0}} .\n\\]\n\n\\emph{HC0 is too small in this design.} Suppose that $y=Xb+u$, where, conditional on $X$, the errors $u$ are independent with mean zero and common variance $\\sigma^2$, and let $h_{ii}$ be the $i$th diagonal entry of $H$. Then $\\mathbb{E}[e_i^2\\mid X]=\\sigma^2(1-h_{ii})$, so\n\\[\n\\mathbb{E}[V_{\\mathrm{HC0}}\\mid X]=\\sigma^{2}A\\operatorname{diag}(1-h)A^{\\top},\\qquad \\operatorname{Var}(\\widehat b\\mid X)=\\sigma^{2}AA^{\\top}.\n\\]\nFor the $j$th coefficient, the expected HC0 variance divided by the true variance is\n\\[\n\\frac{\\sum_i A_{ji}^{2}(1-h_{ii})}{\\sum_i A_{ji}^{2}} .\n\\]\nOn the public data this ratio is $0.619$ for the treatment coefficient of the regression of Table S6, where the mean of $h_{ii}$ is $19/49$, and $0.556$ for the interaction coefficient of the regression of Table S7, where it is $20/49$.\n\n\\emph{The estimates on the public data.} The public-data reconstruction applies the authors' rule for selecting records, builds the network and the outcome as their scripts do, and fits their two covariate-adjusted regressions. The rule selects 49 schools, 26 control and 23 treatment, as in the article (the rule is the subject of \\fid{P037-N007}). The main effect is $-0.0573$ and the interaction coefficient is $-0.841$. The script prints estimates, standard errors and intervals, and no $P$ value; I computed each two-sided $P$ value here from the estimate and its standard error.\n\n\\begin{audittable}{lrrrr}{Standard errors and two-sided $P$ values on the public data (ICPSR 37070, version 2), 49 schools, for the main effect of treatment on disciplinary reports (estimate $-0.0573$; Table S6 prints $-0.06$ with standard error $0.03$) and for the interaction with the proportion of social referent seeds (estimate $-0.841$; Table S7 prints $-0.85$ with standard error $0.46$). The first row is the test of the article. The other rows are sensitivity calculations, not corrected values. In the last row the schools are assigned again within blocks, 30,000 times, under the hypothesis of no effect on any school.}\nVariance and reference & SE, main & $P$, main & SE, interaction & $P$, interaction \\\\\n\\hline\nHC0, normal (the article) & $0.0347$ & $0.099$ & $0.463$ & $0.069$ \\\\\nHC1, normal & $0.0444$ & $0.197$ & $0.601$ & $0.162$ \\\\\nHC2, $t$ (30 and 29 d.f.) & $0.0451$ & $0.214$ & $0.621$ & $0.186$ \\\\\nHC3, normal & $0.0594$ & $0.335$ & $0.850$ & $0.322$ \\\\\nHC0 statistic, re-randomization & & $0.202$ & & $0.152$ \\\\\n\\end{audittable}\n\n\\emph{The null experiments.} The first experiment draws independent standard normal errors as the outcome, 100,000 times for each of the two fixed designs, so that every coefficient is truly zero. The test of the article rejected at the nominal 5\\% level in 13.7\\% of draws for the main effect and in 16.0\\% for the interaction (Monte Carlo standard errors of about 0.11 percentage points). The second experiment assigns two of the four schools of each block to treatment again, 30,000 times over the 56 schools, holding the outcomes, the covariates, the draws of the seeds and the set of 49 schools fixed, as they would be if treatment had no effect on any school. The test rejected in 13.0\\% and 12.4\\% of assignments (Monte Carlo standard errors of about 0.19 percentage points). The last row of the table estimates the share of assignments in which the HC0 statistic was at least as large in absolute value as the observed one, using $(1+\\text{count})/(30000+1)$.\n\nIn the first experiment the variants also miss the nominal level. HC1 with a $t$ reference rejects in 5.2\\% and 6.2\\% of draws, HC2 with a $t$ reference in 5.4\\% and 5.7\\%, and HC3 with a normal reference in 2.0\\% and 1.9\\%. The experiment has normal errors of equal variance, and it does not show how these variants behave for the randomization of the study. The re-randomization calculation is for the hypothesis of no effect on any school, with the outcomes, the covariates, the draws of the seeds and the set of 49 schools held fixed. That hypothesis is stronger than a zero average effect, and stronger than a zero interaction when treatment has some effect.\n\n\\emph{What the printed tables alone allow.} An effect of $-0.0649$ with HC0 standard error $0.0251$ rounds to the entries $-0.06$ and $0.03$ of Table S6. After the rescaling of the standard error by $\\sqrt{49/(49-k)}$ (HC1), its two-sided normal $P$ value is $0.043$ with $k=19$ coefficients and $0.047$ with $k=20$, the two numbers of coefficients on the public data. The rescaling alone does not remove significance for every pair of values that rounds to the printed entries. With the rescaling and a $t$ reference with $49-k$ degrees of freedom, every such pair has $P>0.05$; the smallest values are $0.051$ and $0.055$.\n\n\\emph{Influence-score terminology and current use.} The raw empirical OLS influence covariance is exactly HC0. With $n$ schools, design row $x_i^{\\top}$ and $\\widehat\\phi_i=(X^{\\top}X/n)^{-1}x_i e_i=nA_{\\cdot i}e_i$,\n\\[\n\\frac{1}{n^2}\\sum_i\\widehat\\phi_i\\widehat\\phi_i^{\\top}=A\\operatorname{diag}(e^2)A^{\\top}=V_{\\mathrm{HC0}}.\n\\]\nAn identified current DoubleML implementation also uses an unadjusted empirical influence-score second moment: its nonclustered variance helper computes $\\operatorname{mean}(\\psi_i^2)/(N\\operatorname{mean}(\\psi'_i)^2)$, and its official variance guide uses normal inference under regularity conditions. The sources are the \\href{https://docs.doubleml.org/stable/guide/se_confint.html}{captured stable guide} and \\href{https://github.com/DoubleML/doubleml-for-py/blob/6884b5f9816f180324d854a4d81b22e9d334f5e0/doubleml/utils/_estimation.py}{the pinned Python source}. They establish documented continued use, without measuring its prevalence or identifying the stable documentation with a particular numeric package release. Cross-fitted DoubleML scores generally differ from these in-sample OLS residuals; their shared second-moment form transfers no finite-sample calibration to this school design.\n\nThe corrected passage below retains HC0. Reporting its public-data main-effect standard error as $0.035$ rather than $0.03$ gives an extra rounding digit; it does not provide a more accurate estimator of sampling uncertainty.\n\n\\emph{Limits.} The values above are values on the public data. The authors' own input files are not public, and ICPSR states that some variables of the release were changed. On the public data the coefficient and standard error of Table S6 agree with the printed $-0.06$ and $0.03$, and the standard error and $P$ value of the interaction round to the printed $0.46$ and $0.07$. The interaction coefficient is $-0.841$, which rounds to $-0.84$ where Table S7 prints $-0.85$. The rejection rates are properties of the two experiments on the public-data design. They are not the rejection probability of the study under its own unknown outcomes, and the coverage of the bands of Fig.~2 was not examined.", "problem": "The article says that the school-level confidence intervals and $P$ values use ``robust SEs under a normal approximation'', and it does not name the estimator. The authors' script \\texttt{4. School results-with-correction.R} (lines 162--163 and 208--209) uses the heteroskedasticity-robust variance with no finite-sample adjustment, known as HC0. The wild bootstrap of Tables S6 and S7 (lines 178--195 and 228--248) multiplies the unscaled residuals by standard normal draws. Given the data, its variance is exactly HC0. The row ``Wild Bootstrap SE'' is therefore not a second check on the row ``Heteroskedasticity Robust SE''.\n\nThe regressions for disciplinary reports have 49 schools and many coefficients: block effects for 14 blocks, treatment, the composition terms and covariates. On the public data the regression of Table S6 has 19 estimated coefficients and that of Table S7 has 20, which leaves 30 and 29 residual degrees of freedom. Under a linear model with independent mean-zero errors of equal variance, fitting shrinks the residuals: their expected mean square is $30/49$ and $29/49$ of the error variance, whereas the expected HC0 variance is 0.62 of the variance of the treatment coefficient and 0.56 of the variance of the interaction coefficient. A normal reference does not correct for this, and the test rejects too often in the null experiments below.\n\nBoth results on disciplinary reports rest on this test: the main effect ($P<0.05$) and the interaction with the proportion of social referent seeds ($P=0.07$). In null experiments on the design reconstructed from the public data, the test rejects a true null hypothesis at the nominal 5\\% level in 12 to 16\\% of draws. The two printed $P$ values are weaker evidence than the same values from a calibrated test. On the public data, the test of the article gives $P=0.099$ for the main effect and $P=0.069$ for the interaction, and about $0.20$ and $0.15$ when the schools are re-randomized under the hypothesis of no effect on any school.\n\nThe description in the Methods is accurate as far as it goes, and it claims no exact size. Whether the printed $P$ values are the values of this test on the authors' own files remains unknown, because those files are not public. No replacement test is established as calibrated for the two hypotheses of the article.", "what_survives": "If the variance estimator or the reference distribution is changed, the least-squares estimates, their direction, the assignment of schools and the measurement of the networks do not change. The analysis of peer-to-peer influence computes its variance from simulated assignments and does not use this test. The correction of 30\\% to 25\\% is a separate matter (\\fid{P037-N001}).\n\nThe evidence for the two results on disciplinary reports is weaker than printed. On the public data the main effect is not significant at the 0.05 level under the test of the article ($P=0.099$), in agreement with the last sentence of the correction of 2018; the disagreement between that sentence and the article's $P<0.05$ is the subject of \\fid{P037-N008}. The interaction keeps $P=0.07$ under a test that rejects a true null hypothesis in 12 to 16\\% of draws. The treatment bands of Fig.~2 use the same robust variance (their construction is the subject of \\fid{P037-N026}). No printed claim is shown to be false on the authors' own files. No test here is established as calibrated for a zero average effect and for a zero interaction in this design; the variants in the table weaken the evidence and are not such a test."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N006", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the first nine lines of the legend of Fig. 1, 'Overall school climate results: distribution of disciplinary events throughout school networks, comparing treatment, and control schools.' A red box marks the words 'the 49 schools that provided administrative data (26 in control, 23 in treatment)', which run over two lines of the sentence 'Visualization of the effect of treatment on disciplinary reports of peer conflict among the 49 schools that provided administrative data'.", "caption": "Legend of Fig. 1, p. 568 (PDF p. 3). The marked words describe the 49 schools of the conflict analyses as the schools that provided administrative data, 26 in control and 23 in treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N007.png"}, {"alt": "Crop of six lines of the Methods and Materials. A red box marks the parenthesis '(Administrative data were available for 49 of the 56 schools, and we restricted our attention to these schools for analyses of conflict-related events.)', which runs over three lines. The text before it says that behavior was tracked with the schools' administrative records on peer conflict-related disciplinary events across the entire year.", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked parenthesis says that administrative data were available for 49 of the 56 schools and that the analyses of conflict-related events are restricted to them.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N007-b.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T19", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T19"}, {"id": "P037-T20", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T20"}, {"id": "P037-T22", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T22"}, {"id": "P037-T38", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T38"}], "headline": "The authors' code selects the 49 schools of the conflict analyses by a rule on the outcome (a school mean of the peer-conflict count above $0.02$), not by whether a school provided administrative data, which is the article’s description; on the public data 52 schools have disciplinary records, and the three further schools that the rule removes are treatment schools.", "id": "P037-N007", "location": "Legend of Fig. 1, p. 568 (PDF p. 3); Methods and Materials, p. 567 (PDF p. 2), right column; SI Appendix, p. 10", "pdf_page": 3, "printed_page": "568", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T19", "P037-T20", "P037-T22", "P037-T38"], "text": {"as_printed": "Fig.~1, legend: ``Visualization of the effect of treatment on disciplinary reports of peer conflict among \\hl{the 49 schools that provided administrative data} (26 in control, 23 in treatment).''\n\nMethods and Materials, p.~567: ``(\\hl{Administrative data were available for 49 of the 56 schools}, and we restricted our attention to these schools for analyses of conflict-related events.)''", "corrected": "Fig.~1, legend: ``Visualization of the effect of treatment on disciplinary reports of peer conflict among \\hl{the 49 schools whose mean peer-conflict count per student exceeded 0.02} (26 in control, 23 in treatment).''\n\nMethods and Materials, p.~567: ``(\\hl{Each student's peer-conflict count is the sum of 17 disciplinary category counts, with a missing category counted as zero. We restricted the analyses of conflict-related events to the 49 of the 56 schools whose mean count per student exceeded 0.02.})''\n\nOn the public data this rule selects 49 schools (26 control, 23 treatment), and 52 schools (26 in each arm) have recorded disciplinary fields; the three further schools removed are treatment schools with positive counts. The sentence of the SI Appendix (p.~10) that seven schools provided no data needs evidence of what each school supplied. Of the seven schools removed by the code on the public data, four have no disciplinary entry and three have recorded fields.", "evidence": "The evidence is a reading of the authors' code and a recalculation on the public data. In the cleaning script (\\texttt{2. Cleaning.R}, lines 35--38), the count is the sum of the 17 categories after missing values are set to zero; the list of categories is the subject of \\fid{P037-N018}. The school-level script (\\texttt{4. School results-with-correction.R}, lines 151--153) keeps the schools whose mean count exceeds $0.02$, and it fits the regressions of Tables S6 and S7 on them. The network script (\\texttt{3. Network results.R}, lines 167--173) applies the same rule. The script for the check within treatment schools (Fig.~S5) removes the schools with mean below $0.02$, which selects the same schools here, since no school mean equals $0.02$. The rule is not applied to the survey outcomes.\n\nI ran these lines on the public data, after converting its variable names and codes to the form that the scripts read. The authors' filter on the records keeps 24,191 students in 56 schools, as published.\n\n\\begin{audittable}{lrr}{Schools of the public data in each arm, and the sum of the peer-conflict count over their students.}\n & Control & Treatment \\\\\nSchools randomized & 28 & 28 \\\\\nSchools with recorded disciplinary fields & 26 & 26 \\\\\nSchools with no disciplinary entry & 2 & 2 \\\\\nSchools with mean count above $0.02$ (kept by the code) & 26 & 23 \\\\\nSchools with recorded fields that the rule removes & 0 & 3 \\\\\nSum of the count, all schools & 2,213 & 1,862 \\\\\nSum of the count, schools kept & 2,213 & 1,849 \\\\\n\\end{audittable}\n\nThe counts of the three treatment schools that have records and fail the rule sum to 13. Two of these schools have ordinary amounts of other disciplinary records: $0.51$ and $0.41$ per student, which exceed the rates of 32 and of 28 of the 49 retained schools. Nearly all of these records (245 of 255, and all 228) are generic rule violations, and the third school has one other record. These totals do not show whether the three schools reported peer conflict completely. The public data also do not show which seven schools originally provided no data.\n\nI then fitted the covariate-adjusted regressions of Tables S6 and S7 on the public data twice, changing only the schools included.\n\n\\begin{audittable}{lrrrrrr}{Covariate-adjusted school regressions on the public data under the two rules of inclusion: the main effect of treatment (the model of Table S6) and the interaction of treatment with the proportion of social referents (the model of Table S7), each with its heteroskedasticity-robust (HC0) standard error and two-sided normal $P$ value.}\nSchools included & Main effect & SE & $P$ & Interaction & SE & $P$ \\\\\n49 schools (mean count above $0.02$) & $-0.0573$ & $0.0347$ & $0.099$ & $-0.841$ & $0.463$ & $0.069$ \\\\\n52 schools (recorded disciplinary fields) & $-0.0713$ & $0.0323$ & $0.027$ & $-0.469$ & $0.417$ & $0.260$ \\\\\n\\end{audittable}\n\nThese $P$ values use the article's HC0 normal test. In null experiments on the 49-school public-data design, that test rejects a true null too often (\\fid{P037-N006}). The row for 52 schools is a check of sensitivity and not a corrected estimate.\n\nA constructed example shows that the further condition does not follow from randomization, even when every record is complete. Suppose that a block of four schools has control means $0$, $0.03$, $0.06$ and $0.09$ per student and a constant effect of $+0.03$, and that two of the four schools are treated at random. The first school passes the rule only when it is treated. Over the six assignments, the difference in means between the retained treatment and control schools is $-0.03$, $0$, $0.03$, $-0.015$, $0.03$ and $0.075$. The average is $0.015$, half the effect.\n\nThe public data reproduce the published 24,191 students, the 49 schools and their division into 26 and 23, and the $-0.06$ and $0.03$ of Table S6. For the interaction of Table S7, I find $-0.84$ against the printed $-0.85$, with the printed standard error $0.46$. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The values above are values for the public data, and not the values that the authors would obtain from their own files.", "problem": "The article says that the analyses of disciplinary reports of peer conflict (the conflict analyses) use the 49 schools for which administrative data were available, and the SI Appendix (p.~10) says that seven of the 56 schools provided no data. The authors' code does not select schools in this way. It forms each student's peer-conflict count as the sum of 17 disciplinary categories, with a missing category counted as zero, and it keeps a school when the mean of this count over the school's students is greater than $0.02$. This rule is a condition on the outcome. On the public data (ICPSR 37070 V2) the rule selects 49 schools, 26 control and 23 treatment, and the printed count is not shown to be wrong. The same data have disciplinary records for 52 schools, 26 in each arm. Four schools, two in each arm, have no disciplinary entry. The three further schools that the rule removes are all treatment schools. Each has a recorded count for every student, and their school means, $0.0018$, $0.0145$ and $0.0199$, are positive and below $0.02$. On the public data, ``provided administrative data'' does not describe the selection.\n\nThere are two consequences. First, a rule that keeps a school only when its outcome is large enough can depend on the assignment, since the treatment may change the count. A sufficient additional condition for interpreting the conflict estimates causally is that school inclusion and reporting remain unchanged across assignments, together with the paper's model conditions; an analysis that accounts for selection could also justify that interpretation. The observed data cannot check inclusion under other assignments. Second, the estimates depend on the rule. When the three schools are admitted on the public data, the main effect moves from $-0.057$ to $-0.071$, and the interaction with the proportion of social referents moves from $-0.84$ ($P = 0.07$) to $-0.47$ ($P = 0.26$).", "what_survives": "The number 49 and its division into 26 control and 23 treatment schools are recovered by the authors' rule on the public data. The assignment of the 56 schools is unaffected, the rule is not applied to the survey outcomes (talking about conflict, norms, wristbands), and the main effect on disciplinary reports is negative under both rules. No published estimate is shown to be wrong. What changes is the description of the sample in every analysis of disciplinary reports (Figs.~1, 2D and 3D, the peer-conflict columns of Tables S5--S7, and the checks of the SI Appendix that use the same rule). Interpreting the conflict estimates causally needs the additional justification stated above. Whether seven schools originally provided no data, and whether that failure was unrelated to treatment as the SI Appendix says, cannot be decided without the original records of the schools."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N007", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the opening of the Results section. Red boxes mark the words 'significantly reduced' and the parenthesis '(P < 0.05, heteroskedasticity-robust Wald test)' in the sentence 'Figs. 1 and 2 reveal that the treatment significantly reduced average levels of disciplinary reports of peer conflict in treatment compared with control schools'. The next sentences give 0.20 disciplinary events per student in control schools and an average decrease of 0.06.", "caption": "Results, first paragraph, p. 569 (PDF p. 4). The marked words report the overall reduction in disciplinary reports as significant, with $P < 0.05$ from a heteroskedasticity-robust Wald test.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N008.png"}, {"alt": "Crop of the last five lines of the published correction. After the instruction to replace the number 30 with 25, the final sentence reads 'The estimated overall effects on administrative reports of conflict remain statistically insignificant at the alpha=0.05 level.' Two red boxes mark the words 'remain statistically insignificant at the alpha=0.05 level'.", "caption": "2018 correction, p. E3598 (PDF p. 1), last sentence. The marked words say that the overall effects on administrative reports of conflict remain insignificant at the $0.05$ level.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N008-b.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T3", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T3"}, {"id": "P037-T15", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T15"}], "headline": "The article reports the overall reduction in disciplinary reports as significant ($P < 0.05$), the 2018 correction says that the same effect remains insignificant at $0.05$, and the sentence of the Results was never withdrawn; a conventional two-sided Wald test on the public data gives $P = 0.099$, supporting nonsignificance for that reconstruction.", "id": "P037-N008", "location": "Results, first paragraph, p. 569 (PDF p. 4); 2018 correction, p. E3598 (PDF p. 1), last sentence", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T15", "P037-T3"], "text": {"as_printed": "Article, p.~569: ``Figs. 1 and 2 reveal that the treatment \\hl{significantly reduced} average levels of disciplinary reports of peer conflict in treatment compared with control schools ($\\hl{P < 0.05}$, heteroskedasticity-robust Wald test).''\n\n2018 correction, p.~E3598: ``The estimated overall effects on administrative reports of conflict \\hl{remain} statistically insignificant at the $\\alpha=0.05$ level.''", "corrected": "Article, p.~569: ``Figs. 1 and 2 \\hl{show lower} average levels of disciplinary reports of peer conflict in treatment compared with control schools\\hl{; a reconstruction on the public data gives a reduction that is not statistically significant at the 0.05 level} (\\hl{two-sided} heteroskedasticity-robust Wald test).'' On the public data the estimate is $-0.057$ reports per student, with standard error $0.035$ and two-sided $P = 0.099$.\n\n2018 correction, last sentence: ``The estimated overall effects on administrative reports of conflict \\hl{are} statistically insignificant at the $\\alpha=0.05$ level \\hl{in the two-sided analysis reconstructed on the public data; this replaces the statement of significance ($P < 0.05$) in the Results of the article}.''\n\nIf the $P < 0.05$ of the article came from another test, for instance a one-sided test, the test and its inputs should be stated.", "evidence": "The evidence is a check of the pages, a reading of the authors' code and a recalculation on the public data. Both quotations are as printed, and the correction specifies no replacement test. In the school-level script (\\texttt{4. School results-with-correction.R}), lines 162--163 regress the school mean of the peer-conflict count on treatment, block fixed effects, the proportion of social referents among the seed-eligible students and three school covariates, with HC0 standard errors. The script prints no $P$ value. I ran these lines on the public data, after converting its variable names and codes to the form that the scripts read.\n\n\\begin{audittable}{lrrrrr}{The overall effects of Table S6 and their heteroskedasticity-robust standard errors (SE), as printed and from the authors' code on the public data.}\nOutcome & Printed & SE & Public data & SE & Schools \\\\\nFriends talk about conflict & $0.09$ & $0.02$ & $0.0874$ & $0.0176$ & 56 \\\\\nPrescriptive norms & $-0.01$ & $0.08$ & $-0.0142$ & $0.0781$ & 56 \\\\\nWristband & $0.16$ & $0.01$ & $0.1647$ & $0.0114$ & 56 \\\\\nPeer conflict & $-0.06$ & $0.03$ & $-0.0573$ & $0.0347$ & 49 \\\\\n\\end{audittable}\n\nFor peer conflict on the public data, I obtain\n\\[\nz \\simeq \\frac{-0.0572720}{0.0347214} \\simeq -1.649, \\qquad P = 2\\bigl\\{1 - \\Phi(|z|)\\bigr\\} \\simeq 0.099,\n\\]\nand the 95\\% interval is $[-0.125, 0.011]$. A separate calculation of the same regression by matrix algebra agrees to ten decimals. The code uses another fit for the corrected percentage, the interaction model averaged over the 49 schools; on the public data its contrast is $-0.0566$ with standard error $0.0321$ and $P = 0.078$. A one-sided $P$ value for the public-data coefficient of Table S6 would be $0.0495$, but neither the article nor the code specifies a one-sided test.\n\nThe rounded entries of Table S6 do not decide the question by themselves: an estimate of $-0.057$ with standard error $0.028$ or $0.034$ prints as $-0.06$ and $0.03$, and its $P$ value is $0.042$ or $0.094$.\n\nThe public data reproduce all eight entries of the table above at the printed precision, and the published 24,191 students, 56 schools and 49 schools of the conflict analyses. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The $P$ value of $0.099$ is the value for the public data, and the $P$ value of the original run is not recovered.", "problem": "Both sentences describe the overall school-level effect of the intervention on disciplinary reports of peer conflict. The article reports this effect as a significant reduction, with $P < 0.05$ from a heteroskedasticity-robust Wald test. The correction changes the size of the reduction from 30\\% to 25\\% and ends by saying that the overall effects remain insignificant at the $0.05$ level. It names no new test, and the four passages that it lists for amendment do not include the sentence that reports significance. For one test, the two statements cannot both hold. The word ``remain'' is also inaccurate, since the article did not report the overall effect as insignificant. The corrected publication still carries ``significantly reduced ($P < 0.05$)'' in the Results beside a correction that says the opposite, and the abstract, the Significance statement, the Discussion and the legend of Fig.~2 can still be read as establishing the overall reduction at the $0.05$ level.\n\nThe authors' code identifies the analysis; its reconstruction on the public data (ICPSR 37070 V2) supports nonsignificance under a conventional two-sided Wald test. The code fits the regression of Table S6 on the 49 schools of the conflict analyses and prints the coefficient of treatment with its heteroskedasticity-robust (HC0) standard error. On the public data the estimate is $-0.0573$ reports per student with standard error $0.0347$, and the two-sided normal Wald $P$ value is $0.099$. This supports the correction's nonsignificance statement for that reconstruction.", "what_survives": "The point estimate ($-0.06$ reports per student, standard error $0.03$) is reproduced on the public data, and so is the corrected 25\\% ($25.4\\%$). Nothing here bears on the randomization, the intervention or the data. What changes is the sentence of the Results; readings of the abstract, the Significance statement, the Discussion and the legend of Fig.~2 as establishing the overall reduction at the $0.05$ level need the same qualification for the public-data analysis. Whether this test is reliable with 49 schools is the subject of \\fid{P037-N006}. The results on the other school-climate outcomes, on the composition of the seed group and on peer-to-peer influence rest on separate estimates and tests."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N008", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of the first paragraph of the Results: 'a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year. We estimated an average decrease of 0.06 disciplinary events per student in treatment schools, a 30 percent reduction in peer conflict reports.' A red box marks the number 0.20.", "caption": "Results, first paragraph, p. 569 (PDF p. 4). The marked number is the mean number of disciplinary events per student that the article reports for control schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N009.png"}, {"alt": "Crop of six lines of the published correction, from 'mated average treatment effect (-0.06) divided by the rounded unadjusted control group mean (0.20)' to 'The original estimate of 30 percent was reported in the Abstract (line 11), the Significance'. A red box marks the sentence 'Using both unadjusted estimates, we have an estimate of 23 percent.', which runs over two lines and follows 'Using both covariate adjusted estimates, we have an estimate of 25 percent.'", "caption": "2018 correction, p. E3598 (PDF p. 1). The marked sentence gives 23% as the reduction computed from both unadjusted estimates.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The control mean of $0.20$ in the Results and the “unadjusted” 23% of the 2018 correction are computed in the authors' code over all 56 schools, including the seven schools outside the conflict analyses, whose counts are zero or nearly zero; on the 49-school sample selected by the code, the public data give raw means of $0.21$ and $0.18$ and a raw reduction of 13.6%.", "id": "P037-N009", "location": "Results, first paragraph, p. 569 (PDF p. 4); 2018 correction, p. E3598 (PDF p. 1)", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Article, p.~569: ``In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average \\hl{0.20} times per year.''\n\n2018 correction, p.~E3598: ``Using both covariate adjusted estimates, we have an estimate of 25\\%. Using both \\hl{unadjusted estimates}, we have an estimate of \\hl{23\\%}.''", "corrected": "Article, p.~569: ``In a control school, we estimated that each student in the school was officially disciplined for peer conflict on average 0.20 times per year \\hl{(the unadjusted mean over all 28 control schools, after missing disciplinary fields were coded as zero)}.''\n\n2018 correction: ``Using both unadjusted estimates \\hl{over all 56 schools, including the seven excluded from the conflict analyses, after missing disciplinary fields were coded as zero,} we have an estimate of 23\\%.''\n\nOn the 49-school sample selected by the code, I obtain raw control and treatment means of $0.2124$ and $0.1836$ on the public data, weighting schools equally. These round to $0.21$ and $0.18$ and imply a raw reduction of 13.6\\%, or 14\\% to the nearest whole percent. These are a different comparison from the printed $0.20$ and 23\\%. The adjusted 25\\% uses interaction-model predictions over the 49 schools, not the raw 56-school control mean.", "evidence": "The evidence is a reading of the authors' code and a recalculation on the public data. In the school-level script (\\texttt{4. School results-with-correction.R}), lines 151--153 select schools whose mean exceeds $0.02$ (\\fid{P037-N007}), yielding 49 schools on the public data, and the fits on lines 162 and 208 use them. Lines 264, 273 and 289 compute the raw means and the raw percentage without the rule. Lines 253--256 and 286 compute the adjusted percentage from the interaction model, by averaging its predictions over the 49 schools with treatment set to zero and to one. I ran these lines on the public data, after converting its variable names and codes to the form that the scripts read. Each mean weights the schools equally.\n\n\\begin{audittable}{lrrr}{Mean peer-conflict count per student in control and treatment schools on the public data: raw means over all 56 schools, raw means over the 49 schools of the analyses, and the adjusted means that the code uses for the corrected 25\\%.}\n & Raw, 56 schools & Raw, 49 schools & Adjusted, 49 schools \\\\\nControl schools & 28 & 26 & 26 \\\\\nTreatment schools & 28 & 23 & 23 \\\\\nControl mean & $0.1972$ & $0.2124$ & $0.2225$ \\\\\nTreatment mean & $0.1521$ & $0.1836$ & $0.1660$ \\\\\nDifference & $-0.0451$ & $-0.0288$ & $-0.0566$ \\\\\nReduction & 22.9\\% & 13.6\\% & 25.4\\% \\\\\n\\end{audittable}\n\nOn the public data, the model of Table S6 has fitted rates $0.2258$ and $0.1685$. Multiplying them by the 11,938 students across all 28 treatment schools, I obtain the totals 2,695 and 2,012 of the Discussion after rounding. These totals are fitted values and not raw means. The public data reproduce the published 24,191 students, the 26 control and 23 treatment schools, the $-0.06$ of Table S6 and the rounded $0.20$, 29\\%, 25\\% and 23\\%; the interaction is $-0.84$ against the printed $-0.85$. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The decimals above are values for the public data; the exact historical values and excluded-school membership cannot be established without the authors' original files.", "problem": "The article restricts the analyses of disciplinary reports to 49 schools (p.~567); the authors' code selects 49 schools on the public data by a rule on the school mean. The lines of the code that compute the raw means of the two arms do not apply the restriction. They average over all 56 schools. On the public data (ICPSR 37070 V2), missing disciplinary fields are coded as zero: four of the seven excluded schools have no recorded fields for peer conflict, and three have positive counts. The two excluded control schools have no recorded event, and the five excluded treatment schools have 13 events in all. Including them lowers the treatment mean more than the control mean. On the public data the means over the 56 schools are $0.197$ and $0.152$, a reduction of 22.9\\%, which reproduces the printed $0.20$ and 23\\%. Over the 49 schools the means are $0.212$ and $0.184$, a reduction of 13.6\\%. The correction places the 23\\% beside the adjusted 25\\% as the same comparison without adjustment. On the 49-school sample selected by the code in the public data, the raw reduction is about half of the adjusted one.", "what_survives": "The adjusted estimate and the corrected 25\\% are reproduced at published precision on the public data, as are the 29\\% of the correction and the totals of the Discussion. No defect in the regressions follows from this sample mismatch. What changes is the interpretation of the raw control mean of the Results and the raw percentage of the correction: the code computes them over all 56 schools, rather than its 49-school conflict-analysis sample."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N009", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the end of the second paragraph of the Results. In the sentence 'When 20 percent of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to 37 percent), we estimate that each student was disciplined on average 0.08 times during the year, a reduction of 60 percent compared with control schools and twice the average effect of the anticonflict groups', a red box marks the number 0.08.", "caption": "Results, second paragraph, p. 569 (PDF p. 4). The marked number is the level of disciplinary reports per student that the sentence reports at 20% social referents.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N010.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T13", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T13"}], "headline": "At 20% social referents the fitted model predicts about $0.095$ disciplinary reports per student, not $0.08$.", "id": "P037-N010", "location": "Results, second paragraph, p. 569 (PDF p. 4), sentence beginning “When 20%”", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T13"], "text": {"as_printed": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to 37\\%), we estimate that each student was disciplined on average \\hl{0.08} times during the year, a reduction of 60\\% compared with control schools and twice the average effect of the anticonflict groups.", "corrected": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds \\ldots, we estimate that each student was disciplined on average \\hl{about 0.095} times during the year, a reduction of 60\\% compared with control schools and twice the average effect of the anticonflict groups.\n\nThis is the smallest change; to two decimals the level is $0.10$. A fuller statement would report the fitted control level on the public data, about $0.223$, a reduction of about 57\\%, and an effect of roughly $2.2$ times the average decrease in disciplinary events per student. The parenthesis on the range of the proportions is the subject of \\fid{P037-N027}. In the first paragraph of the Results, label $0.20$ as the unadjusted control mean over all 28 control schools and state that the corrected 25\\% uses an adjusted control level of about $0.2225$ from the interaction regression on the public data; see \\fid{P037-N001} and \\fid{P037-N009}.", "evidence": "The evidence is a recalculation on the public data (ICPSR 37070 V2) with the authors' code through an adapter for that release, and a recalculation from the published table and figure. In the school-level script (\\texttt{4. School results-with-correction.R}), line 208 fits the interaction model on the 49 schools of the conflict analyses, lines 253--255 average its predictions for control, and lines 311--312 and 332--333 compute the plotted level in treatment schools. On the public data I obtain $\\widehat{\\tau}_0 \\approx 0.04083$, $\\widehat{\\beta} \\approx -0.84134$ and a fitted control level of about $0.22254$, and the level at $c = 0.20$ is\n\\[\n0.22254 + 0.04083 + 0.20 \\times (-0.84134) \\approx 0.0951 .\n\\]\nThe difference from control is $-0.1274$, and the reduction is $0.1274/0.2225 \\approx 57.3\\%$. The main effect of Table S6 on the public data is a decrease of $0.0573$, and the decrease at $c = 0.20$ is $2.2$ times as large.\n\nThe published figure agrees. In the vector drawing of Fig.~2D, the control line is at about $0.2225$ and the treatment line is approximately $0.2634 - 0.8414\\,c$, giving about $0.0951$ at $c = 0.20$; before rounding, the two calculations differ by less than $0.000002$ in level.\n\nRounding in Table S7 does not close the gap. By SI equations (4) and (6), the difference between the treatment and control predictions at proportion $c$ is $\\widehat{\\tau}_0 + \\widehat{\\beta} c$, and Table S7 prints $\\widehat{\\tau}_0 = 0.04$ and $\\widehat{\\beta} = -0.85$ for peer conflict. If $\\widehat{\\tau}_0 \\in [0.035, 0.045]$ and $\\widehat{\\beta} \\in [-0.855, -0.845]$, then the effect at $c = 0.20$ lies in $[-0.136, -0.124]$. Since the control level lies in $[0.222, 0.223]$, the treatment level lies in $[0.086, 0.099]$. A number printed as $0.08$ lies in $[0.075, 0.085]$, and the two intervals are disjoint.\n\nThe values from the code are values for the public data. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The agreement with Fig.~2D holds at the precision of the drawing. On the public data the code does not show how the printed $0.08$ arose.", "problem": "The sentence reports a prediction of the fitted model of Fig.~2D and Table S7 of the SI Appendix, at a proportion $c = 0.20$ of social referents in the seed group. The predicted level in treatment schools is the fitted control level plus the treatment effect $\\widehat{\\tau}_0 + \\widehat{\\beta} c$. From the line of Fig.~2D and from the authors' code on the public data, I find a level of about $0.095$, and no values that round to the entries of Table S7 correspond to a level that rounds to $0.08$. The other two numbers of the sentence hold as approximations: by the same calculation, the reduction is about 57\\% and the effect is about $2.2$ times the average decrease.", "what_survives": "The fitted effect at 20\\% referents (about $-0.13$ reports per student), the lines of Fig.~2D, the projected totals of the Discussion and the corrected 25\\% are unchanged, and the effect at 20\\% referents is about twice the average effect, as stated. Only the printed level changes. The slope of Fig.~2D differs from that of Table S7 in the second decimal (\\fid{P037-N011}); with either, the level lies between $0.086$ and $0.099$."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N010", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Fig. 2, four panels with its caption at the right. Each panel plots a horizontal red line for control schools and a sloped black line for treatment schools against the proportion of social referent seeds in the anti-conflict group, with ticks from 0.05 to 0.20, each with dashed curves labeled as 95 percent confidence intervals. Red boxes mark panel B, 'Perceived social norm against conflict', and panel D, 'Peer conflict (disciplinary records)', where the black line falls from about 0.24 to about 0.09. A third red box marks the words 'Table S7' at the end of the caption.", "caption": "Fig. 2 and its caption, p. 569 (PDF p. 4). The marked panels, B (perceived social norm against conflict) and D (disciplinary records of peer conflict), are the two whose lines do not match Table S7, to which the caption refers.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N011.png"}, {"alt": "Crop of Table S7, 'Average school-level climate effects caused by anti-conflict groups, interacted with proportion of social referent seeds in the group', with columns Friends talk about conflict, Prescriptive norms, Wristband and Peer conflict. Red boxes mark, in the Prescriptive norms column, the school effect with 0 percent social referents, 0.05, and the interaction -0.53 with its robust standard error (1.06); and, in the Peer conflict column, the interaction -0.85. A fourth red box marks the sentence of the note 'A graphical version of this table is presented in Figure 2'.", "caption": "SI Appendix, Table S7, p. 42 (supplement PDF p. 42), shown upright; the table is printed sideways on the page. The marked entries are the four that differ from Fig. 2 and from the authors' code on the public data, and the marked sentence of the note calls Fig. 2 the graphical version of the table.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N011-b.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T13", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T13"}], "headline": "Four entries of Table S7 do not match Fig. 2, which the SI Appendix presents as the graphical version of the table; the authors' code on the public data reproduces the fitted lines and treatment bands and gives $0.04$, $-0.51$ and $1.07$ where the table prints $0.05$, $-0.53$ and $1.06$ for prescriptive norms, and $-0.84$ where it prints $-0.85$ for peer conflict.", "id": "P037-N011", "location": "Fig. 2, panels B and D, and its caption, p. 569 (PDF p. 4); SI Appendix, Table S7 and its note, p. 42 (supplement PDF p. 42)", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T13"], "text": {"as_printed": "Fig.~2, caption: ``Lines represent estimates of the conditional average potential outcome with respect to the proportion of social referent seeds, and are surrounded by 95\\% confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit. \\ldots\\ See SI Appendix, \\hl{Table S7} for regression tables.''\n\n\\begin{audittable}{lrr}{Table S7 (SI p.~42), two of its four columns and the rows for the estimates and their heteroskedasticity-robust standard errors.}\n & Prescriptive norms & Peer conflict \\\\\nSchool effect with 0\\% Social Referents & \\hl{0.05} & 0.04 \\\\\nHeteroskedasticity Robust SE & (0.15) & (0.07) \\\\\nInteraction: Proportion of Referents in Treatment group & \\hl{-0.53} & \\hl{-0.85} \\\\\nHeteroskedasticity Robust SE & \\hl{(1.06)} & (0.46) \\\\\n\\end{audittable}\n\nTable S7, note: ``\\hl{A graphical version of this table is presented in Figure 2} in the main text of the paper.''", "corrected": "\\begin{audittable}{lrr}{Table S7, the same rows and columns, with the values from the authors' code on the public data.}\n & Prescriptive norms & Peer conflict \\\\\nSchool effect with 0\\% Social Referents & \\hl{0.04} & 0.04 \\\\\nHeteroskedasticity Robust SE & (0.15) & (0.07) \\\\\nInteraction: Proportion of Referents in Treatment group & \\hl{-0.51} & \\hl{-0.84} \\\\\nHeteroskedasticity Robust SE & \\hl{(1.07)} & (0.46) \\\\\n\\end{audittable}\n\nThese four values agree with the corresponding estimates recovered from Fig.~2 as published. They are values for the public data and should be checked against the authors' own data before they are printed as the entries of the table.\n\nUse the same interaction fit for the table and figure, retaining the 49-school restriction for every peer-conflict covariance calculation on the public data. To retain the predicted-mean labels, calculate each band's variance as $\\hl{a_g(c)^\\top V a_g(c)}$, where $V$ is that fit's full heteroskedasticity-robust covariance matrix and $a_g(c)$ is the prediction design vector averaged over the included schools for group $g$ at proportion $c$. The pointwise interval is the predicted mean plus or minus $1.96\\sqrt{a_g(c)^\\top V a_g(c)}$. The bands of Fig.~2 are also addressed in \\fid{P037-N026}.", "evidence": "The evidence is a recalculation from the published PDF and a recalculation on the public data with the authors' code. In each panel I read the lines and the band around the treatment line from the vector drawing and calibrated them on the axis ticks. In the school-level script (\\texttt{4. School results-with-correction.R}), lines 208--209 fit the model of Table S7, on 56 schools for the survey outcomes and on 49 schools for peer conflict. I ran these lines on the public data, after converting its variable names and codes to the form that the scripts read.\n\n\\begin{audittable}{lrrrr}{The four entries of Table S7 that differ: the printed value, the value read from Fig.~2, the value from the authors' code on the public data (Code), and that value at two decimals (Rounded). SE is the heteroskedasticity-robust standard error.}\nEntry & Table S7 & Fig.~2 & Code & Rounded \\\\\nNorms, effect with 0\\% referents & $0.05$ & $0.042$ & $0.0421$ & $0.04$ \\\\\nNorms, interaction & $-0.53$ & $-0.511$ & $-0.5107$ & $-0.51$ \\\\\nNorms, SE of the interaction & $1.06$ & $1.069$ & $1.0692$ & $1.07$ \\\\\nPeer conflict, interaction & $-0.85$ & $-0.841$ & $-0.8413$ & $-0.84$ \\\\\n\\end{audittable}\n\nThe standard error in the third row is read from the curvature of the band: if $h(c)$ is the half-width of the dashed band around the treatment line, then $\\{h(c)/1.96\\}^2$ is a quadratic in $c$ whose leading coefficient is the variance of the slope. The other 12 estimates and robust standard errors of Table S7 agree with the code on the public data at the printed precision. For the interaction of peer conflict, the fit on the 49 schools has standard error $0.4626$, which is the printed $0.46$, and a two-sided normal $P$ value of $0.069$, which is the $P = 0.07$ of the caption. The band of panel D implies the smaller standard error $0.394$, which comes from the fit on 56 schools.\n\nThe code on the public data matches the treatment band at all 178 plotted proportions in each panel to within $0.000032$ outcome units. The authors' own data file is not public, and ICPSR states that some variables of the public data were changed. The values in the fourth column are values for the public data, and the source of the four printed entries is not known.", "problem": "For each outcome, Table S7 reports the treatment effect when no seed is a social referent, $\\widehat{\\tau}_0$, and the slope $\\widehat{\\beta}$ of the effect in the proportion $c$ of social referents. The note to the table calls Fig.~2 its graphical version. Since the figure plots the predicted means in control and in treatment schools, the gap between the two lines at proportion $c$ is $\\widehat{\\tau}_0 + \\widehat{\\beta} c$. Panels A and C agree with the table at the printed precision. In panel B the gap has intercept $0.042$ and slope $-0.511$, against the printed $0.05$ and $-0.53$, and in panel D the slope is $-0.841$ against the printed $-0.85$. Rounding in the table and in the drawing does not account for these differences. The authors' code on the public data (ICPSR 37070 V2) reproduces the lines and treatment bands of all four panels, and its values are those of the figure. The figure and the code on the public data agree, and four entries of the table differ from them in the second decimal: the three marked entries for prescriptive norms and the interaction for peer conflict.\n\nThe band of panel D is a separate matter. On the public data, the code draws it using the uncertainty of a treatment-control contrast from a fit on all 56 schools around a line fitted on the 49 schools of the conflict analyses, and it is not the interval for the predicted mean that the caption describes; this is the subject of \\fid{P037-N026}. For that reason the band of panel D cannot be used to recover the reported test of the interaction. It is narrower than the same-fit contrast interval at low referent proportions, but not throughout the plotted range.", "what_survives": "Panels A and C, and 12 of the 16 estimates and robust standard errors of Table S7, agree at the printed precision. The interaction for peer conflict is negative in the figure, in the table and on the public data, and the $P = 0.07$ of the caption is reproduced. The qualitative reading of the fitted lines is unchanged. I draw no conclusion here about the average effect or its significance; the bootstrap rows of Table S7 were not included in this comparison. The prediction at 20\\% referents that the Results take from Fig.~2D is the subject of \\fid{P037-N010}."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N011", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of seven lines of the right column of the Results. The sentence 'These statements about the undesirability of conflict behaviors, or prescriptive norms, moved in the same manner as those describing the typicality of conflict behaviors, or descriptive norms' ends with the citation '(SI Appendix, Figs. S3 and S6 and Tables S3 and S4)', which is marked with a red box.", "caption": "Results, p. 569 (PDF p. 4), right column. The marked citation accompanies the statement that prescriptive and descriptive norms moved in the same manner; it does not include Fig. S5.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N013.png"}], "formal_release": null, "formal_statements": [], "headline": "The citation for the statement that descriptive norms moved like prescriptive norms points to four items of the SI Appendix, and none of them shows that comparison; Fig. S5, the only display of descriptive-norm contrasts in the publication, is not cited.", "id": "P037-N013", "location": "Results, p. 569 (PDF p. 4), right column, sentence beginning “These statements about the undesirability of conflict behaviors”", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "These statements about the undesirability of conflict behaviors, or prescriptive norms, moved in the same manner as those describing the typicality of conflict behaviors, or descriptive norms (SI Appendix, \\hl{Figs. S3 and S6 and Tables S3 and S4}).", "corrected": "These statements about the undesirability of conflict behaviors, or prescriptive norms, moved in the same manner as those describing the typicality of conflict behaviors, or descriptive norms (SI Appendix, Figs. S3\\hl{, S5,} and S6 and Tables S3 and S4).\n\nThe four exposure means for descriptive norms and their contrasts are not published. In a reconstruction on the public data using the Rounding sampler, they are the values of the table above: means of $-0.006$, $-0.958$, $-0.362$ and $-0.113$ for control schools, no seed exposure, nonreferent exposure and referent exposure, with contrasts of $0.846$ ($P = 0.009$) for referent against no seed exposure and $0.249$ ($P = 0.30$) for referent against nonreferent exposure.", "evidence": "I checked the cited pages. In Fig.~S5, the contrasts for peers of social referent seeds are positive for both measures: $0.432$ ($p = 0.044$) for prescriptive norms and $0.635$ ($p = 0.013$) for descriptive norms. Fig.~S5 compares students within treatment schools, and it does not reproduce the four exposure means of Table S5.\n\nI also ran the descriptive-norm option of the authors' code on the public data (ICPSR 37070 V2), using an adapter for the released variables and reconstructed exposure inputs. The exposure probabilities were rebuilt from 10,000 simulated assignments with seed 123, the variances from the first 1,000 of them, and the principal components were given the signs of Tables S3 and S4. The table below shows the result with the Rounding sampler that R used before version 3.6. With the current Rejection sampler, the two contrasts are $0.875$ ($P = 0.007$) and $0.277$ ($P = 0.22$), on 2,101 students. In both runs the mean under referent exposure is above the means of the other two exposures in treatment schools, which is the ordering of the prescriptive-norm means, and the difference between referent and nonreferent exposure is not significant.\n\n\\begin{audittable}{lrr}{Descriptive-norm index under the four exposure conditions: a reconstruction with the authors' code on the public data, using the Rounding sampler, 2,094 students. The standard errors are randomization-based, and $P$ is from the two-sided Wald test of the code.}\nQuantity & Estimate & Standard error \\\\\nMean, control school & $-0.006$ & $0.102$ \\\\\nMean, treatment school, no seed exposure & $-0.958$ & $0.293$ \\\\\nMean, nonreferent seed exposure & $-0.362$ & $0.196$ \\\\\nMean, social referent seed exposure & $-0.113$ & $0.145$ \\\\\nReferent minus no seed exposure ($P = 0.009$) & $0.846$ & $0.325$ \\\\\nReferent minus nonreferent exposure ($P = 0.30$) & $0.249$ & $0.239$ \\\\\n\\end{audittable}\n\nThese values are computed from the public data. The corresponding values from the authors' own data files are unknown: those files are not public, and ICPSR states that some variables in the release were changed. Two comparisons show how close the public data come. All 48 loadings of Tables S3 and S4 are reproduced at the printed precision. The prescriptive-norm means from the same run have a largest absolute difference of approximately $0.023$ from Table S5, on 2,273 students where the note to Table S5 states 2,451.", "problem": "None of the four cited items separately identifies an effect on descriptive norms. Fig.~S3 shows scree plots, and Tables S3 and S4 report factor loadings; these describe how the two norm indices were built. Fig.~S6 repeats the analyses of Figs.~2 and 3 with another measure of centrality, and it has no separately identified result for descriptive norms. Table S5, which reports the main exposure estimates, has no column for descriptive norms. The only explicitly identified post-treatment contrasts for descriptive norms in the publication are in Fig.~S5 (SI pp.~30--32), under the alternative specification, and Fig.~S5 is not cited. The authors' analysis code has an option that applies the main four-exposure analysis to the descriptive-norm index, but the script that writes Table S5 reads only the four outcomes of that table, and the publication has no estimate from that option.", "what_survives": "The statement is not shown to be false. The publication documents the parallel between the two measures only under the alternative specification of Fig.~S5. On the public data, the authors' code shows the same qualitative ordering under the main four-exposure analysis, although the difference between referent and nonreferent exposure is not significant for descriptive norms. The estimates from the authors' own files are not available. No reported estimate changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N013", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Two excerpts of a numbered list, separated by a grey horizontal rule. Above the rule: 1. Inappropriate language towards student (written or verbal), marked by a red box, and 2. Verbal altercation with student. Below the rule: 13. Inciting violence/planning fight; 14. Inappropriate language towards student (written or verbal), marked by a red box; and 15. Verbal altercation with student.", "caption": "SI Appendix, list of disciplinary codes: items 1 and 2 from p. 10 (above the grey rule) and items 13 to 15 from p. 11 (below it). The marked item 14 repeats the marked item 1, and the list then runs through the same thirteen categories a second time.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N018.png"}], "formal_release": null, "formal_statements": [], "headline": "In the list of disciplinary codes, thirteen categories are printed twice, as items 1–13 and again as items 14–26, and two categories that the released code sums are missing. The outcome is the sum of seventeen counts, each entered once.", "id": "P037-N018", "location": "SI Appendix, “Codes for administrative disciplinary action (counts summed for each individual and school)”, pp. 10–12 (PDF pp. 10–12 of the SI Appendix), numbered list, items 1–28", "pdf_page": 10, "printed_page": "10–12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``1. Inappropriate language towards student (written or verbal)'' \\ldots ``13. Inciting violence/planning fight'', then ``\\hl{14. Inappropriate language towards student (written or verbal)}'' \\ldots ``26. Inciting violence/planning fight'', then ``27. Number of H.I.B. incidents as offender'' and ``28. Number of H.I.B. incidents as target''.", "corrected": "\\hl{Items 1--13 are deleted}. Items 14--26 keep their wording, which is the more specific of the two versions. \\hl{``Taking or abusing other student property'' and ``Online harassment'' are added} before the two H.I.B. counts, and the list is renumbered 1--17. The outcome for student $i$ is\n\\[\nP_i=\\sum_{j=4}^{16}c_{ij}+c_{i,46}+c_{i,47}+o_i+t_i,\n\\]\nwhere $c_{ij}$ denotes the count of the code \\texttt{de}$j$, $o_i$ the count \\texttt{hibo} of original item 27 and $t_i$ the count \\texttt{hibt} of original item 28, and a missing count is taken as zero. Each of the seventeen counts enters once. The wording of original item 28 is corrected in \\fid{P037-N019}.", "evidence": "The evidence is a check of pp.~10--12, a reading of the released code and a recalculation on the public data. Items 1 and 14, items 2 and 15, and each later pair up to items 13 and 26 name the same category. The investigators' codebook has one variable for each of the thirteen categories (\\texttt{de4} to \\texttt{de16}), one for each of the two H.I.B. counts (\\texttt{hibo} and \\texttt{hibt}), and the variables \\texttt{de46} and \\texttt{de47} with the labels quoted above. The script \\texttt{2. Cleaning.R}, lines 36--38, selects these seventeen variables once each, sets missing entries to zero and sums them for each student. I ran these lines on the public data after adapting the column names and filter variables and applying the first script's filter (line 33), which retains 24,191 of the 24,471 public records and removes 280, the number that the script's own comment states. The thirteen categories sum to 2,281, the two unlisted categories to 44 and 36, and the two H.I.B. counts to 811 and 903. The total is 4,075. The fifteen distinct counts named in the printed list sum to 3,995; the total difference is 80, and each retained student's difference equals \\texttt{de46} plus \\texttt{de47}.", "problem": "Items 14--26 repeat the thirteen categories of items 1--13 in the same order. The wording differs in three places: item 16 reads ``Bullying / H.I.B. report (H.I.B.: harassment, intimidation, and bullying New Jersey state report)'' where item 3 reads ``Bully/HIB'', item 22 has ``can be one way or two way'', and item 23 adds ``NOT pushing/hitting/shoving/tackling''. The heading says that the counts are summed, and the list has 28 entries for 15 distinct counts. The released code sums 17 counts. Besides the thirteen categories and the two H.I.B. counts, it includes two categories that the list does not name, ``Taking or abusing other student property'' (\\texttt{de46}) and ``Online harassment'' (\\texttt{de47}).", "what_survives": "The code enters each of the thirteen repeated categories once, and the repetition in the list is not a double count in the outcome. Correcting the list to the seventeen counts leaves the outcome and every estimate computed from it unchanged. The check does not show that the two unlisted categories should have been left out, and I did not compute how the estimates change without them. The totals are for the public data (ICPSR 37070 V2). The authors' own data file is not public, and ICPSR states that some variables of the release were changed; I cannot confirm that the counts in the two files are equal."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N018", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Four lines of a numbered list: 25. Sexual harassment, inappropriate sexual conduct (inc. sexual behavior in school); 26. Inciting violence/planning fight; 27. Number of H.I.B. incidents as offender; 28. Number of H.I.B. incidents as target. A red box marks the word incidents in item 28.", "caption": "SI Appendix, p. 12 (PDF p. 12), end of the list of disciplinary codes. The marked word gives the unit of item 28 as incidents; the codebook of the released data says that the variable counts writeups, and the released code adds it to the outcome.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N019.png"}, {"alt": "Two excerpts from one column of text, separated by a grey horizontal rule. Above the rule, a red box that runs over three lines marks the words each student in the school was officially disciplined for peer conflict on average 0.20 times per year. Below the rule, a red box that runs over two lines marks the words each student was disciplined on average 0.08 times during the year.", "caption": "Results, p. 569 (PDF p. 4), left column: lines from the first paragraph (above the grey rule) and from the second (below it). The marked words give the level of the outcome, 0.20 in control schools and 0.08 at 20% social referents, as the number of times each student was disciplined.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N019-b.png"}, {"alt": "The caption of Fig. 1, Overall school climate results: distribution of disciplinary events throughout school networks, comparing treatment, and control schools. A red box that runs over three lines marks the words the average number of times each student in the school was disciplined for peer conflict. A second red box, over two lines, marks the words the number of times they were disciplined during the year.", "caption": "Caption of Fig. 1, p. 568 (PDF p. 3). The marked words describe the colour of a school as the average number of times each student was disciplined for peer conflict, and the size of a student’s node as the number of times the student was disciplined.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N019-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The disciplinary outcome is described as the number of times each student was disciplined for peer conflict, but it includes writeups that name the student as the target of harassment: 903 of the 4,075 units of the outcome on the public data. The outcome is an index of administrative records, and item 28 of the SI list counts writeups, not incidents.", "id": "P037-N019", "location": "SI Appendix, “Codes for administrative disciplinary action (counts summed for each individual and school)”, pp. 10–12 (PDF pp. 10–12 of the SI Appendix), item 28; main text, Results, p. 569 (PDF p. 4), left column, first two paragraphs; caption of Fig. 1, p. 568 (PDF p. 3)", "pdf_page": 10, "printed_page": "10–12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "SI Appendix, p.~12: ``28. Number of H.I.B. \\hl{incidents} as target''\n\nResults, p.~569: ``In a control school, we estimated that \\hl{each student in the school was officially disciplined for peer conflict on average 0.20 times per year}.'' Later on the same page: ``we estimate that \\hl{each student was disciplined on average 0.08 times during the year}''.\n\nCaption of Fig.~1, p.~568: ``Color coding reveals \\hl{the average number of times each student in the school was disciplined for peer conflict}, from dark blue (little conflict) to dark orange (many disciplinary events; higher concentration of dark oranges among control schools). Student nodes are colored red when \\hl{the student was disciplined for conflict}, and their node is scaled to \\hl{the number of times they were disciplined during the year}.''", "corrected": "SI Appendix, p.~12: ``28. Number of H.I.B. \\hl{offender writeups naming the student} as target \\hl{(multiple writeups may refer to one incident)}''\n\nThe definition of the outcome is added to the SI Appendix. For student $i$,\n\\[\nP_i=\\sum_{j=4}^{16}c_{ij}+c_{i,46}+c_{i,47}+o_i+t_i,\n\\]\nwhere $c_{ij}$ denotes student $i$'s count in the field \\texttt{de}$j$, $o_i$ is the count of item 27 (H.I.B. incidents as offender), $t_i$ is the number of H.I.B. offender writeups that name the student as target, and a missing count is taken as zero. \\hl{$P_i$ is an index of administrative records of peer conflict; it is not the number of times student $i$ was disciplined.}\n\nResults, p.~569: ``In a control school, \\ldots\\ \\hl{the index of administrative records of peer conflict, which counts disciplinary reports of a student and writeups naming the student as a target, averaged 0.20 per student} per year.'' In the second paragraph: ``\\ldots\\ \\hl{the fitted mean of this index was about 0.095 per student} during the year'' (the adjusted prediction reconstructed on the public data, agreeing with Fig.~2D to plotted precision; \\fid{P037-N010}).\n\nCaption of Fig.~1, p.~568: ``Color coding reveals \\hl{the average per student of the index of administrative records of peer conflict (disciplinary reports of a student and writeups naming the student as a target)}, \\ldots\\ Student nodes are colored red when \\hl{the student's index is positive}, and their node is scaled to \\hl{the value of the index}.''\n\nThe printed 0.20 stays as the rounded unadjusted mean of the index over all 28 control schools, reproduced on the public data. If the number of a student's own disciplinary actions is wanted, the count as target is removed and the remaining counts are reconciled by incident; on the public data the sum without the count as target has the values in the last column of the table above, and that sum is not yet a count of distinct disciplinary actions.", "evidence": "The evidence is a check of the codebook, an exact count, a reading of the released code and a recalculation on the public data.\n\nFor a student $t$ and an incident $e$, let $m(e)$ denote the number of offender writeups for $e$ that name $t$ as target. The variable equals $W=\\sum_e m(e)$, and I write $I$ for the number of incidents with $m(e)>0$, the incidents represented in these writeups. Subtracting, I obtain\n\\[\nW-I=\\sum_{e:\\,m(e)>0}\\bigl(m(e)-1\\bigr)\\ge 0,\n\\]\nwith equality only if $m(e)=1$ for every incident with $m(e)>0$. In the codebook's example one incident has three offenders, each of whom names the same target, and I find $W=3$ and $I=1$.\n\nThe script \\texttt{2. Cleaning.R}, lines 36--38, forms the outcome from the seventeen variables \\texttt{de4} to \\texttt{de16}, \\texttt{de46}, \\texttt{de47}, \\texttt{hibo} and \\texttt{hibt}, and the school and network scripts use this variable when the disciplinary outcome is selected. I ran these lines on the public data (ICPSR 37070 V2), with an adapter for the release's encodings; the filter of the first script retains 24,191 of the 24,471 records, in 56 schools.\n\n\\begin{audittable}{lrr}{Components of the disciplinary outcome on the public data: sums over the 24,191 retained records of the counts that the released code adds, and their shares of the total. The count as target is the number of offender writeups that name the student as target.}\nComponent & Sum & Share \\\\\nThirteen categories of the list (\\texttt{de4} to \\texttt{de16}) & 2,281 & 56.0\\% \\\\\nTaking or abusing other student property (\\texttt{de46}) & 44 & 1.1\\% \\\\\nOnline harassment (\\texttt{de47}) & 36 & 0.9\\% \\\\\nH.I.B. incidents as offender (\\texttt{hibo}, item 27) & 811 & 19.9\\% \\\\\nH.I.B. writeups as target (\\texttt{hibt}, item 28) & 903 & 22.2\\% \\\\\nOutcome: sum of the seventeen counts & 4,075 & 100.0\\% \\\\\nSum without the count as target & 3,172 & 77.8\\% \\\\\n\\end{audittable}\n\nThe 389 students whose outcome is positive through the count as target alone account for 650 of the 903 writeups, and all of them are in the 49 schools of the conflict analysis (26 control, 23 treatment). I then ran the school-level regressions of the script \\texttt{4. School results-with-correction.R} (lines 162 and 208, with the robust covariance of lines 163 and 209) on the public data, once with the outcome as coded and once without the count as target. The same 49 schools pass the script's screen in both cases.\n\n\\begin{audittable}{lrrr}{The disciplinary outcome with and without the count as target, on the public data. Published: the values printed in the Results and in Tables S6 and S7. As coded: the sum of the seventeen counts. Without target: the sum without the count as target. The two right-hand columns are computed from the public data with the released code; they do not establish the values that the authors' own data file would give. The first mean is the script's reporting line, over all 28 control schools; the second is over the 26 control schools of the analysis. Main effect: the school level main effect of Table S6. Interaction: the interaction with the proportion of referents of Table S7. Robust standard errors in parentheses.}\nQuantity & Published & As coded & Without target \\\\\nMean, 28 control schools & 0.20 & 0.197 & 0.154 \\\\\nMean, 26 control schools &  & 0.212 & 0.166 \\\\\nMain effect & $-0.06$ (0.03) & $-0.057$ (0.035) & $-0.053$ (0.029) \\\\\nInteraction & $-0.85$ (0.46) & $-0.841$ (0.463) & $-0.773$ (0.384) \\\\\n\\end{audittable}\n\nOn the public data the main effect and its standard error round to the printed values of Table S6, and the interaction is $-0.84$ where Table S7 prints $-0.85$. The script's reporting line (line 273) averages over all 28 control schools. Its value on the public data is 0.197, which rounds to the printed 0.20; the choice of schools in that average is the subject of \\fid{P037-N009}.", "problem": "The released code builds the outcome as the sum of seventeen administrative counts for each student, with every missing count set to zero: the thirteen categories of the SI list, two categories that the list omits (\\fid{P037-N018}), the H.I.B. count as offender of item 27, and the H.I.B. count as target of item 28. The count of item 28 does not record discipline of the student. The investigators' codebook defines the variable (\\texttt{hibt}) in these words: ``Since HIB incidents were reported by offender, if 3 offenders targeted one other student as part of one event, the target would be listed in 3 writeups, and this value would reflect that number of writeups (3), not the number of incidents (1)''. A student therefore scores on the outcome each time a writeup names the student as the target of harassment by others. On the public data this count contributes 903 of the 4,075 units of the outcome (22.2\\%). There are 526 students with a positive count as target, and 389 of them (74\\%) have zero in the other sixteen counts, so that their outcome is positive through writeups as target alone. The main text describes the level of the outcome as the number of times each student was disciplined (0.20 in control schools, 0.08 at 20\\% social referents), and the caption of Fig.~1 describes the colours and node sizes in the same way. These descriptions are inaccurate. The outcome is an index of administrative records of peer conflict, which counts disciplinary reports of a student and writeups that name the student as target. Item 28 also misstates its own unit: the count is a number of writeups, and it equals the number of incidents represented for that student only if each such incident contributes exactly one writeup naming that student.", "what_survives": "The survey outcomes (perceived norms, conversations with friends, wristbands), the randomization and the network definitions are not affected. The estimates for the disciplinary outcome remain estimates for a defined index of administrative records, and the printed 0.20 is reproduced to rounding on the public data as the mean of that index. Without the count as target, the two coefficients of the school-level regressions keep their sign and a similar size on the public data. This comparison does not establish that the published standard errors or statements of significance are unchanged, and the values come from the public data: the authors' own data file is not public, and ICPSR states that some variables of the release were changed. What changes is the description of the outcome and of its levels as each student's own discipline, in the Results and in the caption of Fig.~1, and the unit of item 28.\n\nThree points are not established. I did not recompute the network estimates of Fig.~3D and Table S5 or the drawing of Fig.~1 without the count as target. The check does not show that one incident was counted both under the code ``Bully/HIB'' (item 3) and under the offender count of item 27; on the public data 58 students have both counts positive, 26 only the first and 608 only the second, and these numbers do not link the counts to incidents. Removing the count as target does not turn the remaining sum into a count of distinct disciplinary actions."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N019", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Five lines of text from page 13 of the SI Appendix. A red box marks the words in percentage points in the phrase the slope of the average causal effect with respect to group composition in percentage points. A second red box marks the end of the parenthesis that follows: c percent is tau-zero plus beta c.", "caption": "SI Appendix, p. 13 (PDF p. 13), text after equation (2). The marked words give the unit of the slope $\\beta$ as percentage points, and the marked formula gives the effect at composition $c\\%$ as $\\tau_0+\\beta c$.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N020.png"}], "formal_release": null, "formal_statements": [], "headline": "The slope $\\beta$ is reported per unit of the proportion of social referents in the seed group, on the scale from 0 to 1, and not per percentage point.", "id": "P037-N020", "location": "SI Appendix, p. 13 (PDF p. 13 of the SI Appendix), text after equations (1) and (2); the same wording after equations (3) and (4), pp. 15–16, and in the note to Table S7 (PDF p. 42)", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``$C_{ij}$ is the composition of the seed group (\\hl{percentage} of seed group that is classified as a social referent)''\n\n``$\\beta$ represents the slope of the average causal effect with respect to group composition \\hl{in percentage points} (i.e., assuming that the slope is constant, the average causal effect of treatment at a given composition $c\\%$ is $\\tau_0+\\hl{\\beta c}$.)''\n\nNote to Table S7: ``the first derivative of the effect size with respect to the \\hl{percentage} of the group that were referents''", "corrected": "``$C_{ij}$ is the composition of the seed group (\\hl{proportion} of seed group that is classified as a social referent)'', and likewise for $C^{E}_{ij}$.\n\n``$\\beta$ represents the slope of the average causal effect with respect to group composition, \\hl{per unit of this proportion} (i.e., assuming that the slope is constant, the average causal effect of treatment at a given composition $c\\%$ is $\\tau_0+\\hl{\\beta c/100}$.)''\n\nNote to Table S7: ``the first derivative of the effect size with respect to the \\hl{proportion} of the group that were referents''", "intended_meaning": "The coefficients of Table S7 and the lines of Fig.~2 use the proportion of social referents in the seed group. The title of Table S7 says ``proportion'', and the horizontal axis of Fig.~2 is labelled ``Proportion of social referent seeds in anti-conflict group'', with ticks from 0.05 to 0.20. With the peer-conflict coefficients $\\tau_0=0.04$ and $\\beta=-0.85$ of Table S7, the effect at 20\\% social referents is $0.04-0.85\\times 0.20=-0.13$. The printed formula with $c=20$ would give $0.04-0.85\\times 20=-16.96$. The slope per percentage point is $\\beta/100=-0.0085$.\n\nThe released code uses the proportion. The first script computes the composition of the seed group and of the seed-eligible group as means of 0/1 indicators of social referents (lines 141--146). The school script enters these proportions in the regression (line 208) and draws the treatment line of Fig.~2 as the adjusted control mean plus $\\tau_0+\\beta p$, where $p$ is the proportion (lines 311--313 and 332--333). With these scripts adapted to the public ICPSR 37070 V2 data, I find $\\tau_0=0.0408$ and $\\beta=-0.841$ per unit of proportion, that is, $-0.00841$ per percentage point, and a fit with the percentage as predictor preserves fitted values within numerical precision. The public data do not reproduce the published fit exactly: the slope rounds to $-0.84$ where Table S7 prints $-0.85$, and the predicted outcome at 20\\% is 0.095 where the main text prints 0.08 (see \\fid{P037-N010}). The authors' own data file is not public, and ICPSR notes that some variables were changed for release; this note does not explain these differences."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N020", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of the SI paragraph beginning 'component probability, pi_i(d_k), is, by definition, equal to the expected proportion of treatment assignments that induce exposure d_k for student i.' Two red boxes mark the words 'achieve an arbitrarily precise estimate of pi_i via J = 10,000', which run over two lines. The sentence continues 'simulated random assignments of Z according to the experimental design', followed by the formula for the estimated probability.", "caption": "SI Appendix, p. 14 (supplement PDF p. 14), the paragraph that introduces the simulated exposure probabilities. The marked words attach “arbitrarily precise” to the fixed choice $J = 10{,}000$.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N021.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T22", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T22"}, {"id": "P037-T29", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T29"}, {"id": "P037-T30", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T30"}], "headline": "A fixed number of $10{,}000$ simulated assignments does not give an “arbitrarily precise” estimate of the exposure probabilities; on the public data, simulation noise changes whether 33 students pass the code’s probability rule, about 1% of those who pass.", "id": "P037-N021", "location": "SI Appendix, p. 14 (supplement PDF p. 14), paragraph on the exposure probabilities; the method is described again on SI p. 18. The corresponding passage of the article is on p. 568 (PDF p. 3).", "pdf_page": 14, "printed_page": "14", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T22", "P037-T29", "P037-T30"], "text": {"as_printed": "We achieve an \\hl{arbitrarily precise} estimate of $\\pi_i$ via $J = 10{,}000$ simulated random assignments of $\\mathbf{Z}$ according to the experimental design: \\ldots", "corrected": "We \\hl{estimate} $\\pi_i$ \\hl{using} $J = 10{,}000$ simulated random assignments of $\\mathbf{Z}$ according to the experimental design\\hl{; increasing $J$ can make the Monte Carlo approximation arbitrarily accurate at any prescribed confidence level.}\n\nThe formula for $\\widehat{\\pi}_i(d_k)$ that follows the colon is corrected separately in \\fid{P037-E003}, and the rule on the estimated probabilities is described in \\fid{P037-N025}.", "evidence": "The evidence is an exact counterexample to the printed sentence and a recalculation on the public data (ICPSR 37070 V2).\n\nFor the counterexample, suppose that the school is treated with probability $1/2$ and has 40 seed-eligible students in four blocks of 10, with five seeds drawn from each block. Let a student who is not eligible have ten eligible peers, five in each of two blocks, one of them a social referent. The student has no seed peer only if the five seeds of each of the two blocks are the five students who are not peers. Since $\\binom{10}{5} = 252$, the four exposure probabilities are\n\\[\n\\pi(d_C) = \\frac{1}{2}, \\qquad \\pi(d_S) = \\frac{1}{2 \\cdot 252^2} = \\frac{1}{127{,}008}, \\qquad \\pi(d_{NI}) = \\frac{31{,}751}{127{,}008}, \\qquad \\pi(d_I) = \\frac{1}{4}.\n\\]\nAll four are positive. An estimate from $10{,}000$ draws is a multiple of $1/10{,}000$. This implies that its error for $\\pi(d_S)$ is at least $1/127{,}008$, with equality only at the estimate zero. With independent draws the estimate is zero with probability $(127{,}007/127{,}008)^{10{,}000} \\approx 0.924$.\n\nIn a reconstruction on the public data, I computed the exposure probabilities from combinatorial formulas and also estimated them from $10{,}000$ draws by adapting the authors' scripts, with their random seed (123) and the sampling method of the R versions of the time (before 3.6). Before the other exclusions, 2,921 students have all four combinatorial probabilities above $0.01$. With the estimated probabilities, 2,908 do: 23 of the 2,921 fail, and 10 other students pass. The two sets differ in 33 students. With the sampling method of current R versions and the same seed, 2,918 students pass and the two sets differ in 25. The reconstruction matches the published totals of 24,191 students in 56 schools. ICPSR reports changes to some variables; the authors' own data file and their saved draws are not public, so these counts describe the public data and not the original run.", "problem": "An estimate from a fixed number of simulated assignments (draws) has a fixed precision. SI p.~18 states the matter correctly: an arbitrarily precise estimate can be achieved by simulated assignments, and the study uses $J = 10{,}000$. The sentence on p.~14 attaches the precision to that fixed $J$. The precision matters in the application, because the authors' code keeps a student in the network analyses only if each of the four estimated exposure probabilities is greater than $0.01$ (\\fid{P037-N025}). For a student whose probability is near $0.01$, simulation noise can change the comparison.", "what_survives": "The statement of SI p.~18 on convergence and the definition of $\\pi_i(d_k)$ stand; the estimator's formula needs the separate correction in \\fid{P037-E003}. No student with probability zero of some exposure passes the rule in either run. I have not established a change to any published estimate; on the public data, using the historical sampler, the 16 exposure means computed with the simulated and combinatorial probabilities differ by at most $0.0281$."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N021", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of panel B of Fig. 2, 'Perceived social norm against conflict', with a red control line, a black treatment line and dashed bands. A red box marks the vertical axis label, 'Perceived Social Norms Against Conflict (First Principal Component, Higher Indicates Norm is More Anti-conflict)'.", "caption": "Fig. 2B, p. 569 (PDF p. 4). The marked axis label calls the index the first principal component and says that higher values are more anticonflict.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N023.png"}, {"alt": "Crop of Table S3, 'Factor loadings for Prescriptive norm items (behaviors perceived to be desirable at school)', with columns Survey Item, Wave 1 and Wave 2 and rows Item 1 to Item 7 and Item 9 to Item 13. A red box marks the two columns of loadings. The loadings are negative for Items 1, 2, 7, 10, 11, 12 and 13 and positive for Items 3, 4, 5, 6 and 9; Item 13 has a loading in Wave 2 only.", "caption": "SI Appendix, Table S3, p. 38 (supplement PDF p. 38). The marked loadings are those of the first principal component before its sign is reversed: they are negative on the items worded against conflict.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N023-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The SI does not state how the prescriptive-norm index is scored: the code uses the items as they were administered and negates the whole first principal component.", "id": "P037-N023", "location": "Figs. 2B and 3B, axis labels, pp. 569–570 (PDF pp. 4–5); SI Appendix, list of prescriptive-norm items, p. 9, and Table S3, p. 38 (supplement PDF pp. 9 and 38)", "pdf_page": 4, "printed_page": "569–570", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Fig.~2B, axis label: ``Perceived Social Norms Against Conflict (First Principal Component, Higher Indicates Norm is More Anti-conflict)''. The axis of Fig.~3B has the same label after the words ``Predicted Mean of''.\n\n\\begin{audittable}{lrr}{Table S3 (SI p.~38), ``Factor loadings for Prescriptive norm items (behaviors perceived to be desirable at school)''.}\nSurvey Item & Wave 1 & Wave 2 \\\\\nItem 1 & $-0.29$ & $-0.27$ \\\\\nItem 2 & $-0.26$ & $-0.26$ \\\\\nItem 3 & $0.32$ & $0.26$ \\\\\nItem 4 & $0.30$ & $0.27$ \\\\\nItem 5 & $0.28$ & $0.23$ \\\\\nItem 6 & $0.27$ & $0.25$ \\\\\nItem 7 & $-0.29$ & $-0.28$ \\\\\nItem 9 & $0.09$ & $0.10$ \\\\\nItem 10 & $-0.36$ & $-0.34$ \\\\\nItem 11 & $-0.34$ & $-0.33$ \\\\\nItem 12 & $-0.40$ & $-0.38$ \\\\\nItem 13 & & $-0.38$ \\\\\n\\end{audittable}\n\nSI p.~9, in the section ``Survey wording and school administrative record description'':\n\nPrescriptive norms scale (\\hl{asked at pretest and posttest}, answered using pictorial scale featured above): ``How many students think it is\\ldots''\n\n\\begin{itemize}\n\\item 3. \\hl{Not funny} to mess with or pick on others\n\\item 4. \\hl{Not OK} to say mean things to girl because of what she does with boys\n\\item 5. \\hl{Not funny} to post online or text mean things\n\\item 6. \\hl{Not OK} to use racial, ethnic jokes\n\\item \\hl{8.} \\hl{OK} to tell adults about conflict\n\\end{itemize}", "corrected": "The SI should identify the list on p.~9 as labels and retain the questionnaire's numbering:\n\n``Prescriptive-norm items collected at pretest and posttest are summarized below by \\hl{labels expressed in the anti-conflict direction, rather than by the administered wording}. \\hl{The administered PN3--PN6 and PN9 have the opposite polarity from these labels.} PN8 concerns academic effort and is omitted here. \\hl{The numbers below retain the PN numbers used in Table S3}; PN13 was collected only in Wave 2.''\n\nThe twelve labels should be numbered 1, 2, 3, 4, 5, 6, 7, \\hl{9, 10, 11, 12, 13}. The same wording and numbering repair is described under \\fid{P037-E006}.\n\nAdd the scoring rule with Table S3 or with that list:\n\n``The prescriptive-norm index uses the item responses as administered; no item is reverse-coded. In each wave, a principal components analysis of the covariance matrix is fitted to the students with complete responses, after entries greater than 10 are set to missing. The index is \\hl{the negative of the first principal component} whose loadings are printed in Table S3,\n\\[\nY_w = \\hl{-}\\,u_w^{\\top}(x_w - \\mu_w),\n\\]\nwhere $x_w$ is the vector of a student's complete responses in wave $w$, $u_w$ is the \\hl{unrounded first principal loading vector, oriented with the signs shown in Table S3}, and $\\mu_w$ is the vector of item means among the complete cases. \\hl{The index's coefficients have the opposite signs to Table S3}, and higher values indicate more anti-conflict norms. Missing baseline scores are replaced by the mean baseline score; missing posttest scores remain missing.''\n\nAnyone who runs the released scripts should fix the orientation explicitly, by requiring a positive coefficient of the index on item 1. The scripts' unconditional change of sign gives the opposite orientation under the R 4.4.2 default of \\texttt{princomp}.", "evidence": "I read the authors' scripts and ran both of their scoring branches, for the school analysis and for the network analysis, on the public data (ICPSR 37070 V2), adapting its variable coding and missing values to the scripts' input format. After the authors' filter the public data have 24,191 students in 56 schools, and the two principal components analyses use 19,630 complete cases in Wave 1 and 19,293 in Wave 2. With \\texttt{fix\\_sign=FALSE} in R 4.4.2, all 23 loadings agree with Table S3 at two decimals. Let $u_w$ be the unrounded first principal loading vector of wave $w$ with the signs of Table S3, let $x_w$ be a student's vector of item answers and let $\\mu_w$ be the vector of item means among the complete cases. For complete responses, both branches return exactly\n\\[\nY_w = -\\,u_w^{\\top}(x_w - \\mu_w) .\n\\]\nSince $u_w$ is negative on the items worded against conflict and positive on the other five, $Y_w$ increases with every move toward an anticonflict answer. Its correlation with item 1 is $0.595$ in Wave 1 and $0.607$ in Wave 2. If $C u = \\lambda u$ for the covariance matrix $C$, then $C(-u) = \\lambda(-u)$ and $(-u)^{\\top} C (-u) = u^{\\top} C u$. This implies that $-u$ is a first principal direction with the same explained variance. A missing baseline score is replaced by the mean baseline score (4,561 students on the public data), and a missing posttest score stays missing.\n\nWith this orientation, the school-level effect on the public data is $-0.014$, where Table S6 prints $-0.01$. The four exposure means from a run of the network analysis on the public data are $0.127$, $-0.823$, $-0.456$ and $0.030$ for control schools, no seed exposure, nonreferent exposure and referent exposure, on 2,273 students. The contrasts of referent exposure with the other two exposures in treatment schools are positive, as in Fig.~3B. These means are not those of Table S5, which prints $0.141$, $-0.825$, $-0.468$ and $0.007$; the authors' own files are not public, and ICPSR states that some variables in the release were changed. Under the R 4.4.2 default of \\texttt{princomp}, the rounded loadings are the negatives of Table S3 and every one of these estimates changes sign.", "problem": "The axes of Figs.~2B and 3B call the index the first principal component of the prescriptive-norm items and say that higher values are more anticonflict. Table S3 prints loadings for the same items. The SI does not say how the index is computed from the items, and the loadings cannot be read against the axes without that rule. The authors' code supplies it. The item columns enter as they were administered, and no item is reverse-coded. In each wave, a principal components analysis of the covariance matrix is fitted to the students with complete answers. The first component is then multiplied by $-1$.\n\nTo obtain the figures' stated anticonflict orientation from the component in Table S3, the index takes its negative. A component's global sign is arbitrary, so its negative is still a first principal component. The printed loadings are negative on the items that are worded against conflict and positive on items 3 to 6 and 9, which the questionnaire words the other way. With the printed signs the component decreases when a student moves toward the anticonflict answer on any item, and the index increases. The list of these items on SI p.~9 presents the five anticonflict labels as administered wording and numbers the items differently from the table (\\fid{P037-E006}). It omits item 8, on academic effort, and numbers items 9 to 13 as 8 to 12. Against that list, the loadings have mixed signs on items that all point the same way.\n\nThe sign also depends on the version of R when the code is run today. Since version 3.5.0, R's \\texttt{princomp} chooses the sign that makes the first loading non-negative, and the first loading of Table S3 is negative. The scripts multiply by $-1$ whatever sign the component has. Under the R 4.4.2 default, the same code returns a pro-conflict index and estimates of the opposite sign. This is established for a run on the public data. It is not shown for the published analysis.", "what_survives": "No published loading, estimate or figure is shown to be wrong. On the public data, the index scored as the code scores it, with the sign convention that reproduces Table S3, has the anticonflict orientation that the axes state, and the school and network results for norms have the published direction. The authors' own files and their original run are not available; my public-data recalculations do not establish their original standard errors or $P$ values for norms. The outcomes for disciplinary reports, wristbands and talking with friends do not use the index."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N023", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of eight lines of the Methods and Materials, beginning 'We considered four conditions of exposure to the seed students in each school'. A red box marks the end of that paragraph, over two lines: 'restricted our network-based analyses to the subpopulation of 2,451 students who had a positive probability of falling into all four levels of exposure.' The last line of the crop begins the next paragraph, 'Twenty-four percent of seed students did not accept our invitation to join'.", "caption": "Methods and Materials, p. 568 (PDF p. 3), right column. The marked sentence restricts the network analyses to 2,451 students with a positive probability of all four levels of exposure.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N025.png"}, {"alt": "Crop of three lines of the SI: 'We restrict ourselves to the subpopulation of students (n=2451) for which there exists some probability that students could receive all four exposures: 0 < pi_i(d_k) < 1 for k in {C, S, NI, I}.' Red boxes mark '(n=2451)' and the inequality '0 < pi_i(d_k) < 1'.", "caption": "SI Appendix, p. 18 (supplement PDF p. 18). The marked terms give the size of the population, $n = 2451$, and its definition by $0 < \\pi_i(d_k) < 1$ for the four exposures.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N025-b.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T21", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T21"}, {"id": "P037-T22", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T22"}, {"id": "P037-T39", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T39"}], "headline": "The network results are not computed on one population of 2,451 students with positive probability of all four exposures: for each outcome the authors' code keeps the students whose four estimated exposure probabilities exceed $0.01$, who were not seeds in the realised assignment and whose outcome and preprocessed pretreatment score are available, with a further school-mean cutoff for disciplinary reports, and it averages over that set.", "id": "P037-N025", "location": "Methods and Materials, p. 568 (PDF p. 3), right column; SI Appendix, pp. 14, 18 and 19, and the note to Table S5, p. 40", "pdf_page": 3, "printed_page": "568", "reaches": [{"effect": "same_error_restated", "effect_text": "a corresponding statement is printed there", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T21", "P037-T22", "P037-T39"], "text": {"as_printed": "Article, p.~568: ``In this particular analysis, we restricted our network-based analyses to the subpopulation of \\hl{2,451 students who had a positive probability} of falling into all four levels of exposure.''\n\nSI Appendix, p.~18: ``We restrict ourselves to the subpopulation of students \\hl{(n=2451)} for which there exists some probability that students could receive all four exposures: $\\hl{0 < \\pi_i(d_k) < 1}$ for $k \\in \\{C,S,NI,I\\}$.''\n\nSI Appendix, note to Table S5, p.~40: ``\\ldots\\ computed on the subpopulation of \\hl{2,451 students who had a positive probability} of falling into all four levels of exposure.''", "corrected": "Article, p.~568: ``In this particular analysis, \\hl{for each outcome} we restricted our network-based analyses to the \\hl{students whose estimated probability of falling into each of the four levels of exposure exceeded 0.01, who were not seeds in the realised assignment and whose outcome and preprocessed pretreatment score were available; for disciplinary reports, only schools whose mean count exceeded 0.02 were used. The regression was fitted and its predictions averaged over the same students}.''\n\nSI Appendix, p.~18: ``\\hl{For each outcome,} we restrict ourselves to the students \\hl{with estimated probabilities} $\\hl{\\widehat{\\pi}_i(d_k) > 0.01}$ for $k \\in \\{C,S,NI,I\\}$\\hl{, who were not directly treated seeds in the realised assignment and whose outcome and preprocessed pretreatment score are available; for disciplinary reports, only schools whose mean count exceeds 0.02 are used. The regression is fitted and its predictions averaged over the same students.}''\n\nThe same restriction replaces the population descriptions on SI pp.~14 and 19 and in the note to Table S5. The single number 2,451 on p.~568, on SI pp.~18 and 19 and in the note to Table S5 should be replaced by the number of students of the original run for each outcome. On the public data these numbers are 2,457 (talk), 2,273 (norms), 2,847 (wristband) and 2,495 (disciplinary reports); they are not the original numbers. In the estimator on SI p.~19, sum over the outcome's analysis set and set $N_s$ equal to its size (\\fid{P037-N033}). If the stated population is to be kept instead, the analysis has to be carried out on it, with a treatment of the students whose outcome is missing; I have not done that analysis.", "evidence": "The evidence is a reading of the authors' code and a recalculation on the public data (ICPSR 37070 V2). In the network script (\\texttt{3. Network results.R}), line 190 sets the value $0.01$, line 195 defines the analysis set by conditions (i)--(iv) and pretreatment-score availability, line 216 fits the regression on it, and lines 223--226 average the fitted values over it. For norms the code imputes missing pretreatment scores; for the other outcomes it uses a constant. The availability condition removes no student on the public data.\n\nAn exact example shows that condition (i) differs from positive probability. Suppose that a student who is not eligible has six eligible peers, one of them a social referent, in a block of 12 eligible students from which six seeds are drawn. There are $\\binom{12}{6} = 924$ equally likely draws, and the student has no seed peer in exactly one of them. With a treatment probability of $1/2$ for the school, the probabilities of the control school, of no seed peer, of a non-referent seed peer only and of a referent seed peer are $1/2$, $1/1848$, $461/1848$ and $1/4$. All are positive, and the second is below $0.01$.\n\nI recalculated these counts on the public data using the scripts, after converting its variable names and codes to the form that the scripts read. I used the random seed of the code and the sampling method of the R versions of the time (before 3.6).\n\n\\begin{audittable}{lrrrr}{Students of the public data at each step from the stated population to the analysis set, for the four outcomes. The first row uses probabilities computed exactly from the design; the article prints 2,451 for every outcome.}\n & Talk & Norms & Wristband & Conflict \\\\\nPositive probability of all four exposures & 3,120 & 3,120 & 3,120 & 3,120 \\\\\n(i) All four estimated probabilities above $0.01$ & 2,908 & 2,908 & 2,908 & 2,908 \\\\\n(ii) and not a seed in the realised assignment & 2,847 & 2,847 & 2,847 & 2,847 \\\\\n(iii) and outcome observed & 2,457 & 2,273 & 2,847 & 2,847 \\\\\n(iv) and school kept for disciplinary reports: the analysis set & 2,457 & 2,273 & 2,847 & 2,495 \\\\\nSchools represented in the analysis set & 56 & 55 & 56 & 49 \\\\\n\\end{audittable}\n\nWith the exact probabilities in condition (i), 2,921 of the 3,120 students pass. Using exact probabilities, the cutoff removes 199 students of the positive-probability population on the public data; the further membership changes from simulation noise are the subject of \\fid{P037-N021}. Condition (ii) removes 61 students. Seed-eligible students who were not drawn as seeds stay in the analysis set: 207 of the 2,847 students of the wristband analysis are seed-eligible. The code holds the analysis set fixed in the $1{,}000$ simulated assignments from which the standard errors are computed. Since no student of the analysis set has the directly treated exposure as observed exposure, the imputation of potential outcomes on SI pp.~19--20 never needs a mean for that exposure.\n\nThe public data reproduce the published 24,191 students in 56 schools. None of the four analysis sets has 2,451 students. The authors' own data file and their saved draws are not public, and ICPSR states that some variables of the public data were changed. The four numbers of the original run are not recovered, and the counts in the table are counts for the public data.", "problem": "The article, the SI Appendix (pp.~14, 18 and 19) and the note to Table S5 describe one population for every network result: the students with positive probability of each of the four exposures, 2,451 in number. This population is fixed before the assignment. The authors' code computes the results on another set, which I call the analysis set of the outcome. For each outcome the code keeps a student when the preprocessed pretreatment score is available and four conditions hold: (i) each of the four exposure probabilities, as estimated from $10{,}000$ simulated assignments, is greater than $0.01$; (ii) the student was not a seed in the realised assignment; (iii) the outcome is observed; and (iv) for disciplinary reports, the school's mean count exceeds $0.02$ (\\fid{P037-N007}). The regression is fitted on the analysis set, and the fitted values are averaged over the same set.\n\nCondition (i) is stricter than positive probability. Condition (ii) depends on the realised assignment, and conditions (iii) and (iv) depend on the outcome. The SI Appendix sets the directly treated exposure aside (p.~17), which accounts for leaving the outcomes of the realised seeds out of the fit. It does not say that these students are also left out of the set over which the means are averaged. The reported means are averages over a set that depends on the assignment and on the outcome, so the number of students must be reported for each outcome.", "what_survives": "Positive probability as a requirement, the four conditions of exposure and the reported network estimates are untouched. No estimate, $P$ value or conclusion of the article is shown to change. The school-level regressions (Tables S6 and S7) do not use the analysis sets. What changes is the population to which Fig.~3 and Table S5 refer, and its size for each outcome: describing the rule correctly replaces the fixed population of the article by a set that depends on the assignment and on the outcome."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N025", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Fig. 2, four panels with its caption at the right. Each panel plots a horizontal red line for control schools and a sloped black line for treatment schools against the proportion of social referent seeds in the anti-conflict group, with labelled ticks from 0.05 to 0.20, each between two dashed curves. Red boxes mark the key of each panel, 'Control schools' predicted mean (and 95 percent CI)' and 'Treatment schools' predicted mean (and 95 percent CI)', and the words of the caption 'and are surrounded by 95 percent confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit'.", "caption": "Fig. 2 and its caption, p. 569 (PDF p. 4). The marked sentence of the caption and the marked keys of the four panels call the dashed bands 95% confidence intervals for the predicted means of control and treatment schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N026.png"}, {"alt": "Crop of the first page of Fig. S6, 'Re-analysis of Fig. 2 and 3 using betweenness in place of indegree': four panels, (a) Friends talked to me about conflict, (b) Perceived social norm against conflict, (c) Wearing anti-conflict wristband and (d) Peer conflict (disciplinary records), each with a red control line, a black treatment line and dashed bands. Red boxes mark the key of each panel, 'Control schools' predicted mean (and 95 percent CI)' and 'Treatment schools' predicted mean (and 95 percent CI)', and the words of the caption 'and are surrounded by 95 percent confidence intervals estimated using heteroskedasticity-robust standard errors for the regression fit'.", "caption": "SI Appendix, Fig. S6, school-level panels (a)–(d), p. 33 (supplement PDF p. 33). The same keys and the corresponding sentence of the caption are marked; a separate plotting loop uses the same band formulas.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N026-b.png"}, {"alt": "Four panels, A to D, one for each outcome of Fig. 2: friends talked to me about conflict (score from 0 to 3), perceived social norm against conflict (first principal component), wearing anti-conflict wristband (proportion reporting wearing) and peer conflict (reports per student). In each panel there are four positions: control schools, and treatment schools at 5, 10 and 20 percent social referent seeds. At each position an orange interval with a square marker, labelled as drawn, stands beside a blue interval with a round marker, labelled as captioned; both are centred on the same predicted mean. The orange interval for control schools is shorter than the blue one in all four panels. For treatment schools the orange interval is longer than the blue one at ten of the twelve positions, and slightly shorter in panel B at 20 percent and in panel D at 5 percent. A note under each panel title says that the line is fitted on 56 schools, or 49 in panel D, and the drawn treatment band on 56.", "caption": "The bands of Fig. 2 on the public data (ICPSR 37070 V2), at control schools and at treatment schools with 5%, 10% and 20% social referent seeds. Orange squares: the interval as the plotting code draws it. Blue circles: the interval for the predicted mean, computed with the robust covariance matrix of the fit of the line, as the caption describes. The half-widths are $1.96$ times the unrounded standard errors summarised in the table above.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N026-bands.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T23", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T23"}, {"id": "P037-T24", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T24"}, {"id": "P037-T40", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T40"}], "headline": "The dashed bands of Fig. 2, and of the school-level panels of Fig. S6, are not the 95% confidence intervals for the two predicted means that the caption and the keys describe: the band around the treatment line uses the standard error of the treatment–control contrast, and the band around the control line uses the variance of the control schools' residuals divided by 56.", "id": "P037-N026", "location": "Fig. 2, panels A–D, keys and caption, p. 569 (PDF p. 4); SI Appendix, Fig. S6, school-level panels (a)–(d) and caption, p. 33 (supplement PDF p. 33)", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T23", "P037-T24", "P037-T40"], "text": {"as_printed": "Fig.~2, caption: ``Lines represent estimates of the conditional average potential outcome with respect to the proportion of social referent seeds, and are \\hl{surrounded by 95\\% confidence intervals estimated using heteroskedasticity-robust SEs for the regression fit}.''\n\nFig.~2, key in each panel: ``Control schools' predicted mean \\hl{(and 95\\% CI)}'' and ``Treatment schools' predicted mean \\hl{(and 95\\% CI)}''.\n\nFig.~S6, caption (SI p.~33): ``Lines represent estimates of the conditional average potential outcome with respect to the proportion of high betweenness seeds, and are \\hl{surrounded by 95\\% confidence intervals estimated using heteroskedasticity-robust standard errors for the regression fit}.''", "corrected": "The caption and the keys can stand if the bands are redrawn. For each panel, let $\\widehat b$ and\n\\[\n\\widehat V = (X^{\\top}X)^{-1} X^{\\top} \\operatorname{diag}(e_1^2, \\ldots, e_n^2)\\, X\\, (X^{\\top}X)^{-1}\n\\]\nbe the coefficients and the heteroskedasticity-robust covariance matrix of the regression that gives the lines, where $X$ contains its \\hl{linearly independent design columns} and $e_i$ are its residuals, fitted on the schools of that panel's analysis (49 for peer conflict, 56 for the other outcomes). With $\\bar z$ the average of the block indicators and covariates over those schools, $\\ell_0 = (1, 0, 0, \\bar z^{\\top})$ and $\\ell_1(c) = (1, 1, c, \\bar z^{\\top})$, the bands are\n\\[\n\\widehat m_0 \\pm 1.96\\sqrt{\\hl{\\ell_0 \\widehat V \\ell_0^{\\top}}}, \\qquad \\widehat m_1(c) \\pm 1.96\\sqrt{\\hl{\\ell_1(c)\\, \\widehat V\\, \\ell_1(c)^{\\top}}} ,\n\\]\nin place of the residual-variance formula for control and the contrast standard error from the plotting refit for treatment. For Fig.~S6, $\\widehat b$, $\\widehat V$, $\\bar z$ and the control mean $\\widehat m_0$ all come from the betweenness regression. These are approximate pointwise intervals under the normal approximation that the paper uses, and no new assumption is needed.\n\nOn the public data the corrected standard errors are those in the last column of the table above. In panel D, for example, the half-width of the control band becomes $0.0445$ in place of $0.0287$, and the half-width of the treatment band at $c = 0.10$ becomes $1.96 \\times 0.0217$ in place of $1.96 \\times 0.0296$. These are values on the public data; the values from the authors' own files are not available.", "evidence": "The evidence is a derivation with an exact counterexample, a recalculation on the public data, and measurements of the printed figures from rendered pages.\n\n\\emph{Derivation.} For a fixed fitting sample and covariates, since\n\\[\n\\tau_0 T_i + \\beta\\, T_i C_i = (\\tau_0 + \\beta c)\\, T_i + \\beta\\, T_i (C_i - c),\n\\]\nthe refit with $C_i - c$ is the same regression in other coordinates. It has the same fitted values and residuals, its coefficient on $T_i$ is $\\widehat\\tau_0 + c\\widehat\\beta$, and the robust variance of that coefficient is $d(c)\\widehat V d(c)^{\\top}$. The variance of the treatment mean is\n\\[\n\\ell_1 \\widehat V \\ell_1^{\\top} = \\ell_0 \\widehat V \\ell_0^{\\top} + d \\widehat V d^{\\top} + 2\\,\\ell_0 \\widehat V d^{\\top},\n\\]\nwhich differs from $d \\widehat V d^{\\top}$ by the variance of the control mean and twice a covariance. Since the covariance may be negative, the drawn band may be wider or narrower than the captioned one.\n\n\\emph{Counterexample.} The synthetic example has 56 schools and no covariates: 14 control schools with $Y = 0$ and 14 with $Y = 1/5$; 7 treatment schools with $(C, Y) = (0, 1/10)$, 7 with $(0, 3/10)$, 7 with $(1/5, 2/5)$ and 7 with $(1/5, 3/5)$. The least squares fit of $Y = \\alpha + \\tau_0 T + \\beta\\, T C$ is $(\\widehat\\alpha, \\widehat\\tau_0, \\widehat\\beta) = (1/10, 1/10, 3/2)$, every residual is $\\pm 1/10$, and the robust covariance matrix is\n\\[\n\\widehat V = \\begin{pmatrix} 1/2800 & -1/2800 & 0 \\\\ -1/2800 & 3/2800 & -1/280 \\\\ 0 & -1/280 & 1/28 \\end{pmatrix} .\n\\]\nAt $c = 1/10$ the two predicted means are $1/10$ and $7/20$. Each has variance $1/2800$, and the contrast has variance $1/1400$, with covariance $-1/2800$ between the control mean and the contrast. The code's treatment band uses $1/1400$, twice the variance of the treatment mean. Its control band uses $s_0^2/56 = (28/100)/(27 \\cdot 56) = 1/5400$, which is $14/27$ of the variance of the control mean. All of these values are exact rational numbers.\n\n\\emph{Recalculation.} I recalculated the authors' school regressions on the public data (ICPSR 37070 V2; 24,191 students in 56 schools after the authors' filter), using an adapter for the public release and current R packages, and compared the standard error that the plotting code uses with the robust standard error of the predicted mean from the fit of each line. The table reports both for the four panels of Fig.~2, and the figure below draws the corresponding intervals. In panel D at $c = 0.05$, the drawn standard error is $0.0374$, where the contrast from the 49-school fit of the line has standard error $0.0482$. For Fig.~S6(d) at $c = 0.20$, the drawn standard error is $0.0476$ and that of the predicted mean from the betweenness fit is $0.0656$.\n\n\\begin{audittable}{lrr}{Standard errors behind the bands of Fig.~2, from the authors' school script on the public data: as the plotting code computes them, and for the predicted mean from the fit of the line. Multiply by $1.96$ for the half-width of a band.}\nPanel and line & As drawn & For the predicted mean \\\\\nA, control & $0.0082$ & $0.0122$ \\\\\nA, treatment at $c = 0.05$ & $0.0229$ & $0.0215$ \\\\\nA, treatment at $c = 0.10$ & $0.0174$ & $0.0129$ \\\\\nA, treatment at $c = 0.20$ & $0.0338$ & $0.0293$ \\\\\nB, control & $0.0343$ & $0.0534$ \\\\\nB, treatment at $c = 0.05$ & $0.1076$ & $0.0772$ \\\\\nB, treatment at $c = 0.10$ & $0.0807$ & $0.0545$ \\\\\nB, treatment at $c = 0.20$ & $0.1164$ & $0.1189$ \\\\\nC, control & $0.0040$ & $0.0055$ \\\\\nC, treatment at $c = 0.05$ & $0.0144$ & $0.0132$ \\\\\nC, treatment at $c = 0.10$ & $0.0110$ & $0.0094$ \\\\\nC, treatment at $c = 0.20$ & $0.0205$ & $0.0199$ \\\\\nD, control & $0.0146$ & $0.0227$ \\\\\nD, treatment at $c = 0.05$ & $0.0374$ & $0.0382$ \\\\\nD, treatment at $c = 0.10$ & $0.0296$ & $0.0217$ \\\\\nD, treatment at $c = 0.20$ & $0.0464$ & $0.0413$ \\\\\n\\end{audittable}\n\n\\emph{Measurement of the printed figure.} The authors' own analysis files are not public, and ICPSR states that some variables in the release were changed. To connect the code to the printed figure, I compared the bands that the code draws on the public data with the bands as printed in the two peer-conflict panels. In the published panel D, the half-width of the control band measures $0.0289$, and the half-widths of the treatment band at $c = 0.05$, $0.10$ and $0.20$ measure $0.0744$, $0.0572$ and $0.0917$. From the code on the public data, I obtain $0.0287$ and $0.0733$, $0.0579$ and $0.0910$. In the published Fig.~S6(d), the measured half-widths are $0.0302$ and $0.0758$, $0.0559$ and $0.0938$, against $0.0302$ and $0.0755$, $0.0576$ and $0.0933$ from the code. Each pair agrees within $0.002$, the tolerance of the measurement. These measurements support the code's construction in the two peer-conflict panels. The captioned interval for the control mean of panel D would have half-width $0.0445$ on the public data.", "problem": "Each panel of Fig.~2 comes from a school-level regression of the outcome $Y_i$ on the treatment indicator $T_i$, its product with the proportion $C_i$ of social referents among the seeds, block indicators and school covariates $z_i$ (SI equation (2), with the adjustments of SI p.~14):\n\\[\nY_i = \\alpha + \\tau_0 T_i + \\beta\\, T_i C_i + z_i^{\\top}\\eta + u_i .\n\\]\nAfter omitting redundant design columns, let $\\bar z$ be the average of $z_i$ over the schools in the fit, and write $\\ell_0 = (1, 0, 0, \\bar z^{\\top})$, $\\ell_1(c) = (1, 1, c, \\bar z^{\\top})$ and $d(c) = \\ell_1(c) - \\ell_0 = (0, 1, c, 0^{\\top})$. The red line is the predicted control mean $\\widehat m_0 = \\ell_0 \\widehat b$, where $\\widehat b$ is the vector of estimated coefficients, and the black line is the predicted treatment mean $\\widehat m_1(c) = \\ell_1(c)\\widehat b$ at proportion $c$. The caption and the keys say that the dashed curves are 95\\% confidence intervals for these two means, computed from the robust covariance matrix $\\widehat V$ of the fit. Such intervals are\n\\[\n\\widehat m_0 \\pm 1.96\\sqrt{\\ell_0 \\widehat V \\ell_0^{\\top}}, \\qquad \\widehat m_1(c) \\pm 1.96\\sqrt{\\ell_1(c)\\, \\widehat V\\, \\ell_1(c)^{\\top}} .\n\\]\nThe plotting code draws something else, in four respects.\n\n\\begin{enumerate}\n\\item The treatment band. For each $c$ the code refits the regression with $C_i$ replaced by $C_i - c$ and takes the robust standard error of the coefficient on $T_i$. When the refit uses the same schools and covariates as the line, that coefficient is $\\widehat\\tau_0 + c\\widehat\\beta$, the difference between the two lines, and its variance is $d(c)\\widehat V d(c)^{\\top}$. The band uses a standard error of the contrast, drawn around the treatment line. It treats the control mean as known.\n\\item The control band. The code uses $1.96\\sqrt{s_0^2/56}$, where $s_0^2$ is the sample variance of the residuals of the control schools. This is not $1.96\\sqrt{\\ell_0 \\widehat V \\ell_0^{\\top}}$, and the divisor is 56 although the fit has 28 control schools, or 26 for peer conflict on the public data.\n\\item Panel D. The line is fitted on the 49 schools of the conflict analyses (\\fid{P037-N007}), but the refit that gives its treatment band uses all 56 schools.\n\\item Fig.~S6. The refit behind the treatment bands controls for the share of social referents by indegree in the seed-eligible group, where the fit of the lines controls for the share by betweenness. The bands of Fig.~S6 share the two variance errors, panel (d) also uses the mismatched sample, and its lines are placed on the control mean of the indegree regression.\n\\end{enumerate}\n\nThe bands are not wrong in one direction. On the public data, the control band is narrower than the captioned interval in every panel. At the three proportions checked (5\\%, 10\\% and 20\\%), the treatment band is wider than the captioned interval in ten of the twelve cases and narrower in two. The Results cite Fig.~2 for the school-level effects, and a judgement of uncertainty from the overlap or the width of these bands does not have the meaning that the caption gives it. The coefficients, their standard errors and the tests are computed elsewhere in the code and do not use these formulas.", "what_survives": "The lines of Fig.~2 do not change. The coefficients for treatment and for its interaction with the proportion of social referents, their robust standard errors, the Wald tests and the bootstrap calculations are computed before the plotting, and none of them uses the band formulas. The peer-exposure analyses of Fig.~3 use other calculations. In Fig.~S6 the slopes and the treatment coefficients stay as they are, but the plotted levels move slightly when the lines are placed on the control mean of the betweenness regression: for peer conflict on the public data that mean is $0.2171$, where the plotting code uses $0.2225$ from the indegree regression. No reading of a panel is shown to reverse. This check does not establish the coverage of any of these intervals; the behaviour of the robust standard errors with 49 schools is the subject of \\fid{P037-N006}. The cells of Table S7 that differ from the fit on the public data are recorded under \\fid{P037-N011}."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N026", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of seven lines of the left column of the Results, ending 'twice the average effect of the anticonflict groups.' A red box marks the parenthesis '(proportions ranged from 0 to 37 percent)' in the sentence 'When 20 percent of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to 37 percent), we estimate that each student was disciplined on average 0.08 times during the year'.", "caption": "Results, second paragraph, p. 569 (PDF p. 4), left column. The marked parenthesis gives the range of the proportion of social referent seeds.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N027.png"}], "formal_release": null, "formal_statements": [], "headline": "Among the anticonflict groups of the 28 treatment schools, the proportion of social referent seeds ranges from 0 to 25% on the public data, not from 0 to 37%; only three of the 28 groups reach the 20% at which the sentence evaluates its projection.", "id": "P037-N027", "location": "Results, second paragraph, p. 569 (PDF p. 4), left column, parenthesis “(proportions ranged from 0 to 37%)”", "pdf_page": 4, "printed_page": "569", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds (proportions ranged from 0 to \\hl{37\\%}), we estimate that each student was disciplined on average 0.08 times during the year, a reduction of 60\\% compared with control schools and twice the average effect of the anticonflict groups.", "corrected": "When 20\\% of a treatment school's anticonflict group is composed of social referent seeds (\\hl{in the public replication data,} proportions \\hl{among assigned seeds in the 28 treatment schools} ranged from 0 to \\hl{25\\%}), we estimate \\ldots\n\nThe qualifier ``in the public replication data'' can be dropped if the authors' own file gives the same range. A range over the lottery groups of all 56 schools would have to be labelled as such; on the public data it is 0 to $35.7\\%$.", "evidence": "The evidence is a recalculation on the public data (ICPSR 37070 V2). I executed the authors' code for the nomination network with an adapter for the public file and applied their rule for social referents: indegree at or above the 90th percentile of the student's school, ties included. On the public data, I find 28 treatment schools with 728 assigned seeds, the published count. The proportions were then compared as exact fractions. Among the treatment schools the smallest is $0$ and the largest is $5/20$. Among all 56 schools the largest is $10/28$.\n\nThe source of the printed 37\\% is not established. The authors' own analysis file is not public, and ICPSR states that some variables in the release were changed; its notes list no change to the variables used here. The range of 0 to 25\\% is the range on the public data.", "problem": "The sentence concerns the anticonflict groups of treatment schools, and the parenthesis reads as the range of their composition: the proportion of social referents among the seeds assigned in each of the 28 treatment schools. On the public data that range is 0 to 25\\%. Three of the 28 groups reach 20\\%, with 5 social referents among 20 seeds in one school and 5 among 24 in two others. The projection at 20\\% is therefore near the top of the observed compositions, which the printed range does not show.\n\nThe code also computes a composition for the students drawn in the seed lottery of each control school, where no anticonflict group met. Over all 56 schools the largest value on the public data is 10 of 28, or $35.7\\%$, in a control school. It rounds to 36\\%, not 37\\%.", "what_survives": "Nothing is computed from the parenthesis. In the regression the seed-group composition enters only multiplied by the treatment indicator, which is zero in every control school, and the projection at 20\\% lies inside the observed range of treatment-group compositions. The fitted models, their tests and the peer-exposure analyses are untouched by this correction. The level of $0.08$ in the same sentence is the subject of \\fid{P037-N010}."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N027", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of a paragraph of ten lines beginning 'Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks.' Red boxes mark the words 'the percentage' in three places: before 'of students identified as white, black, and hispanic', before 'of students identified as having limited English proficiency', and before 'of students receiving free or reduced lunch'.", "caption": "SI Appendix, “Study design”, p. 5 (supplement PDF p. 5). The three marked phrases describe the demographic blocking variables as percentages.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N030.png"}], "formal_release": null, "formal_statements": [], "headline": "The blocks of schools were formed on the numbers of students in each demographic group, not on the percentages that the SI lists, and one of the 14 blocks was not formed by the matching procedure.", "id": "P037-N030", "location": "SI Appendix, “Study design”, p. 5 (supplement PDF p. 5), paragraph beginning “Schools were assigned to blocks of four”", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks. Blocks were composed to maximize balance on the following variables: the latitude and longitude location of each school, the average school population as measured by the number of students who took the pre-randomization student surveys, the 5th, 6th, and 7th grade population during the year prior to the study (2011), \\hl{the percentage} of students identified as white, black, and hispanic, \\hl{the percentage} of students identified as having limited English proficiency, and \\hl{the percentage} of students receiving free or reduced lunch as identified by the New Jersey State Department of Education, and finally the average network clustering coefficient and network density calculated from student network data gathered in the pre-randomization student surveys.", "corrected": "Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks. Blocks were composed to maximize balance on the following variables: the latitude and longitude location of each school, the average school population as measured by the number of students who took the pre-randomization student surveys, the 5th, 6th, and 7th grade population during the year prior to the study (2011), \\hl{the number} of students identified as white, black, and hispanic, \\hl{the number} of students identified as having limited English proficiency, and \\hl{the number} of students receiving free or reduced lunch as identified by the New Jersey State Department of Education, and finally the average network clustering coefficient and network density calculated from student network data gathered in the pre-randomization student surveys. \\hl{Thirteen blocks were formed using the matching procedure on the listed variables; the four schools that already had a block label were assigned to a fourteenth block.}\n\nThe percentages and proportions of the other demographic columns of Table S1 need no change for this correction.", "evidence": "The evidence is a reading of the code, with three checks. First, I executed the statements of the script that build the blocking variables on invented school records and confirmed that the five demographic columns reach the blocking routine as counts. The codebook of the public release describes their components as numbers of students. Second, Table S1 independently rules out percentages for limited English proficiency: for one block it prints a mean of $51.25$ and a standard deviation of $84.74$. Four percentages $x_i$ in $[0, 100]$ with mean $\\mu$ have variance\n\\[\n\\frac{1}{4}\\sum_i x_i^2 - \\mu^2 \\le 100\\mu - \\mu^2 \\le 2500 ,\n\\]\nand this implies that their standard deviation is at most $50$, or about $57.7$ with the divisor 3.\n\nThird, counts and shares are not equivalent under the covariance scaling of the Mahalanobis distance, which is the default of the blocking routine in blockTools version 0.6.6, even when enrollment is among the variables. Suppose that four invented schools have enrollments $100$, $200$, $300$ and $400$ and group counts $10$, $20$, $120$ and $200$, so that the shares are $1/10$, $1/10$, $2/5$ and $1/2$. With the pair (enrollment, count), the squared distance between the second and third schools is $291/122$ and that between the second and fourth is $358/61$. The third school is the nearer match. With the pair (enrollment, share), the two distances are $51/10$ and $22/5$, and the fourth school is nearer.\n\nThe check does not reconstruct the original matching. The school file that the script reads and the original software environment are unavailable; the public school variables are given in ranges.", "problem": "The authors' blocking script does not divide by enrollment. It defines the variables for white, black and Hispanic students and for free or reduced lunch as sums of numbers of students, and it passes them, with the number of students of limited English proficiency, to the routine that forms the blocks. Division by the school total appears only later in the script, for the summaries of Table S1. When schools differ in size, a count is not a rescaled share, and counts also follow school size. Matching on counts is therefore not matching on percentages.\n\nThe script also keeps four schools out of the matching. It forms 13 blocks with the matching routine and carries those four schools, which already had a block label, in as the fourteenth block. The code does not say how that group was chosen.", "what_survives": "The randomization within blocks does not depend on how the blocks were formed. The script assigns two of the four schools of each block to treatment by a random permutation, so the six allocations are equally likely and each school has probability $1/2$. The public data show 56 schools in 14 blocks of four, each with two treatment and two control schools. The analyses use the recorded blocks, so correcting this description changes no estimate or standard error. Nothing here shows that the fourteenth block was poorly balanced, or which blocks would have differed had shares been used."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N030", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of the SI: 'Randomization-based confidence intervals for average potential outcomes are obtained under a maintained hypothesis of constant effects. Specifically, we use 1,000 simulated randomizations (both randomization across schools and within seed-eligibles) to estimate the standard errors under the maintained hypothesis that, for all i,'. A red box marks the words '1,000 simulated randomizations (both randomization across schools and within seed-eligibles)', which run over two lines.", "caption": "SI Appendix, p. 19 (supplement PDF p. 19), the paragraph on randomization-based confidence intervals. The marked words say that the $1{,}000$ simulated randomizations include the randomization across schools.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N031.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T31", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T31"}, {"id": "P037-T32", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T32"}], "headline": "The authors' code draws the school assignments simulated for the network analyses as 28 of the 56 schools without the blocks, and not as two of the four schools of each block as in the experiment; with blocked draws on the public data, the 16 standard errors corresponding to Table S5 change by at most 7% relative to unblocked draws, and none of the 12 referent contrasts crosses $0.05$.", "id": "P037-N031", "location": "SI Appendix, pp. 14 and 18 (supplement PDF pp. 14 and 18), “according to the experimental design”; the paragraph on randomization-based confidence intervals on SI pp. 19–20. The design is described on SI p. 5.", "pdf_page": 14, "printed_page": "14", "reaches": [{"effect": "cannot_determine", "effect_text": "could not be determined", "explanation": null, "package": null, "recorded_as": "P013-N016", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P013-N016", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N016", "work_id": "P013"}], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T31", "P037-T32"], "text": {"as_printed": "SI Appendix, p.~14: ``We achieve an arbitrarily precise estimate of $\\pi_i$ via $J = 10,000$ simulated random assignments of $\\mathbf{Z}$ \\hl{according to the experimental design}:''\n\nSI Appendix, pp.~19--20: ``Specifically, we use 1,000 simulated randomizations (both randomization across schools and within seed-eligibles) to estimate the standard errors under the maintained hypothesis that, for all $i$, $y_i(d_k) = y_i(D_i) + \\hat{\\mu}_R(d_k) - \\hat{\\mu}_R(D_i)$.''\n\nSI Appendix, p.~5: ``Schools were assigned to blocks of four, and randomized to receive the treatment or not within these blocks.''", "corrected": "The school assignments should be drawn within the blocks, by \\texttt{sch.perms <- genperms(schtreat, \\hl{blockvar = schrb,} clustvar = schid, maxiter = niter)}, where \\texttt{schrb} is the block of the school. Under this call each simulated assignment treats two of the four schools of each of the 14 blocks, each of the six choices in a block having probability $1/6$, independently over blocks. The exposure matrices, and the standard errors, confidence intervals and $P$ values of Fig.~3, Table S5, Fig.~S4 and the network panels on SI p.~34, should then be recomputed. With this change the simulations follow the experimental design as stated on SI pp.~14 and 18.\n\nIf the unblocked procedure is retained, replace ``according to the experimental design'' on SI pp.~14 and 18 with ``\\hl{with 28 of the 56 schools drawn uniformly without regard to the school blocks, and seeds drawn among seed-eligibles within the student blocks of each school}''.", "evidence": "I checked the authors' code, derived the assignment laws and recalculated the network inference on the public data (ICPSR 37070 V2), using an adapter for the authors' scripts. The first script (\\texttt{1. Sanitize and generate perms, probs.R}, line 58) generates the school assignments by \\texttt{sch.perms <- genperms(schtreat, clustvar = schid, maxiter = niter)}. The call names the schools as clusters and names no blocks. Applied to eight schools in two blocks, it generates all 70 assignments with four treated schools, of which 34 do not treat two schools in each block.\n\nFor two schools $a$ and $c$ of one block, with treatment indicators $S_a$ and $S_c$, under the law of the experiment I have\n\\[\n\\Pr(S_a = S_c = 1) = \\frac{2}{4}\\cdot\\frac{1}{3} = \\frac{1}{6},\n\\]\nand under the law of the code I have\n\\[\n\\Pr(S_a = S_c = 1) = \\frac{28}{56}\\cdot\\frac{27}{55} = \\frac{27}{110}.\n\\]\nThe exact marginal exposure probabilities agree. Let $U_a$ be the seeds drawn in school $a$, which are drawn independently of the school assignment. Since the network of the public data has no tie between schools, the exposure of a student of school $a$ is a function of $(S_a, U_a)$. Under both laws $\\Pr(S_a = 1) = 1/2$ and $U_a$ is independent of $S_a$. This implies that the pair $(S_a, U_a)$ has the same distribution under both, and that each $\\pi_i(d_k)$ is unchanged.\n\nOn the public data I ran the network script twice, once with the code's school draws and once with school draws made within the blocks, each with $10{,}000$ draws for the exposure probabilities and $1{,}000$ for the variances. The largest change among the 16 standard errors corresponding to Table S5 is $0.0105$, or 7.0\\%. Of the 12 contrasts with referent peer exposure, none crosses $0.05$, and the largest change in a $P$ value is $0.150$, from $0.823$ to $0.673$, for a contrast in disciplinary reports. With the estimated probabilities and the screened sample held fixed, so that only the school draws change, the largest change in a standard error is 5.8\\% and the largest change in a $P$ value is $0.009$. For prescriptive norms, the contrast of referent with non-referent peer exposure, which the article reports with $P < 0.01$, has $P = 0.0101$ with the code's draws, $0.0144$ with blocked draws, and $0.0113$ with blocked draws and the probabilities held fixed. These finite simulations have Monte Carlo uncertainty. The standard errors from the code's draws on the public data are close to those of Table S5 and do not all round to them. Since the authors' own data file and their saved draws are not public, these values are values for the public data, and they do not show whether the original $P$ value was below $0.01$. I did not recompute the placebo panels (Fig.~S4) or the betweenness panels (SI p.~34), which use the same draws.", "problem": "The experiment assigned two of the four schools of each of 14 blocks to treatment. In the authors' code, the simulated school assignments are generated without the block variable, and each one treats 28 schools drawn uniformly from the 56. Under both laws each school is treated with probability $1/2$, but the joint laws differ. The exact exposure probabilities depend only on the assignment of the student's own school and on the seeds drawn in it, and they are the same under both laws. The standard errors, confidence intervals and $P$ values of the network analyses come from the same simulated assignments, and they depend on the joint law. The SI Appendix describes these simulations as following the experimental design.", "what_survives": "The experiment was blocked as described. The exact exposure probabilities, the population with positive probability of all four exposures and the point estimates for fixed probability estimates and screening are unchanged; fresh finite simulations of the probabilities can change the screened sample and point estimates. The school-level regressions (Figs.~1 and 2, Tables S6 and S7) and the permutation check within treatment schools (Fig.~S5) do not use the simulated school assignments. The statement of the SI Appendix that the schools are independent is a separate matter (\\fid{P037-E004}), which blocked draws do not repair."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N031", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of nine lines of text. A red box marks 'equals student i's row in the network adjacency matrix' in the sentence 'We define the exposure mapping as a function, f(z, theta-i), such that the parameter, theta-i, equals student i's row in the network adjacency matrix.' The following sentences define gamma as an N-length vector whose jth element is 1 if student j has an indegree in the top decile of student j's school, and say that z-prime theta-i is the number of student i's peers that are assigned to be seeds.", "caption": "SI Appendix, p. 17. The marked words define the parameter $\\theta_i$ of the exposure mapping. The next sentence defines social referents by indegree, the number of nominations received.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N032.png"}, {"alt": "Crop of seven lines of text. Red boxes mark the sentence 'Our network question (i.e., whom did you choose to spend time with, face to face or online) directly measures who is observing whose behavior.'", "caption": "Methods and Materials, p. 567 (PDF p. 2), right column. The marked sentence says that the network question measures who observes whose behavior, which is a directed relation.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N032-b.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T25", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T25"}, {"id": "P037-T26", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T26"}, {"id": "P037-T41", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T41"}], "headline": "In the authors' primary social influence code, two students are peers when either nominated the other; the supplement defines a student’s exposure by “student $i$'s row in the network adjacency matrix” and does not say that the matrix of nominations was made symmetric.", "id": "P037-N032", "location": "SI Appendix, p. 17 (PDF p. 17 of the supplement), definition of the exposure mapping; Methods and Materials, p. 567 (PDF p. 2), right column, sentence on the network question; exposure conditions (ii)–(iv), p. 568 (PDF p. 3)", "pdf_page": 17, "printed_page": "17", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T25", "P037-T26", "P037-T41"], "text": {"as_printed": "SI p.~17: We define the exposure mapping as a function, $f(\\mathbf{z}, \\theta_i)$, such that the parameter, $\\theta_i$, equals \\hl{student $i$'s row in the network adjacency matrix}.\n\nMain text, p.~567: Our network question (i.e., ``whom did you choose to spend time with, face to face or online'') directly measures who is observing whose behavior.", "corrected": "SI p.~17: \\hl{For the primary social influence analyses,} we define the exposure mapping as a function, $f(\\mathbf{z}, \\theta_i)$, such that the parameter, $\\theta_i$, equals student $i$'s row in \\hl{the symmetrized network adjacency matrix. Let $B_{ij}=1$ if $i\\ne j$ and student $i$ nominated student $j$ before treatment, and $B_{ij}=0$ otherwise, and set $S_{ij}=\\mathbf{1}(B_{ij}+B_{ji}>0)$. Then $\\theta_i$ is the $i$th row of $S$, for the observed exposure and for the exposure probabilities: two students are peers when either nominated the other.} Further let $\\gamma$ be an $N$-length vector denoting referent vs.\\ nonreferent status, so that the $j$th element of $\\gamma$ takes on the value 1 if student $j$ has an indegree in the top decile of student $j$'s school, and 0 otherwise. \\hl{Compute indegree from nominations received from other students, counting repeated nominations separately.}\n\nTo be added on SI p.~17: \\hl{In the analysis by betweenness (Fig.~S6), $\\gamma$ is the indicator of high betweenness in the directed nominations, and $\\theta_i$ is unchanged. The alternative analysis of Fig.~S5 does not use $S$. There the peers of a seed, or of a seed-eligible student who was not selected, are the students whom that student nominated, each counted as often as listed.}\n\nMain text, p.~568, conditions (ii)--(iv): ``peers'' are \\hl{students joined by a nomination in either direction}.", "evidence": "The evidence is a check of the pages, a reading of the authors' scripts, a derivation, and a count on the public release of the data (ICPSR 37070, version 2).\n\n\\emph{The printed identities hold for $S$.} With $\\theta_i$ the $i$th row of $S$, I have\n\\[\n\\mathbf{z}'\\theta_i=\\sum_j z_jS_{ij},\\qquad (\\mathbf{z}\\circ\\gamma)'\\theta_i=\\sum_j z_j\\gamma_jS_{ij}.\n\\]\nThe first sum counts the seeds who nominated student $i$ or were nominated by her, and the second counts the social referents among them. The sentences of SI p.~17 on these two counts are true for $S$, and they are true for $B$ with a different meaning of ``peers''.\n\n\\emph{The exposure depends on the matrix.} A nomination in either direction can only add seed peers. For a student in a treated school who is not a seed, passing from $B$ to $S$ can move her from the condition with no seed peer to one of the two conditions with a seed peer, and from the condition with a seed peer and no social referent among her seed peers to the condition with a seed peer who is a social referent. It can move her in no other way. Under $S$ the probability of the condition with no seed peer is therefore at most its probability under $B$, and the probability of the condition with a seed peer who is a social referent is at least its probability under $B$.\n\n\\emph{The count.} I computed the observed exposure condition of each of the 24,191 students kept by the authors' rule on the public data, once with $B$ and once with $S$. The graph has 181,044 directed ties and 276,652 ordered pairs joined in either direction.\n\n\\begin{audittable}{lrr}{Observed exposure conditions of the 24,191 students on the public data (ICPSR 37070, version 2), under the two definitions of a peer. The third column is the definition in the authors' code.}\nExposure condition & Students whom $i$ nominated ($B$) & Either direction ($S$) \\\\\n\\hline\nControl school & 12,253 & 12,253 \\\\\nSeed & 728 & 728 \\\\\nTreated school, no seed peer & 7,330 & 5,754 \\\\\nSeed peer, none a social referent & 2,879 & 4,306 \\\\\nSeed peer who is a social referent & 1,001 & 1,150 \\\\\n\\end{audittable}\n\nThe condition differs for 1,609 students: 1,460 move from no seed peer to a seed peer who is not a social referent, 116 from no seed peer to a seed peer who is a social referent, and 33 from a seed peer who is not a social referent to one who is. In all, 1,576 students have a seed peer only because a seed nominated them.\n\nThese counts cover all students kept on the public data. They do not identify the sample of 2,451 students reported in the article, which has further restrictions (see \\fid{P037-N025}). I did not estimate any contrast with $B$ in place of $S$, and no published estimate is shown to change. The authors' own input files are not public, and ICPSR reports changes to some released variables. The check does not establish which matrix produced the printed estimates; the released code uses $S$.", "problem": "The network question is directed. Each student nominates the students with whom she chose to spend time, and the main text argues that the question measures who observes whose behavior. On the same page of the supplement, social referents are defined by indegree, the number of nominations received. In this context it is natural to take ``student $i$'s row'' to be the students whom student $i$ nominated, although the words ``network adjacency matrix'' also allow a matrix derived from the nominations.\n\nThe authors' code uses a derived matrix. Let $B_{ij}=1$ if $i\\ne j$ and student $i$ nominated student $j$, and $B_{ij}=0$ otherwise, and set\n\\[\nS_{ij}=\\mathbf{1}\\,(B_{ij}+B_{ji}>0).\n\\]\nThe script \\texttt{1. Sanitize and generate perms, probs.R} forms $S$ (lines 125--126) and takes $\\theta_i$ to be the $i$th row of $S$, for the observed exposure and for the simulated assignments from which the exposure probabilities are estimated (lines 170--181). Two students are peers when either nominated the other. Referent status is still computed from indegree in the directed nominations. Neither the article nor the supplement states this step.\n\nThe exposure conditions, the exposure probabilities and the analysis sample depend on which matrix is used. In the released code, the contrasts corresponding to Fig.~3 and Table S5 compare students by the seeds among those who nominated them or whom they nominated. They cannot be read as effects of exposure only to seeds whom the student chose to spend time with, and the analysis cannot be rebuilt from the printed description.", "what_survives": "The printed identities are correct for $S$. The released code uses $S$ in the same way for the observed exposure and for the simulated assignments, and it computes referent status from directed indegree. In that code, the same definition of a peer serves the placebo outcomes of Fig.~S4 and the analysis by betweenness of Fig.~S6. The school-level results of Figs.~1 and 2 and Tables S6 and S7 do not use the exposure conditions. The word ``peers'' in the statements of the main results is compatible with $S$. What changes is the reading of the within-school contrasts: they cannot be read as effects of seeds whom the student nominated. An analysis with $B$ would be a different analysis, with its own exposure probabilities and its own sample, and I did not carry it out."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N032", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the top of SI page 19. After the words 'define an estimated parameter vector associated with exposure condition d_k', a displayed formula sets xi-hat(d_k) equal to the arg min over xi(d_k) of the sum over i with D_i = d_k of 1 over pi_i(d_k) times the squared difference between y_i(d_k) and g(x_i, xi(d_k)). A red box marks the summation sign with its index 'i: D_i = d_k'. A second red box marks the words 'among students in a given level of exposure.' in the sentence below the formula.", "caption": "SI Appendix, p. 19 (supplement PDF p. 19). The marked sum runs over the students observed in exposure $d_k$ only, and the marked words say that the regression is fitted among the students in a given level of exposure.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N033.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T27", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T27"}, {"id": "P037-T28", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T28"}, {"id": "P037-T42", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T42"}], "headline": "The authors' code computes the numbers of Fig. 3 and Table S5 from one weighted regression over the four exposures with common covariate slopes, and not from the separate weighted regression within each exposure that the SI Appendix displays; the two procedures can give different estimates.", "id": "P037-N033", "location": "SI Appendix, p. 19 (supplement PDF p. 19), the two displays that define the estimator; the corresponding sentence of the article is on p. 568 (PDF p. 3), right column", "pdf_page": 19, "printed_page": "19", "reaches": [{"effect": "cannot_determine", "effect_text": "could not be determined", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P037-T27", "P037-T28", "P037-T42"], "text": {"as_printed": "For covariate adjustment, define an estimated parameter vector associated with exposure condition $d_k$\n\\[\n\\widehat{\\xi}(d_k) = \\arg\\min_{\\xi(d_k)} \\hl{\\sum_{i:D_i=d_k}} \\frac{1}{\\pi_i(d_k)} \\bigl[y_i(d_k) - g\\left(\\mathbf{x}_i, \\xi(d_k)\\right)\\bigr]^2 ,\n\\]\nwhere $g(.)$ is the specification for the regression of $y_i(d_k)$ on $\\mathbf{x}_i$. $\\xi(d_k)$ is the parameter vector arising from an inverse probability weighted regression of outcomes on covariates \\hl{among students in a given level of exposure}. Then the regression estimator for the mean is\n\\[\n\\widehat{\\mu}_R(d_k) = \\hl{\\frac{1}{N_s} \\sum_{i:\\pi_i(d_k)>0,\\forall k\\in\\{C,S,NI,I\\}}} \\left[ \\mathrm{I}(D_i = d_k)\\, \\frac{y_i(d_k) - g\\bigl(\\mathbf{x}_i, \\widehat{\\xi}(d_k)\\bigr)}{\\hat{\\pi}_i(d_k)} + g\\bigl(\\mathbf{x}_i, \\widehat{\\xi}(d_k)\\bigr) \\right],\n\\]\nwhere $N_s$ denotes the number of students such that $0 < \\pi_i(d_k) < 1$ for all $i$ and for $k \\in \\{C,S,NI,I\\}$.", "corrected": "For covariate adjustment, define the estimated intercepts and the estimated common slope vector\n\\[\n\\bigl(\\widehat{\\alpha}_C, \\widehat{\\alpha}_S, \\widehat{\\alpha}_{NI}, \\widehat{\\alpha}_I, \\widehat{\\beta}\\bigr) = \\arg\\min_{\\alpha, \\beta} \\hl{\\sum_{i \\in A}} \\frac{1}{\\widehat{\\pi}_i(D_i)} \\bigl[y_i(D_i) - \\hl{\\alpha_{k(i)} - \\mathbf{x}_i^{\\top}\\beta}\\bigr]^2 ,\n\\]\nwhere $A$ is the outcome-specific set of students with \\hl{all four estimated exposure probabilities greater than $0.01$, no observed direct seed exposure, and a nonmissing outcome and processed pretreatment score; for peer conflict, the observed school mean must also exceed $0.02$} (\\fid{P037-N025}). Here $k(i)$ is the exposure observed for student $i$, so that $D_i = d_{k(i)}$, and $\\widehat{\\pi}_i(D_i)$ is the estimated probability of that exposure. The vector $\\mathbf{x}_i$ contains the estimated probabilities of referent peer, non-referent peer, school and both direct seed exposures; the pretreatment score for norm outcomes (a constant for other outcomes); and three background covariates: self-identification as white, whether friends say the student's house is nice, and whether a language other than English is spoken at home. Missing background values and pretreatment norm scores are replaced by their respective means before selection and fitting. This is one inverse probability weighted regression of outcomes on \\hl{indicators of the four exposures and on covariates, with the same covariate slopes in every exposure}. Then the regression estimator for the mean is\n\\[\n\\widehat{\\mu}_R(d_k) = \\hl{\\frac{1}{|A|} \\sum_{i \\in A}} \\bigl(\\hl{\\widehat{\\alpha}_k + \\mathbf{x}_i^{\\top}\\widehat{\\beta}}\\bigr), \\qquad k \\in \\{C, S, NI, I\\}.\n\\]\nSince the weighted residuals sum to zero within each exposure, adding the term $\\mathrm{I}(D_i = d_k)\\,\\bigl[y_i(D_i) - \\widehat{\\alpha}_k - \\mathbf{x}_i^{\\top}\\widehat{\\beta}\\bigr]/\\widehat{\\pi}_i(d_k)$ inside the sum does not change $\\widehat{\\mu}_R(d_k)$.\n\nThis describes an average over the realized, outcome-specific set $A$, rather than the fixed positive-probability population stated in the SI Appendix. Retaining that fixed population requires revising the averaging set and addressing missing outcomes and selection. If the separate regressions of the printed display are intended instead, the estimates of Fig.~3 and Table S5 and their standard errors have to be recomputed with them.", "evidence": "The evidence is a reading of the authors' code, an exact example and a recalculation on the public data (ICPSR 37070 V2). In the network script (\\texttt{3. Network results.R}), line 216 regresses the outcome on indicators of the exposures, the student's estimated exposure probabilities, the pretreatment outcome and three background covariates. The weights are the inverses of the estimated probabilities of the observed exposures, and the regression has no interaction of an exposure with a covariate. Lines 223--226 average the fitted values over the analysis set.\n\nThe residual term of the display does not account for the difference. A regression with an intercept for each exposure has weighted residuals that sum to zero within each exposure. When the fitting and averaging sets and weights agree, the augmented form of the display equals the average of the fitted values. The difference lies in the slopes.\n\nThe synthetic example has twelve units, each with probability $1/4$ of each exposure, one covariate $x$, and potential outcomes $y(d_C) = y(d_S) = y(d_{NI}) = x$ and $y(d_I) = 2x$. Let the units observed in $d_C$, $d_S$, $d_{NI}$ and $d_I$ have $x$ equal to $0,1,2$; $1,2,3$; $2,3,4$; and $3,4,5$. The mean of $x$ over the twelve units is $5/2$. The separate regressions fit $y = x$ in the first three exposures and $y = 2x$ in the fourth without error, and I obtain the means $5/2$, $5/2$, $5/2$ and $5$. The pooled regression has common slope\n\\[\n\\widehat{\\beta} = \\frac{2 + 2 + 2 + 4}{2 + 2 + 2 + 2} = \\frac{5}{4},\n\\]\nintercepts $-1/4$, $-1/2$, $-3/4$ and $3$, and means $23/8$, $21/8$, $19/8$ and $49/8$. In the display, the numerator sums over the four exposures the cross-products of $x$ and $y$ about their means within the exposure, and the denominator sums the corresponding sums of squares of $x$. The contrast of $d_I$ with $d_{NI}$ is $5/2$ under the separate regressions and $15/4$ under the pooled one, and the residual terms vanish in both. In a second synthetic example, with the probabilities $1/2$, $1/8$, $1/8$ and $1/4$ of a simplified school and seed design, I find the contrasts $5/2$ and $42/11$.\n\nIn a reconstruction on the public data I fitted both procedures with the analysis set and covariates of the code, using inverse probability weights from analytically computed exposure probabilities in both fits, so that only the constraint of common slopes changes.\n\n\\begin{audittable}{lrr}{Estimated mean under referent peer exposure minus estimated mean under non-referent peer exposure on the public data, from the pooled regression of the code and from separate regressions within each exposure.}\nOutcome & Pooled regression & Separate regressions \\\\\nFriends talk about conflict & $0.036$ & $0.027$ \\\\\nPrescriptive norms & $0.468$ & $0.487$ \\\\\nWristband & $0.108$ & $0.099$ \\\\\nPeer conflict & $-0.005$ & $-0.019$ \\\\\n\\end{audittable}\n\nAmong the 16 means, the largest difference is for prescriptive norms under school exposure $d_S$ (a treatment school and no seed peer), $-0.795$ pooled against $-0.905$ separate. This comparison uses the analysis set and not the population stated in the SI Appendix, and the public data do not reproduce Table S5 exactly; ICPSR reports that some released variables were changed. The authors' own data file is not public, and I recomputed no standard error or $P$ value for the separate regressions.", "problem": "The SI Appendix defines, for each exposure $d_k$, a weighted regression fitted to the students observed in that exposure, with its own parameter vector $\\xi(d_k)$. With a linear specification, each exposure then has its own covariate slopes. The authors' code fits one weighted regression to all students of the analysis set of the outcome (the students that the code keeps, described in \\fid{P037-N025}), with an intercept for each exposure and one vector of covariate slopes common to the four exposures. Its estimate of the mean under $d_k$ is the average, over the same set, of the fitted values with the intercept of $d_k$. The two procedures are different estimators. When the fitted slopes differ between exposures they ordinarily return different means, and the SI Appendix does not assume equal slopes. The displayed estimator therefore does not describe how Fig.~3 and Table S5 were computed. The sentence of the article (p.~568) speaks of ``a weighted least-squares regression'' and is compatible with either procedure.", "what_survives": "The displayed estimator is a legitimate estimator, and this discrepancy does not refute the randomization or exposure definitions. Pooling is not shown to bias the estimates, and no $P$ value or conclusion of the article is shown to change. The school-level regressions and the corrected 25\\% school-level conflict reduction use another estimator. What changes is the description of the method behind Fig.~3 and Table S5. The analysis set, over which the means are averaged, is the subject of \\fid{P037-N025}, and the regression used in the simulated assignments is the subject of \\fid{P037-N034}."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N033", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Two excerpts separated by a grey horizontal rule. Above the rule, four lines beginning Randomization-based confidence intervals for average potential outcomes are obtained under a maintained hypothesis of constant effects; two red boxes mark the words to estimate the standard errors under the maintained hypothesis that, for all i. Below the rule, three lines; a red box marks the equation y sub i of d sub k equals y sub i of D sub i plus mu-hat sub R of d sub k minus mu-hat sub R of D sub i, which is followed by Aronow [52] derives the consistency of a variance estimator under this maintained hypothesis.", "caption": "SI Appendix, end of p. 19 (above the grey rule) and beginning of p. 20 (below it; PDF pp. 19–20). The marked words and the marked equation state how the standard errors are estimated: under the hypothesis that each potential outcome is the observed outcome shifted by the difference of the estimated means.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N034.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T33", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T33"}, {"id": "P037-T34", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T34"}], "headline": "The randomization standard errors of the social-influence estimates do not follow the stated constant-effect procedure for the estimator used in the released code: the refits replace two probability controls of the point regression by their sum. Keeping the two controls in every refit recovers the variances of constant-effect imputation followed by that same regression; on the public data the standard errors move by less than 1%.", "id": "P037-N034", "location": "SI Appendix, pp. 19–20 (PDF pp. 19–20 of the SI Appendix), from “Randomization-based confidence intervals” to “using a normal approximation”; the same description on pp. 14–15; it underlies the standard errors of Table S5 (PDF p. 40) and the intervals of Fig. 3 (main text, p. 570)", "pdf_page": 19, "printed_page": "19–20", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T33", "P037-T34"], "text": {"as_printed": "``Randomization-based confidence intervals for average potential outcomes are obtained under a maintained hypothesis of constant effects. Specifically, we use 1,000 simulated randomizations (both randomization across schools and within seed-eligibles) \\hl{to estimate the standard errors under the maintained hypothesis that, for all $i$, $y_i(d_k)=y_i(D_i)+\\hat\\mu_R(d_k)-\\hat\\mu_R(D_i)$}. Aronow [52] derives the consistency of a variance estimator under this maintained hypothesis. Wald-type confidence intervals and $p$-values are then constructed using a normal approximation.''", "corrected": "For the released pooled regression, keep $A$ and $x_i$ fixed across assignments and compute estimable averaged predictions as follows. For each of $B=1{,}000$ simulated assignments $\\mathbf{z}^1,\\ldots,\\mathbf{z}^B$, let $e_i^b$ denote the exposure indicators of student $i$ under $\\mathbf{z}^b$ and $D_i^b$ the exposure, and compute\n\\[\n(\\hat\\tau^b,\\hat\\gamma^b)=\\arg\\min_{\\tau,\\gamma}\\sum_{i\\in A}\\frac{1}{\\hat\\pi_i(D_i^b)}\\bigl(r_i-(e_i^b)^\\top\\tau-x_i^\\top\\gamma\\bigr)^2,\\qquad \\hat m^b(d_k)=\\hat\\tau^b_k+\\bar x^\\top\\hat\\gamma^b ,\n\\]\n\\hl{with the covariate vector $x_i$ of the point regression, in which $\\hat\\pi_i(d_I)$ and $\\hat\\pi_i(d_{NI})$ enter as separate controls}. The standard error of $\\hat\\mu(d_k)$ is the standard deviation of $\\hat m^b(d_k)$ over $b$, and the standard error of $\\hat\\mu(d_k)-\\hat\\mu(d_l)$ is the standard deviation of $\\hat m^b(d_k)-\\hat m^b(d_l)$. This procedure has the same variances as imputing $y_i^b=y_i+\\hat\\mu(D_i^b)-\\hat\\mu(D_i)$ and refitting the same pooled point regression; it does not repair the separate discrepancy in the estimator displayed on p.~19. In the released code, replace \\texttt{pi.01} at line 257 of \\texttt{3. Network results.R} by \\hl{\\texttt{pi.01.inf + pi.01.ninf}}. The standard errors, intervals and Wald tests of Fig.~3 and Table S5 are then recomputed.", "evidence": "The evidence is a derivation, an exact counterexample and a recalculation on the public data.\n\nLet $A$ denote the fixed analysis set. For $i\\in A$, let $e_i$ denote the vector of indicators of the observed exposure $D_i$, and let $x_i$ denote the fixed covariates of the point regression, in which $\\hat\\pi_i(d_I)$ and $\\hat\\pi_i(d_{NI})$ are separate entries. The point regression minimizes $\\sum_{i\\in A}\\hat\\pi_i(D_i)^{-1}(y_i-e_i^\\top\\tau-x_i^\\top\\gamma)^2$. Choose fixed coefficients $(\\hat\\tau,\\hat\\gamma)$ from this fit, giving estimates $\\hat\\mu(d_k)=\\hat\\tau_k+\\bar x^\\top\\hat\\gamma$ with $\\bar x=|A|^{-1}\\sum_{i\\in A}x_i$, and residuals $r_i=y_i-e_i^\\top\\hat\\tau-x_i^\\top\\hat\\gamma$. For a simulated assignment $\\mathbf{z}$ with exposure indicators $e_i^{\\mathbf{z}}$, the imputed outcome under the hypothesis of constant effects is\n\\[\ny_i^{\\mathbf{z}}=y_i+(e_i^{\\mathbf{z}}-e_i)^\\top\\hat\\tau=r_i+(e_i^{\\mathbf{z}})^\\top\\hat\\tau+x_i^\\top\\hat\\gamma .\n\\]\nLet $H_{\\mathbf{z}}$ denote the matrix with rows $((e_i^{\\mathbf{z}})^\\top,x_i^\\top)$ and $W_{\\mathbf{z}}$ the diagonal matrix of the positive weights of the simulated exposures. When $H_{\\mathbf{z}}$ has full column rank, set $L_{\\mathbf{z}}=(H_{\\mathbf{z}}^\\top W_{\\mathbf{z}}H_{\\mathbf{z}})^{-1}H_{\\mathbf{z}}^\\top W_{\\mathbf{z}}$. Since $L_{\\mathbf{z}}H_{\\mathbf{z}}$ is then the identity, I have\n\\[\nL_{\\mathbf{z}}\\,y^{\\mathbf{z}}=L_{\\mathbf{z}}\\,r+(\\hat\\tau,\\hat\\gamma).\n\\]\nI conclude that the estimated means from the refit of the imputed outcomes and from the refit of the residuals differ by the constant $\\hat\\mu(d_k)$ for every assignment, and their variances over the assignments are equal. When columns of $H_{\\mathbf{z}}$ are aliased, the same holds for averaged predictions whose rows lie in its row space (estimable predictions); these are independent of the choice of coefficients. The identity uses that the columns $x_i$ are in the refit. With the two probability controls replaced by their sum, the component $(\\hat\\gamma_I-\\hat\\gamma_{NI})\\hat\\pi_i(d_I)$ of the fitted values need not lie in the span of the refit, and the identity need not hold.\n\nThe following example shows that the variance can change. Eight units form four pairs. Two of the four pairs are drawn with equal probabilities to be exposed to seeds. In such a pair $j$, the first unit has exposure $d_I$ with probability $q_j$ and $d_{NI}$ otherwise, where $(q_1,q_2,q_3,q_4)=(1/10,1/5,3/10,2/5)$, and the second unit has the other of the two exposures. In each remaining pair the two units have $d_C$ and $d_S$, in either order with probability $1/2$. There are 96 assignments. For every unit the probabilities of $d_I$ and $d_{NI}$ sum to $1/2$, so that the sum control is constant and the separate controls add the direction $\\hat\\pi_i(d_I)$. The observed exposures are $(d_I,d_{NI},d_I,d_{NI},d_C,d_S,d_C,d_S)$ and the observed outcomes are $(0,2,1,0,3,1,2,4)$. For the point regression, I find the estimates $46/13$, $19/13$, $-47/39$ and $112/39$ for $d_C$, $d_S$, $d_{NI}$ and $d_I$. Over the 96 assignments, in exact rational arithmetic, the contrast between $d_I$ and $d_{NI}$ has the same variance for the refit of the imputed outcomes and for the refit of the residuals with the separate controls, with standard deviation $3.027$. For the refit of the residuals with the sum control the variance is $1453114769225810453/621302156766720000$, and the standard deviation is $1.529$. The example is an abstract design: its probability of control exposure is $1/4$, where the paper's is $1/2$, and it is not a model of the school experiment.\n\nUsing the authors' scripts with an adapter for the public data (ICPSR 37070 V2), I compared the two versions of the refit on the same 1,000 simulated assignments, for the four outcomes. ICPSR says some variables in this release were changed; the point estimates and analysis sets of this comparison differ from the published ones. The sixteen standard errors of the estimated means change by at most 0.80\\%, and the twelve standard errors of the contrasts between the referent exposure and each other exposure by at most 0.79\\%. None of the twelve Wald tests changes its decision at the level 0.05. The changes are not all in one direction. One estimated $P$ value, for the norms contrast between the referent and the nonreferent exposure, moves from 0.0101 to 0.0099; the change is smaller than its Monte Carlo standard error, and it does not establish a change in a published statement of significance.", "problem": "The passage describes the following computation: impute the potential outcomes of each student by the displayed equation, draw assignments, recompute the reported estimator for each draw, and take the standard deviation of the results. The released code differs from this in two respects, and one of them is a defect. The code refits the residuals of the point regression instead of imputed outcomes. For the released pooled regression, the two computations have the same variances when each refit keeps the analysis set and covariates and determines the averaged predictions uniquely. The refits change the covariates. The point regression (\\texttt{3. Network results.R}, line 216) controls separately for the estimated probabilities of the referent and the nonreferent exposure, $\\hat\\pi_i(d_I)$ and $\\hat\\pi_i(d_{NI})$, and the refits (line 257) control only for their sum. The standard errors, intervals and Wald tests of Fig.~3 and Table S5 therefore come from the randomization distribution of a regression other than the one that produced the estimates. The estimator in question is the pooled regression of the released code; its difference from the estimator displayed on p.~19 is the subject of \\fid{P037-N033}.", "what_survives": "The point estimates of Fig.~3 and Table S5 do not depend on the refits and are unchanged. The school-level analysis, Fig.~2, and Tables S6 and S7 use a separate computation. The displayed equation is a coherent hypothesis, and refitting residuals is equivalent for variances to its imputation for the pooled regression under the conditions above. The check establishes no systematic direction of the change, no failure of coverage and no inconsistency of the variance estimator, and no conclusion of the paper is shown to change. The figures of less than 1\\% are an observation in one simulation on the public data, not a bound; the published standard errors can be recomputed only from the authors' own data file and saved simulations, which are not public. The correction concerns the controls of the refits alone. The simulated school draws are the subject of \\fid{P037-N031} and the description of the analysis set is the subject of \\fid{P037-N025}."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N034", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Fig. S2, Treatment-control balance across pretreatment covariates: a dot plot with fifteen rows, one for each covariate, and a horizontal axis for the p-value from 0 to 1, with a dashed vertical line at p = 0.05. A red box marks the row for the percentage receiving free or reduced-price lunch, the seventh from the top, whose dot is near 0.34. Only the dot of the last row, Network cluster coefficient, lies to the left of the dashed line.", "caption": "SI Appendix, Fig. S2 (PDF p. 25): the $p$-values of the fifteen balance tests. The marked row is labelled as free and reduced-price lunch together; the released code computes it from free lunch alone.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N035.png"}], "formal_release": null, "formal_statements": [], "headline": "The row of Fig. S2 labelled “%Free reduced lunch” is computed in the released code from free lunch alone; the label should name the free lunch ratio, or the test should be recomputed with free and reduced-price lunch together.", "id": "P037-N035", "location": "SI Appendix, Fig. S2 (PDF p. 25 of the SI Appendix), row “%Free reduced lunch”, and the text below the figure", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Row label of Fig.~S2: ``\\%Free \\hl{reduced} lunch''\n\nText below the figure: ``To compute these $p$-values, we use the school-level treatment assignment to predict the 15 pretreatment school-level covariates as outcomes in our regression specification.''", "corrected": "If the released computation is kept, the row label reads ``\\hl{Free lunch count / population included in the study}'', and the text below the figure states that the row is the number of students with free lunch divided by the number of students in the study and that this computation does not test free and reduced-price lunch together.\n\nTo test free and reduced-price lunch together with the same denominator, replace line 133 with \\texttt{prFr <- (free\\_lunch + \\hl{reduced\\_lunch})/incsize}, document that denominator, and recompute the $p$-value and plotted point on the authors' own data. To match Table S1, use \\texttt{row\\_total} in place of \\texttt{incsize}, after checking the relevant population.", "evidence": "I checked the released code and an exact example. With $F$ and $R$ the numbers of students with free and with reduced-price lunch in a school and $D$ the denominator, the two measures differ by $(F+R)/D-F/D=R/D$, which is positive whenever $R>0$. The two tests are not interchangeable. In an invented example of eight schools in two blocks, with two treated schools in each block and $D=100$ in every school, the figure's regression specification has treatment coefficient $0$ for $F/D$ and $1/10$ for $(F+R)/D$, with the same robust variance $1/20000$; the squared test statistics are $0$ and $200$.\n\nI cannot reconstruct the published point from the public data or establish that this released version generated it. ICPSR 37070 V2 reports the lunch counts as interval codes, not as counts, and the authors' own data file is not public. With those codes in place of counts, an adapted calculation of the affected regression on the public data gives $p=0.40$ for free lunch alone and $p=0.45$ for free and reduced-price lunch together, keeping the same denominator of students in the study; these values illustrate the code and do not reconstruct the figure. The two scripts also use different denominators (students in the study in the figure code, total enrolment in Table S1).", "problem": "The released figure script sets the variable of this row to the number of students with free lunch divided by the number of students in the study (\\texttt{8. Balance test and plot.R}, line 133), and tests and plots it under the label ``\\%Free reduced lunch'' (lines 127--128 and 141--155). The codebook of the released data lists the recipients of free lunch and of reduced-price lunch as separate counts. The column of Table S1 with the same name is computed from the sum of the two counts (\\texttt{7. Blocking, randomization, and summary statistics.R}, line 152), and the study design names free or reduced lunch among the blocking variables (SI Appendix, p.~5). The released code therefore tests a different covariate from the one this row's label names.", "what_survives": "Changing this numerator alone leaves the other fourteen balance outcomes unchanged. The released scripts that compute the school-level and the social-influence estimates do not use this variable, and the finding does not call for revising an effect estimate or the randomization. The size of the correction to the plotted point is not established. If the row is recomputed, the statement below the figure that only one of the 15 covariates ``has a significant imbalance at the 0.05 level'' has to be checked again on the authors' own data."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N035", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Five lines of text from page 29 of the SI Appendix, beginning As an additional robustness check of our estimated network effects. A red box marks the words to assess within-school social influence effects. The following sentence reads: We restrict our attention to treated schools, comparing the mean of outcomes of peers of seeds to the mean of outcomes of peers of seed-eligibles who were not selected.", "caption": "SI Appendix, p. 29 (PDF p. 29), first paragraph. The marked words give the purpose of the alternative analysis as assessing social influence effects; the next sentence defines its statistic as a difference between two means of outcomes of peers.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N036.png"}, {"alt": "Seven lines of text from page 32 of the SI Appendix, beginning Our results comport with the estimates found in our primary analysis. A red box marks the words restricted to social influence effects in the last sentence, which reads: Recall that this method does not allow for assessing between-school variation, and thus our comparisons with this methodology are restricted to social influence effects and do not incorporate any differences attributable to the climate of the school.", "caption": "SI Appendix, p. 32 (PDF p. 32), the paragraph after Fig. S5. The marked words say that the comparisons of this analysis are restricted to social influence effects.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N036-b.png"}], "formal_release": null, "formal_statements": [{"id": "P037-T35", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T35"}, {"id": "P037-T36", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-T36"}], "headline": "The alternative analysis of Fig. S5 is said to be restricted to social influence effects, but its statistic counts the outcomes of seeds among the peers and can respond to the effect of a student’s own treatment; its results are evidence against the null hypothesis of no effect of any kind.", "id": "P037-N036", "location": "SI Appendix, p. 29 (PDF p. 29 of the SI Appendix), first paragraph; Fig. S5 (PDF pp. 30–32) and the paragraph after it on p. 32", "pdf_page": 29, "printed_page": "29", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P037-T35", "P037-T36"], "text": {"as_printed": "p.~29: ``As an additional robustness check of our estimated network effects, we employed a variant of the method used by (15) \\hl{to assess within-school social influence effects}. We restrict our attention to treated schools, comparing the mean of outcomes of peers of seeds to the mean of outcomes of peers of seed-eligibles who were not selected. This difference-in-means serves as our test statistic.''\n\np.~32: ``Overall, we found a statistically significant \\hl{influence effect} on wristband usage ($p < 0.05$). Among the peers of referents, we found statistically significant \\hl{influence effects} on wristband usage ($p < 0.01$) and prescriptive and descriptive norms ($p < 0.05$). Recall that this method does not allow for assessing between-school variation, and thus our comparisons with this methodology are \\hl{restricted to social influence effects} and do not incorporate any differences attributable to the climate of the school.''", "corrected": "p.~29: ``\\ldots\\ a variant of the method used by (15), \\hl{a test of the sharp null hypothesis of no treatment effect within treated schools}. \\hl{The peers of a seed-eligible student are the students whom that student named, counted once for each nomination; seeds may be among them.} We restrict our attention to treated schools, comparing the mean of outcomes of peers of seeds to the mean of outcomes of peers of seed-eligibles who were not selected.''\n\np.~32: ``Overall, we found a statistically significant \\hl{difference in} wristband usage ($p < 0.05$). Among the peers of referents, we found statistically significant \\hl{differences in} wristband usage ($p < 0.01$) and prescriptive and descriptive norms ($p < 0.05$). Recall that this method does not allow for assessing between-school variation, and thus our comparisons with this methodology \\hl{are made within treated schools} and do not incorporate any differences attributable to the climate of the school. \\hl{They do not by themselves separate the effect of a student's own assignment as a seed from the influence of seeds on other students. The $p$-value of 0.009 for wristbands among the peers of referents is an estimate from 1,000 permutations and lies within Monte Carlo error of 0.01.}''", "evidence": "I checked the released code, recomputed counts on the public data and enumerated an exact counterexample. Ten seed-eligible students of one treated school form five pairs, and each student names the other member of the pair. Five of the ten are drawn as seeds, with all 252 assignments equally likely, and the outcome of each student is $Y_i(\\mathbf{z})=z_i$. The outcome of a student depends on that student's own assignment alone, and seeds have no influence on other students. Suppose that the seeds are one member of each pair. The students named by seeds have outcome $0$ and the students named by the others have outcome $1$, so that the statistic equals $-1$. With the outcomes held fixed, the statistic has absolute value $1$ for 2 of the 252 assignments, and the two-tailed $p$-value of the permutation test is $2/252=1/126<0.01$. Over the 252 assignments of the experiment, the test reports $p<0.01$ with probability $32/252=8/63$.", "problem": "In the released code (\\texttt{6. Bond et al analysis.R}), the peers of a seed-eligible student are the students whom that student named. The statistic is the mean outcome over the nominations made by seeds minus the mean outcome over the nominations made by seed-eligible students who were not selected. A student enters a mean once for each nomination received, and students who are themselves seeds are kept. In a reconstruction on the public data (ICPSR 37070 V2), before removing missing outcomes, seeds receive 296 of the 5,377 nominations made by seeds (5.5\\%) and 359 of the 5,235 nominations made by seed-eligible students who were not selected (6.9\\%). The statistic therefore contains outcomes of treated students, and an effect of a student's own treatment can move it. The test of p.~29 is a test of the sharp null hypothesis of no effect, and it can reject that hypothesis when seeds have no influence on other students. The statements that the analysis assesses social influence effects, and is restricted to them, claim more than the test supports.", "what_survives": "The statistic and its permutation test are a coherent test of the sharp null hypothesis of no effect, and the statistically significant results of Fig.~S5 are evidence against that hypothesis. The primary analyses of social influence (Fig.~3 and Table S5) and the school-level analysis do not use this statistic, and this finding does not refute their estimates. It does not show that the differences of Fig.~S5 come from the direct effect of treatment. The printed $p=0.009$ for wristbands among peers of social referent seeds is an estimate from 1,000 permutations, with a Monte Carlo standard error of 0.003, and it is not shown to be wrong. In a reconstruction on the public data the five overall statistics of Fig.~S5 round to the printed values and the five statistics for social referents do not, so the public data cannot settle the second column's $p$-values from the authors' own files."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N036", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The last two rows of Table S2, Middle East and Native American, each with four entries of the form mean (standard error) and a paired t-test p, followed by the note. Two red boxes mark the sentence Mean estimates include adjustment for block fixed effects, split across two lines.", "caption": "SI Appendix, Table S2 (PDF p. 37): the last two rows of the table and its note. The marked sentence says that the means of the table are adjusted for block fixed effects; the released code computes unadjusted means.", "url": "https://pmaronow.github.io/llm-audit/works/P037/exhibits/P037-N037.png"}], "formal_release": null, "formal_statements": [], "headline": "The means of Table S2 are unadjusted subgroup means; the sentence of the note that says they include adjustment for block fixed effects should be replaced.", "id": "P037-N037", "location": "SI Appendix, Table S2 (PDF p. 37 of the SI Appendix; the page carries no number), note, second sentence", "pdf_page": 37, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Note: Heteroskedasticity-robust standard errors clustered by school in parentheses. \\hl{Mean estimates include adjustment for block fixed effects.} Paired $t$-test $p$ refers to the $p$-value from a $t$-test comparing each variable's school-level means for social referents against the school-level means for non-social referents, pooled across eligible and non-eligible students.''", "corrected": "``Note: Heteroskedasticity-robust standard errors clustered by school in parentheses. \\hl{Mean estimates are unadjusted subgroup means.} Paired $t$-test $p$ refers to the $p$-value from a $t$-test comparing each variable's school-level means for social referents against the school-level means for non-social referents, pooled across eligible and non-eligible students.''", "evidence": "I checked the released code and recalculated the entries on the public data (ICPSR 37070 V2) using adapted portions of the authors' scripts. For a group $g$ with nonmissing values $y_1,\\ldots,y_n$, the script \\texttt{9. Elig and Ref descriptives.R} (lines 73--80) computes\n\\[\n\\widehat m_g=\\frac{1}{n}\\sum_{i=1}^{n}y_i,\\qquad \\widehat V(\\widehat m_g)=\\frac{M}{M-1}\\,\\frac{\\sum_{s}U_s^2}{n^2},\\qquad U_s=\\sum_{i\\in s}(y_i-\\widehat m_g),\n\\]\nwhere $s$ runs over the $M$ schools with nonmissing values in the group; the standard error comes from a regression on a constant alone. The script fits no block effect and applies no weights. On the public data, all 124 means and all 124 standard errors that can be compared (31 rows and four groups) agree with the printed entries at two decimals. The rows Age, Other, Middle East and Native American cannot be compared, because the public release masks or recodes these variables. For illustration, standardizing each subgroup to equal weights for the 14 school blocks changes 62 of the 124 means at two decimals on the public data. In the row White, for example, the printed means are 0.70, 0.63, 0.71 and 0.66, and the means with equal block weights are 0.70, 0.61, 0.68 and 0.62 on the public data.", "problem": "Table S2 reports a mean and a standard error for each of four groups of students (eligible or ineligible, social referent or not) and each of 35 variables. The note says that the means are adjusted for block fixed effects. The released code computes each mean as the average of the variable over the students of the group with a nonmissing value, and no block enters the computation. The standard errors are clustered by school, as the note says.", "what_survives": "The 31 rows reproduced on the public data retain their unadjusted subgroup means and school-clustered standard errors. The main text (p.~567) cites the table for differences between social referents and other students in dating, in compliments on their house and in having older siblings; on the public data the unadjusted differences are 0.23, 0.11 and 0.08, and their direction is as described. The released scripts contain no computation of the column ``Paired $t$-test $p$'', and I have not settled those values. The school-level and social-influence estimates do not use this table."}, "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-N037", "work": {"arxiv_id": null, "doi": "10.1073/pnas.1514483113", "title": "Changing Climates of Conflict: A Social Network Experiment in 56 Schools", "url": "https://pmaronow.github.io/llm-audit/works/P037/"}, "work_id": "P037"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the Findings paragraph of the abstract and the first line of the Conclusions. A red outline surrounds the word entirely in the sentence: any increase appears to be entirely offset by a lowered risk during non-election hours.", "caption": "Abstract, Findings, first page (PDF p. 1). The outline marks the word “entirely” in the second sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-E001.png"}, {"alt": "Crop of five lines of the Results. A red outline surrounds RR = 1.04 in the sentence: When we included the full 24 h on election days, we found no evidence of an effect of presidential elections, with an RR = 1.04 (non-parametric P = 0.570, Wald-type 95 \\% CI [0.89,1.20]).", "caption": "Results, second paragraph, p. 3 (PDF p. 3), left column. The outline marks the full-day estimate, RR = 1.04.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-E001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "“Entirely offset” in the abstract means that the two hourly estimates lie on opposite sides of 1 and that the relative risk for the full day is not statistically unusual; it does not mean exact cancellation, since the point estimate for the full day is 1.04.", "id": "P038-E001", "location": "Abstract, Findings, second sentence, first page (PDF p. 1; the page carries no number); the three estimates are in the Results, p. 3 (PDF p. 3)", "pdf_page": 1, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Abstract, Findings: Although we find weakly suggestive evidence that presidential elections may increase the risk of driving fatalities during election hours, any increase appears to be \\hl{entirely} offset by a lowered risk during non-election hours.\n\nResults, p.~3: When we included the full 24 h on election days, we found no evidence of an effect of presidential elections, with an RR = 1.04 (non-parametric $P = 0.570$, Wald-type 95~\\% CI [0.89,1.20]).", "corrected": "Abstract, Findings, in place of the second sentence: The point estimates indicate an elevated RR during election hours and a reduced RR during non-election hours. For the full day, the estimated RR was 1.04 (non-parametric $P = 0.570$; Wald-type 95\\% confidence interval $[0.89, 1.20]$). We did not detect a statistically significant full-day association. \\hl{These results do not establish exact cancellation or a negligible full-day association.}\n\nThe caution against causal inference from a non-randomized study remains. If the two hourly categories exhaust the records of the full-day analysis, then the full-day RR is the average of the two hourly RRs weighted by the comparison counts, and the point estimates show a partial offset. An exact offset percentage would require confirming this partition of the records and using unrounded counts. No reported RR, interval or $P$ value is replaced.", "intended_meaning": "The Results give relative risks (RRs) of 1.17 for election hours, 0.83 for non-election hours and 1.04 for the full day. In this context the hedged sentence means that the two hourly estimates lie on opposite sides of 1 and that the full-day RR is not unusual among the RRs of the 200 surrounding Tuesdays ($P = 0.570$). The paper reports this. Read literally, ``entirely'' says more than the point estimates show. If the two periods together contain exactly the records of the full-day analysis, then the deficit in non-election hours equals the excess in election hours if and only if the full-day RR is 1, and the printed estimate is 1.04. For example, comparison counts of 2100 in election hours and 1300 in non-election hours, with election-day counts of 2457 and 1079, reproduce the three printed RRs with an excess of 357 and a deficit of 221; these counts are constructed for illustration. The interval $[0.89, 1.20]$ contains 1, and the estimates are compatible with exact cancellation of the underlying risks. They do not establish that the cancellation is exact or that the full-day association is negligible."}, "url": "https://pmaronow.github.io/llm-audit/works/P038/#P038-E001", "work": {"arxiv_id": null, "doi": "10.1186/s13104-016-2148-6", "title": "Driving Fatalities on US Presidential Election Days: A Reanalysis", "url": "https://pmaronow.github.io/llm-audit/works/P038/"}, "work_id": "P038"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of six lines of the Methods, beginning 200 RRs, we constructed an empirical null distribution. Two red outlines, one on each of two consecutive lines, surround the parenthesis (i.e., RR greater or 1/RR smaller), which follows the words the proportion of RRs more extreme.", "caption": "Methods, first paragraph, p. 2 (PDF p. 2), left column. The outlines mark the parenthesis “(i.e., RR greater or 1/RR smaller)”, which runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-E002.png"}, {"alt": "Crop of the three-line caption of Figure 1. Red outlines surround the words The proportion of red dots outside black dashed lines, at the end of the second line and the start of the third. The sentence continues: is equivalent to the reported non-parametric P value. The sentence before it says that black dashed lines indicate RR and 1/RR for presidential election days.", "caption": "Caption of Fig. 1, p. 2 (PDF p. 2). The outlines mark the words “The proportion of red dots outside black dashed lines”, which state the two-tailed rule.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-E002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The parenthesis “(i.e., RR greater or 1/RR smaller)” stands for the two-tailed rule of the Fig. 1 caption: a reference RR is more extreme when it lies outside the interval between the election-day RR and its reciprocal.", "id": "P038-E002", "location": "Methods, p. 2 (PDF p. 2), left column, first paragraph, the parenthesis after “more extreme”; Fig. 1 caption on the same page", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Methods: Using these 200 RRs, we constructed an empirical null distribution, with non-parametric two-tailed $P$ values computed as the proportion of RRs more extreme \\hl{(i.e., RR greater or 1/RR smaller)} than that of presidential election days.\n\nFig.~1 caption: Black circles indicate RRs and black dashed lines indicate RR and 1/RR for presidential election days. The proportion of red dots outside black dashed lines is equivalent to the reported non-parametric $P$ value", "corrected": "Using these 200 RRs, we constructed an empirical null distribution, with non-parametric two-tailed $P$ values computed as the proportion of RRs more extreme \\hl{(i.e., reference RR smaller than $\\min(R, 1/R)$ or greater than $\\max(R, 1/R)$, where $R$ is the election-day RR)} than that of presidential election days.\n\nIn symbols, let $R > 0$ be the election-day RR for a given time window, and let $r_1, \\ldots, r_{200} > 0$ be the RRs of the reference Tuesdays. Then\n\\[\np_{\\mathrm{emp}} = \\frac{1}{200} \\sum_{j=1}^{200} \\mathbf{1}\\bigl\\{ r_j < \\min(R, 1/R) \\ \\text{or} \\ r_j > \\max(R, 1/R) \\bigr\\},\n\\]\nwhich is the proportion of indices $j$ such that $|\\log r_j| > |\\log R|$. The strict inequalities follow the caption's word ``outside''; another treatment of ties would have to be stated. This wording clarifies the description and changes no reported $P$ value.", "intended_meaning": "Let $R > 0$ be the election-day RR, and let $r > 0$ be the RR of a reference Tuesday, one of the 200 Tuesdays that form the null distribution. The word ``two-tailed'' and the caption fix the rule: $r$ is counted when it lies outside the dashed lines at $R$ and $1/R$, that is, when $r < \\min(R, 1/R)$ or $r > \\max(R, 1/R)$. Since the logarithm is increasing and $\\log(1/R) = -\\log R$, this condition is equivalent to $|\\log r| > |\\log R|$. The parenthesis abbreviates this rule and has no single literal reading. If it compares $r$ with $R$ and $1/r$ with $1/R$, then the two conditions are the same condition, $r > R$. If it compares $1/r$ with $R$, then the condition is $r > \\min(R, 1/R)$. For $R = 1.17$, both readings omit $r = 0.8$, which lies below $1/R$. No reported $P$ value is shown to be miscalculated, and the treatment of exact ties in the computation is not known."}, "url": "https://pmaronow.github.io/llm-audit/works/P038/#P038-E002", "work": {"arxiv_id": null, "doi": "10.1186/s13104-016-2148-6", "title": "Driving Fatalities on US Presidential Election Days: A Reanalysis", "url": "https://pmaronow.github.io/llm-audit/works/P038/"}, "work_id": "P038"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the six lines of footnote 2 below the footnote rule. A red outline surrounds the words surrounding 12, 25 and 50 Tuesdays in the first sentence: We also computed P values for alternative time periods for the null distribution, namely for the surrounding 12, 25 and 50 Tuesdays.", "caption": "Footnote 2, p. 3 (PDF p. 3), left column. The outline marks the words “surrounding 12, 25 and 50 Tuesdays”.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-N004.png"}, {"alt": "Crop of the last three lines of the Results and the heading Discussion. A red outline surrounds the words the surrounding 200 Tuesdays in the sentence: Figure 1 shows the RRs for election days and the surrounding 200 Tuesdays. A superscript 2 follows the sentence.", "caption": "Results, last sentence, p. 3 (PDF p. 3), left column. The outline marks the words “the surrounding 200 Tuesdays”; the footnote mark 2 follows them.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-N004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P038-T1", "url": "https://pmaronow.github.io/llm-audit/works/P038/#P038-T1"}], "headline": "In footnote 2, “the surrounding 12, 25 and 50 Tuesdays” are numbers of Tuesdays on each side of election day, which give 24, 50 and 100 reference RRs; “the surrounding 200 Tuesdays” in the sentence that carries the footnote is a total.", "id": "P038-N004", "location": "Results, last sentence, and footnote 2, first sentence, p. 3 (PDF p. 3), left column; the convention is in the Methods, p. 2 (PDF p. 2)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P038-T1"], "text": {"as_printed": "Results, last sentence: Figure 1 shows the RRs for election days and \\hl{the surrounding 200 Tuesdays}.$^{2}$\n\nFootnote 2: We also computed $P$ values for alternative time periods for the null distribution, namely for the \\hl{surrounding 12, 25 and 50 Tuesdays}. For the 12 surrounding Tuesday specifications, we find for election hours, non-election hours, and the full day, respective $P$ values of 0.04, 0.04, and 0.46. For 25 Tuesdays, we find respectively 0.06, 0.02, and 0.42. For 50 Tuesdays, we find respectively 0.05, 0.08, and 0.47.", "corrected": "We also computed $P$ values for alternative time periods for the null distribution, namely for \\hl{the 12, 25 and 50 Tuesdays before and after presidential election days (24, 50 and 100 reference RRs, respectively)}.\n\nThe later sentences of the footnote then refer to windows of 12, 25 and 50 Tuesdays on each side. No printed $P$ value changes.", "intended_meaning": "The Methods estimate RRs for ``the 100 Tuesdays before and after presidential election days'' and call the result ``these 200 RRs''. This count implies that a window is named by its size on each side. With the same convention, the footnote's 12, 25 and 50 are numbers of Tuesdays on each side, and the null distributions have 24, 50 and 100 reference RRs. The nine printed $P$ values are compatible with this reading: each is a proportion $k/n$ rounded to two decimals, with $k = 1, 1, 11$ for $n = 24$, with $k = 3, 1, 21$ for $n = 50$, and with $k = 5, 8, 47$ for $n = 100$. They are not compatible with totals of 12, 25 and 50. For example, 0.04 is not $k/12$ rounded to two decimals for any integer $k$, and 0.06 is not $k/25$. These counts show compatibility only, and I did not reconstruct the windows from data. The value 0.06 is the subject of \\fid{P038-N005}."}, "url": "https://pmaronow.github.io/llm-audit/works/P038/#P038-N004", "work": {"arxiv_id": null, "doi": "10.1186/s13104-016-2148-6", "title": "Driving Fatalities on US Presidential Election Days: A Reanalysis", "url": "https://pmaronow.github.io/llm-audit/works/P038/"}, "work_id": "P038"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the six lines of footnote 2. A red outline surrounds the value 0.06, the first of three values in the sentence: For 25 Tuesdays, we find respectively 0.06, 0.02, and 0.42.", "caption": "Footnote 2, p. 3 (PDF p. 3), left column. The outline marks 0.06, the election-hours $P$ value for the window of 25 Tuesdays on each side.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-N005.png"}, {"alt": "Crop of the top panel of Figure 1, titled Relative Risk of Driving Fatalities (Election Hours). Red dots show the relative risk, on a vertical axis from 0.50 to 1.50, for each Tuesday from 100 weeks before to 100 weeks after presidential election day. Two horizontal black dashed lines cross the panel, one above 1.00 and one below it. A large red outline encloses the plot between weeks -25 and +25. Inside it, two small red outlines each surround one red dot: a dot just below the lower dashed line at week -14, and a dot above the upper dashed line at week +7. A black circle surrounds the dot at week 0, which sits on the upper dashed line. Every other reference dot between weeks -25 and +25 lies between the dashed lines.", "caption": "Fig. 1, election-hours panel, p. 2 (PDF p. 2). The large outline spans weeks $-25$ to $+25$. The two small outlines mark the only reference RRs in this window that lie outside the dashed lines, at weeks $-14$ and $+7$. The circled point at week 0 is the election day.", "url": "https://pmaronow.github.io/llm-audit/works/P038/exhibits/P038-N005-b.png"}], "formal_release": null, "formal_statements": [{"id": "P038-T1", "url": "https://pmaronow.github.io/llm-audit/works/P038/#P038-T1"}], "headline": "Footnote 2 prints 0.06 for the election-hours $P$ value with 25 Tuesdays on each side, while Fig. 1 shows two of the 50 reference RRs outside the dashed lines, which gives $2/50 = 0.04$; one of the two must be corrected, and the published sources do not show which.", "id": "P038-N005", "location": "Footnote 2, third sentence, first value, p. 3 (PDF p. 3); Fig. 1, election-hours panel, and its caption, p. 2 (PDF p. 2)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P038-T1"], "text": {"as_printed": "Footnote 2, third sentence: For 25 Tuesdays, we find respectively \\hl{0.06}, 0.02, and 0.42.\n\nFig.~1 caption: Black circles indicate RRs and black dashed lines indicate RR and 1/RR for presidential election days. The proportion of red dots outside black dashed lines is equivalent to the reported non-parametric $P$ value", "corrected": "If Fig.~1 shows the final analysis, then the third sentence of footnote 2 reads: For 25 Tuesdays, we find respectively \\hl{0.04}, 0.02, and 0.42.\n\nHere $0.04 = 2/50$. Of the 50 reference RRs for election hours within 25 weeks of election day, the two of weeks $-14$ and $+7$ lie outside the dashed lines at the election-day RR and its reciprocal. If instead 0.06 is the value from the final data, then the footnote stands, and the election-hours panel of Fig.~1 must show a third reference RR outside the dashed lines within 25 weeks of election day. The published figure contains no such point. The choice between the two corrections requires the final data.", "evidence": "The evidence is a recalculation from the published figure. Fig.~1 is embedded in the PDF as an image of $1595 \\times 1740$ pixels. In the election-hours panel, the dashed lines are at pixel rows 170 and 274, counted downward, and one week spans 6.55 pixels. Four red points in the window from week $-25$ to week $+25$ touch a dashed line or lie beyond it.\n\\begin{audittable}{rrll}{Election-hours panel of Fig.~1: the red points within 25 weeks of election day that reach a dashed line. Rows are pixel rows of the embedded image, counted downward.}\nWeek & Center row & Position & Counted \\\\\n$-14$ & 281.5 & 7.5 rows below the lower line & yes \\\\\n$0$ & 170 & on the upper line; the election day & no \\\\\n$+6$ & 267 & 7 rows above the lower line, inside & no \\\\\n$+7$ & 148.5 & 21.5 rows above the upper line & yes \\\\\n\\end{audittable}\nAll other red points in the window lie between rows 174 and 271, inside the lines. The point of week 0 is the election day, which is not a reference Tuesday. The classification is the same for five thresholds that define a red pixel, and it does not change when each center and each line is moved by up to two pixels. A second count, which uses the red ink beyond each line and the median row of each point, finds the same two points. Restricting the count to 12 weeks on each side, I find the point of week $+7$ only, and $1/24 \\approx 0.0417$ agrees with the footnote's 0.04 for that window. If the election day is counted as a third extreme point, then the proportion is $3/51 \\approx 0.0588$, which rounds to 0.06. The same rule gives $2/25 = 0.08$ for the window of 12 Tuesdays, against the printed 0.04. The abstract of the thesis on which the article is based reports no $P$ values for the alternative windows, and the full text of the thesis was not examined.", "problem": "The first value in this sentence is the election-hours $P$ value when the null distribution is restricted to the 25 Tuesdays before and the 25 Tuesdays after election day (see \\fid{P038-N004} for the count). By the Methods and the caption of Fig.~1, this value is the proportion of the 50 reference RRs that lie outside the dashed lines of the election-hours panel. In the published figure, exactly two of the 50 lie outside: the RR of week $-14$ is below the lower line, and the RR of week $+7$ is above the upper line. From the figure, I obtain $2/50 = 0.04$. The printed 0.06 equals $3/50$ and requires a third RR outside the lines. The footnote and the figure cannot both be right under the stated procedure. The published sources do not show which of them reflects the final data; I did not have the final data or analysis code. With 0.04, the value would move from above 0.05 to below it.", "what_survives": "Under the figure-based correction, the election-hours RR of 1.17, with $P = 0.070$ and interval $[0.99, 1.35]$ for 100 Tuesdays on each side, the estimates for non-election hours and the full day, and the conclusion for the full day are unchanged. No later calculation is stated to use this footnote value. If the footnote is the output in error, then the election-hours $P$ values for the windows of 12, 25 and 50 Tuesdays read 0.04, 0.04 and 0.05. If the figure is in error, then its plotted RRs must be checked against the final data. I did not recompute the other reported values from data."}, "url": "https://pmaronow.github.io/llm-audit/works/P038/#P038-N005", "work": {"arxiv_id": null, "doi": "10.1186/s13104-016-2148-6", "title": "Driving Fatalities on US Presidential Election Days: A Reanalysis", "url": "https://pmaronow.github.io/llm-audit/works/P038/"}, "work_id": "P038"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the display and the lines around it. A red box encloses the display: 'Incumbent extremism = [(Incumbent position - Voter mean) + (Challenger position - Voter mean)] / 2 = [Incumbent position + Challenger position] / 2 - Voter mean'. Above it the text reads 'all recentered to a range of -3 to +3, we measure relative incumbent extremism as follows:'. Below it: 'In the special case in which mean voter ideology is 0, this second method of calculating relative candidate extremism reduces to the first.'", "caption": "p. 133 (PDF p. 8). The marked display defines the voter-centered measure of relative incumbent extremism; the sentence before it names the scale, and the sentence after it states the reduction to the first method.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-A001.png"}], "formal_release": null, "formal_statements": [], "headline": "The display for the voter-centered measure leaves implicit that each candidate position is centered at the voter mean before its absolute value is taken and signed by incumbency; the measure that was computed is $(|x_I-m|-|x_C-m|)/2$.", "id": "P039-A001", "location": "p. 133 (PDF p. 8), the display that begins “Incumbent extremism =” and the sentence after it; the sign convention is on p. 132 (PDF p. 7) and in note 11, p. 141 (PDF p. 16)", "pdf_page": 8, "printed_page": "133", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Based on the self-placement and two candidate placement scales---all recentered to a range of $-3$ to $+3$---we measure relative incumbent extremism as follows:\n\\[\n\\begin{aligned}\n\\text{Incumbent extremism}&=\\left[\\begin{array}{l}{}\\hl{(\\text{Incumbent position}-\\text{Voter mean})+}\\\\ {}\\hl{(\\text{Challenger position}-\\text{Voter mean})}\\end{array}\\right]/2\\\\\n&=\\hl{[\\text{Incumbent position}+\\text{Challenger position}]/2-\\text{Voter mean}}\n\\end{aligned}\n\\]\nIn the special case in which mean voter ideology is 0, this second method of calculating relative candidate extremism reduces to the first.''", "corrected": "``Based on the self-placement and two candidate placement scales---all recentered to a range of $-3$ to $+3$---we measure relative incumbent extremism as follows. Let $x_I$ and $x_C$ be the positions of the incumbent-party candidate and of the challenger on this scale, and let $m$ be the voter mean. Each candidate position is first centered at $m$. The absolute value is then taken, and the challenger's receives a negative sign:\n\\[\n\\text{Incumbent extremism}=\\hl{\\bigl(|x_I-m|-|x_C-m|\\bigr)/2}.\n\\]\nIn the special case in which mean voter ideology is 0, this second method of calculating relative candidate extremism reduces to the first.''\n\nThe formula replaces both lines of the printed display; the midpoint of the signed positions minus the voter mean is not kept as a second equality. The display should have made clear that centering at the voter mean precedes the orientation by incumbent and challenger. The scores, Table 1, Table 3 and the conclusions are unchanged, since the archived scores already follow this formula.", "intended_meaning": "Recall that the first method (p.~132 and note 11) takes the absolute value of each candidate's position, assigns a negative sign to the challenger's and takes the midpoint. Let $x_I$ and $x_C$ be the positions of the incumbent-party candidate and of the challenger on the liberal--conservative scale. Then the first measure is $(|x_I|-|x_C|)/2$. The second method moves the reference point from zero to the voter mean $m$. The measure intended is\n\\[\nE_m=\\bigl(|x_I-m|-|x_C-m|\\bigr)/2,\n\\]\nwhich is positive when the incumbent-party candidate is farther from the voter mean than the challenger. This is the measure that was computed. The replication code subtracts the mean self-placement of voters from each candidate position, takes absolute values, signs them by incumbency and averages. The formula for $E_m$ reproduces the 22 archived scores of the voter-centered measures (two measures in each of the eleven elections of 1972--2012) to within $6\\times10^{-15}$. It also reproduces the two correlations with vote share in Table 1, $-.08$ and $-.07$. Setting $m=0$, I obtain $E_0=(|x_I|-|x_C|)/2$, so that the sentence after the display is correct.\n\nThe display does not state these conventions. The sentence before the display refers to the recentered scale, on which liberal positions are negative and conservative positions are positive. Read on this scale, the display is the midpoint of the two signed positions minus the voter mean, $P_m=(x_I+x_C)/2-m$. Let $x_I=-1$, $x_C=2$ and $m=1/4$. Then the distances from the voter mean are $5/4$ and $7/4$, and I have $E_m=-1/4$ and $P_m=1/4$. The display thus reverses which candidate is the more extreme. If the candidate positions are first transformed as in note 11, to $1$ and $-2$, and the voter mean is left on its original scale, then the display equals $-3/4$. In the archived data, $P_m$ agrees with the archived score up to rounding in the seven elections with a Republican incumbent party. In the four elections with a Democratic incumbent party (1980, 1996, 2000 and 2012), it is the negative of the archived score. For the all-voter measure in 1996, I have $x_I=-0.9468$, $x_C=1.1557$ and $m=0.4413$, so that $P_m=-0.3369$, and the archived score is $0.3369$.\n\nOne reading in context makes the display correct. Suppose that the voter mean lies between the two candidates. If $x_C\\le m\\le x_I$, then $E_m=(x_I+x_C)/2-m$, and if $x_I\\le m\\le x_C$, then $E_m=m-(x_I+x_C)/2$. This implies that the display holds when the whole scale, the voter mean included, is reflected in the elections in which the incumbent-party candidate is the more liberal. The voter mean lies between the two candidates in all eleven elections. The printed display can be read correctly, but only with conventions that the text leaves implicit."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-A001", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text. A red box marks the words 'quarters 1-16' in the sentence 'The dependent variable in Panel B is incumbent party share of the two-party vote, and RDI is measured using Hibbs's definition of the variable, with quarters 1-16 discounted by a decay parameter as described above.'", "caption": "p. 136 (PDF p. 11), paragraph on Panel B of Table 2. The marked range is given for the income measure of the bread and peace models.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E001.png"}, {"alt": "Crop of the top of the Bread and Peace panel of Table 2, with the column heads B1, B2 and B3. The first row is labelled 'Delta weighted RDI in Q 1-16', and a red box marks 'Q 1-16'. The row gives the coefficients 3.29, 2.82 and 2.89, each with three asterisks, and the standard errors (.52), (.44) and (.44).", "caption": "Table 2, p. 137 (PDF p. 12), first row of the Bread & Peace panel, turned upright (the table is printed sideways). The marked label gives the same range.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E001-b.png"}], "formal_release": null, "formal_statements": [{"id": "P039-T1", "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-T1"}, {"id": "P039-T2", "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-T2"}], "headline": "The income measure in every bread and peace model that was fitted covers quarters 2–16 of the presidential term, and not quarters 1–16 as the text and Table 2 say.", "id": "P039-E001", "location": "Paragraph on Panel B of Table 2, p. 136 (PDF p. 11); section “The `bread and peace' model”, p. 128 (PDF p. 3); Table 2, Bread & Peace panel, first row label, p. 137 (PDF p. 12)", "pdf_page": 11, "printed_page": "136", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P039-T1", "P039-T2"], "text": {"as_printed": "p.~128: ``While Hibbs, like Achen and Bartels (2016), also employs change in RDI to measure economic performance, bread and peace allows RDI growth from \\hl{every quarter} of a presidential term to factor into the model, albeit discounted over time so that earlier quarters carry less weight. Hibbs applies a decay parameter---$\\lambda$, estimated within the model as a number between 0 and 1---to progressively discount economic growth in earlier quarters of the presidential term. This discounting takes the following form: Because the election is held one-third of the way through the 16th quarter of a presidential term, RDI growth in this quarter is down-weighted by $\\tfrac{1}{3} * \\lambda^{0}$. The 15th quarter of the presidential term is down-weighted by $\\lambda^{1}$, the 14th quarter by $\\lambda^{2}$, and so on for the preceding quarters.''\n\np.~136: ``The dependent variable in Panel B is incumbent party share of the two-party vote, and RDI is measured using Hibbs's definition of the variable, with \\hl{quarters 1--16} discounted by a decay parameter as described above.''\n\nTable 2, p.~137, Bread \\& Peace panel, first row label: ``$\\Delta$ weighted RDI in \\hl{Q 1-16}''", "corrected": "p.~136: ``The dependent variable in Panel B is incumbent party share of the two-party vote, and RDI is measured using Hibbs's definition of the variable, with \\hl{quarters 2--16} discounted by a decay parameter as described above.''\n\nTable 2, Bread \\& Peace panel, first row label: ``$\\Delta$ weighted RDI in \\hl{Q 2-16}''\n\np.~128, to be added after ``and so on for the preceding quarters'': In the analyses reported here, the weighted measure of income growth uses quarters 2--16 of each presidential term, and quarter 1 is excluded. Quarter 16 receives the weight $1/3$, quarters 15 through 2 receive the weights $\\lambda$ through $\\lambda^{14}$, and the weighted sum is divided by the sum of the weights. With $h_q$ the annualized growth of RDI in quarter $q$, the measure is\n\\[\n\\frac{h_{16}/3+\\sum_{k=1}^{14}\\lambda^{k}h_{16-k}}{1/3+\\sum_{k=1}^{14}\\lambda^{k}},\n\\]\nand $\\lambda$ is estimated together with the other coefficients. If note 5 is kept as a description of these models, its exponents end at 14 and not at 15; the note is the subject of \\fid{P039-E006}.\n\nAlternatively, the range 1--16 is kept in the text and the table, and $\\lambda^{15}h_1$ and $\\lambda^{15}$ are added to the numerator and the denominator of each of the nine formulas. Panel B of Table 2, the bread and peace half of Table 3, Figure 4B and the numbers quoted from them are then updated. The extremism coefficient in column B3 becomes $-2.30$, with standard error $2.08$ and p-value $.29$, and the p-value for incumbent fatigue in column B3 becomes $.050$, so that its marker for $p<.05$ is removed.", "evidence": "The evidence is a check of the nine formulas, an exact counterexample which shows that the two measures differ, and a refit of the models. I write $N_{15}$ and $D_{15}$ for the numerator and the denominator of $G_{15}$. The measure as described, with quarter 1 included, is\n\\[\nG_{16}=\\frac{N_{15}+\\lambda^{15}h_1}{D_{15}+\\lambda^{15}}.\n\\]\nSubtracting $G_{15}$, I obtain\n\\[\nG_{16}-G_{15}=\\frac{\\lambda^{15}}{D_{15}+\\lambda^{15}}\\,(h_1-G_{15}).\n\\]\nFor $0<\\lambda<1$, the two measures agree only if $h_1=G_{15}$. For the counterexample, let $\\lambda=9/10$, $h_1=1$ and $h_2=\\cdots=h_{16}=0$. Then $G_{15}=0$ in each of the nine archived formulas, and for the measure as described I find\n\\[\nG_{16}=\\frac{(9/10)^{15}}{1/3+\\sum_{k=1}^{15}(9/10)^{k}}=\\frac{617673396283947}{22440939433444477}\\approx 0.0275.\n\\]\nFitted with the archived formulas, the models reproduce the coefficients of Panel B of Table 2 and the five bread and peace entries of Table 3 to the printed precision. The models were then fitted again with $\\lambda^{15}h_1$ added to the numerator and $\\lambda^{15}$ to the denominator, and nothing else changed. The results of the two fits are in the table below. The p-values are the approximate ones that the fitting routine reports.\n\\begin{audittable}{lrr}{Column B3 of Table 2, the bread and peace models of Table 3 and the fitted line of Figure 4B, with the income measure as archived (quarters 2--16) and with quarter 1 restored (quarters 1--16). Maximum effects are in points of the two-party vote share, with the p-value of the extremism coefficient in parentheses.}\nQuantity & Quarters 2--16 & Quarters 1--16 \\\\\n\\hline\nB3, relative extremism: coefficient & $-2.228$ & $-2.300$ \\\\\nB3, relative extremism: standard error & $2.045$ & $2.078$ \\\\\nB3, relative extremism: p-value & $.301$ & $.294$ \\\\\nB3, incumbent fatigue: p-value & $.0466$ & $.0501$ \\\\\nMaximum effect, all-voter zero-centered & $1.24$ $(.391)$ & $1.27$ $(.389)$ \\\\\nMaximum effect, all-voter voter-centered & $0.30$ $(.848)$ & $0.28$ $(.856)$ \\\\\nMaximum effect, refined zero-centered & $1.38$ $(.301)$ & $1.43$ $(.294)$ \\\\\nMaximum effect, refined voter-centered & $0.82$ $(.639)$ & $0.82$ $(.640)$ \\\\\nMaximum effect, NOMINATE & $1.14$ $(.773)$ & $1.14$ $(.774)$ \\\\\nFigure 4B, fitted line: p-value & $.259$ & $.252$ \\\\\n\\end{audittable}", "problem": "The paragraph on p.~136 says that the income measure in Panel B of Table 2 discounts quarters 1--16, and the row label in the table repeats this range. The description on p.~128 assigns the weight $1/3$ to quarter 16, the weight $\\lambda$ to quarter 15 and the weight $\\lambda^{2}$ to quarter 14, ``and so on for the preceding quarters''. Carried back to the start of the term, this rule implies that quarter 1 has the weight $\\lambda^{15}$. The models that were fitted stop at quarter 2. Let $h_q$ be the annualized growth of real disposable income in quarter $q$ of the term. The replication files contain nine formulas that are fitted by nonlinear least squares, eight for Tables 2 and 3 and one for Figure 4B. In each of them the income measure is\n\\[\nG_{15}=\\frac{h_{16}/3+\\sum_{k=1}^{14}\\lambda^{k}h_{16-k}}{1/3+\\sum_{k=1}^{14}\\lambda^{k}}.\n\\]\nThe growth $h_1$ of the first quarter is in the data for each of the sixteen elections of 1952--2012, and it enters none of the formulas. The stated range is therefore not the range of the fitted models. Hibbs (2012) and the online appendix were not available. I could not determine whether Hibbs's own specification uses fifteen or sixteen quarters, or whether quarter 1 was left out on purpose.", "what_survives": "If the description is corrected to quarters 2--16, this finding changes no number in the paper. The ideology measures, Table 1, Panel A of Table 2, the musical chairs half of Table 3, Figures 1--3 and Figure 4A do not involve the income measure of the bread and peace models. If quarter 1 is restored instead, the entries of Panel B of Table 2, the bread and peace half of Table 3, Figure 4B and the numbers quoted from them on pp.~138--139 change; the table shows the changes for column B3, Table 3 and Figure 4B. The p-value for incumbent fatigue in column B3 moves from $.047$ to $.050$, so that its marker for $p<.05$ would be removed. Under either repair the five bread and peace extremism coefficients are negative and far from significance (with quarter 1 restored, p-values from $.294$ to $.856$ and maximum effects from $0.28$ to $1.43$ points). The paper's finding of scant evidence of an extremism penalty is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-E001", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last five lines of the page. The final sentence begins 'The ability of candidate extremism to explain the remaining variation in vote share' and says that the relationship depicted in panel B remains well short of statistical significance. A red box marks its closing clause, 'the two-tailed p-value for the coefficient is .26.'", "caption": "p. 138 (PDF p. 13), end of the last paragraph. The marked clause gives the $p$-value of the line fitted to the residuals of Figure 4, panel B.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P039-T3", "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-T3"}, {"id": "P039-T4", "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-T4"}], "headline": "The $p$-value of .26 on p. 138 is the nominal $p$-value of the line fitted to the residuals in Figure 4, panel B; the test of extremism given the fundamentals is the test of the extremism coefficient of column B3, whose approximate $p$-value is .30.", "id": "P039-E002", "location": "p. 138 (PDF p. 13), last sentence of the page, on Figure 4, panel B; compare Table 2, column B3, p. 137 (PDF p. 12), and Table 3, row (3), p. 139 (PDF p. 14)", "pdf_page": 13, "printed_page": "138", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P039-T3", "P039-T4"], "text": {"as_printed": "p.~138: ``The ability of candidate extremism to explain the remaining variation in vote share from the bread and peace--based model is stronger than in the musical chairs--based model, but the relationship depicted in panel B remains well short of statistical significance: \\hl{the two-tailed $p$-value for the coefficient is .26}.''\n\np.~139: ``As Table 3 also reports, the $p$-value for the coefficient underlying this estimate is .30 and based on ten degrees of freedom.''", "corrected": "``The ability of candidate extremism to explain the remaining variation in vote share from the bread and peace--based model is stronger than in the musical chairs--based model. \\hl{The negative slope in panel B is descriptive. In the full bread and peace composite model B3, the extremism coefficient is $-2.23$, with an approximate two-tailed $p$-value of .30 on ten degrees of freedom; it remains well short of statistical significance.}''", "evidence": "I checked the inference by recalculating it from the replication data and by an exact counterexample in a linear model. For the regression of the residuals, I find slope $-1.8917$, standard error $1.6067$, $t=-1.177$ and $p=.2587$ on 14 degrees of freedom. The extremism coefficient of B3 is $-2.2285$, with standard error $2.0450$, $t=-1.090$ and $p=.3014$ on 10 degrees of freedom.\n\nFor the counterexample, take six observations and the vectors\n\\[\nu=(1,-1,0,0,0,0),\\qquad v=(0,0,1,-1,0,0),\\qquad w=(0,0,0,0,1,-1).\n\\]\nLet the first-step regressors be the constant and $u$, let $x=v$, and let $y=u+(3/2)v+w$. The first-step residual is $r=(3/2)v+w$. Regressing $r$ on the constant and $x$, I obtain slope $3/2$ and residual sum of squares 2 on 4 degrees of freedom. Then $t^2=9$, and the nominal two-tailed $p$-value is $1-45/(13\\sqrt{13})$, about $.040$. Regressing $y$ on the constant, $u$ and $x$ gives the same slope and the same residual sum of squares on 3 degrees of freedom. Then $t^2=27/4$, and the $p$-value is about $.081$. The regression of the residuals rejects at the 5\\% level, and the full regression does not. With $x=u+v$ in place of $x=v$, the slope in the regression of the residuals is $3/4$ and the coefficient of $x$ in the full regression is $3/2$. In a linear model, the first slope equals the second times a fraction: the part of the centred sum of squares of $x$ that remains after $x$ is adjusted for the first-step regressors. Here this fraction is $1/2$. Under the Gaussian null hypothesis with $x=v$, the test statistic of the regression of the residuals is distributed as $\\sqrt{4/3}\\,T_3$, where $T_k$ denotes a Student variable with $k$ degrees of freedom. Its second moment is 4, and that of the reference variable $T_4$ is 2.\n\nThe paper's model is nonlinear, and the same two effects appear through the Jacobian of B2 at the fitted values, which has rank 5. The centred sum of squares of the extremism variable is $1.0027$. After the variable is adjusted for the five columns of the Jacobian, $.8498$ of it remains, a fraction of $.85$. Neither .26 nor .30 is an exact finite-sample $p$-value for the nonlinear regression.", "problem": "The sentence reports the test of the slope of the line in Figure 4, panel B. That line is an ordinary regression of the residuals of model B2 on the extremism variable, over 16 elections. Its slope is $-1.89$ with standard error $1.61$, and the nominal two-tailed $p$-value on 14 degrees of freedom is .26. The number is reported correctly. The sentence uses it to say how far extremism is from statistical significance once the fundamentals are accounted for, and the regression of the residuals does not test that. There are two reasons. The extremism variable is not adjusted for the fundamentals, so the slope of the line is not the coefficient of the full model. Further, the standard error of the slope and its 14 degrees of freedom take no account of the five parameters already fitted in B2. The test that conditions on the fundamentals is the test of the extremism coefficient of column B3. That coefficient is $-2.23$ with standard error $2.04$, and its approximate $p$-value is .30 on ten degrees of freedom, as Table 3 and p.~139 report. Both $p$-values are far from significance.", "what_survives": "Figure 4, panel B, stands as a descriptive plot of the residuals, with its fitted line. The coefficient of column B3, row (3) of Table 3 and the worked example on p.~139 are unchanged, and no calculation in the paper uses the .26. The conclusion that the relationship is well short of statistical significance is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-E002", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the left half of Table 2, with columns A1, A2 and A3 under the heading Musical Chairs. The first row, Delta RDI in Q14-15, shows 6.40, 6.38 and 6.38, each with three stars, and the standard errors (1.4), (.96) and (1.0) beneath them. The rows below are Incumbent fatigue, War fatalities, Relative extremism with -.46 (5.19) in column A3, Constant, N of cases 17, 17, 17, adjusted r-squared .81, .80, .79, and RMSQ 4.62, 4.70, 4.89. Red boxes mark 6.40 together with (1.4) in column A1, 6.38 in column A2 and 6.38 in column A3.", "caption": "Table 2, p. 137 (PDF p. 12), Musical Chairs panel, turned upright (the table is printed sideways). The marked entries are the three income coefficients and the standard error of column A1.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P039-T7", "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-T7"}], "headline": "In the Musical Chairs panel of Table 2, the income coefficients are 6.39, 6.37 and 6.39, and the standard error in column A1 is .94, not 1.4.", "id": "P039-E003", "location": "Table 2, p. 137 (PDF p. 12), Musical Chairs panel, row “$\\Delta$ RDI in Q14-15”, columns A1–A3", "pdf_page": 12, "printed_page": "137", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P039-T7"], "text": {"as_printed": "\\begin{audittable}{lrrr}{Table 2, Musical Chairs panel, as printed (the income row and the two rows used below).}\n & A1 & A2 & A3 \\\\\n$\\Delta$ RDI in Q14-15 & \\hl{6.40}$^{***}$ & \\hl{6.38}$^{***}$ & \\hl{6.38}$^{***}$ \\\\\n & \\hl{(1.4)} & (.96) & (1.0) \\\\\nN of cases & 17 & 17 & 17 \\\\\nRMSQ & 4.62 & 4.70 & 4.89 \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lrrr}{Table 2, Musical Chairs panel, income row as corrected.}\n & A1 & A2 & A3 \\\\\n$\\Delta$ RDI in Q14-15 & \\hl{6.39}$^{***}$ & \\hl{6.37}$^{***}$ & \\hl{6.39}$^{***}$ \\\\\n & \\hl{(.94)} & (.96) & (1.0) \\\\\n\\end{audittable}\n\nThe standard errors of columns A2 and A3 are correct as printed; at two decimals the latter is 1.01. To four decimals the pairs are $6.3938$ $(.9406)$, $6.3723$ $(.9571)$ and $6.3855$ $(1.0069)$.", "evidence": "I checked the row by recalculating it from the replication data in exact rational arithmetic, and by a derivation that uses the printed entries alone. Fitting the three regressions to the 17 observations in the replication data, I find $6.3938$ $(.9406)$ for A1, $6.3723$ $(.9571)$ for A2 and $6.3855$ $(1.0069)$ for A3. The three regression calls of the authors' script, run unchanged, return the same values.\n\nFor the derivation, let $q$ be the income variable, and let $R_j$ be the residual sum of squares when $q$ is regressed on the other regressors of model $j$: the constant and incumbent fatigue for A1, and these together with war fatalities for A2. Adding a regressor cannot raise this sum, so that $R_2\\le R_1$. The conventional standard error of the coefficient of $q$ in model $j$ is $s_j/\\sqrt{R_j}$, where $s_j$ is the residual standard error, printed as RMSQ. Combining these two facts, I have\n\\[\n\\operatorname{SE}_1\\le \\operatorname{SE}_2\\, s_1/s_2 .\n\\]\nEach printed entry may lie anywhere in its rounding interval, so that $\\operatorname{SE}_2\\le .965$, $s_1\\le 4.625$ and $s_2\\ge 4.695$. I obtain\n\\[\n\\operatorname{SE}_1\\le \\frac{.965\\times 4.625}{4.695}=\\frac{7141}{7512}<.951 .\n\\]\nA standard error printed as 1.4 is at least 1.35. Then no data can give the printed standard error of A1 together with the printed .96, 4.62 and 4.70. If RMSQ denoted $\\sqrt{\\mathrm{SSE}/n}$ instead, the bound would be smaller by the factor $\\sqrt{13/14}$.", "problem": "The row reports the income coefficients of three nested regressions, fitted by least squares to the same 17 elections, with conventional standard errors. The replication data yield 6.39 (.94), 6.37 (.96) and 6.39 (1.01). Each printed coefficient is off by .01. The standard error printed for column A1 is too large by about .46, which rounding cannot explain. The $t$-ratio of that coefficient is 6.80, not the 4.57 that the printed pair implies. The replication script for the table carries a note on this row. It says that, because of ``a minor calculation error'', the published estimates and standard errors for this variable ``differ slightly from those produced below'', and that the values the script produces ``are the correct values''. It is not established whether the printed 1.4 comes from that calculation error or from typesetting.", "what_survives": "The three income coefficients remain positive with $p<.01$, and the significance stars stand. Every other entry of the Musical Chairs panel agrees with the recalculation at the printed precision, including the extremism coefficient of column A3, $-.46$ $(5.19)$. The five rows of Table 3 for the musical chairs model also agree with it; they are computed from the fitted regressions, not from the printed standard error. The Bread \\& Peace models use another income variable. The quarterly income series is not in the replication files, and the construction of the income variable from it was not checked."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-E003", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines of the notes: the last words of note 14, 'have reported.', and the three lines of note 15. A red box marks the first sentence of note 15, 'For the eight measures based on ANES/Rosenstone, this was the 1980 election.' The second sentence, on the NOMINATE measure and the 1964 election, is unmarked.", "caption": "Note 15, p. 142 (PDF p. 17). The marked sentence assigns all eight estimates based on voter perceptions to the 1980 election.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E004.png"}, {"alt": "Crop of five lines of text. A red box marks 'the largest value of that variable.' in the sentence 'Maximum effect is calculated as the coefficient for the variable times the largest value of that variable.' A second red box marks 'The largest value' at the start of the sentence 'The largest value of the zero-centered refined variable occurs in 1980'.", "caption": "p. 139 (PDF p. 14), first paragraph under Table 3. The phrase “largest value” is marked in the definition of the maximum effect and at the start of the worked example; it occurs a third time in the next paragraph.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E004-b.png"}], "formal_release": null, "formal_statements": [{"id": "P039-T5", "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-T5"}, {"id": "P039-T6", "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-T6"}], "headline": "Note 15 should assign the 1980 election to the four estimates from the two zero-centered measures and the 1972 election to the four estimates from the two voter-centered measures; on p. 139, “the largest value” is the score of greatest absolute magnitude, with its sign.", "id": "P039-E004", "location": "Note 15, p. 142 (PDF p. 17), first sentence; p. 139 (PDF p. 14), section “Maximum predicted effect of relative candidate extremism”", "pdf_page": 17, "printed_page": "142", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P039-T5", "P039-T6"], "text": {"as_printed": "p.~139: ``Maximum effect is calculated as the coefficient for the variable $\\times$ \\hl{the largest value} of that variable.'' Later in the section: ``For all five variables, \\hl{the largest value} represents an election in which the incumbent was less extreme than his challenger, so the relative extremism score employed in these calculations is always negative.$^{15}$''\n\nNote 15, p.~142: ``\\hl{For the eight measures based on ANES/Rosenstone, this was the 1980 election.} For the NOMINATE measure, which did not include the 1980 case, the maximum effect estimate is based on candidates' locations in the 1964 election, in which Johnson was rated less extreme than Goldwater.''", "corrected": "p.~139: ``Maximum effect is calculated as the coefficient for the variable $\\times$ \\hl{the observed score of that variable of greatest absolute magnitude, with its sign retained}.'' The same wording replaces ``the largest value'' in the worked example and in the sentence that carries note 15. The product is the fitted contribution of the variable relative to a score of zero.\n\nNote 15: ``\\hl{For the four estimates based on the two zero-centered ANES/Rosenstone measures, this was the 1980 election. For the four estimates based on the two voter-centered ANES measures, this was the 1972 election. For the two estimates based on} the NOMINATE measure, which did not include the 1980 case, the maximum effect estimate is based on candidates' locations in the 1964 election, in which Johnson was rated less extreme than Goldwater.''", "evidence": "I compared the scores in the replication files in exact arithmetic and recalculated the ten models. For the voter-centered measure based on all voters, the score is $-.5118$ in 1972 and $-.1715$ in 1980. For the refined voter-centered measure, it is $-.6022$ in 1972 and $-.2943$ in 1980. For each of these measures, 1972 has the score of greatest absolute magnitude among the eleven elections from 1972 to 2012; the nearest scores are $-.3403$ (2004) and $-.3603$ (2008). The zero-centered measures have theirs in 1980 ($-.4981$ and $-.6210$) and the NOMINATE measure in 1964 ($-.22$), as the note says. With these scores the recalculation agrees with all ten entries of Table 3 at the printed precision.\n\n\\begin{audittable}{llrrr}{Voter-centered entries of Table 3: the coefficient times the 1972 score gives the printed entry, and the coefficient times the 1980 score does not.}\nVote model & Measure & Printed & With the 1972 score & With the 1980 score \\\\\nMusical chairs & (2) All-voter & 1.33 & 1.33 & 0.45 \\\\\nMusical chairs & (4) Refined & 2.28 & 2.28 & 1.11 \\\\\nBread and peace & (2) All-voter & 0.30 & 0.30 & 0.10 \\\\\nBread and peace & (4) Refined & 0.82 & 0.82 & 0.40 \\\\\n\\end{audittable}\n\nFor the wording on p.~139, the score of the refined zero-centered measure is $-.6210$ in 1980 and $.2738$ in 1984, so that the 1980 score is not the largest in signed order.", "problem": "Table 3 has four measures built from voter perceptions, each used in two vote models, which makes eight estimates. Note 15 says that all eight rest on the 1980 election. This holds for the two zero-centered measures. The two voter-centered measures have their score of greatest absolute magnitude in 1972, and the four entries in rows (2) and (4) of Table 3 are computed with the 1972 scores. Two smaller points concern the same sentence of the note. It speaks of eight measures where there are eight estimates from four measures, and the voter-centered measures use ANES data alone. On p.~139, ``the largest value'' is not the largest score in signed order. The score used is negative for every variable, and it is the score of greatest absolute magnitude. The calculation is correct under that reading, which the worked example and the sentence that carries note 15 make clear.", "what_survives": "Every entry of Table 3 is correct as printed, and so is the statement that the score used is negative for all five variables. The worked example for 1980 and the assignment of the NOMINATE estimates to 1964 stand. No model, estimate or conclusion changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-E004", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the running head and first five lines of page 127. Two red boxes mark the words 'a candidate can always' at the end of the first line and 'capture more votes by moving toward the center,' at the start of the second, in the sentence that ends 'away from the extremes' with a superscript 1.", "caption": "Top of p. 127 (PDF p. 2). The marked words belong to the sentence on the median voter theorem that begins at the foot of p. 126; the superscript refers to note 1.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E005.png"}], "formal_release": null, "formal_statements": [], "headline": "The median voter theorem predicts that candidates who seek to win converge at the voter median; it does not say that a candidate “can always capture more votes by moving toward the center”.", "id": "P039-E005", "location": "First sentence of p. 127, continued from p. 126 (PDF pp. 1–2); note 1, p. 141 (PDF p. 16)", "pdf_page": 2, "printed_page": "127", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "pp.~126--127: ``This conventional wisdom finds theoretical support in the median voter theorem, a classic model of candidate positioning that demonstrates that if voters favor a candidate with views closest to their own, a candidate \\hl{can always capture more votes} by moving toward the center, away from the extremes.$^{1}$''\n\nNote 1, p.~141: ``As such, the median voter theorem predicts that if candidates are simply looking to win, they will converge at the center, \\hl{since positions that are more extreme would garner fewer votes}. Many efforts have been made to reconcile this model with the abundant empirical evidence against this prediction.''", "corrected": "p.~127: ``This conventional wisdom finds theoretical support in \\hl{the standard median voter model, which, under its usual one-dimensional, two-candidate assumptions, predicts convergence at the voter median for candidates seeking to win}.$^{1}$''\n\nNote 1: ``As such, the median voter theorem predicts that if candidates are simply looking to win, they will converge at the center. Many efforts have been made to reconcile this model with the abundant empirical evidence against this prediction.'' The clause ``since positions that are more extreme would garner fewer votes'' is deleted.\n\nIn full, the result is the following. Suppose that there are two candidates and an odd number of voters with ideal points on a line, that each voter chooses the nearer candidate, and that the votes of voters equidistant from the two candidates are divided equally. Then a candidate at the median ideal point receives a strict majority against a candidate at any other position. The pair of positions with both candidates at the median is an equilibrium, since a candidate who moves away from the median loses.", "intended_meaning": "The sentence is an introductory gloss on the standard convergence result, as note 1 explains: in the median voter model, candidates who seek to win converge at the voter median. That statement is correct under the model's usual assumptions. Read literally, the sentence asserts more, namely that every move toward the center gains votes. This does not follow from the assumption that voters choose the nearest candidate. Suppose that two candidates are at positions $x>z$ on a line. Then a voter with ideal point $v$ strictly prefers $x$ exactly when $v>(x+z)/2$, and a move of $x$ to the right raises this cutoff, also when the move is toward the median. Two examples show what can happen. First, let the voters be at $-1$, $0$ and $1$, the candidate at $-1/2$ and the opponent at $3$. The candidate receives all three votes at $-1/2$ and at every position between $-1/2$ and the median $0$, so that no move toward the median gains a vote. Second, let the voters be at $-3$, $-5/4$, $0$, $1$ and $2$, the candidate at $-1$ and the opponent at $-2$. The cutoff is $-3/2$, and the candidate receives four votes. When the candidate moves to the median $0$, the cutoff is $-1$, and the candidate receives three votes. No voter is tied in either example, and the candidate wins at each position. Note that the examples do not contradict the convergence result, which concerns winning and not the number of votes at every position. No estimate or conclusion in the paper depends on the sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-E005", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of six lines of the notes: the end of note 4, notes 5 and 6, and the first line of note 7. In note 5 a red box marks 'all quarters receive an equal weight of 1;'. In note 6 red boxes mark 'decay parameter of lambda = 1' and 'lambda = 0'.", "caption": "Notes 5 and 6, p. 141 (PDF p. 16). The marked phrases are the equal weight of 1 for all quarters in note 5, and the decay parameter of $\\lambda=1$ and the value $\\lambda=0$ in note 6.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E006.png"}], "formal_release": null, "formal_statements": [], "headline": "In note 5 the equal weight of 1 refers to the temporal discount factors, and the election quarter keeps its factor of $1/3$; note 6 means that musical chairs uses the 14th and 15th quarters, with equal weight, and no other quarter.", "id": "P039-E006", "location": "Notes 5 and 6, p. 141 (PDF p. 16), called from p. 128 (PDF p. 3)", "pdf_page": 16, "printed_page": "141", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Note 5: ``This is because $1^{15}=1^{14}=\\ldots=1^{0}=1$, so \\hl{all quarters receive an equal weight of 1}; and $0^{15}=0^{14}=\\ldots=0^{1}=0$, but $0^{0}=1$, so all quarters preceding the election quarter receive a weight of 0.''\n\nNote 6: ``For comparison, musical chairs could be thought of as having a \\hl{decay parameter of} $\\hl{\\lambda=1}$ for the 14th and 15th quarter, and $\\hl{\\lambda=0}$ for all other quarters.''\n\nMain text, p.~128: ``Because the election is held one-third of the way through the 16th quarter of a presidential term, RDI growth in this quarter is down-weighted by $1/3 * \\lambda^{0}$. The 15th quarter of the presidential term is down-weighted by $\\lambda^{1}$, the 14th quarter by $\\lambda^{2}$, and so on for the preceding quarters.''", "corrected": "Note 5: ``This is because $1^{15}=1^{14}=\\ldots=1^{0}=1$, so \\hl{all temporal discount factors equal 1, while the election quarter keeps its factor of one third}; and $0^{15}=0^{14}=\\ldots=0^{1}=0$, but $0^{0}=1$, so all quarters preceding the election quarter receive a weight of 0.''\n\nNote 6: ``For comparison, musical chairs \\hl{gives equal weight to the 14th and 15th quarters and a weight of zero to all other quarters}.''", "intended_meaning": "I read note 5 as explaining the temporal discount factors $\\lambda^{k}$ at the two extreme values of $\\lambda$. At $\\lambda=1$ every discount factor equals 1. The weights are then not all equal, because the main text assigns the election quarter the further factor $1/3$. Its weight is $1/3$, and each earlier quarter in the sum has weight 1. At $\\lambda=0$ only the election quarter has a nonzero weight, as the note says. Note 6 is an analogy. Musical chairs uses income growth in the 14th and 15th quarters only, as p.~128 states, and in the replication script the two quarters have equal weight. This selection cannot be written with a decay parameter. With $\\lambda=1$ for those two quarters and $\\lambda=0$ for the others, the formula of p.~128 still assigns the 16th quarter the weight $(1/3)\\,0^{0}=1/3$. No computation depends on either note, and the fitted models keep the factor $1/3$. The quarters that enter the sum in the fitted models are the subject of \\fid{P039-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-E006", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the reference list: the entry for Bartels and Zaller (2001), the entry for Converse (1964) and the entry for Downs (1957). In the Converse entry a red box marks the italic title 'Ideology and its discontents'.", "caption": "References, p. 142 (PDF p. 17). The marked title is the one printed for the volume edited by Apter.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-E008.png"}], "formal_release": null, "formal_statements": [], "headline": "The volume edited by Apter that contains Converse’s chapter is titled Ideology and Discontent, not “Ideology and its discontents”.", "id": "P039-E008", "location": "References, p. 142 (PDF p. 17), entry “Converse, Philip E. 1964”", "pdf_page": 17, "printed_page": "142", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Converse, Philip E. 1964. The nature of belief systems in mass publics. In \\emph{\\hl{Ideology and its discontents}}, ed. David E. Apter. New York, NY: The Free Press of Glencoe.''", "corrected": "``Converse, Philip E. 1964. The nature of belief systems in mass publics. In \\emph{\\hl{Ideology and discontent}}, ed. David E. Apter. New York, NY: The Free Press of Glencoe.''", "evidence": "I checked the page against a public library catalogue (WorldCat, OCLC 193632). The entry for the print book of 1964, published by the Free Press of Glencoe, has the title ``Ideology and discontent''. The book itself was not consulted.", "problem": "The reference adds ``its'' to the title and makes its last word plural.", "what_survives": "The chapter is identifiable from its author, year, title and editor, and nothing in the paper depends on the entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-E008", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of seven lines of text below Figure 1. Two red boxes mark the words 'slightly more moderate than the' at the end of one line and 'median of all Republican nominees in the entire period.' at the start of the next, in the sentence that begins 'Calling himself a compassionate conservative, he was rated in 2000 as a typical conservative Republican; indeed, he was rated'.", "caption": "p. 131 (PDF p. 6), first paragraph below Figure 1. The marked words compare Bush’s rating in 2000 with the median of all Republican nominees.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "Bush’s rating in 2000 is at the median of the seventeen Republican ratings, and not slightly below it.", "id": "P039-N005", "location": "p. 131 (PDF p. 6), first paragraph below Figure 1", "pdf_page": 6, "printed_page": "131", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Calling himself a compassionate conservative, he was rated in 2000 as a typical conservative Republican; indeed, he was rated \\hl{slightly more moderate than} the median of all Republican nominees in the entire period.''", "corrected": "``Calling himself a compassionate conservative, he was rated in 2000 as a typical conservative Republican; indeed, he was rated \\hl{at} the median of all Republican nominees in the entire period.''", "intended_meaning": "The sentence says that Bush's rating in 2000 was typical of Republican nominees, and this is correct. In the refined series plotted in Figure 1, the Republican rating for 2000 is $1.332$. It is the ninth of the seventeen Republican ratings in increasing order, between McCain in 2008 ($1.321$) and Bush in 1992 ($1.363$), and is therefore their median. If the rating for 2000 is set aside, the median of the other sixteen is $1.342$, and the rating for 2000 is $0.010$ below it. The sentence, however, refers to all Republican nominees. Nothing in the paper depends on the comparison."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-N005", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of seven lines of text. A red box marks the value '-.44' in the sentence 'When the two zero-centered measures are limited to just these eight cases, both measures show a correlation with vote share of -.44.' Three lines above, the text gives the NOMINATE correlation as 'a respectable -.48'.", "caption": "p. 134 (PDF p. 9), second paragraph of the section on correlations. The marked value is given for both zero-centered measures on the eight elections with NOMINATE scores.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "On the eight elections with NOMINATE scores, the refined zero-centered measure has a correlation of $-.45$ with vote share, not $-.44$; the value $-.44$ is correct for the all-voter measure.", "id": "P039-N006", "location": "Section “Correlations between incumbent extremism and vote share”, second paragraph, p. 134 (PDF p. 9)", "pdf_page": 9, "printed_page": "134", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``When the two zero-centered measures are limited to just these eight cases, \\hl{both measures show a correlation} with vote share of $\\hl{-.44}$.''", "corrected": "``When the two zero-centered measures are limited to just these eight cases, \\hl{the all-voter and refined measures show correlations} with vote share of $-.44$ \\hl{and} $\\hl{-.45}$\\hl{, respectively}.''", "evidence": "I recalculated the correlations from the replication files, which give the relative extremism scores and the vote shares used for Table 1. The rounding was settled with exact rational arithmetic: the refined correlation $r$ satisfies $-0.4457949780<r<-0.4457949779$. The reason for the printed value is not established.\n\\begin{audittable}{lrrr}{Correlations with the incumbent party's share of the two-party vote on the eight elections with NOMINATE scores, recalculated from the replication files.}\nMeasure & Recalculated & To two decimals & Printed \\\\\n\\hline\nAll-voter, zero-centered & $-0.44216$ & $-.44$ & $-.44$ \\\\\nRefined, zero-centered & $-0.44579$ & $-.45$ & $-.44$ \\\\\nNOMINATE & $-0.47924$ & $-.48$ & $-.48$ \\\\\n\\end{audittable}", "problem": "The eight cases are the elections in which both candidates have NOMINATE scores: 1960, 1964, 1968, 1972, 1984, 1996, 2004 and 2008. On these elections the all-voter zero-centered measure has a correlation of $-0.4422$ with the incumbent party's share of the two-party vote, and the refined zero-centered measure has a correlation of $-0.4458$. The first rounds to $-.44$ and the second to $-.45$. The printed sentence reports $-.44$ for both. The entry for the refined measure in Table 1 ($-.12$, from $-0.1189$ on seventeen elections) is rounded to the nearest hundredth; applying that convention here gives $-.45$.", "what_survives": "The comparison that the passage draws is unchanged: on the eight cases, both zero-centered measures have correlations close to the $-.48$ of the NOMINATE measure. Table 1 does not print these two correlations, and the regressions use the underlying scores and not the rounded value. No other number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-N006", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the right half of Table 2, with columns B1, B2 and B3 under the heading Bread and Peace. The rows are Delta weighted RDI in Q 1-16, Incumbent fatigue, War fatalities, Relative extremism with -2.23 (2.04) in column B3, Decay parameter .88, .83, .83, Constant, N of cases 16, 16, 16, adjusted r-squared .83, .88, .88, and RMSQ 2.30, 1.90, 1.89. Red boxes mark .88 in column B2, .88 in column B3 and 2.30 in column B1.", "caption": "Table 2, p. 137 (PDF p. 12), Bread & Peace panel, turned upright. The marked entries are the adjusted r-squared of columns B2 and B3 and the RMSQ of column B1.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "In the Bread & Peace panel of Table 2, the adjusted $R^2$ of columns B2 and B3 rounds to .89, not .88, and the RMSQ of column B1 rounds to 2.29, not 2.30.", "id": "P039-N008", "location": "Table 2, p. 137 (PDF p. 12), Bread & Peace panel, rows “adjusted r-squared” (columns B2 and B3) and “RMSQ” (column B1)", "pdf_page": 12, "printed_page": "137", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lrrr}{Table 2, Bread \\& Peace panel, as printed (last three rows).}\n & B1 & B2 & B3 \\\\\nN of cases & 16 & 16 & 16 \\\\\nadjusted r-squared & .83 & \\hl{.88} & \\hl{.88} \\\\\nRMSQ & \\hl{2.30} & 1.90 & 1.89 \\\\\n\\end{audittable}\n\nNote to the table: ``RMSQ = Root mean squared error.''", "corrected": "\\begin{audittable}{lrrr}{Table 2, Bread \\& Peace panel, last three rows as corrected.}\n & B1 & B2 & B3 \\\\\nN of cases & 16 & 16 & 16 \\\\\nadjusted r-squared & .83 & \\hl{.89} & \\hl{.89} \\\\\nRMSQ & \\hl{2.29} & 1.90 & 1.89 \\\\\n\\end{audittable}\n\nIf the convention is stated, RMSQ is the residual standard error, and the decay parameter counts among the fitted parameters.", "evidence": "I recalculated the statistics from the replication data and checked the rounding boundaries in exact arithmetic. The adjusted $R^2$ is $1-[\\mathrm{SSE}/(n-p)]/[\\mathrm{SST}/(n-1)]$ with $n=16$ and $p$ the number of fitted parameters, the decay parameter included: $p=5$ for B2 and $p=6$ for B3. RMSQ is the residual standard error $s=\\sqrt{\\mathrm{SSE}/(n-p)}$, which the entries 1.90 and 1.89 follow. With one parameter fewer the adjusted $R^2$ would round to .89 and .90, and with one more to .87 and .87. The printed entries of B3 also disagree with each other. Since the adjusted $R^2$ equals $1-s^2/(\\mathrm{SST}/15)$ and $\\mathrm{SST}/15=31.513$, an RMSQ between 1.885 and 1.895 implies an adjusted $R^2$ between $.8860$ and $.8872$.", "problem": "Three fit statistics of the panel differ in the last digit from the values that the replication data give. When the three model calls of the authors' script are run unchanged, the residual standard error of B1 is $2.2946$, and the adjusted $R^2$ is $.8851$ for B2 and $.8870$ for B3. These round to 2.29, .89 and .89. The paper does not give its formula for adjusted $R^2$, and the replication files do not compute this row. Truncation in place of rounding would give .88 twice, and rounding the three-decimal value 2.295 again would give 2.30, but the cause is not established.", "what_survives": "No coefficient, standard error or test changes. Figure 4 and Table 3 use the fitted models, not these entries. The other fit statistics of the panel agree with the recalculation at the printed precision, and the fit still improves from B1 to B2 and B3."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-N008", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of text from the worked example. In the sentence 'The largest value of the zero-centered refined variable occurs in 1980, when Reagan's ideological location was a relatively extreme 1.58 and Carter's location was -.34', red boxes mark 1.58 and -.34. The next sentence gives the relative incumbent extremism score of -.62, unmarked.", "caption": "p. 139 (PDF p. 14), worked example under Table 3. The two marked numbers are the 1980 locations of Reagan and Carter.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-N010.png"}], "formal_release": null, "formal_statements": [], "headline": "In the worked example on p. 139, the 1980 locations are 1.57 for Reagan and $-.33$ for Carter, not 1.58 and $-.34$.", "id": "P039-N010", "location": "p. 139 (PDF p. 14), first paragraph under Table 3, worked example", "pdf_page": 14, "printed_page": "139", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``The \\hl{largest value} of the zero-centered refined variable occurs in 1980, when Reagan's ideological location was a relatively extreme \\hl{1.58} and Carter's location was $\\hl{-.34}$, the most moderate of any of the thirty-four candidate positions in the dataset. This works out to a relative incumbent extremism score of $-.62$ for that variable in that election, with the negative direction of the score showing that the challenger, Reagan, was more extreme than the incumbent, Carter.''", "corrected": "``The \\hl{score of greatest absolute magnitude} of the zero-centered refined variable occurs in 1980, when Reagan's ideological location was a relatively extreme \\hl{1.57} and Carter's location was $\\hl{-.33}$, the most moderate of any of the thirty-four candidate positions in the dataset.''\n\nI explain the change to ``the largest value'' in \\fid{P039-E004}.", "evidence": "I recalculated the locations from the replication files. These two values appear in two of the files of scores, and the two scripts of the authors that construct the scores return them from the ANES summary scores in the replication files. The estimation from the ANES respondent data could not be repeated. Note that $1.5742<1.575$ and $-.3322>-.335$, so that neither value is a rounding tie. The relative score is half the difference between the incumbent's and challenger's absolute locations. For the printed pair and for the corrected pair, I have\n\\[\n(.34-1.58)/2=-.62,\\qquad (.33-1.57)/2=-.62 .\n\\]\nThe relative score in the replication files is $-.6210$.", "problem": "In the replication files, the refined scores for 1980 are $1.5742$ for Reagan and $-.3322$ for Carter. At two decimals these are 1.57 and $-.33$, so each printed number differs by .01 from the correctly rounded one. The cause is not established.", "what_survives": "The relative score of $-.62$, the maximum effect of $+1.38$, the $p$-value of .30 and the ten degrees of freedom are unchanged. Carter in 1980 remains the most moderate of the thirty-four locations of the refined measure. The tables and figures use the scores at full precision, not these two printed numbers."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-N010", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the reference list: the entry for Abramowitz (1988), the three-line entry for Achen and Bartels (2016) and the entry for Bartels (1998). In the Achen and Bartels entry red boxes mark 'Pocketbook voting and the limits of democratic accountability.', which runs over two lines, and 'eds.' before the names Christopher Achen and Larry M. Bartels.", "caption": "References, p. 142 (PDF p. 17), entry for Achen and Bartels (2016). The marked terms are the subtitle of the chapter and the abbreviation “eds.”", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-N013.png"}, {"alt": "Crop of five lines of the reference list: the entry for Stokes and Iversen (1962) and the three-line entry for Zaller. In the Zaller entry red boxes mark the year '2003.' and the italic title 'The issue of belief: Essays in the intersection of non-attitudes and attitude change', which runs over two lines.", "caption": "References, p. 142 (PDF p. 17), entry for Zaller (2003). The marked terms are the year and the title of the volume.", "url": "https://pmaronow.github.io/llm-audit/works/P039/exhibits/P039-N013-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Achen and Bartels are the authors, not the editors, of Democracy for Realists, and its chapter on musical chairs has another subtitle; Zaller’s chapter was published in 2004, in a volume with another title.", "id": "P039-N013", "location": "References, p. 142 (PDF p. 17), entries “Achen, Christopher H., and Larry M. Bartels. 2016” and “Zaller, John. 2003”; the citation “Zaller (2003)” on p. 130 (PDF p. 5)", "pdf_page": 17, "printed_page": "142", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Achen, Christopher H., and Larry M. Bartels. 2016. Musical chairs: \\hl{Pocketbook voting and the limits of democratic accountability}. In \\emph{Democracy for realists}, \\hl{eds.} Christopher Achen and Larry M. Bartels. Princeton, NJ: Princeton University Press.''\n\n``Zaller, John. \\hl{2003}. Floating voters in U.S. presidential elections, 1948--2000. In \\emph{\\hl{The issue of belief: Essays in the intersection of non-attitudes and attitude change}}, eds. \\hl{Paul Sniderman, and Willem E. Saris}. Princeton, NJ: Princeton University Press.''", "corrected": "``Achen, Christopher H., and Larry M. Bartels. 2016. Musical chairs: \\hl{Economic voting and the specious present}. \\hl{Chap. 6 in} \\emph{Democracy for realists\\hl{: Why elections do not produce responsive government}}. Princeton, NJ: Princeton University Press.''\n\n``Zaller, John. \\hl{2004}. Floating voters in U.S. presidential elections, 1948--2000. In \\emph{\\hl{Studies in public opinion: Attitudes, nonattitudes, measurement error, and change}}, eds. \\hl{Willem E. Saris and Paul M. Sniderman}. Princeton, NJ: Princeton University Press.''\n\nThe citation ``Zaller (2003)'' on p.~130 becomes ``Zaller (\\hl{2004})''.", "evidence": "I checked the page against public bibliographic sources. The library catalogue entry for the 2016 book (WorldCat, OCLC 946459788) lists Achen and Bartels as its authors. The publisher's chapter listings state the title of the chapter on musical chairs, and the volume, year and editors of Zaller's chapter. A curriculum vitae of Bartels lists a conference paper with the printed subtitle, dated September 2004. A 2003 article by Zaller cites the chapter as in press in a volume titled ``The issue of belief''. The books themselves were not consulted. It is not established that no manuscript of 2003 existed, or which version was used for the paper.", "problem": "Read as references to the published books, both entries give wrong details. \\emph{Democracy for Realists: Why Elections Do Not Produce Responsive Government} (2016) is a book by Achen and Bartels, not a volume that they edited. Its chapter 6 is titled ``Musical Chairs: Economic Voting and the Specious Present''; the subtitle printed in the reference is that of a conference paper of September 2004. Zaller's chapter was published in 2004, in \\emph{Studies in Public Opinion: Attitudes, Nonattitudes, Measurement Error, and Change}, edited by Willem E. Saris and Paul M. Sniderman. The title printed in the reference is one under which the chapter was cited while the volume was in press.", "what_survives": "Both works are identifiable from author and chapter title, and no result depends on either entry. If an earlier manuscript was meant, the entry should name it."}, "url": "https://pmaronow.github.io/llm-audit/works/P039/#P039-N013", "work": {"arxiv_id": null, "doi": "10.1177/0002716216660571", "title": "Ideologically Extreme Candidates in U.S. Presidential Elections, 1948-2012", "url": "https://pmaronow.github.io/llm-audit/works/P039/"}, "work_id": "P039"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the fifteen lines of the paragraph. The text reads: In our sample of 15 customers, six spontaneously detected a scent. Once prompted to consciously attend to scent, all but one of the respondents were able to discern the presence of an ambient scent. Six respondents spontaneously identified that ambient scent as chocolate, and another five reported recognizing the scent as chocolate once the enumerator identified the scent as such. The total proportion of our sample reporting a detectable ambient scent was .93 (95 \\% CI: .81, 1.06). One limitation of the small sample size in our manipulation check is that the confidence intervals for our estimates are relatively wide. However, since the lower bound of our estimate indicates that at least 80 \\% of patrons would discern the presence of an ambient scent, we remain confident that the scent was sufficiently detectable. One red outline surrounds (95 \\% CI: .81, 1.06) and a second surrounds the words at least 80 \\% of patrons.", "caption": "Results and discussion, third paragraph, p. 3 (PDF p. 3), right column. The outlines mark the printed interval, “(95 % CI: .81, 1.06)”, and the inference drawn from its lower limit, “at least 80 % of patrons”.", "url": "https://pmaronow.github.io/llm-audit/works/P040/exhibits/P040-E001.png"}, {"alt": "Chart with one horizontal axis for the detection probability p, with labelled ticks from .60 to 1.10, and three horizontal bars. The title reads: 14 of 15 customers detected a scent: the exact 95\\% lower limits, .68 and .72, are below the printed limit .81 and below the minimum of .80 that the paper states. The top bar, in orange, is labelled As printed: 95 \\% CI (.81, 1.06); its endpoints are approximately .807 and 1.060, labelled .81 and 1.06, and it extends into a gray region to the right of 1.00 that is labelled above 1: not a probability. The middle bar, in blue, is labelled Exact 95\\% interval, equal tails; its endpoints are labelled .681 and .998. The bottom bar, in blue, is labelled Exact one-sided 95\\% lower limit; its lower endpoint is labelled .721 and it ends at 1.00. A dashed vertical line at .80 is labelled stated minimum: .80. The orange bar lies entirely to the right of this line, and both blue bars cross it. An open circle on each bar marks the observed proportion 14/15, labelled .933.", "caption": "The nominal interval as printed and the exact binomial limits for 14 detections among 15 respondents, on one axis for the detection probability $p$. The printed interval $(.81, 1.06)$ is drawn at the endpoints, approximately $.807$ and $1.060$, of the nominal normal-approximation interval. Its lower-endpoint interpretation is conventional, while the exact finite-sample calculation exposes poor calibration here. The exact equal-tailed 95% interval rounds to $[.681, .998]$, and the exact one-sided 95% lower limit rounds to $.721$. Both exact lower limits lie to the left of the dashed line at $.80$. The open circles mark the observed proportion $14/15$.", "url": "https://pmaronow.github.io/llm-audit/works/P040/exhibits/P040-E001-intervals.png"}], "formal_release": null, "formal_statements": [{"id": "P040-T1", "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-T1"}, {"id": "P040-T2", "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-T2"}], "headline": "The printed interval agrees with a nominal Wald interval with a conventional lower-endpoint interpretation, but its normal approximation is poorly calibrated with 15 responses: exact coverage at a detection probability of $.80$ is about 81.5%.", "id": "P040-E001", "location": "Results and discussion, third paragraph, p. 3 (PDF p. 3), right column: the interval “(95 % CI: .81, 1.06)” and the last sentence of the paragraph", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P040-T1", "P040-T2"], "text": {"as_printed": "In our sample of 15 customers, six spontaneously detected a scent. Once prompted to consciously attend to scent, all but one of the respondents were able to discern the presence of an ambient scent. Six respondents spontaneously identified that ambient scent as chocolate, and another five reported recognizing the scent as chocolate once the enumerator identified the scent as such. The total proportion of our sample reporting a detectable ambient scent was .93 (95~\\% CI: \\hl{.81, 1.06}). One limitation of the small sample size in our manipulation check is that the confidence intervals for our estimates are relatively wide. However, since the lower bound of our estimate indicates that \\hl{at least 80~\\% of patrons} would discern the presence of an ambient scent, we remain confident that the scent was sufficiently detectable.", "corrected": "Fourteen of the fifteen surveyed customers reported detecting an ambient scent, either spontaneously or after prompting. Under an independent common-probability binomial sampling model, the exact equal-tailed 95\\% confidence interval is approximately \\hl{$[.680515, .998314]$}. The observed detection rate was high, but the exact-binomial lower endpoint is \\hl{about $.68$ rather than above $.80$}, leaving substantial uncertainty about the detection rate; generalization beyond the surveyed customers requires appropriate sampling assumptions.\n\nThe exact-binomial alternative can be specified as follows. Let $X$ be the number of detections among $n = 15$ respondents, and suppose that the responses are independent with a common detection probability $p \\in [0, 1]$, so that $X$ is binomial with parameters $15$ and $p$. For the observed value $X = 14$, the exact equal-tailed 95\\% confidence interval for $p$ is $[L, U]$, where $L$ and $U$ are the unique solutions in $(0, 1)$ of\n\\[\nL^{14} (15 - 14 L) = .025, \\qquad U^{15} = .975 .\n\\]\nThen $L \\approx .680515$ and $U \\approx .998314$, and at the precision of the paper the interval is $[.68, 1.00]$. The exact one-sided 95\\% lower confidence limit is the unique solution $L_1 \\approx .720604$ in $(0, 1)$ of $L_1^{14} (15 - 14 L_1) = .05$. Both lower limits are less than $.80$. These limits concern the detection probability under the stated model. A statement about patrons in general requires in addition that the 15 respondents represent them, and the paper does not describe how the respondents were selected.", "evidence": "I checked the Wald derivation, calculated finite-sample coverage exactly and recalculated the counts from the replication files. The manipulation-check file has 15 respondents, of whom 14 detected a scent. For the calculations, suppose that the 15 responses are independent with a common detection probability $p$, so that the number $X$ of detections is binomial with parameters $15$ and $p$. I assume this sampling model for the calculations; the paper does not specify its interval method.\n\nFor fixed $0<p<1$, the Bernoulli central limit theorem and consistency of $\\hat p$ give\n\\[\n\\frac{\\sqrt n(\\hat p-p)}{\\sqrt{\\hat p(1-\\hat p)}}\\;\\xrightarrow{d}\\;N(0,1).\n\\]\nConsequently the nominal two-sided Wald interval has coverage tending to $.95$, and its lower endpoint is an approximate one-sided 97.5\\% lower confidence limit. This supplies the conventional approximate reading; it gives no small-sample accuracy bound at $n=15$.\n\nNow fix $p=4/5$ and $n=15$. Then\n\\[\n\\mathbb{P}(X = 14) = 15 \\left(\\frac{4}{5}\\right)^{14} \\frac{1}{5} = \\frac{805306368}{6103515625} \\approx .1319 .\n\\]\nThe printed interval at $X = 14$ excludes $.80$; if $.81$ is a rounded value, the limit is still at least $.805$. Consider any rule that assigns an interval to each count and assigns the printed interval to the count 14. This rule fails to contain $p = .80$ whenever $X = 14$, which implies that it contains $.80$ with probability at most $1 - .1319 = .8681$. This probability is less than $.95$, and the bound does not depend on the method that produced the interval. It refutes an exact finite-sample 95\\% guarantee for a deterministic count-only rule with that endpoint, not the usual asymptotic interpretation of a Wald endpoint. For the normal-approximation rule, the interval contains $.80$ exactly when $9 \\le X \\le 13$, so\n\\[\n\\mathbb{P}(9 \\le X \\le 13) = \\frac{24866193408}{30517578125} \\approx .8148 .\n\\]\n\\begin{audittable}{lll}{Exact probabilities for 15 independent responses with detection probability $.80$. The rule in the last two rows is the nominal normal-approximation interval $\\hat p \\pm 1.96 \\sqrt{\\hat p (1 - \\hat p)/15}$.}\nEvent & Probability & Decimal \\\\\n$X = 14$ & $805306368/6103515625$ & $.1319$ \\\\\n$X \\ge 14$ & $5100273664/30517578125$ & $.1671$ \\\\\nThe interval of the rule contains $.80$ ($9 \\le X \\le 13$) & $24866193408/30517578125$ & $.8148$ \\\\\nThe lower limit of the rule is at most $.80$ ($X \\le 13$) & $25417304461/30517578125$ & $.8329$ \\\\\n\\end{audittable}\n\nFor the exact limits, a direct computation gives\n\\[\n\\mathbb{P}_p(X \\ge 14) = 15 p^{14} (1 - p) + p^{15} = p^{14} (15 - 14 p) .\n\\]\nThis function of $p$ is continuous and strictly increasing on $[0, 1]$, since its derivative is $210 p^{13} (1 - p)$. The lower limit $L$ of the exact equal-tailed 95\\% interval is the solution of $L^{14} (15 - 14 L) = .025$. The upper limit $U$ is the solution of $\\mathbb{P}_U(X \\le 14) = 1 - U^{15} = .025$. The exact one-sided 95\\% lower limit $L_1$ is the solution of $L_1^{14} (15 - 14 L_1) = .05$. Their solutions are\n\\[\nL \\approx .680515, \\qquad U \\approx .998314, \\qquad L_1 \\approx .720604 .\n\\]\nFor every $p$, the lower limit of the exact interval exceeds $p$ with probability at most $.025$, and its upper limit is less than $p$ with probability at most $.025$. This implies that the interval contains $p$ with probability at least $.95$. At $p=.80$, $\\mathbb{P}(X \\ge 14) \\approx .1671$, which exceeds $.05$ and $.025$.", "problem": "The count is correct: 14 of the 15 respondents detected a scent, spontaneously or when prompted, and $14/15\\approx .93$. The paper does not name its interval method. The printed limits agree with the normal-approximation, or Wald, interval $\\hat p\\pm z_{.975}\\sqrt{\\hat p(1-\\hat p)/n}$, where $z_{.975}\\approx1.96$ is the standard normal $.975$ quantile, $\\hat p=14/15$ and $n=15$; this gives approximately $(.807,1.060)$. Reading its lower endpoint as approximate confidence-based evidence for a population detection rate above $.80$ is conventional. The passage does not promise certainty or exact finite-sample coverage, and this interpretation is not the defect.\n\nThe defect is the approximation's accuracy in this small sample. Under independent common-probability Bernoulli responses with $n=15$ and $p=.80$, the reconstructed nominal 95\\% interval contains $p$ with probability only $.8148$, and its lower limit is at most $p$ with probability only $.8329$. By contrast, with fixed $0<p<1$ and increasing $n$, those probabilities tend to $.95$ and $.975$. The finite-sample failure does not refute that asymptotic interpretation.\n\nThe nominal Wald test does reject $.80$ here, with a two-sided normal $P$ value of about $.03843$. The exact one-sided binomial tail probability is about $.16713$. An exact equal-tailed 95\\% interval rounds to $[.68,1.00]$, and an exact one-sided 95\\% lower limit rounds to $.72$. These alternatives show the weaker assurance available without the unreliable normal approximation. An upper limit above one is not by itself the error: clipping the printed interval at one changes neither its lower limit nor its coverage for a permissible probability.", "what_survives": "The observed count of 14 detections among 15, the other manipulation-check counts, the reconstructed Wald arithmetic and the conventional lower-confidence-endpoint interpretation survive. The sales regressions, Table~1 and Fig.~1 do not use the survey responses or the interval, so this repair leaves their numerical calculations unchanged; it does not independently validate that analysis.\n\nThe surveyed sample has a high observed detection rate. Under the exact-binomial alternative its 95\\% interval extends down to about $.68$, and its one-sided 95\\% lower limit is about $.72$. The nominal Wald inference above $.80$ therefore remains a conventional but poorly calibrated approximation here. Nothing establishes that the true rate is below $.80$ or that the scent was too weak."}, "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-E001", "work": {"arxiv_id": null, "doi": "10.1186/s40064-016-2303-5", "title": "Chocolate Scents and Product Sales: A Randomized Controlled Trial in a Canadian Bookstore and Cafe", "url": "https://pmaronow.github.io/llm-audit/works/P040/"}, "work_id": "P040"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of the left column of page 3. The text reads: The trial used a between-subjects design in which the 31 experimental days were assigned one of two conditions: chocolate scent dispersion (treatment) and no scent (control). Condition was determined by Bernoulli (coin flip) random assignment with probability .5 independently across days. The crop ends with the fragment Chocolate scent was dispersed. A red outline surrounds the words 31 experimental days.", "caption": "Methods, first sentence on p. 3 (PDF p. 3), left column. The outline marks “31 experimental days”.", "url": "https://pmaronow.github.io/llm-audit/works/P040/exhibits/P040-E002.png"}, {"alt": "Crop of Fig. 1, titled Sales by Type on Treatment and Control Days. The horizontal axis is Day of Experiment, with ticks from 0 to 30, and the vertical axis is Sales in Canadian Dollars, from 0 to 1000. Each observed day has four markers: a circle for total sales, an X for cafe sales, a diamond for book sales and a triangle for bulk sales. Sixteen gray vertical bars mark the treatment days. A tall red outline surrounds the column of the plot at day 5 and the tick label 5 below it; the column is empty. In the caption below the figure, a second red outline surrounds the words 31 days of the experiment. The caption reads: Fig. 1 Sales by type on treatment and control days. All sales data associated with the randomized controlled trial. The x-axis shows each of the 31 days of the experiment, with gray vertical bars indicating days on which the chocolate scent was administered. The y-axis shows sales in Canadian dollars (CAD). The markers indicate sales per day: circles denote total sales, Xs denote cafe sales, diamonds denote book sales, and triangles denote bulk coffee bean/loose tea/spice sales.", "caption": "Fig. 1 with its caption, p. 4 (PDF p. 4). The outline in the caption marks “31 days of the experiment”. The outline in the plot marks the position of day 5, where no markers are plotted; every other day from 1 to 31 has four markers.", "url": "https://pmaronow.github.io/llm-audit/works/P040/exhibits/P040-E002-figure.png"}], "formal_release": null, "formal_statements": [], "headline": "The reported analyses and Fig. 1 cover 30 daily observations, for August 1–4 and August 6–31: the trial spanned 31 calendar days, and the replication data have no entry for August 5.", "id": "P040-E002", "location": "Methods, first sentence on p. 3 (PDF p. 3), left column; Fig. 1 and its caption, p. 4 (PDF p. 4)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~3, Methods.} The trial used a between-subjects design in which the \\hl{31 experimental days} were assigned one of two conditions: chocolate scent dispersion (treatment) and no scent (control). Condition was determined by Bernoulli (coin flip) random assignment with probability .5 independently across days.\n\n\\emph{p.~4.} We present the complete raw results of the trial in Fig.~1.\n\n\\emph{p.~4, caption of Fig.~1.} All sales data associated with the randomized controlled trial. The x-axis shows \\hl{each of the 31 days of the experiment}, with gray vertical bars indicating days on which the chocolate scent was administered.", "corrected": "\\emph{p.~3, Methods, to be added.} \\hl{The trial period spanned August 1--31. The reported analyses use 30 daily observations (August 1--4 and August 6--31), comprising 16 treatment and 14 control days. August 5 is absent from the deposited sales data.}\n\n\\emph{p.~4, caption of Fig.~1.} All sales data associated with the randomized controlled trial. \\hl{The x-axis spans the 31-day trial period. Markers show the 30 observed days; no observation is shown for day 5. Gray vertical bars indicate observed days assigned to the chocolate-scent treatment.}\n\nThe number of days that were assigned a condition and the reason for the absence of August 5 should be stated from the trial records, if these are available. The evidence does not support the replacement of ``31 experimental days were assigned'' by ``30 experimental days were assigned''.", "evidence": "I checked the replication files. The daily sales file has 30 distinct dates, each with complete values of the four outcomes. Computing the eight regressions of the analysis script from these 30 rows in exact arithmetic, I obtain the 16 estimates and the 16 standard errors of Table~1 to the printed decimal. The files do not show whether August 5 was assigned a condition, whether sales were recorded on that day, or why the day is absent. It is not established that only 30 days were randomized, and no change in an estimate and no selection effect is established.", "problem": "The paper states that there were 31 experimental days, and it presents Fig.~1 as the complete raw results, on an axis that shows each of the 31 days. In a trial randomized by day, this suggests that the daily data and the analysis cover 31 observations. The replication data hold 30 daily observations, for August 1--4 and August 6--31, of which 16 are treatment days and 14 are control days. August 5 is absent. The analysis script uses the 30 rows without removing or adding any, and Fig.~1 has no markers at day 5. The paper does not distinguish the 31-day span from the 30 observed days, and it does not explain why August 5 is absent.", "what_survives": "Table~1, the 30 plotted days of Fig.~1 and the conclusions on sales are unchanged. The descriptions of a 31-day trial in the abstract and on p.~2 hold for the calendar span, August 1--31, and the statement on p.~3 that the study has three times as many observations as the original 10-day study holds for 30 days. The correction clarifies the report."}, "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-E002", "work": {"arxiv_id": null, "doi": "10.1186/s40064-016-2303-5", "title": "Chocolate Scents and Product Sales: A Randomized Controlled Trial in a Canadian Bookstore and Cafe", "url": "https://pmaronow.github.io/llm-audit/works/P040/"}, "work_id": "P040"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of eight lines of the left column of page 4. The text reads: between treatment and control. To improve efficiency, the second strategy uses ordinary least squares adjusting for a mean-centered linear time trend as well as mean-centered fixed effects for the day of week, following Lin (2013). In all cases, robust standard errors are used to estimate standard errors, with p values computed under a normal approximation. We present the complete raw results of the trial in. A red outline surrounds the words a normal approximation.", "caption": "p. 4 (PDF p. 4), left column, end of the first paragraph. The outline marks “a normal approximation”.", "url": "https://pmaronow.github.io/llm-audit/works/P040/exhibits/P040-E003.png"}, {"alt": "Crop of eight lines of the left column of page 5. The text reads: ment effect on café sales was slightly negative, though not statistically significant (average estimated decrease of 8.9CAD from a control average of 336.5CAD, two-tailed p = .59) and the direction of treatment on bulk coffee bean/loose tea/spice sales was slightly positive, though not statistically significant (average estimated increase of 23.2CAD from a control average of 71.2CAD, two-tailed p = .29). One red outline surrounds p = .59) and a second surrounds p = .29) and the final period.", "caption": "p. 5 (PDF p. 5), left column, end of the last paragraph. The outlines mark the two printed $p$-values, $.59$ for café sales and $.29$ for bulk sales.", "url": "https://pmaronow.github.io/llm-audit/works/P040/exhibits/P040-E003-pvalues.png"}], "formal_release": null, "formal_statements": [{"id": "P040-T4", "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-T4"}, {"id": "P040-T5", "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-T5"}], "headline": "The printed $p$-values are computed from a Student-$t$ reference distribution with the residual degrees of freedom of each regression, 28 for the unadjusted regressions and 21 for the adjusted regressions, and not under a normal approximation.", "id": "P040-E003", "location": "p. 4 (PDF p. 4), left column, first paragraph, last sentence; the two $p$-values that it governs, p. 5 (PDF p. 5), left column, last paragraph", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P040-T4", "P040-T5"], "text": {"as_printed": "\\emph{p.~4.} In all cases, robust standard errors are used to estimate standard errors, with $p$ values computed under \\hl{a normal approximation}.\n\n\\emph{p.~5.} The direction of the treatment effect on café sales was slightly negative, though not statistically significant (average estimated decrease of 8.9CAD from a control average of 336.5CAD, two-tailed $p = .59$) and the direction of treatment on bulk coffee bean/loose tea/spice sales was slightly positive, though not statistically significant (average estimated increase of 23.2CAD from a control average of 71.2CAD, two-tailed $p = .29$).", "corrected": "\\emph{p.~4.} In all cases, robust standard errors are used to estimate standard errors, with $p$ values computed \\hl{using a Student-$t$ reference distribution with the model's residual degrees of freedom (28 for the unadjusted regressions and 21 for the adjusted regressions)}.\n\nIn full, the corrected statement is the following. For an estimate with ratio $t$ to its robust standard error, the two-tailed $p$-value is $2\\,\\mathbb{P}(T_\\nu \\ge |t|)$, where $T_\\nu$ has the Student-$t$ distribution with $\\nu = 30 - k$ degrees of freedom, and $k$ is the rank of the regression on the 30 daily observations. Without covariates, $k = 2$ and $\\nu = 28$. With the linear time trend and the day-of-week effects, $k = 9$ and $\\nu = 21$. The Student-$t$ distribution serves as a reference distribution: the corrected sentence does not state that the ratio has this distribution exactly under the null hypothesis. With this sentence, the two $p$-values on p.~5 stand as printed. The alternative is to keep the sentence on p.~4, compute the tests with the normal reference, and print $p = .58$ and $p = .28$ on p.~5.", "evidence": "I checked a recalculation from the replication files. With its statistical expressions unchanged, the analysis script reproduces all 32 estimates and standard errors of Table~1 to the printed decimal. Its test of coefficients uses the residual degrees of freedom of the fitted regression. The robust standard errors of the script are the HC1 standard errors. The two adjusted effects were also computed in exact arithmetic, with certified bounds for the two probabilities.\n\\begin{audittable}{lrrrrrr}{The two adjusted treatment effects quoted on p.~5: the estimate and its robust standard error in CAD, their ratio $t$, and the two-tailed $p$-value under the Student-$t$ distribution with 21 degrees of freedom and under the standard normal distribution.}\nOutcome & Estimate & Standard error & $t$ & $p$, Student-$t$ & $p$, normal & Printed \\\\\nCafé sales & $-8.891$ & $16.151$ & $-0.5505$ & $.5878$ & $.5820$ & $.59$ \\\\\nBulk sales & $23.179$ & $21.247$ & $1.0909$ & $.2877$ & $.2753$ & $.29$ \\\\\n\\end{audittable}\nThe Student-$t$ values round to the printed $.59$ and $.29$, and the normal values round to $.58$ and $.28$. The normal value for bulk sales must be computed from the unrounded estimate and standard error. With the rounded entries $23.2$ and $21.2$ of Table~1, I obtain a value near $.274$.", "problem": "The Methods state that the $p$-values are computed under a normal approximation. For an estimate with ratio $t$ to its robust standard error, this is the value $2 \\Phi(-|t|)$, where $\\Phi$ is the standard normal distribution function. The analysis script instead computes each $p$-value from the Student-$t$ distribution with the residual degrees of freedom of the regression, which are 28 for the unadjusted regressions and 21 for the adjusted regressions. The two values printed on p.~5 are these Student-$t$ values. Under the normal approximation they are $.58$ and $.28$.", "what_survives": "The estimates and standard errors of Table~1 and the observations of Fig.~1 are unchanged. No statement of significance changes. Over the eight tests of a treatment effect, the smallest $p$-value is $.225$ with the Student-$t$ reference and $.215$ with the normal reference. The statement on p.~4 that no estimate is significant at the two-tailed $.10$ level holds in both cases. The correction changes the description of the method."}, "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-E003", "work": {"arxiv_id": null, "doi": "10.1186/s40064-016-2303-5", "title": "Chocolate Scents and Product Sales: A Randomized Controlled Trial in a Canadian Bookstore and Cafe", "url": "https://pmaronow.github.io/llm-audit/works/P040/"}, "work_id": "P040"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of eight lines of the reference list. The first three lines are the entry: Dooley R (2013) Keep your customers shopping... with chocolate! Forbes Available: http://www.forbes.com/sites/rogerdooley/2013/07/22/chocolate-scent/. The next three lines are the entry: Doucé L, Poels K, Janssens W, DeBacker C (2013) Smelling the books: the effect of chocolate scent in purchase-related behavior in a bookstore. J Environ Psychol 36:65 to 69. The last two lines are the entry: Herrington JD (1996) Effects of music in service environments: a field study. J Serv Mark 10:26 to 41. One red outline surrounds the name DeBacker, and a second surrounds the word in that follows chocolate scent.", "caption": "References, p. 6 (PDF p. 6), left column: the entry for Doucé et al. (2013), between the entries for Dooley (2013) and Herrington (1996). The outlines mark “DeBacker” and the word “in” of the title.", "url": "https://pmaronow.github.io/llm-audit/works/P040/exhibits/P040-N008.png"}], "formal_release": null, "formal_statements": [], "headline": "In the reference to Doucé et al. (2013), the title should read “the effect of chocolate scent on purchase-related behavior in a bookstore”, and the surname of the fourth author is De Backer.", "id": "P040-N008", "location": "References, p. 6 (PDF p. 6), left column, the entry for Doucé et al. (2013)", "pdf_page": 6, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Doucé L, Poels K, Janssens W, \\hl{DeBacker} C (2013) Smelling the books: the effect of chocolate scent \\hl{in} purchase-related behavior in a bookstore. J Environ Psychol 36:65--69", "corrected": "Doucé L, Poels K, Janssens W, \\hl{De Backer} C (2013) Smelling the books: the effect of chocolate scent \\hl{on} purchase-related behavior in a bookstore. J Environ Psychol 36:65--69", "evidence": "I checked the page against the bibliographic entry for the cited article in the repository of Hasselt University, which has the title with ``on'' and the fourth author as ``De Backer, Charlotte''. I did not read the article itself. It is not settled whether the entry for Herrington (1996), and footnote 1 on p.~1, omit a second author. The publisher lists J.~Duncan Herrington alone, a library catalogue lists Herrington and Capella, and the printed first page of that article was not available.", "problem": "The entry prints ``in'' where the title of the cited article has ``on'', and it prints the surname De Backer as one word.", "what_survives": "The cited work can still be identified from the entry, and nothing in the paper depends on these details."}, "url": "https://pmaronow.github.io/llm-audit/works/P040/#P040-N008", "work": {"arxiv_id": null, "doi": "10.1186/s40064-016-2303-5", "title": "Chocolate Scents and Product Sales: A Randomized Controlled Trial in a Canadian Bookstore and Cafe", "url": "https://pmaronow.github.io/llm-audit/works/P040/"}, "work_id": "P040"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of eight lines of the abstract. Red boxes mark '36\\%' at the start of the line 'more likely to be removed by Wikipedia editors than positive facts within 12 hours and' and '29\\%' at the start of the next line, 'more likely within 3 days'.", "caption": "Abstract, p. 1 (PDF p. 1). The marked figures are the 36% and 29% said to be the increases in the likelihood of removal within 12 hours and within 3 days.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E001.png"}, {"alt": "Crop of the last three lines of page 4. A red box marks '64\\%' in 'uncited (but true) facts are 64\\% more likely to be', where the page ends.", "caption": "Results, “Effects of Citations on Survival”, last lines of p. 4 (PDF p. 4). The marked figure is the 64%; the sentence continues on p. 5 with “removed than cited facts within four hours (hazard ratio = 2.33, $p<0.001$)”.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E001-b.png"}, {"alt": "Listing of lines 30 to 43 of the Stata program. Red boxes mark line 33, 'disp ((abs(75 - 52)/((75 + 52)/2)) * 100)', line 38, 'disp ((abs(92 - 69)/((92 + 69)/2)) * 100)', and line 43, 'disp ((abs(62 - 32)/((62 + 32)/2)) * 100)'. The comment lines above them read 'Negative facts are 36\\% more likely to be removed', 'and 29\\% more likely within 3 days' and 'Uncited facts are 43\\% more likely to be removed'.", "caption": "Replication program Kalla_Aronow_PLOS_2015.do, lines 30–43, typeset from the file. The marked lines compute the three percentages: each divides the difference of the two counts by their mean.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E001-c.png"}, {"alt": "Dot plot with three rows and a horizontal axis labeled from 0 to 100 per cent. Negative against positive facts within 12 hours (75 against 52 of 150 removed): circle labeled 36 per cent, relative increase 44.2 per cent. Negative against positive facts within 3 days (92 against 69 of 150): circle labeled 29 per cent, relative increase 33.3 per cent. Uncited against cited facts within 4 hours (62 against 32 of 100): circle labeled 64 per cent, relative increase 93.8 per cent. In each row the square for the relative increase lies to the right of the open circle for the symmetric difference.", "caption": "The three comparisons, from the recount of the replication data. Open circles: the difference of the two shares removed divided by their mean, labeled with the rounded printed figures. Squares: the increase relative to the share removed in the reference group.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E001-percentages.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T3", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T3"}, {"id": "P041-T4", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T4"}, {"id": "P041-T12", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T12"}], "headline": "The printed 36%, 29% and 64% are differences divided by the mean of the two shares of facts removed; relative to the positive facts and to the cited facts, the increases in the share removed are 44%, 33% and 94%.", "id": "P041-E001", "location": "Abstract, p. 1 (PDF p. 1); Results, “Effects of Citations on Survival”, pp. 4–5 (PDF pp. 4–5); replication program Kalla_Aronow_PLOS_2015.do, lines 33, 38 and 43", "pdf_page": 1, "printed_page": "1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P041-T12", "P041-T3", "P041-T4"], "text": {"as_printed": "Abstract, p.~1: Negative facts are \\hl{36\\%} more likely to be removed by Wikipedia editors than positive facts within 12 hours and \\hl{29\\%} more likely within 3 days.\n\nResults, pp.~4--5: Consistent with Wikipedia's policy, uncited (but true) facts are \\hl{64\\%} more likely to be removed than cited facts within four hours (hazard ratio = 2.33, $p<0.001$).\n\nReplication program, lines 33, 38 and 43:\n\n\\texttt{disp ((abs(75 - 52)/\\hl{((75 + 52)/2)}) * 100)}\n\n\\texttt{disp ((abs(92 - 69)/\\hl{((92 + 69)/2)}) * 100)}\n\n\\texttt{disp ((abs(62 - 32)/\\hl{((62 + 32)/2)}) * 100)}", "corrected": "Abstract, p.~1: Negative facts are \\hl{44\\%} more likely to be removed by Wikipedia editors than positive facts within 12 hours and \\hl{33\\%} more likely within 3 days.\n\nResults, pp.~4--5: Consistent with Wikipedia's policy, uncited (but true) facts are \\hl{94\\%} more likely to be removed than cited facts within four hours (hazard ratio = 2.33, $p<0.001$).\n\nHere the increase is computed relative to the reference group. Let $p_1$ be the share of facts removed by the stated time in the comparison group (negative facts, or uncited facts) and $p_0>0$ the share in the reference group (positive facts, or cited facts). The increase is\n\\[\n100\\Bigl(\\frac{p_1}{p_0}-1\\Bigr)=100\\,\\frac{p_1-p_0}{\\hl{p_0}} ,\n\\]\nwhich gives 44.23\\%, 33.33\\% and 93.75\\% with the stored survival times. In lines 33, 38 and 43 of the replication program, the divisor is the count for the reference group: $100\\,(75-52)/\\hl{52}$, $100\\,(92-69)/\\hl{69}$ and $100\\,(62-32)/\\hl{32}$. An alternative is to keep 36\\%, 29\\% and 64\\% and to describe them as differences relative to the mean of the two shares, which reports a different measure.", "evidence": "The evidence is a recount from the replication data and exact arithmetic. I count a fact as removed by a horizon when its removal is recorded and its stored survival time is at most the horizon, as the program does. For negative against positive facts, Studies 1, 2 and 4 are pooled, with 150 facts in each group. For uncited against cited facts, Studies 1 and 4 are pooled, with 100 facts in each group. The counts agree with the numbers written into the program. Since the groups have equal sizes, I have $p_A/p_B=a/b$, and a direct computation gives\n\\[\n\\begin{aligned}\n100\\Bigl(\\frac{75}{52}-1\\Bigr)&=\\frac{575}{13}=44.23\\ldots,\\\\\n100\\Bigl(\\frac{92}{69}-1\\Bigr)&=\\frac{100}{3}=33.33\\ldots,\\\\\n100\\Bigl(\\frac{62}{32}-1\\Bigr)&=\\frac{375}{4}=93.75 .\n\\end{aligned}\n\\]\nEvaluating the three expressions in the program, I find\n\\[\n\\begin{aligned}\n\\frac{100\\cdot 23}{127/2}&=\\frac{4600}{127}=36.22\\ldots,\\\\\n\\frac{100\\cdot 23}{161/2}&=\\frac{200}{7}=28.57\\ldots,\\\\\n\\frac{100\\cdot 30}{94/2}&=\\frac{3000}{47}=63.83\\ldots .\n\\end{aligned}\n\\]\nEach printed integer is a rounded value from the second display, and none is a rounded value from the first. To compare the two measures in general, let $p_A>p_B>0$ and $d=p_A-p_B$, and write $R=100\\,d/p_B$ and $M=200\\,d/(p_A+p_B)$. Then\n\\[\nR-M=\\frac{100\\,d^{2}}{p_B\\,(p_A+p_B)}>0 .\n\\]\nThis implies that the symmetric difference is the smaller of the two measures. The Harvard and the Yale copies of the replication program contain the same three expressions.\n\n\\begin{audittable}{lrrrrr}{Facts removed in the comparison group and in the reference group (positive facts in the first two rows, cited facts in the third), the printed figure, the difference divided by the mean of the two shares, and the relative increase.}\nComparison & Removed & Reference & Printed & Over the mean & Relative \\\\\nNegative vs positive, 12 hours & 75 of 150 & 52 of 150 & 36\\% & 36.22\\% & 44.23\\% \\\\\nNegative vs positive, 3 days & 92 of 150 & 69 of 150 & 29\\% & 28.57\\% & 33.33\\% \\\\\nUncited vs cited, 4 hours & 62 of 100 & 32 of 100 & 64\\% & 63.83\\% & 93.75\\%\n\\end{audittable}", "problem": "The sentence ``$A$ is $x$\\% more likely to be removed than $B$ within a given time'' compares the shares of facts removed by that time, with $B$ as the reference group. If $p_A$ and $p_B$ are the two shares, it means that $p_A=(1+x/100)\\,p_B$, that is, $x=100\\,(p_A/p_B-1)$. The three printed figures were computed differently. Let $a$ and $b$ be the numbers of facts removed in the two groups. The replication program evaluates $100\\,|a-b|/\\{(a+b)/2\\}$, the difference divided by the mean of the two counts. Since the groups have equal sizes, this equals $100\\,|p_A-p_B|/\\{(p_A+p_B)/2\\}$, a symmetric percentage difference. When $p_A>p_B>0$, the symmetric difference is smaller than the relative increase. The counts are 75 negative against 52 positive facts of 150 at 12 hours, 92 against 69 of 150 at 72 hours, and 62 uncited against 32 cited facts of 100 at four hours. The printed 36\\%, 29\\% and 64\\% are the symmetric differences 36.22\\%, 28.57\\% and 63.83\\%, rounded. The relative increases are 44.23\\%, 33.33\\% and 93.75\\%, and rounding does not account for the gap. The article gives no convention under which ``$x$\\% more likely'' means a symmetric difference. Someone who quotes the three figures as relative increases quotes numbers that are too small.", "what_survives": "Correcting only these percentages leaves the three pairs of counts, the direction of the three comparisons, the Kaplan--Meier curves, and the Cox estimates with their $p$-values (hazard ratios 1.59 and 2.33) unchanged; I have not checked the correctness of those other results here. The corrected increases are larger than the printed ones. If the survival times are rebuilt from the recorded minutes (\\fid{P041-E002}), converted from milliseconds to hours by division by $3{,}600{,}000$, and counted with an inclusive 12-hour cutoff, then 76 negative facts are removed within 12 hours and the first increase is $600/13=46.15\\ldots$\\%."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E001", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the heading 'Estimation of Experimental Effects' and the three lines below it. A red box marks 'survival time of the facts' in 'Our primary dependent variable is the survival time of the facts, the amount of time between when the fact is added to Wikipedia and when it is removed'.", "caption": "“Estimation of Experimental Effects”, p. 4 (PDF p. 4). The marked words name the quantity whose stored values are affected; the definition itself is correct.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E002.png"}, {"alt": "Crop of six lines of the codebook. Red boxes mark the variable names 'time\\_edit:', 'time\\_removed:' and 'time\\_alive:' at the start of their definitions, 'Date and time the edit was made, in Stata format', 'Date and time the edit was removed, in Stata format' and 'Difference between time\\_removed and time\\_edit'.", "caption": "Codebook of the replication files (Kalla_Aronow_PLOS_2015.pdf), p. 1. The marked names are the three variables that the data file stores in single precision.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E002-b.png"}, {"alt": "Listing of lines 11 to 17 of the Stata program. Red boxes mark line 12, 'gen curr\\_time = time\\_alive', and line 15, 'gen hours\\_alive=curr\\_time*2.77778e-7'.", "caption": "Replication program Kalla_Aronow_PLOS_2015.do, lines 11–17, typeset from the file. The marked lines take the stored survival time and convert it to hours with a rounded multiplier.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E002-c.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T4", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T4"}], "headline": "The replication data store the edit and removal times in single precision, so that the stored survival times differ from the recorded ones by up to 129 seconds; one 12-hour count changes from 75 to 76, and the pooled hazard ratios change in the third decimal.", "id": "P041-E002", "location": "“Estimation of Experimental Effects”, p. 4 (PDF p. 4); replication files: the three time variables of the data file, the codebook (p. 1), and lines 12 and 15 of the main Stata program", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P041-T4"], "text": {"as_printed": "Article, p.~4: Our primary dependent variable is the survival time of the facts--the amount of time between when the fact is added to Wikipedia and when it is removed.\n\nCodebook, p.~1:\n\n\\hl{\\texttt{time\\_edit}}: Date and time the edit was made, in Stata format\n\n\\hl{\\texttt{time\\_removed}}: Date and time the edit was removed, in Stata format. Coded as missing if the edit was not removed.\n\n\\hl{\\texttt{time\\_alive}}: Difference between \\texttt{time\\_removed} and \\texttt{time\\_edit}. Amount of time the edit was alive on the Wikipedia page.\n\nReplication program, lines 12 and 15:\n\n\\texttt{gen curr\\_time = time\\_alive}\n\n\\texttt{gen hours\\_alive=curr\\_time*\\hl{2.77778e-7}}", "corrected": "Rebuild \\texttt{time\\_edit} and \\texttt{time\\_removed} from the recorded strings \\texttt{time} and \\texttt{removed} as millisecond datetimes \\hl{in double precision}, keeping a missing value where the fact was not removed. Set \\texttt{time\\_alive} to the difference of the rebuilt times, and compute the hours as this difference \\hl{divided by $3{,}600{,}000$}, in double precision. Converting the stored floats to doubles does not recover the recorded minutes. For the reported Cox recalculation, \\hl{retain the program's censoring time of $5\\times10^9$ ms and replace zero durations by $0.0001$ ms before converting to hours}. Equal rebuilt durations are ties at the recorded resolution of one minute. With these survival times, \\hl{76} of the 150 negative facts of Studies 1, 2 and 4 are removed within 12 hours (stored values: 75), and the pooled hazard ratios of Table A of S6 File are \\hl{$0.628$} and \\hl{$0.428$} (stored values: $0.629$ and $0.429$).", "evidence": "The evidence is an exact counterexample from the data, with a recalculation. Set $q=2^{17}=131{,}072$. For the fact on the page of Jon Tester in Study 5, the recorded times are 18:49 and 20:47 on 6 November 2014, which are 118 minutes apart. In milliseconds the two times are\n\\[\n\\begin{aligned}\nX_1&=1{,}730{,}918{,}940{,}000=13{,}205{,}863\\,q+64{,}864,\\\\\nX_2&=1{,}730{,}926{,}020{,}000=13{,}205{,}917\\,q+66{,}976 .\n\\end{aligned}\n\\]\nSince $64{,}864<q/2<66{,}976$, the first is rounded down and the second up. The stored times are $13{,}205{,}863\\,q$ and $13{,}205{,}918\\,q$, and I obtain\n\\[\n\\texttt{time\\_alive}=55\\,q=7{,}208{,}960\\ \\text{ms},\\qquad X_2-X_1=7{,}080{,}000\\ \\text{ms},\n\\]\nan error of $128{,}960$ ms, or $128.96$ seconds. For the negative, cited fact on the page of Dan Coats in Study 1, the recorded times are 21:23 on 9 July and 09:23 on 10 July 2014, exactly 12 hours apart, and\n\\[\n\\begin{aligned}\nX_1&=1{,}720{,}560{,}180{,}000=13{,}126{,}832\\,q+56{,}096,\\\\\nX_2&=1{,}720{,}603{,}380{,}000=13{,}127{,}162\\,q+2{,}336 .\n\\end{aligned}\n\\]\nBoth are rounded down, so that $\\texttt{time\\_alive}=330\\,q=43{,}253{,}760$ ms, which exceeds $43{,}200{,}000$ ms. The program counts a removal within 12 hours when \\texttt{hours\\_alive} is at most 12, and this fact is not counted. With the recorded minutes it is counted. The number of negative facts removed within 12 hours in Studies 1, 2 and 4 is then 76, against 75 from the stored values, and the number of positive facts stays at 52. Among the horizons of 4, 6, 12, 24, 72 and 168 hours, the only other change is one Study 5 fact at 4 hours, outside the pooled samples. The multiplier alone would also misplace the Coats fact, since $43{,}200{,}000\\times2.77778\\times10^{-7}=12.0000096$.\n\nI re-estimated the two pooled Cox models of Table A of S6 File with survival times rebuilt from the recorded minutes and nothing else changed (Breslow ties, robust standard errors). The hazard ratio for positive facts moves from $0.628942$ to $0.627865$ ($p=0.0013$) and that for cited facts from $0.429227$ to $0.427530$ ($p<0.001$). Their reciprocals are $1.593$ and $2.339$, against the printed 1.59 and 2.33. In the 12-hour regressions of the kind reported in S11 File, the coefficient for positive facts moves from $0.153$ to $0.160$ and that for cited facts from $0.37$ to $0.36$, and both $p$-values stay below $0.01$.", "problem": "The article's definition of the survival time is correct; the stored values are not exact. The data file holds the three variables as Stata floats, which have 24 significant binary digits. A Stata datetime counts milliseconds from 1 January 1960, and the 2014 values lie between $2^{40}$ and $2^{41}$, where consecutive floats are $2^{17}=131{,}072$ milliseconds apart. The edit and removal times were recorded to the minute (the string variables \\texttt{time} and \\texttt{removed}). Each stored time is the recorded minute rounded to the nearest multiple of $131{,}072$ ms, and \\texttt{time\\_alive} is the difference of the two rounded times. Of the 294 survival times, 286 differ from the difference of the recorded minutes, by between $-126{,}752$ and $+128{,}960$ ms, and 15 stored survival times are zero where the recorded minutes give 8. The multiplier $2.77778\\times10^{-7}$ in line 15 is a rounded value of $1/3{,}600{,}000$ and moves a survival time of exactly 12 hours above 12.", "what_survives": "The designs, the treatment assignments and the removal indicators are unchanged. The hazard of removal remains lower for positive facts and for cited facts, and both pooled effects remain statistically significant. The 3-day counts (69 and 92) and the 4-hour citation counts (32 and 62) are unchanged, and no fact changes its classification at 6, 24 or 168 hours. Every analysis of the survival times inherits the rounding, and its entries can change in the last digits; I recomputed only the quantities listed above. The seconds within each minute were never recorded, and the correction does not restore them."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E002", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 11 to 17 of the Stata program. A red box marks 'ss(time\\_edit)' in line 17, 'gen hourentry = ss(time\\_edit)'.", "caption": "Replication program Kalla_Aronow_PLOS_2015.do, lines 11–17, typeset from the file. The marked term defines the variable hourentry with the function that returns seconds.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E003.png"}, {"alt": "Crop of the rows 'Covariates' and 'Studies' of Table E and the six-line note below them. Red boxes mark 'hour of entry' in 'Date of entry, and hour of entry removed to avoid convergence issues' and 'the date and time order' in 'Covariates Yes means controlling for the date and time order in which an edit was randomly assigned to be made'.", "caption": "S6 File, last rows of Table E and its note, from a rendering of the published Word file (the supplement has no fixed page numbers). The marked words name the control as the hour of entry and as part of the date and time order of the edit.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E003-b.png"}, {"alt": "Crop of four lines of page 4. A red box marks 'date of entry, and time of entry' in the list of covariates 'state population, Senate class, character count of page, geographic region, length of incumbency, political party, date of entry, and time of entry'.", "caption": "Results, p. 4 (PDF p. 4). The marked words list entry date and time among possible moderators, providing context for the intended meaning of time of entry.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E003-c.png"}, {"alt": "Dot plot of 20 hazard ratios on a logarithmic axis from 0.2 to 3, in six groups: S6 Table B (pooled), S6 Table E (by study), S7 Table B columns 1 and 2, and S8 columns 3 and 4. Each row has an open circle for the model with the seconds control and a square for the model with the hour of entry, and the two values are printed at the right. No pair lies on opposite sides of 1. Examples include Study 4 Positive in S6 Table E, 0.753 to 0.934; Positive in S7 Table B column 2, 0.708 to 0.781; Cited by Upcoming election in S8 column 4, 0.323 to 0.284; and Upcoming election in S8 column 4, 2.146 to 2.436. The pooled entries of S6 Table B move from 0.670 to 0.691 and from 0.352 to 0.345.", "caption": "The 20 hazard ratios of the adjusted models that include the control, re-estimated with the seconds within the minute (open circles) and with the hour of entry (squares); the two columns of numbers give the values. S7 Table B, column 1, pools Studies 1 and 4, and column 2 adds the first half of Study 5; S8, column 3, pools Studies 1, 2 and 4, and column 4 Studies 1 and 4. The values with the seconds equal the printed entries at three decimals in 19 rows; the exception is “Cited” in column 1 of S7 Table B, printed 0.386.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E003-hazard-ratios.png"}], "formal_release": null, "formal_statements": [], "headline": "The control described as the hour (time) of entry in the covariate-adjusted models is the seconds within the minute of the stored edit time; with the hour of entry the adjusted estimates change, and the 10% marker in column 2 of S7 Table B is lost.", "id": "P041-E003", "location": "Replication program Kalla_Aronow_PLOS_2015.do, line 17; S6 File, Tables B and E and their notes; S7 File, Table B; S8 File, columns 3 and 4; Results, “Survival of Positive and Negative Facts on Wikipedia”, p. 4 (PDF p. 4)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Replication program, line 17:\n\n\\texttt{gen hourentry = \\hl{ss}(time\\_edit)}\n\nS6 File, note to Table E: ${}^{\\wedge}$Date of entry, and \\hl{hour of entry} removed to avoid convergence issues\n\nS6, S7 and S8 Files, notes to the tables: Covariates Yes means controlling for the date and \\hl{time order} in which an edit was randomly assigned to be made, a binary variable for Republicans, Senate class, region (NE, S, W), length of incumbency, log of Wikipedia page character count before Study 1 began, log of state population, and a dichotomous influence variable for party leaders and committee chairs.\n\nResults, p.~4 (context): We find no systematic evidence that this effect is moderated by covariates (state population, Senate class, character count of page, geographic region, length of incumbency, political party, date of entry, and time of entry; see Table A in S9 File and S10 File), as specified in our registered pre-analysis plan.", "corrected": "Replication program, line 17, with the hour of day taken from a double-precision datetime parsed from the recorded edit-time string \\texttt{time}:\n\n\\texttt{\\hl{gen double time\\_edit\\_exact = clock(time, \"hmDMY\")}}\n\n\\texttt{gen \\hl{byte} hourentry = \\hl{hh(time\\_edit\\_exact)}}\n\nThe two lines state the intended computation; the re-estimates use the same construction outside Stata. With this control, the models that include \\texttt{hourentry} are estimated again, and their estimates, standard errors and significance markers replace the printed ones. The pooled entries of Table B of S6 File become \\hl{$0.691$ $(0.097)$} for positive facts and \\hl{$0.345$ $(0.052)$} for cited facts; in column 2 of S7 Table B, ``Positive'' becomes \\hl{$0.781$ $(0.156)$, without a marker}; in column 4 of S8 File the interaction of citation with an upcoming election becomes \\hl{$0.284$ $(0.088)$}, and the main text (p.~5) reads ``interaction term hazard ratio = \\hl{0.28}, $p<0.001$''. The remaining hazard ratios are in the figure. The note to Table E for the first half of Study 5 stands.", "evidence": "The evidence is an exact example from the data and a re-estimation of the affected models. For a datetime $t$ in milliseconds, the two functions are\n\\[\n\\operatorname{ss}(t)=\\frac{t \\bmod 60{,}000}{1000},\\qquad \\operatorname{hh}(t)=\\Bigl\\lfloor \\frac{t}{3{,}600{,}000}\\Bigr\\rfloor \\bmod 24 .\n\\]\nThe Study 5 edit on the page of Al Franken was recorded at 17:18 on 7 November 2014. Its stored time is $t=1{,}730{,}999{,}877{,}632$, that is, 17:17:57.632, and\n\\[\nt=28{,}849{,}997\\times 60{,}000+57{,}632,\\qquad \\operatorname{ss}(t)=57.632,\\qquad \\operatorname{hh}(t)=17 .\n\\]\nThe program assigns $57.632$ to an edit made in hour 17. In all 551 rows the value of \\texttt{ss(time\\_edit)} differs from the hour.\n\nI re-estimated the Cox models in R with Breslow ties and robust standard errors scaled as in Stata. With the seconds as the control, I obtain the printed entries of Table B of S6 File, $0.670$ $(0.095)$ and $0.352$ $(0.055)$. The other hazard ratios shown also agree at three decimals, one entry of S7 Table B aside (printed $0.386$, re-estimated $0.389$). I then replaced the control by the integer hour of day, rebuilt from the strings that record the edit times. The table lists selected changes, and the figure shows all 20. Every hazard ratio keeps its side of 1, and every conclusion at the 5\\% level that I checked is unchanged. The $p$-value of ``Positive'' in column 2 of S7 Table B rises from $0.095$ to $0.214$. Applying \\texttt{hh()} to the stored times is not enough, since rounding moves three edits across an hour boundary. With a fractional hour (hour plus minutes over 60), the two pooled entries are $0.691$ and $0.344$, nearly the same as with the integer hour.\n\n\\begin{audittable}{lrrrr}{Hazard ratios (standard errors) of adjusted models with the control as run and with the hour of entry, and the two $p$-values.}\nEntry & Seconds, as printed & Hour of entry & $p$, seconds & $p$, hour \\\\\nS6 Table B, Positive & 0.670 (0.095) & 0.691 (0.097) & 0.0045 & 0.0085 \\\\\nS6 Table B, Cited & 0.352 (0.055) & 0.345 (0.052) & $<0.001$ & $<0.001$ \\\\\nS6 Table E, Study 4, Positive & 0.753 (0.200) & 0.934 (0.267) & 0.29 & 0.81 \\\\\nS7 Table B, column 2, Positive & 0.708 (0.146) & 0.781 (0.156) & 0.095 & 0.214 \\\\\nS8, column 4, Cited $\\times$ Upcoming Election & 0.323 (0.104) & 0.284 (0.088) & $<0.001$ & $<0.001$\n\\end{audittable}", "problem": "In Stata, \\texttt{ss()} returns the seconds component of a datetime and \\texttt{hh()} returns the hour. Line 17 of the replication program therefore sets the variable \\texttt{hourentry} to the seconds within the minute of the stored edit time. The supplements call this control the ``hour of entry'' (note to Table E of S6 File) and the ``date and time order'' of the edit. The main text speaks of the time of day (p.~3) and of the ``time of entry'' (p.~4). The recorded edit-time strings have zero seconds. The fitted seconds values are the remainders left by the single-precision rounding of the stored times (\\fid{P041-E002}). They range from $0.032$ to $59.904$, and their correlation with the fractional hour of day is $-0.03$. Every adjusted model that includes \\texttt{hourentry} controls for a different variable from the one described. These are the two models of Table B of S6 File, the models of Table E for Studies 1, 2, 4 and the second half of Study 5, the two models of Table B of S7 File, and columns 3 and 4 of S8 File. The printed entries of these models are those of a specification that the paper does not describe. The Table E model for the first half of Study 5 omits the date and the hour, as its note says, and is not affected. The analyses of S9 and S10 Files do not use the variable; on whether a test of the ``time of entry'' as a moderator is reported, see \\fid{P041-N005}.", "what_survives": "The unadjusted estimates of Table A of S6 File, and with them the hazard ratios 1.59 and 2.33 of the main text, do not use the variable. The same holds for the Kaplan--Meier curves, the balance tables, Study 3, the unadjusted interactions, the regressions of S11 File and the tables of S9 and S10 Files. In the affected models the pooled effects of positive and cited facts stay significant, and no sign changes. All 20 hazard ratios of those models change at three decimals. For example, the Study 4 hazard ratio for positive facts in Table E moves from $0.75$ to $0.93$. The interaction of citation with an upcoming election, which the main text quotes as $0.32$ (p.~5), becomes $0.28$, and ``Positive'' in column 2 of S7 Table B is no longer significant at the 10\\% level."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E003", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Three rows of a table of estimates in two columns: Cited X W Region, 1.128 and 1.193; Cited X Length of Incumbency, 1.056 with three stars in both columns, standard errors (0.020) and (0.019); and the label Cited X Log(Page Length). A red rectangle marks the words Length of Incumbency.", "caption": "S9 File, Table B, on the second page of the supplement as rendered from the published Word file. The label “Length of Incumbency” is marked in the row with the two entries 1.056***; the covariate is the year of first election.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E004.png"}, {"alt": "Four lines of the codebook in typewriter type: class, party, lengthofincumbency and leadership. A red rectangle marks the definition in the entry, lengthofincumbency: Year Senator was first elected.", "caption": "Codebook of the replication files (PDF p. 1), entry lengthofincumbency, with its definition marked.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E004-codebook.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T10", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T10"}], "headline": "The covariate labelled length of incumbency in the Cox models of S9 and S10 Files is the year in which the senator was first elected, and a larger value means a shorter tenure: label the rows “First-election year”.", "id": "P041-E004", "location": "S9 File, Tables A and B, rows “Positive X Length of Incumbency” and “Cited X Length of Incumbency”; S10 File, row “Incumbency Length”; cited from p. 4 (PDF p. 4), “length of incumbency”", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P041-T10"], "text": {"as_printed": "\\begin{itemize}\n\\item S9 File, Table A: ``Positive X \\hl{Length of Incumbency}''. Coefficient: 0.989, 0.981. Standard error: (0.019), (0.017).\n\\item S9 File, Table B: ``Cited X \\hl{Length of Incumbency}''. Coefficient: 1.056***, 1.056***. Standard error: (0.020), (0.019).\n\\item S10 File: ``\\hl{Incumbency Length}''. Coefficient: 1.004. Standard error: (0.008).\n\\item Codebook: ``lengthofincumbency: Year Senator was first elected.''\n\\item Do-file, line 8: \\texttt{gen yearsinsenate=2014-lengthofincumbency}\n\\end{itemize}", "corrected": "Label the rows by the variable that was fitted: ``Positive X \\hl{First-election year}'' in Table~A of S9 File, ``Cited X \\hl{First-election year}'' in Table~B of S9 File, and ``\\hl{First-election year}'' in S10 File, with a note that a later year means a shorter tenure. The printed entries are unchanged. The lists of covariates that call the same variable length of incumbency, on p.~4 and in the table notes, change in the same way.\n\nAlternatively, keep the tenure labels and report the models in the tenure $L=2014-Y$. The model $h_0(t)\\exp(bY+aT+gTY+z)$ is the model $h_{0,L}(t)\\exp(b_LL+a_LT+g_LTL+z)$ with\n\\[\nb_L=-b,\\qquad g_L=-g,\\qquad a_L=a+2014\\,g,\\qquad h_{0,L}(t)=h_0(t)\\,e^{2014\\,b},\n\\]\nand the covariance matrix of $(a_L,b_L,g_L)$ is $MVM^\\top$, where $V$ is the covariance matrix of $(a,b,g)$ and\n\\[\nM=\\begin{pmatrix}1&0&2014\\\\0&-1&0\\\\0&0&-1\\end{pmatrix}.\n\\]\nThe hazard ratios per year of tenure are the reciprocals $e^{-g}$ and $e^{-b}$ of the printed ones. For an \\hl{unrounded} ratio $r$ with standard error $s$, the transformed standard error is $s/r^2$; \\hl{round only after transforming the fitted values}. The treatment main effect of each interaction model then refers to tenure zero.", "evidence": "The evidence is a recalculation from the replication data and a derivation. Refitting the five models with the year of first election reproduces all five printed pairs of estimate and standard error to three decimals. Let $Y$ be the year of first election, $L=2014-Y$ the tenure, $T$ the treatment indicator (positive or cited) and $z$ the remaining terms of the linear predictor. I write the fitted hazard as $h(t\\mid T,Y,z)=h_0(t)\\exp(bY+aT+gTY+z)$. Substituting $Y=2014-L$, I obtain\n\\[\nbY+aT+gTY+z=2014\\,b-bL+(a+2014\\,g)\\,T-gTL+z.\n\\]\nThe treatment hazard ratio at tenure $L$ is $\\exp(a+2014\\,g-gL)$, so one more year of tenure multiplies it by $e^{-g}=1/e^{g}$. S10 File has no treatment term, and there the hazard ratio per year of tenure is $e^{-b}$. The five models refitted with $L$ give the reciprocals in the table.\n\n\\begin{audittable}{lrr}{Incumbency rows of S9 and S10 Files: hazard ratio (standard error) as printed, per later year of first election, and per additional year of tenure.}\nRow & As printed, per year of first election & Per year of tenure \\\\\nS9 File, Table A, column 1 & 0.989 (0.019) & 1.011 (0.020) \\\\\nS9 File, Table A, column 2 & 0.981 (0.017) & 1.019 (0.017) \\\\\nS9 File, Table B, column 1 & 1.056 (0.020) & 0.947 (0.018) \\\\\nS9 File, Table B, column 2 & 1.056 (0.019) & 0.947 (0.017) \\\\\nS10 File & 1.004 (0.008) & 0.996 (0.008) \\\\\n\\end{audittable}", "problem": "The codebook defines \\texttt{lengthofincumbency} as the year in which the senator was first elected, and its values in the data are calendar years, 1993 and 2013 for example. The do-file builds a tenure, \\texttt{yearsinsenate}, as 2014 minus that year, but uses it only in the balance checks. The Cox commands behind S9 and S10 Files use the year. A row labelled length of incumbency is read per additional year of tenure. In S9 File the entry is read as the factor by which the hazard ratio of the treatment changes, and in S10 File as the hazard ratio itself. Each of the five printed entries is the corresponding factor per later year of first election, and the factor per year of tenure is its reciprocal. The direction is therefore read backwards in all three rows, among them the one starred row, 1.056***. The entries are hazard ratios, as in the other Cox tables.", "what_survives": "The fitted hazards, the partial likelihood, the test of each incumbency interaction, the test of the incumbency main effect and the joint moderation tests are the same under either coding. The interaction in Table~B of S9 File stays significant, with $p=0.004$ and $p=0.003$ in its two columns, in the opposite direction. The main text draws no conclusion about the direction of an incumbency effect. The comparisons of treatments, the balance tables and the other coefficients of the models that use the year only as an additive control are unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E004", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the first three lines of page 3. Red boxes mark 'left the Sen-' at the end of the first line and 'ate since 1990' at the start of the second, in 'senators who have passed away or otherwise left the Senate since 1990 and do not currently hold any elective office or serve as a Cabinet Secretary'.", "caption": "Methods, first lines of p. 3 (PDF p. 3); the sentence begins on p. 2. The marked words give the rule “left the Senate since 1990”.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E005.png"}, {"alt": "Crop of five lines of page 5. A red box marks 'senators who have left office since 1990' in 'Study 3 conducted a replication on the Wikipedia pages of senators who have left office since 1990'.", "caption": "Results, “Replication on Wikipedia Pages of Dead and Retired Senators”, p. 5 (PDF p. 5). The marked words describe the Study 3 sample.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E005-b.png"}, {"alt": "Crop of the last five lines of the abstract. A red box marks 'recently retired' in 'deceased as well as recently retired but living senators'.", "caption": "Abstract, p. 1 (PDF p. 1). The marked words describe the living senators of Study 3.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E005-c.png"}, {"alt": "Crop of five lines of the Discussion. A red box marks 'recently retired' in 'In a test conducted on the Wikipedia pages of recently retired and deceased senators, we find no editorial bias'.", "caption": "Discussion, p. 5 (PDF p. 5). The marked words describe the living senators of Study 3.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E005-d.png"}], "formal_release": null, "formal_statements": [], "headline": "Study 3 was not restricted to senators who left the Senate since 1990: 20 of its 72 living former senators left before 1990, and S3 File reports a broader eligibility rule.", "id": "P041-E005", "location": "Methods, pp. 2–3 (PDF pp. 2–3); Results, “Replication on Wikipedia Pages of Dead and Retired Senators”, p. 5 (PDF p. 5); Abstract, p. 1 (PDF p. 1); Discussion, p. 5 (PDF p. 5)", "pdf_page": 2, "printed_page": "2–3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Methods, pp.~2--3: Studies 1, 2, and 4 were conducted on the English-language Wikipedia pages of the 100 current U.S. senators, while Study 3 was conducted on the 151 English-language Wikipedia pages of senators who have passed away or otherwise \\hl{left the Senate since 1990} and do not currently hold any elective office or serve as a Cabinet Secretary.\n\nResults, p.~5: To ensure that this more conservative approach does not explain our finding of editorial bias, Study 3 conducted a replication on the Wikipedia pages of \\hl{senators who have left office since 1990}.\n\nAbstract, p.~1: We replicate this study on the Wikipedia pages of deceased as well as \\hl{recently retired} but living senators and find no evidence of an editorial bias in either.\n\nDiscussion, p.~5: In a test conducted on the Wikipedia pages of \\hl{recently retired} and deceased senators, we find no editorial bias, suggesting that this bias is limited to the pages of active politicians and is not systemic to Wikipedia.", "corrected": "Methods: \\ldots while Study 3 was conducted on the 151 English-language Wikipedia pages of \\hl{former U.S. senators for whom both a positive and a negative fact could be collected. S3 File reports an eligible pool of 243 senators who had passed away since 1990, or were living and had retired from the Senate during or before 2009. The living sample included 20 senators whose Senate service ended before 1990.}\n\nResults, p.~5: \\ldots Study 3 conducted a replication on the Wikipedia pages of \\hl{this sample of former senators}.\n\nAbstract and Discussion: replace ``recently retired'' by \\hl{``retired''}. The clause on elective office and Cabinet service should be kept only after it has been checked against the sample.", "evidence": "The evidence is a recalculation from the replication data. I take the last year in the Senate to be the last four characters of the variable \\texttt{term}, as the replication program does; it equals the latest year among the terms listed for each senator. With this definition, I find that the means of the last year are $29831/15\\approx1988.7$ for the 75 positive facts and $151157/76\\approx1988.9$ for the 76 negative facts. These are the two values printed in Table A of S3 File. All 72 living senators left by 2009, in agreement with the rule of S3 File.", "problem": "The Study 3 sample has 151 former senators, 72 living and 79 deceased. The last year of Senate service is before 1990 for 75 of them, 20 living and 55 deceased. A living senator who left before 1990 fits neither ``passed away'' nor ``left the Senate since 1990''; examples are Bill Brock (1971--1977), Dan Quayle (1981--1989) and Paul S. Trible, Jr. (1983--1989). S3 File reports the broader eligibility rule: ``senators who have passed away since 1990 or are living and retired during or before 2009'', of whom 243 qualified and 151 had both facts collected. The living senators left the Senate between 1971 and 2009, with median 1997, so that ``recently retired'' does not convey the full range of departure dates. I did not check the clause ``do not currently hold any elective office'' against dated records of office, and it is neither confirmed nor refuted here.", "what_survives": "No estimate changes. The Study 3 curves in Fig 1 and Table C of S6 File ($N=151$, 72 and 79) describe the sample that was analyzed, and the pooled analyses of Studies 1, 2 and 4 do not use Study 3. What changes is the description of the comparison group: it is the former senators of the S3 rule, and a comparison restricted to departures since 1990 (52 of the 72 living senators) would be a separate analysis."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E005", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of the article. Red boxes mark the words 'cannot be explained' in the sentence 'This suggests that the editorial bias cannot be explained by Wikipedia's culture in general or by policies concerning biographies of living persons in particular', and the words 'limited to active politicians' in the sentence that follows.", "caption": "End of the section “Replication on Wikipedia Pages of Dead and Retired Senators”, p. 5 (PDF p. 5). The marked words carry the two conclusions that Study 3 does not support.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E006.png"}, {"alt": "Five lines of the article. A red box marks the clause 'is limited to the pages of active politicians and is not systemic to Wikipedia', which ends the sentence that begins 'In a test conducted on the Wikipedia pages of recently retired and deceased senators'.", "caption": "Discussion, first paragraph, p. 5 (PDF p. 5). The marked clause repeats the conclusion.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E006-b.png"}, {"alt": "A chart with four horizontal intervals on a logarithmic axis from 0.125 to 4. Sitting senators, Studies 1, 2, 4, 199 of 300 facts removed: 0.629, interval 0.47 to 0.83, a short blue interval that lies entirely below 1. Former senators, Study 3, 20 of 151 facts removed: 0.691, interval 0.28 to 1.70. Study 3, living, 10 of 72 removed: 0.775, interval 0.22 to 2.77. Study 3, deceased, 10 of 79 removed: 0.627, interval 0.18 to 2.23. The three orange intervals for Study 3 are wide and cross both the solid vertical line at 1 and the dashed vertical line at 0.629, the estimate for sitting senators.", "caption": "Hazard ratios of removal for positive against negative facts, with 95% intervals, from a recalculation on the replication files. The four estimates are the entries printed in S6 Tables A and C. The dashed line is the estimate for sitting senators, and the solid line is a ratio of 1.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E006-hazard-ratios.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T1", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T1"}, {"id": "P041-T2", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T2"}, {"id": "P041-T11", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T11"}], "headline": "Study 3’s nondetection and smaller early absolute survival differences do not establish that editorial bias is limited to active politicians or is not systemic to Wikipedia, or exclude Wikipedia’s culture or its policy on biographies of living persons as explanations.", "id": "P041-E006", "location": "“Replication on Wikipedia Pages of Dead and Retired Senators”, last two sentences, p. 5 (PDF p. 5); Discussion, first paragraph, p. 5 (PDF p. 5)", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": ["P041-T1", "P041-T11", "P041-T2"], "text": {"as_printed": "End of the section on Study 3, p.~5: ``This suggests that the editorial bias \\hl{cannot be explained} by Wikipedia's culture in general or by policies concerning biographies of living persons in particular. Instead, our findings suggest the editorial bias is \\hl{limited to active politicians}, the types of people for whom the stakes of a positive public image are the highest.''\n\nDiscussion, p.~5: ``In a test conducted on the Wikipedia pages of recently retired and deceased senators, we find no editorial bias, suggesting that this bias \\hl{is limited to the pages of active politicians and is not systemic to Wikipedia}.''", "corrected": "End of the section on Study 3, in place of the two sentences: ``Study 3 detected no statistically clear difference between positive and negative facts over the full follow-up. Its estimate of the relative effect is imprecise and does not exclude the estimate for sitting senators. Early absolute differences in survival were smaller on the sampled pages of former senators, also when the comparison is restricted to cited facts. These comparisons \\hl{do not establish that relative bias is absent} on the pages of former senators, do not identify political activity as the cause of the bias, and \\hl{do not exclude} Wikipedia's culture or the enforcement of its policy on biographies of living persons as explanations.''\n\nDiscussion, p.~5: ``In a test conducted on the Wikipedia pages of recently retired and deceased senators, \\hl{no editorial bias was detected; the estimate is too imprecise to show} that this bias is limited to the pages of active politicians or that it is not systemic to Wikipedia.''", "evidence": "The evidence is a recalculation from the replication files and an exact counterexample.\n\nThe recalculation reproduces the four printed hazard ratios and standard errors of S6 Tables A and C and the three $p$-values of the main text. The figure shows the estimates with their 95\\% intervals. To compare the two populations, I fit a Cox model to Studies 1--4 with an interaction between valence and Study 3. The ratio between the hazard ratio for former senators and the hazard ratio for sitting senators is 1.12 ($p=0.81$), or 1.09 ($p=0.85$) when the baseline hazard is stratified by study. Treating the samples as independent and approximating the log hazard ratio standard errors by $0.318/0.691$ and $0.091/0.629$, the printed entries give\n\\[\nz=\\frac{\\log 0.691-\\log 0.629}{\\sqrt{(0.318/0.691)^2+(0.091/0.629)^2}}=\\frac{0.0940}{0.4824}=0.19,\\qquad p=0.85 .\n\\]\nAt 12 hours, the difference in survival between positive and negative facts is $23/150=0.153$ for sitting senators and $-51/1900=-0.027$ for former senators, and the gap between the two populations has $p=0.0019$. At 7 days the corresponding figures are 0.127, $-0.054$ and $p=0.0038$. When the comparison for sitting senators is restricted to cited facts, the differences are 0.140 and 0.138, with $p=0.0097$ and $p=0.0091$.\n\nThe counterexample shows that such a gap in absolute differences is compatible with one and the same relative bias. Suppose that, in each population, positive facts are removed at the constant hazard $\\lambda_+=-2\\log(q)/12$ per hour and negative facts at the constant hazard $\\lambda_-=-3\\log(q)/12$ per hour, where $q=4/5$ for sitting senators and $q=99/100$ for former senators. Then $\\lambda_+/\\lambda_-=2/3$ in both populations. The probability that a fact survives 12 hours is $S_+(12)=\\exp(-12\\lambda_+)=q^2$ for a positive fact and $S_-(12)=\\exp(-12\\lambda_-)=q^3$ for a negative fact, which gives $S_+(12)-S_-(12)=q^2(1-q)$.\n\n\\begin{audittable}{llllll}{The counterexample: the same hazard ratio in both populations, and a much smaller absolute difference in survival at 12 hours where facts are removed less often.}\nPages & $q$ & $S_+(12)$ & $S_-(12)$ & $S_+(12)-S_-(12)$ & $\\lambda_+/\\lambda_-$ \\\\\nSitting senators & $4/5$ & $16/25$ & $64/125$ & $16/125=0.128$ & $2/3$ \\\\\nFormer senators & $99/100$ & $9801/10000$ & $970299/1000000$ & $9801/1000000=0.009801$ & $2/3$ \\\\\n\\end{audittable}\n\nThe model illustrates the logical point and is not a fit to the data.", "problem": "Study 3 added facts to the pages of 151 former senators, and 20 of the facts were removed during the follow-up: 8 of 75 positive facts and 12 of 76 negative facts. The article reports that it finds no difference between the rates at which positive and negative facts are removed ($p=0.70$ for living and $p=0.47$ for deceased former senators, and $p=0.42$ pooled). Read as a statement that the tests detected no difference, this is correct. The two passages above go further. They conclude that the bias found on the pages of sitting senators is absent elsewhere, and that Wikipedia's culture and its policy on living persons are excluded as explanations. These conclusions need more than a test that detects no difference. They need evidence that the effect on the pages of former senators is absent, or smaller than the effect on the pages of sitting senators.\n\nOn the primary measure of the article, Study 3 gives no such evidence. The hazard ratio of removal for positive against negative facts is 0.691 for former senators (S6 Table C) and 0.629 for sitting senators (S6 Table A). The 95\\% interval for former senators runs from 0.28 to 1.70 and contains both 1 and 0.629. Comparing the two hazard ratios directly, I find a ratio of 1.12 with $p=0.81$. Study 3 therefore cannot distinguish the absence of bias from a relative bias as large as the one found for sitting senators.\n\nThe data do contain evidence of a narrower kind. In the first days, the difference in survival between positive and negative facts was larger on the pages of sitting senators: 0.153 against $-0.027$ at 12 hours and 0.127 against $-0.054$ at 7 days, and a test rejects equality of these survival differences at both times. A smaller absolute difference does not show that relative bias is absent, because the same relative preference can produce a smaller absolute difference on pages that are edited less often. Further, the experiments randomized the valence of a fact and not the population of pages. The pages of sitting and of former senators differ in more than political activity. These comparisons therefore do not identify political activity as the cause of the bias, and they do not exclude Wikipedia's culture or the enforcement of its policy on living persons as explanations. The second paragraph of the Discussion allows as much when it says that editors may apply the guidelines for living persons more conservatively to sitting senators.", "what_survives": "The bias on the pages of sitting senators stands (hazard ratio 0.629, $p=0.001$). The statement that Study 3 detected no difference also stands, and so does the sentence of the abstract that reports ``no evidence of an editorial bias'' for former senators, read as a statement that none was detected. The early absolute differences in survival were smaller on the pages of former senators, also when the comparison is restricted to cited facts. The findings on citations do not depend on this interpretation. What is given up is the conclusion that the bias is limited to active politicians and is not systemic to Wikipedia, and the exclusion of Wikipedia's culture and of its policy on living persons as explanations. No data and no estimate change."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E006", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the seven-line paragraph 'Implementation on Wikipedia'. A red box marks 'the name of the editing Wikipedia account, and the IP address' in the sentence 'In order to prevent confounding, the order in which the edits were made, the time of day, the name of the editing Wikipedia account, and the IP address used to make the edit were randomly assigned as well'.", "caption": "“Implementation on Wikipedia”, p. 3 (PDF p. 3). The marked words are the two items whose random assignment did not hold in every study.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E007.png"}], "formal_release": null, "formal_statements": [], "headline": "The editing account and the IP address were not randomly assigned in every study: IP addresses were randomized from Study 2 on, and after its first day Study 1 used a single account.", "id": "P041-E007", "location": "“Implementation on Wikipedia”, p. 3 (PDF p. 3), second and third sentences", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "In order to prevent confounding, the order in which the edits were made, the time of day, \\hl{the name of the editing Wikipedia account, and the IP address} used to make the edit were randomly assigned as well. With the exception of Study 1, new accounts were created before each study began and were seeded with true, cited, political, non-Senate edits to Wikipedia. More details are presented in S1--S4 Files.", "corrected": "In order to prevent confounding, the order in which the edits were made and the time of day were randomly assigned as well. \\hl{IP addresses were randomly assigned beginning in Study 2. Accounts were initially randomly assigned; after the first day's edits in Study 1, all subsequent edits used veto118, the account randomly assigned to be first.} With the exception of Study 1, new accounts were created before each study began and were seeded with true, cited, political, non-Senate edits to Wikipedia; \\hl{previously used, autoconfirmed accounts were used for locked pages in Study 4 and the concluding Study 5.} More details are presented in \\hl{S1--S5 Files}.", "evidence": "The evidence is a check of the page against the text of S1, S2, S4 and S5 Files.", "problem": "The supplements state three exceptions that the summary omits. S2 File says that the IP address was ``only introduced beginning in Study 2''. S1 File says that after the first day of Study 1, ``All subsequent edits were made from the same account, veto118'', an account that ``was randomly assigned to be the first account''. S4 and S5 Files say that the ten locked pages were edited from accounts that had ``already been used to make edits in previous rounds of this experiment''. Note that reusing accounts does not contradict the statement that new accounts were created.", "what_survives": "The supplements disclose the exceptions, and no estimate depends on the summary. The assignment of valence and citation, the survival times and every reported estimate are unchanged. The wording alone does not show that any estimate is biased."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E007", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The last six rows of a regression table with two columns of entries: Wald test (p-value) 0.225 and 0.095; N 200 and 250; Main Effects Yes, Yes; Fixed Effects Yes, Yes; Covariates No, No; Studies 1,4 and 1,2,4,5 (1st Half). Red boxes mark the two entries of the row N and the two entries of the row Studies.", "caption": "S9 File, Table B, last rows and the first line of the note, in a rendering of the published Word file. The marked rows state 200 and 250 edits and the Studies “1,4” and “1,2,4,5 (1st Half)”.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E008.png"}], "formal_release": null, "formal_statements": [], "headline": "The entries of S9 Table B were estimated on 300 edits (Studies 1, 2, 4) and 400 edits (Studies 1, 2, 4, 5), not on the 200 and 250 edits that its rows “N” and “Studies” state.", "id": "P041-E008", "location": "S9 File, Table B (“Covariates as Moderators of Cited Edits from Cox Regressions”), rows “N” and “Studies”; the table is cited on p. 5 (PDF p. 5) of the article. The S9 File is a Word document without page numbers.", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lll}{S9 File, Table B, last rows.}\n & Column 1 & Column 2 \\\\\nWald test (p-value) & 0.225 & 0.095 \\\\\nN & \\hl{200} & \\hl{250} \\\\\nMain Effects & Yes & Yes \\\\\nFixed Effects & Yes & Yes \\\\\nCovariates & No & No \\\\\nStudies & \\hl{1,4} & \\hl{1,2,4,5 (1st Half)} \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lll}{S9 File, Table B, last rows, with the samples of the printed estimates.}\n & Column 1 & Column 2 \\\\\nWald test (p-value) & 0.225 & 0.095 \\\\\nN & \\hl{300} & \\hl{400} \\\\\nMain Effects & Yes & Yes \\\\\nFixed Effects & Yes & Yes \\\\\nCovariates & No & No \\\\\nStudies & \\hl{1,2,4} & \\hl{1,2,4,5} \\\\\n\\end{audittable}\n\nAlternatively, the table may report the samples in which citation was randomized. The rows then read N = 200 and 250 and Studies ``1,4'' and ``1,4,5 (1st Half)'', every hazard ratio, standard error and marker is replaced by its value on those samples, and the joint $p$-values are 0.5274 and 0.3110.", "evidence": "The evidence is a recalculation from the replication files. On 300 and 400 edits, the fits reproduce all sixteen printed hazard ratios, all sixteen standard errors and both joint $p$-values to three decimals. On the 200 and 250 edits in which citation was randomized (Studies 1 and 4, and these two with the first half of Study 5), the same model has joint $p$-values 0.527 and 0.311. The interaction with influence then loses its marker, and the interaction with length of incumbency is significant at 5\\% and not at 1\\%.", "problem": "The rows ``N'' and ``Studies'' describe the sample of each regression, and neither column is described correctly. The commands in the replication files restrict the first regression to Studies 1, 2 and 4 (300 edits) and the second to Studies 1, 2, 4 and all of Study 5 (400 edits). The second entry of the row ``Studies'' is also inconsistent with its own ``N'', since Studies 1, 2, 4 and the first half of Study 5 contain 350 edits.", "what_survives": "If the two rows are corrected, the estimates, standard errors and tests in the table are unchanged. The main text (p.~5) cites the table for the statement that there is no systematic evidence that other variables moderate the effect of citations, and this statement is not weakened under either repair."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E008", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The last four rows of a regression table with four columns: N 300, 200, 300, 200; Fixed Effects Yes in all columns; Covariates No, No, Yes, Yes; Studies 1,2,4, then 1,4, then 1,2,4, then 1,4. Below is the note of the table. Red boxes mark the two entries 'Yes' of the row Covariates in columns 3 and 4, and the last clause of the note, 'and a dichotomous influence variable for party leaders and committee chairs.'", "caption": "S8 File, last rows of the table and its note, in a rendering of the published Word file. The marked flags call columns 3 and 4 covariate-adjusted, and the marked clause of the note lists the influence variable among the covariates.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E009.png"}, {"alt": "The last six rows of a regression table with two columns: Wald test (p-value) 0.321 and 0.221; N 300 and 400; Main Effects Yes, Yes; Fixed Effects Yes, Yes; Covariates Yes, No; Studies 1,2,4 and 1,2,4,5. A red box marks the entry 'Yes' of the row Covariates in the first column. Three lines of the note follow.", "caption": "S9 File, Table A, last rows and the start of its note, in a rendering of the published Word file. The marked flag is “Yes” for “Covariates” in the first column.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The note to the table of the S8 File lists a control for influence that its two adjusted models do not contain, and the first column of S9 Table A is marked “Covariates Yes” although its two columns fit the same specification and differ only in the studies included.", "id": "P041-E009", "location": "S8 File, row “Covariates” and note of the table; S9 File, Table A, row “Covariates”; S6 File, note to Table D. The supplementary files are Word documents without page numbers; they are listed on p. 6 (PDF p. 6) of the article, and the main text cites the S8 File on p. 5.", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "S8 File, note to the table: ``Covariates Yes means controlling for \\hl{the date and time order in which an edit was randomly assigned to be made}, a binary variable for Republicans, Senate class, region (NE, S, W), \\hl{length of incumbency}, log of Wikipedia page character count before Study 1 began, log of state population, \\hl{and a dichotomous influence variable for party leaders and committee chairs}.''\n\n\\begin{audittable}{lll}{S9 File, Table A, last rows.}\n & Column 1 & Column 2 \\\\\nN & 300 & 400 \\\\\nMain Effects & Yes & Yes \\\\\nFixed Effects & Yes & Yes \\\\\nCovariates & \\hl{Yes} & No \\\\\nStudies & 1,2,4 & 1,2,4,5 \\\\\n\\end{audittable}\n\nS6 File, note to Table D: ``****Date of entry and hour of entry removed to avoid convergence issues''.", "corrected": "S8 File, note to the table, describing the models as fitted: ``Covariates Yes means controlling for \\hl{date of entry and seconds within the entry minute}, a binary variable for Republicans, Senate class, region (NE, S, W), \\hl{first-election year}, log of Wikipedia page character count before Study 1 began, \\hl{and} log of state population.''\n\nS9 File, Table A, row ``Covariates'': \\hl{No} in both columns. The note may add that both models include the main effects of Republican status, influence, region, \\hl{first-election year}, log page character count and log state population, the interactions of ``Positive'' with each of them, and indicators for study wave; that neither model adds controls for the date of entry, the hour of entry or Senate class; and that the columns differ in the studies included.\n\nS6 File, note to Table D: the clause ``****Date of entry and hour of entry removed to avoid convergence issues'' is deleted.", "evidence": "The evidence is a recalculation from the replication files. The corrected descriptions of the entry-time and incumbency variables follow \\fid{P041-E003} and \\fid{P041-E004}. Without the influence variable, the two adjusted models of the S8 File reproduce the six printed hazard ratios 0.574, 1.557, 0.783 and 0.507, 0.323, 2.146. With it, I obtain 0.580, 1.538, 0.774 and 0.515, 0.294, 2.237. The two specifications of S9 Table A, fitted to 300 and 400 edits, reproduce all sixteen printed hazard ratios of the interactions.", "problem": "In the S8 File, columns 3 and 4 are marked ``Covariates Yes'', and the note says that this includes the influence variable. The commands in the replication files for these two columns do not contain that variable, and the printed entries reproduce without it and not with it. In S9 Table A, the note repeats this definition, and the row ``Covariates'' marks the first column ``Yes'' and the second ``No''. The two commands have the same terms: the interactions of ``Positive'' with the moderators, the main effects of the moderators and indicators for study wave. Neither command adds the date of entry, the hour of entry or Senate class, and the columns differ only in the studies included. The note to S6 Table D defines a marker of four stars that no entry of that table carries. Since Table D is labelled as unadjusted and its command agrees with the label, this definition is superfluous and not false.", "what_survives": "The reproduced hazard ratios are unchanged, and only the notes and the flag change. The interaction between citation and an upcoming election that the main text reports as 0.32 (p.~5) is the entry 0.323 of the S8 File, fitted without the influence variable; with that variable the interaction is 0.294 with $p<0.001$."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E009", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The heading S11 File, Ordinary Least Squares Robustness Check, a paragraph of five lines, and the title Table A: OLS Estimates on Survival at 6 Hours and 12 Hours. A red rectangle marks the words was removed by that point in time in the third line of the paragraph.", "caption": "S11 File, first paragraph, on the first page of the supplement as rendered from the published Word file. The words “was removed by that point in time” are marked; the title of Table A below them says “Survival”.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T9", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T9"}], "headline": "The outcome in Tables A and B of S11 File is survival beyond the stated time, not removal by it: read “survived beyond that point in time”.", "id": "P041-E010", "location": "S11 File, first paragraph, second sentence; cited from p. 4 (PDF p. 4), “further robustness checks are presented in Tables A and B in S11 File”", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P041-T9"], "text": {"as_printed": "These results, presented in Tables A and B estimate the likelihood that an edit \\hl{was removed by} that point in time.", "corrected": "These results, presented in Tables A and B\\hl{,} estimate the likelihood that an edit \\hl{survived beyond} that point in time.", "evidence": "The evidence is an exact recalculation from the replication data. Let $H_i$ be the duration of fact $i$ in hours and $S_i(h)=\\mathbf{1}\\{H_i>h\\}$. With the outcome $S_i(h)$, all twelve printed slopes of the treatment indicators are reproduced to three decimals. For example, the six-hour slope for positive facts in Studies 1, 2 and 4 is $17/150$, printed as 0.113. The removal indicator is $R_i(h)=1-S_i(h)$. Since each regression has an intercept, I have\n\\[\n\\widehat\\beta_R=e_0-\\widehat\\beta_S,\n\\]\nfor the two vectors of coefficients, where $e_0$ is the coordinate vector of the intercept. Every slope changes sign ($-17/150$ in the example), the residuals change sign, and the robust covariance matrix is the same.", "problem": "The two tables are titled ``OLS Estimates on Survival''. They regress an indicator that the fact was still present after 6, 12 or 24 hours or 7 days, and the printed slopes have the signs of that outcome. The sentence calls the outcome removal, for which every slope would have the opposite sign. The paragraph opens by naming survival, and the sentence that follows interprets the slopes correctly. The intended meaning can be recovered from them. The Harvard Dataverse copy (version 1.0) of the do-file codes the four outcomes as survival. The Yale Dataverse copy (version 2.0) codes them as removal by the stated time, and returns every slope with the sign opposite to the printed one.", "what_survives": "All twelve printed slopes and their interpretation stand, and no coefficient changes. The Cox models, the Kaplan--Meier curves and Fig~1 use the duration and the failure indicator, not these four outcomes. The Yale copy of the do-file reproduces the printed signs once the coding of lines 147--154 of the Harvard copy is restored. That coding sets each of \\texttt{hour6}, \\texttt{hour12}, \\texttt{hour24} and \\texttt{day7} to 1, and replaces it by 0 when \\texttt{hours\\_alive} is at or below the stated time."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E010", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Table A, OLS Estimates on Survival at 6 Hours and 12 Hours, with six columns. In the second column the coefficient for Cited is 0.320 with three stars, and a red rectangle marks the standard error (0.645) printed below it. The corresponding 12-hour entry in the fifth column is 0.370 with three stars and standard error (0.065).", "caption": "S11 File, Table A, on the first page of the supplement as rendered from the published Word file. The standard error (0.645) of the six-hour coefficient for “Cited” is marked; the entry to its right, at 12 hours, is (0.065).", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E011.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T7", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T7"}], "headline": "The standard error of the citation coefficient at six hours in Table A of S11 File is 0.064, not 0.645.", "id": "P041-E011", "location": "S11 File, Table A, column 2 (6 hrs; $N=200$; Studies 1,4), standard error of “Cited”; cited from p. 4 (PDF p. 4)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P041-T7"], "text": {"as_printed": "\\begin{audittable}{lr}{Table A: OLS Estimates on Survival at 6 Hours and 12 Hours}\nCoefficient: & 6 hrs \\\\\nCited & \\\\\nCoefficient: & 0.320*** \\\\\nStandard error: & (\\hl{0.645}) \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lr}{Table A: OLS Estimates on Survival at 6 Hours and 12 Hours}\nCoefficient: & 6 hrs \\\\\nCited & \\\\\nCoefficient: & 0.320*** \\\\\nStandard error: & (\\hl{0.064}) \\\\\n\\end{audittable}", "evidence": "The evidence is a recalculation from the replication data in exact arithmetic. I regress the indicator of survival beyond six hours on an intercept, the citation indicator and an indicator for Study~4, in Studies 1 and 4. The regression has $n=200$ and $k=3$, and its citation coefficient is $8/25=0.320$. Each of the four cells of study by citation holds 50 facts, and the sum of squared residuals is $2047/50$. The HC1 variance of the citation coefficient is\n\\[\n\\frac{200}{197}\\cdot\\frac{1}{10000}\\cdot\\frac{2047}{50}=\\frac{2047}{492500},\n\\]\nand taking the square root, I find the standard error $\\sqrt{2047/492500}=0.0644697\\ldots$, which rounds to 0.064. The $t$ statistic is then 4.96, with $p\\approx1.5\\times10^{-6}$ on 197 degrees of freedom, in agreement with the three stars.", "problem": "The printed standard error is about ten times too large. A coefficient of 0.320 with a standard error of 0.645 has $t=64/129\\approx0.496$ and a two-sided $p$ of about 0.62, which contradicts the three stars printed beside the coefficient. The standard errors of the same coefficient at 12 hours, 24 hours and 7 days are printed as 0.065, 0.062 and 0.053.", "what_survives": "The coefficient 0.320 and its three stars stand. The recalculated standard errors at 12 hours, 24 hours and 7 days match their printed cells, and no other calculation uses this cell."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E011", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The top of a regression table titled 'Table B: Covariate Adjusted Cox Regression Pooled Estimates'. Under 'Positive', the coefficient 0.670** with standard error (0.095); under 'Cited', the coefficient 0.352*** with standard error (0.055). A red box marks 0.670**.", "caption": "S6 File, Table B, in a rendering of the published Word file. The marked entry, for “Positive”, carries two stars.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E012.png"}, {"alt": "A regression table titled 'Table E: Study by Study Cox Regression Estimates, with Covariate Adjustment', with five columns for Studies 1, 2, 4, 5 (1st Half) and 5 (2nd Half). Row Positive: 0.532***, 0.574, 0.753, 0.387***, 0.274*, with standard errors (0.108), (0.220), (0.200), (0.139), (0.162). Row Cited: 0.455***, blank, 0.239***, 0.480*, blank, with standard errors (0.090), (0.068), (0.174). Red boxes mark 0.274* in the last column and 0.480* in the fourth column.", "caption": "S6 File, Table E, in a rendering of the published Word file. The marked entries are “Positive” for Study 5 (2nd Half) and “Cited” for Study 5 (1st Half), each with one star. The first line of the image is the end of the note to Table D.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E012-b.png"}, {"alt": "Five rows of a regression table with two columns of entries. Under 'Positive X NE Region', the coefficients 0.316*** and 0.472* with standard errors (0.149) and (0.192). A red box marks 0.316***.", "caption": "S9 File, Table A, rows around “Positive X NE Region”, in a rendering of the published Word file. The marked entry in the first column carries three stars.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E012-c.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T8", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T8"}], "headline": "Four significance markers are one level off: “Positive” in S6 Table B should carry ***, the two entries for Study 5 in S6 Table E should carry **, and “Positive X NE Region” in the first column of S9 Table A should carry ** in place of ***.", "id": "P041-E012", "location": "S6 File, Table B (row “Positive”) and Table E (columns for Study 5); S9 File, Table A (row “Positive X NE Region”, first column). The supplementary files are Word documents without page numbers; they are listed on p. 6 (PDF p. 6) of the article.", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P041-T8"], "text": {"as_printed": "S6 File, Table B, ``Positive'': 0.670\\hl{**} (0.095).\n\nS6 File, Table E, ``Cited'', Study 5 (1st Half): 0.480\\hl{*} (0.174); ``Positive'', Study 5 (2nd Half): 0.274\\hl{*} (0.162).\n\nS9 File, Table A, ``Positive X NE Region'', first column: 0.316\\hl{***} (0.149).\n\nNote to each table: ``*Significant at the 10\\% level; **Significant at the 5\\% level; ***Significant at the 1\\% level''.", "corrected": "S6 File, Table B, ``Positive'': 0.670\\hl{***} (0.095).\n\nS6 File, Table E, ``Cited'', Study 5 (1st Half): 0.480\\hl{**} (0.174); ``Positive'', Study 5 (2nd Half): 0.274\\hl{**} (0.162).\n\nS9 File, Table A, ``Positive X NE Region'', first column: 0.316\\hl{**} (0.149).", "evidence": "The evidence is a calculation from the printed numbers and a recalculation from the replication files. For a printed hazard ratio $\\widehat{H}$ with standard error $s$, the Wald statistic for the log hazard ratio and its two-sided $p$-value are\n\\[\nz=\\frac{\\log \\widehat{H}}{s/\\widehat{H}},\\qquad p=2\\,\\Phi(-|z|),\n\\]\nwhere $\\Phi$ is the standard normal distribution function. The table below lists $z$ and $p$ at the printed values, bounds on $p$ over all hazard ratios and standard errors that round to the printed values, and the $p$-value when the same model is fitted to the replication files. In every row these bounds lie on one side of the thresholds 0.01 and 0.05, so that rounding cannot account for the printed marker.\n\n\\begin{audittable}{lllllll}{The four entries: the printed hazard ratio with its marker and standard error, the Wald statistic and $p$-value that these give, and the marker that follows from the note of the tables. ``S6 B'' is Table B of the S6 File, and ``5 (1st Half)'' is the first half of Study 5; the entry of S9 Table A is in its first column.}\nEntry & Printed & $z$ & $p$ & Bounds on $p$ & Refit $p$ & Marker \\\\\nS6 B, Positive & 0.670** (0.095) & $-2.82$ & 0.0047 & 0.0044--0.0051 & 0.0045 & *** \\\\\nS6 E, Cited, 5 (1st Half) & 0.480* (0.174) & $-2.02$ & 0.0429 & 0.0418--0.0440 & 0.0434 & ** \\\\\nS6 E, Positive, 5 (2nd Half) & 0.274* (0.162) & $-2.19$ & 0.0285 & 0.0276--0.0296 & 0.0285 & ** \\\\\nS9 A, Positive X NE Region & 0.316*** (0.149) & $-2.44$ & 0.0146 & 0.0139--0.0152 & 0.0144 & ** \\\\\n\\end{audittable}", "problem": "The tables print hazard ratios with standard errors on the scale of the hazard ratio, and their notes assign one star to the 10\\% level, two to the 5\\% level and three to the 1\\% level. For the four entries above, the printed marker is not the one that the printed hazard ratio and standard error imply. Three markers understate the level of significance. The fourth, in S9 Table A, claims significance at 1\\% for an interaction that is significant at 5\\%.", "what_survives": "The four estimates and their standard errors are unchanged. The joint tests 0.321 and 0.221 of S9 Table A are also unchanged. The correction leaves in place the statement of the main text that there is no systematic evidence of moderation by covariates. The interaction with the Northeast region remains significant at 5\\%."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E012", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last four lines of the paragraph and the title of Table A. Red boxes mark 'using only the first' at the end of one line and 'randomly-assigned fact that each worker saw' at the start of the next.", "caption": "S12 File, paragraph introducing Table A, from a rendering of the published Word file (the supplement has no fixed page numbers). The marked words describe the restricted sample.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E013.png"}, {"alt": "Crop of Table A, 'Regression Analysis of Mechanical Turk Coding', and its note. Red boxes mark the three column heads 'First Fact', under Positive-Negative (3.26, standard error 0.23, N 112), Relevance (0.03, 0.25, 113) and Favorability (2.08, 0.24, 113), and the words 'column is only for the first fact seen' in the note.", "caption": "S12 File, Table A and its note, from the same rendering. The marked column heads and the marked words of the note label the restricted columns.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E013-b.png"}, {"alt": "Listing of lines 49 to 53 of the Stata program for the Mechanical Turk data. A red box marks 'drop if inputround==\"test\"' in line 49. Lines 51 to 53 read 'sort workerid submittime', 'bysort workerid: egen submitorder=seq()' and 'gen view\\_first=submitorder==1'.", "caption": "Replication program Kalla_Aronow_PLOS_2015_mturk.do, lines 49–53, typeset from the file. The marked command removes the quality-control items before the submissions of each worker are ranked.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E013-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The “First Fact” columns of S12 Table A use each worker’s first substantive fact after the quality-control items have been dropped; seven of the 113 workers had already completed a quality-control item.", "id": "P041-E013", "location": "S12 File, Results, paragraph introducing Table A; Table A, “First Fact” columns and note (Word supplement, no fixed pagination); replication program Kalla_Aronow_PLOS_2015_mturk.do, lines 49–53", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "As an additional robustness check designed to guard against any desirability bias and learning that may occur as Mechanical Turk workers rate up to 50 of these facts, we produce the same regression \\hl{using only the first randomly-assigned fact that each worker saw}. The results are substantively the same.\n\nTable A, ``\\hl{First Fact}'' columns: Positive-Negative, $3.26$ $(0.23)$, $N=112$; Relevance, $0.03$ $(0.25)$, $N=113$; Favorability, $2.08$ $(0.24)$, $N=113$.\n\nTable A, note: For each question, the first column reports the regression across all facts and the second \\hl{column is only for the first fact seen}.\n\nReplication program, lines 49 and 51--53:\n\n\\texttt{\\hl{drop if inputround==\"test\"}}\n\n\\texttt{sort workerid submittime}\n\n\\texttt{bysort workerid: egen submitorder=seq()}\n\n\\texttt{gen view\\_first=submitorder==1}", "corrected": "Either keep the printed numbers, head the restricted columns \\hl{``First substantive response''}, and replace the description of the sample and the note to the table by the following text.\n\nWe repeat the regressions using \\hl{each worker's first submitted substantive fact, after removing quality-control items}. Seven workers had previously completed a quality-control item; the restriction therefore avoids prior substantive responses but \\hl{does not exclude prior quality-control exposure}.\n\nOr restrict the sample to the workers whose first recorded assignment, quality-control items included, was substantive, and replace the three restricted columns by \\hl{$3.34$ $(0.24)$, $N=105$}; \\hl{$-0.02$ $(0.23)$, $N=106$}; and \\hl{$2.13$ $(0.25)$, $N=106$}. The text should then say that the order is that of accepted and submitted assignments, since no log of views exists.", "evidence": "The evidence is an exact counterexample from the replication data, with a recalculation. One worker submitted a quality-control item at 01:51:28 GMT on 9 December 2014 and accepted the selected fact, a Study 4 fact about Ed Markey, at 01:51:35, seven seconds later. For the seven workers the gaps between the submission of the quality-control item and the acceptance of the selected fact are 7, 3, 1, 1, 13, 6620 and 30 seconds. All six printed regressions of Table A reproduce at the printed precision. The slope of a regression on a binary indicator is the difference of the two group means. Without the seven workers, I find that the three slopes are $1535/459$, $-7/351$ and $2987/1404$.\n\n\\begin{audittable}{lrrrr}{Table A, restricted columns: coefficient (standard error clustered by worker) and $N$.}\nQuestion & As printed & $N$ & Without the seven workers & $N$ \\\\\nPositive-Negative & 3.26 (0.23) & 112 & 3.34 (0.24) & 105 \\\\\nRelevance & 0.03 (0.25) & 113 & $-0.02$ (0.23) & 106 \\\\\nFavorability & 2.08 (0.24) & 113 & 2.13 (0.25) & 106\n\\end{audittable}\n\nThe times are those of accepted and submitted assignments and do not show what a worker viewed. One retained worker accepted four substantive facts in the same second, so the order of viewing cannot be established for every worker.", "problem": "S12 File counts the 49 quality-control items among the facts that a worker could be shown: facts were sampled ``from the set of 551 facts included in the five studies, plus an additional 49 facts that were seeded as quality control'', up to 50 per worker. The program drops the quality-control items in line 49 and ranks the submissions of each worker afterwards, in lines 51--53. The variable \\texttt{view\\_first} therefore marks each worker's first submitted substantive fact. Seven of the 113 workers had completed a quality-control item before that fact. For these seven the selected fact is not the first that the worker saw, and the restriction does not exclude earlier exposure to an item, which is the purpose stated for it.", "what_survives": "The six printed regressions are correct for the samples that the program selects. Without the seven workers the pattern is the same: large positive differences for the positive-negative question and for favorability, and a difference near zero for relevance. The columns for all responses and the Wikipedia analyses do not use this restriction."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E013", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "A box plot with two groups on the horizontal axis and seven tick labels, set at an angle, on the vertical axis. Red boxes mark the two group labels, which read 'Nega2ve' and 'Posi2ve', and the top tick label, in which a percent sign is printed over the gap between 'Very' and 'Pos'. From top to bottom, the other tick labels are Pos, Somewhat Pos, Neither, Somewhat Neg, Neg and Very Neg, and each label of two words has a percent sign in place of its space.", "caption": "Figure A of the S12 File, in an earlier rendering of the embedded image. The marked group labels read “Nega2ve” and “Posi2ve”, and in the marked tick label a percent sign stands between “Very” and “Pos”. How the figure appears in Word with its original fonts was not checked.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E014.png"}], "formal_release": null, "formal_statements": [], "headline": "The labels stored in the three figures of the S12 File are malformed: “Negative” and “Positive” are stored with a digit in place of the letters “ti”, and the tick labels have a percent sign in place of a space.", "id": "P041-E014", "location": "S12 File, Figures A–C (images embedded in a Word document without page numbers); the S12 File is listed on p. 6 (PDF p. 6) of the article", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lll}{Text stored in the three figures of the S12 File.}\nFigure & Group labels & Tick labels of two words \\\\\nA & Nega\\hl{2}ve, Posi\\hl{2}ve & Very\\hl{\\%}Neg, Somewhat\\hl{\\%}Neg, Somewhat\\hl{\\%}Pos, Very\\hl{\\%}Pos \\\\\nB & Nega\\hl{3}ve, Posi\\hl{3}ve & Very\\hl{\\%}Irr, Somewhat\\hl{\\%}Irr, Somewhat\\hl{\\%}Rel, Very\\hl{\\%}Rel \\\\\nC & Nega\\hl{5}ve, Posi\\hl{5}ve & Very\\hl{\\%}Unfav, Somewhat\\hl{\\%}Unfav, Somewhat\\hl{\\%}Fav, Very\\hl{\\%}Fav \\\\\n\\end{audittable}", "corrected": "Figures A--C are drawn again from the same data, with the group labels ``\\hl{Negative}'' and ``\\hl{Positive}'' and with spaces in the tick labels: ``Very Neg'', ``Somewhat Neg'', ``Somewhat Pos'' and ``Very Pos'' in Figure A, ``Very Irr'', ``Somewhat Irr'', ``Somewhat Rel'' and ``Very Rel'' in Figure B, and ``Very Unfav'', ``Somewhat Unfav'', ``Somewhat Fav'' and ``Very Fav'' in Figure C.", "evidence": "The evidence is a check of the source file. I read the text that is stored in the three images embedded in the file, which consists of ordinary characters in the font Calibri. I did not check how the labels appear in Word with the original fonts.", "problem": "Each figure of the S12 File is an image embedded in the Word document, and the image stores its labels as text. In all three figures this text is malformed. The commands in the replication files that draw the figures contain the intended labels: ``Negative'' and ``Positive'' for the two groups, and tick labels such as ``Very Neg'' and ``Somewhat Pos''.", "what_survives": "No number depends on the labels, and the two groups and the response scale keep their intended meaning."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E014", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The three-line caption of Figure 1. A red rectangle marks the word fare in the sentence, Results show that among active senators, negative facts are fare more likely to be removed more quickly than positive facts.", "caption": "Fig 1 caption, p. 4 (PDF p. 4), with the word “fare” marked in the last sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E015.png"}, {"alt": "Four lines of text. A red rectangle marks the word pages in the sentence, We created 5 Wikipedia pages and made around 20 political, non-Senate edits using these accounts before beginning the experiment.", "caption": "S3 File, “Description of implementation”, last paragraph, on the single page of the supplement as rendered from the published Word file. The word “pages” is marked; the next line continues “using these accounts”.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E015-S3.png"}, {"alt": "Three lines of bold text beginning, S8 File. Investigates whether Senate class is a moderator of the effect if positive and cited edits. A red rectangle marks the word if at the end of the first line.", "caption": "S8 File, opening sentence, on the single page of the supplement as rendered from the published Word file, with the word “if” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E015-S8.png"}, {"alt": "Eight lines of the codebook in typewriter type, from committeechair to treatment_pre_order. Red rectangles mark the words page count in the entry, pagechar: Senator's Wikipedia page count before experiment began, and the words Study 5 in the entry, treatment_pre_order: Order in which edits were made in Study 5, pre-election.", "caption": "Codebook of the replication files (PDF p. 1), entries pagechar and treatment_pre_order, with “page count” and “Study 5” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E015-codebook.png"}], "formal_release": null, "formal_statements": [], "headline": "Five slips of a word or label: read “far” for “fare”, “accounts” for “pages” and “effect of” for “effect if”, and in the codebook read Study 4 for Study 5 in the entry for the pre-election order and “character count” for “count” in the entry for page length.", "id": "P041-E015", "location": "Fig 1 caption, p. 4 (PDF p. 4), last sentence; S3 File, “Description of implementation”, last paragraph; S8 File, opening sentence; replication codebook, entries treatment_pre_order and pagechar", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{itemize}\n\\item Fig~1 caption: ``Results show that among active senators, negative facts are \\hl{fare} more likely to be removed more quickly than positive facts.''\n\\item S3 File: ``We created 5 Wikipedia \\hl{pages} and made around 20 political, non-Senate edits using these accounts before beginning the experiment.''\n\\item S8 File: ``Investigates whether Senate class is a moderator of the effect \\hl{if} positive and cited edits.''\n\\item Codebook: ``treatment\\_pre\\_order: Order in which edits were made in \\hl{Study 5}, pre-election.''\n\\item Codebook: ``pagechar: Senator's Wikipedia \\hl{page count} before experiment began.''\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Fig~1 caption: ``Results show that among active senators, negative facts are \\hl{far} more likely to be removed more quickly than positive facts.''\n\\item S3 File: ``We created 5 Wikipedia \\hl{accounts} and made around 20 political, non-Senate edits using these accounts before beginning the experiment.''\n\\item S8 File: ``Investigates whether Senate class is a moderator of the effect \\hl{of} positive and cited edits.''\n\\item Codebook: ``treatment\\_pre\\_order: Order in which edits were made in \\hl{Study 4}, pre-election \\hl{(also carried into the corresponding Study 5 records)}.''\n\\item Codebook: ``pagechar: Senator's Wikipedia page \\hl{character} count before \\hl{the} experiment began.''\n\\end{itemize}", "evidence": "The evidence is a check of the page, the supplements and the codebook, and a recalculation from the replication data. In the data, \\texttt{treatment\\_pre\\_order} is missing in Studies 1 to 3, is filled for all 100 Study~4 records, and is copied in the same order into the matching 100 Study~5 records. The values of \\texttt{pagechar} range from 26686 to 631299.", "problem": "The caption and S8 File contain misprints. The sentence of S3 File concerns the accounts created to reach autoconfirmed status and itself continues ``using these accounts''. S2 File describes the same step as creating five Wikipedia accounts. The pre-election study is Study~4 (30--31 October 2014), and Study~5 followed the election (6--7 November 2014). The variable \\texttt{pagechar} is the length of the page in characters, as p.~4 and the notes to the supplementary tables say.", "what_survives": "These five wording changes do not alter any number or fitted result."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E015", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The three Kaplan-Meier panels of Figure 1, each with a horizontal axis from 0 to 250 hours and a vertical axis from 0 to 1, and the legend below them. A red rectangle encloses the lower right corner of each panel, where survival zero meets the 250-hour boundary. No curve or segment lies at survival zero inside the rectangles.", "caption": "Fig 1, p. 4 (PDF p. 4), with the lower right corner of each panel marked. The published panels stop at 250 hours and carry no segment at survival zero there.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E017.png"}, {"alt": "Two renders of Figure 1, one above the other, each with three panels of survival curves against hours from 0 to 250. In the upper render the horizontal axis runs a little past 250, and a short red segment at survival zero sits to the right of the 250 tick in each panel, circled in blue. In the lower render the axis stops at 250 and there is no segment at survival zero.", "caption": "The plotting script run from the replication files. (a) As supplied, with R’s default axes: a segment at survival zero lies to the right of 250 hours in each panel (circled). (b) With the extension drawn only when a curve ends before 250 hours and with xaxs=\"i\": no such segment.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E017-default-axes.png"}], "formal_release": null, "formal_statements": [], "headline": "In the plotting script, each survival curve should be extended to the 250-hour boundary only when the curve ends before 250 hours, and at its last survival level; as written, the loops draw a segment at survival zero beyond 250 hours, which the published Fig 1 does not show.", "id": "P041-E017", "location": "Fig 1, p. 4 (PDF p. 4), right-hand edge of the three panels; R plotting script of the replication files, lines 28 and 34–37, with the same loop at lines 43–46 and 51–54 and in the loops for Figure A of S6 File", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{maxtime <- 250}\n\n\\texttt{for (i in 1:length(fit1\\$strata)) \\{ \\#number of curves}\n\n\\texttt{temp <- fit1[i]}\n\n\\texttt{\\hl{lines}(c(max(temp\\$time), maxtime), rep(min(temp\\$surv),2),}\n\n\\texttt{col=plot\\_colors[i],lty=plot\\_lty[i])}\n\n\\texttt{\\}}", "corrected": "\\texttt{maxtime <- 250}\n\n\\texttt{for (i in 1:length(fit1\\$strata)) \\{ \\#number of curves}\n\n\\texttt{temp <- fit1[i]}\n\n\\texttt{\\hl{if (max(temp\\$time) < maxtime)}}\n\n\\texttt{lines(c(max(temp\\$time), maxtime), rep(\\hl{tail(temp\\$surv,1)},2),}\n\n\\texttt{col=plot\\_colors[i],lty=plot\\_lty[i])}\n\n\\texttt{\\}}\n\nThe same change applies to the loops for \\texttt{fit2} and \\texttt{fit3} and to the loops that draw Figure~A of S6 File. To match the boundary of the published figure, each \\texttt{plot} call also takes \\texttt{\\hl{xaxs=\"i\"}}.", "evidence": "The evidence is a recalculation from the replication files and a check of the page. I ran the supplied plotting code with R's default axes and recorded the coordinates of each extension. For all eight curves the extension runs from 3463.58037 hours to 250 hours at survival zero. The render shows a zero-level segment to the right of 250 hours in each panel. A second render, with the extension drawn only when \\texttt{max(temp\\$time) < maxtime} and with \\texttt{xaxs=\"i\"}, shows none. On the published page, viewed at 220 dpi, there is no mark at survival zero at the right-hand edge of any panel.", "problem": "After each panel is plotted, a loop extends every curve to the display limit of 250 hours. The extension is meant for a curve whose fit stops before 250 hours, but the loop has no such condition. It uses the final fitted survival level, which need not be the level at 250 hours. For the eight curves of Fig~1 the fit ends at 3463.58037 hours with survival zero. This is the time that the script assigns to facts that were never removed (\\fid{P041-N002}). The loop draws a segment at survival zero from 3463.58037 hours back to 250 hours. With R's default axis style, \\texttt{xaxs=\"r\"}, the horizontal range 0 to 250 is widened to $-10$ to 260, and the part of the segment between 250 and 260 hours appears inside each panel. The published Fig~1 stops at 250 hours and shows no such segment.", "what_survives": "The published Fig~1 and the survival curves it shows over 0 to 250 hours are unaffected. Only a new render of the supplied script with default axes shows the extra segments. The same loop, without a condition, draws Figure~A of S6 File."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E017", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the eight-line paragraph that introduces Table A. Red boxes mark 'more positive,' at the end of one line and 'equally relevant, and more favorable at statistically significant levels' at the start of the next.", "caption": "S12 File, paragraph introducing Table A, from a rendering of the published Word file (the supplement has no fixed page numbers). The marked words join “equally relevant” to “at statistically significant levels”.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-E018.png"}], "formal_release": null, "formal_statements": [], "headline": "S12 File detects no difference in mean relevance between positive and negative facts ($-0.03$, standard error $0.07$); the sentence can be read as if equal relevance had been established at a significance level.", "id": "P041-E018", "location": "S12 File, Results, paragraph introducing Table A (Word supplement, no fixed pagination)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Consistent with the box-plots, we see that positive facts were perceived to be more positive, \\hl{equally relevant}, and more favorable \\hl{at statistically significant levels}.", "corrected": "Consistent with the box-plots, positive facts were perceived as more positive and more favorable at statistically significant levels; \\hl{no statistically detectable difference in mean relevance was found}.", "intended_meaning": "The phrase ``at statistically significant levels'' applies to the two differences that Table A shows to be significant, $2.96$ (standard error $0.08$) for the positive-negative question and $2.30$ $(0.09)$ for favorability. For relevance the table shows $-0.03$ $(0.07)$, and the two mean ratings are $5.00$ and $5.03$, so ``equally relevant'' means that no difference in mean relevance was detected. Read in this way the sentence is correct. A test of a zero difference that does not reject is not a test of equivalence, and S12 File reports no equivalence margin or test."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-E018", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "The lower edge of the three panels of Figure 1, with horizontal axes labelled Hours and ticks from 0 to 250, the legend, and the caption. Red rectangles mark the tick label 250 of each panel and the caption title, Fig 1. Kaplan-Meier survival curves across all studies and all conditions.", "caption": "Fig 1, p. 4 (PDF p. 4): the title of the caption and the last tick of each horizontal axis are marked. The panels stop at 250 hours; the removals that the script invents lie at 3463.58 hours, outside the figure.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-N002.png"}], "formal_release": null, "formal_statements": [], "headline": "The R script that draws the Kaplan–Meier figures builds its survival object without an event indicator, and the 257 facts that were never removed are entered as removals at an invented time of 3463.58 hours; line 14 should pass event=!wiki$dead.", "id": "P041-N002", "location": "Fig 1, p. 4 (PDF p. 4), and Figure A of S6 File; R plotting script of the replication files, lines 8–14", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\texttt{wiki\\$dead <- is.na(wiki\\$time\\_alive)}\n\n\\texttt{wiki\\$time\\_alive2 <- wiki\\$time\\_alive}\n\n\\texttt{wiki\\$time\\_alive2[wiki\\$dead] <- max(wiki\\$time\\_alive*2,na.rm=TRUE)}\n\n\\texttt{wiki\\$time\\_alive2[wiki\\$time\\_alive==0] <- 0.1}\n\n\\texttt{wiki\\$hours\\_alive <- wiki\\$time\\_alive2*2.77778e-7}\n\n\\texttt{wiki\\$survdata <- \\hl{Surv(time=wiki\\$hours\\_alive)}\\#,event=dead)}", "corrected": "\\texttt{wiki\\$survdata <- Surv(time=wiki\\$hours\\_alive\\hl{, event=!wiki\\$dead})}\n\nLines 8--13 are unchanged. The indicator is the negation of \\texttt{dead}, since in this script \\texttt{dead} is \\texttt{TRUE} for a fact that was never removed.", "evidence": "The evidence is an exact counterexample and a recalculation from the replication data. I take two records, one removed at raw time 3600000 and one never removed. The script assigns the second record the raw time 7200000, and with the conversion constant $2.77778\\times10^{-7}=138889/500000000000$ the two times are $t_1=1250001/1250000$ and $t_2=2t_1$ hours. The Kaplan--Meier estimator is\n\\[\n\\widehat S(t)=\\prod_{u\\le t,\\ d(u)>0}\\frac{n(u)-d(u)}{n(u)},\n\\]\nwhere $n(u)$ is the number of records with time at least $u$ and $d(u)$ is the number of events at $u$. At $t_1$ I have $n=2$ and $d=1$ whatever the status of the second record, which gives $\\widehat S(t_1)=1/2$. With the statuses $(1,1)$ of the script, $n(t_2)=d(t_2)=1$ and $\\widehat S(t_2)=0$. With the statuses $(1,0)$, no factor is added at $t_2$ and $\\widehat S(t_2)=1/2$.\n\nIn the data, the largest recorded duration is 6234439680 in raw units. The invented time is then $169120135296/48828125=3463.58037086208$ hours, and all 551 statuses are 1. A fact that was never removed has time $3463.58>250$ in the survival object of the script and in the repaired one, and every recorded removal is the same in both. For every $u\\le250$ the two objects then have the same $n(u)$ and $d(u)$. This implies that the Kaplan--Meier products agree on $[0,250]$. In exact arithmetic on the data, the two products agree for all 26 fitted curves, 8 in Fig~1 and 18 in Figure~A of S6 File. The table lists the curves of Fig~1.\n\n\\begin{audittable}{llrrrrr}{Curves of Fig~1, as scripted and with the event indicator repaired. The value at 250 hours is the same in both cases. The last two columns give the value at the invented time of 3463.58 hours.}\nStudies & Condition & $N$ & Never removed & $\\widehat S(250)$ & As scripted & Repaired \\\\\n1 and 4 & Positive, cited & 51 & 16 & $16/51$ & 0 & $16/51$ \\\\\n1 and 4 & Positive, uncited & 49 & 5 & $5/49$ & 0 & $5/49$ \\\\\n1 and 4 & Negative, cited & 49 & 10 & $10/49$ & 0 & $10/49$ \\\\\n1 and 4 & Negative, uncited & 51 & 3 & $1/17$ & 0 & $1/17$ \\\\\n2 & Positive, cited & 50 & 37 & $4/5$ & 0 & $37/50$ \\\\\n2 & Negative, cited & 50 & 30 & $16/25$ & 0 & $3/5$ \\\\\n3 & Positive, cited & 75 & 67 & $68/75$ & 0 & $67/75$ \\\\\n3 & Negative, cited & 76 & 64 & $73/76$ & 0 & $16/19$ \\\\\n\\end{audittable}", "problem": "In R's survival package, \\texttt{Surv} called with a time and no event indicator gives every row status 1, an observed event. The outcome of the paper is the time until a fact is removed. In the data, 257 of the 551 facts were never removed: their \\texttt{time\\_alive} is missing and \\texttt{removed} is coded ``NO''. Line 10 assigns these facts twice the largest recorded duration, that is, 3463.58 hours, and line 14 enters them as removals at that time. The fitted curve of any group that contains such facts then falls to zero at 3463.58 hours, where a Kaplan--Meier curve with these facts censored keeps its last level. The column \\texttt{dead} is the missing-time indicator, not the removal indicator. It is \\texttt{TRUE} for the facts that were never removed, and passing it as \\texttt{event} would invert every status.", "what_survives": "Every Kaplan--Meier value shown in Fig~1 and in Figure~A of S6 File, which stop at 250 hours, is unchanged when only the event indicator is repaired. The Cox regressions, the regression tables and the counts at fixed times come from the Stata do-file and do not use the R object. The do-file sets \\texttt{dead} to 1 for a recorded removal and declares it with \\texttt{stset hours\\_alive, failure(dead)}. The fitted curves beyond 250 hours and any summary of the full fit do change. The repair keeps the time that the script assigns to facts that were never removed. Whether that time matches the actual end of monitoring is a separate question, which this correction does not settle."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-N002", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Four lines of text beginning, We find no systematic evidence that this effect is moderated by covariates. Two red rectangles on the third line mark the words date of entry, and time of entry, and the citation see Table A in S9 File and S10 File.", "caption": "Results, p. 4 (PDF p. 4): the last sentence of the first paragraph, with “date of entry, and time of entry” and the citation “see Table A in S9 File and S10 File” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "The sentence cites Table A of S9 File and S10 File for eight moderators of the valence effect, but the test for Senate class is in S8 File, S10 File is not a moderation analysis, and no test for date or time of entry is reported: cite S8 File and Table A of S9 File, and remove date and time of entry from the list.", "id": "P041-N005", "location": "Results, “Survival of Positive and Negative Facts on Wikipedia”, last sentence of the first paragraph, p. 4 (PDF p. 4)", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We find no systematic evidence that this effect is moderated by covariates (state population, Senate class, character count of page, geographic region, length of incumbency, political party, \\hl{date of entry, and time of entry}; \\hl{see Table A in S9 File and S10 File}), as specified in our registered pre-analysis plan.", "corrected": "We find no systematic evidence that this effect is moderated by covariates (state population, Senate class \\hl{dichotomized as facing reelection in 2014}, character count of page, geographic region, length of incumbency, \\hl{and} political party; \\hl{see S8 File and Table A in S9 File}), as specified in our registered pre-analysis plan.\n\nDate and time of entry may return to the list once their interactions with valence are reported. The covariate called length of incumbency is the year of first election (\\fid{P041-E004}).", "evidence": "The evidence is a check of the page against the twelve supplements and the replication do-files. I write the Cox model with interactions as $h(t\\mid V,Z)=h_0(t)\\exp\\{\\beta V+\\gamma^\\top Z+\\delta^\\top(VZ)\\}$, where $V=1$ for a positive fact and $Z$ is the vector of covariates. Dividing the hazard of a positive fact by the hazard of a negative fact at $Z=z$, I find the hazard ratio $\\exp(\\beta+\\delta^\\top z)$. Moderation is a statement about $\\delta$, and S10 File estimates only $\\gamma$. The joint Wald tests of Table~A in S9 File have $p=0.321$ and $p=0.221$. From the rounded entries of S8 File, the interaction with Senate class has $p\\approx0.075$ without covariates and $p\\approx0.155$ with them.", "problem": "Table~A of S9 File reports interactions of the indicator for positive facts with party, influence, three regions, length of incumbency, log page length and log state population. It has no interaction with Senate class, date of entry or time of entry. S10 File regresses survival on covariates alone, with no valence indicator and no interaction, and therefore cannot test moderation. The one reported test of Senate class as a moderator of the valence effect is in S8 File, which the sentence does not cite. There Senate class is dichotomized as facing reelection in 2014, and the interaction is 1.750* (0.550) without covariates and 1.557 (0.485) with them. No supplement and neither copy of the do-file reports an interaction of valence with date or time of entry. These two variables enter only as additive controls.", "what_survives": "The analyses that are reported do not contradict the statement of no systematic evidence. The joint tests do not reject, and the interaction with Senate class is marginal only without covariates. The cited files do not document three of the eight listed moderators, and the corrected sentence covers six. The pooled valence estimates, the survival curves and the citation results do not depend on this sentence. I could not compare the sentence with the pre-analysis plan, which was not available."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-N005", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the four-line last paragraph of S1 File. Red boxes mark the sentence 'Because the order in which edits were made was randomly assigned, this does not bias our results.', which runs over the last two lines.", "caption": "S1 File, “Description of implementation”, last paragraph, from a rendering of the published Word file (the supplement has no fixed page numbers). The marked sentence is the assurance.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-N011.png"}, {"alt": "Crop of the last five lines of S5 File. Red boxes mark the sentence 'Because the order in which edits were made was randomly assigned, this does not bias our results.', which runs over the last two lines.", "caption": "S5 File, “Description of implementation”, end of the last paragraph, from a rendering of the published Word file. The marked sentence is the same assurance.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-N011-b.png"}], "formal_release": null, "formal_statements": [{"id": "P041-T5", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T5"}, {"id": "P041-T6", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T6"}, {"id": "P041-T13", "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-T13"}], "headline": "Random assignment of the order of the edits does not by itself ensure that speeding up Study 1, and converting the remaining uncited edits of Study 5 to cited ones, leave the comparisons unbiased.", "id": "P041-N011", "location": "S1 File and S5 File, “Description of implementation”, last sentence of each (Word supplements, no fixed pagination); the main text refers to S5 File on p. 3 (PDF p. 3)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P041-T13", "P041-T5", "P041-T6"], "text": {"as_printed": "S1 File: Because other Wikipedia editors became suspicious of this account's uncited edits and threatened to block the account, we sped up the implementation and finished implementing the experiment sooner than anticipated, on 15 July 2014. \\hl{Because the order in which edits were made was randomly assigned, this does not bias our results.}\n\nS5 File: After implementing the first 50 edits on 6 November 2014, other Wikipedia editors became suspicious. We decided to convert all of the uncited edits in the remaining 50 edits to cited edits. Thus the first 50 edits had all four conditions while the last 50 edits only had positive or negative conditions, with all facts cited. \\hl{Because the order in which edits were made was randomly assigned, this does not bias our results.}", "corrected": "S1 File and S5 File, in place of the marked sentence: \\hl{Random assignment of the original order prevents deliberate ordering by treatment. It does not by itself rule out treatment-dependent account responses or effects of the adaptive changes.}\n\nIf the comparisons are read as effects of an edit's own treatment, the text should also state the protocol and the conditions of exposure to which they refer.", "evidence": "The evidence is an exact counterexample to the sufficiency of random order, with a recalculation of two sensitivity checks. I take four pages, two assigned a cited fact (C) and two an uncited fact (U), as in one block of Study 1, and let the 24 orders of insertion be equally likely. Suppose that an uncited edit starts a period in which every later insertion from the account is removed, whatever its own citation status, and that no other removals occur. This response is hypothetical and is not a description of the actual editors. On the planned, slow schedule the period ends before the next insertion, no fact is removed, and the difference $D$ between the removal rates of cited and uncited facts is $0$. On the accelerated schedule, all remaining insertions fall inside the period that the first uncited edit starts. The edit at rank $r$ is then removed if and only if an earlier edit is uncited,\n\\[\nY_r=\\mathbf{1}\\{z_s=\\mathrm{U}\\ \\text{for some}\\ s<r\\},\n\\]\nwhere $z_s$ is the type of the edit at rank $s$. The outcome $Y_r$ does not depend on $z_r$, which implies that the effect of an edit's own citation is zero. The six orders of types are equally likely.\n\n\\begin{audittable}{llrrr}{The six equally likely orders of types on the accelerated schedule.}\nTypes at ranks 1--4 & Removed at ranks 1--4 & Cited rate & Uncited rate & $D$ \\\\\nU U C C & 0 1 1 1 & $1$ & $1/2$ & $1/2$ \\\\\nU C U C & 0 1 1 1 & $1$ & $1/2$ & $1/2$ \\\\\nU C C U & 0 1 1 1 & $1$ & $1/2$ & $1/2$ \\\\\nC U U C & 0 0 1 1 & $1/2$ & $1/2$ & $0$ \\\\\nC U C U & 0 0 1 1 & $1/2$ & $1/2$ & $0$ \\\\\nC C U U & 0 0 0 1 & $0$ & $1/2$ & $-1/2$\n\\end{audittable}\n\nAveraging over the six orders, I obtain\n\\[\n\\mathbb{E}[D]=\\frac{1}{6}\\Bigl(\\frac12+\\frac12+\\frac12+0+0-\\frac12\\Bigr)=\\frac16 ,\n\\]\nagainst $0$ on the slow schedule, although the order is uniformly random and the citation has no effect of its own. With 100 pages, 50 cited and 50 uncited, the same construction has expected removal rates $50/51$ and $49/50$, and I find that $\\mathbb{E}[D]=1/2550$. This value shows that the expectation need not vanish. It is not an estimate of any bias in the paper. If the pattern of removals is fixed over the calendar and does not depend on the treatments, the same enumeration gives $\\mathbb{E}[D]=0$.\n\nThe replication data show removals concentrated in time: in Study 1, 55 of the 93 removals fall between 19:28 and 20:39 on 15 July 2014, and all 36 facts inserted on that day were removed on that day. For the pooled comparison of positive and negative facts (Studies 1, 2 and 4), I re-estimated the Cox model in R. With robust standard errors computed without clustering, as in the replication program, the hazard ratio is $0.629$ and $p=0.0013$. With standard errors clustered by senator, I find $p=0.0008$. Without the 36 Study 1 edits of the last day, the hazard ratio is $0.606$ and $p=0.0016$.", "problem": "In each supplement, ``this'' refers to a change made after editors reacted to the uncited edits: the remaining edits of Study 1 were made faster, and the remaining uncited edits of Study 5 were made cited. Random order balances, in expectation, any pattern over the calendar that does not depend on the treatments. It does not cover the case that the supplements describe, in which the treatment of earlier edits (an uncited fact from the same account) changes how later edits are received or when they are made. In that case the comparison under the protocol that was carried out need not equal the comparison under the planned protocol, or the effect of an edit's own treatment. Random order gives no guarantee here. To keep the assurance, one needs a further assumption that excludes such effects of earlier edits on later ones. An example is the assumption that the removal of an edit does not depend on the treatments and timing of the earlier edits from the account. The supplements state no such assumption, and the replication data do not identify the editors who removed the facts. This is a limit on the assurance; it does not show that any estimate is biased.", "what_survives": "The counts, the survival curves and the fitted coefficients are unchanged, and no numerical correction follows. The pooled difference between positive and negative facts remains under clustering by senator and without the last day of Study 1. These checks do not address an effect of earlier edits on later ones within an account. Study 1 enters both pooled analyses of the main text. Study 5 enters supplementary analyses and is excluded from the main-text pooled estimates. What is withdrawn is the unconditional assurance for Studies 1 and 5."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-N011", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "The top of Table A, Pooled Cox Regression Estimates. Red rectangles mark the word Coefficient in the header row and beside the entries 0.629 with three stars, for Positive, and 0.429 with three stars, for Cited. The standard errors 0.091 and 0.069 are printed in parentheses below.", "caption": "S6 File, Table A, on the first page of the supplement as rendered from the published Word file. The label “Coefficient:” is marked in the header and in the rows “Positive” and “Cited”; the entries 0.629 and 0.429 are hazard ratios.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-N013.png"}], "formal_release": null, "formal_statements": [], "headline": "The entries labelled “Coefficient:” in the Cox tables of S6–S10 Files are hazard ratios, and their standard errors are on the hazard-ratio scale.", "id": "P041-N013", "location": "S6–S10 Files, row label “Coefficient:” in every Cox table (for example S6 File, Table A); cited from p. 4 (PDF p. 4), “hazard ratio = 1.59”", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{lll}{Table A: Pooled Cox Regression Estimates}\n\\hl{Coefficient:} & & \\\\\nPositive & & \\\\\n\\hl{Coefficient:} & 0.629*** & \\\\\nStandard error: & (0.091) & \\\\\nCited & & \\\\\n\\hl{Coefficient:} & & 0.429*** \\\\\nStandard error: & & (0.069) \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lll}{Table A: Pooled Cox Regression Estimates}\n\\hl{Hazard ratio:} & & \\\\\nPositive & & \\\\\n\\hl{Hazard ratio:} & 0.629*** & \\\\\n\\hl{Standard error of the hazard ratio:} & (0.091) & \\\\\nCited & & \\\\\n\\hl{Hazard ratio:} & & 0.429*** \\\\\n\\hl{Standard error of the hazard ratio:} & & (0.069) \\\\\n\\end{audittable}\n\nThe same two labels replace ``Coefficient:'' and ``Standard error:'' in the other Cox tables of S6--S10 Files.", "intended_meaning": "The entries are exponentiated Cox coefficients, that is, removal hazard ratios for the named condition against its reference, and the standard errors are on the same scale. For the row ``Positive'' of Table~A in S6 File, the fitted coefficient on the log-hazard scale is $-0.4637$. Its exponential is 0.629, and 0.629 times the robust standard error of the coefficient is 0.091. The main text prints the reciprocals: $1.59\\approx1/0.629$ on p.~4 for negative against positive facts, and $2.33\\approx1/0.429$ on p.~5 for uncited against cited facts. The supplements do not use the words ``hazard ratio'', but the surrounding text and these reciprocals identify the intended scale."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-N013", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of two paragraphs of S12 File. Red boxes mark 'fairly strong' at the end of one line and 'agreement' at the start of the next, in 'When we calculate the inter-rater reliability across the facts and five raters, we find fairly strong agreement'. The following sentences give the kappas 0.20 and 0.56, 0.04 and 0.08, 0.13 and 0.38.", "caption": "S12 File, Results, first and second paragraphs, from a rendering of the published Word file (the supplement has no fixed page numbers). The marked words summarize the six kappas that follow.", "url": "https://pmaronow.github.io/llm-audit/works/P041/exhibits/P041-N015.png"}], "formal_release": null, "formal_statements": [], "headline": "The six Fleiss kappas reported in S12 File do not show “fairly strong agreement”: the two for relevance are 0.04 and 0.08, and the largest of the six is 0.56.", "id": "P041-N015", "location": "S12 File, Results, second paragraph (Word supplement, no fixed pagination)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "When we calculate the inter-rater reliability across the facts and five raters, we find \\hl{fairly strong agreement}. On the positive-negative question, the Fleiss's kappa across all seven categories is 0.20 and 0.56 when collapsed to three categories. On the relevance question, the Fleiss's kappa across all seven categories is 0.04 and 0.08 when collapsed to three categories. On the favorability question, the Fleiss's kappa across all seven categories is 0.13 and 0.38 when collapsed to three categories.", "corrected": "\\hl{Agreement varied by question and category resolution: it was greatest for valence after collapsing to three categories, while relevance showed little exact}-\\hl{category agreement beyond the marginal}-\\hl{frequency chance benchmark.} On the positive-negative question, the Fleiss's kappa across all seven categories is 0.20 and 0.56 when collapsed to three categories. On the relevance question, the Fleiss's kappa across all seven categories is 0.04 and 0.08 when collapsed to three categories. On the favorability question, the Fleiss's kappa across all seven categories is 0.13 and 0.38 when collapsed to three categories.", "evidence": "The evidence is a recalculation from the replication data, in exact arithmetic. There are 551 facts with five ratings each, and a missing rating is set to the midpoint 4, as in the replication program. Let $P_o$ be the share of agreeing pairs of raters among the $5510$ pairs and $P_e=\\sum_j p_j^2$, where $p_j$ is the pooled share of category $j$. The kappa is $\\kappa=(P_o-P_e)/(1-P_e)$. For relevance with seven categories I obtain\n\\[\nP_o=\\frac{1391}{5510},\\qquad P_e=\\frac{1704971}{7590025},\\qquad \\kappa=\\frac{422263}{11770108}\\approx0.0359 ,\n\\]\nand with three categories\n\\[\nP_o=\\frac{3207}{5510},\\qquad P_e=\\frac{4154083}{7590025},\\qquad \\kappa=\\frac{527119}{6871884}\\approx0.0767 .\n\\]\nThe six kappas are $0.202$ and $0.559$ for the positive-negative question, $0.036$ and $0.077$ for relevance, and $0.134$ and $0.381$ for favorability, and each rounds to the printed value.", "problem": "The sentence summarizes the six kappas that follow it. A Fleiss kappa measures how far the raters agree on the exact category beyond the agreement expected from the pooled frequencies of the categories. For relevance the two values are $0.036$ and $0.077$: the raters agree in $25.2\\%$ of pairs against a benchmark of $22.5\\%$ with seven categories, and in $58.2\\%$ against $54.7\\%$ with three. As a summary that includes relevance, ``fairly strong'' overstates these values. The paper sets no threshold for the phrase, and the finding concerns the wording; the six numbers are correct.", "what_survives": "The six kappas stand. The mean ratings separate the two kinds of facts: positive minus negative is $2.96$ points for the positive-negative question and $2.30$ for favorability, and $-0.03$ for relevance. The description of the facts as positive or negative rests on these means, and the analyses of survival do not use the ratings."}, "url": "https://pmaronow.github.io/llm-audit/works/P041/#P041-N015", "work": {"arxiv_id": null, "doi": "10.1371/journal.pone.0136327", "title": "Editorial Bias in Crowd-Sourced Political Information", "url": "https://pmaronow.github.io/llm-audit/works/P041/"}, "work_id": "P041"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of a paragraph on page 36. A red box marks the citation '(Gerber et al., 2010)' at the end of the sentence 'When social norms are asserted forcefully, the effects tend to be quite large, and even pre-recorded phone calls conveying social pressure messages significantly increase turnout (Gerber et al., 2010).'", "caption": "p. 36 (PDF p. 12), end of the first paragraph. The marked citation fits two entries of the reference list.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-A002.png"}, {"alt": "Crop of the reference list on page 44. Red boxes mark the year '(2010)' in two entries: the starred entry of Gerber, Green and Larimer, 'An experiment testing the relative effectiveness of encouraging voter participation by inducing feelings of pride or shame', and the unstarred entry of Gerber, Green, Kaplan and Kern, 'Baseline, placebo, and treatment: efficient estimation for three-group experiments'.", "caption": "Reference list, p. 44 (PDF p. 20). The two marked entries of 2010 both abbreviate to “Gerber et al., 2010”; the second is the study of pre-recorded calls.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-A002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The citation “(Gerber et al., 2010)” for pre-recorded calls with a social-pressure message refers to Gerber, Green, Kaplan and Kern (2010); the abbreviation also fits a second entry of the reference list.", "id": "P042-A002", "location": "p. 36 (PDF p. 12), first paragraph, last sentence; reference list, p. 44 (PDF p. 20)", "pdf_page": 12, "printed_page": "36", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``When social norms are asserted forcefully, the effects tend to be quite large, and even pre-recorded phone calls conveying social pressure messages significantly increase turnout \\hl{(Gerber et al., 2010)}.''", "corrected": "``\\ldots\\ and even pre-recorded phone calls conveying social pressure messages significantly increase turnout \\hl{(Gerber, Green, Kaplan \\& Kern, 2010)}.''", "intended_meaning": "The reference list has two entries that abbreviate to ``Gerber et al., 2010'': Gerber, Green and Larimer (2010), on feelings of pride or shame, and Gerber, Green, Kaplan and Kern (2010), ``Baseline, placebo, and treatment: efficient estimation for three-group experiments''. The sentence concerns pre-recorded calls, and the second entry is the study of such calls. The replication dataset codes its estimate, 1.85 with standard error 0.89, as a pre-recorded call with a social-pressure message, and the abstract stored there describes pre-recorded phone calls. The statement is correct, and only the abbreviation is ambiguous. The asterisk on the first entry points the same way, since by the legend a starred work is not cited in the text, but the asterisks are not reliable (\\fid{P042-E004})."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-A002", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text on page 32. A red box marks the words 95\\% CI = (2.686, 3.014), which follow the statement that the 29 social pressure treatments generated a weighted average treatment effect of 2.850 percentage points. Two lines below, the gratitude estimate reads 1.332 percentage points, 95\\% CI = (0.916, 1.747).", "caption": "Page 32 (PDF p. 8), first paragraph: the interval for social-pressure mail, marked. The other eight pooled intervals on pp. 31–33 are computed in the same way.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E001.png"}, {"alt": "Crop of six lines of text on page 31. Red boxes mark the words: weighting each study by the precision with which it estimates the treatment effect. This fixed effects weighting scheme.", "caption": "Page 31 (PDF p. 7), second paragraph: the weighting scheme, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E001-b.png"}, {"alt": "Crop of six lines of text on page 31. Red boxes mark the clause that says the two types of mailings have significantly different average effects. The next sentence gives the nonadvocacy estimate 0.194.", "caption": "Page 31 (PDF p. 7), third paragraph: the statement of significance that rests on the same calculation, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E001-c.png"}, {"alt": "Crop of nine lines of text on page 36. A red box marks the words: The control condition received no mail. The surrounding text describes the Civic Duty, Hawthorne and Self treatments of the direct mail experiment of Gerber et al. (2008).", "caption": "Page 36 (PDF p. 12), first paragraph: one control condition, marked, for the four mailings of Gerber et al. (2008). Three of the four contrasts are among the 29 social-pressure treatments.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E001-d.png"}, {"alt": "Chart titled: Shared control groups raise the pooled standard errors: social-pressure mail by 21\\% under the arm sizes in the dataset, and by up to 83\\% at the broader bound. The horizontal axis is the pooled standard error divided by the printed standard error, from 1.0 to 1.9, with a vertical line at 1 labelled as printed. Nine rows give, for the model, the narrower bound and the broader bound: conventional mail 1.01, 1.06, 1.09; nonadvocacy mail 1.00, 1.03, 1.06; advocacy mail 1.07, 1.28, 1.37; social-pressure mail 1.21, 1.51, 1.83, labelled plus 21\\%, plus 51\\% and plus 83\\%; gratitude mail 1.00, 1.34, 1.34; canvassing 1.00, 1.08, 1.10; recorded calls 1.00, 1.08, 1.08; commercial calls 1.00, 1.23, 1.27; volunteer calls 1.00, 1.00, 1.01. The model is drawn as light blue circles, the narrower bound as blue squares and the broader bound as dark blue diamonds.", "caption": "The pooled standard error of each of the nine estimates, divided by the standard error behind the printed interval, under the shared-control model with the arm sizes in the dataset (circles) and at the narrower and broader upper bounds (squares and diamonds). For gratitude mail and recorded calls the two bounds are equal. The values are conditional recalculations and bounds, not the covariances of the actual experiments.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E001-se-ratio.png"}], "formal_release": null, "formal_statements": [{"id": "P042-T1", "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-T1"}, {"id": "P042-T2", "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-T2"}, {"id": "P042-T5", "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-T5"}], "headline": "The pooled 95% intervals on pp. 31–33 treat all pooled contrasts as independent, although several are arms of one experiment that share a control group; the variance of each pooled estimate should include the covariances between such contrasts, and under the standard shared-control model with the arm sizes in the replication dataset the standard error of the social-pressure estimate is 21% larger than the one behind the printed interval.", "id": "P042-E001", "location": "pp. 31–33 (PDF pp. 7–9): the weighting scheme described on p. 31, second paragraph, and every pooled 95% interval that follows, shown here for the social-pressure interval on p. 32 (PDF p. 8), first paragraph. The shared control group is described on p. 36 (PDF p. 12).", "pdf_page": 7, "printed_page": "31–33", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P042-T1", "P042-T2", "P042-T5"], "text": {"as_printed": "p.~31: ``Meta-analysis is a statistical method that pools the experiments in a given domain, weighting each study by the precision with which it estimates the treatment effect. This fixed effects weighting scheme means that smaller experiments tend to be accorded less weight than larger studies.''\n\np.~32: ``The 29 social pressure treatments generated a weighted average treatment effect of 2.850 percentage points, \\hl{95\\% CI} $\\hl{= (2.686, 3.014)}$.''\n\np.~31: ``we find the two types of mailings have \\hl{significantly different} average effects.''", "corrected": "Let a pool consist of contrasts $d_1,\\dots,d_K$ with standard errors $s_1,\\dots,s_K$ and covariance matrix $\\Sigma$, so that $\\Sigma_{jj}=s_j^2$, and let the weights $a_j=s_j^{-2}/\\sum_i s_i^{-2}$ be treated as fixed. The pooled estimate $\\sum_j a_jd_j$ has variance\n\\[\na^{\\mathsf T}\\Sigma a=\\frac{1}{\\sum_j s_j^{-2}}+\\hl{2\\sum_{j<k}a_ja_k\\Sigma_{jk}}.\n\\]\nThe printed intervals use the first term alone. It is the full variance only if the second term vanishes, for instance if no two contrasts of the pool are correlated. Approximate 95\\% coverage also depends on the validity of the normal approximation.\n\nSuppose that $d_j=\\bar Y_j-\\bar Y_0$ and $d_k=\\bar Y_k-\\bar Y_0$ compare treatment groups of $n_j$ and $n_k$ subjects with a common control group of $n_0$ subjects, and that subjects' outcomes are independent with the same variance in all three groups. Then\n\\[\n\\Sigma_{jk}=\\rho_{jk}\\,s_js_k,\\qquad \\rho_{jk}=\\sqrt{\\frac{n_jn_k}{(n_j+n_0)(n_k+n_0)}}.\n\\]\nI refer to these assumptions as the shared-control model. The correlation $\\rho_{jk}$ is the same if each contrast is a fixed positive multiple of the difference in means, as for an effect per mailing.\nFor the difference between the nonadvocacy and advocacy pooled estimates the variance is $b^{\\mathsf T}\\Sigma b$, with $b_j=a_j$ in the first pool and $b_j=-a_j$ in the second.\n\nThe sentence on p.~32 should read: ``The 29 social pressure treatments generated a weighted average treatment effect of 2.850 percentage points. \\hl{If the 29 contrasts are treated as independent,} the \\hl{nominal} 95\\% CI is (2.686, 3.014)\\hl{; using the shared-control model's correlations for the arm sizes in the replication dataset with the stored standard errors, it is (2.651, 3.049).}'' The other eight intervals need the same qualification, with the values in the table above. On p.~31, the two types of mailings have significantly different average effects \\hl{under the independence-based normal approximation ($z=2.54$)}. An interval for the actual experiments requires the covariance of the contrasts of each experiment, which the replication dataset does not contain.", "evidence": "The evidence is a derivation, two exact counterexamples to the program's variance formula, and a recalculation from the replication files.\n\n\\emph{Reproduction.} The pooling commands of the replication program receive an effect and its standard error for each treatment and no covariance. With the variance $1/\\sum_j s_j^{-2}$, all nine estimates and intervals on pp.~31--33 reproduce to the printed precision. For the social-pressure pool the standard error is 0.0838.\n\n\\emph{Derivation.} Let $\\delta_j=d_j-\\mathbb{E}[d_j]$ and $\\Sigma_{jk}=\\mathbb{E}[\\delta_j\\delta_k]$, so that $\\Sigma_{jj}=s_j^2$. For fixed weights, I have\n\\[\n\\operatorname{Var}\\Bigl(\\sum_j a_jd_j\\Bigr)=\\mathbb{E}\\Bigl[\\Bigl(\\sum_j a_j\\delta_j\\Bigr)^2\\Bigr]=\\sum_j a_j^2s_j^2+2\\sum_{j<k}a_ja_k\\Sigma_{jk}.\n\\]\nWith $a_j=s_j^{-2}/\\sum_i s_i^{-2}$, the first sum equals $1/\\sum_j s_j^{-2}$. Let $\\bar Y_j$, $\\bar Y_k$ and $\\bar Y_0$ be the mean outcomes of two treatment groups and of their common control group. If the three means are uncorrelated, then\n\\[\n\\Sigma_{jk}=\\operatorname{Cov}(\\bar Y_j-\\bar Y_0,\\ \\bar Y_k-\\bar Y_0)=\\operatorname{Var}(\\bar Y_0).\n\\]\n\n\\emph{Counterexamples.} Consider six voters whose turnout is $(1,1,1,0,0,0)$ whatever they are sent, and assign two of them at random to each of a control group and two mail groups. Both true treatment effects are zero. Enumerating the 90 equally likely assignments shows that each contrast has variance $3/10$ and that the two contrasts have covariance $3/20$. Their average then has variance $9/40$, and the program's formula gives $3/20$. In the second example, each voter in three independent groups of 100 votes independently with probability $1/2$. Each of the two contrasts has variance $1/200$, and their covariance is $1/400$. Their average has variance $3/800$, where the program's formula gives $1/400$. By exact enumeration, the nominal 95\\% normal interval built from $1/400$ covers the true value zero with probability 0.8887. The examples show that the program's formula can fail for contrasts with a common control group. They do not measure the coverage of the article's intervals.\n\n\\emph{Recalculation.} The dataset records the arm sizes for some groups of contrasts with a common control group. The groups with at least two contrasts in the social-pressure pool hold 50.3\\% of its weight, and the four Kentucky treatments hold 32.4\\%. I use the shared-control model's correlations with the stored standard errors for these groups and take all other contrasts to be independent. For the social-pressure estimate, I obtain a standard error of 0.1014, which is 21\\% above 0.0838. With the Kentucky group alone, the standard error is 0.0997, which is 19\\% above. The three mailings of Gerber et al. (2008) hold 23.4\\% of the weight, each with $a\\approx0.0780$. If their three covariances equal $c$, then they add $6a^2c\\approx0.0365\\,c$ to the variance. The dataset does not determine $c$, since its sample-size fields for these rows cannot be used (\\fid{P042-E003}), and the model treats the three contrasts as independent.\n\nI also computed two upper bounds. At the narrower bound, each group of contrasts that appear to come from one experiment, but lack sizes, has the largest covariance that their standard errors allow, $s_js_k$. The broader bound also allows overlaps that the dataset leaves unresolved, among them a control group common to all seven Kentucky rows. At both bounds, different groups are independent. Neither bound is the covariance of the actual experiments.\n\\begin{audittable}{lrrrr}{The 95\\% intervals of the nine pooled estimates, in percentage points: as printed, under the shared-control model with the arm sizes in the dataset, and at the two upper bounds. The number of treatments is in parentheses.}\nPool & As printed & Shared-control model & Narrower bound & Broader bound \\\\\nConventional mail (110) & $(0.078,\\ 0.247)$ & $(0.077,\\ 0.247)$ & $(0.073,\\ 0.251)$ & $(0.071,\\ 0.254)$ \\\\\nNonadvocacy mail (79) & $(0.106,\\ 0.282)$ & $(0.106,\\ 0.282)$ & $(0.103,\\ 0.285)$ & $(0.101,\\ 0.287)$ \\\\\nAdvocacy mail (31) & $(-0.515,\\ 0.088)$ & $(-0.536,\\ 0.109)$ & $(-0.599,\\ 0.173)$ & $(-0.627,\\ 0.201)$ \\\\\nSocial-pressure mail (29) & $(2.686,\\ 3.014)$ & $(2.651,\\ 3.049)$ & $(2.602,\\ 3.098)$ & $(2.549,\\ 3.150)$ \\\\\nGratitude mail (8) & $(0.916,\\ 1.747)$ & $(0.916,\\ 1.747)$ & $(0.777,\\ 1.887)$ & $(0.777,\\ 1.887)$ \\\\\nCanvassing (71) & $(1.817,\\ 3.255)$ & $(1.817,\\ 3.255)$ & $(1.763,\\ 3.309)$ & $(1.743,\\ 3.329)$ \\\\\nRecorded calls (11) & $(-0.118,\\ 0.430)$ & $(-0.118,\\ 0.430)$ & $(-0.139,\\ 0.451)$ & $(-0.139,\\ 0.451)$ \\\\\nCommercial calls (25) & $(0.504,\\ 1.456)$ & $(0.504,\\ 1.456)$ & $(0.394,\\ 1.567)$ & $(0.378,\\ 1.583)$ \\\\\nVolunteer calls (37) & $(1.298,\\ 2.575)$ & $(1.298,\\ 2.575)$ & $(1.297,\\ 2.576)$ & $(1.290,\\ 2.583)$ \\\\\n\\end{audittable}\nThe difference between the nonadvocacy and advocacy estimates is 0.407. Its variance is $b^{\\mathsf T}\\Sigma b$, where $b_j=a_j$ in the first pool and $b_j=-a_j$ in the second. No group of contrasts in the model or the bounds joins the two pools.\n\\begin{audittable}{lrrrr}{Nonadvocacy minus advocacy mail (difference 0.407): standard error, $z$, two-sided $p$ and 95\\% interval.}\nCovariance & Standard error & $z$ & $p$ & 95\\% interval \\\\\nIndependence, as in the article & 0.160 & 2.54 & 0.011 & $(0.093,\\ 0.722)$ \\\\\nShared-control model & 0.170 & 2.39 & 0.017 & $(0.073,\\ 0.741)$ \\\\\nNarrower bound & 0.202 & 2.01 & 0.044 & $(0.011,\\ 0.804)$ \\\\\nBroader bound & 0.217 & 1.88 & 0.060 & $(-0.017,\\ 0.832)$ \\\\\n\\end{audittable}\nBoth tables use the estimates as stored. I did not repeat the calculation with the estimates corrected in \\fid{P042-N003}, \\fid{P042-N005} and \\fid{P042-N006}.", "problem": "The replication program computes each pooled estimate as a weighted average $\\sum_j a_jd_j$ of the contrasts $d_j$, with weights $a_j$ proportional to $1/s_j^2$, where $s_j$ is the standard error of $d_j$. It takes the variance of this average to be $1/\\sum_j s_j^{-2}$. That is the variance only if the weighted covariance terms sum to zero, as they do for uncorrelated contrasts. Several pooled contrasts are arms of one experiment that share a control group. The Hawthorne, Self and Neighbors mailings of Gerber et al. (2008), which p.~36 describes with one control condition, are three of the 29 social-pressure treatments. Four Kentucky treatments of Mann (2010) share a control group of 19,561 people and hold 32\\% of the weight of that pool. Two contrasts with a common control group both contain the sampling error of the control mean. For unscaled differences in means under independent sampling, their covariance is the variance of that mean, which is positive. The printed intervals omit these terms. How much the intervals' widths should change for the actual experiments cannot be recovered from the dataset, which gives marginal standard errors but no covariances; the size of the omission can be computed under stated models, as below.", "what_survives": "The nine pooled estimates, their counts and their ordering do not depend on the variance and are unchanged. Under the model and at both bounds, each of the nine intervals keeps its position relative to zero. The social-pressure and gratitude normal intervals stay above zero for every covariance matrix that is compatible with the stored standard errors, with minimum lower endpoints of 2.087 and 0.257 to three decimal places. The difference between nonadvocacy and advocacy mail stays significant at the 5\\% level under the model and at the narrower bound. At the broader bound it does not ($z=1.88$), which shows that the printed statement could fail and not that it does. What changes is the stated precision: the printed intervals use the independence-based variance and a normal approximation."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-E001", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Note 5 on page 40. Red boxes mark '1.720' and '(0.306, 3.135)' in the last sentence of the note: 'These studies found stronger effects among Compliers: a weighted average effect of 1.720 percentage points, with a 95 percent CI = (0.306, 3.135).' The sentence before it refers to the two pre-recorded call experiments that confronted recipients with their vote histories.", "caption": "Note 5, p. 40 (PDF p. 16). The marked estimate and interval match a random-effects calculation.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E002.png"}, {"alt": "Crop of a paragraph on page 31. A red box marks the words 'fixed effects' in the sentence 'This fixed effects weighting scheme means that smaller experiments tend to be accorded less weight than larger studies.'", "caption": "p. 31 (PDF p. 7), second paragraph. The marked words declare the pooling method of the article.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P042-T3", "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-T3"}, {"id": "P042-T4", "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-T4"}], "headline": "The figure in Note 5 for the two experiments on pre-recorded calls with a social-pressure message is not the fixed effects average that the article declares on p. 31; the fixed effects average of the two estimates in the replication dataset is 1.866, with 95% interval (0.963, 2.768), and the printed 1.720, with interval (0.306, 3.135), matches a random-effects estimate.", "id": "P042-E002", "location": "Note 5, p. 40 (PDF p. 16), last sentence; the method is declared on p. 31 (PDF p. 7)", "pdf_page": 16, "printed_page": "40", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P042-T3", "P042-T4"], "text": {"as_printed": "``These studies found stronger effects among Compliers: a weighted average effect of \\hl{1.720} percentage points, with a 95\\% CI = \\hl{(0.306, 3.135)}.''\n\nThe method paragraph on p.~31 says: ``Meta-analysis is a statistical method that pools the experiments in a given domain, weighting each study by the precision with which it estimates the treatment effect. This fixed effects weighting scheme means that smaller experiments tend to be accorded less weight than larger studies.''", "corrected": "``These studies found stronger effects among Compliers: a weighted average effect of \\hl{1.866} percentage points, with a 95\\% CI = \\hl{(0.963, 2.768)}.''\n\nThis is the fixed effects average of the two estimates in the replication dataset, with weights equal to the inverse squared standard errors and a normal interval, as declared on p.~31; the replication program should include the command that produces it. If the printed figure is the intended one, the note should instead say that it is a DerSimonian--Laird random-effects estimate, with $\\widehat\\tau^2=0.5819$, and that it is an exception to the fixed effects weighting of p.~31, and the replication program should include that command. Both alternatives keep a positive interval.", "evidence": "The evidence is an exact calculation from the two estimates in the replication dataset. Let $x_1=2.32988$ and $x_2=0.86596$ denote the two estimates and $v_1=0.55712^2$ and $v_2=0.81782^2$ their variances, and set $w_i=1/v_i$. The fixed effects average and its variance are\n\\[\n\\widehat\\mu_F=\\frac{w_1x_1+w_2x_2}{w_1+w_2}=\\frac{114191974724271}{61200765425000}=1.865859,\\qquad V_F=\\frac{1}{w_1+w_2}=0.212000 .\n\\]\nSince $\\widehat\\mu_F>1.7205$, it cannot round to the printed 1.720. For the random-effects estimate, the heterogeneity statistic for two studies is $Q=(x_1-x_2)^2/(v_1+v_2)=2.188557$, and the DerSimonian--Laird estimate of the variance between studies is\n\\[\n\\widehat\\tau^2=\\frac{Q-1}{2/(v_1+v_2)}=\\frac{2909623799}{5000000000}=0.5819247598 .\n\\]\nWith $u_i=1/(v_i+\\widehat\\tau^2)$, I obtain\n\\[\n\\widehat\\mu_R=\\frac{u_1x_1+u_2x_2}{u_1+u_2}=\\frac{12592252377}{7319600000}=1.720347,\\qquad V_R=\\frac{1}{u_1+u_2}=0.520777 .\n\\]\n\\begin{audittable}{lrrr}{The figure of Note~5 and the two calculations from the estimates in the replication dataset, in percentage points.}\nCalculation & Estimate & SE & 95\\% interval \\\\\nPrinted in Note~5 & 1.720 & & (0.306, 3.135) \\\\\nDerSimonian--Laird random effects & 1.720347 & 0.721649 & (0.305941, 3.134753) \\\\\nFixed effects, as declared on p.~31 & 1.865859 & 0.460435 & (0.963423, 2.768294) \\\\\n\\end{audittable}\nThe rounding of both intervals is the same for every multiplier between 1.9599 and 1.9600.", "problem": "The article declares one pooling method, a fixed effects average in which each study is weighted by its precision, and its nine other pooled figures on pp.~31--33 follow it. The figure in Note~5 does not. The two estimates for these experiments in the replication dataset are 2.32988 with standard error 0.55712 (``Robo SP1'') and 0.86596 with standard error 0.81782 (``Robo SP2''), and their fixed effects average is 1.866, with interval (0.963, 2.768). The printed estimate and both limits agree to three decimals with the DerSimonian--Laird random-effects estimate from the same two estimates. The note does not say that the method differs. The replication program has no command that pools the two estimates: its ten pooling commands all specify fixed effects, and its only mention of the two entries excludes them from the ordinary pre-recorded calls. Read alone, the words ``a weighted average'' admit the random-effects calculation. Read with the method declared on p.~31, the figure is wrong. Note that the numerical agreement does not identify the command that produced the printed figure, and it does not show whether the exception was intended.", "what_survives": "Both calculations give a positive estimate with a 95\\% interval above zero, so the statement of the note that these calls had stronger effects stands, as does the statement on p.~36 that such calls increase turnout. The nine other pooled figures of the article agree with the fixed effects calculation and do not change. Whether a third estimate for pre-recorded calls belongs with these two is a separate matter (\\fid{P042-N010}), and I do not change the count."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-E002", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Listing of cells of the replication dataset with their row numbers. First part: rows 7 and 9, author Gillespie, year 2005. Red boxes mark the Notes cells, St. Louis Experiment and Detroit Experiment 2001, and the two Election/Intervention date cells, both reading 2002 Newark NJ mayoral (May) and runoff (June). Second part: rows 256 to 259, Gerber, Green, Larimer, 2008, with notes civic duty, hawthorne, self and neighbors. One red box encloses their N Treatment cells, all 353341, and their N Control cells, 3238, 6307, 6325 and 6325; the N cells are 356579, 359648, 359666 and 359666. Below, unmarked, rows 86 to 88, Gerber, Green, Larimer, 2010: N Treatment 3238, 6307 and 6325, N Control 353341 in each, N 356579, 359648 and 359666.", "caption": "The replication dataset, not a page of the article: rows 7 and 9, and rows 256–259 above rows 86–88. In rows 7 and 9 the marked fields name different places. In rows 256–259 the marked sample sizes are those of rows 86–88 with treatment and control exchanged.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E003.png"}, {"alt": "Crop of the first paragraph of page 36. It says that Gerber et al. (2008) conducted a large-scale direct mail experiment that varied the forcefulness with which the norm of voting was asserted. A red box marks the words: The control condition received no mail. The text goes on to describe the Civic Duty, Hawthorne and Self treatments.", "caption": "Page 36 (PDF p. 12), first paragraph: the article describes the experiment of rows 256–259 as having one control condition, marked, and four mailings.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In the replication dataset, two rows for Gillespie (2005) give conflicting experiment and election labels, and the sample-size fields of the four rows for Gerber, Green and Larimer (2008) repeat those of the 2010 study with treatment and control exchanged; these fields should be checked against the sources and corrected.", "id": "P042-E003", "location": "Replication dataset (Yale Dataverse, release v2.0; the CSV file of the release), rows 7, 9 and 256–259, where the header is row 1. The dataset is not printed in the article; the experiment of rows 256–259 is described on p. 36 (PDF p. 12).", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{audittable}{rll}{Rows 7 and 9 (Gillespie, 2005).}\nRow & Notes & Election/Intervention date \\\\\n7 & \\hl{St. Louis Experiment} & \\hl{2002 Newark NJ mayoral (May) \\& runoff (June)} \\\\\n9 & \\hl{Detroit Experiment 2001} & \\hl{2002 Newark NJ mayoral (May) \\& runoff (June)} \\\\\n\\end{audittable}\n\\begin{audittable}{rlrrr}{Rows 256--259 (Gerber, Green and Larimer, 2008; election field ``2006 primary Michigan'').}\nRow & Notes & N Treatment & N Control & N \\\\\n256 & civic duty & \\hl{353341} & \\hl{3238} & 356579 \\\\\n257 & hawthorne & \\hl{353341} & \\hl{6307} & 359648 \\\\\n258 & self & \\hl{353341} & \\hl{6325} & 359666 \\\\\n259 & neighbors & \\hl{353341} & \\hl{6325} & 359666 \\\\\n\\end{audittable}", "corrected": "Rows 7 and 9: the fields Notes and Election/Intervention date should name the same experiment, as established from Gillespie (2005). Until then, the election field of these two rows should be left blank or marked as unchecked.\n\nRows 256--259: the sizes should be taken from Table~2 of Gerber, Green and Larimer (2008). The sizes proposed for these rows are 38,218, 38,204, 38,218 and 38,201 individuals in the Civic Duty, Hawthorne, Self and Neighbors groups and 191,243 in the common control group, 344,084 in all. If the table confirms them, the rows should read as follows.\n\\begin{audittable}{rlrrr}{Rows 256--259, conditional on Table 2 of Gerber, Green and Larimer (2008).}\nRow & Notes & N Treatment & N Control & N \\\\\n256 & civic duty & \\hl{38218} & \\hl{191243} & \\hl{229461} \\\\\n257 & hawthorne & \\hl{38204} & \\hl{191243} & \\hl{229447} \\\\\n258 & self & \\hl{38218} & \\hl{191243} & \\hl{229461} \\\\\n259 & neighbors & \\hl{38201} & \\hl{191243} & \\hl{229444} \\\\\n\\end{audittable}", "evidence": "The evidence is a check of the replication dataset and a recalculation. I transcribed the fields above from the file. The ten pooling commands of the replication program use only the effect and standard-error columns, the classification flags and two title exclusions. In exact arithmetic, the ten pooled estimates and their variances are the same before and after the election and sample-size fields are changed. Neither Table~2 of Gerber, Green and Larimer (2008) nor Gillespie's dissertation was available for the check, and the correct sizes and dates are not established here. The name of the file carries the date 19 March 2014, and I did not establish whether the file used for the article had the same entries.", "problem": "Each row of the dataset should describe the experiment behind its estimate. Row 7 names a St.~Louis experiment and row 9 a Detroit experiment of 2001, and both carry the Newark elections of 2002 in the election field. At least one of the two fields is wrong in each row, and the dataset does not show which. In rows 256--259 the treatment group has 353,341 members in every row and the control group has 3,238, 6,307, 6,325 and 6,325 members, although the article describes one control group for the four mailings. These are the sizes stored for the 2010 study in rows 86--88 with treatment and control exchanged, and row 259 repeats row 258. The pattern is consistent with a copying slip, but it does not show what the correct sizes are.", "what_survives": "The pooled estimates, intervals and counts on pp.~31--33 do not use these fields and are unchanged. The correction matters to anyone who uses the election or sample-size fields. The replication program also summarizes the column \\texttt{N}, and that summary may change."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-E003", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the head of the reference list on page 40. Red boxes mark the words 'but are not otherwise referenced in the text.' in the legend 'References marked with an asterisk are included in the meta-analysis, but are not otherwise referenced in the text.'", "caption": "References, p. 40 (PDF p. 16). The marked clause of the legend says that a starred work is not cited in the text.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E004.png"}, {"alt": "Crop of the reference list on page 40. A red box marks the asterisk, authors and year of the entry 'Almond, Gabriel A. and Verba, Sidney (1963) The Civic Culture (Princeton: Princeton University Press)', which begins with an asterisk. The entries above and below it, Alford and Lee (1968) and Alvarez, Hopkins and Sinclair (2010), have none.", "caption": "References, p. 40 (PDF p. 16). The marked entry carries an asterisk.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E004-b.png"}, {"alt": "Crop of the top of page 28, with the page number 28 and the running head. A red box marks the citation '(Almond \\& Verba, 1963;' in the line 'socialization and culture (Almond \\& Verba, 1963; Litt, 1963), patronage (Blaydes,'.", "caption": "p. 28 (PDF p. 4), first line. The starred work of Almond and Verba is cited in the text.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E004-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The asterisks in the reference list do not follow the rule stated above the list: five starred works are cited in the text, one of them matches no entry in the replication dataset, and six uncited works that supply estimates there carry no asterisk.", "id": "P042-E004", "location": "References, p. 40 (PDF p. 16): the legend above the list, and the starred entries on pp. 40–48 (PDF pp. 16–24)", "pdf_page": 16, "printed_page": "40", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``References marked with an * are included in the meta-analysis, \\hl{but are not otherwise referenced in the text}.''\n\n``\\hl{*}Almond, Gabriel A. \\& Verba, Sidney (1963) \\emph{The Civic Culture} (Princeton: Princeton University Press).''", "corrected": "``\\del{*}Almond, Gabriel A. \\& Verba, Sidney (1963) \\emph{The Civic Culture} (Princeton: Princeton University Press).''\n\nThe asterisk on Almond and Verba (1963) is removed. The remaining asterisks and the legend are to be regenerated from the list of inputs used for the article; I have not established a complete replacement set of marks.", "evidence": "The evidence is a check of the reference list on pp.~40--48 and of the citations in the text, together with a comparison of the list with the 295 entries of the replication dataset and the selections of the replication program. The replication dataset is the release of March 2014, after publication, and I cannot confirm that it is the list of inputs used for the article. For this reason I give no complete set of corrected marks. The five citations in the text contradict the legend on the printed page and do not depend on the dataset.", "problem": "By the legend, a starred work is an input of the meta-analysis and is cited nowhere in the text. The marks depart from this rule in three ways.\n\\begin{itemize}\n\\item Five starred works are cited in the text: Almond and Verba (1963) on p.~28; Arceneaux and Nickerson (2009) on p.~38; Arceneaux and Nickerson (2010) on p.~35; Gerber and Green (2000a) on pp.~28, 31, 32 and 35; and Sinclair, McConnell and Green (2012) on pp.~36 and 37. Almond and Verba (1963) also matches no entry of the replication dataset.\n\\item Six works that supply estimates to the pooled figures and are not cited in the text carry no asterisk: Arceneaux (2005); Arceneaux, Gerber and Green (2010); Barton, Castillo and Petrie (2012); Mann (2005); Mann and Mayhew (2012); and Nickerson (2007b).\n\\item Shaw, Green, Gimpel and Gerber (2012) supplies two of the estimates for pre-recorded calls and has no entry in the reference list.\n\\end{itemize}\nIn addition, eight starred works supply no usable estimate to the pooled figures in the replication dataset. Seven match no entry: Aker, Collier and Vicente (2010); Bond et al.\\ (2012); Gerber and Green (2000b); Gerber, Karlan and Bergan (2009); Mann and Klofstad (2011); Panagopoulos (2009b); and Panagopoulos and Green (2011). The eighth, Cann et al.\\ (2012), matches two entries dated 2010, without standard errors.", "what_survives": "No formula, estimate or argument of the article depends on the asterisks. The review and the pooled figures are unaffected; the defect is in the bookkeeping of the reference list."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-E004", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of text on page 29. A red box marks the words Redefining Citizenship in the italic book title Mobilizing Inclusion: Redefining Citizenship through Get-Out-the-Vote Campaigns, cited as García Bedolla and Michelson, 2012.", "caption": "Page 29 (PDF p. 5), first paragraph: the subtitle given in the text, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005.png"}, {"alt": "Crop of the reference entry García Bedolla, Lisa and Michelson, Melissa R. (2012). A red box marks the words Transforming the Electorate in the title Mobilizing Inclusion: Transforming the Electorate through Get-Out-the-Vote Campaigns.", "caption": "Reference list, p. 43 (PDF p. 19): the subtitle of the same book in its entry, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-b.png"}, {"alt": "Crop of two consecutive reference entries by Gerber, Alan S. and Green, Donald P., both titled Do phone calls increase voter turnout, A field experiment, Public Opinion Quarterly, 65, pages 75 to 85. Red boxes mark the year (2001) in the first entry and (2001b) in the second. The second title lacks the question mark.", "caption": "Reference list, p. 43 (PDF p. 19): one article entered twice, under the marked years.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-c.png"}, {"alt": "Crop of the reference entry Kelley, Stanley, Jr., Ayres, Richard and Bowen, William C. (1967) Registration and voting: putting first things first. American Political Science Review, 10, pages 233 to 246. A red box marks the volume number 10.", "caption": "Reference list, p. 45 (PDF p. 21): the volume number of the entry for Kelley, Ayres and Bowen (1967), marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-d.png"}, {"alt": "Crop of the reference entry Chong, Alberto, De La O, Ana L., Karlan, Dean and Wantchekron, Leonard (2011). A red box marks the surname Wantchekron.", "caption": "Reference list, p. 42 (PDF p. 18): the misspelled surname, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-e.png"}, {"alt": "Crop of two reference entries. The first is Huckfeldt and Sprague (1992), American Political Science Review, 86. In the second, Ichino, Nahomi and Schündeln, Matthias (2012), a red box marks the word orregularities in the title Deterring or displacing electoral orregularities?", "caption": "Reference list, p. 45 (PDF p. 21): the misspelled word, marked. The entry above gives volume 86 of the American Political Science Review for 1992.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-f.png"}, {"alt": "Crop of five lines of text on page 38. A red box marks the citation Cutts et al. (2010) in the sentence beginning Although studies like Cutts et al. (2010) are designed to test whether results from American experiments apply to the United Kingdom.", "caption": "Page 38 (PDF p. 14): the citation that has no entry in the reference list, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-g.png"}, {"alt": "Crop of the reference entry Michelson, Melissa R., Bedolla, Lisa García and Green, Donald P. (2007) New experiments in minority voter mobilization. A red box marks the words Bedolla, Lisa García.", "caption": "Reference list, p. 46 (PDF p. 22): the inverted surname in the entry for Michelson et al. (2007), marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-h.png"}, {"alt": "Crop of the reference entry Michelson, Melissa R., Bedolla, Lisa García and McConnell, Margaret A. (2009) Heeding the call. A red box marks the words Bedolla, Lisa García.", "caption": "Reference list, p. 47 (PDF p. 23): the inverted surname in the entry for Michelson et al. (2009), marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-E005-i.png"}], "formal_release": null, "formal_statements": [], "headline": "One citation and the reference list contain seven slips: two subtitles for one book, a duplicated entry, a wrong volume number, two misspellings, a citation with no matching entry, and a surname inverted in two entries.", "id": "P042-E005", "location": "p. 29 (PDF p. 5), first paragraph; p. 38 (PDF p. 14); reference list, pp. 42, 43, 45, 46 and 47 (PDF pp. 18, 19, 21, 22 and 23).", "pdf_page": 5, "printed_page": "29", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{enumerate}\n\\item p.~29: ``The recent book \\emph{Mobilizing Inclusion: \\hl{Redefining Citizenship} through Get-Out-the-Vote Campaigns} (García Bedolla \\& Michelson, 2012) alone reports more than 100 experiments.'' Reference list, p.~43: ``García Bedolla, Lisa \\& Michelson, Melissa R. (2012) \\emph{Mobilizing Inclusion: \\hl{Transforming the Electorate} through Get-Out-the-Vote Campaigns} (New Haven: Yale University Press).''\n\\item p.~43: ``*Gerber, Alan S. \\& Green, Donald P. \\hl{(2001)} Do phone calls increase voter turnout? A field experiment. \\emph{Public Opinion Quarterly}, 65, pp.~75--85.'' and ``*Gerber, Alan S. \\& Green, Donald P. \\hl{(2001b)} Do phone calls increase voter turnout A field experiment. \\emph{Public Opinion Quarterly}, 65, pp.~75--85.''\n\\item p.~45: ``Kelley, Stanley, Jr., Ayres, Richard \\& Bowen, William C. (1967) Registration and voting: putting first things first. \\emph{American Political Science Review}, \\hl{10}, pp.~233--246.''\n\\item p.~42: ``Chong, Alberto, De La O, Ana L., Karlan, Dean \\& \\hl{Wantchekron}, Leonard (2011) Looking beyond the incumbent: the effects of exposing corruption on electoral outcomes. NBER Working Paper 17679.''\n\\item p.~45: ``Ichino, Nahomi \\& Schündeln, Matthias (2012) Deterring or displacing electoral \\hl{orregularities}? Spillover effects of observers in a randomized field experiment in Ghana.''\n\\item p.~38: ``Although studies like \\hl{Cutts et al. (2010)} are designed to test whether results from American experiments apply to the United Kingdom, the literature has yet to produce a study that fields the same treatments in different countries.''\n\\item p.~46: ``Michelson, Melissa R., \\hl{Bedolla, Lisa García} \\& Green, Donald P. (2007)''; p.~47: ``Michelson, Melissa R., \\hl{Bedolla, Lisa García} \\& McConnell, Margaret A. (2009)''.\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item One subtitle in both places, as on the title page of the book; the reference list has ``Transforming the Electorate through Get-Out-the-Vote Campaigns''.\n\\item One entry, ``*Gerber, Alan S. \\& Green, Donald P. (2001) Do phone calls increase voter turnout? A field experiment. \\emph{Public Opinion Quarterly}, 65, pp.~75--85.'', with the suffixes of the citations reconciled.\n\\item ``\\emph{American Political Science Review}, \\hl{61}'', with the page range and the authors' initials checked against the article.\n\\item ``\\hl{Wantchekon}, Leonard''.\n\\item ``electoral \\hl{irregularities}?''\n\\item The citation on p.~38 should name the source meant, and the list should contain it; no replacement is established here.\n\\item ``Michelson, Melissa R., \\hl{García Bedolla, Lisa} \\& \\ldots'' in both entries.\n\\end{enumerate}", "evidence": "The evidence is a check of the pages: I read each item on the page and compared it with the rest of the reference list. The book, the article of Kelley et al. (1967) and the source meant by the Cutts citation were not available. The correct subtitle, the page range and the authors' initials of the Kelley entry, and the intended Cutts reference are therefore not settled here. The entry for Bhavnani (2009) on p.~41 has a single page, 23, and I could not check whether a page range is missing.", "problem": "The seven items are these. (1) The text and the reference list give different subtitles for one book. (2) The entries for 2001 and 2001b are the same article. (3) Volume 10 cannot be right for 1967, since the list has volume 50 of the same journal for 1956, 62 for 1968, 72 for 1978 and 80 for 1986, which implies volume 61 for 1967. (4) The surname is Wantchekon on pp.~38 and 48. (5) The word is ``irregularities''. (6) No entry matches ``Cutts et al. (2010)''; the only entry led by Cutts is dated 2009, and the article does not show which source is meant. (7) The list itself has the surname García Bedolla on p.~43.", "what_survives": "The slips affect reference lookup and attribution. No estimate or argument depends on them, and the duplicated entry does not show that a study entered the meta-analysis twice."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-E005", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of text on page 31. Red boxes mark the sentence that begins Using linear regression and ends with the words percentage-point increase in turnout per mailing. The lines above say that 147 distinct experimental treatments assessed the effects of direct mail, sometimes sending several mailings to a given recipient.", "caption": "Page 31 (PDF p. 7), third paragraph: the marked sentence states that the mail estimates are increases in turnout per mailing.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N003.png"}, {"alt": "Crop of ten lines of text on page 31. A red box marks the words 0.162, with a 95\\% confidence interval of (0.078, 0.247). A second red box marks the words minus 0.213 with a 95\\% interval of (minus 0.515, 0.088). Between them the text says that the two types of mailings have significantly different average effects and that nonadvocacy mailings increase turnout by 0.194, with a 95\\% confidence interval of (0.106, 0.282).", "caption": "Page 31 (PDF p. 7), same paragraph: the two pooled estimates that change, marked. The estimate for nonadvocacy mail, 0.194, does not change.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N003-b.png"}, {"alt": "Chart titled: Putting five estimates on the per-mailing scale lowers conventional mail from 0.162 to 0.132 and moves advocacy mail toward zero; nonadvocacy mail is unchanged. The horizontal axis is the pooled effect on turnout in percentage points per mailing, with a vertical line at zero. For conventional mail, 110 treatments, the printed estimate is 0.162 with interval 0.078 to 0.247 and the corrected estimate is 0.132 with interval 0.055 to 0.208. For nonadvocacy mail, 79 treatments, both are 0.194 with interval 0.106 to 0.282. For advocacy mail, 31 treatments, the printed estimate is minus 0.213 with interval minus 0.515 to 0.088 and the corrected estimate is minus 0.066 with interval minus 0.222 to 0.091. Printed values are drawn as orange open circles and corrected values as blue squares.", "caption": "The pooled mail estimates of p. 31 with their 95% intervals, as printed (open circles) and with the five estimates of rows 226–230 on the per-mailing scale (squares). The intervals are computed as in the article.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N003-mail-pools.png"}], "formal_release": null, "formal_statements": [], "headline": "Five of the mail estimates, taken from Gerber, Green and Green (2003), are differences in turnout for whole programmes of four, six or nine mailings and not effects per mailing; on the per-mailing scale the estimate for conventional mail is 0.132 with 95% interval (0.055, 0.208), not 0.162 (0.078, 0.247), and the estimate for advocacy mail is $-0.066$ $(-0.222, 0.091)$, not $-0.213$ $(-0.515, 0.088)$.", "id": "P042-N003", "location": "p. 31 (PDF p. 7), third paragraph: the sentence that begins “Using linear regression” and the pooled estimates for conventional and advocacy mail that follow; replication dataset, rows 226–230, where the header is row 1.", "pdf_page": 7, "printed_page": "31", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "``In all, 147 distinct experimental treatments assessed the effects of direct mail, sometimes sending several mailings to a given recipient.$^{4}$ Using linear regression, we estimated the percentage-point increase in turnout \\hl{per mailing}. [\\ldots] In this restricted set of 110 studies, we find a weighted average treatment effect of $\\hl{0.162}$, with a 95\\% confidence interval of $\\hl{(0.078, 0.247)}$. [\\ldots] whereas the 31 advocacy mailings have negligible effects on turnout: an estimated effect of $\\hl{-0.213}$ with a 95\\% interval of $\\hl{(-0.515, 0.088)}$.''\n\\begin{audittable}{rlrr}{Replication dataset, rows 226--230 (Gerber, Green and Green, 2003), as stored.}\nRow & Notes & Mail ITT & Mail SE \\\\\n226 & CT Mayoral Race & $\\hl{-1.1}$ & \\hl{1.13} \\\\\n227 & 6 mailers, other Dems \\& Independents & \\hl{0.02} & \\hl{1.2} \\\\\n228 & 4 mailers, other registered voters & \\hl{0.45} & \\hl{1.67} \\\\\n229 & 6 mailers, other registered voters & \\hl{1.04} & \\hl{1.49} \\\\\n230 & 4 mailers, prime democrats only & \\hl{1.92} & \\hl{1.47} \\\\\n\\end{audittable}", "corrected": "``In this restricted set of 110 studies, the weighted average treatment effect is $\\hl{0.132}$, with a 95\\% confidence interval of $\\hl{(0.055, 0.208)}$. [\\ldots] whereas the 31 advocacy mailings have negligible effects on turnout: an estimated effect of $\\hl{-0.066}$ with a 95\\% interval of $\\hl{(-0.222, 0.091)}$.''\n\\begin{audittable}{rrrr}{Rows 226--230 on the per-mailing scale: the stored effect and standard error divided by the number of mailings $m$.}\nRow & $m$ & Effect per mailing & Standard error \\\\\n226 & 9 & $\\hl{-0.1222}$ & \\hl{0.1256} \\\\\n227 & 6 & \\hl{0.0033} & \\hl{0.2000} \\\\\n228 & 4 & \\hl{0.1125} & \\hl{0.4175} \\\\\n229 & 6 & \\hl{0.1733} & \\hl{0.2483} \\\\\n230 & 4 & \\hl{0.4800} & \\hl{0.3675} \\\\\n\\end{audittable}", "evidence": "The evidence is a check of the source, a derivation and a recalculation from the replication files.\n\n\\emph{Source.} Gerber, Green and Green (2003) write on p.~572 that ``the four mailings increase turnout from 63.7 to 65.6\\%, or approximately 0.5 percentage-points per mailing.'' On p.~573 the Connecticut treatment group ``received nine pieces of direct mail''. Their Table~3 reports differences in turnout of about 1.9, 0.0, 0.4 and 1.0 points for the groups in New Jersey that were sent 4, 6, 4 and 6 mailings, and of $-1.1$ points for Connecticut. The stored values 1.92, 0.02, 0.45, 1.04 and $-1.1$ are these programme differences. Table~3 reports rounded differences and not every standard error; the stored standard errors are close to binomial standard errors on the programme scale. For the New Jersey sample in which the number of mailings varies, the same article reports a slope of 0.16 per mailing with standard error 0.24 (p.~573), which is not among the stored values.\n\n\\emph{Derivation.} Let a treatment group be assigned $m$ mailings and the control group none, let $D$ indicate assignment to treatment, and let $M=mD$. The regression of turnout on $D$ has slope $\\hat b_D$, the difference in turnout between the two groups. Since $a+b_MM=a+(mb_M)D$, the regression on $M$ has the same fitted values. This implies that\n\\[\n\\hat b_M=\\frac{\\hat b_D}{m},\\qquad \\operatorname{SE}(\\hat b_M)=\\frac{\\operatorname{SE}(\\hat b_D)}{m}.\n\\]\nA stored pair $(e_i,s_i)$ that is a programme difference therefore corresponds to the per-mailing pair $(e_i/m_i,\\ s_i/m_i)$, and its weight $1/s_i^2$ becomes $m_i^2/s_i^2$. Let $W=\\sum_i 1/s_i^2$ and $A=\\sum_i e_i/s_i^2$ over a pool that contains the set $T$ of the five rows, so that the pooled estimate is $A/W$ with the article's standard error $1/\\sqrt{W}$. On the per-mailing scale, I obtain\n\\[\nW'=W+\\sum_{i\\in T}\\frac{m_i^2-1}{s_i^2},\\qquad A'=A+\\sum_{i\\in T}\\frac{(m_i-1)\\,e_i}{s_i^2},\n\\]\nand the pooled estimate $A'/W'$ with standard error $1/\\sqrt{W'}$ by the same calculation.\n\n\\emph{Recalculation.} The pooled estimates as printed reproduce from the dataset. Dividing the five rows by 9, 6, 4, 6 and 4 and keeping every other estimate, I find 0.1317 with standard error 0.0391 for conventional mail and $-0.0657$ with standard error 0.0797 for advocacy mail. The division is exact only for an estimate that compares a fixed number of mailings with none. The other scale checks were Niven's (2006) single-mailing estimate and Ramirez's (2005) two Colorado estimates, all per mailing; I did not examine the remaining estimates' units.", "problem": "The article states that every mail estimate is an increase in turnout per mailing. Five of the 31 advocacy estimates are not on that scale. Rows 226--230 of the dataset are the differences in turnout between a group that was sent a whole programme of mailings and a control group that was sent none: nine mailings in the Connecticut mayoral race, and four or six in the New Jersey assembly race. For a fixed programme of $m$ mailings versus none, the per-mailing regression coefficient and its standard error are the programme contrast and standard error divided by $m$. The five rows therefore entered the pooling with effect magnitudes that are too large and with weights that are too small by the factors 81, 36, 16, 36 and 16. As stored, they hold 0.5\\% of the weight in the estimate for conventional mail, and on the per-mailing scale they hold 18.0\\%. The corrected estimate for conventional mail is 18.8\\% below the printed one.", "what_survives": "The counts and the classification of the treatments are unchanged, as are the estimates for nonadvocacy mail, 0.194 (0.106, 0.282), for social-pressure and gratitude mail, and for canvassing and phone calls. The interval for conventional mail stays above zero, and the interval for advocacy mail still contains zero. With the article's variance calculation, the difference between the two types of mailings stays significant at the 5\\% level: it falls from 0.407 ($z=2.54$) to 0.260 ($z=2.84$), with 95\\% interval (0.080, 0.439). The statements on pp.~31--33 that conventional mail has weak effects and that advocacy mail has little effect on turnout stand. The pooled estimate for all 147 mail treatments in the replication program, which the article does not print, changes from 0.743 (0.669, 0.816) to 0.637 (0.568, 0.705). These intervals omit covariance between contrasts sharing controls; see \\fid{P042-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-N003", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of text on page 32. Red boxes mark the words The 29 social pressure treatments and the number 2.850 in the sentence: The 29 social pressure treatments generated a weighted average treatment effect of 2.850 percentage points, 95\\% CI = (2.686, 3.014).", "caption": "Page 32 (PDF p. 8), first paragraph: the count and the pooled estimate for social-pressure mail, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "Among the 29 estimates for social-pressure mail, three treatments of the 2007 Kentucky experiment of Mann (2010) are counted twice, once for people who did not vote in 2006 and once for a broader sample that contains those people; each treatment should enter once, which leaves 26 estimates with a weighted average of 2.874 or 3.069 according to the sample kept, against the printed 2.850.", "id": "P042-N005", "location": "p. 32 (PDF p. 8), first paragraph: “The 29 social pressure treatments …”; replication dataset, rows 171–176 (Mann, 2010), where the header is row 1.", "pdf_page": 8, "printed_page": "32", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``The \\hl{29} social pressure treatments generated a weighted average treatment effect of $\\hl{2.850}$ percentage points, 95\\% CI $= (2.686, 3.014)$.''\n\\begin{audittable}{rlrr}{Replication dataset, rows 171--176 (Mann, 2010; election field ``2007 KY gubernatorial''), as stored.}\nRow & Notes & Mail ITT & Mail SE \\\\\n171 & Mann\\_2007\\_KY hawthorne (didn't vote 06) & 2.1 & 0.3 \\\\\n172 & Mann\\_2007\\_KY self (didn't vote 06) & 2.1 & 0.3 \\\\\n173 & Mann\\_2007\\_KY hawthorne+survey (didn't vote 06) & 2.2 & 0.3 \\\\\n174 & Mann\\_2007\\_KY hawthorne+survey & 2.5 & 0.35 \\\\\n175 & Mann\\_2007\\_KY hawthorne & 2.6 & 0.35 \\\\\n176 & Mann\\_2007\\_KY self & 3.1 & 0.35 \\\\\n\\end{audittable}", "corrected": "Each Kentucky treatment should enter once, for one stated sample. With the estimates for the 2006 nonvoters, rows 171--173: ``The \\hl{26} social pressure treatments generated a weighted average treatment effect of $\\hl{2.874}$ percentage points.'' With the estimates for the broader sample, rows 174--176, the average is $\\hl{3.069}$. I could not determine which sample was intended. The interval should then account for all dependent contrasts, including the Kentucky contrasts, which share a control group (\\fid{P042-E001}).", "evidence": "The evidence is a check of the source and a recalculation from the replication files. I read Mann (2010) in the author's posted manuscript of 20 April 2010, not in the journal's typesetting. The first sample of the study has 78,441 people who did not vote in 2006. The broader sample has no further restriction (p.~12 of the manuscript), and Table~6 reports turnout of 13.2\\%, 16.3\\%, 15.8\\% and 15.7\\% in it. The differences from the first figure, 3.1, 2.6 and 2.5 points, are the values stored in rows 176, 175 and 174. The analysis of the broader sample leaves out only the 17,609 recipients of a fourth mailing from 95,788 people, so that 60,832 of the 2006 nonvoters are in both analyses. The 19,561 nonvoters in the control group are among the 25,037 controls of the broader sample. The printed 2.850 reproduces from the 29 rows. Without rows 174--176, the 26 remaining rows have a weighted average of 2.874, and without rows 171--173, of 3.069.", "problem": "For the Hawthorne, Self and Hawthorne-plus-survey mailings the dataset holds two estimates each. Rows 171--173 are labelled as estimates for people who did not vote in 2006. Rows 174--176 carry no such label. They are not estimates for the people who did vote in 2006. They are the estimates for the broader sample of the study, which contains the 2006 nonvoters. The same people therefore stand behind two estimates of each mailing, and three treatments are counted twice.", "what_survives": "The effect of social-pressure mail stays large under either choice, and the pooled estimate moves by 1\\% to 8\\%. The pooled estimates for conventional and gratitude mail are unaffected, and no other pooled estimate of pp.~31--33 uses these rows. The overlap adds to the dependence between contrasts that the printed interval omits (\\fid{P042-E001}), and I did not establish a corrected interval."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-N005", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of six lines of text on page 32. Red boxes mark the words 71 canvassing treatments and the words 2.536, with a 95\\% confidence interval of (1.817, 3.255) in the sentence: Analyzing the effects of 71 canvassing treatments reveals a weighted average CACE of 2.536, with a 95\\% confidence interval of (1.817, 3.255).", "caption": "Page 32 (PDF p. 8), third paragraph: the count, the pooled estimate and the interval for canvassing, marked.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "The 71 canvassing estimates include the 1998 New Haven experiment twice, as first reported in 1999 and as corrected in 2005; with the superseded estimate removed, the 70 remaining estimates have a weighted average CACE of 2.434 with nominal 95% interval (1.706, 3.163), not 2.536 (1.817, 3.255).", "id": "P042-N006", "location": "p. 32 (PDF p. 8), third paragraph, first sentence; replication dataset, rows 10 and 11, where the header is row 1.", "pdf_page": 8, "printed_page": "32", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Analyzing the effects of \\hl{71} canvassing treatments reveals a weighted average CACE of $\\hl{2.536}$, with a 95\\% confidence interval of $\\hl{(1.817, 3.255)}$.''\n\\begin{audittable}{rlrrrrl}{Replication dataset, rows 10 and 11, as stored.}\nRow & Author & Year & Canvass ATT & Canvass SE & N & Election/Intervention date \\\\\n10 & Gerber, Green & 2005 & 8.4 & 2.6 & 29380 & 1998 general New Haven \\\\\n11 & Gerber, Green & 1999 & \\hl{6.33} & \\hl{2.27} & 28430 & 1998 general New Haven \\\\\n\\end{audittable}", "corrected": "``Analyzing the \\hl{70 remaining} canvassing \\hl{estimates} reveals a weighted average CACE of $\\hl{2.434}$, with a \\hl{nominal} 95\\% confidence interval of $\\hl{(1.706, 3.163)}$.'' Row 11 of the dataset is removed from the pool and row 10 is kept.", "evidence": "The evidence is a recalculation from the replication files in exact arithmetic. The 71 rows reproduce the printed 2.536 (1.817, 3.255). Without row 11, I obtain 2.4344 with standard error 0.3717 and nominal interval (1.7059, 3.1629) from the 70 remaining rows. That the two rows are one experiment rests on their election fields, on the abstracts stored with them and on p.~31 of the article, which identifies the 1998 New Haven study. I did not read the two source articles. A second pair, for a Detroit experiment of 2001 in rows 9 and 16, is not established as a duplicate, because the fields of row 9 conflict (\\fid{P042-E003}). Removing row 9 as well would give 2.410 (1.681, 3.139) over 69 estimates.", "problem": "Rows 10 and 11 of the dataset are two analyses of one experiment, the canvassing arm of the 1998 New Haven study. Row 11 is the report of 1999, ``Does canvassing increase voter turnout? A field experiment'', and row 10 is the correction of 2005, ``Correction to Gerber and Green (2000), Replication of Disputed Findings, and Reply to Imai (2005)''. The canvassing command of the replication program pools every row that has a canvassing estimate, and both rows enter. The experiment receives 2.31 times the weight of the corrected estimate alone, and a superseded estimate is pooled as if it were a separate treatment.", "what_survives": "The estimate for canvassing stays positive, with an interval above zero, and the qualitative comparison of tactics on p.~33 is unchanged. No other pooled estimate uses these rows. The interval keeps the article's calculation, which is the subject of \\fid{P042-E001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-N006", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the paragraph on GOTV phone calls on page 32. Red boxes mark '0.156', 'eleven' and '-0.118 to 0.430' in the sentences 'Excluding two experiments that tested the effectiveness of a pre-recorded social pressure message, we find a weighted average effect of 0.156 percentage points among the remaining eleven treatments, which tested the effects of messages recorded by politicians, celebrities or local clergy. The confidence interval, which ranges from -0.118 to 0.430 percentage points, indicates that the treatment effects of this tactic are very close to zero.'", "caption": "p. 32 (PDF p. 8). The marked estimate, count and interval are those of the eleven estimates for pre-recorded calls; one of the eleven is coded as a social-pressure call in the replication dataset.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N010.png"}], "formal_release": null, "formal_statements": [], "headline": "The eleven estimates pooled as ordinary pre-recorded calls include one that the replication dataset codes as a social-pressure call; without it the weighted average is 0.113 percentage points, with interval $(-0.164, 0.391)$, over ten treatments, in place of 0.156, with interval $(-0.118, 0.430)$, over eleven.", "id": "P042-N010", "location": "p. 32 (PDF p. 8), paragraph beginning “Studies that assess the effects of GOTV phone calls”, third and fourth sentences", "pdf_page": 8, "printed_page": "32", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Excluding two experiments that tested the effectiveness of a pre-recorded social pressure message, we find a weighted average effect of \\hl{0.156} percentage points among the remaining \\hl{eleven} treatments, which tested the effects of messages recorded by politicians, celebrities or local clergy. The confidence interval, which ranges from $\\hl{-0.118}$ to \\hl{0.430} percentage points, indicates that the treatment effects of this tactic are very close to zero.''", "corrected": "``\\hl{After excluding the social-pressure entries, the remaining ten pre-recorded-call treatment estimates have} a weighted average effect of \\hl{0.113} percentage points, with a 95\\% confidence interval from $\\hl{-0.164}$ to \\hl{0.391} percentage points. \\hl{This summary continues to indicate a small average effect of conventional recorded calls.}''\n\nHere the weighted average is the fixed effects average with weights $s_i^{-2}$, as on p.~31, and the ten estimates are the eleven of the replication program without the estimate of Gerber, Green, Kaplan and Kern (2010). In the replication program, the selection of pre-recorded-call estimates takes the further condition \\texttt{socialpressure != 1}.", "evidence": "The evidence is a recalculation from the replication files in exact rational arithmetic. Let $b_i$ denote the estimates and $s_i$ their standard errors, in percentage points, and set $w_i=s_i^{-2}$. The fixed effects average and its standard error are\n\\[\n\\widehat b=\\frac{\\sum_i w_i b_i}{\\sum_i w_i},\\qquad \\operatorname{SE}(\\widehat b)=\\Bigl(\\sum_i w_i\\Bigr)^{-1/2},\n\\]\nand the 95\\% interval is $\\widehat b\\pm z\\operatorname{SE}(\\widehat b)$ with $z\\approx1.95996398454$. For the eleven estimates, I have $\\sum_i w_i=51.25488$ and $\\sum_i w_i b_i=8.00301$. Then $\\widehat b=0.15614$, with standard error 0.13968 and interval $(-0.11763, 0.42991)$, which is the printed figure. The estimate of Gerber, Green, Kaplan and Kern has $w=0.89^{-2}=1.26247$ and $wb=2.33556$. Removing it, I obtain $\\sum_i w_i=49.99241$ and $\\sum_i w_i b_i=5.66744$, and therefore\n\\[\n\\widehat b=0.11337,\\qquad \\operatorname{SE}(\\widehat b)=0.14143,\\qquad \\widehat b\\pm z\\operatorname{SE}(\\widehat b)=(-0.16384,\\ 0.39057).\n\\]\nThe removed estimate holds 2.46\\% of the weight of the eleven, and the average falls by 0.043 points.", "problem": "The paragraph sets the social-pressure calls aside and reports the average effect of the remaining pre-recorded calls, which it describes as messages recorded by politicians, celebrities or local clergy. The replication program forms this set by dropping two entries, titled ``Robo SP1'' and ``Robo SP2'', from the thirteen estimates for pre-recorded calls. Among the eleven that remain is the estimate of Gerber, Green, Kaplan and Kern (2010), which is 1.85 with standard error 0.89 and which the replication dataset codes as a social-pressure call. This is the study cited on p.~36 for the statement that pre-recorded calls with a social-pressure message increase turnout (see \\fid{P042-A002}). The printed average is not the average for ordinary pre-recorded calls that the paragraph describes. The arithmetic is not at fault. From the eleven estimates, I find 0.156 and $(-0.118, 0.430)$, as printed, and it is true that two entries were excluded.\n\nThree points are not established. The replication files do not show whether the three entries coded or titled as social pressure are three separate experiments, or whether the estimate of Gerber, Green, Kaplan and Kern repeats one of the two excluded entries. They do not show who recorded its message. They also do not show whether the stored 1.85 is an effect of assignment, as the paragraph says of the eleven, or an effect among those reached.", "what_survives": "The conclusion of the paragraph stands: the average effect of ordinary pre-recorded calls is small, and its interval contains zero. The count, the estimate and the interval on p.~32 change, and the rough conversion to a CACE in Note~5, which divides the effect of assignment by 0.8, starts from 0.113. The pooled figures for mail, canvassing, commercial calls and volunteer calls use other estimates and do not change. I do not change the count of two experiments in Note~5, since the replication files do not show that there were three. The figure that Note~5 gives for those two is the subject of \\fid{P042-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-N010", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the top of page 33. Red boxes mark '1.936', '(1.298, 2.575)' and '37' in the sentence 'Excluding experiments that conducted follow-up calls with respondents who had previously indicated an intention to vote (see below), volunteer phone banks generate an average CACE of 1.936 percentage points, with a 95 percent CI = (1.298, 2.575), over 37 experimental treatments.' The sentence before it says that live calls from volunteer phone banks seem to be more effective than commercial calls, whose average CACE is 0.980.", "caption": "p. 33 (PDF p. 9), first paragraph. The marked estimate, interval and count are those of the 37 estimates for volunteer calls, one of which is the overall effect of a two-round campaign with follow-up calls.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N011.png"}, {"alt": "Chart with five horizontal intervals under the fixed effects independence model on an axis in percentage points of turnout, from -0.5 to 3. Commercial calls, 25 treatments, grey diamond: 0.980 (0.504, 1.456). Volunteer calls as printed, 37 treatments, blue circle: 1.936 (1.298, 2.575). Volunteer calls corrected, 36 treatments, orange square: 1.710 (1.064, 2.357). Volunteer minus commercial with the 37 treatments, blue circle: 0.956 (0.160, 1.752). Volunteer minus commercial with the 36 treatments, orange square: 0.730 (-0.072, 1.533), an interval that crosses the vertical line at zero.", "caption": "The pooled CACE for volunteer calls as printed and without the OCAPICA June 2008 estimate, beside the pooled CACE for commercial calls, and the difference between the volunteer and commercial figures in each case, with 95% intervals under the fixed effects independence model. Without the estimate, the interval for the difference includes zero.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N011-pooled.png"}], "formal_release": null, "formal_statements": [], "headline": "The 37 estimates for volunteer calls include the overall effect of a two-round campaign with follow-up calls, which the sentence says is excluded; without it the pooled CACE is 1.710, with 95% interval (1.064, 2.357), over 36 treatments, and under independence the pooled figures no longer separate volunteer from commercial calls at the two-sided 5% level.", "id": "P042-N011", "location": "p. 33 (PDF p. 9), first paragraph, sentence beginning “Excluding experiments that conducted follow-up calls”", "pdf_page": 9, "printed_page": "33", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Live calls from volunteer phone banks seem to be more effective. Excluding experiments that conducted follow-up calls with respondents who had previously indicated an intention to vote (see below), volunteer phone banks generate an average CACE of \\hl{1.936} percentage points, with a 95\\% CI = \\hl{(1.298, 2.575)}, over \\hl{37} experimental treatments.''", "corrected": "``Excluding \\hl{the OCAPICA June 2008 experiment, which} conducted follow-up calls with respondents who had previously indicated an intention to vote (see below), volunteer phone banks generate an average CACE of \\hl{1.710} percentage points, with a 95\\% CI = \\hl{(1.064, 2.357)}, over \\hl{36} experimental treatments.''\n\nThe average is the fixed effects average with weights $s_i^{-2}$, as on p.~31, over the 37 estimates of the replication program without the OCAPICA June 2008 estimate, and the interval assumes that the 36 estimates are independent. The comparison with commercial calls then reads: ``The volunteer pooled CACE is higher than the commercial pooled CACE by \\hl{0.730} percentage points, but under the fixed effects independence model its 95\\% difference interval is $\\hl{(-0.072, 1.533)}$; delivery-mode and authenticity explanations remain tentative.'' Any comparison with the effect of a volunteer phone call, or with the typical CACE of a single call if this pooled figure is meant, uses 1.710. Whether the other nine estimates from the same book meet the exclusion is not established, and I remove none of them.", "evidence": "The evidence has two parts. The first is the text of Michelson, García Bedolla and McConnell (2009), pp.~1556--1557, quoted above, available in indexed excerpts of a repository copy; the typeset Table~7 was not inspected. The second is a recalculation from the replication files using exact rational weights and sums. Let $y_i$ denote the estimates and $s_i$ their standard errors, in percentage points. I set $w_i=s_i^{-2}$, $W=\\sum_i w_i$ and $S=\\sum_i w_i y_i$. The fixed effects average is $m=S/W$, its variance under independence is $1/W$, and the 95\\% interval is $m\\pm zW^{-1/2}$ with $z\\approx1.95996$. For the 37 estimates, I have $W\\approx9.42573$ and $S\\approx18.25079$, giving an average that rounds to the printed 1.936. The OCAPICA estimate has $y_j=11.1$ and $s_j=2.1$, so that $w_j=100/441$ and $w_jy_j=370/147$. Removing it, I obtain\n\\[\nW'=W-\\frac{100}{441}\\approx9.19897,\\qquad S'=S-\\frac{370}{147}\\approx15.73379,\\qquad m'=\\frac{S'}{W'}\\approx1.71039,\n\\]\nwith standard error $(W')^{-1/2}\\approx0.32971$. The change in the average is\n\\[\nm-m'=\\frac{w_j}{W}\\,(y_j-m')\\approx0.22589 .\n\\]\nFor the difference $d$ between two pooled estimates with weight sums $W_A$ and $W_B$, independence implies that $\\operatorname{SE}(d)=(1/W_A+1/W_B)^{1/2}$, and I compare $z=d/\\operatorname{SE}(d)$ with the normal distribution. The commercial calls have weight sum 16.95194.\n\\begin{audittable}{lrrrr}{Pooled CACE in percentage points, and the difference between volunteer and commercial calls, from the replication files.}\nQuantity & Treatments & Estimate & SE & 95\\% interval \\\\\nCommercial calls & 25 & 0.980 & 0.243 & (0.504, 1.456) \\\\\nVolunteer calls, as printed & 37 & 1.936 & 0.326 & (1.298, 2.575) \\\\\nVolunteer calls, corrected & 36 & 1.710 & 0.330 & (1.064, 2.357) \\\\\nVolunteer minus commercial, with the 37 & & 0.956 & 0.406 & (0.160, 1.752) \\\\\nVolunteer minus commercial, with the 36 & & 0.730 & 0.410 & $(-0.072, 1.533)$ \\\\\n\\end{audittable}\nThe two differences have $z=2.353$ and $z=1.783$, with two-sided $p$-values 0.0186 and 0.0746.", "problem": "The sentence says that the pooled estimate for volunteer phone banks leaves out the experiments in which callers made follow-up calls to respondents who had said that they intended to vote. The 37 estimates pooled by the replication program include one from this class, the estimate for the OCAPICA phone bank in the election of June 2008. It is 11.1 with standard error 2.1, and the replication dataset lists its source as the book of García Bedolla and Michelson (2012). The campaign is described by Michelson, García Bedolla and McConnell (2009), the study of two-round phone banks that the article discusses on pp.~34--35. First-round respondents who intended to vote ``were targeted for reminder calls just before Election Day'' (p.~1556), and Table~7 of that study (p.~1557) reports 11.1 (2.1) as ``the effect of one or more contacts'', that is, the effect of the two-round campaign as a whole. The estimate lies 9.4 points above the average of the other 36, and with it the pooled CACE is too high by 0.226 points.\n\nThe comparison drawn in the paragraph is affected as well. The article says that volunteer calls ``seem to be more effective'' than commercial calls, whose pooled CACE is 0.980. Under the independence model that the pooling uses, the difference between the two pooled estimates is 0.956 with $z=2.35$ when the estimate is included, and 0.730 with $z=1.78$ when it is removed. The pooled figures then no longer separate volunteer from commercial calls at the two-sided 5\\% level. The article prints no such test, and it words the comparison tentatively. The correction weakens the numerical support for the comparison, and for the reading on p.~33 that the effect of a live call depends on whether the message is ``delivered in a routinized way by a commercial phone bank or in a more authentic manner by a volunteer phone bank''.\n\nOne point is not established. Nine other estimates among the 37 come from the same book. The replication dataset contains neither the number of rounds of calls nor the contrast behind each estimate, and therefore does not show whether any of the nine is also the overall effect of a two-round campaign. The corrected figure removes the one estimate known to belong to the excluded class.", "what_survives": "The pooled estimate for volunteer calls remains positive, with a 95\\% interval above zero under independence, and its point estimate remains above that of commercial calls. The figures for commercial calls, canvassing, mail and pre-recorded calls do not use the removed estimate and are not changed by this correction. The experiments with follow-up calls reported on pp.~34--35 are also unchanged. If the statement on p.~35 that the CACE in those experiments is ``more than twice as large as the typical CACE of a single phone call'' was valid before correction, it remains valid when the typical CACE is this pooled figure, since the bound to exceed falls from 3.873 to 3.421. The text-message CACE of 4.1 on p.~33, which ``rivals the effect of a volunteer phone call'', is now compared with 1.710. The count of 37 and the estimate 1.936 change, and under independence the pooled figures no longer separate volunteer from commercial calls at the two-sided 5\\% level."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-N011", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of a paragraph on page 34. Red boxes mark 'in 2006 and 2008' and '(Michelson et al., 2007)' in the sentence 'A series of experiments directed at minority voters in California in 2006 and 2008 turned up no evidence of interactions between mailings and phone calls (Michelson et al., 2007).'", "caption": "p. 34 (PDF p. 10), second paragraph of the section on synergy. The marked years and the marked citation do not agree.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N012.png"}, {"alt": "Crop of the reference list on page 46. A red box marks the year '(2007)' in the entry 'Michelson, Melissa R., Bedolla, Lisa García and Green, Donald P. (2007) New experiments in minority voter mobilization, Report for The James Irvine Foundation.'", "caption": "Reference list, p. 46 (PDF p. 22). The cited report is dated 2007.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N012-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The sentence dates the California experiments to 2006 and 2008 but cites only a report dated 2007, which cannot cover experiments of 2008.", "id": "P042-N012", "location": "p. 34 (PDF p. 10), section “Synergy, Diminishing Returns, or Neither?”, second paragraph; reference entry on p. 46 (PDF p. 22)", "pdf_page": 10, "printed_page": "34", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``A series of experiments directed at minority voters in California \\hl{in 2006 and 2008} turned up no evidence of interactions between mailings and phone calls \\hl{(Michelson et al., 2007)}.''\n\nThe reference list has one matching entry: ``Michelson, Melissa R., Bedolla, Lisa García \\& Green, Donald P. (2007) New experiments in minority voter mobilization, Report for The James Irvine Foundation.''", "corrected": "``A series of experiments directed at minority voters in California \\del{in 2006 and 2008} turned up no evidence of interactions between mailings and phone calls (Michelson et al., 2007).''\n\nAlternatively, the sentence keeps the years and adds a later source for the experiments of 2008, once that source has been checked.", "evidence": "The evidence is a check of the two printed pages: the sentence on p.~34 and the single matching entry of the reference list on p.~46.", "problem": "A report issued in 2007 cannot report experiments conducted in 2008, so the years in the sentence and the citation do not agree. Which of the two should change is not established, since I have read neither the report nor a later source for the experiments of 2008. The statement that the experiments showed no interactions is not in question.", "what_survives": "The conclusion of the section, that interactions between treatments tend to be weak, rests on several other studies that the paragraph discusses separately, and it is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-N012", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of a paragraph on page 39. A red box marks the word 'schooling' in the sentence 'For example, a growing literature attempts to gauge the causal effect of schooling on voter turnout by tracing the downstream consequences of random interventions that increase educational attainment (Sondheimer \\& Green, 2010) or family income (De La O, 2012; Doherty et al., 2006).' The sentence before it reads 'Other attributes, such as education or affluence, are also susceptible to experimental manipulation.'", "caption": "p. 39 (PDF p. 15), last paragraph before the Notes. The marked word names one of the two attributes that the sentence goes on to discuss; the preceding sentence names both.", "url": "https://pmaronow.github.io/llm-audit/works/P042/exhibits/P042-N014.png"}], "formal_release": null, "formal_statements": [], "headline": "The sentence names only schooling as the cause under study, although it cites experiments that raise schooling and experiments that raise family income; “schooling or family income” is meant.", "id": "P042-N014", "location": "p. 39 (PDF p. 15), last paragraph before the Notes, sentence beginning “For example, a growing literature”", "pdf_page": 15, "printed_page": "39", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``For example, a growing literature attempts to gauge the causal effect of \\hl{schooling} on voter turnout by tracing the downstream consequences of random interventions that increase educational attainment (Sondheimer \\& Green, 2010) or family income (De La O, 2012; Doherty et al., 2006).''", "corrected": "``For example, a growing literature attempts to gauge the causal effect of schooling \\hl{or family income} on voter turnout by tracing the downstream consequences of random interventions that increase educational attainment (Sondheimer \\& Green, 2010) or family income (De La O, 2012; Doherty et al., 2006).''", "intended_meaning": "The preceding sentence reads: ``Other attributes, such as education or affluence, are also susceptible to experimental manipulation.'' The example covers both attributes. Interventions that raise educational attainment are used to study the effect of schooling, and interventions that raise family income are used to study the effect of income. The sentence does not propose income interventions as a way to measure the effect of schooling. One point is left open. Doherty et al.\\ (2006) is, by its title in the reference list, a study of personal income and attitudes toward redistribution, and its abstract describes effects on political attitudes. I could read only the abstract and cannot say whether the study reports a turnout outcome. This decides whether the citation fits a sentence about voter turnout."}, "url": "https://pmaronow.github.io/llm-audit/works/P042/#P042-N014", "work": {"arxiv_id": null, "doi": "10.1080/17457289.2012.728223", "title": "Field Experiments and the Study of Voter Turnout", "url": "https://pmaronow.github.io/llm-audit/works/P042/"}, "work_id": "P042"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last seven lines of the left column of page 472. A red box marks the last line, 'then we can estimate the local average treatment'. The two lines above it read 'assume that the treatment only influences support for civil liberties indirectly by influencing knowledge,'.", "caption": "Statistical Model, p. 472 (PDF p. 10), foot of the left column. The marked line begins the conclusion drawn from random assignment and the exclusion restriction; it continues at the top of the right column.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002.png"}, {"alt": "Crop of the first four lines of the right column below Table 3. Red boxes mark the words 'effect of knowledge by using randomly assigned curriculum as an instrumental variable', which are followed by the citation '(Angrist, Imbens, and Rubin 1996)'.", "caption": "p. 472 (PDF p. 10), top of the right column, below Table 3. The marked words complete the sentence: the effect of knowledge is to be estimated with the curriculum as an instrument.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002-b.png"}, {"alt": "Crop of the last seven lines of the right column of page 474. A red box marks the last line, 'effects of increases in knowledge are identified only'. The lines above it attribute the interpretation to the work of Angrist, Imbens, and Rubin (1996).", "caption": "Conclusion, p. 474 (PDF p. 12), foot of the right column. The marked line begins the statement that effects of knowledge are identified in a subgroup; it continues on p. 475.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002-c.png"}, {"alt": "Crop of the running head and the first seven lines of the left column of page 475. Red boxes mark the words 'among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum.' The next sentence begins 'It may be that support for civil liberties among students who are affected on the margin'.", "caption": "Conclusion, p. 475 (PDF p. 13), top of the left column. The marked words name the subgroup: the students whose knowledge the curriculum raises.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E002-d.png"}], "formal_release": null, "formal_statements": [{"id": "P043-T4", "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T4"}, {"id": "P043-T5", "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-T5"}], "headline": "Random assignment and the exclusion restriction do not by themselves give a local effect of knowledge among the students whose knowledge the curriculum raises; monotonicity of knowledge in the assignment and a positive first stage suffice to justify this interpretation.", "id": "P043-E002", "location": "Statistical Model, p. 472 (PDF p. 10), foot of the left column to the top of the right column; Conclusion, pp. 474–475 (PDF pp. 12–13)", "pdf_page": 10, "printed_page": "472", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P043-T4", "P043-T5"], "text": {"as_printed": "p.~472: ``If we assume that the treatment only influences support for civil liberties indirectly by influencing knowledge, \\hl{then we can estimate the local average treatment effect of knowledge} by using randomly assigned curriculum as an instrumental variable (Angrist, Imbens, and Rubin 1996).''\n\npp.~474--475: ``This line of argument takes notice of the fact that the statistical effects of increases in knowledge \\hl{are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum}.''", "corrected": "p.~472: ``If we assume that the treatment only influences support for civil liberties indirectly by influencing knowledge, \\hl{and that the treatment lowers no student's knowledge and raises knowledge on average,} then we can estimate the local average treatment effect of knowledge by using randomly assigned curriculum as an instrumental variable (Angrist, Imbens, and Rubin 1996).''\n\npp.~474--475: ``This line of argument takes notice of the fact that\\hl{, if the treatment curriculum lowers no student's knowledge and raises knowledge on average,} the statistical effects of increases in knowledge are identified only among the subgroup of students whose knowledge of civil liberties is augmented by the treatment curriculum.''\n\nIn full, let $Z_i\\in\\{0,1\\}$ be the assignment of student $i$, let $K_i(z)$ be the student's knowledge score under assignment $z$, and let $Y_i(k)$ be the student's support at knowledge score $k$. Assume that $Z_i$ is independent of $(K_i(0),K_i(1),Y_i(\\cdot))$, that \\hl{$0<\\Pr(Z_i=1)<1$}, that assignment affects support only through knowledge, and that the observed scores are $K_i=K_i(Z_i)$ and $Y_i=Y_i(K_i)$. Assume \\hl{in addition that $K_i(1)\\ge K_i(0)$ for every student and that $\\mathbb{E}[K_i(1)-K_i(0)]>0$}. Then, for an integer knowledge score,\n\\[\n\\begin{aligned}\n\\frac{\\mathbb{E}[Y_i\\mid Z_i=1]-\\mathbb{E}[Y_i\\mid Z_i=0]}{\\mathbb{E}[K_i\\mid Z_i=1]-\\mathbb{E}[K_i\\mid Z_i=0]}\n&=\\sum_{j}w_j\\,\\mathbb{E}\\bigl[Y_i(j)-Y_i(j-1)\\mid K_i(0)<j\\le K_i(1)\\bigr],\\\\\nw_j&=\\frac{\\Pr(K_i(0)<j\\le K_i(1))}{\\mathbb{E}[K_i(1)-K_i(0)]},\n\\end{aligned}\n\\]\nwhere the sum is over the score levels $j$ with $\\Pr(K_i(0)<j\\le K_i(1))>0$. The weights $w_j$ are nonnegative and sum to one. Only students whose knowledge the curriculum raises enter the conditional means, and a student who crosses several levels contributes at each of them. Another sufficient restriction on the response of knowledge to the curriculum may replace monotonicity. Without such a restriction, random assignment and exclusion do not identify an effect confined to the students whose knowledge the curriculum raises. No numerical estimate is corrected.", "evidence": "The evidence is an exact counterexample and a derivation. Consider a population with two types of student, in which every knowledge score lies inside the 2008 range of 0 to 18 and every support score inside the range of 0 to 7. The curriculum is assigned with probability $1/2$, independently of type. I write $Z$ for the assignment and $K$ and $Y$ for the observed knowledge and support.\n\n\\begin{audittable}{lccccc}{A population in which the instrumental-variable ratio is zero although knowledge raises support.}\nType & Share & $K_i(0)$ & $K_i(1)$ & $Y_i(8)$ & $Y_i(9)$ \\\\\nKnowledge rises & $2/3$ & 8 & 9 & 3 & 4 \\\\\nKnowledge falls & $1/3$ & 9 & 8 & 2 & 4\n\\end{audittable}\n\nSupport depends on the assignment only through knowledge, as the exclusion restriction requires. By the table, I have $\\mathbb{E}[K\\mid Z=0]=25/3$ and $\\mathbb{E}[K\\mid Z=1]=26/3$, and the first stage is $1/3>0$. I also have $\\mathbb{E}[Y\\mid Z=0]=\\mathbb{E}[Y\\mid Z=1]=10/3$. Then the reduced form is $0$ and $\\beta_{\\mathrm{IV}}=0$. However, every student whose knowledge rises has a causal effect of $4-3=1$ on support. The numerator vanishes by cancellation: $(2/3)(1)+(1/3)(-2)=0$.\n\nOther features of the observed data do not identify this effect in general either. Two further populations, each with five equally likely types, have the same joint distribution of assignment, knowledge and support, with first stage $1/5$ and reduced form $0$. The average effect among the students whose knowledge rises is $1$ in the first population and $4/3$ in the second.\n\nUnder monotonicity, the corrected statement follows by telescoping. For an integer score,\n\\[\n\\begin{aligned}\nY_i(K_i(1))-Y_i(K_i(0))&=\\sum_{j}\\mathbf{1}\\{K_i(0)<j\\le K_i(1)\\}\\,[Y_i(j)-Y_i(j-1)],\\\\\nK_i(1)-K_i(0)&=\\sum_{j}\\mathbf{1}\\{K_i(0)<j\\le K_i(1)\\}.\n\\end{aligned}\n\\]\nThe expectation of the $j$th term of the first sum is $\\Pr(K_i(0)<j\\le K_i(1))$ times the mean of $Y_i(j)-Y_i(j-1)$ among the students such that $K_i(0)<j\\le K_i(1)$. Taking expectations of both identities and dividing the first by the second, I obtain the weighted average in the corrected statement.", "problem": "The first sentence obtains a local average treatment effect of knowledge from two conditions: the curriculum is randomly assigned, and it affects support only through knowledge. The second sentence reads that effect as one identified among the students whose knowledge the curriculum raises. The two conditions do not give this reading. Let $K_i(z)$ be the knowledge score of student $i$ under assignment $z\\in\\{0,1\\}$, and let $Y_i(k)$ be the support of student $i$ at knowledge score $k$. Under the two conditions, the ratio that the instrumental-variable estimator targets is\n\\[\n\\beta_{\\mathrm{IV}}=\\frac{\\mathbb{E}[Y_i(K_i(1))-Y_i(K_i(0))]}{\\mathbb{E}[K_i(1)-K_i(0)]}\n\\]\nwhen the denominator is positive, and nothing fixes the sign of $K_i(1)-K_i(0)$. A student whose knowledge the curriculum lowers enters both expectations and can cancel the contribution of the students whose knowledge it raises. The reading needs a restriction on how knowledge responds to the curriculum, such as monotonicity: $K_i(1)\\ge K_i(0)$ for every student. The paper cites the framework in which this condition is stated but does not state it, and a positive effect on mean knowledge does not imply it.", "what_survives": "The paper reports no instrumental-variable estimate. It chooses the reduced-form regressions described on p.~473, and Table 4 and the online appendix report coefficients on the curriculum. Those estimates and their standard errors are unaffected, as are the findings that the curriculum raised mean knowledge and had no detected effect on support. The interpretation changes. Without monotonicity, a zero effect of the curriculum on support does not show that knowledge has no effect on support among the students whose knowledge the curriculum raised. Monotonicity is an assumption about every student, and the paper does not test it. The check does not show that the curriculum lowered any student's knowledge."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E002", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index with an asterisk, and Support for Civil Liberties Index without an asterisk. The rows are Number of Questions (9, 7, 6), Mean (2010 Survey) (5.36, 3.46, 3.77), Standard Deviation (2010 Survey) (2.31, 1.90, 1.33), Cronbach's Alpha (2010 Survey) (0.70, 0.64, 0.43), Mean (2009 Survey) (4.38, 3.17, 4.06), Standard Deviation (2009 Survey) (2.28, 1.67, 1.27) and Cronbach's Alpha (2009 Survey) (0.66, 0.50, 0.31). A red outline surrounds the three entries 4.06, 1.27 and 0.31 in the last column. Under the table: asterisk, Index consists of a subset of questions originally included. Notes: All questions rescaled to 0 to 1 for additive index. N is 251 for all indices in 2010 Survey. For the 2007 to 2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey.", "caption": "Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2009 Indices”, with the notes under the table. The outline marks the three 2009 statistics of the Support for Civil Liberties Index, which are those of the seven-item index. The heading of that column has no asterisk.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E003.png"}, {"alt": "Crop of Table A5, Index Correlations on 2010 Survey, a nine by nine matrix whose rows and columns are the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index, each for 2007, 2008 and 2009. A red outline surrounds the first eight entries of the last column: 0.218, 0.176, 0.232, 0.088, 0.164, 0.223, 0.346 and 0.289. The ninth entry, on the diagonal, is 1.000. The note under the table reads: Lower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387).", "caption": "Online Appendix, p. 9, Table A5 with its note. The outline marks the eight correlations of the 2009 Support for Civil Liberties Index in the upper triangle.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The 2009 support statistics of Table A4 and eight correlations of Table A5 are those of the seven-item index, where the notes promise the six items asked again in 2010; the six-item values are mean 3.76, standard deviation 1.25 and alpha 0.38.", "id": "P043-E003", "location": "Online Appendix, p. 8, Table A4, Posttreatment Survey 2009 Indices, column “Support for Civil Liberties Index”, and the note under the table; p. 9, Table A5, upper triangle, last column", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table A4, Posttreatment Survey 2009 Indices, Support for Civil Liberties Index} (no asterisk). Number of Questions 6; Mean (2010 Survey) 3.77; Standard Deviation (2010 Survey) 1.33; Cronbach's Alpha (2010 Survey) 0.43; Mean (2009 Survey) \\hl{4.06}; Standard Deviation (2009 Survey) \\hl{1.27}; Cronbach's Alpha (2009 Survey) \\hl{0.31}.\n\n* Index consists of a subset of questions originally included. Notes: All questions rescaled to 0--1 for additive index. N is 251 for all indices in 2010 Survey. For the 2007--2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey.\n\n\\emph{Table A5, upper triangle, column ``Support for Civil Liberties Index 2009'', first eight rows.} \\hl{0.218, 0.176, 0.232, 0.088, 0.164, 0.223, 0.346, 0.289}.\n\nLower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387).", "corrected": "\\emph{Table A4, Posttreatment Survey 2009 Indices.} Support for Civil Liberties Index\\hl{*}: Number of Questions 6; Mean (2009 Survey) \\hl{3.76}; Standard Deviation (2009 Survey) \\hl{1.25}; Cronbach's Alpha (2009 Survey) \\hl{0.38}. The 2010 entries are unchanged.\n\n\\emph{Table A5, upper triangle, column ``Support for Civil Liberties Index 2009'', first eight rows.} \\hl{0.206, 0.164, 0.221, 0.095, 0.155, 0.207, 0.371, 0.267}. The diagonal entry remains 1.000.\n\nThese values use the six-item score $D_6$ defined above, on the same 387 students. The seven-item outcome of the main analysis is unchanged. Alternatively, keep the printed values and revise the item counts and both notes to say that seven items of the 2009 survey are compared with six items of the 2010 survey; the comparison then no longer claims equivalent questions.", "evidence": "The evidence is a check of the pages and of the replication code, and a recalculation from the replication data in exact arithmetic. Let $q_{11}, q_{13}, q_{15}, q_{17}, q_{18}, q_{39b}, q_{44}$ denote the responses to the seven items, after the item mean is imputed for a response coded 9 as in the replication code, and define $w_{18} = 1 + (q_{18} - 1)/2$ and $w_{39b} = 1 + (5 - q_{39b})/4$. The seven-item index of the replication code and the six-item index that matches the 2010 survey are\n\\[\n\\begin{aligned}\nD_7 &= q_{11} + q_{13} + q_{15} + q_{17} + w_{18} + w_{39b} + q_{44} - 7, \\\\\nD_6 &= q_{11} + q_{13} + q_{15} + q_{17} + w_{18} + q_{44} - 6 = D_7 - (w_{39b} - 1).\n\\end{aligned}\n\\]\nThe code computes the 2009 entries of Tables A4 and A5 from $D_7$ and the 2010 entries from six items.\n\\begin{audittable}{lrrrr}{Support for Civil Liberties Index: the recalculated statistics. The first and third rows reproduce the printed column.}\nScore & $N$ & Mean & Standard deviation & Alpha \\\\\n2009 survey, seven items ($D_7$) & 508 & 4.0570 & 1.2732 & 0.3142 \\\\\n2009 survey, six items ($D_6$) & 508 & 3.7573 & 1.2512 & 0.3839 \\\\\n2010 survey, six items & 251 & 3.7703 & 1.3277 & 0.4334 \\\\\n\\end{audittable}\nNote that the sample of 387 students with all nine indices observed is the same for $D_6$ and $D_7$.\n\\begin{audittable}{lrrr}{Table A5, upper triangle, last column: the printed correlations, the recalculation with seven items and the recalculation with six.}\nRow & Printed & Seven items & Six items \\\\\nCivil Liberties Knowledge 2007 & 0.218 & 0.21747 & 0.20567 \\\\\nCivil Liberties Knowledge 2008 & 0.176 & 0.17635 & 0.16437 \\\\\nCivil Liberties Knowledge 2009 & 0.232 & 0.23163 & 0.22063 \\\\\nGeneral Political Knowledge 2007 & 0.088 & 0.08813 & 0.09451 \\\\\nGeneral Political Knowledge 2008 & 0.164 & 0.16396 & 0.15547 \\\\\nGeneral Political Knowledge 2009 & 0.223 & 0.22294 & 0.20694 \\\\\nSupport for Civil Liberties 2007 & 0.346 & 0.34610 & 0.37079 \\\\\nSupport for Civil Liberties 2008 & 0.289 & 0.28927 & 0.26673 \\\\\n\\end{audittable}\nSeven of the eight printed correlations agree with the seven-item recalculation to three decimals. The first is printed as 0.218 and recalculated as 0.217. I could not establish the cause of this difference of 0.001; the original data, before the privacy edit of 2019, are not available.", "problem": "The 2009 Support for Civil Liberties Index has seven items (Table 3, p.~472). The 2010 survey did not ask one of them, the question on allowing a Ku Klux Klan rally, and the 2010 version has six items, as the row ``Number of Questions'' says. By the note, the 2009 statistics in this column should be those of the same six items. The printed values 4.06, 1.27 and 0.31 are instead the statistics of the seven-item index, exactly as in Table 3. The same index enters the eight correlations in the last column of the upper triangle of Table A5, which the note to that table also describes as correlations for the equivalent questions. The heading of the column in Table A4 lacks the asterisk that marks a subset of the original questions.", "what_survives": "The seven-item 2009 outcome of Table 3, the treatment effects of Table 4 and the analyses of single items are unaffected, since the correction introduces a separate score on six items for the comparison in the appendix only. The lower triangle of Table A5 and the entries that do not involve 2009 support stand, and the figure on p.~11 of the Online Appendix concerns knowledge, not support. Footnote 10 (p.~471) cites the 2010 survey in support of the reliability estimates; for 2009 support, that comparison was not made on matched items."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E003", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Table 3, Description of Outcome Variables, with columns for the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index, and blocks of rows for the posttreatment surveys of 2007, 2008 and 2009. Red boxes mark three entries in the rows labelled Cronbach's Alpha: 0.62 and 0.66 in the first two columns of the 2008 block, and 0.68 in the first column of the 2009 block.", "caption": "Table 3, p. 472 (PDF p. 10). The marked entries are the three reliability coefficients corrected here: 0.62 and 0.66 for the two knowledge indices of 2008, and 0.68 for the civil liberties knowledge index of 2009.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E004.png"}], "formal_release": null, "formal_statements": [], "headline": "For the additive knowledge indices that the paper analyses, three reliability coefficients in Table 3 are 0.60, 0.65 and 0.67, not 0.62, 0.66 and 0.68.", "id": "P043-E004", "location": "Table 3, p. 472 (PDF p. 10), rows “Cronbach’s Alpha” of the 2008 and 2009 surveys", "pdf_page": 10, "printed_page": "472", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Table 3, rows ``Cronbach's Alpha''. Posttreatment Survey 2008: Civil Liberties Knowledge Index \\hl{0.62}, General Political Knowledge Index \\hl{0.66}. Posttreatment Survey 2009: Civil Liberties Knowledge Index \\hl{0.68}.", "corrected": "Table 3, rows ``Cronbach's Alpha''. Posttreatment Survey 2008: Civil Liberties Knowledge Index \\hl{0.60}, General Political Knowledge Index \\hl{0.65}. Posttreatment Survey 2009: Civil Liberties Knowledge Index \\hl{0.67}.\n\nThese are the raw coefficients of the items as scored in the indices, computed on the respondents with a complete index ($N=591$, $591$ and $508$). The outcome scores are unchanged.", "evidence": "This is a recalculation from the replication data in exact arithmetic. For $k$ items $x_1,\\dots,x_k$ with sum $y$, on the $n$ respondents with a complete index, the coefficient is\n\\[\n\\alpha=\\frac{k}{k-1}\\left(1-\\frac{\\sum_{j=1}^{k}\\operatorname{Var}(x_j)}{\\operatorname{Var}(y)}\\right).\n\\]\n\n\\begin{audittable}{lrrccc}{Cronbach's alpha of the three scales: as printed, for the items as scored, and with one item reversed.}\nScale & $n$ & $k$ & Printed & Items as scored & One item reversed \\\\\n2008 civil liberties knowledge & 591 & 18 & 0.62 & 0.601442718 & 0.618754678 (\\texttt{Cq20p}) \\\\\n2008 general political knowledge & 591 & 22 & 0.66 & 0.654451008 & 0.657504233 (\\texttt{Cq41p}) \\\\\n2009 civil liberties knowledge & 508 & 16 & 0.68 & 0.674405221 & 0.680637049 (\\texttt{Dq34p})\n\\end{audittable}\n\nReversing a binary item $x_j$ leaves every item variance unchanged and replaces $\\operatorname{Var}(y)$ by $\\operatorname{Var}(y)-4\\operatorname{Cov}(x_j,y-x_j)$, which raises alpha when the covariance is negative. The same calculation reproduces the other three knowledge coefficients of Table 3 (0.67, 0.48 and 0.61) with no reversal, and the sample sizes, means and standard deviations of all six knowledge indices.", "problem": "Table 3 describes the outcome indices, and each knowledge index is the sum of items scored 1 for a correct answer and 0 otherwise. For these sums, on the samples of Table 3, Cronbach's alpha is 0.60, 0.65 and 0.67. The printed 0.62, 0.66 and 0.68 are the values obtained when one item of each scale is reverse-scored: \\texttt{Cq20p}, \\texttt{Cq41p} and \\texttt{Dq34p}, in the names of the replication files. Each of these is the only item of its scale that covaries negatively with the sum of the others. The Stata program computes the three coefficients with the \\texttt{alpha} command and without the option \\texttt{asis}, which keeps the items as scored. The indices add all the items as scored. This is consistent with an automatic reversal of the three items. The rounded entries do not determine every detail of the original computation.", "what_survives": "The outcome indices, the sample sizes, means and standard deviations of Table 3 and the regressions of Table 4 do not use these coefficients and are unchanged. The knowledge scales still have higher alpha than the support scales (0.60 and 0.65 against 0.45 in 2008, and 0.67 against 0.31 in 2009). I also recalculated the support-scale coefficients; they reproduce the printed $0.37$, $0.45$ and $0.31$."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E004", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of one panel of a table. Under the heading General Political Knowledge Index, three pairs of columns are labelled 2007, 2008 and 2009 Posttreatment Survey. The Treatment row gives 0.201 (0.129), 0.141 (0.126), -0.180 (0.315), -0.163 (0.321), -0.308 (0.233) and -0.139 (0.237), with standard errors in parentheses. The Covariates row alternates No and Yes, the Strata row reads Yes throughout, and N is 387 in every column. A red outline surrounds the entry -0.308.", "caption": "Online Appendix, p. 10, Table A6, panel “General Political Knowledge Index”. The outline marks the 2009 coefficient without covariates.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E005.png"}], "formal_release": null, "formal_statements": [], "headline": "The coefficient $-0.308$ should read $-0.031$, with the printed standard error $0.233$.", "id": "P043-E005", "location": "Online Appendix, p. 10, Table A6, General Political Knowledge Index, 2009 Posttreatment Survey, column without covariates", "pdf_page": 10, "printed_page": "10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table A6, General Political Knowledge Index, 2009 Posttreatment Survey; Covariates: No; Strata: Yes; N 387.} Treatment $\\hl{-0.308}$ $(0.233)$.", "corrected": "\\emph{Table A6, General Political Knowledge Index, 2009 Posttreatment Survey; Covariates: No; Strata: Yes; N 387.} Treatment $\\hl{-0.031}$ $(0.233)$.", "evidence": "The evidence is a recalculation from the replication data. For stratum $s$ and arm $t$ (0 for control, 1 for treatment), let $n_{st}$ be the number of students among the 387 and $S_{st}$ the sum of their scores.\n\\begin{audittable}{lrrrr}{Students and sums of the 2009 General Political Knowledge Index among the 387 students, by stratum and arm.}\nStratum $s$ & $n_{s0}$ & $n_{s1}$ & $S_{s0}$ & $S_{s1}$ \\\\\nA & 8 & 9 & 33 & 34 \\\\\nB & 13 & 7 & 78 & 43 \\\\\nI & 15 & 30 & 83 & 148 \\\\\nJ & 16 & 26 & 64 & 107 \\\\\nK & 9 & 23 & 46 & 94 \\\\\nL & 6 & 3 & 25 & 14 \\\\\nM & 9 & 5 & 34 & 19 \\\\\nQ & 78 & 34 & 281 & 141 \\\\\nR & 14 & 15 & 86 & 83 \\\\\nS & 4 & 7 & 17 & 23 \\\\\nT & 28 & 28 & 154 & 159 \\\\\nTotal & 200 & 187 & 901 & 865 \\\\\n\\end{audittable}\nIn this cell the moment estimate of the variance between classrooms is negative ($-0.048$), and the estimator sets it to zero. The random-effects estimate is then the least squares coefficient of treatment in a regression with an intercept for each stratum. Solving the normal equations for the stratum intercepts and inserting them into the equation for the treatment coefficient $b$, I obtain\n\\[\nb = \\frac{\\sum_s (n_{s0} S_{s1} - n_{s1} S_{s0})/(n_{s0} + n_{s1})}{\\sum_s n_{s0} n_{s1}/(n_{s0} + n_{s1})} = -\\frac{49321068}{1600529849} \\approx -0.03082 .\n\\]\nThe residual sum of squares, with $387 - 12 = 375$ degrees of freedom, gives the standard error $0.23323$. The same calculation reproduces the eight other coefficients without covariates in Table A6 and all nine of their standard errors to three decimals. The conclusion does not depend on how the variance between classrooms is estimated. For every nonnegative ratio of that variance to the residual variance, the coefficient from generalized least squares lies strictly between $-1/5$ and $1/5$, which implies that no such ratio gives a value that rounds to $-0.308$.", "problem": "The cell reports the effect of treatment on the 2009 General Political Knowledge Index among the 387 students who completed all three posttreatment surveys, from a generalized least squares regression with random effects for classrooms and indicators for strata, without covariates. That specification gives $-0.0308$, which rounds to $-0.031$. The printed coefficient is ten times as large. The printed standard error is correct.", "what_survives": "The standard error of the cell stands, and the estimate is not significant at the 5\\% level with either value. Table 4 uses larger samples and does not change. Footnote 12 (p.~473) cites Table A6 for the robustness of the results among students present in all three waves; the corrected estimate is closer to zero. This recalculation covers the nine cells of Table A6 without covariates."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E005", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of running text. The text reads: allow the prosecutor to retry Mr. Jones in this case? (Yes/No) 12. Declare a law unconstitutional [Identification: Federal, state, or local branch of government] 13. Police must inform suspected criminals of their constitutional right to remain silent while being arrested. [Matching court case] 14. School districts are allowed to conduct random drug tests on any student involved in extracurricular activities. A red outline surrounds the words while being arrested.", "caption": "Mini Appendix, p. 2, list of the 2008 Civil Liberties Knowledge Index questions, item 13. The outline marks the phrase that places the warning at the time of arrest.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E006.png"}, {"alt": "Crop of four lines of running text. The text reads: of Rights does not safeguard individual liberties.) 8. As a result of the Supreme Court ruling in Miranda v. Arizona (1966), a person accused of a crime is entitled to: (a speedy trial/reasonable bail/a reading of his or her rights at the time of arrest/protection against cruel or unusual punishment.) 9. The clear and present danger ruling of the Supreme Court in Schenck v. United States illustrates the continuing. A red outline in two parts surrounds the words a reading of his or her rights at the time of arrest.", "caption": "Mini Appendix, p. 3, list of the 2009 Civil Liberties Knowledge Index questions, item 8. The outline marks the answer scored as correct; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Two questions of the knowledge indices place the Miranda warnings at the time of arrest; the decision requires them before custodial questioning.", "id": "P043-E006", "location": "Mini Appendix, p. 2 (pages unnumbered), “2008 Civil Liberties Knowledge Index Questions”, item 13; p. 3, “2009 Civil Liberties Knowledge Index Questions”, item 8", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{2008, item 13.} Police must inform suspected criminals of their constitutional right to remain silent \\hl{while being arrested}. [Matching court case]\n\n\\emph{2009, item 8.} As a result of the Supreme Court ruling in Miranda v. Arizona (1966), a person accused of a crime is entitled to: (a speedy trial/reasonable bail/\\hl{a reading of his or her rights at the time of arrest}/protection against cruel or unusual punishment.)", "corrected": "\\emph{2008, item 13.} Police must inform suspected criminals of their constitutional right to remain silent \\hl{before custodial questioning}. [Matching court case]\n\n\\emph{2009, item 8.} As a result of the Supreme Court ruling in Miranda v. Arizona (1966), a person accused of a crime is entitled to: (a speedy trial/reasonable bail/\\hl{warnings about his or her rights before custodial questioning}/protection against cruel or unusual punishment.)", "evidence": "The evidence is a check of the pages and of the opinion. The questionnaires and coding keys in the replication files confirm the wording and the keyed answers. The opinion of the Court in \\emph{Miranda v.\\ Arizona}, 384 U.S. 436 (1966), says at p.~444: ``By custodial interrogation, we mean questioning initiated by law enforcement officers after a person has been taken into custody or otherwise deprived of his freedom of action in any significant way.'' Its summary at p.~479 says: ``He must be warned prior to any questioning that he has the right to remain silent''. The quotations are from the \\href{https://tile.loc.gov/storage-services/service/ll/usrep/usrep384/usrep384436/usrep384436.pdf}{United States Reports}.", "problem": "In both questions the answer scored as correct concerns \\emph{Miranda v.\\ Arizona}, and both describe its rule by the moment of arrest. The opinion requires the warnings before custodial interrogation, that is, before law enforcement questions a person in custody or otherwise significantly deprived of freedom of action. Arrest alone does not trigger the requirement. The two keyed statements overstate the timing and the scope of the rule. In each question the Miranda answer remains the only plausible choice.", "what_survives": "The intended answer is recognizable in both questions. No change in the responses, in the knowledge scores or in the estimated treatment effects has been shown, and none is claimed here."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E006", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of ten lines of text. The passage reads: footnote 10, which included most or all of the index questions on a single questionnaire (See Table A4). After correcting for the changing difficulty of the items over time, we find that the control group means were essentially unchanged during the three postintervention waves, while the treatment group showed an initial surge followed by a gradual decline to the control group mean. The treatment had no effect, positive or negative, on general political knowledge. The estimates hover; the crop ends in the middle of this sentence. Red outlines surround the words essentially unchanged during the three postintervention waves, which run over two lines.", "caption": "p. 473 (PDF p. 11), right column. The outlines mark “essentially unchanged during the three postintervention waves”, the description of the control group means that the appendix figure does not support.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E007.png"}, {"alt": "Plot of six points with vertical bars. The horizontal axis is the year, with ticks at 2007, 2008 and 2009. The vertical axis is the rescaled civil liberties knowledge score, with ticks from 0.7 to 1.1. In each year one point is labelled T and one is labelled C. The T points lie at about 1.06, 0.97 and 0.83. The C points lie at about 0.90, 0.88 and 0.81, and the third is lower than the first two. In 2009 the T and C points are close together and their bars overlap. Red outlines surround the three C points.", "caption": "Online appendix, p. 11: the plot of the unnumbered figure “Rescaled Civil Liberties Knowledge Score Across 2007, 2008 and 2009 Surveys”. T is the treatment group, C is the control group, and the bars are 95% confidence intervals. The outlines mark the three control group means, which lie at about 0.90, 0.88 and 0.81.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "After the paper’s normalization, the control group means are 0.896, 0.884 and 0.805 in 2007, 2008 and 2009: the description “essentially unchanged” omits a decline of about 10% by the third wave, while the gap between the treatment and control means shrinks from 0.161 to 0.089 to 0.022.", "id": "P043-E007", "location": "p. 473 (PDF p. 11), right column, sentence beginning “After correcting for the changing difficulty of the items over time”; the figure of rescaled knowledge scores, p. 11 of the online appendix, and Table A4, p. 8 of the online appendix", "pdf_page": 11, "printed_page": "473", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "After correcting for the changing difficulty of the items over time, we find that the control group means were \\hl{essentially unchanged during the three postintervention waves}, while the treatment group showed an initial surge followed by a gradual decline to the control group mean.", "corrected": "After applying this normalization, the observed control-group means were \\hl{approximately 0.896, 0.884 and 0.805 in 2007, 2008 and 2009, respectively}. The treatment-control gap \\hl{shrank from approximately 0.161 to 0.089 to 0.022} over these waves.\n\nThis wording replaces the whole sentence. A shorter repair keeps the sentence and replaces only its clause on the control group, which then reads ``the observed rescaled control-group means \\hl{declined, particularly in the third postintervention wave}''. Neither wording attributes the decline of the control means to forgetting or to a failure of the normalization.", "evidence": "The evidence is a recalculation from the public replication data in exact arithmetic, together with a check of the appendix figure and of Table~A4. Following the scoring rules of the Stata script in the replication files, I rebuilt the equivalent index of each wave (13, 17 and 9 questions) and the matching index on the 2010 survey. Each index is available for 251 of the 254 respondents in the public file of the 2010 survey, and the three totals over these respondents are $A_{2007} = 1904$, $A_{2008} = 2128$ and $A_{2009} = 1345$. Let $S_{ay}$ be the total of the equivalent index over the $n_{ay}$ students of group $a$ in the survey of year $y$. The mean plotted in the figure is\n\\[\nM_{ay} = \\frac{S_{ay}/n_{ay}}{A_y/251} = \\frac{251\\,S_{ay}}{n_{ay}A_y} .\n\\]\nFor the control group in 2007, I obtain $M_{C,2007} = (3492/514)/(1904/251) = 219123/244664$, which is 0.8956 to four decimals. A second calculation, written independently from the same files, gives the same fractions.\n\\begin{audittable}{llrrrrr}{Rescaled means of the equivalent Civil Liberties Knowledge Index, recalculated from the public data, with the gap between the treatment and control means. The first three rows use every student whose index is available in that wave, and the last three use the 387 students whose index is available in all three waves.}\nStudents & Year & Control $n$ & Control mean & Treatment $n$ & Treatment mean & Gap \\\\\nAll in the wave & 2007 & 514 & 0.8956 & 531 & 1.0566 & 0.1610 \\\\\nAll in the wave & 2008 & 296 & 0.8842 & 295 & 0.9728 & 0.0886 \\\\\nAll in the wave & 2009 & 253 & 0.8055 & 255 & 0.8277 & 0.0222 \\\\\nThe same 387 & 2007 & 200 & 0.9215 & 187 & 1.0772 & 0.1557 \\\\\nThe same 387 & 2008 & 200 & 0.8958 & 187 & 0.9682 & 0.0724 \\\\\nThe same 387 & 2009 & 200 & 0.8192 & 187 & 0.8632 & 0.0440 \\\\\n\\end{audittable}\nThe public files are a release edited for privacy; the numbers of students in the three waves (1045, 591 and 508) agree with Table~4. The 2010 means $A_y/251$ are 7.59, 8.48 and 5.36 to two decimals, and the pooled means of the three waves are 7.41, 7.87 and 4.38; these six values agree with Table~A4. The points of the appendix figure lie at the values of the first three rows: about 0.90, 0.88 and 0.81 for the control group, and about 1.06, 0.97 and 0.83 for the treatment group.\n\nThe control mean falls by 0.0114 from 2007 to 2008 and by 0.0901 from 2007 to 2009. Among the 200 control students whose index is available in all three waves, the decline from 2007 to 2009 is 0.1022, which is 11.09\\% of the 2007 value. Holding the set of students fixed therefore does not remove the decline. The gap is positive and decreasing in both sets of students. The treatment mean falls by 0.2289 from 2007 to 2009. Writing $G_y = M_{T,y} - M_{C,y}$ for the gap, I have\n\\[\nM_{T,2009} - M_{T,2007} = \\bigl(M_{C,2009} - M_{C,2007}\\bigr) + \\bigl(G_{2009} - G_{2007}\\bigr),\n\\]\nand the two terms on the right are $-0.0901$ and $-0.1388$. The identity relates observed means and has no causal interpretation. I ran no test of change over time that accounts for repeated students, classrooms and the uncertainty of the 2010 means. I do not use the confidence bars of the figure as such a test.", "problem": "The sentence summarizes the unnumbered figure on p.~11 of the online appendix. The three survey waves used different questions to measure knowledge. For each wave, the figure takes the equivalent index of Table~A4, which consists of the questions of that wave that were repeated on the 2010 survey. It divides this index by the mean of the same index on the 2010 survey. After this normalization, the control group means are 0.896 in 2007, 0.884 in 2008 and 0.805 in 2009. The control mean is nearly flat between the first two waves and then falls. Its decline from 2007 to 2009 is 0.090, which is 10.06\\% of its 2007 value and about 56\\% of the initial gap of 0.161 between the treatment and control means. The description ``essentially unchanged'' fits the first two waves, and the sentence extends it to the third, where it does not fit. The sentence then presents the treatment group as falling toward a control level that stayed fixed. The figure and the data show that both means fell and that the gap between them shrank. The paper states no tolerance for ``essentially'', so there is no numerical criterion to refute, and I do not read the phrase as exact equality. The error is one of description: the sentence reports a stable control baseline and omits a decline of this size, which the paper's own figure displays. The decline is a difference of observed means. It does not show that students forgot, that the normalization failed, or that a population mean changed.", "what_survives": "This correction leaves the appendix figure, its means and Table~A4 unchanged. The decline of the treatment advantage stands: the gap between the treatment and control means is positive and shrinks from 0.161 to 0.089 to 0.022. Each regression of Table~4 and of the appendix is fitted within one wave and does not assume a constant control mean across waves. The sentence is not an input to those estimates. I have not recalculated those estimates here. What changes is the description of the two trajectories. The control baseline was not stable after the normalization, and the data do not support describing the decline of the treatment group as a fall toward a fixed control level."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E007", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the left column. The text reads: ies that emphasize the importance of cognitive skills or the experience of diversity acknowledge the mediating role of political knowledge (Nie, Junn, and Stehlik-Berry 1996; Stouffer 1955), followed by a footnote mark 3 and the first line of the next paragraph. A red outline surrounds the word Stehlik-Berry.", "caption": "Introduction, p. 464 (PDF p. 2), left column, first paragraph. The outline marks the misspelled surname in the citation.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E008.png"}, {"alt": "Crop of the right column around equation (1). Two lines end with: adding dummy variables for all strata but one. Then: Thus, our model has the form, and the display Y sub i equals beta sub 0 plus beta sub 1 X sub i plus the sum over k from 1 to K minus 1 of gamma sub k D sub k plus u sub i, numbered (1). A red outline surrounds D sub k. Below, the text reads: where X sub i is scored 0 for control and 1 for treatment, the K-1 dummy variables marking each of the K strata are denoted D sub i, and u sub i denotes the unobserved disturbance term. A second red outline surrounds D sub i.", "caption": "Statistical Model, p. 471 (PDF p. 9), right column. The outlines mark the stratum indicator $D_k$ in equation (1) and $D_i$ in the sentence that explains it. The sentence above the display says that there is an indicator for all strata but one.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E008-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The cited surname is Stehlik-Barry, and the stratum indicators of equation (1), written $D_k$ in the display and $D_i$ in the sentence after it, should read $D_{ik}$ in both places.", "id": "P043-E008", "location": "Introduction, p. 464 (PDF p. 2), left column, first paragraph; Statistical Model, p. 471 (PDF p. 9), right column, equation (1) and the sentence after it", "pdf_page": 2, "printed_page": "464", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~464.} \\ldots even studies that emphasize the importance of cognitive skills or the experience of diversity acknowledge the mediating role of political knowledge (Nie, Junn, and \\hl{Stehlik-Berry} 1996; Stouffer 1955).$^{3}$\n\n\\emph{p.~471.} Thus, our model has the form:\n\\[\nY_i = \\beta_0 + \\beta_1 X_i + \\sum_{k=1}^{K-1} \\gamma_k \\hl{D_k} + u_i, \\qquad (1)\n\\]\nwhere $X_i$ is scored 0 for control and 1 for treatment, the $K$-1 dummy variables marking \\hl{each of the $K$ strata} are denoted $\\hl{D_i}$, and $u_i$ denotes the unobserved disturbance term.", "corrected": "\\emph{p.~464.} (Nie, Junn, and \\hl{Stehlik-Barry} 1996; Stouffer 1955).\n\n\\emph{p.~471.} Thus, our model has the form:\n\\[\nY_i = \\beta_0 + \\beta_1 X_i + \\sum_{k=1}^{K-1} \\gamma_k \\hl{D_{ik}} + u_i, \\qquad (1)\n\\]\nwhere $X_i$ is scored 0 for control and 1 for treatment, \\hl{the $K-1$ dummy variables for all strata except the reference stratum are denoted $D_{ik}$, where $D_{ik} = 1$ if student $i$ belongs to stratum $k$ and $D_{ik} = 0$ otherwise}, and $u_i$ denotes the unobserved disturbance term.\n\nHere $Y_i$ is one of the three outcome measures for student $i$, as the paper says after the display, and $K$ is the number of strata.", "evidence": "The evidence is a check of the pages: pp.~464 and 476 for the surname, and p.~471 for the display and the two sentences around it. The replication code enters the stratum as a categorical variable in the model, which is the specification written below. No recalculation was needed.", "problem": "There are two misprints. On p.~464 the surname of the third author is spelled ``Stehlik-Berry''. A second citation on the same page and the entry in the reference list on p.~476 have ``Stehlik-Barry''.\n\nIn equation (1) the stratum indicator is written $D_k$, with no index for the student, and the next sentence calls the same indicators $D_i$, with no index for the stratum. An indicator of membership in a stratum depends on both. The same sentence says that the $K-1$ indicators mark ``each of the $K$ strata''. They mark all strata except one, as the sentence before the display says (``adding dummy variables for all strata but one'').", "what_survives": "Nothing depends on either misprint. The intended model is clear from the sentence before the display, and no estimate or conclusion changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E008", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 696 to 698 in monospaced type with line numbers. Line 696 builds the data frame A from nine columns: BBRIknowE, CBRIknowE, DBRIknowE, BGENknowE, CGENknowE, DGENknowE, Btolerance, Ctolerance and Dtolerance. Line 697 names the nine columns. Line 698 reads cor(A, use=\"pairwise.complete.obs\"). A red outline surrounds use=\"pairwise.complete.obs\".", "caption": "Replication files, R script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 696–698, typeset from the file with line numbers added and long lines wrapped. The outline marks the option that selects pairwise deletion.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E009.png"}, {"alt": "Crop of three lines. The text reads: Lower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387). A red outline surrounds the words using listwise deletion (N = 387).", "caption": "Online Appendix, p. 9, the note under Table A5. The outline marks the deletion rule and the sample size that the R script should implement.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E009-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The R script computes the 2007–2009 correlations of Table A5 with pairwise deletion, where the appendix specifies listwise deletion ($N = 387$); the option should be complete.obs.", "id": "P043-E009", "location": "Replication files, R script Green_et_al_JP_2011_Public_Use_EDITED.R, line 698; the rule that the line should implement is in the note to Table A5, Online Appendix, p. 9", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{R script, line 698.} \\texttt{cor(A, use=\"\\hl{pairwise.complete.obs}\")}\n\n\\emph{Note to Table A5.} Lower triangular matrix indicates correlations in 2010 Survey (N = 251). Upper triangular matrix indicates correlations for the equivalent questions in the 2007-2009 surveys, using listwise deletion (N = 387).", "corrected": "\\emph{R script, line 698.} \\texttt{cor(A, use=\"\\hl{complete.obs}\")}", "evidence": "The evidence is an exact counterexample from the replication data. Let $x$ and $y$ denote the Civil Liberties Knowledge indices of 2007 and 2008. For a sample of size $N$, define $U = N \\sum xy - \\sum x \\sum y$, $V = N \\sum x^2 - (\\sum x)^2$ and $W = N \\sum y^2 - (\\sum y)^2$. Then the correlation is $r = U/\\sqrt{VW}$.\n\\begin{audittable}{lrr}{Civil Liberties Knowledge 2007 and 2008: the correlation under the two deletion rules.}\nQuantity & Pairwise deletion & Listwise deletion \\\\\n$N$ & 551 & 387 \\\\\n$\\sum x$ & 4124 & 2926 \\\\\n$\\sum y$ & 4330 & 3054 \\\\\n$\\sum x^2$ & 34292 & 24456 \\\\\n$\\sum y^2$ & 38220 & 27286 \\\\\n$\\sum xy$ & 34202 & 24401 \\\\\n$U$ & 988382 & 507183 \\\\\n$V$ & 1887516 & 902996 \\\\\n$W$ & 2310320 & 1232766 \\\\\n$r$ & 0.4733 & 0.4807 \\\\\n\\end{audittable}\nThe table prints 0.481, the listwise value. With \\texttt{complete.obs}, 32 of the 36 printed correlations of the upper triangle agree with the recalculation to three decimals, against none with pairwise deletion. The four others differ by 0.001, and the cause of these differences is not established.", "problem": "The data frame \\texttt{A} holds the nine indices of the 2007, 2008 and 2009 surveys whose correlations fill the upper triangle of Table A5. With the option \\texttt{pairwise.complete.obs}, each correlation is computed on the students observed on that pair of indices. These samples range from 413 to 1045 students. The common sample with all nine indices observed has 387. The printed table follows the listwise rule of its note, and the Stata script computes the same matrix with a command that deletes listwise. The defect is therefore in the R script alone. With the indices built as in the script, line 698 returns a matrix in which all 36 correlations differ from the listwise ones, by as much as 0.101.", "what_survives": "The sample size $N = 387$ in Table A5 agrees with the stated listwise rule. Line 698 only displays the matrix. Its result is not stored or used again, and the treatment-effect models do not depend on it. The correction concerns the deletion rule alone; the composition of the 2009 support index in the same matrix is the subject of \\fid{P043-E003}."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E009", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 843 to 849 in monospaced type with line numbers. Lines 843 to 845 reverse three items by subtracting them from 3. A red box marks line 848, which assembles the command that assigns to data.E Ctolerance the sum of the seven items joined by plus signs, with no subtraction.", "caption": "Replication files, R program Green_et_al_JP_2011_Public_Use_EDITED.R, lines 843–849, typeset from the file with line numbers added. The marked line builds the score as the sum of seven items coded from 1 to 2 and subtracts nothing.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E010.png"}, {"alt": "Listing of lines 773 to 777 in monospaced type with line numbers. Line 777 generates Ctolerance as the sum of the same seven items. A red box marks the term 'minus 7' at the end of that line.", "caption": "Replication files, Stata program Green_et_al_JP_2011_Public_Use_EDITED.do, lines 773–777, typeset from the file with line numbers added. The marked term is the subtraction that the R program omits.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E010-b.png"}, {"alt": "Crop of the block 'Posttreatment Survey 2008 Indices' of Table A4, with columns for the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index. A red box marks the entry 3.95 in the row 'Mean (2010 Survey)' of the support column. The same column shows 7 questions, standard deviation 1.44 and alpha 0.44 for the 2010 survey.", "caption": "Online appendix, Table A4, p. 8, block for the 2008 indices. The marked mean of the seven-item support index in the 2010 survey is correct as printed; the R program as deposited gives 10.95 for it.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E010-c.png"}], "formal_release": null, "formal_statements": [], "headline": "The R program in the replication files omits the subtraction of 7 when it builds the seven-item support score of the 2010 survey, and its score runs from 7 to 14 where the paper’s index runs from 0 to 7; the mean of 3.95 printed in Table A4 is correct.", "id": "P043-E010", "location": "Replication files, R program Green_et_al_JP_2011_Public_Use_EDITED.R, lines 847–849 (2010 survey, score Ctolerance); in the paper, Table 3, p. 472 (PDF p. 10), 2008 support index with range 0–7, and Table A4 of the online appendix, p. 8, mean 3.95", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "R program, lines 847--849:\n\n\\texttt{vars <- paste(\"seq\", c(\"28pW\", \"29p\", \"30pW\",\"34p\",\"35p\",\"36pW\", \"37p\"), sep=\"\")}\n\n\\texttt{command <- paste(\"data.E\\$Ctolerance <-\", \\hl{paste(\"data.E\\$\",vars,sep=\"\",collapse=\"+\")})}\n\n\\texttt{eval(parse(text=command))}\n\nTable A4 of the online appendix, p.~8, Posttreatment Survey 2008 Indices, Support for Civil Liberties Index: Number of Questions 7; Mean (2010 Survey) 3.95; Standard Deviation (2010 Survey) 1.44; Cronbach's Alpha (2010 Survey) 0.44.", "corrected": "R program, line 848:\n\n\\texttt{command <- paste(\"data.E\\$Ctolerance <-\", paste(\"data.E\\$\",vars,sep=\"\",collapse=\"+\")\\hl{, \"-7\"})}\n\nWith this line the score runs from 0 to 7, and its mean on the 2010 file is $3.950642$, which rounds to the 3.95 of Table A4. No entry of Table A4 changes.", "evidence": "The evidence is an exact identity and a recalculation. Let $u_1,\\dots,u_7$ be the item codes after the three reversals. Each lies in $[1,2]$. It is 1 or 2 for an answered item, and a value between 1 and 2 where a non-response coded 9 was replaced by the mean of the item. The R score is $S_R=\\sum_{j=1}^{7}u_j$, and the documented score is\n\\[\nS_0=\\sum_{j=1}^{7}(u_j-1)=S_R-7 .\n\\]\nFor example, the answers $(2,1,2,1,1,2,1)$ to items 28, 29, 30, 34, 35, 36 and 37 give $u_j=1$ for every $j$, an R score of 7 and a documented score of 0. On the 251 respondents of the 2010 file who have a score, the lines as deposited give a mean of $10.950642$ and a range of 8 to 14. With the subtraction the mean is $3.950642$ and the range is 1 to 7. The standard deviation is $1.4405$ and Cronbach's alpha is $0.4394$ in both cases. These values round to the 3.95, 1.44 and 0.44 printed in Table A4.", "problem": "The paper's support index for 2008 is a sum of seven items, each rescaled to lie between 0 and 1, and it runs from 0 to 7 (Table 3). In this part of the R program the seven items keep their original codes of 1 and 2. Three of them are reversed by $x\\mapsto 3-x$, which preserves these codes, and the program then adds the seven items without subtracting 7. The score that it computes is the documented score plus 7. The Stata program in the same files subtracts 7 (line 777), and so does the R program where it builds the same index for the main experiment (line 150).", "what_survives": "The mean of 3.95, the standard deviation of 1.44 and the alpha of 0.44 in Table A4 stand as printed. A constant shift leaves standard deviations, alpha and correlations unchanged, including the correlation matrix that the R program computes from this score. The support index of the main experiment is built separately, with the subtraction, and the models of Table 4 are fitted before the 2010 file is read. This line therefore affects none of the experimental results."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E010", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text. The passage reads: partitioning the sample by political interest, frequency of political discussion at home, party ID, age, mother's level of education, employment status and seven factual questions yielded no significant interactions in models predicting knowledge or support for civil liberties, followed by the footnote mark 14 and the first line of the next paragraph, Another interpretation is that knowledge and. A red outline surrounds the words seven factual questions.", "caption": "p. 475 (PDF p. 13), left column. The outline marks “seven factual questions”, the moderator that the replication files build as a sum of raw response codes.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E011.png"}, {"alt": "Crop of two survey questions with their codes. Question 19 reads: If Congress passes a law and the President vetoes it, Congress can enact it anyway with a two-thirds majority vote. Question 20 reads: The Constitution created a weaker central government than the Articles of Confederation. Under each question three codes are listed: 1 True, 2 False, 9 No answer or both checked. Red outlines surround the lines 1 True and 2 False under each of the two questions.", "caption": "Coding document of the pretreatment survey in the replication files, p. 4, questions 19 and 20. The outlines mark the response codes, 1 for True and 2 for False, under both questions. The statement of question 19 is true and that of question 20 is false, so the code of the correct answer is 1 for the first and 2 for the second.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E011-coding.png"}], "formal_release": null, "formal_statements": [], "headline": "The pretest knowledge score that the replication files use as a moderator adds the raw True/False response codes of questions 18 to 24 and is not a count of correct answers; scored by correctness, the six interactions remain non-significant in a recalculation on the public file.", "id": "P043-E011", "location": "Replication files, Green_et_al_JP_2011_Public_Use_EDITED.do, lines 31–37 and 358–366, with the same construction in Green_et_al_JP_2011_Public_Use_EDITED.R; the paper’s counterparts are the sentence on heterogeneous treatment effects, p. 475 (PDF p. 13), left column, and footnote 14", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Stata script, lines 31--37 and 358.}\n\\texttt{forvalues i = 18/24 \\{}\n\n\\texttt{gen Aq`i'}\\texttt{p = }\\hl{\\texttt{Aq`i'}}\n\n\\texttt{replace Aq`i'}\\texttt{p = }\\hl{\\texttt{1}}\\texttt{ if Aq`i' == 9}\n\n\\texttt{replace Aq`i'}\\texttt{p = . if Aq`i' == 10}\n\n\\texttt{\\}}\n\n\\texttt{gen ATOTknow = Aq18p + Aq19p + Aq20p + Aq21p + Aq22p + Aq23p + Aq24p}\n\n\\texttt{gen trpreknow = treatment*ATOTknow}\n\n\\emph{p.~475.} Our search for heterogeneous treatment effects came up with no convincing evidence of subgroup differences; partitioning the sample by political interest, frequency of political discussion at home, party ID, age, mother's level of education, employment status and seven factual questions yielded no significant interactions in models predicting knowledge or support for civil liberties.", "corrected": "Score questions 18 to 24 by correctness, and do not add the raw True/False codes. With the key $(k_{18},\\dots,k_{24}) = (2,1,2,2,2,1,2)$ and the raw code $r_i$ of question $i$, set\n\\[\nK = \\sum_{i=18}^{24} \\hl{\\mathbf{1}\\{r_i = k_i\\}} .\n\\]\nA response with the code 9 (no answer or both boxes checked) receives zero credit. For a student with no pretreatment survey, which the script codes as 10, $K$ is missing. Replace \\texttt{ATOTknow} by $K$ in the interaction with the treatment, fit the six regressions again, and apply the Holm--Bonferroni correction of footnote~14. In the recalculation on the public file, the six $p$-values with the corrected score exceed 0.05. The adjusted models of Table~4 use the questions as separate categorical controls and do not change.", "evidence": "The evidence is a derivation, an exact counterexample and a recalculation on the public file. Let $r_i \\in \\{1,2\\}$ be the response code of question $i$, let $C_F$ be the number of correct answers among the five questions whose answer is False, and let $C_T$ be the number among the two whose answer is True. The script's sum is\n\\[\nS = \\sum_{i=18}^{24} r_i = 7 + \\#\\{i : r_i = 2\\} = 9 + C_F - C_T ,\n\\]\nand the number of correct answers is $K = C_F + C_T$. The response patterns $(2,1,2,2,2,1,2)$ and $(2,2,1,2,2,1,2)$ both have $S = 12$; the first has $K = 7$ and the second $K = 5$. These two patterns imply that $K$ is not a function of $S$, and no rescaling of $S$ recovers it. The pattern $(2,2,2,2,2,2,2)$ has $S = 14$ and $K = 5$. Among the 1095 students of the public file with a pretest, the correlation of $S$ and $K$ is 0.669.\n\nFor the recalculation, I fit the interaction models of the R script with each score, and compute Wald $p$-values for the interaction.\n\\begin{audittable}{lrrrr}{Interaction of treatment with the pretest score: $p$-values with the script's sum $S$ and with the number correct $K$, and Holm-adjusted $p$-values for $K$ within the six tests.}\nOutcome & $n$ & $p$ with $S$ & $p$ with $K$ & Holm $p$ with $K$ \\\\\nSupport for civil liberties, 2007 & 974 & 0.242 & 0.074 & 0.443 \\\\\nSupport for civil liberties, 2008 & 553 & 0.767 & 0.126 & 0.505 \\\\\nSupport for civil liberties, 2009 & 478 & 0.194 & 0.761 & 1.000 \\\\\nCivil liberties knowledge, 2007 & 974 & 0.026 & 0.095 & 0.475 \\\\\nCivil liberties knowledge, 2008 & 553 & 0.382 & 0.977 & 1.000 \\\\\nCivil liberties knowledge, 2009 & 478 & 0.244 & 0.569 & 1.000 \\\\\n\\end{audittable}\nWith $K$, the six $p$-values exceed 0.05, and none is significant after the Holm--Bonferroni correction in any family that contains them. The models follow the R specification and not the Stata estimator. The original family of comparisons, which included the other moderators, was not rebuilt.", "problem": "In the coding of the pretreatment survey, each of the questions 18 to 24 has the code 1 for True, 2 for False and 9 for no answer or both boxes checked. These are response labels and not marks of correctness. The correct answer is False for questions 18, 20, 21, 22 and 24 and True for questions 19 and 23. The script copies the raw code, sets the code 9 to 1, which is the code for True, and adds the seven values. For a fully answered survey the sum is 7 plus the number of False responses. A correct answer raises the sum on five questions and lowers it on the other two. The variable \\texttt{ATOTknow} is therefore not a knowledge score, and the six regressions that interact it with the treatment do not test the moderator that the text names.", "what_survives": "The adjusted models of Table~4 enter the seven questions as separate categorical controls and never use the sum, and the unadjusted models omit the questions. No entry of Table~4 changes. The reported absence of significant interactions holds for pretest knowledge under the corrected score in the recalculation. The moderator and the six auxiliary regressions change (lines 360--362 and 364--366 of the Stata script, and the matching loop of the R script), and their coefficients and $p$-values must be recomputed."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E011", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 140 to 146 in monospaced type with line numbers. Lines 140 to 142 end the loop over the 2008 support items: a command that assigns the item mean to responses coded 9, then eval(parse(text=command)) and a closing brace. A red outline surrounds line 144: data is assigned subset(data, select= c(-Cq25no9, -Cq25avg, -Cq27no9, -Cq27avg, -Cq28no9, -Cq28avg, -Cq29no9, -Cq29avg, -Cq31no9, -Cq31avg, -Cq32no9, -Cq32avg, -Cq33no9, -Cq33avg)). Line 146 assigns Cq28pW as 3 minus Cq28p.", "caption": "Replication files, R script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 140–146, typeset from the file with line numbers added and long lines wrapped. The outline marks line 144, which drops fourteen columns that the loop above does not create.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E012.png"}, {"alt": "Listing of lines 165 to 171 in monospaced type with line numbers. Lines 165 to 167 end the loop over the 2009 support items. A red outline surrounds line 169: data is assigned subset(data, select= c(-Dq11no9, -Dq11avg, -Dq13no9, -Dq13avg, -Dq15no9, -Dq15avg, -Dq17no9, -Dq17avg, -Dq18no9, -Dq18avg, -Dq39bno9, -Dq39bavg, -Dq44no9, -Dq44avg)). Line 171 assigns Dq18pW as Dq18p minus 1, times one half, plus 1.", "caption": "The same R script, lines 165–171, typeset in the same way. The outline marks line 169, the corresponding statement for the 2009 support items.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E012-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Two cleanup statements in the R script name columns that do not exist, and executing those statements stops with an error before the 2008 and 2009 support indices are built; deleting the two statements removes the error.", "id": "P043-E012", "location": "Replication files, R script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 144 and 169", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Line 144.} \\texttt{data <- subset(data, select= c(-\\hl{Cq25no9},-\\hl{Cq25avg},}\\ldots\\texttt{,-\\hl{Cq33no9},-\\hl{Cq33avg}))}\n\n\\emph{Line 169.} \\texttt{data <- subset(data, select= c(-\\hl{Dq11no9},-\\hl{Dq11avg},}\\ldots\\texttt{,-\\hl{Dq44no9},-\\hl{Dq44avg}))}\n\nEach statement lists fourteen names, abbreviated here: the suffixes \\texttt{no9} and \\texttt{avg} attached to \\texttt{Cq25}, \\texttt{Cq27}, \\texttt{Cq28}, \\texttt{Cq29}, \\texttt{Cq31}, \\texttt{Cq32}, \\texttt{Cq33} in line 144, and to \\texttt{Dq11}, \\texttt{Dq13}, \\texttt{Dq15}, \\texttt{Dq17}, \\texttt{Dq18}, \\texttt{Dq39b}, \\texttt{Dq44} in line 169.", "corrected": "Delete lines 144 and 169 of the R script. Alternatively, replace each by a statement that removes only the columns that exist, \\texttt{data <- data[, !(names(data) \\%in\\% cleanup\\_names), drop=FALSE]}, where \\texttt{cleanup\\_names} is the character vector of the fourteen names of the original statement.", "evidence": "The evidence is an exact counterexample and a check on the replication data. Let the data frame have the seven columns \\texttt{Cq25}, \\ldots, \\texttt{Cq33}, each equal to $(1, 9)$. After lines 135--142 are run unchanged, every recoded column equals $(1, 1)$ and no column with the suffix \\texttt{no9} or \\texttt{avg} exists. Line 144 then raises the error. The same holds for lines 159--167 and line 169 with the seven \\texttt{Dq} columns. I checked the data transformations using the replication data (1215 rows, 128 columns), omitting package loads and reporting commands. All fourteen names of each statement are absent, before and after the preceding transformations, and each statement raises the error. With both statements deleted, those transformations build the three support indices, with 1045, 591 and 508 nonmissing values. Removing only the columns that exist gives an identical data frame.", "problem": "Each statement is meant to drop temporary columns after the loop that recodes the support items. The Stata script creates columns with these names and then drops them. The R loops assign the item means directly and never create them, and the data file does not contain them. In R, \\texttt{subset} does not ignore a name that is absent, and the statement stops with the error ``object 'Cq25no9' not found''. When execution reaches line 144, that statement stops it before the 2008 support index (line 150), the 2009 support index (line 173) and the models of Table 4 (from line 181). With line 144 removed, line 169 fails in the same way.", "what_survives": "No printed number, index definition or conclusion is in question, and the Stata script does not have this defect. The correction removes these two errors only. It does not by itself make the R script reproduce the tables; other differences between the R script and the printed analysis are described in \\fid{P043-E009}, \\fid{P043-N020} and \\fid{P043-N021}."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E012", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Listing of lines 1 to 21 of the README in monospaced type with line numbers. Lines 1 to 3 say that the data files were revised on December 3, 2019 to minimize disclosure risk and refer to the program file Green\\_et\\_al\\_JP\\_2011\\_piiremoval.do for the procedure. Line 5 reads: The pii variables of interest are Aq33year and seg3, which are found in the following do file, R file and datasets. A red outline surrounds seg3. Lines 7 to 15 list the do file, the two dta files and the R file. A second red outline surrounds line 18: The dta files have been edited to remove the observations where the person's age is less than 18. Lines 19 to 21 name the two edited files.", "caption": "Replication files, Green_et_al_JP_2011_EDITED_README.txt, lines 1–21, typeset from the file with line numbers added and long lines wrapped. The outlines mark the misnamed variable in line 5 and the sentence in line 18 that says observations were removed.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E013.png"}, {"alt": "Listing of lines 10 to 25 of the script in monospaced type with line numbers. Line 10 loads the main data file. Lines 11 to 16 are a comment: the variable to top code is Aq33year; 2007 is the reference year; the individual's age is to be bottom coded at 18, so the year of birth is top coded at 2007 minus 18, which is 1989; the 44 observations with a birth year of 9999 will not be top-coded. A red outline surrounds line 17: replace Aq33year = 1989 if Aq33year is greater than 1989 and Aq33year is not equal to 9999. Lines 18 and 19 save and export the file. Line 22 loads the Survey E file, and the comment in lines 23 and 24 says that the variable to drop is seq3. A second red outline surrounds line 25: drop seq3.", "caption": "Replication files, privacy script Green_et_al_JP_2011_piiremoval.do, lines 10–25, typeset in the same way. The outlines mark the two commands that change the data: line 17 replaces years of birth, and line 25 drops a variable. Neither removes an observation.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-E013-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The README says that respondents under 18 were removed from the public data, but the privacy script removes no respondent: it sets every year of birth above 1989 other than the code 9999, and every missing year of birth, to 1989, and it deletes one variable from the 2010 survey file.", "id": "P043-E013", "location": "Replication files (public deposit of 2019): Green_et_al_JP_2011_EDITED_README.txt, lines 5 and 18; privacy script Green_et_al_JP_2011_piiremoval.do, lines 17 and 25", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{README, line 5.} The pii variables of interest are Aq33year and \\hl{seg3}, which are found in the following do file, R file and datasets:\n\n\\emph{README, line 18.} The dta files have been edited to \\hl{remove the observations} where the person's age $<$ 18.\n\n\\emph{Privacy script, line 17.} \\texttt{replace Aq33year = 1989 if Aq33year > 1989 \\& Aq33year != 9999}\n\n\\emph{Privacy script, line 25.} \\texttt{drop seq3}", "corrected": "\\emph{README, in place of line 18, with} seq3 \\emph{for} seg3 \\emph{in line 5.} \\hl{The privacy edit retains all observations in both files. In the main file, the script replaces Aq33year with 1989 whenever Aq33year is greater than 1989 and is not 9999; in Stata this condition includes numeric missing values. The code 9999 is retained. In the Survey E file, the variable seq3 is removed. The edited files contain 1215 and 254 observations, respectively.}\n\n\\emph{Privacy script, line 17,} if missing years of birth are to stay missing: \\texttt{replace Aq33year = 1989 if Aq33year > 1989 \\& Aq33year != 9999 \\hl{\\& Aq33year < .}} The public files would then have to be regenerated from the original data, and the README would describe the corrected operation.", "evidence": "The evidence is an exact counterexample and a check of the replication files. The privacy script has eight commands, none of which deletes observations. When line 17 is applied to two students born in 1988 and 1991, aged 19 and 16 in the reference year 2007, the output has two rows, with years 1988 and 1989; removing respondents under 18 would leave one row. For a missing year, both comparisons hold, and the year becomes 1989. In the public main file the year of birth takes the values 1986 (1 student), 1988 (21), 1989 (1149) and 9999 (44). Since the original files are not available, I could not observe the original values for the 120 students who missed the pretreatment survey.", "problem": "The README misdescribes the edit in two ways. It says that observations were removed, and no command of the privacy script removes a row. The script replaces values of the year of birth, \\texttt{Aq33year}, in the main file, and drops the variable \\texttt{seq3} from the file of the 2010 survey. The main file keeps all 1215 students and the 2010 file all 254 respondents. The README also names the second variable \\texttt{seg3}; its name is \\texttt{seq3}.\n\nThe script itself does more than its comment describes. The comment says that the year of birth is to be top-coded at 1989, so that no age below 18 in 2007 can be inferred, and that the code 9999 is left alone. In Stata a missing numeric value is larger than every number. This implies that the condition \\texttt{Aq33year > 1989 \\& Aq33year != 9999} also holds for a missing year of birth, and the script sets such a year to 1989. In the public file all 120 students who did not take the pretreatment survey have year of birth 1989, and every other pretreatment variable is missing for them. The analysis script later recodes a missing pretreatment answer to its own category; a year already set to 1989 is not recoded, and it enters the age interactions as an observed year.", "what_survives": "This finding concerns the deposit of 2019, and it shows no error in the article. The public file keeps the 627 treatment and 588 control students and the response counts 1045, 591 and 508 of the article. The privacy edit leaves the assignment, the outcomes and every analysis without year of birth unchanged, and the analyses of the 2010 survey do not use \\texttt{seq3}. For the models of Table 4 with covariates, the space spanned by the covariates is unchanged when only the 120 missing years are recoded, because the same 120 students are already identified by the missing-answer category of another pretreatment covariate. The public file does not preserve the original year-of-birth categories needed to reconstruct the original covariate-adjusted models, randomization check and age interactions."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-E013", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of the right column. The text reads: who go on to college ranges widely, from 28\\% to 88\\%. All 10 schools participated in the pretreatment survey and the 2007 posttreatment survey. Seven of the ten original schools participated in all four waves of surveys. Attrition occurred in schools IV and VI. A red outline surrounds the word Seven.", "caption": "Research Sites, last paragraph, p. 466 (PDF p. 4), right column. The outline marks the number of schools said to have participated in all four waves of surveys.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N001.png"}, {"alt": "Crop of the first three lines of the right column. The text reads: completed by students in seven schools, taught by 19 of the original 25 teachers. Teachers whose students participated in the follow-up study had an average. A red outline surrounds the words seven schools.", "caption": "Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), top of the right column. The sentence begins at the foot of the left column with “The 2008 and 2009 surveys were thus”. The outline marks the number of schools.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Six schools, not seven, supplied completed surveys in 2008 and in all four waves; seven is the number of schools that remained in the study after 2007.", "id": "P043-N001", "location": "Research Sites, last paragraph, p. 466 (PDF p. 4), right column; Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), sentence that runs from the foot of the left column to the top of the right column", "pdf_page": 4, "printed_page": "466", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~466.} All 10 schools participated in the pretreatment survey and the 2007 posttreatment survey. \\hl{Seven} of the ten original schools participated in all four waves of surveys.\n\n\\emph{p.~469.} The 2008 and 2009 surveys were thus completed by students in \\hl{seven schools}, taught by 19 of the original 25 teachers.", "corrected": "\\emph{p.~466.} Seven of the ten original schools \\hl{remained in the study after 2007; six supplied completed surveys in all four waves}.\n\n\\emph{p.~469.} The 2008 and 2009 surveys were thus completed by students in \\hl{six and seven schools, respectively}, \\ldots\n\nThe number of teachers in the rest of the second sentence is corrected separately in \\fid{P043-N004}.", "evidence": "The evidence is a recalculation from the replication data. I counted the completed surveys by school and by wave. The same counts by stratum and arm agree with all 240 numeric cells of Table 2, the class counts and the total row included.\n\\begin{audittable}{llrrrrrl}{Completed surveys by school, from the replication data. The school codes are those of the data file.}\nSchool & Strata & Assigned & Pretreatment & 2007 & 2008 & 2009 & Remained after 2007 \\\\\n1 & A, L & 86 & 79 & 72 & 49 & 44 & yes \\\\\n2 & C, D & 86 & 80 & 66 & 0 & 11 & yes \\\\\n3 & E, F & 65 & 63 & 61 & 0 & 0 & no \\\\\n4 & H, N & 87 & 79 & 69 & 0 & 0 & no \\\\\n5 & R, S & 225 & 200 & 211 & 145 & 46 & yes \\\\\n6 & G, P & 100 & 81 & 76 & 0 & 0 & no \\\\\n7 & I, J & 146 & 144 & 129 & 112 & 112 & yes \\\\\n8 & K, M & 121 & 102 & 90 & 63 & 73 & yes \\\\\n9 & B, T & 120 & 102 & 105 & 88 & 89 & yes \\\\\n10 & Q & 179 & 165 & 166 & 134 & 133 & yes \\\\\n\\end{audittable}\nFor school 2, the cells of Table 2 for strata C and D (control and treatment) give $20 + 24 + 27 + 15 = 86$ assigned, $0 + 0 + 0 + 0 = 0$ respondents in 2008 and $0 + 2 + 5 + 4 = 11$ in 2009.", "problem": "Three schools left the study after 2007, and seven remained. One of the seven has no completed survey in 2008. Strata C and D are the whole of this school, which has 86 assigned students. Table 2 shows that neither arm of either stratum has a 2008 respondent, and that the school has 11 respondents in 2009. Completed surveys come from six schools in 2008, from seven in 2009, and from six in every wave.\n\nOn p.~466, ``participated'' can be read loosely as ``remained in the study'', and with that meaning seven is correct. The sentence on p.~469 says that the 2008 and 2009 surveys were completed by students in seven schools, and for 2008 this is false. The data show no completed 2008 survey for this school. They do not show whether the survey was attempted there, or why none was returned.", "what_survives": "The wording correction leaves Table 2, the sample sizes 1045, 591 and 508, and the reported estimates unchanged; it changes no survey responses or model specification. The 2009 response rates among the seven schools that remained also stand: $253/477 = 53.0\\%$ and $255/486 = 52.5\\%$. Only the two sentences change."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N001", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the page number 469 and the first five lines of the right column. A red box marks the number 19 at the end of the first line, 'completed by students in seven schools, taught by 19', which continues 'of the original 25 teachers'.", "caption": "Top of the right column, p. 469 (PDF p. 7). The sentence begins at the foot of the left column with “The 2008 and 2009 surveys were thus”. The marked number is the count of teachers in the continuing schools.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N004.png"}], "formal_release": null, "formal_statements": [], "headline": "The seven continuing schools had 18 of the original 25 teachers, not 19.", "id": "P043-N004", "location": "Characteristics of the Courses and Teachers Studied, p. 469 (PDF p. 7), sentence running from the foot of the left column to the top of the right column", "pdf_page": 7, "printed_page": "469", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``The 2008 and 2009 surveys were thus completed by students in seven schools, taught by \\hl{19} of the original 25 teachers.''", "corrected": "``The 2008 and 2009 surveys were thus completed by students in seven schools, taught by \\hl{18} of the original 25 teachers.''", "evidence": "This is a recalculation from the replication data. Each teacher identifier belongs to one school, and the seven continuing schools have $1+2+6+2+1+3+3=18$ teachers. The students who completed the 2008 survey had 16 of these teachers, those who completed the 2009 survey had all 18, and the two groups together had 18. The same data reproduce every entry of Table 2, including its totals.", "problem": "The replication data identify 25 teachers in 10 schools. The three schools that left the study after the 2007 survey had 3, 2 and 2 teachers, which leaves $25-7=18$ teachers in the seven continuing schools. Counting the teachers of the students who answered a follow-up survey does not give 19 either.", "what_survives": "The 25 original teachers, the 10 schools, the seven continuing schools, the 19 strata and the counts of Table 2 are unchanged. No model in the replication code uses the number of teachers, and no estimate or standard error changes. The ages and other attributes of the teachers are not in the replication data, and the description of the follow-up teachers in the next sentences of the paper was not checked."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N004", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the last two lines of the left column and footnote 8 below them. A red box marks the words 'Stratum H' in the first line of the footnote, 'Consider, for example, Stratum H, consisting of three courses taught by an instructor'.", "caption": "Footnote 8, p. 469 (PDF p. 7). The marked stratum is named as the example of one instructor with three courses.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "The example of footnote 8 describes Stratum I (and equally J or K), not Stratum H.", "id": "P043-N006", "location": "Footnote 8, p. 469 (PDF p. 7)", "pdf_page": 7, "printed_page": "469", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Consider, for example, \\hl{Stratum H}, consisting of three courses taught by an instructor. Suppose this instructor indicates that he/she is willing to teach two BRRL courses. We randomly assign two courses of the three to the treatment group and the remaining course to the control group.''", "corrected": "``Consider, for example, \\hl{Stratum I}, consisting of three courses taught by an instructor. Suppose this instructor indicates that he/she is willing to teach two BRRL courses. We randomly assign two courses of the three to the treatment group and the remaining course to the control group.''", "evidence": "This is a recalculation from the replication data, tabulated by stratum, teacher and assignment. The numbers of courses and students that it gives for strata H, I, J and K agree with Table 2 (p.~470).", "problem": "In the replication data, Stratum H has two teachers. One taught its single control course (10 students) and the other taught both of its treatment courses (31 students). The footnote describes one instructor with three courses, two of them treated. Strata I, J and K have this composition.", "what_survives": "The footnote's account of assignment within a stratum is unchanged, and the definition in the text allows a stratum taught by a cluster of teachers. Nothing else depends on the label. The check does not establish how the courses of Stratum H were assigned."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N006", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the end of footnote 13. The text reads: of relevant constitutional principles identified. In the 2007 posttreatment survey, students in the treatment group performed significantly better than students in the control group, indicating that students were able to apply the constitutional knowledge they had gained. On average, treated students identified .182 (SE=0.068) additional rights out of a possible 13. However, there was no significant difference between treatment and control in the 2008 survey (-.007 rights, SE = 0.128). This finding comports with the general trend of declining differences between treatment and control groups over time. Red outlines surround .182, (SE=0.068) and (-.007 rights, SE = 0.128).", "caption": "Footnote 13, p. 473 (PDF p. 11), last ten lines. The outlines mark the two essay estimates with their standard errors, which the essay commands of the replication files do not return.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N010.png"}], "formal_release": null, "formal_statements": [], "headline": "The essay commands in the replication files are not restricted to the students who took each survey, and they do not return the estimates of footnote 13; the printed estimates are not shown to be wrong.", "id": "P043-N010", "location": "Footnote 13, p. 473 (PDF p. 11); the essay commands of the replication files: lines 483 and 485 of the Stata script Green_et_al_JP_2011_Public_Use_EDITED.do, and lines 613 and 615 of the R script Green_et_al_JP_2011_Public_Use_EDITED.R", "pdf_page": 11, "printed_page": "473", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Footnote~13, p.~473.} On average, treated students identified .182 (SE=0.068) additional rights out of a possible 13. However, there was no significant difference between treatment and control in the 2008 survey (-.007 rights, SE = 0.128).\n\n\\emph{Stata script, line 483 (broken here after \\texttt{i.Aq21}); line 485 is the same command for \\texttt{Copenended}.}\n\\[\n\\begin{aligned}\n&\\texttt{xi: xtreg Bopenended treat i.stratum i.Aq1 i.Aq3 i.Aq4  i.Aq18 i.Aq19 i.Aq20 i.Aq21}\\\\\n&\\qquad\\texttt{i.Aq22 i.Aq23 i.Aq24  i.Aq28 i.Aq32 i.Aq33year, re i(classcode)}\n\\end{aligned}\n\\]\n\n\\emph{R script, lines 613 and 615.}\n\\[\n\\begin{aligned}\n&\\texttt{reg.B.open <- lmer(Bopenended \\textasciitilde{} treatment + stratum + (1|classcode), }\\hl{\\texttt{data=data}}\\texttt{)}\\\\\n&\\texttt{reg.C.open <- lmer(Copenended \\textasciitilde{} treatment + stratum + (1|classcode), }\\hl{\\texttt{data=data}}\\texttt{)}\n\\end{aligned}\n\\]", "corrected": "In the Stata script, each essay command takes the students who took that survey:\n\\[\n\\begin{aligned}\n&\\texttt{xi: xtreg Bopenended treat} \\;\\ldots\\; \\texttt{i.Aq33year }\\hl{\\texttt{if Bcompleted == 1}}\\texttt{, re i(classcode)}\\\\\n&\\texttt{xi: xtreg Copenended treat} \\;\\ldots\\; \\texttt{i.Aq33year }\\hl{\\texttt{if Ccompleted == 1}}\\texttt{, re i(classcode)}\n\\end{aligned}\n\\]\nThe R calls take the same two subsets. The files should also document how the essay score was built, how a missing essay is scored, the fixed covariates and the variance estimator. These filters alone do not reproduce both printed pairs, and the numbers of footnote~13 stay as printed.", "evidence": "The evidence is a recalculation on the public file. The R formulas were run as written. Stata was not available, and the lines marked OLS are ordinary least squares with the fixed regressors of the Stata commands and conventional standard errors.\n\\begin{audittable}{llrrr}{Effect of treatment on the essay score: footnote~13 and recalculations on the public file.}\nWave & Calculation & $N$ & Estimate & SE \\\\\n2007 & Footnote~13 & & .182 & 0.068 \\\\\n2007 & R formula as written & 1014 & 0.139609 & 0.076139 \\\\\n2007 & OLS, sample of the Stata command & 1014 & 0.123438 & 0.068956 \\\\\n2007 & OLS, students who took the survey, missing scores set to zero & 1045 & 0.181744 & 0.067518 \\\\\n2008 & Footnote~13 & & $-.007$ & 0.128 \\\\\n2008 & R formula as written & 1215 & $-0.002728$ & 0.100683 \\\\\n2008 & OLS, sample of the Stata command & 1215 & $-0.007565$ & 0.070603 \\\\\n2008 & OLS, students who took the survey & 591 & $-0.007703$ & 0.128322 \\\\\n\\end{audittable}\nThe 2007 pair is recovered at printed precision on the 1045 students who took the survey, with their 201 missing scores set to zero. The 2008 standard error is recovered on the 591 students who took the survey; the estimate there rounds to $-.008$ and not to $-.007$. Note that setting missing scores to zero is a diagnostic coding and not an established rule for the original analysis. The birth-year covariate of the public file was top-coded in 2019 (see \\fid{P043-E013}), which changes one set of regressors.", "problem": "Footnote~12 says that the analyses include all students who took each survey, and Table~3 reports 1045 such students in 2007 and 591 in 2008. Neither script restricts the essay regressions to them. On the public file, the 2007 command selects 1014 students. It includes the 170 students flagged as not having taken the 2007 survey, whose essay score is zero, and it excludes 201 students who took the survey and whose essay score is missing. The 2008 command selects all 1215 students, of whom 624 are flagged as not having taken the 2008 survey. The R formulas also omit the pretest covariates of the Stata commands (see \\fid{P043-N021}). As written, the commands do not return the two pairs printed in footnote~13.", "what_survives": "The estimates of footnote~13 are not shown to be wrong, and the 2007 pair is recovered at printed precision under one coding of the students who took the survey. The models of Table~4 use other outcome variables and other commands, and they do not depend on the essay scores. The essay commands of the replication files change."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N010", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": {"class": "error", "severity": "Moderate"}, "class": "not_confirmed", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of eight lines of text. The passage reads: dissent, or due process. This pattern of results is striking both because this experiment has ample power to detect even small effects and because, as noted above, the statistical test is biased in favor of finding a mediating relationship. The theoretical implications of this finding are profound. Evidently, it is possible to increase awareness and understanding of civil liberties without producing an increase in; the crop ends in the middle of this sentence. Red outlines surround the words the statistical test is biased in favor of finding a mediating relationship, which run over two lines.", "caption": "p. 474 (PDF p. 12), left column below Table 4. The outlines mark the clause that refers back to the preceding positive-other-channel explanation.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N014.png"}, {"alt": "Line chart. The horizontal axis is the net effect of the curriculum on support through channels other than knowledge, D, from -3/25 to 3/25 points of the support index. The vertical axis is the reduced-form effect on support, M + D, from -3/50 to 9/50. A straight line of slope one passes through the point D = 0, reduced form 3/50. Its part with D at least 0 is a solid blue line labelled reduced form at least M, the case of p. 473. Its part with D below 0 is a dashed orange line labelled reduced form below M. A dotted horizontal line marks the effect through knowledge, M = 3/50. A dot on the dashed part at D = -3/50 and reduced form 0 is labelled exact example: D = -3/50, reduced form 0 although M = 3/50.", "caption": "The identity $\\tau = M + D$ with the effect through knowledge fixed at $M = 3/50$, the value in the two examples. The reduced form is at least $M$ when $D \\ge 0$, the case described on p. 473, and smaller than $M$ when $D < 0$. The dot marks the examples outside that sign premise, in which $D = -3/50$ and the reduced form is zero.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N014-sign-condition.png"}], "formal_release": null, "formal_statements": [], "headline": "Whether the favorable-mediation claim omits the sign premise needed for its comparison.", "id": "P043-N014", "location": "p. 474 (PDF p. 12), left column below Table 4, sentence beginning “This pattern of results is striking”; the argument it refers to is on p. 473 (PDF p. 11), left column", "pdf_page": 12, "printed_page": "474", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"reason": "The allegation was that p.~474 drops a necessary sign condition and claims that the reduced-form test always favors finding mediation. The relevant passages are:\n\n\\emph{p.~474.} This pattern of results is striking both because this experiment has ample power to detect even small effects and because, as noted above, the statistical test is biased in favor of finding a mediating relationship.\n\n\\emph{p.~473.} As Figure 1 suggests, estimating the reduced form provides a \\emph{conservative} test, because if the treatment leads to a direct increase in support for civil liberties (i.e., course content encourages support for norms of free expression and association) or indirectly through channels other than knowledge (e.g., course discussion promotes appreciation for dissenting viewpoints), we would attribute these effects to the mediating role of knowledge and understanding.\n\nThe phrase ``as noted above'' refers back to p.~473's positive direct or other-channel effects. Read in context, the claim carries that hypothesized sign premise forward: the curriculum's net effect through channels other than knowledge is nonnegative. On that premise, attributing the reduced form to knowledge weakly favors finding mediation. The published conditional is mathematically valid, so I did not establish a false statement under this natural reading.\n\nThe design does not verify the premise. Random assignment makes the treatment independent of the disturbances of Figure~1 but imposes no sign on the other-channel contribution. The null result on general political knowledge addresses one possible offset, and does not identify the net sign of every other channel. Its application to mediation consequently remains conditional; no actual negative channel or positive mediated effect in the experiment is established here.\n\nThe derivation and two exact examples explain the limitation of an unconditional reading. In a model without feedback from support to knowledge and with well-defined interventions on knowledge, let $K(t)$ be a student's knowledge score under curriculum $t$, with $t = 1$ for the experimental curriculum and $t = 0$ for the usual one, and let $S(t,k)$ be the support score under curriculum $t$ when knowledge is set to $k$. Define\n\\[\n\\begin{aligned}\n\\tau &= \\mathbb{E}\\bigl[S(1,K(1)) - S(0,K(0))\\bigr],\\\\\nM &= \\mathbb{E}\\bigl[S(0,K(1)) - S(0,K(0))\\bigr],\\\\\nD &= \\mathbb{E}\\bigl[S(1,K(1)) - S(0,K(1))\\bigr].\n\\end{aligned}\n\\]\nHere $\\tau$ is the reduced form, $M$ is the effect through knowledge and $D$ is the net effect through the other channels. Adding and subtracting $S(0,K(1))$ gives $\\tau = M + D$. Random assignment identifies $\\tau$ and puts no restriction on the sign of $D$. The identity implies that $\\tau \\ge M$ if and only if $D \\ge 0$.\n\nIn the first example, $T$ takes the values $0$ and $1$, $u_1$ the values $\\pm 1/5$ and $u_2$ the values $\\pm 1/10$, each with probability $1/2$, and $T$, $u_1$, $u_2$ are independent. Set $K(t) = 8 + (6/5)t + u_1$ and $S(t,k) = 23/10 + (k-8)/20 - (3/50)t + u_2$. A direct computation gives $M = 3/50$, $D = -3/50$ and $\\tau = 0$. The observed support score is $23/10 + u_1/20 + u_2$ in both arms.\n\nThe second example has integer scores inside the 2007 ranges of Table~3. Let $W$ be uniform on $\\{1,\\dots,5\\}$ and $V$ uniform on $\\{1,\\dots,100\\}$, with $T$, $W$, $V$ independent. Set $K(t) = 8 + t\\,(1 + \\mathbf{1}\\{W = 5\\})$ and $S(t,k) = 2 + \\mathbf{1}\\{V \\le 36 + 5(k-8) - 6t\\}$. These definitions give $\\mathbb{E}[S(0,K(0))] = 59/25$, $\\mathbb{E}[S(0,K(1))] = 121/50$ and $\\mathbb{E}[S(1,K(1))] = 59/25$.\n\\begin{audittable}{lrr}{The two examples violate the contextual nonnegative-other-channel premise: positive mediation is canceled by a negative effect through other channels.}\nQuantity & First example & Second example \\\\\nMean knowledge change, $\\mathbb{E}[K(1)-K(0)]$ & $6/5$ & $6/5$ \\\\\nExpected support gain, $8\\to9$ or $9\\to10$ & $1/20$ & $1/20$ \\\\\nEffect through knowledge, $M$ & $3/50$ & $3/50$ \\\\\nNet effect through the other channels, $D$ & $-3/50$ & $-3/50$ \\\\\nReduced form, $\\tau = M + D$ & $0$ & $0$ \\\\\n\\end{audittable}\nIn both examples the treatment is independent of every disturbance, as Figure~1 assumes. In the second, a general-knowledge score can be held equal in the two arms, so the null result on general political knowledge does not exclude the example. Note that the examples are not fitted to the data.\n\nBoth examples have $D=-3/50<0$. They satisfy random assignment and the unsigned diagram, but violate the hypothesized sign premise carried forward from p.~473. They therefore refute only an unconditional version of the favorable-mediation comparison, not the paper's contextual conditional. Setting the second example's threshold term $-6t$ to $0t$ or $+6t$ gives, respectively, $(M,D,\\tau)=(3/50,0,3/50)$ and $(3/50,3/50,3/25)$, which satisfy the weak comparison exactly.\n\nTable~4 and the reported total-effect estimates are unchanged by this mediation argument. I separately recalculated the unadjusted estimates (\\fid{P043-N020}); exact reproduction of the adjusted estimates remains blocked (\\fid{P043-E013}). The reported gain in knowledge without a statistically significant increase in support remains a statement about curriculum total effects. The p.~472 conditional exclusion-restriction discussion, the p.~473 positive-channel argument and the Conclusion's cautious multiple interpretations remain. I did not check the ample-power claim.\n\nAn optional clarification would repeat the premise: ``under the positive other-channel effects described above, the reduced-form contrast weakly favors attributing an effect to knowledge.'' More explicitly, $\\tau\\ge M$ if and only if $D\\ge0$, with strict inequality only for $D>0$; the condition concerns the net of the other channels, not each path separately. A negative net contribution can conceal positive mediation, as the examples show. This makes the existing contextual condition explicit without changing an estimator or numerical estimate."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N014", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of footnote 14, four lines. The footnote reads: We tested for heterogeneous treatment effects by interacting the treatment term with each of the individual-level covariates. None of these effects were significant after taking into account multiple comparisons using the Holm-Bonferroni method. Red outlines surround the words each of the individual-level covariates and the words Holm-Bonferroni method.", "caption": "Footnote 14, p. 475 (PDF p. 13). The outlines mark the two parts of the description that the replication scripts do not implement: interactions with each individual-level covariate, and the Holm–Bonferroni adjustment.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N015.png"}], "formal_release": null, "formal_statements": [], "headline": "The replication scripts interact the treatment with one summed pretest score where the text describes each of seven factual questions, and they contain no Holm–Bonferroni adjustment, no balance simulation and no code for Table A6; the scripts cannot reproduce these analyses, and no published result is shown to be wrong.", "id": "P043-N015", "location": "Footnote 14 and the sentence on heterogeneous treatment effects, p. 475 (PDF p. 13), left column; also the balance check on p. 468 (PDF p. 6), footnote 12 on p. 473 (PDF p. 11) and Table A6, p. 10 of the online appendix; read against the Stata and R scripts of the replication files", "pdf_page": 13, "printed_page": "475", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~475.} Our search for heterogeneous treatment effects came up with no convincing evidence of subgroup differences; partitioning the sample by political interest, frequency of political discussion at home, party ID, age, mother's level of education, employment status and \\hl{seven factual questions} yielded no significant interactions in models predicting knowledge or support for civil liberties.\n\n\\emph{Footnote~14, p.~475.} We tested for heterogeneous treatment effects by interacting the treatment term with \\hl{each of the individual-level covariates}. None of these effects were significant after taking into account multiple comparisons using the \\hl{Holm-Bonferroni method}.", "corrected": "The replication files should \\hl{supply the code and output} for the interactions of the treatment with each of the seven factual questions, state the family of comparisons to which the Holm--Bonferroni adjustment was applied and the adjustment itself, and supply the code for the balance simulation and for Table~A6. The description in the paper changes only if the original analysis is found to differ from it.", "evidence": "The evidence is a check of the pages and of the two scripts. I read the heterogeneity blocks (lines 313--408 of the Stata script and lines 283--440 of the R script). I also searched both scripts for a multiplicity adjustment, for simulated assignments and for a restriction to the three-wave sample, and found none. The check does not show that the analyses were not performed outside these scripts. The top-coded year of birth in the public file also prevents an exact reconstruction of the original interactions with age.", "problem": "Each script fits one interaction model for each of seven moderators and six outcomes, 42 models in all. The seven factual questions enter through a single moderator, their sum \\texttt{ATOTknow}, and not through seven separate interactions; the scoring of this sum is the subject of \\fid{P043-E011}. Neither script computes a Holm--Bonferroni adjustment. The scripts also contain no code for the balance check with 20,000 simulated assignments (p.~468), and none for the analysis of Table~A6, which is restricted to the 387 students present in all three posttreatment waves. These analyses cannot be reproduced from the scripts as the paper describes them.", "what_survives": "No published result is refuted, and the code for the main treatment effects is separate from the missing blocks. The replication files change: the family of heterogeneity tests as described, its Holm--Bonferroni adjustment, the balance check and Table~A6 are at present not reproducible from them."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N015", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of seven lines of the reference list. The last line of the preceding entry ends with Political Psychology 26 (3): 333-64. The next entry reads: Galston, William A. 2001. Political Knowledge, Political Engagement, and Civic Education. Annual Review of Political Science 4 (1): 217-41. It is followed by the entry Galston, William A. 2007. Civic Knowledge, Civic Education, and Civic Engagement: A Summary of Recent Research. International Journal of Public Administration 30: 623-42. A red outline surrounds the page range 217-41.", "caption": "References, p. 475 (PDF p. 13), right column, the entries Galston 2001 and Galston 2007. The outline marks the page range “217–41”, which should read “217–34”.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N016.png"}], "formal_release": null, "formal_statements": [], "headline": "The page range of Galston (2001) is 217–34, not 217–41.", "id": "P043-N016", "location": "References, entry Galston 2001, p. 475 (PDF p. 13), right column", "pdf_page": 13, "printed_page": "475", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Galston, William A. 2001. ``Political Knowledge, Political Engagement, and Civic Education.'' \\emph{Annual Review of Political Science} 4 (1): \\hl{217--41}.", "corrected": "Galston, William A. 2001. ``Political Knowledge, Political Engagement, and Civic Education.'' \\emph{Annual Review of Political Science} 4 (1): \\hl{217--34}.", "evidence": "The evidence is a check of the page and of the publisher's page for the article (Annual Reviews, DOI 10.1146/annurev.polisci.4.1.217), which reads ``Vol.~4:217-234''.", "problem": "The cited article ends on page 234 and not on page 241.", "what_survives": "The author, title, year and volume identify the work, and nothing in the paper depends on the page range."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N016", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index, and Support for Civil Liberties Index. The rows are Number of Questions (13, 5, 3), Mean (2010 Survey) (7.59, 2.50, 2.35), Standard Deviation (2010 Survey) (2.21, 1.33, 0.81), Cronbach's Alpha (2010 Survey) (0.55, 0.54, 0.37), Mean (2007 Survey) (7.41, 1.93, 2.36), Standard Deviation (2007 Survey) (2.56, 1.31, 0.80) and Cronbach's Alpha (2007 Survey) (0.65, 0.48, 0.37). Red outlines surround 2.56 and 1.31.", "caption": "Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2007 Indices”. The outlines mark the standard deviations 2.56 and 1.31 in the row for the 2007 survey.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N017.png"}, {"alt": "Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index with an asterisk, and Support for Civil Liberties Index. The rows are Number of Questions (17, 8, 7), Mean (2010 Survey) (8.48, 4.67, 3.95), Standard Deviation (2010 Survey) (2.81, 1.81, 1.44), Cronbach's Alpha (2010 Survey) (0.56, 0.57, 0.44), Mean (2008 Survey) (7.87, 3.88, 4.05), Standard Deviation (2008 Survey) (2.79, 1.68, 1.50) and Cronbach's Alpha (2008 Survey) (0.57, 0.49, 0.45). A red outline surrounds 2.81.", "caption": "Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2008 Indices”. The outline marks the standard deviation 2.81 in the row for the 2010 survey.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N017-b.png"}], "formal_release": null, "formal_statements": [], "headline": "Three standard deviations in Table A4 match truncation at the second decimal rather than rounding: 2.56, 1.31 and 2.81 should read 2.57, 1.32 and 2.82.", "id": "P043-N017", "location": "Online Appendix, p. 8, Table A4: Posttreatment Survey 2007 Indices, row “Standard Deviation (2007 Survey)”; Posttreatment Survey 2008 Indices, row “Standard Deviation (2010 Survey)”", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Posttreatment Survey 2007 Indices.} Standard Deviation (2007 Survey): \\hl{2.56} (Civil Liberties Knowledge Index*), \\hl{1.31} (General Political Knowledge Index).\n\n\\emph{Posttreatment Survey 2008 Indices.} Standard Deviation (2010 Survey): \\hl{2.81} (Civil Liberties Knowledge Index*).", "corrected": "\\emph{Posttreatment Survey 2007 Indices.} Standard Deviation (2007 Survey): \\hl{2.57} (Civil Liberties Knowledge Index*), \\hl{1.32} (General Political Knowledge Index).\n\n\\emph{Posttreatment Survey 2008 Indices.} Standard Deviation (2010 Survey): \\hl{2.82} (Civil Liberties Knowledge Index*).", "evidence": "The evidence is a recalculation from the replication data, with the item sets of the replication code and the sample standard deviation (denominator $N - 1$). It gives 2.568148678 with $N = 1045$, 1.318642601 with $N = 1045$, and 2.817537565 with $N = 251$. The recalculation establishes the numerical pattern only; it does not show how the three cells came to be printed this way.", "problem": "The three standard deviations are 2.568, 1.319 and 2.818 to three decimals. Rounded to two decimals they are 2.57, 1.32 and 2.82. The printed values are the same numbers with the third decimal dropped. Table 3 (p.~472) prints 1.32 for the same five-item 2007 General Political Knowledge Index.", "what_survives": "The definitions of the indices and every finding of the paper are unaffected. Each of the three cells changes by 0.01."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N017", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of a table with three columns: Civil Liberties Knowledge Index with an asterisk, General Political Knowledge Index with an asterisk, and Support for Civil Liberties Index. The rows are Number of Questions (17, 8, 7), Mean (2010 Survey) (8.48, 4.67, 3.95), Standard Deviation (2010 Survey) (2.81, 1.81, 1.44), Cronbach's Alpha (2010 Survey) (0.56, 0.57, 0.44), Mean (2008 Survey) (7.87, 3.88, 4.05), Standard Deviation (2008 Survey) (2.79, 1.68, 1.50) and Cronbach's Alpha (2008 Survey) (0.57, 0.49, 0.45). A red outline surrounds the heading General Political Knowledge Index with its asterisk and the entry 8 below it.", "caption": "Online Appendix, p. 8, Table A4, block “Posttreatment Survey 2008 Indices”. The outline marks the heading of the General Political Knowledge Index column, whose asterisk refers to the note on subsets of questions, and its number of questions, 8.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N018.png"}, {"alt": "Crop of four lines of notes. The text reads: asterisk, Index consists of a subset of questions originally included. Notes: All questions rescaled to 0 to 1 for additive index. N is 251 for all indices in 2010 Survey. For the 2007 to 2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey. A red outline in two parts surrounds the last sentence, from For the 2007 to 2009 surveys to the 2010 survey.", "caption": "Online Appendix, p. 8, the notes under Table A4. The outline marks the sentence that describes the 2007–2009 statistics as those of the equivalent indices made of the questions included on the 2010 survey; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N018-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The 2008 and 2010 versions of the General Political Knowledge Index, each with eight questions, have seven questions in common; the eighth asks for the name of the town’s head official in 2008 and for the name of the governor in 2010, and the column does not compare one fixed set of questions.", "id": "P043-N018", "location": "Online Appendix, p. 8, Table A4, Posttreatment Survey 2008 Indices, column “General Political Knowledge Index*”, and the note under the table; the note to Table A5, p. 9, describes the same indices", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Table A4, Posttreatment Survey 2008 Indices, General Political Knowledge Index*.} Number of Questions 8; Mean (2010 Survey) 4.67; Standard Deviation (2010 Survey) 1.81; Cronbach's Alpha (2010 Survey) 0.57; Mean (2008 Survey) 3.88; Standard Deviation (2008 Survey) 1.68; Cronbach's Alpha (2008 Survey) 0.49.\n\n* Index consists of a subset of questions originally included. Notes: All questions rescaled to 0--1 for additive index. N is 251 for all indices in 2010 Survey. \\hl{For the 2007--2009 surveys, the summary statistics are for the equivalent indices consisting of the subset of questions included on the 2010 survey.}", "corrected": "Keep the entries of Tables A4 and A5 and add to both notes: \\hl{The 2008 General Political Knowledge Index versions share seven questions. The eighth question asks for the town or city head official's name in 2008 and for the governor's name in 2010. The eight-question versions therefore contain different questions, and their mean difference does not compare one fixed set of questions.}\n\nAlternatively, compare the seven shared questions. The column of Table A4 then reads: Number of Questions \\hl{7}; Mean (2010 Survey) \\hl{3.93}; Standard Deviation (2010 Survey) \\hl{1.65}; Cronbach's Alpha (2010 Survey) \\hl{0.55}; Mean (2008 Survey) \\hl{3.51}; Standard Deviation (2008 Survey) \\hl{1.47}; Cronbach's Alpha (2008 Survey) \\hl{0.43}. The sixteen correlations of Table A5 that involve this index change as follows, with the other eight indices as built in the replication code.\n\\begin{audittable}{lrrrr}{Correlations of the 2008 General Political Knowledge Index in Table A5, as printed and with the seven shared questions: the lower triangle (2010 survey) and the upper triangle (2007--2009 surveys).}\nOther index & Lower, printed & Lower, seven & Upper, printed & Upper, seven \\\\\nCivil Liberties Knowledge 2007 & 0.440 & 0.436 & 0.411 & 0.414 \\\\\nCivil Liberties Knowledge 2008 & 0.559 & 0.532 & 0.468 & 0.444 \\\\\nCivil Liberties Knowledge 2009 & 0.616 & 0.610 & 0.420 & 0.426 \\\\\nGeneral Political Knowledge 2007 & 0.615 & 0.509 & 0.487 & 0.441 \\\\\nGeneral Political Knowledge 2009 & 0.486 & 0.481 & 0.408 & 0.389 \\\\\nSupport for Civil Liberties 2007 & 0.247 & 0.267 & 0.106 & 0.111 \\\\\nSupport for Civil Liberties 2008 & 0.215 & 0.256 & 0.133 & 0.129 \\\\\nSupport for Civil Liberties 2009 & 0.205 & 0.233 & 0.164 & 0.157 \\\\\n\\end{audittable}", "evidence": "The evidence is a check of the two questionnaires and of the replication code, and a recalculation from the replication data in exact arithmetic. The six statistics printed in the column reproduce at the printed precision for the scores as built ($N = 591$ in 2008 and $N = 251$ in 2010). The question on the local official's name was answered correctly by 216 of 591 respondents in 2008, and the question on the governor's name by 186 of 251 in 2010. Each score on eight questions is the score on the seven shared questions plus the question that differs. For the 2010 mean minus the 2008 mean, I have\n\\[\n\\frac{1172}{251} - \\frac{764}{197} \\;=\\; \\Bigl( \\frac{986}{251} - \\frac{692}{197} \\Bigr) + \\Bigl( \\frac{186}{251} - \\frac{72}{197} \\Bigr),\n\\qquad\\text{that is,}\\qquad\n\\frac{39120}{49447} = \\frac{20550}{49447} + \\frac{18570}{49447} .\n\\]\nThe three terms are approximately 0.791, 0.416 and 0.376. The two questions that differ account for $619/1304$, about 47.5\\%, of the difference between the means on eight questions. This is arithmetic on two samples. It is not an estimate of what the substitution caused, since the 2010 respondents were not asked for the local official's name.\n\\begin{audittable}{lrrrr}{The column as printed and the same statistics for the seven questions asked in both surveys.}\nScore & Questions & Mean & Standard deviation & Alpha \\\\\n2008 survey, as printed & 8 & 3.88 & 1.68 & 0.49 \\\\\n2010 survey, as printed & 8 & 4.67 & 1.81 & 0.57 \\\\\n2008 survey, seven shared questions & 7 & 3.51 & 1.47 & 0.43 \\\\\n2010 survey, seven shared questions & 7 & 3.93 & 1.65 & 0.55 \\\\\n\\end{audittable}", "problem": "The note says that each index of 2007--2009 is restricted to the questions that were asked again in 2010. For this column that is not the case. The 2008 questionnaire asks for the title of the head official of the respondent's town or city and then asks ``What is that person's name?'' The 2010 questionnaire asks for the same title and then asks the respondent to name the governor of the state; it has no question on the name of the local official. The replication code builds the 2008 score with the question on the local official's name and the 2010 score with the question on the governor's name; the other seven questions are the same.\n\nThe printed statistics are correct for the two scores as built. They do not describe the same eight questions. The difference between the two means, $4.67 - 3.88 = 0.79$, combines a difference between the two groups of respondents with a difference between two questions.", "what_survives": "Every printed number in the column is accurate for the score it was computed from. The figure on p.~11 of the Online Appendix calibrates the Civil Liberties Knowledge indices, which contain neither of these two questions, and the treatment-effect regressions use the full 22-question 2008 index of Table 3; neither changes. What changes is the description of this column, and of the corresponding entries of Table A5, as a comparison of equivalent questions."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N018", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of running text. The text reads: the prosecutor receives new evidence that he believes will result in a conviction. Does the Constitution allow the prosecutor to retry Mr. Jones in this case? (Yes/No) 12. Declare a law unconstitutional [Identification: Federal, state, or local branch of government] 13. Police must inform suspected criminals of their constitutional right to remain silent while being arrested. [Matching court case] 14. School. A red outline surrounds the bracket Identification: Federal, state, or local branch of government.", "caption": "Mini Appendix, p. 2, list of the 2008 Civil Liberties Knowledge Index questions. The outline marks the bracket that describes item 12.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N019.png"}], "formal_release": null, "formal_statements": [], "headline": "Item 12 of the 2008 list asks which branch of the federal government can declare a law unconstitutional; it is not a question about the federal, state or local level.", "id": "P043-N019", "location": "Mini Appendix, p. 2 (pages unnumbered), “2008 Civil Liberties Knowledge Index Questions”, item 12", "pdf_page": 2, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "12. Declare a law unconstitutional \\hl{[Identification: Federal, state, or local branch of government]}", "corrected": "12. Declare a law unconstitutional \\hl{[Identification: Which branch of the federal government?]}", "evidence": "The evidence is a check of the pages: the Mini Appendix, Table A2 of the Online Appendix, and the 2008 questionnaire with its coding key in the replication files.", "problem": "The bracket states the type of question. In the 2008 questionnaire this item stands under the heading Separation of Powers, with the choices Executive, Legislative, Judicial, and Both Executive and Legislative, and the coding key scores Judicial as correct. The questions on the federal, state and local levels of government form a different group of items in that questionnaire. Table A2 of the Online Appendix (p.~4) labels the same item ``[Which branch?]''.", "what_survives": "The question, its key, the coding and the results are unaffected; only the label in the Mini Appendix changes. The numbering of the 2009 list on the same page, which passes from 1 to 3, is not an error, since the request to name two rights is scored as two answers, which Table A2 shows as (1) and (2)."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N019", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of Table 4, Generalized Least Squares Estimates of Experimental Treatment Effects on Civil Liberties Knowledge, General Political Knowledge, and Support for Civil Liberties. Three panels, one for each index, each give six treatment estimates with standard errors in parentheses: without and with covariates for the 2007, 2008 and 2009 posttreatment surveys. Red outlines surround six entries in the columns without covariates: civil liberties knowledge 1.248 (0.181) for 2007 and 0.796 (0.228) for 2008; general political knowledge 0.058 (0.113) for 2007, 0.055 (0.279) for 2008 and -0.004 (0.204) for 2009; support for civil liberties 0.026 (0.150) for 2008. The notes below the table read: Standard errors in parentheses. Clusters for random effects model are at the classroom level.", "caption": "Table 4, p. 474 (PDF p. 12), with its notes. The outlines mark the six unadjusted estimates, with their standard errors, that the R script does not return at three decimals.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N020.png"}], "formal_release": null, "formal_statements": [], "headline": "The R script of the replication files fits the unadjusted models of Table 4 by restricted maximum likelihood and not by the random-effects GLS with a moment estimate of the classroom variance, which reproduces the printed table; six of its nine estimates with standard errors differ from the print in the third decimal, and the printed table is supported.", "id": "P043-N020", "location": "Replication script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 198–205 and 227–234; the paper’s counterparts are Table 4, the “Covariates: No” columns, p. 474 (PDF p. 12), and the description of the estimator on p. 472 (PDF p. 10)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{R script, lines 198, 202 and 203 (line 202 is broken here after its third argument); lines 227--234 repeat the construction for the three support indices.}\n\\[\n\\begin{aligned}\n&\\texttt{ivs <- c(\"treatment\", \"stratum\")}\\\\\n&\\texttt{f <- as.formula(paste(name,\" \\textasciitilde\", paste(ivs, collapse=\"+\"),}\\\\\n&\\qquad\\texttt{\"+ (1|classcode)\", sep=\"\"))}\\\\\n&\\texttt{command <- paste(\"GLS.\",name,\" <- }\\hl{\\texttt{lmer(formula=f, data=data)}}\\texttt{\", sep=\"\");}\n\\end{aligned}\n\\]\n\n\\emph{Statistical model, p.~472.} We use Generalized Least Squares (GLS) regression with random effects at the classroom level to estimate average treatment effects (Green and Vavreck 2008).", "corrected": "The Table~4 block of the R script should compute the moment-based GLS of the Stata commands in place of the call to \\hl{\\texttt{lmer}}, or the replication should be run through the Stata commands. For one outcome, let classroom $i$ have $n_i$ students with outcomes $y_{ij}$ and mean $\\bar y_i$, let $G$ be the number of classrooms and $N = \\sum_i n_i$. Let $x_i$ be the \\hl{column vector} of regressors of classroom $i$ (the intercept, the treatment and the stratum indicators, all constant within a classroom), and let $k$ be the rank of the design. Let $b_B$ be the least-squares coefficient of $\\bar y_i$ on $\\hl{x_i^{\\top}}$ over the $G$ classrooms. Define\n\\[\n\\begin{aligned}\n\\widehat\\sigma_e^2 &= \\frac{1}{N-G}\\sum_i\\sum_j (y_{ij}-\\bar y_i)^2, \\qquad\n\\widehat\\sigma_B^2 = \\frac{1}{G-k}\\sum_i (\\bar y_i - x_i^{\\top} b_B)^2,\\\\\n\\widehat\\sigma_u^2 &= \\max\\Bigl\\{0,\\ \\widehat\\sigma_B^2 - \\widehat\\sigma_e^2\\,\\frac{1}{G}\\sum_i \\frac{1}{n_i}\\Bigr\\}, \\qquad\n\\theta_i = 1 - \\sqrt{\\frac{\\widehat\\sigma_e^2}{\\widehat\\sigma_e^2 + n_i\\,\\widehat\\sigma_u^2}} .\n\\end{aligned}\n\\]\nThe estimate $b$ is the least-squares coefficient of $y^*_{ij} = y_{ij} - \\theta_i \\bar y_i$ on $x^*_{ij} = (1-\\theta_i)\\,x_i$ over the $N$ students. Its covariance matrix is $s^2 (X^{*\\top}X^{*})^{-1}$, where $X^*$ has rows $x^{*\\top}_{ij}$ and $s^2 = \\sum_{i}\\sum_{j} (y^*_{ij} - x^{*\\top}_{ij} b)^2/(N-k)$. Setting \\texttt{REML=FALSE} in \\texttt{lmer} is not this repair, since it selects maximum likelihood. Table~4 stays as printed.", "evidence": "The evidence is a recalculation on the public file and an exact check. The nine models of the R script were fitted as written. The moment-based GLS of the corrected block below was computed on the same outcomes, strata, classrooms and samples.\n\\begin{audittable}{lrrr}{Table~4, ``Covariates: No'': treatment estimate and standard error as printed, by moment-based GLS, and from the R script. Shaded entries differ from the print at three decimals.}\nIndex and wave & Printed & Moment-based GLS & R script (REML) \\\\\nCivil liberties knowledge, 2007 & 1.248 (0.181) & 1.248136 (0.180988) & $\\hl{1.250977\\ (0.186482)}$ \\\\\nCivil liberties knowledge, 2008 & 0.796 (0.228) & 0.796109 (0.227714) & $\\hl{0.794538\\ (0.246765)}$ \\\\\nCivil liberties knowledge, 2009 & 0.492 (0.268) & 0.492164 (0.267887) & 0.492164 (0.267887) \\\\\nGeneral political knowledge, 2007 & 0.058 (0.113) & 0.058406 (0.113359) & $\\hl{0.059166\\ (0.114956)}$ \\\\\nGeneral political knowledge, 2008 & 0.055 (0.279) & 0.054524 (0.278920) & $\\hl{0.059257\\ (0.291241)}$ \\\\\nGeneral political knowledge, 2009 & $-0.004$ (0.204) & $-0.004015$ (0.204283) & $\\hl{-0.003581\\ (0.200669)}$ \\\\\nSupport for civil liberties, 2007 & $-0.061$ (0.050) & $-0.060545$ (0.050440) & $-0.060545$ (0.050440) \\\\\nSupport for civil liberties, 2008 & 0.026 (0.150) & 0.026044 (0.149720) & $\\hl{0.023627\\ (0.159055)}$ \\\\\nSupport for civil liberties, 2009 & 0.022 (0.118) & 0.022227 (0.117634) & 0.022227 (0.117634) \\\\\n\\end{audittable}\nThe moment-based GLS agrees with all nine printed entries to rounding. For the exact check, I take the 2008 model for civil liberties knowledge, which has 591 students, 40 classrooms and a design of rank 12. Computing the moment estimates in exact rational arithmetic, I obtain the variance ratio $\\lambda = \\widehat\\sigma_u^2/\\widehat\\sigma_e^2 \\approx 0.000465 > 0$. The derivative of the profiled REML criterion at this $\\lambda$ is about $-80.48$, and exact arithmetic confirms that it is negative. Since $\\lambda$ is an interior point and the derivative is not zero, I conclude that the moment estimate is not a REML optimum. The two procedures differ on these data, and the differences in the table are not rounding in the optimizer. Stata was not available, and the Stata script was not run.", "problem": "The block of the R script is headed TABLE 4 and builds a table named \\texttt{TABLE4.nocov}. It calls \\texttt{lmer} without the argument \\texttt{REML}, whose default is restricted maximum likelihood (REML). The Stata script of the same files fits these nine models with \\texttt{xtreg} and the option \\texttt{re}, which is a random-effects GLS that estimates the classroom variance by the method of moments; it is called moment-based GLS below. Both procedures are GLS with a classroom random effect in the sense of p.~472, since the REML coefficients are the GLS coefficients for the covariance that REML estimates. The sentence on p.~472 is not wrong. The two procedures estimate the classroom variance differently, and the output of the R script does not reproduce the printed table. Six of the nine estimates with standard errors differ from the print at three decimals, and the standard errors differ by up to 0.019.", "what_survives": "The nine unadjusted entries of Table~4 are supported as printed, and no entry changes. At the two-sided 5\\% level, the decisions under REML and under moment-based GLS agree in all nine models. The statement $p < 0.001$ for the 2008 knowledge effect (p.~473) holds for the unadjusted GLS entry in print, whose normal $p$-value is 0.00047; under the REML fit of the R script it would be 0.0013. The adjusted columns are not touched by this finding. The R script changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N020", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of Table 4 with its notes. Three panels, for the Civil Liberties Knowledge Index, the General Political Knowledge Index and the Support for Civil Liberties Index, each give six treatment estimates with standard errors, and rows for Covariates (No or Yes), Strata (Yes) and N. Red outlines surround the nine columns with Covariates equal to Yes: 1.220 (0.155), 0.762 (0.226) and 0.221 (0.269) for civil liberties knowledge; 0.023 (0.076), 0.051 (0.259) and -0.037 (0.202) for general political knowledge; -0.071 (0.051), -0.073 (0.131) and -0.080 (0.119) for support for civil liberties. The second note reads: Covariates are dummy variables generated from the Pretreatment Survey: survey attendance, year of birth, interest in politics, frequency of political discussions at home, party identification, having a job during the school year, mother's highest level of education, and seven factual questions measuring political knowledge.", "caption": "Table 4, p. 474 (PDF p. 12), with its notes. The outlines mark the nine covariate-adjusted models (estimate, standard error and the entry “Yes” in the Covariates row), which the R script does not fit.", "url": "https://pmaronow.github.io/llm-audit/works/P043/exhibits/P043-N021.png"}], "formal_release": null, "formal_statements": [], "headline": "The R script of the replication files does not fit the nine covariate-adjusted models of Table 4, and it fits the item models without the pretreatment covariates that the tables report and the essay models without those specified in the Stata script; no printed number is shown to be wrong.", "id": "P043-N021", "location": "Replication script Green_et_al_JP_2011_Public_Use_EDITED.R, lines 193 and 224, with the Table 4 block (lines 181–253), the item blocks (lines 442–583) and the essay models (lines 613 and 615); the paper’s counterparts are Table 4, the “Covariates: Yes” columns, p. 474 (PDF p. 12), Tables A1 to A3 of the online appendix, and footnote 13, p. 473 (PDF p. 11)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{R script, lines 193 and 224 (line 224 is broken here after its first sentence).}\n\\[\n\\begin{aligned}\n&\\hl{\\texttt{\\#Cannot do random effects with covariates - matrices will not invert}}\\\\\n&\\hl{\\texttt{\\#Cannot perform random effects regression with covariates.}}\\\\\n&\\qquad\\hl{\\texttt{Too many dummy variables results in uninvertible matrix.}}\n\\end{aligned}\n\\]\n\n\\emph{Table~4, notes, p.~474.} $^{\\dagger}$Covariates are dummy variables generated from the Pretreatment Survey: survey attendance, year of birth, interest in politics, frequency of political discussions at home, party identification, having a job during the school year, mother's highest level of education, and seven factual questions measuring political knowledge.\n\n\\emph{Table~A1, note, p.~2 of the online appendix.} All estimates include covariates and fixed effects for strata.", "corrected": "The R script should \\hl{fit the nine covariate-adjusted models of Table~4} and \\hl{include the pretreatment controls in the item and essay models}. The controls are the categorical variables \\texttt{Aq1}, \\texttt{Aq3}, \\texttt{Aq4}, \\texttt{Aq18} to \\texttt{Aq24}, \\texttt{Aq28}, \\texttt{Aq32} and \\texttt{Aq33year} of the Stata commands, with the same coding of a missing pretest, the same handling of redundant indicator columns and the same estimator. Its output should then be compared with the Stata output on one dataset, since adding the terms to \\texttt{lmer} does not by itself establish numerical agreement. The comments at lines 193 and 224 should be replaced by a statement of what the script does not compute. No entry of Table~4 or of Tables~A1 to~A3 changes.", "evidence": "The evidence is a check of the pages and of the two scripts. Evaluating the formula loops of the R script, with the fitting function replaced by one that records its formula, I obtain 114 formulas ($9 + 17 + 49 + 37 + 2$), each with exactly the variables \\texttt{treatment}, \\texttt{stratum} and \\texttt{classcode}. The matching Stata commands (lines 264--270 and 286--288 for Table~4, lines 419--476 for the items, lines 483 and 485 for the essays) each include the 13 variables \\texttt{Aq1}, \\texttt{Aq3}, \\texttt{Aq4}, \\texttt{Aq18} to \\texttt{Aq24}, \\texttt{Aq28}, \\texttt{Aq32} and \\texttt{Aq33year}. I did not fit a model or run the Stata script.", "problem": "The block of the R script headed TABLE 4 requests nine models, each with the treatment, the strata and a classroom random intercept. Two comments say that the models with covariates are left out because the matrices will not invert. The 17 item models for Table~A1, the 49 and 37 item models for Tables~A2 and~A3 and the two essay models have the same right-hand side. The R script alone therefore cannot reproduce the ``Covariates: Yes'' columns of Table~4 or the item tables of the appendix, whose notes say that all estimates include covariates. The Stata script of the same files specifies each of these models with 13 pretreatment variables as categorical controls. The comments describe a difficulty of this R implementation. They do not show that the adjusted models cannot be fitted, and the paper does not say that the R script produced the printed adjusted entries.", "what_survives": "This omission does not refute the printed tables, the statistical model or the conclusions of the paper. The unadjusted specifications of the R script and the adjusted specifications of the Stata script stand. The R script changes, since at present it cannot serve to check the adjusted results."}, "url": "https://pmaronow.github.io/llm-audit/works/P043/#P043-N021", "work": {"arxiv_id": null, "doi": "10.1017/s0022381611000107", "title": "Does Knowledge of Constitutional Principles Increase Support for Civil Liberties? Results from a Randomized Field Experiment", "url": "https://pmaronow.github.io/llm-audit/works/P043/"}, "work_id": "P043"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of six lines of text. A red box marks the words 'the interactive effect of direct and indirect exposure' in the sentence 'For example, the interactive effect of direct and indirect exposure results from the contrast between estimates of the average potential outcome in the direct + indirect exposure and the average potential outcome in the no exposure condition.'", "caption": "p. 217 (PDF p. 9), last paragraph. The marked phrase names the contrast that the rest of the sentence defines.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-A001.png"}], "formal_release": null, "formal_statements": [], "headline": "The “interactive effect of direct and indirect exposure” on p. 217 is the joint effect of the two exposures, $\\mu_{11}-\\mu_{00}$, and not the additive interaction.", "id": "P044-A001", "location": "p. 217 (PDF p. 9), last paragraph, sentence beginning “For example, the interactive effect”", "pdf_page": 9, "printed_page": "217", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``For example, \\hl{the interactive effect of direct and indirect exposure} results from the contrast between estimates of the average potential outcome in the \\emph{direct + indirect exposure} and the average potential outcome in the \\emph{no exposure} condition.''", "corrected": "``For example, \\hl{the joint effect of direct and indirect exposure, including any interaction,} results from the contrast between estimates of the average potential outcome in the \\emph{direct + indirect exposure} and the average potential outcome in the \\emph{no exposure} condition.''", "intended_meaning": "Let $\\mu_{ab}$ denote the mean potential outcome under direct-exposure indicator $a$ and indirect-exposure indicator $b$. The sentence defines its contrast in words, and that contrast is $\\mu_{11}-\\mu_{00}$, the joint effect of direct and indirect exposure. The estimand, the estimators and the entries of Table 16.1 are those of this contrast, and the wording changes none of them. The term ``interactive effect'' ordinarily denotes the additive interaction $\\mu_{11}-\\mu_{10}-\\mu_{01}+\\mu_{00}$, which is a different contrast. With the chapter's multipliers $(2,1.5,1.25)$ and baseline mean $b$, the joint contrast is $b$ and the additive interaction is $b/4$."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-A001", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of the first paragraph under the heading Network experimental setting and the first two lines of the next paragraph. A red box marks the words 'there are no channels through which j could influence i, even indirectly).' They close the sentence that begins 'A value of 0 in row i and column j means that i's potential outcomes most certainly do not depend on j's treatment status'.", "caption": "Section “Network experimental setting”, first paragraph, p. 210 (PDF p. 2). The marked words say that a 0 in the adjacency matrix leaves no channel of influence, even an indirect one. The sentence before them gives the meaning of a 1.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-A003.png"}, {"alt": "Crop of the first five lines of text on the page. A red box marks the words 'on whether peers of peers are treated,' in the sentence that begins 'One might imagine other possible exposure mappings'.", "caption": "Section “Exposure mappings and causal effects”, p. 212 (PDF p. 4), first lines of the page. The marked words allow exposure mappings in which outcomes depend on the treatment of peers of peers.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-A003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The statement that a 0 in the adjacency matrix excludes all influence, even indirect influence, is correct for a matrix that records every permitted influence; for a matrix of direct ties it holds only when interference is restricted to immediate peers, and the text does not say which matrix is meant.", "id": "P044-A003", "location": "Section “Network experimental setting”, first paragraph, p. 210 (PDF p. 2); compare the sentence on other exposure mappings, p. 212 (PDF p. 4), first paragraph", "pdf_page": 2, "printed_page": "210", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~210: ``A value of 1 in row $i$ and column $j$ in the adjacency matrix means that $i$'s potential outcomes might depend on whether or not $j$ is treated (e.g., they may be immediate peers or close to each other in the network). A value of 0 in row $i$ and column $j$ means that $i$'s potential outcomes most certainly do not depend on $j$'s treatment status (e.g., there are no channels through which $j$ could influence $i$, \\hl{even indirectly}).''\n\np.~212: ``One might imagine other possible exposure mappings that suggest potential outcomes might depend on the number of peers treated, \\hl{on whether peers of peers are treated}, and possibly on other covariate values.''", "corrected": "The text should say whether the adjacency matrix records immediate links or all possible influence under the specified exposure mapping. A wording for p.~210 that does so:\n\n``A value of 0 in row $i$ and column $j$ of \\hl{a matrix of direct ties} means that there is no edge from $j$ to $i$. It excludes dependence of $i$'s potential outcomes on $j$'s treatment status \\hl{only when interference is restricted to immediate peers}. When indirect paths are allowed, the exposure mapping must account for the relevant influence neighborhood, which may include peers of peers.''", "intended_meaning": "The paragraph defines the entries of $\\theta$ by possible influence, and p.~211 calls $\\theta$ the interference network. Under this definition, a 0 in row $i$ and column $j$ means that the treatment of $j$ cannot affect $i$ by any route, and the sentence is correct. The worked example agrees with this definition, since it restricts interference to first-degree connections (p.~215). The same paragraph also identifies the entries of $\\theta$ with the edges of Figure~16.1, and p.~212 allows exposure mappings that depend on whether peers of peers are treated. If $\\theta$ records only direct ties, a 0 does not exclude that dependence. On the path $1$--$2$--$3$, I have $\\theta_{13}=0$ and $(\\theta^2)_{13}=1$, and the outcome $y_1(\\mathbf{z})=I(z_3=1)$ depends on the treatment of unit $3$. The exposure probabilities, the estimates and the conclusions of the worked example do not depend on which matrix is meant."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-A003", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the last five lines of the section. A red box marks the words 'unbiased Horvitz-Thompson estimators of the tau of d, d prime effects' in the sentence that begins 'The probabilities can be used to construct'.", "caption": "Section “Design-based inference”, p. 212 (PDF p. 4), last sentence. The marked words claim unbiased estimation of the population contrast, with no condition on the exposure probabilities.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E001.png"}, {"alt": "Crop of six lines of text. Red boxes mark the words 'unbiased estimates of average potential outcomes' and 'unbiased estimates of contrasts between these averages', on two consecutive lines of the sentence that begins 'This is precisely what inverse probability estimators like the Horvitz-Thompson take into account'.", "caption": "Section “Examples”, p. 216 (PDF p. 8). The marked words repeat the claim of unbiasedness, for averages and for contrasts.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E001-b.png"}, {"alt": "Crop of the heading 'NOTES' and the two notes below it. Red boxes mark Note 1, 'Our networks do not have isolates, but we suggest researchers remove isolates from their analysis.', which runs over two lines. Note 2 reads 'For a formal discussion of the estimators, see Aronow and Samii (2017).'", "caption": "Note 1, p. 221 (PDF p. 13). The marked note is the chapter’s only guidance related to the condition. The note is not itself incorrect, and removing isolates does not guarantee the condition.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E001-c.png"}], "formal_release": null, "formal_statements": [{"id": "P044-T3", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T3"}, {"id": "P044-T4", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T4"}, {"id": "P044-T12", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T12"}], "headline": "The Horvitz–Thompson estimator of the population contrast $\\tau(d,d')$ is unbiased only if every unit has positive probability of each of the two exposures $d$ and $d'$ under the design; the chapter does not state this condition, and neither random assignment nor the absence of isolates guarantees it.", "id": "P044-E001", "location": "Section “Design-based inference”, p. 212 (PDF p. 4), last sentence; Section “Examples”, p. 216 (PDF p. 8), paragraph beginning “We now estimate exposure-specific effects”; Note 1, p. 221 (PDF p. 13)", "pdf_page": 4, "printed_page": "212", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P044-T12", "P044-T3", "P044-T4"], "text": {"as_printed": "p.~212: ``The probabilities can be used to construct \\hl{unbiased} Horvitz-Thompson estimators of the $\\tau(d,d')$ effects or consistent and considerably more efficient Hajek or weighted least squares estimators.''\n\np.~216: ``This is precisely what inverse probability estimators like the Horvitz--Thompson take into account and the reason why we can obtain \\hl{unbiased} estimates of average potential outcomes in a specific exposure condition, and \\hl{unbiased} estimates of contrasts between these averages across different exposures.''\n\nNote~1, p.~221: ``Our networks do not have isolates, but we suggest researchers remove isolates from their analysis.''", "corrected": "p.~212: ``\\hl{Suppose that every unit $i$ in the population has positive probability of each of the two exposures being contrasted, that is, $\\Pr(f(\\mathbf{Z},\\theta_i)=d)>0$ and $\\Pr(f(\\mathbf{Z},\\theta_i)=d')>0$ under the experimental design and the exposure mapping.} The probabilities can \\hl{then} be used to construct unbiased Horvitz-Thompson estimators of the $\\tau(d,d')$ effects.''\n\nIn full, let the exposure mapping be correctly specified, so that $Y_i=y_i(D_i)$ with $D_i=f(\\mathbf{Z},\\theta_i)$, and let $\\pi_i(e)=\\Pr(D_i=e)$ be the exact exposure probabilities under the design. Suppose that $\\pi_i(d)>0$ and $\\pi_i(d')>0$ for every $i\\in U$. Then the estimator\n\\[\n\\widehat\\tau_{HT}(d,d')=\\frac{1}{N}\\sum_{i=1}^N\\left[\\frac{\\mathbf{1}(D_i=d)\\,Y_i}{\\pi_i(d)}-\\frac{\\mathbf{1}(D_i=d')\\,Y_i}{\\pi_i(d')}\\right]\n\\]\nsatisfies $\\mathbb{E}[\\widehat\\tau_{HT}(d,d')]=\\tau(d,d')$.\n\nIf the condition fails, either the assignment design is changed so that it holds, or the target is restricted to the fixed set $S=\\{i\\in U:\\pi_i(d)>0,\\ \\pi_i(d')>0\\}$, provided that $S$ is nonempty, with\n\\[\n\\tau_S(d,d')=\\frac{1}{|S|}\\sum_{i\\in S}\\bigl(y_i(d)-y_i(d')\\bigr),\n\\]\nwhere the original network, design and exposure mapping are retained. This is a different estimand. The same qualification applies to the sentence on p.~216. To Note~1 one may add: ``\\hl{Having no isolates does not by itself guarantee this condition under every design.}''", "evidence": "The evidence is an exact counterexample, obtained by enumerating all assignments in rational arithmetic, and a derivation of the corrected statement. I consider four units in a star: unit 1 is joined to units 2, 3 and 4, and there are no other edges. The degrees are $(3,1,1,1)$, so no unit is an isolate. Exactly two units are treated, chosen uniformly from the six pairs. This is complete random assignment with $N=4$ and $N_1=2$. Under the chapter's mapping with four conditions, the exposure probabilities are as follows.\n\n\\begin{audittable}{lrrrr}{Exposure probabilities in the four-unit star when exactly two units are treated uniformly at random, by exact enumeration of the six assignments.}\nUnit & $\\pi_i(d_{11})$ & $\\pi_i(d_{10})$ & $\\pi_i(d_{01})$ & $\\pi_i(d_{00})$ \\\\\n1 (centre, degree 3) & $1/2$ & $0$ & $1/2$ & $0$ \\\\\n2, 3 and 4 (leaves, degree 1) & $1/6$ & $1/3$ & $1/3$ & $1/6$ \\\\\n\\end{audittable}\n\nUnit 1 never receives $d_{10}$ or $d_{00}$. Indeed, if it is treated, then one of its peers is treated, and if it is untreated, then two of its peers are treated. I compare two schedules of potential outcomes. In the first, $y_i(d)=1$ for every unit and every exposure. The second differs from the first only in that $y_1(d_{00})=5$. Under each of the six assignments, the observed outcomes are $(1,1,1,1)$ in both schedules. The joint distribution of assignments, exposures and outcomes is then the same in both, and any estimator has the same expectation under both. However, $\\tau(d_{11},d_{00})=0$ in the first schedule and $\\tau(d_{11},d_{00})=(1-5)/4=-1$ in the second. No estimator is unbiased under both. If each inverse-weighted sum is taken over the units observed in the exposure and divided by $N=4$, then the expected contrast is $1-3/4=1/4$ under both schedules. The bias is $1/4$ in the first schedule and $5/4$ in the second.\n\nUnder the condition, the estimator is unbiased. Let $D_i=f(\\mathbf{Z},\\theta_i)$, and let $\\widehat\\tau_{HT}(d,d')$ be the difference between the averages over $i$ of $\\mathbf{1}(D_i=d)\\,Y_i/\\pi_i(d)$ and of $\\mathbf{1}(D_i=d')\\,Y_i/\\pi_i(d')$. Since $Y_i=y_i(d)$ on the event $D_i=d$ and $\\mathbb{E}[\\mathbf{1}(D_i=d)]=\\pi_i(d)$, I have\n\\[\n\\mathbb{E}\\left[\\frac{\\mathbf{1}(D_i=d)\\,Y_i}{\\pi_i(d)}\\right]=\\frac{y_i(d)}{\\pi_i(d)}\\,\\pi_i(d)=y_i(d).\n\\]\nAveraging over $i$ and subtracting the same identity for $d'$, I obtain $\\mathbb{E}[\\widehat\\tau_{HT}(d,d')]=\\tau(d,d')$. No condition on joint exposure probabilities, and no independence across units, is used.\n\nFor complete random assignment of $M$ of $N$ units and the mapping with four conditions, a unit with $k$ peers has\n\\[\n\\pi_i(d_{00})=\\binom{N-1-k}{M}\\Big/\\binom{N}{M},\\qquad\n\\pi_i(d_{10})=\\binom{N-1-k}{M-1}\\Big/\\binom{N}{M},\n\\]\nwhere a binomial coefficient is zero when its lower index is negative or exceeds its upper index. For $2\\le M\\le N-1$, all four exposures have positive probability exactly when $1\\le k\\le N-M-1$. If isolates are removed, the lower end of this range is secured, and the upper end is not. In the star, $N=4$, $M=2$ and $k=3$ for the centre. In the chapter's design, $N=1000$ and $M=100$, and the range is $1\\le k\\le 899$.", "problem": "The chapter defines the effect $\\tau(d,d')=N^{-1}\\sum_{i=1}^N(y_i(d)-y_i(d'))$ as an average over the whole population, and it says on p.~212 that the exposure probabilities give unbiased Horvitz--Thompson estimators of it. This holds only under a condition that the chapter does not state. Let $\\pi_i(d)=\\Pr(f(\\mathbf{Z},\\theta_i)=d)$ be the probability that unit $i$ receives exposure $d$ under the design. The condition is that $\\pi_i(d)>0$ and $\\pi_i(d')>0$ for every unit $i$. If $\\pi_j(d)=0$ for some unit $j$, then $y_j(d)$ is never observed. It can be changed without changing the distribution of anything that is observed, while $\\tau(d,d')$ changes with it. In that case no estimator is unbiased for $\\tau(d,d')$ over all potential outcomes, whatever weights it uses.\n\nRandom assignment does not guarantee the condition: there exist networks and designs such that some unit has probability zero of an exposure. Under complete random assignment, for example, a unit with enough peers cannot have all of them untreated. The only related guidance in the chapter is Note~1, which recommends that isolates be removed. The note does not claim that this is sufficient, and it is not sufficient: the example below has no isolates. The sentence on p.~216 needs the same condition when it is read as a general rule. The condition is the usual proviso of the Horvitz--Thompson result. The error is that the chapter states the result in general without it.", "what_survives": "The exposure mapping and the estimand are well defined, and the Horvitz--Thompson estimator is unbiased wherever the condition holds. This counterexample does not establish an error in the estimates, the simulation or the comparison of precision for the two example networks. The degrees displayed in Figure~16.2 are compatible with the range $1\\le k\\le 899$, and no failure of the condition is established for either network. Note~1 is not itself incorrect. What changes is the general statement on p.~212, and the sentence on p.~216 when it is read as a general rule. An application in which some unit has probability zero of one of the two exposures must change the design or the target."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E001", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of text. Two red boxes mark the words 'from an absolute standard normal distribution, which is correlated with the unit's first- and second-order degree.' The sentence begins 'Specifically, we draw values for the baseline no exposure condition', and the following sentences say that the other exposure conditions are obtained with multipliers (2, 1.5, 1.25).", "caption": "p. 216 (PDF p. 8), paragraph beginning “To show how to compute exposure-specific effects”. The marked clause describes the distribution of the baseline outcomes; the code called a few lines below generates a different one.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P044-T5", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T5"}, {"id": "P044-T6", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T6"}, {"id": "P044-T13", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T13"}], "headline": "The baseline outcomes of the simulation are generated as $(k_i+q_i+k_iq_i)\\,|G_i|+E_i$, the absolute value of a standard normal draw scaled by a function of the degrees, plus normal noise; they are not draws from an absolute standard normal distribution.", "id": "P044-E002", "location": "p. 216 (PDF p. 8), paragraph beginning “To show how to compute exposure-specific effects”, second sentence; the call that follows runs on to p. 217 (PDF p. 9)", "pdf_page": 8, "printed_page": "216", "reaches": [{"effect": "same_error_restated", "effect_text": "a corresponding statement is printed there", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [{"id": "P045-E006", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E006", "work_id": "P045"}], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P044-T13", "P044-T5", "P044-T6"], "text": {"as_printed": "``Specifically, we draw values for the baseline \\emph{no exposure} condition \\hl{from an absolute standard normal distribution, which is correlated with the unit's first- and second-order degree}.''\n\npp.~216--217, joined on two lines:\n\n\\texttt{potential\\_outcome\\_sw <- make\\_dilated\\_out(adj\\_matrix\\_sw, make\\_corr\\_out,}\n\n\\texttt{hop = 1, seed = 547)}", "corrected": "Specifically, conditional on the network, the baseline \\emph{no exposure} outcome of unit $i$ is generated as \\hl{$B_i=(k_i+q_i+k_iq_i)\\,|G_i|+E_i$, where $k_i$ is the first-neighbor degree of unit $i$, $q_i$ is the number of other units reachable from $i$ by a walk of two edges, $G_i\\sim N(0,1)$ and $E_i\\sim N(1,1/16)$, with $1/16$ the variance, and the draws are independent conditional on the network}.", "evidence": "I checked the function that the printed call invokes, using an exact counterexample and a short derivation. The function computes\n\n\\texttt{(degree + degree2 + degree * degree2) * abs(rnorm(length(degree))) +}\n\n\\texttt{rnorm(length(degree), 1, 0.25)}\n\nwhere \\texttt{degree2} counts the other units $j$ for which the $(i,j)$ entry of the squared adjacency matrix is positive. For the counterexample, let three units form a triangle. Then $k_i=q_i=2$ and $B_i=8\\,|G_i|+E_i$ for each unit. With the draws $G=(1/64,1,-1/2)$ and $E=(-3/4,1,5/4)$, I obtain $B=(-5/8,\\,9,\\,21/4)$. The functions of the package, run with these draws in place of the random ones, return these values and the corresponding multiples for the other exposures. On the event that $|G_1|<1/32$ and $-1<E_1<-1/2$, which has positive probability, I have $B_1<-1/4$. The absolute value of a standard normal variable is never negative. For the moments, recall that $\\mathbb{E}|G_i|=\\sqrt{2/\\pi}$ and $\\operatorname{Var}(|G_i|)=1-2/\\pi$. Since $G_i$ and $E_i$ are independent given the network, I find $\\mathbb{E}[B_i]=a_i\\sqrt{2/\\pi}+1$ and $\\operatorname{Var}(B_i)=a_i^2(1-2/\\pi)+1/16$ with $a_i=k_i+q_i+k_iq_i$.\n\n\\begin{audittable}{lll}{Baseline outcome of a unit with $k_i=q_i=2$ (a vertex of a triangle): the printed description and the generated variable.}\nQuantity & As described: $|G_i|$ & As generated: $8\\,|G_i|+E_i$ \\\\\nMean & $\\sqrt{2/\\pi}\\approx0.798$ & $8\\sqrt{2/\\pi}+1\\approx7.383$ \\\\\nVariance & $1-2/\\pi$ & $64(1-2/\\pi)+1/16\\approx23.319$ \\\\\nNegative values & impossible & possible: $B_i=-5/8$ at $G_i=1/64$, $E_i=-3/4$ \\\\\n\\end{audittable}\n\nTwo separate recalculations illustrate the scale in the chapter's designs. On the small-world network regenerated from the printed code and seeds under R 4.4.2 and igraph 2.2.2, which need not be the chapter's realization (\\fid{P044-N004}), I obtain the three true effects 467.75, 233.88 and 116.94 from the generated potential outcomes. On the degree-corrected block model they are 623.80, 311.90 and 155.95 (\\fid{P044-N009}).", "problem": "The sentence gives the wrong outcome model for the only simulation of the chapter. The printed call passes \\texttt{make\\_corr\\_out} to \\texttt{make\\_dilated\\_out} with \\texttt{hop = 1}, and the baseline outcome of unit $i$ is then generated as $B_i=(k_i+q_i+k_iq_i)\\,|G_i|+E_i$. Here $k_i$ is the degree of unit $i$, and $q_i$ is the number of other units that a walk of two edges reaches from $i$. The variable $G_i$ is standard normal, and $E_i$ is normal with mean 1 and standard deviation $1/4$. The baseline outcome is not the absolute value of a standard normal variable, and it can be negative. Its mean given the network is $(k_i+q_i+k_iq_i)\\sqrt{2/\\pi}+1$, while the absolute value of a standard normal variable has mean $\\sqrt{2/\\pi}\\approx0.80$. The first exceeds the second by at least 1 for every unit of a network without isolates. With the multipliers $(2,1.5,1.25)$, the three average effects are the mean of the $B_i$, one half of it and one quarter of it. Under the printed description they would be about 0.80, 0.40 and 0.20 in expectation. Table 16.1 reports estimates between 115.79 and 891.08. The sentence also does not say how strongly the outcomes depend on degree. The simulation cannot be rebuilt from the sentence alone, although the printed call and the package determine it.", "what_survives": "The multipliers $(2,1.5,1.25)$ agree with the code. The exposure mapping, the estimators and the design-based argument do not depend on the description of the baseline. The correction changes the sentence and the matching sentence of the package's help page; it changes no code and no reported number. I did not recompute Table 16.1 or Figure 16.3 for this correction, and neither is shown to be wrong by it."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E002", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of text. Red boxes mark the words 'an expected proportion of edges that is within a community of 0.7', which run over two lines of the sentence that begins 'In particular, we generate a network according to a DCBM'.", "caption": "Section “Examples”, p. 214 (PDF p. 6), paragraph below Figure 16.2. The marked words give 0.7 as the expected proportion of edges within a community.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E003.png"}, {"alt": "Crop of the printed R function make_adj_matrix_dcsbm. Red boxes mark 'rho = 0.7' in each of the two calls randnet::BlockModel.Gen(10, N, K=10, rho = 0.7, alpha = 1). The second call is inside a while loop that repeats while some row sum of the adjacency matrix is zero.", "caption": "Code on p. 215 (PDF p. 7), function make_adj_matrix_dcsbm. The marked argument rho = 0.7 appears in both calls to the generator; the argument beta is not passed.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P044-T10", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T10"}], "headline": "In the printed call to randnet::BlockModel.Gen, the argument rho = 0.7 is the probability that a unit receives the low degree weight $0.2$, and every edge of the generated network lies within a community; the proportion of edges within a community is 1, not 0.7.", "id": "P044-E003", "location": "Section “Examples”, p. 214 (PDF p. 6), paragraph below Figure 16.2; code for make_adj_matrix_dcsbm, p. 215 (PDF p. 7)", "pdf_page": 6, "printed_page": "214", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P044-T10"], "text": {"as_printed": "p.~214: ``In particular, we generate a network according to a DCBM with $N=1000$ units, 10 communities (or clusters); where units are assigned to one of the 10 communities uniformly at random, and \\hl{an expected proportion of edges that is within a community of 0.7}.''\n\np.~215, in both calls to the generator (line break of the printed code joined):\n\n\\texttt{randnet::BlockModel.Gen(10, N, K=10, rho = 0.7, alpha = 1)}", "corrected": "``In particular, we generate a network according to a DCBM with $N=1000$ units, 10 communities (or clusters); where units are assigned to one of the 10 communities uniformly at random, and \\hl{all edges are within communities. Before graphs with isolated vertices are rejected, each unit is given the degree weight 0.2 with probability 0.7 and the weight 1 otherwise}.''", "evidence": "I checked the source code of the generator and a recalculation. In \\texttt{randnet} 0.5, the arguments of \\texttt{BlockModel.Gen} have the defaults \\texttt{beta=0}, \\texttt{rho=0}, \\texttt{simple=TRUE} and \\texttt{alpha=5}. The function sets the block matrix to \\texttt{diag(w)} and fills its off-diagonal entries only \\texttt{if(beta > 0)}, and in the simple branch it sets \\texttt{node.degree[runif(n)<rho] <- 0.2}. The manual describes \\texttt{beta} as the ``out-in ratio: the ratio of between-block edges over within-block edges''. The code of the generator is the same in versions 0.4, 0.5 and 0.7. The chapter does not say which version it used.\n\nWhen the printed function is run with seed 42, the loop accepts the 193rd draw. The accepted network has all of its edges within communities and ten connected components. A share $0.723$ of its units have weight $0.2$, its mean degree is $9.858$, its degree variance is $75.87$ and its skewness is $1.075$. It reproduces the six effect estimates and the six variance estimates that are printed for the DCBM on p.~218, to their printed precision.", "problem": "The text reads the argument \\texttt{rho = 0.7} of the generator as the proportion of edges within communities. In \\texttt{randnet::BlockModel.Gen} the argument has a different meaning. Between-community connection probabilities are controlled by \\texttt{beta}, relative to within-community connection probabilities. The printed call does not pass \\texttt{beta}, and its default is $0$. The block probability matrix is then diagonal, and no edge joins two communities. Under the default \\texttt{simple = TRUE}, the argument \\texttt{rho} is the probability that a unit is given the degree weight $0.2$ in place of $1$, and \\texttt{alpha} is not used. The second example network is therefore misdescribed, and it cannot be recovered from the sentence.", "what_survives": "The degree variance of 75 and the skewness of 1.07 quoted in the same paragraph approximate those of the network that the printed code generates. The printed DCBM estimates follow from that network. The definitions, the estimand and the design-based arguments do not depend on the proportion of edges within communities, and the small-world network is unaffected. If a network with 70\\% of its edges within communities was intended, the settings of the generator would have to be changed and the DCBM example recomputed."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E003", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of two paragraphs of running text. A red box marks the opening words 'These exposure probabilities are generalized propensity scores'. In the second paragraph, red boxes mark the words 'unbiased estimates' twice, in the sentence 'the reason why we can obtain unbiased estimates of average potential outcomes in a specific exposure condition, and unbiased estimates of contrasts between these averages across different exposures'.", "caption": "p. 216 (PDF p. 8), the two paragraphs after the make_exposure_prob calls. The first marked phrase refers to the output of the routine; the other two say that the estimates computed from it are unbiased.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E005.png"}], "formal_release": null, "formal_statements": [{"id": "P044-T7", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T7"}, {"id": "P044-T8", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T8"}], "headline": "The Horvitz–Thompson estimators are unbiased with the true positive exposure probabilities of the design; make_exposure_prob returns smoothed frequencies, $(c+1)/(R+1)$, even when every assignment of the design is supplied, and the estimates computed from them are not exactly unbiased.", "id": "P044-E005", "location": "p. 216 (PDF p. 8), the two paragraphs after the make_exposure_prob calls; p. 212 (PDF p. 4), last paragraph. The routine is in the R package interference, R/make_exposure_prob.R, lines 83 and 88, and is not printed in the chapter.", "pdf_page": 8, "printed_page": "216", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [{"id": "P045-N001", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N001", "work_id": "P045"}], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P044-T7", "P044-T8"], "text": {"as_printed": "p.~216: ``\\hl{These exposure probabilities} are generalized propensity scores that can then be used in Horvitz--Thompson estimators or other inverse probability weighted estimators, such as the Hajek estimator of exposure-specific causal effects.''\n\np.~216: ``This is precisely what inverse probability estimators like the Horvitz--Thompson take into account and the reason why we can obtain \\hl{unbiased estimates} of average potential outcomes in a specific exposure condition, and \\hl{unbiased estimates} of contrasts between these averages across different exposures.''\n\np.~212: ``The probabilities can be used to construct unbiased Horvitz-Thompson estimators of the $\\tau(d,d')$ effects or consistent and considerably more efficient Hajek or weighted least squares estimators.''\n\n\\texttt{R/make\\_exposure\\_prob.R}, lines 83 and 88 (same-exposure and cross-exposure matrices):\n\n\\texttt{(I\\_exposure[[i]] \\%*\\% t(I\\_exposure[[i]]) \\hl{+ diag(N)}) / \\hl{(R + 1)}}\n\n\\texttt{(I\\_exposure[[i]] \\%*\\% t(I\\_exposure[[j]])) / R}", "corrected": "Let $\\pi_i(d)=\\Pr(D_i=d)$ be the true exposure probabilities of the design and $Y_i=y_i(D_i)$ the observed outcome. If $\\pi_i(d)>0$ and $\\pi_i(d')>0$ for every unit $i$, then the Horvitz--Thompson estimator\n\\[\n\\widehat\\tau_{HT}(d,d')=\\frac{1}{N}\\sum_{i=1}^{N}\\Bigl(\\frac{\\mathbf{1}\\{D_i=d\\}\\,Y_i}{\\pi_i(d)}-\\frac{\\mathbf{1}\\{D_i=d'\\}\\,Y_i}{\\pi_i(d')}\\Bigr)\n\\]\nis unbiased for $\\tau(d,d')$. The exposure probabilities computed on p.~216 are frequencies over $R=10{,}000$ simulated assignments, smoothed as $(c+1)/(R+1)$, and the estimates computed from them \\hl{have no guarantee of exact unbiasedness}. When all $R$ equally likely assignments of a design are enumerated, the exact probabilities are $\\hl{c_i(d)/R}$ and $\\hl{c_{ij}(d,d')/R}$, and line 83 of the routine should then read\n\n\\texttt{(I\\_exposure[[i]] \\%*\\% t(I\\_exposure[[i]])) / \\hl{R}}", "evidence": "The evidence is an exact counterexample, which I also ran through the unchanged functions of the package. Let eight units form a cycle, and let the design be uniform on the eight rotations of the assignment $(1,1,0,1,0,0,0,0)$. Every assignment places two units in $d_{11}$, one in $d_{10}$, three in $d_{01}$ and two in $d_{00}$, and every unit has the true exposure probabilities $(1/4,1/8,3/8,1/4)$. Supplied with all eight assignments, the routine returns $(1/3,2/9,4/9,1/3)$, which sum to $4/3$. Let every unit have potential outcomes $(2,3/2,5/4,1)$ under $(d_{11},d_{10},d_{01},d_{00})$. Then $\\tau(d_{11},d_{00})=1$, $\\tau(d_{10},d_{00})=1/2$ and $\\tau(d_{01},d_{00})=1/4$. With the returned probabilities, the Horvitz--Thompson contrasts equal $3/4$, $3/32$ and $39/128$ under every assignment, and these values are their expectations. With the true probabilities they equal $1$, $1/2$ and $1/4$.\n\nThe general computation is short. Let $q_i(d)$ be positive weights used in place of $\\pi_i(d)$. Since $\\mathbb{E}[\\mathbf{1}\\{D_i=d\\}]=\\pi_i(d)$, I have\n\\[\n\\mathbb{E}\\Bigl[\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\mathbf{1}\\{D_i=d\\}\\,y_i(d)}{q_i(d)}\\Bigr]=\\frac{1}{N}\\sum_{i=1}^{N}y_i(d)\\,\\frac{\\pi_i(d)}{q_i(d)}.\n\\]\nFor a design of $R$ equally likely assignments, all supplied, $\\pi_i(d)=c_i(d)/R$ and $q_i(d)=(c_i(d)+1)/(R+1)$. The ratio $\\pi_i(d)/q_i(d)$ then equals $c_i(d)(R+1)/\\bigl(R(c_i(d)+1)\\bigr)$, which differs from 1 whenever $0<c_i(d)<R$.", "problem": "The statement is correct for the true exposure probabilities $\\pi_i(d)=\\Pr(D_i=d)$, when they are positive. The routine that computes ``these exposure probabilities'' does not return them. Let $c_i(d)$ denote the number of the $R$ supplied assignments under which unit $i$ has exposure $d$, and let $c_{ij}(d,d')$ denote the number under which unit $i$ has exposure $d$ and unit $j$ has exposure $d'$. The routine returns $(c_i(d)+1)/(R+1)$ for the probability that unit $i$ has exposure $d$, $c_{ij}(d,d)/(R+1)$ for the joint probability of two different units in the same exposure, and $c_{ij}(d,d')/R$ for two different exposures. It does so even when the $R$ assignments are all the assignments of the design, a case that p.~214 allows (``the set of all possible (or a large random sample of all possible) treatment assignments''). The four returned probabilities of a unit sum to $(R+4)/(R+1)$, which exceeds 1. Weighting by their inverses does not give an exactly unbiased estimator. The chapter does not mention the smoothing.", "what_survives": "The exposure mapping and the estimands are unaffected, and the Horvitz--Thompson estimator remains unbiased when the exposure probabilities are the true ones and are positive. The chapter already says that the Hajek estimator is biased. The estimator computed from the package's probabilities has no guarantee of exact unbiasedness, even when all assignments are enumerated. The counterexample does not measure the bias at the chapter's $R=10{,}000$. The recalculation made for \\fid{P044-N009} measures it on the degree-corrected block model generated by the printed code. There, conditional on the chapter's simulated probabilities and with an independent assignment, the expectation of the Horvitz--Thompson estimator exceeds the true effect by 0.5\\%, 1.5\\% and 0.9\\% for $(d_{11},d_{00})$, $(d_{10},d_{00})$ and $(d_{01},d_{00})$. No entry of Table 16.1 and no comparison of Figure 16.3 is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E005", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Code listing under the page header '218 Handbook of computational social science'. Red boxes mark the first line of the call, estimates_sw <- estimates(received_exposure_sw, and the line estimates_sw$tau_ht. The call continues with observed_outcome_sw, prob_exposure_sw and control_condition = 'no'. The output below shows dir_ind1 478.5162, isol_dir 174.5077 and ind1 115.7963.", "caption": "p. 218 (PDF p. 10): the call to estimates() and the Horvitz–Thompson contrasts that it returns. The package line that sets the total of an empty exposure to NA is not printed in the chapter; the marks identify the function call and its point-contrast output.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E006.png"}], "formal_release": null, "formal_statements": [], "headline": "When no unit receives an exposure, the Horvitz–Thompson total for that exposure is zero and the contrast is defined; the package returns a missing value instead.", "id": "P044-E006", "location": "p. 218 (PDF p. 10), the calls to estimates() and the output tau_ht. The line at issue is in the R package interference, R/estimates.R, line 219 (also line 624, and R/estimates_batch.R, line 222), and is not printed in the chapter.", "pdf_page": 10, "printed_page": "218", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": "P013-E018", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P045-E004", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P013-E018", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E018", "work_id": "P013"}, {"id": "P045-E004", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E004", "work_id": "P045"}], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~218, joined on two lines:\n\n\\texttt{estimates\\_sw <- estimates(received\\_exposure\\_sw, observed\\_outcome\\_sw,}\n\n\\texttt{prob\\_exposure\\_sw, control\\_condition = 'no')}\n\n\\texttt{estimates\\_sw\\$tau\\_ht}\n\n\\texttt{R/estimates.R}, lines 216--219 (the first three joined on one line):\n\n\\texttt{yT\\_ht <- rowSums(obs\\_outcome\\_by\\_exposure / obs\\_prob\\_exposure\\_individual\\_kk, na.rm = T)}\n\n\\texttt{yT\\_ht[rowSums(!is.na(obs\\_outcome\\_by\\_exposure)) == 0] <- \\hl{NA}}", "corrected": "Let $\\pi_i(d)>0$ for every unit $i$. The Horvitz--Thompson total under exposure $d$ is $\\widehat T_d=\\sum_{i=1}^{N}\\mathbf{1}\\{D_i=d\\}\\,Y_i/\\pi_i(d)$, which \\hl{equals 0 when no unit receives $d$}. If also $\\pi_i(d')>0$ for every $i$, then $\\widehat\\tau_{HT}(d,d')=(\\widehat T_d-\\widehat T_{d'})/N$ is defined under every assignment and is unbiased for $\\tau(d,d')$. In the package, \\texttt{R/estimates.R}, lines 216--219, \\hl{with line 219 deleted}:\n\n\\texttt{yT\\_ht <- rowSums(obs\\_outcome\\_by\\_exposure / obs\\_prob\\_exposure\\_individual\\_kk, na.rm = T)}\n\nThe same line should be deleted at \\texttt{R/estimates.R}, line 624, and at \\texttt{R/estimates\\_batch.R}, line 222. The change applies when all outcomes are observed; outcomes that are themselves missing need separate handling. The Hajek mean of an empty exposure remains missing, and the variance code is not repaired by this change.", "evidence": "The evidence is an exact counterexample, which I also ran on the unchanged function. Let six units form a cycle, and let two of them be treated, uniformly over the 15 assignments. Every unit has exposure probabilities $(2/15,1/5,7/15,1/5)$ for $(d_{11},d_{10},d_{01},d_{00})$. Let every unit have potential outcomes $(2,3/2,5/4,1)$, so that $\\tau(d_{11},d_{00})=1$. Under the assignment $(1,0,1,0,0,0)$ no unit receives $d_{11}$ and one unit receives $d_{00}$. The totals are $\\widehat T_{d_{11}}=0$ and $\\widehat T_{d_{00}}=5$, and the Horvitz--Thompson contrast is $(0-5)/6=-5/6$. The function returns \\texttt{NA}. Over the 15 assignments, the contrast equals $10/3$ six times (the two treated units are adjacent), $-5/6$ six times and $0$ three times, and its mean is 1. The function returns a value only in the first six cases, and the mean of the returned values is $10/3$. With the overwrite removed, the function returns the exact totals and contrasts under all 15 assignments.\n\nThe general statement takes one line. Since $\\mathbf{1}\\{D_i=d\\}\\,Y_i=\\mathbf{1}\\{D_i=d\\}\\,y_i(d)$ and $\\pi_i(d)>0$, I have $\\mathbb{E}\\bigl[\\mathbf{1}\\{D_i=d\\}\\,Y_i/\\pi_i(d)\\bigr]=y_i(d)$ for each unit, whether or not the exposure occurs under a given assignment. Summing over $i$ gives $\\mathbb{E}[\\widehat T_d]=\\sum_i y_i(d)$, which implies that $\\mathbb{E}\\bigl[(\\widehat T_d-\\widehat T_{d'})/N\\bigr]=\\tau(d,d')$.", "problem": "The package documents \\texttt{yT\\_ht} as the Horvitz--Thompson estimator of the total of the potential outcomes under an exposure, $\\widehat T_d=\\sum_i\\mathbf{1}\\{D_i=d\\}\\,Y_i/\\pi_i(d)$. When no unit receives $d$, every term of the sum is zero and $\\widehat T_d=0$. Line 219 replaces this zero by \\texttt{NA}. The contrast \\texttt{tau\\_ht} is then missing, although the estimator that the chapter calls unbiased (pp.~212 and 216) equals a finite number. An average over the assignments for which the function returns a value conditions on a random event and can have a different expectation. A missing value is appropriate for the Hajek mean, which is $0/0$ for an empty exposure.", "what_survives": "The Horvitz--Thompson estimator remains unbiased when the exposure probabilities are the true ones and are positive. The estimands are unaffected, and so are the point estimates under any assignment in which every exposure occurs. Table 16.1 prints finite estimates, and no entry of it and no result of Figure 16.3 is shown to change. I did not estimate how often an exposure is empty on the chapter's networks of 1000 units. The correction concerns the point totals only and implies nothing about the variance estimators."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E006", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of eight lines of text. The first line ends the definition of the potential outcome y i of z i and z minus i. Four red boxes mark the bold z and the bold z prime that stand as second arguments in 'y i of z i; z equals y i of z i; z prime' and in 'y i of z i; z is not equal to y i of z i; z prime'.", "caption": "Section “Network experimental setting”, p. 211 (PDF p. 3), second paragraph. The four marked second arguments are the full vectors $\\mathbf{z}$ and $\\mathbf{z}'$; the definition in the first line of the crop has $\\mathbf{z}_{-i}$ in that place.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E007.png"}, {"alt": "Crop of three lines of text and a display of four lines. A red box marks the word 'row' in 'unit i's row in the adjacency matrix, theta i'. Four more red boxes mark the product z prime theta i inside the indicator in each of the four lines of the display.", "caption": "Section “Exposure mappings and causal effects”, p. 211 (PDF p. 3), the display defining $f(\\mathbf{z},\\theta_i)$. The marked word says that $\\theta_i$ is a row, and each of the four marked products multiplies the row $\\mathbf{z}'$ by it.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E007-b.png"}], "formal_release": null, "formal_statements": [{"id": "P044-T9", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T9"}], "headline": "In the statements of no interference and of interference, the second argument of the potential outcome is the vector of the other units' assignments, and in the exposure-mapping display the count of treated peers is the scalar $\\theta_i\\mathbf{z}$.", "id": "P044-E007", "location": "Section “Network experimental setting”, p. 211 (PDF p. 3), second paragraph, statements of no interference and of interference; Section “Exposure mappings and causal effects”, p. 211 (PDF p. 3), display defining $f(\\mathbf{z},\\theta_i)$; definition of $\\mathbf{z}$, p. 210 (PDF p. 2)", "pdf_page": 3, "printed_page": "211", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P044-T9"], "text": {"as_printed": "p.~210: ``The $N\\times 1$ vector of treatments for all units is given by $\\mathbf{z}=(z_1,\\ldots,z_N)'$.''\n\np.~211: ``Then we can also denote unit $i$'s potential outcome as $y_i(z_i;\\mathbf{z}_{-i})$. \\ldots\\ Traditional analyses of experiments assume no interference, which would imply that unit $i$'s outcome depends only on the element $z_i$; that is, for any $\\mathbf{z}$ and $\\mathbf{z}'$, $y_i(z_i;\\hl{\\mathbf{z}})=y_i(z_i;\\hl{\\mathbf{z}'})$. Interference means for some unit $i\\in U$ we have $y_i(z_i;\\hl{\\mathbf{z}})\\neq y_i(z_i;\\hl{\\mathbf{z}'})$.''\n\np.~211: ``This would imply an exposure mapping that takes as inputs the assignment profile $\\mathbf{z}$ and unit $i$'s row in the adjacency matrix, $\\theta_i$, as follows:'' The display that follows is\n\\[\nf(\\mathbf{z},\\theta_i)=\\begin{cases}\nd_{11}(\\text{Direct}+\\text{IndirectExposure}): & z_i\\mathbf{I}(\\hl{\\mathbf{z}'\\theta_i}>0)=1,\\\\\nd_{10}(\\text{IsolatedDirectExposure}): & z_i\\mathbf{I}(\\hl{\\mathbf{z}'\\theta_i}=0)=1,\\\\\nd_{01}(\\text{IndirectExposure}): & (1-z_i)\\mathbf{I}(\\hl{\\mathbf{z}'\\theta_i}>0)=1,\\\\\nd_{00}(\\text{NoExposure}): & (1-z_i)\\mathbf{I}(\\hl{\\mathbf{z}'\\theta_i}=0)=1\n\\end{cases}\n\\]", "corrected": "Let $y_i(a;\\mathbf{u})$ denote unit $i$'s potential outcome when its own assignment is $a\\in\\{0,1\\}$ and the assignments of the other $N-1$ units are $\\mathbf{u}\\in\\{0,1\\}^{N-1}$.\n\n``\\ldots\\ that is, for every unit $i$, every $a\\in\\{0,1\\}$ and all $\\mathbf{u},\\mathbf{v}\\in\\{0,1\\}^{N-1}$, $y_i(a;\\hl{\\mathbf{u}})=y_i(a;\\hl{\\mathbf{v}})$. Interference means that there exist a unit $i\\in U$, a value $a\\in\\{0,1\\}$ and vectors $\\mathbf{u},\\mathbf{v}\\in\\{0,1\\}^{N-1}$ such that $y_i(a;\\hl{\\mathbf{u}})\\neq y_i(a;\\hl{\\mathbf{v}})$.''\n\nWith $\\theta_i$ the $i$th row of the adjacency matrix and $\\mathbf{z}$ the $N\\times 1$ column of assignments,\n\\[\nf(\\mathbf{z},\\theta_i)=\\begin{cases}\nd_{11}(\\text{Direct}+\\text{IndirectExposure}): & z_i\\mathbf{I}(\\hl{\\theta_i\\mathbf{z}}>0)=1,\\\\\nd_{10}(\\text{IsolatedDirectExposure}): & z_i\\mathbf{I}(\\hl{\\theta_i\\mathbf{z}}=0)=1,\\\\\nd_{01}(\\text{IndirectExposure}): & (1-z_i)\\mathbf{I}(\\hl{\\theta_i\\mathbf{z}}>0)=1,\\\\\nd_{00}(\\text{NoExposure}): & (1-z_i)\\mathbf{I}(\\hl{\\theta_i\\mathbf{z}}=0)=1,\n\\end{cases}\n\\]\nwhere $\\theta_i\\mathbf{z}=\\sum_{j=1}^N\\theta_{ij}z_j$.", "evidence": "I checked the pages and the following derivation. For $a\\in\\{0,1\\}$ and $\\mathbf{u}\\in\\{0,1\\}^{N-1}$, let $y_i(a;\\mathbf{u})$ denote $y_i$ evaluated at the profile obtained by inserting $a$ in position $i$ of $\\mathbf{u}$. No interference is the statement that $y_i(a;\\mathbf{u})=y_i(a;\\mathbf{v})$ for all $i$, $a$, $\\mathbf{u}$ and $\\mathbf{v}$. Interference is its negation, in which the two vectors of assignments are quantified existentially. For the display, the product of the $1\\times N$ row $\\theta_i$ and the $N\\times 1$ column $\\mathbf{z}$ is the scalar\n\\[\n\\theta_i\\mathbf{z}=\\sum_{j=1}^N\\theta_{ij}z_j .\n\\]\nBy the definition of $\\theta$ on p.~210, its entries are $0$ or $1$ and its diagonal is zero. The sum counts the treated units among those whose treatment may affect unit $i$, and it is positive exactly when at least one of them is treated.", "problem": "Two pieces of notation on p.~211 do not match the definitions given just before them. First, the potential outcome with two arguments is defined as $y_i(z_i;\\mathbf{z}_{-i})$, with the $(N-1)$-vector of the other units' assignments in the second place. The four expressions that follow carry the full vectors $\\mathbf{z}$ and $\\mathbf{z}'$ there, without the subscript $-i$, so that the assignment of unit $i$ itself enters twice. Second, $\\mathbf{z}$ is an $N\\times 1$ column and $\\theta_i$ is unit $i$'s row of the adjacency matrix. Then $\\mathbf{z}'$ and $\\theta_i$ are both $1\\times N$, and the product $\\mathbf{z}'\\theta_i$ is not defined for $N>1$. The intended meaning is clear in both places, and it is correct. Under no interference, the assignments of the other units do not change unit $i$'s outcome when its own assignment is held fixed. The indicator in the display tests whether at least one of unit $i$'s peers is treated.", "what_survives": "Nothing else in the chapter depends on these misprints. The definition of interference, the four exposure conditions, the estimand and all later analysis are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E007", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Code listing. The call reads estimates_sw <- estimates(received_exposure_sw, observed_outcome_sw, prob_exposure_sw, control_condition = 'no'), and a red box marks the argument control_condition = 'no'. Below it are the line estimates_sw$tau_ht and the output dir_ind1 478.5162, isol_dir 174.5077, ind1 115.7963.", "caption": "p. 218 (PDF p. 10): the call to estimates(). The marked argument names the control condition. The chapter supplies 'no', the one value for which the point contrasts are computed as requested.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-E008.png"}], "formal_release": null, "formal_statements": [], "headline": "The function estimates() takes its point contrasts against the exposure named 'no' whatever control_condition is supplied; they should be taken against the supplied control condition, and the chapter’s calls, which supply 'no', are unaffected.", "id": "P044-E008", "location": "p. 218 (PDF p. 10), the two calls estimates(..., control_condition = 'no'). The lines at issue are in the R package interference, R/estimates.R, lines 314, 318 and 333–336, and are not printed in the chapter.", "pdf_page": 10, "printed_page": "218", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P013-N013", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P013-N013", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N013", "work_id": "P013"}], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~218, joined on two lines:\n\n\\texttt{estimates\\_sw <- estimates(received\\_exposure\\_sw, observed\\_outcome\\_sw,}\n\n\\texttt{prob\\_exposure\\_sw, control\\_condition = 'no')}\n\n\\texttt{R/estimates.R}, lines 314 and 318:\n\n\\texttt{tau\\_ht <- (1 / N) * (yT\\_ht - yT\\_ht[\\hl{'no'}])[names(yT\\_ht) != \\hl{'no'}]}\n\n\\texttt{tau\\_h <- (mu\\_h - mu\\_h[\\hl{'no'}])[names(mu\\_h) != \\hl{'no'}]}", "corrected": "Let \\texttt{control\\_condition} name a single existing exposure. In \\texttt{R/estimates.R}, lines 314, 318 and 333--336, each statement on two lines:\n\n\\texttt{tau\\_ht <- (1 / N) * (yT\\_ht - yT\\_ht[\\hl{control\\_condition}])[}\n\n\\texttt{names(yT\\_ht) != \\hl{control\\_condition}]}\n\n\\texttt{tau\\_h <- (mu\\_h - mu\\_h[\\hl{control\\_condition}])[}\n\n\\texttt{names(mu\\_h) != \\hl{control\\_condition}]}\n\n\\texttt{means\\_dsm <- rowMeans(obs\\_outcome\\_by\\_exposure, na.rm = TRUE)}\n\n\\texttt{tau\\_dsm <- (means\\_dsm - means\\_dsm[\\hl{control\\_condition}])[}\n\n\\texttt{names(means\\_dsm) != \\hl{control\\_condition}]}\n\nIn \\texttt{R/estimates\\_batch.R}, lines 248 and 252, the same replacement applies to the Horvitz--Thompson and Hajek contrasts. Lines 262--263 of \\texttt{R/estimates.R} belong to the optional constant-effect variance estimator and would read as follows; I did not run that branch.\n\n\\texttt{diffs <- const\\_eff\\$means - const\\_eff\\$means[[control\\_condition]]}\n\n\\texttt{diffs <- diffs[, names(diffs) != control\\_condition, with = FALSE]}", "evidence": "The evidence is an exact counterexample, which I also ran on the unchanged function. Let eight units form a cycle, let three of them be treated, uniformly over the 56 assignments, and let $y_i(d)=i$ for every exposure $d$. Every unit has exposure probabilities $(11/56,5/28,25/56,5/28)$ for $(d_{11},d_{10},d_{01},d_{00})$. Under the assignment $(1,1,0,0,1,0,0,0)$ every exposure occurs, and the Horvitz--Thompson totals are $(168/11,\\,28,\\,1176/25,\\,196/5)$. With \\texttt{control\\_condition = 'ind1'}, the contrast of $d_{11}$ against $d_{01}$ is $(168/11-1176/25)/8=-1092/275$. The function returns $(168/11-196/5)/8=-329/110$, the contrast against $d_{00}$, in a vector with the names \\texttt{dir\\_ind1}, \\texttt{isol\\_dir}, \\texttt{ind1}. Its variance estimates carry the names \\texttt{dir\\_ind1}, \\texttt{isol\\_dir}, \\texttt{no} and are those of the contrasts against $d_{01}$. When I rename \\texttt{'no'} as \\texttt{'baseline'} throughout and supply \\texttt{control\\_condition = 'baseline'}, each of the three point vectors consists of four \\texttt{NA} values. The three entries of each default variance vector are finite and unchanged. With the corrected lines for \\texttt{tau\\_ht}, \\texttt{tau\\_h} and \\texttt{tau\\_dsm}, the function returns the requested contrasts in both cases, and its output for \\texttt{control\\_condition = 'no'} is unchanged.", "problem": "The argument \\texttt{control\\_condition} names the exposure against which the other exposures are contrasted, and the variance estimates that the function returns by default, \\texttt{var\\_tau\\_ht} and \\texttt{var\\_tau\\_h}, use it. The point contrasts \\texttt{tau\\_ht}, \\texttt{tau\\_h} and \\texttt{tau\\_dsm} do not: they subtract, and exclude, the exposure literally named \\texttt{'no'}. If the no-exposure condition has another name, which an exposure mapping defined by the user (pp.~215--216) may give it, every point contrast is \\texttt{NA}. If another exposure is requested as the control condition, the point contrasts are still taken against \\texttt{'no'}, while the variance estimates are those of the contrasts against the requested control condition. The estimates and their variances then refer to different contrasts.", "what_survives": "The definition of the contrasts $\\tau(d,d')$, the totals and means for each exposure, and calls that supply \\texttt{control\\_condition = 'no'} are unaffected by this defect. The two calls on p.~218 supply \\texttt{'no'}, Table 16.1 takes $d_{00}$ as the reference, and nothing reported in the chapter changes. The correction is to the package."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E008", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of text. A red box marks '5,' in the sentence 'However, unit 2 is also tied to other peers of unit 3: units 5, 4, and 1.'", "caption": "Section “Design-based inference”, p. 213 (PDF p. 5), first paragraph. The marked unit 5 is not tied to unit 2 in Figure 16.1.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N003.png"}, {"alt": "Drawing of a network with ten numbered vertices. A red box marks vertex 5, which lies between vertex 2, above it and to its left, and vertex 4, below it. A grey stroke leaves vertex 2 toward the lower right, disappears behind the disk of vertex 5 and reappears below it, beside a second stroke, to reach vertex 4. Vertex 2 has other strokes to vertices 1, 3 and 10.", "caption": "Figure 16.1, p. 210 (PDF p. 2). The stroke that leaves vertex 2 toward the lower right passes behind the marked vertex 5 and ends at vertex 4; no edge joins vertices 2 and 5.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N003-b.png"}], "formal_release": null, "formal_statements": [], "headline": "In Figure 16.1, unit 2 is tied to units 1, 3, 4 and 10; the other peers of unit 3 to which it is tied are units 4 and 1, and unit 5 is not among them.", "id": "P044-N003", "location": "Section “Design-based inference”, p. 213 (PDF p. 5), first paragraph; Figure 16.1, p. 210 (PDF p. 2)", "pdf_page": 5, "printed_page": "213", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``In the network in Figure 16.1, we can see that units 1 and 2 are both connected to unit 3, creating a dependency on whether these two units will receive indirect exposure. However, unit 2 is also tied to other peers of unit 3: units \\hl{5,} 4, and 1.''", "corrected": "``However, unit 2 is also tied to other peers of unit 3: units \\hl{4 and 1}.''", "evidence": "I checked the figure. In the image embedded in the PDF, a line was fitted to the visible part of the stroke that leaves vertex 2 toward the lower right. The line passes within $0.3$ pixels of the centres of vertices 2 and 4, and it misses the centre of vertex 5 by about $23$ pixels. The stroke continues below vertex 5 to vertex 4.", "problem": "The list includes unit 5, and Figure~16.1 shows no edge between units 2 and 5. The edge from unit 2 to unit 4 is drawn behind the disk of vertex 5, and this makes it look like an edge to unit 5. The point of the passage is that units with shared peers have dependent exposures. That point is correct, and units 4 and 1 illustrate it.", "what_survives": "No estimator, result or calculation depends on the list. The statement about dependence through shared peers stands."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N003", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of five lines of text. A red box marks '0.12' in the words 'than the vertex degree distribution in the small-world network (with a skewness of 0.12)'.", "caption": "Section “Examples”, p. 214 (PDF p. 6). The marked skewness of the small-world network is 0.12; the network regenerated from the printed code under R 4.4.2 and igraph 2.2.2 has 0.07.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N004.png"}, {"alt": "Crop of the printed R function make_adj_matrix_small_world and the line that calls it. Red boxes mark the call igraph::sample_smallworld(dim=1, size=N, nei=5, p=0.1), which runs over two lines, and the argument 'seed = 42' in the line adj_matrix_sw <- make_adj_matrix_small_world(N, seed = 42).", "caption": "Code on p. 215 (PDF p. 7), function make_adj_matrix_small_world. The marked call and the marked seed are all that the chapter gives to determine the small-world network.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N004-b.png"}, {"alt": "Crop of printed R output. Red boxes mark four rows of numbers: 478.5162 174.5077 115.7963 under the heading tau_ht; 26198.147 52201.813 3038.021 under var_tau_ht; 456.9802 187.6371 123.1371 under tau_h; and 16088.628 17991.860 1017.804 under var_tau_h. The columns are headed dir_ind1, isol_dir and ind1.", "caption": "Output on p. 218 (PDF p. 10) for estimates_sw. The four marked rows are the printed small-world estimates and variance estimates; none of them is obtained when the printed code is run under R 4.4.2 and igraph 2.2.2.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N004-c.png"}], "formal_release": null, "formal_statements": [], "headline": "Under R 4.4.2 and igraph 2.2.2, the printed generator and seeds do not reproduce the printed results for the small-world network, and the chapter does not state the software versions needed to reproduce them.", "id": "P044-N004", "location": "Section “Examples”, p. 214 (PDF p. 6), paragraph below Figure 16.2; code for make_adj_matrix_small_world, p. 215 (PDF p. 7); output for estimates_sw, p. 218 (PDF p. 10); Table 16.1, Small-World columns, p. 219 (PDF p. 11)", "pdf_page": 6, "printed_page": "214", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "p.~214: ``\\ldots\\ the vertex degree distribution in this DCBM is also substantially more skewed (with a skewness of 1.07) than the vertex degree distribution in the small-world network (with a skewness of 0.12).''\n\np.~215 (line breaks of the printed code joined):\n\n\\texttt{set.seed(seed)}\n\n\\texttt{g <- igraph::sample\\_smallworld(dim=1, size=N, nei=5, p=0.1)}\n\n\\texttt{adj\\_matrix\\_sw <- make\\_adj\\_matrix\\_small\\_world(N, seed = 42)}\n\np.~218, output in the columns \\texttt{dir\\_ind1}, \\texttt{isol\\_dir}, \\texttt{ind1}:\n\n\\texttt{estimates\\_sw\\$tau\\_ht}: 478.5162, 174.5077, 115.7963\n\n\\texttt{estimates\\_sw\\$var\\_tau\\_ht}: 26198.147, 52201.813, 3038.021\n\n\\texttt{estimates\\_sw\\$tau\\_h}: 456.9802, 187.6371, 123.1371\n\n\\texttt{estimates\\_sw\\$var\\_tau\\_h}: 16088.628, 17991.860, 1017.804", "corrected": "No printed value is replaced. The correction is an addition: \\hl{the chapter should state the versions of R, igraph and the \\texttt{interference} package, the kind of random number generator and the seeds with which the results were obtained, and the small-world network, the realized assignment and the 10,000 sampled assignments should be archived.}\n\nIf instead a new network is generated under R 4.4.2 and igraph 2.2.2, the degree summary on p.~214, Figure~16.2, the output on p.~218 and the Small-World columns of Table~16.1 must all be replaced together. Changing 0.12 to 0.07 alone would not make the printed output reproducible.", "evidence": "I checked a recalculation from the printed code and the source of the \\texttt{interference} package (version 0.1.0). The regenerated network has 5000 edges and degrees from 5 to 15. In exact arithmetic, its degree variance is $233/125$ and its third central moment is $9/50$, which implies that the skewness lies between $0.07072$ and $0.07074$. The adjusted skewness also rounds to 0.07. The two sampling rules of R, the R 4.4.2 default (``Rejection'') and the older one (``Rounding''), give the same network and different assignments, and neither reproduces the printed output.\n\n\\begin{audittable}{lrrr}{Small-world estimates: the output printed on p.~218 and the values obtained from the printed code and seeds under R 4.4.2 and igraph 2.2.2, for each sampling rule.}\nEstimate & Printed & Regenerated, Rejection & Regenerated, Rounding \\\\\nHorvitz--Thompson, $(d_{11},d_{00})$ & 478.5162 & 388.58 & 314.98 \\\\\nHorvitz--Thompson, $(d_{10},d_{00})$ & 174.5077 & 448.62 & 170.37 \\\\\nHorvitz--Thompson, $(d_{01},d_{00})$ & 115.7963 & 152.29 & $-9.21$ \\\\\nHajek, $(d_{11},d_{00})$ & 456.9802 & 529.16 & 389.37 \\\\\nHajek, $(d_{10},d_{00})$ & 187.6371 & 244.61 & 225.87 \\\\\nHajek, $(d_{01},d_{00})$ & 123.1371 & 142.75 & 117.27 \\\\\n\\end{audittable}\n\nThe variance estimates differ in the same way. Table~16.1 agrees with the printed output: its 18 Small-World entries (estimates, standard errors and $t$-statistics) match the printed output after truncation to two decimal places.", "problem": "The small-world half of the worked example cannot be reproduced from the printed code. Under R 4.4.2 and igraph 2.2.2, the printed generator with seed 42 generates a network whose degree skewness is 0.07, where the chapter prints 0.12. Its mean degree is 10 and its degree variance rounds to 2, as printed. When that network is carried through the rest of the printed code with the printed seeds, every estimate and every variance estimate differs from the output on p.~218, and from the Small-World columns of Table~16.1, which agree with that output. A seed fixes the state of a random number generator, and it does not fix the implementation of \\texttt{sample\\_smallworld}. The chapter states no version of R or of igraph, and no archive of the network that it used has been located. This is a gap in reproducibility. It does not show that any printed number is wrong for the network that the chapter used.", "what_survives": "The exposure mapping, the estimands and the design-based arguments are untouched, and so is the role of the example as an illustration. The regenerated network still has mean degree 10, a degree variance near 2 and a tight degree distribution. The DCBM results and Figure~16.3 are not shown wrong by this point. What cannot be done from the printed code alone is to reproduce the degree summary of the small-world network, the output on p.~218 and the Small-World columns of Table~16.1."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N004", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text beginning 'Next, we estimate exposure-specific effects using both the Horvitz-Thompson and Hajek estimators, as well as the variance estimators of these effects'. Two red boxes mark the words 'which are conservative approximations to the exact variances.'", "caption": "p. 217 (PDF p. 9), last paragraph, first sentence. The marked clause calls the variance estimators conservative.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N005.png"}, {"alt": "Crop of three lines of text and a code listing. A red box marks the words 'R = 10,000 random treatment assignments.' The code reads N <- 1000, p <- 0.1, possible_tr_assignment <- make_tr_vec_permutation(N, p, R = 10000, seed = 4224), and a second red box marks 'seed = 4224'.", "caption": "p. 214 (PDF p. 6): the 10,000 simulated assignments from which the exposure probabilities are computed. The marked seed is used again on p. 217.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N005-seed-p214.png"}, {"alt": "Crop of the sentence 'We also need to assign treatment and compute units' received exposure condition according to our assumed exposure scenario.' and a code listing that begins observed_tr_assignment <- make_tr_vec_permutation(N, p, R = 1, seed = 4224). A red box marks 'seed = 4224'.", "caption": "p. 217 (PDF p. 9): the realized assignment. With the marked seed it is the first of the 10,000 simulated assignments of p. 214.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N005-seed-p217.png"}, {"alt": "Bar chart with three groups of three bars, one group for each contrast, and a horizontal line at the ratio 1. For the contrast (d11, d00) the bars are 1.673 with simulated probabilities and the realized assignment among the simulated assignments, 0.617 with simulated probabilities and an independent realized assignment, and 1.745 with exact probabilities. For (d10, d00) they are 1.687, 0.035 and 1.696. For (d01, d00) they are 0.996, 0.995 and 1.008.", "caption": "Expected Horvitz–Thompson variance estimate divided by the variance of the Horvitz–Thompson estimator, for the three contrasts in the small-world design regenerated from the printed code ($N=1000$, 100 units treated). The blue and orange bars use probabilities simulated from $R=10{,}000$ assignments and condition on those assignments; the green bars use exact probabilities. A value below 1 means that the variance estimator is not conservative.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N005-ht-variance-ratio.png"}], "formal_release": null, "formal_statements": [{"id": "P044-T1", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T1"}, {"id": "P044-T2", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T2"}, {"id": "P044-T11", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-T11"}], "headline": "The Horvitz–Thompson variance estimator is conservative with exact exposure probabilities and positive marginal support, but not in general with probabilities simulated from 10,000 assignments, as in the chapter: in a small-world network regenerated from the printed code its expectation is 0.617, 0.035 and 0.995 times the variance of the estimator for the three contrasts when the realized assignment is independent of the simulated ones.", "id": "P044-N005", "location": "p. 217 (PDF p. 9), last paragraph, first sentence; the code lines with seed = 4224 on p. 214 (PDF p. 6) and p. 217", "pdf_page": 9, "printed_page": "217", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}], "related_entries": [{"id": "P045-N016", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N016", "work_id": "P045"}], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": ["P044-T1", "P044-T11", "P044-T2"], "text": {"as_printed": "p.~217: ``Next, we estimate exposure-specific effects using both the Horvitz--Thompson and Hajek estimators, as well as the variance estimators of these effects, \\hl{which are conservative approximations to the exact variances}.''\n\np.~214, joined on one line:\n\n\\texttt{possible\\_tr\\_assignment <- make\\_tr\\_vec\\_permutation(N, p, R = 10000, \\hl{seed = 4224})}\n\np.~217: ``We also need to assign treatment and compute units' received exposure condition according to our assumed exposure scenario.''\n\n\\texttt{observed\\_tr\\_assignment <- make\\_tr\\_vec\\_permutation(N, p, R = 1, \\hl{seed = 4224})}", "corrected": "Next, the exposure-specific effects are estimated using both the Horvitz--Thompson and Hajek estimators, together with variance estimators of these effects. \\hl{Let $\\pi_i(d)$ and $\\pi_{ij}(d,d')$ be the exact exposure probabilities of the design, and suppose that $\\pi_i(d)>0$ and $\\pi_i(d')>0$ for every unit $i$. Then the Horvitz--Thompson variance estimator $\\widehat V_{HT}(d,d')$ computed with these probabilities, including its terms for pairs of units with joint exposure probability zero, is conservative:}\n\\[\n\\mathbb{E}\\bigl[\\widehat V_{HT}(d,d')\\bigr]\\;\\ge\\;\\operatorname{Var}\\bigl(\\widehat\\tau_{HT}(d,d')\\bigr).\n\\]\n\\hl{The exposure probabilities used below are simulated from $R=10{,}000$ assignments, and the variance estimator computed from them need not be conservative, even at $R=10{,}000$ and even if the realized assignment is one of the simulated assignments, as it is in the code below, where both calls use seed 4224. Exact probabilities retain this variance guarantee; a method using simulated probabilities needs a separate justification that accounts for their simulation error. Increasing the number of assignments with suitable accuracy checks is one possible computational remedy; the necessary effort depends on the joint exposure probabilities relevant to the contrasts. This expectation guarantee alone does not guarantee confidence-interval coverage. The Hajek variance estimator is a linearized approximation and has no general finite-sample guarantee.}\n\nFor the design of the examples the exact probabilities have closed forms. Under complete randomization of $m$ of $N$ units, the probability that $o$ given units are treated and $f$ other given units are not is $Q(o,f)=\\binom{N-o-f}{m-o}\\big/\\binom{N}{m}$. For a unit with $g_i$ neighbors, $\\pi_i(d_{10})=Q(1,g_i)$, $\\pi_i(d_{00})=Q(0,g_i+1)$, $\\pi_i(d_{11})=m/N-Q(1,g_i)$ and $\\pi_i(d_{01})=(N-m)/N-Q(0,g_i+1)$. The joint probabilities follow in the same way by inclusion and exclusion over the two neighbor sets.", "evidence": "I checked the claim with a recalculation in the chapter's small-world design, exact counterexamples and a derivation, supplemented by a second recorded 10,000-assignment table and a finite-bootstrap comparison.\n\n\\emph{The small-world design.} The small-world network was regenerated with the printed code ($N=1000$, \\texttt{nei = 5}, rewiring probability 0.1, seed 42), the 10,000 simulated assignments (100 of 1000 units treated, seed 4224) and the potential outcomes (seed 547). The package then computed the exposure probabilities. The regenerated network need not be the chapter's realization: its degree variance is 1.87 and its skewness is 0.07, against 2 and 0.12 on p.~214. Conditional on the 10,000 simulated assignments, the expectation of each variance estimator and the variance of each effect estimator were computed under two distributions of the realized assignment. In the first, the realized assignment is uniform on the 10,000 simulated assignments. In the second, it is drawn by complete randomization, independently of them. For the Horvitz--Thompson estimator, both moments are exact sums over the joint exposure probabilities of the distribution in question, with an empty exposure sum defined as zero; the package instead returns an undefined value when an exposure group is absent. The shares of negative variance estimates, the coverage and the Hajek ratios use all 10,000 simulated assignments in the first case and are estimated from 30,000 independent assignments in the second; all evaluated assignments have all four exposure groups present.\n\n\\begin{audittable}{llrrrr}{Small-world design regenerated from the printed code ($N=1000$, 100 units treated, exposure probabilities simulated from $R=10{,}000$ assignments). HT: Horvitz--Thompson. Ratio: expected variance estimate divided by the variance of the effect estimator, conditional on the simulated assignments. Negative: share of realized assignments under which the variance estimate is negative. Coverage: share under which the interval $\\widehat\\tau_{HT}\\pm1.96\\,\\widehat V_{HT}^{1/2}$ contains the true effect; a negative $\\widehat V_{HT}$ counts as a miss. Independent-assignment negative shares, coverage and Hajek ratios are simulation estimates.}\nRealized assignment & Contrast & HT ratio & HT negative & HT coverage & Hajek ratio \\\\\none of the simulated & $(d_{11},d_{00})$ & 1.673 & 0.00\\% & 97.9\\% & 1.824 \\\\\none of the simulated & $(d_{10},d_{00})$ & 1.687 & 0.17\\% & 96.3\\% & 1.705 \\\\\none of the simulated & $(d_{01},d_{00})$ & 0.996 & 0.00\\% & 94.9\\% & 1.006 \\\\\nindependent & $(d_{11},d_{00})$ & 0.617 & 8.97\\% & 75.8\\% & 1.817 \\\\\nindependent & $(d_{10},d_{00})$ & 0.035 & 40.06\\% & 44.4\\% & 1.824 \\\\\nindependent & $(d_{01},d_{00})$ & 0.995 & 0.00\\% & 94.8\\% & 1.022 \\\\\n\\end{audittable}\n\nAveraging also over the simulated assignments by numerical integration, with a negligible bound on the approximation of sampling without replacement, the Horvitz--Thompson ratios are 1.664, 1.667 and 0.996 in the first case and 0.632, 0.072 and 0.980 in the second. With exact exposure probabilities the ratios are 1.745, 1.696 and 1.008. With exact marginal probabilities and simulated joint probabilities they are 0.648, 0.020 and 1.006 for an independent realized assignment. This shows that the shortfall persists with exact marginal probabilities. The ratios for the degree-corrected block model were not computed.\n\n\\emph{Exact counterexamples.} In each, the exposure mapping is the chapter's, treatment is by complete randomization, and every unit has positive probability of each of the four exposures. The contrast is $(d_{01},d_{00})$. The simulated assignments are a uniform sample of distinct assignments, as \\texttt{make\\_tr\\_vec\\_permutation} draws them, and the moments are exact rational numbers.\n\\begin{enumerate}\n\\item Twelve units form three disjoint cliques of four, two units are treated, all potential outcomes equal 1, and the probabilities are simulated from 65 of the 66 assignments. Over all such samples, the variance of the Horvitz--Thompson estimator is 0.48766 and the expected variance estimate is 0.44710 when the realized assignment is independent of the sample. They are 0.48734 and 0.44874 when it is one of the 65. With exact probabilities both equal $6050/11907$.\n\\item Ten units form a cycle, two units are treated, $y_i(d_{00})=1$ and the other potential outcomes are 0, and the probabilities are simulated from 20 of the 45 assignments. Over all such samples, the expected variance estimate is $-0.0711$ against a variance of 0.0370 when the realized assignment is independent of the sample, and 0.0173 against 0.0334 when it is one of the 20. With exact probabilities the expected variance estimate is $32/245$ and the variance is $3/98$.\n\\item Forty units form eight disjoint cliques of five, four units are treated, all potential outcomes equal 1, and the probabilities are simulated from one recorded sample of $R=10{,}000$ of the 91,390 assignments. Given this sample, the expected variance estimate is 0.12980 against a variance of 0.13262 when the realized assignment is independent of the sample, and 0.13013 against 0.13036 when it is one of the 10,000.\n\\end{enumerate}\n\n\nFor a second probability table for the same forty-unit design, using Python seed 20261019 for its 10,000 distinct assignments, I obtain an expected variance estimate of $0.130527822999$ against variance $0.132664519808$ for an independent realized assignment, and $0.130907208515$ against $0.131130209715$ when the realized assignment is included. Every positive relevant joint has a positive empirical count, with minimum 845; exact probabilities give equality at $7430620902683355/55988922226386688\\approx0.132715912491$. These moments condition on this probability table rather than averaging over all 10,000-assignment samples. Both tables concern this forty-unit design rather than the chapter's published networks, and neither calculation establishes a numerical error in a published table entry.\n\n\\emph{Finite simulation and accuracy.} The literal bootstrap analogy concerns a computational approximation. For fixed observed data $(0,1)$, the ideal variance of a two-observation bootstrap mean is $1/8$. With $B\\ge2$ independent bootstrap means, their sample variance with denominator $B-1$ has expectation $1/8$, but is zero with probability $2(1/4)^B+(1/2)^B>0$; denominator $B$ gives expectation $(B-1)/(8B)$. This illustrates finite computational error without equating its bias mechanism to the nonlinear Horvitz--Thompson plug-in or establishing bootstrap validity for an original sampling distribution.\n\nThe package samples distinct assignments by default. If $M$ assignments are possible and an exposure event has exact probability $q$, its count $C$ in $R$ uniform draws without replacement is hypergeometric, with\n\\[\n\\mathbb{E}[C/R]=q,\\qquad\n\\operatorname{Var}(C/R)=\\frac{q(1-q)}R\\frac{M-R}{M-1}.\n\\]\nThus its relative standard deviation depends on $Rq$, not on $R$ alone. At $R=10{,}000$ and negligible sampling fraction, the illustrative probabilities $.01$, $.001$ and $.0001$ give relative standard deviations about $.0995$, $.3161$ and $.9999$; these are not measurements of the chapter's graph. The routine also uses $(C_i+1)/(R+1)$ for marginals, $C_{ij}/(R+1)$ for same-exposure off-diagonal joints and $C_{ij}/R$ for cross-exposure joints. Smoothing and these distinct normalizers must be included in an accuracy argument, along with support and zero-count handling. A bound holding with high probability does not alone prove conservative expectation without control of the failure event and any needed margin or inflation.\n\nFor independent with-replacement simulation from a fixed finite design, fixed finite outcomes and mapping, positive contrast marginals and finitely many relevant positive joints, the strong law and vanishing smoothing give almost-sure convergence of the coefficient formulas and their fixed-design conditional moments to their exact-probability counterparts once support is correctly distinguished. This is no finite-$R$ inequality, no uniform guarantee over changing designs or shrinking positive joints, no growing-$N$ result, and no coverage guarantee. For the default without-replacement scheme, $R\\le M$ at fixed design; even a full census retains the routine's $R+1$ smoothing. Analytic probabilities or unsmoothed census frequencies are exact alternatives. Sufficiently accurate simulation with justified error accounting can also be adequate.\n\n\\emph{Derivation for exact probabilities.} Let $\\pi_i(d)=\\Pr(D_i=d)$ and $\\pi_{ij}(d,d')=\\Pr(D_i=d,\\,D_j=d')$, with $\\pi_{ii}(d,d)=\\pi_i(d)$, and write $I_i(d)=\\mathbf{1}\\{D_i=d\\}$ and $Y_i=y_i(D_i)$. The Horvitz--Thompson estimator of the total under $d$ is $\\widehat T_d=\\sum_i I_i(d)\\,Y_i/\\pi_i(d)$, and $\\widehat\\tau_{HT}(d,d')=(\\widehat T_d-\\widehat T_{d'})/N$. The variance estimator is $\\widehat V_{HT}(d,d')=N^{-2}\\bigl(\\widehat W_d+\\widehat W_{d'}-2\\,\\widehat C_{dd'}\\bigr)$, where\n\\[\n\\begin{aligned}\n\\widehat W_d={}&\\sum_{i,j:\\,\\pi_{ij}(d,d)>0}I_i(d)\\,I_j(d)\\,\\frac{\\pi_{ij}(d,d)-\\pi_i(d)\\pi_j(d)}{\\pi_{ij}(d,d)}\\,\\frac{Y_iY_j}{\\pi_i(d)\\pi_j(d)}\\\\\n&+\\sum_{i\\neq j:\\,\\pi_{ij}(d,d)=0}\\Bigl(\\frac{I_i(d)\\,Y_i^2}{2\\pi_i(d)}+\\frac{I_j(d)\\,Y_j^2}{2\\pi_j(d)}\\Bigr),\\\\\n\\widehat C_{dd'}={}&\\sum_{i,j:\\,\\pi_{ij}(d,d')>0}I_i(d)\\,I_j(d')\\,\\frac{\\pi_{ij}(d,d')-\\pi_i(d)\\pi_j(d')}{\\pi_{ij}(d,d')}\\,\\frac{Y_iY_j}{\\pi_i(d)\\pi_j(d')}\\\\\n&-\\sum_{i,j:\\,\\pi_{ij}(d,d')=0}\\Bigl(\\frac{I_i(d)\\,Y_i^2}{2\\pi_i(d)}+\\frac{I_j(d')\\,Y_j^2}{2\\pi_j(d')}\\Bigr).\n\\end{aligned}\n\\]\nEach term of the two sums over positive joint probabilities has expectation equal to the corresponding term of $\\operatorname{Var}(\\widehat T_d)$ or of $\\operatorname{Cov}(\\widehat T_d,\\widehat T_{d'})$. For a pair with $\\pi_{ij}(d,d)=0$, the term of the variance is $-y_i(d)\\,y_j(d)$, and the second sum of $\\widehat W_d$ has expectation $\\bigl(y_i(d)^2+y_j(d)^2\\bigr)/2$ for that pair. For a pair with $\\pi_{ij}(d,d')=0$, the term of the covariance is $-y_i(d)\\,y_j(d')$, and the second sum of $\\widehat C_{dd'}$ has expectation $-\\bigl(y_i(d)^2+y_j(d')^2\\bigr)/2$. Collecting terms gives\n\\[\n\\begin{aligned}\n\\mathbb{E}\\bigl[\\widehat V_{HT}(d,d')\\bigr]-\\operatorname{Var}\\bigl(\\widehat\\tau_{HT}(d,d')\\bigr)={}&\\frac{1}{2N^2}\\sum_{i\\neq j:\\,\\pi_{ij}(d,d)=0}\\bigl(y_i(d)+y_j(d)\\bigr)^2\\\\\n&+\\frac{1}{2N^2}\\sum_{i\\neq j:\\,\\pi_{ij}(d',d')=0}\\bigl(y_i(d')+y_j(d')\\bigr)^2\\\\\n&+\\frac{1}{N^2}\\sum_{i,j:\\,\\pi_{ij}(d,d')=0}\\bigl(y_i(d)-y_j(d')\\bigr)^2\\;\\ge\\;0.\n\\end{aligned}\n\\]\nThe first step of this computation uses that the probabilities inside $\\widehat V_{HT}$ are those of the distribution of the realized assignment. With simulated probabilities in their place the identity fails, as the counterexamples show.", "problem": "Recall that a variance estimator is conservative if its expectation over the randomization is at least the variance of the effect estimator. The Horvitz--Thompson variance estimator has this property when it is computed with the exact exposure probabilities of the design and every unit has positive probability of both contrasted exposures. The chapter computes it with exposure probabilities simulated from $R=10{,}000$ assignments, and the property does not carry over. The estimator divides by the simulated probability that two units are jointly in given exposures, and for rare joint exposures this frequency rests on few assignments. In the regenerated network, among pairs of units that are both in isolated direct exposure, the smallest positive count in the 10,000 simulated assignments is 1.\n\nFinite simulation introduces numerical error here, just as a finite number of bootstrap resamples approximates the ideal bootstrap calculation. This familiar issue can be addressed by controlling simulation error; a draw count alone does not show that the resulting Horvitz--Thompson variance estimates remain conservative.\n\nThe worked example uses an assignment from the probability simulation. Both calls to \\texttt{make\\_tr\\_vec\\_permutation} set the seed 4224, which implies that the realized assignment of p.~217 is the first of the 10,000 simulated assignments of p.~214. The text does not say so; it says ``We also need to assign treatment''. In an experiment analyzed by this recipe, the realized assignment is ordinarily drawn independently of the simulated assignments. The two cases differ in the small-world network regenerated from the printed code. When the realized assignment is uniform among the simulated assignments, the expectation of the Horvitz--Thompson variance estimator is 1.673, 1.687 and 0.996 times the variance of the Horvitz--Thompson estimator for the contrasts $(d_{11},d_{00})$, $(d_{10},d_{00})$ and $(d_{01},d_{00})$. When the realized assignment is independent of them, the expectation is 0.617, 0.035 and 0.995 times that variance, and the observed shares of negative variance estimates are 9\\%, 40\\% and 0\\%. Inclusion raises the first two ratios above 1 but leaves the third below 1. Applied to an experiment, the recipe can return a Horvitz--Thompson variance estimate that is a small fraction of the variance of the estimator, or negative. Exact counterexamples show that neither a large $R$ nor a realized assignment among the simulated ones guarantees that the estimator is conservative.", "what_survives": "The exposure mapping, the estimands and the Horvitz--Thompson variance estimator computed with exact exposure probabilities under positive marginal support are unaffected. No entry of Table 16.1 is shown to be wrong. Its Horvitz--Thompson standard errors and $t$-statistics were computed with the realized assignment among the simulated assignments, and they describe that case. Figure 16.3 reports the bias and the RMSE of the point estimators and does not depend on the variance estimators. In the regenerated small-world design, the Hajek variance ratios are approximately between 1.01 and 1.82 in the first case and are estimated to lie approximately between 1.02 and 1.82 in the independent case. These results concern that design and give no general guarantee. What is lost is the assurance that the Horvitz--Thompson variance estimator computed with simulated exposure probabilities is conservative."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N005", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of page 220 from the running head, which reads 220 Handbook of computational social science, to the caption of Figure 16.3, which reads Estimator performance across networks. Two stacked panels share a horizontal axis named estimand, with categories dir\\_ind, ind and isol\\_dir, and a vertical axis named value. A red box marks the strip label of the upper panel, which reads bias (\\%). The ticks of that panel are 0.10, 0.05, 0.00, -0.05, -0.10 and -0.15. Red points are the model dcBM and turquoise points the model small world; circles are the estimator Hajek and triangles the estimator HT. In the upper panel the red circles lie near 0.04, 0.11 and -0.15, the red triangles near -0.01, -0.02 and -0.03, and the turquoise points near zero. The lower panel has the strip label rmse and ticks from 0 to 600. In it each circle lies below the triangle of the same colour, and the red points lie above the turquoise points.", "caption": "Figure 16.3, p. 220 (PDF p. 12). The marked strip label of the upper panel reads “bias (%)”. Its labelled ticks, from $-0.15$ to $0.10$, give proportions of the true effect.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N009.png"}, {"alt": "Dot plot titled: The Hajek biases plotted in Figure 16.3 agree with the recalculation as proportions of the effect, not as percentages (degree-corrected block model). The horizontal axis is the bias of the Hajek estimator as a percentage of the true effect, with ticks from -20 to 20 and a vertical line at zero. There are three rows. Row dir\\_ind, tau(d11, d00): a blue circle at 0.04 percent, which is Figure 16.3 as labelled; an orange square at 4.2 percent, which is the plotted value as a proportion, times 100; a grey diamond at 4.1 percent, which is the recalculation, with a bar from 2.8 to 5.5. Row ind, tau(d01, d00): blue circle at 0.11 percent, orange square at 10.6 percent, grey diamond at 12.5 percent with a bar from 9.5 to 15.5. Row isol\\_dir, tau(d10, d00): blue circle at -0.15 percent, orange square at -14.8 percent, grey diamond at -13.7 percent with a bar from -18.7 to -8.7. Each orange square lies within the horizontal span of the bar in its row. Each blue circle lies near the zero line, far outside that span.", "caption": "The three Hajek biases plotted for the degree-corrected block model in Figure 16.3, read as labelled (in percent) and as proportions (times 100), beside the relative biases recalculated from 100,000 assignments on the regenerated network. Each bar extends two estimated Monte Carlo standard errors for 5000 assignments on either side. The proportion interpretation agrees with the recalculation within Monte Carlo variability; the percentage interpretation does not.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N009-bias-scale.png"}], "formal_release": null, "formal_statements": [], "headline": "The upper panel of Figure 16.3 plots the bias as a proportion of the true effect, not in percent as its label “bias (%)” says: the Hajek biases plotted for the degree-corrected block model, near 0.042, 0.106 and $-0.148$, are about 4%, 11% and $-15\\%$ of the effect, not 0.04%, 0.11% and $-0.15\\%$.", "id": "P044-N009", "location": "Figure 16.3, p. 220 (PDF p. 12), upper panel, strip label “bias (%)”", "pdf_page": 12, "printed_page": "220", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Figure 16.3, upper panel: strip label ``bias \\hl{(\\%)}'', with ticks labelled 0.10, 0.05, 0.00, $-0.05$, $-0.10$, $-0.15$. The points for the Hajek estimator on the degree-corrected block model (red circles) lie at about $0.042$ for dir\\_ind, $0.106$ for ind and $-0.148$ for isol\\_dir.", "corrected": "Figure 16.3, upper panel: strip label ``\\hl{relative} bias \\hl{(proportion)}'', with the ticks 0.10, 0.05, 0.00, $-0.05$, $-0.10$, $-0.15$ unchanged. Here, for each network, each estimator $\\widehat\\tau$ and each estimand $\\tau(d,d')$, I estimate relative bias by\n\\[\n\\hl{\\text{estimated}}\\ \\text{relative bias}=\\frac{\\overline{\\tau}_R(d,d')-\\tau(d,d')}{\\tau(d,d')},\\qquad \\overline{\\tau}_R(d,d')=\\frac{1}{R}\\sum_{r=1}^{R}\\widehat\\tau_r(d,d'),\n\\]\nwhere $\\widehat\\tau_r(d,d')$ is the estimate under the $r$th of the $R=5000$ reassignments of treatment, and $\\tau(d,d')$ is the true average effect computed from the simulated potential outcomes.\n\nAlternatively, the label ``bias (\\%)'' is kept and the plotted values and ticks are multiplied by 100, so that the ticks read $\\hl{10}$, $\\hl{5}$, $0$, $\\hl{-5}$, $\\hl{-10}$, $\\hl{-15}$.", "evidence": "The evidence is a recalculation. Running the printed code and seed for the degree-corrected block model (p.~215) with the generator of \\texttt{randnet} 0.5, I obtain a network. With the potential outcomes, the exposure probabilities and the assignment generated as in the printed code, this network reproduces the six estimates and the six variance estimates that are printed for this model on p.~218, to within $5\\times 10^{-5}$ and $4\\times 10^{-3}$. This agreement ties the recalculation to the chapter's printed example. On this network the true effects are $\\tau(d_{11},d_{00})=623.80$, $\\tau(d_{01},d_{00})=155.95$ and $\\tau(d_{10},d_{00})=311.90$.\n\nI then kept the exposure probabilities fixed, drew $R=100{,}000$ fresh assignments by complete randomization, and computed for each estimand the relative bias defined in the corrected passage below. For the Hajek estimator, I find the following values.\n\\begin{audittable}{lrrrr}{Hajek estimator on the degree-corrected block model: the value plotted in Figure 16.3, and the relative bias recalculated from 100,000 assignments, with its Monte Carlo standard error in parentheses.}\nEstimand & True effect & Plotted value & Recalculated relative bias & In percent \\\\\ndir\\_ind, $\\tau(d_{11},d_{00})$ & 623.80 & $0.042$ & $0.0414$ $(0.0015)$ & 4.14\\% \\\\\nind, $\\tau(d_{01},d_{00})$ & 155.95 & $0.106$ & $0.1249$ $(0.0034)$ & 12.49\\% \\\\\nisol\\_dir, $\\tau(d_{10},d_{00})$ & 311.90 & $-0.148$ & $-0.1367$ $(0.0056)$ & $-13.67$\\% \\\\\n\\end{audittable}\nFor 5000 assignments, the estimated Monte Carlo standard errors are larger than those of the table by the factor $\\sqrt{20}$, namely 0.0068, 0.0152 and 0.0250. Read as proportions, the plotted values lie 0.2, 1.3 and 0.4 of these standard errors from the recalculated ones. Read as percentages, they lie 6.0, 8.2 and 5.4 standard errors away.", "problem": "The strip label of the upper panel reads ``bias (\\%)'', and the labelled ticks run from $-0.15$ to $0.10$. Read as labelled, the largest absolute bias in the panel is about 0.15\\% of the effect. The plotted values are proportions. They agree, within Monte Carlo variability, with a recalculation of the mean estimate minus the true effect, divided by the true effect, with no factor of 100. This implies that the label understates every bias in the panel by a factor of 100. On the degree-corrected block model the plotted Hajek biases are about $+4\\%$, $+11\\%$ and $-15\\%$ of the effect for the three estimands, and not a few hundredths of one percent. The panel is the only evidence in the chapter on the size of this bias, and nothing on the page reveals the scale, since the chapter gives no formula for the plotted quantity (see \\fid{P044-N010}).\n\nI infer the scale from a recalculation. The original plotting code and assignments were not found, so I do not recover the exact original biases.", "what_survives": "The lower panel is unaffected. The RMSE is in outcome units, and the Hajek estimator has the lower RMSE in all six cells. In the recalculation on the degree-corrected block model, I find an RMSE of 303, 168 and 554 for the Hajek estimator, against 396, 269 and 771 for the Horvitz--Thompson estimator, in the order of the figure. The estimands, the estimators and Table 16.1 do not depend on the label. The statements that the bias of the Hajek estimator is ``typically small and worth the variance reduction'' (p.~216), and that the variance reduction ``overwhelms the increase in bias'' (p.~219), are qualitative and stand at either scale. At the corrected scale, the three plotted absolute Hajek biases are about 4\\% to 15\\% of the effect.\n\nI checked the scale only on the three Hajek points for the degree-corrected block model, and I did not recalculate the small-world example. The Horvitz--Thompson points share the axis and are on the same scale. I do not read them as Monte Carlo error alone. With exposure probabilities estimated from 10,000 assignments, the expected relative bias of the Horvitz--Thompson estimator on the regenerated network, computed exactly, is 0.005, 0.009 and 0.015 for the three estimands in the order of the figure."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N009", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of eight lines of text on page 219. Red boxes mark the words replicates the experiment 5000 times by reassigning treatment each time, and the word bias, in the sentences: To talk more about the performance of the estimators across these two networks, we now present a simulation that replicates the experiment 5000 times by reassigning treatment each time. Our evaluation metrics are bias and root mean squared error (RMSE), which is increased by both bias and variance of the estimated exposure-specific effects. The next paragraph begins: Figure 16.3 presents the results of the simulation.", "caption": "p. 219 (PDF p. 11). The marked sentence is all that the chapter says about how the simulation of Figure 16.3 was run, and the marked word “bias” names the quantity that the figure plots without a formula.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N010.png"}], "formal_release": null, "formal_statements": [], "headline": "The chapter gives neither the random seed of the 5000 reassignments nor a formula for the quantity plotted as “bias (%)”, and no code for the simulation or for Figure 16.3 was found, so the plotted points cannot be recomputed exactly.", "id": "P044-N010", "location": "Description of the simulation, p. 219 (PDF p. 11), paragraph beginning “To talk more about the performance”; Figure 16.3, p. 220 (PDF p. 12)", "pdf_page": 11, "printed_page": "219", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``To talk more about the performance of the estimators across these two networks, we now present a simulation that \\hl{replicates the experiment 5000 times by reassigning treatment each time}. Our evaluation metrics are \\hl{bias} and root mean squared error (RMSE), which is increased by both bias and variance of the estimated exposure-specific effects.''\n\nFigure 16.3 (p.~220) shows the results in two panels, labelled ``bias (\\%)'' and ``rmse''.", "corrected": "No number changes. The description on p.~219 should state \\hl{the random seed of the 5000 reassignments}, \\hl{whether the exposure probabilities of p.~214 are used in every repetition}, and \\hl{the formula for the plotted bias}; the formula that the plotted values support is given in \\fid{P044-N009}. The script that ran the simulation and drew the figure should accompany the chapter, with its software versions or saved simulation inputs and outputs. I do not supply the seed or the script here, since neither was found.", "evidence": "The evidence is a check of the pages and of the source of the \\texttt{interference} package, which the chapter cites. The package holds no script for this simulation or for the figure; its one replication script reproduces a table of Aronow and Samii (2017).", "problem": "The chapter prints the code and the seeds for the networks, the potential outcomes, the exposure probabilities and the assignment of Table 16.1. For the simulation it states only the number of repetitions. It gives no seed for the 5000 reassignments, it does not say whether the exposure probabilities estimated from the 10,000 assignments of p.~214 are used in every repetition, and it gives no formula for the plotted bias. No plotted value is shown to be wrong.", "what_survives": "The stated number of repetitions and use of treatment reassignment are unaffected, and the experiment can be run again with new random draws. In the figure the Hajek estimator has the lower RMSE in all six cells, and the RMSE is higher for the degree-corrected block model in every comparison. The scale of the upper panel is the subject of \\fid{P044-N009}."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N010", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of seven lines of the reference list on page 222. A red box marks the word in, in the entry: Imbens, G. W., and Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge University Press. The entries for Hudgens and Halloran (2008) and for Karrer and Newman (2011) are above and below it.", "caption": "References, p. 222 (PDF p. 14). The marked word “in” in the title of Imbens and Rubin (2015) should be “for”.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N012.png"}], "formal_release": null, "formal_statements": [], "headline": "The title of Imbens and Rubin (2015) is Causal inference for statistics, social, and biomedical sciences; the reference list prints “in” for “for”.", "id": "P044-N012", "location": "References, p. 222 (PDF p. 14), entry for Imbens and Rubin (2015)", "pdf_page": 14, "printed_page": "222", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "``Imbens, G. W., \\& Rubin, D. B. (2015). \\emph{Causal inference \\hl{in} statistics, social, and biomedical sciences}. Cambridge University Press.''", "corrected": "``Imbens, G. W., \\& Rubin, D. B. (2015). \\emph{Causal inference \\hl{for} statistics, social, and biomedical sciences}. Cambridge University Press.''", "evidence": "The evidence is a check of the page against the catalogue entry of Cambridge University Press for the 2015 edition (ISBN 9780521885881).", "problem": "The preposition in the title is wrong. The publisher lists the book as \\emph{Causal Inference for Statistics, Social, and Biomedical Sciences}, with the subtitle \\emph{An Introduction}.", "what_survives": "The entry still identifies the book by its authors, year and publisher, and nothing else in the chapter depends on it. I do not count the omission of the subtitle as an error in an abbreviated reference."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N012", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the Bernoulli-design definition and surrounding text. Red boxes mark the words 'Under the Bernoulli assignment experimental design, treatments are assigned through independent Bernoulli draws (e.g., independent coin flips),' which run over two lines.", "caption": "Section “Network experimental setting”, p. 210 (PDF p. 2). The marked words define the Bernoulli design, which the package’s sampler is meant to simulate; the definition itself is correct.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N014.png"}, {"alt": "Listing of lines 163 to 189 in monospaced type with line numbers: the function make_tr_vec_bernoulli_indiv. A red box marks lines 181 to 185, a while loop whose condition is any(duplicated(rbind(vec, tr_vec_sampled[1:i - 1,]))) and whose body assigns a new draw rbinom(N, 1, p) to vec. Above it, lines 169 to 177 stop the function if R exceeds 2 to the power N.", "caption": "Package interference, file R/make_tr_vec_bernoulli.R, lines 163–189, typeset from the source file with line numbers added. The marked loop draws a new profile whenever the proposed one equals an earlier row.", "url": "https://pmaronow.github.io/llm-audit/works/P044/exhibits/P044-N014-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The rows returned by the Bernoulli samplers of the interference package are not independent draws from the Bernoulli design, because the samplers discard every proposed profile that equals an earlier row; to simulate that design, repeated profiles must be kept.", "id": "P044-N014", "location": "Section “Network experimental setting”, p. 210 (PDF p. 2), definition of the Bernoulli design; Section “Design-based inference”, p. 212 (PDF p. 4), recipe for exposure probabilities; package interference, file R/make_tr_vec_bernoulli.R, lines 179–187 and 117–152", "pdf_page": 2, "printed_page": "210", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P013-N012", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "where": null, "work_id": "P045"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P013-N012", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-N012", "work_id": "P013"}], "severity": "Moderate", "subject": "paper", "target_ids": [], "text": {"as_printed": "Chapter, p.~210: ``Under the \\emph{Bernoulli assignment} experimental design, treatments are assigned through independent Bernoulli draws (e.g., independent coin flips), in which case $p_{\\mathbf{z}}>0$ for any of the $2^N$ possible profiles, although $p_{\\mathbf{z}}$ would be vanishingly small for some profiles (such as the profile in which $\\mathbf{z}$ is a vector of only 0's).''\n\nChapter, p.~212: ``Specifically, one can simulate treatment assignments based on the experimental design, feed them through the exposure mapping for each unit, and then determine each unit's probabilities of falling into any of the exposure conditions.''\n\nPackage \\texttt{interference}, file \\texttt{R/make\\_tr\\_vec\\_bernoulli.R}, documentation, lines 10--11 and 13: ``Sampling of treatment vectors is without replacement.'' ``\\texttt{@param p} probability that treatment takes the value 1.''\n\nThe same file, lines 180--186, inside the loop \\texttt{for (i in 1:R)} (indentation and one blank line removed):\n\n\\texttt{vec <- rbinom(N, 1, p)}\n\n\\texttt{\\hl{while (any(duplicated(rbind(vec, tr\\_vec\\_sampled[1:i - 1,]))))}}\n\n\\texttt{\\{}\n\n\\texttt{vec <- rbinom(N, 1, p)}\n\n\\texttt{\\}}\n\n\\texttt{tr\\_vec\\_sampled[i,] <- vec}", "corrected": "In the unit-level sampler I delete the \\texttt{while} loop of lines 181--185, and each row is \\hl{one fresh draw}:\n\n\\texttt{for (i in 1:R) \\{}\n\n\\texttt{vec <- rbinom(N, 1, p)}\n\n\\texttt{tr\\_vec\\_sampled[i,] <- vec}\n\n\\texttt{\\}}\n\nI delete the loop of lines 117--152 in the cluster sampler in the same way, so that each row is one fresh clustering with one fresh draw for each cluster. The tests that stop the function when \\texttt{R} exceeds the number of possible assignments are then unnecessary and are also deleted, along with the corresponding restriction in the documentation. The documentation reads ``Sampling of treatment vectors is \\hl{with} replacement.''\n\nWith this change the rows are independent draws from the design, and the frequency of an exposure condition among them is an unbiased estimate of its probability $\\pi$. The adjustment in \\texttt{make\\_exposure\\_prob} is a separate matter and remains: the reported value $(C+1)/(R+1)$ has expectation $(R\\pi+1)/(R+1)$. If instead every profile in the support is enumerated, each profile must enter with its probability $q(\\mathbf{z})$; in the two-unit example this weighting returns $1/25$, $4/25$, $4/25$ and $16/25$.", "evidence": "I checked the counterexample in exact rational arithmetic and by running the unmodified package functions. Since each proposal has the law $q$, for $\\mathbf{z}\\notin S$ and $q(S)<1$ the probability that $\\mathbf{z}$ is the next row to be accepted is\n\\[\n\\sum_{m=0}^{\\infty}q(S)^m q(\\mathbf{z})=\\frac{q(\\mathbf{z})}{1-q(S)} .\n\\]\n\nLet the network consist of two units joined by an edge, so that neither is an isolate, and let $p=1/5$. The profiles $(0,0)$, $(0,1)$, $(1,0)$ and $(1,1)$ have probabilities $16/25$, $4/25$, $4/25$ and $1/25$, and they place unit 1 in the conditions \\texttt{no}, \\texttt{ind1}, \\texttt{isol\\_dir} and \\texttt{dir\\_ind1}, respectively. With $R=4$, the largest value that the function accepts when $N=2$, each profile is returned exactly once. The routine \\texttt{make\\_exposure\\_prob} reports the marginal probability of a condition as $(C+1)/(R+1)$, where $C$ is the number of rows that place the unit in the condition.\n\n\\begin{audittable}{lrrr}{Unit 1 in the two-unit network with $p=1/5$ and $R=4$: probability of each exposure condition under the Bernoulli design, frequency among the four returned rows, and value reported by \\texttt{make\\_exposure\\_prob}.}\nCondition & Design & Frequency in rows & Reported \\\\\n\\texttt{dir\\_ind1} & $1/25$ & $1/4$ & $2/5$ \\\\\n\\texttt{isol\\_dir} & $4/25$ & $1/4$ & $2/5$ \\\\\n\\texttt{ind1} & $4/25$ & $1/4$ & $2/5$ \\\\\n\\texttt{no} & $16/25$ & $1/4$ & $2/5$ \\\\\n\\end{audittable}\n\nFor two different conditions, the routine reports the plain frequency. The reported probability that unit 1 is in \\texttt{isol\\_dir} and unit 2 is in \\texttt{ind1} equals $1/4$, where the design has $4/25$. Since no adjustment enters this entry, the difference comes from the discarded repeats alone.\n\nWith $R=2$ on the same network, there are fewer rows than profiles. In the first row, unit 1 is treated with probability $1/5$. Conditioning on the first row, which is $(0,0)$, $(0,1)$, $(1,0)$ or $(1,1)$, I obtain for the second row\n\\[\n\\Pr(Z_{2,1}=1)=\\frac{16}{25}\\cdot\\frac{5/25}{9/25}+\\frac{4}{25}\\cdot\\frac{5/25}{21/25}+\\frac{4}{25}\\cdot\\frac{1/25}{21/25}+\\frac{1}{25}\\cdot\\frac{4/25}{24/25}=\\frac{257}{630} .\n\\]\nFor the expected frequency of treatment of unit 1 over the two rows, I find $(1/5+257/630)/2=383/1260$, where the design has $1/5$. The cluster sampler puts both units of this network in one cluster. With $R=2$, it must return $(0,0)$ and $(1,1)$ once each. The condition \\texttt{dir\\_ind1} then has frequency $1/2$, where the design has $1/5$. With a single unit, $p=1/5$ and $R=2$, the function returns one $0$ and one $1$, and the unit is treated in the second row with probability $4/5$.", "problem": "The chapter defines the Bernoulli design and gives a recipe for exposure probabilities: simulate assignments from the design, and count how often each unit falls in each exposure condition. Independent draws suffice, but dependent rows with the right marginal law or suitable sampling weights can also yield unbiased estimates. The function \\texttt{make\\_tr\\_vec\\_bernoulli} of the package does not generally preserve that marginal law. A proposed profile that equals an earlier row is discarded and drawn again, both in the unit-level sampler (lines 181--185) and in the cluster sampler (lines 117--152).\n\nLet $q(\\mathbf{z})=p^{\\sum_i z_i}(1-p)^{N-\\sum_i z_i}$ be the probability of the profile $\\mathbf{z}$ under the unit-level design, and let $S$ be the set of rows already accepted, with $q(S)=\\sum_{\\mathbf{s}\\in S}q(\\mathbf{s})$. While $q(S)<1$, the next row to be accepted equals a profile $\\mathbf{z}\\notin S$ with probability $q(\\mathbf{z})/(1-q(S))$, and it never equals a member of $S$. The first row has the law $q$; later rows are conditioned to exclude earlier profiles. The routine \\texttt{make\\_exposure\\_prob} weights the returned rows equally. This can overrepresent low-probability profiles and distort exposure probabilities. Inverse probability weights and variance estimates that use the distorted probabilities can be affected, and the package gives no warning. The documentation says correctly that sampling is without replacement; it does not explain that the returned rows need not retain the design's marginal law with parameter \\texttt{p}.\n\nThe distortion is not confined to small examples. Among distinct profiles, at most one has no treated unit and at most $N$ have exactly one treated unit. With $N=100$, $p=0.01$ and $R=1000$, at least $899$ of the returned rows have two or more treated units, and the rows contain at least $100+2\\cdot 899=1898$ treated entries among $100{,}000$. The treatment frequency is at least $0.01898$, where the design has $0.01$. When repeated profiles are unlikely over the $R$ draws, the loop usually has no effect.", "what_survives": "This defect does not alter the chapter's definition of the Bernoulli design, its exposure mappings or its design-based identities when the exposure probabilities are those of the design. The worked example summarized in Table~16.1 uses the function \\texttt{make\\_tr\\_vec\\_permutation}, which carries out complete random assignment, and the simulation in Figure~16.3 continues that design. Under a design that is uniform on its support, each row remains marginally uniform when repeats are discarded, so this point does not implicate those results. A single row ($R=1$) from either Bernoulli sampler is unaffected by duplicate rejection. What can change is the calculation of exposure probabilities from several rows of \\texttt{make\\_tr\\_vec\\_bernoulli} under a nonuniform design, with the weights and variance estimates that use them."}, "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N014", "work": {"arxiv_id": null, "doi": "10.4337/9781802207309.00025", "title": "Estimating Causal Effects in Experiments with Network Spillovers", "url": "https://pmaronow.github.io/llm-audit/works/P044/"}, "work_id": "P044"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of seven lines from the left column of page 299. The text reads 'Confidence intervals are based on a large-N normal approximation:' and is followed by the interval tau-hat HT of d_k, d_l, plus or minus z with subscript (1 minus alpha) over 2, times the square root of the estimated variance of tau-hat HT. A red box marks z with its subscript (1 minus alpha)/2.", "caption": "Section 16.4.2, p. 299 (PDF p. 11), left column. The marked term is the critical value $z_{(1-\\alpha)/2}$ in the confidence interval.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-A001.png"}], "formal_release": null, "formal_statements": [], "headline": "For nominal coverage $c$, the confidence interval on p. 299 uses the $(1+c)/2$ quantile of the standard normal distribution; the printed subscript $(1-\\alpha)/2$ comes with no definition of $\\alpha$ or of $z$.", "id": "P045-A001", "location": "Section 16.4.2, p. 299 (PDF p. 11), left column, the confidence interval after “Confidence intervals are based on a large-$N$ normal approximation”", "pdf_page": 11, "printed_page": "299", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Confidence intervals are based on a large-$N$ normal approximation:''\n\\[\n\\widehat{\\tau_{HT}}(d_k,d_l)\\pm \\hl{z_{(1-\\alpha)/2}}\\sqrt{\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau_{HT}}(d_k,d_l)]}.\n\\]", "corrected": "``Confidence intervals are based on a large-$N$ normal approximation: \\hl{for nominal coverage $c$,}''\n\\[\n\\widehat{\\tau_{HT}}(d_k,d_l)\\pm \\hl{z_{(1+c)/2}}\\sqrt{\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau_{HT}}(d_k,d_l)]},\n\\]\n\\hl{where $z_q=\\Phi^{-1}(q)$ is the $q$-quantile of the standard normal distribution}. Equivalently, with error probability $\\alpha=1-c$, the critical value is $z_{1-\\alpha/2}$.", "intended_meaning": "The intended interval is the usual two-sided normal interval. The passage defines neither $\\alpha$ nor $z$, and the printed subscript is correct under one convention and wrong under the usual one. Let $\\alpha$ be the error probability and let $z_q=\\Phi^{-1}(q)$ be the $q$-quantile of the standard normal distribution. Then the interval of level $1-\\alpha$ uses $z_{1-\\alpha/2}$, and for $\\alpha=0.05$ the printed subscript gives $\\Phi^{-1}(0.475)=-0.063$ in place of $\\Phi^{-1}(0.975)=1.960$. Suppose instead that $\\alpha$ is the coverage and that $z_q=\\Phi^{-1}(1-q)$ is the critical value for the upper tail. Then for $\\alpha=0.95$ the printed subscript gives $\\Phi^{-1}(0.975)=1.960$, which is the correct value. A script distributed with the package computes its 95\\% intervals with \\texttt{qnorm(.975)} and its 90\\% intervals with \\texttt{qnorm(.95)}, as the usual interval requires. No result of the chapter depends on the display."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-A001", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of text followed by a displayed formula. A red box marks the word 'unbiased' in the sentence 'Given Bernoulli assignment with probability psi, an unbiased estimator for one over n_g times the sum of y_ig(z; psi), with z = {0, 1}, is'. The display below defines y-hat_g(z; psi) as the sum of Y_ig times the indicator of Z_ig = z, divided by the sum of the indicators.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), left column: the sentence that calls the arm mean $\\widehat{y}_g(z;\\psi)$ unbiased under Bernoulli assignment, with “unbiased” marked, and the display that defines the arm mean.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E001.png"}, {"alt": "Nine lines of text. Red boxes mark the word 'unbiased' in 'Therefore, an unbiased estimator for the average direct effect is tau-hat D(psi) = y-hat(1; psi) minus y-hat(0; psi)' and the word 'Unbiased' that begins the next sentence, 'Unbiased estimators for the other causal estimands of interest are defined analogously'.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), left column, below the second display: the sentences that call the estimators of the direct, indirect, total and overall effects unbiased, with “unbiased” and “Unbiased” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E001-b.png"}, {"alt": "A chart with five rows and a horizontal axis of expected value from minus 1 to 1. Blue vertical ticks mark the targets: 0 for the direct effect at psi = 1/2 and for the direct effect at phi = 1/4, and 3/4 for the indirect, total and overall effects. Orange dots mark the expectation of each estimator as printed: minus 1 for each direct effect, 224/255 for the indirect effect, 37/255 for the total effect, and 3/4 for the overall effect, where the dot lies on the tick and is labelled 'no bias'. In the first four rows a grey line joins the dot to its tick. The legend says that the tick is also the expectation of the Horvitz-Thompson replacement.", "caption": "Targets and expectations in the counterexample of this finding: two groups of four units, outcome equal to the number of treated group members other than the unit, one group at $\\psi=1/2$ and one at $\\phi=1/4$, Bernoulli assignment within groups. The expectations of the printed estimators are conditional on each estimator being defined; all values are exact fractions from enumeration of 512 assignments. The Horvitz–Thompson replacements have unconditional expectations equal to the targets; the overall estimator is unchanged.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E001-bias.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T1", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T1"}, {"id": "P045-T2", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T2"}, {"id": "P045-T25", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T25"}], "headline": "Under Bernoulli assignment within groups, the arm means of Section 16.5.1 are not in general unbiased for the marginal potential outcomes of p. 309, and neither are the estimators of the direct, indirect and total effects built from them; the estimator of the overall effect is unbiased under uniform assignment of fixed positive numbers of groups to saturations.", "id": "P045-E001", "location": "Section 16.5.1, p. 310 (PDF p. 22), left column, the paragraph beginning “Let the observed outcome be” and its two displays", "pdf_page": 22, "printed_page": "310", "reaches": [{"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P045-T1", "P045-T2", "P045-T25"], "text": {"as_printed": "Let the observed outcome be $Y_{ig}=y_{ig}(\\mathbf{Z}_{ig})$ under the group treatment assignment $\\mathbf{Z}_{ig}$ and suppose treatment saturation is $\\psi$. Given Bernoulli assignment with probability $\\psi$, an \\hl{unbiased} estimator for $\\frac{1}{n_g}\\sum_{i=1}^{n_g}y_{ig}(z;\\psi)$, with $z=\\{0,1\\}$, is\n\\[\n\\widehat{y}_g(z;\\psi)=\\frac{\\sum_{i=1}^{n_g}Y_{ig}\\mathbf{I}(Z_{ig}=z)}{\\sum_{i=1}^{n_g}\\mathbf{I}(Z_{ig}=z)},\n\\]\nand for the population average potential outcome, it is\n\\[\n\\widehat{y}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi)\\mathbf{I}(S_g=\\psi)}{\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi)},\n\\]\nwhere $S_g$ is the saturation level of group $g$ (conditional on the denominators of these estimators being nonzero). Therefore, an \\hl{unbiased} estimator for the average direct effect is $\\widehat{\\tau}^{D}(\\psi)=\\widehat{y}(1;\\psi)-\\widehat{y}(0;\\psi)$. \\hl{Unbiased} estimators for the other causal estimands of interest are defined analogously: $\\widehat{\\tau}^{I}(\\psi,\\phi)=\\widehat{y}(0;\\psi)-\\widehat{y}(0;\\phi)$, $\\widehat{\\tau}^{To}(\\psi,\\phi)=\\widehat{y}(1;\\psi)-\\widehat{y}(0;\\phi)$, and $\\widehat{\\tau}^{O}(\\psi,\\phi)=\\widehat{y}(\\psi)-\\widehat{y}(\\phi)$, where $\\widehat{y}(\\psi)$ is the average across groups assigned to saturation $\\psi$ of the average observed outcomes of units in the group under treatment assignment $\\mathbf{Z}_{ig}$.", "corrected": "Let the observed outcome be $Y_{ig}=y_{ig}(\\mathbf{Z}_{ig})$ under the group treatment assignment $\\mathbf{Z}_{ig}$ and suppose treatment saturation is $\\psi$, with $0<\\psi<1$ and, for the indirect and total contrasts, $\\hl{0\\le\\phi<1}$. Given Bernoulli assignment with probability $\\psi$, the arm mean\n\\[\n\\widehat{y}_g(z;\\psi)=\\frac{\\sum_{i=1}^{n_g}Y_{ig}\\mathbf{I}(Z_{ig}=z)}{\\sum_{i=1}^{n_g}\\mathbf{I}(Z_{ig}=z)}\n\\]\nis \\hl{not in general unbiased} for $\\frac{1}{n_g}\\sum_{i=1}^{n_g}y_{ig}(z;\\psi)$, with $z\\in\\{0,1\\}$, \\hl{even conditional on its denominator being nonzero}. An unbiased estimator is\n\\[\n\\widehat{y}^{HT}_g(z;\\psi)=\\frac{\\sum_{i=1}^{n_g}Y_{ig}\\mathbf{I}(Z_{ig}=z)}{\\hl{n_g\\,\\psi^{z}(1-\\psi)^{1-z}}},\n\\]\ncomputed for every assignment, with the value $0$ when no unit of the group has treatment $z$. Suppose that a fixed positive number of the $G$ groups is assigned to each saturation, all such assignments being equally likely. Then\n\\[\n\\widehat{y}^{HT}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}^{HT}_g(z;\\psi)\\mathbf{I}(S_g=\\psi)}{\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi)},\n\\]\nwhere $S_g$ is the saturation level of group $g$, is unbiased for the population average potential outcome. The estimators $\\widehat{\\tau}^{D}(\\psi)=\\widehat{y}^{HT}(1;\\psi)-\\widehat{y}^{HT}(0;\\psi)$, $\\widehat{\\tau}^{I}(\\psi,\\phi)=\\widehat{y}^{HT}(0;\\psi)-\\widehat{y}^{HT}(0;\\phi)$ and $\\widehat{\\tau}^{To}(\\psi,\\phi)=\\widehat{y}^{HT}(1;\\psi)-\\widehat{y}^{HT}(0;\\phi)$ are then unbiased for the average direct, indirect and total effects. The estimator $\\widehat{\\tau}^{O}(\\psi,\\phi)=\\widehat{y}(\\psi)-\\widehat{y}(\\phi)$, where $\\widehat{y}(\\psi)$ is the average across groups assigned to saturation $\\psi$ of the average observed outcomes of units in the group, is unbiased for the average overall effect.\n\nAlternatively, suppose that treatment within each group is by \\hl{complete random assignment}, with a fixed number of treated units and a fixed number of control units, both at least one, and that the marginal potential outcomes $y_{ig}(z;\\psi)$ are defined by averaging over the assignments of that design. Then the arm means $\\widehat{y}_g(z;\\psi)$ and $\\widehat{y}(z;\\psi)$ as printed, and the estimators $\\widehat{\\tau}^{D}$, $\\widehat{\\tau}^{I}$ and $\\widehat{\\tau}^{To}$ built from them, are unbiased for the corresponding quantities under the same assignment of groups to saturations.", "evidence": "An exact counterexample, and derivations for the two corrected forms.\n\n\\emph{The counterexample.} I take two groups of four units and set $y_{ig}(\\mathbf{z})=\\sum_{j\\neq i}z_j$, the number of treated group members other than $i$. Own treatment has no effect, and partial and stratified interference hold. One group, chosen with probability $1/2$, has saturation $\\psi=1/2$ and the other has $\\phi=1/4$. Each unit is then treated independently with the probability of its group. For a unit in a group with treatment probability $p$, each of the three other members is treated with probability $p$ whatever $z$ is, so that $y_{ig}(0;p)=y_{ig}(1;p)=3p$. The group averages of the marginal potential outcomes are $3/2$ at $\\psi$ and $3/4$ at $\\phi$, both direct effects are $0$, and the indirect, total and overall effects are $3/4$.\n\nLet $K$ be the number of treated units in a group. Every treated unit has outcome $K-1$ and every control unit has outcome $K$. The treated arm mean is $K-1$ when $K\\ge 1$, the control arm mean is $K$ when $K\\le 3$, and their difference is $-1$ on every assignment for which both are defined.\n\n\\begin{audittable}{crrcc}{The number $K$ of treated units in a group of four, its probability at each saturation, and the two arm means.}\n$K$ & Probability at $\\psi=1/2$ & Probability at $\\phi=1/4$ & Treated arm mean & Control arm mean \\\\\n0 & 1/16 & 81/256 & undefined & 0 \\\\\n1 & 4/16 & 108/256 & 0 & 1 \\\\\n2 & 6/16 & 54/256 & 1 & 2 \\\\\n3 & 4/16 & 12/256 & 2 & 3 \\\\\n4 & 1/16 & 1/256 & 3 & undefined \\\\\n\\end{audittable}\n\nBy the table, I have\n\\[\n\\begin{aligned}\nE\\bigl[\\widehat{y}_g(1;\\psi)\\mid K\\ge 1\\bigr]&=\\frac{4\\cdot 0+6\\cdot 1+4\\cdot 2+1\\cdot 3}{15}=\\frac{17}{15},\\\\\nE\\bigl[\\widehat{y}_g(0;\\psi)\\mid K\\le 3\\bigr]&=\\frac{1\\cdot 0+4\\cdot 1+6\\cdot 2+4\\cdot 3}{15}=\\frac{28}{15},\n\\end{aligned}\n\\]\nwhile the target of both is $3/2$. At $\\phi$ the two conditional expectations are $81/175$ and $84/85$, and the target is $3/4$. Enumerating the 512 assignments of the two stages in exact fractions, I find the expectation of each estimator over the assignments on which it is defined. It is $-1$ for $\\widehat{\\tau}^{D}(\\psi)$ and for $\\widehat{\\tau}^{D}(\\phi)$, whose targets are $0$. It is $224/255$ for $\\widehat{\\tau}^{I}$ and $37/255$ for $\\widehat{\\tau}^{To}$, whose targets are $3/4$. It is $3/4$ for $\\widehat{\\tau}^{O}$, which equals its target.\n\n\\emph{The Horvitz--Thompson mean.} Since $y_{ig}(z;\\psi)$ is the expected outcome of unit $i$ given $Z_{ig}=z$, I have\n\\[\nE\\bigl[\\mathbf{I}(Z_{ig}=z)\\,Y_{ig}\\bigr]=\\Pr(Z_{ig}=z)\\,E\\bigl[Y_{ig}\\mid Z_{ig}=z\\bigr]=\\psi^{z}(1-\\psi)^{1-z}\\,y_{ig}(z;\\psi).\n\\]\nSumming over $i$ and dividing by $n_g\\,\\psi^{z}(1-\\psi)^{1-z}$, I obtain that the Horvitz--Thompson mean in the correction has expectation $(1/n_g)\\sum_{i}y_{ig}(z;\\psi)$. Note that the expectation is over all assignments, including those with no unit in arm $z$, where the sum is $0$. In the counterexample, when the arm means are replaced by Horvitz--Thompson means, the expectations found by enumeration equal the targets for all five effects.\n\n\\emph{Complete random assignment.} Suppose that exactly $k_z\\ge 1$ units of the group receive treatment $z$, all such assignments being equally likely, and let $y^{\\mathrm{cr}}_{ig}(z)$ be the expected outcome of unit $i$ given $Z_{ig}=z$ under this design. Since each unit has treatment $z$ with probability $k_z/n_g$ and the denominator of the arm mean is the constant $k_z$, I find\n\\[\nE\\bigl[\\widehat{y}_g(z;\\psi)\\bigr]=\\frac{1}{k_z}\\sum_{i=1}^{n_g}\\frac{k_z}{n_g}\\,y^{\\mathrm{cr}}_{ig}(z)=\\frac{1}{n_g}\\sum_{i=1}^{n_g}y^{\\mathrm{cr}}_{ig}(z).\n\\]\nThese targets differ from the Bernoulli ones. In the counterexample with two of the four units treated, I have $y^{\\mathrm{cr}}_{ig}(1)=1$ and $y^{\\mathrm{cr}}_{ig}(0)=2$. The direct effect defined for this design is $-1$, while the direct effect under Bernoulli assignment is $0$.\n\n\\emph{Groups.} Suppose that a fixed number $C\\ge 1$ of the $G$ groups receives saturation $\\psi$, all choices being equally likely. Each group then has saturation $\\psi$ with probability $C/G$. If a quantity computed in group $g$ has expectation $\\mu_g$ when the group has saturation $\\psi$, its mean over the groups with that saturation has expectation $(1/G)\\sum_{g}\\mu_g$. Applied to the Horvitz--Thompson means and to the arm means under complete random assignment, this implies the population statements in the correction. Applied to the mean outcome of all units in a group, it implies that $\\widehat{\\tau}^{O}$ is unbiased.", "problem": "On p.~309 the chapter defines the marginal potential outcome $y_{ig}(z;\\psi)$ by fixing the treatment of unit $i$ at $z$ and averaging its potential outcomes over the treatments of the other group members, each of whom is treated independently with probability $\\psi$. The same page separates this from complete random assignment, which fixes the number of treated units. Under Bernoulli assignment the number of treated units in a group is random, and the arm mean $\\widehat{y}_g(z;\\psi)$, which is the mean of the observed outcomes of the units of group $g$ with treatment $z$, divides by the realized size of the arm. When the outcome of a unit depends on how many other group members are treated, which is the case that Section 16.5 treats, the arm mean then averages the outcomes over a distribution of the other members' treatments that differs from the one in the definition of $y_{ig}(z;\\psi)$. Conditioning on a nonzero denominator does not remove the difference. The step to ``unbiased'' fails for $\\widehat{y}_g(z;\\psi)$ and $\\widehat{y}(z;\\psi)$, and with it the claims for $\\widehat{\\tau}^{D}$, $\\widehat{\\tau}^{I}$ and $\\widehat{\\tau}^{To}$. The estimator $\\widehat{\\tau}^{O}$ uses the mean outcome of all units in a group, has no denominator that depends on the size of an arm, and is unbiased under uniform assignment of fixed positive numbers of groups to saturations.", "what_survives": "The definitions of the marginal causal effects on p.~309 and the estimator of the overall effect are unaffected. The toy example on pp.~310--311 uses complete random assignment in both stages, and Figure 16.6 uses the same two-stage design. For that design the arm means are unbiased for the marginal potential outcomes defined by it. No reported number is known to depend on the claim. What changes is that, under Bernoulli assignment within groups, the printed estimators of the direct, indirect and total effects cannot be called unbiased. No variance estimator for the Horvitz--Thompson means was derived."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E001", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first display in the left column of page 301. The first line defines mu-hat HT,None of 1 as one over N times the sum over i of the indicator of Z_i = 1 times Y_i over pi_i. The second and third lines expand it as one over N times the sum of the indicator of D_i = d11 times y_i(d11) over pi_i(d11), plus the indicator of D_i = d10 times y_i(d10) over pi_i(d10). Below, the text reads 'where pi_i = Pr[Z_i = 1], while the unbiased estimator would be'. Red boxes mark the two denominators pi_i(d11) and pi_i(d10).", "caption": "Section 16.4.3, p. 301 (PDF p. 13), left column, first display. The marked denominators are $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$ in the second equality.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T12", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T12"}, {"id": "P045-T13", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T13"}], "headline": "In the expansion of $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ on p. 301, both denominators are $\\pi_i=\\Pr[Z_i=1]$, not the exposure probabilities $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$.", "id": "P045-E002", "location": "Section 16.4.3, p. 301 (PDF p. 13), left column, first display, second equality, and the sentence that begins “Therefore, the misspecified”", "pdf_page": 13, "printed_page": "301", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T12", "P045-T13"], "text": {"as_printed": "\\[\n\\begin{aligned}\n\\widehat{\\mu_{HT,None}}(\\mathbf{1})&=\\frac{1}{N}\\sum_{i=1}^{N}\\mathbf{I}(Z_i=1)\\frac{Y_i}{\\pi_i}\\\\\n&=\\frac{1}{N}\\sum_{i=1}^{N}\\left(\\mathbf{I}(D_i=d_{11})\\frac{y_i(d_{11})}{\\hl{\\pi_i(d_{11})}}+\\mathbf{I}(D_i=d_{10})\\frac{y_i(d_{10})}{\\hl{\\pi_i(d_{10})}}\\right),\n\\end{aligned}\n\\]\n``where $\\pi_i=\\Pr[Z_i=1]$, while the unbiased estimator would be'' $\\widehat{\\mu_{HT}}(\\mathbf{1})=\\widehat{\\mu_{HT}}(d_{11})$ \\ldots\\ ``Therefore, the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ is biased for \\hl{$\\widehat{\\mu_{HT}}(\\mathbf{1})$} insofar as $y_i(d_{11})\\ne y_i(d_{10})$ for some $i$.''", "corrected": "\\[\n\\begin{aligned}\n\\widehat{\\mu_{HT,None}}(\\mathbf{1})&=\\frac{1}{N}\\sum_{i=1}^{N}\\mathbf{I}(Z_i=1)\\frac{Y_i}{\\pi_i}\\\\\n&=\\frac{1}{N}\\sum_{i=1}^{N}\\left(\\mathbf{I}(D_i=d_{11})\\frac{y_i(d_{11})}{\\hl{\\pi_i}}+\\mathbf{I}(D_i=d_{10})\\frac{y_i(d_{10})}{\\hl{\\pi_i}}\\right),\n\\end{aligned}\n\\]\nwhere $\\pi_i=\\Pr[Z_i=1]$\\hl{${}=\\pi_i(d_{11})+\\pi_i(d_{10})$}. The equality holds for every assignment when the first-degree exposure mapping is correctly specified and $\\pi_i>0$ for every $i$. The closing sentence becomes: Therefore, \\hl{relative to $\\mu(\\mathbf{1})$,} the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ \\hl{has bias}\n\\[\n\\hl{\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\pi_i(d_{10})}{\\pi_i}\\,\\bigl\\{y_i(d_{10})-y_i(d_{11})\\bigr\\}}.\n\\]\nThis formula for the bias holds when, in addition, every unit has at least one direct peer, so that $\\mu(\\mathbf{1})=\\mu(d_{11})$.", "evidence": "The evidence is a derivation and an exact counterexample. For the derivation, suppose that the first-degree exposure mapping of p.~294 is correctly specified and that $\\pi_i$, $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$ are positive for every $i$. By the definition of the exposure conditions, the events $\\{D_i=d_{11}\\}$ and $\\{D_i=d_{10}\\}$ partition the event $\\{Z_i=1\\}$. This implies that $\\pi_i=\\pi_i(d_{11})+\\pi_i(d_{10})$ and that $\\mathbf{I}(Z_i=1)=\\mathbf{I}(D_i=d_{11})+\\mathbf{I}(D_i=d_{10})$. Since $Y_i=y_i(d)$ on the event $\\{D_i=d\\}$, I have\n\\[\n\\mathbf{I}(Z_i=1)\\,Y_i=\\mathbf{I}(D_i=d_{11})\\,y_i(d_{11})+\\mathbf{I}(D_i=d_{10})\\,y_i(d_{10})\n\\]\nfor every assignment. Dividing by $\\pi_i$ and averaging over $i$ gives the expansion in the corrected display below. The potential outcomes are fixed, and the two indicators have expectations $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$. Taking expectations in the first line and in the printed second line, I obtain\n\\[\n\\begin{aligned}\n\\mathbb{E}\\bigl[\\widehat{\\mu_{HT,None}}(\\mathbf{1})\\bigr]&=\\frac{1}{N}\\sum_{i=1}^{N}\\left(\\frac{\\pi_i(d_{11})}{\\pi_i}\\,y_i(d_{11})+\\frac{\\pi_i(d_{10})}{\\pi_i}\\,y_i(d_{10})\\right),\\\\\n\\mathbb{E}[\\text{printed second line}]&=\\frac{1}{N}\\sum_{i=1}^{N}\\bigl(y_i(d_{11})+y_i(d_{10})\\bigr)=\\mu(d_{11})+\\mu(d_{10}).\n\\end{aligned}\n\\]\nNote that in the first expectation the two weights of each unit sum to one. If every unit has a direct peer, then $\\mu(\\mathbf{1})=\\mu(d_{11})$, and subtracting it from the first expectation gives the bias in the corrected sentence below.\n\nFor the counterexample, let ten units form a cycle, let exactly two units be treated, with all 45 assignments equally likely, and let $y_i(d_{11})=2$, $y_i(d_{10})=1$, $y_i(d_{01})=1$ and $y_i(d_{00})=0$ for every unit. Then $\\pi_i=1/5$, $\\pi_i(d_{11})=2/45$ and $\\pi_i(d_{10})=7/45$. On each of the 10 assignments in which the two treated units are adjacent, the first line equals $2$ and the printed second line equals $9$. On each of the other 35 assignments, the first line equals $1$ and the printed second line equals $9/7$. Enumerating the 45 assignments in exact arithmetic, I find that the first line has expectation $11/9$ and the printed second line has expectation $3$, while $\\mu(\\mathbf{1})=2$. The printed equality fails on all 45 assignments, and the corrected equality holds on all 45. The bias of the first line for $\\mu(\\mathbf{1})$ is $11/9-2=-7/9$, which is the value of the corrected formula.", "problem": "The first line of the display defines the estimator that ignores interference. It weights the outcome of every treated unit by $1/\\pi_i$, where $\\pi_i$ is the probability that unit $i$ is treated. The second line is presented as an expansion of the first over the two exposure conditions of a treated unit, $d_{11}$ (a treated direct peer) and $d_{10}$ (none). Such an expansion keeps the denominator $\\pi_i$ in both terms. The printed line divides instead by the exposure probabilities $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$, and the result is a different quantity. Its expectation is $\\mu(d_{11})+\\mu(d_{10})$, the sum of the averages of two potential outcomes. The expectation of the first line is the mixture of $y_i(d_{11})$ and $y_i(d_{10})$ that the surrounding text describes. The closing sentence also calls the estimator biased for another estimator, $\\widehat{\\mu_{HT}}(\\mathbf{1})$, where the estimand $\\mu(\\mathbf{1})$ is meant. The condition ``for some $i$'' in that sentence is the subject of \\fid{P045-E016}.", "what_survives": "The definition of the estimator in the first line, the Horvitz--Thompson estimators of Section 16.4.2 and of the next display, and the explanation that the estimator mixes $y_i(d_{11})$ and $y_i(d_{10})$ are unaffected. In both copies of the package in the replication files, the corresponding code divides the treated total by one common probability. No error in Table 16.2 or in Figure 16.4 is established: they report simulations of Hajek estimators, and I found no dependence of them on the printed expansion. The second equality and the closing sentence change."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E002", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Two lines of text, a displayed formula in two lines, and two more lines of text. The formula sets y-hat(z; psi) equal to a fraction plus a second fraction. Red boxes mark the two denominators: the sum over g of the indicator of S_g = psi psi, and the sum over g of the indicator of S_g = psi phi.", "caption": "Section 16.5.2, p. 312 (PDF p. 24), left column: the display for $\\widehat{y}(z;\\psi)$, with the two separate denominators marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T14", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T14"}, {"id": "P045-T15", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T15"}], "headline": "The pooled estimator $\\widehat{y}(z;\\psi)$ divides the sum of the group means by the number of all groups with saturation $\\psi$; the display prints the sum of two subgroup means, each with its own denominator.", "id": "P045-E003", "location": "Section 16.5.2, p. 312 (PDF p. 24), left column, the display after “the estimator for the population average potential outcome under saturation $\\psi$ is”", "pdf_page": 24, "printed_page": "312", "reaches": [{"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T14", "P045-T15"], "text": {"as_printed": "In other words, the estimator for the population average potential outcome under saturation $\\psi$ is\n\\[\n\\widehat{y}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\psi)\\mathbf{I}(S_g=\\psi\\psi)}{\\hl{\\textstyle\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\psi)}}+\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\phi)\\mathbf{I}(S_g=\\psi\\phi)}{\\hl{\\textstyle\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\phi)}}.\n\\]\nThe average outcome across groups includes a mixture of the average of observed outcomes for units with treatment $z$ in a group with realized saturation $\\psi$ in a tract in which the realized saturation for the other group is $\\psi$, and the average of observed outcomes for units with treatment $z$ in a group with realized saturation $\\psi$ in a tract in which the realized saturation for the other group is $\\phi$.", "corrected": "In other words, the estimator for the population average potential outcome under saturation $\\psi$ is\n\\[\n\\widehat{y}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\psi)\\mathbf{I}(S_g=\\psi\\psi)+\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\phi)\\mathbf{I}(S_g=\\psi\\phi)}{\\hl{\\textstyle\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\psi)+\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\phi)}},\n\\]\nwhich is defined whenever at least one group has saturation $\\psi$ and \\hl{each included group has at least one unit with treatment $z$}. Here $S_g=\\psi\\psi$ means that group $g$ has saturation $\\psi$ and the other group in its tract has saturation $\\psi$, and $S_g=\\psi\\phi$ means that group $g$ has saturation $\\psi$ and the other group has saturation $\\phi$. When both kinds of group occur, the estimator is the average of the two subgroup means weighted by the shares of the two kinds of group.", "evidence": "A derivation and an exact example. For positive $a$ and $b$, I have\n\\[\n\\frac{A}{a}+\\frac{B}{b}-\\frac{A+B}{a+b}=\\frac{bA}{a(a+b)}+\\frac{aB}{b(a+b)},\n\\]\nwhich is positive when $A$ and $B$ are positive. For the example, I take the design of Section 16.5.2, with six groups of 75 units in three tracts, $\\psi=2/3$ and $\\phi=1/3$. I let every baseline outcome be $1$ and use the outcome multipliers of the package function \\texttt{make\\_dilated\\_out\\_hh\\_miss}: in a group with saturation $\\psi$, a treated unit has outcome $9/4$ and an untreated unit $11/8$ when the other group in the tract has saturation $\\psi$, and $2$ and $5/4$ when it has saturation $\\phi$. Let groups 1, 2 and 3 have saturation $\\psi$, with tracts $(1,2)$, $(3,4)$ and $(5,6)$, so that $a=2$ and $b=1$. For treated units the estimator is $(9/2+2)/3=13/6$ and the display gives $9/4+2=17/4$. For untreated units the values are $4/3$ and $21/8$. Of the 20 equally likely choices of the three groups with saturation $\\psi$, 8 place one such group in each tract. Then $a=0$ and the first term of the display is $0/0$. On all 20 choices the corrected form equals the estimator of p.~310, and the package function \\texttt{estimates\\_hierarchical} computes that estimator.", "problem": "On p.~310 the estimator $\\widehat{y}(z;\\psi)$ is the mean of the group means $\\widehat{y}_g(z;\\psi)$ over all groups with saturation $\\psi$. Section 16.5.2 splits these groups by the saturation of the other group in the tract. Among the groups with saturation $\\psi$, let $a$ be the number of groups such that the other group in the tract has saturation $\\psi$, and let $b$ be the number such that it has saturation $\\phi$. Let $A$ and $B$ be the sums of the group means over these two sets of groups. The estimator is $(A+B)/(a+b)$, which is the mixture of the subgroup means $A/a$ and $B/b$ with weights $a/(a+b)$ and $b/(a+b)$ that the next sentence describes. The display prints $A/a+B/b$, the sum without weights. When both subgroups occur and the outcomes are positive, this exceeds the estimator; when one subgroup is empty, it contains $0/0$, although the pooled estimator is still defined.", "what_survives": "The definition on p.~310, the sentence after the display and the argument of Section 16.5.2 are unaffected: pooling groups from tracts of different composition can bias the estimator of $\\tau^{D}(\\psi\\psi)$. The package computes the pooled estimator. The script for Figure 16.6 is not available, and it could not be confirmed that the figure was computed from the pooled estimator and not from the display."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E003", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the middle of page 298: two paragraphs set in two columns, a one-line sentence, three blocks of R code, and the output printed after two of them. Red boxes mark the words 'cannot compute' in the sentence about the estimate of tau HT of d11 and d00, and the entry NA under the column dir_ind1 in each of the two printed outputs, which otherwise read 41.61043 and 39.51643.", "caption": "Section 16.4.2, p. 298 (PDF p. 10). The marked terms are the words “cannot compute” and the entry NA for dir_ind1 in each of the two printed outputs.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T16", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T16"}, {"id": "P045-T17", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T17"}], "headline": "An empty Horvitz–Thompson total is zero, and $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ in the example equals $-13.27099$; the text says that it cannot be computed, and the package returns NA.", "id": "P045-E004", "location": "Section 16.4.2, p. 298 (PDF p. 10): the paragraph before the yT_HT code and the two printed outputs of $\\widehat{\\tau_{HT}}$; the function estimates of the interference package", "pdf_page": 10, "printed_page": "298", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P013-E018", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P044-E006", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [], "same_defect_as": [{"id": "P013-E018", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E018", "work_id": "P013"}, {"id": "P044-E006", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E006", "work_id": "P044"}], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T16", "P045-T17"], "text": {"as_printed": "In this case, we \\hl{cannot compute} $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ -- the interactive effect of direct and indirect exposure -- because in this small-scale example no unit received exposure $d_{11}$.\n\n\\begin{audittable}{rrr}{Output printed twice on p.~298, after \\texttt{tau\\_HT} and after the call to \\texttt{estimates}.}\n\\texttt{dir\\_ind1} & \\texttt{isol\\_dir} & \\texttt{ind1} \\\\\n\\hl{\\texttt{NA}} & 41.61043 & 39.51643 \\\\\n\\end{audittable}", "corrected": "In this case, \\hl{no unit received exposure $d_{11}$. Since every $\\widehat\\pi_i(d_{11})$ is positive, the total $\\widehat{y^T_{HT}}(d_{11})$ is zero, and} $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ -- the interactive effect of direct and indirect exposure -- \\hl{equals $-\\widehat{y^T_{HT}}(d_{00})/N$.}\n\n\\begin{audittable}{rrr}{Corrected output, in both places on p.~298.}\n\\texttt{dir\\_ind1} & \\texttt{isol\\_dir} & \\texttt{ind1} \\\\\n\\hl{$-13.27099$} & 41.61043 & 39.51643 \\\\\n\\end{audittable}\n\nIn general, let $d_k$ be an exposure such that $\\pi_i(d_k)>0$ for all units $i$. Then\n\\[\n\\widehat{y^T_{HT}}(d_k)=\\sum_{i=1}^N\\mathbf I(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)}\n\\]\nis defined for every assignment, and it equals $0$ when no unit has $D_i=d_k$. \\hl{If positive estimates $\\widehat\\pi_i(d_k)$ are used instead, replace $\\pi_i(d_k)$ by $\\widehat\\pi_i(d_k)$ in this formula; an empty total is still zero.} With the true probabilities, its expectation over all assignments is $\\sum_{i=1}^N y_i(d_k)$. The Hajek estimator and the sample mean of an exposure that no unit receives remain undefined. In the package, the function \\texttt{estimates} should keep the zero total and not replace it by \\texttt{NA}.", "evidence": "The evidence is a derivation from the definition, a recalculation of the example with the package, and an exact counterexample.\n\nSuppose that $\\pi_i(d)>0$ for every unit $i$ and that the observed outcomes are finite. If no unit has $D_i=d$, then every summand of $\\widehat{y^T_{HT}}(d)$ is $0\\cdot Y_i/\\pi_i(d)=0$, and the total is $0$. Since $Y_i=y_i(d)$ when $D_i=d$, I have\n\\[\n\\mathbb E\\bigl[\\widehat{y^T_{HT}}(d)\\bigr]=\\sum_{i=1}^N y_i(d)\\,\\frac{\\Pr(D_i=d)}{\\pi_i(d)}=\\sum_{i=1}^N y_i(d),\n\\]\nwhere the expectation is over all assignments, including those that place no unit in $d$. In the last equality, I used that $\\Pr(D_i=d)=\\pi_i(d)$.\n\nFor the example, I ran two versions of the package (commits \\texttt{a5b8dfb} and \\texttt{c3fc7b3} of its repository) with the chapter's seeds and the printed adjacency matrix, under the sampling algorithm that was the default before R 3.6.0. Both reproduce the printed assignment and the printed probabilities, and both return the totals $(\\texttt{NA},\\,548.8142,\\,527.8742,\\,132.7099)$ for $(d_{11},d_{10},d_{01},d_{00})$ and the printed contrasts. When I remove the line of \\texttt{estimates} that sets an empty total to \\texttt{NA}, the first total is $0$ and the first contrast is $(0-132.7099)/10=-13.27099$. The other two contrasts do not change, and the Hajek contrast for $d_{11}$ remains \\texttt{NA}. The same line occurs in the package's function for the full-neighbourhood exposures, which I did not test.\n\nThe counterexample uses the printed network of ten units, with two units assigned to treatment uniformly among the 45 pairs and potential outcomes\n\\[\n\\bigl(y_i(d_{11}),\\,y_i(d_{10}),\\,y_i(d_{01}),\\,y_i(d_{00})\\bigr)=\\bigl(2,\\,3/2,\\,5/4,\\,1\\bigr)\n\\]\nfor every unit. The true total for $d_{11}$ is $20$, and $\\tau(d_{11},d_{00})=1$. A unit with $g_i$ neighbours has $\\pi_i(d_{11})=g_i/45$, and every exposure probability of every unit is positive. The network has 20 edges. The 20 assignments that treat two adjacent units place those two units in $d_{11}$, and the other 25 assignments place no unit there. On the assignment that treats the adjacent units $a$ and $b$, the total is $90(1/g_a+1/g_b)$, and the sum of these totals over the 20 edges is $900$. With empty totals equal to zero, I obtain\n\\[\n\\mathbb E\\bigl[\\widehat{y^T_{HT}}(d_{11})\\bigr]=\\frac{900}{45}=20 ,\n\\]\nwhich is the true total. The mean over the 20 assignments with a unit in $d_{11}$ is $900/20=45$. For the contrast, both $d_{11}$ and $d_{00}$ are received by some unit on 19 assignments. The mean of $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ over those 19 assignments is $1227/380\\approx3.23$. The mean over all 45 assignments, with empty totals equal to zero, is $1$.", "problem": "The chapter defines the Horvitz--Thompson total on p.~297 as a sum over all units,\n\\[\n\\widehat{y^T_{HT}}(d_k)=\\sum_{i=1}^N\\mathbf I(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)},\n\\]\nunder the assumption that every unit has a positive probability of each exposure. When no unit receives $d_{11}$, every indicator in the sum is zero, and I call the total empty. An empty total equals $0$, and the contrast $\\widehat{\\tau_{HT}}(d_{11},d_{00})=-\\widehat{y^T_{HT}}(d_{00})/N$ is a finite number. The sentence on p.~298 says that the contrast cannot be computed, and the package returns \\texttt{NA}, because the function \\texttt{estimates} replaces an empty total by \\texttt{NA}. On such an assignment, the package does not compute the estimator of p.~297. The unbiasedness stated for that estimator refers to the mean over all assignments, with empty totals equal to zero. If the assignments on which the package returns \\texttt{NA} are dropped, the mean of the remaining estimates can differ from the estimand. An empty Hajek ratio or sample mean is $0/0$ and remains undefined.", "what_survives": "The definition of the estimator on p.~297, the estimand and the unbiasedness of the estimator with the true exposure probabilities are not affected. The contrasts $41.61043$ and $39.51643$ of the example stand, as does every Horvitz--Thompson output on an assignment in which each exposure involved is received by some unit. I could not determine whether the simulations with $N=400$ are affected, because the scripts and draws behind their tables are not available and I do not know how they treated empty totals."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E004", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines from the left column of page 303. The text reads 'Note that this example uses a relatively small set of units (N = 400), and the unbiased estimators (in this case, tau-hat H of d11, d00, tau-hat H of d111, d000) work with smaller subsets of the data than the coarser, biased estimators (here, tau-hat H of d1, d0). This is apparent if one looks at the standard deviations in Table 16.2.' A red box marks the word 'unbiased'.", "caption": "Section 16.4.3, p. 303 (PDF p. 15), left column. The marked word is “unbiased”.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E005.png"}], "formal_release": null, "formal_statements": [], "headline": "On p. 303, “the unbiased estimators” are the Hajek estimators whose exposure mapping is adequately specified; a Hajek estimator need not be exactly unbiased in a finite sample.", "id": "P045-E005", "location": "Section 16.4.3, p. 303 (PDF p. 15), left column, the paragraph that begins “Note that this example uses a relatively small set of units”", "pdf_page": 15, "printed_page": "303", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Note that this example uses a relatively small set of units ($N=400$), and the \\hl{unbiased} estimators (in this case, $\\widehat{\\tau_H}(d_{11},d_{00})$, $\\widehat{\\tau_H}(d_{111},d_{000})$) work with smaller subsets of the data than the coarser, biased estimators (here, $\\widehat{\\tau_H}(d_1,d_0)$).''", "corrected": "``Note that this example uses a relatively small set of units ($N=400$), and the \\hl{estimators using adequately specified exposure mappings} (in this case, $\\widehat{\\tau_H}(d_{11},d_{00})$, $\\widehat{\\tau_H}(d_{111},d_{000})$) work with smaller subsets of the data than the coarser, biased estimators (here, $\\widehat{\\tau_H}(d_1,d_0)$).''", "intended_meaning": "The word separates the estimators whose exposure mapping allows for at least as much interference as the data-generating process from the coarser ones. The intended meaning is that the former carry no bias from the exposure mapping, and this is correct. The estimators named are Hajek estimators, which are ratios, and a ratio estimator need not be exactly unbiased in a finite sample. The chapter says so on p.~299, where the Hajek estimator ``improves efficiency with a small cost in terms of finite-sample bias''. Table 16.2 reports biases of 1.29 and 0.85 for $\\widehat{\\tau_H}(d_{11},d_{00})$ when interference is of first degree, and of 4.28 and 2.58 for $\\widehat{\\tau_H}(d_{111},d_{000})$ when it is of second degree. An exact example confirms the point. Let exactly one of two units be treated, with assignment probabilities $(1/4,3/4)$, treated outcomes $(1,3)$, control outcomes zero and no interference. The Hajek contrast has expectation $5/2$ and the Horvitz--Thompson contrast has expectation $2$, which is the value of the estimand. The formulas and the simulation results are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E005", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of six lines near the foot of the right column of page 297, beginning 'Specifically, we generate a variable with random values from'. Red boxes mark the phrase 'an absolute standard normal distribution', which runs over two lines.", "caption": "Section 16.4.2, p. 297 (PDF p. 9), foot of the right column. The marked phrase describes the distribution of the baseline outcome.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E006.png"}], "formal_release": null, "formal_statements": [], "headline": "The baseline outcome is a half-normal variable scaled by degree plus a normal term, as make_corr_out generates it; it is not an absolute standard normal variable.", "id": "P045-E006", "location": "Section 16.4.2, pp. 297–298 (PDF pp. 9–10), last paragraph of p. 297; see also p. 305 (PDF p. 17), right column", "pdf_page": 9, "printed_page": "297–298", "reaches": [], "related_entries": [{"id": "P044-E002", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E002", "work_id": "P044"}], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Specifically, we generate a variable with random values from \\hl{an absolute standard normal distribution} that is correlated with the unit's first- and second-order degree -- the number of peers and peers of peers, respectively (for which we use the arguments \\texttt{adj\\_matrix} and \\texttt{make\\_corr\\_out} in the function below). This variable determines the potential outcome under the $d_{00}$ condition.", "corrected": "Specifically, we generate a variable \\hl{$(k_i+k_i^{(2)}+k_ik_i^{(2)})|X_i|+E_i$, where $k_i$ and $k_i^{(2)}$ are} the unit's first- and second-order degree -- the number of peers and peers of peers, respectively \\hl{-- and $X_i\\sim N(0,1)$ and $E_i\\sim N(1,0.25^2)$ are independent. We use} the arguments \\texttt{adj\\_matrix} and \\texttt{make\\_corr\\_out} in the function below. This variable determines the potential outcome under the $d_{00}$ condition. \\hl{A positive lower bound of the outcomes in a simulated population refers to its realized minimum, not a bound on this distribution.}", "intended_meaning": "The sentence describes a baseline outcome that is correlated with degree and is built by the function \\texttt{make\\_corr\\_out}, which it names. That meaning is clear and correct. This function does not draw from the absolute value of a standard normal variable alone. It sets\n\\[\ny_i(d_{00})=\\bigl(k_i+k_i^{(2)}+k_ik_i^{(2)}\\bigr)\\,|X_i|+E_i,\n\\]\nwhere $k_i$ and $k_i^{(2)}$ are the first- and second-order degree of unit $i$, and $X_i\\sim N(0,1)$ and $E_i\\sim N(1,0.25^2)$ are independent. This distribution has no lower bound, since $E_i$ can be negative. The ``positive lower bound'' of the outcomes on p.~305 then refers to the smallest value in the fixed simulated population, which can be positive, and not to the distribution. The ten baseline values in the example of p.~298 are all positive, with minimum $6.194878$. No reported result depends on the description."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E006", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the right column of page 292. Red boxes mark three terms: the equation that the size of Omega is 2 to the power N, the definition of Omega as the set of assignments z with positive probability p z, and, in the example with N equal to 10, the parenthesis stating that the size of Omega is 45.", "caption": "Section 16.3, p. 292 (PDF p. 4), right column. The marked terms are the two descriptions of $\\Omega$ and its size in the example with ten units.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E007.png"}, {"alt": "Crop of eight lines from the right column of page 292, beginning 'To analyze the effect of different treatment assignments'. A red box marks the expression y i, colon, Omega, arrow, R, which defines the potential outcome as a function from Omega to the real numbers.", "caption": "Section 16.3, p. 292 (PDF p. 4), right column, lower down. The marked term is the domain of the potential outcome function.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E007-b.png"}], "formal_release": null, "formal_statements": [], "headline": "$\\Omega$ is the set of assignments that the design can produce; the count $2^N$ belongs to the full space $\\{0,1\\}^N$, on which the potential outcomes are defined.", "id": "P045-E007", "location": "Section 16.3, p. 292 (PDF p. 4), right column: the definition of $\\Omega$ and the domain of the potential outcomes", "pdf_page": 4, "printed_page": "292", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Let $\\Omega$ be the set of treatment assignment vectors $\\mathbf z$ with $\\hl{|\\Omega|=2^N}$. An \\emph{experimental design} is a plan for randomly selecting a particular value of $\\mathbf z$ from the $2^N$ different possibilities with predetermined probability $p_{\\mathbf z}$ -- for example, Bernoulli assignment (i.e., coin flips) or a completely randomized assignment strategy. Therefore, $\\Omega=\\{\\mathbf z:p_{\\mathbf z}>0\\}$ and the realized treatment assignment $\\mathbf Z=(Z_1,\\ldots,Z_N)'$ is a random vector with support $\\Omega$ and $\\Pr(\\mathbf Z=\\mathbf z)=p_{\\mathbf z}$. For example, with a population of size $N=10$ and an experimental design that randomly assigns without replacement a proportion $p=0.2$ to treatment condition $z_i=1$ with uniform probability, there are $\\binom{N}{pN}=45$ possible treatment assignments ($|\\Omega|=45$) and the realized treatment assignment $\\mathbf Z$ has $p_{\\mathbf z}=\\frac{1}{45}$. \\ldots\\ These potential outcomes are defined for each unit $i$ as the elements in the image of a function that maps assignment vectors to real valued outcomes, $y_i:\\hl{\\Omega}\\to\\mathbb R$.", "corrected": "\\hl{There are $2^N$ treatment assignment vectors $\\mathbf z$ in $\\{0,1\\}^N$.} An \\emph{experimental design} is a plan for randomly selecting a particular value of $\\mathbf z$ from the $2^N$ different possibilities with predetermined probability $p_{\\mathbf z}$ -- for example, Bernoulli assignment (i.e., coin flips) or a completely randomized assignment strategy. \\hl{Let $\\Omega=\\{\\mathbf z:p_{\\mathbf z}>0\\}$ be the support of the design. The} realized treatment assignment $\\mathbf Z=(Z_1,\\ldots,Z_N)'$ is a random vector with support $\\Omega$ and $\\Pr(\\mathbf Z=\\mathbf z)=p_{\\mathbf z}$. \\ldots\\ These potential outcomes are defined for each unit $i$ as the elements in the image of a function that maps assignment vectors to real valued outcomes, $y_i:\\hl{\\{0,1\\}^N}\\to\\mathbb R$.", "intended_meaning": "The set $\\Omega$ is the support of the design, $\\{\\mathbf z:p_{\\mathbf z}>0\\}$, and the chapter uses it in that sense. In the example on the same page, two of ten units are treated, and $|\\Omega|=\\binom{10}{2}=45$. The full space has $2^{10}=1024$ vectors. The count $2^N$ is the number of all vectors in $\\{0,1\\}^N$, and it equals $|\\Omega|$ only when the design gives every vector positive probability. With $\\Omega$ as the support, the map $y_i:\\Omega\\to\\mathbb R$ is defined only on the assignments that the design can produce. In the example these exclude the all-treated and all-control vectors, which define $\\tau(\\mathbf 1,\\mathbf 0)$ on p.~300. The potential outcomes for such assignments are meant to exist, and their domain is $\\{0,1\\}^N$. No estimator or result depends on the count."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E007", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of nine lines from the right column of page 292. A red box marks the first entry of the displayed vector z sub minus i, which is printed as z sub i and is followed by dots, z sub i minus 1, z sub i plus 1, dots, z sub N.", "caption": "Section 16.3, p. 292 (PDF p. 4), right column. The marked term is the first entry of the vector $\\mathbf z_{-i}$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E008.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T18", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T18"}], "headline": "For $i>1$, the vector $\\mathbf z_{-i}$ begins with $z_1$, not with $z_i$.", "id": "P045-E008", "location": "Section 16.3, p. 292 (PDF p. 4), right column: the definition of $\\mathbf z_{-i}$", "pdf_page": 4, "printed_page": "292", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T18"], "text": {"as_printed": "For convenience, let $\\mathbf z_{-i}=(z_{\\hl{i}},\\ldots,z_{i-1},z_{i+1},\\ldots,z_N)'$ denote the $(N-1)$-element vector that removes the $i$th element from $\\mathbf z$.", "corrected": "For convenience, let $\\mathbf z_{-i}=(z_{\\hl{1}},\\ldots,z_{i-1},z_{i+1},\\ldots,z_N)'$ denote the $(N-1)$-element vector that removes the $i$th element from $\\mathbf z$. \\hl{For $N>1$, the vector begins with $z_2$ when $i=1$ and ends with $z_{N-1}$ when $i=N$; for $N=1$, it is empty.}", "evidence": "The evidence is a check of the page. The first subscript in the formula is the letter $i$. The definition $\\mathbf z=(z_1,\\ldots,z_N)'$ at the top of the same column has the numeral $1$ in that place. For $N=4$ and $i=3$, removing the third element of $(z_1,z_2,z_3,z_4)'$ leaves $(z_1,z_2,z_4)'$, and the printed formula begins with $z_3$ instead. The same page uses $\\mathbf z_{-i}$ for ``the treatment assignment of all other households''.", "problem": "The words define $\\mathbf z_{-i}$ as the vector $\\mathbf z$ with its $i$th element removed. The formula begins with $z_i$, the element that is removed. For $i>1$, the correct first entry is $z_1$.", "what_survives": "Nothing else changes. The words beside the formula give the intended definition, and the chapter uses $\\mathbf z_{-i}$ in that sense afterwards."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E008", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Seven lines of text from the top of the left column. A red box marks the expression 'psi (n_g minus 1)' in the parenthesis '(i.e., psi (n_g minus 1) when i is assigned to treatment and psi (n_g) when i is assigned to control)'.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), top of the left column: the parenthesis that counts the treated group members other than $i$, with $\\psi(n_g-1)$ marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E009.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T19", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T19"}], "headline": "Under complete random assignment, a treated unit has $\\psi n_g-1$ treated group members other than itself, not $\\psi(n_g-1)$.", "id": "P045-E009", "location": "Section 16.5.1, p. 310 (PDF p. 22), left column, first lines, the parenthesis beginning “(i.e.,”", "pdf_page": 22, "printed_page": "310", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T19"], "text": {"as_printed": "\\ldots\\ complete random assignment, the individual-level direct effect for unit $i$ would contrast outcomes with different numbers of group members other than $i$ assigned to treatment (i.e., $\\hl{\\psi(n_g-1)}$ when $i$ is assigned to treatment and $\\psi(n_g)$ when $i$ is assigned to control).", "corrected": "\\ldots\\ complete random assignment, the individual-level direct effect for unit $i$ would contrast outcomes with different numbers of group members other than $i$ assigned to treatment (i.e., $\\hl{\\psi n_g-1}$ when $i$ is assigned to treatment and $\\psi(n_g)$ when $i$ is assigned to control).", "evidence": "A derivation, and the example on the same page. When $Z_{ig}=z$, the number of treated group members other than $i$ is\n\\[\n\\sum_{j\\neq i}Z_{jg}=\\psi n_g-z .\n\\]\nIn the toy example, $n_g=3$ and $\\psi=2/3$. Among the three assignments that treat two of the three units, a treated unit has one treated group member other than itself and a control unit has two. The printed expression equals $4/3$ for the treated unit.", "problem": "Complete random assignment at saturation $\\psi$ treats exactly $\\psi n_g$ of the $n_g$ units in the group. If unit $i$ is treated, it is one of them, and $\\psi n_g-1$ of the other units are treated. If unit $i$ is in control, all $\\psi n_g$ treated units are among the others. The printed $\\psi(n_g-1)$ equals $\\psi n_g-\\psi$, which exceeds the count for a treated unit by $1-\\psi$.", "what_survives": "The point of the parenthesis is unaffected: under complete random assignment, the outcomes of a treated unit and of a control unit are compared at different numbers of treated group members. The count $\\psi n_g$ for a control unit is correct, and nothing else in the chapter uses the expression."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E009", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Six lines of text from the foot of the left column. Red boxes mark the words 'at least within epsilon hops' at the end of the fifth line and 'of each other' at the start of the sixth, in the sentence 'clusters are formed by finding a set of units such that all units in the set are at least within epsilon hops of each other, and every unit outside the'.", "caption": "Section 16.4.6, p. 305 (PDF p. 17), foot of the left column: the definition of $\\epsilon$-net clustering, with the phrase “at least within $\\epsilon$ hops of each other” marked. The sentence continues at the top of the right column.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E010.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T20", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T20"}, {"id": "P045-T21", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T21"}], "headline": "In the definition of $\\epsilon$-net clustering, the units in the set are at least $\\epsilon$ hops apart, not “at least within $\\epsilon$ hops of each other”.", "id": "P045-E010", "location": "Section 16.4.6, p. 305 (PDF p. 17), last lines of the left column, continuing at the top of the right column", "pdf_page": 17, "printed_page": "305", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T20", "P045-T21"], "text": {"as_printed": "Ugander et al.\\ (2013) analyze an intuitive graph clustering method, $\\epsilon$-net clustering, where clusters are formed by finding a set of units such that all units in the set are \\hl{at least within $\\epsilon$ hops of each other}, and every unit outside the set is within $\\epsilon-1$ hops of a unit in the set.", "corrected": "Ugander et al.\\ (2013) analyze an intuitive graph clustering method, $\\epsilon$-net clustering, where clusters are formed by finding a set of units such that all units in the set are \\hl{at least $\\epsilon$ hops apart}, and every unit outside the set is within $\\epsilon-1$ hops of a unit in the set.", "evidence": "I checked the page and the intended greedy clustering procedure, and used an exact example.\n\nThe greedy procedure intended by \\texttt{make\\_tr\\_vec\\_bernoulli\\_cluster} in the \\texttt{interference} package picks an unmarked unit, marks every unit within two hops of it, and repeats until no unit is unmarked. These rules give pairwise separation of at least three hops and coverage within two hops. When only one eligible vertex remains, the actual sampler can choose an already-marked unit and violate the separation condition; see \\fid{P045-E013}. On a path with nine vertices and $\\epsilon=3$, none of the 511 nonempty sets of vertices has all its members within three hops of each other and every vertex within two hops of the set, while 13 sets have all their members at least three hops apart and every vertex within two hops of the set. The article of Ugander et al.\\ (2013) was not consulted.", "problem": "The marked phrase joins two conditions that point in opposite directions. ``At least'' is a lower bound on the distance between two units in the set, and ``within'' is an upper bound. The definition needs the lower bound: any two units in the set are at least $\\epsilon$ hops apart, and the second clause supplies the upper bound, on the distance from a unit outside the set to the set. If the phrase is read as an upper bound, the two clauses can be impossible to satisfy together.", "what_survives": "The second clause of the definition and the intended 3-net design in Section 16.4.6 are unaffected by this wording correction. No change to the simulation numbers follows from this phrase alone."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E010", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 16.3 on page 305, 'Comparing unit to cluster randomization', with columns Randomization, Estimator, Estimand, True value, Average value, Bias, SD and RMSE, four rows (Unit and Cluster for Horvitz-Thompson, Unit and Cluster for Hajek) and the note below the table. A red box marks the Estimand column, which reads tau(d1, d0) in all four rows.", "caption": "Table 16.3, p. 305 (PDF p. 17). The marked column gives the estimand, the contrast $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ between the two full-neighborhood exposure conditions.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E011.png"}, {"alt": "Crop of seven lines from the right column of page 305. The text reads 'We compare the Horvitz-Thompson and Hajek estimators and their variance for the causal estimand of full neighborhood exposure, tau(d1, d0), for the case in which units are assigned to treatment via unit-level randomization as opposed to three-net clustering randomization. In both cases,'. Red boxes mark the words 'the causal estimand of full neighborhood' and 'exposure, tau(d1, d0),'.", "caption": "Section 16.4.6, p. 305 (PDF p. 17), right column. The marked words name the estimand of Table 16.3.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E011-b.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T5", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T5"}, {"id": "P045-T6", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T6"}, {"id": "P045-T27", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T27"}], "headline": "The package’s full-neighborhood exposure mapping must code a unit as all control when no member of its closed neighborhood is treated; as written it codes every unit whose closed neighborhood is not entirely treated as all control, so that the contrast it supports is not the $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ of Section 16.4.6.", "id": "P045-E011", "location": "Section 16.4.6 and Table 16.3, p. 305 (PDF p. 17); lines 104–121 of the file R/make_exposure_map_AS.R in the R package interference, version 0.1.0 (code, not printed in the chapter)", "pdf_page": 17, "printed_page": "305", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P045-T27", "P045-T5", "P045-T6"], "text": {"as_printed": "Section 16.4.6, p.~305: ``We compare the Horvitz--Thompson and Hajek estimators and their variance for the causal estimand of full neighborhood exposure, $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$, for the case in which units are assigned to treatment via unit-level randomization as opposed to three-net clustering randomization.''\n\nPackage, documentation of \\texttt{make\\_exposure\\_map\\_full\\_neighborhood}: ``that produces two exposure conditions: All Treat; a unit and 100\\% of its first-degree neighbors are treated, and All Control; a unit and 100\\% of its first-degree neighbors are in the control condition.''\n\\begin{audittable}{rl}{Package, \\texttt{R/make\\_exposure\\_map\\_AS.R}, lines 116--120.}\nLine & Code \\\\\n116 & \\texttt{exposure <- as.numeric(tr\\_vector\\%*\\%adj\\_matrix\\_diag)} \\\\\n117 & \\texttt{B\\_size <- rowSums(adj\\_matrix\\_diag)} \\\\\n118 & \\texttt{return(matrix(as.numeric(c(exposure==B\\_size,} \\\\\n119 & \\texttt{~~~~\\hl{exposure<B\\_size})),} \\\\\n120 & \\texttt{~~N, 2, dimnames = list(NULL, c('all\\_treat', 'all\\_control'))))} \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{rl}{Package, \\texttt{R/make\\_exposure\\_map\\_AS.R}, lines 118--120, corrected.}\nLine & Code \\\\\n118 & \\texttt{return(matrix(as.numeric(c(exposure==B\\_size,} \\\\\n119 & \\texttt{~~~~\\hl{exposure==0})),} \\\\\n120 & \\texttt{~~N, 2, dimnames = list(NULL, c('all\\_treat', 'all\\_control'))))} \\\\\n\\end{audittable}\nFor an undirected network without loops, the function then returns $\\mathbf{I}(s_i=b_i)$ and $\\mathbf{I}(s_i=0)$: a unit is all treated when every member of its closed neighborhood is treated and all control when none is. A unit with a partly treated closed neighborhood is in neither condition, so the two columns no longer sum to one. Recompute exposure probabilities and weights from the corrected events, retaining the original population size and observed outcomes. In simulations, code that uses the two columns to generate outcomes, as the package's example does, must assign mixed-neighborhood outcomes from its outcome model rather than turn them into zeros.", "evidence": "The evidence is a derivation and an exact counterexample. For the derivation, consider an undirected network without loops, with adjacency matrix $A$, and a treatment vector $\\mathbf{z}$ with entries 0 and 1. For each unit $i$, let $B_i=\\{i\\}\\cup\\{j:A_{ij}=1\\}$ be its closed neighborhood, let $b_i=|B_i|$, and let $s_i=\\sum_{j\\in B_i}z_j$. Lines 114--117 compute \\texttt{exposure} $=s_i$ and \\texttt{B\\_size} $=b_i$. Since $0\\le s_i\\le b_i$, I have $s_i=b_i$ exactly when every member of $B_i$ is treated and $s_i=0$ exactly when no member is treated. The condition $s_i<b_i$ holds exactly when at least one member is untreated, and it includes every case with $0<s_i<b_i$. Note that the first indicator of the function is correct and that the second is the complement of the first.\n\nFor the counterexample, let two units be joined by one tie, so that $b_1=b_2=2$. For the assignment $\\mathbf{z}=(1,0)$, I have $s_1=s_2=1$. The function, run unchanged from both copies of the package in the replication files, returns (all treated, all control) $=(0,1)$ for both units, and the documented conditions give $(0,0)$. Let the two units be treated independently with probability $1/2$. Each unit is all treated with probability $1/4$, all control with probability $1/4$, and coded all control by the function with probability $3/4$. Let both units have outcome $Y_i(\\mathbf{z})=1+z_1+z_2$, so that $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})=3-1=2$. The table lists the Horvitz--Thompson contrast of p.~297 for each assignment, with exact probabilities. Its expectation is $4/3$ with the function's events and $2$ with the documented events.\n\\begin{audittable}{crccrr}{Two units joined by one tie, treated independently with probability $1/2$, with outcomes $Y_i(\\mathbf{z})=1+z_1+z_2$. Columns 3 and 4 give the pair (all treated, all control), which is the same for both units, from the function as written and from the documented conditions. Columns 5 and 6 give the Horvitz--Thompson contrast with the events of the function as written and with the documented events.}\n$(z_1,z_2)$ & $s_i$ & Pair, as written & Pair, documented & Contrast, as written & Contrast, documented \\\\\n$(0,0)$ & 0 & $(0,1)$ & $(0,1)$ & $-4/3$ & $-4$ \\\\\n$(0,1)$ & 1 & $(0,1)$ & $(0,0)$ & $-8/3$ & $0$ \\\\\n$(1,0)$ & 1 & $(0,1)$ & $(0,0)$ & $-8/3$ & $0$ \\\\\n$(1,1)$ & 2 & $(1,0)$ & $(1,0)$ & $12$ & $12$ \\\\\nExpectation & & & & $4/3$ & $2$ \\\\\n\\end{audittable}\nWith the threshold outcomes $Y_i(\\mathbf{z})=1+2\\,\\mathbf{I}(z_1=z_2=1)$, the contrast is again $2$, and both versions of the estimator have expectation $2$. This is why the defect does not by itself imply an error in Table 16.3. Over all simple undirected networks on one to four labeled units and all treatment vectors, 1098 configurations in total, the function as written gets 2456 of the 4306 unit classifications wrong, and the corrected indicator agrees with a direct check of the treatment of every member of $B_i$ in every case.", "problem": "Section 16.4.6 and Table 16.3 concern the contrast $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ between two exposure conditions: a unit and all of its first-degree neighbors are treated, and a unit and all of its first-degree neighbors are in control. The section motivates it on p.~304 by the difference between 100\\% and 0\\% treatment saturation. The package has one exposure mapping for these two conditions, and its documentation describes them in the same way. Its second indicator is \\texttt{exposure<B\\_size}, where \\texttt{exposure} is the number of treated units in the closed neighborhood of a unit (the unit and its first-degree neighbors) and \\texttt{B\\_size} is the size of that neighborhood. This indicator is one whenever at least one member of the closed neighborhood is untreated. Then a unit whose closed neighborhood is partly treated is coded all control, and so is a treated unit with one untreated neighbor. The contrast that the function supports is between a closed neighborhood that is entirely treated and everything else.\n\nEvery quantity that the package computes for the comparison condition inherits the wrong event: its exposure probability, the inverse-probability weights, the comparison mean and the variance estimates. When the outcomes in a partly treated neighborhood differ from the outcomes under full control, the Horvitz--Thompson contrast built on the function's two indicators can estimate a different quantity from $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$, and nothing in the output shows it. Whether Table 16.3 was computed with this function is not known, because the script that produced the table is not in the replication files. No entry of Table 16.3 is shown to be wrong. If every unit short of full treatment has its baseline outcome, then the contrast built on the function's indicators has the same expectation as the intended one.", "what_survives": "The indicator for a closed neighborhood that is entirely treated is correct, and the package's mapping into the four conditions $d_{11}$, $d_{10}$, $d_{01}$ and $d_{00}$ is a separate function. The estimators and their theory for a correctly specified exposure mapping, and the argument of Section 16.4.6 for designs that raise the probability of the two full-neighborhood conditions, are unaffected. No entry of Table 16.3 and no ranking in it is shown to be wrong. The function changes, and outputs that use its all-control indicator have to be computed again with the corrected event. The package's 3-net sampler is the subject of \\fid{P045-E013}."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E011", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Figure 16.5 on page 304: twelve histograms of point estimates in three rows, for tau-hat H of d01, d00, of d10, d00 and of d11, d00, and four columns headed 0, 0.025, 0.225 and 0.425, each with a dashed vertical line at the true value. In the first and third rows the histograms of the three right-hand columns sit to the left of the dashed line by a similar amount. The caption reads 'Figure 16.5 The distribution of point estimates for 3000 simulations given different proportions of missing ties for a case of positive spillover. The dashed vertical line represents the true value of the target quantity.' Red boxes mark the words 'given different proportions of' and 'missing ties'.", "caption": "Figure 16.5, p. 304 (PDF p. 16). The columns are labelled 0, 0.025, 0.225 and 0.425 as proportions of missing ties. The marked words describe the intended simulation.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E012.png"}, {"alt": "Crop of eight lines from the right column of page 303. The text reads 'Another type of misspecification is when the network measured by the experimenter is different from the actual interference network. In this case, the bias of the estimator will increase with the proportion of mismeasured ties. For example, with positive spillover and an elicited network with missing ties, one would mistakenly take a mixture of the poten-'. Red boxes mark the sentence 'In this case, the bias of the estimator will increase with the proportion of mismeasured ties.'", "caption": "Section 16.4.4, p. 303 (PDF p. 15), right column, opening of the section. The marked sentence says that the bias increases with the proportion of mismeasured ties.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E012-b.png"}, {"alt": "Crop of eight lines from the right column of page 303. The text reads 'The distribution of point estimates is centered around the target quantity (the dashed line) when the proportion of unmeasured ties is zero, indicating unbiasedness. However, as the proportion of unmeasured ties increases, the distributions shift to the left or right, depending on the net effect of the biases for the estimated potential outcome means.' A red box marks the words 'is zero, indicating unbiasedness.'", "caption": "Section 16.4.4, p. 303 (PDF p. 15), right column, the description of Figure 16.5. The marked words are “is zero, indicating unbiasedness.”", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E012-c.png"}, {"alt": "The title reads 'Figure 16.5 compared with five new simulations using the package's tie-removal function and correctly mirrored random deletion'. Three panels, one for each of tau-hat H of d01, d00, of d10, d00 and of d11, d00. Titles give the printed targets 14.53, 29.06 and 58.12; simulation targets vary by network. Horizontal positions are labelled 0, 2.5 percent, 22.5 percent and 42.5 percent, under the axis label 'Printed missing-tie proportion / simulated deletion probability'; for the simulations these are deletion probabilities, not realized missing shares. The vertical axis runs from minus 13 to 7 units of the outcome. Black squares joined by a line show printed binned means minus dashed-line positions: for d01 about 0.3, minus 7.0, minus 7.0 and minus 6.3; for d10 about minus 0.2, 2.8, 3.6 and 3.9; for d11 about 1.3, minus 9.6, minus 9.1 and minus 8.0. Orange circles and blue triangles, five per position, show simulation means minus each network's true target. Orange circles generally follow the printed pattern. Blue triangles change little at 2.5 percent; by 42.5 percent they range from minus 1.2 to 0.0 for d01, from 2.2 to 3.9 for d10 and from 2.3 to 5.1 for d11.", "caption": "Comparison for the three contrasts of Figure 16.5. Black squares: printed binned histogram mean minus dashed-line position. Orange circles: new simulations with the package’s tie-removal function, one point per network. Blue triangles: new simulations with random deletion of ties, mirrored correctly. Simulation values subtract each network’s own true value; their horizontal positions denote deletion probabilities. The comparison supports but does not establish the source of the printed figure.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E012-missing-ties-r2.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T7", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T7"}, {"id": "P045-T8", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T8"}, {"id": "P045-T9", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T9"}], "headline": "The package’s functions for removing and for adding ties must copy the transpose of the edited upper triangle; as written they in general return an asymmetric network with ties that were never there, Figure 16.5 resembles simulations using these functions although its generating code is unknown, and the general statement on p. 303 that the bias increases with the proportion of mismeasured ties needs qualifying.", "id": "P045-E012", "location": "Figure 16.5, p. 304 (PDF p. 16), and Section 16.4.4, p. 303 (PDF p. 15), right column; lines 36 and 55 of the file R/functions.R in the R package interference, version 0.1.0 (code, not printed in the chapter)", "pdf_page": 16, "printed_page": "304", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [{"classification_after": {"class": "error", "severity": "Moderate"}, "classification_before": {"class": "error", "severity": "Moderate"}, "note": "The wording was revised; the classification remains a Moderate error. The package’s tie-editing functions are defective independently of how the prose is read. The description of approximate centering needs clarification because the Hajek estimator can have small finite-sample bias, and the general claim that bias increases with missing ties needs qualification. The evidence supports but does not establish that the defective functions generated Figure 16.5; its original inputs and generating code are needed to settle that attribution."}], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P045-T7", "P045-T8", "P045-T9"], "text": {"as_printed": "Package, file \\texttt{R/functions.R}, line 36 in the function \\texttt{make\\_adj\\_matrix\\_trunc}, which removes ties, and line 55 in the function \\texttt{make\\_adj\\_matrix\\_add}, which adds ties:\n\n\\texttt{m[lower.tri(m,diag=FALSE)] <- \\hl{m[upper.tri(m,diag=FALSE)]}}\n\nSection 16.4.4, p.~303: ``In this case, \\hl{the bias of the estimator will increase with the proportion of mismeasured ties}.'' and ``The distribution of point estimates is centered around the target quantity (the dashed line) when the proportion of unmeasured ties is zero, \\hl{indicating unbiasedness}. However, as the proportion of unmeasured ties increases, the distributions shift to the left or right, depending on the net effect of the biases for the estimated potential outcome means.''\n\nFigure 16.5, p.~304, caption: ``The distribution of point estimates for 3000 simulations given different proportions of missing ties for a case of positive spillover. The dashed vertical line represents the true value of the target quantity.''", "corrected": "Package, \\texttt{R/functions.R}, lines 36 and 55:\n\n\\texttt{m[lower.tri(m,diag=FALSE)] <- \\hl{t(m)[lower.tri(m,diag=FALSE)]}}\n\nFor a symmetric adjacency matrix with entries 0 and 1 and zero diagonal, and defined, admissible edit probabilities, the two functions then return a symmetric matrix in which ties are only removed, or only added.\n\nSection 16.4.4, p.~303: ``In this case, \\hl{missing or mismeasured ties can introduce bias, whose magnitude need not increase monotonically with the proportion of missing ties}.''\n\nAs a clarification, the description of Figure 16.5 can read: ``The distribution of point estimates is \\hl{approximately} centered around the target quantity (the dashed line) when the proportion of unmeasured ties is zero\\hl{; the Hajek estimator can still have a small finite-sample bias}.''\n\nIf Figure 16.5 was drawn with either defective function, it must be drawn again with the corrected code. Whether it was is not established: its original inputs and generating code are needed to determine whether its columns 0.025, 0.225 and 0.425 show the effect of random deletion of ties.", "evidence": "The evidence is an exact counterexample for the code, a derivation for the repair, a reading of the printed figure, new simulations, and two exact examples that mark the limits of the conclusion.\n\n\\emph{The code.} Let $A$ be the adjacency matrix of the network on four units whose only tie joins units 1 and 4. Each of the two functions, called with $p=0$, leaves the upper triangle unchanged and returns the matrix $B$:\n\\[\nA=\\begin{pmatrix}0&0&0&1\\\\0&0&0&0\\\\0&0&0&0\\\\1&0&0&0\\end{pmatrix},\\qquad\nB=\\begin{pmatrix}0&0&0&1\\\\0&0&0&0\\\\0&1&0&0\\\\0&0&0&0\\end{pmatrix}.\n\\]\nR lists the upper cells in the order $(1,2),(1,3),(2,3),(1,4),(2,4),(3,4)$ and the lower cells in the order $(2,1),(3,1),(4,1),(3,2),(4,2),(4,3)$. The fourth upper cell, $(1,4)$, is written to the fourth lower cell, $(3,2)$, and not to $(4,1)$. The matrix $B$ is not symmetric: the entry $(3,2)$ was added and the entry $(4,1)$ was removed. The same $B$ is returned through the package's function \\texttt{make\\_adj\\_matrix\\_miss\\_ties} with $p=1/40$ and seed 1, for removal and for addition. With $B$ in place of $A$, the exposure mapping of the package moves unit 1 from $d_{01}$ to $d_{00}$ when unit 4 alone is treated, and it moves unit 2 from $d_{00}$ to $d_{01}$ when unit 3 alone is treated. Over all 64 upper triangles on four units, the printed line returns an asymmetric matrix for 32, and the corrected line returns the symmetric matrix for all 64. On the network of ten units that is printed on p.~295, the tie-removal function with $p=9/40$ and seed 56 removes 4 ties from the upper triangle, and its output differs from the input by 9 added and 17 removed entries.\n\n\\emph{The repair.} Let $U$ be the upper triangle after the random edits, with zeros elsewhere. The corrected line writes $m_{ji}$ to each lower cell $(i,j)$ such that $i>j$, and it changes no cell on or above the diagonal. The result is $U+U^{\\mathsf T}$, which is symmetric. For a symmetric input $A$ with entries 0 and 1 and zero diagonal, and upper cells $i<j$, I have $0\\le U_{ij}\\le A_{ij}$ after a removal, so that no entry is added, and $A_{ij}\\le U_{ij}\\le 1$ after an addition, so that no entry is removed.\n\n\\emph{The printed figure.} Figure 16.5 is a vector drawing, and the heights of its bars give the counts of each histogram; the counts of each of the twelve panels sum to 3000. The third column of the table below is the mean of each histogram minus the position of its dashed line. The bins are about 14.4 wide, so these means are those of the binned estimates.\n\n\\emph{New simulations.} I drew five networks of 400 units and 800 ties with the package's generator and simulated the design of pp.~299 and 303: 40 treated units, dilated effects, Hajek estimators, 10,000 draws for the exposure probabilities and 3000 experiments. For each network and each deletion probability, I measured the network once with the package's tie-removal function and once after mirroring the same random deletion decisions correctly. At a deletion probability of 2.5\\% the tie-removal function deleted between 13 and 21 ties, and its output contained between 766 and 780 entries that were not in the network. The last two columns of the table give the range, over the five networks, of the mean of the defined estimates minus each network's true value. Averaging changes from no missing ties over experiments where both estimates are defined, random deletion at probability 2.5\\% moves $\\widehat{\\tau_H}(d_{01},d_{00})$ by $-0.19$ to $+0.09$, $\\widehat{\\tau_H}(d_{10},d_{00})$ by $+0.04$ to $+0.25$ and $\\widehat{\\tau_H}(d_{11},d_{00})$ by $-0.25$ to $+0.10$. The tie-removal function moves them by $-7.22$ to $-6.29$, by $+2.96$ to $+3.41$ and by $-11.53$ to $-10.45$.\n\\begin{audittable}{llrrr}{Binned histogram mean minus dashed-line position in Figure 16.5, and mean estimate minus each network's true value over five new simulations, in units of the outcome.}\nEstimator & Printed proportion / deletion probability & Figure 16.5 as printed & Tie-removal function & Random deletion \\\\\n$\\widehat{\\tau_H}(d_{01},d_{00})$ & 0 & $+0.260$ & $+0.130$ to $+0.560$ & $+0.130$ to $+0.560$ \\\\\n & 2.5\\% & $-7.008$ & $-6.663$ to $-5.891$ & $+0.045$ to $+0.404$ \\\\\n & 22.5\\% & $-7.026$ & $-6.965$ to $-6.113$ & $-0.841$ to $-0.242$ \\\\\n & 42.5\\% & $-6.256$ & $-7.364$ to $-4.817$ & $-1.154$ to $+0.026$ \\\\\n$\\widehat{\\tau_H}(d_{10},d_{00})$ & 0 & $-0.187$ & $-0.459$ to $+0.731$ & $-0.459$ to $+0.731$ \\\\\n & 2.5\\% & $+2.798$ & $+2.540$ to $+3.699$ & $-0.265$ to $+0.985$ \\\\\n & 22.5\\% & $+3.586$ & $+3.060$ to $+4.429$ & $+1.094$ to $+2.475$ \\\\\n & 42.5\\% & $+3.937$ & $+3.591$ to $+5.117$ & $+2.167$ to $+3.891$ \\\\\n$\\widehat{\\tau_H}(d_{11},d_{00})$ & 0 & $+1.299$ & $+0.861$ to $+2.679$ & $+0.861$ to $+2.679$ \\\\\n & 2.5\\% & $-9.587$ & $-10.671$ to $-7.854$ & $+0.715$ to $+2.739$ \\\\\n & 22.5\\% & $-9.082$ & $-10.742$ to $-7.218$ & $+0.739$ to $+2.895$ \\\\\n & 42.5\\% & $-7.952$ & $-10.716$ to $-5.598$ & $+2.303$ to $+5.096$ \\\\\n\\end{audittable}\n\n\\emph{Limits.} The simulations use new networks and new outcomes, since the network, the outcomes and the script of the printed figure are not available. They do not give the values of a corrected figure. They also do not exclude random deletion as the source of the printed figure. Let 400 units form eighty groups of five, with a tie between every two units of a group. Let the baseline outcome be 250 in two of the groups and 53.2 in the others, so that the mean baseline is 58.12. Suppose that the 20 missing ties, out of 800, are exactly the ties within the two groups with the high baseline. With exact exposure probabilities and conditioning separately on each contrast being defined, an exact calculation yields a bias of $-6.70$ for $\\widehat{\\tau_H}(d_{01},d_{00})$ and of $-10.39$ for $\\widehat{\\tau_H}(d_{11},d_{00})$. Configurations arbitrarily close to this example have positive but very small probability under the stated stochastic design.\n\n\\emph{Bias and the proportion of missing ties.} On the ten-unit network of p.~295, let one unit be treated, chosen uniformly, and let the baseline outcome be $5/2$ for unit 2 and $1$ for the other units, with the multipliers of p.~298. Enumerating every assignment and every set of deleted ties in exact arithmetic, I find that the bias of $\\widehat{\\tau_H}(d_{01},d_{00})$ is $+0.0200$ with no tie deleted, $+0.0091$ with one of the 20 ties deleted at random and $-0.0012$ with two deleted. The absolute bias decreases. This is the total bias of the contrast in one small design: it qualifies the general statement on p.~303 and says nothing about the simulation of Figure 16.5.", "problem": "Section 16.4.4 asks what happens when the measured network misses some ties of the interference network, and Figure 16.5 illustrates this. The package provides a function intended to remove each tie with probability \\texttt{p} and a function that adds ties. Both edit the upper triangle of the adjacency matrix and then copy the upper triangle into the lower triangle with the line above. R lists the cells of each triangle column by column, and in that order the $k$th cell of the upper triangle is in general not the transpose of the $k$th cell of the lower triangle. The matrix that the functions return is in general not symmetric. It can lack entries that were not removed and contain entries that were never in the network. A simulation that removes ties with this function studies the altered network, and not a network from which a random share of the ties is missing.\n\nThe printed figure has the pattern that the altered network produces, although this does not show that it was drawn with the defective function. Reading the histograms from the drawing of the page, I find that the binned mean estimate lies about $7.0$ below the true value $14.53$ of $\\tau(d_{01},d_{00})$ and $9.6$ below the true value $58.12$ of $\\tau(d_{11},d_{00})$ in the column labelled 0.025. The binned gaps are about as large in the columns labelled 0.225 and 0.425. I simulated the stated design on five networks. With the package's tie-removal function the same jump appears at a deletion probability of 2.5\\%. With correctly mirrored random deletion, the average changes in the three contrasts, using experiments where both estimates are defined, are less than $0.3$ in magnitude at 2.5\\%, at most $1.8$ at 22.5\\% and at most $3.2$ at 42.5\\%. The printed binned offsets at 2.5\\% are much larger than any change at that probability in these five simulations. Since the simulations use new networks, and the bins of the figure are about 14.4 wide, this comparison does not measure how far the figure departs from the effect of random deletion on its own network. The script that drew the figure is not in the replication files. The five simulations support the attribution of the figure to the tie-removal function, and they do not establish it; an example below shows that the deletion of particular ties, 2.5\\% of the total, can produce shifts of this size.\n\nThe sentence that opens the section, that the bias of the estimator will increase with the proportion of mismeasured ties, is stated generally, before the figure is introduced, and it needs qualifying: the bias need not increase. In an exact example below, the absolute bias of the Hajek contrast decreases when ties are deleted at random. That example concerns the total bias of the estimator in a small network. It does not show that the simulation of Figure 16.5 behaves otherwise than the chapter describes, or that the part of the bias due to the missing ties decreases.\n\nThe words ``indicating unbiasedness'' describe the centring of the histograms in one figure. Page 299 already says that the Hajek estimator improves efficiency ``with a small cost in terms of finite-sample bias'', and Table 16.1, p.~300 (PDF p.~12), reports biases of $0.32$, $-0.14$ and $1.29$ for these three estimators with no missing ties. Read with that qualification, the words say that the figure shows no appreciable bias, not that the estimator is exactly unbiased. They call for a clarification, below, and are not part of the error.", "what_survives": "The estimands, the exposure conditions and the estimator formulas do not depend on the two functions. The point of the section, that mismeasured ties can bias the estimators, stands. The column of Figure 16.5 with no missing ties is not in question: the package's \\texttt{make\\_adj\\_matrix\\_miss\\_ties} function with $p=0$ and type \\texttt{nothing} returns the network unchanged, and the printed histograms are approximately centered, in agreement with Table 16.1. No other table or figure is shown to depend on the two functions. What changes is the code, the confidence in reading the three other columns of Figure 16.5 as the effect of random deletion at the stated proportions, and the general statement on p.~303 about increasing bias. The words ``indicating unbiasedness'' need only the clarification below."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E012", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Table 16.3 on page 305, 'Comparing unit to cluster randomization', with columns Randomization, Estimator, Estimand, True value, Average value, Bias, SD and RMSE, four rows and the note below the table. Red boxes mark the two Cluster rows (Horvitz-Thompson: 58.12, 57.52, minus 0.60, 61.31, 61.30; Hajek: 58.12, 55.01, minus 3.10, 28.89, 29.05) and the words 'Cluster = three-net clustering.' in the note.", "caption": "Table 16.3, p. 305 (PDF p. 17). The marked rows are the two rows for cluster randomization, and the marked words of the note say that Cluster is three-net clustering.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E013.png"}, {"alt": "A chart of two distributions of the number of clusters in one clustering of the 400 units. The horizontal axis runs from 45 to 200 clusters and the vertical axis gives the share of clusterings in percent, from 0 to 14. The blue distribution, the repaired sampler, is narrow: it lies between 48 and 72 clusters and peaks near 60 at about 14 percent; its mean is 60.4. The orange distribution, the package's sampler as written, peaks near 62 at about 4 percent and has a long right tail that passes 100 and 150 clusters; its mean is 83.8. A note says that 0.9 percent of the clusterings of the package's sampler have more than 200 clusters and that the largest has 326.", "caption": "Number of clusters in one clustering of 400 units, for the package’s sampler as written (orange) and for the repaired sampler (blue): 50,000 clusterings per sampler, on five networks.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E013-cluster-counts.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T10", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T10"}, {"id": "P045-T11", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T11"}, {"id": "P045-T29", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T29"}], "headline": "The package’s 3-net sampler must draw each new center by index from the unmarked units; as written it draws from $1,\\dots,n$ when a single unmarked unit $n$ remains, and in new simulations of the design of Table 16.3 it returns centers closer than three hops in about nine clusterings out of ten.", "id": "P045-E013", "location": "Table 16.3, rows Cluster, and Section 16.4.6, p. 305 (PDF p. 17); the file R/make_tr_vec_bernoulli.R, lines 77 and 123, in the R package interference, version 0.1.0 (code, not printed in the chapter)", "pdf_page": 17, "printed_page": "305", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P045-T10", "P045-T11", "P045-T29"], "text": {"as_printed": "\\begin{audittable}{lllrrrrr}{Table 16.3, p.~305, rows Cluster.}\nRandomization & Estimator & Estimand & True value & Average value & Bias & SD & RMSE \\\\\nCluster & Horvitz--Thompson & $\\tau(d_1,d_0)$ & 58.12 & 57.52 & $-0.60$ & 61.31 & 61.30 \\\\\nCluster & Hajek & $\\tau(d_1,d_0)$ & 58.12 & 55.01 & $-3.10$ & 28.89 & 29.05 \\\\\n\\end{audittable}\nTable 16.3, note: ``Unit = unit-level randomization. Cluster = \\hl{three-net clustering}.''\n\\begin{audittable}{rl}{Package, \\texttt{R/make\\_tr\\_vec\\_bernoulli.R}, lines 76--82. Lines 122--128 repeat the loop.}\nLine & Code \\\\\n76 & \\texttt{while (any(!marked)) \\{} \\\\\n77 & \\texttt{~~v\\_j <- \\hl{sample(which(!marked), 1)}} \\\\\n78 & \\texttt{~~cluster\\_centers[v\\_j] <- j} \\\\\n79 & \\texttt{~~marked[v\\_j]~~<- TRUE} \\\\\n80 & \\texttt{~~marked[unlist(B\\_2[v\\_j])]~~<- TRUE} \\\\\n81 & \\texttt{~~j <- j + 1} \\\\\n82 & \\texttt{\\}} \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{rl}{Package, \\texttt{R/make\\_tr\\_vec\\_bernoulli.R}, line 77, corrected. The same two lines replace line 123.}\nLine & Code \\\\\n77 & \\texttt{~~\\hl{remaining <- which(!marked)}} \\\\\n & \\texttt{~~v\\_j <- \\hl{remaining[sample.int(length(remaining), 1)]}} \\\\\n\\end{audittable}\nFor every finite undirected network, the loop then returns centers that are pairwise at least three hops apart, with every unit within two hops of a center. The treatment assignments have to be drawn again and the exposure probabilities recomputed under the corrected sampler.", "evidence": "The evidence is an exact counterexample, a derivation, and new simulations of the design of Table 16.3.\n\n\\emph{Counterexample.} Let four units form the path $1$--$2$--$3$--$4$, and let each cluster be treated independently with probability $1/2$. A 3-net has either the single center 2, or the single center 3, or the two centers 1 and 4, and the possible treatment vectors are $0000$, $0011$, $1100$ and $1111$. Suppose that the sampler first draws unit 1, which has probability $1/4$. It marks units 1, 2 and 3, and unit 4 alone remains. The next draw is uniform on $\\{1,2,3,4\\}$. A draw of 1 leaves the center set unchanged, and each of 2, 3 and 4 ends the loop, so that each is the last draw with probability $1/3$. If the last draw is 2, the centers are 1 and 2, at distance one, and the clusters are $\\{1\\}$ and $\\{2,3,4\\}$. This partition has probability $1/12$, and its contribution to the probabilities of $1000$ and $0111$ is $1/48$ each. Neither vector is possible under 3-net clustering. Enumerating the sampler in exact arithmetic, I find the probabilities $3/8$, $5/48$, $1/48$, $1/48$, $5/48$ and $3/8$ for $0000$, $0011$, $0111$, $1000$, $1100$ and $1111$ as written, and $3/8$, $1/8$, $0$, $0$, $1/8$ and $3/8$ after the repair. The function, run unchanged from both copies of the package in the replication files using R 4.4.2's default random-number settings and seed 19, returns $1000$ on this path.\n\n\\emph{Derivation.} Suppose that the loop reaches a single unmarked unit $n>1$. Let $S$ be the set of units $v\\le n$ that are within two hops of $n$, with $s=|S|$, and let $c_0$ be the number of earlier centers $v\\le n$. Each later draw is uniform on $\\{1,\\dots,n\\}$. A draw in $S$ marks $n$ and ends the loop, and a draw outside $S$ leaves $n$ unmarked, so the number of draws is geometric with mean $n/s$. Every unit other than $n$ is already marked, and a marked unit that is not a center is within two hops of a center, so that it cannot become a center of a 3-net. A draw of an earlier center only overwrites its label. This implies that the final centers form a 3-net exactly when every draw before the last selects an earlier center and the last draw selects $n$. Summing the geometric series, I obtain\n\\[\n\\Pr(\\text{the final centers form a 3-net})=\\sum_{k=0}^{\\infty}\\Bigl(\\frac{c_0}{n}\\Bigr)^{k}\\frac{1}{n}=\\frac{1}{n-c_0}.\n\\]\nFor the repair, the loop condition makes the vector of unmarked units nonempty, and \\texttt{sample.int} returns a position in that vector, so the new center is unmarked. The marked units are the units within two hops of the earlier centers. Then the new center is at least three hops from each of them. Each pass marks at least the new center, so the loop ends after at most $N$ passes, and at the end every unit is within two hops of a center. This holds for every finite undirected network.\n\n\\emph{New simulations.} I drew five networks of 400 units and mean degree 4 with the package's generator. On each network I ran the sampler as written and the repaired sampler 10,000 times to average the exposure probabilities conditional on each clustering, and I estimated the contrast $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ in 3000 further experiments for each sampler and for unit-level randomization, with the package's dilated outcomes. The two exposure conditions are those of the documentation, a closed neighborhood that is entirely treated or entirely in control (see \\fid{P045-E011}). The first table reports how often the defect occurs and how it changes the design. Over the 50,000 clusterings the sampler as written produced 23.4 more centers on average than the repaired sampler continued from the same state, and 92.7 pairs of centers within two hops of each other.\n\\begin{audittable}{rrrrrrr}{The sampler as written and the repaired sampler on five networks of 400 units, 10,000 clusterings each. Networks are named by the seed of the generator. Columns 2 to 4 and 6 are for the sampler as written. Column 2: share of clusterings that reach a single unmarked unit $n>1$. Column 3: share whose final centers are closer than three hops. Columns 4 and 5: mean number of clusters. Columns 6 and 7: probability that the closed neighborhood of a unit is entirely treated, averaged over units.}\nNetwork & Reached & Not a 3-net & Clusters & Clusters, repaired & Prob. & Prob., repaired \\\\\n357 & 92.50\\% & 90.61\\% & 84.116 & 60.338 & 0.170822 & 0.195066 \\\\\n492 & 93.58\\% & 92.36\\% & 83.503 & 60.829 & 0.168758 & 0.192388 \\\\\n104729 & 92.00\\% & 89.91\\% & 81.824 & 60.266 & 0.168589 & 0.191379 \\\\\n202603 & 93.57\\% & 92.17\\% & 88.622 & 61.576 & 0.168027 & 0.195761 \\\\\n8675309 & 92.09\\% & 90.82\\% & 81.030 & 59.145 & 0.169020 & 0.192765 \\\\\n\\end{audittable}\nThe second table reports the standard deviations of the two estimators. The repair lowers the estimated standard deviation under cluster randomization by 4.41\\% to 10.96\\% for the Horvitz--Thompson estimator and by 6.57\\% to 13.61\\% for the Hajek estimator. Under both samplers the ratio of the standard deviation under cluster randomization to the unit-level value is between 0.28 and 0.70 on every network. For comparison, the printed ratios of the Cluster to the Unit standard deviation in Table 16.3 are $61.31/139.65\\approx0.439$ and $28.89/52.89\\approx0.546$.\n\\begin{audittable}{rlrrr}{Standard deviation of the estimates of $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ in the new simulations. The unit-level value is exact for the Horvitz--Thompson estimator and comes from 3000 experiments for the Hajek estimator; the cluster values come from 3000 experiments. All Hajek estimates were defined.}\nNetwork & Estimator & Unit-level & Cluster, as written & Cluster, repaired \\\\\n357 & Horvitz--Thompson & 116.407 & 73.682 & 67.867 \\\\\n357 & Hajek & 60.069 & 27.605 & 24.681 \\\\\n492 & Horvitz--Thompson & 106.384 & 74.172 & 68.388 \\\\\n492 & Hajek & 55.743 & 29.293 & 25.306 \\\\\n104729 & Horvitz--Thompson & 130.143 & 74.126 & 70.859 \\\\\n104729 & Hajek & 60.962 & 28.160 & 26.309 \\\\\n202603 & Horvitz--Thompson & 274.282 & 86.745 & 77.240 \\\\\n202603 & Hajek & 83.574 & 44.851 & 39.581 \\\\\n8675309 & Horvitz--Thompson & 132.629 & 73.636 & 68.371 \\\\\n8675309 & Hajek & 56.640 & 28.501 & 25.546 \\\\\n\\end{audittable}\nThe networks, the outcomes and the seeds of Table 16.3 are not available. The simulations measure the defect in the design of the table, and they do not give corrected entries for it.", "problem": "Table 16.3 compares unit-level randomization with cluster randomization, where the clusters come from 3-net clustering: the centers are at least three hops apart, every unit is within two hops of a center, each unit joins a nearest center, and treatment is assigned by cluster. The package's sampler builds the centers in a loop. It draws a center from the units that are not yet marked, marks the center and every unit within two hops of it, and repeats until every unit is marked. The draw is \\texttt{sample(which(!marked), 1)}. When exactly one unmarked unit $n$ remains, \\texttt{which(!marked)} is the single number $n$, and R then draws uniformly from $1,\\dots,n$ and not from the one remaining unit. Each draw that misses the units within two hops of $n$ writes a center on a unit that is already marked. Unless that unit is already a center, the new center lies within two hops of an earlier one. The loop continues until a draw lands within two hops of $n$.\n\nThis is not a rare case in new simulations of the design of Table 16.3. In 50,000 clusterings of five networks of 400 units, the loop reached a single unmarked unit $n>1$ in 92.75\\% of the clusterings, and the final centers were closer than three hops in 91.17\\%. The sampler as written returned 83.8 clusters on average, against 60.4 for the repaired sampler, and the estimated probability that the closed neighborhood of a unit is entirely treated, averaged over units, was 0.169, against 0.193. The design that the package draws under the name of 3-net clustering is therefore a different design. If the rows Cluster of Table 16.3 were computed with the package, they are not results for 3-net clustering. The script and the seeds of the table are not in the replication files, and no printed entry is shown to be wrong. In the new simulations the repaired sampler has estimated standard deviations that are lower by 4\\% to 14\\%.", "what_survives": "The rows Unit of Table 16.3 do not use the sampler. The estimand and the unbiasedness of the Horvitz--Thompson estimator with a correctly specified exposure mapping and the actual, positive exposure probabilities are unaffected. In my simulations under both samplers, the estimated standard deviation under cluster randomization is at most 0.70 of the unit-level value on every network. No printed entry is shown to be wrong. The sampler changes; if the package generated the rows Cluster, they have to be computed again before they are read as results for 3-net clustering. The printed definition of $\\epsilon$-net clustering on the same page is the subject of \\fid{P045-E010}."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E013", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "A grid of ten histograms of point estimates, in five columns and two rows labelled 'General interference' and 'Partial interference', each with a dashed vertical line. Red boxes mark the labels above the first two columns, which read tau-hat D(phi) and tau-hat D(psi). In the bottom row the dashed line of the first column is near 0.6 and that of the second column is near 0.4. The caption below ends 'The dashed vertical line represents the value of the estimand.'", "caption": "Figure 16.6, p. 312 (PDF p. 24), with its caption. The labels above the first two columns are marked: the first reads $\\widehat{\\tau}^{D}(\\phi)$ and the second reads $\\widehat{\\tau}^{D}(\\psi)$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E014.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T24", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T24"}], "headline": "The labels of the first two columns of Figure 16.6 are interchanged: the first column shows the direct effect at $\\psi$ and the second shows the direct effect at $\\phi$.", "id": "P045-E014", "location": "Figure 16.6, p. 312 (PDF p. 24), the labels above the first two columns", "pdf_page": 24, "printed_page": "312", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T24"], "text": {"as_printed": "Column labels, from left to right: $\\widehat{\\tau}^{D}(\\hl{\\phi})$, $\\widehat{\\tau}^{D}(\\hl{\\psi})$, $\\widehat{\\tau}^{I}(\\psi,\\phi)$, $\\widehat{\\tau}^{To}(\\psi,\\phi)$, $\\widehat{\\tau}^{O}(\\psi,\\phi)$. The caption ends: The dashed vertical line represents the value of the estimand.", "corrected": "Column labels, from left to right: $\\widehat{\\tau}^{D}(\\hl{\\psi})$, $\\widehat{\\tau}^{D}(\\hl{\\phi})$, $\\widehat{\\tau}^{I}(\\psi,\\phi)$, $\\widehat{\\tau}^{To}(\\psi,\\phi)$, $\\widehat{\\tau}^{O}(\\psi,\\phi)$.", "evidence": "A check of the page against the outcome model stated in the text. The positions of the ten dashed lines were measured from the drawing in the PDF. In the bottom row they are $0.595$, $0.397$, $0.199$, $0.794$ and $0.463$ from left to right. With $b=0.794$, the position of the line for the total effect, these agree to within $0.002$ with $3b/4$, $b/2$, $b/4$, $b$ and $7b/12$, the values of $\\tau^{D}(\\psi)$, $\\tau^{D}(\\phi)$, $\\tau^{I}$, $\\tau^{To}$ and $\\tau^{O}$ under the outcome model. In both rows the line for the total effect minus the line for the indirect effect agrees, within measurement precision, with the line of the first column, as the identity $\\tau^{To}=\\tau^{D}(\\psi)+\\tau^{I}$ requires, and differs from the line of the second column by more than $0.1$. The package also returns the direct effect at $\\psi$ before the direct effect at $\\phi$, which is the order of the output printed on p.~311. The simulation behind the figure was not run again, because its script and seeds are not available.", "problem": "Each dashed line marks the estimand of the estimator named above its column. In the bottom row, the outcome model on p.~310 gives $\\tau^{D}(\\psi)=(2-1.25)\\,b=3b/4$ and $\\tau^{D}(\\phi)=(1.5-1)\\,b=b/2$, where $b$ is the population mean of $y_{ig}(0,\\phi)$. The line for $\\phi$ should therefore stand at two-thirds of the line for $\\psi$. In the figure the line of the first column is at about $0.595$ and the line of the second is at about $0.397$, a ratio of $1.50$. The two labels are in the reverse order. With the printed labels, each direct effect is attached to the wrong saturation, and so is the direction of its bias in the upper row.", "what_survives": "The histograms, the dashed lines and the other three columns are unchanged. So is the lesson of the figure, that the estimators are unbiased under partial interference and biased when interference crosses groups. Only the two labels change."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E014", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the last six lines of the left column of page 303. The text reads 'the data than the coarser, biased estimators (here, tau-hat H of d1, d0). This is apparent if one looks at the standard deviations in Table 16.2. As such, with these sample sizes, the root-mean-square error is swamped by estimation variance relative to bias. As N gets larger, this'. Red boxes mark the words 'swamped by estimation' and, on the next line, 'variance relative to bias.'", "caption": "Section 16.4.3, p. 303 (PDF p. 15), left column, end of the column. The marked words are “swamped by estimation variance relative to bias”.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E015.png"}, {"alt": "Crop of Table 16.2 on page 301, 'Misspecifying exposure conditions', with columns Spillover, Estimand, Estimator, True value, Bias, SD and RMSE and twelve rows. Red boxes mark the Bias and SD entries of the four rows whose estimator is tau-hat H of d1, d0: minus 23.33 and 19.06; minus 32.21 and 21.42; 14.87 and 14.73; 23.92 and 13.65.", "caption": "Table 16.2, p. 301 (PDF p. 13). The marked entries are the Bias and the SD in the four rows with estimator $\\widehat{\\tau_H}(d_1,d_0)$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E015-b.png"}], "formal_release": null, "formal_statements": [], "headline": "At $N=400$ the greater variance of the finer exposure mappings often outweighs their reduction in squared bias; within the four rows of Table 16.2 that ignore interference, the squared bias exceeds the variance.", "id": "P045-E015", "location": "Section 16.4.3, p. 303 (PDF p. 15), left column, the sentence that begins “As such, with these sample sizes”; Table 16.2, p. 301 (PDF p. 13), the four rows with estimator $\\widehat{\\tau_H}(d_1,d_0)$", "pdf_page": 15, "printed_page": "303", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``This is apparent if one looks at the standard deviations in Table 16.2. As such, with these sample sizes, the root-mean-square error is \\hl{swamped by estimation variance relative to bias}. As $N$ gets larger, this would change: the standard deviation would get smaller, but the bias would remain.''", "corrected": "``\\hl{With these sample sizes, the greater variance of finer exposure mappings often outweighs their reduction in squared bias when comparing root-mean-square errors.}''", "intended_meaning": "The sentences before this one compare the finer exposure mappings, which use fewer units and have larger variance, with the coarser mapping, which has more bias. In that context the sentence says that, at $N=400$, the variance added by a finer mapping often outweighs the squared bias that it removes. Table 16.2 supports this reading: the added variance exceeds the reduction in squared bias in seven of the eight comparisons between the estimator that ignores interference and a finer one. The exception is negative spillover with second-degree interference, where the first-degree estimator has the smaller root-mean-square error (22.21 against 27.54). Read as a statement about the two components of each root-mean-square error, the sentence does not hold for the four rows with estimator $\\widehat{\\tau_H}(d_1,d_0)$. There the squared bias exceeds the squared standard deviation: 544.29 against 363.28, 1037.48 against 458.82, 221.12 against 216.97 and 572.17 against 186.32. Each of these inequalities survives the rounding of the printed entries. The entries of the table are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E015", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of five lines from the left column of page 301. The text reads 'where pi_i(d11) = Pr[Z_i I(Z' theta_i > 0) = 1]. Therefore, the misspecified mu-hat HT,None of 1 is biased for mu-hat HT of 1 insofar as y_i(d11) is not equal to y_i(d10) for some i.' Red boxes mark the words 'insofar as y_i(d11) not equal to y_i(d10)' and, on the next line, 'for some i'.", "caption": "Section 16.4.3, p. 301 (PDF p. 13), left column, the sentence after the display for $\\widehat{\\mu_{HT}}(\\mathbf{1})$. The marked words give the condition for bias.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E016.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T13", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T13"}], "headline": "The estimator that ignores interference is biased when a weighted sum of the differences $y_i(d_{10})-y_i(d_{11})$ is nonzero; a difference for some unit $i$ with $\\pi_i(d_{10})>0$ suffices when all the differences have one sign, as they do in the chapter’s examples.", "id": "P045-E016", "location": "Section 16.4.3, p. 301 (PDF p. 13), left column, the sentence after the display for $\\widehat{\\mu_{HT}}(\\mathbf{1})$", "pdf_page": 13, "printed_page": "301", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P045-T13"], "text": {"as_printed": "``Therefore, the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ is biased for $\\widehat{\\mu_{HT}}(\\mathbf{1})$ \\hl{insofar as $y_i(d_{11})\\ne y_i(d_{10})$ for some $i$}.''", "corrected": "\\hl{For a correctly specified first-degree exposure mapping, with $\\pi_i=\\Pr[Z_i=1]>0$ and at least one direct peer for every unit,} the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ \\hl{has bias relative to $\\mu(\\mathbf{1})$}\n\\[\n\\hl{\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\pi_i(d_{10})}{\\pi_i}\\,\\bigl\\{y_i(d_{10})-y_i(d_{11})\\bigr\\}}.\n\\]\n\\hl{This bias is nonzero when the differences $y_i(d_{10})-y_i(d_{11})$ have one sign and at least one unit with $\\pi_i(d_{10})>0$ has a nonzero difference, as in the examples considered here.}", "intended_meaning": "The sentence explains that the estimator that ignores interference averages together the outcomes of treated units with and without a treated direct peer. In the chapter's simulations with dilated effects, the differences $y_i(d_{10})-y_i(d_{11})$ have one sign. In that setting a difference for some unit with $\\pi_i(d_{10})>0$ gives a nonzero bias, and the statement is correct. As a general condition it is incomplete. For a correctly specified first-degree exposure mapping, positive treatment probabilities and at least one direct peer per unit, the bias relative to $\\mu(\\mathbf{1})$ is the weighted sum $N^{-1}\\sum_{i}[\\pi_i(d_{10})/\\pi_i]\\{y_i(d_{10})-y_i(d_{11})\\}$, and differences of opposite sign can cancel. For example, let two units be joined by one tie and be treated independently with probability $1/2$, with $y(d_{11})=(2,1)$, $y(d_{10})=(1,2)$ and outcome zero for an untreated unit. The estimator takes the values $0$, $2$, $1$ and $3$ on the four assignments. Its expectation is $3/2=\\mu(\\mathbf{1})$, although $y_i(d_{11})\\ne y_i(d_{10})$ for both units. The discussion of the simulations on p.~303 concerns spillovers of one sign and needs no change. The denominators of the display before this sentence are the subject of \\fid{P045-E002}."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E016", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the first five lines of the left column of page 297, beginning 'The columns capture pi i, the expected proportion of treatment assignments that result in each exposure'. A red box marks the bold symbol pi sub i.", "caption": "Section 16.4.1, p. 297 (PDF p. 9), first lines of the left column. The marked symbol is $\\boldsymbol{\\pi}_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N001.png"}], "formal_release": null, "formal_statements": [], "headline": "The columns of the printed output are adjusted Monte Carlo estimates of the exposure probabilities, $(C_{ik}+1)/(R+1)$, and each column sums to $34/31$ before rounding.", "id": "P045-N001", "location": "Section 16.4.1, p. 297 (PDF p. 9), first sentence of the page, on the output of make_prob_exposure_cond printed on p. 296 (PDF p. 8)", "pdf_page": 9, "printed_page": "297", "reaches": [], "related_entries": [{"id": "P044-E005", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E005", "work_id": "P044"}], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "The columns capture $\\hl{\\boldsymbol{\\pi}_i}$, the expected proportion of treatment assignments that result in each exposure.", "corrected": "The columns \\hl{contain adjusted Monte Carlo estimates $\\widehat{\\boldsymbol{\\pi}}_i$ of each unit's exposure probabilities: the entry for exposure $d_k$ is $(C_{ik}+1)/(R+1)$, where $C_{ik}$ is the number of the $R$ sampled assignments that place unit $i$ in $d_k$. The four entries of a column sum to $(R+4)/(R+1)$, here $34/31$, and not to one.}", "intended_meaning": "Each column of the output on p.~296 holds estimates $\\widehat{\\boldsymbol{\\pi}}_i$ of the four exposure probabilities of one unit, computed from the $R=30$ sampled assignments. Page 296 says that the example works with 30 random draws from $\\Omega$, and pp.~297--298 distinguish the estimates $\\widehat\\pi_i(d_k)$ from the true values $\\pi_i(d_k)$, so that this meaning is clear and correct. The chapter does not say that the package adds one to each count and to the number of draws. The entry for unit $i$ and exposure $d_k$ is $(C_{ik}+1)/(R+1)$, where $C_{ik}$ is the number of sampled assignments that place unit $i$ in $d_k$. Since the four counts of a unit sum to $R$, I find that each column sums to $(R+4)/(R+1)=34/31$ before rounding, and not to one. For unit 1 the unrounded entries are $(3,6,13,12)/31$, the sample proportions are $(2,5,12,11)/30$, and the exact probabilities over the 45 assignments are $(3,6,21,15)/45$. The contrast $41.61043$ on p.~298 is reproduced from these estimates, and no reported number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N001", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines from the right column of page 299. The text reads 'ance is with little cost in bias, as indicated by a smaller root-mean-square error. Moreover, the variance estimator is consistent, given that the mean of the standard error estimate approaches the empirical standard deviation from the simulation.' A red box marks the line 'the variance estimator is consistent, given'.", "caption": "Section 16.4.2, p. 299 (PDF p. 11), right column, last sentence of the section. The marked line draws the conclusion of consistency.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N002.png"}, {"alt": "Crop of Table 16.1 on page 300, 'Comparing Horvitz-Thompson to Hajek estimators using approximate exposure probabilities', with columns Estimator, Estimand, True value, Average value, Bias, SD, RMSE and Mean SE, six rows and the note below the table. Red boxes mark four entries in the two rows for tau(d10, d00): SD 26.60 and Mean SE 33.11 for Horvitz-Thompson, and SD 21.55 and Mean SE 25.09 for Hajek.", "caption": "Table 16.1, p. 300 (PDF p. 12). The marked entries are the SD and the Mean SE for $\\tau(d_{10},d_{00})$, for the Horvitz–Thompson estimator and for the Hajek estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N002-b.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T22", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T22"}, {"id": "P045-T23", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T23"}], "headline": "Table 16.1 does not show that the variance estimator is consistent: at $N=400$ the mean standard error is within 7.1 percent of the empirical standard deviation for two contrasts and exceeds it by 16.4 and 24.5 percent for the third.", "id": "P045-N002", "location": "Section 16.4.2, p. 299 (PDF p. 11), right column, last sentence; Table 16.1, p. 300 (PDF p. 12), columns SD and Mean SE", "pdf_page": 11, "printed_page": "299", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P045-T22", "P045-T23"], "text": {"as_printed": "``Moreover, \\hl{the variance estimator is consistent, given that} the mean of the standard error estimate approaches the empirical standard deviation from the simulation.''", "corrected": "``\\hl{At $N=400$, the mean standard error estimates are within 7.1 percent of the empirical standard deviations for $\\tau(d_{11},d_{00})$ and $\\tau(d_{01},d_{00})$, and exceed them by 16.4 percent for Hajek and 24.5 percent for Horvitz--Thompson for $\\tau(d_{10},d_{00})$. These finite-sample comparisons do not establish asymptotic consistency.}''", "evidence": "The evidence is a recalculation from Table 16.1 and an exact counterexample to the inference. The ratios of the printed entries are in the table below. For $\\tau(d_{10},d_{00})$ they are $33.11/26.60=473/380$ and $25.09/21.55=2509/2155$, and both remain more than 16 percent above one when each entry is allowed to vary by its rounding error. The other four ratios are within 7.1 percent of one.\n\\begin{audittable}{llrrr}{Table 16.1, p.~300: the mean standard error estimate against the empirical standard deviation.}\nEstimator & Estimand & SD & Mean SE & Mean SE relative to SD \\\\\nHorvitz--Thompson & $\\tau(d_{11},d_{00})$ & 45.32 & 45.94 & $+1.4\\%$ \\\\\nHorvitz--Thompson & $\\tau(d_{10},d_{00})$ & 26.60 & 33.11 & $+24.5\\%$ \\\\\nHorvitz--Thompson & $\\tau(d_{01},d_{00})$ & 10.45 & 10.33 & $-1.1\\%$ \\\\\nHajek & $\\tau(d_{11},d_{00})$ & 36.66 & 34.07 & $-7.1\\%$ \\\\\nHajek & $\\tau(d_{10},d_{00})$ & 21.55 & 25.09 & $+16.4\\%$ \\\\\nHajek & $\\tau(d_{01},d_{00})$ & 8.38 & 8.41 & $+0.4\\%$ \\\\\n\\end{audittable}\nFor the counterexample, let $X_1,\\dots,X_N$ be independent signs, each equal to $1$ or $-1$ with probability $1/2$, and let $T_N=N^{-1}\\sum_{i=1}^N X_i$ estimate $\\theta=0$. Then $T_N$ is consistent and $V_N=\\operatorname{Var}(T_N)=1/N$. Define $B=1/2$ if $X_1=-1$ and $B=3/2$ if $X_1=1$, and define the variance estimate $\\widehat V_N=B^2/N$. Since $\\mathbb{E}[B]=1$, I have\n\\[\n\\mathbb{E}\\Bigl[\\sqrt{\\widehat V_N}\\Bigr]=\\frac{1}{\\sqrt N}=\\sqrt{V_N}\\qquad\\text{for every } N,\n\\]\nso that the mean standard error equals the true standard deviation at every sample size. The estimate is also conservative in expectation, since $\\mathbb{E}[\\widehat V_N]=5/(4N)\\ge V_N$. But $\\widehat V_N/V_N=B^2$ takes the values $1/4$ and $9/4$ with probability $1/2$ each. This implies that\n\\[\n\\Pr\\bigl(\\,\\bigl|\\widehat V_N/V_N-1\\bigr|>1/2\\,\\bigr)=1\\qquad\\text{for every } N,\n\\]\nand the ratio does not tend to one. Note that in this example the difference $\\widehat V_N-V_N$ does tend to zero, since $|\\widehat V_N-V_N|\\le 5/(4N)$. The counterexample concerns the inference only. It is not one of the chapter's variance estimators.", "problem": "The sentence infers that the variance estimator is consistent from the fact that, in Table 16.1, the mean of the standard error estimates is close to the empirical standard deviation of the point estimates. The conclusion does not follow from the table. Table 16.1 reports one design with $N=400$, so it shows nothing about convergence as $N$ grows. The two columns also disagree for one of the three contrasts: for $\\tau(d_{10},d_{00})$ the mean standard error exceeds the empirical standard deviation by 24.5 percent for the Horvitz--Thompson estimator and by 16.4 percent for the Hajek estimator. Finally, agreement between the mean standard error and the standard deviation does not imply consistency, even if it holds at every sample size. The property that matters for a sampling distribution whose scale shrinks is that the ratio of the variance estimate to the true variance tends to one in probability, and the mean of the standard errors does not control that ratio. This finding concerns the inference in the sentence. It does not show that either variance estimator is inconsistent.", "what_survives": "All six rows of Table 16.1 stand, and so does the comparison that the table supports: the Hajek estimator has the smaller standard deviation and the smaller root-mean-square error for each contrast. No later result is derived from the sentence. Whether the two variance estimators are consistent under suitable asymptotic assumptions is not settled either way by this finding, and the large-$N$ confidence interval on p.~299 rests on separate theory."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N002", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "A block of R code and output in typewriter type. The first line reads 'estimates <- estimates_hierarchical(post_tr_data)'. The output lists five point estimates and then five variances. A red box marks the entries 'var_direct_psi_hat 0.71255260' and 'var_direct_phi_hat 0.04854851'. The other variances are var_indirect_hat 0.28000088, var_total_hat 0.88936810 and var_overall_hat 0.39005445.", "caption": "Toy example, p. 311 (PDF p. 23): the call to estimates_hierarchical and its output, with the two direct-effect variance entries marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N008.png"}, {"alt": "Twelve lines of text from the right column. Red boxes mark the words '18 units equally divided into' at the end of the second line and 'six groups' at the start of the third. The following lines say that two-thirds of units in groups with saturation psi are assigned to the treatment condition and one-third to the control condition, and one-third of units in groups with saturation phi are assigned to treatment.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), right column: the design of the toy example, with “18 units equally divided into six groups” marked. Each group has three units, two in one arm and one in the other.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N008-p310.png"}, {"alt": "Two panels, each with five rows and a horizontal axis of variance. Blue vertical ticks mark the true variance and orange dots the expected value of the reported variance. Left panel, constant additive effect: for the direct effects at psi and at phi the dot is at 0.33 and the tick at 0.50, labelled 'ratio 2/3'; for the indirect, total and overall effects the dot lies on the tick, at 0.28, 0.11 and 0. Right panel, outcome model of the example: for the direct effect at psi the dot is at 0.28 and the tick at 0.34, labelled 'ratio 0.80'; for the direct effect at phi the dot is at 0.16 and the tick at 0.27, labelled 'ratio 0.60'; for the indirect, total and overall effects the dot is at or slightly to the right of the tick, near 0.24, 0.30 and 0.21.", "caption": "True variance of each point estimator and expected value of the variance that estimates_hierarchical reports, over the 14,580 assignments of the toy design. Left: every group has baseline outcomes $(1,2,3)$ and a constant additive effect $d=1$. Right: the outcome model of the example, on data consistent with the printed output. The ratio is the expected reported variance divided by the true variance. The values are exact sums over the assignments.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N008-variance.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T3", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T3"}, {"id": "P045-T4", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T4"}, {"id": "P045-T26", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T26"}], "headline": "In the toy example every group has an arm with one unit, so the variance estimator for a direct effect that p. 310 describes is undefined; the two direct-effect variances in the output omit the undefined terms and are too small in expectation.", "id": "P045-N008", "location": "Section 16.5.1, toy example, p. 311 (PDF p. 23), the two direct-effect variances in the printed output; with the description of the variance estimators and of the design on p. 310 (PDF p. 22)", "pdf_page": 23, "printed_page": "311", "reaches": [{"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013"}, {"effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044"}], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P045-T26", "P045-T3", "P045-T4"], "text": {"as_printed": "\\emph{p.~310.} When stratified interference holds, their proposed variance estimators are unbiased if unit causal effects are additive (e.g., $y_{ig}(1,\\psi)=y_{ig}(0,\\psi)+d$, where $d$ is constant) and otherwise positively biased. \\ldots\\ Suppose we have 18 units equally divided into six groups. In the first stage, half of the groups are assigned to saturation $\\psi$ and the other half to $\\phi$ with equal probability. In the second stage, using complete random assignment, two-thirds of units in groups with saturation $\\psi$ are assigned to the treatment condition and one-third to the control condition, and one-third of units in groups with saturation $\\phi$ are assigned to treatment, while two-thirds are assigned to control.\n\n\\emph{p.~311.} The estimators and their variance are computed under the stratified interference assumption with the following function from the \\texttt{interference} package: \\ldots\n\n\\begin{audittable}{lr}{The variance output of \\texttt{estimates\\_hierarchical(post\\_tr\\_data)} on p.~311, as printed.}\nEntry & Value \\\\\n\\texttt{var\\_direct\\_psi\\_hat} & \\hl{\\texttt{0.71255260}} \\\\\n\\texttt{var\\_direct\\_phi\\_hat} & \\hl{\\texttt{0.04854851}} \\\\\n\\texttt{var\\_indirect\\_hat} & \\texttt{0.28000088} \\\\\n\\texttt{var\\_total\\_hat} & \\texttt{0.88936810} \\\\\n\\texttt{var\\_overall\\_hat} & \\texttt{0.39005445} \\\\\n\\end{audittable}", "corrected": "\\emph{p.~310.} \\hl{Under stratified interference and the two-stage randomization specified here, the proposed variance estimator for a direct effect is unbiased exactly when unit causal effects are constant within each group; the constants may differ across groups. A common additive effect} (e.g., $y_{ig}(1,\\psi)=y_{ig}(0,\\psi)+d$, where $d$ is constant) \\hl{is one such case. Otherwise its bias is positive. The first stage chooses a fixed number of groups uniformly for each saturation. For each fixed saturation, the complete random assignments within groups are independent across groups and independent of which groups are selected for that saturation. The variance estimator for a direct effect uses the sample variance of the outcomes in each arm of each group. It requires at least two treated and at least two control units in every group at that saturation, that is, $2\\le K_g\\le n_g-2$ where $K_g$ is the number of treated units in group $g$, and at least two groups at each saturation.}\n\n\\emph{p.~311.} \\hl{The two direct-effect variances are not available in this design, because every group has an arm with one unit.}\n\n\\begin{audittable}{lr}{The variance output on p.~311, corrected.}\nEntry & Value \\\\\n\\texttt{var\\_direct\\_psi\\_hat} & \\hl{\\texttt{NA}} \\\\\n\\texttt{var\\_direct\\_phi\\_hat} & \\hl{\\texttt{NA}} \\\\\n\\texttt{var\\_indirect\\_hat} & \\texttt{0.28000088} \\\\\n\\texttt{var\\_total\\_hat} & \\texttt{0.88936810} \\\\\n\\texttt{var\\_overall\\_hat} & \\texttt{0.39005445} \\\\\n\\end{audittable}\n\nIn the package, \\texttt{estimates\\_hierarchical} should return \\texttt{NA} with a warning when a sample variance that the formula needs is undefined. If numerical direct-effect variances are wanted in the example, use the uniform and independent randomization specified above with six groups of six units, treating four units per group at one saturation and two at the other. This keeps both saturations and has \\hl{at least} two units in every arm. The printed numbers would then change.", "evidence": "I checked the package output on data consistent with the example, an exact counterexample and a derivation of the bias.\n\n\\emph{The printed output.} Data were found that match the six printed rows and all ten printed outputs to every printed digit. On these data both versions of \\texttt{estimates\\_hierarchical} that I examined return the printed output with no warning. The six arms with one unit have undefined sample variance. The two direct-effect variances in the output are $0.695937+0.016616=0.712553$ and $0.012457+0.036092=0.048549$, where the first term is the part between groups and the second is the part within groups from the arms with two units.\n\n\\emph{The counterexample.} In each of the six groups let the units have $y_{ig}(0;s)=x_i$ and $y_{ig}(1;s)=x_i+1$ at both saturations $s$, with $(x_1,x_2,x_3)=(1,2,3)$. Unit effects are constant and additive with $d=1$, and partial and stratified interference hold. The design has $20\\cdot 3^{6}=14{,}580$ equally likely assignments. Summing over them in exact fractions, I obtain the following values.\n\n\\begin{audittable}{lcc}{Constant additive effect $d=1$, baseline outcomes $(1,2,3)$ in every group: true variance of each estimator and expected value of the reported variance.}\nEstimator & True variance & Expected reported variance \\\\\nDirect effect at $\\psi$ & $1/2$ & $1/3$ \\\\\nDirect effect at $\\phi$ & $1/2$ & $1/3$ \\\\\nIndirect effect & $5/18$ & $5/18$ \\\\\nTotal effect & $1/9$ & $1/9$ \\\\\nOverall effect & $0$ & $0$ \\\\\n\\end{audittable}\n\n\\emph{The bias.} Fix a saturation, and let $G$ be the number of groups. At the first stage, a fixed number of at least two groups is chosen uniformly for this saturation. The complete random assignments within groups at this saturation are independent across groups and independent of the first-stage selection. In group $g$, with $n$ units of which $m$ are treated, let $S^{2}_{a,g}$, $S^{2}_{b,g}$ and $S^{2}_{\\delta,g}$ be the variances, with divisor $n-1$, of the potential outcomes under treatment, of the potential outcomes under control and of the unit effects at this saturation. The difference in means in group $g$ has variance\n\\[\nV_g=\\frac{S^{2}_{a,g}}{m}+\\frac{S^{2}_{b,g}}{n-m}-\\frac{S^{2}_{\\delta,g}}{n}.\n\\]\nIf the within-group term of the variance estimator has expectation $B_g$ in group $g$, the estimator has bias $G^{-2}\\sum_{g}(B_g-V_g)$. When both arms have at least two units, $B_g=S^{2}_{a,g}/m+S^{2}_{b,g}/(n-m)$, and the bias is $G^{-2}\\sum_{g}S^{2}_{\\delta,g}/n$. Under this design the bias is nonnegative, and it is zero exactly when unit effects are constant within every group; the constants may differ across groups. Otherwise it is positive. In the toy design $G=6$ and $n=3$, with $m=2$ at $\\psi$ and $m=1$ at $\\phi$. When the term of the arm with one unit is dropped, $B_g=S^{2}_{a,g}/2$ at $\\psi$ and $B_g=S^{2}_{b,g}/2$ at $\\phi$, and I obtain\n\\[\n\\operatorname{Bias}_{\\psi}=\\frac{1}{G^{2}}\\sum_{g=1}^{G}\\Bigl(\\frac{S^{2}_{\\delta,g}}{3}-S^{2}_{b,g}\\Bigr),\n\\qquad\n\\operatorname{Bias}_{\\phi}=\\frac{1}{G^{2}}\\sum_{g=1}^{G}\\Bigl(\\frac{S^{2}_{\\delta,g}}{3}-S^{2}_{a,g}\\Bigr).\n\\]\nUnder constant effects $S^{2}_{\\delta,g}=0$; the bias at a saturation is negative if outcomes vary within any group at that saturation. Under the outcome model of the example, with $S^{2}_{x,g}$ the variance of the baseline outcomes in group $g$, the two biases are $-(11/8)\\,G^{-2}\\sum_{g}S^{2}_{x,g}$ and $-(13/6)\\,G^{-2}\\sum_{g}S^{2}_{x,g}$. On the data consistent with the printed output, the expected reported variances are $0.276$ and $0.161$, and the true variances are $0.344$ and $0.267$; the ratios are $0.80$ and $0.60$. In the other direction, if $y_{ig}(0;s)=0$ and $y_{ig}(1;s)=x_i$ with $(x_1,x_2,x_3)=(1,2,3)$ in every group, the expected reported variance at $\\psi$ is $1/9$ and the true variance is $1/18$.", "problem": "The variance estimator that Hudgens and Halloran (2008) propose for a direct effect, and that p.~310 describes, uses the sample variance of the observed outcomes in each arm of each group, and a sample variance needs at least two observations. In the toy design each group has three units, two in one arm and one in the other. One of the two sample variances is then undefined in every group. The function \\texttt{estimates\\_hierarchical} sums the terms that are defined and drops the others without a warning. In each of the two direct-effect variances in the output, the part within groups therefore contains only the terms from the arms with two units. This quantity is not the estimator described, and it does not have the property stated on p.~310: in this design its expectation can fall below the true variance even with constant additive effects, and it does so under the outcome model of the example. The statement concerns the expectation over assignments and not the two printed numbers themselves. With other potential outcomes the omission can also bias the reported variance upward.", "what_survives": "The five point estimates are unaffected, since an arm mean is defined for an arm with one unit. The variances of the indirect, total and overall effects use sample variances across the three groups at each saturation and are unaffected. Figure 16.6 shows point estimates only, in a design with 75 units per group and arms of 50 and 25. The direct-effect variance guarantee holds under stratified interference when a fixed number of at least two groups is selected uniformly for each saturation, the within-group complete random assignments at a fixed saturation are independent across groups and independent of the first-stage selection, and every arm has at least two units. Under these conditions the bias is zero exactly when unit effects are constant within each group, with possibly different constants across groups, and otherwise positive. What changes is the two direct-effect variances of the example and, for a user of the function, the variance reported for a direct effect in any design that has an arm with one unit."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N008", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Nine lines of text from the right column. A red box marks the word 'municipalities' in the sentence 'In this case, only the first stage is randomized, assigning municipalities to treatment or control, while in the second stage a subset of individuals are offered treatment based on their income.'", "caption": "Section 16.5.3, p. 313 (PDF p. 25), right column: the description of the Progresa design, with “municipalities” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "The Progresa evaluation analyzed by Angelucci and De Giorgi (2009) randomized villages, not municipalities.", "id": "P045-N009", "location": "Section 16.5.3, p. 313 (PDF p. 25), right column, the sentences on Angelucci and De Giorgi (2009)", "pdf_page": 25, "printed_page": "313", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Similarly, Angelucci and De Giorgi (2009) analyze the indirect effects of the cash transfer program Progresa on consumption. In this case, only the first stage is randomized, assigning \\hl{municipalities} to treatment or control, while in the second stage a subset of individuals are offered treatment based on their income.", "corrected": "Similarly, Angelucci and De Giorgi (2009) analyze the indirect effects of the cash transfer program Progresa on consumption. In this case, only the first stage is randomized, assigning \\hl{villages (localities)} to treatment or control, while in the second stage a subset of individuals are offered treatment based on their income.", "evidence": "A check against the cited study. The abstract of the published article (\\emph{American Economic Review} 99(1): 486--508) says that the authors ``use a village-level randomization''. The version of April 2006 (UCL Discussion Papers in Economics 06-01) states that ``The randomization is conducted at the village level, with 186 villages randomized out'', and its Figure~1 shows 506 rural villages, 320 randomized in and 186 randomized out. The design section of the published article was not read, and the counts are those of the 2006 version.", "problem": "The unit of randomization is misnamed. In the cited study the randomization is at the level of the village (locality): of 506 rural villages, 320 were randomized in and 186 were randomized out. In the 2006 version of the study, municipalities appear in the description of the wider program, not as the unit that was randomized.", "what_survives": "The point of the example is unaffected: geographic groups are randomized in the first stage, and treatment within them follows eligibility by income. The chapter's conclusion about what this design identifies does not depend on the name of the unit."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N009", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Two entries of the reference list, four lines each. A red box marks 'arXiv:1709.0963v1' at the end of the third line of the entry 'Taylor, Sean J., and Dean Eckles. 2017. Randomized Experiments to Detect and Estimate Social Influence in Networks.' The next entry is Tchetgen and VanderWeele 2012.", "caption": "References, p. 319 (PDF p. 31), left column: the entry for Taylor and Eckles (2017), with the arXiv identifier marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N010.png"}], "formal_release": null, "formal_statements": [], "headline": "The arXiv identifier in the reference to Taylor and Eckles (2017) is malformed: it has four digits after the point, and identifiers of September 2017 have five.", "id": "P045-N010", "location": "References, p. 319 (PDF p. 31), left column, the entry for Taylor and Eckles (2017)", "pdf_page": 31, "printed_page": "319", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Taylor, Sean J., and Dean Eckles. 2017. ``Randomized Experiments to Detect and Estimate Social Influence in Networks.'' \\hl{\\emph{arXiv:1709.0963v1}} \\emph{[cs.SI]}.", "corrected": "Taylor, Sean J., and Dean Eckles. 2017. ``Randomized Experiments to Detect and Estimate Social Influence in Networks.'' \\emph{arXiv:1709.0963\\hl{6}v1} \\emph{[cs.SI]}.", "evidence": "I checked the printed identifier against \\href{https://info.arxiv.org/help/arxiv_identifier.html}{arXiv's identifier format}. The chapter's other identifiers of this period, 1809.09561 and 1711.06399, have five digits after the point. I then compared the authors, title, year and version with \\href{https://arxiv.org/abs/1709.09636v1}{arXiv's entry}, which gives 1709.09636v1 [cs.SI]. I did not test whether the malformed string resolves on arXiv.", "problem": "An arXiv identifier issued from January 2015 on has the form YYMM.NNNNN, with five digits after the point and an optional version suffix. The printed identifier has four digits after the point and is not a valid identifier.", "what_survives": "The authors, title and year identify the work, and the chapter's discussion of it is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N010", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": {"class": "error", "severity": "Moderate"}, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the foot of page 298, in two columns. The left column has ten lines that begin 'The estimator tau-hat HT of d_k, d_l is unbiased when it is estimated with pi_i(d_k) rather than pi-hat_i(d_k)'; a red box marks its last line, 'These are conservative approximations of'. The right column continues 'the exact variances that are guaranteed to have nonnegative bias relative to the variance of the randomization distribution of the estimators'; red boxes mark the words 'guaranteed to' at the end of its first line and 'have nonnegative bias' at the start of its second. The right column goes on with 'Because unbiased estimators for Var of y-hat-T HT of d_k are only identified when the joint exposure probabilities of every pair of units are positive' and ends with 'because the Cov of the two estimated totals is always'.", "caption": "Section 16.4.2, p. 298 (PDF p. 10), last paragraph, in both columns. The sentence begins on the marked last line of the left column, “These are conservative approximations of”, and continues at the top of the right column, where “guaranteed to have nonnegative bias” is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016.png"}, {"alt": "Crop of five lines from the top of the left column of page 299. The text reads 'unidentified, we use an approximation. Both of these corrections contribute to the nonnegative bias of the variance estimator. We also implement the constant effects variance estimator derived in Aronow'. Red boxes mark the two lines of the sentence 'Both of these corrections contribute to the nonnegative bias of the variance estimator.'", "caption": "Section 16.4.2, p. 299 (PDF p. 11), top of the left column: the end of the paragraph. The marked sentence says that the two corrections contribute to the nonnegative bias of the variance estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016-p299.png"}, {"alt": "Crop of six lines from the right column of page 297. The text reads 'In cases where the size of Omega is high, and therefore we use sampling from Omega to obtain estimates pi-hat_i, those estimates are used in place of the true pi_i values. A Horvitz-Thompson estimator of the average unit-level causal effect of exposure k versus l, tau(d_k, d_l), is therefore'. Red boxes mark the words 'those estimates are used in place of the true' and, on the next line, 'pi_i values.'", "caption": "Section 16.4.2, p. 297 (PDF p. 9), right column, below the display of the Horvitz–Thompson estimator of a total. The marked words say that estimated exposure probabilities are used in place of the true ones.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016-p297.png"}, {"alt": "A chart titled 'With simulated exposure probabilities, the expectation of the variance estimator is below the variance that it estimates', with four rows and a horizontal axis of variance from 0.40 to 0.56. Blue vertical ticks mark the variance of the Horvitz-Thompson estimator of the contrast and orange dots mark the expectation of its variance estimator; a grey line joins the two. Row 'Simulated; the 66 samples equally likely': dot at 0.447, tick at 0.488, label 'bias minus 0.0406'. Row 'Simulated; the omitted assignment is within one clique (18 samples)': dot at 0.421, tick at 0.487, label 'bias minus 0.0660'. Row 'Simulated; the omitted assignment is across two cliques (48 samples)': dot at 0.457, tick at 0.488, label 'bias minus 0.0309'. Row 'Exact probabilities': the dot lies on the tick at 0.508, label 'no bias'.", "caption": "The finite-sample counterexample of this note: twelve units in three disjoint cliques of four, two units treated, all potential outcomes equal to 1, contrast $(d_{01},d_{00})$, exposure probabilities simulated from 65 of the 66 assignments. In each row the tick is the variance of the Horvitz–Thompson estimator of the contrast, the dot is the expectation of its variance estimator, and the label is the expectation minus the variance. In the first row the moments are over the sample of 65 assignments and the realized assignment. In the second and third rows the sample is fixed, and the assignment that it omits treats two units of one clique or one unit in each of two cliques. The values are exact fractions from the enumeration, shown as decimals. The horizontal axis starts at 0.40.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016-variance-bias.png"}], "formal_release": null, "formal_statements": [{"id": "P045-T32", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T32"}, {"id": "P045-T33", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T33"}], "headline": "The guarantee is exact with true marginal and joint exposure probabilities. The context clearly treats Monte Carlo probabilities as numerical approximations; the guarantee should state this scope explicitly.", "id": "P045-N016", "location": "Section 16.4.2, last paragraph of p. 298 (PDF p. 10): the sentence that begins “These are conservative approximations” on the last line of the left column and continues in the right column; and the sentence “Both of these corrections …” at the top of p. 299 (PDF p. 11)", "pdf_page": 10, "printed_page": "298", "reaches": [], "related_entries": [{"id": "P044-N005", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N005", "work_id": "P044"}], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P045-T32", "P045-T33"], "text": {"as_printed": "The estimator $\\widehat{\\tau}_{HT}(d_k,d_l)$ is unbiased when it is estimated with $\\pi_i(d_k)$ rather than $\\widehat{\\pi}_i(d_k)$. As mentioned above, when estimated with the latter, the estimator is not unbiased, but the bias becomes negligible with a sufficiently large number of replicates $R$. We implement variance estimators for $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ and $\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]$ as derived in equation 11 of Aronow and Samii (2017). These are conservative approximations of the exact variances that are \\hl{guaranteed to have nonnegative bias} relative to the variance of the randomization distribution of the estimators. Because unbiased estimators for $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ are only identified when the joint exposure probabilities of every pair of units are positive, it is necessary to add a correction term to $\\widehat{\\operatorname{Var}}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$. Second, because the $\\operatorname{Cov}\\bigl[\\widehat{y^T}_{HT}(d_k),\\widehat{y^T}_{HT}(d_l)\\bigr]$ is always unidentified, we use an approximation. Both of these corrections contribute to \\hl{the nonnegative bias} of the variance estimator.", "corrected": "The variance estimators for $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ and $\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]$ are guaranteed to have nonnegative bias \\hl{with the exact marginal and joint exposure probabilities of the design}, provided that $\\pi_i(d_k)>0$ and $\\pi_i(d_l)>0$ for every unit $i$. With these probabilities,\n\\[\n\\begin{aligned}\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]\\Bigr]&\\;\\ge\\;\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr],\\\\\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]\\Bigr]&\\;\\ge\\;\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)].\n\\end{aligned}\n\\]\n\\hl{With exact probabilities,} both corrections for unidentified outcome products contribute to the nonnegative bias. Using estimated exposure probabilities approximates this result when Monte Carlo error is small; \\hl{a finite number $R$ of simulated assignments need not preserve the exact nonnegative-bias guarantee}.", "intended_meaning": "A variance estimator has nonnegative bias if its expectation over the randomization is at least the variance that it estimates. The guarantee holds when the variance estimators are computed with the exact marginal and joint exposure probabilities of the design and every unit has positive probability of the exposures concerned. This is the natural intended meaning. Page~294 explains that Monte Carlo probabilities approximate the exact probabilities with arbitrary precision and discusses keeping their bias small. Immediately before the guarantee on p.~298, the chapter distinguishes $\\pi_i(d_k)$ from $\\widehat\\pi_i(d_k)$ and says that the point estimator's finite-$R$ bias becomes negligible with sufficiently many replicates. Read together, these passages describe the exact-probability result and an accurate numerical approximation to it.\n\nThe examples do use simulated exposure probabilities: 30 assignments drawn without replacement from 45 in the running example (p.~296), and $R=10{,}000$ assignments in Section 16.4.2 (p.~299). Page~297 says that the estimates replace the true probabilities. These finite calculations need not preserve an exact sign guarantee. The example below shows that the expected variance estimate can be below the variance of the contrast computed with those same simulated probabilities. It does not test the numerical accuracy of the chapter's $R=10{,}000$ calculations.\n\nThe distinction follows from the expectation of a product of exposure indicators, which is the exact joint exposure probability. With that probability in the denominator of a positive-joint-probability term, the two cancel. A simulated denominator does not give this cancellation. The corrections for zero joint probabilities control the missing outcome products, not the Monte Carlo error. Adding the exact-probability restriction to the guarantee and to the following sentence about the corrections would make their intended scope explicit. The related qualification for the chapter ``Estimating Causal Effects in Experiments with Network Spillovers'' is listed as \\fid{P044-N005}.\n\nThe evidence is an exact counterexample, a check of the counterexample against the package, and a derivation for exact exposure probabilities.\n\n\\emph{The counterexample.} Twelve units form three disjoint cliques of four: two distinct units are adjacent exactly when they belong to the same clique. Two units are treated, and the 66 assignments are equally likely. The exposure mapping is the chapter's. A unit has exposure $d_{11}$ if it and a neighbor are treated, $d_{10}$ if it is treated and no neighbor is, $d_{01}$ if it is untreated and a neighbor is treated, and $d_{00}$ otherwise. Every potential outcome equals 1, so that the exposure mapping fully characterizes interference and $\\tau(d_{01},d_{00})=0$. For every unit, the exact probabilities of $d_{11}$, $d_{10}$, $d_{01}$ and $d_{00}$ are $3/66$, $8/66$, $27/66$ and $28/66$. The exposure probabilities are simulated from a sample of $R=65$ distinct assignments, drawn uniformly without replacement from the 66 and independently of the realized assignment. The example on p.~296 also draws its 30 assignments without replacement, from 45. There are 66 such samples, one for each omitted assignment.\n\nFix two exposures, and let $c_{ij}$ be the number of sampled assignments under which unit $i$ has the first exposure and unit $j$ has the second. The package estimates the joint probability by $(c_{ij}+\\mathbf{1}\\{i=j\\})/(R+1)$ when the two exposures are the same and by $c_{ij}/R$ when they differ. The marginal estimates are the diagonal entries in the first case (see \\fid{P045-N001}). In the counterexample the simulated probability of $d_{01}$ is $27/66$ for a unit that has exposure $d_{01}$ under the omitted assignment and $28/66$ for the other units. The simulated probability of $d_{00}$ is $28/66$ for a unit that has exposure $d_{00}$ under the omitted assignment and $29/66$ for the other units. For the two contrasted exposures, a simulated joint probability is zero exactly when the exact joint probability is zero. The failure does not come from a pair of exposures that is possible but absent from the sample.\n\nI enumerated the $66\\times 66=4356$ pairs of a sample and a realized assignment in exact rational arithmetic. For each pair I computed $\\widehat{\\tau}_{HT}(d_{01},d_{00})$ and the variance estimator displayed below, with the simulated probabilities in place of the exact ones.\n\n\\begin{audittable}{lrrrrr}{The counterexample. Mean and Variance: of $\\widehat{\\tau}_{HT}(d_{01},d_{00})$. Expectation: of its variance estimator. Bias: the expectation minus the variance. In the first row the 66 samples are equally likely. In the second and third rows the sample is fixed, and the assignment that it omits treats two units of one clique or one unit in each of two cliques. The decimals are exact fractions rounded to six places.}\nExposure probabilities & Samples & Mean & Variance & Expectation & Bias \\\\\nSimulated, all samples & 66 & $-67/53592$ & 0.487658 & 0.447098 & $-0.040560$ \\\\\nSimulated, one clique & 18 & $-89/4872$ & 0.487267 & 0.421232 & $-0.066035$ \\\\\nSimulated, two cliques & 48 & $25/4872$ & 0.487655 & 0.456797 & $-0.030858$ \\\\\nExact & & $0$ & 0.508104 & 0.508104 & $0$ \\\\\n\\end{audittable}\n\nIn the first row the moments are over the sample and the realized assignment, and the exact values are\n\\[\n\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_{01},d_{00})]=\\frac{113448938731}{232640299584},\n\\qquad\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_{01},d_{00})]\\Bigr]=\\frac{1040129476423}{2326402995840}.\n\\]\nTheir difference is $-953130413/23499020160$, and the expectation is about 8.3 percent below the variance. In the second and third rows the moments are over the realized assignment. The bias is negative for each of the 66 samples. With the exact probabilities in the same formulas, the variance and the expectation of the variance estimator both equal $6050/11907$.\n\n\\emph{The package.} I ran the unchanged functions of two versions of the \\texttt{interference} package (commits \\texttt{a5b8dfb} and \\texttt{c3fc7b3}) for the exposure mapping, the exposure probabilities, the variance and the covariance. For each of the 4356 pairs, I compared two quantities with the enumeration: the contrast computed with the package's simulated probabilities, and the variance estimate that the package returns. In both versions they agree up to floating-point rounding.\n\n\\emph{Derivation for exact exposure probabilities.} Let $\\pi_i(d)=\\Pr(D_i=d)$ and $\\pi_{ij}(d,d')=\\Pr(D_i=d,\\,D_j=d')$, with $\\pi_{ii}(d,d)=\\pi_i(d)$, and write $I_i(d)=\\mathbf{1}\\{D_i=d\\}$. Let $\\widehat W_k$ denote the variance estimator $\\widehat{\\operatorname{Var}}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ and $\\widehat C_{kl}$ the estimator of $\\operatorname{Cov}\\bigl[\\widehat{y^T}_{HT}(d_k),\\widehat{y^T}_{HT}(d_l)\\bigr]$. The package computes\n\\[\n\\begin{aligned}\n\\widehat W_k={}&\\sum_{i,j:\\,\\pi_{ij}(d_k,d_k)>0}I_i(d_k)\\,I_j(d_k)\\,\\frac{\\pi_{ij}(d_k,d_k)-\\pi_i(d_k)\\pi_j(d_k)}{\\pi_{ij}(d_k,d_k)}\\,\\frac{Y_iY_j}{\\pi_i(d_k)\\pi_j(d_k)}\\\\\n&+\\sum_{i\\neq j:\\,\\pi_{ij}(d_k,d_k)=0}\\Bigl(\\frac{I_i(d_k)\\,Y_i^2}{2\\pi_i(d_k)}+\\frac{I_j(d_k)\\,Y_j^2}{2\\pi_j(d_k)}\\Bigr),\\\\\n\\widehat C_{kl}={}&\\sum_{i,j:\\,\\pi_{ij}(d_k,d_l)>0}I_i(d_k)\\,I_j(d_l)\\,\\frac{\\pi_{ij}(d_k,d_l)-\\pi_i(d_k)\\pi_j(d_l)}{\\pi_{ij}(d_k,d_l)}\\,\\frac{Y_iY_j}{\\pi_i(d_k)\\pi_j(d_l)}\\\\\n&-\\sum_{i,j:\\,\\pi_{ij}(d_k,d_l)=0}\\Bigl(\\frac{I_i(d_k)\\,Y_i^2}{2\\pi_i(d_k)}+\\frac{I_j(d_l)\\,Y_j^2}{2\\pi_j(d_l)}\\Bigr),\n\\end{aligned}\n\\]\nand $\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]=N^{-2}\\bigl(\\widehat W_k+\\widehat W_l-2\\,\\widehat C_{kl}\\bigr)$. The sums are over ordered pairs of units. The pairs with $i=j$ belong to the first sum of $\\widehat W_k$ and to the second sum of $\\widehat C_{kl}$.\n\nSuppose that the probabilities are exact and that $\\pi_i(d_k)>0$ and $\\pi_i(d_l)>0$ for every unit $i$. Since $\\mathbb{E}\\bigl[I_i(d)\\,I_j(d')\\bigr]=\\pi_{ij}(d,d')$, each term of a sum over positive joint probabilities has expectation equal to the corresponding term of $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ or of the covariance. For a pair with $\\pi_{ij}(d_k,d_k)=0$, the term of the variance is $-y_i(d_k)\\,y_j(d_k)$, and the second sum of $\\widehat W_k$ has expectation $\\bigl(y_i(d_k)^2+y_j(d_k)^2\\bigr)/2$ for that pair. For a pair with $\\pi_{ij}(d_k,d_l)=0$, the term of the covariance is $-y_i(d_k)\\,y_j(d_l)$, and the second sum of $\\widehat C_{kl}$ has expectation $-\\bigl(y_i(d_k)^2+y_j(d_l)^2\\bigr)/2$. Collecting terms, I obtain\n\\[\n\\mathbb{E}\\bigl[\\widehat W_k\\bigr]-\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]=\\frac{1}{2}\\sum_{i\\neq j:\\,\\pi_{ij}(d_k,d_k)=0}\\bigl(y_i(d_k)+y_j(d_k)\\bigr)^2\\;\\ge\\;0\n\\]\nand\n\\[\n\\begin{aligned}\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]\\Bigr]-\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]={}&\\frac{1}{2N^2}\\sum_{i\\neq j:\\,\\pi_{ij}(d_k,d_k)=0}\\bigl(y_i(d_k)+y_j(d_k)\\bigr)^2\\\\\n&+\\frac{1}{2N^2}\\sum_{i\\neq j:\\,\\pi_{ij}(d_l,d_l)=0}\\bigl(y_i(d_l)+y_j(d_l)\\bigr)^2\\\\\n&+\\frac{1}{N^2}\\sum_{i,j:\\,\\pi_{ij}(d_k,d_l)=0}\\bigl(y_i(d_k)-y_j(d_l)\\bigr)^2\\;\\ge\\;0.\n\\end{aligned}\n\\]\nIn the counterexample no two units have joint probability zero of $d_{01}$ or of $d_{00}$, and every difference $y_i(d_{01})-y_j(d_{00})$ is zero. The last display then implies that the bias with exact probabilities is zero, in agreement with the last row of the table. In the first step of the derivation, I used that the probabilities inside the variance estimator are those of the distribution of the realized assignment. With simulated probabilities, the expectation of $I_i(d)\\,I_j(d')$ is still the exact joint probability, the denominator is the simulated one, and the identities do not hold.\n\nThe exact-probability result, exposure mapping, estimands and statement about the point estimator's bias are unaffected. The counterexample concerns the Horvitz--Thompson variance estimator of a contrast. The applications that follow take the maximum of this estimator and the constant-effects variance estimator (p.~299), and I did not examine that maximum. No published simulation value is shown to be wrong, and no failure of confidence-interval coverage or of an asymptotic statement is established. A large finite $R$ alone supplies no general exact sign guarantee: convergence to a nonnegative bias limit need not imply nonnegative bias at finite $R$. The finite calculation therefore needs a precision qualification; this clarifies the intended exact-probability result."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N016", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of eight lines from the right column of page 307. The text reads 'only a few randomized experiments. These typically rely on imposing partial interference assumptions (see below) so that outcomes of, for example, different, villages or schools can be treated as independent observations. Some studies used random or haphazard assignment of treatment to analyze what types of units produce the largest spillovers.' A red box marks the fifth line, 'can be treated as independent observations.'", "caption": "Section 16.4.7, p. 307 (PDF p. 19), right column, paragraph on seeding. The marked line ends the sentence on partial interference: “can be treated as independent observations.”", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N018.png"}], "formal_release": null, "formal_statements": [], "headline": "Partial interference rules out spillovers between villages or schools, which can then be analyzed as separate groups under the actual randomization; it does not by itself make their outcomes statistically independent.", "id": "P045-N018", "location": "Section 16.4.7, p. 307 (PDF p. 19), right column, paragraph on seeding: the sentence that begins “These typically rely”", "pdf_page": 19, "printed_page": "307", "reaches": [], "related_entries": [{"id": "P037-E004", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E004", "work_id": "P037"}], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "These typically rely on imposing partial interference assumptions (see below) so that outcomes of, for example, different, villages or schools \\hl{can be treated as independent observations}.", "corrected": "These typically rely on imposing partial interference assumptions (see below) \\hl{to rule out spillovers between villages or schools and analyze them as separate groups, with inference based on the actual randomization law. In the fixed-population framework used in this chapter, mutually independent complete group assignment vectors are sufficient to guarantee mutually independent group outcome vectors; partial interference alone does not guarantee that independence.}", "intended_meaning": "The sentence says that the randomized experiments on seeding typically assume partial interference. Under this assumption, the treatment of one village or school does not affect the outcomes of another. I read ``independent observations'' here as meaning that the groups can be analyzed as separate units, with inference based on the actual randomization. This meaning is correct. Read literally, the sentence says that the outcomes of different groups are statistically independent. Partial interference alone does not imply that they are independent. It restricts the potential outcomes, which the chapter takes as fixed, and not the random assignment. An example with the design of p.~310 of the chapter shows that the outcomes of two groups can be dependent. In this design, 18 units are divided into six groups of three. Exactly three of the groups receive the saturation $\\psi$, under which two of the three units of a group are treated. The other three groups receive the saturation $\\phi$, under which one unit of a group is treated. Suppose that the outcome of each unit is $1$ if at least two units of its group are treated and $0$ otherwise. Then partial interference holds, and the mean outcome of a group is $1$ if the group receives $\\psi$ and $0$ otherwise. The mean outcome of each group equals $1$ with probability $1/2$, and the mean outcomes of two given groups both equal $1$ with probability $1/5$, where independence would require $1/4$. I found no estimator, variance formula, simulation or reported number in the chapter that depends on this sentence. The paragraph goes on to cite the experiment of Paluck et al. (2016). The related statement about the schools of that experiment is the subject of \\fid{P037-E004}."}, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N018", "work": {"arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/"}, "work_id": "P045"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines. The first two lines end the preceding paragraph: This confidence interval is finite-sample valid, meaning that its coverage is guaranteed to be greater or equal to its nominal level even in small samples. Proposition 3 follows: Suppose the propensity scores are known. Then a finite-sample-valid 1 minus alpha confidence interval for mu is mu-hat plus or minus the square root of xi log(2/alpha)/2n, where xi is the mean of the square inverse of the propensity scores. A red outline surrounds the square root of xi log(2/alpha)/2n.", "caption": "Proposition 3, p. 7 (PDF p. 6), with the two lines that precede it. The outline marks the radius of the printed interval.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E001.png"}, {"alt": "Crop of three lines of text and one displayed inequality. The text reads: The Horvitz-Thompson estimator can be written mu-hat equals the sum of the A-i, where A-i equals X-i Y-i divided by n pi(W-i). Because the outcome is binary, the bound 0 at most A-i at most 1 over n pi(W-i) holds with probability one. Recall that E[mu-hat] = mu, so by Hoeffding's inequality. The display states that the probability, conditional on W, that the absolute value of mu-hat minus mu is at least the square root of xi log(2/alpha)/2n is at most 2 exp of minus 2 xi log(2/alpha)/2n divided by the sum over i of n pi(W-i) to the power minus 2. Red outlines surround E[mu-hat] = mu in the text and the second mu inside the absolute value in the display.", "caption": "Proof of Proposition 3, Appendix A, p. 16 (PDF p. 15). The outlines mark the unconditional identity $\\mathrm{E}[\\hat\\mu] = \\mu$ that the proof recalls and the centre $\\mu$ of the conditional bound.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E001-proof.png"}, {"alt": "Two panels that share a horizontal axis, K from 0 to 100, under the title: Proposition 3 as printed: the probability that mu is not in the interval equals 1.198 times 10 to the minus 7 at nominal level alpha equal to 10 to the minus 7. The top panel is a blue curve, on a logarithmic vertical axis, of the probability that mu is not in the interval given K. The curve equals 1 for K below about 17, falls to about 3 times 10 to the minus 13 near K equal to 64, and rises again to about 0.3 at K equal to 100. A dashed orange horizontal line marks the nominal level 10 to the minus 7; the curve is above this line for K at most 50 and for K at least 72. A dotted vertical line at K equal to 60 is labelled: the conditional mean of mu-hat equals mu. The bottom panel is a bar chart of the probability that K equals k and mu is not in the interval, in units of 10 to the minus 9. The bars form two groups, one between K equal to 29 and 54 with a maximum of about 12.9 at K equal to 44, and a smaller one between K equal to 70 and 86 with a maximum of about 5 at K equal to 76. A label gives the exact sum over K, 1.198 times 10 to the minus 7, above the nominal level, 1.000 times 10 to the minus 7.", "caption": "The counterexample for P046-E001 ($n = 100$, $\\alpha = 10^{-7}$), by the number $K$ of units such that $W_i \\le 3/5$. Top: the probability, given $K$, that the printed interval fails to contain $\\mu$, on a logarithmic scale; the dashed line is the nominal level $\\alpha$. Bottom: the contribution of each value of $K$ to that probability; the contributions sum to $1.198 \\times 10^{-7}$. Given $K$, the estimator $\\hat\\mu$ has mean $K/100$, which equals $\\mu = 3/5$ only when $K = 60$.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E001-noncoverage.png"}], "formal_release": null, "formal_statements": [{"id": "P046-T1", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T1"}, {"id": "P046-T2", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T2"}, {"id": "P046-T3", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T3"}, {"id": "P046-T6", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T6"}], "headline": "The interval of Proposition 3 is not finite-sample valid at every level, and its proof centres a conditional Hoeffding bound at $\\mu$ where the conditional mean of $\\hat\\mu$ is required; the interval with radius $(\\sqrt{\\xi}+1)\\sqrt{\\log(4/\\alpha)/(2n)}$ is valid.", "id": "P046-E001", "location": "Proposition 3, p. 7 (PDF p. 6); its proof in Appendix A, pp. 15–16 (PDF pp. 14–15), at the display after “Recall that $\\mathrm{E}[\\hat\\mu] = \\mu$, so by Hoeffding’s inequality,” on p. 16", "pdf_page": 6, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P046-T1", "P046-T2", "P046-T3", "P046-T6"], "text": {"as_printed": "\\emph{Proposition 3 (p.~7).} Suppose the propensity scores are known. Then a finite-sample-valid $1-\\alpha$ confidence interval for $\\mu$ is $\\hat\\mu \\pm \\hl{\\sqrt{\\xi \\log(2/\\alpha)/2n}}$, where $\\xi = n^{-1}\\sum_{i=1}^n [\\pi(W_i)]^{-2}$ is the mean of the square inverse of the propensity scores.\n\n\\emph{Proof, Appendix A (p.~16).} The Horvitz--Thompson estimator can be written $\\hat\\mu = \\sum_{i=1}^n A_i$, where $A_i = [n\\pi(W_i)]^{-1}X_iY_i$. Because the outcome is binary, the bound $0 \\le A_i \\le [n\\pi(W_i)]^{-1}$ holds with probability one. Recall that $\\mathrm{E}[\\hat\\mu] = \\mu$, \\hl{so by Hoeffding's inequality},\n\\[\n\\Pr\\Bigl( |\\hat\\mu - \\hl{\\mu}| \\ge \\sqrt{\\xi \\log(2/\\alpha)/2n} \\Bigm| \\mathbf{W} \\Bigr) \\le 2\\exp\\biggl( \\frac{-2\\xi \\log(2/\\alpha)/2n}{\\sum_{i=1}^n [n\\pi(W_i)]^{-2}} \\biggr).\n\\]", "corrected": "\\emph{Proposition 3.} Suppose the propensity scores are known, and let $0 < \\alpha < 1$. Under the assumptions of Section 2 (an i.i.d.\\ sample, binary $Y_i$ and $X_i$, consistency, unconfoundedness and positivity), a finite-sample-valid $1-\\alpha$ confidence interval for $\\mu = \\mathrm{E}[Y_i(1)]$ is\n\\[\n\\hat\\mu \\pm \\hl{(\\sqrt{\\xi}+1)\\sqrt{\\log(4/\\alpha)/(2n)}},\n\\]\nwhere $\\xi = n^{-1}\\sum_{i=1}^n [\\pi(W_i)]^{-2}$ is the mean of the square inverse of the propensity scores. \\hl{If a number $b > 0$ is known such that $\\pi(W_i) \\ge b$ with probability one, then $\\hat\\mu \\pm b^{-1}\\sqrt{\\log(2/\\alpha)/(2n)}$ is also a finite-sample-valid $1-\\alpha$ confidence interval for $\\mu$.}\n\n\\emph{Proof, in place of the conditional display.} Let $m(w) = \\mathrm{E}[Y_i(1) \\mid W_i = w]$ be the outcome regression function. Recall that\n\\[\n\\hl{\\mathrm{E}[\\hat\\mu \\mid \\mathbf{W}] = \\mu_{\\mathbf{W}}}, \\qquad \\text{where } \\mu_{\\mathbf{W}} = \\frac{1}{n}\\sum_{i=1}^n m(W_i),\n\\]\nso by Hoeffding's inequality, with $c = \\sqrt{\\log(4/\\alpha)/(2n)}$,\n\\[\n\\Pr\\Bigl( |\\hat\\mu - \\hl{\\mu_{\\mathbf{W}}}| \\ge \\sqrt{\\xi}\\,c \\Bigm| \\mathbf{W} \\Bigr) \\le 2\\exp\\biggl( \\frac{-2\\xi c^2}{\\sum_{i=1}^n [n\\pi(W_i)]^{-2}} \\biggr) = \\frac{\\alpha}{2}.\n\\]\n\\hl{The variables $m(W_i)$ are i.i.d., take values in $[0,1]$ and have mean $\\mu$, so a second application of Hoeffding's inequality gives $\\Pr(|\\mu_{\\mathbf{W}} - \\mu| \\ge c) \\le \\alpha/2$. The triangle inequality and a union bound then give $\\Pr(|\\hat\\mu - \\mu| \\ge (\\sqrt{\\xi}+1)c) \\le \\alpha$.}", "evidence": "The evidence is an exact counterexample to the printed statement and a derivation of the corrected one.\n\n\\emph{Counterexample.} Let $n = 100$ and $\\alpha = 10^{-7}$. Let $W_i$ be uniform on $(0,1)$, and set $Y_i(1) = \\mathbf{1}\\{W_i \\le 3/5\\}$ and $Y_i(0) = 0$. Let $\\pi(w) = 13/20$ for $w \\le 3/5$ and $\\pi(w) = 99/100$ for $w > 3/5$, and assign the treatments independently given the covariates. The observations are i.i.d., the outcome and the treatment are binary, the covariate is continuous, consistency and unconfoundedness hold, positivity holds with $\\delta = 1/200$, and $\\mu = 3/5$. Let $K$ be the number of units such that $W_i \\le 3/5$, and let $S$ be the number of these units that are treated. In this design $\\hat\\mu = S/65$ and $\\xi = [400K/169 + 10000(100-K)/9801]/100$, which implies that the printed interval depends on the sample only through $(K, S)$. For $0 \\le s \\le k \\le 100$, I have\n\\[\n\\Pr(K = k,\\, S = s) = \\binom{100}{k}\\binom{k}{s}\\frac{39^s\\, 21^{k-s}\\, 40^{100-k}}{100^{100}}.\n\\]\nThe printed interval fails to contain $\\mu$ exactly when $200(S/65 - 3/5)^2/\\xi > \\log(2 \\cdot 10^7)$. This inequality holds for 1946 of the 5151 pairs $(k,s)$. Summing their probabilities in exact rational arithmetic, with rational bounds on the logarithm, I obtain\n\\[\n\\Pr\\Bigl( |\\hat\\mu - \\mu| > \\sqrt{\\xi \\log(2/\\alpha)/2n} \\Bigr) = 1.19792910697956\\ldots \\times 10^{-7} > 10^{-7} = \\alpha.\n\\]\nThe conditional display in the proof fails more directly. On the event that $W_i > 3/5$ for every $i$, which has probability $(2/5)^{100} > 0$, the estimator $\\hat\\mu$ equals $0$ and $\\xi = (100/99)^2$. On this event $|\\hat\\mu - \\mu| = 3/5$ exceeds the printed radius, the left side of the display equals $1$, and the right side equals $10^{-7}$.\n\nThe table lists, at three levels, the exact probability that each interval fails to contain $\\mu$ in this design. Only the first entry of the first row exceeds its nominal level.\n\\begin{audittable}{lll}{Exact probability that the interval fails to contain $\\mu$ in the counterexample ($n = 100$), for the printed radius and for the corrected radius.}\nNominal level $\\alpha$ & Printed interval & Corrected interval \\\\\n$10^{-7}$ & $\\hl{1.198 \\times 10^{-7}}$ & $1.207 \\times 10^{-23}$ \\\\\n$0.05$ & $0.01381$ & $2.324 \\times 10^{-6}$ \\\\\n$0.01$ & $0.003614$ & $2.396 \\times 10^{-8}$ \\\\\n\\end{audittable}\n\n\\emph{Derivation of the corrected interval.} Fix $\\alpha \\in (0,1)$ and set $c = \\sqrt{\\log(4/\\alpha)/(2n)}$. By consistency, I have $X_iY_i = X_iY_i(1)$. Using unconfoundedness and the independence of the units, I find\n\\[\n\\mathrm{E}[A_i \\mid \\mathbf{W}] = \\frac{\\mathrm{E}[X_i \\mid W_i]\\,\\mathrm{E}[Y_i(1) \\mid W_i]}{n\\pi(W_i)} = \\frac{m(W_i)}{n},\n\\]\nand summing over $i$, I obtain $\\mathrm{E}[\\hat\\mu \\mid \\mathbf{W}] = \\mu_{\\mathbf{W}}$. Conditional on $\\mathbf{W}$, the variables $A_i$ are independent and satisfy $0 \\le A_i \\le [n\\pi(W_i)]^{-1}$. Since $\\sum_{i=1}^n [n\\pi(W_i)]^{-2} = \\xi/n$, Hoeffding's inequality implies that\n\\[\n\\Pr\\bigl( |\\hat\\mu - \\mu_{\\mathbf{W}}| \\ge t \\bigm| \\mathbf{W} \\bigr) \\le 2\\exp(-2nt^2/\\xi)\n\\]\nfor every $t > 0$ that is a function of $\\mathbf{W}$. Setting $t = \\sqrt{\\xi}\\,c$ and taking the expectation over $\\mathbf{W}$, I obtain\n\\[\n\\Pr\\bigl( |\\hat\\mu - \\mu_{\\mathbf{W}}| \\ge \\sqrt{\\xi}\\,c \\bigr) \\le 2\\exp(-2nc^2) = \\alpha/2.\n\\]\nNext, the variables $m(W_i)$ are i.i.d., take values in $[0,1]$ and have mean $\\mu$. Applying Hoeffding's inequality to their mean, I have $\\Pr(|\\mu_{\\mathbf{W}} - \\mu| \\ge c) \\le 2\\exp(-2nc^2) = \\alpha/2$. If neither event occurs, then by the triangle inequality I have\n\\[\n|\\hat\\mu - \\mu| \\le |\\hat\\mu - \\mu_{\\mathbf{W}}| + |\\mu_{\\mathbf{W}} - \\mu| < (\\sqrt{\\xi}+1)c.\n\\]\nBy a union bound, I conclude that $\\Pr(|\\hat\\mu - \\mu| \\ge (\\sqrt{\\xi}+1)c) \\le \\alpha$. Positivity implies that $\\sqrt{\\xi} < \\delta^{-1}$, and the width of the corrected interval is $O(n^{-1/2})$ for fixed $\\alpha$. For the alternative with a known bound $b > 0$ such that $\\pi(W_i) \\ge b$ with probability one, the variables $X_iY_i/\\pi(W_i)$ are i.i.d., take values in $[0, b^{-1}]$ and have mean $\\mu$, and Hoeffding's inequality applied to their mean $\\hat\\mu$ gives the radius $b^{-1}\\sqrt{\\log(2/\\alpha)/(2n)}$.", "problem": "The proof conditions on the covariate vector $\\mathbf{W} = (W_1, \\dots, W_n)$ and applies Hoeffding's inequality to the deviation of $\\hat\\mu$ from $\\mu$. Conditional on $\\mathbf{W}$, however, the mean of $\\hat\\mu$ need not equal $\\mu$. Let $m(w) = \\mathrm{E}[Y_i(1) \\mid W_i = w]$ and $\\mu_{\\mathbf{W}} = n^{-1}\\sum_{i=1}^n m(W_i)$. Then $\\mathrm{E}[\\hat\\mu \\mid \\mathbf{W}] = \\mu_{\\mathbf{W}}$; taking the expectation over $\\mathbf{W}$ gives the unconditional identity $\\mathrm{E}[\\hat\\mu] = \\mu$ that the proof recalls. Hoeffding's inequality bounds the conditional probability that $|\\hat\\mu - \\mu_{\\mathbf{W}}|$ exceeds the printed radius. It does not bound $|\\hat\\mu - \\mu|$, and the printed radius does not account for the variation of $\\mu_{\\mathbf{W}}$ around $\\mu$.\n\nThe proposition itself fails as stated. It claims coverage at least $1-\\alpha$ and states no restriction on $\\alpha$. In the design that I give below, which satisfies every assumption of Section 2, the printed interval fails to contain $\\mu$ with probability $1.198 \\times 10^{-7}$ at $\\alpha = 10^{-7}$. This level is far from the levels used in practice. The counterexample does not show that the printed interval undercovers at the 95\\% or 99\\% level: in the same design the probability that it fails to contain $\\mu$ is below $\\alpha$ at both levels. These calculations do not settle whether the printed interval is valid at such levels in every design. To repair the proposition I keep the estimand, the estimator and the assumptions, and I widen the interval. A second Hoeffding bound controls $\\mu_{\\mathbf{W}} - \\mu$, and I allot error probability $\\alpha/2$ to each of the two bounds. The corrected interval still has width of order $n^{-1/2}$.", "what_survives": "The claim of the abstract and of p.~7 survives: with a binary outcome and a known propensity score, there exist finite-sample valid confidence intervals for $\\mu$ whose width decreases at the rate $n^{-1/2}$. The corrected interval is one. The printed radius remains valid when the propensity score is constant, by unconditional Hoeffding's inequality. Propositions 1, 2 and 4 and the numerical illustration of Section 3 do not use Proposition 3. What changes is the radius in Proposition 3, which becomes larger, the statement on p.~7 that the printed interval guarantees uniform validity, and the proof in Appendix A."}, "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-E001", "work": {"arxiv_id": "2108.11342", "doi": "10.1353/obs.2025.a956837", "title": "Nonparametric identification is not enough, but randomized controlled trials are", "url": "https://pmaronow.github.io/llm-audit/works/P046/"}, "work_id": "P046"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Proposition 4 from its first line through part (c). The first line reads: Proposition 4. Suppose the propensity scores are known. Then. Part (a): mu-hat-R is finite-sample unbiased for mu. Part (b): If g-hat-0 and g-hat-1 are convergent in mean square (but not necessarily consistent for the true outcome regression function), then mu-hat-R is root-n consistent for mu. Part (c): If g-hat-0 and g-hat-1 are parameterized by a finite number of parameters and estimated by minimizing the empirical weighted mean square error of the outcome regression with weights w-i equal to (1 minus pi(W-i)) divided by pi(W-i) squared, then mu-hat-R is at least as asymptotically efficient as the Horvitz-Thompson estimator mu-hat. A red outline surrounds the words parameterized by a finite number of parameters.", "caption": "Proposition 4, parts (a) to (c), p. 8 (PDF p. 7). The outline marks the hypothesis of part (c) on the working family.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E002.png"}], "formal_release": null, "formal_statements": [{"id": "P046-T4", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T4"}, {"id": "P046-T5", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T5"}], "headline": "Proposition 4(c) holds when the working family of outcome regressions contains the zero function and the fitted regressions converge; for other families the adjusted estimator can be less efficient than the Horvitz–Thompson estimator.", "id": "P046-E002", "location": "Proposition 4(c), p. 8 (PDF p. 7)", "pdf_page": 7, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P046-T4", "P046-T5"], "text": {"as_printed": "\\textbf{Proposition 4} Suppose the propensity scores are known. Then: \\ldots\n\n(c) If $\\hat g_0$ and $\\hat g_1$ are \\hl{parameterized by a finite number of parameters} and estimated by minimizing the empirical weighted mean square error of the outcome regression with weights $w_i = [\\pi(W_i)]^{-2}[1 - \\pi(W_i)]$, then $\\hat\\mu_R$ is at least as asymptotically efficient as the Horvitz--Thompson estimator $\\hat\\mu$.", "corrected": "\\textbf{Proposition 4} Suppose the propensity scores are known. Then: \\ldots\n\n(c) \\hl{Suppose that} $\\hat g_0$ and $\\hat g_1$ are estimated by minimizing the empirical weighted mean square error of the outcome regression with weights $w_i = [\\pi(W_i)]^{-2}[1 - \\pi(W_i)]$ \\hl{over a fixed working family that is parameterized by a finite number of parameters and contains the zero function (or has the zero function in its weighted $L^2$ closure), and that both converge in $L^2$ to a deterministic minimizer over this family of}\n\\[\n\\hl{R(g) = \\mathrm{E}\\biggl[ \\frac{1 - \\pi(W_i)}{\\pi(W_i)}\\bigl( Y_i(1) - g(W_i) \\bigr)^2 \\biggr].}\n\\]\n\\hl{Then} $\\hat\\mu_R$ is at least as asymptotically efficient as the Horvitz--Thompson estimator $\\hat\\mu$.", "evidence": "The evidence is an exact counterexample to the printed statement and a derivation of the corrected one.\n\n\\emph{Counterexample.} Let $W_i$ be uniform on $(0,1)$, let $L_i = \\mathbf{1}\\{W_i \\le 1/2\\}$, and let $Z_i$ be a Bernoulli variable with mean $1/10$ that is independent of $W_i$. Set $Y_i(0) = Y_i(1) = L_iZ_i$. Let $X_i$ be independent of $Z_i$ given $W_i$, with $\\pi(W_i) = 1/4$ if $L_i = 1$ and $\\pi(W_i) = 3/4$ if $L_i = 0$. The assumptions of Section 2 hold, with $\\delta = 1/8$ in the positivity condition, and $\\mu = 1/20$. Consider the working family of the functions $g_\\theta(w) = (1/2)\\mathbf{1}\\{w \\le 1/2\\} + \\theta\\,\\mathbf{1}\\{w > 1/2\\}$ with $\\theta \\in \\mathbb{R}$. It has one parameter and does not contain the zero function. The printed weights equal $12$ where $L_i = 1$ and $4/9$ where $L_i = 0$. Since $Y_i = 0$ where $L_i = 0$, the weighted criterion on any training fold equals a term that does not depend on $\\theta$ plus $(4/9)N\\theta^2$, where $N$ is the number of treated units of the training fold such that $L_i = 0$. The value $\\theta = 0$ minimizes it for every training fold (I take $\\theta = 0$ when $N = 0$), and $\\theta = 0$ is the unique minimizer of the population criterion $R(g_\\theta) = 3/8 + \\theta^2/6$. With $g_*(w) = (1/2)\\mathbf{1}\\{w \\le 1/2\\}$, I therefore have $\\hat g_0 = \\hat g_1 = g_*$, and $\\hat\\mu_R$ is the mean of the i.i.d.\\ variables $A_i = X_i[Y_i - g_*(W_i)]/\\pi(W_i) + g_*(W_i)$. The table lists the values of $A_i$ and of the Horvitz--Thompson summand $H_i = X_iY_i/\\pi(W_i)$.\n\\begin{audittable}{llll}{The summands of the two estimators in the counterexample.}\nEvent & Probability & $A_i$ & $H_i$ \\\\\n$L_i = 1$, $X_i = 0$ & $3/8$ & $1/2$ & $0$ \\\\\n$L_i = 1$, $X_i = 1$, $Z_i = 0$ & $9/80$ & $-3/2$ & $0$ \\\\\n$L_i = 1$, $X_i = 1$, $Z_i = 1$ & $1/80$ & $5/2$ & $4$ \\\\\n$L_i = 0$ & $1/2$ & $0$ & $0$ \\\\\n\\end{audittable}\nA direct computation gives $\\mathrm{E}[A_i] = \\mathrm{E}[H_i] = 1/20$, $\\mathrm{E}[A_i^2] = 17/40$ and $\\mathrm{E}[H_i^2] = 1/5$. For every $n$, I obtain\n\\[\n\\operatorname{Var}(\\hat\\mu_R) = \\frac{169}{400n} > \\frac{79}{400n} = \\operatorname{Var}(\\hat\\mu).\n\\]\nIn this example $\\operatorname{Var}(Y_i(1)) = 19/400$, $R(0) = 3/20$ and $R(g_*) = 3/8$.\n\n\\emph{Derivation of the corrected statement.} Fix a square-integrable function $g$ and write $A_g = Y_i(1) + (X_i/\\pi(W_i) - 1)(Y_i(1) - g(W_i))$ for the summand of $\\hat\\mu_R$. By unconfoundedness, conditional on $(W_i, Y_i(1))$ the variable $A_g$ has mean $Y_i(1)$ and variance $(1 - \\pi(W_i))\\pi(W_i)^{-1}(Y_i(1) - g(W_i))^2$. By the law of total variance, I have $\\operatorname{Var}(A_g) = \\operatorname{Var}(Y_i(1)) + R(g)$. For $g = 0$ this is the variance of the Horvitz--Thompson summand. If the working family contains the zero function, then a minimizer $g_*$ of $R$ over the family satisfies $R(g_*) \\le R(0)$. Suppose that $\\hat g_0$ and $\\hat g_1$ converge to $g_*$ in $L^2$. Conditional on the training fold, the summands of the difference between $\\hat\\mu_R$ and the mean of the variables $A_{g_*}$ have mean zero and second moment $\\mathrm{E}[(1 - \\pi(W_i))\\pi(W_i)^{-1}(\\hat g_j(W_i) - g_*(W_i))^2]$ over the evaluation fold, which implies that this difference is $o_p(n^{-1/2})$. Then $\\sqrt{n}(\\hat\\mu_R - \\mu)$ has asymptotic variance $\\operatorname{Var}(Y_i(1)) + R(g_*)$, which is at most the asymptotic variance $\\operatorname{Var}(Y_i(1)) + R(0)$ of $\\sqrt{n}(\\hat\\mu - \\mu)$.", "problem": "Note that the Horvitz--Thompson estimator is the adjusted estimator $\\hat\\mu_R$ with the zero function in place of $\\hat g_0$ and $\\hat g_1$. For a fixed function $g$, the summand of $\\hat\\mu_R$ has variance $\\operatorname{Var}(Y_i(1)) + R(g)$, where\n\\[\nR(g) = \\mathrm{E}\\biggl[ \\frac{1 - \\pi(W_i)}{\\pi(W_i)}\\bigl( Y_i(1) - g(W_i) \\bigr)^2 \\biggr],\n\\]\nand the printed weights make the population weighted least-squares criterion on the treated units proportional to $R(g)$. The fit minimizes $R$ only over the family of functions in which the two regressions are fitted, which I call the working family. The comparison with the Horvitz--Thompson estimator requires that the limit $g_*$ of the fit satisfy $R(g_*) \\le R(0)$. This holds when the working family contains the zero function, and it can fail otherwise. Part (c) as printed admits every family with a finite number of parameters. Every linear model with freely fitted coefficients contains the zero function, and the usual working models satisfy this condition.", "what_survives": "The sentence on p.~7 that adjustments ``can be constructed so that they cannot harm, but can dramatically improve, asymptotic precision'' asserts that suitable adjustments exist, and it stands. No other result of the paper relies on part (c), and the paper states on p.~10 that part (c) does not apply to its HTLOESS estimator. In the counterexample both estimators remain unbiased and root-$n$ consistent. What changes is the hypothesis of part (c)."}, "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-E002", "work": {"arxiv_id": "2108.11342", "doi": "10.1353/obs.2025.a956837", "title": "Nonparametric identification is not enough, but randomized controlled trials are", "url": "https://pmaronow.github.io/llm-audit/works/P046/"}, "work_id": "P046"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of part (d) of Proposition 4 and the first four lines of the next paragraph. Part (d) reads: If g-hat-0 and g-hat-1 are consistent for the true regression function, then mu-hat-R attains the semiparametric efficiency bound. Red outlines surround the words the semi-parametric efficiency bound, which run over a line break. The paragraph reads: All parts of Proposition 4 are previously known results. For completeness, we provide a proof for Proposition 4(a) in the appendix. See van der Vaart (1998) for Proposition 4(b). See Section 3 of Cao et al. (2009) for Proposition 4(c); see also Rubin and van der Laan (2008). See Theorem 5.1 in Chernozhukov et al. (2018) for Proposition 4(d).", "caption": "Proposition 4(d) and the paragraph that follows the proposition, p. 8 (PDF p. 7). The outline marks “the semiparametric efficiency bound”; the last line shown cites Theorem 5.1 of Chernozhukov et al. (2018) for part (d).", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E003.png"}], "formal_release": null, "formal_statements": [{"id": "P046-T9", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T9"}], "headline": "“The semiparametric efficiency bound” in Proposition 4(d) is the bound of Hahn (1998) for an unrestricted covariate distribution, and with this meaning the statement holds whether or not the covariate distribution is known.", "id": "P046-E003", "location": "Proposition 4(d), p. 8 (PDF p. 7), with the set-up on p. 5 (PDF p. 4)", "pdf_page": 7, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P046-T9"], "text": {"as_printed": "(d) If $\\hat g_0$ and $\\hat g_1$ are consistent for the true regression function, then $\\hat\\mu_R$ attains \\hl{the semiparametric efficiency bound}.", "corrected": "(d) If \\hl{$\\|\\hat g_j - m\\|_{L^2(F_W)} \\to 0$ in probability for $j = 0, 1$, where $m(w) = \\mathrm{E}[Y_i(1) \\mid W_i = w]$}, then $\\hat\\mu_R$ attains \\hl{the efficiency bound of Hahn (1998) for $\\mu$ in the model with a known propensity score and an unrestricted covariate distribution $F_W$, which equals $\\mathrm{E}[m(W_i)(1 - m(W_i))/\\pi(W_i)] + \\operatorname{Var}(m(W_i))$}.", "intended_meaning": "The paragraph after Proposition 4 cites Theorem 5.1 of Chernozhukov et al.\\ (2018) for part (d). That theorem, as it appears in the preprint arXiv:1608.00060v6, names the efficiency bound of Hahn (1998), which is the bound in the model with an unrestricted distribution $F_W$ of the covariate. Let $m(w) = \\mathrm{E}[Y_i(1) \\mid W_i = w]$. For $\\mu$ and a known propensity score this bound equals $\\mathrm{E}[m(W_i)(1 - m(W_i))/\\pi(W_i)] + \\operatorname{Var}(m(W_i))$, and $\\hat\\mu_R$ attains it when $\\hat g_0$ and $\\hat g_1$ converge to $m$ in $L^2(F_W)$, whether or not $F_W$ is known. The set-up on p.~5 allows $F_W$ to be ``either known or unknown''. In the smaller model in which $F_W$ is known the bound is lower by $\\operatorname{Var}(m(W_i))$, and part (d) does not claim that $\\hat\\mu_R$ attains that lower bound."}, "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-E003", "work": {"arxiv_id": "2108.11342", "doi": "10.1353/obs.2025.a956837", "title": "Nonparametric identification is not enough, but randomized controlled trials are", "url": "https://pmaronow.github.io/llm-audit/works/P046/"}, "work_id": "P046"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the heading Abstract and the first eight lines of the abstract. The third sentence reads: First, in settings with at least one continuous confounder, there exists no estimator of the average treatment effect that is uniformly consistent unless the propensity score is known or additional assumptions are made on the complexity of the propensity score function. A red outline surrounds the words the propensity score function, at the start of the sixth line.", "caption": "Abstract, p. 3 (PDF p. 2). The outline marks the phrase that names the propensity score function as the only object of the additional assumptions.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E004.png"}], "formal_release": null, "formal_statements": [{"id": "P046-T10", "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-T10"}], "headline": "Assumptions on the outcome regression alone also permit a uniformly consistent estimator; the abstract names only assumptions on the propensity score function.", "id": "P046-E004", "location": "Abstract, third sentence, p. 3 (PDF p. 2)", "pdf_page": 2, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P046-T10"], "text": {"as_printed": "First, in settings with at least one continuous confounder, there exists no estimator of the average treatment effect that is uniformly consistent unless the propensity score is known or additional assumptions are made on the complexity of \\hl{the propensity score function}.", "corrected": "First, in settings with at least one continuous confounder, there exists no estimator of the average treatment effect that is uniformly consistent unless the propensity score is known or additional assumptions are made on the complexity of \\hl{the outcome regression and/or propensity score functions}.", "intended_meaning": "The sentence summarizes Proposition 1, which states the impossibility ``Without further assumptions'', and with this meaning it is correct. Read literally, the clause after ``unless'' lists every remedy, and the list is incomplete: suitable assumptions on the outcome regression alone also suffice, with the propensity score unknown and unrestricted. For example, suppose that $\\mathrm{E}[Y_i(x) \\mid W_i = w] = a_x + h(w)$ for $x \\in \\{0,1\\}$, where $h(w) = \\min\\{1, \\max\\{0, w\\}\\}/4$ is known and $a_0, a_1 \\in [1/4, 1/2]$ are unknown. Then the difference of the within-arm means of $Y_i - h(W_i)$ (zero for an empty arm) estimates the average treatment effect with mean squared error at most $(3/4)^n + 25/[2(n+1)]$, for every measurable propensity score such that $1/4 < \\pi(W_i) < 3/4$. The body of the paper allows for such assumptions (the paragraph after Proposition 1 and footnote 1, p.~6), and no result depends on the literal reading."}, "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-E004", "work": {"arxiv_id": "2108.11342", "doi": "10.1353/obs.2025.a956837", "title": "Nonparametric identification is not enough, but randomized controlled trials are", "url": "https://pmaronow.github.io/llm-audit/works/P046/"}, "work_id": "P046"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the last six lines of page 9. A red outline surrounds the sentence: The second estimator, labelled LOESS, uses an LOESS regression for the predictions. The text continues: We use LOESS regression with locally linear fits and the smoothing parameter, alpha, is set to 5 times N to the power minus one third, where N is the number of observations used in the regression. In this case, N is approximately n/2, because this is the number of observations with X-i equal to 1.", "caption": "Section 3.2, p. 9 (PDF p. 8). The outline marks the sentence that introduces the LOESS estimator; the lines after it give the degree and span of the fits and no rule for missing predictions.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E005.png"}, {"alt": "Crop of two lines of text, a displayed formula and three more lines. The text says that both estimators are so-called double robust estimators (Robins et al., 1994) of the form: one over n times the sum over i of X-i times (Y-i minus g-hat(W-i)) divided by pi-hat(W-i), plus g-hat(W-i). It continues: The first of these estimators, labelled DRLOG, uses ordinary linear logistic regression to construct both g-hat(W-i) and pi-hat(W-i). The second estimator, labelled DRLOESS, instead uses LOESS regression for both functions. We use the same type of LOESS regression here as above. A red outline surrounds the last sentence.", "caption": "Section 3.2, p. 10 (PDF p. 9). The outline marks the last sentence of the description of the two estimators that use an estimated propensity score; the description states no lower limit for the estimated scores.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E005-dr.png"}], "formal_release": null, "formal_statements": [], "headline": "The simulation code applies two rules that Section 3.2 does not state: a missing LOESS prediction is set to $0.5$, and an estimated propensity score below $0.01$ is set to $0.01$.", "id": "P046-E005", "location": "Section 3.2, pp. 9–10 (PDF pp. 8–9)", "pdf_page": 8, "printed_page": "9–10", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~9.} The second estimator, labelled \\texttt{LOESS}, uses an LOESS regression for the predictions. We use LOESS regression with locally linear fits and the smoothing parameter, $\\alpha$, is set to $5N^{-1/3}$, where $N$ is the number of observations used in the regression.\n\n\\emph{p.~10.} The first of these estimators, labelled \\texttt{DRLOG}, uses ordinary linear logistic regression to construct both $\\hat g(W_i)$ and $\\hat\\pi(W_i)$. The second estimator, labelled \\texttt{DRLOESS}, instead uses LOESS regression for both functions. We use the same type of LOESS regression here as above.", "corrected": "Add to Section 3.2: \\hl{For every LOESS fit, missing predictions are replaced by $0.5$. Estimated propensity scores used in \\texttt{DRLOG} and \\texttt{DRLOESS} are floored at $0.01$ from below only, with no upper cap.}", "intended_meaning": "The passages name the regression families, the degree and span of the LOESS fits and the estimator formulas, and nothing in them is false. They do not fully specify the estimators behind Figure 3. In the replication code (\\texttt{inesims.R}), every LOESS prediction that is missing is replaced by $0.5$, which enters \\texttt{LOESS}, \\texttt{DRLOESS} and \\texttt{HTLOESS}, and the estimated propensity scores of \\texttt{DRLOG} and \\texttt{DRLOESS} are raised to $0.01$ when they fall below it, with no upper cap. Both rules can take effect. Their effect on the reported RMSEs was not measured, and no reported value is shown to change."}, "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-E005", "work": {"arxiv_id": "2108.11342", "doi": "10.1353/obs.2025.a956837", "title": "Nonparametric identification is not enough, but randomized controlled trials are", "url": "https://pmaronow.github.io/llm-audit/works/P046/"}, "work_id": "P046"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of three lines. The passage begins: trade-off: when the assignment mechanism is unknown, the strength of our statistical procedures critically depend on the assumptions brought to bear. A red outline surrounds the word depend.", "caption": "Item 1: Section 2, p. 6 (PDF p. 5). The outline marks “depend”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007.png"}, {"alt": "Crop of three lines. The passage includes: To ground these ideas, we consider a regression-based estimator of mu that use a simple type of cross-fitting. A red outline surrounds the word use.", "caption": "Item 2: Section 2, p. 7 (PDF p. 6). The outline marks “use”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007-2.png"}, {"alt": "Crop of three lines. The passage includes: in Proposition 4, where g-hat-0 and g-hat-1 are cross-fitted LOESS regression of the same type as above. A red outline surrounds the word regression.", "caption": "Item 3: Section 3.2, p. 10 (PDF p. 9). The outline marks “regression”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007-3.png"}, {"alt": "Crop of three lines. The sentence reads: For the third data generating process, which is the highly discontinuous function, only the two estimators that use of the true propensity score perform well. A red outline surrounds the words use of.", "caption": "Item 4: Section 3.3, p. 11 (PDF p. 10). The outline marks “use of”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007-4.png"}, {"alt": "Crop of three lines. The sentence reads: The explanation is that it is essentially impossible to capture the outcome regression function here, and trying to do so introduce unnecessary noise, which makes the precision worse. A red outline surrounds the word introduce.", "caption": "Item 5: Section 3.3, p. 11 (PDF p. 10). The outline marks “introduce”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007-5.png"}, {"alt": "Crop of three lines. The sentence reads: Knowledge of the propensity score can enormously beneficial in such settings. A red outline surrounds the words can enormously beneficial, at the start of the second line.", "caption": "Item 6: Section 4, p. 12 (PDF p. 11). The outline marks “can enormously beneficial”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007-6.png"}, {"alt": "Crop of three lines. The sentence reads: The estimation difficulties are often profound because the input space tend to be high-dimensional, and the treatment assignment function and the outcome regression tend to be highly non-smooth with discontinuities. A red outline surrounds the first word tend, at the start of the second line.", "caption": "Item 7: Section 4, p. 12 (PDF p. 11). The outline marks “tend”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007-7.png"}, {"alt": "Crop of the three lines of the reference by P. M. Aronow and Joel A. Middleton, A class of unbiased estimators of the average treatment effect in randomized experiments, Journal of Causal Inference, 2013. The third line reads: 2013. doi: doi:10.1515/jci-2012-0009, followed by the URL. A red outline surrounds the words doi: doi:.", "caption": "Item 8: References, p. 13 (PDF p. 12). The outline marks the repeated label “doi: doi:”.", "url": "https://pmaronow.github.io/llm-audit/works/P046/exhibits/P046-E007-8.png"}], "formal_release": null, "formal_statements": [], "headline": "Seven sentences have errors of agreement or wording, and one reference repeats the label “doi:”.", "id": "P046-E007", "location": "Seven sentences on pp. 6, 7, 10, 11 and 12 (PDF pp. 5, 6, 9, 10 and 11), and the Aronow–Middleton entry of the reference list, p. 13 (PDF p. 12)", "pdf_page": 5, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\begin{enumerate}\n\\item \\emph{p.~6.} the strength of our statistical procedures critically \\hl{depend} on the assumptions brought to bear.\n\\item \\emph{p.~7.} we consider a regression-based estimator of $\\mu$ that \\hl{use} a simple type of cross-fitting.\n\\item \\emph{p.~10.} where $\\hat g_0$ and $\\hat g_1$ are cross-fitted LOESS \\hl{regression} of the same type as above.\n\\item \\emph{p.~11.} only the two estimators that \\hl{use of} the true propensity score perform well.\n\\item \\emph{p.~11.} and trying to do so \\hl{introduce} unnecessary noise\n\\item \\emph{p.~12.} Knowledge of the propensity score \\hl{can enormously} beneficial in such settings.\n\\item \\emph{p.~12.} because the input space \\hl{tend} to be high-dimensional\n\\item \\emph{p.~13.} Peter M. Aronow and Joel A. Middleton. A class of unbiased estimators of the average treatment effect in randomized experiments. \\emph{Journal of Causal Inference}, 1(1):135--154, 2013. \\hl{doi: doi:}10.1515/jci-2012-0009. URL \\url{https://doi.org/10.1515/jci-2012-0009}.\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item the strength of our statistical procedures critically \\hl{depends} on the assumptions brought to bear.\n\\item we consider a regression-based estimator of $\\mu$ that \\hl{uses} a simple type of cross-fitting.\n\\item where $\\hat g_0$ and $\\hat g_1$ are cross-fitted LOESS \\hl{regressions} of the same type as above.\n\\item only the two estimators that \\hl{use} the true propensity score perform well.\n\\item and trying to do so \\hl{introduces} unnecessary noise\n\\item Knowledge of the propensity score \\hl{can be enormously} beneficial in such settings.\n\\item because the input space \\hl{tends} to be high-dimensional\n\\item P. M. Aronow and Joel A. Middleton. A class of unbiased estimators of the average treatment effect in randomized experiments. \\emph{Journal of Causal Inference}, 1(1):135--154, 2013. \\hl{doi:} 10.1515/jci-2012-0009. URL \\url{https://doi.org/10.1515/jci-2012-0009}.\n\\end{enumerate}", "evidence": "I checked the pages. On p.~7 the paper calls $\\hat g_0$ and $\\hat g_1$ ``two estimators of the regression function'', which supports the plural in item 3.", "problem": "Items 1 to 7 are errors of grammar: a singular subject with a plural verb in items 1, 2, 5 and 7, a singular noun for the two fitted regressions in item 3, an extra ``of'' in item 4, and a missing ``be'' in item 6. The reference in item 8 prints the label ``doi:'' twice.", "what_survives": "The intended meaning is clear in every case, and no statement, formula or number changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P046/#P046-E007", "work": {"arxiv_id": "2108.11342", "doi": "10.1353/obs.2025.a956837", "title": "Nonparametric identification is not enough, but randomized controlled trials are", "url": "https://pmaronow.github.io/llm-audit/works/P046/"}, "work_id": "P046"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of Section 2.6. The sentence at issue reads: The reason the U.S. Food and Drug Administration requires evidence from RCTs for drug approval is not because it successfully adjudicates the question at hand, but because such evidence is effective in producing desirable policy outcomes. A red outline surrounds the words requires evidence from RCTs for drug approval. The next sentence reads: RCTs are preferable over tea leaf readings because of their evidence-producing qualities.", "caption": "Section 2.6, third paragraph, p. 89 (PDF p. 6). The outline marks the statement that the FDA requires evidence from RCTs for drug approval.", "url": "https://pmaronow.github.io/llm-audit/works/P047/exhibits/P047-E001.png"}], "formal_release": null, "formal_statements": [], "headline": "The FDA does not require evidence from RCTs for every drug approval; it ordinarily expects randomized evidence of efficacy and allows other adequate and well-controlled designs.", "id": "P047-E001", "location": "Section 2.6, third paragraph, p. 89 (PDF p. 6), sentence beginning “The reason the U.S. Food and Drug Administration”", "pdf_page": 6, "printed_page": "89", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "The reason the U.S. Food and Drug Administration \\hl{requires evidence from RCTs for drug approval} is not because it successfully adjudicates the question at hand, but because such evidence is effective in producing desirable policy outcomes.", "corrected": "The reason the U.S. Food and Drug Administration \\hl{ordinarily expects randomized evidence of efficacy for drug approval, while allowing other adequate and well-controlled designs,} is not because it successfully adjudicates the question at hand, but because such evidence is effective in producing desirable policy outcomes.\n\nA wording that makes no statement about the FDA's rule also keeps the argument: ``The value of evidence from RCTs in drug approval lies not merely in adjudicating the question at hand, but in helping produce desirable policy outcomes.''", "evidence": "I checked public sources in editions that precede the rejoinder. The statute, \\href{https://www.govinfo.gov/content/pkg/USCODE-2023-title21/html/USCODE-2023-title21-chap9-subchapV-partA-sec355.htm}{21 U.S.C.~355(d)}, defines substantial evidence as ``evidence consisting of adequate and well-controlled investigations, including clinical investigations''. The regulation, \\href{https://www.govinfo.gov/content/pkg/CFR-2024-title21-vol5/pdf/CFR-2024-title21-vol5-sec314-126.pdf}{21 CFR~314.126}, lists ``Historical control'' in paragraph (b)(2)(v) and says in paragraph (b)(4) that ``Ordinarily, in a concurrently controlled study, assignment is by randomization, with or without stratification.'' The FDA \\href{https://content.govdelivery.com/accounts/USFDA/bulletins/2c424af}{approved Nulibry (fosdenopterin) on 26 February 2021} on a comparison of 13 treated patients with 18 matched, untreated patients, and its \\href{https://www.accessdata.fda.gov/drugsatfda_docs/nda/2021/214018Orig1s000OtherR.pdf}{epidemiology review} describes the population of the efficacy study as ``Nulibry patients from 3 single-arm trials and untreated natural history controls''. The same application contained a randomized, blinded, placebo-controlled phase 1 study in healthy volunteers. The approval shows efficacy established without randomization; it is not an example of an application with no RCT evidence.", "problem": "The sentence says without qualification that the FDA requires evidence from RCTs for drug approval. Read as a rule for every approval, this is false. The statute asks for ``adequate and well-controlled investigations'', and the regulation that defines such studies lists a historical control among the accepted designs. The sentence does not say ``every approval'', and it can be read as a description of what the FDA ordinarily expects, which the sources do not contradict. The unqualified verb overstates the rule.", "what_survives": "Only this illustration uses the statement. The distinction between producing evidence and merely reaching a decision, and the comparison of RCTs with tea leaf readings, do not depend on a requirement without exceptions. The correction weakens the description of the FDA's rule, and it does not address the reason that the sentence gives for the FDA's practice."}, "url": "https://pmaronow.github.io/llm-audit/works/P047/#P047-E001", "work": {"arxiv_id": null, "doi": "10.1353/obs.2025.a956844", "title": "Rejoinder: Nonparametric Identification Is Not Enough, but Randomized Controlled Trials Are", "url": "https://pmaronow.github.io/llm-audit/works/P047/"}, "work_id": "P047"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of Section 2.4. The third line reads: settings, where upon the assignment mechanism becomes relevant also to a Bayesian. They. A red outline surrounds the two words where upon.", "caption": "Section 2.4, p. 87 (PDF p. 4). The outline marks “where upon”.", "url": "https://pmaronow.github.io/llm-audit/works/P047/exhibits/P047-E002.png"}, {"alt": "Crop of six lines of Section 2.5. The sentence at issue reads: They claim that these assumptions are realistic, but they provide no justification for this position other than a cursorily declaration that the function class is fairly large. A red outline surrounds the word cursorily, at the start of the fourth line.", "caption": "Section 2.5, third paragraph, p. 88 (PDF p. 5). The outline marks “cursorily”.", "url": "https://pmaronow.github.io/llm-audit/works/P047/exhibits/P047-E002-b.png"}], "formal_release": null, "formal_statements": [], "headline": "On p. 87, “where upon” should read “whereupon”; on p. 88, “a cursorily declaration” should read “a cursory declaration”.", "id": "P047-E002", "location": "Section 2.4, p. 87 (PDF p. 4), sentence beginning “They highlight that these assumptions”; Section 2.5, third paragraph, p. 88 (PDF p. 5), sentence beginning “They claim that these assumptions are realistic”", "pdf_page": 4, "printed_page": "87", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{p.~87.} They highlight that these assumptions do not hold in many settings, \\hl{where upon} the assignment mechanism becomes relevant also to a Bayesian.\n\n\\emph{p.~88.} They claim that these assumptions are realistic, but they provide no justification for this position other than a \\hl{cursorily} declaration that the function class is fairly large.", "corrected": "\\emph{p.~87.} They highlight that these assumptions do not hold in many settings, \\hl{whereupon} the assignment mechanism becomes relevant also to a Bayesian.\n\n\\emph{p.~88.} They claim that these assumptions are realistic, but they provide no justification for this position other than a \\hl{cursory} declaration that the function class is fairly large.", "evidence": "I checked the two pages. Both forms also occur in the manuscript source.", "problem": "On p.~87 the conjunction ``whereupon'' is split into two words. On p.~88 the adverb ``cursorily'' stands where the adjective ``cursory'' is needed to modify ``declaration''.", "what_survives": "The meaning of both sentences is unchanged, and no statement of the rejoinder depends on either word."}, "url": "https://pmaronow.github.io/llm-audit/works/P047/#P047-E002", "work": {"arxiv_id": null, "doi": "10.1353/obs.2025.a956844", "title": "Rejoinder: Nonparametric Identification Is Not Enough, but Randomized Controlled Trials Are", "url": "https://pmaronow.github.io/llm-audit/works/P047/"}, "work_id": "P047"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of one line at the foot of the cover page. At the left is a Creative Commons icon with the labels BY, NC and ND. To its right, grey text reads: This work is licensed under a. A red outline surrounds the words that follow: Creative Commons Attribution 4.0 International License.", "caption": "Foot of the cover page (PDF p. 1). The outline marks the licence as named in words, Creative Commons Attribution 4.0 International License. The icon at the left of the same line is labelled BY NC ND.", "url": "https://pmaronow.github.io/llm-audit/works/P047/exhibits/P047-N001.png"}, {"alt": "Crop of the two-line copyright footer of the first page of the rejoinder. It reads: Copyright 2025 P. M. Aronow, James M. Robins, Theo Saarinen, Fredrik Sävje and Jasjeet Sekhon. This is an open access article distributed under the terms of a Creative Commons license (CC BY-NC-ND 4.0). A red outline surrounds the words CC BY-NC-ND 4.0.", "caption": "Copyright footer of p. 85 (PDF p. 2), for comparison. The outline marks the licence named there, CC BY-NC-ND 4.0.", "url": "https://pmaronow.github.io/llm-audit/works/P047/exhibits/P047-N001-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The cover page names the licence in words as Creative Commons Attribution 4.0, while the icon beside those words and the footer of p. 85 give CC BY-NC-ND 4.0.", "id": "P047-N001", "location": "Cover page added by the distributor (PDF p. 1), licence line at the foot; compare the copyright footer of p. 85 (PDF p. 2)", "pdf_page": 1, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Cover page, beside an icon labelled BY NC ND.} This work is licensed under a Creative Commons \\hl{Attribution} 4.0 International License.\n\n\\emph{Footer of p.~85.} ©2025 P. M. Aronow, James M. Robins, Theo Saarinen, Fredrik Sävje and Jasjeet Sekhon. This is an open access article distributed under the terms of a Creative Commons license (CC BY-NC-ND 4.0).", "corrected": "This work is licensed under a Creative Commons \\hl{Attribution-NonCommercial-NoDerivatives} 4.0 International License.", "evidence": "I compared the two pages. They establish the mismatch, and they do not establish which licence governs the rejoinder.", "problem": "The published PDF identifies its licence in three places, and the three do not agree. The words on the cover page name the Attribution licence. The icon on the same line and the footer of the first page of the rejoinder give a different licence, Attribution-NonCommercial-NoDerivatives. The cover page is a wrapper added by the distributor, and it is not part of the text of the rejoinder.", "what_survives": "No statement of the rejoinder is affected. The correction below makes the words of the cover page agree with the icon beside them and with the footer of p.~85."}, "url": "https://pmaronow.github.io/llm-audit/works/P047/#P047-N001", "work": {"arxiv_id": null, "doi": "10.1353/obs.2025.a956844", "title": "Rejoinder: Nonparametric Identification Is Not Enough, but Randomized Controlled Trials Are", "url": "https://pmaronow.github.io/llm-audit/works/P047/"}, "work_id": "P047"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [], "formal_release": null, "formal_statements": [], "headline": "The address of the Harvard School of Public Health should read Boston, MA 02115, not Cambridge, MA 02115.", "id": "P047-N002", "location": "Author block, address under “James M. Robins”, p. 85 (PDF p. 2)", "pdf_page": 2, "printed_page": "85", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "James M. Robins\n\nHarvard School of Public Health\n\n\\hl{Cambridge}, MA 02115, USA", "corrected": "James M. Robins\n\nHarvard School of Public Health\n\n\\hl{Boston}, MA 02115, USA", "evidence": "I checked the page against two public sources. The \\href{https://hsph.harvard.edu/contact/}{contact page of the school}, now the Harvard T.H. Chan School of Public Health, gives the primary mailing address above. An \\href{https://pe.usps.com/Archive/HTML/DMMArchive20050106/print/L007.htm}{archived labelling list of the U.S. Postal Service} pairs the ZIP code 02115 with the label ``BOSTON MA 02115''.", "problem": "The city does not match the school and the ZIP code that are printed with it. The school gives its primary mailing address as 677 Huntington Avenue, Boston, MA 02115. The manuscript source has the same line as the published page.", "what_survives": "Only the city changes. The correction makes no claim about a street address or an office, and no statement of the rejoinder is affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P047/#P047-N002", "work": {"arxiv_id": null, "doi": "10.1353/obs.2025.a956844", "title": "Rejoinder: Nonparametric Identification Is Not Enough, but Randomized Controlled Trials Are", "url": "https://pmaronow.github.io/llm-audit/works/P047/"}, "work_id": "P047"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of four lines at the top of page 86. The first two lines end the second bullet: the practitioner must impose untestable assumptions on either the propensity score function or the conditional expectation function of the outcomes. A red outline surrounds the word untestable. The next two lines begin a new paragraph: These insights are decades old, going back to Robins and Ritov (1997), but they are not yet common knowledge.", "caption": "Top of p. 86 (PDF p. 3): the end of the second bullet of Section 1 and the two lines that follow it. The outline marks the word “untestable”.", "url": "https://pmaronow.github.io/llm-audit/works/P047/exhibits/P047-N003.png"}, {"alt": "Crop of the three lines of the second bullet at the foot of page 85. They read: If the propensity score is not known, such as in an observational study, there exist neither uniformly consistent estimators nor uniform (i.e., honest) large-sample confidence intervals whose widths are shrinking with the sample size. To achieve these properties, and the page ends there. A red outline surrounds the words To achieve these properties.", "caption": "Foot of p. 85 (PDF p. 2): the second bullet of Section 1. The outline marks the opening words of the sentence at issue, which continues on p. 86.", "url": "https://pmaronow.github.io/llm-audit/works/P047/exhibits/P047-N003-b.png"}], "formal_release": null, "formal_statements": [{"id": "P047-T3", "url": "https://pmaronow.github.io/llm-audit/works/P047/#P047-T3"}], "headline": "In the second bullet, “untestable” means that the data cannot verify the added assumptions; it does not mean that those assumptions have no refutable implications.", "id": "P047-N003", "location": "Section 1, second bullet, last sentence, pp. 85–86 (PDF pp. 2–3); the word is in the first line of p. 86", "pdf_page": 2, "printed_page": "85–86", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P047-T3"], "text": {"as_printed": "To achieve these properties, the practitioner must impose \\hl{untestable} assumptions on either the propensity score function or the conditional expectation function of the outcomes.", "corrected": "To achieve these properties, the practitioner must impose \\hl{complexity-reducing} assumptions on either the propensity score function or the conditional expectation function of the outcomes.\n\nThe replacement is the rejoinder's own term on p.~87. The sentence may instead keep ``untestable'' and say that it means not verifiable uniformly from the observed data, which does not exclude refutable implications.", "intended_meaning": "The bullet concerns an unknown propensity score $\\pi(w)$, the probability of treatment at the covariate value $w$. The sentence says that uniform consistency and honest intervals of shrinking width then require further restrictions on $\\pi$ or on the outcome regression, and that the data cannot certify such restrictions. Sections 2.3 and 2.5 support this reading, since they discuss how hard it is to judge whether a propensity score belongs to a given functional class. The sentence is correct on this reading. I take a uniform covariate and two laws that differ only in the propensity score, with $\\pi = 1/2$ in the first, and $\\pi(w) = 1/2 - \\eta$ for $w < 1/2$ and $\\pi(w) = 1/2 + \\eta$ for $w \\ge 1/2$ in the second. In samples of $n$ independent and identically distributed units, the total variation distance between the two laws is at most $n\\eta$. Letting $\\eta \\to 0$, I find that a procedure that certifies a constant propensity score with probability at most $\\alpha$ under every nonconstant one also certifies it with probability at most $\\alpha$ under the first law.\n\nThe word is too strong if it is read as ``having no refutable implications''. Under this sampling assumption and the bullet's ignorability assumption, a propensity score equal to an unknown constant $p \\in [1/5, 4/5]$ is a restriction under which both properties hold for outcomes in $[0,1]$, and the data can refute it. For $n = 20$, an exact one-sided Fisher test of association between treatment and the indicator of $w \\ge 1/2$ has size at most $1/20$ for every $p$ and every covariate distribution. With a uniform covariate, its power against a propensity score that jumps from $1/5$ to $4/5$ at $w = 1/2$ is\n\\[\n\\frac{72825613746176}{95367431640625} \\approx 0.764 .\n\\]\nThe example restricts the model, and the impossibility result of the bullet is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P047/#P047-N003", "work": {"arxiv_id": null, "doi": "10.1353/obs.2025.a956844", "title": "Rejoinder: Nonparametric Identification Is Not Enough, but Randomized Controlled Trials Are", "url": "https://pmaronow.github.io/llm-audit/works/P047/"}, "work_id": "P047"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the four lines of the reference entry Hendrycks D, Burns C, Basart S, et al. (2021) Measuring massive Multitask Language Understanding. In: International Conference on Learning Representations, Singapore, Thu Apr 24 - Mon Apr 28th, 2025. A red outline surrounds Singapore, Thu Apr at the end of the third line and another surrounds 24 - Mon Apr 28th, 2025. in the fourth line.", "caption": "Reference list, p. 5 (PDF p. 5), right column, entry for Hendrycks et al. (2021). The outlines mark the place and date line ending in 2025.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-E001.png"}], "formal_release": null, "formal_statements": [], "headline": "The reference dated 2021 carries a place and date line for 2025, which should be deleted.", "id": "P048-E001", "location": "Reference list, p. 5 (PDF p. 5), right column, entry “Hendrycks D, Burns C, Basart S, et al. (2021)”", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Hendrycks D, Burns C, Basart S, et al. (2021) Measuring massive Multitask Language Understanding. In: International Conference on Learning Representations, \\hl{Singapore, Thu Apr 24 -- Mon Apr 28th, 2025}.", "corrected": "Hendrycks D, Burns C, Basart S, et al. (2021) Measuring massive Multitask Language Understanding. In: International Conference on Learning Representations\\del{, Singapore, Thu Apr 24 -- Mon Apr 28th, 2025}.", "evidence": "The evidence is a check of the page and of the calendar: 24 and 28 April fall on a Thursday and a Monday in 2025, and on a Saturday and a Wednesday in 2021. I conclude that the printed days are those of 2025. I did not check the place and dates of the 2021 meeting against its proceedings, and the correction omits them.", "problem": "The entry is dated 2021, and the text on p.~4 cites it as ``Hendrycks et al., 2021'', but its last line has the place and dates of a meeting in 2025.", "what_survives": "The benchmark is identified by its title, authors and year, and the argument on p.~4 about adaptation to benchmarks is unaffected."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-E001", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the three lines of the reference entry Björkegren D, Blumenstock JE and Samsun K (2020) Training machine learning to anticipate manipulation. arXiv preprint arXiv:2004.03865. A red outline surrounds the name Samsun K.", "caption": "Reference list, p. 5 (PDF p. 5), left column, last entry. The outline marks “Samsun K”.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-E002.png"}], "formal_release": null, "formal_statements": [], "headline": "The third author of the reference is Samsun Knight, and “Samsun K” should read “Knight S”.", "id": "P048-E002", "location": "Reference list, p. 5 (PDF p. 5), left column, entry “Björkegren D, Blumenstock JE and Samsun K (2020)”", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Björkegren D, Blumenstock JE and \\hl{Samsun K} (2020) ``Training machine learning to anticipate manipulation''. arXiv preprint arXiv:2004.03865.", "corrected": "Björkegren D, Blumenstock JE and \\hl{Knight S} (2020) ``Training machine learning to anticipate manipulation''. arXiv preprint arXiv:2004.03865.", "evidence": "I base this correction on the arXiv listing for arXiv:2004.03865, whose author line reads ``Daniel Björkegren, Joshua E. Blumenstock, Samsun Knight''.", "problem": "The entry lists its authors by surname and initials, and it prints the third author's given name as the surname and the initial of the surname as the given name.", "what_survives": "The arXiv identifier locates the work, and the discussion of strategic adaptation on p.~4 does not depend on the entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-E002", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the first four lines of note 4 on page 5. The note begins: Following Lin (2013)'s, then in quotation marks Agnostic notes on regression adjustments to experimental, then literal belief in assumptions is not requisite to learning from settings in which they are applied. A red outline surrounds the word experimental and the closing quotation mark that follows it directly. No comma follows the quotation mark.", "caption": "Note 4, p. 5 (PDF p. 5), left column, first four lines. The outline marks the end of the quoted title, which stops at “experimental”.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-E003.png"}, {"alt": "Crop of the three lines of the reference entry Lin W (2013) Agnostic notes on regression adjustments to experimental data: reexamining Freedman's critique. Annals of Applied Statistics 7(1): 295-318. A red outline surrounds the words data: reexamining Freedman's critique.", "caption": "Reference list, p. 5 (PDF p. 5), right column, entry for Lin (2013). The outline marks the part of the title that note 4 omits.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-E003-reference.png"}], "formal_release": null, "formal_statements": [], "headline": "Note 4 quotes the title of Lin (2013) only as far as “experimental” and has no comma after it; the note should cite Lin (2013) without the title.", "id": "P048-E003", "location": "Note 4, p. 5 (PDF p. 5), left column, first sentence", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Following Lin (2013)\\hl{’s ``Agnostic notes on regression adjustments to experimental''} literal belief in assumptions is not requisite to learning from settings in which they are applied:", "corrected": "Following Lin (2013)\\hl{,} literal belief in assumptions is not requisite to learning from settings in which they are applied:", "evidence": "I compared note 4 with the entry for Lin (2013) in the reference list on the same page.", "problem": "The quoted title ends at the adjective ``experimental''. The reference list on the same page prints the title as ``Agnostic notes on regression adjustments to experimental data: reexamining Freedman's critique''. The sentence also has no comma between the introductory phrase and ``literal belief''.", "what_survives": "The quotation about OLS adjustment that follows in the note is unaffected, as is the passage on p.~2 that the note supports."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-E003", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the three lines of the reference entry Hastie T, Tibshirani R, Friedman JH, et al. (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer, Vol. 2. A red outline surrounds et al. in the first line and another surrounds Vol. 2. at the end of the third line.", "caption": "Reference list, p. 5 (PDF p. 5), right column, entry for Hastie, Tibshirani and Friedman (2009). The outlines mark “et al.” and “Vol. 2.”", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-E004.png"}], "formal_release": null, "formal_statements": [], "headline": "The book has three authors and is a second edition: “et al.” should be deleted, and “Vol. 2” should read “2nd ed.”", "id": "P048-E004", "location": "Reference list, p. 5 (PDF p. 5), right column, entry “Hastie T, Tibshirani R, Friedman JH, et al. (2009)”", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Hastie T, Tibshirani R, Friedman JH, \\hl{et al.} (2009) \\emph{The Elements of Statistical Learning: Data Mining, Inference, and Prediction}. New York: Springer, \\hl{Vol. 2}.", "corrected": "Hastie T, Tibshirani R \\hl{and} Friedman JH (2009) \\emph{The Elements of Statistical Learning: Data Mining, Inference, and Prediction}. \\hl{2nd ed.} New York: Springer.", "evidence": "The evidence is a public source. The title page of the book, as the publisher hosts it, names Trevor Hastie, Robert Tibshirani and Jerome Friedman and reads ``Second Edition'', and the publisher's bibliographic information for the book gives ``Edition number: 2''.", "problem": "The entry adds ``et al.'' after the names of all three authors of the book, and it describes the second edition of the book as a second volume.", "what_survives": "The book is identifiable by its title, authors and year, and the definition of prediction in note 6 does not depend on the entry."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-E004", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of the right column of page 2: item 3 of the numbered list, We cannot have knowledge of the future, and the first three lines of the next paragraph, which begins We agree that if the only criteria for a successful research program is the guaranteed transportability of a causal parameter. A red outline surrounds the word criteria.", "caption": "Section “How knowledge travels”, p. 2 (PDF p. 2), right column, after the numbered list. The outline marks “criteria”.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-E005.png"}], "formal_release": null, "formal_statements": [], "headline": "“The only criteria … is” should read “the only criterion … is”.", "id": "P048-E005", "location": "Section “How knowledge travels”, p. 2 (PDF p. 2), right column, first sentence after the numbered list", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We agree that if the only \\hl{criteria} for a successful research program is the guaranteed transportability of a causal parameter, then this contradiction would be troubling.", "corrected": "We agree that if the only \\hl{criterion} for a successful research program is the guaranteed transportability of a causal parameter, then this contradiction would be troubling.", "evidence": "I checked the printed page.", "problem": "The sentence names one condition, the guaranteed transportability of a causal parameter, and its verb is the singular ``is'', but its noun is the plural ``criteria''.", "what_survives": "The conditional argument of the sentence is unchanged, and nothing else depends on the word."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-E005", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the left column of page 2. The paragraph that begins To contextualize this stance says that standard analyses of RCTs do not make zero-modeling assumptions, with the last two words in quotation marks and followed by the note number 5. A red outline surrounds the hyphenated word zero-modeling, which stands inside one line.", "caption": "Section “Agnosticism and causality”, p. 2 (PDF p. 2), left column. The outline marks “zero-modeling” in the short quotation from Munger (2023).", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-N001.png"}, {"alt": "Crop of the first five lines of the left column of page 2. The quotation reads: fully agnostic: they ensure unconfoundedness and positivity (internal validity) through research design, with zero modeling assumptions about the structure of the world, followed by the citation Munger 2023: p. 3. A red outline surrounds the two words zero modeling, which have a space between them and no hyphen.", "caption": "The same column, lines 1–5 of the page. The outline marks “zero modeling”, printed without a hyphen in the long quotation from Munger (2023: p. 3).", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-N001-quotation.png"}], "formal_release": null, "formal_statements": [], "headline": "The short quotation from Munger (2023) should read “zero modeling assumptions”, without a hyphen, as the longer quotation above it does.", "id": "P048-N001", "location": "Section “Agnosticism and causality”, p. 2 (PDF p. 2), left column, first sentence of the paragraph beginning “To contextualize this stance”", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "To contextualize this stance, we first note that standard analyses of RCTs do not make ``\\hl{zero-modeling} assumptions.''$^{5}$", "corrected": "To contextualize this stance, we first note that standard analyses of RCTs do not make ``\\hl{zero modeling} assumptions.''$^{5}$", "evidence": "I base this correction on the page and the published text of Munger (2023), which prints ``zero modeling'' on its p.~3.", "problem": "The article quotes the same words of Munger (2023, p.~3) twice in this column. The long quotation at the top of the column has ``zero modeling assumptions''. The short quotation has ``zero-modeling assumptions'', and the hyphen is not a line-break hyphen, since it falls inside a line.", "what_survives": "Both forms are read as ``no modeling assumptions'', and the argument of the section is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-N001", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the heading Why knowledge decays and the first five lines of the section in the right column of page 3. The first sentence says that Munger provides the concept of a decay rate r, which, even in a best case scenario, is both positive and unpredictable, with the citation p 6. A red outline surrounds the word best at the end of the second line and another surrounds the word case at the start of the third line. No hyphen follows best.", "caption": "Section “Why knowledge decays”, p. 3 (PDF p. 3), right column, first lines. The outlines mark “best” and “case”, printed without a hyphen at the end of one line and the start of the next.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The quotation from Munger (2023, p. 6) should read “best-case scenario”, with a hyphen.", "id": "P048-N003", "location": "Section “Why knowledge decays”, p. 3 (PDF p. 3), right column, first sentence", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "In formalizing the idea of knowledge decay, Munger provides the concept of a decay rate $r$, which, even in a ``\\hl{best case} scenario\\ldots\\ is both positive and unpredictable'' (p 6).", "corrected": "In formalizing the idea of knowledge decay, Munger provides the concept of a decay rate $r$, which, even in a ``\\hl{best-case} scenario\\ldots\\ is both positive and unpredictable'' (p 6).", "evidence": "I base this correction on a comparison with the published text of Munger (2023), \\emph{Research \\& Politics} 10(3).", "problem": "Munger (2023, p.~6) prints ``best-case scenario'', and the quotation drops the hyphen. The other passages that were compared with that article agree with it, apart from the hyphen in \\fid{P048-N001}: the three premises of the contradiction on p.~2, the words ``assumption-free'', the symbol $r$, the long quotation from its p.~3, the sentence quoted in note 7, and the page numbers 2, 3 and 6.", "what_survives": "The quoted words keep their meaning, and the article's account of Munger's argument is unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-N003", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the notes on page 5: the last line of note 6, the three lines of note 7 and the first line of note 8. Note 7 reads: Munger acknowledges that beyond prediction, There are other goals, of course, and social science is no stranger to methodological pluralism (p 2). The quotation opens before There and closes after the full stop that follows (p 2). A red outline surrounds (p 2), the full stop and the closing quotation mark.", "caption": "Note 7, p. 5 (PDF p. 5), left column. The outline marks the page reference “(p 2).” and the closing quotation mark printed after it.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "In note 7 the closing quotation mark should come before the page reference “(p 2)”.", "id": "P048-N007", "location": "Note 7, p. 5 (PDF p. 5), left column", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Munger acknowledges that beyond prediction, ``There are other goals, of course, and social science is no stranger to methodological pluralism \\hl{(p 2).''}", "corrected": "Munger acknowledges that beyond prediction, ``There are other goals, of course, and social science is no stranger to methodological pluralism\\hl{'' (p 2).}", "evidence": "I checked the printed page.", "problem": "The closing quotation mark follows the page reference and the full stop, and the reference is printed inside the quotation. On p.~3 the same construction closes the quotation first: ``\\ldots\\ is both positive and unpredictable'' (p 6).", "what_survives": "The quotation is read as ending at ``pluralism'', and nothing else changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-N007", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the notes on page 5: the five lines of note 8 and the first line of note 9. The last sentence of note 8 reads: Einstein himself was emphatically skeptical of the eventual possibility of using the atom to generate energy Moszkowski (1921/2014). A red outline surrounds Moszkowski (1921/2014) and the full stop after it.", "caption": "Note 8, p. 5 (PDF p. 5), left column. The outline marks the citation “Moszkowski (1921/2014)”, which follows the sentence without enclosing parentheses.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-N008.png"}, {"alt": "Crop of the three lines of the reference entry Moszkowski A (2014). Einstein the searcher: his work explained from dialogues with einstein. In: Translated from the Original German by Henry L. Brose. Oxfordshire: Routledge. The title is in roman type and the words Translated from the Original German by Henry L. Brose are in italics. A red outline surrounds the passage from einstein, with a lower-case e, through In: to Brose.", "caption": "Reference list, p. 5 (PDF p. 5), right column, entry for Moszkowski. The outline marks the lower-case “einstein”, the “In:” and the translator credit set in italics.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-N008-reference.png"}], "formal_release": null, "formal_statements": [], "headline": "The citation of Moszkowski in note 8 lacks its parentheses, and the reference entry prints the translator credit as a containing work after “In:” and the second “Einstein” of the title in lower case.", "id": "P048-N008", "location": "Note 8, last sentence, p. 5 (PDF p. 5), left column; and the entry “Moszkowski A (2014)” in the reference list, p. 5, right column", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Note 8.} Einstein himself was emphatically skeptical of the eventual possibility of using the atom to generate energy \\hl{Moszkowski (1921/2014)}.\n\n\\emph{Reference list.} Moszkowski A (2014). Einstein the searcher: his work explained from dialogues with \\hl{einstein}. \\hl{In: \\emph{Translated from the Original German by Henry L. Brose}}. Oxfordshire: Routledge.", "corrected": "\\emph{Note 8.} Einstein himself was emphatically skeptical of the eventual possibility of using the atom to generate energy \\hl{(Moszkowski, 1921/2014)}.\n\n\\emph{Reference list.} Moszkowski A (2014). \\emph{Einstein the searcher: his work explained from dialogues with \\hl{Einstein}}. \\del{In:} \\hl{Translated from the Original German by Henry L. Brose}. Oxfordshire: Routledge.\n\nIn the entry the title of the book is set in italics and the translator credit in roman.", "evidence": "I checked the printed page.", "problem": "In note 8 the citation is appended to the sentence without the parentheses that a citation outside the grammar of the sentence takes. In the reference entry the translator credit follows ``In:'' and is set in italics, as the title of a containing work would be, while the title of the book is set in roman, and the second ``Einstein'' of the title is in lower case.", "what_survives": "The book remains identifiable, and the dates are compatible with an original year paired with the year of the cited edition. The argument about basic science and later innovation does not depend on the form of the citation."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-N008", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the three lines of the reference entry Goodhart CAE (1984) Problems of monetary management: the UK experience. In: Monetary Theory and Practice. Berlin: Springer, 91-121. A red outline surrounds Berlin: at the end of the second line and another surrounds Springer, with its comma, at the start of the third line.", "caption": "Reference list, p. 5 (PDF p. 5), right column, entry for Goodhart (1984). The outlines mark “Berlin:” and “Springer”.", "url": "https://pmaronow.github.io/llm-audit/works/P048/exhibits/P048-N009.png"}], "formal_release": null, "formal_statements": [], "headline": "The 1984 book Monetary Theory and Practice was published by Macmillan Press in London, not by Springer in Berlin.", "id": "P048-N009", "location": "Reference list, p. 5 (PDF p. 5), right column, entry “Goodhart CAE (1984)”", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Goodhart CAE (1984) Problems of monetary management: the UK experience. In: \\emph{Monetary Theory and Practice}. \\hl{Berlin: Springer}, 91--121.", "corrected": "Goodhart CAE (1984) Problems of monetary management: the UK experience. In: \\emph{Monetary Theory and Practice}. \\hl{London: Macmillan Press}, 91--121.", "evidence": "I base this correction on the front matter of a reprint of the first edition: its title page reads ``MACMILLAN PRESS'' and ``LONDON'', and its copyright page reads ``First published 1984 by'' and ``London and Basingstoke''.", "problem": "The entry gives Berlin and Springer as the place and publisher of the 1984 book, which was published by Macmillan Press in London. Springer hosts a later reprint.", "what_survives": "The entry identifies the chapter and its pages 91--121 correctly, and the discussion of Goodhart's law on pp.~3--4 does not depend on the imprint."}, "url": "https://pmaronow.github.io/llm-audit/works/P048/#P048-N009", "work": {"arxiv_id": null, "doi": "10.1177/20531680241307942", "title": "Gnostic Notes on Temporal Validity", "url": "https://pmaronow.github.io/llm-audit/works/P048/"}, "work_id": "P048"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines at the top of the right column of page 484. The third line reads: have testable implications. Pearl is quick to point the first part. A red outline surrounds the two words point the. The fourth line begins: while neglecting the second.", "caption": "Page 484 (PDF p. 4), top of the right column. The outline marks “point the” in the third line, where the word “out” is missing between the two words.", "url": "https://pmaronow.github.io/llm-audit/works/P050/exhibits/P050-E001.png"}], "formal_release": null, "formal_statements": [], "headline": "The word “out” is missing: the sentence should read “Pearl is quick to point out the first part while neglecting the second.”", "id": "P050-E001", "location": "p. 484 (PDF p. 4), right column, first (continued) paragraph, third line", "pdf_page": 4, "printed_page": "484", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Pearl is quick to \\hl{point the} first part while neglecting the second.", "corrected": "Pearl is quick to point \\hl{out} the first part while neglecting the second.", "evidence": "I checked the printed page.", "problem": "The phrase ``point out'' has lost its second word. The preceding sentence supplies the two parts: the additional independence assumptions make Pearl's model more powerful, and they have no testable implications.", "what_survives": "The meaning is clear from the preceding sentence, and nothing in the review depends on the missing word."}, "url": "https://pmaronow.github.io/llm-audit/works/P050/#P050-E001", "work": {"arxiv_id": "2003.11635", "doi": "10.1080/01621459.2020.1721245", "title": "Review of The Book of Why: The New Science of Cause and Effect", "url": "https://pmaronow.github.io/llm-audit/works/P050/"}, "work_id": "P050"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the four lines that end the review on page 485. The last two lines read: question whether, opening quotation mark, causal diagrams provide a complete and systematic way of finding a solution [to confounding], full stop, closing quotation mark. A red outline surrounds the single word a, between finding and solution.", "caption": "Page 485 (PDF p. 5), left column, the last sentence of the review. The outline marks the word “a” in “finding a solution [to confounding]”.", "url": "https://pmaronow.github.io/llm-audit/works/P050/exhibits/P050-N006.png"}, {"alt": "Crop of the five-line block quotation on page 483. It reads: There is now an almost universal consensus, at least among epidemiologists, philosophers and social scientists, that (1) confounding needs and has a causal solution, and (2) causal diagrams provide a complete and systematic way of finding that solution. (p. 141). A red outline surrounds the word that, at the start of the last line.", "caption": "Page 483 (PDF p. 3), left column, the block quotation cited to p. 141 of the book, for comparison. The outline marks the word “that” in “finding that solution”.", "url": "https://pmaronow.github.io/llm-audit/works/P050/exhibits/P050-N006-b.png"}], "formal_release": null, "formal_statements": [], "headline": "The closing sentence quotes p. 141 of the book as “finding a solution”, while the block quotation on p. 483 has “finding that solution”; the change of word is not marked.", "id": "P050-N006", "location": "p. 485 (PDF p. 5), left column, last sentence of the review; compare the block quotation on p. 483 (PDF p. 3), left column", "pdf_page": 5, "printed_page": "485", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "\\emph{Block quotation, p.~483.} There is now an almost universal consensus, at least among epidemiologists, philosophers and social scientists, that (1) confounding needs and has a causal solution, and (2) causal diagrams provide a complete and systematic way of finding \\hl{that} solution. (p.~141)\n\n\\emph{Last sentence, p.~485.} \\ldots\\ they might question whether ``causal diagrams provide a complete and systematic way of finding \\hl{a} solution [to confounding].''", "corrected": "\\ldots\\ they might question whether ``causal diagrams provide a complete and systematic way of finding \\hl{[a]} solution [to confounding].''", "evidence": "I compared the two printed pages, which show that the quotations differ. I did not read p.~141 of the book, so I cannot establish which quotation matches it.", "problem": "The review quotes the same sentence of the book twice, and the two quotations differ in one word. In the closing sentence the square brackets mark ``to confounding'' as an insertion, but ``a'' stands outside the brackets where the block quotation has ``that''. The two quotations cannot both reproduce the wording of the book.", "what_survives": "The sense of the quotation is unchanged, and the argument of the closing paragraph does not depend on the word. I use the block quotation on p.~483 as the reference wording below and mark the substitution with brackets; the final wording should be compared with p.~141 of the book."}, "url": "https://pmaronow.github.io/llm-audit/works/P050/#P050-N006", "work": {"arxiv_id": "2003.11635", "doi": "10.1080/01621459.2020.1721245", "title": "Review of The Book of Why: The New Science of Cause and Effect", "url": "https://pmaronow.github.io/llm-audit/works/P050/"}, "work_id": "P050"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines at the top of the right column of page 348. They read: for completely randomized experiments, if outcomes are binary, then under the sharp null hypothesis of no treatment effect, all test statistics may be written as a function of the difference-in-means statistic. Stronger statements are likely available. A red outline surrounds the words all test statistics.", "caption": "Section 4, p. 348 (PDF p. 3), top of the right column: the claim, which ends the sentence. The outline marks “all test statistics”.", "url": "https://pmaronow.github.io/llm-audit/works/P053/exhibits/P053-E001.png"}, {"alt": "Crop of four lines at the foot of the left column of page 348. They read: ference in means (e.g., the absolute value of tau-hat). For such test statistics, the resulting inferences may substantively differ. Here, we state a weaker claim verifiable at full generality, which has been known since Copas [(1973), equation (9)]: A red outline surrounds the words at full generality and the comma after them.", "caption": "Section 4, p. 348 (PDF p. 3), foot of the left column: the start of the same sentence. The outline marks “at full generality”.", "url": "https://pmaronow.github.io/llm-audit/works/P053/exhibits/P053-E001-b.png"}, {"alt": "Crop of five lines in the right column of page 346. They read: one of the constituent hypotheses is false. (iii) We also note an error in Ding's (2017) Theorem 7 and suggest a refinement that is correct at full generality. Note, we consider only asymptotic results, and do not discuss the finite N differences between exact. A red outline surrounds the words at full generality and the full stop after them.", "caption": "Introduction, item (iii), p. 346 (PDF p. 1), right column. The outline marks “at full generality”.", "url": "https://pmaronow.github.io/llm-audit/works/P053/exhibits/P053-E001-c.png"}], "formal_release": null, "formal_statements": [{"id": "P053-T1", "url": "https://pmaronow.github.io/llm-audit/works/P053/#P053-T1"}, {"id": "P053-T2", "url": "https://pmaronow.github.io/llm-audit/works/P053/#P053-T2"}], "headline": "In the replacement for Ding’s Theorem 7, “all test statistics” should read “all test statistics based on the two-by-two treatment-by-outcome table”, which is the class that the claim concerns and for which it is correct.", "id": "P053-E001", "location": "Section 4, p. 348 (PDF p. 3), the sentence beginning “Here, we state a weaker claim”, from the foot of the left column to the top of the right column; Introduction, item (iii), p. 346 (PDF p. 1)", "pdf_page": 3, "printed_page": "348", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P053-T1", "P053-T2"], "text": {"as_printed": "Section 4, p.~348: Here, we state a weaker claim verifiable \\hl{at full generality}, which has been known since Copas [(1973), equation (9)]: for completely randomized experiments, if outcomes are binary, then under the sharp null hypothesis of no treatment effect, \\hl{all test statistics} may be written as a function of the difference-in-means statistic. Stronger statements are likely available.\n\nIntroduction, item (iii), p.~346: We also note an error in Ding's (2017) Theorem 7 and suggest a refinement that is correct \\hl{at full generality}.", "corrected": "Section 4, p.~348: \\ldots for completely randomized experiments, if outcomes are binary, then under the sharp null hypothesis of no treatment effect, all test statistics \\hl{based on the two-by-two treatment-by-outcome table} may be written as a function of the difference-in-means statistic.\n\nIntroduction, item (iii), p.~346: \\ldots a refinement that is correct at full generality \\hl{for statistics based on the two-by-two treatment-by-outcome table}.\n\nThe statement in full: in a completely randomized experiment with fixed positive treatment and control counts $N_1$ and $N_0$ and binary outcomes, under the sharp null hypothesis of no treatment effect, every test statistic that depends on the data only through the two-by-two treatment-by-outcome table is a function of the difference-in-means statistic $\\widehat\\tau$. With $K$ total successes and $a$ treated successes,\n\\[\na = \\frac{N_1 N_0 \\widehat\\tau + N_1 K}{N_1 + N_0},\n\\]\nwhich determines all four cells of the table.", "intended_meaning": "The sentence takes the words ``all test statistics'' from Ding's Theorem 7, which the comment quotes a few lines above, and it replaces ``are equivalent to'' by ``may be written as a function of''. In the preprint of Ding's article (arXiv:1402.0142v4, Supplementary Material, p.~38), the proof of Theorem 7 begins ``All test statistics are functions of the two by two table''. The preprint was not compared with the published article and its supplement. Read with that proof, ``all test statistics'' ranges over the statistics based on the two-by-two treatment-by-outcome table, and ``at full generality'' covers all of them, including those that are not monotone in the difference in means $\\widehat\\tau$. The comment's own example, $|\\widehat\\tau|$, is such a statistic. The article of Copas that the sentence cites is titled ``Randomization models for the matched and unmatched $2 \\times 2$ tables''; its equation (9) was not read. With this reading the claim is correct.\n\nTo see this, let $N_1$ and $N_0$ be the fixed positive numbers of treated and control units, and let $N = N_1 + N_0$. Under the sharp null hypothesis, unit $i$ has one binary outcome $y_i = Y_i(1) = Y_i(0)$, and the number of successes $K = \\sum_i y_i$ is the same for every assignment. If $a$ of the treated units are successes, then the table has cells $a$, $N_1 - a$, $K - a$ and $N_0 - K + a$, and\n\\[\n\\widehat\\tau = \\frac{a}{N_1} - \\frac{K - a}{N_0} = \\frac{N a - N_1 K}{N_1 N_0}.\n\\]\nSolving for $a$, I obtain\n\\[\na = \\frac{N_1 N_0 \\widehat\\tau + N_1 K}{N_1 + N_0}.\n\\]\nThen $\\widehat\\tau$ determines $a$, and $a$ determines the four cells. This implies that every statistic based on the table is a function of $\\widehat\\tau$, whether or not it is monotone in $\\widehat\\tau$. A statistic that depends on the treated outcomes and on the control outcomes, and not on their order within either sample, is based on the table, since a binary sample is determined up to order by its number of successes.\n\nThe sentence does not name this class. If ``all test statistics'' is taken to include statistics that use the labels of the units or fixed covariates, then the claim is false. For example, suppose that there are $N = 4$ units with $N_1 = N_0 = 2$, that the outcomes are $y = (1,1,0,0)$ under both treatments, and that $X = (0,1,2,3)$ is a fixed covariate. Let $Z$ be the vector of treatment indicators, and let $\\widehat\\beta$ be the coefficient on $Z$ in the least-squares regression of the outcome on a constant, $Z$ and $X$. The six assignments are equally likely.\n\\begin{audittable}{llrrr}{The six assignments in the example with four units. The cells of the two-by-two table are listed as (treated successes, treated failures, control successes, control failures).}\nAssignment $Z$ & Cells & $\\widehat\\tau$ & $\\widehat\\beta$ & $Z_1 y_1$ \\\\\n\\hline\n$(1,1,0,0)$ & $(2,0,0,2)$ & $1$ & $1$ & $1$ \\\\\n$(1,0,1,0)$ & $(1,1,1,1)$ & $0$ & $-1/2$ & $1$ \\\\\n$(1,0,0,1)$ & $(1,1,1,1)$ & $0$ & $0$ & $1$ \\\\\n$(0,1,1,0)$ & $(1,1,1,1)$ & $0$ & $0$ & $0$ \\\\\n$(0,1,0,1)$ & $(1,1,1,1)$ & $0$ & $1/2$ & $0$ \\\\\n$(0,0,1,1)$ & $(0,2,2,0)$ & $-1$ & $-1$ & $0$ \\\\\n\\end{audittable}\nThe assignments $(1,0,1,0)$ and $(0,1,0,1)$ have the same cells and $\\widehat\\tau = 0$, but $\\widehat\\beta$ is $-1/2$ at the first and $1/2$ at the second. The statistic $Z_1 y_1$, which uses no covariate, is $1$ at the first and $0$ at the second. Neither statistic is a function of $\\widehat\\tau$. Repeating the four units $m$ times gives the same values at every $N = 4m$, so the asymptotic conditions on p.~347 do not change this.\n\nNothing else in the comment uses the sentence. The distinction between functional dependence and identical inference, the variance comparisons of Sections 2 and 3, and the conclusion are unaffected. In the example, the assignment with $\\widehat\\tau = 1$ has upper-tail p-value $1/6$ for $\\widehat\\tau$ and $1/3$ for $|\\widehat\\tau|$ in the randomization test. This is the comment's own point: a function of $\\widehat\\tau$ need not give the same inference as $\\widehat\\tau$."}, "url": "https://pmaronow.github.io/llm-audit/works/P053/#P053-E001", "work": {"arxiv_id": null, "doi": "10.1214/16-STS582", "title": "Understanding Ding's Apparent Paradox", "url": "https://pmaronow.github.io/llm-audit/works/P053/"}, "work_id": "P053"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of six lines in the right column of page 346. They read: ear model; Samii and Aronow (2012) give the implied variance of this test a randomization basis, by establishing that the implied variance is equivalent to the randomization distribution of the difference-in-means estimator if the treatment effect is assumed to be constant and equal to the observed estimate. Gerber and. Two red outlines surround the words equivalent to the, at the end of the third line, and randomization distribution, at the start of the fourth line.", "caption": "Section 2, first paragraph, p. 346 (PDF p. 1), right column. The outlines mark “equivalent to the randomization distribution”, which runs over two lines.", "url": "https://pmaronow.github.io/llm-audit/works/P053/exhibits/P053-N001.png"}], "formal_release": null, "formal_statements": [{"id": "P053-T6", "url": "https://pmaronow.github.io/llm-audit/works/P053/#P053-T6"}], "headline": "The implied variance is equivalent to the variance of the randomization distribution of the difference-in-means estimator; the words “the variance of” are missing.", "id": "P053-N001", "location": "Section 2, first paragraph, p. 346 (PDF p. 1), right column, the clause that follows “linear model;”", "pdf_page": 1, "printed_page": "346", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P053-T6"], "text": {"as_printed": "Samii and Aronow (2012) give the implied variance of this test a randomization basis, by establishing that the implied variance is \\hl{equivalent to the randomization distribution} of the difference-in-means estimator if the treatment effect is assumed to be constant and equal to the observed estimate.", "corrected": "Samii and Aronow (2012) give the implied variance of this test a randomization basis, by establishing that the implied variance is equivalent to \\hl{the variance of} the randomization distribution of the difference-in-means estimator if the treatment effect is assumed to be constant and equal to the observed estimate.", "intended_meaning": "As printed, the sentence compares a variance with a distribution. The intended comparison is with the variance of that distribution. The equivalence is asymptotic, which is the scope that the Introduction (p.~346) sets for the whole comment, and with this reading the statement is correct. Let $\\text{SSR}$ be the residual sum of squares from the regression of the outcome on a constant and the treatment indicator. When the outcomes are imputed under a constant effect equal to the observed estimate, the randomization distribution of the difference in means has variance $N\\,\\text{SSR}/(N_1 N_0 (N-1))$. This is $(N-2)/(N-1)$ times the usual estimate $\\widehat{V}(\\text{OLS}) = N\\,\\text{SSR}/(N_1 N_0 (N-2))$, and the factor tends to $1$. No result in the comment depends on the sentence. Samii and Aronow (2012) was not consulted."}, "url": "https://pmaronow.github.io/llm-audit/works/P053/#P053-N001", "work": {"arxiv_id": null, "doi": "10.1214/16-STS582", "title": "Understanding Ding's Apparent Paradox", "url": "https://pmaronow.github.io/llm-audit/works/P053/"}, "work_id": "P053"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of the second paragraph of the first column of page 223, 23 lines, from 'To answer this question, we undertook a nationally representative survey through the 2014 Cooperative Congressional Election Study (n=1384)' to '12 per cent (10-14) believed that no federal background checks are required.' Red outlines surround five ranges: 75-79, which follows '77 per cent (95 per cent CI', and the ranges in parentheses (51-56), (38-43), (44-50) and (10-14), which follow 53, 41, 47 and 12 per cent.", "caption": "Column 1, second paragraph, p. 223 (PDF p. 1). The outlines mark the first five printed intervals: 75–79, 51–56, 38–43, 44–50 and 10–14.", "url": "https://pmaronow.github.io/llm-audit/works/P054/exhibits/P054-N001.png"}, {"alt": "Crop of the first 22 lines of the second column of page 223, from 'existing laws predicted support for stricter gun laws' to 'laws. However, the role of policy'. Red outlines surround four ranges: 69-79, which follows '74 per cent (95 per cent CI'; (86-92), which follows 89 per cent; 65-78, which follows '71 per cent (95 per cent CI'; and (30-37), which follows 34 per cent. Between them the text reads 'We recorded no such relation among those who did not strongly favour universal background checks' and 'We also identified evidence of an engagement gap: among voters who thought gun laws should be less strict than at present'.", "caption": "Column 2, first 22 lines, p. 223 (PDF p. 1). The outlines mark the other four printed intervals: 69–79, 86–92, 65–78 and 30–37.", "url": "https://pmaronow.github.io/llm-audit/works/P054/exhibits/P054-N001-b.png"}, {"alt": "Chart titled 'Allowing for the survey weights widens all nine 95 per cent intervals', with the subtitle 'Standard errors are 1.28 to 1.47 times larger; the nine percentages do not change'. The horizontal axis is percentage points below or above the estimate, from -9 to +9. Each of nine rows has an orange interval, which rounds to the printed one, above a wider blue interval, which allows for the survey weights; both are centred at zero. To the right of each row are the group size, the printed estimate and interval, and the corrected one. All respondents, n = 1384: favour universal background checks, 77 (75-79) and 77 (74-80); prefer stricter gun laws, 53 (51-56) and 53 (50-57); believe checks are already universal, 41 (38-43) and 41 (37-45); say checks cover only some purchases, 47 (44-50) and 47 (43-51); believe no checks are required, 12 (10-14) and 12 (10-15). Strong supporters of universal checks, share preferring stricter gun laws: believe checks are already universal, n = 306, 74 (69-79) and 74 (67-81); know some sales are exempt, n = 334, 89 (86-92) and 89 (85-93). Share who would never vote for someone with a different position on gun control: want less strict gun laws, n = 191, 71 (65-78) and 71 (63-80); want stricter gun laws, n = 746, 34 (30-37) and 34 (29-39). The widest blue interval, for n = 191, reaches about 8.6 points on each side of the estimate, against about 6.4 for the orange one.", "caption": "The nine 95% intervals, each drawn relative to its estimate. Orange: $\\hat p\\pm1.96\\sqrt{V_0}$, which rounds to the printed interval in every row. Blue: $\\hat p\\pm1.96\\sqrt{V}$, the stated independent-respondent survey linearisation allowing for the weights. Orange reconstructs the printed arithmetic without identifying the historical method; in the stricter-law row, the unrounded blue lower endpoint is $49.51388\\%$. The columns at the right give the size of the group, the estimate and interval as printed, and the corrected interval, in per cent.", "url": "https://pmaronow.github.io/llm-audit/works/P054/exhibits/P054-N001-intervals.png"}], "formal_release": null, "formal_statements": [{"id": "P054-T1", "url": "https://pmaronow.github.io/llm-audit/works/P054/#P054-T1"}, {"id": "P054-T2", "url": "https://pmaronow.github.io/llm-audit/works/P054/#P054-T2"}], "headline": "Under the stated independent-respondent survey linearisation, all nine intervals widen and the 53% stricter-law interval includes 50%; the nine estimates and the three main positive comparisons remain.", "id": "P054-N001", "location": "Columns 1 and 2, the nine percentages printed with 95% confidence intervals, from “77% (95% CI 75–79)” to “34% (30–37)”, p. 223 (PDF p. 1)", "pdf_page": 1, "printed_page": "223", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P054-T1", "P054-T2"], "text": {"as_printed": "To answer this question, we undertook a nationally representative survey through the 2014 Cooperative Congressional Election Study (n=1384). [\\ldots] 77\\% (95\\% CI \\hl{75--79}) of Americans favoured universal background checks, yet only 53\\% (\\hl{51--56}) reported preferring stricter gun control laws. [\\ldots] We found that 41\\% (\\hl{38--43}) of Americans believed that universal background checks are already required by federal law, whereas 47\\% (\\hl{44--50}) correctly reported that a federal background check is required only for some gun purchases, and 12\\% (\\hl{10--14}) believed that no federal background checks are required.\n\n[\\ldots] Whereas 74\\% (95\\% CI \\hl{69--79}) of those who incorrectly believed that the USA has universal background checks supported stricter gun laws, 89\\% (\\hl{86--92}) of those who knew that some gun sales do not require background checks supported stricter laws. [\\ldots] 71\\% (95\\% CI \\hl{65--78}) said they would never vote for a political candidate who did not share their position on gun control, compared with just 34\\% (\\hl{30--37}) of those who support stricter gun laws.", "corrected": "77\\% (95\\% CI \\hl{74--80}) of Americans favoured universal background checks, yet only 53\\% (\\hl{50--57}) reported preferring stricter gun control laws. [\\ldots] An estimated 41\\% (\\hl{37--45}) of Americans believed that universal background checks are already required by federal law, whereas 47\\% (\\hl{43--51}) correctly reported that a federal background check is required only for some gun purchases, and 12\\% (\\hl{10--15}) believed that no federal background checks are required.\n\n[\\ldots] \\hl{Among those who strongly supported universal background checks,} 74\\% (95\\% CI \\hl{67--81}) of those who incorrectly believed that the USA has universal background checks supported stricter gun laws, \\hl{whereas} 89\\% (\\hl{85--93}) of those who knew that some gun sales do not require background checks supported stricter laws. [\\ldots] \\hl{Among respondents who thought gun laws should be less strict than at present,} 71\\% (95\\% CI \\hl{63--80}) said they would never vote for a political candidate who did not share their position on gun control, compared with just 34\\% (\\hl{29--39}) of those who support stricter gun laws.\n\nThe letter should also state the release of the data, the respondents selected, the weights and the interval method that were used. If they are those of the recalculation, a sentence such as the following would serve: \\hl{Estimates use the team weight of the February 2015 release of the survey data, on the 1384 respondents with valid answers to the three questions on gun laws, on universal background checks and on existing federal law. The 95\\% confidence intervals use a linearised variance that allows for the weights and treats respondents as independent.}\n\nThese replacement intervals are conditional on the following approximation; they are not independently established full-design national intervals. The corrected intervals are $\\hat p\\pm1.96\\sqrt V$, with $\\hat p$ and $V$ as defined under Evidence. They are approximate 95\\% intervals under three conditions: respondents can be treated as independent draws with the released weights; the linear and normal approximations are adequate; and the released weights identify the national population, with the retained respondents, including those answering the engagement question, identifying the same proportions in each group. If the variance under the full design of the survey is available, it should replace $V$. The printed limits could be kept only with a documented inferential justification that accounts for the survey design and response selection.", "evidence": "I recalculated the nine estimates from the public survey files (2014 Cooperative Congressional Election Study, Yale team module, Harvard Dataverse, \\url{https://doi.org/10.7910/DVN/FBXY8V}). Exact rational comparisons confirmed the rounded percentages and endpoints of both intervals in the table.\n\nThe following notation is introduced for the recalculation; the letter prints no formula. Fix one of the nine estimates. For respondent $i$, let $d_i\\in\\{0,1\\}$ indicate that $i$ belongs to the group and gave a valid answer, let $y_i\\in\\{0,1\\}$ be the answer, and let $w_i>0$ be the weight. The sums below run over the $N=1384$ respondents. Set $n=\\sum_i d_i$, $W=\\sum_i d_iw_i$ and $\\hat p=\\sum_i d_iw_iy_i/W$, and define\n\\[\nV_0=\\frac{\\hat p(1-\\hat p)}{n},\\qquad V=\\frac{N}{N-1}\\cdot\\frac{\\sum_i d_iw_i^2(y_i-\\hat p)^2}{W^2}.\n\\]\nThe variance $V$ comes from linearising the ratio $\\hat p=\\bar a/\\bar b$, where $\\bar a=N^{-1}\\sum_i d_iw_iy_i$ and $\\bar b=W/N$. To first order, a change in $(\\bar a,\\bar b)$ changes the ratio by $(\\mathrm{d}\\bar a-\\hat p\\,\\mathrm{d}\\bar b)/\\bar b$, so respondent $i$ enters through the residual $z_i=d_iw_i(y_i-\\hat p)$. These residuals sum to zero. Their empirical variance is $\\sum_i z_i^2/(N-1)$, and the estimated variance of their mean is $\\sum_i z_i^2/(N(N-1))$. Dividing by $\\bar b^2=W^2/N^2$ gives $V$. If all weights are equal, then $\\sum_i d_iw_i^2(y_i-\\hat p)^2/W^2=\\hat p(1-\\hat p)/n$, and $V$ equals $V_0$ up to the factor $N/(N-1)$. The standard errors $\\sqrt V$ agree with those of the survey package of R for a design with independent respondents and these weights.\n\nFor an intercept-only weighted least-squares fit with classical standard errors, binary responses give\n\\[\n\\sum_i d_iw_i(y_i-\\hat p)^2=W\\hat p(1-\\hat p),\\qquad\nV_{\\mathrm{classical}}=\\frac{\\hat p(1-\\hat p)}{n-1}.\n\\]\nThus entering the team weights in R's \\texttt{lm} leaves a classical intercept variance differing from $V_0$ only by the factor $n/(n-1)$. Numerical agreement with the print establishes compatibility with the method I report using, not its historical use or the inverse-variance assumptions that would justify its classical SEs. Both matching formulas remain distinct from the survey-ratio variance $V$.\n\nThe estimates are coded as follows. Favouring universal checks is the answer ``strongly favor'' or ``favor'', and strong support is ``strongly favor''. Preferring stricter laws is the answer ``more strict'' to the question with three answers. The last two estimates are the share answering ``No'' to the question whether one could still vote for someone standing for office with whom one disagrees on gun control. Respondents without a valid answer to it are left out.\n\n\\begin{audittable}{lrrlllr}{The nine estimates in the order of the letter. Printed: percentage and 95\\% interval as printed. $n$: respondents in the group. $\\hat p$: weighted percentage. Fourth and fifth columns: the two intervals, in per cent. Rounded: the fifth column at the precision of the letter. Last column: ratio of the two standard errors.}\nPrinted & $n$ & $\\hat p$ (\\%) & $\\hat p\\pm1.96\\sqrt{V_0}$ & $\\hat p\\pm1.96\\sqrt{V}$ & Rounded & $\\sqrt{V/V_0}$ \\\\\n77 (75--79) & 1384 & 76.82 & 74.60--79.05 & 73.68--79.97 & 74--80 & 1.41 \\\\\n53 (51--56) & 1384 & 53.24 & 50.61--55.87 & 49.51--56.96 & 50--57 & 1.42 \\\\\n41 (38--43) & 1384 & 40.82 & 38.23--43.41 & 37.08--44.56 & 37--45 & 1.44 \\\\\n47 (44--50) & 1384 & 47.05 & 44.42--49.68 & 43.34--50.76 & 43--51 & 1.41 \\\\\n12 (10--14) & 1384 & 12.13 & 10.41--13.85 & 9.67--14.58 & 10--15 & 1.43 \\\\\n74 (69--79) & 306 & 73.90 & 68.98--78.82 & 66.86--80.95 & 67--81 & 1.43 \\\\\n89 (86--92) & 334 & 89.15 & 85.81--92.48 & 84.89--93.40 & 85--93 & 1.28 \\\\\n71 (65--78) & 191 & 71.16 & 64.74--77.59 & 62.52--79.81 & 63--80 & 1.35 \\\\\n34 (30--37) & 746 & 33.82 & 30.43--37.22 & 28.85--38.80 & 29--39 & 1.47 \\\\\n\\end{audittable}\n\nIn every row the fourth column rounds to the printed interval and the fifth does not. The ratio $V/V_0$ ranges from approximately $1.626$ to $2.147$. It differs from one estimate to the next, since it depends on the answers as well as on the weights. No common rescaling factor was used. Without weights, or with the weights of the August 2015 release, the nine percentages are not all reproduced: the rounded values are 77, 54, 39, 49, 12, 76, 88, 71, 33 and 75, 53, 40, 48, 12, 73, 88, 73, 33. The five tested interval methods for a proportion in the survey package of R, applied to the same design, reproduce none of the nine printed intervals. If each group is treated as a separate sample, the factor $N/(N-1)$ becomes $n/(n-1)$; eight rounded intervals are unchanged, and the eighth row becomes 62--80.", "problem": "The letter states neither its weights, nor the rule that gives $n=1384$, nor how its intervals were computed. The public survey files reproduce all nine printed percentages and intervals under the reconstruction below. With the team weight of the February 2015 release, on the 1384 respondents with valid answers to the first three questions about guns, each of the nine percentages agrees with the print after rounding. Let $\\hat p$ be one of these weighted proportions and let $n$ be the unweighted number of respondents in its group. Then the interval $\\hat p\\pm1.96\\sqrt{\\hat p(1-\\hat p)/n}$ rounds to the printed interval in all nine cases, at both endpoints.\n\nThis interval uses the weights in $\\hat p$ but does not account for their unequal influence in the variance. It treats $\\hat p$ as the proportion in a simple random sample of $n$ respondents with equal weights. The weights of the 1384 respondents are unequal: they range from $0.05$ to $7.00$. The usual linearised variance of a weighted proportion, $V$ in the display under Evidence, allows for them. With $V$, every standard error is larger, by a factor ranging from approximately $1.28$ to $1.47$, and every interval is wider. For example, where the letter prints 74\\% (69--79), the interval becomes 74\\% (67--81). Under this approximation, the printed intervals overstate the precision of all nine estimates. No percentage changes.\n\nTwo limits apply. First, the rounded figures do not identify the original calculation uniquely. An actual intercept-only weighted least-squares fit in R, with classical standard errors and either normal or default $t$ critical values, also gives the same nine intervals after rounding, and six other rules for selecting respondents reproduce every printed percentage and interval. I report using classical intercept-only weighted least squares in R for the original analysis. That method is compatible with every printed number, while its historical use is not independently established by the letter, the available deposit or rounded agreement. Each matching calculation omits the squared-weight residual term from the survey-ratio variance, and under each of the seven rules none of the nine intervals based on the same linearisation rounds to the printed one. Second, $V$ is the ordinary approximation for independent respondents. Each release of the public files holds at most one weight and no strata, clusters or replicate weights, and the files do not document how the weight was built. The variance under the full design of the survey cannot be computed from them, and whether any of these intervals covers the national proportion 95\\% of the time is not established.", "what_survives": "All nine percentages are reproduced and stand, and the three knowledge percentages still sum to 100. The three comparisons that the letter draws keep 95\\% intervals that exclude zero when the same linearisation is applied to each difference. The difference between 77\\% and 53\\% is $23.59$ percentage points ($19.83$ to $27.34$). The difference between 89\\% and 74\\% among strong supporters is $15.24$ points ($7.01$ to $23.47$), and the difference between 71\\% and 34\\% is $37.34$ points ($27.36$ to $47.32$). The printed precision of each estimate is lost under this approximation. One confidence inference also changes: the matching stricter-law interval is $50.60848$--$55.86594\\%$, while the interval using $V$ is $49.51388$--$56.96055\\%$. The latter contains $50\\%$, so a simple majority preferring stricter laws is no longer established by the reported two-sided 95\\% interval criterion. The paper does not explicitly report a majority hypothesis test, and this change does not reverse the $77\\%$-versus-$53\\%$ comparison. Neither calculation establishes a causal education effect or statistical equivalence to zero among non-strong supporters. The group to which the last two estimates refer is corrected in \\fid{P054-N005}."}, "url": "https://pmaronow.github.io/llm-audit/works/P054/#P054-N001", "work": {"arxiv_id": null, "doi": "10.1016/S0140-6736(16)00042-8", "title": "Policy Misperceptions and Support for Gun Control Legislation", "url": "https://pmaronow.github.io/llm-audit/works/P054/"}, "work_id": "P054"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [{"alt": "Crop of twelve lines of the second column of page 223. The first two lines end the preceding paragraph: 'those who did not strongly favour universal background checks.' The rest reads: 'We also identified evidence of an engagement gap: among voters who thought gun laws should be less strict than at present, 71 per cent (95 per cent CI 65-78) said they would never vote for a political candidate who did not share their position on gun control, compared with just 34 per cent (30-37) of those who support stricter gun laws. However, the role of policy'. A red outline surrounds the word voters.", "caption": "Column 2, second paragraph, p. 223 (PDF p. 1), with the last two lines of the paragraph before it. The outline marks the word “voters”.", "url": "https://pmaronow.github.io/llm-audit/works/P054/exhibits/P054-N005.png"}], "formal_release": null, "formal_statements": [], "headline": "In the engagement comparison, “among voters” should read “among respondents”: the 71% and 34% are reproduced without a voter restriction, in groups that include people who did not vote.", "id": "P054-N005", "location": "Column 2, second paragraph, sentence beginning “We also identified evidence of an engagement gap”, p. 223 (PDF p. 1)", "pdf_page": 1, "printed_page": "223", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "We also identified evidence of an engagement gap: among \\hl{voters} who thought gun laws should be less strict than at present, 71\\% (95\\% CI \\hl{65--78}) said they would never vote for a political candidate who did not share their position on gun control, compared with just 34\\% (\\hl{30--37}) of those who support stricter gun laws.", "corrected": "We also identified evidence of an engagement gap: among \\hl{respondents} who thought gun laws should be less strict than at present, 71\\% (95\\% CI \\hl{63--80}) said they would never vote for a political candidate who did not share their position on gun control, compared with just 34\\% (\\hl{29--39}) of those who support stricter gun laws.\n\nIf a comparison among voters is wanted instead, the restriction must be defined and both figures recomputed. Among respondents who report that they definitely voted in November 2014, the two shares are 72\\% and 31\\%.", "evidence": "The evidence is a recalculation from the public files of the survey (2014 Cooperative Congressional Election Study, Yale team module, Harvard Dataverse, \\url{https://doi.org/10.7910/DVN/FBXY8V}). The rounding of the percentages and endpoints quoted here was confirmed in exact rational arithmetic. Each share is the weighted proportion $p$ answering ``No'' to the question whether one could still vote for someone standing for office with whom one disagrees on gun control. I use the team weight of the February 2015 release and reproduce the printed intervals with $p\\pm1.96\\sqrt{p(1-p)/n}$ for a group of $n$ respondents. I repeated the calculation in the three releases of the data, with the team weight where a release has one and without weights. I also restricted the groups by reported turnout, by registration and, in the August 2015 release, by the match to the voter file. Among the calculations I tried, both printed percentages are reproduced only with the team weight of the February 2015 release and no restriction. The rounded figures do not fix the sample or interval method exactly. Without the requirement of valid answers to the first three questions about guns, the groups have 192 and 750 respondents, and the rounded figures are the same. The analysis code of the letter is not available.", "problem": "The letter does not define ``voters''. From the public files of the survey, I reproduce the printed 71\\% (65--78) and 34\\% (30--37) with no restriction to voters. Using the 1384 respondents with valid answers to the first three questions about guns and excluding missing engagement answers, the two groups have 191 and 746 respondents. They include 18 and 149 respondents who report that they did not vote in November 2014, and 14 and 59 who report that they are not registered to vote. When I restrict the groups to respondents who report that they definitely voted in that election, I obtain 72\\% (65--79) and 31\\% (27--35) by the same calculation. These are not the printed figures. As a description of people who voted, the word ``voters'' therefore misdescribes these groups.", "what_survives": "The comparison stands as one among respondents, 71\\% against 34\\%. Among respondents who report that they definitely voted, the gap is of similar size, 72\\% against 31\\%. Changing this population label does not alter any other estimate in the letter. The corrected intervals below allow for the survey weights under the independent-respondent approximation described in \\fid{P054-N001}."}, "url": "https://pmaronow.github.io/llm-audit/works/P054/#P054-N005", "work": {"arxiv_id": null, "doi": "10.1016/S0140-6736(16)00042-8", "title": "Policy Misperceptions and Support for Gun Control Legislation", "url": "https://pmaronow.github.io/llm-audit/works/P054/"}, "work_id": "P054"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the reference list on page 223: the last two lines of reference 1, which end with the access date March 30, 2015, and the four lines of reference 2. Reference 2 reads: Fleegler EW, Lee LK, Monteaux MC, Hemenway D, Mannix R. Firearm legislation and firearm-related fatalities in the United States. JAMA Intern Med 2013; 173: 732-40. A red outline surrounds the word Monteaux.", "caption": "Reference 2, p. 223 (PDF p. 1), foot of the second column. The outline marks the surname “Monteaux”.", "url": "https://pmaronow.github.io/llm-audit/works/P054/exhibits/P054-N006.png"}], "formal_release": null, "formal_statements": [], "headline": "In reference 2 the surname of the third author should read “Monuteaux”, not “Monteaux”.", "id": "P054-N006", "location": "Reference 2, p. 223 (PDF p. 1), foot of the second column, third author", "pdf_page": 1, "printed_page": "223", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Fleegler EW, Lee LK, \\hl{Monteaux} MC, Hemenway D, Mannix R. Firearm legislation and firearm-related fatalities in the United States. \\emph{JAMA Intern Med} 2013; \\textbf{173:} 732--40.", "corrected": "Fleegler EW, Lee LK, \\hl{Monuteaux} MC, Hemenway D, Mannix R. Firearm legislation and firearm-related fatalities in the United States. \\emph{JAMA Intern Med} 2013; \\textbf{173:} 732--40.", "evidence": "I checked the page against a public source. On the publisher's page for the article (\\url{https://doi.org/10.1001/jamainternmed.2013.1286}), the byline names the third author as ``Michael C. Monuteaux, ScD'', and the citation of the article prints ``Monuteaux MC''. The title, journal, year, volume and pages of reference 2 agree with that page.", "problem": "The third author of the cited article is Michael C. Monuteaux. The reference omits the ``u'' that follows ``Mon'' in his surname.", "what_survives": "The article remains identifiable from its title, journal, volume and pages. No estimate or argument in the letter depends on the spelling."}, "url": "https://pmaronow.github.io/llm-audit/works/P054/#P054-N006", "work": {"arxiv_id": null, "doi": "10.1016/S0140-6736(16)00042-8", "title": "Policy Misperceptions and Support for Gun Control Legislation", "url": "https://pmaronow.github.io/llm-audit/works/P054/"}, "work_id": "P054"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of six lines of the reference list on page 223: the last line of reference 2, the four lines of reference 3, and the first line of reference 4, Quinnipiac University Polling Institute. Reference 3 reads: Webster D, Crifasi KC, Vernick JS. Effects of the repeal of Missouri's handgun purchaser licensing law on homicides. J Urban Health 2014; 91: 293-302. A red outline surrounds the initials KC and the comma after them.", "caption": "Reference 3, p. 223 (PDF p. 1), foot of the second column. The outline marks the initials “KC”.", "url": "https://pmaronow.github.io/llm-audit/works/P054/exhibits/P054-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "In reference 3 the initials of the second author should read “CK”, not “KC”.", "id": "P054-N007", "location": "Reference 3, p. 223 (PDF p. 1), foot of the second column, second author", "pdf_page": 1, "printed_page": "223", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Webster D, Crifasi \\hl{KC}, Vernick JS. Effects of the repeal of Missouri’s handgun purchaser licensing law on homicides. \\emph{J Urban Health} 2014; \\textbf{91:} 293--302.", "corrected": "Webster D, Crifasi \\hl{CK}, Vernick JS. Effects of the repeal of Missouri’s handgun purchaser licensing law on homicides. \\emph{J Urban Health} 2014; \\textbf{91:} 293--302.", "evidence": "I checked the page against a public source. On the publisher's page for the article (\\url{https://doi.org/10.1007/s11524-014-9865-8}), the byline names the second author as ``Cassandra Kercher Crifasi'', and the citation of the article prints ``Crifasi, C.K.'' The publisher's page for the erratum to the article (\\url{https://doi.org/10.1007/s11524-014-9882-7}) has the same byline and the same form of citation. The title, journal, year, volume and pages of reference 3 agree with the publisher's page for the article.", "problem": "The second author of the cited article is Cassandra Kercher Crifasi, whose initials are CK. The reference prints them in the reverse order.", "what_survives": "The article remains identifiable from its title, journal, volume and pages. No estimate or argument in the letter depends on the order of the initials."}, "url": "https://pmaronow.github.io/llm-audit/works/P054/#P054-N007", "work": {"arxiv_id": null, "doi": "10.1016/S0140-6736(16)00042-8", "title": "Policy Misperceptions and Support for Gun Control Legislation", "url": "https://pmaronow.github.io/llm-audit/works/P054/"}, "work_id": "P054"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of five lines of the Introduction. A red box marks the citation '(Harshaw, Sävje, and Wang 2022)'.", "caption": "Introduction, p. 2 (PDF p. 2). The marked citation gives the order Harshaw, Sävje, Wang.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N002.png"}, {"alt": "Crop of three entries of the reference list. A red box marks the authors and year 'Harshaw, C., Y. Wang, and F. Sävje. 2022.' at the start of the second entry.", "caption": "Reference list, p. 15 (PDF p. 15). The marked entry gives the order Harshaw, Wang, Sävje.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N002-p15.png"}], "formal_release": null, "formal_statements": [], "headline": "The citation on p. 2 and the entry in the reference list on p. 15 give the authors of Harshaw et al. (2022) in different orders, and one of the two must be changed to agree with the cited work.", "id": "P061-N002", "location": "Introduction, p. 2 (PDF p. 2), paragraph on recent advances; reference list, p. 15 (PDF p. 15)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": {"as_printed": "Introduction, p.~2: ``More general methods for randomization tests under interference (Basse, Feller, and Toulis 2019) and Riesz estimators (\\hl{Harshaw, Sävje, and Wang} 2022) have also been proposed.''\n\nReference list, p.~15: \\hl{Harshaw, C., Y. Wang, and F. Sävje.} 2022. ``A Design-Based Riesz Representation Framework for Randomized Experiments.'' In \\textit{NeurIPS 2022 Workshop on Causality for Real-World Impact}. \\url{https://openreview.net/forum?id=GM16YCLkEmz}.", "corrected": "The order of the authors must be the same in both places, after a check against the cited work. If the reference list is right, the citation on p.~2 reads ``(Harshaw, \\hl{Wang, and Sävje} 2022)''. If the citation is right, the entry on p.~15 begins ``Harshaw, C., \\hl{F. Sävje, and Y. Wang}. 2022.''", "evidence": "I compared the author orders on the two pages. I did not compare either order with the cited work, so I have not established the correct order.", "problem": "The citation places Sävje before Wang, and the entry in the reference list places Wang before Sävje. At least one of the two is misprinted. The article does not determine which order is that of the cited work.", "what_survives": "The cited work can be identified from either order. No argument in the article depends on the citation."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N002", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of the heading '3.1. Design' and the first three lines of the section. A red box marks the word 'bijective' in 'where Z is a bijective random vector Z: Omega to R^N'.", "caption": "Section 3.1, p. 3 (PDF p. 3), first sentence. The marked word describes the map $\\mathbf Z$.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N003.png"}], "formal_release": null, "formal_statements": [], "headline": "The word “bijective” means injective here: $\\mathbf Z$ is a bijection from $\\Omega$ onto the design space $\\mathcal Z$, not onto $\\mathbb R^N$.", "id": "P061-N003", "location": "Section 3.1, p. 3 (PDF p. 3), first sentence", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``We define the design space $\\mathcal Z=\\mathbf Z(\\Omega)$, where $\\mathbf Z$ is a \\hl{bijective} random vector $\\mathbf Z:\\Omega\\to\\mathbb R^N$. The \\emph{design} is defined as the joint p.m.f. of $\\mathbf Z$, $f:\\mathbb R^N\\to[0,1]$.''", "corrected": "``We define the design space $\\mathcal Z=\\mathbf Z(\\Omega)$, where $\\mathbf Z$ is an \\hl{injective} random vector $\\mathbf Z:\\Omega\\to\\mathbb R^N$ \\hl{(a bijection from $\\Omega$ onto $\\mathcal Z$)}. The \\emph{design} is defined as the joint p.m.f. of $\\mathbf Z$, $f:\\mathbb R^N\\to[0,1]$.''", "intended_meaning": "A bijection from $\\Omega$ onto $\\mathbb R^N$ would force $\\mathcal Z=\\mathbf Z(\\Omega)=\\mathbb R^N$. This excludes the finite design spaces that the article uses, such as $\\mathcal Z\\subseteq\\{0,1,2\\}^N$ in Section 6.2 (p.~11). The article needs $\\mathbf Z$ to be one-to-one, so that $\\mathbf Z^{-1}(\\mathbf z)$ is defined for every $\\mathbf z\\in\\mathcal Z$. Equation (1) on p.~4 applies the inverse only on $\\mathcal Z$, and every definition is unchanged under this reading."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N003", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the display that defines f(z) for the school experiment and the line below it. A red box marks the condition of the first case, that the sum of z_k over k from 1 to N equals the ceiling of 0.2N.", "caption": "Section 3.1, p. 4 (PDF p. 4): the design of the school experiment. The marked condition defines the positive case.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N004.png"}], "formal_release": null, "formal_statements": [], "headline": "The probability mass function of the school design is meant for binary assignment vectors, and its positive case needs the condition $\\mathbf z\\in\\{0,1\\}^N$.", "id": "P061-N004", "location": "Section 3.1, p. 4 (PDF p. 4), display at the top of the page; its lead-in is at the foot of p. 3", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "``Letting $\\lceil 0.2N\\rceil$ be the number of observations in treatment, the design is''\n\\[\nf(\\mathbf z)=\\begin{cases}\\dfrac{(\\lceil 0.2N\\rceil)!(N-\\lceil 0.2N\\rceil)!}{N!} & :\\ \\hl{\\sum_{k=1}^{N}z_k=\\lceil 0.2N\\rceil}\\\\ 0 & :\\ \\text{otherwise,}\\end{cases}\n\\]\n``where the corresponding design space is $\\mathcal Z\\supseteq\\{0,1\\}^N$.''", "corrected": "``Letting $\\lceil 0.2N\\rceil$ be the number of observations in treatment, the design is''\n\\[\nf(\\mathbf z)=\\begin{cases}\\dfrac{(\\lceil 0.2N\\rceil)!(N-\\lceil 0.2N\\rceil)!}{N!} & :\\ \\hl{\\mathbf z\\in\\{0,1\\}^N\\text{ and }}\\sum_{k=1}^{N}z_k=\\lceil 0.2N\\rceil\\\\ 0 & :\\ \\text{otherwise,}\\end{cases}\n\\]\n``where the corresponding design space is $\\mathcal Z\\supseteq\\{0,1\\}^N$.''", "intended_meaning": "The text fixes the design: $\\lceil 0.2N\\rceil$ of the $N$ students are assigned to the program uniformly at random, and the exposure mappings on p.~5 take $\\mathbf z\\in\\{0,1\\}^N$. For binary vectors the display is correct. With $k=\\lceil 0.2N\\rceil$, there are $\\binom{N}{k}$ binary vectors with $k$ ones, and each has mass $k!(N-k)!/N!=1/\\binom{N}{k}$. As printed, the positive case also covers vectors that are not binary and have the same sum, since $f$ is defined on $\\mathbb R^N$ and the design space is only required to contain $\\{0,1\\}^N$. For $N=2$, the vectors $(0,1)$, $(1,0)$ and $(1/2,1/2)$ each receive mass $1/2$, and these three masses sum to $3/2$. For $N=10$ and $\\mathcal Z=\\{0,1,2\\}^{10}$, the 45 binary vectors with two ones and the 10 vectors with a single entry equal to 2 give total mass $55/45=11/9$. Nothing after the display depends on the omission."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N004", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [{"alt": "Crop of the heading '4. Inferential targets' and the first five lines of the section. A red box marks the words 'new inferential targets' in the first sentence.", "caption": "Section 4, p. 5 (PDF p. 5), first sentence. The marked words call the targets new.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N006.png"}, {"alt": "Crop of four lines of Section 3.1. A red box marks the words 'novel estimands' in the sentence 'However, we will use this setup to define novel estimands in Section 4.'", "caption": "Section 3.1, p. 3 (PDF p. 3). The marked words call the estimands novel.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N006-p3.png"}, {"alt": "Crop of five lines of the last paragraph of the Introduction. A red box marks the words 'new inferential targets' in the sentence about Section 4.", "caption": "Introduction, p. 2 (PDF p. 2): the outline of the article. The marked words repeat the description.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N006-p2.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T6", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T6"}], "headline": "The novelty claim for the targets of Section 4 needs qualification: the conditional mean of Equation (5) and the contrast of Equation (8) coincide on their shared domain with the expected exposure outcome and the expected exposure effect of Sävje (2024), and under an individualistic binary exposure the AEED is the average distributional shift effect of Sävje, Aronow, and Hudgens (2021).", "historical_evidence_type": "lean_proof", "id": "P061-N006", "location": "Section 4, p. 5 (PDF p. 5), first sentence; Section 3.1, p. 3 (PDF p. 3); Introduction, p. 2 (PDF p. 2), last paragraph", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P061-T6"], "text": {"as_printed": "Section 4, p.~5: ``We define \\hl{new} inferential targets for the design-based framework.''\n\nSection 3.1, p.~3: ``However, we will use this setup to define \\hl{novel} estimands in Section 4.''\n\nIntroduction, p.~2: ``In Section 4, we define a set of \\hl{new} inferential targets for the design-based setting.''", "corrected": "Section 4, p.~5: ``We \\hl{formulate design-conditional} inferential targets\\hl{, building on the expected exposure outcomes and effects of Sävje (2024). Under individualistic binary exposure, the AEED is the average distributional shift effect of Sävje, Aronow, and Hudgens (2021).}''\n\nSection 3.1, p.~3: ``However, we will use this setup to define \\del{novel} estimands in Section 4.''\n\nIntroduction, p.~2: ``In Section 4, we define a set of \\del{new} inferential targets for the design-based setting.''", "evidence": "The evidence is a comparison with the earlier definitions and a derivation of the identities. I read Sävje (2024) in its author manuscript of 13 March 2023 (arXiv:2103.06471v2); a manuscript of the same paper dated 26 November 2019 has the same two definitions. I did not obtain the typeset text of the journal article. I read Definition 4 of Sävje, Aronow, and Hudgens (2021) in the author manuscript of the published article and in arXiv:1711.06399v4.\n\n\\begin{audittable}{lll}{The targets of Section 4 beside the earlier definitions. The locators for Sävje (2024) are those of the author manuscript of 2023.}\nIn the article & Formula & Earlier definition \\\\\nEPO, Equation (5) & $\\overline{y}_i(d)=\\operatorname{E}[y_i(\\mathbf Z)\\mid D_i=d]$ & Sävje (2024), Section 4.3 \\\\\nAEED, Equation (8) & $N^{-1}\\sum_{i}[\\overline{y}_i(d)-\\overline{y}_i(d')]$ & Sävje (2024), Definition 1 \\\\\nAEED with $g_i(\\mathbf z)=z_i$ & $N^{-1}\\sum_{i}\\{\\operatorname{E}[Y_i\\mid Z_i=1]-\\operatorname{E}[Y_i\\mid Z_i=0]\\}$ & Sävje et al. (2021), Definition 4 \\\\\n\\end{audittable}\n\nThe identities hold for every design with finitely many assignments. By the definition of conditional probability, the ratio in Equation (5) equals $\\operatorname{E}[y_i(\\mathbf Z)\\mid D_i=d]$ whenever $\\pi_i(d)>0$. Substituting this identity into Equations (6)--(8), I obtain the contrast of Definition 1. If $g_i(\\mathbf z)=z_i$ and the treatment is binary, then $D_i=Z_i$ and $Y_i=y_i(\\mathbf Z)$, and the contrast $\\tau(1,0)$ is the ADSE. The three identities are also proved in Lean, as identities between weighted sums over an arbitrary finite set of assignments.", "problem": "The article calls the targets of Section 4 new or novel in three places. Earlier work defines the same conditional mean and the same contrast on their shared domain, and the article does not say so. Sävje (2024) defines the conditional mean of Equation (5), with the expectation taken over the design, and the average contrast of Equation (8), for an arbitrary exposure mapping. When the exposure of a unit is its own binary treatment, $g_i(\\mathbf z)=z_i$, the AEED is the average distributional shift effect (ADSE) of Sävje, Aronow, and Hudgens (2021). The article cites both works for other purposes, on p.~5 and in footnote 8, and footnote 7 credits Hudgens and Halloran (2008) with a special case of the EPO. The defect lies in the claim of novelty and in the missing attribution. Equations (5)--(8) are correct as definitions.", "what_survives": "Equations (5)--(8), the values of the targets, the analysis of NURVA and SUTVA, the examples, Tables 1--5, Figure 1 and the estimation results of the supplement do not depend on whether the targets are new, and none of them changes. I did not assess the novelty of the framework as a whole, of the survey-sampling interpretation or of NURVA. Footnotes 7 and 8 are not affected."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N006", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of text from the top of page 7. A red box marks the word implies in the sentence: Our previously introduced exposure mapping for the survey setting implies that the number of members reported for this group should be unaffected by the act of surveying other groups.", "caption": "Section 4.1 (continued), p. 7 (PDF p. 7), first paragraph. The marked verb credits the exposure mapping with an invariance that only a stability assumption supplies.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N007.png"}], "formal_release": null, "formal_statements": [], "headline": "The exposure mapping only labels assignments, and the invariance described in the sentence comes from a stability assumption on that mapping, not from the mapping itself.", "id": "P061-N007", "location": "Section 4.1 (continued), first paragraph of p. 7 (PDF p. 7), sentence beginning “Our previously introduced exposure mapping”", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "While we do not typically think of surveys as requiring SUTVA-like assumptions, survey outcomes are not generally well-defined in their absence. Consider a quantity such as ``the potential outcome of rebel group three, when it is surveyed.'' Our previously introduced exposure mapping for the survey setting \\hl{implies} that the number of members reported for this group should be unaffected by the act of surveying other groups (no interference) and whether the group is surveyed first, second or third (no hidden treatment variations).", "corrected": "\\hl{For our previously introduced exposure mapping for the survey setting to yield a stable outcome for the surveyed exposure on the design support, one would have to assume} that the number of members reported for this group should be unaffected by the act of surveying other groups (no interference) and whether the group is surveyed first, second or third (no hidden treatment variations).", "intended_meaning": "The paragraph describes what a SUTVA-like assumption must assert for the outcome of a group ``when it is surveyed'' to be a single number, and that description is correct. The mapping of Section 3.3, $g_i(\\mathbf z)=1$ if $z_i>0$ and $g_i(\\mathbf z)=0$ otherwise, only labels each group as surveyed or not surveyed. For example, the survey orders $(1,2,3,0,\\ldots,0)$ and $(3,2,1,0,\\ldots,0)$ each have probability $1/720$ under the design of p.~3, and group three has the exposure $1$ under both. With the outcome rule $y_3(\\mathbf z)=z_3$, its outcomes are $3$ and $1$. The invariance holds once stability is assumed for this mapping, as in Condition 5.1 (NURVA). Nothing in the paper depends on the verb."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N007", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of the middle of page 13, from the sentence that defines the corresponding AEED to part of the sentence on additional stability assumptions. Red boxes mark four terms: the italic subscript Z prime of tau, the italic z inside g i of z in the display, and two occurrences of y i of d that have no bar over the y.", "caption": "Section 6.3, p. 13 (PDF p. 13). Four terms are marked: the italic subscript in $\\tau_{Z'}(2,1)$, the italic argument of $g_i$, and the two EPOs printed as $y_i(d)$ without an overbar.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N008.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T9", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T9"}], "headline": "In Section 6.3 the two EPOs should carry their overbar, $\\overline y_i(d)$, and the argument of $g_i$ and the first subscript $\\mathbf Z'$ should be bold.", "id": "P061-N008", "location": "Section 6.3, p. 13 (PDF p. 13): the sentence defining the corresponding AEED, the display defining $g_i$, and the paragraph below it", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P061-T9"], "text": {"as_printed": "Using the same exposure mapping, define the corresponding AEED $\\tau_{\\hl{Z'}}(2,1)$.$^{10}$\n\nWe define the exposure mapping as\n\\[\ng_i(\\hl{z})=\\begin{cases}2&:z_i=2\\\\1&:z_i=1\\\\0&:z_i=0\\end{cases},\\ \\forall i\\in\\{1,\\ldots,N\\},\\mathbf z\\in\\mathcal Z.\n\\]\nEven if NURVA holds for both $\\mathbf Z$ and $\\mathbf Z'$, there is no guarantee that $\\tau_{\\mathbf Z}(2,1)$ will equal $\\tau_{\\mathbf Z'}(2,1)$: changing either $n$ to $n'$ or $k$ to $k'$ changes the distribution of other units' sampling or treatment statuses conditional on $D_i=d$, and hence may change the EPOs $\\hl{y_i(d)}$ that enter the AEPOs. Learning what would happen under alternative values of $(n,k)$ or in alternative target populations requires additional stability assumptions on how $\\hl{y_i(d)}$ varies with the sampling and treatment scheme, as in the broader generalizability and transportability literature.", "corrected": "Using the same exposure mapping, define the corresponding AEED $\\tau_{\\hl{\\mathbf Z'}}(2,1)$.$^{10}$\n\nWe define the exposure mapping as\n\\[\ng_i(\\hl{\\mathbf z})=\\begin{cases}2&:z_i=2\\\\1&:z_i=1\\\\0&:z_i=0\\end{cases},\\ \\forall i\\in\\{1,\\ldots,N\\},\\mathbf z\\in\\mathcal Z.\n\\]\n\\ldots\\ and hence may change the EPOs $\\hl{\\overline y_i(d)}$ that enter the AEPOs. Learning what would happen under alternative values of $(n,k)$ or in alternative target populations requires additional stability assumptions on how $\\hl{\\overline y_i(d)}$ varies with the sampling and treatment scheme, as in the broader generalizability and transportability literature.", "evidence": "The evidence is a check of p.~13 against Equation (5) on p.~6 and the vector notation on p.~4, including the fonts of the marked symbols.", "problem": "Equation (5) writes the EPO as $\\overline y_i(d)$, and the paper reserves $y_i(\\cdot)$ for raw potential outcomes of an assignment vector. The two quantities that this passage calls EPOs are printed without the overbar. In the same passage the argument of $g_i$ is an italic $z$ where the quantifier has the bold vector $\\mathbf z$, and the first subscript $Z'$ is italic where the next paragraph has the bold $\\mathbf Z'$.", "what_survives": "The argument of Section 6.3 is unaffected. NURVA for each of two designs does not make their EPOs or AEEDs equal, and additional stability assumptions are needed."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N008", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text from the conclusion on page 14. Red boxes mark the phrase asymptotically equivalent and, on the next line, the phrase sufficiently local.", "caption": "Section 8 (Conclusion), p. 14 (PDF p. 14), second paragraph. The marked phrases assert an equivalence of inference under a condition, “sufficiently local”, that the paper does not specify.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N009.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T4", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T4"}, {"id": "P061-T5", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T5"}], "headline": "For the same estimator and uncertainty scale, the standardized statistics centred on the AEED and the average direct effect are asymptotically equal only when the target difference is $o(s_N)$; local dependence and local interference do not imply this, and the cited central limit theorem in arXiv version 2 is centred on the AEED.", "id": "P061-N009", "location": "Section 8 (Conclusion), second paragraph, sentence beginning “There is additional hope”, p. 14 (PDF p. 14)", "pdf_page": 14, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P061-T4", "P061-T5"], "text": {"as_printed": "There is additional hope for further refining inference on targets such as AEEDs from Sävje (2024); inference on AEEDs is \\hl{asymptotically equivalent} to inference on the ``average direct effect'' $N^{-1}\\sum_{i=1}^{N}\\bigl(\\mathrm E[y_i(d;\\mathbf D_{-i})-y_i(d';\\mathbf D_{-i})]\\bigr)$ if both statistical dependence in $\\mathbf D$ and unmodeled interference are \\hl{sufficiently local}.", "corrected": "There is additional hope for further refining inference on targets such as AEEDs from Sävje (2024). For the individualistic exposure mapping $g_i(\\mathbf z)=z_i$, let $\\tau_N=\\tau(d,d')$ be the AEED, let $A_N=N^{-1}\\sum_{i=1}^{N}\\mathrm E[y_i(d;\\mathbf Z_{-i})-y_i(d';\\mathbf Z_{-i})]$ be the ``average direct effect'', and let $\\widehat\\tau_N$ be an estimator of $\\tau_N$ with standard error $s_N>0$ and estimated standard error $\\widehat s_N>0$. \\hl{Assume that the assignments used to define $A_N$ are feasible and both target means are finite.} Suppose that $(\\widehat\\tau_N-\\tau_N)/s_N$ converges in distribution to a random variable $W$ \\hl{with a continuous distribution}, that $\\widehat s_N/s_N\\to1$ in probability, and that\n\\[\n\\hl{|\\tau_N-A_N|=o(s_N).}\n\\]\nThen $(\\widehat\\tau_N-A_N)/\\widehat s_N$ converges in distribution to $W$, so confidence intervals based on this limit have the same limiting coverage for either target. If $(\\tau_N-A_N)/s_N\\to c\\ne0$, the limit is $W+c$. \\hl{Local dependence in $\\mathbf Z$ and local unmodeled interference do not imply the displayed condition.} \\hl{Under the hypotheses of Lemma A20 of Sävje, Aronow, and Hudgens (2021), with its bound $k$ fixed,} the condition $\\alpha_{\\mathrm{int}}=o(Ns_N)$ is sufficient for it; when $s_N$ is of order $N^{-1/2}$, this reads $\\alpha_{\\mathrm{int}}=o(\\sqrt N)$.", "evidence": "The evidence is a derivation, an exact example, and a check of the cited works. For the derivation, let $\\tau_N$ be the AEED, $A_N$ the average direct effect, $\\delta_N=\\tau_N-A_N$, and $\\widehat\\tau_N$ an estimator with standard error $s_N>0$. Both targets are deterministic, and\n\\[\n\\frac{\\widehat\\tau_N-A_N}{s_N}-\\frac{\\widehat\\tau_N-\\tau_N}{s_N}=\\frac{\\delta_N}{s_N}.\n\\]\nIf $(\\widehat\\tau_N-\\tau_N)/s_N$ converges in distribution to $W$ and $\\widehat s_N/s_N\\to1$ in probability, then Slutsky's theorem implies that $(\\widehat\\tau_N-A_N)/\\widehat s_N$ converges in distribution to $W$ when $\\delta_N/s_N\\to0$, and to $W+c$ when $\\delta_N/s_N\\to c\\ne0$.\n\nFor the example, fix an integer $m\\ge2$ and take $m^8$ disjoint pairs of units, so that $N=2m^8$. The pairs are assigned independently. Within a pair the assignments $(0,0)$ and $(1,1)$ each have probability $(1+1/m)/4$, and $(0,1)$ and $(1,0)$ each have probability $(1-1/m)/4$. Let $g_i(\\mathbf z)=z_i$, and let $y_i(\\mathbf z)=1+z_j/m$, where $j$ is the other unit in the pair of $i$. Each unit has $\\Pr[D_i=1]=1/2$, the outcomes lie in $[1,3/2]$, and each assignment and each outcome depends on at most one other unit. The correlation of the two assignments in a pair is $1/m$, and the assignment of the other unit in the pair changes the outcome by $1/m$. The outcome does not depend on the unit's own assignment, and hence $A_N=0$. Since $\\Pr[D_j=1\\mid D_i=1]=(1+1/m)/2$ and $\\Pr[D_j=1\\mid D_i=0]=(1-1/m)/2$, I have $\\tau_N=1/m^2$ and $\\delta_N=1/m^2\\to0$. The Horvitz--Thompson estimator of the AEED is unbiased, with variance\n\\[\ns_N^2=\\frac{V_m}{4m^8},\\qquad V_m=8+\\frac{16}{m}+\\frac{14}{m^2}+\\frac{2}{m^3}-\\frac{4}{m^4},\\qquad 8\\le V_m<20 .\n\\]\nA direct computation gives $\\delta_N^2/s_N^2=4m^4/V_m\\ge m^4/5$. The difference between the two targets is of order $N^{-1/4}$ and the standard error is of order $N^{-1/2}$. By Chebyshev's inequality, the interval $\\widehat\\tau_N\\pm5s_N$ covers the AEED with probability at least $24/25$, and for $m\\ge5$ it covers the average direct effect with probability at most $V_m/m^4\\le20/m^4$.\n\\begin{audittable}{rrcccc}{Exact values in the example with $N=2m^8$ units. The last column is the upper bound $V_m/m^4$ on the probability that $\\widehat\\tau_N\\pm5s_N$ covers the average direct effect, which applies for $m\\ge5$; decimal values are rounded.}\n$m$ & $N$ & $\\delta_N$ & $\\delta_N^2/s_N^2$ & $\\delta_N/s_N$ & Bound on coverage \\\\\n$2$ & $512$ & $1/4$ & $128/39$ & $1.81$ & --- \\\\\n$3$ & $13122$ & $1/9$ & $6561/302$ & $4.66$ & --- \\\\\n$5$ & $781250$ & $1/25$ & $390625/1839$ & $14.6$ & $7356/390625\\approx0.019$ \\\\\n$10$ & $2\\times10^{8}$ & $1/100$ & $5\\times10^{7}/12177$ & $64.1$ & $12177/12500000\\approx0.00097$ \\\\\n$100$ & $2\\times10^{16}$ & $1/10^{4}$ & $10^{16}/204035049$ & $7000.8$ & $204035049/(2.5\\times10^{15})\\approx8.2\\times10^{-8}$ \\\\\n\\end{audittable}\nThe example violates NURVA. It satisfies the conditions of the central limit theorem in the version of Sävje (2024) that was read, which is Proposition S8 of the supplement to arXiv version 2. The outcomes are bounded, the exposure probabilities equal $1/2$, and the variance is of order $N^{-1}$. Each unit depends on its own pair only, and hence Condition S6, $d_{\\max}=o(n^{1/4})$, holds. Propositions S7 and S8 of that supplement keep the limit centred on the expected exposure effect. Lemma A20 of Sävje, Aronow, and Hudgens (2021), read in Supplement A of arXiv version 4, bounds the difference between the corresponding two targets of that paper by $6k^2n^{-1}\\alpha_{\\mathrm{int}}$ in its notation, and this bound tends to zero only when $\\alpha_{\\mathrm{int}}=o(n)$. Both works were read as preprints, and the published journal versions were not compared.", "problem": "The sentence cites Sävje (2024) for an equivalence between inference on AEEDs and inference on the average direct effect when the dependence in the design and the unmodeled interference are ``sufficiently local'', and neither the paper nor its supplement says what that requires. I take the individualistic exposure mapping $g_i(\\mathbf z)=z_i$, with feasible assignments and finite means defining the average direct effect (see \\fid{P061-N019}), and an estimator of the AEED with standard error $s_N$. The statistic standardized at the AEED and the statistic standardized at the average direct effect differ by $\\delta_N/s_N$, where $\\delta_N$ is the difference between the two targets. These statistics are asymptotically equal if and only if $\\delta_N=o(s_N)$. It is not enough that $\\delta_N$ tends to zero. In arXiv version 2 of Sävje (2024), including its supplement, the central limit theorem under local dependence is centred on the AEED, and I found no result there transferring inference to the average direct effect. Locality does not imply the rate. In the example below, each unit depends on one other unit only and the conditions of that theorem hold, while $\\delta_N/s_N$ diverges. Conditions stronger than locality can make the printed sentence true, and the sentence is not shown to be false under every interpretation of ``sufficiently local''.", "what_survives": "The definitions of the AEPO and the AEED, their interpretation, the examples, the finite-sample unbiasedness of the Horvitz--Thompson estimators, and the variance and consistency results stated under NURVA do not use this comparison. The closing remark changes. Using the same estimator and uncertainty scale for the average direct effect requires the rate condition below to preserve the first-order limit. Whether the journal version of Sävje (2024) contains a result of this kind was not checked."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N009", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of three lines of text and Condition 1.1 on page 5 of the supplement. A red box marks the words 'the variance bound is o(N^2).' at the end of the sentence that begins 'To provide intuition for Condition 1.3'.", "caption": "Supplementary Appendix, p. 5: the sentence that gives the intuition for Condition 1.3, and Condition 1.1. The marked words state the rate.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N010.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T21", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T21"}], "headline": "The requirement behind Condition 1.3 is that the bound on the variance of the Horvitz–Thompson estimator is $o(1)$; the quantity that is $o(N^2)$ is the bound $b_{(N)}c_{(N)}^2$ before division by $N^2$.", "id": "P061-N010", "location": "Supplementary Appendix, p. 5 (PDF p. 5 of the supplement), sentence before Condition 1.1", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P061-T21"], "text": {"as_printed": "``To provide intuition for Condition 1.3, what we require jointly from our regularity conditions is that \\hl{the variance bound is $o(N^2)$}.''\n\n``\\textbf{Condition 1.3.} (Interaction condition) $b_{(N)}c_{(N)}^2=o(N^2)$''", "corrected": "``To provide intuition for Condition 1.3, what we require jointly from our regularity conditions is that \\hl{the bound on the variance of the Horvitz--Thompson mean estimator is $o(1)$, equivalently $b_{(N)}c_{(N)}^2=o(N^2)$}.''", "intended_meaning": "Under NURVA, Condition 1.1 implies that $|\\overline{y}_i(d)|/\\pi_i(d)\\le c_{(N)}$ for every $i$. Inserting this bound into supplement Equation (5) and using Condition 1.2, I obtain\n\\[\n\\operatorname{Var}[\\widehat{\\overline{y}}(d)]\\le\\frac{c_{(N)}^2}{N^2}\\sum_{i=1}^{N}\\sum_{j=1}^{N}|\\pi_{ij}(d)-\\pi_i(d)\\pi_j(d)|=\\frac{b_{(N)}c_{(N)}^2}{N^2}.\n\\]\nBy Condition 1.3, the right side is $o(1)$, and Chebyshev's inequality gives Proposition 1. The sentence therefore refers to $b_{(N)}c_{(N)}^2$, which bounds the variance of the total $N\\widehat{\\overline{y}}(d)$. A bound of order $o(N^2)$ on the variance of $\\widehat{\\overline{y}}(d)$ itself would not give consistency. Condition 1.3 and Proposition 1 are correct as stated."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N010", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [{"alt": "Crop of four lines of text from page 6 of the supplement. A red box marks the rate little o of n to the power one quarter in the sentence: the maximum degree d max grows at a rate of o of n to the one quarter.", "caption": "Supplementary Material, p. 6 (PDF p. 6), “Central limit theorems” paragraph. The marked rate is written in $n$, while the population size in the supplement is $N$.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N011.png"}, {"alt": "Crop of the last four lines of text on page 6 of the supplement. A red box marks the rate capital O of n to the power one half, at the end of the sentence: the maximum degree d max is O of n to the one half.", "caption": "Supplementary Material, p. 6 (PDF p. 6), last paragraph. The second rate is also written in $n$.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N011-b.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T10", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T10"}], "headline": "The two degree rates on p. 6 of the supplement should be written in the population size $N$, as $o(N^{1/4})$ and $O(N^{1/2})$.", "id": "P061-N011", "location": "Supplementary Material, p. 6 (PDF p. 6 of the supplement), “Central limit theorems” paragraph and last paragraph of the page", "pdf_page": 6, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P061-T10"], "text": {"as_printed": "Recent advancements in this area have been made under weakened assumptions about the growth of the maximum degree in the dependency graph, specifically that the maximum degree $d_{\\max}$ grows at a rate of $o(\\hl{n}^{1/4})$ (Ogburn et al. 2024; Han, Basse, and Bojinov 2024). \\ldots\n\nHowever, there are relevant limitations to note. Sävje, Aronow, and Hudgens (2021) prove that Chebyshev's inequality remains asymptotically sharp under strong restrictions on the growth of the degree sequence, even when the average degree $d_{\\mathrm{avg}}$ is $O(1)$ and the maximum degree $d_{\\max}$ is $O(\\hl{n}^{1/2})$.", "corrected": "\\ldots\\ specifically that the maximum degree $d_{\\max}$ grows at a rate of $o(\\hl{N}^{1/4})$ (Ogburn et al. 2024; Han, Basse, and Bojinov 2024). \\ldots\\ even when the average degree $d_{\\mathrm{avg}}$ is $O(1)$ and the maximum degree $d_{\\max}$ is $O(\\hl{N}^{1/2})$.", "evidence": "The evidence is a check of pp.~1, 5 and 6 of the supplement.", "problem": "Both rates are written in $n$, which the passage does not define. The supplement indexes its populations by their size $N$ and lets $N$ tend to infinity (p.~5), and on p.~1 it uses $n$ for the number of units sampled into exposure $d$. The rates refer to the number of units in the dependency graph, which is $N$.", "what_survives": "Nothing depends on the symbol. The correction concerns the notation only. It does not assess whether the exponents are those of the cited works."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N011", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the sentence that introduces the EPO and of Equation (5). A red box marks the words 'Given that relevant individual positivity assumptions hold,' at the start of the sentence.", "caption": "Section 4.1, p. 6 (PDF p. 6): the definition of the EPO, Equation (5). The marked words give the only condition under which the EPO is defined.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N014.png"}, {"alt": "Crop of two lines of text and Condition 4.1 (Individual positivity). A red box marks the inequality that says c is positive and at most pi_i(d) for every unit i.", "caption": "Section 4.1, p. 6 (PDF p. 6): Condition 4.1. The marked inequality is the whole of the condition.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N014-cond41.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T5", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T5"}, {"id": "P061-T6", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T6"}, {"id": "P061-T7", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T7"}, {"id": "P061-T8", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T8"}, {"id": "P061-T11", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T11"}, {"id": "P061-T16", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T16"}, {"id": "P061-T20", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T20"}], "headline": "Individual positivity alone does not make the EPO of Equation (5) a finite number when the design has infinitely many assignments; the definition also needs a finite first moment for each targeted unit and exposure, a condition that every design with finitely many assignments satisfies.", "id": "P061-N014", "location": "Section 4.1, p. 6 (PDF p. 6), Condition 4.1 and Equation (5); Equations (6)–(8), p. 7; statements on pp. 2 and 14", "pdf_page": 6, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": ["P061-T11", "P061-T16", "P061-T20", "P061-T5", "P061-T6", "P061-T7", "P061-T8"], "text": {"as_printed": "``\\textbf{Condition 4.1} (\\textbf{Individual positivity}). An exposure $d$ satisfies \\emph{individual positivity} if there exists a $c$ such that $0<c\\le\\pi_i(d)\\ \\forall i\\in\\{1,\\ldots,N\\}$.''\n\n``\\hl{Given that relevant individual positivity assumptions hold}, the EPO for unit $i$ under exposure $d$ is defined as''\n\\[\n\\begin{aligned}\n\\overline{y}_i(d)&=\\frac{\\sum_{\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z):g_i(\\mathbf z)=d}y_i(\\mathbf z)\\Pr[\\mathbf Z=\\mathbf z]}{\\sum_{\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z):g_i(\\mathbf z)=d}\\Pr[\\mathbf Z=\\mathbf z]}\\\\\n&=\\operatorname{E}[y_i(\\mathbf Z)\\mid g_i(\\mathbf Z)=d]\\\\\n&=\\operatorname{E}[y_i(\\mathbf Z)\\mid D_i=d].\n\\end{aligned}\n\\qquad(5)\n\\]\n\nIntroduction, p.~2: ``Our defined contrasts coincide with the usual average treatment effects when SUTVA holds, but these targets \\hl{remain well-defined} as combinations of potential outcomes, even under interference or misspecification of exposures---as long as the \\emph{design} is known.''\n\nConclusion, p.~14: ``These quantities \\hl{remain well-defined} under arbitrary interference or exposure misspecification''", "corrected": "\\textbf{Condition 4.1} (\\textbf{Individual positivity}). An exposure $d$ satisfies \\emph{individual positivity} if there exists a $c$ such that $0<c\\le\\pi_i(d)\\ \\forall i\\in\\{1,\\ldots,N\\}$.\n\n\\hl{In addition to individual positivity, assume that, for each targeted unit $i$ and exposure $d$,}\n\\[\n\\hl{\\sum_{\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z):g_i(\\mathbf z)=d}|y_i(\\mathbf z)|\\Pr[\\mathbf Z=\\mathbf z]<\\infty.}\n\\]\nGiven that \\hl{these assumptions} hold, the EPO for unit $i$ under exposure $d$ is defined as\n\\[\n\\begin{aligned}\n\\overline{y}_i(d)&=\\frac{\\sum_{\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z):g_i(\\mathbf z)=d}y_i(\\mathbf z)\\Pr[\\mathbf Z=\\mathbf z]}{\\sum_{\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z):g_i(\\mathbf z)=d}\\Pr[\\mathbf Z=\\mathbf z]}\\\\\n&=\\operatorname{E}[y_i(\\mathbf Z)\\mid g_i(\\mathbf Z)=d]\\\\\n&=\\operatorname{E}[y_i(\\mathbf Z)\\mid D_i=d].\n\\end{aligned}\n\\qquad(5)\n\\]\n\\hl{Under these conditions, the EPOs in Equation (5), their averages and contrasts in Equations (6)--(8), and the corresponding Horvitz--Thompson first-moment unbiasedness statements are well-defined and finite. This condition is automatic for finite assignment support and under NURVA at the targeted exposures.}\n\nIntroduction, p.~2: ``Our defined contrasts coincide with the usual average treatment effects when SUTVA holds, but these targets remain well-defined as combinations of potential outcomes, even under interference or misspecification of exposures---as long as the \\emph{design} is known \\hl{and the targeted first moments are finite}.''\n\nConclusion, p.~14: ``These quantities remain well-defined under arbitrary interference or exposure misspecification\\hl{, provided that the targeted first moments are finite}''\n\nSupplementary Appendix, p.~2: ``The most important of these is that it is finite-$N$ unbiased for the AEPO\\hl{, provided that the first-moment condition holds for exposure $d$ and every unit}:''", "evidence": "The evidence is an exact counterexample that satisfies the setup of Section 3 and Condition 4.1. Let the population consist of one unit, and let $\\Omega=\\mathbb R$ and $\\mathbf Z(\\omega)=\\omega$. Let $p(0)=1/2$, let $p(k)=2^{-k-1}$ for each integer $k\\ge1$, and let $p(z)=0$ otherwise, so that the total mass is $1$. Let $g_1(z)=1$ if $z$ is a positive integer and $g_1(z)=0$ otherwise. Then $\\pi_1(0)=\\pi_1(1)=1/2$, and Condition 4.1 holds with $c=1/2$. Let $y_1(k)=(-1)^{k+1}2^{k+1}$ for each integer $k\\ge1$ and $y_1(z)=0$ otherwise. Every outcome is a finite real number, and $y_1(k)p(k)=(-1)^{k+1}$.\n\n\\begin{audittable}{rrrrr}{The first six assignments with exposure 1 in the counterexample.}\n$k$ & $p(k)$ & $y_1(k)$ & $y_1(k)p(k)$ & Sum up to $k$ \\\\\n1 & $1/4$ & $4$ & $1$ & $1$ \\\\\n2 & $1/8$ & $-8$ & $-1$ & $0$ \\\\\n3 & $1/16$ & $16$ & $1$ & $1$ \\\\\n4 & $1/32$ & $-32$ & $-1$ & $0$ \\\\\n5 & $1/64$ & $64$ & $1$ & $1$ \\\\\n6 & $1/128$ & $-128$ & $-1$ & $0$ \\\\\n\\end{audittable}\n\nUp to assignment $2m$, the positive terms of $\\sum_k y_1(k)p(k)$ sum to $m$ and the negative terms sum to $-m$. Both parts diverge, and the partial sums alternate between $1$ and $0$. The numerator of Equation (5) at $d=1$ is undefined, and the denominator is $1/2$. The EPO at $d=0$ is $0$. Since $N=1$, the AEPO at $d=1$, the EED and the AEED of $d=1$ against $d=0$ are undefined as well. The Horvitz--Thompson estimator is $2Y_1\\mathbf{1}\\{D_1=1\\}$, which is finite at every assignment and has no expectation.\n\nThe corrected statement holds by the following argument. Let\n\\[\nM_{i,d}=\\sum_{\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z):g_i(\\mathbf z)=d}|y_i(\\mathbf z)|\\Pr[\\mathbf Z=\\mathbf z],\n\\]\nand suppose that $M_{i,d}<\\infty$ and $\\pi_i(d)>0$. Then $\\operatorname{E}[|Y_i|\\mid D_i=d]=M_{i,d}/\\pi_i(d)<\\infty$. The numerator of Equation (5) converges absolutely, and $\\overline{y}_i(d)$ is the finite conditional expectation. Since $N$ is finite, Equations (6)--(8) are finite sums and differences of finite numbers. For the estimator, let $T_i=Y_i\\mathbf{1}\\{D_i=d\\}/\\pi_i(d)$. I have $\\operatorname{E}|T_i|=M_{i,d}/\\pi_i(d)<\\infty$ and $\\operatorname{E}[T_i]=\\overline{y}_i(d)$, and linearity of expectation gives $\\operatorname{E}[\\widehat{\\overline{y}}(d)]=\\overline{y}(d)$. The condition $M_{i,d}<\\infty$ holds whenever the support is finite. It also holds under NURVA at a targeted exposure: there $y_i$ takes a single value $a$ on the assignments in the support with exposure $d$, and $M_{i,d}=|a|\\pi_i(d)$. For the bounded outcomes $y_1(k)=(-1)^{k+1}$ on the same design, I find $M_{1,1}=1/2$, and the EPO at $d=1$ and the expectation of the Horvitz--Thompson estimator both equal $1/3$.", "problem": "Section 3 fixes a finite population, real-valued outcomes and a probability measure that is a sum of point masses. It does not restrict the design to finitely many assignments, and it does not bound the outcomes. If the support of the design is countably infinite, then the sum in the numerator of Equation (5) need not converge absolutely, so a finite conditional mean is not guaranteed. Individual positivity controls only the denominator. In the generality of Section 3, positivity does not ensure that the EPO, the AEPO, the EED and the AEED exist, or that the Horvitz--Thompson estimator of supplement Equation (1) has an expectation. Every design in the article has finitely many assignments, and for those designs the targets are well-defined as stated.", "what_survives": "Equations (5)--(8) are unchanged, and the Horvitz--Thompson estimator is unbiased for the AEPO whenever the targeted first moments are finite. The survey, experiment and sampling-treatment examples of the article and Tables 1--5 have finitely many assignments and are unaffected, as are the distinction between NURVA and SUTVA and the variance formulas under NURVA. Proposition 1 of the supplement assumes NURVA and the bound of Condition 1.1, which imply that the targeted first moments are finite. The added condition is needed in the statements on pp.~2, 6--7 and~14 that the targets are well-defined, in the unbiasedness derivation of supplement Equation (3), and for the AEPO and the Horvitz--Thompson expectation used in supplement Equation (10). The proof of supplement Equation (10) also needs the expectations of the predictions in it to be finite."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N014", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text from page 6. A red box marks the word always in the sentence: in practice, researchers can always trim the population to a subset where the assumption is valid.", "caption": "Section 4.1, p. 6 (PDF p. 6), paragraph after Condition 4.1. The marked word “always” omits the case in which no unit is eligible for every exposure in the target.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N015.png"}], "formal_release": null, "formal_statements": [], "headline": "Trimming recovers positivity only when some unit has positive probability of every exposure that the target requires, and the word “always” omits this proviso.", "id": "P061-N015", "location": "Section 4.1, paragraph after Condition 4.1, p. 6 (PDF p. 6), sentence beginning “While this may seem like a stringent assumption”", "pdf_page": 6, "printed_page": "6", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "In our setting, each individual that we conduct inference over must have a non-zero probability of being in the relevant exposure condition. While this may seem like a stringent assumption, in practice, researchers can \\hl{always} trim the population to a subset where the assumption is valid to arrive at interpretable functions of EPOs---although this changes the target population and therefore the estimand.", "corrected": "While this may seem like a stringent assumption, in practice, \\hl{when a nonempty subset of units has positive exposure probabilities for every exposure required by the target,} researchers can trim the \\hl{target} population to \\hl{that subset} to arrive at interpretable functions of EPOs---although this changes the target population and therefore the estimand.", "intended_meaning": "The advice is to restrict the target population to the units that are eligible for the target, and it is correct whenever such units exist. For a finite set $T$ of target exposures, let $U_T=\\{i:\\pi_i(d)>0\\text{ for every }d\\in T\\}$. If $U_T$ is nonempty, then $c=\\min_{i\\in U_T,\\,d\\in T}\\pi_i(d)$ is positive because $U_T$ is finite. Then Condition 4.1 holds on $U_T$, and the targets are averages over $U_T$ with its size as denominator. If $U_T$ is empty, no trimmed population exists. For example, let there be two units with equally likely assignments $(0,0)$ and $(1,1)$, and let $g_1(\\mathbf z)=z_1$ and $g_2(\\mathbf z)=1+z_2$. Then each of the exposures $0$ and $2$ has probability $1/2$ for one unit and probability $0$ for the other. No definition or result in the paper uses the remark."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N015", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of Condition 5.2 on page 8 with two lines of text above it and two below. A red box marks the last equality in the display, equals y bar i of g i of z, which stands before the quantifier for all z and z prime in the design space.", "caption": "Condition 5.2 (SUTVA), p. 8 (PDF p. 8). The marked equality identifies the common raw outcome with the EPO $\\overline y_i(g_i(\\mathbf z))$, which Equation (5) defines only when the exposure has positive probability under the design.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N016.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T12", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T12"}, {"id": "P061-T13", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T13"}], "headline": "Condition 5.2 should require stability of the raw potential outcomes on all of $\\mathcal Z$; its last equality refers to an EPO that Equation (5) leaves undefined at exposures the design never produces.", "id": "P061-N016", "location": "Condition 5.2 (SUTVA), display, p. 8 (PDF p. 8); Equation (5), p. 6", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P061-T12", "P061-T13"], "text": {"as_printed": "Condition 5.2 (SUTVA).\n\\[\ng_i(\\mathbf z)=g_i(\\mathbf z')\\implies y_i(\\mathbf z)=y_i(\\mathbf z')\\hl{{}=\\overline y_i(g_i(\\mathbf z))},\\quad\\forall\\mathbf z,\\mathbf z'\\in\\mathcal Z.\n\\]", "corrected": "Condition 5.2 (SUTVA). For every $i\\in\\{1,\\ldots,N\\}$ and every $\\mathbf z,\\mathbf z'\\in\\mathcal Z$,\n\\[\ng_i(\\mathbf z)=g_i(\\mathbf z')\\implies y_i(\\mathbf z)=y_i(\\mathbf z').\n\\]\nThis defines a common raw outcome $h_i(d)$ for each $d\\in g_i(\\mathcal Z)$. \\hl{When $\\pi_i(d)>0$,} Equation (5) implies $\\overline y_i(d)=h_i(d)$.", "intended_meaning": "The intended condition is that, for every unit and every pair of feasible assignments, equal exposures imply equal raw potential outcomes. Sections 5 and 6 use this condition, and it is correct. The EPO of Equation (5) is a conditional mean under the implemented design, defined only when $\\pi_i(d)>0$, while Condition 5.2 ranges over all of $\\mathcal Z$. For example, let $N=2$ and $\\mathcal Z=\\{0,1\\}^2$, let the assignments $(0,0)$, $(0,1)$, $(1,0)$, $(1,1)$ have probabilities $1/2$, $1/4$, $1/4$, $0$, and let $g_i(\\mathbf z)=z_1+z_2$ and $y_i(\\mathbf z)=i(z_1+z_2)$. The raw potential outcomes satisfy the intended condition on $\\mathcal Z$, and the exposures $0$ and $1$ each have probability $1/2$. Since $\\pi_i(2)=0$, the last term of the display is $0/0$ at $\\mathbf z=\\mathbf z'=(1,1)$. No such gap occurs in Condition 5.1, and no supported EPO, table or later argument changes."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N016", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of seven lines of the Introduction. A red box marks the clause 'but only SUTVA permits off-support interpretations.' at the end of the paragraph that introduces NURVA.", "caption": "Introduction, p. 2 (PDF p. 2): the end of the paragraph that introduces NURVA. The marked clause is the statement about off-support interpretations.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N017.png"}, {"alt": "Crop of six lines of the Conclusion. Red boxes mark the sentence 'SUTVA is required for those off-support interpretations.', which runs from the end of one line to the start of the next.", "caption": "Conclusion, p. 14 (PDF p. 14), first paragraph. The marked sentence repeats the statement.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N017-p14.png"}], "formal_release": null, "formal_statements": [], "headline": "The statements that “only SUTVA permits off-support interpretations” (p. 2) and that SUTVA “is required” for them (p. 14) mean that NURVA alone does not guarantee such an interpretation and that SUTVA suffices for one; a weaker restriction can suffice for a particular target, as in the partial-interference example on p. 9.", "id": "P061-N017", "location": "Introduction, p. 2 (PDF p. 2), last sentence of the paragraph that introduces NURVA; Conclusion, p. 14 (PDF p. 14), first paragraph", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "Introduction, p.~2: ``When NURVA holds but SUTVA does not, contrasts remain causal under the design; but \\hl{only SUTVA permits} off-support interpretations.''\n\nConclusion, p.~14: ``What NURVA buys in such cases is a within-support causal interpretation; what it does not buy is extrapolation to alternative allocations, protocols or populations. \\hl{SUTVA is required} for those off-support interpretations.''", "corrected": "Introduction, p.~2: ``When NURVA holds but SUTVA does not, contrasts remain causal under the design; but \\hl{NURVA alone does not guarantee} off-support interpretations. \\hl{Global exposure stability (Condition 5.2) is sufficient for invariance across designs on the same population, with a fixed exposure mapping and positive probabilities for the relevant exposures, while weaker restrictions linking the particular target assignments can suffice, including the partial-interference case discussed on p.~9.}''\n\nConclusion, p.~14: ``What NURVA buys in such cases is a within-support causal interpretation; what it does not buy is extrapolation to alternative allocations, protocols or populations. \\hl{Such off-support interpretations require additional restrictions; full SUTVA is sufficient for exposure stability on the specified feasible set, but target-specific restrictions may suffice.}''", "intended_meaning": "On p.~8 the article says what SUTVA adds to NURVA. Under Condition 5.2, the outcome of a unit depends on the assignment only through the exposure of that unit, over the whole design space $\\mathcal Z$. The EED can then be interpreted without the qualifiers about the design. NURVA requires this stability only on the support of the design. In that sense both sentences are correct: NURVA alone does not guarantee an off-support interpretation, and the interpretation without qualifiers holds exactly when Condition 5.2 does.\n\nRead as statements about every off-support target, the words ``only'' and ``required'' say too much, and Section 5.2 (p.~9) contains an exception. I consider four units in the clusters $\\{1,2\\}$ and $\\{3,4\\}$, with the design space $\\{0,1\\}^4$, the exposure mapping $g_i(\\mathbf z)=z_i$ and the outcomes $y_i(\\mathbf z)=z_i+z_{m(i)}$, where $m(i)$ is the other unit in the cluster of $i$. Let the design put probability $1/2$ on each of $(1,1,0,0)$ and $(0,0,1,1)$. Individual positivity and NURVA hold. SUTVA fails, since $y_1(1,0,0,0)=1$ and $y_1(1,1,0,0)=2$ at the same exposure. The AEED is $\\tau(1,0)=2$, and it equals the average effect of the assignment $(1,1,1,1)$ against $(0,0,0,0)$, two assignments outside the support. The equality holds for all outcomes of the form $y_i(\\mathbf z)=F_i(z_i,z_{m(i)})$, because under this design $\\overline{y}_i(d)=F_i(d,d)$. NURVA alone does not guarantee this equality. If the outcomes at $(1,1,1,1)$ are changed to $0$, then the distribution of $(\\mathbf Z,\\mathbf Y)$ under the design and the AEED are unchanged, and the average effect of $(1,1,1,1)$ against $(0,0,0,0)$ becomes $0$. No calculation that I inspected uses the stronger reading as a premise."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N017", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of six lines of text from page 9. A red box marks the phrase single exposure level in the sentence: NURVA will hold with respect to that outcome on all designs where all units in a cluster are assigned to a single exposure level.", "caption": "Section 5.2, p. 9 (PDF p. 9), the partial-interference illustration. The marked phrase is correct for a binary treatment with exposure equal to own treatment, and too broad for a coarse exposure label.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N018.png"}], "formal_release": null, "formal_statements": [], "headline": "The partial-interference illustration holds for a binary treatment with the individualistic exposure mapping $g_i(\\mathbf z)=z_i$ and clusters assigned wholly to treatment or wholly to control; a common but coarser exposure label does not suffice.", "id": "P061-N018", "location": "Section 5.2, p. 9 (PDF p. 9), paragraph beginning “Whether each stability assumption holds”, sentence beginning “In such a setting, NURVA will hold”; footnote 9", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": {"as_printed": "For example, consider a setting where there is partial interference on a given outcome. That is, there may be arbitrary interference within, but not across mutually exclusive clusters of units. In such a setting, NURVA will hold with respect to that outcome on all designs where all units in a cluster are assigned to a \\hl{single exposure level}, and the AEED will be equivalent to the global average treatment effect (the effect of treating all subjects relative to no subjects). However, NURVA will not necessarily hold for designs that result in within-cluster variation in treatment assignment.$^{9}$", "corrected": "In such a setting, \\hl{for a binary treatment with the individualistic exposure mapping $g_i(\\mathbf z)=z_i$,} partial interference implies NURVA with respect to that outcome on any design that assigns each cluster \\hl{wholly to treatment or wholly to control}. \\hl{Under the individual positivity required in Condition 4.1, with all-treated and all-control assignments feasible,} the AEED $\\tau(1,0)$ equals the global average treatment effect (the effect of treating all subjects relative to no subjects).", "intended_meaning": "The setting meant is the conventional one, as the contrast with within-cluster variation and footnote 9 indicate: a binary treatment, the individualistic exposure mapping $g_i(\\mathbf z)=z_i$, and a design that assigns each cluster wholly to treatment or wholly to control. In that setting, with all-treated and all-control assignments feasible, the claim is correct. Two assignments in the support under which unit $i$ has the same exposure $d$ agree on the cluster of $i$, and partial interference implies that $y_i$ takes the value $y_i(d\\mathbf 1_N)$ at both. Under the positivity of Condition 4.1, I have $\\overline y_i(d)=y_i(d\\mathbf 1_N)$ by Equation (5), and hence $\\tau(1,0)=N^{-1}\\sum_{i=1}^N[y_i(\\mathbf 1_N)-y_i(\\mathbf 0_N)]$. The wording is too broad only if ``a single exposure level'' may be a coarse label. For example, let the clusters $\\{1,2\\}$ and $\\{3,4\\}$ be assigned raw levels $0$, $1$ or $2$ independently with probability $1/3$ each, let $g_i(\\mathbf z)=\\mathbf 1\\{z_i>0\\}$, and let $y_i(\\mathbf z)$ be the sum of the raw levels of unit $i$ and of the other unit in its cluster. Under the assignments $(1,1,0,0)$ and $(2,2,0,0)$, unit $1$ has the exposure $1$ and the outcomes $2$ and $4$, and NURVA fails. The AEED is $3$, while treating every unit at raw level $1$ or at raw level $2$ has effect $2$ or $4$. No result in the paper uses the sentence."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N018", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of five lines of text from the conclusion on page 14. Red boxes mark the two terms of the formula for the average direct effect: y i of d and bold D minus i at the end of one line, and y i of d prime and bold D minus i at the start of the next line.", "caption": "Section 8 (Conclusion), p. 14 (PDF p. 14), second paragraph. In the two marked terms, the arguments of the raw potential outcome $y_i$, which Equation (1) defines on assignment vectors, are an exposure $d$ or $d'$ and the other units' exposures $\\mathbf D_{-i}$.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N019.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T1", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T1"}, {"id": "P061-T2", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T2"}, {"id": "P061-T3", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T3"}], "headline": "The “average direct effect” of the conclusion is defined for the individualistic exposure mapping $g_i(\\mathbf z)=z_i$, where it equals the AEED under independent assignments; for the general exposure mappings that the paper allows, the expression $y_i(d;\\mathbf D_{-i})$ need not be defined, and the AEED of a coarsened exposure can differ from the average direct effect at every population size.", "id": "P061-N019", "location": "Section 8 (Conclusion), second paragraph, second sentence, p. 14 (PDF p. 14): the formula for the “average direct effect”", "pdf_page": 14, "printed_page": "14", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": ["P061-T1", "P061-T2", "P061-T3"], "text": {"as_printed": "There is additional hope for further refining inference on targets such as AEEDs from Sävje (2024); inference on AEEDs is asymptotically equivalent to inference on the ``average direct effect'' $N^{-1}\\sum_{i=1}^{N}\\bigl(\\mathrm E[y_i(d;\\hl{\\mathbf D_{-i}})-y_i(d';\\hl{\\mathbf D_{-i}})]\\bigr)$ if both statistical dependence in $\\mathbf D$ and unmodeled interference are sufficiently local.", "corrected": "There is additional hope for further refining inference on targets such as AEEDs from Sävje (2024). \\hl{For the individualistic exposure mapping $g_i(\\mathbf z)=z_i$,} define the ``average direct effect'' as\n\\[\nN^{-1}\\sum_{i=1}^{N}\\mathrm E\\bigl[y_i(d;\\hl{\\mathbf Z_{-i}})-y_i(d';\\hl{\\mathbf Z_{-i}})\\bigr],\n\\]\nwhere $y_i(d;\\mathbf z_{-i})$ is the raw potential outcome at the assignment with $i$th coordinate $d$ and remaining coordinates $\\mathbf z_{-i}$, as in footnote 8, and both assignments are assumed to lie in $\\mathcal Z$. Suppose that, for every $i\\in\\{1,\\ldots,N\\}$, $\\pi_i(d)>0$ and $\\pi_i(d')>0$, the assignment \\hl{$Z_i$ is independent of $\\mathbf Z_{-i}$}, and $y_i(\\mathbf Z)$ is integrable. Then the average direct effect \\hl{equals the AEED} $\\tau(d,d')$. For dependent assignments the two targets can differ, and inference on one is inference on the other only under further conditions.\n\nFor a general exposure mapping the same equality holds under an added assumption. Suppose that, for every $i$, there exists a function $H_i$ of the exposure vector such that $H_i(\\mathbf D)$ is integrable and $y_i(\\mathbf z)=H_i(g_1(\\mathbf z),\\ldots,g_N(\\mathbf z))$ for every $\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z)$, that $\\pi_i(d)>0$ and $\\pi_i(d')>0$, and that $D_i$ is independent of $\\mathbf D_{-i}$. Then\n\\[\n\\tau(d,d')=N^{-1}\\sum_{i=1}^{N}\\mathrm E\\bigl[H_i(d,\\mathbf D_{-i})-H_i(d',\\mathbf D_{-i})\\bigr].\n\\]\nNeither statement asserts that inference procedures are equivalent under weaker conditions on the dependence.", "evidence": "The evidence is two exact examples and a derivation. In the first example, let $N=2$ and $\\mathcal Z=\\{0,1\\}^2$, let each assignment have probability $1/4$, and let the two units be adjacent in the network. The mapping of Equation (3) is then $g_1(\\mathbf z)=z_1+2z_2$ and $g_2(\\mathbf z)=z_2+2z_1$. Let $y_i(\\mathbf z)=z_i$.\n\\begin{audittable}{cccc}{The exposure mapping of Equation (3) on two adjacent units.}\nAssignment $\\mathbf z$ & Exposure vector $(D_1,D_2)$ & Outcomes $(y_1,y_2)$ & Probability \\\\\n$(0,0)$ & $(0,0)$ & $(0,0)$ & $1/4$ \\\\\n$(0,1)$ & $(2,1)$ & $(0,1)$ & $1/4$ \\\\\n$(1,0)$ & $(1,2)$ & $(1,0)$ & $1/4$ \\\\\n$(1,1)$ & $(3,3)$ & $(1,1)$ & $1/4$ \\\\\n\\end{audittable}\nEach exposure has probability $1/4$ for each unit, and the AEED $\\tau(1,0)$ equals $1$. Each value of $D_2$ occurs with exactly one value of $D_1$. For no assignment are both $(1,D_2)$ and $(0,D_2)$ exposure vectors, and the contrast for unit $1$ lacks a term in all four cases. The same holds for unit $2$. If $(d,D_2)$ is read as an assignment instead, it lies in $\\{0,1\\}^2$ only when $D_2\\in\\{0,1\\}$, which has probability $1/2$. The same failure occurs for every even $N$ in a population of independent copies of this pair, although each exposure depends on one other exposure only and no outcome depends on another unit's assignment.\n\nIn the second example, let $N\\ge1$, let $Z_1,\\ldots,Z_N$ be independent and uniform on $\\{0,1,2\\}$, and let $g_i(\\mathbf z)=\\mathbf 1\\{z_i>0\\}$, as in Section 6.2, and $y_i(\\mathbf z)=z_i$. Then $\\pi_i(1)=2/3$ and $\\pi_i(0)=1/3$. By Equation (5), I have $\\overline y_i(1)=(1+2)/2=3/2$ and $\\overline y_i(0)=0$. Using Equation (8), I obtain $\\tau(1,0)=3/2$ for every $N$. With $d=1$ and $d'=0$ read as assignment coordinates, the contrast $y_i(1;\\cdot)-y_i(0;\\cdot)$ equals $1$ for every unit and every draw, and the average direct effect is $1$. The two targets differ by $1/2$ at every $N$. The Horvitz--Thompson estimator of the AEED is $(3/(2N))\\sum_{i=1}^NZ_i$, with expectation $3/2$ and variance $3/(2N)$, and it converges in probability to $3/2$. The exposures do not determine a unique outcome either. For $N=2$, the assignments $(1,0)$ and $(2,0)$ have the same exposure vector $(1,0)$, and the outcomes of unit $1$ are $1$ and $2$.\n\\begin{audittable}{ccccc}{Exact values in the second example. The Horvitz--Thompson estimator targets the AEED at every $N$.}\n$N$ & Expectation of the estimator & Variance of the estimator & AEED $\\tau(1,0)$ & Average direct effect \\\\\n$1$ & $3/2$ & $3/2$ & $3/2$ & $1$ \\\\\n$2$ & $3/2$ & $3/4$ & $3/2$ & $1$ \\\\\n$4$ & $3/2$ & $3/8$ & $3/2$ & $1$ \\\\\n$6$ & $3/2$ & $1/4$ & $3/2$ & $1$ \\\\\n\\end{audittable}\nFor the derivation, suppose that $g_i(\\mathbf z)=z_i$, that $\\pi_i(d)>0$ and $\\pi_i(d')>0$, that $Z_i$ is independent of $\\mathbf Z_{-i}$, and that $y_i(\\mathbf Z)$ is integrable. For each $\\mathbf w$ such that $\\Pr[\\mathbf Z_{-i}=\\mathbf w]>0$, the assignment $(d;\\mathbf w)$ has probability $\\Pr[Z_i=d]\\Pr[\\mathbf Z_{-i}=\\mathbf w]>0$, and hence it lies in the support of the design. By Equation (5), I find\n\\[\n\\overline y_i(d)=\\sum_{\\mathbf w}y_i(d;\\mathbf w)\\Pr[\\mathbf Z_{-i}=\\mathbf w\\mid Z_i=d]=\\sum_{\\mathbf w}y_i(d;\\mathbf w)\\Pr[\\mathbf Z_{-i}=\\mathbf w]=\\mathrm E[y_i(d;\\mathbf Z_{-i})].\n\\]\nIn the second equality, I used that $Z_i$ is independent of $\\mathbf Z_{-i}$. The same identity holds for $d'$. Subtracting the two identities and averaging over $i$, I obtain\n\\[\n\\tau(d,d')=\\frac1N\\sum_{i=1}^N\\mathrm E\\bigl[y_i(d;\\mathbf Z_{-i})-y_i(d';\\mathbf Z_{-i})\\bigr].\n\\]\nThe identity uses neither SUTVA nor the absence of interference. In an example with three units, independent binary assignments with probabilities $1/3$, $1/2$, $2/3$ and outcomes $y_i(\\mathbf z)=z_i+2z_{i+1}+3z_iz_{i+1}$ with cyclic indices, both sides equal $5/2$. The second statement in the correction follows from the same computation with $\\mathbf D$ in place of $\\mathbf Z$ and $H_i$ in place of $y_i$.", "problem": "Equation (1) defines the raw potential outcome $y_i$ on assignment vectors $\\mathbf z\\in\\mathcal Z$, and Equation (2) defines the exposure vector $\\mathbf D=(g_1(\\mathbf Z),\\ldots,g_N(\\mathbf Z))$ through exposure mappings that the researcher chooses. In the marked expression, the arguments of $y_i$ are an exposure $d$ for unit $i$ and the exposures $\\mathbf D_{-i}$ of the other units. The paper does not define $y_i$ on such arguments. For the individualistic exposure mapping $g_i(\\mathbf z)=z_i$, I have $\\mathbf D=\\mathbf Z$, and the expression is the average of the contrast in footnote 8 (p.~7), which holds fixed the treatments assigned to the other units. The sentence does not state this restriction, and the paragraph before it speaks of arbitrary exposure misspecification. For a general exposure mapping the comparison fails in two ways. First, the vector $(d,\\mathbf D_{-i})$ need not lie in $\\mathcal Z$ and need not be the exposure vector of any assignment. Further, two assignments with the same exposure vector can have different outcomes, so the exposures need not determine a unique outcome. Second, when $d$ and $d'$ are read as assignment coordinates and the expression has a value, the AEED of a coarsened exposure can differ from it by the same amount at every $N$, even if the assignments are independent and no outcome depends on another unit's assignment. The comparison is correct for the individualistic exposure mapping under independent assignments, where the two targets are equal.", "what_survives": "The assignment space, the definitions of the EPO, AEPO, EED and AEED, the interpretation under NURVA and SUTVA, the examples and the arguments on external validity do not use this formula. The results on estimation and inference in the supplement do not use it either. No later result, table or figure depends on the sentence. Only the closing remark changes. The comparison with the average direct effect is restricted to the individualistic exposure mapping, or to exposure mappings for which the outcome is a function of the exposure vector. For dependent assignments a further condition is needed, which is the subject of \\fid{P061-N009}."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N019", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [{"alt": "Crop of three lines of the derivation in Equation (3) of the supplement. A red box marks the second term of the numerator in the fourth line, the conditional expectation of Y_i 1{D_i = d} given D_i not equal to d, multiplied by Pr[D_i not equal to d].", "caption": "Supplementary Appendix, p. 2: the third, fourth and fifth lines of Equation (3). The marked term conditions on the event $D_i\\neq d$.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N020.png"}, {"alt": "Crop of note 1 of the supplement. A red box marks the term E[Y_i 1{D_i = d} | D_i not equal to d] in the second line of the note.", "caption": "Supplementary Appendix, p. 10: note 1. The marked term is the same conditional expectation.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N020-note1.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T16", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T16"}], "headline": "The fourth line of supplement Equation (3), and note 1, condition on the event $D_i\\neq d$, which has probability zero when $\\pi_i(d)=1$; the term is meant to contribute zero, and the derivation holds without it.", "id": "P061-N020", "location": "Supplementary Appendix, p. 2 (PDF p. 2 of the supplement), Equation (3), fourth line; note 1, p. 10 (PDF p. 10 of the supplement)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P061-T16"], "text": {"as_printed": "Equation (3), fourth line:\n\\[\n=\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\operatorname{E}[Y_i\\mathbf{1}\\{D_i=d\\}\\mid D_i=d]\\Pr[D_i=d]+\\hl{\\operatorname{E}[Y_i\\mathbf{1}\\{D_i=d\\}\\mid D_i\\neq d]\\Pr[D_i\\neq d]}}{\\pi_i(d)}\n\\]\n\nNote 1, p.~10: ``The term involving $D_i\\neq d$ of Equation 3 demonstrates why the coding of the potential outcome for unsurveyed groups is irrelevant, because the term \\hl{$\\operatorname{E}[Y_i\\mathbf{1}\\{D_i=d\\}\\mid D_i\\neq d]$} will always be equal to zero.''", "corrected": "Equation (3), with the third line followed directly by the fifth:\n\\[\n\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\operatorname{E}[Y_i\\mathbf{1}\\{D_i=d\\}]}{\\pi_i(d)}=\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\operatorname{E}[Y_i\\mid D_i=d]\\Pr[D_i=d]}{\\pi_i(d)},\n\\]\n\\hl{where $\\operatorname{E}[Y_i\\mathbf{1}\\{D_i=d\\}]=\\pi_i(d)\\operatorname{E}[Y_i\\mid D_i=d]$, including when $\\pi_i(d)=1$.}\n\nNote 1: ``\\hl{The coding of the potential outcome for unsurveyed groups is irrelevant, because $Y_i\\mathbf{1}\\{D_i=d\\}$ is zero whenever $D_i\\neq d$.}''", "intended_meaning": "Condition 4.1 requires $\\pi_i(d)\\ge c>0$ and allows $\\pi_i(d)=1$, for instance in a census. Then the event $D_i\\neq d$ has probability zero, and the expectation conditional on it is not defined as a ratio. The term is meant to contribute zero, because $Y_i\\mathbf{1}\\{D_i=d\\}$ vanishes whenever $D_i\\neq d$. The derivation does not need the split. Summing over the assignments with $g_i(\\mathbf z)=d$, I have\n\\[\n\\operatorname{E}[Y_i\\mathbf{1}\\{D_i=d\\}]=\\sum_{\\mathbf z\\in\\operatorname{Supp}(\\mathbf Z):g_i(\\mathbf z)=d}y_i(\\mathbf z)\\Pr[\\mathbf Z=\\mathbf z]=\\pi_i(d)\\operatorname{E}[Y_i\\mid D_i=d],\n\\]\nand this identity holds when $\\pi_i(d)=1$. It uses only the first moments that the AEPO already requires (see \\fid{P061-N014}). Finite-$N$ unbiasedness and the remark that the coding of unsurveyed groups is irrelevant are unchanged."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N020", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}
{"before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [{"alt": "Crop of the foot of page 2 of the supplement. A red box marks the words 'we remove all implicit probabilistic dependence of Y_i on Z:' in the sentence beginning 'Under NURVA'. Equation (4) follows below it.", "caption": "Supplementary Appendix, p. 2: the sentence that introduces Equation (4), and the equation. The marked words describe what NURVA removes.", "url": "https://pmaronow.github.io/llm-audit/works/P061/exhibits/P061-N021.png"}], "formal_release": null, "formal_statements": [{"id": "P061-T18", "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-T18"}], "headline": "Under NURVA the outcome does not vary within an exposure on the support of the design; it still depends on the assignment through the exposure, and Equation (4) uses only the identity within an exposure.", "id": "P061-N021", "location": "Supplementary Appendix, p. 2 (PDF p. 2 of the supplement), sentence before Equation (4)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": ["P061-T18"], "text": {"as_printed": "``Under NURVA, we \\hl{remove all implicit probabilistic dependence of $Y_i$ on $\\mathbf Z$}:''\n\\[\n\\widehat{\\overline{y}}(d)=\\frac{1}{N}\\sum_{i=1}^{N}\\frac{Y_i\\mathbf{1}\\{D_i=d\\}}{\\pi_i(d)}=\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\overline{y}_i(d)\\mathbf{1}\\{D_i=d\\}}{\\Pr[D_i=d]}.\\qquad(4)\n\\]", "corrected": "``Under NURVA, \\hl{there is no residual outcome variation within exposure $d$ on the support of the design, so that $Y_i\\mathbf{1}\\{D_i=d\\}=\\overline{y}_i(d)\\mathbf{1}\\{D_i=d\\}$}:''\n\\[\n\\widehat{\\overline{y}}(d)=\\frac{1}{N}\\sum_{i=1}^{N}\\frac{Y_i\\mathbf{1}\\{D_i=d\\}}{\\pi_i(d)}=\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\overline{y}_i(d)\\mathbf{1}\\{D_i=d\\}}{\\Pr[D_i=d]}.\\qquad(4)\n\\]\nEquation (4) is unchanged.", "intended_meaning": "Under NURVA (Condition 5.1), the outcome of unit $i$ is the same at all assignments in the support of the design that have the same exposure. Hence $Y_i\\mathbf{1}\\{D_i=d\\}=\\overline{y}_i(d)\\mathbf{1}\\{D_i=d\\}$ with probability one. Equation (4) uses this identity and no more, and it is correct. NURVA does not make $Y_i$ independent of $\\mathbf Z$. With one unit, a fair binary assignment, $g_1(\\mathbf z)=z_1$ and $y_1(\\mathbf z)=z_1$, both NURVA and SUTVA hold and $Y_1=Z_1$, so that $\\Pr[Y_1=1\\mid Z_1=1]=1$ and $\\Pr[Y_1=1]=1/2$. In each summand of Equation (4), the remaining assignment randomness is carried by the indicator $\\mathbf{1}\\{D_i=d\\}$."}, "url": "https://pmaronow.github.io/llm-audit/works/P061/#P061-N021", "work": {"arxiv_id": "2505.10519", "doi": "10.1017/pan.2026.10045", "title": "On the Foundations of the Design-Based Approach", "url": "https://pmaronow.github.io/llm-audit/works/P061/"}, "work_id": "P061"}