{ "claims": [ { "claim": "With a binary instrument and the five potential-outcome assumptions, IV estimation identifies the average treatment effect among compliers (the LATE), which can differ from the ATE for the whole population when effects are heterogeneous.", "claim_id": "P029-H1", "related_corrections": [ "P029-E001", "P029-E002" ], "tied_to": "Abstract; Section 2; equations (3)-(4); footnote 2; the gender example in Section 3.2" }, { "claim": "Weighting units by the inverse of the compliance score and then applying IV/2SLS (ICSW) consistently estimates the ATE, given a randomized instrument and the usual IV assumptions, latent ignorability of compliance given covariates (Assumption 1), a consistent compliance-score estimator (Assumption 2) and compliance scores bounded away from zero (Assumption 3).", "claim_id": "P029-H2", "related_corrections": [ "P029-E003", "P029-E004", "P029-E006", "P029-E009" ], "tied_to": "Abstract; introduction p. 493; Sections 3.3-3.4; Assumptions 1-3; Appendix 'ICSW Consistency', eqs. (8)-(14)" }, { "claim": "A new nested maximum-likelihood estimator of the compliance score handles two-sided noncompliance (both always-takers and never-takers), reduces to binomial (probit) regression on the assigned-to-treatment units under one-sided noncompliance, and reduces to strata means under saturation.", "claim_id": "P029-H3", "related_corrections": [ "P029-E005", "P029-E008", "P029-E009", "P029-E014", "P029-E016", "P029-E017", "P029-N010" ], "tied_to": "Introduction p. 493; Section 3.1, eqs. (5)-(7), footnote 3; Assumption 2; the compliance-score code (Dataverse icsw.R and the icsw package)" }, { "claim": "Under Assumptions 2 and 3 and the five IV assumptions, if latent ignorability fails, the paper says ICSW still converges to an internally valid causal effect: the complier effect reweighted to the covariate distribution of the full population, $\\int E(Y_1-Y_0\\mid D_1>D_0,\\mathbf{X}=\\mathbf{x})f(\\mathbf{x})\\,d\\mathbf{x}$. The paper presents 2SLS as the special case with no covariates in the compliance model.", "claim_id": "P029-H4", "related_corrections": [ "P029-E004" ], "tied_to": "Introduction p. 493; Section 3.4 pp. 499-500; Appendix eqs. (10) and (13); the two integral displays on p. 499" }, { "claim": "In Green, Gerber and Nickerson (2003), the paper reports estimated turnout effects of $5.6$ percentage points among compliers (interval $2.2$ to $8.9$) and $6.7$ points in the full population (interval $3.2$ to $10.3$), making the ATE estimate $19\\%$ larger than the LATE estimate.", "claim_id": "P029-H5", "related_corrections": [ "P029-E010", "P029-E021", "P029-N004" ], "tied_to": "Section 4.1, pp. 500-502; Table 1; footnotes 10-11; supplement Table A7" }, { "claim": "In Albertson and Lawrence (2009), the paper reports estimated effects of watching the debate on self-reported knowledge of $0.28$ among compliers and $0.40$ in the full population (approximately $45\\%$ larger), while the estimated effects on support for the proposition are similar ($-0.07$ and $-0.05$) and indistinguishable from zero.", "claim_id": "P029-H6", "related_corrections": [ "P029-E010", "P029-E012", "P029-N006", "P029-N010" ], "tied_to": "Section 4.2, pp. 502-503; Table 2; footnotes 12-13; supplement Table A8" }, { "claim": "The paper recommends the Winsorization constant $\\alpha=0.275$ based on its simulations. Its supplementary tables report lower RMSEs for ICSW than for 2SLS under the correctly specified, “Wrong Model” and “Only X1” designs, and similar RMSEs under the two noise designs. The paper says that the bootstrap has conservative coverage in small samples and proper coverage in larger ones.", "claim_id": "P029-H7", "related_corrections": [ "P029-E009", "P029-E011", "P029-A001", "P029-N009" ], "tied_to": "Footnotes 6 and 7, p. 498; supplement Figures A1-A3 and Tables A1-A6" }, { "claim": "The two applications show that the LATE has the potential to differ substantively from the ATE, so researchers should attend to the estimand; when the ATE is the target, ICSW may be a more sensible estimator than standard IV, given the additional assumptions and the finite-sample uncertainty the paper describes.", "claim_id": "P029-H8", "related_corrections": [ "P029-E004", "P029-E006", "P029-E011", "P029-E016", "P029-N009" ], "tied_to": "Abstract; introduction p. 493; Section 3.4 pp. 499-500; conclusion p. 503; Tables 1 and 2; footnotes 11 and 13" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of equation (2) and the eight lines below it. Equation (2) reads D sub i = gamma 0 + gamma 1 Z sub i + delta 1 Q sub 1i + delta 2 Q sub 2i + ... + delta K Q sub Ki + e sub i. The text explains two-stage least squares and ends: Given the model in equations (1) and (2), the SEM IV framework relies on two main assumptions for consistency. First, the correlation between Z sub i and u sub i must approach zero as n grows. Second, the covariance between Z sub i and D sub i must converge to a nonzero quantity as n grows. Red outlines surround the two sentences that begin First and Second.", "caption": "Section 2.1, p. 494 (PDF p. 3). The outlines mark the two conditions for consistency of 2SLS. Equation (2), at the top of the crop, contains the covariates $Q_{1i},\\dots,Q_{Ki}$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E001.png" }, { "alt": "Crop of four lines. The first sentence reads: We are able to consistently estimate beta 1 using ordinary least squares (OLS) regression when D sub i is asymptotically uncorrelated with the error term (one over n times the sum of D sub i u sub i converges in probability to 0). The next sentence says that this requirement is violated when D sub i is endogenous. The crop ends partway through the sentence about finding an instrument, after Y sub i. A red outline surrounds the words asymptotically uncorrelated with the error term and the formula in parentheses.", "caption": "Section 2.1, p. 494 (PDF p. 3), first lines of the page. The outline marks the condition for consistency of OLS.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E001-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The consistency conditions of Section 2.1 are stated for $D_i$ and $Z_i$ themselves, although equations (1) and (2) contain covariates; the conditions meant are the usual ones for $D_i$ and $Z_i$ after the covariates are partialled out.", "id": "P029-E001", "location": "Section 2.1, p. 494 (PDF p. 3), first sentence of the page and the two sentences after “two main assumptions for consistency”; equation (1) is on p. 493 (PDF p. 2)", "pdf_page": 3, "printed_page": "494", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "We are able to consistently estimate $\\beta_1$ using ordinary least squares (OLS) regression when $D_i$ is asymptotically uncorrelated with the error term ($\\hl{\\frac{1}{n}\\Sigma D_i u_i \\to_p 0}$). This requirement is violated, however, when $D_i$ is endogenous. [\\ldots] Given the model in equations (1) and (2), the SEM IV framework relies on two main assumptions for consistency. First, the \\hl{correlation between $Z_i$ and $u_i$} must approach zero as $n$ grows. Second, the \\hl{covariance between $Z_i$ and $D_i$} must converge to a nonzero quantity as $n$ grows.", "corrected": "Suppose that \\hl{the observations are independent and identically distributed}, that the second moments are finite and that the second-moment matrix of the intercept and the covariates is nonsingular. Let $d_i$ and $z_i$ be the residuals from the linear projections of $D_i$ and $Z_i$ on an intercept and $Q_{1i},\\dots,Q_{Ki}$. OLS is consistent for $\\beta_1$ when \\hl{$E(d_iu_i) = 0$ and $E(d_i^2) > 0$}. 2SLS with the single instrument $Z_i$ is consistent for $\\beta_1$ when \\hl{$E(z_iu_i) = 0$ and $E(z_iD_i) \\neq 0$}. For other sampling schemes, require stable probability limits for the sample moments and a nonsingular limiting controls matrix, with the same zero-moment and nonzero-denominator conditions using sample projection residuals. Orthogonality of $u_i$ to every regressor, or to every instrument, together with the usual rank condition, is a stronger sufficient condition.", "intended_meaning": "The passage is a short account of OLS and 2SLS with one endogenous treatment. It takes for granted that the covariates $Q_{1i},\\dots,Q_{Ki}$ are valid controls, which serve as their own instruments, and that the instrument is related to the treatment when the covariates are held fixed. With these conventions the account is correct, and the phrase ``two main assumptions'' does not claim to be a complete list.\n\nThe printed conditions do not suffice if they are read literally, as conditions on $D_i$ and $Z_i$ alone. Let $d_i$ and $z_i$ be the residuals from the linear projections of $D_i$ and $Z_i$ on an intercept and the covariates. For independent, identically distributed sampling with finite second moments, and a nonsingular second-moment matrix of the intercept and the covariates, the two estimators have the limits\n\\[\n\\operatorname{plim}\\widehat\\beta_{1}^{\\mathrm{OLS}} = \\beta_1 + \\frac{E(d_iu_i)}{E(d_i^2)}, \\qquad \\operatorname{plim}\\widehat\\beta_{1}^{\\mathrm{IV}} = \\beta_1 + \\frac{E(z_iu_i)}{E(z_iD_i)},\n\\]\nwhen the denominators are not zero. An example shows the difference. Let $Z$ equal $0$ or $1$ with probability $1/2$ each, let $U$ and $V$ equal $-1$ or $1$ with probability $1/2$ each, and let $Z$, $U$ and $V$ be independent. Set $D = Z$, $Q = D + U/2 + V/4$, $u = U$ and $Y = 2 + D + U$. Equations (1) and (2) hold with $\\beta_1 = 1$ and $\\lambda_1 = 0$. I have $E(Du) = 0$, $\\operatorname{Cov}(Z,u) = 0$ and $\\operatorname{Cov}(Z,D) = 1/4$, and the moment matrices of the regressors and of the instruments are nonsingular, with determinant $5/64$. The covariate is correlated with the error, since $E(Qu) = 1/2$. A direct computation gives $E(zu) = -2/9$ and $E(zD) = 5/36$. Substituting these values into the two limits, I find that both OLS and 2SLS converge to $1 + (-2/9)/(5/36) = -3/5$. In a second example the covariate is exogenous and $\\operatorname{Cov}(Z,D) = 1/16$, but $E(zD) = 0$, and $\\beta_1$ is not identified.\n\nNo later proof or calculation in the paper is shown to rely on the literal reading. From Section~2.2 on, the instrument is randomized and independent of the pretreatment covariates, and the argument for the LATE and for ICSW does not use the conditions of Section~2.1." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E001", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of eight lines of text. A red box marks the words 'these assumptions are considerably weaker than (and subsumed by) those of the SEM framework.' The lines above state the fourth assumption, that there are no defiers, and the fifth, that Z is randomly assigned.", "caption": "Section 2.2, p. 495 (PDF p. 4), end of the paragraph that lists the five IV assumptions. The marked sentence compares them with the assumptions of the SEM framework.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E002.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T11", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T11" }, { "id": "P029-T12", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T12" } ], "headline": "The five potential-outcome assumptions are not weaker than, or subsumed by, the SEM assumptions of Section 2.1: neither set of assumptions implies the other.", "id": "P029-E002", "location": "Section 2.2, p. 495 (PDF p. 4), last sentence of the paragraph that lists the five IV assumptions", "pdf_page": 4, "printed_page": "495", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T11", "P029-T12" ], "text": { "as_printed": "Fourth, we assume that the population contains no defiers, i.e., $\\Pr(D_0 > D_1) = 0$. Fifth, we assume that $Z_i$ is randomly assigned and thus independent of all potential outcomes. As no parametric assumptions are necessary, these assumptions are \\hl{considerably weaker than (and subsumed by)} those of the SEM framework.", "corrected": "Fourth, we assume that the population contains no defiers, i.e., $\\Pr(D_0 > D_1) = 0$. Fifth, we assume that $Z_i$ is randomly assigned and thus independent of all potential outcomes. \\hl{These assumptions permit heterogeneous treatment effects but impose random assignment and absence of defiers, neither of which follows from the SEM conditions described above. The two sets of causal assumptions are therefore distinct and non-nested.}", "evidence": "The evidence is three exact counterexamples. Each is a finite population with binary $Z$ and $D$, no covariates and independently sampled units, in which $D=D_Z$ and $Y=Y_D$.\n\\begin{itemize}\n\\item \\emph{The SEM conditions hold and the population contains defiers.} A unit is a complier with probability $3/4$ and a defier with probability $1/4$, and $Z\\sim\\mathrm{Bernoulli}(1/4)$ is independent of the type. I set $u=3/4$ for a defier and $u=-1/4$ for a complier, $Y_0=u$ and $Y_1=u+1$. Then $Y=D+u$ with $\\beta_1=1$, and $D=1/4+Z/2+e$ with $\\mathrm{E}(e\\mid Z)=0$. I have $\\operatorname{Var}(Z)=\\operatorname{Var}(u)=3/16$, $\\operatorname{Cov}(Z,u)=0$ and $\\operatorname{Cov}(Z,D)=3/32$. The instrument is independent of all potential outcomes and the exclusion restriction holds, but $\\Pr(D_0>D_1)=1/4$.\n\\item \\emph{The SEM conditions hold and $Z$ is not independent of the potential outcomes.} Let $U$ be uniform on $\\{-2,-1,1,2\\}$, and set $Z=\\mathbf{1}\\{|U|=2\\}$, $D_0=\\mathbf{1}\\{U>0\\}$, $D_1=1$, $Y_0=U$ and $Y_1=U+1$. Then $Y=D+U$, and $D=1/2+Z/2+e$ with $\\mathrm{E}(e\\mid Z)=0$. I have $\\mathrm{E}(U\\mid Z)=0$, $\\operatorname{Cov}(Z,U)=0$ and $\\operatorname{Cov}(Z,D)=1/8$, and there are no defiers. But $\\Pr(|Y_0|=2\\mid Z=1)=1$ and $\\Pr(|Y_0|=2\\mid Z=0)=0$.\n\\item \\emph{The five assumptions hold and the effect is not constant.} Let $H$ be uniform on $\\{1,2\\}$, and let $Z\\sim\\mathrm{Bernoulli}(1/2)$ be independent of $H$. Every unit is a complier, $Y_0=0$ and $Y_1=H$. The effect is $1$ for half of the units and $2$ for the other half, while a causal reading of equation (1) requires that $Y_1-Y_0=\\beta_1$ for every unit.\n\\end{itemize}\nNote that in the first two populations the treatment is endogenous ($\\operatorname{Cov}(D,u)$ is $3/32$ and $1/4$), and the population IV ratio equals the common effect $1$. These populations are not counterexamples to IV estimation under a constant effect.", "problem": "The sentence asserts that the SEM conditions of Section 2.1 imply the five assumptions just listed. For the model in equations (1) and (2), with a common effect $\\beta_1$, Section 2.1 requires that the correlation between $Z_i$ and $u_i$ tend to zero and that the covariance between $Z_i$ and $D_i$ tend to a nonzero limit. These conditions do not exclude defiers, and a zero correlation does not make $Z_i$ independent of the potential outcomes. In the other direction, the five assumptions let the effect $Y_1-Y_0$ vary across units, and the SEM formulation does not. Neither set of assumptions implies the other. The point that the passage serves is correct: the potential-outcome formulation permits heterogeneous treatment effects.", "what_survives": "Equation (3) is stated under the five assumptions themselves, and Section 3.4 invokes the IV assumptions directly for ICSW. Neither uses the comparison, and no simulation or application result depends on it. Only the comparison changes. The claim that one set of assumptions contains the other is withdrawn." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E002", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of five lines of text. The first lines display the two equalities of the typical ignorability assumption, for Y1 and for Y0 given D = 1 and D = 0. Two red boxes mark the words 'Assumption 1 is less restrictive than the typical ignorability assumption', which are followed by 'as there is no need to account for baseline selection bias'.", "caption": "Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1. The marked words rank Assumption 1 below the ignorability assumption displayed in the lines above.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E003.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T2", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T2" }, { "id": "P029-T20", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T20" } ], "headline": "Assumption 1 is not less restrictive than the ignorability assumption displayed beside it: under two-sided noncompliance neither assumption implies the other.", "id": "P029-E003", "location": "Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1", "pdf_page": 8, "printed_page": "499", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P029-T2", "P029-T20" ], "text": { "as_printed": "Assumption 1 entails that, for all $\\mathbf{x} \\in \\mathrm{Supp}(\\mathbf{X})$, $\\mathrm{E}(Y_1 - Y_0|\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_1 - Y_0|D_1 > D_0, \\mathbf{X}=\\mathbf{x})$: [\\ldots]\n\nHowever, Assumption 1 may be distinguished from the typical ignorability assumption (Rubin 1978), which requires that both $\\mathrm{E}(Y_1|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_1|D=0,\\mathbf{X}=\\mathbf{x})$ and $\\mathrm{E}(Y_0|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_0|D=0,\\mathbf{X}=\\mathbf{x})$. \\hl{Assumption 1 is less restrictive than the typical ignorability assumption} as there is no need to account for baseline selection bias, only the determinants of treatment effect heterogeneity.", "corrected": "However, Assumption 1 may be distinguished from the typical ignorability assumption (Rubin 1978), which requires that both $\\mathrm{E}(Y_1|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_1|D=0,\\mathbf{X}=\\mathbf{x})$ and $\\mathrm{E}(Y_0|D=1,\\mathbf{X}=\\mathbf{x}) = \\mathrm{E}(Y_0|D=0,\\mathbf{X}=\\mathbf{x})$. \\hl{Assumption 1 and the typical ignorability assumption displayed here impose different restrictions and are not generally ordered. Assumption 1 permits baseline selection bias, provided that the conditional average treatment effect for compliers equals that for the full population.}\n\n\\hl{Under one-sided noncompliance ($D_0=0$), random assignment and positive probabilities of both assignment arms and compliance at each covariate profile, the typical ignorability assumption implies Assumption 1.}", "evidence": "I use two exact counterexamples with a single covariate profile. In both, a unit is a complier ($C$), an always-taker ($A$) or a never-taker ($N$) with probabilities $1/2$, $1/4$ and $1/4$, and $Z\\sim\\mathrm{Bernoulli}(1/2)$ is independent of the type. The five IV assumptions hold, the difference in treatment rates between the arms is $1/2$, and Assumption 3 holds with $\\epsilon=1/4$. Half of the treated units and half of the untreated units are compliers.\n\\begin{audittable}{lrrr}{Potential outcomes $(Y_0,Y_1)$ by type in the two populations.}\nType & Probability & Population 1 & Population 2 \\\\\n\\hline\nComplier & $1/2$ & $(0,1)$ & $(0,1)$ \\\\\nAlways-taker & $1/4$ & $(0,0)$ & $(1,2)$ \\\\\nNever-taker & $1/4$ & $(0,0)$ & $(0,1)$ \\\\\n\\end{audittable}\n\\begin{audittable}{lrr}{The two assumptions in the two populations.}\nQuantity & Population 1 & Population 2 \\\\\n\\hline\n$\\mathrm{E}(Y_0|D=1)$ and $\\mathrm{E}(Y_0|D=0)$ & $0$ and $0$ & $1/2$ and $0$ \\\\\n$\\mathrm{E}(Y_1|D=1)$ and $\\mathrm{E}(Y_1|D=0)$ & $1/2$ and $1/2$ & $3/2$ and $1$ \\\\\nIgnorability, as displayed & holds & fails \\\\\n$\\mathrm{E}(Y_1-Y_0)$ and $\\mathrm{E}(Y_1-Y_0|D_1>D_0)$ & $1/2$ and $1$ & $1$ and $1$ \\\\\nAssumption 1 & fails & holds \\\\\nLimit of ICSW minus the ATE & $1/2$ & $0$ \\\\\n\\end{audittable}\nNote that in population 1 the pair $(Y_0,Y_1)$ is independent of $D$, which is stronger than the displayed equalities. In general, with $\\mu_{td}$ the mean of $Y_d$ for type $t$ and $\\tau_t=\\mu_{t1}-\\mu_{t0}$, ignorability holds in these populations if and only if $\\mu_{Ad}=\\mu_{Nd}$ for $d=0,1$, and Assumption 1 holds if and only if $\\tau_C=(\\tau_A+\\tau_N)/2$.\n\nThe implication holds in one case. Suppose that noncompliance is one-sided, $D_0=0$ for every unit, with random assignment and positive probabilities of both assignment arms and compliance at each covariate profile $\\mathbf{x}$. If ignorability holds, let $m_d$ be the common value of $\\mathrm{E}(Y_d|D,\\mathbf{X}=\\mathbf{x})$, so that $\\mathrm{E}(Y_d|\\mathbf{X}=\\mathbf{x})=m_d$. The treated units are the compliers assigned to treatment, and I have $m_d=\\mathrm{E}(Y_d|D_1>D_0,Z=1,\\mathbf{X}=\\mathbf{x})=\\mathrm{E}(Y_d|D_1>D_0,\\mathbf{X}=\\mathbf{x})$. Subtracting the identity for $d=0$ from the identity for $d=1$, I obtain Assumption 1.", "problem": "The sentence ranks two conditions at a fixed covariate profile. The ignorability assumption equates the mean of each potential outcome between the units that received treatment and the units that did not. Assumption 1 equates the mean effect among compliers with the mean effect among all units. To say that Assumption 1 is less restrictive is to say that ignorability implies it. The reason given, that Assumption 1 need not account for baseline selection, shows only that Assumption 1 can hold when ignorability fails. Under two-sided noncompliance the implication fails as well. The treated group mixes compliers with always-takers, and the untreated group mixes compliers with never-takers. In the two populations below, ignorability compares always-takers with never-takers and places no restriction on complier means. Assumption 1 compares compliers with everyone else. The method covers two-sided noncompliance (p.~496), and the Albertson and Lawrence application has it. Someone choosing between ICSW and a method that assumes selection on observables is told that the ICSW assumption is the weaker of the two. It is a different assumption.", "what_survives": "Assumption 1 and the consistency argument, which invokes Assumption 1 between equations (10) and (11), are unaffected. The observation that Assumption 1 permits baseline selection and the example of the medical trial also stand. This correction changes no estimate. The ranking is withdrawn. In an application with two-sided noncompliance, the reading of the ICSW estimate as an ATE rests on Assumption 1 itself, and ignorability of the treatment received does not supply it. Whether Assumption 1 holds in either application is not established here." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E003", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of six lines of text. Red boxes mark the sentence 'More generally, even if treatment effects are heterogeneous, but still statistically independent of compliance, Assumption 1 would necessarily be satisfied, and both 2SLS and ICSW would be consistent for the ATE.'", "caption": "Section 3.4, p. 499 (PDF p. 8). The marked sentence states a sufficient condition for Assumption 1 and for the consistency of both estimators.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E004.png" }, { "alt": "Two rows of panels with a common horizontal axis, the probability limit of the estimator, marked at 3/8, 7/16, 1/2, 9/16 and 5/8; a vertical line stands at the ATE of 1/2. In population A, effects independent of compliance but not given X, the 2SLS point (blue circle) lies on the line at 1/2 and the ICSW point (orange diamond) lies to the left at 7/16. In population B, effects independent of compliance given X but not marginally, the 2SLS point lies to the right at 5/8 and the ICSW point lies on the line at 1/2.", "caption": "Probability limits of 2SLS and ICSW in the two exact populations, against the ATE of $1/2$ (vertical line). The marginal and conditional readings of the sentence on p. 499 each fail for one estimator. In population A the ICSW limit is $7/16$, and in population B the 2SLS limit is $5/8$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E004-limits.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T3", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T3" }, { "id": "P029-T4", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T4" }, { "id": "P029-T21", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T21" } ], "headline": "Independence of the treatment effects from compliance does not imply Assumption 1: the sentence requires that the mean effect among compliers equal the population mean effect both unconditionally, for 2SLS, and given every covariate profile, for Assumption 1 and ICSW.", "id": "P029-E004", "location": "Section 3.4, p. 499 (PDF p. 8), last sentence of the paragraph beginning “We can describe sufficient conditions for and consequences of Assumption 1”", "pdf_page": 8, "printed_page": "499", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P029-T21", "P029-T3", "P029-T4" ], "text": { "as_printed": "More generally, even if treatment effects are heterogeneous, but still \\hl{statistically independent of compliance, Assumption 1 would necessarily be satisfied}, and both 2SLS and ICSW would be consistent for the ATE.", "corrected": "More generally, \\hl{with heterogeneous treatment effects, if the mean treatment effect among compliers equals the population mean both unconditionally and conditional on every covariate profile $\\mathbf{X}$ used in the compliance score}, Assumption 1 \\hl{holds} and, \\hl{under the other stated assumptions and regularity conditions}, both 2SLS and ICSW \\hl{are} consistent for the ATE.\n\nIn symbols, the two conditions are $\\mathrm{E}(Y_1-Y_0|D_1>D_0)=\\mathrm{E}(Y_1-Y_0)$ and $\\mathrm{E}(Y_1-Y_0|D_1>D_0,\\mathbf{X}=\\mathbf{x})=\\mathrm{E}(Y_1-Y_0|\\mathbf{X}=\\mathbf{x})$ for every $\\mathbf{x}\\hl{\\in\\operatorname{Supp}(\\mathbf{X})}$. The first suffices for 2SLS, and the second, which is Assumption 1, suffices for ICSW.", "evidence": "The evidence is two exact counterexamples and the identities that explain them. I write $\\tau=Y_1-Y_0$, $f(\\mathbf{x})=\\Pr(\\mathbf{X}=\\mathbf{x})$, $p(\\mathbf{x})=\\Pr(D_1>D_0|\\mathbf{X}=\\mathbf{x})$ and $\\mu(\\mathbf{x})=\\mathrm{E}(\\tau|D_1>D_0,\\mathbf{X}=\\mathbf{x})$. Under the IV assumptions, with a discrete covariate and correct scores, the limits of 2SLS and of ICSW are\n\\[\n\\theta_{\\mathrm{IV}}=\\frac{\\sum_{\\mathbf{x}}f(\\mathbf{x})p(\\mathbf{x})\\mu(\\mathbf{x})}{\\Pr(D_1>D_0)},\\qquad\n\\theta_{\\mathrm{ICSW}}=\\sum_{\\mathbf{x}}f(\\mathbf{x})\\mu(\\mathbf{x}),\\qquad\n\\theta_{\\mathrm{IV}}-\\theta_{\\mathrm{ICSW}}=\\frac{\\operatorname{Cov}\\{p(\\mathbf{X}),\\mu(\\mathbf{X})\\}}{\\Pr(D_1>D_0)} .\n\\]\nAssumption 1 implies that $\\theta_{\\mathrm{ICSW}}$ equals the ATE, $\\sum_{\\mathbf{x}}f(\\mathbf{x})\\mathrm{E}(\\tau|\\mathbf{X}=\\mathbf{x})$.\n\nBoth populations have a binary covariate $X$, a complier indicator $C$, an effect $\\tau\\in\\{0,1\\}$, outcomes $Y_0=0$ and $Y_1=\\tau$, and $Z\\sim\\mathrm{Bernoulli}(1/2)$ independent of everything else. A unit that is not a complier is an always-taker or a never-taker with probability $1/2$ each, independently. In population A, $C$ and $\\tau$ are independent $\\mathrm{Bernoulli}(1/2)$ variables, and $\\Pr(X=1|C,\\tau)$ is $3/4$ if $C=\\tau=1$ and $1/4$ otherwise. In population B, $X\\sim\\mathrm{Bernoulli}(1/2)$ and, given $X$, the variables $C$ and $\\tau$ are independent with $\\Pr(C=1|X)=\\Pr(\\tau=1|X)$ equal to $1/4$ at $X=0$ and $3/4$ at $X=1$.\n\\begin{audittable}{lrrrr}{The two populations by covariate value.}\nPopulation and stratum & $f(x)$ & $p(x)$ & $\\mathrm{E}(\\tau|X=x)$ & $\\mu(x)$ \\\\\n\\hline\nA, $X=0$ & $5/8$ & $2/5$ & $2/5$ & $1/4$ \\\\\nA, $X=1$ & $3/8$ & $2/3$ & $2/3$ & $3/4$ \\\\\nB, $X=0$ & $1/2$ & $1/4$ & $1/4$ & $1/4$ \\\\\nB, $X=1$ & $1/2$ & $3/4$ & $3/4$ & $3/4$ \\\\\n\\end{audittable}\n\\begin{audittable}{lrr}{The sentence of p.~499 in the two populations.}\nQuantity & Population A & Population B \\\\\n\\hline\n$\\tau$ independent of compliance type & yes & no \\\\\n$\\tau$ independent of compliance type given $X$ & no & yes \\\\\nAssumption 1 & fails & holds \\\\\nATE & $1/2$ & $1/2$ \\\\\nLimit of 2SLS (the LATE) & $1/2$ & $5/8$ \\\\\nLimit of ICSW & $7/16$ & $1/2$ \\\\\n\\end{audittable}\nIn population A, I have $\\theta_{\\mathrm{ICSW}}=(5/8)(1/4)+(3/8)(3/4)=7/16$. In population B, I have $\\theta_{\\mathrm{IV}}=(5/16)/(1/2)=5/8$. Assumption 3 holds with $\\epsilon=1/3$ in A and $\\epsilon=1/8$ in B, and the saturated compliance model of equations (5)--(7) fits both populations, so that the scores are correct and bounded. Solving the IV moment equations in exact arithmetic, with and without $X$ as a regressor, I find the same four limits.", "problem": "Assumption 1 is a condition given the covariates: the mean effect among compliers with profile $\\mathbf{x}$ equals the mean effect among all units with profile $\\mathbf{x}$. Independence of the effects from compliance, without qualification, is independence in the whole population. It implies that the LATE equals the ATE, and 2SLS is then consistent for the ATE. It does not give the conditional equality, because effects and compliance that are independent overall can be dependent among the units that share a covariate value. ICSW with correct scores then converges to the average over $\\mathbf{x}$ of the complier effects, which is not the ATE in general. If the independence is read as conditional on the covariates, Assumption 1 holds and ICSW is consistent, but 2SLS converges to the LATE, which then differs from the ATE in general. Neither of these two readings makes the whole sentence true. The sentence suggests that ICSW cannot hurt when 2SLS already recovers the ATE. The first population below shows that weighting on covariates can introduce a bias.", "what_survives": "The consistency result in the appendix, which applies Assumption 1 between equations (10) and (11), is unaffected by this finding, and so are the preceding sentences on a constant effect, the estimator and every reported number. The sufficient condition is withdrawn. The repair adds a restriction on the effects among compliers, which the applications may not satisfy. Whether they satisfy it is not established here." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E004", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of Assumption 2 with its five lines of text. The text reads: Consistency of P-hat C i for P C i. Assumption 2 amounts to requiring proper specification of the ML estimator. Application of a flexible parametrization of the covariates in X, as is possible with large samples, will necessarily yield consistency, so long as the rate of growth in the number of estimated parameters is sufficiently slow in n. In the case of a saturated model and F(.) as the normal CDF, P-hat C i will be unbiased for the conditional probability of compliance. A red outline surrounds the word unbiased in the last line.", "caption": "Assumption 2, p. 500 (PDF p. 9). The outline marks “unbiased” in the last sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E005.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T13", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T13" }, { "id": "P029-T14", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T14" } ], "headline": "The maximum-likelihood compliance score of a saturated model is consistent but not unbiased: within a covariate stratum it is the difference between the treatment-receipt rates of the two instrument arms, truncated at zero, and under two-sided noncompliance its expectation exceeds the compliance score in every finite sample with both arms nonempty.", "id": "P029-E005", "location": "Assumption 2, last sentence, p. 500 (PDF p. 9); the estimator is defined in Section 3.1, equations (5)–(7), pp. 496–497 (PDF pp. 5–6)", "pdf_page": 9, "printed_page": "500", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T13", "P029-T14" ], "text": { "as_printed": "In the case of a saturated model and $F(.)$ as the normal CDF, $\\hat{P}_{Ci}$ will be \\hl{unbiased} for the conditional probability of compliance.", "corrected": "In the case of a saturated model, with a fixed finite set of covariate strata and the number of observations in both instrument arms of every stratum tending to infinity, $\\hat{P}_{Ci}$, allowing boundary fitted probabilities, will be \\hl{consistent} for the conditional probability of compliance. For a unit in stratum $x$ with \\hl{both arms nonempty}, it equals $\\max(\\bar D_{1x} - \\bar D_{0x}, 0)$, the difference between the treatment-receipt rates of the two instrument arms in that stratum truncated below at zero, and it \\hl{need not be unbiased} in finite samples. Under one-sided noncompliance it equals the receipt rate $\\bar D_{1x}$ of the arm $Z = 1$ and is unbiased when that arm is nonempty. These statements hold for any $F$ that maps onto $(0,1)$.", "evidence": "The evidence is a derivation of the estimator and an exact counterexample. For the derivation, I fix a stratum with $n_1 > 0$ units in the arm $Z = 1$ and $n_0 > 0$ units in the arm $Z = 0$, of which $k_1$ and $k_0$ receive treatment. I write $p = P_{A,C}$ and $q = P_{A,C} P_{A|A,C}$ for the two receipt probabilities of the model. Up to a constant, the likelihood (7) of the stratum is\n\\[\nL(p, q) = p^{k_1} (1-p)^{n_1 - k_1} \\, q^{k_0} (1-q)^{n_0 - k_0}, \\qquad 0 \\le q \\le p \\le 1 ,\n\\]\nand the fitted score is $p - q$ at the maximum. If $k_0/n_0 \\le k_1/n_1$, then the unconstrained maximizer $(k_1/n_1, k_0/n_0)$ satisfies the constraint. If $k_0/n_0 > k_1/n_1$, then $k_0 > 0$ and $k_1 < n_1$, so $L$ vanishes on the edges $q = 0$ and $p = 1$. Further, $L$ has no stationary point such that $0 < q < p < 1$, since such a point would be $(k_1/n_1, k_0/n_0)$. The maximum then lies on the diagonal $q = p = t$, where $L(t,t) = t^{k_1 + k_0}(1-t)^{n_1 + n_0 - k_1 - k_0}$ is largest at $t = (k_1 + k_0)/(n_1 + n_0)$. Combining the two cases, I obtain the formula above. A probit model with finite coefficients has $0 < q < p < 1$, and a sample whose maximizer lies on the boundary has a supremum that is not attained. The formula takes the maximum over the closure.\n\nFor the counterexample, I take a single stratum in which a unit is an always-taker with probability $1/4$, a complier with probability $1/2$ and a never-taker with probability $1/4$, with no defiers. One of two independently sampled units is assigned to each arm at random. The receipt probabilities are $p_1 = 3/4$ in the arm $Z = 1$ and $p_0 = 1/4$ in the arm $Z = 0$, and the compliance score is $1/2$. The model is correctly specified, with $P_{A,C} = 3/4$ and $P_{A|A,C} = 1/3$.\n\\begin{audittable}{llll}{The four possible samples with one unit in each arm. Here $d_1$ and $d_0$ are the receipts in the arms $Z = 1$ and $Z = 0$.}\nReceipts $(d_1, d_0)$ & Probability & Fitted $(p, q)$ & Fitted score \\\\\n$(0, 0)$ & $3/16$ & $(0, 0)$ & $0$ \\\\\n$(0, 1)$ & $1/16$ & $(1/2, 1/2)$ & $0$ \\\\\n$(1, 0)$ & $9/16$ & $(1, 0)$ & $1$ \\\\\n$(1, 1)$ & $3/16$ & $(1, 1)$ & $0$ \\\\\n\\end{audittable}\nThe expectation of the fitted score is $9/16$, and the bias is $1/16$. The untruncated difference $d_1 - d_0$ has expectation $3/4 - 1/4 = 1/2$. The estimator replaces its value $-1$ in the second row by $0$.\n\nThe bias is not special to one unit per arm. Let $T = \\bar D_{1x} - \\bar D_{0x}$, and let $c$ be the compliance score. Since $\\max(T, 0) = T + \\max(-T, 0)$ and $\\mathbb{E}[T] = c$, I have\n\\[\n\\mathbb{E}\\bigl[\\hat P_{Ci}\\bigr] - c = \\mathbb{E}\\bigl[\\max(-T, 0)\\bigr] \\ge 0 ,\n\\]\nand the inequality is strict for all arm sizes whenever $p_0 > 0$ and $p_1 < 1$. With five units in each arm, $p_1 = 1/2$ and $p_0 = 3/10$, I enumerate the 36 pairs of counts and find the expectation $15659/64000 \\approx 0.245$, against a compliance score of $1/5$.\n\nConsistency holds. For $c \\ge 0$ and every real $t$, I have $|\\max(t, 0) - c| \\le |t - c|$. Chebyshev's inequality then implies that, for every $\\delta > 0$,\n\\[\n\\Pr\\bigl(|\\hat P_{Ci} - c| > \\delta\\bigr) \\le \\frac{1}{\\delta^2}\\left(\\frac{p_1(1-p_1)}{n_1} + \\frac{p_0(1-p_0)}{n_0}\\right) ,\n\\]\nand the right side tends to zero as $n_1$ and $n_0$ tend to infinity. Under one-sided noncompliance, $p_0 = 0$, so $k_0 = 0$ and $\\hat P_{Ci} = \\bar D_{1x}$, which is unbiased when the arm $Z = 1$ is not empty.", "problem": "The estimator $\\hat P_{Ci}$ is the maximum-likelihood estimator of Section~3.1. By equations (5)--(7), the model sets the probability of receiving treatment to $P_{A,C,i}$ for a unit with $Z_i = 1$ and to $P_{A,C,i} P_{A|A,C,i}$ for a unit with $Z_i = 0$. The compliance score is the difference of the two. Since the second probability is at most the first, the model cannot fit a covariate stratum in which the arm $Z = 0$ receives treatment at a higher rate than the arm $Z = 1$. In a saturated model with both arms nonempty in stratum $x$, let $\\bar D_{1x}$ and $\\bar D_{0x}$ be the treatment-receipt rates in stratum $x$, that is, the shares of units that receive treatment in the arm $Z = 1$ and in the arm $Z = 0$. If $\\bar D_{0x} \\le \\bar D_{1x}$, then the fitted probabilities are these two rates, and the fitted score is $\\bar D_{1x} - \\bar D_{0x}$. If $\\bar D_{0x} > \\bar D_{1x}$, then both fitted probabilities equal the pooled rate, and the fitted score is zero. Hence\n\\[\n\\hat P_{Ci} = \\max\\bigl(\\bar D_{1x} - \\bar D_{0x},\\, 0\\bigr) \\qquad \\text{for every unit } i \\text{ in stratum } x .\n\\]\nThe difference $\\bar D_{1x} - \\bar D_{0x}$ is unbiased for the compliance score, and the truncation replaces its negative values by zero. The expectation of $\\hat P_{Ci}$ is therefore larger than the compliance score whenever a negative difference has positive probability. This is the case in every finite sample when the stratum contains always-takers and never-takers. The printed sentence is true under one-sided noncompliance, where the arm $Z = 0$ never receives treatment and the truncation cannot bind. Note that the normal CDF plays no role: the formula holds for any $F$ that maps onto $(0,1)$.", "what_survives": "Assumption~2 requires that $\\hat P_{Ci}$ be consistent, not that it be unbiased, and the consistency argument of the Appendix uses the assumption in that form. The estimands, the ICSW estimator and its identification argument do not depend on the sentence. The two applications and the simulations fit parametric models and not saturated ones, and none of their calculations uses the sentence. I did not recompute them for this finding. The phrase ``strata means under saturation'' on p.~496 needs the same qualification, since the two fitted probabilities are the separate arm means only when $\\bar D_{0x} \\le \\bar D_{1x}$. What changes is the claim that the score is unbiased, which becomes a claim that it is consistent." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E005", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of footnote 6, seven lines. A red box marks the words 'because c tends to 0 as n tends to infinity' in the sentence 'Asymptotically, assuming that alpha remains fixed, the Winsorization has no impact on estimation'. Earlier in the footnote c is defined as the 1 over n to the alpha quantile of the estimated compliance scores.", "caption": "Footnote 6, p. 498 (PDF p. 7). The marked clause gives the reason for the asymptotic irrelevance of Winsorization.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E006.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T15", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T15" }, { "id": "P029-T16", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T16" } ], "headline": "The Winsorization cut-off $c$ need not tend to zero; the fraction of scores changed tends to zero, and under suitable regularity conditions Winsorization leaves the probability limit unchanged.", "id": "P029-E006", "location": "Footnote 6, p. 498 (PDF p. 7), sentence beginning “Asymptotically, assuming that $\\alpha$ remains fixed”", "pdf_page": 7, "printed_page": "498", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T15", "P029-T16" ], "text": { "as_printed": "Define $c$ as the $1/n^{\\alpha}$th quantile of $\\hat{P}_{Ci}$, where $\\alpha$ is a positive calibrating constant. We replace all $\\hat{P}_{Ci}0$, the fraction of observations whose scores are changed tends to zero. Under Assumption 3 and regularity conditions ensuring bounded estimated inverse scores with probability tending to one and integrable outcomes (for example, uniform score consistency and $\\mathrm{E}|Y|<\\infty$), Winsorization does not change the probability limit of the displayed ICSW estimator. The cutoff $c$ need not tend to zero. This consistency argument does not establish first-order distributional equivalence.}\n\nThe rest of the footnote is unchanged.", "evidence": "The evidence is an exact counterexample to the limit and a derivation of the corrected statement.\n\n\\emph{Counterexample.} Let the covariate be constant. A unit is an always-taker with probability $1/4$, a complier with probability $1/2$ and a never-taker with probability $1/4$, the assignment $Z\\sim\\mathrm{Bernoulli}(1/2)$ is independent of the type, and $Y_0=0$, $Y_1=1$. The IV assumptions hold, Assumption 1 holds because the effect is $1$ for every unit, and Assumption 3 holds with $\\epsilon=1/4$ because the compliance score is $1/2$. With the exact scores, every quantile of the scores is $1/2$, and $c=1/2$ for every $n$ and every $\\alpha$. With the ML estimator of equations (5)--(7), the fitted score is the same for all units and eventually equals the difference between the proportions treated in the two arms, once those proportions are interior and correctly ordered. By the strong law, $c$ converges almost surely to $3/4-1/4=1/2$. No score is below $c$, the weights are unchanged, and the ICSW ratio is $1$, the ATE, whenever its empirical first stage is nonzero. Note that the example contradicts the reason given and not the conclusion.\n\n\\emph{Derivation.} Let $p_n=n^{-\\alpha}$, let $c_n$ be the cut-off, and let $m_n$ be the number of scores strictly below $c_n$. For the quantile rule that the code uses, I have $m_n/n\\le p_n+1/n$. Suppose that $\\max_{i\\le n}|\\hat{P}_{Ci}-P_{Ci}|\\to0$ in probability and that $\\mathrm{E}|Y|<\\infty$. By Assumption 3 and this convergence, every estimated score exceeds $\\epsilon/2$ with probability tending to one. On that event the weights $w_i=1/\\hat{P}_{Ci}$ and the Winsorized weights $w_i^*=1/\\max(\\hat{P}_{Ci},c_n)$ satisfy $0\\le w_i-w_i^*\\le2/\\epsilon$, and they differ for at most $m_n$ units. For any variables with $|V_i|\\le K$, I obtain\n\\[\n\\Bigl|\\frac1n\\sum_{i=1}^n(w_i-w_i^*)V_i\\Bigr|\\le\\frac{2K}{\\epsilon}\\Bigl(p_n+\\frac1n\\Bigr)\\to0 .\n\\]\nThis bound covers the weighted sums of $\\mathbf{1}\\{Z_i=z\\}$ and $\\mathbf{1}\\{Z_i=z\\}D_i$. For the weighted sums of the outcome, truncating $Y_i$ at a level $M$ and using $\\mathrm{E}|Y|<\\infty$, I find that the change tends to zero in probability. Since the weighted difference in treatment rates has the positive limit $1/\\mathrm{E}\\{1/P_C(\\mathbf{X})\\}$, I conclude that the Winsorized and original estimators have the same probability limit. The bound is of order $n^{-\\alpha}$, which for $\\alpha=0.275$ is not $o(n^{-1/2})$. The argument does not show that the two estimators have the same first-order distribution, and it does not show that they differ.", "problem": "The cut-off $c$ is a quantile of the estimated compliance scores, taken at the level $n^{-\\alpha}$. As $n$ grows the level tends to zero, but the quantile need not, since Assumption 3 keeps the true scores above some $\\epsilon>0$. The sentence's conclusion holds for a different reason. The fraction of scores that Winsorization changes is at most $n^{-\\alpha}+1/n$, and the weights that it changes stay bounded under conditions of the kind that the article already invokes.", "what_survives": "The Winsorization rule, the code, which computes a quantile, and every estimate are unchanged. The appendix does not use the limit $c\\to0$, and consistency holds under the conditions stated below. No application estimate and no simulation result is shown to be wrong. Only the reason given in the footnote changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E006", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of Assumption 3 with its eight lines of text. The assumption reads: For all x in Supp(X), Pr(D1 greater than D0 given X = x) lies in the interval from epsilon, excluded, to 1, included, where epsilon is greater than 0. The text continues: Note that IV estimation already requires the assumption of nonzero compliance in the full population, and we generalize this assumption to every covariate profile. This assumption has both theoretical and practical implications. If Assumption 3 fails, then the ATE cannot generally be estimated using the variation induced by Z: the instrument simply does not cause any variation in the treatment received by some types of units. In practice, if a unit has zero probability of compliance, the weighting procedure will produce infinite weights asymptotically, leading to an undefined estimate. Conditional on the validity of Assumptions 1 and 2, Assumption 3 is therefore verifiable in large samples. Red outlines surround the sentence from If Assumption 3 fails to some types of units, which runs over three lines.", "caption": "Assumption 3 and its explanation, p. 500 (PDF p. 9). The outlines mark the sentence that describes the failure of the assumption.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E007.png" } ], "formal_release": null, "formal_statements": [], "headline": "Assumption 3 can fail although every covariate profile has compliers; the explanation that follows it describes only the case of a profile with no compliers.", "id": "P029-E007", "location": "Assumption 3 and the explanation that follows it, p. 500 (PDF p. 9)", "pdf_page": 9, "printed_page": "500", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "For all $\\mathbf{x} \\in \\operatorname{Supp}(\\mathbf{X})$, $\\Pr(D_1 > D_0 \\mid \\mathbf{X} = \\mathbf{x}) \\in (\\epsilon, 1]$, where $\\epsilon > 0$. Note that IV estimation already requires the assumption of nonzero compliance in the full population, and we generalize this assumption to every covariate profile. This assumption has both theoretical and practical implications. \\hl{If Assumption 3 fails}, then the ATE cannot generally be estimated using the variation induced by $Z$: \\hl{the instrument simply does not cause any variation in the treatment received by some types of units}. In practice, if a unit has zero probability of compliance, the weighting procedure will produce infinite weights asymptotically, leading to an undefined estimate.", "corrected": "\\hl{If compliance is zero for a set of covariate profiles with positive population probability}, the ATE cannot generally be \\hl{identified} using the variation induced by $Z$. \\hl{Assumption 3 is stronger than this pointwise overlap requirement: a uniformly positive lower bound also rules out arbitrarily large inverse compliance weights.}", "intended_meaning": "The intended point is that a covariate profile without compliers cannot be reweighted: the instrument moves no unit's treatment there, and the inverse of the compliance score is infinite. This is correct, and ``cannot generally'' does not claim an impossibility for every law that violates the assumption. Read literally, however, the sentence equates the failure of Assumption~3 with the existence of such a profile. The assumption asks for more, namely a lower bound $\\epsilon$ that holds for all profiles at once, and the failure of a uniform bound need not imply a profile with zero compliance.\n\nFor an example, let $\\Pr(X = k) = 2^{-k}$ for $k = 1, 2, \\ldots$, and let the compliance score at $X = k$ be $1/k$. Every profile has compliers, and no $\\epsilon > 0$ bounds the scores from below. Suppose further that there are no always-takers or defiers, that $Z$ is assigned independently as a fair coin, that the outcomes are bounded and that the treatment effect is $1$ for every unit. The weights $w = k$ formed from the true scores satisfy $\\mathbb{E}[w] = 2$ and $\\mathbb{E}[w^2] = 6$, and the weighted IV ratio is consistent for the ATE, which is $1$. The example concerns the true scores and says nothing about estimated scores. Assumption~3 as stated, and every result that uses it, is unchanged." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E007", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of footnote 3. A red box marks the plus sign between '(F(theta A,C x sub i) Z sub i) to the power D sub i' and '(1 minus F(theta A,C x sub i) Z sub i) to the power 1 minus D sub i' in the displayed likelihood. The next sentence calls it the standard likelihood function for binomial regression.", "caption": "Footnote 3, p. 497 (PDF p. 6). The marked plus sign joins the two Bernoulli factors of the one-sided likelihood.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E008.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T17", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T17" } ], "headline": "In the one-sided likelihood of footnote 3 the two Bernoulli factors are multiplied, not added.", "id": "P029-E008", "location": "Footnote 3, p. 497 (PDF p. 6), displayed likelihood", "pdf_page": 6, "printed_page": "497", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T17" ], "text": { "as_printed": "\\[\nL(\\theta_{\\mathbf{C}}|D,Z) = \\Pi_{i=1}^{n}\\bigl[(F(\\hl{\\theta_{\\mathbf{A,C}}}\\mathbf{x_i})Z_i)^{D_i} \\;\\hl{+}\\; (1 - F(\\hl{\\theta_{\\mathbf{A,C}}}\\mathbf{x_i})Z_i)^{1-D_i}\\bigr].\n\\]\nThis likelihood function is a familiar one: it is the standard likelihood function for binomial regression, if only applied to units in the treatment condition.", "corrected": "\\[\nL(\\theta_{\\mathbf{C}}|D,Z) = \\Pi_{i=1}^{n}\\bigl[(F(\\hl{\\theta_{\\mathbf{C}}}\\mathbf{x_i})Z_i)^{D_i} \\;\\hl{\\times}\\; (1 - F(\\hl{\\theta_{\\mathbf{C}}}\\mathbf{x_i})Z_i)^{1-D_i}\\bigr].\n\\]", "evidence": "The evidence is a derivation. Setting $P_{A|A,C,i}=0$ in equation (7), I obtain the likelihood for unit $i$,\n\\[\n(P_{A,C,i}Z_i)^{D_i}(1-P_{A,C,i}Z_i)^{1-D_i}.\n\\] For an assigned unit with $F(\\cdot)=3/4$, the printed factor is $5/4$ when $D_i=0$ and $7/4$ when $D_i=1$, in place of $1/4$ and $3/4$. The one-sided likelihood in the replication code is the product.", "problem": "With a plus sign every factor of the product is at least $1$, and the display is not a likelihood. The reduction of equation (7) and the likelihood for binomial regression, which the next sentence names, both have a product. The parameter is also written $\\theta_{\\mathbf{C}}$ on the left and $\\theta_{\\mathbf{A,C}}$ on the right. With no always-takers these index the same probability, and one subscript is meant.", "what_survives": "Equation (7), the definition of the ML estimate, the sentence on binomial regression and every computed result are unaffected." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E008", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of equation (6) and the five lines after it. A red box marks 'P sub C,i = Pr(D sub 1i greater than D sub 0i)' in the sentence 'Therefore, we may define the compliance score as'.", "caption": "Section 3.1, p. 496 (PDF p. 5), after equation (6). The marked definition of the compliance score has no conditioning on the covariates.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009.png" }, { "alt": "Crop of a two-line display and the three lines of text after it. A red box marks the left side 'Pr(D = 1 given X = x sub i)'. The right side is 'Pr(D1 greater than D0 given X = x sub i) times Z sub i plus Pr(D0 = 1 given X = x sub i)'.", "caption": "Section 3.1, p. 497 (PDF p. 6), first display. The marked left side conditions on the covariates only, while the right side depends on $Z_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-b.png" }, { "alt": "Crop of equation (9) for the weighted ITT with one line of text above and one below. A red box marks the vertical bar between 'D1 greater than D0' and 'X = x' inside the expectation of Y1 minus Y0 in the numerator.", "caption": "Appendix, p. 504 (PDF p. 13), equation (9). The marked bar is the second conditioning bar in the expectation.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-c.png" }, { "alt": "Crop of one line of text, ending 'a consistent estimator of the weighted ratio will be', and of the ratio estimator written as four weighted sums over four weighted sums. A red box marks the weight 'w sub Ci' with a double hat in the second sum of the denominator; every other weight has one hat.", "caption": "Appendix, p. 504 (PDF p. 13), display after equation (14). The marked weight carries two hats.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-d.png" }, { "alt": "Crop of six displayed lines defining X1, X2, X3, the coefficients a to h, AC sub i, A sub i, C sub i and D sub i. Red boxes mark the closing bracket of the indicator in the line for AC sub i, the closing bracket of the indicator in the line for A sub i, and the first factor 'D sub i' on the right of 'D sub i = D sub i Z sub i + A sub i'.", "caption": "Supplement, p. A1 (PDF p. 1), baseline data generating process. The marked brackets close indicators that have no test $>0$, and the marked $D_i$ stands where $C_i$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-f.png" }, { "alt": "Crop of the heading 'Heterogeneous Treatment Effects, Model is Incorrect', the line 'Same as the baseline DGP, except:' and the two lines defining AC sub i and A sub i with a Logistic(0,3) term. Red boxes mark the closing bracket of each indicator.", "caption": "Supplement, p. A2 (PDF p. 2), design “Heterogeneous Treatment Effects, Model is Incorrect”. The marked brackets close indicators that have no test $>0$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E009-g.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T18", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T18" }, { "id": "P029-T19", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T19" }, { "id": "P029-T26", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T26" } ], "headline": "Six misprints of notation: the compliance score and the receipt probability each lack a conditioning variable, equation (9) has a second conditioning bar, one weight has two hats, and the simulation recipe prints $D_i$ for $C_i$ and omits the test $>0$ in its indicators.", "id": "P029-E009", "location": "Section 3.1, pp. 496–497 (PDF pp. 5–6); Appendix, p. 504 (PDF p. 13), equation (9) and the display after equation (14); Supplement, pp. A1–A2 (PDF pp. 1–2)", "pdf_page": 5, "printed_page": "496–497", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T18", "P029-T19", "P029-T26" ], "text": { "as_printed": "\\begin{enumerate}\n\\item p.~496: ``Therefore, we may define the compliance score as $P_{C,i} = \\hl{\\Pr(D_{1i} > D_{0i})}$.''\n\\item p.~497: $\\hl{\\Pr(D=1|\\mathbf{X}=\\mathbf{x}_i)} = \\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x}_i)Z_i + \\Pr(D_0=1|\\mathbf{X}=\\mathbf{x}_i) = P_{A,C,i}(1-P_{A|A,C,i})Z_i + P_{A,C,i}P_{A|A,C,i}$.\n\\item p.~504, equation (9):\n\\[\n\\mathrm{ITT}^w = \\frac{1}{\\overline{w_c}}\\int_\\Omega \\frac{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})\\,\\mathrm{E}(Y_1 - Y_0|D_1 > D_0\\,\\hl{|}\\,\\mathbf{X}=\\mathbf{x})}{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})}f(\\mathbf{x})d\\mathbf{x}.\n\\]\n\\item p.~504, display after equation (14), first ratio of the denominator:\n\\[\n\\Bigl(\\sum_{i=1}^{n}\\hat{w}_{Ci}Z_iD_i\\Bigr)\\Big/\\Bigl(\\sum_{i=1}^{n}\\hl{\\hat{\\hat{w}}_{Ci}}Z_i\\Bigr).\n\\]\n\\item Supplement, p.~A1: $AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\mathbf{N}(0,3)\\hl{]}$, $A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\mathbf{N}(0,3)\\hl{]}AC_i$, $C_i = AC_i - A_i$, $D_i = \\hl{D_i}Z_i + A_i$.\n\\item Supplement, p.~A2: $AC_i = \\mathbf{I}[2 + a + (1 + bX1_i)(1 + cX2_i) + dX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{]}$, $A_i = \\mathbf{I}[-2 + e + (1 + fX1_i)(1 + gX2_i) + hX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{]}AC_i$.\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item p.~496: $P_{C,i} = \\Pr(D_{1i} > D_{0i}\\hl{\\,|\\,\\mathbf{X}=\\mathbf{x}_i})$.\n\\item p.~497: $\\Pr(\\hl{D_i}=1|\\mathbf{X}=\\mathbf{x}_i\\hl{,Z_i}) = \\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x}_i)Z_i + \\Pr(D_0=1|\\mathbf{X}=\\mathbf{x}_i) = P_{A,C,i}(1-P_{A|A,C,i})Z_i + P_{A,C,i}P_{A|A,C,i}$.\n\\item p.~504, equation (9):\n\\[\n\\mathrm{ITT}^w = \\frac{1}{\\overline{w_c}}\\int_\\Omega \\frac{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})\\,\\mathrm{E}(Y_1 - Y_0|D_1 > D_0\\hl{,}\\,\\mathbf{X}=\\mathbf{x})}{\\Pr(D_1 > D_0|\\mathbf{X}=\\mathbf{x})}f(\\mathbf{x})d\\mathbf{x}.\n\\]\n\\item p.~504, display after equation (14), first ratio of the denominator:\n\\[\n\\Bigl(\\sum_{i=1}^{n}\\hat{w}_{Ci}Z_iD_i\\Bigr)\\Big/\\Bigl(\\sum_{i=1}^{n}\\hl{\\hat{w}_{Ci}}Z_i\\Bigr).\n\\]\n\\item Supplement, p.~A1: $AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\mathbf{N}(0,3)\\hl{>0}]$, $A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\mathbf{N}(0,3)\\hl{>0}]AC_i$, $C_i = AC_i - A_i$, $D_i = \\hl{C_i}Z_i + A_i$.\n\\item Supplement, p.~A2: $AC_i = \\mathbf{I}[2 + a + (1 + bX1_i)(1 + cX2_i) + dX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{>0}]$, $A_i = \\mathbf{I}[-2 + e + (1 + fX1_i)(1 + gX2_i) + hX3_i^3/3 + \\mathrm{Logistic}(0,3)\\hl{>0}]AC_i$.\n\\end{enumerate}\nThe supplement may also state that assignment is balanced, with $n/2$ units in each arm, as in the code, and the step before equation (11) may be attributed to the law of total expectation. Neither omission is an error.", "evidence": "The evidence is a check of the pages against the neighbouring formulas and the simulation code, with two short calculations. With no always-takers, compliance probability $1/2$ and $Z\\sim\\mathrm{Bernoulli}(1/2)$, I have $\\Pr(D=1|\\mathbf{X})=1/4$, while the printed right side is $0$ for $Z_i=0$ and $1/2$ for $Z_i=1$. For an always-taker assigned to treatment, the identity printed on p.~A1 reads $D_i=D_i+1$. All six simulation scripts in the replication files test whether the latent index is positive and set \\texttt{D <- C*Z + A}.", "problem": "Each item is a slip of notation, and the neighbouring formulas and the simulation code give the intended meaning. The compliance score is conditional on the covariates, since p.~496 first defines $P_{Ci}=\\Pr(D_1>D_0|\\mathbf{X}=\\mathbf{x}_i)$ and p.~497 uses that definition again. In the first display on p.~497 the right side depends on $Z_i$, so that the left side must also condition on $Z_i$. In equation (9) the second bar stands for a comma, as in equations (8) and (10). The weight with two hats is $\\hat{w}_{Ci}$, as in Section 3.3 and in the display at the top of p.~505. In the supplement each indicator is the indicator that the latent index is positive, and the treatment received is $C_iZ_i+A_i$. As printed, $D_i$ is defined in terms of itself.", "what_survives": "The definitions used elsewhere, the argument in the appendix and the simulations are unaffected. Two further points about these passages are not errors. The receipt probability lies strictly in $(0,1)$ under the probit specification of equations (5) and (6). The step before equation (11) is attributed to the law of total probability. It is a correct step, by the law of total expectation." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E009", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of three rows of Table 1, each with a coefficient and an interval in two columns. Age missing: 12.7 with interval (-16.5, 44.1), and 17.7 with interval (-15.9, 50.3). Democrat: 2.6 with interval (0.6, 4.7), and 2.6 with interval (0.7, 04.8). Republican: 3.8 with interval (1.3, 6.2), and 3.2 with interval (0.8, 5.8). A red outline surrounds the upper limit 04.8 of the Democrat interval in the second column.", "caption": "Table 1, rows “Age missing”, “Democrat” and “Republican”, p. 501 (PDF p. 10). The two columns of numbers are LATE (2SLS), (1), and ATE (ICSW), (2). The outline marks the upper limit “04.8” in column (2).", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E010.png" }, { "alt": "Crop of the supplement section headed (3) Bootstrap Confidence Interval Coverage. The text reads: In Table A6, we present percent-sign coverage with the bootstrap for n in 500, 2500, 5000 assuming that the model is correct and alpha = 0.275. Due to computational limitations, we use only 500 bootstrap iterations and 500 simulations for each n. Below it, Table A6, Bootstrap Coverage, has the columns n, 90 percent Interval Coverage and 95 percent Interval Coverage, with the rows 500, 0.94, 0.98; 2500, 0.91, 0.94; 5000, 0.92, 0.95. A red outline surrounds the percent sign and the word coverage in the first line.", "caption": "Supplement, p. A9 (supplement PDF p. 9): the sentence that introduces Table A6, and the table. The outline marks “% coverage”; the entries of the table are proportions.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E010-A6.png" } ], "formal_release": null, "formal_statements": [], "headline": "Three labels do not match what they label: the Democrat interval in column (2) of Table 1 ends at 4.8, printed “04.8”; Table A6 reports coverage as proportions and not as percentages; and the simulation scripts carry the comment “ATE is always 1”, while the average effect in the simulated population is 5.", "id": "P029-E010", "location": "Table 1, row “Democrat”, column (2), p. 501 (PDF p. 10); Section 4.2, p. 502 (PDF p. 11), with supplement Table A8, p. A12 (supplement PDF p. 12); supplement p. A9 (supplement PDF p. 9), the sentence that introduces Table A6; the simulation scripts of the replication files", "pdf_page": 10, "printed_page": "501", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Table~1, row ``Democrat'', p.~501.} Column (1): 2.6 $(0.6, 4.7)$. Column (2): 2.6 $(0.7, \\hl{0}4.8)$.\n\n\\emph{Section~4.2, p.~502.} The compliance score most strongly correlates with political interest, newspaper reading, and education (see the supplementary appendix for a full correlation matrix).\n\n\\emph{Supplement, p.~A9.} In Table A6, we present \\hl{\\% coverage} with the bootstrap for $n \\in 500, 2500, 5000$ assuming that the model is correct and $\\alpha = 0.275$.\n\\begin{audittable}{rrr}{Supplement Table~A6: Bootstrap Coverage.}\n$n$ & 90\\% Interval Coverage & 95\\% Interval Coverage \\\\\n500 & 0.94 & 0.98 \\\\\n2500 & 0.91 & 0.94 \\\\\n5000 & 0.92 & 0.95 \\\\\n\\end{audittable}\n\\emph{Simulation scripts.} \\texttt{\\# ATE is always \\hl{1}}", "corrected": "\\emph{Table~1, row ``Democrat'', column (2).} 2.6 $(0.7, \\hl{4.8})$.\n\n\\emph{Supplement, p.~A9.} In Table A6, we present \\hl{coverage proportions} with the bootstrap for $n \\in 500, 2500, 5000$ assuming that the model is correct and $\\alpha = 0.275$.\n\n\\emph{Simulation scripts.} \\texttt{\\# Superpopulation ATE is \\hl{5}; heterogeneous sample-average effects vary, while the homogeneous sample effect is 5.}\n\n\\emph{Optional clarification of Section~4.2.} The \\hl{Winsorized} compliance score correlates most strongly with political interest and newspaper reading, \\hl{followed by party identification and education} (see the supplementary appendix for a full correlation matrix). Alternatively, the printed sentence may be kept and labelled as referring to the score before Winsorization.", "evidence": "The evidence is a check of the pages and two recalculations. Integrating each of the two effect functions over the three covariates, I obtain 5. From the replication data, the correlations of the score before Winsorization are 0.71 with political interest, 0.70 with newspaper reading, 0.40 with education and 0.32 with party ID. After Winsorization with $\\alpha = 0.275$ they are 0.65, 0.62, 0.36 and 0.37, which are the entries of Table~A8. The comment is at line 44 of \\texttt{Simulations Alpha 6-20.R} and of the three \\texttt{Simulations Misspecification} scripts named \\texttt{Right}, \\texttt{Wrong Model} and \\texttt{Only X1}, and at line 45 of \\texttt{Simulations Misspecification Noise 6-20.R}.", "problem": "In Table~1 the upper limit of the Democrat interval in column (2) is printed with a leading zero. The supplement announces ``\\% coverage'', and the entries of Table~A6 are proportions: 0.94 stands for 94\\%. The five simulation scripts carry the comment that the ATE is always 1. They generate the treatment effect $5(1 + X_1 + X_2 + X_3)$ in the baseline design and $5(1 + X_1 + X_2 + X_1 X_2 + X_3)$ in the design with an interaction, with independent covariates that are uniform on $(-1, 1)$. The average effect in the population is then 5, and the average effect in a simulated sample varies from sample to sample. In the homogeneous design the effect is 5 for every unit.\n\nThe sentence of Section~4.2 is not an error; the optional clarification below makes explicit which score it describes. The sentence names political interest, newspaper reading and education as the strongest correlates of the compliance score, while Table~A8 reports a correlation of 0.37 for party ID and of 0.36 for education. Table~A8 is computed with the Winsorized score. For the score before Winsorization, the three strongest correlates are those of the sentence. The sentence does not say which score it describes.", "what_survives": "No estimate, interval or fit changes. The label of Table~A6 and the comment do not enter a computation. The limit 4.8 is the printed value without its leading zero; I obtain 4.7515 in the full bootstrap, which rounds to 4.8." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E010", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of a table with the caption Table A3: Noise (Homogeneous). The columns are n, OLS (Z), OLS (D), 2SLS and ICSW. The rows read: 500: 2.50, 2.64, 0.53, 0.58. 1000: 2.50, 2.62, 0.37, 0.39. 2500: 2.49, 2.61, 0.23, 0.23. 5000: 2.49, 2.64, 0.16, 0.16. 10000: 2.47, 2.61, 0.11, 0.11. 25000: 2.48, 2.63, 0.07, 0.07. One red outline surrounds all 24 entries of the four estimator columns.", "caption": "Supplement, Table A3, p. A7 (supplement PDF p. 7). The outline marks the 24 RMSE entries; their replication formulas subtract the heterogeneous sample-average effect $B$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011.png" }, { "alt": "Crop of three lines. A bold heading reads Homogeneous Treatment Effects, Conditioning on Random Noise. The next line reads Same as the previous DGP, except: and the last line reads tau sub i = 5, for all i. A red outline surrounds the last line.", "caption": "Supplement, p. A2 (supplement PDF p. 2). The outline marks the line that sets the treatment effect of every unit to $5$ in the homogeneous design.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011-b.png" }, { "alt": "Listing of R code with line numbers, in two parts. In the first part, line 22 sets tau1 to 5*(1+X1+X2+X3), line 24 sets Y2 to Y0 + 5*(D), line 26 returns list(D,Y1,tau1,C,Y2), line 83 sets output[1] to mean(dgpout[[3]]), and lines 91 to 94 store in output[9] to output[12] the four estimates computed from Y2. Lines 125 and 127 compute t500 and t1000 as the square root of the column means of (x - simoutput[1,isN]) squared over rows 9:12; a red outline surrounds simoutput[1,isN] on each of the two lines. In the second part, line 60 of the display script has the same statement for t500 with the same red outline, line 72 calls xtable on the six results, and line 74 plots them with the title Noise (Homogeneous).", "caption": "Source code of the replication files, not a page of the article or of the supplement: Simulations Misspecification Noise 6-20.R, lines 22–26, 83, 91–94 and 124–127, and Sim-Display Misspec.R, lines 59–60 and 72–74. The outlines mark the benchmark simoutput[1,isN] in the RMSE statements for rows 9 to 12.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011-c.png" }, { "alt": "Line chart. The horizontal axis is the sample size n on a logarithmic scale, with ticks at 500, 1,000, 2,500, 5,000, 10,000 and 25,000. The vertical axis is labelled RMSE or standard deviation (units of the outcome) and runs from 0 to about 0.65. The title reads: Table A3 measures error about a benchmark that is itself random, where the effect in the homogeneous design is exactly 5. An orange line with square markers gives the ICSW RMSE printed in Table A3: 0.58, 0.39, 0.23, 0.16, 0.11 and 0.07. A blue line with round markers gives the printed 2SLS RMSE: 0.53, 0.37, 0.23, 0.16, 0.11 and 0.07. A dashed green line with triangular markers gives the standard deviation of the benchmark used by the code, 5 divided by the square root of n: 0.224, 0.158, 0.100, 0.071, 0.050 and 0.032. A note below the chart reads: Corrected RMSEs are not shown: they need the simulation output or a new run.", "caption": "The RMSEs of 2SLS and ICSW printed in Table A3 and the standard deviation $5/\\sqrt{n}$ of the benchmark that the code subtracts from each estimate, by sample size. The benchmark has mean $5$, the true effect, and its standard deviation falls from $0.224$ at $n = 500$ to $0.032$ at $n = 25{,}000$. Corrected RMSEs are not shown, because they are not established.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E011-benchmark.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T5", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T5" }, { "id": "P029-T6", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T6" }, { "id": "P029-T22", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T22" } ], "headline": "The replication code for Table A3 and the “Noise (Homogeneous)” panel of Figure A3 computes RMSEs about the sample-average effect of the companion heterogeneous design, a random quantity with mean $5$ and variance $25/n$; the effect in the homogeneous design is exactly $5$, and the RMSEs should be computed about $5$.", "id": "P029-E011", "location": "Supplement: Table A3, p. A7 (supplement PDF p. 7); Figure A3, panel “Noise (Homogeneous)”, p. A6 (supplement PDF p. 6); definition of the design, p. A2 (supplement PDF p. 2). Replication files: Simulations Misspecification Noise 6-20.R, lines 125–135, and Sim-Display Misspec.R, lines 60–70", "pdf_page": 7, "printed_page": "A7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T22", "P029-T5", "P029-T6" ], "text": { "as_printed": "\\emph{Supplement, p.~A2.} \\textbf{Homogeneous Treatment Effects, Conditioning on Random Noise}\n\nSame as the previous DGP, except: $\\tau_i = 5, \\forall i$.\n\n\\emph{Supplement, p.~A7.}\n\\begin{audittable}{rrrrr}{Table A3: Noise (Homogeneous)}\n$n$ & OLS ($Z$) & OLS ($D$) & 2SLS & ICSW \\\\\n500 & 2.50 & 2.64 & 0.53 & 0.58 \\\\\n1000 & 2.50 & 2.62 & 0.37 & 0.39 \\\\\n2500 & 2.49 & 2.61 & 0.23 & 0.23 \\\\\n5000 & 2.49 & 2.64 & 0.16 & 0.16 \\\\\n10000 & 2.47 & 2.61 & 0.11 & 0.11 \\\\\n25000 & 2.48 & 2.63 & 0.07 & 0.07 \\\\\n\\end{audittable}\n\n\\emph{Replication files, \\texttt{Simulations Misspecification Noise 6-20.R}, lines 22, 24, 26, 83 and 125. Line 125 is broken here after \\texttt{function(x)}. Lines 127, 129, 131, 133 and 135 repeat it for the other sample sizes, and lines 60 to 70 of \\texttt{Sim-Display Misspec.R} repeat all six.}\n\\[\n\\begin{aligned}\n&\\texttt{tau1<- 5*(1+X1+X2+X3)}\\\\\n&\\texttt{Y2 <- Y0 + 5*(D)}\\\\\n&\\texttt{return(list(D,Y1,tau1,C,Y2))}\\\\\n&\\texttt{output[1] <- mean(dgpout[[3]])}\\\\\n&\\texttt{t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)}\\\\\n&\\qquad\\texttt{(x - }\\hl{\\texttt{simoutput[1,isN]}}\\texttt{)\\textasciicircum 2))\\textasciicircum .5,mean(simoutput[7,isN]))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files.} In the six RMSE statements for rows 9 to 12 of \\texttt{Simulations Misspecification Noise 6-20.R} (lines 125, 127, 129, 131, 133 and 135) and in the six corresponding statements of \\texttt{Sim-Display Misspec.R} (lines 60, 62, 64, 66, 68 and 70), the benchmark is the constant $5$:\n\\[\n\\begin{aligned}\n&\\texttt{t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)}\\\\\n&\\qquad\\texttt{(x - }\\hl{\\texttt{5}}\\texttt{)\\textasciicircum 2))\\textasciicircum .5,mean(simoutput[7,isN]))}\n\\end{aligned}\n\\]\nRow~1 and the statements for rows 2 to 5, which score the heterogeneous design, stay as they are.\n\n\\emph{Supplement.} For the design with $\\tau_i = 5$ for all $i$, the RMSE of estimator $j$ at sample size $n$ is\n\\[\n\\mathrm{RMSE}_j = \\Bigl(\\frac{1}{R}\\sum_{r=1}^{R}(t_{jr} - \\hl{5})^2\\Bigr)^{1/2},\n\\]\nwhere $t_{jr}$ is the estimate in the simulated dataset $r$ and $R = 2{,}500$ is the number of simulated datasets. Regenerate Table~A3 and the ``Noise (Homogeneous)'' panel of Figure~A3 with this formula, from the saved output of the simulation or from a new run. The corrected entries are not established here.", "evidence": "The evidence is an exact counterexample and a derivation, together with a reading of the two scripts. Let $t_{jr}$ be the estimate of estimator $j$ in the simulated dataset $r$, let $B_r$ be the benchmark of that dataset, and let $R$ be the number of simulated datasets for a given $n$. Both scripts return\n\\[\n\\Bigl(\\frac{1}{R}\\sum_{r=1}^{R}(t_{jr} - B_r)^2\\Bigr)^{1/2} \\quad\\text{where the design requires}\\quad \\Bigl(\\frac{1}{R}\\sum_{r=1}^{R}(t_{jr} - 5)^2\\Bigr)^{1/2}.\n\\]\n\nFor the counterexample, I take $n = 500$ units in $50$ identical blocks of ten. Set $a = \\dots = h = 0$, let the latent covariates be $(X1_i, X2_i, X3_i) = (1/5, 0, 0)$ for every unit, and let all three disturbances be zero. In each block the observed covariates take the five values $(0,0,0)$, $(1/2,0,0)$, $(0,1/2,0)$, $(0,0,1/2)$ and $(-1/2,-1/2,-1/2)$, each once with $Z_i = 0$ and once with $Z_i = 1$. Then $AC_i = 1$ and $A_i = 0$ for every unit, so that $D_i = Z_i$, $Y0_i = 5$ and $Y2_i = 5 + 5Z_i$. The regressors and the instruments coincide and have full rank, and the 2SLS coefficient on $D$ is exactly $5$. Since $\\texttt{tau1}_i = 5(1 + 1/5) = 6$ for every unit, I have $B = 6$. The code assigns the squared error $(5-6)^2 = 1$ to an estimate that equals the true effect. A second input differs only in the disturbance of the outcome, which equals $Z_i$. For this input, I have $Y2_i = 5 + 6Z_i$, the 2SLS estimate is $6$, and again $B = 6$. The code assigns the squared error $0$ to an estimate that is off by $1$.\n\\begin{audittable}{lrrrr}{The two exact inputs with $n = 500$: the 2SLS estimate $t$, the benchmark $B$ that the code subtracts, and the two squared errors.}\nInput & $t$ & $B$ & $(t-B)^2$, as coded & $(t-5)^2$, correct \\\\\nFirst & 5 & 6 & 1 & 0 \\\\\nSecond & 6 & 6 & 0 & 1 \\\\\n\\end{audittable}\nThe original functions of the replication files, run on the first input, return $5$ for all four homogeneous fits, and the RMSE expressions of both scripts return $1$ for 2SLS; with the benchmark replaced by $5$ they return $0$. Since the thresholds hold with strict inequality and the designs are nonsingular in both inputs, the same discrepancy occurs on a set of inputs of positive probability.\n\nI compute the variance of the benchmark from the design. A variable that is uniform on $(-1,1)$ has mean $0$ and variance $1/3$, and $B - 5$ is $5/n$ times a sum of $3n$ independent such variables. I obtain\n\\[\nE(B) = 5, \\qquad \\operatorname{Var}(B) = \\frac{25}{n^2}\\cdot 3n\\cdot\\frac{1}{3} = \\frac{25}{n}.\n\\]\nFor any estimate $t$, I have\n\\[\n(t - B)^2 - (t - 5)^2 = (B - 5)^2 - 2(t - 5)(B - 5).\n\\]\nIf $t$ has a finite second moment, taking expectations gives $E[(t-B)^2] - E[(t-5)^2] = 25/n - 2\\operatorname{Cov}(t, B)$. The two inputs above show that the difference can have either sign in a single dataset. The size of the change in each entry of Table~A3, and whether the entry rises or falls, is therefore not established.", "problem": "In the homogeneous design every unit has the treatment effect $5$. This implies that the sample-average effect, the ATE and the LATE are all equal to $5$, and that the RMSE of an estimator is its root mean squared error about $5$. The simulation script generates the homogeneous outcome $Y2 = Y0 + 5D$ together with the outcome of the heterogeneous design, and it stores the four estimates from $Y2$ in rows 9 to 12 of its output. It scores these rows against row~1. Row~1 holds the mean of $\\texttt{tau1}_i = 5(1 + X1_i + X2_i + X3_i)$, the sample-average effect of the heterogeneous design. This benchmark is\n\\[\nB = 5 + \\frac{5}{n}\\sum_{i=1}^{n}(X1_i + X2_i + X3_i),\n\\]\na random quantity with mean $5$ and variance $25/n$. Both reporting scripts therefore compute a root mean squared distance from $B$ for every estimator and sample size in Table~A3 and the corresponding panel of Figure~A3, instead of the required RMSE about $5$. The corrected values cannot be computed from the printed ones, because the difference depends on how each estimate varies with $B$ across the simulated datasets. They require the saved output of the simulation, which is not among the replication files, or a new run.", "what_survives": "The definition of the homogeneous design and the fitted estimates are not affected, and 2SLS and ICSW both estimate the constant effect $5$ in this design. The correction changes the scoring of one supplementary design: Table~A3 and one panel of Figure~A3 have to be regenerated. The identification and consistency arguments, the two applications, the other simulation designs, the study of the Winsorization constant and the bootstrap study do not depend on it. Whether the corrected RMSEs of 2SLS and ICSW remain close to each other is not established." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E011", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the top of Table 2, titled 2SLS and ICSW estimates for Albertson and Lawrence (2009). Under Knowledge are the columns LATE (2SLS), (1), and ATE (ICSW), (2); under Opinion are the columns LATE (2SLS), (3), and ATE (ICSW), (4). Row Watching debate: 0.28 with interval (-0.04, 0.59); 0.40 with interval (0.01, 1.26); -0.07 with interval (-0.25, 0.13); -0.05 with interval (-0.31, 0.24). Row Intercept: 1.81 with interval (1.33, 2.26); 1.90 with interval (1.34, 2.50); 1.01 with interval (0.70, 1.30); 0.99 with interval (0.60, 1.29). A red outline surrounds column (4), from its heading ATE (ICSW) to the interval (0.60, 1.29).", "caption": "Table 2, title, column headings and first two rows, p. 503 (PDF p. 12). The outline marks column (4), the opinion ICSW model, whose upper limits line 195 of the script is written to assemble. The printed limits are finite.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E012.png" }, { "alt": "Image of R code in a monospaced face, under the heading albertsonlawrence.R, lines 193-195 (replication files). Line 193, over three rows: mat1 <- round(cbind(apply(ests,2,quantile,c(0.025))[1:10],apply(ests,2,quantile,c(0.025))[11:20],apply(ests,2,quantile,c(0.025))[21:30],apply(ests,2,quantile,c(0.025))[31:40]),digits=2). Line 194 is empty. Line 195, over three rows: mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],apply(ests,2,quantile,c(0.975))[11:20],apply(ests,2,quantile,c(0.975))[21:30],apply(ests,2,quantile,c(0.975))[310]),digits=2). A red outline surrounds the subscript [310] near the end of line 195.", "caption": "Replication script albertsonlawrence.R, lines 193–195, typeset from the replication files (a script has no published page). The outline marks the subscript [310] in line 195; line 193 has [31:40] in the same place.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E012-code.png" } ], "formal_release": null, "formal_statements": [], "headline": "In the line of the replication script that assembles the upper bootstrap limits for Table 2, the last subscript [310] should read [31:40]; as deposited, the line returns missing values for the ten upper limits of the opinion ICSW model, and no printed limit of Table 2 is shown to be wrong.", "id": "P029-E012", "location": "Replication script albertsonlawrence.R, line 195; the paper’s counterpart is Table 2, column (4), upper confidence limits, p. 503 (PDF p. 12)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Script, lines 193 and 195 (each is one line of code, broken here after each argument).}\n\\[\n\\begin{aligned}\n&\\texttt{mat1 <- round(cbind(apply(ests,2,quantile,c(0.025))[1:10],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.025))[11:20],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.025))[21:30],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.025))[31:40]),digits=2)}\\\\\n&\\texttt{mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[11:20],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[21:30],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))}\\hl{\\texttt{[310]}}\\texttt{),digits=2)}\n\\end{aligned}\n\\]\n\n\\emph{Table~2, column (4), Opinion, ATE (ICSW), p.~503.} Watching debate: $-0.05$ $(-0.31, 0.24)$.", "corrected": "\\emph{Script, line 195 (one line of code).}\n\\[\n\\begin{aligned}\n&\\texttt{mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[11:20],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))[21:30],}\\\\\n&\\qquad\\texttt{apply(ests,2,quantile,c(0.975))}\\hl{\\texttt{[31:40]}}\\texttt{),digits=2)}\n\\end{aligned}\n\\]", "evidence": "The evidence is an exact counterexample. I ran the two lines as deposited on a matrix with two rows and 40 columns such that the entries in column $j$ are $j$ and $j + 40$. The quantiles of column $j$ at $0.025$ and $0.975$ are $j + 1$ and $j + 39$. Line 193 returns the four intended columns. Line 195 returns $40, \\ldots, 49$, then $50, \\ldots, 59$, then $60, \\ldots, 69$, and ten missing values in the fourth column. With \\texttt{[31:40]} it returns $70, \\ldots, 79$ there and leaves the other columns unchanged. Neither \\texttt{mat1} nor \\texttt{mat2} is used elsewhere in the replication scripts, and the script saves its workspace at line 155, before both lines.", "problem": "The bootstrap matrix \\texttt{ests} of the script has 40 columns: ten coefficients for each of four models, in the order knowledge 2SLS, knowledge ICSW, opinion 2SLS and opinion ICSW. Line 193 assembles the lower limits from the columns 1 to 10, 11 to 20, 21 to 30 and 31 to 40. Line 195 does the same for the upper limits, except that its last subscript is \\texttt{[310]}. The vector of 40 quantiles has no element 310. R returns a single missing value there, and \\texttt{cbind} repeats it ten times. The fourth column of \\texttt{mat2}, which should hold the upper limits of the opinion ICSW model in column (4) of Table~2, is then missing in all ten rows.", "what_survives": "The typo comes after the point-estimation and bootstrap calculations of the script. The point estimates, the bootstrap draws, the quantiles of the differences in coefficients, the matrix of lower limits and the first three columns of upper limits do not pass through it. Table~2 prints ten finite upper limits in column (4), and none is shown to be wrong. How the table was assembled from the output is not known. Bootstrap draws with missing coefficients are a separate question, which the repair does not address, and I did not establish that any occurred." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E012", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Listing of R code with line numbers. Lines 39 and 40 read return( and -sum(weights*log((p.D)^D * (1 - p.D)^(1 - D)))). After a gap, line 45 begins the one-sided function with if (onesided == TRUE) complier.neg.logLik.fnc <- function (par), lines 46 to 48 set theta, p.C and p.D <- p.C * Z, and lines 49 and 50 read return( and -sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights))). A red outline surrounds the characters * weights on line 50.", "caption": "Source code of the replication files, not a page of the article: icsw.R, lines 39–41 and 45–52. The outline marks the weights inside the logarithm on line 50; on line 40 the weights multiply the logarithm.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E014.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T8", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T8" } ], "headline": "In the one-sided branch of the function complier.score, the observation weights are inside the logarithm, so positive weights leave the likelihood minimizers unchanged; the weights should multiply the logarithm, as in the two-sided branch.", "id": "P029-E014", "location": "Replication files, icsw.R, function complier.score, branch for one-sided noncompliance, lines 45–52 (objective on line 50); the objective for two-sided noncompliance is on lines 39–40", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T8" ], "text": { "as_printed": "\\emph{Replication files, \\texttt{icsw.R}, lines 39--40 (two-sided noncompliance) and lines 49--50 (one-sided noncompliance).}\n\\[\n\\begin{aligned}\n&\\texttt{return(}\\\\\n&\\quad\\texttt{-sum(weights*log((p.D)\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D))))}\\\\\n&\\texttt{return(}\\\\\n&\\quad\\texttt{-sum(log((p.D)\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D)}\\hl{\\texttt{ * weights}}\\texttt{)))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files, \\texttt{icsw.R}, lines 49--50.}\n\\[\n\\begin{aligned}\n&\\texttt{return(}\\\\\n&\\quad\\texttt{-sum(}\\hl{\\texttt{weights[weights > 0]*}}\\texttt{log(((p.D)\\textasciicircum D *}\\\\\n&\\qquad\\texttt{(1 - p.D)\\textasciicircum (1 - D))}\\hl{\\texttt{[weights > 0]}}\\texttt{)))}\n\\end{aligned}\n\\]\nIn full, for weights $w_i \\ge 0$ the one-sided branch minimizes over $\\theta$\n\\[\n-\\sum_{i:\\,w_i > 0} w_i \\log L_i(\\theta), \\qquad L_i(\\theta) = p_i(\\theta)^{D_i}\\bigl(1 - p_i(\\theta)\\bigr)^{1 - D_i}, \\qquad p_i(\\theta) = Z_i\\,F(x_i^{\\top}\\theta),\n\\]\nwhere $x_i$ consists of an intercept and the standardized covariates of unit $i$, and $F$ is the distribution function passed as \\texttt{link}, by default $\\Phi$. Observations with weight zero are left out before the logarithm is taken. The logarithm is evaluated as $\\log p_i$ when $D_i = 1$ and as $\\log(1 - p_i)$ when $D_i = 0$, which avoids the product $0\\cdot\\log 0$ for units with $Z_i = 0$.", "evidence": "The evidence is an exact counterexample and a run of the original function. The example has six observations with one covariate $W_i$.\n\\begin{audittable}{rrrrr}{Six observations with one-sided noncompliance and positive weights.}\n$i$ & $W_i$ & $Z_i$ & $D_i$ & $w_i$ \\\\\n1 & $-1$ & 1 & 0 & 1 \\\\\n2 & $-1$ & 1 & 1 & 3 \\\\\n3 & $1$ & 1 & 0 & 3 \\\\\n4 & $1$ & 1 & 1 & 1 \\\\\n5 & $-1$ & 0 & 0 & 5 \\\\\n6 & $1$ & 0 & 0 & 7 \\\\\n\\end{audittable}\nWith an intercept and one slope, the fitted probabilities $p_-$ and $p_+$ of compliance at $W_i = -1$ and $W_i = 1$ range freely over $(0,1)^2$, and the two control observations have $L_i = 1$. For $a, b > 0$, the function $-a\\log p - b\\log(1-p)$ has its unique minimum at $p = a/(a+b)$. The function on line 50 is $-\\log\\bigl(p_-(1-p_-)\\,p_+(1-p_+)\\bigr)$ plus a constant, and its minimum is at $(p_-, p_+) = (1/2, 1/2)$. The weighted negative log likelihood is $-\\log\\bigl(p_-^3(1-p_-)\\,p_+(1-p_+)^3\\bigr)$, and its minimum is at $(3/4, 1/4)$. The weighted likelihood $\\prod_i L_i^{w_i}$ equals $1/256$ at the first point and $729/65536$ at the second.\n\nThe original function was also run on eight observations: at each of $W_i = -1$ and $W_i = 1$, three units with $Z_i = 1$ and outcomes $D_i = 0, 0, 1$, with weights $1, 1, 6$ and $3, 3, 2$, and one unit with $Z_i = 0$. It returns the score $1/3$ for every unit, with unit weights and with these weights. The weighted fit is $3/4$ at $W_i = -1$ and $1/4$ at $W_i = 1$; the corrected objective and a weighted probit regression both return it. With a weight equal to zero, the original objective is infinite at the starting value.", "problem": "Let $w_i$ be the weight of observation $i$ and let $L_i(\\theta)$ be its Bernoulli likelihood contribution. Line 50 computes\n\\[\n-\\sum_i \\log\\bigl(w_i L_i(\\theta)\\bigr) = -\\sum_i \\log L_i(\\theta) - \\sum_i \\log w_i .\n\\]\nThe last sum does not depend on $\\theta$. This implies that, for positive weights, the minimizers of this function are those of the unweighted negative log likelihood. A zero weight makes the function infinite. The function meant is the weighted negative log likelihood $-\\sum_i w_i \\log L_i(\\theta)$, which line 40 computes for two-sided noncompliance. No script of the replication files calls \\texttt{complier.score} with \\texttt{onesided=TRUE}, and no reported number is affected. Unequal weights therefore do not weight the fit as intended.", "what_survives": "The canvassing application fits its scores by a weighted probit regression on the units with $Z_i = 1$ and does not use this branch. The debate application and the simulations use the two-sided branch, whose objective weights the logarithm correctly. All 15 calls to \\texttt{complier.score} in the replication files leave \\texttt{onesided} at its default \\texttt{FALSE}. No table, simulation or result of the paper changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E014", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Listing of R code with line numbers. Line 15 declares complier.score with the arguments D, Z, W, weights = NULL, link = pnorm, genoud = TRUE, num.iter and onesided=FALSE. Lines 32 and 33 set p.AC and p.A to link(rowSums(...)) and line 47 sets p.C to link(rowSums(...)). Line 86 sets CA.score to pnorm(rowSums(...)), line 87 sets A.score to pnorm(rowSums(...)) times CA.score, line 88 sets C.score to CA.score - A.score, and line 98 sets C.score to pnorm(rowSums(...)) in the one-sided branch. Red outlines surround the word pnorm on lines 86, 87 and 98.", "caption": "Source code of the replication files, not a page of the article: icsw.R, lines 15, 32–33, 47, 86–89 and 98–99. The outlines mark pnorm in the three statements that compute the returned scores; the likelihood on lines 32, 33 and 47 uses link.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E015.png" } ], "formal_release": null, "formal_statements": [], "headline": "The function complier.score fits the likelihood with the distribution function passed as link and then computes the returned scores with the normal distribution function; the two agree for the default probit, which every analysis of the paper uses.", "id": "P029-E015", "location": "Replication files, icsw.R, function complier.score, lines 86–87 and 98; the argument link is declared on line 15 and used in the likelihood on lines 32–33 and 47", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Replication files, \\texttt{icsw.R}, lines 32--33 and 47 (likelihood) and lines 86, 87 and 98 (returned scores). The last three lines are broken here after the multiplication sign.}\n\\[\n\\begin{aligned}\n&\\texttt{p.AC <- link(rowSums(t(t(W1) * theta)))}\\\\\n&\\texttt{p.A <- link(rowSums(t(t(W1) * theta.A)))}\\\\\n&\\texttt{p.C <- link(rowSums(t(t(W1) * theta)))}\\\\\n&\\texttt{CA.score <- (}\\hl{\\texttt{pnorm}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par)/2)]))))}\\\\\n&\\texttt{A.score <- (}\\hl{\\texttt{pnorm}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[(length(opt.log\\$par)/2+1):(length(opt.log\\$par))])))) * CA.score}\\\\\n&\\texttt{C.score <- (}\\hl{\\texttt{pnorm}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par))]))))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files, \\texttt{icsw.R}, lines 86, 87 and 98.}\n\\[\n\\begin{aligned}\n&\\texttt{CA.score <- (}\\hl{\\texttt{link}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par)/2)]))))}\\\\\n&\\texttt{A.score <- (}\\hl{\\texttt{link}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[(length(opt.log\\$par)/2+1):(length(opt.log\\$par))])))) * CA.score}\\\\\n&\\texttt{C.score <- (}\\hl{\\texttt{link}}\\texttt{(rowSums(t(t(W1) *}\\\\\n&\\qquad\\texttt{opt.log\\$par[1:(length(opt.log\\$par))]))))}\n\\end{aligned}\n\\]\nIn full, with fitted coefficient vectors $\\widehat\\theta$ and $\\widehat\\theta_A$ and the distribution function $F$ passed as \\texttt{link}, the two-sided branch returns the always-taker score $F(x_i^{\\top}\\widehat\\theta)\\,F(x_i^{\\top}\\widehat\\theta_A)$ and the complier score $F(x_i^{\\top}\\widehat\\theta)\\bigl(1 - F(x_i^{\\top}\\widehat\\theta_A)\\bigr)$, and the one-sided branch returns the complier score $F(x_i^{\\top}\\widehat\\theta)$. Calls with the default probit are unchanged.", "evidence": "The evidence is an exact counterexample and a run of the original function. Let $F(t) = 1/(1 + 3e^{-t})$, which is a distribution function with $F(0) = 1/4$, and recall that $\\Phi(0) = 1/2$. The data have $64$ observations with one covariate $W_i = \\pm 1$ and $16$ observations in each of the four cells of $(W_i, Z_i)$. In each cell with $Z_i = 0$ one unit is treated, and in each cell with $Z_i = 1$ four units are treated. At the zero coefficient vector, the model assigns probability $q = F(0) = 1/4$ to being a complier or an always-taker and probability $r = F(0) = 1/4$ to being an always-taker given either type. The fitted probabilities of treatment are then $qr = 1/16$ when $Z_i = 0$ and $q = 1/4$ when $Z_i = 1$. Since these equal the observed proportions in all four cells, the zero vector maximizes the likelihood, and it is the unique maximizer because $F$ is strictly increasing. The fitted model has always-taker score $qr = 1/16$ and complier score $q(1-r) = 3/16$. The function returns $\\Phi(0)\\Phi(0) = 1/4$ and $\\Phi(0)(1 - \\Phi(0)) = 1/4$. For the one-sided branch, I use the same data with no treated unit among those with $Z_i = 0$. The fitted complier score is $F(0) = 1/4$, and the function returns $\\Phi(0) = 1/2$. A run of the original function on both datasets reproduces the returned values.", "problem": "The article defines the fitted compliance score with the same distribution function $F$ as the likelihood (p.~497), and it takes $F = \\Phi$ for its own calculations. In the function, a distribution function supplied through \\texttt{link} enters the likelihood, but the returned scores apply \\texttt{pnorm} to the fitted coefficients. With the default \\texttt{link = pnorm} the two coincide for every coefficient vector. With another distribution function, the returned scores can differ from the scores of the fitted model.", "what_survives": "All 15 calls to \\texttt{complier.score} in the replication files omit \\texttt{link} and use the probit, for which the returned scores are those of the fitted model. The canvassing application obtains its scores from a probit regression. The article's formula for the score with a general $F$ is correct as printed. No reported number changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E015", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the last six lines of Section 3.3. Red boxes mark the sentence 'To characterize uncertainty, we recommend bootstrapping the entire process---from computing the compliance score to weighted IV estimation (Abadie 2002).', which ends with the marker of footnote 7.", "caption": "Section 3.3, p. 498 (PDF p. 7), last paragraph. The marked sentence recommends bootstrapping the whole procedure, the compliance score included. The recommendation is correct; the error is in the package that implements it.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E016.png" }, { "alt": "Listing of eighteen lines of R code. A red box marks the call 'fit <- glm(D ~ W, df, subset = (Z == 1), family = binomial(link=\"probit\"))' on lines 43 to 47, which has no weights argument. A second red box marks '* weights' inside the logarithm in line 91, 'return(-sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights)))'.", "caption": "Package icsw 1.0.0, R/compliance.score.R, lines 43–48 and 81–92, typeset from the source file with line numbers added. The marked call fits the one-sided probit without weights; the marked factor multiplies the likelihood contribution by the weights inside the logarithm.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E016-code.png" }, { "alt": "Two panels, for the strata W = 0 and W = 1, with the bootstrap replicate on the horizontal axis (original fit, then 1 to 4) and the fitted compliance score from 0 to 1 on the vertical axis. Blue circles, the published code, lie at 0.25 in every replicate in the left panel and at 0.75 in the right panel. Orange diamonds, a probit refitted with the replicate weights, have the same score as the circles at the original fit and then move: about 0.33, 0.12, 0.22 and 0.19 on the left, and about 0.99, 0.88, 0.67 and 0.63 on the right.", "caption": "Fitted compliance scores in the original fit and in four bootstrap replicates of the one-sided code of icsw 1.0.0, for the 16-observation example. The published code returns $0.25$ and $0.75$ every time; a probit that uses the replicate weights returns scores that change with the weights. Four replicates show the path through the code and do not estimate a sampling distribution.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E016-bootstrap-scores.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T7", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T7" }, { "id": "P029-T8", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T8" }, { "id": "P029-T23", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T23" } ], "headline": "In the R package icsw (version 1.0.0) the observation weights must enter the fit of the compliance score: as published, the one-sided probit omits them and the two-sided objective puts them inside the logarithm, so that the bootstrap does not refit the scores to its weights.", "id": "P029-E016", "location": "Section 3.3, p. 498 (PDF p. 7), last paragraph; R package icsw, version 1.0.0: R/compliance.score.R, lines 43–47 and 91, R/icsw.tsls.R, lines 28–37, and the help page of icsw.tsls", "pdf_page": 7, "printed_page": "498", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P029-T23", "P029-T7", "P029-T8" ], "text": { "as_printed": "Article, p.~498: To characterize uncertainty, we recommend bootstrapping the entire process---from computing the compliance score to weighted IV estimation (Abadie 2002).$^{7}$\n\nPackage, \\texttt{R/compliance.score.R}, lines 43--47 (line breaks removed; the call has no \\texttt{weights} argument):\n\n$\\texttt{fit <- }\\hl{\\texttt{glm(D \\textasciitilde{} W, df, subset = (Z == 1), family = binomial(link=\"probit\"))}}$\n\nLine 91:\n\n$\\texttt{return(-sum(log((p.D)\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D)}\\hl{\\texttt{ * weights}}\\texttt{)))}$\n\n\\texttt{R/icsw.tsls.R}, lines 29--32 (line breaks removed; the call continues):\n\n\\texttt{weights.r <- rexp(n.obs, 1) \\# Bayesian bootstrap}\n\n\\texttt{coefs.r <- icsw.tsls.fit(D, X, Y, Z, W, weights = weights * weights.r,}\n\nHelp page of \\texttt{icsw.tsls}: Estimate average treatment effects using two-stage least squares with a binary instrument and binary treatment and weighting with inverse complier scores (probabilities of compliance). Optionally, bootstrap the \\hl{entire} estimation process for the purpose of hypothesis testing and constructing confidence intervals.", "corrected": "In \\texttt{R/compliance.score.R}, lines 43--47:\n\n$\\texttt{fit <- glm(D \\textasciitilde{} W, df, subset = (Z == 1),}\\hl{\\texttt{ weights = weights,}}$\n\n$\\texttt{family = binomial(link=\"probit\"))}$\n\nLine 91:\n\n$\\texttt{return(-sum(}\\hl{\\texttt{weights * }}\\texttt{log(p.D\\textasciicircum D * (1 - p.D)\\textasciicircum (1 - D))))}$\n\nLine 89 keeps \\texttt{p.D} in $[10^{-5},1-10^{-5}]$, so that every logarithm is finite and a weight of zero removes the observation from the sum.", "evidence": "The evidence is a derivation for the two-sided objective, an exact counterexample for the one-sided branch, and a run of the package's code. Let $L_i(\\theta)=p_i(\\theta)^{D_i}\\{1-p_i(\\theta)\\}^{1-D_i}$, where the code keeps $p_i(\\theta)$ in $[10^{-5},1-10^{-5}]$. For positive weights $w_i$, the objective on line 91 is\n\\[\n-\\sum_i\\log\\{L_i(\\theta)w_i\\}=-\\sum_i\\log L_i(\\theta)-\\sum_i\\log w_i ,\n\\]\nand the last sum does not depend on $\\theta$. The weighted negative log-likelihood is $-\\sum_i w_i\\log L_i(\\theta)$. If some $w_i=0$, then $\\log\\{L_i(\\theta)w_i\\}=-\\infty$ for every $\\theta$.\n\nThe counterexample has 16 observations, four in each cell of a binary covariate $W$ and the assignment $Z$. No unit assigned to control is treated. Among the assigned units, one of four is treated at $W=0$ and three of four at $W=1$. Each treated unit has weight $9$ and every other unit has weight $1$. A probit with an intercept and $W$ is saturated, so that the fitted score at each value of $W$ is the proportion treated among the assigned units, weighted when the fit uses the weights. The outcome is $Y=D(1+W)$, and, with an intercept and no other covariates in the IV fit, the weighted IV estimate reduces to $\\sum_{Z_i=1}v_iY_i/\\sum_{Z_i=1}v_iD_i$ with $v_i$ the weight divided by the score.\n\\begin{audittable}{lrr}{Exact one-sided example with 16 observations and weights 9 and 1.}\nQuantity & Package 1.0.0 & Weights in the probit \\\\\n\\hline\nCompliance score, $W=0$ & $1/4$ & $9/(9+3)=3/4$ \\\\\nCompliance score, $W=1$ & $3/4$ & $27/(27+1)=27/28$ \\\\\nICSW coefficient on $D$ & $(36+72)/(36+36)=3/2$ & $(12+56)/(12+28)=17/10$ \\\\\n\\end{audittable}\nRunning the unmodified package code on these data, I find scores $(0.25,0.75)$ with and without the weights and a coefficient of $1.5$. With \\texttt{weights = weights} added to the call, the coefficient is $1.7$. The first step also matters for the bootstrap. Starting from unit weights, I change the weight of one untreated assigned unit at $W=0$ from $1$ to $t$. The package's coefficient stays at $3/2$, while refitting the score gives $(t+11)/(t+7)$, with derivative $-1/16$ at $t=1$. In a trace of four bootstrap replicates of the one-sided code on the same $D$, $Z$ and $W$, the scores are identical in the original fit and in every replicate. Each bootstrap coefficient equals the second stage computed with the scores held fixed.\n\nThree points are not established. The size and the direction of the error in the standard errors are unknown. Leaving out the first step removes a variance and a covariance, and the sign of their sum is not determined. In the two-sided branch the weights do enter a regression for starting values, and a numerical optimizer may return different approximate fits from different starts, although the objective has the same minimizers. The default optimizer for the two-sided branch (\\texttt{rgenoud}) was not run, and I do not state what the example on the help page returns.", "problem": "The article recommends that the bootstrap repeat the whole procedure, the fit of the compliance score included, and that recommendation is correct. The package says on its help page that it bootstraps ``the entire estimation process'', and it does so by reweighting. Each replicate draws exponential weights, multiplies them into the observation weights and calls the fitting routine again. The compliance model is not fitted with those weights. In the one-sided branch the probit is fitted by \\texttt{glm} without a \\texttt{weights} argument, and the fitted scores are the same in every replicate. In the two-sided branch the weights multiply the likelihood contribution inside the logarithm. This adds a term to the objective that does not depend on the parameters. The set of minimizers is then the same for every vector of positive weights. There are three consequences. The package's bootstrap standard errors leave out the variation of the first step, contrary to the article and to the help page. Sampling weights or missing-data weights that the user supplies are not used to weight the compliance-model log likelihood, and this can change the estimate itself. A weight of zero makes the two-sided objective infinite at every parameter value. The example on the help page sets the weight to zero for the 9 of its 507 observations with a missing outcome.", "what_survives": "This package defect does not implicate the article's mathematical results or reported estimates. The article's own scripts do not use this package. The one-sided application fits a weighted probit directly and refits it on resampled units. The two-sided application and the bootstrap simulations resample rows and use the helper in the replication files, whose two-sided objective has the weights outside the logarithm. The one-sided branch of that helper has the same misplaced weights, but the article's one-sided application does not use it. With unit weights and no bootstrap, the package's compliance models use ordinary unweighted objectives. Package results with nonconstant weights, and all bootstrap standard errors from the package, are to be recomputed after the repair." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E016", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Listing of R code with line numbers, in two parts. In the first part, lines 111 to 114 call minqa::bobyqa with starting, complier.neg.logLik.fnc and control = list(maxit = num.iter, trace = 0), and line 115 reads if (opt.log$convergence > 0) warning(\"Nelder-Mead did not converge!\"). Red outlines surround maxit = num.iter, trace = 0 on line 113, and opt.log$convergence and Nelder-Mead on line 115. The second part shows line 81 of icsw.R, which calls bobyqa with control=list(iprint=0,maxfun=num.iter), lower=-5, upper=5, without marks.", "caption": "Source code, not a page of the article: package icsw, R/compliance.score.R, lines 110–116, and for comparison line 81 of icsw.R in the replication files. The outlines mark the two control names, the field that the optimizer does not return, and the name of another optimizer in the warning.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E017.png" } ], "formal_release": null, "formal_statements": [], "headline": "With genoud = FALSE, the package function compliance.score stops with an error under two-sided noncompliance: it passes control names that the BOBYQA optimizer does not use, and it tests a field that the optimizer does not return.", "id": "P029-E017", "location": "R package icsw, version 1.0.0, file R/compliance.score.R, lines 110–116 (the branch genoud = FALSE of the fit for two-sided noncompliance)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Package \\texttt{icsw}, \\texttt{R/compliance.score.R}, lines 111--115.}\n\\[\n\\begin{aligned}\n&\\texttt{opt.log <- minqa::bobyqa(}\\\\\n&\\quad\\texttt{starting, complier.neg.logLik.fnc,}\\\\\n&\\quad\\texttt{control = list(}\\hl{\\texttt{maxit}}\\texttt{ = num.iter, }\\hl{\\texttt{trace}}\\texttt{ = 0)}\\\\\n&\\quad\\texttt{)}\\\\\n&\\texttt{if (opt.log\\$}\\hl{\\texttt{convergence}}\\texttt{ > 0) warning(\"}\\hl{\\texttt{Nelder-Mead}}\\texttt{ did not converge!\")}\n\\end{aligned}\n\\]", "corrected": "\\emph{Package \\texttt{icsw}, \\texttt{R/compliance.score.R}, lines 111--115.}\n\\[\n\\begin{aligned}\n&\\texttt{opt.log <- minqa::bobyqa(}\\\\\n&\\quad\\texttt{starting, complier.neg.logLik.fnc,}\\\\\n&\\quad\\texttt{control = list(}\\hl{\\texttt{maxfun}}\\texttt{ = num.iter, }\\hl{\\texttt{iprint}}\\texttt{ = 0)}\\\\\n&\\quad\\texttt{)}\\\\\n&\\texttt{if (opt.log\\$}\\hl{\\texttt{ierr}}\\texttt{ > 0) warning(\"}\\hl{\\texttt{BOBYQA}}\\texttt{ did not converge!\")}\n\\end{aligned}\n\\]", "evidence": "The evidence is a derivation from the interface of the optimizer and a run of the package source, under R~4.4.2 with \\texttt{minqa}~1.2.8. The example has 32 units with two-sided noncompliance, eight in each cell of a binary covariate and the assignment, with treatment rates $1/4$ and $3/4$. The unmodified function issues the warning ``unused control arguments ignored'' and stops with ``argument is of length zero''. The function \\texttt{icsw.tsls.fit} with \\texttt{genoud = FALSE} stops with the same error. With the corrections below, the function returns 32 finite scores and \\texttt{ierr} equal to $0$. With \\texttt{num.iter = 1} it returns \\texttt{ierr} equal to $1$ and the corrected warning. The function \\texttt{complier.score} in \\texttt{icsw.R} of the replication files, which the analyses of the paper use, calls \\texttt{bobyqa} with \\texttt{maxfun} and \\texttt{iprint}, has no such test, and returns scores on the same example.", "problem": "The function \\texttt{bobyqa} of the package \\texttt{minqa} takes the controls \\texttt{maxfun} and \\texttt{iprint}, and it returns a list with the components \\texttt{par}, \\texttt{fval}, \\texttt{feval}, \\texttt{ierr} and \\texttt{msg}. The names \\texttt{maxit} and \\texttt{trace} are ignored with a warning, and a value of \\texttt{num.iter} chosen by the user has no effect. The returned list has no component \\texttt{convergence}. The expression \\texttt{opt.log\\$convergence} is then \\texttt{NULL}, the comparison with $0$ has length zero, and \\texttt{if} stops with the error ``argument is of length zero''. No scores are returned. The warning names Nelder--Mead, and the optimizer is BOBYQA. The default \\texttt{genoud = TRUE} and the fit for one-sided noncompliance do not pass through these lines.", "what_survives": "The defect is in a branch of the package that is not the default, and it stops the computation without returning a number. The scripts for the simulations and the applications of the paper use the function in the replication files and not the package. No definition, result or reported number of the paper is affected. The correction restores the branch; it does not show that BOBYQA finds the global maximum of the likelihood." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E017", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Listing of R code with line numbers. Line 8 reads low <- simoutput[6,] < 0.33 and line 10 reads RMSE <- apply(simoutput[7:nrow(simoutput),low],1,function(x) mean((x - simoutput[1,low])^2)^.5). Lines 15 and 17 repeat the pair for med, defined as (simoutput[6,] < 0.50) and (0.33 <= simoutput[6,]), and lines 22 and 24 repeat it for high, defined as simoutput[6,] >= 0.50. Red outlines surround simoutput[7:nrow(simoutput),low] on line 10, simoutput[7:nrow(simoutput),med] on line 17 and simoutput[7:nrow(simoutput),high] on line 24.", "caption": "Source code of the replication files, not a page of the supplement: Sim-Display Alpha.R, lines 8–10, 15–17 and 22–24. The outlines mark the three extractions that lose the matrix shape when a group has one dataset.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E019.png" } ], "formal_release": null, "formal_statements": [], "headline": "The script that draws Figures A1 and A2 stops with an error when a compliance group contains exactly one simulated dataset, because the extraction of the group loses its matrix shape; adding drop=FALSE repairs it.", "id": "P029-E019", "location": "Replication files, Sim-Display Alpha.R, lines 10, 17 and 24; the script draws Figures A1 and A2 of the supplement (supplement PDF pp. 3–4)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Replication files, \\texttt{Sim-Display Alpha.R}, lines 8 and 10. Line 10 is broken here after \\texttt{function(x)}. Lines 17 and 24 repeat it with \\texttt{med} and \\texttt{high} in place of \\texttt{low}.}\n\\[\n\\begin{aligned}\n&\\texttt{low <- simoutput[6,] < 0.33}\\\\\n&\\texttt{RMSE <- apply(}\\hl{\\texttt{simoutput[7:nrow(simoutput),low]}}\\texttt{,1,function(x)}\\\\\n&\\qquad\\texttt{mean((x - simoutput[1,low])\\textasciicircum 2)\\textasciicircum .5)}\n\\end{aligned}\n\\]", "corrected": "\\emph{Replication files, \\texttt{Sim-Display Alpha.R}, line 10, and likewise lines 17 and 24 with \\texttt{med} and \\texttt{high}.}\n\\[\n\\begin{aligned}\n&\\texttt{RMSE <- apply(simoutput[7:nrow(simoutput),low}\\hl{\\texttt{,drop=FALSE}}\\texttt{],1,function(x)}\\\\\n&\\qquad\\texttt{mean((x - simoutput[1,low])\\textasciicircum 2)\\textasciicircum .5)}\n\\end{aligned}\n\\]\nThe script may also skip an empty group and report its RMSE as undefined.", "evidence": "The evidence is an exact counterexample, obtained by evaluating the statements of the script. On a matrix with nine rows and three columns whose compliance row is $(0.2, 0.4, 0.6)$, each group contains one dataset, and all three statements stop with the error above. With \\texttt{drop=FALSE} added, each statement returns an RMSE of $1$ in each of its three rows. On a matrix with two datasets in each group, the original and the repaired statements return the same values.", "problem": "When \\texttt{low} selects exactly one column, R returns a vector in place of a matrix with one column, and \\texttt{apply} stops with the error ``dim(X) must have a positive length''. The same holds for \\texttt{med} and \\texttt{high}. No group with a single dataset is shown to occur in the published runs of 2,500 simulated datasets. With two or more datasets in a group the statements work, and the RMSE formulas are right.", "what_survives": "The RMSE formulas and all calculations for groups with two or more datasets are unaffected, and the published figures are not shown to be affected." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E019", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of six lines of the introduction. The text reads: Several influential papers, including Deaton (2009) and Heckman and Urzua (2010), have expressed concern about the interpretation of the LATE. Both papers argue that the LATE is not usually the parameter of interest, as we are unlikely to learn much about the process at work by estimating the LATE (Deaton 2009, 430) and the LATE is often very difficult to interpret as an answer to an interesting economic question (Heckman and Urzua 2010, 35). The crop ends inside the next sentence. One red outline surrounds the word process, and another surrounds the citation (Deaton 2009, 430).", "caption": "Section 1, p. 492 (PDF p. 1). The outlines mark the word “process” in the quotation and the citation with the page number 430.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020.png" }, { "alt": "Crop of thirteen lines of the reference list, from the entry for Angrist and Fernandez-Val (2013) to the entry for Deaton (2009). In the first entry the editors are printed as Manuel Arellano Daron Acemoglu, and Eddie Deke, with the pages 401-36. Red outlines surround the words Arellano Daron and the words Deke, 401-36. The entries for Angrist and Pischke, Arceneaux and Nickerson (2009 and 2010) and Aronow and Carnegie follow without marks. In the last entry, Deaton, Angus. 2009. Instruments of development, Proceedings of the British Academy, 2008 Lectures, a red outline surrounds 162:123-60.", "caption": "Reference list, p. 505 (PDF p. 14). The outlines mark the two editors' names that run together, the misspelled name with the page range of the chapter, and the page range of the edition of Deaton’s essay that the list gives.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020-b.png" }, { "alt": "Crop of the two lines of one reference entry. The text reads: Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til Strmer, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. American Journal of Epidemiology 173(7):761-7. A red outline surrounds the words Til Strmer.", "caption": "Reference list, p. 505 (PDF p. 14). The outline marks the misspelled name in the entry for Funk et al.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020-c.png" }, { "alt": "Crop of sixteen lines at the top of the last page of the reference list, from the entry for Joffe and Brensinger (2003) to the entry for Verba, Schlozman and Brady (1995). Four red outlines surround: 22(1) in Statistics in Medicine 22(1):1285-303; Robert C. in Putnam, Robert C. 2000; the word renewal in Bowling alone: The collapse and renewal of American Community; and the title Voluntarism in American Politics. The entries for Joffe, Ten Have and Brensinger, for Rosenstone and Hansen, for Roy, Hogan and Marcus and the two entries for Rubin carry no marks.", "caption": "Reference list, p. 506 (PDF p. 15). The outlines mark the issue number in the entry for Joffe and Brensinger, the middle initial and the word “renewal” in the entry for Putnam, and the shortened title in the entry for Verba, Schlozman and Brady.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E020-d.png" } ], "formal_release": null, "formal_statements": [], "headline": "The quotation from Deaton reads “processes” in the original and is on p. 130 of the edition listed, and five entries of the reference list have a wrong name, title, issue or page range.", "id": "P029-E020", "location": "Section 1, p. 492 (PDF p. 1), quotation from Deaton; reference list, pp. 505–506 (PDF pp. 14–15), entries for Angrist and Fernandez-Val, Deaton, Funk et al., Joffe and Brensinger, Putnam, and Verba, Schlozman and Brady", "pdf_page": 1, "printed_page": "492", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Section~1, p.~492.} Both papers argue that the LATE is not usually the parameter of interest, as ``we are unlikely to learn much about the \\hl{process} at work'' by estimating the LATE (Deaton 2009, \\hl{430}) [\\ldots]\n\n\\emph{Reference list, pp.~505--506.}\n\\begin{itemize}\n\\item Angrist, Joshua D., and Ivan Fernandez-Val. 2013. ExtrapoLATE-ing: External validity and overidentification in the LATE framework. In \\emph{Advances in economics and econometrics: Tenth world congress}, Vol.~3, eds. Manuel \\hl{Arellano Daron} Acemoglu, and Eddie \\hl{Deke}, 401--\\hl{36}. Cambridge: Cambridge University Press.\n\\item Deaton, Angus. 2009. Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development. \\emph{Proceedings of the British Academy, 2008 Lectures} 162:123--60.\n\\item Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til \\hl{Strmer}, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. \\emph{American Journal of Epidemiology} 173(7):761--7.\n\\item Joffe, Marshall M., and Colleen Brensinger. 2003. Weighting in instrumental variables and G-estimation. \\emph{Statistics in Medicine} 22(\\hl{1}):1285--303.\n\\item Putnam, Robert \\hl{C.} 2000. \\emph{Bowling alone: The collapse and \\hl{renewal} of American Community}. New York: Simon and Schuster.\n\\item Verba, Sidney, Kay Lehman Schlozman, and Henry E. Brady. 1995. \\emph{\\hl{Voluntarism in American Politics}}. Cambridge, MA: Harvard University Press.\n\\end{itemize}", "corrected": "\\emph{Section~1, p.~492.} Both papers argue that the LATE is not usually the parameter of interest, as ``we are unlikely to learn much about the \\hl{processes} at work'' by estimating the LATE (Deaton 2009, \\hl{130}) [\\ldots]\n\nAn alternative is to cite and list the 2010 version of the essay, \\emph{Journal of Economic Literature} 48(2):424--55, and to keep the page 430.\n\n\\emph{Reference list.}\n\\begin{itemize}\n\\item Angrist, Joshua D., and Ivan Fernandez-Val. 2013. ExtrapoLATE-ing: External validity and overidentification in the LATE framework. In \\emph{Advances in economics and econometrics: Tenth world congress}, Vol.~3, eds. Manuel \\hl{Arellano, Daron} Acemoglu, and Eddie \\hl{Dekel}, 401--\\hl{34}. Cambridge: Cambridge University Press.\n\\item Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til \\hl{Stürmer}, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. \\emph{American Journal of Epidemiology} 173(7):761--7.\n\\item Joffe, Marshall M., and Colleen Brensinger. 2003. Weighting in instrumental variables and G-estimation. \\emph{Statistics in Medicine} 22(\\hl{8}):1285--303.\n\\item Putnam, Robert \\hl{D.} 2000. \\emph{Bowling alone: The collapse and \\hl{revival} of American Community}. New York: Simon and Schuster.\n\\item Verba, Sidney, Kay Lehman Schlozman, and Henry E. Brady. 1995. \\emph{\\hl{Voice and Equality: Civic} Voluntarism in American Politics}. Cambridge, MA: Harvard University Press.\n\\end{itemize}", "evidence": "The evidence is a check of the page against the cited works and the records of their publishers. In both editions of Deaton's essay the sentence reads ``We are unlikely to learn much about the processes at work''. It is on p.~430 of the 2010 version (\\emph{Journal of Economic Literature} 48(2), 424--455) and on p.~130 of the 2009 version in the \\emph{Proceedings of the British Academy}. The publisher's record of the chapter by Angrist and Fernandez-Val names the editors Daron Acemoglu, Manuel Arellano and Eddie Dekel, with the pages 401--434. In the publisher's record of the article by Joffe and Brensinger, the issue is Volume~22, Issue~8. Bibliographic records give the names Til Stürmer and Robert D. Putnam, and the titles \\emph{Bowling Alone: The Collapse and Revival of American Community} and \\emph{Voice and Equality: Civic Voluntarism in American Politics}.\n\nOne point is not settled. The same page of the paper cites ``Imbens (2009, 403)'', and the reference list has an NBER working paper for this citation. Page 403 lies in the page range 399--423 of the version of that comment in the \\emph{Journal of Economic Literature} in 2010. I could not confirm the page of the quoted words in either edition.", "problem": "The quotation changes a word of the original, and its page number does not belong to the edition in the reference list, whose pages are 123 to 160. Page 430 is the page of the passage in the version of the essay that appeared in the \\emph{Journal of Economic Literature} in 2010. In the reference list, two editors' names run together and a third is misspelled, one chapter ends at p.~434 and not at p.~436, an author's name has lost a letter, an issue number and a middle initial are wrong, a subtitle has ``renewal'' for ``revival'', and one title has lost its first words. Each cited work can still be identified.", "what_survives": "The argument of the paper does not depend on any of these details, and the sense of the quotation is unchanged. No correction is proposed for the citation of Imbens until the edition is checked." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E020", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of five displayed lines. The first reads X1 sub i, X2 sub i, X3 sub i distributed as U(-1, 1). The second reads a, b, c, d, e, f, g, h distributed as U(-2, 2) and has a red outline. The third reads AC sub i = I[2 + a + b X1 sub i + c X2 sub i + d X3 sub i + N(0, 3)], with a red outline around N(0, 3). The fourth reads A sub i = I[-2 + e + f X1 sub i + g X2 sub i + h X3 sub i + N(0, 3)] AC sub i, with a red outline around N(0, 3). The fifth reads C sub i = AC sub i - A sub i.", "caption": "Supplement, p. A1 (supplement PDF p. 1), first lines of the baseline design. The outlines mark the two terms $\\mathbf{N}(0,3)$, whose scale is not stated, and the line for the coefficients, which does not say when they are drawn.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-A001.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T19", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T19" } ], "headline": "In the simulation design, $\\mathbf{N}(0,3)$ is a normal draw with standard deviation $3$, and the coefficients $a,\\dots,h$ are drawn once for each simulated dataset; the supplement states neither convention.", "id": "P029-A001", "location": "Supplement, p. A1 (supplement PDF p. 1), “Baseline Data Generating Process”: the line for $a,\\dots,h$ and the two terms $\\mathbf{N}(0,3)$", "pdf_page": 1, "printed_page": "A1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P029-T19" ], "text": { "as_printed": "\\[\n\\begin{aligned}\n& X1_i, X2_i, X3_i \\sim \\mathbf{U}(-1,1)\\\\\n& \\hl{a, b, c, d, e, f, g, h \\sim \\mathbf{U}(-2,2)}\\\\\n& AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\hl{\\mathbf{N}(0,3)}]\\\\\n& A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\hl{\\mathbf{N}(0,3)}]AC_i\n\\end{aligned}\n\\]", "corrected": "\\[\n\\begin{aligned}\n& X1_i, X2_i, X3_i \\sim \\mathbf{U}(-1,1)\\\\\n& a, b, c, d, e, f, g, h \\sim \\mathbf{U}(-2,2)\\\\\n& AC_i = \\mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \\mathbf{N}(0,3)]\\\\\n& A_i = \\mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \\mathbf{N}(0,3)]AC_i\n\\end{aligned}\n\\]\n\\hl{Here $\\mathbf{N}(0,3)$ denotes a normal draw with mean zero and standard deviation $3$. The scalar coefficients $a,\\dots,h$ are drawn once for each simulated dataset and are shared by all of its observations.}\n\nAn alternative for the first sentence is to write $\\mathbf{N}(0,3^2)$ and to state that the second argument is the variance.", "evidence": "The evidence is a check of the page against the replication files. In \\texttt{Simulations Alpha 6-20.R}, the function \\texttt{dgp} generates one simulated dataset. It draws each disturbance with \\texttt{rnorm(N,0,3)}, where the third argument of \\texttt{rnorm} is the standard deviation. Each coefficient is drawn with \\texttt{runif(1,-2,2)}, once for each simulated dataset, and the eight numbers are passed to \\texttt{dgp}.", "problem": "The supplement does not say whether the second argument of $\\mathbf{N}(0,3)$ is a variance or a standard deviation. A common convention in mathematics takes it as the variance. The simulation code draws the disturbances with standard deviation $3$. The choice changes the compliance rates of the design: at zero covariates and $a = 0$, the probability that $AC_i = 1$ is $\\Phi(2/3) = 0.7475$ with standard deviation $3$ and $\\Phi(2/\\sqrt{3}) = 0.8759$ with variance $3$. The supplement also does not say when the coefficients $a,\\dots,h$ are drawn. They carry no index $i$, which shows that all units share them. In the code they are drawn once for each simulated dataset. Someone who implements the design from the supplement alone has to guess both conventions. No reported number is shown to be wrong.", "what_survives": "The simulation design in the code and all reported simulation results are not shown to be affected. The notation of the supplement already indicates that the coefficients are common to all units. The correction adds two sentences to the description of the design." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-A001", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of two lines. The text reads: A1 and A2. We subclassify each of these plots to DGPs with low (mean(P sub C) < 0.33), medium (0.33 is at most mean(P sub C) < 0.50) and high (0.50 is at most mean(P sub C)) compliance. Red outlines surround the three occurrences of mean(P sub C).", "caption": "Supplement, p. A3 (supplement PDF p. 3), the two lines above Figure A1. The outlines mark the three occurrences of the label $\\operatorname{mean}(P_C)$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-A002.png" } ], "formal_release": null, "formal_statements": [], "headline": "The compliance groups of Figures A1 and A2 are formed from the realized fraction of compliers in each simulated dataset, $\\bar C = N^{-1}\\sum_i C_i$, which the supplement labels $\\operatorname{mean}(P_C)$.", "id": "P029-A002", "location": "Supplement, p. A3 (supplement PDF p. 3), sentence above Figure A1", "pdf_page": 3, "printed_page": "A3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "We subclassify each of these plots to DGPs with low ($\\hl{\\operatorname{mean}(P_C)} < 0.33$), medium ($0.33 \\le \\hl{\\operatorname{mean}(P_C)} < 0.50$) and high ($0.50 \\le \\hl{\\operatorname{mean}(P_C)}$) compliance.", "corrected": "We subclassify each of these plots to DGPs with low ($\\hl{\\bar C} < 0.33$), medium ($0.33 \\le \\hl{\\bar C} < 0.50$) and high ($0.50 \\le \\hl{\\bar C}$) compliance\\hl{, where $\\bar C = N^{-1}\\sum_i C_i$ is the realized fraction of compliers in each simulated dataset}.", "intended_meaning": "The sentence sorts the simulated datasets by their overall compliance, with the cut-offs $0.33$ and $0.50$, and the code does the same. In the article, $P_{Ci}$ is the probability that unit $i$ is a complier given its covariates (p.~496), so the label suggests an average of probabilities. The code stores, for each dataset, the mean of the indicators $C_i = AC_i - A_i$ that unit $i$ is a complier, and the display script sorts on this mean. The realized fraction and the mean probability have the same conditional expectation, but they differ in a finite dataset. In one simulated dataset of $100$ units, with all covariates and coefficients equal to zero, the realized fraction is $0.53$ and the probability is $\\Phi(2/3)(1-\\Phi(-2/3)) = 0.5588$. The thresholds and the meaning of low, medium and high compliance are as printed, and no figure changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-A002", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of three rows of Table 1, each with a coefficient and an interval in two columns. Age: 0.2 with interval (0.2, 0.2), and 0.2 with interval (0.1, 0.2). Age missing: 12.7 with interval (-16.5, 44.1), and 17.7 with interval (-15.9, 50.3). Democrat: 2.6 with interval (0.6, 4.7), and 2.6 with interval (0.7, 04.8). Red outlines surround the label Age missing and the two coefficients 12.7 and 17.7.", "caption": "Table 1, rows “Age”, “Age missing” and “Democrat”, p. 501 (PDF p. 10). The two columns of numbers are LATE (2SLS), (1), and ATE (ICSW), (2). The outlines mark the label “Age missing” and its two coefficients, which the script as deposited does not reproduce.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E021.png" }, { "alt": "Image of six lines of R code in a monospaced face, under the heading greengerbernickerson.R, lines 33-36 and 45 (replication files). Line 33: age[age>101] <- mean(age[age <= 101],na.rm=TRUE). Line 34: age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE). Line 35: ageM <- rep(0,N). Line 36: ageM[age==mean(age[age <= 101],na.rm=TRUE)] <- 1. Line 45, after an ellipsis: covs <- cbind(makeind(data.frame(factor(city))),famsize,white,black,voted00,voted99a,primary,age,ageM,dem,rep,ind)[,2:17]. A red outline surrounds the subscript of line 36, from the opening bracket before age==mean to the closing bracket.", "caption": "Replication script greengerbernickerson.R, lines 33–36 and 45, typeset from the replication files (a script has no published page). The outline marks the comparison in line 36.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-E021-code.png" } ], "formal_release": null, "formal_statements": [], "headline": "The replication script rebuilds the age-missing indicator by testing whether an imputed age equals a recomputed mean, which on a current system flags 1 of the 2,488 records with a missing or impossible age and gives “Age missing” coefficients near 97 where Table 1 prints 12.7 and 17.7; the printed point estimates are reproduced when the indicator is set before the imputation.", "id": "P029-E021", "location": "Replication script greengerbernickerson.R, lines 33–36 and 45; the paper’s counterparts are Table 1, row “Age missing”, p. 501 (PDF p. 10), and supplement Table A7, column “Age M.” and row “Age Miss.”, p. A11 (supplement PDF p. 11)", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Script, lines 33--36 and 45 (line 45 is one line of code, broken here).}\n\\[\n\\begin{aligned}\n&\\texttt{age[age>101] <- mean(age[age <= 101],na.rm=TRUE)}\\\\\n&\\texttt{age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)}\\\\\n&\\texttt{ageM <- rep(0,N)}\\\\\n&\\texttt{ageM[}\\hl{\\texttt{age==mean(age[age <= 101],na.rm=TRUE)}}\\texttt{] <- 1}\\\\\n&\\texttt{covs <- cbind(makeind(data.frame(factor(city))),famsize,white,black,voted00,}\\\\\n&\\qquad\\texttt{voted99a,primary,age,ageM,dem,rep,ind)[,2:17]}\n\\end{aligned}\n\\]\n\n\\emph{Table~1, p.~501.}\n\\begin{audittable}{lll}{Table~1, row ``Age missing'', with the bootstrap 95\\% confidence intervals.}\n & LATE (2SLS), (1) & ATE (ICSW), (2) \\\\\nAge missing & 12.7 $(-16.5, 44.1)$ & 17.7 $(-15.9, 50.3)$ \\\\\n\\end{audittable}\n\\emph{Supplement Table~A7, p.~A11.} Comp. Score, Age M.: $-0.55$.", "corrected": "In \\texttt{greengerbernickerson.R}, the indicator should be set before age is imputed and kept in \\texttt{covs} (line 45), and lines 35--36 should be deleted:\n\\[\n\\begin{aligned}\n&\\hl{\\texttt{ageM <- as.numeric(is.na(age) | age > 101)}}\\\\\n&\\texttt{age[age>101] <- mean(age[age <= 101],na.rm=TRUE)}\\\\\n&\\texttt{age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)}\n\\end{aligned}\n\\]\nOn R 4.4.2 for arm64, the comparison in the script as deposited flags 1 of the 2,488 records with a missing or impossible age. The indicator set before the imputation flags all of them, and it reproduces the printed point estimates of Table~1 and the corresponding entries of Table~A7.", "evidence": "The evidence is a recalculation from the replication data, in which I compare the exact rational representations of the three floating-point means. The data hold 18,933 records, of which 2,487 have a missing age and one has an age above 101. The 16,445 valid ages have mean $742242/16445$. I evaluated the script's expressions for the compliance score and for the two estimators twice, once with the indicator of lines 35--36 and once with the indicator set to 1 for the 2,488 records before the imputation.\n\\begin{audittable}{lrrr}{Results from the replication data under the two constructions of the age-missing indicator. Coefficients are in percentage points.}\nQuantity & Script as deposited & Indicator set before imputation & Printed \\\\\nRecords flagged & 1 & 2,488 & \\\\\n2SLS, Age missing & 97.35 & 12.65 & 12.7 \\\\\nICSW, Age missing & 96.60 & 17.71 & 17.7 \\\\\n2SLS, Campaign contact & 5.64 & 5.61 & 5.6 \\\\\nICSW, Campaign contact & 6.71 & 6.67 & 6.7 \\\\\nCorrelation of score with indicator & $-0.01$ & $-0.55$ & $-0.55$ \\\\\n\\end{audittable}\nWith the indicator set before the imputation, I find that all 24 point estimates of Table~1 agree with the printed values to one decimal. The 17 entries of the column ``Age M.'' and the 12 entries of the row ``Age Miss.'' of Table~A7 agree to two decimals. With the script as deposited, the ICSW coefficient on ``Voted in 1999 general election'' also changes, from 27.2 to 27.3.\n\nFor the bootstrap, I took one draw that the resampling of line 94 can produce, in which the flagged record is replaced by another record of its stratum. With the script as deposited, the coefficient on the indicator is missing in both models and both quantile calls stop with an error. With the indicator set before the imputation, the same draw runs. I did not recompute the 2,500-draw confidence intervals, and I do not know how or on what system the published run was made.", "problem": "The note to Table~1 lists a dummy variable for missing data on age among the covariates. The script first replaces the ages above 101, and then the missing ages, by the mean of the valid ages (lines 33 and 34). Only afterwards does it set the indicator to 1 for every record whose age equals that mean, which line 36 computes a third time. In exact arithmetic the three means agree, since adding copies of the mean to a sample does not change its mean. In floating-point arithmetic they need not agree to the last bit, and line 36 tests exact equality.\n\nOn R 4.4.2 for arm64 under macOS, the first and third means are equal and the second is smaller by $3/2^{47}$. The one age above 101 (the value 2001) was replaced by the first mean and passes the test. The 2,487 missing ages were replaced by the second mean and fail it. The indicator is then 1 for one record, where it should be 1 for 2,488. With this indicator the script returns ``Age missing'' coefficients of 97.3 and 96.6 percentage points, against the printed 12.7 and 17.7. The correlation between the compliance score and the indicator is $-0.01$, against the $-0.55$ of Table~A7. The bootstrap of the script can also stop. A draw that omits the one flagged record leaves a column of zeros, the coefficient on that column is missing, and the quantile calls of the script do not remove missing values.\n\nThe test is unreliable in exact arithmetic too, in the other direction. For the ages $20$, $30$, $40$ and one missing value, the mean is $30$ at every stage, and the test flags the observed age $30$ along with the imputed one. I checked one system, and the number of flagged records may differ on another.", "what_survives": "Table~1 and Table~A7 are not shown to be wrong: the indicator set before the imputation reproduces every printed point estimate of Table~1 and the entries of Table~A7 that involve the indicator. Under both constructions the effects of contact round to 5.6 and 6.7 percentage points, and the ATE is 19\\% larger than the LATE. In this script the compliance-score fit, the adjusted coefficients, the correlation matrix, the bootstrap and the permutation diagnostic all use the indicator. The confidence intervals and the diagnostic were not recomputed. The theory, the simulations and the application to Albertson and Lawrence (2009) do not use this script." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-E021", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of the four lines of footnote 1. The text reads: Other examples include: Hidalgo et al. (2010) show that the effects of economic shocks on conflict are larger in regions with high levels of preexisting inequality; Altonji and Dunn (1996) demonstrate that the returns to education are larger for students in high-quality schools; Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are larger for high-propensity voters. A red outline surrounds the last clause, from Arceneaux and Nickerson (2009) to the end.", "caption": "Footnote 1, p. 494 (PDF p. 3). The outline marks the clause on Arceneaux and Nickerson (2009).", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N001.png" } ], "formal_release": null, "formal_statements": [], "headline": "Arceneaux and Nickerson (2009) find larger effects of canvassing for high-propensity voters in low-salience elections only; in high-salience elections they find the opposite.", "id": "P029-N001", "location": "Footnote 1, p. 494 (PDF p. 3), last clause", "pdf_page": 3, "printed_page": "494", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Other examples include: Hidalgo et al. (2010) show that the effects of economic shocks on conflict are larger in regions with high levels of preexisting inequality; Altonji and Dunn (1996) demonstrate that the returns to education are larger for students in high-quality schools; Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are \\hl{larger for high-propensity voters}.", "corrected": "[\\ldots] Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are larger for high-propensity voters \\hl{in low-salience elections}.\n\nThe footnote may add that the pattern is reversed in high-salience elections.", "evidence": "The evidence is a check of the footnote against the cited article (\\emph{American Journal of Political Science} 53(1):1--16). In its discussion of Figure~4, the article has the phrase ``boost turnout among high-propensity voters but not low-propensity voters in low-salience elections'' on p.~10, and it continues on p.~11: ``The opposite takes place in high-salience elections.'' Its Table~3 (p.~12) reports effects of 3.3\\%, 11.7\\% and 12.3\\% for low-, medium- and high-propensity voters in low-salience elections, and of 12.4\\%, 11.6\\% and 3.2\\% in high-salience elections.", "problem": "The clause reads as a general statement about the direction of the heterogeneity. In the cited article the direction depends on the salience of the election. The larger effect for high-propensity voters is a finding of that article for low-salience elections, and the footnote omits that the pattern reverses in high-salience elections.", "what_survives": "The footnote lists examples of heterogeneous treatment effects, and the cited article is such an example. Nothing in the paper depends on the direction of the heterogeneity. The corrected clause is narrower than the printed one." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N001", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of seven lines of text. The text reads: We begin by estimating compliance scores for the sample. As recommended above, we use all observed pretreatment covariates, which include the covariates listed in Table 1 and dummy variables for the six cities in the sample. Although the set of available covariates may or may not satisfy the ignorability assumption, we will still recover an estimate for a population that has a covariate profile that matches that of the full population. We see that compliers are more likely to have voted previously, be members of a major party, and identify as white (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates). We now apply ICSW; the crop ends in the middle of this sentence. A red outline surrounds the words be members of a major party, and identify as white.", "caption": "Section 4.1, last paragraph of p. 501 (PDF p. 10). The outline marks the description of the compliers by party and race.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N004.png" }, { "alt": "Crop of the top of a correlation table with the column headings C. Score, Fam. Size, White, Black, Voted 00, Voted 99, Voted Pri., Age, Age M., Dem, Rep and Ind. Row Comp. Score: 1.00, 0.37, 0.61, 0.32, 0.40, 0.43, 0.25, 0.23, -0.55, 0.38, 0.31, -0.47. Row Columbus: -0.55, -0.17, -0.20, -0.07, -0.15, -0.05, -0.19, -0.00, 1.00, -0.11, -0.03, 0.79. Row Detroit: 0.05, 0.03, -0.30, -0.11, -0.00, -0.24, 0.23, 0.16, -0.23, -0.28, -0.19, -0.18. Row Minneapolis: -0.45, -0.24, -0.21, -0.08, -0.06, -0.14, -0.01, -0.14, -0.16, -0.20, -0.14, -0.13. Row Raleigh: 0.75, 0.26, 0.89, 0.33, 0.13, 0.17, -0.00, 0.02, -0.22, 0.46, 0.43, -0.18. Row St. Paul: 0.07, 0.15, -0.18, -0.07, 0.16, 0.30, 0.06, -0.08, -0.14, -0.17, -0.12, -0.11. Red outlines surround three entries of the row Comp. Score: 0.61 under White, 0.38 under Dem and 0.31 under Rep.", "caption": "Supplement Table A7, p. A11 (supplement PDF p. 11): the column headings and the first six rows. The table is printed sideways and is turned upright here. The outlines mark the correlations of the compliance score with White, Dem and Rep. In the row “Raleigh”, the entry under “C. Score” is 0.75 and the entry under “White” is 0.89.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N004-A7.png" }, { "alt": "Figure with two panels of horizontal bars, titled: The White indicator is 1 only in Raleigh, the city with the highest scores; within Raleigh, voters recorded as White have the lowest mean score. Upper panel, mean estimated compliance score. Five grey bars are cities where race is not recorded and the White indicator is 0 for every voter: Columbus (n = 2,478) 0.147, Minneapolis (n = 2,827) 0.185, Bridgeport (n = 1,806) 0.279, Detroit (n = 4,954) 0.309, St. Paul (n = 2,208) 0.320. One orange bar is Raleigh voters recorded as White, indicator 1 (n = 3,884): 0.432. Two blue bars are Raleigh voters not recorded as White, indicator 0: recorded Black (n = 646) 0.487, and other or unknown (n = 130) 0.529. Lower panel, correlation of the estimated compliance score with the White indicator: a solid dark bar for all six cities, the published value of Table A7, at plus 0.61, and a hatched bar for the correlation within Raleigh from this check, at minus 0.41.", "caption": "Mean estimated compliance score by city and, within Raleigh, by recorded race (upper panel), and the correlation between the score and the White indicator over the six cities, as in Table A7, and within Raleigh (lower panel). The values are from the recalculation with the replication data.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N004-scores-by-group.png" } ], "formal_release": null, "formal_statements": [ { "id": "P029-T9", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T9" }, { "id": "P029-T10", "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T10" } ], "headline": "The data do not support the statement that compliers in the canvassing study are more likely to be members of a major party and to identify as white: the race and party indicators are zero wherever the label is not recorded, race is recorded only in Raleigh, and within Raleigh the voters recorded as White have lower estimated compliance scores than the others.", "id": "P029-N004", "location": "Section 4.1, last paragraph of p. 501 (PDF p. 10), sentence beginning “We see that compliers”; supplement Table A7, p. A11 (supplement PDF p. 11)", "pdf_page": 10, "printed_page": "501", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P029-T10", "P029-T9" ], "text": { "as_printed": "We see that compliers are more likely to have voted previously, \\hl{be members of a major party, and identify as white} (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates).\n\n\\begin{audittable}{lrrrrrrrrr}{Supplement Table~A7, rows ``Comp. Score'' and ``Raleigh'', nine of the twelve columns.}\n & C. Score & White & Black & Voted 00 & Voted 99 & Voted Pri. & Dem & Rep & Ind \\\\\nComp. Score & 1.00 & 0.61 & 0.32 & 0.40 & 0.43 & 0.25 & 0.38 & 0.31 & $-0.47$ \\\\\nRaleigh & 0.75 & 0.89 & 0.33 & 0.13 & 0.17 & $-0.00$ & 0.46 & 0.43 & $-0.18$ \\\\\n\\end{audittable}", "corrected": "\\emph{Section~4.1, p.~501.} Estimated compliance scores are positively associated with prior voting. The pooled correlations with \\hl{recorded} White, \\hl{Dem and Rep} indicators \\hl{also reflect differences between cities in contact rates and covariate availability}. Within Raleigh, the only city supplying race labels, voters recorded as White have \\hl{lower} mean estimated compliance scores than the remaining records (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates).\n\n\\emph{Note to add to supplement Table~A7.} Blank race and party labels are coded zero in the respective indicators; race is unavailable outside Raleigh and party is unavailable throughout Detroit, Minneapolis, and St.~Paul, with additional blank party records in Columbus. Age, primary-vote history, and 1999-vote history are unavailable in particular cities and are imputed. The correlations in the table do not change.", "evidence": "The evidence is a recalculation from the replication data and an exact counterexample to the inference from recorded labels. The recalculation fits the compliance model of the script. It reproduces the correlations of the compliance score in Table~A7 and the twelve point estimates in each column of Table~1 at the printed precision, with the age-missing indicator set before the imputation (see \\fid{P029-E021}). The first table lists the blank labels and the mean score in each city.\n\\begin{audittable}{lrrrr}{Voters, blank race and party labels, and mean estimated compliance score, by city.}\nCity & Voters & Race blank & Party blank & Mean score \\\\\nBridgeport & 1,806 & 1,806 & 0 & 0.279 \\\\\nColumbus & 2,478 & 2,478 & 489 & 0.147 \\\\\nDetroit & 4,954 & 4,954 & 4,954 & 0.309 \\\\\nMinneapolis & 2,827 & 2,827 & 2,827 & 0.185 \\\\\nRaleigh & 4,660 & 0 & 0 & 0.442 \\\\\nSt.~Paul & 2,208 & 2,208 & 2,208 & 0.320 \\\\\n\\end{audittable}\nIn Raleigh, 21 of the 4,660 race labels are unknown or unspecified. The covariance between the score $s$ and an indicator $A$ splits exactly into a part within cities and a part between city means,\n\\[\n\\operatorname{Cov}(s, A) = \\mathbb{E}\\bigl[(s - s_G)(A - A_G)\\bigr] + \\mathbb{E}\\bigl[(s_G - \\bar s)(A_G - \\bar A)\\bigr] ,\n\\]\nwhere $s_G$ and $A_G$ are the means in the voter's city and $\\bar s$ and $\\bar A$ are the overall means.\n\\begin{audittable}{lrrr}{The pooled covariance of the estimated compliance score with each indicator, and its two parts.}\nIndicator & Pooled & Between cities & Within cities \\\\\nWhite & $0.0269$ & $0.0290$ & $-0.0021$ \\\\\nDem & $0.0158$ & $0.0145$ & $0.0013$ \\\\\nRep & $0.0099$ & $0.0093$ & $0.0007$ \\\\\nInd & $-0.0142$ & $-0.0133$ & $-0.0009$ \\\\\n\\end{audittable}\nFor White the part within cities is negative, and all of it comes from Raleigh.\n\nFor the counterexample, I complete the unobserved quantities in two ways that agree with every recorded label and every observed variable. In both, a voter assigned to canvassing is a complier if contacted, and a control voter, whose status is not observed, is not a complier. I assign the 21 unknown or unspecified Raleigh race labels as not White in both completions. There are 2,551 compliers. In the first completion every voter with a blank race label is not White. Then 3,884 voters are White, 602 of them are compliers, and the compliance rate of the White voters exceeds that of the others by $744791/29225158 \\approx 0.025$. In the second completion every voter with a blank race label is White. Then 18,157 voters are White, 2,391 of them are compliers, and the difference is $-131213/1761229 \\approx -0.075$. With the estimated scores held fixed, the correlation between the score and race is $0.61$ in the first completion and $-0.37$ in the second. For party, assigning every blank label as outside the two major parties gives a positive association; assigning blank labels as major-party only for noncompliers gives a negative association, with recorded LIB affiliations outside the two major parties in both cases. Both completions agree with the data. This implies that the data do not determine the sign, and they do not show which completion is closer to the truth.", "problem": "The sentence reads a description of the compliers off the correlations in Table~A7 between the estimated compliance score and the covariates: 0.61 for White, 0.38 for Dem and 0.31 for Rep. These numbers are right. They do not measure race or party membership in the six cities. The replication script sets each indicator to 1 when the label of the voter matches and to 0 otherwise, and a blank label counts as 0. Race is blank for all 14,273 voters outside Raleigh. The White indicator is then 1 only for the 3,884 Raleigh voters with the label W, and Raleigh has the highest mean estimated score of the six cities, 0.44 against 0.15 to 0.32. The pooled correlation mostly separates Raleigh from the other cities. Table~A7 itself shows that the Raleigh indicator has correlation 0.89 with White and 0.75 with the score, and that the Black indicator is also positively correlated with the score (0.32).\n\nWithin Raleigh, the one city where race is recorded, the correlation between the score and the White indicator is $-0.41$. The mean estimated score is 0.432 for the voters recorded as White, 0.487 for those recorded as Black and 0.529 for the 130 voters with another or an unknown label. Among the Raleigh voters assigned to canvassing, 602 of the 1,370 recorded as White were contacted (43.9\\%), against 160 of the 315 others (50.8\\%).\n\nParty is blank for every voter in Detroit, Minneapolis and St.~Paul and for 489 voters in Columbus. More than 90\\% of the pooled covariance between the score and each party indicator is covariance between city means. For party the sign does not reverse within Raleigh, where the correlation of the score with the combined Dem/Rep indicator (which includes the code LIB in Dem) is 0.18, against 0.53 in the pooled data.\n\nI conclude that the correlations cited do not support the description of the compliers by race and party. I do not conclude that the description is false. The association between race or party and compliance over the six cities depends on the labels that were not recorded, and these data cannot determine it. The part of the sentence about previous voting is supported.", "what_survives": "The correlations of Table~A7 are numerically right. The statement that compliers are more likely to have voted previously stands: the score is positively correlated with turnout in 2000 and in 1999 in the pooled data and within cities. The sentence is not an input to any calculation. A change of wording leaves the point estimates of Table~1, the comparison of 5.6 with 6.7 percentage points and the bootstrap as they are. I did not recompute the intervals. The method, the simulations and the application to Albertson and Lawrence (2009) do not depend on the sentence. What changes is the description of the compliers, which must be limited to labels that were recorded." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N004", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of footnote 6, seven lines. One red box marks 'but Winsorizing may sometimes increase finite sample bias.' A second red box, on the next line, marks 'because c tends to 0 as n tends to infinity.'", "caption": "Footnote 6, p. 498 (PDF p. 7). The upper mark is the article’s caution that Winsorizing can add finite-sample bias; the lower mark is the false limit.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N006.png" }, { "alt": "Crop of the head of Table 2, '2SLS and ICSW estimates for Albertson and Lawrence (2009)', with columns Knowledge LATE (2SLS), Knowledge ATE (ICSW), Opinion LATE (2SLS) and Opinion ATE (ICSW). In the row 'Watching debate' the entries are 0.28 (-0.04, 0.59), 0.40 (0.01, 1.26), -0.07 (-0.25, 0.13) and -0.05 (-0.31, 0.24). A red box marks 0.40 (0.01, 1.26).", "caption": "Table 2, p. 503 (PDF p. 12), first rows. The marked entry is the ICSW estimate of the effect of watching the debate on knowledge, with its bootstrap interval.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N006-table2.png" } ], "formal_release": null, "formal_statements": [], "headline": "The knowledge estimate of Table 2 depends on the Winsorization constant, and I reproduce the printed point estimates; the error is the statement of footnote 6 that $c\\to0$, the same error that P029-E006 corrects.", "id": "P029-N006", "location": "Footnote 6, p. 498 (PDF p. 7); Table 2, row “Watching debate”, column (2), and the paragraph that discusses it, pp. 502–503 (PDF pp. 11–12)", "pdf_page": 7, "printed_page": "498", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Footnote 6: Winsorizing is performed because very low probabilities can introduce instability in the estimates resulting from IPW (Elliott 2009), but Winsorizing may sometimes increase finite sample bias. Asymptotically, assuming that $\\alpha$ remains fixed, the Winsorization has no impact on estimation, \\hl{because $c\\to0$ as $n\\to\\infty$}.\n\nTable 2, row ``Watching debate'': $0.28$ $(-0.04, 0.59)$ in column (1), Knowledge, LATE (2SLS); $0.40$ $(0.01, 1.26)$ in column (2), Knowledge, ATE (ICSW).\n\nText: Although Albertson and Lawrence (2009) find that compliers are 0.28 points more informed after viewing the program, we find that viewers in the entire subject population are 0.40 points more informed after viewing. We therefore estimate that the ATE is approximately 45\\% larger than the LATE.", "corrected": "Footnote 6, in place of the sentence beginning ``Asymptotically'': \\hl{For fixed $\\alpha > 0$, the quantile level $1/n^{\\alpha}$ tends to zero, so the fraction of scores changed vanishes. Under Assumption 3 and regularity conditions ensuring bounded estimated inverse scores with probability tending to one, integrable outcomes and a nonzero limiting weighted first stage, Winsorization does not change the probability limit of the displayed ICSW estimator.}\n\n\\fid{P029-E006} gives the same correction with the regularity conditions written out. Table 2 and the comparison of $45\\%$ are unchanged.", "evidence": "The evidence is a recalculation from the replication data and an exact counterexample to the limit. The recalculation fits the scores once and varies $\\alpha$. It reproduces all 40 point estimates of Table 2 to two decimals. The excess of the ICSW estimate over the 2SLS estimate is $44.9\\%$ with unrounded coefficients. The bootstrap intervals were not recomputed.\n\\begin{audittable}{rrrrr}{Knowledge estimate for the debate application as $\\alpha$ varies ($n=507$; scores fitted once).}\n$\\alpha$ & Cut-off $c$ & Scores raised & Estimate & Excess over 2SLS \\\\\n\\hline\n$0$ & $0.717$ & $506$ & $0.276$ & $0.0\\%$ \\\\\n$0.275$ & $0.290$ & $92$ & $0.400$ & $44.9\\%$ \\\\\n$0.5$ & $0.162$ & $23$ & $0.505$ & $83.2\\%$ \\\\\n$1$ & $0.065$ & $1$ & $0.612$ & $122.0\\%$ \\\\\n$\\infty$ & $0.063$ & $0$ & $0.612$ & $121.9\\%$ \\\\\n\\end{audittable}\nFor the limit, I consider one covariate profile and one-sided noncompliance with $D_0=0$ and $D_1=C$, where $\\Pr(C=1)=1/2$, outcomes $Y_0=0$ and $Y_1=1$, and $Z\\sim\\mathrm{Bernoulli}(1/2)$ independent of $C$. The article's assumptions hold, with $\\epsilon=1/4$ in Assumption 3. Every true score is $1/2$, and $c=1/2$ for every $n$ with the exact scores. The fitted score is the proportion treated among the assigned units, which converges almost surely to $1/2$, and $c$ converges to $1/2$ with it. No score is changed, and the ICSW ratio equals the ATE of $1$ whenever its empirical first stage is nonzero.", "problem": "The point raised is that the knowledge estimate depends on the constant $\\alpha$ and that the article does not report this dependence. With $n=507$ the rule in footnote 6 raises the 92 lowest scores, $18.1\\%$ of the sample, to $0.290$. The estimate is $0.40$ at $\\alpha=0.275$ and $0.61$ without Winsorization, and it equals the 2SLS estimate at $\\alpha=0$. This dependence is not an error in Table 2. The estimates are computed by the rule that the article states, and the footnote says that Winsorizing can add finite-sample bias. The article does not claim that the estimate is free of the choice of $\\alpha$. The error is in the same footnote: the cut-off $c$ does not tend to zero. This is the statement corrected in \\fid{P029-E006}.", "what_survives": "The Winsorization rule and its implementation, the 40 point estimates of Table 2 and the comparison of $45\\%$ stand. The conclusion that Winsorization does not change the limit of the estimator holds under positivity and regularity conditions. The true ATE, the size of the bias from Winsorization and the best $\\alpha$ for these data are not established." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N006", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of footnote 7, five lines. Red boxes mark the sentence 'In our own simulation studies, assuming alpha = 0.275, we find that the bootstrap has conservative coverage for ICSW in small samples, and proper coverage in larger samples.'", "caption": "Footnote 7, p. 498 (PDF p. 7). The marked sentence summarizes the bootstrap coverage found in the supplement without naming the quantity covered.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N009.png" }, { "alt": "Crop of the section heading and six lines of text. A red box marks 'We present tables of RMSE vs' at the end of the first line, which continues 'n in 500, 1000, 2500, 5000, 10000, 25000 under the 5 different models, including misspecification.'", "caption": "Supplement, p. A5 (PDF p. 5), section “(2) Properties of ICSW with and without Misspecification”. The marked words introduce the RMSE tables; the target of the RMSE is not stated.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N009-A5.png" }, { "alt": "Crop of the section heading, three lines of text and Table A6. A red box marks 'In Table A6, we present percent coverage with the bootstrap'. Table A6 has rows n = 500, 2500 and 5000, with 90 percent interval coverage 0.94, 0.91 and 0.92 and 95 percent interval coverage 0.98, 0.94 and 0.95.", "caption": "Supplement, p. A9 (PDF p. 9), section “(3) Bootstrap Confidence Interval Coverage”, with Table A6. The marked words introduce the coverage; the quantity covered is not stated.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N009-A9.png" } ], "formal_release": null, "formal_statements": [], "headline": "The heterogeneous-effect RMSEs and bootstrap coverage in the supplement are measured against the sample average effect of each simulated data set, $n^{-1}\\sum_i\\tau_i$, and not against the population ATE of 5; the supplement and footnote 7 should say so.", "id": "P029-N009", "location": "Supplement, pp. A5 and A9 (PDF pp. 5 and 9), with Tables A1, A2 and A4–A6 and Figures A1–A3; article, footnote 7, p. 498 (PDF p. 7)", "pdf_page": 5, "printed_page": "A5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Supplement, p.~A5:\n\nWe assume $\\alpha = 0.275$, as recommended in the text. We present tables of \\hl{RMSE} vs $n \\in 500, 1000, 2500, 5000, 10000, 25000$ under the 5 different models, including misspecification.\n\nSupplement, p.~A9:\n\nIn Table A6, we present \\hl{\\% coverage} with the bootstrap for $n \\in 500, 2500, 5000$ assuming that the model is correct and $\\alpha = 0.275$.\n\nArticle, footnote 7:\n\nIn our own simulation studies, assuming $\\alpha = 0.275$, we find that the bootstrap has conservative \\hl{coverage} for ICSW in small samples, and proper \\hl{coverage} in larger samples.\n\nSimulation scripts in the replication files: \\texttt{output[1] <- mean(dgpout[[3]])}", "corrected": "Supplement, added to p.~A5 and p.~A9: \\hl{In the heterogeneous-effect simulations, RMSE is computed relative to the realized sample average treatment effect, $S_n=n^{-1}\\sum_i\\tau_i$. Table A6 likewise reports bootstrap interval coverage of $S_n$. These summaries do not report RMSE or coverage for the population ATE, $\\mathrm{E}\\,\\tau_i=5$.}\n\nArticle, footnote 7: In our own simulation studies, assuming $\\alpha = 0.275$, we find that the bootstrap has conservative coverage \\hl{of the realized sample average treatment effect} for ICSW in small samples, and proper coverage in larger samples.", "evidence": "The evidence is a derivation and a check of the scoring lines in the simulation scripts. For $X$ uniform on $(-1,1)$, $\\mathrm{E}X=0$ and $\\mathrm{E}X^2=1/3$. By independence of the three covariates, I have $\\mathrm{E}\\tau_i=5$ and $\\operatorname{Var}(\\tau_i)=25(1/3+1/3+1/3)=25$, and $\\operatorname{Var}(S_n)=25/n$. In the design ``Model is Incorrect'', $\\tau_i=5(1+X1_i+X2_i+X1_iX2_i+X3_i)$, and $\\operatorname{Var}(S_n)=250/(9n)$. Let $R_S$ and $R_\\theta$ denote the RMSE of an estimator $\\hat\\theta$ around $S_n$ and around $5$. Then\n\\[\nR_\\theta^2-R_S^2=\\operatorname{Var}(S_n)+2\\,\\mathrm{E}\\{(\\hat\\theta-S_n)(S_n-5)\\},\n\\]\nand the cross term need not vanish. Adding $25/n$ to a printed squared RMSE is not a valid conversion. In the scripts, the first stored quantity is \\texttt{mean(dgpout[[3]])}, the RMSE lines subtract it from each estimate, and the coverage lines in \\texttt{sims-BS.R} test whether it lies between the percentile endpoints. Applied to test inputs, the scripts' scoring lines and the same lines with $5$ as the target give different RMSEs and different coverage.\n\nThe saved simulation output is not in the replication files, and the tables were not recomputed against $5$. No printed entry, no ranking of the estimators and no statement about coverage is shown to change.", "problem": "The article defines the ATE as the population expectation $\\mathrm{E}(Y_1-Y_0)$. In the baseline simulation design with heterogeneous effects, $\\tau_i=5(1+X1_i+X2_i+X3_i)$ with covariates uniform on $(-1,1)$, and the population ATE is $5$. The heterogeneous-effect scripts store, for each simulated data set, the sample average $S_n=n^{-1}\\sum_i\\tau_i$. They compute the RMSE of each estimator around $S_n$, and they compute coverage as the frequency with which the bootstrap percentile interval contains $S_n$. The sample average $S_n$ is random. Scoring an estimator against it is legitimate, but the supplement does not say that it does so, and the definition in the article points to the population ATE. The RMSE around $S_n$ and the RMSE around $5$ are different quantities, and so are the two coverage probabilities.", "what_survives": "The heterogeneous-effect scoring formulas are valid for the sample average effect. Since $S_n$ converges to $5$, an estimator is consistent for one target if and only if it is consistent for the other. The identification argument, the estimator and the applications are unaffected. What changes is the reading of Tables A1, A2, A4 and A5, the heterogeneous panels of Figure A3, Figures A1 and A2, Table A6, and footnotes 6 and 7, which cite them. This evidence concerns the sample average effect. This finding does not cover Table A3, the design with a constant effect." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N009", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the sentence 'Thus the ML estimate of theta A,C and theta A given A,C' the display that follows and one line after it. A red box marks the display, in which the pair of estimates equals the argmax over both coefficient vectors of the likelihood L given D and Z.", "caption": "Section 3.1, p. 497 (PDF p. 6). The marked display defines the ML estimate as the maximizer of the likelihood over all coefficient vectors. The display is correct as printed.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N010.png" }, { "alt": "Listing of seven lines of R code from the function complier.score, three of them continued on a second row. Red boxes mark '1e4' in the default 'num.iter = ifelse(genoud, 200, 1e4)' on line 15, and 'maxfun=num.iter' and 'lower=-5,upper=5' in the call to bobyqa on line 81. Lines 86 to 89 compute CA.score, A.score and C.score from the returned parameters and return them.", "caption": "Replication files, icsw.R, lines 15, 80–81 and 86–89, typeset from the source file with line numbers added; long lines are continued after an arrow. The marked terms set the cap of 10,000 evaluations and the bounds $\\pm5$.", "url": "https://pmaronow.github.io/llm-audit/works/P029/exhibits/P029-N010-code.png" } ], "formal_release": null, "formal_statements": [], "headline": "The replication code for the two-sided compliance score confines each standardized coefficient to $[-5,5]$, stops after 10,000 evaluations and uses the result without a convergence check, so it can return a fit that is not the ML estimate defined on p. 497.", "id": "P029-N010", "location": "Section 3.1, p. 497 (PDF p. 6), display that defines the ML estimate; replication files, icsw.R, function complier.score, lines 15, 81 and 86–89, called at lines 37 and 92 of albertsonlawrence.R", "pdf_page": 6, "printed_page": "497", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Article, p.~497: Thus the ML estimate of $\\{\\theta_{\\mathbf{A,C}}, \\theta_{\\mathbf{A|A,C}}\\}$,\n\\[\n\\{\\hat\\theta_{\\mathbf{A,C}}, \\hat\\theta_{\\mathbf{A|A,C}}\\} = \\operatorname*{argmax}_{\\theta_{\\mathbf{A,C}},\\,\\theta_{\\mathbf{A|A,C}}}\\bigl[L(\\theta_{\\mathbf{A,C}}, \\theta_{\\mathbf{A|A,C}}|D,Z)\\bigr].\n\\]\n\nReplication files, \\texttt{icsw.R}, line 15 (one line in the file, shown here in two pieces):\n\n\\texttt{complier.score <- function(D, Z, W, weights = NULL, link = pnorm,}\n\n\\texttt{genoud = TRUE, num.iter = ifelse(genoud, 200, \\hl{1e4}),onesided=FALSE) \\{}\n\nLine 81 (also in two pieces):\n\n\\texttt{opt.log <- bobyqa(starting, complier.neg.logLik.fnc,control=list(iprint=0,}\n\n\\texttt{\\hl{maxfun=num.iter}),\\hl{lower=-5,upper=5})}\n\nLine 86, the first of the four lines that compute and return the scores (in two pieces):\n\n\\texttt{CA.score <- (pnorm(rowSums(t(t(W1) *}\n\n\\texttt{opt.log\\$par[1:(length(opt.log\\$par)/2)]))))}", "corrected": "The definition on p.~497 is unchanged. In \\texttt{icsw.R} and its callers, I would test the optimizer status before using the scores, and continue or report failed fits. I would remove or expand coefficient bounds that exclude better fits. A status of $0$ does not by itself establish a global maximum; I would describe the scores as numerical approximations to the ML scores unless their accuracy is established.", "evidence": "The evidence is an exact counterexample, a run of the code on it, and a run of the code on four bootstrap draws of the application.\n\nThe counterexample has 96 observations and two covariates, $W_1=U$ and $W_2=U+V/50$ with $U,V\\in\\{-1,1\\}$. Each of the eight cells $(U,V,Z)$ holds twelve observations. For either value of $U$, the number of observations with $D=1$ is $1$ for $(V,Z)=(-1,0)$, $2$ for $(-1,1)$, $5$ for $(1,0)$ and $10$ for $(1,1)$. Let $t$ be such that $\\Phi(t)=5/6$. The model of equations (5)--(7) holds with coefficients $\\theta_{\\mathbf{A,C}}=(0,-50t,50t)$ and $\\theta_{\\mathbf{A|A,C}}=(0,0,0)$ on the covariates $(1,W_1,W_2)$. Since these coefficients fit the proportion treated in every cell, they attain the maximum of the likelihood. In the standardized coordinates that the code uses, the third coefficient exceeds $100/3$. Let $M$ be the maximum of the likelihood and $B$ an upper bound for the likelihood over the box, namely\n\\[\nM=\\Bigl(\\frac56\\Bigr)^{40}\\Bigl(\\frac16\\Bigr)^{8}B_0,\\qquad B=\\Bigl(\\frac{11}{20}\\Bigr)^{40}\\Bigl(\\frac{9}{20}\\Bigr)^{8}B_0,\\qquad B_0=\\Bigl(\\frac{5}{12}\\Bigr)^{10}\\Bigl(\\frac{7}{12}\\Bigr)^{14}\\Bigl(\\frac{1}{12}\\Bigr)^{2}\\Bigl(\\frac{11}{12}\\Bigr)^{22}.\n\\]\nThe bound $B$ holds for the printed likelihood and for the likelihood with the probabilities clipped as in the code. Comparing the two in exact rational arithmetic, I find that $M>5000B$. Run on these data, the unchanged helper returns status $0$ after 253 evaluations, with two coefficients at the bound $5$ and negative log-likelihood $56.378$. The value at the maximizer is $44.812$.\n\nThe run on the application uses the script's seed and R 4.4.2 with \\texttt{minqa} 1.2.8. The fit on the full data ends with status $0$ after 1,096 evaluations. Of the first four bootstrap draws, the fourth stops at the cap (status $1$, 10,000 evaluations), and its scores are finite and are used. Continuing that search for 37,259 more evaluations lowers the negative log-likelihood from $197.12769$ to $197.12729$. A separate recalculation of the full bootstrap reported 39 capped fits among the 2,500 draws. I have not confirmed that count.", "problem": "The article defines the estimate as the maximizer of the likelihood over all coefficient vectors. The helper \\texttt{complier.score}, called with \\texttt{genoud=FALSE}, standardizes the covariates and searches with BOBYQA inside the box $[-5,5]$ for every coefficient. It stops after at most 10,000 evaluations of the objective and computes the scores from the last iterate without testing the status code that the optimizer returns. The box can exclude the unrestricted maximizer, and stopping at the cap does not establish convergence. The script nevertheless uses the scores without checking the returned status. The Albertson and Lawrence script calls this branch for the point estimate (line 37) and in every bootstrap replicate (line 92), so that the ICSW columns of Table 2 depend on these fits.", "what_survives": "The display on p.~497 is correct and stands as printed. No value in Table 2 is shown to change. A recalculation of the bootstrap that kept the capped fits, as the authors' code does, agreed with the printed intervals within Monte Carlo error. The intervals have not been recomputed with converged fits, and the software versions used for the article are not known. The 2SLS columns do not use the scores, and the consistency result is conditional on consistent scores. The finding concerns only the branch \\texttt{genoud=FALSE} of the two-sided fit in \\texttt{icsw.R}." }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-N010", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpt013", "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "url": "https://pmaronow.github.io/llm-audit/works/P029/" }, "work_id": "P029" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P029.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:99b64fac9c6bcf6b20f07aa663ae8b471b7f715ba22e13fa4cb42258a2e5a202", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.ignorability_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L197", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:99b64fac9c6bcf6b20f07aa663ae8b471b7f715ba22e13fa4cb42258a2e5a202", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.potential_pair_independent_of_receipt", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2402", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:99b64fac9c6bcf6b20f07aa663ae8b471b7f715ba22e13fa4cb42258a2e5a202", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T1" }, "html": { "body": "

Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1. The six-atom law satisfies the IV and received-treatment ignorability conditions, including potential-pair independence, but fails latent ignorability and has the stated ATE and Wald values.

", "headline": "Section 3.4, the sentence ‘Assumption 1 is less restrictive than the typical ignorability assumption’, set against Assumption 1 and the two displayed ignorability equalities; the witness is the validator's and is not in the paper", "location": "p. 499 (PDF p. 8), paragraph after Assumption 1 beginning ‘Ignorability assumptions, while strong’" }, "id": "P029-T1", "text": { "body": "Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1. The six-atom law satisfies the IV and received-treatment ignorability conditions, including potential-pair independence, but fails latent ignorability and has the stated ATE and Wald values.", "headline": "Section 3.4, the sentence ‘Assumption 1 is less restrictive than the typical ignorability assumption’, set against Assumption 1 and the two displayed ignorability equalities; the witness is the validator's and is not in the paper", "location": "p. 499 (PDF p. 8), paragraph after Assumption 1 beginning ‘Ignorability assumptions, while strong’" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T1" }, { "declarations": [ { "claim_revision_id": "catalog:5b000171abcc0deea3fff5569759d94be13fab89f5aa15064d81c89060bfd009", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.baseline_selection_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L230", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5b000171abcc0deea3fff5569759d94be13fab89f5aa15064d81c89060bfd009", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.corrected_ignorability_incomparability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2380", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5b000171abcc0deea3fff5569759d94be13fab89f5aa15064d81c89060bfd009", "conditional_on": [], "credit_reason": "The apparent extra no-defiers and global relevance premises in IVAssumptions follow from D0=0 almost surely, positive compliance at a positive-mass x, and randomized nondegenerate assignment; they do not exclude a target case.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.one_sided_ignorability_implication_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B2b.lean#L279", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:5b000171abcc0deea3fff5569759d94be13fab89f5aa15064d81c89060bfd009", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T2", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T2" }, "html": { "body": "

Section 3.4, p. 499 (PDF p. 8), comparison following Assumption 1. The two ignorability conditions are incomparable, as witnessed by the stated six-atom laws with type masses (1/2, 1/4, 1/4). The baseline-selection law has constant effect one and received-treatment means (1/2, 0) and (3/2, 1). Under randomized one-sided noncompliance, received-treatment ignorability implies latent ignorability at every positive-mass profile with positive compliance. The usual measure-theoretic conditions are assumed.

", "headline": "Section 3.4, comparison of Assumption 1 with the typical ignorability assumption, corrected: the two conditions are not ordered in general; the typical condition implies Assumption 1 under one-sided noncompliance", "location": "p. 499 (PDF p. 8), paragraph after Assumption 1" }, "id": "P029-T2", "text": { "body": "Section 3.4, p. 499 (PDF p. 8), comparison following Assumption 1. The two ignorability conditions are incomparable, as witnessed by the stated six-atom laws with type masses (1/2, 1/4, 1/4). The baseline-selection law has constant effect one and received-treatment means (1/2, 0) and (3/2, 1). Under randomized one-sided noncompliance, received-treatment ignorability implies latent ignorability at every positive-mass profile with positive compliance. The usual measure-theoretic conditions are assumed.", "headline": "Section 3.4, comparison of Assumption 1 with the typical ignorability assumption, corrected: the two conditions are not ordered in general; the typical condition implies Assumption 1 under one-sided noncompliance", "location": "p. 499 (PDF p. 8), paragraph after Assumption 1" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T2" }, { "declarations": [ { "claim_revision_id": "catalog:1fb6606f7e13aa43f2b40d543365e0714680c8d77d793c312cb4be2bb4a206fe", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.conditional_independence_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L351", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1fb6606f7e13aa43f2b40d543365e0714680c8d77d793c312cb4be2bb4a206fe", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.conditional_witness_not_marginally_independent", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2413", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1fb6606f7e13aa43f2b40d543365e0714680c8d77d793c312cb4be2bb4a206fe", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.marginal_independence_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L297", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1fb6606f7e13aa43f2b40d543365e0714680c8d77d793c312cb4be2bb4a206fe", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.marginal_witness_nested_probabilities", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2423", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1fb6606f7e13aa43f2b40d543365e0714680c8d77d793c312cb4be2bb4a206fe", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T3" }, "html": { "body": "

Section 3.4, p. 499 (PDF p. 8), last sentence of the sufficiency paragraph. The explicit binary-covariate laws refute the marginal and conditional independence readings, with the stated nested probabilities, arm differences, ATEs, LATEs and population Wald ratios.

", "headline": "Section 3.4, the sentence ‘More generally, even if treatment effects are heterogeneous, but still statistically independent of compliance, Assumption 1 would necessarily be satisfied, and both 2SLS and ICSW would be consistent for the ATE’; the two witnesses are the validator's and are not in the paper", "location": "p. 499 (PDF p. 8), last sentence of the paragraph beginning ‘We can describe sufficient conditions for and consequences of Assumption 1’" }, "id": "P029-T3", "text": { "body": "Section 3.4, p. 499 (PDF p. 8), last sentence of the sufficiency paragraph. The explicit binary-covariate laws refute the marginal and conditional independence readings, with the stated nested probabilities, arm differences, ATEs, LATEs and population Wald ratios.", "headline": "Section 3.4, the sentence ‘More generally, even if treatment effects are heterogeneous, but still statistically independent of compliance, Assumption 1 would necessarily be satisfied, and both 2SLS and ICSW would be consistent for the ATE’; the two witnesses are the validator's and are not in the paper", "location": "p. 499 (PDF p. 8), last sentence of the paragraph beginning ‘We can describe sufficient conditions for and consequences of Assumption 1’" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T3" }, { "declarations": [ { "claim_revision_id": "catalog:5fa11705af98f87e10f72dd12ef5ee072c067ddc5dde5925fc7bdc67793cce5b", "conditional_on": [], "credit_reason": "This counterexample_proved row is a separation witness for the corrected target T4(vi), not a proof of its positive sufficiency implications; those are covered by R189-10 and R189-11.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.conditional_independence_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L351", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5fa11705af98f87e10f72dd12ef5ee072c067ddc5dde5925fc7bdc67793cce5b", "conditional_on": [], "credit_reason": "The target itself explicitly fixes finite covariate support and integrable potential outcomes. Null profile labels can be removed or assigned arbitrary positive weight extensions; replacing pointwise no-defiers by almost-sure no-defiers leaves every moment unchanged. The statement proves population identities, which are the specified component.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.corrected_weighted_limits_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B4.lean#L372", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5fa11705af98f87e10f72dd12ef5ee072c067ddc5dde5925fc7bdc67793cce5b", "conditional_on": [], "credit_reason": "The target itself requires finite covariate support and explicitly proposes these two joint-independence alternatives. Boolean decide(complier) represents exactly the binary C; no additional independence claim is hidden.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.joint_independence_suffices_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B4.lean#L621", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5fa11705af98f87e10f72dd12ef5ee072c067ddc5dde5925fc7bdc67793cce5b", "conditional_on": [], "credit_reason": "This counterexample_proved row is a separation witness for the corrected target T4(vi), not a proof of its positive sufficiency implications; those are covered by R189-10 and R189-11.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.marginal_independence_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L297", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5fa11705af98f87e10f72dd12ef5ee072c067ddc5dde5925fc7bdc67793cce5b", "conditional_on": [], "credit_reason": "The transported corrected_weighted_limits_proof already supplies the population covariance-gap identity; this finite-profile helper is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.profile_covariance_gap", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2236", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:5fa11705af98f87e10f72dd12ef5ee072c067ddc5dde5925fc7bdc67793cce5b", "conditional_on": [], "credit_reason": "The transported corrected_weighted_limits_proof already supplies inverse-score cancellation; this finite-profile helper is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.profile_inverse_score_cancellation", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2215", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:5fa11705af98f87e10f72dd12ef5ee072c067ddc5dde5925fc7bdc67793cce5b", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T4", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T4" }, "html": { "body": "

Section 3.4, p. 499 (PDF p. 8), corrected heterogeneous-effect sufficiency statement; Appendix equations (10) and (13), p. 504 (PDF p. 13). For finite positive-mass covariate support, the weighted Wald identities, LATE and true-score ICSW formulas, covariance gap, separate ATE sufficiency implications, both separation laws, and both joint-independence sufficient conditions hold. For iid sampling and measurable profile-based consistent score estimates, ordinary IV and estimated-score ICSW converge in probability to the respective population ratios. The respective mean-equality premises give ATE consistency; imposing both gives consistency of both estimators. The usual measure-theoretic conditions are assumed.

", "headline": "Section 3.4, the sufficiency sentence for heterogeneous effects, corrected: the mean-independence condition each estimator needs, with the exact relation between the two limits", "location": "p. 499 (PDF p. 8), last sentence of the paragraph beginning ‘We can describe sufficient conditions’; the limits are the displays on p. 499 and Appendix equations (10) and (13), p. 504 (PDF p. 13)" }, "id": "P029-T4", "text": { "body": "Section 3.4, p. 499 (PDF p. 8), corrected heterogeneous-effect sufficiency statement; Appendix equations (10) and (13), p. 504 (PDF p. 13). For finite positive-mass covariate support, the weighted Wald identities, LATE and true-score ICSW formulas, covariance gap, separate ATE sufficiency implications, both separation laws, and both joint-independence sufficient conditions hold. For iid sampling and measurable profile-based consistent score estimates, ordinary IV and estimated-score ICSW converge in probability to the respective population ratios. The respective mean-equality premises give ATE consistency; imposing both gives consistency of both estimators. The usual measure-theoretic conditions are assumed.", "headline": "Section 3.4, the sufficiency sentence for heterogeneous effects, corrected: the mean-independence condition each estimator needs, with the exact relation between the two limits", "location": "p. 499 (PDF p. 8), last sentence of the paragraph beginning ‘We can describe sufficient conditions’; the limits are the displays on p. 499 and Appendix equations (10) and (13), p. 504 (PDF p. 13)" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T4" }, { "declarations": [ { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "conditional_on": [], "credit_reason": "Witness for a scoring-rule mismatch on admissible realized inputs, not a refutation of historical numerical entries. Assignment indexing in the opposite input is a realized error array, as in the target; it does not assert a population law with treatment affecting baseline outcomes. No change from the blind claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.homogeneous_rmse_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1018", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "conditional_on": [], "credit_reason": "The selected counterexample and positive-weight theorem already export coefficient recovery; the generic matrix helper is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.normal_equation_exact_recovery", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L998", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "conditional_on": [], "credit_reason": "Auxiliary determinant certificate for the fixed witness, not a standalone refutation.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.rmse_design_determinant", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L992", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "conditional_on": [], "credit_reason": "The selected homogeneous_rmse_counterexample already exports this exact Gram identity, which is used to transport the outcome cross-products.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.rmse_design_gram", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L982", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "conditional_on": [], "credit_reason": "Strictly positive weights are part of T5(d), not an extra assumption added by Lean; no fitted-score existence claim is required.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.rmse_positive_weight_recovery", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1040", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "conditional_on": [], "credit_reason": "This row covers the stated uniform-support, threshold and balance checks, not a theorem constructing an entire random simulation law.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.rmse_witness_admissible", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2450", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "conditional_on": [], "credit_reason": "The row accurately limits itself to the two matrix-vector identities and benchmark; the effect and observed-outcome identities are in R190-01.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.rmse_witness_outcomes", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1005", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:2a2a03b5dc31d0b84acd4885fb1e6ca7466ee9650e0b2a0bb720e2a99cc6e15d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T5", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T5" }, "html": { "body": "

supplement pp. A2, A5 and A7, homogeneous-noise design, Table A3 and Figure A3. Two admissible 500-record homogeneous-effect inputs have the stated nonsingular design, cross-products and recovered coefficients, and scoring about six instead of five reverses the squared-error discrepancy even though positive weights preserve exact recovery.

", "headline": "Supplement Table A3 ‘Noise (Homogeneous)’ and the ‘Noise (Homogeneous)’ panel of Figure A3, presented as RMSE for the design with τi=5\\tau_i = 5τi​=5; the scoring rule is in the deposited code and the witnesses are the adjudicating validator's", "location": "Supplement p. A2 (design), p. A5 (description of the RMSE tables), p. A6 (Figure A3), p. A7 (Table A3); deposited code ‘Simulations Misspecification Noise 6-20.R’, lines 14–27, 70–94 and 124–135, and ‘Sim-Display Misspec.R’, lines 59–70" }, "id": "P029-T5", "text": { "body": "supplement pp. A2, A5 and A7, homogeneous-noise design, Table A3 and Figure A3. Two admissible 500-record homogeneous-effect inputs have the stated nonsingular design, cross-products and recovered coefficients, and scoring about six instead of five reverses the squared-error discrepancy even though positive weights preserve exact recovery.", "headline": "Supplement Table A3 ‘Noise (Homogeneous)’ and the ‘Noise (Homogeneous)’ panel of Figure A3, presented as RMSE for the design with $\\tau_i = 5$; the scoring rule is in the deposited code and the witnesses are the adjudicating validator's", "location": "Supplement p. A2 (design), p. A5 (description of the RMSE tables), p. A6 (Figure A3), p. A7 (Table A3); deposited code ‘Simulations Misspecification Noise 6-20.R’, lines 14–27, 70–94 and 124–135, and ‘Sim-Display Misspec.R’, lines 59–70" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T5" }, { "declarations": [ { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "The L2 hypotheses are exactly the finite-second-moment assumptions in T6(d); they ensure all squared-error and covariance integrals have their usual meanings.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.expected_mse_discrepancy", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L908", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "This is a finite auxiliary lemma, not a claim to replace the general probability-space version in R190-09.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.finite_expected_mse_discrepancy", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L893", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "The empty-run extension is harmless; positive-length runs have the ordinary empirical-average meaning. No change from the claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.finite_run_mse_discrepancy", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L883", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "Measurability is intrinsic to the claimed random-variable laws, and full independence of all 3n coordinates is explicitly in T6(c). No conditional published premise is used.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.heterogeneous_benchmark_moments_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B5.lean#L124", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "Specializing C to the measurable complier event yields exactly the asserted LATE; the integral is of the constant treatment difference, not of the separate potentially nonintegrable outcomes. No change from the claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.homogeneous_population_effect", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L857", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "This narrower row states only R=2500; the all-positive-R formula instead follows by unfolding the existing parametric rmse definition, copied under its SAP name.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.homogeneous_rmse_rule", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L872", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "The nonempty-sample assumption is explicitly in T6(a) and prevents the empty-average convention.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.homogeneous_sample_effect", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L850", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "Auxiliary exact algebra retaining the cross term; no independence or asymptotic approximation is assumed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.squared_error_discrepancy", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L878", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "conditional_on": [], "credit_reason": "The actual continuous uniform law is used; no assumed moment specification or finite-support substitution appears. No change from the claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.uniform_covariate_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L938", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:ea076d83b4478157a68fe43f95e1fc66a630c4c03c5822380180d2c676ae446b", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T6", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T6" }, "html": { "body": "

supplement pp. A2, A5 and A7, corrected homogeneous-noise scoring rule. The effect is five, the corrected RMSE uses five for every positive replicate count, the coded benchmark has the stated uniform moments, and the finite and population error-discrepancy identities permit both signs on the admissible inputs.

", "headline": "Supplement Table A3 and the ‘Noise (Homogeneous)’ panel of Figure A3, corrected scoring rule: RMSE about the constant effect 5, with the exact relation to the coded statistic", "location": "Supplement pp. A2, A5–A7; deposited code ‘Simulations Misspecification Noise 6-20.R’, lines 124–135, and ‘Sim-Display Misspec.R’, lines 59–70" }, "id": "P029-T6", "text": { "body": "supplement pp. A2, A5 and A7, corrected homogeneous-noise scoring rule. The effect is five, the corrected RMSE uses five for every positive replicate count, the coded benchmark has the stated uniform moments, and the finite and population error-discrepancy identities permit both signs on the admissible inputs.", "headline": "Supplement Table A3 and the ‘Noise (Homogeneous)’ panel of Figure A3, corrected scoring rule: RMSE about the constant effect 5, with the exact relation to the coded statistic", "location": "Supplement pp. A2, A5–A7; deposited code ‘Simulations Misspecification Noise 6-20.R’, lines 124–135, and ‘Sim-Display Misspec.R’, lines 59–70" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T6" }, { "declarations": [ { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "This row is only the derivative component, not a standalone assertion about refitting or a full refutation; the earlier review's generic witness caveat is unnecessary for this exact component.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.bootstrap_first_step_derivative", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L828", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:2847 evaluates the Wald estimator using bootstrapRefitScores, whose definition at P029/P029_Audit.lean:2842 merely prescribes 1/(3+t) and 3/4. T7(v) and the row assert that refitting the weighted compliance model produces these scores and hence the displayed estimator, but the declaration contains no fitting or likelihood-optimality conclusion. The missing link is substantive: an identity after substituting chosen scores does not establish the effect of refitting, even though the intended scores are correct and the separate general cell-minimum theorem at P029/P029_Audit.lean:816 can establish it after specialization.', 'suggested_fix': 'Add a conclusion that bootstrapRefitScores t uniquely minimizes the observation-weighted negative Bernoulli log likelihood on the sampleZ = true records over both interior W-cell probabilities. Specialize saturated_weighted_objective_minimum with success/failure totals (1, 2+t) and (3, 1), then retain the existing Wald identities.', 'second_review': 'T7(v) requires the scores and Wald ratio obtained by refitting the compliance model to the perturbed observation weights. P029/P029_Audit.lean:2847 evaluates the ratio at bootstrapRefitScores, but P029/P029_Audit.lean:2842 defines that array by the prescribed numbers 1/(3+t) and 3/4 without an optimization property. Thus the estimator arithmetic omits the substantive identification of these scores as the weighted fit. Append that identification for the actual Z = 1 records, using the cell totals (1, 2+t) and (3, 1) and the separate saturated optimum theorem at P029/P029_Audit.lean:816.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.bootstrap_refit_ratio", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2470", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "Harmless genuine generalization; strict likelihood positivity follows from the clipping in the target.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.inside_log_minimizers", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L533", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "Harmless genuine generalization; the parameter-independent interpretation follows by applying this identity at each parameter.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.inside_log_weights_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L522", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P029/P029_Audit.lean:947 establishes the component identified by this row: both specified cell arrays are interior and ordered, their unweighted likelihood ratio is 729/1500625 < 1, and the weighted compliance scores are 3/20 and 3/14. These are the corresponding arithmetic and feasibility assertions in T7(iv); requiring this component also to prove both global fitting optima would extend its stated role. Ordered receipt rates faithfully represent nested feasibility: for each x take a = q(x,true) and b = q(x,false)/a, as characterized by nested_receipt_parameters at P029/P029_Audit.lean:2241; intercept-plus-binary-W probit coefficients can represent any such interior a and b. The flag therefore identifies no missing conclusion within this comparison component, and the theorem's names are not themselves optimization assertions.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.two_sided_weight_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L811", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:925 states two rational identities for correctedScores and evaluates Wald at correctedScores and unweightedScores, which are prescribed constants at P029/P029_Audit.lean:919. T7(iii) and the row explicitly assert that these are the weighted and unweighted fitted scores, including the cell-maximizer property. No conclusion identifies either array as a likelihood optimum on the actual Z = 1 records, so the statement establishes estimator arithmetic without the claimed fitting counterexample. The separate saturated_weighted_objective_minimum theorem at P029/P029_Audit.lean:816 supplies a possible repair but does not add that missing conclusion to this declaration.', 'suggested_fix': 'Append unique-minimum conclusions for the one-sided Bernoulli objective on sampleZ = true records: unweightedScores for unit weights and correctedScores for oneSidedWeight. Specialize the saturated objective theorem to the actual two W cells, with unit-weight totals (1,3), (3,1) and weighted totals (9,3), (27,1).', 'second_review': 'T7(iii) asserts that the unweighted and observation-weighted likelihood fits produce the specified scores and consequently different weighted IV estimates. P029/P029_Audit.lean:925 establishes the numerical Wald values and two rational score identities, while unweightedScores and correctedScores at P029/P029_Audit.lean:919 are simply prescribed constants. No conclusion identifies either array as a likelihood optimum on the treated-assignment records of oneSidedSample at P029/P029_Audit.lean:914. Specializing the saturated objective theorem at P029/P029_Audit.lean:816 to the actual weighted and unweighted cell totals would supply the missing fitting conclusion and repair the counterexample statement.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.weighted_score_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L789", "status": "counterexample_proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "The +∞ assertion is definition-level because the target itself specifies log 0=−∞; this is the intended convention and does not render a substantive positive-likelihood case empty.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.zero_weight_infinite_contribution", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L556", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "conditional_on": [], "credit_reason": "The definition only takes finite or +∞ values, so no −∞ summand can cancel the selected +∞. The record's observation about unused proof premises is not a reason to remove premises from the stated theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.zero_weight_objective_infinite", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L563", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:7196c75d2d4209602ac4ce027733152e4b41cf971386e7d00100bbce478ebc0f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T7", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T7" }, "html": { "body": "

package icsw 1.0.0, compliance.score.R lines 43–48 and 81–92, icsw.tsls.R lines 28–37 and package help; article Section 3.3, p. 498 (PDF p. 7). Positive weights inside the logarithm leave all minimizers unchanged; a zero weight gives infinite extended objective. In the sixteen-record one-sided witness, the actual unweighted and weighted probit fits have scores (1/4,3/4) and (3/4,27/28), giving Wald estimates 3/2 and 17/10. Every ordered raw two-sided receipt array has finite nested-probit coefficients; the stated two-sided arrays are the actual clipped package and corrected fits, with the stated score differences and likelihood ratio. Under the positive weight perturbation, actual refitting yields scores (1/(3+t),3/4) and the stated Wald curve with derivative -1/16 at one, while the fixed-score estimate has derivative zero. These facts refute the package help's interpretation that the supplied weights refit the compliance model. The usual measure-theoretic conditions are assumed.

", "headline": "R package icsw 1.0.0 (the published implementation of the method): the compliance-score objective with observation weights and the reweighting bootstrap, set against Section 3.3's ‘bootstrapping the entire process’ and the package help; the witnesses are the validators'", "location": "Not in the article PDF. Staged package source: packages/icsw/icsw/R/compliance.score.R, lines 43–48 and 81–92; R/icsw.tsls.R, lines 28–37; man/icsw.tsls.Rd, lines 9–13 and 42–43. The recommendation it implements: p. 498 (PDF p. 7), Section 3.3" }, "id": "P029-T7", "text": { "body": "package icsw 1.0.0, compliance.score.R lines 43–48 and 81–92, icsw.tsls.R lines 28–37 and package help; article Section 3.3, p. 498 (PDF p. 7). Positive weights inside the logarithm leave all minimizers unchanged; a zero weight gives infinite extended objective. In the sixteen-record one-sided witness, the actual unweighted and weighted probit fits have scores (1/4,3/4) and (3/4,27/28), giving Wald estimates 3/2 and 17/10. Every ordered raw two-sided receipt array has finite nested-probit coefficients; the stated two-sided arrays are the actual clipped package and corrected fits, with the stated score differences and likelihood ratio. Under the positive weight perturbation, actual refitting yields scores (1/(3+t),3/4) and the stated Wald curve with derivative -1/16 at one, while the fixed-score estimate has derivative zero. These facts refute the package help's interpretation that the supplied weights refit the compliance model. The usual measure-theoretic conditions are assumed.", "headline": "R package icsw 1.0.0 (the published implementation of the method): the compliance-score objective with observation weights and the reweighting bootstrap, set against Section 3.3's ‘bootstrapping the entire process’ and the package help; the witnesses are the validators'", "location": "Not in the article PDF. Staged package source: packages/icsw/icsw/R/compliance.score.R, lines 43–48 and 81–92; R/icsw.tsls.R, lines 28–37; man/icsw.tsls.Rd, lines 9–13 and 42–43. The recommendation it implements: p. 498 (PDF p. 7), Section 3.3" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T7" }, { "declarations": [ { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:2847 substitutes the explicitly defined bootstrapRefitScores into fWald and obtains (t+11)/(t+7), but does not assert any optimization property of those scores. T8(c), through its reference to T7(v), claims this is the estimate obtained under the corrected weighted fit, and the row explicitly describes refitting. Without identifying the prescribed array as that fit, the declaration omits a substantive part of its credited component, despite the correct numerical formula. The general cell-minimum result at P029/P029_Audit.lean:816 is separate from this statement.', 'suggested_fix': 'Append unique optimality of bootstrapRefitScores t for the corrected weighted Bernoulli objective restricted to the Z = 1 records, using the W-cell success/failure totals (1, 2+t) and (3, 1); keep the existing estimator identities.', 'second_review': 'T8(c), through T7(v), identifies (t+11)/(t+7) as the estimate under the corrected observation-weighted likelihood fit. P029/P029_Audit.lean:2847 instead evaluates fWald using bootstrapRefitScores, which P029/P029_Audit.lean:2842 defines explicitly without asserting minimization of the corrected weightedObjective at P029/P029_Audit.lean:649. Identifying these scores as the actual fit is a substantive missing conclusion of the credited component, even though the numerical ratio is correct. Append optimality for the Z = 1 sample with cell totals (1, 2+t) and (3, 1), specializing P029/P029_Audit.lean:816.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.bootstrap_refit_ratio", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2470", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "The constant-weight part of T8(d) is the component covered here; this row does not purport to supply its separate nonconstant-weight witnesses.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.constant_weight_minimizers", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L766", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "Zero-likelihood and negative-input extensions rely on totalized log, but none is needed by the target, which assumes positive contributions and instructs omission of zero-weight records.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.corrected_objective_zero_unit", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L583", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "Zero multiplicities use the standard natural-power convention L^0=1, including the empty replicated data set.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.integer_weight_replication", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L591", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "The component is T8(b) for free cell probabilities, not a claim about a numerical optimizer or an arbitrary unsaturated probit model; the target itself includes the positive-cell-total restriction.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.saturated_weighted_objective_minimum", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L680", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P029/P029_Audit.lean:947 proves the ordered-interior-rate and compliance-score component of T8(c), together with the likelihood comparison inherited from T7(iv), exactly as the row describes. The target's specified weighted rates are the constants in weightedCells at P029/P029_Audit.lean:940, and the statement gives their scores 3/20 and 3/14 without restricting any cases of this finite comparison. The optimization property is a distinct component, not an additional conclusion promised by this row's feasibility and likelihood-ratio description. Nor is there a material coefficient-realization gap: q(x,true) and q(x,false)/q(x,true) are valid nested parameters by P029/P029_Audit.lean:2241 and can be represented with a probit intercept and binary W.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.two_sided_weight_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L811", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "Strict positivity of S and F and the open parameter interval are explicitly part of the target, so no boundary cases are omitted.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.weighted_cell_strict_concavity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L650", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "This is the unique-maximizer component; strict concavity is deliberately a separate credited statement.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.weighted_cell_unique_max", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L606", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:925 evaluates the weighted IV estimate at the prescribed correctedScores and verifies the fractions 9/(9+3) and 27/(27+1). T8(c) claims that the corrected objective fits those scores, and the row expressly calls them cell maximizers, but the theorem contains no likelihood inequality or minimizer conclusion tying them to that objective on the sample. Consequently its arithmetic does not establish the credited fitted-score assertion, even though the intended scores and 17/10 estimate are correct. The general optimization theorem at P029/P029_Audit.lean:816 must be specialized and its consequence included in this statement.', 'suggested_fix': \"Append that correctedScores is the unique minimizer of the corrected weighted negative Bernoulli log likelihood on the Z = 1 records, with W-cell success/failure totals (9,3) and (27,1); include the unweighted fit's optimality if retaining the row's fitting contrast.\", 'second_review': \"T8(c) requires that the corrected objective's one-sided fit be (3/4, 27/28), as well as that the resulting weighted IV estimate be 17/10. P029/P029_Audit.lean:925 establishes the fractions and Wald values using the hard-coded score arrays at P029/P029_Audit.lean:919 and P029/P029_Audit.lean:920, but contains no minimizer conclusion for weightedObjective, defined at P029/P029_Audit.lean:649. The general unique-minimizer theorem at P029/P029_Audit.lean:816 permits a local repair by specializing it to the Z=1 records and identifying their actual cell totals as (9,3) and (27,1), with corresponding unweighted optimality if retaining the fitting contrast. The missing link to likelihood fitting is a substantive part of the target claim, so changing only the description would not repair the credited statement.\", 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.weighted_score_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L789", "status": "counterexample_proved", "support_scope": "limited", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:857bf13ce10bcfea048588896f1924b4176ce18425b31a43fc93794408fe6770", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T8", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T8" }, "html": { "body": "

package icsw 1.0.0 and deposited icsw.R helper, corrected observation weights; article Section 3.1, pp. 496–497 (PDF pp. 5–6). Observation weights outside the logarithm satisfy the zero/unit and replication rules, give unique free-cell fits, and preserve the stated constant-weight and numerical witness conclusions.

", "headline": "R package icsw 1.0.0, compliance-score fit with observation weights, corrected: the weighted log likelihood −∑iwilog⁡Li(θ)-\\sum_i w_i\\log L_i(\\theta)−∑i​wi​logLi​(θ) (also the corrected one-sided objective of the Dataverse helper, finding P029-E014)", "location": "Not in the article PDF. Staged package source: packages/icsw/icsw/R/compliance.score.R, lines 43–47 and 91; Dataverse helper code/dataverse-UCFNHX-v2.0-original/icsw.R, lines 49–50" }, "id": "P029-T8", "text": { "body": "package icsw 1.0.0 and deposited icsw.R helper, corrected observation weights; article Section 3.1, pp. 496–497 (PDF pp. 5–6). Observation weights outside the logarithm satisfy the zero/unit and replication rules, give unique free-cell fits, and preserve the stated constant-weight and numerical witness conclusions.", "headline": "R package icsw 1.0.0, compliance-score fit with observation weights, corrected: the weighted log likelihood $-\\sum_i w_i\\log L_i(\\theta)$ (also the corrected one-sided objective of the Dataverse helper, finding P029-E014)", "location": "Not in the article PDF. Staged package source: packages/icsw/icsw/R/compliance.score.R, lines 43–47 and 91; Dataverse helper code/dataverse-UCFNHX-v2.0-original/icsw.R, lines 49–50" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T8" }, { "declarations": [ { "claim_revision_id": "catalog:2b22f69ea9331ac51e13cdd0969c5350ddc6fa02c1c9a45124464df434d5b305", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1369 asserts positivity of exactly the probability of the recorded assignment vector under independent Bernoulli(e) assignment, for every 0 < e < 1. This supplies the assignment-support component of T9: both fixed completions can generate the recorded assignment with positive probability. Independence from fixed potential responses and demographics follows from that randomization construction; requiring empirical independence of the realized record list would impose a different condition.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.canvass_assignment_supported", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1210", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2b22f69ea9331ac51e13cdd0969c5350ddc6fa02c1c9a45124464df434d5b305", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1291 equates record-level and count-weighted cell-level conditional means for every cell-constant value function and cell event. The masses at P029_Audit.lean:1252 count actual records, so this is the compression needed for T9 rather than a change to a nine-person population. Totalizing an empty-event conditional mean to zero on both sides loses no positive-event case, and the race comparisons in T9 condition on nonempty groups.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.canvass_cell_conditional_mean", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1132", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2b22f69ea9331ac51e13cdd0969c5350ddc6fa02c1c9a45124464df434d5b305", "conditional_on": [], "credit_reason": "The selected conditional-mean compression theorem already incorporates the expectation compression; no separate invocation is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.canvass_cell_expectation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1097", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2b22f69ea9331ac51e13cdd0969c5350ddc6fa02c1c9a45124464df434d5b305", "conditional_on": [], "credit_reason": "The selected conditional-mean compression theorem already incorporates the probability compression; no separate invocation is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.canvass_cell_probability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1110", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2b22f69ea9331ac51e13cdd0969c5350ddc6fa02c1c9a45124464df434d5b305", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1315 realizes 18,933 records, preserves the recorded Boolean race labels, receipts and arbitrary record-specific outcomes, and exports the two conditional-mean gaps 744791/29225158 > 0 and -131213/1761229 < 0. Its potential-unit definition at P029_Audit.lean:1303 makes compliance equal recorded contact, including zero compliance for controls, exactly as T9 prescribes. Raw-code retention and assignment support are companion components at P029_Audit.lean:1385 and P029_Audit.lean:1369; the Boolean representation does not exclude a completion required by the target.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.racial_completion_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1156", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2b22f69ea9331ac51e13cdd0969c5350ddc6fa02c1c9a45124464df434d5b305", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1385 retains each raw code and requires both completed White indicators to agree with code = \"W\" whenever a code is present. The hR premise only links the supplied raw strings to the recorded Boolean labels used by T9; every compatible non-W code, including the Raleigh unknown codes, is permitted. Retaining the raw coordinate by definition is appropriate bookkeeping, and the additional classification conclusion establishes the substantive compatibility requirement.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.racial_raw_code_preservation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1226", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:2b22f69ea9331ac51e13cdd0969c5350ddc6fa02c1c9a45124464df434d5b305", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T9", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T9" }, "html": { "body": "

Section 4.1, p. 501 (PDF p. 10), and supplement Table A7. The two 18,933-record completions preserve the specified recorded labels, compatible raw codes and arbitrary outcomes, yet give opposite strict compliance/race associations in both directions.

", "headline": "Section 4.1, ‘We see that compliers are more likely to have voted previously, be members of a major party, and identify as white’, read off the correlations of supplement Table A7; the non-identification witness is the validator's and is not in the paper", "location": "p. 501 (PDF p. 10), last paragraph; supplement Table A7, p. A11" }, "id": "P029-T9", "text": { "body": "Section 4.1, p. 501 (PDF p. 10), and supplement Table A7. The two 18,933-record completions preserve the specified recorded labels, compatible raw codes and arbitrary outcomes, yet give opposite strict compliance/race associations in both directions.", "headline": "Section 4.1, ‘We see that compliers are more likely to have voted previously, be members of a major party, and identify as white’, read off the correlations of supplement Table A7; the non-identification witness is the validator's and is not in the paper", "location": "p. 501 (PDF p. 10), last paragraph; supplement Table A7, p. A11" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T9" }, { "declarations": [ { "claim_revision_id": "catalog:ea8fa87783817854e835ccf0e7304fe0869e7f4044ec0907ae521337d4028b98", "conditional_on": [], "credit_reason": "The selected binary_population_count_sign gives the formulas for actual finite binary variables and counts; this abstract count-algebra helper is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.binary_count_sign_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1247", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea8fa87783817854e835ccf0e7304fe0869e7f4044ec0907ae521337d4028b98", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.binary_population_count_sign", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1294", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea8fa87783817854e835ccf0e7304fe0869e7f4044ec0907ae521337d4028b98", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1369 gives a strictly positive independent Bernoulli probability to the recorded assignment vector for every assignment probability in (0,1). This is also the support fact needed for the two completions in T10(c), whose claim explicitly reuses the completions of T9. The T9 label identifies that shared construction; absence of a separately named independence predicate does not change the product randomization or remove a required case.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.canvass_assignment_supported", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1210", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea8fa87783817854e835ccf0e7304fe0869e7f4044ec0907ae521337d4028b98", "conditional_on": [], "credit_reason": "Acceptable generalization. The assigned component is T10(d), not empirical fitted-score signs. The record’s reference to a separate held-metadata typo about assignment does not describe this row’s original text, which is accurate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.grouped_covariance_decomposition_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B1.lean#L107", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea8fa87783817854e835ccf0e7304fe0869e7f4044ec0907ae521337d4028b98", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1315 supplies precisely the two completions invoked in T10(c), preserving observed races, receipts and outcomes while producing the count numerators 1489582 and -1049704. The underlying population contains all 18,933 individual records, and its outcome vector is unrestricted. Boolean race labels, with raw-code preservation handled at P029_Audit.lean:1385, change representation without weakening the claimed non-identification result.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.racial_completion_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1156", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ea8fa87783817854e835ccf0e7304fe0869e7f4044ec0907ae521337d4028b98", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.raleigh_recorded_comparison", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1269", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:ea8fa87783817854e835ccf0e7304fe0869e7f4044ec0907ae521337d4028b98", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T10", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T10" }, "html": { "body": "

Section 4.1, p. 501 (PDF p. 10), corrected interpretation of supplement Table A7. Finite binary counts determine both association directions, the two Raleigh rational rate comparisons are negative, compatible completions have opposite signs, and covariance decomposes into the stated within-group and between-group terms.

", "headline": "Section 4.1, description of the compliers, corrected (formal content of the narrowed claim): what the recorded race labels do and do not determine", "location": "p. 501 (PDF p. 10), last paragraph; supplement Table A7, p. A11" }, "id": "P029-T10", "text": { "body": "Section 4.1, p. 501 (PDF p. 10), corrected interpretation of supplement Table A7. Finite binary counts determine both association directions, the two Raleigh rational rate comparisons are negative, compatible completions have opposite signs, and covariance decomposes into the stated within-group and between-group terms.", "headline": "Section 4.1, description of the compliers, corrected (formal content of the narrowed claim): what the recorded race labels do and do not determine", "location": "p. 501 (PDF p. 10), last paragraph; supplement Table A7, p. A11" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T10" }, { "declarations": [ { "claim_revision_id": "catalog:2b15aa7802f3672e41b20ba33940f98936ca18b42f46a299b0de23cb54978503", "conditional_on": [], "credit_reason": "The target explicitly requests population SEM conditions and a finite-law witness. Positive variances make the correlation meaningful; taking iid samples of this bounded law realizes the sample moment limits, without needing that separate convergence theorem in this witness row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sem_defier_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1448", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:2b15aa7802f3672e41b20ba33940f98936ca18b42f46a299b0de23cb54978503", "conditional_on": [], "credit_reason": "The target explicitly requests population SEM conditions and a finite-law witness. The conditional means and positive variances make the exogeneity claim nondegenerate; independence fails because |Y0| determines Z.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sem_dependence_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1499", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:2b15aa7802f3672e41b20ba33940f98936ca18b42f46a299b0de23cb54978503", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T11", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T11" }, "html": { "body": "

Section 2.2, p. 495 (PDF p. 4), comparison with the Section 2.1 SEM framework. Both explicit constant-effect SEM laws have the stated exogeneity, relevance and endogeneity moments, while one has defiers and the other lacks instrument/potential-outcome independence.

", "headline": "Section 2.2, ‘these assumptions are considerably weaker than (and subsumed by) those of the SEM framework’; the witnesses are the validator's and are not in the paper", "location": "p. 495 (PDF p. 4), last sentence of the paragraph listing the five IV assumptions; the SEM conditions are on pp. 493–494 (PDF pp. 2–3)" }, "id": "P029-T11", "text": { "body": "Section 2.2, p. 495 (PDF p. 4), comparison with the Section 2.1 SEM framework. Both explicit constant-effect SEM laws have the stated exogeneity, relevance and endogeneity moments, while one has defiers and the other lacks instrument/potential-outcome independence.", "headline": "Section 2.2, ‘these assumptions are considerably weaker than (and subsumed by) those of the SEM framework’; the witnesses are the validator's and are not in the paper", "location": "p. 495 (PDF p. 4), last sentence of the paragraph listing the five IV assumptions; the SEM conditions are on pp. 493–494 (PDF pp. 2–3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T11" }, { "declarations": [ { "claim_revision_id": "catalog:111b4f9f4b92bcd967bda7f012e87f224f8a16f318563caf92cbf67138b5cc9d", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"P029/P029_Audit.lean:1733 states FIV, heterogeneous effects, ATE = 3/2 and Wald = 3/2, but does not include the LATE = 3/2 equality in T12(c) and the row description. This is a literal missing conclusion, but its mathematical extent is trivial: heterogeneousUnit at P029/P029_Audit.lean:1727 makes every unit a complier, and FIV supplies a normalized probability law. Thus fCE at P029/P029_Audit.lean:70 reduces that LATE to the already stated fE divided by 1, with positive conditioning mass. The heterogeneity witness is complete, so the reviewer's major assessment overstates this omission.\", 'suggested_fix': \"Append fCE (randomized (fun _ : Bool => 1/2) (1/2)) (fun i => (heterogeneousUnit i.1).tau) (fun i => (heterogeneousUnit i.1).complier) = 3/2 to the theorem's conjunction; the conditioning event has probability 1.\", 'second_review': 'T12(c) explicitly includes E(tau | D1 > D0) = 3/2, whereas P029/P029_Audit.lean:1733 exports the IV assumptions, heterogeneous effects, ATE and Wald values without this conditional-mean conclusion. Every heterogeneousUnit is a complier by P029/P029_Audit.lean:1727, and fCE at P029/P029_Audit.lean:70 therefore reduces to the exported expectation divided by total probability one. Appending the LATE equality is a trivial statement repair; the heterogeneity witness itself is complete.', 'severity': 'minor'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.heterogeneous_iv_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1546", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:111b4f9f4b92bcd967bda7f012e87f224f8a16f318563caf92cbf67138b5cc9d", "conditional_on": [], "credit_reason": "This is the witness component T12(a) specifies, not a claim to prove every corrected comparison alone. The zero correlation has two strictly positive variances; the target states population moment conditions, so no additional sampling-limit theorem is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sem_defier_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1448", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:111b4f9f4b92bcd967bda7f012e87f224f8a16f318563caf92cbf67138b5cc9d", "conditional_on": [], "credit_reason": "The counterexample violates random assignment, not mean exogeneity. The positive variances prevent totalized correlation from supplying a false zero. This is a witness for the relevant corrected-statement component; its status counterexample_proved is consistent with that purpose.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sem_dependence_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1499", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:111b4f9f4b92bcd967bda7f012e87f224f8a16f318563caf92cbf67138b5cc9d", "conditional_on": [], "credit_reason": "The common-error and common-covariate hypotheses are exactly the causal potential-outcome reading explicitly required by T12(b), not an inference from an arbitrary observational linear projection.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sem_forces_constant_effect", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1532", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:111b4f9f4b92bcd967bda7f012e87f224f8a16f318563caf92cbf67138b5cc9d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T12", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T12" }, "html": { "body": "

Section 2.2, p. 495 (PDF p. 4), corrected comparison with Section 2.1. The SEM witnesses and the heterogeneous all-complier IV law establish non-nesting, while the SEM potential-outcome equations force a constant effect and the IV witness has ATE and LATE 3/2.

", "headline": "Section 2.2, comparison of the five IV assumptions with the SEM assumptions, corrected: the two sets are distinct and non-nested; the potential-outcome assumptions permit heterogeneous effects", "location": "p. 495 (PDF p. 4), last sentence of the paragraph listing the five IV assumptions" }, "id": "P029-T12", "text": { "body": "Section 2.2, p. 495 (PDF p. 4), corrected comparison with Section 2.1. The SEM witnesses and the heterogeneous all-complier IV law establish non-nesting, while the SEM potential-outcome equations force a constant effect and the IV witness has ATE and LATE 3/2.", "headline": "Section 2.2, comparison of the five IV assumptions with the SEM assumptions, corrected: the two sets are distinct and non-nested; the potential-outcome assumptions permit heterogeneous effects", "location": "p. 495 (PDF p. 4), last sentence of the paragraph listing the five IV assumptions" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T12" }, { "declarations": [ { "claim_revision_id": "catalog:45ff9b332e1caed34961d2a9c8935acc3527770d7a463456a97abed7741a9bc2", "conditional_on": [], "credit_reason": "The selected saturated_ml_bias_full_counterexample already exports the population IV, latent-ignorability and overlap facts needed here.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.constant_score_counterexample_law", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1736", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:45ff9b332e1caed34961d2a9c8935acc3527770d7a463456a97abed7741a9bc2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P029_Audit.lean:1844 gives exactly T13(iv)'s positive-part-score expectation 15659/64000, bias 2859/64000, and inverted-count probability 257079/1600000. The product-binomial law and armRateScore definitions encode the stated independent counts and max(k1/5 - k0/5, 0). This row covers that exact enumeration component; T13(iv) itself specifies the score formula and references T14(a) for its ML characterization, so no ML-identification conclusion is omitted here.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.five_per_arm_bias_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1642", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:45ff9b332e1caed34961d2a9c8935acc3527770d7a463456a97abed7741a9bc2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P029_Audit.lean:2970 combines the valid constant-effect IV population, its true score 1/2 and receipt rates 3/4 and 1/4, globally maximizing feasible fits, and their expectation 9/16. T13's own Reading explicitly admits the closed triangle 0 <= q <= p <= 1 and the 0^0 = 1 convention, so these are not restrictions disclosed only in metadata. The claim specifies the maximizing pairs rather than requiring a separate uniqueness theorem, and this counterexample loses none of its stated cases.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.saturated_ml_bias_full_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2547", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:45ff9b332e1caed34961d2a9c8935acc3527770d7a463456a97abed7741a9bc2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1817 gives the prescribed fitted-score values, expectation 9/16, bias 1/16, and unbiased raw-difference expectation 1/2 under the independent two-receipt law, matching T13(ii)-(iii). Its score is defined from the same fitted pairs whose actual closed-likelihood maximality is established by the companion declaration at P029_Audit.lean:1796. The closure convention appears in T13 itself, so evaluating those boundary pairs does not replace the target with another estimator.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.saturated_unbiasedness_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1615", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:45ff9b332e1caed34961d2a9c8935acc3527770d7a463456a97abed7741a9bc2", "conditional_on": [], "credit_reason": "The selected saturated_ml_bias_full_counterexample already exports maximality for every two-receipt data set.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.two_observation_ml_maxima", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1594", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:45ff9b332e1caed34961d2a9c8935acc3527770d7a463456a97abed7741a9bc2", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T13", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T13" }, "html": { "body": "

Assumption 2, p. 500 (PDF p. 9), with the estimator on pp. 496–497. The one-observation-per-arm closed-domain likelihood maxima give a biased compliance score, with the stated certificates, raw-difference expectation and five-per-arm positive-part example; boundary probabilities and the natural-power convention are part of the claim.

", "headline": "Assumption 2, last sentence: ‘In the case of a saturated model and F(.)F(.)F(.) as the normal CDF, P^Ci\\hat{P}_{Ci}P^Ci​ will be unbiased for the conditional probability of compliance’; the witness is the validator's and is not in the paper", "location": "p. 500 (PDF p. 9), Assumption 2; the estimator is defined on pp. 496–497 (PDF pp. 5–6)" }, "id": "P029-T13", "text": { "body": "Assumption 2, p. 500 (PDF p. 9), with the estimator on pp. 496–497. The one-observation-per-arm closed-domain likelihood maxima give a biased compliance score, with the stated certificates, raw-difference expectation and five-per-arm positive-part example; boundary probabilities and the natural-power convention are part of the claim.", "headline": "Assumption 2, last sentence: ‘In the case of a saturated model and $F(.)$ as the normal CDF, $\\hat{P}_{Ci}$ will be unbiased for the conditional probability of compliance’; the witness is the validator's and is not in the paper", "location": "p. 500 (PDF p. 9), Assumption 2; the estimator is defined on pp. 496–497 (PDF pp. 5–6)" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T13" }, { "declarations": [ { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "conditional_on": [], "credit_reason": "This is the interior parameterization component, not independently the likelihood-maximization certificate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.nested_receipt_parameters", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2004", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "conditional_on": [], "credit_reason": "The corrected prose mentions only a nonempty treated arm, but its detailed mathematical model expressly fixes n0>=1 as well; this row matches that detailed component.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.one_sided_saturated_unbiasedness_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B1.lean#L275", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "conditional_on": [], "credit_reason": "The binomial bias row already supplies the target-specific identity and sign.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.positive_part_bias_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1662", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "conditional_on": [], "credit_reason": "Harmless generalization; substituting the empirical rate difference and true compliance gives exactly the target contraction.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.positive_part_error_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1651", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "conditional_on": [], "credit_reason": "This row is a proof of the corrected finite bias/bound component, not the printed unrestricted unbiasedness assertion. The separate contraction and asymptotic conclusions are covered by R193-07 and R193-04.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.saturated_binomial_bias_bound_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B1.lean#L444", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "conditional_on": [ "C. van Eeden, \"Maximum likelihood estimation of ordered probabilities\", Proc. KNAW Ser. A 59 = Indagationes Math. 18 (1956), Theorem I, p. 445" ], "credit_reason": "Its proof requires an external ordered-binomial theorem as a hypothesis; the supplied unconditional whole proof establishes optimization instead.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.saturated_ml_characterization_conditional", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Cond/CondP027P029Proofs.lean#L92", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "conditional_on": [], "credit_reason": "The target metadata says asymptotic=false even though T14(c) expressly includes this asymptotic conclusion; the mathematical target and row purpose are consistent. Positivity at each sample size follows the detailed positive-arm model.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.saturated_score_consistency_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B2b.lean#L36", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c0f468ecca687ce1f02a952ec90ce68ba180aad65fb0ba07208079c6733a4301", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T14", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T14" }, "html": { "body": "

p. 500, Assumption 2; pp. 496–497, Section 3.1. With boundary probabilities allowed, the saturated ordered binomial likelihood has the unique separate-rate or pooled maximizer and compliance score max(r1-r0,0). The interior nested parameters, finite bias and extreme-event probability, contraction and tail bound, per-stratum and simultaneous finite-stratum consistency, and one-sided unbiasedness (requiring only a nonempty treated arm) all hold. The usual measure-theoretic conditions are assumed.

", "headline": "Assumption 2, last sentence, and Section 3.1 ‘strata means under saturation’, corrected: the saturated ML compliance score is the receipt-rate difference truncated at zero; it is consistent, biased upward under two-sided noncompliance and unbiased under one-sided noncompliance", "location": "p. 500 (PDF p. 9), Assumption 2; p. 496 (PDF p. 5), Section 3.1, third paragraph" }, "id": "P029-T14", "text": { "body": "p. 500, Assumption 2; pp. 496–497, Section 3.1. With boundary probabilities allowed, the saturated ordered binomial likelihood has the unique separate-rate or pooled maximizer and compliance score max(r1-r0,0). The interior nested parameters, finite bias and extreme-event probability, contraction and tail bound, per-stratum and simultaneous finite-stratum consistency, and one-sided unbiasedness (requiring only a nonempty treated arm) all hold. The usual measure-theoretic conditions are assumed.", "headline": "Assumption 2, last sentence, and Section 3.1 ‘strata means under saturation’, corrected: the saturated ML compliance score is the receipt-rate difference truncated at zero; it is consistent, biased upward under two-sided noncompliance and unbiased under one-sided noncompliance", "location": "p. 500 (PDF p. 9), Assumption 2; p. 496 (PDF p. 5), Section 3.1, third paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T14" }, { "declarations": [ { "claim_revision_id": "catalog:7519b0943f75f83f2ac1ba931eb99a3b64920782d6f603b1b542419edf75fb00", "conditional_on": [], "credit_reason": "Harmless generalization of the constant-sample auxiliary lemma. This row alone is not credited as a complete population counterexample.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.constant_quantile", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1701", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7519b0943f75f83f2ac1ba931eb99a3b64920782d6f603b1b542419edf75fb00", "conditional_on": [], "credit_reason": "This is the population-law component of the witness. The cutoff nonconvergence and oracle-score sequence are separately exported in R193-10 and R193-11; fScore=1/2 already entails zero error for scores equal to 1/2.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.constant_score_counterexample_law", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1736", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7519b0943f75f83f2ac1ba931eb99a3b64920782d6f603b1b542419edf75fb00", "conditional_on": [], "credit_reason": "This is the quantile/nonconvergence component, paired with the valid population law in R193-09/R193-11. T15's asymptotic=false metadata does not reflect its explicit cutoff nonconvergence statement.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.cutoff_not_zero", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1720", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:7519b0943f75f83f2ac1ba931eb99a3b64920782d6f603b1b542419edf75fb00", "conditional_on": [], "credit_reason": "The counterexample refutes the stated cutoff-to-zero rationale, not the claim that Winsorization can preserve consistency. The target explicitly permits oracle scores; it does not require this row to formalize the intercept-only fitted-ML sequence.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.winsorization_cutoff_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2521", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:7519b0943f75f83f2ac1ba931eb99a3b64920782d6f603b1b542419edf75fb00", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T15", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T15" }, "html": { "body": "

p. 498, footnote 6. The constant-score counterexample satisfies the five IV assumptions and Assumptions 1–3, with receipt-arm means 3/4 and 1/4 and ATE and population ICSW equal to one. Both inverse empirical-CDF and type-7 quantiles of its nonempty half-score vectors exist and equal one half, including at every positive power level n^(-alpha). The cutoff therefore does not tend to zero; no scores or weights change. The usual measure-theoretic conditions are assumed.

", "headline": "Footnote 6: ‘Asymptotically, assuming that α\\alphaα remains fixed, the Winsorization has no impact on estimation, because c→0c \\to 0c→0 as n→∞n \\to \\inftyn→∞’; the witness is the validator's and is not in the paper", "location": "p. 498 (PDF p. 7), footnote 6" }, "id": "P029-T15", "text": { "body": "p. 498, footnote 6. The constant-score counterexample satisfies the five IV assumptions and Assumptions 1–3, with receipt-arm means 3/4 and 1/4 and ATE and population ICSW equal to one. Both inverse empirical-CDF and type-7 quantiles of its nonempty half-score vectors exist and equal one half, including at every positive power level n^(-alpha). The cutoff therefore does not tend to zero; no scores or weights change. The usual measure-theoretic conditions are assumed.", "headline": "Footnote 6: ‘Asymptotically, assuming that $\\alpha$ remains fixed, the Winsorization has no impact on estimation, because $c \\to 0$ as $n \\to \\infty$’; the witness is the validator's and is not in the paper", "location": "p. 498 (PDF p. 7), footnote 6" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T15" }, { "declarations": [ { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "The boundedness premise is appropriate for the auxiliary bounded-outcome step; the separate tail inequality and limit theorem cover the integrable-outcome argument, so this row is not credited with the full unbounded-outcome limit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.bounded_weighted_mean_change", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1853", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "Reciprocal-error auxiliary is used within the existing limit argument; it is not a separate claim clause.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.inverse_score_accuracy", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1889", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "Tail-truncation auxiliary is already contained in the existing corrected-limit proof.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.outcome_tail_weight_bound", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1912", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "This statement concerns the fraction changed, not the numerical cutoff; constant positive scores are therefore consistent with its conclusion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.quantile_changed_fraction_tends_zero", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1810", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "An arbitrary finite real vector has a real common lower bound, so supplying b introduces no substantive restriction for the count claim.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.quantile_count_and_lower_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1763", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "Ratio algebra is an auxiliary already subsumed by the corrected-limit proof.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.ratio_perturbation", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1953", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "Harmless generalization of the exact pointwise weight bound; neither the cutoff nor score upper bound is needed for its specialization.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.weight_change_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1836", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "conditional_on": [], "credit_reason": "The expanded ICSWConditions fields do not assume any estimator limit. The printed footnote is being corrected; finite support and uniform consistency occur in T16(iv) itself, so their disclosure is not being used to excuse a narrower theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.winsorization_corrected_limit_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GLP029B1.lean#L429", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c7469f5b866a01d174b259f2d942a1a3f73ff57d2beaca7539fabb836728b641", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T16", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T16" }, "html": { "body": "

p. 498, footnote 6, corrected. The type-7 changed count is at most floor(1+(n-1)lambda), and its fraction is at most lambda+1/n. The inverse empirical-CDF convention obeys the same bounds. At fixed positive power levels both changed fractions vanish. Quantiles preserve common lower bounds; half-score cutoffs exist and stay one half. The pointwise inverse-weight and bounded weighted-mean bounds hold. Under the printed finite- profile iid, overlap, integrability and uniform score consistency conditions, ordinary and Winsorized ICSW have the stated common limit and vanishing difference, with ATE identification under Assumption 1. The usual measure-theoretic conditions, including measurable scores and cutoffs, are assumed. Ratios use their real-valued extension on undefined samples; no eventual-definedness assumption is imposed.

", "headline": "Footnote 6, the asymptotic justification of Winsorization, corrected: the fraction of scores changed tends to zero and, with bounded estimated inverse scores and integrable outcomes, the probability limit is unchanged; the cut-off need not tend to zero", "location": "p. 498 (PDF p. 7), footnote 6" }, "id": "P029-T16", "text": { "body": "p. 498, footnote 6, corrected. The type-7 changed count is at most floor(1+(n-1)lambda), and its fraction is at most lambda+1/n. The inverse empirical-CDF convention obeys the same bounds. At fixed positive power levels both changed fractions vanish. Quantiles preserve common lower bounds; half-score cutoffs exist and stay one half. The pointwise inverse-weight and bounded weighted-mean bounds hold. Under the printed finite- profile iid, overlap, integrability and uniform score consistency conditions, ordinary and Winsorized ICSW have the stated common limit and vanishing difference, with ATE identification under Assumption 1. The usual measure-theoretic conditions, including measurable scores and cutoffs, are assumed. Ratios use their real-valued extension on undefined samples; no eventual-definedness assumption is imposed.", "headline": "Footnote 6, the asymptotic justification of Winsorization, corrected: the fraction of scores changed tends to zero and, with bounded estimated inverse scores and integrable outcomes, the probability limit is unchanged; the cut-off need not tend to zero", "location": "p. 498 (PDF p. 7), footnote 6" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T16" }, { "declarations": [ { "claim_revision_id": "catalog:d4a029c35c7d0be30d2fab0c210831a7d019a700b494b9adb84b3afe6089b0e8", "conditional_on": [], "credit_reason": "Equality is stronger than equality of maximizer sets: for arbitrary theta, substitute pi=pi(theta) into the theorem. Thus the unexported argmax notation does not omit mathematical information. No final verdict change from claim-stage.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.one_sided_likelihood_reduction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1966", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d4a029c35c7d0be30d2fab0c210831a7d019a700b494b9adb84b3afe6089b0e8", "conditional_on": [], "credit_reason": "Harmless extension from probability profiles to arbitrary real pi. This is a substitution lemma in the corrected likelihood argument, not a claim that values outside [0,1] define probabilities.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.one_sided_unit_likelihood", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1959", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d4a029c35c7d0be30d2fab0c210831a7d019a700b494b9adb84b3afe6089b0e8", "conditional_on": [], "credit_reason": "This is an exact witness to the printed formula error, not a quantified theorem giving its general value for every pi; that remaining part of T17(c) is recorded as uncovered in targets.json.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.printed_one_sided_factor_countercheck", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1981", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:d4a029c35c7d0be30d2fab0c210831a7d019a700b494b9adb84b3afe6089b0e8", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T17", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T17" }, "html": { "body": "

p. 497, footnote 3 and equation (7), corrected. The one-sided nested Bernoulli likelihood reduces to the treated-arm likelihood, and their maximizer predicates agree. For every interior probability the printed additive factor has the stated two formulas and is at least one; at three quarters it gives seven quarters and five quarters instead of the Bernoulli probabilities. The usual measure-theoretic conditions are assumed.

", "headline": "Footnote 3, the likelihood under one-sided noncompliance, corrected (product of the two Bernoulli factors), and the reduction to binomial regression on the units assigned to treatment", "location": "p. 497 (PDF p. 6), footnote 3; equation (7) on the same page" }, "id": "P029-T17", "text": { "body": "p. 497, footnote 3 and equation (7), corrected. The one-sided nested Bernoulli likelihood reduces to the treated-arm likelihood, and their maximizer predicates agree. For every interior probability the printed additive factor has the stated two formulas and is at least one; at three quarters it gives seven quarters and five quarters instead of the Bernoulli probabilities. The usual measure-theoretic conditions are assumed.", "headline": "Footnote 3, the likelihood under one-sided noncompliance, corrected (product of the two Bernoulli factors), and the reduction to binomial regression on the units assigned to treatment", "location": "p. 497 (PDF p. 6), footnote 3; equation (7) on the same page" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T17" }, { "declarations": [ { "claim_revision_id": "catalog:dcb69c09610188420418083f8fd1df5efab6fb4e195ccdbe53f7c4eda32caba8", "conditional_on": [], "credit_reason": "Pointwise Boolean auxiliary fact; the target’s zero-defier-probability assumption makes it applicable almost everywhere.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.compliance_type_partition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2015", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dcb69c09610188420418083f8fd1df5efab6fb4e195ccdbe53f7c4eda32caba8", "conditional_on": [], "credit_reason": "Auxiliary pointwise identity, sufficient for the integrand cancellation rather than a standalone consistency theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.corrected_equation_nine_integrand", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2023", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dcb69c09610188420418083f8fd1df5efab6fb4e195ccdbe53f7c4eda32caba8", "conditional_on": [], "credit_reason": "This is the corrected target, including boundary nested probabilities where the target identities permit them; no claim about the literal misprinted conditioning is made.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.corrected_nested_receipt_model_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B2b.lean#L141", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dcb69c09610188420418083f8fd1df5efab6fb4e195ccdbe53f7c4eda32caba8", "conditional_on": [], "credit_reason": "The corrected target itself includes both interior-probability premises.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.nested_probability_range", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L1989", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dcb69c09610188420418083f8fd1df5efab6fb4e195ccdbe53f7c4eda32caba8", "conditional_on": [], "credit_reason": "The actual conditional-probability and positive-AC identities are supplied by the whole proof; this separate algebraic parameterization lemma is unnecessary.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.nested_receipt_parameters", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2004", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:dcb69c09610188420418083f8fd1df5efab6fb4e195ccdbe53f7c4eda32caba8", "conditional_on": [], "credit_reason": "The fixed witness suffices for the row’s countercheck; this is not credited as the full universal corrected probability theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.receipt_conditioning_countercheck", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2438", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:dcb69c09610188420418083f8fd1df5efab6fb4e195ccdbe53f7c4eda32caba8", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T18", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T18" }, "html": { "body": "

pp. 496–497, Section 3.1; p. 504, equation (9) and the estimator display. Under random assignment and no defiers, the type partition, conditional compliance and receipt formulas, positive-AC nested representation, and interior probability range hold. Every law with no always-takers, conditional compliance probability one half at the profile, and assignment probability one half gives the printed misconditioning discrepancy. With existing weighted expectations and positive compliance almost surely, weighted ITT equals the corrected covariate-law integral in (9), which cancels to (10). The single-hat sample display equals Section 3.3. The usual measure-theoretic conditions are assumed, including measurable versions of the conditional means and integrability of the displayed expectations. No uniform positive lower bound on compliance is required for the integral identity.

", "headline": "Section 3.1 (the second definition of the compliance score and the receipt-probability display) and Appendix (equation (9) and the estimator display), corrected notation", "location": "p. 496 (PDF p. 5), sentence after equation (6); p. 497 (PDF p. 6), first display and the sentence after it; p. 504 (PDF p. 13), equation (9) and the display after equation (14)" }, "id": "P029-T18", "text": { "body": "pp. 496–497, Section 3.1; p. 504, equation (9) and the estimator display. Under random assignment and no defiers, the type partition, conditional compliance and receipt formulas, positive-AC nested representation, and interior probability range hold. Every law with no always-takers, conditional compliance probability one half at the profile, and assignment probability one half gives the printed misconditioning discrepancy. With existing weighted expectations and positive compliance almost surely, weighted ITT equals the corrected covariate-law integral in (9), which cancels to (10). The single-hat sample display equals Section 3.3. The usual measure-theoretic conditions are assumed, including measurable versions of the conditional means and integrability of the displayed expectations. No uniform positive lower bound on compliance is required for the integral identity.", "headline": "Section 3.1 (the second definition of the compliance score and the receipt-probability display) and Appendix (equation (9) and the estimator display), corrected notation", "location": "p. 496 (PDF p. 5), sentence after equation (6); p. 497 (PDF p. 6), first display and the sentence after it; p. 504 (PDF p. 13), equation (9) and the display after equation (14)" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T18" }, { "declarations": [ { "claim_revision_id": "catalog:62ba8560c3de212c3251f11774ee93c6f42eb8ab7b35496b8d5e1c3cf0bdedf7", "conditional_on": [], "credit_reason": "The target's mathematical content explicitly quantifies any real latent indices, so the earlier record's claim of a stronger arbitrary-index domain is not a difference from this component. Noise laws and balanced assignment are recipe specifications and immaterial to these pointwise conclusions.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.corrected_simulation_types", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2048", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:62ba8560c3de212c3251f11774ee93c6f42eb8ab7b35496b8d5e1c3cf0bdedf7", "conditional_on": [], "credit_reason": "The selected refutation is exact for binary A,Z,D; using real D in the impossibility statement strengthens it without changing the printed reading. Both required binary solutions are exported in the nonunique case.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.printed_simulation_equation_failure", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2064", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:62ba8560c3de212c3251f11774ee93c6f42eb8ab7b35496b8d5e1c3cf0bdedf7", "conditional_on": [], "credit_reason": "This is only T19(iv)'s observed-outcome algebra, with the potential-outcome substitution Y1=Y0+τ; it does not purport to prove a stochastic exclusion assumption from observational data.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.simulation_outcome_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2076", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:62ba8560c3de212c3251f11774ee93c6f42eb8ab7b35496b8d5e1c3cf0bdedf7", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T19", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T19" }, "html": { "body": "

supplement pp. A1-A2, baseline and misspecified simulation designs. Independent uniform covariates and shared coefficients, normal errors of standard deviation three (baseline), logistic errors of location zero and scale three (nonlinear design), standard normal outcome errors, and independent uniformly balanced assignment realize the corrected recipes for every even sample size. Thresholded indices give binary disjoint types, monotone potential treatments and the stated type characterizations. The corrected treatment is the realized potential treatment; an assigned complier receives treatment. The printed treatment equation can have no solution or two solutions. The printed outcome and effect formulas obey the potential-outcome identity. The usual measure-theoretic conditions are assumed.

", "headline": "Supplement, ‘Baseline Data Generating Process’ and ‘Heterogeneous Treatment Effects, Model is Incorrect’, corrected: the treatment equation Di=CiZi+AiD_i = C_iZ_i + A_iDi​=Ci​Zi​+Ai​ and the threshold tests in the latent-index indicators", "location": "Supplement p. A1 (lines for ACiAC_iACi​, AiA_iAi​, CiC_iCi​, DiD_iDi​) and p. A2 (lines for ACiAC_iACi​, AiA_iAi​ of ‘Model is Incorrect’)" }, "id": "P029-T19", "text": { "body": "supplement pp. A1-A2, baseline and misspecified simulation designs. Independent uniform covariates and shared coefficients, normal errors of standard deviation three (baseline), logistic errors of location zero and scale three (nonlinear design), standard normal outcome errors, and independent uniformly balanced assignment realize the corrected recipes for every even sample size. Thresholded indices give binary disjoint types, monotone potential treatments and the stated type characterizations. The corrected treatment is the realized potential treatment; an assigned complier receives treatment. The printed treatment equation can have no solution or two solutions. The printed outcome and effect formulas obey the potential-outcome identity. The usual measure-theoretic conditions are assumed.", "headline": "Supplement, ‘Baseline Data Generating Process’ and ‘Heterogeneous Treatment Effects, Model is Incorrect’, corrected: the treatment equation $D_i = C_iZ_i + A_i$ and the threshold tests in the latent-index indicators", "location": "Supplement p. A1 (lines for $AC_i$, $A_i$, $C_i$, $D_i$) and p. A2 (lines for $AC_i$, $A_i$ of ‘Model is Incorrect’)" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T19" }, { "declarations": [ { "claim_revision_id": "catalog:895bba0522208348b3f95dfc90bea1c30e759f3766faa96643826779b8d5d85f", "conditional_on": [], "credit_reason": "This is the valid existence example for latent ignorability without typical ignorability, not a positive proof of the false I-implies-L ordering.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.baseline_selection_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L230", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:895bba0522208348b3f95dfc90bea1c30e759f3766faa96643826779b8d5d85f", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029/Wave1/IgnorabilityOrdering.lean:16 negates the universal implication from the two printed received-treatment mean equalities to latent ignorability, the component of the claim saying that Assumption 1 is \"less restrictive\" than typical ignorability. The positive-mass and positive-score guards occur inside the negated universal: establishing a failure even on this smaller class suffices to refute the printed all-support-profile implication. The definitions in P029/P029_Audit.lean:417 and :436 use the stated IV assumptions and treatment received D, and the finite law at :181 has constant X, profile mass 1, compliance score 1/2, both received-treatment arms of mass 1/2, ordinary conditional effect 1/2 and complier effect 1. Thus neither zero-probability conditioning nor universe size invalidates the counterexample; refuting the forward implication suffices without refuting the separate baseline-selection example.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.printed_ignorability_ordering_refuted", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/IgnorabilityOrdering.lean#L16", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:895bba0522208348b3f95dfc90bea1c30e759f3766faa96643826779b8d5d85f", "conditional_on": [], "credit_reason": "The direct positive-profile universal ordering is already refuted by the selected measure-theoretic row; a second kernel presentation is unnecessary.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.printed_ignorability_ordering_all_profiles_refuted", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/IgnorabilityAllProfiles.lean#L16", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:895bba0522208348b3f95dfc90bea1c30e759f3766faa96643826779b8d5d85f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T20", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T20" }, "html": { "body": "

The printed claim is false: P029-T20; p. 499, Assumption 1 and the following paragraph. Typical mean ignorability need not imply latent ignorability under the IV assumptions, and a separate baseline-selection law satisfies latent ignorability with unequal received-treatment potential-outcome means.

", "headline": "Assumption 1 and the paragraph comparing it with the typical ignorability assumption, as printed", "location": "p. 499 (PDF p. 8), Assumption 1 and the following paragraph" }, "id": "P029-T20", "text": { "body": "The printed claim is false: P029-T20; p. 499, Assumption 1 and the following paragraph. Typical mean ignorability need not imply latent ignorability under the IV assumptions, and a separate baseline-selection law satisfies latent ignorability with unequal received-treatment potential-outcome means.", "headline": "Assumption 1 and the paragraph comparing it with the typical ignorability assumption, as printed", "location": "p. 499 (PDF p. 8), Assumption 1 and the following paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T20" }, { "declarations": [ { "claim_revision_id": "catalog:b790a39f19aa28dbb96ca420b083c4a5db516bfede86e8fb75ed529de8c010e3", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P029/Wave1/Gap/GP029B2b.lean:580 requires a common epsilon > 0 with epsilon < c(X) almost surely, whereas T21(a) says constant effects give the conditional means and both population limits equal to tau without imposing uniform overlap. This hypothesis excludes nontrivial constant-effect populations: take Pr(X=k)=3*4^(-(k+1)), c(k)=2^(-(k+1)), Y0=0 and Y1=tau, with randomized assignment and one-sided compliance; E[1/c(X)]=3 and both population Wald ratios are tau, but no such epsilon exists. Applying the same global bound to the conditional-mean and ordinary-Wald conclusions also restricts conclusions that do not need it. The positive-mass event formulation at P029/Wave1/Gap/GP029B2b.lean:582 and the almost-sure conditional-expectation version are legitimate qualifications for null conditioning events, rather than a further major defect.', 'suggested_fix': 'Separate the constant-effect conditional-mean and ordinary-Wald conclusions from the inverse-score result. State conditional equalities on positive-score profiles without a global epsilon; for the inverse-score Wald identity use almost-sure positive scores and finiteness of the required inverse-weighted moments instead of a uniform lower bound.', 'second_review': 'T21(a) asserts the constant-effect conditional means and both population limits equal tau without requiring uniformly positive compliance scores. P029/Wave1/Gap/GP029B2b.lean:580 instead requires one epsilon > 0 below c(X) almost surely, restricting even its conditional-mean and ordinary-Wald conclusions. For example, profile masses 3*4^(-(k+1)) and scores 2^(-(k+1)) have finite mean inverse score 3 but no uniform positive lower bound, so substantive cases with well-defined inverse-weighted moments are excluded. The statement should separate the conclusions and use positivity and the necessary finite weighted moments for the inverse-score result.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.Wave1.constant_effect_population_limits_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B2b.lean#L564", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:b790a39f19aa28dbb96ca420b083c4a5db516bfede86e8fb75ed529de8c010e3", "conditional_on": [], "credit_reason": "The conditional reading is explicitly a separate part of the target; the original informal wording is ambiguous between this and the marginal reading. Bounded outcomes and finite positive overlap make the population inequality a regular counterexample to the specified limit assertion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.printed_conditional_sufficiency_refuted", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/ConditionalSufficiency.lean#L19", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:b790a39f19aa28dbb96ca420b083c4a5db516bfede86e8fb75ed529de8c010e3", "conditional_on": [], "credit_reason": "The record’s marginal/conditional distinction agrees with the independent reading; no constant-effect assertion is refuted.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.printed_marginal_sufficiency_refuted", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/MarginalSufficiency.lean#L17", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:b790a39f19aa28dbb96ca420b083c4a5db516bfede86e8fb75ed529de8c010e3", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T21", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T21" }, "html": { "body": "

The printed claim is false: P029-T21; p. 499, the sufficient-conditions paragraph. Both printed independence implications fail, while constant effects give the stated population identities under uniform positive overlap. The constant-effect component adds measurable conditional-mean versions and a uniform positive lower bound for the compliance score, and asserts the profile identities only at profiles of positive probability and positive score. It does not assert the same constant-effect result under positivity without that uniform bound, or sample limits. The marginal refutation is stated for independence of the complier indicator.

", "headline": "Section 3.4, the paragraph on sufficient conditions for Assumption 1 (constant effects; effects independent of compliance), as printed", "location": "p. 499 (PDF p. 8), paragraph beginning ‘We can describe sufficient conditions for and consequences of Assumption 1’" }, "id": "P029-T21", "text": { "body": "The printed claim is false: P029-T21; p. 499, the sufficient-conditions paragraph. Both printed independence implications fail, while constant effects give the stated population identities under uniform positive overlap. The constant-effect component adds measurable conditional-mean versions and a uniform positive lower bound for the compliance score, and asserts the profile identities only at profiles of positive probability and positive score. It does not assert the same constant-effect result under positivity without that uniform bound, or sample limits. The marginal refutation is stated for independence of the complier indicator.", "headline": "Section 3.4, the paragraph on sufficient conditions for Assumption 1 (constant effects; effects independent of compliance), as printed", "location": "p. 499 (PDF p. 8), paragraph beginning ‘We can describe sufficient conditions for and consequences of Assumption 1’" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T21" }, { "declarations": [ { "claim_revision_id": "catalog:2ae5fe95044c74448fe9e0f349476851ea7360ad97bd05945528cbdede3598b4", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P029/P029_Audit.lean:1018 states exactly the target's scoring formula sqrt(sum_r (t_r - 5)^2 / 2500), using the ordinary RMSE definition at P029/P029_Audit.lean:986. The component allocated to this row is the scoring rule, not verification of the historical numerical entries, and an arbitrary t includes every possible vector of simulated estimates. Unfolding a correct definition for this definitional component is not a vacuity caused by an empty domain, zero denominator, or an artificial convention that changes the paper's quantity. The fixed replicate count is positive and the homogeneous benchmark is exactly the target's 5, so the flag identifies no fidelity defect.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.homogeneous_rmse_rule", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L872", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:2ae5fe95044c74448fe9e0f349476851ea7360ad97bd05945528cbdede3598b4", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T22", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T22" }, "html": { "body": "

supplement p. A2 (design), p. A5 (text), p. A6 (Figure A3, Noise (Homogeneous) panel), and p. A7 (Table A3). For a supplied array of 2500 estimates and a supplied benchmark of five, empirical RMSE is the square root of the average squared error about that benchmark.

", "headline": "Supplement Table A3 ‘Noise (Homogeneous)’ with the design it reports on, as printed", "location": "Supplement p. A2 (design), p. A5 (text), p. A6 (Figure A3, panel ‘Noise (Homogeneous)’), p. A7 (Table A3)" }, "id": "P029-T22", "text": { "body": "supplement p. A2 (design), p. A5 (text), p. A6 (Figure A3, Noise (Homogeneous) panel), and p. A7 (Table A3). For a supplied array of 2500 estimates and a supplied benchmark of five, empirical RMSE is the square root of the average squared error about that benchmark.", "headline": "Supplement Table A3 ‘Noise (Homogeneous)’ with the design it reports on, as printed", "location": "Supplement p. A2 (design), p. A5 (text), p. A6 (Figure A3, panel ‘Noise (Homogeneous)’), p. A7 (Table A3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T22" }, { "declarations": [ { "claim_revision_id": "catalog:1a87066719220074f22fbbd3554184984a4f1be4b84c393e0561b49c2d6ed0ad", "conditional_on": [], "credit_reason": "The record additionally mentions likelihood clipping to [10^-5,1-10^-5]. Both displayed fits lie in its interior and cellwise likelihood optima remain optimal against clipped competitors, so that fact does not change the witness verdict. Packaging/build checks are outside this semantic review.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.nested_bernoulli_package_weighted_fit_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/FidP029.lean#L20", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1a87066719220074f22fbbd3554184984a4f1be4b84c393e0561b49c2d6ed0ad", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T23", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T23" }, "html": { "body": "

The printed claim is false: P029-T23; p. 498, Section 3.3; package compliance.score.R, lines 88-91, and icsw.tsls.R, lines 28-37. An exact minimizer of the package's inside-log criterion in an interior nested Bernoulli model with positive observation weights does not minimize the weighted negative log likelihood. This refutes the exact observation-weight fitting contract, and makes no assertion against the article's bootstrap recommendation or about numerical solver performance.

", "headline": "Section 3.3, the recommendation to bootstrap the entire process, with its implementation in the R package icsw 1.0.0 (help text and source), as printed", "location": "p. 498 (PDF p. 7), Section 3.3, last paragraph; staged package: packages/icsw/icsw/man/icsw.tsls.Rd, lines 9–13 and 42–43; R/compliance.score.R, lines 43–48 and 88–91; R/icsw.tsls.R, lines 28–37; R/icsw.tsls.fit.R, lines 8 and 21–24" }, "id": "P029-T23", "text": { "body": "The printed claim is false: P029-T23; p. 498, Section 3.3; package compliance.score.R, lines 88-91, and icsw.tsls.R, lines 28-37. An exact minimizer of the package's inside-log criterion in an interior nested Bernoulli model with positive observation weights does not minimize the weighted negative log likelihood. This refutes the exact observation-weight fitting contract, and makes no assertion against the article's bootstrap recommendation or about numerical solver performance.", "headline": "Section 3.3, the recommendation to bootstrap the entire process, with its implementation in the R package icsw 1.0.0 (help text and source), as printed", "location": "p. 498 (PDF p. 7), Section 3.3, last paragraph; staged package: packages/icsw/icsw/man/icsw.tsls.Rd, lines 9–13 and 42–43; R/compliance.score.R, lines 43–48 and 88–91; R/icsw.tsls.R, lines 28–37; R/icsw.tsls.fit.R, lines 8 and 21–24" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T23" }, { "declarations": [ { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Probability-law auxiliary is already included in the existing population identity proof.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.bernoulli_measure_probability", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L474", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Algebraic auxiliary is already included in the sample coefficient identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.bivariate_ratio_algebra", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2160", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Covers only the population equation-(3) component. Outcome integrability is expressly in the target, and positive complier mass follows rather than being assumed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.late_identity_paper_scope", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/LateIdentity.lean#L18", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P029/P029_Audit.lean:2448 states all three numerical ratios in the worked-example component: ATE 1/2, LATE 2/17, and inverse-compliance-score weighted effect 1/2. It also states 0.115 ≤ 2/17 < 0.125, precisely the rounding interpretation expressly specified by the target for the printed 0.12. The denominators are respectively 1, 17/40 and 1, so these are substantive arithmetic equalities, and the row does not claim to establish the other components of T25.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.gender_worked_example", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2200", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Duplicate export of the same population identity; use the paper-scope declaration directly.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.late_identity_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GLP029B2.lean#L25", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Finite-arm auxiliary is already subsumed by the general population identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.randomized_conditional_mean", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2486", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Randomization auxiliary is already included in the existing population identity proof.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.randomized_probability_independent", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/RandomizedProbability.lean#L15", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Finite-law auxiliary is already subsumed by the general population identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.randomized_valid", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L140", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Raw-sum auxiliary is already included in the sample coefficient identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sample_arm_unweighted_raw", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2142", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Centering auxiliary is already included in the sample coefficient identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sample_cross_raw", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2131", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "The target itself defines the bivariate 2SLS component by this centered scalar ratio; an additional matrix/minimizer theorem is therefore not an omitted conclusion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sample_wald_bivariate_2sls", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2175", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "conditional_on": [], "credit_reason": "Unitwise auxiliary is already included in the population identity proof.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.unit_itt_identities", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2508", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f0c56fbadd7971349e549104b68e1d47d0c5dc8c370b8138c8b69e32aea18b96", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T25", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T25" }, "html": { "body": "

p. 495, equations (3)-(4); pp. 497-498, the gender example. The five IV assumptions give both ITT identities and the LATE ratio, the defined sample Wald estimator equals the centered bivariate IV coefficient, and the gender example has values one half, 2/17 (rounding to 0.12), and one half.

", "headline": "Equation (3) (the LATE identity under the five IV assumptions, attributed to Angrist, Imbens, and Rubin 1996), the estimator (4) as bivariate 2SLS, and the worked example of Section 3.2", "location": "p. 495 (PDF p. 4), equations (3) and (4); pp. 497–498 (PDF pp. 6–7), the gender example" }, "id": "P029-T25", "text": { "body": "p. 495, equations (3)-(4); pp. 497-498, the gender example. The five IV assumptions give both ITT identities and the LATE ratio, the defined sample Wald estimator equals the centered bivariate IV coefficient, and the gender example has values one half, 2/17 (rounding to 0.12), and one half.", "headline": "Equation (3) (the LATE identity under the five IV assumptions, attributed to Angrist, Imbens, and Rubin 1996), the estimator (4) as bivariate 2SLS, and the worked example of Section 3.2", "location": "p. 495 (PDF p. 4), equations (3) and (4); pp. 497–498 (PDF pp. 6–7), the gender example" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T25" }, { "declarations": [ { "claim_revision_id": "catalog:534c4149c5d1795f3145af6dce2c86c7c69cd55e951e2bb65c258e9eee16219c", "conditional_on": [], "credit_reason": "Finite-support specialization is unnecessary: the selected population theorem already supplies the general integral identities.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.appendix_finite_identities", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2247", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "catalog:534c4149c5d1795f3145af6dce2c86c7c69cd55e951e2bb65c258e9eee16219c", "conditional_on": [], "credit_reason": "ScoreVersions is a genuine conditional-expectation specification; A1Versions is exactly the optional printed latent-ignorability premise, not an assumption of the weighted arm identity. Uniform overlap is supplied by Assumption 3 here.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.appendix_population_identities_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B3b.lean#L268", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:534c4149c5d1795f3145af6dce2c86c7c69cd55e951e2bb65c258e9eee16219c", "conditional_on": [], "credit_reason": "{'source': 'dispute', 'reason': \"P029/Wave1/Gap/GP029B3b.lean:708 concludes convergence in probability to the ATE, and icswSequence at P029/P029_Audit.lean:2595 expands through sampleWald at P029/P029_Audit.lean:2335 to the Section 3.3 estimator specified by T26. ICSWConditions at P029/P029_Audit.lean:2580 supplies iid sampling, the IV assumptions, actual conditional-expectation versions, overlap, and measurable fitted scores in (0,1], with maximum in-sample score error converging in probability to zero at P029/P029_Audit.lean:2590. Maximum-error convergence is stronger than pointwise Assumption 2 and must not be represented as following from it alone. However, T26 expressly includes 'suitable regularity conditions', and the explicit sufficient specification at P029/P029_Audit.lean:2573 is noncircular and does not assume convergence of the estimator or its sample moments. The additional hypotheses at P029/Wave1/Gap/GP029B3b.lean:714, P029/Wave1/Gap/GP029B3b.lean:716 and P029/Wave1/Gap/GP029B3b.lean:717 represent conditional population effects, Assumption 1 and Assumption 3, so the row correctly proves this qualified consistency claim with the disclosed regularity retained.\", 'defensible_statement': 'For iid full potential-outcome/assignment observations with second-countable Borel covariates, satisfying the binary IV assumptions and integrability of both potential outcomes, assume measurable conditional-mean versions satisfy Assumption 1 and the true compliance score c is bounded away from zero on the covariate support. If for each N the measurable fitted scores s_Ni lie in (0,1] almost surely and max_{1<=i<=N}|s_Ni-c(X_i)| converges in probability to zero, the Section 3.3 ICSW estimator with weights 1/s_Ni converges in probability to E[Y1-Y0].'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.icsw_consistency_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B3b.lean#L703", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:534c4149c5d1795f3145af6dce2c86c7c69cd55e951e2bb65c258e9eee16219c", "conditional_on": [], "credit_reason": "Finite-sum cancellation auxiliary adds no coverage beyond the selected general population/limit statements.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P029.profile_inverse_score_cancellation", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2215", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "catalog:534c4149c5d1795f3145af6dce2c86c7c69cd55e951e2bb65c258e9eee16219c", "conditional_on": [], "credit_reason": "Common-scale identity is an auxiliary, not a separate clause of the retained consistency claim.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sample_arm_scale", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2093", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:534c4149c5d1795f3145af6dce2c86c7c69cd55e951e2bb65c258e9eee16219c", "conditional_on": [], "credit_reason": "Common-scale auxiliary is unnecessary for the selected direct constant-score identity and consistency statements.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sample_wald_scale", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2111", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:534c4149c5d1795f3145af6dce2c86c7c69cd55e951e2bb65c258e9eee16219c", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T26", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T26" }, "html": { "body": "

p. 498, Section 3.4; pp. 504-505, Appendix equations (8)-(14). The population weighted-moment identities hold and ICSW converges to the ATE with uniform consistency of the scores over the growing observed sample.

", "headline": "Consistency of ICSW for the ATE under Assumptions 1–3: Section 3.4 (opening sentences) and the Appendix ‘ICSW Consistency’, equations (8)–(14)", "location": "p. 498 (PDF p. 7), Section 3.4, first paragraph; pp. 504–505 (PDF pp. 13–14), Appendix" }, "id": "P029-T26", "text": { "body": "p. 498, Section 3.4; pp. 504-505, Appendix equations (8)-(14). The population weighted-moment identities hold and ICSW converges to the ATE with uniform consistency of the scores over the growing observed sample.", "headline": "Consistency of ICSW for the ATE under Assumptions 1–3: Section 3.4 (opening sentences) and the Appendix ‘ICSW Consistency’, equations (8)–(14)", "location": "p. 498 (PDF p. 7), Section 3.4, first paragraph; pp. 504–505 (PDF pp. 13–14), Appendix" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T26" }, { "declarations": [ { "claim_revision_id": "catalog:633b9a012efd71dfebf2743880f88d86eb3a195a3b0f6aba12dbacd03d9f4cae", "conditional_on": [], "credit_reason": "ScoreVersions gives defining conditional-expectation identities rather than assuming the Bayes law or the final integral equality. Standard measurability and integrability make the conditional means genuine; overlap is unnecessary for this component.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.complier_covariate_distribution_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GP029B3b.lean#L818", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:633b9a012efd71dfebf2743880f88d86eb3a195a3b0f6aba12dbacd03d9f4cae", "conditional_on": [], "credit_reason": "Finite Bayes auxiliary is subsumed by the selected genuine complier pushforward law.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.complier_profile_bayes", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2274", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:633b9a012efd71dfebf2743880f88d86eb3a195a3b0f6aba12dbacd03d9f4cae", "conditional_on": [], "credit_reason": "The row assumes exact constancy of fitted scores, matching T27(c); it does not claim constancy follows from any fitting algorithm.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.constant_scores_equal_iv", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2120", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:633b9a012efd71dfebf2743880f88d86eb3a195a3b0f6aba12dbacd03d9f4cae", "conditional_on": [], "credit_reason": "{'source': 'dispute', 'reason': \"P029/Wave1/Gap/GLP029B2.lean:44 concludes convergence in probability to the expectation of m(X) under the full population law, exactly the probability-limit component T27(b). ScoreVersions at P029/P029_Audit.lean:2547 identifies c(X) with E[1_C|X] and c(X)m(X) with E[1_C tau|X], so positivity makes m the conditional complier effect and its population expectation is the target's integral against f. The maximum-error condition at P029/P029_Audit.lean:2590 strengthens pointwise score consistency, but T27(b) explicitly qualifies its conclusion by 'suitable regularity conditions', for which ICSWConditions at P029/P029_Audit.lean:2580 supplies a noncircular sufficient specification. Thus the failure of pointwise consistency alone to imply maximum-error consistency does not contradict the qualified target or establish an omitted conclusion. The declaration assumes neither latent ignorability nor its negation, as appropriate for a limit derived without Assumption 1, and the row expressly claims only T27(b), leaving T27(a) and T27(c) outside its credit.\", 'defensible_statement': 'Let c(x) and m(x) be measurable versions of P(C=1|X=x) and E[Y1-Y0|C=1,X=x], with second-countable Borel covariates. For iid full potential-outcome/assignment observations satisfying the binary IV assumptions and integrable potential outcomes, with c bounded away from zero on the covariate support and measurable fitted scores s_Ni in (0,1] almost surely whose maximum in-sample error converges in probability to zero, the Section 3.3 ICSW estimator converges in probability to E[m(X)] under the full population covariate law. This conclusion requires no latent-ignorability assumption.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.Wave1.icsw_complier_effect_limit_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/Wave1/Gap/GLP029B2.lean#L41", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:633b9a012efd71dfebf2743880f88d86eb3a195a3b0f6aba12dbacd03d9f4cae", "conditional_on": [], "credit_reason": "Finite-sum cancellation auxiliary adds no coverage beyond the selected general population/limit statements.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.profile_inverse_score_cancellation", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2215", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:633b9a012efd71dfebf2743880f88d86eb3a195a3b0f6aba12dbacd03d9f4cae", "conditional_on": [], "credit_reason": "Common-scale auxiliary is unnecessary for the selected direct constant-score identity and consistency statements.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P029.sample_wald_scale", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P029/P029_Audit.lean#L2111", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:633b9a012efd71dfebf2743880f88d86eb3a195a3b0f6aba12dbacd03d9f4cae", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P029-T27", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P029-T27" }, "html": { "body": "

The printed claim is false: P029-T27; pp. 499-500, paragraph on failure of Assumption 1. Failure of latent ignorability need not produce asymptotic bias. An iid randomized twelve-atom law has strict positive overlap, interior nested receipt probabilities, heterogeneous effects, and failure of Assumption 1 at both profiles. Its ATE and every observed outcome are zero. Saturated nested ML scores are consistent on an event of probability tending to one, and the zero extension of ICSW converges to the zero ATE. The usual measure-theoretic conditions are satisfied.

", "headline": "The limit of ICSW when Assumption 1 fails (complier effects reweighted to the population covariate distribution), the LATE as an integral against the complier covariate density, and 2SLS as the special case with no covariates", "location": "p. 499 (PDF p. 8), paragraph beginning ‘But what if treatment effects are heterogeneous’ and the two displays; pp. 499–500 (PDF pp. 8–9), the following paragraph; Appendix equations (10) and (13), p. 504 (PDF p. 13)" }, "id": "P029-T27", "text": { "body": "The printed claim is false: P029-T27; pp. 499-500, paragraph on failure of Assumption 1. Failure of latent ignorability need not produce asymptotic bias. An iid randomized twelve-atom law has strict positive overlap, interior nested receipt probabilities, heterogeneous effects, and failure of Assumption 1 at both profiles. Its ATE and every observed outcome are zero. Saturated nested ML scores are consistent on an event of probability tending to one, and the zero extension of ICSW converges to the zero ATE. The usual measure-theoretic conditions are satisfied.", "headline": "The limit of ICSW when Assumption 1 fails (complier effects reweighted to the population covariate distribution), the LATE as an integral against the complier covariate density, and 2SLS as the special case with no covariates", "location": "p. 499 (PDF p. 8), paragraph beginning ‘But what if treatment effects are heterogeneous’ and the two displays; pp. 499–500 (PDF pp. 8–9), the following paragraph; Appendix equations (10) and (13), p. 504 (PDF p. 13)" }, "url": "https://pmaronow.github.io/llm-audit/works/P029/#P029-T27" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "full", "targets": [ { "agreement": "agrees_in_part", "location": "p. 501 (PDF p. 10)", "method": "authors' code with changes: all point estimates and 2,500-draw intervals agree to rounding after correcting the age-missing indicator; with deposited preprocessing, the Age missing coefficients are 97.3 and 96.6 rather than 12.7 and 17.7 (P029-E021)", "status": "reproduced", "target": "Table 1" }, { "agreement": "within_mc_error", "location": "p. 503 (PDF p. 12)", "method": "authors' code with changes: point estimates and all 2,500-draw bootstrap intervals; capped optimizer fits retained as in the script (P029-N010)", "status": "reproduced", "target": "Table 2" }, { "agreement": "to_rounding", "location": "p. 502, first paragraph and footnote 11 (PDF p. 11)", "method": "authors' code with changes: relative increase from unrounded coefficients and paired difference intervals from all 2,500 bootstrap draws, with the age-missing indicator corrected", "status": "reproduced", "target": "Canvassing ATE versus LATE" }, { "agreement": "within_mc_error", "location": "p. 503, text below Table 2 and footnote 13 (PDF p. 12)", "method": "authors' code with changes: relative increase from unrounded coefficients and paired difference intervals from all 2,500 bootstrap draws; capped optimizer fits retained as in the script (P029-N010)", "status": "reproduced", "target": "Debate-viewing ATE versus LATE" } ] }, "summary": "The paper asks how to estimate the average treatment effect for a whole population when instrumental variables estimation recovers only the effect among compliers. It weights units by the inverse of an estimated compliance score, shows consistency under its stated assumptions, and reanalyses two experiments. 23 entries are classified as errors (4 Moderate, 19 Minor), and 3 are clarity notes; 2 of the error entries, \\fid{P029-E006} and \\fid{P029-N006}, concern the same statement of footnote~6; 7 other possible errors were not confirmed, are undetermined or lie outside the paper. The finding that matters most is \\fid{P029-E004}: when treatment effects are marginally independent of compliance, the weighted estimator can be inconsistent for the ATE although two-stage least squares is consistent.", "what_was_checked": "I checked the following through derivations, recalculation from the replication files or comparison with the pages; they held up, subject to the corrections cited.\n\\begin{itemize}\n\\item Equations (3) and (4): under the five assumptions of Section~2.2, the ratio of the two intention-to-treat effects is the LATE, and the sample ratio equals 2SLS without covariates. Footnote~2: with a randomized instrument, 2SLS with covariates is consistent for the LATE.\n\\item Section~3.1: the probabilities in equations (5) to (7) sum to one and give the Bernoulli likelihood. Under one-sided noncompliance the likelihood reduces to a probit on the units assigned to treatment, after the misprint of \\fid{P029-E008} is corrected.\n\\item The worked example on pp.~497--498: the ATE is $0.5$, the LATE is $0.05/0.425 = 2/17 \\approx 0.12$, and the reweighted value is $0.5$.\n\\item Section~3.3 and the Appendix: the weighted ratio agrees with weighted 2SLS. Each step of equations (8) to (14) holds under Assumptions~1 and~3 and regularity conditions, after the corrections of notation in \\fid{P029-E009}. The weighted estimator converges to $E(Y_1 - Y_0)$ when $Z$ is independent of the covariates and the potential outcomes and the scores are consistent.\n\\item Section~3.4: the integral formula for the LATE on p.~499 and the limit of ICSW when Assumption~1 fails, which follows from equations (10) and (13). Constant treatment effects satisfy Assumption~1 and give the same limit for 2SLS and ICSW. Footnote~4: with saturated covariates and a constant share assigned to treatment, ICSW equals the average of the Wald estimates of the strata, weighted by the shares of the strata.\n\\item Table~1: all point estimates follow from the data and the script when the indicator for missing age is set from the raw data (\\fid{P029-E021}). The estimates for contact are $5.61$ and $6.67$, with ratio $1.189$. With the full 2,500 bootstrap draws, I find $5.607$ $(2.230, 8.892)$ and $6.668$ $(3.204, 10.292)$, which agree with the printed intervals to rounding. For the intervals $(-0.2, 2.3)$ and $(0.0, 2.2)$ of footnote~11, I find $(-0.17, 2.32)$ and $(0.01, 2.15)$.\n\\item Table~2: all point estimates follow from the data ($0.276$, $0.400$, $-0.070$ and $-0.050$, with ratio $1.449$). The bootstrap intervals, including those of footnote~13, agree within Monte Carlo error on the full 2,500 draws.\n\\item Sample facts: $n = 18{,}933$ in six cities, with no contact among controls; $n = 507$, where 54.8\\% of those assigned did not watch and 4.4\\% of those not assigned watched.\n\\item Footnotes~6 and~12: the Winsorization in the code is the one described, with quantile levels $n^{-0.275}$ approximately $0.067$ and $0.180$ in the two applications. The reason stated in footnote~6 for ignoring Winsorization in the limit is corrected in \\fid{P029-E006}. The weighting for missing outcomes is as described and is inside the bootstrap.\n\\item The intervals quoted in the text agree with Tables~1 and~2, and footnotes~11 and~13 state that the 95\\% intervals for the differences between the two estimates include zero.\n\\item Supplement: Tables~A7 and~A8 follow from the data, Table~A7 with the indicator for missing age set from the raw data. The simulation designs agree with the code apart from the notation slips (\\fid{P029-E009}), the scale and coefficient-draw omissions (\\fid{P029-A001}) and the benchmark of the homogeneous design (\\fid{P029-E011}). The large-sample limits $0.731$ for 2SLS, $2.49$ for OLS on $Z$ and $0.60$ for ICSW with only $X1$ agree with Tables~A1 and~A5. The three panels of Figure~A1 for $N = 500$ were obtained again by running the script. The coverage in Table~A6 agrees with footnote~7, and it is coverage of the sample-average effect (\\fid{P029-N009}).\n\\item Every work cited in the text has an entry in the reference list, and every entry is cited.\n\\end{itemize}\n\nI did not rerun the simulations for $n \\ge 1{,}000$, the four simulations with misspecification or the bootstrap simulation; I read their code against the supplement and compared the tables with the figures. The saved outputs of the simulations are not among the replication files, so I could not compute the corrected Table~A3. I did not run the path of the package \\texttt{icsw} that uses genetic optimization. The versions of R and of \\texttt{minqa} used in 2013 are not stated; I used R~4.4.2 and \\texttt{minqa}~1.2.8. The full-bootstrap output reported 39 of 2,500 compliance-score fits at the evaluation cap, retained as in the script; I did not independently confirm that count. I confirmed one capped fit among the first four draws (\\fid{P029-N010}).\n\nOf the works cited, I consulted only those needed for particular questions: Deaton's essay in both editions and the publishers' records for the entries of \\fid{P029-E020}, Arceneaux and Nickerson (2009), Green, Gerber and Nickerson (2003) and Albertson and Lawrence (2009). The other cited works were not read, and the classification of the 34 articles behind the 6\\% in the introduction was not available. Three questions about the coding of variables in the debate study are undetermined, because the cited article and the data files do not settle which description of the variables is right. I did not search for published errata. That this audit lists no further corrections is not a finding that the rest of the paper is correct.", "work": { "arxiv_id": null, "audit_complete": true, "authors": [ "P. M. Aronow", "Allison Carnegie" ], "category": "Articles - Theory and Methodology", "citation": "P. M. Aronow and Allison Carnegie (2013). Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable. Political Analysis 21(4):492–506.", "doi": "10.1093/pan/mpt013", "edition": { "pages_in_audited_file": 15, "sha256": "d1956303080df64032982932790f3499309cff77117db63f967e2c71efd418c6", "source_url": "https://doi.org/10.1093/pan/mpt013", "venue": "Political Analysis", "year": "2013" }, "id": "P029", "later_materials": null, "title": "Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P029/", "venue": "Political Analysis", "year": "2013" } }