{ "claims": [ { "claim": "Replacing BFP's Kolmogorov-Smirnov statistic by a sum-of-squared-residuals statistic that also uses the fixed network (regression of the adjusted outcome on own treatment, number of treated neighbours and degree: 'SSR + Design') improves the power of sharp-hypothesis tests of BFP's two-parameter spillover model; in the Fig. 3 example SSR + Design has much better power than the other five statistics.", "claim_id": "P016-H1", "related_corrections": [ "P016-E002", "P016-E006", "P016-E008", "P016-E009" ], "tied_to": "title; abstract (p. 395); equation (2); Fig. 3; Section 2.2 (p. 400); Section 4 (p. 402)" }, { "claim": "Sharp hypotheses about treatment propagation can be assessed by mapping observed outcomes to the uniformity trial implied by the hypothesis and comparing a test statistic with its distribution under the known treatment assignment design.", "claim_id": "P016-H2", "related_corrections": [ "P016-E001", "P016-E008", "P016-N002" ], "tied_to": "abstract (p. 395); Section 1, four-step procedure on p. 396; equation (3)" }, { "claim": "In a simulation without interference, with 256 units and no true effect, SSR has more power than KS for a Normal outcome and less for a skewed outcome under both the additive and multiplicative hypotheses. Figure 2 reports powers of $0.70$ versus $0.54$ and $0.86$ versus $0.74$ for the Normal outcome, and $0.70$ versus $1$ and $0.85$ versus $1$ for the jittered geometric outcome.", "claim_id": "P016-H3", "related_corrections": [ "P016-E002", "P016-E004", "P016-E006", "P016-E011", "P016-N003" ], "tied_to": "Section 2, pp. 397-399; equation (1); Fig. 1; Fig. 2 and its caption (p. 398)" }, { "claim": "All six tests in the network simulation reject the true hypothesis $(\\tau,\\beta)=(0.5,2)$ no more than $5\\%$ of the time at $\\alpha=0.05$, and are conservative: the reported null rejection rates are about $4\\%$ for SSR tests and $2\\%$ for KS tests.", "claim_id": "P016-H4", "related_corrections": [ "P016-E006", "P016-E008", "P016-E009", "P016-E011" ], "tied_to": "Section 2.2, p. 400; Fig. 3" }, { "claim": "Adding a quantity from the true model, the number of treated neighbours, to the KS statistic does not raise its power to that of SSR + Design; in the network simulation the KS tests appear less powerful in general than the SSR tests.", "claim_id": "P016-H5", "related_corrections": [], "tied_to": "Section 2.2, p. 400, last sentence of the second paragraph; Fig. 3 (bottom row against top row)" }, { "claim": "In a re-analysis of Coppock's (2014) spillover model for the Butler–Nickerson legislative experiment, SSR bounds the plausible direct and indirect effects roughly by the rectangle with corners $(-0.6,0.2)$ and $(0.3,-0.4)$. KS rejects more hypotheses near $(0,0)$ but leaves an unbounded band of unrejected hypotheses.", "claim_id": "P016-H6", "related_corrections": [ "P016-E006", "P016-E010", "P016-E011" ], "tied_to": "Section 3, pp. 400-402; Fig. 4 and its caption (p. 401); the unnumbered application model on p. 400" }, { "claim": "Which hypotheses are rejected or look comparatively plausible depends on the test statistic as well as the causal model, so researchers should check operating characteristics and account for the choice of statistic when interpreting a $p$-value surface; SSR from ordinary least squares is not always appropriate.", "claim_id": "P016-H7", "related_corrections": [], "tied_to": "Section 1, pp. 396-397; Section 4, p. 402; Figs 2-4" }, { "claim": "Conjecture: the SSR from an inverse-probability-weighted least squares regression is a more generally sensible test statistic for models with interference, and other dependence measures (a weighted Brownian distance covariance or another E statistic) may help with nonlinear deviations.", "claim_id": "P016-H8", "related_corrections": [], "tied_to": "Section 4, p. 402; footnote 8" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Eight lines of text from page 396 that give steps (2), (3) and (4) of the testing procedure. A red box marks the expression T(Y_i, Z_i) in the words 'comparing the observed test statistic, T(Y_i, Z_i) against the distribution of that test statistic'.", "caption": "Steps (2) to (4) of the testing procedure, p. 396. The mark shows the observed test statistic of step (4), printed with the raw outcome $Y_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E001.png" } ], "formal_release": null, "formal_statements": [ { "id": "P016-T7", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T7" }, { "id": "P016-T8", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T8" } ], "headline": "In step (4) the observed test statistic is $\\mathcal{T}(\\tilde y_{i,Z_i=0},Z_i)$, computed from the outcome adjusted under the hypothesis, and not $\\mathcal{T}(Y_i,Z_i)$.", "id": "P016-E001", "location": "Section 1, second paragraph, step (4), p. 396 (PDF p. 2)", "pdf_page": 2, "printed_page": "396", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P016-T7", "P016-T8" ], "text": { "as_printed": "(2) using this result to adjust the observed outcome to represent what would be implied by the hypothesis for a given $\\tau_0$ such that $\\tilde y_{i,Z_i=0}=Y_i-Z_i\\tau_0$; and (3) under the hypothesis, $\\tilde y_{i,Z_i=0}$ should have no systematic relationship with treatment assignment, so we can summarize this relationship with a test statistic, $\\mathcal{T}(\\tilde y_{i,Z_i=0},Z_i)$. A distribution of values for this test statistic arises from repetitions of the treatment assignment process (new draws of $\\mathbf{z}$ from all of the ways that such treatment assignment vectors could have been produced); and finally (4) a $p$ value arises by comparing the observed test statistic, $\\hl{\\mathcal{T}(Y_i,Z_i)}$ against the distribution of that test statistic that characterizes the hypothesis.", "corrected": "(4) a $p$ value arises by comparing the observed test statistic, $\\hl{\\mathcal{T}(\\tilde y_{i,Z_i=0},Z_i)}$, evaluated at the realized assignment, against the distribution of that test statistic that characterizes the hypothesis.\n\nIn full: let $Y$ be the vector of observed outcomes, $Z$ the observed assignment and $u_i=Y_i-\\tau_0Z_i$ the outcome adjusted under the hypothesized additive effect $\\tau_0$. Compare the observed $\\mathcal{T}(u,Z)$ with the distribution of $\\mathcal{T}(u,Z')$, where $Z'$ follows the experimental assignment law and $u$ is held fixed.", "evidence": "The evidence is a derivation, an exact counterexample to the rule as printed, and a check of the replication code.\n\nLet $Z$ be the realized assignment and $Y$ the vector of observed outcomes. Suppose that the hypothesis $y_{i,Z_i=1}=y_{i,Z_i=0}+\\tau_0$ holds, with no interference. Inserting it into the identity $Y_i=Z_iy_{i,Z_i=1}+(1-Z_i)y_{i,Z_i=0}$, I obtain $Y_i=y_{i,Z_i=0}+\\tau_0Z_i$. This implies that $u_i=Y_i-\\tau_0Z_i$ equals $y_{i,Z_i=0}$, which does not depend on the assignment. Under the hypothesis, the statistic at the realized assignment is $\\mathcal{T}(u,Z)$, and its randomization distribution is that of $\\mathcal{T}(u,Z')$, where $Z'$ follows the assignment law of the experiment and $u$ is held fixed. For the difference in means, I have $\\mathcal{T}(Y,Z)=\\mathcal{T}(u,Z)+\\tau_0$.\n\nFor the counterexample, I take four units, two of them treated, with the six assignments equally likely, $u=(0,1,2,3)$, true effect $\\tau_0=10$, and the difference in means as the statistic. The six values of $\\mathcal{T}(u,Z')$ are $-2,-1,0,0,1,2$. At $Z=(0,0,1,1)$ the observed outcomes are $Y=(0,1,12,13)$, the raw statistic is $12$, and the adjusted statistic is $2$. The two-sided $p$ value is the proportion of the six values of $\\mathcal{T}(u,Z')$ that are at least as large in absolute value as the observed statistic. It is $0$ for the raw statistic and $1/3$ for the adjusted statistic. Under each of the six assignments the raw statistic lies between $8$ and $12$, and its $p$ value is $0$. The rule as printed rejects the true hypothesis $\\tau_0=10$ with probability one.\n\nThe randomization routine of the replication code adjusts the outcome once, evaluates the observed statistic on the adjusted outcome, and compares it with the statistic evaluated on the same adjusted outcome under the sampled assignments.", "problem": "Steps (2) and (3) define the adjusted outcome $\\tilde y_{i,Z_i=0}=Y_i-Z_i\\tau_0$ and build the randomization distribution from it. Step (4) then writes the observed statistic with the raw outcome $Y_i$. In general $\\mathcal{T}(Y,Z)$ differs from $\\mathcal{T}(\\tilde y,Z)$; the two coincide when $\\tau_0=0$ or when the statistic is unchanged by the adjustment. As printed, the observed statistic is then compared with the distribution of a different quantity. Steps (2) and (3) show which statistic is meant, and the replication code computes it.", "what_survives": "The procedure of steps (1) to (3) and the inspected randomization routine are unchanged. I found no reported result that depends on the raw statistic. Only the symbol in step (4) changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E001", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/" }, "work_id": "P016" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Footnotes 2, 3 and 4 of page 396. In footnote 3, a red box marks the bracketed expression F_1(y_i,0) - F_0(y_i,0) that follows 'max over i = 1,...,n' in the definition of the KS test statistic.", "caption": "Footnotes 2 to 4, p. 396. The mark shows the difference of the two ECDFs in the KS statistic of footnote 3, printed in square brackets without an absolute value.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E002.png" }, { "alt": "Four lines of text followed by displayed equation (1), T(y_0, z)_SSR defined as the sum over i of (y-tilde_i,Zi=0 minus y-tilde-bar_i,Zi=0) squared. Red boxes mark the words 'sums the difference' in the text and the overbarred term y-tilde-bar_i,Zi=0 in the equation.", "caption": "The sentence that introduces equation (1), and equation (1), p. 397. The marks show the words “sums the difference” and the mean that is subtracted in the summand.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E002-b.png" }, { "alt": "Four lines of text followed by displayed equation (2), T(y_0, z)_SSR defined as the sum over i of (y-tilde_i,Zi=0 minus X beta-hat) squared. Red boxes mark the words 'Z_i, z-transpose S, and 1-transpose S into a matrix X,' in the text and the term X beta-hat in the equation.", "caption": "The sentence that introduces equation (2), and equation (2), p. 399. The marks show the list of columns of $\\mathbf{X}$, which has no constant, and the fitted vector $\\mathbf{X}\\hat{\\boldsymbol{\\beta}}$, printed without a row index.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E002-c.png" } ], "formal_release": null, "formal_statements": [ { "id": "P016-T5", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T5" }, { "id": "P016-T9", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T9" }, { "id": "P016-T10", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T10" } ], "headline": "The three displayed definitions of the test statistics need the absolute value in the KS statistic, the mean of the unit’s own assignment group in equation (1), and the row index and the intercept in equation (2).", "id": "P016-E002", "location": "Footnote 3, p. 396 (PDF p. 2); equation (1) and the sentence before it, p. 397 (PDF p. 3); equation (2) and the sentence before it, p. 399 (PDF p. 5)", "pdf_page": 2, "printed_page": "396", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P016-T10", "P016-T5", "P016-T9" ], "text": { "as_printed": "Footnote 3, p.~396: If the empirical cumulative distribution function (ECDF) of the treated units is $F_1$ and the ECDF of the control units is $F_0$, then the KS test statistic is $\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{KS}}=\\max_{i=1,\\ldots,n}\\hl{[F_1(y_{i,0})-F_0(y_{i,0})]}$, where $F(x)=(1/n)\\sum_{i=1}^{n}I(x_i\\le x)$ records the proportion of the distribution of $x$ at or below $x_i$ (Hollander 1999, section 5.4).\n\np.~397: The simplest version of the sum-of-squared-residual (SSR) test statistic merely \\hl{sums the difference} between the mean of the outcome implied by the hypothesis and individual outcomes, thereby including the correlation between treated and control outcomes as a part of the distribution of the statistic:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(\\tilde y_{i,Z_i=0}-\\hl{\\overline{\\tilde y}_{i,Z_i=0}})^2 \\qquad (1)\n\\]\n\np.~399: If we collected \\hl{$Z_i$, $\\mathbf{z}^{T}\\mathbf{S}$, and $\\mathbf{1}^{T}\\mathbf{S}$} into a matrix $\\mathbf{X}$, and fit the $\\tilde y_{i,Z_i=0}$ as a linear function of $\\mathbf{X}$ with fitted coefficients $\\hat{\\boldsymbol{\\beta}}$ then we could define the test statistic as\n\\[\n\\mathcal{T}(y_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(\\tilde y_{i,Z_i=0}-\\hl{\\mathbf{X}\\hat{\\boldsymbol{\\beta}}})^2. \\qquad (2)\n\\]", "corrected": "Write $u_i=\\tilde y_{i,Z_i=0}$. For $g\\in\\{0,1\\}$, let $n_g=\\#\\{i:z_i=g\\}>0$, let $\\bar u_g=n_g^{-1}\\sum_{i:z_i=g}u_i$, and let $F_g(t)=n_g^{-1}\\sum_{i:z_i=g}I(u_i\\le t)$.\n\nFootnote 3:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{KS}}=\\max_{i=1,\\ldots,n}\\hl{|}F_1(u_i)-F_0(u_i)\\hl{|}=\\sup_t|F_1(t)-F_0(t)|.\n\\]\n\nEquation (1): The simplest version of the sum-of-squared-residual (SSR) test statistic sums the \\hl{squared differences} between individual outcomes and \\hl{the mean of the outcome in the unit's own assignment group}:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(u_i-\\hl{\\bar u_{z_i}})^2. \\qquad (1)\n\\]\n\nEquation (2): Let $\\mathbf{X}(\\mathbf{z})$ be the matrix with rows $x_i^{T}=(\\hl{1},z_i,(\\mathbf{z}^{T}\\mathbf{S})_i,(\\mathbf{1}^{T}\\mathbf{S})_i)$, and let $\\hat{\\boldsymbol{\\beta}}$ be a least squares coefficient vector. Then\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\min_b\\|u-\\mathbf{X}(\\mathbf{z})b\\|_2^2=\\sum_i\\{u_i-\\hl{(\\mathbf{X}(\\mathbf{z})\\hat{\\boldsymbol{\\beta}})_i}\\}^2. \\qquad (2)\n\\]", "evidence": "The evidence is exact arithmetic on small examples, a derivation, and a check of the functions in the replication code on the same examples.\n\nFor the KS statistic, let the adjusted outcomes be $u=(0,1,2,3)$, with $z=(0,0,1,1)$. At the points $0,1,2,3$ the differences $F_1-F_0$ are $-1/2,-1,-1/2,0$. The display as printed gives $0$, and the two-sided statistic is $1$.\n\nFor equation (1), an intercept and $z$ fit one constant $c_g$ to each assignment group $g$. For any constant $c$, I have $\\sum_{i:z_i=g}(u_i-c)^2=\\sum_{i:z_i=g}(u_i-\\bar u_g)^2+n_g(\\bar u_g-c)^2$, where $\\bar u_g$ is the mean of the $n_g$ units of group $g$. The fitted constants are therefore the group means, and the residual sum of squares is $\\sum_i(u_i-\\bar u_{z_i})^2$. With $u=(0,1,2,3)$ and two of the four units treated, the total sum of squares is $5$ under all six assignments, and the residual sum of squares takes the values $1,4,5,5,4,1$. At the assignment that treats units 1 and 2, the lower-tail $p$ value is $1/3$ for the residual sum of squares and $1$ for the total sum of squares.\n\nFor equation (2), take six units with the single edge $\\{1,2\\}$, $z=(1,0,1,0,1,0)$ and $u=(1,1,1,1,1,1)$. The residual sum of squares is $0$ with the intercept and $2$ without it.\n\nOn these inputs the functions of the replication code return $1$ for the KS statistic, $1,4,5,5,4,1$ for the simple SSR statistic and $0$ for the network SSR statistic. These are the values of the corrected definitions.", "problem": "Read literally, the KS display differs from the statistic used, equation (1) leaves its mean undefined, and equation (2) omits the component index of its fitted vector.\n\\begin{enumerate}\n\\item Footnote 3 has no absolute value, and the display is the one-sided statistic $\\max_i\\{F_1-F_0\\}$. The statistic described, which is sensitive to any difference between the two distributions, and the statistic coded are the two-sided $\\max_i|F_1-F_0|$.\n\\item In equation (1) the overbar carries the unit index $i$ and is not defined. The statistic meant is the residual sum of squares from the regression of the adjusted outcome on an intercept and $z$, in which the mean subtracted from unit $i$ is the mean of its own assignment group. If the overbar is read as the mean over all units, the display is the total sum of squares, which does not depend on $\\mathbf{z}$. The sentence before the display says ``sums the difference'' where the squared difference is meant.\n\\item In equation (2) the summand subtracts the whole vector $\\mathbf{X}\\hat{\\boldsymbol{\\beta}}$ from the scalar $\\tilde y_{i,Z_i=0}$, since the row index is missing. The list of columns of $\\mathbf{X}$ has no constant. The regression in the replication code has an intercept.\n\\end{enumerate}\nThe surrounding text on least squares regression identifies the statistics that are meant.", "what_survives": "The inspected functions of the replication code compute the corrected definitions. These notation repairs do not change their outputs, and I found no error in the comparisons of Figs.~2 to 4 caused by these defects. Note that the signed maximum of footnote 3 is a valid one-sided statistic; it is not the statistic used. The power recalculations are described below under what I checked." }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E002", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/" }, "work_id": "P016" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Seven lines of text from page 397, beginning 'example,', followed by displayed equation (1). Red boxes mark the words 'thereby ignoring information about the joint distribution' and the words 'thereby including the correlation between treated and control outcomes as a part of the distribution of the statistic'.", "caption": "The contrast before equation (1), p. 397; the crop begins after “For”. The marks show the statement that the KS test ignores information about the joint distribution, and the statement that the simplest SSR statistic includes the correlation between treated and control outcomes.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E004.png" } ], "formal_release": null, "formal_statements": [ { "id": "P016-T1", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T1" }, { "id": "P016-T2", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T2" }, { "id": "P016-T5", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T5" } ], "headline": "The simple SSR statistic of equation (1) uses no correlation between treated and control outcomes that the KS statistic ignores: both are functions of the two group distributions and their sizes, and with fixed adjusted outcomes and group sizes the SSR test is the test based on the absolute difference in group means.", "id": "P016-E004", "location": "Section 2, second paragraph, the two sentences before equation (1), p. 397 (PDF p. 3)", "pdf_page": 3, "printed_page": "397", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P016-T1", "P016-T2", "P016-T5" ], "text": { "as_printed": "For example, the KS test uses the maximum difference in the empirical cumulative distributions of each treatment group calculated without regard for the relationship between the treated and control distributions, \\hl{thereby ignoring information about the joint distribution} that could increase the precision of the test. The simplest version of the sum-of-squared-residual (SSR) test statistic merely sums the difference between the mean of the outcome implied by the hypothesis and individual outcomes, \\hl{thereby including the correlation between treated and control outcomes as a part of the distribution of the statistic}:\n\\[\n\\mathcal{T}(\\mathbf{y}_0,\\mathbf{z})_{\\mathrm{SSR}}\\equiv\\sum_i(\\tilde y_{i,Z_i=0}-\\bar{\\tilde y}_{i,Z_i=0})^2 \\qquad (1)\n\\]", "corrected": "For example, the KS test uses the maximum difference in the empirical cumulative distributions of each treatment group \\del{calculated without regard for the relationship between the treated and control distributions, thereby ignoring information about the joint distribution that could increase the precision of the test}. The simplest version of the sum-of-squared-residual (SSR) test statistic \\hl{is the residual sum of squares from a regression of the adjusted outcome on an intercept and treatment. It is determined by the first two empirical moments and the size of each treatment group. With fixed adjusted outcomes and group sizes, smaller SSR values correspond exactly to larger absolute differences in group means. Its sensitivity differs from that of the KS statistic, but it does not acquire pairing information absent from the two group distributions. Adding fixed-network regressors can use information about the alignment of outcomes with the network. The randomization distribution of either statistic also depends on the assignment design.}", "evidence": "I use a derivation and exact finite examples.\n\nLet $a_i=\\tilde y_{i,Z_i=0}$ be the adjusted outcomes for one hypothesis, held fixed. Let $z\\in\\{0,1\\}^n$ have $n_1>0$ ones and $n_0>0$ zeros, with $n=n_0+n_1$. I write $m_g$ for the mean of $a_i$ over the units with $z_i=g$, $m$ for the mean over all units, and $T(a)=\\sum_i(a_i-m)^2$. For any group $g$ and any constant $c$, I have\n\\[\n\\sum_{i:z_i=g}(a_i-c)^2=\\sum_{i:z_i=g}(a_i-m_g)^2+n_g(m_g-c)^2,\n\\]\nsince the cross term vanishes. The least squares fit on an intercept and $z$ therefore has fitted values $m_0$ and $m_1$, and its residual sum of squares is $R(a,z)=\\sum_g\\sum_{i:z_i=g}(a_i-m_g)^2$. Taking $c=m$ and summing over the two groups, I obtain $T(a)=R(a,z)+n_0(m_0-m)^2+n_1(m_1-m)^2$. Since $m_0-m=-n_1(m_1-m_0)/n$ and $m_1-m=n_0(m_1-m_0)/n$, I conclude that\n\\[\nR(a,z)=T(a)-\\frac{n_0n_1}{n}(m_1-m_0)^2.\n\\]\nThe term $T(a)$ and the factor $n_0n_1/n$ do not depend on the assignment. For any two assignments $z$ and $w$ with these group sizes, I have $R(a,w)\\le R(a,z)$ if and only if $|m_1(w)-m_0(w)|\\ge|m_1(z)-m_0(z)|$, ties included. This implies that, for every assignment law $P_Z$ on such assignments,\n\\[\nP_Z\\{R(a,Z)\\le R(a,z_{\\mathrm{obs}})\\}=P_Z\\{|m_1(Z)-m_0(Z)|\\ge|m_1(z_{\\mathrm{obs}})-m_0(z_{\\mathrm{obs}})|\\}.\n\\]\nFurther, I have $R=n_0(q_0-m_0^2)+n_1(q_1-m_1^2)$, where $q_g$ is the mean of $a_i^2$ in group $g$. The two group distributions and their sizes therefore determine $R$.\n\nThe table shows the example $a=(0,1,2,3)$ with two of the four units treated.\n\\begin{audittable}{lrrrrrr}{The six assignments of two treated units among four, with $a=(0,1,2,3)$ and each assignment of probability $1/6$. Here $\\delta=m_1-m_0$ and $n_0n_1/n=1$, so the SSR equals $5-\\delta^2$. The lower-tail $p$ value of the SSR equals the upper-tail $p$ value of $|\\delta|$ at every assignment. All values are exact.}\nTreated units & $\\delta$ & $T(a)$ & SSR & KS & $p_{\\mathrm{SSR}}=p_{|\\delta|}$ & $p_{\\mathrm{KS}}$ \\\\\n1, 2 & $-2$ & 5 & 1 & 1 & $1/3$ & $1/3$ \\\\\n1, 3 & $-1$ & 5 & 4 & $1/2$ & $2/3$ & 1 \\\\\n1, 4 & 0 & 5 & 5 & $1/2$ & 1 & 1 \\\\\n2, 3 & 0 & 5 & 5 & $1/2$ & 1 & 1 \\\\\n2, 4 & 1 & 5 & 4 & $1/2$ & $2/3$ & 1 \\\\\n3, 4 & 2 & 5 & 1 & 1 & $1/3$ & $1/3$ \\\\\n\\end{audittable}\nThree further exact computations bear on the two printed statements. First, for treated values $(0,2)$ and control values $(1,3)$, the SSR is $4$ and the KS statistic is $1/2$ whichever control value is paired with which treated value, although the empirical covariance of the pairs, with divisor $2$, is $1$ under one pairing and $-1$ under the other. Second, both randomization distributions depend on the design: at the assignment that treats units 1 and 2, both $p$ values are $1/3$ under complete randomization and $1/2$ when one unit is chosen uniformly and independently for treatment in each of the blocks $\\{1,3\\}$ and $\\{2,4\\}$. Third, the SSR is not a function of the KS value: when units 1 and 3 are treated, the KS statistic is $1/2$ and the SSR is $4$, and when units 1 and 4 are treated, the KS statistic is $1/2$ and the SSR is $5$.", "problem": "The two sentences give a reason to prefer the simple SSR statistic to the KS statistic: the KS statistic is said to ignore information about the joint distribution of treated and control outcomes, and the SSR statistic is said to include their correlation. This reason does not hold. The statistic meant by equation (1), and computed by the replication code, is the residual sum of squares from the regression of the adjusted outcome on an intercept and the treatment indicator (see \\fid{P016-E002}). It is determined by the group sizes and the first two moments of the adjusted outcome within each treatment group. Like the KS statistic, it is a function of the two group distributions and their sizes, and no pairing of treated with control units and no covariance between them enters either statistic. With the adjusted outcomes and the group sizes fixed, a smaller SSR corresponds exactly to a larger absolute difference in group means. The SSR test is then the test based on the absolute difference in means. A narrower statement is true: the randomization distribution of the SSR statistic reflects the dependence between the two groups that the assignment design induces. The same holds for the randomization distribution of the KS statistic, and the statement does not distinguish the two. The two statistics differ in power because they summarize the two group distributions differently.", "what_survives": "The simple SSR statistic and the validity of its exact randomization test are unaffected by this correction, and no simulation, figure or part of the application depends on this explanation. The SSR and KS statistics remain different summaries with different power (Fig.~2). The statistics with network regressors do use information beyond the two group distributions: in an exact four-unit example, two outcome vectors with the same group distributions have SSR $0$ and SSR $4$ once degree is added to the regression. I correct only the two explanatory sentences here. I have not recomputed the reported power values." }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E004", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/" }, "work_id": "P016" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Eight lines of text from the top of page 397. Red boxes mark the words 'with this property (\"effect increasing\" test statistics)' and the words 'any effect increasing test statistic ought to be a valid test'.", "caption": "The end of the quotation and the two sentences that follow it, p. 397. The first mark shows the mistaken identification of the quoted property with “effect increasing”. The second marks a sufficient validity statement, whose conclusion remains correct.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E005.png" } ], "formal_release": null, "formal_statements": [ { "id": "P016-T8", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T8" }, { "id": "P016-T11", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T11" }, { "id": "P016-T12", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T12" }, { "id": "P016-T14", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T14" } ], "headline": "A statistic that is small when the two distributions are similar and large when they diverge need not be “effect increasing”, and a randomization test of a sharp null hypothesis is valid for any test statistic, with or without that property.", "id": "P016-E005", "location": "Section 1, last paragraph, the two sentences after the quotation, p. 397 (PDF p. 3); the quotation begins on p. 396", "pdf_page": 3, "printed_page": "397", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P016-T11", "P016-T12", "P016-T14", "P016-T8" ], "text": { "as_printed": "``Select a test statistic [$\\mathcal{T}$] that will be small when the treated and control distributions in the adjusted data \\ldots\\ are similar, and large when the distributions diverge.'' Rosenbaum (2002, Proposition 4 and 5, section 2.9) presents results proving that test statistics \\hl{with this property (``effect increasing'' test statistics)} produce an unbiased test of the hypothesis of no effects or positive effects when the positive effects involve one parameter. Such results mean that a test of the sharp null of no effects based on a known randomization using \\emph{any} effect increasing test statistic ought to be a valid test (in that the test should produce $p$ values less than $\\alpha$ no more than $100\\alpha\\%$ of the time when the null is true) even though different test statistics may imply different power against false hypotheses.", "corrected": "\\hl{A statistic is effect increasing if increasing treated responses and decreasing control responses, at fixed assignment, cannot decrease it. The quoted advice about distributional divergence does not imply this directional condition. Under a sharp null hypothesis, calibrating any specified test statistic under the known assignment law gives a valid test; unbiasedness against alternatives is a separate question.}", "evidence": "The evidence is an exact counterexample and a derivation.\n\nFix the assignment $z=(1,1,0,0)$ and change $y=(0,1,2,3)$ to $y'=(2,3,2,3)$. Both treated responses rise by $2$ and the control responses are unchanged. The two-sided KS statistic is $1$ for $y$, where every treated value lies below every control value, and $0$ for $y'$, where both groups have the values $2$ and $3$. Since the statistic decreases, it is not effect increasing. The KS function of the replication code returns the same two values, and the signed maximum printed in footnote 3 also falls from $1$ to $0$.\n\nFor validity, let $\\Omega$ be the finite set of assignments, with known probabilities $q(z)>0$. Under the sharp null hypothesis of no effects, the outcome vector $y^0$ is the same under every assignment. Any statistic then has fixed scores $s(z)=\\mathcal{T}(y^0,z)$. Define the $p$ value $p(z)=\\sum_{w\\in\\Omega:s(w)\\ge s(z)}q(w)$. Let $v_1>\\cdots>v_k$ be the distinct scores, let $m_j$ be the probability that $s(Z)=v_j$, and let $M_j=m_1+\\cdots+m_j$. An assignment with score $v_j$ has $p(z)=M_j$, and $M_j$ is strictly increasing in $j$. Fix $\\alpha\\in[0,1]$. If $M_1>\\alpha$, the event $\\{p(Z)\\le\\alpha\\}$ is empty. Otherwise, let $r$ be the largest index such that $M_r\\le\\alpha$. Then the event consists of the assignments with scores $v_1,\\ldots,v_r$, and its probability is $M_r\\le\\alpha$. I conclude that $\\Pr\\{p(Z)\\le\\alpha\\}\\le\\alpha$ for every statistic. The argument does not use the effect increasing property.\n\nThe cited propositions and the quotation were not compared with the two works cited.", "problem": "The passage identifies the property in the quotation, being small for similar distributions and large for divergent ones, with the effect increasing property. The two are different. A statistic is effect increasing if, at a fixed assignment, raising treated responses and lowering control responses cannot decrease it. This condition is directional. An unsigned measure of the difference between two distributions need not satisfy it, and the two-sided KS statistic does not. The next sentence gives a correct sufficient condition for validity, but validity holds for every test statistic calibrated under the known assignment law, and whether a test is unbiased against alternatives is a separate question.", "what_survives": "The conclusion that the randomization test of the sharp null hypothesis is valid stands, for a more general reason. Nothing later in the note uses the effect increasing property: the statistics, the simulations and the application are unaffected, and the note does not claim that its SSR statistics are effect increasing. Only these explanatory sentences change." }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E005", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/" }, "work_id": "P016" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Six lines from page 400: the sentence 'With distance matrix Gamma, Coppock evaluated the model:', the displayed model y_0 = y_z - beta_1 z - beta_2 g(Gamma z), and the four lines that follow. Red boxes mark the words 'distance matrix', the term g(Gamma z) in the display, and the words 'normalizes the sums of distances to treated neighbors to have unit variance.'.", "caption": "The application model, Section 3, p. 400. The marks show the words “distance matrix”, the term $g(\\Gamma\\mathbf{z})$ of the model, and the description of $g$ as a normalization of sums of distances to unit variance.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E006.png" }, { "alt": "Five lines of text from page 400. A red box marks the words 'less than .05' in the sentence 'The proportion of p values from that test less than .05 is plotted in Fig. 3'.", "caption": "Section 2.2, p. 400. The mark shows the rejection rule of the simulations as printed, “less than .05”.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E006-b.png" }, { "alt": "Scatter plot. Horizontal axis: raw exposure (Sz)_i, the sum of similarities to treated legislators, from 0 to 25. Vertical axis: exposure in standard deviations across the 70 legislators, from -5 to 2. A dashed grey line rising from about -4.8 at raw exposure 0 to about 1 at raw exposure 24 shows the raw sums centred and scaled to unit sample variance, labelled as described on page 400. Orange circles (35 treated legislators) and blue squares (35 control legislators) show the exposure in the replication files, labelled as coded. The points form a cloud between raw exposures 14 and 24 and between -1.9 and 1.6 on the vertical axis, with no visible linear relation to the dashed line; at raw exposure 0 and vertical value near 0, a blue square covers an orange circle at the same position. The title reads 'The exposure used for Fig. 4 is not the raw sum normalized to unit variance: the correlation between the two is 0.066'.", "caption": "The exposure used in the application against the raw exposure, for the 70 legislators of the application table in the replication files. If p. 400’s normalization is read as centring and scaling the raw similarity sum $(Sz)_i$ to unit sample variance, the points would lie on the dashed line. The correlation between the two standardized exposures is $0.0665$. Two legislators have raw exposure $0$ and identical standardized exposure; their symbols overlap.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E006-exposure.png" } ], "formal_release": null, "formal_statements": [ { "id": "P016-T3", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T3" }, { "id": "P016-T4", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T4" }, { "id": "P016-T6", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T6" } ], "headline": "The model behind Fig. 4 uses a similarity exposure from which the unit’s mean exposure among saved assignments with the same own treatment status is subtracted before standardization, not normalized sums of distances; the simulations use a two-sided $p$ value and reject when it is at most $.05$, and the application uses one-sided $p$ values.", "id": "P016-E006", "location": "Section 3, the model that begins “With distance matrix $\\Gamma$”, p. 400 (PDF p. 6); Section 2.2, “less than .05”, p. 400; also p. 398 and the caption of Fig. 3, p. 401", "pdf_page": 6, "printed_page": "400", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P016-T3", "P016-T4", "P016-T6" ], "text": { "as_printed": "With \\hl{distance matrix} $\\Gamma$, Coppock evaluated the model:\n\\[\n\\mathbf{y}_0=\\mathbf{y}_{\\mathbf{z}}-\\beta_1\\mathbf{z}-\\beta_2\\hl{g(\\Gamma\\mathbf{z})},\n\\]\nwhere $g$ is a function that \\hl{normalizes the sums of distances to treated neighbors to have unit variance}. This model has a direct effect ($\\beta_1$) and an indirect effect ($\\beta_2$) that is linear in the distances to treated neighbors. Coppock evaluated this spillover model using the SSR + Design statistic as presented in this article.\n\nSection 2.2, same page: The proportion of $p$ values from that test \\hl{less than .05} is plotted in Fig. 3: darker values show fewer rejections, lighter values record more rejections.", "corrected": "With \\hl{similarity matrix $S$}, where $S_{ij}=1-|s_i-s_j|/2$ for the W-NOMINATE scores of legislators $i\\ne j$ and the diagonal and undefined entries are zero, the model evaluated is\n\\[\n\\mathbf{y}_0=\\mathbf{y}_{\\mathbf{z}}-\\beta_1\\mathbf{z}-\\beta_2\\hl{e(\\mathbf{z})},\\qquad \\hl{e_i(z)=\\frac{h_i(z)-\\bar h(z)}{s_h(z)}},\\qquad \\hl{h_i(z)=(Sz)_i-z_i\\mu_{i1}-(1-z_i)\\mu_{i0}}.\n\\]\nHere $\\mu_{ia}$ is the mean of $(Sz)_i$ over those of the 10,000 saved assignments of the matched design in which $z_i=a$, and $\\bar h(z)$ and $s_h(z)>0$ are the mean and the sample standard deviation of $h_i(z)$ across the 70 legislators. The indirect effect $\\beta_2$ multiplies this \\hl{centred, standardized similarity exposure}, and Fig.~4 and the rectangle on p.~402 refer to this model. For each hypothesis the adjusted outcome $u=Y-\\beta_1Z-\\beta_2e(Z)$ is held fixed. For each reference assignment $z$ the exposure $e(z)$ is recomputed, and the SSR statistic is the residual sum of squares from the regression of $u$ on \\hl{an intercept, $z$ and $e(z)$, with no separate degree column}, \\hl{using the 67 legislators with observed outcomes}. The application uses 500 reference assignments drawn from the saved assignments. Its $p$ value is the proportion of reference statistics that are at most the observed statistic for SSR, and at least the observed statistic for KS. For each axis, the SSR bounds are taken at \\hl{the second coordinate of each adjacent grid pair where the largest $p$ value along the other axis changes from at most $.05$ to greater than $.05$, or conversely}.\n\nI describe the procedure used for the published figures as follows. Section 2.2: The proportion of $p$ values from that test \\hl{less than or equal to} .05 is plotted in Fig. 3, \\hl{where the $p$ value is twice the smaller of the proportions of sampled statistics that are at most, and at least, the observed statistic, capped at $1$}. The same descriptive change applies to ``lower than $\\alpha=0.05$'' on p.~398 and to the caption of Fig.~3. This describes the original rule used for Figs.~2 and 3. Figures for the repaired procedure in \\fid{P016-E008} require recomputation.", "evidence": "I base this correction on the replication code, exact arithmetic on the application table of the replication files, and exact small examples.\n\nLet $r_i=(Sz)_i$ be the raw exposure and $h_i$ the exposure after the subtraction. If the subtraction could be absorbed by an intercept, the direct effect and a rescaling of the indirect effect, then there would exist constants $a$, $b$ and $c$ such that $h_i=a+bz_i+cr_i$ for all $i$, and the four columns $(1,z,r,h)$ would have rank at most three. For the first four rows of the application table, the determinant of these four columns, computed in exact rational arithmetic from the decimals of the table, is $-1.4030\\ldots\\ne0$. No such constants exist. Across the 70 legislators the correlation between the standardized raw exposure and the standardized exposure used is $0.0665$, and the largest difference between the two is $4.76$ standard deviations.\n\nThe same holds in an example in which all assignments are enumerated. Let six units have scores $(-1,-3/4,-1/2,0,1/4,1)$ and form the pairs $(1,2)$, $(3,4)$ and $(5,6)$, with one treated unit in each pair, a design with eight assignments. At $z=(1,0,1,0,1,0)$ the raw exposures $r$ and the exposures $h$ after the subtraction are\n\\[\nr=(9/8,\\ 9/4,\\ 11/8,\\ 17/8,\\ 1,\\ 7/8),\\qquad h=(5/16,\\ 5/16,\\ 1/8,\\ 1/8,\\ -3/16,\\ -3/16).\n\\]\nUnits 1 and 2 have equal exposure after the subtraction and unequal raw exposure, while units 1 and 3 differ in both. This implies that $h$ is not of the form $a+cr$, and the rank of $(1,z,r,h)$ is four. For the outcome $y=(0,1,0,1,1,0)$ the residual sum of squares is $4/3$ from the regression on $(1,z,h)$ and $655/708$ from the regression on $(1,z,r)$.\n\nFor the rejection rule, take the reference values $1,2,\\ldots,1000$. At the observed value $25$ the lower tail proportion is $1/40$ and the two-sided $p$ value of the simulation code is exactly $.05$. The code counts a rejection, and the printed rule does not. At the observed value $40$ the lower tail proportion is $.04$ and the two-sided $p$ value is $.08$; at $980$ they are $.98$ and $.042$. Neither the one-sided rule nor the two-sided rule rejects whenever the other does.\n\\begin{audittable}{lll}{How $p$ values are formed and used, in the text and in the replication code. $L$ and $U$ are the proportions of the sampled statistics that are at most, and at least, the observed statistic.}\n & Text & Replication code \\\\\nSimulations, $p$ value & not stated & $\\min\\{1,2\\min(L,U)\\}$ \\\\\nSimulations, rejection & $p<.05$ & $p\\le.05$ \\\\\nApplication, $p$ value & not stated & $L$ for SSR, $U$ for KS, 500 draws \\\\\nApplication, bounds of the SSR region & not stated & transitions in whether the largest $p$ along the other axis is $>.05$ \\\\\n\\end{audittable}\nThe numerical comparison uses the application table that is among the replication files. The data file that the figure script loads is not among them, and I did not recompute Fig.~4.", "problem": "The description of the application does not specify the model to which Fig.~4 and the rectangle on p.~402 refer. In the replication code the matrix is a similarity matrix $S$, with $S_{ij}=1-|s_i-s_j|/2$ for the W-NOMINATE scores $s_i$ and $s_j$, so that ideologically closer legislators have larger weights; the diagonal and undefined entries are zero. The exposure of legislator $i$ is not $(Sz)_i$ normalized to unit variance. The code first subtracts from $(Sz)_i$ its mean over those of 10,000 saved assignments in which legislator $i$ has the same treatment status, and only then centres and scales across legislators. This subtraction is not a recentring or a rescaling. The exposure used is not an affine function of the raw exposure, even when an intercept and a change in the direct effect are allowed, and in the application table of the replication files the two standardized exposures have correlation $0.07$. The indirect effect $\\beta_2$ of Fig.~4 therefore multiplies a different variable from the one that the text describes, and its sign and its unit can be read only with the coded definition. The regression in the SSR statistic has an intercept, own assignment and this exposure, with no separate degree column, and the phrase ``SSR + Design statistic as presented in this article'' does not convey this.\n\nThe text also does not say how the $p$ values are formed. The simulations behind Figs.~2 and 3 use a two-sided $p$ value, twice the smaller of the two tail proportions, capped at $1$, and they count a rejection when $p\\le.05$. The text and the caption of Fig.~3 say ``less than .05''. The application uses one-sided $p$ values from 500 draws, the lower tail for SSR and the upper tail for KS. The two rules define different tests, and the note does not distinguish them.", "what_survives": "This correction changes the description of the model and testing rules used by the replication code. The note's comparison of a bounded SSR region and an unbounded KS region in Fig.~4 concerns the coded model. I have not established a numerical error in Fig.~4 or in the rectangle, and the absence of a degree column is not shown to make the test invalid. Figs.~2 and 3 are results for the two-sided rule with rejection at $p\\le.05$; they cannot be relabelled as results for another rule without recomputation, and my recalculated null rejection rates are compatible with the reported rates of about 4\\% and 2\\% within Monte Carlo error; they do not establish a level guarantee. The meaning of $\\beta_2$ changes: it is the coefficient of a centred, standardized similarity exposure, not of normalized sums of distances." }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E006", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/" }, "work_id": "P016" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Four lines of text from page 400. A red box marks the words 'we sampled 1000 permutations. We executed that test 10,000 times'.", "caption": "Section 2.2, first paragraph, p. 400. The mark shows the description of the simulation: 1000 sampled permutations for each test, and 10,000 tests for each pair of parameters.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E009.png" }, { "alt": "Four lines of text from page 400. A red box marks the words 'of the 10,000 simulations' in 'the SSR-based tests rejected the truth roughly 4 percent of the 10,000 simulations but the KS tests rejected the truth roughly 2 percent of the time'.", "caption": "Section 2.2, second paragraph, p. 400. The mark shows the number of simulations to which the rejection rates of about 4% and 2% are referred.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E009-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The text describes the simulation behind Fig. 3 as 10,000 experiments of 1000 sampled permutations each, and the replication code defaults to 1000 experiments of 10,000 sampled assignments each; which setting produced the published figure is not known.", "id": "P016-E009", "location": "Section 2.2, first and second paragraphs, p. 400 (PDF p. 6)", "pdf_page": 6, "printed_page": "400", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "For each test statistic, we tested the hypothesis $\\tau=\\tau_0,\\ \\beta=\\tau_0$ by using a simulated permutation test: \\hl{we sampled 1000 permutations. We executed that test 10,000 times} for each pair of parameters.\n\nAgain, all of the tests are valid in the sense of rejecting the truth no more than 5\\% of the time when $\\alpha=.05$ although all of these tests are conservative: the SSR-based tests rejected the truth roughly 4\\% \\hl{of the 10,000 simulations} but the KS tests rejected the truth roughly 2\\% of the time.", "corrected": "The deposited driver for Fig.~3 defaults to \\hl{1,000 simulated experiments with 10,000 sampled assignments for each permutation distribution}. This allocation differs from the 10,000 experiments with 1,000 sampled permutations described on p.~400. The counts used to produce the published figure cannot be recovered from the deposited files, because saved results and run logs are not supplied; the published counts should be reconciled against the original invocation and results.", "evidence": "The evidence is a recalculation from the replication code and a check of the page. Running the count expressions, the sampler and the test routine of both released versions of the code, I obtain a matrix of 1000 assignments of the 256 units, and each test receives 10,000 samples. The standard errors are $\\sqrt{p(1-p)/N}$ at $p=1/25$, which is $0.006197$ for $N=1000$ and $0.001960$ for $N=10{,}000$. I obtained these counts from the code settings and calls, without simulating rejection rates.", "problem": "The text and the replication code disagree about how the simulation behind Fig.~3 is divided. In the code, when no setting is supplied, the number of simulated experiments is 1000 and each test samples 10,000 assignments, and the driver files of the archive supply no other setting. The environment variable that the code provides changes only the number of experiments; with 10,000 experiments each test still samples 10,000 assignments, which again differs from the text. Since saved results and run logs are not among the replication files, the setting used for the published figure cannot be recovered. The difference bears on the precision of the reported rates: at a rejection rate of $0.04$ the Monte Carlo standard error is $0.0062$ with 1000 experiments and $0.0020$ with 10,000.", "what_survives": "No rejection rate, power comparison or conclusion is shown to be wrong. The discrepancy establishes that the text and the code disagree, and it does not establish which setting was run. Fig.~2 is unaffected, since its script sets 1000 sampled assignments itself, and so are the statistics, the framework and the application." }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E009", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/" }, "work_id": "P016" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Seven lines of text from page 400. Red boxes mark the words 'less than .05' and the sentence part 'All of these test statistics are valid - they reject the true null of tau = .5, beta = 2 no more than 5 percent of the time at alpha = .05 -'.", "caption": "Section 2.2, first paragraph, p. 400. The marks show the rejection rule as printed, “less than .05”, and the statement that all six tests reject the true null no more than 5% of the time at $\\alpha=.05$. The statement is correct for the rule as printed; the simulation code applies a different rule.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E008.png" } ], "formal_release": null, "formal_statements": [ { "id": "P016-T13", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T13" }, { "id": "P016-T14", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T14" }, { "id": "P016-T21", "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T21" } ], "headline": "The printed rule, which rejects when $p<.05$, is valid at level $.05$; the simulation code rejects when $p\\le.05$ with tail proportions that do not count the observed statistic, and that rule can have size slightly above $.05$.", "id": "P016-E008", "location": "Section 2.2, first two paragraphs, p. 400 (PDF p. 6), with the validity statement on p. 397 (PDF p. 3); the rejection rule is in the simulation scripts of the replication files", "pdf_page": 6, "printed_page": "400", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P016-T13", "P016-T14", "P016-T21" ], "text": { "as_printed": "The proportion of $p$ values from that test less than .05 is plotted in Fig. 3: darker values show fewer rejections, lighter values record more rejections. All of these test statistics are valid---they reject the true null of $\\tau=.5,\\ \\beta=2$ no more than 5\\% of the time at $\\alpha=.05$---and the plots are darkest in the area where the two lines showing the true parameters intersect.\n\nSimulation scripts of the replication files: \\texttt{m <- m \\hl{<=} ALPHA}, with \\texttt{ALPHA <- 0.05}.", "corrected": "The validity statement on p.~397 stands. For a repaired simulation procedure with the existing tail estimator, \\hl{1,000 or 10,000 independent reference draws and a fixed statistic}, count a rejection when $p\\,\\hl{<}\\,.05$. The published Figs.~2 and 3 describe the original $p\\le.05$ rule (\\fid{P016-E006}); they have not been recomputed for the repaired procedure. Figures for the repaired procedure require recomputation.\n\nFor a $p$ value that is valid at every threshold, let $L$ and $U$ be the numbers of the $B$ reference statistics that are at most, and at least, the observed statistic, and set\n\\[\np_L=\\frac{\\hl{1+}L}{B\\hl{+1}},\\qquad p_U=\\frac{\\hl{1+}U}{B\\hl{+1}},\\qquad p=\\min\\{1,2\\min(p_L,p_U)\\}.\n\\]\nIf the observed statistic and the $B$ reference statistics are exchangeable under the null hypothesis, as they are for a fixed statistic and independent draws from the assignment law, then $\\Pr(p\\le\\alpha)\\le\\alpha$ for every $\\alpha\\in[0,1]$. Sampling from a fixed saved set of assignments, as in the application, requires a separate justification of exchangeability.", "evidence": "The evidence is an exact counterexample, computed by enumeration and rational arithmetic, and a derivation.\n\nFor the counterexample, I take sixteen units, eight of them treated, with all $\\binom{16}{8}=12870$ assignments equally likely, the sharp null hypothesis of no effect with outcomes $2^i$ for $i=0,\\ldots,15$, and the simple SSR statistic. The statistic takes 6435 distinct values over these assignments, each shared by an assignment and its complement. The observed assignment and the $B$ reference assignments are drawn independently from the design.\n\\begin{audittable}{lrr}{Exact null rejection probability at nominal level $.05$ in the sixteen-unit example, rounded to six decimals.}\nReference draws $B$ & Coded rule, $p\\le.05$ & Printed rule, $p<.05$ \\\\\n$1000$ & $0.051793$ & $0.049795$ \\\\\n$10{,}000$ & $0.050040$ & $0.049840$ \\\\\n\\end{audittable}\nThe probability that the coded $p$ value equals $0$ is $0.0018$ for $B=1000$.\n\nFor the derivation, let $T_0$ be the observed statistic and $T_1,\\ldots,T_B$ the reference statistics, and suppose that these $B+1$ statistics are exchangeable under the null hypothesis. Set $L=\\#\\{b:T_b\\le T_0\\}$ and $U=\\#\\{b:T_b\\ge T_0\\}$. With ties broken at random, the rank of $T_0$ among the $B+1$ statistics is uniform on $\\{1,\\ldots,B+1\\}$, and this rank is at most $L+1$. I therefore have $\\Pr(L\\le c)\\le(c+1)/(B+1)$ for every integer $c\\ge0$, and the same bound for $U$. The event $2\\min(L,U)/B<\\alpha$ requires that $L\\le c$ or $U\\le c$ with $c=\\lceil\\alpha B/2\\rceil-1$, and by the union bound its probability is at most $2(c+1)/(B+1)$. At $\\alpha=.05$ and $B=1000$, I have $c=24$, and the bound is $50/1001<.05$. The coded rule allows $c=\\lfloor\\alpha B/2\\rfloor=25$, for which the same bound is $52/1001>.05$; the counterexample shows that the size can exceed $.05$. Next, set $p_L=(1+L)/(B+1)$ and $p_U=(1+U)/(B+1)$. For every $a\\in[0,1]$, I have $\\Pr(p_L\\le a)=\\Pr(L+1\\le\\lfloor a(B+1)\\rfloor)\\le\\lfloor a(B+1)\\rfloor/(B+1)\\le a$, and the same holds for $p_U$. For $\\alpha<1$ the event $\\min\\{1,2\\min(p_L,p_U)\\}\\le\\alpha$ is the union of $\\{p_L\\le\\alpha/2\\}$ and $\\{p_U\\le\\alpha/2\\}$. Its probability is at most $\\alpha$.", "problem": "The validity statements on pp.~397 and 400 are correct for the rule as printed. With a fixed statistic and $B$ fresh reference draws, the rule that rejects when the $p$ value is less than $.05$ has null rejection probability at most $50/1001$ for $B=1000$ and at most $500/10001$ for $B=10{,}000$. The simulation code applies a different rule. Each tail is the proportion of the $B$ reference statistics that are at most, or at least, the observed statistic, and the observed statistic is not counted among them. The $p$ value is twice the smaller tail, capped at $1$, and a rejection is counted when $p\\le.05$. This rule is not a test of level $.05$ in general: its size can exceed $.05$ slightly, and its $p$ value can equal $0$.", "what_survives": "The validity statement for the exact randomization test on p.~397 and the printed rule at level $.05$ stand, as do the statistics and the framework. No reported rejection rate, power ranking or figure is shown to change. Note that the counterexample is a constructed design of sixteen units and not the simulation of 256 units in the note, and the excess over $.05$ is a fraction of a percentage point. The $p$ values of the application are computed from a fixed saved set of assignments; they need a separate argument and were not examined here." }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-E008", "work": { "arxiv_id": null, "doi": "10.1093/pan/mpw018", "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "url": "https://pmaronow.github.io/llm-audit/works/P016/" }, "work_id": "P016" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Five lines of text from page 402. A red box, in two parts across a line break, marks the words 'outside the rectangle with corners (-0.6, 0.2) and (0.3, -0.4).'.", "caption": "Section 3, last sentence, p. 402. The mark shows the rectangle that summarizes the SSR region. The defect is in the endpoint summaries that the application script saves; the printed rectangle is not shown to come from them.", "url": "https://pmaronow.github.io/llm-audit/works/P016/exhibits/P016-E010.png" } ], "formal_release": null, "formal_statements": [], "headline": "The application script saves, as the upper endpoint of an interior accepted range, the first rejected grid point and not the last accepted one; the $p$ values, Fig. 4 and the printed rectangle are not shown to depend on this.", "id": "P016-E010", "location": "Section 3, last sentence, p. 402 (PDF p. 8); the endpoint helper find_breaks is in the application script of the replication files, lines 12–18, 77–78 and 83–84", "pdf_page": 8, "printed_page": "402", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "it would be very unlikely that the data were generated from the model with parameters outside the rectangle with corners $(-0.6, 0.2)$ and $(0.3, -0.4)$.\n\nHelper \\texttt{find\\_breaks} of \\texttt{CoppockJEPS\\_figure2code.R}, the application script of the replication files:\n\n\\texttt{breaks[i+1] <- x[i]!=x[i+1]}\n\n\\texttt{return(which(breaks))}\n\nIn the main part of the script, and likewise for the indirect effect:\n\n\\texttt{direct\\_breaks <- find\\_breaks(apply(pmat.ssr, MARGIN=1, FUN=max) >.05)}\n\n\\texttt{directs.95.2 <- directs[\\hl{direct\\_breaks}]}", "corrected": "For each parameter, the endpoints of the accepted range are the extremes of the grid coordinates whose projected $p$ value exceeds $.05$, that is \\texttt{range(grid[accepted])} in place of the transition positions, and an empty accepted set is reported as empty. When a projection is accepted at positions $r,\\ldots,s$, its endpoints are the grid coordinates $g_r$ and $\\hl{g_s}$, not $g_r$ and $g_{s+1}$. If the accepted set is not connected, the product of the two ranges is a bounding rectangle and not the accepted set itself. This corrects the endpoint extraction only; it does not alter the matrices of $p$ values or establish corrected values for the rounded rectangle on p.~402.", "evidence": "I checked the derivation and an exact counterexample using the original code.\n\nLet $a_i$ indicate that the largest $p$ value in row $i$ of the matrix exceeds $.05$, for $i=1,\\ldots,n$. The helper sets entry $i+1$ of its output to the truth value of $a_i\\ne a_{i+1}$ for $i=1,\\ldots,n-1$, and it returns the positions of the true entries. If the accepted positions are exactly $r,\\ldots,s$ with $1p. 397 (PDF p. 3), Section 2, before equation (1). The pairing and assignment-design witnesses give both statistic tables, both tail masses and both design covariances.

", "headline": "Section 2, the two sentences that lead into equation (1): the KS test ‘ignoring information about the joint distribution’ and the simplest SSR ‘including the correlation between treated and control outcomes as a part of the distribution of the statistic’, as printed; the witnesses are the validator's constructions and are not in the paper", "location": "p. 397 (PDF p. 3), Section 2, second paragraph, with equation (1)" }, "id": "P016-T1", "text": { "body": "p. 397 (PDF p. 3), Section 2, before equation (1). The pairing and assignment-design witnesses give both statistic tables, both tail masses and both design covariances.", "headline": "Section 2, the two sentences that lead into equation (1): the KS test ‘ignoring information about the joint distribution’ and the simplest SSR ‘including the correlation between treated and control outcomes as a part of the distribution of the statistic’, as printed; the witnesses are the validator's constructions and are not in the paper", "location": "p. 397 (PDF p. 3), Section 2, second paragraph, with equation (1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T1" }, { "declarations": [ { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "This is a positive exact witness in corrected target T2(G), and also shows why equality of the scalar KS statistic is weaker than equality of both empirical margins.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_scalar_ks_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L350", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "The infimum has a nonempty range bounded below by zero, and the exported equality and coefficient equivalence identify the attained unique minimum.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_least_squares", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L783", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "The hypothesis n≥2 is redundant under the two positive group-size assumptions, and the claim itself includes all three.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_decomposition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L838", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "Fixed outcomes and fixed group sizes are explicit claim hypotheses, not added restrictions.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_tail_equivalence", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L857", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "The assignment-map formulation covers every finite target design, and the inclusive tails match. The public description states exactly these restrictions and this equality.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_tail_probability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L885", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "The group second moments use divisor n_g, exactly as the target formula requires.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_group_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L899", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "Redundant: the used t2_margin_invariance already gives both SSR and KS equality.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_rss_margin_invariance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L938", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "Equal multiset cardinalities imply nonempty groups for the second assignment; ties and multiplicities are retained. The public description accurately states this invariance theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_margin_invariance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1068", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "An arbitrary finite probability space with an assignment map represents every target design by its pushforward; nonuniform and zero weights are allowed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_design_dependence", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1141", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "conditional_on": [], "credit_reason": "The graph column is degree only, precisely the SSR + Degree witness in T2(F).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_network_alignment", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1168", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:d495c131035f86d02a1921d0c22e40272a77031ae072da6eb5057c4479da0699", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T2", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T2" }, "html": { "body": "

p. 397 (PDF p. 3), Section 2, before equation (1), corrected claim. The simple SSR has the least-squares, moment, ordering, margin and design identities, with the network-alignment and scalar-KS witnesses.

", "headline": "Section 2, the information contrast before equation (1), as repaired by Sol: the simple SSR is a function of the two group margins and its lower-tail test is the absolute difference-in-means test", "location": "p. 397 (PDF p. 3), Section 2, second paragraph, with equation (1)" }, "id": "P016-T2", "text": { "body": "p. 397 (PDF p. 3), Section 2, before equation (1), corrected claim. The simple SSR has the least-squares, moment, ordering, margin and design identities, with the network-alignment and scalar-KS witnesses.", "headline": "Section 2, the information contrast before equation (1), as repaired by Sol: the simple SSR is a function of the two group margins and its lower-tail test is the absolute difference-in-means test", "location": "p. 397 (PDF p. 3), Section 2, second paragraph, with equation (1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T2" }, { "declarations": [ { "claim_revision_id": "catalog:6c69fc5e4c71bde3e936d61563324a7dae4d9f25bf25fefd8a5b3097a897c644", "conditional_on": [], "credit_reason": "The row establishes the assigned affine-normalization witness; it makes no full refutation claim for an arbitrary unspecified vector-valued g.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t3_four_unit_exposure", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L536", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6c69fc5e4c71bde3e936d61563324a7dae4d9f25bf25fefd8a5b3097a897c644", "conditional_on": [], "credit_reason": "The target itself requires reading the displayed decimals as exact rationals; this is a direct match, so I retain faithful rather than the record’s faithful_with_notes classification.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t3_determinants", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L572", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6c69fc5e4c71bde3e936d61563324a7dae4d9f25bf25fefd8a5b3097a897c644", "conditional_on": [], "credit_reason": "The target is explicitly about the recorded rational rows, so no unavailable full-precision score reconstruction is required.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t3_table_not_affine", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L586", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6c69fc5e4c71bde3e936d61563324a7dae4d9f25bf25fefd8a5b3097a897c644", "conditional_on": [], "credit_reason": "The row is a numerical witness for the explicitly specified tail rules, not a claim about a particular simulated SSR dataset or general test validity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t3_cutoff_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1387", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6c69fc5e4c71bde3e936d61563324a7dae4d9f25bf25fefd8a5b3097a897c644", "conditional_on": [], "credit_reason": "All least-squares minima are exported as values of the actual infimum definition, rather than merely values at chosen coefficients.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t3_six_unit_exposure", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1480", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:6c69fc5e4c71bde3e936d61563324a7dae4d9f25bf25fefd8a5b3097a897c644", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T3" }, "html": { "body": "

p. 400 (PDF p. 6), Section 3 and Section 2.2; Fig. 3, p. 401 (PDF p. 7). The four-unit, six-unit, recorded-table and cutoff witnesses establish the stated affine obstructions, regression minima and rejection-rule discrepancies.

", "headline": "Section 3, the application model y0=yz−β1z−β2g(Γz)\\mathbf{y}_0 = \\mathbf{y}_{\\mathbf{z}} - \\beta_1\\mathbf{z} - \\beta_2 g(\\Gamma\\mathbf{z})y0​=yz​−β1​z−β2​g(Γz) with ‘ggg ... normalizes the sums of distances to treated neighbors to have unit variance’, and the rejection rule ‘ppp values ... less than .05’ of Section 2.2, as printed; the witnesses are the validators' constructions and are not in the paper", "location": "p. 400 (PDF p. 6), Section 3, the unnumbered display and the two sentences after it; p. 400, Section 2.2, and the caption of Fig. 3, p. 401 (PDF p. 7), for the rejection rule" }, "id": "P016-T3", "text": { "body": "p. 400 (PDF p. 6), Section 3 and Section 2.2; Fig. 3, p. 401 (PDF p. 7). The four-unit, six-unit, recorded-table and cutoff witnesses establish the stated affine obstructions, regression minima and rejection-rule discrepancies.", "headline": "Section 3, the application model $\\mathbf{y}_0 = \\mathbf{y}_{\\mathbf{z}} - \\beta_1\\mathbf{z} - \\beta_2 g(\\Gamma\\mathbf{z})$ with ‘$g$ ... normalizes the sums of distances to treated neighbors to have unit variance’, and the rejection rule ‘$p$ values ... less than .05’ of Section 2.2, as printed; the witnesses are the validators' constructions and are not in the paper", "location": "p. 400 (PDF p. 6), Section 3, the unnumbered display and the two sentences after it; p. 400, Section 2.2, and the caption of Fig. 3, p. 401 (PDF p. 7), for the rejection rule" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T3" }, { "declarations": [ { "claim_revision_id": "catalog:a310d7df4092992c541153a42e5275124790709a593c5011948fab487366dd8d", "conditional_on": [], "credit_reason": "The empty-design specialization is harmless: the row covers every nonempty target design as well. The forward-model premise is precisely T4(a), rather than an imported assumption of a substantive result.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t4_application_mapping", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L600", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:a310d7df4092992c541153a42e5275124790709a593c5011948fab487366dd8d", "conditional_on": [], "credit_reason": "The general real-order identity specializes directly to the target p2 and loses no cases.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t4_cutoff_difference", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L656", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:a310d7df4092992c541153a42e5275124790709a593c5011948fab487366dd8d", "conditional_on": [], "credit_reason": "B>0 is the implicit requirement for the target proportions and is satisfied by the stated application and simulation sample sizes. There is no sampled-test calibration conclusion in this row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t4_tail_coverage", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1347", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:a310d7df4092992c541153a42e5275124790709a593c5011948fab487366dd8d", "conditional_on": [], "credit_reason": "The infimum formulation has its ordinary meaning: the finite least-squares objective range is nonempty and nonnegative, and a finite-dimensional least-squares minimum is attained. No rank hypothesis is added.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t4_standardization", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1419", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:a310d7df4092992c541153a42e5275124790709a593c5011948fab487366dd8d", "conditional_on": [], "credit_reason": "This is an explicit witness for need-not-be-affine, not a universal assertion that centering always changes the span.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t4_centering_not_rescaling", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1578", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:a310d7df4092992c541153a42e5275124790709a593c5011948fab487366dd8d", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P016/P016_Interference.lean:1645 places hm, requiring every bank assignment to have m treated units, before the entire three-part conclusion. T4(c) asserts the raw identity (Sz)_i = m - z_i - (Dz)_i/2 whenever the current z has m treated units, and introduces the fixed-size-bank condition only for the centered and standardized identities. A bank containing the all-control and all-treated assignments meets the standing coverage condition but fails hm for a mixed z, so this nontrivial case of the raw identity is excluded. The centered and standardized conclusions otherwise match their claimed scope.', 'suggested_fix': 'Move hm into an implication governing only the centered and standardized conclusions, leaving the raw-sum identity outside that implication; alternatively split off and credit a raw-identity declaration with no fixed-count-bank premise.', 'second_review': 'P016/P016_Interference.lean:1645 places the fixed-count-bank hypothesis hm at line 1648 before all three conclusions, including the raw similarity-sum identity. T4(c) requires that identity for every current assignment with m treated units and introduces the bank condition only for the centered and standardized identities. A bank containing the all-control and all-treated assignments satisfies coverage but fails hm for a mixed current assignment, excluding an allowed class of instances of the raw identity. Export the raw identity separately without hm, or move hm into an implication governing the final two conclusions.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P016.t4_similarity_orientation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1613", "status": "proved", "support_scope": "limited", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:a310d7df4092992c541153a42e5275124790709a593c5011948fab487366dd8d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T4", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T4" }, "html": { "body": "

p. 400 (PDF p. 6), Sections 2.2 and 3; pp. 398 and 401, corrected claim. The bank-centered model has the stated cancellation, standardization, regression invariance, orientation, nonaffine witnesses and inclusive-tail cutoff identities.

", "headline": "Section 3, the application model and the description of the tests, as repaired by Sol: the centred similarity exposure, the application regression, and the two ppp-value rules", "location": "p. 400 (PDF p. 6), Section 3, the unnumbered display and the two sentences after it; p. 400, Section 2.2, p. 398 (PDF p. 4) and the caption of Fig. 3, p. 401 (PDF p. 7), for the rejection rule" }, "id": "P016-T4", "text": { "body": "p. 400 (PDF p. 6), Sections 2.2 and 3; pp. 398 and 401, corrected claim. The bank-centered model has the stated cancellation, standardization, regression invariance, orientation, nonaffine witnesses and inclusive-tail cutoff identities.", "headline": "Section 3, the application model and the description of the tests, as repaired by Sol: the centred similarity exposure, the application regression, and the two $p$-value rules", "location": "p. 400 (PDF p. 6), Section 3, the unnumbered display and the two sentences after it; p. 400, Section 2.2, p. 398 (PDF p. 4) and the caption of Fig. 3, p. 401 (PDF p. 7), for the rejection rule" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T4" }, { "declarations": [ { "claim_revision_id": "catalog:f8ca38a9dfe4ff4002b4e7945bd3a82e71d2313f09ed7903aeb0cf1d4e5bcfa9", "conditional_on": [], "credit_reason": "Marked wrong_credit: fixed empirical-margin invariance does not refute the printed distribution-level information comparison.", "credit_status": "wrong_credit", "fidelity_status_baseline": "wrong_credit", "lean_name": "AronowAudit.P016.t5_printed_contrast_refuted", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1218", "status": "counterexample_proved", "support_scope": "withdrawn", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:f8ca38a9dfe4ff4002b4e7945bd3a82e71d2313f09ed7903aeb0cf1d4e5bcfa9", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T5", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T5" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Section 2, the comparison of the information used by the KS and by the simplest SSR statistic, with equation (1); as printed", "location": "p. 397 (PDF p. 3), Section 2, second paragraph, and equation (1)" }, "id": "P016-T5", "text": { "body": "No current theorem is supplied for this target.", "headline": "Section 2, the comparison of the information used by the KS and by the simplest SSR statistic, with equation (1); as printed", "location": "p. 397 (PDF p. 3), Section 2, second paragraph, and equation (1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T5" }, { "declarations": [ { "claim_revision_id": "catalog:1fdc5c94863f5cf59a6e1571ff58d9871200380c229d94d2bdedbf86862f0d0b", "conditional_on": [], "credit_reason": "The used general finite-design theorem contains this single-assignment cancellation, with the same sample-variance restriction.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P016.t6_printed_model_consistency", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L623", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "catalog:1fdc5c94863f5cf59a6e1571ff58d9871200380c229d94d2bdedbf86862f0d0b", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P016/P016_Interference.lean:659 states printedReject p iff p < 1/20, with printedReject defined at line 539 by precisely that strict inequality. T6's component here is the printed rejection convention, expressed as p values less than .05, and 1/20 is exactly .05. A definitional transcription supplies all the content claimed for this component; the row promises neither a size bound nor verification of the released implementation. Its definitional proof therefore creates no fidelity defect.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t6_printed_cutoff", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L634", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1fdc5c94863f5cf59a6e1571ff58d9871200380c229d94d2bdedbf86862f0d0b", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P016/P016_Interference.lean:667 requires sampleVariance(g(Γz)) = 1 on every design assignment through hg at line 670; sampleVariance fixes divisor n-1 at line 505. Printed T6 only requires g to normalize exposure to unit variance and does not restrict it to this divisor. For two units with Γ = [[0,1],[1,0]] and the two one-treated assignments, population standardization gives vectors (-1,1) and (1,-1), whose population variance is one but sample variance is two. These nondegenerate normalizers are excluded, although the same model-consistency cancellation holds at the original g and coefficient values.', 'suggested_fix': 'Remove hg from the credited consistency theorem while retaining the displayed-model premise hy, so consistency holds for every g. Normalization can be specified separately without restricting this algebraic result to the n-1 convention.', 'second_review': \"P016/P016_Interference.lean:667 requires sampleVariance(g(Gamma z)) = 1 throughout the design at line 670, with divisor n-1 fixed at line 505. T6's printed claim describes g as normalizing to unit variance without specifying that divisor, so it also permits population normalization. For two units with off-diagonal distances one and the two one-treated assignments, population normalization yields (-1,1) and (1,-1), each of population variance one but sample variance two, excluding nondegenerate model instances at their given parameters. Remove hg from the consistency statement and retain the model premise hy; the cancellation applies to arbitrary g.\", 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P016.t6_printed_general_model_consistency", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L642", "status": "proved", "support_scope": "limited", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1fdc5c94863f5cf59a6e1571ff58d9871200380c229d94d2bdedbf86862f0d0b", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T6", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T6" }, "html": { "body": "

p. 400 (PDF p. 6), Section 3 and Section 2.2; pp. 398 and 401, as printed. For any map g with across-unit sample variance one on every assignment of a nonempty finite design, the displayed generation equation implies recovery of the baseline outcomes; the printed rejection convention is p < 1/20.

", "headline": "Section 3, the application model, and the rejection rule of the simulations; as printed", "location": "p. 400 (PDF p. 6), Section 3, the unnumbered display and the sentences around it; p. 400, Section 2.2; caption of Fig. 3, p. 401 (PDF p. 7); p. 398 (PDF p. 4)" }, "id": "P016-T6", "text": { "body": "p. 400 (PDF p. 6), Section 3 and Section 2.2; pp. 398 and 401, as printed. For any map g with across-unit sample variance one on every assignment of a nonempty finite design, the displayed generation equation implies recovery of the baseline outcomes; the printed rejection convention is p < 1/20.", "headline": "Section 3, the application model, and the rejection rule of the simulations; as printed", "location": "p. 400 (PDF p. 6), Section 3, the unnumbered display and the sentences around it; p. 400, Section 2.2; caption of Fig. 3, p. 401 (PDF p. 7); p. 398 (PDF p. 4)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T6" }, { "declarations": [ { "claim_revision_id": "catalog:92ab1de07c3f88f89bfe33a3247bab1f548546a7b594d66c1806376a560c6c28", "conditional_on": [], "credit_reason": "The row is the numerical witness component of T7(i); the separate credited row R114-11 exports rejection with probability one for every positive level.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t7_mean_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L405", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:92ab1de07c3f88f89bfe33a3247bab1f548546a7b594d66c1806376a560c6c28", "conditional_on": [], "credit_reason": "This witnesses the literal raw-argument recipe using the target absolute-tail convention; it does not refute the intended adjusted procedure or every possible choice of tail.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t7_mean_rejects_truth", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L419", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:92ab1de07c3f88f89bfe33a3247bab1f548546a7b594d66c1806376a560c6c28", "conditional_on": [], "credit_reason": "The target specifies inclusive upper tails and a strict .05 cutoff for this witness. The paper itself gives no tail formula; this is the stated literal-argument witness, not a claim that every interpretation of the printed comparison is invalid. Metadata status proved and resolution theorem_proved refer to a proved witness theorem and do not conflict with target_kind counterexample.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.Wave1.t7_ks_witness_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/Wave1/Gap/GLP027P016P031P016B2.lean#L439", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:92ab1de07c3f88f89bfe33a3247bab1f548546a7b594d66c1806376a560c6c28", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T7", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T7" }, "html": { "body": "

p. 396 (PDF p. 2), Section 1, steps (2)-(4). The four-unit mean and seventy-assignment KS witnesses show probability-one rejection of true additive hypotheses by the literal raw-versus-adjusted comparison, with the stated corrected reference probabilities.

", "headline": "Section 1, step (4) of the testing procedure, ‘comparing the observed test statistic, T(Yi,Zi)\\mathcal{T}(Y_i, Z_i)T(Yi​,Zi​) against the distribution of that test statistic that characterizes the hypothesis’, as printed; the witnesses are the validator's constructions and are not in the paper", "location": "p. 396 (PDF p. 2), Section 1, second full paragraph, steps (2)–(4)" }, "id": "P016-T7", "text": { "body": "p. 396 (PDF p. 2), Section 1, steps (2)-(4). The four-unit mean and seventy-assignment KS witnesses show probability-one rejection of true additive hypotheses by the literal raw-versus-adjusted comparison, with the stated corrected reference probabilities.", "headline": "Section 1, step (4) of the testing procedure, ‘comparing the observed test statistic, $\\mathcal{T}(Y_i, Z_i)$ against the distribution of that test statistic that characterizes the hypothesis’, as printed; the witnesses are the validator's constructions and are not in the paper", "location": "p. 396 (PDF p. 2), Section 1, second full paragraph, steps (2)–(4)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T7" }, { "declarations": [ { "claim_revision_id": "catalog:8fc33a0ac34c7eceb1bab5fadf76cce17791886fde1f2b75054c696edb9c3682", "conditional_on": [], "credit_reason": "Both identities apply to arbitrary finite baselines and additive parameters, exactly as required by T8(a).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t8_additive_mapping", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L398", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:8fc33a0ac34c7eceb1bab5fadf76cce17791886fde1f2b75054c696edb9c3682", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P016_Interference.lean:1780 gives both the equality of statistic laws and the inclusive upper-tail bound for every alpha in [0,1], as required by T8(b). By the definitions at lines 84 and 88, true additive generation followed by adjustment returns y0 for every assignment, so the observed-assignment index in the inner reference expression does not make the adjusted vector vary across reference draws. Every target assignment design is recovered by taking its finite assignment subtype as Omega and the inclusion as assign; allowing duplicate indices, zero weights, or index-sensitive statistics excludes no target case. Finite support also makes arbitrary event predicates harmless.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t8_corrected_comparison", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1748", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:8fc33a0ac34c7eceb1bab5fadf76cce17791886fde1f2b75054c696edb9c3682", "conditional_on": [], "credit_reason": "The control-nonempty hypothesis is included in the target even though only treated positivity is needed for this shift identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t8_raw_mean_shift", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1825", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:8fc33a0ac34c7eceb1bab5fadf76cce17791886fde1f2b75054c696edb9c3682", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T8", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T8" }, "html": { "body": "

p. 396 (PDF p. 2), Section 1, step (4), corrected claim. The additive adjustment fixes the true baseline, gives the same observed and reference law with calibrated inclusive upper tails, and removes exactly the raw mean shift.

", "headline": "Section 1, step (4) of the testing procedure, corrected: the observed statistic is T(y~i,Zi=0,Zi)\\mathcal{T}(\\tilde y_{i,Z_i=0}, Z_i)T(y~​i,Zi​=0​,Zi​) at the realized assignment", "location": "p. 396 (PDF p. 2), Section 1, second full paragraph, step (4)" }, "id": "P016-T8", "text": { "body": "p. 396 (PDF p. 2), Section 1, step (4), corrected claim. The additive adjustment fixes the true baseline, gives the same observed and reference law with calibrated inclusive upper tails, and removes exactly the raw mean shift.", "headline": "Section 1, step (4) of the testing procedure, corrected: the observed statistic is $\\mathcal{T}(\\tilde y_{i,Z_i=0}, Z_i)$ at the realized assignment", "location": "p. 396 (PDF p. 2), Section 1, second full paragraph, step (4)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T8" }, { "declarations": [ { "claim_revision_id": "catalog:cc61c29ab65a83b0e15b4b6b76736c9223898c420c706672d948adfef547e96d", "conditional_on": [], "credit_reason": "The signed statistic is a legitimate one-sided maximum. The target component is its discrepancy from the described two-sided statistic, rather than invalidity of one-sided KS testing.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t9_signed_ks_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L356", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:cc61c29ab65a83b0e15b4b6b76736c9223898c420c706672d948adfef547e96d", "conditional_on": [], "credit_reason": "The row is a discrepancy witness under the pooled interpretation, not a universal formal theorem about every u and every design. The target note expressly identifies the pooled interpretation as conditional on a meaning of the undefined bar.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t9_pooled_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L367", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:cc61c29ab65a83b0e15b4b6b76736c9223898c420c706672d948adfef547e96d", "conditional_on": [], "credit_reason": "The intended component is the discrepancy witness under the explicitly listed-regressors-only interpretation. As the target note warns, this does not establish that an unstated intercept is itself a paper error.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t9_intercept_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2064", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:cc61c29ab65a83b0e15b4b6b76736c9223898c420c706672d948adfef547e96d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T9", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T9" }, "html": { "body": "

footnote 3, p. 396; equation (1), p. 397; equation (2), p. 399. The signed-KS, pooled-mean and missing-intercept readings give the three specified discrepancies, including the general constant-statistic law and inclusive tails and the complete six-assignment SSR table.

", "headline": "The three displayed definitions of the test statistics as printed: the KS formula of footnote 3 (no absolute value), equation (1) (mean not specified) and equation (2) (no row index; no intercept listed); the witnesses are the validator's constructions and are not in the paper", "location": "footnote 3, p. 396 (PDF p. 2); equation (1), p. 397 (PDF p. 3); equation (2) and the sentence before it, p. 399 (PDF p. 5)" }, "id": "P016-T9", "text": { "body": "footnote 3, p. 396; equation (1), p. 397; equation (2), p. 399. The signed-KS, pooled-mean and missing-intercept readings give the three specified discrepancies, including the general constant-statistic law and inclusive tails and the complete six-assignment SSR table.", "headline": "The three displayed definitions of the test statistics as printed: the KS formula of footnote 3 (no absolute value), equation (1) (mean not specified) and equation (2) (no row index; no intercept listed); the witnesses are the validator's constructions and are not in the paper", "location": "footnote 3, p. 396 (PDF p. 2); equation (1), p. 397 (PDF p. 3); equation (2) and the sentence before it, p. 399 (PDF p. 5)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T9" }, { "declarations": [ { "claim_revision_id": "catalog:319376436952627b39ef48a4077db9226449fc0ab3b9f5da3ed13468ef630d46", "conditional_on": [], "credit_reason": "Attainment and uniqueness of the fitted vector hold for every real design matrix on exactly the nonempty-group outcome domain of T10(b).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t10_general_least_squares", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1294", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:319376436952627b39ef48a4077db9226449fc0ab3b9f5da3ed13468ef630d46", "conditional_on": [], "credit_reason": "The sample KS maximum is equated with the threshold supremum, with label symmetry and the full [0,1] range as required by T10(c).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t10_ks_supremum", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1303", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:319376436952627b39ef48a4077db9226449fc0ab3b9f5da3ed13468ef630d46", "conditional_on": [], "credit_reason": "The symmetric binary-network identities retain diagonal entries and exactly establish T10(d) on the stated nonempty-group domain.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.Wave1.t10_symmetric_network", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/Wave1/Gap/FidP016.lean#L9", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:319376436952627b39ef48a4077db9226449fc0ab3b9f5da3ed13468ef630d46", "conditional_on": [], "credit_reason": "The statement establishes the group-centered minimum and unique intercept/slope coefficients of T10(a), with only the redundant n≥2 premise.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t2_least_squares", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L783", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:319376436952627b39ef48a4077db9226449fc0ab3b9f5da3ed13468ef630d46", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T10", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T10" }, "html": { "body": "

footnote 3, p. 396; equation (1), p. 397; equation (2), p. 399, corrected claim. The corrected SSR minima are attained with unique fitted values, KS equals its threshold supremum and is label invariant in [0,1], and symmetric adjacency gives the stated network columns.

", "headline": "Footnote 3 and equations (1) and (2), corrected: the two-sided KS distance, the group-mean residual sum of squares and the indexed least-squares residual sum of squares with an intercept", "location": "footnote 3, p. 396 (PDF p. 2); equation (1), p. 397 (PDF p. 3); equation (2) and the sentence before it, p. 399 (PDF p. 5)" }, "id": "P016-T10", "text": { "body": "footnote 3, p. 396; equation (1), p. 397; equation (2), p. 399, corrected claim. The corrected SSR minima are attained with unique fitted values, KS equals its threshold supremum and is label invariant in [0,1], and symmetric adjacency gives the stated network columns.", "headline": "Footnote 3 and equations (1) and (2), corrected: the two-sided KS distance, the group-mean residual sum of squares and the indexed least-squares residual sum of squares with an intercept", "location": "footnote 3, p. 396 (PDF p. 2); equation (1), p. 397 (PDF p. 3); equation (2) and the sentence before it, p. 399 (PDF p. 5)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T10" }, { "declarations": [ { "claim_revision_id": "catalog:5251e9fbd4d7c3adac9ceb11fe677510863af68d57a13274e4a472fae1431fc2", "conditional_on": [], "credit_reason": "The actual vectors meet the directional effect-increasing antecedent; no definition of effect increasing is assumed as a premise. This says nothing about bias or validity of randomization p values.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t11_ks_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L374", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5251e9fbd4d7c3adac9ceb11fe677510863af68d57a13274e4a472fae1431fc2", "conditional_on": [], "credit_reason": "The explicit unchanged control coordinates supply the required control direction even though the exported coordinatewise inequalities alone do not express the general effect-increasing antecedent. The target note says the paper does not claim SSR is effect increasing; this is an auxiliary validator witness, as credited.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t11_rss_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L387", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:5251e9fbd4d7c3adac9ceb11fe677510863af68d57a13274e4a472fae1431fc2", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T11", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T11" }, "html": { "body": "

pp. 396-397 (PDF pp. 2-3), the quoted statistic criterion and following sentence. The weak and strict KS witnesses and the two opposite SSR changes refute effect-increasing monotonicity for both KS versions and both SSR orientations.

", "headline": "Section 1, ‘test statistics with this property (“effect increasing” test statistics)’, which identifies a statistic that is small for similar and large for divergent distributions with an effect increasing statistic, as printed; the witnesses are the validator's constructions and are not in the paper", "location": "pp. 396–397 (PDF pp. 2–3): the quotation that runs from the last line of p. 396 to the first lines of p. 397, and the sentence after it" }, "id": "P016-T11", "text": { "body": "pp. 396-397 (PDF pp. 2-3), the quoted statistic criterion and following sentence. The weak and strict KS witnesses and the two opposite SSR changes refute effect-increasing monotonicity for both KS versions and both SSR orientations.", "headline": "Section 1, ‘test statistics with this property (“effect increasing” test statistics)’, which identifies a statistic that is small for similar and large for divergent distributions with an effect increasing statistic, as printed; the witnesses are the validator's constructions and are not in the paper", "location": "pp. 396–397 (PDF pp. 2–3): the quotation that runs from the last line of p. 396 to the first lines of p. 397, and the sentence after it" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T11" }, { "declarations": [ { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "conditional_on": [], "credit_reason": "The inclusive upper-tail calibration bound matches the upper-tail component of T12(iii).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t12_upper_valid", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L290", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "conditional_on": [], "credit_reason": "The inclusive lower-tail bound uses the strictly positive finite law and [0,1] level domain explicitly required by T12(iii).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t12_lower_valid", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L298", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "conditional_on": [], "credit_reason": "The inclusive capped doubled-minimum calibration bound matches the two-tail component of T12(iii).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t12_two_tail_valid", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L306", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "conditional_on": [], "credit_reason": "The six exact upper p-values equal the table required by the fixed example in T12(iv).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t12_ks_p_table", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L431", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "conditional_on": [], "credit_reason": "The exact rejection probability has precisely the three threshold ranges specified in T12(iv).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t12_ks_size", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L438", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "conditional_on": [], "credit_reason": "The definition expands to the directional mean-difference monotonicity on nonempty groups required by T12(i). The n<2 cases are empty-domain cases, but the universally quantified family has nondegenerate instances for every n≥2.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t12_mean_effect_increasing", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1713", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "conditional_on": [], "credit_reason": "Its conclusion uses the total extension of KS to empty groups. The solution instead instantiates its existing exported four-unit witness to state failure on nonempty groups.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t12_ks_not_effect_increasing", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1732", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:d5fab4570267f3f3b05424775062754a5f7d9444ab14ed3917c798a636003591", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T12", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T12" }, "html": { "body": "

p. 397 (PDF p. 3), first paragraph, second and third sentences. The mean contrast is effect increasing on nonempty groups, whereas both KS statistics fail this directional condition. Every statistic calibrated under a positive finite assignment law has valid inclusive upper, lower, and doubled minimum tails, at weak and strict cutoffs. For the uniform two-of-four design and outcomes (0,1,2,3), the KS scores are (1,1/2,1/2,1/2,1/2,1), the upper p values are (1/3,1,1,1,1,1/3), and the rejection probabilities are 0, 1/3, and 1 on the stated threshold ranges.

", "headline": "Section 1, the two sentences after the quotation of BFP's advice, as repaired by Sol: the effect increasing property stated, and sharp-null validity for any test statistic", "location": "p. 397 (PDF p. 3), first paragraph, second and third sentences" }, "id": "P016-T12", "text": { "body": "p. 397 (PDF p. 3), first paragraph, second and third sentences. The mean contrast is effect increasing on nonempty groups, whereas both KS statistics fail this directional condition. Every statistic calibrated under a positive finite assignment law has valid inclusive upper, lower, and doubled minimum tails, at weak and strict cutoffs. For the uniform two-of-four design and outcomes (0,1,2,3), the KS scores are (1,1/2,1/2,1/2,1/2,1), the upper p values are (1/3,1,1,1,1,1/3), and the rejection probabilities are 0, 1/3, and 1 on the stated threshold ranges.", "headline": "Section 1, the two sentences after the quotation of BFP's advice, as repaired by Sol: the effect increasing property stated, and sharp-null validity for any test statistic", "location": "p. 397 (PDF p. 3), first paragraph, second and third sentences" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T12" }, { "declarations": [ { "claim_revision_id": "catalog:4e6ad1e145e4b1bff0b0e75ad6ccc4a1b99bfb2bd73823d2201527c286ba89dd", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"P016_Interference.lean:2365 correctly expresses the uniform rank law, the almost-sure count identities, and the weak-cutoff size formula of T13(i) under exchangeability and almost-sure distinctness. Its extra alpha < 1 hypothesis excludes the target's admissible alpha = 1 case when B is odd; then c = (B-1)/2, rawP <= 1 always, and 2(c+1)/(B+1) = 1. Nonnegative thresholds above 1 that still satisfy 2c < B give the same trivial identity. Thus the noted restriction is a literal but mathematically trivial narrowing; I partly confirm it as narrower_than_claim/minor under the stated exception, with no substantive failure of the rank calculation.\", 'suggested_fix': 'Remove hα1 : alpha < 1 from t13_no_ties, retaining 0 <= alpha, c = Nat.floor (alpha*B/2), and 2*c < B; handle alpha >= 1 by the probability-one identity. All substantive cases and the .05 witnesses already match the target.', 'second_review': 'P016/P016_Interference.lean:2365 adds alpha < 1 at line 2367, whereas T13(i) requires the size formula whenever c = floor(alpha B/2) and 2c < B. At alpha = 1 with odd B, the target permits c = (B-1)/2, rawP is always at most one by its definition at line 1374, and both sides of the formula equal one. Thus the statement omits only a trivial boundary case within the usual level range; remove the strict upper-bound premise and handle that case separately.', 'severity': 'minor'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P016.t13_no_ties", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2306", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:4e6ad1e145e4b1bff0b0e75ad6ccc4a1b99bfb2bd73823d2201527c286ba89dd", "conditional_on": [], "credit_reason": "The finite-design declaration already includes the same moment and RSS identity on the balanced design used in this claim.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t13_rss_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2365", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4e6ad1e145e4b1bff0b0e75ad6ccc4a1b99bfb2bd73823d2201527c286ba89dd", "conditional_on": [], "credit_reason": "The fixed power-of-two population has exactly the balanced support, moments, RSS identity, image cardinality, and complementary-pair fibers required by T13(ii). This row is an auxiliary or mechanism component, not a standalone full refutation of a printed validity claim.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t13_finite_design", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2386", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4e6ad1e145e4b1bff0b0e75ad6ccc4a1b99bfb2bd73823d2201527c286ba89dd", "conditional_on": [], "credit_reason": "The exact independent finite-design mixture, weak and strict cutoff inequalities, and positive zero-p probability establish the finite witness components T13(ii)–(iii), with no assertion refuting the printed strict rule. This is a counterexample to the target’s explicitly specified archived weak-cutoff algorithm, not to the paper’s strict rejection rule or its reported simulation rates.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.Wave1.t13_finite_rejection_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/Wave1/Gap/GLP027P016P031P016B1.lean#L911", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4e6ad1e145e4b1bff0b0e75ad6ccc4a1b99bfb2bd73823d2201527c286ba89dd", "conditional_on": [], "credit_reason": "The closed rational calculations are the advertised arithmetic auxiliaries for the no-ties size and finite-witness lower bound, without purporting to prove a probability law. This row is an auxiliary or mechanism component, not a standalone full refutation of a printed validity claim.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t13_numeric_certificates", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2417", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:4e6ad1e145e4b1bff0b0e75ad6ccc4a1b99bfb2bd73823d2201527c286ba89dd", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T13", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T13" }, "html": { "body": "

p. 400 (PDF p. 6), Section 2.2, and p. 397 (PDF p. 3), validity parenthesis; archived simulation weak-cutoff rule. Tie-free exchangeable scores have uniform ranks and raw weak-cutoff rejection probability 2(c+1)/(B+1) whenever c=floor(alpha B/2) and 2c<B. At .05 the sizes for B=1000 and 10000 exceed .05. In the uniform eight-of-sixteen design with fixed outcomes 2^i, the SSR has the stated moment formula and 6435 values, each on a complementary pair. Independent uniform observed and reference draws give the exact binomial-mixture rejection laws and bounds, including every positive B for the weak .05 rule. All five reported decimal prefixes are certified by open rational intervals. The strict .05 sizes are below .05, and raw p=0 has the stated positive power-sum probability. These witnesses concern the archived weak cutoff; they do not refute the printed strict rule.

", "headline": "The simulated permutation tests of Section 2.2 as the archived code implements them (raw empirical tail proportions; rejection when p≤.05p \\le .05p≤.05), set against ‘they reject the true null ... no more than 5% of the time at α=.05\\alpha = .05α=.05’; the witnesses are the validator's constructions and are not in the paper", "location": "p. 400 (PDF p. 6), Section 2.2, first and second paragraphs; p. 397 (PDF p. 3), the parenthesis ‘(in that the test should produce ppp values less than α\\alphaα no more than 100α%100\\alpha\\%100α% of the time when the null is true)’; code: pkgs/RItools/R/zzpvalues.R, simulation/teststat-parallel-results.R" }, "id": "P016-T13", "text": { "body": "p. 400 (PDF p. 6), Section 2.2, and p. 397 (PDF p. 3), validity parenthesis; archived simulation weak-cutoff rule. Tie-free exchangeable scores have uniform ranks and raw weak-cutoff rejection probability 2(c+1)/(B+1) whenever c=floor(alpha B/2) and 2cp. 400 (PDF p. 6), Section 2.2, and the validity parenthesis on p. 397 (PDF p. 3). Exchangeable references give rank bounds, the printed strict-cutoff guarantees, valid corrected tails, and iid permutation invariance. Exact-tail clause (e) uses the strictly positive finite-law domain of its explicit reference to canonical T12(iii); the iid exchangeability clause permits zero-weight points.

", "headline": "The level of the simulated permutation tests of Section 2.2, as repaired by Sol: the printed strict cutoff, and the rank-corrected Monte Carlo ppp value", "location": "p. 400 (PDF p. 6), Section 2.2, first and second paragraphs; p. 397 (PDF p. 3), the parenthesis on validity" }, "id": "P016-T14", "text": { "body": "p. 400 (PDF p. 6), Section 2.2, and the validity parenthesis on p. 397 (PDF p. 3). Exchangeable references give rank bounds, the printed strict-cutoff guarantees, valid corrected tails, and iid permutation invariance. Exact-tail clause (e) uses the strictly positive finite-law domain of its explicit reference to canonical T12(iii); the iid exchangeability clause permits zero-weight points.", "headline": "The level of the simulated permutation tests of Section 2.2, as repaired by Sol: the printed strict cutoff, and the rank-corrected Monte Carlo $p$ value", "location": "p. 400 (PDF p. 6), Section 2.2, first and second paragraphs; p. 397 (PDF p. 3), the parenthesis on validity" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T14" }, { "declarations": [ { "claim_revision_id": "catalog:42bcd028d29a9874004d843ce56b1e855b9984ffe52efcb27492b33ad0de7f7b", "conditional_on": [], "credit_reason": "The two explicit adjusted vectors and treated ECDF values give the required T15(a) witness, with unchanged controls visible in the vector equalities. The displayed vector equalities explicitly preserve the two control coordinates; no missing separate control-ECDF theorem is needed for this witness.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t15_adjustment_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2794", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:42bcd028d29a9874004d843ce56b1e855b9984ffe52efcb27492b33ad0de7f7b", "conditional_on": [], "credit_reason": "The ECDF values at 2 and 3 for sample (1,3) exactly give the T15(b) witness.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t15_ecdf_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2810", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:42bcd028d29a9874004d843ce56b1e855b9984ffe52efcb27492b33ad0de7f7b", "conditional_on": [], "credit_reason": "The diagonal-family equivalence and exclusion of (1/2,2) are precisely T15(c).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t15_diagonal_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2815", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:42bcd028d29a9874004d843ce56b1e855b9984ffe52efcb27492b33ad0de7f7b", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T15", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T15" }, "html": { "body": "

p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); and p. 400 (PDF p. 6), Section 2.2. The specified adjustment, ECDF, and diagonal-hypothesis witnesses establish all three local mathematical errors.

", "headline": "Three local statements as printed: ‘the hypotheses only have different implications for the control group distribution’ (p. 398); the ECDF ‘records the proportion of the distribution of xxx at or below xix_ixi​’ (footnote 3); ‘we tested the hypothesis τ=τ0,β=τ0\\tau = \\tau_0, \\beta = \\tau_0τ=τ0​,β=τ0​’ (p. 400); the witnesses are the validator's constructions and are not in the paper", "location": "p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); p. 400 (PDF p. 6), Section 2.2, first paragraph" }, "id": "P016-T15", "text": { "body": "p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); and p. 400 (PDF p. 6), Section 2.2. The specified adjustment, ECDF, and diagonal-hypothesis witnesses establish all three local mathematical errors.", "headline": "Three local statements as printed: ‘the hypotheses only have different implications for the control group distribution’ (p. 398); the ECDF ‘records the proportion of the distribution of $x$ at or below $x_i$’ (footnote 3); ‘we tested the hypothesis $\\tau = \\tau_0, \\beta = \\tau_0$’ (p. 400); the witnesses are the validator's constructions and are not in the paper", "location": "p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); p. 400 (PDF p. 6), Section 2.2, first paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T15" }, { "declarations": [ { "claim_revision_id": "catalog:c4b6d26cd3cb71a2b5e2b93c7384674b5b6456f47081595201b933c6884d911c", "conditional_on": [], "credit_reason": "The nonzero observed response qualification in the multiplicative parameter-dependence assertion is explicitly present in both the corrected target and Lean.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t16_group_adjustments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2828", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4b6d26cd3cb71a2b5e2b93c7384674b5b6456f47081595201b933c6884d911c", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P016_Interference.lean:2941 states that both additive and multiplicative adjustments leave the control ECDF unchanged at every real threshold, exactly the empirical-distribution component of T16(a). The nonzero multiplicative parameter matches the target's domain and restricts no additive parameter, since t is independent and a can always be chosen as 1. With a nonempty control group the ecdf definition at line 64 is the ordinary empirical proportion; its value 0 for an empty group is only an additional total-division convention. The ordinary nonempty-group identity does not hold merely by that convention.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t16_control_ecdf", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2851", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4b6d26cd3cb71a2b5e2b93c7384674b5b6456f47081595201b933c6884d911c", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P016_Interference.lean:2953 equates sampleECDF u x with the number of indices satisfying u i <= x divided by n, precisely T16(b)'s corrected explanation of the argument x. The definition at line 68 is the target's indicator-sum formula, so converting that sum to a cardinality expresses the intended mathematical identity. Total division at n = 0 extends the identity by 0 = 0, while every positive-size sample retains its usual proportion; this makes neither the whole statement vacuous nor the target domain narrower.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t16_ecdf_argument", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2863", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4b6d26cd3cb71a2b5e2b93c7384674b5b6456f47081595201b933c6884d911c", "conditional_on": [], "credit_reason": "The weak-looking existential is sufficient here because the target component is precisely membership of the fixed truth (1/2,2), together with its exclusion from the diagonal.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t16_joint_hypothesis", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L2870", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c4b6d26cd3cb71a2b5e2b93c7384674b5b6456f47081595201b933c6884d911c", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T16", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T16" }, "html": { "body": "

p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); p. 400 (PDF p. 6), Section 2.2, first paragraph. Under additive and nonzero multiplicative hypotheses, control outcomes and control empirical distributions stay fixed. Treated outcomes have the stated parameter dependence, with nonzero response for the multiplicative comparison. The sample ECDF counts observations at or below its argument. Empirical distributions are asserted on their nonempty-sample domains. The joint family with separate parameter indices contains the truth (1/2,2), which belongs to no member of the printed diagonal family.

", "headline": "The three local statements of p. 398, footnote 3 and p. 400, corrected: ‘treated group distribution’; ‘at or below xxx’; ‘τ=τ0,β=β0\\tau = \\tau_0, \\beta = \\beta_0τ=τ0​,β=β0​’", "location": "p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); p. 400 (PDF p. 6), Section 2.2, first paragraph" }, "id": "P016-T16", "text": { "body": "p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); p. 400 (PDF p. 6), Section 2.2, first paragraph. Under additive and nonzero multiplicative hypotheses, control outcomes and control empirical distributions stay fixed. Treated outcomes have the stated parameter dependence, with nonzero response for the multiplicative comparison. The sample ECDF counts observations at or below its argument. Empirical distributions are asserted on their nonempty-sample domains. The joint family with separate parameter indices contains the truth (1/2,2), which belongs to no member of the printed diagonal family.", "headline": "The three local statements of p. 398, footnote 3 and p. 400, corrected: ‘treated group distribution’; ‘at or below $x$’; ‘$\\tau = \\tau_0, \\beta = \\beta_0$’", "location": "p. 398 (PDF p. 4), last paragraph; footnote 3, p. 396 (PDF p. 2); p. 400 (PDF p. 6), Section 2.2, first paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T16" }, { "declarations": [ { "claim_revision_id": "catalog:bce5ecf69cbb07586d47fef06dccccbfe4008864b83ca5a364b304295118676c", "conditional_on": [], "credit_reason": "The printed treated symbol is undefined; the target explicitly makes the u+t interpretation diagnostic and conditional. This row faithfully proves those conditional algebraic identities, not a full refutation of a well-defined printed assertion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t17_adjusted_comparison", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1839", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:bce5ecf69cbb07586d47fef06dccccbfe4008864b83ca5a364b304295118676c", "conditional_on": [], "credit_reason": "The target itself states this witness in terms of u,τ,Z, so no additional raw-outcome premise or probability law is owed by this row. It is an illustrative exact instance, not a refutation of the undefined printed symbol.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t17_comparison_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L1859", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:bce5ecf69cbb07586d47fef06dccccbfe4008864b83ca5a364b304295118676c", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T17", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T17" }, "html": { "body": "

p. 397 (PDF p. 3), Section 2, first paragraph. The corrected comparison uses two adjusted group means, whereas the conditional imputed-treated reading gives the raw contrast, with the stated 2-versus-12 instance.

", "headline": "Section 2, first paragraph, ‘Comparing the mean of y~i,Zi=1∣Zi=1\\tilde y_{i,Z_i=1}|Z_i = 1y~​i,Zi​=1​∣Zi​=1 to the mean of y~i,Zi=0∣Zi=0\\tilde y_{i,Z_i=0}|Z_i = 0y~​i,Zi​=0​∣Zi​=0’, corrected: the first term is y~i,Zi=0∣Zi=1\\tilde y_{i,Z_i=0}|Z_i = 1y~​i,Zi​=0​∣Zi​=1", "location": "p. 397 (PDF p. 3), Section 2, first paragraph" }, "id": "P016-T17", "text": { "body": "p. 397 (PDF p. 3), Section 2, first paragraph. The corrected comparison uses two adjusted group means, whereas the conditional imputed-treated reading gives the raw contrast, with the stated 2-versus-12 instance.", "headline": "Section 2, first paragraph, ‘Comparing the mean of $\\tilde y_{i,Z_i=1}|Z_i = 1$ to the mean of $\\tilde y_{i,Z_i=0}|Z_i = 0$’, corrected: the first term is $\\tilde y_{i,Z_i=0}|Z_i = 1$", "location": "p. 397 (PDF p. 3), Section 2, first paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T17" }, { "declarations": [ { "claim_revision_id": "catalog:32ee6a487779aa316e5194baee367f9ad76689b2995504e3af4c08e85fffb2ad", "conditional_on": [], "credit_reason": "The target explicitly asks for constructed data, population variance with divisor 256, and the ordinary even median; all match directly. Zero is a global lower bound for a median of absolute discrepancies for every parameter, so the stated zero median is a minimum without further population hypotheses; the record’s historical proof-status remarks were not used in this statement-only audit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t18_calibration_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3219", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:32ee6a487779aa316e5194baee367f9ad76689b2995504e3af4c08e85fffb2ad", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T18", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T18" }, "html": { "body": "

p. 398 (PDF p. 4), last paragraph and Fig. 2; p. 399 (PDF p. 5), first sentences; released calibration code. The specified 256-unit witness attains the global minimum zero of the median criterion despite 127 unmatched treated units and unequal generated and adjusted mean shifts.

", "headline": "The choice of the multiplicative parameter of Fig. 2 (‘we chose the two parameters for the additive and multiplicative models which produced very similar implications’), read as a match of the strength of the two hypotheses; the witness is the validator's construction and is not in the paper", "location": "p. 398 (PDF p. 4), last paragraph, and the labels of Fig. 2 on the same page; p. 399 (PDF p. 5), first sentences; code: simulation/ksvsssr-setup.R, lines 52–61" }, "id": "P016-T18", "text": { "body": "p. 398 (PDF p. 4), last paragraph and Fig. 2; p. 399 (PDF p. 5), first sentences; released calibration code. The specified 256-unit witness attains the global minimum zero of the median criterion despite 127 unmatched treated units and unequal generated and adjusted mean shifts.", "headline": "The choice of the multiplicative parameter of Fig. 2 (‘we chose the two parameters for the additive and multiplicative models which produced very similar implications’), read as a match of the strength of the two hypotheses; the witness is the validator's construction and is not in the paper", "location": "p. 398 (PDF p. 4), last paragraph, and the labels of Fig. 2 on the same page; p. 399 (PDF p. 5), first sentences; code: simulation/ksvsssr-setup.R, lines 52–61" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T18" }, { "declarations": [ { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "Positive m is implicit in the target median framework; for this coordinate identity the excluded m=0 case has no coordinate i and contributes no missing assertion. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_discrepancy_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3018", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "This is the component discrepancy calculation in part (c); median zero and global minimality are separately exported by R117-02. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_root_discrepancies", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3026", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "The calibration-roots declaration already includes global nonnegativity for clause (c); this row is used separately to certify the minimum in T18.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_median_nonneg", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3036", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "The sorted median uses both middle order statistics with multiplicities; m>0 makes both accesses valid, and the infimum is an attained minimum. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_median_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3146", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "The outcome y_j≠0 is exactly the root construction requirement; there is no extra positivity condition on the multiplier, and the criterion is minimized over all real multipliers. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_calibration_roots", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3165", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "This is an acceptable generalization: the target balanced identities follow by specialization. It is not a narrowing and does not change either adjustment formula. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_mean_reductions", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3192", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "Fixing y across assignments expresses the no-effect schedule. UniformExpectation averages the full complete balanced design, and m>0 ensures nonempty groups and support. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_expected_reductions", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3409", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P016_Interference.lean:3527 states (1-1/a)*mu = t iff a = mu/(mu-t), with exactly T19(e)'s hypotheses mu != 0 and mu != t. Abstracting the population mean to an arbitrary real scalar imposes no restriction, and the statement matches the expected adjusted mean reductions rather than forward model shifts. At the additional a = 0 case, Lean's total division makes the left side mu = t, which is false, and the right side is also false because mu/(mu-t) is nonzero. Thus the convention adds no false matching parameter and loses no meaningful case.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_matched_parameter", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3438", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "This auxiliary arithmetic component should be read together with R117-01 and the matching equivalence R117-05; no claim about a constructed population is assumed as a hypothesis here. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t19_mismatch_numbers", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3460", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "conditional_on": [], "credit_reason": "The counterexample_proved status identifies the witness supporting the corrected statement, not a refutation of T19 itself. Its variance 131/128 is not an extra hypothesis on T19, whose mathematical formulation permits arbitrary y. Global minimum follows by combining its median zero with the separately credited nonnegativity theorem. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t18_calibration_witness", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3219", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:e8a2adad7852625d979cf7a0739c2926911185766fb39d6ed8ad3d2fa9c201b6", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T19", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T19" }, "html": { "body": "

p. 398 (PDF p. 4), last paragraph and Fig. 2; p. 399 (PDF p. 5), first sentences. The median criterion sees only the least treated discrepancy, has the stated roots, and does not match mean reductions, whose unique matching parameter is identified.

", "headline": "The comparison of the additive and multiplicative columns of Fig. 2, as repaired by Sol: the two hypotheses are not matched in expected mean shift; the calibration criterion and the matched parameter", "location": "p. 398 (PDF p. 4), last paragraph, and the labels of Fig. 2; p. 399 (PDF p. 5), first sentences" }, "id": "P016-T19", "text": { "body": "p. 398 (PDF p. 4), last paragraph and Fig. 2; p. 399 (PDF p. 5), first sentences. The median criterion sees only the least treated discrepancy, has the stated roots, and does not match mean reductions, whose unique matching parameter is identified.", "headline": "The comparison of the additive and multiplicative columns of Fig. 2, as repaired by Sol: the two hypotheses are not matched in expected mean shift; the calibration criterion and the matched parameter", "location": "p. 398 (PDF p. 4), last paragraph, and the labels of Fig. 2; p. 399 (PDF p. 5), first sentences" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T19" }, { "declarations": [ { "claim_revision_id": "catalog:61d8a869e66d36f4c85a2c76813cb19a59ae0b97e5c359d73b240f39d7559d44", "conditional_on": [], "credit_reason": "Bool with its real 0/1 indicator represents the printed binary assignment exactly.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t20_additive_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3478", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:61d8a869e66d36f4c85a2c76813cb19a59ae0b97e5c359d73b240f39d7559d44", "conditional_on": [], "credit_reason": "The nonnegative-matrix and positive-β domain is expressly part of T20(b), although it is more explicit than the original printed model domain.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t20_model_positive", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3485", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:61d8a869e66d36f4c85a2c76813cb19a59ae0b97e5c359d73b240f39d7559d44", "conditional_on": [], "credit_reason": "The original description’s restored domain assumptions coincide with T20(b), and ensure the inverse map has ordinary nonzero denominators even though the displayed zero-assignment algebra alone needs less.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t20_zero_assignment", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3508", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:61d8a869e66d36f4c85a2c76813cb19a59ae0b97e5c359d73b240f39d7559d44", "conditional_on": [], "credit_reason": "The target formal reading already assumes S≥0 and β>0; arbitrary real τ and lack of symmetry/binary/diagonal restrictions therefore agree with the target rather than narrow it. Positivity excludes total-division artifacts.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t20_model_composition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3522", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:61d8a869e66d36f4c85a2c76813cb19a59ae0b97e5c359d73b240f39d7559d44", "conditional_on": [], "credit_reason": "hmodel is the target’s explicit forward-model hypothesis, not an assumption of the recovered identity. The separate multiplier positivity result justifies division, and the public description accurately says generated outcomes.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t20_model_uniformity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3535", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:61d8a869e66d36f4c85a2c76813cb19a59ae0b97e5c359d73b240f39d7559d44", "conditional_on": [], "credit_reason": "This is the additive-vector consequence reused for T20(a); the residual-shift identity additionally spells out adjustment under a possibly false hypothesized parameter.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t8_additive_mapping", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L398", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:61d8a869e66d36f4c85a2c76813cb19a59ae0b97e5c359d73b240f39d7559d44", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T20", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T20" }, "html": { "body": "

p. 396 (PDF p. 2), Section 1, steps (1)–(2); equation (3), p. 399 (PDF p. 5). The additive identity and all four network-mapping clauses recover assignment-independent uniformity outcomes on the stated nonnegative-network and positive-multiplier domain.

", "headline": "Section 1, steps (1)–(2) of the testing procedure: the mapping of observed outcomes to the uniformity trial implied by a sharp hypothesis (yi,Zi=0=Yi−Ziτy_{i,Z_i=0} = Y_i - Z_i\\tauyi,Zi​=0​=Yi​−Zi​τ), with the model of equation (3)", "location": "p. 396 (PDF p. 2), Section 1, second full paragraph, steps (1)–(2); equation (3), p. 399 (PDF p. 5)" }, "id": "P016-T20", "text": { "body": "p. 396 (PDF p. 2), Section 1, steps (1)–(2); equation (3), p. 399 (PDF p. 5). The additive identity and all four network-mapping clauses recover assignment-independent uniformity outcomes on the stated nonnegative-network and positive-multiplier domain.", "headline": "Section 1, steps (1)–(2) of the testing procedure: the mapping of observed outcomes to the uniformity trial implied by a sharp hypothesis ($y_{i,Z_i=0} = Y_i - Z_i\\tau$), with the model of equation (3)", "location": "p. 396 (PDF p. 2), Section 1, second full paragraph, steps (1)–(2); equation (3), p. 399 (PDF p. 5)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T20" }, { "declarations": [ { "claim_revision_id": "catalog:58e8a39620de197add22aff816a6c9978880756b610b1b2d23d3701d2cd32e74", "conditional_on": [], "credit_reason": "Every model denominator is positive for the stated binary network and true parameters. The weak and strict bounds are both exported, and neither equality of size to .05 nor empirical 4%/2% frequencies is asserted. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t21_exact_level", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3548", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:58e8a39620de197add22aff816a6c9978880756b610b1b2d23d3701d2cd32e74", "conditional_on": [], "credit_reason": "The raw tails count ties inclusively, divide by B with no plus-one correction, and use the printed strict cutoff. B is positive in both exported bounds; no asymptotic or empirical-rate assertion is substituted. No change from the saved claim-stage verdict.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P016.t21_sampled_level", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P016/P016_Interference.lean#L3582", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:58e8a39620de197add22aff816a6c9978880756b610b1b2d23d3701d2cd32e74", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P016-T21", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P016-T21" }, "html": { "body": "

p. 400 (PDF p. 6), Section 2.2; Fig. 3 and caption, p. 401 (PDF p. 7). At the stated true network parameters, adjustment recovers uniformity outcomes, all three exact tails control rejection at .05, and fresh iid references give the bounds 50/1001 and 500/10001 for the printed strict rule.

", "headline": "Section 2.2: ‘All of these test statistics are valid—they reject the true null of τ=.5,β=2\\tau = .5, \\beta = 2τ=.5,β=2 no more than 5% of the time at α=.05\\alpha = .05α=.05’, repeated as ‘all of the tests are valid in the sense of rejecting the truth no more than 5% of the time when α=.05\\alpha = .05α=.05’", "location": "p. 400 (PDF p. 6), Section 2.2, first and second paragraphs; Fig. 3 and its caption, p. 401 (PDF p. 7)" }, "id": "P016-T21", "text": { "body": "p. 400 (PDF p. 6), Section 2.2; Fig. 3 and caption, p. 401 (PDF p. 7). At the stated true network parameters, adjustment recovers uniformity outcomes, all three exact tails control rejection at .05, and fresh iid references give the bounds 50/1001 and 500/10001 for the printed strict rule.", "headline": "Section 2.2: ‘All of these test statistics are valid—they reject the true null of $\\tau = .5, \\beta = 2$ no more than 5% of the time at $\\alpha = .05$’, repeated as ‘all of the tests are valid in the sense of rejecting the truth no more than 5% of the time when $\\alpha = .05$’", "location": "p. 400 (PDF p. 6), Section 2.2, first and second paragraphs; Fig. 3 and its caption, p. 401 (PDF p. 7)" }, "url": "https://pmaronow.github.io/llm-audit/works/P016/#P016-T21" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "partial", "targets": [ { "agreement": "within_mc_error", "location": "p. 398 (PDF p. 4)", "method": "authors' code with changes; all eight printed powers compared at the displayed hypotheses, with 1000 experiments and 1000 reference draws per test", "status": "reproduced", "target": "Figure 2" }, { "agreement": "agrees_in_part", "location": "pp. 400-401 (PDF pp. 6-7)", "method": "Authors' code with changes: the approximate null rejection rates agree within Monte Carlo error. On a reduced 15-point power grid, SSR + Design has large advantages at some alternatives, but estimated KS power is 0.9725 versus 0.935 for SSR + Design at $\\beta=4,\\tau=1.25$. The full power surfaces were not reproduced.", "status": "reproduced_reduced", "target": "Figure 3 and in-text null rejection rates" }, { "agreement": "agrees", "location": "pp. 401-402 (PDF pp. 7-8)", "method": "Only the outer rectangle was compared: inspection of Figure 4 agrees with the printed bounds to rounding, and SSR bounds recalculated using a network reconstructed from the deposited table lie inside it. The original inputs and p-value surfaces were not recovered.", "status": "recalculated", "target": "Figure 4 and in-text plausible-region bounds" } ] }, "summary": "For randomization tests of sharp hypotheses about interference, the note proposes a sum of squared residuals from a regression that includes the fixed network, in place of the Kolmogorov--Smirnov statistic of Bowers, Fredrickson and Panagopoulos (2013). Simulations and a re-analysis of a legislative experiment illustrate gains in power and a bounded region of plausible hypotheses. 11 entries are classified as errors (2 Moderate, 9 Minor). The error that matters most is \\fid{P016-E006}: the application model as printed is not the model computed for Fig.~4. No reported number is shown to be wrong.", "what_was_checked": "I checked the following points.\n\\begin{itemize}\n\\item The adjustment $y_{i,Z_i=0}=Y_i-Z_i\\tau$ on p.~396 follows from the additive model, and the randomization routine of the replication code applies it in the same way to the observed and the sampled statistics.\n\\item Equation (3) agrees with the coded model in both directions, and the verbal description fits the truth $\\beta=2$, $\\tau=.5$.\n\\item The network described on p.~399 is reproduced from the seed in the replication files with the sampling convention of R before version 3.6: 256 nodes, 512 edges, mean degree 4, degrees from 0 to 10, and five isolated nodes. Treatment is assigned to 128 of the 256 nodes by complete randomization.\n\\item I traced the six statistics of Fig.~3 in the code. Their definitions are clarified in \\fid{P016-E002}; the coded regressions include an intercept.\n\\item Fig.~1: both outcomes are reproduced from the seed, with the printed bandwidths $0.2946$ and $0.2886$.\n\\item Fig.~2: the eight power values printed in the figure are reproduced within Monte Carlo error by the replication code with its own $p$ value. A recalculation gives $0.700$ and $0.542$, $0.865$ and $0.729$, $0.701$ and $1.000$, $0.852$ and $1.000$ for SSR and KS, against the printed $0.70$ and $0.54$, $0.86$ and $0.74$, $0.70$ and $1$, $0.85$ and $1$.\n\\item Fig.~3: on the reduced 15-point grid, SSR + Design had large advantages at $(\\beta_0,\\tau_0)=(4,.8)$ and $(4,1)$, but many alternatives gave ties near power $1$, and KS had more power at $(4,1.25)$. SSR + Design also had the highest estimated power at six further alternatives. At the true parameters the recomputed rejection rates, from 1000 experiments, were between $0.045$ and $0.059$ for the SSR statistics and between $0.022$ and $0.029$ for the KS statistics. These are compatible with the published ``roughly 4\\%'' and ``roughly 2\\%'' within Monte Carlo error.\n\\item The application data have 70 legislators in 35 matched pairs, 35 treated, and 67 observed outcomes. The published Fig.~4 agrees with the rectangle on p.~402 up to rounding, and a recalculation using the network reconstructed from the deposited table places the SSR region inside the rectangle.\n\\item The discussion limits its statements about power to the examples, and it labels the weighted SSR statistic as a conjecture.\n\\end{itemize}\nI did not recompute the full power surfaces of Fig.~3: my recalculations cover a reduced grid of 15 points and six further alternatives, with fewer experiments than the text describes. I did not recompute Fig.~4 from its original inputs; I compared SSR bounds using a network reconstructed from the deposited table; the recalculated SSR region lies inside the printed rectangle. Of the cited works, only Coppock (2014), its corrigendum and the catalogue record of Hollander and Wolfe (1999) were consulted; Rosenbaum (2002), Bowers, Fredrickson and Panagopoulos (2013) and Butler and Nickerson (2011) were not read. Parts of the bundled RItools package that the simulations do not call, and some scripts of the other study in the application folder, were not examined. The conjecture of Section 4 offers no result to check.\n\nThe following are missing from the deposited replication files: the saved simulation output behind Figs.~2 to 4 and the run logs; the data files that the script for Fig.~4 loads; and the W-NOMINATE scores, which are all missing in the application table.\n\nThese checks do not establish that the rest of the paper is correct.", "work": { "arxiv_id": null, "audit_complete": true, "authors": [ "Jake Bowers", "Mark M. Fredrickson", "P. M. Aronow" ], "category": "Articles - Theory and Methodology", "citation": "Jake Bowers, Mark M. Fredrickson, and P. M. Aronow (2016). Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units. Political Analysis 24(3):395–403.", "doi": "10.1093/pan/mpw018", "edition": { "pages_in_audited_file": 9, "sha256": "ac0da6bc6675eae54c4e3c3f58f048fee49f170552a6bd65fdc65cc629bc8717", "source_url": "https://doi.org/10.1093/pan/mpw018", "venue": "Political Analysis", "year": "2016" }, "id": "P016", "later_materials": null, "title": "Research Note: A More Powerful Test Statistic for Reasoning about Interference between Units", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P016/", "venue": "Political Analysis", "year": "2016" } }