{ "claims": [ { "claim": "With a known interference network and a correctly specified exposure mapping, exposure probabilities follow from the design, and exposure-specific average potential outcomes and their contrasts are estimated without bias by Horvitz-Thompson weighting (Aronow and Samii 2017); the interference R package carries this out in a 10-unit toy example.", "claim_id": "P045-H1", "related_corrections": [ "P045-E004", "P045-E007", "P045-E008", "P045-A001", "P045-N001" ], "tied_to": "Sections 16.4.1-16.4.2; exposure-mapping and HT displays on pp. 294 and 297; Figure 16.3; toy example code and output on pp. 295-298" }, { "claim": "In a 400-unit small-world simulation the Hajek estimator is more efficient than Horvitz-Thompson at little cost in bias (smaller RMSE), and the chapter states that the variance estimator is consistent because the mean standard error approaches the empirical standard deviation.", "claim_id": "P045-H2", "related_corrections": [ "P045-N002", "P045-E005" ], "tied_to": "Section 16.4.2, p. 299; Table 16.1; Hajek display" }, { "claim": "If the assumed exposure mapping is coarser than the true interference, estimators of the all-versus-none effect $\\tau(\\mathbf{1},\\mathbf{0})$ are biased (downward under positive spillover, upward under negative spillover); if it is as fine or finer, they are centred on the truth but more variable.", "claim_id": "P045-H3", "related_corrections": [ "P045-E002", "P045-E005", "P045-E015", "P045-E016" ], "tied_to": "Section 16.4.3, pp. 299–303; Table 16.2; Figure 16.4; the $\\widehat{\\mu}_{HT,\\mathrm{None}}(\\mathbf{1})$ display on p. 301; Proposition 8.1 of Aronow and Samii (2017)" }, { "claim": "When the measured network is missing ties, the estimators are biased and the bias grows with the proportion of missing ties; the distributions are centred at 0% missing and shift as the proportion rises.", "claim_id": "P045-H4", "related_corrections": [ "P045-E012" ], "tied_to": "Section 16.4.4, p. 303; Figure 16.5" }, { "claim": "For the full-neighbourhood contrast, graph cluster randomization (3-net clustering) gives substantially lower estimator variability (lower SD and RMSE) than unit-level Bernoulli$(1/2)$ randomization, at no cost in bias.", "claim_id": "P045-H5", "related_corrections": [ "P045-E011", "P045-E010", "P045-E013", "P045-E006" ], "tied_to": "Section 16.4.6, pp. 304-306; Table 16.3; full-neighbourhood exposure mapping" }, { "claim": "Under partial interference and a two-stage randomized saturation design, the chapter states that the Hudgens–Halloran arm means and direct, indirect, total and overall effect estimators are unbiased conditional on their denominators being nonzero, without knowledge of the within-group network. It states that the variance estimators under stratified interference are unbiased with additive effects and conservative otherwise; the package function estimates_hierarchical computes them in an 18-unit example.", "claim_id": "P045-H6", "related_corrections": [ "P045-E001", "P045-E009", "P045-N008" ], "tied_to": "Section 16.5.1, pp. 308-311; marginal-effect and estimator displays on pp. 309-310; toy output on p. 311" }, { "claim": "If interference also runs across groups within tracts while the analyst assumes partial interference at the group level, the direct, indirect, total and overall estimators are biased; they are unbiased when partial interference holds.", "claim_id": "P045-H7", "related_corrections": [ "P045-E003", "P045-E014" ], "tied_to": "Section 16.5.2, pp. 311–312; Figure 16.6; the $\\widehat{y}(z;\\psi)$ mixture display on p. 312" }, { "claim": "Contagion is one (outcome-mediated) mechanism of interference; the indirect effect decomposes into a contagion effect and other mechanisms, $\\theta=\\gamma(1)+\\eta(0)=\\gamma(0)+\\eta(1)$, and identifying the decomposition needs mediation assumptions (sequential ignorability, a cross-world assumption) beyond randomized treatment assignment.", "claim_id": "P045-H8", "related_corrections": [], "tied_to": "Figure 16.1; Section 16.6, pp. 314–316; displays for $\\theta$, $\\gamma(z)$ and $\\eta(z)$ on p. 315" }, { "claim": "The reviewed empirical studies show that spillovers matter (voting, petitioning, student behaviour, prejudice, economic outcomes), that inter-group comparisons can conceal within-group spillovers and that effects depend on the assignment regime, and they illustrate designs that implement the two frameworks.", "claim_id": "P045-H9", "related_corrections": [ "P045-N009", "P045-N010" ], "tied_to": "Figure 16.2; Sections 16.2, 16.4.7, 16.5.3; conclusion p. 316" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Six lines of text followed by a displayed formula. A red box marks the word 'unbiased' in the sentence 'Given Bernoulli assignment with probability psi, an unbiased estimator for one over n_g times the sum of y_ig(z; psi), with z = {0, 1}, is'. The display below defines y-hat_g(z; psi) as the sum of Y_ig times the indicator of Z_ig = z, divided by the sum of the indicators.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), left column: the sentence that calls the arm mean $\\widehat{y}_g(z;\\psi)$ unbiased under Bernoulli assignment, with “unbiased” marked, and the display that defines the arm mean.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E001.png" }, { "alt": "Nine lines of text. Red boxes mark the word 'unbiased' in 'Therefore, an unbiased estimator for the average direct effect is tau-hat D(psi) = y-hat(1; psi) minus y-hat(0; psi)' and the word 'Unbiased' that begins the next sentence, 'Unbiased estimators for the other causal estimands of interest are defined analogously'.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), left column, below the second display: the sentences that call the estimators of the direct, indirect, total and overall effects unbiased, with “unbiased” and “Unbiased” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E001-b.png" }, { "alt": "A chart with five rows and a horizontal axis of expected value from minus 1 to 1. Blue vertical ticks mark the targets: 0 for the direct effect at psi = 1/2 and for the direct effect at phi = 1/4, and 3/4 for the indirect, total and overall effects. Orange dots mark the expectation of each estimator as printed: minus 1 for each direct effect, 224/255 for the indirect effect, 37/255 for the total effect, and 3/4 for the overall effect, where the dot lies on the tick and is labelled 'no bias'. In the first four rows a grey line joins the dot to its tick. The legend says that the tick is also the expectation of the Horvitz-Thompson replacement.", "caption": "Targets and expectations in the counterexample of this finding: two groups of four units, outcome equal to the number of treated group members other than the unit, one group at $\\psi=1/2$ and one at $\\phi=1/4$, Bernoulli assignment within groups. The expectations of the printed estimators are conditional on each estimator being defined; all values are exact fractions from enumeration of 512 assignments. The Horvitz–Thompson replacements have unconditional expectations equal to the targets; the overall estimator is unchanged.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E001-bias.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T1", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T1" }, { "id": "P045-T2", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T2" }, { "id": "P045-T25", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T25" } ], "headline": "Under Bernoulli assignment within groups, the arm means of Section 16.5.1 are not in general unbiased for the marginal potential outcomes of p. 309, and neither are the estimators of the direct, indirect and total effects built from them; the estimator of the overall effect is unbiased under uniform assignment of fixed positive numbers of groups to saturations.", "id": "P045-E001", "location": "Section 16.5.1, p. 310 (PDF p. 22), left column, the paragraph beginning “Let the observed outcome be” and its two displays", "pdf_page": 22, "printed_page": "310", "reaches": [ { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" }, { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P045-T1", "P045-T2", "P045-T25" ], "text": { "as_printed": "Let the observed outcome be $Y_{ig}=y_{ig}(\\mathbf{Z}_{ig})$ under the group treatment assignment $\\mathbf{Z}_{ig}$ and suppose treatment saturation is $\\psi$. Given Bernoulli assignment with probability $\\psi$, an \\hl{unbiased} estimator for $\\frac{1}{n_g}\\sum_{i=1}^{n_g}y_{ig}(z;\\psi)$, with $z=\\{0,1\\}$, is\n\\[\n\\widehat{y}_g(z;\\psi)=\\frac{\\sum_{i=1}^{n_g}Y_{ig}\\mathbf{I}(Z_{ig}=z)}{\\sum_{i=1}^{n_g}\\mathbf{I}(Z_{ig}=z)},\n\\]\nand for the population average potential outcome, it is\n\\[\n\\widehat{y}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi)\\mathbf{I}(S_g=\\psi)}{\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi)},\n\\]\nwhere $S_g$ is the saturation level of group $g$ (conditional on the denominators of these estimators being nonzero). Therefore, an \\hl{unbiased} estimator for the average direct effect is $\\widehat{\\tau}^{D}(\\psi)=\\widehat{y}(1;\\psi)-\\widehat{y}(0;\\psi)$. \\hl{Unbiased} estimators for the other causal estimands of interest are defined analogously: $\\widehat{\\tau}^{I}(\\psi,\\phi)=\\widehat{y}(0;\\psi)-\\widehat{y}(0;\\phi)$, $\\widehat{\\tau}^{To}(\\psi,\\phi)=\\widehat{y}(1;\\psi)-\\widehat{y}(0;\\phi)$, and $\\widehat{\\tau}^{O}(\\psi,\\phi)=\\widehat{y}(\\psi)-\\widehat{y}(\\phi)$, where $\\widehat{y}(\\psi)$ is the average across groups assigned to saturation $\\psi$ of the average observed outcomes of units in the group under treatment assignment $\\mathbf{Z}_{ig}$.", "corrected": "Let the observed outcome be $Y_{ig}=y_{ig}(\\mathbf{Z}_{ig})$ under the group treatment assignment $\\mathbf{Z}_{ig}$ and suppose treatment saturation is $\\psi$, with $0<\\psi<1$ and, for the indirect and total contrasts, $\\hl{0\\le\\phi<1}$. Given Bernoulli assignment with probability $\\psi$, the arm mean\n\\[\n\\widehat{y}_g(z;\\psi)=\\frac{\\sum_{i=1}^{n_g}Y_{ig}\\mathbf{I}(Z_{ig}=z)}{\\sum_{i=1}^{n_g}\\mathbf{I}(Z_{ig}=z)}\n\\]\nis \\hl{not in general unbiased} for $\\frac{1}{n_g}\\sum_{i=1}^{n_g}y_{ig}(z;\\psi)$, with $z\\in\\{0,1\\}$, \\hl{even conditional on its denominator being nonzero}. An unbiased estimator is\n\\[\n\\widehat{y}^{HT}_g(z;\\psi)=\\frac{\\sum_{i=1}^{n_g}Y_{ig}\\mathbf{I}(Z_{ig}=z)}{\\hl{n_g\\,\\psi^{z}(1-\\psi)^{1-z}}},\n\\]\ncomputed for every assignment, with the value $0$ when no unit of the group has treatment $z$. Suppose that a fixed positive number of the $G$ groups is assigned to each saturation, all such assignments being equally likely. Then\n\\[\n\\widehat{y}^{HT}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}^{HT}_g(z;\\psi)\\mathbf{I}(S_g=\\psi)}{\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi)},\n\\]\nwhere $S_g$ is the saturation level of group $g$, is unbiased for the population average potential outcome. The estimators $\\widehat{\\tau}^{D}(\\psi)=\\widehat{y}^{HT}(1;\\psi)-\\widehat{y}^{HT}(0;\\psi)$, $\\widehat{\\tau}^{I}(\\psi,\\phi)=\\widehat{y}^{HT}(0;\\psi)-\\widehat{y}^{HT}(0;\\phi)$ and $\\widehat{\\tau}^{To}(\\psi,\\phi)=\\widehat{y}^{HT}(1;\\psi)-\\widehat{y}^{HT}(0;\\phi)$ are then unbiased for the average direct, indirect and total effects. The estimator $\\widehat{\\tau}^{O}(\\psi,\\phi)=\\widehat{y}(\\psi)-\\widehat{y}(\\phi)$, where $\\widehat{y}(\\psi)$ is the average across groups assigned to saturation $\\psi$ of the average observed outcomes of units in the group, is unbiased for the average overall effect.\n\nAlternatively, suppose that treatment within each group is by \\hl{complete random assignment}, with a fixed number of treated units and a fixed number of control units, both at least one, and that the marginal potential outcomes $y_{ig}(z;\\psi)$ are defined by averaging over the assignments of that design. Then the arm means $\\widehat{y}_g(z;\\psi)$ and $\\widehat{y}(z;\\psi)$ as printed, and the estimators $\\widehat{\\tau}^{D}$, $\\widehat{\\tau}^{I}$ and $\\widehat{\\tau}^{To}$ built from them, are unbiased for the corresponding quantities under the same assignment of groups to saturations.", "evidence": "An exact counterexample, and derivations for the two corrected forms.\n\n\\emph{The counterexample.} I take two groups of four units and set $y_{ig}(\\mathbf{z})=\\sum_{j\\neq i}z_j$, the number of treated group members other than $i$. Own treatment has no effect, and partial and stratified interference hold. One group, chosen with probability $1/2$, has saturation $\\psi=1/2$ and the other has $\\phi=1/4$. Each unit is then treated independently with the probability of its group. For a unit in a group with treatment probability $p$, each of the three other members is treated with probability $p$ whatever $z$ is, so that $y_{ig}(0;p)=y_{ig}(1;p)=3p$. The group averages of the marginal potential outcomes are $3/2$ at $\\psi$ and $3/4$ at $\\phi$, both direct effects are $0$, and the indirect, total and overall effects are $3/4$.\n\nLet $K$ be the number of treated units in a group. Every treated unit has outcome $K-1$ and every control unit has outcome $K$. The treated arm mean is $K-1$ when $K\\ge 1$, the control arm mean is $K$ when $K\\le 3$, and their difference is $-1$ on every assignment for which both are defined.\n\n\\begin{audittable}{crrcc}{The number $K$ of treated units in a group of four, its probability at each saturation, and the two arm means.}\n$K$ & Probability at $\\psi=1/2$ & Probability at $\\phi=1/4$ & Treated arm mean & Control arm mean \\\\\n0 & 1/16 & 81/256 & undefined & 0 \\\\\n1 & 4/16 & 108/256 & 0 & 1 \\\\\n2 & 6/16 & 54/256 & 1 & 2 \\\\\n3 & 4/16 & 12/256 & 2 & 3 \\\\\n4 & 1/16 & 1/256 & 3 & undefined \\\\\n\\end{audittable}\n\nBy the table, I have\n\\[\n\\begin{aligned}\nE\\bigl[\\widehat{y}_g(1;\\psi)\\mid K\\ge 1\\bigr]&=\\frac{4\\cdot 0+6\\cdot 1+4\\cdot 2+1\\cdot 3}{15}=\\frac{17}{15},\\\\\nE\\bigl[\\widehat{y}_g(0;\\psi)\\mid K\\le 3\\bigr]&=\\frac{1\\cdot 0+4\\cdot 1+6\\cdot 2+4\\cdot 3}{15}=\\frac{28}{15},\n\\end{aligned}\n\\]\nwhile the target of both is $3/2$. At $\\phi$ the two conditional expectations are $81/175$ and $84/85$, and the target is $3/4$. Enumerating the 512 assignments of the two stages in exact fractions, I find the expectation of each estimator over the assignments on which it is defined. It is $-1$ for $\\widehat{\\tau}^{D}(\\psi)$ and for $\\widehat{\\tau}^{D}(\\phi)$, whose targets are $0$. It is $224/255$ for $\\widehat{\\tau}^{I}$ and $37/255$ for $\\widehat{\\tau}^{To}$, whose targets are $3/4$. It is $3/4$ for $\\widehat{\\tau}^{O}$, which equals its target.\n\n\\emph{The Horvitz--Thompson mean.} Since $y_{ig}(z;\\psi)$ is the expected outcome of unit $i$ given $Z_{ig}=z$, I have\n\\[\nE\\bigl[\\mathbf{I}(Z_{ig}=z)\\,Y_{ig}\\bigr]=\\Pr(Z_{ig}=z)\\,E\\bigl[Y_{ig}\\mid Z_{ig}=z\\bigr]=\\psi^{z}(1-\\psi)^{1-z}\\,y_{ig}(z;\\psi).\n\\]\nSumming over $i$ and dividing by $n_g\\,\\psi^{z}(1-\\psi)^{1-z}$, I obtain that the Horvitz--Thompson mean in the correction has expectation $(1/n_g)\\sum_{i}y_{ig}(z;\\psi)$. Note that the expectation is over all assignments, including those with no unit in arm $z$, where the sum is $0$. In the counterexample, when the arm means are replaced by Horvitz--Thompson means, the expectations found by enumeration equal the targets for all five effects.\n\n\\emph{Complete random assignment.} Suppose that exactly $k_z\\ge 1$ units of the group receive treatment $z$, all such assignments being equally likely, and let $y^{\\mathrm{cr}}_{ig}(z)$ be the expected outcome of unit $i$ given $Z_{ig}=z$ under this design. Since each unit has treatment $z$ with probability $k_z/n_g$ and the denominator of the arm mean is the constant $k_z$, I find\n\\[\nE\\bigl[\\widehat{y}_g(z;\\psi)\\bigr]=\\frac{1}{k_z}\\sum_{i=1}^{n_g}\\frac{k_z}{n_g}\\,y^{\\mathrm{cr}}_{ig}(z)=\\frac{1}{n_g}\\sum_{i=1}^{n_g}y^{\\mathrm{cr}}_{ig}(z).\n\\]\nThese targets differ from the Bernoulli ones. In the counterexample with two of the four units treated, I have $y^{\\mathrm{cr}}_{ig}(1)=1$ and $y^{\\mathrm{cr}}_{ig}(0)=2$. The direct effect defined for this design is $-1$, while the direct effect under Bernoulli assignment is $0$.\n\n\\emph{Groups.} Suppose that a fixed number $C\\ge 1$ of the $G$ groups receives saturation $\\psi$, all choices being equally likely. Each group then has saturation $\\psi$ with probability $C/G$. If a quantity computed in group $g$ has expectation $\\mu_g$ when the group has saturation $\\psi$, its mean over the groups with that saturation has expectation $(1/G)\\sum_{g}\\mu_g$. Applied to the Horvitz--Thompson means and to the arm means under complete random assignment, this implies the population statements in the correction. Applied to the mean outcome of all units in a group, it implies that $\\widehat{\\tau}^{O}$ is unbiased.", "problem": "On p.~309 the chapter defines the marginal potential outcome $y_{ig}(z;\\psi)$ by fixing the treatment of unit $i$ at $z$ and averaging its potential outcomes over the treatments of the other group members, each of whom is treated independently with probability $\\psi$. The same page separates this from complete random assignment, which fixes the number of treated units. Under Bernoulli assignment the number of treated units in a group is random, and the arm mean $\\widehat{y}_g(z;\\psi)$, which is the mean of the observed outcomes of the units of group $g$ with treatment $z$, divides by the realized size of the arm. When the outcome of a unit depends on how many other group members are treated, which is the case that Section 16.5 treats, the arm mean then averages the outcomes over a distribution of the other members' treatments that differs from the one in the definition of $y_{ig}(z;\\psi)$. Conditioning on a nonzero denominator does not remove the difference. The step to ``unbiased'' fails for $\\widehat{y}_g(z;\\psi)$ and $\\widehat{y}(z;\\psi)$, and with it the claims for $\\widehat{\\tau}^{D}$, $\\widehat{\\tau}^{I}$ and $\\widehat{\\tau}^{To}$. The estimator $\\widehat{\\tau}^{O}$ uses the mean outcome of all units in a group, has no denominator that depends on the size of an arm, and is unbiased under uniform assignment of fixed positive numbers of groups to saturations.", "what_survives": "The definitions of the marginal causal effects on p.~309 and the estimator of the overall effect are unaffected. The toy example on pp.~310--311 uses complete random assignment in both stages, and Figure 16.6 uses the same two-stage design. For that design the arm means are unbiased for the marginal potential outcomes defined by it. No reported number is known to depend on the claim. What changes is that, under Bernoulli assignment within groups, the printed estimators of the direct, indirect and total effects cannot be called unbiased. No variance estimator for the Horvitz--Thompson means was derived." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E001", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Two lines of text, a displayed formula in two lines, and two more lines of text. The formula sets y-hat(z; psi) equal to a fraction plus a second fraction. Red boxes mark the two denominators: the sum over g of the indicator of S_g = psi psi, and the sum over g of the indicator of S_g = psi phi.", "caption": "Section 16.5.2, p. 312 (PDF p. 24), left column: the display for $\\widehat{y}(z;\\psi)$, with the two separate denominators marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E003.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T14", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T14" }, { "id": "P045-T15", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T15" } ], "headline": "The pooled estimator $\\widehat{y}(z;\\psi)$ divides the sum of the group means by the number of all groups with saturation $\\psi$; the display prints the sum of two subgroup means, each with its own denominator.", "id": "P045-E003", "location": "Section 16.5.2, p. 312 (PDF p. 24), left column, the display after “the estimator for the population average potential outcome under saturation $\\psi$ is”", "pdf_page": 24, "printed_page": "312", "reaches": [ { "effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" }, { "effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T14", "P045-T15" ], "text": { "as_printed": "In other words, the estimator for the population average potential outcome under saturation $\\psi$ is\n\\[\n\\widehat{y}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\psi)\\mathbf{I}(S_g=\\psi\\psi)}{\\hl{\\textstyle\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\psi)}}+\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\phi)\\mathbf{I}(S_g=\\psi\\phi)}{\\hl{\\textstyle\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\phi)}}.\n\\]\nThe average outcome across groups includes a mixture of the average of observed outcomes for units with treatment $z$ in a group with realized saturation $\\psi$ in a tract in which the realized saturation for the other group is $\\psi$, and the average of observed outcomes for units with treatment $z$ in a group with realized saturation $\\psi$ in a tract in which the realized saturation for the other group is $\\phi$.", "corrected": "In other words, the estimator for the population average potential outcome under saturation $\\psi$ is\n\\[\n\\widehat{y}(z;\\psi)=\\frac{\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\psi)\\mathbf{I}(S_g=\\psi\\psi)+\\sum_{g=1}^{G}\\widehat{y}_g(z;\\psi\\phi)\\mathbf{I}(S_g=\\psi\\phi)}{\\hl{\\textstyle\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\psi)+\\sum_{g=1}^{G}\\mathbf{I}(S_g=\\psi\\phi)}},\n\\]\nwhich is defined whenever at least one group has saturation $\\psi$ and \\hl{each included group has at least one unit with treatment $z$}. Here $S_g=\\psi\\psi$ means that group $g$ has saturation $\\psi$ and the other group in its tract has saturation $\\psi$, and $S_g=\\psi\\phi$ means that group $g$ has saturation $\\psi$ and the other group has saturation $\\phi$. When both kinds of group occur, the estimator is the average of the two subgroup means weighted by the shares of the two kinds of group.", "evidence": "A derivation and an exact example. For positive $a$ and $b$, I have\n\\[\n\\frac{A}{a}+\\frac{B}{b}-\\frac{A+B}{a+b}=\\frac{bA}{a(a+b)}+\\frac{aB}{b(a+b)},\n\\]\nwhich is positive when $A$ and $B$ are positive. For the example, I take the design of Section 16.5.2, with six groups of 75 units in three tracts, $\\psi=2/3$ and $\\phi=1/3$. I let every baseline outcome be $1$ and use the outcome multipliers of the package function \\texttt{make\\_dilated\\_out\\_hh\\_miss}: in a group with saturation $\\psi$, a treated unit has outcome $9/4$ and an untreated unit $11/8$ when the other group in the tract has saturation $\\psi$, and $2$ and $5/4$ when it has saturation $\\phi$. Let groups 1, 2 and 3 have saturation $\\psi$, with tracts $(1,2)$, $(3,4)$ and $(5,6)$, so that $a=2$ and $b=1$. For treated units the estimator is $(9/2+2)/3=13/6$ and the display gives $9/4+2=17/4$. For untreated units the values are $4/3$ and $21/8$. Of the 20 equally likely choices of the three groups with saturation $\\psi$, 8 place one such group in each tract. Then $a=0$ and the first term of the display is $0/0$. On all 20 choices the corrected form equals the estimator of p.~310, and the package function \\texttt{estimates\\_hierarchical} computes that estimator.", "problem": "On p.~310 the estimator $\\widehat{y}(z;\\psi)$ is the mean of the group means $\\widehat{y}_g(z;\\psi)$ over all groups with saturation $\\psi$. Section 16.5.2 splits these groups by the saturation of the other group in the tract. Among the groups with saturation $\\psi$, let $a$ be the number of groups such that the other group in the tract has saturation $\\psi$, and let $b$ be the number such that it has saturation $\\phi$. Let $A$ and $B$ be the sums of the group means over these two sets of groups. The estimator is $(A+B)/(a+b)$, which is the mixture of the subgroup means $A/a$ and $B/b$ with weights $a/(a+b)$ and $b/(a+b)$ that the next sentence describes. The display prints $A/a+B/b$, the sum without weights. When both subgroups occur and the outcomes are positive, this exceeds the estimator; when one subgroup is empty, it contains $0/0$, although the pooled estimator is still defined.", "what_survives": "The definition on p.~310, the sentence after the display and the argument of Section 16.5.2 are unaffected: pooling groups from tracts of different composition can bias the estimator of $\\tau^{D}(\\psi\\psi)$. The package computes the pooled estimator. The script for Figure 16.6 is not available, and it could not be confirmed that the figure was computed from the pooled estimator and not from the display." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E003", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the middle of page 298: two paragraphs set in two columns, a one-line sentence, three blocks of R code, and the output printed after two of them. Red boxes mark the words 'cannot compute' in the sentence about the estimate of tau HT of d11 and d00, and the entry NA under the column dir_ind1 in each of the two printed outputs, which otherwise read 41.61043 and 39.51643.", "caption": "Section 16.4.2, p. 298 (PDF p. 10). The marked terms are the words “cannot compute” and the entry NA for dir_ind1 in each of the two printed outputs.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E004.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T16", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T16" }, { "id": "P045-T17", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T17" } ], "headline": "An empty Horvitz–Thompson total is zero, and $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ in the example equals $-13.27099$; the text says that it cannot be computed, and the package returns NA.", "id": "P045-E004", "location": "Section 16.4.2, p. 298 (PDF p. 10): the paragraph before the yT_HT code and the two printed outputs of $\\widehat{\\tau_{HT}}$; the function estimates of the interference package", "pdf_page": 10, "printed_page": "298", "reaches": [ { "effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P013-E018", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" }, { "effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": "P044-E006", "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044" } ], "related_entries": [], "revisions": [], "same_defect_as": [ { "id": "P013-E018", "url": "https://pmaronow.github.io/llm-audit/works/P013/#P013-E018", "work_id": "P013" }, { "id": "P044-E006", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E006", "work_id": "P044" } ], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T16", "P045-T17" ], "text": { "as_printed": "In this case, we \\hl{cannot compute} $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ -- the interactive effect of direct and indirect exposure -- because in this small-scale example no unit received exposure $d_{11}$.\n\n\\begin{audittable}{rrr}{Output printed twice on p.~298, after \\texttt{tau\\_HT} and after the call to \\texttt{estimates}.}\n\\texttt{dir\\_ind1} & \\texttt{isol\\_dir} & \\texttt{ind1} \\\\\n\\hl{\\texttt{NA}} & 41.61043 & 39.51643 \\\\\n\\end{audittable}", "corrected": "In this case, \\hl{no unit received exposure $d_{11}$. Since every $\\widehat\\pi_i(d_{11})$ is positive, the total $\\widehat{y^T_{HT}}(d_{11})$ is zero, and} $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ -- the interactive effect of direct and indirect exposure -- \\hl{equals $-\\widehat{y^T_{HT}}(d_{00})/N$.}\n\n\\begin{audittable}{rrr}{Corrected output, in both places on p.~298.}\n\\texttt{dir\\_ind1} & \\texttt{isol\\_dir} & \\texttt{ind1} \\\\\n\\hl{$-13.27099$} & 41.61043 & 39.51643 \\\\\n\\end{audittable}\n\nIn general, let $d_k$ be an exposure such that $\\pi_i(d_k)>0$ for all units $i$. Then\n\\[\n\\widehat{y^T_{HT}}(d_k)=\\sum_{i=1}^N\\mathbf I(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)}\n\\]\nis defined for every assignment, and it equals $0$ when no unit has $D_i=d_k$. \\hl{If positive estimates $\\widehat\\pi_i(d_k)$ are used instead, replace $\\pi_i(d_k)$ by $\\widehat\\pi_i(d_k)$ in this formula; an empty total is still zero.} With the true probabilities, its expectation over all assignments is $\\sum_{i=1}^N y_i(d_k)$. The Hajek estimator and the sample mean of an exposure that no unit receives remain undefined. In the package, the function \\texttt{estimates} should keep the zero total and not replace it by \\texttt{NA}.", "evidence": "The evidence is a derivation from the definition, a recalculation of the example with the package, and an exact counterexample.\n\nSuppose that $\\pi_i(d)>0$ for every unit $i$ and that the observed outcomes are finite. If no unit has $D_i=d$, then every summand of $\\widehat{y^T_{HT}}(d)$ is $0\\cdot Y_i/\\pi_i(d)=0$, and the total is $0$. Since $Y_i=y_i(d)$ when $D_i=d$, I have\n\\[\n\\mathbb E\\bigl[\\widehat{y^T_{HT}}(d)\\bigr]=\\sum_{i=1}^N y_i(d)\\,\\frac{\\Pr(D_i=d)}{\\pi_i(d)}=\\sum_{i=1}^N y_i(d),\n\\]\nwhere the expectation is over all assignments, including those that place no unit in $d$. In the last equality, I used that $\\Pr(D_i=d)=\\pi_i(d)$.\n\nFor the example, I ran two versions of the package (commits \\texttt{a5b8dfb} and \\texttt{c3fc7b3} of its repository) with the chapter's seeds and the printed adjacency matrix, under the sampling algorithm that was the default before R 3.6.0. Both reproduce the printed assignment and the printed probabilities, and both return the totals $(\\texttt{NA},\\,548.8142,\\,527.8742,\\,132.7099)$ for $(d_{11},d_{10},d_{01},d_{00})$ and the printed contrasts. When I remove the line of \\texttt{estimates} that sets an empty total to \\texttt{NA}, the first total is $0$ and the first contrast is $(0-132.7099)/10=-13.27099$. The other two contrasts do not change, and the Hajek contrast for $d_{11}$ remains \\texttt{NA}. The same line occurs in the package's function for the full-neighbourhood exposures, which I did not test.\n\nThe counterexample uses the printed network of ten units, with two units assigned to treatment uniformly among the 45 pairs and potential outcomes\n\\[\n\\bigl(y_i(d_{11}),\\,y_i(d_{10}),\\,y_i(d_{01}),\\,y_i(d_{00})\\bigr)=\\bigl(2,\\,3/2,\\,5/4,\\,1\\bigr)\n\\]\nfor every unit. The true total for $d_{11}$ is $20$, and $\\tau(d_{11},d_{00})=1$. A unit with $g_i$ neighbours has $\\pi_i(d_{11})=g_i/45$, and every exposure probability of every unit is positive. The network has 20 edges. The 20 assignments that treat two adjacent units place those two units in $d_{11}$, and the other 25 assignments place no unit there. On the assignment that treats the adjacent units $a$ and $b$, the total is $90(1/g_a+1/g_b)$, and the sum of these totals over the 20 edges is $900$. With empty totals equal to zero, I obtain\n\\[\n\\mathbb E\\bigl[\\widehat{y^T_{HT}}(d_{11})\\bigr]=\\frac{900}{45}=20 ,\n\\]\nwhich is the true total. The mean over the 20 assignments with a unit in $d_{11}$ is $900/20=45$. For the contrast, both $d_{11}$ and $d_{00}$ are received by some unit on 19 assignments. The mean of $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ over those 19 assignments is $1227/380\\approx3.23$. The mean over all 45 assignments, with empty totals equal to zero, is $1$.", "problem": "The chapter defines the Horvitz--Thompson total on p.~297 as a sum over all units,\n\\[\n\\widehat{y^T_{HT}}(d_k)=\\sum_{i=1}^N\\mathbf I(D_i=d_k)\\frac{Y_i}{\\pi_i(d_k)},\n\\]\nunder the assumption that every unit has a positive probability of each exposure. When no unit receives $d_{11}$, every indicator in the sum is zero, and I call the total empty. An empty total equals $0$, and the contrast $\\widehat{\\tau_{HT}}(d_{11},d_{00})=-\\widehat{y^T_{HT}}(d_{00})/N$ is a finite number. The sentence on p.~298 says that the contrast cannot be computed, and the package returns \\texttt{NA}, because the function \\texttt{estimates} replaces an empty total by \\texttt{NA}. On such an assignment, the package does not compute the estimator of p.~297. The unbiasedness stated for that estimator refers to the mean over all assignments, with empty totals equal to zero. If the assignments on which the package returns \\texttt{NA} are dropped, the mean of the remaining estimates can differ from the estimand. An empty Hajek ratio or sample mean is $0/0$ and remains undefined.", "what_survives": "The definition of the estimator on p.~297, the estimand and the unbiasedness of the estimator with the true exposure probabilities are not affected. The contrasts $41.61043$ and $39.51643$ of the example stand, as does every Horvitz--Thompson output on an assignment in which each exposure involved is received by some unit. I could not determine whether the simulations with $N=400$ are affected, because the scripts and draws behind their tables are not available and I do not know how they treated empty totals." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E004", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of seven lines from the left column of page 303. The text reads 'Note that this example uses a relatively small set of units (N = 400), and the unbiased estimators (in this case, tau-hat H of d11, d00, tau-hat H of d111, d000) work with smaller subsets of the data than the coarser, biased estimators (here, tau-hat H of d1, d0). This is apparent if one looks at the standard deviations in Table 16.2.' A red box marks the word 'unbiased'.", "caption": "Section 16.4.3, p. 303 (PDF p. 15), left column. The marked word is “unbiased”.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E005.png" } ], "formal_release": null, "formal_statements": [], "headline": "On p. 303, “the unbiased estimators” are the Hajek estimators whose exposure mapping is adequately specified; a Hajek estimator need not be exactly unbiased in a finite sample.", "id": "P045-E005", "location": "Section 16.4.3, p. 303 (PDF p. 15), left column, the paragraph that begins “Note that this example uses a relatively small set of units”", "pdf_page": 15, "printed_page": "303", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "``Note that this example uses a relatively small set of units ($N=400$), and the \\hl{unbiased} estimators (in this case, $\\widehat{\\tau_H}(d_{11},d_{00})$, $\\widehat{\\tau_H}(d_{111},d_{000})$) work with smaller subsets of the data than the coarser, biased estimators (here, $\\widehat{\\tau_H}(d_1,d_0)$).''", "corrected": "``Note that this example uses a relatively small set of units ($N=400$), and the \\hl{estimators using adequately specified exposure mappings} (in this case, $\\widehat{\\tau_H}(d_{11},d_{00})$, $\\widehat{\\tau_H}(d_{111},d_{000})$) work with smaller subsets of the data than the coarser, biased estimators (here, $\\widehat{\\tau_H}(d_1,d_0)$).''", "intended_meaning": "The word separates the estimators whose exposure mapping allows for at least as much interference as the data-generating process from the coarser ones. The intended meaning is that the former carry no bias from the exposure mapping, and this is correct. The estimators named are Hajek estimators, which are ratios, and a ratio estimator need not be exactly unbiased in a finite sample. The chapter says so on p.~299, where the Hajek estimator ``improves efficiency with a small cost in terms of finite-sample bias''. Table 16.2 reports biases of 1.29 and 0.85 for $\\widehat{\\tau_H}(d_{11},d_{00})$ when interference is of first degree, and of 4.28 and 2.58 for $\\widehat{\\tau_H}(d_{111},d_{000})$ when it is of second degree. An exact example confirms the point. Let exactly one of two units be treated, with assignment probabilities $(1/4,3/4)$, treated outcomes $(1,3)$, control outcomes zero and no interference. The Hajek contrast has expectation $5/2$ and the Horvitz--Thompson contrast has expectation $2$, which is the value of the estimand. The formulas and the simulation results are unchanged." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E005", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of six lines near the foot of the right column of page 297, beginning 'Specifically, we generate a variable with random values from'. Red boxes mark the phrase 'an absolute standard normal distribution', which runs over two lines.", "caption": "Section 16.4.2, p. 297 (PDF p. 9), foot of the right column. The marked phrase describes the distribution of the baseline outcome.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E006.png" } ], "formal_release": null, "formal_statements": [], "headline": "The baseline outcome is a half-normal variable scaled by degree plus a normal term, as make_corr_out generates it; it is not an absolute standard normal variable.", "id": "P045-E006", "location": "Section 16.4.2, pp. 297–298 (PDF pp. 9–10), last paragraph of p. 297; see also p. 305 (PDF p. 17), right column", "pdf_page": 9, "printed_page": "297–298", "reaches": [], "related_entries": [ { "id": "P044-E002", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E002", "work_id": "P044" } ], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "Specifically, we generate a variable with random values from \\hl{an absolute standard normal distribution} that is correlated with the unit's first- and second-order degree -- the number of peers and peers of peers, respectively (for which we use the arguments \\texttt{adj\\_matrix} and \\texttt{make\\_corr\\_out} in the function below). This variable determines the potential outcome under the $d_{00}$ condition.", "corrected": "Specifically, we generate a variable \\hl{$(k_i+k_i^{(2)}+k_ik_i^{(2)})|X_i|+E_i$, where $k_i$ and $k_i^{(2)}$ are} the unit's first- and second-order degree -- the number of peers and peers of peers, respectively \\hl{-- and $X_i\\sim N(0,1)$ and $E_i\\sim N(1,0.25^2)$ are independent. We use} the arguments \\texttt{adj\\_matrix} and \\texttt{make\\_corr\\_out} in the function below. This variable determines the potential outcome under the $d_{00}$ condition. \\hl{A positive lower bound of the outcomes in a simulated population refers to its realized minimum, not a bound on this distribution.}", "intended_meaning": "The sentence describes a baseline outcome that is correlated with degree and is built by the function \\texttt{make\\_corr\\_out}, which it names. That meaning is clear and correct. This function does not draw from the absolute value of a standard normal variable alone. It sets\n\\[\ny_i(d_{00})=\\bigl(k_i+k_i^{(2)}+k_ik_i^{(2)}\\bigr)\\,|X_i|+E_i,\n\\]\nwhere $k_i$ and $k_i^{(2)}$ are the first- and second-order degree of unit $i$, and $X_i\\sim N(0,1)$ and $E_i\\sim N(1,0.25^2)$ are independent. This distribution has no lower bound, since $E_i$ can be negative. The ``positive lower bound'' of the outcomes on p.~305 then refers to the smallest value in the fixed simulated population, which can be positive, and not to the distribution. The ten baseline values in the example of p.~298 are all positive, with minimum $6.194878$. No reported result depends on the description." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E006", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of the right column of page 292. Red boxes mark three terms: the equation that the size of Omega is 2 to the power N, the definition of Omega as the set of assignments z with positive probability p z, and, in the example with N equal to 10, the parenthesis stating that the size of Omega is 45.", "caption": "Section 16.3, p. 292 (PDF p. 4), right column. The marked terms are the two descriptions of $\\Omega$ and its size in the example with ten units.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E007.png" }, { "alt": "Crop of eight lines from the right column of page 292, beginning 'To analyze the effect of different treatment assignments'. A red box marks the expression y i, colon, Omega, arrow, R, which defines the potential outcome as a function from Omega to the real numbers.", "caption": "Section 16.3, p. 292 (PDF p. 4), right column, lower down. The marked term is the domain of the potential outcome function.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E007-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "$\\Omega$ is the set of assignments that the design can produce; the count $2^N$ belongs to the full space $\\{0,1\\}^N$, on which the potential outcomes are defined.", "id": "P045-E007", "location": "Section 16.3, p. 292 (PDF p. 4), right column: the definition of $\\Omega$ and the domain of the potential outcomes", "pdf_page": 4, "printed_page": "292", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "Let $\\Omega$ be the set of treatment assignment vectors $\\mathbf z$ with $\\hl{|\\Omega|=2^N}$. An \\emph{experimental design} is a plan for randomly selecting a particular value of $\\mathbf z$ from the $2^N$ different possibilities with predetermined probability $p_{\\mathbf z}$ -- for example, Bernoulli assignment (i.e., coin flips) or a completely randomized assignment strategy. Therefore, $\\Omega=\\{\\mathbf z:p_{\\mathbf z}>0\\}$ and the realized treatment assignment $\\mathbf Z=(Z_1,\\ldots,Z_N)'$ is a random vector with support $\\Omega$ and $\\Pr(\\mathbf Z=\\mathbf z)=p_{\\mathbf z}$. For example, with a population of size $N=10$ and an experimental design that randomly assigns without replacement a proportion $p=0.2$ to treatment condition $z_i=1$ with uniform probability, there are $\\binom{N}{pN}=45$ possible treatment assignments ($|\\Omega|=45$) and the realized treatment assignment $\\mathbf Z$ has $p_{\\mathbf z}=\\frac{1}{45}$. \\ldots\\ These potential outcomes are defined for each unit $i$ as the elements in the image of a function that maps assignment vectors to real valued outcomes, $y_i:\\hl{\\Omega}\\to\\mathbb R$.", "corrected": "\\hl{There are $2^N$ treatment assignment vectors $\\mathbf z$ in $\\{0,1\\}^N$.} An \\emph{experimental design} is a plan for randomly selecting a particular value of $\\mathbf z$ from the $2^N$ different possibilities with predetermined probability $p_{\\mathbf z}$ -- for example, Bernoulli assignment (i.e., coin flips) or a completely randomized assignment strategy. \\hl{Let $\\Omega=\\{\\mathbf z:p_{\\mathbf z}>0\\}$ be the support of the design. The} realized treatment assignment $\\mathbf Z=(Z_1,\\ldots,Z_N)'$ is a random vector with support $\\Omega$ and $\\Pr(\\mathbf Z=\\mathbf z)=p_{\\mathbf z}$. \\ldots\\ These potential outcomes are defined for each unit $i$ as the elements in the image of a function that maps assignment vectors to real valued outcomes, $y_i:\\hl{\\{0,1\\}^N}\\to\\mathbb R$.", "intended_meaning": "The set $\\Omega$ is the support of the design, $\\{\\mathbf z:p_{\\mathbf z}>0\\}$, and the chapter uses it in that sense. In the example on the same page, two of ten units are treated, and $|\\Omega|=\\binom{10}{2}=45$. The full space has $2^{10}=1024$ vectors. The count $2^N$ is the number of all vectors in $\\{0,1\\}^N$, and it equals $|\\Omega|$ only when the design gives every vector positive probability. With $\\Omega$ as the support, the map $y_i:\\Omega\\to\\mathbb R$ is defined only on the assignments that the design can produce. In the example these exclude the all-treated and all-control vectors, which define $\\tau(\\mathbf 1,\\mathbf 0)$ on p.~300. The potential outcomes for such assignments are meant to exist, and their domain is $\\{0,1\\}^N$. No estimator or result depends on the count." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E007", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of nine lines from the right column of page 292. A red box marks the first entry of the displayed vector z sub minus i, which is printed as z sub i and is followed by dots, z sub i minus 1, z sub i plus 1, dots, z sub N.", "caption": "Section 16.3, p. 292 (PDF p. 4), right column. The marked term is the first entry of the vector $\\mathbf z_{-i}$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E008.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T18", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T18" } ], "headline": "For $i>1$, the vector $\\mathbf z_{-i}$ begins with $z_1$, not with $z_i$.", "id": "P045-E008", "location": "Section 16.3, p. 292 (PDF p. 4), right column: the definition of $\\mathbf z_{-i}$", "pdf_page": 4, "printed_page": "292", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T18" ], "text": { "as_printed": "For convenience, let $\\mathbf z_{-i}=(z_{\\hl{i}},\\ldots,z_{i-1},z_{i+1},\\ldots,z_N)'$ denote the $(N-1)$-element vector that removes the $i$th element from $\\mathbf z$.", "corrected": "For convenience, let $\\mathbf z_{-i}=(z_{\\hl{1}},\\ldots,z_{i-1},z_{i+1},\\ldots,z_N)'$ denote the $(N-1)$-element vector that removes the $i$th element from $\\mathbf z$. \\hl{For $N>1$, the vector begins with $z_2$ when $i=1$ and ends with $z_{N-1}$ when $i=N$; for $N=1$, it is empty.}", "evidence": "The evidence is a check of the page. The first subscript in the formula is the letter $i$. The definition $\\mathbf z=(z_1,\\ldots,z_N)'$ at the top of the same column has the numeral $1$ in that place. For $N=4$ and $i=3$, removing the third element of $(z_1,z_2,z_3,z_4)'$ leaves $(z_1,z_2,z_4)'$, and the printed formula begins with $z_3$ instead. The same page uses $\\mathbf z_{-i}$ for ``the treatment assignment of all other households''.", "problem": "The words define $\\mathbf z_{-i}$ as the vector $\\mathbf z$ with its $i$th element removed. The formula begins with $z_i$, the element that is removed. For $i>1$, the correct first entry is $z_1$.", "what_survives": "Nothing else changes. The words beside the formula give the intended definition, and the chapter uses $\\mathbf z_{-i}$ in that sense afterwards." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E008", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Seven lines of text from the top of the left column. A red box marks the expression 'psi (n_g minus 1)' in the parenthesis '(i.e., psi (n_g minus 1) when i is assigned to treatment and psi (n_g) when i is assigned to control)'.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), top of the left column: the parenthesis that counts the treated group members other than $i$, with $\\psi(n_g-1)$ marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E009.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T19", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T19" } ], "headline": "Under complete random assignment, a treated unit has $\\psi n_g-1$ treated group members other than itself, not $\\psi(n_g-1)$.", "id": "P045-E009", "location": "Section 16.5.1, p. 310 (PDF p. 22), left column, first lines, the parenthesis beginning “(i.e.,”", "pdf_page": 22, "printed_page": "310", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T19" ], "text": { "as_printed": "\\ldots\\ complete random assignment, the individual-level direct effect for unit $i$ would contrast outcomes with different numbers of group members other than $i$ assigned to treatment (i.e., $\\hl{\\psi(n_g-1)}$ when $i$ is assigned to treatment and $\\psi(n_g)$ when $i$ is assigned to control).", "corrected": "\\ldots\\ complete random assignment, the individual-level direct effect for unit $i$ would contrast outcomes with different numbers of group members other than $i$ assigned to treatment (i.e., $\\hl{\\psi n_g-1}$ when $i$ is assigned to treatment and $\\psi(n_g)$ when $i$ is assigned to control).", "evidence": "A derivation, and the example on the same page. When $Z_{ig}=z$, the number of treated group members other than $i$ is\n\\[\n\\sum_{j\\neq i}Z_{jg}=\\psi n_g-z .\n\\]\nIn the toy example, $n_g=3$ and $\\psi=2/3$. Among the three assignments that treat two of the three units, a treated unit has one treated group member other than itself and a control unit has two. The printed expression equals $4/3$ for the treated unit.", "problem": "Complete random assignment at saturation $\\psi$ treats exactly $\\psi n_g$ of the $n_g$ units in the group. If unit $i$ is treated, it is one of them, and $\\psi n_g-1$ of the other units are treated. If unit $i$ is in control, all $\\psi n_g$ treated units are among the others. The printed $\\psi(n_g-1)$ equals $\\psi n_g-\\psi$, which exceeds the count for a treated unit by $1-\\psi$.", "what_survives": "The point of the parenthesis is unaffected: under complete random assignment, the outcomes of a treated unit and of a control unit are compared at different numbers of treated group members. The count $\\psi n_g$ for a control unit is correct, and nothing else in the chapter uses the expression." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E009", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of Table 16.3 on page 305, 'Comparing unit to cluster randomization', with columns Randomization, Estimator, Estimand, True value, Average value, Bias, SD and RMSE, four rows (Unit and Cluster for Horvitz-Thompson, Unit and Cluster for Hajek) and the note below the table. A red box marks the Estimand column, which reads tau(d1, d0) in all four rows.", "caption": "Table 16.3, p. 305 (PDF p. 17). The marked column gives the estimand, the contrast $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ between the two full-neighborhood exposure conditions.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E011.png" }, { "alt": "Crop of seven lines from the right column of page 305. The text reads 'We compare the Horvitz-Thompson and Hajek estimators and their variance for the causal estimand of full neighborhood exposure, tau(d1, d0), for the case in which units are assigned to treatment via unit-level randomization as opposed to three-net clustering randomization. In both cases,'. Red boxes mark the words 'the causal estimand of full neighborhood' and 'exposure, tau(d1, d0),'.", "caption": "Section 16.4.6, p. 305 (PDF p. 17), right column. The marked words name the estimand of Table 16.3.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E011-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T5", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T5" }, { "id": "P045-T6", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T6" }, { "id": "P045-T27", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T27" } ], "headline": "The package’s full-neighborhood exposure mapping must code a unit as all control when no member of its closed neighborhood is treated; as written it codes every unit whose closed neighborhood is not entirely treated as all control, so that the contrast it supports is not the $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ of Section 16.4.6.", "id": "P045-E011", "location": "Section 16.4.6 and Table 16.3, p. 305 (PDF p. 17); lines 104–121 of the file R/make_exposure_map_AS.R in the R package interference, version 0.1.0 (code, not printed in the chapter)", "pdf_page": 17, "printed_page": "305", "reaches": [ { "effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" }, { "effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P045-T27", "P045-T5", "P045-T6" ], "text": { "as_printed": "Section 16.4.6, p.~305: ``We compare the Horvitz--Thompson and Hajek estimators and their variance for the causal estimand of full neighborhood exposure, $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$, for the case in which units are assigned to treatment via unit-level randomization as opposed to three-net clustering randomization.''\n\nPackage, documentation of \\texttt{make\\_exposure\\_map\\_full\\_neighborhood}: ``that produces two exposure conditions: All Treat; a unit and 100\\% of its first-degree neighbors are treated, and All Control; a unit and 100\\% of its first-degree neighbors are in the control condition.''\n\\begin{audittable}{rl}{Package, \\texttt{R/make\\_exposure\\_map\\_AS.R}, lines 116--120.}\nLine & Code \\\\\n116 & \\texttt{exposure <- as.numeric(tr\\_vector\\%*\\%adj\\_matrix\\_diag)} \\\\\n117 & \\texttt{B\\_size <- rowSums(adj\\_matrix\\_diag)} \\\\\n118 & \\texttt{return(matrix(as.numeric(c(exposure==B\\_size,} \\\\\n119 & \\texttt{~~~~\\hl{exposurej$, and it changes no cell on or above the diagonal. The result is $U+U^{\\mathsf T}$, which is symmetric. For a symmetric input $A$ with entries 0 and 1 and zero diagonal, and upper cells $i1$. Let $S$ be the set of units $v\\le n$ that are within two hops of $n$, with $s=|S|$, and let $c_0$ be the number of earlier centers $v\\le n$. Each later draw is uniform on $\\{1,\\dots,n\\}$. A draw in $S$ marks $n$ and ends the loop, and a draw outside $S$ leaves $n$ unmarked, so the number of draws is geometric with mean $n/s$. Every unit other than $n$ is already marked, and a marked unit that is not a center is within two hops of a center, so that it cannot become a center of a 3-net. A draw of an earlier center only overwrites its label. This implies that the final centers form a 3-net exactly when every draw before the last selects an earlier center and the last draw selects $n$. Summing the geometric series, I obtain\n\\[\n\\Pr(\\text{the final centers form a 3-net})=\\sum_{k=0}^{\\infty}\\Bigl(\\frac{c_0}{n}\\Bigr)^{k}\\frac{1}{n}=\\frac{1}{n-c_0}.\n\\]\nFor the repair, the loop condition makes the vector of unmarked units nonempty, and \\texttt{sample.int} returns a position in that vector, so the new center is unmarked. The marked units are the units within two hops of the earlier centers. Then the new center is at least three hops from each of them. Each pass marks at least the new center, so the loop ends after at most $N$ passes, and at the end every unit is within two hops of a center. This holds for every finite undirected network.\n\n\\emph{New simulations.} I drew five networks of 400 units and mean degree 4 with the package's generator. On each network I ran the sampler as written and the repaired sampler 10,000 times to average the exposure probabilities conditional on each clustering, and I estimated the contrast $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ in 3000 further experiments for each sampler and for unit-level randomization, with the package's dilated outcomes. The two exposure conditions are those of the documentation, a closed neighborhood that is entirely treated or entirely in control (see \\fid{P045-E011}). The first table reports how often the defect occurs and how it changes the design. Over the 50,000 clusterings the sampler as written produced 23.4 more centers on average than the repaired sampler continued from the same state, and 92.7 pairs of centers within two hops of each other.\n\\begin{audittable}{rrrrrrr}{The sampler as written and the repaired sampler on five networks of 400 units, 10,000 clusterings each. Networks are named by the seed of the generator. Columns 2 to 4 and 6 are for the sampler as written. Column 2: share of clusterings that reach a single unmarked unit $n>1$. Column 3: share whose final centers are closer than three hops. Columns 4 and 5: mean number of clusters. Columns 6 and 7: probability that the closed neighborhood of a unit is entirely treated, averaged over units.}\nNetwork & Reached & Not a 3-net & Clusters & Clusters, repaired & Prob. & Prob., repaired \\\\\n357 & 92.50\\% & 90.61\\% & 84.116 & 60.338 & 0.170822 & 0.195066 \\\\\n492 & 93.58\\% & 92.36\\% & 83.503 & 60.829 & 0.168758 & 0.192388 \\\\\n104729 & 92.00\\% & 89.91\\% & 81.824 & 60.266 & 0.168589 & 0.191379 \\\\\n202603 & 93.57\\% & 92.17\\% & 88.622 & 61.576 & 0.168027 & 0.195761 \\\\\n8675309 & 92.09\\% & 90.82\\% & 81.030 & 59.145 & 0.169020 & 0.192765 \\\\\n\\end{audittable}\nThe second table reports the standard deviations of the two estimators. The repair lowers the estimated standard deviation under cluster randomization by 4.41\\% to 10.96\\% for the Horvitz--Thompson estimator and by 6.57\\% to 13.61\\% for the Hajek estimator. Under both samplers the ratio of the standard deviation under cluster randomization to the unit-level value is between 0.28 and 0.70 on every network. For comparison, the printed ratios of the Cluster to the Unit standard deviation in Table 16.3 are $61.31/139.65\\approx0.439$ and $28.89/52.89\\approx0.546$.\n\\begin{audittable}{rlrrr}{Standard deviation of the estimates of $\\tau(d_{\\mathbf{1}},d_{\\mathbf{0}})$ in the new simulations. The unit-level value is exact for the Horvitz--Thompson estimator and comes from 3000 experiments for the Hajek estimator; the cluster values come from 3000 experiments. All Hajek estimates were defined.}\nNetwork & Estimator & Unit-level & Cluster, as written & Cluster, repaired \\\\\n357 & Horvitz--Thompson & 116.407 & 73.682 & 67.867 \\\\\n357 & Hajek & 60.069 & 27.605 & 24.681 \\\\\n492 & Horvitz--Thompson & 106.384 & 74.172 & 68.388 \\\\\n492 & Hajek & 55.743 & 29.293 & 25.306 \\\\\n104729 & Horvitz--Thompson & 130.143 & 74.126 & 70.859 \\\\\n104729 & Hajek & 60.962 & 28.160 & 26.309 \\\\\n202603 & Horvitz--Thompson & 274.282 & 86.745 & 77.240 \\\\\n202603 & Hajek & 83.574 & 44.851 & 39.581 \\\\\n8675309 & Horvitz--Thompson & 132.629 & 73.636 & 68.371 \\\\\n8675309 & Hajek & 56.640 & 28.501 & 25.546 \\\\\n\\end{audittable}\nThe networks, the outcomes and the seeds of Table 16.3 are not available. The simulations measure the defect in the design of the table, and they do not give corrected entries for it.", "problem": "Table 16.3 compares unit-level randomization with cluster randomization, where the clusters come from 3-net clustering: the centers are at least three hops apart, every unit is within two hops of a center, each unit joins a nearest center, and treatment is assigned by cluster. The package's sampler builds the centers in a loop. It draws a center from the units that are not yet marked, marks the center and every unit within two hops of it, and repeats until every unit is marked. The draw is \\texttt{sample(which(!marked), 1)}. When exactly one unmarked unit $n$ remains, \\texttt{which(!marked)} is the single number $n$, and R then draws uniformly from $1,\\dots,n$ and not from the one remaining unit. Each draw that misses the units within two hops of $n$ writes a center on a unit that is already marked. Unless that unit is already a center, the new center lies within two hops of an earlier one. The loop continues until a draw lands within two hops of $n$.\n\nThis is not a rare case in new simulations of the design of Table 16.3. In 50,000 clusterings of five networks of 400 units, the loop reached a single unmarked unit $n>1$ in 92.75\\% of the clusterings, and the final centers were closer than three hops in 91.17\\%. The sampler as written returned 83.8 clusters on average, against 60.4 for the repaired sampler, and the estimated probability that the closed neighborhood of a unit is entirely treated, averaged over units, was 0.169, against 0.193. The design that the package draws under the name of 3-net clustering is therefore a different design. If the rows Cluster of Table 16.3 were computed with the package, they are not results for 3-net clustering. The script and the seeds of the table are not in the replication files, and no printed entry is shown to be wrong. In the new simulations the repaired sampler has estimated standard deviations that are lower by 4\\% to 14\\%.", "what_survives": "The rows Unit of Table 16.3 do not use the sampler. The estimand and the unbiasedness of the Horvitz--Thompson estimator with a correctly specified exposure mapping and the actual, positive exposure probabilities are unaffected. In my simulations under both samplers, the estimated standard deviation under cluster randomization is at most 0.70 of the unit-level value on every network. No printed entry is shown to be wrong. The sampler changes; if the package generated the rows Cluster, they have to be computed again before they are read as results for 3-net clustering. The printed definition of $\\epsilon$-net clustering on the same page is the subject of \\fid{P045-E010}." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E013", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "A grid of ten histograms of point estimates, in five columns and two rows labelled 'General interference' and 'Partial interference', each with a dashed vertical line. Red boxes mark the labels above the first two columns, which read tau-hat D(phi) and tau-hat D(psi). In the bottom row the dashed line of the first column is near 0.6 and that of the second column is near 0.4. The caption below ends 'The dashed vertical line represents the value of the estimand.'", "caption": "Figure 16.6, p. 312 (PDF p. 24), with its caption. The labels above the first two columns are marked: the first reads $\\widehat{\\tau}^{D}(\\phi)$ and the second reads $\\widehat{\\tau}^{D}(\\psi)$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E014.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T24", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T24" } ], "headline": "The labels of the first two columns of Figure 16.6 are interchanged: the first column shows the direct effect at $\\psi$ and the second shows the direct effect at $\\phi$.", "id": "P045-E014", "location": "Figure 16.6, p. 312 (PDF p. 24), the labels above the first two columns", "pdf_page": 24, "printed_page": "312", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T24" ], "text": { "as_printed": "Column labels, from left to right: $\\widehat{\\tau}^{D}(\\hl{\\phi})$, $\\widehat{\\tau}^{D}(\\hl{\\psi})$, $\\widehat{\\tau}^{I}(\\psi,\\phi)$, $\\widehat{\\tau}^{To}(\\psi,\\phi)$, $\\widehat{\\tau}^{O}(\\psi,\\phi)$. The caption ends: The dashed vertical line represents the value of the estimand.", "corrected": "Column labels, from left to right: $\\widehat{\\tau}^{D}(\\hl{\\psi})$, $\\widehat{\\tau}^{D}(\\hl{\\phi})$, $\\widehat{\\tau}^{I}(\\psi,\\phi)$, $\\widehat{\\tau}^{To}(\\psi,\\phi)$, $\\widehat{\\tau}^{O}(\\psi,\\phi)$.", "evidence": "A check of the page against the outcome model stated in the text. The positions of the ten dashed lines were measured from the drawing in the PDF. In the bottom row they are $0.595$, $0.397$, $0.199$, $0.794$ and $0.463$ from left to right. With $b=0.794$, the position of the line for the total effect, these agree to within $0.002$ with $3b/4$, $b/2$, $b/4$, $b$ and $7b/12$, the values of $\\tau^{D}(\\psi)$, $\\tau^{D}(\\phi)$, $\\tau^{I}$, $\\tau^{To}$ and $\\tau^{O}$ under the outcome model. In both rows the line for the total effect minus the line for the indirect effect agrees, within measurement precision, with the line of the first column, as the identity $\\tau^{To}=\\tau^{D}(\\psi)+\\tau^{I}$ requires, and differs from the line of the second column by more than $0.1$. The package also returns the direct effect at $\\psi$ before the direct effect at $\\phi$, which is the order of the output printed on p.~311. The simulation behind the figure was not run again, because its script and seeds are not available.", "problem": "Each dashed line marks the estimand of the estimator named above its column. In the bottom row, the outcome model on p.~310 gives $\\tau^{D}(\\psi)=(2-1.25)\\,b=3b/4$ and $\\tau^{D}(\\phi)=(1.5-1)\\,b=b/2$, where $b$ is the population mean of $y_{ig}(0,\\phi)$. The line for $\\phi$ should therefore stand at two-thirds of the line for $\\psi$. In the figure the line of the first column is at about $0.595$ and the line of the second is at about $0.397$, a ratio of $1.50$. The two labels are in the reverse order. With the printed labels, each direct effect is attached to the wrong saturation, and so is the direction of its bias in the upper row.", "what_survives": "The histograms, the dashed lines and the other three columns are unchanged. So is the lesson of the figure, that the estimators are unbiased under partial interference and biased when interference crosses groups. Only the two labels change." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E014", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the last six lines of the left column of page 303. The text reads 'the data than the coarser, biased estimators (here, tau-hat H of d1, d0). This is apparent if one looks at the standard deviations in Table 16.2. As such, with these sample sizes, the root-mean-square error is swamped by estimation variance relative to bias. As N gets larger, this'. Red boxes mark the words 'swamped by estimation' and, on the next line, 'variance relative to bias.'", "caption": "Section 16.4.3, p. 303 (PDF p. 15), left column, end of the column. The marked words are “swamped by estimation variance relative to bias”.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E015.png" }, { "alt": "Crop of Table 16.2 on page 301, 'Misspecifying exposure conditions', with columns Spillover, Estimand, Estimator, True value, Bias, SD and RMSE and twelve rows. Red boxes mark the Bias and SD entries of the four rows whose estimator is tau-hat H of d1, d0: minus 23.33 and 19.06; minus 32.21 and 21.42; 14.87 and 14.73; 23.92 and 13.65.", "caption": "Table 16.2, p. 301 (PDF p. 13). The marked entries are the Bias and the SD in the four rows with estimator $\\widehat{\\tau_H}(d_1,d_0)$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E015-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "At $N=400$ the greater variance of the finer exposure mappings often outweighs their reduction in squared bias; within the four rows of Table 16.2 that ignore interference, the squared bias exceeds the variance.", "id": "P045-E015", "location": "Section 16.4.3, p. 303 (PDF p. 15), left column, the sentence that begins “As such, with these sample sizes”; Table 16.2, p. 301 (PDF p. 13), the four rows with estimator $\\widehat{\\tau_H}(d_1,d_0)$", "pdf_page": 15, "printed_page": "303", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "``This is apparent if one looks at the standard deviations in Table 16.2. As such, with these sample sizes, the root-mean-square error is \\hl{swamped by estimation variance relative to bias}. As $N$ gets larger, this would change: the standard deviation would get smaller, but the bias would remain.''", "corrected": "``\\hl{With these sample sizes, the greater variance of finer exposure mappings often outweighs their reduction in squared bias when comparing root-mean-square errors.}''", "intended_meaning": "The sentences before this one compare the finer exposure mappings, which use fewer units and have larger variance, with the coarser mapping, which has more bias. In that context the sentence says that, at $N=400$, the variance added by a finer mapping often outweighs the squared bias that it removes. Table 16.2 supports this reading: the added variance exceeds the reduction in squared bias in seven of the eight comparisons between the estimator that ignores interference and a finer one. The exception is negative spillover with second-degree interference, where the first-degree estimator has the smaller root-mean-square error (22.21 against 27.54). Read as a statement about the two components of each root-mean-square error, the sentence does not hold for the four rows with estimator $\\widehat{\\tau_H}(d_1,d_0)$. There the squared bias exceeds the squared standard deviation: 544.29 against 363.28, 1037.48 against 458.82, 221.12 against 216.97 and 572.17 against 186.32. Each of these inequalities survives the rounding of the printed entries. The entries of the table are unchanged." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E015", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of five lines from the left column of page 301. The text reads 'where pi_i(d11) = Pr[Z_i I(Z' theta_i > 0) = 1]. Therefore, the misspecified mu-hat HT,None of 1 is biased for mu-hat HT of 1 insofar as y_i(d11) is not equal to y_i(d10) for some i.' Red boxes mark the words 'insofar as y_i(d11) not equal to y_i(d10)' and, on the next line, 'for some i'.", "caption": "Section 16.4.3, p. 301 (PDF p. 13), left column, the sentence after the display for $\\widehat{\\mu_{HT}}(\\mathbf{1})$. The marked words give the condition for bias.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E016.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T13", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T13" } ], "headline": "The estimator that ignores interference is biased when a weighted sum of the differences $y_i(d_{10})-y_i(d_{11})$ is nonzero; a difference for some unit $i$ with $\\pi_i(d_{10})>0$ suffices when all the differences have one sign, as they do in the chapter’s examples.", "id": "P045-E016", "location": "Section 16.4.3, p. 301 (PDF p. 13), left column, the sentence after the display for $\\widehat{\\mu_{HT}}(\\mathbf{1})$", "pdf_page": 13, "printed_page": "301", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [ "P045-T13" ], "text": { "as_printed": "``Therefore, the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ is biased for $\\widehat{\\mu_{HT}}(\\mathbf{1})$ \\hl{insofar as $y_i(d_{11})\\ne y_i(d_{10})$ for some $i$}.''", "corrected": "\\hl{For a correctly specified first-degree exposure mapping, with $\\pi_i=\\Pr[Z_i=1]>0$ and at least one direct peer for every unit,} the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ \\hl{has bias relative to $\\mu(\\mathbf{1})$}\n\\[\n\\hl{\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\pi_i(d_{10})}{\\pi_i}\\,\\bigl\\{y_i(d_{10})-y_i(d_{11})\\bigr\\}}.\n\\]\n\\hl{This bias is nonzero when the differences $y_i(d_{10})-y_i(d_{11})$ have one sign and at least one unit with $\\pi_i(d_{10})>0$ has a nonzero difference, as in the examples considered here.}", "intended_meaning": "The sentence explains that the estimator that ignores interference averages together the outcomes of treated units with and without a treated direct peer. In the chapter's simulations with dilated effects, the differences $y_i(d_{10})-y_i(d_{11})$ have one sign. In that setting a difference for some unit with $\\pi_i(d_{10})>0$ gives a nonzero bias, and the statement is correct. As a general condition it is incomplete. For a correctly specified first-degree exposure mapping, positive treatment probabilities and at least one direct peer per unit, the bias relative to $\\mu(\\mathbf{1})$ is the weighted sum $N^{-1}\\sum_{i}[\\pi_i(d_{10})/\\pi_i]\\{y_i(d_{10})-y_i(d_{11})\\}$, and differences of opposite sign can cancel. For example, let two units be joined by one tie and be treated independently with probability $1/2$, with $y(d_{11})=(2,1)$, $y(d_{10})=(1,2)$ and outcome zero for an untreated unit. The estimator takes the values $0$, $2$, $1$ and $3$ on the four assignments. Its expectation is $3/2=\\mu(\\mathbf{1})$, although $y_i(d_{11})\\ne y_i(d_{10})$ for both units. The discussion of the simulations on p.~303 concerns spillovers of one sign and needs no change. The denominators of the display before this sentence are the subject of \\fid{P045-E002}." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E016", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of seven lines from the left column of page 299. The text reads 'Confidence intervals are based on a large-N normal approximation:' and is followed by the interval tau-hat HT of d_k, d_l, plus or minus z with subscript (1 minus alpha) over 2, times the square root of the estimated variance of tau-hat HT. A red box marks z with its subscript (1 minus alpha)/2.", "caption": "Section 16.4.2, p. 299 (PDF p. 11), left column. The marked term is the critical value $z_{(1-\\alpha)/2}$ in the confidence interval.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-A001.png" } ], "formal_release": null, "formal_statements": [], "headline": "For nominal coverage $c$, the confidence interval on p. 299 uses the $(1+c)/2$ quantile of the standard normal distribution; the printed subscript $(1-\\alpha)/2$ comes with no definition of $\\alpha$ or of $z$.", "id": "P045-A001", "location": "Section 16.4.2, p. 299 (PDF p. 11), left column, the confidence interval after “Confidence intervals are based on a large-$N$ normal approximation”", "pdf_page": 11, "printed_page": "299", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "``Confidence intervals are based on a large-$N$ normal approximation:''\n\\[\n\\widehat{\\tau_{HT}}(d_k,d_l)\\pm \\hl{z_{(1-\\alpha)/2}}\\sqrt{\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau_{HT}}(d_k,d_l)]}.\n\\]", "corrected": "``Confidence intervals are based on a large-$N$ normal approximation: \\hl{for nominal coverage $c$,}''\n\\[\n\\widehat{\\tau_{HT}}(d_k,d_l)\\pm \\hl{z_{(1+c)/2}}\\sqrt{\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau_{HT}}(d_k,d_l)]},\n\\]\n\\hl{where $z_q=\\Phi^{-1}(q)$ is the $q$-quantile of the standard normal distribution}. Equivalently, with error probability $\\alpha=1-c$, the critical value is $z_{1-\\alpha/2}$.", "intended_meaning": "The intended interval is the usual two-sided normal interval. The passage defines neither $\\alpha$ nor $z$, and the printed subscript is correct under one convention and wrong under the usual one. Let $\\alpha$ be the error probability and let $z_q=\\Phi^{-1}(q)$ be the $q$-quantile of the standard normal distribution. Then the interval of level $1-\\alpha$ uses $z_{1-\\alpha/2}$, and for $\\alpha=0.05$ the printed subscript gives $\\Phi^{-1}(0.475)=-0.063$ in place of $\\Phi^{-1}(0.975)=1.960$. Suppose instead that $\\alpha$ is the coverage and that $z_q=\\Phi^{-1}(1-q)$ is the critical value for the upper tail. Then for $\\alpha=0.95$ the printed subscript gives $\\Phi^{-1}(0.975)=1.960$, which is the correct value. A script distributed with the package computes its 95\\% intervals with \\texttt{qnorm(.975)} and its 90\\% intervals with \\texttt{qnorm(.95)}, as the usual interval requires. No result of the chapter depends on the display." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-A001", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the first display in the left column of page 301. The first line defines mu-hat HT,None of 1 as one over N times the sum over i of the indicator of Z_i = 1 times Y_i over pi_i. The second and third lines expand it as one over N times the sum of the indicator of D_i = d11 times y_i(d11) over pi_i(d11), plus the indicator of D_i = d10 times y_i(d10) over pi_i(d10). Below, the text reads 'where pi_i = Pr[Z_i = 1], while the unbiased estimator would be'. Red boxes mark the two denominators pi_i(d11) and pi_i(d10).", "caption": "Section 16.4.3, p. 301 (PDF p. 13), left column, first display. The marked denominators are $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$ in the second equality.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E002.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T12", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T12" }, { "id": "P045-T13", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T13" } ], "headline": "In the expansion of $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ on p. 301, both denominators are $\\pi_i=\\Pr[Z_i=1]$, not the exposure probabilities $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$.", "id": "P045-E002", "location": "Section 16.4.3, p. 301 (PDF p. 13), left column, first display, second equality, and the sentence that begins “Therefore, the misspecified”", "pdf_page": 13, "printed_page": "301", "reaches": [ { "effect": "implementation_differs_from_paper", "effect_text": "the code implements something other than the printed method", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" }, { "effect": "uses_only_what_survives", "effect_text": "relies only on the part that survives", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T12", "P045-T13" ], "text": { "as_printed": "\\[\n\\begin{aligned}\n\\widehat{\\mu_{HT,None}}(\\mathbf{1})&=\\frac{1}{N}\\sum_{i=1}^{N}\\mathbf{I}(Z_i=1)\\frac{Y_i}{\\pi_i}\\\\\n&=\\frac{1}{N}\\sum_{i=1}^{N}\\left(\\mathbf{I}(D_i=d_{11})\\frac{y_i(d_{11})}{\\hl{\\pi_i(d_{11})}}+\\mathbf{I}(D_i=d_{10})\\frac{y_i(d_{10})}{\\hl{\\pi_i(d_{10})}}\\right),\n\\end{aligned}\n\\]\n``where $\\pi_i=\\Pr[Z_i=1]$, while the unbiased estimator would be'' $\\widehat{\\mu_{HT}}(\\mathbf{1})=\\widehat{\\mu_{HT}}(d_{11})$ \\ldots\\ ``Therefore, the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ is biased for \\hl{$\\widehat{\\mu_{HT}}(\\mathbf{1})$} insofar as $y_i(d_{11})\\ne y_i(d_{10})$ for some $i$.''", "corrected": "\\[\n\\begin{aligned}\n\\widehat{\\mu_{HT,None}}(\\mathbf{1})&=\\frac{1}{N}\\sum_{i=1}^{N}\\mathbf{I}(Z_i=1)\\frac{Y_i}{\\pi_i}\\\\\n&=\\frac{1}{N}\\sum_{i=1}^{N}\\left(\\mathbf{I}(D_i=d_{11})\\frac{y_i(d_{11})}{\\hl{\\pi_i}}+\\mathbf{I}(D_i=d_{10})\\frac{y_i(d_{10})}{\\hl{\\pi_i}}\\right),\n\\end{aligned}\n\\]\nwhere $\\pi_i=\\Pr[Z_i=1]$\\hl{${}=\\pi_i(d_{11})+\\pi_i(d_{10})$}. The equality holds for every assignment when the first-degree exposure mapping is correctly specified and $\\pi_i>0$ for every $i$. The closing sentence becomes: Therefore, \\hl{relative to $\\mu(\\mathbf{1})$,} the misspecified $\\widehat{\\mu_{HT,None}}(\\mathbf{1})$ \\hl{has bias}\n\\[\n\\hl{\\frac{1}{N}\\sum_{i=1}^{N}\\frac{\\pi_i(d_{10})}{\\pi_i}\\,\\bigl\\{y_i(d_{10})-y_i(d_{11})\\bigr\\}}.\n\\]\nThis formula for the bias holds when, in addition, every unit has at least one direct peer, so that $\\mu(\\mathbf{1})=\\mu(d_{11})$.", "evidence": "The evidence is a derivation and an exact counterexample. For the derivation, suppose that the first-degree exposure mapping of p.~294 is correctly specified and that $\\pi_i$, $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$ are positive for every $i$. By the definition of the exposure conditions, the events $\\{D_i=d_{11}\\}$ and $\\{D_i=d_{10}\\}$ partition the event $\\{Z_i=1\\}$. This implies that $\\pi_i=\\pi_i(d_{11})+\\pi_i(d_{10})$ and that $\\mathbf{I}(Z_i=1)=\\mathbf{I}(D_i=d_{11})+\\mathbf{I}(D_i=d_{10})$. Since $Y_i=y_i(d)$ on the event $\\{D_i=d\\}$, I have\n\\[\n\\mathbf{I}(Z_i=1)\\,Y_i=\\mathbf{I}(D_i=d_{11})\\,y_i(d_{11})+\\mathbf{I}(D_i=d_{10})\\,y_i(d_{10})\n\\]\nfor every assignment. Dividing by $\\pi_i$ and averaging over $i$ gives the expansion in the corrected display below. The potential outcomes are fixed, and the two indicators have expectations $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$. Taking expectations in the first line and in the printed second line, I obtain\n\\[\n\\begin{aligned}\n\\mathbb{E}\\bigl[\\widehat{\\mu_{HT,None}}(\\mathbf{1})\\bigr]&=\\frac{1}{N}\\sum_{i=1}^{N}\\left(\\frac{\\pi_i(d_{11})}{\\pi_i}\\,y_i(d_{11})+\\frac{\\pi_i(d_{10})}{\\pi_i}\\,y_i(d_{10})\\right),\\\\\n\\mathbb{E}[\\text{printed second line}]&=\\frac{1}{N}\\sum_{i=1}^{N}\\bigl(y_i(d_{11})+y_i(d_{10})\\bigr)=\\mu(d_{11})+\\mu(d_{10}).\n\\end{aligned}\n\\]\nNote that in the first expectation the two weights of each unit sum to one. If every unit has a direct peer, then $\\mu(\\mathbf{1})=\\mu(d_{11})$, and subtracting it from the first expectation gives the bias in the corrected sentence below.\n\nFor the counterexample, let ten units form a cycle, let exactly two units be treated, with all 45 assignments equally likely, and let $y_i(d_{11})=2$, $y_i(d_{10})=1$, $y_i(d_{01})=1$ and $y_i(d_{00})=0$ for every unit. Then $\\pi_i=1/5$, $\\pi_i(d_{11})=2/45$ and $\\pi_i(d_{10})=7/45$. On each of the 10 assignments in which the two treated units are adjacent, the first line equals $2$ and the printed second line equals $9$. On each of the other 35 assignments, the first line equals $1$ and the printed second line equals $9/7$. Enumerating the 45 assignments in exact arithmetic, I find that the first line has expectation $11/9$ and the printed second line has expectation $3$, while $\\mu(\\mathbf{1})=2$. The printed equality fails on all 45 assignments, and the corrected equality holds on all 45. The bias of the first line for $\\mu(\\mathbf{1})$ is $11/9-2=-7/9$, which is the value of the corrected formula.", "problem": "The first line of the display defines the estimator that ignores interference. It weights the outcome of every treated unit by $1/\\pi_i$, where $\\pi_i$ is the probability that unit $i$ is treated. The second line is presented as an expansion of the first over the two exposure conditions of a treated unit, $d_{11}$ (a treated direct peer) and $d_{10}$ (none). Such an expansion keeps the denominator $\\pi_i$ in both terms. The printed line divides instead by the exposure probabilities $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$, and the result is a different quantity. Its expectation is $\\mu(d_{11})+\\mu(d_{10})$, the sum of the averages of two potential outcomes. The expectation of the first line is the mixture of $y_i(d_{11})$ and $y_i(d_{10})$ that the surrounding text describes. The closing sentence also calls the estimator biased for another estimator, $\\widehat{\\mu_{HT}}(\\mathbf{1})$, where the estimand $\\mu(\\mathbf{1})$ is meant. The condition ``for some $i$'' in that sentence is the subject of \\fid{P045-E016}.", "what_survives": "The definition of the estimator in the first line, the Horvitz--Thompson estimators of Section 16.4.2 and of the next display, and the explanation that the estimator mixes $y_i(d_{11})$ and $y_i(d_{10})$ are unaffected. In both copies of the package in the replication files, the corresponding code divides the treated total by one common probability. No error in Table 16.2 or in Figure 16.4 is established: they report simulations of Hajek estimators, and I found no dependence of them on the printed expansion. The second equality and the closing sentence change." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E002", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Six lines of text from the foot of the left column. Red boxes mark the words 'at least within epsilon hops' at the end of the fifth line and 'of each other' at the start of the sixth, in the sentence 'clusters are formed by finding a set of units such that all units in the set are at least within epsilon hops of each other, and every unit outside the'.", "caption": "Section 16.4.6, p. 305 (PDF p. 17), foot of the left column: the definition of $\\epsilon$-net clustering, with the phrase “at least within $\\epsilon$ hops of each other” marked. The sentence continues at the top of the right column.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-E010.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T20", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T20" }, { "id": "P045-T21", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T21" } ], "headline": "In the definition of $\\epsilon$-net clustering, the units in the set are at least $\\epsilon$ hops apart, not “at least within $\\epsilon$ hops of each other”.", "id": "P045-E010", "location": "Section 16.4.6, p. 305 (PDF p. 17), last lines of the left column, continuing at the top of the right column", "pdf_page": 17, "printed_page": "305", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T20", "P045-T21" ], "text": { "as_printed": "Ugander et al.\\ (2013) analyze an intuitive graph clustering method, $\\epsilon$-net clustering, where clusters are formed by finding a set of units such that all units in the set are \\hl{at least within $\\epsilon$ hops of each other}, and every unit outside the set is within $\\epsilon-1$ hops of a unit in the set.", "corrected": "Ugander et al.\\ (2013) analyze an intuitive graph clustering method, $\\epsilon$-net clustering, where clusters are formed by finding a set of units such that all units in the set are \\hl{at least $\\epsilon$ hops apart}, and every unit outside the set is within $\\epsilon-1$ hops of a unit in the set.", "evidence": "I checked the page and the intended greedy clustering procedure, and used an exact example.\n\nThe greedy procedure intended by \\texttt{make\\_tr\\_vec\\_bernoulli\\_cluster} in the \\texttt{interference} package picks an unmarked unit, marks every unit within two hops of it, and repeats until no unit is unmarked. These rules give pairwise separation of at least three hops and coverage within two hops. When only one eligible vertex remains, the actual sampler can choose an already-marked unit and violate the separation condition; see \\fid{P045-E013}. On a path with nine vertices and $\\epsilon=3$, none of the 511 nonempty sets of vertices has all its members within three hops of each other and every vertex within two hops of the set, while 13 sets have all their members at least three hops apart and every vertex within two hops of the set. The article of Ugander et al.\\ (2013) was not consulted.", "problem": "The marked phrase joins two conditions that point in opposite directions. ``At least'' is a lower bound on the distance between two units in the set, and ``within'' is an upper bound. The definition needs the lower bound: any two units in the set are at least $\\epsilon$ hops apart, and the second clause supplies the upper bound, on the distance from a unit outside the set to the set. If the phrase is read as an upper bound, the two clauses can be impossible to satisfy together.", "what_survives": "The second clause of the definition and the intended 3-net design in Section 16.4.6 are unaffected by this wording correction. No change to the simulation numbers follows from this phrase alone." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-E010", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the first five lines of the left column of page 297, beginning 'The columns capture pi i, the expected proportion of treatment assignments that result in each exposure'. A red box marks the bold symbol pi sub i.", "caption": "Section 16.4.1, p. 297 (PDF p. 9), first lines of the left column. The marked symbol is $\\boldsymbol{\\pi}_i$.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N001.png" } ], "formal_release": null, "formal_statements": [], "headline": "The columns of the printed output are adjusted Monte Carlo estimates of the exposure probabilities, $(C_{ik}+1)/(R+1)$, and each column sums to $34/31$ before rounding.", "id": "P045-N001", "location": "Section 16.4.1, p. 297 (PDF p. 9), first sentence of the page, on the output of make_prob_exposure_cond printed on p. 296 (PDF p. 8)", "pdf_page": 9, "printed_page": "297", "reaches": [], "related_entries": [ { "id": "P044-E005", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-E005", "work_id": "P044" } ], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "The columns capture $\\hl{\\boldsymbol{\\pi}_i}$, the expected proportion of treatment assignments that result in each exposure.", "corrected": "The columns \\hl{contain adjusted Monte Carlo estimates $\\widehat{\\boldsymbol{\\pi}}_i$ of each unit's exposure probabilities: the entry for exposure $d_k$ is $(C_{ik}+1)/(R+1)$, where $C_{ik}$ is the number of the $R$ sampled assignments that place unit $i$ in $d_k$. The four entries of a column sum to $(R+4)/(R+1)$, here $34/31$, and not to one.}", "intended_meaning": "Each column of the output on p.~296 holds estimates $\\widehat{\\boldsymbol{\\pi}}_i$ of the four exposure probabilities of one unit, computed from the $R=30$ sampled assignments. Page 296 says that the example works with 30 random draws from $\\Omega$, and pp.~297--298 distinguish the estimates $\\widehat\\pi_i(d_k)$ from the true values $\\pi_i(d_k)$, so that this meaning is clear and correct. The chapter does not say that the package adds one to each count and to the number of draws. The entry for unit $i$ and exposure $d_k$ is $(C_{ik}+1)/(R+1)$, where $C_{ik}$ is the number of sampled assignments that place unit $i$ in $d_k$. Since the four counts of a unit sum to $R$, I find that each column sums to $(R+4)/(R+1)=34/31$ before rounding, and not to one. For unit 1 the unrounded entries are $(3,6,13,12)/31$, the sample proportions are $(2,5,12,11)/30$, and the exact probabilities over the 45 assignments are $(3,6,21,15)/45$. The contrast $41.61043$ on p.~298 is reproduced from these estimates, and no reported number changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N001", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of six lines from the right column of page 299. The text reads 'ance is with little cost in bias, as indicated by a smaller root-mean-square error. Moreover, the variance estimator is consistent, given that the mean of the standard error estimate approaches the empirical standard deviation from the simulation.' A red box marks the line 'the variance estimator is consistent, given'.", "caption": "Section 16.4.2, p. 299 (PDF p. 11), right column, last sentence of the section. The marked line draws the conclusion of consistency.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N002.png" }, { "alt": "Crop of Table 16.1 on page 300, 'Comparing Horvitz-Thompson to Hajek estimators using approximate exposure probabilities', with columns Estimator, Estimand, True value, Average value, Bias, SD, RMSE and Mean SE, six rows and the note below the table. Red boxes mark four entries in the two rows for tau(d10, d00): SD 26.60 and Mean SE 33.11 for Horvitz-Thompson, and SD 21.55 and Mean SE 25.09 for Hajek.", "caption": "Table 16.1, p. 300 (PDF p. 12). The marked entries are the SD and the Mean SE for $\\tau(d_{10},d_{00})$, for the Horvitz–Thompson estimator and for the Hajek estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N002-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T22", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T22" }, { "id": "P045-T23", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T23" } ], "headline": "Table 16.1 does not show that the variance estimator is consistent: at $N=400$ the mean standard error is within 7.1 percent of the empirical standard deviation for two contrasts and exceeds it by 16.4 and 24.5 percent for the third.", "id": "P045-N002", "location": "Section 16.4.2, p. 299 (PDF p. 11), right column, last sentence; Table 16.1, p. 300 (PDF p. 12), columns SD and Mean SE", "pdf_page": 11, "printed_page": "299", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P045-T22", "P045-T23" ], "text": { "as_printed": "``Moreover, \\hl{the variance estimator is consistent, given that} the mean of the standard error estimate approaches the empirical standard deviation from the simulation.''", "corrected": "``\\hl{At $N=400$, the mean standard error estimates are within 7.1 percent of the empirical standard deviations for $\\tau(d_{11},d_{00})$ and $\\tau(d_{01},d_{00})$, and exceed them by 16.4 percent for Hajek and 24.5 percent for Horvitz--Thompson for $\\tau(d_{10},d_{00})$. These finite-sample comparisons do not establish asymptotic consistency.}''", "evidence": "The evidence is a recalculation from Table 16.1 and an exact counterexample to the inference. The ratios of the printed entries are in the table below. For $\\tau(d_{10},d_{00})$ they are $33.11/26.60=473/380$ and $25.09/21.55=2509/2155$, and both remain more than 16 percent above one when each entry is allowed to vary by its rounding error. The other four ratios are within 7.1 percent of one.\n\\begin{audittable}{llrrr}{Table 16.1, p.~300: the mean standard error estimate against the empirical standard deviation.}\nEstimator & Estimand & SD & Mean SE & Mean SE relative to SD \\\\\nHorvitz--Thompson & $\\tau(d_{11},d_{00})$ & 45.32 & 45.94 & $+1.4\\%$ \\\\\nHorvitz--Thompson & $\\tau(d_{10},d_{00})$ & 26.60 & 33.11 & $+24.5\\%$ \\\\\nHorvitz--Thompson & $\\tau(d_{01},d_{00})$ & 10.45 & 10.33 & $-1.1\\%$ \\\\\nHajek & $\\tau(d_{11},d_{00})$ & 36.66 & 34.07 & $-7.1\\%$ \\\\\nHajek & $\\tau(d_{10},d_{00})$ & 21.55 & 25.09 & $+16.4\\%$ \\\\\nHajek & $\\tau(d_{01},d_{00})$ & 8.38 & 8.41 & $+0.4\\%$ \\\\\n\\end{audittable}\nFor the counterexample, let $X_1,\\dots,X_N$ be independent signs, each equal to $1$ or $-1$ with probability $1/2$, and let $T_N=N^{-1}\\sum_{i=1}^N X_i$ estimate $\\theta=0$. Then $T_N$ is consistent and $V_N=\\operatorname{Var}(T_N)=1/N$. Define $B=1/2$ if $X_1=-1$ and $B=3/2$ if $X_1=1$, and define the variance estimate $\\widehat V_N=B^2/N$. Since $\\mathbb{E}[B]=1$, I have\n\\[\n\\mathbb{E}\\Bigl[\\sqrt{\\widehat V_N}\\Bigr]=\\frac{1}{\\sqrt N}=\\sqrt{V_N}\\qquad\\text{for every } N,\n\\]\nso that the mean standard error equals the true standard deviation at every sample size. The estimate is also conservative in expectation, since $\\mathbb{E}[\\widehat V_N]=5/(4N)\\ge V_N$. But $\\widehat V_N/V_N=B^2$ takes the values $1/4$ and $9/4$ with probability $1/2$ each. This implies that\n\\[\n\\Pr\\bigl(\\,\\bigl|\\widehat V_N/V_N-1\\bigr|>1/2\\,\\bigr)=1\\qquad\\text{for every } N,\n\\]\nand the ratio does not tend to one. Note that in this example the difference $\\widehat V_N-V_N$ does tend to zero, since $|\\widehat V_N-V_N|\\le 5/(4N)$. The counterexample concerns the inference only. It is not one of the chapter's variance estimators.", "problem": "The sentence infers that the variance estimator is consistent from the fact that, in Table 16.1, the mean of the standard error estimates is close to the empirical standard deviation of the point estimates. The conclusion does not follow from the table. Table 16.1 reports one design with $N=400$, so it shows nothing about convergence as $N$ grows. The two columns also disagree for one of the three contrasts: for $\\tau(d_{10},d_{00})$ the mean standard error exceeds the empirical standard deviation by 24.5 percent for the Horvitz--Thompson estimator and by 16.4 percent for the Hajek estimator. Finally, agreement between the mean standard error and the standard deviation does not imply consistency, even if it holds at every sample size. The property that matters for a sampling distribution whose scale shrinks is that the ratio of the variance estimate to the true variance tends to one in probability, and the mean of the standard errors does not control that ratio. This finding concerns the inference in the sentence. It does not show that either variance estimator is inconsistent.", "what_survives": "All six rows of Table 16.1 stand, and so does the comparison that the table supports: the Hajek estimator has the smaller standard deviation and the smaller root-mean-square error for each contrast. No later result is derived from the sentence. Whether the two variance estimators are consistent under suitable asymptotic assumptions is not settled either way by this finding, and the large-$N$ confidence interval on p.~299 rests on separate theory." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N002", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "A block of R code and output in typewriter type. The first line reads 'estimates <- estimates_hierarchical(post_tr_data)'. The output lists five point estimates and then five variances. A red box marks the entries 'var_direct_psi_hat 0.71255260' and 'var_direct_phi_hat 0.04854851'. The other variances are var_indirect_hat 0.28000088, var_total_hat 0.88936810 and var_overall_hat 0.39005445.", "caption": "Toy example, p. 311 (PDF p. 23): the call to estimates_hierarchical and its output, with the two direct-effect variance entries marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N008.png" }, { "alt": "Twelve lines of text from the right column. Red boxes mark the words '18 units equally divided into' at the end of the second line and 'six groups' at the start of the third. The following lines say that two-thirds of units in groups with saturation psi are assigned to the treatment condition and one-third to the control condition, and one-third of units in groups with saturation phi are assigned to treatment.", "caption": "Section 16.5.1, p. 310 (PDF p. 22), right column: the design of the toy example, with “18 units equally divided into six groups” marked. Each group has three units, two in one arm and one in the other.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N008-p310.png" }, { "alt": "Two panels, each with five rows and a horizontal axis of variance. Blue vertical ticks mark the true variance and orange dots the expected value of the reported variance. Left panel, constant additive effect: for the direct effects at psi and at phi the dot is at 0.33 and the tick at 0.50, labelled 'ratio 2/3'; for the indirect, total and overall effects the dot lies on the tick, at 0.28, 0.11 and 0. Right panel, outcome model of the example: for the direct effect at psi the dot is at 0.28 and the tick at 0.34, labelled 'ratio 0.80'; for the direct effect at phi the dot is at 0.16 and the tick at 0.27, labelled 'ratio 0.60'; for the indirect, total and overall effects the dot is at or slightly to the right of the tick, near 0.24, 0.30 and 0.21.", "caption": "True variance of each point estimator and expected value of the variance that estimates_hierarchical reports, over the 14,580 assignments of the toy design. Left: every group has baseline outcomes $(1,2,3)$ and a constant additive effect $d=1$. Right: the outcome model of the example, on data consistent with the printed output. The ratio is the expected reported variance divided by the true variance. The values are exact sums over the assignments.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N008-variance.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T3", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T3" }, { "id": "P045-T4", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T4" }, { "id": "P045-T26", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T26" } ], "headline": "In the toy example every group has an arm with one unit, so the variance estimator for a direct effect that p. 310 describes is undefined; the two direct-effect variances in the output omit the undefined terms and are too small in expectation.", "id": "P045-N008", "location": "Section 16.5.1, toy example, p. 311 (PDF p. 23), the two direct-effect variances in the printed output; with the description of the variance estimators and of the design on p. 310 (PDF p. 22)", "pdf_page": 23, "printed_page": "311", "reaches": [ { "effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P013/", "where": null, "work_id": "P013" }, { "effect": "implementation_wrong", "effect_text": "the package or replication code has the reported defect", "explanation": null, "package": null, "recorded_as": null, "to": "work", "url": "https://pmaronow.github.io/llm-audit/works/P044/", "where": null, "work_id": "P044" } ], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P045-T26", "P045-T3", "P045-T4" ], "text": { "as_printed": "\\emph{p.~310.} When stratified interference holds, their proposed variance estimators are unbiased if unit causal effects are additive (e.g., $y_{ig}(1,\\psi)=y_{ig}(0,\\psi)+d$, where $d$ is constant) and otherwise positively biased. \\ldots\\ Suppose we have 18 units equally divided into six groups. In the first stage, half of the groups are assigned to saturation $\\psi$ and the other half to $\\phi$ with equal probability. In the second stage, using complete random assignment, two-thirds of units in groups with saturation $\\psi$ are assigned to the treatment condition and one-third to the control condition, and one-third of units in groups with saturation $\\phi$ are assigned to treatment, while two-thirds are assigned to control.\n\n\\emph{p.~311.} The estimators and their variance are computed under the stratified interference assumption with the following function from the \\texttt{interference} package: \\ldots\n\n\\begin{audittable}{lr}{The variance output of \\texttt{estimates\\_hierarchical(post\\_tr\\_data)} on p.~311, as printed.}\nEntry & Value \\\\\n\\texttt{var\\_direct\\_psi\\_hat} & \\hl{\\texttt{0.71255260}} \\\\\n\\texttt{var\\_direct\\_phi\\_hat} & \\hl{\\texttt{0.04854851}} \\\\\n\\texttt{var\\_indirect\\_hat} & \\texttt{0.28000088} \\\\\n\\texttt{var\\_total\\_hat} & \\texttt{0.88936810} \\\\\n\\texttt{var\\_overall\\_hat} & \\texttt{0.39005445} \\\\\n\\end{audittable}", "corrected": "\\emph{p.~310.} \\hl{Under stratified interference and the two-stage randomization specified here, the proposed variance estimator for a direct effect is unbiased exactly when unit causal effects are constant within each group; the constants may differ across groups. A common additive effect} (e.g., $y_{ig}(1,\\psi)=y_{ig}(0,\\psi)+d$, where $d$ is constant) \\hl{is one such case. Otherwise its bias is positive. The first stage chooses a fixed number of groups uniformly for each saturation. For each fixed saturation, the complete random assignments within groups are independent across groups and independent of which groups are selected for that saturation. The variance estimator for a direct effect uses the sample variance of the outcomes in each arm of each group. It requires at least two treated and at least two control units in every group at that saturation, that is, $2\\le K_g\\le n_g-2$ where $K_g$ is the number of treated units in group $g$, and at least two groups at each saturation.}\n\n\\emph{p.~311.} \\hl{The two direct-effect variances are not available in this design, because every group has an arm with one unit.}\n\n\\begin{audittable}{lr}{The variance output on p.~311, corrected.}\nEntry & Value \\\\\n\\texttt{var\\_direct\\_psi\\_hat} & \\hl{\\texttt{NA}} \\\\\n\\texttt{var\\_direct\\_phi\\_hat} & \\hl{\\texttt{NA}} \\\\\n\\texttt{var\\_indirect\\_hat} & \\texttt{0.28000088} \\\\\n\\texttt{var\\_total\\_hat} & \\texttt{0.88936810} \\\\\n\\texttt{var\\_overall\\_hat} & \\texttt{0.39005445} \\\\\n\\end{audittable}\n\nIn the package, \\texttt{estimates\\_hierarchical} should return \\texttt{NA} with a warning when a sample variance that the formula needs is undefined. If numerical direct-effect variances are wanted in the example, use the uniform and independent randomization specified above with six groups of six units, treating four units per group at one saturation and two at the other. This keeps both saturations and has \\hl{at least} two units in every arm. The printed numbers would then change.", "evidence": "I checked the package output on data consistent with the example, an exact counterexample and a derivation of the bias.\n\n\\emph{The printed output.} Data were found that match the six printed rows and all ten printed outputs to every printed digit. On these data both versions of \\texttt{estimates\\_hierarchical} that I examined return the printed output with no warning. The six arms with one unit have undefined sample variance. The two direct-effect variances in the output are $0.695937+0.016616=0.712553$ and $0.012457+0.036092=0.048549$, where the first term is the part between groups and the second is the part within groups from the arms with two units.\n\n\\emph{The counterexample.} In each of the six groups let the units have $y_{ig}(0;s)=x_i$ and $y_{ig}(1;s)=x_i+1$ at both saturations $s$, with $(x_1,x_2,x_3)=(1,2,3)$. Unit effects are constant and additive with $d=1$, and partial and stratified interference hold. The design has $20\\cdot 3^{6}=14{,}580$ equally likely assignments. Summing over them in exact fractions, I obtain the following values.\n\n\\begin{audittable}{lcc}{Constant additive effect $d=1$, baseline outcomes $(1,2,3)$ in every group: true variance of each estimator and expected value of the reported variance.}\nEstimator & True variance & Expected reported variance \\\\\nDirect effect at $\\psi$ & $1/2$ & $1/3$ \\\\\nDirect effect at $\\phi$ & $1/2$ & $1/3$ \\\\\nIndirect effect & $5/18$ & $5/18$ \\\\\nTotal effect & $1/9$ & $1/9$ \\\\\nOverall effect & $0$ & $0$ \\\\\n\\end{audittable}\n\n\\emph{The bias.} Fix a saturation, and let $G$ be the number of groups. At the first stage, a fixed number of at least two groups is chosen uniformly for this saturation. The complete random assignments within groups at this saturation are independent across groups and independent of the first-stage selection. In group $g$, with $n$ units of which $m$ are treated, let $S^{2}_{a,g}$, $S^{2}_{b,g}$ and $S^{2}_{\\delta,g}$ be the variances, with divisor $n-1$, of the potential outcomes under treatment, of the potential outcomes under control and of the unit effects at this saturation. The difference in means in group $g$ has variance\n\\[\nV_g=\\frac{S^{2}_{a,g}}{m}+\\frac{S^{2}_{b,g}}{n-m}-\\frac{S^{2}_{\\delta,g}}{n}.\n\\]\nIf the within-group term of the variance estimator has expectation $B_g$ in group $g$, the estimator has bias $G^{-2}\\sum_{g}(B_g-V_g)$. When both arms have at least two units, $B_g=S^{2}_{a,g}/m+S^{2}_{b,g}/(n-m)$, and the bias is $G^{-2}\\sum_{g}S^{2}_{\\delta,g}/n$. Under this design the bias is nonnegative, and it is zero exactly when unit effects are constant within every group; the constants may differ across groups. Otherwise it is positive. In the toy design $G=6$ and $n=3$, with $m=2$ at $\\psi$ and $m=1$ at $\\phi$. When the term of the arm with one unit is dropped, $B_g=S^{2}_{a,g}/2$ at $\\psi$ and $B_g=S^{2}_{b,g}/2$ at $\\phi$, and I obtain\n\\[\n\\operatorname{Bias}_{\\psi}=\\frac{1}{G^{2}}\\sum_{g=1}^{G}\\Bigl(\\frac{S^{2}_{\\delta,g}}{3}-S^{2}_{b,g}\\Bigr),\n\\qquad\n\\operatorname{Bias}_{\\phi}=\\frac{1}{G^{2}}\\sum_{g=1}^{G}\\Bigl(\\frac{S^{2}_{\\delta,g}}{3}-S^{2}_{a,g}\\Bigr).\n\\]\nUnder constant effects $S^{2}_{\\delta,g}=0$; the bias at a saturation is negative if outcomes vary within any group at that saturation. Under the outcome model of the example, with $S^{2}_{x,g}$ the variance of the baseline outcomes in group $g$, the two biases are $-(11/8)\\,G^{-2}\\sum_{g}S^{2}_{x,g}$ and $-(13/6)\\,G^{-2}\\sum_{g}S^{2}_{x,g}$. On the data consistent with the printed output, the expected reported variances are $0.276$ and $0.161$, and the true variances are $0.344$ and $0.267$; the ratios are $0.80$ and $0.60$. In the other direction, if $y_{ig}(0;s)=0$ and $y_{ig}(1;s)=x_i$ with $(x_1,x_2,x_3)=(1,2,3)$ in every group, the expected reported variance at $\\psi$ is $1/9$ and the true variance is $1/18$.", "problem": "The variance estimator that Hudgens and Halloran (2008) propose for a direct effect, and that p.~310 describes, uses the sample variance of the observed outcomes in each arm of each group, and a sample variance needs at least two observations. In the toy design each group has three units, two in one arm and one in the other. One of the two sample variances is then undefined in every group. The function \\texttt{estimates\\_hierarchical} sums the terms that are defined and drops the others without a warning. In each of the two direct-effect variances in the output, the part within groups therefore contains only the terms from the arms with two units. This quantity is not the estimator described, and it does not have the property stated on p.~310: in this design its expectation can fall below the true variance even with constant additive effects, and it does so under the outcome model of the example. The statement concerns the expectation over assignments and not the two printed numbers themselves. With other potential outcomes the omission can also bias the reported variance upward.", "what_survives": "The five point estimates are unaffected, since an arm mean is defined for an arm with one unit. The variances of the indirect, total and overall effects use sample variances across the three groups at each saturation and are unaffected. Figure 16.6 shows point estimates only, in a design with 75 units per group and arms of 50 and 25. The direct-effect variance guarantee holds under stratified interference when a fixed number of at least two groups is selected uniformly for each saturation, the within-group complete random assignments at a fixed saturation are independent across groups and independent of the first-stage selection, and every arm has at least two units. Under these conditions the bias is zero exactly when unit effects are constant within each group, with possibly different constants across groups, and otherwise positive. What changes is the two direct-effect variances of the example and, for a user of the function, the variance reported for a direct effect in any design that has an arm with one unit." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N008", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Nine lines of text from the right column. A red box marks the word 'municipalities' in the sentence 'In this case, only the first stage is randomized, assigning municipalities to treatment or control, while in the second stage a subset of individuals are offered treatment based on their income.'", "caption": "Section 16.5.3, p. 313 (PDF p. 25), right column: the description of the Progresa design, with “municipalities” marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N009.png" } ], "formal_release": null, "formal_statements": [], "headline": "The Progresa evaluation analyzed by Angelucci and De Giorgi (2009) randomized villages, not municipalities.", "id": "P045-N009", "location": "Section 16.5.3, p. 313 (PDF p. 25), right column, the sentences on Angelucci and De Giorgi (2009)", "pdf_page": 25, "printed_page": "313", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Similarly, Angelucci and De Giorgi (2009) analyze the indirect effects of the cash transfer program Progresa on consumption. In this case, only the first stage is randomized, assigning \\hl{municipalities} to treatment or control, while in the second stage a subset of individuals are offered treatment based on their income.", "corrected": "Similarly, Angelucci and De Giorgi (2009) analyze the indirect effects of the cash transfer program Progresa on consumption. In this case, only the first stage is randomized, assigning \\hl{villages (localities)} to treatment or control, while in the second stage a subset of individuals are offered treatment based on their income.", "evidence": "A check against the cited study. The abstract of the published article (\\emph{American Economic Review} 99(1): 486--508) says that the authors ``use a village-level randomization''. The version of April 2006 (UCL Discussion Papers in Economics 06-01) states that ``The randomization is conducted at the village level, with 186 villages randomized out'', and its Figure~1 shows 506 rural villages, 320 randomized in and 186 randomized out. The design section of the published article was not read, and the counts are those of the 2006 version.", "problem": "The unit of randomization is misnamed. In the cited study the randomization is at the level of the village (locality): of 506 rural villages, 320 were randomized in and 186 were randomized out. In the 2006 version of the study, municipalities appear in the description of the wider program, not as the unit that was randomized.", "what_survives": "The point of the example is unaffected: geographic groups are randomized in the first stage, and treatment within them follows eligibility by income. The chapter's conclusion about what this design identifies does not depend on the name of the unit." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N009", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Two entries of the reference list, four lines each. A red box marks 'arXiv:1709.0963v1' at the end of the third line of the entry 'Taylor, Sean J., and Dean Eckles. 2017. Randomized Experiments to Detect and Estimate Social Influence in Networks.' The next entry is Tchetgen and VanderWeele 2012.", "caption": "References, p. 319 (PDF p. 31), left column: the entry for Taylor and Eckles (2017), with the arXiv identifier marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N010.png" } ], "formal_release": null, "formal_statements": [], "headline": "The arXiv identifier in the reference to Taylor and Eckles (2017) is malformed: it has four digits after the point, and identifiers of September 2017 have five.", "id": "P045-N010", "location": "References, p. 319 (PDF p. 31), left column, the entry for Taylor and Eckles (2017)", "pdf_page": 31, "printed_page": "319", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Taylor, Sean J., and Dean Eckles. 2017. ``Randomized Experiments to Detect and Estimate Social Influence in Networks.'' \\hl{\\emph{arXiv:1709.0963v1}} \\emph{[cs.SI]}.", "corrected": "Taylor, Sean J., and Dean Eckles. 2017. ``Randomized Experiments to Detect and Estimate Social Influence in Networks.'' \\emph{arXiv:1709.0963\\hl{6}v1} \\emph{[cs.SI]}.", "evidence": "I checked the printed identifier against \\href{https://info.arxiv.org/help/arxiv_identifier.html}{arXiv's identifier format}. The chapter's other identifiers of this period, 1809.09561 and 1711.06399, have five digits after the point. I then compared the authors, title, year and version with \\href{https://arxiv.org/abs/1709.09636v1}{arXiv's entry}, which gives 1709.09636v1 [cs.SI]. I did not test whether the malformed string resolves on arXiv.", "problem": "An arXiv identifier issued from January 2015 on has the form YYMM.NNNNN, with five digits after the point and an optional version suffix. The printed identifier has four digits after the point and is not a valid identifier.", "what_survives": "The authors, title and year identify the work, and the chapter's discussion of it is unaffected." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N010", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": { "class": "error", "severity": "Moderate" }, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the foot of page 298, in two columns. The left column has ten lines that begin 'The estimator tau-hat HT of d_k, d_l is unbiased when it is estimated with pi_i(d_k) rather than pi-hat_i(d_k)'; a red box marks its last line, 'These are conservative approximations of'. The right column continues 'the exact variances that are guaranteed to have nonnegative bias relative to the variance of the randomization distribution of the estimators'; red boxes mark the words 'guaranteed to' at the end of its first line and 'have nonnegative bias' at the start of its second. The right column goes on with 'Because unbiased estimators for Var of y-hat-T HT of d_k are only identified when the joint exposure probabilities of every pair of units are positive' and ends with 'because the Cov of the two estimated totals is always'.", "caption": "Section 16.4.2, p. 298 (PDF p. 10), last paragraph, in both columns. The sentence begins on the marked last line of the left column, “These are conservative approximations of”, and continues at the top of the right column, where “guaranteed to have nonnegative bias” is marked.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016.png" }, { "alt": "Crop of five lines from the top of the left column of page 299. The text reads 'unidentified, we use an approximation. Both of these corrections contribute to the nonnegative bias of the variance estimator. We also implement the constant effects variance estimator derived in Aronow'. Red boxes mark the two lines of the sentence 'Both of these corrections contribute to the nonnegative bias of the variance estimator.'", "caption": "Section 16.4.2, p. 299 (PDF p. 11), top of the left column: the end of the paragraph. The marked sentence says that the two corrections contribute to the nonnegative bias of the variance estimator.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016-p299.png" }, { "alt": "Crop of six lines from the right column of page 297. The text reads 'In cases where the size of Omega is high, and therefore we use sampling from Omega to obtain estimates pi-hat_i, those estimates are used in place of the true pi_i values. A Horvitz-Thompson estimator of the average unit-level causal effect of exposure k versus l, tau(d_k, d_l), is therefore'. Red boxes mark the words 'those estimates are used in place of the true' and, on the next line, 'pi_i values.'", "caption": "Section 16.4.2, p. 297 (PDF p. 9), right column, below the display of the Horvitz–Thompson estimator of a total. The marked words say that estimated exposure probabilities are used in place of the true ones.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016-p297.png" }, { "alt": "A chart titled 'With simulated exposure probabilities, the expectation of the variance estimator is below the variance that it estimates', with four rows and a horizontal axis of variance from 0.40 to 0.56. Blue vertical ticks mark the variance of the Horvitz-Thompson estimator of the contrast and orange dots mark the expectation of its variance estimator; a grey line joins the two. Row 'Simulated; the 66 samples equally likely': dot at 0.447, tick at 0.488, label 'bias minus 0.0406'. Row 'Simulated; the omitted assignment is within one clique (18 samples)': dot at 0.421, tick at 0.487, label 'bias minus 0.0660'. Row 'Simulated; the omitted assignment is across two cliques (48 samples)': dot at 0.457, tick at 0.488, label 'bias minus 0.0309'. Row 'Exact probabilities': the dot lies on the tick at 0.508, label 'no bias'.", "caption": "The finite-sample counterexample of this note: twelve units in three disjoint cliques of four, two units treated, all potential outcomes equal to 1, contrast $(d_{01},d_{00})$, exposure probabilities simulated from 65 of the 66 assignments. In each row the tick is the variance of the Horvitz–Thompson estimator of the contrast, the dot is the expectation of its variance estimator, and the label is the expectation minus the variance. In the first row the moments are over the sample of 65 assignments and the realized assignment. In the second and third rows the sample is fixed, and the assignment that it omits treats two units of one clique or one unit in each of two cliques. The values are exact fractions from the enumeration, shown as decimals. The horizontal axis starts at 0.40.", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N016-variance-bias.png" } ], "formal_release": null, "formal_statements": [ { "id": "P045-T32", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T32" }, { "id": "P045-T33", "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T33" } ], "headline": "The guarantee is exact with true marginal and joint exposure probabilities. The context clearly treats Monte Carlo probabilities as numerical approximations; the guarantee should state this scope explicitly.", "id": "P045-N016", "location": "Section 16.4.2, last paragraph of p. 298 (PDF p. 10): the sentence that begins “These are conservative approximations” on the last line of the left column and continues in the right column; and the sentence “Both of these corrections …” at the top of p. 299 (PDF p. 11)", "pdf_page": 10, "printed_page": "298", "reaches": [], "related_entries": [ { "id": "P044-N005", "url": "https://pmaronow.github.io/llm-audit/works/P044/#P044-N005", "work_id": "P044" } ], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [ "P045-T32", "P045-T33" ], "text": { "as_printed": "The estimator $\\widehat{\\tau}_{HT}(d_k,d_l)$ is unbiased when it is estimated with $\\pi_i(d_k)$ rather than $\\widehat{\\pi}_i(d_k)$. As mentioned above, when estimated with the latter, the estimator is not unbiased, but the bias becomes negligible with a sufficiently large number of replicates $R$. We implement variance estimators for $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ and $\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]$ as derived in equation 11 of Aronow and Samii (2017). These are conservative approximations of the exact variances that are \\hl{guaranteed to have nonnegative bias} relative to the variance of the randomization distribution of the estimators. Because unbiased estimators for $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ are only identified when the joint exposure probabilities of every pair of units are positive, it is necessary to add a correction term to $\\widehat{\\operatorname{Var}}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$. Second, because the $\\operatorname{Cov}\\bigl[\\widehat{y^T}_{HT}(d_k),\\widehat{y^T}_{HT}(d_l)\\bigr]$ is always unidentified, we use an approximation. Both of these corrections contribute to \\hl{the nonnegative bias} of the variance estimator.", "corrected": "The variance estimators for $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ and $\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]$ are guaranteed to have nonnegative bias \\hl{with the exact marginal and joint exposure probabilities of the design}, provided that $\\pi_i(d_k)>0$ and $\\pi_i(d_l)>0$ for every unit $i$. With these probabilities,\n\\[\n\\begin{aligned}\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]\\Bigr]&\\;\\ge\\;\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr],\\\\\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]\\Bigr]&\\;\\ge\\;\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)].\n\\end{aligned}\n\\]\n\\hl{With exact probabilities,} both corrections for unidentified outcome products contribute to the nonnegative bias. Using estimated exposure probabilities approximates this result when Monte Carlo error is small; \\hl{a finite number $R$ of simulated assignments need not preserve the exact nonnegative-bias guarantee}.", "intended_meaning": "A variance estimator has nonnegative bias if its expectation over the randomization is at least the variance that it estimates. The guarantee holds when the variance estimators are computed with the exact marginal and joint exposure probabilities of the design and every unit has positive probability of the exposures concerned. This is the natural intended meaning. Page~294 explains that Monte Carlo probabilities approximate the exact probabilities with arbitrary precision and discusses keeping their bias small. Immediately before the guarantee on p.~298, the chapter distinguishes $\\pi_i(d_k)$ from $\\widehat\\pi_i(d_k)$ and says that the point estimator's finite-$R$ bias becomes negligible with sufficiently many replicates. Read together, these passages describe the exact-probability result and an accurate numerical approximation to it.\n\nThe examples do use simulated exposure probabilities: 30 assignments drawn without replacement from 45 in the running example (p.~296), and $R=10{,}000$ assignments in Section 16.4.2 (p.~299). Page~297 says that the estimates replace the true probabilities. These finite calculations need not preserve an exact sign guarantee. The example below shows that the expected variance estimate can be below the variance of the contrast computed with those same simulated probabilities. It does not test the numerical accuracy of the chapter's $R=10{,}000$ calculations.\n\nThe distinction follows from the expectation of a product of exposure indicators, which is the exact joint exposure probability. With that probability in the denominator of a positive-joint-probability term, the two cancel. A simulated denominator does not give this cancellation. The corrections for zero joint probabilities control the missing outcome products, not the Monte Carlo error. Adding the exact-probability restriction to the guarantee and to the following sentence about the corrections would make their intended scope explicit. The related qualification for the chapter ``Estimating Causal Effects in Experiments with Network Spillovers'' is listed as \\fid{P044-N005}.\n\nThe evidence is an exact counterexample, a check of the counterexample against the package, and a derivation for exact exposure probabilities.\n\n\\emph{The counterexample.} Twelve units form three disjoint cliques of four: two distinct units are adjacent exactly when they belong to the same clique. Two units are treated, and the 66 assignments are equally likely. The exposure mapping is the chapter's. A unit has exposure $d_{11}$ if it and a neighbor are treated, $d_{10}$ if it is treated and no neighbor is, $d_{01}$ if it is untreated and a neighbor is treated, and $d_{00}$ otherwise. Every potential outcome equals 1, so that the exposure mapping fully characterizes interference and $\\tau(d_{01},d_{00})=0$. For every unit, the exact probabilities of $d_{11}$, $d_{10}$, $d_{01}$ and $d_{00}$ are $3/66$, $8/66$, $27/66$ and $28/66$. The exposure probabilities are simulated from a sample of $R=65$ distinct assignments, drawn uniformly without replacement from the 66 and independently of the realized assignment. The example on p.~296 also draws its 30 assignments without replacement, from 45. There are 66 such samples, one for each omitted assignment.\n\nFix two exposures, and let $c_{ij}$ be the number of sampled assignments under which unit $i$ has the first exposure and unit $j$ has the second. The package estimates the joint probability by $(c_{ij}+\\mathbf{1}\\{i=j\\})/(R+1)$ when the two exposures are the same and by $c_{ij}/R$ when they differ. The marginal estimates are the diagonal entries in the first case (see \\fid{P045-N001}). In the counterexample the simulated probability of $d_{01}$ is $27/66$ for a unit that has exposure $d_{01}$ under the omitted assignment and $28/66$ for the other units. The simulated probability of $d_{00}$ is $28/66$ for a unit that has exposure $d_{00}$ under the omitted assignment and $29/66$ for the other units. For the two contrasted exposures, a simulated joint probability is zero exactly when the exact joint probability is zero. The failure does not come from a pair of exposures that is possible but absent from the sample.\n\nI enumerated the $66\\times 66=4356$ pairs of a sample and a realized assignment in exact rational arithmetic. For each pair I computed $\\widehat{\\tau}_{HT}(d_{01},d_{00})$ and the variance estimator displayed below, with the simulated probabilities in place of the exact ones.\n\n\\begin{audittable}{lrrrrr}{The counterexample. Mean and Variance: of $\\widehat{\\tau}_{HT}(d_{01},d_{00})$. Expectation: of its variance estimator. Bias: the expectation minus the variance. In the first row the 66 samples are equally likely. In the second and third rows the sample is fixed, and the assignment that it omits treats two units of one clique or one unit in each of two cliques. The decimals are exact fractions rounded to six places.}\nExposure probabilities & Samples & Mean & Variance & Expectation & Bias \\\\\nSimulated, all samples & 66 & $-67/53592$ & 0.487658 & 0.447098 & $-0.040560$ \\\\\nSimulated, one clique & 18 & $-89/4872$ & 0.487267 & 0.421232 & $-0.066035$ \\\\\nSimulated, two cliques & 48 & $25/4872$ & 0.487655 & 0.456797 & $-0.030858$ \\\\\nExact & & $0$ & 0.508104 & 0.508104 & $0$ \\\\\n\\end{audittable}\n\nIn the first row the moments are over the sample and the realized assignment, and the exact values are\n\\[\n\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_{01},d_{00})]=\\frac{113448938731}{232640299584},\n\\qquad\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_{01},d_{00})]\\Bigr]=\\frac{1040129476423}{2326402995840}.\n\\]\nTheir difference is $-953130413/23499020160$, and the expectation is about 8.3 percent below the variance. In the second and third rows the moments are over the realized assignment. The bias is negative for each of the 66 samples. With the exact probabilities in the same formulas, the variance and the expectation of the variance estimator both equal $6050/11907$.\n\n\\emph{The package.} I ran the unchanged functions of two versions of the \\texttt{interference} package (commits \\texttt{a5b8dfb} and \\texttt{c3fc7b3}) for the exposure mapping, the exposure probabilities, the variance and the covariance. For each of the 4356 pairs, I compared two quantities with the enumeration: the contrast computed with the package's simulated probabilities, and the variance estimate that the package returns. In both versions they agree up to floating-point rounding.\n\n\\emph{Derivation for exact exposure probabilities.} Let $\\pi_i(d)=\\Pr(D_i=d)$ and $\\pi_{ij}(d,d')=\\Pr(D_i=d,\\,D_j=d')$, with $\\pi_{ii}(d,d)=\\pi_i(d)$, and write $I_i(d)=\\mathbf{1}\\{D_i=d\\}$. Let $\\widehat W_k$ denote the variance estimator $\\widehat{\\operatorname{Var}}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ and $\\widehat C_{kl}$ the estimator of $\\operatorname{Cov}\\bigl[\\widehat{y^T}_{HT}(d_k),\\widehat{y^T}_{HT}(d_l)\\bigr]$. The package computes\n\\[\n\\begin{aligned}\n\\widehat W_k={}&\\sum_{i,j:\\,\\pi_{ij}(d_k,d_k)>0}I_i(d_k)\\,I_j(d_k)\\,\\frac{\\pi_{ij}(d_k,d_k)-\\pi_i(d_k)\\pi_j(d_k)}{\\pi_{ij}(d_k,d_k)}\\,\\frac{Y_iY_j}{\\pi_i(d_k)\\pi_j(d_k)}\\\\\n&+\\sum_{i\\neq j:\\,\\pi_{ij}(d_k,d_k)=0}\\Bigl(\\frac{I_i(d_k)\\,Y_i^2}{2\\pi_i(d_k)}+\\frac{I_j(d_k)\\,Y_j^2}{2\\pi_j(d_k)}\\Bigr),\\\\\n\\widehat C_{kl}={}&\\sum_{i,j:\\,\\pi_{ij}(d_k,d_l)>0}I_i(d_k)\\,I_j(d_l)\\,\\frac{\\pi_{ij}(d_k,d_l)-\\pi_i(d_k)\\pi_j(d_l)}{\\pi_{ij}(d_k,d_l)}\\,\\frac{Y_iY_j}{\\pi_i(d_k)\\pi_j(d_l)}\\\\\n&-\\sum_{i,j:\\,\\pi_{ij}(d_k,d_l)=0}\\Bigl(\\frac{I_i(d_k)\\,Y_i^2}{2\\pi_i(d_k)}+\\frac{I_j(d_l)\\,Y_j^2}{2\\pi_j(d_l)}\\Bigr),\n\\end{aligned}\n\\]\nand $\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]=N^{-2}\\bigl(\\widehat W_k+\\widehat W_l-2\\,\\widehat C_{kl}\\bigr)$. The sums are over ordered pairs of units. The pairs with $i=j$ belong to the first sum of $\\widehat W_k$ and to the second sum of $\\widehat C_{kl}$.\n\nSuppose that the probabilities are exact and that $\\pi_i(d_k)>0$ and $\\pi_i(d_l)>0$ for every unit $i$. Since $\\mathbb{E}\\bigl[I_i(d)\\,I_j(d')\\bigr]=\\pi_{ij}(d,d')$, each term of a sum over positive joint probabilities has expectation equal to the corresponding term of $\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]$ or of the covariance. For a pair with $\\pi_{ij}(d_k,d_k)=0$, the term of the variance is $-y_i(d_k)\\,y_j(d_k)$, and the second sum of $\\widehat W_k$ has expectation $\\bigl(y_i(d_k)^2+y_j(d_k)^2\\bigr)/2$ for that pair. For a pair with $\\pi_{ij}(d_k,d_l)=0$, the term of the covariance is $-y_i(d_k)\\,y_j(d_l)$, and the second sum of $\\widehat C_{kl}$ has expectation $-\\bigl(y_i(d_k)^2+y_j(d_l)^2\\bigr)/2$. Collecting terms, I obtain\n\\[\n\\mathbb{E}\\bigl[\\widehat W_k\\bigr]-\\operatorname{Var}\\bigl[\\widehat{y^T}_{HT}(d_k)\\bigr]=\\frac{1}{2}\\sum_{i\\neq j:\\,\\pi_{ij}(d_k,d_k)=0}\\bigl(y_i(d_k)+y_j(d_k)\\bigr)^2\\;\\ge\\;0\n\\]\nand\n\\[\n\\begin{aligned}\n\\mathbb{E}\\Bigl[\\widehat{\\operatorname{Var}}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]\\Bigr]-\\operatorname{Var}\\,[\\widehat{\\tau}_{HT}(d_k,d_l)]={}&\\frac{1}{2N^2}\\sum_{i\\neq j:\\,\\pi_{ij}(d_k,d_k)=0}\\bigl(y_i(d_k)+y_j(d_k)\\bigr)^2\\\\\n&+\\frac{1}{2N^2}\\sum_{i\\neq j:\\,\\pi_{ij}(d_l,d_l)=0}\\bigl(y_i(d_l)+y_j(d_l)\\bigr)^2\\\\\n&+\\frac{1}{N^2}\\sum_{i,j:\\,\\pi_{ij}(d_k,d_l)=0}\\bigl(y_i(d_k)-y_j(d_l)\\bigr)^2\\;\\ge\\;0.\n\\end{aligned}\n\\]\nIn the counterexample no two units have joint probability zero of $d_{01}$ or of $d_{00}$, and every difference $y_i(d_{01})-y_j(d_{00})$ is zero. The last display then implies that the bias with exact probabilities is zero, in agreement with the last row of the table. In the first step of the derivation, I used that the probabilities inside the variance estimator are those of the distribution of the realized assignment. With simulated probabilities, the expectation of $I_i(d)\\,I_j(d')$ is still the exact joint probability, the denominator is the simulated one, and the identities do not hold.\n\nThe exact-probability result, exposure mapping, estimands and statement about the point estimator's bias are unaffected. The counterexample concerns the Horvitz--Thompson variance estimator of a contrast. The applications that follow take the maximum of this estimator and the constant-effects variance estimator (p.~299), and I did not examine that maximum. No published simulation value is shown to be wrong, and no failure of confidence-interval coverage or of an asymptotic statement is established. A large finite $R$ alone supplies no general exact sign guarantee: convergence to a nonnegative bias limit need not imply nonnegative bias at finite $R$. The finite calculation therefore needs a precision qualification; this clarifies the intended exact-probability result." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N016", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of eight lines from the right column of page 307. The text reads 'only a few randomized experiments. These typically rely on imposing partial interference assumptions (see below) so that outcomes of, for example, different, villages or schools can be treated as independent observations. Some studies used random or haphazard assignment of treatment to analyze what types of units produce the largest spillovers.' A red box marks the fifth line, 'can be treated as independent observations.'", "caption": "Section 16.4.7, p. 307 (PDF p. 19), right column, paragraph on seeding. The marked line ends the sentence on partial interference: “can be treated as independent observations.”", "url": "https://pmaronow.github.io/llm-audit/works/P045/exhibits/P045-N018.png" } ], "formal_release": null, "formal_statements": [], "headline": "Partial interference rules out spillovers between villages or schools, which can then be analyzed as separate groups under the actual randomization; it does not by itself make their outcomes statistically independent.", "id": "P045-N018", "location": "Section 16.4.7, p. 307 (PDF p. 19), right column, paragraph on seeding: the sentence that begins “These typically rely”", "pdf_page": 19, "printed_page": "307", "reaches": [], "related_entries": [ { "id": "P037-E004", "url": "https://pmaronow.github.io/llm-audit/works/P037/#P037-E004", "work_id": "P037" } ], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "These typically rely on imposing partial interference assumptions (see below) so that outcomes of, for example, different, villages or schools \\hl{can be treated as independent observations}.", "corrected": "These typically rely on imposing partial interference assumptions (see below) \\hl{to rule out spillovers between villages or schools and analyze them as separate groups, with inference based on the actual randomization law. In the fixed-population framework used in this chapter, mutually independent complete group assignment vectors are sufficient to guarantee mutually independent group outcome vectors; partial interference alone does not guarantee that independence.}", "intended_meaning": "The sentence says that the randomized experiments on seeding typically assume partial interference. Under this assumption, the treatment of one village or school does not affect the outcomes of another. I read ``independent observations'' here as meaning that the groups can be analyzed as separate units, with inference based on the actual randomization. This meaning is correct. Read literally, the sentence says that the outcomes of different groups are statistically independent. Partial interference alone does not imply that they are independent. It restricts the potential outcomes, which the chapter takes as fixed, and not the random assignment. An example with the design of p.~310 of the chapter shows that the outcomes of two groups can be dependent. In this design, 18 units are divided into six groups of three. Exactly three of the groups receive the saturation $\\psi$, under which two of the three units of a group are treated. The other three groups receive the saturation $\\phi$, under which one unit of a group is treated. Suppose that the outcome of each unit is $1$ if at least two units of its group are treated and $0$ otherwise. Then partial interference holds, and the mean outcome of a group is $1$ if the group receives $\\psi$ and $0$ otherwise. The mean outcome of each group equals $1$ with probability $1/2$, and the mean outcomes of two given groups both equal $1$ with probability $1/5$, where independence would require $1/4$. I found no estimator, variance formula, simulation or reported number in the chapter that depends on this sentence. The paragraph goes on to cite the experiment of Paluck et al. (2016). The related statement about the schools of that experiment is the subject of \\fid{P037-E004}." }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-N018", "work": { "arxiv_id": "2001.05444", "doi": "10.1017/9781108777919.021", "title": "Spillover Effects in Experimental Data", "url": "https://pmaronow.github.io/llm-audit/works/P045/" }, "work_id": "P045" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P045.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:148b8833b5980fc04360588dd0af5cb1290526c8c54b7e2c004d6bc765c0906b", "conditional_on": [], "credit_reason": "Combining this row with R257-01 gives the arm numbers as well as the direct, indirect and total failure. All conditioning events have positive probability; the overall contrast remains unbiased.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.bernoulli_ratio_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L823", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:148b8833b5980fc04360588dd0af5cb1290526c8c54b7e2c004d6bc765c0906b", "conditional_on": [], "credit_reason": "Witness for the direct-unbiasedness clause, not a full statement about all four estimators; partial and stratified interference hold by the explicit peer-count definition.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.bernoulli_direct_failure", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L728", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:148b8833b5980fc04360588dd0af5cb1290526c8c54b7e2c004d6bc765c0906b", "conditional_on": [], "credit_reason": "This is the conditional direct-moment component of the same valid witness; the zero target is exported by the companion row R257-02.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.bernoulli_direct_failure_means", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L691", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:148b8833b5980fc04360588dd0af5cb1290526c8c54b7e2c004d6bc765c0906b", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'AronowAudit/P045/P045_Spillover.lean:586 states the eight conditional arm means in T1(iii), the two both-arm domain probabilities, and the constant count-arm contrast. The binomial4 mass at line 543 is the treated-count distribution of the four-unit Bernoulli design, as expressed by bernoulli4_count_law at line 568; co_members_arm_mean at line 495 identifies countArm with the actual mean on each nonempty arm. Passing to the count distribution loses no cases or conditioning information for this component, and every conditioning event has positive probability.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.bernoulli_count_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L580", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:148b8833b5980fc04360588dd0af5cb1290526c8c54b7e2c004d6bc765c0906b", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T1" }, "html": { "body": "

p. 310 (PDF p. 22); definitions on pp. 308–309 (PDF pp. 20–21). The four-unit, two-group co-member schedule refutes Bernoulli arm-ratio unbiasedness. It has the stated individual, group and population estimands, pointwise arm formulas, positive conditioning domains and every own-arm and both-arm conditional expectation. Direct, indirect and total ratios fail as quantified, whereas the overall estimator is unbiased. The usual measure-theoretic conditions are assumed; here all laws and sums are finite.

", "headline": "Section 16.5.1: ‘Given Bernoulli assignment with probability ψ\\psiψ, an unbiased estimator for 1ng∑i=1ngyig(z;ψ)\\frac{1}{n_g}\\sum_{i=1}^{n_g}y_{ig}(z;\\psi)ng​1​∑i=1ng​​yig​(z;ψ) […] is’ y^g(z;ψ)\\widehat y_g(z;\\psi)y​g​(z;ψ), and the unbiasedness claimed for τ^D\\widehat\\tau^DτD, τ^I\\widehat\\tau^IτI and τ^To\\widehat\\tau^{To}τTo (p. 310); the witness is the adjudicator's construction and is not in the paper", "location": "p. 310 (PDF p. 22); definitions on pp. 308–309 (PDF pp. 20–21)" }, "id": "P045-T1", "text": { "body": "p. 310 (PDF p. 22); definitions on pp. 308–309 (PDF pp. 20–21). The four-unit, two-group co-member schedule refutes Bernoulli arm-ratio unbiasedness. It has the stated individual, group and population estimands, pointwise arm formulas, positive conditioning domains and every own-arm and both-arm conditional expectation. Direct, indirect and total ratios fail as quantified, whereas the overall estimator is unbiased. The usual measure-theoretic conditions are assumed; here all laws and sums are finite.", "headline": "Section 16.5.1: ‘Given Bernoulli assignment with probability $\\psi$, an unbiased estimator for $\\frac{1}{n_g}\\sum_{i=1}^{n_g}y_{ig}(z;\\psi)$ […] is’ $\\widehat y_g(z;\\psi)$, and the unbiasedness claimed for $\\widehat\\tau^D$, $\\widehat\\tau^I$ and $\\widehat\\tau^{To}$ (p. 310); the witness is the adjudicator's construction and is not in the paper", "location": "p. 310 (PDF p. 22); definitions on pp. 308–309 (PDF pp. 20–21)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T1" }, { "declarations": [ { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "conditional_on": [], "credit_reason": "No verdict change after reading record.json. The combined identities have the required quantifiers; contrast conclusions follow by substitution and finite expectation linearity, without an additional statistical hypothesis.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.marginal_estimator_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1441", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.bernoulli_group_ht_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L976", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "conditional_on": [], "credit_reason": "Harmless algebraic generalization. The probability interpretation and the expectation theorem require the interior probability domain, which R257-06 explicitly imposes.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.co_members_group_ht_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1002", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.co_members_group_ht_expectation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1031", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "conditional_on": [], "credit_reason": "The aggregate correction supplies the one-group overall identity used in the Solution.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.overall_group_unbiased", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1496", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "conditional_on": [], "credit_reason": "The aggregate correction supplies the uniform selection identity specialized in the Solution.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.selected_mean_expect", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1233", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.cr_group_arm_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1363", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:193800aef589c17163cf56aa64c417cb37a0477417a466a2f50e37e409b7eaab", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T2", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T2" }, "html": { "body": "

p. 310 (PDF p. 22). Bernoulli arm ratios can be biased even conditional on a positive denominator. HT group means, uniform first-stage averages and their direct, indirect and total contrasts are unbiased for the Bernoulli marginal targets. Fixed-count assignment instead makes the original ratios unbiased for that design's marginal targets, which differ under interference. Overall means remain unbiased for arbitrary local laws. The joint results require only the stated conditional group marginals, allow cross-group dependence, and include empty Bernoulli arms. All four-unit formulas and the numerical HT contrast expectations are retained. The usual measure-theoretic conditions are assumed; all spaces here are finite.

", "headline": "Section 16.5.1, the unbiasedness statements for the marginal-effect estimators (p. 310), corrected: the within-arm means can be biased under Bernoulli assignment; unconditional Horvitz–Thompson means are unbiased for the Bernoulli targets, and the within-arm means are unbiased under fixed-count assignment for that design's own marginal outcomes", "location": "p. 310 (PDF p. 22)" }, "id": "P045-T2", "text": { "body": "p. 310 (PDF p. 22). Bernoulli arm ratios can be biased even conditional on a positive denominator. HT group means, uniform first-stage averages and their direct, indirect and total contrasts are unbiased for the Bernoulli marginal targets. Fixed-count assignment instead makes the original ratios unbiased for that design's marginal targets, which differ under interference. Overall means remain unbiased for arbitrary local laws. The joint results require only the stated conditional group marginals, allow cross-group dependence, and include empty Bernoulli arms. All four-unit formulas and the numerical HT contrast expectations are retained. The usual measure-theoretic conditions are assumed; all spaces here are finite.", "headline": "Section 16.5.1, the unbiasedness statements for the marginal-effect estimators (p. 310), corrected: the within-arm means can be biased under Bernoulli assignment; unconditional Horvitz–Thompson means are unbiased for the Bernoulli targets, and the within-arm means are unbiased under fixed-count assignment for that design's own marginal outcomes", "location": "p. 310 (PDF p. 22)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T2" }, { "declarations": [ { "claim_revision_id": "catalog:cb73eb13bcb8a7224a3a60b66e7a0f9c28d9ffc21bf7a6ce9b387e74c0fde6fc", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P045/P045_Spillover.lean:1859 gives the additive, dilated, and upward-bias moments advertised by this row and T3. selectedLaw at line 1056 gives unused groups the same treatment count as the selected groups, but directGroup and directVarianceEstimate at lines 1525 and 1538 depend only on selected groups. Summing out the normalized unused draws gives exactly the corresponding one-saturation statistic law in T3's mixed-saturation experiment, so none of the asserted moments changes. Calling this augmented law the full toy experiment is imprecise, rather than a material narrowing of the moment claims.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.singleton_variance_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1853", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:cb73eb13bcb8a7224a3a60b66e7a0f9c28d9ffc21bf7a6ce9b387e74c0fde6fc", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.singleton_group_finite_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1814", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:cb73eb13bcb8a7224a3a60b66e7a0f9c28d9ffc21bf7a6ce9b387e74c0fde6fc", "conditional_on": [], "credit_reason": "The selected six-group counterexample already exports the estimator moments; this general identical-group calculation is a helper.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.identical_group_selected_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1756", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:cb73eb13bcb8a7224a3a60b66e7a0f9c28d9ffc21bf7a6ce9b387e74c0fde6fc", "conditional_on": [], "credit_reason": "The additive local and full counterexample rows already export the retained moments; the general affine formulas are not part of the retained statement.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.three_unit_affine_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1833", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:cb73eb13bcb8a7224a3a60b66e7a0f9c28d9ffc21bf7a6ce9b387e74c0fde6fc", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T3" }, "html": { "body": "

p. 310 (PDF p. 22) and p. 311 (PDF p. 23). In the actual six-group joint toy design, all 14,580 supported outcomes are equally likely. The additive schedule has direct estimand and estimator mean one, but variance 1/2 and expected singleton-drop variance estimate 1/3 at either regime. The singleton and direct-statistic distributions, two-unit sample variance and expected between-group sample variance give the stated bias -1/6 and ratio 2/3. The dilated schedules are also downward biased, and the zero-control schedule gives upward bias. The dropping convention is part of the specified estimator. The usual measure-theoretic conditions are assumed; all laws here are finite.

", "headline": "Section 16.5.1: ‘When stratified interference holds, their proposed variance estimators are unbiased if unit causal effects are additive […] and otherwise positively biased’ (p. 310), for the direct-effect variance estimates of the toy example (p. 311: var_direct_psi_hat, var_direct_phi_hat), whose design has a one-unit arm in every group; the witness is the validator's and is not in the paper", "location": "p. 310 (PDF p. 22) and p. 311 (PDF p. 23)" }, "id": "P045-T3", "text": { "body": "p. 310 (PDF p. 22) and p. 311 (PDF p. 23). In the actual six-group joint toy design, all 14,580 supported outcomes are equally likely. The additive schedule has direct estimand and estimator mean one, but variance 1/2 and expected singleton-drop variance estimate 1/3 at either regime. The singleton and direct-statistic distributions, two-unit sample variance and expected between-group sample variance give the stated bias -1/6 and ratio 2/3. The dilated schedules are also downward biased, and the zero-control schedule gives upward bias. The dropping convention is part of the specified estimator. The usual measure-theoretic conditions are assumed; all laws here are finite.", "headline": "Section 16.5.1: ‘When stratified interference holds, their proposed variance estimators are unbiased if unit causal effects are additive […] and otherwise positively biased’ (p. 310), for the direct-effect variance estimates of the toy example (p. 311: var_direct_psi_hat, var_direct_phi_hat), whose design has a one-unit arm in every group; the witness is the validator's and is not in the paper", "location": "p. 310 (PDF p. 22) and p. 311 (PDF p. 23)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T3" }, { "declarations": [ { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "This judges the corrected T4(a)-(c), not the unrestricted printed variance prose; its minimum arm sizes and groupwise equality condition are already in the target.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.direct_variance_correction_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/Wave1/Gap/GLP041P042P045P045B2.lean#L485", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "This proves the specified component T4(d); it is not credited as a proof of T4(a)-(c) or as a claim printed verbatim in the chapter.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.singleton_drop_bias_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P041/Wave1/Gap/GP041P042P045B2.lean#L369", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "This proves the specified specialization of T4(d), for every real baseline; it does not certify a particular random baseline dataset.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.singleton_drop_dilated_bias_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P041/Wave1/Gap/GP041P042P045B2.lean#L387", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "The retained toy theorem directly states undefined required variance on every admissible toy assignment.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.singleton_sample_variance_undefined", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1967", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P045/P045_Spillover.lean:1566 covers both allowed treatment counts, every selected three-group set, every supported selected-group assignment, and arbitrary real outcome schedules. Every three-unit group with treatment count one or two has a singleton arm, so the domain guard in requiredDirectVariance at line 1555 returns none, matching T4's repaired toy-output component. This row claims that mathematical undefinedness, not execution of the R edit or issuance of a warning, so their absence is not an omitted conclusion of this component.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.toy_required_direct_variance_undefined", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1560", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "Nonnegativity is already exported by the correction and singleton bias rows.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.pop_var_nonnegative", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1580", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "The correction row already exports the equality characterization.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.pop_var_zero_iff_constant", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1591", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "The selected dilated-bias row already exports the scaled variance calculations.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.pop_var_scale", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1623", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "Its mean identity is included in the complete direct-variance correction row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.direct_group_cr_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1905", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "conditional_on": [], "credit_reason": "Its selected mean identity is included in the complete direct-variance correction row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.selected_direct_cr_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L1952", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:543f7c17433612f68feb416d55e8c9c409e61c5a6637934a87bf6b8c01441681", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T4", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T4" }, "html": { "body": "

p. 310 (PDF p. 22); toy example, pp. 310–311 (PDF pp. 22–23). The corrected direct variance formula is conservative on its proper domain, with the stated equality characterization and singleton-drop bias formulas.

", "headline": "Section 16.5.1, the property of the direct-effect variance estimator (p. 310), corrected: it holds when every arm of every group has at least two units; with a one-unit arm the within-arm variance is undefined and the estimate computed in the toy example is not covered", "location": "p. 310 (PDF p. 22); toy example, pp. 310–311 (PDF pp. 22–23)" }, "id": "P045-T4", "text": { "body": "p. 310 (PDF p. 22); toy example, pp. 310–311 (PDF pp. 22–23). The corrected direct variance formula is conservative on its proper domain, with the stated equality characterization and singleton-drop bias formulas.", "headline": "Section 16.5.1, the property of the direct-effect variance estimator (p. 310), corrected: it holds when every arm of every group has at least two units; with a one-unit arm the within-arm variance is undefined and the estimate computed in the toy example is not covered", "location": "p. 310 (PDF p. 22); toy example, pp. 310–311 (PDF pp. 22–23)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T4" }, { "declarations": [ { "claim_revision_id": "catalog:b30e44d9635b6212f566067d52456910982cd154b06a047f87a69264e0253b18", "conditional_on": [], "credit_reason": "This is a witness against the package event interpretation and its HT contrast. It neither refutes the correctly worded all-control definition nor asserts that the chapter table entries are wrong.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.full_neighborhood_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2018", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:b30e44d9635b6212f566067d52456910982cd154b06a047f87a69264e0253b18", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T5", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T5" }, "html": { "body": "

p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 104–121. The stipulated all-control event is the complement of all-treated, and differs from the documented all-untreated event. The two-vertex count/event tables, probabilities and all four HT values for the linear and threshold schedules exhibit the exact failure and the case where the expectation does not move. The usual measure-theoretic conditions are assumed; all spaces here are finite.

", "headline": "Section 16.4.6 and Table 16.3, ‘the causal estimand of full neighborhood exposure, τ(d1,d0)\\tau(d_{\\mathbf 1},d_{\\mathbf 0})τ(d1​,d0​)’ (p. 305), against the package's only full-neighbourhood exposure mapping (make_exposure_map_full_neighborhood), whose ‘all_control’ indicator is the complement of ‘all_treat’; the witness is the validators' and is not in the paper", "location": "p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 104–121 (identical in both snapshots)" }, "id": "P045-T5", "text": { "body": "p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 104–121. The stipulated all-control event is the complement of all-treated, and differs from the documented all-untreated event. The two-vertex count/event tables, probabilities and all four HT values for the linear and threshold schedules exhibit the exact failure and the case where the expectation does not move. The usual measure-theoretic conditions are assumed; all spaces here are finite.", "headline": "Section 16.4.6 and Table 16.3, ‘the causal estimand of full neighborhood exposure, $\\tau(d_{\\mathbf 1},d_{\\mathbf 0})$’ (p. 305), against the package's only full-neighbourhood exposure mapping (make_exposure_map_full_neighborhood), whose ‘all_control’ indicator is the complement of ‘all_treat’; the witness is the validators' and is not in the paper", "location": "p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 104–121 (identical in both snapshots)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T5" }, { "declarations": [ { "claim_revision_id": "catalog:500c8657057b336a080ff07dab0fa83405059f1345aa74eb5891810a2af2a8b3", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.full_neighborhood_events", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2050", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:500c8657057b336a080ff07dab0fa83405059f1345aa74eb5891810a2af2a8b3", "conditional_on": [], "credit_reason": "The complement event has positive probability automatically because it contains the positive-probability all-control event. The outcome assumptions match T6(e) and do not constrain mixed-neighborhood outcomes.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.full_neighborhood_ht_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2141", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:500c8657057b336a080ff07dab0fa83405059f1345aa74eb5891810a2af2a8b3", "conditional_on": [], "credit_reason": "The conventional conditional-mean definition reduces this auxiliary claim to a finite algebraic identity; the main statistical theorem separately supplies genuine designs and positive events.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.event_ht_conditional", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L233", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:500c8657057b336a080ff07dab0fa83405059f1345aa74eb5891810a2af2a8b3", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T6", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T6" }, "html": { "body": "

p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 112–121. Closed-neighborhood event characterizations justify the corrected extreme-event HT contrast and the complement-event conditional-mean formula.

", "headline": "Full-neighbourhood exposure conditions (Section 16.4.6, p. 305; package function make_exposure_map_full_neighborhood), corrected: ‘all control’ is the event that the unit and all its first-degree neighbours are untreated", "location": "p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 112–121" }, "id": "P045-T6", "text": { "body": "p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 112–121. Closed-neighborhood event characterizations justify the corrected extreme-event HT contrast and the complement-event conditional-mean formula.", "headline": "Full-neighbourhood exposure conditions (Section 16.4.6, p. 305; package function make_exposure_map_full_neighborhood), corrected: ‘all control’ is the event that the unit and all its first-degree neighbours are untreated", "location": "p. 305 (PDF p. 17); staged package R/make_exposure_map_AS.R, lines 112–121" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T6" }, { "declarations": [ { "claim_revision_id": "catalog:4abcb42da3345ff6ec66603bab8794c7b045bf5b8419532faa5325372f512e88", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P045/P045_Spillover.lean:2398 states the column-major triangle lists, dimension distinction, 32 asymmetric outputs, exposure changes, and deletion-probability witness advertised by the row. For the specified single-edge input, deleting the only upper-triangle edge before faultyCopy at line 2266 gives exactly the Boolean mixture of the zero matrix and the faulty output, with probabilities q and 1-q. matrixExposure at line 2311 reads columns as required by T7's package convention, so neither the Boolean mixture nor the exposure representation changes the witness.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.faulty_copy_full_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2336", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4abcb42da3345ff6ec66603bab8794c7b045bf5b8419532faa5325372f512e88", "conditional_on": [], "credit_reason": "This is the described T7(ii) witness, not a separate assertion of all T7 enumeration clauses.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.faulty_copy_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2221", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:4abcb42da3345ff6ec66603bab8794c7b045bf5b8419532faa5325372f512e88", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T7", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T7" }, "html": { "body": "

p. 303 (PDF p. 15), Figure 16.5 p. 304 (PDF p. 16); staged R/functions.R, lines 36 and 55. Storage-order copying has the stated triangle-order mismatch from dimension four. The single-edge witness produces exactly the two asserted nonzero entries, is asymmetric, changes both exposures, and 32 of the 64 binary upper triangles fail symmetry. At each deletion probability q in [0,1), the complete output law is (1-q) at the faulty matrix and q at zero; intended deletion instead has (1-q) at the original matrix and q at zero. No other outputs occur. The usual measure-theoretic conditions are assumed; these are finite laws.

", "headline": "Section 16.4.4 and Figure 16.5, ‘a network that is randomly missing varying proportions of ties’ (p. 303), against the package's tie-removal and tie-addition helpers (make_adj_matrix_trunc, make_adj_matrix_add), whose last line copies the upper triangle into the lower triangle in storage order; the witness is the validators' and is not in the paper", "location": "p. 303 (PDF p. 15) and Figure 16.5, p. 304 (PDF p. 16); staged package R/functions.R, lines 36 and 55 (identical in both snapshots)" }, "id": "P045-T7", "text": { "body": "p. 303 (PDF p. 15), Figure 16.5 p. 304 (PDF p. 16); staged R/functions.R, lines 36 and 55. Storage-order copying has the stated triangle-order mismatch from dimension four. The single-edge witness produces exactly the two asserted nonzero entries, is asymmetric, changes both exposures, and 32 of the 64 binary upper triangles fail symmetry. At each deletion probability q in [0,1), the complete output law is (1-q) at the faulty matrix and q at zero; intended deletion instead has (1-q) at the original matrix and q at zero. No other outputs occur. The usual measure-theoretic conditions are assumed; these are finite laws.", "headline": "Section 16.4.4 and Figure 16.5, ‘a network that is randomly missing varying proportions of ties’ (p. 303), against the package's tie-removal and tie-addition helpers (make_adj_matrix_trunc, make_adj_matrix_add), whose last line copies the upper triangle into the lower triangle in storage order; the witness is the validators' and is not in the paper", "location": "p. 303 (PDF p. 15) and Figure 16.5, p. 304 (PDF p. 16); staged package R/functions.R, lines 36 and 55 (identical in both snapshots)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T7" }, { "declarations": [ { "claim_revision_id": "catalog:dd70372fd007f0d185f476410974aa9532b67f146029cbde0cfc6da1f5f59e29", "conditional_on": [], "credit_reason": "This is the specified finite counterexample to the general bias assertion, not a claim to reproduce Figure 16.5; the degree bounds imply positive ratio domains and R258-13 explicitly exports the cell certificate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.nonmonotone_hajek_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2769", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dd70372fd007f0d185f476410974aa9532b67f146029cbde0cfc6da1f5f59e29", "conditional_on": [], "credit_reason": "Its zero-deletion value and bounds are already included in the nonmonotone row; the domain row covers every retained deletion count.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.zero_deletion_hajek_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2624", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dd70372fd007f0d185f476410974aa9532b67f146029cbde0cfc6da1f5f59e29", "conditional_on": [], "credit_reason": "Both bounds are already included in the nonmonotone counterexample row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.one_deletion_bias_bounds", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2678", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dd70372fd007f0d185f476410974aa9532b67f146029cbde0cfc6da1f5f59e29", "conditional_on": [], "credit_reason": "Both bounds are already included in the nonmonotone counterexample row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.two_deletion_bias_bounds", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2688", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dd70372fd007f0d185f476410974aa9532b67f146029cbde0cfc6da1f5f59e29", "conditional_on": [], "credit_reason": "Degree positivity and nonempty realized cells exclude all zero-denominator conventions from the T8 witness.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.deletion_domain_certificate", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2735", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:dd70372fd007f0d185f476410974aa9532b67f146029cbde0cfc6da1f5f59e29", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T8", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T8" }, "html": { "body": "

p. 303 (PDF p. 15); network of p. 295 (PDF p. 7). In the printed-network witness every Hajek ratio is defined, yet absolute bias strictly decreases over zero, one, and two deleted ties. This refutes a universal monotonic-bias reading of the missing-tie guidance.

", "headline": "Section 16.4.4: ‘In this case, the bias of the estimator will increase with the proportion of mismeasured ties’ and ‘centered around the target quantity […] when the proportion of unmeasured ties is zero, indicating unbiasedness’ (p. 303); the witness is from the check that settled the doubt and is not in the paper", "location": "p. 303 (PDF p. 15); network of p. 295 (PDF p. 7)" }, "id": "P045-T8", "text": { "body": "p. 303 (PDF p. 15); network of p. 295 (PDF p. 7). In the printed-network witness every Hajek ratio is defined, yet absolute bias strictly decreases over zero, one, and two deleted ties. This refutes a universal monotonic-bias reading of the missing-tie guidance.", "headline": "Section 16.4.4: ‘In this case, the bias of the estimator will increase with the proportion of mismeasured ties’ and ‘centered around the target quantity […] when the proportion of unmeasured ties is zero, indicating unbiasedness’ (p. 303); the witness is from the check that settled the doubt and is not in the paper", "location": "p. 303 (PDF p. 15); network of p. 295 (PDF p. 7)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T8" }, { "declarations": [ { "claim_revision_id": "catalog:09fd31373e5fe3270e3803bbb9aaded571f1a32d1575e09397722e1cb47ffc53", "conditional_on": [], "credit_reason": "This is the same entrywise transpose identity for every finite dimension, with an arbitrary extension outside the finite index range.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.corrected_copy_symmetric", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2239", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:09fd31373e5fe3270e3803bbb9aaded571f1a32d1575e09397722e1cb47ffc53", "conditional_on": [], "credit_reason": "The lower-triangle freedom and infinite extension are representational; no finite claimed case is excluded. Symmetry is covered by R259-02, and zero diagonal follows immediately from correctedCopy and the assumed U diagonal.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.corrected_copy_monotone", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2386", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:09fd31373e5fe3270e3803bbb9aaded571f1a32d1575e09397722e1cb47ffc53", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.deletion_exposure_inclusions", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2432", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:09fd31373e5fe3270e3803bbb9aaded571f1a32d1575e09397722e1cb47ffc53", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.missing_ties_ht_bias", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2457", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:09fd31373e5fe3270e3803bbb9aaded571f1a32d1575e09397722e1cb47ffc53", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.hajek_bias_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L2797", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:09fd31373e5fe3270e3803bbb9aaded571f1a32d1575e09397722e1cb47ffc53", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T9", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T9" }, "html": { "body": "

p. 303 (PDF p. 15); staged package R/functions.R, lines 36 and 55. The transposed copy preserves the stated deletion/addition bounds, and missing ties give the stated HT mixture bias without universal Hajek monotonicity.

", "headline": "Section 16.4.4 (p. 303) and the package's tie-editing helpers, corrected: the lower triangle is the transpose of the edited upper triangle; missing ties can introduce bias, whose magnitude need not increase with the proportion of missing ties", "location": "p. 303 (PDF p. 15); staged package R/functions.R, lines 36 and 55" }, "id": "P045-T9", "text": { "body": "p. 303 (PDF p. 15); staged package R/functions.R, lines 36 and 55. The transposed copy preserves the stated deletion/addition bounds, and missing ties give the stated HT mixture bias without universal Hajek monotonicity.", "headline": "Section 16.4.4 (p. 303) and the package's tie-editing helpers, corrected: the lower triangle is the transpose of the edited upper triangle; missing ties can introduce bias, whose magnitude need not increase with the proportion of missing ties", "location": "p. 303 (PDF p. 15); staged package R/functions.R, lines 36 and 55" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T9" }, { "declarations": [ { "claim_revision_id": "catalog:e5a9e6d8e23e8f5a4bc544836044162fc37c17d21dac8f74aab13a6378c6c662", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"AronowAudit/P045/Wave1/Gap/GLP041P042P045P045B1.lean:511 exports the coded centre and treatment laws and the six repaired treatment-vector comparisons, but omits T10(iii)'s cluster-partition probabilities 1/12, 5/12, and 1/2. It also omits T10(v)'s assertion that the three listed centre sets are exactly all separated, covering three-nets, and does not state the repaired treatment law's zero mass on every other vector. The row says it resolves T10, whose claim explicitly includes these conclusions; limiting coverage to the exported comparisons in the description does not supply them. The reviewer identified missing conclusions but treated them as minor, whereas the required complete distribution and net-characterization conclusions need to be included under the stated standard.\", 'suggested_fix': \"Extend the credited theorem's conjunction with the radius-two ball table, the three cluster-partition event probabilities defined using nearestCentre, the equivalence epsilonNet d 3 S iff S belongs to {{0,3},{1},{2}}, and a universal zero-support assertion for the repaired treatment law outside 0000, 1111, 1100, and 0011. Existing centre-law and treatment-law helpers can support these additional conclusions without changing the sampler model.\", 'second_review': \"P045/Wave1/Gap/GLP041P042P045P045B1.lean:511 exports the centre masses and treatment-vector comparisons, but omits T10(iii)'s cluster-partition probabilities and T10(v)'s assertion that the listed centre sets are the only three-nets, as well as the full repaired zero-support assertion. These are conclusions of the target claim itself, so limiting the row description does not remove the statement omission. However, nearestCentre at P045/P045_Spillover.lean:3124 gives the omitted partition masses by the elementary sums 1/12, 1/12+1/3=5/12, and 1/4+1/4=1/2; the ball, net and repaired-support checks require only the four-vertex construction defined at lines 2877 and 3091. The missing conclusions therefore warrant a minor statement repair, rather than the adjudicator's major rating.\", 'severity': 'minor'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P045.Wave1.cluster_sampler_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/Wave1/Gap/GLP041P042P045P045B1.lean#L510", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:e5a9e6d8e23e8f5a4bc544836044162fc37c17d21dac8f74aab13a6378c6c662", "conditional_on": [], "credit_reason": "The aggregate row already supplies the full separation-failure probability and treatment-law counterexample.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.singleton_draw_bad_trace", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3158", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:e5a9e6d8e23e8f5a4bc544836044162fc37c17d21dac8f74aab13a6378c6c662", "conditional_on": [], "credit_reason": "This supports the existing sampler proof; the retained statement does not export the separate ball certificate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.four_path_marking_certificate", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3076", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e5a9e6d8e23e8f5a4bc544836044162fc37c17d21dac8f74aab13a6378c6c662", "conditional_on": [], "credit_reason": "This retry calculation supports the existing sampler proof; the retained statement exports its terminal law.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.four_path_repeat_trace_mass", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3097", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e5a9e6d8e23e8f5a4bc544836044162fc37c17d21dac8f74aab13a6378c6c662", "conditional_on": [], "credit_reason": "This convergent retry sum supports the existing sampler proof; the retained statement exports its terminal law.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.four_path_repeat_trace_sum", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3147", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:e5a9e6d8e23e8f5a4bc544836044162fc37c17d21dac8f74aab13a6378c6c662", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T10", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T10" }, "html": { "body": "

p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 76–82 and 122–128. On the four-path, use labels 0 through 3, nearest-centre clusters with ties to the smallest label, and independent fair cluster treatment. The radius-two balls are {0,1,2}, all vertices, all vertices, and {1,2,3}. Each initial centre has probability 1/4. After initial centre 0, each retry draw is uniform on all four vertices and each eventual second distinct centre has conditional probability 1/3; initial centre 3 forces 0, and centres 1 or 2 finish at once. The coded terminal law is {0,1}:1/12, {0,2}:1/12, {0,3}:1/3, {1}:1/4, {2}:1/4, zero elsewhere, and normalized; separation fails with probability 1/6. The cluster partitions {0}|{1,2,3}, {0,1}|{2,3}, and {0,1,2,3} have masses 1/12, 5/12, 1/2 and exhaust the law. The coded treatment law is 0000:3/8, 1111:3/8, 1100:5/48, 0011:5/48, 1000:1/48, 0111:1/48, zero elsewhere. The repaired terminal law is {0,3}:1/2, {1}:1/4, {2}:1/4, zero elsewhere; these are exactly the separated covering three-nets. Its treatment law is 0000:3/8, 1111:3/8, 1100:1/8, 0011:1/8, zero elsewhere. All terminal series converge, and the treatment distributions differ. The usual measure-theoretic conditions are assumed.

", "headline": "Section 16.4.6 and Table 16.3, ‘Cluster = three-net clustering’ (p. 305), against the package's 3-net sampler (make_tr_vec_bernoulli_cluster), which can choose an already marked vertex as a cluster centre when one unmarked vertex remains; the witness is the validator's and is not in the paper", "location": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 76–82 and 122–128 (identical in both snapshots)" }, "id": "P045-T10", "text": { "body": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 76–82 and 122–128. On the four-path, use labels 0 through 3, nearest-centre clusters with ties to the smallest label, and independent fair cluster treatment. The radius-two balls are {0,1,2}, all vertices, all vertices, and {1,2,3}. Each initial centre has probability 1/4. After initial centre 0, each retry draw is uniform on all four vertices and each eventual second distinct centre has conditional probability 1/3; initial centre 3 forces 0, and centres 1 or 2 finish at once. The coded terminal law is {0,1}:1/12, {0,2}:1/12, {0,3}:1/3, {1}:1/4, {2}:1/4, zero elsewhere, and normalized; separation fails with probability 1/6. The cluster partitions {0}|{1,2,3}, {0,1}|{2,3}, and {0,1,2,3} have masses 1/12, 5/12, 1/2 and exhaust the law. The coded treatment law is 0000:3/8, 1111:3/8, 1100:5/48, 0011:5/48, 1000:1/48, 0111:1/48, zero elsewhere. The repaired terminal law is {0,3}:1/2, {1}:1/4, {2}:1/4, zero elsewhere; these are exactly the separated covering three-nets. Its treatment law is 0000:3/8, 1111:3/8, 1100:1/8, 0011:1/8, zero elsewhere. All terminal series converge, and the treatment distributions differ. The usual measure-theoretic conditions are assumed.", "headline": "Section 16.4.6 and Table 16.3, ‘Cluster = three-net clustering’ (p. 305), against the package's 3-net sampler (make_tr_vec_bernoulli_cluster), which can choose an already marked vertex as a cluster centre when one unmarked vertex remains; the witness is the validator's and is not in the paper", "location": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 76–82 and 122–128 (identical in both snapshots)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T10" }, { "declarations": [ { "claim_revision_id": "catalog:1cadd499fc6354b71c91958d3c20692b4ec9d6adbecd2612d5813b20fd35ab25", "conditional_on": [], "credit_reason": "The deterministic component is faithful; the uncredited stochastic clause T11(e) remains uncovered here. The generic Boolean-assignment/weighted-sum boilerplate in modelling_choices does not describe this graph theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.greedy_three_net_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3181", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1cadd499fc6354b71c91958d3c20692b4ec9d6adbecd2612d5813b20fd35ab25", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T11", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T11" }, "html": { "body": "

p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 77 and 123. Every admissible repaired greedy run has distinct separated centers, strict progress, a length bound, and a covering completion.

", "headline": "The package's 3-net sampler, corrected: each centre is drawn uniformly from the unmarked vertices, and the centres then form a 3-net", "location": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 77 and 123" }, "id": "P045-T11", "text": { "body": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 77 and 123. Every admissible repaired greedy run has distinct separated centers, strict progress, a length bound, and a covering completion.", "headline": "The package's 3-net sampler, corrected: each centre is drawn uniformly from the unmarked vertices, and the centres then form a 3-net", "location": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 77 and 123" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T11" }, { "declarations": [ { "claim_revision_id": "catalog:359e3fac35e20ba1d4d7115472a8e15945270996bb8d4bea377e3ec05ac71428", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'AronowAudit/P045/P045_Spillover.lean:3440 states the 45-pair support, exposure probabilities, adjacent/nonadjacent point values, pointwise inequality, and expectations 11/9 and 3 required by this T12 component. The literal 1/5 in cycleL at line 3342 is the actual own-treatment probability: each unit belongs to nine of the 45 unordered treatment pairs. It therefore evaluates the printed left expression under the specified fixed-count design, with no substituted sampling law or extra hypothesis.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.naive_denominator_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3345", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:359e3fac35e20ba1d4d7115472a8e15945270996bb8d4bea377e3ec05ac71428", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T12", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T12" }, "html": { "body": "

p. 301 (PDF p. 13). The ten-cycle with exactly two uniformly chosen treated units and exposure outcomes (2,1,1,0) is the exact counterexample to the printed second equality. Each own-treatment probability is 1/5; the d11, d10, d01, d00 probabilities are 2/45, 7/45, 1/3, 7/15. All five are positive and the two treated-exposure probabilities sum to 1/5. Of the 45 assignments, ten treat adjacent units and give (L,R,H)=(2,9,9); the other 35 give (1,9/7,0), where L and R are the two printed sides and H is the correct d11 HT mean. Thus L differs from R on every assignment. The all-treated mean and mu(d11) are 2, and mu(d10)=1. E[L]=11/9, E[R]=3=mu(d11)+mu(d10), and E[H]=2. The bias of L is 11/9-2=-7/9, also exactly the negative mean of (pi(d10)/pi)(y(d11)-y(d10)). The usual measure-theoretic conditions are assumed.

", "headline": "Section 16.4.3, first display of p. 301, second equality (the expansion of μHT,None^(1)\\widehat{\\mu_{HT,None}}(\\mathbf 1)μHT,None​​(1) with denominators πi(d11)\\pi_i(d_{11})πi​(d11​) and πi(d10)\\pi_i(d_{10})πi​(d10​)); the witness is the validator's and is not in the paper", "location": "p. 301 (PDF p. 13)" }, "id": "P045-T12", "text": { "body": "p. 301 (PDF p. 13). The ten-cycle with exactly two uniformly chosen treated units and exposure outcomes (2,1,1,0) is the exact counterexample to the printed second equality. Each own-treatment probability is 1/5; the d11, d10, d01, d00 probabilities are 2/45, 7/45, 1/3, 7/15. All five are positive and the two treated-exposure probabilities sum to 1/5. Of the 45 assignments, ten treat adjacent units and give (L,R,H)=(2,9,9); the other 35 give (1,9/7,0), where L and R are the two printed sides and H is the correct d11 HT mean. Thus L differs from R on every assignment. The all-treated mean and mu(d11) are 2, and mu(d10)=1. E[L]=11/9, E[R]=3=mu(d11)+mu(d10), and E[H]=2. The bias of L is 11/9-2=-7/9, also exactly the negative mean of (pi(d10)/pi)(y(d11)-y(d10)). The usual measure-theoretic conditions are assumed.", "headline": "Section 16.4.3, first display of p. 301, second equality (the expansion of $\\widehat{\\mu_{HT,None}}(\\mathbf 1)$ with denominators $\\pi_i(d_{11})$ and $\\pi_i(d_{10})$); the witness is the validator's and is not in the paper", "location": "p. 301 (PDF p. 13)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T12" }, { "declarations": [ { "claim_revision_id": "catalog:c4a77c6ba9b0f48c7c6ebcfd1aad1e86ec467ea3e549bce2094e5a33e754575d", "conditional_on": [], "credit_reason": "The additional signed-weight and arbitrary-adjacency cases preserve exactly the required specialization. Zero division and the empty-population convention do not replace any claimed positive-probability case.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.naive_expansion_correct", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3399", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4a77c6ba9b0f48c7c6ebcfd1aad1e86ec467ea3e549bce2094e5a33e754575d", "conditional_on": [], "credit_reason": "Scope is the partition-probability identity, mixture/bias/sign statements, and summed exposure-weight expectation; the single-HT unbiasedness clause is not separately exported. It follows under the target's two-exposure positivity assumptions by applying the universal summed identity with all d10 outcomes set to zero. The corrected target uses mu(d11), avoiding an unjustified identification with all-treated outcomes at isolated vertices.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.naive_bias_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3412", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4a77c6ba9b0f48c7c6ebcfd1aad1e86ec467ea3e549bce2094e5a33e754575d", "conditional_on": [], "credit_reason": "The counterexample_proved status concerns the refutation of the printed some-unit-difference suffices assertion, while target_kind=corrected_statement denotes the correction that includes this witness; these metadata describe compatible roles.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.spillover_cancellation_instance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3519", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c4a77c6ba9b0f48c7c6ebcfd1aad1e86ec467ea3e549bce2094e5a33e754575d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T13", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T13" }, "html": { "body": "

p. 301 (PDF p. 13); lead-in on p. 300 (PDF p. 12). For any positive finite population, fixed undirected loopless network, correctly specified first-degree outcomes and design with positive own-treatment probabilities, the two treated-exposure indicators partition own treatment. The corrected pointwise expansion uses the common own-treatment denominator. Its expectation is the unitwise mixture of y(d11) and y(d10), with nonnegative weights pi(d11)/pi and pi(d10)/pi summing to one. Its bias for mu(d11) is the negative mean of (pi(d10)/pi)(y(d11)-y(d10)). Nonnegative spillovers give nonpositive bias, strictly negative exactly when some unit has both pi(d10)>0 and strictly positive spillover; nonpositive spillovers give nonnegative bias. When both treated-exposure probabilities are positive, the individual d11 HT mean has expectation mu(d11), and the sum of the d11 and d10 means has expectation mu(d11)+mu(d10). With no isolated units, the all-treated mean equals mu(d11). Two adjacent independently fair-treated units with y(d11)=(2,1), y(d10)=(1,2), and untreated outcomes zero give naive estimates 0,2,1,3 and mean 3/2=mu(d11), despite unequal treated-exposure outcomes at both units. The usual measure-theoretic conditions are assumed.

", "headline": "Section 16.4.3, the expansion of the no-interference Horvitz–Thompson mean μHT,None^(1)\\widehat{\\mu_{HT,None}}(\\mathbf 1)μHT,None​​(1) and the sentence on its bias (pp. 300–301), corrected: the common denominator πi\\pi_iπi​ in both terms, and the exact bias for μ(1)\\mu(\\mathbf 1)μ(1)", "location": "p. 301 (PDF p. 13); lead-in on p. 300 (PDF p. 12)" }, "id": "P045-T13", "text": { "body": "p. 301 (PDF p. 13); lead-in on p. 300 (PDF p. 12). For any positive finite population, fixed undirected loopless network, correctly specified first-degree outcomes and design with positive own-treatment probabilities, the two treated-exposure indicators partition own treatment. The corrected pointwise expansion uses the common own-treatment denominator. Its expectation is the unitwise mixture of y(d11) and y(d10), with nonnegative weights pi(d11)/pi and pi(d10)/pi summing to one. Its bias for mu(d11) is the negative mean of (pi(d10)/pi)(y(d11)-y(d10)). Nonnegative spillovers give nonpositive bias, strictly negative exactly when some unit has both pi(d10)>0 and strictly positive spillover; nonpositive spillovers give nonnegative bias. When both treated-exposure probabilities are positive, the individual d11 HT mean has expectation mu(d11), and the sum of the d11 and d10 means has expectation mu(d11)+mu(d10). With no isolated units, the all-treated mean equals mu(d11). Two adjacent independently fair-treated units with y(d11)=(2,1), y(d10)=(1,2), and untreated outcomes zero give naive estimates 0,2,1,3 and mean 3/2=mu(d11), despite unequal treated-exposure outcomes at both units. The usual measure-theoretic conditions are assumed.", "headline": "Section 16.4.3, the expansion of the no-interference Horvitz–Thompson mean $\\widehat{\\mu_{HT,None}}(\\mathbf 1)$ and the sentence on its bias (pp. 300–301), corrected: the common denominator $\\pi_i$ in both terms, and the exact bias for $\\mu(\\mathbf 1)$", "location": "p. 301 (PDF p. 13); lead-in on p. 300 (PDF p. 12)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T13" }, { "declarations": [ { "claim_revision_id": "catalog:f3458211946cfb968893505baf4b0bbfda661bdd260a0a01f02a38c59941db37", "conditional_on": [], "credit_reason": "This is the arithmetic component of the counterexample; R259-16 supplies the actual population and assignment. The target's positivity and equal-subclass-mean observations are direct algebraic consequences of the universal discrepancy identity.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.pooling_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3540", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f3458211946cfb968893505baf4b0bbfda661bdd260a0a01f02a38c59941db37", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.pooling_population_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3628", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f3458211946cfb968893505baf4b0bbfda661bdd260a0a01f02a38c59941db37", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.pooling_empty_subclass", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3678", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f3458211946cfb968893505baf4b0bbfda661bdd260a0a01f02a38c59941db37", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T14", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T14" }, "html": { "body": "

p. 312 (PDF p. 24); the estimator is defined on p. 310 (PDF p. 22). The six-group tract witness separates the pooled estimator from the printed sum and exhibits empty subclasses.

", "headline": "Section 16.5.2, the display for y^(z;ψ)\\widehat y(z;\\psi)y​(z;ψ) on p. 312 (a sum of two separately normalized subgroup means); the witness is the validator's and is not in the paper", "location": "p. 312 (PDF p. 24); the estimator is defined on p. 310 (PDF p. 22)" }, "id": "P045-T14", "text": { "body": "p. 312 (PDF p. 24); the estimator is defined on p. 310 (PDF p. 22). The six-group tract witness separates the pooled estimator from the printed sum and exhibits empty subclasses.", "headline": "Section 16.5.2, the display for $\\widehat y(z;\\psi)$ on p. 312 (a sum of two separately normalized subgroup means); the witness is the validator's and is not in the paper", "location": "p. 312 (PDF p. 24); the estimator is defined on p. 310 (PDF p. 22)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T14" }, { "declarations": [ { "claim_revision_id": "catalog:523f4a9e4177dcce956dbb6e1587f79ea0945419330c4ef6cc89f3d963170071", "conditional_on": [], "credit_reason": "The A=0 boundary hypothesis is already implied by the claim's empty-sum definition, so it is not a narrowing. The first Option equality unfolds the chosen representation, while the weighted-mixture identity remains substantive algebra.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.pooled_ratio_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3698", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:523f4a9e4177dcce956dbb6e1587f79ea0945419330c4ef6cc89f3d963170071", "conditional_on": [], "credit_reason": "Six groups are explicitly fixed in the claim's setting. Quantifying over every H in that setting loses no stated first-stage assignment; the scalar ratio theorem R259-18 supplies the denominator algebra.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.tract_partition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3718", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:523f4a9e4177dcce956dbb6e1587f79ea0945419330c4ef6cc89f3d963170071", "conditional_on": [], "credit_reason": "The auxiliary all-psi law agrees with the focal-tract psi-psi intervention because tractOutcome depends only on that tract. The record's missing record-linkage compile-time example is a provenance check, not a lost mathematical hypothesis or conclusion, and is outside this statement-only review.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.tract_mixture_bias_identified", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/Wave1/Gap/FidP045.lean#L315", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:523f4a9e4177dcce956dbb6e1587f79ea0945419330c4ef6cc89f3d963170071", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T15", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T15" }, "html": { "body": "

p. 312 (PDF p. 24); setting on p. 311 (PDF p. 23). The own-psi groups are the disjoint union of those with psi partners and those with phi partners, for any finite group population and tract pairing. Counts and sums partition accordingly. With positive total count, the selected-group mean is the pooled ratio; with both subclass counts positive, its positive group shares sum to one and give the weighted mixture of subclass means. An empty first subclass has zero numerator and leaves the other mean. The scalar pooling identities and the complete normalized 450-unit two-stage witness are included: the two group direct effects are 7/8 and 3/4, the psi-psi target is 7/8, expected shares are 2/5 and 3/5, and the mean estimator is 4/5 with bias -3/40. The usual measure-theoretic conditions are assumed.

", "headline": "Section 16.5.2, the display for y^(z;ψ)\\widehat y(z;\\psi)y​(z;ψ) (p. 312), corrected: one ratio with the two sums added in numerator and denominator (a share-weighted mixture of the two subgroup means)", "location": "p. 312 (PDF p. 24); setting on p. 311 (PDF p. 23)" }, "id": "P045-T15", "text": { "body": "p. 312 (PDF p. 24); setting on p. 311 (PDF p. 23). The own-psi groups are the disjoint union of those with psi partners and those with phi partners, for any finite group population and tract pairing. Counts and sums partition accordingly. With positive total count, the selected-group mean is the pooled ratio; with both subclass counts positive, its positive group shares sum to one and give the weighted mixture of subclass means. An empty first subclass has zero numerator and leaves the other mean. The scalar pooling identities and the complete normalized 450-unit two-stage witness are included: the two group direct effects are 7/8 and 3/4, the psi-psi target is 7/8, expected shares are 2/5 and 3/5, and the mean estimator is 4/5 with bias -3/40. The usual measure-theoretic conditions are assumed.", "headline": "Section 16.5.2, the display for $\\widehat y(z;\\psi)$ (p. 312), corrected: one ratio with the two sums added in numerator and denominator (a share-weighted mixture of the two subgroup means)", "location": "p. 312 (PDF p. 24); setting on p. 311 (PDF p. 23)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T15" }, { "declarations": [ { "claim_revision_id": "catalog:45d645beb49cbddb34933e8d3b557076cdbacdc7ba444c8c202fa167f0bde4ab", "conditional_on": [], "credit_reason": "The empty HT total is zero as an ordinary finite sum, not a totalized ratio; positivity of every unit inclusion probability is exported explicitly.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.empty_cell_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3873", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:45d645beb49cbddb34933e8d3b557076cdbacdc7ba444c8c202fa167f0bde4ab", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T16", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T16" }, "html": { "body": "

p. 298 (PDF p. 10); definition on p. 297 (PDF p. 9); network on p. 295 (PDF p. 7). The printed-network HT contrast remains numerical for an empty exposure cell, and the stated support restrictions change its expectation.

", "headline": "Section 16.4.2: ‘In this case, we cannot compute τHT^(d11,d00)\\widehat{\\tau_{HT}}(d_{11},d_{00})τHT​​(d11​,d00​) […] because in this small-scale example no unit received exposure d11d_{11}d11​’ and the entries ‘NA’ of the two printed outputs (p. 298); the witness is the validators' and is not in the paper", "location": "p. 298 (PDF p. 10); definition on p. 297 (PDF p. 9); network on p. 295 (PDF p. 7)" }, "id": "P045-T16", "text": { "body": "p. 298 (PDF p. 10); definition on p. 297 (PDF p. 9); network on p. 295 (PDF p. 7). The printed-network HT contrast remains numerical for an empty exposure cell, and the stated support restrictions change its expectation.", "headline": "Section 16.4.2: ‘In this case, we cannot compute $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ […] because in this small-scale example no unit received exposure $d_{11}$’ and the entries ‘NA’ of the two printed outputs (p. 298); the witness is the validators' and is not in the paper", "location": "p. 298 (PDF p. 10); definition on p. 297 (PDF p. 9); network on p. 295 (PDF p. 7)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T16" }, { "declarations": [ { "claim_revision_id": "catalog:660c41c5839f6863a9d89e4cf7ba73b0fb64ff215a35a60b967cc1a774191e99", "conditional_on": [], "credit_reason": "A valid algebraic extension of T17(a); every intended positive-size distinct-label case is included.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.empty_ht_total", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L321", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:660c41c5839f6863a9d89e4cf7ba73b0fb64ff215a35a60b967cc1a774191e99", "conditional_on": [], "credit_reason": "A valid generalization of T17(d); Option.none preserves the distinction between an undefined ratio and an observed zero.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.empty_hajek", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L332", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:660c41c5839f6863a9d89e4cf7ba73b0fb64ff215a35a60b967cc1a774191e99", "conditional_on": [], "credit_reason": "No cases of the single-cell claim are excluded by recoding labels; this does not supply a general four-label representation of the entire original outcome schedule.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.nonempty_ht_conditional", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3928", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:660c41c5839f6863a9d89e4cf7ba73b0fb64ff215a35a60b967cc1a774191e99", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T17", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T17" }, "html": { "body": "

pp. 297–299 (PDF pp. 9–11). Empty realized HT cells contribute zero without being discarded; unconditional and nonempty-event conditional expectations have the stated identities, while empty Hajek ratios and ordinary sample means for realized exposure cells are undefined.

", "headline": "Section 16.4.2, the sentence on τHT^(d11,d00)\\widehat{\\tau_{HT}}(d_{11},d_{00})τHT​​(d11​,d00​) in the toy example (p. 298), corrected: the Horvitz–Thompson total of an exposure that no unit received is zero, so the contrast is computable", "location": "p. 298 (PDF p. 10); definition on p. 297 (PDF p. 9)" }, "id": "P045-T17", "text": { "body": "pp. 297–299 (PDF pp. 9–11). Empty realized HT cells contribute zero without being discarded; unconditional and nonempty-event conditional expectations have the stated identities, while empty Hajek ratios and ordinary sample means for realized exposure cells are undefined.", "headline": "Section 16.4.2, the sentence on $\\widehat{\\tau_{HT}}(d_{11},d_{00})$ in the toy example (p. 298), corrected: the Horvitz–Thompson total of an exposure that no unit received is zero, so the contrast is computable", "location": "p. 298 (PDF p. 10); definition on p. 297 (PDF p. 9)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T17" }, { "declarations": [ { "claim_revision_id": "catalog:538e858094196421d6f0a2d913fb5f4854036d360f43512d053e34d2149593cd", "conditional_on": [], "credit_reason": "Covers the ordered formula, own-coordinate independence and bijection in T18(i)-(iii), including both endpoints.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.coordinate_removal_correct", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3958", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:538e858094196421d6f0a2d913fb5f4854036d360f43512d053e34d2149593cd", "conditional_on": [], "credit_reason": "Completes the existence-and-uniqueness portion of T18(iii).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.reindexed_outcome_unique", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3980", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:538e858094196421d6f0a2d913fb5f4854036d360f43512d053e34d2149593cd", "conditional_on": [], "credit_reason": "The recorded difference mentions an older lean.json description of first/last-coordinate examples, but no such public-description text is supplied in this packet; the supplied original row correctly says third coordinate.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.coordinate_removal_instance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3994", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:538e858094196421d6f0a2d913fb5f4854036d360f43512d053e34d2149593cd", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T18", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T18" }, "html": { "body": "

p. 292 (PDF p. 4). Ordered coordinate removal ignores own treatment and gives a bijective reindexing with a unique outcome function and the four-coordinate example.

", "headline": "Section 16.3, the definition of z−i\\mathbf z_{-i}z−i​ (p. 292), with the misprinted first entry corrected (z1z_1z1​ for ziz_izi​)", "location": "p. 292 (PDF p. 4)" }, "id": "P045-T18", "text": { "body": "p. 292 (PDF p. 4). Ordered coordinate removal ignores own treatment and gives a bijective reindexing with a unique outcome function and the four-coordinate example.", "headline": "Section 16.3, the definition of $\\mathbf z_{-i}$ (p. 292), with the misprinted first entry corrected ($z_1$ for $z_i$)", "location": "p. 292 (PDF p. 4)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T18" }, { "declarations": [ { "claim_revision_id": "catalog:4551f7a6121a0de42615612d01f794f870a723c3a11950f39f12c979e5936356", "conditional_on": [], "credit_reason": "The integral-saturation premise is exactly the target assumption; n=3 and psi=2/3 give its displayed toy instance.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.fixed_count_peers_correct", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4002", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:4551f7a6121a0de42615612d01f794f870a723c3a11950f39f12c979e5936356", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T19", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T19" }, "html": { "body": "

pp. 309–310 (PDF pp. 21–22). With a fixed treated count, treated peers number K minus own treatment, giving the corrected saturation formula and the three-member example.

", "headline": "Section 16.5.1, the number of other group members treated under complete random assignment (p. 310, top), corrected: ψng−1\\psi n_g - 1ψng​−1 for a treated unit", "location": "p. 310 (PDF p. 22); the sentence begins on p. 309 (PDF p. 21)" }, "id": "P045-T19", "text": { "body": "pp. 309–310 (PDF pp. 21–22). With a fixed treated count, treated peers number K minus own treatment, giving the corrected saturation formula and the three-member example.", "headline": "Section 16.5.1, the number of other group members treated under complete random assignment (p. 310, top), corrected: $\\psi n_g - 1$ for a treated unit", "location": "p. 310 (PDF p. 22); the sentence begins on p. 309 (PDF p. 21)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T19" }, { "declarations": [ { "claim_revision_id": "catalog:b7a37b87a15b049c2f8b74c50b9e9f2e2ce6cbac0a0726c4e62ab65837955cbb", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'AronowAudit/P045/P045_Spillover.lean:3314 refutes exactly the upper-bound condition W together with coverage C that T20 explicitly defines, and gives both the count and complete list of thirteen corrected nets. Including the empty subset changes neither assertion because that subset cannot cover the nonempty nine-path. The malformed printed phrase does not determine a unique interpretation, but the target itself expressly specifies W, so the Lean statement is faithful to the chosen claim rather than silently selecting a different condition.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.within_hops_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3219", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:b7a37b87a15b049c2f8b74c50b9e9f2e2ce6cbac0a0726c4e62ab65837955cbb", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T20", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T20" }, "html": { "body": "

p. 305 (PDF p. 17). On the nine-vertex path no set satisfies the literal pairwise-within-three reading and the covering clause, whereas exactly the thirteen listed sets satisfy the corrected separation reading.

", "headline": "Section 16.4.6, the definition of ϵ\\epsilonϵ-net clustering: ‘a set of units such that all units in the set are at least within ϵ\\epsilonϵ hops of each other, and every unit outside the set is within ϵ−1\\epsilon - 1ϵ−1 hops of a unit in the set’ (p. 305), read literally (‘within’ as an upper bound); the witness is the validator's and is not in the paper", "location": "p. 305 (PDF p. 17)" }, "id": "P045-T20", "text": { "body": "p. 305 (PDF p. 17). On the nine-vertex path no set satisfies the literal pairwise-within-three reading and the covering clause, whereas exactly the thirteen listed sets satisfy the corrected separation reading.", "headline": "Section 16.4.6, the definition of $\\epsilon$-net clustering: ‘a set of units such that all units in the set are at least within $\\epsilon$ hops of each other, and every unit outside the set is within $\\epsilon - 1$ hops of a unit in the set’ (p. 305), read literally (‘within’ as an upper bound); the witness is the validator's and is not in the paper", "location": "p. 305 (PDF p. 17)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T20" }, { "declarations": [ { "claim_revision_id": "catalog:1ecb168df3a867bdec3c818b3d2ed1c590ff636f0b9256080734ca344e5a7d56", "conditional_on": [], "credit_reason": "Covers T21(i)-(ii); the separately credited nine-path instance is outside this chunk.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.epsilon_net_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L3194", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1ecb168df3a867bdec3c818b3d2ed1c590ff636f0b9256080734ca344e5a7d56", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T21", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T21" }, "html": { "body": "

p. 305 (PDF p. 17). Corrected epsilon-nets are maximal separated sets, exist by the bounded finite greedy construction, and have exactly the thirteen listed instances on the nine-vertex path.

", "headline": "Section 16.4.6, the definition of ϵ\\epsilonϵ-net clustering (p. 305), corrected: the units of the set are pairwise at least ϵ\\epsilonϵ hops apart", "location": "p. 305 (PDF p. 17)" }, "id": "P045-T21", "text": { "body": "p. 305 (PDF p. 17). Corrected epsilon-nets are maximal separated sets, exist by the bounded finite greedy construction, and have exactly the thirteen listed instances on the nine-vertex path.", "headline": "Section 16.4.6, the definition of $\\epsilon$-net clustering (p. 305), corrected: the units of the set are pairwise at least $\\epsilon$ hops apart", "location": "p. 305 (PDF p. 17)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T21" }, { "declarations": [ { "claim_revision_id": "catalog:1447994d9c3f97fb5e03e469659f8137366f5b01fb3c43d5889c72905820a5ec", "conditional_on": [], "credit_reason": "Metadata inconsistency: T22 has asymptotic=false although its statement and this row explicitly contain convergence and nonconvergence in probability. This does not change fidelity to its stated witness.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.mean_se_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4265", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1447994d9c3f97fb5e03e469659f8137366f5b01fb3c43d5889c72905820a5ec", "conditional_on": [], "credit_reason": "The moments concern the actual N-sign mean, not an assumed two-point distribution of that mean.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.sign_mean_finite_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4065", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1447994d9c3f97fb5e03e469659f8137366f5b01fb3c43d5889c72905820a5ec", "conditional_on": [], "credit_reason": "Included by the complete mean_se_counterexample assembly.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.sign_mean_chebyshev", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4133", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1447994d9c3f97fb5e03e469659f8137366f5b01fb3c43d5889c72905820a5ec", "conditional_on": [], "credit_reason": "An auxiliary finite lemma, not a substitute for the full asymptotic witness; R260-08 exports the latter.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.mean_se_finite_obstruction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4154", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1447994d9c3f97fb5e03e469659f8137366f5b01fb3c43d5889c72905820a5ec", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T22", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T22" }, "html": { "body": "

pp. 299–300 (PDF pp. 11–12). The fair-sign witness has exact mean-SE agreement and positive variance-estimate bias but fails relative consistency, while its point estimate and absolute variance error are consistent. This is the audit witness, rather than either estimator in Table 16.1.

", "headline": "Section 16.4.2: ‘Moreover, the variance estimator is consistent, given that the mean of the standard error estimate approaches the empirical standard deviation from the simulation’ (p. 299): the inference from agreement of mean standard error and standard deviation to consistency; the witness is the validator's and is not in the paper", "location": "p. 299 (PDF p. 11); Table 16.1, p. 300 (PDF p. 12)" }, "id": "P045-T22", "text": { "body": "pp. 299–300 (PDF pp. 11–12). The fair-sign witness has exact mean-SE agreement and positive variance-estimate bias but fails relative consistency, while its point estimate and absolute variance error are consistent. This is the audit witness, rather than either estimator in Table 16.1.", "headline": "Section 16.4.2: ‘Moreover, the variance estimator is consistent, given that the mean of the standard error estimate approaches the empirical standard deviation from the simulation’ (p. 299): the inference from agreement of mean standard error and standard deviation to consistency; the witness is the validator's and is not in the paper", "location": "p. 299 (PDF p. 11); Table 16.1, p. 300 (PDF p. 12)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T22" }, { "declarations": [ { "claim_revision_id": "catalog:be447021cd93af8f15316f6b0a8d22d784de2330996cb939ee9ef808f6ac9643", "conditional_on": [], "credit_reason": "The target concerns the printed rounded table entries, so exact arithmetic on their rational transcriptions is sufficient.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.table_standard_error_comparison", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4294", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:be447021cd93af8f15316f6b0a8d22d784de2330996cb939ee9ef808f6ac9643", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"At P045/P045_Spillover.lean:4399, the Fintype assumption represents the finite randomization already imposed by standing assumption S1 (a design on {0,1}^N with finite-sum expectations) and S2 (fixed potential outcomes and assignment as the only randomness). The sample spaces and designs may vary with n, so this representation covers all estimators in that standing model. The hypotheses at P045/P045_Spillover.lean:4401 reproduce the positivity, nonnegativity and two limit premises of T23(b2), and the conclusion at P045/P045_Spillover.lean:4405, using InProbability at P045/P045_Spillover.lean:4116, expresses exactly the claimed convergence of Vhat_N - V_N to zero in probability. The reviewer's continuous-U example lies outside the supplied standing assumptions and therefore does not demonstrate an excluded case of this claim.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.absolute_variance_consistency", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4305", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:be447021cd93af8f15316f6b0a8d22d784de2330996cb939ee9ef808f6ac9643", "conditional_on": [], "credit_reason": "A witness for failure of T23(b2) without vanishing true variance, not a claim that the chapter's own estimator is inconsistent.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.nonvanishing_variance_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4342", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:be447021cd93af8f15316f6b0a8d22d784de2330996cb939ee9ef808f6ac9643", "conditional_on": [], "credit_reason": "M>0 is the meaningful domain of the displayed arithmetic mean; nonnegative inputs are already in T23(b4).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.mean_square_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4363", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:be447021cd93af8f15316f6b0a8d22d784de2330996cb939ee9ef808f6ac9643", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T23", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T23" }, "html": { "body": "

pp. 299–300 (PDF pp. 11–12), Table 16.1. The six table ratios have the corrected finite-sample comparisons; the witness, the vanishing-variance implication, the nonvanishing-variance counterexample, and the mean-square inequality delimit what mean-SE agreement implies.

", "headline": "Section 16.4.2, the closing sentence on the variance estimator (p. 299), corrected: a finite-sample comparison of mean standard errors with empirical standard deviations in Table 16.1, which does not establish consistency", "location": "p. 299 (PDF p. 11); Table 16.1, p. 300 (PDF p. 12)" }, "id": "P045-T23", "text": { "body": "pp. 299–300 (PDF pp. 11–12), Table 16.1. The six table ratios have the corrected finite-sample comparisons; the witness, the vanishing-variance implication, the nonvanishing-variance counterexample, and the mean-square inequality delimit what mean-SE agreement implies.", "headline": "Section 16.4.2, the closing sentence on the variance estimator (p. 299), corrected: a finite-sample comparison of mean standard errors with empirical standard deviations in Table 16.1, which does not establish consistency", "location": "p. 299 (PDF p. 11); Table 16.1, p. 300 (PDF p. 12)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T23" }, { "declarations": [ { "claim_revision_id": "catalog:0e7d95a9740b588f4f212c562e0a31030681f1dc5371d9433b79bed6eb692036", "conditional_on": [], "credit_reason": "The mixture is in the target's own mathematical content, not an extra hypothesis justified only by the row description. The theorem gives the effect algebra, not a verification of the plotted reference-line locations.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.dilated_effect_labels", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4420", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:0e7d95a9740b588f4f212c562e0a31030681f1dc5371d9433b79bed6eb692036", "conditional_on": [], "credit_reason": "This is the general marginal-effect identity T24(iii); it requires neither the dilated model nor a law on assignments.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.total_direct_indirect_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4395", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:0e7d95a9740b588f4f212c562e0a31030681f1dc5371d9433b79bed6eb692036", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T24", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T24" }, "html": { "body": "

p. 312 (PDF p. 24), Figure 16.6; model on p. 310. The dilated model gives the five effect formulas and the ordering and ratio of direct effects, and total equals direct at psi plus indirect for every outcome schedule.

", "headline": "Figure 16.6 (p. 312), strip labels of the first two columns, corrected: τ^D(ψ)\\widehat\\tau^D(\\psi)τD(ψ) first and τ^D(ϕ)\\widehat\\tau^D(\\phi)τD(ϕ) second; formal content: the order of the two direct-effect estimands under the outcome model of p. 310", "location": "p. 312 (PDF p. 24); outcome model on p. 310 (PDF p. 22)" }, "id": "P045-T24", "text": { "body": "p. 312 (PDF p. 24), Figure 16.6; model on p. 310. The dilated model gives the five effect formulas and the ordering and ratio of direct effects, and total equals direct at psi plus indirect for every outcome schedule.", "headline": "Figure 16.6 (p. 312), strip labels of the first two columns, corrected: $\\widehat\\tau^D(\\psi)$ first and $\\widehat\\tau^D(\\phi)$ second; formal content: the order of the two direct-effect estimands under the outcome model of p. 310", "location": "p. 312 (PDF p. 24); outcome model on p. 310 (PDF p. 22)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T24" }, { "declarations": [ { "claim_revision_id": "catalog:9d5f1ca85b1badcac339dc016d614c6bed5504d6a1061dc30796f173463c2c4e", "conditional_on": [], "credit_reason": "Negation of the larger conjunction does not isolate the group assertion; its already proved N1/N2 witness facts give the direct refutation.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.bernoulli_unbiasedness_as_printed_refuted", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/Wave1/BernoulliUnbiasedness.lean#L17", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:9d5f1ca85b1badcac339dc016d614c6bed5504d6a1061dc30796f173463c2c4e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T25", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T25" }, "html": { "body": "

The printed claim is false: within a group under Bernoulli assignment, the arm mean conditional on a nonempty arm need not be unbiased for the group average of marginal potential outcomes. p. 310 (PDF p. 22). The theorem negates the printed within-group assertion itself, and hence refutes the printed conjunction without quantifying over first-stage designs.

", "headline": "Section 16.5.1, the estimators of the marginal potential-outcome means and of the direct, indirect, total and overall effects, with their unbiasedness under Bernoulli assignment, as printed", "location": "p. 310 (PDF p. 22)" }, "id": "P045-T25", "text": { "body": "The printed claim is false: within a group under Bernoulli assignment, the arm mean conditional on a nonempty arm need not be unbiased for the group average of marginal potential outcomes. p. 310 (PDF p. 22). The theorem negates the printed within-group assertion itself, and hence refutes the printed conjunction without quantifying over first-stage designs.", "headline": "Section 16.5.1, the estimators of the marginal potential-outcome means and of the direct, indirect, total and overall effects, with their unbiasedness under Bernoulli assignment, as printed", "location": "p. 310 (PDF p. 22)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T25" }, { "declarations": [ { "claim_revision_id": "catalog:f5df365924e0e094a449dbc656d4c859a3e1c43e1e14a57638dca81dea8543ff", "conditional_on": [], "credit_reason": "Singleton arms are allowed on the printed toy domain; the row refutes the returned package quantity on that domain and does not refute the corrected Hudgens-Halloran formula on its valid arm-size domain.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.direct_variance_as_printed_refuted", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/Wave1/DirectVariance.lean#L16", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f5df365924e0e094a449dbc656d4c859a3e1c43e1e14a57638dca81dea8543ff", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T26", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T26" }, "html": { "body": "

pp. 310–311 (PDF pp. 22–23). The universal conjunction of additive-effect unbiasedness and conservativeness fails for the package statistic with singleton-arm variance contributions dropped.

", "headline": "Section 16.5.1, the property stated for the Hudgens–Halloran variance estimators and the toy example's variance output, as printed", "location": "p. 310 (PDF p. 22) and p. 311 (PDF p. 23)" }, "id": "P045-T26", "text": { "body": "pp. 310–311 (PDF pp. 22–23). The universal conjunction of additive-effect unbiasedness and conservativeness fails for the package statistic with singleton-arm variance contributions dropped.", "headline": "Section 16.5.1, the property stated for the Hudgens–Halloran variance estimators and the toy example's variance output, as printed", "location": "p. 310 (PDF p. 22) and p. 311 (PDF p. 23)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T26" }, { "declarations": [ { "claim_revision_id": "catalog:16db538d96f4ee31d279cd6b633fa9066faa026286bf2083c7240129df904060", "conditional_on": [], "credit_reason": "The documented package indicator is the component refuted; the chapter's intended full-neighborhood estimand and the provenance of Table 16.3 are not refuted.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.all_control_as_documented_refuted", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/Wave1/AllControl.lean#L16", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:16db538d96f4ee31d279cd6b633fa9066faa026286bf2083c7240129df904060", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T27", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T27" }, "html": { "body": "

The printed claim is false: the documented full-neighborhood control indicator differs from the indicator returned by the package. pp. 304–305 (PDF pp. 16–17), staged full-neighborhood mapping. The theorem negates their asserted equivalence on every simple undirected network and assignment.

", "headline": "Section 16.4.6, the full-neighbourhood estimand of Table 16.3, as printed, with the package function that implements its two exposure conditions", "location": "pp. 304–305 (PDF pp. 16–17); staged package R/make_exposure_map_AS.R, lines 104–121" }, "id": "P045-T27", "text": { "body": "The printed claim is false: the documented full-neighborhood control indicator differs from the indicator returned by the package. pp. 304–305 (PDF pp. 16–17), staged full-neighborhood mapping. The theorem negates their asserted equivalence on every simple undirected network and assignment.", "headline": "Section 16.4.6, the full-neighbourhood estimand of Table 16.3, as printed, with the package function that implements its two exposure conditions", "location": "pp. 304–305 (PDF pp. 16–17); staged package R/make_exposure_map_AS.R, lines 104–121" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T27" }, { "declarations": [ { "claim_revision_id": "catalog:1414546e3a5b8ed7ab55d17449905fc065acc48c0b482f1de320016e76808815", "conditional_on": [], "credit_reason": "Full refutation of the documented sampler's universal 3-net contract, not a numerical audit of Table 16.3; normalization need not be false to refute the conjunction.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.Wave1.cluster_sampler_as_documented_refuted", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/Wave1/ClusterSampler.lean#L24", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1414546e3a5b8ed7ab55d17449905fc065acc48c0b482f1de320016e76808815", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T29", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T29" }, "html": { "body": "

The printed claim is false: the package's cluster sampler as coded does not always return a three-net. p. 305 (PDF p. 17), staged cluster sampler. The theorem negates the documentation's universal contract for that implementation, with the actual singleton-number draw and retry traces included. The intended sampler that draws uniformly among unmarked vertices is outside this refutation.

", "headline": "Section 16.4.6 and Table 16.3, the cluster design ‘three-net clustering’, as printed, with the package's sampler for it", "location": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 57–115" }, "id": "P045-T29", "text": { "body": "The printed claim is false: the package's cluster sampler as coded does not always return a three-net. p. 305 (PDF p. 17), staged cluster sampler. The theorem negates the documentation's universal contract for that implementation, with the actual singleton-number draw and retry traces included. The intended sampler that draws uniformly among unmarked vertices is outside this refutation.", "headline": "Section 16.4.6 and Table 16.3, the cluster design ‘three-net clustering’, as printed, with the package's sampler for it", "location": "p. 305 (PDF p. 17); staged package R/make_tr_vec_bernoulli.R, lines 57–115" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T29" }, { "declarations": [ { "claim_revision_id": "catalog:9062e20dc2e3fecf7188c2964099202d15d27d225f63c8785ba187e73cfe4e3f", "conditional_on": [], "credit_reason": "The claim's formal statement already imposes the relevant positivity, and observed=y_i(D_i) represents correct specification. Empty realized cells remain empty sums, rather than being discarded.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ht_unbiased_headline", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L311", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:9062e20dc2e3fecf7188c2964099202d15d27d225f63c8785ba187e73cfe4e3f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T30", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T30" }, "html": { "body": "

pp. 297–298 (PDF pp. 9–10). Positive exact marginal exposure probabilities make the HT total and the exposure contrast unbiased under a correctly specified mapping.

", "headline": "Section 16.4.2, unbiasedness of the Horvitz–Thompson estimators of the exposure-specific total and of the contrast τ(dk,dl)\\tau(d_k,d_l)τ(dk​,dl​)", "location": "p. 297 (PDF p. 9) and p. 298 (PDF p. 10)" }, "id": "P045-T30", "text": { "body": "pp. 297–298 (PDF pp. 9–10). Positive exact marginal exposure probabilities make the HT total and the exposure contrast unbiased under a correctly specified mapping.", "headline": "Section 16.4.2, unbiasedness of the Horvitz–Thompson estimators of the exposure-specific total and of the contrast $\\tau(d_k,d_l)$", "location": "p. 297 (PDF p. 9) and p. 298 (PDF p. 10)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T30" }, { "declarations": [ { "claim_revision_id": "catalog:6f4b437a136b1c10772d1baea27c91e13ee8328a19fe7b91f35d6040c72c4496", "conditional_on": [], "credit_reason": "No causal identification theorem is claimed; the target is the potential-outcome decomposition itself.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.contagion_decomposition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_Spillover.lean#L4470", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:6f4b437a136b1c10772d1baea27c91e13ee8328a19fe7b91f35d6040c72c4496", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T31", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T31" }, "html": { "body": "

p. 315 (PDF p. 27). The indirect household effect has both contagion decompositions, and an unchanged mediator gives zero household contagion contribution.

", "headline": "Section 16.6, the decomposition of the indirect effect into a contagion effect and other mechanisms: θ=γ(1)+η(0)=γ(0)+η(1)\\theta = \\gamma(1) + \\eta(0) = \\gamma(0) + \\eta(1)θ=γ(1)+η(0)=γ(0)+η(1)", "location": "p. 315 (PDF p. 27)" }, "id": "P045-T31", "text": { "body": "p. 315 (PDF p. 27). The indirect household effect has both contagion decompositions, and an unchanged mediator gives zero household contagion contribution.", "headline": "Section 16.6, the decomposition of the indirect effect into a contagion effect and other mechanisms: $\\theta = \\gamma(1) + \\eta(0) = \\gamma(0) + \\eta(1)$", "location": "p. 315 (PDF p. 27)" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T31" }, { "declarations": [ { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"P045/P045_ExposureVariance.lean:1531 states positivity of retained smoothed marginals and moments calculated with cliqueWeights/cliqueJoints, but its conclusion contains neither their identification with the actual retained-table inputs nor probability-law normalization. T32 defines its estimators using the simulated counts, and this row expressly claims 'actual simulated-table formulas' and 'both normalized observation laws'. Those conclusions are supplied only by clique_retained_weights at line 754 and clique_probability_laws at line 1154, which are separately credited declarations; they are not conclusions of this compound declaration. Under the required row-level standard this is a material coverage omission, despite the fixed witness satisfying those facts and the numerical shortfalls being stated correctly.\", 'suggested_fix': \"Append to clique_variance_counterexample the retained-input equalities stated by clique_retained_weights and the nonnegativity/normalization conclusions stated by clique_probability_laws, using those existing declarations to discharge the added conjuncts. This preserves the row's promised complete witness without changing its finite-simulation scope.\", 'second_review': \"P045/P045_ExposureVariance.lean:1531 states T32(ii)-(v)'s exact moments and shortfalls and the two representative fixed-table values. Its estimators use the retained-count specializations at :713 and :719, identified with the target's table formulas at :754, and cliqueJointMass at :797 is already exactly the target's uniform 4356-pair or 4290-pair law. T32 defines these inputs and observation laws as the setting; it does not require separate input-identification and normalization certificates as additional conclusions of this moment component. The row description overattributes those companion certificates (notably :754 and :1154) to the compound declaration, so correct that description rather than append mathematically redundant conjuncts.\", 'severity': None}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_variance_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1511", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "The all-fixed-table result includes both representative tables.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_fixed_table_counterexamples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1053", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P045/P045_ExposureVariance.lean:727 quantifies over every unit and exposure for the marginal counts, and every ordered pair with both labels in {d01,d00} for the joint counts, including diagonals. With cliqueFullCount and cliqueFullJoint at lines 698 and 703, these are exactly the counting component of T32(i); the two-label restriction excludes no joint count requested for this contrast. The target's own opening identifies a stronger finite-simulation extension, so this auxiliary count theorem is not being required to refute the contextual exact-probability guarantee.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_full_counts", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L706", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P045/P045_ExposureVariance.lean:1154 states nonnegativity and normalization for both joint masses and normalization for every conditional observation mass, as this row promises. The definitions at lines 797 and 802 give mass 1/4356 to every independent pair, 1/4290 to each included pair with distinct coordinates, and conditional mass 1/66 or 1/65 on the specified support. Conditional nonnegativity follows immediately from those nonnegative constants; using rational finite masses loses none of T32's probability-law component. The target's stated stronger-extension scope also prevents the suggested attribution issue from becoming a fidelity defect in this row.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_probability_laws", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1133", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P045/P045_ExposureVariance.lean:989 states assignment injectivity and exactly two treated units, symmetry and a loopless graph without isolates, exposure-cell sizes (2,8) or (6,4), and the 18/48 profile split. These are exactly the structural assertions in T32(i) and in this row's description, with zero-based labels for the same twelve-unit graph. It introduces no additional assumption, and this structural component does not purport to establish a refutation of the chapter's exact-probability result.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_structure", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L968", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P045/P045_ExposureVariance.lean:754 identifies every retained marginal weight and all four contrasted joint arrays with the smoothed inputs computed from retainedF. smoothedWeight and smoothedJoint at lines 402 and 408 use exactly the claim's denominators R+1=66 and R=65 and the same-exposure diagonal correction. Restricting the joint labels to d01 and d00 loses no input used by T32's specified contrast, and rational representation is lossless. The stronger finite-simulation restriction comes from the claim's own opening, rather than being added only by the row description.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_retained_weights", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L733", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"AronowAudit/P045/P045_ExposureVariance.lean:1046 states the exact-input contrast for every treatment profile, with values -220/189 and 55/126 according to whether the treated pair shares a clique, as in T32(ii). Its marginal input is the full-design count divided by 66, independent of the nominal omission parameter because included is false. The 18/48 multiplicities are the elementary counts of same-clique and other unordered pairs in three four-cliques, also stated in clique_structure at line 989, so the representation excludes no profiles. The only correction needed here is to T32's shared closing attribution, which contradicts the target's corrected scope rather than these profilewise values.\", 'suggested_fix': \"In shared target T32, replace the final attribution that the chapter's assertion is false with: 'This disproves the stronger exact finite-simulation conservativeness extension for the specified simulated inputs; it does not establish failure of the chapter's contextual exact-probability guarantee or its R=10000 analysis.'\", 'second_review': 'P045/P045_ExposureVariance.lean:1046 gives T32(ii)\\'s exact-input contrast values -220/189 and 55/126 for every profile according to whether its treated pair shares a clique. cliqueObservationMarginal at P045/P045_ExposureVariance.lean:814 reduces to full-design counts divided by 66 when included is false; the 18/48 multiplicities are elementary counts in the explicit pair enumeration at :615, also recorded at :989. Thus the credited profilewise component is faithful, while the shared target\\'s final attribution goes beyond its opening \"stronger exact finite-simulation\" scope. The repair is the minor shared target-wording correction.', 'severity': None}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_exact_profile_estimates", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1025", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "Included by clique_variance_counterexample.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_exact_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1077", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "Included by clique_variance_counterexample.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_joint_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1419", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P045/P045_ExposureVariance.lean:1261 covers every omitted profile and supplies graph preservation together with assignment and exposure identities under the explicit permutation. Representatives 0 and 3 are the profiles treating zero-based pairs (0,1) and (0,4), hence the claim's units (1,2) and (1,5). The permutation's inverse gives the opposite orientation of the same classification, so this matches T32(vi) without restricting its omissions; the target-level attribution caveat creates no missing case in this component.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_relabeling_verified", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1240", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"AronowAudit/P045/P045_ExposureVariance.lean:1508 quantifies over every omitted assignment and states the actual variance, expected estimate, negative difference, and strict shortfall with the two exact cases in T32(vi). cliqueT and cliqueV at lines 786 and 791 use the specified smoothed marginal and joint inputs, while the observation law is the independent uniform law on all 66 assignments. Rational arithmetic loses no part of this fixed rational witness, and the casting identities at lines 1599-1658 preserve its real interpretation. The shared target's closing attribution to a refutation of the chapter conflicts with its opening and accepted correction scope, but that wording error does not invalidate this row's finite-simulation component.\", 'suggested_fix': \"In shared target T32, replace the final attribution that the chapter's assertion is false with: 'This disproves the stronger exact finite-simulation conservativeness extension for the specified simulated inputs; it does not establish failure of the chapter's contextual exact-probability guarantee or its R=10000 analysis.'\", 'second_review': 'P045/P045_ExposureVariance.lean:1508 states the two exact variance, expected-estimate and negative-difference cases for every omitted assignment, matching the fixed-table component of T32(vi). The definitions cliqueT and cliqueV at P045/P045_ExposureVariance.lean:786 and :791 use the specified retained-count inputs, whose identification is established at :754. The target opens with a \"stronger exact finite-simulation conservativeness extension\" but its closing attribution calls the chapter\\'s conservativeness assertion false, exceeding that corrected scope. Aligning that shared closing sentence with the existing opening is a minor claim-text correction and requires no change to this declaration.', 'severity': None}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_all_fixed_table_counterexamples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1487", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-02:P045-T32:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P045/P045_ExposureVariance.lean:1673 uses real-valued versions of the actual HT contrast and variance functional and states, for each of the two normalized nonnegative laws, that the expected estimate is below both the actual variance and 6050/11907. These are the strict-inequality conclusions of T32(iii)-(v), without added hypotheses or excluded realizations. T32 itself opens by specifying a counterexample to a stronger finite-simulation extension; its conditional closing 'Read as' sentence need not be read as an additional claim to refute the contextual exact-input theorem.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.clique_real_variance_shortfall", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L1635", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "CT-02:P045-T32:accepted", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T32", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T32" }, "html": { "body": "

pp. 298–299 (PDF pp. 10–11), audit counterexample to a stronger finite-simulation extension. The specified twelve-unit, three-clique witness has all the stated exposure counts, exact and simulated moments, strict shortfalls, and fixed-table and relabeling facts. Exact rational certificates use canonical real embedding; the final clause states the shortfall for the real-valued estimator under both sampling regimes. This refutes the stronger exact finite-simulation reading, without refuting the contextual exact-probability assertion.

", "headline": "A counterexample to an exact finite-simulation extension of the nonnegative-bias guarantee in Section 16.4.2", "location": "pp. 298-299 (PDF pp. 10-11), Section 16.4.2" }, "id": "P045-T32", "text": { "body": "pp. 298–299 (PDF pp. 10–11), audit counterexample to a stronger finite-simulation extension. The specified twelve-unit, three-clique witness has all the stated exposure counts, exact and simulated moments, strict shortfalls, and fixed-table and relabeling facts. Exact rational certificates use canonical real embedding; the final clause states the shortfall for the real-valued estimator under both sampling regimes. This refutes the stronger exact finite-simulation reading, without refuting the contextual exact-probability assertion.", "headline": "A counterexample to an exact finite-simulation extension of the nonnegative-bias guarantee in Section 16.4.2", "location": "pp. 298-299 (PDF pp. 10-11), Section 16.4.2" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T32" }, { "declarations": [ { "claim_revision_id": "catalog:43bfd6fa1da1003d3d159f60a92485f5b73783b759f6e394ef11508675d47403", "conditional_on": [], "credit_reason": "T33's exact surplus component; every custom functional was expanded, and the cross sum includes i=j. No theorem is assumed as a premise.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.exact_variance_surplus", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L551", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:43bfd6fa1da1003d3d159f60a92485f5b73783b759f6e394ef11508675d47403", "conditional_on": [], "credit_reason": "T33's nonnegative-bias component; exact marginals and joints are used as estimator inputs, with structural zeros allowed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.exact_variance_conservative", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L580", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:43bfd6fa1da1003d3d159f60a92485f5b73783b759f6e394ef11508675d47403", "conditional_on": [], "credit_reason": "T33's unbiasedness component; N>0, distinct exposures and positivity are explicit in the target, and finite Ω is a lossless atom representation of the finite design.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P045.ExposureVariance.ht_unbiased", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P045/P045_ExposureVariance.lean#L350", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:43bfd6fa1da1003d3d159f60a92485f5b73783b759f6e394ef11508675d47403", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P045-T33", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P045-T33" }, "html": { "body": "

pp. 298–299 (PDF pp. 10–11), Section 16.4.2. With positive exact marginal probabilities, the adjusted HT contrast variance estimate has precisely the structural-zero squared-outcome surplus, is conservative in expectation, and the point estimator is unbiased.

", "headline": "The exact-probability nonnegative-bias guarantee of Section 16.4.2, with its contextual scope explicit", "location": "pp. 298-299 (PDF pp. 10-11), Section 16.4.2" }, "id": "P045-T33", "text": { "body": "pp. 298–299 (PDF pp. 10–11), Section 16.4.2. With positive exact marginal probabilities, the adjusted HT contrast variance estimate has precisely the structural-zero squared-outcome surplus, is conservative in expectation, and the point estimator is unbiased.", "headline": "The exact-probability nonnegative-bias guarantee of Section 16.4.2, with its contextual scope explicit", "location": "pp. 298-299 (PDF pp. 10-11), Section 16.4.2" }, "url": "https://pmaronow.github.io/llm-audit/works/P045/#P045-T33" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "selected_recalculation", "targets": [ { "agreement": "to_rounding", "location": "p. 291 (PDF p. 3)", "method": "I checked the printed consumption figures: at 50% saturation, the rounded means of 235 USD and 188 USD average to 211.5 USD, compatible with the reported rounded pooled treatment-village mean of 211 USD; 217 USD is the control-village mean. This is an internal arithmetic comparison only. I could not rerun the underlying analysis without the household consumption data and calculation code.", "status": "recalculated", "target": "Figure 16.2 and accompanying consumption figures" }, { "agreement": "to_rounding", "location": "p. 300 (PDF p. 12)", "method": "I recalculated the RMSE identities from the printed bias and SD, allowing for rounding and the finite-replication correction for 3,000 simulations. I also checked that the printed average minus the true value equals the bias. These arithmetic comparisons agree to rounding. I could not rerun the original simulation without its driver, network, outcomes and seeds.", "status": "recalculated", "target": "Table 16.1" }, { "agreement": "to_rounding", "location": "p. 301 (PDF p. 13)", "method": "I recalculated the RMSE identities from the printed bias and SD, allowing for rounding and the finite-replication correction for 3,000 simulations. These arithmetic comparisons agree to rounding. I could not rerun the original simulation without its driver, network, outcomes and seeds.", "status": "recalculated", "target": "Table 16.2" }, { "agreement": "agrees", "location": "p. 302 (PDF p. 14); interpretation p. 303 (PDF p. 15)", "method": "I inspected and digitized all twelve panels. The dashed lines agree with the Table 16.2 true values within 0.1; the binned means minus the dashed lines agree with its biases within 0.45; the binned standard deviations agree with its SDs within 0.7. Agreement refers to these comparisons only. I could not regenerate the histograms without the original simulation driver, networks, outcomes, seeds and settings.", "status": "inspected_only", "target": "Figure 16.4" }, { "agreement": "agrees_in_part", "location": "p. 304 (PDF p. 16); description p. 303 (PDF p. 15)", "method": "I inspected and digitized the twelve histograms and compared the binned means with new simulations (P045-E012). The no-missing-ties column is approximately centered and consistent with Table 16.1; the other columns resemble simulations using the defective tie-removal function. This supports but does not establish that the function generated the figure, and the new simulations do not give a corrected figure. Agreement in part refers to these qualified comparisons only. I could not regenerate the original histograms or establish their source without the original inputs and generating code.", "status": "inspected_only", "target": "Figure 16.5" }, { "agreement": "agrees_in_part", "location": "p. 305 (PDF p. 17)", "method": "I recalculated the RMSE identities from the printed bias and SD, allowing for rounding and the finite-replication correction for 3,000 simulations. I also checked that the printed average minus the true value equals the bias. These arithmetic comparisons agree to rounding. I could not rerun the original simulation without its driver, network, outcomes and seeds. Separately, new simulations reproduce the ranking of designs and estimators but give smaller SDs and different biases; the missing original script prevents resolving the difference. Agreement in part covers both comparisons.", "status": "recalculated", "target": "Table 16.3" }, { "agreement": "disagrees", "location": "p. 312 (PDF p. 24); simulation specification pp. 311-312 (PDF pp. 23-24)", "method": "I inspected and digitized the figure and checked its labels against the stated outcome model. The first two column labels are interchanged (P045-E014); disagreement refers to that label check. After exchanging them and inferring a common baseline mean from the figure, the dashed lines agree with the implied estimands to plotting precision and the binned means agree with the derived estimator expectations within 0.003. I could not regenerate the histograms without the original driver, baseline outcomes or seed, assignment seeds and scenario settings.", "status": "inspected_only", "target": "Figure 16.6" } ] }, "summary": "This chapter reviews methods for estimating direct and spillover effects in randomized experiments under interference. It covers exposure mappings with Horvitz--Thompson and Hajek estimators on a known network, the Hudgens--Halloran estimators under partial interference, and the relation between interference and contagion, and it illustrates them with the \\texttt{interference} R package. 15 entries are classified as errors (5 Moderate, 10 Minor), and 9 are clarity notes; 5 further concerns were not confirmed, 3 remain unresolved and 4 fall outside the scope of this chapter. With Bernoulli assignment inside groups, the within-arm sample means of Section 16.5.1 are not generally unbiased under interference (\\fid{P045-E001}). I clarify the distinction between the exact-probability variance guarantee and its intended Monte Carlo approximation (\\fid{P045-N016}).", "what_was_checked": "I read all 31 pages of the chapter, with every display, table, figure, footnote and reference, together with the source code of two versions of the \\texttt{interference} package. The following points were checked and held up.\n\\begin{itemize}\n\\item The example with ten units (pp.~295--298). The printed adjacency matrix agrees with Figure 16.3 edge by edge. Taking that matrix as input and using the sampling algorithm that was the default before R 3.6.0, I reproduced the treatment assignment, the exposure matrix, the printed exposure probabilities and the contrasts $41.61043$ and $39.51643$ from the package code. The remaining entry of that output is the subject of \\fid{P045-E004}, and the form of the printed probabilities is the subject of \\fid{P045-N001}.\n\\item The Horvitz--Thompson estimator of p.~297 is exactly unbiased over the 45 assignments of that example when the exact exposure probabilities are used.\n\\item The variance code of the package implements the variance and covariance expressions described on pp.~298--299. For the Horvitz--Thompson estimators, the nonnegative-bias guarantee holds with exact marginal and joint exposure probabilities; finite Monte Carlo substitution need not preserve that exact guarantee (\\fid{P045-N016}).\n\\item Tables 16.1, 16.2 and 16.3. In every row of the three tables the RMSE equals $\\sqrt{\\text{Bias}^2+(2999/3000)\\text{SD}^2}$ up to rounding, with SD interpreted as a sample standard deviation over 3000 replications, and in every row of Tables 16.1 and 16.3 the bias equals the average value minus the true value. The true values of Tables 16.1 and 16.2 are in the ratios that the multipliers imply. In Table 16.1 the Hajek estimator has the smaller SD and RMSE in every comparison, and in Table 16.2 the biases have the signs that the text states.\n\\item Figure 16.4 agrees with Table 16.2 in all twelve panels: the dashed lines agree with the tabulated true values within $0.1$, the means estimated from the histogram bins minus the dashed lines agree with the tabulated biases within $0.45$, and the standard deviations estimated from those bins agree with the tabulated SDs within $0.7$.\n\\item The example with eighteen units (p.~311). The package reproduces all five point estimates and all five variances on a reconstructed dataset compatible with the six printed rows and every reported digit; this does not identify the original dataset or seed. The identities between the total, direct, indirect and overall estimates hold within rounding. Under the two-stage complete randomization of that example and stratified interference, the five point estimators are exactly unbiased over its 14,580 assignments. \\fid{P045-N008} concerns the two direct-effect variances.\n\\item Figure 16.6. After swapping the first two column labels (\\fid{P045-E014}) and using a common baseline mean inferred from the figure, the dashed lines agree with the estimands implied by the multipliers of the package to plotting precision, and the means estimated from the histogram bins agree with the expectations of the estimators derived from them within $0.003$.\n\\item The decomposition $\\theta=\\gamma(1)+\\eta(0)=\\gamma(0)+\\eta(1)$ on p.~315 follows from the printed definitions.\n\\item The arithmetic in the summaries of the empirical studies (Figure 16.2 and the experiment of Nickerson) agrees with the reported figures up to rounding.\n\\end{itemize}\nI did not have the scripts, seeds and draws behind Tables 16.1--16.3 and Figures 16.4--16.6, so I could not recompute those results from their source. I checked them by arithmetic, by reading the figures and by independent simulations on new network draws. These simulations reproduce the ordering of the estimators in Table 16.1 with similar magnitudes, give biases close to the printed ones in the cases of Table 16.2 that were simulated, and reproduce the ranking of designs and estimators in Table 16.3. They do not reproduce the magnitudes of Table 16.3: the simulated standard deviations are smaller and the biases differ, and without the script behind that table the difference could not be resolved. The works that the chapter cites and the empirical studies that it summarizes were consulted only on a few specific points. Otherwise, only the internal consistency of what the chapter reports about them was checked. I also examined the chapter's description of the experiment of Paluck, Shepherd and Aronow (2016); \\fid{P045-N018} clarifies the wording on p.~307. The chapter accurately attributes the school-composition finding to that study, with its causal interpretation conditional on the study's response model; no correction of that attribution is established.\n\nNo appendix or supplement to the chapter was identified. The replication script distributed with the package reproduces a table of Aronow and Samii (2017) and not the results of this chapter. These checks cover the points listed above. They do not establish that the rest of the chapter is correct.\n\nThe context of the variance discussion on pp.~294 and 298--299 naturally distinguishes an exact-probability theorem from accurate Monte Carlo approximation (\\fid{P045-N016}). The finite-draw counterexample shows that Monte Carlo substitution need not preserve the exact sign guarantee; it does not establish an error in the chapter's 10,000-draw calculations, its maximum variance estimator or a published simulation value, nor a confidence-interval coverage failure.", "work": { "arxiv_id": "2001.05444", "audit_complete": true, "authors": [ "P. M. Aronow", "Dean Eckles", "Cyrus Samii", "Stephanie Zonszein" ], "category": "Book Chapters", "citation": "P. M. Aronow, Dean Eckles, Cyrus Samii, and Stephanie Zonszein (2021). Spillover Effects in Experimental Data. In Advances in Experimental Political Science.", "doi": "10.1017/9781108777919.021", "edition": { "pages_in_audited_file": 31, "sha256": "0db5a6f02a106e561bd0aa527292dafd2f6f9852abb2374c959a64d40bcde5df", "source_url": "https://doi.org/10.1017/9781108777919.021", "venue": "Advances in Experimental Political Science", "year": "2021" }, "id": "P045", "later_materials": null, "title": "Spillover Effects in Experimental Data", "type": "Book chapter", "url": "https://pmaronow.github.io/llm-audit/works/P045/", "venue": "Advances in Experimental Political Science", "year": "2021" } }