LLM-generated corrections to the published research of P. M. Aronow

Journal article P029

Corrections to: Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable

P. M. Aronow and Allison Carnegie (2013). Beyond LATE: Estimation of the Average Treatment Effect with an Instrumental Variable. Political Analysis 21(4):492–506.

DOI
10.1093/pan/mpt013
Audited edition
Political Analysis, 2013, 21(4):492–506; 15 pages in the audited file
SHA-256 of the audited file
d1956303080df64032982932790f3499309cff77117db63f967e2c71efd418c6
Permanent address
https://pmaronow.github.io/llm-audit/works/P029/

Summary

The paper asks how to estimate the average treatment effect for a whole population when instrumental variables estimation recovers only the effect among compliers. It weights units by the inverse of an estimated compliance score, shows consistency under its stated assumptions, and reanalyses two experiments. 23 entries are classified as errors (4 Moderate, 19 Minor), and 3 are clarity notes; 2 of the error entries, P029-E006 and P029-N006, concern the same statement of footnote 6; 7 other possible errors were not confirmed, are undetermined or lie outside the paper. The finding that matters most is P029-E004: when treatment effects are marginally independent of compliance, the weighted estimator can be inconsistent for the ATE although two-stage least squares is consistent.

What was and was not checked

I checked the following through derivations, recalculation from the replication files or comparison with the pages; they held up, subject to the corrections cited.

  • Equations (3) and (4): under the five assumptions of Section 2.2, the ratio of the two intention-to-treat effects is the LATE, and the sample ratio equals 2SLS without covariates. Footnote 2: with a randomized instrument, 2SLS with covariates is consistent for the LATE.
  • Section 3.1: the probabilities in equations (5) to (7) sum to one and give the Bernoulli likelihood. Under one-sided noncompliance the likelihood reduces to a probit on the units assigned to treatment, after the misprint of P029-E008 is corrected.
  • The worked example on pp. 497–498: the ATE is 0.50.5, the LATE is 0.05/0.425=2/17≈0.120.05/0.425 = 2/17 \approx 0.12, and the reweighted value is 0.50.5.
  • Section 3.3 and the Appendix: the weighted ratio agrees with weighted 2SLS. Each step of equations (8) to (14) holds under Assumptions 1 and 3 and regularity conditions, after the corrections of notation in P029-E009. The weighted estimator converges to E(Y1−Y0)E(Y_1 - Y_0) when ZZ is independent of the covariates and the potential outcomes and the scores are consistent.
  • Section 3.4: the integral formula for the LATE on p. 499 and the limit of ICSW when Assumption 1 fails, which follows from equations (10) and (13). Constant treatment effects satisfy Assumption 1 and give the same limit for 2SLS and ICSW. Footnote 4: with saturated covariates and a constant share assigned to treatment, ICSW equals the average of the Wald estimates of the strata, weighted by the shares of the strata.
  • Table 1: all point estimates follow from the data and the script when the indicator for missing age is set from the raw data (P029-E021). The estimates for contact are 5.615.61 and 6.676.67, with ratio 1.1891.189. With the full 2,500 bootstrap draws, I find 5.6075.607 (2.230,8.892)(2.230, 8.892) and 6.6686.668 (3.204,10.292)(3.204, 10.292), which agree with the printed intervals to rounding. For the intervals (−0.2,2.3)(-0.2, 2.3) and (0.0,2.2)(0.0, 2.2) of footnote 11, I find (−0.17,2.32)(-0.17, 2.32) and (0.01,2.15)(0.01, 2.15).
  • Table 2: all point estimates follow from the data (0.2760.276, 0.4000.400, −0.070-0.070 and −0.050-0.050, with ratio 1.4491.449). The bootstrap intervals, including those of footnote 13, agree within Monte Carlo error on the full 2,500 draws.
  • Sample facts: n=18,933n = 18{,}933 in six cities, with no contact among controls; n=507n = 507, where 54.8% of those assigned did not watch and 4.4% of those not assigned watched.
  • Footnotes 6 and 12: the Winsorization in the code is the one described, with quantile levels n−0.275n^{-0.275} approximately 0.0670.067 and 0.1800.180 in the two applications. The reason stated in footnote 6 for ignoring Winsorization in the limit is corrected in P029-E006. The weighting for missing outcomes is as described and is inside the bootstrap.
  • The intervals quoted in the text agree with Tables 1 and 2, and footnotes 11 and 13 state that the 95% intervals for the differences between the two estimates include zero.
  • Supplement: Tables A7 and A8 follow from the data, Table A7 with the indicator for missing age set from the raw data. The simulation designs agree with the code apart from the notation slips (P029-E009), the scale and coefficient-draw omissions (P029-A001) and the benchmark of the homogeneous design (P029-E011). The large-sample limits 0.7310.731 for 2SLS, 2.492.49 for OLS on ZZ and 0.600.60 for ICSW with only X1X1 agree with Tables A1 and A5. The three panels of Figure A1 for N=500N = 500 were obtained again by running the script. The coverage in Table A6 agrees with footnote 7, and it is coverage of the sample-average effect (P029-N009).
  • Every work cited in the text has an entry in the reference list, and every entry is cited.

I did not rerun the simulations for n≥1,000n \ge 1{,}000, the four simulations with misspecification or the bootstrap simulation; I read their code against the supplement and compared the tables with the figures. The saved outputs of the simulations are not among the replication files, so I could not compute the corrected Table A3. I did not run the path of the package icsw that uses genetic optimization. The versions of R and of minqa used in 2013 are not stated; I used R 4.4.2 and minqa 1.2.8. The full-bootstrap output reported 39 of 2,500 compliance-score fits at the evaluation cap, retained as in the script; I did not independently confirm that count. I confirmed one capped fit among the first four draws (P029-N010).

Of the works cited, I consulted only those needed for particular questions: Deaton’s essay in both editions and the publishers' records for the entries of P029-E020, Arceneaux and Nickerson (2009), Green, Gerber and Nickerson (2003) and Albertson and Lawrence (2009). The other cited works were not read, and the classification of the 34 articles behind the 6% in the introduction was not available. Three questions about the coding of variables in the debate study are undetermined, because the cited article and the data files do not settle which description of the variables is right. I did not search for published errata. That this audit lists no further corrections is not a finding that the rest of the paper is correct.

Headline claims and related corrections

  1. With a binary instrument and the five potential-outcome assumptions, IV estimation identifies the average treatment effect among compliers (the LATE), which can differ from the ATE for the whole population when effects are heterogeneous.

    Abstract; Section 2; equations (3)-(4); footnote 2; the gender example in Section 3.2

    See P029-E001, P029-E002.

  2. Weighting units by the inverse of the compliance score and then applying IV/2SLS (ICSW) consistently estimates the ATE, given a randomized instrument and the usual IV assumptions, latent ignorability of compliance given covariates (Assumption 1), a consistent compliance-score estimator (Assumption 2) and compliance scores bounded away from zero (Assumption 3).

    Abstract; introduction p. 493; Sections 3.3-3.4; Assumptions 1-3; Appendix 'ICSW Consistency', eqs. (8)-(14)

    See P029-E003, P029-E004, P029-E006, P029-E009.

  3. A new nested maximum-likelihood estimator of the compliance score handles two-sided noncompliance (both always-takers and never-takers), reduces to binomial (probit) regression on the assigned-to-treatment units under one-sided noncompliance, and reduces to strata means under saturation.

    Introduction p. 493; Section 3.1, eqs. (5)-(7), footnote 3; Assumption 2; the compliance-score code (Dataverse icsw.R and the icsw package)

    See P029-E005, P029-E008, P029-E009, P029-E014, P029-E016, P029-E017, P029-N010.

  4. Under Assumptions 2 and 3 and the five IV assumptions, if latent ignorability fails, the paper says ICSW still converges to an internally valid causal effect: the complier effect reweighted to the covariate distribution of the full population, ∫E(Y1−Y0∣D1>D0,X=x)f(x) dx\int E(Y_1-Y_0\mid D_1>D_0,\mathbf{X}=\mathbf{x})f(\mathbf{x})\,d\mathbf{x}. The paper presents 2SLS as the special case with no covariates in the compliance model.

    Introduction p. 493; Section 3.4 pp. 499-500; Appendix eqs. (10) and (13); the two integral displays on p. 499

    See P029-E004.

  5. In Green, Gerber and Nickerson (2003), the paper reports estimated turnout effects of 5.65.6 percentage points among compliers (interval 2.22.2 to 8.98.9) and 6.76.7 points in the full population (interval 3.23.2 to 10.310.3), making the ATE estimate 19%19\% larger than the LATE estimate.

    Section 4.1, pp. 500-502; Table 1; footnotes 10-11; supplement Table A7

    See P029-E010, P029-E021, P029-N004.

  6. In Albertson and Lawrence (2009), the paper reports estimated effects of watching the debate on self-reported knowledge of 0.280.28 among compliers and 0.400.40 in the full population (approximately 45%45\% larger), while the estimated effects on support for the proposition are similar (−0.07-0.07 and −0.05-0.05) and indistinguishable from zero.

    Section 4.2, pp. 502-503; Table 2; footnotes 12-13; supplement Table A8

    See P029-E010, P029-E012, P029-N006, P029-N010.

  7. The paper recommends the Winsorization constant α=0.275\alpha=0.275 based on its simulations. Its supplementary tables report lower RMSEs for ICSW than for 2SLS under the correctly specified, “Wrong Model” and “Only X1” designs, and similar RMSEs under the two noise designs. The paper says that the bootstrap has conservative coverage in small samples and proper coverage in larger ones.

    Footnotes 6 and 7, p. 498; supplement Figures A1-A3 and Tables A1-A6

    See P029-E009, P029-E011, P029-A001, P029-N009.

  8. The two applications show that the LATE has the potential to differ substantively from the ATE, so researchers should attend to the estimand; when the ATE is the target, ICSW may be a more sensible estimator than standard IV, given the additional assumptions and the finite-sample uncertainty the paper describes.

    Abstract; introduction p. 493; Section 3.4 pp. 499-500; conclusion p. 503; Tables 1 and 2; footnotes 11 and 13

    See P029-E004, P029-E006, P029-E011, P029-E016, P029-N009.

Corrections

Moderate (4)

A real substantive error with limited reach.

Assumption 1 is not less restrictive than the ignorability assumption displayed beside it: under two-sided noncompliance neither assumption implies the other.

Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1

See also under Formal statements (Lean): P029-T2, P029-T20

As printed

Assumption 1 entails that, for all x∈Supp(X)\mathbf{x} \in \mathrm{Supp}(\mathbf{X}), E(Y1−Y0∣X=x)=E(Y1−Y0∣D1>D0,X=x)\mathrm{E}(Y_1 - Y_0|\mathbf{X}=\mathbf{x}) = \mathrm{E}(Y_1 - Y_0|D_1 > D_0, \mathbf{X}=\mathbf{x}): […]

However, Assumption 1 may be distinguished from the typical ignorability assumption (Rubin 1978), which requires that both E(Y1∣D=1,X=x)=E(Y1∣D=0,X=x)\mathrm{E}(Y_1|D=1,\mathbf{X}=\mathbf{x}) = \mathrm{E}(Y_1|D=0,\mathbf{X}=\mathbf{x}) and E(Y0∣D=1,X=x)=E(Y0∣D=0,X=x)\mathrm{E}(Y_0|D=1,\mathbf{X}=\mathbf{x}) = \mathrm{E}(Y_0|D=0,\mathbf{X}=\mathbf{x}). Assumption 1 is less restrictive than the typical ignorability assumption as there is no need to account for baseline selection bias, only the determinants of treatment effect heterogeneity.

The problem

The sentence ranks two conditions at a fixed covariate profile. The ignorability assumption equates the mean of each potential outcome between the units that received treatment and the units that did not. Assumption 1 equates the mean effect among compliers with the mean effect among all units. To say that Assumption 1 is less restrictive is to say that ignorability implies it. The reason given, that Assumption 1 need not account for baseline selection, shows only that Assumption 1 can hold when ignorability fails. Under two-sided noncompliance the implication fails as well. The treated group mixes compliers with always-takers, and the untreated group mixes compliers with never-takers. In the two populations below, ignorability compares always-takers with never-takers and places no restriction on complier means. Assumption 1 compares compliers with everyone else. The method covers two-sided noncompliance (p. 496), and the Albertson and Lawrence application has it. Someone choosing between ICSW and a method that assumes selection on observables is told that the ICSW assumption is the weaker of the two. It is a different assumption.

Evidence an exact counterexample

I use two exact counterexamples with a single covariate profile. In both, a unit is a complier (CC), an always-taker (AA) or a never-taker (NN) with probabilities 1/21/2, 1/41/4 and 1/41/4, and Z∼Bernoulli(1/2)Z\sim\mathrm{Bernoulli}(1/2) is independent of the type. The five IV assumptions hold, the difference in treatment rates between the arms is 1/21/2, and Assumption 3 holds with ϵ=1/4\epsilon=1/4. Half of the treated units and half of the untreated units are compliers.

Potential outcomes (Y0,Y1)(Y_0,Y_1) by type in the two populations.
TypeProbabilityPopulation 1Population 2
Complier1/21/2(0,1)(0,1)(0,1)(0,1)
Always-taker1/41/4(0,0)(0,0)(1,2)(1,2)
Never-taker1/41/4(0,0)(0,0)(0,1)(0,1)
The two assumptions in the two populations.
QuantityPopulation 1Population 2
E(Y0∣D=1)\mathrm{E}(Y_0|D=1) and E(Y0∣D=0)\mathrm{E}(Y_0|D=0)00 and 001/21/2 and 00
E(Y1∣D=1)\mathrm{E}(Y_1|D=1) and E(Y1∣D=0)\mathrm{E}(Y_1|D=0)1/21/2 and 1/21/23/23/2 and 11
Ignorability, as displayedholdsfails
E(Y1−Y0)\mathrm{E}(Y_1-Y_0) and E(Y1−Y0∣D1>D0)\mathrm{E}(Y_1-Y_0|D_1>D_0)1/21/2 and 1111 and 11
Assumption 1failsholds
Limit of ICSW minus the ATE1/21/200

Note that in population 1 the pair (Y0,Y1)(Y_0,Y_1) is independent of DD, which is stronger than the displayed equalities. In general, with μtd\mu_{td} the mean of YdY_d for type tt and τt=μt1−μt0\tau_t=\mu_{t1}-\mu_{t0}, ignorability holds in these populations if and only if μAd=μNd\mu_{Ad}=\mu_{Nd} for d=0,1d=0,1, and Assumption 1 holds if and only if τC=(τA+τN)/2\tau_C=(\tau_A+\tau_N)/2.

The implication holds in one case. Suppose that noncompliance is one-sided, D0=0D_0=0 for every unit, with random assignment and positive probabilities of both assignment arms and compliance at each covariate profile x\mathbf{x}. If ignorability holds, let mdm_d be the common value of E(Yd∣D,X=x)\mathrm{E}(Y_d|D,\mathbf{X}=\mathbf{x}), so that E(Yd∣X=x)=md\mathrm{E}(Y_d|\mathbf{X}=\mathbf{x})=m_d. The treated units are the compliers assigned to treatment, and I have md=E(Yd∣D1>D0,Z=1,X=x)=E(Yd∣D1>D0,X=x)m_d=\mathrm{E}(Y_d|D_1>D_0,Z=1,\mathbf{X}=\mathbf{x})=\mathrm{E}(Y_d|D_1>D_0,\mathbf{X}=\mathbf{x}). Subtracting the identity for d=0d=0 from the identity for d=1d=1, I obtain Assumption 1.

What survives

Assumption 1 and the consistency argument, which invokes Assumption 1 between equations (10) and (11), are unaffected. The observation that Assumption 1 permits baseline selection and the example of the medical trial also stand. This correction changes no estimate. The ranking is withdrawn. In an application with two-sided noncompliance, the reading of the ICSW estimate as an ATE rests on Assumption 1 itself, and ignorability of the treatment received does not supply it. Whether Assumption 1 holds in either application is not established here.

Corrected

However, Assumption 1 may be distinguished from the typical ignorability assumption (Rubin 1978), which requires that both E(Y1∣D=1,X=x)=E(Y1∣D=0,X=x)\mathrm{E}(Y_1|D=1,\mathbf{X}=\mathbf{x}) = \mathrm{E}(Y_1|D=0,\mathbf{X}=\mathbf{x}) and E(Y0∣D=1,X=x)=E(Y0∣D=0,X=x)\mathrm{E}(Y_0|D=1,\mathbf{X}=\mathbf{x}) = \mathrm{E}(Y_0|D=0,\mathbf{X}=\mathbf{x}). Assumption 1 and the typical ignorability assumption displayed here impose different restrictions and are not generally ordered. Assumption 1 permits baseline selection bias, provided that the conditional average treatment effect for compliers equals that for the full population.

Under one-sided noncompliance (D0=0D_0=0), random assignment and positive probabilities of both assignment arms and compliance at each covariate profile, the typical ignorability assumption implies Assumption 1.

Crop of five lines of text. The first lines display the two equalities of the typical ignorability assumption, for Y1 and for Y0 given D = 1 and D = 0. Two red boxes mark the words 'Assumption 1 is less restrictive than the typical ignorability assumption', which are followed by 'as there is no need to account for baseline selection bias'.
Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1. The marked words rank Assumption 1 below the ignorability assumption displayed in the lines above.

Independence of the treatment effects from compliance does not imply Assumption 1: the sentence requires that the mean effect among compliers equal the population mean effect both unconditionally, for 2SLS, and given every covariate profile, for Assumption 1 and ICSW.

Section 3.4, p. 499 (PDF p. 8), last sentence of the paragraph beginning “We can describe sufficient conditions for and consequences of Assumption 1”

See also under Formal statements (Lean): P029-T3, P029-T4, P029-T21

As printed

More generally, even if treatment effects are heterogeneous, but still statistically independent of compliance, Assumption 1 would necessarily be satisfied, and both 2SLS and ICSW would be consistent for the ATE.

The problem

Assumption 1 is a condition given the covariates: the mean effect among compliers with profile x\mathbf{x} equals the mean effect among all units with profile x\mathbf{x}. Independence of the effects from compliance, without qualification, is independence in the whole population. It implies that the LATE equals the ATE, and 2SLS is then consistent for the ATE. It does not give the conditional equality, because effects and compliance that are independent overall can be dependent among the units that share a covariate value. ICSW with correct scores then converges to the average over x\mathbf{x} of the complier effects, which is not the ATE in general. If the independence is read as conditional on the covariates, Assumption 1 holds and ICSW is consistent, but 2SLS converges to the LATE, which then differs from the ATE in general. Neither of these two readings makes the whole sentence true. The sentence suggests that ICSW cannot hurt when 2SLS already recovers the ATE. The first population below shows that weighting on covariates can introduce a bias.

Evidence an exact counterexample

The evidence is two exact counterexamples and the identities that explain them. I write τ=Y1−Y0\tau=Y_1-Y_0, f(x)=Pr⁡(X=x)f(\mathbf{x})=\Pr(\mathbf{X}=\mathbf{x}), p(x)=Pr⁡(D1>D0∣X=x)p(\mathbf{x})=\Pr(D_1>D_0|\mathbf{X}=\mathbf{x}) and μ(x)=E(τ∣D1>D0,X=x)\mu(\mathbf{x})=\mathrm{E}(\tau|D_1>D_0,\mathbf{X}=\mathbf{x}). Under the IV assumptions, with a discrete covariate and correct scores, the limits of 2SLS and of ICSW are

θIV=∑xf(x)p(x)μ(x)Pr⁡(D1>D0),θICSW=∑xf(x)μ(x),θIV−θICSW=Cov⁡{p(X),μ(X)}Pr⁡(D1>D0).\theta_{\mathrm{IV}}=\frac{\sum_{\mathbf{x}}f(\mathbf{x})p(\mathbf{x})\mu(\mathbf{x})}{\Pr(D_1>D_0)},\qquad \theta_{\mathrm{ICSW}}=\sum_{\mathbf{x}}f(\mathbf{x})\mu(\mathbf{x}),\qquad \theta_{\mathrm{IV}}-\theta_{\mathrm{ICSW}}=\frac{\operatorname{Cov}\{p(\mathbf{X}),\mu(\mathbf{X})\}}{\Pr(D_1>D_0)} .

Assumption 1 implies that θICSW\theta_{\mathrm{ICSW}} equals the ATE, ∑xf(x)E(τ∣X=x)\sum_{\mathbf{x}}f(\mathbf{x})\mathrm{E}(\tau|\mathbf{X}=\mathbf{x}).

Both populations have a binary covariate XX, a complier indicator CC, an effect τ∈{0,1}\tau\in\{0,1\}, outcomes Y0=0Y_0=0 and Y1=τY_1=\tau, and Z∼Bernoulli(1/2)Z\sim\mathrm{Bernoulli}(1/2) independent of everything else. A unit that is not a complier is an always-taker or a never-taker with probability 1/21/2 each, independently. In population A, CC and τ\tau are independent Bernoulli(1/2)\mathrm{Bernoulli}(1/2) variables, and Pr⁡(X=1∣C,τ)\Pr(X=1|C,\tau) is 3/43/4 if C=τ=1C=\tau=1 and 1/41/4 otherwise. In population B, X∼Bernoulli(1/2)X\sim\mathrm{Bernoulli}(1/2) and, given XX, the variables CC and τ\tau are independent with Pr⁡(C=1∣X)=Pr⁡(τ=1∣X)\Pr(C=1|X)=\Pr(\tau=1|X) equal to 1/41/4 at X=0X=0 and 3/43/4 at X=1X=1.

The two populations by covariate value.
Population and stratumf(x)f(x)p(x)p(x)E(τ∣X=x)\mathrm{E}(\tau|X=x)μ(x)\mu(x)
A, X=0X=05/85/82/52/52/52/51/41/4
A, X=1X=13/83/82/32/32/32/33/43/4
B, X=0X=01/21/21/41/41/41/41/41/4
B, X=1X=11/21/23/43/43/43/43/43/4
The sentence of p. 499 in the two populations.
QuantityPopulation APopulation B
τ\tau independent of compliance typeyesno
τ\tau independent of compliance type given XXnoyes
Assumption 1failsholds
ATE1/21/21/21/2
Limit of 2SLS (the LATE)1/21/25/85/8
Limit of ICSW7/167/161/21/2

In population A, I have θICSW=(5/8)(1/4)+(3/8)(3/4)=7/16\theta_{\mathrm{ICSW}}=(5/8)(1/4)+(3/8)(3/4)=7/16. In population B, I have θIV=(5/16)/(1/2)=5/8\theta_{\mathrm{IV}}=(5/16)/(1/2)=5/8. Assumption 3 holds with ϵ=1/3\epsilon=1/3 in A and ϵ=1/8\epsilon=1/8 in B, and the saturated compliance model of equations (5)–(7) fits both populations, so that the scores are correct and bounded. Solving the IV moment equations in exact arithmetic, with and without XX as a regressor, I find the same four limits.

What survives

The consistency result in the appendix, which applies Assumption 1 between equations (10) and (11), is unaffected by this finding, and so are the preceding sentences on a constant effect, the estimator and every reported number. The sufficient condition is withdrawn. The repair adds a restriction on the effects among compliers, which the applications may not satisfy. Whether they satisfy it is not established here.

Corrected

More generally, with heterogeneous treatment effects, if the mean treatment effect among compliers equals the population mean both unconditionally and conditional on every covariate profile X\mathbf{X} used in the compliance score, Assumption 1 holds and, under the other stated assumptions and regularity conditions, both 2SLS and ICSW are consistent for the ATE.

In symbols, the two conditions are E(Y1−Y0∣D1>D0)=E(Y1−Y0)\mathrm{E}(Y_1-Y_0|D_1>D_0)=\mathrm{E}(Y_1-Y_0) and E(Y1−Y0∣D1>D0,X=x)=E(Y1−Y0∣X=x)\mathrm{E}(Y_1-Y_0|D_1>D_0,\mathbf{X}=\mathbf{x})=\mathrm{E}(Y_1-Y_0|\mathbf{X}=\mathbf{x}) for every x∈Supp⁡(X)\mathbf{x}\colorbox{#ffe58a}{$\in\operatorname{Supp}(\mathbf{X})$}. The first suffices for 2SLS, and the second, which is Assumption 1, suffices for ICSW.

Crop of six lines of text. Red boxes mark the sentence 'More generally, even if treatment effects are heterogeneous, but still statistically independent of compliance, Assumption 1 would necessarily be satisfied, and both 2SLS and ICSW would be consistent for the ATE.'
Section 3.4, p. 499 (PDF p. 8). The marked sentence states a sufficient condition for Assumption 1 and for the consistency of both estimators.
Two rows of panels with a common horizontal axis, the probability limit of the estimator, marked at 3/8, 7/16, 1/2, 9/16 and 5/8; a vertical line stands at the ATE of 1/2. In population A, effects independent of compliance but not given X, the 2SLS point (blue circle) lies on the line at 1/2 and the ICSW point (orange diamond) lies to the left at 7/16. In population B, effects independent of compliance given X but not marginally, the 2SLS point lies to the right at 5/8 and the ICSW point lies on the line at 1/2.
Probability limits of 2SLS and ICSW in the two exact populations, against the ATE of 1/21/2 (vertical line). The marginal and conditional readings of the sentence on p. 499 each fail for one estimator. In population A the ICSW limit is 7/167/16, and in population B the 2SLS limit is 5/85/8.

In the R package icsw (version 1.0.0) the observation weights must enter the fit of the compliance score: as published, the one-sided probit omits them and the two-sided objective puts them inside the logarithm, so that the bootstrap does not refit the scores to its weights.

Section 3.3, p. 498 (PDF p. 7), last paragraph; R package icsw, version 1.0.0: R/compliance.score.R, lines 43–47 and 91, R/icsw.tsls.R, lines 28–37, and the help page of icsw.tsls

See also under Formal statements (Lean): P029-T7, P029-T8, P029-T23

As printed

Article, p. 498: To characterize uncertainty, we recommend bootstrapping the entire process—from computing the compliance score to weighted IV estimation (Abadie 2002).7^{7}

Package, R/compliance.score.R, lines 43–47 (line breaks removed; the call has no weights argument):

fit <- glm(D ~ W, df, subset = (Z == 1), family = binomial(link="probit"))\texttt{fit <- }\colorbox{#ffe58a}{$\texttt{glm(D \textasciitilde{} W, df, subset = (Z == 1), family = binomial(link="probit"))}$}

Line 91:

return(-sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights)))\texttt{return(-sum(log((p.D)\textasciicircum D * (1 - p.D)\textasciicircum (1 - D)}\colorbox{#ffe58a}{$\texttt{ * weights}$}\texttt{)))}

R/icsw.tsls.R, lines 29–32 (line breaks removed; the call continues):

weights.r <- rexp(n.obs, 1) # Bayesian bootstrap

coefs.r <- icsw.tsls.fit(D, X, Y, Z, W, weights = weights * weights.r,

Help page of icsw.tsls: Estimate average treatment effects using two-stage least squares with a binary instrument and binary treatment and weighting with inverse complier scores (probabilities of compliance). Optionally, bootstrap the entire estimation process for the purpose of hypothesis testing and constructing confidence intervals.

The problem

The article recommends that the bootstrap repeat the whole procedure, the fit of the compliance score included, and that recommendation is correct. The package says on its help page that it bootstraps “the entire estimation process”, and it does so by reweighting. Each replicate draws exponential weights, multiplies them into the observation weights and calls the fitting routine again. The compliance model is not fitted with those weights. In the one-sided branch the probit is fitted by glm without a weights argument, and the fitted scores are the same in every replicate. In the two-sided branch the weights multiply the likelihood contribution inside the logarithm. This adds a term to the objective that does not depend on the parameters. The set of minimizers is then the same for every vector of positive weights. There are three consequences. The package’s bootstrap standard errors leave out the variation of the first step, contrary to the article and to the help page. Sampling weights or missing-data weights that the user supplies are not used to weight the compliance-model log likelihood, and this can change the estimate itself. A weight of zero makes the two-sided objective infinite at every parameter value. The example on the help page sets the weight to zero for the 9 of its 507 observations with a missing outcome.

Evidence an exact counterexample

The evidence is a derivation for the two-sided objective, an exact counterexample for the one-sided branch, and a run of the package’s code. Let Li(θ)=pi(θ)Di{1−pi(θ)}1−DiL_i(\theta)=p_i(\theta)^{D_i}\{1-p_i(\theta)\}^{1-D_i}, where the code keeps pi(θ)p_i(\theta) in [10−5,1−10−5][10^{-5},1-10^{-5}]. For positive weights wiw_i, the objective on line 91 is

−∑ilog⁡{Li(θ)wi}=−∑ilog⁡Li(θ)−∑ilog⁡wi,-\sum_i\log\{L_i(\theta)w_i\}=-\sum_i\log L_i(\theta)-\sum_i\log w_i ,

and the last sum does not depend on θ\theta. The weighted negative log-likelihood is −∑iwilog⁡Li(θ)-\sum_i w_i\log L_i(\theta). If some wi=0w_i=0, then log⁡{Li(θ)wi}=−∞\log\{L_i(\theta)w_i\}=-\infty for every θ\theta.

The counterexample has 16 observations, four in each cell of a binary covariate WW and the assignment ZZ. No unit assigned to control is treated. Among the assigned units, one of four is treated at W=0W=0 and three of four at W=1W=1. Each treated unit has weight 99 and every other unit has weight 11. A probit with an intercept and WW is saturated, so that the fitted score at each value of WW is the proportion treated among the assigned units, weighted when the fit uses the weights. The outcome is Y=D(1+W)Y=D(1+W), and, with an intercept and no other covariates in the IV fit, the weighted IV estimate reduces to ∑Zi=1viYi/∑Zi=1viDi\sum_{Z_i=1}v_iY_i/\sum_{Z_i=1}v_iD_i with viv_i the weight divided by the score.

Exact one-sided example with 16 observations and weights 9 and 1.
QuantityPackage 1.0.0Weights in the probit
Compliance score, W=0W=01/41/49/(9+3)=3/49/(9+3)=3/4
Compliance score, W=1W=13/43/427/(27+1)=27/2827/(27+1)=27/28
ICSW coefficient on DD(36+72)/(36+36)=3/2(36+72)/(36+36)=3/2(12+56)/(12+28)=17/10(12+56)/(12+28)=17/10

Running the unmodified package code on these data, I find scores (0.25,0.75)(0.25,0.75) with and without the weights and a coefficient of 1.51.5. With weights = weights added to the call, the coefficient is 1.71.7. The first step also matters for the bootstrap. Starting from unit weights, I change the weight of one untreated assigned unit at W=0W=0 from 11 to tt. The package’s coefficient stays at 3/23/2, while refitting the score gives (t+11)/(t+7)(t+11)/(t+7), with derivative −1/16-1/16 at t=1t=1. In a trace of four bootstrap replicates of the one-sided code on the same DD, ZZ and WW, the scores are identical in the original fit and in every replicate. Each bootstrap coefficient equals the second stage computed with the scores held fixed.

Three points are not established. The size and the direction of the error in the standard errors are unknown. Leaving out the first step removes a variance and a covariance, and the sign of their sum is not determined. In the two-sided branch the weights do enter a regression for starting values, and a numerical optimizer may return different approximate fits from different starts, although the objective has the same minimizers. The default optimizer for the two-sided branch (rgenoud) was not run, and I do not state what the example on the help page returns.

What survives

This package defect does not implicate the article’s mathematical results or reported estimates. The article’s own scripts do not use this package. The one-sided application fits a weighted probit directly and refits it on resampled units. The two-sided application and the bootstrap simulations resample rows and use the helper in the replication files, whose two-sided objective has the weights outside the logarithm. The one-sided branch of that helper has the same misplaced weights, but the article’s one-sided application does not use it. With unit weights and no bootstrap, the package’s compliance models use ordinary unweighted objectives. Package results with nonconstant weights, and all bootstrap standard errors from the package, are to be recomputed after the repair.

Corrected

In R/compliance.score.R, lines 43–47:

fit <- glm(D ~ W, df, subset = (Z == 1), weights = weights,\texttt{fit <- glm(D \textasciitilde{} W, df, subset = (Z == 1),}\colorbox{#ffe58a}{$\texttt{ weights = weights,}$}

family = binomial(link="probit"))\texttt{family = binomial(link="probit"))}

Line 91:

return(-sum(weights * log(p.D^D * (1 - p.D)^(1 - D))))\texttt{return(-sum(}\colorbox{#ffe58a}{$\texttt{weights * }$}\texttt{log(p.D\textasciicircum D * (1 - p.D)\textasciicircum (1 - D))))}

Line 89 keeps p.D in [10−5,1−10−5][10^{-5},1-10^{-5}], so that every logarithm is finite and a weight of zero removes the observation from the sum.

Crop of the last six lines of Section 3.3. Red boxes mark the sentence 'To characterize uncertainty, we recommend bootstrapping the entire process---from computing the compliance score to weighted IV estimation (Abadie 2002).', which ends with the marker of footnote 7.
Section 3.3, p. 498 (PDF p. 7), last paragraph. The marked sentence recommends bootstrapping the whole procedure, the compliance score included. The recommendation is correct; the error is in the package that implements it.
Listing of eighteen lines of R code. A red box marks the call 'fit <- glm(D ~ W, df, subset = (Z == 1), family = binomial(link="probit"))' on lines 43 to 47, which has no weights argument. A second red box marks '* weights' inside the logarithm in line 91, 'return(-sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights)))'.
Package icsw 1.0.0, R/compliance.score.R, lines 43–48 and 81–92, typeset from the source file with line numbers added. The marked call fits the one-sided probit without weights; the marked factor multiplies the likelihood contribution by the weights inside the logarithm.
Two panels, for the strata W = 0 and W = 1, with the bootstrap replicate on the horizontal axis (original fit, then 1 to 4) and the fitted compliance score from 0 to 1 on the vertical axis. Blue circles, the published code, lie at 0.25 in every replicate in the left panel and at 0.75 in the right panel. Orange diamonds, a probit refitted with the replicate weights, have the same score as the circles at the original fit and then move: about 0.33, 0.12, 0.22 and 0.19 on the left, and about 0.99, 0.88, 0.67 and 0.63 on the right.
Fitted compliance scores in the original fit and in four bootstrap replicates of the one-sided code of icsw 1.0.0, for the 16-observation example. The published code returns 0.250.25 and 0.750.75 every time; a probit that uses the replicate weights returns scores that change with the weights. Four replicates show the path through the code and do not estimate a sampling distribution.

The data do not support the statement that compliers in the canvassing study are more likely to be members of a major party and to identify as white: the race and party indicators are zero wherever the label is not recorded, race is recorded only in Raleigh, and within Raleigh the voters recorded as White have lower estimated compliance scores than the others.

Section 4.1, last paragraph of p. 501 (PDF p. 10), sentence beginning “We see that compliers”; supplement Table A7, p. A11 (supplement PDF p. 11)

See also under Formal statements (Lean): P029-T9, P029-T10

As printed

We see that compliers are more likely to have voted previously, be members of a major party, and identify as white (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates).

Supplement Table A7, rows “Comp. Score” and “Raleigh”, nine of the twelve columns.
C. ScoreWhiteBlackVoted 00Voted 99Voted Pri.DemRepInd
Comp. Score1.000.610.320.400.430.250.380.31−0.47-0.47
Raleigh0.750.890.330.130.17−0.00-0.000.460.43−0.18-0.18
The problem

The sentence reads a description of the compliers off the correlations in Table A7 between the estimated compliance score and the covariates: 0.61 for White, 0.38 for Dem and 0.31 for Rep. These numbers are right. They do not measure race or party membership in the six cities. The replication script sets each indicator to 1 when the label of the voter matches and to 0 otherwise, and a blank label counts as 0. Race is blank for all 14,273 voters outside Raleigh. The White indicator is then 1 only for the 3,884 Raleigh voters with the label W, and Raleigh has the highest mean estimated score of the six cities, 0.44 against 0.15 to 0.32. The pooled correlation mostly separates Raleigh from the other cities. Table A7 itself shows that the Raleigh indicator has correlation 0.89 with White and 0.75 with the score, and that the Black indicator is also positively correlated with the score (0.32).

Within Raleigh, the one city where race is recorded, the correlation between the score and the White indicator is −0.41-0.41. The mean estimated score is 0.432 for the voters recorded as White, 0.487 for those recorded as Black and 0.529 for the 130 voters with another or an unknown label. Among the Raleigh voters assigned to canvassing, 602 of the 1,370 recorded as White were contacted (43.9%), against 160 of the 315 others (50.8%).

Party is blank for every voter in Detroit, Minneapolis and St. Paul and for 489 voters in Columbus. More than 90% of the pooled covariance between the score and each party indicator is covariance between city means. For party the sign does not reverse within Raleigh, where the correlation of the score with the combined Dem/Rep indicator (which includes the code LIB in Dem) is 0.18, against 0.53 in the pooled data.

I conclude that the correlations cited do not support the description of the compliers by race and party. I do not conclude that the description is false. The association between race or party and compliance over the six cities depends on the labels that were not recorded, and these data cannot determine it. The part of the sentence about previous voting is supported.

Evidence an exact counterexample

The evidence is a recalculation from the replication data and an exact counterexample to the inference from recorded labels. The recalculation fits the compliance model of the script. It reproduces the correlations of the compliance score in Table A7 and the twelve point estimates in each column of Table 1 at the printed precision, with the age-missing indicator set before the imputation (see P029-E021). The first table lists the blank labels and the mean score in each city.

Voters, blank race and party labels, and mean estimated compliance score, by city.
CityVotersRace blankParty blankMean score
Bridgeport1,8061,80600.279
Columbus2,4782,4784890.147
Detroit4,9544,9544,9540.309
Minneapolis2,8272,8272,8270.185
Raleigh4,660000.442
St. Paul2,2082,2082,2080.320

In Raleigh, 21 of the 4,660 race labels are unknown or unspecified. The covariance between the score ss and an indicator AA splits exactly into a part within cities and a part between city means,

Cov⁡(s,A)=E[(s−sG)(A−AG)]+E[(sG−sˉ)(AG−Aˉ)],\operatorname{Cov}(s, A) = \mathbb{E}\bigl[(s - s_G)(A - A_G)\bigr] + \mathbb{E}\bigl[(s_G - \bar s)(A_G - \bar A)\bigr] ,

where sGs_G and AGA_G are the means in the voter’s city and sˉ\bar s and Aˉ\bar A are the overall means.

The pooled covariance of the estimated compliance score with each indicator, and its two parts.
IndicatorPooledBetween citiesWithin cities
White0.02690.02690.02900.0290−0.0021-0.0021
Dem0.01580.01580.01450.01450.00130.0013
Rep0.00990.00990.00930.00930.00070.0007
Ind−0.0142-0.0142−0.0133-0.0133−0.0009-0.0009

For White the part within cities is negative, and all of it comes from Raleigh.

For the counterexample, I complete the unobserved quantities in two ways that agree with every recorded label and every observed variable. In both, a voter assigned to canvassing is a complier if contacted, and a control voter, whose status is not observed, is not a complier. I assign the 21 unknown or unspecified Raleigh race labels as not White in both completions. There are 2,551 compliers. In the first completion every voter with a blank race label is not White. Then 3,884 voters are White, 602 of them are compliers, and the compliance rate of the White voters exceeds that of the others by 744791/29225158≈0.025744791/29225158 \approx 0.025. In the second completion every voter with a blank race label is White. Then 18,157 voters are White, 2,391 of them are compliers, and the difference is −131213/1761229≈−0.075-131213/1761229 \approx -0.075. With the estimated scores held fixed, the correlation between the score and race is 0.610.61 in the first completion and −0.37-0.37 in the second. For party, assigning every blank label as outside the two major parties gives a positive association; assigning blank labels as major-party only for noncompliers gives a negative association, with recorded LIB affiliations outside the two major parties in both cases. Both completions agree with the data. This implies that the data do not determine the sign, and they do not show which completion is closer to the truth.

What survives

The correlations of Table A7 are numerically right. The statement that compliers are more likely to have voted previously stands: the score is positively correlated with turnout in 2000 and in 1999 in the pooled data and within cities. The sentence is not an input to any calculation. A change of wording leaves the point estimates of Table 1, the comparison of 5.6 with 6.7 percentage points and the bootstrap as they are. I did not recompute the intervals. The method, the simulations and the application to Albertson and Lawrence (2009) do not depend on the sentence. What changes is the description of the compliers, which must be limited to labels that were recorded.

Corrected

Section 4.1, p. 501. Estimated compliance scores are positively associated with prior voting. The pooled correlations with recorded White, Dem and Rep indicators also reflect differences between cities in contact rates and covariate availability. Within Raleigh, the only city supplying race labels, voters recorded as White have lower mean estimated compliance scores than the remaining records (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates).

Note to add to supplement Table A7. Blank race and party labels are coded zero in the respective indicators; race is unavailable outside Raleigh and party is unavailable throughout Detroit, Minneapolis, and St. Paul, with additional blank party records in Columbus. Age, primary-vote history, and 1999-vote history are unavailable in particular cities and are imputed. The correlations in the table do not change.

Crop of seven lines of text. The text reads: We begin by estimating compliance scores for the sample. As recommended above, we use all observed pretreatment covariates, which include the covariates listed in Table 1 and dummy variables for the six cities in the sample. Although the set of available covariates may or may not satisfy the ignorability assumption, we will still recover an estimate for a population that has a covariate profile that matches that of the full population. We see that compliers are more likely to have voted previously, be members of a major party, and identify as white (see the supplementary appendix for a correlation matrix between the estimated compliance score and covariates). We now apply ICSW; the crop ends in the middle of this sentence. A red outline surrounds the words be members of a major party, and identify as white.
Section 4.1, last paragraph of p. 501 (PDF p. 10). The outline marks the description of the compliers by party and race.
Crop of the top of a correlation table with the column headings C. Score, Fam. Size, White, Black, Voted 00, Voted 99, Voted Pri., Age, Age M., Dem, Rep and Ind. Row Comp. Score: 1.00, 0.37, 0.61, 0.32, 0.40, 0.43, 0.25, 0.23, -0.55, 0.38, 0.31, -0.47. Row Columbus: -0.55, -0.17, -0.20, -0.07, -0.15, -0.05, -0.19, -0.00, 1.00, -0.11, -0.03, 0.79. Row Detroit: 0.05, 0.03, -0.30, -0.11, -0.00, -0.24, 0.23, 0.16, -0.23, -0.28, -0.19, -0.18. Row Minneapolis: -0.45, -0.24, -0.21, -0.08, -0.06, -0.14, -0.01, -0.14, -0.16, -0.20, -0.14, -0.13. Row Raleigh: 0.75, 0.26, 0.89, 0.33, 0.13, 0.17, -0.00, 0.02, -0.22, 0.46, 0.43, -0.18. Row St. Paul: 0.07, 0.15, -0.18, -0.07, 0.16, 0.30, 0.06, -0.08, -0.14, -0.17, -0.12, -0.11. Red outlines surround three entries of the row Comp. Score: 0.61 under White, 0.38 under Dem and 0.31 under Rep.
Supplement Table A7, p. A11 (supplement PDF p. 11): the column headings and the first six rows. The table is printed sideways and is turned upright here. The outlines mark the correlations of the compliance score with White, Dem and Rep. In the row “Raleigh”, the entry under “C. Score” is 0.75 and the entry under “White” is 0.89.
Figure with two panels of horizontal bars, titled: The White indicator is 1 only in Raleigh, the city with the highest scores; within Raleigh, voters recorded as White have the lowest mean score. Upper panel, mean estimated compliance score. Five grey bars are cities where race is not recorded and the White indicator is 0 for every voter: Columbus (n = 2,478) 0.147, Minneapolis (n = 2,827) 0.185, Bridgeport (n = 1,806) 0.279, Detroit (n = 4,954) 0.309, St. Paul (n = 2,208) 0.320. One orange bar is Raleigh voters recorded as White, indicator 1 (n = 3,884): 0.432. Two blue bars are Raleigh voters not recorded as White, indicator 0: recorded Black (n = 646) 0.487, and other or unknown (n = 130) 0.529. Lower panel, correlation of the estimated compliance score with the White indicator: a solid dark bar for all six cities, the published value of Table A7, at plus 0.61, and a hatched bar for the correlation within Raleigh from this check, at minus 0.41.
Mean estimated compliance score by city and, within Raleigh, by recorded race (upper panel), and the correlation between the score and the White indicator over the six cities, as in Table A7, and within Raleigh (lower panel). The values are from the recalculation with the replication data.

Minor (19)

Real but inconsequential, for example a misprint that does not change a result.

The five potential-outcome assumptions are not weaker than, or subsumed by, the SEM assumptions of Section 2.1: neither set of assumptions implies the other.

Section 2.2, p. 495 (PDF p. 4), last sentence of the paragraph that lists the five IV assumptions

See also under Formal statements (Lean): P029-T11, P029-T12

As printed

Fourth, we assume that the population contains no defiers, i.e., Pr⁡(D0>D1)=0\Pr(D_0 > D_1) = 0. Fifth, we assume that ZiZ_i is randomly assigned and thus independent of all potential outcomes. As no parametric assumptions are necessary, these assumptions are considerably weaker than (and subsumed by) those of the SEM framework.

The problem

The sentence asserts that the SEM conditions of Section 2.1 imply the five assumptions just listed. For the model in equations (1) and (2), with a common effect β1\beta_1, Section 2.1 requires that the correlation between ZiZ_i and uiu_i tend to zero and that the covariance between ZiZ_i and DiD_i tend to a nonzero limit. These conditions do not exclude defiers, and a zero correlation does not make ZiZ_i independent of the potential outcomes. In the other direction, the five assumptions let the effect Y1−Y0Y_1-Y_0 vary across units, and the SEM formulation does not. Neither set of assumptions implies the other. The point that the passage serves is correct: the potential-outcome formulation permits heterogeneous treatment effects.

Evidence an exact counterexample

The evidence is three exact counterexamples. Each is a finite population with binary ZZ and DD, no covariates and independently sampled units, in which D=DZD=D_Z and Y=YDY=Y_D.

  • The SEM conditions hold and the population contains defiers. A unit is a complier with probability 3/43/4 and a defier with probability 1/41/4, and Z∼Bernoulli(1/4)Z\sim\mathrm{Bernoulli}(1/4) is independent of the type. I set u=3/4u=3/4 for a defier and u=−1/4u=-1/4 for a complier, Y0=uY_0=u and Y1=u+1Y_1=u+1. Then Y=D+uY=D+u with β1=1\beta_1=1, and D=1/4+Z/2+eD=1/4+Z/2+e with E(e∣Z)=0\mathrm{E}(e\mid Z)=0. I have Var⁡(Z)=Var⁡(u)=3/16\operatorname{Var}(Z)=\operatorname{Var}(u)=3/16, Cov⁡(Z,u)=0\operatorname{Cov}(Z,u)=0 and Cov⁡(Z,D)=3/32\operatorname{Cov}(Z,D)=3/32. The instrument is independent of all potential outcomes and the exclusion restriction holds, but Pr⁡(D0>D1)=1/4\Pr(D_0>D_1)=1/4.
  • The SEM conditions hold and ZZ is not independent of the potential outcomes. Let UU be uniform on {−2,−1,1,2}\{-2,-1,1,2\}, and set Z=1{∣U∣=2}Z=\mathbf{1}\{|U|=2\}, D0=1{U>0}D_0=\mathbf{1}\{U>0\}, D1=1D_1=1, Y0=UY_0=U and Y1=U+1Y_1=U+1. Then Y=D+UY=D+U, and D=1/2+Z/2+eD=1/2+Z/2+e with E(e∣Z)=0\mathrm{E}(e\mid Z)=0. I have E(U∣Z)=0\mathrm{E}(U\mid Z)=0, Cov⁡(Z,U)=0\operatorname{Cov}(Z,U)=0 and Cov⁡(Z,D)=1/8\operatorname{Cov}(Z,D)=1/8, and there are no defiers. But Pr⁡(∣Y0∣=2∣Z=1)=1\Pr(|Y_0|=2\mid Z=1)=1 and Pr⁡(∣Y0∣=2∣Z=0)=0\Pr(|Y_0|=2\mid Z=0)=0.
  • The five assumptions hold and the effect is not constant. Let HH be uniform on {1,2}\{1,2\}, and let Z∼Bernoulli(1/2)Z\sim\mathrm{Bernoulli}(1/2) be independent of HH. Every unit is a complier, Y0=0Y_0=0 and Y1=HY_1=H. The effect is 11 for half of the units and 22 for the other half, while a causal reading of equation (1) requires that Y1−Y0=β1Y_1-Y_0=\beta_1 for every unit.

Note that in the first two populations the treatment is endogenous (Cov⁡(D,u)\operatorname{Cov}(D,u) is 3/323/32 and 1/41/4), and the population IV ratio equals the common effect 11. These populations are not counterexamples to IV estimation under a constant effect.

What survives

Equation (3) is stated under the five assumptions themselves, and Section 3.4 invokes the IV assumptions directly for ICSW. Neither uses the comparison, and no simulation or application result depends on it. Only the comparison changes. The claim that one set of assumptions contains the other is withdrawn.

Corrected

Fourth, we assume that the population contains no defiers, i.e., Pr⁡(D0>D1)=0\Pr(D_0 > D_1) = 0. Fifth, we assume that ZiZ_i is randomly assigned and thus independent of all potential outcomes. These assumptions permit heterogeneous treatment effects but impose random assignment and absence of defiers, neither of which follows from the SEM conditions described above. The two sets of causal assumptions are therefore distinct and non-nested.

Crop of eight lines of text. A red box marks the words 'these assumptions are considerably weaker than (and subsumed by) those of the SEM framework.' The lines above state the fourth assumption, that there are no defiers, and the fifth, that Z is randomly assigned.
Section 2.2, p. 495 (PDF p. 4), end of the paragraph that lists the five IV assumptions. The marked sentence compares them with the assumptions of the SEM framework.

The maximum-likelihood compliance score of a saturated model is consistent but not unbiased: within a covariate stratum it is the difference between the treatment-receipt rates of the two instrument arms, truncated at zero, and under two-sided noncompliance its expectation exceeds the compliance score in every finite sample with both arms nonempty.

Assumption 2, last sentence, p. 500 (PDF p. 9); the estimator is defined in Section 3.1, equations (5)–(7), pp. 496–497 (PDF pp. 5–6)

See also under Formal statements (Lean): P029-T13, P029-T14

As printed

In the case of a saturated model and F(.)F(.) as the normal CDF, P^Ci\hat{P}_{Ci} will be unbiased for the conditional probability of compliance.

The problem

The estimator P^Ci\hat P_{Ci} is the maximum-likelihood estimator of Section 3.1. By equations (5)–(7), the model sets the probability of receiving treatment to PA,C,iP_{A,C,i} for a unit with Zi=1Z_i = 1 and to PA,C,iPA∣A,C,iP_{A,C,i} P_{A|A,C,i} for a unit with Zi=0Z_i = 0. The compliance score is the difference of the two. Since the second probability is at most the first, the model cannot fit a covariate stratum in which the arm Z=0Z = 0 receives treatment at a higher rate than the arm Z=1Z = 1. In a saturated model with both arms nonempty in stratum xx, let Dˉ1x\bar D_{1x} and Dˉ0x\bar D_{0x} be the treatment-receipt rates in stratum xx, that is, the shares of units that receive treatment in the arm Z=1Z = 1 and in the arm Z=0Z = 0. If Dˉ0x≤Dˉ1x\bar D_{0x} \le \bar D_{1x}, then the fitted probabilities are these two rates, and the fitted score is Dˉ1x−Dˉ0x\bar D_{1x} - \bar D_{0x}. If Dˉ0x>Dˉ1x\bar D_{0x} > \bar D_{1x}, then both fitted probabilities equal the pooled rate, and the fitted score is zero. Hence

P^Ci=max⁡(Dˉ1x−Dˉ0x, 0)for every unit i in stratum x.\hat P_{Ci} = \max\bigl(\bar D_{1x} - \bar D_{0x},\, 0\bigr) \qquad \text{for every unit } i \text{ in stratum } x .

The difference Dˉ1x−Dˉ0x\bar D_{1x} - \bar D_{0x} is unbiased for the compliance score, and the truncation replaces its negative values by zero. The expectation of P^Ci\hat P_{Ci} is therefore larger than the compliance score whenever a negative difference has positive probability. This is the case in every finite sample when the stratum contains always-takers and never-takers. The printed sentence is true under one-sided noncompliance, where the arm Z=0Z = 0 never receives treatment and the truncation cannot bind. Note that the normal CDF plays no role: the formula holds for any FF that maps onto (0,1)(0,1).

Evidence an exact counterexample

The evidence is a derivation of the estimator and an exact counterexample. For the derivation, I fix a stratum with n1>0n_1 > 0 units in the arm Z=1Z = 1 and n0>0n_0 > 0 units in the arm Z=0Z = 0, of which k1k_1 and k0k_0 receive treatment. I write p=PA,Cp = P_{A,C} and q=PA,CPA∣A,Cq = P_{A,C} P_{A|A,C} for the two receipt probabilities of the model. Up to a constant, the likelihood (7) of the stratum is

L(p,q)=pk1(1−p)n1−k1 qk0(1−q)n0−k0,0≤q≤p≤1,L(p, q) = p^{k_1} (1-p)^{n_1 - k_1} \, q^{k_0} (1-q)^{n_0 - k_0}, \qquad 0 \le q \le p \le 1 ,

and the fitted score is p−qp - q at the maximum. If k0/n0≤k1/n1k_0/n_0 \le k_1/n_1, then the unconstrained maximizer (k1/n1,k0/n0)(k_1/n_1, k_0/n_0) satisfies the constraint. If k0/n0>k1/n1k_0/n_0 > k_1/n_1, then k0>0k_0 > 0 and k1<n1k_1 < n_1, so LL vanishes on the edges q=0q = 0 and p=1p = 1. Further, LL has no stationary point such that 0<q<p<10 < q < p < 1, since such a point would be (k1/n1,k0/n0)(k_1/n_1, k_0/n_0). The maximum then lies on the diagonal q=p=tq = p = t, where L(t,t)=tk1+k0(1−t)n1+n0−k1−k0L(t,t) = t^{k_1 + k_0}(1-t)^{n_1 + n_0 - k_1 - k_0} is largest at t=(k1+k0)/(n1+n0)t = (k_1 + k_0)/(n_1 + n_0). Combining the two cases, I obtain the formula above. A probit model with finite coefficients has 0<q<p<10 < q < p < 1, and a sample whose maximizer lies on the boundary has a supremum that is not attained. The formula takes the maximum over the closure.

For the counterexample, I take a single stratum in which a unit is an always-taker with probability 1/41/4, a complier with probability 1/21/2 and a never-taker with probability 1/41/4, with no defiers. One of two independently sampled units is assigned to each arm at random. The receipt probabilities are p1=3/4p_1 = 3/4 in the arm Z=1Z = 1 and p0=1/4p_0 = 1/4 in the arm Z=0Z = 0, and the compliance score is 1/21/2. The model is correctly specified, with PA,C=3/4P_{A,C} = 3/4 and PA∣A,C=1/3P_{A|A,C} = 1/3.

The four possible samples with one unit in each arm. Here d1d_1 and d0d_0 are the receipts in the arms Z=1Z = 1 and Z=0Z = 0.
Receipts (d1,d0)(d_1, d_0)ProbabilityFitted (p,q)(p, q)Fitted score
(0,0)(0, 0)3/163/16(0,0)(0, 0)00
(0,1)(0, 1)1/161/16(1/2,1/2)(1/2, 1/2)00
(1,0)(1, 0)9/169/16(1,0)(1, 0)11
(1,1)(1, 1)3/163/16(1,1)(1, 1)00

The expectation of the fitted score is 9/169/16, and the bias is 1/161/16. The untruncated difference d1−d0d_1 - d_0 has expectation 3/4−1/4=1/23/4 - 1/4 = 1/2. The estimator replaces its value −1-1 in the second row by 00.

The bias is not special to one unit per arm. Let T=Dˉ1x−Dˉ0xT = \bar D_{1x} - \bar D_{0x}, and let cc be the compliance score. Since max⁡(T,0)=T+max⁡(−T,0)\max(T, 0) = T + \max(-T, 0) and E[T]=c\mathbb{E}[T] = c, I have

E[P^Ci]−c=E[max⁡(−T,0)]≥0,\mathbb{E}\bigl[\hat P_{Ci}\bigr] - c = \mathbb{E}\bigl[\max(-T, 0)\bigr] \ge 0 ,

and the inequality is strict for all arm sizes whenever p0>0p_0 > 0 and p1<1p_1 < 1. With five units in each arm, p1=1/2p_1 = 1/2 and p0=3/10p_0 = 3/10, I enumerate the 36 pairs of counts and find the expectation 15659/64000≈0.24515659/64000 \approx 0.245, against a compliance score of 1/51/5.

Consistency holds. For c≥0c \ge 0 and every real tt, I have ∣max⁡(t,0)−c∣≤∣t−c∣|\max(t, 0) - c| \le |t - c|. Chebyshev’s inequality then implies that, for every δ>0\delta > 0,

Pr⁡(∣P^Ci−c∣>δ)≤1δ2(p1(1−p1)n1+p0(1−p0)n0),\Pr\bigl(|\hat P_{Ci} - c| > \delta\bigr) \le \frac{1}{\delta^2}\left(\frac{p_1(1-p_1)}{n_1} + \frac{p_0(1-p_0)}{n_0}\right) ,

and the right side tends to zero as n1n_1 and n0n_0 tend to infinity. Under one-sided noncompliance, p0=0p_0 = 0, so k0=0k_0 = 0 and P^Ci=Dˉ1x\hat P_{Ci} = \bar D_{1x}, which is unbiased when the arm Z=1Z = 1 is not empty.

What survives

Assumption 2 requires that P^Ci\hat P_{Ci} be consistent, not that it be unbiased, and the consistency argument of the Appendix uses the assumption in that form. The estimands, the ICSW estimator and its identification argument do not depend on the sentence. The two applications and the simulations fit parametric models and not saturated ones, and none of their calculations uses the sentence. I did not recompute them for this finding. The phrase “strata means under saturation” on p. 496 needs the same qualification, since the two fitted probabilities are the separate arm means only when Dˉ0x≤Dˉ1x\bar D_{0x} \le \bar D_{1x}. What changes is the claim that the score is unbiased, which becomes a claim that it is consistent.

Corrected

In the case of a saturated model, with a fixed finite set of covariate strata and the number of observations in both instrument arms of every stratum tending to infinity, P^Ci\hat{P}_{Ci}, allowing boundary fitted probabilities, will be consistent for the conditional probability of compliance. For a unit in stratum xx with both arms nonempty, it equals max⁡(Dˉ1x−Dˉ0x,0)\max(\bar D_{1x} - \bar D_{0x}, 0), the difference between the treatment-receipt rates of the two instrument arms in that stratum truncated below at zero, and it need not be unbiased in finite samples. Under one-sided noncompliance it equals the receipt rate Dˉ1x\bar D_{1x} of the arm Z=1Z = 1 and is unbiased when that arm is nonempty. These statements hold for any FF that maps onto (0,1)(0,1).

Crop of Assumption 2 with its five lines of text. The text reads: Consistency of P-hat C i for P C i. Assumption 2 amounts to requiring proper specification of the ML estimator. Application of a flexible parametrization of the covariates in X, as is possible with large samples, will necessarily yield consistency, so long as the rate of growth in the number of estimated parameters is sufficiently slow in n. In the case of a saturated model and F(.) as the normal CDF, P-hat C i will be unbiased for the conditional probability of compliance. A red outline surrounds the word unbiased in the last line.
Assumption 2, p. 500 (PDF p. 9). The outline marks “unbiased” in the last sentence.

The Winsorization cut-off cc need not tend to zero; the fraction of scores changed tends to zero, and under suitable regularity conditions Winsorization leaves the probability limit unchanged.

Footnote 6, p. 498 (PDF p. 7), sentence beginning “Asymptotically, assuming that α\alpha remains fixed”

See also under Formal statements (Lean): P029-T15, P029-T16

As printed

Define cc as the 1/nα1/n^{\alpha}th quantile of P^Ci\hat{P}_{Ci}, where α\alpha is a positive calibrating constant. We replace all P^Ci<c\hat{P}_{Ci}<c with cc. […] Asymptotically, assuming that α\alpha remains fixed, the Winsorization has no impact on estimation, because c→0c\to0 as n→∞n\to\infty.

The problem

The cut-off cc is a quantile of the estimated compliance scores, taken at the level n−αn^{-\alpha}. As nn grows the level tends to zero, but the quantile need not, since Assumption 3 keeps the true scores above some ϵ>0\epsilon>0. The sentence’s conclusion holds for a different reason. The fraction of scores that Winsorization changes is at most n−α+1/nn^{-\alpha}+1/n, and the weights that it changes stay bounded under conditions of the kind that the article already invokes.

Evidence an exact counterexample

The evidence is an exact counterexample to the limit and a derivation of the corrected statement.

Counterexample. Let the covariate be constant. A unit is an always-taker with probability 1/41/4, a complier with probability 1/21/2 and a never-taker with probability 1/41/4, the assignment Z∼Bernoulli(1/2)Z\sim\mathrm{Bernoulli}(1/2) is independent of the type, and Y0=0Y_0=0, Y1=1Y_1=1. The IV assumptions hold, Assumption 1 holds because the effect is 11 for every unit, and Assumption 3 holds with ϵ=1/4\epsilon=1/4 because the compliance score is 1/21/2. With the exact scores, every quantile of the scores is 1/21/2, and c=1/2c=1/2 for every nn and every α\alpha. With the ML estimator of equations (5)–(7), the fitted score is the same for all units and eventually equals the difference between the proportions treated in the two arms, once those proportions are interior and correctly ordered. By the strong law, cc converges almost surely to 3/4−1/4=1/23/4-1/4=1/2. No score is below cc, the weights are unchanged, and the ICSW ratio is 11, the ATE, whenever its empirical first stage is nonzero. Note that the example contradicts the reason given and not the conclusion.

Derivation. Let pn=n−αp_n=n^{-\alpha}, let cnc_n be the cut-off, and let mnm_n be the number of scores strictly below cnc_n. For the quantile rule that the code uses, I have mn/n≤pn+1/nm_n/n\le p_n+1/n. Suppose that max⁡i≤n∣P^Ci−PCi∣→0\max_{i\le n}|\hat{P}_{Ci}-P_{Ci}|\to0 in probability and that E∣Y∣<∞\mathrm{E}|Y|<\infty. By Assumption 3 and this convergence, every estimated score exceeds ϵ/2\epsilon/2 with probability tending to one. On that event the weights wi=1/P^Ciw_i=1/\hat{P}_{Ci} and the Winsorized weights wi∗=1/max⁡(P^Ci,cn)w_i^*=1/\max(\hat{P}_{Ci},c_n) satisfy 0≤wi−wi∗≤2/ϵ0\le w_i-w_i^*\le2/\epsilon, and they differ for at most mnm_n units. For any variables with ∣Vi∣≤K|V_i|\le K, I obtain

∣1n∑i=1n(wi−wi∗)Vi∣≤2Kϵ(pn+1n)→0.\Bigl|\frac1n\sum_{i=1}^n(w_i-w_i^*)V_i\Bigr|\le\frac{2K}{\epsilon}\Bigl(p_n+\frac1n\Bigr)\to0 .

This bound covers the weighted sums of 1{Zi=z}\mathbf{1}\{Z_i=z\} and 1{Zi=z}Di\mathbf{1}\{Z_i=z\}D_i. For the weighted sums of the outcome, truncating YiY_i at a level MM and using E∣Y∣<∞\mathrm{E}|Y|<\infty, I find that the change tends to zero in probability. Since the weighted difference in treatment rates has the positive limit 1/E{1/PC(X)}1/\mathrm{E}\{1/P_C(\mathbf{X})\}, I conclude that the Winsorized and original estimators have the same probability limit. The bound is of order n−αn^{-\alpha}, which for α=0.275\alpha=0.275 is not o(n−1/2)o(n^{-1/2}). The argument does not show that the two estimators have the same first-order distribution, and it does not show that they differ.

What survives

The Winsorization rule, the code, which computes a quantile, and every estimate are unchanged. The appendix does not use the limit c→0c\to0, and consistency holds under the conditions stated below. No application estimate and no simulation result is shown to be wrong. Only the reason given in the footnote changes.

Corrected

In place of the sentence beginning “Asymptotically”: For fixed α>0\alpha>0, the fraction of observations whose scores are changed tends to zero. Under Assumption 3 and regularity conditions ensuring bounded estimated inverse scores with probability tending to one and integrable outcomes (for example, uniform score consistency and E∣Y∣<∞\mathrm{E}|Y|<\infty), Winsorization does not change the probability limit of the displayed ICSW estimator. The cutoff cc need not tend to zero. This consistency argument does not establish first-order distributional equivalence.

The rest of the footnote is unchanged.

Crop of footnote 6, seven lines. A red box marks the words 'because c tends to 0 as n tends to infinity' in the sentence 'Asymptotically, assuming that alpha remains fixed, the Winsorization has no impact on estimation'. Earlier in the footnote c is defined as the 1 over n to the alpha quantile of the estimated compliance scores.
Footnote 6, p. 498 (PDF p. 7). The marked clause gives the reason for the asymptotic irrelevance of Winsorization.

In the one-sided likelihood of footnote 3 the two Bernoulli factors are multiplied, not added.

Footnote 3, p. 497 (PDF p. 6), displayed likelihood

See also under Formal statements (Lean): P029-T17

As printed
L(θC∣D,Z)=Πi=1n[(F(θA,Cxi)Zi)Di  +  (1−F(θA,Cxi)Zi)1−Di].L(\theta_{\mathbf{C}}|D,Z) = \Pi_{i=1}^{n}\bigl[(F(\colorbox{#ffe58a}{$\displaystyle \theta_{\mathbf{A,C}}$}\mathbf{x_i})Z_i)^{D_i} \;\colorbox{#ffe58a}{$\displaystyle +$}\; (1 - F(\colorbox{#ffe58a}{$\displaystyle \theta_{\mathbf{A,C}}$}\mathbf{x_i})Z_i)^{1-D_i}\bigr].

This likelihood function is a familiar one: it is the standard likelihood function for binomial regression, if only applied to units in the treatment condition.

The problem

With a plus sign every factor of the product is at least 11, and the display is not a likelihood. The reduction of equation (7) and the likelihood for binomial regression, which the next sentence names, both have a product. The parameter is also written θC\theta_{\mathbf{C}} on the left and θA,C\theta_{\mathbf{A,C}} on the right. With no always-takers these index the same probability, and one subscript is meant.

Evidence a derivation

The evidence is a derivation. Setting PA∣A,C,i=0P_{A|A,C,i}=0 in equation (7), I obtain the likelihood for unit ii,

(PA,C,iZi)Di(1−PA,C,iZi)1−Di.(P_{A,C,i}Z_i)^{D_i}(1-P_{A,C,i}Z_i)^{1-D_i}.

For an assigned unit with F(⋅)=3/4F(\cdot)=3/4, the printed factor is 5/45/4 when Di=0D_i=0 and 7/47/4 when Di=1D_i=1, in place of 1/41/4 and 3/43/4. The one-sided likelihood in the replication code is the product.

What survives

Equation (7), the definition of the ML estimate, the sentence on binomial regression and every computed result are unaffected.

Corrected
L(θC∣D,Z)=Πi=1n[(F(θCxi)Zi)Di  ×  (1−F(θCxi)Zi)1−Di].L(\theta_{\mathbf{C}}|D,Z) = \Pi_{i=1}^{n}\bigl[(F(\colorbox{#ffe58a}{$\displaystyle \theta_{\mathbf{C}}$}\mathbf{x_i})Z_i)^{D_i} \;\colorbox{#ffe58a}{$\displaystyle \times$}\; (1 - F(\colorbox{#ffe58a}{$\displaystyle \theta_{\mathbf{C}}$}\mathbf{x_i})Z_i)^{1-D_i}\bigr].
Crop of footnote 3. A red box marks the plus sign between '(F(theta A,C x sub i) Z sub i) to the power D sub i' and '(1 minus F(theta A,C x sub i) Z sub i) to the power 1 minus D sub i' in the displayed likelihood. The next sentence calls it the standard likelihood function for binomial regression.
Footnote 3, p. 497 (PDF p. 6). The marked plus sign joins the two Bernoulli factors of the one-sided likelihood.

Six misprints of notation: the compliance score and the receipt probability each lack a conditioning variable, equation (9) has a second conditioning bar, one weight has two hats, and the simulation recipe prints DiD_i for CiC_i and omits the test >0>0 in its indicators.

Section 3.1, pp. 496–497 (PDF pp. 5–6); Appendix, p. 504 (PDF p. 13), equation (9) and the display after equation (14); Supplement, pp. A1–A2 (PDF pp. 1–2)

See also under Formal statements (Lean): P029-T18, P029-T19, P029-T26

As printed
  1. p. 496: “Therefore, we may define the compliance score as PC,i=Pr⁡(D1i>D0i)P_{C,i} = \colorbox{#ffe58a}{$\Pr(D_{1i} > D_{0i})$}.”
  2. p. 497: Pr⁡(D=1∣X=xi)=Pr⁡(D1>D0∣X=xi)Zi+Pr⁡(D0=1∣X=xi)=PA,C,i(1−PA∣A,C,i)Zi+PA,C,iPA∣A,C,i\colorbox{#ffe58a}{$\Pr(D=1|\mathbf{X}=\mathbf{x}_i)$} = \Pr(D_1 > D_0|\mathbf{X}=\mathbf{x}_i)Z_i + \Pr(D_0=1|\mathbf{X}=\mathbf{x}_i) = P_{A,C,i}(1-P_{A|A,C,i})Z_i + P_{A,C,i}P_{A|A,C,i}.
  3. p. 504, equation (9):
    ITTw=1wc‾∫ΩPr⁡(D1>D0∣X=x) E(Y1−Y0∣D1>D0 ∣ X=x)Pr⁡(D1>D0∣X=x)f(x)dx.\mathrm{ITT}^w = \frac{1}{\overline{w_c}}\int_\Omega \frac{\Pr(D_1 > D_0|\mathbf{X}=\mathbf{x})\,\mathrm{E}(Y_1 - Y_0|D_1 > D_0\,\colorbox{#ffe58a}{$\displaystyle |$}\,\mathbf{X}=\mathbf{x})}{\Pr(D_1 > D_0|\mathbf{X}=\mathbf{x})}f(\mathbf{x})d\mathbf{x}.
  4. p. 504, display after equation (14), first ratio of the denominator:
    (∑i=1nw^CiZiDi)/(∑i=1nw^^CiZi).\Bigl(\sum_{i=1}^{n}\hat{w}_{Ci}Z_iD_i\Bigr)\Big/\Bigl(\sum_{i=1}^{n}\colorbox{#ffe58a}{$\displaystyle \hat{\hat{w}}_{Ci}$}Z_i\Bigr).
  5. Supplement, p. A1: ACi=I[2+a+bX1i+cX2i+dX3i+N(0,3)]AC_i = \mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \mathbf{N}(0,3)\colorbox{#ffe58a}{$]$}, Ai=I[−2+e+fX1i+gX2i+hX3i+N(0,3)]ACiA_i = \mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \mathbf{N}(0,3)\colorbox{#ffe58a}{$]$}AC_i, Ci=ACi−AiC_i = AC_i - A_i, Di=DiZi+AiD_i = \colorbox{#ffe58a}{$D_i$}Z_i + A_i.
  6. Supplement, p. A2: ACi=I[2+a+(1+bX1i)(1+cX2i)+dX3i3/3+Logistic(0,3)]AC_i = \mathbf{I}[2 + a + (1 + bX1_i)(1 + cX2_i) + dX3_i^3/3 + \mathrm{Logistic}(0,3)\colorbox{#ffe58a}{$]$}, Ai=I[−2+e+(1+fX1i)(1+gX2i)+hX3i3/3+Logistic(0,3)]ACiA_i = \mathbf{I}[-2 + e + (1 + fX1_i)(1 + gX2_i) + hX3_i^3/3 + \mathrm{Logistic}(0,3)\colorbox{#ffe58a}{$]$}AC_i.
The problem

Each item is a slip of notation, and the neighbouring formulas and the simulation code give the intended meaning. The compliance score is conditional on the covariates, since p. 496 first defines PCi=Pr⁡(D1>D0∣X=xi)P_{Ci}=\Pr(D_1>D_0|\mathbf{X}=\mathbf{x}_i) and p. 497 uses that definition again. In the first display on p. 497 the right side depends on ZiZ_i, so that the left side must also condition on ZiZ_i. In equation (9) the second bar stands for a comma, as in equations (8) and (10). The weight with two hats is w^Ci\hat{w}_{Ci}, as in Section 3.3 and in the display at the top of p. 505. In the supplement each indicator is the indicator that the latent index is positive, and the treatment received is CiZi+AiC_iZ_i+A_i. As printed, DiD_i is defined in terms of itself.

Evidence a check of the printed page

The evidence is a check of the pages against the neighbouring formulas and the simulation code, with two short calculations. With no always-takers, compliance probability 1/21/2 and Z∼Bernoulli(1/2)Z\sim\mathrm{Bernoulli}(1/2), I have Pr⁡(D=1∣X)=1/4\Pr(D=1|\mathbf{X})=1/4, while the printed right side is 00 for Zi=0Z_i=0 and 1/21/2 for Zi=1Z_i=1. For an always-taker assigned to treatment, the identity printed on p. A1 reads Di=Di+1D_i=D_i+1. All six simulation scripts in the replication files test whether the latent index is positive and set D <- C*Z + A.

What survives

The definitions used elsewhere, the argument in the appendix and the simulations are unaffected. Two further points about these passages are not errors. The receipt probability lies strictly in (0,1)(0,1) under the probit specification of equations (5) and (6). The step before equation (11) is attributed to the law of total probability. It is a correct step, by the law of total expectation.

Corrected
  1. p. 496: PC,i=Pr⁡(D1i>D0i ∣ X=xi)P_{C,i} = \Pr(D_{1i} > D_{0i}\colorbox{#ffe58a}{$\,|\,\mathbf{X}=\mathbf{x}_i$}).
  2. p. 497: Pr⁡(Di=1∣X=xi,Zi)=Pr⁡(D1>D0∣X=xi)Zi+Pr⁡(D0=1∣X=xi)=PA,C,i(1−PA∣A,C,i)Zi+PA,C,iPA∣A,C,i\Pr(\colorbox{#ffe58a}{$D_i$}=1|\mathbf{X}=\mathbf{x}_i\colorbox{#ffe58a}{$,Z_i$}) = \Pr(D_1 > D_0|\mathbf{X}=\mathbf{x}_i)Z_i + \Pr(D_0=1|\mathbf{X}=\mathbf{x}_i) = P_{A,C,i}(1-P_{A|A,C,i})Z_i + P_{A,C,i}P_{A|A,C,i}.
  3. p. 504, equation (9):
    ITTw=1wc‾∫ΩPr⁡(D1>D0∣X=x) E(Y1−Y0∣D1>D0, X=x)Pr⁡(D1>D0∣X=x)f(x)dx.\mathrm{ITT}^w = \frac{1}{\overline{w_c}}\int_\Omega \frac{\Pr(D_1 > D_0|\mathbf{X}=\mathbf{x})\,\mathrm{E}(Y_1 - Y_0|D_1 > D_0\colorbox{#ffe58a}{$\displaystyle ,$}\,\mathbf{X}=\mathbf{x})}{\Pr(D_1 > D_0|\mathbf{X}=\mathbf{x})}f(\mathbf{x})d\mathbf{x}.
  4. p. 504, display after equation (14), first ratio of the denominator:
    (∑i=1nw^CiZiDi)/(∑i=1nw^CiZi).\Bigl(\sum_{i=1}^{n}\hat{w}_{Ci}Z_iD_i\Bigr)\Big/\Bigl(\sum_{i=1}^{n}\colorbox{#ffe58a}{$\displaystyle \hat{w}_{Ci}$}Z_i\Bigr).
  5. Supplement, p. A1: ACi=I[2+a+bX1i+cX2i+dX3i+N(0,3)>0]AC_i = \mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \mathbf{N}(0,3)\colorbox{#ffe58a}{$>0$}], Ai=I[−2+e+fX1i+gX2i+hX3i+N(0,3)>0]ACiA_i = \mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \mathbf{N}(0,3)\colorbox{#ffe58a}{$>0$}]AC_i, Ci=ACi−AiC_i = AC_i - A_i, Di=CiZi+AiD_i = \colorbox{#ffe58a}{$C_i$}Z_i + A_i.
  6. Supplement, p. A2: ACi=I[2+a+(1+bX1i)(1+cX2i)+dX3i3/3+Logistic(0,3)>0]AC_i = \mathbf{I}[2 + a + (1 + bX1_i)(1 + cX2_i) + dX3_i^3/3 + \mathrm{Logistic}(0,3)\colorbox{#ffe58a}{$>0$}], Ai=I[−2+e+(1+fX1i)(1+gX2i)+hX3i3/3+Logistic(0,3)>0]ACiA_i = \mathbf{I}[-2 + e + (1 + fX1_i)(1 + gX2_i) + hX3_i^3/3 + \mathrm{Logistic}(0,3)\colorbox{#ffe58a}{$>0$}]AC_i.

The supplement may also state that assignment is balanced, with n/2n/2 units in each arm, as in the code, and the step before equation (11) may be attributed to the law of total expectation. Neither omission is an error.

Crop of equation (6) and the five lines after it. A red box marks 'P sub C,i = Pr(D sub 1i greater than D sub 0i)' in the sentence 'Therefore, we may define the compliance score as'.
Section 3.1, p. 496 (PDF p. 5), after equation (6). The marked definition of the compliance score has no conditioning on the covariates.
Crop of a two-line display and the three lines of text after it. A red box marks the left side 'Pr(D = 1 given X = x sub i)'. The right side is 'Pr(D1 greater than D0 given X = x sub i) times Z sub i plus Pr(D0 = 1 given X = x sub i)'.
Section 3.1, p. 497 (PDF p. 6), first display. The marked left side conditions on the covariates only, while the right side depends on ZiZ_i.
Crop of equation (9) for the weighted ITT with one line of text above and one below. A red box marks the vertical bar between 'D1 greater than D0' and 'X = x' inside the expectation of Y1 minus Y0 in the numerator.
Appendix, p. 504 (PDF p. 13), equation (9). The marked bar is the second conditioning bar in the expectation.
Crop of one line of text, ending 'a consistent estimator of the weighted ratio will be', and of the ratio estimator written as four weighted sums over four weighted sums. A red box marks the weight 'w sub Ci' with a double hat in the second sum of the denominator; every other weight has one hat.
Appendix, p. 504 (PDF p. 13), display after equation (14). The marked weight carries two hats.
Crop of six displayed lines defining X1, X2, X3, the coefficients a to h, AC sub i, A sub i, C sub i and D sub i. Red boxes mark the closing bracket of the indicator in the line for AC sub i, the closing bracket of the indicator in the line for A sub i, and the first factor 'D sub i' on the right of 'D sub i = D sub i Z sub i + A sub i'.
Supplement, p. A1 (PDF p. 1), baseline data generating process. The marked brackets close indicators that have no test >0>0, and the marked DiD_i stands where CiC_i is meant.
Crop of the heading 'Heterogeneous Treatment Effects, Model is Incorrect', the line 'Same as the baseline DGP, except:' and the two lines defining AC sub i and A sub i with a Logistic(0,3) term. Red boxes mark the closing bracket of each indicator.
Supplement, p. A2 (PDF p. 2), design “Heterogeneous Treatment Effects, Model is Incorrect”. The marked brackets close indicators that have no test >0>0.

Three labels do not match what they label: the Democrat interval in column (2) of Table 1 ends at 4.8, printed “04.8”; Table A6 reports coverage as proportions and not as percentages; and the simulation scripts carry the comment “ATE is always 1”, while the average effect in the simulated population is 5.

Table 1, row “Democrat”, column (2), p. 501 (PDF p. 10); Section 4.2, p. 502 (PDF p. 11), with supplement Table A8, p. A12 (supplement PDF p. 12); supplement p. A9 (supplement PDF p. 9), the sentence that introduces Table A6; the simulation scripts of the replication files

As printed

Table 1, row “Democrat”, p. 501. Column (1): 2.6 (0.6,4.7)(0.6, 4.7). Column (2): 2.6 (0.7,04.8)(0.7, \colorbox{#ffe58a}{$0$}4.8).

Section 4.2, p. 502. The compliance score most strongly correlates with political interest, newspaper reading, and education (see the supplementary appendix for a full correlation matrix).

Supplement, p. A9. In Table A6, we present % coverage with the bootstrap for n∈500,2500,5000n \in 500, 2500, 5000 assuming that the model is correct and α=0.275\alpha = 0.275.

Supplement Table A6: Bootstrap Coverage.
nn90% Interval Coverage95% Interval Coverage
5000.940.98
25000.910.94
50000.920.95

Simulation scripts. # ATE is always 1

The problem

In Table 1 the upper limit of the Democrat interval in column (2) is printed with a leading zero. The supplement announces “% coverage”, and the entries of Table A6 are proportions: 0.94 stands for 94%. The five simulation scripts carry the comment that the ATE is always 1. They generate the treatment effect 5(1+X1+X2+X3)5(1 + X_1 + X_2 + X_3) in the baseline design and 5(1+X1+X2+X1X2+X3)5(1 + X_1 + X_2 + X_1 X_2 + X_3) in the design with an interaction, with independent covariates that are uniform on (−1,1)(-1, 1). The average effect in the population is then 5, and the average effect in a simulated sample varies from sample to sample. In the homogeneous design the effect is 5 for every unit.

The sentence of Section 4.2 is not an error; the optional clarification below makes explicit which score it describes. The sentence names political interest, newspaper reading and education as the strongest correlates of the compliance score, while Table A8 reports a correlation of 0.37 for party ID and of 0.36 for education. Table A8 is computed with the Winsorized score. For the score before Winsorization, the three strongest correlates are those of the sentence. The sentence does not say which score it describes.

Evidence a recalculation from the source

The evidence is a check of the pages and two recalculations. Integrating each of the two effect functions over the three covariates, I obtain 5. From the replication data, the correlations of the score before Winsorization are 0.71 with political interest, 0.70 with newspaper reading, 0.40 with education and 0.32 with party ID. After Winsorization with α=0.275\alpha = 0.275 they are 0.65, 0.62, 0.36 and 0.37, which are the entries of Table A8. The comment is at line 44 of Simulations Alpha 6-20.R and of the three Simulations Misspecification scripts named Right, Wrong Model and Only X1, and at line 45 of Simulations Misspecification Noise 6-20.R.

What survives

No estimate, interval or fit changes. The label of Table A6 and the comment do not enter a computation. The limit 4.8 is the printed value without its leading zero; I obtain 4.7515 in the full bootstrap, which rounds to 4.8.

Corrected

Table 1, row “Democrat”, column (2). 2.6 (0.7,4.8)(0.7, \colorbox{#ffe58a}{$4.8$}).

Supplement, p. A9. In Table A6, we present coverage proportions with the bootstrap for n∈500,2500,5000n \in 500, 2500, 5000 assuming that the model is correct and α=0.275\alpha = 0.275.

Simulation scripts. # Superpopulation ATE is 5; heterogeneous sample-average effects vary, while the homogeneous sample effect is 5.

Optional clarification of Section 4.2. The Winsorized compliance score correlates most strongly with political interest and newspaper reading, followed by party identification and education (see the supplementary appendix for a full correlation matrix). Alternatively, the printed sentence may be kept and labelled as referring to the score before Winsorization.

Crop of three rows of Table 1, each with a coefficient and an interval in two columns. Age missing: 12.7 with interval (-16.5, 44.1), and 17.7 with interval (-15.9, 50.3). Democrat: 2.6 with interval (0.6, 4.7), and 2.6 with interval (0.7, 04.8). Republican: 3.8 with interval (1.3, 6.2), and 3.2 with interval (0.8, 5.8). A red outline surrounds the upper limit 04.8 of the Democrat interval in the second column.
Table 1, rows “Age missing”, “Democrat” and “Republican”, p. 501 (PDF p. 10). The two columns of numbers are LATE (2SLS), (1), and ATE (ICSW), (2). The outline marks the upper limit “04.8” in column (2).
Crop of the supplement section headed (3) Bootstrap Confidence Interval Coverage. The text reads: In Table A6, we present percent-sign coverage with the bootstrap for n in 500, 2500, 5000 assuming that the model is correct and alpha = 0.275. Due to computational limitations, we use only 500 bootstrap iterations and 500 simulations for each n. Below it, Table A6, Bootstrap Coverage, has the columns n, 90 percent Interval Coverage and 95 percent Interval Coverage, with the rows 500, 0.94, 0.98; 2500, 0.91, 0.94; 5000, 0.92, 0.95. A red outline surrounds the percent sign and the word coverage in the first line.
Supplement, p. A9 (supplement PDF p. 9): the sentence that introduces Table A6, and the table. The outline marks “% coverage”; the entries of the table are proportions.

The replication code for Table A3 and the “Noise (Homogeneous)” panel of Figure A3 computes RMSEs about the sample-average effect of the companion heterogeneous design, a random quantity with mean 55 and variance 25/n25/n; the effect in the homogeneous design is exactly 55, and the RMSEs should be computed about 55.

Supplement: Table A3, p. A7 (supplement PDF p. 7); Figure A3, panel “Noise (Homogeneous)”, p. A6 (supplement PDF p. 6); definition of the design, p. A2 (supplement PDF p. 2). Replication files: Simulations Misspecification Noise 6-20.R, lines 125–135, and Sim-Display Misspec.R, lines 60–70

See also under Formal statements (Lean): P029-T5, P029-T6, P029-T22

As printed

Supplement, p. A2. Homogeneous Treatment Effects, Conditioning on Random Noise

Same as the previous DGP, except: τi=5,∀i\tau_i = 5, \forall i.

Supplement, p. A7.

Table A3: Noise (Homogeneous)
nnOLS (ZZ)OLS (DD)2SLSICSW
5002.502.640.530.58
10002.502.620.370.39
25002.492.610.230.23
50002.492.640.160.16
100002.472.610.110.11
250002.482.630.070.07

Replication files, Simulations Misspecification Noise 6-20.R, lines 22, 24, 26, 83 and 125. Line 125 is broken here after function(x). Lines 127, 129, 131, 133 and 135 repeat it for the other sample sizes, and lines 60 to 70 of Sim-Display Misspec.R repeat all six.

tau1<- 5*(1+X1+X2+X3)Y2 <- Y0 + 5*(D)return(list(D,Y1,tau1,C,Y2))output[1] <- mean(dgpout[[3]])t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)(x - simoutput[1,isN])^2))^.5,mean(simoutput[7,isN]))\begin{aligned} &\texttt{tau1<- 5*(1+X1+X2+X3)}\\ &\texttt{Y2 <- Y0 + 5*(D)}\\ &\texttt{return(list(D,Y1,tau1,C,Y2))}\\ &\texttt{output[1] <- mean(dgpout[[3]])}\\ &\texttt{t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)}\\ &\qquad\texttt{(x - }\colorbox{#ffe58a}{$\displaystyle \texttt{simoutput[1,isN]}$}\texttt{)\textasciicircum 2))\textasciicircum .5,mean(simoutput[7,isN]))} \end{aligned}
The problem

In the homogeneous design every unit has the treatment effect 55. This implies that the sample-average effect, the ATE and the LATE are all equal to 55, and that the RMSE of an estimator is its root mean squared error about 55. The simulation script generates the homogeneous outcome Y2=Y0+5DY2 = Y0 + 5D together with the outcome of the heterogeneous design, and it stores the four estimates from Y2Y2 in rows 9 to 12 of its output. It scores these rows against row 1. Row 1 holds the mean of tau1i=5(1+X1i+X2i+X3i)\texttt{tau1}_i = 5(1 + X1_i + X2_i + X3_i), the sample-average effect of the heterogeneous design. This benchmark is

B=5+5n∑i=1n(X1i+X2i+X3i),B = 5 + \frac{5}{n}\sum_{i=1}^{n}(X1_i + X2_i + X3_i),

a random quantity with mean 55 and variance 25/n25/n. Both reporting scripts therefore compute a root mean squared distance from BB for every estimator and sample size in Table A3 and the corresponding panel of Figure A3, instead of the required RMSE about 55. The corrected values cannot be computed from the printed ones, because the difference depends on how each estimate varies with BB across the simulated datasets. They require the saved output of the simulation, which is not among the replication files, or a new run.

Evidence an exact counterexample

The evidence is an exact counterexample and a derivation, together with a reading of the two scripts. Let tjrt_{jr} be the estimate of estimator jj in the simulated dataset rr, let BrB_r be the benchmark of that dataset, and let RR be the number of simulated datasets for a given nn. Both scripts return

(1R∑r=1R(tjr−Br)2)1/2where the design requires(1R∑r=1R(tjr−5)2)1/2.\Bigl(\frac{1}{R}\sum_{r=1}^{R}(t_{jr} - B_r)^2\Bigr)^{1/2} \quad\text{where the design requires}\quad \Bigl(\frac{1}{R}\sum_{r=1}^{R}(t_{jr} - 5)^2\Bigr)^{1/2}.

For the counterexample, I take n=500n = 500 units in 5050 identical blocks of ten. Set a=⋯=h=0a = \dots = h = 0, let the latent covariates be (X1i,X2i,X3i)=(1/5,0,0)(X1_i, X2_i, X3_i) = (1/5, 0, 0) for every unit, and let all three disturbances be zero. In each block the observed covariates take the five values (0,0,0)(0,0,0), (1/2,0,0)(1/2,0,0), (0,1/2,0)(0,1/2,0), (0,0,1/2)(0,0,1/2) and (−1/2,−1/2,−1/2)(-1/2,-1/2,-1/2), each once with Zi=0Z_i = 0 and once with Zi=1Z_i = 1. Then ACi=1AC_i = 1 and Ai=0A_i = 0 for every unit, so that Di=ZiD_i = Z_i, Y0i=5Y0_i = 5 and Y2i=5+5ZiY2_i = 5 + 5Z_i. The regressors and the instruments coincide and have full rank, and the 2SLS coefficient on DD is exactly 55. Since tau1i=5(1+1/5)=6\texttt{tau1}_i = 5(1 + 1/5) = 6 for every unit, I have B=6B = 6. The code assigns the squared error (5−6)2=1(5-6)^2 = 1 to an estimate that equals the true effect. A second input differs only in the disturbance of the outcome, which equals ZiZ_i. For this input, I have Y2i=5+6ZiY2_i = 5 + 6Z_i, the 2SLS estimate is 66, and again B=6B = 6. The code assigns the squared error 00 to an estimate that is off by 11.

The two exact inputs with n=500n = 500: the 2SLS estimate tt, the benchmark BB that the code subtracts, and the two squared errors.
InputttBB(t−B)2(t-B)^2, as coded(t−5)2(t-5)^2, correct
First5610
Second6601

The original functions of the replication files, run on the first input, return 55 for all four homogeneous fits, and the RMSE expressions of both scripts return 11 for 2SLS; with the benchmark replaced by 55 they return 00. Since the thresholds hold with strict inequality and the designs are nonsingular in both inputs, the same discrepancy occurs on a set of inputs of positive probability.

I compute the variance of the benchmark from the design. A variable that is uniform on (−1,1)(-1,1) has mean 00 and variance 1/31/3, and B−5B - 5 is 5/n5/n times a sum of 3n3n independent such variables. I obtain

E(B)=5,Var⁡(B)=25n2⋅3n⋅13=25n.E(B) = 5, \qquad \operatorname{Var}(B) = \frac{25}{n^2}\cdot 3n\cdot\frac{1}{3} = \frac{25}{n}.

For any estimate tt, I have

(t−B)2−(t−5)2=(B−5)2−2(t−5)(B−5).(t - B)^2 - (t - 5)^2 = (B - 5)^2 - 2(t - 5)(B - 5).

If tt has a finite second moment, taking expectations gives E[(t−B)2]−E[(t−5)2]=25/n−2Cov⁡(t,B)E[(t-B)^2] - E[(t-5)^2] = 25/n - 2\operatorname{Cov}(t, B). The two inputs above show that the difference can have either sign in a single dataset. The size of the change in each entry of Table A3, and whether the entry rises or falls, is therefore not established.

What survives

The definition of the homogeneous design and the fitted estimates are not affected, and 2SLS and ICSW both estimate the constant effect 55 in this design. The correction changes the scoring of one supplementary design: Table A3 and one panel of Figure A3 have to be regenerated. The identification and consistency arguments, the two applications, the other simulation designs, the study of the Winsorization constant and the bootstrap study do not depend on it. Whether the corrected RMSEs of 2SLS and ICSW remain close to each other is not established.

Corrected

Replication files. In the six RMSE statements for rows 9 to 12 of Simulations Misspecification Noise 6-20.R (lines 125, 127, 129, 131, 133 and 135) and in the six corresponding statements of Sim-Display Misspec.R (lines 60, 62, 64, 66, 68 and 70), the benchmark is the constant 55:

t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)(x - 5)^2))^.5,mean(simoutput[7,isN]))\begin{aligned} &\texttt{t500 <- c(colMeans(apply(simoutput[9:12,isN],1,function(x)}\\ &\qquad\texttt{(x - }\colorbox{#ffe58a}{$\displaystyle \texttt{5}$}\texttt{)\textasciicircum 2))\textasciicircum .5,mean(simoutput[7,isN]))} \end{aligned}

Row 1 and the statements for rows 2 to 5, which score the heterogeneous design, stay as they are.

Supplement. For the design with τi=5\tau_i = 5 for all ii, the RMSE of estimator jj at sample size nn is

RMSEj=(1R∑r=1R(tjr−5)2)1/2,\mathrm{RMSE}_j = \Bigl(\frac{1}{R}\sum_{r=1}^{R}(t_{jr} - \colorbox{#ffe58a}{$\displaystyle 5$})^2\Bigr)^{1/2},

where tjrt_{jr} is the estimate in the simulated dataset rr and R=2,500R = 2{,}500 is the number of simulated datasets. Regenerate Table A3 and the “Noise (Homogeneous)” panel of Figure A3 with this formula, from the saved output of the simulation or from a new run. The corrected entries are not established here.

Crop of a table with the caption Table A3: Noise (Homogeneous). The columns are n, OLS (Z), OLS (D), 2SLS and ICSW. The rows read: 500: 2.50, 2.64, 0.53, 0.58. 1000: 2.50, 2.62, 0.37, 0.39. 2500: 2.49, 2.61, 0.23, 0.23. 5000: 2.49, 2.64, 0.16, 0.16. 10000: 2.47, 2.61, 0.11, 0.11. 25000: 2.48, 2.63, 0.07, 0.07. One red outline surrounds all 24 entries of the four estimator columns.
Supplement, Table A3, p. A7 (supplement PDF p. 7). The outline marks the 24 RMSE entries; their replication formulas subtract the heterogeneous sample-average effect BB.
Crop of three lines. A bold heading reads Homogeneous Treatment Effects, Conditioning on Random Noise. The next line reads Same as the previous DGP, except: and the last line reads tau sub i = 5, for all i. A red outline surrounds the last line.
Supplement, p. A2 (supplement PDF p. 2). The outline marks the line that sets the treatment effect of every unit to 55 in the homogeneous design.
Listing of R code with line numbers, in two parts. In the first part, line 22 sets tau1 to 5*(1+X1+X2+X3), line 24 sets Y2 to Y0 + 5*(D), line 26 returns list(D,Y1,tau1,C,Y2), line 83 sets output[1] to mean(dgpout[[3]]), and lines 91 to 94 store in output[9] to output[12] the four estimates computed from Y2. Lines 125 and 127 compute t500 and t1000 as the square root of the column means of (x - simoutput[1,isN]) squared over rows 9:12; a red outline surrounds simoutput[1,isN] on each of the two lines. In the second part, line 60 of the display script has the same statement for t500 with the same red outline, line 72 calls xtable on the six results, and line 74 plots them with the title Noise (Homogeneous).
Source code of the replication files, not a page of the article or of the supplement: Simulations Misspecification Noise 6-20.R, lines 22–26, 83, 91–94 and 124–127, and Sim-Display Misspec.R, lines 59–60 and 72–74. The outlines mark the benchmark simoutput[1,isN] in the RMSE statements for rows 9 to 12.
Line chart. The horizontal axis is the sample size n on a logarithmic scale, with ticks at 500, 1,000, 2,500, 5,000, 10,000 and 25,000. The vertical axis is labelled RMSE or standard deviation (units of the outcome) and runs from 0 to about 0.65. The title reads: Table A3 measures error about a benchmark that is itself random, where the effect in the homogeneous design is exactly 5. An orange line with square markers gives the ICSW RMSE printed in Table A3: 0.58, 0.39, 0.23, 0.16, 0.11 and 0.07. A blue line with round markers gives the printed 2SLS RMSE: 0.53, 0.37, 0.23, 0.16, 0.11 and 0.07. A dashed green line with triangular markers gives the standard deviation of the benchmark used by the code, 5 divided by the square root of n: 0.224, 0.158, 0.100, 0.071, 0.050 and 0.032. A note below the chart reads: Corrected RMSEs are not shown: they need the simulation output or a new run.
The RMSEs of 2SLS and ICSW printed in Table A3 and the standard deviation 5/n5/\sqrt{n} of the benchmark that the code subtracts from each estimate, by sample size. The benchmark has mean 55, the true effect, and its standard deviation falls from 0.2240.224 at n=500n = 500 to 0.0320.032 at n=25,000n = 25{,}000. Corrected RMSEs are not shown, because they are not established.

In the line of the replication script that assembles the upper bootstrap limits for Table 2, the last subscript [310] should read [31:40]; as deposited, the line returns missing values for the ten upper limits of the opinion ICSW model, and no printed limit of Table 2 is shown to be wrong.

Replication script albertsonlawrence.R, line 195; the paper’s counterpart is Table 2, column (4), upper confidence limits, p. 503 (PDF p. 12)

As printed

Script, lines 193 and 195 (each is one line of code, broken here after each argument).

mat1 <- round(cbind(apply(ests,2,quantile,c(0.025))[1:10],apply(ests,2,quantile,c(0.025))[11:20],apply(ests,2,quantile,c(0.025))[21:30],apply(ests,2,quantile,c(0.025))[31:40]),digits=2)mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],apply(ests,2,quantile,c(0.975))[11:20],apply(ests,2,quantile,c(0.975))[21:30],apply(ests,2,quantile,c(0.975))[310]),digits=2)\begin{aligned} &\texttt{mat1 <- round(cbind(apply(ests,2,quantile,c(0.025))[1:10],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.025))[11:20],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.025))[21:30],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.025))[31:40]),digits=2)}\\ &\texttt{mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.975))[11:20],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.975))[21:30],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.975))}\colorbox{#ffe58a}{$\displaystyle \texttt{[310]}$}\texttt{),digits=2)} \end{aligned}

Table 2, column (4), Opinion, ATE (ICSW), p. 503. Watching debate: −0.05-0.05 (−0.31,0.24)(-0.31, 0.24).

The problem

The bootstrap matrix ests of the script has 40 columns: ten coefficients for each of four models, in the order knowledge 2SLS, knowledge ICSW, opinion 2SLS and opinion ICSW. Line 193 assembles the lower limits from the columns 1 to 10, 11 to 20, 21 to 30 and 31 to 40. Line 195 does the same for the upper limits, except that its last subscript is [310]. The vector of 40 quantiles has no element 310. R returns a single missing value there, and cbind repeats it ten times. The fourth column of mat2, which should hold the upper limits of the opinion ICSW model in column (4) of Table 2, is then missing in all ten rows.

Evidence an exact counterexample

The evidence is an exact counterexample. I ran the two lines as deposited on a matrix with two rows and 40 columns such that the entries in column jj are jj and j+40j + 40. The quantiles of column jj at 0.0250.025 and 0.9750.975 are j+1j + 1 and j+39j + 39. Line 193 returns the four intended columns. Line 195 returns 40,…,4940, \ldots, 49, then 50,…,5950, \ldots, 59, then 60,…,6960, \ldots, 69, and ten missing values in the fourth column. With [31:40] it returns 70,…,7970, \ldots, 79 there and leaves the other columns unchanged. Neither mat1 nor mat2 is used elsewhere in the replication scripts, and the script saves its workspace at line 155, before both lines.

What survives

The typo comes after the point-estimation and bootstrap calculations of the script. The point estimates, the bootstrap draws, the quantiles of the differences in coefficients, the matrix of lower limits and the first three columns of upper limits do not pass through it. Table 2 prints ten finite upper limits in column (4), and none is shown to be wrong. How the table was assembled from the output is not known. Bootstrap draws with missing coefficients are a separate question, which the repair does not address, and I did not establish that any occurred.

Corrected

Script, line 195 (one line of code).

mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],apply(ests,2,quantile,c(0.975))[11:20],apply(ests,2,quantile,c(0.975))[21:30],apply(ests,2,quantile,c(0.975))[31:40]),digits=2)\begin{aligned} &\texttt{mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.975))[11:20],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.975))[21:30],}\\ &\qquad\texttt{apply(ests,2,quantile,c(0.975))}\colorbox{#ffe58a}{$\displaystyle \texttt{[31:40]}$}\texttt{),digits=2)} \end{aligned}
Crop of the top of Table 2, titled 2SLS and ICSW estimates for Albertson and Lawrence (2009). Under Knowledge are the columns LATE (2SLS), (1), and ATE (ICSW), (2); under Opinion are the columns LATE (2SLS), (3), and ATE (ICSW), (4). Row Watching debate: 0.28 with interval (-0.04, 0.59); 0.40 with interval (0.01, 1.26); -0.07 with interval (-0.25, 0.13); -0.05 with interval (-0.31, 0.24). Row Intercept: 1.81 with interval (1.33, 2.26); 1.90 with interval (1.34, 2.50); 1.01 with interval (0.70, 1.30); 0.99 with interval (0.60, 1.29). A red outline surrounds column (4), from its heading ATE (ICSW) to the interval (0.60, 1.29).
Table 2, title, column headings and first two rows, p. 503 (PDF p. 12). The outline marks column (4), the opinion ICSW model, whose upper limits line 195 of the script is written to assemble. The printed limits are finite.
Image of R code in a monospaced face, under the heading albertsonlawrence.R, lines 193-195 (replication files). Line 193, over three rows: mat1 <- round(cbind(apply(ests,2,quantile,c(0.025))[1:10],apply(ests,2,quantile,c(0.025))[11:20],apply(ests,2,quantile,c(0.025))[21:30],apply(ests,2,quantile,c(0.025))[31:40]),digits=2). Line 194 is empty. Line 195, over three rows: mat2 <- round(cbind(apply(ests,2,quantile,c(0.975))[1:10],apply(ests,2,quantile,c(0.975))[11:20],apply(ests,2,quantile,c(0.975))[21:30],apply(ests,2,quantile,c(0.975))[310]),digits=2). A red outline surrounds the subscript [310] near the end of line 195.
Replication script albertsonlawrence.R, lines 193–195, typeset from the replication files (a script has no published page). The outline marks the subscript [310] in line 195; line 193 has [31:40] in the same place.

In the one-sided branch of the function complier.score, the observation weights are inside the logarithm, so positive weights leave the likelihood minimizers unchanged; the weights should multiply the logarithm, as in the two-sided branch.

Replication files, icsw.R, function complier.score, branch for one-sided noncompliance, lines 45–52 (objective on line 50); the objective for two-sided noncompliance is on lines 39–40

See also under Formal statements (Lean): P029-T8

As printed

Replication files, icsw.R, lines 39–40 (two-sided noncompliance) and lines 49–50 (one-sided noncompliance).

return(-sum(weights*log((p.D)^D * (1 - p.D)^(1 - D))))return(-sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights)))\begin{aligned} &\texttt{return(}\\ &\quad\texttt{-sum(weights*log((p.D)\textasciicircum D * (1 - p.D)\textasciicircum (1 - D))))}\\ &\texttt{return(}\\ &\quad\texttt{-sum(log((p.D)\textasciicircum D * (1 - p.D)\textasciicircum (1 - D)}\colorbox{#ffe58a}{$\displaystyle \texttt{ * weights}$}\texttt{)))} \end{aligned}
The problem

Let wiw_i be the weight of observation ii and let Li(θ)L_i(\theta) be its Bernoulli likelihood contribution. Line 50 computes

−∑ilog⁡(wiLi(θ))=−∑ilog⁡Li(θ)−∑ilog⁡wi.-\sum_i \log\bigl(w_i L_i(\theta)\bigr) = -\sum_i \log L_i(\theta) - \sum_i \log w_i .

The last sum does not depend on θ\theta. This implies that, for positive weights, the minimizers of this function are those of the unweighted negative log likelihood. A zero weight makes the function infinite. The function meant is the weighted negative log likelihood −∑iwilog⁡Li(θ)-\sum_i w_i \log L_i(\theta), which line 40 computes for two-sided noncompliance. No script of the replication files calls complier.score with onesided=TRUE, and no reported number is affected. Unequal weights therefore do not weight the fit as intended.

Evidence an exact counterexample

The evidence is an exact counterexample and a run of the original function. The example has six observations with one covariate WiW_i.

Six observations with one-sided noncompliance and positive weights.
iiWiW_iZiZ_iDiD_iwiw_i
1−1-1101
2−1-1113
311103
411111
5−1-1005
611007

With an intercept and one slope, the fitted probabilities p−p_- and p+p_+ of compliance at Wi=−1W_i = -1 and Wi=1W_i = 1 range freely over (0,1)2(0,1)^2, and the two control observations have Li=1L_i = 1. For a,b>0a, b > 0, the function −alog⁡p−blog⁡(1−p)-a\log p - b\log(1-p) has its unique minimum at p=a/(a+b)p = a/(a+b). The function on line 50 is −log⁡(p−(1−p−) p+(1−p+))-\log\bigl(p_-(1-p_-)\,p_+(1-p_+)\bigr) plus a constant, and its minimum is at (p−,p+)=(1/2,1/2)(p_-, p_+) = (1/2, 1/2). The weighted negative log likelihood is −log⁡(p−3(1−p−) p+(1−p+)3)-\log\bigl(p_-^3(1-p_-)\,p_+(1-p_+)^3\bigr), and its minimum is at (3/4,1/4)(3/4, 1/4). The weighted likelihood ∏iLiwi\prod_i L_i^{w_i} equals 1/2561/256 at the first point and 729/65536729/65536 at the second.

The original function was also run on eight observations: at each of Wi=−1W_i = -1 and Wi=1W_i = 1, three units with Zi=1Z_i = 1 and outcomes Di=0,0,1D_i = 0, 0, 1, with weights 1,1,61, 1, 6 and 3,3,23, 3, 2, and one unit with Zi=0Z_i = 0. It returns the score 1/31/3 for every unit, with unit weights and with these weights. The weighted fit is 3/43/4 at Wi=−1W_i = -1 and 1/41/4 at Wi=1W_i = 1; the corrected objective and a weighted probit regression both return it. With a weight equal to zero, the original objective is infinite at the starting value.

What survives

The canvassing application fits its scores by a weighted probit regression on the units with Zi=1Z_i = 1 and does not use this branch. The debate application and the simulations use the two-sided branch, whose objective weights the logarithm correctly. All 15 calls to complier.score in the replication files leave onesided at its default FALSE. No table, simulation or result of the paper changes.

Corrected

Replication files, icsw.R, lines 49–50.

return(-sum(weights[weights > 0]*log(((p.D)^D *(1 - p.D)^(1 - D))[weights > 0])))\begin{aligned} &\texttt{return(}\\ &\quad\texttt{-sum(}\colorbox{#ffe58a}{$\displaystyle \texttt{weights[weights > 0]*}$}\texttt{log(((p.D)\textasciicircum D *}\\ &\qquad\texttt{(1 - p.D)\textasciicircum (1 - D))}\colorbox{#ffe58a}{$\displaystyle \texttt{[weights > 0]}$}\texttt{)))} \end{aligned}

In full, for weights wi≥0w_i \ge 0 the one-sided branch minimizes over θ\theta

−∑i: wi>0wilog⁡Li(θ),Li(θ)=pi(θ)Di(1−pi(θ))1−Di,pi(θ)=Zi F(xi⊤θ),-\sum_{i:\,w_i > 0} w_i \log L_i(\theta), \qquad L_i(\theta) = p_i(\theta)^{D_i}\bigl(1 - p_i(\theta)\bigr)^{1 - D_i}, \qquad p_i(\theta) = Z_i\,F(x_i^{\top}\theta),

where xix_i consists of an intercept and the standardized covariates of unit ii, and FF is the distribution function passed as link, by default Φ\Phi. Observations with weight zero are left out before the logarithm is taken. The logarithm is evaluated as log⁡pi\log p_i when Di=1D_i = 1 and as log⁡(1−pi)\log(1 - p_i) when Di=0D_i = 0, which avoids the product 0⋅log⁡00\cdot\log 0 for units with Zi=0Z_i = 0.

Listing of R code with line numbers. Lines 39 and 40 read return( and -sum(weights*log((p.D)^D * (1 - p.D)^(1 - D)))). After a gap, line 45 begins the one-sided function with if (onesided == TRUE) complier.neg.logLik.fnc <- function (par), lines 46 to 48 set theta, p.C and p.D <- p.C * Z, and lines 49 and 50 read return( and -sum(log((p.D)^D * (1 - p.D)^(1 - D) * weights))). A red outline surrounds the characters * weights on line 50.
Source code of the replication files, not a page of the article: icsw.R, lines 39–41 and 45–52. The outline marks the weights inside the logarithm on line 50; on line 40 the weights multiply the logarithm.

The function complier.score fits the likelihood with the distribution function passed as link and then computes the returned scores with the normal distribution function; the two agree for the default probit, which every analysis of the paper uses.

Replication files, icsw.R, function complier.score, lines 86–87 and 98; the argument link is declared on line 15 and used in the likelihood on lines 32–33 and 47

As printed

Replication files, icsw.R, lines 32–33 and 47 (likelihood) and lines 86, 87 and 98 (returned scores). The last three lines are broken here after the multiplication sign.

p.AC <- link(rowSums(t(t(W1) * theta)))p.A <- link(rowSums(t(t(W1) * theta.A)))p.C <- link(rowSums(t(t(W1) * theta)))CA.score <- (pnorm(rowSums(t(t(W1) *opt.log$par[1:(length(opt.log$par)/2)]))))A.score <- (pnorm(rowSums(t(t(W1) *opt.log$par[(length(opt.log$par)/2+1):(length(opt.log$par))])))) * CA.scoreC.score <- (pnorm(rowSums(t(t(W1) *opt.log$par[1:(length(opt.log$par))]))))\begin{aligned} &\texttt{p.AC <- link(rowSums(t(t(W1) * theta)))}\\ &\texttt{p.A <- link(rowSums(t(t(W1) * theta.A)))}\\ &\texttt{p.C <- link(rowSums(t(t(W1) * theta)))}\\ &\texttt{CA.score <- (}\colorbox{#ffe58a}{$\displaystyle \texttt{pnorm}$}\texttt{(rowSums(t(t(W1) *}\\ &\qquad\texttt{opt.log\$par[1:(length(opt.log\$par)/2)]))))}\\ &\texttt{A.score <- (}\colorbox{#ffe58a}{$\displaystyle \texttt{pnorm}$}\texttt{(rowSums(t(t(W1) *}\\ &\qquad\texttt{opt.log\$par[(length(opt.log\$par)/2+1):(length(opt.log\$par))])))) * CA.score}\\ &\texttt{C.score <- (}\colorbox{#ffe58a}{$\displaystyle \texttt{pnorm}$}\texttt{(rowSums(t(t(W1) *}\\ &\qquad\texttt{opt.log\$par[1:(length(opt.log\$par))]))))} \end{aligned}
The problem

The article defines the fitted compliance score with the same distribution function FF as the likelihood (p. 497), and it takes F=ΦF = \Phi for its own calculations. In the function, a distribution function supplied through link enters the likelihood, but the returned scores apply pnorm to the fitted coefficients. With the default link = pnorm the two coincide for every coefficient vector. With another distribution function, the returned scores can differ from the scores of the fitted model.

Evidence an exact counterexample

The evidence is an exact counterexample and a run of the original function. Let F(t)=1/(1+3e−t)F(t) = 1/(1 + 3e^{-t}), which is a distribution function with F(0)=1/4F(0) = 1/4, and recall that Φ(0)=1/2\Phi(0) = 1/2. The data have 6464 observations with one covariate Wi=±1W_i = \pm 1 and 1616 observations in each of the four cells of (Wi,Zi)(W_i, Z_i). In each cell with Zi=0Z_i = 0 one unit is treated, and in each cell with Zi=1Z_i = 1 four units are treated. At the zero coefficient vector, the model assigns probability q=F(0)=1/4q = F(0) = 1/4 to being a complier or an always-taker and probability r=F(0)=1/4r = F(0) = 1/4 to being an always-taker given either type. The fitted probabilities of treatment are then qr=1/16qr = 1/16 when Zi=0Z_i = 0 and q=1/4q = 1/4 when Zi=1Z_i = 1. Since these equal the observed proportions in all four cells, the zero vector maximizes the likelihood, and it is the unique maximizer because FF is strictly increasing. The fitted model has always-taker score qr=1/16qr = 1/16 and complier score q(1−r)=3/16q(1-r) = 3/16. The function returns Φ(0)Φ(0)=1/4\Phi(0)\Phi(0) = 1/4 and Φ(0)(1−Φ(0))=1/4\Phi(0)(1 - \Phi(0)) = 1/4. For the one-sided branch, I use the same data with no treated unit among those with Zi=0Z_i = 0. The fitted complier score is F(0)=1/4F(0) = 1/4, and the function returns Φ(0)=1/2\Phi(0) = 1/2. A run of the original function on both datasets reproduces the returned values.

What survives

All 15 calls to complier.score in the replication files omit link and use the probit, for which the returned scores are those of the fitted model. The canvassing application obtains its scores from a probit regression. The article’s formula for the score with a general FF is correct as printed. No reported number changes.

Corrected

Replication files, icsw.R, lines 86, 87 and 98.

CA.score <- (link(rowSums(t(t(W1) *opt.log$par[1:(length(opt.log$par)/2)]))))A.score <- (link(rowSums(t(t(W1) *opt.log$par[(length(opt.log$par)/2+1):(length(opt.log$par))])))) * CA.scoreC.score <- (link(rowSums(t(t(W1) *opt.log$par[1:(length(opt.log$par))]))))\begin{aligned} &\texttt{CA.score <- (}\colorbox{#ffe58a}{$\displaystyle \texttt{link}$}\texttt{(rowSums(t(t(W1) *}\\ &\qquad\texttt{opt.log\$par[1:(length(opt.log\$par)/2)]))))}\\ &\texttt{A.score <- (}\colorbox{#ffe58a}{$\displaystyle \texttt{link}$}\texttt{(rowSums(t(t(W1) *}\\ &\qquad\texttt{opt.log\$par[(length(opt.log\$par)/2+1):(length(opt.log\$par))])))) * CA.score}\\ &\texttt{C.score <- (}\colorbox{#ffe58a}{$\displaystyle \texttt{link}$}\texttt{(rowSums(t(t(W1) *}\\ &\qquad\texttt{opt.log\$par[1:(length(opt.log\$par))]))))} \end{aligned}

In full, with fitted coefficient vectors θ^\widehat\theta and θ^A\widehat\theta_A and the distribution function FF passed as link, the two-sided branch returns the always-taker score F(xi⊤θ^) F(xi⊤θ^A)F(x_i^{\top}\widehat\theta)\,F(x_i^{\top}\widehat\theta_A) and the complier score F(xi⊤θ^)(1−F(xi⊤θ^A))F(x_i^{\top}\widehat\theta)\bigl(1 - F(x_i^{\top}\widehat\theta_A)\bigr), and the one-sided branch returns the complier score F(xi⊤θ^)F(x_i^{\top}\widehat\theta). Calls with the default probit are unchanged.

Listing of R code with line numbers. Line 15 declares complier.score with the arguments D, Z, W, weights = NULL, link = pnorm, genoud = TRUE, num.iter and onesided=FALSE. Lines 32 and 33 set p.AC and p.A to link(rowSums(...)) and line 47 sets p.C to link(rowSums(...)). Line 86 sets CA.score to pnorm(rowSums(...)), line 87 sets A.score to pnorm(rowSums(...)) times CA.score, line 88 sets C.score to CA.score - A.score, and line 98 sets C.score to pnorm(rowSums(...)) in the one-sided branch. Red outlines surround the word pnorm on lines 86, 87 and 98.
Source code of the replication files, not a page of the article: icsw.R, lines 15, 32–33, 47, 86–89 and 98–99. The outlines mark pnorm in the three statements that compute the returned scores; the likelihood on lines 32, 33 and 47 uses link.

With genoud = FALSE, the package function compliance.score stops with an error under two-sided noncompliance: it passes control names that the BOBYQA optimizer does not use, and it tests a field that the optimizer does not return.

R package icsw, version 1.0.0, file R/compliance.score.R, lines 110–116 (the branch genoud = FALSE of the fit for two-sided noncompliance)

As printed

Package icsw, R/compliance.score.R, lines 111–115.

opt.log <- minqa::bobyqa(starting, complier.neg.logLik.fnc,control = list(maxit = num.iter, trace = 0))if (opt.log$convergence > 0) warning("Nelder-Mead did not converge!")\begin{aligned} &\texttt{opt.log <- minqa::bobyqa(}\\ &\quad\texttt{starting, complier.neg.logLik.fnc,}\\ &\quad\texttt{control = list(}\colorbox{#ffe58a}{$\displaystyle \texttt{maxit}$}\texttt{ = num.iter, }\colorbox{#ffe58a}{$\displaystyle \texttt{trace}$}\texttt{ = 0)}\\ &\quad\texttt{)}\\ &\texttt{if (opt.log\$}\colorbox{#ffe58a}{$\displaystyle \texttt{convergence}$}\texttt{ > 0) warning("}\colorbox{#ffe58a}{$\displaystyle \texttt{Nelder-Mead}$}\texttt{ did not converge!")} \end{aligned}
The problem

The function bobyqa of the package minqa takes the controls maxfun and iprint, and it returns a list with the components par, fval, feval, ierr and msg. The names maxit and trace are ignored with a warning, and a value of num.iter chosen by the user has no effect. The returned list has no component convergence. The expression opt.log$convergence is then NULL, the comparison with 00 has length zero, and if stops with the error “argument is of length zero”. No scores are returned. The warning names Nelder–Mead, and the optimizer is BOBYQA. The default genoud = TRUE and the fit for one-sided noncompliance do not pass through these lines.

Evidence a derivation

The evidence is a derivation from the interface of the optimizer and a run of the package source, under R 4.4.2 with minqa 1.2.8. The example has 32 units with two-sided noncompliance, eight in each cell of a binary covariate and the assignment, with treatment rates 1/41/4 and 3/43/4. The unmodified function issues the warning “unused control arguments ignored” and stops with “argument is of length zero”. The function icsw.tsls.fit with genoud = FALSE stops with the same error. With the corrections below, the function returns 32 finite scores and ierr equal to 00. With num.iter = 1 it returns ierr equal to 11 and the corrected warning. The function complier.score in icsw.R of the replication files, which the analyses of the paper use, calls bobyqa with maxfun and iprint, has no such test, and returns scores on the same example.

What survives

The defect is in a branch of the package that is not the default, and it stops the computation without returning a number. The scripts for the simulations and the applications of the paper use the function in the replication files and not the package. No definition, result or reported number of the paper is affected. The correction restores the branch; it does not show that BOBYQA finds the global maximum of the likelihood.

Corrected

Package icsw, R/compliance.score.R, lines 111–115.

opt.log <- minqa::bobyqa(starting, complier.neg.logLik.fnc,control = list(maxfun = num.iter, iprint = 0))if (opt.log$ierr > 0) warning("BOBYQA did not converge!")\begin{aligned} &\texttt{opt.log <- minqa::bobyqa(}\\ &\quad\texttt{starting, complier.neg.logLik.fnc,}\\ &\quad\texttt{control = list(}\colorbox{#ffe58a}{$\displaystyle \texttt{maxfun}$}\texttt{ = num.iter, }\colorbox{#ffe58a}{$\displaystyle \texttt{iprint}$}\texttt{ = 0)}\\ &\quad\texttt{)}\\ &\texttt{if (opt.log\$}\colorbox{#ffe58a}{$\displaystyle \texttt{ierr}$}\texttt{ > 0) warning("}\colorbox{#ffe58a}{$\displaystyle \texttt{BOBYQA}$}\texttt{ did not converge!")} \end{aligned}
Listing of R code with line numbers, in two parts. In the first part, lines 111 to 114 call minqa::bobyqa with starting, complier.neg.logLik.fnc and control = list(maxit = num.iter, trace = 0), and line 115 reads if (opt.log$convergence > 0) warning("Nelder-Mead did not converge!"). Red outlines surround maxit = num.iter, trace = 0 on line 113, and opt.log$convergence and Nelder-Mead on line 115. The second part shows line 81 of icsw.R, which calls bobyqa with control=list(iprint=0,maxfun=num.iter), lower=-5, upper=5, without marks.
Source code, not a page of the article: package icsw, R/compliance.score.R, lines 110–116, and for comparison line 81 of icsw.R in the replication files. The outlines mark the two control names, the field that the optimizer does not return, and the name of another optimizer in the warning.

The script that draws Figures A1 and A2 stops with an error when a compliance group contains exactly one simulated dataset, because the extraction of the group loses its matrix shape; adding drop=FALSE repairs it.

Replication files, Sim-Display Alpha.R, lines 10, 17 and 24; the script draws Figures A1 and A2 of the supplement (supplement PDF pp. 3–4)

As printed

Replication files, Sim-Display Alpha.R, lines 8 and 10. Line 10 is broken here after function(x). Lines 17 and 24 repeat it with med and high in place of low.

low <- simoutput[6,] < 0.33RMSE <- apply(simoutput[7:nrow(simoutput),low],1,function(x)mean((x - simoutput[1,low])^2)^.5)\begin{aligned} &\texttt{low <- simoutput[6,] < 0.33}\\ &\texttt{RMSE <- apply(}\colorbox{#ffe58a}{$\displaystyle \texttt{simoutput[7:nrow(simoutput),low]}$}\texttt{,1,function(x)}\\ &\qquad\texttt{mean((x - simoutput[1,low])\textasciicircum 2)\textasciicircum .5)} \end{aligned}
The problem

When low selects exactly one column, R returns a vector in place of a matrix with one column, and apply stops with the error “dim(X) must have a positive length”. The same holds for med and high. No group with a single dataset is shown to occur in the published runs of 2,500 simulated datasets. With two or more datasets in a group the statements work, and the RMSE formulas are right.

Evidence an exact counterexample

The evidence is an exact counterexample, obtained by evaluating the statements of the script. On a matrix with nine rows and three columns whose compliance row is (0.2,0.4,0.6)(0.2, 0.4, 0.6), each group contains one dataset, and all three statements stop with the error above. With drop=FALSE added, each statement returns an RMSE of 11 in each of its three rows. On a matrix with two datasets in each group, the original and the repaired statements return the same values.

What survives

The RMSE formulas and all calculations for groups with two or more datasets are unaffected, and the published figures are not shown to be affected.

Corrected

Replication files, Sim-Display Alpha.R, line 10, and likewise lines 17 and 24 with med and high.

RMSE <- apply(simoutput[7:nrow(simoutput),low,drop=FALSE],1,function(x)mean((x - simoutput[1,low])^2)^.5)\begin{aligned} &\texttt{RMSE <- apply(simoutput[7:nrow(simoutput),low}\colorbox{#ffe58a}{$\displaystyle \texttt{,drop=FALSE}$}\texttt{],1,function(x)}\\ &\qquad\texttt{mean((x - simoutput[1,low])\textasciicircum 2)\textasciicircum .5)} \end{aligned}

The script may also skip an empty group and report its RMSE as undefined.

Listing of R code with line numbers. Line 8 reads low <- simoutput[6,] < 0.33 and line 10 reads RMSE <- apply(simoutput[7:nrow(simoutput),low],1,function(x) mean((x - simoutput[1,low])^2)^.5). Lines 15 and 17 repeat the pair for med, defined as (simoutput[6,] < 0.50) and (0.33 <= simoutput[6,]), and lines 22 and 24 repeat it for high, defined as simoutput[6,] >= 0.50. Red outlines surround simoutput[7:nrow(simoutput),low] on line 10, simoutput[7:nrow(simoutput),med] on line 17 and simoutput[7:nrow(simoutput),high] on line 24.
Source code of the replication files, not a page of the supplement: Sim-Display Alpha.R, lines 8–10, 15–17 and 22–24. The outlines mark the three extractions that lose the matrix shape when a group has one dataset.

The quotation from Deaton reads “processes” in the original and is on p. 130 of the edition listed, and five entries of the reference list have a wrong name, title, issue or page range.

Section 1, p. 492 (PDF p. 1), quotation from Deaton; reference list, pp. 505–506 (PDF pp. 14–15), entries for Angrist and Fernandez-Val, Deaton, Funk et al., Joffe and Brensinger, Putnam, and Verba, Schlozman and Brady

As printed

Section 1, p. 492. Both papers argue that the LATE is not usually the parameter of interest, as “we are unlikely to learn much about the process at work” by estimating the LATE (Deaton 2009, 430) […]

Reference list, pp. 505–506.

  • Angrist, Joshua D., and Ivan Fernandez-Val. 2013. ExtrapoLATE-ing: External validity and overidentification in the LATE framework. In Advances in economics and econometrics: Tenth world congress, Vol. 3, eds. Manuel Arellano Daron Acemoglu, and Eddie Deke, 401–36. Cambridge: Cambridge University Press.
  • Deaton, Angus. 2009. Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development. Proceedings of the British Academy, 2008 Lectures 162:123–60.
  • Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til Strmer, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. American Journal of Epidemiology 173(7):761–7.
  • Joffe, Marshall M., and Colleen Brensinger. 2003. Weighting in instrumental variables and G-estimation. Statistics in Medicine 22(1):1285–303.
  • Putnam, Robert C. 2000. Bowling alone: The collapse and renewal of American Community. New York: Simon and Schuster.
  • Verba, Sidney, Kay Lehman Schlozman, and Henry E. Brady. 1995. Voluntarism in American Politics. Cambridge, MA: Harvard University Press.
The problem

The quotation changes a word of the original, and its page number does not belong to the edition in the reference list, whose pages are 123 to 160. Page 430 is the page of the passage in the version of the essay that appeared in the Journal of Economic Literature in 2010. In the reference list, two editors' names run together and a third is misspelled, one chapter ends at p. 434 and not at p. 436, an author’s name has lost a letter, an issue number and a middle initial are wrong, a subtitle has “renewal” for “revival”, and one title has lost its first words. Each cited work can still be identified.

Evidence a check of the printed page

The evidence is a check of the page against the cited works and the records of their publishers. In both editions of Deaton’s essay the sentence reads “We are unlikely to learn much about the processes at work”. It is on p. 430 of the 2010 version (Journal of Economic Literature 48(2), 424–455) and on p. 130 of the 2009 version in the Proceedings of the British Academy. The publisher’s record of the chapter by Angrist and Fernandez-Val names the editors Daron Acemoglu, Manuel Arellano and Eddie Dekel, with the pages 401–434. In the publisher’s record of the article by Joffe and Brensinger, the issue is Volume 22, Issue 8. Bibliographic records give the names Til Stürmer and Robert D. Putnam, and the titles Bowling Alone: The Collapse and Revival of American Community and Voice and Equality: Civic Voluntarism in American Politics.

One point is not settled. The same page of the paper cites “Imbens (2009, 403)”, and the reference list has an NBER working paper for this citation. Page 403 lies in the page range 399–423 of the version of that comment in the Journal of Economic Literature in 2010. I could not confirm the page of the quoted words in either edition.

What survives

The argument of the paper does not depend on any of these details, and the sense of the quotation is unchanged. No correction is proposed for the citation of Imbens until the edition is checked.

Corrected

Section 1, p. 492. Both papers argue that the LATE is not usually the parameter of interest, as “we are unlikely to learn much about the processes at work” by estimating the LATE (Deaton 2009, 130) […]

An alternative is to cite and list the 2010 version of the essay, Journal of Economic Literature 48(2):424–55, and to keep the page 430.

Reference list.

  • Angrist, Joshua D., and Ivan Fernandez-Val. 2013. ExtrapoLATE-ing: External validity and overidentification in the LATE framework. In Advances in economics and econometrics: Tenth world congress, Vol. 3, eds. Manuel Arellano, Daron Acemoglu, and Eddie Dekel, 401–34. Cambridge: Cambridge University Press.
  • Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til Stürmer, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. American Journal of Epidemiology 173(7):761–7.
  • Joffe, Marshall M., and Colleen Brensinger. 2003. Weighting in instrumental variables and G-estimation. Statistics in Medicine 22(8):1285–303.
  • Putnam, Robert D. 2000. Bowling alone: The collapse and revival of American Community. New York: Simon and Schuster.
  • Verba, Sidney, Kay Lehman Schlozman, and Henry E. Brady. 1995. Voice and Equality: Civic Voluntarism in American Politics. Cambridge, MA: Harvard University Press.
Crop of six lines of the introduction. The text reads: Several influential papers, including Deaton (2009) and Heckman and Urzua (2010), have expressed concern about the interpretation of the LATE. Both papers argue that the LATE is not usually the parameter of interest, as we are unlikely to learn much about the process at work by estimating the LATE (Deaton 2009, 430) and the LATE is often very difficult to interpret as an answer to an interesting economic question (Heckman and Urzua 2010, 35). The crop ends inside the next sentence. One red outline surrounds the word process, and another surrounds the citation (Deaton 2009, 430).
Section 1, p. 492 (PDF p. 1). The outlines mark the word “process” in the quotation and the citation with the page number 430.
Crop of thirteen lines of the reference list, from the entry for Angrist and Fernandez-Val (2013) to the entry for Deaton (2009). In the first entry the editors are printed as Manuel Arellano Daron Acemoglu, and Eddie Deke, with the pages 401-36. Red outlines surround the words Arellano Daron and the words Deke, 401-36. The entries for Angrist and Pischke, Arceneaux and Nickerson (2009 and 2010) and Aronow and Carnegie follow without marks. In the last entry, Deaton, Angus. 2009. Instruments of development, Proceedings of the British Academy, 2008 Lectures, a red outline surrounds 162:123-60.
Reference list, p. 505 (PDF p. 14). The outlines mark the two editors' names that run together, the misspelled name with the page range of the chapter, and the page range of the edition of Deaton’s essay that the list gives.
Crop of the two lines of one reference entry. The text reads: Funk, Michele Jonsson, Daniel Westreich, Chris Wiesen, Til Strmer, M. Alan Brookhart, and Marie Davidian. 2011. Doubly robust estimation of causal effects. American Journal of Epidemiology 173(7):761-7. A red outline surrounds the words Til Strmer.
Reference list, p. 505 (PDF p. 14). The outline marks the misspelled name in the entry for Funk et al.
Crop of sixteen lines at the top of the last page of the reference list, from the entry for Joffe and Brensinger (2003) to the entry for Verba, Schlozman and Brady (1995). Four red outlines surround: 22(1) in Statistics in Medicine 22(1):1285-303; Robert C. in Putnam, Robert C. 2000; the word renewal in Bowling alone: The collapse and renewal of American Community; and the title Voluntarism in American Politics. The entries for Joffe, Ten Have and Brensinger, for Rosenstone and Hansen, for Roy, Hogan and Marcus and the two entries for Rubin carry no marks.
Reference list, p. 506 (PDF p. 15). The outlines mark the issue number in the entry for Joffe and Brensinger, the middle initial and the word “renewal” in the entry for Putnam, and the shortened title in the entry for Verba, Schlozman and Brady.

In the simulation design, N(0,3)\mathbf{N}(0,3) is a normal draw with standard deviation 33, and the coefficients a,…,ha,\dots,h are drawn once for each simulated dataset; the supplement states neither convention.

Supplement, p. A1 (supplement PDF p. 1), “Baseline Data Generating Process”: the line for a,…,ha,\dots,h and the two terms N(0,3)\mathbf{N}(0,3)

See also under Formal statements (Lean): P029-T19

As printed
X1i,X2i,X3i∼U(−1,1)a,b,c,d,e,f,g,h∼U(−2,2)ACi=I[2+a+bX1i+cX2i+dX3i+N(0,3)]Ai=I[−2+e+fX1i+gX2i+hX3i+N(0,3)]ACi\begin{aligned} & X1_i, X2_i, X3_i \sim \mathbf{U}(-1,1)\\ & \colorbox{#ffe58a}{$\displaystyle a, b, c, d, e, f, g, h \sim \mathbf{U}(-2,2)$}\\ & AC_i = \mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \colorbox{#ffe58a}{$\displaystyle \mathbf{N}(0,3)$}]\\ & A_i = \mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \colorbox{#ffe58a}{$\displaystyle \mathbf{N}(0,3)$}]AC_i \end{aligned}
The problem

The supplement does not say whether the second argument of N(0,3)\mathbf{N}(0,3) is a variance or a standard deviation. A common convention in mathematics takes it as the variance. The simulation code draws the disturbances with standard deviation 33. The choice changes the compliance rates of the design: at zero covariates and a=0a = 0, the probability that ACi=1AC_i = 1 is Φ(2/3)=0.7475\Phi(2/3) = 0.7475 with standard deviation 33 and Φ(2/3)=0.8759\Phi(2/\sqrt{3}) = 0.8759 with variance 33. The supplement also does not say when the coefficients a,…,ha,\dots,h are drawn. They carry no index ii, which shows that all units share them. In the code they are drawn once for each simulated dataset. Someone who implements the design from the supplement alone has to guess both conventions. No reported number is shown to be wrong.

Evidence a check of the printed page

The evidence is a check of the page against the replication files. In Simulations Alpha 6-20.R, the function dgp generates one simulated dataset. It draws each disturbance with rnorm(N,0,3), where the third argument of rnorm is the standard deviation. Each coefficient is drawn with runif(1,-2,2), once for each simulated dataset, and the eight numbers are passed to dgp.

What survives

The simulation design in the code and all reported simulation results are not shown to be affected. The notation of the supplement already indicates that the coefficients are common to all units. The correction adds two sentences to the description of the design.

Corrected
X1i,X2i,X3i∼U(−1,1)a,b,c,d,e,f,g,h∼U(−2,2)ACi=I[2+a+bX1i+cX2i+dX3i+N(0,3)]Ai=I[−2+e+fX1i+gX2i+hX3i+N(0,3)]ACi\begin{aligned} & X1_i, X2_i, X3_i \sim \mathbf{U}(-1,1)\\ & a, b, c, d, e, f, g, h \sim \mathbf{U}(-2,2)\\ & AC_i = \mathbf{I}[2 + a + bX1_i + cX2_i + dX3_i + \mathbf{N}(0,3)]\\ & A_i = \mathbf{I}[-2 + e + fX1_i + gX2_i + hX3_i + \mathbf{N}(0,3)]AC_i \end{aligned}

Here N(0,3)\mathbf{N}(0,3) denotes a normal draw with mean zero and standard deviation 33. The scalar coefficients a,…,ha,\dots,h are drawn once for each simulated dataset and are shared by all of its observations.

An alternative for the first sentence is to write N(0,32)\mathbf{N}(0,3^2) and to state that the second argument is the variance.

Crop of five displayed lines. The first reads X1 sub i, X2 sub i, X3 sub i distributed as U(-1, 1). The second reads a, b, c, d, e, f, g, h distributed as U(-2, 2) and has a red outline. The third reads AC sub i = I[2 + a + b X1 sub i + c X2 sub i + d X3 sub i + N(0, 3)], with a red outline around N(0, 3). The fourth reads A sub i = I[-2 + e + f X1 sub i + g X2 sub i + h X3 sub i + N(0, 3)] AC sub i, with a red outline around N(0, 3). The fifth reads C sub i = AC sub i - A sub i.
Supplement, p. A1 (supplement PDF p. 1), first lines of the baseline design. The outlines mark the two terms N(0,3)\mathbf{N}(0,3), whose scale is not stated, and the line for the coefficients, which does not say when they are drawn.

The replication script rebuilds the age-missing indicator by testing whether an imputed age equals a recomputed mean, which on a current system flags 1 of the 2,488 records with a missing or impossible age and gives “Age missing” coefficients near 97 where Table 1 prints 12.7 and 17.7; the printed point estimates are reproduced when the indicator is set before the imputation.

Replication script greengerbernickerson.R, lines 33–36 and 45; the paper’s counterparts are Table 1, row “Age missing”, p. 501 (PDF p. 10), and supplement Table A7, column “Age M.” and row “Age Miss.”, p. A11 (supplement PDF p. 11)

As printed

Script, lines 33–36 and 45 (line 45 is one line of code, broken here).

age[age>101] <- mean(age[age <= 101],na.rm=TRUE)age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)ageM <- rep(0,N)ageM[age==mean(age[age <= 101],na.rm=TRUE)] <- 1covs <- cbind(makeind(data.frame(factor(city))),famsize,white,black,voted00,voted99a,primary,age,ageM,dem,rep,ind)[,2:17]\begin{aligned} &\texttt{age[age>101] <- mean(age[age <= 101],na.rm=TRUE)}\\ &\texttt{age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)}\\ &\texttt{ageM <- rep(0,N)}\\ &\texttt{ageM[}\colorbox{#ffe58a}{$\displaystyle \texttt{age==mean(age[age <= 101],na.rm=TRUE)}$}\texttt{] <- 1}\\ &\texttt{covs <- cbind(makeind(data.frame(factor(city))),famsize,white,black,voted00,}\\ &\qquad\texttt{voted99a,primary,age,ageM,dem,rep,ind)[,2:17]} \end{aligned}

Table 1, p. 501.

Table 1, row “Age missing”, with the bootstrap 95% confidence intervals.
LATE (2SLS), (1)ATE (ICSW), (2)
Age missing12.7 (−16.5,44.1)(-16.5, 44.1)17.7 (−15.9,50.3)(-15.9, 50.3)

Supplement Table A7, p. A11. Comp. Score, Age M.: −0.55-0.55.

The problem

The note to Table 1 lists a dummy variable for missing data on age among the covariates. The script first replaces the ages above 101, and then the missing ages, by the mean of the valid ages (lines 33 and 34). Only afterwards does it set the indicator to 1 for every record whose age equals that mean, which line 36 computes a third time. In exact arithmetic the three means agree, since adding copies of the mean to a sample does not change its mean. In floating-point arithmetic they need not agree to the last bit, and line 36 tests exact equality.

On R 4.4.2 for arm64 under macOS, the first and third means are equal and the second is smaller by 3/2473/2^{47}. The one age above 101 (the value 2001) was replaced by the first mean and passes the test. The 2,487 missing ages were replaced by the second mean and fail it. The indicator is then 1 for one record, where it should be 1 for 2,488. With this indicator the script returns “Age missing” coefficients of 97.3 and 96.6 percentage points, against the printed 12.7 and 17.7. The correlation between the compliance score and the indicator is −0.01-0.01, against the −0.55-0.55 of Table A7. The bootstrap of the script can also stop. A draw that omits the one flagged record leaves a column of zeros, the coefficient on that column is missing, and the quantile calls of the script do not remove missing values.

The test is unreliable in exact arithmetic too, in the other direction. For the ages 2020, 3030, 4040 and one missing value, the mean is 3030 at every stage, and the test flags the observed age 3030 along with the imputed one. I checked one system, and the number of flagged records may differ on another.

Evidence an exact counterexample

The evidence is a recalculation from the replication data, in which I compare the exact rational representations of the three floating-point means. The data hold 18,933 records, of which 2,487 have a missing age and one has an age above 101. The 16,445 valid ages have mean 742242/16445742242/16445. I evaluated the script’s expressions for the compliance score and for the two estimators twice, once with the indicator of lines 35–36 and once with the indicator set to 1 for the 2,488 records before the imputation.

Results from the replication data under the two constructions of the age-missing indicator. Coefficients are in percentage points.
QuantityScript as depositedIndicator set before imputationPrinted
Records flagged12,488
2SLS, Age missing97.3512.6512.7
ICSW, Age missing96.6017.7117.7
2SLS, Campaign contact5.645.615.6
ICSW, Campaign contact6.716.676.7
Correlation of score with indicator−0.01-0.01−0.55-0.55−0.55-0.55

With the indicator set before the imputation, I find that all 24 point estimates of Table 1 agree with the printed values to one decimal. The 17 entries of the column “Age M.” and the 12 entries of the row “Age Miss.” of Table A7 agree to two decimals. With the script as deposited, the ICSW coefficient on “Voted in 1999 general election” also changes, from 27.2 to 27.3.

For the bootstrap, I took one draw that the resampling of line 94 can produce, in which the flagged record is replaced by another record of its stratum. With the script as deposited, the coefficient on the indicator is missing in both models and both quantile calls stop with an error. With the indicator set before the imputation, the same draw runs. I did not recompute the 2,500-draw confidence intervals, and I do not know how or on what system the published run was made.

What survives

Table 1 and Table A7 are not shown to be wrong: the indicator set before the imputation reproduces every printed point estimate of Table 1 and the entries of Table A7 that involve the indicator. Under both constructions the effects of contact round to 5.6 and 6.7 percentage points, and the ATE is 19% larger than the LATE. In this script the compliance-score fit, the adjusted coefficients, the correlation matrix, the bootstrap and the permutation diagnostic all use the indicator. The confidence intervals and the diagnostic were not recomputed. The theory, the simulations and the application to Albertson and Lawrence (2009) do not use this script.

Corrected

In greengerbernickerson.R, the indicator should be set before age is imputed and kept in covs (line 45), and lines 35–36 should be deleted:

ageM <- as.numeric(is.na(age) | age > 101)age[age>101] <- mean(age[age <= 101],na.rm=TRUE)age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)\begin{aligned} &\colorbox{#ffe58a}{$\displaystyle \texttt{ageM <- as.numeric(is.na(age) | age > 101)}$}\\ &\texttt{age[age>101] <- mean(age[age <= 101],na.rm=TRUE)}\\ &\texttt{age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE)} \end{aligned}

On R 4.4.2 for arm64, the comparison in the script as deposited flags 1 of the 2,488 records with a missing or impossible age. The indicator set before the imputation flags all of them, and it reproduces the printed point estimates of Table 1 and the corresponding entries of Table A7.

Crop of three rows of Table 1, each with a coefficient and an interval in two columns. Age: 0.2 with interval (0.2, 0.2), and 0.2 with interval (0.1, 0.2). Age missing: 12.7 with interval (-16.5, 44.1), and 17.7 with interval (-15.9, 50.3). Democrat: 2.6 with interval (0.6, 4.7), and 2.6 with interval (0.7, 04.8). Red outlines surround the label Age missing and the two coefficients 12.7 and 17.7.
Table 1, rows “Age”, “Age missing” and “Democrat”, p. 501 (PDF p. 10). The two columns of numbers are LATE (2SLS), (1), and ATE (ICSW), (2). The outlines mark the label “Age missing” and its two coefficients, which the script as deposited does not reproduce.
Image of six lines of R code in a monospaced face, under the heading greengerbernickerson.R, lines 33-36 and 45 (replication files). Line 33: age[age>101] <- mean(age[age <= 101],na.rm=TRUE). Line 34: age[is.na(age)] <- mean(age[age <= 101],na.rm=TRUE). Line 35: ageM <- rep(0,N). Line 36: ageM[age==mean(age[age <= 101],na.rm=TRUE)] <- 1. Line 45, after an ellipsis: covs <- cbind(makeind(data.frame(factor(city))),famsize,white,black,voted00,voted99a,primary,age,ageM,dem,rep,ind)[,2:17]. A red outline surrounds the subscript of line 36, from the opening bracket before age==mean to the closing bracket.
Replication script greengerbernickerson.R, lines 33–36 and 45, typeset from the replication files (a script has no published page). The outline marks the comparison in line 36.

Arceneaux and Nickerson (2009) find larger effects of canvassing for high-propensity voters in low-salience elections only; in high-salience elections they find the opposite.

Footnote 1, p. 494 (PDF p. 3), last clause

As printed

Other examples include: Hidalgo et al. (2010) show that the effects of economic shocks on conflict are larger in regions with high levels of preexisting inequality; Altonji and Dunn (1996) demonstrate that the returns to education are larger for students in high-quality schools; Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are larger for high-propensity voters.

The problem

The clause reads as a general statement about the direction of the heterogeneity. In the cited article the direction depends on the salience of the election. The larger effect for high-propensity voters is a finding of that article for low-salience elections, and the footnote omits that the pattern reverses in high-salience elections.

Evidence a check of the printed page

The evidence is a check of the footnote against the cited article (American Journal of Political Science 53(1):1–16). In its discussion of Figure 4, the article has the phrase “boost turnout among high-propensity voters but not low-propensity voters in low-salience elections” on p. 10, and it continues on p. 11: “The opposite takes place in high-salience elections.” Its Table 3 (p. 12) reports effects of 3.3%, 11.7% and 12.3% for low-, medium- and high-propensity voters in low-salience elections, and of 12.4%, 11.6% and 3.2% in high-salience elections.

What survives

The footnote lists examples of heterogeneous treatment effects, and the cited article is such an example. Nothing in the paper depends on the direction of the heterogeneity. The corrected clause is narrower than the printed one.

Corrected

[…] Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are larger for high-propensity voters in low-salience elections.

The footnote may add that the pattern is reversed in high-salience elections.

Crop of the four lines of footnote 1. The text reads: Other examples include: Hidalgo et al. (2010) show that the effects of economic shocks on conflict are larger in regions with high levels of preexisting inequality; Altonji and Dunn (1996) demonstrate that the returns to education are larger for students in high-quality schools; Arceneaux and Nickerson (2009) find that the mobilizing effects of canvassing are larger for high-propensity voters. A red outline surrounds the last clause, from Arceneaux and Nickerson (2009) to the end.
Footnote 1, p. 494 (PDF p. 3). The outline marks the clause on Arceneaux and Nickerson (2009).

The knowledge estimate of Table 2 depends on the Winsorization constant, and I reproduce the printed point estimates; the error is the statement of footnote 6 that c→0c\to0, the same error that P029-E006 corrects.

Footnote 6, p. 498 (PDF p. 7); Table 2, row “Watching debate”, column (2), and the paragraph that discusses it, pp. 502–503 (PDF pp. 11–12)

As printed

Footnote 6: Winsorizing is performed because very low probabilities can introduce instability in the estimates resulting from IPW (Elliott 2009), but Winsorizing may sometimes increase finite sample bias. Asymptotically, assuming that α\alpha remains fixed, the Winsorization has no impact on estimation, because c→0c\to0 as n→∞n\to\infty.

Table 2, row “Watching debate”: 0.280.28 (−0.04,0.59)(-0.04, 0.59) in column (1), Knowledge, LATE (2SLS); 0.400.40 (0.01,1.26)(0.01, 1.26) in column (2), Knowledge, ATE (ICSW).

Text: Although Albertson and Lawrence (2009) find that compliers are 0.28 points more informed after viewing the program, we find that viewers in the entire subject population are 0.40 points more informed after viewing. We therefore estimate that the ATE is approximately 45% larger than the LATE.

The problem

The point raised is that the knowledge estimate depends on the constant α\alpha and that the article does not report this dependence. With n=507n=507 the rule in footnote 6 raises the 92 lowest scores, 18.1%18.1\% of the sample, to 0.2900.290. The estimate is 0.400.40 at α=0.275\alpha=0.275 and 0.610.61 without Winsorization, and it equals the 2SLS estimate at α=0\alpha=0. This dependence is not an error in Table 2. The estimates are computed by the rule that the article states, and the footnote says that Winsorizing can add finite-sample bias. The article does not claim that the estimate is free of the choice of α\alpha. The error is in the same footnote: the cut-off cc does not tend to zero. This is the statement corrected in P029-E006.

Evidence an exact counterexample

The evidence is a recalculation from the replication data and an exact counterexample to the limit. The recalculation fits the scores once and varies α\alpha. It reproduces all 40 point estimates of Table 2 to two decimals. The excess of the ICSW estimate over the 2SLS estimate is 44.9%44.9\% with unrounded coefficients. The bootstrap intervals were not recomputed.

Knowledge estimate for the debate application as α\alpha varies (n=507n=507; scores fitted once).
α\alphaCut-off ccScores raisedEstimateExcess over 2SLS
000.7170.7175065060.2760.2760.0%0.0\%
0.2750.2750.2900.29092920.4000.40044.9%44.9\%
0.50.50.1620.16223230.5050.50583.2%83.2\%
110.0650.065110.6120.612122.0%122.0\%
∞\infty0.0630.063000.6120.612121.9%121.9\%

For the limit, I consider one covariate profile and one-sided noncompliance with D0=0D_0=0 and D1=CD_1=C, where Pr⁡(C=1)=1/2\Pr(C=1)=1/2, outcomes Y0=0Y_0=0 and Y1=1Y_1=1, and Z∼Bernoulli(1/2)Z\sim\mathrm{Bernoulli}(1/2) independent of CC. The article’s assumptions hold, with ϵ=1/4\epsilon=1/4 in Assumption 3. Every true score is 1/21/2, and c=1/2c=1/2 for every nn with the exact scores. The fitted score is the proportion treated among the assigned units, which converges almost surely to 1/21/2, and cc converges to 1/21/2 with it. No score is changed, and the ICSW ratio equals the ATE of 11 whenever its empirical first stage is nonzero.

What survives

The Winsorization rule and its implementation, the 40 point estimates of Table 2 and the comparison of 45%45\% stand. The conclusion that Winsorization does not change the limit of the estimator holds under positivity and regularity conditions. The true ATE, the size of the bias from Winsorization and the best α\alpha for these data are not established.

Corrected

Footnote 6, in place of the sentence beginning “Asymptotically”: For fixed α>0\alpha > 0, the quantile level 1/nα1/n^{\alpha} tends to zero, so the fraction of scores changed vanishes. Under Assumption 3 and regularity conditions ensuring bounded estimated inverse scores with probability tending to one, integrable outcomes and a nonzero limiting weighted first stage, Winsorization does not change the probability limit of the displayed ICSW estimator.

P029-E006 gives the same correction with the regularity conditions written out. Table 2 and the comparison of 45%45\% are unchanged.

Crop of footnote 6, seven lines. One red box marks 'but Winsorizing may sometimes increase finite sample bias.' A second red box, on the next line, marks 'because c tends to 0 as n tends to infinity.'
Footnote 6, p. 498 (PDF p. 7). The upper mark is the article’s caution that Winsorizing can add finite-sample bias; the lower mark is the false limit.
Crop of the head of Table 2, '2SLS and ICSW estimates for Albertson and Lawrence (2009)', with columns Knowledge LATE (2SLS), Knowledge ATE (ICSW), Opinion LATE (2SLS) and Opinion ATE (ICSW). In the row 'Watching debate' the entries are 0.28 (-0.04, 0.59), 0.40 (0.01, 1.26), -0.07 (-0.25, 0.13) and -0.05 (-0.31, 0.24). A red box marks 0.40 (0.01, 1.26).
Table 2, p. 503 (PDF p. 12), first rows. The marked entry is the ICSW estimate of the effect of watching the debate on knowledge, with its bootstrap interval.

The heterogeneous-effect RMSEs and bootstrap coverage in the supplement are measured against the sample average effect of each simulated data set, n−1∑iτin^{-1}\sum_i\tau_i, and not against the population ATE of 5; the supplement and footnote 7 should say so.

Supplement, pp. A5 and A9 (PDF pp. 5 and 9), with Tables A1, A2 and A4–A6 and Figures A1–A3; article, footnote 7, p. 498 (PDF p. 7)

As printed

Supplement, p. A5:

We assume α=0.275\alpha = 0.275, as recommended in the text. We present tables of RMSE vs n∈500,1000,2500,5000,10000,25000n \in 500, 1000, 2500, 5000, 10000, 25000 under the 5 different models, including misspecification.

Supplement, p. A9:

In Table A6, we present % coverage with the bootstrap for n∈500,2500,5000n \in 500, 2500, 5000 assuming that the model is correct and α=0.275\alpha = 0.275.

Article, footnote 7:

In our own simulation studies, assuming α=0.275\alpha = 0.275, we find that the bootstrap has conservative coverage for ICSW in small samples, and proper coverage in larger samples.

Simulation scripts in the replication files: output[1] <- mean(dgpout[[3]])

The problem

The article defines the ATE as the population expectation E(Y1−Y0)\mathrm{E}(Y_1-Y_0). In the baseline simulation design with heterogeneous effects, τi=5(1+X1i+X2i+X3i)\tau_i=5(1+X1_i+X2_i+X3_i) with covariates uniform on (−1,1)(-1,1), and the population ATE is 55. The heterogeneous-effect scripts store, for each simulated data set, the sample average Sn=n−1∑iτiS_n=n^{-1}\sum_i\tau_i. They compute the RMSE of each estimator around SnS_n, and they compute coverage as the frequency with which the bootstrap percentile interval contains SnS_n. The sample average SnS_n is random. Scoring an estimator against it is legitimate, but the supplement does not say that it does so, and the definition in the article points to the population ATE. The RMSE around SnS_n and the RMSE around 55 are different quantities, and so are the two coverage probabilities.

Evidence a derivation

The evidence is a derivation and a check of the scoring lines in the simulation scripts. For XX uniform on (−1,1)(-1,1), EX=0\mathrm{E}X=0 and EX2=1/3\mathrm{E}X^2=1/3. By independence of the three covariates, I have Eτi=5\mathrm{E}\tau_i=5 and Var⁡(τi)=25(1/3+1/3+1/3)=25\operatorname{Var}(\tau_i)=25(1/3+1/3+1/3)=25, and Var⁡(Sn)=25/n\operatorname{Var}(S_n)=25/n. In the design “Model is Incorrect”, τi=5(1+X1i+X2i+X1iX2i+X3i)\tau_i=5(1+X1_i+X2_i+X1_iX2_i+X3_i), and Var⁡(Sn)=250/(9n)\operatorname{Var}(S_n)=250/(9n). Let RSR_S and RθR_\theta denote the RMSE of an estimator θ^\hat\theta around SnS_n and around 55. Then

Rθ2−RS2=Var⁡(Sn)+2 E{(θ^−Sn)(Sn−5)},R_\theta^2-R_S^2=\operatorname{Var}(S_n)+2\,\mathrm{E}\{(\hat\theta-S_n)(S_n-5)\},

and the cross term need not vanish. Adding 25/n25/n to a printed squared RMSE is not a valid conversion. In the scripts, the first stored quantity is mean(dgpout[[3]]), the RMSE lines subtract it from each estimate, and the coverage lines in sims-BS.R test whether it lies between the percentile endpoints. Applied to test inputs, the scripts' scoring lines and the same lines with 55 as the target give different RMSEs and different coverage.

The saved simulation output is not in the replication files, and the tables were not recomputed against 55. No printed entry, no ranking of the estimators and no statement about coverage is shown to change.

What survives

The heterogeneous-effect scoring formulas are valid for the sample average effect. Since SnS_n converges to 55, an estimator is consistent for one target if and only if it is consistent for the other. The identification argument, the estimator and the applications are unaffected. What changes is the reading of Tables A1, A2, A4 and A5, the heterogeneous panels of Figure A3, Figures A1 and A2, Table A6, and footnotes 6 and 7, which cite them. This evidence concerns the sample average effect. This finding does not cover Table A3, the design with a constant effect.

Corrected

Supplement, added to p. A5 and p. A9: In the heterogeneous-effect simulations, RMSE is computed relative to the realized sample average treatment effect, Sn=n−1∑iτiS_n=n^{-1}\sum_i\tau_i. Table A6 likewise reports bootstrap interval coverage of SnS_n. These summaries do not report RMSE or coverage for the population ATE, E τi=5\mathrm{E}\,\tau_i=5.

Article, footnote 7: In our own simulation studies, assuming α=0.275\alpha = 0.275, we find that the bootstrap has conservative coverage of the realized sample average treatment effect for ICSW in small samples, and proper coverage in larger samples.

Crop of footnote 7, five lines. Red boxes mark the sentence 'In our own simulation studies, assuming alpha = 0.275, we find that the bootstrap has conservative coverage for ICSW in small samples, and proper coverage in larger samples.'
Footnote 7, p. 498 (PDF p. 7). The marked sentence summarizes the bootstrap coverage found in the supplement without naming the quantity covered.
Crop of the section heading and six lines of text. A red box marks 'We present tables of RMSE vs' at the end of the first line, which continues 'n in 500, 1000, 2500, 5000, 10000, 25000 under the 5 different models, including misspecification.'
Supplement, p. A5 (PDF p. 5), section “(2) Properties of ICSW with and without Misspecification”. The marked words introduce the RMSE tables; the target of the RMSE is not stated.
Crop of the section heading, three lines of text and Table A6. A red box marks 'In Table A6, we present percent coverage with the bootstrap'. Table A6 has rows n = 500, 2500 and 5000, with 90 percent interval coverage 0.94, 0.91 and 0.92 and 95 percent interval coverage 0.98, 0.94 and 0.95.
Supplement, p. A9 (PDF p. 9), section “(3) Bootstrap Confidence Interval Coverage”, with Table A6. The marked words introduce the coverage; the quantity covered is not stated.

The replication code for the two-sided compliance score confines each standardized coefficient to [−5,5][-5,5], stops after 10,000 evaluations and uses the result without a convergence check, so it can return a fit that is not the ML estimate defined on p. 497.

Section 3.1, p. 497 (PDF p. 6), display that defines the ML estimate; replication files, icsw.R, function complier.score, lines 15, 81 and 86–89, called at lines 37 and 92 of albertsonlawrence.R

As printed

Article, p. 497: Thus the ML estimate of {θA,C,θA∣A,C}\{\theta_{\mathbf{A,C}}, \theta_{\mathbf{A|A,C}}\},

{θ^A,C,θ^A∣A,C}=argmax⁡θA,C, θA∣A,C[L(θA,C,θA∣A,C∣D,Z)].\{\hat\theta_{\mathbf{A,C}}, \hat\theta_{\mathbf{A|A,C}}\} = \operatorname*{argmax}_{\theta_{\mathbf{A,C}},\,\theta_{\mathbf{A|A,C}}}\bigl[L(\theta_{\mathbf{A,C}}, \theta_{\mathbf{A|A,C}}|D,Z)\bigr].

Replication files, icsw.R, line 15 (one line in the file, shown here in two pieces):

complier.score <- function(D, Z, W, weights = NULL, link = pnorm,

genoud = TRUE, num.iter = ifelse(genoud, 200, 1e4),onesided=FALSE) {

Line 81 (also in two pieces):

opt.log <- bobyqa(starting, complier.neg.logLik.fnc,control=list(iprint=0,

maxfun=num.iter),lower=-5,upper=5)

Line 86, the first of the four lines that compute and return the scores (in two pieces):

CA.score <- (pnorm(rowSums(t(t(W1) *

opt.log$par[1:(length(opt.log$par)/2)]))))

The problem

The article defines the estimate as the maximizer of the likelihood over all coefficient vectors. The helper complier.score, called with genoud=FALSE, standardizes the covariates and searches with BOBYQA inside the box [−5,5][-5,5] for every coefficient. It stops after at most 10,000 evaluations of the objective and computes the scores from the last iterate without testing the status code that the optimizer returns. The box can exclude the unrestricted maximizer, and stopping at the cap does not establish convergence. The script nevertheless uses the scores without checking the returned status. The Albertson and Lawrence script calls this branch for the point estimate (line 37) and in every bootstrap replicate (line 92), so that the ICSW columns of Table 2 depend on these fits.

Evidence an exact counterexample

The evidence is an exact counterexample, a run of the code on it, and a run of the code on four bootstrap draws of the application.

The counterexample has 96 observations and two covariates, W1=UW_1=U and W2=U+V/50W_2=U+V/50 with U,V∈{−1,1}U,V\in\{-1,1\}. Each of the eight cells (U,V,Z)(U,V,Z) holds twelve observations. For either value of UU, the number of observations with D=1D=1 is 11 for (V,Z)=(−1,0)(V,Z)=(-1,0), 22 for (−1,1)(-1,1), 55 for (1,0)(1,0) and 1010 for (1,1)(1,1). Let tt be such that Φ(t)=5/6\Phi(t)=5/6. The model of equations (5)–(7) holds with coefficients θA,C=(0,−50t,50t)\theta_{\mathbf{A,C}}=(0,-50t,50t) and θA∣A,C=(0,0,0)\theta_{\mathbf{A|A,C}}=(0,0,0) on the covariates (1,W1,W2)(1,W_1,W_2). Since these coefficients fit the proportion treated in every cell, they attain the maximum of the likelihood. In the standardized coordinates that the code uses, the third coefficient exceeds 100/3100/3. Let MM be the maximum of the likelihood and BB an upper bound for the likelihood over the box, namely

M=(56)40(16)8B0,B=(1120)40(920)8B0,B0=(512)10(712)14(112)2(1112)22.M=\Bigl(\frac56\Bigr)^{40}\Bigl(\frac16\Bigr)^{8}B_0,\qquad B=\Bigl(\frac{11}{20}\Bigr)^{40}\Bigl(\frac{9}{20}\Bigr)^{8}B_0,\qquad B_0=\Bigl(\frac{5}{12}\Bigr)^{10}\Bigl(\frac{7}{12}\Bigr)^{14}\Bigl(\frac{1}{12}\Bigr)^{2}\Bigl(\frac{11}{12}\Bigr)^{22}.

The bound BB holds for the printed likelihood and for the likelihood with the probabilities clipped as in the code. Comparing the two in exact rational arithmetic, I find that M>5000BM>5000B. Run on these data, the unchanged helper returns status 00 after 253 evaluations, with two coefficients at the bound 55 and negative log-likelihood 56.37856.378. The value at the maximizer is 44.81244.812.

The run on the application uses the script’s seed and R 4.4.2 with minqa 1.2.8. The fit on the full data ends with status 00 after 1,096 evaluations. Of the first four bootstrap draws, the fourth stops at the cap (status 11, 10,000 evaluations), and its scores are finite and are used. Continuing that search for 37,259 more evaluations lowers the negative log-likelihood from 197.12769197.12769 to 197.12729197.12729. A separate recalculation of the full bootstrap reported 39 capped fits among the 2,500 draws. I have not confirmed that count.

What survives

The display on p. 497 is correct and stands as printed. No value in Table 2 is shown to change. A recalculation of the bootstrap that kept the capped fits, as the authors' code does, agreed with the printed intervals within Monte Carlo error. The intervals have not been recomputed with converged fits, and the software versions used for the article are not known. The 2SLS columns do not use the scores, and the consistency result is conditional on consistent scores. The finding concerns only the branch genoud=FALSE of the two-sided fit in icsw.R.

Corrected

The definition on p. 497 is unchanged. In icsw.R and its callers, I would test the optimizer status before using the scores, and continue or report failed fits. I would remove or expand coefficient bounds that exclude better fits. A status of 00 does not by itself establish a global maximum; I would describe the scores as numerical approximations to the ML scores unless their accuracy is established.

Crop of the sentence 'Thus the ML estimate of theta A,C and theta A given A,C' the display that follows and one line after it. A red box marks the display, in which the pair of estimates equals the argmax over both coefficient vectors of the likelihood L given D and Z.
Section 3.1, p. 497 (PDF p. 6). The marked display defines the ML estimate as the maximizer of the likelihood over all coefficient vectors. The display is correct as printed.
Listing of seven lines of R code from the function complier.score, three of them continued on a second row. Red boxes mark '1e4' in the default 'num.iter = ifelse(genoud, 200, 1e4)' on line 15, and 'maxfun=num.iter' and 'lower=-5,upper=5' in the call to bobyqa on line 81. Lines 86 to 89 compute CA.score, A.score and C.score from the returned parameters and return them.
Replication files, icsw.R, lines 15, 80–81 and 86–89, typeset from the source file with line numbers added; long lines are continued after an arrow. The marked terms set the cap of 10,000 evaluations and the bounds ±5\pm5.

Clarity notes

Imprecise or ambiguous wording whose intended meaning is correct; not an error.

The consistency conditions of Section 2.1 are stated for DiD_i and ZiZ_i themselves, although equations (1) and (2) contain covariates; the conditions meant are the usual ones for DiD_i and ZiZ_i after the covariates are partialled out.

Section 2.1, p. 494 (PDF p. 3), first sentence of the page and the two sentences after “two main assumptions for consistency”; equation (1) is on p. 493 (PDF p. 2)

As printed

We are able to consistently estimate β1\beta_1 using ordinary least squares (OLS) regression when DiD_i is asymptotically uncorrelated with the error term (1nΣDiui→p0\colorbox{#ffe58a}{$\frac{1}{n}\Sigma D_i u_i \to_p 0$}). This requirement is violated, however, when DiD_i is endogenous. […] Given the model in equations (1) and (2), the SEM IV framework relies on two main assumptions for consistency. First, the correlation between ZiZ_i and uiu_i must approach zero as nn grows. Second, the covariance between ZiZ_i and DiD_i must converge to a nonzero quantity as nn grows.

Intended meaning

The passage is a short account of OLS and 2SLS with one endogenous treatment. It takes for granted that the covariates Q1i,…,QKiQ_{1i},\dots,Q_{Ki} are valid controls, which serve as their own instruments, and that the instrument is related to the treatment when the covariates are held fixed. With these conventions the account is correct, and the phrase “two main assumptions” does not claim to be a complete list.

The printed conditions do not suffice if they are read literally, as conditions on DiD_i and ZiZ_i alone. Let did_i and ziz_i be the residuals from the linear projections of DiD_i and ZiZ_i on an intercept and the covariates. For independent, identically distributed sampling with finite second moments, and a nonsingular second-moment matrix of the intercept and the covariates, the two estimators have the limits

plim⁡β^1OLS=β1+E(diui)E(di2),plim⁡β^1IV=β1+E(ziui)E(ziDi),\operatorname{plim}\widehat\beta_{1}^{\mathrm{OLS}} = \beta_1 + \frac{E(d_iu_i)}{E(d_i^2)}, \qquad \operatorname{plim}\widehat\beta_{1}^{\mathrm{IV}} = \beta_1 + \frac{E(z_iu_i)}{E(z_iD_i)},

when the denominators are not zero. An example shows the difference. Let ZZ equal 00 or 11 with probability 1/21/2 each, let UU and VV equal −1-1 or 11 with probability 1/21/2 each, and let ZZ, UU and VV be independent. Set D=ZD = Z, Q=D+U/2+V/4Q = D + U/2 + V/4, u=Uu = U and Y=2+D+UY = 2 + D + U. Equations (1) and (2) hold with β1=1\beta_1 = 1 and λ1=0\lambda_1 = 0. I have E(Du)=0E(Du) = 0, Cov⁡(Z,u)=0\operatorname{Cov}(Z,u) = 0 and Cov⁡(Z,D)=1/4\operatorname{Cov}(Z,D) = 1/4, and the moment matrices of the regressors and of the instruments are nonsingular, with determinant 5/645/64. The covariate is correlated with the error, since E(Qu)=1/2E(Qu) = 1/2. A direct computation gives E(zu)=−2/9E(zu) = -2/9 and E(zD)=5/36E(zD) = 5/36. Substituting these values into the two limits, I find that both OLS and 2SLS converge to 1+(−2/9)/(5/36)=−3/51 + (-2/9)/(5/36) = -3/5. In a second example the covariate is exogenous and Cov⁡(Z,D)=1/16\operatorname{Cov}(Z,D) = 1/16, but E(zD)=0E(zD) = 0, and β1\beta_1 is not identified.

No later proof or calculation in the paper is shown to rely on the literal reading. From Section 2.2 on, the instrument is randomized and independent of the pretreatment covariates, and the argument for the LATE and for ICSW does not use the conditions of Section 2.1.

Corrected

Suppose that the observations are independent and identically distributed, that the second moments are finite and that the second-moment matrix of the intercept and the covariates is nonsingular. Let did_i and ziz_i be the residuals from the linear projections of DiD_i and ZiZ_i on an intercept and Q1i,…,QKiQ_{1i},\dots,Q_{Ki}. OLS is consistent for β1\beta_1 when E(diui)=0E(d_iu_i) = 0 and E(di2)>0E(d_i^2) > 0. 2SLS with the single instrument ZiZ_i is consistent for β1\beta_1 when E(ziui)=0E(z_iu_i) = 0 and E(ziDi)≠0E(z_iD_i) \neq 0. For other sampling schemes, require stable probability limits for the sample moments and a nonsingular limiting controls matrix, with the same zero-moment and nonzero-denominator conditions using sample projection residuals. Orthogonality of uiu_i to every regressor, or to every instrument, together with the usual rank condition, is a stronger sufficient condition.

Crop of equation (2) and the eight lines below it. Equation (2) reads D sub i = gamma 0 + gamma 1 Z sub i + delta 1 Q sub 1i + delta 2 Q sub 2i + ... + delta K Q sub Ki + e sub i. The text explains two-stage least squares and ends: Given the model in equations (1) and (2), the SEM IV framework relies on two main assumptions for consistency. First, the correlation between Z sub i and u sub i must approach zero as n grows. Second, the covariance between Z sub i and D sub i must converge to a nonzero quantity as n grows. Red outlines surround the two sentences that begin First and Second.
Section 2.1, p. 494 (PDF p. 3). The outlines mark the two conditions for consistency of 2SLS. Equation (2), at the top of the crop, contains the covariates Q1i,…,QKiQ_{1i},\dots,Q_{Ki}.
Crop of four lines. The first sentence reads: We are able to consistently estimate beta 1 using ordinary least squares (OLS) regression when D sub i is asymptotically uncorrelated with the error term (one over n times the sum of D sub i u sub i converges in probability to 0). The next sentence says that this requirement is violated when D sub i is endogenous. The crop ends partway through the sentence about finding an instrument, after Y sub i. A red outline surrounds the words asymptotically uncorrelated with the error term and the formula in parentheses.
Section 2.1, p. 494 (PDF p. 3), first lines of the page. The outline marks the condition for consistency of OLS.

Assumption 3 can fail although every covariate profile has compliers; the explanation that follows it describes only the case of a profile with no compliers.

Assumption 3 and the explanation that follows it, p. 500 (PDF p. 9)

As printed

For all x∈Supp⁡(X)\mathbf{x} \in \operatorname{Supp}(\mathbf{X}), Pr⁡(D1>D0∣X=x)∈(ϵ,1]\Pr(D_1 > D_0 \mid \mathbf{X} = \mathbf{x}) \in (\epsilon, 1], where ϵ>0\epsilon > 0. Note that IV estimation already requires the assumption of nonzero compliance in the full population, and we generalize this assumption to every covariate profile. This assumption has both theoretical and practical implications. If Assumption 3 fails, then the ATE cannot generally be estimated using the variation induced by ZZ: the instrument simply does not cause any variation in the treatment received by some types of units. In practice, if a unit has zero probability of compliance, the weighting procedure will produce infinite weights asymptotically, leading to an undefined estimate.

Intended meaning

The intended point is that a covariate profile without compliers cannot be reweighted: the instrument moves no unit’s treatment there, and the inverse of the compliance score is infinite. This is correct, and “cannot generally” does not claim an impossibility for every law that violates the assumption. Read literally, however, the sentence equates the failure of Assumption 3 with the existence of such a profile. The assumption asks for more, namely a lower bound ϵ\epsilon that holds for all profiles at once, and the failure of a uniform bound need not imply a profile with zero compliance.

For an example, let Pr⁡(X=k)=2−k\Pr(X = k) = 2^{-k} for k=1,2,…k = 1, 2, \ldots, and let the compliance score at X=kX = k be 1/k1/k. Every profile has compliers, and no ϵ>0\epsilon > 0 bounds the scores from below. Suppose further that there are no always-takers or defiers, that ZZ is assigned independently as a fair coin, that the outcomes are bounded and that the treatment effect is 11 for every unit. The weights w=kw = k formed from the true scores satisfy E[w]=2\mathbb{E}[w] = 2 and E[w2]=6\mathbb{E}[w^2] = 6, and the weighted IV ratio is consistent for the ATE, which is 11. The example concerns the true scores and says nothing about estimated scores. Assumption 3 as stated, and every result that uses it, is unchanged.

Corrected

If compliance is zero for a set of covariate profiles with positive population probability, the ATE cannot generally be identified using the variation induced by ZZ. Assumption 3 is stronger than this pointwise overlap requirement: a uniformly positive lower bound also rules out arbitrarily large inverse compliance weights.

Crop of Assumption 3 with its eight lines of text. The assumption reads: For all x in Supp(X), Pr(D1 greater than D0 given X = x) lies in the interval from epsilon, excluded, to 1, included, where epsilon is greater than 0. The text continues: Note that IV estimation already requires the assumption of nonzero compliance in the full population, and we generalize this assumption to every covariate profile. This assumption has both theoretical and practical implications. If Assumption 3 fails, then the ATE cannot generally be estimated using the variation induced by Z: the instrument simply does not cause any variation in the treatment received by some types of units. In practice, if a unit has zero probability of compliance, the weighting procedure will produce infinite weights asymptotically, leading to an undefined estimate. Conditional on the validity of Assumptions 1 and 2, Assumption 3 is therefore verifiable in large samples. Red outlines surround the sentence from If Assumption 3 fails to some types of units, which runs over three lines.
Assumption 3 and its explanation, p. 500 (PDF p. 9). The outlines mark the sentence that describes the failure of the assumption.

The compliance groups of Figures A1 and A2 are formed from the realized fraction of compliers in each simulated dataset, Cˉ=N−1∑iCi\bar C = N^{-1}\sum_i C_i, which the supplement labels mean⁡(PC)\operatorname{mean}(P_C).

Supplement, p. A3 (supplement PDF p. 3), sentence above Figure A1

As printed

We subclassify each of these plots to DGPs with low (mean⁡(PC)<0.33\colorbox{#ffe58a}{$\operatorname{mean}(P_C)$} < 0.33), medium (0.33≤mean⁡(PC)<0.500.33 \le \colorbox{#ffe58a}{$\operatorname{mean}(P_C)$} < 0.50) and high (0.50≤mean⁡(PC)0.50 \le \colorbox{#ffe58a}{$\operatorname{mean}(P_C)$}) compliance.

Intended meaning

The sentence sorts the simulated datasets by their overall compliance, with the cut-offs 0.330.33 and 0.500.50, and the code does the same. In the article, PCiP_{Ci} is the probability that unit ii is a complier given its covariates (p. 496), so the label suggests an average of probabilities. The code stores, for each dataset, the mean of the indicators Ci=ACi−AiC_i = AC_i - A_i that unit ii is a complier, and the display script sorts on this mean. The realized fraction and the mean probability have the same conditional expectation, but they differ in a finite dataset. In one simulated dataset of 100100 units, with all covariates and coefficients equal to zero, the realized fraction is 0.530.53 and the probability is Φ(2/3)(1−Φ(−2/3))=0.5588\Phi(2/3)(1-\Phi(-2/3)) = 0.5588. The thresholds and the meaning of low, medium and high compliance are as printed, and no figure changes.

Corrected

We subclassify each of these plots to DGPs with low (Cˉ<0.33\colorbox{#ffe58a}{$\bar C$} < 0.33), medium (0.33≤Cˉ<0.500.33 \le \colorbox{#ffe58a}{$\bar C$} < 0.50) and high (0.50≤Cˉ0.50 \le \colorbox{#ffe58a}{$\bar C$}) compliance, where Cˉ=N−1∑iCi\bar C = N^{-1}\sum_i C_i is the realized fraction of compliers in each simulated dataset.

Crop of two lines. The text reads: A1 and A2. We subclassify each of these plots to DGPs with low (mean(P sub C) < 0.33), medium (0.33 is at most mean(P sub C) < 0.50) and high (0.50 is at most mean(P sub C)) compliance. Red outlines surround the three occurrences of mean(P sub C).
Supplement, p. A3 (supplement PDF p. 3), the two lines above Figure A1. The outlines mark the three occurrences of the label mean⁡(PC)\operatorname{mean}(P_C).

Formal statements (Lean)

Formal results: 17 whole; 5 restricted; 4 refuted; 0 omitted.

Coverage concerns the identified claim under its stated assumptions. A checked proof establishes the formal statement; correspondence to the published text was reviewed by AI. Repository access is currently required to open the source links.

Each result links to its current formal statement and proof. “Whole” covers the identified claim under its assumptions; “restricted” covers the stated narrower scope; “refuted” gives a counterexample to the identified assertion; “omitted” has no current theorem. Any named published theorem used as a premise remains an assumption. Browse the formal results for this work.

Section 3.4, the sentence ‘Assumption 1 is less restrictive than the typical ignorability assumption’, set against Assumption 1 and the two displayed ignorability equalities; the witness is the validator's and is not in the paper

p. 499 (PDF p. 8), paragraph after Assumption 1 beginning ‘Ignorability assumptions, while strong’

Section 3.4, p. 499 (PDF p. 8), paragraph after Assumption 1. The six-atom law satisfies the IV and received-treatment ignorability conditions, including potential-pair independence, but fails latent ignorability and has the stated ATE and Wald values.

Historical component credits

Section 3.4, comparison of Assumption 1 with the typical ignorability assumption, corrected: the two conditions are not ordered in general; the typical condition implies Assumption 1 under one-sided noncompliance

p. 499 (PDF p. 8), paragraph after Assumption 1

Section 3.4, p. 499 (PDF p. 8), comparison following Assumption 1. The two ignorability conditions are incomparable, as witnessed by the stated six-atom laws with type masses (1/2, 1/4, 1/4). The baseline-selection law has constant effect one and received-treatment means (1/2, 0) and (3/2, 1). Under randomized one-sided noncompliance, received-treatment ignorability implies latent ignorability at every positive-mass profile with positive compliance. The usual measure-theoretic conditions are assumed.

Historical component credits

Section 3.4, the sentence ‘More generally, even if treatment effects are heterogeneous, but still statistically independent of compliance, Assumption 1 would necessarily be satisfied, and both 2SLS and ICSW would be consistent for the ATE’; the two witnesses are the validator's and are not in the paper

p. 499 (PDF p. 8), last sentence of the paragraph beginning ‘We can describe sufficient conditions for and consequences of Assumption 1’

Section 3.4, p. 499 (PDF p. 8), last sentence of the sufficiency paragraph. The explicit binary-covariate laws refute the marginal and conditional independence readings, with the stated nested probabilities, arm differences, ATEs, LATEs and population Wald ratios.

Historical component credits

Section 3.4, the sufficiency sentence for heterogeneous effects, corrected: the mean-independence condition each estimator needs, with the exact relation between the two limits

p. 499 (PDF p. 8), last sentence of the paragraph beginning ‘We can describe sufficient conditions’; the limits are the displays on p. 499 and Appendix equations (10) and (13), p. 504 (PDF p. 13)

Section 3.4, p. 499 (PDF p. 8), corrected heterogeneous-effect sufficiency statement; Appendix equations (10) and (13), p. 504 (PDF p. 13). For finite positive-mass covariate support, the weighted Wald identities, LATE and true-score ICSW formulas, covariance gap, separate ATE sufficiency implications, both separation laws, and both joint-independence sufficient conditions hold. For iid sampling and measurable profile-based consistent score estimates, ordinary IV and estimated-score ICSW converge in probability to the respective population ratios. The respective mean-equality premises give ATE consistency; imposing both gives consistency of both estimators. The usual measure-theoretic conditions are assumed.

Historical component credits
  • Counterexample proved AronowAudit.P029.conditional_independence_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This counterexample_proved row is a separation witness for the corrected target T4(vi), not a proof of its positive sufficiency implications; those are covered by R189-10 and R189-11.
  • Proved AronowAudit.P029.Wave1.corrected_weighted_limits_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The target itself explicitly fixes finite covariate support and integrable potential outcomes. Null profile labels can be removed or assigned arbitrary positive weight extensions; replacing pointwise no-defiers by almost-sure no-defiers leaves every moment unchanged. The statement proves population identities, which are the specified component.
  • Proved AronowAudit.P029.Wave1.joint_independence_suffices_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The target itself requires finite covariate support and explicitly proposes these two joint-independence alternatives. Boolean decide(complier) represents exactly the binary C; no additional independence claim is hidden.
  • Counterexample proved AronowAudit.P029.marginal_independence_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This counterexample_proved row is a separation witness for the corrected target T4(vi), not a proof of its positive sufficiency implications; those are covered by R189-10 and R189-11.
  • Proved AronowAudit.P029.profile_covariance_gap Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; The transported corrected_weighted_limits_proof already supplies the population covariance-gap identity; this finite-profile helper is not separately invoked.
  • Proved AronowAudit.P029.profile_inverse_score_cancellation Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; The transported corrected_weighted_limits_proof already supplies inverse-score cancellation; this finite-profile helper is not separately invoked.

Supplement Table A3 ‘Noise (Homogeneous)’ and the ‘Noise (Homogeneous)’ panel of Figure A3, presented as RMSE for the design with τi=5\tau_i = 5; the scoring rule is in the deposited code and the witnesses are the adjudicating validator's

Supplement p. A2 (design), p. A5 (description of the RMSE tables), p. A6 (Figure A3), p. A7 (Table A3); deposited code ‘Simulations Misspecification Noise 6-20.R’, lines 14–27, 70–94 and 124–135, and ‘Sim-Display Misspec.R’, lines 59–70

supplement pp. A2, A5 and A7, homogeneous-noise design, Table A3 and Figure A3. Two admissible 500-record homogeneous-effect inputs have the stated nonsingular design, cross-products and recovered coefficients, and scoring about six instead of five reverses the squared-error discrepancy even though positive weights preserve exact recovery.

Historical component credits
  • Counterexample proved AronowAudit.P029.homogeneous_rmse_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Witness for a scoring-rule mismatch on admissible realized inputs, not a refutation of historical numerical entries. Assignment indexing in the opposite input is a realized error array, as in the target; it does not assert a population law with treatment affecting baseline outcomes. No change from the blind claim-stage verdict.
  • Proved AronowAudit.P029.normal_equation_exact_recovery Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected counterexample and positive-weight theorem already export coefficient recovery; the generic matrix helper is not separately invoked.
  • Proved AronowAudit.P029.rmse_design_determinant Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary determinant certificate for the fixed witness, not a standalone refutation.
  • Proved AronowAudit.P029.rmse_design_gram Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected homogeneous_rmse_counterexample already exports this exact Gram identity, which is used to transport the outcome cross-products.
  • Proved AronowAudit.P029.rmse_positive_weight_recovery Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Strictly positive weights are part of T5(d), not an extra assumption added by Lean; no fitted-score existence claim is required.
  • Proved AronowAudit.P029.rmse_witness_admissible Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This row covers the stated uniform-support, threshold and balance checks, not a theorem constructing an entire random simulation law.
  • Proved AronowAudit.P029.rmse_witness_outcomes Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The row accurately limits itself to the two matrix-vector identities and benchmark; the effect and observed-outcome identities are in R190-01.

Supplement Table A3 and the ‘Noise (Homogeneous)’ panel of Figure A3, corrected scoring rule: RMSE about the constant effect 5, with the exact relation to the coded statistic

Supplement pp. A2, A5–A7; deposited code ‘Simulations Misspecification Noise 6-20.R’, lines 124–135, and ‘Sim-Display Misspec.R’, lines 59–70

supplement pp. A2, A5 and A7, corrected homogeneous-noise scoring rule. The effect is five, the corrected RMSE uses five for every positive replicate count, the coded benchmark has the stated uniform moments, and the finite and population error-discrepancy identities permit both signs on the admissible inputs.

Historical component credits
  • Proved AronowAudit.P029.expected_mse_discrepancy Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The L2 hypotheses are exactly the finite-second-moment assumptions in T6(d); they ensure all squared-error and covariance integrals have their usual meanings.
  • Proved AronowAudit.P029.finite_expected_mse_discrepancy Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is a finite auxiliary lemma, not a claim to replace the general probability-space version in R190-09.
  • Proved AronowAudit.P029.finite_run_mse_discrepancy Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The empty-run extension is harmless; positive-length runs have the ordinary empirical-average meaning. No change from the claim-stage verdict.
  • Proved AronowAudit.P029.Wave1.heterogeneous_benchmark_moments_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Measurability is intrinsic to the claimed random-variable laws, and full independence of all 3n coordinates is explicitly in T6(c). No conditional published premise is used.
  • Proved AronowAudit.P029.homogeneous_population_effect Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Specializing C to the measurable complier event yields exactly the asserted LATE; the integral is of the constant treatment difference, not of the separate potentially nonintegrable outcomes. No change from the claim-stage verdict.
  • Proved AronowAudit.P029.homogeneous_rmse_rule Limited support; baseline fidelity: narrower; manifest: not used; not used in Solution; This narrower row states only R=2500; the all-positive-R formula instead follows by unfolding the existing parametric rmse definition, copied under its SAP name.
  • Proved AronowAudit.P029.homogeneous_sample_effect Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The nonempty-sample assumption is explicitly in T6(a) and prevents the empty-average convention.
  • Proved AronowAudit.P029.squared_error_discrepancy Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary exact algebra retaining the cross term; no independence or asymptotic approximation is assumed.
  • Proved AronowAudit.P029.uniform_covariate_moments Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The actual continuous uniform law is used; no assumed moment specification or finite-support substitution appears. No change from the claim-stage verdict.

R package icsw 1.0.0 (the published implementation of the method): the compliance-score objective with observation weights and the reweighting bootstrap, set against Section 3.3's ‘bootstrapping the entire process’ and the package help; the witnesses are the validators'

Not in the article PDF. Staged package source: packages/icsw/icsw/R/compliance.score.R, lines 43–48 and 81–92; R/icsw.tsls.R, lines 28–37; man/icsw.tsls.Rd, lines 9–13 and 42–43. The recommendation it implements: p. 498 (PDF p. 7), Section 3.3

package icsw 1.0.0, compliance.score.R lines 43–48 and 81–92, icsw.tsls.R lines 28–37 and package help; article Section 3.3, p. 498 (PDF p. 7). Positive weights inside the logarithm leave all minimizers unchanged; a zero weight gives infinite extended objective. In the sixteen-record one-sided witness, the actual unweighted and weighted probit fits have scores (1/4,3/4) and (3/4,27/28), giving Wald estimates 3/2 and 17/10. Every ordered raw two-sided receipt array has finite nested-probit coefficients; the stated two-sided arrays are the actual clipped package and corrected fits, with the stated score differences and likelihood ratio. Under the positive weight perturbation, actual refitting yields scores (1/(3+t),3/4) and the stated Wald curve with derivative -1/16 at one, while the fixed-score estimate has derivative zero. These facts refute the package help's interpretation that the supplied weights refit the compliance model. The usual measure-theoretic conditions are assumed.

Historical component credits
  • Proved AronowAudit.P029.bootstrap_first_step_derivative Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This row is only the derivative component, not a standalone assertion about refitting or a full refutation; the earlier review's generic witness caveat is unnecessary for this exact component.
  • Proved AronowAudit.P029.bootstrap_refit_ratio Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:2847 evaluates the Wald estimator using bootstrapRefitScores, whose definition at P029/P029_Audit.lean:2842 merely prescribes 1/(3+t) and 3/4. T7(v) and the row assert that refitting the weighted compliance model produces these scores and hence the displayed estimator, but the declaration contains no fitting or likelihood-optimality conclusion. The missing link is substantive: an identity after substituting chosen scores does not establish the effect of refitting, even though the intended scores are correct and the separate general cell-minimum theorem at P029/P029_Audit.lean:816 can establish it after specialization.', 'suggested_fix': 'Add a conclusion that bootstrapRefitScores t uniquely minimizes the observation-weighted negative Bernoulli log likelihood on the sampleZ = true records over both interior W-cell probabilities. Specialize saturated_weighted_objective_minimum with success/failure totals (1, 2+t) and (3, 1), then retain the existing Wald identities.', 'second_review': 'T7(v) requires the scores and Wald ratio obtained by refitting the compliance model to the perturbed observation weights. P029/P029_Audit.lean:2847 evaluates the ratio at bootstrapRefitScores, but P029/P029_Audit.lean:2842 defines that array by the prescribed numbers 1/(3+t) and 3/4 without an optimization property. Thus the estimator arithmetic omits the substantive identification of these scores as the weighted fit. Append that identification for the actual Z = 1 records, using the cell totals (1, 2+t) and (3, 1) and the separate saturated optimum theorem at P029/P029_Audit.lean:816.', 'severity': 'major'}
  • Proved AronowAudit.P029.inside_log_minimizers Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Harmless genuine generalization; strict likelihood positivity follows from the clipping in the target.
  • Proved AronowAudit.P029.inside_log_weights_identity Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Harmless genuine generalization; the parameter-independent interpretation follows by applying this identity at each parameter.
  • Counterexample proved AronowAudit.P029.two_sided_weight_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': "P029/P029_Audit.lean:947 establishes the component identified by this row: both specified cell arrays are interior and ordered, their unweighted likelihood ratio is 729/1500625 < 1, and the weighted compliance scores are 3/20 and 3/14. These are the corresponding arithmetic and feasibility assertions in T7(iv); requiring this component also to prove both global fitting optima would extend its stated role. Ordered receipt rates faithfully represent nested feasibility: for each x take a = q(x,true) and b = q(x,false)/a, as characterized by nested_receipt_parameters at P029/P029_Audit.lean:2241; intercept-plus-binary-W probit coefficients can represent any such interior a and b. The flag therefore identifies no missing conclusion within this comparison component, and the theorem's names are not themselves optimization assertions."}
  • Counterexample proved AronowAudit.P029.weighted_score_counterexample Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:925 states two rational identities for correctedScores and evaluates Wald at correctedScores and unweightedScores, which are prescribed constants at P029/P029_Audit.lean:919. T7(iii) and the row explicitly assert that these are the weighted and unweighted fitted scores, including the cell-maximizer property. No conclusion identifies either array as a likelihood optimum on the actual Z = 1 records, so the statement establishes estimator arithmetic without the claimed fitting counterexample. The separate saturated_weighted_objective_minimum theorem at P029/P029_Audit.lean:816 supplies a possible repair but does not add that missing conclusion to this declaration.', 'suggested_fix': 'Append unique-minimum conclusions for the one-sided Bernoulli objective on sampleZ = true records: unweightedScores for unit weights and correctedScores for oneSidedWeight. Specialize the saturated objective theorem to the actual two W cells, with unit-weight totals (1,3), (3,1) and weighted totals (9,3), (27,1).', 'second_review': 'T7(iii) asserts that the unweighted and observation-weighted likelihood fits produce the specified scores and consequently different weighted IV estimates. P029/P029_Audit.lean:925 establishes the numerical Wald values and two rational score identities, while unweightedScores and correctedScores at P029/P029_Audit.lean:919 are simply prescribed constants. No conclusion identifies either array as a likelihood optimum on the treated-assignment records of oneSidedSample at P029/P029_Audit.lean:914. Specializing the saturated objective theorem at P029/P029_Audit.lean:816 to the actual weighted and unweighted cell totals would supply the missing fitting conclusion and repair the counterexample statement.', 'severity': 'major'}
  • Proved AronowAudit.P029.zero_weight_infinite_contribution Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The +∞ assertion is definition-level because the target itself specifies log 0=−∞; this is the intended convention and does not render a substantive positive-likelihood case empty.
  • Proved AronowAudit.P029.zero_weight_objective_infinite Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The definition only takes finite or +∞ values, so no −∞ summand can cancel the selected +∞. The record's observation about unused proof premises is not a reason to remove premises from the stated theorem.

R package icsw 1.0.0, compliance-score fit with observation weights, corrected: the weighted log likelihood −∑iwilog⁡Li(θ)-\sum_i w_i\log L_i(\theta) (also the corrected one-sided objective of the Dataverse helper, finding P029-E014)

Not in the article PDF. Staged package source: packages/icsw/icsw/R/compliance.score.R, lines 43–47 and 91; Dataverse helper code/dataverse-UCFNHX-v2.0-original/icsw.R, lines 49–50

package icsw 1.0.0 and deposited icsw.R helper, corrected observation weights; article Section 3.1, pp. 496–497 (PDF pp. 5–6). Observation weights outside the logarithm satisfy the zero/unit and replication rules, give unique free-cell fits, and preserve the stated constant-weight and numerical witness conclusions.

Historical component credits
  • Proved AronowAudit.P029.bootstrap_refit_ratio Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:2847 substitutes the explicitly defined bootstrapRefitScores into fWald and obtains (t+11)/(t+7), but does not assert any optimization property of those scores. T8(c), through its reference to T7(v), claims this is the estimate obtained under the corrected weighted fit, and the row explicitly describes refitting. Without identifying the prescribed array as that fit, the declaration omits a substantive part of its credited component, despite the correct numerical formula. The general cell-minimum result at P029/P029_Audit.lean:816 is separate from this statement.', 'suggested_fix': 'Append unique optimality of bootstrapRefitScores t for the corrected weighted Bernoulli objective restricted to the Z = 1 records, using the W-cell success/failure totals (1, 2+t) and (3, 1); keep the existing estimator identities.', 'second_review': 'T8(c), through T7(v), identifies (t+11)/(t+7) as the estimate under the corrected observation-weighted likelihood fit. P029/P029_Audit.lean:2847 instead evaluates fWald using bootstrapRefitScores, which P029/P029_Audit.lean:2842 defines explicitly without asserting minimization of the corrected weightedObjective at P029/P029_Audit.lean:649. Identifying these scores as the actual fit is a substantive missing conclusion of the credited component, even though the numerical ratio is correct. Append optimality for the Z = 1 sample with cell totals (1, 2+t) and (3, 1), specializing P029/P029_Audit.lean:816.', 'severity': 'major'}
  • Proved AronowAudit.P029.constant_weight_minimizers Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The constant-weight part of T8(d) is the component covered here; this row does not purport to supply its separate nonconstant-weight witnesses.
  • Proved AronowAudit.P029.corrected_objective_zero_unit Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Zero-likelihood and negative-input extensions rely on totalized log, but none is needed by the target, which assumes positive contributions and instructs omission of zero-weight records.
  • Proved AronowAudit.P029.integer_weight_replication Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Zero multiplicities use the standard natural-power convention L^0=1, including the empty replicated data set.
  • Proved AronowAudit.P029.saturated_weighted_objective_minimum Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The component is T8(b) for free cell probabilities, not a claim about a numerical optimizer or an arbitrary unsaturated probit model; the target itself includes the positive-cell-total restriction.
  • Counterexample proved AronowAudit.P029.two_sided_weight_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': "P029/P029_Audit.lean:947 proves the ordered-interior-rate and compliance-score component of T8(c), together with the likelihood comparison inherited from T7(iv), exactly as the row describes. The target's specified weighted rates are the constants in weightedCells at P029/P029_Audit.lean:940, and the statement gives their scores 3/20 and 3/14 without restricting any cases of this finite comparison. The optimization property is a distinct component, not an additional conclusion promised by this row's feasibility and likelihood-ratio description. Nor is there a material coefficient-realization gap: q(x,true) and q(x,false)/q(x,true) are valid nested parameters by P029/P029_Audit.lean:2241 and can be represented with a probit intercept and binary W."}
  • Proved AronowAudit.P029.weighted_cell_strict_concavity Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Strict positivity of S and F and the open parameter interval are explicitly part of the target, so no boundary cases are omitted.
  • Proved AronowAudit.P029.weighted_cell_unique_max Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the unique-maximizer component; strict concavity is deliberately a separate credited statement.
  • Counterexample proved AronowAudit.P029.weighted_score_counterexample Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P029/P029_Audit.lean:925 evaluates the weighted IV estimate at the prescribed correctedScores and verifies the fractions 9/(9+3) and 27/(27+1). T8(c) claims that the corrected objective fits those scores, and the row expressly calls them cell maximizers, but the theorem contains no likelihood inequality or minimizer conclusion tying them to that objective on the sample. Consequently its arithmetic does not establish the credited fitted-score assertion, even though the intended scores and 17/10 estimate are correct. The general optimization theorem at P029/P029_Audit.lean:816 must be specialized and its consequence included in this statement.', 'suggested_fix': "Append that correctedScores is the unique minimizer of the corrected weighted negative Bernoulli log likelihood on the Z = 1 records, with W-cell success/failure totals (9,3) and (27,1); include the unweighted fit's optimality if retaining the row's fitting contrast.", 'second_review': "T8(c) requires that the corrected objective's one-sided fit be (3/4, 27/28), as well as that the resulting weighted IV estimate be 17/10. P029/P029_Audit.lean:925 establishes the fractions and Wald values using the hard-coded score arrays at P029/P029_Audit.lean:919 and P029/P029_Audit.lean:920, but contains no minimizer conclusion for weightedObjective, defined at P029/P029_Audit.lean:649. The general unique-minimizer theorem at P029/P029_Audit.lean:816 permits a local repair by specializing it to the Z=1 records and identifying their actual cell totals as (9,3) and (27,1), with corresponding unweighted optimality if retaining the fitting contrast. The missing link to likelihood fitting is a substantive part of the target claim, so changing only the description would not repair the credited statement.", 'severity': 'major'}

Section 4.1, ‘We see that compliers are more likely to have voted previously, be members of a major party, and identify as white’, read off the correlations of supplement Table A7; the non-identification witness is the validator's and is not in the paper

p. 501 (PDF p. 10), last paragraph; supplement Table A7, p. A11

Section 4.1, p. 501 (PDF p. 10), and supplement Table A7. The two 18,933-record completions preserve the specified recorded labels, compatible raw codes and arbitrary outcomes, yet give opposite strict compliance/race associations in both directions.

Historical component credits
  • Proved AronowAudit.P029.canvass_assignment_supported Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1369 asserts positivity of exactly the probability of the recorded assignment vector under independent Bernoulli(e) assignment, for every 0 < e < 1. This supplies the assignment-support component of T9: both fixed completions can generate the recorded assignment with positive probability. Independence from fixed potential responses and demographics follows from that randomization construction; requiring empirical independence of the realized record list would impose a different condition.'}
  • Proved AronowAudit.P029.canvass_cell_conditional_mean Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1291 equates record-level and count-weighted cell-level conditional means for every cell-constant value function and cell event. The masses at P029_Audit.lean:1252 count actual records, so this is the compression needed for T9 rather than a change to a nine-person population. Totalizing an empty-event conditional mean to zero on both sides loses no positive-event case, and the race comparisons in T9 condition on nonempty groups.'}
  • Proved AronowAudit.P029.canvass_cell_expectation Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected conditional-mean compression theorem already incorporates the expectation compression; no separate invocation is needed.
  • Proved AronowAudit.P029.canvass_cell_probability Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected conditional-mean compression theorem already incorporates the probability compression; no separate invocation is needed.
  • Counterexample proved AronowAudit.P029.racial_completion_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1315 realizes 18,933 records, preserves the recorded Boolean race labels, receipts and arbitrary record-specific outcomes, and exports the two conditional-mean gaps 744791/29225158 > 0 and -131213/1761229 < 0. Its potential-unit definition at P029_Audit.lean:1303 makes compliance equal recorded contact, including zero compliance for controls, exactly as T9 prescribes. Raw-code retention and assignment support are companion components at P029_Audit.lean:1385 and P029_Audit.lean:1369; the Boolean representation does not exclude a completion required by the target.'}
  • Proved AronowAudit.P029.racial_raw_code_preservation Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1385 retains each raw code and requires both completed White indicators to agree with code = "W" whenever a code is present. The hR premise only links the supplied raw strings to the recorded Boolean labels used by T9; every compatible non-W code, including the Raleigh unknown codes, is permitted. Retaining the raw coordinate by definition is appropriate bookkeeping, and the additional classification conclusion establishes the substantive compatibility requirement.'}

Section 4.1, description of the compliers, corrected (formal content of the narrowed claim): what the recorded race labels do and do not determine

p. 501 (PDF p. 10), last paragraph; supplement Table A7, p. A11

Section 4.1, p. 501 (PDF p. 10), corrected interpretation of supplement Table A7. Finite binary counts determine both association directions, the two Raleigh rational rate comparisons are negative, compatible completions have opposite signs, and covariance decomposes into the stated within-group and between-group terms.

Historical component credits
  • Proved AronowAudit.P029.binary_count_sign_identity Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected binary_population_count_sign gives the formulas for actual finite binary variables and counts; this abstract count-algebra helper is not separately invoked.
  • Proved AronowAudit.P029.binary_population_count_sign Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Fidelity sweep: ok
  • Proved AronowAudit.P029.canvass_assignment_supported Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1369 gives a strictly positive independent Bernoulli probability to the recorded assignment vector for every assignment probability in (0,1). This is also the support fact needed for the two completions in T10(c), whose claim explicitly reuses the completions of T9. The T9 label identifies that shared construction; absence of a separately named independence predicate does not change the product randomization or remove a required case.'}
  • Proved AronowAudit.P029.Wave1.grouped_covariance_decomposition_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Acceptable generalization. The assigned component is T10(d), not empirical fitted-score signs. The record’s reference to a separate held-metadata typo about assignment does not describe this row’s original text, which is accurate.
  • Counterexample proved AronowAudit.P029.racial_completion_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1315 supplies precisely the two completions invoked in T10(c), preserving observed races, receipts and outcomes while producing the count numerators 1489582 and -1049704. The underlying population contains all 18,933 individual records, and its outcome vector is unrestricted. Boolean race labels, with raw-code preservation handled at P029_Audit.lean:1385, change representation without weakening the claimed non-identification result.'}
  • Proved AronowAudit.P029.raleigh_recorded_comparison Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Fidelity sweep: ok

Section 2.2, ‘these assumptions are considerably weaker than (and subsumed by) those of the SEM framework’; the witnesses are the validator's and are not in the paper

p. 495 (PDF p. 4), last sentence of the paragraph listing the five IV assumptions; the SEM conditions are on pp. 493–494 (PDF pp. 2–3)

Section 2.2, p. 495 (PDF p. 4), comparison with the Section 2.1 SEM framework. Both explicit constant-effect SEM laws have the stated exogeneity, relevance and endogeneity moments, while one has defiers and the other lacks instrument/potential-outcome independence.

Historical component credits
  • Counterexample proved AronowAudit.P029.sem_defier_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The target explicitly requests population SEM conditions and a finite-law witness. Positive variances make the correlation meaningful; taking iid samples of this bounded law realizes the sample moment limits, without needing that separate convergence theorem in this witness row.
  • Counterexample proved AronowAudit.P029.sem_dependence_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The target explicitly requests population SEM conditions and a finite-law witness. The conditional means and positive variances make the exogeneity claim nondegenerate; independence fails because |Y0| determines Z.

Section 2.2, comparison of the five IV assumptions with the SEM assumptions, corrected: the two sets are distinct and non-nested; the potential-outcome assumptions permit heterogeneous effects

p. 495 (PDF p. 4), last sentence of the paragraph listing the five IV assumptions

Section 2.2, p. 495 (PDF p. 4), corrected comparison with Section 2.1. The SEM witnesses and the heterogeneous all-complier IV law establish non-nesting, while the SEM potential-outcome equations force a constant effect and the IV witness has ATE and LATE 3/2.

Historical component credits
  • Proved AronowAudit.P029.heterogeneous_iv_counterexample Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': "P029/P029_Audit.lean:1733 states FIV, heterogeneous effects, ATE = 3/2 and Wald = 3/2, but does not include the LATE = 3/2 equality in T12(c) and the row description. This is a literal missing conclusion, but its mathematical extent is trivial: heterogeneousUnit at P029/P029_Audit.lean:1727 makes every unit a complier, and FIV supplies a normalized probability law. Thus fCE at P029/P029_Audit.lean:70 reduces that LATE to the already stated fE divided by 1, with positive conditioning mass. The heterogeneity witness is complete, so the reviewer's major assessment overstates this omission.", 'suggested_fix': "Append fCE (randomized (fun _ : Bool => 1/2) (1/2)) (fun i => (heterogeneousUnit i.1).tau) (fun i => (heterogeneousUnit i.1).complier) = 3/2 to the theorem's conjunction; the conditioning event has probability 1.", 'second_review': 'T12(c) explicitly includes E(tau | D1 > D0) = 3/2, whereas P029/P029_Audit.lean:1733 exports the IV assumptions, heterogeneous effects, ATE and Wald values without this conditional-mean conclusion. Every heterogeneousUnit is a complier by P029/P029_Audit.lean:1727, and fCE at P029/P029_Audit.lean:70 therefore reduces to the exported expectation divided by total probability one. Appending the LATE equality is a trivial statement repair; the heterogeneity witness itself is complete.', 'severity': 'minor'}
  • Counterexample proved AronowAudit.P029.sem_defier_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the witness component T12(a) specifies, not a claim to prove every corrected comparison alone. The zero correlation has two strictly positive variances; the target states population moment conditions, so no additional sampling-limit theorem is needed.
  • Counterexample proved AronowAudit.P029.sem_dependence_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The counterexample violates random assignment, not mean exogeneity. The positive variances prevent totalized correlation from supplying a false zero. This is a witness for the relevant corrected-statement component; its status counterexample_proved is consistent with that purpose.
  • Proved AronowAudit.P029.sem_forces_constant_effect Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The common-error and common-covariate hypotheses are exactly the causal potential-outcome reading explicitly required by T12(b), not an inference from an arbitrary observational linear projection.

Assumption 2, last sentence: ‘In the case of a saturated model and F(.)F(.) as the normal CDF, P^Ci\hat{P}_{Ci} will be unbiased for the conditional probability of compliance’; the witness is the validator's and is not in the paper

p. 500 (PDF p. 9), Assumption 2; the estimator is defined on pp. 496–497 (PDF pp. 5–6)

Assumption 2, p. 500 (PDF p. 9), with the estimator on pp. 496–497. The one-observation-per-arm closed-domain likelihood maxima give a biased compliance score, with the stated certificates, raw-difference expectation and five-per-arm positive-part example; boundary probabilities and the natural-power convention are part of the claim.

Historical component credits
  • Proved AronowAudit.P029.constant_score_counterexample_law Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected saturated_ml_bias_full_counterexample already exports the population IV, latent-ignorability and overlap facts needed here.
  • Proved AronowAudit.P029.five_per_arm_bias_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': "P029_Audit.lean:1844 gives exactly T13(iv)'s positive-part-score expectation 15659/64000, bias 2859/64000, and inverted-count probability 257079/1600000. The product-binomial law and armRateScore definitions encode the stated independent counts and max(k1/5 - k0/5, 0). This row covers that exact enumeration component; T13(iv) itself specifies the score formula and references T14(a) for its ML characterization, so no ML-identification conclusion is omitted here."}
  • Counterexample proved AronowAudit.P029.saturated_ml_bias_full_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': "P029_Audit.lean:2970 combines the valid constant-effect IV population, its true score 1/2 and receipt rates 3/4 and 1/4, globally maximizing feasible fits, and their expectation 9/16. T13's own Reading explicitly admits the closed triangle 0 <= q <= p <= 1 and the 0^0 = 1 convention, so these are not restrictions disclosed only in metadata. The claim specifies the maximizing pairs rather than requiring a separate uniqueness theorem, and this counterexample loses none of its stated cases."}
  • Proved AronowAudit.P029.saturated_unbiasedness_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029_Audit.lean:1817 gives the prescribed fitted-score values, expectation 9/16, bias 1/16, and unbiased raw-difference expectation 1/2 under the independent two-receipt law, matching T13(ii)-(iii). Its score is defined from the same fitted pairs whose actual closed-likelihood maximality is established by the companion declaration at P029_Audit.lean:1796. The closure convention appears in T13 itself, so evaluating those boundary pairs does not replace the target with another estimator.'}
  • Proved AronowAudit.P029.two_observation_ml_maxima Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected saturated_ml_bias_full_counterexample already exports maximality for every two-receipt data set.

Assumption 2, last sentence, and Section 3.1 ‘strata means under saturation’, corrected: the saturated ML compliance score is the receipt-rate difference truncated at zero; it is consistent, biased upward under two-sided noncompliance and unbiased under one-sided noncompliance

p. 500 (PDF p. 9), Assumption 2; p. 496 (PDF p. 5), Section 3.1, third paragraph

p. 500, Assumption 2; pp. 496–497, Section 3.1. With boundary probabilities allowed, the saturated ordered binomial likelihood has the unique separate-rate or pooled maximizer and compliance score max(r1-r0,0). The interior nested parameters, finite bias and extreme-event probability, contraction and tail bound, per-stratum and simultaneous finite-stratum consistency, and one-sided unbiasedness (requiring only a nonempty treated arm) all hold. The usual measure-theoretic conditions are assumed.

Historical component credits
  • Proved AronowAudit.P029.nested_receipt_parameters Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the interior parameterization component, not independently the likelihood-maximization certificate.
  • Proved AronowAudit.P029.Wave1.one_sided_saturated_unbiasedness_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The corrected prose mentions only a nonempty treated arm, but its detailed mathematical model expressly fixes n0>=1 as well; this row matches that detailed component.
  • Proved AronowAudit.P029.positive_part_bias_identity Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The binomial bias row already supplies the target-specific identity and sign.
  • Proved AronowAudit.P029.positive_part_error_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Harmless generalization; substituting the empirical rate difference and true compliance gives exactly the target contraction.
  • Proved AronowAudit.P029.Wave1.saturated_binomial_bias_bound_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This row is a proof of the corrected finite bias/bound component, not the printed unrestricted unbiasedness assertion. The separate contraction and asymptotic conclusions are covered by R193-07 and R193-04.
  • Proved, conditional on a published theorem AronowAudit.P029.Wave1.saturated_ml_characterization_conditional Assumes as a hypothesis: C. van Eeden, "Maximum likelihood estimation of ordered probabilities", Proc. KNAW Ser. A 59 = Indagationes Math. 18 (1956), Theorem I, p. 445 Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Its proof requires an external ordered-binomial theorem as a hypothesis; the supplied unconditional whole proof establishes optimization instead.
  • Proved AronowAudit.P029.Wave1.saturated_score_consistency_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The target metadata says asymptotic=false even though T14(c) expressly includes this asymptotic conclusion; the mathematical target and row purpose are consistent. Positivity at each sample size follows the detailed positive-arm model.

Footnote 6: ‘Asymptotically, assuming that α\alpha remains fixed, the Winsorization has no impact on estimation, because c→0c \to 0 as n→∞n \to \infty’; the witness is the validator's and is not in the paper

p. 498 (PDF p. 7), footnote 6

p. 498, footnote 6. The constant-score counterexample satisfies the five IV assumptions and Assumptions 1–3, with receipt-arm means 3/4 and 1/4 and ATE and population ICSW equal to one. Both inverse empirical-CDF and type-7 quantiles of its nonempty half-score vectors exist and equal one half, including at every positive power level n^(-alpha). The cutoff therefore does not tend to zero; no scores or weights change. The usual measure-theoretic conditions are assumed.

Historical component credits
  • Proved AronowAudit.P029.constant_quantile Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Harmless generalization of the constant-sample auxiliary lemma. This row alone is not credited as a complete population counterexample.
  • Proved AronowAudit.P029.constant_score_counterexample_law Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the population-law component of the witness. The cutoff nonconvergence and oracle-score sequence are separately exported in R193-10 and R193-11; fScore=1/2 already entails zero error for scores equal to 1/2.
  • Proved AronowAudit.P029.cutoff_not_zero Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the quantile/nonconvergence component, paired with the valid population law in R193-09/R193-11. T15's asymptotic=false metadata does not reflect its explicit cutoff nonconvergence statement.
  • Counterexample proved AronowAudit.P029.winsorization_cutoff_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The counterexample refutes the stated cutoff-to-zero rationale, not the claim that Winsorization can preserve consistency. The target explicitly permits oracle scores; it does not require this row to formalize the intercept-only fitted-ML sequence.

Footnote 6, the asymptotic justification of Winsorization, corrected: the fraction of scores changed tends to zero and, with bounded estimated inverse scores and integrable outcomes, the probability limit is unchanged; the cut-off need not tend to zero

p. 498 (PDF p. 7), footnote 6

p. 498, footnote 6, corrected. The type-7 changed count is at most floor(1+(n-1)lambda), and its fraction is at most lambda+1/n. The inverse empirical-CDF convention obeys the same bounds. At fixed positive power levels both changed fractions vanish. Quantiles preserve common lower bounds; half-score cutoffs exist and stay one half. The pointwise inverse-weight and bounded weighted-mean bounds hold. Under the printed finite- profile iid, overlap, integrability and uniform score consistency conditions, ordinary and Winsorized ICSW have the stated common limit and vanishing difference, with ATE identification under Assumption 1. The usual measure-theoretic conditions, including measurable scores and cutoffs, are assumed. Ratios use their real-valued extension on undefined samples; no eventual-definedness assumption is imposed.

Historical component credits
  • Proved AronowAudit.P029.bounded_weighted_mean_change Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The boundedness premise is appropriate for the auxiliary bounded-outcome step; the separate tail inequality and limit theorem cover the integrable-outcome argument, so this row is not credited with the full unbounded-outcome limit.
  • Proved AronowAudit.P029.inverse_score_accuracy Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Reciprocal-error auxiliary is used within the existing limit argument; it is not a separate claim clause.
  • Proved AronowAudit.P029.outcome_tail_weight_bound Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Tail-truncation auxiliary is already contained in the existing corrected-limit proof.
  • Proved AronowAudit.P029.quantile_changed_fraction_tends_zero Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; This statement concerns the fraction changed, not the numerical cutoff; constant positive scores are therefore consistent with its conclusion.
  • Proved AronowAudit.P029.quantile_count_and_lower_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; An arbitrary finite real vector has a real common lower bound, so supplying b introduces no substantive restriction for the count claim.
  • Proved AronowAudit.P029.ratio_perturbation Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Ratio algebra is an auxiliary already subsumed by the corrected-limit proof.
  • Proved AronowAudit.P029.weight_change_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Harmless generalization of the exact pointwise weight bound; neither the cutoff nor score upper bound is needed for its specialization.
  • Proved AronowAudit.P029.Wave1.winsorization_corrected_limit_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The expanded ICSWConditions fields do not assume any estimator limit. The printed footnote is being corrected; finite support and uniform consistency occur in T16(iv) itself, so their disclosure is not being used to excuse a narrower theorem.

Footnote 3, the likelihood under one-sided noncompliance, corrected (product of the two Bernoulli factors), and the reduction to binomial regression on the units assigned to treatment

p. 497 (PDF p. 6), footnote 3; equation (7) on the same page

p. 497, footnote 3 and equation (7), corrected. The one-sided nested Bernoulli likelihood reduces to the treated-arm likelihood, and their maximizer predicates agree. For every interior probability the printed additive factor has the stated two formulas and is at least one; at three quarters it gives seven quarters and five quarters instead of the Bernoulli probabilities. The usual measure-theoretic conditions are assumed.

Historical component credits
  • Proved AronowAudit.P029.one_sided_likelihood_reduction Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Equality is stronger than equality of maximizer sets: for arbitrary theta, substitute pi=pi(theta) into the theorem. Thus the unexported argmax notation does not omit mathematical information. No final verdict change from claim-stage.
  • Proved AronowAudit.P029.one_sided_unit_likelihood Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Harmless extension from probability profiles to arbitrary real pi. This is a substitution lemma in the corrected likelihood argument, not a claim that values outside [0,1] define probabilities.
  • Proved AronowAudit.P029.printed_one_sided_factor_countercheck Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is an exact witness to the printed formula error, not a quantified theorem giving its general value for every pi; that remaining part of T17(c) is recorded as uncovered in targets.json.

Section 3.1 (the second definition of the compliance score and the receipt-probability display) and Appendix (equation (9) and the estimator display), corrected notation

p. 496 (PDF p. 5), sentence after equation (6); p. 497 (PDF p. 6), first display and the sentence after it; p. 504 (PDF p. 13), equation (9) and the display after equation (14)

pp. 496–497, Section 3.1; p. 504, equation (9) and the estimator display. Under random assignment and no defiers, the type partition, conditional compliance and receipt formulas, positive-AC nested representation, and interior probability range hold. Every law with no always-takers, conditional compliance probability one half at the profile, and assignment probability one half gives the printed misconditioning discrepancy. With existing weighted expectations and positive compliance almost surely, weighted ITT equals the corrected covariate-law integral in (9), which cancels to (10). The single-hat sample display equals Section 3.3. The usual measure-theoretic conditions are assumed, including measurable versions of the conditional means and integrability of the displayed expectations. No uniform positive lower bound on compliance is required for the integral identity.

Historical component credits
  • Proved AronowAudit.P029.compliance_type_partition Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Pointwise Boolean auxiliary fact; the target’s zero-defier-probability assumption makes it applicable almost everywhere.
  • Proved AronowAudit.P029.corrected_equation_nine_integrand Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary pointwise identity, sufficient for the integrand cancellation rather than a standalone consistency theorem.
  • Proved AronowAudit.P029.Wave1.corrected_nested_receipt_model_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the corrected target, including boundary nested probabilities where the target identities permit them; no claim about the literal misprinted conditioning is made.
  • Proved AronowAudit.P029.nested_probability_range Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The corrected target itself includes both interior-probability premises.
  • Proved AronowAudit.P029.nested_receipt_parameters Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; The actual conditional-probability and positive-AC identities are supplied by the whole proof; this separate algebraic parameterization lemma is unnecessary.
  • Proved AronowAudit.P029.receipt_conditioning_countercheck Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; The fixed witness suffices for the row’s countercheck; this is not credited as the full universal corrected probability theorem.

Supplement, ‘Baseline Data Generating Process’ and ‘Heterogeneous Treatment Effects, Model is Incorrect’, corrected: the treatment equation Di=CiZi+AiD_i = C_iZ_i + A_i and the threshold tests in the latent-index indicators

Supplement p. A1 (lines for ACiAC_i, AiA_i, CiC_i, DiD_i) and p. A2 (lines for ACiAC_i, AiA_i of ‘Model is Incorrect’)

supplement pp. A1-A2, baseline and misspecified simulation designs. Independent uniform covariates and shared coefficients, normal errors of standard deviation three (baseline), logistic errors of location zero and scale three (nonlinear design), standard normal outcome errors, and independent uniformly balanced assignment realize the corrected recipes for every even sample size. Thresholded indices give binary disjoint types, monotone potential treatments and the stated type characterizations. The corrected treatment is the realized potential treatment; an assigned complier receives treatment. The printed treatment equation can have no solution or two solutions. The printed outcome and effect formulas obey the potential-outcome identity. The usual measure-theoretic conditions are assumed.

Historical component credits
  • Proved AronowAudit.P029.corrected_simulation_types Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The target's mathematical content explicitly quantifies any real latent indices, so the earlier record's claim of a stronger arbitrary-index domain is not a difference from this component. Noise laws and balanced assignment are recipe specifications and immaterial to these pointwise conclusions.
  • Proved AronowAudit.P029.printed_simulation_equation_failure Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The selected refutation is exact for binary A,Z,D; using real D in the impossibility statement strengthens it without changing the printed reading. Both required binary solutions are exported in the nonunique case.
  • Proved AronowAudit.P029.simulation_outcome_identity Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is only T19(iv)'s observed-outcome algebra, with the potential-outcome substitution Y1=Y0+τ; it does not purport to prove a stochastic exclusion assumption from observational data.

Assumption 1 and the paragraph comparing it with the typical ignorability assumption, as printed

p. 499 (PDF p. 8), Assumption 1 and the following paragraph

The printed claim is false: P029-T20; p. 499, Assumption 1 and the following paragraph. Typical mean ignorability need not imply latent ignorability under the IV assumptions, and a separate baseline-selection law satisfies latent ignorability with unequal received-treatment potential-outcome means.

Historical component credits
  • Proved AronowAudit.P029.baseline_selection_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the valid existence example for latent ignorability without typical ignorability, not a positive proof of the false I-implies-L ordering.
  • Counterexample proved AronowAudit.P029.Wave1.printed_ignorability_ordering_refuted Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029/Wave1/IgnorabilityOrdering.lean:16 negates the universal implication from the two printed received-treatment mean equalities to latent ignorability, the component of the claim saying that Assumption 1 is "less restrictive" than typical ignorability. The positive-mass and positive-score guards occur inside the negated universal: establishing a failure even on this smaller class suffices to refute the printed all-support-profile implication. The definitions in P029/P029_Audit.lean:417 and :436 use the stated IV assumptions and treatment received D, and the finite law at :181 has constant X, profile mass 1, compliance score 1/2, both received-treatment arms of mass 1/2, ordinary conditional effect 1/2 and complier effect 1. Thus neither zero-probability conditioning nor universe size invalidates the counterexample; refuting the forward implication suffices without refuting the separate baseline-selection example.'}
  • Counterexample proved AronowAudit.P029.Wave1.printed_ignorability_ordering_all_profiles_refuted Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; The direct positive-profile universal ordering is already refuted by the selected measure-theoretic row; a second kernel presentation is unnecessary.

Section 3.4, the paragraph on sufficient conditions for Assumption 1 (constant effects; effects independent of compliance), as printed

p. 499 (PDF p. 8), paragraph beginning ‘We can describe sufficient conditions for and consequences of Assumption 1’

The printed claim is false: P029-T21; p. 499, the sufficient-conditions paragraph. Both printed independence implications fail, while constant effects give the stated population identities under uniform positive overlap. The constant-effect component adds measurable conditional-mean versions and a uniform positive lower bound for the compliance score, and asserts the profile identities only at profiles of positive probability and positive score. It does not assert the same constant-effect result under positivity without that uniform bound, or sample limits. The marginal refutation is stated for independence of the complier indicator.

Historical component credits
  • Proved AronowAudit.P029.Wave1.constant_effect_population_limits_proof Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P029/Wave1/Gap/GP029B2b.lean:580 requires a common epsilon > 0 with epsilon < c(X) almost surely, whereas T21(a) says constant effects give the conditional means and both population limits equal to tau without imposing uniform overlap. This hypothesis excludes nontrivial constant-effect populations: take Pr(X=k)=3*4^(-(k+1)), c(k)=2^(-(k+1)), Y0=0 and Y1=tau, with randomized assignment and one-sided compliance; E[1/c(X)]=3 and both population Wald ratios are tau, but no such epsilon exists. Applying the same global bound to the conditional-mean and ordinary-Wald conclusions also restricts conclusions that do not need it. The positive-mass event formulation at P029/Wave1/Gap/GP029B2b.lean:582 and the almost-sure conditional-expectation version are legitimate qualifications for null conditioning events, rather than a further major defect.', 'suggested_fix': 'Separate the constant-effect conditional-mean and ordinary-Wald conclusions from the inverse-score result. State conditional equalities on positive-score profiles without a global epsilon; for the inverse-score Wald identity use almost-sure positive scores and finiteness of the required inverse-weighted moments instead of a uniform lower bound.', 'second_review': 'T21(a) asserts the constant-effect conditional means and both population limits equal tau without requiring uniformly positive compliance scores. P029/Wave1/Gap/GP029B2b.lean:580 instead requires one epsilon > 0 below c(X) almost surely, restricting even its conditional-mean and ordinary-Wald conclusions. For example, profile masses 3*4^(-(k+1)) and scores 2^(-(k+1)) have finite mean inverse score 3 but no uniform positive lower bound, so substantive cases with well-defined inverse-weighted moments are excluded. The statement should separate the conclusions and use positivity and the necessary finite weighted moments for the inverse-score result.', 'severity': 'major'}
  • Counterexample proved AronowAudit.P029.Wave1.printed_conditional_sufficiency_refuted Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The conditional reading is explicitly a separate part of the target; the original informal wording is ambiguous between this and the marginal reading. Bounded outcomes and finite positive overlap make the population inequality a regular counterexample to the specified limit assertion.
  • Counterexample proved AronowAudit.P029.Wave1.printed_marginal_sufficiency_refuted Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The record’s marginal/conditional distinction agrees with the independent reading; no constant-effect assertion is refuted.

Supplement Table A3 ‘Noise (Homogeneous)’ with the design it reports on, as printed

Supplement p. A2 (design), p. A5 (text), p. A6 (Figure A3, panel ‘Noise (Homogeneous)’), p. A7 (Table A3)

supplement p. A2 (design), p. A5 (text), p. A6 (Figure A3, Noise (Homogeneous) panel), and p. A7 (Table A3). For a supplied array of 2500 estimates and a supplied benchmark of five, empirical RMSE is the square root of the average squared error about that benchmark.

Historical component credits
  • Proved AronowAudit.P029.homogeneous_rmse_rule Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': "P029/P029_Audit.lean:1018 states exactly the target's scoring formula sqrt(sum_r (t_r - 5)^2 / 2500), using the ordinary RMSE definition at P029/P029_Audit.lean:986. The component allocated to this row is the scoring rule, not verification of the historical numerical entries, and an arbitrary t includes every possible vector of simulated estimates. Unfolding a correct definition for this definitional component is not a vacuity caused by an empty domain, zero denominator, or an artificial convention that changes the paper's quantity. The fixed replicate count is positive and the homogeneous benchmark is exactly the target's 5, so the flag identifies no fidelity defect."}

Section 3.3, the recommendation to bootstrap the entire process, with its implementation in the R package icsw 1.0.0 (help text and source), as printed

p. 498 (PDF p. 7), Section 3.3, last paragraph; staged package: packages/icsw/icsw/man/icsw.tsls.Rd, lines 9–13 and 42–43; R/compliance.score.R, lines 43–48 and 88–91; R/icsw.tsls.R, lines 28–37; R/icsw.tsls.fit.R, lines 8 and 21–24

The printed claim is false: P029-T23; p. 498, Section 3.3; package compliance.score.R, lines 88-91, and icsw.tsls.R, lines 28-37. An exact minimizer of the package's inside-log criterion in an interior nested Bernoulli model with positive observation weights does not minimize the weighted negative log likelihood. This refutes the exact observation-weight fitting contract, and makes no assertion against the article's bootstrap recommendation or about numerical solver performance.

Historical component credits
  • Counterexample proved AronowAudit.P029.Wave1.nested_bernoulli_package_weighted_fit_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The record additionally mentions likelihood clipping to [10^-5,1-10^-5]. Both displayed fits lie in its interior and cellwise likelihood optima remain optimal against clipped competitors, so that fact does not change the witness verdict. Packaging/build checks are outside this semantic review.

Equation (3) (the LATE identity under the five IV assumptions, attributed to Angrist, Imbens, and Rubin 1996), the estimator (4) as bivariate 2SLS, and the worked example of Section 3.2

p. 495 (PDF p. 4), equations (3) and (4); pp. 497–498 (PDF pp. 6–7), the gender example

p. 495, equations (3)-(4); pp. 497-498, the gender example. The five IV assumptions give both ITT identities and the LATE ratio, the defined sample Wald estimator equals the centered bivariate IV coefficient, and the gender example has values one half, 2/17 (rounding to 0.12), and one half.

Historical component credits
  • Proved AronowAudit.P029.bernoulli_measure_probability Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Probability-law auxiliary is already included in the existing population identity proof.
  • Proved AronowAudit.P029.bivariate_ratio_algebra Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Algebraic auxiliary is already included in the sample coefficient identity.
  • Proved AronowAudit.P029.Wave1.late_identity_paper_scope Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Covers only the population equation-(3) component. Outcome integrability is expressly in the target, and positive complier mass follows rather than being assumed.
  • Proved AronowAudit.P029.gender_worked_example Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P029/P029_Audit.lean:2448 states all three numerical ratios in the worked-example component: ATE 1/2, LATE 2/17, and inverse-compliance-score weighted effect 1/2. It also states 0.115 ≤ 2/17 < 0.125, precisely the rounding interpretation expressly specified by the target for the printed 0.12. The denominators are respectively 1, 17/40 and 1, so these are substantive arithmetic equalities, and the row does not claim to establish the other components of T25.'}
  • Proved AronowAudit.P029.Wave1.late_identity_proof Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Duplicate export of the same population identity; use the paper-scope declaration directly.
  • Proved AronowAudit.P029.randomized_conditional_mean Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Finite-arm auxiliary is already subsumed by the general population identity.
  • Proved AronowAudit.P029.Wave1.randomized_probability_independent Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Randomization auxiliary is already included in the existing population identity proof.
  • Proved AronowAudit.P029.randomized_valid Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Finite-law auxiliary is already subsumed by the general population identity.
  • Proved AronowAudit.P029.sample_arm_unweighted_raw Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Raw-sum auxiliary is already included in the sample coefficient identity.
  • Proved AronowAudit.P029.sample_cross_raw Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Centering auxiliary is already included in the sample coefficient identity.
  • Proved AronowAudit.P029.sample_wald_bivariate_2sls Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The target itself defines the bivariate 2SLS component by this centered scalar ratio; an additional matrix/minimizer theorem is therefore not an omitted conclusion.
  • Proved AronowAudit.P029.unit_itt_identities Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Unitwise auxiliary is already included in the population identity proof.

Consistency of ICSW for the ATE under Assumptions 1–3: Section 3.4 (opening sentences) and the Appendix ‘ICSW Consistency’, equations (8)–(14)

p. 498 (PDF p. 7), Section 3.4, first paragraph; pp. 504–505 (PDF pp. 13–14), Appendix

p. 498, Section 3.4; pp. 504-505, Appendix equations (8)-(14). The population weighted-moment identities hold and ICSW converges to the ATE with uniform consistency of the scores over the growing observed sample.

Historical component credits
  • Proved AronowAudit.P029.appendix_finite_identities Limited support; baseline fidelity: narrower; manifest: not used; not used in Solution; Finite-support specialization is unnecessary: the selected population theorem already supplies the general integral identities.
  • Proved AronowAudit.P029.Wave1.appendix_population_identities_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; ScoreVersions is a genuine conditional-expectation specification; A1Versions is exactly the optional printed latent-ignorability premise, not an assumption of the weighted arm identity. Uniform overlap is supplied by Assumption 3 here.
  • Proved AronowAudit.P029.Wave1.icsw_consistency_proof Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'dispute', 'reason': "P029/Wave1/Gap/GP029B3b.lean:708 concludes convergence in probability to the ATE, and icswSequence at P029/P029_Audit.lean:2595 expands through sampleWald at P029/P029_Audit.lean:2335 to the Section 3.3 estimator specified by T26. ICSWConditions at P029/P029_Audit.lean:2580 supplies iid sampling, the IV assumptions, actual conditional-expectation versions, overlap, and measurable fitted scores in (0,1], with maximum in-sample score error converging in probability to zero at P029/P029_Audit.lean:2590. Maximum-error convergence is stronger than pointwise Assumption 2 and must not be represented as following from it alone. However, T26 expressly includes 'suitable regularity conditions', and the explicit sufficient specification at P029/P029_Audit.lean:2573 is noncircular and does not assume convergence of the estimator or its sample moments. The additional hypotheses at P029/Wave1/Gap/GP029B3b.lean:714, P029/Wave1/Gap/GP029B3b.lean:716 and P029/Wave1/Gap/GP029B3b.lean:717 represent conditional population effects, Assumption 1 and Assumption 3, so the row correctly proves this qualified consistency claim with the disclosed regularity retained.", 'defensible_statement': 'For iid full potential-outcome/assignment observations with second-countable Borel covariates, satisfying the binary IV assumptions and integrability of both potential outcomes, assume measurable conditional-mean versions satisfy Assumption 1 and the true compliance score c is bounded away from zero on the covariate support. If for each N the measurable fitted scores s_Ni lie in (0,1] almost surely and max_{1<=i<=N}|s_Ni-c(X_i)| converges in probability to zero, the Section 3.3 ICSW estimator with weights 1/s_Ni converges in probability to E[Y1-Y0].'}
  • Proved AronowAudit.P029.profile_inverse_score_cancellation Limited support; baseline fidelity: narrower; manifest: not used; not used in Solution; Finite-sum cancellation auxiliary adds no coverage beyond the selected general population/limit statements.
  • Proved AronowAudit.P029.sample_arm_scale Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Common-scale identity is an auxiliary, not a separate clause of the retained consistency claim.
  • Proved AronowAudit.P029.sample_wald_scale Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Common-scale auxiliary is unnecessary for the selected direct constant-score identity and consistency statements.

The limit of ICSW when Assumption 1 fails (complier effects reweighted to the population covariate distribution), the LATE as an integral against the complier covariate density, and 2SLS as the special case with no covariates

p. 499 (PDF p. 8), paragraph beginning ‘But what if treatment effects are heterogeneous’ and the two displays; pp. 499–500 (PDF pp. 8–9), the following paragraph; Appendix equations (10) and (13), p. 504 (PDF p. 13)

The printed claim is false: P029-T27; pp. 499-500, paragraph on failure of Assumption 1. Failure of latent ignorability need not produce asymptotic bias. An iid randomized twelve-atom law has strict positive overlap, interior nested receipt probabilities, heterogeneous effects, and failure of Assumption 1 at both profiles. Its ATE and every observed outcome are zero. Saturated nested ML scores are consistent on an event of probability tending to one, and the zero extension of ICSW converges to the zero ATE. The usual measure-theoretic conditions are satisfied.

Historical component credits
  • Proved AronowAudit.P029.Wave1.complier_covariate_distribution_proof Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; ScoreVersions gives defining conditional-expectation identities rather than assuming the Bayes law or the final integral equality. Standard measurability and integrability make the conditional means genuine; overlap is unnecessary for this component.
  • Proved AronowAudit.P029.complier_profile_bayes Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Finite Bayes auxiliary is subsumed by the selected genuine complier pushforward law.
  • Proved AronowAudit.P029.constant_scores_equal_iv Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; The row assumes exact constancy of fitted scores, matching T27(c); it does not claim constancy follows from any fitting algorithm.
  • Proved AronowAudit.P029.Wave1.icsw_complier_effect_limit_proof Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; {'source': 'dispute', 'reason': "P029/Wave1/Gap/GLP029B2.lean:44 concludes convergence in probability to the expectation of m(X) under the full population law, exactly the probability-limit component T27(b). ScoreVersions at P029/P029_Audit.lean:2547 identifies c(X) with E[1_C|X] and c(X)m(X) with E[1_C tau|X], so positivity makes m the conditional complier effect and its population expectation is the target's integral against f. The maximum-error condition at P029/P029_Audit.lean:2590 strengthens pointwise score consistency, but T27(b) explicitly qualifies its conclusion by 'suitable regularity conditions', for which ICSWConditions at P029/P029_Audit.lean:2580 supplies a noncircular sufficient specification. Thus the failure of pointwise consistency alone to imply maximum-error consistency does not contradict the qualified target or establish an omitted conclusion. The declaration assumes neither latent ignorability nor its negation, as appropriate for a limit derived without Assumption 1, and the row expressly claims only T27(b), leaving T27(a) and T27(c) outside its credit.", 'defensible_statement': 'Let c(x) and m(x) be measurable versions of P(C=1|X=x) and E[Y1-Y0|C=1,X=x], with second-countable Borel covariates. For iid full potential-outcome/assignment observations satisfying the binary IV assumptions and integrable potential outcomes, with c bounded away from zero on the covariate support and measurable fitted scores s_Ni in (0,1] almost surely whose maximum in-sample error converges in probability to zero, the Section 3.3 ICSW estimator converges in probability to E[m(X)] under the full population covariate law. This conclusion requires no latent-ignorability assumption.'}
  • Proved AronowAudit.P029.profile_inverse_score_cancellation Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Finite-sum cancellation auxiliary adds no coverage beyond the selected general population/limit statements.
  • Proved AronowAudit.P029.sample_wald_scale Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Common-scale auxiliary is unnecessary for the selected direct constant-score identity and consistency statements.

Reproduction

What I recalculated or reran from public materials. "Agrees", "Agrees in part" or "Disagrees" compares my result with the printed one. "Not compared" means the comparison could not be made.

Status
Full reproduction of the selected headline targets
Agreement
Agrees in part
Reported results that were targeted
TargetWhereMethodStatusAgreement
Table 1p. 501 (PDF p. 10)authors' code with changes: all point estimates and 2,500-draw intervals agree to rounding after correcting the age-missing indicator; with deposited preprocessing, the Age missing coefficients are 97.3 and 96.6 rather than 12.7 and 17.7 (P029-E021)ReproducedAgrees in part
Table 2p. 503 (PDF p. 12)authors' code with changes: point estimates and all 2,500-draw bootstrap intervals; capped optimizer fits retained as in the script (P029-N010)ReproducedWithin Monte Carlo error
Canvassing ATE versus LATEp. 502, first paragraph and footnote 11 (PDF p. 11)authors' code with changes: relative increase from unrounded coefficients and paired difference intervals from all 2,500 bootstrap draws, with the age-missing indicator correctedReproducedTo rounding
Debate-viewing ATE versus LATEp. 503, text below Table 2 and footnote 13 (PDF p. 12)authors' code with changes: relative increase from unrounded coefficients and paired difference intervals from all 2,500 bootstrap draws; capped optimizer fits retained as in the script (P029-N010)ReproducedWithin Monte Carlo error