Corrections to: Nonparametric identification is not enough, but randomized controlled trials are

The problem

The proof conditions on the covariate vector $\mathbf{W} = (W_1, \dots, W_n)$ and applies Hoeffding’s inequality to the deviation of $\hat\mu$ from $\mu$. Conditional on $\mathbf{W}$, however, the mean of $\hat\mu$ need not equal $\mu$. Let $m(w) = \mathrm{E}[Y_i(1) \mid W_i = w]$ and $\mu_{\mathbf{W}} = n^{-1}\sum_{i=1}^n m(W_i)$. Then $\mathrm{E}[\hat\mu \mid \mathbf{W}] = \mu_{\mathbf{W}}$; taking the expectation over $\mathbf{W}$ gives the unconditional identity $\mathrm{E}[\hat\mu] = \mu$ that the proof recalls. Hoeffding’s inequality bounds the conditional probability that $|\hat\mu - \mu_{\mathbf{W}}|$ exceeds the printed radius. It does not bound $|\hat\mu - \mu|$, and the printed radius does not account for the variation of $\mu_{\mathbf{W}}$ around $\mu$.

The proposition itself fails as stated. It claims coverage at least $1-\alpha$ and states no restriction on $\alpha$. In the design that I give below, which satisfies every assumption of Section 2, the printed interval fails to contain $\mu$ with probability $1.198 \times 10^{-7}$ at $\alpha = 10^{-7}$. This level is far from the levels used in practice. The counterexample does not show that the printed interval undercovers at the 95% or 99% level: in the same design the probability that it fails to contain $\mu$ is below $\alpha$ at both levels. These calculations do not settle whether the printed interval is valid at such levels in every design. To repair the proposition I keep the estimand, the estimator and the assumptions, and I widen the interval. A second Hoeffding bound controls $\mu_{\mathbf{W}} - \mu$, and I allot error probability $\alpha/2$ to each of the two bounds. The corrected interval still has width of order $n^{-1/2}$.

Evidence an exact counterexample

The evidence is an exact counterexample to the printed statement and a derivation of the corrected one.

Counterexample. Let $n = 100$ and $\alpha = 10^{-7}$. Let $W_i$ be uniform on $(0,1)$, and set $Y_i(1) = \mathbf{1}\{W_i \le 3/5\}$ and $Y_i(0) = 0$. Let $\pi(w) = 13/20$ for $w \le 3/5$ and $\pi(w) = 99/100$ for $w > 3/5$, and assign the treatments independently given the covariates. The observations are i.i.d., the outcome and the treatment are binary, the covariate is continuous, consistency and unconfoundedness hold, positivity holds with $\delta = 1/200$, and $\mu = 3/5$. Let $K$ be the number of units such that $W_i \le 3/5$, and let $S$ be the number of these units that are treated. In this design $\hat\mu = S/65$ and $\xi = [400K/169 + 10000(100-K)/9801]/100$, which implies that the printed interval depends on the sample only through $(K, S)$. For $0 \le s \le k \le 100$, I have

The printed interval fails to contain $\mu$ exactly when $200(S/65 - 3/5)^2/\xi > \log(2 \cdot 10^7)$. This inequality holds for 1946 of the 5151 pairs $(k,s)$. Summing their probabilities in exact rational arithmetic, with rational bounds on the logarithm, I obtain

The conditional display in the proof fails more directly. On the event that $W_i > 3/5$ for every $i$, which has probability $(2/5)^{100} > 0$, the estimator $\hat\mu$ equals $0$ and $\xi = (100/99)^2$. On this event $|\hat\mu - \mu| = 3/5$ exceeds the printed radius, the left side of the display equals $1$, and the right side equals $10^{-7}$.

The table lists, at three levels, the exact probability that each interval fails to contain $\mu$ in this design. Only the first entry of the first row exceeds its nominal level.

Derivation of the corrected interval. Fix $\alpha \in (0,1)$ and set $c = \sqrt{\log(4/\alpha)/(2n)}$. By consistency, I have $X_iY_i = X_iY_i(1)$. Using unconfoundedness and the independence of the units, I find

and summing over $i$, I obtain $\mathrm{E}[\hat\mu \mid \mathbf{W}] = \mu_{\mathbf{W}}$. Conditional on $\mathbf{W}$, the variables $A_i$ are independent and satisfy $0 \le A_i \le [n\pi(W_i)]^{-1}$. Since $\sum_{i=1}^n [n\pi(W_i)]^{-2} = \xi/n$, Hoeffding’s inequality implies that

for every $t > 0$ that is a function of $\mathbf{W}$. Setting $t = \sqrt{\xi}\,c$ and taking the expectation over $\mathbf{W}$, I obtain

Next, the variables $m(W_i)$ are i.i.d., take values in $[0,1]$ and have mean $\mu$. Applying Hoeffding’s inequality to their mean, I have $\Pr(|\mu_{\mathbf{W}} - \mu| \ge c) \le 2\exp(-2nc^2) = \alpha/2$. If neither event occurs, then by the triangle inequality I have

By a union bound, I conclude that $\Pr(|\hat\mu - \mu| \ge (\sqrt{\xi}+1)c) \le \alpha$. Positivity implies that $\sqrt{\xi} < \delta^{-1}$, and the width of the corrected interval is $O(n^{-1/2})$ for fixed $\alpha$. For the alternative with a known bound $b > 0$ such that $\pi(W_i) \ge b$ with probability one, the variables $X_iY_i/\pi(W_i)$ are i.i.d., take values in $[0, b^{-1}]$ and have mean $\mu$, and Hoeffding’s inequality applied to their mean $\hat\mu$ gives the radius $b^{-1}\sqrt{\log(2/\alpha)/(2n)}$.

What survives

The claim of the abstract and of p. 7 survives: with a binary outcome and a known propensity score, there exist finite-sample valid confidence intervals for $\mu$ whose width decreases at the rate $n^{-1/2}$. The corrected interval is one. The printed radius remains valid when the propensity score is constant, by unconditional Hoeffding’s inequality. Propositions 1, 2 and 4 and the numerical illustration of Section 3 do not use Proposition 3. What changes is the radius in Proposition 3, which becomes larger, the statement on p. 7 that the printed interval guarantees uniform validity, and the proof in Appendix A.

The problem

Items 1 to 7 are errors of grammar: a singular subject with a plural verb in items 1, 2, 5 and 7, a singular noun for the two fitted regressions in item 3, an extra “of” in item 4, and a missing “be” in item 6. The reference in item 8 prints the label “doi:” twice.

Evidence a check of the printed page

I checked the pages. On p. 7 the paper calls $\hat g_0$ and $\hat g_1$ “two estimators of the regression function”, which supports the plural in item 3.

What survives

The intended meaning is clear in every case, and no statement, formula or number changes.

The problem

Note that the Horvitz–Thompson estimator is the adjusted estimator $\hat\mu_R$ with the zero function in place of $\hat g_0$ and $\hat g_1$. For a fixed function $g$, the summand of $\hat\mu_R$ has variance $\operatorname{Var}(Y_i(1)) + R(g)$, where

and the printed weights make the population weighted least-squares criterion on the treated units proportional to $R(g)$. The fit minimizes $R$ only over the family of functions in which the two regressions are fitted, which I call the working family. The comparison with the Horvitz–Thompson estimator requires that the limit $g_*$ of the fit satisfy $R(g_*) \le R(0)$. This holds when the working family contains the zero function, and it can fail otherwise. Part (c) as printed admits every family with a finite number of parameters. Every linear model with freely fitted coefficients contains the zero function, and the usual working models satisfy this condition.

Evidence an exact counterexample

The evidence is an exact counterexample to the printed statement and a derivation of the corrected one.

Counterexample. Let $W_i$ be uniform on $(0,1)$, let $L_i = \mathbf{1}\{W_i \le 1/2\}$, and let $Z_i$ be a Bernoulli variable with mean $1/10$ that is independent of $W_i$. Set $Y_i(0) = Y_i(1) = L_iZ_i$. Let $X_i$ be independent of $Z_i$ given $W_i$, with $\pi(W_i) = 1/4$ if $L_i = 1$ and $\pi(W_i) = 3/4$ if $L_i = 0$. The assumptions of Section 2 hold, with $\delta = 1/8$ in the positivity condition, and $\mu = 1/20$. Consider the working family of the functions $g_\theta(w) = (1/2)\mathbf{1}\{w \le 1/2\} + \theta\,\mathbf{1}\{w > 1/2\}$ with $\theta \in \mathbb{R}$. It has one parameter and does not contain the zero function. The printed weights equal $12$ where $L_i = 1$ and $4/9$ where $L_i = 0$. Since $Y_i = 0$ where $L_i = 0$, the weighted criterion on any training fold equals a term that does not depend on $\theta$ plus $(4/9)N\theta^2$, where $N$ is the number of treated units of the training fold such that $L_i = 0$. The value $\theta = 0$ minimizes it for every training fold (I take $\theta = 0$ when $N = 0$), and $\theta = 0$ is the unique minimizer of the population criterion $R(g_\theta) = 3/8 + \theta^2/6$. With $g_*(w) = (1/2)\mathbf{1}\{w \le 1/2\}$, I therefore have $\hat g_0 = \hat g_1 = g_*$, and $\hat\mu_R$ is the mean of the i.i.d. variables $A_i = X_i[Y_i - g_*(W_i)]/\pi(W_i) + g_*(W_i)$. The table lists the values of $A_i$ and of the Horvitz–Thompson summand $H_i = X_iY_i/\pi(W_i)$.

A direct computation gives $\mathrm{E}[A_i] = \mathrm{E}[H_i] = 1/20$, $\mathrm{E}[A_i^2] = 17/40$ and $\mathrm{E}[H_i^2] = 1/5$. For every $n$, I obtain

In this example $\operatorname{Var}(Y_i(1)) = 19/400$, $R(0) = 3/20$ and $R(g_*) = 3/8$.

Derivation of the corrected statement. Fix a square-integrable function $g$ and write $A_g = Y_i(1) + (X_i/\pi(W_i) - 1)(Y_i(1) - g(W_i))$ for the summand of $\hat\mu_R$. By unconfoundedness, conditional on $(W_i, Y_i(1))$ the variable $A_g$ has mean $Y_i(1)$ and variance $(1 - \pi(W_i))\pi(W_i)^{-1}(Y_i(1) - g(W_i))^2$. By the law of total variance, I have $\operatorname{Var}(A_g) = \operatorname{Var}(Y_i(1)) + R(g)$. For $g = 0$ this is the variance of the Horvitz–Thompson summand. If the working family contains the zero function, then a minimizer $g_*$ of $R$ over the family satisfies $R(g_*) \le R(0)$. Suppose that $\hat g_0$ and $\hat g_1$ converge to $g_*$ in $L^2$. Conditional on the training fold, the summands of the difference between $\hat\mu_R$ and the mean of the variables $A_{g_*}$ have mean zero and second moment $\mathrm{E}[(1 - \pi(W_i))\pi(W_i)^{-1}(\hat g_j(W_i) - g_*(W_i))^2]$ over the evaluation fold, which implies that this difference is $o_p(n^{-1/2})$. Then $\sqrt{n}(\hat\mu_R - \mu)$ has asymptotic variance $\operatorname{Var}(Y_i(1)) + R(g_*)$, which is at most the asymptotic variance $\operatorname{Var}(Y_i(1)) + R(0)$ of $\sqrt{n}(\hat\mu - \mu)$.

What survives

The sentence on p. 7 that adjustments “can be constructed so that they cannot harm, but can dramatically improve, asymptotic precision” asserts that suitable adjustments exist, and it stands. No other result of the paper relies on part (c), and the paper states on p. 10 that part (c) does not apply to its HTLOESS estimator. In the counterexample both estimators remain unbiased and root-$n$ consistent. What changes is the hypothesis of part (c).

Intended meaning

The sentence summarizes Proposition 1, which states the impossibility “Without further assumptions”, and with this meaning it is correct. Read literally, the clause after “unless” lists every remedy, and the list is incomplete: suitable assumptions on the outcome regression alone also suffice, with the propensity score unknown and unrestricted. For example, suppose that $\mathrm{E}[Y_i(x) \mid W_i = w] = a_x + h(w)$ for $x \in \{0,1\}$, where $h(w) = \min\{1, \max\{0, w\}\}/4$ is known and $a_0, a_1 \in [1/4, 1/2]$ are unknown. Then the difference of the within-arm means of $Y_i - h(W_i)$ (zero for an empty arm) estimates the average treatment effect with mean squared error at most $(3/4)^n + 25/[2(n+1)]$, for every measurable propensity score such that $1/4 < \pi(W_i) < 3/4$. The body of the paper allows for such assumptions (the paragraph after Proposition 1 and footnote 1, p. 6), and no result depends on the literal reading.

Intended meaning

The paragraph after Proposition 4 cites Theorem 5.1 of Chernozhukov et al. (2018) for part (d). That theorem, as it appears in the preprint arXiv:1608.00060v6, names the efficiency bound of Hahn (1998), which is the bound in the model with an unrestricted distribution $F_W$ of the covariate. Let $m(w) = \mathrm{E}[Y_i(1) \mid W_i = w]$. For $\mu$ and a known propensity score this bound equals $\mathrm{E}[m(W_i)(1 - m(W_i))/\pi(W_i)] + \operatorname{Var}(m(W_i))$, and $\hat\mu_R$ attains it when $\hat g_0$ and $\hat g_1$ converge to $m$ in $L^2(F_W)$, whether or not $F_W$ is known. The set-up on p. 5 allows $F_W$ to be “either known or unknown”. In the smaller model in which $F_W$ is known the bound is lower by $\operatorname{Var}(m(W_i))$, and part (d) does not claim that $\hat\mu_R$ attains that lower bound.

Intended meaning

The passages name the regression families, the degree and span of the LOESS fits and the estimator formulas, and nothing in them is false. They do not fully specify the estimators behind Figure 3. In the replication code (inesims.R), every LOESS prediction that is missing is replaced by $0.5$, which enters LOESS, DRLOESS and HTLOESS, and the estimated propensity scores of DRLOG and DRLOESS are raised to $0.01$ when they fall below it, with no upper cap. Both rules can take effect. Their effect on the reported RMSEs was not measured, and no reported value is shown to change.