Corrections to: Sharp bounds on the variance in randomized experiments

The problem

Lemma 2 makes two assertions about the distribution functions $G_N$ and $F_N$ of the potential outcomes in the population $U_N$ and their sample analogues $\hat G_N$ and $\hat F_N$. The first is that $G_N$ and $F_N$ converge uniformly to their limits $G$ and $F$. The second is that the probability of a uniform error of at least $\eta$ between a sample distribution function and its limit is at most of order $1/N$. Neither follows from conditions 1–3. Weak convergence (condition 2) implies that $G_N(y)\to G(y)$ at the points where $G$ is continuous, and condition 3 adds the points where $G$ jumps. Uniform convergence also requires the left limits to converge at the jumps, that is, $G_N(y-)\to G(y-)$, and no condition supplies this. The proof on p. 861 needs $|G(s_i-)-G_N(s_i-)|<\eta_1/2$ for large $N$ and obtains it “by conditions 2 and 3”. At a jump of $G$ this step can fail. If the outcomes of a population approach a jump point of $G$ from below without reaching it, then conditions 1–4 can hold while the uniform error stays at $1/2$ or above for every $N$ and every assignment.

The lemma underlies Remark 1, the proof outline on p. 856, stages (i) and (ii) in the proof of Proposition 1 on p. 862, and the bounds for the last terms in the proof of Proposition 2 on pp. 867–869. Those steps are not justified by the stated conditions when $G$ or $F$ has a jump. The conclusions of Propositions 1 and 2 are not affected, since both hold under their printed hypotheses by arguments that do not pass through uniform convergence. To repair the lemma, one regularity condition must be added. That condition holds when $G$ and $F$ are continuous, and it holds with probability one when the units are independent and identically distributed draws from a superpopulation.

Evidence an exact counterexample

The evidence is an exact counterexample to the lemma, together with derivations of the corrected statements.

The counterexample. For $i=1,2,\ldots$, let $(a_i,b_i)$ run through the cycle $(0,0),(0,1),(1,0),(1,1)$, and define

Let $U_N$ consist of the first $N$ units, for $N\ge 4$, and set $n=N$ and $m=\lfloor N/2\rfloor$. The populations are nested and the outcomes are fixed. Condition 1 holds with $\theta=1$ and $\rho=1/2$. Condition 2 holds: $H_N$ converges weakly to the distribution $H$ with mass $1/4$ at each of the points $(0,0)$, $(0,1)$, $(1,0)$ and $(1,1)$. Both marginals of $H$ have mass $1/2$ at $0$ and at $1$, so that $G(x)=F(x)=1/2$ for $0\le x<1$. Condition 3 holds as well. Since $y_{1i}\le 0$ exactly when $a_i=0$, I have $G_N(0)\to 1/2 = G(0)$, and since every outcome is below $1$, I have $G_N(1)=1=G(1)$; the same holds for $F_N$. All outcomes lie in $(-1,1)$, which implies condition 4, the boundedness assumed in Proposition 2 and the moment condition of Corollary 1. The distribution $H$ is not concentrated on a line.

Set $t_N = 1 - 1/\{2(N+2)\}$. For $i\le N$, I have $y_{1i}\le 1-1/(N+2) < t_N$ and $y_{0i}\le 1-1/(N+2) < t_N$, and $t_N<1$. Then $G_N(t_N)=F_N(t_N)=1$, and $\hat G_N(t_N)=\hat F_N(t_N)=1$ for every assignment, while $G(t_N)=F(t_N)=1/2$. This implies that, for every $N\ge 4$ and every assignment,

and likewise for $F$, $F_N$ and $\hat F_N$. The first assertion of the lemma fails. For $\eta_1=\eta_0=1/4$, the two probabilities in the second assertion equal $1$, and $N$ times each of them tends to infinity, while the right-hand sides equal $16K_1(1/4)$ and $16K_0(1/4)$. There exist no integers $K_1(1/4)$ and $K_0(1/4)$ for which the second assertion holds. Note that the left limits that the proof requires do not converge: $G_N(0-)\to 1/2$ while $G(0-)=0$, and $G_N(1-)=1$ while $G(1-)=1/2$. The table lists exact values at three population sizes, obtained by enumerating every assignment.

The added condition suffices. I write $P$ for $G$ or $F$, $P_N$ for the corresponding distribution function in the population, and $\hat P_N$ for the one in the sample, with $k$ sampled units and $\alpha=\lim k/N>0$. By weak convergence, I have $P_N(x)\to P(x)$ and $P_N(x-)\to P(x-)$ at every continuity point $x$ of $P$. At the discontinuity points, both limits hold by condition 3 and the added condition. Fix $\eta$ such that $0<\eta\le 1$, and set $R=\lceil 4/\eta\rceil$. The points $\pm\infty$ and the distinct quantiles of $P$ at the levels $j/R$, for $1\le j\le R-1$, form a partition with at most $R$ intervals such that $P(t-)-P(s)\le 1/R$ for adjacent points $s<t$. Since distribution functions are monotone, for every distribution function $A$, I have

where the maximum is over the adjacent points $s<t$ of the partition. Taking $A=P_N$, I obtain the uniform convergence. For $A=\hat P_N$, each of the at most $2R$ values $\hat P_N(s)$ and $\hat P_N(t-)$ has mean $P_N(s)$ or $P_N(t-)$ and variance at most $(N-k)/\{4(N-1)k\}$. Using Chebyshev’s inequality and the union bound, I obtain

This is the printed bound with $K(\eta)=2\lceil 4/\eta\rceil$. For $\eta>1$ the event is empty.

Proposition 1 without the lemma. Let $c>0$ be such that $m\ge cN$ and $n-m\ge cN$ for all large $N$. For fixed $x$, the value $\hat G_N(x)$ has mean $G_N(x)$ and variance at most $1/(4cN)$, so that $\hat G_N(x)-G_N(x)\to 0$ in probability. By condition 2, $\hat G_N$ converges weakly to $G$ in probability, and the same holds for $\hat F_N$ and $F$. For every sample and every $T>0$, I have

and by condition 4 the sampled distributions are uniformly square-integrable. Weak convergence and uniform square-integrability imply that the quantile functions converge in $L^2(0,1)$, in probability for the samples. The means, the second moments and the two integrals of products of quantile functions in (7)–(9) are continuous in the quantile functions with respect to that norm. I conclude that $NV_N^H$ and $NV_N^L$ have the limits stated in Proposition 1, and that $N(\hat V_N^B-V_N^B)\to 0$ in probability for $B\in\{H,L\}$. Condition 3 is not used.

Proposition 2 without the lemma. Suppose $|y_{1i}|\le C$ and $|y_{0i}|\le C$, and write $P_{1N}=G_N$, $P_{0N}=F_N$, $k_1=m$ and $k_0=n-m$. For $a\in\{0,1\}$, define

By the Cauchy–Schwarz inequality and the bound on the variance above, I have $\mathrm{E}\,D_a^2\le C^2/(cN)$, so that $\mathrm{E}(D_1+D_0)^2\le 4C^2/(cN)$. The quantile functions $q_A$ and $q_P$ of two distributions on $[-C,C]$ with distribution functions $A$ and $P$ satisfy $\int_0^1|q_A(u)-q_P(u)|\,du=\int_{-C}^{C}|A(x)-P(x)|\,dx$. It follows that the error of the sample mean in arm $a$ is at most $D_a$, the error of the unbiased variance estimator is at most $8CD_a+C^2/(k_a-1)$, and the error of each estimated covariance bound in (8) is at most $2C(D_1+D_0)$. Substituting these bounds into (9), I obtain, for $N\ge 2/c$ and $B\in\{H,L\}$,

Markov’s inequality then implies the bound stated in the correction below.

What survives

Lemma 1 and the sharpness of the interval $[V_N^L,V_N^H]$, the bounds and estimators (7)–(9), and the ordering of $\hat V_N^H$ and $\hat V_N^L$ relative to Neyman’s bounds do not depend on Lemma 2. Propositions 1 and 2 keep their hypotheses and conclusions. With them stand the consistency of $\hat V_N^H$ used in Corollary 1 and the argument for asymptotically conservative Wald intervals. This error does not change the computations in the application, the simulations or Appendix C. Three things change: Lemma 2 gains a hypothesis, Remark 1 is reworded, and the proof outline and the steps in the proofs of Propositions 1 and 2 that rest on the lemma are replaced.

The problem

The variance $\sigma_N^2(I\{y_1\le y\})=G_N(y)\{1-G_N(y)\}$ is at most $1/4$, and it equals $1/4$ only when $G_N(y)=1/2$. The last relation in (11) is therefore an inequality. The sentence before the display and the companion display (12) state the bound correctly.

Evidence an exact counterexample

I used an exact counterexample. Let $N=4$ and $m=2$, and take $y$ below every potential outcome under treatment. Then $\hat G_N(y)=0$ in each of the six assignments, so that $\operatorname{Var}_X\hat G_N(y)=0$, while the right-hand side of (11) is $(4-2)/(4\cdot 3\cdot 2)=1/12$.

What survives

The proof substitutes (11) into (10), and that step uses only the upper bound. The bound that follows is unchanged. The separate problem with Lemma 2 is P022-E001.

The problem

Proposition 2 bounds $\mathrm{P}\{N|\hat V_N^H-V_N^H|\ge\varepsilon\}$ for an arbitrary $\varepsilon>0$, with $\varepsilon_1+\varepsilon_2+\varepsilon_3=\varepsilon$. The proof splits $N(\hat V_N^H-V_N^H)$ into seven terms. The first six receive the tolerances $\nu_1,\ldots,\nu_8$, whose sum is $\varepsilon_1$, and the last receives $1-\varepsilon_1$. The tolerances add to $1$ and not to $\varepsilon$. As printed, the argument bounds the event $\{N|\hat V_N^H-V_N^H|\ge 1\}$. When $\varepsilon_1\ge 1$ the tolerance $1-\varepsilon_1$ is not positive, and (23) cannot hold with a finite $\kappa_1(\varepsilon_1)$. The display after (23) has a second slip: it names the true variance $V_N$ where the sharp upper bound $V_N^H$ is meant. Read literally, it says that $N\hat V_N^H$ concentrates around $NV_N$, which is false when the bound is not attained.

Evidence a derivation

The evidence is a derivation with two exact examples. For the tolerance, take $\varepsilon=3$ and $\varepsilon_1=\varepsilon_2=\varepsilon_3=1$. The event in (23) is then that a nonnegative quantity is at least $0$. It has probability $1$, and the left-hand side of (23) is infinite. For the target, let $N=n=4r$ and $m=N/2$, with $r$ units at each of the points $(0,0)$, $(0,1)$, $(1,0)$ and $(1,1)$. Both marginal variances are $1/4$, the covariance is $0$ and its sharp upper bound is $1/4$, so that $NV_N=N/\{2(N-1)\}$ and $NV_N^H=N/(N-1)$. Since $N\hat V_N^H\to 1$ in probability, $N|\hat V_N^H-V_N|$ tends to $1/2$ and not to $0$.

The repair rests on the inequality that the proof already uses. Since the square $[-C,C]^2$ has area $4C^2$, I have

A tolerance $x>0$ for the last term therefore corresponds to the threshold $(N-1)x/(8NC^2)$ for the supremum. With $x=\varepsilon_1/2$ and $\nu_1+\cdots+\nu_8=\varepsilon_1/2$, the union bound controls the event $\{N|\hat V_N^H-V_N^H|\ge\varepsilon_1\}$, which contains the event at $\varepsilon$ because $\varepsilon_1<\varepsilon$.

What survives

The statement of Proposition 2 and its $O(1/N)$ rate are unaffected. The decomposition into seven terms and the bounds (17)–(22) stand, and the argument for $\hat V_N^L$ keeps its own tolerance $\varepsilon_2+\varepsilon_3$. The use of Lemma 2 in the same passage is a separate matter, treated in P022-E001, where the rate is obtained from a bound that does not use the lemma.

The problem

R ends a statement at a line break when the expression read so far is complete. The first of the two lines is a complete function definition. The second then begins a new statement with a binary operator, and the parser stops with the message unexpected '*' at line 5, column 5 of the function.

Evidence a recalculation from the source

I checked the code by running it. The block copied with its printed line breaks fails to parse at the place stated. With the operator moved to the end of the preceding line and no other change, the function parses and runs.

What survives

The formula intended for FPvar and the estimator are unchanged by the syntax repair. The reported numbers were not recomputed. Two further defects of the same function are P022-E002 and P022-E003.

The problem

Equation (8) weights each product of order statistics by the width $p_i-p_{i-1}$ of a cell of the partition $P_{m,n}$ of $[0,1]$, and the widths sum to $1$. The function subtracts $e=\sqrt{\texttt{.Machine\$double.eps}}=2^{-26}$ from every partition point, resets the first point to $e$, and uses the shifted points both to select the order statistics and as weights. When every cell is wider than $2e$, the shifted points select the order statistics of (8). The first weight, however, becomes $p_1-p_0-2e$, and the weights sum to $1-2e=1-2^{-25}$. A fraction $2e$ of the first product is lost from the sum. The output is then not invariant to adding a constant to every outcome, as the estimate (9) is, and it can be negative. The discrepancy is about $6\times10^{-8}$ times the product of two extreme outcomes, divided by $N-1$. For the application, with $N=1561$ and outcomes between $0 and $150, it is at most $8.6\times10^{-7}$ USD$^2$, against a reported estimate of 0.186. It vanishes for the upper bound when the smallest outcome in either arm is zero.

Evidence an exact counterexample

I checked an exact counterexample in rational arithmetic and reproduced it in R. Let $k=n-m$, and suppose every cell of the partition is wider than $2e$, so that the shifted points select the same order statistics as (8). Let $a_1$ be the smallest outcome under treatment, and let $b_*$ be the smallest outcome under control for the upper bound and the largest for the lower bound. Since only the first weight changes, the output of the printed function differs from the estimate (9) by exactly

With all outcomes equal to $1$, $N=n=4$ and $m=2$, the randomization variance and both estimates (9) are $0$, and the printed function returns $-4e/3=-1/50331648$ for both bounds; R returns $-1.9868\times10^{-8}$. The table lists this case and three others.