Corrections to: A General Method for Detecting Interference Between Units in Randomized Experiments

The problem

The rule uses only the lower-tail probability $p=\Pr(t_0\le t)$ of Equation (2), which includes the observed value. On the lower side the rule rejects when $p<\alpha/2$, and under $H_0$ this event has probability at most $\alpha/2$. On the upper side it rejects when $p>1-\alpha/2$, that is, when $\Pr(t_0>t)<\alpha/2$. This is a condition on the strict upper tail, which leaves out the probability that the reference distribution gives to the observed value. When the reference distribution is discrete, this probability can be large. In particular, the largest value of the statistic has $p=1$ and is always rejected, however probable it is. The rule can then reject with probability greater than $\alpha$ when there is no interference, and the statement that the two-tailed procedure is exact does not hold. Excluding ties does not remove the problem. With $n$ equally likely distinct values, the rule rejects with probability $(2\lceil n\alpha/2\rceil-1)/n$, which is $1/6$ for $n=6$ and $\alpha=0.05$. Not every discrete distribution leads to a rejection probability above the nominal level: for $n=5{,}000$ the same expression equals $249/5000$. The one-tailed $p$ value of Equation (2) is not affected.

Evidence an exact counterexample

The evidence is three exact counterexamples and a derivation of the corrected rule. All probabilities are exact, and $\alpha=1/20$ throughout.

First, I consider five units on a line at the positions $0,4,8,-1,-2$. The units at $0,4,8$ form the fixed subset; they are never treated, and their outcomes are $0,1,2$ under every assignment. Exactly one of the units at $-1$ and $-2$ is treated, each with probability $1/2$, and both have outcome $0$ under every assignment. No outcome depends on any treatment, so $H_0$ holds. The distances from the fixed units to the treated unit are $(1,5,9)$ or $(2,6,10)$, and $\rho=1$ under both assignments. The example therefore satisfies Equation (1) even as it is printed (see P030-E002). Then $p=1>39/40$ under both assignments, and the printed rule rejects with probability 1.

Second, I keep the fixed units and place 42 other units at $-1,-2,\ldots,-41$ and $9$, each with outcome $0$ under every assignment. One of them is treated, each with probability $1/42$. The statistic equals $1$ when the treated unit is at a negative position and $-1$ when it is at $9$. For $\rho=-1$ I have $p=1/42<1/40$, and for $\rho=1$ I have $p=1$. The printed rule rejects both values, again with probability 1, although the statistic is not constant.

Third, the reference distribution of Table 1 arises without interference when the ten outcomes $(1,2,2,2,2,3,2,3,2,1)$ are the same under every assignment. (The article’s example generates these outcomes with interference; the population used here is a separate one.) The table below lists the six equally likely assignments. The printed rule rejects the three with $p=1$, which have total probability $1/2$.

For the corrected rule, let the reference distribution have the distinct values $x_1<\cdots<x_m$ with probabilities $q_1,\ldots,q_m$, and set $C_0=0$ and $C_j=q_1+\cdots+q_j$. Under $H_0$ the observed statistic $t$ is a draw from this distribution. If $t=x_j$, then $p_L=C_j$ and $p_U=1-C_{j-1}$. Fix $u\in[0,1]$. Since the $C_j$ increase, the event $p_L\le u$ consists of the values $x_1,\ldots,x_k$, where $k$ is the largest index with $C_k\le u$, and its probability is $C_k\le u$; the event is empty if there is no such index. In the same way, the event $p_U\le u$ has probability at most $u$. Taking $u=\alpha/2$ with $0<\alpha<1$, I obtain

In the first inequality, I used the union bound. In the same notation, I have $p_U=1-p_L+\Pr(t_0=t)$. The printed rule compares $1-p_L$ with $\alpha/2$ on the upper side, where the corrected rule compares $p_U$.

What survives

The conditional randomization construction and the one-tailed $p$ value of Equation (2) are valid under the same assumptions, and Table 1 with its $p=0.333$ is unchanged. The application reports a lower-tail $p<0.001$ on p. 13, which the correction does not change; taking that value as given, the corrected two-tailed $p$ value is below $0.002$. The correction needs no new assumption. Its cost is that some two-tailed rejections are withdrawn. With strict endpoints, every rejection by the corrected rule is also a rejection by the printed rule, and in the first example no rejection remains. The article reports two-tailed rejection rates for its simulations (83% on p. 11, and Figure 2). The simulation code is not available, and how these rates change under the corrected rule has not been established.

The problem

Figure 2 reports, for each of the three models, how often the test detects interference as $N$ grows. For the linear model the plotted rates lie above the rates that the model produces as it is printed on p. 10: weights $2 - d_{ij}$ out to distance 2, the unit’s own term left out, and $\Theta$ and $\Psi$ each normalized to mean one. At $\alpha = 0.05$ the figure shows about 0.53 at $N = 400$ and 0.83 at $N = 800$. For the printed model, I obtain 0.27 and 0.49. At the four checked values of $N$, the same recalculation comes within 0.024 of every reading for the Gaussian panel and the panel of the third model. The large gap is specific to the linear panel at those values. Either the plotted linear curves or the printed linear model is not what was simulated, and the paper’s simulation code is not available to say which. A cutoff of $\sqrt{2}$ in place of 2 comes closer but overshoots the figure (0.60 at $N = 400$ and 0.88 at $N = 800$). It does not identify the simulated model either. Taken from this panel, the power of the test under linear decay is overstated for the model as printed.

Evidence a simulation

A recalculation by simulation of the model as printed. At each of $N = 100$, 200, 400 and 800, I ran 1,200 independent simulations, each with 5,000 conditional randomizations and the two-tailed rule of p. 7. The design follows the text: units placed uniformly on a square of area $N$, $N/2$ units treated, a random half of the controls fixed, standard normal noise, and the rank correlation between the outcomes of the fixed units and the distance to the nearest treated unit as the statistic. The table sets the rates read from the published figure beside the recomputed rates.

The readings carry a tolerance of 0.01 for diamonds and circles and 0.02 for triangles. At $\alpha = 0.05$ the figure’s value lies above the 95% interval by more than that tolerance at every checked $N$, and at $N = 400$ and 800 this holds at all three values of $\alpha$. With 20,000 conditional randomizations at $N = 400$, in 400 simulations, I find 0.275 at $\alpha = 0.05$ (standard error 0.022). The number of randomizations does not explain the gap. I checked the code against a direct calculation on all 495 conditional assignments of an example with 16 units. This is a statistical comparison and not an exact calculation of power. It does not say whether the figure or the formula departs from what was simulated.

What survives

The test and its construction do not use these simulated rates. The Gaussian simulation of Figure 1, the other two panels of Figure 2 and the voter-turnout application do not use the linear model. The recomputed linear rates still rise across the four checked values of $N$, supporting the qualitative trend of increasing detection. For the printed linear model, at $\alpha = 0.05$ and $N = 400$ the estimated rate is 0.27 where the figure shows 0.53. My recalculations stop at $N = 800$ and do not test the remark that the rate appears to converge to 100%.

The problem

The hypothesis of no interference, $H_{0''}$ on p. 6, concerns outcomes: the outcome of a unit does not change when its own treatment does not. Since every assignment in $\Omega'$ leaves the treatment of the fixed units unchanged, $H_{0''}$ implies that $Y_{\mathit{fixed}}$ is the same for all $Z\in\Omega'$. It does not imply that the statistic is the same. The statistic $t_Z$ depends on $Z_{\mathit{variant}}$ as well as on $Y_{\mathit{fixed}}$, for example through $D_{\mathit{nearest}}$, and it can vary over $\Omega'$ whether or not there is interference; the two sentences before the display say as much. Equation (1) does not follow from them, and it is not the hypothesis that the procedure tests. If Equation (1) held as printed, the reference distribution would be a point mass at the observed value and the $p$ value of Equation (2) would always equal 1. The article itself uses the hypothesis on outcomes. Table 1 lists three different values of the statistic under $H_0$, and p. 10 says that “in each randomization, we are assuming $H_0$, so $Y_{\mathit{fixed}}$ remains constant across randomizations”.

Evidence an exact counterexample

The evidence is a counterexample computed in exact arithmetic. I consider ten units at the positions $1,\ldots,10$, of which one is treated, each with probability $1/10$. Let the outcome of unit $i$ be $b_i$ under every assignment, where $b=(1,2,2,2,2,3,2,3,2,1)$. No outcome depends on any treatment, so there is no interference. Fix the units $1,8,9,10$ and suppose that unit 7 is treated. Then $\Omega'$ consists of the six assignments that treat one of the units $2,\ldots,7$, each with conditional probability $1/6$, and $Y_{\mathit{fixed}}=(1,3,2,1)$ under all six. The Spearman correlation between $Y_{\mathit{fixed}}$ and the distance to the treated unit is $-1/\sqrt{90}$ when the treated unit is 2, 3 or 4, it is $-5/6$ when the treated unit is 5, and it is $-3/\sqrt{10}$ when the treated unit is 6 or 7. These are the values $-0.105$, $-0.833$ and $-0.949$ of Table 1. Since $1/90\ne 25/36$, I have $t_Z\ne t_{Z'}$ for two assignments in $\Omega'$, although the outcomes do not depend on the assignment. Counting the two assignments with $t_Z\le -3/\sqrt{10}$, I obtain $p=2/6$, which is the article’s $0.333$.

What survives

The correction adds no assumption and changes no number. The hypothesis $H_{0''}$ and the conditioning on $Z_{\mathit{fixed}}$ are not affected, and Table 1 and $p=0.333$ are already computed under the hypothesis on outcomes. The simulations and the application are described with a reference distribution that varies over the assignments, which agrees with the corrected hypothesis; their numbers were not recomputed for this correction.

The problem

The sentence names the Pearson correlation of the ranks, which divides the covariance by the square root of the product of the variances. The displayed expression divides by the product itself, and it is not a correlation. The next sentence is too broad when tied values are given average ranks. A relationship that is monotone but not strictly monotone can produce ties in one variable and not in the other, and then $|\rho|<1$.

Evidence a derivation

The evidence is a recalculation from Table 1 and a short derivation. In the last row of Table 1, the ranks of $D_{\mathit{nearest}}=(6,1,2,3)$ are $(4,1,2,3)$ and the ranks of $Y_{\mathit{fixed}}=(1,3,2,1)$ are $(3/2,4,3,3/2)$. With divisor $n$, their covariance is $-9/8$ and their variances are $5/4$ and $9/8$. The printed expression gives $(-9/8)/(45/32)=-4/5$, and $-3/5$ with divisor $n-1$. The Pearson correlation is $(-9/8)/\sqrt{45/32}=-3/\sqrt{10}$, which is the value $-0.949$ in the table. All six rows of Table 1 agree with the formula that has the square root.

For the second sentence, let $D=(1,2,3)$ and $Y=(0,0,1)$, so that $Y$ is a nondecreasing function of $D$. The average ranks are $(1,2,3)$ and $(3/2,3/2,3)$, and $\rho=\sqrt{3}/2$. The case of equality in the Cauchy–Schwarz inequality implies that $|\rho|=1$ exactly when the centered vectors of ranks are proportional.

What survives

Table 1, $p=0.333$ and the test itself use the Pearson correlation of the ranks and do not change. I found no reported number from the simulations or the application that is shown to depend on the printed expression.

The problem

For $d>0$ the decay function is $1/(1.5d)^2=4/(9d^2)$, a power of the distance. It falls by a factor of 4 whenever the distance doubles, and it diverges as $d\to 0$, as the article notes. An exponential decay $\exp(-cd)$ is bounded at $d=0$ and falls by a constant factor for each unit of distance. The model is called exponential on pp. 10 and 11 and in the title of the lower-left panel of Figure 2. As described, the simulations do not include an exponential decay. Separately, the sums run over all $j$. For $j=i$ the summand is $(1-\delta_{ii})/(1.5d_{ii})^2=0/0$, which the factor $1-\delta_{ij}$ does not define; the sums meant are over $j\ne i$.

Evidence a derivation

The evidence is a derivation from the printed formula and a check of the pages. Let $k(d)=4/(9d^2)$. Then $k(2d)/k(d)=1/4$ for every $d>0$, and the derivative of $\log k(d)$ is $-2/d$, which is not constant as it is for an exponential decay. The curve in the lower-left panel of Figure 2 agrees with $k(2/3)=1$, $k(1)=4/9$ and $k(2)=1/9$. The page prints no restriction to $j\ne i$.

What survives

The formula and the plotted curve agree on the decay function, and no reported detection rate is shown to change. The simulation code is not available. The correction concerns the name and the notation, and it is not a check of the implementation. The simulations cover Gaussian, linear and inverse-square decay; they do not show how the method behaves under exponential decay.

The problem

In the paper $N$ is the total number of units (p. 5). The text introduces Figure 1 with $N = 400$, of which 200 units are treated and 200 are controls, and the caption of the same figure states $N = 200$.

Evidence a check of the printed page

A check of the two pages. The simulation code and its output are not available, and the two pages do not settle which sample size produced the plot.

What survives

Only the sample size attributed to Figure 1 is in doubt. The plot, the 83% rejection rate and the Kolmogorov–Smirnov $p = 0.661$ are not shown to be wrong.

The problem

The fixed, control and treatment groups are disjoint and together make up the analysis sample, so their counts sum to $N$. A direct computation gives $114{,}629 + 191{,}228 + 38{,}198 = 344{,}055$. The counts are inconsistent by one; the arithmetic does not identify which entries are wrong.

Evidence an exact counterexample

An exact calculation with the printed counts. The log of the source study records 344,084 voters before the stratum was omitted: 114,640 in the three fixed conditions, 191,243 in control and 38,201 in the neighbors condition. Then the printed $N$ implies that 28 voters were omitted, and the printed group counts imply 29. I did not examine the voter-level data, and the omitted stratum and the incorrect counts are not identified.

What survives

The method and the simulations do not use these counts. The discrepancy alone does not show that the estimate of the direct effect, the rank correlation, Figure 3 or the $p$ value rests on a wrong sample. I did not recompute those, since the distances and the analysis code are not available.

The problem

The figure is a histogram of the rank correlation under the null hypothesis. Its horizontal axis is labelled “Rank Correlation”, and p. 13 introduces it as “$T_0$, the approximate distribution of $\rho_0$ associated with $H_0$”. The symbol in the caption is the italic $p$ that the caption of Figure 1 uses for $p$ values.

Evidence a check of the printed page

A check of the page: the caption, the axis label and the sentence on p. 13.

What survives

Only the symbol in the caption changes. The histogram and the results drawn from it are unaffected.

The problem

Two curves in the Gaussian panel carry the same marker, and the legend does not identify the lowest one. Its position does. Since the test rejects less often at a smaller $\alpha$, the lowest curve is the $\alpha = 0.01$ series.

Evidence a check of the printed page

A check of the page. Enlarged, the markers of the lowest Gaussian curve at $N = 100$ to $400$, where the curves are separated, are circles.

What survives

Only the marker is wrong. No plotted value is shown to be wrong, and nothing else in the paper depends on it.

The problem

As printed, each condition compares the treatment of unit $i$ with the treatment of a unit $i'$ in the same vector $Z$, and no unit $i'$ is defined. The condition meant is that unit $i$ has the same treatment under the two assignments $Z$ and $Z'$.

Evidence a check of the printed page

The evidence is a check of the page. The units are indexed by $i\in\{1,2,\ldots,N\}$ on p. 4, and no index $i'$ is introduced. Two lines below the statement of $H_{0''}$, the same condition for the fixed subset is printed with the prime on the vector: $Z_{\mathit{fixed}}=Z'_{\mathit{fixed}}$, $\forall Z,Z'\in\Omega'$.

What survives

Nothing depends on the misprint. The article uses $\Omega'$ and $H_{0''}$ in their intended form in everything that follows.

The problem

The sentence concerns the third model, which the paper calls exponential and whose kernel is the inverse square $1/(1.5 d_{ij})^2$ (see P030-E006). It does not say whether its test is one-tailed or two-tailed, and “approximately 5,000” has no stated tolerance. Since the paragraph before it reports a rejection rate “when using a two-tailed test”, the natural reading is the two-tailed rule of p. 7 at $\alpha = 0.05$.

Under that reading the stated rate is too high. In a recalculation of the design as printed, the two-tailed test detects interference in 93.0% of simulations at $N = 5{,}000$ and in 92.25% at $N = 6{,}000$. With the one-tailed $p$ value of Equation (2), the rates are 94.83% and 94.5%, which neither bear out “over 95%” nor contradict it. At $N = 8{,}000$ the estimate of the two-tailed rate is 97.00%.

Evidence a simulation

A recalculation by simulation of the design printed on pp. 10–11. Each unit has coordinates drawn uniformly on $[0, \sqrt{N}]^2$ and noise $\epsilon_i \sim N(0,1)$. The outcome is

where $\Theta$ and $\Psi$ are each divided by their mean over the $N$ units. Half of the units are treated, and a random half of the controls is the fixed subset. The statistic is the rank correlation between $Y_{\mathit{fixed}}$ and the distance to the nearest treated unit, and each simulation uses 5,000 conditional randomizations, the number that the paper reports for Figure 1. The one-tailed test rejects when the $p$ value of Equation (2) is below 0.05. The two-tailed test rejects when it is below 0.025 or above 0.975.

At $N = 5{,}000$ and $N = 6{,}000$ the exact 95% intervals for the two-tailed rate are $[0.907, 0.949]$ and $[0.892, 0.947]$, and both end below 0.95. Testing the two counts together, 927 rejections in 1,000 simulations, with an exact binomial test against a rate of at least 0.95 at both values of $N$, I find $p = 0.001$. The one-tailed intervals, $[0.927, 0.965]$ and $[0.918, 0.965]$, contain 0.95. I checked the code against an exhaustive enumeration of the 84 conditional assignments of a small example.

The original simulation code and output are not available. The rates above are estimates from this recalculation, with the Monte Carlo error stated, and three values of $N$ do not establish the rate at every larger $N$.

What survives

The method, the worked example, the Gaussian result and the plotted curves of Figure 2 do not depend on this sentence. The application reports $p < 0.001$, which rejects under either tail rule. Only the sentence itself changes: near $N = 5{,}000$ the detection rate of the two-tailed test under this model is about 93%, and the one-tailed rate is not shown to exceed 95%.

Intended meaning

The numerator is written as the size of a set of values of the statistic, and a set in the usual sense does not repeat equal values. The count meant is over assignments. The paragraph before the display builds $T_0$ from one $t_0$ for each $Z\in\Omega'$ and calls it a distribution, and the worked example counts “2 of the 6 conditional randomizations” on p. 10, although the six assignments of Table 1 have only three distinct values of $\rho_0$. Read in this way, the display is the lower-tail probability, and I obtain the article’s $p=2/6=0.333$. Counting distinct values would give $1/6$.

Intended meaning

The exclamation mark is printed before the existential quantifier. This is not the usual order for the symbol of unique existence, and the mark could be taken for a negation. The article defines no such convention, and Table 1 lists one $\rho_0$ for each of the six assignments. The sentence means that each $Z\in\Omega'$ determines exactly one rank correlation under $H_0$, and nothing false is printed.