Corrections to: Spillover Effects in Experimental Data

The problem

On p. 309 the chapter defines the marginal potential outcome $y_{ig}(z;\psi)$ by fixing the treatment of unit $i$ at $z$ and averaging its potential outcomes over the treatments of the other group members, each of whom is treated independently with probability $\psi$. The same page separates this from complete random assignment, which fixes the number of treated units. Under Bernoulli assignment the number of treated units in a group is random, and the arm mean $\widehat{y}_g(z;\psi)$, which is the mean of the observed outcomes of the units of group $g$ with treatment $z$, divides by the realized size of the arm. When the outcome of a unit depends on how many other group members are treated, which is the case that Section 16.5 treats, the arm mean then averages the outcomes over a distribution of the other members' treatments that differs from the one in the definition of $y_{ig}(z;\psi)$. Conditioning on a nonzero denominator does not remove the difference. The step to “unbiased” fails for $\widehat{y}_g(z;\psi)$ and $\widehat{y}(z;\psi)$, and with it the claims for $\widehat{\tau}^{D}$, $\widehat{\tau}^{I}$ and $\widehat{\tau}^{To}$. The estimator $\widehat{\tau}^{O}$ uses the mean outcome of all units in a group, has no denominator that depends on the size of an arm, and is unbiased under uniform assignment of fixed positive numbers of groups to saturations.

Evidence an exact counterexample

An exact counterexample, and derivations for the two corrected forms.

The counterexample. I take two groups of four units and set $y_{ig}(\mathbf{z})=\sum_{j\neq i}z_j$, the number of treated group members other than $i$. Own treatment has no effect, and partial and stratified interference hold. One group, chosen with probability $1/2$, has saturation $\psi=1/2$ and the other has $\phi=1/4$. Each unit is then treated independently with the probability of its group. For a unit in a group with treatment probability $p$, each of the three other members is treated with probability $p$ whatever $z$ is, so that $y_{ig}(0;p)=y_{ig}(1;p)=3p$. The group averages of the marginal potential outcomes are $3/2$ at $\psi$ and $3/4$ at $\phi$, both direct effects are $0$, and the indirect, total and overall effects are $3/4$.

Let $K$ be the number of treated units in a group. Every treated unit has outcome $K-1$ and every control unit has outcome $K$. The treated arm mean is $K-1$ when $K\ge 1$, the control arm mean is $K$ when $K\le 3$, and their difference is $-1$ on every assignment for which both are defined.

By the table, I have

while the target of both is $3/2$. At $\phi$ the two conditional expectations are $81/175$ and $84/85$, and the target is $3/4$. Enumerating the 512 assignments of the two stages in exact fractions, I find the expectation of each estimator over the assignments on which it is defined. It is $-1$ for $\widehat{\tau}^{D}(\psi)$ and for $\widehat{\tau}^{D}(\phi)$, whose targets are $0$. It is $224/255$ for $\widehat{\tau}^{I}$ and $37/255$ for $\widehat{\tau}^{To}$, whose targets are $3/4$. It is $3/4$ for $\widehat{\tau}^{O}$, which equals its target.

The Horvitz–Thompson mean. Since $y_{ig}(z;\psi)$ is the expected outcome of unit $i$ given $Z_{ig}=z$, I have

Summing over $i$ and dividing by $n_g\,\psi^{z}(1-\psi)^{1-z}$, I obtain that the Horvitz–Thompson mean in the correction has expectation $(1/n_g)\sum_{i}y_{ig}(z;\psi)$. Note that the expectation is over all assignments, including those with no unit in arm $z$, where the sum is $0$. In the counterexample, when the arm means are replaced by Horvitz–Thompson means, the expectations found by enumeration equal the targets for all five effects.

Complete random assignment. Suppose that exactly $k_z\ge 1$ units of the group receive treatment $z$, all such assignments being equally likely, and let $y^{\mathrm{cr}}_{ig}(z)$ be the expected outcome of unit $i$ given $Z_{ig}=z$ under this design. Since each unit has treatment $z$ with probability $k_z/n_g$ and the denominator of the arm mean is the constant $k_z$, I find

These targets differ from the Bernoulli ones. In the counterexample with two of the four units treated, I have $y^{\mathrm{cr}}_{ig}(1)=1$ and $y^{\mathrm{cr}}_{ig}(0)=2$. The direct effect defined for this design is $-1$, while the direct effect under Bernoulli assignment is $0$.

Groups. Suppose that a fixed number $C\ge 1$ of the $G$ groups receives saturation $\psi$, all choices being equally likely. Each group then has saturation $\psi$ with probability $C/G$. If a quantity computed in group $g$ has expectation $\mu_g$ when the group has saturation $\psi$, its mean over the groups with that saturation has expectation $(1/G)\sum_{g}\mu_g$. Applied to the Horvitz–Thompson means and to the arm means under complete random assignment, this implies the population statements in the correction. Applied to the mean outcome of all units in a group, it implies that $\widehat{\tau}^{O}$ is unbiased.

What survives

The definitions of the marginal causal effects on p. 309 and the estimator of the overall effect are unaffected. The toy example on pp. 310–311 uses complete random assignment in both stages, and Figure 16.6 uses the same two-stage design. For that design the arm means are unbiased for the marginal potential outcomes defined by it. No reported number is known to depend on the claim. What changes is that, under Bernoulli assignment within groups, the printed estimators of the direct, indirect and total effects cannot be called unbiased. No variance estimator for the Horvitz–Thompson means was derived.

The problem

Section 16.4.6 and Table 16.3 concern the contrast $\tau(d_{\mathbf{1}},d_{\mathbf{0}})$ between two exposure conditions: a unit and all of its first-degree neighbors are treated, and a unit and all of its first-degree neighbors are in control. The section motivates it on p. 304 by the difference between 100% and 0% treatment saturation. The package has one exposure mapping for these two conditions, and its documentation describes them in the same way. Its second indicator is exposure<B_size, where exposure is the number of treated units in the closed neighborhood of a unit (the unit and its first-degree neighbors) and B_size is the size of that neighborhood. This indicator is one whenever at least one member of the closed neighborhood is untreated. Then a unit whose closed neighborhood is partly treated is coded all control, and so is a treated unit with one untreated neighbor. The contrast that the function supports is between a closed neighborhood that is entirely treated and everything else.

Every quantity that the package computes for the comparison condition inherits the wrong event: its exposure probability, the inverse-probability weights, the comparison mean and the variance estimates. When the outcomes in a partly treated neighborhood differ from the outcomes under full control, the Horvitz–Thompson contrast built on the function’s two indicators can estimate a different quantity from $\tau(d_{\mathbf{1}},d_{\mathbf{0}})$, and nothing in the output shows it. Whether Table 16.3 was computed with this function is not known, because the script that produced the table is not in the replication files. No entry of Table 16.3 is shown to be wrong. If every unit short of full treatment has its baseline outcome, then the contrast built on the function’s indicators has the same expectation as the intended one.

Evidence an exact counterexample

The evidence is a derivation and an exact counterexample. For the derivation, consider an undirected network without loops, with adjacency matrix $A$, and a treatment vector $\mathbf{z}$ with entries 0 and 1. For each unit $i$, let $B_i=\{i\}\cup\{j:A_{ij}=1\}$ be its closed neighborhood, let $b_i=|B_i|$, and let $s_i=\sum_{j\in B_i}z_j$. Lines 114–117 compute exposure $=s_i$ and B_size $=b_i$. Since $0\le s_i\le b_i$, I have $s_i=b_i$ exactly when every member of $B_i$ is treated and $s_i=0$ exactly when no member is treated. The condition $s_i<b_i$ holds exactly when at least one member is untreated, and it includes every case with $0<s_i<b_i$. Note that the first indicator of the function is correct and that the second is the complement of the first.

For the counterexample, let two units be joined by one tie, so that $b_1=b_2=2$. For the assignment $\mathbf{z}=(1,0)$, I have $s_1=s_2=1$. The function, run unchanged from both copies of the package in the replication files, returns (all treated, all control) $=(0,1)$ for both units, and the documented conditions give $(0,0)$. Let the two units be treated independently with probability $1/2$. Each unit is all treated with probability $1/4$, all control with probability $1/4$, and coded all control by the function with probability $3/4$. Let both units have outcome $Y_i(\mathbf{z})=1+z_1+z_2$, so that $\tau(d_{\mathbf{1}},d_{\mathbf{0}})=3-1=2$. The table lists the Horvitz–Thompson contrast of p. 297 for each assignment, with exact probabilities. Its expectation is $4/3$ with the function’s events and $2$ with the documented events.

With the threshold outcomes $Y_i(\mathbf{z})=1+2\,\mathbf{I}(z_1=z_2=1)$, the contrast is again $2$, and both versions of the estimator have expectation $2$. This is why the defect does not by itself imply an error in Table 16.3. Over all simple undirected networks on one to four labeled units and all treatment vectors, 1098 configurations in total, the function as written gets 2456 of the 4306 unit classifications wrong, and the corrected indicator agrees with a direct check of the treatment of every member of $B_i$ in every case.

What survives

The indicator for a closed neighborhood that is entirely treated is correct, and the package’s mapping into the four conditions $d_{11}$, $d_{10}$, $d_{01}$ and $d_{00}$ is a separate function. The estimators and their theory for a correctly specified exposure mapping, and the argument of Section 16.4.6 for designs that raise the probability of the two full-neighborhood conditions, are unaffected. No entry of Table 16.3 and no ranking in it is shown to be wrong. The function changes, and outputs that use its all-control indicator have to be computed again with the corrected event. The package’s 3-net sampler is the subject of P045-E013.

The problem

Section 16.4.4 asks what happens when the measured network misses some ties of the interference network, and Figure 16.5 illustrates this. The package provides a function intended to remove each tie with probability p and a function that adds ties. Both edit the upper triangle of the adjacency matrix and then copy the upper triangle into the lower triangle with the line above. R lists the cells of each triangle column by column, and in that order the $k$th cell of the upper triangle is in general not the transpose of the $k$th cell of the lower triangle. The matrix that the functions return is in general not symmetric. It can lack entries that were not removed and contain entries that were never in the network. A simulation that removes ties with this function studies the altered network, and not a network from which a random share of the ties is missing.

The printed figure has the pattern that the altered network produces, although this does not show that it was drawn with the defective function. Reading the histograms from the drawing of the page, I find that the binned mean estimate lies about $7.0$ below the true value $14.53$ of $\tau(d_{01},d_{00})$ and $9.6$ below the true value $58.12$ of $\tau(d_{11},d_{00})$ in the column labelled 0.025. The binned gaps are about as large in the columns labelled 0.225 and 0.425. I simulated the stated design on five networks. With the package’s tie-removal function the same jump appears at a deletion probability of 2.5%. With correctly mirrored random deletion, the average changes in the three contrasts, using experiments where both estimates are defined, are less than $0.3$ in magnitude at 2.5%, at most $1.8$ at 22.5% and at most $3.2$ at 42.5%. The printed binned offsets at 2.5% are much larger than any change at that probability in these five simulations. Since the simulations use new networks, and the bins of the figure are about 14.4 wide, this comparison does not measure how far the figure departs from the effect of random deletion on its own network. The script that drew the figure is not in the replication files. The five simulations support the attribution of the figure to the tie-removal function, and they do not establish it; an example below shows that the deletion of particular ties, 2.5% of the total, can produce shifts of this size.

The sentence that opens the section, that the bias of the estimator will increase with the proportion of mismeasured ties, is stated generally, before the figure is introduced, and it needs qualifying: the bias need not increase. In an exact example below, the absolute bias of the Hajek contrast decreases when ties are deleted at random. That example concerns the total bias of the estimator in a small network. It does not show that the simulation of Figure 16.5 behaves otherwise than the chapter describes, or that the part of the bias due to the missing ties decreases.

The words “indicating unbiasedness” describe the centring of the histograms in one figure. Page 299 already says that the Hajek estimator improves efficiency “with a small cost in terms of finite-sample bias”, and Table 16.1, p. 300 (PDF p. 12), reports biases of $0.32$, $-0.14$ and $1.29$ for these three estimators with no missing ties. Read with that qualification, the words say that the figure shows no appreciable bias, not that the estimator is exactly unbiased. They call for a clarification, below, and are not part of the error.

Evidence an exact counterexample

The evidence is an exact counterexample for the code, a derivation for the repair, a reading of the printed figure, new simulations, and two exact examples that mark the limits of the conclusion.

The code. Let $A$ be the adjacency matrix of the network on four units whose only tie joins units 1 and 4. Each of the two functions, called with $p=0$, leaves the upper triangle unchanged and returns the matrix $B$:

R lists the upper cells in the order $(1,2),(1,3),(2,3),(1,4),(2,4),(3,4)$ and the lower cells in the order $(2,1),(3,1),(4,1),(3,2),(4,2),(4,3)$. The fourth upper cell, $(1,4)$, is written to the fourth lower cell, $(3,2)$, and not to $(4,1)$. The matrix $B$ is not symmetric: the entry $(3,2)$ was added and the entry $(4,1)$ was removed. The same $B$ is returned through the package’s function make_adj_matrix_miss_ties with $p=1/40$ and seed 1, for removal and for addition. With $B$ in place of $A$, the exposure mapping of the package moves unit 1 from $d_{01}$ to $d_{00}$ when unit 4 alone is treated, and it moves unit 2 from $d_{00}$ to $d_{01}$ when unit 3 alone is treated. Over all 64 upper triangles on four units, the printed line returns an asymmetric matrix for 32, and the corrected line returns the symmetric matrix for all 64. On the network of ten units that is printed on p. 295, the tie-removal function with $p=9/40$ and seed 56 removes 4 ties from the upper triangle, and its output differs from the input by 9 added and 17 removed entries.

The repair. Let $U$ be the upper triangle after the random edits, with zeros elsewhere. The corrected line writes $m_{ji}$ to each lower cell $(i,j)$ such that $i>j$, and it changes no cell on or above the diagonal. The result is $U+U^{\mathsf T}$, which is symmetric. For a symmetric input $A$ with entries 0 and 1 and zero diagonal, and upper cells $i<j$, I have $0\le U_{ij}\le A_{ij}$ after a removal, so that no entry is added, and $A_{ij}\le U_{ij}\le 1$ after an addition, so that no entry is removed.

The printed figure. Figure 16.5 is a vector drawing, and the heights of its bars give the counts of each histogram; the counts of each of the twelve panels sum to 3000. The third column of the table below is the mean of each histogram minus the position of its dashed line. The bins are about 14.4 wide, so these means are those of the binned estimates.

New simulations. I drew five networks of 400 units and 800 ties with the package’s generator and simulated the design of pp. 299 and 303: 40 treated units, dilated effects, Hajek estimators, 10,000 draws for the exposure probabilities and 3000 experiments. For each network and each deletion probability, I measured the network once with the package’s tie-removal function and once after mirroring the same random deletion decisions correctly. At a deletion probability of 2.5% the tie-removal function deleted between 13 and 21 ties, and its output contained between 766 and 780 entries that were not in the network. The last two columns of the table give the range, over the five networks, of the mean of the defined estimates minus each network’s true value. Averaging changes from no missing ties over experiments where both estimates are defined, random deletion at probability 2.5% moves $\widehat{\tau_H}(d_{01},d_{00})$ by $-0.19$ to $+0.09$, $\widehat{\tau_H}(d_{10},d_{00})$ by $+0.04$ to $+0.25$ and $\widehat{\tau_H}(d_{11},d_{00})$ by $-0.25$ to $+0.10$. The tie-removal function moves them by $-7.22$ to $-6.29$, by $+2.96$ to $+3.41$ and by $-11.53$ to $-10.45$.

Limits. The simulations use new networks and new outcomes, since the network, the outcomes and the script of the printed figure are not available. They do not give the values of a corrected figure. They also do not exclude random deletion as the source of the printed figure. Let 400 units form eighty groups of five, with a tie between every two units of a group. Let the baseline outcome be 250 in two of the groups and 53.2 in the others, so that the mean baseline is 58.12. Suppose that the 20 missing ties, out of 800, are exactly the ties within the two groups with the high baseline. With exact exposure probabilities and conditioning separately on each contrast being defined, an exact calculation yields a bias of $-6.70$ for $\widehat{\tau_H}(d_{01},d_{00})$ and of $-10.39$ for $\widehat{\tau_H}(d_{11},d_{00})$. Configurations arbitrarily close to this example have positive but very small probability under the stated stochastic design.

Bias and the proportion of missing ties. On the ten-unit network of p. 295, let one unit be treated, chosen uniformly, and let the baseline outcome be $5/2$ for unit 2 and $1$ for the other units, with the multipliers of p. 298. Enumerating every assignment and every set of deleted ties in exact arithmetic, I find that the bias of $\widehat{\tau_H}(d_{01},d_{00})$ is $+0.0200$ with no tie deleted, $+0.0091$ with one of the 20 ties deleted at random and $-0.0012$ with two deleted. The absolute bias decreases. This is the total bias of the contrast in one small design: it qualifies the general statement on p. 303 and says nothing about the simulation of Figure 16.5.

What survives

The estimands, the exposure conditions and the estimator formulas do not depend on the two functions. The point of the section, that mismeasured ties can bias the estimators, stands. The column of Figure 16.5 with no missing ties is not in question: the package’s make_adj_matrix_miss_ties function with $p=0$ and type nothing returns the network unchanged, and the printed histograms are approximately centered, in agreement with Table 16.1. No other table or figure is shown to depend on the two functions. What changes is the code, the confidence in reading the three other columns of Figure 16.5 as the effect of random deletion at the stated proportions, and the general statement on p. 303 about increasing bias. The words “indicating unbiasedness” need only the clarification below.

The problem

Table 16.3 compares unit-level randomization with cluster randomization, where the clusters come from 3-net clustering: the centers are at least three hops apart, every unit is within two hops of a center, each unit joins a nearest center, and treatment is assigned by cluster. The package’s sampler builds the centers in a loop. It draws a center from the units that are not yet marked, marks the center and every unit within two hops of it, and repeats until every unit is marked. The draw is sample(which(!marked), 1). When exactly one unmarked unit $n$ remains, which(!marked) is the single number $n$, and R then draws uniformly from $1,\dots,n$ and not from the one remaining unit. Each draw that misses the units within two hops of $n$ writes a center on a unit that is already marked. Unless that unit is already a center, the new center lies within two hops of an earlier one. The loop continues until a draw lands within two hops of $n$.

This is not a rare case in new simulations of the design of Table 16.3. In 50,000 clusterings of five networks of 400 units, the loop reached a single unmarked unit $n>1$ in 92.75% of the clusterings, and the final centers were closer than three hops in 91.17%. The sampler as written returned 83.8 clusters on average, against 60.4 for the repaired sampler, and the estimated probability that the closed neighborhood of a unit is entirely treated, averaged over units, was 0.169, against 0.193. The design that the package draws under the name of 3-net clustering is therefore a different design. If the rows Cluster of Table 16.3 were computed with the package, they are not results for 3-net clustering. The script and the seeds of the table are not in the replication files, and no printed entry is shown to be wrong. In the new simulations the repaired sampler has estimated standard deviations that are lower by 4% to 14%.

Evidence an exact counterexample

The evidence is an exact counterexample, a derivation, and new simulations of the design of Table 16.3.

Counterexample. Let four units form the path $1$–$2$–$3$–$4$, and let each cluster be treated independently with probability $1/2$. A 3-net has either the single center 2, or the single center 3, or the two centers 1 and 4, and the possible treatment vectors are $0000$, $0011$, $1100$ and $1111$. Suppose that the sampler first draws unit 1, which has probability $1/4$. It marks units 1, 2 and 3, and unit 4 alone remains. The next draw is uniform on $\{1,2,3,4\}$. A draw of 1 leaves the center set unchanged, and each of 2, 3 and 4 ends the loop, so that each is the last draw with probability $1/3$. If the last draw is 2, the centers are 1 and 2, at distance one, and the clusters are $\{1\}$ and $\{2,3,4\}$. This partition has probability $1/12$, and its contribution to the probabilities of $1000$ and $0111$ is $1/48$ each. Neither vector is possible under 3-net clustering. Enumerating the sampler in exact arithmetic, I find the probabilities $3/8$, $5/48$, $1/48$, $1/48$, $5/48$ and $3/8$ for $0000$, $0011$, $0111$, $1000$, $1100$ and $1111$ as written, and $3/8$, $1/8$, $0$, $0$, $1/8$ and $3/8$ after the repair. The function, run unchanged from both copies of the package in the replication files using R 4.4.2’s default random-number settings and seed 19, returns $1000$ on this path.

Derivation. Suppose that the loop reaches a single unmarked unit $n>1$. Let $S$ be the set of units $v\le n$ that are within two hops of $n$, with $s=|S|$, and let $c_0$ be the number of earlier centers $v\le n$. Each later draw is uniform on $\{1,\dots,n\}$. A draw in $S$ marks $n$ and ends the loop, and a draw outside $S$ leaves $n$ unmarked, so the number of draws is geometric with mean $n/s$. Every unit other than $n$ is already marked, and a marked unit that is not a center is within two hops of a center, so that it cannot become a center of a 3-net. A draw of an earlier center only overwrites its label. This implies that the final centers form a 3-net exactly when every draw before the last selects an earlier center and the last draw selects $n$. Summing the geometric series, I obtain

For the repair, the loop condition makes the vector of unmarked units nonempty, and sample.int returns a position in that vector, so the new center is unmarked. The marked units are the units within two hops of the earlier centers. Then the new center is at least three hops from each of them. Each pass marks at least the new center, so the loop ends after at most $N$ passes, and at the end every unit is within two hops of a center. This holds for every finite undirected network.

New simulations. I drew five networks of 400 units and mean degree 4 with the package’s generator. On each network I ran the sampler as written and the repaired sampler 10,000 times to average the exposure probabilities conditional on each clustering, and I estimated the contrast $\tau(d_{\mathbf{1}},d_{\mathbf{0}})$ in 3000 further experiments for each sampler and for unit-level randomization, with the package’s dilated outcomes. The two exposure conditions are those of the documentation, a closed neighborhood that is entirely treated or entirely in control (see P045-E011). The first table reports how often the defect occurs and how it changes the design. Over the 50,000 clusterings the sampler as written produced 23.4 more centers on average than the repaired sampler continued from the same state, and 92.7 pairs of centers within two hops of each other.

The second table reports the standard deviations of the two estimators. The repair lowers the estimated standard deviation under cluster randomization by 4.41% to 10.96% for the Horvitz–Thompson estimator and by 6.57% to 13.61% for the Hajek estimator. Under both samplers the ratio of the standard deviation under cluster randomization to the unit-level value is between 0.28 and 0.70 on every network. For comparison, the printed ratios of the Cluster to the Unit standard deviation in Table 16.3 are $61.31/139.65\approx0.439$ and $28.89/52.89\approx0.546$.

The networks, the outcomes and the seeds of Table 16.3 are not available. The simulations measure the defect in the design of the table, and they do not give corrected entries for it.

What survives

The rows Unit of Table 16.3 do not use the sampler. The estimand and the unbiasedness of the Horvitz–Thompson estimator with a correctly specified exposure mapping and the actual, positive exposure probabilities are unaffected. In my simulations under both samplers, the estimated standard deviation under cluster randomization is at most 0.70 of the unit-level value on every network. No printed entry is shown to be wrong. The sampler changes; if the package generated the rows Cluster, they have to be computed again before they are read as results for 3-net clustering. The printed definition of $\epsilon$-net clustering on the same page is the subject of P045-E010.

The problem

The variance estimator that Hudgens and Halloran (2008) propose for a direct effect, and that p. 310 describes, uses the sample variance of the observed outcomes in each arm of each group, and a sample variance needs at least two observations. In the toy design each group has three units, two in one arm and one in the other. One of the two sample variances is then undefined in every group. The function estimates_hierarchical sums the terms that are defined and drops the others without a warning. In each of the two direct-effect variances in the output, the part within groups therefore contains only the terms from the arms with two units. This quantity is not the estimator described, and it does not have the property stated on p. 310: in this design its expectation can fall below the true variance even with constant additive effects, and it does so under the outcome model of the example. The statement concerns the expectation over assignments and not the two printed numbers themselves. With other potential outcomes the omission can also bias the reported variance upward.

Evidence an exact counterexample

I checked the package output on data consistent with the example, an exact counterexample and a derivation of the bias.

The printed output. Data were found that match the six printed rows and all ten printed outputs to every printed digit. On these data both versions of estimates_hierarchical that I examined return the printed output with no warning. The six arms with one unit have undefined sample variance. The two direct-effect variances in the output are $0.695937+0.016616=0.712553$ and $0.012457+0.036092=0.048549$, where the first term is the part between groups and the second is the part within groups from the arms with two units.

The counterexample. In each of the six groups let the units have $y_{ig}(0;s)=x_i$ and $y_{ig}(1;s)=x_i+1$ at both saturations $s$, with $(x_1,x_2,x_3)=(1,2,3)$. Unit effects are constant and additive with $d=1$, and partial and stratified interference hold. The design has $20\cdot 3^{6}=14{,}580$ equally likely assignments. Summing over them in exact fractions, I obtain the following values.

The bias. Fix a saturation, and let $G$ be the number of groups. At the first stage, a fixed number of at least two groups is chosen uniformly for this saturation. The complete random assignments within groups at this saturation are independent across groups and independent of the first-stage selection. In group $g$, with $n$ units of which $m$ are treated, let $S^{2}_{a,g}$, $S^{2}_{b,g}$ and $S^{2}_{\delta,g}$ be the variances, with divisor $n-1$, of the potential outcomes under treatment, of the potential outcomes under control and of the unit effects at this saturation. The difference in means in group $g$ has variance

If the within-group term of the variance estimator has expectation $B_g$ in group $g$, the estimator has bias $G^{-2}\sum_{g}(B_g-V_g)$. When both arms have at least two units, $B_g=S^{2}_{a,g}/m+S^{2}_{b,g}/(n-m)$, and the bias is $G^{-2}\sum_{g}S^{2}_{\delta,g}/n$. Under this design the bias is nonnegative, and it is zero exactly when unit effects are constant within every group; the constants may differ across groups. Otherwise it is positive. In the toy design $G=6$ and $n=3$, with $m=2$ at $\psi$ and $m=1$ at $\phi$. When the term of the arm with one unit is dropped, $B_g=S^{2}_{a,g}/2$ at $\psi$ and $B_g=S^{2}_{b,g}/2$ at $\phi$, and I obtain

Under constant effects $S^{2}_{\delta,g}=0$; the bias at a saturation is negative if outcomes vary within any group at that saturation. Under the outcome model of the example, with $S^{2}_{x,g}$ the variance of the baseline outcomes in group $g$, the two biases are $-(11/8)\,G^{-2}\sum_{g}S^{2}_{x,g}$ and $-(13/6)\,G^{-2}\sum_{g}S^{2}_{x,g}$. On the data consistent with the printed output, the expected reported variances are $0.276$ and $0.161$, and the true variances are $0.344$ and $0.267$; the ratios are $0.80$ and $0.60$. In the other direction, if $y_{ig}(0;s)=0$ and $y_{ig}(1;s)=x_i$ with $(x_1,x_2,x_3)=(1,2,3)$ in every group, the expected reported variance at $\psi$ is $1/9$ and the true variance is $1/18$.

What survives

The five point estimates are unaffected, since an arm mean is defined for an arm with one unit. The variances of the indirect, total and overall effects use sample variances across the three groups at each saturation and are unaffected. Figure 16.6 shows point estimates only, in a design with 75 units per group and arms of 50 and 25. The direct-effect variance guarantee holds under stratified interference when a fixed number of at least two groups is selected uniformly for each saturation, the within-group complete random assignments at a fixed saturation are independent across groups and independent of the first-stage selection, and every arm has at least two units. Under these conditions the bias is zero exactly when unit effects are constant within each group, with possibly different constants across groups, and otherwise positive. What changes is the two direct-effect variances of the example and, for a user of the function, the variance reported for a direct effect in any design that has an arm with one unit.

The problem

On p. 310 the estimator $\widehat{y}(z;\psi)$ is the mean of the group means $\widehat{y}_g(z;\psi)$ over all groups with saturation $\psi$. Section 16.5.2 splits these groups by the saturation of the other group in the tract. Among the groups with saturation $\psi$, let $a$ be the number of groups such that the other group in the tract has saturation $\psi$, and let $b$ be the number such that it has saturation $\phi$. Let $A$ and $B$ be the sums of the group means over these two sets of groups. The estimator is $(A+B)/(a+b)$, which is the mixture of the subgroup means $A/a$ and $B/b$ with weights $a/(a+b)$ and $b/(a+b)$ that the next sentence describes. The display prints $A/a+B/b$, the sum without weights. When both subgroups occur and the outcomes are positive, this exceeds the estimator; when one subgroup is empty, it contains $0/0$, although the pooled estimator is still defined.

Evidence an exact counterexample

A derivation and an exact example. For positive $a$ and $b$, I have

which is positive when $A$ and $B$ are positive. For the example, I take the design of Section 16.5.2, with six groups of 75 units in three tracts, $\psi=2/3$ and $\phi=1/3$. I let every baseline outcome be $1$ and use the outcome multipliers of the package function make_dilated_out_hh_miss: in a group with saturation $\psi$, a treated unit has outcome $9/4$ and an untreated unit $11/8$ when the other group in the tract has saturation $\psi$, and $2$ and $5/4$ when it has saturation $\phi$. Let groups 1, 2 and 3 have saturation $\psi$, with tracts $(1,2)$, $(3,4)$ and $(5,6)$, so that $a=2$ and $b=1$. For treated units the estimator is $(9/2+2)/3=13/6$ and the display gives $9/4+2=17/4$. For untreated units the values are $4/3$ and $21/8$. Of the 20 equally likely choices of the three groups with saturation $\psi$, 8 place one such group in each tract. Then $a=0$ and the first term of the display is $0/0$. On all 20 choices the corrected form equals the estimator of p. 310, and the package function estimates_hierarchical computes that estimator.