Journal article P024

Corrections to: Field Experimental Designs for the Study of Media Effects

Donald P. Green, Brian R. Calfano, and P. M. Aronow (2014). Field Experimental Designs for the Study of Media Effects. Political Communication 31(1):168–180.

DOI: 10.1080/10584609.2013.828142
Audited edition: Political Communication, 2014, 31(1):168–180; 14 pages in the audited file
SHA-256 of the audited file: 3bdff39a04294d8a5e1cd0016d214e7f9fbcaf445676cdb53e53990f9e0cad09
Permanent address: https://pmaronow.github.io/llm-audit/works/P024/

Summary

The article reviews designs for randomized field experiments on media effects: individual and clustered assignment, restricted assignment, stepped-wedge and crossover designs, and a single-medium design with several messages and outcomes. It has no theorems, tables or data analysis of its own. 19 entries are classified as errors (19 Minor), and 3 are clarity notes. Two of the errors are in statements about methods (P024-E002 and P024-A002), and one is an estimate misquoted from a cited study (P024-E001). The other 16 are one misprinted verb (P024-E015) and 15 errors in citations and the reference list.

What was and was not checked

The article has no formulas, theorems, tables, figures or original data analysis. Its technical content consists of verbal statements about estimators and tests, and of descriptions of cited studies. I checked the following statements, in small exact examples or against the cited source, and they held up.

p. 170: under uniform complete individual randomization with nonempty arms and assumptions (a)–(c), the difference in means is unbiased for the average effect in the subject pool. Enumerating every allocation, I find that the expected difference in means equals the average effect in a six-unit example with heterogeneous effects, and in an example with two of four units treated.
p. 171: under uniform complete cluster randomization with nonempty arms, the individual-level difference in means can be biased when cluster sizes are unequal. In one example with cluster sizes $(1,1,2,6)$ its expectation is $547/504$ against an average effect of $7/5$ , and in another with sizes $(1,2,3,6)$ it is $191/504$ against $1/2$ . The estimator is unbiased when the clusters have equal sizes. With uniform fixed treatment counts within blocks, blocking on cluster size and weighting the block estimates by their numbers of subjects removes the bias when each block holds clusters of one size and both arms are nonempty. The consistency sentence of the same paragraph is the subject of P024-E002.
pp. 171–172: standard errors computed as if individuals had been assigned can understate the uncertainty of a clustered design. In a worked case with four markets of 1 million people and no treatment effect, the standard deviation of the estimate over the six allocations is $0.0913$ . The standard error computed as for individual assignment is $0.0005$ . The wording of the illustration is the subject of P024-N002.
p. 172: a randomization test of the sharp null hypothesis is valid in the four-market design. With six equally likely allocations its usual inclusive two-sided $p$ -value based on the absolute difference in means cannot be smaller than $1/3$ , and the article does not claim otherwise.
p. 172: when clusters are assigned and one outcome is observed in each cluster, the methods for a between-subjects design apply to the cluster-level effect.
p. 173 and Note 4: with uniform randomization of fixed, nonzero treatment and control counts within each block, the average of the block estimates weighted by block size equals the inverse-probability-weighted estimator and is unbiased. The pooled comparison can be biased when the probability of treatment differs across blocks.
p. 173: the counts of the rollout are consistent. The weekly additions $5+7+4=16$ cover 16 of the 18 experimental markets. With the two excluded markets, the 18 experimental markets make the 20 markets of pp. 169, 173 and 174. The weekly counts 5, 7 and 4 and the exclusion of the two largest markets from the randomization agree with Gerber et al. (2011).
p. 174: assigning each of six organizations to a different week gives $720$ orderings. Under uniform randomization over these orderings, a test of the global sharp null hypothesis, including no subsequent effects, is valid whatever the differences between weeks. The sentence on the assumptions of a within-subjects design is the subject of P024-A002.
Note 1: for a fixed set of strata with positive limiting population shares, uniformly bounded potential outcomes and a common treatment fraction bounded away from zero and one, complete randomization followed by post-stratification has the same first-order variance as the blocked design.
The roadmap on p. 170 matches the sections, Notes 1–5 attach to the right sentences, and the counts and durations that are repeated across pages agree.
The reference list: 58 of the 64 entries are cited in the text or the notes. Every citation in the text has an entry, apart from the two years corrected in P024-E005 and P024-N001. The six entries that are not cited are the subject of P024-N007.

The checks had the following limits. I had no supplement, correction or replication file for this article, and it reports no result of its own that could be recomputed. Of the studies that the article describes, I read Bond et al. (2012) with its supplement, Gerber et al. (2011), the description of the allocation in Green and Vavreck (2008), the listed article of Arceneaux and Nickerson (2009), and the abstract of their second article of that year. For the other reference entries I correct, I compared titles, names, volumes, page ranges and one DOI with published bibliographic records or the published articles. I did not check the substantive results in the remaining sources: Broockman and Green (in press), the studies of Paluck and of Panagopoulos and Green, Gerber and Green (2012), and the working papers Aronow, Calfano and Green (2013) and Paluck et al. (2013). I checked the descriptions of those studies only for consistency within the article. Two details are unresolved. Gerber et al. (2011) give political importance as the reason for excluding the two largest markets and do not mention a demand for a high volume of ads there. This detail of p. 173 is therefore neither confirmed nor shown to be false. The sources also disagree on how the 85 cable systems of Green and Vavreck (2008) were divided between treatment and control (see P024-E003). I have not established that the rest of the article is correct.

Headline claims and related corrections

Field experimentation is a promising but seldom used way to study media effects, combining the internal validity of randomized trials with realistic treatments and unobtrusive outcome measurement; the practical obstacle is usually the cooperation of research partners, and researchers can keep the advantages of random assignment by adapting the design to the constraints of partners and of the media environment.
Abstract and introduction, pp. 168–170; conclusion, pp. 176–177. The article has no numbered results, tables or figures; the designs are supplied by the sections on pp. 170–175 and the roadmap paragraph on p. 170.
See P024-E005, P024-N001, P024-E001.
In the basic between-subjects design (individual assignment with identical probability), if (a) the assignment is implemented as planned and outcomes are observed for all subjects, (b) the groups differ systematically only in the intended treatment and (c) there is no interference between subjects, the difference in group means is an unbiased estimator of the average causal effect in the subject pool.
Section 'Between-Subjects Design', pp. 170–171; Notes 1–3
See P024-E001.
Under clustered assignment with clusters of different sizes, comparing treatment and control averages is no longer unbiased for the individual-level average effect; the difference-in-means estimator is consistent as the number of clusters grows; the bias can matter with a few dozen clusters; and blocking on cluster size mitigates it.
Section 'Between-Subjects Clustered Design', p. 171, first and second paragraphs; restated in the conclusion, p. 176
See P024-E002, P024-E004, P024-E003, P024-N001.
Standard errors computed as if individuals had been assigned are grossly misleading under clustered assignment; sampling variability must be estimated in a way that respects the clustering (cluster-robust, random-effects, hierarchical or design-based methods); such estimates are unreliable with few clusters, and randomization inference is an option in such settings; when the outcome is itself measured at the cluster level, the usual methods for a between-subjects design apply.
Four-media-market example and the paragraphs that follow, pp. 171–172; conclusion, p. 176
See P024-N002, P024-N003.
Restrictions imposed by research partners or budgets on who may be treated can be accommodated by excluding units from the randomization and blocking or otherwise restricting assignment among the rest; this narrows the scope of the experiment and requires an analysis that respects the blocks, the overall estimate being in effect a weighted average of block-level estimates (inverse-probability weighting).
Section 'Restricted Random Assignment', pp. 172–173; Note 4; Gerber et al. (2011) and Panagopoulos and Green (2008) examples
Stepped-wedge and crossover (delayed-treatment) designs, in which treatment is rolled out in randomly ordered stages and outcomes are tracked over time, gain precision from repeated observation of each cluster and ease partners' reluctance to withhold treatment; the paper notes that the crossover design limits what can be learned about long-term outcomes because eventually both groups receive treatment.
Section 'Crossover and Stepped-Wedge Designs', pp. 173–174; Gerber et al. (2011) and Paluck (2009) examples; roadmap, p. 170
When only a single media outlet is available, a single-medium multi-outcome crossover design (several messages, each with its own outcome measure, aired in randomly assigned periods) is stronger than a single-message within-subjects design with randomized timing, permits randomization tests of the sharp null of no effect and rank tests, and improves on before-after comparisons whose timing is not randomized.
Section 'Single-Medium Multi-Outcome Crossover Design', pp. 174–175; Aronow, Calfano and Green (2013) and Paluck et al. (2013) examples
See P024-A002.
Sustaining a research partnership rests on three things: building the partner's motivation for a randomized study, reducing its perceived costs and risks (resources, staffing, pilot studies), and formalizing the relationship in a memorandum of understanding under which results are made public, with embargo and anonymity as safeguards for the partner.
Section 'Building and Sustaining Research Collaborations', p. 175; Note 5
Randomized field experiments on media effects are still few, but momentum is growing: low-cost media units and online clusters make clustered designs feasible, and tools now exist for calculating statistical power and for handling non-compliance and attrition.
Section 'Conclusion', pp. 176–177; citations to Raudenbush et al. (2011), Angrist, Imbens and Rubin (1996), and Gerber and Green (2012, chapters 5–7)
See P024-E008, P024-N007, P024-E014, P024-E016, P024-E010, P024-E011, P024-E012, P024-N008.

Corrections

Minor (19)

Real but inconsequential, for example a misprint that does not change a result.

P024-E001 Minor

The effect on turnout attributed to Bond et al. (2012) should read approximately 0.4 percentage points, not approximately 0.7: the cited article reports 0.39.

Section “Between-Subjects Design”, second paragraph on p. 171 (PDF p. 5), sentence beginning “Public voting records revealed”

As printed

“In order to estimate the average causal effect of an application that both invited Facebook users to indicate whether they had voted and showed which of their friends had voted, Bond et al. simply compared those assigned to receive the application to those who were assigned not to receive it. Public voting records revealed a significantly higher voting rate among the treatment group, with an average treatment effect of approximately 0.7 percentage points. A different application that invited randomly selected Facebook users to indicate whether they had voted but did not indicate whether their friends had voted had no apparent effect.”

The problem

The sentence gives the size of the direct effect that Bond et al. (2012) estimated. This effect is the difference in turnout, taken from public voting records, between users assigned to the application that showed which friends had voted and users not assigned to it. Bond et al. call that application the social message, and they call the application that did not show the votes of friends the informational message. They report the difference between users who received the social message and users who received no message as 0.39 percentage points, with standard error 0.17 percentage points and $P=0.02$ . They report the same estimate when the comparison group is the users who received the informational message. The printed value is larger by 0.31 percentage points, a factor of $70/39\approx 1.8$ . It is not a rounding of the reported estimate, which rounds to 0.4. A different comparison group does not account for it, since both comparisons give 0.39. The estimates of Bond et al. for the friends of treated users concern other comparisons, and the article discusses them separately, in the next paragraph and in Note 2. It is not established where the value 0.7 came from.

Evidence a derivation

I checked the cited article and compared the two values exactly. On p. 296 of Bond et al. (2012), the paragraph on validated voting states that users “who received the social message were 0.39% (s.e.m., 0.17%; $t$ -test, $P=0.02$ ) more likely to vote than users who received no message at all”, and that the difference between the social message and the informational message was also 0.39%. Figure 1 of that article and p. 8 of its supplementary information agree. The table lists the three reported comparisons.

Differences in validated turnout reported by Bond et al. (2012), in percentage points, and the value printed in the article.
Comparison	Estimate	Standard error	$P$
Social message against no message	0.39	0.17	0.02
Social message against informational message	0.39	0.17	0.02
Informational message against no message	0.00	0.28	0.98
As printed on p. 171 of the article	0.7

I found no overall direct estimate of 0.7 percentage points in the article of Bond et al. or in its supplement. The numeral 0.7 occurs in Table S6 of the supplement (p. 20), where it is an interaction threshold and not an average effect on turnout. A value that rounds to 0.39 lies in $[0.385,0.395)$ , and a value that rounds to 0.7 lies in $[0.65,0.75)$ ; the two intervals are $0.255$ apart.

What survives

The qualitative statements of the paragraph stand. Turnout was higher among users who received the social message, and the informational message had no apparent effect. Only the size of the effect changes. The value 0.7 occurs once in the article, and I found no later numerical use of it.

Corrected

“Public voting records revealed a significantly higher voting rate among the treatment group, with an average treatment effect of approximately 0.4 percentage points.”

The estimate reported by Bond et al. (2012, p. 296) is 0.39 percentage points, with standard error 0.17 percentage points and $P=0.02$ , for users who received the social message compared with users who received no message.

Crop of the end of the paragraph on the experiment of Bond et al. A red box marks the words 'approximately 0.7 percentage points.' in the sentence 'Public voting records revealed a significantly higher voting rate among the treatment group, with an average treatment effect of approximately 0.7 percentage points.' The next sentence ends 'had no apparent effect.' — Section “Between-Subjects Design”, p. 171 (PDF p. 5). The marked words give the size of the effect that the article attributes to Bond et al. (2012).

Crop of six lines of the cited article by Bond et al. A red box marks the figure 0.39 percent in the sentence 'who received the social message were 0.39 percent (s.e.m., 0.17 percent; t-test, P = 0.02) more likely to vote than users who received no message at all.' The following sentence says that the difference in voting between those who received the social message and those who received the informational message was 0.39 percent (s.e.m., 0.17 percent; t-test, P = 0.02). — The cited article, Bond et al. (2012), *Nature* 489, p. 296, left column. The marked estimate is the difference in validated turnout between users who received the social message and users who received no message.

Dot plot with a horizontal axis from 0 to 0.8 percentage points of validated turnout and three rows. In the row 'Social message minus no message' a blue circle at 0.39 is the estimate reported by Bond et al. and an orange diamond at 0.7 is the value printed in the article; an arrow between them is labelled 'printed value is 0.31 higher'. In the row 'Social message minus informational message' a blue circle is at 0.39. In the row 'Informational message minus no message' a blue circle is at 0.00. The title reads 'The article prints 0.7 percentage points; Bond et al. (2012) report 0.39'. — The value printed on p. 171 beside the three differences in validated turnout that Bond et al. (2012) report, in percentage points.

P024-E002 Minor

The difference-in-means estimator is consistent as the number of clusters grows only under further conditions, the main one being that no cluster keeps a nonvanishing share of all subjects.

Section “Between-Subjects Clustered Design”, second paragraph, p. 171 (PDF p. 5), sentence beginning “The difference-in-means estimator is consistent”

See also under Formal statements (Lean): P024-T4, P024-T5

As printed

“Clustered assignment poses two statistical complications. The first is that when clusters contain different numbers of subjects, the standard estimation approach—comparing treatment and control group averages—no longer gives unbiased estimates of the average treatment effect among all subjects (Middleton, 2008, Middleton & Aronow, 2011). The difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of clusters becomes very large, but bias may be appreciable in studies with only a few dozen clusters of varying size.”

The problem

The sentence says that the difference between the mean outcomes of treated and control subjects converges to the average treatment effect among all subjects as the number of clusters grows. A large number of clusters is not sufficient for this. If one cluster keeps a nonvanishing share of all subjects, then the assignment of that cluster can continue to move the estimate, however many other clusters are added. The article does not say how the clusters grow, and it states no condition that excludes such a cluster. Under the usual regularity conditions for a law of large numbers the sentence is correct, and the correction supplies them.

Evidence an exact counterexample

The evidence is an exact counterexample to the sentence as printed and a derivation of the corrected statement.

Counterexample. Fix an integer $K\ge 2$ . The population has one cluster of $K^2$ subjects and $K-1$ clusters of one subject each. Every subject in the large cluster has potential outcomes $Y(0)=1/2$ and $Y(1)=3/4$ , and each of the other subjects has $Y(0)=1/4$ and $Y(1)=1/2$ . Every subject has treatment effect $1/4$ , so that the average treatment effect is $\tau=1/4$ for every $K$ . All outcomes lie strictly between $0$ and $1$ , all are observed, and there is no interference. The share of the subjects in the large cluster is $K^2/(K^2+K-1)$ , which tends to $1$ .

Let $\widehat\tau$ denote the difference in means, and let $q$ denote the number of one-subject clusters in the same arm as the large cluster. Suppose that both arms are nonempty, so that $0\le q\le K-2$ and the other arm holds only one-subject clusters. If the large cluster is treated, then the mean outcome of the treated arm is $((3/4)K^2+q/2)/(K^2+q)$ and that of the control arm is $1/4$ . If the large cluster is in control, then the two means are $1/2$ and $((1/2)K^2+q/4)/(K^2+q)$ . Subtracting, I obtain

\begin{aligned} \widehat\tau&=\frac14+\frac14\,\frac{K^2}{K^2+q}&&\text{if the large cluster is treated,}\\ \widehat\tau&=\frac14-\frac14\,\frac{K^2}{K^2+q}&&\text{if it is in control.} \end{aligned}

In both cases

|\widehat\tau-\tau|\ge\frac14\,\frac{K^2}{K^2+K-2}>\frac18 .

Suppose first that the $K$ clusters are assigned independently, each to treatment with probability $1/2$ . Then both arms are nonempty with probability $1-2^{1-K}$ , and I obtain $\mathbb{P}(|\widehat\tau-\tau|>1/8)\ge 1-2^{1-K}\to 1$ . Suppose next that $K$ is even and that exactly $K/2$ clusters are treated, all such allocations being equally likely. Then $q=K/2-1$ , and $\widehat\tau=1/4\pm r_K/4$ with probability $1/2$ each, where $r_K=K^2/(K^2+K/2-1)$ . In this design the estimator is unbiased for every $K$ , its variance $r_K^2/16$ tends to $1/16$ , and it is not consistent. For $K=6$ the twenty allocations give the estimate $37/76$ ten times and $1/76$ ten times. The table lists exact values for $K\le 12$ , found by enumerating every allocation in rational arithmetic.

The counterexample for $K\le 12$ . Second column: the variance of $\widehat\tau$ when $K/2$ clusters are treated; in this design $\widehat\tau$ has mean $1/4$ and differs from $\tau=1/4$ by more than $1/8$ in every allocation. Third and fourth columns: independent assignment with probability $1/2$ ; the smallest error is taken over the allocations with both arms nonempty.
$K$	Variance of $\widehat\tau$	$\mathbb{P}$ (both arms nonempty)	Smallest $\|\widehat\tau-\tau\|$
2	$1/16$	$1/2$	$1/4$
4	$16/289$	$7/8$	$2/9$
6	$81/1444$	$31/32$	$9/40$
8	$256/4489$	$127/128$	$8/35$
10	$625/10816$	$511/512$	$25/108$
12	$1296/22201$	$2047/2048$	$18/77$
Limit	$1/16$	$1$	$1/4$

Derivation of the corrected statement. For each $K$ , let the population consist of $K$ clusters of sizes $n_1,\dots,n_K$ , and set $N=\sum_j n_j$ and $w_j=n_j/N$ . Let $\bar Y_j(z)$ denote the mean potential outcome of cluster $j$ under condition $z\in\{0,1\}$ , and set $\mu_z=\sum_j w_j\bar Y_j(z)$ , so that the average treatment effect is $\tau_K=\mu_1-\mu_0$ . Let $Z_j$ denote the treatment indicator of cluster $j$ , and define

A=\sum_j Z_jw_j\bar Y_j(1),\qquad C=\sum_j(1-Z_j)w_j\bar Y_j(0),\qquad D=\sum_j Z_jw_j .

Then the difference in means is $\widehat\tau_K=A/D-C/(1-D)$ . Suppose that there exist constants $B$ and $\eta>0$ , not depending on $K$ , such that every potential outcome is at most $B$ in absolute value and each cluster is treated with the same probability $p_K$ , where $\eta\le p_K\le 1-\eta$ . Suppose also that $\max_j w_j\to 0$ .

Since the weights $w_j$ are nonnegative and sum to one, I have

\Bigl(\max_j w_j\Bigr)^2\le\sum_j w_j^2\le\max_j w_j ,

so that $\max_j w_j\to 0$ if and only if $\sum_j w_j^2\to 0$ . The expectations of $A$ , $C$ and $D$ are $p_K\mu_1$ , $(1-p_K)\mu_0$ and $p_K$ . Let $x_1,\dots,x_K$ be fixed numbers. If the clusters are assigned independently, then $\operatorname{Var}\bigl(\sum_j Z_jx_j\bigr)=p_K(1-p_K)\sum_j x_j^2$ . If instead $m_K=p_KK$ clusters are treated and all such allocations are equally likely, then $\operatorname{Cov}(Z_j,Z_l)=-p_K(1-p_K)/(K-1)$ for $j\ne l$ , and expanding the variance gives

\operatorname{Var}\Bigl(\sum_j Z_jx_j\Bigr)=p_K(1-p_K)\,\frac{K}{K-1}\Bigl[\sum_j x_j^2-\frac1K\Bigl(\sum_j x_j\Bigr)^2\Bigr]\le 2\,p_K(1-p_K)\sum_j x_j^2 .

In the last inequality, I used $K/(K-1)\le 2$ and dropped a nonnegative term. Since $p_K(1-p_K)\le 1/4$ and $|\bar Y_j(z)|\le B$ , in both designs the variances of $A$ and $C$ are at most $(B^2/2)\sum_j w_j^2$ , and the variance of $D$ is at most $(1/2)\sum_j w_j^2$ . These bounds tend to zero, and Chebyshev’s inequality implies that $A-p_K\mu_1$ , $C-(1-p_K)\mu_0$ and $D-p_K$ converge to zero in probability. On the event $|D-p_K|<\eta/2$ , both $D$ and $1-D$ are at least $\eta/2$ . Using $|\mu_z|\le B$ , I find

\begin{aligned} \Bigl|\frac{A}{D}-\mu_1\Bigr|&\le\frac{2}{\eta}\bigl(|A-p_K\mu_1|+B\,|D-p_K|\bigr),\\ \Bigl|\frac{C}{1-D}-\mu_0\Bigr|&\le\frac{2}{\eta}\bigl(|C-(1-p_K)\mu_0|+B\,|D-p_K|\bigr) \end{aligned}

on this event. I conclude that $\widehat\tau_K-\tau_K\to 0$ in probability. An allocation with an empty arm lies in the event $|D-p_K|\ge\eta/2$ , whose probability tends to zero. The conclusion therefore does not depend on how the estimator is defined on such allocations. Note that in the counterexample $\max_j w_j=K^2/(K^2+K-1)\to 1$ , while the other hypotheses hold.

What survives

The rest of the paragraph stands. The difference in means can be biased when cluster sizes vary, the bias can matter with a few dozen clusters, and blocking on cluster size is a remedy. The discussion of standard errors and of randomization inference, and the distinction between individual-level and cluster-level effects, are unaffected. No later statement of the article is derived from the consistency sentence, and no estimate of a cited study changes. The check does not show that any cited application has a cluster of this kind.

Corrected

“With uniformly bounded potential outcomes and a common cluster treatment probability bounded away from zero and one, the difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of independently assigned clusters becomes very large, provided that no cluster accounts for a nonvanishing share of all subjects, but bias may be appreciable in studies with only a few dozen clusters of varying size.”

A full statement, which also covers complete randomization of the clusters, is as follows. For each $K$ , consider a finite population of $N$ subjects in $K$ clusters of sizes $n_1,\dots,n_K$ , with fixed potential outcomes and no interference between subjects. Suppose that the following four conditions hold.

Every potential outcome is at most $B$ in absolute value, where $B$ does not depend on $K$ .
Either the clusters are assigned to treatment independently with a common probability $p_K$ , or $m_K$ of the $K$ clusters are treated, all such allocations being equally likely, and $p_K=m_K/K$ .
There exists $\eta>0$ , not depending on $K$ , such that $\eta\le p_K\le 1-\eta$ .
$\max_j n_j/N\to 0$ as $K\to\infty$ , or equivalently $\sum_j(n_j/N)^2\to 0$ .

Let $\widehat\tau_K$ denote the difference between the mean observed outcome of the treated subjects and that of the control subjects, and let $\tau_K$ denote the average treatment effect among the $N$ subjects. Then $\mathbb{P}(|\widehat\tau_K-\tau_K|>\varepsilon)\to 0$ as $K\to\infty$ , for every $\varepsilon>0$ . The four conditions are sufficient, and I do not claim that they are necessary. A growing number of clusters alone is not sufficient when one cluster keeps a nonvanishing share of the subjects.

Crop of six lines of the paragraph that begins 'Clustered assignment poses two statistical complications'. Red boxes mark the words 'difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of clusters becomes very large,' which run over two lines. The sentence continues 'but bias may be appreciable in studies with only a few dozen clusters of varying size.' — Section “Between-Subjects Clustered Design”, second paragraph, p. 171 (PDF p. 5). The marked clause states consistency with no condition other than a large number of clusters.

P024-E005 Minor

The citation “Arceneaux (2009)” on p. 168 should read “Arceneaux (2010)”, the year of the review in the reference list.

Introduction, first paragraph, second sentence, p. 168 (PDF p. 2); reference list, p. 177 (PDF p. 11)

As printed

p. 168: “A vast research literature has accumulated, but as Arceneaux (2009) notes in his recent review, this literature is hampered by methodological shortcomings.”

Reference list, p. 177: “Arceneaux, K. (2010). The benefits of experimental methods for the study of campaign effects. Political Communication, 27, 199–215.”

The problem

The year in the text matches no entry in the reference list. The only work listed under Arceneaux alone is the review of 2010. The only Arceneaux entry dated 2009 is Arceneaux and Nickerson (2009), which is a different work and is cited once, on p. 172.

Evidence a check of the printed page

This is a check of the page. I read p. 168 and the reference list on p. 177, and I looked at every occurrence of the name in the article. I did not compare the year of the review with a source outside the article. The article alone does not show whether a review of 2009 was meant and its entry omitted.

What survives

Everything else is unaffected. No statement depends on the year.

Corrected

p. 168: “… as Arceneaux (2010) notes in his recent review …”

Crop of five lines of the first paragraph of the article. A red box marks the citation 'Arceneaux (2009)' in the sentence 'A vast research literature has accumulated, but as Arceneaux (2009) notes in his recent review, this literature is hampered by methodological shortcomings.' — Introduction, first paragraph, p. 168 (PDF p. 2). The marked citation gives the year 2009.

Crop of seven lines of the reference list. A red box marks 'Arceneaux, K. (2010).' at the start of the entry 'The benefits of experimental methods for the study of campaign effects. Political Communication, 27, 199-215.' The next entry is by Arceneaux and Nickerson, is dated 2009 and has the title 'Who is mobilized to vote? A re-analysis of 11 field experiments.' — Reference list, p. 177 (PDF p. 11). The marked entry is the only work listed under Arceneaux alone, and it is dated 2010. The entry below it, dated 2009, has a second author.

P024-E006 Minor

The DOI in the entry for Broockman and Green (in press) is missing a digit and should read 10.1007/s11109-013-9239-z.

References, p. 178 (PDF p. 12), entry for Broockman and Green (in press): the DOI

As printed

Broockman, D. E., & Green, D. P. (in press). Do online advertisements increase political candidates’ name recognition or favorability? Evidence from randomized field experiments. Political Behavior. doi: 10.1007/s1109-013-9239-z

The problem

The journal code in the DOI is printed as s1109, and one digit 1 is missing from it.

Evidence a check of the printed page

The evidence is a check of the page against the publisher’s record. The record of Springer for the article gives the DOI as 10.1007/s11109-013-9239-z.

What survives

The article remains identifiable from its authors, title and journal, and no statement about the study in the text depends on the DOI.

Corrected

Crop of the three-line entry for Broockman and Green (in press) and the two-line entry for Chong and Druckman (2010) below it. A red box marks 's1109' in the DOI, which is printed as 10.1007/s1109-013-9239-z. — References, p. 178 (PDF p. 12), first entry on the page. The marked journal code in the DOI is missing a digit.

The evidence is a check of the page against the record of SAGE for the article.

What survives

The year, volume and pages identify the article, and the description of the study on p. 169 does not quote the title.

Corrected

Paluck, E. L. (2010). Is it better not to talk? Group polarization, extended contact, and perspective taking in Eastern Democratic Republic of Congo. Personality and Social Psychology Bulletin, 36, 1170–1185.

Crop of four lines of the reference list: the last line of the entry for Paluck (2009) and the three-line entry for Paluck (2010). Red boxes mark the word 'contacts', the hyphenated word 'perspective-taking', which is broken over two lines, and the word 'the' before 'Eastern Democratic Republic of Congo'. — References, p. 179 (PDF p. 13), entry for Paluck (2010). The three marked items differ from the published title.

P024-E010 Minor

The surname Cappella is misspelled “Capella” in one citation and two reference entries.

Conclusion, p. 176 (PDF p. 10), first paragraph; reference list, p. 179 (PDF p. 13), entries Kang et al. (2009) and Lee et al. (2011)

As printed

Conclusion, p. 176: “the effects of media campaigns designed to discourage the use of drugs (Fishbein et al., 2002; Kang, Capella, & Fishbein, 2009; Werb et al., 2011)”

Reference list, p. 179: “Kang, Y., Capella, J. N., & Fishbein, M. (2009). The effect of marijuana scenes in anti-marijuana public service announcements on adolescents' evaluation of ad effectiveness. Health Communication, 24, 483–493.”

Reference list, p. 179: “Lee, S., Capella, J. N., Lerman, C., & Strasser, A. A. (2011). Smoking cues, argument strength, and perceived effectiveness of anti-smoking PSAs. Nicotine and Tobacco Research, 13, 282–290.”

The problem

The second author of both cited articles is Joseph N. Cappella, whose surname has two p’s.

Evidence a check of the printed page

The evidence is a check of the pages and of the author lists of the two cited articles. In PubMed, both articles list the second author as “Joseph N Cappella”, and the publisher’s author list for the article of Kang et al. (doi:10.1080/10410230903104269) agrees.

What survives

The titles, years, journals and page ranges identify the intended articles, and nothing else is affected. Only these three places were checked; the other names in the article were not.

Corrected

Conclusion, p. 176: “(Fishbein et al., 2002; Kang, Cappella, & Fishbein, 2009; Werb et al., 2011)”

Reference list, p. 179: “Kang, Y., Cappella, J. N., & Fishbein, M. (2009).” The rest of the entry is unchanged.

Reference list, p. 179: “Lee, S., Cappella, J. N., Lerman, C., & Strasser, A. A. (2011).” The rest of the entry is unchanged.

“A stronger design would be to deploy a series of different messages in this media market and gauge distinct outcomes that measure responses to each message.”

Crop of five lines of a paragraph on page 174. A red box marks the word 'gauges' in the line 'to deploy a series of different messages in this media market and gauges distinct outcomes'. — Section “Single-Medium Multi-Outcome Crossover Design”, p. 174 (PDF p. 8), first paragraph. The marked verb is coordinated with “to deploy” and needs the base form.

P024-E016 Minor

In the reference list, two author names have wrong punctuation (“Nickerson. D. W.” and “Murdock G.”), and two entries are out of alphabetical order: Fishbein et al. (2002) follows Fisher and Franz, and Imai, King and Nall (2009) follows Jensen and Kang.

Reference list, pp. 177–179 (PDF pp. 11–13): entries Arceneaux and Nickerson (2009), Deacon et al. (2007), Fishbein et al. (2002), and Imai, King and Nall (2009)

As printed

P. 177: “Arceneaux, K., & Nickerson. D. W. (2009). Who is mobilized to vote? A re-analysis of 11 field experiments. American Journal of Political Science, 53, 1–16.”

P. 178: “Deacon, D., Murdock G., Pickering, M., & Golding, P. (2007). Researching communications: A practical guide to methods in media and cultural analysis. New York, NY: Bloomsbury USA.”

P. 178, order of entries: Fisher, R. A. (1935); Franz, M. M., Freedman, P. B., Goldstein, K. M., & Ridout, T. N. (2007); Fishbein, M., Hall-Jamieson, K., Zimmer, E., von Haeften, I., & Nabi, R. (2002); Gaines, B. J., Kuklinski, J. H., & Quirk, P. J. (2007).

Pp. 178–179, order of entries: Jensen, K. B. (Ed.). (2011), the last entry of p. 178; Kang, Y., Capella, J. N., & Fishbein, M. (2009); Imai, K., King, G., & Nall, C. (2009); Lee, S., Capella, J. N., Lerman, C., & Strasser, A. A. (2011).

The problem

Elsewhere in the list a comma separates each surname from the initials, and the entries follow the alphabetical order of the first authors. “Fishbein” precedes “Fisher” and “Franz”, and “Imai” precedes “Jensen” and “Kang”.

Evidence a check of the printed page

The evidence is a check of the pages. In the printed order of the 64 entries of the list, entries 20 to 22 are Fisher, Franz and Fishbein, and entries 35 to 37 are Jensen, Kang and Imai.

What survives

The four entries identify the intended works, and nothing else is affected. The other bibliographic details of these entries were not checked. The spelling “Capella” in two of the quoted entries is corrected in P024-E010.

Corrected

P. 177: “Arceneaux, K., & Nickerson, D. W. (2009).” The rest of the entry is unchanged.

P. 178: “Deacon, D., Murdock, G., Pickering, M., & Golding, P. (2007).” The rest of the entry is unchanged.

Order of entries: the entry for Fishbein et al. (2002) is placed immediately before Fisher (1935), so that the order is Farquhar et al. (1985), Fishbein et al. (2002), Fisher (1935), Franz et al. (2007), Gaines et al. (2007). The entry for Imai, King and Nall (2009) is placed immediately before Jensen (2011), so that the order is Hudgens and Halloran (2008), Imai, King and Nall (2009), Jensen (2011), Kang et al. (2009), Lee et al. (2011).

Crop of five lines of the reference list on page 177. A red box marks 'Nickerson. D. W.' in the entry 'Arceneaux, K., and Nickerson. D. W. (2009). Who is mobilized to vote?'. The entries above and below it, for Arceneaux (2010) and for Aronow, Calfano and Green (2013), have a comma after each surname. — Reference list, p. 177 (PDF p. 11). In the marked name a full stop stands where the list has a comma between surname and initials.

Crop of five lines of the reference list on page 178. A red box marks 'Murdock G.' in the entry 'Deacon, D., Murdock G., Pickering, M., and Golding, P. (2007)'. The other names in the entry have a comma after the surname. — Reference list, p. 178 (PDF p. 12). The marked name lacks the comma between surname and initial.

Crop of seven lines of the reference list on page 178. The entries appear in the order Fisher, Franz, Fishbein, Gaines. A red box marks 'Fishbein, M.' at the start of the third entry. — Reference list, p. 178 (PDF p. 12). The marked entry, Fishbein et al. (2002), is printed after Fisher (1935) and Franz et al. (2007).

Crop of the first eight lines of the reference list on page 179. The entries appear in the order Kang, Imai, Lee. A red box marks 'Imai, K.' at the start of the second entry. — Reference list, top of p. 179 (PDF p. 13). The marked entry, Imai, King and Nall (2009), is printed after Kang et al. (2009); the entry for Jensen (2011) closes p. 178.

P024-A002 Minor

A design that randomizes the timing of the treatment does not require the treatment to be the sole systematic factor that changes over time; I narrow the sentence to a warning about unadjusted before-after comparisons.

Section “Single-Medium Multi-Outcome Crossover Design”, p. 174 (PDF p. 8), first paragraph, sentence beginning “In order to attribute a change in outcomes to the treatment”

See also under Formal statements (Lean): P024-T1, P024-T2

As printed

“This constraint obviously rules out between-subjects designs. One alternative is to deploy a within-subjects design that randomizes the timing of the treatment. This approach can be informative, but it relies on fairly strong assumptions (see Gerber & Green, 2012, chapter 8). In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time. A stronger design would be to deploy a series of different messages in this media market and gauges distinct outcomes that measure responses to each message.”

The problem

The paragraph introduces “a within-subjects design that randomizes the timing of the treatment”, says that this approach relies on fairly strong assumptions, and then states the condition. Read in this order, the sentence says that the design with randomized timing can attribute a change in outcomes to the treatment only if nothing else systematic changes over time. This is not required. Suppose that the timing is drawn by a known randomization mechanism. Then the probability of treatment in each period is known. If the outcome in a period depends on the timing only through whether the treatment has begun, then an estimator that weights the outcomes by these probabilities is unbiased for the average effect over the periods in which both treatment and control have positive probability, whatever the calendar trend. Further, under the sharp null hypothesis of no effect, the outcome path is the same for every possible timing. A randomization test of this hypothesis is then valid whatever the trend. An unadjusted before-after difference can nevertheless reflect factors other than treatment. That warning is sound, but the paragraph does not limit the sentence to such comparisons. The next sentence calls the multi-message design “stronger”. The ground for this comparison needs the same qualification, since the single-message design with randomized timing does not rest on an assumption of no trend. The word “gauges” in the last quoted sentence is a misprint; see P024-E015.

Evidence an exact counterexample

The evidence is an exact counterexample, computed in rational arithmetic. Consider one outlet, one outcome observed in the periods $t=1,2,3,4$ , and a treatment that is permanent once it starts. The onset $S$ is uniform on $\{2,3,4\}$ , and $Z_t=\mathbf{1}\{t\ge S\}$ indicates treatment in period $t$ . The potential outcomes are

Y_t(S)=t^2+2\cdot\mathbf{1}\{t\ge S\},\qquad t=1,2,3,4.

The outcome depends on the onset only through $Z_t$ , and I write $Y_t(0)=t^2$ and $Y_t(1)=t^2+2$ . The effect of the treatment is $2$ in every period, and the untreated outcomes $1,4,9,16$ follow a fixed trend. The treatment is therefore not the sole systematic factor that changes over time. Set $p_t=\mathbb{P}(Z_t=1)$ , so that $(p_1,p_2,p_3,p_4)=(0,1/3,2/3,1)$ . Treatment and control both have positive probability in periods 2 and 3 only. I define

\tau=\frac{1}{2}\sum_{t=2}^{3}\bigl[Y_t(1)-Y_t(0)\bigr]=2,\qquad \widehat{\tau}=\frac{1}{2}\sum_{t=2}^{3}\Bigl\{\frac{Z_tY_t}{p_t}-\frac{(1-Z_t)Y_t}{1-p_t}\Bigr\},

where $Y_t$ is the observed outcome. Since $Z_tY_t=Z_tY_t(1)$ , $(1-Z_t)Y_t=(1-Z_t)Y_t(0)$ and $\mathbb{E}[Z_t]=p_t$ , I have

\mathbb{E}\Bigl[\frac{Z_tY_t}{p_t}\Bigr]=Y_t(1),\qquad \mathbb{E}\Bigl[\frac{(1-Z_t)Y_t}{1-p_t}\Bigr]=Y_t(0),

and hence $\mathbb{E}[\widehat{\tau}]=\tau$ . Note that no step uses the form of $Y_t(0)$ . For the three onsets, I obtain $\widehat{\tau}=69/4$ , $21/4$ and $-33/2$ , and the mean of these values is $2$ .

For the unadjusted comparison, let $D(s)$ be the mean outcome over the periods $t\ge s$ minus the mean outcome over the periods $t<s$ . Under the sharp null hypothesis of no effect, $Y_t=t^2$ for every onset, and a direct computation gives $D(2)=26/3$ , $D(3)=10$ and $D(4)=34/3$ . The expectation of $D(S)$ is $10$ , although the effect is zero. With the effect of $2$ restored, the expectation is $12$ . Randomizing the onset does not make the unadjusted difference unbiased, and this is the content of the narrower warning.

Under the sharp null hypothesis, the outcome path is the same for every onset, and the distribution of $D(S)$ is therefore known. The upper-tail $p$ value $p(s)=\#\{u:D(u)\ge D(s)\}/3$ takes the values $1$ , $2/3$ and $1/3$ , and $\mathbb{P}(p(S)\le k/3)=k/3$ for $k=0,1,2,3$ . The test is valid despite the trend. The same argument applies to any outcome path and to $m$ equally likely onsets. If $k>0$ onsets have $p$ values at most $\alpha$ , then the smallest statistic among them has at least $k$ onsets at or above it, and its $p$ value is at least $k/m$ . This implies that $k/m\le\alpha$ , and the probability of rejection is $k/m$ . If $k=0$ , the probability of rejection is zero.

The counterexample, by onset. The weighted estimate is computed with a true effect of 2. The unadjusted difference and its $p$ value are computed with a true effect of 0.
Onset $S$	Weighted estimate $\widehat{\tau}$	Unadjusted difference $D$	$p$ value of $D$
$2$	$69/4$	$26/3$	$1$
$3$	$21/4$	$10$	$2/3$
$4$	$-33/2$	$34/3$	$1/3$
Mean over onsets	$2$	$10$

An independent recalculation agrees with these values. In a second example, with six periods, onsets uniform on $\{2,\dots,6\}$ and untreated outcomes $10t+3t^2$ , I obtain the estimates $2505/16$ , $1665/16$ , $695/16$ , $-785/16$ and $-245$ , whose mean is again the true effect $2$ . The four-period example has limits. The estimator $\widehat{\tau}$ is unbiased and imprecise, with variance $1561/8$ . The example uses the fact that the outcome in a period depends on the onset only through whether the treatment has begun, and it estimates the effect only at the periods in which both treatment and control have positive probability. It does not show that the multi-message design has no advantage in precision or robustness.

What survives

The warning about unadjusted before-after comparisons stands. The radio example with six organizations, the television example with nine segments and the randomization tests of the sharp null hypothesis that are proposed for them remain viable. Such a test is valid when it uses the actual randomization mechanism and the stated sharp null hypothesis, and it does not require the absence of calendar trends. The effect estimators and the assumptions about treatment history in those two studies were not examined. There are two changes. The necessity claim becomes a warning, and the reason for calling the multi-message design stronger can no longer be that randomized timing requires the absence of a trend. No number, table or later result in the article is derived from the sentence, and the other design sections do not depend on it.

Corrected

The sentence “In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time.” is replaced by the following three sentences.

“An unadjusted before-after change can reflect other factors that vary over time. With a known randomization mechanism, randomized timing can support design-based effect estimation and sharp-null randomization inference despite fixed calendar trends. Effect estimation must specify the estimand, ensure the relevant assignment probabilities are positive, and address treatment history, anticipation, and carryover in its estimand or assumptions.”

The second of these sentences rests on the following two statements. Let the onset $S$ of a permanent treatment be drawn by a known randomization mechanism, and let $Z_t=\mathbf{1}\{t\ge S\}$ . Suppose that the potential outcomes are fixed and that the outcome in period $t$ depends on $S$ only through $Z_t$ , with values $Y_t(1)$ and $Y_t(0)$ . Set $p_t=\mathbb{P}(Z_t=1)$ , and let $\mathcal{T}$ be a nonempty finite set of periods such that $0<p_t<1$ for all $t\in\mathcal{T}$ . Then

\mathbb{E}\biggl[\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\Bigl\{\frac{Z_tY_t}{p_t}-\frac{(1-Z_t)Y_t}{1-p_t}\Bigr\}\biggr] =\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\bigl[Y_t(1)-Y_t(0)\bigr],

with no condition on how $Y_t(0)$ varies with $t$ . Next, suppose that $S$ is uniform on a finite set of $m$ onsets and that the outcome path is the same for every onset, which is the sharp null hypothesis. For a statistic $D$ , set $p(s)=\#\{u:D(u)\ge D(s)\}/m$ . Then $\mathbb{P}(p(S)\le\alpha)\le\alpha$ for every $\alpha\in[0,1]$ , whatever the outcome path. This test needs no separate assumption about trends, anticipation or carryover, because the sharp null hypothesis fixes the outcomes under every onset.

Crop of eight lines of a paragraph on page 174. Red boxes mark the sentence 'In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time', which runs over two lines. The lines above it introduce a within-subjects design that randomizes the timing of the treatment and say that this approach relies on fairly strong assumptions. The text after it begins 'A stronger design would be to deploy a series of different messages'. — Section “Single-Medium Multi-Outcome Crossover Design”, p. 174 (PDF p. 8), first paragraph. The marked sentence states the condition directly after the paragraph has introduced the design that randomizes the timing of the treatment.

P024-N001 Minor

The citation “Green and Vavreck (2007)” on p. 169 should read “Green and Vavreck (2008)”, as on pp. 172 and 173 and in the reference list.

Introduction, paragraph beginning “Some recent examples of field experiments”, second sentence, p. 169 (PDF p. 3); reference list, p. 178 (PDF p. 12)

As printed

p. 169: “Green and Vavreck (2007) randomly assigned cable TV markets to receive Rock The Vote ads designed to encourage young people to vote in the 2004 presidential election.”

p. 172: “Green and Vavreck (2008) assigned 85 cable TV systems of varying sizes to receive Rock The Vote ads encouraging voter participation.”

Reference list, p. 178: “Green, D. P., & Vavreck, L. (2008). Analysis of cluster-randomized experiments: A comparison of alternative estimation approaches. Political Analysis, 16, 138–152.”

The problem

The citation of 2007 has no entry in the reference list. The same Rock The Vote experiment is described under Green and Vavreck (2008) on p. 172 and cited as Green & Vavreck (2008) on p. 173, and the only entry for these authors is dated 2008.

Evidence a check of the printed page

This is a check of the page. I read pp. 169, 172, 173 and 178, and I searched the article for every occurrence of “Vavreck”. I did not consult a source outside the article, so an earlier report of the experiment dated 2007 is not excluded.

What survives

Everything else is unaffected. No statement depends on the year.

Corrected

p. 169: “Green and Vavreck (2008) randomly assigned cable TV markets to receive Rock The Vote ads …”

Crop of five lines of the paragraph that begins 'Some recent examples of field experiments'. A red box marks the citation 'Green and Vavreck (2007)' in the sentence 'Green and Vavreck (2007) randomly assigned cable TV markets to receive Rock The Vote ads designed to encourage young people to vote in the 2004 presidential election.' — Introduction, p. 169 (PDF p. 3). The marked citation gives the year 2007.

Crop of five lines of the reference list. A red box marks the authors and year, Green, D. P., and Vavreck, L. (2008), at the start of the entry 'Analysis of cluster-randomized experiments: A comparison of alternative estimation approaches. Political Analysis, 16, 138-152.' The entries above and below are Gerber and Green (2012) and Gueron (2002). — Reference list, p. 178 (PDF p. 12). The marked entry is the only work by Green and Vavreck in the list, and it is dated 2008.

P024-N003 Minor

The reference for hierarchical modeling should be Arceneaux and Nickerson (2009), “Modeling certainty with clustered data: A comparison of methods”, Political Analysis 17, 177–190; the listed article by the same authors fits no hierarchical model.

First paragraph of p. 172 (PDF p. 6), sentence beginning “Options include”; reference list, p. 177 (PDF p. 11)

As printed

p. 172: “Options include the estimation of robust cluster standard errors (Wooldridge, 2003), random effects estimation (Green & Vavreck, 2008), hierarchical modeling (Arceneaux & Nickerson, 2009), or design-based estimates (Hansen & Bowers, 2009; Middleton & Aronow, 2011; Gerber & Green, 2012).”

Reference list, p. 177: “Arceneaux, K., & Nickerson. D. W. (2009). Who is mobilized to vote? A re-analysis of 11 field experiments. American Journal of Political Science, 53, 1–16.”

The problem

The citation resolves to the only entry for these authors, the article in the American Journal of Political Science. The listed article estimates probit and bivariate probit models with individual covariates and the salience of the election. It treats dependence by a bootstrap across individuals and elections, and it does not fit or illustrate a hierarchical model. The same authors published a second article in 2009, in Political Analysis, which compares hierarchical linear models with other methods for clustered data. The second article supports the statement and is not in the reference list. That it is the one intended is an inference from its subject, its authors and its year. The full stop after “Nickerson” in the entry is as printed (see P024-E016).

Evidence a check of the printed page

I checked the page, the listed article and the abstract of the second article. In the listed article, the Method section (p. 5) “presents the probit model”, and note 13 (p. 8) says that “bootstrapping occurs not only among individuals within an experiment but also between elections included in the analysis”. The abstract of the second article names the methods that it compares: “clustered robust standard errors (SEs), random effects, hierarchical linear model, and aggregated ordinary least squares”.

What survives

The list of ways to allow for clustering stands, hierarchical modeling included, and so does the warning that these estimates are unreliable with few clusters. Only the reference changes. The citation occurs once in the article.

Corrected

The citation on p. 172 stands, and the entry in the reference list is replaced. If the listed article is also kept, the two works of 2009 need distinguishing letters.

Reference list: “Arceneaux, K., & Nickerson, D. W. (2009). Modeling certainty with clustered data: A comparison of methods. Political Analysis, 17, 177–190.”

Crop of five lines of the first paragraph of the page. A red box marks the words 'hierarchical modeling', followed by the citation of Arceneaux and Nickerson, 2009, in parentheses. The list of options begins 'the estimation of robust cluster standard errors (Wooldridge, 2003), random effects estimation'. — First paragraph of p. 172 (PDF p. 6). The marked citation is offered for hierarchical modeling.

Crop of six lines of the reference list. Red boxes mark the two lines of the entry by Arceneaux and Nickerson, dated 2009: 'Who is mobilized to vote? A re-analysis of 11 field experiments. American Journal of Political Science, 53, 1-16.' The entry above it is Arceneaux, K. (2010), and the entry below it is Aronow, Calfano and Green (2013). — Reference list, p. 177 (PDF p. 11). The marked entry is the only work by Arceneaux and Nickerson in the list.

P024-N007 Minor

Six of the 64 entries in the reference list are not cited anywhere in the abstract, the text or the notes.

References, pp. 177–179 (PDF pp. 11–13): the entries for Abrajano (2010), Dong and Maynard (2013), Franz et al. (2007), Hillygus and Shields (2009), Nickerson (2005) and Prior (2009)

As printed

p. 177: Abrajano, M. (2010). Campaigning to the new American electorate: Advertising to Latino voters. Palo Alto, CA: Stanford University Press.
p. 178: Dong, N., & Maynard, R. (2013). PowerUp!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6, 24–67.
p. 178: Franz, M. M., Freedman, P. B., Goldstein, K. M., & Ridout, T. N. (2007). Campaign advertising and American democracy. Philadelphia, PA: Temple University Press.
p. 178: Hillygus, D. S., & Shields, T. G. (2009). The persuadable voter: Wedge issues in presidential campaigns. Princeton, NJ: Princeton University Press.
p. 179: Nickerson, D. (2005). Scalable protocols offer efficient design for field experiments. Political Analysis, 13, 233–252.
p. 179: Prior, M. (2009). Improving media effects research through better measurement of news exposure. Journal of Politics, 71, 893–908.

The problem

The reference list contains six works that the article never cites. The only occurrence of the name Nickerson before the reference list is the citation “Arceneaux & Nickerson, 2009” on p. 172, which belongs to a different entry. The article does not show whether the six entries remain from an earlier draft or were meant as background reading.

Evidence a check of the printed page

The evidence is a check of the pages. Searching the abstract, the text and Notes 1–5 (pp. 168–177) for the surname of each first author, I found no occurrence of Abrajano, Dong, Franz, Hillygus or Prior as a name, and only the occurrence of Nickerson quoted above. The same pages do not mention any of the six works by its title alone.

What survives

Everything else is unaffected, since no statement of the article depends on these entries.

Corrected

The six entries are removed from the reference list. An entry that was meant to be cited is instead kept, and a citation to it is added at the passage that it supports.

Crop of the heading 'References' and the first three lines of the list. A red box marks the author and year that open the first entry, 'Abrajano, M. (2010).'; the title of the entry is Campaigning to the new American electorate: Advertising to Latino voters. — References, p. 177 (PDF p. 11), first entry of the list. The marked entry, Abrajano (2010), is not cited in the article.

Crop of five lines of the reference list: the entry for Deacon, Murdock, Pickering and Golding (2007) and the entry for Dong and Maynard (2013). A red box marks the authors and year that open the second entry, Dong, N., and Maynard, R. (2013); the title of the entry begins PowerUp!: A tool for calculating minimum detectable effect sizes. — References, p. 178 (PDF p. 12). The marked entry, Dong and Maynard (2013), is not cited in the article.

Crop of three lines of the reference list: the entry for Fisher (1935) and the two-line entry for Franz et al. A red box marks the authors and year that open the second entry, Franz, M. M., Freedman, P. B., Goldstein, K. M., and Ridout, T. N. (2007); the title of the entry is Campaign advertising and American democracy. — References, p. 178 (PDF p. 12). The marked entry, Franz, Freedman, Goldstein and Ridout (2007), is not cited in the article.

Crop of three lines of the reference list: the last line of the entry for Hansen and Bowers (2009) and the two-line entry for Hillygus and Shields. A red box marks the authors and year that open the entry, Hillygus, D. S., and Shields, T. G. (2009); the title of the entry is The persuadable voter: Wedge issues in presidential campaigns. — References, p. 178 (PDF p. 12). The marked entry, Hillygus and Shields (2009), is not cited in the article.

Crop of three lines of the reference list: the last line of the entry for Mutz and Reeves (2005) and the two-line entry for Nickerson. A red box marks the author and year that open the entry, 'Nickerson, D. (2005).'; the title of the entry is Scalable protocols offer efficient design for field experiments. — References, p. 179 (PDF p. 13). The marked entry, Nickerson (2005), is not cited in the article.

Crop of three lines of the reference list: the last line of the entry for the Pew Center for Civic Journalism (1999) and the two-line entry for Prior. A red box marks the author and year that open the entry, 'Prior, M. (2009).'; the title of the entry is Improving media effects research through better measurement of news exposure. — References, p. 179 (PDF p. 13). The marked entry, Prior (2009), is not cited in the article.

P024-N008 Minor

Two reference titles depart from the published ones: the 1949 book is titled Experiments on mass communication, and the title of Panagopoulos and Green (2008) has no colon after “Field experiments”.

References, p. 178 (PDF p. 12), entry for Hovland, Lumsdaine and Sheffield (1949), and p. 179 (PDF p. 13), entry for Panagopoulos and Green (2008): the titles

As printed

p. 178: Hovland, C., Lumsdaine, A. A., & Sheffield, F. (1949). Experiments in mass communications. Princeton, NJ: Princeton University Press.
p. 179: Panagopoulos, C., & Green, D. P. (2008). Field experiments: Testing the impact of radio advertisements on electoral competition. American Journal of Political Science, 52, 156–168.

The problem

The book of Hovland, Lumsdaine and Sheffield is titled Experiments on mass communication, with “on” and the singular. The published title of Panagopoulos and Green (2008) is “Field experiments testing the impact of radio advertisements on electoral competition”, and the colon in the entry changes only the punctuation.

Evidence a check of the printed page

The evidence is a check of the pages against two records. The first is the catalogue record of the University of Canterbury library for the edition of the book published by Princeton University Press in 1949, and the second is the record of Wiley for the article.

What survives

Both works remain identifiable, and the citations of them in the text do not quote their titles.

Corrected

Hovland, C., Lumsdaine, A. A., & Sheffield, F. (1949). Experiments on mass communication. Princeton, NJ: Princeton University Press.
Panagopoulos, C., & Green, D. P. (2008). Field experiments testing the impact of radio advertisements on electoral competition. American Journal of Political Science, 52, 156–168.

Crop of three lines of the reference list: the last line of the entry for Horvitz and Thompson (1952) and the two-line entry for Hovland, Lumsdaine and Sheffield (1949). Red boxes mark the words 'in' and 'communications' in the italic title, which is printed as Experiments in mass communications. — References, p. 178 (PDF p. 12), entry for Hovland, Lumsdaine and Sheffield (1949). The two marked words of the title differ from the published title.

Crop of three lines of the reference list: the word 'University.' that ends the preceding entry and the two-line entry for Panagopoulos and Green (2008). A red box marks 'Field experiments:' at the start of the title, which continues Testing the impact of radio advertisements on electoral competition. — References, p. 179 (PDF p. 13), entry for Panagopoulos and Green (2008). The marked words end in a colon that the published title does not have.

Clarity notes

Imprecise or ambiguous wording whose intended meaning is correct; not an error.

“To get an intuitive sense of this issue, consider the contrast between an experiment in which two of four media markets, each comprising 1 million inhabitants, are assigned either to treatment or control. If the four media markets tend to have different outcomes, this design will generate highly variable results because two high- or low-scoring media markets have a good chance of being assigned to the same condition. On the other hand, a study with 4 million individually assigned subjects would have negligible sampling variability.”

Intended meaning

The passage compares two designs for the same 4 million people. The first is an experiment that assigns four media markets of 1 million inhabitants each, two to treatment and two to control. The second is a study that assigns the 4 million subjects individually. The first sentence does not say this well. The word “between” introduces one experiment, and the other arrives two sentences later. The phrase “two of four media markets … are assigned either to treatment or control” leaves the division into two treated and two control markets implicit. The meaning is recoverable. The first paragraph of the section defines the assignment of clusters to treatment or control. On p. 172 the article calls the illustration “the four-market experiment” and names the study with which it is compared. The point of the illustration is correct, and the warning about standard errors does not change.

Corrected

“To get an intuitive sense of this issue, consider an experiment in which four media markets, each comprising 1 million inhabitants, are assigned two to treatment and two to control. If the four media markets tend to have different outcomes, this design will generate highly variable results because two high- or low-scoring media markets have a good chance of being assigned to the same condition. By contrast, a study with 4 million individually assigned subjects would have negligible sampling variability.”

Crop of six lines near the bottom of the page. Red boxes mark the words 'consider the contrast between an experiment in which two of four media markets, each comprising 1 million inhabitants, are assigned either to treatment or control.', which run over three lines. The next sentence begins 'If the four media markets tend to have different outcomes'. — Section “Between-Subjects Clustered Design”, last paragraph of p. 171 (PDF p. 5). The marked sentence opens a contrast and names only one of the two experiments.

Crop of the running head '172 Donald P. Green et al.' and the first four lines of the page. Red boxes mark the words 'On the other hand, a study with 4 million individually assigned subjects', which run over two lines; the sentence continues 'would have negligible sampling variability.' — Top of p. 172 (PDF p. 6). The marked words supply the second half of the contrast, two sentences later.

Formal statements (Lean)

Formal results: 5 whole; 2 restricted; 0 refuted; 1 omitted.

Coverage concerns the identified claim under its stated assumptions. A checked proof establishes the formal statement; correspondence to the published text was reviewed by AI. Repository access is currently required to open the source links.

Each result links to its current formal statement and proof. “Whole” covers the identified claim under its assumptions; “restricted” covers the stated narrower scope; “refuted” gives a counterexample to the identified assertion; “omitted” has no current theorem. Any named published theorem used as a premise remains an assumption. Browse the formal results for this work.

P024-T1

Section ‘Single-Medium Multi-Outcome Crossover Design’, first paragraph, sentence beginning ‘In order to attribute a change in outcomes to the treatment’, as printed; the witness is the validator's construction and is not in the paper

p. 174 (PDF p. 8)

p. 174 (PDF p. 8), first paragraph of the single-medium crossover section. The four-period randomized-onset witness establishes the canonical counterexample. It refutes the reading that design-unbiased effect estimation or valid sharp-null inference under randomized timing requires a constant untreated outcome path. The fixed nonconstant trend admits the stated unbiased estimator, variance, and exactly calibrated valid sharp-null test.

Historical component credits

Proved AronowAudit.P024.calendar_ht_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Role: numerical witness for T1(i)/(ii), not a general theorem about arbitrary timing designs; all custom definitions were traced to finite sums and current binary exposure.
Proved AronowAudit.P024.calendar_null_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Role: T1(iii) numerical null witness and size bound; the baseline’s nonconstancy is explicit in its definition and in R154-01/04.
Proved AronowAudit.P024.calendar_p_grid Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Role: the grid-calibration component of T1(iii); R154-02 supplies validity at every level in [0,1].
Counterexample proved AronowAudit.P024.randomized_timing_no_trend_counterexample Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication', 'decision': 'rejected', 'what': 'P024/P024_Designs.lean:1041-1047 asserts a nonconstant untreated path, eligible average effect 2, design-unbiased estimation of that effect, and sharp-null test validity for every alpha in [0,1], matching T1(iv). The definitions at lines 174-195 encode uniform onsets 2-4, baseline t^2, treated path t^2+2, the overlap periods 2 and 3, and the baseline before-after statistic specified by the claim. Using the effect-2 path for estimation and the separate no-effect path for test calibration is expressly required by claim parts (ii) and (iii), rather than an unannounced change of model. The target itself specifies the broad design-level necessity reading of the printed sentence; the Lean statement supplies that witness and makes no claim about authorial intent or unbiasedness of the raw before-after contrast.'}

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:32d86df0c69b9c5b8b5e312d7af8808072449a05a2299ce331988320e98c44ad

Standing assumptions: The article (Political Communication 31:168–180, 2014; PDF pp. 2–14, PDF p. 1 being the publisher's cover page) is an overview of experimental designs. It has no theorem, no displayed formula, no mathematical notation, no table or figure and no formal model; every statistical claim is made in words. The setting those claims inherit, as printed (read from the page images): [Basic design and its assumptions: section ‘Between-Subjects Design’, p. 170 (PDF p. 4)] “The most basic experimental design is one in which subjects are assigned individually and with identical probability to treatment or control conditions. Assuming that (a) the random assignments are implemented as planned and outcomes are observed for all subjects, (b) the only systematic factor that causes the treatment group to differ from the control group is the intended treatment, and (c) subjects respond solely to the treatment that they receive (and are unaffected by treatments administered to others), the between-subjects design provides the basis for unbiased estimation of the average causal effect of the treatment in the subject pool (Gerber & Green, 2012). Indeed, the estimation of the average causal effect is simple and intuitive: The average outcome in the control group is subtracted from the average outcome in the treatment group.” [Clustered design: section ‘Between-Subjects Clustered Design’, p. 171 (PDF p. 5)] “Accordingly, researchers studying the effects of TV or radio randomly assign their treatments to geographic clusters. In effect, all individuals living within the same geographic cluster are assigned together to either treatment or control conditions. In the between-subjects clustered design, clusters are assigned independently, but individuals within clusters are not.” [Estimand under clustering: p. 172 (PDF p. 6)] “The design suggestions given above presuppose that a researcher’s aim in conducting a clustered experiment is the estimation of an \emph{individual-level} average treatment effect.” [Randomization inference: p. 172 (PDF p. 6)] “One option in such settings is hypothesis testing based on the method of randomization inference (Fisher, 1935; Rosenbaum, 2002). Under the null hypothesis of no treatment effect whatsoever, one may simulate the sampling distribution of possible estimated treatment effects under all possible random assignments. The $p$ value of the observed estimate is calculated based on its location relative to this randomization distribution.” [No interference: Note 2, p. 177 (PDF p. 11), attached on p. 171 to the discussion of spillovers in Bond et al.] “2. Modeling spillovers in this way in effect relaxes the assumption that subjects are affected solely by their own treatment status and not the status of others.” Not stated anywhere in the article: potential-outcome notation; a formula for any estimator; an assignment mechanism beyond the words quoted (complete randomization of a fixed number of units and independent assignment with a common probability both fit “assigned individually and with identical probability” and “clusters are assigned independently”); a sequence of populations for the consistency statement, or any bound on outcomes or on cluster sizes; a sampling model. The formal renderings in the targets below are therefore the audit's, taken from the validators' derivations and from the two readers' exact checks, in finite-population potential-outcome notation: units $i = 1, \ldots, N$ with fixed potential outcomes $Y_i(0)$, $Y_i(1)$; an assignment $Z \in \{0,1\}^N$ that is the only source of randomness (the article's condition (b)); observed outcome $Y_i = Z_iY_i(1) + (1-Z_i)Y_i(0)$ for every unit (conditions (a) and (c)); average effect $\tau = \frac{1}{N}\sum_i [Y_i(1) - Y_i(0)]$. What is and is not a target. Two findings are confirmed errors whose correction changes a mathematical statement: P024-A002 (Moderate; p. 174, the stated necessity of a no-other-change assumption under randomized timing) and P024-E002 (Minor; p. 171, consistency stated without a regularity condition). Each has a counterexample and a corrected statement below. P024-E001 (Moderate) is not a target: it is a misreported statistic from a cited study (p. 171 gives the Bond et al. (2012) effect as approximately 0.7 percentage points where the source reports 0.39), a reporting finding with no mathematical statement to correct, and no claim of the article is derived from the number. The other sixteen error-class findings (P024-E005 to P024-E016, P024-N001, P024-N003, P024-N007, P024-N008) are citation, reference-list, spelling and grammar misprints. Of the nine headline claims, P024-H2, P024-H3 and P024-H5 contain a mathematical statement (unbiasedness; bias and consistency; a weighted-average identity) and are targets; P024-H1, P024-H4, P024-H6, P024-H8 and P024-H9 are framing, qualitative or practical claims; the formal content of P024-H7 (sharp-null randomization tests under a randomized schedule) is part (b) of P024-T2.

Effective claim (source TeX): [Printed statement refuted: p. 174 (PDF p. 8)] “One alternative is to deploy a within-subjects design that randomizes the timing of the treatment. This approach can be informative, but it relies on fairly strong assumptions (see Gerber & Green, 2012, chapter 8). In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time.” [Reading refuted] For the randomized-timing design of the preceding sentence, causal attribution (a design-unbiased estimate of an average effect, or a valid test of the sharp null of no effect) requires that the untreated outcome not change systematically over time. [Witness: validator's construction, not in the paper] One media outlet; calendar periods $t = 1, 2, 3, 4$; one outcome per period; one intervention that is permanent from its onset. The onset $S$ is drawn uniformly from $\{2, 3, 4\}$ and is the only randomness; $Z_t(s) = \mathbf{1}\{t \ge s\}$. For every onset $s \in \{2, 3, 4\}$ and every period $t$ the potential outcome is $$Y_t(s) = t^2 + 2\,\mathbf{1}\{t \ge s\},$$ so that the outcome depends on the schedule only through current exposure (no anticipation, no carryover), with untreated values $Y_t(0) = t^2$, that is $(1, 4, 9, 16)$, and treated values $Y_t(1) = t^2 + 2$, that is $(3, 6, 11, 18)$. The observed outcome in period $t$ is $Y_t = Y_t(S)$, and all four are observed. The exposure probabilities are $p_t = \Pr(Z_t(S) = 1) = (0, 1/3, 2/3, 1)$ for $t = 1, \ldots, 4$. [Claims] (i) [The treatment is not the sole systematic factor that changes over time.] The untreated outcome $t^2$ is not constant in $t$: it is $1, 4, 9, 16$ whatever the onset. (ii) [A design-unbiased effect estimator exists.] Let $\tau = \frac{1}{2}\sum_{t=2}^{3}[Y_t(1) - Y_t(0)] = 2$ and $$\widehat{\tau}(S) = \frac{1}{2}\sum_{t=2}^{3}\left\{\frac{Z_t(S)\,Y_t}{p_t} - \frac{[1 - Z_t(S)]\,Y_t}{1 - p_t}\right\}.$$ Then $\widehat{\tau}(2) = 69/4$, $\widehat{\tau}(3) = 21/4$, $\widehat{\tau}(4) = -33/2$, and $\mathrm{E}[\widehat{\tau}(S)] = \frac{1}{3}\left(\frac{69}{4} + \frac{21}{4} - \frac{33}{2}\right) = 2 = \tau$. Its variance is $1561/8$. (iii) [A valid sharp-null test exists.] Replace the potential outcomes by $Y_t(s) = t^2$ for every $s$ (no effect of any kind, the same trend). For the before-after statistic $$D(s) = \frac{1}{5-s}\sum_{t=s}^{4} t^2 - \frac{1}{s-1}\sum_{t=1}^{s-1} t^2,$$ $D(2) = 26/3$, $D(3) = 10$, $D(4) = 34/3$. The randomization $p$-value $p(s) = \#\{u \in \{2,3,4\} : D(u) \ge D(s)\}/3$ takes the values $p(2) = 1$, $p(3) = 2/3$, $p(4) = 1/3$, so $\Pr(p(S) \le k/3) = k/3$ for $k = 0, 1, 2, 3$ and $\Pr(p(S) \le \alpha) \le \alpha$ for every $\alpha \in [0, 1]$. (iv) [Consequence.] There is a single-outlet within-subjects design with randomized timing, with all outcomes observed and the schedule the only randomness, in which the treatment is not the sole systematic factor that changes over time and yet (ii) and (iii) hold. The assumption is therefore not necessary for causal attribution under that design.

: SAP.P024.T1Statement

Theorem: SAP.P024.T1

Statement hash: cc7579e8e9b3f6617818f2dbe3636227d499441455370f3e1dec17351b87eaae

Statement definition: SAP.P024.T1Statement

Certificate. 20261104T192849Z

Declaration review: faithful; 2026-11-04T19:44:53Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P024-T2

Section ‘Single-Medium Multi-Outcome Crossover Design’, first paragraph, sentence beginning ‘In order to attribute a change in outcomes to the treatment’, as repaired by Sol

p. 174 (PDF p. 8)

corrected timing sentence, p. 174 (PDF p. 8). Current-exposure overlap permits unbiased timing estimation and fixed sharp-null paths permit valid uniform randomization tests, while the actual raw before-after contrast is biased.

Historical component credits

Proved AronowAudit.P024.expectation_add Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.expectation_sub Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.expectation_mul Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.expectation_const Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.expectation_sum Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.expectation_div Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.ht_coordinate_unbiased Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.timing_ht_unbiased Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in timing_overlap_unbiased or randomization_test_valid; no separate invocation.
Proved AronowAudit.P024.timing_overlap_unbiased Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The finite current-exposure model, nonempty full overlap set, arbitrary calendar trajectories and arbitrary finite design law match T2(a).
Proved AronowAudit.P024.randomization_test_valid Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; A fixed sharp-null path gives one common function D(ω;y) on all schedules, which is precisely the theorem’s D. Uniformity and ties included in the upper tail match T2(b).
Proved AronowAudit.P024.raw_before_after_bias Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The covered component is the two expectation calculations in T2(c); the identification of D+2 with the actual observed statistic is exported by R154-14.
Proved AronowAudit.P024.raw_observed_effect_bias Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The identity linking the actual effect-2 observed statistic to the shifted null statistic is explicit in the conclusion, not hidden inside a proof.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:d38c993d9b92d67771c0d9e9dc44d66df6e81d7487740e3e58747639084eadf3

Effective claim (source TeX): Sol's corrected text (validated.json, P024-A002, prior_correction.corrected_text; smallest_repair gives the same repair as an instruction). It replaces the sentence “In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time.” Verbatim: “An unadjusted before-after change can reflect other factors that vary over time. With a known randomization mechanism, randomized timing can support design-based effect estimation and sharp-null randomization inference despite fixed calendar trends. Effect estimation must specify the estimand, ensure the relevant assignment probabilities are positive, and address treatment history, anticipation, and carryover in its estimand or assumptions.” Mathematical content, with every hypothesis (validator derivation, sections 4–6). [Setting] One unit (one media outlet) observed in periods $t = 1, \ldots, T$. A finite set $\Omega$ of treatment schedules with a known probability distribution; schedule $\omega$ gives exposure indicators $Z_t(\omega) \in \{0, 1\}$ (in the validator's case $\Omega$ is a set of onsets and $Z_t(s) = \mathbf{1}\{t \ge s\}$). The schedule is the only randomness and all outcomes are observed. (a) [Design-based estimation despite a fixed calendar trend.] Hypotheses: fixed real numbers $Y_t(0)$ and $Y_t(1)$ for each period, with no restriction on how either varies with $t$; the observed outcome under schedule $\omega$ is $Y_t = Y_t(Z_t(\omega))$ (the outcome depends on the schedule only through current exposure: no anticipation, no carryover, no dependence on duration); $p_t = \Pr(Z_t = 1)$; the set $E = \{t : 0 < p_t < 1\}$ is nonempty. Conclusion: $$\mathrm{E}\left[\frac{1}{|E|}\sum_{t \in E}\left\{\frac{Z_tY_t}{p_t} - \frac{(1 - Z_t)Y_t}{1 - p_t}\right\}\right] = \frac{1}{|E|}\sum_{t \in E}[Y_t(1) - Y_t(0)].$$ No independence of the $Z_t$ across periods is used. (b) [Sharp-null randomization inference despite a fixed calendar trend.] Hypotheses: $\Omega$ has $m \ge 1$ schedules, each with probability $1/m$; under the sharp null the outcome path $y = (y_1, \ldots, y_T)$ is the same under every schedule and is otherwise arbitrary; $D(\omega) = D(\omega; y)$ is any real-valued statistic; $p(\omega) = \#\{\upsilon \in \Omega : D(\upsilon) \ge D(\omega)\}/m$. Conclusion: for the realized schedule $W$, uniform on $\Omega$, $\Pr(p(W) \le \alpha) \le \alpha$ for every $\alpha \in [0, 1]$. (c) [An unadjusted before-after change can reflect other factors.] In the population of P024-T1 with no effect ($Y_t(s) = t^2$ for every $s$) and $S$ uniform on $\{2, 3, 4\}$, the before-after statistic $D$ of P024-T1 has $\mathrm{E}[D(S)] = \frac{1}{3}\left(\frac{26}{3} + 10 + \frac{34}{3}\right) = 10 \ne 0$. With the constant effect 2 restored ($Y_t(s) = t^2 + 2\,\mathbf{1}\{t \ge s\}$) the same statistic takes the value $D(s) + 2$ for each onset and has expectation $12 \ne 2$. Randomizing the onset does not make the raw before-after difference unbiased for the effect.

: SAP.P024.T2Statement

Theorem: SAP.P024.T2

Statement hash: d624836f6b83de138d42e0ae7ed573295a5ec398a946e05cf8d426c20b438b1c

Statement definition: SAP.P024.T2Statement

Certificate. 20261104T192849Z

Declaration review: faithful; 2026-11-04T19:44:53Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Manifest annotation reconciliation: Effective use follows the source-bound dependency observation; original annotations are retained in the crosswalk.

AronowAudit.P024.expectation_const: observed used

AronowAudit.P024.expectation_div: observed used

AronowAudit.P024.expectation_mul: observed used

AronowAudit.P024.expectation_sub: observed used

AronowAudit.P024.expectation_sum: observed used

AronowAudit.P024.ht_coordinate_unbiased: observed used

AronowAudit.P024.timing_ht_unbiased: observed used

Source and evidence metadata · sap-2026-11-08-assured-release

P024-T3

Section ‘Single-Medium Multi-Outcome Crossover Design’, first paragraph, as printed

p. 174 (PDF p. 8); related sentences on p. 174 (PDF p. 8) and p. 175 (PDF p. 9)

No current theorem is supplied for this target.

Historical component credits

Not applicable Excluded association; baseline fidelity: None; manifest: not applicable; Solution use unverified; Historical excluded association.

Formal result: omitted

Scope: No Lean declarations are credited to this target; no theorem establishes its multi-message design comparison, robustness assertion, sharp-null procedure, or claimed advance over prior work.

Omission: not formalized

Reason: No Lean declarations are credited to this target; no theorem establishes its multi-message design comparison, robustness assertion, sharp-null procedure, or claimed advance over prior work.

Omission explanation

Assumptions and verification

Claim revision: catalog:8f252f1da3a677d49f225f00f996626eb1b2f604ccce1c9e870c6472e203ae3b

Effective claim (source TeX): [First paragraph of the section, p. 174 (PDF p. 8)] “In extreme cases, researchers have access to just a single media outlet, either because of resource constraints, a narrow geographic focus, or a highly consolidated media market. This constraint obviously rules out between-subjects designs. One alternative is to deploy a within-subjects design that randomizes the timing of the treatment. This approach can be informative, but it relies on fairly strong assumptions (see Gerber & Green, 2012, chapter 8). In order to attribute a change in outcomes to the treatment, one must assume that the treatment is the sole systematic factor that changes over time. A stronger design would be to deploy a series of different messages in this media market and gauges distinct outcomes that measure responses to each message. We refer to this design as a \emph{single-medium multi-outcome crossover} design.” [What the comparison is used for, second paragraph, p. 174 (PDF p. 8)] “The attractive feature of this design is that it allows the researcher to estimate the average effect of the weeklong radio announcements in a more robust manner than would be the case if there were just one PSA for one social service group. A nonparametric test of the sharp null hypothesis of no effect from the PSAs (during the week they air or in subsequent weeks) may be conducted using randomization inference.” [Closing sentence of the section, p. 175 (PDF p. 9)] “This design represents an advance over prior work, which tended to rely on before-after comparisons where the timing of the intervention is not randomly assigned.”

Unit certificate (omission has no certified theorem). 20261104T192849Z

Declaration review: omission_ok; 2026-11-04T19:33:51Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Not supplied by this export. Manifest uses are reported separately; actual use is unverified.

Source and evidence metadata · sap-2026-11-08-assured-release

P024-T4

Section ‘Between-Subjects Clustered Design’, second paragraph, sentence ‘The difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of clusters becomes very large’, as printed; the witness is the validator's construction and is not in the paper

p. 171 (PDF p. 5)

consistency sentence, p. 171 (PDF p. 5). The dominant-cluster populations establish the canonical counterexample. They refute the reading that a growing number of independently assigned clusters alone, without a condition controlling their population shares, guarantees consistency of the individual-level difference in means. The theorem includes the independent and complete-design formulas, limits, conditional mean, population shares, and six-cluster numerical checks.

Historical component credits

Proved AronowAudit.P024.sum_two_values Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in the retained exact formulas, design distribution, or numerical checks; no separate invocation.
Proved AronowAudit.P024.cluster_dim_subset Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in the retained exact formulas, design distribution, or numerical checks; no separate invocation.
Proved AronowAudit.P024.dominant_means Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Its mean calculations are already inside the exact-value theorem; the concrete population specifies the individual outcomes.
Proved AronowAudit.P024.dominant_individual_outcomes Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The fixed-outcome witness is constructed explicitly; no boundedness or effect identity is smuggled in as a hypothesis.
Proved AronowAudit.P024.dominant_effect Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This verifies the individual-weighted estimand, rather than an unweighted cluster estimand.
Proved AronowAudit.P024.dominant_exact_value Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The only restrictions are exactly the target’s nonempty-arm restrictions; positive natural cluster sizes guarantee positive treatment and control population denominators.
Proved AronowAudit.P024.dominant_error_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This auxiliary real-variable inequality genuinely applies to both branches of every nonempty integer allocation once the exact-value row is used; no assignment law or unproved statistical result is assumed.
Proved AronowAudit.P024.uniform_subsets_independent Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The conclusion specifies all joint masses and therefore full independence, not merely identical marginals or zero covariances. Inclusion of K=0 loses no case required by the K≥2 witness.
Proved AronowAudit.P024.nonempty_arm_probability Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Uniform subsets are the independent Bernoulli(1/2) law, as established by R151-12; K=k+2 excludes precisely cluster counts outside the target domain.
Proved AronowAudit.P024.dominant_probability_lower_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The finite lower bound holds uniformly over all empty-arm values. The actual limit is exported by R150-03 and R150-02, rather than inferred as an additional conclusion of this row.
Proved AronowAudit.P024.dominant_cluster_inconsistent Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the limit component for the same exact witness, not a generic theorem about every clustered population. Its concrete definitions and the other reviewed witness rows verify the printed hypotheses. No verdict change after the record.
Counterexample proved AronowAudit.P024.clustered_consistency_counterexample Historical support retained; baseline fidelity: ok; manifest: not used; not used in Solution; Its assembled witness is obtained here from its separately credited components; no separate invocation.
Proved AronowAudit.P024.dominant_conditional_unbiased Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The conditioning event has probability 1−2^(1−K)>0 for every K≥2; uniform conditioning is valid and does not add a restriction on the unconditioned counterexample.
Proved AronowAudit.P024.probability_indicator_expected Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in the retained exact formulas, design distribution, or numerical checks; no separate invocation.
Proved AronowAudit.P024.expectation_if_constant Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in the retained exact formulas, design distribution, or numerical checks; no separate invocation.
Proved AronowAudit.P024.probability_all Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in the retained exact formulas, design distribution, or numerical checks; no separate invocation.
Proved AronowAudit.P024.uniform_fixed_subset_marginal Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in the retained exact formulas, design distribution, or numerical checks; no separate invocation.
Proved AronowAudit.P024.dominant_complete_distribution Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The complete-design component is ancillary: fixed-count cluster assignments are dependent, so this row is not used as the independent-assignment witness refuting the printed statement. Its probability-one error bound itself entails inconsistency along the even sequence.
Proved AronowAudit.P024.half_dominant_ratio_limit Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The same ratio limit is included in dominant_complete_limits.
Proved AronowAudit.P024.dominant_complete_limits Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The two conclusions are the ratio limit and variance limit; no stronger distributional limit is credited.
Proved AronowAudit.P024.dominant_total_size Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is a direct witness component. Coercing natural sizes to real numbers loses none of the asserted size identity.
Proved AronowAudit.P024.dominant_max_weight Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; The description’s nonvanishing-share comment is consistent with the exact formula. The convergence-to-one component is separately credited in the next split, so this row is assessed as the finite identity.
Proved AronowAudit.P024.dominant_share_limit Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Ancillary witness component, not a standalone refutation: the maximum is over every cluster of exactly the target population, and k+2 covers all K≥2.
Proved AronowAudit.P024.dominant_six_checks Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This is the numerical auxiliary component advertised by row_says. In particular its last equality evaluates a variance expression; it does not independently identify that expression with the actual estimator variance.
Proved AronowAudit.P024.dominant_six_actual_checks Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P024/P024_Designs.lean:1932 exports the specified sizes, variance, universal error lower bound, and existence of an allocation attaining 9/40, so those parts of T4(v) match. Its last conjunct at P024/P024_Designs.lean:1938 does not identify the allocation: it says only that some nonempty proper subset attains the bound. The row additionally says that treating every cluster except cluster 6 attains it, and that particular equality is absent from the statement. The defect is confined to this advertised named minimizer, not the minimum or the counterexample itself.', 'suggested_fix': 'Add the conclusion that for s = (Finset.univ : Finset (Fin 6)).erase 5, s is nonempty and proper and |populationEstimator (dominantPopulation 4) (fun _ => 0) s - 1/4| = 9/40; the existing existential conclusion can then remain or follow from it.', 'second_review': "P024/P024_Designs.lean:1933-1939 exports the sizes, variance, universal error bound and attainment required by T4(v)'s claim that the smallest absolute error is 9/40. Only row_says additionally names the allocation treating every cluster except cluster 6; the target does not require that identification. This concrete finite-instance detail warrants a minor description repair, not the adjudicator's major statement repair.", 'severity': None}

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:e6f2c4640611c2625a04160fbd1b3d9fcc44888778d16c1ead47bef2baa3bb5d

Effective claim (source TeX): [Printed statement refuted: p. 171 (PDF p. 5)] “The difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of clusters becomes very large, but bias may be appreciable in studies with only a few dozen clusters of varying size.” [Reading refuted] Under clustered assignment with the article's conditions (a)–(c) and a common assignment probability, the individual-level difference in means converges in probability to the average treatment effect among all subjects whenever the number of clusters tends to infinity. [Witness: validator's construction, not in the paper] For each integer $K \ge 2$ the population has $K$ clusters: cluster 1 with $n_1 = K^2$ individuals and clusters $2, \ldots, K$ with one individual each, so $N_K = K^2 + K - 1$. Potential outcomes are fixed: every individual in cluster 1 has $Y(0) = 1/2$ and $Y(1) = 3/4$; every singleton has $Y(0) = 1/4$ and $Y(1) = 1/2$. Every individual effect is $1/4$, so the average effect is $\tau_K = 1/4$ for every $K$, and every potential outcome lies in $(0, 1)$. Clusters are assigned as wholes, $Z_j \in \{0, 1\}$; all outcomes are observed and there is no interference. With $\bar{Y}_j(z)$ the mean of $Y(z)$ in cluster $j$, the estimator on allocations with both arms nonempty is $$\widehat{\tau}_K = \frac{\sum_j Z_j n_j \bar{Y}_j(1)}{\sum_j Z_j n_j} - \frac{\sum_j (1 - Z_j) n_j \bar{Y}_j(0)}{\sum_j (1 - Z_j) n_j}.$$ [Claims] (i) [Exact values.] On an allocation with $m$ treated clusters, $1 \le m \le K - 1$: if cluster 1 is treated, $\widehat{\tau}_K = \frac{1}{4} + \frac{1}{4}\cdot\frac{K^2}{K^2 + m - 1}$; if cluster 1 is in control, $\widehat{\tau}_K = \frac{1}{4} - \frac{1}{4}\cdot\frac{K^2}{K^2 + K - m - 1}$. (ii) [Independent assignment, the design the article describes.] Let $Z_1, \ldots, Z_K$ be independent with $\Pr(Z_j = 1) = 1/2$. On every allocation with both arms nonempty, $$|\widehat{\tau}_K - \tau_K| \ge \frac{1}{4}\cdot\frac{K^2}{K^2 + K - 2} > \frac{1}{8},$$ and these allocations have total probability $1 - 2^{1-K}$. Hence $\Pr(|\widehat{\tau}_K - \tau_K| > 1/8) \ge 1 - 2^{1-K} \to 1$ as $K \to \infty$, whatever value the estimator is given on the two empty-arm allocations; $\widehat{\tau}_K - \tau_K$ does not converge to 0 in probability. Conditionally on both arms being nonempty, $\mathrm{E}[\widehat{\tau}_K] = 1/4$: the failure is not one of bias. (iii) [Complete randomization of half the clusters.] For even $K$, let the treated set be uniform over the subsets of $K/2$ clusters, and put $r_K = K^2/(K^2 + K/2 - 1)$. Then $\widehat{\tau}_K = \frac{1}{4} + \frac{1}{4}r_K$ with probability $1/2$ and $\widehat{\tau}_K = \frac{1}{4} - \frac{1}{4}r_K$ with probability $1/2$; $\mathrm{E}[\widehat{\tau}_K] = 1/4$; $\mathrm{Var}(\widehat{\tau}_K) = r_K^2/16 \to 1/16$; and, since $r_K > 1/2$, $\Pr(|\widehat{\tau}_K - \tau_K| > 1/8) = 1$ for every even $K \ge 2$. (iv) [What fails.] The largest cluster's share of the population is $n_1/N_K = K^2/(K^2 + K - 1) \to 1$. (v) [Instance $K = 6$.] Cluster sizes $(36, 1, 1, 1, 1, 1)$. Under (iii) the 20 equally likely allocations give $37/76$ ten times and $1/76$ ten times, with variance $81/1444$. Under (ii), 62 of the 64 allocations have both arms nonempty and the smallest absolute error among them is $9/40$.

: SAP.P024.T4Statement

Theorem: SAP.P024.T4

Statement hash: d49e6051e05a6d4eb78dda5dbadd439989d7766e56d5241b9f498dc30eecbbb8

Statement definition: SAP.P024.T4Statement

Certificate. 20261104T192849Z

Declaration review: faithful; 2026-11-04T19:44:53Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Manifest annotation reconciliation: Effective use follows the source-bound dependency observation; original annotations are retained in the crosswalk.

AronowAudit.P024.cluster_dim_subset: observed used

AronowAudit.P024.dominant_means: observed used

AronowAudit.P024.expectation_if_constant: observed used

AronowAudit.P024.half_dominant_ratio_limit: observed used

AronowAudit.P024.probability_all: observed used

AronowAudit.P024.probability_indicator_expected: observed used

AronowAudit.P024.sum_two_values: observed used

AronowAudit.P024.uniform_fixed_subset_marginal: observed used

Source and evidence metadata · sap-2026-11-08-assured-release

P024-T5

Section ‘Between-Subjects Clustered Design’, second paragraph, the consistency sentence, corrected

p. 171 (PDF p. 5)

corrected consistency sentence, p. 171 (PDF p. 5). Uniformly bounded individual potential outcomes, a common cluster treatment probability uniformly bounded away from zero and one, and a vanishing largest cluster share imply consistency: one fixed eta > 0 satisfies eta <= p_K <= 1-eta for every K. This holds under independent assignment or complete cluster randomization. The squared-share condition is equivalent. Bias can still be appreciable with a few dozen unequal clusters: a 24-cluster, 12-treated example with outcomes in [0,1] has effect 2/3, mean estimate 23/57 and bias -5/19, of magnitude greater than 1/4. Without no dominance, bounded outcomes and interior independent assignment do not suffice: the dominant-cluster sequence has error probability tending to one for every empty-arm extension. Populations, effects, and probabilities may change, while the outcome bound and probability margin remain uniform over the sequence.

Historical component credits

Proved AronowAudit.P024.weight_square_bounds Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.finite_chebyshev Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.ratio_error_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.two_arm_ratio_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.expectation_mono Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.probability_mono Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.probability_bounds Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.uniform_probability_card Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.probability_union_le Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.finite_ratio_good_event Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.finite_sample_probability_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.no_dominance_iff_squared_weights Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; K=k+2 enumerates every target population count K≥2. General normalized weights include all actual shares; the equivalence does not assume either limit.
Proved AronowAudit.P024.probability_not Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.independent_exclusion_probability Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.independent_marginal Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.independent_pair_moment Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.finite_variance_identity Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.variance_const_sub Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.assignment_total_indicator Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.assignment_total_complement Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.exchangeable_weighted_moments Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.independent_weighted_moments Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.complete_probability_count Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.complete_joint_probability Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.complete_marginal Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.complete_pair_inclusion Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.complete_weighted_moments Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.normalized_total_moments Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.population_total_positive Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.population_weights Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This row establishes the properties of shares used by the other lemmas; it does not assume or conclude no dominance for a sequence.
Proved AronowAudit.P024.finite_mean_abs_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.weighted_mean_abs_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.population_effect_normalized Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.population_estimator_normalized Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.cluster_population_probability_bound Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary already incorporated in corrected_cluster_consistency or the retained weight equivalence; no separate invocation.
Proved AronowAudit.P024.corrected_cluster_consistency Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; This proves corrected T5 rather than the article’s unqualified printed assertion; the added regularity assumptions are in the corrected target itself. K=k+2 is a bijective reindexing of the target K=2,3,…, and the limit uses every natural k. No claim-stage verdict changed after reading the record.

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:c7730f348ece475275a0d920e3eb610ed296b8be2c4c159c7a5b06eacd471669

Effective claim (source TeX): Sol's replacement for the consistency clause (validated.json, P024-E002, smallest_repair), verbatim: “With uniformly bounded potential outcomes and a common cluster treatment probability bounded away from zero and one, the difference-in-means estimator is consistent as the number of independently assigned clusters grows, provided that no cluster accounts for a nonvanishing share of all subjects.” Sol adds: “For complete cluster randomization, use a treatment fraction bounded away from zero and one. The no-dominance condition is max_j n_j/N -> 0, equivalently sum_j (n_j/N)^2 -> 0.” The rest of the printed sentence (“but bias may be appreciable in studies with only a few dozen clusters of varying size”) is unchanged. [Corrected statement in full; validator derivation, section 5] [Setting] A sequence of finite populations indexed by $K = 2, 3, \ldots$ The $K$-th population has $K$ clusters with sizes $n_j \ge 1$ ($j = 1, \ldots, K$), $N = \sum_j n_j$ individuals, and fixed potential outcomes $Y_{ij}(0)$, $Y_{ij}(1)$ for individual $i$ of cluster $j$; sizes and outcomes may change with $K$. Put $w_j = n_j/N$, $\bar{Y}_j(z) = n_j^{-1}\sum_i Y_{ij}(z)$, $\mu_z = \sum_j w_j\bar{Y}_j(z)$ and $\tau_K = \mu_1 - \mu_0$, the average treatment effect among all individuals. Clusters are assigned as wholes with indicators $Z_j \in \{0,1\}$; every individual's outcome is observed and equals $Y_{ij}(Z_j)$ (no interference, assignment implemented as planned). The estimator is $$\widehat{\tau}_K = \frac{\sum_j Z_j n_j \bar{Y}_j(1)}{\sum_j Z_j n_j} - \frac{\sum_j (1 - Z_j) n_j \bar{Y}_j(0)}{\sum_j (1 - Z_j) n_j}$$ when both arms are nonempty, and any real number otherwise. [Hypotheses] (i) [Bounded outcomes] there is $B < \infty$, not depending on $K$, with $|Y_{ij}(z)| \le B$ for all $K$, $j$, $i$, $z$. (ii) [Design] either $Z_1, \ldots, Z_K$ are independent with a common probability $\Pr(Z_j = 1) = p_K$, or the treated set is uniform over the subsets of $m_K$ clusters and $p_K = m_K/K$. (iii) [Interior probability] there is $\eta > 0$, not depending on $K$, with $\eta \le p_K \le 1 - \eta$. (iv) [No dominant cluster] $\max_j w_j \to 0$ as $K \to \infty$. [Conclusion] For every $\epsilon > 0$, $\Pr(|\widehat{\tau}_K - \tau_K| > \epsilon) \to 0$ as $K \to \infty$. Neither $\tau_K$ nor $p_K$ is required to converge. [Equivalent form of (iv)] $\max_j w_j \to 0$ if and only if $\sum_j w_j^2 \to 0$. [What does not hold] Hypotheses (i)–(iii) and $K \to \infty$ without (iv) do not imply the conclusion: P024-T4.

: SAP.P024.T5WholeStatement

Theorem: SAP.P024.T5

Statement hash: 79e03a1316615f5249fd6c1d0b0d3f298a4f058997fe227a171a245cde65737f

Statement definition: SAP.P024.T5WholeStatement

Certificate. 20261104T192849Z

Declaration review: faithful; 2026-11-04T19:44:53Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Manifest annotation reconciliation: Effective use follows the source-bound dependency observation; original annotations are retained in the crosswalk.

AronowAudit.P024.assignment_total_complement: observed used

AronowAudit.P024.assignment_total_indicator: observed used

AronowAudit.P024.cluster_population_probability_bound: observed used

AronowAudit.P024.complete_joint_probability: observed used

AronowAudit.P024.complete_marginal: observed used

AronowAudit.P024.complete_pair_inclusion: observed used

AronowAudit.P024.complete_probability_count: observed used

AronowAudit.P024.complete_weighted_moments: observed used

AronowAudit.P024.exchangeable_weighted_moments: observed used

AronowAudit.P024.expectation_mono: observed used

AronowAudit.P024.finite_chebyshev: observed used

AronowAudit.P024.finite_mean_abs_bound: observed used

AronowAudit.P024.finite_ratio_good_event: observed used

AronowAudit.P024.finite_sample_probability_bound: observed used

AronowAudit.P024.finite_variance_identity: observed used

AronowAudit.P024.independent_exclusion_probability: observed used

AronowAudit.P024.independent_marginal: observed used

AronowAudit.P024.independent_pair_moment: observed used

AronowAudit.P024.independent_weighted_moments: observed used

AronowAudit.P024.normalized_total_moments: observed used

AronowAudit.P024.population_effect_normalized: observed used

AronowAudit.P024.population_estimator_normalized: observed used

AronowAudit.P024.population_total_positive: observed used

AronowAudit.P024.probability_bounds: observed used

AronowAudit.P024.probability_mono: observed used

AronowAudit.P024.probability_not: observed used

AronowAudit.P024.probability_union_le: observed used

AronowAudit.P024.ratio_error_bound: observed used

AronowAudit.P024.two_arm_ratio_bound: observed used

AronowAudit.P024.uniform_probability_card: observed used

AronowAudit.P024.variance_const_sub: observed used

AronowAudit.P024.weight_square_bounds: observed used

AronowAudit.P024.weighted_mean_abs_bound: observed used

Source and evidence metadata · sap-2026-11-08-assured-release

P024-T6

Section ‘Between-Subjects Design’, first paragraph (unnumbered statement that the difference in group means is unbiased for the average causal effect)

p. 170 (PDF p. 4)

between-subjects design, p. 170 (PDF p. 4). For explicit subclasses of the paper's equal-probability individual-assignment description, the treatment-arm outcome mean minus the control-arm outcome mean is unbiased for the finite subject-pool average effect. Potential outcomes are fixed, assignment is implemented as planned, all outcomes are observed, there is no interference, and assignment is the sole randomness. The design is complete individual randomization with 1 <= m < N, or independent common-p individual assignment, 0 < p < 1, conditional on both arms being nonempty. The result also covers fixed counts almost surely with actual common probabilities m/N, and every nonempty-arm law uniform conditional on the realized count. The Bernoulli conditioning event has positive probability; its conditional masses are the parent masses divided by that probability. Population sizes are at least two and both realized arms have positive denominators. The usual measure-theoretic conditions are assumed; all laws, outcomes, and expectations in this finite formulation are measurable and finite.

Historical component credits

Proved AronowAudit.P024.fixed_dim_ht Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Auxiliary fixed-divisor algebra is already incorporated in the retained observed-denominator result.
Proved AronowAudit.P024.individual_dim_unbiased Limited support; baseline fidelity: narrower; manifest: used; used in Solution; The retained observed-denominator result already incorporates this fixed-divisor intermediate.
Proved AronowAudit.P024.individual_observed_dim_unbiased Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': "P024/P024_Designs.lean:1053 does use the actual observed group denominators through unit-weight clusterDIM, matching the estimator named in T6. Nevertheless, hcount at P024/P024_Designs.lean:1056 still requires exactly the same m treated subjects on every assignment. Identical assignment probabilities and assumptions (a)-(c) do not imply that restriction; for example, an exchangeable law supported on both singleton and two-subject treated sets is excluded. Thus changing the denominator representation did not recover the printed claim's random-count cases.", 'suggested_fix': 'Add a random-count observed-DIM theorem under a sufficient law such as conditional equal inclusion probabilities within each treated-count stratum, including independent common-p assignment conditioned on both arms being nonempty. Explicitly state that design/domain correction in T6 rather than claiming that arbitrary equal unconditional marginals suffice.', 'second_review': "P024/P024_Designs.lean:1053-1058 uses actual observed denominators through unit-weight clusterDIM, whose definition is at P024/P024_Designs.lean:248. Nevertheless, hcount at line 1056 still requires the same m treated subjects on every assignment, whereas T6 requires identical assignment probabilities without fixing the count. An exchangeable law on the six nonempty proper subsets of three subjects satisfies the target's probability condition and leaves both arms nonempty but cannot instantiate this theorem, so the narrowing needs a substantive statement repair.", 'severity': 'major'}
Proved AronowAudit.P024.complete_individual_dim_unbiased Limited support; baseline fidelity: narrower; manifest: not used; not used in Solution; Uniform fixed-count assignment is subsumed by the retained fixed-count common-marginal result.

Formal result: restricted

Scope: Require either a treatment count m fixed almost surely, 0<m<N, with every actual unit marginal m/N, or an assignment law on proper nonempty subsets uniform conditional on its treatment count. This includes complete individual randomization and independent common-p individual assignment conditioned on nonempty arms. Arbitrary equal-marginal dependent laws with varying counts are not covered.

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:7197efe3a83e2f3a1e70b19ee3f9e72fd18dc1a0c843182c6aa42379541da67c

Effective claim (source TeX): “The most basic experimental design is one in which subjects are assigned individually and with identical probability to treatment or control conditions. Assuming that (a) the random assignments are implemented as planned and outcomes are observed for all subjects, (b) the only systematic factor that causes the treatment group to differ from the control group is the intended treatment, and (c) subjects respond solely to the treatment that they receive (and are unaffected by treatments administered to others), the between-subjects design provides the basis for unbiased estimation of the average causal effect of the treatment in the subject pool (Gerber & Green, 2012). Indeed, the estimation of the average causal effect is simple and intuitive: The average outcome in the control group is subtracted from the average outcome in the treatment group.”

: SAP.P024.T6WholeStatement_W2

Theorem: SAP.P024.T6

Statement hash: 343354f22fbc93e56f9016b7e71179eb7e34690b2a93d394713a1d352d92bf55

Statement definition: SAP.P024.T6WholeStatement_W2

Certificate. 20261104T192849Z

Declaration review: faithful; 2026-11-04T19:44:53Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Manifest annotation reconciliation: Effective use follows the source-bound dependency observation; original annotations are retained in the crosswalk.

AronowAudit.P024.fixed_dim_ht: observed used

AronowAudit.P024.individual_dim_unbiased: observed used

Source and evidence metadata · sap-2026-11-08-assured-release

P024-T7

Section ‘Between-Subjects Clustered Design’, second paragraph; audit correction: conditional unbiasedness for positive equal-size clusters and a conditional unequal-size bias witness under an independent Bernoulli parent assignment

p. 171 (PDF p. 5); the estimand is named on p. 172 (PDF p. 6)

corrected conditional clustering component, p. 171 (PDF p. 5), with the individual-level estimand on p. 172 (PDF p. 6). For K >= 2 independent common-p Bernoulli cluster assignments, 0 < p < 1, the nonempty-arm event has probability 1-p^K-(1-p)^K > 0, for arbitrary positive cluster sizes. Conditional masses are the actual parent masses divided by that probability. Equal positive cluster sizes give unbiased individual difference in means conditionally on this event, and each allocation equals the difference of the two cluster means. For sizes (1,2), zero control outcomes and treatment outcomes zero in the small cluster and one in the large cluster, the fair parent law instead gives effect 2/3 and conditional mean 1/2, with the two conditional allocations equiprobable and estimates zero and one. Every empty-arm extension is allowed. Independence refers to the parent law.

Historical component credits

Proved AronowAudit.P024.Wave1.independent_unequal_size_bias_on_defined_domain Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': "P024/Wave1/Gap/DecisionT7.lean:58 gives an independent four-allocation parent and explicitly identifies its nonempty-arm conditional law, but the expectation 1/2 and its disagreement with the individual effect 2/3 at P024/Wave1/Gap/DecisionT7.lean:70 are under that conditional law. This leaves an unconditioned reading of the supplied printed target unestablished, so the target-to-statement mismatch is real. The reviewer's not_faithful characterization overstates the independence problem: conditioning induces dependence, but the actual parent randomization remains fully independent and the theorem says so explicitly. Since record.json identifies precisely this conditional bias witness as the approved repair, the necessary change is to encode that correction in the target rather than reject the witness as a dependent parent design.", 'suggested_fix': "State the corrected unequal-size target explicitly: under an independent Bernoulli(1/2) parent for two clusters of sizes 1 and 2, with all control outcomes zero and treatment outcomes zero in the small cluster and one in the large cluster, E[DIM | both arms nonempty] = 1/2 != 2/3. Preserve the independent parent and normalized conditional-law identification already exported by Lean, and remove any claim of an unconditioned expectation from this row's target.", 'second_review': "P024/Wave1/Gap/DecisionT7.lean:58-77 exports the independent four-allocation parent, its explicitly normalized nonempty-arm law, and conditional expectation 1/2 versus individual effect 2/3 for sizes 1 and 2. ConditionalNonemptyDesign at P024/P024_Designs.lean:1571 is uniform on nonempty proper subsets, so the dependent conditional assignments do not contradict the separately asserted independent parent. This is the approved conditional unequal-size witness identified in the supplied row and adjudication, but T7's claim text still supplies only the printed unconditional prose; record the corrected conditional bias claim in the target.", 'severity': None}
Proved AronowAudit.P024.equal_size_dim Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; Judged as the described auxiliary size-cancellation identity. It neither assumes nor exports unbiasedness; for any nonempty two-arm allocation one may choose m to be its treated count.
Proved AronowAudit.P024.Wave1.equal_cluster_independent_population_on_defined_domain Historical support retained; baseline fidelity: ok; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P024/Wave1/Gap/DecisionT7.lean:17 asserts independence for the Bernoulli parent, but its unbiasedness conclusion at P024/Wave1/Gap/DecisionT7.lean:29 is only under bernoulliNonemptyDesign, the normalized nonempty-arm law defined at P024/P024_IndependentCompletion.lean:351. This does not establish an unconditioned expectation under the printed independent-assignment claim: with two equal one-person clusters, y0=(0,0), y1=(0,1), p=1/2 and zero empty-arm extensions, the parent expectation is 1/4 while the conditional expectation and effect are 1/2. However, record.json expressly identifies this row as the approved nonempty-arm conditional repair, and the Lean statement covers that corrected component without losing equal-size finite populations on its defined domain. The real mismatch is that the supplied target still gives the printed text instead of the recorded correction; the approved conditional theorem itself does not need an unconditional conclusion.', 'suggested_fix': "Replace the credited equal-size component's target text with the explicit audit correction: for K >= 2 positive equal-size clusters, arbitrary fixed individual potential outcomes, and independent common-p assignment with 0 < p < 1, E[DIM | both arms nonempty] = the individual-level average effect; no unconditional equality for arbitrary empty-arm extensions is asserted. Keep the existing Lean statement attached to that corrected target.", 'second_review': "P024/Wave1/Gap/DecisionT7.lean:17-31 establishes an independent common-p parent, a positive nonempty-arm event, its normalized conditional law, and conditional unbiasedness for arbitrary positive equal-size individual populations. The conditional law is explicitly defined at P024/P024_IndependentCompletion.lean:351, so this matches the approved conditional correction identified in the supplied row and adjudication. The supplied T7 claim text instead retains the printed group-averages paragraph and omits that correction; replace the credited component's target with the conditional statement, since unconditional unbiasedness with arbitrary empty-arm extensions is a materially different assertion.", 'severity': None}

Formal result: whole

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: CT-02:P024-T7:accepted

Effective claim (source TeX): [Printed statement] “Clustered assignment poses two statistical complications. The first is that when clusters contain different numbers of subjects, the standard estimation approach—comparing treatment and control group averages—no longer gives unbiased estimates of the average treatment effect among all subjects (Middleton, 2008, Middleton & Aronow, 2011). The difference-in-means estimator is consistent, which is to say that it converges to the correct answer as the number of clusters becomes very large, but bias may be appreciable in studies with only a few dozen clusters of varying size. In order to mitigate the threat of bias and to improve efficiency, researchers working with clusters of varying sizes may wish to block on cluster size before randomly assigning clusters in each block (Imai, King, & Nall, 2009; Broockman & Green, in press).” [Corrected formal component: the audit's nonempty-arm conditional reading.] Let $K \ge 2$ clusters have positive integer sizes $n_j$ and arbitrary fixed individual potential outcomes $Y_{ij}(0),Y_{ij}(1)$, under the paper's conditions (a)–(c). The parent assignment treats whole clusters independently with a common probability $p$, $0 < p < 1$: $Z_j \sim \mathrm{Bernoulli}(p)$. Put $N = \sum_{j=1}^{K} n_j$, $\tau = N^{-1}\sum_{j=1}^{K}\sum_{i=1}^{n_j}[Y_{ij}(1)-Y_{ij}(0)]$, and $A = \{0 < \sum_j Z_j < K\}$. Then $\Pr(A)=1-p^K-(1-p)^K>0$. On $A$, define the individual-level difference in means $$\widehat\tau = \frac{\sum_j Z_j\sum_i Y_{ij}(1)}{\sum_j Z_j n_j} - \frac{\sum_j(1-Z_j)\sum_i Y_{ij}(0)}{\sum_j(1-Z_j)n_j}.$$ (i) [Equal cluster sizes.] If $n_j=n>0$ for every $j$, then $\mathrm E[\widehat\tau\mid A]=\tau$. For each allocation in $A$, with $m=\sum_j Z_j$, the estimator is also $m^{-1}\sum_{j:Z_j=1}\bar Y_j(1)-(K-m)^{-1}\sum_{j:Z_j=0}\bar Y_j(0)$. (ii) [Unequal cluster sizes; bias is possible.] Take $K=2$, $(n_1,n_2)=(1,2)$ and an independent $\mathrm{Bernoulli}(1/2)$ parent assignment. Set all control outcomes to zero, the small cluster's treatment outcome to zero, and both treatment outcomes in the large cluster to one. Here $\tau=2/3$ and $\Pr(A)=1/2$; conditional on $A$ the two allocations are equiprobable and give $\widehat\tau=0$ and $1$, so $\mathrm E[\widehat\tau\mid A]=1/2\ne 2/3$. [Scope] These expectations use the normalized conditional law $\Pr(\mathbf Z=\mathbf z\mid A)=\Pr(\mathbf Z=\mathbf z)/\Pr(A)$ for $\mathbf z\in A$. Independence describes the parent assignment, not its conditional law. Arbitrary extensions of the estimator to empty-arm allocations do not affect either conditional conclusion; no unconditional equality for such extensions is asserted. The printed consistency clause and blocking recommendation are not conclusions of this corrected formal component.

: SAP.P024.T7WholeStatement

Theorem: SAP.P024.T7

Statement hash: 9618e111b63be8e61f87cbdd6a0d3892bc2752aac7930e39a5a271ce3994572e

Statement definition: SAP.P024.T7WholeStatement

Certificate. 20261104T192849Z

Declaration review: faithful; 2026-11-04T19:42:53Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Source and evidence metadata · sap-2026-11-08-assured-release

P024-T8

Section ‘Restricted Random Assignment’, first paragraph, with Note 4 (unnumbered statement that under a blocked design the overall estimate is a weighted average of the estimates for each block)

p. 173 (PDF p. 7); Note 4 on p. 177 (PDF p. 11)

fixed-quota specialization of weighted blocked estimation, p. 173 (PDF p. 7), and Note 4, p. 177 (PDF p. 11). The actual-probability Horvitz–Thompson estimate equals on every allocation the population-share weighted average of ordinary within-block differences in means, and both estimators are unbiased for the individual average effect among randomized units.

Historical component credits

Proved AronowAudit.P024.blocked_weighted_identity Limited support; baseline fidelity: narrower; manifest: used; used in Solution; The retained actual-observed-means identity incorporates this fixed-divisor intermediate.
Proved AronowAudit.P024.blocked_estimator_unbiased Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': 'P024/P024_Designs.lean:426 assumes fixed nonempty block counts and equal within-block marginal treatment probabilities m_b/N_b at P024/P024_Designs.lean:430. T8 and Note 4 describe weighted block estimation and inverse-probability weighting without restricting the design to those fixed counts or equal within-block probabilities. The theorem therefore omits blocked designs with random counts or unequal actual unit probabilities, even though HT unbiasedness applies to such designs when both treatment probabilities are positive. Allowing dependence across blocks is appropriate, and restricting a finite design to its positive-mass support resolves the separate zero-mass-schedule issue without repairing this narrowing.', 'suggested_fix': "Parameterize the block HT estimator by each unit's actual marginal probability, assume 0 < pi_{b,i} < 1, and prove its unbiasedness and the corresponding block-HT weighted-average result without hcount or equal-marginal assumptions. Keep the existing fixed-count raw difference-in-means conclusion as a specialized corollary.", 'second_review': 'P024/P024_Designs.lean:426-436 assumes both fixed nonempty block counts and common within-block marginal probabilities m_b/N_b before asserting unbiasedness of the HT and block-weighted estimators. The credited inverse-probability estimation component of T8 and Note 4 does not impose those restrictions; actual-probability HT estimation remains unbiased with random counts and unequal positive unit probabilities. The specialized blockedHT definition at P024/P024_Designs.lean:389 therefore needs an actual-unit-probability generalization, while the current raw block-mean result can remain a corollary.', 'severity': 'major'}
Proved AronowAudit.P024.blocked_actual_means_identity Limited support; baseline fidelity: narrower; manifest: used; used in Solution; {'source': 'adjudication+second review', 'adjudicator': "P024/P024_Designs.lean:1069 requires a fixed count m_b with 0 < m_b < N_b in every block on every assignment, whereas T8's printed weighted-average claim and Note 4 do not impose that restriction on randomization within blocks. Moreover, blockedHT at P024/P024_Designs.lean:389 uses stipulated fractions m_b/N_b, and this law-free identity has no premise identifying those fractions with actual treatment probabilities. The algebra is valid for its stipulated denominators, but it is not the general actual-probability HT/block decomposition named by the target. Variable block counts and unequal treatment probabilities within a block are substantive omitted cases, not indexing conventions.", 'suggested_fix': 'State the general decomposition using actual unit probabilities pi_{b,i} and block-specific HT estimates: the overall HT estimate equals the sum of block HT estimates weighted by N_b/N, without fixed-count assumptions. Retain the current raw block-mean identity as a corollary for fixed nonempty counts and probabilities pi_{b,i}=m_b/N_b, explicitly identifying those probabilities whenever it is described as the actual HT estimator.', 'second_review': "P024/P024_Designs.lean:1069-1076 requires fixed, strictly positive treatment and control counts in every block to identify actual block group means. T8's weighted-average statement and Note 4's inverse-probability weighting impose no such count restriction, and blockedHT at P024/P024_Designs.lean:389 substitutes m_b/N_b without identifying it with actual unit treatment probabilities. Generalize to actual unit probabilities and weighted block HT estimates, retaining the raw-means identity as the fixed-count, equal-probability corollary; the excluded random-count and unequal-probability designs are substantive.", 'severity': 'major'}

Formal result: restricted

Scope: Assume a predetermined interior treatment quota m_b in every block on every assignment and actual common within-block treatment probabilities m_b/N_b. Establish the population-share weighted raw-mean identity and unbiasedness under these conditions; no result for variable block treatment counts or unequal within-block propensities is asserted. Uniform subsets and independence across blocks are not required.

Formal statement · Proof · Definitions

Assumptions and verification

Claim revision: catalog:a209c5c32f62371bfc007848898a044ec72d1bc46a6f80655aa6552d29388f7b

Effective claim (source TeX): [p. 173 (PDF p. 7)] “In order to accommodate the campaign, Gerber et al. (2011) excluded the two largest markets from the random assignment and blocked the remaining markets so that a certain degree of geographic and political balance was ensured. The first restriction effectively limited the broadcast TV experiment to the 18 smaller markets. The second restriction meant that data had to be analyzed in a manner that took the blocked design into account; in effect, the overall estimate of the average causal effect represents a weighted average of the estimate for each block.$^{4}$” [Note 4, p. 177 (PDF p. 11)] “4. For a step-by-step explanation of the inverse-probability-weighted (Horvitz & Thompson, 1952) estimation of average causal effects under a blocked design or restricted random assignment, see Gerber and Green (2012, chapters 3 and 4, with technical details in Aronow & Middleton, 2013).”

: SAP.P024.T8WholeStatement_W3

Theorem: SAP.P024.T8

Statement hash: c9087c95bf71db79a5bd4bbf25f92c3c5d4f1048c0ffdbd96b315142cd3eda5e

Statement definition: SAP.P024.T8WholeStatement_W3

Certificate. 20261104T192849Z

Declaration review: faithful; 2026-11-04T19:42:53Z; binding v2

Evidence scope: Certification checks formal proof objects. Statement fidelity is a model judgment. Author review is separate.

Proof dependencies: Independently extracted

Manifest annotation reconciliation: Effective use follows the source-bound dependency observation; original annotations are retained in the crosswalk.

AronowAudit.P024.blocked_weighted_identity: observed used

Source and evidence metadata · sap-2026-11-08-assured-release

Reproduction

What I recalculated or reran from public materials. "Agrees", "Agrees in part" or "Disagrees" compares my result with the printed one. "Not compared" means the comparison could not be made.

Status: Not applicable
Agreement: Not applicable