Corrections to: The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial

The problem

The experiment assigned children to a side of the porch. The Obama side had a large photograph of Michelle Obama and a poster with the name “Obama”; I refer to assignment to this side as the Michelle Obama porch condition. The comparison side had a photograph of Ann Romney (2012) or of Hillary Clinton (2014, 2015), or no photograph (2014, 2015), and posters with the name “Romney” or “Clinton”. Exposure to the Let’s Move campaign, Michelle Obama’s initiative to reduce childhood obesity, was not varied, and the recorded variables include no measure of campaign exposure, recall or the proposed reminder pathway. Under the assumptions of the design (valid assignment, consistency and no interference), the contrast between the two sides estimates the effect of the Michelle Obama porch condition. The first sentence of the Conclusions attributes that effect to the campaign. The attribution requires further assumptions linking at least part of the contrast to the campaign. Recognition of the person, a political association, or another response to the name or the photograph could account for the difference without any campaign contribution. The Introduction explicitly motivates campaign reinforcement through the figurehead’s association with the health message. It proposes this pathway without imposing or testing a restriction that excludes other pathways, or otherwise ensures that some of the contrast comes from the campaign.

The paper says that it cannot evaluate the entire nationwide campaign, and it limits its conclusion to Halloween in New Haven. Neither qualification supplies the missing restriction, since the attribution is not identified even for the children on this porch. The Introduction states a research aim and a proposed explanation: a subconscious reminder may reinforce the campaign’s message. That passage need not be corrected as though it asserted a separately demonstrated mechanism. The categorical attribution is in the abstract’s Conclusions; p. 5/6 continues it by calling the estimate “the treatment effect of her campaign” in its conditional discussion of extrapolation.

The correction weakens the conclusion and changes no number. To establish some campaign contribution, one could assume that noncampaign contributions are zero, or bound their combined contribution below a positive porch-condition effect. The paper supplies neither restriction. The correction does not say that the campaign had no influence, that the proposed explanation is unreasonable, or that there is no contextual evidence favoring it. These data do not settle the actual campaign contribution in either direction. I also do not count the wording of the Results sentence as an error. Since the photograph and the name poster were always assigned together, the design cannot separate their contributions. But “viewing a photograph of Michelle Obama’s face” can reasonably be read as a label for the Michelle Obama porch condition.

Evidence an exact counterexample

The evidence is an exact counterexample: two models of the experiment that agree on everything it can observe and disagree on the effect of the campaign. The mathematical notation is introduced for this check; the paper itself names the Obama and Comparison conditions without defining a mediator or campaign-effect parameter. Fix a year $y$, and let $p_{y0}$ and $p_{y1}$ be the proportions choosing fruit on the comparison side and on the Obama side in the released data. Let $L_y$ be the least common multiple of their denominators, and set $t_{y0} = L_y p_{y0}$ and $t_{y1} = L_y p_{y1}$, which are integers. Each child has a latent value $R$, uniform on $\{1, \dots, L_y\}$ and independent of the assigned side $Z \in \{0,1\}$, where $Z = 1$ denotes the Obama side. Let $A \in \{0,1\}$ indicate whether the campaign is present, so that $A = 1$ throughout the study. This counterfactual holds the specified porch displays fixed when the campaign is absent. I write $1$ for fruit and $0$ for candy, and I define the potential outcomes in the direct-display model and in the campaign-mediated model by

Setting $a = 1$, I have $Y^{\mathrm{D}}_y(z,1;r) = Y^{\mathrm{M}}_y(z,1;r)$ for every $z$ and $r$. Since $A = 1$ throughout the study, the two models give each child the same outcome on either side, and they give the same distribution to every data set that the experiment can produce, at any sample size. In both models the probability of choosing fruit is $t_{y0}/L_y = p_{y0}$ on the comparison side and $t_{y1}/L_y = p_{y1}$ on the Obama side.

The two models differ in the effect of the campaign. In the direct-display model, $a$ does not enter the definition, which implies that $Y^{\mathrm{D}}_y(z,1;r) - Y^{\mathrm{D}}_y(z,0;r) = 0$ for every $z$ and $r$. In the campaign-mediated model, the difference $Y^{\mathrm{M}}_y(1,1;r) - Y^{\mathrm{M}}_y(1,0;r)$ equals $1$ if $t_{y0} < r \le t_{y1}$ and $0$ otherwise. Taking the mean over the latent value, I obtain

The right side is positive in each of the three years, as the table shows.

A quantity that the experiment identifies takes the same value in any two models with the same observable distribution. The effect of the campaign is $0$ in the direct-display model and positive in the campaign-mediated model. I conclude that the experiment does not identify it. Both models assume random assignment, consistency and no interference, and in both the Michelle Obama porch condition has an effect. They can also share the stated public association between Michelle Obama and the campaign: in the direct-display model, that association does not affect the choice. Calibration to the observed proportions specifies an exact counterexample; it does not assert that the unknown population probabilities equal those sample proportions.

The construction also respects the disclosed comparison conditions. Applying it separately to each comparison protocol reproduces every integer allocation compatible with S1 Table and the pooled counts: Romney in 2012; Clinton and no photograph in 2014; and either compatible Clinton/no-photograph split in 2015. Averaging over protocols leaves the campaign effect on the Obama side equal to $p_{y1}-p_{y0}$ in the campaign-mediated model and zero in the direct-display model. The compatible splits are not recovered participant labels. Both models can include the disclosed 2014 prompting and the separately randomized 2015 prompt with no effect on choice. The paper does not require a nonzero prompt effect, so this completion is permitted. Pooling the prompt is not itself a defect, and these comparisons do not resolve the separate questions about queue assignment or interference.

A recalculation from the released data, with the estimator in the replication code, reproduces the printed estimates. The correction changes none of them.

What survives

The yearly proportions on each side, the pooled contrast with inverse probability weights, the confidence intervals, the $p$-value and Fig. 1 are unaffected. The 19% relative increase stands as an estimate of the effect of the Michelle Obama porch condition, conditional on the assumptions about assignment, consistency and interference; the questions about assignment in 2014 and 2015 are separate from this correction, and neither the paper nor the released data settle them. What changes is the categorical campaign conclusion and the campaign-effect wording on p. 5/6. The Introduction’s proposed reminder mechanism and stated research purpose can remain, together with the explicit description of the randomized intervention and the geographic, Halloween, repeat-participation and priming qualifications. No later mathematical result depends on the campaign attribution.

The problem

The parenthetical describes Class 1 obesity by a fixed range of body mass index. Reference [4] concerns US children aged 2 to 19 and defines this class, which it writes as class I, as BMI at or above the 95th percentile for age and sex, with each weight category including the higher ones; fixed values of BMI appear in its definitions only as alternatives for classes II (35) and III (40).

Evidence a check of the printed page

The evidence is a check of the cited source, Skinner, Perrin and Skelton, Obesity 24(5):1116–1123 (2016), in the version of record. Its Methods define “obesity as $\ge$95th percentile”, and the first footnote of its Table 1 states that “Each weight category is inclusive of the higher weight categories”. By its Table 2, the prevalence of this class was 14.6% in 1999–2000 and 17.4% in 2013–2014, which gives a relative increase of $(17.4 - 14.6)/14.6 \approx 0.192$.

What survives

The 19% increase stands, and no quantity in the trial uses this prevalence or the BMI range; the 19% in the abstract is a different quantity, the relative increase in fruit choice.

Intended meaning

Children were allocated to two sides of the porch. In 2014 and 2015 the comparison side had two conditions, a photograph of Hillary Clinton or no photograph, and the analysis pools these conditions and the 2015 priming assignment. This reading agrees with the page, and every entry of S1 Table is compatible with the released data. The paper does not say how or when the two conditions were allocated, and the released data record neither the comparison condition nor the priming assignment. The Clinton and No photo rows of S1 Table cannot be regenerated from the released data: in 2015, fruit counts of 55 and 58 or of 56 and 57 in these two rows both match the printed means and standard errors. The pooled estimates do not use these labels.

Intended meaning

In this parenthesis, “dropped from analysis” means that the participant is omitted from the outcome means and the outcome regressions, which use 1222 rows; the replication code does this. Her assignment still enters the estimate of the treatment probabilities, which uses all 1223 rows and gives the 2012 probability $96/165$ in place of $95/164$. The weighting method, as the paper describes it, allows this. Deleting her before the weights are estimated changes the pooled difference from 0.045414 to 0.045371 and the $p$-value from 0.0908 to 0.0910, and no printed number changes.