{ "claims": [ { "claim": "Viewing a photograph of Michelle Obama's face, relative to control conditions, caused children to be 19% more likely to choose fruit over candy.", "claim_id": "P035-H1", "related_corrections": [ "P035-E001" ], "tied_to": "Abstract, Results (p. 2/6); Introduction (p. 2/6); ratio of the two pooled means in Fig 1 and the Results text" }, { "claim": "Pooled over three Halloweens, 28.1% (95% CI 23.8-32.4%) of participants in the Obama treatment chose fruit, against 23.5% (20.5-26.5%) in the pooled comparison group.", "claim_id": "P035-H2", "related_corrections": [ "P035-N004" ], "tied_to": "Results and conclusions (pp. 4/6-5/6); Fig 1, pooled rows; replication script halloweenforest_PLOS_final.R" }, { "claim": "The pooled difference is statistically significant at the 0.10 level (two-sided $p=0.091$).", "claim_id": "P035-H3", "related_corrections": [ "P035-N004" ], "tied_to": "Results and conclusions (p. 5/6); replication script, section '# pvalues' (lmfit1)" }, { "claim": "Michelle Obama's initiative to reduce childhood obesity has influenced children's dietary preferences; the photograph experiment evaluates the initiative's microfoundations.", "claim_id": "P035-H4", "related_corrections": [ "P035-E001", "P035-N003" ], "tied_to": "Abstract, Conclusions and Introduction (p. 2/6); pooled contrast in Fig 1; intervention description (p. 3/6); S1 Table" }, { "claim": "The study is a multi-year randomized control trial in which trick-or-treaters were randomly assigned to the Michelle Obama side or the Comparison side of the porch.", "claim_id": "P035-H5", "related_corrections": [], "tied_to": "Title; Abstract, Design and Intervention (p. 1/6); Methods, first paragraph (pp. 2/6-3/6); S1 Table" }, { "claim": "There were 1223 participants (165 in 2012, 422 in 2014, 636 in 2015), on average 8.5 years old and 53% male, with yearly averages 7.8/58%, 8.5/56% and 8.7/49%.", "claim_id": "P035-H6", "related_corrections": [], "tied_to": "Abstract, Participants (p. 1/6); Methods (pp. 2/6-3/6); released data; S1 Table ($N$ column)" }, { "claim": "Eligible participants were trick-or-treaters over the age of three who approached the house.", "claim_id": "P035-H7", "related_corrections": [], "tied_to": "Abstract, Eligibility (p. 1/6); Methods, second paragraph (p. 3/6); released data (age)" }, { "claim": "The yearly arm means and the disaggregated comparison arms are as shown in Fig 1 and S1 Table.", "claim_id": "P035-H8", "related_corrections": [ "P035-N003" ], "tied_to": "Results and conclusions (p. 5/6, 'We present the estimates for each year in Fig 1'); Methods (p. 3/6, pointer to S1 Table); Fig 1; S1 Table" }, { "claim": "Findings are substantively unchanged by imputing either a fruit or a candy outcome for the one participant with a missing outcome.", "claim_id": "P035-H9", "related_corrections": [ "P035-N004" ], "tied_to": "Methods, last paragraph (p. 3/6); released data and script" }, { "claim": "Whether the observed effect extends beyond these Halloween visitors at this New Haven porch requires further research.", "claim_id": "P035-H10", "related_corrections": [], "tied_to": "Abstract, Conclusions (p. 2/6); Results and conclusions (p. 5/6); the stated setting and sample design" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the end of the abstract. Under the heading Results, the text reads: We estimate that viewing a photograph of Michelle Obama's face relative to control conditions caused children to be 19 percent more likely to choose fruit over candy. Under the heading Conclusions, a red outline surrounds the first sentence: Michelle Obama's initiative to reduce childhood obesity has influenced children's dietary preferences. The remaining sentence of the Conclusions, on whether this influence extends beyond Halloween in New Haven, is not outlined.", "caption": "Abstract, Results and Conclusions, p. 2/6 (PDF p. 2). The outline marks the first sentence of the Conclusions, which attributes the estimated effect to the initiative.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E001.png" }, { "alt": "Crop of five lines from page 5. The sentence reads: In particular, this is a strongly Democratic neighborhood, and if children in Democratic households are either more aware of Michelle Obama's campaign or more favorably disposed to it, we would generate an overestimate of the treatment effect of her campaign on the general population of those in the United States. A red outline surrounds the words the treatment effect of her campaign.", "caption": "Results and conclusions, second paragraph, p. 5/6 (PDF p. 5). The outline marks the phrase that calls the estimate the treatment effect of the campaign.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E001-p5.png" }, { "alt": "Two dot plots side by side under the title: Two models give the same fruit choices on each side of the porch but different effects of the campaign. The left plot, headed Observable in the experiment, the two models agree, has six rows, one for each year and side. In each row a filled blue dot for the direct-display model sits inside an open orange ring for the campaign-mediated model at the same value: 20.3 percent and 25.3 percent for the comparison and Obama sides in 2012, 21.9 and 23.6 percent in 2014, 25.5 and 31.8 percent in 2015. The right plot, headed Not observable, the two models differ, has three rows, one for each year. The blue dot for the direct-display model is at zero in every row, and the orange ring for the campaign-mediated model is at 5.0 percentage points in 2012, 1.7 in 2014 and 6.3 in 2015.", "caption": "The counterexample for P035-E001, by year. Left: the probability of choosing fruit on each side of the porch, which is the same in the direct-display model (filled dot) and in the campaign-mediated model (open ring) and equals the proportion in the released data. Right: the mean effect of the campaign on fruit choice on the Obama side, which is zero in the direct-display model and 5.0, 1.7 and 6.3 percentage points in the campaign-mediated model.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E001-models.png" } ], "formal_release": null, "formal_statements": [ { "id": "P035-T1", "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-T1" }, { "id": "P035-T3", "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-T3" } ], "headline": "The experiment estimates the effect of assignment to the Michelle Obama porch condition; it does not establish that her initiative to reduce childhood obesity caused the difference.", "id": "P035-E001", "location": "Abstract, Conclusions, first sentence, p. 2/6 (PDF p. 2), and the campaign-effect wording in Results and conclusions, p. 5/6 (PDF p. 5); read with the hypothesis and rationale in the Introduction, third paragraph, p. 2/6", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": [ "P035-T1", "P035-T3" ], "text": { "as_printed": "\\emph{Abstract, Results (p.~2/6).} We estimate that viewing a photograph of Michelle Obama's face relative to control conditions caused children to be 19\\% more likely to choose fruit over candy.\n\n\\emph{Abstract, Conclusions (p.~2/6).} \\hl{Michelle Obama's initiative to reduce childhood obesity has influenced children's dietary preferences.} Whether this influence extends beyond Halloween trick-or-treating in New Haven, CT on the porch of an economics professor requires further research.\n\n\\emph{Introduction, context (p.~2/6).} While the campaign gained publicity, an experimental evaluation of the effects of the entire campaign would not be possible due to the initiative being rolled out across the United States through mass media.\n\nHowever, we have experimentally assessed the microfoundations of Let's Move in order to evaluate whether the public health campaign altered dietary decision-making. When children are exposed to the image of a figurehead regularly associated with the public health campaign's message, the subconscious reminder may reinforce the message.\n\n\\emph{Results and conclusions (p.~5/6).} In particular, this is a strongly Democratic neighborhood, and if children in Democratic households are either more aware of Michelle Obama's campaign or more favorably disposed to it, we would generate an overestimate of \\hl{the treatment effect of her campaign} on the general population of those in the United States.", "corrected": "\\emph{Abstract, Conclusions.} \\hl{These trials estimate that assignment to the implemented Michelle Obama porch condition increased the probability of choosing raisins rather than candy relative to the pooled comparison conditions. Campaign reinforcement is a possible explanation; the experiment does not establish that the initiative or its health message caused the difference.} Whether \\hl{the estimated porch-condition effect} extends beyond Halloween trick-or-treating in New Haven, CT on the porch of an economics professor requires further research.\n\n\\emph{Results and conclusions, p.~5/6.} In particular, this is a strongly Democratic neighborhood, and if children in Democratic households are either more aware of Michelle Obama's campaign or more favorably disposed to it, we \\hl{could} generate an overestimate of \\hl{the effect of the implemented Michelle Obama porch condition} on the general population of those in the United States.", "evidence": "The evidence is an exact counterexample: two models of the experiment that agree on everything it can observe and disagree on the effect of the campaign. The mathematical notation is introduced for this check; the paper itself names the Obama and Comparison conditions without defining a mediator or campaign-effect parameter. Fix a year $y$, and let $p_{y0}$ and $p_{y1}$ be the proportions choosing fruit on the comparison side and on the Obama side in the released data. Let $L_y$ be the least common multiple of their denominators, and set $t_{y0} = L_y p_{y0}$ and $t_{y1} = L_y p_{y1}$, which are integers. Each child has a latent value $R$, uniform on $\\{1, \\dots, L_y\\}$ and independent of the assigned side $Z \\in \\{0,1\\}$, where $Z = 1$ denotes the Obama side. Let $A \\in \\{0,1\\}$ indicate whether the campaign is present, so that $A = 1$ throughout the study. This counterfactual holds the specified porch displays fixed when the campaign is absent. I write $1$ for fruit and $0$ for candy, and I define the potential outcomes in the direct-display model and in the campaign-mediated model by\n\\[\nY^{\\mathrm{D}}_y(z,a;r) = \\mathbf{1}\\{ r \\le t_{y0} + z (t_{y1} - t_{y0}) \\},\n\\qquad\nY^{\\mathrm{M}}_y(z,a;r) = \\mathbf{1}\\{ r \\le t_{y0} + a z (t_{y1} - t_{y0}) \\}.\n\\]\nSetting $a = 1$, I have $Y^{\\mathrm{D}}_y(z,1;r) = Y^{\\mathrm{M}}_y(z,1;r)$ for every $z$ and $r$. Since $A = 1$ throughout the study, the two models give each child the same outcome on either side, and they give the same distribution to every data set that the experiment can produce, at any sample size. In both models the probability of choosing fruit is $t_{y0}/L_y = p_{y0}$ on the comparison side and $t_{y1}/L_y = p_{y1}$ on the Obama side.\n\nThe two models differ in the effect of the campaign. In the direct-display model, $a$ does not enter the definition, which implies that $Y^{\\mathrm{D}}_y(z,1;r) - Y^{\\mathrm{D}}_y(z,0;r) = 0$ for every $z$ and $r$. In the campaign-mediated model, the difference $Y^{\\mathrm{M}}_y(1,1;r) - Y^{\\mathrm{M}}_y(1,0;r)$ equals $1$ if $t_{y0} < r \\le t_{y1}$ and $0$ otherwise. Taking the mean over the latent value, I obtain\n\\[\n\\mathbb{E}\\bigl[ Y^{\\mathrm{M}}_y(1,1;R) - Y^{\\mathrm{M}}_y(1,0;R) \\bigr] = \\frac{t_{y1} - t_{y0}}{L_y} = p_{y1} - p_{y0} > 0 .\n\\]\nThe right side is positive in each of the three years, as the table shows.\n\\begin{audittable}{lllll}{The two models in each year. Both models reproduce the proportions choosing fruit in the released data (second and third columns). The last two columns give the mean effect of the campaign on the Obama side in each model.}\nYear & Comparison side, $p_{y0}$ & Obama side, $p_{y1}$ & Direct-display model & Campaign-mediated model \\\\\n2012 & $14/69 \\approx 0.203$ & $24/95 \\approx 0.253$ & $0$ & $326/6555 \\approx 0.050$ \\\\\n2014 & $58/265 \\approx 0.219$ & $37/157 \\approx 0.236$ & $0$ & $699/41605 \\approx 0.017$ \\\\\n2015 & $113/444 \\approx 0.255$ & $61/192 \\approx 0.318$ & $0$ & $449/7104 \\approx 0.063$ \\\\\n\\end{audittable}\nA quantity that the experiment identifies takes the same value in any two models with the same observable distribution. The effect of the campaign is $0$ in the direct-display model and positive in the campaign-mediated model. I conclude that the experiment does not identify it. Both models assume random assignment, consistency and no interference, and in both the Michelle Obama porch condition has an effect. They can also share the stated public association between Michelle Obama and the campaign: in the direct-display model, that association does not affect the choice. Calibration to the observed proportions specifies an exact counterexample; it does not assert that the unknown population probabilities equal those sample proportions.\n\nThe construction also respects the disclosed comparison conditions. Applying it separately to each comparison protocol reproduces every integer allocation compatible with S1 Table and the pooled counts: Romney in 2012; Clinton and no photograph in 2014; and either compatible Clinton/no-photograph split in 2015. Averaging over protocols leaves the campaign effect on the Obama side equal to $p_{y1}-p_{y0}$ in the campaign-mediated model and zero in the direct-display model. The compatible splits are not recovered participant labels. Both models can include the disclosed 2014 prompting and the separately randomized 2015 prompt with no effect on choice. The paper does not require a nonzero prompt effect, so this completion is permitted. Pooling the prompt is not itself a defect, and these comparisons do not resolve the separate questions about queue assignment or interference.\n\nA recalculation from the released data, with the estimator in the replication code, reproduces the printed estimates. The correction changes none of them.\n\\begin{audittable}{lll}{The printed estimates beside the recalculation from the released data.}\nQuantity & Printed & Recalculated \\\\\nFruit choice, Obama side (95\\% interval) & 28.1\\% (23.8--32.4\\%) & 0.28066 (0.23751, 0.32381) \\\\\nFruit choice, comparison side (95\\% interval) & 23.5\\% (20.5--26.5\\%) & 0.23525 (0.20512, 0.26538) \\\\\nRelative increase & 19\\% & 0.19305 \\\\\nTwo-sided $p$ & 0.091 & 0.09078 \\\\\n\\end{audittable}", "problem": "The experiment assigned children to a side of the porch. The Obama side had a large photograph of Michelle Obama and a poster with the name ``Obama''; I refer to assignment to this side as the Michelle Obama porch condition. The comparison side had a photograph of Ann Romney (2012) or of Hillary Clinton (2014, 2015), or no photograph (2014, 2015), and posters with the name ``Romney'' or ``Clinton''. Exposure to the \\emph{Let's Move} campaign, Michelle Obama's initiative to reduce childhood obesity, was not varied, and the recorded variables include no measure of campaign exposure, recall or the proposed reminder pathway. Under the assumptions of the design (valid assignment, consistency and no interference), the contrast between the two sides estimates the effect of the Michelle Obama porch condition. The first sentence of the Conclusions attributes that effect to the campaign. The attribution requires further assumptions linking at least part of the contrast to the campaign. Recognition of the person, a political association, or another response to the name or the photograph could account for the difference without any campaign contribution. The Introduction explicitly motivates campaign reinforcement through the figurehead's association with the health message. It proposes this pathway without imposing or testing a restriction that excludes other pathways, or otherwise ensures that some of the contrast comes from the campaign.\n\nThe paper says that it cannot evaluate the entire nationwide campaign, and it limits its conclusion to Halloween in New Haven. Neither qualification supplies the missing restriction, since the attribution is not identified even for the children on this porch. The Introduction states a research aim and a proposed explanation: a subconscious reminder may reinforce the campaign's message. That passage need not be corrected as though it asserted a separately demonstrated mechanism. The categorical attribution is in the abstract's Conclusions; p.~5/6 continues it by calling the estimate ``the treatment effect of her campaign'' in its conditional discussion of extrapolation.\n\nThe correction weakens the conclusion and changes no number. To establish some campaign contribution, one could assume that noncampaign contributions are zero, or bound their combined contribution below a positive porch-condition effect. The paper supplies neither restriction. The correction does not say that the campaign had no influence, that the proposed explanation is unreasonable, or that there is no contextual evidence favoring it. These data do not settle the actual campaign contribution in either direction. I also do not count the wording of the Results sentence as an error. Since the photograph and the name poster were always assigned together, the design cannot separate their contributions. But ``viewing a photograph of Michelle Obama's face'' can reasonably be read as a label for the Michelle Obama porch condition.", "what_survives": "The yearly proportions on each side, the pooled contrast with inverse probability weights, the confidence intervals, the $p$-value and Fig.~1 are unaffected. The 19\\% relative increase stands as an estimate of the effect of the Michelle Obama porch condition, conditional on the assumptions about assignment, consistency and interference; the questions about assignment in 2014 and 2015 are separate from this correction, and neither the paper nor the released data settle them. What changes is the categorical campaign conclusion and the campaign-effect wording on p.~5/6. The Introduction's proposed reminder mechanism and stated research purpose can remain, together with the explicit description of the randomized intervention and the geographic, Halloween, repeat-participation and priming qualifications. No later mathematical result depends on the campaign attribution." }, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-E001", "work": { "arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/" }, "work_id": "P035" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of the heading Introduction and the first five lines below it. The second sentence reads: Based on an analysis of the National Health and Nutrition Examination Survey, 1999 to 2014, researchers show an estimated 19 percent increase in the prevalence of Class 1 obesity (low risk obesity, BMI 30.0 to 34.9) in the United States from 1999 to 2014, with citation 4. Red outlines surround the two halves of the parenthetical, low risk obesity, BMI 30.0 to 34.9, at the end of one line and the start of the next.", "caption": "Introduction, first paragraph, p. 2/6 (PDF p. 2). The outline marks the parenthetical that describes Class 1 obesity by the BMI range 30.0 to 34.9; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-E002.png" } ], "formal_release": null, "formal_statements": [], "headline": "Reference [4] defines Class 1 obesity in children as BMI at or above the age- and sex-specific 95th percentile, including higher obesity classes, and not as BMI 30.0 to 34.9; the 19% increase is correct.", "id": "P035-E002", "location": "Introduction, first paragraph, second sentence, p. 2/6 (PDF p. 2)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Based on an analysis of the National Health and Nutrition Examination Survey, 1999--2014, researchers show an estimated 19\\% increase in the prevalence of Class 1 obesity (\\hl{low risk obesity, BMI 30.0 to 34.9}) in the United States from 1999 to 2014 [4].", "corrected": "Based on an analysis of the National Health and Nutrition Examination Survey, 1999--2014, researchers show an estimated 19\\% increase in the prevalence of Class 1 obesity (\\hl{BMI at or above the age- and sex-specific 95th percentile, including higher obesity classes}) in the United States from 1999 to 2014 [4].", "evidence": "The evidence is a check of the cited source, Skinner, Perrin and Skelton, \\emph{Obesity} 24(5):1116--1123 (2016), in the version of record. Its Methods define ``obesity as $\\ge$95th percentile'', and the first footnote of its Table~1 states that ``Each weight category is inclusive of the higher weight categories''. By its Table~2, the prevalence of this class was 14.6\\% in 1999--2000 and 17.4\\% in 2013--2014, which gives a relative increase of $(17.4 - 14.6)/14.6 \\approx 0.192$.", "problem": "The parenthetical describes Class 1 obesity by a fixed range of body mass index. Reference [4] concerns US children aged 2 to 19 and defines this class, which it writes as class I, as BMI at or above the 95th percentile for age and sex, with each weight category including the higher ones; fixed values of BMI appear in its definitions only as alternatives for classes II (35) and III (40).", "what_survives": "The 19\\% increase stands, and no quantity in the trial uses this prevalence or the BMI range; the 19\\% in the abstract is a different quantity, the relative increase in fruit choice." }, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-E002", "work": { "arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/" }, "work_id": "P035" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of nine lines of the Methods, beginning One side of the porch, denoted Obama, had a large photo of Michelle Obama in front of the children. Red outlines surround two passages, each running over a line break. The first is the parenthetical: See S1 Table for the disaggregation of comparison groups. The second is the sentence: We pool our analysis, irrespective of this additional source of variation.", "caption": "Methods, third paragraph, p. 3/6 (PDF p. 3). The outlines mark the pointer to S1 Table for the comparison groups and the sentence that pools the analysis over the 2015 priming assignment.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-N003.png" } ], "formal_release": null, "formal_statements": [], "headline": "The paper does not say how the Clinton and no-photo conditions were allocated on the comparison side, and the released data record neither the comparison condition nor the 2015 priming assignment.", "id": "P035-N003", "location": "Methods, third paragraph, p. 3/6 (PDF p. 3); S1 Table", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "The other side--denoted \\emph{Comparison}--had a photo of Ann Romney (2012), a photo of Hillary Clinton (2014, 2015), or no photo (2014, 2015). \\hl{(See S1 Table for the disaggregation of comparison groups.)} Other posters were taped to the wall that said either ``Obama'' or ``Romney''/``Clinton'' on the respective sides of the porch. In 2014, we primed the trick-or-treater by pointing to the photo and asking if they knew who that was; in 2015, trick-or-treaters were individually randomly assigned (using a pre-coded Excel spreadsheet) to receive or not receive the additional priming. \\hl{We pool our analysis, irrespective of this additional source of variation.}", "corrected": "(See S1 Table for the disaggregation of comparison groups. \\hl{The released data record each child's side of the porch but not the comparison condition or the 2015 priming assignment.})\n\nA complete statement would also say how and when the Clinton and no-photo conditions were allocated on the comparison side. The published sources do not contain this information.", "intended_meaning": "Children were allocated to two sides of the porch. In 2014 and 2015 the comparison side had two conditions, a photograph of Hillary Clinton or no photograph, and the analysis pools these conditions and the 2015 priming assignment. This reading agrees with the page, and every entry of S1 Table is compatible with the released data. The paper does not say how or when the two conditions were allocated, and the released data record neither the comparison condition nor the priming assignment. The Clinton and No photo rows of S1 Table cannot be regenerated from the released data: in 2015, fruit counts of 55 and 58 or of 56 and 57 in these two rows both match the printed means and standard errors. The pooled estimates do not use these labels." }, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-N003", "work": { "arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/" }, "work_id": "P035" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the last five lines of page 3. The text reads: For analysis, we use the difference-in-means estimates for each year. We had missing outcome data on one 12 year old, female participant in 2012; this participant is dropped from analysis. Our findings are substantively unchanged by imputing either a fruit or candy outcome for the participant. Red outlines surround the words this participant is dropped from analysis, at the end of one line and the start of the next.", "caption": "Methods, last paragraph, p. 3/6 (PDF p. 3). The outline marks the statement that the participant with the missing outcome is dropped from analysis; it runs over a line break.", "url": "https://pmaronow.github.io/llm-audit/works/P035/exhibits/P035-N004.png" } ], "formal_release": null, "formal_statements": [], "headline": "The participant with the missing outcome is excluded from the outcome regressions, and her assignment still enters the estimated treatment probabilities.", "id": "P035-N004", "location": "Methods, last paragraph, p. 3/6 (PDF p. 3)", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "(We had missing outcome data on one 12 year old, female participant in 2012; this participant is \\hl{dropped from analysis}. Our findings are substantively unchanged by imputing either a fruit or candy outcome for the participant.) To compute the pooled estimates, we aggregate across years using inverse probability of treatment weighting.", "corrected": "(We had missing outcome data on one 12 year old, female participant in 2012; this participant is \\hl{excluded from the outcome regressions, while treatment probabilities are estimated using all observed assignments}. Our findings are substantively unchanged by imputing either a fruit or candy outcome for the participant.)", "intended_meaning": "In this parenthesis, ``dropped from analysis'' means that the participant is omitted from the outcome means and the outcome regressions, which use 1222 rows; the replication code does this. Her assignment still enters the estimate of the treatment probabilities, which uses all 1223 rows and gives the 2012 probability $96/165$ in place of $95/164$. The weighting method, as the paper describes it, allows this. Deleting her before the weights are estimated changes the pooled difference from 0.045414 to 0.045371 and the $p$-value from 0.0908 to 0.0910, and no printed number changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-N004", "work": { "arxiv_id": null, "doi": "10.1371/journal.pone.0189693", "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "url": "https://pmaronow.github.io/llm-audit/works/P035/" }, "work_id": "P035" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P035.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:1a14f00780847df782ccfe100c8563494fc3c514ebd51bb5293195e5dc1c5530", "conditional_on": [], "credit_reason": "T1 explicitly asks for a validator nonidentification witness, not a full refutation of the factual empirical attribution. All clauses (i)-(v), exact constants, and granted design assumptions are exported. Rational weights cause no gap in the any-mechanism conclusion: the separately exported pointwise observed-choice equality also implies equality for arbitrary real probability weights. All sums use the finite support; all denominators are positive and natural threshold subtraction is untruncated in these instances. No status/kind inconsistency.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P035.campaign_nonidentification_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P035/P035_Nonidentification.lean#L470", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1a14f00780847df782ccfe100c8563494fc3c514ebd51bb5293195e5dc1c5530", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P035-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P035-T1" }, "html": { "body": "

Abstract Conclusions, p. 2/6, and Results and conclusions, p. 5/6. The validator's two finite worlds reproduce all six frequencies and every observed choice, but have the specified zero versus positive campaign effects, so the observed law does not identify the campaign effect. This is a witness about identification, not a decision about the initiative's actual effect; its overall mixture uses raw enrollment shares.

", "headline": "Abstract, Conclusions, first sentence (the campaign attribution); the witness is the validators' construction and is not in the paper", "location": "Abstract Conclusions, p. 2/6 (PDF p. 2), and Results and conclusions, p. 5/6 (PDF p. 5); the Introduction, p. 2/6, supplies hypothesis and rationale, not a separate categorical error" }, "id": "P035-T1", "text": { "body": "Abstract Conclusions, p. 2/6, and Results and conclusions, p. 5/6. The validator's two finite worlds reproduce all six frequencies and every observed choice, but have the specified zero versus positive campaign effects, so the observed law does not identify the campaign effect. This is a witness about identification, not a decision about the initiative's actual effect; its overall mixture uses raw enrollment shares.", "headline": "Abstract, Conclusions, first sentence (the campaign attribution); the witness is the validators' construction and is not in the paper", "location": "Abstract Conclusions, p. 2/6 (PDF p. 2), and Results and conclusions, p. 5/6 (PDF p. 5); the Introduction, p. 2/6, supplies hypothesis and rationale, not a separate categorical error" }, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-T1" }, { "declarations": [ { "claim_revision_id": "catalog:79ff3888dc20ffefc96ae74ed28528005d05cce38db19b50eee5f259353c4ca4", "conditional_on": [], "credit_reason": "Faithful to the two mathematical campaign clauses identified in T2 and claimed by this row. The fixed product masses give independent randomized assignment for every potential-outcome function; no added design hypothesis or assumed identification result is hidden in a structure. The empirical estimate sentence, separate health-message mediation, and an editorial instruction are not asserted as new population causal theorems. No status/kind inconsistency.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P035.corrected_campaign_interpretation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P035/P035_Nonidentification.lean#L526", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:79ff3888dc20ffefc96ae74ed28528005d05cce38db19b50eee5f259353c4ca4", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P035-T2", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P035-T2" }, "html": { "body": "

corrected Abstract Conclusions, p. 2/6, and campaign wording, p. 5/6. Two calibrated models with the same observed law have zero versus positive campaign effects, and full campaign enablement of the porch contrast is possible. Restricted to mathematical clauses (a)-(b): this does not assert the retained empirical estimate that porch assignment increased raisin choice, and it does not formalize the instruction to replace the campaign wording on p. 5/6. The counterfactual is initiative presence, with no separate health-message mediator.

", "headline": "Abstract, Conclusions, first sentence, and 'the treatment effect of her campaign' (p. 5/6), as repaired by Sol", "location": "p. 2/6 (PDF p. 2) and p. 5/6 (PDF p. 5)" }, "id": "P035-T2", "text": { "body": "corrected Abstract Conclusions, p. 2/6, and campaign wording, p. 5/6. Two calibrated models with the same observed law have zero versus positive campaign effects, and full campaign enablement of the porch contrast is possible. Restricted to mathematical clauses (a)-(b): this does not assert the retained empirical estimate that porch assignment increased raisin choice, and it does not formalize the instruction to replace the campaign wording on p. 5/6. The counterfactual is initiative presence, with no separate health-message mediator.", "headline": "Abstract, Conclusions, first sentence, and 'the treatment effect of her campaign' (p. 5/6), as repaired by Sol", "location": "p. 2/6 (PDF p. 2) and p. 5/6 (PDF p. 5)" }, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-T2" }, { "declarations": [ { "claim_revision_id": "catalog:ec64db3197be09cae82b0bcfbe64dc189b8ae04d63fef6455c85bb1416b41f50", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:ec64db3197be09cae82b0bcfbe64dc189b8ae04d63fef6455c85bb1416b41f50", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P035-T3", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P035-T3" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Abstract, Conclusions (first sentence), with Abstract, Results", "location": "Abstract Conclusions/Results, p. 2/6 (PDF p. 2), and campaign-effect wording in Results and conclusions, p. 5/6 (PDF p. 5); read with the Introduction's contextual hypothesis and rationale" }, "id": "P035-T3", "text": { "body": "No current theorem is supplied for this target.", "headline": "Abstract, Conclusions (first sentence), with Abstract, Results", "location": "Abstract Conclusions/Results, p. 2/6 (PDF p. 2), and campaign-effect wording in Results and conclusions, p. 5/6 (PDF p. 5); read with the Introduction's contextual hypothesis and rationale" }, "url": "https://pmaronow.github.io/llm-audit/works/P035/#P035-T3" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees", "status": "full", "targets": [ { "agreement": "agrees", "location": "Results and conclusions, p. 4 (PDF p. 4), Figure 1", "method": "I ran the authors' code with package-library and working-directory changes. The pooled means and confidence intervals agree to rounding; the six yearly points and intervals agree at the resolution of the published graph, which gives no exact annual numeric labels.", "status": "reproduced", "target": "Figure 1: pooled and yearly fruit-choice means and 95% CIs" }, { "agreement": "to_rounding", "location": "Results and conclusions, p. 5 (PDF p. 5), first paragraph", "method": "I ran the authors' code with package-library and working-directory changes. The two-sided p-value of the pooled difference is 0.0908 (standard error 0.0269), which rounds to the printed 0.091.", "status": "reproduced", "target": "Pooled treatment-effect significance" }, { "agreement": "to_rounding", "location": "Abstract Results, p. 2 (PDF p. 2)", "method": "I calculated 100 times the ratio of the unrounded pooled means minus 100, using estimates from the authors' code. The result is 19.305%, rounding to the reported 19%; the released script does not itself calculate this ratio.", "status": "recalculated", "target": "Abstract: relative increase in fruit choice" }, { "agreement": "to_rounding", "location": "Abstract Participants, p. 1 (PDF p. 1); Methods, pp. 2-3 (PDF pp. 2-3)", "method": "I summarized counts, ages and recorded genders directly from the released data, adding calculations to the authors' analysis. All counts agree exactly and the age and male-percentage summaries agree to rounding. The released script does not calculate these summaries.", "status": "recalculated", "target": "Participant counts and demographics" } ] }, "summary": "On three Halloweens in New Haven, 1223 trick-or-treaters were assigned to a side of a porch showing a photograph of Michelle Obama or to a comparison side; fruit-or-candy choices were recorded for 1222. The pooled estimates of fruit choice were 28.1\\% against 23.5\\%, a 19\\% relative increase ($p = 0.091$). 2 entries are classified as errors (1 Major, 1 Minor), and 2 are clarity notes; 1 further concern was not confirmed and 2 remain unresolved. The porch-condition experiment alone does not identify the campaign contribution behind the categorical abstract attribution (\\fid{P035-E001}). The Introduction proposes reinforcement through a reminder as a possible pathway, which the counterexample does not refute.", "what_was_checked": "I checked the following calculations and compatibility claims against the released data and the replication code.\n\\begin{itemize}\n\\item The sample sizes: $165 + 422 + 636 = 1223$ enrolled, and 1222 analysed after the one disclosed missing outcome ($164 + 422 + 636$ in S1 Table).\n\\item The descriptive statistics: mean age 8.52 (printed 8.5) and 52.6\\% male among the 1222 with gender recorded (printed 53\\%), and by year 7.84 and 57.6\\%, 8.53 and 56.2\\%, 8.69 and 49.0\\% (printed 7.8 and 58\\%, 8.5 and 56\\%, 8.7 and 49\\%).\n\\item The pooled estimates 28.1\\% (23.8--32.4\\%) and 23.5\\% (20.5--26.5\\%), recalculated as 0.28066 (0.23751, 0.32381) and 0.23525 (0.20512, 0.26538).\n\\item The two-sided $p = 0.091$, recalculated as 0.0908 with standard error 0.0269; under that calculation, the difference is significant at the 0.10 level, as the paper states.\n\\item The 19\\% relative increase: the ratio of the pooled proportions is 1.193. It is not an increase of 19 percentage points.\n\\item The eight points of Fig.~1 and their interval endpoints, to the resolution of the published graph.\n\\item The $N$, mean and standard error in each Obama and Romney row of S1 Table reproduce from the data. The Clinton and No photo rows are compatible with the pooled comparison counts, but the released data do not authenticate those subgroup assignments (see \\fid{P035-N003}).\n\\item The statement that the findings are substantively unchanged by imputing the missing outcome: imputing fruit gives $p = 0.084$ and imputing candy gives $p = 0.093$.\n\\item The described method, which the replication code follows: yearly assignment shares as treatment probabilities, normal-approximation intervals and heteroskedasticity-robust standard errors, with the one qualification in \\fid{P035-N004}. The numerical reproduction used R 4.4.2 and its installed packages.\n\\end{itemize}\nI read all six pages. The paper has no proofs or mathematical displays, and every numbered citation in the text has an entry in the reference list.\n\nI opened the zip file nested inside the data release and found its three replication files byte-identical to the copies supplied separately.\n\nThe following were not checked. The cited works other than reference [4], which was consulted only for \\fid{P035-E002}, were not read; in particular, the 79\\% and 82\\% informal votes attributed to reference [11] were not compared with their source. The sources contain no record of how children were allocated to the sides in 2014 and 2015 beyond the description in the Methods, of the comparison condition each child saw, or of the 2015 priming assignment. The questions about assignment in 2014 and 2015, namely whether the queue-based allocation was random and whether it left assignments dependent across children or related to their choices, and the question in \\fid{P035-N003} therefore could not be settled. These checks do not show that the paper is correct in the respects that were not examined.\n\nThe identification check preserves the Introduction's proposed reminder pathway as a hypothesis. Its exact hypothetical models show what the porch contrast alone does not identify about a campaign contribution even granting valid assignment; they neither refute that pathway nor establish that the campaign had no effect.", "work": { "arxiv_id": null, "audit_complete": true, "authors": [ "P. M. Aronow", "Dean Karlan", "Lauren E. Pinson" ], "category": "Articles - Empirics", "citation": "P. M. Aronow, Dean Karlan, and Lauren E. Pinson (2018). The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial. PLOS ONE 13(1):e0189693.", "doi": "10.1371/journal.pone.0189693", "edition": { "pages_in_audited_file": 6, "sha256": "6e86409b33b193413652f023ceafa1c1c4c3ff88a46d05ed27e18f2002bb24a3", "source_url": "https://doi.org/10.1371/journal.pone.0189693", "venue": "PLOS ONE", "year": "2018" }, "id": "P035", "later_materials": null, "title": "The Effect of Images of Michelle Obama's Face on Trick-or-Treaters' Dietary Choices: A Randomized Control Trial", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P035/", "venue": "PLOS ONE", "year": "2018" } }