{ "claims": [ { "claim": "Adding the deepfake warning above a deepfake video raised the belief that the video was fake by 0.5 points on the 7-point scale ($p<0.001$), from $-0.4$ to $0.1$, shifting responses toward both uncertainty and agreement (Study 1).", "claim_id": "P034-H1", "related_corrections": [ "P034-E003" ], "tied_to": "Figure 2, left panel; Table OA4a; article p. 7 (PDF p. 7)" }, { "claim": "The warning increased disbelief in accompanying real political videos in both experiments: by 0.73 points with the real version of the Study 1 video ($p<0.001$) and by 0.26 points with the real video of Romney in Study 2 ($p<0.001$).", "claim_id": "P034-H2", "related_corrections": [], "tied_to": "Figure 2, middle and right panels; Tables OA4b and OA4c; article p. 8 (PDF p. 8)" }, { "claim": "Participants could not tell the real video from its deepfake version, with or without the warning, and telling them about deepfakes did not improve their ability to do so (in Study 1 the interaction of the deepfake and information treatments is not significant and has the opposite sign from the ideal).", "claim_id": "P034-H3", "related_corrections": [ "P034-E002" ], "tied_to": "Figure OA1 and Appendix Section 3; Figure 2, left and middle panels; interaction test on article p. 8 (PDF p. 8) and footnote 17" }, { "claim": "Pooling both studies (real-video conditions, study fixed effect), the warning raised belief that the video was fabricated by 0.40 points ($p<0.001$), raised 'unconvinced' by 0.13 points on a 3-point scale ($p<0.001$) and raised 'don't know' by 6.0 percentage points ($p=0.001$).", "claim_id": "P034-H4", "related_corrections": [ "P034-E003" ], "tied_to": "article pp. 8–9 (PDF pp. 8–9) and footnote 19; Figures OA2a and OA2b" }, { "claim": "The warning changed what participants associated with Romney: 3.1 percentage points fewer named a pro-choice position in the open-ended question ($p<0.001$), with a 3.2-point effect under the liberal coding of footnote 20, and 14.1 points fewer marked 'true' that he had ever supported women's access to abortion ($p<0.001$).", "claim_id": "P034-H5", "related_corrections": [ "P034-E006", "P034-E007", "P034-E010", "P034-N010" ], "tied_to": "Table OA3; article p. 9 (PDF p. 9) and footnote 20" }, { "claim": "The effects were large and consistent across both studies.", "claim_id": "P034-H6", "related_corrections": [], "tied_to": "article p. 3 (PDF p. 3); Figure 2; footnotes 14, 16, 18" }, { "claim": "Warning the public about deepfakes erodes trust: information about deepfakes can be used to dismiss genuine political footage, and warned voters may come to distrust political video in general, so public warnings warrant care.", "claim_id": "P034-H7", "related_corrections": [], "tied_to": "Abstract; article pp. 2–3 (PDF pp. 2–3) and the conclusion (pp. 9-11); Figure 2; Table OA3; Figure OA3 and Table OA7 (neither cited in the main text)" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of Table OA8 with its title: Table OA8: Study 2 - Republicans appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video. Two red boxes mark the words 'increases in favorability', split over two lines. The table has the columns Vote and Favor and three rows: Information*Republican, 0.06 and 0.24; 95 percent Upper Bound, 0.20 and 0.53; 95 percent Lower Bound, minus 0.09 and minus 0.06.", "caption": "Appendix p. 15 (PDF p. 15 of the Appendix), Table OA8. The marked words of the title report increases in favorability; the table prints only the interaction of the information with the Republican indicator.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E001.png" }, { "alt": "Crop of Table OA9 with its title: Table OA9: Study 2 - Participants who are pro-life appear to have directional (non-significant) increases in favorability when information about deepfakes is added to Romney video. A red box marks the words 'increases in favorability'. The table has the columns Vote and Favor and three rows: Information*Pro-Life, 0.02 and 0.13; 95 percent Upper Bound, 0.17 and 0.43; 95 percent Lower Bound, minus 0.13 and minus 0.17.", "caption": "Appendix p. 15 (PDF p. 15 of the Appendix), Table OA9. The marked words of the title report increases in favorability; the table prints only the interaction of the information with the pro-life indicator.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E001-b.png" }, { "alt": "Crop of seven lines of Section 5 of the Appendix. The text says that pro-life Republicans should be less likely to favor Romney if they believe that he was or is pro-choice, that favorability and vote intention levels in the Information condition should be higher than in the No Information condition, and that this pattern should be reversed for pro-choice Democrats. Two red boxes mark the words 'consistent with this theory in terms of estimate directions', split over two lines, in the sentence 'Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.'", "caption": "Appendix p. 19 (PDF p. 19 of the Appendix), Section 5, “Pooled”. The marked words say that the results agree with the theory in the direction of the estimates; the theory stated before them predicts higher favorability and vote intention with the information.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E001-c.png" } ], "formal_release": null, "formal_statements": [], "headline": "The titles of Tables OA8 and OA9 report “increases” among Republicans and among pro-life participants, but the estimated effects of the information within these two groups are slightly negative and not distinguishable from zero; the tables show differences between groups that are positive and not significant.", "id": "P034-E001", "location": "Appendix, titles of Tables OA8 and OA9, p. 15 (PDF p. 15 of the Appendix), repeated in the contents, p. 1; Section 5, “Pooled”, p. 19 (PDF p. 19 of the Appendix)", "pdf_page": 15, "printed_page": "15", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Appendix p.~15: ``Table OA8: Study 2 -- Republicans appear to have directional (non-significant) \\hl{increases in favorability} when information about deepfakes is added to Romney video''. The table has the columns Vote and Favor. Information*Republican: 0.06, 0.24. 95\\% Upper Bound: 0.20, 0.53. 95\\% Lower Bound: $-0.09$, $-0.06$.\n\nAppendix p.~15: ``Table OA9: Study 2 -- Participants who are pro-life appear to have directional (non-significant) \\hl{increases in favorability} when information about deepfakes is added to Romney video''. Information*Pro-Life: 0.02, 0.13. 95\\% Upper Bound: 0.17, 0.43. 95\\% Lower Bound: $-0.13$, $-0.17$.\n\nAppendix p.~19: ``For instance, in Study 2, pro-life Republicans should be less likely to favor Romney if they believe that he was or is pro-choice. And since information about deepfakes increases disbelief that Romney was ever pro-choice, favorability and vote intention levels in the Information condition should be higher than in the No Information condition. This pattern should be reversed for pro-choice Democrats. Our results (see Section 3, Tables 6-7) are \\hl{consistent with this theory in terms of estimate directions} but are not statistically significant.''", "corrected": "Appendix pp.~1 and~15: ``Table OA8: Study 2 -- \\hl{Information effects on vote intention and favorability are estimated to be less negative among Republicans than Democrats; the differences are not statistically significant}''.\n\nAppendix pp.~1 and~15: ``Table OA9: Study 2 -- \\hl{Information effects on vote intention and favorability are estimated to be less negative among pro-life than pro-choice participants; the differences are not statistically significant}''.\n\nAppendix p.~19, in place of the last sentence quoted above: ``\\hl{The interaction point estimates of Tables OA8--OA9 are positive, consistent with the pre-specified comparative predictions, but the differences are not statistically significant. The effects within Republicans and within pro-life participants are estimated to be negative on both outcomes, with 95\\% intervals that include zero; these tables do not estimate a separate effect for participants who are both pro-life and Republican.}''\n\nThe sentences before it can stay as a statement of the theory. An effect within a group is the sum $b+d$, and it is to be reported with an interval that uses the covariance of the two coefficients, as in the table above. The cross-reference ``Section 3, Tables 6-7'' of the printed sentence is corrected in \\fid{P034-E015}.", "evidence": "The evidence is a recalculation from the public Study 2 data. The final cleaning code and the four regressions of the replication files were translated from Stata into Python; Stata was not run. The recalculation gives the analysis sample of 1,925 respondents, 966 without and 959 with information, and it reproduces the four printed interactions with their printed bounds. The regressions of Table OA8 exclude Independents and use 1,750 respondents. Each interval below is the estimate $\\pm 1.96$ robust standard errors, as in the Appendix. The standard error of $b+d$ uses the covariance of the two coefficients, that is, $\\operatorname{Var}(\\hat b+\\hat d)=\\operatorname{Var}(\\hat b)+2\\operatorname{Cov}(\\hat b,\\hat d)+\\operatorname{Var}(\\hat d)$.\n\\begin{audittable}{lrrrrrl}{The coefficient $b$ on the information, the interaction $d$ and the sum $b+d$ in the four regressions of Tables OA8 and OA9. The entries are rounded from the unrounded estimates, so a sum can differ in the last digit from the sum of the rounded coefficients.}\nTable, outcome & $N$ & $b$ & $d$ & $b+d$ & SE of $b+d$ & 95\\% interval for $b+d$ \\\\\n\\hline\nOA8, Vote & 1,750 & $-0.099$ & $0.055$ & $-0.043$ & $0.052$ & $[-0.146,\\,0.059]$ \\\\\nOA8, Favor & 1,750 & $-0.243$ & $0.238$ & $-0.005$ & $0.110$ & $[-0.221,\\,0.211]$ \\\\\nOA9, Vote & 1,925 & $-0.076$ & $0.021$ & $-0.055$ & $0.061$ & $[-0.175,\\,0.065]$ \\\\\nOA9, Favor & 1,925 & $-0.179$ & $0.130$ & $-0.049$ & $0.127$ & $[-0.298,\\,0.200]$ \\\\\n\\end{audittable}\nThe vote is coded $-1$, $0$, $1$ and favorability from $-3$ to $3$, and the entries are points on these scales. The signs do not depend on rounding: solving the normal equations of the four regressions in exact rational arithmetic, I find that $b0$ in each.", "problem": "Each table reports, for each outcome, one coefficient with its bounds: the interaction of the information about deepfakes with the group. In each regression, let $b$ be the coefficient on the information and let $d$ be the coefficient on the interaction. The fitted outcome is $a+bT+cG+dTG+x'\\theta$, where $T$ indicates the information, $G$ the group and $x$ the covariates. The estimated effect of the information is then $b$ in the comparison group, which is Democrats in Table OA8 and pro-choice participants in Table OA9, and $b+d$ in the group that the title names. The tables print $d$ and not $b$. A positive $d$ says that the estimated effect is larger in the named group than in the comparison group. It does not say that the effect in the named group is positive.\n\nIn the Study 2 data all four sums $b+d$ are negative. They are $-0.043$ for the vote and $-0.005$ for favorability among Republicans, and $-0.055$ and $-0.049$ among pro-life participants. Each 95\\% interval for a sum contains zero. The titles describe as increases the two favorability estimates that are slightly negative, and the data establish neither an increase nor a decrease within either group.\n\nRead as a statement about the levels that the theory predicts, the sentence of Section 5 is not supported either: the theory predicts higher favorability and vote intention with the information among pro-life Republicans. The estimates do agree with the prediction of both pre-analysis plans of Study 2, which is comparative. The plans expect the interaction of the treatment with a Republican indicator, and with a pro-life indicator, to be positive. All four interactions are positive, and none is statistically significant. Tables OA8 and OA9 come from separate regressions, and neither estimates a separate effect for respondents who are both pro-life and Republican.", "what_survives": "The four interactions, their printed bounds and their lack of statistical significance stand, and their signs agree with the predictions of both pre-analysis plans. The regressions are unchanged; only their description changes. The pooled estimates of Section 5 and the results of the main text come from other regressions and do not depend on these titles. What changes is the reading of the tables: the estimates do not show increases within either group, and the tables do not support the levels that the theory of Section 5 predicts. The check does not establish a decrease within either group." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E001", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of five lines of the Introduction. A red box marks the words 'were unable to discriminate' in the sentence 'We found that participants were unable to discriminate between real and deepfaked videos even when they were informed about the existence of deepfakes.'", "caption": "Introduction, p. 3 (PDF p. 3). The marked words state as an inability what the tests report as a difference that was not detected.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002.png" }, { "alt": "Crop of six lines of the abstract. Three red boxes mark the clause 'did not enhance participants' ability to successfully spot manipulated videos', which runs over two lines, in the sentence 'We conducted two online survey experiments, and found that informing participants about deepfakes did not enhance participants' ability to successfully spot manipulated videos but consistently induced them to believe the videos they watched were fake, even when they were real.'", "caption": "Abstract, p. 1 (PDF p. 1). The marked clause summarizes the test of the interaction on p. 8, which was not significant.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-e.png" }, { "alt": "Crop of the last four footnote lines of the page. A red box marks the words 'were not able to discern' in footnote 15: 'Participants were not able to discern between a real video and a deepfake version of the same video without the information treatment (see Section 3 of the Appendix for more details).'", "caption": "Footnote 15, p. 7 (PDF p. 7). The marked words make the same statement for participants who saw no warning.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-b.png" }, { "alt": "Crop of the first four lines of Section 3 of the Appendix. Two red boxes mark the words 'we show that participants were unable to do so.', split over two lines, after the sentence 'One key question is whether American voters can discern between deepfakes and real videos without being informed about deepfakes.'", "caption": "Appendix, Section 3, p. 6 (PDF p. 6 of the Appendix). The marked sentence says that Figure OA1 shows the inability.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-c.png" }, { "alt": "Crop of the title of Figure OA1 and the line above it. A red box marks the words 'were unable to discriminate' in the title 'Figure OA1: Voters were unable to discriminate between a real video and a deepfake (Study 1)'.", "caption": "Appendix p. 6, title of Figure OA1. The marked words repeat the statement in the title.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E002-d.png" } ], "formal_release": null, "formal_statements": [], "headline": "“Unable to discriminate” and “did not enhance” mean that no difference and no improvement were detected; the tests reported do not establish that there was none.", "id": "P034-E002", "location": "Abstract, p. 1 (PDF p. 1); Introduction, p. 3 (PDF p. 3); footnote 15, p. 7 (PDF p. 7); Appendix, Section 3 and the title of Figure OA1, p. 6 (PDF p. 6 of the Appendix)", "pdf_page": 1, "printed_page": "1", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "Abstract, p.~1: ``We conducted two online survey experiments, and found that informing participants about deepfakes \\hl{did not enhance participants' ability to successfully spot manipulated videos} but consistently induced them to believe the videos they watched were fake, even when they were real.''\n\np.~3: ``We found that participants \\hl{were unable to discriminate} between real and deepfaked videos even when they were informed about the existence of deepfakes.''\n\np.~7, footnote 15: ``Participants \\hl{were not able to discern} between a real video and a deepfake version of the same video without the information treatment (see Section 3 of the Appendix for more details).''\n\nAppendix p.~6: ``One key question is whether American voters can discern between deepfakes and real videos without being informed about deepfakes. In Figure OA1, \\hl{we show that participants were unable to do so}.''\n\nAppendix p.~6, title: ``Figure OA1: Voters \\hl{were unable to discriminate} between a real video and a deepfake (Study 1)''", "corrected": "Abstract: ``We conducted two online survey experiments. \\hl{For the selected Study 1 videos, we did not detect an improvement in discrimination between the real and manipulated videos when participants were informed about deepfakes; the information} consistently induced them to believe the videos they watched were fake, even when they were real.''\n\np.~3: ``\\hl{For the selected Study 1 videos, we did not detect a statistically significant difference in participants' ratings of the real and deepfaked videos, with or without information about deepfakes.}''\n\np.~7, footnote 15: ``\\hl{Without the information treatment, we did not detect a statistically significant difference in participants' ratings of the real video and its deepfake version} (see Section 3 of the Appendix for more details).''\n\nAppendix p.~6: ``In Figure OA1, \\hl{we report no statistically significant difference between the real and deepfake conditions for the selected stimulus}.''\n\nAppendix p.~6, title: ``Figure OA1: \\hl{No statistically significant real/deepfake difference in participants' ratings} (Study 1, \\hl{without deepfake information})''\n\nThese are results of tests of a difference and not of an equivalence analysis. The statements of the paper about power and about generalization stay as they are.", "intended_meaning": "The passages summarize tests of differences in mean ratings for the two videos of Study 1. They mean that the study detected no significant difference between the ratings of the real video and of its deepfake version, with or without the warning, and no significant improvement due to the warning. This is correct, and p.~8 states it: the interaction of the two treatments ``is not statistically significant'', and its estimate has the opposite sign from an improvement. Recalculating from the Study 1 data, I obtain the following values. Without the warning, the adjusted difference (deepfake minus real) in the belief that the video is a deepfake is $0.062$, with $p=0.59$. Its complete-case 95\\% interval is $[-0.164,0.287]$, and its interval with Manski bounds is $[-0.384,0.469]$, as in Figure OA1. With the warning, the difference is $-0.130$, with $p=0.26$ and interval $[-0.478,0.215]$ with Manski bounds. The interaction is $-0.214$, with $p=0.19$ as in footnote 17 and complete-case interval $[-0.534,0.107]$.\n\nRead literally, ``we show that participants were unable'' says more, namely that the tests establish that participants could not tell the videos apart. A test that does not reject does not establish this, and the intervals above contain positive values. For example, suppose that every rating is $\\pm1$, and that the probability of $+1$ is $2/5$ for the real video and $3/5$ for the deepfake without the warning, and $3/10$ and $7/10$ with it. Then the mean difference is $2/5$ without the warning and $4/5$ with it. With ten participants in each of the four cells, the event that every cell has five ratings of each sign has positive probability (about $0.0004$). On this event every sample difference and the sample interaction are zero, with $p=1$. This example does not show that the study's participants could discriminate or that the warning helped them. No estimate, test or conclusion of the paper changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E002", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of six lines of the plan. Two red boxes mark the words 'we conclude that missingness occurs at random', split over two lines, in the sentence 'If there is no differential attrition, we conclude that missingness occurs at random and exclude all affected observations from analysis.'", "caption": "Amended pre-analysis plan for Study 2, Section 8, second paragraph (PDF p. 5 of the plan). The marked words draw the conclusion from the absence of differential attrition.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017.png" }, { "alt": "Crop of the last four lines of the note to Figure OA2a. Two red boxes mark the words 'as there was no differential attrition.', split over two lines, in the sentence 'The rightmost treatment effect graph uses covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.'", "caption": "Appendix p. 9 (PDF p. 9 of the Appendix), note to Figure OA2a. The marked clause gives the absence of differential attrition as the reason for the complete-case interval of Study 2.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017-c.png" }, { "alt": "Crop of four lines of the note to Figure OA3. A red box marks the words 'as there was no differential attrition.' in the sentence 'The confidence intervals in Study 2 are estimated using covariate-adjusted standard errors from an OLS regression, as there was no differential attrition.'", "caption": "Appendix p. 18 (PDF p. 18 of the Appendix), note to Figure OA3. The marked clause gives the same reason.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017-d.png" }, { "alt": "Crop of the last four lines of Section 2.2. A red box marks the words 'we found no evidence of differential attrition:' in the sentence 'In Study 2, we found no evidence of differential attrition: survey completion rates did not differ significantly across treatment arms.'", "caption": "Section 2.2 of the article, p. 6 (PDF p. 6). The marked sentence reports the result of the test. It is not in error; the inference from it is made in the plan and in the two notes of the Appendix.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E017-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P034-T1", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T1" }, { "id": "P034-T2", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T2" } ], "headline": "The absence of differential attrition does not show that outcomes are missing at random; the complete-case analysis of Study 2 needs a separate assumption about missingness, which the chi-squared test cannot supply.", "id": "P034-E017", "location": "Amended pre-analysis plan for Study 2, Section 8 (Missing Data), second paragraph (PDF p. 5 of the plan, whose pages are not numbered); notes to Figure OA2a, Appendix p. 9, and to Figure OA3, Appendix p. 18; related sentence in Section 2.2 of the article, p. 6 (PDF p. 6)", "pdf_page": 5, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P034-T1", "P034-T2" ], "text": { "as_printed": "Plan, Section 8: ``Since there may be individuals who are randomized but do not finish the survey (and thus do not provide outcome responses), we will check for differential attrition using a chi-squared test (see line 13 in the Stata code). If there is no differential attrition, \\hl{we conclude that missingness occurs at random} and exclude all affected observations from analysis. If there is differential attrition (i.e., chi-squared p-value$<$.05), we will use inverse probability weights to weight the population of all subjects who passed the attention checks.''\n\nAppendix p.~9, note to Figure OA2a: ``The rightmost treatment effect graph uses covariate-adjusted standard errors from an OLS regression, \\hl{as there was no differential attrition}.''\n\nAppendix p.~18, note to Figure OA3: ``The confidence intervals in Study 2 are estimated using covariate-adjusted standard errors from an OLS regression, \\hl{as there was no differential attrition}.''", "corrected": "Plan, Section 8: ``\\hl{If the chi-squared test does not reject, we will report no detected difference in completion rates across treatment arms. This result does not establish random or ignorable outcome missingness. Complete-case covariate-adjusted OLS requires a separate identifying condition; for example, within the target population, completion under each treatment is independent of potential outcomes and pretreatment covariates, with positive completion probability. The test does not verify that condition. If a suitable identifying condition cannot be justified, we will report attrition bounds or sensitivity analyses.}''\n\nIn the notes to Figures OA2a and OA3, the clause ``as there was no differential attrition'' should be replaced in the same way. The intervals of Study 2 are complete-case intervals, no difference in completion rates was detected, and reading them as intervals for the randomized sample rests on the separate condition.\n\nIn full, one sufficient condition for the complete-case analysis is the following. Let $T\\in\\{0,1\\}$ be the treatment, let $Y(0)$ and $Y(1)$ be real potential outcomes with finite second moments, let $R(0),R(1)\\in\\{0,1\\}$ be the potential completion indicators, and let $X$ be a vector of pretreatment covariates with finite second moments. Set $Y=Y(T)$, $R=R(T)$ and $\\tau=\\mathbb{E}[Y(1)-Y(0)]$. Suppose that (i) $T$ is independent of $(Y(0),Y(1),R(0),R(1),X)$ and $00$. Then $\\mathbb{E}[Y\\mid T=t,R=1]=\\mathbb{E}[Y(t)]$ for $t=0,1$, the law of $X$ given $\\{T=t,R=1\\}$ is the law of $X$, and every solution of the normal equations of the complete-case regression of $Y$ on $(1,T,X)$ has coefficient $\\tau$ on $T$. Condition (ii) does not follow from (i) and equal completion rates. In the first case of the table, (i) holds and both completion rates are $1/2$, while $\\tau=0$ and the coefficient on $T$ is $12/25$.", "evidence": "The evidence is an exact counterexample, computed in rational arithmetic. Let $00$. Among the complete cases $X$ is independent of $(T,Y)$ and the two arms are equally likely. Solving the normal equations of the complete-case regression of $Y$ on $(1,T,X)$, I find that the coefficient on $T$ is also $\\delta$. Two choices of $a$ and $b$ are listed in the table. For each choice the table shows a sample that the process produces with positive probability, and in this sample the chi-squared statistic for completion by arm is exactly zero.\n\\begin{audittable}{lrr}{Two cases of the counterexample. The true effect is zero in both. The sample of each column is divided evenly between $X=0$ and $X=1$ in every cell.}\n & First case & Second case \\\\\n\\hline\n$a$, $b$ & $37/50$, $13/50$ & $499/500$, $489/500$ \\\\\nAttrition in each arm & $1/2$ & $3/250$ ($1.2\\%$) \\\\\nComplete-case mean of $Y$, control and treated & $13/50$, $37/50$ & $489/988$, $499/988$ \\\\\nComplete-case coefficient on $T$ & $12/25=0.48$ & $5/494\\approx0.0101$ \\\\\nSample size & $400$ & $4000$ \\\\\nMissing and observed in each arm & $100$, $100$ & $24$, $1976$ \\\\\nObserved $Y=0$ and $Y=1$, control & $74$, $26$ & $998$, $978$ \\\\\nObserved $Y=0$ and $Y=1$, treated & $26$, $74$ & $978$, $998$ \\\\\nPearson chi-squared statistic, $p$-value & $0$, $1$ & $0$, $1$ \\\\\n\\end{audittable}\nIn both cases the distribution of the observed data, that is, of $(T,X,R)$ and of $Y$ when $R=1$, is the same as under a second process in which outcomes are missing completely at random and the true effect is $\\delta$. This implies that no test computed from the observed data can tell the two processes apart.\n\nThe corrected statement below has two hypotheses: (i) the treatment is randomized, and (ii) completion under each treatment is independent of the potential outcomes and the covariates and has positive probability. It holds for the following reason. By (i), the event $\\{T=t,R=1\\}$ equals $\\{T=t,R(t)=1\\}$ and has probability $P(T=t)P(R(t)=1)>0$. By (i) and (ii), the law of $(Y(t),X)$ given this event is its unconditional law. This implies the first two conclusions. Let $p=P(T=1\\mid R=1)$, and let $(\\beta_0,\\beta_T,\\beta_X)$ solve the normal equations. Subtracting $p$ times the equation for the intercept from the equation for $T$, I obtain\n\\[\n\\mathbb{E}\\bigl[(T-p)(Y-\\beta_0-\\beta_TT-\\beta_X'X)\\mid R=1\\bigr]=0.\n\\]\nNote that $\\mathbb{E}[(T-p)X\\mid R=1]=0$, since $X$ has the same law in both arms of the complete cases, and that $\\mathbb{E}[(T-p)T\\mid R=1]=p(1-p)$. The display then reads $p(1-p)\\{\\mathbb{E}[Y\\mid T=1,R=1]-\\mathbb{E}[Y\\mid T=0,R=1]\\}=\\beta_T\\,p(1-p)$, and I conclude that $\\beta_T=\\tau$.", "problem": "The plan treats a chi-squared test that does not reject equal completion rates as showing that outcomes are missing at random. On that basis it analyzes only the respondents who completed the survey. The conclusion does not follow. I write $T$ for the randomized treatment, $Y(0)$ and $Y(1)$ for the potential outcomes, $R(0)$ and $R(1)$ for the indicators that the outcome would be observed under each assignment, and $X$ for the pretreatment covariates. The outcome $Y=Y(T)$ is observed when $R=R(T)$ equals $1$. Under randomization the test addresses the hypothesis\n\\[\nP(R=1\\mid T=0)=P(R=1\\mid T=1),\n\\]\nthat is, $\\mathbb{E}[R(0)]=\\mathbb{E}[R(1)]$. This hypothesis concerns how many outcomes are missing in each arm, and not which outcomes are missing. The complete-case difference in means converges to $\\tau_{\\mathrm{CC}}=\\mathbb{E}[Y(1)\\mid R(1)=1]-\\mathbb{E}[Y(0)\\mid R(0)=1]$. For the average effect $\\tau=\\mathbb{E}[Y(1)-Y(0)]$, I have\n\\[\n\\tau_{\\mathrm{CC}}-\\tau=\\bigl\\{\\mathbb{E}[Y(1)\\mid R(1)=1]-\\mathbb{E}[Y(1)]\\bigr\\}-\\bigl\\{\\mathbb{E}[Y(0)\\mid R(0)=1]-\\mathbb{E}[Y(0)]\\bigr\\}.\n\\]\nEqual completion rates do not make either bracket vanish. The implication fails even when the two rates are known to be equal, and the gap is therefore not one of statistical power.\n\nThe rule of the plan was applied. The analysis program of the replication files tests completion by arm and carries the comment ``attrition not differential, so as per PAP we drop missings''. It then drops the respondents with a missing outcome before the Study 2 regressions and the pooled regressions. The notes to Figures OA2a and OA3 say that the intervals of Study 2 are complete-case intervals because there was no differential attrition.", "what_survives": "The sentence of the article on p.~6 reports the result of a test, and it stands; I did not repeat the test for this finding. The randomized design, the arithmetic of the regressions and the comparisons among respondents who completed the survey are unaffected. The bounds used for Study 1 do not rely on this inference. No bias, wrong estimate, wrong $p$-value or failed interval in Study 2 is established. What changes is the justification. The Study 2 and pooled estimates describe the randomized sample only under an additional assumption about missingness, and the test cannot supply it. If the assumption is stated, it is a new hypothesis that the data may not satisfy. If bounds replace it, the conclusions may become weaker." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E017", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the six lines of footnote 20. The footnote says that the most conservative coding is reported, that as a robustness check the same analysis is conducted with a liberal interpretation of the same responses, and continues: Using this liberal interpretation, we recover a slightly larger treatment effect of 3.2 percentage points (p less than 0.001). A red outline surrounds the words 3.2 percentage points (p less than 0.001). The last sentence points to Section 8 of the Appendix for the full list of ambiguous responses.", "caption": "Footnote 20, p. 9 (PDF p. 9). The outline marks the estimate and $p$-value reported for the liberal coding.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E006.png" }, { "alt": "Listing of sixteen numbered lines of Stata code under the heading ORIGINAL CODE: 3-analysis_FINAL.do, SOURCE LINES 394-409. Line 394 renames FR_abortion_flag to FR_abortion_flag_conservative. Lines 395 to 398 preserve the data, set the global frmethod to liberal, run free_response_cleanup.do and restore. Line 399 reads merge 1:1 responseid using data/study2/study2_coded_free_responses_to_merge, gen(_merge_handcode_lib). Lines 401 to 407 generate FR_abortion_flag equal to 1 and replace it by 0 if any of the three final codes is curr_prochoice or past_prochoice. Line 409 regresses FR_abortion_flag on tr_warn and the macro covars for Study 2 participants with romney_party equal to 1. Nothing is outlined.", "caption": "Replication code, 3-analysis_FINAL.do, lines 394–409. Line 399 merges the liberal codes without replacing the conservative ones already in memory, and line 409 runs the regression with the macro covars.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E006-code.png" } ], "formal_release": null, "formal_statements": [], "headline": "The replication code for the robustness check of footnote 20 keeps the conservative codes and changes the covariate list, so it does not carry out the check that the footnote describes; a recalculation with the liberal codes and Study 2 covariates reproduces the printed 3.2 percentage points (see P034-E010).", "id": "P034-E006", "location": "Footnote 20, p. 9 (PDF p. 9); replication code, 3-analysis_FINAL.do, lines 393–413, with lines 227–228, 296, 317 and 366", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Footnote 20, p.~9.} We report the most conservative coding so that ambiguous responses like ``support women'' and ``fight for women's causes'' are coded as ``other.'' As a robustness check, we conduct the same analysis but with a liberal interpretation of the same responses (i.e., we code these types of ambiguous responses as ``pro-choice''). Using this liberal interpretation, we recover a slightly larger treatment effect of 3.2 percentage points ($p<0.001$).\n\n\\emph{\\texttt{3-analysis\\_FINAL.do}, lines 399 and 409.}\n\n\\texttt{\\hl{merge} 1:1 responseid using} [\\ldots]\\texttt{, gen(\\_merge\\_handcode\\_lib)}\n\n\\texttt{reg FR\\_abortion\\_flag tr\\_warn \\hl{\\$covars} if study==2 \\& romney\\_party==1, robust}", "corrected": "\\emph{\\texttt{3-analysis\\_FINAL.do}, lines 398--409.} The conservative code variables are dropped before the second merge, and the regression names the covariates of Study 2:\n\n\\texttt{restore}\n\n\\texttt{\\hl{drop final\\_code1 final\\_code2 final\\_code3}}\n\n\\texttt{merge 1:1 responseid using} [\\ldots]\\texttt{, gen(\\_merge\\_handcode\\_lib)}\n\nLines 401--407, which construct \\texttt{FR\\_abortion\\_flag} from the merged codes, remain between the merge and the regression:\n\n\\texttt{reg FR\\_abortion\\_flag tr\\_warn \\hl{c.age i.gender i.party i.region c.hhi} ///}\n\n\\texttt{\\hl{elder-pro\\_vax conlib imputed\\_conlib bigfive* pol\\_know\\_total polinterest} ///}\n\n\\texttt{if study==2 \\& romney\\_party==1, robust}\n\n\\emph{Footnote 20.} I reproduce the printed estimate and $p$-value in the translated calculation of \\fid{P034-E010}. The repaired Stata code should be run on the Study 2 analysis data to verify its output against that calculation.", "evidence": "The evidence is a trace of the code and an exact check on the file of hand-coded responses in the replication files, which has 7,671 responses from 2,557 participants. Applying the conservative rules to this file, I find 103 pro-choice responses from 101 participants; the liberal rules give 138 from 134. The two sets of codes differ for 36 responses from 36 participants, and the outcome would change for 33 participants; with the merge at line 399 it changes for none. For example, one participant names the facts ``MAN'', ``WHITE'' and ``FOR WOMEN'S RIGHTS''. The outcome variable \\texttt{FR\\_abortion\\_flag} equals $1$ when no response has a pro-choice code. With the conservative codes \\texttt{demog}, \\texttt{demog} and \\texttt{other} of this participant, it equals $1$. With the liberal codes \\texttt{demog}, \\texttt{demog} and \\texttt{curr\\_prochoice}, it equals $0$, and the block returns $1$. The counts above refer to the whole file and not to the regression sample. I checked the documented default merge rule without running Stata. In the recalculation reported in \\fid{P034-E010}, I generated the conservative and liberal outcomes separately and matched them to the Study 2 sample, avoiding the merge that retains the conservative codes. With the liberal outcome, the expanded Study 2 covariates and HC1 standard errors, I obtain a drop of 3.20 percentage points ($p=0.0006$) on $N=1{,}925$. The calculation translates the relevant Stata regression into Python and uses the supplied cleaned covariates; Stata was not run.", "problem": "Footnote 20 describes the analysis behind the estimate of 3.1 percentage points, repeated with one change: ambiguous responses are coded as pro-choice. The block of the replication code that produces the regression of the footnote does not do this, for two reasons. First, the conservative code variables \\texttt{final\\_code1}, \\texttt{final\\_code2} and \\texttt{final\\_code3} are merged into the analysis data at line 296 and are never dropped. The block writes the liberal codes to a file and merges that file at line 399 with a plain one-to-one merge. Under Stata's default rule for a merge, a matched observation keeps the values already in memory for every variable that both files contain. The liberal codes therefore never enter the data, and the new outcome equals the conservative outcome on every row. Second, the regression at line 409 takes its covariates from the global macro \\texttt{covars}. Line 366 has reset this macro to the six covariates of Study 1. The conservative regression at line 317 used the expanded list of Study 2 that lines 227--228 define. The block thus fits the conservative outcome with a different covariate list, and its output cannot show what the liberal coding does to the estimate.", "what_survives": "The conservative estimate of 3.1 percentage points at line 317, the closed-ended result of 14.1 percentage points and the main analyses of both studies are computed before this block and do not depend on it. I reproduce the 3.2 percentage points and $p<0.001$ of footnote 20 in the translated calculation of \\fid{P034-E010}. The deposited replication code still needs both repairs to carry out the stated robustness check." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E006", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of nine lines of the article, beginning: We also evaluated whether information about deepfakes can affect how participants internalize the content of the video. The marked sentence reads: We found that information about deepfakes did not significantly change how likely participants were to mention abortion (p equals 0.95), but there was a major shift in what participants perceived Romney's position on abortion to be. A red outline surrounds the words mention abortion (p equals 0.95). The following sentence says that the information about deepfakes caused a 3.1-percentage-point drop in the percentage of participants who associated Romney with a pro-choice position (p less than 0.001).", "caption": "Section 3, second paragraph of p. 9 (PDF p. 9). The outline marks the outcome described as mentioning abortion, with its $p$-value of 0.95; the next sentence gives the drop of 3.1 percentage points.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E007.png" }, { "alt": "Listing of fourteen numbered lines of Stata code under the heading ORIGINAL CODE: 3-analysis_FINAL.do, SOURCE LINES 302-315. Line 302 generates FR_abortion_flag equal to 1 and line 304 generates mention_abort equal to 0. A loop over i from 1 to 3 replaces FR_abortion_flag by 0 if final_code i equals curr_prochoice or past_prochoice, and replaces mention_abort by 1 if final_code i equals curr_prolife or past_prolife or FR_abortion_flag equals 0. Line 313 regresses mention_abort on tr_warn and the covariates for Study 2 participants with romney_party equal to 1, and the comment on lines 314 and 315 quotes the sentence of page 9 with p equals 0.95. Nothing is outlined.", "caption": "Replication code, 3-analysis_FINAL.do, lines 302–315. Lines 304 and 309–310 build mention_abort from the pro-choice and pro-life codes, and line 313 is the regression behind $p = 0.95$.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E007-code.png" } ], "formal_release": null, "formal_statements": [], "headline": "Two hand-coded responses that support abortion choice are coded as pro-life, and the outcome described as mentioning abortion is an indicator of a pro-choice or pro-life code, which misses six participants who name abortion.", "id": "P034-E007", "location": "Section 3, second paragraph of p. 9 (PDF p. 9); Appendix, Table OA3, p. 11, and Section 8, p. 23; file of hand-coded responses in the replication files; 3-analysis_FINAL.do, lines 302–313", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Section 3, p.~9.} We found that information about deepfakes did not significantly change how likely participants were to \\hl{mention abortion} ($p=0.95$), but there was a major shift in what participants perceived Romney's position on abortion to be. The information about deepfakes caused a 3.1-percentage-point drop in the percentage of participants who associated Romney with a pro-choice position ($p<0.001$).\n\n\\emph{Hand-coded responses in the replication files.} ``HE WILL FIGHT FOR WOMEN'S RIGHTS TO CHOOSE'': \\hl{\\texttt{curr\\_prolife}}. ``HE’S PRO ABORTION'': \\hl{\\texttt{curr\\_prolife}}.", "corrected": "\\emph{Section 3, p.~9.} We found that information about deepfakes did not significantly change how likely participants were to \\hl{give a response coded as a current or past pro-choice or pro-life position} ($p=0.95$), but there was a major shift in what participants perceived Romney's position on abortion to be.\n\n\\emph{Hand-coded responses.} ``HE WILL FIGHT FOR WOMEN'S RIGHTS TO CHOOSE'': \\hl{\\texttt{curr\\_prochoice}}. ``HE’S PRO ABORTION'': \\hl{\\texttt{curr\\_prochoice}}. The response ``HE WILL LET WOMEN HAVE THE RIGHT TO CHOOSE FOR ABORTION'' carries the same reversed code and is to be recoded with them; the hand codes were not reviewed exhaustively. Table OA3 and the estimate of 3.1 percentage points are then to be recomputed on the analysis sample.", "evidence": "The evidence is an exact check on the file of hand-coded responses, which has 7,671 responses from 2,557 participants, and a derivation from the code. For $j = 1, 2, 3$, I set $P_j = 1$ if response $j$ has a current or past pro-choice code and $P_j = 0$ otherwise, and I define $L_j$ in the same way for the pro-life codes. Lines 302--311 of \\texttt{3-analysis\\_FINAL.do} start with \\texttt{FR\\_abortion\\_flag} equal to $1$ and \\texttt{mention\\_abort} equal to $0$. For each $j$ in turn, they set the first variable to $0$ if $P_j = 1$, and then set the second to $1$ if $L_j = 1$ or the first variable is $0$. I write $F$ and $M$ for the final values of the two variables. By induction on $j$, I obtain\n\\[\nF = \\prod_{j=1}^{3} (1 - P_j), \\qquad M = 1 - \\prod_{j=1}^{3} (1 - P_j - L_j) .\n\\]\nThe second equation shows that $M$ indicates a pro-choice or pro-life code, and not a mention of the topic. One participant's responses have the codes \\texttt{other}, \\texttt{dontknow} and \\texttt{dontknow}, and the first response is the word ``ABORTION''. Since $P_j = L_j = 0$ for every $j$, I have $M = 0$ for this participant. The six participants who name abortion and have $M = 0$ gave the responses ``ABORTION'' (four participants), ``MADE CLAIMS ABOUT ABORTION'' and ``MORMON CHURCH DOES NOT SUPPORT ABORTION''.\n\nIn the whole file, the two recodes raise the number of participants with a pro-choice code from 101 to 103 and leave $M$ unchanged. There are 150 participants with $M = 1$, and counting the six participants as well gives 156. The 53 rules of the conservative cleanup, applied to the file, reproduce all 7,671 codes saved in the replication files. The file covers 2,557 participants, and the analysis sample has 1,925. I did not establish analysis-sample membership for all of these participants or compute the corrected Table OA3 and regressions.", "problem": "The two analyses in this passage use hand codes for the three facts that each participant named. First, two responses that state support for abortion choice carry the code for a current pro-life position. With the correct code, each of the two participants counts as associating Romney with a pro-choice position, which bears on the estimate of 3.1 percentage points and on the pro-choice and pro-life rows of Table OA3. These two responses are not the only ones: the response ``HE WILL LET WOMEN HAVE THE RIGHT TO CHOOSE FOR ABORTION'' has the same reversed code. Second, the outcome behind $p = 0.95$ does not measure whether a participant mentions abortion. The variable \\texttt{mention\\_abort} equals $1$ exactly when at least one of the three responses has a current or past pro-choice code or a current or past pro-life code. A response that names abortion and receives another code does not count. The conservative cleanup printed on p.~23 of the Appendix assigns the code ``other'' to the bare response ``ABORTION''. Six participants in the file of hand-coded responses name abortion and have \\texttt{mention\\_abort} equal to $0$.", "what_survives": "The two recodes do not change \\texttt{mention\\_abort}, and describing this variable accurately changes no number. The result $p = 0.95$ stands as a statement about pro-choice and pro-life codes. The outcomes for disbelief in the video and the closed-ended result of 14.1 percentage points use other variables. The estimate of 3.1 percentage points, its $p$-value and the pro-choice and pro-life rows of Table OA3 are to be recomputed after the recodes. The recodes concern a few responses, and their effect on these numbers was not computed." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E007", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the top of Table OA3, titled Study 2 - Hand-coded Romney facts across conditions, with the columns No Info about Deepfakes, Info about Deepfakes and Total. The first row, Don't Know / No Response Given, has the percentages 28.2, 27.0 and 27.6 percent and, below them, the counts N 818, 777 and 1,595. Red boxes mark the counts 777 and 1,595. The second row, Is Republican, has 16.5 percent in each column and the counts 477, 475 and 952.", "caption": "Appendix, Table OA3, p. 11 (PDF p. 11 of the Appendix): the head of the table and its first two rows. The marked counts, 777 and 1,595, include one substantive answer whose text was lost before hand-coding.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E010.png" } ], "formal_release": null, "formal_statements": [], "headline": "Six free-response answers from five respondents of Study 2 were lost or cut short before they were hand-coded; one substantive answer in the analysis sample was coded as “don’t know”, and the count 777 in the first row of Table OA3 should be 776, with total 1,594.", "id": "P034-E010", "location": "Appendix, Table OA3, first row, p. 11 (PDF p. 11 of the Appendix); the paragraph on the open-ended question and footnote 20, p. 9 (PDF p. 9); replication files: the raw survey file of Study 2, the cleaned data, the file of hand-coded responses, and 2-study2_data_cleanup_FINAL.do, line 2", "pdf_page": 11, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Appendix p.~11, ``Table OA3: Study 2 - Hand-coded Romney facts across conditions'', first row, under the columns ``No Info about Deepfakes'', ``Info about Deepfakes'' and ``Total''.\n\nDon't Know / No Response Given: 28.2\\%, 27.0\\%, 27.6\\%. $N$: 818, \\hl{777}, \\hl{1,595}.\n\nFile of hand-coded responses in the replication files, third answer of one respondent in the arm with information about deepfakes: the text is \\hl{empty}, and the code is \\hl{\\texttt{dontknow}}.", "corrected": "Appendix p.~11, Table OA3, first row.\n\nDon't Know / No Response Given: 28.2\\%, 27.0\\%, 27.6\\%. $N$: 818, \\hl{776}, \\hl{1,594}.\n\nThe answer \\texttt{Canceled \"Big Bird\"} is counted in a substantive row. With the code \\texttt{other}, the row ``Other'' reads 8.9\\%, \\hl{8.4\\%}, 8.6\\% and $N$: 258, \\hl{241}, \\hl{499}.\n\nIn the replication files, the seven damaged responses in the cleaned data and the six corresponding entries in the file of hand-coded responses are to be restored from the raw survey file, matched by respondent and by the position of the answer. The six hand-coded entries are to be coded again from the full text. Three keep the code \\texttt{dontknow}, the sentence on health care keeps the code \\texttt{policy}, and the two substantive answers receive substantive non-abortion codes. The raw text should be kept beside any normalized text that is used for coding. The conservative and liberal hand-code files are then to be generated separately, followed by Table OA3 and the analyses of the open-ended question.", "evidence": "The evidence is a recalculation from the replication files. I compared the raw survey file of Study 2 with the cleaned data and with the file of hand-coded responses, matching the answers by respondent and by their position among the three. The cleaned data hold 8,973 answers from 2,991 respondents, and seven differ from the raw file. The file of hand-coded responses holds 7,671 answers from 2,557 respondents, and six differ. The seventh belongs to a respondent who fails the age check and was not hand-coded. Every other answer agrees with the raw file up to capitalization and spacing. The six answers are the following.\n\\begin{enumerate}\n\\item \\texttt{He asked to be called \"Ike\"}. The hand-coded entry is empty, with the code \\texttt{dontknow}. The respondent is outside the analysis sample.\n\\item \\texttt{'don't know\"}. The entry is empty, with the code \\texttt{dontknow}. The respondent is in the analysis sample, in the arm with information.\n\\item \\texttt{Canceled \"Big Bird\"}. The entry is empty, with the code \\texttt{dontknow}. The respondent is in the analysis sample, in the arm with information.\n\\item \\texttt{\" don't know \"}, written twice by one respondent, as the second and the third answer. Both entries are empty, with the code \\texttt{dontknow}. The respondent is in the analysis sample, in the arm without information.\n\\item A sentence on the health care plan that Romney supported in Massachusetts, which ends \\texttt{which was said to be the \"model\" for Obamacare}. The entry is ``FOR OBAMACARE'', with the code \\texttt{policy}. The respondent is in the analysis sample, in the arm with information.\n\\end{enumerate}\nWith the codes as they are in the replication files, I reproduce every count of Table OA3. The answer of item 3 is substantive, and the table changes as follows when it leaves the first row. Which substantive row receives it is a coding judgment, and the table gives two choices.\n\\begin{audittable}{lll}{The rows of Table OA3 that change when the answer of item 3 is coded as substantive. Each entry gives the counts for the columns ``No Info about Deepfakes'', ``Info about Deepfakes'' and ``Total''.}\nRow & Printed & After the answer is restored \\\\\n\\hline\nDon't Know / No Response Given & 818, 777, 1,595 & 818, 776, 1,594 \\\\\nOther, if the answer is coded \\texttt{other} & 258, 240, 498 & 258, 241, 499 \\\\\nStance on Other Policies, if it is coded \\texttt{policy} & 24, 25, 49 & 24, 26, 50 \\\\\n\\end{audittable}\nThe denominators 2,898, 2,877 and 5,775 do not change, and the percentages of the first row still round to 28.2\\%, 27.0\\% and 27.6\\%. With the code \\texttt{other}, the percentage of the row ``Other'' in the column with information becomes 8.4\\% (printed 8.3\\%).\n\nNone of the six answers has a pro-choice or pro-life code, and none receives one when its text is restored. This implies that the two outcomes that the analyses of p.~9 build from the hand codes are the same before and after for each of the 2,557 respondents. This holds under both the conservative coding and the liberal coding of footnote 20. I read the files with a strict parser of comma-separated text in Python and repeated the comparison in R. Stata was not run. The check does not identify what in the import of the raw file (\\texttt{insheet}, line 2 of \\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}) causes the loss.", "problem": "Each participant of Study 2 was asked to name three facts about Romney, and each answer was coded by hand. The codes are the basis of Table OA3 and of the analyses of the open-ended question on p.~9. Some answers did not reach the coders as they were written. Among the answers of the 2,991 respondents in the cleaned data, seven contain a double quotation mark in the raw survey file. In the cleaned data, each of the seven consists only of the text after its last quotation mark. Six of the seven are among the 7,671 answers that were hand-coded, and they come from five respondents. Five of these entries are empty, and the sixth reads ``FOR OBAMACARE'' in place of a full sentence.\n\nAll five empty entries have the code \\texttt{dontknow}. For three of them the code is right, since the respondent wrote ``don't know'' between quotation marks. The other two respondents gave substantive answers, \\texttt{He asked to be called \"Ike\"} and \\texttt{Canceled \"Big Bird\"}. The first of these two respondents did not identify Romney's party and is outside the analysis sample. The second is in the analysis sample, in the arm with information about deepfakes. The first row of Table OA3 counts this answer as ``Don't Know / No Response Given'', and its entries 777 and 1,595 are too large by one.", "what_survives": "Restoring these responses changes no reported effect. The result $p=0.95$ and the drop of 3.1 percentage points on p.~9, and the drop of 3.2 percentage points in footnote 20, use outcomes that are identical before and after the six answers are restored. In the recalculation these are $p=0.953$, 3.09 points ($p=0.0003$) and 3.20 points ($p=0.0006$). The denominators of Table OA3 and its rows for pro-choice and pro-life positions are not affected by this correction. Study 1, the closed-ended outcomes of Study 2 and the pooled analyses do not use the hand codes. What changes is one count and one total in the first row of Table OA3, the row that receives the restored answer, and the text of seven responses in the cleaned data, including six in the file of hand-coded responses. Other corrections to Table OA3 are given in \\fid{P034-E007} and \\fid{P034-N010}." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E010", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the Education and Partisanship blocks of Table OA1, with three columns of percentages. A red box marks the row 'Postgrad 17 percent, 25 percent, 22 percent'. The rows above read 'HS or less 18, 16, 17', 'Some College 37, 29, 32' and 'Bachelor's Degree 27, 29, 29'; the rows below give partisanship.", "caption": "Appendix p. 7 (PDF p. 7 of the Appendix), Table OA1, the rows for education and partisanship; the columns are Study 1, Study 2 and Pooled. The marked row is the postgraduate share, whose stored count for Study 1 includes five respondents with missing education.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E009.png" } ], "formal_release": null, "formal_statements": [], "headline": "The stored count behind the Study 1 postgraduate share in Table OA1 includes five respondents whose education is missing (243 for 238); the printed 17% is unchanged, and the pooled 22% is not shown to be wrong.", "id": "P034-E009", "location": "Appendix, Table OA1 (Summary statistics), p. 7 (PDF p. 7 of the Appendix), row “Postgrad”; replication files, tables/sum_stats_table.xlsx", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\begin{audittable}{lrrr}{Table OA1, Education block, last row, and the last row of the table.}\n & Study 1 & Study 2 & Pooled \\\\\n\\hline\nPostgrad & 17\\% & 25\\% & 22\\% \\\\\nTotal N & 1396 & 1925 & 3321 \\\\\n\\end{audittable}\nFor this row, the workbook of the replication files stores the shares $\\hl{0.1740687679}$ (Study 1), $0.2493506494$ (Study 2) and $\\hl{0.2177055104}$ (pooled).", "corrected": "The row should be defined and its denominator stated: ``Postgrad'' is the share of all participants with education code 7 or 8, and a missing response is not counted in the numerator. For Study 1 the count is \\hl{238} of 1,396, and not 243, and the entry stays at 17\\%. The entries for Study 2 and for the pooled sample should be recomputed from the Study 2 data with the same definition before the printed 25\\% and 22\\% are kept or changed.", "evidence": "The evidence is a recalculation from the Study 1 raw data and from the stored workbook. With the final rules for eligibility, attention checks, duplicates and randomization, the counts of the education codes 1 to 8 are 41, 217, 57, 300, 155, 383, 218 and 20, and 5 respondents have no code. These sum to 1,396, and $218+20=238$. Multiplying the stored shares by the totals 1396, 1925 and 3321, I obtain 243, 480 and 723.", "problem": "The stored share for Study 1 equals $243/1396$. In the Study 1 raw data, after the final exclusions, there are 1,396 randomized respondents, of whom 238 have a postgraduate education (codes 7 and 8 of the education variable) and 5 have no recorded education. The stored count is $238+5$: the five respondents with missing education are counted as postgraduates. The stored pooled share equals $723/3321$, and the stored share for Study 2 equals $480/1925$. Since $723=243+480$, the pooled count carries the same five respondents.\n\nThe printed entry for Study 1 does not change: $238/1396=17.05\\%$, and $238/1391=17.11\\%$ if the five are also removed from the denominator. Whether the pooled entry changes cannot be decided from the Study 1 data. Removing the five respondents, I obtain $718/3321=21.62\\%$, which still rounds to 22\\%. The pooled entry would round to 21\\% only if at least four of the 480 respondents counted for Study 2 also had missing education. I did not check this education count against the Study 2 data. The program for the supplementary materials calls a second program to write the workbook, and that second program is not among the replication files. The step that produced the count is therefore not identified.", "what_survives": "The printed 17\\% for Study 1 is unchanged, and the printed pooled 22\\% is not shown to be wrong. No treatment effect or conclusion of the paper is shown to depend on the postgraduate count." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E009", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the first lines under the heading Pooled. The text reads: Information about deepfakes appears to have some impact on favorability and intention to vote for the politician featured in the video clip. Overall, information about deepfakes decreases intention to vote for the politician by 3.8 percentage points (p<0.05), with little difference in effect sizes across studies. Similarly, information about deepfakes decreases the politician's favorability by 0.17 points on a 7-point scale (from extremely unfavorable to extremely favorable) (p<0.05). The crop ends in the middle of the next sentence, which begins There are, however, strong theoretical reasons. A red outline surrounds the words 3.8 percentage points (p<0.05).", "caption": "Online Appendix, Section 5, “Pooled”, p. 19 (Appendix PDF p. 19). The outline marks the estimate that the binary vote regression produces.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E011.png" }, { "alt": "Listing of six lines of Stata code with line numbers 851 to 856. Line 851 is preserve, line 852 is blank, and line 853 is a keep if command with the condition study==1 and tr_d==0, or study==2 and romney_party==1. A red outline surrounds line 854, gen vote_bin=vote==1. Lines 855 and 856 regress vote_bin on tr_warn and the covariates pro_vax, acupuncture, conlib, imputed_conlib, i.party and hhi with robust standard errors, first pooled with i.study and then by study.", "caption": "Source code of the replication files, not a page of the article or of the Appendix: 4-Supplementary-Materials_FINAL.do, lines 851–856. The outline marks line 854, which sets the outcome to 0 when the vote question was not answered.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E011-code.png" } ], "formal_release": null, "formal_statements": [], "headline": "The binary vote outcome behind the pooled estimate of 3.8 percentage points gives the value 0 to participants who did not answer the vote question; it should be defined only for participants who answered, which removes 11 participants from the Study 1 regression and moves its coefficient from about $-0.045$ to about $-0.052$.", "id": "P034-E011", "location": "Replication file 4-Supplementary-Materials_FINAL.do, line 854; its printed result is in the online Appendix, Section 5, “Pooled”, second sentence, p. 19 (Appendix PDF p. 19)", "pdf_page": 19, "printed_page": "19", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Online Appendix, p.~19.} Overall, information about deepfakes decreases intention to vote for the politician by 3.8 percentage points (p$<$0.05), with little difference in effect sizes across studies.\n\n\\emph{Replication file \\texttt{4-Supplementary-Materials\\_FINAL.do}, lines 853--856 (lines 855 and 856 are each one line of code, broken here before \\texttt{imputed\\_conlib}).}\n\\[\n\\begin{aligned}\n&\\texttt{keep if (study==1 \\& tr\\_d==0 ) | (study==2 \\& romney\\_party==1)}\\\\\n&\\texttt{gen vote\\_bin=}\\hl{\\texttt{vote==1}}\\\\\n&\\texttt{reg vote\\_bin tr\\_warn~ pro\\_vax acupuncture conlib}\\\\\n&\\qquad\\texttt{imputed\\_conlib i.party hhi i.study, robust}\\\\\n&\\texttt{bysort study:\\ reg vote\\_bin tr\\_warn~ pro\\_vax acupuncture conlib}\\\\\n&\\qquad\\texttt{imputed\\_conlib i.party hhi, robust}\n\\end{aligned}\n\\]", "corrected": "Line 854 should read\n\n\\texttt{gen vote\\_bin=\\hl{(vote==1) if vote<.}}\n\nThe condition \\texttt{if !missing(vote)} is equivalent. The outcome \\texttt{vote\\_bin} then equals $1$ for ``Yes'' and $0$ for ``No'' and ``Don't Know / Not Sure'', and it is missing when the question was not answered. The regressions of lines 855 and 856 then use the participants who answered. For Study~1, the regression of line 856 has $N = 685$ and a coefficient on the information of about $-0.052$ (robust standard error $0.033$), in place of $N = 696$ and about $-0.045$.\n\nThe sentence on p.~19 should report the coefficient of the pooled regression of line 855, computed with this outcome on the data of both studies, and should describe it as an effect among the participants who answered the vote question. The corrected line keeps unanswered questions missing but does not address attrition, and a statement about all randomized participants needs a separate argument.", "evidence": "The evidence is a reading of the code and an exact recalculation from the Study~1 raw data, in which I applied the exclusions and imputations of the cleaning code.\n\\begin{audittable}{lrrrrrr}{Answers to the vote question in the two real-video arms of Study~1.}\nArm & Yes & No & Don't know & Unanswered & All & Answered \\\\\nNo Information & 99 & 158 & 81 & 9 & 347 & 338 \\\\\nInformation about Deepfakes & 81 & 183 & 83 & 2 & 349 & 347 \\\\\n\\end{audittable}\nWithout covariates, the difference between the two arms in the share of ``Yes'' is\n\\[\n\\frac{81}{349}-\\frac{99}{347} = -\\frac{6444}{121103} \\approx -0.0532\n\\]\nwith the outcome as coded, and\n\\[\n\\frac{81}{347}-\\frac{99}{338} = -\\frac{6975}{117286} \\approx -0.0595\n\\]\namong the participants who answered. For the Study~1 regression of line 856, with the covariates of the code and robust (HC1) standard errors, I obtain the following.\n\\begin{audittable}{lrrrr}{The Study~1 regression of line 856 under the two definitions of the outcome.}\nOutcome & $N$ & Coefficient on the information & Robust s.e. & $p$ \\\\\nAs coded (unanswered set to 0) & 696 & $-0.04486$ & $0.03289$ & $0.173$ \\\\\nParticipants who answered & 685 & $-0.05245$ & $0.03326$ & $0.115$ \\\\\n\\end{audittable}\nThe coefficients are computed in exact rational arithmetic, and the standard errors and $p$-values in floating point. The calculation does not imitate the single-precision storage of Stata, and the last digits may differ from Stata's output. The pooled regression of line 855 and its Study~2 component were not recalculated.", "problem": "The vote question has three answers, coded $-1$ for ``No'', $0$ for ``Don't Know / Not Sure'' and $1$ for ``Yes'' (Appendix p.~15). The cleaning code leaves the variable \\texttt{vote} missing when the question was not answered. In Stata a missing value is not equal to $1$, which implies that the expression \\texttt{vote==1} evaluates to $0$ when \\texttt{vote} is missing. Line 854 then gives a participant who did not answer the same value as a participant who answered ``No'' or ``Don't Know / Not Sure''. A missing outcome would be dropped from a regression, and a value of $0$ is kept. The regressions of lines 855 and 856 estimate the effect of information about deepfakes on an outcome that equals $1$ when a ``Yes'' was recorded and $0$ otherwise, while the sentence on p.~19 reports a decrease in the intention to vote for the politician.\n\nThe case occurs in the data. In the two real-video arms of Study~1, 11 participants did not answer the vote question, 9 of 347 in the No Information arm and 2 of 349 in the Information about Deepfakes arm. All 11 enter the regression with the value $0$. If the outcome is defined only for participants who answered, the Study~1 regression has 685 observations in place of 696, and its coefficient on the information changes from about $-0.045$ to about $-0.052$. The corrected pooled estimate is not given here. I did not recalculate the pooled regression of line 855 or its Study~2 component. The estimate of 3.8 percentage points and its significance at the 5\\% level are neither reproduced nor replaced.", "what_survives": "The correction touches the binary vote regressions of lines 855 and 856 and the sentence on p.~19 that reports them. The favorability regressions that follow, the analyses of the three-level vote outcome with bounds in Tables~OA5 and OA6, and the results of the main text on disbelief, trust and policy knowledge do not use this outcome. Whether the pooled estimate stays near 3.8 percentage points and significant at the 5\\% level is not established; in Study~1 the coefficient moves away from zero." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E011", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of the title of Figure OA2a and the headings of its three panels. The title reads: Figure OA2a: Study 2 - Impacts of information about deepfakes on disbelief (3-point scale): unconvinced that politician believes what is being said. A red outline surrounds the words Study 2. The panel headings read: Impact of Information Above Deepfake Video (Study 1), Impact of Information Above Real Video (Study 1) and Impact of Information Above Real Video (Study 2).", "caption": "Appendix, Figure OA2a, title and panel headings, p. 9. The outline marks “Study 2” in the title; the first two panels are headed Study 1.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012.png" }, { "alt": "Crop of the title of Figure OA2b and the headings of its three panels. The title reads: Figure OA2b: Study 2 - Impacts of information about deepfakes on disbelief (binary): don't know if politician truly believes what is being said. A red outline surrounds the words Study 2. The panel headings read: Impact of Information Above Deepfake Video (Study 1), Impact of Information Above Real Video (Study 1) and Impact of Information Above Real Video (Study 2).", "caption": "Appendix, Figure OA2b, title and panel headings, p. 10. The outline marks “Study 2” in the title; the first two panels are headed Study 1.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012-b.png" }, { "alt": "Crop of the eight-line italic note under Figure OA2b. Red outlines surround, over three lines, the words: how convinced are you that the politician believes what is being said? (3-point scale from 1 equals not at all convinced to minus 1 equals very convinced). The sentence that contains them begins: All three columns measure impact in terms of the same outcome. The rest of the note describes the rows of the figure and says that the two leftmost graphs of treatment effects use Manski bounds and the rightmost uses standard errors from an OLS regression.", "caption": "Appendix, note to Figure OA2b, p. 10. The outlines mark the definition of the three-point outcome, which belongs to Figure OA2a; it runs over three lines.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012-c.png" }, { "alt": "Crop of the six-line italic note under Figure OA3. The third line reads: range from 0 equals not at all confident to 4 equals completely confident, and a red outline surrounds the digit 4. The last sentence reads: The pooled estimate is estimated using a regression with shared covariates; it excludes all participants in the deepfake condition in Study 2, and a second red outline surrounds the final words Study 2.", "caption": "Appendix, note to Figure OA3, p. 18. The outlines mark the upper end of the trust scale, printed as 4, and the study named for the excluded deepfake condition, printed as Study 2.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E012-d.png" } ], "formal_release": null, "formal_statements": [], "headline": "Figures OA2a and OA2b show Studies 1 and 2, the note to Figure OA2b should define its binary outcome, the trust item ranges from 0 to 3, and the pooled estimate in Figure OA3 excludes the deepfake arm of Study 1.", "id": "P034-E012", "location": "Appendix: title of Figure OA2a, p. 9; title and note of Figure OA2b, p. 10; note to Figure OA3, p. 18; the two titles as repeated in the contents, p. 1; Study 1 pre-analysis plan, p. 4", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Figure OA2a, title, p.~9.} Figure OA2a: \\hl{Study 2} - Impacts of information about deepfakes on disbelief (3-point scale): unconvinced that politician believes what is being said\n\n\\emph{Figure OA2b, title and note, p.~10.} Figure OA2b: \\hl{Study 2} - Impacts of information about deepfakes on disbelief (binary): don't know if politician truly believes what is being said\n\n[\\ldots] All three columns measure impact in terms of the same outcome: \\hl{how convinced are you that the politician believes what is being said? (3-point scale from 1=''not at all convinced'' to -1=''very convinced'')}.\n\n\\emph{Figure OA3, note, p.~18.} Responses range from 0 = ``not at all confident'' to \\hl{4} = ``completely confident.'' [\\ldots] The pooled estimate is estimated using a regression with shared covariates; it excludes all participants in the deepfake condition in \\hl{Study 2}.\n\n\\emph{Study 1 pre-analysis plan, p.~4.} Trust is measured on a 4-point scale (ranging from ``0 = not at all confident'' to ``\\hl{4} = completely confident'').", "corrected": "\\emph{Figures OA2a and OA2b, titles (and the contents, p.~1).} Figure OA2a: \\hl{Studies 1 and 2} - Impacts of information about deepfakes on disbelief (3-point scale) [\\ldots] Figure OA2b: \\hl{Studies 1 and 2} - Impacts of information about deepfakes on disbelief (binary) [\\ldots]\n\n\\emph{Figure OA2b, note.} All three columns measure \\hl{an indicator equal to 1 for ``I don't know if the politician truly believes what is said in the video,'' and 0 for either alternative response}. The label of the vertical axis becomes \\hl{Don't know if politician truly believes (binary)}.\n\n\\emph{Figure OA3, note.} Responses range from 0 = ``not at all confident'' to \\hl{3} = ``completely confident.'' [\\ldots] it excludes all participants in the deepfake condition in \\hl{Study 1}.\n\n\\emph{Study 1 pre-analysis plan, p.~4.} Trust is measured on a 4-point scale (ranging from ``0 = not at all confident'' to ``\\hl{3} = completely confident'').", "evidence": "The evidence is a check of the pages against the survey instruments, the pre-analysis plans and the replication code. Both survey instruments list four options for the trust item, and both cleaning scripts code them 0 to 3. The code for Figure OA2b plots the indicator of the answer ``I don't know if the politician truly believes what is said in the video'', and the pooled regression for Figure OA3 selects the real-video participants of Study 1 and the Study 2 participants who knew Romney's party.", "problem": "These are misprints in titles and notes. In Figures OA2a and OA2b the first two panels are headed Study 1 and the third is headed Study 2. The note to Figure OA2b repeats the definition of the three-point outcome of Figure OA2a, but Figure OA2b plots a binary outcome, as its title says. Its axis label ``Dont Know if Real'' is also inexact, since the question asks whether the politician truly believes the statement. The trust item has four response options, coded 0 to 3. Study 2 has no deepfake condition, and the pooled estimate in Figure OA3 excludes the deepfake arm of Study 1, as footnote 19 of the article says.", "what_survives": "The plotted estimates, the coding of the outcomes and the exclusion used in the pooled regression are unaffected, and only the titles, notes and axis label change. The Study 2 estimates in these figures were not recalculated." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E012", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the lower edge of Figure 1, its caption and the first line of the next paragraph. A red box marks the word 'Schreenshot' in the caption 'Figure 1: Schreenshot of the video used in Study 1.'", "caption": "Article p. 4 (PDF p. 4), caption of Figure 1, item 2. The marked word is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015.png" }, { "alt": "Crop of five lines of the Introduction. A red box marks the words 'YouGov (according to 2021)' in the sentence 'A cursory search of the five most popular news websites in the U.S. YouGov (according to 2021) for the term deepfake on the news aggregator Google News returned 11,700 news articles discussing deepfakes'.", "caption": "Article p. 2 (PDF p. 2), second paragraph, item 1. In the marked words the name of the source stands outside the parenthesis and the year inside it.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-p2.png" }, { "alt": "Crop of five footnote lines. A red box marks the word 'reponded' in the sentence '27 percent thought the politician was a Democrat, 16 percent thought he was a Republican, and 27 percent reponded that he was Independent.'", "caption": "Article p. 4 (PDF p. 4), footnote 8, item 3. The marked word is misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-p4.png" }, { "alt": "Crop of the top of the page. A red box marks the word 'SECITON' in the heading 'SECITON 3: Discernment of Deepfakes Without Deepfake Information'. Two lines of text follow.", "caption": "Appendix p. 6, heading of Section 3, item 4. The marked word is misspelled; the contents on p. 1 of the Appendix have the same spelling.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a06.png" }, { "alt": "Crop of a footnote: '1 See Swaim, B. \\& Nussbaum, J. (2016). The Perfect Presidential Stump Speech. FiveThirtyEight. November. Retrieved from' followed by a link. A red box marks the word 'November.'", "caption": "Appendix p. 3, footnote 1, item 5. The marked month is the date given for the FiveThirtyEight article.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a03.png" }, { "alt": "Crop of the reference entry 'Swaim, Barton, and Jeff Nussbaum. 2016. The Perfect Presidential Stump Speech (October 3, 2016).' followed by a link, and of the first line of the next entry. A red box marks '(October 3, 2016)'.", "caption": "Article p. 14 (PDF p. 14), reference list, item 5. The marked date is given for the same article.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-p14.png" }, { "alt": "Crop of the last two rows of Table OA2 (totals 369, 406, 370, 215 and 1,360) and the note under the table: 'Lucid asks for participants' political party before they are able to enroll in any surveys. ANOVA f-statistic=2.84 (p less than .0369).' A red box marks the parenthesis with the p-value.", "caption": "Appendix p. 8, note to Table OA2, item 6. The marked expression gives the rounded $p$-value as an upper bound.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a08.png" }, { "alt": "Crop of six rows of Table OA3 with their percentages and counts. Red boxes mark the word 'trusworthy' in the label 'Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.)' and the word 'bussinessman' in the label 'Demographic Fact (e.g., white, male, bussinessman)'.", "caption": "Appendix p. 11, Table OA3, item 7. The two marked words in the row labels are misspelled.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a11.png" }, { "alt": "Crop of the last row of Table OA5 (95 percent lower bound: -0.73, -0.31, -0.75) and the note under it. Red boxes mark the words 'This is defined as ALL participants EXCEPT', split over two lines, in the note 'As mentioned in the PAP, this analysis excludes participants who are likely supporters of alternative medicine. This is defined as ALL participants EXCEPT those who BOTH oppose government mandates for vaccination AND support government subsidies for acupuncture procedures.'", "caption": "Appendix p. 13, note to Table OA5, item 8; the note to Table OA6 on p. 14 is the same. After the marked words the note describes the participants who are kept, and the sentence before them speaks of those who are excluded.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a13.png" }, { "alt": "Crop of footnote 4 at the foot of the page, with the page number 18. A red box marks the words 'See Table OA5.' at the end of the footnote, after 'was used in Study 2 and produced very similar treatment effects.'", "caption": "Appendix p. 18, footnote 4, item 9. The marked reference is to Table OA5; the results for the alternative wording are in Table OA7.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a18.png" }, { "alt": "Crop of three lines. A red box marks the words 'Section 3, Tables 3-4,' in the sentence 'As shown in Section 3, Tables 3-4, we saw no statistically significant differences.'", "caption": "Appendix p. 19, Section 5, Study 1, item 10. The marked reference names Section 3 and Tables 3-4; the results are in Section 4, Tables OA5 and OA6.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a19a.png" }, { "alt": "Crop of five lines. A red box marks the words 'Section 3, Table OA9,' in the sentence 'As seen in Section 3, Table OA9, we find that participants surveyed before the election have higher levels (0.14 points more) of belief that the video they watched is deepfaked than participants surveyed after the election'.", "caption": "Appendix p. 19, Section 5, Study 2, item 11. The marked reference names Section 3 and Table OA9; the estimate of 0.14 points is in Section 4, Table OA11.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a19b.png" }, { "alt": "Crop of three lines. A red box marks the words 'Section 3, Tables 6-7)' in the sentence 'Our results (see Section 3, Tables 6-7) are consistent with this theory in terms of estimate directions but are not statistically significant.'", "caption": "Appendix p. 19, Section 5, Pooled, item 12. The marked reference names Section 3 and Tables 6-7; the interactions are in Section 4, Tables OA8 and OA9.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a19c.png" }, { "alt": "Crop of the last eight lines of Section 6. Red boxes mark 'Table OA8)' in 'the interaction of the information treatment and Republican self-identification (see Table OA8)' and the first 'pro-choice' in 'it is not clear that all rank-and-file Republicans identify being pro-choice as a clearly Republican position; 31 percent of surveyed Republicans identified themselves as pro-choice in 2019.'", "caption": "Appendix p. 20, Section 6, items 13 and 14. The marked reference is to Table OA8, and the marked “pro-choice” is followed by the share of Republicans who call themselves pro-choice.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E015-a20.png" } ], "formal_release": null, "formal_statements": [], "headline": "Misprints and wrong cross-references: five references in the Appendix point to the wrong table or section, the last sentence of Appendix Section 6 says “pro-choice” where “pro-life” is meant, and the note to Table OA2 prints $p<.0369$ for $p=.0369$.", "id": "P034-E015", "location": "Article: p. 2 (PDF p. 2); Figure 1 caption and footnote 8, p. 4 (PDF p. 4); reference list, p. 14 (PDF p. 14). Appendix: contents, p. 1; footnote 1, p. 3; heading of Section 3, p. 6; note to Table OA2, p. 8; Table OA3, p. 11; notes to Tables OA5 and OA6, pp. 13–14; footnote 4, p. 18; Section 5, p. 19; Section 6, p. 20 (the same PDF pages of the Appendix)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\begin{enumerate}\n\\item Article p.~2: ``A cursory search of the five most popular news websites in the U.S. \\hl{YouGov (according to 2021)} for the term \\ldots''\n\\item Article p.~4, caption: ``Figure 1: \\hl{Schreenshot} of the video used in Study 1.''\n\\item Article p.~4, footnote 8: ``27\\% \\hl{reponded} that he was Independent''\n\\item Appendix pp.~1 and~6, heading: ``\\hl{SECITON} 3: Discernment of Deepfakes Without Deepfake Information''\n\\item Appendix p.~3, footnote 1: ``Swaim, B. \\& Nussbaum, J. (2016). The Perfect Presidential Stump Speech. FiveThirtyEight. \\hl{November}.'' Article p.~14, reference list: Swaim, Barton, and Jeff Nussbaum. 2016. ``The Perfect Presidential Stump Speech'' (\\hl{October 3, 2016}).\n\\item Appendix p.~8, note to Table OA2: ``ANOVA f-statistic=2.84 ($p\\hl{<}.0369$).''\n\\item Appendix p.~11, Table OA3, row labels: ``Subjective Evaluation of Character (e.g., \\hl{trusworthy}, traitor, etc.)'' and ``Demographic Fact (e.g., white, male, \\hl{bussinessman})''\n\\item Appendix pp.~13 and~14, notes to Tables OA5 and OA6: ``As mentioned in the PAP, this analysis excludes participants who are likely supporters of alternative medicine. \\hl{This is defined as} ALL participants EXCEPT those who BOTH oppose government mandates for vaccination AND support government subsidies for acupuncture procedures.''\n\\item Appendix p.~18, footnote 4: ``\\ldots\\ was used in Study 2 and produced very similar treatment effects. See \\hl{Table OA5}.''\n\\item Appendix p.~19: ``As shown in \\hl{Section 3, Tables 3-4}, we saw no statistically significant differences.''\n\\item Appendix p.~19: ``As seen in \\hl{Section 3, Table OA9}, we find that participants surveyed before the election have higher levels (0.14 points more) of belief that the video they watched is deepfaked than participants surveyed after the election ($p<.05$).''\n\\item Appendix p.~19: ``Our results (see \\hl{Section 3, Tables 6-7}) are consistent with this theory in terms of estimate directions but are not statistically significant.''\n\\item Appendix p.~20: ``While we do include results illustrating the interaction of the information treatment and Republican self-identification (see \\hl{Table OA8}), we cannot directly address the question of motivated reasoning with this data for three reasons.''\n\\item Appendix p.~20: ``And finally, it is not clear that all rank-and-file Republicans identify being \\hl{pro-choice} as a clearly Republican position; 31\\% of surveyed Republicans identified themselves as pro-choice in 2019.''\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item ``the five most popular news websites in the U.S. \\hl{(according to YouGov 2021)} for the term \\ldots''\n\\item ``Figure 1: \\hl{Screenshot} of the video used in Study 1.''\n\\item ``27\\% \\hl{responded} that he was Independent''\n\\item ``\\hl{SECTION} 3: Discernment of Deepfakes Without Deepfake Information'', in the contents and in the heading\n\\item The two entries should give the same date. The reference list of the article gives October 3, 2016, and the Appendix should agree with it once the date has been checked against the article.\n\\item ``ANOVA F-statistic=2.84 ($p\\hl{=}.0369$).''\n\\item ``\\hl{trustworthy}'' and ``\\hl{businessman}''\n\\item ``\\hl{The retained analysis sample comprises all participants except} those who both oppose government mandates for vaccination and support government subsidies for acupuncture procedures.''\n\\item ``See \\hl{Table OA7}.''\n\\item ``As shown in \\hl{Section 4, Tables OA5--OA6}, we saw no statistically significant differences.''\n\\item ``As seen in \\hl{Section 4, Table OA11}, we find that participants surveyed before the election \\ldots''\n\\item ``Our results (see \\hl{Section 4, Tables OA8--OA9}) are consistent with this theory \\ldots''\n\\item ``(see \\hl{Table OA10})''\n\\item ``\\ldots\\ identify being \\hl{pro-life} as a clearly Republican position; 31\\% of surveyed Republicans identified themselves as pro-choice in 2019.''\n\\end{enumerate}", "evidence": "The evidence is a check of the pages against the contents of the Appendix, with a recalculation for item 6 and, for item 8, a reading of the program for the supplementary materials. For item 6, I computed the one-way analysis of variance from the counts printed in Table OA2. The participant's party is scored 1 (Republican), 2 (Independent) and 3 (Democrat), as in the replication files, and the four guesses are the groups. A direct computation gives $F(3,1356)=2.836964$ and $p=0.036942$. For item 8, the condition in that program excludes a participant exactly when the participant both supports subsidies for acupuncture and opposes vaccine mandates.", "problem": "Each item is a slip in spelling, in a label or in a cross-reference, and the surrounding text shows what was meant. In item 6 the printed expression is false as written.\n\\begin{enumerate}\n\\item The source of the ranking is YouGov (2021). The name and the year have changed places with the words ``according to''.\n\\item The word is ``Screenshot''.\n\\item The word is ``responded''.\n\\item The word is ``SECTION'', as in the headings of the other sections.\n\\item The two entries date the same article differently. I did not check the date against the article.\n\\item From the twelve counts of Table OA2, the statistic is $F(3,1356)=2.837$ with $p=0.03694$. This value rounds to $.0369$ and is larger than $.0369$, and the printed inequality is false. The note prints the rounded $p$-value as a bound where an equality is meant.\n\\item The words are ``trustworthy'' and ``businessman''.\n\\item ``This'' has no clear antecedent. The clause that follows describes the participants who are kept, and the sentence before it speaks of those who are excluded. The program for the supplementary materials keeps every participant except those who both support subsidies for acupuncture and oppose vaccine mandates, as the clause says. The note can be read as defining the excluded group by that clause, which would reverse the sample.\n\\item The results for the alternative wording are in Table OA7 (p.~14), whose title and note quote that wording. Table OA5 reports favorability, vote choice and support for essential oils in Study 1.\n\\item Section 3 of the Appendix contains only Figure OA1. The secondary results for Study 1 are in Section 4, Tables OA5 and OA6 (pp.~13--14).\n\\item The estimate for the election, $-0.14$, is in Section 4, Table OA11 (p.~16). Table OA9 reports the interaction with being pro-life.\n\\item The interactions with party and with the stance on abortion are in Section 4, Tables OA8 and OA9 (p.~15).\n\\item Section 6 concerns dismissing the video as a deepfake. The interaction of the warning with Republican identification for those outcomes is in Table OA10 (p.~16), and Table OA8 reports it for vote choice and favorability.\n\\item The clause after the semicolon says that 31\\% of Republicans call themselves pro-choice. That share is a reason to doubt that all Republicans regard the pro-life position as their party's position. It is no reason to doubt this of the pro-choice position, which the article (p.~6) calls ``an unusual stance for a Republican''.\n\\end{enumerate}", "what_survives": "No estimate, interval or conclusion changes. The $p$-value of Table OA2 remains below $.05$. The analyses of Tables OA5 and OA6 use the sample that the notes intend. The Appendix still says, correctly, that the data cannot settle the question of motivated reasoning." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E015", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of three entries of the reference list. The middle entry reads: Harwell, Drew. 2021. Remember the 'deepfake cheerleader mom'? Prosecutors now admit they can't prove fake-video claims (March 14, 2021). https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/. A red outline surrounds the word March. The entry above is H.R.6395, the National Defense Authorization Act for Fiscal Year 2021, and the entry below is Kuklinski et al. 2000.", "caption": "References, p. 13 (PDF p. 13): the Harwell entry between its two neighbours. The outline marks the month “March”; the link printed in the same entry carries 2021/05/14.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E018-b.png" }, { "alt": "Crop of three entries of the reference list. The middle entry reads: Ternovski, John, and Lilla Orr. 2022. Evidence of rising rates of inattentiveness on Lucid in 2020, https://osf.io/preprints/socarxiv/8sbe4/. A red outline surrounds the words Ternovski, John, and Lilla Orr. 2022. The entry above is Swire et al. 2017 and the entry below is Thalen 2021, which is dated May 17, 2021.", "caption": "References, p. 14 (PDF p. 14): the entry cited in footnote 6, between its two neighbours. The outline marks the authors and year, which are those of the 2022 journal article; the title and link that follow are those of the 2020 memo.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E018.png" }, { "alt": "Crop of footnote 6. It reads: Lucid is an increasingly popular alternative to Amazon Mechanical Turk for social science survey research. Many well-known findings have been replicated on Lucid, suggesting the platform is capable of providing high-quality data (Coppock and McClellan 2019; Peyton, Huber, and Coppock 2021). During the COVID-19 pandemic, Ternovski and Orr (2022) found that Lucid data can provide reliable data when researchers screen on attentiveness, which we do here. A red outline surrounds the words Ternovski and Orr (2022).", "caption": "Footnote 6, p. 3 (PDF p. 3). The outline marks the citation “Ternovski and Orr (2022)”, which the footnote gives as the source of its statement about screening on attentiveness.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E018-c.png" } ], "formal_release": null, "formal_statements": [], "headline": "The Harwell reference should be dated May 14, 2021, not March 14, and the reference “Ternovski and Orr 2022” joins the authors and year of a 2022 journal article to the title and link of a 2020 memo by four authors.", "id": "P034-E018", "location": "References, entry “Harwell, Drew. 2021.”, p. 13 (PDF p. 13), and entry “Ternovski, John, and Lilla Orr. 2022.”, p. 14 (PDF p. 14), which is cited in footnote 6, p. 3 (PDF p. 3)", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{References, p.~13.} Harwell, Drew. 2021. ``Remember the `deepfake cheerleader mom'? Prosecutors now admit they can't prove fake-video claims'' (\\hl{March} 14, 2021). \\url{https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/}.\n\n\\emph{References, p.~14.} \\hl{Ternovski, John, and Lilla Orr. 2022.} ``Evidence of rising rates of inattentiveness on Lucid in 2020,'' \\url{https://osf.io/preprints/socarxiv/8sbe4/}.\n\n\\emph{Footnote~6, p.~3.} During the COVID-19 pandemic, Ternovski and Orr (2022) found that Lucid data can provide reliable data when researchers screen on attentiveness, which we do here.", "corrected": "\\emph{References, p.~13.} Harwell, Drew. 2021. ``Remember the `deepfake cheerleader mom'? Prosecutors now admit they can't prove fake-video claims'' (\\hl{May} 14, 2021). \\url{https://www.washingtonpost.com/technology/2021/05/14/deepfake-cheer-mom-claims-dropped/}.\n\n\\emph{References, p.~14.} The entry should cite one work in full. If the memo is the source of footnote~6, then footnote~6 cites Aronow et al.\\ (2020) and the entry is\n\n\\hl{Aronow, P. M., Joshua Kalla, Lilla Orr, and John Ternovski. 2020.} ``Evidence of rising rates of inattentiveness on Lucid in 2020,'' \\url{https://osf.io/preprints/socarxiv/8sbe4/}.\n\nIf the journal article is the source, then the entry is\n\nTernovski, John, and Lilla Orr. 2022. ``\\hl{A Note on Increases in Inattentive Online Survey-Takers Since 2020}.'' \\emph{Journal of Quantitative Description: Digital Media} 2. \\url{https://doi.org/10.51685/jqd.2022.002}.\n\nIn the second case, footnote~6 should describe screening on attentiveness as a way to reduce measurement error, to be used with continuing checks of data quality, and not as sufficient to validate the data.", "evidence": "The evidence is a check of the pages against the public pages of the cited sources. The page of the Harwell article at the publisher displays the date May 14, 2021 below the headline. The journal lists Ternovski and Orr as the authors of ``A Note on Increases in Inattentive Online Survey-Takers Since 2020'' (\\emph{Journal of Quantitative Description: Digital Media} 2, DOI 10.51685/jqd.2022.002, published on 7 February 2022). A library catalogue entry for the memo, with DOI 10.31235/osf.io/8sbe4, lists the four authors and the date 15 September 2020. The full text of the memo and the history of its versions on OSF were not obtained. These sources do not exclude that the two works are stages of one project or that the author list of the memo changed between versions.", "problem": "The reference list has two slips. In the Harwell entry, the date in parentheses is March 14, 2021, while the publisher dates the article May 14, 2021, as the link of the entry itself does (\\texttt{2021/05/14}).\n\nThe second entry joins the data of two publications. The title ``Evidence of rising rates of inattentiveness on Lucid in 2020'' and the SocArXiv identifier \\texttt{8sbe4} belong to a memo that is catalogued under four authors, Aronow, Kalla, Orr and Ternovski (2020). The pre-analysis plan of Study~2 cites the memo under those four names (p.~6, footnote~3). The authors and year of the entry, Ternovski and Orr (2022), belong to the journal article ``A Note on Increases in Inattentive Online Survey-Takers Since 2020'', which has a different title and a different DOI. The entry does not identify which of the two footnote~6 relies on, and the evidence does not determine which one the entry was meant to cite.", "what_survives": "No result of the paper depends on either entry. I did not establish whether the statement of footnote~6 about Lucid is correct. Both works favour screening on attentiveness. The journal article adds that attention checks ``do contain drawbacks and are not a panacea'' (pp.~25--26), and the abstract of the memo recommends stringent attention checks." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E018", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of six lines of the article under the heading on heterogeneous treatment effects. The marked sentence reads: We found some evidence that participants surveyed before the election displayed higher levels of distrust in political videos than those surveyed afterwards (p less than 0.05). A red outline surrounds the parenthetical p less than 0.05.", "caption": "Section 3.1, p. 9 (PDF p. 9). The outline marks the $p$-value reported for the difference between participants surveyed before and after the election.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019.png" }, { "alt": "Crop of Table OA11 with its title: Study 2, Some evidence of more distrust immediately before a major election. The table has columns Deepfake?, Unconvinced and Don't Know if Real. Red outlines surround the three entries of the first column: minus 0.14 for the Post Election Indicator, minus 0.01 for the 95 percent Upper Bound and minus 0.27 for the 95 percent Lower Bound. The other columns read minus 0.05, 0.02, minus 0.12 and minus 0.01, 0.04, minus 0.05.", "caption": "Appendix, Table OA11, p. 16. The outlines mark the three entries of the column “Deepfake?”, which differ from the table saved in the replication files.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019-b.png" }, { "alt": "Listing of four numbered lines of Stata code under the heading ORIGINAL CODE: 2-study2_data_cleanup_FINAL.do, SOURCE LINES 312-315. Line 313 is the comment: respondent's device: computer fl_25 ; mobile fl_87. Line 314 generates mobile equal to 0 if fl_25 is empty and fl_87 is not empty. Line 315 replaces mobile by 1 if fl_25 is not empty and fl_87 is empty. Nothing is outlined.", "caption": "Replication code, 2-study2_data_cleanup_FINAL.do, lines 312–315. The comment on line 313 gives the mapping of the two routes to devices, and lines 314–315 assign mobile the opposite values.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019-code-a.png" }, { "alt": "Listing of numbered excerpts of Stata code under the heading ORIGINAL CODE: 4-Supplementary-Materials_FINAL.do. Lines 3 to 7 include local figures=0 and local tables=0. Lines 190 to 193 open a block with the condition that figures equals 1. Lines 207 to 211 define the global macro covars as c.age i.gender i.party i.region c.hhi elder-pro_vax conlib imputed_conlib bigfive* pol_know_total polinterest and run a regression with it. Lines 597 to 604 set up the saved table trust_SM_table.xlsx and, on line 603, regress trust_tv_actually on tr_warn and the macro covars for Study 2 participants with romney_party equal to 1. Nothing is outlined.", "caption": "Replication code, 4-Supplementary-Materials_FINAL.do, excerpts. Lines 6–7 set both switches to zero, lines 207–208 define covars inside the block for figures that opens at line 190, and line 603 is a table regression that reads the macro.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E019-code-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "In the replication files, the device indicator of Study 2 is reversed, the supplementary script uses the covariate macro before defining it when only tables are requested, and the saved tables for Tables OA7, OA11 and OA12 differ from the printed ones; no printed estimate is shown to be wrong.", "id": "P034-E019", "location": "Section 3.1, p. 9 (PDF p. 9); Appendix, Tables OA7 (p. 14), OA11 (p. 16) and OA12 (p. 17), and the saved tables for them; replication code, lines 313–315 of 2-study2_data_cleanup_FINAL.do and lines 6–7, 593–766 and 840–841 of 4-Supplementary-Materials_FINAL.do", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{\\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}, lines 313--315.}\n\n\\texttt{/*respondent's device: computer fl\\_25 ; mobile fl\\_87*/}\n\n\\texttt{gen mobile=\\hl{0} if fl\\_25==\"\" \\& fl\\_87!=\"\"}\n\n\\texttt{replace mobile=\\hl{1} if fl\\_25!=\"\" \\& fl\\_87==\"\"}\n\n\\emph{\\texttt{4-Supplementary-Materials\\_FINAL.do}, lines 6, 7 and 594.}\n\n\\texttt{local figures=0}\n\n\\texttt{local tables=0}\n\n\\texttt{reg trust\\_tv\\_actually tr\\_warn \\hl{\\$covars} if romney\\_party==1 \\& study==2, robust}\n\n\\emph{Appendix, Table OA11, p.~16, column ``Deepfake?''.} Post Election Indicator: $-0.14$. 95\\% Upper Bound: $-0.01$. 95\\% Lower Bound: $-0.27$.\n\n\\emph{Section 3.1, p.~9.} We found some evidence that participants surveyed before the election displayed higher levels of distrust in political videos than those surveyed afterwards ($p<0.05$).", "corrected": "\\emph{\\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}, lines 314--315.}\n\n\\texttt{gen mobile=\\hl{1} if fl\\_25==\"\" \\& fl\\_87!=\"\"}\n\n\\texttt{replace mobile=\\hl{0} if fl\\_25!=\"\" \\& fl\\_87==\"\"}\n\n\\emph{\\texttt{4-Supplementary-Materials\\_FINAL.do}.} The definition of lines 840--841 is to be placed before line 593, outside the blocks for figures:\n\n\\texttt{\\hl{global covars c.age i.gender i.party i.region c.hhi elder-pro\\_vax conlib ///}}\n\n\\texttt{\\hl{imputed\\_conlib bigfive* pol\\_know\\_total polinterest}}\n\nI reproduce the printed tables in the translated calculation of \\fid{P034-N011}. After the native scripts are repaired, the saved workbooks for Tables OA7, OA11 and OA12 should be regenerated from the Study 2 data and replaced with verified outputs, with any remaining difference from the printed tables explained.", "evidence": "I checked the two device assignments, traced the supplementary script and compared the saved tables with the printed ones. I then recalculated Tables OA7, OA11 and OA12 from the Study 2 data in \\fid{P034-N011}, translating the relevant code and using the supplied cleaned covariates. Stata was not run. The recalculation reproduces the printed tables under the code sample; it does not establish how the discrepant saved workbooks were produced.\n\\begin{audittable}{llll}{The columns of Tables OA7, OA11 and OA12 in which the saved tables differ from the printed ones. Each entry is an estimate with its lower and upper bounds, and the saved values are rounded to three decimals.}\nTable & Column & Saved in the replication files & Printed \\\\\nOA7 & Trust TV & $-0.012$ $[-0.095, 0.070]$ & $0.01$ $[-0.07, 0.09]$ \\\\\nOA7 & Trust Online & $-0.032$ $[-0.117, 0.052]$ & $-0.04$ $[-0.12, 0.04]$ \\\\\nOA11 & Deepfake? & $-0.121$ $[-0.255, 0.012]$ & $-0.14$ $[-0.27, -0.01]$ \\\\\nOA11 & Don't Know if Real & $-0.003$ $[-0.047, 0.042]$ & $-0.01$ $[-0.05, 0.04]$ \\\\\nOA12 & Deepfake? & $0.132$ $[-0.135, 0.400]$ & $0.12$ $[-0.14, 0.39]$ \\\\\n\\end{audittable}\nIn the first column of Table OA11, the width of the saved interval implies a standard error of $0.0681$. The saved estimate is then $1.777$ standard errors from zero, which is less than $1.96$. The saved tables for Tables OA5, OA6, OA8, OA9 and OA10 agree with the printed tables in every entry.", "problem": "There are three defects in the replication files. First, the cleaning script of Study 2 assigns \\texttt{mobile} the values opposite to its own comment and to the cleaning script of Study 1: a participant with only the mobile route \\texttt{fl\\_87} filled gets $0$, and a participant with only the computer route \\texttt{fl\\_25} filled gets $1$. Second, the supplementary script has one switch for figures and one for tables, and every definition of the global macro \\texttt{covars} before line 840 lies inside a block that runs only when the switch for figures is on. With the switch for tables on and the switch for figures off, 16 regressions in the table blocks read the macro before it is defined, among them the regressions for Tables OA7, OA11 and OA12. The macro is then empty in a fresh Stata session, and it holds the six covariates of Study 1 if the main analysis script has run first, as the README instructs; in neither case do these regressions use the expanded covariate list of Study 2. Third, the saved tables for Tables OA7, OA11 and OA12 differ from the printed tables in 5 of 6, 4 of 9 and 3 of 9 entries. In the first column of Table OA11, the saved estimate is $-0.121$ with the interval $[-0.255, 0.012]$, which contains zero. The printed entry is $-0.14$ with the interval $[-0.27, -0.01]$, and the sentence on p.~9 reports $p<0.05$. The replication files do not show which specification or data produced the saved tables, or whether the undefined macro explains the difference.", "what_survives": "No printed estimate, interval or $p$-value is shown to be wrong. The device indicator enters no regression. In Tables OA7 and OA12, every interval contains zero in the saved version and in the printed one. The main analysis script and the supplementary script each define the covariates of Study 2 before their own regression for the comparison across the election, so the model behind the sentence on p.~9 is specified apart from the table blocks. In the recalculation of \\fid{P034-N011}, I reproduce all 24 printed entries of Tables OA7, OA11 and OA12 with the code sample; the election comparison has $p=0.038$, and $p=0.047$ with the plan sample, so the $p<0.05$ sentence stands. This does not establish how the discrepant saved tables were produced." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E019", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of six lines of the article. The text reads: The leftmost panel of Figure 2 on the next page illustrates that when the information about deepfakes was randomly added to a deepfake video, participants were 0.5 points more likely to believe the video they were watching was fabricated (p less than 0.001), followed by footnote mark 14 and the start of the next sentences. A red outline surrounds the parenthetical p less than 0.001.", "caption": "Section 3, p. 7 (PDF p. 7). The outline marks the $p$-value printed for the effect of 0.5 points in the deepfake-video arm of Study 1.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E003.png" }, { "alt": "Crop of the eight-line paragraph of the Appendix that begins: The major deviation is the use of Manski bounds for confidence intervals and statistical significance. A red outline surrounds the words for confidence intervals and statistical significance. The paragraph goes on to say that the distribution of outcomes that least favors the hypothesis is imposed on all missing outcomes, the survey instrument maximum in the control and the minimum in treatment, that a covariate-adjusted regression with robust standard errors is run, and that the lower bound of its 95 percent confidence interval is the lower bound of the reported interval, with the same done for a best-case distribution to obtain the upper bound.", "caption": "Appendix, Section 2, Study 1, p. 5. The outline marks the statement that the bounds are used for confidence intervals and statistical significance; the rest of the paragraph describes how missing outcomes are filled in.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E003-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The $p<0.001$ printed for the Study 1 effect in the deepfake-video arm is the complete-case value; in the worst-case regression that the Appendix prescribes for statistical significance, $p \\approx 0.003$, and the pooled $p$-values are computed without the bounds.", "id": "P034-E003", "location": "Section 3, paragraph beginning “The leftmost panel of Figure 2”, p. 7 (PDF p. 7); Appendix, Section 2, Study 1, p. 5; pooled estimates on pp. 8–9 and in the Appendix, Section 5, p. 19", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Section 3, p.~7.} The leftmost panel of Figure 2 on the next page illustrates that when the information about deepfakes was randomly added to a deepfake video, participants were 0.5 points more likely to believe the video they were watching was fabricated ($\\hl{p<0.001}$).\n\n\\emph{Appendix, Section 2, Study 1, p.~5.} The major deviation is the use of Manski bounds for confidence intervals \\hl{and statistical significance}. [\\ldots] We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment), run a covariate-adjusted OLS regression with robust standard errors over all outcomes, and use the lower bound of the resulting 95\\% confidence interval as the lower bound of our reported 95\\% CI.", "corrected": "\\emph{Section 3, p.~7.} [\\ldots] participants were 0.5 points more likely to believe the video they were watching was fabricated (\\hl{complete-case regression, $n = 685$:} $p<0.001$\\hl{; with missing outcomes set to the values that least favor the hypothesis, as in Section 2 of the Appendix, $n = 700$: estimate $0.35$, $p \\approx 0.003$; the 95\\% interval formed from the worst- and best-case regressions is $[0.12, 0.85]$}).\n\n\\emph{Section 3, pp.~8--9, and Appendix, Section 5, p.~19.} \\hl{The pooled estimates and their $p$-values come from robust regressions with a study fixed effect and shared covariates, without the Study 1 bounds; the pooled regression for intention to vote codes a missing vote as zero.}", "evidence": "The evidence is a recalculation from the Study 1 raw data, following the cleaning and analysis code in the replication files, together with an exact lower bound for the $p$-value of the worst-case regression. The deepfake-video arm has 349 control participants, of whom 6 have a missing outcome, and 351 treated participants, of whom 9 have a missing outcome. The worst-case regression keeps every observed outcome, sets the 6 missing control outcomes to $3$ and sets the 9 missing treated outcomes to $-3$; these are the ends of the scale. Each regression has an intercept, the treatment indicator and seven covariate columns, and its standard errors are the robust ones that Stata reports.\n\\begin{audittable}{lrrrrr}{The effect of the information treatment in the deepfake-video arm of Study 1 in the three regressions.}\nRegression & $n$ & Estimate & Robust SE & Degrees of freedom & Two-sided $p$ \\\\\nComplete case & 685 & 0.4988516 & 0.1163098 & 676 & 0.0000206 \\\\\nWorst case & 700 & 0.3549310 & 0.1193355 & 691 & 0.0030396 \\\\\nBest case & 700 & 0.6235811 & 0.1178226 & 691 & 0.0000002 \\\\\n\\end{audittable}\nFollowing the rule of the Appendix, with the multiplier $1.96$ that the program uses, I obtain the ends of the 95\\% interval:\n\\[\n0.3549310 - 1.96 \\times 0.1193355 \\approx 0.121033, \\qquad 0.6235811 + 1.96 \\times 0.1178226 \\approx 0.854513 .\n\\]\nThis interval agrees with the one drawn in the left panel of Figure 2, and it excludes zero.\n\nIn the worst-case regression, the $t$ statistic is $2.974$ on 691 degrees of freedom. The bound $p > 0.001$ does not depend on a numerical evaluation of the $t$ distribution. In exact rational arithmetic, the estimate $b$ and its robust variance $\\widehat{V}$ satisfy $b > 0$ and $b^2/\\widehat{V} < 9$ (the ratio is $8.846$), which implies that $0 < t < 3$. The density of the $t$ distribution with 691 degrees of freedom is\n\\[\nf(x) = \\frac{C}{\\pi \\sqrt{691}} \\Bigl( 1 + \\frac{x^2}{691} \\Bigr)^{-346}, \\qquad C = \\frac{4^{345} (345!)^2}{690!},\n\\]\nand it is decreasing for $x > 0$. Since $t < 3$, I have\n\\[\np = 2 \\int_t^\\infty f(x) \\, dx > 2 \\int_3^{33/10} f(x) \\, dx \\ge \\frac{3}{5} f\\Bigl( \\frac{33}{10} \\Bigr) > \\frac{3}{5} \\cdot \\frac{C}{(22/7) \\cdot 27} \\Bigl( 1 + \\frac{(33/10)^2}{691} \\Bigr)^{-346} .\n\\]\nIn the last inequality, I used that $\\pi < 22/7$ and $\\sqrt{691} < 27$. The right side is a rational number, approximately $0.0010409$, and it exceeds $1/1000$. A two-sided normal test gives $0.0029$ and a one-sided $t$ test gives $0.0015$; neither restores the printed threshold. One admissible completion of the missing outcomes suffices for this conclusion; I do not claim that $0.0030$ is the largest $p$-value over all completions.\n\nIn the real-video arm of Study 1, I find $p = 0.0000165$ in the worst-case regression, with the interval $[0.2877, 1.0999]$. The code shows that the pooled regressions omit the bounds. I reproduced the three pooled estimates of pp.~8--9 without bounds, as reported in Reproduction. I did not compute the pooled regressions with the Study 1 bounds.", "problem": "The Appendix states that, in Study 1, bounds are used for confidence intervals and for statistical significance. To form the lower end of an interval, each missing outcome is set to the value that least favors the hypothesis, and a covariate-adjusted regression with robust standard errors is fitted to all randomized participants. I call this the worst-case regression, after the ``worst-case bounds'' of p.~5 of the article, and I call the regression with the opposite values the best-case regression. The $p<0.001$ on p.~7 does not come from this procedure. It is the $p$-value of the complete-case regression, which drops the 15 participants in the deepfake-video arm whose outcome is missing. In the worst-case regression the two-sided $p$-value is $0.0030$. Under the method that the Appendix announces, the effect is therefore significant at the 1\\% level and not at the 0.1\\% level. The page does not say which regression the parenthetical reports. The program that computes the bounds returns a point estimate and the two ends of the interval, and no $p$-value.\n\nThe pooled estimates raise a second point. The estimates of 0.40 points, 0.13 points and 6.0 percentage points on pp.~8--9, and of 3.8 percentage points and 0.17 points in Section 5 of the Appendix, come from robust regressions with a study fixed effect and shared covariates. These regressions include the real-video arm of Study 1, where 17 control participants and 4 treated participants did not complete the survey, and they do not use the Study 1 bounds. Four of the five regressions drop participants whose outcome is missing. The fifth, for intention to vote, uses an indicator that the vote equals yes, which codes the 11 missing votes in the real-video arm of Study 1 as zero (see \\fid{P034-E011}). The paper does not promise bounds for the pooled analyses, and the note to Figure OA3 separates the Study 1 bounds from the pooled regression for the outcomes of that figure. The pooled $p$-values are not shown to be wrong, but the text does not say that they are computed without the bounds.", "what_survives": "The complete-case estimate of 0.5 points, the intervals in Figure 2 and the positive effect in the deepfake-video arm at the 5\\% level are unaffected, and the effect remains significant at the 1\\% level in the worst-case regression. In the real-video arm of Study 1, $p<0.001$ holds in the worst-case regression as well. The Study 2 analyses do not use the bounds. No pooled estimate or pooled $p$-value is shown to be wrong; whether the pooled results remain significant with bounds was not checked. What changes is the significance level on p.~7 under the method of the Appendix, and the description of the pooled analyses." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E003", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of six lines of Section 2 of the Appendix. Two red boxes mark the words 'onto all missing outcomes', split over two lines, in the sentence 'We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment)'.", "caption": "Appendix, Section 2, p. 5 (PDF p. 5 of the Appendix). The marked words say which values are replaced by the extremes: the missing outcomes.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E005.png" }, { "alt": "Rendering of fourteen numbered lines of Stata code under the headings 'ORIGINAL CODE: manski_ci.ado' and 'SOURCE LINES 14-27'. Red boxes mark the condition complete==0, written with Stata macro quotes, at the end of line 20, 'replace outcomenew=max if treat==0 and complete==0', and of line 21, 'replace outcomenew=min if treat==1 and complete==0'. Each of the two lines carries the comment 'replace missing outcomes with bounds'.", "caption": "Replication files, manski_ci.ado, lines 14–27, in a rendering of the released file. The two marked conditions select every respondent whose completion flag is zero, whether or not the outcome analyzed is missing.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-E005-code.png" } ], "formal_release": null, "formal_statements": [], "headline": "The bounding program replaces every outcome of the 36 respondents who did not answer the last survey item, including answers that they gave, and ten intervals for secondary outcomes of Study 1 are wider than under the procedure described in the Appendix.", "id": "P034-E005", "location": "Appendix, Section 2, Study 1, p. 5 (PDF p. 5 of the Appendix), sentence beginning “We impose the distribution of outcomes”; replication files, manski_ci.ado, lines 19–21 and 36–38. The article refers to this section on p. 5 (PDF p. 5)", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Appendix p.~5: ``We impose the distribution of outcomes that least favors our hypothesis \\hl{onto all missing outcomes} (i.e., the survey instrument maximum for all missing outcomes in the control and the survey instrument minimum for all missing outcomes in treatment), run a covariate-adjusted OLS regression with robust standard errors over all outcomes, and use the lower bound of the resulting 95\\% confidence interval as the lower bound of our reported 95\\% CI. We do the same with a best-case distribution of missing outcomes to estimate the upper bound of our reported 95\\% CI.''\n\n\\texttt{manski\\_ci.ado}, lines 19--21 (the worst case):\n\n\\texttt{gen `outcomenew'=`outcome'}\n\n\\texttt{replace `outcomenew'=`max' if `treat'==0 \\& \\hl{`complete'==0}}\n\n\\texttt{replace `outcomenew'=`min' if `treat'==1 \\& \\hl{`complete'==0}}\n\nLines 36--38 (the best case) are the same with \\texttt{max} and \\texttt{min} exchanged.", "corrected": "Either the description or the program can be changed. To keep the printed numbers, the Appendix should say what the program does: ``\\hl{For every outcome, we treat all answers of a respondent who did not answer the final survey item as missing, including answers given to earlier questions.} We impose the distribution of outcomes that least favors our hypothesis onto all missing outcomes \\ldots''\n\nTo carry out the procedure as described, each of the four replacement conditions (lines 20, 21, 37 and 38 of \\texttt{manski\\_ci.ado}) should test the outcome itself. For example, line 20 becomes\n\n\\texttt{replace `outcomenew'=`max' if `treat'==0 \\& \\hl{missing(`outcome')}}\n\nThe Appendix can then read: ``\\hl{For each outcome, we retain every observed answer and replace only missing values} with the instrument maximum in control and minimum in treatment for the lower endpoint, reversing these assignments for the upper endpoint. We fit the same adjusted regressions with robust standard errors \\hl{and recompute the affected intervals}.''\n\nIn full, let $[L,U]$ be the range of the instrument. The two imputed outcomes are\n\\[\nY_i^{\\mathrm{low}}=\\begin{cases}Y_i&\\text{if }M_i=0,\\\\U&\\text{if }M_i=1\\text{ and }T_i=0,\\\\L&\\text{if }M_i=1\\text{ and }T_i=1,\\end{cases}\n\\qquad\nY_i^{\\mathrm{up}}=\\begin{cases}Y_i&\\text{if }M_i=0,\\\\L&\\text{if }M_i=1\\text{ and }T_i=0,\\\\U&\\text{if }M_i=1\\text{ and }T_i=1.\\end{cases}\n\\]\nThe reported interval is $[\\hat\\beta^{\\mathrm{low}}-1.96\\,\\widehat{\\mathrm{se}}^{\\mathrm{low}},\\ \\hat\\beta^{\\mathrm{up}}+1.96\\,\\widehat{\\mathrm{se}}^{\\mathrm{up}}]$, where $\\hat\\beta$ and $\\widehat{\\mathrm{se}}$ are the coefficient on $T_i$ and its robust standard error in the covariate-adjusted regression of the imputed outcome. With this rule the intervals of Table OA5 become, at the printed precision, $[-0.61,0.17]$ for favorability and $[-0.26,0.06]$ for the vote, and those of Table OA6 become $[-0.22,0.59]$ and $[-0.12,0.22]$. The intervals for essential oils are unchanged.", "evidence": "The evidence is one respondent in the data who shows the discrepancy exactly, and a recalculation from the Study 1 raw data. A respondent in an arm without the warning did not answer the last item and answered ``Extremely unfavorable'' ($-3$) to the question on favorability. In the fit for the lower endpoint, the program replaces this answer by $+3$. The rule of the Appendix keeps $-3$ in both fits.\n\nI rebuilt the Study 1 analysis file from the raw data with the released cleaning rules. I then computed each interval twice, once as the program does and once replacing only missing values. The first computation agrees with all 18 printed entries of Tables OA5 and OA6. Of the 17 intervals computed, ten change.\n\\begin{audittable}{lrr}{The ten intervals that change, as the program computes them and when only missing values are replaced. The Appendix prints the first four to two decimals and shows the other six in figures.}\nInterval & As coded & Only missing values replaced \\\\\n\\hline\nTable OA5, favorability & $[-0.732,\\,0.248]$ & $[-0.611,\\,0.167]$ \\\\\nTable OA5, vote & $[-0.311,\\,0.079]$ & $[-0.265,\\,0.058]$ \\\\\nTable OA6, favorability & $[-0.322,\\,0.669]$ & $[-0.217,\\,0.595]$ \\\\\nTable OA6, vote & $[-0.162,\\,0.235]$ & $[-0.119,\\,0.220]$ \\\\\nFigure OA1, unconvinced & $[-0.169,\\,0.187]$ & $[-0.156,\\,0.176]$ \\\\\nFigure OA1, don't know & $[-0.123,\\,0.085]$ & $[-0.117,\\,0.079]$ \\\\\nFigure OA2a, real video & $[0.014,\\,0.358]$ & $[0.033,\\,0.340]$ \\\\\nFigure OA2b, real video & $[-0.045,\\,0.157]$ & $[-0.036,\\,0.148]$ \\\\\nFigure OA3, trust in footage on TV & $[-0.094,\\,0.233]$ & $[-0.087,\\,0.230]$ \\\\\nFigure OA3, trust in footage online & $[-0.221,\\,0.103]$ & $[-0.216,\\,0.098]$ \\\\\n\\end{audittable}\nEach interval as coded contains the interval obtained when only missing values are replaced. The seven intervals that do not change are the two of Figure 2, the first column of Figure OA1, the panels for the deepfake video in Figures OA2a and OA2b, and the columns for essential oils in Tables OA5 and OA6. No point estimate changes, and each of the 17 intervals contains zero under one rule if and only if it does under the other.", "problem": "The Appendix describes the bounds outcome by outcome: the extreme values are imposed on the missing values of the outcome that is analyzed, and the observed answers are kept. The program does something else. I write $Y_i$ for the outcome analyzed, $T_i$ for the treatment indicator passed to the program, and $M_i$ for the indicator that $Y_i$ is missing. Let $C_i$ denote the completion flag, which in every call is the indicator that respondent $i$ answered the last item of the survey, the guess of the politician's party. The program replaces $Y_i$ by an extreme value when $C_i=0$, and not when $M_i=1$. A respondent who answered an earlier question and then left the survey has $C_i=0$ and $M_i=0$ for that question, and the answer is overwritten in at least one of the two fits. The two rules agree for an outcome only if $M_i=1$ exactly when $C_i=0$.\n\nIn Study 1, 36 of the 1,396 randomized respondents did not answer the last item. All 36 are missing on the main outcome, the belief that the video is a deepfake, and on the question about essential oils, and for these two outcomes the rules coincide. For the other outcomes they do not. Of the 36, 11 answered the questions on favorability and on the vote, 6 answered the two questions on whether the politician believes what is said, and 2 answered each of the two questions on trust.", "what_survives": "The main result is untouched: for the outcome of Figure 2 the two rules replace the same 36 values, and the fits are identical. All point estimates stand. The conclusions drawn from the ten intervals also stand, since none changes its position relative to zero. In particular, the interval for the real video in Figure OA2a stays above zero. The Study 2 regressions and the pooled regressions do not use the program. What changes is the width of ten intervals for secondary outcomes of Study 1. That the wider intervals contain the narrower ones in these data is not a statement about coverage, and no failure of coverage is established." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-E005", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [ { "alt": "Crop of Table OA5 with its title: Study 1, Information about deepfakes has no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). The table has columns Favor, Vote and More Essential Oils in Medicine, and three rows. A red outline surrounds the label of the first row, Information Treatment Effect, whose entries are minus 0.22, minus 0.10 and minus 0.25. The row 95 percent Upper Bound has 0.25, 0.08 and 0.26, and the row 95 percent Lower Bound has minus 0.73, minus 0.31 and minus 0.75.", "caption": "Appendix, Table OA5, p. 13. The outline marks the row label “Information Treatment Effect”, which does not say that the effect is taken for the real video.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-A001.png" }, { "alt": "Crop of Table OA6 with its title: Study 1, Deepfakes have no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). The table has columns Favor, Vote and More Essential Oils in Medicine, and three rows. A red outline surrounds the label of the first row, Deepfake Treatment Effect, whose entries are 0.19, 0.05 and minus 0.14. The row 95 percent Upper Bound has 0.67, 0.24 and 0.39, and the row 95 percent Lower Bound has minus 0.32, minus 0.16 and minus 0.63.", "caption": "Appendix, Table OA6, p. 14. The outline marks the row label “Deepfake Treatment Effect”, which does not say that the effect is taken without the information treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-A001-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P034-T3", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T3" }, { "id": "P034-T4", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T4" } ], "headline": "Table OA5 reports the effect of the information treatment for the real video, and Table OA6 the effect of the deepfake without the information treatment; these are the main coefficients of models that contain the interaction of the two treatments, and not effects averaged over the other treatment.", "historical_evidence_type": "lean_proof", "id": "P034-A001", "location": "Appendix, Table OA5, p. 13, and Table OA6, p. 14 (titles, row labels and notes); cited from Section 3.1, p. 9 (PDF p. 9)", "pdf_page": 13, "printed_page": "13", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P034-T3", "P034-T4" ], "text": { "as_printed": "\\emph{Appendix, p.~13.} Table OA5: Study 1 -- Information about deepfakes has no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). \\hl{Information Treatment Effect}: $-0.22$ (Favor), $-0.10$ (Vote), $-0.25$ (More Essential Oils in Medicine?). 95\\% Upper Bound: $0.25$, $0.08$, $0.26$. 95\\% Lower Bound: $-0.73$, $-0.31$, $-0.75$.\n\n\\emph{Appendix, p.~14.} Table OA6: Study 1 -- Deepfakes have no impact on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). \\hl{Deepfake Treatment Effect}: $0.19$ (Favor), $0.05$ (Vote), $-0.14$ (More Essential Oils in Medicine?). 95\\% Upper Bound: $0.67$, $0.24$, $0.39$. 95\\% Lower Bound: $-0.32$, $-0.16$, $-0.63$.", "corrected": "\\emph{Table OA5.} Title: Study 1 -- Information about deepfakes has no \\hl{statistically significant effect, for the real video,} on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Row label: Information Treatment Effect \\hl{for the real video (deepfake $= 0$)}.\n\n\\emph{Table OA6.} Title: Study 1 -- Deepfakes have no \\hl{statistically significant effect, without information about deepfakes,} on favorability, vote choice, or support for essential oils in medicine (but we are underpowered). Row label: Deepfake Treatment Effect \\hl{without information about deepfakes (information $= 0$)}.\n\n\\emph{Note to both tables.} \\hl{Each estimate is a main coefficient of a covariate-adjusted model that contains the information treatment, the video type and their interaction, and the intervals use the bounds of Section 2. Table OA5 compares information with no information for the real video. Table OA6 compares the deepfake with the real video without information.} Averaging over the other treatment would define different contrasts; the displayed coefficients need only be labelled explicitly.", "evidence": "The evidence is a derivation of the two contrasts and a recalculation from the Study 1 raw data. Let $w$ and $v$ be the indicators of the information treatment and of the deepfake video, and let $z$ be the vector of covariates. I write the fitted model as\n\\[\nm(w, v, z) = a + b w + c v + d w v + \\gamma^{\\mathsf T} z .\n\\]\nFor fixed $z$, I have\n\\[\nm(1, v, z) - m(0, v, z) = b + d v, \\qquad m(w, 1, z) - m(w, 0, z) = c + d w .\n\\]\nTable OA5 reports $b$, which is the first contrast at $v = 0$, and Table OA6 reports $c$, which is the second contrast at $w = 0$. Suppose that a fraction $q$ of the target population has $v = 1$. Then the first contrast, averaged over $v$, is $(1 - q) b + q (b + d) = b + q d$. Similarly, the second contrast averaged over $w$ is $c + p d$, where $p$ is the fraction with $w = 1$. The main coefficient $b$ equals the averaged effect only if $q d = 0$, and $c$ only if $p d = 0$. For example, if the four cell means for $(w, v) = (0,0), (1,0), (0,1), (1,1)$ are $0, 0, 0, 2$, then $b = 0$ and $d = 2$, and the average with $q = 1/2$ is $1$.\n\nThe recalculation reproduces all 18 printed entries from the raw data, in the sample of 1,252 participants that remains after the exclusion stated in the notes. The fitted interaction coefficients are $-0.153$ for favorability, $-0.065$ for vote and $-0.041$ for essential oils.\n\\begin{audittable}{llll}{The main coefficients printed in Tables OA5 and OA6, beside the averages $b + d/2$ and $c + d/2$ from the same models. Each entry is an estimate with its interval, and the intervals for the averages apply the paper's procedure for the bounds to those contrasts.}\nContrast & Outcome & Main coefficient & Average, equal weights \\\\\nInformation & Favor & $-0.219$ $[-0.732, 0.248]$ & $-0.295$ $[-0.670, 0.063]$ \\\\\nInformation & Vote & $-0.102$ $[-0.311, 0.079]$ & $-0.134$ $[-0.285, 0.006]$ \\\\\nInformation & Oils & $-0.251$ $[-0.752, 0.263]$ & $-0.271$ $[-0.652, 0.109]$ \\\\\nDeepfake & Favor & $0.188$ $[-0.322, 0.669]$ & $0.111$ $[-0.254, 0.479]$ \\\\\nDeepfake & Vote & $0.049$ $[-0.162, 0.235]$ & $0.017$ $[-0.133, 0.159]$ \\\\\nDeepfake & Oils & $-0.137$ $[-0.634, 0.388]$ & $-0.157$ $[-0.531, 0.230]$ \\\\\n\\end{audittable}\nEvery interval in the table contains zero. In the complete cases, the averaged effects of the information treatment have $p$-values of $0.004$, $0.004$ and $0.014$. These tests leave out the missing outcomes that the bounds are meant to cover, and they do not overturn the reading of ``no impact'' given above.", "problem": "The row labels read as the effects of the two treatments. Neither the titles nor the notes say at which level of the other treatment each effect is taken. The models behind the tables contain both treatments and their interaction, and each table reports a main coefficient. Table OA5 therefore reports the effect of the information treatment for participants who saw the real video, and Table OA6 reports the effect of the deepfake for participants who did not receive the information treatment. An effect averaged over the other treatment is a different contrast, since it adds a share of the interaction coefficient to the main coefficient. For favorability, the $-0.22$ of Table OA5 can be taken for the averaged effect, which is $-0.295$ with equal weights. The secondary hypotheses of the Study 1 pre-analysis plan compare information with no information, and the deepfake with the real video, without conditioning on the other treatment; the tables do not report those contrasts. All 18 printed entries are correct for the contrasts that were fitted. The meaning of ``no impact'' is clear from the parenthetical in each title and from Section 5 of the Appendix: no difference is statistically significant with the bounds.", "what_survives": "All 18 entries stand as main coefficients with their bounds, and so does the reading that no difference is statistically significant with the bounds. The main results on disbelief in the video, the Study 2 results and the pooled regressions for favorability and vote are separate calculations and do not use these entries. What changes is the labelling: each table reports the effect of one treatment at the reference level of the other, and neither reports the averaged contrasts of the pre-analysis plan." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-A001", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of six lines of the Introduction. Two red boxes mark the citation '(Wittenberg et al. 2021', which is split over two lines, in the sentence 'recent randomized experiments on the impacts of deepfakes in American politics have found no evidence that people believe the content of the manipulated videos (Wittenberg et al. 2021; Vaccari and Chadwick 2020).'", "caption": "Introduction, first paragraph of p. 2 (PDF p. 2). The marked citation is given, together with Vaccari and Chadwick (2020), for experiments on deepfakes that found no evidence of belief in the manipulated videos.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N001.png" }, { "alt": "Crop of one entry of the reference list, which begins 'Wittenberg, Chloe, Ben M Tappin, Adam J Berinsky, and David G Rand. 2021.' A red box marks the title '(minimal) persuasive advantage of political video over text', which is followed by 'Proceedings of the National Academy of Sciences 118 (47)' and a DOI link.", "caption": "Reference list, p. 14 (PDF p. 14). The marked title of the cited article describes a comparison of political video with text.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N001-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "Wittenberg et al. (2021) is not an experiment on deepfakes and does not support the statement that such experiments found no evidence that people believe manipulated videos.", "id": "P034-N001", "location": "Introduction, first paragraph of p. 2 (PDF p. 2), sentence beginning “However, recent randomized experiments”; reference entry on p. 14 (PDF p. 14)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "p.~2: ``However, recent randomized experiments on the impacts of deepfakes in American politics have found no evidence that people believe the content of the manipulated videos (\\hl{Wittenberg et al. 2021}; Vaccari and Chadwick 2020). Barari, Lucas, and Munger (2021) found that deepfakes were persuasive, but that their effects were comparable to those of textual misinformation.''\n\np.~14, reference list: Wittenberg, Chloe, Ben M Tappin, Adam J Berinsky, and David G Rand. 2021. ``The (minimal) persuasive advantage of political video over text.'' \\emph{Proceedings of the National Academy of Sciences} 118 (47).", "corrected": "p.~2: ``However, \\hl{a recent randomized experiment on a deepfake of American political figures found that viewers were not significantly more likely to be deceived when the deepfake was shown without an educational reveal than with one, although they were more uncertain (Vaccari and Chadwick 2020)}. Barari, Lucas, and Munger (2021) found that deepfakes were persuasive, but that their effects were comparable to those of textual misinformation.''\n\nIf Wittenberg et al.\\ (2021) is cited, it should be cited separately, as a comparison of ordinary political video with text and not as an experiment on deepfakes. The comparison on p.~7 can stay qualitative; it is clearer if it names the comparator of that study, the same deepfake shown with the reveal.", "evidence": "The evidence is a check of the cited sources. I read Wittenberg et al.\\ (2021) in the publisher's typeset version posted by one of its authors (\\emph{PNAS} 118(47), seven pages), and Vaccari and Chadwick (2020) in the version of record (\\emph{Social Media + Society} 6(1), pp.~1--13). The passages quoted above are on p.~5 of the first and on p.~4 of the second. On p.~6 of the second, the analysis reads ``A smaller, though by no means negligible, group (16\\%) was deceived''.", "problem": "The sentence attributes to two studies the finding that randomized experiments on deepfakes found no evidence that people believe the manipulated videos. The first of the two, Wittenberg et al.\\ (2021), did not study deepfakes. Its experiments compared ordinary videos, on politics and on COVID-19, with transcripts and with controls. Its Discussion (p.~5) describes the subject as ``belief in unmanipulated content'' and says that ``future work should explore whether our results replicate in the context of misinformation, including deepfakes''. One of the two citations therefore does not support the sentence.\n\nThe second citation is an experiment on a deepfake, and its result is narrower than the sentence. Vaccari and Chadwick (2020) showed British respondents a deepfake of American political figures, with or without an educational reveal. They report that 16\\% of the respondents were deceived, that the difference in deception between the versions was not statistically significant, and that uncertainty was higher without the reveal. The design has no control group (p.~4: ``Our study does not include a control group'').", "what_survives": "The two experiments of the paper, their estimates and their bounds do not use the cited findings. What changes is the summary of earlier work in the motivation, which has to be narrowed. I did not establish that the phrase ``prior research has found that such videos have minimal effects'' in the abstract is wrong, or that the comparison with Vaccari and Chadwick on p.~7 is wrong. I could not obtain the 2021 preprint of Barari, Lucas and Munger, and the sentence about it was not checked against its text." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N001", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of five lines of the Introduction. A red box marks the citation 'Pennycook, Bear, et al. 2020' in the sentence 'Official warnings about fake news similarly induce participants to disbelieve true headlines (Clayton et al. 2020; Pennycook, Bear, et al. 2020).'", "caption": "Introduction, third paragraph of p. 2 (PDF p. 2). The marked citation is one of the two given for the statement that warnings lead participants to disbelieve true headlines.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N002.png" }, { "alt": "Crop of the first six lines of Section 4. Red boxes mark the words 'consistently found' and the citation 'Pennycook, Bear, et al. 2020' in the sentence 'Our findings add to the growing body of literature on textual misinformation, which has consistently found that warnings about fake news may help readers reject misinformation but may have the undesirable effect of increasing Americans' disbelief in true news stories (e.g., Pennycook, Bear, et al. 2020; Clayton et al. 2020).'", "caption": "Section 4, first sentence, p. 9 (PDF p. 9). The marked words “consistently found” and the marked citation repeat the attribution.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N002-b.png" }, { "alt": "Crop of one entry of the reference list, which begins 'Pennycook, Gordon, Adam Bear, Evan T Collins, and David G Rand. 2020.' Two red boxes mark the words 'increases perceived accuracy of headlines without warnings', split over two lines, at the end of the title 'The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings.'", "caption": "Reference list, p. 13 (PDF p. 13). The marked words of the title of the cited article state an increase in the perceived accuracy of headlines without warnings.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N002-c.png" } ], "formal_release": null, "formal_statements": [], "headline": "Pennycook, Bear, Collins and Rand (2020) does not report that warnings lower belief in true headlines; of the two works cited for that statement, only Clayton et al. (2020) supports it.", "id": "P034-N002", "location": "Introduction, third paragraph of p. 2 (PDF p. 2); Section 4, first sentence, p. 9 (PDF p. 9); reference entry on p. 13 (PDF p. 13)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "p.~2: ``Official warnings about fake news similarly induce participants to disbelieve true headlines (Clayton et al.\\ 2020; \\hl{Pennycook, Bear, et al. 2020}).''\n\np.~9: ``Our findings add to the growing body of literature on textual misinformation, which \\hl{has consistently found} that warnings about fake news may help readers reject misinformation but may have the undesirable effect of increasing Americans' disbelief in true news stories (e.g., \\hl{Pennycook, Bear, et al. 2020}; Clayton et al.\\ 2020).''\n\np.~13, reference list: Pennycook, Gordon, Adam Bear, Evan T Collins, and David G Rand. 2020. ``The implied truth effect: Attaching warnings to a subset of fake news headlines increases perceived accuracy of headlines without warnings.'' \\emph{Management Science} 66 (11): 4944--57.", "corrected": "p.~2: ``Official warnings about fake news similarly induce participants to disbelieve true headlines (\\hl{Clayton et al. 2020}).''\n\np.~9: ``Our findings add to the growing body of literature on textual misinformation, which \\hl{has found that general warnings about fake news can reduce belief in both false and true headlines (e.g., Clayton et al. 2020)}.''\n\nIf Pennycook, Bear, et al.\\ (2020) is kept, it should be described separately: when warnings were attached to a subset of false headlines, the headlines without warnings were rated as more accurate.", "evidence": "The evidence is a check of the cited sources. I read the version of record of Pennycook et al.\\ (\\emph{Management Science} 66(11), pp.~4944--4957), including the methods and results of both of its studies. For Clayton et al.\\ (\\emph{Political Behavior} 42, pp.~1073--1095) I read the publisher's abstract only, and I did not obtain the full text.", "problem": "Both sentences cite Pennycook, Bear, Collins and Rand (2020) for the finding that warnings about fake news lower belief in true headlines. On belief, that article reports the opposite. In its Study 1, the proportion rating true headlines accurate was .542 when warnings were attached to a subset of the false headlines and .530 in the control (p.~4948): ``headlines that were not tagged in the warning treatment were rated as more accurate than those in the control''. The article does report one adverse result for true headlines, in its Study 2 (p.~4953): untagged true headlines were shared at a rate of 35.2\\% under warnings plus verification and 40.1\\% in the control. That outcome is sharing and not belief. The other citation supports the statement: the abstract of Clayton et al.\\ (2020) says that ``exposure to a general warning decreased belief in the accuracy of true headlines''. The phrase ``has consistently found'' on p.~9 thus rests on one of the two works that it cites.", "what_survives": "The analogy with textual misinformation stands on Clayton et al.\\ (2020), and no design, estimate or test in the paper depends on the citation. The check does not show that the wider literature disagrees on this point; I did not review it." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N002", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the top of Table OA3, titled Study 2 - Hand-coded Romney facts across conditions, with the columns No Info about Deepfakes, Info about Deepfakes and Total. The rows shown, with percentages in the three columns and counts below them, are Don't Know / No Response Given (28.2, 27.0, 27.6 percent; N 818, 777, 1,595), Is Republican (16.5, 16.5, 16.5 percent; N 477, 475, 952), Ran for President (9.7, 10.7, 10.2 percent; N 281, 309, 590) and Subjective Evaluation of Character (e.g., trusworthy, traitor, etc.) (9.2, 9.3, 9.3 percent; N 266, 268, 534). A red outline surrounds the last percentage of the fourth row, 9.3 percent, in the Total column.", "caption": "Online Appendix, Table OA3, p. 11: the head of the table and its first four categories. The outline marks the Total percentage of “Subjective Evaluation of Character”, printed as 9.3% above the count 534.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N010.png" }, { "alt": "Crop of the bottom of Table OA3. The rows shown, with percentages in the three columns and counts below them, are Stance on Other Policies (0.8, 0.9, 0.9 percent; N 24, 25, 49), Is/Was Pro-Life (0.3, 1.4, 0.8 percent; N 9, 39, 48), Believe Video is Real (0.3, 0.5, 0.4 percent; N 9, 15, 24), Believe Video is Fake (0.2, 0.1, 0.2 percent; N 5, 4, 9) and Total (100, 100, 100 percent; N 2,898, 2,877, 5,775). A red outline surrounds the percentage 0.9 percent in the Total column of the first row.", "caption": "Online Appendix, Table OA3, p. 11: the last four categories and the foot of the table. The outline marks the Total percentage of “Stance on Other Policies”, printed as 0.9% above the count 49; the last row gives the column totals 2,898, 2,877 and 5,775.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N010-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P034-T5", "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T5" } ], "headline": "In the Total column of Table OA3, the percentage for “Subjective Evaluation of Character” should be 9.2%, not 9.3%, and the percentage for “Stance on Other Policies” should be 0.8%, not 0.9%.", "id": "P034-N010", "location": "Online Appendix, Table OA3, column “Total”, rows “Subjective Evaluation of Character” and “Stance on Other Policies”, p. 11 (Appendix PDF p. 11)", "pdf_page": 11, "printed_page": "11", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P034-T5" ], "text": { "as_printed": "\\begin{audittable}{lrrr}{Table OA3: Study 2 - Hand-coded Romney facts across conditions. The two rows at issue and the last row, each with its row of counts.}\n & No Info about Deepfakes & Info about Deepfakes & Total \\\\\nSubjective Evaluation of Character (e.g., trusworthy, traitor, etc.) & 9.2\\% & 9.3\\% & \\hl{9.3\\%} \\\\\nN & 266 & 268 & 534 \\\\\nStance on Other Policies & 0.8\\% & 0.9\\% & \\hl{0.9\\%} \\\\\nN & 24 & 25 & 49 \\\\\nTotal & 100\\% & 100\\% & 100\\% \\\\\nN & 2,898 & 2,877 & 5,775 \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{lrrr}{Table OA3, the two rows as corrected.}\n & No Info about Deepfakes & Info about Deepfakes & Total \\\\\nSubjective Evaluation of Character (e.g., trusworthy, traitor, etc.) & 9.2\\% & 9.3\\% & \\hl{9.2\\%} \\\\\nN & 266 & 268 & 534 \\\\\nStance on Other Policies & 0.8\\% & 0.9\\% & \\hl{0.8\\%} \\\\\nN & 24 & 25 & 49 \\\\\n\\end{audittable}\nEach percentage in the Total column is the count in its row divided by 5,775, the count at the foot of the column, rounded to one decimal place.", "evidence": "The evidence is a recalculation from the printed counts in exact decimal arithmetic. I have\n\\[\n\\frac{534}{5775}\\times 100 = 9.246753\\ldots, \\qquad \\frac{49}{5775}\\times 100 = 0.848484\\ldots,\n\\]\nand at one decimal place the first value rounds to $9.2$ and the second to $0.8$. The other 37 percentages of the table agree with their counts, and the counts of each column sum to the printed totals 2,898, 2,877 and 5,775.", "problem": "Each percentage in the table is the count in its row divided by the count at the foot of its column, rounded to one decimal place. In the Total column the counts are 534 and 49. Dividing them by 5,775, I find 9.2\\% and 0.8\\%. The table prints 9.3\\% and 0.9\\%.", "what_survives": "Only these two cells change. The counts, the other percentages of Table~OA3 and every analysis of the paper are unaffected." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N010", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the end of the passage on Study 1 and the passage on Study 2 in Section 2 of the Appendix. The last paragraph on Study 1 reads: We also had two minor deviations from the PAP. To be consistent with Study 2's PAP, we 1) dropped the one IP address that was randomized three times (there were no other IP duplicates that watched our video) and 2) used robust standard errors. These modifications have negligible impacts on the reported results. Under the heading Study 2, a red box marks the sentence 'There were no deviations from the PAP in Study 2.'", "caption": "Appendix, Section 2, p. 5 (PDF p. 5 of the Appendix). The marked sentence says that Study 2 had no deviations from its pre-analysis plan.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N011.png" }, { "alt": "Crop of Section 9, Exclusions, of the amended plan. After the sentence 'We exclude all participants who satisfy any one of the following conditions:' come four bullets: did not consent or meet the eligibility criteria; 'Have duplicate IP addresses and reached randomization.', marked with a red box; failed any one of the audio, visual, and attention checks; self-reported age was not within one year of their age according to Lucid. After a sentence on participants who could not identify Mitt Romney's political party, the text reads 'For SQ3_election1 and SQ4_election2, we exclude respondents who:' followed by two bullets, each marked with a red box: 'Take any portion of the survey between 8am and midnight November 3rd' and 'Or take any portion of the survey after 8am November 10th'.", "caption": "Amended pre-analysis plan for Study 2, Section 9, “Exclusions” (PDF p. 6 of the plan). The marked lines are the exclusion for duplicate IP addresses, which names respondents who reached randomization, and the two exclusions by time for the analyses before and after the election.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N011-b.png" }, { "alt": "Crop of Table OA11 with its title: Table OA11: Study 2 - Some evidence of more distrust immediately before a major election. The columns are Deepfake?, Unconvinced and Don't Know if Real. The row Post Election Indicator reads minus 0.14, minus 0.05, minus 0.01. The row 95 percent Upper Bound reads minus 0.01, 0.02, 0.04. The row 95 percent Lower Bound reads minus 0.27, minus 0.12, minus 0.05. Red boxes mark the first two entries of the column Deepfake?, minus 0.14 and minus 0.01.", "caption": "Appendix, Table OA11, p. 16 (PDF p. 16 of the Appendix). The table is computed on the 1,830 respondents who started the survey on November 1, 2, 8 or 9. With the exclusions of the amended plan, which leave 1,839 respondents, the two marked entries become $-0.13$ and $0.00$.", "url": "https://pmaronow.github.io/llm-audit/works/P034/exhibits/P034-N011-c.png" } ], "formal_release": null, "formal_statements": [], "headline": "The Appendix says that Study 2 had no deviations from its pre-analysis plan, but the analyses before and after the election drop nine eligible respondents whom the amended plan keeps (1,830 analyzed instead of 1,839), and the cleaning code drops rows with duplicated IP addresses without requiring that they reached randomization, a rule that the plan does not state.", "id": "P034-N011", "location": "Appendix, Section 2, “Study 2”, p. 5 (PDF p. 5 of the Appendix); Tables OA11 and OA12, Appendix pp. 16–17; Section 3.1, p. 9 (PDF p. 9); amended pre-analysis plan for Study 2, Section 9 (PDF p. 6 of the plan, whose pages are not numbered); replication files, lines 334–336 of 3-analysis_FINAL.do and lines 23–30 and 195–196 of 2-study2_data_cleanup_FINAL.do", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Appendix p.~5, Section 2, under ``Study 2'': ``\\hl{There were no deviations from the PAP in Study 2.}''\n\nAmended pre-analysis plan for Study 2, Section 9: ``We exclude all participants who satisfy any one of the following conditions:'' The second of the four conditions is ``Have duplicate IP addresses and reached randomization.'' The section continues: ``For \\textit{SQ3\\_election1} and \\textit{SQ4\\_election2}, we exclude respondents who:'' with the two conditions ``Take any portion of the survey between 8am and midnight November 3$^{\\text{rd}}$'' and ``Or take any portion of the survey after 8am November 10$^{\\text{th}}$''.\n\n\\texttt{3-analysis\\_FINAL.do}, lines 334--336:\n\n\\texttt{reg deepfake i.tr\\_warn i.post\\_election\\_ind \\$covars if romney\\_party==1 ///}\n\n\\texttt{\\& study==2 \\& \\hl{(stata\\_start==mdy(11,1,2020) | stata\\_start==mdy(11,2,2020) ///}}\n\n\\texttt{\\hl{| stata\\_start==mdy(11,8,2020) | stata\\_start==mdy(11,9,2020))}, robust}\n\n\\texttt{2-study2\\_data\\_cleanup\\_FINAL.do}, lines 23--25:\n\n\\texttt{duplicates tag ip, gen(dup)}\n\n\\texttt{tab dup}\n\n\\texttt{\\hl{drop if dup>0}}", "corrected": "Either the Appendix reports the two departures, or the code follows the plan.\n\nTo keep the printed tables, Appendix p.~5 can read: ``\\hl{Study 2 had two deviations from the PAP. First, the analyses for SQ3\\_election1 and SQ4\\_election2 (Tables OA11 and OA12) include only respondents who started the survey on November 1, 2, 8 or 9, 2020. This rule excludes nine respondents whom the exclusions of the PAP retain, so that these analyses use 1,830 respondents and not 1,839. Second, after removing previews and responses without respondent identifiers, we removed every remaining response whose IP address occurred more than once, whether or not it reached randomization, before imputing missing household income with the mean of the cleaned sample.}''\n\nTo follow the plan for these data, let $s_i$ and $e_i$ be the recorded start and end of the survey of respondent $i$, in 2020. The restriction to four start dates is replaced by the exclusion of respondent $i$ when\n\\[\n\\hl{\\bigl(s_i<\\text{November 4, 0:00 and } e_i\\ge\\text{November 3, 8:00}\\bigr)\\quad\\text{or}\\quad e_i>\\text{November 10, 8:00}}.\n\\]\nAt the precision shown, Tables OA11 and OA12 then take the values in the last column of the table above. In the cleaning, keep the existing population used to count duplicates, and identify randomized rows from the recorded treatment branches. Restrict \\texttt{drop if dup>0} \\hl{to those randomized rows}, and impute household income after that step. \\hl{Restrict the following \\texttt{isid rid} and \\texttt{isid ip} uniqueness checks to randomized rows as well.} The Appendix should also say among which rows a duplicate is counted.", "evidence": "The evidence is a recalculation from the public Study 2 data. The relevant cleaning and regression calculations were translated from Stata into Python, using the supplied cleaned data for other covariates; Stata was not run. The translated cleaning gives the 2,991 respondents of the cleaned data and the 1,925 of the analysis sample, 966 without and 959 with information. With the rules of the code, I reproduce all 24 printed entries of Tables OA7, OA11 and OA12. (The tables stored in the replication files differ from the printed ones; see \\fid{P034-E019}.) All 1,925 surveys have recorded start and end times; no endpoint lies on a boundary, and no survey spans the whole November 3 exclusion period. I then computed Tables OA11 and OA12 again with the exclusions of the plan.\n\\begin{audittable}{lll}{Tables OA11 and OA12 with the sample of the code and with the sample of the amended plan. Each entry is the coefficient with its interval of $\\pm 1.96$ robust standard errors, as in the Appendix.}\nTable and column & Code, $N=1{,}830$ & Plan, $N=1{,}839$ \\\\\n\\hline\nOA11, Deepfake? & $-0.140$ $[-0.272,\\,-0.008]$ & $-0.134$ $[-0.266,\\,-0.002]$ \\\\\nOA11, Unconvinced & $-0.047$ $[-0.116,\\,0.022]$ & $-0.047$ $[-0.116,\\,0.022]$ \\\\\nOA11, Don't Know if Real & $-0.009$ $[-0.053,\\,0.036]$ & $-0.005$ $[-0.050,\\,0.039]$ \\\\\nOA12, Deepfake? & $0.123$ $[-0.140,\\,0.386]$ & $0.133$ $[-0.129,\\,0.395]$ \\\\\nOA12, Unconvinced & $-0.059$ $[-0.195,\\,0.077]$ & $-0.057$ $[-0.193,\\,0.079]$ \\\\\nOA12, Don't Know if Real & $0.029$ $[-0.059,\\,0.116]$ & $0.027$ $[-0.061,\\,0.114]$ \\\\\n\\end{audittable}\nFor the first row, the $p$-value is $0.038$ with the sample of the code and $0.047$ with the sample of the plan. At the printed precision, six entries change with the sample of the plan. In Table OA11 the first column becomes $-0.13$ with upper bound $0.00$ (printed $-0.14$ and $-0.01$). In Table OA12 the first column becomes $0.13$ with lower bound $-0.13$ (printed $0.12$ and $-0.14$), the lower bound of the second column becomes $-0.19$ (printed $-0.20$), and the upper bound of the third becomes $0.11$ (printed $0.12$).\n\nFor the second departure, I repeated the cleaning and removed only the duplicated rows that reached randomization. The effect of the information on the main outcome of Study 2 is $0.263595$ with either mean of income, and no printed entry of Tables OA7, OA11 and OA12 changes.", "problem": "The code of Study 2 departs from the amended pre-analysis plan in two places, and the Appendix reports no deviation.\n\nThe first departure concerns the analyses before and after the election, which are Tables OA11 and OA12 and the sentence of Section 3.1 on p.~9. The plan excludes a respondent who took any portion of the survey between 8am and midnight on November 3, or after 8am on November 10. The code keeps a respondent only if the survey was started on November 1, 2, 8 or 9; the regressions behind the two tables, on lines 728--766 of \\texttt{4-Supplementary-Materials\\_FINAL.do}, carry the same restriction. Of the 1,925 respondents in the analysis sample, the code drops 95 and the plan drops 86. The nine respondents whom only the code drops took the survey outside the two excluded periods. Six started and finished on November 3 before 8am, one took it on the evening of November 4, and two started and finished on November 10 before 8am. The code keeps no respondent whom the plan excludes. The tables are computed on 1,830 respondents, and the plan specifies 1,839.\n\nThe second departure concerns duplicate IP addresses. The plan excludes participants who ``Have duplicate IP addresses and reached randomization.'' After removing previews and rows without respondent identifiers, the code removes every remaining row whose IP address occurs more than once, whether or not it reached randomization. At that step 176 rows have a duplicated address; 62 of them reached randomization and 114 did not. If duplicates are counted among these rows under both rules, then the two rules leave the same 1,925 respondents in the analysis sample. They do not leave the same cleaned file. Nine of the 114 rows pass every other step of the cleaning. The code replaces each missing value of household income by the mean over the cleaned file, and it does so before it selects the analysis sample. That mean is $387/37\\approx 10.459$ without the nine rows and $29877/2858\\approx 10.454$ with them, and 75 of the 1,925 respondents receive it. The plan does not say whether a duplicate is counted among all rows or only among those that reached randomization. On the second reading, 13 more respondents would enter the analysis sample.", "what_survives": "The sentence of Section 3.1 stands: the difference between respondents surveyed before and after the election has $p<0.05$ with either sample. Every interval of Table OA12 contains zero with either sample. The printed entries of Tables OA11 and OA12 are correct for the respondents who started the survey on November 1, 2, 8 or 9. The main effect of Study 2 remains $0.26$ with $p<0.001$ under either mean of income; I did not recompute every other coefficient or results under the alternative duplicate-counting rule. Study 1 does not pass through this code, and the other analyses of Study 2 and the pooled analyses do not use the restriction by date." }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-N011", "work": { "arxiv_id": null, "doi": "10.54501/jots.v1i2.28", "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "url": "https://pmaronow.github.io/llm-audit/works/P034/" }, "work_id": "P034" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P034.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:923ad8c1abf4a0a2f5e774ff85cd2d502c625295d778c033aeab2259e2a40e1c", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P034_Attrition.lean:353 bundles the specified witness in T1(i)-(ix): normalized positive laws, randomization, equal completion rates, outcome-dependent selection, the nonzero complete-case contrast and adjusted coefficient, the nonrejecting sample, and observational equivalence. Boolean coding and rational finite sums exactly represent this fixed binary construction, and the cell multiplicities at P034_Attrition.lean:210 preserve all 400 sample records relevant to the claimed statistics. Although the explicit coefficient quantifiers at P034_Attrition.lean:355 and :359 use rationals, the included determinant identity at P034_Attrition.lean:174 and sample scaling at :263 ensure the same unique solution over the reals, so no additional coefficient solutions are left open. The target explicitly requests this validator-supplied construction, and no material narrowing or omitted conclusion remains.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P034.missingness_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P034/P034_Attrition.lean#L349", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:923ad8c1abf4a0a2f5e774ff85cd2d502c625295d778c033aeab2259e2a40e1c", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P034-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P034-T1" }, "html": { "body": "

amended Study 2 pre-analysis plan, Section 8, PDF pp. 5-6; Appendix p. 9, Figure OA2a note, and p. 18, Figure OA3 note. The specified randomized finite population has equal completion rates but outcome-dependent missingness and a biased complete-case OLS coefficient, including the nonrejecting sample and the observationally equivalent independent-completion population with a different treatment effect. Probabilities and moments use exact finite rational sums; both OLS uniqueness assertions quantify over real coefficient vectors.

", "headline": "Amended Study 2 pre-analysis plan, Section 8 (Missing Data): the rule “If there is no differential attrition, we conclude that missingness occurs at random”, repeated as a justification in the notes to Appendix Figures OA2a and OA3. The witness is the validator's construction and is not in the paper.", "location": "Amended Study 2 pre-analysis plan (staged as code/pap/DF_Study2_PAP_Amended.pdf), PDF p. 5 of 6 (pages unnumbered), Section 8, second paragraph, ending on PDF p. 6; online Appendix p. 9 (PDF p. 9), note to Figure OA2a, and p. 18 (PDF p. 18), note to Figure OA3; related sentence in the article, p. 6 (PDF p. 6), Section 2.2" }, "id": "P034-T1", "text": { "body": "amended Study 2 pre-analysis plan, Section 8, PDF pp. 5-6; Appendix p. 9, Figure OA2a note, and p. 18, Figure OA3 note. The specified randomized finite population has equal completion rates but outcome-dependent missingness and a biased complete-case OLS coefficient, including the nonrejecting sample and the observationally equivalent independent-completion population with a different treatment effect. Probabilities and moments use exact finite rational sums; both OLS uniqueness assertions quantify over real coefficient vectors.", "headline": "Amended Study 2 pre-analysis plan, Section 8 (Missing Data): the rule “If there is no differential attrition, we conclude that missingness occurs at random”, repeated as a justification in the notes to Appendix Figures OA2a and OA3. The witness is the validator's construction and is not in the paper.", "location": "Amended Study 2 pre-analysis plan (staged as code/pap/DF_Study2_PAP_Amended.pdf), PDF p. 5 of 6 (pages unnumbered), Section 8, second paragraph, ending on PDF p. 6; online Appendix p. 9 (PDF p. 9), note to Figure OA2a, and p. 18 (PDF p. 18), note to Figure OA3; related sentence in the article, p. 6 (PDF p. 6), Section 2.2" }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T1" }, { "declarations": [ { "claim_revision_id": "catalog:b056c5d9e1dd32fd15c0eef7bf9d9e16d73cd5a33762a6793d206d42618075dd", "conditional_on": [], "credit_reason": "Matches T2(a)–(e) on an arbitrary probability space. The second moments and joint independence assumptions are in the target itself; no identifying result is assumed. The unrestricted normal-equation conclusion is conditional on being a solution, exactly as in T2(d); nonsingularity is required only for the unique-existence conclusion. Reporting instructions are outside the mathematical component. No change after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P034.corrected_complete_case_identification", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P034/P034_Attrition.lean#L936", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:b056c5d9e1dd32fd15c0eef7bf9d9e16d73cd5a33762a6793d206d42618075dd", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P034-T2", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P034-T2" }, "html": { "body": "

corrected amended Study 2 pre-analysis plan, Section 8, PDF pp. 5-6, and Appendix pp. 9 and 18. Randomization, overlap, finite second moments and independent positive potential completion preserve arm means and covariate laws, identify the OLS treatment coefficient, and give a unique solution when nonsingular; the fixed counterexample shows that equal completion rates do not suffice.

", "headline": "Amended Study 2 pre-analysis plan, Section 8 (Missing Data), second paragraph, as repaired by Sol; the same repair applies to the clause “as there was no differential attrition” in the notes to Appendix Figures OA2a and OA3", "location": "Amended Study 2 pre-analysis plan (staged as code/pap/DF_Study2_PAP_Amended.pdf), PDF p. 5 of 6 (pages unnumbered), Section 8; online Appendix p. 9 (PDF p. 9) and p. 18 (PDF p. 18)" }, "id": "P034-T2", "text": { "body": "corrected amended Study 2 pre-analysis plan, Section 8, PDF pp. 5-6, and Appendix pp. 9 and 18. Randomization, overlap, finite second moments and independent positive potential completion preserve arm means and covariate laws, identify the OLS treatment coefficient, and give a unique solution when nonsingular; the fixed counterexample shows that equal completion rates do not suffice.", "headline": "Amended Study 2 pre-analysis plan, Section 8 (Missing Data), second paragraph, as repaired by Sol; the same repair applies to the clause “as there was no differential attrition” in the notes to Appendix Figures OA2a and OA3", "location": "Amended Study 2 pre-analysis plan (staged as code/pap/DF_Study2_PAP_Amended.pdf), PDF p. 5 of 6 (pages unnumbered), Section 8; online Appendix p. 9 (PDF p. 9) and p. 18 (PDF p. 18)" }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T2" }, { "declarations": [ { "claim_revision_id": "catalog:810791f0e0586dcdd104a9781995f071cb38346534d8386edce986f6707cc846", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P034_Contrasts.lean:108 states the unique real saturated fit (0,0,0,2) for the specified four cell outcomes and both equal-weight contrasts equal to one while the main coefficients are zero, exactly as T3(i)-(iii) requests. The target expressly presents unique interpolation as equivalent to the four-unit least-squares formulation; zero covariate dimension is part of this particular counterexample, not a restriction of a universally quantified claim. The target also explicitly identifies the coefficient/average identity as an interpretation of the printed labels and attributes the construction to the validator. Consequently the theorem's failure to refute the actual table entries or a literally printed identity is not a fidelity defect in this row.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P034.factorial_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P034/P034_Contrasts.lean#L108", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:810791f0e0586dcdd104a9781995f071cb38346534d8386edce986f6707cc846", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P034-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P034-T3" }, "html": { "body": "

online Appendix pp. 13-14, Tables OA5-OA6. The four specified factorial cells have a unique saturated interpolation with both main coefficients zero and both equally weighted contrasts one, separating the averaged-effect interpretation from the printed row labels.

", "headline": "Tables OA5 and OA6, first rows (“Information Treatment Effect”, “Deepfake Treatment Effect”), read as effects averaged over the other randomized factor. The witness is the validator's construction and is not in the paper.", "location": "Online Appendix p. 13 (PDF p. 13), Table OA5, and p. 14 (PDF p. 14), Table OA6; the article, p. 9 (PDF p. 9), Section 3.1, refers the reader to these Appendix sections" }, "id": "P034-T3", "text": { "body": "online Appendix pp. 13-14, Tables OA5-OA6. The four specified factorial cells have a unique saturated interpolation with both main coefficients zero and both equally weighted contrasts one, separating the averaged-effect interpretation from the printed row labels.", "headline": "Tables OA5 and OA6, first rows (“Information Treatment Effect”, “Deepfake Treatment Effect”), read as effects averaged over the other randomized factor. The witness is the validator's construction and is not in the paper.", "location": "Online Appendix p. 13 (PDF p. 13), Table OA5, and p. 14 (PDF p. 14), Table OA6; the article, p. 9 (PDF p. 9), Section 3.1, refers the reader to these Appendix sections" }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T3" }, { "declarations": [ { "claim_revision_id": "catalog:3d1b895e8614de5ba6c32a9852d9d880363a17565e3567ae8e8e863c7087df95", "conditional_on": [], "credit_reason": "Harmless strengthening with no excluded target cases. Weights are arbitrary reals in the target already, and the positive-weight equivalences correctly handle the exceptional zero-weight case. This is fitted-function algebra, not a causal identification theorem or a verification of empirical intervals. The claim-stage verdict is unchanged.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P034.corrected_factorial_contrasts", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P034/P034_Contrasts.lean#L46", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:3d1b895e8614de5ba6c32a9852d9d880363a17565e3567ae8e8e863c7087df95", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P034-T4", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P034-T4" }, "html": { "body": "

online Appendix pp. 13-14, corrected Tables OA5-OA6 labels. The fitted interaction model has the stated reference and weighted contrasts, with an average equal to its main coefficient exactly when the weighted interaction is zero, or, for positive weights, when the interaction is zero.

", "headline": "Tables OA5 and OA6, titles, first-row labels and notes, as repaired by Sol", "location": "Online Appendix p. 13 (PDF p. 13), Table OA5, and p. 14 (PDF p. 14), Table OA6" }, "id": "P034-T4", "text": { "body": "online Appendix pp. 13-14, corrected Tables OA5-OA6 labels. The fitted interaction model has the stated reference and weighted contrasts, with an average equal to its main coefficient exactly when the weighted interaction is zero, or, for positive weights, when the interaction is zero.", "headline": "Tables OA5 and OA6, titles, first-row labels and notes, as repaired by Sol", "location": "Online Appendix p. 13 (PDF p. 13), Table OA5, and p. 14 (PDF p. 14), Table OA6" }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T4" }, { "declarations": [ { "claim_revision_id": "catalog:99d675fde73c4d0cd26d355c5cd738f77f7d146457e46370258634d147c9d101", "conditional_on": [], "credit_reason": "Exact match to T5(i)–(iv), including all 39 corrected cells and all count totals. The half-open rounding interval is the target’s own convention, and total division has no effect because the denominators are positive. The claim-stage verdict is unchanged.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P034.corrected_oa3_table", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P034/P034_Rounding.lean#L46", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:99d675fde73c4d0cd26d355c5cd738f77f7d146457e46370258634d147c9d101", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P034-T5", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P034-T5" }, "html": { "body": "

online Appendix p. 11, Table OA3, corrected Total column. The two corrected percentages round to 9.2 and 0.8, the two printed values fail the rounding rule, and all 39 corrected category percentages and all row and column count totals satisfy the stated arithmetic.

", "headline": "Table OA3 (Study 2 - Hand-coded Romney facts across conditions), Total column, rows “Subjective Evaluation of Character” and “Stance on Other Policies”, corrected", "location": "Online Appendix p. 11 (PDF p. 11)" }, "id": "P034-T5", "text": { "body": "online Appendix p. 11, Table OA3, corrected Total column. The two corrected percentages round to 9.2 and 0.8, the two printed values fail the rounding rule, and all 39 corrected category percentages and all row and column count totals satisfy the stated arithmetic.", "headline": "Table OA3 (Study 2 - Hand-coded Romney facts across conditions), Total column, rows “Subjective Evaluation of Character” and “Stance on Other Policies”, corrected", "location": "Online Appendix p. 11 (PDF p. 11)" }, "url": "https://pmaronow.github.io/llm-audit/works/P034/#P034-T5" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "full", "targets": [ { "agreement": "agrees_in_part", "location": "p. 8 (PDF p. 8), associated text p. 7", "method": "I translated the Stata analysis into R. The effect, response shares, Cohen's d and complete-case p-value agree to rounding, and the plotted interval agrees visually. The printed $p<0.001$ differs from $p=0.00304$ under the appendix's stated worst-case attrition procedure; the effect remains significant at 5%.", "status": "reproduced", "target": "Figure 2, left panel: Study 1 deepfake video" }, { "agreement": "to_rounding", "location": "p. 8 (PDF p. 8), footnote 16", "method": "I translated the Stata analysis into R. The effect of 0.726, its complete-case $p$-value, response shares and Cohen's $d$ agree to rounding; the plotted interval agrees visually.", "status": "reproduced", "target": "Figure 2, middle panel: Study 1 real video" }, { "agreement": "to_rounding", "location": "p. 8 (PDF p. 8), footnote 18", "method": "I translated the Stata analysis into R. The effect of 0.264, its $p$-value, response shares and Cohen's $d$ agree to rounding; the plotted interval agrees visually.", "status": "reproduced", "target": "Figure 2, right panel: Study 2 real Romney video" }, { "agreement": "to_rounding", "location": "p. 8 (PDF p. 8), paragraph after Figure 2 and footnote 17", "method": "I translated the Stata analysis into R. The interaction of $-0.214$ and its $p$-value of 0.191 agree with the printed direction and $p=0.19$; I did not compute a separate interaction test with attrition bounds.", "status": "reproduced", "target": "No improvement in discernment: treatment interaction" }, { "agreement": "to_rounding", "location": "pp. 8-9 (PDF pp. 8-9), footnote 19", "method": "I translated the Stata analysis into R. The effects of 0.398 and 0.125 scale points and 6.0 percentage points, and their $p$-values, agree with the printed values to rounding; these regressions omit attrition bounds.", "status": "reproduced", "target": "Pooled effects on belief and uncertainty" }, { "agreement": "to_rounding", "location": "p. 9 (PDF p. 9), paragraph 2 and footnote 20; refers to Appendix Table OA3", "method": "I translated the Stata analysis into R and regenerated the conservative hand codes. The 3.1-percentage-point reduction in pro-choice associations and its $p$-value agree to rounding, as does $p=0.95$ for receiving a pro-choice or pro-life code. These comparisons use the original codes before the recodes proposed in P034-E007.", "status": "reproduced", "target": "Study 2 open-ended policy associations" }, { "agreement": "to_rounding", "location": "p. 9 (PDF p. 9), paragraph 2", "method": "I translated the Stata analysis into R. Reversing the sign of the coded disbelief outcome gives a 14.1-percentage-point decrease in true responses; the effect and its $p$-value agree to rounding.", "status": "reproduced", "target": "Study 2 closed-ended policy knowledge" } ] }, "summary": "The paper asks whether warning voters about deepfakes makes them disbelieve real political videos. It reports two online survey experiments, one with a fictional politician and one with a real video of Mitt Romney. A warning above the video raised the belief that it was fake, whether it was real or a deepfake, and no improvement in telling the two apart was detected. 18 entries are classified as errors (18 Minor), and 1 is a clarity note. Five possible errors were not confirmed, and three remain undetermined.", "what_was_checked": "I checked the following results, which held up with the qualifications noted below.\n\\begin{itemize}\n\\item Table 1 and attrition in Study 1 (p.~5). The cell sizes 347, 349, 349 and 351 and the total of 1,396 follow from the Study 1 raw data. The one IP address that was randomized three times is the only duplicate removed, as Section 2 of the Appendix says. For the test of attrition, I find a Pearson statistic of $11.31$ on 3 degrees of freedom, with $p=0.010$. Attrition is 4.9\\% in the arm with the real video and no warning, against 1.8\\% (19 of 1,049) in the other three arms.\n\\item The main estimates of Study 1 (pp.~7--8, Figure 2). The means are $-0.408$ and $0.102$ for the deepfake video and $-0.485$ and $0.235$ for the real video. The adjusted effects are $0.499$ and $0.726$, each with robust standard error $0.116$. The intervals with Manski bounds, $[0.121,0.855]$ and $[0.288,1.100]$, agree with Figure 2. Both effects stay significant at the 5\\% level under the adverse imputation. The estimates change little with the covariates: $0.497$ and $0.719$ with the planned covariates, and $0.511$ and $0.720$ with none. The interaction of the two treatments is $-0.214$ with standard error $0.163$ and $p=0.191$, as in footnote 17. The printed $p<0.001$ for the deepfake video is the subject of \\fid{P034-E003}.\n\\item The response shares on pp.~7--8 agree with Tables OA4a and OA4c and, for Study 1, with the raw data. In Tables OA4a--OA4c all counts, sums, percentages and totals recompute. Cohen's $d$ in footnotes 14, 16 and 18 recomputes as $0.337$, $0.480$ and $0.197$.\n\\item Footnote 8 and Table OA2: every count, row percentage and total follows from the raw data, as do the shares 29.9\\%, 27.1\\%, 15.8\\% and 27.2\\% and Cramer's $V=0.082$. The $p$-value printed in the note to the table is corrected in \\fid{P034-E015}.\n\\item Section 3 of the Appendix and Figure OA1: the mean of $-0.48$ for the real video, the adjusted difference of $0.062$, and the intervals $[-0.384,0.469]$, $[-0.169,0.187]$ and $[-0.123,0.085]$ for the three outcomes. The wording drawn from these numbers is the subject of \\fid{P034-E002}. The Study 1 panels of Figures OA2a and OA2b and the Study 1 points of Figure OA3 also recompute. Which observations the bounding program replaces is the subject of \\fid{P034-E005}.\n\\item Table OA1: the Study 1 column follows from the raw data, and the printed median income band of Study 1 is correct for the data before imputation. The Study 2 and pooled columns agree with the stored workbook. The postgraduate row is the subject of \\fid{P034-E009}.\n\\item Table OA3: the column totals are $2{,}898=3\\times966$, $2{,}877=3\\times959$ and $5{,}775=3\\times1{,}925$, and the percentages recompute except in two cells (\\fid{P034-N010}). The drop of 3.1 points follows from 49 of 966 against 19 of 959, and 58 responses with pro-choice or pro-life codes in each arm agree with $p=0.95$; the outcome description and some hand codes are corrected in \\fid{P034-E007}, and the loss of free-response text before hand-coding in \\fid{P034-E010}.\n\\item Tables OA5 and OA6: the printed coefficients and bounds follow from the replication programs. Their labels are the subject of \\fid{P034-A001}. In Tables OA7--OA12 each printed interval is symmetric about its estimate up to rounding, and Tables OA8--OA10 agree with the stored workbook. Tables OA7--OA9, OA11 and OA12 also recompute from the Study 2 data for the samples actually used; the subgroup interpretation is corrected in \\fid{P034-E001} and departures from the plan in \\fid{P034-N011}. The stored values for Tables OA7, OA11 and OA12 are the subject of \\fid{P034-E019}. In Section 5 of the Appendix, the printed $F(34,1795)=1.75$ corresponds to $p=0.005$, which agrees with the printed $p<.01$.\n\\item I computed seven headline analyses again from the public data of both studies, with the Stata programs translated into R. The three effects of Figure 2 are $0.499$, $0.726$ and $0.264$. The pooled effects are $0.398$, $0.125$ and $6.0$ percentage points. The effect on naming a pro-choice position is $-3.1$ points, the test for giving a pro-choice or pro-life code has $p=0.95$, and the effect on the closed question is $-14.1$ points. These agree with the printed values up to rounding; the one difference is the $p$-value treated in \\fid{P034-E003}.\n\\item Design and coding: the 65,000 training iterations are the sum of 50,000 and 15,000 (Section 1 of the Appendix). The warning text in Tables 1 and 2 agrees with the script of Study 2. The restrictions of Study 2 use questions asked before treatment. The final party variable corrects the coding defect that the Appendix acknowledges. The code printed in Section 8 of the Appendix agrees with the replication file, and the closed question is coded as the plan specifies. For the main outcome, the bounding program does what Section 2 of the Appendix describes.\n\\item The pre-analysis plans: the target of 362 participants per arm in Study 1 follows from the stated standard deviation and effect. The amendment to the Study 2 plan adds only the last exclusion by date. The rule of that plan for missing data is the subject of \\fid{P034-E017}.\n\\end{itemize}\nThe paper contains no theorem, proof or derivation. The checks had the following limits. Study 1 was rebuilt from its raw file. Beyond the seven headline analyses, I used the public Study 2 data to check free-response text and coding, subgroup and alternative trust effects, and the duplicate-IP and election exclusions. Other supplementary Study 2 results were compared only with the printed tables, the figures and the stored workbooks. The seven headline analyses use an R translation of the authors' programs, and the later Study 2 checks use Python translations rather than native Stata execution; not all supplementary analyses were recalculated. The intended population for counting duplicate IP addresses remains unclear. The two videos of Study 1 were not viewed. For the video of Study 2, I inspected the file information and two frames and read the transcript. Of the works cited, I read only those needed for particular questions: Wittenberg et al.\\ (2021), Vaccari and Chadwick (2020), Pennycook et al.\\ (2020) and the abstract of Clayton et al.\\ (2020) for \\fid{P034-N001} and \\fid{P034-N002}, and the sources of two reference entries for \\fid{P034-E018}. I could not obtain the 2021 preprint of Barari, Lucas and Munger or the chapter on attrition of Gerber and Green (2012). The other cited works were not consulted, and the returns of the news searches described on p.~2 were not available. The undetermined questions require the approved Study 1 protocol and fielded survey, authenticated question wording for both studies, or a source that I could not obtain. That I list no further corrections is not a finding that the rest of the paper is correct.", "work": { "arxiv_id": null, "audit_complete": true, "authors": [ "John Ternovski", "Joshua Kalla", "P. M. Aronow" ], "category": "Articles - Empirics", "citation": "John Ternovski, Joshua Kalla, and P. M. Aronow (2022). The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments. Journal of Online Trust and Safety 1(2).", "doi": "10.54501/jots.v1i2.28", "edition": { "pages_in_audited_file": 16, "sha256": "698d62dd33fb7b0c1c16ffee884c581db1ca798edb0cbc0db5b77415c47b4a36", "source_url": "https://doi.org/10.54501/jots.v1i2.28", "venue": "Journal of Online Trust and Safety", "year": "2022" }, "id": "P034", "later_materials": null, "title": "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P034/", "venue": "Journal of Online Trust and Safety", "year": "2022" } }